we’ll all sit down every Friday and look at the P 99 of like the most token exhaustive custom agent transcript and just look at why it didn’t do well and cut a bunch of tasks.
Sarah SachsManages Notion's core AI capabilities and infrastructure team · Leader, 
Notion
Podcast: Notion's Token Town: 5 Rebuilds, 100+ Tools, MCP vs CLIs and the Software Factory Future (Latent Space), Latent Space, April 15, 2026, at 00:09:31. Reported by Swyx.
Sarah Sachs, Manages Notion's core AI capabilities and infrastructure team. Head of AI at Notion, which makes the Custom Agents she discusses; Notion sells AI features and has reason to present them favorably.
Notion sells AI or agent tools or services, so it has a reason to publish flattering numbers about AI. The passage stays in the archive; read it with that in mind.
Checked against the source by a first reader, then confirmed independently by a second reader who fetched the source without seeing the first reader's notes (Independent review agent (Claude Sonnet), blind second pass from passage text and citation only, no repository access, 2026-11-03).
Sarah Sachs speaking at [00:09:31]. The passage starts mid-sentence at 'we'll all sit down every Friday' and matches exactly. The source's lowercase 'we'll' and the 'P 99' spelling are preserved.
Sarah Sachs (Notion), “Notion's Token Town: 5 Rebuilds, 100+ Tools, MCP vs CLIs and the Software Factory Future (Latent Space)”, Latent Space, 2026-04-15, at 00:09:31. https://www.latent.space/p/notion Archived at https://the-ai-record.com/passages/notion-friday-review-p99-token-heavy-agent-2026
basically it just means can you create sort of like a, as automated as possible, a workflow for developing debugging. Mm-hmm. Merging, reviewing, and maintaining a code base
Simon LastCo-founder and head of AI · Executive, Notion
Interview: Notion's Token Town: 5 Rebuilds, 100+ Tools, MCP vs CLIs and the Software Factory Future (Latent Space), Latent Space (independent AI engineering podcast), April 15, 2026.
in the latest version of the agent, I, it’s not in custom agents yet, but it will be like, like next week, a week after or so, um, there’s now like over a hundred tools. Just for all, all the crazy notion stuff.
Simon LastCo-founder and head of AI · Executive, Notion
Podcast: Notion's Token Town: 5 Rebuilds, 100+ Tools, MCP vs CLIs and the Software Factory Future (Latent Space), Latent Space, April 15, 2026, at 00:54:57. Reported by Swyx.
Literally over 30 of them
Simon LastCo-founder and head of AI · Executive, Notion
Interview: Notion's Token Town: 5 Rebuilds, 100+ Tools, MCP vs CLIs and the Software Factory Future (Latent Space), Latent Space (independent AI engineering podcast), April 15, 2026.
Notions last exam is a big thing inside the company and we have people, full-time staff to it exclusively. Mm. We have a data scientist, a model behavior engineer, and an full-time, um, evals engineer just dedicated to the evals that we pass 30% of the time.
Sarah SachsManages Notion's core AI capabilities and infrastructure team · Leader, Notion
Podcast: Notion's Token Town: 5 Rebuilds, 100+ Tools, MCP vs CLIs and the Software Factory Future (Latent Space), Latent Space, April 15, 2026, at 00:24:23. Reported by Swyx.
Regression test. Those live in ci, those have to pass a certain percent, you know, within some stochastic error rate. Then we have, as you’re building a product, evals of these aren’t passing right now, and this is launch quality. So we have a report card and we need to, on these categories, you know, be it 80 or 90% of all of these user journeys to launch, and then what we have what we call frontier or headroom evals, where we actively wanna be at 30% pass rate.
Sarah SachsManages Notion's core AI capabilities and infrastructure team · Leader, Notion
Podcast: Notion's Token Town: 5 Rebuilds, 100+ Tools, MCP vs CLIs and the Software Factory Future (Latent Space), Latent Space, April 15, 2026, at 00:24:23. Reported by Swyx.
Something wrong? Report a correction. JSON: /api/v1/passages/notion-friday-review-p99-token-heavy-agent-2026.json