Sarah Sachs

“Notion's auto model does this really well. We have state-of-the-art models available always. But we also have an auto model there at the top that handles about 75% of our traffic, right? We have the ability to switch between models and our product, and we also offer it to our customers … they have access to these models without vendor lock-in.”

Sarah SachsLeads engineering teams for AI, Notion · Leader, Notion

“That expertise on use cases is also very valuable to Frontier Labs. We find that our evals and our Early Access Program partnerships actually help us a lot with Frontier Labs and it's something that we can exchange instead of extraordinary large commits. And I don't think the discount is ever worth the lost in optionality.”

Sarah SachsLeads engineering teams for AI, Notion · Leader, Notion

“And the last thing is CPUs over GPUs. We've recently launched something at Notion called Workers. I don't think that the GPU is necessary for every job. A lot of the jobs that we have are actually serving discrete pieces of code. Like you don't need an LLM to turn a CSV into a PDF. You don't need an LLM to talk to Notion tool calls if we have a CLI. You definitely don't need an LLM to do deterministic SQL queries. This is where people become token poor very quick.”

Sarah SachsLeads engineering teams for AI, Notion · Leader, Notion

“And not all traffic is equal. It is a huge miss to send all of these to the latest Opus model. Some of these absolutely. Large scale data analysis, when you do it on Notion, will recommend Opus, right? When you triage an email inbox, if we're charging you to do that on Opus, we're ripping you off, and ourselves.”

Sarah SachsLeads engineering teams for AI, Notion · Leader, Notion

“This is how it works today internally at Notion. Almost all of our polish and large feedback like this is actually coordinated through our software factories, both in terms of writing to the right teams and also having coding agents take the first step.”

Sarah SachsLeads engineering teams for AI, Notion · Leader, Notion

“We posted this review when we announced our partnership with parallel as our web search provider. If you were to look at just latency of a single call or just cost, parallel might not be the cheapest. But if you have expertise in entire web search trajectories, you'll see how it differs. The granularity of this eval is what lets us make the best decisions for our customers, because we understand all of the trade-offs on entire trajectories”

Sarah SachsLeads engineering teams for AI, Notion · Leader, Notion

“Regression test. Those live in ci, those have to pass a certain percent, you know, within some stochastic error rate. Then we have, as you’re building a product, evals of these aren’t passing right now, and this is launch quality. So we have a report card and we need to, on these categories, you know, be it 80 or 90% of all of these user journeys to launch, and then what we have what we call frontier or headroom evals, where we actively wanna be at 30% pass rate.”

Sarah SachsManages Notion's core AI capabilities and infrastructure team · Leader, Notion

“we’ll all sit down every Friday and look at the P 99 of like the most token exhaustive custom agent transcript and just look at why it didn’t do well and cut a bunch of tasks.”

Sarah SachsManages Notion's core AI capabilities and infrastructure team · Leader, Notion

“Notions last exam is a big thing inside the company and we have people, full-time staff to it exclusively. Mm. We have a data scientist, a model behavior engineer, and an full-time, um, evals engineer just dedicated to the evals that we pass 30% of the time.”

Sarah SachsManages Notion's core AI capabilities and infrastructure team · Leader, Notion