How do organizations choose and switch AI models?

“Fine-tuning Gemma4-E2B on hundreds of thousands of domain-specific examples lifts its F1 from 0.563 to 0.902, on par with the best proprietary models, while median latency stays at 35ms.”

AirbnbPublished by the organization itself (post), no named author

“Proprietary models only win at low traffic: below roughly 15 to 30 RPM, Claude Sonnet 5.5 and GPT-6.1 Sol are cheaper than a dedicated GPU.”

AirbnbPublished by the organization itself (post), no named author

“Notion's auto model does this really well. We have state-of-the-art models available always. But we also have an auto model there at the top that handles about 75% of our traffic, right? We have the ability to switch between models and our product, and we also offer it to our customers … they have access to these models without vendor lock-in.”

Sarah SachsLeads engineering teams for AI, Notion · Leader, Notion

“we now have a number of distilled models where we take a teacher model, usually a big frontier model, then teach a specific use case to a smaller model”

Harley FinkelsteinPresident · Executive, Shopify

“GLOWC, an internal framework used by engineers at Airbnb, makes it easy to fine-tune open-weight language (and vision-language) models into fast, accurate classifiers. In this blog post, we put one of these classifiers head-to-head with open-weight and proprietary models and find that it matches frontier-level quality in under 100ms at roughly 1% of the cost.”

AirbnbPublished by the organization itself (post), no named author

“You have labeled data. Examples teach distinctions and edge cases that instructions struggle to spell out. Hundreds of thousands of in-domain examples did what a 2,700 word prompt couldn’t.”

AirbnbPublished by the organization itself (post), no named author

“Our platform is provider agnostic by definition. So if we come back to the diagram, the little blue … green agent box there can be very easily swapped for any model. We can dynamically choose which model best fits the purposes that we have. So we are not specifically tied to any provider by definition”

Felippe Felisola CasoMonzo (conference affiliation, 2026) · Practitioner, Monzo

“In a later randomized split, Opus 5.5 at medium showed no measurable change in review outcomes or critical findings compared with Opus 5, so we switched.”

Ryan SkidmoreAuthor of Cloudflare's "Orchestrating AI Code Review at scale" blog post · Practitioner, Cloudflare

“Today, we have 800 plus projects internally going through this, cumulatively handling more than 100 million model requests per day. This includes both the frontier models and also open source models.”

Uday Kiran MedisettyDistinguished Engineer, Uber · Practitioner, Uber

“The planning agent is responsible for the actual thinking, so it should run on your most powerful model. I initially used Fable for this purpose when it was available, as it excels in UI/UX design. When Fable was no longer accessible, I switched to Opus. Since the planner writes the specifications, the implementation can be handled by a more cost-effective and faster model like Sonnet.”

Noah BlauensteinerAuthor, Xebia articles · Practitioner, Xebia

“And not all traffic is equal. It is a huge miss to send all of these to the latest Opus model. Some of these absolutely. Large scale data analysis, when you do it on Notion, will recommend Opus, right? When you triage an email inbox, if we're charging you to do that on Opus, we're ripping you off, and ourselves.”

Sarah SachsLeads engineering teams for AI, Notion · Leader, Notion

“And since all the requests go through one place, we could also add like audit logging, session traces, and a whole bunch of infrastructure around this. Every model request at Uber goes through this. So there's no model request that doesn't go through this. So that gives us like one single entry point for across all the models.”

Uday Kiran MedisettyDistinguished Engineer, Uber · Practitioner, Uber

“Ultimately, we opted to build the agent loop, tools, and the system for gathering context ourselves, and we connect directly to model providers instead of relying on a broad agent framework. That's because a framework we initially explored lagged behind the provider APIs we wanted to use and included more structure than Reclaim needed.”

DropboxPublished by the organization itself (post), no named author

“And that's why we have Cursor Bench. We have this private eval set that is mostly made up of things that happen in our code base, which is held out from the … So we ensure that the models aren't trained on it. And it's based on those real world engineering tasks.”

Lee RobinsonMachine Learning Engineer, Model Behavior, Cursor · Practitioner, Cursor (Anysphere)

“And then the self review, which is kind of interesting because we have this, we created this review pool. So we have one model, which is a Sonnet that does one review. … then we have kind of a … cheaper model that also does a review. And then we have the third model, … right now, it kind of consolidates this and verifies the findings in the reviews and decides if it should change something or not. And when this review pool is kind of finished and happy, … will assign it to a human.”

Hallstein BrøtanSenior consultant, Novanet; developer on Nordic Corporate Bank's system · Practitioner, Nordic Corporate Bank