we found the most effective route to improve outcomes was simply brute force: retry steps multiple times until they passed or we reached a limit.
Charles Covey-BrandtAuthor of the Airbnb engineering blog post on LLM test migration (role not stated on the page) · Practitioner, 
Airbnb
Post: Accelerating Large-Scale Test Migration with LLMs, Airbnb Engineering & Data Science (own site), March 13, 2025, at 2. Retry Loops & Dynamic Prompting.
Charles Covey-Brandt, Author of the Airbnb engineering blog post on LLM test migration (role not stated on the page). Wrote the post describing the migration the employer carried out, so the account is first-hand; it is also Airbnb's own account of its project.
Checked against the source by a first reader, then confirmed independently by a second reader who fetched the source without seeing the first reader's notes (Independent review agent (Claude Sonnet), blind second pass from passage text and citation only, no repository access, 2026-11-04).
Exact match in the Retry Loops & Dynamic Prompting section of the Airbnb post, bylined Charles Covey-Brandt.
Charles Covey-Brandt (Airbnb), “Accelerating Large-Scale Test Migration with LLMs”, Airbnb Engineering & Data Science (own site), 2025-03-13, at 2. Retry Loops & Dynamic Prompting. https://airbnb.tech/?p=366 Archived at https://the-ai-record.com/passages/airbnb-found-most-effective-route-improve-outcomes-2025
when we ran our first bulk run, we successfully migrated 75% of our target files in just four hours.
Charles Covey-BrandtAuthor of the Airbnb engineering blog post on LLM test migration (role not stated on the page) · Practitioner, Airbnb
Post: Accelerating Large-Scale Test Migration with LLMs, Airbnb Engineering & Data Science (own site), March 13, 2025, at 3. Increasing the Context.
the main success driver we saw was choosing the right related files (finding nearby files, good example files from the same project, filtering the dependencies for files that were relevant to the component, etc.), rather than getting the prompt engineering perfect.
Charles Covey-BrandtAuthor of the Airbnb engineering blog post on LLM test migration (role not stated on the page) · Practitioner, Airbnb
Post: Accelerating Large-Scale Test Migration with LLMs, Airbnb Engineering & Data Science (own site), March 13, 2025, at 3. Increasing the Context.
We'd originally estimated this would take 1.5 years of engineering time to do by hand, but — using a combination of frontier models and robust automation — we finished the entire migration in just 6 weeks.
Charles Covey-BrandtAuthor of the Airbnb engineering blog post on LLM test migration (role not stated on the page) · Practitioner, Airbnb
Post: Accelerating Large-Scale Test Migration with LLMs, Airbnb Engineering & Data Science (own site), March 13, 2025.
we reached 97% of the 3.5K original Enzyme files
Charles Covey-BrandtAuthor of the Airbnb engineering blog post on LLM test migration (role not stated on the page) · Practitioner, Airbnb
Post: Accelerating Large-Scale Test Migration with LLMs, Airbnb engineering blog, March 2025.
Something wrong? Report a correction. JSON: /api/v1/passages/airbnb-found-most-effective-route-improve-outcomes-2025.json