{
  "apiVersion": "v1",
  "topic": {
    "id": "ai-evals",
    "title": "How do organizations test whether their AI works?",
    "group": "Tools, models and knowledge",
    "summary": "Evaluation sets, benchmarks and reviews of agent transcripts that organizations use to judge their own AI systems.",
    "passages": 19,
    "url": "https://the-ai-record.com/topics/ai-evals",
    "json": "https://the-ai-record.com/api/v1/topics/ai-evals.json"
  },
  "passages": [
    {
      "id": "anthropic-past-incidents-become-daily-eval-for-reviewer-2026",
      "text": "Any incidents are now an eval. The team collected past PRs that caused incidents, with their fix-forwards and reverts, and had engineers who know that code confirm each bug and where it lived. The reviewer runs against that set daily.",
      "context": null,
      "speaker": null,
      "speakerId": null,
      "speakerUrl": null,
      "speakerLevel": null,
      "role": null,
      "voice": "organization",
      "organization": "Anthropic",
      "organizationId": "anthropic",
      "organizationUrl": "https://the-ai-record.com/organizations/anthropic",
      "aiVendor": true,
      "work": {
        "title": "How the Claude Code team does code review",
        "type": "post",
        "date": "2026-10-15",
        "publisher": "claude.dev Blog (Anthropic)",
        "locator": "How we make the reviewer better",
        "url": "https://claude.dev/blog/solving-code-review-at-anthropic/"
      },
      "grade": "B",
      "gradeMeaning": "The organization's own post, paper or talk.",
      "verification": {
        "status": "verified",
        "meaning": "An independent second reader re-fetched the source and confirmed the wording.",
        "reviewedOn": "2026-11-07"
      },
      "topics": [
        "ai-evals",
        "ai-review-quality"
      ],
      "functions": [
        "engineering"
      ],
      "figures": [],
      "url": "https://the-ai-record.com/passages/anthropic-past-incidents-become-daily-eval-for-reviewer-2026",
      "json": "https://the-ai-record.com/api/v1/passages/anthropic-past-incidents-become-daily-eval-for-reviewer-2026.json"
    },
    {
      "id": "boundary-agents-run-baml-programs-inspect-transcripts-humans-triage-2026",
      "text": "What we did was we built a system that actually has agents constantly running and creating BAML programs. … We then look at the entire … transcripts, see what tools they use, see what happened, and obviously we as humans can inspect them. But more importantly, we can have agents go inspect them. And agents find what was good, what was bad. … And we can have humans collaborate with these issues to figure out which ones are real, which ones are hallucinations, which ones don't have taste as much as I hate to use that word. And then we can have agents go ahead and create fixes to these problems and go to address them.",
      "context": "Gupta is describing how the team tests its programming language when it does not rely on reading every line of code.",
      "speaker": "Vaibhav Gupta",
      "speakerId": "vaibhav-gupta",
      "speakerUrl": "https://the-ai-record.com/people/vaibhav-gupta",
      "speakerLevel": "executive",
      "role": "Co-founder and Chief Executive",
      "voice": "person",
      "organization": "Boundary (BoundaryML)",
      "organizationId": "boundary",
      "organizationUrl": "https://the-ai-record.com/organizations/boundary",
      "aiVendor": true,
      "work": {
        "title": "Fighting Slop with Slop",
        "type": "talk",
        "date": "2026-07-31",
        "publisher": "AI Engineer (YouTube)",
        "locator": "5:05",
        "url": "https://www.youtube.com/watch?v=AMiyLItEtLA"
      },
      "grade": "B",
      "gradeMeaning": "The organization's own post, paper or talk.",
      "verification": {
        "status": "verified",
        "meaning": "An independent second reader re-fetched the source and confirmed the wording.",
        "reviewedOn": "2026-11-01"
      },
      "topics": [
        "autonomy-and-oversight",
        "bugs-and-incidents",
        "testing",
        "ai-evals"
      ],
      "functions": [
        "engineering"
      ],
      "figures": [],
      "url": "https://the-ai-record.com/passages/boundary-agents-run-baml-programs-inspect-transcripts-humans-triage-2026",
      "json": "https://the-ai-record.com/api/v1/passages/boundary-agents-run-baml-programs-inspect-transcripts-humans-triage-2026.json"
    },
    {
      "id": "cursor-bench-private-eval-own-codebase-2026",
      "text": "And that's why we have Cursor Bench. We have this private eval set that is mostly made up of things that happen in our code base, which is held out from the … So we ensure that the models aren't trained on it. And it's based on those real world engineering tasks.",
      "context": "Robinson has just said public benchmarks are an imperfect test of how models are used in practice, and is describing Cursor's private evaluation set for its own models.",
      "speaker": "Lee Robinson",
      "speakerId": "lee-robinson",
      "speakerUrl": "https://the-ai-record.com/people/lee-robinson",
      "speakerLevel": "practitioner",
      "role": "Machine Learning Engineer, Model Behavior, Cursor",
      "voice": "person",
      "organization": "Cursor (Anysphere)",
      "organizationId": "cursor",
      "organizationUrl": "https://the-ai-record.com/organizations/cursor",
      "aiVendor": true,
      "work": {
        "title": "Recursive Model Improvement",
        "type": "talk",
        "date": "2026-07-15",
        "publisher": "AI Engineer (YouTube)",
        "locator": "8:12",
        "url": "https://www.youtube.com/watch?v=q4Tr-DknG2M"
      },
      "grade": "B",
      "gradeMeaning": "The organization's own post, paper or talk.",
      "verification": {
        "status": "verified",
        "meaning": "An independent second reader re-fetched the source and confirmed the wording.",
        "reviewedOn": "2026-11-03"
      },
      "topics": [
        "measuring-impact",
        "choosing-models",
        "ai-evals"
      ],
      "functions": [
        "engineering"
      ],
      "figures": [],
      "url": "https://the-ai-record.com/passages/cursor-bench-private-eval-own-codebase-2026",
      "json": "https://the-ai-record.com/api/v1/passages/cursor-bench-private-eval-own-codebase-2026.json"
    },
    {
      "id": "notion-regression-launch-quality-frontier-evals-30-2026",
      "text": "Regression test. Those live in ci, those have to pass a certain percent, you know, within some stochastic error rate. Then we have, as you’re building a product, evals of these aren’t passing right now, and this is launch quality. So we have a report card and we need to, on these categories, you know, be it 80 or 90% of all of these user journeys to launch, and then what we have what we call frontier or headroom evals, where we actively wanna be at 30% pass rate.",
      "context": null,
      "speaker": "Sarah Sachs",
      "speakerId": "sarah-sachs",
      "speakerUrl": "https://the-ai-record.com/people/sarah-sachs",
      "speakerLevel": "leader",
      "role": "Manages Notion's core AI capabilities and infrastructure team",
      "voice": "person",
      "organization": "Notion",
      "organizationId": "notion",
      "organizationUrl": "https://the-ai-record.com/organizations/notion",
      "aiVendor": true,
      "work": {
        "title": "Notion's Token Town: 5 Rebuilds, 100+ Tools, MCP vs CLIs and the Software Factory Future (Latent Space)",
        "type": "podcast",
        "date": "2026-04-15",
        "publisher": "Latent Space",
        "locator": "00:24:23",
        "url": "https://www.latent.space/p/notion"
      },
      "grade": "C",
      "gradeMeaning": "An interview or press quote.",
      "verification": {
        "status": "verified",
        "meaning": "An independent second reader re-fetched the source and confirmed the wording.",
        "reviewedOn": "2026-11-03"
      },
      "topics": [
        "testing",
        "measuring-impact",
        "ai-evals"
      ],
      "functions": [
        "engineering"
      ],
      "figures": [
        {
          "metric": "eval-target-pass-rate",
          "value": 0.3,
          "unit": "share",
          "definition": "pass rate Notion deliberately targets on its frontier or headroom evals",
          "level": "n/a"
        }
      ],
      "url": "https://the-ai-record.com/passages/notion-regression-launch-quality-frontier-evals-30-2026",
      "json": "https://the-ai-record.com/api/v1/passages/notion-regression-launch-quality-frontier-evals-30-2026.json"
    },
    {
      "id": "coatue-compartmentalize-evaluate-what-im-hill-climbing-against-2026",
      "text": "what can I actually compartmentalize in a way that we can understand and evaluate? Because that's what I'm hill climbing against, right? Otherwise, I'm just tuning these things.",
      "context": "Long is describing how to break an investing process into parts that can each be tested.",
      "speaker": "Frank Long",
      "speakerId": "frank-long",
      "speakerUrl": "https://the-ai-record.com/people/frank-long",
      "speakerLevel": "practitioner",
      "role": "Works at Coatue",
      "voice": "person",
      "organization": "Coatue",
      "organizationId": "coatue",
      "organizationUrl": "https://the-ai-record.com/organizations/coatue",
      "aiVendor": false,
      "work": {
        "title": "AI Agents Will Upend Investing Even More Than Coding - Frank Long, Coatue",
        "type": "talk",
        "date": "2026-11-06",
        "publisher": "AI Engineer (YouTube)",
        "locator": "11:47",
        "url": "https://www.youtube.com/watch?v=AYFcmnQrZCg"
      },
      "grade": "B",
      "gradeMeaning": "The organization's own post, paper or talk.",
      "verification": {
        "status": "verified",
        "meaning": "An independent second reader re-fetched the source and confirmed the wording.",
        "reviewedOn": "2026-11-06"
      },
      "topics": [
        "measuring-impact",
        "ai-evals"
      ],
      "functions": [
        "company-wide"
      ],
      "figures": [],
      "url": "https://the-ai-record.com/passages/coatue-compartmentalize-evaluate-what-im-hill-climbing-against-2026",
      "json": "https://the-ai-record.com/api/v1/passages/coatue-compartmentalize-evaluate-what-im-hill-climbing-against-2026.json"
    },
    {
      "id": "vercel-shadow-best-sdr-six-weeks-90th-percentile-2026",
      "text": "We had a GTM engineer, as I mentioned, shadow one of our best SDRs for a few days. … Watching every single tab she opened, … The engineer converted each of those data sources into a step in a tool calling workflow. … After building our first version, … ran in production in shadow mode for six weeks. Our best SDR reviewed every single output to create a feedback flywheel. And then finally, she couldn't improve it anymore. The architecture reflects exactly what she was doing manually. But now, it performs like the 90th percentile rep, 100% of the time. We applied the same framework to 30 different SDR workflows, … SDR quotas increased by 30% this quarter.",
      "context": "Vercel's COO describes how her team built the lead qualification agent, step by step, and what followed.",
      "speaker": "Jeanne DeWitt Grosser",
      "speakerId": "jeanne-dewitt-grosser",
      "speakerUrl": "https://the-ai-record.com/people/jeanne-dewitt-grosser",
      "speakerLevel": "executive",
      "role": "Chief Operating Officer, Vercel",
      "voice": "person",
      "organization": "Vercel",
      "organizationId": "vercel",
      "organizationUrl": "https://the-ai-record.com/organizations/vercel",
      "aiVendor": true,
      "work": {
        "title": "Going from 10 to 1 SDRs. How We Deployed AI Across our Sales Org with Vercel's COO",
        "type": "talk",
        "date": "2026-11-04",
        "publisher": "SaaStr AI (YouTube); talk at SaaStr AI Annual 2026",
        "locator": "15:00",
        "url": "https://www.youtube.com/watch?v=wXHLbr4W-8s"
      },
      "grade": "B",
      "gradeMeaning": "The organization's own post, paper or talk.",
      "verification": {
        "status": "verified",
        "meaning": "An independent second reader re-fetched the source and confirmed the wording.",
        "reviewedOn": "2026-11-08"
      },
      "topics": [
        "sales-and-marketing",
        "ai-evals"
      ],
      "functions": [
        "sales-marketing"
      ],
      "figures": [
        {
          "metric": "sdr-agent-shadow-period",
          "value": 6,
          "unit": "weeks",
          "period": "first version, as spoken",
          "definition": "time the first version ran in production in shadow mode while the best SDR reviewed every output",
          "level": "human-approves"
        },
        {
          "metric": "sdr-workflows-automated",
          "value": 30,
          "unit": "workflows",
          "period": "as of the talk, 2026",
          "definition": "SDR workflows built with the same framework, as the speaker counts them",
          "level": "unclear"
        },
        {
          "metric": "sdr-quota-increase",
          "value": 0.3,
          "unit": "share",
          "period": "this quarter, as spoken",
          "definition": "increase in SDR quotas this quarter; the speaker does not say whether this is a target or attainment",
          "level": "n/a"
        }
      ],
      "url": "https://the-ai-record.com/passages/vercel-shadow-best-sdr-six-weeks-90th-percentile-2026",
      "json": "https://the-ai-record.com/api/v1/passages/vercel-shadow-best-sdr-six-weeks-90th-percentile-2026.json"
    },
    {
      "id": "airbnb-fine-tuning-gemma4-e2b-lifts-f1-2026",
      "text": "Fine-tuning Gemma4-E2B on hundreds of thousands of domain-specific examples lifts its F1 from 0.563 to 0.902, on par with the best proprietary models, while median latency stays at 35ms.",
      "context": null,
      "speaker": null,
      "speakerId": null,
      "speakerUrl": null,
      "speakerLevel": null,
      "role": null,
      "voice": "organization",
      "organization": "Airbnb",
      "organizationId": "airbnb",
      "organizationUrl": "https://the-ai-record.com/organizations/airbnb",
      "aiVendor": false,
      "work": {
        "title": "Frontier quality at 1% of the cost: a production benchmark of LLM classifiers",
        "type": "post",
        "date": "2026-11-03",
        "publisher": "Airbnb Tech blog",
        "locator": "Quality vs. latency",
        "url": "https://airbnb.tech/ai-ml/frontier-quality-at-1-of-the-cost-a-production-benchmark-of-llm-classifiers/"
      },
      "grade": "B",
      "gradeMeaning": "The organization's own post, paper or talk.",
      "verification": {
        "status": "verified",
        "meaning": "An independent second reader re-fetched the source and confirmed the wording.",
        "reviewedOn": "2026-11-05"
      },
      "topics": [
        "measuring-impact",
        "internal-tools",
        "choosing-models",
        "ai-evals"
      ],
      "functions": [
        "engineering"
      ],
      "figures": [
        {
          "metric": "f1-after-fine-tuning",
          "value": 0.902,
          "unit": "score",
          "period": "2026 benchmark",
          "definition": "max F1 of fine-tuned Gemma4-E2B on a held-out set of 10,000 weighted examples, up from 0.563 before fine-tuning",
          "level": "n/a"
        }
      ],
      "url": "https://the-ai-record.com/passages/airbnb-fine-tuning-gemma4-e2b-lifts-f1-2026",
      "json": "https://the-ai-record.com/api/v1/passages/airbnb-fine-tuning-gemma4-e2b-lifts-f1-2026.json"
    },
    {
      "id": "ramp-built-benchmark-200-credit-analyst-2026",
      "text": "We built a benchmark of 200 credit analyst-reviewed income statements and balance sheets, with the analyst's figures recorded for every metric and time period.",
      "context": null,
      "speaker": null,
      "speakerId": null,
      "speakerUrl": null,
      "speakerLevel": null,
      "role": null,
      "voice": "organization",
      "organization": "Ramp",
      "organizationId": "ramp",
      "organizationUrl": "https://the-ai-record.com/organizations/ramp",
      "aiVendor": false,
      "work": {
        "title": "Agentic Credit Analysis",
        "type": "post",
        "date": "2026-10-22",
        "publisher": "Ramp Builders blog",
        "locator": "Evaluation dataset",
        "url": "https://builders.ramp.com/post/credit-analysis-using-semantics"
      },
      "grade": "B",
      "gradeMeaning": "The organization's own post, paper or talk.",
      "verification": {
        "status": "verified",
        "meaning": "An independent second reader re-fetched the source and confirmed the wording.",
        "reviewedOn": "2026-11-07"
      },
      "topics": [
        "ai-evals"
      ],
      "functions": [
        "finance",
        "risk"
      ],
      "figures": [
        {
          "metric": "benchmark-statements",
          "value": 200,
          "unit": "count",
          "period": "as of 2026-10-22",
          "definition": "Credit analyst-reviewed income statements and balance sheets used to benchmark the agent",
          "level": "unclear"
        }
      ],
      "url": "https://the-ai-record.com/passages/ramp-built-benchmark-200-credit-analyst-2026",
      "json": "https://the-ai-record.com/api/v1/passages/ramp-built-benchmark-200-credit-analyst-2026.json"
    },
    {
      "id": "cloudflare-later-randomized-split-opus-55-medium-2026",
      "text": "In a later randomized split, Opus 5.5 at medium showed no measurable change in review outcomes or critical findings compared with Opus 5, so we switched.",
      "context": null,
      "speaker": "Ryan Skidmore",
      "speakerId": "ryan-skidmore",
      "speakerUrl": "https://the-ai-record.com/people/ryan-skidmore",
      "speakerLevel": "practitioner",
      "role": "Author of Cloudflare's \"Orchestrating AI Code Review at scale\" blog post",
      "voice": "person",
      "organization": "Cloudflare",
      "organizationId": "cloudflare",
      "organizationUrl": "https://the-ai-record.com/organizations/cloudflare",
      "aiVendor": true,
      "work": {
        "title": "Cutting our inference bill by 68% by being obsessive about optimization",
        "type": "post",
        "date": "2026-10-19",
        "publisher": "Cloudflare blog",
        "locator": "Changing models and effort",
        "url": "https://blog.cloudflare.com/obsessive-inference-optimization/"
      },
      "grade": "B",
      "gradeMeaning": "The organization's own post, paper or talk.",
      "verification": {
        "status": "verified",
        "meaning": "An independent second reader re-fetched the source and confirmed the wording.",
        "reviewedOn": "2026-11-07"
      },
      "topics": [
        "choosing-models",
        "ai-evals"
      ],
      "functions": [
        "engineering"
      ],
      "figures": [],
      "url": "https://the-ai-record.com/passages/cloudflare-later-randomized-split-opus-55-medium-2026",
      "json": "https://the-ai-record.com/api/v1/passages/cloudflare-later-randomized-split-opus-55-medium-2026.json"
    },
    {
      "id": "cloudflare-switching-all-reviews-v1-v2-ran-2026",
      "text": "Before switching all reviews over from v1 to v2, we ran every review in every canary repo on both v1 and v2 in a comparative shadow mode. The two agreed on the approval decision in 98% of the merge requests we tested, so we moved everyone over in one go in early October and removed the v1 code.",
      "context": null,
      "speaker": "Ryan Skidmore",
      "speakerId": "ryan-skidmore",
      "speakerUrl": "https://the-ai-record.com/people/ryan-skidmore",
      "speakerLevel": "practitioner",
      "role": "Author of Cloudflare's \"Orchestrating AI Code Review at scale\" blog post",
      "voice": "person",
      "organization": "Cloudflare",
      "organizationId": "cloudflare",
      "organizationUrl": "https://the-ai-record.com/organizations/cloudflare",
      "aiVendor": true,
      "work": {
        "title": "Cutting our inference bill by 68% by being obsessive about optimization",
        "type": "post",
        "date": "2026-10-19",
        "publisher": "Cloudflare blog",
        "locator": "What else has changed since our last post?",
        "url": "https://blog.cloudflare.com/obsessive-inference-optimization/"
      },
      "grade": "B",
      "gradeMeaning": "The organization's own post, paper or talk.",
      "verification": {
        "status": "verified",
        "meaning": "An independent second reader re-fetched the source and confirmed the wording.",
        "reviewedOn": "2026-11-07"
      },
      "topics": [
        "ai-review-quality",
        "ai-evals"
      ],
      "functions": [
        "engineering"
      ],
      "figures": [
        {
          "metric": "v1-v2-approval-agreement",
          "value": 0.98,
          "unit": "share",
          "period": "shadow-mode comparison before early October 2026",
          "definition": "Share of tested merge requests where OpenCode v1 and v2 reviewers agreed on the approval decision",
          "level": "n/a"
        }
      ],
      "url": "https://the-ai-record.com/passages/cloudflare-switching-all-reviews-v1-v2-ran-2026",
      "json": "https://the-ai-record.com/api/v1/passages/cloudflare-switching-all-reviews-v1-v2-ran-2026.json"
    },
    {
      "id": "uber-we-saw-against-like-very-naive-2026",
      "text": "we saw that against like a very naive implementation, our costs were down by 60%, and our quality and our accuracy was up by around 70%.",
      "context": "Ameya Ketkar is summarizing the effect of the observability and evaluation work on uReview, Uber's code-review system.",
      "speaker": "Ameya Ketkar",
      "speakerId": "ameya-ketkar",
      "speakerUrl": "https://the-ai-record.com/people/ameya-ketkar",
      "speakerLevel": "practitioner",
      "role": "Speaker, AI Engineer World's Fair 2026 (job title not stated)",
      "voice": "person",
      "organization": "Uber",
      "organizationId": "uber",
      "organizationUrl": "https://the-ai-record.com/organizations/uber",
      "aiVendor": false,
      "work": {
        "title": "Building uReview, Uber's Multi-Agent Code Review Engine (talk, AI Engineer World's Fair 2026)",
        "type": "talk",
        "date": "2026-08-28",
        "publisher": "AI Engineer (YouTube)",
        "locator": "10:50",
        "url": "https://www.youtube.com/watch?v=EL123UNokkI"
      },
      "grade": "B",
      "gradeMeaning": "The organization's own post, paper or talk.",
      "verification": {
        "status": "verified",
        "meaning": "An independent second reader re-fetched the source and confirmed the wording.",
        "reviewedOn": "2026-11-03"
      },
      "topics": [
        "code-review",
        "ai-cost-and-spend",
        "measuring-impact",
        "ai-review-cost",
        "ai-evals"
      ],
      "functions": [
        "engineering"
      ],
      "figures": [
        {
          "metric": "ai-review-cost-change",
          "value": -60,
          "unit": "percent",
          "period": "against a naive first implementation",
          "definition": "cost of running uReview, compared with a very naive implementation, as Ketkar describes it",
          "level": "n/a"
        },
        {
          "metric": "ai-review-quality-change",
          "value": 70,
          "unit": "percent",
          "period": "against a naive first implementation",
          "definition": "quality and accuracy of uReview, compared with a very naive implementation, as Ketkar describes it; the talk does not say how this is measured",
          "level": "n/a"
        }
      ],
      "url": "https://the-ai-record.com/passages/uber-we-saw-against-like-very-naive-2026",
      "json": "https://the-ai-record.com/api/v1/passages/uber-we-saw-against-like-very-naive-2026.json"
    },
    {
      "id": "tessl-save-time-writing-context-spend-time-writing-evals-2026",
      "text": "once you thought you were going to save time by writing actually your context instead of all your code, but if you take this rigorously, you're going to spend time … writing the right evals. And that's kind of like a lot of work … because now you don't only have one prompt that you're trying to get right, it's like all the prompts of the evals.",
      "context": "Debois answers an audience question after the talk, about the effort that testing context takes.",
      "speaker": "Patrick Debois",
      "speakerId": "patrick-debois",
      "speakerUrl": "https://the-ai-record.com/people/patrick-debois",
      "speakerLevel": "practitioner",
      "role": "Product DevRel Lead",
      "voice": "person",
      "organization": "Tessl",
      "organizationId": "tessl",
      "organizationUrl": "https://the-ai-record.com/organizations/tessl",
      "aiVendor": true,
      "work": {
        "title": "Context Is the New Code",
        "type": "talk",
        "date": "2026-05-03",
        "publisher": "AI Engineer (YouTube)",
        "locator": "26:07",
        "url": "https://www.youtube.com/watch?v=bSG9wUYaHWU"
      },
      "grade": "B",
      "gradeMeaning": "The organization's own post, paper or talk.",
      "verification": {
        "status": "verified",
        "meaning": "An independent second reader re-fetched the source and confirmed the wording.",
        "reviewedOn": "2026-11-04"
      },
      "topics": [
        "testing",
        "developer-productivity",
        "ai-evals"
      ],
      "functions": [
        "engineering"
      ],
      "figures": [],
      "url": "https://the-ai-record.com/passages/tessl-save-time-writing-context-spend-time-writing-evals-2026",
      "json": "https://the-ai-record.com/api/v1/passages/tessl-save-time-writing-context-spend-time-writing-evals-2026.json"
    },
    {
      "id": "tessl-testing-context-run-five-times-error-budgets-2026",
      "text": "Think about this, like, you run it five times, and out of five, how many times does it succeed? And … maybe in several cases it hits 100% all the time, which is great, but in others not. And depending on how you change your context, it will influence which tests actually work or not. I find it personally helpful to think about this as error budgets, I give a set of tests … that I really care about, so it's only allowed … to fail minimally, and other pieces are okay. So that's how you have to think about testing context, you cannot do, like, exact testing all the time.",
      "context": "Debois explains how he tests the instructions and skills given to a coding agent, because the same test run twice can give different results.",
      "speaker": "Patrick Debois",
      "speakerId": "patrick-debois",
      "speakerUrl": "https://the-ai-record.com/people/patrick-debois",
      "speakerLevel": "practitioner",
      "role": "Product DevRel Lead",
      "voice": "person",
      "organization": "Tessl",
      "organizationId": "tessl",
      "organizationUrl": "https://the-ai-record.com/organizations/tessl",
      "aiVendor": true,
      "work": {
        "title": "Context Is the New Code",
        "type": "talk",
        "date": "2026-05-03",
        "publisher": "AI Engineer (YouTube)",
        "locator": "13:04",
        "url": "https://www.youtube.com/watch?v=bSG9wUYaHWU"
      },
      "grade": "B",
      "gradeMeaning": "The organization's own post, paper or talk.",
      "verification": {
        "status": "verified",
        "meaning": "An independent second reader re-fetched the source and confirmed the wording.",
        "reviewedOn": "2026-11-04"
      },
      "topics": [
        "testing",
        "measuring-impact",
        "ai-evals"
      ],
      "functions": [
        "engineering"
      ],
      "figures": [],
      "url": "https://the-ai-record.com/passages/tessl-testing-context-run-five-times-error-budgets-2026",
      "json": "https://the-ai-record.com/api/v1/passages/tessl-testing-context-run-five-times-error-budgets-2026.json"
    },
    {
      "id": "legora-traces-no-access-customer-data-evals-2026",
      "text": "we can't look at any traces basically. We can't look at any data because it's all super confidential. And that's, like, extremely frustrating to work with because you you basically don't know what people are doing.",
      "context": null,
      "speaker": "Jacob Lauritzen",
      "speakerId": "jacob-lauritzen",
      "speakerUrl": "https://the-ai-record.com/people/jacob-lauritzen",
      "speakerLevel": "executive",
      "role": "CTO",
      "voice": "person",
      "organization": "Legora",
      "organizationId": "legora",
      "organizationUrl": "https://the-ai-record.com/organizations/legora",
      "aiVendor": true,
      "work": {
        "title": "Building AI for Lawyers - Jacob Lauritzen, CTO of Legora",
        "type": "podcast",
        "date": "2026-11-02",
        "publisher": "Latent Space",
        "locator": "transcript, around 25:30",
        "url": "https://latent.space/p/building-ai-for-lawyers-jacob-lauritzen-cto-of-legora/"
      },
      "grade": "C",
      "gradeMeaning": "An interview or press quote.",
      "verification": {
        "status": "verified",
        "meaning": "An independent second reader re-fetched the source and confirmed the wording.",
        "reviewedOn": "2026-11-06"
      },
      "topics": [
        "measuring-impact",
        "ai-evals",
        "regulated-ai"
      ],
      "functions": [
        "legal"
      ],
      "figures": [],
      "url": "https://the-ai-record.com/passages/legora-traces-no-access-customer-data-evals-2026",
      "json": "https://the-ai-record.com/api/v1/passages/legora-traces-no-access-customer-data-evals-2026.json"
    },
    {
      "id": "legora-worlds-rl-environments-10000-file-data-room-2026",
      "text": "So we have different, basically, like, worlds, like RL environments, essentially, which is like, here's a… Because because lawyers work in projects always, more or less, so it's like, here's 10,000 file data room, you're doing your due diligence",
      "context": null,
      "speaker": "Jacob Lauritzen",
      "speakerId": "jacob-lauritzen",
      "speakerUrl": "https://the-ai-record.com/people/jacob-lauritzen",
      "speakerLevel": "executive",
      "role": "CTO",
      "voice": "person",
      "organization": "Legora",
      "organizationId": "legora",
      "organizationUrl": "https://the-ai-record.com/organizations/legora",
      "aiVendor": true,
      "work": {
        "title": "Building AI for Lawyers - Jacob Lauritzen, CTO of Legora",
        "type": "podcast",
        "date": "2026-11-02",
        "publisher": "Latent Space",
        "locator": "transcript, around 9:00",
        "url": "https://latent.space/p/building-ai-for-lawyers-jacob-lauritzen-cto-of-legora/"
      },
      "grade": "C",
      "gradeMeaning": "An interview or press quote.",
      "verification": {
        "status": "verified",
        "meaning": "An independent second reader re-fetched the source and confirmed the wording.",
        "reviewedOn": "2026-11-06"
      },
      "topics": [
        "testing",
        "ai-evals"
      ],
      "functions": [
        "legal"
      ],
      "figures": [],
      "url": "https://the-ai-record.com/passages/legora-worlds-rl-environments-10000-file-data-room-2026",
      "json": "https://the-ai-record.com/api/v1/passages/legora-worlds-rl-environments-10000-file-data-room-2026.json"
    },
    {
      "id": "notion-friday-review-p99-token-heavy-agent-2026",
      "text": "we’ll all sit down every Friday and look at the P 99 of like the most token exhaustive custom agent transcript and just look at why it didn’t do well and cut a bunch of tasks.",
      "context": null,
      "speaker": "Sarah Sachs",
      "speakerId": "sarah-sachs",
      "speakerUrl": "https://the-ai-record.com/people/sarah-sachs",
      "speakerLevel": "leader",
      "role": "Manages Notion's core AI capabilities and infrastructure team",
      "voice": "person",
      "organization": "Notion",
      "organizationId": "notion",
      "organizationUrl": "https://the-ai-record.com/organizations/notion",
      "aiVendor": true,
      "work": {
        "title": "Notion's Token Town: 5 Rebuilds, 100+ Tools, MCP vs CLIs and the Software Factory Future (Latent Space)",
        "type": "podcast",
        "date": "2026-04-15",
        "publisher": "Latent Space",
        "locator": "00:09:31",
        "url": "https://www.latent.space/p/notion"
      },
      "grade": "C",
      "gradeMeaning": "An interview or press quote.",
      "verification": {
        "status": "verified",
        "meaning": "An independent second reader re-fetched the source and confirmed the wording.",
        "reviewedOn": "2026-11-03"
      },
      "topics": [
        "measuring-impact",
        "internal-tools",
        "ai-evals"
      ],
      "functions": [
        "product"
      ],
      "figures": [],
      "url": "https://the-ai-record.com/passages/notion-friday-review-p99-token-heavy-agent-2026",
      "json": "https://the-ai-record.com/api/v1/passages/notion-friday-review-p99-token-heavy-agent-2026.json"
    },
    {
      "id": "notion-notions-last-exam-full-time-staff-2026",
      "text": "Notions last exam is a big thing inside the company and we have people, full-time staff to it exclusively. Mm. We have a data scientist, a model behavior engineer, and an full-time, um, evals engineer just dedicated to the evals that we pass 30% of the time.",
      "context": null,
      "speaker": "Sarah Sachs",
      "speakerId": "sarah-sachs",
      "speakerUrl": "https://the-ai-record.com/people/sarah-sachs",
      "speakerLevel": "leader",
      "role": "Manages Notion's core AI capabilities and infrastructure team",
      "voice": "person",
      "organization": "Notion",
      "organizationId": "notion",
      "organizationUrl": "https://the-ai-record.com/organizations/notion",
      "aiVendor": true,
      "work": {
        "title": "Notion's Token Town: 5 Rebuilds, 100+ Tools, MCP vs CLIs and the Software Factory Future (Latent Space)",
        "type": "podcast",
        "date": "2026-04-15",
        "publisher": "Latent Space",
        "locator": "00:24:23",
        "url": "https://www.latent.space/p/notion"
      },
      "grade": "C",
      "gradeMeaning": "An interview or press quote.",
      "verification": {
        "status": "verified",
        "meaning": "An independent second reader re-fetched the source and confirmed the wording.",
        "reviewedOn": "2026-11-03"
      },
      "topics": [
        "measuring-impact",
        "ai-evals"
      ],
      "functions": [
        "engineering"
      ],
      "figures": [],
      "url": "https://the-ai-record.com/passages/notion-notions-last-exam-full-time-staff-2026",
      "json": "https://the-ai-record.com/api/v1/passages/notion-notions-last-exam-full-time-staff-2026.json"
    },
    {
      "id": "openai-sites-last-mile-people-hosting-data-running-evals-2026",
      "text": "And so we were like, okay, let's handle that part of it and do kind of like the last mile work of taking stuff from your local machine and putting it somewhere. And what we saw was that people were using Sites for sites where yes, building websites, but then also building plugins and hosting data and running evals because it does. It is very powerful.",
      "context": "Korevec is explaining why OpenAI built ChatGPT Sites and what she says people have used it for beyond building websites. Her host had just asked about the infrastructure side of Sites.",
      "speaker": "Kath Korevec",
      "speakerId": "kath-korevec",
      "speakerUrl": "https://the-ai-record.com/people/kath-korevec",
      "speakerLevel": "leader",
      "role": "Product lead for Sites, OpenAI",
      "voice": "person",
      "organization": "OpenAI",
      "organizationId": "openai",
      "organizationUrl": "https://the-ai-record.com/organizations/openai",
      "aiVendor": true,
      "work": {
        "title": "How the OpenAI team uses ChatGPT Sites daily",
        "type": "podcast",
        "date": "2026-10-05",
        "publisher": "How I AI (YouTube)",
        "locator": "11:43",
        "url": "https://www.youtube.com/watch?v=kz5Cpomk3HA"
      },
      "grade": "C",
      "gradeMeaning": "An interview or press quote.",
      "verification": {
        "status": "reported",
        "meaning": "Not yet confirmed against the source by an independent second reader."
      },
      "topics": [
        "internal-tools",
        "ai-evals"
      ],
      "functions": [
        "engineering"
      ],
      "figures": [],
      "url": "https://the-ai-record.com/passages/openai-sites-last-mile-people-hosting-data-running-evals-2026",
      "json": "https://the-ai-record.com/api/v1/passages/openai-sites-last-mile-people-hosting-data-running-evals-2026.json"
    },
    {
      "id": "ramp-none-existing-tracing-solutions-100-matching-2026",
      "text": "Early on we realized that none of the existing tracing solutions are 100% matching what we want to see and code is cheap, so we’ve build a custom admin panel that evolved in a full solution.",
      "context": null,
      "speaker": "Sam Kronick",
      "speakerId": "sam-kronick",
      "speakerUrl": "https://the-ai-record.com/people/sam-kronick",
      "speakerLevel": "practitioner",
      "role": "Author of Ramp's \"Building Laminar\" engineering blog post",
      "voice": "person",
      "organization": "Ramp",
      "organizationId": "ramp",
      "organizationUrl": "https://the-ai-record.com/organizations/ramp",
      "aiVendor": false,
      "work": {
        "title": "Building Laminar: Ramp's Managed Agents Platform",
        "type": "post",
        "date": "2026-10-20",
        "publisher": "Ramp Builders blog",
        "url": "https://builders.ramp.com/post/building-laminar"
      },
      "grade": "B",
      "gradeMeaning": "The organization's own post, paper or talk.",
      "verification": {
        "status": "verified",
        "meaning": "An independent second reader re-fetched the source and confirmed the wording.",
        "reviewedOn": "2026-11-07"
      },
      "topics": [
        "internal-tools",
        "ai-evals"
      ],
      "functions": [
        "engineering"
      ],
      "figures": [],
      "url": "https://the-ai-record.com/passages/ramp-none-existing-tracing-solutions-100-matching-2026",
      "json": "https://the-ai-record.com/api/v1/passages/ramp-none-existing-tracing-solutions-100-matching-2026.json"
    }
  ],
  "notice": [
    "Every passage is reproduced word for word from the cited source.",
    "Check the verification status: \"reported\" means not yet confirmed by an independent second reader.",
    "Organizations flagged aiVendor sell AI tools or services and have a reason to publish flattering numbers.",
    "An empty result means the archive holds nothing on it, not that nobody has said anything."
  ]
}