How is AI used for testing?

“Stripe’s enormous preexisting battery of tests—over three million of them—can provide this feedback.”

Alistair GraySoftware engineer, Leverage team · Practitioner, Stripe

“we reached 97% of the 3.5K original Enzyme files”

Charles Covey-BrandtAuthor of the Airbnb engineering blog post on LLM test migration (role not stated on the page) · Practitioner, Airbnb

“every time there's a pull request, we have a swarm of agents. And they go and look at every file that's changed, every file that's related to the files that change to figure out if there's bugs. And it also spins up your code in a sandbox, … dependencies, spins up local host, clicks around to try to break things, mocks inputs, and everything else to figure out if the code is broken.”

Daksh GuptaCo-founder · Executive, Greptile

“So for example, when … playwright … came out instead of having humans navigate the code now, the agent can explore the code. That was a big win unlock for productivity in a lot of our team.”

Eyal BlumSoftware Engineer · Practitioner, Figma

“What we did was we built a system that actually has agents constantly running and creating BAML programs. … We then look at the entire … transcripts, see what tools they use, see what happened, and obviously we as humans can inspect them. But more importantly, we can have agents go inspect them. And agents find what was good, what was bad. … And we can have humans collaborate with these issues to figure out which ones are real, which ones are hallucinations, which ones don't have taste as much as I hate to use that word. And then we can have agents go ahead and create fixes to these problems and go to address them.”

Vaibhav GuptaCo-founder and Chief Executive · Executive, Boundary (BoundaryML)

“We’ve also internally done a big shift to make a lot of our core, components of code testable as purely local dev without needing to actually, integrate with, any live services for this reason. And honestly, the older the company, the more you have to change to shift in this direction. But you can use AI to help you perform this migration nowadays.”

Walden YanCo-founder and Chief Product Officer · Executive, Cognition

“Over roughly six weeks, we’ve run thousands of scans, produced over 300 findings ranging from defense-in-depth improvements to resolved security incidents.”

Zack DeveauTitle not stated on Deveau's author page; the post is written from Shopify's Application Security team · Practitioner, Shopify

“we are allowing the bot to merge that directly into our main branch because we have integration tests, unit tests, etc., canary tests.”

Xiucheng QuekSoftware engineer · Practitioner, Genentech