Everyone building a software factory wants the same proof
Engineering leaders from HumanLayer, Warp, AWS and Asana teach us how software factories succeed, and how they fail
In February 2026, StrongDM’s three-person AI team demonstrated a pipeline built on two rules: code must not be written by humans, and code must not be reviewed by humans. To hold the second rule the team built behavioral clones of Okta, Jira and Slack and ran thousands of scenarios an hour against work nobody at the company had read. Tokens ran about $1,000 per engineer per day, roughly $20,000 per engineer per month.
This month, Darko Mesaroš, Distinguished Developer Advocate at AWS, described a repository called Kiro Crew. “Every pull request runs a gauntlet before a human even looks at it,” he told us: “Grep-level checks that only take seconds, about 27,000 tests, and four AI reviewers that are only allowed to block on concrete, provable defects.” All of it folds into a single green-or-red verdict, “which is how the project merges about 66 PRs a day with a human still approving every single one.”
Both are heavily automated. What separates them is where the human sits, and in six months it has moved: outside the loop in February, at the merge decision in August. What neither team can show you is what happened to that work after it merged.
Simon Willison watched the StrongDM demo and his skepticism landed on the bill rather than the automation. “If these patterns really do add $20,000/month per engineer, they’re far less interesting to me,” he wrote, wondering whether he had watched “a business model exercise” rather than a breakthrough. What he wanted was harder to build: a way “to have agents prove that their code works” without a human reading every line.
That question is still open. Teams everywhere are building software factories, pipelines where agents triage, spec, implement, review and ship while a human steers rather than types. We put a set of questions to three people building them:
Dex Horthy, who ran the fully automated version at HumanLayer and published where it breaks;
Zach Lloyd, who is building the cloud factory at Warp and has written down the equation he wants to solve;
and Darko Mesaroš, who answered on ownership at AWS and where the work goes once the coding gets faster.
Who owns the work when an agent opens the pull request?
We asked all three the same question, and we put our own data inside it. In LinearB’s 2026 mid-year benchmarks, agentic pull requests merge within 30 days at 37% in the organizations at the bottom of the community, against 81% for all pull requests in those same organizations. Hold the organization constant and the agent’s work is the work that stalls.
Horthy’s answer was straightforward:
“Every PR needs a human owner. This is non-negotiable,”
Mesaroš reached for an analogy. “As a developer, you’re responsible for the capability or product that ships—whether that’s created by you, you and AI, or AI under your supervision,” he said. “Think of AI agents like calculators. A calculator automates solving math equations, but you’re still responsible for making sure the answer is correct before adding that number to a status report for your team. An agent is a tool that helps you scale the work, but it doesn’t transfer accountability.”
The mechanics follow from there: “when an agent opens a pull request, it follows the same path that a developer would, and a human owns the decision to merge it. You need independent checks, whether that’s automated testing, another agent, or a human reviewer, as well as someone with the right expertise to stand behind the change.”
Zach Lloyd runs it as a policy setting rather than a rule. “It’s configurable by the company. At Warp, we currently have agent attribution, so the agent typically owns a PR that comes out of the factory,” he said, with human ownership and dual review required on sensitive parts of the codebase. Mesaroš draws the same line by risk: “While low-risk and well-understood changes can move through more automated paths, changes involving sensitive data, infrastructure, or new logic need stronger guardrails, like property-based testing and neurosymbolic AI checks, and deeper review.”
Amritansh Raghav, Head of Engineering at Asana, moves the question up a level, from the pull request to the feature. “An agent might autonomously ship a PR through the full development lifecycle with no human involvement, but the feature is still owned by a human. That person — or set of people — will always own that feature and be accountable for its quality, reliability, and longevity.”
Take note that none of these experts reach for model quality to explain why agent-opened work stalls. All three reach for the systems and people around the agent, rather than the agent itself.
A benchmark can score a model and still tell you nothing about your factory
Ownership settles who answers for a change. It says nothing about whether the change was any good.
Horthy has been asking a tighter question than most people arguing about AI coding. Not whether a model can write the change, but whether it leaves the codebase in a state anyone can work in months later. Nothing in the standard toolkit answers that. Tests go green on code that’s quietly harder to change than what it replaced.
In July he found a partial answer in SlopCodeBench, a long-horizon benchmark from Gabriel Orlanski’s lab at UW Madison that withholds the problem instead of handing it over up front, so a solution has to keep working as new requirements arrive. Horthy ran three Claude models through a 17-checkpoint subset and published the results himself, caveats included: a three-problem subset, Python-only detectors, and his own note that some of the rules flagging his code as slop are probably “a bit over-aggressive.”
The number he kept coming back to was density rather than pass rate. Comparing lights-off output against HumanLayer’s own reviewed TypeScript codebase, he found “over 11 times more slop triggers per kLOC” in the unreviewed solutions, a gap he attributes to review and context rather than to the model. His conclusion, which he repeats twice in the post: “today’s models can’t be relied on to run lights-off without steering.” He hasn’t closed the door on it, and he set his own bar for reopening it: “if/when models can score 80%+ on a (well-held-out) benchmark like SlopCodeBench which measures iteration over time, I’ll feel a LOT better about setting them loose with the lights off.”
While SlopCodeBench scores whether a model degrades a benchmark codebase, it’s not yet a tool-like layer that can score whether your own factory degrades your codebase. That’s an engineering task still ahead for new software factories as they form-fit them to their codebases.
The equation is only as good as the data
Zach Lloyd has spent the past two months publishing the clearest public description of how a cloud software factory works, and he’s been precise about what he hasn’t measured yet. “The promise of the factory approach is that it creates the foundation for measuring and improving the ROI of software development over time,” he wrote in July.
“This only works though if you measure your factory throughput and efficiency, where factory efficiency = (shipped product) / (token cost).”
He’s named the gap directly. Writing about the code review agent he built, he said, “For this to work really well, you should couple it with metrics and evals to see how much you are spending on code review, how many cycles it takes, and how often the reviewer has to be corrected. I’ll show how to create evals for these loops in a future post.” That post hasn’t shipped.
Asked how he’d defend a factory’s ROI today, he gave us a practitioner’s version of the same equation. “The simplest way to defend the ROI is to look at how much work the factory is actually automating, estimate how much human time that work would have taken, and compare that with the cost of the automation,” he said.
“You can also measure whether it’s improving over time by asking if the percentage of PRs being automated is going up and if the cost per PR is going down.”
On drift he pointed at a method rather than a number: “The best way to enable that is to define the factory as versioned code. Then you can track changes to skills, MCPs, models, and other parts of the factory and see which changes are affecting performance.”
The bottleneck moved to review, and everyone is feeling the strain
Zach Lloyd wrote into his build guide: “Once agents produce code at volume, review becomes the bottleneck.” Asked where the work goes once coding gets faster, he was straightforward:
“Right now, review is definitely the biggest bottleneck. We’re solving that with computer-use verification and better specing, where agents check for spec adherence.”
Mesaroš described the same constraint as a systems property. “When you put agents into a workflow, you accelerate some steps and expose constraints in others,” he said. “You cannot have one part of your workflow moving at AI speed while your other processes are still operating at human speed—you have to optimize the full pipeline.”
His fix starts earlier than review: “making sure the validation questions get answered early: Not just ‘Does it compile?’ but ‘Does it do what the spec and developer intended? Does it behave correctly in the real product?’ If those answers still depend on manual review late in the process or if what you’re reviewing is not quality code, that is where teams slow down regardless of how fast the code was written.”
Raghav follows it further downstream:
“Review is definitely where pressure builds first, especially for domain owners and tech leads who feel accountability for code quality and what ships. But it doesn’t stop there. As code generation accelerates, engineers feel more pain across everything that isn’t generating code — verification workflows, testing, getting a PR ready to ship. Then downstream, if you’re shipping more work with the same deployment practices, engineers start feeling it in deployment systems and ops.”
Horthy pushes the same trajectory further. “There will always be alpha in reviewing things, otherwise you get the exact same software as everybody else,” he told us. “What we review will continue to evolve. Today it still includes the code, some teams are also reviewing specs and designs, someday we might only review those things, or only review user requests, or something else entirely.” Lloyd sees the same move up the stack: “I think the bottleneck will eventually move more toward product definition, and making sure what you’re shipping is actually solving user problems and that they’re being solved in the right way.”
None of them are saying that code review is disappearing. All see it moving, and Horthy names the property that decides how far any of this can go: “The way these factories work is tightly tied to how easy a task is to verify, so if you can easily verify whether a task was completed correctly, it’s much easier to automate it.” Zach’s research estimates only up to 30% of engineering tasks fall in this category today, but that still represents a sizable opportunity for any team.
The ownership puzzle and proof beyond pull requests
Some really incredible proofs have happened in public: Ona built a factory in public over 10 days and posted the result: 375 pull requests merged, 67,000 lines of code, 1,067 tests, no human-written production code. Read that list again as evidence. Pull requests merged, lines written, tests passing. It’s throughput, and it’s the shape of almost every factory result you can find. None of those numbers says whether the work held up after it shipped.
LinearB’s 2026 mid-year AI benchmarks draw on 2.7 million pull requests, 7.5 million commits and 83,000 developers across 253 engineering organizations, and they’re built around the question all three practitioners keep circling: what happens to the code after the agent is done with it.
Agentic pull requests merge at 79% in the strongest organizations and 37% in the weakest, where all pull requests merge at 90% and 81% respectively. The agent’s work is where the spread opens up.
Turning on AI code review moves the number for agentic PR velocity:
At the top of the benchmarks percentile, pull requests reviewed by AI and pull requests reviewed only by a human both merge at 92%.
Lower down the two come apart, by three points in the middle band and by seven at the bottom, 86% against 79%.
Review capacity is where that difference comes from, and it’s thinnest exactly where it’s needed most:
57% of pull requests at the top of the community get an AI review,
against 26% and 8% further down.
Horthy, asked what data he’d reach for to defend a factory’s ROI, answered with a ladder rather than a metric. “If you can measure business success (cash ROI), do that. If you can’t measure biz success, measure customer outcomes (faster workflows, more throughput on valuable tasks). If you can’t measure customer outcomes, measure DORA-shaped things like throughput, lead time, failure rate, and time-to-recover. If you can’t measure those, I guess measure PR’s or LOC but at that point you’re probably better off just steering off vibes - if the factory is working really well, you will FEEL it.”
This is the start of a conversation the industry hasn’t had yet
Intrigued by the evolving discussion? On August 27, Dex Horthy and Zach Lloyd join a first-of-its-kind live roundtable hosted by LinearB and Dev Interrupted to carry that conversation on in the open, in front of an audience asking their own version of the same thing. You’re building a software factory, so let’s unpack the playbook for proving it works.
