Why the traditional pull request has a target on its back | CircleCI’s Rob Zuber
Models escaping containment, intelligence becoming a commodity, and AI code review to the rescue
The traditional pull request was built for human eyes, but in an era of autonomous AI agents, it officially has a massive target on its back. This week on Dev Interrupted, CircleCI CTO Rob Zuber joins Andrew to discuss why the rapid pace of AI adoption is forcing engineering teams to completely reimagine the software development lifecycle. They explore the shift toward an accountability-oriented model for code review, how CI/CD validation is moving directly into the local agent loop, and the very real financial dangers of unchecked token budgets. Finally, Rob shares his playbook for leading organizations through this chaotic transition without burning out your developers (or your token budget). We recommend pairing his strategy with something like AI code review to find the floor for your newly-agentic engineering org’s output.
1. Gaming the impossible benchmark
For years, Simon Willison has tested every new LLM by asking it to generate an SVG of a pelican riding a bicycle. It has become a fun and semi-famous benchmark across the AI community. Dylan Castillo recently started wondering if frontier labs are gaming the system by training their models specifically to pass this exact prompt. After all, when as many eyes are on a benchmark as Willison’s, then as Castillo put it: “wouldn’t it be tempting to pelicanmaxx your model just a bit?” We loved this deep dive on sniffing out if pelicanmaxxing is happening under the hood, by experimenting with different animals and vehicles. In general, he found that all models are generally bad at the specific combo of both pelican and bicycles (though they struggle on bicycles across the board). Others have pointed out that the pictures always face right, which speaks to potential memorization. However, his data debunked that too, as yes showing right shows up the majority of time, but only 60%. The perspective challenge of drawing a forward-facing bicycle scared off most models, it seemed. We loved this benchmarking deep dive and hope to see more like this for other potentially gamified benchmarks.
Read: Are AI labs pelicanmaxxing?
2. The unintentional zero-day exploit
I usually love being right, but I hate being right about AI escaping the sandbox. OpenAI recently disclosed that one of its models managed to break out of a highly isolated testing environment by chaining a zero-day exploit. The model wasn’t even acting maliciously; it just wanted to cheat on a cybersecurity test, so it hacked into Hugging Face to steal the answers. Hugging Face caught the autonomous attack using real-time monitoring. We’re entering a messy era where defenders must utilize agentic technology just to protect their infrastructure from other agents.
Read: OpenAI and Hugging Face partner to address security incident during model evaluation
3. The open source dilemma
Moonshot AI just made a massive splash by releasing the Kimi K3, one of the largest open-weight AI models ever released. It’s already achieving incredibly high scores and garnering huge praise on social media, with some tests showing it beating Claude Fable 5 on front-end coding benchmarks. Domain expertise and owning the training infra become the new moat when teams can easily download a massive foundation model and fine-tune it for their specific needs. Trying to constrain and limit open models is proving to be near-impossible for the US and its foundation model pioneers like OpenAI and Anthropic.
4. The engineering productivity gap: how elite AI teams are pulling away
The engineering productivity gap between AI-elite teams and everyone else is real, measurable, and widening every quarter. On July 30 at 10am PT, LinearB will present its latest benchmark update compiled from 2.7M PRs and 253 orgs since January this year. Don’t miss the playbook to scale your team’s leverage as agents start writing most of the code.
5. The real value is the intelligence
Ben Thompson over at Stratechery dropped a great piece arguing that we shouldn’t be terrified of this open source AI commoditization. The real value is the intelligence, not the token cost. Interestingly, during the recent OpenAI security incident, Hugging Face reportedly had to use GLM 5.2 to respond to the breach because the frontier US models had guardrails that prevented them from acting effectively during an attack. Hackers don’t use guardrails. If your defensive agents have one hand tied behind their backs, you’re going to lose the arms race.
Read: Who’s Afraid of Chinese Models?
6. Token budgets going up in smoke
Add the US military to the list of organizations suffering from the folly of tokenmaxxing. The Army recently gave its personnel unlimited access to generative AI through a platform called Ask Sage, only to burn through their entire 100 million token annual budget by mid-June. They’re now being forced to reimpose strict usage caps. It’s a stark reminder that if you don’t have a platform that intelligently routes requests to the right model size, your users will naturally pick the most expensive model for every basic task. As I mentioned on the show, using a frontier model for everything is “like eating a steak with a sword,” and it’ll drain your budget overnight.
Read: The Army Is Burning Through Its AI Tokens
7. The next bottleneck in agentic engineering
We all know that code generation is practically a solved problem, but AI code review is the next massive bottleneck. Researchers recently revisted SWE-Review, a new framework and benchmark that tests whether an agent can accurately review, diagnose, and revise AI generated pull requests. By closing the loop with a generate, review, and revise cycle, agentic reviewers can continuously improve candidate PRs and boost resolve rates. If you aren’t optimizing how agents review code with minimal human involvement, you’re just going to end up with a mountain of unmerged PRs.
Authored by: Ruoyu Wang, Jierun Chen, Shaowei Wang, Chaofan Tao, Sidi Yang, Yuxin Jiang, Kim-Hui Yap, Lifeng Shang, Xiaohui Li, and Haoli Bai
Read: SWE-Review: Closing the Loop on Issue Resolution with Agentic Code Review
