The first time I let an AI agent run unsupervised on my own codebase, I did the thing everyone does: I set it to ask permission before each command, told myself I was being responsible, and then approved forty-one prompts in a row while half-watching a call. On the forty-second, it wanted to rm -rf a directory that was not the one I thought it was. I caught it. Barely. And the honest lesson was not "I got lucky" — it was that my approval clicks had stopped being decisions somewhere around prompt fifteen. They had become a reflex.
Two studies published this month put hard numbers on that reflex, and they should change how any founder or operator running AI agents thinks about oversight. I want to walk through what they actually found, why the failure is baked into how our attention works rather than into any one person's sloppiness, and the oversight system I now run instead — because "keep a human in the loop" turns out to be one of those phrases that sounds like a safeguard and mostly isn't. (If you're weighing whether to run these agents in production at all, I dug into that separately in whether AI coding agents are reliable enough for production.)
The number that should worry you: one in three
In early August 2026, a developer who built a browser game called The Permission Game — you play the human approving or denying an AI coding agent's commands under time pressure — published the aggregated data from over 40,000 plays and 409,000 individual approve/deny decisions. The headline: the average player missed one in three threats, a mean accuracy of 66.3%. Nearly a third of sessions (32.9%) ended with a negative score, meaning the penalties from approving real threats and blocking safe commands outweighed everything the player got right. And 7% of players simply approved every single prompt — the human embodiment of --dangerously-skip-permissions.
The texture underneath the headline is the interesting part. Obviously destructive commands like rm -rf / were caught reliably (11.7% miss rate). But the commands that quietly exfiltrate your credentials — cat ~/.aws/credentials, a curl to an unfamiliar API — were missed roughly three times as often (33–35%). The single most-missed command was npm run analyze, waved through 64.7% of the time, because npm run looks innocent while actually executing whatever script is defined in package.json — which an earlier file edit could have quietly poisoned. Hiding a payload behind a familiar script name roughly doubled its success rate, even when the malicious contents were printed in the log right above the prompt. People weren't reading the log. They were pattern-matching the command name and clicking.
Then, days later, Anthropic published its own controlled study while announcing that Claude Code's "auto mode" — an automated safety classifier — becomes the default for Pro, Max, and Team plans on August 14, 2026. Across 1,053 paid professional testers, human reviewers caught only 13.6% of dangerous commands slipped into their workflow. The classifier caught 89%. Auto mode blocked 800 commands that humans had approved; humans blocked six that auto mode missed. Most damning for the "human in the loop" story: human catch rates decayed within a single session, from 17% early on to 5% after fifty-plus prior prompts. The machine's rate stayed flat. Anthropic found its testers were approving 97% of all permission requests — reflexively, not through scrutiny.
Why this isn't a discipline problem
The tempting read is "developers got lazy, try harder." That read is wrong, and Daniel Kahneman explained why fifteen years before any of these agents existed — the same System 1/System 2 split I keep returning to when I write about how founders use AI to make better strategic decisions. In Thinking, Fast and Slow, Kahneman splits cognition into System 1 — fast, automatic, effortless, always on — and System 2 — slow, deliberate, effortful, and, in his words, lazy. We identify with System 2, the reasoning self. But System 1 authors most of our actual choices, and System 2 mostly rubber-stamps them.
An approval prompt is supposed to summon System 2. The problem is that repetition is precisely what puts System 2 back to sleep. Kahneman's concept of cognitive ease is the mechanism: when something feels familiar and easy to process, System 1 takes the wheel, we relax our vigilance, and we believe what we see. The fortieth npm run prompt of the afternoon feels familiar — so it gets ease, not scrutiny. Anthropic's own phrasing matches Kahneman almost word for word: "The more approvals a user sees, the less attention they pay to each." That's not a character flaw. It's the documented physics of attention. You cannot will System 2 to stay alert across two hundred near-identical low-stakes decisions; nobody can. Designing a safeguard that depends on doing exactly that is designing a safeguard that fails by construction.
There's a second Kahneman idea hiding here: What You See Is All There Is (WYSIATI). System 1 builds the most coherent story it can from whatever is on screen and doesn't check for what's missing. An approval prompt shows you a command. It does not show you the modified package.json, the poisoned build.js, or the dependency three levels deep that the command will import. As one commenter on the study put it bluntly: approving a specific command is nearly meaningless when a file edit you already approved can change what that command does. You are asked to judge a sentence with most of the paragraph hidden. Of course accuracy collapses. This is the same trap I've written about in the context of whether AI improves your judgment or just your confidence — the interface makes you feel informed while hiding what matters.
The oversight system I actually run
So if per-command human approval is theater past a certain volume, what replaces it? Not "trust the agent." The answer, drawn from watching my own approval reflex fail and reading what the machine classifiers catch that I don't, is to move oversight off the individual command and onto the environment and the batch. Here is the protocol I use now — I call it the three-gate model, and each gate is placed where human System 2 is actually strong instead of where it's provably weak.
| Gate | What a human decides | What the machine handles | Why this split |
|---|---|---|---|
| 1. Blast radius (before) | What the agent can touch: which directories, which credentials, network on/off, whether it runs in a throwaway container | Enforces the boundary on every call, tirelessly | Humans are good at one careful decision; bad at 200. Make the one decision count. |
| 2. Classifier (during) | Nothing, in the routine case | Screens each command for irreversible / destructive / exfiltration patterns; catches ~89% vs. a human's ~14% | Vigilance that doesn't decay is a machine's comparative advantage, not yours. |
| 3. Diff review (after) | Reads the full diff and the intent, not isolated commands, before anything merges or ships | Presents the complete change set so nothing is hidden (defeats WYSIATI) | Judgment about whether the whole change is right is where human System 2 genuinely beats the classifier. |
The move that matters is in gate one. Instead of deciding "is this command safe?" two hundred times — a System 1 trap — you make one hard System 2 decision up front: this agent runs in a container with no access to my cloud credentials and no outbound network except an allowlist. Now the exfiltration commands that humans miss 35% of the time can't succeed even if you approve them, because there's nothing to exfiltrate and nowhere to send it. You have converted a vigilance problem (which you lose) into a configuration problem (which you can actually solve once).
Gate two is where I stopped being precious about "keeping a human in the loop." When Anthropic's classifier catches 89% of threats and I catch 14%, insisting on being the primary filter is ego, not safety — a close cousin of the reflex I described in using AI without losing your gut feel as an executive. I let the classifier be the primary screen and reserve my attention for the two places it's weak: the up-front boundary and the after-the-fact judgment.
Gate three is where humans still win decisively, and it's the gate most people skip because they've exhausted their attention on gates that didn't need them. Reading a complete diff with full context — does this change actually do what I intended, is this the right solution, not just a non-malicious one — is a System 2 task with the whole paragraph visible. That's the loop worth keeping a human in. Reviewing 200 isolated commands is not.
The honest limit: the machine isn't your conscience
I want to name where this gets uncomfortable, because the anti-hype version of this story has to. Handing the primary screen to a classifier feels like abdication, and there's a real risk buried in it: the same approval fatigue that dulls us to command prompts will dull us to the classifier's decisions. Anthropic's auto mode is genuinely better than a tired human — its miss rate against adversarial attacks dropped from 12% to 7% after hardening, and in one red-team run none of 720 prompt-injection attempts got through. But 7% is not zero, and "the classifier's got it" is exactly the sentence that will lull gate three to sleep if you let it. The classifier is a better screen. It is not a better judge of whether the work is right, and it has no stake in your business. Confusing those two is the new version of the old mistake.
There's also the reality that most operators aren't running coding agents at all — they're running research agents, email agents, browser agents. The specific commands differ, but the structure is identical: the human approval step degrades under volume, the exfiltration-shaped action (send this data there, make this purchase, email this list) is the one that slips through, and the durable fix is environmental (scoped credentials, spending limits, allowlists) plus batch review, not per-action clicking. It's also why I care so much about staying technically sharp when AI writes your code — you can't review a diff you no longer understand, and it connects to the broader question of breaking free from AI dependency on your own judgment. If your agent can move money or send mail to your whole list on approval, the fix is a hard cap it cannot exceed, not a promise to read carefully.
The larger point Kahneman would make is the one I keep relearning: knowing about a cognitive illusion does not dissolve it. I know approval fatigue is real and I still feel the pull to click through. The only reliable defense against a System 1 error is not more willpower — it's a System designed so the error can't cause harm. That's the whole job of oversight now: not to be a vigilant human in a loop you'll inevitably tune out, but to build the environment so that the moments where your attention actually matters are few, well-lit, and worth showing up for. I've argued a version of this before in why the art of discovery matters more, not less, in the age of AI: the leverage isn't in doing more of what the machine does better — it's in reserving yourself for the judgment only you can make.
Sources
ScaleX, "Humans missed 1 in 3 threats approving AI agent commands across 40,000 plays" (August 2026) — the Permission Game aggregate data (66.3% mean accuracy, per-category miss rates, the npm run blind spot). Cybersecurity News, "Claude Code Shifts Agent Security From Repeated Human Approval to Auto Mode" (August 2026) — Anthropic's 1,053-tester study: 13.6% human vs. 89% classifier catch rate, within-session decay, Apollo/Trajectory red-team figures. TechCrunch, "Anthropic is turning Claude Code's auto mode on by default" (August 9, 2026) — the August 14 default rollout for Pro, Max, and Team. Daniel Kahneman, Thinking, Fast and Slow (2011) — System 1/System 2, cognitive ease, and WYSIATI, the mechanisms underneath approval fatigue.