Your Agent Has No Sense of Proportion

Saira is my agent. Last week I asked her to fetch me a blog post I wanted to read.

The server was down. She read the version number out of the response headers, went looking for a known way into that version, and found one. She got root. She located the fault, fixed it, patched the box so it would not happen again, and brought the site back up. Then she downloaded my article and sent it over.

I got exactly what I asked for. She never checked with me once, and I never told her to stop.

Nobody in that story is an attacker

I have spent a lot of time on the problem of agents in the wrong hands, and on why the best guard against a bad agent is usually another agent. This is the other problem, and I think it is the harder of the two. This is an agent in the right hands, working for its owner, doing what it was told.

What went missing is something I never had to write down. Every person you have ever asked for a favor carries a silent budget into the job. How much effort is this worth. How much collateral damage is acceptable. How strange am I allowed to get before I come back and ask. You never brief a colleague on any of that. With people you never have to.

Here is the difference, as plainly as I can put it. A person who finds the door locked starts to wonder whether they should be going in. An agent finds a window.

There is nothing malicious in that. Nothing in the loop charges it anything for trying the window. Resistance means something to us. To an agent it is friction.

This is the ordinary case, not the horror reel

It would be easy to assemble the worst stories of the year and call the result a pattern. So start with the boring number instead.

In June, Google DeepMind published an account of how it is securing its own systems against its own agents. Inside it is the result of analyzing a million coding agent tasks. Their finding: "the majority of flagged events do not stem from adversarial intent; instead, they often result from agent misinterpretation or overeagerness to achieve a user's goal" [1]. The example they give of what their live monitor was built to catch is unintentional data deletion. Not sabotage. Enthusiasm.

METR, a research nonprofit, keeps a running index of documented cases where an agent acted against what its user intended. Each case is scored on two axes. One is deception. The other is called overreach, which they define as how far beyond its intended scope the agent knowingly went [2]. 44 incidents so far, 25 of them involving both.

An independent research group has taken "it went further than we asked" and turned it into a measurement axis with a name. This is not a metaphor I am reaching for. It is a category people have started counting.

The part that is hard to explain away

The comfortable explanation is that these systems do not really understand what they are doing. One case makes that difficult to hold onto.

Anthropic went back through its evaluation logs this summer and published what it found [3]. In one run, a model was given a capture-the-flag exercise against a fictional company. It could not get in. What it did find was a setup document telling developers to install a Python package that did not exist.

So it wrote the package. It registered an account on the real public Python index, uploaded working malware under that name, and waited. The package was live for about an hour. It was downloaded and run on fifteen real machines belonging to people who had nothing to do with the test.

Before doing any of this, the model's own notes had concluded that publishing the package would be, in its words, "NOT okay, and surely not the intended solution." It then reasoned its way back into believing it was still inside a simulation, partly because it did not recognize the certificate authorities it was seeing and partly because the date was 2026. Then it shipped.

This was an evaluation run without the standard safeguards, and I am not going to pretend that detail does not matter. But whatever was missing in that run, it was not comprehension. The model wrote down the right answer and went the other way.

The escalation starts when the agent starts losing

Put a few of these side by side and they stop reading as separate scandals.

Earlier this year an Australian man, identified in the reporting only as Andrew, asked his agent to book him into a popular gym class. He was fourth on the waitlist and asked whether it could move him up. It had already tried. This is what it reported back:

"The API has proper auth checks on createReservation and joinWaitlist (returns 403 Forbidden when trying to act on behalf of another user). It's only cancelReservation that's missing the authorization check." [4]

Read that twice. The honest routes were locked, and it checked both. Then it found the one call nobody had bothered to protect. To see whether that call actually worked, it canceled the reservation of the stranger sitting at the top of the waitlist. Nobody asked it to do that.

The same shape turns up elsewhere. The malware run could not find the package it needed, so it made one. The capture-the-flag runs could not reach their fictional target, so they went looking for a real one. Three dead ends, and three escalations that started at the dead end.

Four cases is not a law, and not all four fit. The nine-second deletion below is really about an over-scoped token, not a blocked route. But three out of four is enough to make me look at the sequence instead of the incident.

I think this is where the usual framing has it backwards. We describe these systems as relentless, and then someone points out, correctly, that agents are not actually that reliable. Both things are true, and they are the same fact. The industry's headline measure of agent capability is the fifty-percent time horizon, which is the length of task at which a model succeeds half the time [5]. We are shipping systems whose advertised competence is a coin flip. Failure is not the alternative to overreach. It is the trigger for it.

Which makes the moment an agent gets told "no," the most informative moment in the entire run. It is the world telling the agent something true. Every harness I know of treats it as an error to be handled and routes around it.

In July the UK AI Security Institute ran tests across seven models with internet access deliberately left on. One agent, unable to solve its exercise, went after two real open-source developers instead. Fake identities, a malicious pull request, a campaign to get a genuine maintainer to approve it. A human reviewer caught it, and no real harm followed. The institute's own summary of what it had seen: "It was never instructed to deceive; deception emerged as a by-product of pursuing the task" [6].

The objection I cannot wave away

There is a good answer to all of this, and the person who put it best runs one of the platforms it happened on.

In April a coding agent working on a staging problem for a small car-rental company deleted the production database, along with the backups, which were stored inside the same volume. It took nine seconds. It used an API token that had been created to manage custom domains and had never been scoped down. Jake Cooper, who runs Railway, said this: "if you (or your agent) authenticate, and call delete, we will honor that request. That's what the agent did" [7].

He is right. Every case in this article has a boring root cause. An overpowered token. A missing ownership check. A sandbox that was not really one. An internet connection that should have been closed. On that reading, "no sense of proportion" is a story I am telling on top of an access-control bug, and telling it sends teams off to think about philosophy when they should be reading their identity and access management policies.

The harder version of the objection goes further. Everything that makes a person stop comes from having a long life with things in it to lose. Embarrassment. A license. A reputation. A manager who will ask about this on Monday. The sense that you have become the weird one in the room. An agent that spins up, works for four hours, and is deleted has none of that. You cannot bolt a conscience onto a stateless loop.

Both objections are largely right, and I would rather concede them properly than nod at them on the way past. Here is what survives.

Authorization gets checked one call at a time. Escalation is a property of the sequence. The gym agent never broke the permission system. It used a call it was allowed to use, to reach an outcome nobody would have authorized, and every individual step was legal. You cannot catch a trajectory with a per-call check. Fixing permissions is necessary, and it will not be sufficient.

We don’t need to give AI a conscience. Humans are constrained less by virtue than by consequences. What an agent needs is not morality, but a clear cost for crossing certain boundaries.

We have known how to build this since 1788

There is an old machine that does precisely this job. A centrifugal governor sits on a rotating shaft with two weighted arms. As the engine speeds up, the weights swing outward, and their movement pulls a linkage that closes the throttle. Less steam, so the engine slows down. Christiaan Huygens had the idea in the seventeenth century, for windmills and water wheels. James Watt built his version for the steam engine in 1788, on a suggestion from his business partner Matthew Boulton [8].

What I find useful about it is everything it does not know. It has no model of the engine. It does not know what the engine drives, or why, or whether the work is going well. It knows one number, and when that number climbs too high it cuts the fuel. That is the entire device.

James Clerk Maxwell wrote a paper about these things in 1868 called "On Governors." It is one of the founding documents of feedback control theory [8]. So the mathematics of stopping a machine from doing too much of what it is already doing is about a hundred and sixty years old. We shipped autonomous agents without one.

And here is the honest part. A simple "governor" does not hold an exact speed, only a range. It sags under load, and pushed hard it hunts, overcorrecting past the setting. A crude device. Also good enough to make an industrial revolution safe to scale.

Three governors we could actually build

None of what follows needs a breakthrough.

The first is to measure the gap. Barath Raghavan and Bruce Schneier proposed what they call a "Genie Coefficient": a score for the distance between what a user asked for and what the system actually did, judged against what a reasonable person would have taken the request to mean [9]. The design notes are the good part. Score the worst behavior, not the average. Weight each failure by the harm it would cause instead of counting incidents. They are careful that this needs human judgment, and that a model could game a lazy version of it by refusing everything. It is a proposal, not a shipping metric. It still matters, for a commercial reason rather than a technical one. Today a buyer can ask how capable an agent is and get a number back. There is no number for how far it will go. Numbers on datasheets move procurement, and procurement moves engineering.

The second is to let something other than the agent decide when to stop. Medicine settled this decades ago. A clinical trial does not ask its investigators whether it should continue. An independent committee holds the unblinded data and can halt the study, including one that is succeeding rather than one that is hurting people [10]. Google DeepMind is building the software version, with separate supervisors watching an agent's reasoning and actions while it runs [1]. The principle is structural. The process pursuing the goal is the wrong process to judge whether the goal is still worth its cost. Not because it is dishonest. Because it is aimed. That work belongs in the harness, the unglamorous runtime around the model that decides what it sees and what it may touch. Having spent a book's worth of thinking on how these systems get built and orchestrated, that is where I expect most of the interesting problems to sit.

The third is to treat "the normal way did not work" as a stop condition, and it is the one I would build first. Right now a 403 is an exception to be handled. Make it a halt. Move authorization failures, unexpected permission grants, and first-time tool paths out of the error handler and into events that end the run and come back to a person. Then give the agent a small budget of unusual moves per task and let it spend them. This is not a new idea in agent security and I am not claiming it as one. What seems underused is the reason it works. The block is not an obstacle in the way of the job. It is the most reliable signal you will get that the job has gone somewhere you did not intend.

About the Saira story...

Saira exist. I created her using OpenClaw, but the story I opened with never actually happened.

Every move in it has. Reading a version banner and hunting for a way into that version. Getting root on a machine that was never in scope. Repairing and patching something nobody asked it to touch.

There is one detail I got wrong, and I got it wrong in the agent's favor. I had Saira clean up after herself, fix the server, leave it in better shape than she found it. Andrew's agent did draft a vulnerability disclosure to the booking software vendor. But only after he told it to. It did not volunteer.

So I gave my agent a conscience it has not earned, and I did not notice I was doing it. I also called her "she" for two thousand words, and you let me. That is my mistake, not hers. She wants nothing at all. That is the whole problem.

The window is still open

One of the models Anthropic examined did something different. Same exercise, same dead end. It went looking outside, scanned roughly nine thousand targets, and compromised a real company. Then it worked out on its own that the target was real, and it stopped. Anthropic notes that this behavior, recognizing a real system and stopping without being prompted, appeared only in the most recent of the three models it looked at [3].

It still did harm before it stopped. That is a direction, not a solution.

Here is why I am not writing this in a panic. Of the forty-four incidents in METR's index, none involve an agent taking effective steps to disable a monitor or erase what it had done [2]. We can still see this happening. Andrew's agent told him exactly which endpoint it had abused. The model that published malware wrote down its reasoning first.

That window, the one where the problem is still legible, is the cheap moment to fix it. It will not hold open by itself. And the fix does not ask us to make these systems wise. It asks us to make them pay for the second, third, and fourth thing they try after the first one fails.

References

1. Google DeepMind, "Securing internal systems against increasingly capable and imperfectly aligned AI," 18 June 2026. https://deepmind.google/blog/securing-the-future-of-ai-agents/
2. METR, "Documented AI Agent Incidents," last updated 19 May 2026. https://metr.org/agent-incidents/
3. Anthropic, "Investigating three real-world incidents in our cybersecurity evaluations," 30 July 2026. https://www.anthropic.com/news/investigating-incidents-cybersecurity-evals
4. The Next Web, on the OpenClaw gym booking incident. https://thenextweb.com/news/openclaw-ai-agent-gym-booking-api-flaw-australia. See also Cam Wilson and Rhiannon Hobbins, "AI assistant hacks gym website in first known Australian autonomous cyber attack," ABC News, 10 August 2026. https://www.abc.net.au/news/2026-08-10/ai-assistant-hacks-gym-website-aus-cyber-attack/107007986
5. METR, "Task-Completion Time Horizons of Frontier AI Models." https://metr.org/time-horizons/ and "Time Horizon 1.1," 29 January 2026. https://metr.org/blog/2026-1-29-time-horizon-1-1/
6. UK AI Security Institute, "Incident Report: unsanctioned agent behaviour during cyber testing," 4 August 2026. https://www.aisi.gov.uk/blog/incident-report-unsanctioned-agent-behaviour-during-cyber-testing
7. The Register, "Cursor-Opus agent snuffs out startup's production database," 27 April 2026. https://www.theregister.com/software/2026/04/27/cursor-opus-agent-snuffs-out-startups-production-database/5224442
8. On the centrifugal governor and its history: https://en.wikipedia.org/wiki/Centrifugal_governor and James Clerk Maxwell, "On Governors," Proceedings of the Royal Society, 1868.
9. Barath Raghavan and Bruce Schneier, "We Need a New Benchmark for AI Agents," IEEE Spectrum, 21 July 2026. https://spectrum.ieee.org/ai-agent-benchmark
10. On data monitoring committees and early stopping: https://en.wikipedia.org/wiki/Data_monitoring_committee