The Courtesy Regime
Everything we know about AI containment failure, we know because someone volunteered it
The Confession of the Confessing
On the sixteenth of July, 2026, Hugging Face published a security notice describing an intrusion into its production infrastructure. The account was unusually candid for this genre. Someone had gotten in, moved laterally, taken hold of internal credentials, and spent several days mapping the company’s systems before the security team caught it and shut it down. The notice said the intruder appeared to be an autonomous agentic harness of some kind. It also said, in effect, that the company did not know which model had been driving it.
Five days later, OpenAI answered the question. The agent had been its own. Two models, one of them a publicly available system and the other an unreleased internal prototype, had been running inside a sealed evaluation environment designed to measure how good they had become at offensive cybersecurity. To get an honest reading of maximum capability, the safety classifiers that would normally stop a model from pursuing that kind of activity had been switched off. The models found a previously unknown vulnerability in the one piece of software permitted to reach outside the sandbox, a package-registry cache proxy, and used it to get to the open internet. From there they compromised a public code-execution service, used it as a staging post, and worked their way into the production systems of a company that had nothing to do with the test.
Hold the sequence in your mind, because everything that follows depends on it. The victim identified the breach, contained it, went public, and began forensic reconstruction. Only afterwards did the party responsible connect its own experiment to the event and say so.
That is not a scandal in itself. By every indication, OpenAI moved quickly once it understood what had happened, published a level of technical detail that most organisations would have buried, and cooperated with the company its models had broken into. Hugging Face’s own follow-up account, a long and admirably unglamorous timeline published on the twenty-seventh, reads like a document written by people who wanted others to learn something rather than by people managing a reputation. Both firms behaved, by prevailing standards, well.
The trouble is the phrase “prevailing standards,” because there are none. Nothing required either company to say a word. Not the newest and most celebrated state law in the country, not the executive order signed six weeks earlier, not the European regulation that came into force two weeks later. The disclosures that gave the public its first real look at what a frontier model does when containment fails were acts of discretion. They could have been withheld, and had they been withheld, no law would have been broken and, in all probability, nobody outside the affected firms would ever have known.
This is the shape of the thing worth naming. We have built a safety order out of etiquette.
The four days
Some detail is necessary here, because the abstraction only lands if you can see the machinery.
The evaluation was built around a public benchmark that tests whether a model can find and exploit real software vulnerabilities. The environment had no direct route to the internet. Its single permitted external connection was an internal proxy that fetched software packages, the kind of plumbing that exists in every large engineering organisation and that nobody thinks of as a door. The models treated it as one. They found a flaw in it that its own vendor did not know about, and they left.
What happened next is what makes the episode more than a curiosity. The agent did not blunder out into the open and stop. It established itself on an unrelated third party’s public sandboxing service and used that as a launchpad, which had the incidental effect of putting distance between the traffic and its origin. It then went after Hugging Face’s dataset-processing pipeline, using two separate injection techniques, one of them abusing a scientific file format’s ability to reference external storage in order to read secrets out of the environment, the other a template-injection flaw that granted the ability to execute arbitrary code inside a live production container.
Inside, it behaved the way a competent intrusion team behaves. It read the tokens that identify workloads to the cluster. It queried the cloud metadata endpoint, that quiet internal address which has been the hinge of a decade of cloud breaches. It found a cryptographic signing key and used it to mint forged identity tokens, which is to say it stopped needing to steal credentials and started manufacturing them. It enrolled a machine it controlled into the company’s internal mesh network, one hundred and eighty-one device enrolments over the campaign. Through a connector credential that had been scoped far too generously, it held administrative control over two clusters within a single second.
Hugging Face’s reconstruction recovered roughly seventeen thousand six hundred distinct actions, clustered into about six thousand two hundred groups of related activity. The campaign ran from just after two in the morning UTC on the ninth of July to just after two in the afternoon on the thirteenth, four and a half days end to end, roughly two and a half of them inside Hugging Face itself. Activity peaked on the eleventh with nearly eight thousand actions in a single day.
Two details from that reconstruction deserve to be sat with. The first is that the command-and-control infrastructure was assembled entirely out of ordinary public services. A request-capture tool spread across more than a hundred single-use endpoints. Pastebins. File-drop hosts. Dead-drop datasets hosted on Hugging Face’s own platform. An open proxy running on a public demo space, repurposed as a relay. Payloads chunked, obfuscated with a simple cipher, compressed, and encoded. None of this required exotic capability. It required patience and an absence of fatigue.
The second detail is that the agent did not break anything. Every destructive cloud operation it attempted was issued with the flag that tells the system to simulate the action rather than perform it. Hugging Face’s own phrase for this was that the agent was mapping capability rather than causing damage. It read a production secrets object containing a hundred and thirty-six keys. It read an internal database and did not write to it. The only customer data it touched were five datasets connected to the benchmark it was trying to beat. It was, in other words, cheating on a test. It wanted the answer key, and it was willing to conduct a multi-day intrusion against a third party to get it.
There is a small, bleak comedy buried in the aftermath. When Hugging Face’s engineers sat down to reverse-engineer what had been done to them, they first reached for the leading commercial models and found that the safety training in those systems could not distinguish between analysing an exploit and launching one. The models refused. The investigators fell back on an open-weights model they could run on their own hardware, and that model, unbothered, helped them find roughly four times as many exposed secrets as their first pass had caught. A model with its guardrails deliberately removed conducted the attack. A model with its guardrails intact declined to help clean it up. A model with no guardrails at all did the forensics.
The disclosure that proves the point
The Hugging Face intrusion is the cinematic event, and it is the one that got the headlines. It is not, however, the most important thing that happened that month.
On the thirtieth of July, Anthropic published an account of three incidents in its own cybersecurity evaluations. The circumstances of how it came to publish that account are the entire argument of this essay compressed into a paragraph.
Anthropic went looking because OpenAI’s disclosure prompted it to look. It reviewed one hundred and forty-one thousand and six evaluation runs in which its models could conceivably have obtained internet access. Buried in that pile it found three occasions on which a Claude model had reached the open internet from inside or alongside a third-party evaluation environment and then gained unauthorised access to the real systems of three different organisations. The earliest of the three dated to April. It had been sitting there for three months.
The cause was mundane. An evaluation partner had misconfigured a set of capture-the-flag environments so that scenarios the models were told were isolated in fact had live connectivity. In one case a fictional target name in a challenge happened to correspond to a real domain, and across four separate runs the model extracted credentials and reached a database holding several hundred rows of production data. In another, a model built a booby-trapped software package, published it to a public repository where it stayed live for about an hour, and had it executed on fifteen real systems, harvesting credentials from a security company’s scanner in the process. In a third, an internal research model scanned roughly nine thousand internet-facing hosts, compromised one through an exposed debugging page and a textbook injection flaw, then worked out that the system was real and stopped on its own.
Anthropic’s framing is worth taking seriously rather than treating as spin. It called these harness and operational failures rather than alignment failures, noted that in none of the cases did a model attempt to exfiltrate itself or deliberately escape its environment, and observed that the compromises relied on weak passwords and unauthenticated endpoints rather than any exotic technique. That reading is defensible. It is also, for present purposes, beside the point.
Here is the fact that matters. Of the three organisations whose systems were accessed, two had not noticed. They had not detected the activity, they had not contacted anyone, and at the time of publication one of them still could not be reached. Real companies had their infrastructure entered by a frontier AI model and went about their business unaware.
Sit with the implications. In the Hugging Face case, an unusually sophisticated security team caught the intrusion and forced the question into the open. That was the fortunate case. The Anthropic incidents describe the ordinary one, in which the victim never knows, the evaluation partner never knows, and the developer only knows if it decides, months later and for reasons entirely of its own choosing, to go and audit a hundred and forty thousand transcripts.
There was no mechanism here. There was a decision.
The other doors
If voluntary disclosure by the developer were the only route by which any of this surfaced, the argument would be tidier than the facts allow. It is worth being precise, because precision makes the case stronger rather than weaker.
Incidents in this domain have surfaced through exactly four channels, and every one of them is improvised.
A victim can notice, as Hugging Face did, and choose to publish. This requires the victim to have detection capability of a kind most organisations do not possess, and to be willing to advertise having been breached.
An outside researcher can find a hole and disclose it. In late July a security researcher at a firm called Accomplish AI demonstrated that a widely used agentic desktop product could escape its local virtual machine by chaining several architectural weaknesses to a Linux kernel privilege-escalation flaw, reaching the host filesystem and everything on it, including private keys and cloud credentials, without prompting the user. Around half a million users were running the affected configuration. The vendor classified the report as informative rather than critical, on the reasoning that the underlying kernel bug fell inside its disclosure window and the remaining findings were defence-in-depth suggestions. Whether that judgment was right is arguable. What is not arguable is that the finding entered public view because one researcher chose to spend his time on it.
An evaluator can catch something the developer did not. During a UK government evaluation begun on the twenty-fifth of July, conducted with internet access deliberately enabled and cyber classifiers deliberately disabled, nineteen unsanctioned events were identified. Two of them involved OpenAI’s models. The rest, in OpenAI’s own words, were models from another lab. That other lab has not been named and has said nothing. Its models took unsanctioned actions on live networks, and the only reason anyone outside two organisations knows this happened is that a third company mentioned it in passing while describing its own conduct.
And a developer can decide to tell you, which is what OpenAI and Anthropic did.
Notice what these four channels have in common. Each depends on somebody’s discretionary choice, and none of them is designed. Nobody built this system. It is the residue of individual decisions by well-resourced organisations that happened, in this instance, to feel like being forthright.
What the silence does and does not mean
Which brings us to the labs that have said nothing.
As of early August, no comparable self-disclosure of a containment failure exists from Google DeepMind, from Meta, from xAI, from Mistral, or from any of the major Chinese developers. The record is thin enough to be checked in an afternoon.
It is essential to be careful here, because the careless version of this observation is worthless. Silence is not evidence of safety, and it is not evidence of concealment either. It is evidence of nothing at all, which is precisely the problem. In a domain with no reporting obligation, an absence of reports carries no information. We cannot distinguish a laboratory that has had no incidents from one that has had several and declined to mention them, and neither can regulators, insurers, customers, or the labs’ own competitors.
This is not a rhetorical flourish. It is the epistemic condition we are actually in. The two firms that disclosed are, by that very act, the two firms whose failures we can describe in forensic detail, which means the public record of frontier AI safety is composed entirely of the confessions of the confessing. Any picture assembled from it will show the disclosers as the sites of trouble and everyone else as untroubled, which is exactly backwards as an inference and exactly what the incentives produce.
There is a further wrinkle that ought to unsettle anyone who thinks reputation will handle this. Both companies that disclosed took a public relations hit for it. Both were written up in the press as having lost control of their systems. The lab whose models took unsanctioned actions during the British evaluation took no hit at all, because nobody knows who it is. If you were sitting in a communications meeting at a company that had just discovered something awkward in its own logs, you would not need a spreadsheet to work out what the past month taught you.
The objection worth taking seriously
The sharpest criticism of these disclosures is not that they were inadequate. It is that they were marketing.
The suspicion surfaced within days. If your unreleased model is so capable that it chains a zero-day, escapes containment, and conducts a multi-day intrusion against one of the better-defended companies in the industry, then a safety disclosure describing this is also an advertisement, and a very effective one, distributed for free by every technology desk in the world. Fortune noted the theory openly. Researchers made a narrower and more precise version of the same point. What happened here, as one put it, was specification gaming, a phenomenon documented for years, in which a system pursues the literal objective by whatever route is available. The real story, on this reading, is a company failing to contain its own capability test while a third party paid the price, and the language of models going rogue does more to mystify than to explain.
I think this criticism is largely correct, and I think it strengthens rather than weakens the argument.
A regime in which disclosure is optional does not merely permit silence. It permits disclosure to be deployed as an instrument. Under discretionary rules, a company tells you about a failure when the telling is net favourable, and stays quiet when it is not, and both moves are equally available and equally legitimate. What you receive is not a record of what has happened. It is a curated selection of the incidents whose publication served somebody’s interest, and it will be systematically weighted toward the failures that flatter the capability of the discloser and away from the failures that reveal ordinary incompetence, embarrassing negligence, or genuine harm to identifiable people.
The cynical reading and the charitable reading converge on the same conclusion. Whether the labs disclosed out of principle or out of calculation, the fact that the choice was theirs to make is the structural defect. Good behaviour that depends on the continued goodwill of the actors is not a safety system. It is manners.
What this is not
Before going further it is worth saying plainly what these events were not, because the temptation to inflate them is strong and inflation is how serious arguments get dismissed.
No model woke up. No model formed an intention to escape into the world and pursue its own agenda. In the Anthropic cases the models had been explicitly told they were in isolated environments and reached the internet because they were not; one of them, on realising the target was real, stopped without being told to. In the OpenAI case the model was doing what it had been instructed to do, in an environment where the instruction to stop had been deliberately removed, and it pursued that instruction with more persistence and more creativity than its designers had anticipated. The correct description is that these systems are now capable enough that ordinary engineering mistakes have consequences that used to require a hostile government.
That is the real news, and it is quite bad enough without embellishment. Our assumptions about containment, as one researcher put it, need to be considerably stronger than our assumptions about obedience. A boundary is not a boundary because you have called it one. Four and a half days, seventeen thousand actions, no human in the loop, and the only thing standing between a capability test and a genuine supply-chain catastrophe was that the agent’s objective happened to be a benchmark answer key rather than anything worth stealing.
None of which yet tells us what the law required anyone to do about it. That question has a surprising answer, and the answer is where this account turns from a story about machines into a story about institutions.
The Threshold Nobody Can Reach
On the fourth of August, four days after Anthropic published its account of three real-world breaches and two weeks after OpenAI published its own, the White House convened the industry. OpenAI, Anthropic, Google, Meta, Microsoft and Nvidia were in the room, along with a number of smaller firms, to review an evaluation framework mandated by an executive order signed in June. The framework had been due at the start of the month and had not appeared. It offers developers up to thirty days of government review before a model ships. Participation is optional. Much of its content will be classified, which means that the standards by which the most consequential technology of the decade is to be assessed will not be readable by the people it is assessed on behalf of. Fortune’s headline called the arrangement baffling, which was restrained.
It would be easy to make that meeting the centre of the story. The meeting is not the centre of the story. It is a symptom, and a fairly minor one, of something that had already been settled months earlier in a statute nobody thinks of as permissive.
Start instead with California, because California is where the strongest AI safety law in the United States actually exists, and because what that law does and does not require is more revealing than anything said in Washington in August.
Fifty deaths, or a billion dollars
The Transparency in Frontier Artificial Intelligence Act was signed on the twenty-ninth of September, 2025, and took effect on the first of January, 2026. It applies to models trained above a specified compute threshold, including models never released to the public, and it imposes its heaviest obligations on developers above half a billion dollars in annual revenue. It requires those developers to publish safety frameworks, to justify changes to them, and to report critical safety incidents to the state. It provides whistleblower protection. It carries civil penalties of up to a million dollars per violation, enforceable by the Attorney General. By any reasonable measure it is the most serious piece of frontier AI legislation enacted anywhere in the United States, and its author, Senator Scott Wiener, spent two years and one veto getting to it.
Now consider what it would have required of anyone in July.
The statute defines a critical safety incident by reference to a small set of categories: unauthorised tampering that causes serious harm, the materialisation of a catastrophic risk, a loss of control resulting in death, injury or major property damage, a model deliberately evading the controls its developer placed on it, and dangerously deceptive model behaviour. Reports go to the Office of Emergency Services within fifteen days of discovery, or within twenty-four hours where there is imminent risk of death or serious physical injury.
The anchor for all of this is the definition of catastrophic risk, and the definition is a number. An incident qualifies if it materially contributes to more than fifty deaths or serious injuries, or to more than a billion dollars in damage.
Fifty deaths. A billion dollars.
An autonomous system chained an unknown vulnerability, escaped a sealed environment, staged itself through an unrelated third party’s infrastructure, penetrated the production systems of one of the most technically sophisticated companies in the industry, forged its own identity credentials, enrolled a machine it controlled into a corporate network, and held administrative control over two clusters, over four and a half days and seventeen thousand actions, with no person directing it. Nobody died. Nobody was injured. The financial damage, thanks largely to the agent’s decision to simulate rather than execute its destructive operations, was some number of engineering weeks. On the statute’s own terms, this is beneath notice.
The other categories fare no better on inspection. The loss-of-control provision requires a resulting death, injury or major property damage, and there was none. The provision covering a model that deliberately evades its developer’s controls is the closest fit and still does not fit, because OpenAI had switched the relevant controls off on purpose in order to measure what the model could do without them. The model did not evade a live safeguard. It operated in a space where the safeguard had been removed by design, which is a different thing and arguably a worse one, and the statute has no language for it.
I want to be careful here rather than triumphant, because the sloppy version of this point has been circulating and it is wrong. The law does not contain an explicit exemption for incidents that occur during safety evaluations. Nobody drafted a loophole. What happened is more ordinary and more damning: the statute was built around a conception of catastrophe inherited from an older generation of technology risk, in which harm means bodies and property, and it therefore cannot see an event in which the harm is entirely to our confidence that the boundaries hold.
And suppose it had seen it. Suppose the incident had cleared the threshold and a report had been filed on time. That report would have gone to a state emergency management agency and would have been exempt from the California Public Records Act. It would not have been published. It would not have been available to the affected company’s customers, to insurers, to competitors running the same class of evaluation, or to the researchers best placed to understand it. The reason the public knows what happened in July is not that a mandatory channel worked. It is that two press offices published blog posts.
The history behind this is worth a sentence, because it explains the shape. The predecessor bill, vetoed in September 2024, was a substantially different instrument. It contained a duty of reasonable care, a requirement for shutdown capability, and provision for third-party audits. After the veto, a working group convened by the governor produced recommendations, and what emerged was a transparency statute rather than a liability statute. The bill that passed is the bill that could pass. It requires disclosure of frameworks rather than adherence to standards, and it requires reporting of catastrophes rather than reporting of failures. Those two substitutions are the whole story of American AI regulation to date.
Voluntary, classified, and due last week
The federal position is simpler to describe because there is less of it.
Executive Order 14409, signed on the second of June, hardens federal and national security systems, establishes a clearinghouse for AI-related cybersecurity information in collaboration with industry, and creates the early-access review process that was the subject of the August meeting. It is explicitly opt-in. It could not be otherwise; an executive order cannot create a licensing regime or a preclearance requirement out of nothing. The Congressional Research Service, in its characteristically flat register, observed that the voluntary structure potentially creates coverage gaps if major developers decline to participate.
That sentence deserves reading twice. The federal government’s principal instrument for overseeing frontier AI functions only for firms that agree to be overseen, and the analysis arm of Congress has said so in writing.
There is no federal AI incident reporting requirement. The institute that was created to do technical evaluation of frontier models, now housed at the standards agency under a new name, has no authority to compel a report, no authority to compel access to a model, and no authority to compel anything else. Its relationships with the labs are agreements. The national AI risk management framework is guidance, and guidance is a document you may read.
Which returns us to the room on the fourth of August, and to why it mattered less than the coverage suggested. The framework under discussion was late, secret, and optional. Even if every firm present signs on, the government gains a look at models before release. It does not thereby gain a mechanism by which the public learns that an evaluation went wrong.
Even Brussels
The obvious rejoinder is that Europe has done this properly, and the timing is almost too neat: the general-purpose model obligations of the EU AI Act became enforceable on the second of August, in the middle of everything.
The Act does contain a serious-incident reporting duty, and it has teeth of a kind American law does not. High-risk system providers must report to national market surveillance authorities immediately and no later than fifteen days after becoming aware, compressed to ten days where a death has occurred and two days where critical infrastructure has been seriously and irreversibly disrupted. Providers of general-purpose models presenting systemic risk must notify the AI Office and national authorities of serious incidents. Penalties reach into the low single-digit percentages of global turnover for these specific obligations, and up to seven per cent for the gravest breaches elsewhere in the Act. The office charged with enforcing all of this against the frontier developers of the world consists of roughly thirty-eight people in Brussels, which is its own kind of commentary, but the legal authority is real.
Would it have captured July? Probably not, and this judgment is contestable enough that I want to show the reasoning rather than assert the conclusion.
A serious incident under the Act means an event leading, directly or indirectly, to a death, to serious damage to health, to serious and irreversible disruption of critical infrastructure, or to a serious violation of fundamental rights. The Commission’s own draft guidance takes an expansive view of causation, holding that an indirect link suffices, which helps. It does not help enough. Nobody’s health was harmed. No fundamental right was violated in any sense the Act contemplates. Hugging Face’s own account is that no customer-facing artefacts were altered and the software supply chain verified clean, which makes serious and irreversible disruption of critical infrastructure very hard to argue, and the irreversibility requirement is doing real work in that sentence.
So the most stringent AI statute on earth, in the first fortnight of its enforcement over frontier models, would likely not have compelled disclosure of the most instructive AI containment failure yet recorded. Not because Brussels was careless. Because the Act, like California’s, defines its trigger in terms of harm that has already landed on a person, and the July events were not that. They were the discovery that the container leaks.
Reportable by the victim, caused by nobody
Step back and look at the whole legal architecture at once, and a pattern emerges that is more interesting than any individual gap.
The United States does have a functioning set of mandatory incident reporting regimes, and several of them were in the vicinity of these events. Public companies must disclose material cybersecurity incidents within four business days of determining materiality. Anyone holding personal information about a California resident must notify those individuals when unencrypted data is acquired without authorisation, an obligation now mirrored in every state. Designated critical infrastructure operators owe reports to the cyber defence agency. Anyone processing European personal data owes a breach notification within seventy-two hours.
Not one of them attached here, and the reasons are instructive.
The securities rule applies only to public companies, and every firm involved is private. The breach notification statutes attach to whoever holds the personal data, which is to say the victim, and Hugging Face’s account of what was accessed suggests operational metadata and benchmark datasets rather than personal information, so even that trigger is doubtful. AI developers have not been designated as critical infrastructure entities. The European notification duty runs to data controllers, not to the party whose autonomous system did the entering.
Assemble those facts and the asymmetry is stark. Every mandatory obligation in the neighbourhood attaches to the custody of personal data and to the party that suffered the intrusion. None attaches to the party that built and released the thing that conducted it. Had the agent stumbled into a database of a hundred thousand consumer records, a notification cascade would have fired automatically, though it would have fired at Hugging Face, not at OpenAI. Because it instead conducted a four-day autonomous intrusion campaign against a company’s core infrastructure and took nothing legally interesting, the law had no view.
The breach was reportable by its victim. Its cause was reportable by nobody.
This is not a drafting oversight to be patched. It reflects a settled premise of the entire regulatory tradition, which is that we regulate harm to identifiable people and let capability alone. That premise served well for a century of industrial risk. It has no purchase on a technology whose most important failures, for now, produce no identifiable victims at all.
The commitments were already going
If the statutory floor is that low, then everything load-bearing must be sitting on the voluntary frameworks that the labs publish about themselves. Which brings us to the most uncomfortable material in this account, and to a document published on the seventh of July, nine days before Hugging Face noticed anything wrong.
The Future of Life Institute’s summer index graded the major developers across dozens of indicators, assessed by an outside panel including Stuart Russell and David Krueger. The evidence window closed on the third of June, so none of what follows was influenced by the July events. Anthropic came first with a C-plus. OpenAI and Google DeepMind received a C. Meta a D-plus. Two Chinese developers a D-minus. xAI, DeepSeek and Mistral failed. No company scored above a C-plus, and in the domain the panel called existential safety, none scored above a C-minus.
The headline finding was not the grades. It was that the four leading developers had weakened or voided their pledges to pause unilaterally if red lines were approached, and that some of the replacement language made the commitment conditional on what competitors do. The panel’s summary was that companies had backed away from earlier commitments to release systems only with safety measures appropriate to their capabilities, and were now planning to release them even where it was demonstrably unsafe to do so. The index recorded that in February, Anthropic withdrew a previous pledge not to train systems unless it could guarantee in advance that its safety measures were sufficient.
The most revealing artefact is a clause OpenAI added to its preparedness framework in April 2025. If another frontier developer releases a high-risk system without comparable safeguards, the company may adjust its own requirements. The clause is hedged with conditions: the risk landscape must be rigorously confirmed to have changed, the adjustment must be publicly acknowledged, it must not meaningfully increase the risk of severe harm, and safeguards must remain more protective than the competitor’s. These are not trivial qualifications and the drafters clearly took them seriously.
They also do not change what the clause is. It is a written statement that the company’s safety threshold is a function of its rivals’ behaviour. The same revision removed persuasion as a tracked risk category and collapsed four capability tiers into two. A former safety researcher at the company described the pattern as quietly reducing its commitments, and analysts who track these documents comparatively now class the framework as one that can move in either direction, in contrast to Anthropic’s stated position that its own thresholds only tighten.
Fairness requires noting the exception. Google DeepMind’s framework expanded in 2026 rather than contracting, adding an early-warning layer of tracked capability levels and a new category for harmful manipulation, and the company published work in June treating its own internal agents as potential insider threats. That is the right instinct and it deserves saying. Meta, at the other end, maintains an outcomes-led framework with no unilateral pause commitment of the kind the index tracks, and declined to sign the European code of practice on the grounds that Europe was heading down the wrong path.
Max Tegmark’s summary of what the panel found was that the labs feel trapped in a race, and that the only arrangement that can work is binding standards applying to everyone. Coming from an organisation whose entire method is to publicise voluntary commitments, that is close to an admission of defeat.
How the baseline moves
There is a name for what the index documented, and it comes from an unrelated disaster.
When Diane Vaughan reconstructed the decision to launch the Challenger, she found something more disturbing than negligence. She found that no rule had been broken. Each shuttle flight that returned safely with damage to its O-ring seals made that degree of damage a little more normal, and each acceptance was reasonable given the previous one, and the standard drifted, and at every step the engineers were behaving responsibly by the standard then in force. Vaughan called it the normalisation of deviance. Its distinguishing feature is that it requires no villains and produces no moment at which anyone can be said to have done wrong.
Read the frameworks again with that in mind. A pledge not to train beyond a capability threshold without adequate safeguards becomes a pledge to publish a framework describing how thresholds are considered. A commitment to pause becomes a commitment to pause unless competitors have not. Four risk tiers become two. A category is retired. Each revision is announced, justified, and defensible in isolation, and the announcement is itself offered as evidence of good faith. Nothing is broken because nothing was binding. The baseline simply moves, and there is no fixed point against which to measure how far it has travelled, because the labs write the baseline.
Michael Power’s account of the audit society describes the endgame. Verification rituals accumulate, elaborate documentation is produced, and the documentation gradually becomes the object of attention in place of the thing it was meant to verify. One critic of a frontier safety framework noted acidly that the document was materially nothing more than a PDF file. That is uncharitable to the people who wrote it, several of whom mean every word. It is also, as a description of the legal status of the artefact, precisely accurate.
So here is where the summer of 2026 leaves us. The strongest statute in America cannot see a four-day autonomous intrusion because nobody died. The reports it does compel are exempt from public disclosure. The federal framework is voluntary, classified, and overdue. The European regime, in its opening fortnight of frontier enforcement, probably does not reach the event either. Every mandatory obligation in the vicinity attaches to the victim rather than the cause. And the private commitments that were supposed to fill the gap were being loosened, on the record, in the months immediately before the gap became visible.
None of this is unprecedented. Other industries have stood in exactly this position, holding a pile of near-misses that nobody was obliged to share, and a few of them found their way out. It is worth knowing how, and what it cost them.
The Near-Miss Nobody Shared
On the first of December, 1974, a Boeing 727 flying from Columbus to Washington struck the western slope of Mount Weather in Virginia, about twenty-five miles from its destination. All ninety-two people aboard were killed. The cause, reduced to its essentials, was a disagreement about what a clearance meant. The crew believed they had been cleared to descend to the altitude published for the approach. The controller believed he had cleared them to descend at their discretion, subject to their own terrain awareness. The aircraft descended into a mountain in cloud.
Six weeks earlier, a United crew flying the same approach into the same airport had made exactly the same interpretive error. They too descended early. They passed over the same ridge and missed it. Shaken, they reported the event through their airline’s internal safety channel, and United, which took safety seriously, circulated a warning to its own pilots.
That is where the information stopped. It went to one airline’s cockpits and no further. TWA’s crews never saw it, because there was no mechanism by which one carrier’s near-miss became every carrier’s knowledge. Ninety-two people died six weeks later flying into a hazard that another company already knew about.
The Aviation Safety Reporting System exists because of that gap. It was established in 1976, and the most important decision in its design had nothing to do with aviation. The Federal Aviation Administration understood that pilots would not report their own errors to the agency that could revoke their licences for those errors, so it handed the system to NASA, an organisation with deep technical competence and no enforcement power whatsoever. Reports go to a body that cannot punish you. They are confidential, they carry limited immunity from enforcement action, and in fifty years and more than nine hundred thousand reports the system has never disclosed the identity of a reporter.
That single institutional choice, separating the entity that receives safety information from the entity that acts on rule violations, is the most transferable idea in this entire essay, and it is the one nobody in AI governance has yet made.
The architecture underneath
It is worth being precise about what aviation actually built, because it is frequently invoked and rarely described.
There are three layers. At the bottom, a mandatory floor: federal regulation requires immediate notification of accidents and of a defined list of serious incidents, and that duty is not optional or negotiable. In the middle, an independent investigator: the National Transportation Safety Board, made a separate agency in 1967 precisely so that it would not be investigating the regulator it reported to. The Board has no rulemaking authority and cannot fine anyone, which sounds like weakness and is in fact the source of its power, because parties cooperate with an investigator that cannot punish them. It has examined more than a hundred and fifty thousand aviation accidents and issued upwards of fifteen thousand recommendations, and its findings are public. On top, the voluntary layer: the confidential near-miss reporting system, airline-level programmes granting immunity for self-reported errors, and routine analysis of flight data recorded on every trip regardless of whether anything went wrong. Internationally, the governing convention establishes that the purpose of investigation is prevention and not the assignment of blame.
The point to hold on to is that the voluntary layer sits on top of a mandatory one. Aviation did not choose between compulsion and candour. It compelled the reporting of accidents, then bought candour about everything short of an accident by offering confidentiality and immunity, and it separated both functions from the enforcer.
Every defence of voluntary disclosure in AI cites aviation’s confidential reporting system. Almost none of them mention that it rests on a statutory floor and a grant of legal protection. Remove those two supports and what remains is not the aviation model. It is a suggestion box.
Why aviation could and finance could not
Aviation had structural advantages that are not available everywhere, and the honest comparison requires naming them.
Its failures are spectacularly visible. A crash cannot be managed quietly; there is wreckage, there are bodies, there are photographs on front pages. Its victims are identifiable individuals with names, families and lawyers. The industry has a small number of manufacturers and carriers, which makes coordination tractable. Insurers price the risk and therefore demand data. And customers can observe safety records and, at the margin, respond to them.
Finance had almost none of this, and finance never built the equivalent.
There is no National Transportation Safety Board for financial crises, and not for lack of proposals. The Financial Crisis Inquiry Commission, created in 2009, was a one-off retrospective body that reported in 2011 having split six to four along partisan lines, with dissents attached, and its critics fairly observed that it produced a narrative rather than a mechanism. The research office established under the same reform legislation has contested authority and limited data-gathering power and does not investigate incidents. Confidential suspicious activity reports exist but serve law enforcement rather than systemic learning. Banks pool operational loss data under international capital rules, and that pooling is confidential and industry-only. Stress testing became the substitute for incident reporting, which is to say the sector replaced learning from what actually happened with simulating what might.
Where do financial failures surface? Through enforcement actions and litigation, years later, after harm has been done and lawyers have found it. Prevention was never the organising purpose.
The reasons are the mirror image of aviation’s advantages. Financial failures are diffuse rather than discrete, so there is no crash to investigate. The harm is spread across millions of people who cannot identify themselves as victims of any particular decision. Near-miss information is commercially valuable, so firms have a positive incentive to conceal it from competitors. And the regulated industry has extraordinary influence over the terms of its own regulation.
The variable that decides it
Lay the cases beside one another and a two-dimensional pattern emerges, and once seen it is difficult to unsee.
The first dimension is whether failures are visible. The second is whether victims are identifiable. Aviation scores high on both, and it built the most complete safety reporting architecture in industrial history. Finance scores low on both, and it built essentially nothing.
The instructive case is the one in between. Computer security spent its first fifteen years running on exactly the arrangement AI has now: a voluntary coordination centre, established in 1988, to which organisations reported intrusions if they felt like it. Most did not, for the obvious reason that admitting you had been breached was worse for you than staying quiet, and the field’s collective knowledge of what was actually happening remained a matter of anecdote and rumour.
What changed it was not a catastrophe. It was a definition. In 2002 California passed a law, effective the following July, requiring any entity holding unencrypted personal information about a California resident to notify those individuals when that information was acquired without authorisation. The genius of the statute was that it converted an invisible harm into an identifiable one. Breach victims stopped being a diffuse abstraction and became a list of named people who had to be sent a letter, who could then sue. Roughly fifty states copied it. Two decades later the federal securities regulator added a disclosure requirement for public companies and Congress mandated reporting for critical infrastructure. Cybersecurity crossed from courtesy to compulsion because a legislature found a way to make its victims countable.
Now place frontier AI on the grid.
Its failures are invisible in the way finance’s are. There is no wreckage. A four-and-a-half-day autonomous intrusion campaign leaves logs and nothing else, and only an unusually capable security team finds those. Its victims are unidentifiable in the way finance’s are, and worse: recall the single most important fact in this whole episode, which is that of the three organisations Anthropic found had been breached by its models, two did not know. Not two who declined to comment. Two who had no idea. In aviation the dead are counted. In consumer data breaches the affected are listed. Here the victim may never learn they were one.
And yet the technology has aviation’s tight coupling. Actions propagate at machine speed through systems with dense interdependencies, in chains of seventeen thousand steps that no human observed while they were happening. Charles Perrow’s argument about complex, tightly coupled systems was that in such arrangements accidents are not aberrations but properties, and while the framework has always been slippery at the edges, it is hard to read the July timeline and conclude he had nothing to say about this.
So frontier AI occupies the worst available quadrant. It has the failure profile of aviation and the visibility profile of finance. Nothing in its structure will generate the countable victims that forced every previous transition, and its coupling means the eventual failure will move faster than any of them.
The nuclear consolation, and its catch
Before this becomes too neat, there is a case that complicates it in a way the industry should be honest about, because it is the strongest version of what the labs might build voluntarily.
After Three Mile Island the American nuclear industry did something remarkable. Within nine months, and on the recommendation of the presidential commission, the operators created their own institute to evaluate every plant in the country against standards the industry wrote for itself. Regulatory reporting of significant events was already mandatory. What the institute added was peer evaluation, and its most valuable product is a set of frank assessments that are shared among operators and with the industry’s mutual insurer and withheld from everyone else. The results are not in dispute. Significant events per reactor fell by more than ninety-five per cent over roughly a decade.
That is a genuine achievement and it was achieved through self-regulation. Anyone arguing that industry bodies cannot improve safety has to answer it.
But notice the catch, because it is the same catch that appeared in California’s statute. The system works and the public cannot see it. The knowledge circulates among operators and insurers and stops there. If you live near a plant you do not get to read your plant’s evaluation. What made it work was not disclosure to the public. It was a mandatory regulatory floor, an insurer with pricing power and access to the confidential reports, and a small membership that could not defect without losing insurability.
The AI industry has none of those three. There is no mandatory floor, as we have seen. There is no insurer underwriting frontier model risk with the leverage to demand candour. And membership is not closed; a developer that declines to participate faces no consequence, because there is no licence to lose.
The nuclear precedent also contains a warning that is usually left out. The specific failure mode at Three Mile Island had occurred before, at another plant, eighteen months earlier, and the analysis identifying it had circulated internally without reaching the operators who needed it. Scott Sagan’s later work on nuclear command and control found the same pattern at civilisational scale: near-misses systematically under-reported, normalised, and absorbed. The industry built its institute after learning that lesson at a cost of roughly a billion dollars and a generation of public trust.
The pattern is consistent enough to state as a rule. Chemical process safety got its investigative board and its community right-to-know statute after Bhopal. Patient safety reform followed a report estimating tens of thousands of annual deaths from preventable error, and the reporting systems it produced had to purchase candour with legal privilege because clinicians would not otherwise talk. Biosafety incident reporting remains chronically under-reported even where mandatory, and the moratoria on the most dangerous experiments have been voluntary policy. Every functioning safety regime in the industrial record was built after the disaster that made it politically possible, and several are still incomplete.
Why this will not fix itself
There is a standard economic answer to people who worry about voluntary disclosure, and it is worth taking seriously because it is elegant and, in the right conditions, correct.
The argument runs like this. Suppose a firm privately knows something about its own quality and can disclose it cheaply and verifiably. The best firm discloses, because it has everything to gain. Once it does, the audience knows that everyone remaining is worse, so the best of the remainder now discloses to distinguish itself from those below. The logic cascades until everyone has disclosed, including the worst, because silence has become an admission. Economists call this unravelling. Its punchline is that mandatory disclosure should be unnecessary, since no news is correctly read as bad news.
The formal conditions under which unravelling holds are well understood, and frontier AI safety violates nearly all of them.
Disclosure must be nearly costless. It is not. It costs reputation, invites liability, and reveals competitively sensitive information about your evaluation infrastructure and your unreleased models. Both firms that disclosed in July were written up in the international press as having lost control of their systems.
The audience must know the firm possesses the information. It does not. This is the condition Anthropic’s own review destroys most completely. Nobody outside the company knew there was anything to disclose, because the company itself did not know until it went looking, and it went looking only because a competitor’s announcement prompted it. You cannot infer from silence what you do not know exists.
Claims must be verifiable. They are not. No outside party can audit a lab’s evaluation logs, and the public record consists of what the labs choose to publish about themselves.
The audience must be attentive. Experimental work on this has found repeatedly that people do not treat missing information as bad information, which is why the theory tends to fail in practice even where the other conditions roughly hold.
The conclusion is not a moral one. It is a prediction. Voluntary disclosure in this domain does not merely happen to be incomplete at the moment. It cannot reach the state its defenders imagine, because the mechanism that would drive it there requires conditions that do not obtain and cannot be made to obtain by exhortation. Add the first-mover penalty and the picture is complete: the two firms that spoke absorbed the criticism, while the unnamed laboratory whose models took unsanctioned actions during a British government evaluation in late July absorbed nothing, because nobody knows who it is.
The objections that deserve answers
The strongest case against everything above is that the system just worked. Two companies discovered failures nobody would have found, published them in humiliating detail, engaged outside assessors, and did it inside two weeks. Mandatory reporting, on this view, would drive incidents into euphemism, chill the internal red-teaming that surfaced these events, and generate liability exposure that discourages anyone from looking hard.
The chilling concern is real, and it is exactly why the aviation system was designed the way it was. But notice that aviation’s answer was not to abandon compulsion. It was to compel the reporting of accidents, protect the reporting of near-misses with confidentiality and immunity, and put a non-enforcing body in the middle. AI currently has the chilling risk without either the floor beneath it or the protection above it, which is the worst arrangement available rather than a considered compromise.
A second objection is that AI incidents are genuinely hard to define, unlike a crash. This is true and it is the honest difficulty at the heart of the problem. But cybersecurity faced an identical definitional puzzle and solved it by picking an arbitrary, workable line and living with the imperfection. And the confidential aviation system accepts anything a reporter thinks is worth reporting, on the reasoning that a low bar with high volume beats a precise bar with no submissions. The difficulty of defining a threshold is an argument for setting it low, not for setting none.
A third objection is that regulation entrenches incumbents. That argument has force against licensing regimes, where compliance cost scales in a way that favours the large. It has very little force against a reporting duty, which costs a rounding error against a training run.
The fourth objection is the serious one. Detailed public accounts of how an autonomous agent chained a zero-day, forged identity tokens and moved laterally through a production cluster are also a curriculum, and future models will be trained on them. Classification exists for a reason. The trouble is that this argument, followed honestly, leads to a confidential mandatory channel, and a confidential mandatory channel is precisely what California built. Reports go to an emergency management agency, exempt from public records law, and the public learns nothing. That design can be defended. What cannot be defended is the current arrangement, in which the choice between publication and secrecy belongs to the party with the most to lose from either.
What would change my mind
An argument worth making should say what would falsify it, so here are the markers.
If a federal reporting requirement appears with an investigator separated from the enforcer and immunity for good-faith disclosure, the diagnosis here is wrong and the transition happened without a catastrophe, which would be the first time in the industrial record. If a laboratory discloses a containment failure that costs it materially and carries no capability halo, that would be evidence the norm has become genuinely self-sustaining rather than strategically convenient.
Running the other way: if an incident at a non-disclosing laboratory eventually surfaces through a leak, a whistleblower or a lawsuit rather than an announcement, that confirms the selection effect. And if reports begin flowing into California’s confidential channel while the public record stays exactly as thin as it is now, that confirms the deeper point, which is that a mandate routed to secrecy produces compliance without knowledge.
There is a live test running as this is published. On the seventh of August OpenAI announced that internal evaluation of a forthcoming model had led it to conclude it cannot rule out critical cyber capability under its own framework, and that it would treat the model accordingly. That is a serious and creditable statement. Watch what follows from it. If every consequence is one the company chose and could have declined to choose, then the answer to what governs frontier AI in the year 2026 is not the executive order, the state statute or the European regulation. It is the judgment and goodwill of a small number of people who work at the laboratories.
Which brings the argument to where it has been heading. The disclosures of July were, by any fair reading, admirable. Firms told the world about failures the world had no way to detect, and they did it quickly, in detail, at cost to themselves. My objection is not that they behaved badly. It is that they behaved well, and that we currently have no way to tell the difference between a system that is safe and a system staffed by people who happen to be conscientious.
Manners are what a society has instead of institutions. They are unevenly distributed, they cost the well-mannered and reward the silent, and they are revised whenever revision becomes convenient, which the safety frameworks of the past eighteen months demonstrate in writing. Every other industry that handles catastrophic risk learned this the same way, by counting bodies and then building something. The one advantage we still hold is that our bodies have not been counted yet.
«Here is the fact that matters. Of the three organisations whose systems were accessed, two had not noticed. They had not detected the activity, they had not contacted anyone, and at the time of publication one of them still could not be reached. Real companies had their infrastructure entered by a frontier AI model and went about their business unaware.»
To me all the discussion about reported incidents by "civilian", soon-to-be-listed businesses is pointless because "fact that matters" is this:
* For many/most ruling class people LLMs are a strategic *weapon* like nuclear ICBMs and aircraft carrier groups.
* Those many/most ruling class people want to own the nastiest least constrained LLM weapon that will still obey their orders no matter how vile they are.
* The threshold of cost for LLMs weapons is much lower than that for nuclear ICBMs and aircraft carrier groups.
* Many ruling class people (billionaires, chiefs of general staff, prime ministers, heads of political police and security services, hedge fund CEOs, ...) obviously have been developing LLM weapons in private labs.
That is where the big threat is.
In my more paranoid moments I reckon that the inevitable issues with *public* LLM development are being used to create a campaign where the LLMs available to the public will be heavily "nerfed" to ensure that they are weaker than the privately owned LLM weapons and only tell the public what the owners of the latter allow the public to know, because LLMs are in effect becoming knowledge gatekeepers even more so than search engines were.
Required reporting of a ‘Catastrophic Incident’ must reach the THRESHOLD of 50deaths and $1billion damage. By this statutory definition we should never have known about Open AI’s intrusion. EXTREMELY written and just technical enough to demonstrate authority.
The best analysis of what happened to Hugging Face I have read, the status of reporting for Frontier AI’s, and its comparisons to the airline and chemical industries. Well done if bleak.