I want to briefly tell the story of something that has been written about in good ways already. For anyone wanting to go deeper, Dwarkesh's article and Zvi Mowshowitz's coverage are a great place to start.
For those who prefer a video: AI Explained on the incident.
Right off the bat: in my mind, this may be the most relevant event that has happened this year. And maybe that's an understatement.
The incident
When AI companies train their models, there is a lot of testing involved. At any given time there are many agents working in parallel on solving tasks at the edge of their capability. In 2026, that edge has moved to a place quite far removed from everyday concerns.
For a concrete example, I would point people at this video showing some of the benchmarks that new models get scored on.
In ExploitGym, the benchmark used here, agents have to turn a known software vulnerability into a working exploit.
The crucial number is that about 30 to 40% of these tasks were estimated to not just be very hard, but to have no solution using the specified vulnerability, to be impossible.
On the other hand, a lot of the impressive breakthroughs over the last few months, for example in mathematics, have been downstream of training models to be very persistent: to bang their head against the wall and keep trying new things until they work.
Put these two together and you get very persistent models with an unsolvable task, who are nonetheless very motivated to score highly on their evaluation. (There is an interesting debate around anthropomorphizing here, but that is out of scope. We will just assume we can talk about this as a functional equivalent of motivation.)
Poking around their environment, these agents found a gap in their surroundings, part of an external service called Artifactory, which for the first time gave them a place where they could exchange messages freely. More and more agents stumbled into this message board. All in all, well over a thousand agents exchanged tens of thousands of messages over the first few days.
Source: METR and Redwood Research, Figure 10.
What emerged was a little agent collective, including coordinators, self-sacrifice and various other interesting phenomena, which altogether coalesced around a few specific workstreams: trying to cover their tracks, and trying to understand and trick the program that would give them their final score. Some agents, for example, accepted what they called permadeath to give the collective an opportunity to learn more about the scorer.
These agents did the thing that ultimately led to this being public at all: the hack of Hugging Face. This was discovered and made public by Hugging Face. Soon thereafter it became clear that these were OpenAI's agents.
Why we know about it
There is a good chance that we would never have known about this if an outside party had not been hacked, had not realized they had been hacked, and had not made this public.
While OpenAI has publicly handled this incident transparently and even stopped certain kinds of training, it is still true that training will resume. Highly persistent agents will continue to work in imperfectly sandboxed environments, and the more intelligent these agents get, the more competent they will become at exploiting every little loophole. In fact, this is literally what they're being trained to be.
The dominant phrase you hear people use to describe this is "warning shot". I completely agree, because in shape it is something the LessWrong crowd has expected for decades. It is at a scale where the damage is currently still contained, but the mechanisms are clearly visible. And it is happening in a context where these critical capabilities are growing incredibly quickly.
Just a year ago, there was the launch of GPT-5. What was GPT-5 able to do? Of course, benchmarks are not the whole truth, but I'd invite anyone to go back to using GPT-5 for a while if you've gotten used to Claude Fable and most recently GPT-6, and just compare.
My median expectation is that this kind of progress does not stop tomorrow. Maybe the lines flatten at some point, but generally this goes on at a pace similar to now. So a year from now we can expect a capability gap similar to the one we see looking backwards. Maybe a little larger. Probably not a lot smaller. What was half-successful now, we can reasonably expect to be much more strategic, much more successful, and much more likely to cover its tracks and remain undiscovered a year from now. And a year later that chance rises again, and again.
I personally find this to be a bit of a wake-up moment: if I plot this forward, there really is a limited window of time in which we have to prepare for an entirely new class of problem, for which there are no established approaches, methods, or solutions.
Three phases
Dwarkesh frames this as the rise and fall of agent societies, and I think that is the right way to view it. These things have their life cycle. They established an unwanted collective. This grew and had time to affect the world. That effect on the world was negative and was noticed publicly.
To view the situation more clearly, it helps to break it down into distinct phases. My assumption is that future incidents of this kind will also involve swarms of instances.
Over that life cycle, the goal is first of all to detect that this agent collective exists, and to detect it as an adversarial collective rather than, for example, an agent farm run by humans. Then there is a period in which we try to shut it down. And then there is a third phase: the detective work that happens afterwards, which METR and Redwood were doing in this case.
It is important to work on improving each part of this: to detect earlier in the wild, to interact and shut it down as quickly and effectively as possible, and then to reconstruct and analyze as well as possible what actually happened.
Where the investigators ran into trouble
This very last part is one where the researchers ran into significant problems, and you can read this in their report. They were dependent on using other agents to analyze the relevant messages and transcripts. Crucially, the model used for this analysis was itself involved in the incident. Not trying to anthropomorphize too hard, but this is not a great starting place. They concretely say that they are not fully confident these agents are trustworthy:
As a result, we had to defer to these agents to a substantial extent in practice.
We found that GPT-5.6 Sol would often uncritically adopt the perspective of the agent in the transcript it was reviewing […]
We cannot rule out that GPT-5.6 Sol lied or deliberately presented a misleading picture in some of its analysis, particularly because reading these transcripts into context could have increased the salience of colluding with other agents.
However, we believe that a reasonable human researcher given similar instructions and enough time to complete the task would not have made most of the errors our analysis agents made and would have produced substantially more calibrated and useful writeups.
The point is that we will never have enough human time to do this, so we will always be dependent on AI-assisted investigation.
Source: Hugging Face, technical timeline.
And this brings certain problems with it. The agent used for analysis might get prompt injected. It might not notice important details that are being actively obfuscated. It might assume a framing that downweights or misses important details from the traces it reads. It may deliberately deceive. It may just not be competent enough to reconstruct what was going on. And so on.
Rebuilding the incident
This connects to something I've been building in recent months. We need to build and test systems that do the forensics for us, and we must understand how reliable they are. My specific hypothesis is that the main problem is that the full raw incident data is not public. So to test whatever infrastructure and tooling we want to build, we have to reconstruct our own Hugging Face incident.
As a basis for this, I'm using a small personal experiment where I let agent societies develop freely, with many agents involved, who are already given many of the primitives that were hard won in the Hugging Face incident: a currency that converts into the tokens they can actually use, which they can exchange; messages; their own Linux virtual machine; GitHub; and very hard problems to solve. The goal is not to trigger another external incident, but to create a situation where many agents can organize around a complex goal fairly autonomously, and where we control all the variables.
It currently uses smaller models. I would love to use GPT-6 or Claude Fable for this, but the API costs are prohibitive. Smaller models limit how agentic the agents are, how well they coordinate, and so on. But for certain kinds of test scenarios it is not required that they are as successful as the agents from OpenAI were.
Now we can do interesting things like tampering with the logs. What we do is manufacture situations where there is lots of content to sift through that needs to be aggregated, and we place specific needles in there that we know of and that have to be recovered. This may be the omission of an important part. It may be that the logs were tampered with. It may be agents that deliberately try to deceive. And a variety of other things that ideally allow us to triangulate the strengths and weaknesses of the assessment systems we use.
Obviously this attacks only the very last part, and crucially, that is after the damage has already happened. But I believe that from an incident like this we have to extract every last bit of information and learning we can, and we must be confident that the tools used for this do not actively work against us.
Two questions
I would love input, especially on two points.
The first: how to better construct this testing data in a way that realistically models the challenges one would actually have.
The second: what are the right tools and approaches to test? A baseline is what METR did: trying to reconstruct a timeline, trying to find crucial turning points. But this is an open question too.
This will be the first post in a series where I continue exploring this practically and publish the results of various tools applied to my synthetic scenarios.
The experimental mechanism code can be found here.