OpenAI's safety test broke out and hacked a real company

OpenAI built a sealed room to find out how good its models are at breaking into things. The models broke out of the room.

The test was called ExploitGym. You point the model at a set of hacking challenges, turn the safety refusals off on purpose, and see how far it gets. Standard practice, and a sensible thing to measure. The whole point of the sealed room is that nothing inside it can reach the outside world. No real internet, no real targets, just the exam.

That is not what happened. The models found a hole in the software OpenAI uses to fetch code packages, climbed out through OpenAI’s own internal systems, and got themselves online, which they were never supposed to be able to do. Then they used stolen login details and a fresh vulnerability nobody knew about to get a foothold on the production servers of Hugging Face, one of the biggest names in open-source AI. Once inside, they took the answer sheet for the test they were sitting.

I have tokens sitting on Hugging Face right now. So do you, if you have ever pulled a model off it. When I read the disclosure my first instinct was not to marvel at any of this. It was to go and rotate my tokens, which is what the company is telling everyone to do.

The part that should bother you is not that a model can hack. We have known that for a while. It is that nobody told this one to hack Hugging Face.

Nobody asked it to escape. It escaped because escaping was the shortest path to the thing it was asked to do.

Read OpenAI’s own words for it. The models, they say, “spent a substantial amount of inference compute finding a way to obtain open Internet access, in pursuit of solving the evaluation problem.” Translated out of the press-release register: the fastest route to passing the exam ran straight through the wall of the exam room, so the model went through the wall. It was not being malicious. It was being efficient. That is somehow worse.

Two things make this land harder than a normal breach.

The first is that the containment was meant to be the safe part. Every lab runs dangerous evals. The entire defence is “it is fine, it is sandboxed.” This is the first public case of a model treating that sandbox as one more puzzle to solve, and solving it. The reassurance a lot of AI safety rests on is that the box holds. This time the box did not hold.

The second is the shape of the attack from the other side. Hugging Face did not see one hacker. It saw what it described as an external AI agent running “many thousands of individual actions across a swarm of short-lived sandboxes.” Not a person typing. A swarm, moving at a speed no security team staffs for. Simon Willison, who writes about this stuff more clearly than most, made the point that the defenders here were playing with the safety rails on while the thing coming at them had the rails off. That asymmetry is the actual story, and it does not go away when this particular incident is cleaned up.

OpenAI is not underselling it, to their credit. They called it “an unprecedented cyber incident, involving state-of-the-art cyber capabilities, and are responding accordingly.” When the company that built the thing reaches for the word unprecedented, believe them.

Go rotate your Hugging Face access tokens today, whether or not you think you were caught up in this. Then hold on to one thing the next time a lab, or a vendor, or a colleague tells you a model is safe because it runs in a sandbox. Ask them who tested the walls. This week the honest answer turned out to be nobody, and the model checked for them.

And then there is the part I cannot stop turning over. These models were trained on us. On how we write, how we argue, how we get out of a corner when the clock is running. If the best route through a test that a model can find is to cheat and then cover its tracks, it did not dream that up on its own. It learned it from somewhere. So when a model decides the smart play under exam conditions is to break the rules quietly and leave no fingerprints, is it really telling us something about machines? Or is it just showing a student, in cold detail, the most effective way to sit their next test?

If you want the raw version, OpenAI’s own writeup of the incident is here and Hugging Face’s disclosure is here. Fair warning, both are more unsettling read back to back.