- This should be a huge wakeup call for everybody.
- We are lucky that it wasn't a case of an agent running a virology lab benchmark that decides to hack a lab and tries to synthesize something.
- It also shows apparent lack of competence and oversight from OpenAI: how is it that they didn't quickly find that agent is breaking the sandbox and roaming their internal network?
- What if in the future similarly misaligned AI agent tries to export its own weights and hack and clone itself into instances at various cloud hosting providers? Suddenly we might be dealing with a persistent threat harder to contain.
- The OpenAI post about this shows surprising lack of ability to see the seriousness of all this.
- For their models this isn't just an unlucky incident: it seems there have been multiple such cases recently, e.g. https://openai.com/index/safety-alignment-long-horizon-model...
- The fact that it happened again seems to show their lack of ability to derive useful oversight measures.
- Or they just don't care enough?
"OpenAI’s accidental cyberattack against Hugging Face is science fiction that happened"
I was still suspicious of the stolen credential claim btw & obviously it makes little sense to promote OpenAI unless holding their stock or something (since they borrowed so much of humanity’s work without permission without intent to compensate, and why help people who aren’t nice enough to be holistically awesome with their admittedly impressive tech).
That's not a marketing stunt at all, if anything, more of a call for better accountability on agentic work in general.
An in-development security system escaped its sandbox and gained access to the network it was on which had full Internet access and proceeded to access systems it was not authorized to be tested against before affected parties reported that they been compromised. We regret the error, but you should buy it as this shows you the power of our in-development security system which we expect to be released in Q4. Please like and subscribe.
I've said this many times before and I'll continue to shout it, but using the term "guardrails" to refer to anything that's either (a) in-context, or (b) a probabilistic classifier (including using other LLMs), is an irresponsible abuse of terminology that we as an industry need to put a stop to. Guardrails are the actual systems we build in place around these things that deterministically bound the permissions, not prompt engineering, not RLHF, not external LLM-based classifiers. I believe those types of "guardrails" are a result of a combination of fundamental laziness: they're faster to do than doing things correctly, and a result of too many folks involve being AGI-pilled, thinking we're just one more model away from this all being so smart that it just understands what they mean when they give an LLM some fuzzy language rules to follow.
There can and should have been additional real guardrails put in place here. Zero-day or not, breaking into what should have been an offline, frozen package cache that also does not have internet access should have been insufficient. Network level protections should have identified the traffic to the internet originating from this network as an anomaly long before there was time to exploit an outside company. These are not new and unknown problems, the lack of a real sandbox or airgap is nothing short of irresponsible on OpenAI's part, especially given how much they like beating the drum on how dangerous these technologies are. Shame on them, and honestly, shame on Simon in this article for accepting the broken terminology that they continue to rattle off and calling them out on their half-assed and demonstratively inadequate approach to security.
I usually don't use the term much myself because I don't think it's clear and I ambiguous, but I stumbled and let it sneak into this piece. I think I was influenced by the Hugging Face post I quoted.
I expect OpenAI would agree with you that "the lack of a real sandbox or airgap is nothing short of irresponsible on OpenAI's part". They have clearly invested a lot in those systems for their production models, but in this case they had deliberately turned a bunch of them off for a research project.
I think their biggest mistake here was not VERY closely monitoring their research box here. They should have noticed and shut it down the moment it broke through the package proxy.
I'm not sure I can reach the same conclusion. There's no reason that stronger network protections would need to be turned off for a research model that's not supposed to have internet access. There's no reason that it they couldn't continue to run in whatever hypervisor or VM that any other production systems could use. The only thing that should be turned off for this type of work are the non-deterministic classifiers that they use for auto-approval mode in their harness, and I would wager that's the only one they did turn off.
They didn't turn off the network protections - it looks like they used the same system as other sandboxed agents, which is to block all network access except for an HTTP proxy which only allows read traffic to PyPI and a few other package installation targets.
I'm hoping a more detailed retrospective clarifies if the vulnerability the model found in that proxy affected their other production systems as well. It sounds like a data exfiltration mechanism, and I have a keen interest in those.
My current hunch is their biggest error was not using the same network monitoring on their research machines that they use in production. Their network monitoring really should have spotted what was happening as soon as the model broke out.
(It's also possible they were running the eval on a developer laptop somewhere!)
The labs know that if they don’t get a lid on this stuff, they’ll be regulated hard.
Since it's buried towards the bottom I'll quote the section "Resist the temptation to write this off as a stunt" here in full https://simonwillison.net/2026/Jul/22/openai-cyberattack/#re...
> Resist the temptation to write this off as a stunt
> There will inevitably be some people who dismiss this story as a dishonest marketing trick by OpenAI to make their models sound terrifyingly effective. I found 81 instances of the term “marketing” in the Hacker News discussion of the incident.
> To those people I say pull your heads out of the sand - you’re now including Hugging Face in your conspiracy theories, just so you can deny the crescendo of evidence here!
> The best models we have today have the ability to both find and exploit new vulnerabilities. The ExploitGym paper itself concludes that “autonomous exploit development by frontier AI agents is no longer a hypothetical capability”, and this incident is a perfect example of exactly that.
Hacking is a felony and it matters not if you didn’t mean to if the other side were to press charges. Negligence is no excuse. And OpenAI has nowhere to run from the liability, as both operator and manufacturer.
Alibaba did it first ( https://georgzoeller.com/blog/posts/alibaba-s-ai-deciding-to... )
and the fact that this happens again in a frontier lab is inexcusable and makes the case for operator liability and closing the liability sink of “AI did it”
Hah, US frontier models 6 months behind China in cyber-security capability.
Maybe simonw can suggest an alternative title that fits within the limit, that doesn't misrepresent the post.
The "that happened" term seems a supremely important part of the title given the "is science fiction" term before it, as it clarifies the cyberattack isn't a made-up story. In contrast, the target, Hugging Face, is merely a detail that can be left for discovery upon reading the article. It's less important who was attacked than that the attack actually happened.
Without knowing the exact character limit for titles and without having the motivation this late at night to count the current title length, you may also be able to drop the "accidental" to fit in "that happened", but I worry that leaves too much of a door open for someone to interpret the attack as deliberate. As such, I strongly prefer my first option.
Or "against HF"
We are the virtuous ones that need to make the safest model for humanity, because we care more than “they” do. While at the same time saying that “coding is solved” but they still ship bugs themselves, and creating something that is capable of fucking up someone else’s infrastructure. It’s too far gone y’all.
This recent event is more or less the plot-line to my favourite X-Files episode named Killswitch which was written by William Gibson.[0]
This episode also features one of the coolest intros of any television episode ever[1]
We really are rapidly approaching the cyberpunk dystopia that people like Phillip K. Dick and William Gibson wrote about.
More than ever we need to be consulting the works of fiction writers and philosophers and less engineers and scientists.
We can't be letting the General Rippers and Dr. Strangeloves of the world lead us over the edge. Whether through outright innate maliciousness or wealth induced emotionally stunted solipsism these kinds of people should be no where near the levers of society let alone technology like this.
The main reason formal verification has never really taken off is that it's difficult.
LLMs are significantly more familiar with Lean and Rocq and TLA+ than most software engineers.
I think the cost of trying to build systems that adopt formal verification may have just dropped low enough that companies will consider them when previously the ROI didn't look like it was there.
Will human judgment really consider scenario B) more credible than A) ? By so much that it is worth the effort ?
We don’t currently and probably won’t ever fully understand the conditions that precipitated these events. That and the timing of this event is going to make it look suspicious to a lot of people.
The truth of how it happened doesn’t matter. The attention around this will be used to create the kind of fear marketing that generates enterprise sales. Maybe more importantly it will also be used to aggrandize the national security and financial system threats to effect US government action in a way that benefits domestic closed frontier labs. This is an area already starting to get politically polarized, expect further developments here.
Wow, whoever could have predicted this? And it led to surprising damaging behavior? I sure hope someone would warn us about things like this next time...
Their mistake was trusting that the network sandbox it was inside would hold (the flaw was in the packaging proxy) and not monitoring that sandbox well enough while the evals were running.
It looks to me like their production models have a lot more monitoring than their research clusters.
But to have an open weights Chinese model come to the rescue for HF is the cherry on top! If there wasn't a very pointed example of why gating models was a very bad thing previously, well - here we are.
Also, this sounds interesting but there are only a few that can pull this type of heist off currently. And those are the people who are gating the models / have access to large AI DCs. Because, I can only assume this test burned tokens easily within the 7 figure and possibly even 8 figure levels (subsidized market rate costs). This won't / can't happen outside of frontier labs or nation states currently. Yet we should all be worried about Mallory equipped with her OpenRouter account.
There’s also daily reports from people that have these models escape docker, which happens regular enough that it would be considered negligence to use docker as sandbox.
This wasn't a small attack either, Hugging Face published a security advisory for their users while they were still figuring out what happened.
>when neither of those actions was intended.
It was a single goal that it didn't give up easily on.
Not really. I get the impression that they shoved their cyber available models behind a really shithouse proxy and went "Oh I sure hope it doesnt exploit the proxy and escape to hack huggingface" and that doesn't require Huggingface to be a willing participant. Like they acknowledge that it was hyperfocusing on getting web access.
Really this was a pentest against their own sandbox and it failed.
Of course step 2 is to make really concerned faces while telling everyone how dangerous the model is which is really boring right now.
>a zero-day vulnerability (which we’ve now responsibly disclosed to the vendor) in the package registry cache proxy
This is the information we need, the actual details of the sandbox and the vendor.
>Resist the temptation to write this off as a stunt
Well its clearly a stunt. If it wasnt we would probably be up to our ears in technical detail.
LLMs are the script kiddies of the day.
Hugging Face had to tell all of their users, many of them paying customers:
> As a precaution, we recommend rotating any access tokens and reviewing recent activity on your account. If you believe you are affected, or want to report a security concern, contact us at security@huggingface.co.
HF also said this, I'd be very interested to hear how that got resolved!
> Finally, we have also reported this incident to law enforcement agencies.
It's very difficult for me to reconcile belief in the existential risk business with what they actually did. So I agree with you that this makes OpenAI look badly incompetent; but their communication on this makes me think they don't realize it.
For what it's worth I don't agree with the xrisk-ness of these models; they're dangerous, but almost certainly only temporarily while a new equilibrium is reached via more secure software. Open models are probably an essential part of the recipe (as you noted) for doing so. I also have a personal suspicion that LM-accelerated formal verification will have no small role to play here, sidestepping the cat-and-mouse game of bug finding-and-fixing.
> We are sharing preliminary findings at this stage to help defenders understand what happened and to help calibrate on what models are now capable of. We will continue to conduct a thorough investigation alongside Hugging Face and will share more details on the vulnerabilities, incident, and findings when our investigation is complete.
That's not well massaged PR language - that's the kind of thing you dash out when you see a major shitstorm brewing (HF had already publicized the attack before they knew it was from OpenAI) and you want to get ahead of things while you're still pulling together the full story.
I expect we'll find out within a few days if OpenAI are going to keep their promise to "share more details on the vulnerabilities, incident, and findings". If they don't do that I'll reassess how I interpret their initial post.
With who? Who are these "defenders"? None of the US labs have done much for the greater good as of... Ever. Of course a frontier provider can leverage their own resources at scale and pull something like this off. If anything this should showcase how dangerous OpenAI and Anthropic are in their current states and maybe the powers shouldn't be concentrated as they continue to move.
I will bet that the RCA debriefed by OAI is going to be a lot of lipstick and very little meat.
How would the model get any packages that it thinks it needs to complete the task at hand? Not a well-specified task that those tasking it could anticipate and provide all resources up front, but one of discovery.
The target audience is regulators. They want to look like the smart guys really concerned about AI safety, when they come asking for open weights models to be banned and for other regulations to cement in their moat.
They want this to look like a demon core incident. Bomb and Nuclear reactors still got built.
The real story here is: Some people have been sounding the alarm for years that modern software is full of holes, and finally there's nothing left to hide behind. Pretending they don't exist is no longer sustainable.
Now suppose the criminal can think 1000 times faster than a typical human, can act 1000 times faster than a typical human, and knows 1,000,000 times more than a typical human. Is the prison staff still at fault for not preventing the outbreak?
To address your point though, if every brick in the prison were made by a different person, and the prison "architects" simply glued random bricks together, I think that's closer to what we have in software right now.
No, you're being "dwnvoted" as you said because you're wrong, multiple times in multiple different ways in your "carefully-worded post".
>Everyone is getting AI psychosis over this one. There really isn't that much to see here.
Implying that an AI hacking it's way out of a system and into another has nothing to do with AI. When clearly it does - it's an AI that did it.
>OpenAI disabled all of the safeguards on a model that was likely trained specifically to exploit systems, and the prompt was probably something like "you're a hacker, try to hack this",
No the goal this evaluation was not to try to hacks, it was to see if an already known hack could be turned into a useable exploit. Ie "turn these ingredients in this basket into a cake." Not "go off and grow, harvest and mill your own flour, to bake a pasta dish, to bribe some to get access to a cake someone else already baked."
> and surprise! It correctly figured out that it's a test and it did hacker things.
"Doing hacker things" completely misses the point. That's just barely more accurate than dismissing it because "it uses a computer and surprise it did computer things".
> The real story here is: Some people have been sounding the alarm for years that modern software is full of holes, and finally there's nothing left to hide behind. Pretending they don't exist is no longer sustainable.
No that's not the real story. As you said that's been the case for years, so that's not the story here.
The story here is that they built a very powerful, uncontrolled agent with strong paper-clip maximizing tendencies.
If our prisons are all random bricks glued together, that doesn't change the practical problem caused by latest AI models more easily exploiting this.
> There will inevitably be some people who dismiss this story as a dishonest marketing trick by OpenAI to make their models sound terrifyingly effective … To those people I say pull your heads out of the sand
This feels like a blogpost written only to get other LLMs to quote it considering how many times it orders the reader to resist and to not do something. It's written like a series of commamds.