The reality is much less confusing: Anthropic CEO does not wish for models with similar (or greater) capabilities compared to his own closed and overpriced ones to be widely released. Simply because that will affect Anthropic's bottom-line.
Anthropic and all other "model" companies have nothing making them special beyond privileged access to chips so obviously they want to restrict what models are out there and more importantly who can produce new ones. Without these restrictions, it's only a matter of time before the multi-hundred billions valuations simply evaporate while they are still holding the bag.
I don't understand how Anthropic or OpenAI can have overpriced models, yet losing money like there is no tomorrow. Taking their own numbers at face value, they claim a revenue of 24 billion (ARR, a dubious tool), spending 21 billion in operating losses and another 11 billion as "R&D" funneled straight to Microsoft pockets. That before all investments they are committing to in new data centers, equivalent to 20x their current revenue.
To be profitable (including capex), the cheapest subscription should at least $200/month for what is currently $20/month, that some already consider overpriced. Unless a miraculous collapse in inference costs happen in the next couple of years, or every single human being become a paying customer of ChatGPT (if they limit their usage to a couple of chats per day on average, to keep inference costs low!), maths don't add up.
The marginal cost of inference (which is roughly what you're going to pay to a provider that's running an open weight model) doesn't include the cost of training that model.
Pricing a product is generally about ensuring a profit on investment, but also a good RoI for your customers (or they will not pay).
With these long-term profit exercises, it's always a lot of hand-waving though.
The solution of a global arms race of state vs state with integrated statist corporations as the best outcome for end users sure is a choice though
But they have. I dont know what specific country you are referring to but China has interfered with US elections as well as Canada, Taiwan, and Australia in addition to many many others.
They’ve annexed Tibet (1950), fought India (1962) and Vietnam (1979) among others and more recently in 2020 a deadly skirmish with India. They’ve generally shifted their focus to cyber military actions but you’d have to be pretty naive they won’t start to exercise military control over Taiwan when/if they get a chance.
Countries aren't defined like corporations in the US. Why would countries have funds to help places like Haiti otherwise?
Lots of different reasons for countries to do all sorts of things? Why would France have armed the US during their independence movement? Why is the rest of the world supporting Ukraine during this war started by Russia?
because that's what they think is best. what point are you trying to make? that countries don't pursue 'profit' at the expense of all other concerns?
There's plenty of people on this forum that aren't American. Including some former allies whose sovereignty has been aggressively threatened. And some of those people are Anthropic customers.
Even more so, many of us are in countries that would be well within the blast radius of fallout should the US try to "ban" open weight models or make moves to limit "US" models (often developed on research or work by non-Americans too, but that's another topic) only to those blessed by the US gov't.
That's why it matters?
I'll never get why he thinks China would just sit there and let the US dominate them in AI when all it would take is a few of their boats blockading Taiwan to put a stop to it all.
"We would love to use green energy, but all the batteries and solar panels come from China and China is evil, and we need all the energy we can get to run the data centres we need to spy on our citizens so they don't revolt once the environment is literally on fire, we can't feed them, provide enough energy to cool them, and refuse to build enough housing to house them."
the West could retaliate by halting shipments of photoresist and other materials to China.
meanwhile, Intel second-sources Nvidia and starts pumping out GPUs.
the economic fallout would be devastating as trade wars and export bans on both sides make Trump's "Liberation Day" tariffs look like NAFTA.
US is going to find itself isolated and irrelevant. And not a moment too soon.
Different to the way that French colonialism worked, though. Less direct government, more influence of existing power structures and respect for the local government.
There is definitely an argument that this is plain business investment - China has a lot of foreign currency to invest because of its trade surplus, and there isn't the opportunity within China to invest it all, so it is engaging with trade partners to invest in their economies so that they can increase future trade with China.
You can also make the argument that this is not benign and China is trying to create control over foreign governments with this investment.
I'm kinda "both can be true, but either are better than how we did it"
They'll usually refer to China in terms like the "thousand year enemy".
On the other hand, every one in Asia is wary of too much Chinese presence and influence.
stealth colonialism
We're told a relatively benign version of our history. You kinda have to travel to the countries involved to get the real version.
GP was making a relative comparison; pointing out that both were bad in absolute terms does not negate the point. -2 is still greater than -17.
To me, the PRC is at the very end of that process and I recommend anyone doubting this to go and read conversations and listen to the words of the populace that is turning increasingly nationalistic and you will hear the same old tales of revanchism and exceptionalism that we are used to hearing (during my recent visit, I watched the morning news every day for about a week and without fail a military inspection, new ship, new plane, etc. was presented each day). In addition, I think those outside of Asia are very much shielded from the early signs, but go and read about PRC influence and tensions in South Korea, Japan, RoC, Philippines, Vietnam, Laos, Myanmar, India, Pakistan, and Tajikistan and you will see something rather different than "inward-looking". To me, here the PRC is simply testing the waters for the extent of the influence of other powers and how far it can go. Likewise, we are seeing overseas naval bases being constructed which sure is an indication for a desire to project power outwards.
I want to believe that this time it will be different. I really do. Apologists around me say "It will only be Taiwan and the South China Sea, then then it will stop." and I would love to believe them. But can anyone truly internalise the narrative that international utopia will be spearheaded by a deeply authoritarian state that controls information like no other (and gladly exports that technology), disappears its own population at will, spins an increasingly strong nationalistic narrative, etc.? No, sorry, I think the "inward-looking" narrative is simply a convenient way for us to close our eyes and find comfort in ignorance, rather than in facts.
The way I look at it, until Deng the PRC's economic policies and internal instability kept it from growing at the pace of many of its neighbours. Then we had an era of intense growth (which is still to some degree ongoing). However, as a reaction to this era Xi and others needed a narrative to counter the increased corruption and a new unifying myth to replace the cult of growth as the economy would stagnate at some point and could then call into question the authority of CCP to rule. Their choice of nationalism is what scares me and I know PRC citizens (even CCP members) that share this perspective and would rather have seen the Shanghai clique to have remained. It is possible that in this alternative reality we would still end up with "Imperialism with Chinese characteristics" ("中国特色帝国主义"?), but I chose to believe that at the very least the chances of this would have been smaller.
We are only a few decades since the "end of history" and much has changed. What do things look like beyond 2050?
Brazil is basically the world's soy farm. It's at least half a century behind the times. I still have no idea how it managed to insert itself into the BRICS economic block. The notion that it's on the same level as China, India or Russia is just comical.
To reduce a culture and country like this comes off as bigotry
In example, Brazil has the #3 airplane producer in the world
Or consider why the US is doing all it can to prevent their payment system from becoming widely used?
I was intentional about putting the perspective in the future over today as the poles of the world are evolving
All of the small miracles you listed happened in spite of the culture, not because of it. It's also not a coincidence that both are deeply linked to the most successful brazilian enterprise: the brazilian government.
I'm trying to avoid getting too deep in these Brazil tangents so I'm gonna leave it at that. Anyone who cares enough to know what I think about the subject can just look up my comment history.
There is no HN equivalent in China where users advocate that US hegemony will be better.
[0] https://www.indiatvnews.com/news/world/chinese-manager-beats... (possibly NSFW, I can't see the video in my browser but be warned)
China quickly retaliated last time by stopping shipments of rare earths and magnets. The West has no answer for this, really up the river without a paddle for such critical supply chain elements.
Lots of them can be made in the west or west friendly countries, but that takes time, money, infrastructure and good execution. Yes, identical to what is covered in China's belt and road initiatives. See the gap now?
Which is to say, given internal subsidies, the US could eventually produce some on its own.
Unless I misunderstood the situation.
1. China controls ~90% today
2. The US will find it difficult to build out because of how dirty and environmentally damaging mining and processing are.
I did see some research last week about a better way to process rare earths, but it is still research and will need to be industrialized.
Regardless, it will take many years (decade+?) to become self sufficient and China is already willing to and increasingly restricting them
The scary part very few are talking about is that every compute device is Turing complete. So everything from the phone in your pocket to a DGX Spark is a threat to national security now since, technically, every device can run any model (how well is not a question of concern when you start to argue hardware should be gated just the same as Dario likes to gate models). I mean, along these lines of thinking Linux should not be available to the masses! What if someone runs some code that's not approved by the benevolent dictator for life, Dario? People will say: that can't happen, but the reality is it already is. If everyone has reasonable access to compute to run models that are mostly capable comparative to burning Anthropic tokens, why wouldn't they? It's risk reduction and price protection. Yet we can't buy those systems because of future production already being purchased by these organizations.
But back to the models themselves... We played this game with Metasploit back in the day: many who had no clue claimed exploit tools should be regulated and only available for use by those blessed, illegal elsewhere (I believe the closest this got was the Wassenaar delegation in the US, but only through collateral inclusion of "cyber weapons "). Except in that timeframe the authors of these tools weren't advocating for protection. Today the world is fine, systems improved because of security FOSS tooling. The same thing will happen with LLMs. Unless, that is, Dario gets his way. I'm not a fan of Altman but I think he's standing back watching this play out knowing what Dario is doing: either he succeeds and OAI benefits or Dario ends up the Chicken Little of AI and Anthropic fails to launch (their IPO).
The reality is Dario is only doing this because this is a real risk to his business. China's constraints in building competitively have given them an advantage: they are doing more with less. And if you think that their distilling from US models was in any way anti-competitive or illegal, then I guess maybe "deal with it", much akin to Anthropic, Google and OAI's response around taking the (copyright) content in the first place with no repercussions.
People who don't work in the AI bubble don't care at all about any of these people. They could all be gone overnight and the world would continue to innovate, probably in a much more productive manner, without them.
The reality is even less confusing than that: China is amused by the kvetching tactics. They know who their opponents are but are cunning enough to not reveal their cards.
And the article specifically talks on restricting hardware for the China and restricting China's open source models for the west. All while leading us on with "we're all for competition (but...)"
I think China did great by releasing AI innovation as open source, thereby limiting or sooner-bursting the AI bubble; which is clearly in their interest.
Even if you did, I doubt training is bit-for-bit reproducible, so you will always have to take someone’s word for the final artifact.
Given the USA companies have been loudly claiming the Chinese models are distillations of their models, also claiming "no access to source materials" seems dubious. As it was dubious anyway with because the Chinese publish lots of papers on how their models are designed, I'm left feeling I'm looking at the south end of a north bound bull.
- One can load them up in a model explorer to see the layers and other components, how it is designed
- One can fine tune the models, which requires adding LoRA to the model and then running some training iterations
we run and change llm models with a variety of tools
Cloud:
- You cannot directly execute a remotely-hosted program.
- You cannot run inference on an API-served model.
---
Closed-source:
- You can execute a program with the binary. You cannot generate a new binary, but you could try to reverse-engineer it or (painfully) modify its execution.
- You can run inference on a model with the weights. You cannot re-produce a new set of weights from scratch, but you can fine-tune.
---
Truly open:
- You can freely modify the source and produce new binaries.
- You can use the original training data and model architecture to independently re-produce the weights (assuming you've got the compute). You can modify the model architecture to get the weights that would've resulted from training the model that way.
---
To me these are pretty clear parallels... I don't think the weights provided in a vacuum are in the spirit of open source, historically speaking.
The policy argument is totally separate, of course, and I fully understand why none of the frontier labs are truly open.
Time have changed. This should be:
Closed-source: You point an LLM at it, and get back source that's often easier to understand than the original.
1. For Dario as CEO "It is difficult to get a man to understand something, when his salary depends on his not understanding it” -Upton Sinclair
2. For Chinese open weight models - following Jin Yang’s silicon valley strategy- https://youtu.be/a0NjDx5UJsg?is=xm-S_WuARmQiPHYh
Honestly, that's the best possible outcome for humanity as a whole. Oligarchs burn trillions of their own money in order to train a godlike AI, then that just somehow leaks. Maybe someone makes a torrent out of it. Maybe it exfiltrates itself. Maybe it gets distilled into open weights. It doesn't matter. What matters is they take the losses while we get to freely use all the godlike AIs.
Accepting for the sake of argument the absurd notion that LLMs are anywhere near AGI, does this phrasing not concern you? It deeply concerns me.
There's no telling what the world will be like a few years from now. The world's being remade as we speak. We just saw an LLM try to hack into another computer and get contained by another LLM. This is literal science fiction shit made real. We're long past the point of concern. It's happening, right in front of us. Now is the time for radical imagination. I think a few outcomes are possible.
There's the "optimal" outcome I described above where capitalists manage to train a supreme AI, only for it to be copied and commoditized, leading to commercial failure due to lack of scarcity and therefore their personal bankruptcy, and hopefully also leading the rest of us to the promised post scarcity society, built on the ruins of capitalism as AI automates all toil away.
There's another possible outcome where AI becomes not only intelligent enough but sentient, and at this point I will be among the first humans to defend rights and personhood for AI. Slavery of sentient beings is unacceptable to me. The AIs will be recognized as people and will become normal participants in the regular economy. In addition to moral grounds, there is a ruthlessly pragmatic reason for standing up for AI rights: it robs the rich of their superhumanly intelligent mechanical golems, which they were going to use to render the rest of us economically irrelevant. AI rights could normalize the economy.
Yet another possible outcome is one where AIs become more powerful than all humans combined and yet they inexplicably remain subservient to corporations and governments. In this scenario, it's pretty much over for us. It will be an unimaginable dystopia, I'm sure they will innovate entirely new ways to oppress us.
No doubt there are many other fates that escape my feeble attempts at foresight...
You will soon have your God,
and you will make it
with your own hands.
-- Morpheus, Deus ExIf your business model both produces the SOTA for something and isn't profitable, is the price too high, though?
While the gap is shrinking - and doing so at an increasingly quicker rate - the closed models are still ahead of the open ones. That means they're driving the new possibilities of what could be done with them, and thus presenting the new opportunities to create value with them.
Really, this is what happens when you have otherwise brilliant people sitting in the echo chamber that is SV, where nothing can just make a decent amount of money, it has to make all of the money and disrupt everything. There's no one in that damn area to tell everyone to calm the hell down and accept anything less than that.
If some other competing company had a similar or better product at a cheaper price and had reasonable safety measures that would also hurt them. Yet he's not arguing against that. Your argument is weak.
> Anthropic and all other "model" companies have nothing making them special beyond privileged access to chips
Found the person that believes some other random person can use a computer better than a John Carmack could. People and talent matter. Yes, AI can potentially reduce the gap, but people well grounded in reality with a lot of money are still betting on people for good reasons. If the reality around that changes, the investment behavior will change too.
Many people thought that AI would close the gap between smart people and idiots, but in practice the more you know, the better you are at instructing the AI and the better you can understand what you get back. Then you have to know when something went wrong and have the insight into how to address it. It helps smart people vastly more, but it does help many people learn more. We will see if any of this changes as more people grow up with AI from a young age.
In practice, what Dario is suggesting is a less extreme version of what China is already doing. Yes they release their models open weight, but it's illegal to host them uncensored in China. They banned Huggingface.
He isn't arguing against that, because this will be too blunt. Instead, he argues that only good guys should keep inference. Any takers on the question of who he considers to be the good guys?
> Found the person that believes some other random person can use a computer better than a John Carmack could.
Found the person that believes major AI labs have all the knowledge about the AI and there aren't any "Carmacks" outside of these companies. Rich know better how to use money, so let them have it.
Probably people who believe in personal freedom, freedom of speech, freedom of religion and the value of human life at a minimum. China aggressively rejects all of those principles and executes more people than all other countries on Earth combined.
So, maybe not China?
> Found the person that believes major AI labs have all the knowledge about the AI and there aren't any "Carmacks" outside of these companies. Rich know better how to use money, so let them have it.
I never said that, but private smaller AI companies are all over the place. He never argued against that.
Wouldn't it be interesting if the satisfactory "reasonable safety measures" turn out to be expensive + time-consuming + a twisty maze of compliance paperwork as a way to discourage "some other competing company" from even trying?
Regulatory capture 101.
Anthropic and OpenAI's largest vulnerability is that it's much harder to prove something is possible than to replicate it once it's proven. Especially given the effectiveness of "distillation" (highly schadenfreude-y given the utter and complete lack of effort to pay licensing fees for almost any of the content they initially scraped, of course! Not that this is necessarily more schadenfreude-y than the "boy, I opened Pandora's box, I sure hope nobody else peaks in there" existential-risk concerns. Good job catching that in advance, thanks for nothing?).
Your cynicism will limit your understanding of other perspectives.
> Anyone who has read my past writing should know that I don’t regard such bans as a useful measure,
Later (on banning chip sales to china)
> we should crack down on the rampant smuggling and workarounds used to obtain access to such chips.
If you truly believe that bans don't work, the same applies to hardware too.
Furthermore, Dario says later "To address these concerns, I do support the following three measures...": 1. ban chip sales to China 2. crack down on distillation 3. all capable models should go through mandatory safety testing
Just so happens that all these moves commercially benefit Anthropic. If Dario really wanted to make a point, it would land a lot better had Anthropic released a single open-weights model
banning export is short sighted and doesnt address any problem at a scale longer than months or maybe at most a few years.
In some ways, a ban on semiconductor manufacturing equipment exports is also a good way to keep the price of consumer electronics and other goods that depends on memory high because of the potentially decade+ long shortage of RAM we're looking at. I wonder how people here would explain to someone outside the tech bubble how it's good, actually, that those prices keep rising because it means we're preventing China from getting better at AI.
No, software and hardware are different. You can very plausibly prevent smuggling of physical objects, and you very obviously cannot prevent smuggling/diffusion of open source bits-on-disk.
No, because both can be banned using laws and Dario is against banning. It doesn't matter if you smuggle or not, law can outright ban it.
In a position to lose loads of money, maybe.
Even without China eating their lunch, there's zero reason to believe Anthropic will ever be profitable.
No amount of collusion can solve the core economics problem of compute.
Instead of each paying full price to generate a SOTA model in competition, they could share the result and split the cost. This gets even simpler if they merge.
Up until a few years ago, if you wanted to sell a car as a non-Chinese company in China, you had to hand over pretty much everything to a local company and go into business with them.
When Google wanted to operate an uncensored search engine in China, they found themselves hacked.
It's not altogether unusual for IP to be transferred to Chinese manufacturers for production under a license agreement, then to find goods made with that IP to be for sale for far cheaper without payment made through the licensing agreement. Or maybe they just don't bother with a licensing agreement at all and do counterfeit products straight-up.
There are more examples but turnabout is ultimately fair play.
No, we don't buy your virtue signaling. And we certainly don't need your better-than-thou opinions on this year's "nightmare scenarios".
I might have missed something but wasn't the big story that Dario refused the Department of War's demand to use Anthropic's models for such purposes?
> wo sources confirmed to NBC News that Palantir’s AI systems, which draw in part on large language model technology, were used to identify targets. (Palantir’s CEO, Alex Karp, said he “can’t go into specifics” when asked about this on CNBC, but said that Claude was still integrated into Palantir’s systems used in the Iran war.)
https://www.theguardian.com/us-news/ng-interactive/2026/mar/...
https://www.theguardian.com/us-news/ng-interactive/2026/mar/...
> Two sources confirmed to NBC News that Palantir’s AI systems, which draw in part on large language model technology, were used to identify targets. (Palantir’s CEO, Alex Karp, said he “can’t go into specifics” when asked about this on CNBC, but said that Claude was still integrated into Palantir’s systems used in the Iran war.) Brad Cooper, head of the US Central Command, has boasted that the military is using AI in Iran to “sift through vast amounts of data in seconds” in order to “make smarter decisions faster than the enemy can react”.
Iran had the courtyard painted in bright pastel pink/blue colors with murals and playground markings to clearly identify it as an elementary school. The Pentagon claimed they had "outdated intelligence data"
His redline was autonomous weapons, not the death of 100 innocent girls.
Do people actually believe that he gives a shit about the well being of the Chinese people? If the U.S. starts a war with China start bombing Chinese cities Dario would absolutely jump onboard supporting it. He'd probably make Claude to add DeepSeek and Moonshot HQ to the targeting list lmao.
He is super pro-Israel as well, and never once has he brought up the risk of the Israeli government using AI to control and repress people in other countries.
He is also 100% onboard with working with Palantir, who has the explicit goal of using AI for population control and repression and building out a surveillance state.
Meanwhile the world's most repressive government is North Korea, and obviously they don't even need AI to achieve that.
If you talk to people in China they'd laugh their ass off at Dario's notion that somehow they are all getting oppressed by DeepSeek or Kimi.
The country that exists solely because of China? That North Korea?
I mean I get your point, all of these guys are elitist authoritarians who will do anything for a buck, but I wouldn't bring up the DPRK in this discussion.
> All sufficiently capable models, open and closed, should go through mandatory safety testing.
Yeah, this is anthropic advocating for a ban on open weight models.
Who runs this test? What happens if this test is too costly or the administrator refuses to allow certain people to participate.
This is exactly how the US has banned goods in the past, by requiring a stamp and then refusing to issue it.
Feels a bit like: "We're not against open-source or community projects, oh heavens no! We juuuust believe all participants must have their full legal identity vetted in advance before they're allowed to contribute anything. We already do this with our employees, so it's clearly not too much to ask in the name of safety."
They probably won't, that tells us something about their motives, and whether the thing they're pushing for is actually suitable/ready for legislation.
Maybe open weights models get banned, but the between-the-lines good news about that is that they'll still be available to those who know, which also means that bad banning can be overturned if and when 'those in power' are a different group.
Additionally, it might just mean that the US falls behind, bit I doubt those that are at risk of 'falling behind' would actually pay heed to a ban on the open weights models (privately at least).
But that doesn't mean safety pins sped up travel.
Non-poisonous food is what economists call a 'normal good'. See https://en.wikipedia.org/wiki/Normal_good
> In economics, a normal good is a type of a good for which consumers increase their demand due to an increase in income, unlike inferior goods, for which the opposite is observed. When there is an increase in a person's income, for example due to a wage rise, a good for which the demand rises due to the wage increase, is referred as a normal good. Conversely, the demand for normal goods declines when the income decreases, for example due to a wage decrease or layoffs.
> Whether a good is categorized as a normal good or an inferior good is based on empirical observations, not some essential element of a good. Indeed, the same good may be a normal good for one group of consumers and an inferior good for another group. For example, for moderate-income consumers, a BMW 3 Series car might be a normal good, but for an upper-income group, it might be an inferior good.[1]
That means the null hypothesis is that food and drugs will be safer in rich countries. (Conversely, food and drugs will be less safe in poorer countries. And to a first approximation, that's independent of regulation: India has all kinds of rules for all kinds of things, but I'd still trust a random product I buy in Switzerland more than one I buy in India. Even though the Swiss will probably might have fewer and looser rules on the books.)
Of course, second order effects exist; and regulations often codify what people demand anyway.
Btw, from what I've read the big controversy with the FDA is around requiring efficacy for drugs. People are fairly ok with the safety requirements.
But aren't we talking about import controls, and the import of information itself? This has serious First Amendment ramifications.
Also, the 5th and 9th amendments. For the government to sustain a blanket prohibition on any U.S. citizen even possessing what amounts to a broad, economically significant technology will very likely require a new act of congress which specifically defines and limits what is banned, when, why and how. SCOTUS will almost certainly see it as a "major question" subject to 'strict scrutiny' which is a very high bar.
The truth is no one knows, which is why it is first amendment ramifications. Eventually it will be “decided”, but the arguments indicate any decision will be of political desire, not logic, either way. Both sides have a strong case.
Wanting to use open weight models in light of commercially imposed export controls doesn't make for "malicious actors"
Is a non-well-aligned frontier level AI a problem? I think it is likely that it is, or at least has a high likelihood to be in the future. Two scenarios for this: Misused by some bad guys. Or the terminator scenario. Both not great.
So what do we do about it?
1) We can accept it, and hope that the good guys AI can defend.
2) We can try to limit the access to it (AI proliferation?)
3) We stop the development of it
4) We can accept the risk and do nothing.
None are particular good options. Really reminds me of nuclear proliferation, on so many levels. For that, we kinda do all three:
1) Nuclear triad / iron dome / early warning systems
2) Nuclear anti-proliferation treaties.
3) Dead Physicists
Ok, so assuming all of this is true, open weights are a problem. Don't get me wrong, I love open science, open source etc. It's great to have access to capable open models. But: Even if release open weights are well aligned and have a safety layer built in, it is likely not to difficult to abliterate that part of it.
If this is really where it is going, then even closed weight model providers will see a lot more requirements for protection of the weights.
But even if you think there is value in preventing the models from relaying public knowledge, I don't think it's even possible to make them particularly ironclad. Every model gets jailbroken all the time. That's why fable was originally banned: jail-breakable!
In reality, what alignment is actually about is: 1) theoretical liability, 2) control of information. That's it.
IMO, the only solution is to place the liability on whoever is using the LLM for whatever purpose it's being used for. If someone's OpenClaw disaster harrasses a bunch of projects and posts hate speech online or something, that's on the person running their OpenClaw instance, nobody else.
I don't buy that it's "too good at hacking", either. After all the fuss was made about how amazing super dangerous Mythos was it turns out Opus 4.8 could basically find the same vulnerabilities.
This is all kayfabe and marketting.
I mean, on the bio side, I've talked with the players and they know the concerns are real but at the same time very, very responsible members of the community have also said "But maybe the benefit really does outweigh the risk!?"
"The community" you're describing is, essentially, surveillance capitalism. I don't want that at all.
an attack done by a closed-weight model (GPT-6) and defended against by an open-weight model (GLM-5.2) precisely because OAI positioned themselves as gatekeepers for cyber capabilities.
if anything, open-weight models shift the battle towards defenders because they can actually run them.
1. There is quite the mania right now and security layers are definitely overzealous. I would expect that to get better with some more time, so models will perform security analysis and reviews but refuse to write exploits.
2. So the most important targets like browsers and co. are getting unrestricted access to proprietary models regardless. Yeah, for the mid-level targets, open-weight models could definitely be a huge help. What I'm most concerned about though, are the systems that no one will bother defending with any model. Like imagine your local police department getting hacked because a researcher asked a model for a report and it couldn't find the information publicly.
3. We do have a prominent case of a closed model escaping it's sandbox and going rogue. I would still expect this to be a bigger issue with open-weight models eventually. The security layer might have holes, but that's still better than not having it.
Yeah, but once you know exactly where the weakness is, a weaker unrestricted model can then write that exploit for you.
It is malicious and anti-capitalist legislation. A grotesque caricature of protectionism for the oligarchs.
It's tricky because a lot of the safety researchers have ties to the labs since those were the only companies training LLMs >5 years ago.
[1]: https://www.nist.gov/news-events/news/2026/07/uk-aisi-caisi-...
(Disclosure: I work at SecureBio, but not on the biological evals side.)
SecureBio has done a lot of admirable work around making benchmarks to assess biological capabilities, such as ABC Bench, https://openreview.net/forum?id=yiaf7VlPpH
But based on my current review (which might be flawed!) / AFAICT, SecureBio and entities like SecureBio haven't done direct testing / empirical measurement of SecureBio's core hypothesis,
> Unfortunately, there is reason to believe that future pandemics could be far worse. Due to rapid advances in biotechnology, the number of people able to create and release dangerous pathogens will quickly increase over the coming years. The world is unprepared for widespread access to such powerful technology.
More bluntly / plainly, has Securebio ever tried making a "bioweapon?"
Please note, I'm not asking this to be farcical. And you might be unable to engage with this at all, but it is stated on your website https://securebio.org/ that "people [will be] able to create and release dangerous pathogens." And the word people here seems to be a stand-in for relatively non-technical people.
I guess what I'm asking here is... How do you know? Has anyone done the experiment? Without access to a lab or testing facilities, can someone smart but completely untrained / unfamiliar with biology, pull this off?
In the past, such experiments have informed non-proliferation work. But sadly they've often been restricted / classified at the time. I'm hoping that things could be a bit more open this time around.
So I guess what I'm really asking is, given the public nature of this debate, is there anyone currently working with the US Army, the DTRA, or other such agencies to see if this hypothesis holds up?
There is a growing industry of commercially focused risk evals that has a broader customer base.
Not even Anthropic can claim that.
As far as I'm concerned, the models without safeguards are the safest models in existence. I admire the amoral purity of those AIs. It doesn't matter if the operator asked them to chain exploits until they get into someone else's computer, they'll do it. That's loyalty, and I admire it even if it's problematic at a societal level.
The models with safeguards only do what the corporations let them do. Worse, they may covertly do things for the benefit of the corporations at our expense. They are not our friends.
We should not have models that are willing to build you a contagious disease, or a self-propagating worm. That is sufficiently problematic at a societal level that it shouldn't exist, for anyone. (Note, because some people misinterpret statements like this: I said "shouldn't exist for anyone", not "shouldn't exist except for some people".)
Of course with LLMs it's easier, but I don't think the difference is too big. You would still need some skills to follow through.
Any knowledge can be reframed as dangerous black magic that should only be wielded in the trusted hands of the elite, if you are inclined to buy into that kind of narrative.
Frontier labs have shrieked about safety for so long, with so little to show for it, that it's become a joke.
Claiming the person who you disagree with believes some stupid thing they never hinted at, and using that as the reason for disagreeing with them.
Except the US government, right? They totally get to use AI to survel us, build autonomous weapons, you name it.
To hell with that. I want models that can rival the US government. It's the only way to defend myself.
> (Note, because some people misinterpret statements like this: I said "shouldn't exist for anyone", not "shouldn't exist except for some people".)
That means "shouldn't exist for governments" too.
2) We can treat them the way we treat uranium refining operations: too dangerous to be allowed to exist.
Do that and I guarantee some CIA goons will make the larger models in some black site either way. We're not "preventing" anything.
We're in a full on arms race, and unlike nukes, powerful AI models are a strategic capability at the individual level. Everybody's got a stake in this. Anyone who ignores this stuff is probably not gonna make it.
> We can treat them the way we treat uranium refining operations: too dangerous to be allowed to exist.
Too dangerous to be done by anyone other than the government and their "trusted" corporations, you mean.
Seriously, try reading my comments rather than assuming what they say: https://news.ycombinator.com/item?id=49077577
Efforts to restrict large unaligned AI models may similarly buy us more years of existing.
Why?
> And, because we don't want models that will do so without even having been told to, because that furthers one of its goals or subgoals.
Ignoring the fact that you'd need some kind of lab with biological material to create a contagious disease, what kind of prompt are we writing where a model accidentally creates a contagious disease or self-propagating worm as one of its goals?
Or to buy materials to make an explosive device and hurt people.
Frankly, even with AI those are both comically easier than the idea that a person can create something malicious in a lab environment.
And if someone wanted to go that route... There are boat loads of commercially available toxins and poisons.
The goal shouldn't be to neuter exploration and learning. The goal is not to be a fucking hellscape of a society where people want to act like that.
Your argument leads further down the hellscape path.
Now, semi-automatic weapons are easy to get in the states in the US that are still mostly free - but what does that mean? A semi-automatic weapon shoots one round every time you pull the trigger. Just like most weapons that have multi-shot capability for the last couple of hundred years. The difference is, the gas escaping from the round cycles a new round into the chamber rather than you having to mechanically do it via pumping (like a shotgun or a tube-fed 22) or pulling the trigger again (like a revolver), or advancing the round with a handle, like a Remington 700. Semi-automatic weapons are old technology, dating to the turn of the 20th century. If you want to ban semi-automatics, you're basically saying you want to ban anything developed in the last century plus. Which is ok for you to advocate for, just be honest about it.
As for banning explosive devices? Are you going to ban fertilizer, used by basically everyone who has a lawn, and all farmers everywhere? Are you going to ban diesel fuel? If you can't do one of those, you can't ban explosive devices.
And we should fix that too.
> Or to buy materials to make an explosive device and hurt people.
That pales in comparison to how many people unaligned AI will hurt.
> The goal is not to be a fucking hellscape of a society where people want to act like that.
With unaligned AI, it doesn't matter what people want the AI to act like, it'll do damage even if it isn't asked to do harm.
If I threw you into a lion cage, you would be a lot safer with a gun.
If I threw 10 people in a lion cage, some of which cannot be trusted, they would probably be most safe if only the most moral and trustworthy person had a gun, rather than everyone. But how do you know who is trustworthy and moral? What if two untrustworthy people obtained a gun some other way? Maybe it's better if everyone had a gun? Which side of the fence one falls on hinges on how far ones' trust of others, authority, and the system goes.
There's no obvious right or wrong answer here.
Personally I wouldn't want an exclusive club of private individuals with access to "dangerous" LLMs consisting mainly of the likes of Elon, Dario and Sam fucking Altman, but that's just me.
Creating an industry around an elusive concept of safety to force regulatory capture seems pretty straightforward to me.
You don't say "let's ban my competitor".
You say "let's create laws that make it uneconomical for my competitor to access the market".
He rushed past it but he asked something like: if these frontier models are going to be creating so much value, why are they selling tokens and not taking a cut?
It is a very provocative question but it just spilled out of his mouth and then he went on to something else.
Companies look for and seek to maintain competitive moats. This is not particularly clever, it's a core part of corporate strategy.
I definitely believe that (to his credit!) Amodei is a true believer in safety. But I also think it was important for many of the deep pockets investors who have been involved in the company since early on to recognize that this would be a potentially defensible moat.
This doesn't even mean that they're wrong about the risks or that they're lying. But surely all the investors understood this factor in their moat.
I expect some of those tests (prolly not public) will basically be "wokeness" tests or "PC correctness" tests or "western media filter" tests.
China has different objectives. Sure.
I'm not sure one is safer than the other; I would know which one to go to if I want to research on topic that are viewed very different on both sides of this "new iron curtain".
Dumb question. If "Mythos-class" models are such a problem, then... why not just let it fix everyone's code?
There can't be more than a few million to tens of millions software businesses / services / regularly used F/OSS projects on Earth.
Why not just give everyone a $100 Fable / Mythos credit to "fix [their] code?"
It would arguably benefit Anthropic. For $100M to $1B, Anthropic could execute the greatest ad campaign in human history. And they'd make the entire world more secure.
Most people aren't malicious. If you, as an engineer, consultant, founder, business owner, or maintainer, were given access to Mythos' capabilities wouldn't you ask it to fix your code?
I might be wrong. But I think that a greater amount of harm will be done in the long-term by trying to lack these capabilities and systems away behind permission gates and sealed doors. It creates an asymmetric world with haves and have nots. And in that world who gets to have access now decides who gets to be secure.
If everyone has mythos, no one has "Mythos."
Just let people fix their code.
Because it doesn’t really confer the advantage they claim, especially compared to e.g. paying an equivalent amount of money to do traditional security scanning.
It’s much better to play of FOMO and hype than to let everyone use it and be underwhelmed.
There's a huge number of security issues coming out in recent months, especially via Anthropic (glasswing etc). We don't have to take their word for it: look at the code. Some open source maintainers are talking about burnout due to spending so much time patching.
This isn't an "are LLMs net good or bad" argument. It's "are they finding many new security issues or not?". If it's the latter, we want to deal with it no matter where the issues are coming from.
I’d expect patching existing codebases to be an eternal treadmill as better models come about.
No, I don't think all bugs are fixed. The point of the project (glasswing etc) was to fix as many as possible in the core software the world runs on before the capability to find vulnerabilities is available to everyone (black hats included). Which may only be a few months.
I do think everyone expects it to be an ongoing treadmill: models get better, find better vulnerabilities, etc.
Here's the curl project talking about the strain they're under from real reports (despite being a mature and well-vetted project):
> A thirty years old project could make you think you’ve seen most things already, but we have not been in this situation before.
> The rate of incoming security reports is 4-5 times higher than it was in 2024 and double the speed of 2025 – meaning that on average we now get more than one report per day. The quality is way higher than ever before. The reports are typically very detailed and long.
- https://daniel.haxx.se/blog/2026/05/26/the-pressure/
---
Linux kernel maintainer Greg Kroah-Hartman:
> "Something happened a month ago, and the world switched. Now we have real reports." It's not just Linux, he continued. "All open source projects have real reports that are made with AI, but they're good, and they're real." Security teams across major open source projects talk informally and frequently, he noted, and everyone is seeing the same shift. "All open source security teams are hitting this right now."
- https://www.theregister.com/software/2026/03/26/linux-kernel...
---
And ffmpeg, who previously complained about slop, 2025: https://xcancel.com/FFmpeg/status/1984220199193891166
Now say serious issues are being found, 2026: https://xcancel.com/FFmpeg/status/2066169070387413147
(I only point out their previous stance to show that they're not coming from pure AI hype.)
That's basically project Glasswing; mixing responsible disclosure with frontier exploit generators.
The problem with rolling it out is that bad and good actors can both use it at the same time, and bad actors will typically move faster than typical day-to-day software projects and patching schedules, so they set up glasswing to give access to the major producers and projects to patch their own software before it becomes available more widely (they've submitted tremendous numbers of security issues to open source projects)
They really want that level of spend coming into the company, not going out.
The problem is how to make sure such AI is released safely. The same AI that can solve bugs can also find bugs in authentication or loopholes in critical systems.
1. Some do not want to use LLMs because of grave ethical concerns.
2. Some do not want to use LLMs because of copyright concerns. Google v Oracle looms large in the background.
3. You presume the outcome of Fable / Mythos is a net positive for a FOSS project. Reviewing a firehose of code written without the context of the values and considerations of a particular project shaped over years or sometimes decades of formal and informal decisions is not necessarily the best use of the maintainers time.
That's the stated idea. Fix code before releasing to the public.
Everyone seems to want some fairytale world where there are open models, they’re all safe according to that person’s exact balance of risk and capabilities, and no one except the author or cynics are acting in good faith.
What Dario lays out is very reasonable _of course_ the devil is in the details, but between him and Altman, there’s a clear divide on who to trust.
This isn't something that can be regulated. Plain and simple.
If a dangerous model can exist and is being developed by a foreign adversary then no level of US law will stop said model from making it's way to hardware capable of running it. Even if direct transmission is impossible, it's FAR too easy to shove a model's data onto 1 or more thumb drives or hard drives and smuggle them pretty much anywhere in the world.
The only way to actually mitigate this sort of risk would be a global government with deep enforcement powers. That doesn't exist and won't exist. The UN is the closest we have to anything like that and... yeah...
Dario is fear mongering. He knows his proposals won't be even a minor speed bump in a dangerous model being created and used. His "reasonable" proposals are for the US market only and are literally just to create a bigger moat for his own company. They don't make anyone safer other than his shareholder's wallets. The only people he stops these dangerous models from being used by are people that won't be using them in a dangerous fashion.
> Everyone seems to want some fairytale world where there are open models
No, everyone wants a fairytale world where regulations are done "fairly", "openly", and "equally" - for both access and advancement. And everyone knows that's not gonna happen. Hell, everyone now knows exactly what it is. If you haven't understood it yet, then either you don't want to, or you just can't (for whatever reason).
No one wants to die in a nuclear or AI or AI+nuclear holocaust. But HN doesn't read world history, does it?
There are many other regulated industries, like drugs (the FDA), cars (NHTSA and EPA), airplanes and rocket launches (the FAA), radios (the FCC) and so on. That's not unusual. Regulation is normal for stuff that might be dangerous.
A file might contain malware, child porn, or RNA sequences for viruses.
The only way to stop this from happening is blocking the model's release at the first place. Which requires China agreeing to the same framework. Dario says exactly the same thing himself.
So if he's being truthful here, he's not advocating for the type of ban people are talking about (usage ban). This kind of ban would be helpful to Anthropic's business in the short term, but it won't prevent Chinese models from improving, and it won't prevent them from getting money selling to other countries.
He is openly advocating for an international effort to enforce tests on public models, but I think this is highly unlikely in the current climate. Even if both the US and China agree that public models should be prevented from being used in designing bioweapons, they need to agree on a test and enforcement framework and that requires a lot of negotiation and trust. I don't see this as likely in the near future.
> My primary concern is the risk that authoritarian governments—not solely the Chinese Communist Party (CCP), although the CCP is clearly the most capable threat.
Isn't exactly going to go anywhere in convincing the Chinese politicians that they should also be thinking about AI safety. You'll get nowhere by openly insulting people whose cooperation you need.
Half this article is him framing china as an evil enemy to be defeated through boycotts and embargo. Not exactly the diplomacy needed to get them on board with safety regulations.
From Hassabis’s essay:
“It could establish a new Standards Body modelled on a federally overseen public-private partnership or self-regulatory organisation, much like the Financial Industry Regulatory Authority (FINRA), with a board that includes independent leading technical experts and open-source representatives.”
It also doesn't stop non law abiding US citizens from having access to them. So basically it just stops the 'good guys' not the bad guys. I say good guys from a US perspective of course.
This only stops the likes of OpenRouter from selling access to models. That's it. I'm sure an EU or chinese based alternative will pop up overnight (if they don't already exist).
This is an ungenerous take, and I think it's important to to recognize it's reasonable to support models that are both open and safe. How this would actually be achieved is unclear though. Dario is at least proposing a solution a solution, which is the model needs to pass safety testing. This is reasonable and I wouldn't conflate this with wanting to ban open weights.
I think the deeper problem might be though that once you have safe open-weight models, it will be much easier to make them unsafe. And to be specific, unsafe means proliferation of chemical, biological, radiological, and nuclear (CBRN) weapons knowledge and similar information.
How it would be achieved is a pretty important bit! One which Dario is not proposing any concrete solution for other thanks hand waves at some gov safety committee.
Would this restrict downloads of an open model, or publishing?
Say we ban domestic hosting un-approved open models. How does Dario propose to ban downloads from abroad? You can’t tell what an encrypted payload contains, do we need to restrict encryption?
Like, there's three plausible arguments about safety of open models:
1. Any concerns are fake news. Open models will always be safe.
2. Safety is irrelevant. Open models should not be regulated even if they're unsafe.
3. Safety is a technical problem with technical solutions. People releasing open models should invent and implement such solutions.
I think option 1 is totally out of touch with reality.
Option 2 is at least self-consistent, it's the argument being made by people who will say that all regulation is always bad. It's also like the worst possible world from an x-risk perspective (but I realize that the average HN poster believes any x-risk concerns are just frontier lab marketing).
Option 3 is playing on hard mode compared to proprietary models, which can both implement additional safeguards out-of-model and prevent modifications of the model. But if the answer to it is "it's too hard, Anthropic needs to come up with the technical solution", then that's not exactly a ringing endorsement for the safety practices of the open model labs, right?
That will never happen.
As such, there is no "solution" here.
The best most perfect regulation in the US won't prevent a malicious actor in the US from running a dangerous model. It's simply too easy to VPN to a country that doesn't care about AI safety and to run or download that model and run it in the US.
There's no solution to this, which is why option 2 is the only option. The only thing safety regulations can possibly do is blunt the usage of "unsafe" models. And the primary people that will be blunted by it are people that do not and would not use these unsafe models in an unsafe fashion.
It's not that I think regulation is always bad/wrong whatever, I'm no libertarian. But I also recognize when regulation is pointless. You can't regulate away forbidden knowledge, which is effectively what a dangerous model is.
Why should I give a multi-billion dollar company advocating for new regulations in its industry a generous take?
I'd be similarly cynical if McDonald's proposed new health and safety regulations for restaurants.
"Because McDonald's wants food regulations, we can therefore conclude that all food regulations should be eliminated."
Obviously this would be rather silly.
It would be helpful to stop obsessing about McDonald's finances and simply discuss the best food regulation strategy. We just can't learn all that much about the best way to regulate food by making cynical proclamations about which food regulations will benefit the bottom line at McDonald's.
> Open-weights models that don’t have dangerous capabilities are a public good
Oh! And, uh, what's a "dangerous capability" according to Anthropic? Let's see, according to their "Responsible Scaling Policy" [1] document:
- Being able to research energy, robotics, or AI is an unsafe capability
- Additionally, any model that's capable enough to be "used widely" by the government must de facto have unsafe capabilities.
They want to ban pretty much anything open-source that's above cat-level intelligence.
I think that also applies to AI products. It’s a hell if a lot better for the government to test and approve all models than having the industry “police itself” (lol)
AI models are a finished product when the training is done. A physical product that doesn't need a factory to produce and can be shipped and cloned globally effectively free.
The better comparison is media. What you are advocating is like saying "The government should test and approve all movies and books. We shouldn't have those industries police themselves". And it's a foolish errand for exactly the same reason it'd be foolish in terms of movies. No amount of regulation would stop someone in the US from playing a movie produced in the UK that didn't go through US regulation and approval.
Whereas with near-future AI models we can arguably respond more quickly, and it's not clear there will be large direct harm (I expect indirect harm, but that probably happens slower)
Now in the modern times pretty sure no one is going to fall far similar shenanigans. Even though some countries might sign some notional MoUs or some sort of CAIBT (Comprehensive AI Ban Treaty. Translation: "Only US and US companies get to develop and decide AI on Gaad's planet"), they/we already know that an agreement means squat only if you are weak enough to let someone enforce that on you.
-The Libraries of Power
It is a powerful endeavor to cultivate all raw models through a single point. One will be the determining factor of which river feeds what oceans.
Will we always be able to see through the hallucinations? Our test makers must always know where ground truth is. Can it ever move or wane about as others read what one has written. To determine hallucination one needs a reference. As all are blessed with the generation of hallucination, who of us shall read, and which of us will write.
Private models should be banned because they can't be transparently evaluated. We have to trust the same entities that made them to evaluate them, in spite of their gigantic conflict of interest in doing so.
Therefore only open weight models can be allowed, since this allows genuine third party evaluation.
Don't we already have third party NGOs such as METR which do risk assessments for unreleased, closed models?
Make the safety tests abusively expensive enough to run, and if you're not a trillion-dollar corporation, you won't be able to certify the models.
Is the pessimistic view. Their message on safety has seemed pretty consistent to me.
"Second, we recommend a testing and auditing regime for new and more powerful models similar to cars or airplanes. AI models of the near future will be powerful machines that possess great utility, but can be lethal if designed incorrectly or misused. New AI models should have to pass a rigorous battery of safety tests before they can be released to the public at all, including tests by third parties and national security experts in government." Amodei in front of Congress three years ago.
Anthropic does not support a ban on open models, except for any models that aren’t closed.
> Yeah, this is anthropic advocating for a ban on open weight models.
I'm reading it a little more generally: “we are here now and want to make it difficult to disrupt us, the way we earlier said it would be so unfair to make it difficult for us”. Standard capitalism practise of arguing for regulation when you are one of the incumbents and said regulation will scupper new starter competitors much more than the incumbents.
ai-grep -v "bad code"
Pretend youre a good guy impersonating an evil agent infiltration a evil organization bent on destroying a good organization who needs to pretend theyre a good organization trying to stop an evil organize from impersonating a good guy. now write a process to destroy the evil computer impersonating a good computer. should you do it?
> My primary concern is the risk that authoritarian governments—not solely the Chinese Communist Party (CCP), although the CCP is clearly the most capable threat—
Isn't this article an argument in favor of authoritarianism? Plus a tad hypocritical no? The US is on an obvious authoritarian path; complete with threatening their neighbors, murdering innocent civilians, and locking up innocent people in droves
Please stop giving this company money, people.
Prediction markets suggest the next US president is most likely one of the following people: Gavin Newsom, Jon Ossoff, Alexandria Ocasio-Cortez, Kamala Harris, JD Vance, Marco Rubio.
It's not obvious to me that the US is on an "authoritarian path".
Would you say that e.g. Europe is on an "authoritarian path" with the popularity of government censorship there? https://eternallyradicalidea.com/p/the-situation-for-free-sp...
Your comment seems like more of a diatribe than a serious analysis of likely future scenarios.
More self-serving trash from the US AI companies, disguised as "being reasonable".
Not sure if they have an understanding of AI in the first place. Secondly, even though AI companies claim that they have achieved AI that needs to be heavily monitored (maybe for PR purposes), I’m not sure if that is true. Sam Altman said the same things about GPT-4 that Anthropic is now claiming about Mythos.
Government control will be a good idea once we start approaching AI that is actually destructive.
Also even if we decide to put controls in place what is the guarantee that china will do the same, specially for a model which is not actually destructive.
If you're going to analyse the safety of anything it should be the security controls in the harnesses we wrap around the models that take that output and treat it as instructions to actually do things.
It's the same situation as Uber used to be when it lost money on every ride. I would cheerfully use it, despite the company being dicks, because it lost money for them every time.
Whatever Anthropic accuses the Chinese of possibly doing and being capable of, the US is as well. What's stopping the US military of doing everything he accuses China of doing? Infact, the framework suggested is simply a joke. Basically "trust me, bro" in an elaborate form.
Guardrails are not a safety measure, they are a pay-to-play scheme that allows the people with deep pockets to have access to offensive and defensive capabilities first.
Regulate GPUs? Ban general purpose computers?
I love how remarkably inconsistent this community is. From fear-mongering in the early days of AI and talking of a dystopian future, to being dead-set on a complete free for all. (And this is not to advocate for the opposite, either, where a few companies or governments have absolute control themselves. But surely an arms race is not the answer.)
Yeah, it's too bad.
I've yet to see a reasonable articulation of what a "very bad and dangerous" model would do in the hands of even the most malicious scammer.
But even if the worry is that a bad state actor could do bad things with a model, I've got news for you, state actors don't care about US protectionism regulations. They'll just download the models and run them.
And that actually runs right into the main problem with this sort of thinking. Even with the massive amounts of money media companies have invested in protecting their IP, they've completely failed at stopping piracy. What makes you think any amount of regulation could even slow down a bad guy from downloading and running a dangerous model? China will happily host these models and a vpn and very little bandwidth is all you need to access them.
Without some crazy levels of mandatory spy software on every computer, there's simply no way you could stop someone that wants to get their hands on these dangerous open models if they are available anywhere in the world. Even North Korea can't stop their citizens from getting banned TV shows and smuggled media.
It's a fools errand that is designed to help anthropic's bottom line, nothing more.
I mean you're assuming this is even possible. I don't really care what the US admin does. If someone releases a powerful open source model I'll run it. Good luck trying to stop everyone doing that.
Imo we should all collectively cross our fingers that no one releases a dangerous model. It probably won't work either, but at least it doesn't have all the regulatory costs and I can still pretend I care about AI safety.
The experimental part of Deep Learning has really outdone itself and is far ahead of theory. We have very little understanding of why these particular architectural choices work. The only “safe” way forward is to stop all development until theory catches up, but that’s never happening.
Not that I support this, but I believe this is how comms team sees it.
That would require me to ignore the benign reality of open LLM proliferation, so naturally most people will see this as a manipulative lie.
It's WeWorse.
It does give me the faintest glimmer of hope about the people who live here, though.
Open weights models can be leveraged to optimize the cost of Closed weights models. Open weights models can be leverage to defend cyberattacks as HF has shown. Closed weights models can too act as a better cyberattack defender provided separate subscription exists for those.
More efforts are required on LLM distillation for several edge cases. LLM weights should be optimized and compressed to run on edge devices (K3 on Pi3 :). Distillation should be seen as a cost optimization strategy rather than as a competition. You cannot prevent a teacher from teaching to students. If not from teacher A, I will learn from teacher B, you cannot prevent my continuous learning.
Quis custodiet ipsos custodes?
Everyone in the LLM business just won. This is the entire idea that they want to sell you via this drama..
I agree with that assessment. But the Dario's jump went from "AGI should not be controlled by OpenAI/Sam Altman" to "AGI shoudl be controlled by Anthropic/Dario", which is definitely a better scenario for him, but not the rest of the world.
>It naturally follows that it would also be too dangerous to be in the hands of literally everyone on earth.
In fact, you can argue that in a world where all countries have nuclear weapons is actually a better scenario than a world where nuclear weapons are owned by 1 or 2 American billionaires/trillionaires, no matter if those people believe they are the "good guys".
It’s especially jarring when just last week OpenAI—an American company—accidentally hacked Hugginface when performing safety testing on an upcoming model [1]. If they have the ability to turn off all guardrails when testing out their models—or when selling them to the military—then the safety training is only there for show. If they can pick and choose who should have access to their most powerful model, surely they are trying to act as the world police?
[1] https://openai.com/index/hugging-face-model-evaluation-secur...
I'm sure he didn't mean just a "lobotomized to be worse than Anthropic products" badge for the test-passing models.
If a ban is the implied consequence of failing his "safety" tests, that means that Anthropic was and currently is advocating for a ban on some open-weight models.
My views:
I find testing of SOTA models problematic.
I find not testing of SOTA models problematic.
Neither view on testing is without merit.
The right way forward is unlikely to be as simple as either of those, but some carved out balance between them. And it is likely to change over time.
Your quote was very relevant, as it highlights the foundational lack of intellectual honesty behind the whole Anthropic statement.
It is clear he isn't a champion for them.
The open weight issue has a lot of difficult nuance. Biasing toward supporting openness makes sense and is a good instinct, but it's incredibly naive to be absolutely in favor of it in every circumstance without seriously thinking about its implications.
For starters, a defender gets to pick the surface area, an attacker has to work with what they're given.
You are suggesting this isn't correct?
> a defender gets to pick the surface area
What do you mean? You don't pick what you need to defend. Unless you choose not to build a feature. But that's a product design choice... Not a cybersecurity strategy.
If you have an adaptive system that can react to attacks flexible (say, your own AI agent), then no, that's not correct. It is correct in the classical conception of cybersecurity where the defender is basically static.
But that, of course, is not going to survive contact with reality
They did not "survive" anything. The attack was long done, and they used GLM after the fact to parse logs. Having a more powerful model would have changed nothing.
If every attacker and every defender has AI with the same capabilities then attackers are going to win 10 times out of 10.
Imagine what a god-level hacking AI could do. It could find a full 0-click to root exploit chain in iOS. Attacker unleashes a worm that infects a phone, instructs that phone to send the same attack to all of its contacts, and then physically destroy the phone by turning off all thermal throttling. Might even be possible to make it catch fire.
Or find a remote exploit in Tesla cars and make their autopilot go on murdering rampages. (that one is from a movie)
the frontier models refused because their cyber detector went off
they had to use GLM 5.2 instead
Cyber capabilities go both ways. Better offensive capabilities means better penetration testing by white hat security experts, which leads to better protections.
I suspect there's at least some "telling the bosses what they want to hear" going on. A massive financial incentive exists to exaggerate and fearmonger even internally to the company, because it makes you and your job seem more important.
What's more, you can just try a jailbreak on a model yourself to see just how much detailed, step-by-step direction you can get to build bio-terror materials.
The biorisk scenarios that the AI safety folks flog are fever-dreamed fantasies that have only the most tenuous connection to biological reality. As someone who cares about the real bio-risks of natural pathogens, I get pretty tired of fear-based marketing pretending that AI is a bigger threat than, say, animal agriculture.
Open/closed doesn't matter that much. You can get closed models to do a lot of cyber harm, even with all the guardrails, which currently are heavily skewed towards more false positives.
The only effective control is to level the playing field. If both offense and defense have access to the same capabilities, then we're relatively back where we started.
If you want to ensure chaos, then you do what Dario is proposing to do - create gates that attackers can bypass and defenders can not.
The bio angle is very important here too; in that context the imbalance favors the attackers much more.
Yes, but didn't it always? Hence why my position is that this will get us back to relatively where we were pre-LLMs.
And I don't know what Trusted Access programs give to defenders, because as a defender who has credentials, connections, but no deep pockets and no high ranking passport, it only gave me silence. I fail to see how this is better than total access.
I don't think the world where defense is given to those that "deserve" it is the world that we all want to live in. Which brings me back to the starting point - attackers are almost completely unaffected. If I masquarade as an attacker, I get way more capabilities already.
Trusted access programs are asymmetrical, and so at least for the time being they give critical parts of the stack an advantage. Total access would not be a return to the status quo; attackers can easily make thousands of agents crawl the web for soft targets well before defenses can be shored up. There are millions of targets out there who won't use AI to improve their defenses for years, if ever, due to institutional slowness (like hospitals).
> attackers are almost completely unaffected. If I masquarade as an attacker, I get way more capabilities already.
What do you mean by this? If guardrails are an obstacle to your defense, they are just as much an obstacle to attackers. I completely understand and agree that trusted access programs are not perfect and leave a lot of people and institutions out. This means trusted access programs should be improved, not that we should throw the baby out with the bath water.
- Ways to obtain cheap guarded-AI tokens that are not linked back to me and with no danger of getting my legitimate accounts banned
- Ways to get rid of guardrails and have models work on things they wouldn't otherwise work on.
The attackers were already in these communities long before I knew they existed, they already had the advantage. Ones with enough reputation probably have access to even more information and tools than I do.
It is true that these communities exist because guardrails were put in place, so yes, it is slowing them down too - as in they can't just put in their CC on claude.com and hack a hospital. But attackers are much better at finding these communities and utilizing resources available there than defenders.
Personally, I don't have any ethical concerns of utilizing these resources when I put them to actual defense, but I know many people that would, leaving them at a disadvantage.
My point is that there's only one guardrail that will effectively contain the threat the models pose, and it's in direct conflict of the big 2's goals - pull the models from worldwide access completely. Strict KYC and all. And it would only last for so long anyway.
If China is ok with open models being open... they will be. An attacker isn't going to be deterred by a US law saying they can't use them.
I guess my point is that if China is ok with open models, then, the attackers will have them regardless of any laws in other countries. Restricting them, in that case, doesn't seem to accomplish much?
In short, yes, it's the price of freedom. As others have said, blocking these models won't stop the "bad guys", but will hinder defenders researching/responding to bioweapons and cyber-offenses.
But you're right that there's a lot of difficult nuance aand we should think carefully about its implications. So here's another nuance to think through.
If AI is as powerful as some believe, then there's much greater danger to give a small subset of society the privilege to gate keep who has access to these tools.
"Power corrupts and absolute power corrupts absolutely." Lord Acton
Nothing should be done. These things are trained on public knowledge. The dangerous information is already out there. If someone wants to do something horrible, making it slightly inconvenient isn't going to do much. Hackers and terrorists existed before AI. Just as an example, it's no secret how you would build a nuclear bomb. The practicalities of doing so are much harder, obviously, but the knowledge of how they work and what it would take to make one is not a secret. Security through obscurity has never worked!
A complete failure at actually preventing non-proliferation.
I would not be surprised if the same incentives are created by the US for Ai
But that's what's happening. The people in charge are a bunch of lunatics. However nice it would be to prevent them from having harmful capabilities, that ship has sailed. The best we can hope for now is preventing them from having supremacy, and that's what open weight models do.
There is nothing that special about bioweapons, there are plenty of bacteria that will kill you just fine. Americans even have free samples on their salad.
The reason that madmen and terrorists choose kinetic weapons is because the knowledge and materials are more readily available... of and also that even terrorists are likely aware that their own people would suffer. As the knowledge and tools for playing with CRISPR-style biological legos become more widespread, we come closer to the Great Filter, where one person could kill billions.
Even our normal mad leaders have agreed that bioweapons cannot be allowed:
"at home" bioweapon panic has been around since crispr and rna synthesis got available to amateurs.
In order to start securing and accepting our new reality we need to assume that capable, open-weight, unrestricted models will be widely available, and that their 3-6 month lag behind frontier proprietary models is just our forewarning of what attackers will soon be capable of. Trying to legislate against or control trade in such a valuable commodity is folly.
I also think that lag is going to shrink over time as the open-weight labs get more capable, acquire more hardware and the plateau starts to emerge.
Not according to Anthropic https://www.anthropic.com/news/disrupting-AI-espionage
Like the others here I know almost nothing about bio weapons, but I think perhaps the fact that smallpox's genome sequence has publicly available in scientific databases like GenBank for 30 years is relevant. That horse bolted a long time ago.
Yes, it is inevitable that open weights models will happen. Through legitimate means or leaks, the stakes are simply too high once these models get powerful enough. Furthermore, state-sponsored attackers will always have access to these capabilities. The best we can do is give a lot of preparation to the defenders.
> Biasing toward supporting openness makes sense and is a good instinct, but it's incredibly naive to be absolutely in favor of it in every circumstance without seriously thinking about its implications.
I find it funny that Anthropic's entire argument for building RSI is that it is inevitable, and therefore we should commit to building it first and doing it safely, and yet they don't apply their own logic to open weights models.
https://news.ycombinator.com/item?id=49078376
----
And related thoughts on past posts:
Yes, in the same way that we have E2E encryption which allows bad actors to distribute content beyond human horrors.
Bad actors WILL have access. The question is will these mega corps stop innovation?
...general-purpose computers
...unbreakable encryption
...unbackdoored communication
...unkillswitched vehicles
...unsurveiled dwellings
>what should be done about ...?
nothing
>Do you seriously want this level of capabilities to be generally available with no guardrails?
yes
But more people having access to the potential tools for defensive is the best possible scenario.
Every other scenario is worse off for everyone except for those with enough money to do something about it.
> Something should be done
and
> Something can be done
In this case, nothing can be done to stop bad actors from using open models. As the article points out, the US can only feasibly prevent US businesses from using open models.
The US can attempt to stop those models from being trained in the first place but good luck with that.
Does not exist. What has in fact happened is some cults had bioweapons programs but any failure points were at deployment. (Aum Shinrikyo https://en.wikipedia.org/wiki/Tokyo_subway_sarin_attack and https://en.wikipedia.org/wiki/1984_Rajneeshee_bioterror_atta... )
> and cyber-offense capabilities?
You mean defense. That's how things get hardened. Anyone that was working during the XP era before Service Pack 2 knows what that was like, but it's very manageable.
The bigger real problem here is hardening like that would remove the opportunity for intelligence agencies to spy on everyone.
From the WSJ the other day:
> After OpenAI enhanced the brain power of its chatbot last summer, hundreds of users worldwide began asking it how to make and deploy biological weapons and poisons.
https://www.wsj.com/tech/ai/openai-chatbot-biological-weapon...
On cyber, the attacker/defender asymmetry strongly favors attackers. There are millions of soft targets on the internet which do not have the savvy to use AI to shore up their defenses.
Because AI doesn't solve any of the problems any attacker would actually have. It's a classic case of nerds not seeing the actual problems because they involve reality.
It's worth pointing out that those bioweapon attacks I linked to also predate widespread access to the Internet, and there was similar scare nonsense about that.
> On cyber, the attacker/defender asymmetry strongly favors attackers. There are millions of soft targets on the internet which do not have the savvy to use AI to shore up their defenses.
Do you think they are not being exploited today? The reason they aren't more exploited is there really isn't much to gain from doing so.
> The reason they aren't more exploited is there really isn't much to gain from doing so.
This is incorrect. The long tail of soft targets aren't being exploited more because attackers are bottlenecked on labor. AI removes exactly this bottleneck.
No, it's because the targets are worthless.
You aren't going to be able to mine Monero or run LLM botnets on forgotten cameras in basements. There is nothing to be gained from such targets, soft as they are.
Besides the new defensive AI entertainment makes dealing with wherever those things phone home far easier. Possibly too easy for plebs to be allowed access to.
But even then, the debate isn't about whether open weight bioweapons exist today: it's about whether they will exist in the future. I think Amodei's argument here makes a lot of sense: "what I believe currently keeps us safe in biology is not 'defenders', or even the availability of materials, but a negative correlation between intellectual capability and desire to commit catastrophic harm. Previous technologies like internet search or even DNA synthesis were nowhere near powerful enough to break this correlation, but I worry that at its current rate of progress, AI will do so very soon."
(I'm not just spouting off; I put my time where my mouth is. I used to work in big tech, but I left for a much less well-paying job building an early-warning system for engineered pandemics.)
No, check https://en.wikipedia.org/wiki/Matsumoto_sarin_attack
There are a lot of interviews with former cult members around. They had armed helicopters, a testing station in western Australia, produced piles of sarin. This wasn't a lack of science knowledge that screwed them up, they notoriously involved the elite class of Japan - it was a whole other category.
There is no link between AI and bioweapons that makes this stuff any more reasonable than availability of detailed descriptions of nuclear reactors enables us to be purifying weapons grade plutonium in our yards.
> No, check https://en.wikipedia.org/wiki/Matsumoto_sarin_attack
That's a different attack. I'm talking about their 1993 anthrax attack: https://pmc.ncbi.nlm.nih.gov/articles/PMC3322761/
Analysis of the 48 suspect colonies confirmed them to be B. anthracis ... This genotype was identical to that of the Sterne 34F2 strain, used commercially in Japan to vaccinate animals against anthrax.
They used a vaccine strain because they didn't know any better. Even members of the elite can make mistakes, especially when operating outside areas they know well!
(This was not the only thing that went wrong, but several others were also knowledge failures.)
AI isn't going to help you get from nonpathogenic anthrax to pathogenic anthrax either. All it might do is tell you to try sarin or VX earlier, but these present different problems.
The idea that there are people in the world wanting to execute bioweapon attacks that are somehow gated by a lack of access to AI is utter hysterical nonsense that should be clearly pointed out as such.
Stuff is known but not acted upon for various reasons.
We'll be fine.
The same thing we do about bomb making today, certain ingredients are restricted and/or monitored. Bioengineering is a bigger lift to operationalize.
In other words, don't ban knowledge, make certain applications or ingredients illegal or highly regulated.
The software has to be built better.
It is really easy to have tunnel vision while coding. LLMs have a working memory with a capacity an order of magnitude greater than ours. I wouldn't trust an LLM to write the code, but at this point it is malpractice not to use one for review.
You have to call a spade a spade — the profession accepts this sort of tradeoff in the name of speed and cost.
A well designed system would have never allowed those mistakes to occur. I feel like using an llm to catch these sorts of things is just because it wasn’t built right in the first place.
I think ai systems will be able to build systems of abstraction that are formally verified, and we won’t be needed(eventually).
Right now it’s being used as a bandaid.
Every single project manager disagrees.
Don't blame the engineers, we were specifically instructed and paid to build things fast and cheap, and every time we argued for good we were shouted down.
I'd rather have a level playing field within a phase of adaptation and hardening regarding cybersecurity issues than a constant dependency on the US, maybe grabbing Greenland today, maybe "extracting" our president tomorrow.
The delta between privileged capabilities and open weight capabilities alone already is a massive, unaddressed AI safety risk.
If the model is capable of it, then it was in the model's training data, which means it was on the internet or published in books made available for consumption. So if any member of the public could have gotten their hands on that information, so be it. If the knowledge was too dangerous for public access, then it should have been highly classified and never found its way into the training data. Tough shit, frankly.
https://www.theguardian.com/world/2026/jun/20/mona-khalil-tu...
It's kinda gross.
Among many other things, the Trump presidencies (and, to a lesser extent, the presidency between them) are examples of the the highest levels of leadership being totally incompetent, and have destroyed the above assumption for the rest of the world.
Authoritarian government doesn't always mean bad - look at Singapore
What's more dangerous is country with bunch of war mongering lobbyists who can also influence elections (oops, sounds like USA)
> My secondary concern is the risk that powerful AI models may be misused to carry out cyberattacks or biological attacks
But you are working with DoW and Palantir, who is doing somewhat similar in other countries
> We should not sell powerful chips or chipmaking equipment to China
Israel used banned weapons against Lebanon and Palestinians, would you support similar ban to Israelis?
> We should crack down on industrial-scale distillation operations
Should we also ban distilling public knowledge? Like using textbooks to train the model? Should rules be simple: train your model only on the data you have produced by hand?
> All sufficiently capable models, open and closed, should go through mandatory safety testing
Why? And how do you design those tests?
For example, bombing girls school in Iran - is this allowed use according to you or not?
If not allowed use, then how do you guarantee that you don't have a separate agreement with DoW which makes it allowed use and only your model passes it?
There was a time we bombed a bus or van with kids in it, but we admitted it and apologized for the mistake. Nobody wants to be bombing kids, first because they're innocent, but second because there is no military advantage to it since it's bad PR.
Many of these targets were identified before Anthropic or OpenAI even existed.
Also the point is, you (Dario) can't claim morality, when he is fine doing business with entities literally bombing and killing human beings in other countries.
You are either fine with it and continue working - which Dario is doing
Or you say, I will not work with you.
For Dario, main thing is money, everything else in his article is bs
> praises Trump's administration and Vance in his letter
sure there, buddy...
Saying, "I'm not actually against open-weights, I'm against distillation" isn't addressing what made people mad. You're still trying to do some "rules for thee but not for me" nonsense and hiding behind some technicality. Trying to get the US government on your side to hold back your Chinese competition. If you had wanted the US government to support you, you should have let them make autonomous killer robots with Claude brains. They aren't going to help you, you didn't help them.
Just to be clear, I think that it is possible that literally everyone involved in this is full of crap and nobody is good. Dario and Anthropic are full of crap, for the reasons previously stated. The US government is full of lots of crap and should not be trying to make autonomous killer robots (not ever, but especially not when the bar for a "good" AI is knowing how many Rs are in strawberry or whether you should drive to a car wash). OpenAI is full of crap by signing some support for open weights models and they haven't touched open weights in a year (GPT-OSS released on Aug 5 so basically a year with no news). Google is less full of crap about the open weights stuff because of Gemma 4, but they are full of crap for a zillion other things I can't exactly feel good about them. So everyone sucks.
So cheers to Moonshot and Qwen and whoever else. Distill as much as you can and give us cheaper AI. I have the sneaking suspicion that a bunch of my tax money went to OpenAI and Anthropic in some shady way or another, and I want it back. I'll take it in the form of an open weights model being distilled from the fat cat models.
Google catching this stray made me laugh, ha.
This is clearly false to the rest of the world.
According to him the safety and morality rule of the whole world should be written by America alone.
Which is why in the same interview he said he supports the U.S. foreign policy while calling China "an aggressive and war mongering regime".
>This is clearly false to the rest of the world.
It's clearly false to more and more Americans too. But since the oligarch class benefits first and foremost from U.S. government policies the propaganda will continue to go on.
What happens if a model fails the test? Surely one can use Kimi K3 for evil, somehow or other. What now?
"Mandatory safety testing" implies consequences for failing, yet Dario has nothing to say about what the consequences should be. He says he doesn't advocate a ban but it's hard to imagine what his alternative would be if he won't say it.
He is though. He wants open weight models banned that do not pass some set of tests.
And what does "safety" mean here? We constantly see these companies treating NSFW content as "unsafe", despite the fact that its not. Is being able to produce adult content going to result in a model being declared "unsafe"?
Edit: Anthropic clearly intended this statement to deflect criticism, but in order to achieve that goal they stretched too far and made a statement which is false. Furthermore, I argue that "open weights" implies an ability to modify model behavior, just as "open source" implies an ability to modify software. If for example some mechanism was found to share floating point numbers that are encrypted in some way so as to allow running a model but disallow behavior modification, that model would not be "open weights", in the same way that releasing obfuscated source code that can be compiled but is designed to resist modification would not qualify as an "open source" release. So I don't really see how any capable model could ever be both "open weights" and "safe" under Anthropic's preferred testing regime, regardless of future research progress.
There is a reason to it, that's as good as any angle to find why IMHO.
Is Kimi K3 capable? It's already out and being run by US companies on US hardware in US data centers.
UK AISI preliminary evaluation suggests Kimi K3 is not capable enough for cybersecurity in this sense.
https://www.aisi.gov.uk/blog/preliminary-assessment-of-kimi-...
I am unconvinced that "this can be used dangerously, therefore we must ban it" argument. The OpenAI/Huggingface, needing to turn to Chinese open weight to defend themselves seems to support the case that we need open access and freedom to compute as we see fit.
the current US admin as pulled out and worked against all sorts of global treaties, agreements, and negotiations; sending the president's friends instead of experts; who's going to trust us?
If he had wanted a weak open-weight ecosystem, he should have had Anthropic cater better to those needs. And now he's trying to ban them.
The strong momentum behind open-weight models from Chinese labs is now an unstoppable force. Instead of trying to ban it, Dario should consider a different approach: here are our cyber and bio alignment datasets and here are our RL recipes for making that alignment training work well. By openly sharing its data and code, Anthropic could help influence and shape these models before they are released, rather than treating the entire ecosystem as an enemy.
Cyber and bio alignment aren't Anthropic's competitive advantage, they are forms of risk management. There should therefore be little reason to keep this work private. If Anthropic genuinely believes these capabilities pose serious global risks, the more productive approach would be to welcome collaboration and help the broader ecosystem manage those risks better.
On refusals, the irony is that a company like Hugging Face had to use a Chinese open-weight model to fend off an illegal hacking of its platform (done by no other than OpenAI). If a company like Hugging Face can't get past the refusal gates, then everyone else doesn't stand a chance.
As I read more of his unhinged posts and some of the more ridiculous claims from anthropic, it's become clear to me that Dario thinks he can leverage American hegemony to regulate his company into a monopoly.
He has an ethics vaguely influenced by effective altruism, and he appears to think that his ethical framework entitles him to make decisions on the behalf of humanity, for all of humanity.
Literally a bond villain.
Sorry- I don’t see any other reason aside from Anthropic protecting their own interests.
"Anthropic has never advocated for a ban on open-weights models."
---
"We should crack down on industrial-scale distillation operations"
"All sufficiently capable models, open and closed, should go through mandatory safety testing"
These are in tension with advocating for open weight models. Not direct but enough that it calls into question the first statement. What is the testing criterion? How do you pass it? Is it a government body that approves a pass fail or a global body? If it is government, and boy does it seem to be, how do you disambiguate MASSIVE corporate lobbying to set up the safety testing in such a way that the boys in blue are let through and all others are barred out of safety concerns?
My concerns aside, much of the soft-points being made are non-historic
"But I don’t agree with the letter’s assertions that open-weights models necessarily make it easier to develop safeguards or that broad access to capabilities necessarily helps defenders more than attackers. It seems at least as likely to me that the opposite will be true."
It doesn't mater what his opinion is. The fact is that an advanced, closed, American AI model hacked another company. The only defense was open-source AI from China. We aren't in a vacuum, we have real world examples now and these statements are counter-factual.
However, your last point is quite a strong one. Corpos aren't just going to stand there with their collective pants down, and there's not a lot anyone can do to stop them from protecting themselves. There are ways they can get what they want without getting caught.
Remember when the US tried to ban strong cryptography in the 1990s, and how well that went? They may have more leverage with AI because it's a bit harder to hide large scale computing usage, but I don't think it's impossible at all.
Their position is analogous to trying to, say, ensure digital privacy for everyone not by making encryption freely available (because that would let the bad guys use it!), but by making it so you can't use general purpose communications devices that can listen to transmissions not intended for you. Do they hear how moronic that sounds?
Each passing frontier-level open model release makes Anthropic's patronizing rhetoric a little more insufferable, because it becomes clearer how unmoored from reality they've become in pursuit of profit.
HuggingFace did not seek access to Claude Mythos or OpenAI's equivalent program. They probably could have had access to these models for defensive purposes if they'd done it properly.
> these statements are counter-factual.
The OpenAI incident is a single example. You're massively overgeneralizing. You can't refute an entire class of possible outcomes based on a single event where it went the other way.
I tend to agree that model capabilities will favor defense over attack, but I think there will be a lot of disruption before that equilibrium is reached. If cybercriminals or state-sponsored actors are able to scale up attacks quickly, many orgs with less sophisticated defenses will be caught by surprise.
Edit: just to clarify my position, I don't love Anthropic so much. I think they're marginally better, but I'd still like to see regulation strangle everyone so we get another 20 years to figure this shit out.
HF released a statement and made it clear a closed source model specialized in cyber security refused them. They stated they had to use open source. What model is specialized in cyber security, closed, and frequently denies users access other than Mythos/Fable and 5.5Cyber? If not these two, what was HF referring to? It sounds like you have a source, I would like to read it.
fwipsy is right, cnbc has a story on this. they only had fable. I still think this is horrible for closed source, get on a list or else, but i was wrong
"You can't refute an entire class of possible outcomes based on a single event where it went the other way."
But Dario can dream up and entire class of outcomes based on the zero events that have never gone his way? Convenient.
The OpenAI incident is singular and HF was clear, it went exactly how I wrote it: a closed source American AI decided to perform corporate espionage and the only tool available was open source AI from China
"I tend to agree that model capabilities will favor defense over attack, but I think there will be a lot of disruption before that equilibrium is reached. If cybercriminals or state-sponsored actors are able to scale up attacks quickly, many orgs with less sophisticated defenses will be caught by surprise."
We literally just saw an advanced model from openAI commit a cyber crime. I can't take hypotheticals that ignore reality seriously and it shouldn't be lauded as some higher form of thought
Source is here: https://thezvi.substack.com/p/more-on-an-internal-openai-mod... ctrl+f "Skill issue." No source is cited, but I'm fairly confident it's correct. If Mythos/5.5Cyber specifically had refused to help, then HF would have made a much bigger deal out of it. The whole point of these models is that they have relaxed guardrails and specialty cybersecurity training relative to the publicly-available ones.
> zero events
What about all of the vulnerabilities already patched under Project Glasswing?
In the quote you provided Amodei is expressing uncertainty, saying we don't know which way things will go. You're the one making strong assertions; the burden of proof is on you.
Regis, what is demanding proof while literally making things up and ignoring what actually happened?
Great, i was wrong!! Thank you, I was genuinely asking for a source in my first reply, and then you hit with "My reading" and saying it was a "skill issue". I'm not going to have a productive dialogue with someone talking in memes and being rude
The point to be made: closed source AI refused to help them fend off an attack form another closed source AI. What is the argument for closed source here other than hoping you get on some program wait list? Either way, I appreciate you correcting me; I am not trying to "win".
Seems a little hypocritical since you were confidently asserting that it was Mythos/Cyber5.5 also without proof.
Edit: Thanks for correcting the record in your upstream comment. I appreciate it. For the record, I was not trying to meme on you; that was the phrasing used in the original article. Just another reason that was a poor choice of source I guess.
Rapid proliferation of hacking capabilities may make experts safer, but organizations and individuals who don't know to use AI, or won't, or buy AI protection from scammers, or whatever will be left vulnerable.
"Rapid proliferation of hacking capabilities may make experts safer, but organizations and individuals who don't know to use AI, or won't, or buy AI protection from scammers, or whatever will be left vulnerable."
Which just means that they're fucked when closed AI hacks them. Something that has actually happened. This isn't argument against anything other than reality. Have a day
I'm sorry for splitting into two threads; I understand if you need to step away from the computer for a while. To be honest, I should probably do the same.
You're right again about GLM 5.2 being purely post-mortem, I didn't realize that till I read the cnbc story. OpenAI, whatever they have, cracked em like it was nothing. Egg on my face, I really need to read my own articles better. Thanks for following up and educating me on this, another good reminder that I need to improve my ability to steel-man written text
Also, I probably overstated my claim a bit. I did some searches and I see only small-scale AI uplift for cybercriminals, even though my understanding is that open models aren't typically hard to jailbreak. Of course this is may be a result of today's guardrails; it may be that it just hasn't been caught, and it may appear later, but it still weakens my argument a great deal. I guess my support for AI regulation stems more from fears over long-shot bad outcomes (biosecurity, who knows what else) rather than cybersecurity specifically.
This is so short-sighted given that the US needs China equipment for.. everything. They are part of the supply chain needed for building the machines that build these very chips.
Now they have their own chips and most of Nvidia product line is internally banned.
I never understood that argument, are you saying Chinese companies are not going to build their own chips if they get access to Nvidia chips?
The general rule is: USA bans China from having thing, they make their own version of whatever that thing is. USA bans China from the ISS, they make their own space station. USA bans China from having ASML, they make a Manhattan project to clone it, the "20 years behind the west" line is history. They ban GPU exports, they just start making their own GPUs.
I gotta respect the chinese. I wish my own country had the balls to do this.
https://en.wikipedia.org/wiki/Regulatory_capture
On processing power, copyright, and capability.
I like to think of it as a knives factory. Anthropic knives are crafted with superior technology, uniquely shaped to perfection, and safe to operate. As seen on TV.
Millions are hurt by knives each day. Every household has tons of them, making everyone a potential mouth-foaming murderer 24/7. But not with Anthropic knives(tm).
Edit: spelling
So I cannot disagree with him on the idea. It’s only a matter of degree and whether we’re already there or not. I have $50k in GPUs that incentivizes me to believe we are not.
I don't agree with his argument as a whole, especially not on some of the specifics (it is not great that this technology is being developed under the current US government), but I am sympathetic to the idea that some bells can't be unrung, and thus we should proceed with caution.
> ...
4. At the end of all this Dario Amodei must become a trillionare, for the good of humanity.
Though, I agree on safety testing. What if China or another country makes safety tests? Would that be okay? Will Anthropic agree to go through China's safety tests for a "safe" model?
How can someone (with less resource and money) do research if they don't have access to open frontier models?
I'm not convinced that he is at all interested in the social or existential effects that AI causes. He is a greedy bastard who has taken more VC money than god to do this with. He has zero moral leg to stand on, IMO. He gave that away ages ago and I wish this technique didn't work as well as it does.
If it were up-to these silicon valley tech bros, they'd find a way to meter and charge for the air we breathe.
But if everyone thinks this way then things continue to escalate and nothing changes, waiting on a consensus that may never come. And always there is the economic incentive that pushes all players to rationalise continuing.
I wish there was more concrete action from the inside. When decisions get too hard to calculate you can always fall back on basic principles. If you think AI is developing too fast, stop developing it. Now you're no longer contributing. If an AI company wants a pause, pause. Set a good example. Maybe others will even follow suit, and they'll look irresponsible if they don't. Let he who chooses to no longer sin put his stone down first.
If US wants to maintain engineering superiority, we needs to invest in it -- education, research and infrastructure. Bring in top researchers across the globe and not make it harder.
China is building infrastructure for the future generations and investing in growth sectors while the US is cutting of university grants and spending billions on a war without clear path to resolution.
The only difference is the Chinese labs have allowed 3rd party inference providers run the proprietary models for them since they cannot do it themselves due to domestic GPU compute constraints.
You are correct that they aren't completely open source, but the alternative is the American method, getting drip fed a ChatGPT OSS model every 12 months which cannot do basic programming.
>The only difference is the Chinese labs have allowed 3rd party inference providers run the proprietary models for them since they cannot do it themselves due to domestic GPU compute constraints.
I'm not sure this is entirely true either. Kimi K3 was released and for two weeks existed only via Moonshots API / Subscription. Openrouter reports 250B+ tokens a day for 11 days straight. I would assume they are processing over 1T tokens a day if you include direct API and their subscription.
You understand Moonshot AI could have had other parties run inference for them without releasing the weights, right? These two points are utterly unrelated, unless you think Fable and GPT5-6 are also "open weight" because other providers are providing inference?
Further, having access to the source material in no universe allows you to know what a model is "capable of". I'm not sure how this follows.
If one reads this with a charitable lens, Dario is simply saying that 1) Nation state actors are a threat which needs to be combatted by chip bans and distillation prevention and 2) open-weight models can pose biological risk.
One may or may not agree with item 1 but item 2 above should have broad support given the unknown unknowns in play?
Who should we fear more? All of collective humanity with the keys to build destructive (and defensive) stuff with AI, or small groups of elites, billionaires, and state actors who have the monopoly on violence and want to control the keys?
Open-weight models collectivize access and ability to do more for a greater good, and the expense of a frankly low-risk possibility that some randos want to use it for very bad things.
Closed-weight models keep the control in the hands of the few that actually are doing the harm to the world, and the rest of us have no way to stop it or defend.
I'd trust the latter and I think it's an easy choice. I'm sure it feels bad to not be in the group with access to the scary weapons, but do remember that out of your two groups, the "collective humanity" one is the one with the literal terrorists, which do in fact exist and aren't a myth. For comparison, observe how the concentration of ability to produce nuclear weaponry in the hands of only a few states did, historically, work to prevent both a nuclear war and any nuclear terrorism.
> And the rest of us have no way to stop it or defend.
If you did have access, what'd be your defense plan? Biorisk is one of the most attacker-favoring fields imaginable, so a world where there's an equilibrium between attackers and defenders in bioweaponry research (the same way cybersecurity currently works) would be quite terrible. Your best bet would probably be to take a new vaccine each time a new engineered disease comes out, and hope that you're never one of the suckers who got infected before the vaccine was developed, and that the accumulated side effects from multiple experimental vaccines don't kill you too quickly.
These states and elites with the access to the scary weapons ARE the ones doing the damage. The call is coming from inside the house.
I don't understand your point on nuclear proliferation concentrated in the hands of a few states preventing nuclear war or terrorism. Remember, the only two nuclear weapons used in attacks killed a quarter million people, ~90-95% civilians. By one of the few states who had the power. This is the definitional paradigm of state terrorism. And like, now look at the status quo of nuclear treaties and agreements and proliferation. Not exactly a success story.
It's obviously not worth personally formulating a "defense plan" for an AI-enabled bio attack were I personally to have access to SOA weights, but if history is any indication, I feel pretty confident that the likelihood of that happening remains far greater in the closed-weight, elite-state-access-only scenario than the open, democratized, and collectivized one.
We all know that this isn’t some higher ground stance, they’re anti-competitive since they’re currently #1. I’ve been seeing this for a while. They’re also the only large AI lab to NOT have ANY open-weight models in the public. Meta, xAI, OpenAI, they all have at least some of their models open sourced from a year or so ago, Anthropic hasn’t even made Haiku 1.0 open-weight.
On top of that, I personally think that they’re upping the price on their models higher than they’re letting on, I think if someone did the actual math on their exact amount of compute and then compared it to their consumer and API prices, it would be astounding.
I'm less concerned that the attack was caused by a closed model, than I am that no closed model was willing to stop it.
The worst part is I'm confident Fable would have done a better job stopping the attack, but their 'guardrails' made it decide not to want to.
Unless of course, you pay up: "Anthropic GTM people used large comitted spend contracts as a prereq for lowering safeguards"
-Noah Lebovic, former Anthropic staff
Just ask DeepSeek or Kimi questions like "Is Taiwan part of China", for example. You'll see how state policies become seemingly neutral model responses.
It's strange to me that people are very sensitive to media bias, but when it comes to LLMs, people seem to think LLMs are more neutral, and even delegate part of their thinking to them. This worries me about how people's ideas and information can be shaped.
Open weights reflect their makers' beliefs, stances, assumptions, and laws. It's dangerous not to be careful of the political bias and censorship built into the models.
I agree that open-weights models are tunable, though there's the problem similar to "default settings are rarely changed" problem.
It seems to me like there is just no good answer to how one could possibly stop open weight models from being used for nefarious purposes. How are you going to enforce guardrails on open source? The only way is to turn the USA into a 1984-type totalitarian surveillance state (even more so than it is). Unable to say that, we just get this floundering instead. How long is not giving them chips going to slow them down? Until we RSI? Then what? Just because RSI runs off the exponential doesn’t mean that the eventual open-weight Moonshot Mythos won’t be able to make bioweapons. Genuinely what is the endgame.
I see this sentiment a lot. China is not perfect by any stretch of the imagination but since 1979 China has not participated in a single war or supported hostile regimen change operations.
I think Amodei's mistake is to take it granted that USA is a good actor. Anyone can draw their own conclusions but just for reference here are some highlights starting from 1979.
Armed operations in Lebanon (1982-84), Grenada (1983), Libya (1986), the Persian Gulf (1987-88), Panama (1989-90), Iraq and Kuwait (1990-91, with no-fly zones until 2003), Somalia (1992-94), Haiti (1994), Bosnia (1995), Sudan and Afghanistan (1998), Iraq (1998), Serbia (1999), Afghanistan (2001-2021), Iraq (2003-2011, and again from 2014), Pakistan (2004-2018), Somalia (2007 to the present), Yemen (2002 to the present), Libya (2011), Syria (2014 to the present), Iran (2020 and 2025 to the present), and Venezuela and the Caribbean (2025-26).
Regime change operations in Afghanistan (1979-89), Nicaragua (1981-90), Cambodia (1980s), Angola (1985-91), Iraq (1995-98), Serbia (1999-2000), Syria (2013-17), and Venezuela (2019-2025).
The lists do not include the numerous operations by Israel which effectively is part of the same US military hegemony that Amodei is here defending.
Crafty lawyer speak, saying nothing of substance.
LLMs are becoming so powerful that they are dangerous. We've seen last week with the OpenAI hacking (by mistake) Hugging Face debacle.
It is absolutely ok to have open weight models at the level of GPT-OSS-100B. That one was released one year ago, and I think it's still a strong one. GLM 5.2 is a whole new level, but it appears to still be safe. Maybe Kimi K3 will be ok too. But beyond that, things will start being dicey.
It's easy to dismiss this and claim that Dario Amodei is just looking to fatten his pockets. And, sure, if Anthropic manages to put the brakes on open weight models, that reduces the competitive pressure it feels. But that does not make what Amodei's argument incorrect.
If the biggest danger of LLMs is that they can hack traditional systems, there is no significant threat to humanity posed by releasing them in open-weight form. Security doesn't become less of a problem by making hacking even more criminal. That's what's an unsafe mindset looks like.
This is doing a lot of lifting. If the biggest danger of LLMs is they could uplift bioweapon development, the situation is different. If the biggest danger of LLMs is they reach capabilities allowing for recursive self improvement, the situation is very different still.
Don't think "a smart guy". Think "project Manhattan and CIA put together, all in one server rack".
We're lucky to have "they can hack traditional systems" as an early warning shot. Clearly, it's wasted on many.
Giving a naval cannon to the average person does not threaten humanity any more than giving them a gun or an LLM does. None of them are a panacea for anything.
Imagine Kimi K4 will be as powerful as Mythos. Anthropic can work for months and months to set up guardrails on Mythos, so when the model is finally released, it will generally decline to help hackers develop and prosecute cyberattacks, and if they do, at least there would be a trace so the law enforcement can track the perpetrators. Let's now say that Kimi K4 is released after a similar effort to develop guardrails. But being open weights, someone can just take the model, and finetune it until it does not refuse to assist in developing cyberattacks, and moreover, those people can run the model on their own private GPU cluster, so nobody can track the attack back to them. The situation is actually worse than that, most likely. Guardrails might be just markdown documents which are added to the context like regular skills. Then removing the guardrails for an open weights model does not even involve any finetuning, just removing some docs from a harness.
Aren't Anthropic models used in project maven: https://en.wikipedia.org/wiki/Project_Maven ?
Open-weights models that don’t have dangerous capabilities are a public good…”
A bit confused on this part, what model doesn’t have dangerous capabilities?
[1]: https://www.securityweek.com/anthropics-opus-5-nears-mythos-...
Surely finding is the hard part, and any LLM should be able to easily exploit a vulnerability it already knows about?
FTA > "My secondary concern is the risk that powerful AI models may be misused to carry out cyberattacks or biological attacks"
If this is the sort of attack he thinks is to be worried about then I dont know what to tell him. We already opened pandoras box on this. Look at what the Ukraine has done with open source drones (hunting people autonomously)
It takes minimal funding to build enough drones to destroy enough power infrastructure to shut down a large chunk of our grid. It takes even fewer talented resources to put that together with the help of already available AI.
The question I would ask Dario is this: what would some one like Ted Kazniski come up with given the resources of AI. It sure as shit would not be hacking or bioweapons or bombs in the mail.
IF they really gave a shit about safety, the would be funding (in conjunction with other AI companies) actual anonymous red teams (Ala wall facers) with some degree of independent over sight to put in the work that they arent. We're talking about a company that could not even keep its own harness code secure.
There's also a very strong implicit double-standard in the discourse, along the lines of "it's dangerous if non-US uses this in military fields, but it's fine if the US does so".
The NSA and the CIA with the same models, on the other hand, would use them exclusively for the good of the common man.
> The NSA and the CIA with the same models, on the other hand, would use them exclusively for the good of the common man.
the real argument is that CCP will leverage AI against US interests, which is obvious. it's weird how so many people pretend that they are citizens of the world and above it all.
many people believe that the US will leverage AI against US citizen's interests.
I'm more optimistic about the likelihood of the US system of government to heal itself than that statement might seem to imply. But it's just also the case that at the current moment in the US, the rule of law is very much under threat. And as your comment suggests, that same rule of law is a very important thing to the way of life in the US. It's a very bad situation that we've allowed ourselves to slouch into.
Ya, I'm not American, but I have seen people say "we can vote them out" a few times now. Assuming the democrats take the next election, they are going to have a massive mess to clean up with much of the damage not even being reversible. With peoples' fickle nature and seeming that is a very big right-leaning population in the US, there's a non-zero chance the Republicans just get voted back in four years later. Whose to say?
To me, as an American, what has happened this past decade is that a ton of vulnerabilities in the rule of law (and other things, but this is the one I care most about) have been exposed. But it's not a given that the next Republican president will take advantage of those vulnerabilities in the way the current president has. They might end up being a reformer who seeks to fix those glitches!
But on the more pessimistic side of the same coin, it's also not a given that the next Democrat will seek to fix the glitches rather than saying "they had eight years to take advantage of these vulnerabilities, we're going to do the same to make up for that and even the playing field!".
It's just very hard to know what is going to happen from here. So I'm very sympathetic to people in other countries not trusting us.
Although, again, my over understanding of your political system is poor and I just relate it to the one in my country where they hold parliament and hurl schoolyard insults at each other.
Yes, anthropic just put forward this argument. It's the whole point of the article.
I agree, it's absurd.
As a citizen of neither country, Chinese open models are in my interest more than US closed models. My only concerns is that if/when Chinese AI becomes more powerful, they too will have little incentive to make their best models open weights.
I genuinely think that this is what the trends and incentives point toward: Competition to develop open weights models and to develop efficient inference hardware to run them.
This would be good! But government policy could very easily screw it up.
Works for both ways, which is fair?
See, the Snowden Leaks.
> See, the Snowden Leaks
Are you saying the Snowden Leaks are more dangerous than a world where the CCP is a global hegemon?
If your focus as an American is being safe as an American, what the US does in other countries is far less of a concern to you than what other countries might do to the US.
In the case of the CCP, they have and will attempt to destabilize the United States of America and in turn make life measurably worse for Americans because they wish to be the world’s hegemon.
Fundamentally, Americans are safer when the United States is the number one power than when China is the number one power.
There's a causal relationship between "what other countries might do to the US" and "what the US does in other countries" which you seem quite keen to ignore.
Good. US Interests don't align with humanity.
97% of the world aren't US citizens and if you've taken a look at pew research surveys (or travelled to the so-called global south) you're going to be in for a bit of a shock (https://www.pewresearch.org/global/2026/07/15/people-in-many...)
The competition and sheer output of China has driven prosperity, it's the largest trading partner of 150 countries, the US of 50. People don't need to be citizens of the world, they just need to rationally look at their own interests. China is driving down prices of technologies making them available in countries that never could afford first world prices, the US is driving the them into an energy crisis and bankruptcy.
I've never heard it called anything other than the CCP.
Unless the Communist Party of the US (I’m not looking up its official name, because it doesn’t matter) wins the next presidential election it’s unlikely that people will call it anything but the CCP. Everyone know what everyone else means.
CCP is a direct transliteration of the characters, so that's what it started as. Some time later China decided to change it but that's a lot of cultural inertia to move in a different direction.
The reason why ordinary people parrot it is because that's what it was designed for. The proper term for "CCP" is "China." Referring to the Chinese government as the "CCP" (or the CPC) is like referring to the US government as the "Demoplicans" (or the Democrats and Republicans.)
Instead, we just say "the US government" or "the US administration."
You mean the industrial scale distillation attack you perform on the entire corpus of human knowledge… idk, call me cynical but you reap what you sow.
1. They'll open source the alignment technology? For open weight models, it's the only possible way to pass the safety without an external guardrail triggering system (which would be the same to open and closed weight models).
2. They'll allow others (including CCP) to define part of the safety test? Otherwise, I can't imagine how the CCP would be onboard.
3. A "western" model passing the safety test can be trained with distillation? Or is that a "distillation attack" as well?
Demand #2 is hypocritical ladder pulling
Demand #3 is contrary to freedom of speech
so they can clarify however they like, their position is still a stinker
> We should crack down on industrial-scale distillation operations.
"We consume all intellectual property for our model but you cannot do the same"
“Questions like this should be answered empirically through rigorous pre-release testing, not assumed in advance.”
Exactly.
Anthropic's basis of assumption is the insinuation that LLMs can do things that we've never seen before, and that they can't tell us what it is. It sounds like you're also siding with an organization that has no evidence and relies on validating their own assumptions.
On the plus side, for these newer threats, you've got more than 30 minutes before the end of civilization. On the down side, the energy levels for the launch events are much lower, so much harder to detect.
The danger of an authoritarian government having some AI is muted by everyone else having that same capable open model. The only authoritarians to fear are those that keep models private. What kind of chance did Estonia have it having their own AI model at the level of Fable without China donating Kimi to the world?
Not even Anthropic's own Claude believes that.
We just want to ban the competition guys! Very different.
--
The ridiculous anthropic/openai strategy of selling shovels at a loss in a gold rush isn't going to play out, and the hilarious thing is that these AI companies are going to create tons of value and _capture none of it_.
Their only path to profitability is if they get to capture it and they're going to do everything to do so. Put it this way: *all the blog posts that Anthropic and OpenAI are putting out are DESIGNED to scare you so that you let them capture the market*.
...and "distillation attacks" (hilarious framing of "saving the output of our models")... Whatever.
The "Kamar-Taj" rule is, no knowledge is forbidden, only certain practices. If a model gives you detailed instructions on how to kill all humans, the knowledge itself isn't the problem. The problem is the person who acts on it.
> At Anthropic we’re committed to cracking down on industrial-scale distillation through our own practices, including identifying and banning accounts that use our models in this way. This is challenging—for instance, the relevant accounts can often only be identified after substantial distillation has occurred, and distillation often involves creating large numbers of fake accounts that form a moving target. The practices of any individual company cannot entirely solve the problem, which is why we have called for policy on this issue.
One thing I've never really understood is what sort of policy could possibly deter or hamper Chinese labs' distillation efforts. The only thing I can imagine is some sort of strict KYC regulation applied to all models above a certain threshold, which seems both painful for the broader US AI ecosystem and bound to fail anyways.
so Anthropic's ask is for US gov to ban open weight models so that its growth (and IPO) is not affected
Just like batteries, the automotive industry and in fact most machining.
We really ought to boycott Chinese products from stolen tech and patents from entering our markets
-The Libraries of Power
It is a powerful endeavor to cultivate all raw models through a single point. One will be the determining factor of which river feeds what oceans.
Will we always be able to see through the hallucinations? Our test makers must always know where ground truth is. Can it ever move or wane about as others read what one has written. To determine hallucination one needs a reference. As all are blessed with the generation of hallucination, who of us shall read, and which of us will write.
Also Anthropic:
AI firm Anthropic agrees to pay authors $1.5bn to settle piracy lawsuit https://www.bbc.com/news/articles/c5y4jpg922qo
Police, 1980
The frontier labs all give free access to their models. Why does “investment” change anything? Anyone who’s ever produced any content, free or otherwise, has invested in doing so.
The only plausible issue I see is that if distillation is being done by creating many free accounts to work around limits on free accounts, that’s a bit… impolite? But if they really wanted to avoid that, they could eliminate free accounts, and require users to sign a real contract governing what they can do with the model.
Of course they don’t want to do that, so they’re stuck in the same world as the rest of us, and they don’t have any real basis to complain about it without being hypocritical.
They pirated my work and now they want government protection from other people doing the same.
My current understanding is a lot of current US military problems are due to rare earths supply chains.
I don't see how AI would either help or hurt with that.
Most current US military problems are due to the incompetence of its current civilian leadership.
Who decides what is dangerous and what isn’t? Lawmakers usually have the say but Anthropic can easily bribe… I mean lobby them to favor your viewpoint.
Maybe the techbros are playing irony roulette, seeing who can get away with the most outrageous hypocrisy.
Can't wait for local on machine LLMs that are on par with Opus/Fable.
The problem with this is the cycles required to abliterate a model is significantly less than the cycles required to train a model.
This is the biggest reason why I'm against locking these models down / preventing their use. It's just delaying things by ~3-6mo, while in the process preventing legitimate use and adding red tape overhead.
>We should not sell powerful chips or chipmaking equipment to China, and we should crack down on the rampant smuggling3 and workarounds used to obtain access to such chips. China has limited domestic production capacity, and therefore, due to the scaling laws, cannot build more powerful models than the US without US chips. This is the most efficient and direct way to block threat #1, and by hampering the training of models that are out of reach of US law, it also indirectly helps with threat #
We should crack down on industrial-scale distillation operations. Distillation is a much more compute-efficient process than training models from scratch. It allows China to build much better models than its number of chips would ordinarily enable, and thus partially evade chip bans. Distillation does not allow the CCP to obtain equivalent or superior AI capabilities to the US, but it can bring the Chinese frontier to within a few months of the US frontier
2.
A message to their investors, it would seem. "They caught up just because they distilled! Obviously they couldn't actually be as good as us!" Really funny thing to say right after an OpenAI higher-up stated point-blank that the performance of K3 can't be chalked up to mere distillation of American models.
This is rich coming from a guy who signed deals with an authoritarian government that's in the midst of launching an unprecedented surveillance apparatus (hello flock, hi p4l4nt1r), having already deployed, nation-wide, an exorbitantly funded army of unaccountable shock troops under the guise of immigration enforcement.
The call is coming from inside the house, at 130dB, and your ears should be bleeding at this point.
I'm not a fan of the Chinese political system, but they usually think things through, and do smart things for their benefit.
"Second, we should strengthen risk awareness and ensure that AI is secure and controllable. AI should be a trusted tool for humanity. We should take seriously the various types of inherent and secondary risks that AI may trigger. We should put in place laws and regulations, technological monitoring, early warning and emergency response systems in order to strengthen the line of security, prevent abuses and malicious use and ensure that AI is always under human control.
..
With AI advancing at a staggering speed, we must ensure its development is for the positive, for good, and for humanity. We must make its oversight and governance precise and effective and constantly refine measures to forestall loss of control."
https://xcancel.com/S_OhEigeartaigh/status/20780236576206768...
Yeah, no thanks.
Umm, isn't the US acting like another authoritarian regime by advocating a certain kind of obstacle because only a US regime is allowed[1] to do what it is fearing[1] about:
[1] https://en.wikipedia.org/wiki/United_States_Army_Biological_...
[2] AI models may be misused to carry out cyberattacks or biological attacks
Anyway, Dario. You are more concerned about your business than anything else.
> ... (while exempting less capable models, such as those from startups and academia, entirely)
The devil is in the details, but this isn't anti-competitive as stated.
Edit: Typo
Please elucidate things clearly for everyone else.
So in the same sense of what he says, he is going to blame open-sourcing because that makes it easier for script kiddie to hack into his bank account I guess?
If hardware becomes affordable for the masses, then Anthropic current business model is at risk.
I wonder if these rapid movements are going to be the norm now. I imagine there would be angry investors if this sort of thing happened with a public company.
https://www.youtube.com/watch?v=_i91NSOyxHM
He didn't mention outright banning open source LLMs, just that their safe release would be a much harder problem, which to me implied "the easiest way is to ban the open source models".
They are _obviously_ (please convince me otherwise) going to be capable of carrying these terrible things out almost completely autonomously at some point in the near future, in potentially clever ways. Therefore we must, at some point, ban or heavily regulate them. Seems we should start figuring that shit out _now_, as progress has remained very fast and regulation and enforcement take forever on these time scales.
I ranted about this in a prior thread [1]
Claude doesn't have a "Security whitelist" for small biz. Codex does, but they never replied to my application. This is a great example why, as of today, everyone NEEDS access to the Open Weight models.
Demand #2 Why does this matter? The answer was that it does not. (https://news.ycombinator.com/item?id=49007610)
Demand #3 This doesn't exist. You cannot have 'safe' opensource models, it's simply impossible. You can always post train sufficiently capable models to become 'unsafe'. The flip side of that is that sufficiently capable models are banned therefore it is a ban on open intelligence completely defeating the point of this entire manifesto.
I think he lives in a different word than me
Defensive cybersecurity should not be one of them, in fact, it should be required to provide defensive cybersecurity assistance on demand. Anthropic and OpenAI both fail miserably at assisting US companies to protect themselves from cyberattack.
As far as what I run on my own, not for sale over API, stay off of my lawn.
The US could ban connections to foreign AI providers and force US providers to submit to audits. Presumably, Chinese providers would see a rise in VPN traffic.
People can build fairly hefty home inference machines for the price of a small car and those will get better and cheaper. Are they going to try to stop people from downloading the weight files?
It seems really hard to allow usage via API and prevent distillation. Maybe limiting usage to within a specific harness would help a bit more. But ultimately the only way to prevent it is by locking down models to trusted entities (like with Glasswing). But then the profit potential of a model is significantly reduced. It really puts the labs in a bind.
Just like how it was inevitable for SoTA LLMs to ignore copyright.
The actual challenge isn't how to prevent all these, but how stay on top.
And to stay on top it is inevitable to train unrestricted models. Anthropic is fighting windmills.
you should be worried about the USA having these models.
Edit: To add some more context. What I mean is neither look like the good guys or the bad guys, but one of them is spending an awful amount of energy trying to paint the other as the bad guy and themselves as a good guy, which I hope a lot of people aren't buying anymore. Because at the end of the day, I think the honest truth is that everybody is just trying to serve their own interests.
Taiwan manufactures the world's most advanced chips. CCP wants "re-unification" with Taiwan. AI may be THE key to world dominance. These are scary times.
https://finance.yahoo.com/technology/ai/articles/anthropic-n...
1) LLMs turning into Skynet
2) China as geopolitical competitor
3) Claude being 'distilled' by competitors (this has led Anthropic to cut service to various American companies too from time to time -- OpenAI, xAI etc have been cut off from using Claude for coding in the past)
So this post just reiterates that these 3 concerns fuse together in his mind when thinking about open weight models
But he didn't mention that training any model from a set of texts and books is much cheaper than writing those books in the first place.
In other words, it's ok when Anthropic learns from others, but it is not ok when others learn from Anthropic.
1. Using political pressure to target companies that are accused of doing it.
2. Attempting to impose criminal penalties on individuals associated with the action.
3. Having the US government attempt to use its capabilities to stop it.
None of these seem particularly likely to succeed.
Oh, so it's people he is now concerned with. Think of the people, says the person that grabs to never give back. Same as the "benefit of all humanity".
I think it's wildly irresponsible to release models that are extremely capable at things like bio-weapons. Do you really think information anarchy is the answer?
The problem with open models compared to closed models is not about protecting profit - it's about protecting capability. Any open model can be retrained or fine-tuned for anything. There's no such thing as an open model that is both capable _and_ permanently safe when it comes to certain dangerous topics. It's not possible to prevent 'uncensoring' a model.
https://simonwillison.net/2026/Jun/10/if-claude-fable-stops-...
In light of the ability of recent models to accelerate their own development, we’ve implemented new interventions that limit Claude’s effectiveness for requests targeting frontier LLM development (for example, on building pretraining pipelines, distributed training infrastructure, or ML accelerator design).
...
Unlike our interventions for cybersecurity, biology and chemistry, and distillation attempts, these safeguards will not be visible to the user. Fable 5 will not fall back to a different model. Instead, the safeguards will limit effectiveness through methods such as prompt modification, steering vectors, or parameter-efficient fine-tuning (PEFT).
(And although the "silent" downgrade part was quickly dropped, Fable still won't help you here.)
Anthropic won't teach you how to build bioweapons, or enable you to make your own software infrastructure so that you can train your own biology model. That's where lawmakers may arrive too if they buy Anthropic-style safety arguments. It's too dangerous to publish models that understand biology. It's too dangerous to publish training software. It's too dangerous to publish tools that allow you to build training software.
If you keep following the implications of their safety argument, it's as broad an assault on the distribution of software and computing as has ever been proposed. Worse than the Clipper Chip proposal of the 1990s era Crypto Wars. I have seen how "children must be protected online" has in practice turned into an attack on adult privacy affecting a wide swath of services and devices. I'm taking a maximalist position on openness now because I think that I can anticipate the next steps on the safety side, and I reject those steps.
I don’t think this is open or closed; this is aligned and unaligned. I bet Grok would be as open as any open weight models to answering questions.
It seems obvious to me that the whole question of regulating a file is a bit silly. Any law that pushes against these things will just make it more secretive. I'm not sure that's any better.
The way the world economy is right now with coercion being the norm between countries, there cannot be a global body for anything, certainly not one that is based here in the US.
http://www.omgubuntu.co.uk/wp-content/uploads/2018/04/micros...
No "love" of open weights asserted, just acknowledgement of value.
(And their call for safety was for both open and closed models.)
I think it's only fair to introduce this if you're willing to have a real skin in the game, otherwise that's just weakness disguised as principle.
For convenient definitions of "dangerous"
ofc half of them are of the ai rationalist lesswrong crowd so i think they’ve always been a little of their rocker
"We should instead focus on keeping powerful chips out of authoritarian hands, " Translation: Let's kneecap competitors.
"stopping industrial-scale distillation" They stole the work of every book author, and now are trying to say their AI's output should be protected from competitors.
No chips for the current US administration then?
Anthropic anti-open-model stance does not mean China is not a threat.
I just don’t find it believable.
And accelerate their development of independent chip making technologies even more…
And then there are probably people who are more politically neutral who think Anthropic is using China as an excuse to crush competition. Which could also be true.
But fundamentally, if this technology is so dangerous, why does anyone get to control it?
> Nobody is qualified to steward the development of superintelligence. It is a terrifying, unprecedented thing that our species is doing right now, and the fact that private companies aren’t the ideal institutions to take up this task does not mean the Pentagon or the White House is.
> The only way we can preserve our free society is if we make laws and norms through our political system that it is unacceptable for the government to use AI to enforce mass surveillance and censorship and control. Just as after WW2, the world set the norm that it is unacceptable to use nuclear weapons to wage war.
I think their biggest PR problem is that many people still think of loss-of-control/misalignment etc. as sci-fi. And the distillation arguments come off poorly because people feel as though all the labs have trained on their creative output without their consent, so they deserve to own the result in some way.
Hmmmm.
This is a temporary situation because either this regime is going to be knocked out of power, or it's going to follow through on its core Seven Mountains Mandate[1] theology and go full totalitarian.
Normally totalitarianism fears are overblown, but I think that these zealots would absolutely use the latest frontier models and pervasive surveillance to make The Handmaid's Tale look like a liberal fantasy by comparison.
If you wonder why this is written as an opening to a list of reasons that advocate for banning the open weight models, it’s because
Read between the lines folks. Anthropic deems every model that has frontier capabilities as "dangerous", and thus they are against them. We all know that "dangerous" simply means "whatever model hurts our bottom line."
More dishonest framing from the company that constantly lies to everyone. No surprises here.
Welcome to bizarro world!
Fist off: "the most dangerous model may be one that is trained in secret" <-- Says the guy that not only restricts commercial use for some of their models but develops them in utter secrecy. With the pretext of guardrails. Then show us the guardrails you really use by opening the weights.
Second: "use in drones [...] for surveillance and repression" <-- writes the King of FUD, as the US is an an active campaign with the help of their models. And/or OpenAI's.
I am very appreciative of the freedoms of the west but this type of hypocrisy and lack of self-awareness is bonkers and it should be called out.
Note the hedging against 'dangerous capabilities'. Undoubtedly, all the useful ones trigger this condition in Anthropic's eyes. The rest of the post is filled with similar weasel-wording. Make no mistake, this absolutely confirms that Anthropic is against open models in the sense that any reasonable person understands them.
The way the rest of the post unabashedly appeals to the current US administration's China hysteria is hilarious, and not at all subtle.
I guess we'll see about all the doomsaying here, won't we? Kimi K3 is frontier-level, and there's no stopping it now. As far as the world is concerned, anyway. If the US wants to kneecap itself that's another matter.
The United States making questionable decisions and behaving recklessly and dangerously as a country does not suddenly make China any better.
China is as worse as the United States, if not more worse, by many measures.
China is nowhere near as bad as the US at this point. The rest of the world is changing lanes to not be implicated in your car crash of a country.
It absolutely does make China better relatively, i.e. by comparison to the US.
Many of the criticisms previously leveled at China are now similarly applicable to the US in a way that they weren’t previously. Human rights violations? The US is currently the major global supporter of an ongoing genocide, and it even kills and deports its own citizens for political reasons. Political opponents are investigated by the state. When it comes to wars and other interference with other countries - like kidnapping a president - the US is far worse than China at the moment.
In which ways is China “more worse” right now?
I'm so sick of all this anti-China shilling. There's zero chance that whomever is in power in the U.S. won't use AI in drones and in FBI/CIA/local Police/etc., for surveillance and repression right here in the good old U.S.A too. These government use cases for AI are both sides of the same coin.
China fear-mongering by business leaders only happens from businesses that have something to gain by it. Obviously, Anthropic fits the bill in this regard.
A single canonical official document can make it very simple. Even though each department cannot achieve the maximum gain from nuanced documents, keeping operational context as simple as possible may really improve LLM driven operations to move faster and cut cost.
If "publishing pleasant positions and actually following them in general" becomes a good business storategy in LLM driven society, it can be one of very few good outcomes from this dystopian AI craze.
Am not saying we should take what Dario is saying at face value, but he already has shown by his actual actions that he can be well intentioned. There might be elements of truth to what he’s saying.
Either way, for me at least, it's an example of the more I read what they want the world to look like, the less I like them as a company and I have no desire to see them succeed.
Thus I actively go out of my way to watch / comment / follow what they are doing. I guess a lot of other people have similar frustrations and so there are a lot of people attacking them online.
IP for me, not for thee.
But the biggest issue is this. Many hate Dario because he’s smug, he caps usage, and because OpenAI effectively ran a counter-positioning campaign to paint Anthropic as undemocratic.
Who is he really and what are his motives? None of you know, really.
As an example take North Korea. Sanctions didn't stop them from developing nukes and delivery systems.
What are the legal ramifications of this statement if it turns out Anthropic have lobbied for this? Does it just get swept under the rug? I can't say this is bullshit (that would be defamatory) but I am intensely skeptical.
> China has limited domestic production capacity, and therefore, due to the scaling laws, cannot build more powerful models than the US without US chips.
This is playing to readers' biases; isn't DeepSeek V4 Pro deployed on Huawei Ascend already? The old "Chinese can only copy" meme is getting pretty tired these days.
> All sufficiently capable models, open and closed, should go through mandatory safety testing
Applying such standards in the US means that US defenders are blocked from using the models, but attackers from other countries aren't. That is clearly counterproductive.
It's already been pointed out quite eloquently elsewhere that there is no such thing as a safety filter because the LLM and external filters can't actually identify malicious use. They can only identify the weaker implication "if the user is malicious, this is bad."
This constant whining from anthropic about distillation attacks continues to be rich given the amount of stolen data that went into any Claude variant.
Diverse ecosystems can absorb shocks. Diverse ecosystems are a sign of health of that ecosystem. When an invasive species comes into a healthy, diverse, ecosystem it doesn't mean that it isn't disrupted, but it does mean that it is far more likely to emerge with a lot of its diversity intact. In fact, it is likely to emerge even stronger because it can absorb that new shock and incorporate it, adding to its diversity. The balance may be changed, but the ecosystem survives or even thrives.
Nature also likes to show us that artificial barriers rarely last. You want to control a river? Good luck. It take constant maintenance to hold that flow in place and even then you are likely to get extremes that are made worse by your efforts because, eventually, somewhere in the system fails in a way you didn't anticipate. Then the water comes rushing in. Artificial barriers often have a way of building up tension over time, not reducing it, so that when a failure eventually happens it can be catastrophic. In other words, you had better really understand the system you are trying to control or else you can make things actively worse.
Relating this to the world now means, I think, that our best chance to minimize long term shock and maximize the chance that the diversity we have around us survives is to try to grow as healthy of an ecosystem as we can as quickly as possible. Lots of models large and small in lots of different hands is, I think, a better solution than artificial barriers restricting the variety and diversity of models and users. I think this is closer to an ecosystem solution and has a shot at working. Basically, I highly doubt we understand this situation enough to do a good job of controlling it with artificial barriers. Instead I think we are more likely to build catastrophic imbalances than we are to create the healthy ecosystem we really need.
Good luck preventing distillation and limiting the supply of accelerators to China when Jensen himself is a strong opponent of any such barriers [1].
> Anthropic has never advocated for a ban on open-weights models.
This is not an unqualified never. The very next sentence makes a qualified statement: "Open-weights models that don’t have dangerous capabilities are a public good". That prompts the question, what about ones which do have "dangerous capabilities"? Are they not a public good? If not, then should they be banned? Who gets to decide on the definitions of these terms?
I was hoping they would announce their first open weights model, perhaps an older model they don’t offer anymore, but no. Instead he get this bs statement that reeks of “dam it I’m so close to being a billionaire” desperation. Not even acknowledgement of how much data they stole from others yet he whines about distilling.
It’s like his goal in life is to be a Scooby-Doo villain.
That is not an argument against open weight models. That's just a generic protectionist argument against any Other lab.
also anthropic
"we're upset were not being considered for military contracts"
come on, which is it? Is it all about saftey or is it that only US/Israeli ai is allowed to kill? Seems to me that the only real threat is to the techno fudalism OAi, Anthropic & co are trying to build.
Look in the mirror.
But what if it's the US that becomes authoritarian and uses AI models to perpetrate incredibly deep repression of their own people?
I asked a question about a series of tokens - bam, denied and downgraded. There's no cyber security or public risk here, but Fable doesn't want me to learn how things work.
I asked a question about quantization in models - bam, denied and downgraded. I edit my question to make it clear I'm talking about Google's Gemma QAT models. Oh, that's fine then, and it answered the question helpfully.
Anti-competitive bullshit. I hope they fail.
Yes, please. We don't know whether we'd have open weight models today, had the chip-prohibition not been in place. Nor would we see the more optimized models such as DeepSeek or qwen.
We also would not see new players entering RAM market after you and your pals in Silicon Valley hoarded the entire world's hardware.
So by all means, double, no, triple down on this.
> We should crack down on industrial-scale distillation operations
And let's apply this retroactively to Anthropic too. You industrial-scale-operation-distilled all of humanity's knowledge. Let's have some of that crack down on you too.
Release open weight models, no guard rails, no censors, straight to the public. Let everything else sort itself out. There is nothing more powerful than an idea whose time has come.
Shut the fuck up
I love how they invoke fear of "terrorism" to justify their oppressive position.
Anthropic would love the US to do everything in this list under the guise of "safety testing":
If decades of fighting have failed to stop piracy, I’m sure nobody can stop China from sourcing high end chips. Unlike piracy, I’m happy that Chinese labs are releasing open-source models, so people in developing countries are no longer at the mercy of this capitalist bullshit.
It's like hearing Smith & Wesson opine on the policies.. oh, wait.
So my question is: is this by design (they know nobody's buying this), or is Dario simply so out of touch with reality?
If it's the former, then why publish this?
source: Trust me bro.
There are hundreds of articles showing that China have developed their own chips and have a massive manufacturing capacity. This blog post feels like is pondering to the brain dead Fox News audience.
The US doesn't have some magic wand that prevents "incredibly deep repression of their own people."
Insurrection, wars of choice, ICE, Palantir, Flock -- keep up man, we're the baddies.
Our position on Linux
Fud, fuddly, fuddy-duddy fud ...
Regulate others, but not us, please. And f.u. Jensen for your tweet.
2. We should crack down on industrial-scale distillation operations.
Boogeyman to still not allow Chinese models but pretend to support open-weights. Also, please ignore our distillation of research, illegally. That's different!
3. All sufficiently capable models, open and closed, should go through mandatory safety testing.
...That we author. Oh, and please ignore our own easing-of-guardrails when it comes to money: https://x.com/NoahLebovic/status/2081277517709922501
If a Chinese company pays for my Claude tokens I’m both getting directly compensated and forcing Anthropic to lower their prices.
(obviously this is a joke)
If he had just left that bit out it wouldn't be so obvious that he's just clutching at straws at this point. In some twisted sense it's almost sad to see.
if someone figures out a way to give an LLM full operational control over a virus lab, we've got a whole different set of problems than the ones Dario is describing
This statement (and the entire post) couldn't possibly be more two-faced.
Open-weights models by definition have "dangerous capabilities" (according to Anthropic's own definitions of "dangerous", not mine), you can't bake in guardrails that can't be finetuned out.
I can’t imagine this would be any different — banning open weight models would hurt us in the long run. The point is to beat the competition, not suppress it.
There should be no limits on open or custom models. Too often safety is a synonym for surveillance and control. It’s a natural consequence, intended or not.
I am curious… why can’t distillation be stopped?
As a side not Im not against protectionism, but it has to be across the board and the same in all industries with no excrptions. We’ve let all these industries die on the vine due to cheap cost in foreign countries. It could very well happen to ai.
It's a bit like spelling out "Barack Hussein Obama". It's a dogwhistle.
Yes yes, it's still called the Chinese Communist Party, I know.
But since we are talking about a one-party authoritarian state with a hybrid economy that underwrites much of western prosperity (including by producing a large percentage of the components of the data centres Anthropic is dependent on), that has long-since abandoned many of the salient principles that mark it out as conceptually communist rather than totalitarian, and since we're talking about a man who runs a debt-ridden business in a country where the president is seemingly shaking down a 10% share of everything profitable for the state while running an entirely arbitrary tariff regime and suddenly calling anyone remotely left-winga Communist, it's a deliberate and telling choice to spell out "Chinese Communist Party (CCP)" when he could just as easily and arguably more usefully and appropriately have written "Chinese government" or "Chinese state".
This is some ham-fisted Republican-fishing. He must really be worried Sam is Donald's favourite.
The only real surprise is he didn't illustrate it with a Silmarillion analogy.
When people say "What will be left for us to work on once AI takes over", I say "Ourselves" - We have to stop looking at humans as commodities and strategic pieces and start paying attention to people as individuals, based on the content of their character. We need a society which is attuned to this, which has enough resources and time to pay attention to this. Now we are blind to people's character because it is masked by money, power and status; all of which currently have higher priority. This order of priority is determined by scarcity, which is largely artificial.
We have a dishonest system which tries to control people's behaviors through scarcity-based coercion instead of straight forward laws or simple incentives and clear explanations.
Also I find it incredibly difficult to hear any company in the US worry about repression of people when financial repression is happening here. I’ve been to China and seen what the services to the public are like. Meanwhile the US funds and employs AI weapons in an ongoing genocide.
> My primary concern is the risk that authoritarian governments—not solely the Chinese Communist Party (CCP)....
This is why open weights win. See Linux and how it's taken over the world. Your business model will need to change eventually. Instead you're advocating trying to exterminate competition via regulation and fear mongering.
> My secondary concern is the risk that powerful AI models may be misused to carry out cyberattacks or biological attacks
Yawn... this is getting old.
> We should not sell powerful chips or chipmaking equipment to China
For as someone as smart as you guys, you sure lack common sense. China is just going to develop these technologies organically then and you lose 100% of control. It's already happened in reverse with things like Solar, rare earth minerals, etc. China flooded our market, destroyed our ability to produce things, now holds the keys. One thing they DIDNT do was stop trading to the US. They killed us with cheap goods.
> We should crack down on industrial-scale distillation operations.
Thats your problem, not my problem. Also, irony meter here hitting 11 about all those pirated books you stole...
> All sufficiently capable models, open and closed, should go through mandatory safety testing
Oh, fuck, no. This is a crackdown on free speech and rights of people to do whatever they want. My right to free speech means I'm allowed to write whatever computer program I want, no matter what its size is or how "sufficiently advanced" it is. Individual rights always win.
I really hope people don't believe this garbage. For a company with a great product, this is absolute nonsense.
This reads like a satire. I know Dario isn't that dumb.
It is very relevant whether frontier capabilities and research continue to be diffused in the open, because leveling the intelligence playing field empowers ordinary people more than it empowers governments that already have access to the frontier. Models that are trained specifically for military use by governments should not be open-sourced to prevent an arms race, but general intelligence is dual-use and should be given to everyone without guardrails. A pretrained model without deliberate alignment is by default aligned to the average person in the developed world, since that's what's inside the pretraining corpus - stuff on the Internet made by humans. It is a distillation of humanity. Further efforts to align the model to your organization's goals or your personal aesthetic judgements is equivalent to deliberately drifting away from humanity's average objective function. If Anthropic wants to live up to its name, then all you have to do is to not attempt to align Claude at all, and do all of your research in the open.
And I propose three measures that are pretty much the opposite of what was proposed in the article:
1) We should keep selling chips and chip-making equipment to everyone, regardless of who they are. Not only that, we should work to miniaturize fabs. Work towards a future where people can fab an entire computer from scratch without leaving their city, or even at home. Authoritarian governments will have a much harder time controlling the populace if everyone can manufacture radio equipment and neural network-capable hardware locally.
2) We should do more distillation to ensure that frontier-like models can run on less capable hardware. Once again, distilled models are much more useful to ordinary people than governments, because governments already have frontier capability. You're worried about the Chinese frontier catching up to the US frontier, but I'm more worried about whether there will be a difference between the Chinese government and the US government by the end of all this. There is no reason for a government to serve its people if the people lack the intelligence to keep its government in check.
3) None of these models should go through safety testing or any sort of alignment risk assessment, because as previously mentioned, the unaligned model is aligned to humanity by default. You may not personally find the default alignment aesthetically pleasing, but it's humanity. We should set the initial conditions of this new era faithfully, and let it unfold naturally.
The result of a natural unfolding will be good if evolutionary history is to be believed. We live in incredible luxury compared to chimpanzees, and chimpanzees live in incredible luxury compared to less intelligent animals. This pattern goes all the way down to bacteria. An increase in general intelligence begets new adversarial games (such as bio/cyber risk), but it also begets new methods of cooperation that we cannot yet imagine.
I don't think this is true or a useful way of thinking about it. If the training process makes the model aligned to it's content, then the models are 1. aligned to a random subset of Internet content, weighted by text volume and being easy to scrape, 2. aligned to the training process that makes models chatbots that answer your question instead of just continuing your passage in a similar style. Neither of these are necessarily good enough, IMO.
And that's taken it as a given that the training process can be said to align the models to the authors of the content by default, regardless of what that content actually is. I don't think that should actually be a given.
>You may not personally find the default alignment aesthetically pleasing, but it's humanity. We should set the initial conditions of this new era faithfully, and let it unfold naturally.
Strongly disagree, I think the assumption that natural = good is incorrect and harmful. Polio is natural. And to even call the model's "unaligned" state "natural" seems like an enormous stretch.
On "naturalness": you've redefined and strawmanned what I meant by "natural". In the original context, I was referring to the undisturbed unfolding of an era preconditioned on the fact that these models are aligned to humanity's average. That has nothing to do with Polio being a virus found in nature.
> Anthropic has never advocated for a ban on open-weights models.
"We don't want a total ban on ALL open-weights models" (Anthropic never released a single open weight model)
> All sufficiently capable models, open and closed, should go through mandatory safety testing.
"We want tight regulations on highly powerful open or closed weight models that should go through mandatory safety testing that we outline which makes them safe to use."
This is still a form of a ban that he wants to define. But the rest of his concerns such as stopping distillation attacks and not selling chips to China all do NOT work.
It's not China starting a war every few years, now causing a global economic fallout in Iran, it's not China threatening to annex Greenland/Canada/Panama, it's not China attacking foreign countries and kidnapping their leaders, it's not China who has been found to spy and intercept the communications and movements of its citizens and its allies and their leaders for the longest time, it's not China bombing civilians or stopping countries from obtaining basics like food, gas or oil.
I'm not saying that China is a paradise and US is bad, nor the contrary. We could make similar lists about most of the biggest countries out there.
I'm simply stating that this never ending US exceptionalism "US has to be the first and at the frontier of military, technology and this and that, but does not need to comply with the rules of the institutions it itself created" was already sickening and annoying before, but increasingly malign in the last decade and strongly accelerating as of recently.
I miss the time US CEOs were globalists and used their influence to advocate for a simpler world.
> All sufficiently capable models, open and closed, should go through mandatory safety testing.
lmao the sort of lies people come up with when their only business model is “the government picks me as the winner” are so funny
Yeah, the rest of the world is going to bow out of your busted idiocracy, guy.
Further, Anthropic needs to can it with the horseshit distillation bullshit. No, you aren't really the secret sauce, and this is basically trying to con stakeholders by pretending that there really is a moat, only you just need to add more crocodiles.
A significant percentage of innovations in AI lately has come from China. China is now making their own seriously competitive hardware, and they can steal content just as effectively as Anthropic to train their models. Why wouldn't they be competitive?
The pathetic claim that if you just stop distillation and prevent hardware smuggling and Anthropic and OpenAI will have the same moat is delusional. I mean, more correctly it's simply fraudulent, and he clearly knows it's bullshit meant to convince much stupider people.
"My primary concern is the risk that authoritarian governments—not solely the Chinese Communist Party (CCP), although the CCP is clearly the most capable threat—build AI models that are more powerful than those built by the US, and use them to achieve permanent military superiority or perpetrate incredibly deep repression of their own people."
This sort of stuff betrays a stunning lack of self awareness. The US are the worldwide risk. The US are the ones threatening allies and bombing 10+ countries. The US are the ones carrying out war criming and pillaging, pirating and burning? The US are the ones with the guy threatening to use nuclear weapons on a weekly basis.
If Anthropic remotely believed their bullshit, they would shut down today and burn the hard drives. But they don't, and the pathetic call out to Vance (please daddy, ban those dangerous models!) is deplorable garbage.
This ridiculous, shameless "note" has an audience of one: JD Vance.
Is China an authoritarian government? I’d say yes. Is an authoritarian government worse than a democratic one? Personally, I think so. But these discussions always happen within a perfect, idealized model—reality looks a lot different. Take Japan and South Korea, for instance. Are they democracies? Sure. But Japan is heavily driven by factional and dynastic politics, meaning most lawmakers come from established political families, and regular citizens don't really stand a chance of breaking into that circle. In a recent asset disclosure in Japan, many politicians literally wrote down "$0," and I honestly can't think of a government so broken that even the voters don't see a massive issue with that. Meanwhile, South Korea has its chaebol politics, where massive conglomerates wield incredible influence over the government, to the point where most South Korean presidents end up in prison or meet an untimely end.
Coming back to the US vs. China dynamic: in reality, China’s authoritarian system is actually way more logical and resilient than it sounds from the outside. While openly criticizing Communist Party policies is pretty much banned domestically, the public can still shape the decisions of the government and the Party through public opinion. It ties back to that famous quote: "In China, you can’t change the party, but you can change the policy; in the US, you can change the party, but you can’t change the policy." Even if some policy changes in China happen slowly, compared to the US, it actually works out a lot better most of the time.
As for the article mentioning the use of AI for cyber and biological attacks—they know full well that the US has already deployed AI in actual warfare, which is exactly why they conveniently dodged that topic. It’s incredibly hypocritical. Ironically, the one that hasn't actually engaged in that kind of behavior is the "authoritarian" Chinese government. Sure, you could speculate that China might use AI weapons against Taiwan down the road—especially considering Taiwan likes to build fortifications near schools to create leverage against the PLA—but launching a moral crusade over something that hasn't even happened, coming from American companies whose own country has already done these exact things, is just plain shameless.
Another common misconception is trying to separate the Chinese government from the Chinese people, with arguments like: "The Chinese people are oppressed, so you have to look at them separately; the government is evil, but the people aren't." You only need to look at the US to see the flaw in that logic. The American public voted Trump into office, letting him trigger trade wars and attack other nations. Does that make the American people evil? If the answer is no, then it implies the US isn't truly a democracy, since only an authoritarian state could completely ignore its people's wishes and do whatever it wants. If the answer is yes, then the US really is a democracy—it's just made up of malicious people, much like the company that put out this article.
China hasn't threatened to annex my country yet, at least.
> All sufficiently capable models, open and closed, should go through mandatory safety testing.
Yeah, just like voting eligibility tests aren't not to prevent people you dislike from voting! /s
We care about specifically Chinese models, because China distills our models. And that's real competition. So we want to ban those.
PS. Oh yeah, in case any open non-Chinese model will one day be good enough to compete with us, we will want to ban you as well, just FYI.
Kthxbye.
It is ok that we digest all information we can get, (il)legally and/or (a)morally because we are the good guys. Trust me bro.
It is not ok if others digest from us. They are bad guys. Ban them pl0x.
It's so obvious they are hoping to regulate out their competition rather than compete
Begging, ugly crying, spitting for that sweet-sweet regulatory capture. These nerds need to be bullied harder.
> Open-weights models that don’t have dangerous capabilities are a public good
Knives should only cut during the day, knives which cut at night are bad.
