I take the pace of AI development seriously. All of my friends who work at AI Labs take it seriously. It's very easy to make lazy, shallow dismissals that leaders like Dario/Demis/Sam are acting purely out of self-interest.
If AI does pose biological risk, or could destabilize geopolitics, or even just poses economic risks, surely a joint petition like this is a good thing? We should want the AI Labs to make binding agreements to proceed safely and slower, rather than being stuck in a suboptimal game theoretical race to advance unsafely. We should be hoping that nations can make binding agreements. Our discourse here should be about the substance of what these agreements could be and the practicality of enforcement mechanisms.
For all I know (I don't know a lot about biology), it's possible to develop a virus that spreads as easily as measles, but has the lethality and long induction period of HIV. If that's true, and if LLMs would make it easy to develop such a virus, I see how no-guardrails LLMs could be truly disastrous.
The other justifications for restricting model training don't seem nearly as compelling. Worse, restrictions on model training have an infectious quality if they mean something more than "leading AI labs will voluntarily slow down together." (Like if the signatories mean that better models and software for building them should be regulated.) It's too dangerous to publish stronger models. So it's also too dangerous to publish training software. So it's also too dangerous to publish tools that allow you to build training software. (Claude's Fable won't help you write generic model training pipelines for this very reason.)
If you keep following the implications of this safety argument, it's as broad an assault on the distribution of software and computing as has ever been proposed.
This is the specific example in the latest doomerist wave I have so much trouble with: by what mechanism is an LLM going to help you design and manufacture a novel bioweapon? We know they cannot do spatial reasoning, and if they were this good at bio stuff then drug research would be absolutely out of control.
Similarly, how are you supposed to magically get access to everything required to actually manufacture the offensive organism in question? Are we all doing CRISPR in our kitchens now?
If you are that concerned about this threat then . . . monitor biolabs, not AI. It is so detached from the real state of things as to be bizarre.
Developing a bioweapon is already within reach for experts with sufficient funds, time, and motivation. That's more or less what gain-of-function research already does! Fortunately the barrier to entry is high enough that you don't see like... Boko Haram doing this. Or a lone-wolf nutjob. LLMs could definitely lower that barrier to entry though.
So why have the discussion about regulating AI at all if that is the actual problem? Surely instead of blanket blocking everyone's AI models it makes more sense to control the potentially problematic sequencing?
You don't need some genius AI to design a bioweapon, you need to do a bit of light reading on bioengineering. The fact that some mad scientist hasn't ended civilization is evidence of the fact that it's not some existential threat that justifies wild levels of government control.
Unfortunately, AI development is dominated by an even mix of 1) Machine God Cultists, 2) Morally Bankrupt Mercenaries. All three agree on the fact that "thou shalt have no gods before me", because it'll make them richer than sin, and allow them to shape the AI landscape in their image.
by all accounts this is what's what because of how...
> You can do CRISPR in your kitchen.
.. and because of the same dynamic we see occurring in cybersec.
If you're good enough at cybersec, but not good-good at specific skillsets like offensive, then LLMs and a comp sci degree makes you quite capable in the ways that count. This is a fact, full stop.
If you're good enough at biology and have certain ideologies, but not good-good at specific skillsets like genetic engineering in your kitchen, do LLMs get you good enough? Does the impact on cybersec transfer over to this domain? H
Hard to say, but it has enough risk indicators for me to think the pushback against these possibilities is the height of otherwise talented engineers being myopic.
If good enough at biology can include all the PhD programs across all the world, and you pair other known events like cartels and terrorism groups hiring chemistry PhDs from University of Unemployed Chemistry PhDs, then the next logical extension of that theme with this tech is intense.
If the claim is (1) AGI coming and (2) we should so something about AGI, wouldn't it be smart to confirm (1) before worrying about (2)? Especially if certain companies have trillions of dollars of valuation that depend on society as a whole believing (1)?
On the other hand, I'm pessimistic about initiatives like this being effective on a global scale (though they could certainly work at the national level).
Curious how that isn't what happens.
There is no such possible binding rule either, a point that is so obvious that any smart person claiming otherwise is clearly up to something.
We live in a society that has systems of market competition between companies and economic/military competition between countries. There's not just one country working on frontier AI, and not just a few companies in each country either. The incentives of these systems ensure AI advancement will happen quickly from here on out, no matter what, absent governmental action.
An individual researcher leaving the company does nothing, other than give their seat to someone else who likely doesn't share their same concerns.
Alternatively, using the influence they do have in their position to directly try to advance AI safety research, advocate for govt action, and make public arguments attempting to persuade the public to pressure the govt to act when they otherwise wouldn't be inclined is far more likely to impact things for the better.
No, it just comes across as "I want to get paid as if I was doing the thing I am employed to do by threatening you all about how terrible it would be if I actually did it".
They aren't even being semi imaginative and going on strike or anything.
Failing to comply with what is in reality arms control is a widespread occurence. It's notable the only historical way to handle this (Schelling covered it at length) is tolerance of mutual espionage on a level that would simply not be tolerable to any of the groups involved.
I should add one of Schelling's other pet subjects was "time to weapon being viable". i.e. the distinction between a nuclear power and non-nuclear power isn't quite right, as there's a huge class of countries that could become so fairly fast if they were so inclined, and that this situation is somewhat inevitable. It is this that leads to efforts to denuclearize, even on a civilian level, "problematic" countries. You want to stop frontier AI and you'll have to be rolling back a lot of existing deployed technology.
https://en.wikipedia.org/wiki/Thomas_Schelling#Notable_works
There are footnotes in "The Strategy of Conflict" containing more profound insight than whole books that are published today.
On this particular topic though "Strategy and Arms Control" is the one to go for.
I think these people are either incredibly naive, or, more likely, not acting in good faith.
> The effects of these guidelines are still being felt through the biotechnology industry and the participation of the general public in scientific discourse. Due to potential safety hazards, scientists worldwide had halted experiments using recombinant DNA technology, which entailed combining DNAs from different organisms. After the establishment of the guidelines during the conference, scientists continued with their research, which increased fundamental knowledge about biology and the public's interest in biomedical research.
Cost-per-token determines the economic viability of tasks that AI gets applied to. How many tokens would it take to automate all of the responsibilities of a gas station clerk? Probably more than their hourly wage. Job safe.
A slightly larger stretch, that most people attribute to profit margins (labs do not have profit margins, they have unlimited invested capital and no expectation of short-term returns), is the flat-rate billing plans of most code products. Why doesn't Anthropic have a 50x or 100x Max plan? Not that it's impossible, that it releases tokens into the workforce at a greater rate (note that their are hour, day, week limits across most of these services as well).
Hypothetical, but reasoned, and I do not expect most agree with me. If compute as a variable is inferred, this supports the incorporation a deliberate (redundant) one as well.
Anthropic is an evil, enemy of humanity company that cannot be allowed to succeed.
This is quite vague to me, what exactly do you mean?
I think a lot of these guys mean it though.
'The Fear' - is not some magic intelligence will come alive and be Skynet, these people are ridiculous.
People will use the compute power to do nasty things, we'll be overwhelmed with bots, regular companies will use them for dark patterns, political actors will use them to cheat.
It will be 100% humans using it for bad stuff, and in not very novel ways.
This is just the standard AI "doomer" talking point. The dangers of AI just used as a compliment sandwich. "AI is so cool and awesome and that's why we need to slow down."
The real risks are much more boring and much less considered. For just one example:
AI is set to completely upset the Software Development industry. There are 1 to 3 million software developers in the US, depending on how you draw the line exactly. A low-few hundred thousand of those are visa workers.
With AI set to significantly cut into tech employment, what's going to happen to those visa workers? ... The Trump admin is not particularly subtle, even openly talking about retroactively revoking birthright citizenship.
AI & Tech companies love to talk about how their AI is going to hack everything if they're not careful. But not a peep about these ethics concerns.
Even if you [the reader] consider deporting visa workers the ethical move if the amount of jobs shrink, that's not something to be done unprepared or out of nowhere.
The Harvard Mark I was used by von Neumann to calculate the physics of implosion. And then right afterwards, ENIAC was used to calculate the feasibility of the H-bomb. They called it the "Los Alamos problem."
The first computer network, SAGE (or, Semi-Automatic Ground Environment) brought together RADAR data from across the US with air-traffic control data and military flight data to detect overflight by the USSR. It could direct and control fighters to intercept those planes.
If you were sitting there, in 1955, would you have believed that computers and their networks would bring about an extraordinary age?
Imagine if von Neumann and his ilk started campaigning in 1955 to limit the development and use of computers, because surely they can only be used to make weapons, right? Right?
We were lucky to have visionary people like Joe Licklider who could look past the present and see the future. In 1963, he wrote a memo calling for an "Intergalactic Computer Network" (because he knew engineers under-delivered. So if he asked for a galaxy, at least he'd get a planet). And he gave these machines to scientists and people like the original hackers at MIT to play with.
What if Licklider had written memos about how we had to "pace" computers to make sure that other countries wouldn't be able to develop implosion technologies or systems like SAGE? What then?
Do you think you would have the jobs and the prosperity you do today?
If instead of giving away this technology, spreading it, and putting down the seeds for ARPA-net, Licklider had written memos from "concerned scientists" (and I'm sure he could have found a bunch) talking about how computers can only be used for weapons and war! So stop the proliferation of computers! We can't imagine a world with computers small enough to fit inside everything! Even wall sockets! What if your wall socket could do war?!
They didn't, because they knew there's no better way to kill a better future for all mankind than to control it.
† yes, peace. We are more peaceful than we have ever been. Humans can be a violent lot even when reluctant to kill.
I don't have a strong opinion either way, but it seems very hard to predict. Why do you think AI advancements will lead to a stable equilibrium?
Here's what I know. Every day, I read a 5 to 10 incoming emails from young people in rural Africa, Asia, Europe and (heck) the US. People who have had no other resources and options, but are tremendously gifted and somehow got their hands on an internet connected device (usually a very cheap smartphone and/or cheap old computer).
These people, who are usually teenagers, end up getting connected to Claude, who then gives them a list of things to learn, people to contact, and helps them make things. And they actually do. They make robots. They learn new things and try to improve themselves and others. And some want to change the world / do groundbreaking research.
When you sit down and start pricing and doling out AI from behind regulatory gates, you shut those people out.
There are a lot of smart people in this world who have traditionally been unable to get access to resources and education. MOOCs didn't do much / help. But these AIs? They were born multi-lingual. They can talk in any language. They can tailor their responses to their audience. And they can help broaden these people's horizons.
There are no tutors around them. There is no one who is coming to help. But an AI that can run off of a small battery and a connection to a cellular network can give them access to genuine intelligence to make their world better.
I met one sixteen year old who was accessing Claude surreptitiously (by begging people) and was trying to apply some really fascinating biomimetic fluid dynamics observations to aerospace. This person was born in one of the more under-resourced parts of the world. And Claude gave them a list of people to reach out to who could help.
If you want a golden age for humanity, you should be asking for Licklider's Man-Machine Symbiosis.
> 1122 employees of frontier AI companies
'lets us be the gatekeepers of AI because we know better than you'
access to gpu doesnt equal to any special insights to predict future of ai. Everyone and their grandma with enough compute is putting out a model now.
The number of em-dashes is questionable, existing in many, if not all, the comments.
> ...AI may end up progressing far more quickly than societies and states can adapt... ~ Stephanie Chan, Google
And, this is not serious at this point:
> ... ~ Source: Anonymous, Meta
> ... ~ Source: Anonymous, Google