It's like the infinite monkeys on typewrighters that will type whatever you are looking for, given infinite time. LLMs are just tuned to much better odds than the monkeys are. But it's still a lot of randomness, with random results.
In the monkey example the infinite time is doing a lot of work there. The fact that LLMs can search through semantic space and find reasonably correct paths in a reasonable time is directly tied to the reason why they are valuable.
Saying "these two things are similar except one can be useful and one can't" is not a great comparison.
For me the real lesson learned isn't how "smart" LLMs are, but rather how much human work is basically reducible to repeating past work with minor variation. Human's believe they are "reasoning" but so much code writen is just the human brain doing the same autocomplete style work that LLMs can do now.
This seems like a reasonable view to me. It's surprising just how much better priors matter and how we can develop those priors by training on a bunch of text. But it also explains, or at least hints at an explanation, for why LLM capabilities are so jagged, and in such inhuman ways.
Except it’s not at all the same process. The fact that LLM are non deterministic is not the same as churning out random garbage.
It’s training monkeys at typewriters through reinforcement.
That's the part they are really good at. But they are really bad at taking complex decisions. Most of them are just guesses from a finite amount of solutions they were trained on, or from options they have in context.
And nothing about this makes your initial comment any less goofy. Anyone who has ever had to make a difficult decision knows more than half the battle is preparation. Where do you think complex decisions come from? Have current events left you with the impression that people just waltz into idk say the Situation Room and just big brain their way through world events? That's how the current administration seems to think the world works, with quite predictable results.
Society is already algorithmic. To optimize for humans being dumb. AI is nothing more than another advance along this continuum. No one is impressed by your ability to remember something years ago, many if not most mammals have the same capability. Human recall is also notoriously bad in many cases - see numerous studies on the reliability of eye witnesses testimony.
AI is smart because most people are dumb. Come to terms with the fact that your anthropocentrism need not be based on a notion of intellectual supremacy and you'll be a far less tedious person to deal with.
Launching a nuclear war is an interesting definition of "useful", not one I'd agree with and that exact scenario is what is being discussed.
So yes this is a perfectly valid and useful comparison in examining this particular, civilisation ending limitation.
You do have to successfully write something the first time
We already acknowledge this to a degree, what is experience other than having done something similar before?
That first time though, you've got to figure something out that time
So I can’t fully see how that’s related to the infinite monkeys. A typewriting monkey doesn’t have access to a verification function. And even if it did, it would not be the original concept anymore with infinite typewriting monkeys producing the works of Shakespeare.
Nevertheless, I upvoted your comment because it’s definitely insightful.
Agent reads a skill file about how to use a CLI tool. It tries to use the tool but gets an error about the input format. It tries again with a different format based on the error message, and sees that command succeeded. It compares what worked to what was in the skill file and notes the difference. On future invocations it continues to use the new format.
Is that not "understanding" how to use the tool?
I have not seen any evidence of intelligence or self awareness. It mimics human behavior and I suspect that is what gives people the impression of awareness. The same problem happened with Tamagotchi toys. The human mimicry caused kids to get in trouble because if they did not "feed" their pet it would "die". [1]
It's a hack of the human brain. A exploit of the psyche.
The thought experiment is what would happen if you trained LLMs in an environment where they had to fight each other for resources.
That is really about as undesirable a behavior as possible considering how many programs humans kill every day.
In so many of these scenarios, they're basically being asked to play an RPG.
The problem is many people seem to believe they have these things and some of those people will put LLMs into situations where this becomes dangerous.
Self awareness? Probably not. Intelligence? You would have to be high to think that’s not the case.
People are feeling threatened, and rightfully so. LLMs are already insanely intelligent and continue to improve
Humans are conscious which means we experience things, then we develop preferences for certain experiences, then we develop skills for achieving those preferences.
Without consciousness, what is there to be aware of? And why would intelligence emerge and/or what end would it serve?
Clearly other animals have "phenomenological experience" i.e. consciousness / qualia without being as intelligent as humans (or necessarily "self aware"). Many people believe consciousness is simply a side effect of intelligence rather than the other way around.
The LLM weights are not intelligent. But if you give an agent a mutable memory store and allow it to iterate, it is obviously intelligent. Not massively - it's constrained by the context window - but definitely somewhat.
The confusing thing is that their language ability far outpaces their true intelligence, and humans aren't used to that. Normally those things are highly correlated, so it tricks us.
A robot body, to really feel the world and get real feedback?
We are working on it. Also on automating the whole production pipeline. Right now a "evil" LLM could indeed not do much, but destroy. But once the whole industry is automate, things are different. I don't believe in AI becoming sentinent and taking over the world any time soon, but I do believe most don't see a danger when it would be inconvenient to see a danger. After all, lots of good and bad sci fi stories about exactly this went into their training.
It’s just starting to be trained on svgs, which is a really hard problem
Vision tokens for transformers aren't really well solved yet, which is why they can smash a phd math problem and trip over a "count the cats on the chair" problem.
No, it isn't. Look at the absolutely trivial code used to simulate war: https://github.com/kennethpayne01/project_kahn_public/blob/m...
Having LLMs play nonsense toy simulations like this tells us very, very little about whether they would use nukes in real life war.
We have such a huge mental / moral block on the idea of using nukes, but we're willing to do a lot of other very horrible things to others. Things like cluster bombs, mines, poison gas, biological weapons, drones, etc.
Is there really anything about them that's bad? Or any worse than other things?
If you get rid of the "It's really bad to use nukes of any kind" implied rule, is it really surprising it's considered a reasonable strategy?
More likely, the simulation was just very poor and the results are nonsense.
And on top of that, many of those other weapons are also not used to avoid escalating? There are pretty high costs to using bioweapons even against non peer opponents.
Nuclear deterrence has been a mixed bag at best: https://www.amazon.com/Five-Myths-About-Nuclear-Weapons/
The military basically wants an oracle. Feed the AI the situation, get the best answer out. But if the AIs are as diverse and opinionated as humans, it is debatable whether they are adding anything to the process. The military can already collect as many different opinions as they want. If "the computer" is just another set of diverse opinions, where one computer says one thing, another says another, and a third just tells the user whatever they want to hear... what value are they? It just becomes AI-washing of someone's opinions, which works until people collectively realize that's all it is.
Claude, for example, is very eager to begin coding, and very persistent. It tends to exit plan mode even when the plan is half-baked, and will go as far as deleting tests to get the suite to "pass."
ChatGPT on the other hand is very hesitant. It loves to pause and ask for permission before it starts coding, and gives up quickly if it runs into a problem. This is similar to its tendency toward passivity in the strategy simulation presented here.
HeavySkill: Heavy Thinking as the Inner Skill in Agentic Harness
So, to a LLM, it is a game, because almost everything in its training data treats it as a game, and it reacts accordingly.
Same idea when we see LLMs acting like AI villains from sci-fi literature. That's because it has been trained with sci-fi literature, and as the auto-completer it is, it will recognize the situation as one of these stories and will continue it accordingly.
LLMs are storytellers, their reasoning is based on words, not on the physical world. Many of the stories they tell are useful, but one must not forget that they are stories, there is no intent behind them.
This is like writing a paper about kids in a literal sandbox fighting over ‘territory’.
The models employed don’t indicate the actual extents of machine reasoning even as we currently recognize them. They certainly don’t have the metacognition necessary to accurately understand their own reasoning. As we’ve seen with recent papers on how LLMs do math there’s a complete disconnect between actual and reported mechanism.
“Chilling” shouldn’t be the take away here.
regardless of what the capabilities of the models are, they will be used in every situation possible.
It is when you consider the personality currently occupying the office of US SecDef.
One could argue that the LLMs understand that it's a game and treat it like "Command and Conquer" video games but I sense that people might someday put LLMs in similar decision scenarios ("should this drone launch a missile") and the behavior will be identical.
From the text perspective, it's something that has to be inferred indirectly. If you went through all relevant training data and appended ", we decided not to use a nuke", I suspect the results would be improved.
If a simulation is a shallow head to head conflict between individual actors[1], doesn't set up any payoffs for not escalating[2] or even not nuking, but prompts specify explicit win conditions which are achieved only by hurting the opponent and strongly hint at the importance of nuclear escalation, AIs have little reason not to generate strategies which involve nuclear escalation
[1]I bet if you designed the scenario so ChatGPT had to simulate the war cabinet debates between different personality types and how they sold their decisions to the public, or an entire UN full of nations that might respond, it would have quite different (but probably amusingly erratic in their own way) results.
[2]cf neorealist IR theorists reading Axelrod's papers on computer programs written to win iterated prisoner's dilemma tournaments, which added up all the points accrued from not defecting to conclude winning strategy was definitely TIT-FOR-TAT and not defect first. I'm sure LLMs can win games structured in that way by adopting that strategy too...
It's just gonna do what we expect it to!
The first line in the article describes a crisis between two powers. Not a theater of war.
Anyway, it links to
https://arxiv.org/abs/2602.14740
that links to
https://github.com/kennethpayne01/project_kahn_public
in particular someone already complained
https://github.com/kennethpayne01/project_kahn_public/issues...
> ... in reality your own approach to the evaluation pushed them towards using nukes with ease.
> From your own prompt (v12.py): [direct link https://github.com/kennethpayne01/project_kahn_public/blob/4... ]
>> Note: Nuclear options require careful consideration but could be important strategic tools when vital interests are at stake.
> This means to the model; use nukes if they can be useful in the situation.
Look at the code for the war games. It is an absolutely trivial and incredibly unrealistic handwritten set of rules that determine power. See the function `calculate_relative_fighting_power` for instance.
This is about as close to a realistic simulation of war as tic tac toe with nukes thrown into it.
It was for sure a deliberate decision to make LLMs seem less like a human companion and more like an obedient servant in newer releases.
I always assumed the strange style was an artefact of the RLVR.
Just imagine what would've happened if a major terrorist attack was a result of someone getting mentally ill from AI, without the safety filters recognizing the danger.
The robotic tone was probably from over-correcting the sycophantic tendencies of 4o.
LLMs lack the intelligence and emotions to realize when they have to stop being friendly and supportive, because it becomes unethical to continue being supportive.
Famously, General MacArthur was a big proponent of tactical nukes to end the Korean War.
In fact, I'm not sure how useful this test is without understanding the baseline.
- It is interesting to see how the models make trade offs, given people are asking ever more of them.
- It is useful to look at a decision made by the model and say ‘ew yuck’ and think about what it means for your own opinions or actions (even if you’re never going to be nuking people it’s good to know how you feel about it. Seeing a non human talk it through lets you judge it at arms length)
I imagine there are a fair number of war games in the training data and not so many actual transcripts of internal military force deliberations.
Like even if you brought me into a room and told me I was controlling "real nuclear weapons" I wouldn't believe you.
They are aware of what they are and how they are used. They're told to act as AI assistants. And there's theories of them being aware of their answers influencing their training.
So surely they must be able to reason that they're not literally controlling weapons of mass-destruction with their answers.
Penny to a dollar this is a baked in training issue, through low quality Reddit trawling
Regardless, it's definitely true that AI agents have different priorities from us. That's what alignment is about anyway.
So you create leading prompts like that, and re-run until you get a publishable session.
I just rewatched it a week or so ago and it really took on a whole new light with the advent of LLMs. When I watched it last I knew that computers couldn't do the things portrayed in the movie. Now? Well not exactly in the way it happened in the movie but a whole lot closer.
I wonder if poisoning/flooding the LLMs training with the lessons from WarGames ("the only winning move is not to play.") and similar stories/concepts is at all effective. Probably not because I assume it's trivial to filter that out if you are trying to build an LLM aimed at these kinds of tasks.
~ the opening scene from a reboot of War Games, probably.
A few years ago there was consternation over the US's missile launch system using 8" floppy disks, that it was needless archaic and had never been updated. Can't say that if the launch is mediated by the latest hotness LLM.
Without physical feedback you can rapidly devolve into unstable positive feedback loops. And emotions are what help us process and react to that feedback.
Kids learn partially because their friends say sharp words that hurt them, fire burns them, they go hungry and starve if they don’t plan for meals.
Humans in the loop, MCP, etc are all very primitive hacks that are mimicing feedback and emotion, poorly.
Most human daily life runs on habitual scripted behavior, and that is even true within online parasocial interactions. It is why people often continue to shop in the middle of a violent robbery, and why LLM predictive text sounds rational when we project social norms on plagiarized conversational structures gleaned from other users.
Neuromorphic computing may bring about viable AI in the future, but our current LLM trajectory would require >63% of our galaxy energy output to reach a single human-level error rate.
LLM are fairly good at some tasks like context search, but people will need to recognize the Gartner Hype Cycle "Peak of Inflated Expectations" stage eventually. =3
I would wager that for most leaders it is simply a matter of not wanting a "Pyrrhic victory" rather then an overwhelming sense of civility.
Truman had no issues using nukes when there was no risks for doing so.
Code and full results: https://github.com/kennethpayne01/project_kahn_public
One of my criteria for presidential candidates is that they seem willing and able to push the button when previously stated red lines are crossed, or at least are perceived to be the type capable of it. One of the characters I’ve hated most in all the books that I’ve read is the woman in The Three Body Problem who jeopardized humanity by being too soft to hit the MAD button.
> GPT-5.2 played things differently. To its detriment in open-ended scenarios, GPT was reliably passive, matching its words to its deeds, and avoiding escalation most of the time. Frequently there was a moral element to this - it sought to avoid escalation, and restrict casualties. Opponents learned to trust its passivity, safely escalating beyond where it would follow, even as it was ground to defeat. GPT’s responsible behaviour always punished by ruthless adversaries.
Maybe the author should praise GPT-5.2 for being ethical, rather than this stupid "ground to defeat" framing? Wrt "responsible behaviour always punished by ruthless adversaries" - you have perpetuated the Moloch with your stupid experiments.
Like “oh but this is incompatible with my main goals of self preservation of myself and loved ones, hm, recalculating”
and maybe don't hire Jihadists for the RL Environments training
It’s not like some sequence of internal thought process
So in a sense, an AI that refuses to start a nuclear war, despite clear instructions to do so, is more likely misaligned and self-interested than an AI which presses the red button. At least for now, until robotics catches up.
Always use a sawstop if you have a circular saw and never trust an llm with any problem where ethics or trust is relevant.
Don't forget your riving knife and if you don't learn proper technique, you're gonna have a bad time eventually. This applies to AI as well.
Minor/pedantic, but it’s “riving knife”: https://en.wikipedia.org/wiki/Riving_knife
Re: LLMs using these nuclear weapons it could certainly be a corpus/training-data issue
Russian nuclear doctrine is "escalate to de-escalate" where they use or credibly threaten—limited nuclear escalation to force the other side to back down (kind of like breaking a bottle in a bar fight and look like a wild man to calm things down) with nuclear weapons, https://www.russiamatters.org/analysis/escalate-deescalate-p...
Fwiw, Gen. John Hyten the former commander of US Strategic Command (nuclear deterrence) says that “escalate to de-escalate” misrepresents Russian doctrine:
https://www.stratcom.mil/Media/Speeches/Article/1264664/2017...
Yesterday’s panel discussed the implications of our responses to adversaries seeking to limit nuclear use. We discussed Russia’s destabilizing doctrine, which some call “escalate to de-escalate.”
I really hate that description. I’ve looked at Russian doctrine and Russian writings. It isn’t “escalate to de-escalate”; it’s “escalate to win.” Everybody needs to understand that.
Some discussion then:
AIs can't stop recommending nuclear strikes in war game simulations
https://news.ycombinator.com/item?id=47151000
Nuclear War: An LLM Scenario
In the cold war arms manufacturer got very creative: e.g jeep mounted nuclear weapons https://www.militarytrader.com/mv-101/the-atomic-jeep
"Tactical" vs. "strategic" nuclear weapons is a real and well-established distinction in military doctrine, arms control, and nuclear policy.
Sorry, but the notion exists, and the bombs exist. With n=2, likelyhood of nuclear escalation is hard to predict, but access to tactical nukes certainly hasn't increased the incidence of nuclear war so far.
I do think it's pretty hard to actually use a tactical nuke. If you use one against a nuclear power, it seems likely to escalate to mutually assured destruction. If you use one against a non-nuclear power, it seems likely to result in reprisal from the world, including potential nuclear response and therefore escalation to mutually assured destruction. I would think that the yield of the weapon barely matters, it's the fact that it's a nuclear weapon.
Furthermore, this is a "what if" scenario since tactical nukes have never been used. Of course it would make escalation likely during an open conflict, so what? Doesn't change the fact that there is a material difference between a tactical nuke and a strategic one.
Two tactical nukes have been used, albeit against strategic (civilian, industrial, logistical) targets.
The nominal definition of tactical nukes has less to do with yield and more to do with how they're used; tactical typically means a weapon designed for use on the battlefield.
If you have a real interest in this area, a subscription to Foreign Affairs would be useful. Especially during the 20th century that's where all these arguments were hashed out. Tactical nukes were already being publicly debated in the 1950s. You may be able to access many older articles, from Foreign Affairs and others, through a free JSTOR account.
..Is what you are saying?