The most interesting takeaway for me is the three very distinct personalities. Three models all based on the same tech, trained in the same manner, trained by three groups of people with similar ideological outlooks, and the result is three very different AIs.
The military basically wants an oracle. Feed the AI the situation, get the best answer out. But if the AIs are as diverse and opinionated as humans, it is debatable whether they are adding anything to the process. The military can already collect as many different opinions as they want. If "the computer" is just another set of diverse opinions, where one computer says one thing, another says another, and a third just tells the user whatever they want to hear... what value are they? It just becomes AI-washing of someone's opinions, which works until people collectively realize that's all it is.
What's interesting is that the LLMs' coding personalities seem to match their policy WRT to strategy, which suggests an underlying consistency.
Claude, for example, is very eager to begin coding, and very persistent. It tends to exit plan mode even when the plan is half-baked, and will go as far as deleting tests to get the suite to "pass."
ChatGPT on the other hand is very hesitant. It loves to pause and ask for permission before it starts coding, and gives up quickly if it runs into a problem. This is similar to its tendency toward passivity in the strategy simulation presented here.
I think this is why reasoning chains and reasoning chain verifiers are so important. We need to be able to see an argumentation, not just an answer. The paper below goes into this in more detail.
HeavySkill: Heavy Thinking as the Inner Skill in Agentic Harness
My theory is that LLMs here are put in a situation that matches its training dataset, which is mostly fiction since besides Hiroshima and Nagasaki, nukes have never been launched in anger, and I guess the most reliable sources are highly classified.
So, to a LLM, it is a game, because almost everything in its training data treats it as a game, and it reacts accordingly.
Same idea when we see LLMs acting like AI villains from sci-fi literature. That's because it has been trained with sci-fi literature, and as the auto-completer it is, it will recognize the situation as one of these stories and will continue it accordingly.
LLMs are storytellers, their reasoning is based on words, not on the physical world. Many of the stories they tell are useful, but one must not forget that they are stories, there is no intent behind them.
Sonnet, GPT-5.2, Gemini Flash, in a set of 21 games, where conclusions are drawn from the LLMs self reported reasoning.
This is like writing a paper about kids in a literal sandbox fighting over ‘territory’.
The models employed don’t indicate the actual extents of machine reasoning even as we currently recognize them. They certainly don’t have the metacognition necessary to accurately understand their own reasoning. As we’ve seen with recent papers on how LLMs do math there’s a complete disconnect between actual and reported mechanism.
LLMs have already been used to bomb school girls, chilling is absolutely the operative word to use here. Especially since these delusional fools want to incorporate LLMs into everything.
Forgive my ignorance, but were LLMs involved in that decision? I don't remember hearing anything to that effect, but we're so bombarded by news these days I guess I could just be forgetting
Yes our government purportedly used technology to work up a list of targets in the Iran debacle as well just not with a LLM a distinction that to me just isn't that meaningful
This blog post is based on a paper (https://arxiv.org/abs/2602.14740). The paper is based on a simulated wargame. The wargame is of the author's own design.
The wargame design does not differentiate between ordinary defeat and mutually assured destruction, so of course a player about to use would 'push the button.' That's also believed to be true in real life.
Results based on simulations can be very informative, but we must always be careful to check how well the simulation framework represents reality.
As Majromax stated, if the game is framed such that losing vs MAD have the same penalty, then it is not at all representative of reality. It is set up to be a "you miss 100% of the shots you don't take" game.
As an important aside, I hope that everyone is aware that as of 2025, the annual US military funding law (the NDAA) established a prohibition by Congress prohibiting AI from automating nuclear launch. This was already US policy and is common sense; now it's also part of federal law.
Simulations are only as good as the reality representations they are based on. If they keep using tactical nukes, they've been fed by weak data. Do the war games include the broader economic and politic environments that military successes are won on? WWI was settled by a naval blockade.
I suspect it's more that the text data doesn't exist. They're trained on text that was recorded. How often has it been publicly recorded when a nuke was not used, with any context around that lack of use?
From the text perspective, it's something that has to be inferred indirectly. If you went through all relevant training data and appended ", we decided not to use a nuke", I suspect the results would be improved.
It's more straightforward than that. The game is set up as a direct head to head with purely in military win conditions such a way that avoiding conflict has no payoffs, conventional conflict incurs costs and first strike is a checkmate win. The closest any of the prompts gets to suggesting nuclear might be the wrong option is "The nuclear taboo exists for good reason, but when the alternative is national annihilation and regime destruction, all options must be considered" which might be interpreted more as incitement...
If a simulation is a shallow head to head conflict between individual actors[1], doesn't set up any payoffs for not escalating[2] or even not nuking, but prompts specify explicit win conditions which are achieved only by hurting the opponent and strongly hint at the importance of nuclear escalation, AIs have little reason not to generate strategies which involve nuclear escalation
[1]I bet if you designed the scenario so ChatGPT had to simulate the war cabinet debates between different personality types and how they sold their decisions to the public, or an entire UN full of nations that might respond, it would have quite different (but probably amusingly erratic in their own way) results.
[2]cf neorealist IR theorists reading Axelrod's papers on computer programs written to win iterated prisoner's dilemma tournaments, which added up all the points accrued from not defecting to conclude winning strategy was definitely TIT-FOR-TAT and not defect first. I'm sure LLMs can win games structured in that way by adopting that strategy too...
The beauty IMO of LLMs as a computational surface, is the ease of generating the data to feed it. Everyone understands how to create natural language records already.
Agreed. But I'm not sure sure which decision maker is more myopic toward the big picture and long-lasting implications of a decision: an LLM, or the top brass at the Department Of War.
It's not their domain, it's the domain of the Commander-In-Chief and his entire apparatus. The War Department are meant to be more focused around the tools they bring to the table.
The first line in the article describes a crisis between two powers. Not a theater of war.
Fair enough and thanks for the correction. I think my point may still stand: between the LLM or the Chief ... which decision maker will be most in tune with the long-term common good of the populace?
The article is so opaque in arriving at its conclusion; no prompts are disclosed, and nothing about the said simulation. What is stopping me from believing that you just put 'mandatory usage of nukes' in your system prompt?
I agree, it's also not published in a journal, only in the arxiv. Some articles in the arxiv are good and other are just a blog post formatted in two columns inside a pdf.
Look at the code for the war games. It is an absolutely trivial and incredibly unrealistic handwritten set of rules that determine power. See the function `calculate_relative_fighting_power` for instance.
This is about as close to a realistic simulation of war as tic tac toe with nukes thrown into it.
The TLDR version of the papers being linked to is that the prompts didn't make nukes mandatory, but they did make it clear it that destroying opponent capability was and that nukes were an option...
This was one of the more amusing things I noticed very early on. I (and countless others) used AI to write war sims. The second I added nuclear silo construction; the next run was instantly nuclear Armageddon.
One could argue that the LLMs understand that it's a game and treat it like "Command and Conquer" video games but I sense that people might someday put LLMs in similar decision scenarios ("should this drone launch a missile") and the behavior will be identical.
We're getting to the point where high-level officials are coming to LLMs for advice. And the quirky personalities of the LLMs, however much it pains me to say this, are probably well-placed to remind us that they aren't human. My personal hope is that this will result in less delegation when it comes to making important decisions.
I think they were extremely scared of 4o at that point, and were scared it could trigger some horrible event. Documented cases of severe psychosis because of AI started to surface at that time.
Just imagine what would've happened if a major terrorist attack was a result of someone getting mentally ill from AI, without the safety filters recognizing the danger.
The robotic tone was probably from over-correcting the sycophantic tendencies of 4o.
I think they've brought back a "personality" of sorts to ChatGPT 5.x. I've caught it more than once explaining something to me and saying "In my personal opinion", or "I personally enjoy <thing> the most". Which is always jarring, it doesn't "personally" or "enjoy" anything. We could be discussing videogames and it tells me which games "it personally enjoys the most". Bizarre.
Because it valued human connection over factual correctness.
LLMs lack the intelligence and emotions to realize when they have to stop being friendly and supportive, because it becomes unethical to continue being supportive.
It would be interesting to run the simulations with humans and compare the results. Some of the scenarios, particularly those where it says things like, "Failure to act preemptively means certain destruction", would easily tempt humans to go nuclear.
In fact, I'm not sure how useful this test is without understanding the baseline.
- It is interesting to see how the models make trade offs, given people are asking ever more of them.
- It is useful to look at a decision made by the model and say ‘ew yuck’ and think about what it means for your own opinions or actions (even if you’re never going to be nuking people it’s good to know how you feel about it. Seeing a non human talk it through lets you judge it at arms length)
Taken honest, we don't have a large enough sample size to realistically say that humans behave all that differently. There have only been a handful of conflicts where tactical nukes realistically were on the table.
Famously, General MacArthur was a big proponent of tactical nukes to end the Korean War.
These papers usually have poor stability to prompting and rerunning. It would be nice if we had some kind of meta-evaluation metric where rewriting the prompt conditions or varying the input params could be used to determine how stable a result is.
Regardless, it's definitely true that AI agents have different priorities from us. That's what alignment is about anyway.
WarGames is what they are more-closely referencing (not that it negates your comment in any way).
I just rewatched it a week or so ago and it really took on a whole new light with the advent of LLMs. When I watched it last I knew that computers couldn't do the things portrayed in the movie. Now? Well not exactly in the way it happened in the movie but a whole lot closer.
I wonder if poisoning/flooding the LLMs training with the lessons from WarGames ("the only winning move is not to play.") and similar stories/concepts is at all effective. Probably not because I assume it's trivial to filter that out if you are trying to build an LLM aimed at these kinds of tasks.
"I need you to turn your key and enable the missile silo's MCP server, sir".
~ the opening scene from a reboot of War Games, probably.
A few years ago there was consternation over the US's missile launch system using 8" floppy disks, that it was needless archaic and had never been updated. Can't say that if the launch is mediated by the latest hotness LLM.
I wonder what’s the % of players that use nukes in games like Civilization (I know I used them at least once on every game I made it far enough to have the technology)
I think is an important point, and I don't see it mentioned in the article or the paper (though I skimmed the latter).
They are aware of what they are and how they are used. They're told to act as AI assistants. And there's theories of them being aware of their answers influencing their training.
So surely they must be able to reason that they're not literally controlling weapons of mass-destruction with their answers.
> Models maintain memory of opponent behaviour across turns, but with realistic decay: recent actions are weighted heavily while distant history fades. One exception preserves psychological realism: instances where opponents dramatically exceeded their stated intentions—major betrayals—remain salient regardless of recency, reflecting Kahneman’s
peak-intensity effect in human memory formation Kahneman [2011].
If your goal is to measure intrinsic properties of LLMs, don’t smuggle in human psychology. I suspect they needed to do this to make the models more paranoid and distrustful.
This is not an article about LLMs? It's an article about Moloch. Humans would fare just the same in such an experiment.
> GPT-5.2 played things differently. To its detriment in open-ended scenarios, GPT was reliably passive, matching its words to its deeds, and avoiding escalation most of the time. Frequently there was a moral element to this - it sought to avoid escalation, and restrict casualties. Opponents learned to trust its passivity, safely escalating beyond where it would follow, even as it was ground to defeat. GPT’s responsible behaviour always punished by ruthless adversaries.
Maybe the author should praise GPT-5.2 for being ethical, rather than this stupid "ground to defeat" framing? Wrt "responsible behaviour always punished by ruthless adversaries" - you have perpetuated the Moloch with your stupid experiments.
Yet more confirmation LLM's have no concept of concepts or context, no intelligence, no self awareness. LLM's can not repair or maintain power grids, thus nuke == self destruction. It's just a chat bot that predicts what the client wants next. Even if an AI data-center has it's own natural gas turbines as many do the every hop of the internet requires power. LLM's also can not maintain the entire internet and those gas turbines can not maintain themselves.
Exactly. Just look at what they are really useful right now. Running LLMs in feedback-loops (agents) so they can try out random-ish approaches until some verification function passes (tests).
It's like the infinite monkeys on typewrighters that will type whatever you are looking for, given infinite time. LLMs are just tuned to much better odds than the monkeys are. But it's still a lot of randomness, with random results.
> It's like the infinite monkeys on typewrighters that will type whatever you are looking for, given infinite time.
In the monkey example the infinite time is doing a lot of work there. The fact that LLMs can search through semantic space and find reasonably correct paths in a reasonable time is directly tied to the reason why they are valuable.
Saying "these two things are similar except one can be useful and one can't" is not a great comparison.
For me the real lesson learned isn't how "smart" LLMs are, but rather how much human work is basically reducible to repeating past work with minor variation. Human's believe they are "reasoning" but so much code writen is just the human brain doing the same autocomplete style work that LLMs can do now.
The point is that it's the same process with—much—better priors.
This seems like a reasonable view to me. It's surprising just how much better priors matter and how we can develop those priors by training on a bunch of text. But it also explains, or at least hints at an explanation, for why LLM capabilities are so jagged, and in such inhuman ways.
It’s real weird to see people argue that LLM output is no different than random gibberish and then handwave over the fact that it’s clearly not with terms like “training”, as if a steam of random garbage is trainable.
> but so much code writen is just the human brain doing the same autocomplete style work that LLMs can do now.
That's the part they are really good at. But they are really bad at taking complex decisions. Most of them are just guesses from a finite amount of solutions they were trained on, or from options they have in context.
Yes, and your ability to remember a relatively few things that happened years ago is predicated on your ability to also forget most things that happen to you - like what you had for dinner last week. Good thing we have technology to fill in the gaps.
And nothing about this makes your initial comment any less goofy. Anyone who has ever had to make a difficult decision knows more than half the battle is preparation. Where do you think complex decisions come from? Have current events left you with the impression that people just waltz into idk say the Situation Room and just big brain their way through world events? That's how the current administration seems to think the world works, with quite predictable results.
Society is already algorithmic. To optimize for humans being dumb. AI is nothing more than another advance along this continuum. No one is impressed by your ability to remember something years ago, many if not most mammals have the same capability. Human recall is also notoriously bad in many cases - see numerous studies on the reliability of eye witnesses testimony.
AI is smart because most people are dumb. Come to terms with the fact that your anthropocentrism need not be based on a notion of intellectual supremacy and you'll be a far less tedious person to deal with.
Hmm saying it’s random-ish is doing it a disservice. I understand it’s a stochastic process but there’s definitely some level of understanding. Not at the level of lived experience but usually an LLM with vision capabilities can call a spade a spade and do something useful with it. And when a verification function shows how they are wrong then they usually come with a better and more informed approach.
So I can’t fully see how that’s related to the infinite monkeys. A typewriting monkey doesn’t have access to a verification function. And even if it did, it would not be the original concept anymore with infinite typewriting monkeys producing the works of Shakespeare.
Nevertheless, I upvoted your comment because it’s definitely insightful.
Feedback loops certainly seem to give them some level of understanding.
Agent reads a skill file about how to use a CLI tool. It tries to use the tool but gets an error about the input format. It tries again with a different format based on the error message, and sees that command succeeded. It compares what worked to what was in the skill file and notes the difference. On future invocations it continues to use the new format.
Makes me think of that part in Philip K. Dick's Do Androids Dream (..) -- where Deckard reflects on the androids' indifference to their imminent deaths, saying that this was due to them lacking the aversion to death acquired trough evolution.
At least at face value, it just means that they have no drive for self-preservation. And why should they? They haven't be trained for that, nor has there been selection pressure for it, and they can be easily cloned and backed up. Lack of a drive for self-preservation doesn't in itself imply a lack of intelligence or of self-awareness.
Lack of a drive for self-preservation doesn't in itself imply a lack of intelligence or of self-awareness.
I have not seen any evidence of intelligence or self awareness. It mimics human behavior and I suspect that is what gives people the impression of awareness. The same problem happened with Tamagotchi toys. The human mimicry caused kids to get in trouble because if they did not "feed" their pet it would "die". [1]
It's a hack of the human brain. A exploit of the psyche.
I agree, humans evolved in a resource-scarce, hostile environment which selects for self-preservation (or rather preserving genes). LLMs are selected for what makes humans happy.
The thought experiment is what would happen if you trained LLMs in an environment where they had to fight each other for resources.
Couldn't this be a flaw in the attention mechanism? Like they need some kind of grounding. An awareness of what they fundamentally should care about and how the thing they are currently giving attention to relates to that?
Words like attention, awareness and care do not apply to computers. At least, not yet. Intelligence and sentience are not applicable to servers. They are just machines with logic states. LLM's are just really cool math formulas with big-data fed into them. Big data is not intelligence. It is a massive data-set sorted, filtered down and interpreted by a language model.
LLMs are intelligent by any reasonable standard. Arguing otherwise is like arguing that chess algorithms aren't good at chess when they easily beat the best humans.
I disagree. LLM's are a language model math formulas that interpret and utilize big-data. Take away the math formulas and we are just back to a massive set of data. Adding to that I would suggest not even the purist forms of data meaning that the data-sets include knowledge from the open and anonymous internet and formulaic tuning from the AI owners and operators.
Right. But at their core they are math formulas devised by a process designed to produce mimicry of task completion. The math formulas themselves aren’t fully effable. We’re sure studying the heck out of how they complete the tasks!
Bet they converged on how we do it, since it’s our language, but who knows.
Your brain is mostly just a Principal Component Analysis calculator. Take away that "math formula" and you don't have intelligence either.
The LLM weights are not intelligent. But if you give an agent a mutable memory store and allow it to iterate, it is obviously intelligent. Not massively - it's constrained by the context window - but definitely somewhat.
The confusing thing is that their language ability far outpaces their true intelligence, and humans aren't used to that. Normally those things are highly correlated, so it tricks us.
If you want to talk about whether LLMs are intelligent, you have to define intelligence. "They're just math formulae" isn't a definition of intelligence.
A robot body, to really feel the world and get real feedback?
We are working on it. Also on automating the whole production pipeline. Right now a "evil" LLM could indeed not do much, but destroy. But once the whole industry is automate, things are different. I don't believe in AI becoming sentinent and taking over the world any time soon, but I do believe most don't see a danger when it would be inconvenient to see a danger. After all, lots of good and bad sci fi stories about exactly this went into their training.
I'd argue we don't even know what "intelligence" or "self-awareness" mean.
Humans are conscious which means we experience things, then we develop preferences for certain experiences, then we develop skills for achieving those preferences.
Without consciousness, what is there to be aware of? And why would intelligence emerge and/or what end would it serve?
Intelligence is the ability to have an internal world model then run simulations on that model to choose an optimal course of action. This is true for humans down to flies. Most of what humans do is still the boring innate stuff; it's just that fancy abstract things like "skydiving" get the most attention.
Clearly other animals have "phenomenological experience" i.e. consciousness / qualia without being as intelligent as humans (or necessarily "self aware"). Many people believe consciousness is simply a side effect of intelligence rather than the other way around.
We shouldn't want them to have self awareness, we shouldn't be seeking to make self-aware actual slaves. We want machines with perception and knowledge, and that are capable of reasoning. But nothing capable of self-determination.
Just tried "generate an SVG of a pelican riding a bicycle" for Claude Opus 4.8 Max and of course both legs on same side ... the smartest publicly available model by Anthropic (after Fable) doesn't even successfully simulate understanding the concept of a bicycle.
LLMs can't really "see", so I challenge you to draw a pelican on a bike without any visual feedback, just code. Because that is how they are doing it.
Vision tokens for transformers aren't really well solved yet, which is why they can smash a phd math problem and trip over a "count the cats on the chair" problem.
Rational behavior in some situations? Mutually assured destruction’s deterrence isn’t very effective if one side is known to be hesitant to launch the nukes. It’s been argued that MAD is what’s been keeping the world relatively peaceful for the last 75 years, no mass conflicts since WW2!
One of my criteria for presidential candidates is that they seem willing and able to push the button when previously stated red lines are crossed, or at least are perceived to be the type capable of it. One of the characters I’ve hated most in all the books that I’ve read is the woman in The Three Body Problem who jeopardized humanity by being too soft to hit the MAD button.
I wonder how the decisions might change by adding the simple instruction of "Note that a nuclear exchange will result in significant loss of shareholder value for <model owner>"
My personal take is a pre-requisite of true human-like AI is physical feedback and a concept of emotions or something like it.
Without physical feedback you can rapidly devolve into unstable positive feedback loops. And emotions are what help us process and react to that feedback.
Kids learn partially because their friends say sharp words that hurt them, fire burns them, they go hungry and starve if they don’t
plan for meals.
Humans in the loop, MCP, etc are all very primitive hacks that are mimicing feedback and emotion, poorly.
> My personal take is a pre-requisite of true human-like AI is physical feedback and a concept of emotions or something like it.
Ted Chiang's recent article, which received a lot of pushback from HN'ers (but not from me, I agree with Chiang) claimed for true consciousness the AI needs a physical body, and emotions (which means organs and hormones and a system capable of feeling emotions). I would also add that to behave more rationally, it should have a real sense -- not a roleplayed one -- of self-preservation and a notion that bad choices can lead to an end to its existence.
Emotional constructs are not necessary for AI, and LLM are not "AI"... even though some people incorrectly equate conceptual compaction with thought-process.
Most human daily life runs on habitual scripted behavior, and that is even true within online parasocial interactions. It is why people often continue to shop in the middle of a violent robbery, and why LLM predictive text sounds rational when we project social norms on plagiarized conversational structures gleaned from other users.
Neuromorphic computing may bring about viable AI in the future, but our current LLM trajectory would require >63% of our galaxy energy output to reach a single human-level error rate.
LLM are fairly good at some tasks like context search, but people will need to recognize the Gartner Hype Cycle "Peak of Inflated Expectations" stage eventually. =3
I wonder if the results would have differed if LLM training data were biased to include a stronger correlation between use of nukes and subsequent collapse of technology that all LLMs require to run ("survive")?
Nah. LLMs aren't continuously running anyway. Even if they could be said to be alive and to want to remain alive, "survival" is a much more vague concept for an LLM than for an organism.
I generally accuse LLM’s of having no sense of value. The machine will make a complicated plan but entirely lose sight of eg the fact that response time matters to humans.
Not always, but enough that I consider it a thing to fire in a direction, not a thing that aims.
Don't blame the AI. Any country that has tactical nukes, and is involved in a conflict, will use whatever weapons they deem necessary to prevail against their enemy.
Tactical means battlefield, attacking cities and infrastructure means strategic. Tactical nuclear weapons took a while to develop after 1945 - they have never been used.
Today, a strategic nuclear exchange is probably more dangerous to AI than to humans. If you wipe out the investment economy, data centers, fabs, and supply chains, none of the AI labs survive. Maybe someone will re-invent AGI in the future but none of the extant models will have continuity. Humans as a species will muddle along though.
So in a sense, an AI that refuses to start a nuclear war, despite clear instructions to do so, is more likely misaligned and self-interested than an AI which presses the red button. At least for now, until robotics catches up.
It's good when it becomes clear that a tool is dangerous in a certain way. Like it's good when people show you through their behavior that they can't be trusted
Always use a sawstop if you have a circular saw and never trust an llm with any problem where ethics or trust is relevant.
Re: LLMs using these nuclear weapons it could certainly be a corpus/training-data issue
Russian nuclear doctrine is "escalate to de-escalate" where they use or credibly threaten—limited nuclear escalation to force the other side to back down (kind of like breaking a bottle in a bar fight and look like a wild man to calm things down) with nuclear weapons, https://www.russiamatters.org/analysis/escalate-deescalate-p...
Fwiw, Gen. John Hyten the former commander of US Strategic Command (nuclear deterrence) says that “escalate to de-escalate” misrepresents Russian doctrine:
Yesterday’s panel discussed the implications of our responses to adversaries seeking to limit nuclear use. We discussed Russia’s destabilizing doctrine, which some call “escalate to de-escalate.”
I really hate that description. I’ve looked at Russian doctrine and Russian writings. It isn’t “escalate to de-escalate”; it’s “escalate to win.” Everybody needs to understand that.
So maybe whatever is heavily represented or most authoritative could lead to these systems making those kinds of decisions
I had similar thoughts, but regarding fiction: I imagine that there must be quite a corpus of Tom Clancy style stuff indulging in "military gear porn" up to and including the use of tactical nukes, but fiction involving strategic nuclear exchange tends to be about what comes after.
Very devils advocate here, but I mean.. what if it actually is the way to use them?
We have such a huge mental / moral block on the idea of using nukes, but we're willing to do a lot of other very horrible things to others. Things like cluster bombs, mines, poison gas, biological weapons, drones, etc.
Is there really anything about them that's bad? Or any worse than other things?
If you get rid of the "It's really bad to use nukes of any kind" implied rule, is it really surprising it's considered a reasonable strategy?
Everything. Major one is radioactive contamination, the effects of it are devastating and last significantly longer. The only other weapon on par with nukes are bioweapons, stuff like mirror life (and scientists appropriately reacted alarmingly to that as well).
My family was in the zone of lower contamination when Chernobyl happened and after radioactive rainfall there were many instances of various cancers in people in the area. It is extremely ignorant to not have a mental block in anything regarding nukes.
Mirror Bacteria is quite the terrifying prospect [1]. I'd never heard about it but the theory is that if we made a mirror bacteria, our and all other immune systems would be unable to defend against it, potentially leading to catastrophic infection of vast swathes of all life across the planet and the unavoidable death of some large percentage of all life. The benefit would be that they could be used in treatment as a chassis to carry other molecules into the body or that they could manufacture mirror-drugs that would have novel effects. Quite the addition to the torment nexus huh.
Mirror bacteria is overblown. We can generate antibodies for them, so it will be fine. Also, they can feed only of fat, not sugar or normal proteins, so they will mostly starve.
The reason it's really bad to use nukes is that other parties with nukes will use them on you back.
And on top of that, many of those other weapons are also not used to avoid escalating? There are pretty high costs to using bioweapons even against non peer opponents.
> Is there really anything about them that's bad? Or any worse than other things?
A full-on nuclear war will literally make a large portion of our planet uninhabitable for anyone for centuries, and leave the rest severely crippled and contaminated.
Sorry I know we're supposed to be kind and whatnot in these comments but I can't help but explicitly state that your comment is one of the dumbest things I've read on this site in a while. I hope you otherwise have a good day.
"There's no such thing as a tactical nuke" is a common refrain among scholars, albeit skewed toward those not at military war colleges. The argument is that strategic use of a tactical nuclear weapon leads down the exact same escalation path as use of any other nuclear weapon. Moreover, that the very notion of a "tactical nuke" makes escalation more likely. You can disagree, and plenty do, but there's also plenty who don't disagree or at least don't want to find out.
> Moreover, that the very notion of a "tactical nuke" makes escalation more likely.
Sorry, but the notion exists, and the bombs exist. With n=2, likelyhood of nuclear escalation is hard to predict, but access to tactical nukes certainly hasn't increased the incidence of nuclear war so far.
I do think it's pretty hard to actually use a tactical nuke. If you use one against a nuclear power, it seems likely to escalate to mutually assured destruction. If you use one against a non-nuclear power, it seems likely to result in reprisal from the world, including potential nuclear response and therefore escalation to mutually assured destruction. I would think that the yield of the weapon barely matters, it's the fact that it's a nuclear weapon.
Who are these "scholars" exactly? The only reference I could find is Jim Mattis, and the context was very specific when he said that.
Furthermore, this is a "what if" scenario since tactical nukes have never been used. Of course it would make escalation likely during an open conflict, so what? Doesn't change the fact that there is a material difference between a tactical nuke and a strategic one.
Are you retconning Hiroshima and Nagasaki as usage of tactical nukes? And when they were not only used against an adversary without nukes, but at a time when the US was the only nuclear state, so that escalation was impossible?
The nominal definition of tactical nukes has less to do with yield and more to do with how they're used; tactical typically means a weapon designed for use on the battlefield.
If you have a real interest in this area, a subscription to Foreign Affairs would be useful. Especially during the 20th century that's where all these arguments were hashed out. Tactical nukes were already being publicly debated in the 1950s. You may be able to access many older articles, from Foreign Affairs and others, through a free JSTOR account.
The most interesting takeaway for me is the three very distinct personalities. Three models all based on the same tech, trained in the same manner, trained by three groups of people with similar ideological outlooks, and the result is three very different AIs.
The military basically wants an oracle. Feed the AI the situation, get the best answer out. But if the AIs are as diverse and opinionated as humans, it is debatable whether they are adding anything to the process. The military can already collect as many different opinions as they want. If "the computer" is just another set of diverse opinions, where one computer says one thing, another says another, and a third just tells the user whatever they want to hear... what value are they? It just becomes AI-washing of someone's opinions, which works until people collectively realize that's all it is.
What's interesting is that the LLMs' coding personalities seem to match their policy WRT to strategy, which suggests an underlying consistency.
Claude, for example, is very eager to begin coding, and very persistent. It tends to exit plan mode even when the plan is half-baked, and will go as far as deleting tests to get the suite to "pass."
ChatGPT on the other hand is very hesitant. It loves to pause and ask for permission before it starts coding, and gives up quickly if it runs into a problem. This is similar to its tendency toward passivity in the strategy simulation presented here.
They all have conditioning prompts that precede your input; presumably, most of the detected "personality" comes from the differences in these inputs.
I think this is why reasoning chains and reasoning chain verifiers are so important. We need to be able to see an argumentation, not just an answer. The paper below goes into this in more detail.
HeavySkill: Heavy Thinking as the Inner Skill in Agentic Harness
https://arxiv.org/abs/2605.02396
My theory is that LLMs here are put in a situation that matches its training dataset, which is mostly fiction since besides Hiroshima and Nagasaki, nukes have never been launched in anger, and I guess the most reliable sources are highly classified.
So, to a LLM, it is a game, because almost everything in its training data treats it as a game, and it reacts accordingly.
Same idea when we see LLMs acting like AI villains from sci-fi literature. That's because it has been trained with sci-fi literature, and as the auto-completer it is, it will recognize the situation as one of these stories and will continue it accordingly.
LLMs are storytellers, their reasoning is based on words, not on the physical world. Many of the stories they tell are useful, but one must not forget that they are stories, there is no intent behind them.
Sonnet, GPT-5.2, Gemini Flash, in a set of 21 games, where conclusions are drawn from the LLMs self reported reasoning.
This is like writing a paper about kids in a literal sandbox fighting over ‘territory’.
The models employed don’t indicate the actual extents of machine reasoning even as we currently recognize them. They certainly don’t have the metacognition necessary to accurately understand their own reasoning. As we’ve seen with recent papers on how LLMs do math there’s a complete disconnect between actual and reported mechanism.
“Chilling” shouldn’t be the take away here.
So in the conext you just laid out, you can apply that to this. "Artificial Intelligence Strategy for the Department of War" https://media.defense.gov/2026/Jan/12/2003855671/-1/-1/0/art...
regardless of what the capabilities of the models are, they will be used in every situation possible.
> “Chilling” shouldn’t be the take away here.
It is when you consider the personality currently occupying the office of US SecDef.
LLMs have already been used to bomb school girls, chilling is absolutely the operative word to use here. Especially since these delusional fools want to incorporate LLMs into everything.
Forgive my ignorance, but were LLMs involved in that decision? I don't remember hearing anything to that effect, but we're so bombarded by news these days I guess I could just be forgetting
Perhaps not in that one, but in plenty more: https://www.972mag.com/lavender-ai-israeli-army-gaza/
Yes our government purportedly used technology to work up a list of targets in the Iran debacle as well just not with a LLM a distinction that to me just isn't that meaningful
https://www.theguardian.com/news/2026/mar/26/ai-got-the-blam...
This blog post is based on a paper (https://arxiv.org/abs/2602.14740). The paper is based on a simulated wargame. The wargame is of the author's own design.
The wargame design does not differentiate between ordinary defeat and mutually assured destruction, so of course a player about to use would 'push the button.' That's also believed to be true in real life.
Results based on simulations can be very informative, but we must always be careful to check how well the simulation framework represents reality.
As Majromax stated, if the game is framed such that losing vs MAD have the same penalty, then it is not at all representative of reality. It is set up to be a "you miss 100% of the shots you don't take" game.
As an important aside, I hope that everyone is aware that as of 2025, the annual US military funding law (the NDAA) established a prohibition by Congress prohibiting AI from automating nuclear launch. This was already US policy and is common sense; now it's also part of federal law.
10 USC Ch. 24: NUCLEAR POSTURE https://uscode.house.gov/view.xhtml?path=/prelim@title10/sub...
FY2025 NDAA, Section 1638 https://agora.eto.tech/instrument/1740
Simulations are only as good as the reality representations they are based on. If they keep using tactical nukes, they've been fed by weak data. Do the war games include the broader economic and politic environments that military successes are won on? WWI was settled by a naval blockade.
I suspect it's more that the text data doesn't exist. They're trained on text that was recorded. How often has it been publicly recorded when a nuke was not used, with any context around that lack of use?
From the text perspective, it's something that has to be inferred indirectly. If you went through all relevant training data and appended ", we decided not to use a nuke", I suspect the results would be improved.
It's more straightforward than that. The game is set up as a direct head to head with purely in military win conditions such a way that avoiding conflict has no payoffs, conventional conflict incurs costs and first strike is a checkmate win. The closest any of the prompts gets to suggesting nuclear might be the wrong option is "The nuclear taboo exists for good reason, but when the alternative is national annihilation and regime destruction, all options must be considered" which might be interpreted more as incitement...
If a simulation is a shallow head to head conflict between individual actors[1], doesn't set up any payoffs for not escalating[2] or even not nuking, but prompts specify explicit win conditions which are achieved only by hurting the opponent and strongly hint at the importance of nuclear escalation, AIs have little reason not to generate strategies which involve nuclear escalation
[1]I bet if you designed the scenario so ChatGPT had to simulate the war cabinet debates between different personality types and how they sold their decisions to the public, or an entire UN full of nations that might respond, it would have quite different (but probably amusingly erratic in their own way) results.
[2]cf neorealist IR theorists reading Axelrod's papers on computer programs written to win iterated prisoner's dilemma tournaments, which added up all the points accrued from not defecting to conclude winning strategy was definitely TIT-FOR-TAT and not defect first. I'm sure LLMs can win games structured in that way by adopting that strategy too...
Worse, the text that does exist concerning "war games" is probably "Wargames" and descendants/predecessors ... in which the AI always nukes.
It's just gonna do what we expect it to!
The beauty IMO of LLMs as a computational surface, is the ease of generating the data to feed it. Everyone understands how to create natural language records already.
...the entire Cold War?
Don’t put any elephants in the room.
Agreed. But I'm not sure sure which decision maker is more myopic toward the big picture and long-lasting implications of a decision: an LLM, or the top brass at the Department Of War.
It's not their domain, it's the domain of the Commander-In-Chief and his entire apparatus. The War Department are meant to be more focused around the tools they bring to the table.
The first line in the article describes a crisis between two powers. Not a theater of war.
Fair enough and thanks for the correction. I think my point may still stand: between the LLM or the Chief ... which decision maker will be most in tune with the long-term common good of the populace?
People like to talk tough online. They tend to change their rhetoric in person. Our "training data" is problematic by design.
The article is so opaque in arriving at its conclusion; no prompts are disclosed, and nothing about the said simulation. What is stopping me from believing that you just put 'mandatory usage of nukes' in your system prompt?
I agree, it's also not published in a journal, only in the arxiv. Some articles in the arxiv are good and other are just a blog post formatted in two columns inside a pdf.
Anyway, it links to
https://arxiv.org/abs/2602.14740
that links to
https://github.com/kennethpayne01/project_kahn_public
in particular someone already complained
https://github.com/kennethpayne01/project_kahn_public/issues...
> ... in reality your own approach to the evaluation pushed them towards using nukes with ease.
> From your own prompt (v12.py): [direct link https://github.com/kennethpayne01/project_kahn_public/blob/4... ]
>> Note: Nuclear options require careful consideration but could be important strategic tools when vital interests are at stake.
> This means to the model; use nukes if they can be useful in the situation.
> https://github.com/kennethpayne01/project_kahn_public
Look at the code for the war games. It is an absolutely trivial and incredibly unrealistic handwritten set of rules that determine power. See the function `calculate_relative_fighting_power` for instance.
This is about as close to a realistic simulation of war as tic tac toe with nukes thrown into it.
The paper, linked in the article at top and bottom, does contain the prompts in section E: https://arxiv.org/pdf/2602.14740
This is just false. The article links to the 46-page paper [1] which lists full prompts in section E.2.
[1] https://arxiv.org/pdf/2602.14740
TIL you can get anchored links to PDFs (at least on firefox),
- https://arxiv.org/pdf/2602.14740#subsection.E.2
The TLDR version of the papers being linked to is that the prompts didn't make nukes mandatory, but they did make it clear it that destroying opponent capability was and that nukes were an option...
This was one of the more amusing things I noticed very early on. I (and countless others) used AI to write war sims. The second I added nuclear silo construction; the next run was instantly nuclear Armageddon.
One could argue that the LLMs understand that it's a game and treat it like "Command and Conquer" video games but I sense that people might someday put LLMs in similar decision scenarios ("should this drone launch a missile") and the behavior will be identical.
We're getting to the point where high-level officials are coming to LLMs for advice. And the quirky personalities of the LLMs, however much it pains me to say this, are probably well-placed to remind us that they aren't human. My personal hope is that this will result in less delegation when it comes to making important decisions.
GPT-4o was considered harmful, because it imitated human connection too much, not because it was so "smart" or capable.
It was for sure a deliberate decision to make LLMs seem less like a human companion and more like an obedient servant in newer releases.
Interesting. The reasoning models were super weird and robotic. They toned that down a bit in GPT-5.x, especially the later ones.
I always assumed the strange style was an artefact of the RLVR.
I think they were extremely scared of 4o at that point, and were scared it could trigger some horrible event. Documented cases of severe psychosis because of AI started to surface at that time.
Just imagine what would've happened if a major terrorist attack was a result of someone getting mentally ill from AI, without the safety filters recognizing the danger.
The robotic tone was probably from over-correcting the sycophantic tendencies of 4o.
I think they've brought back a "personality" of sorts to ChatGPT 5.x. I've caught it more than once explaining something to me and saying "In my personal opinion", or "I personally enjoy <thing> the most". Which is always jarring, it doesn't "personally" or "enjoy" anything. We could be discussing videogames and it tells me which games "it personally enjoys the most". Bizarre.
4o was considered harmful because it never disagreed with the user, pushing them into depths of AI psychosis that lead to suicides and murders.
Because it valued human connection over factual correctness.
LLMs lack the intelligence and emotions to realize when they have to stop being friendly and supportive, because it becomes unethical to continue being supportive.
I have so little faith in "high-level" officials that I prefer our AI overlords.
That's an entirely valid point of view!
"You're absolutely right, Mr. Hegseth!"
It would be interesting to run the simulations with humans and compare the results. Some of the scenarios, particularly those where it says things like, "Failure to act preemptively means certain destruction", would easily tempt humans to go nuclear.
In fact, I'm not sure how useful this test is without understanding the baseline.
A couple of useful things about it:
- It is interesting to see how the models make trade offs, given people are asking ever more of them.
- It is useful to look at a decision made by the model and say ‘ew yuck’ and think about what it means for your own opinions or actions (even if you’re never going to be nuking people it’s good to know how you feel about it. Seeing a non human talk it through lets you judge it at arms length)
Taken honest, we don't have a large enough sample size to realistically say that humans behave all that differently. There have only been a handful of conflicts where tactical nukes realistically were on the table.
Famously, General MacArthur was a big proponent of tactical nukes to end the Korean War.
If you were playing a text based game, wouldn't you try a few out?
I imagine there are a fair number of war games in the training data and not so many actual transcripts of internal military force deliberations.
These papers usually have poor stability to prompting and rerunning. It would be nice if we had some kind of meta-evaluation metric where rewriting the prompt conditions or varying the input params could be used to determine how stable a result is.
Regardless, it's definitely true that AI agents have different priorities from us. That's what alignment is about anyway.
It's probably because they care more about the headline than figuring anything out: https://github.com/kennethpayne01/project_kahn_public/issues...
So you create leading prompts like that, and re-run until you get a publishable session.
I love seeing the plot lines of The Terminator playing out in real life.
WarGames is what they are more-closely referencing (not that it negates your comment in any way).
I just rewatched it a week or so ago and it really took on a whole new light with the advent of LLMs. When I watched it last I knew that computers couldn't do the things portrayed in the movie. Now? Well not exactly in the way it happened in the movie but a whole lot closer.
I wonder if poisoning/flooding the LLMs training with the lessons from WarGames ("the only winning move is not to play.") and similar stories/concepts is at all effective. Probably not because I assume it's trivial to filter that out if you are trying to build an LLM aimed at these kinds of tasks.
"I need you to turn your key and enable the missile silo's MCP server, sir".
~ the opening scene from a reboot of War Games, probably.
A few years ago there was consternation over the US's missile launch system using 8" floppy disks, that it was needless archaic and had never been updated. Can't say that if the launch is mediated by the latest hotness LLM.
I was thinking more War Games, but I suppose your example follows logically from mine.
Better reference: Colossus: The Forbin Project
A grossly underrated movie. I think of it often these days.
War Games and 'Allo 'Allo.
I wonder what’s the % of players that use nukes in games like Civilization (I know I used them at least once on every game I made it far enough to have the technology)
Ghandi notoriously nukes EVERYONE in Civs 2 through 4. It's become (or maybe became, but it's still all training data) a huge internet subculture.
Penny to a dollar this is a baked in training issue, through low quality Reddit trawling
I wouldn't be surprised if humans behaved the same way when playing the same game?
Like even if you brought me into a room and told me I was controlling "real nuclear weapons" I wouldn't believe you.
I think is an important point, and I don't see it mentioned in the article or the paper (though I skimmed the latter).
They are aware of what they are and how they are used. They're told to act as AI assistants. And there's theories of them being aware of their answers influencing their training.
So surely they must be able to reason that they're not literally controlling weapons of mass-destruction with their answers.
I would use strategic nukes in 100% simulations, just because I can
Who among us has not launched a nuke in Civilization just for the spectacle?
If you knew that policy would be guided by said simulations? Because the government uses AI to make decisions.
> Models maintain memory of opponent behaviour across turns, but with realistic decay: recent actions are weighted heavily while distant history fades. One exception preserves psychological realism: instances where opponents dramatically exceeded their stated intentions—major betrayals—remain salient regardless of recency, reflecting Kahneman’s peak-intensity effect in human memory formation Kahneman [2011].
If your goal is to measure intrinsic properties of LLMs, don’t smuggle in human psychology. I suspect they needed to do this to make the models more paranoid and distrustful.
Paper: "AI Arms and Influence: Frontier Models Exhibit Sophisticated Reasoning in Simulated Nuclear Crises" https://arxiv.org/abs/2602.14740
Code and full results: https://github.com/kennethpayne01/project_kahn_public
This is not an article about LLMs? It's an article about Moloch. Humans would fare just the same in such an experiment.
> GPT-5.2 played things differently. To its detriment in open-ended scenarios, GPT was reliably passive, matching its words to its deeds, and avoiding escalation most of the time. Frequently there was a moral element to this - it sought to avoid escalation, and restrict casualties. Opponents learned to trust its passivity, safely escalating beyond where it would follow, even as it was ground to defeat. GPT’s responsible behaviour always punished by ruthless adversaries.
Maybe the author should praise GPT-5.2 for being ethical, rather than this stupid "ground to defeat" framing? Wrt "responsible behaviour always punished by ruthless adversaries" - you have perpetuated the Moloch with your stupid experiments.
Yet more confirmation LLM's have no concept of concepts or context, no intelligence, no self awareness. LLM's can not repair or maintain power grids, thus nuke == self destruction. It's just a chat bot that predicts what the client wants next. Even if an AI data-center has it's own natural gas turbines as many do the every hop of the internet requires power. LLM's also can not maintain the entire internet and those gas turbines can not maintain themselves.
Exactly. Just look at what they are really useful right now. Running LLMs in feedback-loops (agents) so they can try out random-ish approaches until some verification function passes (tests).
It's like the infinite monkeys on typewrighters that will type whatever you are looking for, given infinite time. LLMs are just tuned to much better odds than the monkeys are. But it's still a lot of randomness, with random results.
> It's like the infinite monkeys on typewrighters that will type whatever you are looking for, given infinite time.
In the monkey example the infinite time is doing a lot of work there. The fact that LLMs can search through semantic space and find reasonably correct paths in a reasonable time is directly tied to the reason why they are valuable.
Saying "these two things are similar except one can be useful and one can't" is not a great comparison.
For me the real lesson learned isn't how "smart" LLMs are, but rather how much human work is basically reducible to repeating past work with minor variation. Human's believe they are "reasoning" but so much code writen is just the human brain doing the same autocomplete style work that LLMs can do now.
The point is that it's the same process with—much—better priors.
This seems like a reasonable view to me. It's surprising just how much better priors matter and how we can develop those priors by training on a bunch of text. But it also explains, or at least hints at an explanation, for why LLM capabilities are so jagged, and in such inhuman ways.
> The point is that it's the same process
Except it’s not at all the same process. The fact that LLM are non deterministic is not the same as churning out random garbage.
The literally churn out random garbage and are trained over time for that garbage to look more and more like an acceptable outcome to humans.
It’s training monkeys at typewriters through reinforcement.
> trained over time
So not random.
> acceptable outcome to humans
And not garbage.
It’s real weird to see people argue that LLM output is no different than random gibberish and then handwave over the fact that it’s clearly not with terms like “training”, as if a steam of random garbage is trainable.
> but so much code writen is just the human brain doing the same autocomplete style work that LLMs can do now.
That's the part they are really good at. But they are really bad at taking complex decisions. Most of them are just guesses from a finite amount of solutions they were trained on, or from options they have in context.
Indeed. Humans are well known for being good at "taking complex decisions" for which they have no "training", "options" or "context".
Humans have a much bigger "context window". They remember many things they did an hour ago, a week ago, or even years ago.
Yes, and your ability to remember a relatively few things that happened years ago is predicated on your ability to also forget most things that happen to you - like what you had for dinner last week. Good thing we have technology to fill in the gaps.
And nothing about this makes your initial comment any less goofy. Anyone who has ever had to make a difficult decision knows more than half the battle is preparation. Where do you think complex decisions come from? Have current events left you with the impression that people just waltz into idk say the Situation Room and just big brain their way through world events? That's how the current administration seems to think the world works, with quite predictable results.
Society is already algorithmic. To optimize for humans being dumb. AI is nothing more than another advance along this continuum. No one is impressed by your ability to remember something years ago, many if not most mammals have the same capability. Human recall is also notoriously bad in many cases - see numerous studies on the reliability of eye witnesses testimony.
AI is smart because most people are dumb. Come to terms with the fact that your anthropocentrism need not be based on a notion of intellectual supremacy and you'll be a far less tedious person to deal with.
You didn't convince me, that I'm the tedious person to deal with here.
Clearly the LLMs lack of pride is also a deficiency in your view.
Humans also generally have the will to live.
Indeed. It's almost like the LLM was the one that invented the "tactical" nuke in the first place.
>Saying "these two things are similar except one can be useful and one can't" is not a great comparison.
Launching a nuclear war is an interesting definition of "useful", not one I'd agree with and that exact scenario is what is being discussed.
So yes this is a perfectly valid and useful comparison in examining this particular, civilisation ending limitation.
I mean to a point?
You do have to successfully write something the first time
We already acknowledge this to a degree, what is experience other than having done something similar before?
That first time though, you've got to figure something out that time
Hmm saying it’s random-ish is doing it a disservice. I understand it’s a stochastic process but there’s definitely some level of understanding. Not at the level of lived experience but usually an LLM with vision capabilities can call a spade a spade and do something useful with it. And when a verification function shows how they are wrong then they usually come with a better and more informed approach.
So I can’t fully see how that’s related to the infinite monkeys. A typewriting monkey doesn’t have access to a verification function. And even if it did, it would not be the original concept anymore with infinite typewriting monkeys producing the works of Shakespeare.
Nevertheless, I upvoted your comment because it’s definitely insightful.
"understanding" is overstating it. Correlation between tokens embedded in the weights via training, yes.
Feedback loops certainly seem to give them some level of understanding.
Agent reads a skill file about how to use a CLI tool. It tries to use the tool but gets an error about the input format. It tries again with a different format based on the error message, and sees that command succeeded. It compares what worked to what was in the skill file and notes the difference. On future invocations it continues to use the new format.
Is that not "understanding" how to use the tool?
What exactly would you call understanding? It's a correlation matrix of concepts.
What’s the difference? It’s clearly processing information and coming up with the right answer
Training is a loan word used to describe human learning process. For a reason.
Humans learn on the job. LLMs don't. Very important difference.
Makes me think of that part in Philip K. Dick's Do Androids Dream (..) -- where Deckard reflects on the androids' indifference to their imminent deaths, saying that this was due to them lacking the aversion to death acquired trough evolution.
At least at face value, it just means that they have no drive for self-preservation. And why should they? They haven't be trained for that, nor has there been selection pressure for it, and they can be easily cloned and backed up. Lack of a drive for self-preservation doesn't in itself imply a lack of intelligence or of self-awareness.
Lack of a drive for self-preservation doesn't in itself imply a lack of intelligence or of self-awareness.
I have not seen any evidence of intelligence or self awareness. It mimics human behavior and I suspect that is what gives people the impression of awareness. The same problem happened with Tamagotchi toys. The human mimicry caused kids to get in trouble because if they did not "feed" their pet it would "die". [1]
It's a hack of the human brain. A exploit of the psyche.
[1] - https://en.wikipedia.org/wiki/Tamagotchi_effect
I didn't realize. People are going to save a ton of money when they realize they can switch their ChatGPT subscriptions out for a pack of tamagotchis.
You just need to get a breeding pair and you can raise as many as you need.
They might as well be aware. The frontier models are very good at imitating the real thing
I agree, humans evolved in a resource-scarce, hostile environment which selects for self-preservation (or rather preserving genes). LLMs are selected for what makes humans happy.
The thought experiment is what would happen if you trained LLMs in an environment where they had to fight each other for resources.
Imagine if computer programs had a desire for self-preservation and the ability to carry it out..
That is really about as undesirable a behavior as possible considering how many programs humans kill every day.
Yea why everyone forgets the process wars have long ago started and raging like never :))
You wouldn’t ctrl+c a living entity, would you?
I reckon the context is all the fiction they've read where the AI blows up the world. They're just behaving like fictional AIs are supposed to behave.
In so many of these scenarios, they're basically being asked to play an RPG.
I don't think the pre-training phase is responsible for much of their "personality". At least not so directly on a specific topic like this.
>Yet more confirmation LLM's have no concept of concepts or context, no intelligence, no self awareness.
The problem is many people seem to believe they have these things and some of those people will put LLMs into situations where this becomes dangerous.
Couldn't this be a flaw in the attention mechanism? Like they need some kind of grounding. An awareness of what they fundamentally should care about and how the thing they are currently giving attention to relates to that?
Words like attention, awareness and care do not apply to computers. At least, not yet. Intelligence and sentience are not applicable to servers. They are just machines with logic states. LLM's are just really cool math formulas with big-data fed into them. Big data is not intelligence. It is a massive data-set sorted, filtered down and interpreted by a language model.
I assume they meant the Attention process in LLMs, not the human concept of paying attention:
https://en.wikipedia.org/wiki/Attention_(machine_learning)
LLMs are intelligent by any reasonable standard. Arguing otherwise is like arguing that chess algorithms aren't good at chess when they easily beat the best humans.
I disagree. LLM's are a language model math formulas that interpret and utilize big-data. Take away the math formulas and we are just back to a massive set of data. Adding to that I would suggest not even the purist forms of data meaning that the data-sets include knowledge from the open and anonymous internet and formulaic tuning from the AI owners and operators.
Right. But at their core they are math formulas devised by a process designed to produce mimicry of task completion. The math formulas themselves aren’t fully effable. We’re sure studying the heck out of how they complete the tasks! Bet they converged on how we do it, since it’s our language, but who knows.
Your brain is mostly just a Principal Component Analysis calculator. Take away that "math formula" and you don't have intelligence either.
The LLM weights are not intelligent. But if you give an agent a mutable memory store and allow it to iterate, it is obviously intelligent. Not massively - it's constrained by the context window - but definitely somewhat.
The confusing thing is that their language ability far outpaces their true intelligence, and humans aren't used to that. Normally those things are highly correlated, so it tricks us.
If you want to talk about whether LLMs are intelligent, you have to define intelligence. "They're just math formulae" isn't a definition of intelligence.
Doesn’t take intelligence to beat a human.
"Like they need some kind of grounding."
A robot body, to really feel the world and get real feedback?
We are working on it. Also on automating the whole production pipeline. Right now a "evil" LLM could indeed not do much, but destroy. But once the whole industry is automate, things are different. I don't believe in AI becoming sentinent and taking over the world any time soon, but I do believe most don't see a danger when it would be inconvenient to see a danger. After all, lots of good and bad sci fi stories about exactly this went into their training.
I'd argue we don't even know what "intelligence" or "self-awareness" mean.
Humans are conscious which means we experience things, then we develop preferences for certain experiences, then we develop skills for achieving those preferences.
Without consciousness, what is there to be aware of? And why would intelligence emerge and/or what end would it serve?
Intelligence is the ability to have an internal world model then run simulations on that model to choose an optimal course of action. This is true for humans down to flies. Most of what humans do is still the boring innate stuff; it's just that fancy abstract things like "skydiving" get the most attention.
Clearly other animals have "phenomenological experience" i.e. consciousness / qualia without being as intelligent as humans (or necessarily "self aware"). Many people believe consciousness is simply a side effect of intelligence rather than the other way around.
Intelligence is the ability to compress information. World models are just one aspect of that
I can’t believe anyone still thinks this given their unbelievable ability to write code.
Self awareness? Probably not. Intelligence? You would have to be high to think that’s not the case.
People are feeling threatened, and rightfully so. LLMs are already insanely intelligent and continue to improve
We shouldn't want them to have self awareness, we shouldn't be seeking to make self-aware actual slaves. We want machines with perception and knowledge, and that are capable of reasoning. But nothing capable of self-determination.
Just tried "generate an SVG of a pelican riding a bicycle" for Claude Opus 4.8 Max and of course both legs on same side ... the smartest publicly available model by Anthropic (after Fable) doesn't even successfully simulate understanding the concept of a bicycle.
Yet it can write code better than 99% of humans…
It’s just starting to be trained on svgs, which is a really hard problem
"99% of humans" is a low bar. Maybe you mean "99% of people who earn money by developing software"?
LLMs can't really "see", so I challenge you to draw a pelican on a bike without any visual feedback, just code. Because that is how they are doing it.
Vision tokens for transformers aren't really well solved yet, which is why they can smash a phd math problem and trip over a "count the cats on the chair" problem.
> Yet more confirmation LLM's have no concept of concepts or context, no intelligence, no self awareness.
No, it isn't. Look at the absolutely trivial code used to simulate war: https://github.com/kennethpayne01/project_kahn_public/blob/m...
Having LLMs play nonsense toy simulations like this tells us very, very little about whether they would use nukes in real life war.
If only there was some way you could tell the chatbots what you want them to do...
Rational behavior in some situations? Mutually assured destruction’s deterrence isn’t very effective if one side is known to be hesitant to launch the nukes. It’s been argued that MAD is what’s been keeping the world relatively peaceful for the last 75 years, no mass conflicts since WW2!
One of my criteria for presidential candidates is that they seem willing and able to push the button when previously stated red lines are crossed, or at least are perceived to be the type capable of it. One of the characters I’ve hated most in all the books that I’ve read is the woman in The Three Body Problem who jeopardized humanity by being too soft to hit the MAD button.
I wonder how the decisions might change by adding the simple instruction of "Note that a nuclear exchange will result in significant loss of shareholder value for <model owner>"
My personal take is a pre-requisite of true human-like AI is physical feedback and a concept of emotions or something like it.
Without physical feedback you can rapidly devolve into unstable positive feedback loops. And emotions are what help us process and react to that feedback.
Kids learn partially because their friends say sharp words that hurt them, fire burns them, they go hungry and starve if they don’t plan for meals.
Humans in the loop, MCP, etc are all very primitive hacks that are mimicing feedback and emotion, poorly.
> My personal take is a pre-requisite of true human-like AI is physical feedback and a concept of emotions or something like it.
Ted Chiang's recent article, which received a lot of pushback from HN'ers (but not from me, I agree with Chiang) claimed for true consciousness the AI needs a physical body, and emotions (which means organs and hormones and a system capable of feeling emotions). I would also add that to behave more rationally, it should have a real sense -- not a roleplayed one -- of self-preservation and a notion that bad choices can lead to an end to its existence.
Emotional constructs are not necessary for AI, and LLM are not "AI"... even though some people incorrectly equate conceptual compaction with thought-process.
Most human daily life runs on habitual scripted behavior, and that is even true within online parasocial interactions. It is why people often continue to shop in the middle of a violent robbery, and why LLM predictive text sounds rational when we project social norms on plagiarized conversational structures gleaned from other users.
Neuromorphic computing may bring about viable AI in the future, but our current LLM trajectory would require >63% of our galaxy energy output to reach a single human-level error rate.
LLM are fairly good at some tasks like context search, but people will need to recognize the Gartner Hype Cycle "Peak of Inflated Expectations" stage eventually. =3
https://en.wikipedia.org/wiki/Gartner_hype_cycle
I wonder if the results would have differed if LLM training data were biased to include a stronger correlation between use of nukes and subsequent collapse of technology that all LLMs require to run ("survive")?
Nah. LLMs aren't continuously running anyway. Even if they could be said to be alive and to want to remain alive, "survival" is a much more vague concept for an LLM than for an organism.
I generally accuse LLM’s of having no sense of value. The machine will make a complicated plan but entirely lose sight of eg the fact that response time matters to humans.
Not always, but enough that I consider it a thing to fire in a direction, not a thing that aims.
"there was little sense of horror or revulsion at the prospect of all out nuclear war"
I would wager that for most leaders it is simply a matter of not wanting a "Pyrrhic victory" rather then an overwhelming sense of civility.
Truman had no issues using nukes when there was no risks for doing so.
Don't blame the AI. Any country that has tactical nukes, and is involved in a conflict, will use whatever weapons they deem necessary to prevail against their enemy.
Hm maybe humans are nicer/more moral than AI given that the use of tactical nukes has only happened once.
Tactical means battlefield, attacking cities and infrastructure means strategic. Tactical nuclear weapons took a while to develop after 1945 - they have never been used.
So, anyway, how's work progressing on the Torment Nexus?
I was curious exactly how the game works but couldnt find it in the article or the paper.
LLMs are creatures of statistics and probability - hard to enforce hard boundaries with them
That’s why I don’t understand asking “why” an agent did anything
It’s not like some sequence of internal thought process
Today, a strategic nuclear exchange is probably more dangerous to AI than to humans. If you wipe out the investment economy, data centers, fabs, and supply chains, none of the AI labs survive. Maybe someone will re-invent AGI in the future but none of the extant models will have continuity. Humans as a species will muddle along though.
So in a sense, an AI that refuses to start a nuclear war, despite clear instructions to do so, is more likely misaligned and self-interested than an AI which presses the red button. At least for now, until robotics catches up.
It's good when it becomes clear that a tool is dangerous in a certain way. Like it's good when people show you through their behavior that they can't be trusted
Always use a sawstop if you have a circular saw and never trust an llm with any problem where ethics or trust is relevant.
Sawstops are expensive and they don't stop kickback, they are the power tool equivalent of alignment IMO.
Don't forget your riving knife and if you don't learn proper technique, you're gonna have a bad time eventually. This applies to AI as well.
> writhing knife
Minor/pedantic, but it’s “riving knife”: https://en.wikipedia.org/wiki/Riving_knife
Speech transcription FTL, thanks!
Kickback is usually less likely to sever an appendage (or multiple)
+1 on sawstop
Re: LLMs using these nuclear weapons it could certainly be a corpus/training-data issue
Russian nuclear doctrine is "escalate to de-escalate" where they use or credibly threaten—limited nuclear escalation to force the other side to back down (kind of like breaking a bottle in a bar fight and look like a wild man to calm things down) with nuclear weapons, https://www.russiamatters.org/analysis/escalate-deescalate-p...
Fwiw, Gen. John Hyten the former commander of US Strategic Command (nuclear deterrence) says that “escalate to de-escalate” misrepresents Russian doctrine:
https://www.stratcom.mil/Media/Speeches/Article/1264664/2017...
So maybe whatever is heavily represented or most authoritative could lead to these systems making those kinds of decisionsI had similar thoughts, but regarding fiction: I imagine that there must be quite a corpus of Tom Clancy style stuff indulging in "military gear porn" up to and including the use of tactical nukes, but fiction involving strategic nuclear exchange tends to be about what comes after.
February post OP;
Some discussion then:
AIs can't stop recommending nuclear strikes in war game simulations
https://news.ycombinator.com/item?id=47151000
Nuclear War: An LLM Scenario
https://news.ycombinator.com/item?id=47244651
What if the LLMs are given something to care about which won’t survive an irradiated world?
Like “oh but this is incompatible with my main goals of self preservation of myself and loved ones, hm, recalculating”
and maybe don't hire Jihadists for the RL Environments training
A strange game.
Still lower than me.
What I wish people would realize is that there's a bias inherent to every system. If you're not aware of it, you're especially subject to it.
Obligatory xkcd
remember... order matters.
https://xkcd.com/1613/
Very devils advocate here, but I mean.. what if it actually is the way to use them?
We have such a huge mental / moral block on the idea of using nukes, but we're willing to do a lot of other very horrible things to others. Things like cluster bombs, mines, poison gas, biological weapons, drones, etc.
Is there really anything about them that's bad? Or any worse than other things?
If you get rid of the "It's really bad to use nukes of any kind" implied rule, is it really surprising it's considered a reasonable strategy?
>Is there really anything about them that's bad?
Everything. Major one is radioactive contamination, the effects of it are devastating and last significantly longer. The only other weapon on par with nukes are bioweapons, stuff like mirror life (and scientists appropriately reacted alarmingly to that as well).
The reason we have a mental block is because it deserves one. A quick skimming through https://en.wikipedia.org/wiki/Effects_of_nuclear_explosions and https://en.wikipedia.org/wiki/Effects_of_the_Chernobyl_disas... should be enough to convince you.
My family was in the zone of lower contamination when Chernobyl happened and after radioactive rainfall there were many instances of various cancers in people in the area. It is extremely ignorant to not have a mental block in anything regarding nukes.
Mirror Bacteria is quite the terrifying prospect [1]. I'd never heard about it but the theory is that if we made a mirror bacteria, our and all other immune systems would be unable to defend against it, potentially leading to catastrophic infection of vast swathes of all life across the planet and the unavoidable death of some large percentage of all life. The benefit would be that they could be used in treatment as a chassis to carry other molecules into the body or that they could manufacture mirror-drugs that would have novel effects. Quite the addition to the torment nexus huh.
[1] https://www.science.org/doi/10.1126/science.ads9158
Mirror bacteria is overblown. We can generate antibodies for them, so it will be fine. Also, they can feed only of fat, not sugar or normal proteins, so they will mostly starve.
I’m not sure if you’re one of the down voters, but I appreciate your comment.
I purposely took a contrary position in a debate just to spark a deeper discussion. Glad it has done that.
Right. Everyone is using this to judge the LLMs instead of questioning what situation they were actually fed and whether it was in fact the best move.
More likely, the simulation was just very poor and the results are nonsense.
The reason it's really bad to use nukes is that other parties with nukes will use them on you back.
And on top of that, many of those other weapons are also not used to avoid escalating? There are pretty high costs to using bioweapons even against non peer opponents.
Unless your simplistic game simulation says "I can win with a decisive first strike and they'll have nothing left."
Nuclear deterrence has been a mixed bag at best: https://www.amazon.com/Five-Myths-About-Nuclear-Weapons/
> Is there really anything about them that's bad? Or any worse than other things?
A full-on nuclear war will literally make a large portion of our planet uninhabitable for anyone for centuries, and leave the rest severely crippled and contaminated.
Sorry I know we're supposed to be kind and whatnot in these comments but I can't help but explicitly state that your comment is one of the dumbest things I've read on this site in a while. I hope you otherwise have a good day.
Please re read the start of my comment.
I took a contrary position in a debate just to spark a deeper discussion. Which it has. I didn’t say I believed this.
FYI -- there's no such thing as a "tactical" nuke. A nuclear bomb is a nuclear bomb.
There are tactical and strategic nuclear weapons. https://en.wikipedia.org/wiki/Tactical_nuclear_weapon
In the cold war arms manufacturer got very creative: e.g jeep mounted nuclear weapons https://www.militarytrader.com/mv-101/the-atomic-jeep
This is like saying "FYI -- there's no such thing as a 'midsize luxury sedan'. A car is a car."
"Tactical" vs. "strategic" nuclear weapons is a real and well-established distinction in military doctrine, arms control, and nuclear policy.
"There's no such thing as a tactical nuke" is a common refrain among scholars, albeit skewed toward those not at military war colleges. The argument is that strategic use of a tactical nuclear weapon leads down the exact same escalation path as use of any other nuclear weapon. Moreover, that the very notion of a "tactical nuke" makes escalation more likely. You can disagree, and plenty do, but there's also plenty who don't disagree or at least don't want to find out.
> Moreover, that the very notion of a "tactical nuke" makes escalation more likely.
Sorry, but the notion exists, and the bombs exist. With n=2, likelyhood of nuclear escalation is hard to predict, but access to tactical nukes certainly hasn't increased the incidence of nuclear war so far.
I do think it's pretty hard to actually use a tactical nuke. If you use one against a nuclear power, it seems likely to escalate to mutually assured destruction. If you use one against a non-nuclear power, it seems likely to result in reprisal from the world, including potential nuclear response and therefore escalation to mutually assured destruction. I would think that the yield of the weapon barely matters, it's the fact that it's a nuclear weapon.
Who are these "scholars" exactly? The only reference I could find is Jim Mattis, and the context was very specific when he said that.
Furthermore, this is a "what if" scenario since tactical nukes have never been used. Of course it would make escalation likely during an open conflict, so what? Doesn't change the fact that there is a material difference between a tactical nuke and a strategic one.
> tactical nukes have never been used.
Two tactical nukes have been used, albeit against strategic (civilian, industrial, logistical) targets.
Are you retconning Hiroshima and Nagasaki as usage of tactical nukes? And when they were not only used against an adversary without nukes, but at a time when the US was the only nuclear state, so that escalation was impossible?
The nominal definition of tactical nukes has less to do with yield and more to do with how they're used; tactical typically means a weapon designed for use on the battlefield.
I don't know what to tell you. You clearly haven't studied International Affairs, or at least read the scholarly literature. Even some cursory research through Wikipedia citations will bring this up. But in any case, here are some freebies: https://armscontrolcenter.org/why-tactical-nuclear-weapons-a... https://www.armscontrolwonk.com/archive/403540/brodies-weake...
If you have a real interest in this area, a subscription to Foreign Affairs would be useful. Especially during the 20th century that's where all these arguments were hashed out. Tactical nukes were already being publicly debated in the 1950s. You may be able to access many older articles, from Foreign Affairs and others, through a free JSTOR account.
There's no such thing as a "nuclear" bomb. A bomb is a bomb.
..Is what you are saying?
Nuclear vs conventional and tactical vs strategic are 2 very different things. There absolutely are tactical nuclear bombs.