Special Investigations

Press Freedom Defense Fund

AI Almost Started a U.S.–China War — and No One Seems to Cares

The tech world is too busy dreaming up imaginary doomsday scenarios to focus on a very real one that flared up between nuclear powers.

The global tech industry, politicians at home and abroad, and international media coverage are transfixed by one question: Will rogue AI end humanity? Few seem to be asking this question of the Department of Defense.

The artificial intelligence sector has rocked itself in recent weeks following a string of disclosures from the frontier AI labs and some of their personnel. OpenAI and Anthropic have both disclosed incidents in which their software, during routine internal testing, unexpectedly broke into the networks of other corporations. The tests were akin to a disastrous demonstration of a guided missile: After humans hit launch, the autonomous technology veered off course in deeply alarming ways.

Observers and industry figures quickly interpreted the incidents not simply as an indicator of the software’s power. Instead, they argued that it presaged an age of computers possessing something no computer ever has: bad intentions.

If computer systems could behave in ways that are unexpected — or even shocking — without being directly steered by humans, it seems only a short step until computers might have desires, ambitions, and even malice. If OpenAI’s tools could unexpectedly crack the servers of a rival corporation to accomplish a task posed to it by engineers, couldn’t it also surprise us with actions that result in harm to people, or even deaths?

Frontier lab employees and executives have answered with an emphatic yes. On September 8, Anthropic researcher Jacob Coxon announced his resignation on X, writing, “The people building AI earnestly believe that it could kill us all by the end of the decade. This is not a marketing stunt.” Coxon’s former Anthropic colleague Evan Hubinger replied, casually agreed: “Jacob is correct here — we really do earnestly believe AI could kill all humans!” Hubinger noted he personally puts the odds of the species’ extermination by AI, somehow, at over 10 percent.

Tech CEOs’ Doomsaying Is a Distraction from Real, Existing Harms of AI

This has all resulted, unsurprisingly, in a global panic, with a sudden flurry of calls for regulatory intervention, from legislation that would require an emergency “kill switch” for AI systems, to a proposal by Sen. Bernie Sanders to halt AI development altogether.

Throughout this period of alarm, how exactly a large language model could literally end humanity has remained vague. Some have speculated that this technology could foster the creation of some sort of novel bioweapon; others worry many thousands of AI agents could somehow hack all vital infrastructure simultaneously. In both those supposed doomsdays, the details are still fuzzy.

It should have come as quite the shock, then, when CNN reported on September 18 of a recent incident in which the use of artificial intelligence could have genuinely led to the extinction of the species.

It was under direct human supervision that a large language model almost misled the U.S. into instigating a war with his country.

It was under direct human supervision that a large language model almost misled the U.S. into instigating a war with his country.

This past spring, according to the news outlet, a U.S. Special Operations Command analyst used a large language model to generate an intelligence report that indicated “a Chinese ship in the Middle East was transporting components of a nuclear weapons program.” The finding prompted the U.S. military to make quick preparations to intercept the vessel by force. But this AI-generated intelligence, according to one source who spoke to CNN, was “entirely false.” Yet, the source said, it “almost started a war.”

The Dawn of a New Cold War

A shooting war between the U.S. and China could play out in an incalculably wide variety of ways. But one entirely plausible path would be an exchange between the world’s first and third largest nuclear weapons arsenals — an event that would transcend warfare into global cataclysm.

CNN’s reporting garnered attention, but was not followed by nearly the degree of sustained, grave concern of the AI safety news cycles that preceded it. It didn’t seem to prompt company scientists to question their careers, nor did it spur calls for regulatory intervention or self-imposed limits by the companies who furnish the Pentagon with this technology. After weeks of discussion of how AI could hypothetically kill everyone, the public learned of a concrete way in which AI really could have started a war that might have killed everyone, and the world quickly lost interest.

“This [China] incident and the lack of response is an attention problem that the nuclear field has been grappling with for decades.”

“This [China] incident and the lack of response is an attention problem that the nuclear field has been grappling with for decades.”

At a summit........

© The Intercept