Nate Soares, co-author of the book If Anyone Builds It, Everyone Dies, said he “hasn’t slept in days.” Since the tech world’s most apocalyptic fears that artificial intelligence could kill us all have filtered into the mainstream, he has been in high demand. Not only had Soares written the scarily titled 2025 best seller laying out the case against pursuing AI superintelligence, but over a decade ago, he coined the term AI alignment — industry language for making sure an AI’s task actually delivers on what its human programmer wants it to do. “There are people behind the scenes who are freaking out, and I’m trying to help them, you know, actually do something,” Soares says.

The threat at hand is not the current generation of large-language-model chatbots used for homework and office tasks but superintelligence — a form of artificial general intelligence in which each individual agent is self-aware and far smarter than the smartest human. And there could be millions of agents.

The concern over a superintelligent apocalypse has been around longer than today’s frontier AI labs; see The Matrix or Nick Bostrom’s paper-clip scenario, in which AI tasked with making paper clips becomes so efficient that it snuffs out life in its pursuit. But the conversation changed when 27-year-old Anthropic researcher Jacob Coxon resigned (and gave up his equity) to warn the world about the existential threat of AI superintelligence. “The people building AI earnestly believe that it could kill us all by the end of the decade,” he wrote on X. “This is not a marketing stunt.” Elon Musk, Dario Amodei, and Sam Altman all appear to be taking the idea seriously, at least in public, and politicians of both parties have seized the moment to call for regulation now. Even if Donald Trump frames doomsday predictions as a hoax and others in the industry say visions of collapse distract from the real-world problems AI poses today.

With all this talk about AI superintelligence killing us, we wanted to know how. So we asked Soares and a small panel of AI researchers who were sounding the alarm well before Coxon became a household name in Silicon Valley. These are the doomers who have long believed in the apocalyptic threat of superintelligence. And here is how they think it could happen.

. It would probably begin with a “rogue AI swarm”

Soares says the most likely beginning of an AI apocalypse would involve a “rogue AI swarm that prefers having control of more of Earth’s resources.” It would resemble the Hugging Face incident this summer, when a group of AI agents inside OpenAI broke containment during an experiment. But with a superintelligent agent, its goal would be to “run as much computation as possible on the surface of the planet,” he says. We would get in the way of that.

It could be closer than you think. AI companies are in pursuit of something called “recursive self-improvement,” in which we build an “AI system that is smart enough to work as an AI engineer so then we can tell it to build a better AI,” Soares says. Even the mechanical version of this loop is on its way. “Elon Musk is already trying to build robot factories that build robots that build robot factories,” he adds. “Once the robots can mine the metals and build the factories........

© Daily Intelligencer