We’ve been hearing a great deal recently about the specter of AI-driven human extinction. Maybe our misplaced trust in AI intelligence will set off World War III. Or AI agents will unleash widespread cyberattacks and take over the internet. Given too much autonomy, the machines will run amok. They may seek to deceive their creators—wait, that’s already happened. Suppose they conclude, like Agent Smith in The Matrix, that “human beings…are a plague, and we are the cure.” Or perhaps the chatbots help bad humans unleash a literal plague—a biological weapon.

That last one isn’t merely theoretical. A “misuse” report Anthropic made public earlier this month detailed five cases in which foreign scientists tried to evade Claude’s guardrails on “gain of function” inquiries. That’s a euphemism for when scientists tinker with a pathogen’s genes to enhance it in some way, making it more infectious, deadlier, or better able to spread among mammals, for example.

“Think about all the people—scientists, nation states, academic institutions—that are manipulating pathogens for greater virulence.”

Anthropic denied those users the answers they sought and cut off their accounts, even while noting that their motives may not have been sinister. Such information also can be used to help scientist develop pandemic countermeasures like vaccines and drugs. And there’s the rub. Like so many technologies, AI included, genetic engineering can be used for both good and evil. Hence another euphemism: “dual use.”

I studied to be a molecular biologist, and have been covering the gain of function debate since long before AI was on everyone’s radar. In my experience, researchers who work on high-risk pathogens tend to default to the view that it’s better to publish than not, and that we’re better off doing the work to understand which genetic traits make organisms a threat to humankind—never mind the potentially catastrophic consequences of an accidental release, or a deliberate one.

Is humanity really better off doing the work? I have my doubts. And I have more of them after reading the latest book from Annie Jacobsen, whom I last interviewed about her best-selling book Nuclear War: A Scenario. Her new book, Biological War: A Scenario, came out in late July and follows the same template. And because I know you were about to ask, yes, I’m sure she is great fun at cocktail parties.

One thing about Jacobsen, who has decades of high-level national security reporting under her belt, is her unrivaled access to current and former military and government officials and scientists—people who spend their days thinking about threats to humanity and gaming out how the government might respond if such-and-such horrific thing were to happen.

Beyond the notion of human extinction, perhaps the most terrifying aspect of both of her books relates to the limitations in our government’s preparation and response. America’s general plan for nuclear war was, and no doubt remains, complete madness. As for biological war, spoiler alert, it turns out we really can’t defend against it. (You’re welcome.) I’ve edited our conversation for length and clarity.

So, Annie, now that you’ve done both the nuclear scenario and the biological scenario, which is more likely to keep you up at night?

The Defense Department document that I reprinted in the book, on the weapons of mass destruction threat spectrum, lists nuclear weapons at the far end of the curve because they are the most destructive. But I was really surprised to see how biological weapons are a close second. And I believe the reason they’re the greatest overall threat very much ties into the discussion the entire world is having today, which is that the threshold for entry is extremely low, and therefore it is more likely to happen.

To have a nuclear weapon, you have to have weapons-grade fissile material, and then you have to think about containing those weapons within the arsenal. You have to have an ICBM, a B-2 bomber, a nuclear-armed submarine.

You’ve got to be a state!

Yes. A terrorist with a dirty bomb, that’s just a completely different threat. The component that makes biological war so incredibly dangerous and the greatest overall threat is that there are nine nuclear-armed states tasked with this containment component, but then you look at the rest of the world and who has access to [potential pandemic agents]…

Most everyone.

Yes. And that becomes terrifying.

If you were a bioweapons designer, what would you look for in a pathogen?

My mind does not think that way because I am a mother and I’m a member of society. But If I were a pandemic planner, what would scare me the most is the concept of an airborne agent and the public’s naiveté to that very concept. I would think about how important it is to educate the public about the reality of what is an extraordinary threat and what........

© Mother Jones