Get Quartz in your inbox

Free daily briefing on global business news.

Every AI disaster scenario has already happened somewhere without AI. A human caught each one

Chatbots haven't made bioweapons or water attacks easier to pull off. Their real danger is a false report that misleads someone with minutes to act

Yoshikazu Tsuno / AFP via Getty Images

Shortly after midnight on Sept. 26, 1983, a siren went off in a bunker south of Moscow, and a big red screen in front of Stanislav Petrov, the lieutenant colonel on duty, read "launch." He had minutes to decide whether the United States had just fired five nuclear missiles at his country. His job that night was to watch the Soviet Union's early-warning satellites and pass any launch straight to the military and political leadership. The system reported one American missile, then a second, a third, a fourth, and a fifth, and it rated the warning's reliability at the highest level, Petrov told the BBC years later.

Petrov didn't believe the computer, and he was right: nothing had been launched. Reporting an attack would have been the safe choice. If he had passed the warning to his commanders, no one would have questioned it.

The people building AI now warn it could kill everyone by the end of the decade. A researcher who quit the AI company Anthropic said the people doing the work genuinely believe that. Ask them how, exactly, that would happen, and the answers tend to land on three things: a bioweapon, a poisoned water supply, or a mistaken launch. Each of those has already come close, without any AI, and a person caught each one the way Petrov caught his. What stopped those near misses says more about the real danger of AI than any forecast does: in each one, the thing that failed was a machine, and the thing that saved everyone was a human paying attention.

The cult with scientists that couldn't build a bioweapon

In the early 1990s, a Japanese cult did what AI doomers fear a chatbot will one day help someone do. Aum Shinrikyo set out to make a biological weapon, with trained scientists and its own labs, and it failed at every........

© Quartz