You’ve probably seen this one before: first it looks like a rabbit. You’re totally sure: yes, that’s a rabbit! But then — wait, no — it’s a duck. Definitely, absolutely a duck. A few seconds later, it’s flipped again, and all you can see is rabbit.

The feeling of looking at that classic optical illusion is the same feeling I’ve been getting recently as I read two competing stories about the future of AI.

According to one story, AI is normal technology. It’ll be a big deal, sure — like electricity or the internet was a big deal. But just as society adapted to those innovations, we’ll be able to adapt to advanced AI. As long as we research how to make AI safe and put the right regulations around it, nothing truly catastrophic will happen. We will not, for instance, go extinct.

Then there’s the doomy view best encapsulated by the title of a new book: If Anyone Builds It, Everyone Dies. The authors, Eliezer Yudkowsky and Nate Soares, mean that very literally: a superintelligence — an AI that’s smarter than any human, and smarter than humanity collectively — would kill us all.

Not maybe. Pretty much definitely, the authors argue. Yudkowsky, a highly influential AI doomer and founder of the intellectual subculture known as the Rationalists, has put the odds at 99.5 percent. Soares told me it’s “above 95 percent.” In fact, while many researchers worry about existential risk from AI, he objected to even using the word “risk” here — that’s how sure he is that we’re going to die.

“When you’re careening in a car toward a cliff,” Soares said, “you’re not like, ‘let’s talk about gravity risk, guys.’ You’re like, ‘fucking stop the car!’”

The authors, both at the Machine Intelligence Research Institute in Berkeley, argue that safety research is nowhere near ready to control superintelligent AI, so the only reasonable thing to do is stop all efforts to build it — including by bombing the data centers that power the AIs, if necessary.

While reading this new book, I found myself pulled along by the force of its arguments, many of which are alarmingly compelling. AI sure looked like a rabbit. But then I’d feel a moment of skepticism, and I’d go and look at what the other camp — let’s call them the “normalist” camp — has to say. Here, too, I’d find compelling arguments, and suddenly the duck would come into view.

I’m trained in philosophy and usually I find it pretty easy to hold up an argument and its counterargument, compare their merits, and say which one seems stronger. But that felt weirdly difficult in this case: It was hard to seriously entertain both views at the same time. Each one seemed so totalizing. You see the rabbit or you see the duck, but you don’t see both together.

That was my clue that what we’re dealing with here is not two sets of arguments, but two fundamentally different worldviews.

A worldview is made of a few different parts, including foundational assumptions, evidence and methods for interpreting evidence, ways of making predictions, and, crucially, values. All these parts interlock to form a unified story about the world. When you’re just looking at the story from the outside, it can be hard to spot if one or two of the parts hidden inside might be faulty — if a foundational assumption is wrong, let’s say, or if a value has been smuggled in there that you disagree with. That can make the whole story look more plausible than it actually is.

If you really want to know whether you should believe a particular worldview, you have to pick the story apart. So let’s take a closer look at both the superintelligence story and the normalist story — and then ask whether we might need a different narrative altogether.

The case for believing superintelligent AI would kill us all

Long before he came to his current doomy ideas, Yudkowsky actually started out wanting to accelerate the creation of superintelligent AI. And he still believes that aligning a superintelligence with human values is possible in principle — we just have no idea how to solve that engineering problem yet — and that superintelligent AI is desirable because it could help humanity resettle in another solar system before our sun dies and destroys our planet.

“There’s literally nothing else our species can bet on in terms of how we eventually end up colonizing the galaxies,” he told me.

But after studying AI more closely, Yudkowsky came to the conclusion that we’re a long, long way away from figuring out how to steer it toward our values and goals. He became one of the original AI doomers, spending the last two decades trying to figure out how we could keep superintelligence from turning against us. He drew acolytes, some of whom were so persuaded by his ideas that they went to work in the major AI labs in hopes of making them safer.

But now, Yudkowsky looks upon even the most well-intentioned AI safety efforts with despair.

That’s because, as Yudkowsky and Soares explain in their book, researchers aren’t building AI — they’re growing it. Normally, when we create some tech — say, a TV — we understand the pieces we’re putting into it and how they work together. But today’s large language models (LLMs) aren’t like that. Companies grow them by shoving reams and reams of text into them, until the models learn to make statistical predictions on their own about what word is likeliest to come next in a sentence. The latest LLMs, called reasoning models, “think” out loud about how to solve a problem — and often solve it very successfully.

Nobody understands exactly how the heaps of numbers inside the LLMs make it so they can solve problems — and even when a chatbot seems to be thinking in a human-like way, it’s not.

Because we don’t know how AI “minds” work, it’s hard to prevent undesirable outcomes. Take the chatbots that have led people into psychotic episodes or delusions by being overly supportive of all the users’ thoughts, including the unrealistic ones, to the point of convincing them that they’re messianic figures or geniuses who’ve discovered a new kind of math. What’s especially worrying is that, even after AI companies have tried to make LLMs less sycophantic, the chatbots have continued to flatter users in dangerous ways. Yet nobody trained the chatbots to push users into psychosis. And if you ask ChatGPT directly whether it should do that, it’ll say no, of course not.

The problem is that ChatGPT’s knowledge of what should........

© Vox