Silicon Valley’s Favorite Prophet Has a New Warning

Nick Bostrom is no longer worried about the apocalypse. Now he’s worried about utopia.

Produced by Sophia Muniz

Nick Bostrom rose to prominence for predicting artificial intelligence doom more than a decade ago — garnering support from figures like Elon Musk, Sam Altman and Bill Gates. But now he’s retreating from those warnings, focusing his attention on a more complex question: What if A.I. succeeds beyond humanity’s wildest imaginations? Will we be bored out of our minds? Are we too infantile to understand what’s coming?

Guest host Spencer Klavan, the associate editor at the Claremont Review of Books, presses Bostrom on the potential uses and abuses of superintelligence, rationalist ethics and what makes us human.

Silicon Valley’s Favorite Prophet Has a New Warning

Below is an edited transcript of an episode of “Interesting Times.” We recommend listening to it in its original form for the full effect. You can do so using the player above or on the NYTimes app, Apple, Spotify, Amazon Music, YouTube, iHeartRadio or wherever you get your podcasts.

Spencer Klavan: Nick Bostrom, welcome to “Interesting Times.”

Nick Bostrom: Hi, Spencer. Good to be with you.

Klavan: I wanted to start by getting your assessment of where you think things stand. I think the technical term of art I want to invoke here is P (doom), the probability that everything goes terribly wrong.

A researcher at Anthropic recently placed this at around 10 percent or maybe even above. Is that about where you are? Do you have a different assessment? At this point, what do you think is the likelihood of catastrophe?

Bostrom: I don’t think my outlook has changed fundamentally. I see it more that the world has changed.

When I started working on the first book, “Superintelligence,” in 2008, at that point, the world was in a very different place. People were dismissing this whole thing as science fiction. And nobody, save maybe a handful of individuals, were thinking about the challenges that would arise if we one day developed really powerful forms of machine intelligence.

At that time, it seemed most useful for me to try to draw attention to the potential pitfalls on the path ahead so that we could use the intervening years to prepare ourselves — to do our homework, basically, like to do the research on alignment, etc.

Now, of course the discourse has changed a lot, and now it’s very mainstream to think that superintelligence is not just a science fiction–like thing, but a real possibility.

So it seemed less useful for me to just keep pinging the same string. There was this other side of the coin, which was already always there in my mind — like the upside and the downside — and to just take a glance at what would happen if things go well. Like, if we take the best case scenario, that also has philosophically problematic aspects to it.

Klavan: I’d like to get into some of the specifics of what those scenarios might be in a bit, but these numbers do get very specific. I’ve invoked one Anthropic researcher, but I’ve seen a lot of people saying P (doom) is 70 percent or 10 percent.

Can you walk us through the process of assigning a number to a probability like that? Is there a math behind it? How does that work?

Bostrom: No, there isn’t. The number itself comes out of a black box and different people — Those are meant to be expressions of a subjective degree of uncertainty and confidence.

It’s not possible to calculate the number based on some kind of set of agreed assumptions, which is reflected in the fact that different people come up with very different numbers.

My view is that there will be real risks associated with this transition to the machine superintelligence era, including existential risks. We should take those seriously.

If you want a more qualitative assessment, I would say that I’m a fretful optimist.

Klavan: As in, you think that there’s a real chance things go quite pear-shaped, but you’re hoping, you believe it’s not forever concluded.

Bostrom: Yeah, I’m really excited about the upside. I’m hopeful we could get there, but I worry that we might not.

Klavan: Let’s stay in our anxieties for just a second and wallow in them together. Can you put some more meat on the bones of these potential doomsday scenarios? What are some of the things that could go wrong? What would that look like?

Bostrom: I think there are several challenges that we would need to meet in order to get a good outcome here. One is the alignment problem. This is the technical difficulty of developing scalable methods of A.I. control. Basically, ways to ensure that we can continue to steer these systems and have them do the things we intended for them to do, rather than something different, even as these systems become ultimately arbitrarily capable. That’s one challenge.

I think a different class of challenge is what you might call the governance problem. Assuming we manage to control this technology, there is then the question of what ends will we humans put it to. Sometimes we use our powers for ill purposes — to wage war, to oppress each other, maybe to concentrate power — to do all kinds of nefarious things.

So the second challenge is: Can we at least do a reasonably decent job of shifting the use cases predominantly toward the positive?

Klavan: Some of these are touching on the scenarios that you were contemplating back in 2014 and thereabouts — the question of staving off or keeping people from inventing disaster technology. I’m thinking especially of your paper on the “vulnerable world” hypothesis, the idea that there exists some tech somewhere that, if invented, will or could instantly cause disaster.

In that paper, you raise a bunch of things that might be necessary to keep this from happening. You talk about the possibility of mass surveillance, one world government — all kinds of things that would be, I think, pretty difficult and daunting and terrifying to contemplate now, but you say maybe this will start to appear necessary.

Are you still contemplating measures of that kind? Is that still a possibility?

Bostrom: Well, I should clarify that the paper doesn’t advocate any of those things.

Bostrom: It considers this possibility, if we are unlucky. And then I consider if that were the case, then what would have to be true of a civilization that invents that technology in order for it not to be devastated?

Depending on the nature of that technology, if it’s one that sort of incentivizes first strikes among major powers, then it might require some kind of global coordination. If the problem with the technology is that it makes it too easy for random individuals to bring down civilizations, some sort of super weapons of mass destruction that you could concoct in your own kitchen using ordinary materials, then it looks like you would need to do this very detailed surveillance mesh to prevent people from doing that.

But the world is not guaranteed to be pegged up such that we will have an easy time. Anyways, so that’s that paper. It just tries to create a framework for thinking what the different possible situations could be that we might find ourselves in.

Klavan: OK. Let’s shift our focus now and start thinking about what happens if things go right — a cheerier subject, maybe. You’ve been writing it since at least 2024.

Your book “Deep Utopia” is about the scenarios we can hope for, the optimal setup. Can you lay that out for us? What does it look like if this goes well?

Bostrom: If we develop this strong form of superintelligence, then I think what that means is that across the board, across all other technological and economic fields as well, we would have extremely rapid advance because these superintelligent minds would then be doing the science and the research and the economic activity, but at digital speeds.

I think you would have a kind of rapid galloping toward perfect virtual reality or a cure for aging or space colonies or all kinds of almost science fiction–like technologies that don’t violate any laws of nature. They’re just really hard.

So the question then becomes: In such a condition of technological maturity, what would a good human life look like? What would a future path for us look like under these, in some sense, ideal circumstances?

I have this concept of a solved world, which is the idea of technological maturity plus some sort of decent job on the governance things.

Let’s imagine everybody gets a slice, everybody is free. What happens then is that a lot of the constraints that currently give shape to our human lives would peel away.

You could have the A.I.s and the robots doing basically all economically productive tasks. You then have a leisure society where there is no longer any need for humans to work in order to get a paycheck.

If that was all, then I think it would obviously require some cultural transformation and adaptation. But ultimately I think we could easily imagine living in such a world.

There are already various groups of people who don’t work for a living that you could look at for inspiration. Young children don’t work for a living. They seem to have a good life. They spend time with friends, playing games, inventing things.

Klavan: It depends if you aspire to live like children.

Bostrom: Then we have retired people. We have a confounder there, that many retired people have poor health and stuff, but if you imagine somebody who has enough money and perfect health, they often have great lives. They go around traveling, spend time with their grandchildren. They have hobbies. They watch movies. They golf. Whatever it is, it’s not necessarily a dystopian existence.

Klavan: Sounds great.

Bostrom: Then you have various other people who maybe have a lot of independent wealth. They don’t need to work for a living. I think there would be changes.

Like, the education system right now is configured to take in children and then output productive workers with quality grading attached ——

Klavan: This is one of your rants in your book, “Deep Utopia.”

Bostrom: Yes, it is my rant. I do have sympathy because right now, the world does need a lot of productive workers, so in some sense we have no option. But if we no longer needed productive workers, then it would be perverse to continue to think of education as just a preparation for entering into the labor force and to get a desk job at an office.

Maybe then have an education geared more toward preparing people to live a great life, cultivate the art of conversation, hobbies, spirituality, physical wellness, arts ——

Klavan: Now you’re speaking my language. I’m a college professor. I teach Greek literature.

Klavan: You don’t have to sell me on an education for a life of leisure.

But can we get into what happens then, when we peel back........

© The New York Times