The context you need, when you need it

When news breaks, you need to understand what actually matters — and what to do about it. At Vox, our mission to help you make sense of the world has never been more vital. But we can’t do it on our own.

We rely on readers like you to fund our journalism. Will you support our work and become a Vox Member today?

Anthropic could just stop

Unless the AI company has proof that its safety strategy is working, now’s the time to quit racing.

We’re living through an AI-fueled “Black October” and things are changing fast. Hackers are using AI to breach countless companies. AI tools have uncovered security flaws that could leave 70 percent of all websites vulnerable. Governments are scrambling to secure their sites. Banks, too. And because cyberhacks can hurt supply chains, some longtime AI watchers are going full prepper, stocking up on food, cash, and more.

In the center of this story is Anthropic, one of the world’s leading AI companies, which was explicitly founded on the promise that it would be a “good guy” company in a world of dubious tech giants.

Anthropic says all the latest developments are very serious. Its security chief wrote last week that things are “going to get a lot worse before they get better.” Two weeks ago the company warned that AI models could soon self-replicate on the internet-connected devices in your home, like smart fridges and robot vacuums. And its co-founder now says “the singularity” — when AI systems speed up technical progress in a way that’s hard for humans to comprehend — “is here.”

Explore the big, complicated problems the world faces and the most efficient ways to solve them. Sent twice a week.

You might think, given its view of the risks, that Anthropic would be pumping the brakes on its cutting-edge AI progress. Just the opposite. Anthropic filed for a blockbuster initial public offering on Monday, chasing a valuation of $2 trillion even as it warns in the IPO filing that AI could “pose catastrophic or existential risks to humanity.” Every day, it is racing toward recursive self-improvement (RSI), where AI models are smart enough to build the next, even smarter AI models by themselves.

How does Anthropic justify that?

As its CEO, Dario Amodei, reminded the world last week, he built the company on one big bet: that instead of driving a race to the bottom, where companies skimp on safety in their rush to beat competitors in the market, Anthropic could actually catalyze a “race to the top.”

Why I wrote this story

As I’ve watched Anthropic double down on its “race to the top” strategy, I’ve been uncomfortably reminded of another very risky gambit: Sam Bankman-Fried’s crypto fraud of 2022. There are important differences between the two; this is not a perfect parallel. But both involve a tiny handful of figures, strongly influenced by effective altruism, making huge bets on behalf of many, many other people.

EA culture prides itself on offering a counter to commonsense morality. The community has shown a willingness to take big risks and justify them with complex calculations. Sometimes that pays off. But believing you are intelligent enough to adopt a counter-intuitive, galaxy-brained strategy and end up saving the world with it often involves some hubris. And hubris makes me nervous in a field as high-stakes as AI.

The idea is that if Anthropic proves it can build AI that’s both powerful and safe, the market will expect both — so other AI companies will then feel compelled to compete on safety, too. That may sound nice in theory, but it means Anthropic has to keep building ever-more-powerful AIs.

“The forces that drive the race to the top are hugely more effective if the company practicing them is at the frontier,” Amodei wrote. “And looking at that strategy as a whole, I’m proud of our track record and stand by it.”

In other words, Amodei is sticking to his bet.

But Amodei is gambling with, by his and his company’s own telling, the future of humanity. If AI goes wrong, it could lead to our gradual disempowerment, a galling concentration of power, or even complete extinction.

So there’s a very basic question that the public deserves a good answer to right at this pivotal moment: What is the evidence that Anthropic’s stated strategy — compete at the frontier so as to drive a “race to the top” — is actually successful overall?

From the outside, it looks like Anthropic has done more, on balance, to drive a race to the bottom. (Disclosure: Future Perfect is funded in part by the BEMC Foundation, whose major funder was also an early investor in Anthropic; they don’t have any editorial input into our........

© Vox