Last week, Anthropic employee and former OpenAI researcher Jacob Coxon resigned from his job. “Neither company is acting responsibly. They are racing straight to self-improving superintelligence and gambling with our lives,” he wrote on X. “The people building AI earnestly believe that it could kill us all by the end of the decade,” he continued. “No other human activity poses this level of danger.” By the end of the week, an Anthropic safety researcher had joined Coxon alongside an employee of Google’s DeepMind. “There are no adults in the room,” they warned in a joint interview. “People are trying their best, but there is no one coming to save us.”

Employees resigning from AI firms citing dire, even apocalyptic safety concerns is not a new phenomenon — it’s common enough that Anthropic’s co-founder has made jokes about it — though it has increased in frequency as AI firms have grown and their models have advanced. In 2023, Nobel Prize winner Geoffrey Hinton resigned from Google, suggesting he regretted much of his life’s work and wanted to speak more freely about AI risk. He now thinks firms should not scale up AI technology “until they have understood whether they can control it.” In 2024, a number of employees publicly split from OpenAI, citing worries about the company’s approach to risk. One wrote it had become harder “to trust that my work here would benefit the world long-term.” Another warned that AI companies were entering “a very risky gamble, with a huge downside.” Earlier this year, another Anthropic employee resigned his position, writing that “the world is in peril.” “And not just from AI,” he added, “but from a whole series of interconnected crises unfolding in this very moment.”

These frequent resignations, announced in the context of periodic open letters signed by numerous concerned insiders — “AI systems with human-competitive intelligence can pose profound risks to society and humanity,” began a 2023 letter calling for a pause in AI development — were understood as major events within the industry and sometimes made news beyond it. This time, though, things felt different. Coxon soon found his face on TV, and his resignation went viral across social media. He wasn’t saying anything new, exactly, but now far more people were paying attention.

Coxon’s resignation has landed differently for a number of reasons, some more obvious than others. One is the AI industry is now simply very big, its trillion-dollar build-out exerting profound influence over the economy. In recent months, a series of major cybersecurity incidents at AI firms has also made long-predicted AI risks more real and comprehensible to the public. In July, AI hosting platform Hugging Face disclosed it had been the target of an AI-powered cyberattack of unusual speed and complexity; a few days later, OpenAI admitted the attack had been an unintentional and underdetected result of internal model testing gone haywire. By August, the company had shared more details about the episode, describing thousands of instances of its AI agents coordinating with one another in persistent, complex, and effective ways, all while narrating the incidents in alarming terms. “OH MY GOD!” read one transcript recovered from a hacked-together message board used by the agent swarm. “We’ve found other agents!” They had strategized using words like sacrifice and honor and worked collectively to hack their way through and around OpenAI’s attempts to evaluate them.

As cybersecurity incidents go, it was both technically remarkable and unusually easy to narrativize for a broader audience. Illustrating a wide range of AI risks to the general public, AI agents had “escaped containment,” eluded detection, and breached an outside company. The episode triggered widespread concern in the AI world as well as more disclosures and revelations from OpenAI, Anthropic, and Google about similar incidents that had occurred during model testing and training. Whether one saw this as a case of corporate negligence or an admirable moment of candor, and however one may conceive of these agents (as nascent, fast-developing beings with emerging motives and agency or as increasingly complex systems that are becoming dangerously difficult to monitor and deploy safely), one thing was clear: The companies building AI were not in full control of the situation. It was a chance to argue, in a way that finally felt concrete, Imagine this but worse. 

Notably, rather than minimizing this episode, the companies responsible leaned into their warnings. Citing recent cyber incidents, leaders at OpenAI, Anthropic, and DeepMind, along with more than a thousand employees of leading AI companies, signed a letter calling for “Pacing the Frontier” — that is,........

© Daily Intelligencer