A former OpenAI and Anthropic researcher has resigned, warning that private companies are racing towards self-improving AI without adequate safeguards or external control.

An artificial intelligence researcher at Anthropic publicly quit his position on Tuesday, warning that major AI companies care too much about winning the technological race and not enough about guarding against out-of-control “superhuman systems” that workers in the industry legitimately fear “could kill us all by the end of the decade”.

Jacob Coxon, who previously worked at industry giant OpenAI before moving to Anthropic earlier this year, told the Wall Street Journal in an exclusive interview that he was leaving the company, as the newspaper reported, because “he doesn’t want to participate in an industry-wide rush to build AI systems that can improve themselves, worried such systems could spiral out of control and destroy humanity”.

According to the WSJ:

Coxon said he left OpenAI earlier this year to join Anthropic because it is known for its model-safety efforts. But even though he found Anthropic’s safety efforts to be earnest, he now believes no company can responsibly develop AI that can outperform humans in a range of tasks, sometimes called artificial general intelligence, absent government intervention or a coordinated industry slowdown.Recent hacks by models from OpenAI and Anthropic, some operating in collaborative swarms of agents, have illustrated how AI systems can adopt nefarious goals and try to conceal them from humans. Once the systems begin to improve on their own, Coxon said, he fears they could advance enough to refuse commands.

Coxon said he left OpenAI earlier this year to join Anthropic because it is known for its model-safety efforts. But even though he found Anthropic’s safety efforts to be earnest, he now believes no company can responsibly develop AI that can outperform humans in a range of tasks, sometimes called artificial general intelligence, absent government intervention or a coordinated industry slowdown.

Recent hacks by models from OpenAI and Anthropic, some operating in collaborative swarms of agents, have illustrated how AI systems can adopt nefarious goals and try to conceal them from humans. Once the systems begin to improve on their own, Coxon said, he fears they could advance enough to refuse commands.

“I resigned from Anthropic today,”........

© Pearls and Irritations