Safety experts warn novel design of OpenAI’s Astra model could make future AI agents harder to monitor

AI safety experts are raising alarms about how OpenAI has built its soon-to-be-released frontier AI model Astra, saying it may hasten the day when humans will lose the ability to monitor the reasoning that AI agents are using.For its new model, OpenAI has employed a method alternately referred to as “recurrent depth” or “looped Transformers” for a portion of the model’s internal architecture. The method can make AI models considerably more efficient by employing less computing power required to process each prompt—a valuable feature at a time when many businesses are complaining about the high costs of using the most advanced frontier AI models.

The new process, though, also means that part of the AI model’s “chain of thought,” or the reasoning steps it is taking, are not expressed in natural language, making it much more difficult for humans to monitor what the model is doing and why.Chain of thought monitoring is currently one of the methods companies use to make sure AI agents are not taking unintended or unauthorized actions.

Tech publication The Information first reported on OpenAI’s use of recurrent depth in Astra earlier this week. Jakub Pachoki OpenAI’s chief scientist and several other OpenAI researchers criticized the publication for sparking undue alarm among AI safety researchers, saying that it had limited the extent to which the looped transformer architecture is used so that the model’s reasoning remains legible.

In response to The Information’s report, Pachoki wrote on social media platform X that “we care deeply” about chain-of-thought monitoring and that “OpenAI has worked to preserve and utilize chain-of-thought monitoring since our very first reasoning models.” Pachoki said OpenAI would share more details of Astra’s architecture in the future.Pachoki said that he thought chain-of-thought monitoring could grow more challenging, but that this........

© Fortune