Anthropic’s AI oversight isn’t democratic enough

Anthropic recently announced the onboarding of six external evaluators from the nonprofit Model Evaluation and Threat Research to provide external evaluation of AI models throughout the development process. This is an important advance in our society’s ability to manage AI development.

It is also ridiculous. The current implementation is akin to a student choosing which subjects to be tested on and then choosing which of their friends to test them on that subject. 

External evaluators should be embedded at AI companies, but they should not be restricted to AI safety researchers, and they should not be selected by the company being evaluated. Rather, they should represent communities across society, address a much broader set of anticipated disruptions and be selected by the public.

Were AI developing more slowly, its development could be managed by the usual democratic processes. But the incredible pace at which these technologies are advancing requires new ways for the public to manage this industry. It requires the development of new infrastructures of democracy. And independent embedded evaluations should be part of that new infrastructure.

I am a mathematician studying the internal geometry of AI systems. For years, I have been concerned that the way AI researchers are developing AI is growing untethered from the needs and desires of the public. 

Two years ago, I co-organized a workshop at NeurIPS — one of the largest annual AI conferences — to facilitate dialogue between AI researchers and communities being impacted because it is........

© The Hill