The science fiction author Isaac Asimov once came up with a set of laws that we humans should program into our robots. In addition to a first, second, and third law, he also introduced a “zeroth law,” which is so important that it precedes all the others: “A robot may not injure a human being or, through inaction, allow a human being to come to harm.”

This month, the computer scientist Yoshua Bengio — known as the “godfather of AI” because of his pioneering work in the field — launched a new organization called LawZero. As you can probably guess, its core mission is to make sure AI won’t harm humanity.

Even though he helped lay the foundation for today’s advanced AI, Bengio is increasingly worried about the technology over the past few years. In 2023, he signed an open letter urging AI companies to press pause on state-of-the-art AI development. Both because of AI’s present harms (like bias against marginalized groups) and AI’s future risks (like engineered bioweapons), there are very strong reasons to think that slowing down would have been a good thing.

But companies are companies. They did not slow down. In fact, they created autonomous AIs known as AI agents, which can view your computer screen, select buttons, and perform tasks — just like you can. Whereas ChatGPT needs to be prompted by a human every step of the way, an agent can accomplish multistep goals with very minimal prompting, similar to a personal assistant. Right now, those goals are simple — create a website, say — and the agents don’t work that well yet. But Bengio worries that giving AIs agency is an inherently risky move: Eventually, they could escape human control and go “rogue.”

So now, Bengio is pivoting to a backup plan. If he can’t get companies to stop trying to build AI that matches human smarts (artificial general intelligence, or AGI) or even surpasses human smarts (artificial superintelligence, or ASI), then he wants to build something that will block those AIs from harming humanity. He calls it “Scientist AI.”

Scientist AI won’t be like an AI agent — it’ll have no autonomy and no goals of its own. Instead, its main job will be to calculate the probability that some other AI’s action would cause harm — and, if the action is too risky, block it. AI companies could overlay Scientist AI onto their models to stop them from doing something dangerous, akin to how we put guardrails along highways to stop cars from veering off course.

I talked to Bengio about why he’s so disturbed by today’s AI systems, whether he regrets doing the research that led to their creation, and whether he thinks throwing yet more AI at the problem will be enough to solve it. A transcript of our unusually candid conversation, edited for length and clarity, follows.

Sigal Samuel

When people express worry about AI, they often express it as a worry about artificial general intelligence or superintelligence. Do you think that’s the wrong thing to be........

© Vox