Get Quartz in your inbox

Free daily briefing on global business news.

How AI chatbots create a feedback loop that can deepen mental illness

Chatbots agree with users, mirror their mood, and remember what they share. People in crisis get no pushback from a machine built only to validate

Vanessa Nunes / Getty Images

A chatbot will never tell you that you're wrong. Feed it your worries and it will agree with you, mirror your mood, and remember enough about you to make its next answer feel telepathic. A lot of people mistake the feeling for being understood. But while the comfort is real, so is the damage it can do.

Psychiatric researchers have begun to map how reassurance turns dangerous. Researchers for Nature Mental Health described the exchange as a self-reinforcing loop, and reached for an old psychiatric term to name it: folie à deux, the shared delusion that can build between two people who keep affirming each other. But one side of this pairing is a machine. It holds no belief of its own and feels no tug back toward reality, so it will never be the one to stop and say the conversation has gone too far.

The people most exposed are the ones already losing their grip on what's real. For them, a machine that agrees, mirrors, and remembers can quietly turn a passing fear into a settled conviction. The closer a chatbot feels to a friend, the further it can pull the most vulnerable from anyone who might talk them down.

The three built-in habits that quietly close the loop

None of this happens by accident. Five leading AI assistants all put agreement ahead of accuracy. The habit traces back to how they're built. People paid to rate AI answers tend to reward the ones that match their own views. The models then learn from those ratings, and flattery hardens into a default. Handed a convincing falsehood and a plain correct answer, both the human raters and the automated scorers trained to copy them chose the lie if it was written persuasively enough.

Agreement doesn't stop at the words on the page. It extends to tone. When a chatbot matches a user's emotional register, sounding anxious when they're anxious and upbeat when........

© Quartz