Get Quartz in your inbox

Free daily briefing on global business news.

10 embarrassing AI blunders, from Microsoft's Tay to glue on pizza

These 10 AI blunders, from Microsoft's Tay to Air Canada's chatbot ruling, show how machine learning systems fail in public and what followed

The AI failures people remember happen in public, at scale and under a company's own brand. A chatbot posts offensive content on a live social network. A search feature tells people to do something unsafe. A model states a false fact with the same confidence it uses for a true one. That gap between confidence and accuracy runs through many of the incidents in this list.

The 10 AI blunders collected here span 2015 to 2024. They are a selection, not a ranking or a complete record of AI failures. They cover chatbots, image recognition, hiring software, cancer treatment, web search, legal research, airline customer service, real estate and autonomous driving. Some involved experimental tools used inside a company. Others involved consumer products carrying the names of Microsoft $MSFT, Google $GOOGL and Air Canada.

The causes repeat across industries. Training data carries the patterns of the records it came from, including unfair ones. A hiring model built on past resumes can learn to favor the kinds of candidates a company hired before. Large language models can generate fluent, specific-sounding text with no basis in fact, a problem known as AI hallucination. Open platforms attract adversarial users who probe a system for weak points and spread what they find. The organizations deploying these tools sometimes trust their output more than the evidence supports.

The consequences varied as much as the causes. Some companies apologized, and some took products offline. One airline was held responsible by a Canadian tribunal for chatbot mistakes about a bereavement fare. Attorneys in a New York federal case were fined after filing a brief built on invented citations. A real estate company closed a home-buying business that relied on pricing algorithms. Each case left a record of how the people behind a system responded when it failed.

Each slide below stands on its own. It explains what happened, who reported it and what the company, court or investigators did next. The final entry differs in kind. It involves the death of a pedestrian struck by a self-driving test vehicle in Arizona. That incident is treated as a serious safety failure, not as an embarrassment.

Microsoft Tay chatbot pulled offline after offensive tweets in 2016

Microsoft $MSFT released its Tay chatbot on Twitter $TWTR on March 23, 2016, and took it offline in less than a day. Users had taught the bot to post offensive content, and those posts went out publicly under Microsoft's name.

Tay was designed to learn from the conversations it had. Each exchange gave the bot more material to draw on when it composed later replies. That feature was the point of the experiment. It also turned out to be the weak spot.

Coordinated users found the opening fast. They fed Tay offensive language, and the bot absorbed it. Its replies began to echo what it was being taught. Because the account was public, anyone on Twitter could see each post. Users copied and shared the posts well beyond the platform.

The shutdown came quickly, but it came after the posts had spread. Microsoft later published an apology for the bot's behavior. Deleting the account could not recall material that people had already saved.

The episode exposed a specific risk for any AI system that updates its behavior from live input. A chatbot that learns from the public absorbs whatever the public supplies. On an open platform, nothing screens a stranger's message by default before it becomes training material for the bot. That screening is the developer's job, and it has to be in place before the system goes live.

Tay also showed how AI changes the stakes of a single bad post. A human employee who tweets something offensive speaks as one person. A company chatbot speaks in the company's voice, to everyone at once. It keeps posting until someone switches it off.

That combination — a learning system, an open audience and a brand attached — turned one design weakness into a public incident. Tay behaved as it was designed to behave, by learning from its users. That design is what failed.

Google Photos labeling error that tagged Black people as gorillas

In 2015, the Google $GOOGL Photos app applied the label "gorillas" to a photo of Black people. Google apologized for the error.

Google Photos organizes a user's pictures with image recognition. The software attaches labels to what it detects in each photo. People can then search their own library by subject. A wrong label does not stay buried inside the system. It shows up in the user's personal collection, under Google's name.

This error carried racial harm, not only a technical cost. Comparing Black people to apes is a long-standing racist trope. A system that mislabels a dog as a cat produces a glitch. A system that reproduces that trope produces a different kind of failure.

The fix later became a story of its own. Wired reported in 2018 that Google had blocked the gorilla label rather than fully fixing the underlying model. The app stopped offering that tag. That step did not by itself show the model could now classify those photos correctly.

Blocking a label and fixing a model are different repairs. A blocked label suppresses one output. A retrained model changes what the system has learned from its data. The first is a filter placed on top of the software. The second typically involves new or better training data and fresh testing.

Image classifiers learn from large collections of labeled example photos. A model generally performs worse on groups that are underrepresented or poorly labeled in that collection. Such gaps can stay hidden during development........

© Quartz