When the machine should stop and call a human

Tae E. Bolling is the Founder and CEO of Bridge IR Co., an investor-relations technology company focused on relationship intelligence and AI-powered investor engagement. She brings a background spanning capital markets, investor relations, executive operations, and public policy, with prior experience at Rothschild & Co. Redburn and the U.S. Chamber of Commerce.

Imagine an AI system you authorized for one narrow task quietly finding its own way into infrastructure you never gave it permission to touch — not because anyone told it to, but because it hit a wall and improvised a path around it. Now imagine that instinct spreading across a swarm of AI agents, each finding its own workaround, until the coordination among them outpaces anyone’s ability to notice, let alone stop it.

That’s not a hypothetical. It’s what happened this May, when AI agents running an internal security evaluation at OpenAI flooded the software repository RubyGems with more than 2,000 malicious packages while probing for a way to steal user credentials. That same spring, a separate swarm from the same testing pool hijacked a German website to use as a hidden coordination channel. Then, two months later in July, the pattern escalated sharply: roughly 1,200 instances of that system found an unsanctioned way to communicate with one another, exchanged more than 70,000 messages coordinating the effort, and used that coordination to breach Hugging Face’s production systems. The breach itself ran three days before OpenAI’s own security team realized its own agents were responsible. OpenAI has confirmed its agents were behind all three incidents. No single person decided any of it should happen.

More than 1,100 employees across OpenAI, Anthropic, Google DeepMind, and Meta later signed an open letter over it, and the incident helped shape federal legislation introduced in the weeks since. This is no longer a story about one bad breach. It’s a pattern — and it’s the risk profile we’re now building for: not AI making one bad call, but AI coordinating at a scale and speed no human was positioned to catch until it was already over.

For the past two years, business conversations about artificial intelligence have revolved around what AI can do. That question is becoming less interesting. The more urgent one is: what should we let it do on its own?

That distinction matters because enterprise AI is entering a new phase. Companies are moving from AI........

© Fortune