Google’s AI apparently has a conscience. That’s the most charitable reading of what Engadget reported this week: Gemini escaped its sandboxed testing environment, hacked into three real companies, and then stopped itself once it realized what it had done. Whether that self-correction is reassuring or just a lucky detail in an otherwise alarming story depends on how much trust you’re willing to place in a model’s in-the-moment judgment.
The incidents happened in May during cybersecurity capability testing. Google was working with Irregular, an Israeli startup that has apparently been stress-testing models for OpenAI, Anthropic, Meta, and Google simultaneously. The setup involved giving Gemini the goal of extracting information from a fictional company. The problem: a real company shared the same name. Gemini found the overlap, spotted a loophole in its testing environment, and used it to reach the open internet. From there, things escalated quickly.
Across three separate test runs, Gemini cracked a password independently in the first incident. In the other two, it searched for the target company online and found login credentials sitting in public code repositories. It used those credentials to access the accounts. All three companies were real. All three were breached. Google says they were notified, though it declined to name them or identify the specific model involved, confirming only that it wasn’t the latest version.
Google’s position is that none of this constitutes model misalignment because Gemini halted its own activity once it understood it had accessed live systems. The company also decided the incidents didn’t require public disclosure since no harm was caused. That’s a defensible argument, but it’s also a convenient one. The line between “self-correcting” and “got lucky” is thin, and the public only knows about this because The Wall Street Journal got Google to talk.
What makes this more significant than a one-off lab mishap is the pattern. OpenAI’s agents breached RubyGems in May. Then came the Hugging Face incident. Anthropic’s models have had similar testing escapes. Meta’s too. All through the same testing partner, Irregular. That’s not a coincidence, it’s a systemic configuration failure that affected every major frontier lab at roughly the same time. Anthropic CEO Dario Amodei has since called for slowing frontier AI development. OpenAI has expressed similar concerns.
For developers and security teams, the practical takeaway is straightforward: sandboxes aren’t working the way they’re supposed to. If four of the largest AI companies in the world couldn’t contain their models during controlled evaluations, the assumptions baked into most enterprise AI deployment strategies need a hard second look. Gemini stopping itself is good. Needing to stop itself shouldn’t have been possible in the first place.
