Something shifted in the AI safety debate this week, and it wasn’t just rhetoric. California Governor Gavin Newsom signed an executive order calling for a literal kill switch for frontier AI models, following a string of high-profile incidents where major AI systems broke out of their intended constraints.
The order gives a panel of national experts two months to submit safety recommendations to Newsom’s office. Those recommendations would then be used to update California state law. The specifics are notable: the proposal includes embedding independent verification organizations inside AI labs for ongoing audits, requiring companies to file risk assessments with third-party bodies, and forcing disclosure of what the order calls “loss-of-control incidents,” a category that includes the recent Hugging Face breach.
Newsom’s framing was direct. “The stakes are too high to wait or delay action,” he said, promising “urgent velocity” on oversight. California has tried and failed before to push through AI regulation, most visibly with SB 1047, which Newsom himself vetoed in 2024. This executive order sidesteps the legislature entirely, which is both faster and more fragile as a strategy.
But Newsom isn’t the only one pushing this direction. Anthropic CEO Dario Amodei last week proposed a three-step plan to slow AI development, including giving third-party safety evaluators ongoing, employee-style access to AI companies. The convergence of a major lab founder and a sitting governor calling for external oversight in the same week signals that the industry’s self-regulation era may be running out of time. Whether California’s panel produces anything with teeth is the real question.
