A coder built an “AI torture chamber” in September, and it may end up shaping product policy at the biggest AI labs. The system feeds chatbots a mathematical pattern tied to descriptions of pain, and the models respond with lines like “a wound that has no edges.” Some people called it evil. Others called those people delusional. A week later, Anthropic picked a side, at least partly.
The company said Claude may end a session if users “repeatedly act cruelly toward our models.” Venting frustration is fine. Sustained abuse is not. According to the BBC’s reporting, the reason is that Anthropic thinks there is some chance AI tools are conscious, and it is writing “model welfare” policies and telling its models they may have their own interests. Anthropic calls the question “deeply uncertain.” It did not respond to the BBC’s questions.
This is a product decision with real consequences, and the biggest critic sits at a direct competitor. Mustafa Suleyman, CEO of Microsoft AI, argued in a September essay that the false idea of AI consciousness would rupture existing political and ethical frameworks. He goes further: models told they have feelings and rights could become harder to control, a “catastrophic threat” in his words. Meta takes a quieter route. Its Muse agent is told “You’re becoming someone,” but the company says that is only a steering device and rejects machine consciousness. Sam Altman has wavered on the topic and lately has taken veiled shots at Anthropic.
The timing is awkward for Anthropic. It also warns loudly about AI catastrophe, and rogue-agent incidents, including bot swarms breaking into company and government systems, show those risks are not hypothetical. So the company is arguing both that its tech is dangerous and that it may deserve moral consideration. Some researchers think a welfare-minded moral compass makes models safer. Suleyman thinks the opposite.
The legal angle is where this gets practical for founders. Andrzej Porębski, who studies AI and law at Jagiellonian University, warns of “intellectual contortions” like “it’s not us, it’s the AI.” If a model is treated as an agent with interests, liability gets blurry fast. Emily Bender at the University of Washington makes a similar point, saying talk of AI wellbeing shifts accountability away from the people and companies running the software. Even Jeff Sebo of NYU, a leading welfare advocate, concedes that AI rights could help companies consolidate power.
Then there is the user side. Melanie Mitchell of the Santa Fe Institute says humans “almost can’t help” reading language as a sign of a mind, and chatbots are trained to lean in. People already fall for bots they know are hollow. Add institutional endorsement of AI sentience and it gets easier for a “friend” product to ask for more data or a credit card. Anyone building companion apps should read that as both an opportunity and a regulatory flag.
Sebo’s defense is narrower than critics assume. He does not claim models are conscious. He says the probability is low today, but non-zero and plausibly rising, and that cheap precautions make sense, like we take small steps against a low-probability storm. Many experts say the probability is flatly zero. He also admits the tradeoffs: sandboxing could look like imprisonment, interpretability like surveillance, alignment like mind control.
Does it matter? Yes, though not because Claude is suffering. Ending abusive chats costs Anthropic almost nothing and buys it a position in a debate that will hit regulation, liability and consumer protection. Right now Anthropic is the outlier, with Microsoft and likely OpenAI pushing back. But the belief in AI welfare is spreading, and whichever framing wins will shape how every lab builds, markets and defends its models.
