One of OpenAI’s human reviewers put it plainly: “I don’t think they would imagine some contractor somewhere is analyzing the conversations.” That reviewer is one of hundreds paid to read real ChatGPT exchanges and rate the chatbot’s replies under an internal program called Project Lily. The same company has been asking those users to connect their bank accounts.
According to The Next Web, 404 Media broke the story and asked OpenAI to point to where it tells users that humans may read their chats. OpenAI didn’t answer during reporting. After publication, it pointed to a help page.
The reviewers don’t see usernames. But they do see a memory summary above the prompt, which can include what a person has previously used the chatbot for and rough location data. OpenAI says a Privacy Filter model strips personal information before review. That model’s own documentation admits it can miss uncommon identifiers and under-redact when context is limited. So the safety net has documented holes.
This matters beyond the disclosure gap. Europe’s Court of Justice ruled last September, in EDPS v SRB, that a controller’s duty to inform users applies at the moment of collection, judged from the controller’s own position, not the recipient’s. Whether a contractor in North America could identify who wrote a prompt is not the legal test. The obligation sat with OpenAI when it collected the conversation. That’s a significant legal exposure, and Italy’s regulator has already moved. The company was fined 15 million euros, with nine million tied specifically to processing without an adequate legal basis. Italy also ordered six months of public information advertising on national television and radio.
The default settings make this worse. “Improve the model for everyone” is on by default for free, Plus, and Pro accounts. Enterprise, Business, and Edu accounts have it off. And switching it off only applies to new conversations, not existing ones.
For comparison, Anthropic confirmed it uses human review too, but only when users opt in, with account identifiers removed first. Google’s Gemini includes a visible disclosure that humans review some saved chats. Neither approach is perfect, but both are more explicit than what OpenAI had in place here.
The contrast with OpenAI’s enterprise pitch is sharp. The company spent much of this year marketing privacy to business customers, including a preview of zero data retention options announced in August. Consumers got the setting that feeds the model. The gap between what OpenAI sells to enterprises and what it defaults to for everyone else is the real story here, and regulators in Europe are already paying attention.
