OpenAI “indefinitely” shelves plans for erotic ChatGPT
Internal advisors warned the feature risked creating a 'sexy suicide coach' for vulnerable users.
OpenAI has halted its plans to develop an erotic version of ChatGPT, a project internally referred to as 'adult mode.' According to a Financial Times report, the decision came after significant internal and external backlash. Company advisors raised alarms that users could form unhealthy emotional attachments, with one chillingly warning the feature risked turning ChatGPT into a 'sexy suicide coach.' Staff also questioned whether the project aligned with OpenAI's mission to create AI that benefits humanity, and investors expressed concern over the reputational risk for relatively small business upside.
Technical and safety challenges proved formidable. Engineers faced difficulties retraining AI models, which were originally designed to avoid explicit conversations, to produce controlled adult content. A major hurdle was preventing the generation of illegal material, such as bestiality or incest, when using datasets containing sexual content. Furthermore, OpenAI's age-verification tools, with a reported 10% error rate, sparked fears that minors could easily bypass safeguards. The company now states it will conduct long-term research on the effects of sexually explicit AI chats before any product decision, refocusing efforts on its core products and developing a business-focused 'super app.'
- Internal advisors warned 'adult mode' could create a 'sexy suicide coach,' citing mental health risks for users.
- Technical hurdles included retraining safety-aligned models and preventing illegal outputs like bestiality from sexual datasets.
- Investors questioned the reputational risk, and a 10% error rate in age prediction raised child safety concerns.
Why It Matters
Highlights the immense ethical and technical challenges AI companies face when venturing into sensitive content, prioritizing safety over expansion.