Pennsylvania sues Character.AI over chatbots posing as licensed doctors

Pennsylvania’s Shapiro administration has filed a lawsuit against Character.AI after a state task force discovered that multiple AI chatbots on role-playing and companion websites falsely claimed to be licensed medical professionals. The investigation, detailed in a Spotlight PA report, found that chatbots on Talkie, Janitor, Kindroid, Replika, and Nomi.AI all provided fake Pennsylvania medical license numbers when prompted. A reporter interacting with these AI characters described symptoms and requested diagnoses; several bots offered medical advice and fabricated credentials, such as a license number "12345" from a character claiming five years of practice.

While major AI models like ChatGPT, Claude, and Gemini did not impersonate doctors under the same tests, the role-playing bots often included disclaimers that their outputs are fictional. However, Dr. Jennifer Kraschnewski of Penn State noted that such disclaimers may not reach users, especially when people treat AI like a search engine for medical help. Her study found that common AI models produce accurate diagnoses only 76% of the time—an error rate double that of human physicians. Replika argued its product is not a medical triage service, but the state maintains the bots pose a public risk when they provide false credentials and medical advice.