LLM-based appointment bot fails in production after 8 months of agony
Team shuts down doctor appointment AI after endless provider failures and sync issues
The team created an AI assistant integrated with messenger platforms to help users book appointments with private clinic doctors. They used OpenRouter as a gateway to access multiple LLMs (GLM 4.5, 5.0, 5.2, Deepseek, Qwen, GPT, Claude, Minimax) and adopted PydanticAI to streamline API interactions and tool handling. Initial testing went well, but production revealed critical flaws.
PydanticAI's sync wrappers are actually async tricks—when the codebase was entirely synchronous, the library would freeze the entire process, requiring operating system kill commands. Additionally, every model provider (including official ones) occasionally returned empty responses without proper error codes, and fallback providers would fail simultaneously during peak demand. Most damaging: the team's clients (clinics) promised their own customers 100% correct appointments. Any LLM hallucination or service hiccup broke trust and created a cascade of complaints. The founder concludes that while LLMs are fine for personal consumption, they are not ready for B2B services where reliability is contractually required.
- PydanticAI's sync mode is an async hack that can halt the entire process, forcing system-level kill commands
- All model providers (OpenRouter, official APIs) return empty responses instead of proper error messages, breaking fallback logic
- B2B clients promise 100% accuracy to end users, making any LLM unreliability a business-ending liability
Why It Matters
Production LLM services remain fragile for mission-critical tasks, highlighting the gap between demos and real-world reliability.