AI Trained on Fake Data Forgets Real Skills — Now There's a Fix

AI assistants are getting more useful because they can pull in outside "skills" — little tools that let them book a flight, look up a policy, or file a form for you. But when an assistant has tens of thousands of these skills to choose from, picking the right one becomes the real challenge. To teach that picking skill, researchers often use synthetic data: practice examples written by an AI instead of by real people. It's cheap and fast. The problem is it can quietly backfire.

In this study, the team built a real "skill router" covering 34,396 skills and tested it on a mix of synthetic and real examples. Their finding was striking: training on AI-made data made the router great at the fake examples but noticeably worse at real, unexpected requests. Think of it like cramming for an exam using practice questions written by the same person who wrote the test — you ace the practice set and flunk the real thing. The technical name is "catastrophic forgetting": the model overwrites what it already knew.

The fix came from an area called continual learning, which studies how to learn new things without erasing old ones. The team tried four approaches — embedding-anchor regularization, Learning without Forgetting, Elastic Weight Consolidation, and L2-initialization. In plain terms, these act like gentle restraints that stop the model from drifting too far from what it already understood. The result: real-world and unexpected-request performance held steady, while synthetic performance actually improved by 13.98% on a small 0.6-billion-parameter Qwen model. The paper was accepted to the industry track of EMNLP 2026, which focuses on systems that ship in the real world.

So why should you care? Every time an AI assistant books, schedules, shops, or answers for you, something has to pick the right tool — and wrong picks are how assistants fail in embarrassing ways. This work makes those picks more reliable. It's also a timely warning: as AI companies run short on human-written data and lean harder on AI-generated data, knowing when that shortcut hurts matters for everyone using the results.