New Method Spots AI That Silently Breaks When Rules Change

Companies change their rules all the time — new privacy laws, updated lending guidelines, revised safety standards. But the AI models they built under the old rules don't automatically notice. They keep running, quietly out of step with current policy. The author of a new paper calls these "learned artifacts": AI components shaped by rules that may no longer exist. The hard question is figuring out which ones are now invalid — without throwing out everything and starting over.

His first experiment shows why the obvious answer doesn't work. He built four small AI add-ons (called LoRA adapters — think plug-in modules bolted onto a bigger model, each about 1 GB of data) trained under different versions of the same rules. The outdated versions and the freshly retrained versions gave identical answers on all 80 test questions. In other words, you cannot tell the difference just by testing what the AI says. What did work was a paper trail: recording where each model came from and independently re-checking every kept component against the current rulebook. That process rejected the stale versions.

A second experiment tested six policy changes in a controlled setting. Following the trail of dependencies caught all 40 genuinely invalidated items while flagging 56 for human review — compared with 120 if you simply invalidate everything touched by a changed document. That's less than half the review workload. But the trail has to be complete: removing just one key link per policy change dropped detection from 40 down to 34.

The author is careful about limits. This is a controlled lab demonstration, not proof of what happens inside a real company. It doesn't show cost savings, doesn't prove AI can "unlearn" anything, and doesn't claim to be the fastest workflow. The takeaway is simpler: keep records of how your AI was built, and re-verify rather than assume.