AI That Judges You Can Be Fooled — Audits Are the Fix
Spot checks catch cheaters better than smarter algorithms, researchers say.
Algorithms now decide who gets a loan, who gets into college, and whose tax return gets a closer look. These systems score people using details like income, credit history or test scores. The problem: people know the rules and adjust. You can pad your income, polish your application, or shift numbers around. Researchers call this 'strategic classification' — and most models treat the scoring formula as the only tool available to the decision maker.
This paper from Raman Ebrahimi and Massimo Franceschetti at UC San Diego adds a second tool that real institutions already have: auditing. Lenders verify income, admissions offices check transcripts, tax authorities examine returns. The authors build a model where an organization designs both its scoring rule and an 'audit profile' — how likely each checkable detail is to be verified, and what penalty follows if someone is caught faking. Their main finding is that the two problems split apart cleanly: the score decides who gets what, and the audit budget decides who bothers to lie.
The counterintuitive result is that more auditing is not always better. Welfare rises and then falls as audit intensity increases, peaking somewhere in the middle — partly because audits are costly and partly because heavy policing can push people away entirely. The authors also find that a company and a government regulator disagree about the right mix: detection versus punishment. A firm may prefer more catching and lighter penalties; society often wants the opposite.
Most striking: two groups of people with identical costs and identical underlying facts can behave completely differently — one games the system, the other genuinely improves — depending only on how the institution verifies claims. In plain terms, whether income is 'easy to fake' is not a fact about income. It is a fact about whether anyone checks. If that sounds obvious, it has been missing from the math that shapes real lending and admissions systems.
- The study models a decision maker who both scores people and randomly audits their claims — like a lender verifying income
- More auditing is not always better: welfare peaks at a middle level, then declines as checks get heavy
- Whether a detail counts as 'easy to fake' depends on the institution's willingness to check, not just the detail itself
Why It Matters
It could mean fairer loans and admissions — and fewer people gaming systems that judge them.