Researchers Warn AI Chatbots Can't Be Trusted With Money or Medicine
If AI is helping decide your loan, diagnosis, or security, read this warning.
Many companies assume that as AI chatbots get smarter, they'll also get better at serious number-crunching jobs: pricing insurance, deciding who gets a loan, triaging hospital patients, or spotting a hack on a corporate network. A new paper from three researchers argues that assumption is wrong — and that scaling up today's chatbots won't fix it.
The problem is the raw material. Chatbots learn from human writing, and writing is a lossy copy of reality. A doctor's note might say "blood sugar was high" — but the actual reading of 187 is gone. A loan file might summarize "risky borrower" without the exact payment history. Once the numbers are squeezed into words, no model, no matter how huge, can squeeze them back out. The authors say this is a property of the data, not a lack of computing power.
They also list three things serious decisions demand that chatbots can't reliably deliver. First, reproducibility: ask twice, get the same answer twice. Second, lineage: tracing every output back to the exact source records that produced it — essential when a regulator or a court asks "why?" Third, calibrated uncertainty: the model knowing and stating how unsure it is, instead of sounding equally confident when it's right and when it's guessing.
Their fix is a new category they call a Large Quantitative Model: an AI trained on the numbers themselves, designed to repeat itself, show its work, and admit doubt. It's worth stressing this is a proposal, not a product — no such system exists yet, and the paper is an argument, not a demo. But it's a useful warning for anyone tempted to hand a chatbot their mortgage application, their medical history, or their company's security logs. Treat confident-sounding AI numbers as a draft to verify, not a verdict.
- Chatbots learn from human writing, and writing loses the exact numbers that matter — so bigger AI won't automatically make better medical or financial calls.
- The authors demand three things high-stakes AI needs: repeatable answers, traceable sources, and honest uncertainty.
- No such 'Large Quantitative Model' exists yet — this is an argument for how AI should be built, not a tool you can buy.
Why It Matters
Before trusting AI with your money, health, or security, know it may be guessing at numbers it never really learned.