AI Still Can't Really Watch Dementia Care Videos, New Test Finds
57 million families need help — but AI is mostly guessing from textbooks.
About 57 million people worldwide live with dementia, and for most families the hardest part isn't forgetting names — it's the difficult behaviors: agitation, wandering, refusing help, late-afternoon confusion. The same behavior can mean completely different things depending on what happened five minutes earlier. So a team of researchers asked a simple question: can today's AI actually watch a caregiving video and understand it well enough to advise a family?
To find out, they built the first test of its kind. They took 56 professionally made caregiver training videos, cut them into 94 clips covering nine types of difficult behavior, and wrote over 2,000 questions. Each question was checked under four different viewing conditions, so the researchers could measure exactly how much video the AI needed to answer — rather than assuming it.
The results were humbling. The best AI models hit around 85% overall, which sounds impressive. But that average was carried by questions an AI could answer from general medical knowledge alone, without ever watching anything. When questions genuinely required understanding the order of events in the clip, accuracy fell by about 17 points on average. One strong open model performed at chance — essentially guessing — when asked whether a caregiver had responded appropriately.
The good news: this isn't unfixable. The team gave one model a light tune-up using their new training data, and its reliance on actual video content flipped from slightly negative to clearly positive. That means the weakness the test exposes can be repaired, not just measured. The honest catch is that all of this is still lab work — no caregiver app or hospital tool is using it yet, and the videos were training materials, not real family homes. AI that genuinely helps you care for a parent is still years away, but at least now there's a way to measure progress.
- This is the first test built specifically to check whether AI understands dementia caregiving videos, not just dementia facts.
- AI models look 85% accurate overall, but accuracy drops 17 points when they must actually follow the video's sequence of events.
- The researchers proved the problem is fixable — a light tune-up made one model genuinely depend on the video rather than guesswork.
Why It Matters
Millions of families care for dementia patients; this shows AI helpers aren't ready yet — and gives a way to build better ones.