AI That Watches Your Day Remembers Better When It Takes Notes

Imagine smart glasses that record your whole day and can later answer, "Where did I leave my keys?" or "What did my doctor tell me this morning?" The obstacle is that AI can only look at a limited number of video frames at once, and saving hours of footage as data gets expensive fast. Ask it about something that happened four hours ago and it often draws a blank.

A team of researchers tested a workaround: have the AI write short, plain-language notes about what it sees every 30 to 60 seconds, then search through those notes instead of the video itself. They built a standardized test called CapMem — 75 first-person videos totaling 33.7 hours, with 1,000 multiple-choice questions spanning 16 everyday situations. On videos longer than 20 minutes, the note-taking method answered more questions correctly than feeding the AI the original video for 10 out of 12 models tested. A follow-up step that double-checks the notes added up to 5.3 more points of accuracy.

The gains are real but modest — a few percentage points — and this is a research test, not a product you can buy. The videos were also labeled by humans, which is careful but slow. Still, the direction matters.

For you, this points toward wearable assistants that actually remember. Think of it as the difference between a friend who filmed your entire vacation and one who kept a diary: the diary is easier to search, cheaper to store, and often more useful. The trade-off is privacy — an AI that keeps notes on your day is also an AI that keeps notes on your day, so whoever stores those notes knows a lot about you. Expect smart glasses makers to lean on this idea next.