AI Struggles to Understand More Than One Thing at Once
AI can't even pass a test when asked to handle photos, audio, and text together
Imagine trying to take a picture, record a voice note, and type a question all at once — and then answer it correctly. That’s the kind of challenge researchers just gave to the smartest AI systems. They built a new test called the Modality Maturity Index (MMI) to see if AI can handle multiple types of input — like photos, audio, video, and text — and respond in the right way.
The results were underwhelming. Even the top AI model only got about 35% of the answers right when it had to combine different kinds of information. The test included 893 carefully designed questions that required AI to show it understood multiple inputs and could respond in the correct format. For example, an AI might be asked to describe a photo, summarize an audio clip, and then answer a question about both — all in one go.
The researchers also found that many AI systems didn’t even try to use all the requested types of outputs. Some ignored the audio part entirely or skipped the photo description. To measure this, they created another score called the Modality Presence Score (MPS), which tracks whether the AI actually produced the expected kinds of output. The best AI scored only 35%, and the worst just 16%.
This isn’t just about tech geek stuff — it affects how useful AI will be in real life. If AI can’t handle combining a photo with text or audio, it won’t be able to help you write a report based on a spreadsheet and a voice memo, or explain what’s in your vacation photos when you ask it aloud. Until these systems get better at juggling multiple types of information, their real-world usefulness will be limited.
- AI currently fails when asked to combine photos, audio, and text in one task — even the best model only gets 35% right
- The new test, Modality Maturity Index (MMI), was designed with 893 questions to simulate real-world multi-sensory tasks
- AI often ignores parts of the question entirely, meaning it skips important information when responding
Why It Matters
AI still can't reliably combine photos, audio, and text — so it won't be a true assistant until it improves