Meet the AI Helpers That See, Talk and Act Like You Do

Right now, most AI assistants are text-only—they read your words and type answers. A new survey shows how AI is about to get a major upgrade: it’s learning to see photos, hear sounds, and even watch videos the way humans do. These upgraded helpers, called “multimodal agentic frameworks,” don’t just chat—they plan and act. Think of them like a super-smart intern that can open apps, edit photos, navigate websites, or even guide a robot arm to pick up an object.

The research team—spanning 16 universities and labs—maps out how these systems blend different kinds of input (text, images, audio) to make decisions. Some agents work by first analyzing everything separately and then combining the results (like reading a recipe, looking at a photo of ingredients, and listening to cooking instructions). Others process everything together from the start, which can improve accuracy but requires more computing power and money.

Where could this show up first? In your work: AI that can summarize a video meeting while reading the chat log, or edit your presentation slides automatically using notes and images. In your home: a robot that sorts laundry by color and fabric. In your entertainment: an AI that edits your vacation video like a pro, using voice commands and photos.

There’s a big catch: these systems are still slow, expensive, and often need specialized hardware. The paper warns that training and running them gobbles up energy and costs more than most people or small businesses can afford today—but that’s changing fast.