New AI Turns Third-Person Video Into Your Own Point of View

If you've ever watched a sports replay and wished you could see exactly what the player saw, this new research is for you. Scientists created an AI system that takes a normal video shot from outside (like a stadium camera) and generates a video from the point of view of a specific person inside the scene. The result is creepily realistic first-person footage that could make VR experiences, sports training, and even remote medical guidance far more immersive.

The challenge is that when someone turns their head or walks behind an object, the AI has to guess what they'd see — information that simply doesn't exist in the original recording. Older methods tried to use 3D geometry alone, but that fails when the view changes drastically. The new approach, called Grounded-Exo2Ego, uses two parallel systems: one tracks the scene's 3D structure and camera position, while the second "understands" objects — like knowing a chair exists even when it's blocked from view. This lets the AI fill in hidden details based on what it knows about the world.

The team also discovered that a hidden flaw ruined earlier attempts: tiny mismatches between where cameras thought they were and where they actually were. They built a clever re-alignment routine that fixes this, which improved every quality metric they tested. Plus, they created a fully automated system that generates synthetic training videos using 3D characters, so the AI never runs out of realistic practice footage.

On the challenging EgoExo4D benchmark, the new method beat recent state-of-the-art approaches by large margins. For regular people, this means more realistic VR social spaces, smarter sports coaching tools that show you exactly what the pro saw, and future AI assistants that can guide you through tasks from your own perspective.