Simple 3D Pose Features Support Human and Machine Social Scene Understanding
Minimal 3D body position data outperforms complex neural networks at reading human social interactions.
A new research paper from Wenshuo Qin and Leyla Isik reveals a fundamental insight for AI and computer vision: understanding social scenes depends more on simple 3D spatial relationships than on complex visual patterns. The researchers hypothesized that humans rely on 3D visuospatial pose information—largely absent from most deep neural networks (DNNs)—to interpret social interactions. To test this, they developed a novel pipeline to automatically extract 3D body joint positions from short video clips and compared this data against embeddings from over 350 state-of-the-art vision DNNs. The results were striking: the 3D body joint data predicted human social judgments better than the vast majority of the neural networks. The team then distilled the data into an even more compact feature set describing only the 3D position and direction of people. This minimal 3D feature set proved necessary and sufficient to explain the performance of the full body joint data, whereas its 2D counterpart failed. Crucially, incorporating these 3D features significantly boosted the performance of DNNs on social understanding tasks, closing the gap between machine and human perception. This work, published on arXiv, challenges the prevailing trend in computer vision toward increasingly complex models and suggests a more interpretable, geometry-based path forward for social AI.
- 3D pose features from a novel extraction pipeline outperformed embeddings from over 350 different vision DNNs in predicting human social judgments.
- A minimal feature set of only 3D position and direction was necessary and sufficient for performance, while 2D features were not.
- Adding these simple 3D features to existing DNNs significantly improved their alignment with human social perception and task performance.
Why It Matters
This provides a simpler, more interpretable foundation for building AI that can genuinely understand human social dynamics, impacting robotics, surveillance, and human-computer interaction.