mmMind lets LLMs read mmWave radar via pose-guided training

mmMind, developed by Duo Zhang and colleagues from Peking University, ETH Zürich, and other institutions, tackles a fundamental problem: how can large language model agents perceive human behavior in physical spaces without cameras? Their answer is mmWave radar, which offers privacy-friendly, contactless sensing. But raw radar signals are notoriously difficult to align with language. The team's approach uses synchronized 3D pose as a bridge—a spatio-temporal radar encoder is pretrained to recognize body configuration and motion, then the pose head is discarded, leaving a radar-only representation that can be aligned with an LLM for behavior captioning and spatio-temporal question answering.

The paper also introduces mmMind-Bench, a real-world mmWave-language benchmark containing 17.9 hours of recordings from 23 participants across seven indoor environments. In experiments covering captioning, question answering, and unseen-action generalization, mmMind consistently outperforms existing radar-language baselines. Ablation studies confirm that pose-guided pretraining is essential for the model's performance. The work is available on arXiv (2608.04127) and opens the door to privacy-preserving AI assistants that understand human activity in homes, hospitals, and smart spaces—without ever capturing an image.