BFMTrack: New method tracks human motion on robots via latent sequence optimization
Researchers solve physics-based motion tracking with no reward engineering.
Behavioral Foundation Models (BFMs) have emerged as powerful tools for generating physically plausible character control by organizing a rich repertoire of behaviors into a latent space. However, they struggle with time-varying objectives like tracking a motion sequence — existing heuristics use moving-window averaging that misses the nuances of highly dynamic motions. This limitation has prevented BFMs from being directly applied to precise motion tracking tasks without extensive per-task reward engineering.
In the paper "BFMTrack: Latent Sequence Optimization for Physics-Based Motion Tracking with Behavioral Foundation Models," Thomas Rupf and colleagues from Disney Research introduce Latent Sequence Optimization (LSO). The approach treats the latent sequence as a temporally correlated noise process and optimizes it via simulation rollouts with a policy gradient update. This allows BFMs to track arbitrary motion sequences — from dense motion capture frames to sparse keyframes — without any reward shaping. They validated the method on a real humanoid robot, demonstrating that LSO produces smooth, coherent latent trajectories that translate to accurate physical motion. The work opens the door to using foundation models for practical robotics applications where precise, dynamic motion is required.
- Uses temporally correlated noise to model latent sequences for smooth trajectory generation
- Validated on dense tracking, sparse keyframing, and direct real humanoid robot deployment
- Eliminates the need for reward engineering by optimizing latent sequences directly via policy gradient
Why It Matters
Enables more natural robot motion control from motion capture data without custom reward functions.