GaussFly: Contrastive Reinforcement Learning for Visuomotor Policies in 3D Gaussian Fields

A research team led by Yuhang Zhang has introduced GaussFly, a groundbreaking framework designed to solve a core challenge in robotics: training vision-based autonomous drones that can seamlessly transfer from simulation to the real world. The system employs a novel 'real-to-sim-to-real' paradigm. First, it uses 3D Gaussian Splatting (3DGS)—a high-fidelity rendering technique—to reconstruct real-world training scenes in simulation, augmented with explicit geometric constraints for accuracy. This creates a photorealistic and physically plausible digital twin for training.

Second, GaussFly leverages these simulated environments for contrastive representation learning. This process trains an encoder to extract compact, low-dimensional, and noise-resilient latent features from rendered RGB images, rather than relying on raw, high-dimensional pixel data. By feeding these robust features to a downstream reinforcement learning policy, the computational burden is slashed and the policy's inherent resistance to visual noise and domain shifts is dramatically enhanced. Extensive experiments show that policies trained with GaussFly achieve superior sample efficiency and can be deployed 'zero-shot' into complex, unseen real-world environments, effectively bridging the notorious sim-to-real gap that has plagued end-to-end learning approaches.