Animal behavioral analysis and neural encoding with transformer-based self-supervised pretraining

A research team from The International Brain Laboratory, Columbia University, and other institutions has introduced BEAST, a transformative framework for analyzing animal behavior and its neural underpinnings. The core innovation addresses a major bottleneck in neuroscience: traditional video analysis requires massive amounts of manually labeled data, which is slow and expensive to produce. BEAST overcomes this by using self-supervised learning—specifically masked autoencoding and temporal contrastive learning—to pretrain experiment-specific vision transformers directly on raw, unlabeled video footage. This allows the model to learn rich representations of behavior without human annotation, establishing a powerful backbone for diverse neuro-behavioral tasks.

The technical approach enables BEAST to excel in three critical areas: extracting behavioral features that predict neural activity, estimating animal pose (like tracking limbs), and segmenting continuous video into discrete actions. Evaluated across multiple species, the framework demonstrates superior performance, particularly in data-scarce scenarios common in lab settings. By drastically reducing dependency on labeled data, BEAST accelerates the pace of discovery in systems neuroscience, allowing researchers to more efficiently link brain function to observable behavior. This scalable tool promises to become a standard in labs worldwide, unlocking deeper insights into how neural circuits generate complex actions.