Transformer decodes visual categories from brain signals with just 50 samples

A new study from researchers at Monash University and Osaka University demonstrates that a Transformer-based deep learning model can decode what people are watching from their brain activity with remarkable efficiency. Using electrocorticography (ECoG) data from 17 patients with drug-resistant epilepsy, the team built an end-to-end framework that requires fewer than 50 training examples per visual category. The best-performing system combined a Transformer encoder with mixup data augmentation and focused on high-gamma frequency bands (80-150 Hz) within a 900-millisecond post-stimulus window. This approach outperformed traditional handcrafted feature methods and achieved promising decoding accuracy from dynamic video stimuli.

The model's interpretability offered neuroscience insights: early visual cortex (areas V2-V4), the ventral visual stream, MT+ complex, and lateral temporal cortex contributed most to decoding performance. This matches established knowledge about hierarchical visual processing. The study, accepted for presentation at an upcoming machine learning conference, highlights how modern architectures can extract semantic information from noisy biological signals without manual feature engineering. While still preclinical, the work points toward more practical brain-computer interfaces that could eventually help paralyzed patients communicate or control devices directly from visual thoughts.