NEURRATOR decodes visual cortex spiking into semantic narration at single-cell resolution

NEURRATOR, developed by researchers from Harvard and MIT, addresses the long-standing challenge of identifying what individual neurons encode in higher-order visual cortex. Traditional methods rely on black-box deep-network embeddings that resist intuitive interpretation. The framework employs a learned encoder to project spike trains from arbitrary subsets of simultaneously-recorded neurons into the patch-embedding space of a frozen CLIP model. From there, a multimodal language model paired with a sparse autoencoder generates a free-form natural-language description of the viewed scene—without any training on the language side. This allows the system to produce semantic narrations directly from neural activity at single-cell resolution.

Applied to Neuropixel recordings of mouse visual cortex during natural-movie viewing, NEURRATOR demonstrates remarkable flexibility: it can narrate from recordings of thousands of neurons, individual cortical regions, local populations, or even molecularly-defined cell types. The team used this property to quantify how decoding fidelity scales with population size and cortical region, and to “neurrate” in plain language what individual neurons and genetically-tagged inhibitory cell-types contribute to visual representation. The work recasts cell identity from a static classification target into a functional probe, offering a new unit of biological insight in neural systems and opening the door to interpretable, language-based interrogation of brain activity.