IM-LEPP model mimics multimodal cognition with energy-based hierarchy

Subir Varma's new arXiv paper (2608.12398) introduces IM-LEPP (Integrated Multimodal Latent Energy-based Predictive Processing), a 48-page hierarchical energy-based model that extends his prior single-modality LEPP framework into full multimodal cognition. Rather than simulating neural circuits, IM-LEPP treats generative neural networks as effective theories of cognitive dynamics, analogous to statistical mechanics in physics. The architecture uses a hub-and-spoke hierarchy grounded in Lambon Ralph's controlled semantic cognition framework, where separate predictive-coding pipelines for visual objects, scenes, and linguistic units converge on a shared amodal hub modeled on the anterior temporal lobe. Each pipeline's predictions are conditioned by, not overwritten by, the hub state, preserving modality-specific identity while incorporating full multimodal context.

The model provides mechanistic accounts of well-known phenomena like inattentional blindness and Necker-cube bistability. It also recovers or motivates established psycholinguistic findings including surprisal theory, the N400/P600 ERP components, and garden-path reanalysis. Critically, IM-LEPP offers a falsifiable contrast with transformer language models, particularly on trajectory-sensitivity in next-word prediction, and discusses data-efficient language acquisition compared to LLMs. The paper further outlines a semantic/episodic memory subsystem and compares the framework against predictive coding, the free-energy principle, JEPA, and Hierarchical Temporal Memory, ending with concrete experimental predictions to test its central claims.