KDA uses LLMs to decompose actions for state-of-the-art recognition

A team of researchers led by Tianci Wu has introduced KDA (Knowledge-guided Disentanglement with Atomic Actions), a novel framework for multi-label action recognition in complex video scenes. Traditional methods rely on holistic representations that fail to capture fine-grained interactions between concurrent actions. KDA overcomes this by first using Large Language Models (LLMs) to break down action labels into atomic actions—discrete, semantically meaningful steps like 'grasp', 'lift', or 'rotate'—which provide explicit spatial-temporal cues. The framework then employs a Knowledge Injection Module (KIM) to embed these atomic action descriptions into video feature representations. Subsequently, a Knowledge Disentanglement Module (KDM) separates the atomic action knowledge from the enhanced features, guided by a dedicated Knowledge Disentanglement Loss (KD Loss) that promotes clear separation of knowledge components. This layered approach enables the model to distinguish subtle differences between overlapping actions.

Extensive experiments on standard multi-label action recognition benchmarks show that KDA significantly improves feature discriminability over prior methods, achieving state-of-the-art performance. Notably, both KIM and KDM are designed as plug-and-play modules that can be seamlessly integrated into existing action recognition architectures, offering broad applicability. The paper, accepted at ACMMM 26 and available on arXiv, demonstrates that leveraging LLMs for fine-grained semantic decomposition is a powerful strategy for disentangling complex actions. This work has practical implications for video surveillance, sports analytics, human-computer interaction, and any domain requiring precise understanding of simultaneous activities. The combination of explicit knowledge injection and disentanglement loss provides a clear path for future research in fine-grained video understanding.