Neuro-Symbolic Manipulation Understanding with Enriched Semantic Event Chains

Robots operating in human environments face a fundamental challenge: they must understand not just static scenes but how object interactions evolve over time. A new neuro-symbolic framework called eSEC-LAM, developed by Fatemeh Ziaeetabar, tackles this by transforming classical enriched Semantic Event Chains (eSECs) into explicit, uncertainty-aware symbolic states. The system augments traditional eSECs with confidence-aware predicates, functional object roles, and affordance priors, enabling robots to reason about current actions and predict next manipulation steps using lightweight symbolic reasoning over primitive pre- and post-conditions.

The framework was rigorously evaluated on three benchmark datasets—EPIC-KITCHENS-100, EPIC-KITCHENS VISOR, and Assembly101—across multiple tasks. Results show eSEC-LAM achieves competitive action recognition while substantially improving next-primitive prediction accuracy by up to 40% compared to both classical symbolic and end-to-end video baselines. Crucially, the system maintains robustness under degraded perceptual conditions and provides temporally consistent explanation traces grounded in explicit relational evidence. This demonstrates that enriched Semantic Event Chains can serve not just as interpretable descriptors but as effective internal states for neuro-symbolic action reasoning, bridging the gap between neural perception and symbolic planning.