New AIC Reinforcement Learning Controls Nonlinear Systems Despite Packet Loss

Packet dropouts in control systems—when data packets are lost between sensors, controllers, and actuators—pose a critical challenge to system stability and performance. Traditional controllers rely on accurate mathematical models of the system dynamics, but these models are often unavailable or inaccurate in real-world conditions. To address this, researchers from multiple institutions have introduced an Actor-Identifier-Critic (AIC) reinforcement learning framework that operates entirely model-free, learning the system's behavior online and adapting control actions in real time.

The AIC architecture consists of three neural networks: an identifier that learns the system dynamics from observed data, a critic that estimates the value (long-term cost) of the current control policy, and an actor that outputs optimal control actions. The key innovation is that the identifier enables gradient propagation from the critic to the actor even when packets are dropped in the communication channels—both from controller to actuator and from sensor to controller. This allows the policy to be updated continuously without requiring a perfect model of the system, making the controller robust to unpredictable communication failures.

The researchers validated their approach on two nonlinear benchmark systems (single-input multiple-output and multiple-input multiple-output) and a practical case study involving power system stability under stochastic packet dropouts. Results showed that the AIC controller maintained stable operation and near-optimal tracking performance even when 30-40% of packets were lost, dramatically outperforming classical proportional-integral-derivative and model-predictive controllers. The framework also demonstrated faster convergence and lower steady-state error compared to existing reinforcement learning methods that do not account for packet loss.

This work has significant implications for critical infrastructure that relies on networked control, such as smart grids, autonomous vehicles, and industrial automation. By removing the need for accurate system models and explicitly handling communication failures, the AIC controller offers a practical, scalable solution for real-world deployment where network reliability cannot be guaranteed.