Auto-differentiable data assimilation: Co-learning of states, dynamics, and filtering algorithms

A team of researchers led by Melissa Adrian, Daniel Sanz-Alonso, and Rebecca Willett has published a novel framework titled 'Auto-differentiable data assimilation: Co-learning of states, dynamics, and filtering algorithms.' The core innovation is 'auto-differentiable filtering,' a method that uses gradient-based optimization and auto-differentiation to simultaneously learn three critical components from incomplete and noisy observational data: the current state of a dynamical system, the underlying physical laws (dynamics), and the optimal parameters for the filtering algorithm itself. This unified approach tackles a major bottleneck in fields like weather forecasting and aerospace engineering, where traditional data assimilation requires expensive, expert-driven tuning and relies on having a perfectly accurate model.

The framework's versatility was demonstrated across multiple scientific domains, including experiments on the Clohessy-Wiltshire equations (aerospace), the chaotic Lorenz-96 system (atmospheric science), and the generalized Lotka-Volterra equations (systems biology). By providing a theoretically grounded loss function, the method allows practitioners to customize the learning process based on their specific observation models, accuracy needs, and computational constraints. This work effectively turns the complex, multi-step process of data assimilation into a more automated, end-to-end learning problem, potentially accelerating research and application in simulation-heavy sciences by reducing reliance on manual calibration and imperfect models.