New SDE Model Explains Error Floor in TD Learning with Linear Features
A diffusion approximation reveals why constant-stepsize TD(0) hits an error floor under Markovian noise.
Deep Dive
A stochastic differential equation (SDE) approximation is introduced for linear TD(0) under Markovian noise. Unlike the traditional ordinary differential equation (ODE), this model distinguishes the contraction dynamics governed by the projected Bellman operator from the influence of Markovian sampling. Consequently, it explains the constant-stepsize error floor through the interaction between Markovian long-run covariance and the contraction geometry of the projected Bellman operator.
Key Points
- Introduces an SDE approximation for linear TD(0) that captures stochastic fluctuations beyond the classical ODE.
- Separates the projected Bellman operator's contraction dynamics from Markovian sampling effects.
- Explains the constant-stepsize error floor via the interaction of Markovian long-run covariance and contraction geometry.
Why It Matters
Provides a more accurate theoretical model for TD learning, helping researchers tune algorithms in reinforcement learning.