Flow-Corrected Thompson Sampling beats forgetting in non-stationary bandits
New Bayesian method reuses past rewards with drift correction, outperforming standard forgetting baselines.
A new paper from AmirHossein Naghdi and Ali Baheri tackles the challenge of non-stationary contextual bandits—where reward models drift over time, making standard algorithms brittle due to systematic bias. The authors introduce Flow-Corrected Thompson Sampling (fcTS), a Bayesian method that reuses historical data by transporting past rewards to the present using an explicit drift model. Each transported observation is assigned a confidence weight reflecting transport reliability, allowing efficient reuse without discarding data.
fcTS specialises in three non-stationarity patterns: linear parameter drift via online slope estimation, periodic variation via phase-aware cross-cycle reuse, and recurring regime switches via changepoint detection with regime-specific memory. The posterior updates remain closed-form under a linear Gaussian model, implemented with truncated, incrementally updated sufficient statistics. In five controlled case studies and a semi-synthetic portfolio-selection benchmark with multiple overlapping non-stationarities, fcTS significantly outperformed forgetting-based baselines (discounting, sliding windows, periodic restarts). The largest gains occurred in settings with recurring temporal structure, demonstrating that correcting and reweighting historical observations offers substantially better sample efficiency than uniformly discarding them.
- fcTS uses an explicit drift model to transport past rewards to the present with confidence weights.
- Handles three non-stationarity types: linear drift, periodic cycles, and regime switches.
- Outperforms discounting, sliding windows, and restarts across five case studies and a portfolio-selection benchmark.
Why It Matters
Enables more sample-efficient AI decision-making in dynamic environments like financial markets and recommendation systems.