Forecast collapse of transformer-based models under squared loss in financial time series

A new theoretical paper by Pierre Andreoletti provides a formal explanation for why powerful Transformer-based AI models, the architecture behind systems like GPT-4 and Llama 3, often fail at financial time series forecasting. The research identifies a 'forecast collapse' regime common in finance, where the conditional expectation of future price movements is effectively degenerate. In simpler terms, the mathematically optimal prediction for prices is often flat, and for returns is zero. In this scenario, increasing model complexity offers no benefit for reducing bias.

Instead of converging on this simple optimal forecast, highly expressive Transformer models introduce what the paper calls 'spurious trajectory fluctuations.' These are non-predictive patterns where the model essentially learns to replicate and reuse noise from the training data. This mechanism increases prediction variance without any corresponding reduction in bias, leading to worse overall performance. The theory was validated with numerical experiments on high-frequency EUR/USD exchange rate data, where Transformer-based forecasts yielded larger errors than a basic linear benchmark on a large majority of test windows, confirming the variance-driven degradation.