Sequential correlations reshape in-context learning, Harvard study finds

The classic sandbox for studying in-context learning (ICL) has been linear regression with independently distributed examples—a clean but unrealistic assumption for natural sequences. A new theoretical paper from Harvard researchers generalizes this to sequentially correlated data, a fundamental property of real-world prompts like time series or conversational context. They first derive a solvable model based on linear attention and then verify predictions on realistic transformer architectures. When the query token is independent of its context, within-context correlations effectively shorten the prompt: correlated examples behave like a smaller set of i.i.d. examples, reducing the usable sample size for learning. This means that longer but highly correlated prompts do not provide proportional benefit, effectively truncating the model's accessible context length.

More strikingly, when the query token is itself correlated with the context, the story flips—test error can actually decrease. This benefit is especially pronounced for softmax attention compared to linear attention, revealing an architectural mismatch: softmax attention, with its data-dependent weighting, exploits query-context correlations more effectively. The findings suggest that optimal attention architecture depends on the correlation structure of real data, not just on raw depth or parameter count. For practitioners, this means that models optimized for independent tokens (common in benchmarking) may underperform on correlated real-world sequences. The paper provides a theoretical foundation for understanding when softmax attention's extra complexity is warranted, potentially guiding more efficient model design for tasks like financial forecasting, dialogue systems, and code completion where sequential dependencies dominate.