The unreasonable difficulty of time series forecasting

Summary

Preview I’ve been thinking recently about what makes time series forecasting problems so difficult compared to other sequence learning tasks or IID Machine Learning problems. First some motivating baselines illustrating the difficulty of the forecasting problem. I ran a series of models statistical (Naive, AutoARIMA, Theta, MSTL, Seasonal-Naive) linear/transformer neural (DLinear, NLinear, PatchTST) a gradient-boosted tree (LightGBM) zero-shot foundation models (Chronos, TimesFM, TTM) “AI” LLM prompted to continue numeric sequence (Claude Opus, Haiku) against the benchmark datasets m4_hourly, m4_daily etth exchange electricity bitcoin These are interactive plots, click legend to hide a line, double click to isolate. Note that the historical sections have been truncated to 3x the forecasting window, for better visualisation. Results There are many lines to look at, but the benchmark study’s main takeaway is that for series with strong, stable seasonality such as m4 hourly, the simple statistical baselines (Seasonal-Naive, MSTL) and the zero-shot foundation models win by large margins. However, the sophisticated models tend to be completely off for many time series tasks and are not much better than naive predictors because their predictions may trend in the completely wrong direction. We probably have some intuitions like, the forecast doesnt take into account causal information which is necessary for prediction, especially with real world data like exchange rates. But even with those “features” people are not naively making a killing in the market. Also, for many ML problems, we don’t have full information about the casual factors either yet are able to do something reasonable, or at least better than a random walk. Hence, this post investigates the nature of the forecasting problem and what makes it so much more difficult than classical machine learning. What’s going on here? Why Time Series (Forecasting) is hard Problem Definition Forecasting is the act of estimat...