Time series forecasting

Weak stationarity requires a constant mean, constant (finite) variance, and an autocovariance that depends only on the lag, not on absolute time. ARMA models assume stationarity because their parameters describe a fixed autocovariance structure; fitting them to non-stationary data yields spurious or unstable estimates. Differencing or detrending is applied first to induce stationarity (the 'I' in ARIMA).

For a pure AR(p) process the PACF cuts off (drops to ~0) after lag p while the ACF tails off gradually. For a pure MA(q) process the ACF cuts off after lag q while the PACF tails off. Mixed ARMA processes show both tailing off, so identification relies more on information criteria.

The ADF test has a null hypothesis of a unit root (non-stationarity), so rejecting it suggests stationarity. The KPSS test reverses this: its null is (trend-)stationarity, so rejecting it suggests non-stationarity. Using both together is good practice-agreement increases confidence, and conflicting results often indicate a series that is difference- vs trend-stationary.

Standard k-fold shuffles data, allowing future observations into the training set used to predict the past-this leaks information and inflates performance. The correct approach is forward-chaining (rolling/expanding window) cross-validation where each fold trains only on data preceding the validation period. Scikit-learn's TimeSeriesSplit implements this.

from sklearn.model_selection import TimeSeriesSplit
tscv = TimeSeriesSplit(n_splits=5)
for train_idx, test_idx in tscv.split(X):
    X_tr, X_te = X[train_idx], X[test_idx]

SARIMA models a univariate seasonal series using its own past values, errors, and seasonal terms. SARIMAX adds eXogenous regressors-external predictors (e.g., price, promotions, weather) that influence the target. You choose SARIMAX when known external drivers carry predictive signal beyond the series' own history.

Holt-Winters maintains three smoothed components: level, trend, and seasonality, each updated with its own smoothing parameter (alpha, beta, gamma). The additive form assumes seasonal swings of roughly constant magnitude, while the multiplicative form assumes seasonal amplitude scales with the level. Use multiplicative when seasonal fluctuations grow proportionally with the series.

A first difference (y_t - y_{t-1}) removes a stochastic trend/unit root at the regular frequency. A seasonal difference (y_t - y_{t-s}) removes seasonal non-stationarity by subtracting the value one full season ago, eliminating recurring seasonal patterns and seasonal unit roots. Series often need both a regular and a seasonal difference (orders d and D in SARIMA).

MAPE divides by actuals, so it explodes or is undefined near zero values and is asymmetric: an over-forecast can produce an unbounded percentage error while an under-forecast is capped at 100%, biasing model selection toward forecasts that are too low. Alternatives include sMAPE (symmetric), MASE (scaled by a naive benchmark, scale-free and robust to zeros), and RMSE/MAE for absolute-scale comparison. MASE is often preferred for comparing across series of different scales.

MASE divides the model's mean absolute error by the in-sample MAE of a naive (random-walk or seasonal-naive) forecast. A MASE below 1 means the model beats the naive benchmark on average; above 1 means it is worse than simply predicting the last (or last-seasonal) value. Being scale-free, MASE allows comparison across series with different units.

Prophet models trend as a piecewise-linear (or logistic-growth) function whose slope can change at automatically-detected or user-specified changepoints. A sparse (Laplace) prior on the rate adjustments controls flexibility, preventing overfitting. Seasonality is added separately via Fourier series and holidays via dummy regressors, making Prophet an additive decomposable model.

Prophet represents each seasonal cycle as a truncated Fourier series-a sum of sine/cosine pairs of a fixed period. Increasing the Fourier order adds higher-frequency terms, letting the seasonal curve capture sharper, more complex patterns, but also raises the risk of overfitting noise. The default is 10 for yearly and 3 for weekly seasonality.

N-BEATS is a deep, pure feed-forward (no recurrence/attention) architecture of stacked blocks, each producing a backcast and a forecast via doubly-residual stacking. Each block fits and subtracts what it can explain, passing the residual onward, which yields strong accuracy and an interpretable trend/seasonality decomposition variant. It set state-of-the-art results on M3/M4 without time-series-specific feature engineering.

DeepAR trains a single global autoregressive RNN jointly across many related series, learning shared patterns and enabling cold-start forecasts for series with little history. It outputs probabilistic forecasts by predicting the parameters of a likelihood (e.g., Gaussian or negative binomial) and sampling. ARIMA, by contrast, is fit locally per series and cannot share statistical strength across them.

The pinball loss for quantile tau penalizes under-prediction by tau*(y-yhat) and over-prediction by (1-tau)*(yhat-y), making its minimizer the conditional tau-quantile. Training a model to output several quantiles with their respective pinball losses produces calibrated prediction intervals directly. It is asymmetric, which is exactly what is needed to target non-median quantiles.

import numpy as np
def pinball(y, yhat, tau):
    e = y - yhat
    return np.mean(np.maximum(tau * e, (tau - 1) * e))

For many near-random-walk financial and high-noise series, the conditional expectation of the next value is close to the current value, so the naive forecast is hard to beat and complex models mostly fit noise. This is why every forecasting evaluation should include naive/seasonal-naive baselines; a model that cannot beat them adds no value. MASE explicitly encodes this benchmark comparison.

Recursive forecasting trains one one-step model and feeds its predictions back as inputs to forecast further horizons, which can accumulate errors but reuses a single model. Direct forecasting trains a separate model for each horizon h, avoiding error propagation but ignoring dependencies between horizons and costing more to train. Hybrid (DirRec) and multi-output strategies trade off between them.

TFT uses gated variable selection networks at each timestep to learn which static, known-future, and observed inputs matter, suppressing irrelevant features. Its interpretable multi-head attention shares value weights across heads so attention weights reveal which past timesteps drive a forecast. Together they give strong accuracy plus feature- and temporal-importance interpretability, with quantile outputs for uncertainty.

X Granger-causes Y if past values of X improve the prediction of Y beyond Y's own past, tested via nested regressions (an F-test). Its limitation is that it captures predictive precedence, not true causation-a confounder driving both, or omitted variables, can produce spurious Granger causality. It also assumes (linear) stationarity and the right lag specification.

Lag order is typically chosen by minimizing an information criterion (AIC, BIC/SC, or HQIC) over candidate lags, balancing fit against parameter count (VAR parameters grow with k^2 per lag). Adequacy is checked by testing residuals for remaining autocorrelation (e.g., Portmanteau/Ljung-Box) and confirming stability-the companion-matrix eigenvalues must lie inside the unit circle (equivalently, the characteristic roots lie outside it). Cointegration tests determine whether a VECM is more appropriate.

The Ljung-Box test checks whether a group of residual autocorrelations up to lag m are jointly zero (null = no autocorrelation, i.e., residuals are white noise). For a well-specified forecasting model you want a high p-value (fail to reject), indicating residuals contain no leftover predictable structure. A low p-value signals the model missed autocorrelation that should be modeled.

ARIMA models the conditional mean; GARCH models the conditional variance, capturing volatility clustering where large changes follow large changes. A GARCH(p,q) makes today's variance a function of past squared shocks and past variances, producing time-varying, mean-reverting volatility forecasts. They are typically fit on the residuals of a mean model (ARIMA-GARCH) for financial returns.

Target leakage occurs when a feature encodes information not available at prediction time, producing optimistic backtests that collapse in production. Examples include using a centered rolling mean (which peeks ahead), normalizing with statistics computed over the whole series including the future, or joining a feature recorded after the forecast timestamp. All windows, scalers, and aggregations must use only past data, computed within each training fold.

A Box-Cox/log transform stabilizes variance (heteroskedasticity) and can make multiplicative seasonality additive and the series more Gaussian, improving model fit and interval validity. You forecast in the transformed space then invert the transform, but a naive inverse of the mean is biased-back-transforming requires a bias correction (e.g., adding half the forecast variance before exponentiating for a log) to recover the mean rather than the median.

Regressing two independent integrated (I(1)) series often yields a high R-squared and significant t-stats purely from shared trending, even with no real relationship-classic spurious regression. The fix is to test for cointegration (e.g., Engle-Granger or Johansen): if cointegrated, use a Vector Error Correction Model; otherwise, model the differenced (stationary) series. Checking that residuals are stationary is the key diagnostic.

STL (Seasonal-Trend decomposition using Loess) uses iterative locally-weighted regression to extract trend and seasonal components, allowing the seasonal pattern to change over time and being robust to outliers via robustness weights. Naive additive/multiplicative decomposition uses fixed moving averages and assumes a constant seasonal shape. STL handles any seasonal period, evolving seasonality, and tolerates outliers better, though classical STL assumes additive composition (log first for multiplicative effects).

Concept drift is a change over time in the statistical relationship the model learned (e.g., shifting demand patterns, regime changes), degrading a static model's accuracy. Mitigations include periodic or triggered retraining, rolling/sliding training windows that down-weight old data, online/incremental learning, and monitoring error or distribution-shift metrics to detect drift. Maintaining a champion-challenger setup helps decide when to refresh the model.

Standard smoothing/regression on intermittent series is biased and reacts poorly because zeros dominate and demand arrives sporadically. Croston's method separately smooths the non-zero demand sizes and the inter-arrival intervals between demands, then combines them into a rate forecast. Variants like SBA (Syntetos-Boylan Approximation) correct Croston's known positive bias.

The scaler (mean/std, min/max) must be fit only on each fold's training data and then applied to its validation/test slice; fitting on the whole series leaks future statistics into the past. Because validation always lies in the future relative to training in forward-chaining CV, any global preprocessing breaks temporal causality. Use pipeline objects so the transform is refit inside every fold.

GBMs handle many engineered features (lags, rolling stats, calendar/exogenous variables) and complex non-linear interactions with little tuning, often winning tabular forecasting competitions. Their main limitation is the inability to extrapolate beyond the training range-because trees predict piecewise-constant values, a strong upward trend must be removed by differencing/detrending before modeling. They also need explicit lag/feature construction since they have no built-in temporal memory.

AIC = 2k - 2*ln(L) rewards goodness-of-fit (log-likelihood L) while penalizing the number of parameters k; lower is better. BIC = k*ln(n) - 2*ln(L) imposes a stronger, sample-size-dependent penalty, so it favors simpler (more parsimonious) models and is consistent for selecting the true model, whereas AIC is asymptotically efficient for predictive performance. They can disagree, with BIC choosing fewer terms on large samples.

Walk-forward evaluation repeatedly trains on data up to an origin, forecasts the next horizon, then advances the origin and refits/updates, aggregating errors across many origins. It mimics real deployment where the model is periodically retrained as new data arrives, giving a robust, low-variance estimate of out-of-sample accuracy across regimes. A single train/test split, by contrast, can be lucky or unlucky depending on the chosen cut point.

Forecasting each level independently produces incoherent totals (children don't sum to parents). Reconciliation methods enforce coherence: bottom-up sums leaf forecasts; top-down disaggregates a total by proportions; and optimal reconciliation (MinT) projects base forecasts onto the coherent subspace using the forecast-error covariance, typically improving accuracy at all levels. MinT is the modern default because it uses information from every level.

A 90% prediction interval is calibrated if, over many forecasts, the actual value falls inside it about 90% of the time. You check this with empirical coverage (the realized hit rate per nominal level) and reliability/PIT diagrams; coverage well below nominal means overconfident intervals, above means too wide. Proper scoring rules like CRPS or the interval/Winkler score jointly assess sharpness and calibration.

If non-stationarity comes from a deterministic trend rather than a unit root, differencing is unnecessary and can be harmful; detrending (or including a trend regressor) is more appropriate. Over-differencing inflates the variance of the series, introduces an artificial (non-invertible) MA unit root, and complicates the error structure, hurting forecasts. The KPSS-vs-ADF distinction (trend- vs difference-stationary) guides the correct remedy.