Rolling-origin backtesting with multiple forecast horizons

A single holdout score can hide when and where a forecast fails. Rolling-origin backtesting reconstructs several historical decisions: fit or update using only the data available at an origin, forecast forward, move the origin, and repeat. The evaluation therefore respects time while sampling more than one operating condition.

The method is sometimes called walk-forward validation or time-series cross-validation. Names vary; the essential contract is that every prediction must be generated without observations later than its own forecast origin.

Separate horizons before averaging

The local lab applies a 12-month seasonal-naïve rule at three origins and four horizons. The measured MAEs are:

The last origin crosses the authored structural break and degrades sharply. One grand average would erase that information. It would also mix scores based on different decision distances: a one-month replenishment forecast and a 12-month capacity forecast solve different problems.

from timeseries_cases import rolling_case

result = rolling_case()
assert result["origins"] == 3
assert len(result["evaluations"]) == 12
latest = [row for row in result["evaluations"] if row["origin_period"] == "2024-12-01"]
print([(row["horizon"], round(row["mae"], 2)) for row in latest])

The printed horizon-score pairs are (1, 19.14), (3, 22.30), (6, 19.73) and (12, 20.56) after rounding.

Match the historical production process

Choose expanding windows when all prior history would have remained available. Choose sliding windows when older regimes should expire or storage and latency impose a fixed lookback. Refit at the cadence production would use. A model retrained monthly in a backtest does not represent a production model refreshed quarterly.

Record how overlapping targets are aggregated. Monthly origins with a 12-month horizon score some calendar months repeatedly. That is legitimate if it mirrors repeated planning decisions, but each observation is no longer an independent replicate. Report counts by horizon and, where useful, error by origin date.

Features need the same walk-forward treatment. A rolling mean for the target month must end before that target. Known future covariates must genuinely have been known at each historical origin. Reconstruct versioned promotion plans where possible; a cleaned final calendar can leak schedule changes made later.

Use the backtest to answer decisions

Specify the primary horizon and metric before comparing candidates. Retain breakdowns for volatile periods, sparse items and high-value segments. Estimate operational cost if asymmetric errors matter. Then make one final assessment on a test period that was not used to decide window size, feature set or model family.

The fixture deliberately reveals its break in generator metadata for teaching. In an actual backtest, analysts see the error pattern first and investigate the cause; they should not claim the mechanism from residuals alone.

The Data Science course places this evaluation design beside model building so that a forecasting project can be audited from cutoff to score.

Exercise

Add origins every six months. Create a result table with origin, horizon, number of targets and MAE. Then weight errors using a declared planning-volume column and compare the conclusion with unweighted MAE.

Continue learning

This article is part of the Forecasting and time-series analysis sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Seasonal naive forecasting as a serious baseline.
• Continue with MAE, MAPE and WAPE when demand contains zeros.

Reference: Forecasting: Principles and Practice on time-series cross-validation.