Forecast evaluation during structural breaks

A structural break changes the relationship that generated earlier observations: level, trend, seasonality, variance or feature effects may shift. A model can pass historical validation and then become biased because its old pattern no longer represents current operations.

Forecast evaluation should show this temporal behavior. One aggregate score across pre- and post-break dates can make a severe recent failure look acceptable.

A controlled before-and-after result

The local lab gives a 12-month seasonal-naïve forecast two consecutive 24-month evaluations. Validation is before an authored break; test begins at the break on 1 January 2025.

from timeseries_cases import break_case

result = break_case()
assert result["break_start"] == "2025-01-01"
assert result["test_mae_post_break"] > result["validation_mae_pre_break"]
print(round(result["validation_mae_pre_break"], 2), round(result["test_mae_post_break"], 2))

The post-break MAE is almost three times the pre-break value and the positive bias shows persistent underforecasting. Because the fixture generator labels the intervention, the lab knows the mechanism. In real work, a change in residuals can flag a break but cannot establish whether pricing, supply, competition or data collection caused it.

Detect change with a declared policy

Monitor signed error, absolute error, interval coverage and feature distributions in time order. Define the comparison window, minimum observations and alert threshold before examining a suspected incident. Segment by product and region so a portfolio average does not hide a local break.

Distinguish target delay from model delay. If actuals arrive weeks later, a monitoring dashboard cannot detect failure immediately. Separate provisional sales from settled demand and preserve revision timestamps.

Respond without erasing evidence

When an alert fires, check data completeness, calendar versions and operational constraints first. Compare the model with its baseline. Create a post-event evaluation slice, but retain the original locked test result in the model record. Retraining on the new regime can help after enough observations arrive; a one-month anomaly may instead lead to overreaction.

Possible responses include an approved level override, heavier weighting of recent data, a shorter training window, a regime indicator known at forecast time, or a replacement model. Backtest the response across earlier pseudo-breaks where possible. Record who can override, how long an override lasts and how the system rolls back.

The Data Science course integrates this monitoring logic with temporal evaluation and deployment rather than treating an initial test score as permanent.

Exercise

Create rolling signed-error and MAE charts around the authored break. Define an alert using only information available at each month, measure its detection delay, and compare a level override with refitting after 3, 6 and 12 new observations.

Continue learning

This article is part of the Forecasting and time-series analysis sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Detect data leakage in holiday and promotion features.
• Continue with Compare direct and recursive multi-step forecasting.

Reference: Forecasting: Principles and Practice on time-series features and changes.