ARIMA order selection without tuning on the future

ARIMA summarizes three choices as (p, d, q): autoregressive lags, differencing order and moving-average error lags. Searching those choices is model selection. If the search repeatedly consults the final test period, the resulting test score is no longer an honest assessment of an unseen future.

A defensible small experiment declares candidate orders first, fits them on training data, chooses using chronological validation, then evaluates once on a locked test period. Information criteria can supplement the comparison, but they do not replace an out-of-sample score tied to the forecasting decision.

An executed four-order comparison

The local lab fits four nonseasonal candidates on 96 months and forecasts the following 24 validation months.

Validation RMSE selects (0,1,1). AIC would select (1,1,0) in this run. That disagreement is useful: AIC estimates relative information loss under model assumptions, while validation RMSE measures performance on these future dates. The evaluation plan named validation RMSE as the selection criterion.

from timeseries_cases import arima_selection_case

result = arima_selection_case()
assert result["selection_metric"] == "validation_rmse"
assert result["selected_order"] == [0, 1, 1]
print(result["selected_order"])

This code executes under statsmodels 0.15.0 in the project-local extension environment. The lab supplies a linear trend term appropriate to its integrated specifications and stores the complete candidate table.

Keep the search space accountable

Do not expand candidates until one happens to look good without accounting for the extra search. Start from plots, domain cadence, differencing diagnostics and residuals. Record failed fits and convergence warnings rather than dropping them from the comparison. When automated search is used, freeze its ranges and criterion.

Check residual structure after selection. Remaining seasonal autocorrelation is a signal that a nonseasonal ARIMA family may be inadequate. Parameter significance alone does not establish forecast value, and a low training AIC does not guarantee resilience to a level shift.

Refit only after selection

After (p,d,q) is fixed, the selected model can be refit on training plus validation for a single test forecast. That refit uses more history without allowing the test score to choose the order. Preserve both the validation evidence and the final refitted artifact so another analyst can reproduce the sequence.

The fixture is authored and small. The difference of 0.04 RMSE between the top two orders is not evidence of a general advantage; it calls for stability checks across origins. The Data Science course teaches this boundary between model fitting, selection and assessment as part of an evaluated workflow.

Exercise

Add (2,1,0) and (0,1,2) to a predeclared second experiment. Run rolling validation and count order wins by origin and horizon. Keep the original locked test untouched until the new selection rule is fixed.

Continue learning

This article is part of the Forecasting and time-series analysis sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Stationarity: what differencing changes and what it loses.
• Continue with SARIMA for recurring seasonal demand.

References: statsmodels ARIMA API and Forecasting: Principles and Practice on ARIMA models.