Early stopping with a reproducible validation policy

Early stopping is model selection over training epochs. It needs the same discipline as choosing architecture: a fixed validation population, one monitored quantity, an improvement rule, patience, a maximum budget and a saved best checkpoint. Stopping without restoring the best parameters leaves the model from a later, worse epoch.

An executed stopping trace

The neural lab fits a noisy-feature classifier for at most 500 epochs and stops after 25 epochs without sufficient validation improvement.

from deep_learning_cases import early_stopping_case

result = early_stopping_case()
assert result["patience"] == 25
assert result["stopped_epoch"] == result["best_epoch"] + result["patience"]
assert result["restored_validation_log_loss"] <= result["final_validation_log_loss"]
print(result["best_epoch"], result["stopped_epoch"])

The best epoch is 135 and training stops at 160. Restored validation log loss is 0.36276, compared with 0.36342 for the last weights. The numerical improvement is small; the important result is that the selected artifact matches the declared rule.

Write the policy before fitting

Specify whether an epoch starts at zero or one, how ties are handled, the minimum change, patience and the direction of improvement. Record how often validation runs. Patience measured in validations differs from patience measured in optimizer steps.

Keep preprocessing fitted on training data. For grouped, temporal or subject-level data, construct validation with the same separation required for test. A random internal validation fraction can violate that design.

Test remains closed

Changing patience after seeing test performance tunes on test. Select patience alongside other training settings on validation or nested resampling. Then evaluate the restored checkpoint once on the holdout.

Early stopping also makes training budget variable. Report both best and stopped epochs, optimizer steps and wall time. When comparing candidates, give all of them the same maximum opportunity and consistent monitoring cadence.

Save the model state, preprocessing, optimizer or scheduler state if resumption is required, selection metric and software versions. A plot without the checkpoint cannot reproduce the released model.

The Data Science course treats checkpoint selection and reload verification as evidence, not notebook housekeeping.

Exercise

Compare patience 5, 25 and 50 over several seeds. Select patience using validation only, report compute and restored loss, then verify that serialization produces identical predictions.

Continue learning

This article is part of the Deep learning and computer vision sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Dropout and weight decay: different regularization mechanisms.
• Continue with CNN image classification with a clean data split.

Reference: scikit-learn early-stopping parameters for MLPClassifier.