Gradient boosting: why weak learners can overfit together

Small base trees do not make the combined model incapable of overfitting. As gradient boosting adds trees, the ensemble can represent increasingly detailed patterns, including noise that does not improve predictions on new observations.

Our original synthetic duration example uses depth-two trees, learning rate 0.1 and a fixed seed. We inspect five declared checkpoints: 10, 50, 100, 300 and 600 trees. Training and validation use the same fixed partitions as the supervised-model lab.

Compare the two error paths

Open the full-size SVG for zooming. The lines join the five evaluated checkpoints; they do not show every intermediate boosting stage.

Validation selects 50 trees from this declared set. The 600-tree ensemble fits training observations more closely but generalizes less well to the validation sample.

Reproduce staged selection

import numpy as np
from tree_cases import boosting_case

r = boosting_case()
assert r['selected_trees']==50
rows=r['candidates']
assert [row['trees'] for row in rows]==[10,50,100,300,600]
assert all(a['training_rmse']>=b['training_rmse'] for a,b in zip(rows,rows[1:]))
assert rows[-1]['validation_rmse']>rows[1]['validation_rmse']
assert np.isclose(r['selected_test_rmse'],1.9758852473201953)
print(r)

The lab fits the full sequence and retrieves predictions at the declared stages. It then uses the validation-selected stage for the test calculation. This is staged model selection, not an implementation that stops training early through a callback.

Scikit-learn's GradientBoostingRegressor reference documents staged_predict, which exposes successive ensemble predictions for this comparison.

Explain what additional trees are doing

For squared-error boosting, successive learners address residual structure left by the current ensemble. Early additions can capture useful signal. Later additions may increasingly fit sample-specific variation, depending on the data and regularization.

The known synthetic mean is curved and observation noise varies with load. These controlled conditions explain the example, but its selected tree count is not a universal setting for other problems.

Learning rate, tree depth, minimum leaf size, subsampling and the number of trees interact. A smaller learning rate can require more stages to fit useful structure; it does not eliminate the need for validation. Changing several settings creates a larger selection procedure that should be recorded and assessed accordingly.

Preserve the assessment boundary

The test result for the selected 50-tree stage is RMSE about 1.975885 in this instructional fixture. The test score did not select the checkpoint in this exercise. The shared dataset across algorithm lessons is not an independent production benchmark for choosing a universally best family.

Do not increase the checkpoint search repeatedly after inspecting test performance while retaining the claim that the final assessment was untouched. Use development evidence to refine a procedure and obtain appropriate new assessment evidence when needed.

Exercise: declare a second learning rate and a small checkpoint grid before running it. Compare validation behavior and computational effort, preserve all attempted settings, and explain how you would evaluate the selected procedure without using test labels for the choice.

NeuraPath's Data Science course connects ensemble algorithms with practical diagnostics. A convincing explanation of boosting includes the stage at which added training fit stops helping the validation result.

Continue learning

This article is part of the Supervised learning methods sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Random forests: understand out-of-bag estimates and their limits.
• Continue with XGBoost validation: early stopping without test-set leakage.