Decision tree depth: visualize overfitting on a small dataset

A deeper regression tree can make smaller partitions and follow more detail in the training observations. Some of that detail may be noise. Choose complexity using appropriate held-out development evidence rather than training error alone.

Our original synthetic duration fixture has 160 training rows, 80 validation rows and 80 test rows. We compare three declared candidates: maximum depth two, maximum depth four and unrestricted depth. Only load index is a predictor; the known generator mean is reserved for teaching diagnostics.

Compare fitted shapes and measured errors

Open the full-size SVG for zooming. The green mean curve is known because this is a generated teaching dataset; it would not be directly available in a real prediction problem.

The unrestricted tree reaches fitted depth 15 and one leaf per training observation. Its perfect training fit does not produce the best validation result. Depth four wins among these three candidates.

Reproduce the selection rule

import numpy as np
from tree_cases import depth_case

r = depth_case()
assert r['selected_depth']==4
assert [row['leaves'] for row in r['candidates']]==[4,16,160]
assert r['candidates'][-1]['training_rmse']==0
assert r['candidates'][-1]['fitted_depth']==15
assert r['candidates'][1]['validation_rmse'] < r['candidates'][-1]['validation_rmse']
assert np.isclose(r['selected_test_rmse'],2.0738093439716376)
print(r)

Run from the supervised-model lab. The selected depth-four model has test RMSE about 2.073809 in this fixture. Test error did not choose the depth in this exercise.

The DecisionTreeRegressor reference documents depth and leaf controls. The figure, data and numerical comparison above are original.

Explain what the result supports

The depth-two fit is too coarse to represent parts of the curved mean well. The unrestricted fit captures much of the observation-level variation. Depth four provides a better validation tradeoff in this particular sample.

This does not establish depth four as a universal recommendation. Sample size, noise, feature dimension and the target relationship all influence useful tree complexity. The shared instructional fixture is also not an independent production benchmark for every algorithm discussed in the article series.

Consider more than maximum depth

Minimum leaf size, minimum split size and pruning can constrain a tree in different ways. A depth limit bounds the number of successive decisions; a leaf-size requirement constrains how many training observations support a prediction region.

Inspect performance on meaningful slices and near feature-range boundaries. Standard regression trees return leaf values and do not automatically extrapolate a growing trend beyond the observed training range.

If you expand the candidate search after inspecting results, record that development. Do not repeatedly tune on the final test set while continuing to label it untouched.

Exercise: compare two prespecified minimum-leaf-size settings with the depth-four reference using validation data. Report leaf counts, errors and which regions changed most. Explain what fresh evidence would be needed after selecting a revised procedure.

NeuraPath's Data Science course connects tree structure with evaluation. A useful overfitting explanation shows both the fitted behavior and the held-out evidence that supports the complexity choice.

Continue learning

This article is part of the Supervised learning methods sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Elastic net: combine shrinkage and feature selection.
• Continue with Random forests: understand out-of-bag estimates and their limits.