Learning curves: decide whether more data may help

A learning curve compares model performance as training-set size changes. It can reveal whether a method is still improving over the observed range, whether its training-validation gap is shrinking, and how sensitive a conclusion is to the sampled rows. It cannot promise performance at a larger unobserved size.

Preserve the validation role

Our experiment uses the synthetic duration fixture. The same 80 validation rows score every fit. For each of ten seeded permutations of the 160 training rows, we fit nested subsets of 40, 80 and 160 observations. No test row enters the curve.

We compare a misspecified straight-line regression with a depth-four regression tree. The duration generator contains a curved relationship, so the pair exposes different behaviour.

The tree's validation median improves and its train-validation gap narrows across the observed sizes. The linear model changes little because additional samples do not repair its missing curvature. These statements apply to this construction and range.

The scikit-learn learning-curve documentation describes score computation across training sizes. Our lab uses explicit nested subsets so the identities and fixed validation role remain easy to inspect.

from decision_cases import learning_case

r = learning_case()
assert len(r['runs']) == 60
assert len(r['summary']) == 6
tree = [row for row in r['summary'] if row['model'] == 'depth_four']
assert [row['size'] for row in tree] == [40, 80, 160]
assert tree[-1]['validation_rmse_median'] < tree[0]['validation_rmse_median']
print(r['summary'])

Run the reproduction in the supervised-model lab. The ten values at sizes 40 and 80 reuse the same training pool and validation rows. Their min-max ranges are descriptive; they are not confidence intervals from independent experiments. At size 160, every nested subset is the full training set, so all ten fits coincide.

Diagnose before buying more labels

A large training advantage that shrinks as size grows may suggest variance and a possible benefit from more representative data. High, similar training and validation error can point to model bias, weak features, label noise or an unsuitable target. It does not automatically mean “use a larger model.”

Curves should follow the deployment split. Random row subsets can exaggerate performance if customers repeat or time changes. For grouped data, grow the number of groups; for forecasting, expand historical windows without allowing future information backward. Fit preprocessing inside each training subset.

Before collecting more data, inspect where it would come from. More rows from an overrepresented easy segment may lower aggregate error while leaving the critical slice unchanged. Add per-slice curves, label-quality audits, collection cost and outcome-maturity delay to the decision.

Exercise: reproduce the curve with group-sized or chronological subsets on a suitable dataset. Predeclare the metric improvement that would justify another collection batch, and explain why extrapolating a smooth power law beyond the observed range would require assumptions and uncertainty bounds.

NeuraPath's Data Science course teaches learning curves as evidence for a data decision. The chart becomes useful when sample identity, split rules and limits travel with it.

Continue learning

This article is part of the Supervised learning methods sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Cost-sensitive classification with an explicit loss table.
• Continue with Partial dependence: recognize correlated-feature limitations.