Bias and variance with repeated training samples

Bias describes systematic prediction error across repeated training samples. Variance describes how predictions change across those samples. A single fitted model and its training score cannot directly reveal both quantities.

We can measure them in a simulation because we specify the true relationship. Our original experiment uses f(x)=sin(pi*x), 25 fixed training locations between minus one and one, and independent normal training noise with standard deviation 0.3. Five hundred training responses share those same locations. We fit polynomial degrees one, three and nine to each response sample.

Hold the comparison conditions fixed

The three model families receive the same 500 noisy response arrays, generated with integer seed 20261002. Predictions are evaluated at 101 fixed locations inside the training range. This paired design reduces irrelevant differences between the model comparisons.

Only training noise varies here. The results are conditional on the fixed design, chosen function, sample size and estimator. They do not summarize every possible random training dataset or prove that a particular polynomial degree is generally best.

Read the executed results

The straight line misses the curved relationship and has high bias. Degree nine approximates the mean relationship closely but varies more with training noise. Degree three has the smallest expected squared error among these three choices in this experiment.

These are recorded simulation results, not real customer predictions or a general leaderboard. The scikit-learn bias–variance example provides primary conceptual context; the function, fixtures and comparison above are our own.

Verify the decomposition

Run from the mathematics lab:

import numpy as np
from model_geometry import bias_variance

rows = bias_variance()
for row in rows:
    assert np.isclose(row['squared_bias']+row['variance'],
                      row['mse_against_noiseless_truth'])
    assert np.isclose(row['expected_mse_fresh_noise'],
                      row['mse_against_noiseless_truth']+.09)
assert rows[2]['squared_bias'] < rows[1]['squared_bias']
assert rows[2]['variance'] > rows[1]['variance']
assert rows[1]['expected_mse_fresh_noise'] < rows[2]['expected_mse_fresh_noise']
print(rows)

The empirical identity uses variance with divisor 500, equivalent to ddof=0, because it decomposes the mean squared error over this finite collection of predictions. Switching only the variance calculation to ddof=1 changes the identity.

Fresh independent outcome noise has variance 0.09 and mean zero. Adding 0.09 gives expected error over that fresh noise, conditional on these fitted predictions. We did not draw a separate noisy test set and claim that its finite observed error must equal this expectation exactly.

Use the lesson without pretending to know real-world truth

In a real project, the true regression function is unavailable. Learning curves, repeated suitable splits and residual analysis can diagnose problems, but they do not automatically identify exact bias and variance components.

A flexible model may also extrapolate badly outside the observed feature range. Our grid stays inside minus one to one, so this experiment offers no evidence about extrapolation.

Exercise: change training noise to 0.6 while preserving the seed and design. Recompute every component and update the irreducible noise term to 0.36. Explain why reporting only the degree-nine bias would give an incomplete recommendation.

NeuraPath's Data Science course connects these foundations with practical model evaluation. The aim is to explain the error tradeoff that supports a model choice, with the experiment's scope visible.

Continue learning

This article is part of the Mathematics and statistical foundations sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Regularization as a constraint on model complexity.
• Continue with Entropy and information gain in a small decision tree.