Random forests: understand out-of-bag estimates and their limits

An out-of-bag prediction for a training row uses trees whose bootstrap samples omitted that row. It provides an internal assessment mechanism for a bootstrap forest, but it is not automatically a substitute for the grouping or time boundary required by deployment.

Our original example fits 300 regression trees on 160 independent synthetic duration rows, using a fixed seed and bootstrap sampling. There is one predictor, load index. We retain the separate validation set and report its result alongside the out-of-bag calculation.

Calculate why rows are omitted

When a tree samples n rows with replacement from n available rows, a particular row's omission probability is (1-1/n)**n. At n=160, this is approximately 0.366727.

The commonly quoted roughly 36.8% is a limiting approximation, not a guarantee that each tree omits exactly that fraction or each row has the same number of eligible trees. In this 300-tree run, individual rows have between 90 and 130 out-of-bag trees.

Name the score before interpreting it

For this scikit-learn regression forest, the default oob_score_ is R-squared. Its recorded value is approximately 0.942889. That is not an RMSE and should not be reported with the target's units.

Computing RMSE from the stored out-of-bag predictions gives approximately 2.159532 illustrative minutes. Separate validation RMSE is approximately 1.740242. Different estimates need not be identical because they use different observations and prediction ensembles.

The RandomForestRegressor reference documents the bootstrap, out-of-bag score and prediction attributes used here.

import numpy as np
from tree_cases import forest_case

r = forest_case()
assert r['trees']==300 and r['training_rows']==160
assert np.isclose(r['oob_probability_per_tree'],(159/160)**160)
assert r['oob_trees_per_row_min']==90 and r['oob_trees_per_row_max']==130
assert np.isclose(r['default_oob_r2'],r['recomputed_oob_r2'])
assert np.isclose(r['oob_rmse'],2.1595317990132537)
assert np.isclose(r['validation_rmse'],1.7402417236820396)
print(r)

Run from the supervised-model lab. The implementation checks bootstrap sample membership directly and confirms that every row has at least one eligible out-of-bag prediction.

Understand which boundary is protected

Omitting one row does not necessarily omit its customer, patient, device or near-duplicate records. If related rows remain in a tree's training sample, an out-of-bag score can answer a different question from performance on entirely new entities.

Likewise, a bootstrap sample can contain observations later than the row being predicted. It does not recreate a historical prediction time. Use a split that matches the entity and temporal deployment requirements when those matter.

Preprocessing can introduce another issue. If a learned transformation was fitted on all training rows before forest bootstrapping, an omitted row may still have influenced that transformation. The current example has no learned preprocessing, so that complication is absent here rather than solved generally.

Keep model selection in the record

Repeatedly selecting settings using an out-of-bag score makes that score development evidence. Assess the selected procedure under a suitable final design instead of treating the optimized internal score as an untouched external result.

Exercise: reduce the number of trees substantially and count out-of-bag coverage per row before calculating a summary. Then explain how you would evaluate the same modelling idea if several rows belonged to each customer and the claim concerned new customers.

NeuraPath's Data Science course connects ensemble mechanics with validation. A clear out-of-bag report names the metric, verifies coverage and states which forms of dependence it does not address.

Continue learning

This article is part of the Supervised learning methods sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Decision tree depth: visualize overfitting on a small dataset.
• Continue with Gradient boosting: why weak learners can overfit together.