Entropy and information gain in a small decision tree

Information gain measures how much a split reduces class-label entropy in the data reaching a tree node. It is a training-data criterion. A large reduction does not, by itself, establish accurate predictions on new observations.

Consider an original ten-record teaching node with six positive and four negative labels. A candidate rule sends four positives and no negatives left, and two positives plus four negatives right. These counts are sufficient to calculate the split score without inventing row-level features.

Calculate the parent uncertainty

For class proportions p and 1-p, entropy in bits is -p*log2(p)-(1-p)*log2(1-p), with a zero-probability contribution defined as zero. At proportions 0.6 and 0.4, the parent entropy is approximately 0.970951 bits.

A pure child has entropy zero. A balanced binary node has entropy one bit. These describe class mixture, not a percentage accuracy or the difficulty of every possible classification task. The SciPy entropy reference documents the calculation and logarithm-base option.

Weight each child by its record count

The left child has four records and zero entropy. The right child has six records with class proportions one third and two thirds; its entropy is approximately 0.918296 bits.

Weighted child entropy is (4/10)*0 + (6/10)*0.918296 = 0.550978. Subtracting this from the parent gives information gain approximately 0.419973 bits.

An unweighted average of child entropies would describe equal weighting of nodes, not equal weighting of the ten observations. That would score this split incorrectly under the stated criterion.

Reproduce the score and reject invalid counts

import numpy as np
from inference_core import entropy_bits, information_gain

result = information_gain([6,4],[[4,0],[2,4]])
assert np.isclose(entropy_bits([6,4]),.9709505944546686)
assert np.isclose(result['weighted_child_entropy_bits'],.5509775004326937)
assert np.isclose(result['gain_bits'],.4199730940219749)
assert np.isclose(information_gain([6,4],[[3,2],[3,2]])['gain_bits'],0)
try:
    information_gain([6,4],[[4,0],[3,4]])
except ValueError:
    pass
else:
    raise AssertionError('children must partition the parent')
print(result)

Run from the included lab. The deliberately invalid split contains seven positives in its children even though the parent contains six. The helper rejects it. Empty children are also rejected by this teaching helper rather than treated as useful candidate splits.

The alternative children [3,2] and [3,2] preserve the parent's class proportions, so their information gain is zero. Splitting rows into two groups is not enough; the class mixture must change to reduce this entropy.

Keep split quality separate from evaluation quality

A feature that contains the answer, such as a post-outcome status, can create a pure split. Its training gain may look excellent while the feature is unavailable when predictions are needed. Check feature availability before celebrating the score.

Repeatedly searching many candidate splits can also fit accidental patterns. Tree depth, minimum leaf sizes and other controls affect the fitted structure, while suitable validation evaluates the resulting model. This count example does not choose those settings.

Exercise: score the valid alternative children [5,1] and [1,3]. Compare their weighted entropy with the original split. Submit the parent-to-child count reconciliation, score calculation and one reason the better training split might still fail in deployment.

NeuraPath's Data Science course connects decision-tree mathematics with leakage checks and model evaluation. A sound explanation includes both how the split score is calculated and what that score cannot establish.

Continue learning

This article is part of the Mathematics and statistical foundations sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Bias and variance with repeated training samples.
• Continue with Cosine similarity versus Euclidean distance after scaling.