Maximum likelihood estimation with a coin example

For n independent Bernoulli observations with k heads and a constant head probability p, the maximum likelihood estimate is k/n. The useful lesson is how that result follows from a declared model, how to verify an optimizer against it, and what happens at the boundaries.

Our original fixture has seven heads in ten tosses. The fitted value is 0.7. This is a teaching calculation, not evidence that a real coin has been measured accurately.

Derive the interior solution

Ignoring a parameter-independent combinatorial constant, the log likelihood is:

ell(p) = k*log(p) + (n-k)*log(1-p).

For 0<k<n, differentiating inside 0<p<1 gives k/p - (n-k)/(1-p). Setting this to zero produces p=k/n. The second derivative is negative throughout the interior, so the stationary point maximizes this log likelihood.

The NIST binomial reference gives this estimator. The logarithm preserves the ordering of positive likelihoods and replaces products with sums, which is useful when long products become numerically tiny.

Check the numerical optimizer independently

An optimizer returning successfully is one check. Agreement with a separately derived solution is stronger evidence for this small case. Run from the lab directory:

from scipy.optimize import minimize_scalar
from inference_core import coin_log_likelihood, coin_mle

fit = minimize_scalar(
    lambda p: -coin_log_likelihood(p,7,10),
    bounds=(1e-9,1-1e-9), method='bounded',
    options={'xatol':1e-12})
assert fit.success
assert abs(fit.x-coin_mle(7,10)) < 1e-7
assert coin_mle(0,10)==0
assert coin_mle(10,10)==1
try:
    coin_mle(0,0)
except ValueError:
    pass
else:
    raise AssertionError('empty sample must be rejected')
print({'numerical_estimate':float(fit.x),'analytic_estimate':.7})

The recorded numerical estimate is approximately 0.6999999990. The discrepancy from 0.7 is numerical tolerance, not a meaningful scientific difference. SciPy's minimize_scalar documentation explains the bounded minimization interface; the minus sign converts maximization into minimization.

Treat boundaries as part of the problem

For all tails, the maximum occurs at p=0. For all heads, it occurs at p=1. The derivative argument for an interior optimum is not enough to handle these samples. A numerical search restricted to epsilon and one minus epsilon cannot return the exact endpoints, which is why the analytical boundary tests are separate.

With zero observations, k/n is undefined and every p has the same empty-product likelihood. The helper rejects this input rather than manufacturing an estimate. It also rejects negative counts and a head count larger than the number of trials.

Separate fitting from uncertainty and prediction

The estimate from seven heads out of ten and from seven hundred out of one thousand is the same. Their information content is not. A point estimate alone does not express uncertainty, and neither sample ensures that the next ten observations will contain seven heads.

If you select a different model after inspecting the data, record that decision. Dependence, a changing process or selective recording can undermine the Bernoulli assumptions even when the derivative and optimizer agree perfectly.

Exercise: derive and test the estimate for two heads in five trials. Then test both endpoint samples. Your submission passes only if it includes valid interior behavior, boundary behavior and an explicit empty-sample policy.

NeuraPath's Data Science course links estimation mathematics with reproducible implementation. Keep the analytical result as a small reference test when moving on to models whose optimum has no closed form.

Continue learning

This article is part of the Mathematics and statistical foundations sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Likelihood versus probability in a fitted model.
• Continue with Regularization as a constraint on model complexity.