Data science interview: interpret a failed experiment

“The metric did not improve” is an outcome, not an interpretation. A strong interview answer shows what the experiment was expected to change, what acceptance rule was set before seeing the result and what evidence now updates the next decision.

Preserve both parts of the gate

The career evidence lab tests an invented class-weight experiment.

from career_cases import failed_experiment_case

result = failed_experiment_case()
assert result["gate_passed"] is False
assert result["failure_preserved"] is True
assert result["record"]["observed"] == {"recall": 0.68, "precision": 0.21}

The hypothesis required minority recall of at least 0.60 without precision falling below 0.30. Recall reached 0.68, but precision was 0.21, so the candidate was rejected. Reporting only the recall improvement would rewrite the rule after the outcome.

Structure the answer as evidence

1. 1Context: name the decision, data period and baseline.
2. 2Hypothesis: explain the mechanism you expected to help.
3. 3Protocol: state the split, metric and gate chosen before evaluation.
4. 4Result: give denominators and uncertainty, including adverse slices.
5. 5Decision: accept, reject or gather more evidence.
6. 6Next test: change one assumption that the result calls into question.

For this fixture, a defensible next step is validation-only threshold analysis plus label-quality review. It is not immediate retuning against the test set. If class weights changed probability quality, compare calibration and the number of cases sent to review.

Separate learning from rescue

A failed candidate can still expose that the baseline is strong, the sample is too small, labels are delayed, the intervention is capacity-bound or the proposed mechanism was wrong. Preserve the configuration, predictions and error examples so a reviewer can verify that conclusion.

Avoid claiming business lessons that the experiment did not test. An offline precision decline does not prove users dislike alerts. It supports a narrower statement about labelled evaluation under the chosen threshold.

The Data Science course treats rejected experiments as assessed evidence. Learners defend the gate and next decision, not just their best score.

Exercise

Write a six-part answer for one project failure. Remove every adjective that is unsupported by a metric or artifact. Then identify one tempting post-hoc change and explain how a new validation experiment would test it without contaminating the locked test set.

Continue learning

This article is part of the Data science careers and portfolio decisions sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Data science interview: choose a metric from business costs.
• Continue with Prepare a model project for a technical viva.

Reference: NIST guidance on experiment design.