Evaluate a model by meaningful data slices

An aggregate score can conceal different behavior across product plans, missing-data conditions or usage patterns. Choose slices that connect to deployment decisions, then report their denominators and limitations alongside the metrics.

Our original synthetic inactivity model has80 March test snapshots. We inspect plan, whether the historical ticket count is missing, and whether days since activity is below seven. These are descriptive diagnostic slices, not demographic fairness conclusions or verified causes of model error.

Start with counts and target prevalence

The observed recall differs, but so do sample size, prevalence and potentially the feature mix. This table does not establish that the plan itself causes the difference or that the exact rates will persist.

The two plans partition the80 snapshots. Their row-weighted log losses recover the overall0.702749. An unweighted mean would give each plan equal influence, which is a different estimand from the average loss per snapshot.

Verify the slice report

import numpy as np
from assessment_cases import slices

rows = slices()
plans = {row['value']:row['metrics'] for row in rows if row['field']=='plan'}
assert plans['individual']['n']==52 and plans['team']['n']==28
assert plans['individual']['positive_count']==38 and plans['team']['positive_count']==17
assert np.isclose(plans['individual']['recall'],12/38)
assert np.isclose(plans['team']['recall'],10/17)
weighted = sum(row['n']*row['log_loss'] for row in plans.values())/80
assert np.isclose(weighted,.7027491222927091)
missing = next(row['metrics'] for row in rows if row['field']=='ticket_missing' and row['value']=='True')
assert missing['n']==7 and missing['positive_count']==3
print({'plans':plans,'ticket_missing_slice':missing})

Run in the evaluation lab. The recorded assessment report includes all three slice families and reconciles each partition to the full set.

Treat tiny slices as limited evidence

Only seven test snapshots have missing ticket counts, and only three are positive. Their observed recall is2/3. A percentage without those counts could appear much more stable than the evidence supports.

Each customer can contribute repeated observations, so seven rows do not necessarily represent seven independent entities. For uncertainty estimates, consider the relevant dependence structure rather than assuming every row is independent.

If a slice has no positives, recall is undefined. If the model selects nobody, precision is undefined. Preserve those states instead of replacing them with a number that looks directly comparable to a well-supported rate.

Distinguish planned checks from exploratory findings

Product plan and missingness can be sensible predefined monitoring slices. Further combinations discovered after reviewing test errors are exploratory. Searching many small slices increases the chance of finding extreme-looking values by accident.

Record the slice definition, reason for inspection, support threshold and whether it was selected before or after looking at the results. When an exploratory pattern informs a model change, use new or otherwise appropriate assessment evidence for the revised claim.

The Model Cards paper motivates reporting evaluation conditions and disaggregated behavior. This lab's small synthetic slices are a worked reporting example, not evidence of comprehensive fairness or deployment suitability.

Exercise: create a plan-by-missingness table, including empty combinations. Identify which cells have too little evidence for a stable recommendation, reconcile all cells to80 snapshots, and propose how to collect or evaluate additional evidence without silently dropping difficult groups.

NeuraPath's Data Science course connects aggregate metrics with meaningful error analysis. A useful slice report reveals where the evidence is weak as clearly as where a score is high.

Continue learning

This article is part of the Machine learning workflow and evaluation sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Build an error taxonomy for a trained model.
• Continue with Document dataset provenance in an experiment manifest.