# Evaluated inactivity model: teaching project and acceptance rubric

**Local review draft.** Supports the Data Science prediction/evaluation articles. No production use or course-hour increase is implied.

## Decision and deliverable

Build a reproducible evaluation of seven-day account inactivity risk. The deliverable is an evidence package and a decision memo, not a live reminder system. Inactivity is not cancellation, and predicting it does not estimate the benefit of intervention.

Use the [lab](../labs/ds-evaluation/README.md), included320-snapshot CSV and fixed contract. Preserve synthetic/conditional-snapshot limitations. A real-data adaptation must separately establish event coverage, as-of availability and identity rules; this fixture does not supply a continuous event history.

## Required workflow

1. Reconcile rows, prediction grain, target horizon and label maturity. Inject a future-arriving feature and demonstrate rejection.
2. Reproduce the160/80/80 chronological split. Explain why all40 customer identities overlap and which alternative question group folds answer.
3. Fit the prior baseline and complete preprocessing/model pipeline on training only. Recover the training medians and declared C selection from validation results.
4. Preserve all test predictions and confusion counts. Report the baseline comparison, paired customer uncertainty, error taxonomy and slice denominators.
5. Reproduce the reference in an isolated environment, validate hashes, and write a model card and acceptance decision.

Core commands from the lab: `python evaluation_core.py`, `python verify.py`, `python metric_cases.py`, `python assessment_cases.py`, `python reproduce.py`, and `python provenance.py` after the reference manifest exists. Rebuilding the manifest uses `python provenance.py --build`; do that only for an intentional new reference, not to conceal a mismatch.

## Assessment rubric:100 points

| Evidence | Points | Passing evidence |
|---|---:|---|
| Target and source contract |20|Grain, horizon, coverage limitation, label delay and forbidden features explained|
| Split and preprocessing isolation |20|Entity/time scope and training-only learned statistics verified|
| Baseline and declared selection |20|Same-row comparison, all candidates, frozen test role preserved|
| Error/uncertainty/slice interpretation |25|Includes weak recall, interval crossing zero and small denominators|
| Reproduction and decision record |15|Input/code hashes, versions, numerical checks, model card and explicit readiness decision|

Hard failures require correction regardless of points:future information as a predictor; fitting transformations on assessment data; test-driven tuning presented as untouched testing; fictitious execution or performance; synthetic results represented as real customer outcomes; unverified causal or calibration claims.

## Worked decision under illustrative pilot criteria

For teaching, suppose a proposed pilot requires recall at least80%, no more than30 selected cases per80 scored snapshots, and a model-minus-baseline log-loss interval wholly below zero. These are authored acceptance criteria for this exercise, not agreed NeuraPath business requirements.

At the frozen test threshold.5, recall is40% (**fails**), selected cases23 (**meets the sample workload criterion**), and the percentile interval[-.103310,.001455] crosses zero (**fails**). Therefore the reference is **not ready for that proposed pilot**. Passing code checks does not override unmet decision criteria.

The technical educational artifact can pass its reproducibility rubric while the model fails a pilot gate. Do not change the threshold on the test set to manufacture acceptance. Use development data to propose a revised procedure, document its costs and obtain fresh assessment evidence. A real pilot would additionally require source-system validation, access controls, monitoring, fallback/rollback and an evaluation of the intervention itself.

## Extension exercise

Design a new evaluation for future predictions on previously unseen customers. Specify both temporal and entity boundaries, report the reduced sample support, and explain why neither the existing chronological result nor the January group result alone answers that combined question. Keep the original reference intact and label the extension as a new experiment.