# Overnight reporting failure: teaching runbook

Scope: the local synthetic `report-automation` lab. No production service, real credentials, scheduler registration or external distribution is configured. The role called **report owner** below is the team member assigned to the real workflow when this teaching pattern is adopted; this document assigns no real person or contact.

## Expected outcome

For the supplied January 5–12 UTC period, seven raw rows become six unique events and three selected paid events totaling 3,500 paise. The complete local bundle contains selected events, metrics and a manifest with status `prepared_for_review`, distribution `not_sent`.

An identical rerun reuses a verified bundle. Preparation success does not imply human approval or delivery. A January report is not a current-data report merely because it was generated today.

## First response

1. Identify the intended reporting period and source version. Preserve the attempt log and any completed bundle; do not overwrite them to make the error disappear.
2. Establish whether the scheduler attempted the run. If no attempt exists, inspect the scheduler/host/account separately from application logic. This lab does not configure that monitoring.
3. If an attempt exists, inspect its final exit code and the fixed `report_failed` diagnostic. The wrapper logs start, child output and finish; an interrupted wrapper may lack a finish record.
4. Verify whether a completed bundle exists and passes `verify_bundle`. Ignore temporary staging directories as report candidates.
5. Classify the failure and choose the bounded recovery below. Keep downstream distribution blocked until the correct artifact is reviewed.

## Failure guide

| Diagnostic | Meaning in this lab | Next action |
|---|---|---|
| `source_hash_mismatch` | Data bytes differ from the declared manifest | Obtain a consistent source delivery; preserve both versions. Do not change the expected hash merely to suppress the error. |
| `source_row_count_mismatch` | Raw count differs from the declaration | Reconcile the export and expected count at the source. |
| `source_watermark_before_period_end` | Declared coverage does not reach the required end | Wait for a valid source completion signal or explicitly revise the reporting decision; do not invent zero activity. |
| `source_extract_outside_allowed_window` | Extract timing violates the configured allowance | Review source delay and version/correction policy with the report owner. |
| `conflicting_event_id` | One ID has incompatible payloads | Reconcile source identity/version semantics; never choose an arbitrary last row. |
| `invalid_amount`, `invalid_event_enum`, `source_header_mismatch` | Source schema/value contract failed | Inspect a restricted sample and agree a source correction or reviewed schema change. |
| `invalid_region`, `invalid_period`, `invalid_config_fields` | Requested scope is invalid | Correct the configuration with the intended period and scope; retain the failed attempt. |
| `bundle_checksum_mismatch` | A retained output changed against its manifest | Preserve the artifact, investigate modification and restore/rebuild under a documented new decision. |
| wrapper exit 124 | Child exceeded the local timeout | Inspect resource/IO behavior and the artifact state before retrying. A timeout is not proof that no work occurred. |
| `cursor_cycle`, `snapshot_changed`, `declared_total_mismatch` | Simulated API completeness contract failed | Restart only under a valid stable source snapshot; do not release partial collection. |
| `RetryDeferred` | Required retry delay exceeds local waiting budget | Arrange a later bounded attempt; do not shorten the provider's requested delay. |

Authentication failures in a real API require authorized access repair. This lab has no credential or network integration and cannot demonstrate that repair.

## Local reproduction commands

Run from this directory with the intended Python environment:

```powershell
python verify.py
python verify_api.py
python pipeline.py --data events.csv --source-manifest source-manifest.json --config report-config.json --output example-runs
python scheduled_run.py --config report-config.json --output example-runs --logs example-logs
```

The tests use temporary directories and simulated API calls. The last two commands prepare local output and logs only. Do not run `build_fixture.py` over a real incident extract: that command regenerates the synthetic teaching data and would replace the evidence being investigated if misapplied.

## Recovery acceptance

- The intended period and region match the request.
- Source identity, watermark, schema and row-count checks pass.
- Selected records and grouped amounts reconcile; the fixture's reference total remains 3,500 paise when using its original inputs.
- The completed bundle verifies, and the attempt ends with the appropriate success code.
- Any correction creates a traceable version; approval applies to that exact artifact and scope.
- Distribution status remains `not_sent` in this lab. No external notification is implied by these checks.

## Incident record

Record the expected period, source/run IDs, diagnostic, affected outputs, cause supported by evidence, recovery action, resulting version and prevention check. Avoid copying credentials, raw personal data or unrestricted source payloads into the incident note.

For example, a simulated source-watermark incident can be closed when the source provides a consistent extract whose declared coverage reaches the required end and the rerun verifies. It should not be closed merely because a later command returned zero for a different period.