Handle exceptions without silently dropping failed rows

Handle an invalid record only when the workflow defines what recovery means. If a report continues, preserve a rejection reason, reconcile input counts and label the result as partial. Catching every exception and continuing can turn a programming defect into an apparently successful but incomplete report.

An analyst should be able to answer two separate questions: did the program finish, and is the resulting dataset acceptable for this decision?

Classify the failure before catching it

The Python reporting lab distinguishes three outcomes. Valid unique orders enter the accepted dataset. An invalid customer or amount creates a rejection record. An incompatible file schema or conflicting duplicate order ID stops the run.

An identical replay is counted separately and excluded from the accepted order total. It is neither a newly accepted order nor a malformed record.

These are teaching policies. A production report may require every row to pass, but it must say so explicitly.

Reconcile all input records

from pathlib import Path
from report import load_orders

with Path("raw_orders.csv").open(encoding="utf-8-sig", newline="") as handle:
    accepted, rejected, replays, raw_count = load_orders(handle)
assert raw_count == len(accepted) + len(rejected) + replays
assert (raw_count, len(accepted), len(rejected), replays) == (11, 7, 3, 1)
assert {r["order_id"] for r in rejected} == {"R8", "R9", "R10"}
assert {r["reason"] for r in rejected} == {"invalid_amount_format", "invalid_customer_id"}
print("All eleven records have an explicit outcome")

The record-count equation catches disappearance, but it does not establish that rejected amounts are economically immaterial. N/A has no known amount; assigning zero would manufacture a value. Review the rejected population before using the report for a consequential decision.

Catch near the operation that can fail

Inside the loader, the recoverable try block covers field validation. The conflicting-key check happens afterward and propagates as a fatal ValueError. That placement expresses policy: this particular conflict is not another disposable bad row.

Keep such boundaries small. If a broad try block covers parsing, aggregation and output writing, a missing output directory or broken calculation can accidentally be classified as a record-quality issue.

Python's exceptions tutorial describes matching exception handlers and propagation. Choose exception classes according to the operation, and avoid a bare except around an entire pipeline.

Communicate partial success to the caller

The supplied command-line program writes a summary with quality_status set to partial_rejected_records and returns exit code 2 when records were rejected. Fatal processing errors return 1; a clean completed report returns 0.

Those meanings are local conventions, not universal Python rules. The scheduler or calling process must understand them. A nonzero exit should not trigger uncontrolled retries that repeatedly produce the same rejected rows.

The summary and exit code serve different audiences: the summary explains quality to a reviewer, while the code lets automation choose its next action.

Preserve useful evidence without copying everything

The rejection CSV contains record number, order ID and reason. The raw input remains available locally for diagnosis. Ordinary logs do not need full customer records, and a console traceback is not a substitute for an auditable rejection dataset.

For a larger system, attach a run identifier and source version to both accepted and rejected outputs. A separate review process can correct the source and rerun it; silently editing the rejected value inside the reporting script obscures what happened.

Exercise: change the second R3 amount in a source copy from 10.00 to 11.00. Confirm that the loader raises a conflicting-key error rather than selecting the first or last version. Explain who would be authorized to resolve that conflict.

NeuraPath's Data Analytics with Generative AI course connects exception handling with data-quality accountability. A reliable analyst can explain every excluded record and distinguish an executable report from an approved result.

Continue learning

This article is part of the Python foundations for analysts sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Python datetime: distinguish dates, local times and instants.
• Continue with Build a command-line report with argparse.