Review AI-generated Python before using its output

Treat AI-generated Python as a proposed implementation. Define the required behavior, inspect the code and test it against independently calculated cases before trusting its output. A script can run successfully while counting the wrong rows or changing missing values into invented facts.

The review standard should follow the task's consequences and data access. For a local teaching report, a small synthetic fixture can expose several important mistakes without involving real customer data.

Give the model a contract worth reviewing

A useful request specifies input schema, row grain, identifier types, accepted statuses, date boundaries, duplicate policy, rejection handling and expected outputs. Asking only for a sales report leaves those choices to the generated code.

The Python reporting lab provides a concrete contract: eleven raw records become seven accepted unique orders, three rejections and one identical replay. January contains four valid paid orders totaling INR 47.50 for two customers.

Those controls are known before evaluating a proposed implementation.

Catch plausible mistakes with a tiny case

The following deliberately flawed candidate sums every supplied order. It ignores both the requested month and payment status. The test demonstrates why successful execution is insufficient.

from datetime import date
from decimal import Decimal
from report import Order, paid_summary

orders = [
    Order("A", "0012", date(2026, 1, 31), Decimal("19.00"), "Paid"),
    Order("B", "0042", date(2026, 2, 1), Decimal("10.00"), "Paid"),
    Order("C", "0012", date(2026, 1, 31), Decimal("8.00"), "Pending"),
]
def flawed_candidate(orders, month):
    return sum((order.amount_inr for order in orders), Decimal("0.00"))

expected = Decimal("19.00")
assert flawed_candidate(orders, "2026-01") == Decimal("37.00")
assert flawed_candidate(orders, "2026-01") != expected
assert Decimal(paid_summary(orders, "2026-01")["paid_amount_inr"]) == expected
print("The review fixture rejects a plausible but incorrect candidate")

This is an authored failure example, not a claim about a particular model's measured behavior. It shows the kind of error a review must detect.

Inspect transformations that can conceal problems

Look carefully at broad exception handlers, dropna calls, fillna(0), conversions of identifiers to integers, dictionary-based deduplication and joins without cardinality checks. These operations may be valid under a contract, but they can also make troublesome records disappear.

Ask for a reconciliation table rather than accepting a clean-looking final dataframe. Every raw record should have an explained outcome. A lower row count needs a reason; a matching total does not prove the population is correct.

Use the library's primary documentation to verify unfamiliar APIs. For this lab, the CSV reference and unittest reference support parsing and test mechanics. The business rules still come from the reporting contract.

Keep the oracle independent

If the same generated code produces both the result and its expected value, the test may repeat the same mistake. Calculate small controls manually or through a separately reasoned method, such as a simple SQL query at the same grain.

Test changed input order, duplicate replay, conflicting duplicate values, empty periods and boundary dates. These cases challenge assumptions that a happy-path example may never exercise.

A model's explanation of why its code is correct is useful review material, but it is not execution evidence. Record the command, environment and actual result when claiming that a workflow was tested.

Review execution scope as well as arithmetic

Before running a proposed script, inspect file writes, network requests, package installation commands and credential handling. A report that only needs a local CSV should not require unexplained uploads or destructive cleanup.

Use a controlled copy and explicit output directory for evaluation. The lab protects its input against a direct output-path collision, and its tests verify that behavior. It does not claim to sandbox arbitrary generated code.

Exercise: ask for an alternative implementation of the monthly summary, then evaluate it with the same fixed controls. Record any changed assumptions and one test that failed before correction. Do not report quality from whether the prose sounded confident.

NeuraPath's Data Analytics with Generative AI course connects AI-assisted coding with verification. The analyst remains responsible for explaining the population, calculation and evidence behind the final number.

Continue learning

This article is part of the Python foundations for analysts sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the preceding task in Use Git to review changes to an analysis script.
• Return to the cluster foundation in Python environments: reproduce an analysis on another laptop.