Verify an AI-generated revenue answer against source rows

Before trusting an AI-generated revenue answer, define what “revenue” means in the available dataset and reproduce the number from eligible source rows. A confident explanation and plausible SQL do not establish that the amount, period or unit is correct.

This example uses a narrower measure: January completed-order amount. It is not a claim about recognized revenue, cash collection or profit.

Freeze the metric contract

The original analyst AI lab reuses the synthetic commerce orders. Include completed orders from January 1, 2026 at midnight inclusive through February 1 exclusive, interpreted as local Asia/Kolkata timestamps for this teaching exercise.

Each order contributes its supplied order_total_paise once. Do not subtract the separate discount column again, and do not drop an eligible order merely because its customer is missing from a dimension table.

The eight included orders contribute 12,000, 18,000, 10,000, 25,000, 10,000, 8,000, 12,000 and 9,000 paise. Their total is 104,000 paise, or ₹1,040.

Compare a seeded wrong answer with the reference

The lab's candidate answers are deliberately authored examples, not recorded outputs from a live model. They test the verification workflow and must not be presented as a model accuracy benchmark.

from calculator import calculate,seeded_cases,check_candidate

evidence = calculate()
assert evidence['value']==104000 and evidence['eligible_order_count']==8
assert sum(row['amount_paise'] for row in evidence['source_rows'])==104000
assert evidence['evidence_order_ids']==[
    'O1001','O1002','O1003','O1005','O1006','O1007','O1008','O1009']
cases = {case['case_id']:case for case in seeded_cases()}
wrong = cases['header_join_fanout']['answer']
assert wrong['value']==171000
result = check_candidate(wrong,evidence)
assert result['structured_checks_passed'] is False
assert 'value_mismatch' in result['issues']
correct = check_candidate(cases['correct_structured_answer']['answer'],evidence)
assert correct['structured_checks_passed'] is True
assert correct['human_review_required'] is True
print({'reference_paise':104000,'seeded_wrong_paise':171000,
       'wrong_answer_issues':result['issues'],'human_review_still_required':True})

The 171,000-paise answer is a known join-fan-out error: order-header amounts are repeated across item rows. It is not repaired by a more persuasive paragraph. The query or calculation must change and the result must be rechecked.

Verify more than the headline total

The answer contract also checks period, timezone, currency, unit, eligible-order count and evidence IDs. A correct-looking numeric value can still answer the wrong period or omit a record while another error compensates for it.

Check units explicitly. Reporting 104000 INR would be one hundred times the intended major-currency amount. A schema that records currency: INR and unit: paise makes the distinction visible.

The unmatched-customer order O1009 contributes 9,000 paise. An inner customer join would reduce the amount to 95,000, violating this metric's eligibility rule. Evidence IDs help locate that omission.

Separate arithmetic checks from narrative support

A structured checker can compare exact amounts and counts. It does not necessarily understand whether a sentence invents a cause. The lab intentionally includes a causal sentence mislabeled as descriptive that passes the narrow structural checks but must fail semantic review.

That blind spot is part of the demonstration. Do not describe a structural pass as a fully verified analytical answer. Review the prose against the actual evidence and ask whether the claimed comparison or mechanism was measured.

Anthropic's evaluation guidance discusses complementary grader approaches. Here, deterministic checks handle exact facts while human review remains necessary for meaning and decision suitability.

Preserve the evidence behind the check

The calculator records source and contract hashes with the eligible rows. If a source correction changes the reference, retain the new version and explain the change rather than editing an expected value to make a candidate pass.

The supplied refund ledger has no refund timestamps. Its refund-adjusted amount cannot establish January cash-flow timing. A reliable assistant should state that limit instead of inferring a date-specific net-revenue answer.

Exercise: test the seeded wrong-unit and missing-evidence candidates. Then create a sentence with correct arithmetic but an unsupported causal explanation and explain why numeric validation alone is insufficient.

NeuraPath's Data Analytics with Generative AI course connects AI-assisted analysis with deterministic verification. The useful skill is tracing an answer to its metric contract and source evidence before using it in a decision.

Continue learning

This article is part of the Generative AI for verified analyst work sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Continue with Write an analyst prompt with a metric definition and evidence contract.
• Then apply it in AI-generated SQL: test the query before trusting the explanation.