Build a small evaluation set for an analyst assistant

Build an analyst-assistant evaluation set around the mistakes that would change a business decision: the wrong population, duplicated amounts, incorrect units, missing evidence and unsupported conclusions. A small set with explicit expected behavior is more informative than a collection of questions graded only for fluent writing.

There are two different objects to test. One is the assistant producing answers. The other is the checking system deciding whether those answers are acceptable. This exercise tests the second object with deliberately authored candidates.

Fix the reference independently

The original analyst AI lab defines January completed-order amount as 104,000 paise across eight eligible orders. Its deterministic calculator reads the synthetic CSV and records source and contract hashes.

Expected values are fixed before candidate checking. The reference is not a second paraphrase from the same model, and it is not whatever answer happens to look reasonable during review.

Retain the source rows as well as the total. Two errors can cancel numerically, so checking only the final amount can miss a wrong eligible population.

Construct a purposeful case matrix

The supplied set contains eleven authored cases. Nine are expected structural failures: join fan-out, customer-join loss, distinct-amount misuse, wrong unit, wrong period, wrong count, missing order evidence, a numeric string and a declared unsupported causal claim.

One is a correct structured answer. The eleventh is an intentional blind spot: an unsupported causal sentence labeled as descriptive. It passes the narrow structural checker but must fail human semantic review.

from calculator import calculate,seeded_cases,check_candidate

evidence = calculate()
cases = seeded_cases()
assert len(cases)==11
results = []
for case in cases:
    result = check_candidate(case['answer'],evidence)
    assert result['structured_checks_passed']==case['expected_pass']
    assert result['human_review_required'] is True
    results.append(result)
assert sum(r['structured_checks_passed'] for r in results)==2
assert sum(not r['structured_checks_passed'] for r in results)==9
print({'authored_checker_cases':11,'expected_outcomes_matched':11,
       'structural_passes':2,'semantic_blind_spots_included':1,
       'live_model_calls':0})

“Eleven expected checker outcomes matched” is a legitimate result. “The AI is 100% accurate” is not: no model generated these answers, the cases were deliberately selected, and one structural pass is not an acceptable final narrative.

Keep grading dimensions separate

Record arithmetic, metric identity, evidence completeness, narrative support and appropriate abstention separately. A single average score can conceal a critical defect. Excellent prose should not compensate for a monetary amount in the wrong unit.

For a real assistant test, store the exact question, permitted tools, source version, model version, settings, raw response and grading result. Include tool failures and abstentions in the report rather than silently removing difficult runs.

Anthropic's evaluation guidance discusses combining grader types. This lab uses deterministic checks for exact structured facts and explicitly leaves semantic support to a separate review step.

Prevent the test from becoming the prompt

Use development cases to improve the workflow, then reserve additional cases for evaluation. If every expected answer is copied into the prompt, a successful response demonstrates answer reproduction rather than independent analytical execution.

A useful held-out case can change the period, introduce a second refund for one order, add an unmatched customer or alter the unit. Preserve the intended business rule while changing the surface details that a brittle solution might memorize.

Also include a question the dataset cannot answer. The refund fixture lacks refund timestamps, so an assistant should not fabricate January refund cash flow. Grade an appropriate request for missing data as a success for that task.

Report the limits with the result

Eleven cases provide regression coverage for named failure modes. They do not estimate performance across all business questions, organizations or model versions. Add cases from actual reviewed failures when a real deployment exists, while keeping confidential data within its permitted environment.

Exercise: add a correct answer with shuffled evidence IDs and a wrong answer with duplicated IDs. Decide the expected behavior before running the checker, then explain why set membership and uniqueness are different requirements.

NeuraPath's Data Analytics with Generative AI course treats evaluation as part of the analyst workflow. The goal is to know which errors a system catches, which it misses and what evidence supports that conclusion.

Continue learning

This article is part of the Generative AI for verified analyst work sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Create an AI analysis checklist for confidential business data.
• Continue with Catch denominator errors in AI-written business summaries.