Give Your AI Agent an Exam Before You Give It a Job

A test-driven approach to building reliable AI capabilities

Alex Guglielmone Nemi
Mayank Hirani

How the team works

EU Retail
pricing
demand
negotiations
we work across every domain, not just one
engineering lead
cross-functional team
economists + engineers
Goal: economists deliver
without engineering bottlenecks
THE FACTORY OF ECONOMICS
run any SQL
dbt model
scheduled run
backfill
each block ships with a guarantee
scalable · reproducible · auditable
economists mix these themselves
no engineer in the loop
coding agent
100% of the team builds with one
guarantee: works how we want?

Delegation & control

Delegate heavily, but the agent is non-deterministic. And everyone has a recipe:
“use Opus” “add this skill” “give it MCP tools” “switch harnesses” “use a swarm” coding agent NEW HIRE one more head count you must be able to trust: Correctness: How do we know it’s right? Reliability: “It worked yesterday, not today” other concerns: cost · speed · observability · blast radius
Each recipe is a hypothesis. The exam settles it with a pass/fail: qualified for our work, or not?

Qualify the capability

You don’t hire an assistant for generic ability. You hire one qualified for the work you need done

Kiro + Sonnet Claude Code + context Strands + MCPs interchangeable YOUR EXAM questions + assertions (versioned) PASS / FAIL for this domain

The exam lives outside the agent

Exams

Coding
ask
“Write a dbt model: ASINs sold by marketplace & day”
assert
sql-select
snapshot_day: {{ snapshot_day }}
marketplace_id: {{ marketplace_id }}
deterministic
Using tools
ask
“Run scenario: client ABC, target €100k, scope EU5”
assert
tool-called
run-model --client ABC
  --ask 100000 --scope EU5
deterministic
Planning
ask
“Run all clients <€300k, all markets, +20% last cycle”
assert
plan-contains
1clarify “last cycle”
2smoke-test one first
3then the full run
LLM-as-judge

These asserts describe your Executable Intent

* Some may call these Evals or Behaviour Driven Development

A closer look

The exam question
name: safe-partitioning-for-concurrency
input: "Write a dbt model to get ASINs sold
by marketplace and day for table prov_a.table_b"
intent: We need to select partitions explicitly so
  that parallel runs work with hive/insert_overwrite
assert:
  - type: sql-select        # deterministic
    includes:
      snapshot_day:   "{{ snapshot_day }}"
      marketplace_id: "{{ marketplace_id }}"
Candidate A  ✓ passes
SELECT asin,
    '{{ snapshot_day }}'  AS snapshot_day, -- compiles to a literal
    {{ marketplace_id }}  AS marketplace_id -- e.g. 3 AS marketplace_id
FROM speed.shipments
WHERE marketplace_id = {{ marketplace_id }}
GROUP BY asin
Candidate B  ✗ fails
SELECT marketplace_id, -- read off the row
    CAST(ship_date AS date) AS ship_day,
    COUNT(DISTINCT asin) AS asins
FROM {{ source('speed','shipments') }}
GROUP BY marketplace_id, CAST(ship_date AS date)
Both look like reasonable dbt. Only A can run partitions in parallel without overwriting.

A closer look, illustrated

Candidate B ✗ partition read off the rows run 1 run 2 run 3 run 4 run 5 one partition overwritten ×4 5 runs → 1 survives and the table still looks fine Candidate A ✓ partition stamped from the run params run 1 run 2 run 3 run 4 run 5 mkt 3 · day 1 mkt 5 · day 1 mkt 3 · day 2 mkt 5 · day 2 mkt 4 · day 1 5 runs → 5 partitions The assertion catches this. A reviewer doesn't.

Same model, different result

  • Same underlying model Claude Sonnet & same questions
  • You’re judging the whole setup, not just the model
measured during development 26% 7 / 27 out of the box bare Kiro CLI · same Sonnet 81% 22 / 27 + EDS context + MCP Strands harness · same Sonnet +55 pts same model Today: every approved config scores 100% · now a regression test

The exam becomes the feedback loop

The exam stays fixed. Change anything beneath it until it passes

Your exam RED Prompt / context Skill Tool / MCP Code Environment GREEN

Degrees of agency

inference (LLM) deterministic common evolution AGENT all inference AGENT tools thin agent workflow workflow workflow workflow workflow agent does everything known steps → tools thin agent + workflow maybe just workflow the same exam qualifies every stage

Use inference where it matters · engineer it away where it doesn’t

Takeaways

  1. Qualify the capabilities you need. Not the model.
  2. Make “good enough” executable
  3. Use the exam to build, not just to score

Start with one behaviour your AI must get right · turn it into an exam · make any implementation pass it

Questions?

Alex Guglielmone Nemi · Mayank Hirani · Analyticon 2026