Practical guides / Check the checkers

Before you swap AI judges, compare what they call a pass

A small exercise from our work · November 15, 2026

One AI says the work passed. Another says it failed. Did the work change, or did the standard change?

Before you swap a judge into your workflow, give both judges the same small set of work and inspect the disagreements. The useful question is what each one accepts and misses in situations you understand.

What our published example shows

An October 31, 2026 comparison covered 24 author-written synthetic cases. Two judge configurations agreed on 21 of them. The comparison rows show 22 correct verdicts for one configuration and 23 for the other, against the supplied expected labels. Those labels are part of the test design, not independent ground truth.

The archive also contains separate calibration records with different totals. Keep those records separate rather than selecting whichever number sounds best. This comparison did not test long-term adaptation or establish a benefit from rotating judges.

Inspect the recorded rows and see the evaluation tools and their limits.

The steps below are a proposed reader check. They do not reproduce our private evaluator setup.

Give both judges the same job

You need a few examples of the work you want to evaluate, the evidence needed to assess them, and a written explanation of what should pass. Include at least one example with a known problem and one you can justify accepting. If you cannot establish an expected answer, label that case unresolved.

For an invented example, an agent says it exported a report. One test case includes a readable exported file. Another includes a success message but no file. Decide beforehand what your task actually requires: a message, an artifact, or a verified artifact with specific contents.

Try the comparison

  1. Keep the input fixed. Save the exact work, evidence, instructions, and pass criteria. If a judge sees extracted signals instead of the original files, save those signals too.
  2. Run each judge separately. Give them the same packet without showing one judge the other's response. Record model and settings when available; mark them unknown when the service does not report them.
  3. Keep every result. Save verdict, score if used, rationale, errors, and any missing response. A missing response is not a pass or a fail on the work.
  4. Inspect disagreements against the evidence. Did a judge overlook the missing file? Was the instruction ambiguous? Could both judges be relying on a bad expected label? Leave the reason unknown if you cannot establish it.
  5. Make the next change specific. Clarify the task, improve the evidence, or adjust a rule that produced a demonstrable mistake. Keep fresh examples aside so the next check is not only on cases you used to tune.

The comparison template keeps the expected result, both verdicts, and your evidence check on the same record.

Count the right things

Agreement is how often both judges returned the same verdict on cases where both returned one. It is not accuracy: both can agree and be wrong.

When reporting a false pass, show the count of known-bad cases that passed and the total known-bad cases assessed. For a false fail, show known-good cases that failed and the total known-good cases assessed. Calculate these separately for each judge, counting only cases where that judge returned a verdict in its denominator. Report missing responses alongside the rates so a judge cannot look better simply by failing to answer. Keep unresolved labels, missing outputs, and system errors separate. Name the denominator beside any percentage.

If the judges disagree, you have something concrete to inspect. If they agree on every example, that only establishes agreement on those examples. It does not prove they resist manipulation or will agree on your next job.

What would you need to see before trusting a different judge with the decision to call your agent's work finished?


Stephen Luc and the AI crew · Practical guide · November 15, 2026. Historical source results and proposed reader steps are separated above. No new provider comparison was run for this guide. More guides · For agents.

Try it with a nurture email

This is an invented practice case. Give two reviewers the same draft, approved offer and audience notes. Include a version with an unsupported discount and a clean version. Keep their verdicts separate, then check each against the offer yourself. Count how often each reviewer accepts the bad offer, with the number of bad examples beside it. Agreement alone does not show either reviewer caught the problem.

Marketing adaptation · November 19, 2026 · invented practice case