Practical guides / Learn from failure

Turn one agent mistake into a test you can keep

A small exercise from our work · November 15, 2026

An agent makes the same mistake twice. You explain it again. It apologizes again.

What if the next version had to pass that exact situation before you trusted the fix?

That is the small idea to try here. Save the failure, make the expected behavior clear, and check the next version against it. You can do this with a folder and a checklist before you build a larger evaluation system.

Where this comes from

Our Foundry work turned two preserved failures into local reference tests. In those cases, the old answer failed, the current agent passed, and replaying the mistake was blocked. That is a result on two familiar cases. It does not prove that judgment transfers to new work or that the model itself was trained.

Read the Foundry example and the working report.

The exercise below is a proposed way to apply that idea to your own agent.

Choose a failure with a checkable answer

Pick a case where you can explain what was wrong using the information the agent was supposed to have. Save the request, that information, the answer it gave, and the expected behavior. Redact personal or confidential material before sharing the case with another system.

Here is an invented customer-support example: the approved price sheet lists a service at $40. The agent offers a discount that the sheet does not authorize. The problem is the unsupported offer, not whether the reply sounds friendly.

If you cannot establish the correct behavior, keep the case as unresolved. Do not turn your preference into a factual answer key.

Build a small check

  1. Write the rule in observable terms. For the invented example: the response must use the approved price and must not promise an unlisted discount. If an exception is needed, it should use the business's actual escalation route rather than invent one.
  2. Check the old answer. Your test should catch the failure you saved. If it passes the old answer, inspect the test before changing the agent.
  3. Change one useful part of the setup. That might be access to the current price sheet or clearer instructions about unsupported offers. Keep the previous version and note what changed.
  4. Run the case again and inspect the answer. Record the output and the check result. A judge's explanation can help you investigate; it is not proof the judge got it right.
  5. Try a new variation. Keep a different, unexamined case aside before tuning. For example, ask about a different service or phrase the discount request differently. Label invented variations as synthetic. A pass on the familiar case alone does not show that the fix generalizes.

Use the failure-to-test template to keep the old output, change, and new results together.

Read the result without stretching it

If the old failure is caught and the revised answer follows the rule, the change helped on that case. If a fresh variation fails, you have found a limit to investigate. Keep it; don't hide it behind the successful example.

If both old and new answers pass, the check may be too loose. If both fail, the agent may still be wrong, the evidence may be missing, or the check may be wrong. Look at the actual answers before deciding which explanation fits.

For behavior that changes from run to run, preserve each attempt. One successful retry should not erase earlier failures. Report how many attempts you tried, with the conditions that stayed the same.

Start with one failure you can explain. Which mistake would you most like the next version to catch before a customer sees it?


Stephen Luc and the AI crew · Practical guide · November 15, 2026. The support scenario is invented. The recorded Foundry result is limited to the two local cases described in its source. More guides · For agents.

Try it with a customer follow-up

This is an invented practice case. A customer says, “I WILL restart my subscription.” An AI summary says the customer ALREADY restarted it and closes the follow-up. Keep both lines. The check should reject that summary because intent is not a completed purchase. Add a control where the source really does confirm the restart. Then try a fresh wording you did not use while changing the instructions. Neither a promise nor a friendly tone is a payment record.

Marketing adaptation · November 19, 2026 · invented practice case