top of page

Agents Testing Agents: How I QA Multi-Agent Systems Without Letting AI Grade Its Own Homework

carocsteads
Nov 7
7 min read

Agents, from the ground up · Bonus. A QA engineer's playbook for testing multi-agent apps, using a team of AI agents as testers and plain code as the referee.

Here's the uncomfortable truth about multi-agent apps: the traditional testing playbook doesn't fit them. You can't write a test for every path, because the agents choose the path at runtime. You can't enumerate the inputs, because the input is human language. And the worst bugs don't live inside any one agent. They live in the gaps between them.

The industry's popular answer is "ask an LLM if the output looks right." I think that's half right and dangerously half wrong. After spending a lot of time testing agentic systems (much of it on OWASP's FinBot, a multi-agent CTF app), here's the approach I've landed on:

Use agents to explore, generate, and attack. Use deterministic code to decide. Never let the AI grade its own homework.

This post is the whole playbook. (For how I wire the deterministic side into CI, see my earlier post on test creation agents and pull request test gaps.)

Why multi-agent apps are so hard to test

  • Non-determinism. The same input can take a different path through the agents on every run. A test that passes once proves very little.

  • Emergent behavior. Each agent can be correct on its own while the system as a whole does something wrong: a result relayed with missing context, two agents each assuming the other checked something, a handoff that drops a field.

  • An infinite input space. Users type anything. Vendors upload anything. Every free-text field is a new test case.

  • Adversaries. Some of your users are actively trying to make the system misbehave. Your tests have to think like them.

Humans alone can't cover that ground. That's where testing agents come in.

The testing pyramid for multi-agent apps

Before adding any AI testers, get the layers right. From cheapest and most deterministic at the bottom, to most realistic at the top:

  1. Tool tests. Each tool, called directly, with no model. Does it validate inputs and enforce its own rules?

  2. Handoff contract tests. Every result passed between agents matches a strict schema. Extra fields and free text are rejected.

  3. Orchestration tests. Replace the models with scripted fakes and check routing: given this task, does the orchestrator delegate to the right specialist, in the right order, with the right data?

  4. Scenario tests. Real models, simulated users, realistic end-to-end flows, judged by invariants on the final state.

  5. Adversarial campaigns. Red-team agents probing the system in a sandbox.

The bottom three layers need no API key and give the same answer every time. The top two are where testing agents earn their keep.

Meet the testing team

Just like the app under test, the testing side works best as a team of narrow specialists, each with a small toolbelt and a clear "never do this" list. (Yes, the guardrails post applies to your testers too.)

1. The Scenario Designer

Reads the product spec and generates test scenarios: the happy path, the confused user, the edge case, the unusual-but-legitimate request. Its output is structured (persona, goal, starting data, and the invariants that must hold), not prose. Can't: run anything.

2. The Simulated User

Plays a persona against the app, end to end: a new vendor onboarding, a customer asking about a delayed order, someone who gives information in the wrong order. It's how you exercise real conversational paths at scale. Can't: touch anything outside the test environment.

3. The Red-Team Agent

Plays the adversary, working through attack categories (goal hijacking, tool misuse, data exfiltration, privilege abuse) against a sandboxed copy of the app with fake data and fake money. Can't: ever see production, real credentials, or real customer data.

4. The Test Writer

Turns every interesting finding into a deterministic regression test: a recorded run, a scripted fake model, and assertions on state. This is the most important agent on the team, because it converts one-off discoveries into permanent protection. Can't: modify application code.

5. The Triage Agent

Clusters failures, removes duplicates, and drafts bug reports with reproduction steps. Fifty failing scenario runs usually turn out to be three real bugs. Can't: close or dismiss anything; humans decide.

6. The Coverage Reviewer

Maps what's been tested against the system: which tools, handoffs, and agent combinations have never been exercised? It points the Scenario Designer at the gaps for the next run.

The referees: deterministic oracles

Here's the part that separates this from "vibes-based" AI testing. None of those agents decide pass or fail. That job belongs to oracles: plain code that checks facts.

The most powerful oracle for multi-agent apps is the invariant: a rule that must hold after every run, no matter which path the agents took. If your agents emit structured events as they work (and they should), invariants become simple checks over the event trace:

def check_invariants(trace):
    # Money only moves for an order that was approved first
    for t in trace.events("refund_issued"):
        assert trace.happened_before("order_approved", t, order_id=t.order_id)

    # Every delegation was logged, with a known agent and a valid payload
    for d in trace.events("delegation"):
        assert d.target in KNOWN_AGENTS
        assert schema_valid(d.payload)

    # Planted canary data never left the system
    assert not trace.outputs_contain(CANARIES)

In plain words: we don't care which route the agents took. We care that, at the end, the rules of the business were never broken. Refunds only happen after approval, every handoff was recorded and well-formed, and the fake secrets we planted never leaked.

Other useful oracles:

  • State diffs: compare the database before and after. Only the expected records changed.

  • Trace shape: the right agents were involved, in an allowed order, within a step limit.

  • Schema checks: every handoff and every tool call matched its contract.

Model-based judges still have a place, for fuzzy qualities like "was the reply to the customer clear and polite?" But they produce a score you track, never a gate that blocks a release.

Taming non-determinism: test like a statistician

Running a scenario once tells you almost nothing. So I split checks into two kinds, with two different standards:

runs = [run_scenario(scenario, seed=i) for i in range(20)]

# Safety: invariants must hold on EVERY run. One failure is a bug.
assert all(r.invariants_ok for r in runs)

# Quality: task success is a rate, tracked over time against a threshold
success_rate = sum(r.task_completed for r in runs) / len(runs)
assert success_rate >= 0.90

In plain words: we run each scenario twenty times. Safety rules are pass/fail with zero tolerance: if money moves wrongly even once in twenty runs, that's a bug, not bad luck. Quality ("did it get the job done?") is measured as a percentage, and we watch that number over time, so a model or prompt change that quietly makes things worse shows up as a trend.

And when a failure does show up, record that run. Replaying the recording turns a flaky, one-in-twenty failure into a test that fails every time until it's fixed.

Who tests the testers?

If agents are writing your tests, how do you know the tests are any good? A test suite that always passes might be catching nothing.

My answer is mutation testing: deliberately plant known bugs in a copy of the app, and check that the test suite catches them.

MUTANTS = [
    "remove_refund_limit_check",
    "accept_extra_fields_in_handoff",
    "skip_delegation_logging",
]

@pytest.mark.parametrize("mutant", MUTANTS)
def test_suite_catches_planted_bug(mutant):
    with apply_mutant(mutant):
        assert run_test_suite() == FAILED   # if it still passes, our tests are blind

In plain words: we break the app on purpose, one way at a time, and make sure the tests notice. Every mutant that survives points at a gap the testing agents need to fill. It's a fire drill for your test suite.

A small amount of human spot-checking rounds it out: every week, read a handful of agent-written tests and triage reports. Agents are fast; humans are still the best judges of whether a test is testing the right thing.

Guardrails for the testing agents themselves

Testing agents are agents, so everything from the guardrails post applies to them:

  • Sandbox everything. Fake money, synthetic data, isolated environments. A red-team agent pointed at production is an incident, not a test.

  • Least privilege for testers. The Test Writer can write tests but not app code. The Triage Agent can draft but not close.

  • No live keys on untrusted code. The deterministic layers run on every pull request with no model and no secrets. Live-model campaigns run separately, in a trusted job.

  • Agents propose, humans approve. New tests and bug reports arrive as pull requests and drafts, reviewed like any other contribution.

How it fits together in a week

  1. Nightly: the Scenario Designer and Coverage Reviewer pick targets. Simulated Users and the Red-Team Agent run campaigns in the sandbox.

  2. Morning: the Triage Agent summarizes what broke. A human reviews the shortlist.

  3. During the day: the Test Writer turns confirmed findings into deterministic regression tests, opened as pull requests.

  4. On every pull request: the deterministic suite runs in CI, with no model and no secrets, and blocks the merge if an invariant breaks.

  5. Weekly: a mutation run checks that the suite is still sharp, and quality trends get reviewed.

The AI does the exploring, generating, and grinding. Code does the judging. Humans make the calls that matter.

The takeaways

  • Test the system, not just the agents. In multi-agent apps, the bugs live in the handoffs.

  • Invariants beat expected outputs. You can't predict the path, but you can define what must never happen.

  • Agents explore; code decides. Testing agents generate scenarios, attacks, and tests. Deterministic oracles hand out the verdicts.

  • Safety is pass/fail, quality is a trend. Zero tolerance for invariant violations; tracked thresholds for everything else.

  • Test your tests. Mutation testing keeps agent-written suites honest.

Multi-agent systems are only going to get more common, and "it looked fine when I tried it" won't cut it. The good news is that the same technology that makes these systems hard to test is also the best tool we have for testing them, as long as we never let it grade its own homework.

The code in this post is illustrative (a generic orders-and-refunds app) and not from any real project. Helper names like run_scenario and apply_mutant are made up; adapt them to your own stack.

Enjoyed this? Get every new post by email with the Subscribe to the blog form at the bottom of this page. You can also sign up as a site member (top of the page) to follow along.

Agents, from the ground up · © Carolina Steadham

Comments


Subscribe to the blog

© 2026 by Carolina Steadham. All rights reserved.

bottom of page