Test Creation Agents: Making Every Pull Request Show Its Own Test Gaps
My latest FinBot pull request (#615) takes on a problem every growing open-source project hits. Code arrives faster than anyone can review it, especially now that so much of it is AI-assisted.
Why
At the time, FinBot's CI ran only four test files. A pull request could change a detector or an agent without running any related tests. As a concrete example, three tests in the specialized-agents suite had been failing on main since they were merged, and nothing caught it, because CI never ran that file.
What I built
scripts/verify_change.py, a pre-merge check. It finds the tests that cover each changed file and runs them. It validates changed challenge YAML (schema, detector registered, config accepted, prerequisites exist). With --coverage, it lists the changed lines that no test runs. Each change gets a verdict: CLEARED, NEEDS TESTS, BLOCKED, or NO TESTS APPLY.
A CI job that needs no model and no secrets. It runs only on pull requests, with a read-only token and LLM_PROVIDER=mock, so it works on forks. It writes the test gaps to the job summary with a ready-made prompt for each one.
Agent guides. An AGENTS.md that Codex, Copilot, Cursor, and Gemini CLI read automatically, plus Claude Code skills and three agents: test-writer, bug-hunter, and requirements-reviewer.
The key design decision is that agents write tests on the contributor's laptop and CI only checks them. No model runs in CI, for two reasons. Fork pull requests don't get secrets, and an agent reading pull-request text is exposed to prompt injection, which is a fitting concern for a project about attacking AI agents. File names and errors from a pull request are neutralized before they reach any summary or prompt.
How I verified it
The check has 26 tests of its own, including seven end-to-end runs in a throwaway git repo. In a mutation run that planted 12 bugs in the script, the tests caught all 12.
test-writer produced 30 tests for the communication agent. After two assertions were tightened, they caught 10 of 10 planted bugs.
bug-hunter found five issues in the guardrail-prevention detector, including a webhook that blocks every call completing a challenge.
requirements-reviewer wrote acceptance criteria for an open issue and flagged that the issue named a file that doesn't exist.
I also opened #616, which makes CI install dependencies from uv.lock. Unpinned installs had pulled in Starlette 1.x and broken a test on every pull request.
Why it matters
AI can write tests quickly, but the hard part is trusting them. This setup keeps people in charge: the gaps are visible on every pull request, the generated tests have to prove they catch real bugs, and the gate itself is deterministic and costs nothing to run.
Skills: test automation, CI/CD (GitHub Actions), coverage analysis, mutation testing, AI-assisted development, secure CI design



Comments