TDD inside the agent loop

Definition

Böckeler separates three ways TDD meets an agent: a human writes the tests; the agent writes a failing test and a human reviews it before the implementation; or the agent runs the whole red-green-refactor loop alone. The third is the most common, and it is the one her eval tested.

Numbers (reported, not re-measured here)

Three greenfield business-logic tasks. Sonnet 4.6 wrote the code, a Sonnet 4.6 judge checked TDD adherence from the transcripts, and Opus 4.8 ranked the solutions without knowing how each was made. Every run had to reach 80% coverage.

The token counts are running totals that include cache reads, so the dollar overhead is smaller than the multiplier. It is still several times.

Reading the transcripts, Opus found that the no-TDD runs designed the whole thing first: data types, edge cases, contracts. The TDD runs let the first test fix the shape and rarely revisited it, and behaviour nobody wrote a test for was never built. Ivett Ördög’s explanation: models were trained on finished functions, not on step-by-step TDD sessions.

Why red-green loses its meaning

A red test proves a regression check works only when someone checks why it went red. When the agent writes the test and watches it fail, all that proves is that the agent ran it. In her runs agents skipped or faked the red step, and they implemented ahead of the current test so the next one passed without ever going red. Test-first also did not prevent the tautological test: one TDD run computed its “expected” value by re-running the implementation. That is the third of the four lying tests in skills/build/references/tdd-seams.md.

/solo:build used to say “a test that has never been red proves nothing.” That holds for a human. For an agent the stronger statement holds: a test the agent saw go red proves nothing either. The line was removed on 27 Oct 2026.

Rule

• Test-first only where the test has an owner other than the implementing agent: a seam the human confirmed (the build skill already asks for this), a contract or a set of examples written before the code (see harness-cost-by-model-tier), or a human checkpoint between the test and the implementation.
• Measure the suite rather than the ritual. make mutants in solo-factory asks “does the suite go red when a line is wrong”, which is the question red-green was meant to answer. Böckeler made the same switch, to mutation testing as the regression sensor.
• Ask for the design up front: data model, edge cases, contracts. That is what moved quality in her runs, and step-by-step TDD works against it.
• For agent-only tasks, default docs/workflow.md to moderate or none rather than strict, and put the saved tokens into a mutation run at the end.

Exceptions

• One person’s exploratory eval: 5 batches, one judge model, greenfield business logic only. Legacy code, concurrency and UI were not tested.
• Quality is Opus’s rubric, built on the fly. Her criteria were deliberately loose.
• When a human writes the tests or reviews the failing test (her first two modes), none of this applies. Seam-first and vertical slices in dev-principles-summary are about what gets tested and stay as they are.

Adopted here, 27 Oct 2026

The rule above replaced Red→Green→Refactor as the default everywhere it was written: the personal CLAUDE.md, solo-factory’s dev-principles.md, /solo:build 2.4.0 (strict = the user approves each failing test; moderate, the default = design first, code and tests together at agreed seams, a mutation check per phase), /solo:setup and /solo:init. dev-principles-summary and writing-claude-md follow. cli-first-testing stays as it was: make integration is a sensor with a human-owned expectation. And agent-mistake-fix-harness applies: a procedure that drifts across model releases belongs in the harness, not the prompt.

Learned from

Birgitta Böckeler, “TDD inside the agent loop: theater or actual value?”, Exploring Gen AI series on martinfowler.com. The series index dates it 10 August 2026; the article page prints no date. Raw runs are on GitHub at birgitta410/tdd-comparisons. Her mutation-testing setup is in “Sensors for coding agents” (martinfowler.com, 27 May 2026).