2026-02-07 harness agents methodology context-engineering ai

Harness Engineering — Development in the Age of Agents

Synthesis of three key sources: OpenAI experiment (Ryan Lopopolo), Mitchell Hashimoto’s adoption journey, and Birgitta Böckeler’s analysis (Thoughtworks/Martin Fowler).

Definition

Harness Engineering — the discipline of designing environments, tools, and feedback loops so AI agents do reliable work. Humans steer, agents execute.

“When the agent struggles, we treat it as a signal: identify what is missing — tools, guardrails, documentation — and feed it back into the repository.” — OpenAI

“Harness engineering: anytime an agent makes a mistake, you take the time to engineer a solution such that the agent never makes that mistake again.” — Mitchell Hashimoto

Three Components of Harness (Böckeler/Fowler)

1. Context Engineering — context as code

Continuously improved knowledge base in the repository:

AGENTS.md / CLAUDE.md — map, not encyclopedia (~100 lines, links deeper)
docs/ — structured system of records: design docs, execution plans, product specs
Dynamic context — observability (logs, metrics, traces), browser via CDP, screenshots
Progressive disclosure — agent starts with a small stable entry point and knows where to look next

AGENTS.md           <- table of contents (~100 lines)
ARCHITECTURE.md     <- domain and layer map
docs/
├── design-docs/    <- design decisions (with verification status)
├── exec-plans/     <- active and completed plans
│   ├── active/
│   └── completed/
├── product-specs/  <- product specifications
├── references/     <- llms.txt for dependencies
├── QUALITY_SCORE.md
├── RELIABILITY.md
└── SECURITY.md

Key OpenAI insight: One big AGENTS.md is an anti-pattern. Pollutes context, rots instantly, impossible to validate mechanically.

2. Architectural Constraints — boundaries + freedom

Hard boundaries + freedom inside:

Layered domain architecture: Types -> Config -> Repo -> Service -> Runtime -> UI
Directed dependencies — custom linters validate direction
Parse at the boundary — validate data on entry (Zod, Pydantic)
Structural tests (ArchUnit-style) — check dependency graphs
Taste invariants — structured logging, naming conventions, file size limits

“Enforce boundaries centrally, allow autonomy locally.” — OpenAI

3. Garbage Collection — fighting entropy

Agents replicate existing patterns, including bad ones. Without GC, code degrades.

Golden principles — opinionated rules living in the repository
Recurring cleanup agents — background tasks scanning for deviations
Doc-gardening agent — finds stale documentation, opens fix-up PRs
Quality grades — each domain has a score tracked over time
Rule: tech debt as credit — better to pay continuously in small installments

OpenAI: previously 20% of time (Fridays) went to manual cleanup of “AI slop.” Doesn’t scale.

Feedback Loops — the agent’s only ground truth

An agent cannot see. Everything it believes about the running system comes through a loop: a command it runs that returns a verdict. The tightness of that loop sets the ceiling on everything else — context engineering and constraints only decide what the agent tries, the loop decides whether it finds out it was wrong.

A loop is tight when it is:

Red-capable — it can fail on the thing you care about, and you have watched it fail. A loop that has only ever been green proves nothing; it may not reach the code path at all.
Deterministic — same verdict every run. Pin time, seed RNG, isolate the filesystem, freeze the network.
Fast — seconds, not minutes. A 30-second flaky loop is barely better than none; a 2-second deterministic one is a superpower.
Agent-runnable — runs unattended. A human in the loop is a last resort, and even then driven by a script so answers come back structured.

Build the loop before the work, not after. The pull is always to read code and form a theory first — that’s the failure mode. Whatever the task, name the command that will tell you whether it worked, run it once to see its output, and only then start.

Treat the loop as a product. Once you have one, tighten it: cache setup, skip unrelated init, narrow scope, assert the specific symptom instead of “didn’t crash”. The loop gets used hundreds of times; two seconds saved compounds.

When it can’t be deterministic, raise the reproduction rate instead of chasing a clean repro: loop the trigger 100×, parallelise, add stress, inject sleeps. A 50%-flake is debuggable; 1% is not.

The catalogue of loop constructions — failing test, curl script, CLI+snapshot diff, headless browser, trace replay, throwaway harness, fuzz loop, bisect harness, differential run, HITL script — lives in skills/diagnose/SKILL.md Phase 1, ordered by preference. It’s written for bugs but the constructions are general: any task where you need a verdict picks from the same list.

Loops elsewhere in the factory: TDD’s red→green is a loop with a seam agreed up front (skills/build/references/tdd-seams.md). make integration is the CLI-first loop over business logic. /review is the slow loop over a whole change. /retro → ~/.solo/evolution.md is the loop over the factory itself. When a stage feels unreliable, ask what its loop is and whether it can go red.

What a loop costs is part of its design

A loop nobody runs guards nothing, and cost is what decides whether it gets run. Four rules, each learned by paying for the opposite:

Split hooks by cost, not by importance. A pre-commit hook that runs the whole test suite takes minutes, so it gets bypassed with --no-verify — and a bypassed hook checks nothing. Put seconds-long tripwires on commit and the suites on push. The tripwires are worth writing by hand: one per bug that actually shipped, phrased as “this file must still contain that guard”.

Measure per test, not per suite. A suite’s total time names the suite, not the culprit. Trimming what looked like the expensive loop in one suite saved three seconds; the real cost was a different test in the same file, and one sort over per-test timings found it in a minute:

<test command> 2>&1 | grep -E "passed|failed" | sort -t'(' -k2 -rn

A test guards a code path, not an amount of data. The same route through the system is exercised by three fixtures or by nine; the extra six buy nothing but wall-clock. Fixture volume is a dial, and its default should be the smallest value that still reaches the path.

Repetition is a deliberate act, not a default. Tests that repeat an operation to catch a flake are worth having and worth switching off: put the count behind an environment variable, default it to one, and raise it before a release or while chasing the flake. Derive the assertions from the fixture too — a hardcoded expectation (“the output is at least 25 seconds”) turns the next deliberate trim red for the wrong reason.

Verify parallelism instead of assuming it. Test runners that parallelise by cloning a whole environment can be slower than serial when the work is not CPU-bound: measured 11m32s against 3m on one iOS suite. And inside a test that measures duration, concurrency destroys the very number being asserted.

6 Steps of Adoption (Mitchell Hashimoto)

Step 1: Drop the chatbot

Chat interface (ChatGPT, Gemini web) is a dead end for serious development. Use an agent: LLM that reads files, runs programs, makes HTTP requests.

Step 2: Reproduce your own work

Do a task manually, then make the agent do the same with the same quality. Painful, but builds expertise:

Break sessions into separate, clear, actionable tasks
Separate planning from execution
Give the agent verification tools. It will self-correct

Negative space value: understanding when not to use the agent saves the most time.

Step 3: End-of-day agents

Block 30 minutes at end of day for agent runs. Don’t try to do more during work hours. Do more in off hours.

What works:

Deep research sessions — library reviews, competitor analysis
Parallel agents on unclear ideas — illuminate unknown unknowns
Issue/PR triage — agent with gh CLI compiles report (but does NOT respond)

Step 4: Outsource slam dunks

Tasks where agent almost certainly succeeds: let it run in background. Turn off desktop notifications. Human decides when to context-switch.

“Turn off agent desktop notifications. Context switching is expensive.”

Step 5: Engineer the Harness

Every agent mistake -> engineering solution so it never happens again. Two mechanisms:

AGENTS.md / CLAUDE.md — for simple problems (wrong commands, wrong APIs)
Programmatic tools — scripts, screenshots, filtered tests

“Each line in that file is based on a bad agent behavior, and it almost completely resolved them all.”

Step 6: Always have an agent running

Goal: agent always running. If not, ask: “what could the agent be doing for me?”

Preference: slow, thoughtful models (Amp deep mode / GPT-5.2-Codex). 30+ min per task, but high quality. One agent, not parallel.

OpenAI Experiment: Numbers

Metric	Value
Engineers	3 -> 7
Duration	5 months
Code	~1M lines
Pull requests	~1,500
PR/engineer/day	3.5
Human code	0 lines
Estimated speedup	~10x
Max single Codex run	6+ hours

Autonomy Levels (achieved)

One prompt -> agent can:

Validate codebase state
Reproduce bug
Record demo video
Implement fix
Validate fix via UI
Record second video
Open PR
Respond to feedback (agent and human)
Detect and fix build failures
Escalate to human only when judgment needed
Merge

Tools for Legibility

App per worktree — isolated instance per change
Chrome DevTools Protocol -> DOM snapshots, screenshots, navigation
Local observability stack — LogQL, PromQL, TraceQL (ephemeral per worktree)
Custom linters — errors contain remediation instructions for agent
Ralph Wiggum Loop — agent reviews its own changes, requests additional review, iterates until all satisfied

Practical Recommendations

CLAUDE.md as table of contents — keep ~100 lines with links deeper
docs/ as system of record — design docs, execution plans, quality scores
Custom linters with agent-friendly messages — remediation instructions right in the error
Structural tests — check dependency direction, file sizes
Doc-gardening — periodic agent for cleaning stale docs
End-of-day agents — issue triage, research, background tasks
Harness per project — each project = its own CLAUDE.md + docs/ + linters
“Boring” tech — prefer stable, composable technologies

Harness Health Checklist

[ ] AGENTS.md/CLAUDE.md — table of contents, not encyclopedia?
[ ] Pre-commit hooks exist and work?
[ ] Custom linters with remediation instructions?
[ ] Architectural constraints checked automatically?
[ ] Documentation versioned with code?
[ ] Entropy fighting mechanism exists (GC agents, quality grades)?
[ ] Agent can self-validate its work (tests, screenshots)?

Anti-Patterns

One giant AGENTS.md (“graveyard of stale rules”)
Knowledge in Slack/Google Docs (invisible to agent)
Micromanaging implementation instead of enforcing boundaries
Manual cleanup of “AI slop” instead of automation
Trying to “generate anything.” Constraints are multipliers, not brakes

Predictions (Böckeler/Fowler)

Harness as new service templates — organizations will create harness templates for main stacks
Tech stack convergence — AI pushes toward fewer stacks, “AI-friendliness” as selection criterion
Topology convergence — project structures will become more standard (stable data shapes, modular boundaries)
Two worlds — greenfield with harness vs. retrofit on legacy (different approaches)