Teams adopting coding agents hit the same wall in the same order. The agent produces plausible work. The suite is green. Something ships that should not have. The usual response is to change the model or add another review step, and neither helps, because the failure was never in the model.
The method below is built on one claim, and everything else follows from it: a loop's right to run without a human comes from properties of its oracle, not from the quality of its model. An oracle is the check that decides whether the work was done correctly. If it is cheap, frequent and unfakeable, autonomy can be granted. If it is any two of the three, it cannot.
We call the operating system built around that claim an AI Factory. On the classes of change that pass its gates, our clients run routine delivery roughly 3x cheaper and 5x faster than through a human review queue, because the expensive part of routine work was never the typing. It was the waiting: for a reviewer, for an environment, for someone who remembers why the module is shaped that way.
The headline numbers apply to routine change classes that clear the gates. The six measurements in the image above are what the audits themselves recorded, and they are the more useful half.
Three levels, and confusing them is the usual implementation error
Most failed agent rollouts are a category error: a problem at one level is treated with a rule from another. Three definitions fix that, and they have to be introduced in this order because each one is built on the one before it.
| Level | What it is | Unit of work |
|---|---|---|
| Loop | one agent doing one job in a circle: gather context, act, check, repeat until a condition is met | one change |
| Harness | the walls around the loop: sandbox, available tools, memory between runs, and the gate that defines done | one session |
| Factory | many harnessed loops at once, fed by a queue, admitted to trunk through an enforced update protocol, watched by a recovery loop, with a human owning the whole | a stream |
People tune the model when the bottleneck is the harness or the gate. Name the level first. A rule that governs one change, applied in minutes, does not pay down the debt a stream accumulates over weeks, and no session protocol makes an untrustworthy oracle trustworthy.
Each level has its own unit of work, its own horizon and its own owner. The rules that govern them do not compete, and they do not substitute for each other.
A useful consequence: a working session in which an agent edits your code is a factory in miniature. The loop is one turn. The harness is permissions, tools and memory. The factory layer is several concurrent sessions in one working tree. That is why the same audit finds the same three defects in a session that it finds in a repository.
The oracle is the only thing that grants autonomy
An oracle is a fast, durable check that decides whether the task was done correctly. It has to be all three of the following at once:
- Cheap, it runs on every change without spending human attention.
- Frequent, it fires inside the short cycle, not at the end of the week.
- Unfakeable, typed gates, property tests, invariants, or a reviewer bound to an external source of truth and an explicit rubric.
Back pressure: grant exactly as much autonomy as you can verify cheaply and reliably, and no more.
The interesting part is not building an oracle. It is that an oracle stops being one without announcing it, and this is far more common than model failure. Four ways, three of them carrying measurements we took on real systems:
It lives inside what the agent edits. The agent authors both the code and the proof of its correctness. Read this wider than "tests in the same repository": the check's configuration and the conditions under which it runs are usually inside the change area too. On one hosted platform we measured a protected branch where history rewrites were blocked, while the workflow file itself could be changed by an ordinary push and the run skipped by a directive in the commit message. The check was a detector wearing a gate's uniform.
It checks something the consumer never sees. Verification runs against the full artifact while the consumer receives a truncated one. Measured: an instruction string of 10542 bytes where the client reads the first 2048. Eighty per cent was invisible, three of the names that routing depends on sat past the cut, and the tests searched the whole string. The test could not fail.
It does not execute. Measured: 63 of 93 failing tests never ran at all, because a test plugin was absent while the project's own documentation declared it present. A zero exit code means the process finished, not that the checks you were counting on ran.
It measures a side effect of the repair instead of the goal. A watchdog recreated a stalled process supervisor and then proved success by having a canary validate a certificate, but the canary was launched by the watchdog itself, inside a fresh empty supervisor. The check was true, the claim was false, and not one application was running. The rule that follows: the success criterion must name the goal state, not the absence of the original symptom.
How to test an oracle. Disable the behaviour in production code and run. Exactly the tests about that behaviour must go red, and no others. Without that negative control, green is not evidence. This applies to a check you have had for five years as much as to one you wrote yesterday.
Autonomy is granted to a class of change, never to a repository
An audit produces two outcomes, not one. The first is a maturity level for the repository. The second is a binary decision for one class of change: may it update trunk unattended, and under what limits. A mature repository may stay fully human-gated by policy; a mid-maturity repository may earn autonomy for exactly one narrow class of routine work.
The landing event is the trunk update, not the merge. Under review-gated flow that moment is a merge; under trunk-based development it is the push, and the recovery move after it is a revert. Nine gates decide it, and each one has a named human fallback rather than a percentage.
L3 is necessary and not sufficient. It supplies the minimum machinery for cheap, frequent enforcement on every change, and it establishes nothing about whether the oracle is unfakeable or relevant to this particular task. That is what the other gates are for, which is why a level on its own never authorises anything.
Two of the nine carry most of the weight in practice. G4 Oracle asks whether the check is cheap, frequent and unfakeable at once. G7 Independent landing check asks who controls the checker: either prevention, where a protected check with an external source of truth passes on the exact candidate revision before the branch accepts it, or containment, where protected checks run on the exact pushed revision, only one uncertified push range exists at a time, and failure stops the line. A hook on a developer's laptop does not pass this gate; it can be skipped with a flag and it lives on one machine.
G8 deserves its own sentence, because it is the one teams skip. Whoever lands on trunk must not be able, alone, to deploy production or flip a flag. Landing and exposure are two authorities. Collapse them and every landing is a release.
Two ladders, and the link runs one way
There are two constraints on agent work and they are frequently confused. The simplicity ladder constrains the size of a change before it is written: do not build at all, take what already exists, write the minimal clear version of your own. The maturity ladder constrains the right of a change to leave without a human.
The link runs one way: the lower the maturity, the stricter the simplicity ladder must be. At the bottom of the maturity ladder there is no proof of correctness, so smallness and reversibility are the only protection you have. Near the top, a large change is acceptable, because there is finally something to check it with.
Security, trust-boundary validation, accessibility and data-loss handling sit outside anything the simplicity ladder may reduce.
Which brings up the size ceiling, and the one number people most want to import from a book. Do not. Set the ceiling from the repository's own history. Measured across three repositories of a single team on one day: medians of 3, 3 and 7 changed files, and p75 of 5, 6 and 12. The spread is threefold, and a single number would have been wrong for all three. The ceiling is neither a target nor a prohibition: a larger change is allowed, it simply loses the right to land without a human reading it.
The maturity model: eleven pillars, five levels
Scoring runs on an agent-readiness model of five levels and eleven pillars. The levels describe a posture rather than a toolset:
| Level | Posture | What an agent can do here |
|---|---|---|
| L1 Functional | code runs, setup is manual, validation is ad hoc, context lives in people's heads | generate snippets, answer questions |
| L2 Documented | basic docs and scripts exist, coverage is patchy, nothing is enforced | scoped single-file changes under review |
| L3 Standardized | processes defined, documented and enforced by automation on every change | routine autonomous work: bug fixes, refactors, dependency upgrades, tests |
| L4 Optimized | fast feedback, structured observability, measurable performance | multi-file features, performance-aware changes, triage with telemetry |
| L5 Autonomous | orchestration, evaluation harnesses, experimentation, explicit safety boundaries | end-to-end delivery; the human reviews policy, not every diff |
Two scoring rules matter more than the pillar list itself. A pillar takes the highest level at which at least 80% of its items pass, with no partial credit. And the repository takes the minimum across pillars, because agents fail on the weakest dimension first. The average is a direction of travel; the minimum is the gate.
L3 Standardized is the bar for unattended routine work. Most teams should aim there first: going from L1 to L3 unlocks most of the value, and L4 and L5 compound over time.
1. Style and Validation
Deterministic static checks that catch defects in seconds: a compiler, a type checker, or a defect-focused linter. This pillar follows the defect, not the tool, so a strict type checker in CI outscores a configured-but-never-run linter.
L1 to L5: from mistakes caught informally in review, to project-specific rules that encode architectural and domain constraints. What a weak pillar costs: a configuration module called a subprocess without importing it, the resulting error was swallowed, and a memory probe silently reported 8192 MB on a machine with 49152 MB. One linter rule found it in a minute.
2. Build System
Whether a fresh checkout builds deterministically with one documented command and pinned dependencies. Without it an agent cannot verify its own change, and neither can a new engineer.
L1 to L5: from a build that works on the original author's machine and setup that is folklore, to hermetic reproducible builds with a pinned toolchain. A real failure mode we measured: a lockfile present in one package and absent in the backend, so the larger half of the repository was not reproducible at all.
3. Testing
Whether the agent can verify its change without asking a human. This is the single highest-leverage pillar, and it is where the oracle failures above land.
L1 to L5: from tests that exist but are hard to run, to mutation testing, property-based tests and formal verification on the critical logic. The pillar needs one addition most models omit: checks must run against the same artifact the consumer receives, accounting for truncation, serialization and transport.
4. Documentation and Agent Context
Whether the context an agent needs is written down, discoverable and current. This pillar has the strongest leverage of all, because it converts tribal knowledge into something a machine can read on every run.
L1 to L5: from a README with everything else in people's heads, to living documentation where docs link to code, code links to docs, and both are updated as part of the change. It is also where comprehension debt should be measured: the share of code changed this quarter and never read by a human.
5. Development Environment
Whether a fresh agent, or a fresh human, gets a working environment in minutes rather than hours. Reproducible environments remove an entire class of failure that otherwise looks like a code defect.
L1 to L5: from manual order-dependent setup, to identical development, CI and preview environments per change. Measured consequence of a weak pillar: 316 green tests that were green only on a machine already set up by hand, which devalues the testing score sitting above it.
6. Debugging and Observability
Whether anyone, human or agent, can find out what the code actually did at runtime. Structured logs, traces and metrics turn "it failed" into a specific diagnosis, and they are what an evidence bundle is assembled from.
L1 to L5: from unstructured prints that are never aggregated, to telemetry an agent can query programmatically. Below L3 the agent guesses, and a guess dressed as a diagnosis is the most expensive artifact in this whole method.
7. Security and Governance
The guardrails that stop an agent, or a person, doing something unsafe without noticing: pushing secrets, bypassing checks, landing without review. It also carries the branch-protection items that decide whether G7 and G8 can pass at all.
L1 to L5: from no enforced branch policy, to policy-as-code, permission boundaries per agent identity, and supply-chain attestations. One rule worth naming: a prohibition aimed at one agent is not a prohibition, so authorization has to be checked at the delegation boundary and account for who initiated the chain.
8. Task Discovery
Whether the backlog is machine-readable. An agent that can read titles, labels and acceptance criteria can pick up routine work; an agent that cannot will wait to be told, which puts a human back in the loop for scheduling.
L1 to L5: from issues living in someone's head or a chat thread, to a backlog agents query and triage through an interface, with acceptance criteria and reproduction steps attached.
9. Product and Experimentation
Whether feature impact is measurable and rollout is safe. This is what lets anything ship confidently, and it is the pillar that makes G8 enforceable: exposure has to be a separate control from landing.
L1 to L5: from all-or-nothing releases with no analytics, to staged rollouts, working kill switches, and a production signal that changes the queue and the rules.
10. Agent Tooling and Interfaces
The agent-facing surface itself: how context arrives, which tools can be called, what the guardrails are. This is the difference between an agent that can read your repository and an agent that can work in it.
L1 to L5: from default settings and nothing project-specific, to an evaluation harness that measures agent performance on this repository over time, with tool descriptions tuned rather than left as boilerplate. Tool descriptions are read on every invocation; a placeholder is a recurring runtime cost.
11. Multi-Agent Coordination
Readiness to host parallel workers: isolated working copies, scoped permissions per role, and a checker with a source of truth outside the artifact it judges. Multi-agent work is opt-in, not a default, and it burns several times the tokens of a single session, so the cost buys independence of opinion rather than parallelism.
L1 to L5: from one agent opened once in the repository, to orchestration patterns with documented failure modes. If a team runs no multi-agent workloads, this pillar is scored honestly and then marked "Not pursued" and excluded from the minimum. Skip that rule and the model produces a level that describes an artifact of scoring rather than the state of the repository.
Where the human boundary sits, and why it does not move
The inner loop belongs to the agent: explore, diagnose, implement, run checks, report. The outer loop belongs to a person: decide whether the approach is right, verify the diagnosis, approve, own the consequences. The boundary between them is evidence, not another prompt: a diff, tests, logs, and a short explanation tying them to the task condition.
A decision may leave a human only when an oracle exists for it, the oracle's feedback arrives no slower than the failure it must catch, and being wrong is reversible inside the loop's own authority.
The middle condition is the one that gets skipped, and it is why architecture is structurally human rather than human "until the models improve". Tests answer in seconds. Architectural erosion shows up over weeks. A check can only hold a gate against failures that appear no slower than it measures, so a suite that answers in seconds can hold a gate against a compile error and cannot hold one against erosion, because it never measures the thing that is moving.
Two more things worth stating plainly. Splitting maker and checker inside the same system yields claims, not proofs: a checker running on the same model with the same context returns correlated confidence, not independent verification. And at the top of the ladder, "the human reviews policy, not every diff" does not mean there is no human. Control moves to policy, boundaries, sampling, exceptions and consequences. Authority and accountability stay human at every level, including the top one, and accountability in particular moves by contract, never by pipeline.
What we have measured, and what we refuse to promise
Audits produce uncomfortable numbers, and the uncomfortable ones are the useful ones. From three repositories of one team, audited in a single day and every row bound to a revision:
- 93 to 13 red tests in one repository in one day, after the missing test plugin was restored and undefined names were removed and gated. A work list, not a scorecard.
- 1 of 8 parallel sessions publishing a live status, while the mechanism to do so already existed and was ignored.
- 6 attempts needed to confirm that a concurrent external reviewer was answering other sessions' prompts, which makes it not a checker.
That last group came from auditing the environment the sessions run in, treated as a repository of its own. It scored worse on parallel coordination than any of the three product repositories it maintains, and the three incidents that day were exactly the failures that pillar exists to catch: one session's commit absorbing 105 lines of another session's test code, stale measurements handed over as current, and a checker answering somebody else's question. Audit the environment, not only the product.
What this method does not promise. It does not promise autonomy. The honest output of an assessment is often "this class of change may land unattended and nothing else may", and that is a useful answer. It does not paper over open questions either: how to tell a stale test from a real regression, how to measure comprehension debt when the concept is defined and the metric is not, and what to do on a large legacy codebase, where published successes are thin and the cautious answer is to keep the ceiling low until an oracle exists.
How an engagement runs
- Interrogate the plan. Before assessing anything, establish what you believe you are building and why. That conversation produces the scope of the assessment.
- Audit the repository by looking at it. Every answer comes from the repository, not from an interview. An audit assembled from what a team says it does measures the team's self-image, which is a different and much less useful thing.
- Produce two outcomes. A repository level, and a separate landing decision for one class of change with its nine gates either closed or not.
- Run a measured loop first. One day, a baseline, one decisive metric with its direction written down, and an append-only ledger where a crash is a row with a status rather than a deletion. No class of change gets an unattended landing before a team has operated a loop whose results survive being read by somebody else.
- Set the ceiling from your own p75, install the rules where the agents will read them, and re-assess quarterly or after any major architectural change.
The first delivery path is the expensive one, because the method, the conventions and the discipline are established at the same time as the mechanism. Every later path is cheaper: the scope decision, the risk exclusions, the oracle and its negative control, the size ceiling and the gates repeat per path, while the method itself, the context conventions and the evaluation discipline are built once and shared.
Every measured incident described here is de-identified. The measurements, dates and failure mechanics are unchanged, because they are the evidence; repository names, revisions and product names have been removed.