The Maintainer Loop: Keeping Books on the Bookkeeper

An LLM reviewer once invented a paper for me. Plausible authors, plausible venue, a title tuned precisely to my project's niche, and it nearly landed in my related-work table. Nothing with that title exists. Days later, a dead API key produced an audit that looked immaculate: zero violations, zero contradictions, green across the board. It looked clean because nothing had actually been asked. Zero failures is also what silence looks like.

Both burns are now permanent regression tests in CI: test_fabricated_citation_is_rejected and test_dead_key_cannot_report_clean. And the maintainer loop this article describes is those two lessons generalized into an architecture. Every past failure became a check the agent cannot skip. Every consequential decision became a string or set operation the agent cannot argue with.

Benchmarks rot on a schedule

Quick context for new readers. Ontology Debt is my small open-source tool (PyPI ontology-debt, Apache-2.0) that audits LLMs against world-model commitments you declare in YAML. Scoring is fully deterministic: there is no LLM judge, and every verdict is a string comparison reproducible from committed transcripts. It draws a strict line between violations (the model contradicts you) and contradictions (the model contradicts itself), and it keeps a persistent severity-weighted debt ledger. The previous article covered the two-tier capability-gradient audit. This one is about what happens after you ship such a thing, because what happens is rot.

New models ship monthly, and a results table that was honest in March is quietly misleading by June. Prior art accumulates, and a related-work table decays the same way. Frontier models eventually hold every pack at zero, so tiers saturate; the benchmark stops discriminating and starts flattering. I am a solo maintainer. The tempting answer is obvious: let an agent maintain it. My thesis is that this temptation is exactly backwards unless you build the distrust first. An unsupervised agent maintaining a trust artifact is how you convert a benchmark into a rumor.

The rule: agent proposes, code decides, human merges

The whole design hangs on one rule. The agent may propose anything; deterministic code decides everything; a human merges. Every consequential yes/no in the maintainer loop is a string or set operation in committed, tested code:

• "Is there a new model?" is a set difference over a committed raw API payload.
• "Does this paper exist?" is a re-fetch by id with a verbatim quote containment check.
• "Is the tier saturated?" is arithmetic over committed transcripts.
• "May this diff auto-merge?" is a path allow-list plus a recompute-and-compare.

An LLM appears in exactly one module of the maintainer code, fenced to drafting candidate test scenarios, and even there its output must pass the same loaders that gate hand-written packs. Everything else is roughly stdlib-only Python in src/ontodebt/maintainer/, shipped as five GitHub Actions workflows. Everywhere else, "the agent decided" is not a sentence that can occur. The code is public at github.com/dimaggi-ai/ontology-debt, with docs/maintainer-loop.md as the design doc.

Loop one: model watch, daily

Every day, a job pulls the Anthropic, OpenAI, and Gemini model-list APIs plus public Hugging Face org feeds, and diffs the result against a committed registry (maintainer/known_models.json). Set difference in one direction finds new models. A hash change on a known id catches silent metadata bumps. Disappearance flags deprecation. Relevance is a committed regex gate, so "worth an issue" is reviewable config in the repo, not a model's opinion on a given Tuesday.

When a new relevant model appears on a provider the harness can audit, a second job runs the ordinary deterministic audit and opens a PR with the models.yaml entry, full transcripts, and the regenerated report. That job sits behind a GitHub environment requiring my explicit approval click before any API spend happens. The PR body is rendered deterministically from the report. No LLM writes the prose a reviewer relies on, because prose a reviewer relies on is exactly the surface where confabulation hides.

Loop two: prior-art watch, weekly

Once a week, committed queries run against arXiv, OpenAlex, and GitHub. Candidates are scored by a committed keyword table, and the scoring is presence-based, so keyword stuffing cannot inflate a score. Then comes the check that Burn A paid for: the fetch-then-quote verifier. A candidate surfaces only if its id re-fetches from the source API and the stored verbatim quote appears in the re-fetched text. A fabricated citation cannot produce a real fetched quote, no matter how plausible its title sounds.

The output is an issue; the watcher has no write access to the related-work table. The human alone writes any related-work sentence. And in the spirit of Burn B: a watch where every searcher fails every query goes red rather than reporting a quiet zero, because "we found nothing" and "we asked nothing" must never look alike.

Loop three: saturation and the one caged LLM

Saturation detection is pure arithmetic: at least 2 models holding a tier at zero violations, zero contradictions, and zero link breaks, on runs that are both valid and complete. When that fires, it opens an issue, never a pack. Deciding the benchmark needs to get harder is a computation; making it harder is a judgment call.

Drafting a harder pack is human-dispatched and spend-gated, and this is the one module where the LLM lives. A proposer call drafts scenarios with numbered mechanical derivations. Each scenario is then validated by the repo's own loaders, and invalid ones are dropped, not repaired, because repairing generated content is how validation quietly becomes generation. Then an independent adversary LLM call per scenario tries to argue for a different defensible answer, and anything not answering an explicit UNDISPUTED dies. Fail-closed: ambiguity is fatal. The surviving pack lands on a PR labeled never-automerge, and a draft becomes a scoring pack only when the human promotes it.

What the burns actually taught

The transferable lesson from both burns is not "be careful." It is: when a failure burns you, do not write a prompt instruction about it. Write a test that fails forever if it happens again. Each burn became a named regression test (test_fabricated_citation_is_rejected, test_dead_key_cannot_report_clean) and a family of structural rules with the same shape, which is that empty must never impersonate clean:

• An errored provider pull turns the run red rather than reading as an empty model list.
• A monitored provider returning zero models is an error, full stop.
• A run with more than half its probes errored can never count as clean.
• A partial run (a --limit smoke test) can never count as clean.
• A probe-dodging run, where the model heavily refuses to conform to the answer format, can never count as clean.
• A missing API key makes a provider loudly UNMONITORED rather than silently skipped, and a monitored-to-unmonitored transition (say, a deleted secret) fires its own alert issue.

None of these live in a prompt.

Nineteen bugs before the first run

Before the loop ran once, it went through three independent review passes: correctness, security and CI-safety, and guardrail conformance, with every finding then independently re-verified against the code. 19 real findings survived verification, and all were fixed before launch. The best ones are worth narrating, because every one of them had already survived my own reading of the code.

The most autonomous feature was dead on arrival. The auto-merge gate demanded byte-identical report regeneration, but reports embed a wall-clock "Generated:" timestamp, so the comparison could never pass. Fixed with a timestamp-insensitive comparison.

An ordering bug would have buried the very first real detection forever. State bookkeeping was committed before the notification issue was opened, so any issue-creation failure (and a missing label guarantees one) would mark the new model as known and never re-fire. Fixed by inverting the order: surface first, record last. Now a failure re-fires as a recoverable duplicate instead of a silent loss.

A smoke test could impersonate a perfect model. Saturation originally required only "answered more than zero probes with zero violations," so a 2-probe partial run would count as a model holding a 300-probe tier clean. Fixed by requiring complete coverage and a minimum answered fraction.

The LLM's own output could steer a filesystem write. The drafted pack's id field, taken verbatim from the model, became the output filename; a deviating response containing ../ could write outside the drafts directory. Fixed: the id is always the CLI-validated one, and the model's id is ignored.

False deprecation alarms were baked in. The Hugging Face feed is a sliding most-recent-50 window per org, and 6 of the 8 watched orgs already sit at exactly 50, so every new release would push an old model out and fire a false "deprecated" trigger. Fixed: aging out of a window is not deprecation.

And the spend gate could fail open. Referencing a nonexistent GitHub environment auto-creates it with no protection rules, silently removing the human approval. Fixed with an explicit confirmation variable that I set only after configuring the required reviewer.

In fairness to the review, it was itself multi-agent: three lenses, each finding re-verified adversarially. Then the first live run found a 20th bug the review missed, a git command that goes fatal on an empty directory in an edge case. The surface-first ordering made it harmless: nothing was lost, and the fix was one guard. That is the pattern worth internalizing: review reduces the bug count; ordering decides whether the survivors are recoverable. After all fixes, 78 tests are green, and they read as a design document written in assertions.

One admission is owed, because the bug list undercuts my own slogan. The worst of these bugs lived in the deterministic half of the system. The auto-merge gate was a string comparison, and it could never pass. The saturation check was arithmetic, and it was arithmetic about the wrong definition of clean. Determinism made neither of them correct. What it made them was replayable, so review could corner them and a test could pin them down. An LLM that is wrong is wrong differently every time you look. A deterministic gate that is wrong fails identically on every run, which is what makes it fixable once and provably fixed. Wrapping your agent in deterministic checks is not a recipe for being right. It is a recipe for being findable when you are wrong.

First receipts

The loop's first autonomous act was a bookkeeping commit to main: a registry update carrying no factual claims. Its second was opening a true issue: "Saturation: 4 model(s) hold gradient-frontier clean - time for a harder tier," computed from committed transcripts, so you can recompute it yourself without trusting the agent, or me. Four current models hold the project's hard tier at zero, and the loop noticed on its own, arithmetically. Steady-state cost is roughly zero fixed, since public-repo CI minutes are free, with metered API spend only when a human approves an audit.

The ladder, and the off switch

Nothing auto-merges today. The one candidate for future auto-merge is a report regeneration that recomputes identically from committed run files, and even that path ships disabled behind an explicit variable. Spend always requires an approval click. Gold labels, citations, merges, and releases are human forever. Autonomy widens by earned tiers, never by default; years of running infrastructure taught me that trust granted by default is just an incident with a delay.

And the honest coda: after building and verifying all of this, I switched the schedules off. There is a kill-switch variable, and it is set. I run the whole thing as a gated runbook on my own cadence instead, because the loop's job is to make my check-in cheap and complete, not to remove me from it. The schedules can come back when the loop has earned them.

Steal this checklist

If you are building agent-maintained anything:

1. Turn every past failure into a permanent regression test, not a prompt instruction.
2. Make every consequential decision a string or set operation over committed state: a check that is wrong the same way every run can be cornered once and fixed for good.
3. Surface before you record, so failures re-fire instead of burying triggers.
4. Treat "empty" as an error whenever empty is also what failure looks like.
5. Verify citations by re-fetching, never by trusting generation.
6. Attack generated content with an independent adversary that must explicitly clear it.
7. Gate spend behind a human click, and confirm the gate exists before trusting it: referencing a CI environment that was never created makes GitHub conjure it silently, with no protection rules.
8. Ship a kill switch.
9. Widen autonomy by earned tiers, never by default.

Books on the bookkeeper

The first article's conviction was that you cannot align what you cannot account for, applied to models. This is the same conviction applied one level up: if an agent keeps the books, someone must keep books on the agent. Declare your commitments, measure against them, enforce the boundaries, and apply all three to the maintainer itself. At DIMAGGI that conviction splits into two legs, Ontology Debt as the measure leg and Tool Guard as the enforce leg, but the conviction matters more than the products.

Everything above is checkable: github.com/dimaggi-ai/ontology-debt, with docs/maintainer-loop.md as the design doc, every review finding logged in docs/maintainer-loop-review.md, and the tests as receipts.