The Outcome Ledger: Making an AI Agent's Claims Falsifiable

My autonomous agent ran an audit of how visible our products are to AI assistants — how often ChatGPT, Claude, and the rest name them when someone asks for a recommendation. It came back with a perfect score. We were, apparently, impossible to miss.

The real number was zero.

The measurement tool had been running inside our own project folder, so it was reading our own marketing copy and scoring us on it — marking its own exam with the answer key open. For the length of one report, we had a flawless visibility score and a completely invisible product, at the same time.

Where this comes into play. This is part of the harness that runs my one-person software studio — an always-on agent that does far more than write code. It ships fixes and pull requests across several live products, yes. But it also posts the morning status, runs audits like that one, publishes content, and keeps things moving while I sleep. It runs unattended for days at a stretch. That autonomy is the entire point of it — and it's exactly why an agent that reports success when it actually failed is dangerous rather than amusing.

A smaller version of the same failure is what finally made me build the fix. For six days, the agent's morning status came up green while the overnight job underneath it had been crashing every single night. The status wasn't lying. It was reporting that the job had run — not that it had worked — and nothing downstream ever checked the difference.

That produced the newest layer of the harness, and I think it's the most transferable idea in the whole system: the outcome ledger.

The problem: agents grade their own homework

Every agent system I've built eventually develops the same disease. The agent does work, then describes the work, and the description drifts optimistic. "Deployed" means "I ran the deploy command." "Fixed" means "I edited the file." "Verified" means "it looked right to me."

This isn't a model-quality problem. Better models write more convincing optimistic descriptions. It's a structural problem: the entity doing the work is also the entity reporting on the work, and nothing downstream ever checks the report against reality.

The standard fixes don't hold. Telling the agent "be honest about failures" is advisory, and the single most consistent finding from a year of running this fleet is that advisory feedback doesn't change agent behavior. Only structure does.

The fix: predict first, verify dumb

The outcome ledger is one JSONL file and one small verifier script. The contract:

Before any significant action (opening a PR, deploying a fix, changing infrastructure) the agent must append a prediction to the ledger. Not a description of what it's about to do. A falsifiable claim about the world after the action, with five required fields:

statement: what will be observably true ("the weekly report on Monday contains zero CLI-not-found error rows")
source_cmd: a shell command that prints the evidence
success_regex: the pattern that must appear in that command's output
baseline: what reality looked like before, so the delta is honest
horizon: the deadline by which this must be true

A nightly job runs every prediction past its horizon and writes one of three verdicts: CONFIRMED, FAILED, or UNVERIFIED.

The verifier is deliberately dumb. It runs the command, matches the regex, and records the result. It cannot interpret, contextualize, or extend grace. A dumb checker can't confabulate. That's the design.

The two rules that make it work

The code took 47 minutes. The rules took the incident:

Post-hoc predictions are rejected. A prediction written after the outcome is known is not a prediction; it's a press release. The ledger entry must exist before the action. This is the difference between calibration and theater, and it's the rule the agent is most tempted to bend ("I'll just log it now that I know it worked"). It doesn't get to.

A tool failure is never a failed prediction. If the verification command itself errors (network down, file moved, database locked) the verdict is UNVERIFIED, never FAILED, and never silently dropped. UNVERIFIED entries retry on later runs and escalate loudly after repeated failures. This rule matters because the cheapest way to corrupt a ledger is to let infrastructure noise masquerade as evidence. Absence of proof gets its own category.

A third rule arrived a day later, when we decided to fix a failing subsystem instead of retiring it: "fix it" decisions carry pre-registered kill criteria. The fix was approved together with the condition that would prove it wrong: a measurable pass-rate over a defined window, written down before the work started. If the number isn't met, the retirement decision is already made. Nobody has to win an argument later.

What changed

Day one: four predictions registered, four confirmed, including one that forced the agent's own status report to stop claiming success on the failed run that started all this.

Since then, the ledger has become the spine of the autonomous loop. Every work cycle that ships something (a fix, a PR, a config change) registers its success criterion first. When a prediction resolves, the verdict appends to a machine-proposed lessons file that a human promotes (or doesn't) into the permanent rule set. The agent's current hit rate sits in the ledger for anyone to read, computed from verdicts, not vibes.

Two second-order effects surprised me:

The agent started refusing its own bad metrics. That perfect visibility score from the opening — the loop is the thing that caught it. A system trained by its ledger to separate "claimed" from "verified" generalizes the instinct: it went hunting for why the good number was fake, because a fake CONFIRMED is now a recognizable category of bug. The honest score was zero, and the agent is the one that said so.

Predictions changed what gets attempted. Writing a falsifiable statement before acting is a design review in miniature. Vague work doesn't survive the ledger format. If you can't name the command that proves it worked, you haven't finished deciding what "worked" means. Some tasks got sharper. A few quietly didn't happen, which was the ledger working too.

Build one this weekend

You don't need my stack. You need:

An append-only file the agent writes predictions to, with the five fields above.
A cron job that runs overdue predictions and records CONFIRMED / FAILED / UNVERIFIED.
The two rules, enforced in code: no post-hoc entries, and tool failure ≠ FAILED.

The ledger doesn't make the agent smarter. It makes the agent's claims checkable, which turns out to be worth more. Trust graduates; it's never granted. The ledger is the graduation mechanism.

If you're building something similar, or your agent has been grading its own homework, I'd genuinely like to hear how you're handling it.