The failure that costs real money is not a model refusing to answer. It is a model writing a confident paragraph with one number in it that came from nowhere, and a human pasting that paragraph into a board deck.
This checks the narrow thing that can actually be checked with certainty: does every figure in the text appear in the data it was supposedly written from. No model, no API key, no network call. Deterministic string and number matching, which is the point. A checker that needs a model to check the model has just moved the problem.
Four summaries written against the same synthetic quarter of sales data. One of them is clean.
A pass rate on its own is an unverified claim wearing a number. Every figure checked is listed, so the residual is counted rather than implied.
| Figure | Verdict | What the source says |
|---|
Every other button on this page shows the subject failing. These three show the tool failing, which is the only failure that would cost you anything. Click them. They are wired to the same code as everything else, and they are wrong on purpose.
Why these are here. On the day this shipped I had it attacked by an adversarial reviewer that was handed the files and nothing else, no context and no explanation of what they were for, and told to refute rather than to check. It found nine defects. Six are fixed and the fixes are commented in the source with the failing case that produced them. The three above are not defects, they are the honest edge of a lexical tool, so they are on the page instead.
What it caught that is now fixed: a sign-blind regex that passed -298966.50 as traced, which on a checker recommended as a build gate meant signing off on a revenue sign inversion. A fabricated figure tracing clean to the 1 in a filename. A correct document failing because its date said 31 March 2026 instead of 2026-03-31. And the quote box on this page splitting sentences at the decimal point, so it printed 7 percent from a claim that read 23.7 percent, which was visible on page load to anybody who scrolled.
My own end to end gate passed all four of those. It tested whether the tool catches a fabrication and never once tested whether a wrong number can slip through, and the expectations were written by the same person who wrote the test cases. That is a mirror, not a check. Finding it cost one reviewer and about ten minutes, which is the cheapest thing on this page.
Catches: a figure that appears nowhere in the source. A figure that is close to a real one but not equal, which is the rounding drift that survives a skim. A date or a link that was never in the data. These are the three that get past a human reader, because the sentence around them reads perfectly.
Does not catch: a figure that is present in the source but describes the wrong thing. If the source says returns were 4.2% and the summary says growth was 4.2%, this passes it, because the number is genuinely there. That is a semantic error and this is a lexical tool. Saying so is not a disclaimer, it is the specification.
Also does not catch: a claim with no numbers in it at all. "Performance was strong" cannot be traced, so it is not scored. A summary that passes with zero figures checked has told you nothing, and the scoreboard says so rather than showing a green tick.
Wire this in front of any generated summary and it becomes a build gate: nonzero untraceable figures, nonzero exit code, the document does not ship. That is roughly a hundred lines in a build pipeline, and it is the difference between trusting a model and verifying it.