Do our own measurement harnesses measure what they claim to?

Eight did not. One compared generated output against its own generator and reported a clean zero.

Outcome

Divergent

Published

22 August 2026

Question fixed

18 August 2026· stated by us, not proven

What came back.

Eight did not. Each returned a confident, precise, wrong number — and in every case the number was reassuring, which is why none was questioned at the time. A harness that reports a clean zero stops the search, and that is the shape of the harm.

Harnesses found defective: 8
Of those, reporting a false clean result: 5
Worst single miss: 11 live routes reported as unreached
Mutations that did not apply and read as passing guards: 2
Repeat of the same root cause: 3 self-referential scans

The system under test.

A harness returns a number, and a number ends an argument. This check treats the harnesses themselves as systems under test and asks the only question that matters about a measuring instrument: can it detect the thing it is looking for, and has anyone demonstrated that it can?

Population: All measurement harnesses in one production codebase
Method: Mutation — break the thing each harness claims to detect, and require the harness to notice
Period: Findings accumulated across roughly six weeks of ordinary work

The stated constraints

H1A harness must be able to produce a non-zero result — demonstrated, not assumed
H2A harness must not read a source derived from the thing it is comparing against
H3An assertion must be able to fail for the reason it was written
H4A harness must not match its own text
H5The granularity of a measurement must be the granularity of the claim

The second instrument.

Independence is a claim. The table below is the evidence for it — including the two things that ARE shared.

First instrument

The harness under test

Runs and reports a number: routes unwired, clauses uncovered, cycles present, rows ambiguous.

Second instrument

A deliberate mutation, and a check that the mutation applied

The defect the harness exists to find is introduced on purpose. The harness must then fail. Critically, the mutation itself is verified to have landed — a mutation that silently did not apply and a guard that genuinely held produce identical output, and we were fooled by that twice.

Shared?	What	Note
Not shared	The mutation and the harness	The mutation is applied to the subject, not to the measurement. It can therefore falsify the harness.
Not shared	The anchor check	Verifying that a mutation applied is a separate assertion about the edit, not about the code under test.
Shared	The author	The same person wrote most of the harnesses and most of the mutations. A blind spot in one is likely present in the other, and at least two of the eight below were found by accident rather than by this procedure.
Shared	The codebase	Every harness and every mutation reads the same source tree.

The reasoning

Five steps, each checkable without a computer.

This is the part that does not require you to trust the solver.

01A sweep compared a specification against code generated from that specification.Every name it looked for appeared in the generated file by construction, so the comparison was a tautology and it reported zero gaps. The generated artifact had to be excluded before the sweep meant anything.

02The same sweep matched whole paths, and the code assembles paths in pieces.A path built by joining two fragments matches no search for the complete string. Eleven routes in daily use were reported as reaching nothing. Matching on a distinctive final segment fixed it.

03A measurement of ambiguity manufactured its own first answer.It split each labelled row at the first digit — which deletes the section reference that disambiguates the row. Every ambiguous case it reported was an artifact of the split, and changing the split changed the entire result set.

04Three scans matched their own text.A test asserting that a module does not mention another module fails the moment the module explains, in a comment, why it does not. The subject and the instrument were the same bytes. Reading the code structurally rather than as a string is the fix.

05An assertion was true whatever the code did.It compared a value against something the function could never return, so it passed identically against a correct implementation and a broken one. It had been green for months and was measuring nothing.

06A count was right at one granularity and wrong at the one that was claimed.Aggregating modules into packages turns two unrelated dependencies into an apparent cycle. Five at package level, one at module level — and only the module-level figure answered the question being asked.

07A selector picked the wrong one of two similar objects.Two structures shared three field names; the harness selected on those three and got the internal one rather than the one that leaves the building.

08And twice, a mutation did not apply at all.Shell quoting mangled the pattern being introduced, so nothing changed and the harness passed — which is exactly what a working guard looks like. Every mutation now asserts that its anchor was found before it concludes anything.

What this does not settle.

Eight is a count of what was found, not of what exists.: The procedure that found these is the same one that missed them for months. There is no basis for claiming the remainder are sound, and the honest reading of this entry is that the number is a floor.
Two were found by accident, not by the method.: A finding arriving by luck is evidence that the systematic route did not cover that case.
The mutations were written by the person who wrote the harnesses.: A defect neither imagines is invisible to both. This is the weakest joint in the check and it is not repairable from inside.
It says nothing about whether the underlying code was correct.: A harness can be broken while the thing it measures is fine. Three of these were measuring code that turned out to be right; the harm was the false confidence, not a defect underneath.

What would change the conclusion.

A harness with a positive control — one that fails when the detector is disabled — is not vulnerable to the clean-zero class at all, and several now carry one.
Structural reading instead of text search removes the self-referential class entirely.
Somebody outside the codebase re-running these measurements would address the shared-author limit, which nothing internal can.
A ninth instance would not change the conclusion. A stretch with none, found by a procedure known to work, would.

Artifacts

identifiedThe corrected harnesses and their positive controls: Each of the eight, with the self-check that makes a silent failure impossible. Identified by content hash; not yet published as downloadable files.
unavailableThe mutation log: What was broken, whether the anchor was confirmed to have applied, and whether the harness noticed. Held internally; not published.

Revisions

22 Aug 2026
Published.

A corrected figure stays in the record. Silently fixing one would make every other figure unverifiable.

← All checks The method →