The method, stated so it can be checked against the register
A computed answer earns belief only from an instrument that could not have made the same mistake.
Everything below is a rule we hold ourselves to. Each one is visible in the published work — if a check in the register breaks one of these, that is a defect in the check, and we would want to be told.
Every check reports one of three states.
Never two. This is the part of the method most systems get wrong, and it is the part that matters most.
Established
Both instruments agree. The result holds, and the check says how it was reached.
Divergent
They disagree. The check names which was right, and why the other was wrong.
Unestablished
The check could not be completed — and this is an OUTPUT, not a blank. It states precisely what evidence would settle it.
Worked example
The two-state habit, in a protocol you already know.
Almost every address validator ever written reports valid or invalid. The specification supports neither reading — and we made that mistake on this page before we caught it.
You want to know whether an email address exists. You can ask the mail server that would receive it. It answers with a number, and the number is where the trouble starts.
- 250
What it means
The server accepted the forward-path for delivery. That is a statement about what it is willing to receive.
What it does NOT mean
That a mailbox exists. A server may accept everything at the boundary and reject or discard it afterwards — and many deliberately do, precisely to defeat this question.
- 550
What it means
The server refused the address at this moment, from this sender, on this connection.
What it does NOT mean
That the address is invalid. A refusal can be reputation, rate limiting, geography, or a policy about senders it has never seen before. The address may work perfectly for somebody else.
- 4xx / timeout
What it means
Greylisted, deferred, or no answer at all. The server has declined to settle the question now.
What it does NOT mean
Anything about the address. This is the state most validators do not have, so they file it under whichever of the other two is more convenient — and that choice is invisible to whoever reads the result.
What this page used to say
That a 250 establishes the mailbox exists and a 550 establishes it does not. Two states, stated confidently, on the page arguing for three.
The correction
Read against RFC 5321, which defines a 250 at RCPT as the server accepting and storing the forward-path — not as proof of a mailbox.
Accepted, rejected, indeterminate — and none of the three settles the question that was actually asked. It is dated in the corrections register, because a practice that quietly edits its own mistakes is asking for a trust it does not extend to anybody else.
The general shape: a system with two outcomes must put “could not tell” somewhere, and it files it under whichever one is cheaper to build. Nothing downstream can then distinguish a fact from a default — which is what makes the failure silent rather than loud.
What makes the second instrument independent.
Four rules. Break any one and you have a second run, not a second instrument.
- 01It is written from the specification, not from the first implementation.
- A checker written by reading the code under test inherits that code’s misreadings. The statute, the constraint list, the protocol — the source both instruments answer to — is what the second one is built from.Check 0031 demonstrates this →
- 02It shares no code with the thing it checks.
- A checker that imports the model’s own constraint objects verifies that the solver ran. It does not verify that the answer is right. No shared helpers, no shared constants, no shared parser.Check 0028 demonstrates this →
- 03It uses a different representation.
- Whole rupees against integer paisa. Plain tables against a compiled rules engine. A re-derivation from the finished output against a model that produced it. Different representations fail differently, which is the entire point.Check 0030 demonstrates this →
- 04Its result is compared figure by figure, not in aggregate.
- Two totals can agree while their parts disagree in opposite directions. A comparison that only checks the bottom line is a checker with a blind spot the size of its own arithmetic.Check 0029 demonstrates this →
The shape of every check
One question, two instruments, and a comparison that can come back three ways.
04 · From a question to an instrument
Eight moves, and three of them are about stopping.
This is the part that turns an idea into something that works. Each move below is what was actually done, on a real system, with the example attached.
01
Measure before designing anything
The exploratory pass runs before a single model is chosen, because its findings are allowed to overrule the defaults.
Where this comes from
Sixteen instruments were tested for normality and sixteen rejected it, on two tests, at p<0.01. Fat tails everywhere meant squared-error loss — the default in every framework — was the wrong objective, and every neural component moved to Huber or quantile loss. The measurement preceded the decision instead of justifying it.
02
Check the critique before acting on it
A review is an input, not an instruction. The first thing done with a criticism is to establish whether it is true.
Where this comes from
A reviewer stated the data source was capped at six or seven months of minute data, which would have halved the corpus. Checked: false. The limit is a hundred-day PAGINATION window, not a depth limit, against an allowance of a hundred thousand requests a day. The original design stood, and it stood on evidence rather than on defending it.
03
Accept the idea; reject the implementation
A good argument for a change is not an argument for the specific thing proposed. These are separable and almost nobody separates them.
Where this comes from
The same reviewer was right that a point estimate should carry its uncertainty, and wrong about the library — the one proposed had no export path, which meant raw interpreted inference inside a process architecture that could not absorb it. The idea was kept and reached another way, with conformal intervals on a model that exports cleanly. Same benefit, no deployment risk.
04
Choose parts that cannot fail the same way
Two components that agree are only worth having if they could have disagreed. That has to be designed in, not hoped for.
Where this comes from
Two models were selected to attend to orthogonal dimensions of the same input — one across time, one across features — so a pattern with no confirming indicators is caught by the second, and confirming indicators on an exhausted pattern are caught by the first. The anomaly pair splits the same way: one sees sequences, one sees single points. Neither can make the other’s mistake.
05
Reason from the data, not from the default
A default is somebody else’s answer to a question about their data. It is worth re-deriving against yours.
Where this comes from
Storage moved from eight-byte to four-byte floats after working out what precision the source actually carries: prices with two decimals leave at most four or five meaningful digits after computation, against seven the smaller format provides. Roughly 125 GB saved, a maximum precision loss of 0.0000059%, and better compression because simpler bit patterns encode smaller.
06
Name what will bite before it does
The things that sink a build are usually not the interesting decisions. They are the unglamorous requirements nobody wrote down.
Where this comes from
Five were identified and recorded as gaps before implementation, none of them a choice of model. The most consequential: without explicit class weighting, a system whose positive cases are one to two percent of the data learns to answer "no" to everything, reports ninety-eight percent accuracy, and produces zero output. It would have looked like a success.
07
Keep the record of what was rejected
Fifteen numbered revisions, and the useful half of that document is the discards — with the reason each one failed.
Where this comes from
A Bayesian aggregator, rejected for double-counting inputs that were correlated. An end-to-end model, rejected for debugging opacity. A reinforcement-learned execution layer, rejected for the gap between simulation and reality. Two graph networks, rejected because eighteen pairs is a lookup table rather than a graph. A deep order-book model, rejected because a NumPy calculation reached ninety-five percent of the signal at two percent of the compute.
08
Establish when to stop
Refusing to settle is only a virtue while the next change still moves the answer. After that it is superstition, and the way to tell them apart is to measure.
Where this comes from
A horizon formula was found to be algebraically broken — two ratios multiplied so that the middle term cancelled, pinning ninety-seven to ninety-nine percent of cases to the maximum. The "dynamic" parameter was static, and nothing had thrown. Six rounds of research produced the replacement, then showed that every remaining variant correlated above 0.99 with it and moved the result by about five units. Further work was measured to be worthless, and stopped.
None of these is specific to a domain. They are what happens between someone saying “nobody can answer this” and there being something that does — and the reason an instrument takes months rather than weeks is that six of the eight are about being wrong on purpose, early, where it is cheap.
When a check is publishable.
Four conditions, and the last is the one that keeps a register honest.
The apparatus that measures correctness can itself be silently wrong — check 0027 is eight of ours that were. That is the strongest argument for a second instrument we can make, and it is against ourselves.
01
The question was decided before the check ran.
A check written after the result is known is a description, not a check.
02
The method is stated in enough detail to be re-run.
By someone else, without asking us anything.
03
Agreement is reported as loudly as disagreement.
A register that only publishes divergence is a bug list.
04
What could not be established is named, with what would settle it.
An unfinished check is a result. An unmentioned one is a gap.
What the method does not do
A second instrument narrows the space of possible errors. It does not empty it.
Two instruments can share a misreading of the specification itself. Both can be given the same wrong input. Neither can check a fact nobody recorded. Where that is the case, the check comes back unestablished and says so — which is the whole reason the third state exists.
If a computation in your business is wrong today, what would tell you?