Products · Causal measurement over high-frequency time series
CIPHER
A measure that reports a relationship in pure noise — and the correction that stops it.
Transfer entropy is asymmetric: unlike correlation it can distinguish which of two series carries information about the other. It is also POSITIVELY BIASED — a histogram entropy estimator finds finite-sample patterns in noise, so unrelated series return a non-zero value. Most of the engineering here is the correction rather than the measurement, and most of the architecture record is a list of approaches that were built, examined and rejected.
What a silent failure looks like
A download hits a rate limit. The code catches it, moves on, and reports success. Nothing throws, no count looks wrong, and the missing range is invisible because there is nothing there to be noticed. Every measurement built on top of it is computed correctly on data that is quietly incomplete. It surfaced only when a later run with exponential backoff filled the holes.
432,072
rows recovered by fixing a retry that failed quietly · nothing had reported them missing
Why it exists
This domain met all four conditions.
The fourth is the one that disqualifies most software.
- 01The answer matters
- A measure that says two series are related when they are not is acted on exactly like one that is right. Everything downstream inherits the error and none of it can see the inheritance.
- 02Nothing independent checks it
- A biased estimator marks its own work. Nothing in the number it returns distinguishes a real relationship from the bias floor, which is why the floor has to be computed separately and removed.
- 03A different second computation is possible
- The bias can be estimated two genuinely different ways: by destroying the relationship and measuring what survives, or by an analytical shrinkage with a closed form. Both were built. They agree, and only one of them is deterministic.
- 04The failure is silent
- Non-zero on noise looks exactly like a weak real signal. And the ingestion beneath it failed the same way — a rate limiter quietly dropping data with nothing raised.
Built for
Anyone measuring directional dependence in high-frequency series.
Not for
Anyone looking for a signal to act on.
This page describes what the instrument MEASURES and how the measurement is corrected. What anyone does with a measurement is a different question, requiring registration this practice does not hold, and it is deliberately not addressed here or anywhere on this site.
Four things it will not do.
Design decisions with reasons, not missing features. Each one is a thing the instrument could do and will not.
- It does not publish a measurement it cannot separate from noise.
- The bias floor is computed and removed before a value is reported. A raw transfer entropy figure is not a finding; it is an estimator output with a known positive offset.
- It does not use a random correction where an analytical one exists.
- The shuffle-based correction returned a different answer on a different seed. A result that moves when nothing about the input moved cannot be replayed, and replay is the whole basis of the record.
- It does not treat a passing integrity suite as clean data.
- The audit reports pass, fail AND informational — and the informational findings are carried forward with the remedy attached rather than closed. Three are open by name.
- It does not claim the architecture is finished.
- The validation stage is specified and unbuilt, and the record says so. An architecture presented as settled while its own audit layer is missing is the failure this instrument exists to measure.
The machinery
Eight layers, and the part of each that is actually hard.
None of this is specific to one domain. It is the same machinery we build into somebody else’s system, in whatever domain their expensive answer lives.
01
Ingestion that cannot silently lose a row
Every insert is idempotent on a composite key, so a download can be run a hundred times and produce one result. Downloads resume bidirectionally from where they stopped, and a corrections table records every value the cleaner changed.
The hard part
The original code met a rate limit and SKIPPED that range without raising anything. The gap was invisible until a later run with backoff recovered 432,072 rows. A retry that gives up quietly is worse than one that fails loudly, and it is the same defect class the whole instrument exists to measure.
02
A columnar store that fits the machine it runs on
236,734,006 rows of minute and daily bars, compressed columnar, segmented per symbol so a query decompresses one symbol’s blocks rather than a whole chunk. Timestamps use delta-of-delta encoding, which is near-free on regular one-minute intervals.
The hard part
A write into a compressed chunk lands in an overflow area and is merged on read, so the choice is between slow writes at 25GB peak or fast writes that reached 150-200GB and exhausted the disk. The design keeps compression and pays the write penalty — the alternative crashed the database mid-load, and the recovery forensics are in the record.
03
Bias correction, moved from sampling to solving
The reported value is transfer entropy minus its estimator’s own floor. That floor is now derived analytically — a James-Stein shrinkage of the probability estimate toward uniform, with a closed-form optimal intensity (Hausser & Strimmer, JMLR 2009).
The hard part
The version it replaced was not wrong, it was STOCHASTIC. Twenty permutations per measurement, a different answer per seed, and roughly 80% of the runtime. Trading it for a closed form made the result deterministic and about twenty times cheaper — and gave up the one thing the shuffle had, which is that it makes no assumption about the distribution.
04
Look-ahead defended where it is easiest to commit
Volatility used to set a day’s thresholds is lagged, because that day’s own range has not happened yet at the moment it would be used. Rolling windows that would otherwise peek were replaced with stride-subsampled, strictly backward-looking ones.
The hard part
Look-ahead does not announce itself — it announces itself as unusually good results. Every defence here is reasoned and implemented, and NONE of it is yet checked by anything independent, which is why this page says the instrument is in build.
05
An architecture that mostly records what was rejected
Fifteen numbered revisions. A Bayesian aggregator was built and rejected for double-counting correlated inputs; an end-to-end sequence model was built and rejected for debugging opacity; a reinforcement-learned execution layer was rejected for the gap between simulation and reality; two graph networks were rejected because eighteen pairs is a lookup table rather than a graph, and a deep order-book model because a NumPy calculation reached 95% of the signal at 2% of the compute.
The hard part
The record also carries the risks in what SURVIVED, which is the harder half. The chosen meta-learner has no production-stable library and a conventional network reaches roughly 80% of its benefit; the chosen sequence model has little published evidence on this market, so the simpler predecessor is kept as a fallback. Naming those is what stops a converged architecture from becoming a settled one.
06
The gap the record names as the most consequential
A formal framework defining what the models are actually optimising for — what counts as an outcome, measured from what moment, net of what costs. It did not exist in any of the first thirteen revisions.
The hard part
Without it, every model optimises a slightly different implicit objective and the ensemble averages disagreements it cannot see. The record calls it the single most consequential design decision in the system, and identifies it as a gap rather than presenting the architecture as complete without it. That sentence is the reason this instrument is worth writing about.
How it is checked
Every figure is computed twice, by engines that share no code.
A checker that imports the model verifies that the solver ran. It does not verify that the answer is right.
Two independent estimates of the same bias
The original correction destroyed the relationship by permuting the source series twenty times and subtracted what survived. The replacement derives the same floor analytically, by shrinking the probability estimate toward uniform with a closed-form intensity. One samples the null; the other solves it.
A 24-check integrity suite that is allowed to say "warning"
21 passed and 3 returned informational findings — 99,016 equity and 56,731 futures bars outside market hours, 120,309 negative open-interest rows, and 398 zero-price bars on illiquid contracts. Each is carried with what to do about it rather than being closed.
An append-only corrections table
35,740 corrections recorded by type, with the original and replacement values kept. The audit survives a re-run rather than being overwritten by it, so the question "what did we change" has an answer months later.
A measurement that changed the architecture
The exploratory pass tested 16 stocks for normality and 16 rejected it, on two tests, at p<0.01. Fat tails everywhere meant squared-error loss was the wrong default, and every neural component moved to Huber or quantile loss. The finding preceded the decision rather than justifying it.
Published checks involving CIPHER.
Including the ones where it was wrong.
Starting
What you need before the first run.
- An account
- CIPHER is behind a sign-in. The data inside belongs to whoever signed in, and it is not shared between accounts.
- One real case
- Not a sample. The instrument is only interesting on something you already have an answer for, because the first useful result is the two engines agreeing — or not — on a figure you can check yourself.
- Somewhere to disagree
- If the second engine contradicts you, that is the product working. It is worth deciding in advance who looks at a disagreement, because an alert nobody owns is an alert nobody reads.