Measuring correctly in noisy data

1. 01

   No look-ahead

   Every feature has to be knowable at decision time. Decisions are made on day t and positions are taken on day t+1, and tests lock that order in place.

2. 02

   After costs

   Nothing counts until fees, measured spreads and market impact estimated from the order book are taken out.

3. 03

   No survivorship bias

   The universe is rebuilt every day from that day's information; coins that were later delisted stay in until the day they died.

4. 04

   Pre-registration

   Hypothesis, metric, pass threshold and expected effect are written down before a test runs, and not changed afterwards.

5. 05

   Independent reproduction

   Every number that drives a decision is produced a second time, independently. A number that doesn't match can't be used.

6. 06

   Mutation testing

   We check that critical safeguards really catch errors by breaking the code on purpose. A test that catches nothing doesn't count.

7. 07

   Power first

   A test isn't run unless it has a reasonable chance of detecting the expected effect. “Didn't pass” from a weak test carries no information.

8. 08

   Degrees-of-freedom budget

   A model's effective degrees of freedom are capped by what the information in the data can support.

9. 09

   One statistics core

   Sharpe ratios, confidence intervals and power come from a single shared module, so the same number is never computed two different ways.

10. 10

    Degenerate-solution checks

    Every robustness technique is assessed together with a check that the model has not collapsed to a constant or neutral forecast.

11. 11

    P&L in simple returns

    Profit and loss is computed from simple returns, not from summed log returns.

Every idea follows the same path. An idea that fails a step does not move on to the next, and every result is recorded.

1. 01

   Hypothesis

   What should work, why, and how large the effect is expected to be.

2. 02

   Pre-registration

   Metric, threshold and power calculation are written down before the data is seen.

3. 03

   Synthetic lab

   On data with a known signal: does the model actually capture information?

4. 04

   Walk-forward test

   On real data, trained only on the past and measured going forward.

5. 05

   Independent reproduction

   Numbers that drive decisions are reproduced independently in a second run.

6. 06

   Live forward test

   On real-time market data, with results read over a window fixed in advance.

In noisy data, the errors that inflate a result usually sit in the measurement, not in the model. These are a selection of the ones we ran into during this programme and now block with automated tests. Each card shows the apparent value next to the properly measured one.

• Sharpe +1.176 → −0.437

  Balancing tricks can collapse a model to neutral

  With label smoothing and class weights used together, the model learned to say “neutral” every day. A result that looked good in the report was negative once measured properly.

• t +23.5 → +1.0

  Overlapping windows inflate t-statistics

  When multi-day returns are sampled daily, the observations get counted as if they were independent. Measured without overlap, the effect disappeared.

• t 3.0 → 0.8

  Summed log returns are not P&L

  Adding up log returns systematically flatters the short side. Computed with simple returns, the same signal lost its significance.

• 20–50×

  Counting every asset as an independent bet

  Assets move together, so treating each name in the cross-section as an independent bet inflates the information ratio twenty- to fifty-fold.

• IC t = 22 → gross −0.21

  Rank correlation is not profitability

  A statistically very strong ranking signal lost money even before costs, because returns concentrate in the tails. That is why dollar-weighted measurement and a decile profile are mandatory.

• Sharpe 12.3 → 1.25

  Scoring a decision on the same day's return

  Without a one-day shift the result was impossibly good. Adding a single day of delay showed the real picture.

• score 0.80 → 0.51

  Row-order leakage in cross-validation

  Splitting folds by row order instead of time left neighbouring dates in the training set. Split by time blocks, the score dropped.

• Sharpe gap up to 1.36

  Hardware changes results too

  The same model trained on CPU and on GPU (MPS) gave different results. Numbers that drive decisions are now produced on one device with fixed seeds.

• |z| = 158

  Unscaled inputs poison the exponential gate

  One un-winsorised feature showed a value 158 standard deviations out. In an exponentially gated model, a single observation like that takes over the memory, which is why inputs are converted to cross-sectional ranks.

Every idea we try goes into an experiment catalogue, whatever the outcome. That keeps the same idea from being retried unknowingly and lets us account for the effect of multiple comparisons. The 1,583 model runs from the synthetic lab are not included in this count.