How this was built

Most of the work was finding out what was wrong.

Every project reports what it found. This one kept what it got wrong too — several of those errors would have left the numbers looking perfectly reasonable.

9problems found and traced to a cause
5rules fixed before the answer was known
4pieces that survived their own checks

Half of the entries are things that went wrong

Eighteen recorded entries across six phases of the build

rule
one square = one entry · orange = something went wrong012345Setting upReading the dataMeasuringPredictingWhat we foundWhere it leadsSetting up — decidedSetting up — decidedReading the data — went wrongReading the data — went wrongReading the data — went wrongMeasuring — went wrongMeasuring — went wrongMeasuring — decidedMeasuring — builtMeasuring — builtPredicting — went wrongPredicting — decidedPredicting — builtWhat we found — went wrongWhat we found — went wrongWhat we found — went wrongWhat we found — decidedWhere it leads — builtentries in this phase
  • went wrong
  • decided in advance
  • built and survived its checks
Details & limits
Every orange square is an error that ran without complaint until someone went looking. Three of them would have left the published numbers looking entirely reasonable.
01

Setting up

First, make it rebuildable.

Before any biology: could a stranger rebuild this and get the same numbers?

DecidedADR-0001

4languages in one lockfile

One lockfile for four languages

Four languages and a dozen tools in one lockfile, so a stranger reproduces a number rather than approximating it.

DecidedADR-0002

0 MBlarge files committed

Nothing large goes into version control

Large files are refetched from checksummed recipes, never committed. Reproducibility rests on rebuilding, not storing.

02

Reading the data

Public data is not ready to use.

Three errors in how public data described itself. Each would have run without complaint.

Went wrongADR-0004

72 ntupstream context, measured not quoted

The published reporter window was wrong in both places

The preprint said 150, the journal said 147, the data said neither. An off-by-three shifts every feature silently.

Went wrong

0reads matching the recorded adapter

The adapter the archive recorded was not the adapter used

Uncaught, every read carries seventeen stray bases into alignment, nothing errors, and the rhythm the project rests on vanishes.

Went wrong

62.6%structural RNA in one library

Most of a footprint library was not messenger RNA

Annotation-based filtering caught a sixth of it. The obvious implementation looks reasonable and removes a fraction of its target.

03

Measuring

Decide the rule before you look.

Ribosome data can be talked into showing anything, so the pass rule was frozen first.

BuiltADR-0008

24.8Mribosome positions mapped

Ribosome positions, resolved to the codon

Reading position is inferred from each fragment's length and ends, turning reads into a map of where ribosomes sat.

DecidedADR-0009 · ADR-0010

3conditions fixed before the data

The pass rule was written before the answer was known

Density past a stop rises for reasons that are not readthrough, so three conditions were fixed in writing beforehand.

BuiltADR-0010

3 of 3conditions met by G418; 0 of 3 by SRI-37240

The assay tells a readthrough drug from a stalling one

SRI-37240 failed in the direction its stalling mechanism predicts. An assay that cannot separate the two cannot report a null.

Went wrong

1confounded cell line

The drug was also treating a contamination

G418 is an antibacterial, and one cell line was infected. The drug under study was also clearing the contamination.

Went wrong

6,724transcripts the fixed filter removes

A safeguard that never ran, for the whole project

It searched for a feature name the annotation does not use, so it excluded nothing. Correcting it changed no conclusion.

04

Predicting

Reproduce someone else, exactly.

Scoring seventy thousand variants is only worth doing if the scoring is checked independently.

Built

<1e-8agreement with the independent oracle

A published model, reproduced and checked against an independent implementation

Checked against an independent R implementation over sixty rounds. Reproducing someone exactly is the cheapest way to catch a misreading.

Went wrongADR-0015

25,917variants under a placeholder condition

A quarter of the diseases were one non-disease

A placeholder meaning “condition not provided” looks like a real identifier. It slipped the filter and topped the list.

DecidedADR-0014 · ADR-0016

4columns, never one score

Four separate conditions, never multiplied together

Any one at zero makes the product zero, so they stay four columns. A single score would hide which one failed.

05

What we found

Three answers, and none was the one we wanted.

Two hypotheses came back negative and a dataset could not answer its question.

Went wrong

5 / 70,384variants where one drug is distinguishable

Honest uncertainty dissolved the ranking

With intervals rather than point estimates, the ranking dissolves. Uncertainty was not decoration on the result — it was the result.

Went wrongADR-0020 · ADR-0021

p = 0.21the control that decided it

Our own hypothesis did not survive its decisive control

Two shuffles supported it. The third, scrambling only within an amino acid, reproduced most of the gain — the signal is the residue.

DecidedADR-0022

30 sto widen the rule; it stayed

The pre-registration cost us the answer we wanted

Three positive differences and a published result agreeing, with one interval in the way. Widening the rule would have taken seconds.

Went wrongADR-0019

4libraries per arm actually needed

One dataset could not answer the question it was collected for

It missed our quality floor by 0.015%. The threshold did not move — and depth was never the constraint; replication was.

06

Where it leads

One question sequence can still answer.

Where readthrough stays uncertain, geometry can still settle a different question.

BuiltADR-0023

31.2%reachable under the declared editor panel

A third route, decidable from sequence alone

Can an editor be placed on this stop? Just under a third can, and relaxing the targeting rule reaches 89% — the limit is placement.