How the data was made

Every stage from raw collection to the labeled unit databases that all atlas figures are computed from — scripts, models, parameters, seeds, counts, costs, failure modes, and the exact commands. Scope ends at the data layer; the visualizations downstream evolve continuously and are documented in the atlas itself. All counts verified 2026-08-18 by read-only queries.

This is the room next door to the atlas — the bench where the specimens were prepared, with the jars still labelled. It is in four parts: the build, which is everything needed to make the data again; the instrument tested, which is how well the reading reads and where it cannot be trusted; the record, which is what was actually done, in what order, and on whose prompting; and the bill.

Prerequisites & environment

• Language & tooling: Python ≥ 3.12 managed with uv; all commands below are uv run python … from the project root. Tests use unittest (uv run python -m unittest discover -s tests; 69 tests, all synthetic fixtures — they never touch production data).
• Key dependencies: openreview-py ≥ 2.4.1 (collection), openai ≥ 2.53 (OpenAI-compatible clients for DeepSeek/DashScope), jsonschema, and for the taxonomy stage sentence-transformers, umap-learn, hdbscan.
• Credentials (environment variables; never printed or committed): OPENREVIEW_USERNAME/OPENREVIEW_PASSWORD for collection; DEEPSEEK_API_KEY for Layer II; DASHSCOPE_API_KEY for Layer III (in production resolved once per run from 1Password CLI by a supervisor wrapper and exported to the child process).
• Directory contract: data/raw/ is written once by collection and thereafter opened read-only by everything else (verified by grep over all scripts: only the normalizer references it, with mode=ro). Every analysis run lives in its own directory under data/analysis/iclr/ with a manifest.json, a state.sqlite3 request ledger, and per-request outputs/, provider/, validations/ evidence files.

Collection — OpenReview → raw SQLite

Script: scripts/collect_iclr_forums.py (380 lines). One command per year:

# default --years covers all nine (2018..2026); resumable per year
uv run python scripts/collect_iclr_forums.py --years 2018 2019 2020 2021 2022 2023 2024 2025 2026

• Scope decision: ICLR only, 2018–2026, OpenReview as the single source. ICLR was chosen because it publishes reviews of rejected papers; ICML/NeurIPS expose mostly accepted-paper reviews. 2017 was deliberately excluded (not verifiable as an official OpenReview venue). The year range was fixed at the very start of the project; only extraction methods evolved later.
• API version handling: runtime detection, not a hardcoded table. venue_config() first queries OpenReview API v2 for the ICLR.cc/{year}/Conference group; if content.submission_id exists it stays on v2, otherwise falls back to v1 ({venue_id}/-/Blind_Submission). Empirically: 2018–2023 resolved to v1, 2024–2026 to v2.
• Pagination & resilience: cursor pagination (after=<last note id>, sort="id", page size 100) with details="replies" to fetch the reply graph in the same call; up to 7 retries with exponential backoff (1s→30s cap); a RuntimeError guard if a page makes no progress. Resumable via the after_id cursor persisted per year in collection_status; forums upserts on (year, forum_id) so reruns cannot duplicate.
• No anonymization: notes are serialized as-is (.to_json()), signatures included. The data is public; no PII masking exists anywhere in the pipeline.
• Output: data/raw/iclr/openreview.sqlite3 — forums 52,460 rows (2018:935 · 2019:1,419 · 2020:2,213 · 2021:2,594 · 2022:2,617 · 2023:3,792 · 2024:7,404 · 2025:11,672 · 2026:19,814); collection_status 9 rows, all completed=1 with stored == expected.

Normalization — provenance only

Script: scripts/normalize_iclr.py (334 lines). Deterministic and idempotent — per year it deletes and rebuilds inside one transaction; no wall-clock timestamps or randomness (times come from the raw cdate/mdate).

uv run python scripts/normalize_iclr.py

• What it adds — provenance only, no interpretation: classify_kind() strips the API-v2 "edit" wrapper then matches an ordered pattern list (official_review → meta_review → desk_rejection → withdrawal → decision → official_comment → public_comment → comment → unknown); classify_role() reads signature strings (/authors, /reviewer_, /area_chair_, /program_chair, public ~ profiles) with kind-based fallback.
• Output: data/processed/iclr/analysis.sqlite3 — papers 52,460 · messages 792,703. Kind breakdown: official_comment 507,543 · official_review 199,031 · decision 40,821 · meta_review 30,108 · withdrawal 8,235 · public_comment 5,563 · desk_rejection 980 · comment 422 · unknown 0.

Layer II — DeepSeek analytic memos

Engine: scripts/run_deepseek_pilot.py, model deepseek-v4-flash, thinking disabled, temperature=0.4, no seed. A persistent SQLite budget ledger (budget_state, api_call_attempts) with an fcntl lock keeps spend consistent across interruptions; every run declares a cost cap up front.

Two production runs share this engine:

• Prompt contract (canonical text in docs/prompts.md; the rendered prompt of every job is stored verbatim in each DB's jobs table): frame the task as qualitative metascience; never re-review the paper or invent taxonomies; cite evidence by line ID (R-<note_id>:L###); at the initial stage, deliberately hide outcomes and the other reviews.
• Leakage control: decisions, scores, and other outcome metadata are never given to any extraction stage; they are joined only at display time, downstream.
• Method choice: a blind same-model judge preferred Layered over Direct 69–31 in the pilot, but Direct was retained for the nine-year track on cost and error-propagation grounds; Layered feeds the high-resolution 2026 track. This trade-off is recorded, not hidden.

Layer III — Qwen structured units

Model (both tracks): qwen3.7-flash via DashScope's OpenAI-compatible endpoint https://dashscope-intl.aliyuncs.com/compatible-mode/v1, realtime API, enable_thinking=false, strict JSON-schema output. No temperature or seed is set — provider defaults apply (a documented reproducibility gap; see §08). Each request reads only the Layer II memo — never the raw review. Manifests pin protocol_sha256 and schema hashes.

Track A — Compact (ICLR 2026, review-level)

# input: episode-lite-2026-full-single (one review per shard, 75,859 shards)
uv run python scripts/qwen_review_logic_batch.py prepare --billing-mode realtime \
  --output data/analysis/iclr/review-logic-qwen-2026-full …
uv run python scripts/qwen_review_logic_batch.py run-realtime \
  --output data/analysis/iclr/review-logic-qwen-2026-full

Schema review-logic-compact-v0.1.json: per review, logic_units[] of {inspected_object, observation, reasoning, judgment, valence, suggested_improvement, evidence_refs, support_status, confidence, missing_links} plus a summary and unresolved tensions. Result: 74,380 / 75,859 complete (98.1%), 1,479 failed, $28.79 against a declared $40 cap. The prompt was frozen at pilot v3 after ~10 gated pilot/control runs (batch-API attempts failed with model_not_found; realtime succeeded).

Track B — Direct (2018–2026, forum-level)

# prepare declares model, schema hashes and a required --cost-cap-usd; run executes with resume
uv run python scripts/qwen_reviewer_logic_direct.py prepare \
  --output data/analysis/iclr/reviewer-logic-direct-qwen-full-v1 --cost-cap-usd … 
uv run python scripts/qwen_reviewer_logic_direct.py run \
  --output data/analysis/iclr/reviewer-logic-direct-qwen-full-v1
# subcommands: prepare · run · status · reprocess (local re-validation, no API spend)

Schema reviewer-logic-direct-v0.1.json: per forum, reviewer_records[] each with logic units carrying temporal_position (initial / post_author_response / meta_assessment / cross_reviewer), update_trigger, and judgment_change (strengthened / weakened / reversed / clarified / not_observed) — the fields behind the rebuttal and drift analyses. Validation is local and strict: JSON-schema plus evidence-reference checking against the memo's line IDs; a response can be HTTP-200 and still be recorded as failed.

Retries, merging, and the failure tail

The Direct track's first pass left truncation failures. Two retry rounds re-ran only rows with no parseable provider JSON, at raised output budgets:

# each round: prepare → run → dry-run merge → apply merge
uv run python scripts/prepare_qwen_direct_retry.py …
uv run python -m scripts.merge_qwen_direct_retry \
  --source-run data/analysis/iclr/reviewer-logic-direct-qwen-full-v1 \
  --retry-run  data/analysis/iclr/reviewer-logic-direct-qwen-retry-32k-v1 [--apply]

The merge tool verifies the retry's ancestry chain through nested manifests (up to 32 levels), refuses to merge unfinished runs, snapshots the target DB via SQLite's online-backup API before applying (state-pre-retry-merge-<utc>.sqlite3), and never overwrites first-attempt provider evidence. Semantic-validation failures (HTTP-200 rows) are deliberately excluded from API retries — enforced by test.

Final coverage (frozen 2026-08-18): strict-complete 50,861 / 51,813 = 98.16%. A weaker "JSON-available" definition (99.73%) exists but is used only with explicit flags. The 952-row tail, exactly classified:

Unit extraction & the induced taxonomy

The final data layer flattens both tracks into per-unit SQLite tables and assigns every unit to an induced taxonomy. Everything here is local compute — zero API cost — and fully scripted:

# 1) flatten complete records into unit tables
uv run python scripts/extract_logic_units_2026.py     # → unit-taxonomy-2026-v1/units.sqlite3 · 410,586 units
uv run python scripts/extract_logic_units_direct.py   # → unit-taxonomy-direct-v1/units.sqlite3 · 1,009,592 units

# 2) induce the taxonomy from a 12,000-unit sample (seed 7)
uv run python scripts/induce_unit_taxonomy.py --sample-size 12000 --min-cluster-size 60 --seed 7

# 3) hand-write taxonomy-v1.json from the cluster digests, then assign all units
uv run python scripts/assign_unit_taxonomy.py          # 2026 track
uv run python scripts/assign_unit_taxonomy_direct.py   # direct track, reusing the same centroids

• Selection rule: only status='complete' rows of each source run are extracted (outputs also exist for some failed rows; those are excluded).
• Embedding: BAAI/bge-small-en-v1.5, normalized, batch 256, local. Fields inspected_object and reasoning embedded separately.
• Clustering: UMAP (15 components, 30 neighbors, min_dist 0, cosine, random_state 7) → HDBSCAN (min_cluster_size 60, min_samples 10, EOM). Yield: 36 object clusters (37.6% noise) and 17 reasoning clusters (61.5% noise).
• Human step: cluster exemplars were read and merged by hand into 12 objects × 12 standards, with definitions, recorded in taxonomy-v1.json. Two heterogeneous clusters were excluded from centroid construction. No prior scheme (including the project's own Atlas cards) was used as seed.
• Assignment: category centroid = normalized mean of member-cluster sample embeddings (the seeded pipeline re-run reproduces cluster membership exactly — verified); every unit gets its nearest centroid by cosine, with the similarity stored as per-unit confidence. High-confidence (≥0.75) share: objects 94.5%, standards 87.9% (2026 track); per-year mean similarity on the direct track is flat at ~0.81, so cross-year comparisons are not a transfer artifact.
• Caveat: the taxonomy was induced on 2026 compact memos and transferred to the direct track; identical centroids, different memo style. The flat similarity profile is the evidence this transfer holds.
• Scheme validity: the four-part unit grain and the 12 × 12 taxonomy are one induced carving of the review text — internally consistent by the checks above, and externally anchored once (the construct-validity grid in §07: the categories dent the venue's own sub-scores where they should) — but no check establishes them as the right carving. A different grain or a different merge would draw a different map; every downstream count is a count within this scheme.

Everything downstream of these two labeled databases — aggregation files and the atlas's plates — is a separate, continuously evolving layer, documented in the atlas appendix itself.

Instrument checks — reading the reader

Everything above describes how the data was made; this section asks how far the machine reading can be trusted. Two checks, both computed from the shipped unit databases. They are also shown in the atlas, but they belong here: they are statements about the instrument, not findings about reviewers.

Support status — where the instrument speaks with its own voice

Every unit carries a support_status field assigned at structuring time: reviewer_explicit (grounded in the reviewer's stated words), memo_inferred (articulated by the DeepSeek memo from context rather than quoted), or mixed. The two tracks differ sharply by construction — the review-level 2026 track runs 19.8% explicit / 77.1% inferred / 3.1% mixed, while the forum-level track runs 57.4% / 40.8% / 1.7% — because the Layered prompts ask the memo to synthesize, and the Direct prompts demand line-ID citations. The share is not uniform across categories or years:

Categories and years where the violet deepens are where downstream figures speak more with the instrument's voice than the reviewer's. Rebuild: uv run python scripts/build_panel_data.py → panel-data.json (key reliability).

Construct validity — do the induced categories track the venue's own decomposition?

The taxonomy was induced from embeddings, not designed, so it needs an external criterion. ICLR asks reviewers for three sub-scores (soundness · presentation · contribution, 1–4; the fields exist only since 2024). For each object of scrutiny we compare mean sub-scores of reviews that carry at least one negative unit on that object against reviews at the same overall rating without one, weight the per-rating deltas by prevalence (cells with fewer than 30 reviews on either side are dropped), and read whether each criticism dents the sub-score it should. Computed over all 74,380 rated 2026 reviews.

The grid largely agrees with the venue's own decomposition — clarity criticism dents presentation specifically (−0.35), novelty dents contribution (−0.12) — with one telling leak: theory criticism dents presentation more than soundness, as if "I could not follow the theory" were filed under presentation. Rebuild: uv run python scripts/build_construct_data.py → construct-data.json.

The moving ruler — nine years of the rating scale

The venue's own instrument did not hold still. Enumerating every rating string in the record (all 199,031 official reviews) gives four distinct scoring forms, and any figure that compares scores across years has to reckon with them:

The elements audit — three blind readers

Plate X decomposes "not novel" into elements by clustering (data-driven terrain) and analyst merging (hand-drawn borders). To audit the borders, three independent machine readers (Claude Haiku agents) were each given 60 raw novelty denials and no taxonomy, and asked what each denial actually says. All three independently re-derived the plate's main families — the obvious combination, the insufficient increment, undifferentiated prior art, anticipation, restating the known, mere transfer, the discounted non-method contribution, and the venue-indexed bar — and one reader independently reproduced the decision to keep the imported evidence rule as its own element. Two elements the clustering had folded away surfaced in the audit and are recorded here as finer clauses: the false-priority accusation (the paper claims to be first, and the reviewer says the claim is untrue — an accusation of attribution, not of similarity; ≈3–7% of audited samples, folded inside "restating the known"), and the unlocated contribution (the reviewer cannot find what the paper claims is new; ≈5–10%, all three readers surfaced it). The second is best read the way the imported evidence rule is read: the charge is filed in novelty's docket but the operative law is presentation's — with one substantive twist that keeps it from being a mere clarity issue: reviewers treat novelty that cannot be located as novelty that does not exist, which is to say the venue's operative novelty is a property of the artifact plus its argument, never the artifact alone — consistent with the differentiation rule's burden-of-articulation clause, and with the fact that the dominant remedy for novelty is work of writing. Audit protocol and transcripts ship with the repository.

How the atlas handles this: score distributions are always drawn per year on that year's own levels (Figs. 11d, 11e) and the scale changes are named in the captions; the score-spread claim (Fig. 10c) uses entropy normalized to each year's own level count, so a change of form cannot masquerade as a change of behavior; the intraclass correlation (Fig. 10a) is a variance ratio and so survives relabeling, but coarser grids mechanically deflate it — the 2026 point is dashed and flagged for exactly this reason. Raw-point quantities (means, reviewer-pair gaps) are never compared across form boundaries. Sub-scores (soundness · presentation · contribution) exist only from 2024, which is why the construct-validity check above is confined to recent years.

What cannot be reproduced

• LLM sampling: DeepSeek runs at temperature 0.4 without a seed; Qwen sets neither temperature nor seed. Re-running regenerates structurally similar but textually different memos and units. Every seed in the project governs input selection, not model output.
• One-off manual repairs: two hardcoded fix scripts (28 episode IDs; 7 chain templates) patched earlier pilot artifacts and would need conscious re-application after any from-scratch rerun.
• Human judgment points: prompt-freeze timing, calibration-config adoption, and stop decisions are documented but not mechanically reproducible.
• Upstream drift: OpenReview does not expose full edit history; the raw snapshot cannot detect post-collection changes on the platform.
• Hash coverage: protocol/schema SHA-256 pinning covers the Qwen-era runs (16 manifests); older pilot artifacts rely on documentary cross-reference only.

The actual path — pilots, failures, decisions

The raw, sanitized session history behind this chronicle — every command and decision as it happened, 21,386 events — is published in the project repository (codex-history/), alongside the full pipeline code and these pages. The sections above describe the pipeline as it finally ran. This section records how it actually got there — the small trials, dead ends, and mid-course corrections, in order. Nothing here is retrospective varnish; each item is backed by an artifact directory or manifest still on disk.

Every production stage was piloted first

• Nothing ran at full scale untried. Pilot ledgers (all read-only verifiable): initial_blind 395 pilot jobs ($0.33), trajectory 327 ($0.51), paper_synthesis 100 ($0.52), forum_direct 100 ($0.32), and a dedicated 100-forum method-comparison run ($0.40).
• The method fork: Layered (3 DeepSeek calls per review chain) vs Direct (whole forum in one call) were compared head-to-head on 100 forums, judged blind by the same model — Layered won 69–31. Direct was chosen anyway for the nine-year corpus, on cost and because layered errors propagate downstream. The quality trade was accepted knowingly and recorded, not hidden; Layered still powers the high-resolution 2026 track.

The episode detour (a productive failure)

• An intermediate "evaluation episode" schema (v0.1, then a leaner v0.2 Lite/Deep) was designed and validated on curated sets: 1,000 stratified reviews (selection seed 20260816) plus 63 deep extractions — both kept as calibration assets.
• Scaling it to all 75,859 reviews failed three ways in sequence: a first full run was rejected for shallow extraction; a v2 was rejected for templated, copy-paste fields; three calibration configs were then run head-to-head (r5-low, r15-medium, r8-medium-v3) and r8-medium-v3 adopted — but the projected wall-clock for the full corpus was roughly a week, and the run was deliberately stopped mid-flight (~9,774 shards planned).
• The agentic Luna engine itself was then retired for this job: it had completed 876 of 75,859 with 186 failures when it was stopped "safely" and the whole extraction was re-designed around a lighter compact schema on Qwen. The 876 Luna outputs were kept as a comparison baseline, not merged.

The Qwen gauntlet (compact track)

• Batch API first: 0 of 27 pilot requests succeeded (model_not_found on the batch endpoint). Switched to the realtime endpoint: 22/27. Two single-review control runs followed (46/54, then 53/54), a two-reviews-per-request packing was tried and rejected for coverage loss, and a 206-request load test (203 ok) served as the final go/no-go gate before the 75,859-request production run.

Direct-track fits and starts

• Pilot v1: 0/90 — a dead end. v2 stalled with most requests stuck in prepared. v3 completed exactly one forum. v4 reached 79/90 and froze the method; only then did the 51,813-forum production run launch.

The truncation ladder

• The first full pass (16k output tokens) left 7,166 truncated responses. A retry round at 24k re-ran 7,270 eligible rows and recovered 6,223. The 959 still failing were retried at 32k, recovering 812 more. 107 rows never parsed at any budget and remain excluded.
• A separate cluster of 32 rows failed identically at 16k, 24k, and 32k with the provider's content-policy refusal (DataInspectionFailed, HTTP 400) — proof this failure mode has nothing to do with token budgets. They were not retried further; only successes were merged, each merge preceded by an automatic online-backup snapshot of the target database.
• Semantic-validation failures (813 rows with valid JSON that failed local evidence/schema checks) were deliberately excluded from every retry round — re-billing the API cannot fix a local validation rule; that exclusion is enforced by a unit test.

Small scars, recorded

• The 32k retry's supervisor shell script crashed after all 959 rows were committed — it used status, a reserved zsh variable, as a local. Renamed to exit_status; zero data impact, but the lesson is written down.
• Two one-off manual repairs patched earlier pilot artifacts (28 malformed episode IDs migrated via a hardcoded table; 7 placeholder chain-templates restored by hand). Both are irreproducible-by-construction and flagged in §08.
• The taxonomy stage had its own iteration: reasoning-side clustering came back with 61.5% noise (vs 37.6% for objects), so the standard taxonomy was accepted as a skeleton induced from the clusterable ~40% — reported as a limitation rather than silently smoothed over.

The order of the plates is not the order of the work

A page like this can read as though every analysis had been foreseen and each plate had waited for the one before it. It did not happen that way, and saying otherwise would be a research practice worth naming: inventing the motivation after seeing the result. So, plainly:

• The sequence is editorial, and it was imposed after the fact. The analyses were built over weeks in the order questions occurred to us; the plates were then grouped into acts and ordered for reading. That arrangement has now been revised three times — from a loose sequence into six acts; split into seven when the act on judgment was carrying two different questions; and regrouped again so that each act opens with the question its plates share (the current form). None of these rearrangements changed any analysis. No plate's position is evidence that it was done for the plate before it.
• Some plates began as a stated question, before any number existed. The nine plates added in the seven-act revision — the syntax of reasoning, the early verdict, the combination clause, the unnamed precedent, the law's price, the charge sheet, the moves, the borrowed verdict, the unamended code — were each written down as a question first, and computed afterwards. Two of them came back as nulls and are reported as nulls (the rebuttal moves; the interrogative). One candidate question was dropped before running, as too obviously answerable in advance: whether reviewers had grown a clause about prompt engineering in the LLM years. The timelessness test that occupies that plate replaced it.
• Some plates are frankly exploratory. The script, the ladder and the oracle exist because the data supported the description, not because a hypothesis demanded them. Their ledes say so rather than manufacturing a motive.
• One published claim was wrong and was corrected in place. The early-verdict plate first stated that a review's verdict was readable by its sixth unit "against a median review length of eleven units". The median is five, which makes the original claim close to circular. The analysis was re-run restricted to reviews of ten units or more — where the question is real — and the plate now reports that curve, which saturates at eight or nine. The error and the fix are recorded here rather than quietly overwritten.
• A second claim was corrected the day it was published. The Watermark’s first posted reading (2026-08-27, morning) presented the rise in co-reviewers’ object-set overlap after 2023 as content-level convergence, noting its turn “lands on the watermark’s arrival year.” A size-matched null run the same day — same forums, same set sizes, contents drawn at random from the year’s mix — absorbed the rise: reviewers simply charge more objects per review in the LLM years (mean set size 3.4 → 3.8), and the excess of choice over size is flat (+0.037–0.054, maximum in 2018). Fig. 29c now draws the null; Fig. 29d reports the two further structural nulls (mix entropy, reasoning-embedding gap) run at the same time. The uncorrected reading survived on the live page for roughly half a day.
• A third claim fell to the same kind of null, hours later. The corrected plate’s Fig. 29d stated that an individual reviewer’s own attention entropy “rises 0.742 → 0.766: the individual reviewer looks at slightly more, not fewer, kinds of things.” An adversarial audit run the same evening — looking deliberately for siblings of the set-size artifact — found this rise mostly mechanical too: an n-matched null (same reviewers’ unit counts, objects drawn at random from the year’s own mix) rises almost in step, because reviewers file more units per review in the LLM years and the year’s mix itself grew more even. Only +0.008 of the +0.024 rise survives the null, with no trend. Fig. 29d now draws that null and claims only what survives: no narrowing after 2023. The same audit sharpened, rather than weakened, the wording-convergence panel: split by section, the within-paper rise concentrates in the summaries (+20%, still climbing) while the criticism prose rises +6% and stalls after 2025 — both decompositions now on the plate. The affirmative “spreads more” reading survived on the live page for under a day.
• Where a caption gives a reason for an analysis, it is the reason the analysis was run — a question we had — or it is marked as what the result turned out to bear on. Reasons of the second kind are written as consequences ("what this bears on"), never as motives.

Where each plate came from

The table below records, for every plate, the question it answers and what we can honestly say about when that question was formed. It exists so that a reader can tell the difference between a question we asked and then answered, and a description we produced and then found interesting. "Not recorded" is a real answer and appears where it appears honestly. Where a record exists, it was recovered from the project's own working history — the full conversation log between the researcher and the assistant, timestamped — not from memory. The log is kept in Japanese; the excerpts quoted below are translated. That log will stay private (decided 2026-08-27): the code-side session history is published in codex-history/, but the conversation log contains material that cannot be responsibly scrubbed at full length, so the dated excerpts quoted in this table are the parts of it that appear. Reconstructing a motive with no record would be exactly the practice this table is meant to prevent.

• stated first — the question was written down before the numbers existed
• foundational — built together with the taxonomy itself; no separate hypothesis
• exploratory — run because the data allowed the description; nothing predicted
• part stated — partly specified in advance, partly added afterwards
• not recorded — we do not have a reliable record of what prompted it, and are not going to invent one

The tally over all 34 ledger rows, before the row-by-row record — a one-glance answer to "how honest is this project about its own process":

Notes for the added plates

Nine plates were added in the seven-act revision (the Syntax, the Early Verdict, the Combination Clause, the Unnamed Precedent, the Law’s Price, the Charge Sheet, the Moves, the Borrowed Verdict, the Unamended Code), plus the Door, which is navigation and makes no claims. (In the 2026-08-24 restructure the Law's Price merged into the Price plate, the Gauntlet into the Consequence, and the Script into the Itinerary as its prologue; their instruments are unchanged.) One later addition, the Watermark (2026-08-27), closes the list. Their instruments, and where each can mislead:

• The Syntax — form labels come from the Plate III classifier (5-fold CV accuracy 0.685, macro-F1 in the file); transitions are compared against a null that shuffles each review internally, so the reported lifts are within-review order effects only, and they are small — the plate says so rather than dressing them up.
• The Early Verdict — "readable by unit six" is a statement about a classifier over coarse features (object × valence counts), not about when the reviewer decided; reviewers may draft conclusions first. Target is the 2026 even scale at 6+, 3-fold CV, seed fixed. The 2026-08-21 follow-ups share that target and CV: the raw-text bar is TF-IDF (1–2-grams, min_df 5) over the concatenated review fields; the tally weights are L2-logistic coefficients on whole-review per-topic negative counts (plus total positive and hedged counts) — descriptive coefficients on correlated counts, reported with their fold spread.
• The Combination Clause — sub-clusters are k-means (k=8, seed 46) over bge-small embeddings; names were drafted by a machine reader over exemplars and reviewed by the analyst; four near-duplicate referent clusters were merged into one generic charge and the merge is recorded in the builder. Raw clusters ship with the dataset. The referents are clustered on the units' distilled observations, so the headline share (46% name no parts) was audited against source reviews: in a 30-case sample of no-name observations, wherever the source charge sentence could be located its specificity matched the observation's — generic in the reviewer's own words whenever the observation was generic, with one ambiguous exception where a descriptive sentence named parts the charging sentence did not. The audit transcript ships with the repository (notes/combo-audit-30.txt).
• The Unnamed Precedent — the extractor is a citation regex plus arXiv IDs, with venue acronyms excluded; prose references ("the BERT paper") are missed, so all lit shares are floors — the conclusion (no canon) survives any plausible extractor. The (surname, year) key deliberately over-merges, which only strengthens the null. The measurement was run twice, and both runs are reported (design of the second declared before its results): first on the distilled unit text (5 / 29 / 66; naming 8.4%→3.5% by year; max key 10 forums), then re-measured on the raw review text with the unit serving only to locate the charge — every sentence mentioning novelty ±2 sentences, in the best-token-overlap review of the unit's forum (window found for 79% of units; the window is deliberately generous to naming, since it can absorb a neighbouring criticism's citation). The raw reading gives 12 / 25 / 63, naming 20%→11% by year, max key 21 of 23,251 forums. The naming level is instrument-sensitive (floor 5%, generous window 12%); the two headline facts are not: about two thirds of objections carry no referent under either instrument, the naming rate halves 2018→2026 under both, and no canon appears under either. Builder: build_canon_native.py.
• The Law’s Price & the Charge Sheet (today Plate XIV · The Price and Plate XIII · The Charge Sheet, after the 2026-08-24 merge) — per-unit law assignments come from the sampled sentence clusters (40k per docket), so absolute co-filing rates are thinned; the plates therefore report only ratios (lift), which the independent-sampling design leaves unbiased to first order, and per-law mean deficits, which sampling thins but does not bias. Deficits are normalized by each year’s rating standard deviation; ratings join reviewer keys to OpenReview signatures by suffix match. The charge sheet’s within-review lifts were additionally tested against a paper-quality confound (build_chargesheet_control.py, declared before results): every displayed pair re-measured across two different reviewers of the same paper. Attractions kept a median ×1.44 across reviewers against ×1.87 within one review (part paper, part reviewer); repulsions showed no cross-reviewer suppression at all (×0.97 against ×0.75 within), placing the one-filing habit with the reviewer, not the paper. Cross-docket pairs cannot share a unit, so multi-law sentences cannot inflate either list.
• The Moves — the outcome instrument counts only judgment changes the reviewer recorded in post-response units ("weakened/reversed"); silent score raises are invisible to it. Markers are deterministic regexes, quoted in the builder. The contest’s premium is the association of a chosen move — authors contest when they hold a strong hand. A parallel embedding clustering of the same 722,406 reply sentences (k=12) separated topics more than tactics and found the same null; it ships as moves-raw.
• The Borrowed Verdict — echo tiers are cosine conventions (0.75 / 0.90) over cached reasoning embeddings; moving the thresholds moves the levels, not the comparisons. Nearest-neighbor attribution measures textual proximity, not causal influence — an AC and a reviewer may independently echo the paper. Concentration is computed only on forums with three or more echoed units.
• The Unamended Code — the timelessness claim is calibrated by a split-half control (a code fitted on half of 2026, held-out half at 0.152); every year sits within 0.008 of it. Two resolution limits are owned in the caption: sentence embeddings can miss register-preserving semantic shifts, and the transfer clause rides on the code’s one genuinely mixed cluster (majority register transfer; differentiation a defensible second reading), so its 5–7% band is the least certain of the six. The reverse fit (a 2018–19 code applied forward) uses k=6 over only 417 sentences and is reported as weak corroboration, not evidence.
• The Watermark (added 2026-08-27, after a literature survey of LLM-in-peer-review work) — corpus-level only, by design: per-review AI detectors are excluded on the record of non-native-speaker false positives (Liang et al., Patterns 2023) and instance-level unreliability (Yu et al., ICLR 2026). The instrument: a thirteen-word marker list frozen verbatim from Liang et al. (ICML 2024) and Kobak et al. (Science Advances 2025) before any counting; each year’s document frequency against its own 2018–2022 linear extrapolation (Kobak’s excess-vocabulary design); 2023 as a built-in placebo year and a frozen ten-word negative-control list run through identical machinery. Free text only — the five review-form eras’ enumerated fields are stripped, and similarity levels are never compared across a form boundary; the headline convergence claim lives inside the constant-form 2024–2026 window under one shared TF–IDF vocabulary. The correlate battery stratifies by length deciles (presence measures are length-sensitive — the control list’s below-trend drift shows the ambient tide), benchmarks the score comparison against the paper’s longest review, and permutes marked labels within papers 200× before reading any quality gradient (conditioning on the comparison group manufactures one from mean reversion alone). Plan, hypotheses and decision rules: notes/llm-era-analysis-plan.md, frozen before computation; addenda after run 1 (fixed vocabulary, length stratification, permutation control, quality strata after Sharma et al. arXiv:2601.20920) are labeled in the plan file. Prior art cited on the plate: Liang et al. 2024; Kobak et al. 2025 and Gray, arXiv:2512.01560 (the excess-vocabulary method’s ancestors — papers at large, not reviews); Latona et al. 2024; Wu et al. arXiv:2604.19578 (nearest neighbour — same venues, aspect-coverage not similarity); Sharma et al. 2026; Kim et al. arXiv:2605.20668 (AI reviewers’ 21%-vs-3% mutual overlap, by expert annotation); Baumann et al. arXiv:2605.03202 (the “hivemind effect”); and arXiv:2607.10511, which reads a 2022–23 shift in ICLR review-text diagnostics — the timing neighbour this plate’s placebo disagrees with by one cycle. A Semantic Scholar + arXiv search on 2026-08-27 found no published within-paper co-review similarity-over-years measurement on real OpenReview data; that is recorded as a search result, not a proof of novelty. Builders: build_llmtrace_data.py (raw text), build_llmtrace_mix.py (addendum E+F: category-mix entropy, size-matched overlap null, reasoning-embedding dispersion — the instruments behind Fig. 29d and the same-day correction of Fig. 29c recorded in § 10).

Instrument extensions (2026-08-24 review pass)

During the plate-by-plate review, several caption claims turned out to live only in a builder’s one-off print output — verifiable at the time, unverifiable once the run scrolled away. The rule adopted: any number in a caption or stat rail must either sit in the shipped JSON or be recomputable by a named script. The following were persisted accordingly:

• build_acecho_data.py — now emits gap_by_cutoff: per-valence shares of meta-review units at nearest-neighbor cosine ≥ 0.70 / 0.75 / 0.80, backing the Borrowed Verdict’s claim that criticism is echoed more than praise at every cutoff (+7.0 / +6.6 / +3.3 pp).
• build_score_depth.py — now emits dispersion.accept_unan_div: acceptance for unanimous vs divided panels at exact means 4.0 and 6.0 (the banded version confounds range with mean inside the band and is retained only as the naive contrast). Backs the Measurement’s “the decision is a mean, not a variance” (92.1% vs 91.8% at mean 6.0; 12.1% vs 13.1% at 4.0).
• build_timeless_data.py — now emits dist_se: per-year sentence-bootstrap SEs of the mean distance to the 2026 code (2018: 0.0034 … 2026: 0.0006), so the Unamended Code’s null is stated with its resolution: the early years sit measurably (+2–3 SE) but minutely (+0.006) above calibration; sentences cluster within reviews, so even those SEs are optimistic.
• scripts/verify_drift_ci.py (new) — reviewer-clustered bootstrap for the Drift’s four named movers (clarity, theory, compute cost, statistical rigor): all four sit at 8–16× their clustered standard errors (2,721 vs 73,427 reviewer clusters, 1,000 draws).
• scripts/verify_remedy_quartiles.py (new) — recomputes the Elements’ length-quartile robustness check from units.sqlite3: novelty offers a fix for 60–61% of its denials in every quartile of unit length, against 76–94% for the other eleven objects, and novelty units are slightly longer than the pooled rest (221.5 vs 203.0 chars) — terseness ruled out. Supersedes the earlier 63–66%/80–92% figures, which predated a pipeline rebuild.
• scripts/build_mirage_data.py (new) — permanent rebuild of the multinomial null that deleted the Kinds plate (2026-08-21), now feeding Appendix III’s centerpiece figure: real reviewer roses vs roses clustered from synthetic profiles drawn i.i.d. from one shared field mix (seed 46, matched unit counts). Reproduces the recorded numbers exactly: k-means explained variance 35.8% real vs 31.0% null; per-standard overdispersion median 1.15, max 1.58.
• build_moves_markers.py — now emits delta_by_quartile_all and n_by_quartile, so the Moves’ effort-stratified premiums disclose the sub-threshold stratum (contest Q1, +1.43pp, drawn faint) instead of silently dropping it.
• build_overrule_data.py — now emits margins: the mercy asymmetry of the 2026 bench recomputed at overrule margins ¼–1 pt (×2.4 / ×2.7 / ×3.9 / ×3.9), backing the Panel’s “survives any margin” claim.

Naming provenance for the added plates: the novelty direct-track clause map and the combination sub-cluster names follow the same workflow as the eleven-docket elements — machine-drafted over exemplars, analyst-reviewed (one contested cluster, the transfer/differentiation mixture, is flagged in both the map file and the caption). All raw cluster exports, per-unit assignments, and the builders that made them ship with the repository and the HF dataset.

The long rule and the cast shadow — instruments, not plates

Two persistent graphics recur across the atlas and are neither plates nor ledger entries. They are editorial apparatus, and the honest thing is to say so here rather than let a reader mistake them for figures: neither claims anything about the corpus. Each has exactly one real dependency on the data, and it is named below.

The long rule, a folding rule in the right margin, is built from the document itself: its hinges mark the act boundaries, its segment lengths are proportional to each act's plate count — counted at runtime from the page, not hand-tuned — and its graduations are one per plate, each one clickable. What is engraved along it is a vocabulary of magnifications (SENTENCE · REVIEW · CLAIM · COST · ROOM · COURT · RATING · YEARS · RECORD), one word per act, naming the size of the object that act holds under the lens. That vocabulary is editorial: it is the authors' own description of the book's structure, not a measurement of anything, and it should be read the way a table of contents is read, not the way a plate is read.

The cast shadow is a shadow thrown across the page when a heading arrives, inside which real sentences from the corpus develop and fade. It is decorative. The sentences are genuine corpus text — that much is a real data dependency — but nothing is claimed by which ones appear; no frequency, ranking or representativeness should be read into the selection. The whole layer is absent under prefers-reduced-motion and in print, which is itself a statement that it carries no content the page depends on.

Cost ledger

Pilot-phase spend (episode schemas, calibrations, judge comparisons) is additional and itemized in the project's internal provenance ledger; there is deliberately no single "grand total" field — sums must be composed explicitly from the stages used.