Beyond Model Leaderboards: The Operational Supplement

This is the operational companion to Beyond Model Leaderboards. It has its own card because the checklist, frozen defaults and source map form part of the protocol itself.

• Read the main protocol paper

• Download the version 1.2 supplement as PDF

• Verify the release checksums

Article type: Protocol paper supplement
Publisher: nullhex research
Published: 23 July 2026
Version: 1.2

This supplement carries the operational lookup material for the Worker Model Tournament protocol paper. The main paper states the rationale, estimands, decision rules, design history, and threats to validity. This document preserves the adaptation checklist, reference parameters, statistical-amendment requirements, companion-artifact map, and publication source map without repeating them in the narrative.

S1. Adaptation checklist

Before using the protocol, a reproducing team should be able to answer yes to each item.

Decision

• Are the roles and deployment decision explicit?

• Is the treatment a complete configuration or workflow policy?

• Are quality, reliability, and economics separate?

• Are superiority, equivalence, no-winner, and inconclusive rules frozen?

Routes and data

• Is every requested and resolved route pinned and auditable?

• Does the least-permissive route set the common corpus tier?

• Are task archives allowlisted, scanned, inspected, and hashed?

• Are account, retention, training, telemetry, and fallback policies recorded?

Harness and sandbox

• Do candidates receive identical task bytes and equivalent capability classes?

• Has each adapter passed the same conformance suite?

• Are credentials, network, hidden tests, history, memory, and other workspaces inaccessible?

• Are sessions, workspaces, and provider conversations fresh?

Tasks and evaluation

• Does every task have a machine-readable manifest?

• Do base, oracle, revert, unaffected, flake, alternative-correct, and mutant checks pass?

• Are requirement groups weighted by behavior rather than assertion count?

• Are severe defects and integrity violations hard gates?

Review and analysis

• Are judges identity-masked, anchored, calibrated, licensed, and independently adjudicated?

• Are candidate failures retained in the intention-to-treat population?

• Does one frozen inferential_family_id bind the coarsest task or incident cluster throughout the analysis?

• Is the randomization, interval method, margin, seed, and analysis frozen?

• Are the pilot freeze and confirmatory freeze distinct, with no candidate response informing the latter?

Evidence and publication

• Are attempts journaled before dispatch and responses persisted atomically?

• Can every aggregate row trace to a task, response, patch, evaluation, and review?

• Can deterministic analysis run without provider credentials?

• Does the release contain source, HTML, manifests, schemas, checksums, licence, and citation metadata?

• Does the exact release have a persistent version identifier?

S2. Reference instantiation parameters

These values define the reference instantiation. They are decision-policy defaults, not scientific constants. An adaptation may change them only before assignment, must document the rationale, and must include the changed values in its sensitivity analysis.

1. Qualification2 tasks per configuration, 1 per role (new code and bug fixing)
2. Judge calibration and licensing40 calibration patches, disjoint from tasks and the licensing holdout; 140 licensing patches, including at least 60 critical and 60 integrity examples
3. Design pilot8 task families per role, 2 reference controls, 2 repetitions: 64 rows
4. Screening9 primary candidates on 10 tasks, plus 2 controls on 6 tasks each: at most 102 rows. Eligible to advance with a role mean of at least 70, at least 4 usable role assignments and no quarantine or confirmed critical defect
5. Sealed final16 tasks per role, 2 finalists, 2 repetitions, plus reference controls: at most 192 rows
6. Workflow confirmation8 tasks per role, at most 2 workers, 2 repetitions, 3 arms: at most 192 rows, plus up to 192 bounded repair dispatches

Six-step sequence of the reference instantiation, which has two roles, new code and bug fixing. One, qualification: 2 tasks per configuration, 1 per role. Two, judge calibration and licensing: 40 calibration patches, disjoint from tasks and the licensing holdout, and 140 licensing patches including at least 60 critical and 60 integrity examples. Three, design pilot: 8 task families per role, 2 reference controls and 2 repetitions, 64 rows. Four, screening: 9 primary candidates on 10 tasks plus 2 controls on 6 tasks each, at most 102 rows; a candidate is eligible to advance with a role mean of at least 70, at least 4 usable role assignments and no quarantine or confirmed critical defect. Five, sealed final: 16 tasks per role, 2 finalists and 2 repetitions plus reference controls, at most 192 rows. Six, workflow confirmation: 8 tasks per role, at most 2 workers, 2 repetitions and 3 arms, at most 192 rows, plus up to 192 bounded repair dispatches.

S3. Minimum statistical amendment

The dated amendment is a required confirmatory-freeze artifact. It contains:

• The aims, data-generating mechanisms, estimands, candidate methods, selected method and implementation, frozen method-selection rule when applicable, and performance measures for the simulation study

• Separate design alternatives for worker superiority, workflow superiority, workflow non-inferiority, and operational equivalence

• The exact family and repetition layouts, workflow-variance basis, simulation count, Monte Carlo uncertainty method, and numerical acceptance criteria by endpoint, including the reference design and smaller-family stress scenarios

• Bounded score distributions with floor and ceiling mass, heterogeneous family effects and variances, correlated repetitions, score-cap mass, and candidate-attributable zero-score failures

• Explicit null and decision-boundary scenarios, including symmetric evaluator re-evaluation and quarantined-family scenarios, with the latter represented in both the failure-coded and family-removed sensitivity analyses

• The complete role gates, missingness rules, hard defects, reliability preference, equivalence logic, and no-winner or inconclusive outcomes

• Interval coverage and width, false superiority, false non-inferiority, false equivalence at both boundaries, power, correct-decision and correct-equivalence classification, no-winner, co-winner, hard-gate, quarantine, and inconclusive rates, with Monte Carlo uncertainty for every frozen scenario

• Threshold-sensitivity results for every adapted decision-policy value

• Executable code, dependencies, generator parameters, seeds, software versions, raw aggregate counts, machine-readable results, and a human-readable operating-characteristic report

A method authorizes a formal decision only if it passes every frozen acceptance criterion applicable to that endpoint and decision. Otherwise the relevant interval is descriptive and the affected decision is inconclusive.

S4. Companion applications and claim boundaries

The repository includes three bounded applications that informed the design history. They are separate research objects with their own estimands and do not expand the reference protocol's claim boundary.

• Lean worker race. .planning/lean-worker-race-2026-07-12.html and artifacts/pilot-2026-07-12/ document a small public-synthetic methodology pilot. Its claim boundary is finite-bank operational screening, evaluator behavior, review, and repair evidence, not formal non-inferiority or a universal model ranking.

• Matched workflow study. .planning/confirmatory-workflow-study-results-2026-07-13.html and artifacts/confirmatory-2026-07-13/ document three complete workflow policies under a frozen route and acceptance contract. Its treatment is the complete workflow, not a model or isolated workflow component.

• Matched follow-up. .planning/sol-solo-followup-results-2026-07-13.html and its ledger-bound JSON document a direct-versus-delegated historical comparison. Its claim boundary is a descriptive matched extension, not an isolated causal effect or a general claim about delegation.

Each application retains its own plan, evidence, failure lineage, analysis, and reporting language. The protocol paper cites them as design-history and executability evidence without pooling their outcomes.

	Supports	Does not support
Lean worker race	Finite-bank operational screening, evaluator behaviour, review and repair evidence from a small public-synthetic methodology pilot	Formal non-inferiority or a universal model ranking
Matched workflow study	The complete workflow as the treatment, across three workflow policies under a frozen route and acceptance contract	Claims about a model or an isolated workflow component
Matched follow-up	A descriptive matched extension of a direct-versus-delegated historical comparison	An isolated causal effect or a general claim about delegation

Grid of the three companion applications against what each supports and does not support. Lean worker race, a small public-synthetic methodology pilot: supports finite-bank operational screening, evaluator behaviour, review and repair evidence; does not support formal non-inferiority or a universal model ranking. Matched workflow study: supports the complete workflow as the treatment, across three workflow policies under a frozen route and acceptance contract; does not support claims about a model or an isolated workflow component. Matched follow-up: supports a descriptive matched extension of a direct-versus-delegated historical comparison; does not support an isolated causal effect or a general claim about delegation.

S5. Publication and source map

The private implementation repository and supporting evidence are available upon request from nullhex, subject to exclusions for credentials, customer data, private source, sensitive transcripts, and operational identifiers. The publication lineage includes:

• Comprehensive protocol specification: .planning/worker-model-tournament-2026-07-11.md

• Pragmatic incumbent-versus-challenger comparator: .planning/craft/run-pragmatic-worker-comparator-2026-07-12.md

• Supplemental route and recovery protocol: .planning/worker-model-comparator-v2-2026-07-12.md

• Machine-readable plans: tasks/stage0-process.plan.json, tasks/lean-worker-race.plan.json, tasks/confirmatory-workflow-study.plan.json, and tasks/sol-solo-followup.plan.json

• Runner, ledger, artifact builder, deterministic analysis, and reproduction commands documented in README.md

A complete archival release contains the paper and supplement sources and editions, protocol and amendment history, machine-readable plans and schemas, public or licensed task packages, evaluator source and validation evidence, randomization and expected-assignment manifests, pinned runner metadata, sanitized evidence and invalidation lineage, deterministic analysis, reports, checksums, licences, citation metadata, and a precise data-availability statement.

The release artifact manifest, not the private repository tree, defines the public research object. Credentials, customer data, private source, sensitive transcripts, operational identifiers, private tasks, temporary workspaces, and operational rollout state remain outside the citable bundle. The fixed paper bundle is complete for paper reconstruction and protocol inspection. Hosted calls cannot be reproduced bit-for-bit; a live re-execution is a new dated experiment with its own route and policy manifest.

1. 2026-07-11
   Comprehensive protocol specification
2. 2026-07-12
   Pragmatic incumbent-versus-challenger comparator
3. 2026-07-12
   Supplemental route and recovery protocol
4. 2026-07-12
   Lean worker race

   small public-synthetic methodology pilot

5. 2026-07-13
   Matched workflow study

   three complete workflow policies

6. 2026-07-13
   Matched follow-up

   direct-versus-delegated historical comparison

7. 23 July 2026
   Version 1.2 published
8. specifications and companion applications

Timeline from 11 to 23 July 2026, with dates taken from file names in the source map. On 2026-07-11, the comprehensive protocol specification. On 2026-07-12, the pragmatic incumbent-versus-challenger comparator, the supplemental route and recovery protocol, and the lean worker race, a small public-synthetic methodology pilot. On 2026-07-13, the matched workflow study of three complete workflow policies and the matched follow-up, a direct-versus-delegated historical comparison. A bracket spans 11 to 13 July, labelled specifications and companion applications. On 23 July 2026, version 1.2 of the supplement is published.