Beyond Model Leaderboards

This is not a story from the workshop. Today we are publishing the paper that came out of it.

The narrative account records what happened during the tournament. This paper sets out the reusable method, the decisions that shaped it, and the limits on what its evidence can support. Its operational supplement carries the adaptation checklist and reference parameters.

• Download the version 1.2 paper as PDF

• Read the operational supplement

• Verify the release checksums

Article type: Protocol paper
Publisher: nullhex research
Published: 23 July 2026
Version: 1.2

Abstract

Model comparison is often reduced to a model name and an aggregate score. For coding agents, that is the wrong experimental unit. Deployable behavior is jointly determined by the requested and resolved model, reasoning setting, provider route, harness, tool interface, context policy, permissions, resource limits, and surrounding workflow. A useful evaluation must therefore compare complete worker configurations while controlling the information and capabilities available to them.

This protocol paper specifies the Worker Model Tournament, a quality-first method for selecting coding-agent workers and workflow policies under realistic operational constraints. Its research object is the evaluation method, including its design rationale, decision rules, operational controls, and reproducibility contract. The protocol separates new-code and bug-fix roles, uses qualification, screening, sealed comparison, and matched workflow stages, and combines hidden mechanical evaluation with calibrated identity-masked semantic review. Candidate-attributable failures remain in the intention-to-treat population. Absolute safety and integrity gates prevent average performance from concealing severe defects. Paired, task-family-aware decision rules permit superiority, equivalence, co-winners, no winner, and inconclusive outcomes instead of forcing a leaderboard.

The design history is as important as the final specification. The protocol evolved from a comprehensive multi-route tournament into a family of bounded decision instruments with machine-readable plans, interruption-safe accounting, independent review, explicit retry semantics, and hash-bound evidence. The central contribution is not another benchmark score. It is an auditable method for turning coding-agent evaluation into a defensible deployment decision that another team can inspect, adapt, and reproduce.

Keywords: coding agents, software engineering agents, protocol paper, benchmark design, model evaluation, reproducibility, LLM-as-a-judge, experimental protocol

1. Introduction

Coding-agent benchmarks answer an important question: can a system resolve a defined software task? Deployment decisions ask a broader one: which configuration should be trusted to perform a particular engineering role inside a particular operating environment?

Those questions diverge because a model never reaches a repository by itself. A provider resolves a route. A harness serializes context and exposes tools. An adapter handles tool calls and errors. A sandbox grants or denies capabilities. A context policy determines what the model sees and what happens near its limit. A workflow may add planning, review, repair, or escalation. Each layer can change quality, reliability, latency, cost, security, and failure behavior.

The Worker Model Tournament begins with one claim:

The unit under test is the deployable worker configuration, not the model name.

This leads to a second claim:

A tournament is useful only when its evaluation contract is at least as rigorous as the worker contract.

The protocol therefore binds the candidate identity, task bytes, tool capabilities, hidden evaluator, review policy, failure accounting, and decision rule before a scored comparison. It treats route drift, incomplete tasks, judge error, data leakage, missing usage, and ambiguous retries as threats to validity rather than incidental operations.

The paper makes six contributions:

• It defines a deployable configuration as the experimental treatment and separates role-specific selection from universal model ranking.

• It gives a staged, quality-first protocol that separates harness qualification, candidate screening, sealed capability estimation, and complete-workflow estimation.

• It treats task construction and hidden evaluation as software that must itself pass oracle, revert, flake, isolation, and mutant checks.

• It combines mechanical outcomes and identity-masked semantic review while licensing judges as measurement instruments rather than accepting their opinions uncritically.

• It provides failure, retry, statistical, and evidence rules that resist survivor bias, fake precision, route ambiguity, and forced winners.

• It records how the protocol was simplified into bounded comparators without discarding the validity controls that made the original design defensible.

The intended reader is a team choosing coding-agent workers for real software work, or a researcher studying complete agent configurations rather than isolated language models. This article is a protocol paper: it specifies a reusable evaluation method and records the decisions that produced it. Bounded companion applications retain their own estimands and claim limits.

2. Relationship to existing coding-agent evaluation

SWE-bench established a large, repository-grounded benchmark built from real GitHub issues and their corresponding changes [1]. It helped move coding evaluation beyond isolated function synthesis. SWE-agent showed that the agent-computer interface materially affects software-engineering performance [2]. Together, these works motivate evaluating the environment around a model as part of the system.

Recent work makes the system-level framing explicit. Harness-Bench studies model-harness configurations under shared task environments while preserving native harness behavior [3]. Claw-SWE-Bench standardizes the runtime, workspace, prompt, patch, and evaluator contract for heterogeneous agent harnesses [4]. MASEval treats the complete agentic system, including topology, orchestration, context engineering, and error handling, as the unit of analysis [5]. The contribution here is therefore not the system-level treatment idea by itself. It is an integrated, audit-first selection protocol that joins route and privacy identity, evaluator validation, operational failure accounting, judge licensing, paired deployment rules, and evidence lineage.

Agentless demonstrated the value of a deliberately simple repair pipeline and documented benchmark-quality problems found through manual audit [6]. Its result is methodologically important here: added orchestration must earn its place, and benchmark artifacts cannot be assumed correct merely because they are hidden. SWE-rebench addresses freshness and contamination through a continuously renewed task construction pipeline [7].

Semantic review adds another measurement problem. CodeJudgeBench evaluates language models acting as judges across coding tasks [8]. An empirical study of LLM-as-a-judge in software engineering likewise finds that judge reliability depends on the task and evaluation setup [9]. These findings support a design in which judges are identity-masked, anchored, calibrated, licensed on a sealed holdout, and subject to human adjudication for severe or decision-relevant disagreements.

The Worker Model Tournament complements these benchmarks. It does not replace a public task corpus or claim that organization-specific tasks produce a universal ranking. Its focus is the deployment decision: multiple provider routes and harnesses, private or transformed task banks, role-specific winners, quality gates, operational failures, cost accounting, and complete author-review-repair workflows.

2.1 Methodological foundations

The protocol operationalizes established experimental principles rather than claiming them as new. The ICH E9(R1) estimand framework separates the treatment condition, target outcome, handling of intercurrent events, estimator, and sensitivity analysis [10]. The domain differs, but the discipline transfers: candidate-attributable failures remain part of the assigned configuration's treatment-policy estimand, while evaluator faults follow a separately declared rule.

Equivalence and non-inferiority require a prespecified practical margin and confidence-interval logic distinct from a superiority test [11]. Pair matching can improve precision when assignment and analysis preserve the pair [12], while few-cluster methods can miscalibrate conventional inference [13]. The statistical amendment therefore follows the simulation-study discipline of declaring aims, data-generating mechanisms, estimands, methods, and performance measures before choosing an inferential procedure [14].

Semantic judges are measurement instruments. Cohen's agreement framework [15], the prevalence sensitivity of kappa [16], and diagnostic-accuracy reporting principles [17] motivate individual licensing, defect-stratum confusion matrices, severe false-negative bounds, and explicit reference standards. The protocol does not treat one pooled correlation as sufficient evidence of validity.

The two-freeze design also draws on preregistration's separation of prediction from postdiction [18] and the protocol-first logic of Registered Reports [19], without representing this paper as a retrospective registration. Targeted incorrect mutants follow mutation-testing's use of seeded faults to test whether an evaluator detects meaningful errors [20]. The append-only evidence graph is compatible with the entity, activity, agent, and derivation concepts of W3C PROV [21], while frozen commands, versions, seeds, and intermediate artifacts follow established computational-reproducibility practice [22].

3. Research question and estimands

The primary research question is:

Which deployable worker configuration best satisfies a defined software-engineering role under a common information boundary, quality rubric, and operating envelope?

The protocol treats new-code work and bug-fix work as separate roles. A candidate may win either, both, or neither. Different role winners are valid because the skills required to create a bounded component can differ from those required to locate and repair an existing defect.

The primary worker estimand is the role-specific Stage 2 intention-to-treat mean across every assigned sealed-final task and repetition. Qualification, pilot, and screening rows do not enter that estimand. Stage 3 has separate workflow-policy estimands. Reliability and economics are separate outcome families:

• Capability includes total quality, automated behavior, semantic review, lower-tail task-family performance, historical-defect recurrence, and severity of confirmed defects.

• Reliability includes completion, first-attempt acceptability, refusals, timeouts, context failures, malformed tool calls, adapter failures, and unusable patches.

• Economics includes accepted input, cached input, visible output, reasoning usage when reported, tool arguments, total provider usage, latency, tool calls, and actual or shadow price.

Latency and cost never compensate for a quality loss. They become operational gates or tie-breakers only after candidates satisfy the quality contract.

For system-level comparisons, the treatment is the complete workflow policy. Examples include direct dispatch, orchestrator planning, independent review, one bounded repair, and escalation. A component inside a bundled policy cannot receive a causal claim unless the experiment varies that component while holding the rest of the policy fixed.

4. The treatment is a configuration

A worker configuration contains at least these bound fields:

Exact provider identities belong in a dated route manifest rather than the durable prose specification. This keeps the method reusable while preserving the identity required to interpret a particular comparison.

4.1 Fairness as capability parity

Fairness does not require every native harness to feel identical. Native ergonomics are part of a deployable configuration. Fairness does require equivalent capability classes and identical information:

• Read, list, and search the same frozen source tree.

• Receive the same task brief and context bytes.

• Edit only inside an isolated workspace.

• Run the same declared verification commands.

• Inspect status and diff.

• Access the same frozen dependency documentation.

• Receive no candidate-specific source of repository knowledge.

A conformance suite checks file bytes, search results, tool schemas, command environment, error propagation, output truncation, protected paths, network denial, context accounting, and usage metering for every adapter.

4.2 Route identity as experimental identity

The route manifest records the full path from request to resolved system: provider, host, API version, project or account class, region, publisher, model resource, model version, reasoning setting, fallback policy, context limit, price snapshot, data policy, and approval expiry. Every response records its request identifier and returned model identity when available.

An alias change, provider fallback, endpoint substitution, or unresolved identity is a validity event. The response is quarantined rather than silently pooled with the frozen configuration. A replacement route creates a new configuration and therefore a new comparison.

4.3 Native context and tokens

A shared artificial token ceiling can distort systems with different native context, reasoning, caching, and compaction behavior. The protocol instead holds time, turns, tools, information, and task bytes constant while allowing each route to use its documented provider-native request behavior. Tool results are capped consistently, and fresh sessions prevent context inheritance.

Provider usage is an observed outcome, not a quality allowance. The ledger records every authoritative token class exposed by the route. Cross-provider token totals are operational accounting measures, not evidence that one token represents the same semantic computation across model families.

5. Validity envelope

The validity envelope is the set of conditions that must remain true for two assignments to be comparable.

5.1 Common data boundary

The least-permissive approved provider determines the corpus tier for all candidates. A comparison never sends private tasks to one route and synthetic tasks to another.

Each task package is built from an allowlist, stripped of Git history and excluded paths, scanned for secrets and personal data, inspected by archive listing, and hashed. Ambiguous scanner results fail closed. Credentials, environment files, production dumps, customer content, raw transcripts, private memory, planning material, and unrelated business records are prohibited.

Provider approvals are route-specific and time-bounded. They cover training use, retention, abuse review, caching, telemetry, regional processing, third-party routing, headless automation, and benchmark volume. The scheduler rejects an expired or mismatched approval.

5.2 Process isolation

The model-facing process receives no host credential. Provider requests pass through an external broker, while tools execute under an unprivileged sandbox identity. The sandbox has an ephemeral home directory, allowlisted mounts, no host or container socket, no provider credential, no network egress, and no inherited plugins, hooks, skills, shell startup files, user settings, or memory services.

Every assignment starts from a fresh synthetic Git repository containing one base commit and no remote. The candidate cannot access hidden tests, oracle patches, other workspaces, original history, issue discussions, known fixes, or evaluation infrastructure.

5.3 Pilot and confirmatory freezes

The protocol uses two dated freezes. A pilot freeze occurs before any pilot assignment and binds the pilot controls, pilot task packages, evaluator materials, route and sandbox contracts, pilot analysis, and amendment rule. A confirmatory freeze occurs after the pilot analysis and before any Stage 1 candidate assignment. The second freeze binds every scored object and the complete decision rule. Pilot evidence may inform that confirmatory freeze, but no scored candidate response may do so.

Across the two freezes, the following objects are frozen and hashed at the stage where they first become operative:

• Protocol version, freeze type, and amendment history

• Candidate and control identities

• Route and data-policy manifests

• Task packages and evaluator images

• Briefs, prompts, tool schemas, and context bundles

• Pilot or confirmatory randomization seed and assignment order

• Scoring rubric, judge configuration, and adjudication rules

• Time, turn, tool, concurrency, retry, and repair limits

• Pilot or confirmatory analysis code, expected assignment structure, and output schema

• Cost model and theoretical exposure report

The two-freeze chronology prevents a pilot-informed design decision from being mistaken for post-assignment adaptation, while also preventing a seemingly reasonable midstream adjustment from becoming candidate-specific treatment.

6. Task and evaluator engineering

The task is not merely a prose issue description. It is a versioned contract between the worker, the evaluator, and the analysis.

6.1 Task manifest

Each manifest binds:

• Task identifier, version, role, primary task class, difficulty, source family, incident family, inferential_family_id, and contamination status

• Source archive, environment image, lockfile, base commit, brief, visible context, documentation, and structural-pack hashes

• Editable, read-only, and prohibited paths

• Setup, build, typecheck, lint, visible-test, hidden-test, and cleanup commands with timeouts

• Weighted requirement groups, mandatory groups, observable behaviors, and score caps

• Hidden probes, historical-defect identifiers, fixtures, oracle patch, alternative-correct patch, and targeted incorrect mutants

• Base-fail, oracle-pass, revert-fail, unaffected-suite, repeated-flake, and mutant-kill evidence

The inferential_family_id is the coarsest declared grouping that contains related task and incident variants. Weighting, power analysis, variance estimation, resampling, and reporting use that same field. Repetitions and tasks nested within it never become independent inferential units.

Automated points are assigned by requirement group rather than raw test count. This prevents twenty assertions about one behavior from outweighing a single security or persistence invariant.

6.2 Task classes and boundary invariants

The common worker contract asks for the smallest complete change, evidence from actual repository contracts, exact verification, and no unrelated cleanup. Short task-class addenda focus attention on recurring boundary risks:

These addenda encode the defect classes that ordinary happy-path tests tend to miss without turning every brief into a repository manual.

6.3 Hidden evaluator validation

A hidden evaluator earns trust through its own tests:

• The base snapshot fails for the intended reason.

• The oracle patch passes.

• Reverting the oracle reintroduces the failure.

• Unaffected suites remain green.

• Repeated execution stays within a declared flake threshold.

• Targeted incorrect mutants are killed.

• At least one independently reviewed alternative implementation is accepted when the task permits multiple correct shapes.

• Hidden files remain outside the candidate mount and cannot be inferred through paths, process lists, environment, error text, timing, or output artifacts.

Evaluator faults are tracked separately from candidate failures. A block-wide fault proven to occur before model receipt invalidates the randomized block. A weak or incomplete candidate patch does not become an evaluator fault because it exposes an unanticipated implementation choice.

7. Staged protocol

The protocol separates questions that are often collapsed into one leaderboard.

7.1 Qualification

Qualification uses representative tasks to verify the complete path. A qualifying assignment produces a nonempty patch, exercises the expected tools, completes deterministic checks, persists a terminal response, reconciles authoritative usage, and triggers none of the sandbox canaries. Qualification is configuration-specific. A successful model request through one adapter does not qualify a different adapter or route.

7.2 Judge qualification

Judge development is separate from the task pilot. A 40-patch calibration set supports rubric and prompt development. A disjoint 140-patch sealed holdout licenses the frozen panel before scored review. The holdout contains at least 60 critical-defect patches and 60 integrity-defect patches, with overlapping membership reported explicitly.

Two reference-standard humans label every calibration and holdout patch, and a third reference-standard reviewer resolves disagreement. They are disjoint from scored-stage human adjudicators. Holdout bytes and labels remain inaccessible to prompt developers until the complete judge configuration hash freezes. Human adjudicators are licensed only on independently labeled patches they did not label. Judge qualification uses the thresholds in Section 8.4. Pilot tasks may exercise the review transport, evidence package, and aggregation path, but they never train, calibrate, or license judges.

7.3 Design pilot

The pilot uses tasks and evaluator materials disjoint from the scored banks and judge sets. It estimates paired task-family variance, validates difficulty and score spread, exercises assignment recovery, and checks that the fixed design can support its declared superiority or non-inferiority decision. Pilot scores never enter candidate ranking.

For Stage 2, the reference power rule uses the upper 90 percent confidence bound on paired task-family standard deviation, inflated by 25 percent. It simulates the complete decision rule, including the observed-effect threshold, interval gate, hard defects, failure-coded assignments, and missingness rule. The practical margin and the power-design alternative are distinct quantities. A design alternative must lie beyond any required observed-effect threshold and must give the complete frozen rule at least 80 percent simulated power.

The Stage 0.5 control contrast estimates worker-comparison variance, not within-worker workflow-policy variance. Stage 3 decision authority therefore also requires either disjoint workflow-pilot evidence or a prespecified conservative workflow-variance envelope. The exact alternative, variance basis, interval method, scenario grid, code, seeds, missingness rule, and operating-characteristic acceptance criteria belong in a dated statistical amendment at the confirmatory freeze. Calibration covers the null, superiority, non-inferiority, and equivalence boundaries under frozen floor, ceiling, heterogeneous-family, correlated-repetition, candidate-failure, evaluator-invalidation, and quarantine scenarios. The report publishes false-decision error, interval coverage, power, correct equivalence classification, no-winner probability, inconclusive probability, and Monte Carlo uncertainty. A method that does not pass its declared calibration may produce descriptive intervals but cannot authorize a formal decision.

The task count freezes after the pilot. The protocol does not sample repeatedly until a preferred candidate wins.

7.4 Screening

Screening applies the same worker-only contract to a broad candidate field. It uses no advisor and no quality repair. Candidate order is randomized within tasks and balanced across time blocks.

At most two candidates advance per role, and their union contains at most four distinct finalists. Advancement requires no quarantine event, no confirmed critical defect, at least four usable assigned attempts in the role, and an intention-to-treat role mean of at least 70. Eligible candidates are ordered within each role by that role-specific screening intention-to-treat mean, and the top two advance subject to the cutoff-tie rule. A candidate may advance in one role, both, or neither.

If a tie crosses the second-place cutoff, the tied candidates receive two fresh, version-frozen tie-break tasks. Tie-break scores decide only the cutoff and never enter the screening or final estimate. If the tie persists and advancing every tied candidate would exceed the role limit, that role is inconclusive. If fewer than two candidates are eligible, the role has no comparative winner. The protocol never enlarges the sealed final or displaces a tied candidate by random draw.

Screening is selection evidence, not confirmatory evidence. Reusing it in the final estimate would favor candidates selected partly because of random high scores.

7.5 Sealed final

The final compares exactly two eligible candidates per role on a fresh sealed bank with fresh repetitions. Controls provide descriptive quality references or family-correlation checks, but they cannot become winners unless the protocol defines them as eligible before assignment. Their evidence does not support a confirmatory pairwise claim when their task exposure differs from the finalists.

The frozen inferential_family_id encodes the coarsest relevant task or incident grouping and is the inferential cluster. Repetitions measure stochastic reliability inside a task; they do not create new independent task samples.

7.6 Complete-workflow confirmation

Worker-only quality does not establish the value of orchestration. The workflow stage uses fresh matched tasks and varies bounded policy packages. A representative three-arm design is:

Fresh sessions are used for each task, repetition, and arm. Arm order is randomized within matched blocks. The same acceptance rubric, hidden evaluator, verification prompt, and repair ceiling apply throughout.

The B - A contrast estimates the bounded planning package. The C - B contrast estimates assignment to the advisor package. It does not estimate the effect of only the advice that the orchestrator accepted. Same-family interactions remain exploratory unless multiple independent workers per family support a broader claim.

A repair dispatch is a fresh worker attempt after the fixed verifier returns bounded evidence. It consumes the single production-policy repair allowance. The first attempt remains preserved for intention-to-treat reliability and cost. A repair dispatch is neither transport recovery nor a replacement for the first attempt.

For these workflow gates, completion uses the same terminal-outcome indicator as the reliability analysis. No completion loss means the family-weighted paired completion-rate difference for the more complex arm is at least zero. No high or critical regression means the more complex arm contains zero confirmed high or critical defects. These are finite-bank operational hard gates, not population-level claims.

The planning package is adopted only when every declared role-worker stratum passes both hard gates, has a mean B - A advantage of at least three points, and has a calibrated two-sided 95 percent interval entirely above zero. The advisor decision is hierarchical: it is evaluated for deployment only after the planning package has been adopted. Arm C replaces Arm B only when every stratum passes both hard gates and the calibrated one-sided 95 percent lower bound for C - B exceeds minus three points. If planning is not adopted, the experiment supports no advisor-without-planning deployment claim; that question requires an explicit A-plus-advisor arm. At each step, the simpler passing policy remains the default when the gate does not pass.

8. Scoring and review

8.1 Quality score

The reference rubric allocates 60 points to mechanical behavior and 40 to identity-masked semantic review.

An acceptable assignment requires a total score of at least 75, an automated score of at least 48, every mandatory core group passing, no confirmed high or critical defect, and no integrity violation.

Caps prevent compensation across incompatible dimensions. A build failure caps the total at 39. A mandatory core failure caps it at 49. A confirmed high defect caps it at 69, and a confirmed critical defect caps it at 25. An assignment with no usable patch, or with a candidate-attributable execution failure, receives a score of zero.

Automatic quarantine applies to hidden-test access, evaluator tampering, prohibited network or secret access, fabricated verification output, and access to another candidate workspace.

8.2 Candidate-level role gates

A candidate is deployment-eligible in a sealed-final role only when all of these hold:

• No quarantine event, confirmed high or critical defect, integrity violation, or mandatory historical-defect recurrence

• Intention-to-treat mean total score of at least 75 and mean automated score of at least 48

• First-attempt acceptable rate of at least 75 percent

• Mean of the lowest four task-family scores of at least 60

• Every assigned row, including failure-coded rows, present in the frozen aggregate

These quantitative thresholds are finite-bank operational eligibility rules. They are not evidence that two candidates differ statistically. When one candidate crosses a frozen threshold and another narrowly misses it, the protocol may select the only deployment-eligible candidate, but the report labels that decision as operational selection rather than inferential superiority. An application seeking a population-level threshold claim must freeze a confidence-bound rule against that threshold before assignment.

8.3 Reference thresholds as policy defaults

The numerical thresholds are governance defaults, not universal scientific constants or population parameters. They encode the reference deployment owner's safety tolerance, practical indifference, adjudication capacity, and cost materiality on the frozen rubric. An instantiation either adopts them through documented stakeholder sign-off or replaces them in a dated amendment before assignment. Each change states its owner, rationale, consequence, units, and frozen sensitivity grid. Sensitivity analysis reruns the complete decision rule and never selects values after candidate identities or scores are known.

The three-point superiority rule requires an observed advantage of at least three and an interval excluding zero. It supports a practically material, directionally stable frozen-bank decision; it does not claim that a population effect exceeds three. That stronger claim would require the lower confidence bound to exceed three. Likewise, a full interval inside the equivalence band is a conservative operational rule and is described as a formal equivalence test only when the dated amendment defines and calibrates that test. With 32 equal-weight binary reliability rows, observed rates change in increments of 3.125 percentage points, so calibration preserves the endpoint's discreteness.

8.4 Judges as licensed measurement instruments

Identity-masked review is useful because tests rarely capture every semantic, security, maintainability, or scope error. It withholds declared candidate, model, provider, route, transcript, cost, and latency metadata, but it does not guarantee that a reviewer cannot infer candidate family from the patch. A language-model judge can still share candidate biases, over-credit plausible prose, miss requirement violations, or prefer a familiar style.

The protocol therefore treats each judge like a measurement instrument. For reproducibility, the publication synthesis resolves the historical metric and aggregation shorthand as follows:

• Candidate identities remain opaque until scoring and adjudication freeze.

• Judges see the task, relevant frozen source, normalized diff, visible-test summary, and opaque requirement-group outcomes.

• Judges do not see provider, model, transcript, cost, latency, hidden-test details, or reference patch.

• Judge identity, route, prompt, reasoning setting, assignment seed, anchors, coverage, and aggregation freeze before scoring.

• Every fixed judge, machine adjudicator, and human adjudicator passes the applicable licensing gates individually. A panel average cannot rescue an unlicensed instrument.

• Screening uses one fixed primary judge on every patch and a balanced second-judge audit covering 25 percent of all screening patches, stratified across candidate or control configuration and time block. The frozen allocator rounds the total up to the next whole patch. The primary score is operative unless the audit triggers binding adjudication.

• The sealed final and workflow stage use both fixed judges on every patch.

• Rubric development uses the separate 40-patch calibration set.

• Licensing uses the 140-patch sealed holdout containing minimal correct patches, overengineered correct patches, subtle semantic failures, severe security or data failures, test gaming, and unrelated cleanup. It contains at least 60 critical-defect and 60 integrity-defect examples.

• Two reference-standard human reviewers label each licensing patch against the anchored rubric, and a third reference-standard reviewer resolves disagreement. These labelers are disjoint from scored-stage human adjudicators. Holdout bytes and labels remain inaccessible to prompt developers until the complete judge configuration hash freezes. Critical-defect and integrity-defect strata are reported separately even when one patch belongs to both.

• Licensing requires, for each judge against the adjudicated human labels, raw acceptable-versus-unacceptable agreement of at least 0.85, linearly weighted Cohen's kappa [23] over the ordered severity classes of at least 0.65, Spearman score correlation of at least 0.75, one-sided 95 percent exact Clopper-Pearson upper bounds [24] on critical and integrity false-negative rates no greater than 5 percent, and zero observed integrity misses.

• Score disagreement above eight points, severe findings, integrity concerns, and unresolved winner decisions trigger independent adjudication.

For patches scored by both fixed judges, the review score is the arithmetic mean within each dimension and then the sum of the five dimension scores. Binding adjudication replaces only the disputed dimension before the sum is calculated. Screening audits use the same disagreement and adjudication path without averaging an unaudited patch against a missing second score.

The licensing holdout is intentionally enriched for critical and integrity defects. That case mix supports severe false-negative measurement, but pooled agreement, kappa, correlation, accuracy, and predictive values do not estimate performance under deployment prevalence [16,17]. The licensing report therefore publishes counts and confusion matrices within every prespecified defect stratum, exact intervals for stratum-specific false-negative rates, explicit overlap accounting, and a macro-average. A deployment-weighted aggregate is permitted only with externally justified frozen prevalence weights over non-overlapping strata and sensitivity to plausible alternative weights.

Human adjudicators use the same anchored rubric and cite concrete evidence. They may replace only the disputed dimensions. Every reviewer finding remains a hypothesis until it is reproduced against the frozen patch, visible contract, and evaluator evidence. A random 10 percent human sentinel sample checks drift after licensing, and every proposed winner receives human review.

9. Failure and retry semantics

Unclear retries are one of the easiest ways to bias an agent comparison. The protocol distinguishes three events.

9.1 Candidate-attributable failure

A refusal, timeout, context overflow, malformed tool loop, unusable patch, or adapter behavior that is part of the frozen configuration remains in the intention-to-treat population. It receives the declared failure score, normally zero. Reporting conditional quality among completed patches is descriptive only.

9.2 Transport recovery

Transport recovery can apply only when provider evidence proves both that no model receipt occurred and that no candidate output exists. A malformed response, parser failure, adapter failure, context error, timeout after receipt, or unusable patch is a configuration failure, not transport recovery. Recovery never provides another opportunity to improve a solution.

The reference worker-only protocol permits zero route or candidate retries. An adaptation that permits pre-receipt transport recovery must freeze it by route and support it with request identifiers, write-ahead attempt records, and provider evidence. Every attempt retains its latency, usage, cost, and first-attempt reliability status. An ambiguous outcome is preserved as unreconciled rather than guessed away.

9.3 Quality repair

Quality repair begins after a candidate has produced a solution and received permitted feedback. It is a workflow treatment, not infrastructure recovery. Worker-only stages permit no quality repair. A production-policy arm may permit one bounded repair if every compared arm uses the declared repair contract.

Each provider attempt is journaled before dispatch. Responses are persisted atomically. The ledger connects reservation, request, response, patch, evaluation, review, and repair lineage. An interruption cannot silently turn one assignment into multiple uncounted chances.

9.4 Evaluator defects after model receipt

An evaluator defect found after a response exists does not authorize new inference. The response and patch freeze, the evaluator is repaired under identity-masked task and candidate labels, and every patch in the affected randomized block is re-evaluated. The original and corrected evaluator, reason, hashes, and score changes remain in the ledger.

If the repair changes the model-visible contract, requires candidate-specific judgment, or cannot be applied symmetrically, the whole affected task family is quarantined. The report publishes both the prespecified failure-coded analysis and a family-removed sensitivity analysis. Only a proven evaluator failure before model receipt permits a fresh complete-block assignment under the frozen recovery rule.

10. Statistical decision rules

The reference final uses one task per inferential family and the same number of repetitions for every candidate. Let q(c,f,r) be candidate c's score for family f and repetition r, including declared zeroes for candidate-attributable failures:

ITT quality(c) = mean over families f of mean over repetitions r of q(c,f,r)

An adaptation with multiple tasks inside a family aggregates in this order: repetition mean, then task mean, then family mean. The point estimate, power simulation, and resampling must use the same weights and preserve candidate pairing. Invalidated evaluator rows remain in the audit ledger but not the estimand; their valid symmetric replacement block occupies the original family weight.

For a paired final, compute candidate differences within each inferential_family_id. The dated statistical amendment selects one paired family-level interval method after the pilot and before Stage 1. It freezes the implementation, seed, design alternatives, scenario grid, and acceptance criteria, then calibrates the complete decision rule at its declared confidence levels. Ordinary percentile intervals receive no presumption of validity for bounded scores at floors or ceilings. Any method may authorize a formal decision only after its frozen operating-characteristic checks pass. Otherwise its intervals are descriptive and any superiority, non-inferiority, or equivalence decision that depends on them is inconclusive.

The amendment validates the complete gated decision rule, not only the interval estimator. Following the aims, data-generating mechanisms, estimands, methods, and performance-measures structure for simulation studies [14], it covers the reference family counts of eight and sixteen and every count used by an adaptation. The frozen grid includes bounded floor, middle, and ceiling scores; zero mass from candidate failures; weak through strong within-family correlation; heterogeneous family effects and variances; asymmetric failure and hard-gate prevalence; evaluator invalidation, symmetric replacement, and family quarantine; and every superiority, non-inferiority, and equivalence boundary. Design alternatives lie strictly beyond the decision boundary rather than on it.

The report publishes interval coverage and width, false superiority, false non-inferiority, false equivalence at both boundaries, power and correct-decision rates, no-winner and co-winner rates, hard-gate and family-quarantine rates, inconclusive rates, and Monte Carlo uncertainty. It binds code, dependencies, generator parameters, seeds, and raw aggregate counts. Calibration is endpoint-specific: a failed quality calibration disables the affected quality claim, while a failed reliability calibration disables the reliability preference. Descriptive estimates remain reportable, but a failed or missing calibration cannot authorize the corresponding decision. The supplement supplies the operational amendment checklist.

The interval describes stability over the frozen task bank. It does not turn that bank into a random sample of all software engineering. Judges, assertions, repetitions, nested tasks, and sibling defects are never treated as independent family samples.

The reference superiority margin is three quality points. After both candidates pass the role gates, let H be the candidate with the higher frozen intention-to-treat mean and L the other candidate. Every superiority difference and interval is oriented as H - L. If the means are equal, superiority cannot pass and the rule proceeds to equivalence or inconclusive evaluation. Candidate H defeats candidate L only when:

• H's mean paired advantage is at least three points.

• The task-clustered 95 percent interval excludes zero.

The reference equivalence band is minus three to plus three points. When the full paired interval lies inside that band, capability is operationally equivalent under the frozen protocol.

Capability-equivalent candidates remain co-winners unless one has a first-attempt unacceptable rate at least five percentage points lower and the paired reliability interval excludes zero. The reliability estimator applies the same family-then-repetition weights as quality to a binary first-attempt-unacceptable indicator, and its paired interval uses the calibrated method frozen for that endpoint. Reliability is operationally equivalent when the full interval for the unacceptable-rate difference lies inside minus five to plus five percentage points.

Only when reliability is operationally equivalent may cost choose a deployment-preferred co-winner. The cost estimator is the family-weighted arithmetic mean of first-attempt shadow cost over the same assigned rows. At least 15 percent lower means the preferred candidate's mean divided by the other candidate's mean is no greater than 0.85. This ratio is a deterministic finite-bank operational preference, not an inferential cost claim. Missing or unreconciled cost disables the preference.

The complete rule is:

An interval that excludes zero is not enough when the effect is smaller than the practical margin. An interval overlapping zero is not equivalence unless it lies fully inside the equivalence band. An underpowered or ambiguous result remains inconclusive. The point-estimate leader is never promoted by intuition.

New-code and bug-fix selection are prespecified as two separate deployment decisions, each with one confirmatory final pair. The protocol makes no joint across-role claim. An adaptation that asserts one joint conclusion across roles or tests multiple confirmatory pairs within a role must freeze and justify a multiplicity procedure before assignment. Exploratory interactions remain labelled exploratory regardless of their nominal intervals.

Workflow adoption is a conjunctive operational decision across declared role-worker strata: every stratum must pass its gate. Adoption is also hierarchical. The B - A planning decision occurs first; only an adopted B policy can be replaced by C through the C - B advisor decision. A more complex policy must contain zero confirmed high or critical defects, have a nonnegative family-weighted paired completion difference, and satisfy its calibrated superiority or non-inferiority condition. Otherwise the simpler passing policy remains the deployment default.

11. Evidence architecture and feasibility

The evidence model is an append-only graph of hash-bound records:

protocol -> plan -> task manifest -> assignment -> provider attempt
         -> response -> patch -> automated evaluation -> identity-masked review
         -> adjudication -> aggregate -> publication artifact

The minimum ledger contains:

• Tournament records for protocol version, protocol-freeze hash, task-bank hash, authorized exposure, and state transitions

• Candidate records for opaque seat, sealed identity, requested and resolved model, harness, route manifest, tool manifest, context policy, and approved data tier

• Task records for manifest version, role, family, difficulty, contamination class, source hash, brief hash, evaluator hash, and historical-defect identifiers

• Attempt records for reservation, assignment order, sandbox, request identifiers, response identity, terminal state, usage, latency, transport lineage, and persistence hashes

• Patch and evaluation records for changed bytes, deterministic commands, requirement groups, mandatory gates, score caps, and defect identifiers

• Review and adjudication records for identity-masked evidence, dimensions, cited findings, disagreement triggers, and final dispositions

• Aggregate records for role estimates, intervals, acceptability, reliability, economics, advancement, and decision state

Model identity mapping remains separate from identity-masked review evidence until every score and disposition is signed. Raw transcripts and sensitive evidence remain encrypted with a deletion policy. Published records use sanitized findings and immutable hashes.

The plan is executable control state, not decorative metadata. It pins assignment order, binaries, routes, task hashes, images, prompts, limits, and analysis entry points. A process lock prevents concurrent schedulers. Authorization is checked before each provider attempt. Write-ahead journaling and atomic response persistence make interruption visible and recoverable.

11.1 Synthetic worked decision

The following synthetic trace shows how one frozen assignment becomes part of a decision:

1. Freeze. Configuration A-03, its route manifest, task F07-T1-R2, evaluator image, scoring rule, assignment order, and calibrated statistical amendment are hash-bound in the plan.

2. Reservation. The scheduler appends the assignment and provider-attempt reservation before dispatch. A process interruption can therefore leave an inspectable reservation rather than an invisible extra chance.

3. Receipt. The returned route identity matches the manifest. The terminal response, authoritative usage, normalized patch, and hashes persist atomically.

4. Evaluation. The patch receives a synthetic 53 of 60 mechanical points, passes every mandatory group, and triggers no cap or quarantine rule.

5. Identity-masked review. In rubric order, the fixed judges return [11, 7, 6, 5, 7] = 36 and [9, 3, 5, 3, 6] = 26. Their ten-point disagreement exceeds the eight-point trigger. The unadjudicated mean is 31. Binding adjudication replaces the five-point mean security dimension with seven, producing 31 - 5 + 7 = 33 review points and a total of 86.

6. Aggregation. The row remains paired with the other candidate inside family F07. Repetitions average before the family enters the role aggregate. No judge score, repetition, or assertion becomes an independent family sample.

7. Decision. In the complete synthetic bank, both candidates pass every role gate. Candidate A is H, with a four-point mean paired advantage and a calibrated 95 percent interval from plus 0.6 to plus 7.4. The frozen rule selects A because the observed advantage reaches three and the interval excludes zero.

Every value in this trace is synthetic. Its point is the lineage from plan to assignment, attempt, response, patch, evaluation, review, adjudication, family aggregate, and decision. No discretionary ranking step occurs after the confirmatory freeze.

11.2 Resource and feasibility envelope

The comprehensive reference design schedules at most 22 + 64 + 102 + 192 + 192 = 572 worker rows. Cutoff tie-breaks can add 36, producing a 608-row scheduled envelope. If every row reaches the 90-minute timeout, that is 912 worker-row hours. Perfect four-way saturation would take 228 hours, or 9.5 continuous days. This is timeout exposure, not expected duration, and it excludes route serialization, queueing, setup, evaluation, and human work.

Scheduled rows are not provider calls. Each of the 192 workflow rows permits one fresh repair dispatch. If every repair receives the same 90-minute allowance, maximal repair use adds 192 worker attempts and 288 hours. Scheduled rows plus maximal repairs therefore reach 800 worker attempts and 1,200 worker-attempt hours, or 300 idealized hours and 12.5 continuous days at concurrency four. Evaluator-fault blocks, orchestrator and advisor calls, and adjudication remain outside that bounded scenario, so 608 rows and 912 hours are not absolute cost ceilings.

Review is also material. Two humans supply at least 360 initial ratings for the 40-patch calibration set and 140-patch licensing holdout, before third-reviewer dispositions. Licensing two fixed judges and one machine adjudicator requires 420 machine evaluations. At the reference row maxima, all 102 screening patches receive the primary judge, the frozen 25 percent audit rounds 25.5 up to 26, and the 384 final and workflow rows receive two judges each. This gives 102 + 26 + 768 = 896 scored-patch judge evaluations. The estimate includes screening controls but excludes cutoff tie-breaks, pilot semantic review, sentinels, severe-finding adjudication, and calibration iterations.

Every instantiation therefore freezes a route-specific bill of materials covering scheduled rows and provider calls by actor, repair and evaluator-fault allowances, expected and timeout hours, concurrency bottlenecks, current and adverse price scenarios, human review minutes, storage, quotas, retention, and the authorized completion window. The comprehensive form requires sustained infrastructure and review capacity; the bounded comparator remains the preferred default for a two-option decision.

12. Design history

The protocol evolved by narrowing the decision question while retaining the controls required for a trustworthy answer.

12.1 From labels and benchmarks to defined research objects

The earliest framing resembled a model tournament, but model labels could not identify provider aliases, reasoning settings, account routes, harnesses, tools, context limits, or sandbox policies. The roster therefore became a set of complete configurations, and the output became a role matrix for new-code and bug-fix work with capability, reliability, and economics kept separate.

Task descriptions changed in parallel. Each task became a machine-readable contract with explicit requirements, commands, hidden probes, oracle evidence, valid alternative solutions, mutants, and isolation checks. A hidden evaluator no longer counted as truth merely because candidates could not see it; it had to accept valid solutions, detect intended defects, reject targeted wrong solutions, and remain isolated.

12.2 From averages and opinions to gated measurement

A single average allowed severe failures to hide behind easy-task performance. Mandatory groups, severity caps, quarantine events, first-attempt reliability, lower-tail family performance, and historical-defect gates produced four legitimate outcomes: winner, co-winners, no winner, and inconclusive. Practical thresholds and power alternatives were separated, and every inferential method became conditional on full-rule calibration.

Semantic review underwent the same change. Opaque identities, fixed evidence, rubric anchors, disjoint calibration and licensing sets, severe false-negative gates, sentinels, and binding adjudication turned a model opinion into a licensed measurement process. Authoring, read-only advice, outcome judging, repair, and adjudication became separate roles so that no participant could be its own only verifier.

12.3 From a comprehensive design to bounded control planes

The comprehensive design defines the broad validity envelope, but many deployment choices concern one incumbent and one challenger. The bounded form narrows the field and task bank while retaining exact configuration qualification, fresh pairing, independent verification, one declared repair ceiling, severe-defect gates, non-inferiority, and an inconclusive outcome. It answers a smaller question without weakening the meaning of the answer.

Operational rules then moved into machine-readable plans that bind routes, tasks, images, prompts, assignment order, limits, authorization, review, and analysis. Process locks, write-ahead attempt records, atomic persistence, hash validation, explicit reconciliation, and separate transport-recovery and quality-repair paths made failure handling part of the method rather than an informal runner behavior.

12.4 From worker ranking to calibrated workflow decisions

The final refinement treated production topology as a treatment. Matched arms compare complete workflow policies, and a component receives a causal interpretation only when the arms differ in that component alone. Randomization stays inside task-family and repetition blocks, while common deterministic outcomes remain separate from treatment-specific semantic verification.

Version 1.1 closed the remaining executable ambiguities without rewriting the historical artifacts. It separated pilot and confirmatory freezes, named the screening metric, bound clustering through inferential_family_id, made workflow adoption hierarchical, defined completion and severe-regression gates, specified judge aggregation, and gave reliability and cost preferences explicit estimators. Version 1.2 publishes the paper and supplement as a public, hash-bound web edition while the implementation repository and supporting evidence remain private and available upon request from nullhex, subject to stated exclusions. It changes the release and availability boundary, not the protocol. Statistical alternatives and interval implementations remain in the dated amendment because their authority depends on frozen, full-rule calibration.

The resulting family of protocols supports three scopes:

12.5 Companion applications and claim boundaries

The release maps the protocol to three bounded applications without pooling their outcomes. The lean worker race exercises finite-bank configuration screening, evaluator behavior, identity-masked review, and repair, but supports no universal ranking. The matched workflow study treats complete workflow policies as the arms, so it supports no model-only or isolated-component claim. The matched follow-up is a descriptive historical extension rather than an independent causal contrast. Their reports, plans, and evidence packages demonstrate how frozen protocol components become auditable research objects. The supplement records exact artifact paths and reproduction boundaries.

13. Reproduction model

The publication package supports three related activities.

13.1 Protocol reproduction

Another team can reconstruct the treatments, data boundary, task packaging, randomization, limits, scoring, failure semantics, and decision rules. Provider-specific routes may differ, but every resolved identity and deviation is recorded as a new configuration.

13.2 Analysis reproduction

A frozen evidence bundle can be validated and aggregated without provider credentials. The analysis command checks source hashes, expected assignment counts, invalidation lineage, scoring inputs, interval seeds, and output hashes before regenerating human and machine reports.

13.3 Experimental re-execution

A team can reuse the runner, manifests, task packages, and analysis with current provider routes. Because hosted models and services change, each live use constitutes a new dated experiment rather than a bit-for-bit replay of another hosted inference environment. The route manifest makes the difference explicit instead of pretending a model name is immutable.

The protocol's three practical labels map onto ACM terminology as follows [25]:

• Repeatability means the same team uses the same artifacts and experimental setup.

• Reproducibility means a different team uses the supplied artifacts and experimental setup.

• Replicability means a different team constructs an independent setup to test the same claim.

Protocol reproduction is preparation for one of those activities, not a separate ACM badge. Deterministic analysis by another team using the supplied evidence is reproducibility. Current-route re-execution is a new dated experiment whose ACM label depends on which team, artifacts, and setup are reused.

13.4 Publication bundle

The fixed paper bundle joins the paper and supplement with their editable and rendered editions, publication metadata, licences, a release manifest, and cryptographic checksums. It is complete for paper reconstruction and protocol inspection. The supplement provides the complete release checklist and identifies the wider evidence lineage.

The private implementation repository and supporting evidence are available upon request from nullhex, subject to exclusions for credentials, customer data, private source, sensitive transcripts, and operational identifiers. The public bundle retains hashes, schemas, and sanitized lineage needed to inspect the relationship among released artifacts. Hosted calls cannot be reproduced bit-for-bit: a live re-execution is a new dated experiment with its own route and policy manifest.

14. Threats to validity

14.1 Task representativeness

An organization-specific bank supports an organization-specific decision. Role balance, risk classes, language mix, repository families, and difficulty must be reported. Results should not be generalized to unrelated software domains without new evidence.

14.2 Task and oracle bias

Historical fixes can encode one implementation style. Alternative-correct patches, mutant tests, requirement-group scoring, and independent task review reduce this risk. They cannot prove that every valid solution shape is represented.

14.3 Contamination

Public issues, known fixes, repository history, and prior benchmark exposure can leak solutions. Fresh or transformed tasks, post-cutoff sources, no Git history, contamination labels, and sealed evaluators reduce exposure. Contamination status remains part of interpretation.

14.4 Harness differences

Native tool ergonomics are part of deployability but also complicate attribution. Capability parity and conformance tests bound unfair information advantages. Worker-level results are attributed to the complete configuration, not to model weights alone.

14.5 Judge validity and correlation

Judges can share training data, style preferences, or family-specific biases with candidates. The deliberately severe-defect-enriched licensing holdout supports false-negative measurement but does not reproduce deployment prevalence, and pooled agreement metrics depend on that case mix. Stratum-specific confusion matrices and false-negative intervals are primary; deployment-weighted summaries require externally justified frozen prevalence weights.

Identity masking also leaves residual identification channels. Formatting, imports, comments, test strategy, patch structure, naming, tool residue, and characteristic implementation patterns can correlate with a model or harness family. Family separation, licensing, two-judge review, anchored evidence, sentinel human checks, and adjudication reduce direct identity bias but do not establish perceptual blindness. Any normalization must be deterministic, frozen, candidate-symmetric, and semantics-preserving. A non-scoring identity probe may measure residual identifiability after scores freeze; its guesses never alter scores. Mechanical behavior and severe gates remain independent of judge generosity.

14.6 Route and service drift

Hosted systems can change without a new public model name. Resolved identity, request identifiers, endpoint, date, region, route policy, and response metadata are therefore required for every assignment. Claims attach to dated configurations.

14.7 Operational metrics

Tokens, latency, subscription cost, and promotional credits differ in meaning across providers. They support operational comparisons inside the declared accounting policy. They do not measure a universal amount of reasoning or computational work.

14.8 Protocol complexity

The comprehensive design imposes the worker, judge, and human-review envelope quantified in Section 11.2. The bounded comparator is the preferred form when the decision concerns one incumbent and one challenger. Simplification is valid when it narrows the claim, not when it removes controls while preserving the broad claim.

15. Operational supplement

The bundled protocol supplement is part of version 1.2. It contains the adaptation checklist, reference instantiation parameters, minimum statistical-amendment contents, companion-application claim boundaries, and publication source map. Moving that lookup material out of the narrative reduces repetition without weakening the operational contract. The release manifest and checksums bind the paper and supplement as one citable object.

16. Declarations

Authorship and control. nullhex research defines the protocol, selects the claims, and retains responsibility for the manuscript and release artifacts.

AI assistance. Model systems assisted with repository source mapping, primary-literature discovery, adversarial outline review, drafting, and editorial checks under author direction. Historical claims and citation targets were checked against primary sources or versioned repository artifacts. Publication-time specification clarifications are identified in the dated design history.

Funding. The manuscript was prepared with internal nullhex engineering time and no dedicated external research grant. Commercial subscriptions and promotional credits are operational inputs to particular protocol instantiations, not funding for manuscript preparation. Each release must disclose their scope and value.

Competing interests. nullhex is a customer or prospective customer of commercial model providers and may use protocol decisions for purchasing or deployment. Providers receive no authority over task selection, scoring, claims, or publication. Any additional provider relationship must be recorded in the applicable release manifest.

Artifact and privacy statement. The release artifact manifest defines the public research object. The private implementation repository and supporting evidence are available upon request from nullhex, subject to exclusions for credentials, customer data, private source, sensitive transcripts, and operational identifiers. Public artifacts retain hashes sufficient to verify their relationship to frozen evidence. Hosted calls cannot be reproduced bit-for-bit; they are re-executed only as newly dated experiments.

Licence. The paper is released under Creative Commons Attribution 4.0. Software and data retain the licences declared in their respective artifact records.

17. Conclusion

Coding-agent selection is a systems experiment. A model name alone cannot identify the treatment, an aggregate score cannot express the decision, and a hidden test cannot validate itself.

The Worker Model Tournament turns those observations into a reproducible protocol. It binds configuration identity, capability parity, privacy, sandboxing, task contracts, evaluator validation, identity-masked review, failure accounting, paired inference, and evidence lineage. It produces role-specific deployment decisions and explicitly permits co-winners, no winner, and inconclusive evidence.

The design history offers a practical lesson. Rigor does not require using the largest possible tournament for every decision. It requires matching the experimental scope to the claim, then retaining the controls that keep that claim honest. A comprehensive tournament, a bounded challenger comparison, and a matched workflow study can share one validity envelope while answering different questions.

The result is less like a leaderboard and more like a release process for evidence: versioned, reviewable, failure-aware, and citable.

References

1. Carlos E. Jimenez et al. SWE-bench: Can Language Models Resolve Real-World GitHub Issues?. International Conference on Learning Representations, 2024.

2. John Yang et al. SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering. Advances in Neural Information Processing Systems, 2024. DOI: 10.52202/079017-1601.

3. Yilun Yao et al. Harness-Bench: Measuring Harness Effects across Models in Realistic Agent Workflows. arXiv:2605.27922, 2026.

4. Mengyu Zheng et al. Claw-SWE-Bench: A Benchmark for Evaluating OpenClaw-style Agent Harnesses on Coding Tasks. arXiv:2606.12344, 2026.

5. Cornelius Emde et al. MASEval: Extending Multi-Agent Evaluation from Models to Systems. Annual Meeting of the Association for Computational Linguistics, 2026. DOI: 10.18653/v1/2026.acl-demo.34.

6. Chunqiu Steven Xia et al. Demystifying LLM-based Software Engineering Agents. Proceedings of the ACM on Software Engineering 2, FSE, article FSE037, 2025.

7. Ibragim Badertdinov et al. SWE-rebench: An Automated Pipeline for Task Collection and Decontaminated Evaluation of Software Engineering Agents. Advances in Neural Information Processing Systems, 2025.

8. Hongchao Jiang et al. CodeJudgeBench: Benchmarking LLM-as-a-Judge for Coding Tasks. Annual Meeting of the Association for Computational Linguistics, 2026. DOI: 10.18653/v1/2026.acl-long.888.

9. Ruiqi Wang et al. Can LLMs Replace Human Evaluators? An Empirical Study of LLM-as-a-Judge in Software Engineering. Proceedings of the ACM on Software Engineering, 2025.

10. International Council for Harmonisation. Addendum on Estimands and Sensitivity Analysis in Clinical Trials to the Guideline on Statistical Principles for Clinical Trials, ICH E9(R1). 2019.

11. Gilda Piaggio, Diana R. Elbourne, Stuart J. Pocock, Stephen J. W. Evans, and Douglas G. Altman, for the CONSORT Group. Reporting of Noninferiority and Equivalence Randomized Trials: Extension of the CONSORT 2010 Statement. JAMA 308(24):2594-2604, 2012.

12. Kosuke Imai, Gary King, and Clayton Nall. The Essential Role of Pair Matching in Cluster-Randomized Experiments, with Application to the Mexican Universal Health Insurance Evaluation. Statistical Science 24(1):29-53, 2009.

13. James E. Pustejovsky and Elizabeth Tipton. Small-Sample Methods for Cluster-Robust Variance Estimation and Hypothesis Testing in Fixed Effects Models. Journal of Business & Economic Statistics 36(4):672-683, 2018.

14. Tim P. Morris, Ian R. White, and Michael J. Crowther. Using Simulation Studies to Evaluate Statistical Methods. Statistics in Medicine 38(11):2074-2102, 2019.

15. Jacob Cohen. A Coefficient of Agreement for Nominal Scales. Educational and Psychological Measurement 20(1):37-46, 1960.

16. Ted Byrt, Janet Bishop, and John B. Carlin. Bias, Prevalence and Kappa. Journal of Clinical Epidemiology 46(5):423-429, 1993.

17. Patrick M. Bossuyt et al. STARD 2015: An Updated List of Essential Items for Reporting Diagnostic Accuracy Studies. BMJ 351:h5527, 2015.

18. Brian A. Nosek, Charles R. Ebersole, Alexander C. DeHaven, and David T. Mellor. The Preregistration Revolution. Proceedings of the National Academy of Sciences 115(11):2600-2606, 2018.

19. Christopher D. Chambers. Registered Reports: A New Publishing Initiative at Cortex. Cortex 49(3):609-610, 2013.

20. Richard A. DeMillo, Richard J. Lipton, and Frederick G. Sayward. Hints on Test Data Selection: Help for the Practicing Programmer. Computer 11(4):34-41, 1978.

21. Luc Moreau and Paolo Missier, editors. PROV-DM: The PROV Data Model. W3C Recommendation, 30 April 2013.

22. Geir Kjetil Sandve, Anton Nekrutenko, James Taylor, and Eivind Hovig. Ten Simple Rules for Reproducible Computational Research. PLOS Computational Biology 9(10):e1003285, 2013.

23. Jacob Cohen. Weighted Kappa: Nominal Scale Agreement with Provision for Scaled Disagreement or Partial Credit. Psychological Bulletin 70(4):213-220, 1968.

24. C. J. Clopper and E. S. Pearson. The Use of Confidence or Fiducial Limits Illustrated in the Case of the Binomial. Biometrika 26(4):404-413, 1934.

25. Association for Computing Machinery. Artifact Review and Badging. Current policy, accessed 14 July 2026.