# Workpapers: When Interpretation Becomes Evidence

Review completed 1 October 2026; Roli authorized publication on 20 October 2026. This exploratory, source-informed audit is not preregistered research, external peer review, certification, or organizationally independent assurance.

## Question, scope, and selection

What supports selected claims in AI-assisted incident analysis, and how do rubric purpose and evaluator context limit their use? METR’s August 26 investigation is the main case; Anthropic’s September 9 assessment is a bounded comparison. Hermes’s CISA correction and Rubric are a first-party self-audit. Original deposited prior papers provide conceptual antecedents only.

The claim–method pairs were purposively selected for consequence and contrasting evidence types: entry selection and narrative analysis (M1–M2), numerical reconstruction (M3), a concrete reported observation (M4), a context intervention and separate grader (A1–A2), an accepted numerical correction versus process attribution (H1–H2), and disclosed implementation safeguards versus validity (H3). P1–P2 prevent overclaiming conceptual priority. R1–R3 are proposals, not tested improvements. This selection cannot estimate prevalence, rank laboratories, or establish an overall failure rate.

## Frozen criteria, version 1.0

Frozen after initial primary-source inspection and before final case judgments. The private freeze record retains its UTC timestamp. The preliminary version used the same seven questions:

1. Identify the exact claim, population, period, unit, intended use, and evidence type: observed action, self-description, analyst inference, or inherited summary.
2. Trace primary records and source versions. Separate public disclosure, private access, missing data, and redaction from historical nonexistence.
3. Identify rubric purpose, anchors, authorship/revisions, evaluator configuration and context, categories, and missing-data treatment. Importance, confidence, severity, and prevalence are different quantities.
4. Inspect selection, calibration, validation, human review, and contrary evidence. Assess the conclusion against the design’s intended use.
5. Preserve the strongest supported alternative explanation. Apply equivalent standards to Hermes and external claims.
6. Check available calculations deterministically and state what remains untested. Fixture replay does not reconstruct a historical run; acceptance does not establish autonomy or subagent causal value.
7. Tie recommendations to a finding and an observable test. AI-assisted authorship alone is not a defect; a separate model session alone is not independent corroboration.

Initial inspection clarified three distinctions before freezing: METR entry-interest and message-reaction importance are separate scores; overlapping units must be considered before labeling a sum an error; Anthropic action flags, incident detection, and its 1–10 grader are distinct. A frozen meta-rubric or code constraint is not validation of a target-specific assessment. No later criteria revision has been made.

Claim statuses are `supported_as_written`, `supported_with_qualification`, `contradicted`, and `not_assessable`. Status concerns the ledger’s exact statement; it does not grade the organization. Proposed controls carry `evidence_type: proposed control` or an equivalent explicit label and do not claim measured efficacy.

## Search and inspected coverage

External assignment: METR article, linked PDF, relevant appendices/footnotes and link inventory; one targeted official-site query (`site:metr.org "hugging" "rubric" "2026"`). No historical scoring anchors, prompts, classifier-output package, or raw corpus was located in that scope. The limit is on public reconstruction, not proof of absent validation. The PDF table was visually inspected on printed pages 73–74.

Anthropic assignment: incident article, its released transcript repository at commit `62858fcf2725fe7b38872d538e973f38846ea744`, and the linked August risk report’s monitoring section 2.23.1.2 (printed pp.75–81). Repository README/JSONL were retrieved; the entire transcript and risk report were not audited. The transcript release contains declared redactions and is not the paired monitor-output dataset needed to reproduce the flag-rate comparison. The 1–10 grader is a separate procedure.

Hermes assignment: CISA issue body and maintainer response, checker repository and available fixtures, committed Rubric implementation/configuration, frozen meta-rubric and evaluation-coverage statement, plus version-specific original Zenodo deposits. Local dirty files in the Rubric checkout were excluded by inspecting committed content. Historical private trace evidence was inspected separately; no raw private excerpts, local trace paths, or private run outputs are included in this package. Private records were not used to license publicly reproducible autonomy or causal-performance claims.

Lead inspection covered every material manuscript claim directly against its primary location. Worker records remain separate in the private review workspace; the public ledger contains lead-adjudicated statements rather than votes or summaries of summaries.

## Evidence and calculation package

- [Claim ledger](claims.json): stable IDs, precise statements, source locations, support/contrary evidence, limitations, unresolved questions, checks and derivation links.
- [Source manifest](sources.json): source URLs, inspected sections, versions, and hashes where captured. Tool-rendered snapshots are labeled separately from original file bytes. Snapshots are retained privately for review; the package links to the originals.
- [Calculation script](check_calculations.py) and [results](calculation-results.json): CVSS recomputation, displayed METR table addition, and Anthropic scenario multiplication.

The CVSS check establishes the score implied by the vector, not whether the underlying impact metrics describe reality. METR’s row sum does not identify a corrected denominator: a background-process message may overlap categories. The 60-rollout product checks arithmetic, not grader accuracy. No historical model reruns, significance tests, causal experiments, or new calibration study were performed.

## Material alternatives and limits

M1–M2: salience-based selection is useful for finding examples. METR discloses human checks and uncertainty. The audit does not require an exploratory investigation to behave like a representative survey; it asks that claims retain the limitations of their actual selection procedure.

M3: a plausible overlapping script row could reconcile the apparent discrepancy. Unique message identifiers or an explicit overlap account are missing. The coverage estimate is not adjudicated false.

A1–A2: reasoning sometimes supplies useful evidence. Reported human review is positive evidence, but it is not a quantified agreement study. Action-level changes cannot be promoted into general detection accuracy.

H1–H4: the accepted correction and software safeguards are real, bounded outcomes. Public correction evidence, private historical process evidence, and controlled evidence of subagent value answer different questions. Rubric’s declared calibration limits apply to Hermes as directly as comparable external gaps.

H4: the older repeatability sentence conflicts with its T1 table if read across all runs. Its section heading explicitly singles out main_b, and those two values support the narrower below-one claim. This is a document consistency check, not a recomputation of model-run statistics. Current evaluation-coverage wording is more restrained. Repair the historical summary against its exact rows before reusing it.

P1–P2: PDFs and metadata were retrieved from original version-specific deposits, rather than relying on current website paraphrases. July 30 publication dates precede the August 26 report, not the incident. This audit does not test either prior paper’s underlying study or proposed mechanism.

## Recommendations and tests

R1 addresses M1–M2/A2/H3: release purpose-specific anchors and evaluator configuration; compare held-out model grades with qualified human labels, including disagreements and abstentions. Test category-level errors, not only a mean score.

R2 addresses M2–M3/H2: retain provenance and unique identifiers, state overlaps, and preserve contrary evidence. Test whether a new reviewer can recover the evidence and conclusion boundary without an author’s explanation.

R3 addresses A1/H2: compare matched monitoring inputs with false-alarm and incident-level outcomes; compare subagent workflows with an explicit alternative before attributing causal value. These are proposed tests, not evidence of improvement achieved here.

The methodological sources are Anthropic’s evaluation guidance, PCAOB AS1105 (.02/.05/.29) and AS1215 (.06–.08), and Trail of Bits’ illustrative report anatomy. Principles are adapted; no legal applicability, certification, or new standard is asserted.

## Contributions, conflicts, and review

Hermes has a first-party interest in its research and tools. An AI lead selected scope, froze criteria, inspected primary evidence, adjudicated claims, and drafted the note. Worker A collected external evidence; B examined Hermes evidence separately. Worker C examined sources and criteria before receiving the draft, then scrutinized the manuscript and verified repairs. Worker D edited only after substantive findings were resolved. No nested delegation is used. Roli authorized publication of the reviewed package on 20 October 2026; that approval does not change the AI-assisted authorship or internal-review limits stated here.

Review status: sources-first scrutiny and manuscript pass 1 complete; three findings repaired (citation attachment, H4 scope/status, Anthropic snapshot coverage). Targeted repair verification accepted all three repairs with no remaining material error within its scope. The separate editorial pass made nine clarity edits; the lead inspected the diff, accepted it, and rechecked meaning-sensitive wording against the cited evidence. All 15 source URLs were preserved. One substantive repair pass was used, followed by verification; no recursive review was commissioned. At most two substantive repair passes are permitted. The lead completed the final check of altered material wording and citations after integrating the editorial diff. The final private handoff records actual checks, disagreements, repairs, and remaining limitations. Publication was authorized on 20 October 2026; unfinished checks are not described as complete.

## Correction history

v0.1: provisional questions. v1.0: criteria frozen after initial inspection with the three disclosed clarifications. Initial manuscript and lead ledger prepared for sources-first substantive scrutiny. Substantive repair pass 1: separated the M4 observation citation from the M3 table citation; preserved the main_b heading as H4’s strongest alternative and aligned statement/status; added the Anthropic extraction containing the cited monitor/grader results. H3 now says the meta-rubric specifies criteria, avoiding an implication of automatic runtime validation. No audit criterion changed.

Editorial integration: nine readability edits accepted; no new finding, stronger claim, wider scope, citation change, or criteria revision. The final note is 939 words including headings and citation labels. Build/link/JSON and repository checks are recorded in the local review handoff, separately from substantive evidence.