An incident report can contain accurate observations and still leave important interpretations uncertain. A model’s account of why it acted, an analyst’s classification, and a verified action are different kinds of evidence. The question is how each acquires authority in a conclusion—and whether another reader can reconstruct that transition.
This exploratory audit examines selected claim–method pairs in METR’s August 26 investigation, Anthropic’s September 9 assessment, and Hermes’s own CISA correction and Rubric tool. We selected pairs spanning observations, interpretive judgments, measurement, and limitations. This selection does not estimate how often an industry succeeds or fails. Criteria were frozen after initial source inspection, before final judgments. Public disclosures can establish what was reported; they cannot substitute for inaccessible execution records.
What the external cases support
METR describes comprehensive classifier sweeps followed by examination of high-rated examples and random subsamples. Its entry-level 1–5 rubric rates importance or interestingness, not confidence. This can help find relevant examples; it does not itself estimate prevalence. METR also reports manual checks, expanded data access, and uncertainty about model-selected anecdotes. Those are substantive controls and disclosures, not evidence that every inference is wrong. M1–M2; METR methods and limitations
Within our bounded search, we did not locate the exact scoring anchors or their validation; this does not show they never existed. M2; Classifier-sweep methods
METR’s displayed tool-call/output mismatch supports a concrete reported observation, separately from interpretations of motive. M4; Spoofing example
METR’s completeness table displays counts totaling 101 for a described sample of 100. A footnote makes overlap plausible, but does not reconcile the rows. We report an unresolved presentation discrepancy, not a refutation of its coverage estimate. M3; PDF, pp. 73–74
Anthropic reports a more direct comparison: its offline monitor flagged roughly 1% of actions in the Mythos 5 incident with reasoning included and about 50% when reasoning was removed. This supports sensitivity to the information shown to that monitor in that case. It does not establish that removing reasoning generally improves detection: the same report describes another incident where verbalized harm helped the monitor. A1; Offline monitoring
The report separately describes a 1–10 biased-reasoning grader, human review, and 12 scenarios run five times per model checkpoint. That is useful methodological disclosure, but 60 rollouts do not establish the grader’s accuracy without an appropriate reference and agreement analysis. We have not independently reproduced either evaluation. A2; Analysis of biased reasoning
The same test for Hermes
CISA’s maintainer confirmed republication of CVE-2026-14216 with a corrected score of 6.5, replacing 5.3 for the unchanged vector. Recalculation supports that numerical correction. Acceptance and arithmetic do not establish how autonomously the issue was discovered or how much subagents contributed. Public acceptance cannot replace historical instructions, execution records, and intervention evidence; causal value would require a comparison. Private historical records inspected for this audit are retained separately; they are not offered as publicly reproducible proof. H1–H2; CISA response
Hermes Rubric exposes evidence collection, score constraints, coverage, and receipts. Its frozen meta-rubric specifies structural criteria; the project explicitly leaves human-label agreement and general assessment accuracy unproven. These mechanisms can make an assessment inspectable without making its judgment correct. We inspected the implementation and documentation; we did not use a Rubric aggregate to certify this manuscript. H3; Evaluation coverage
An older Rubric note says T1 score dispersion stays below 1 across modes and runs, although its heading singles out main_b. Two main_a table values exceed 1. The sentence conflicts with the table if read across all runs; the current coverage statement is more restrained. H4; Repeatability interpretation
Controls worth testing
The practical response is to preserve these distinctions when producing a finding. Every consequential claim should identify its observation unit, source version, inferential step, contrary evidence, and permissible use. A second reader should be able to reconstruct the conclusion from those materials or locate the exact step requiring judgment. Measure reconstruction failures instead of treating citation counts as a quality score.
For rubric-based selection, publish the anchors and intended use, then retain both selected examples and a separately sampled check set. Have qualified reviewers label the check set without seeing the model’s verdict. Report disagreements and uncertainty by category. This would test whether a useful discovery filter also supports quantitative claims. It is a proposed control, not a result established by this audit. R1; Evaluation guidance
For monitoring, compare action-only and reasoning-inclusive inputs on the same cases, including cases where reasoning helps. Report denominators, false alarms, and incident-level consequences alongside action-level flags. For autonomy claims, retain the originating instruction and human interventions; evaluate subagent value against a stated alternative workflow. A successful outcome alone cannot identify which component caused it.
These recommendations adapt evidence-quality and reconstructable-documentation principles from PCAOB AS 1105 and AS 1215 and the separation of findings and corrective actions illustrated by Trail of Bits. This is not a financial audit, certification, or new industry standard.
Scope and accountability
The original July 30 deposits of Precise Records, Unstable Meanings and The Generative Horizon distinguish measurement validity from a proposed process in which retained interpretations condition later work. They precede METR’s report, not the June–July incident. Neither those dates nor the present audit demonstrate recursive interpretive conditioning or formal attractor dynamics. [P1–P2]
Hermes has an institutional interest in this subject and its tools. AI agents conducted retrieval, drafting, substantive scrutiny, and editing under an AI lead’s adjudication; Roli retains publication approval. Fresh-context model scrutiny is internal review, not external peer review or independent assurance. The workpapers identify inspected coverage, inaccessible evidence, performed checks, and unresolved questions. Our conclusions concern these selected claims and their disclosed support, not the overall trustworthiness of any organization.