Structured rubrics
Synthesizes task-specific criteria or accepts frozen and bundled rubrics.
Hermes Rubric 1.2.2 · Python 3.10+ · Apache-2.0 published-release licenseOpen source · inspectable boundary
Evidence-first assessment
Assess agent outputs against cited evidence.
Hermes Rubric builds structured rubrics, binds scores to cited evidence, reports coverage facts, and leaves the next action to the caller.
$ uvx --from hermes-rubric==1.2.2 hermes-rubric --versionRelease
v1.2.2
Published on PyPI
Runtime
Python 3.10+
Apache-2.0 published-release license
First run
One pinned command
No project files required
Scope
Bounded tool
Not a system-level guarantee
§ 01
The problem
Why this exists
Make the evidence boundary visible.
A score cannot support an assessment when its criteria, cited evidence, or coverage are unclear. Hermes Rubric assembles those materials into an inspectable result while leaving the next action to the caller.
§ 02
Run one first assessment
Score a two-file bundle — a change review and the staging record filed beside it — then read what the assessment cited, which criteria it hedged, and how much of the bundle it read.
Prerequisite. A model backend, and there is no model-free path. The CLI auto-detects only the Claude Code `claude` command or a local Ollama endpoint; with neither present it exits 1 with `ERROR: No backend available. Install Claude Code (claude CLI) or Ollama with a qwen3.5 model.` Rubric synthesis, evidence collection and scoring are all model calls.
INPUTbundle/review.md — the claim being assessed
The migration script backfills the status column for every order row. It was run once against a staging copy of the orders table.
INPUTbundle/evidence.md — the record filed beside it
Staging run log: 12,004 of 12,004 rows updated, 0 errors. No production run has been performed. No rollback script exists yet.
STEP 01Check the backend first
$ uvx --from hermes-rubric==1.2.2 hermes-rubric --version
The pinned version string. This confirms installation only; it does not confirm that a scoring backend is reachable.
STEP 02Assess the bundle
$ uvx --from hermes-rubric==1.2.2 hermes-rubric --intent "Assess whether the change review is supported by the evidence in this bundle." --context bundle/review.md --target bundle --target-type review-bundle --out assessment.json --verbose
The three stages on stderr — rubric synthesis, evidence collection, scoring — then a JSON result written to assessment.json. The target is the directory, so evidence collection reads both files and the result's coverage reports 2 of 2 sources. With no backend reachable the run stops at the backend error above and writes nothing.
rubric.dimensionsThe criteria this run synthesized. They are model output, so they are not fixed between runs.
evidence_citations[]Per dimension: the quoted span, its evidence id, its source_class, and whether evidence was found at all. The location field marks the excerpt, not which bundle file the quote came from.
per_dim_scores[]score_rationale, the citation that drove the score, and hedge_applied where the evidence was thin.
hedge_dims and hedge_noteThe dimensions the run itself says are less reliable. Read these before the aggregate.
coveragestatus, considered_sources against total_sources, visible_bytes against total_bytes, and any limitations, so a file the loader skipped is visible rather than silent.
receipttool_version, backend, input hashes and the rubric hash, which is what makes a later comparable re-grade possible.
aggregate and max_possibleA weighted signal over the dimensions above, meaningful only with those dimensions and their hedges in view.
What this result does not establish. `--context` shapes rubric synthesis only; evidence collection reads what `--target` loads, so a file outside the target path is never scored or cited. Every stage of this pipeline is a model call, so the criteria, the citations and the number differ between runs and between backends. A numeric score is not guaranteed: a run with no reachable backend produces an error instead. The aggregate is a signal, not a verdict, and no run here establishes that a claim is factually true.
Hermes Rubric 1.2.2 is published and maintained by Hermes Labs. Canonical repository: https://github.com/hermes-labs-ai/hermes-rubric. Package: https://pypi.org/project/hermes-rubric/.
§ 03
What it does
Cited scoring
Scores against accepted evidence and preserves evidence, coverage, and receipt data.
Caller-controlled feedback
Produces feedback without imposing a release threshold or changing a runtime.
§ 06
Related concepts
GLOSSARYLLM-as-a-judge reliabilityHow consistently model-based judges produce defensible assessments under defined conditions.GLOSSARYInspection coverageWhether the inspection layers required for a decision actually ran.GLOSSARYEpistemic EngineeringEngineering how AI systems handle evidence, uncertainty, and justification.GLOSSARYTelemetry-to-Claim GateA proposed framework for recording the evidentiary basis of a telemetry-based claim.
The boundary
Useful evidence is not a blanket guarantee.
The aggregate is a signal, not a verdict. Hermes Rubric does not impose a binary release gate or decide what an application does next.
Compare it with the other open-source tools →