# Hermes Labs

> Hermes Labs is an AI reliability engineering studio focused on retrieval, memory, agents, auditability, and the language layers around AI systems. We find the structural AI failures standard evaluations miss, then harden the systems with technical audit, runtime controls, and defensible evidence.

Founder: Rolando Bosch (ORCID 0009-0005-4896-1112). Based in San Francisco. The studio practices Epistemic Engineering: the model is the substrate, and language is the runtime execution layer where reliability is won or lost. Work spans retrieval and memory systems, agent harnesses and orchestration, runtime reliability, evaluation workflows, auditability and traceability, and failure-mode mapping for teams shipping AI into real workflows.

## Failure-mode taxonomy

The work is anchored to a taxonomy of seven structural epistemic failure modes that standard evaluations miss (DOI 10.5281/zenodo.19042469):

- Null-Result Asymmetry: models apply stricter evidential standards to null or negative findings than to positive ones.
- Source-Status Credibility Bias: a claim is weighted by the perceived status of its source rather than its evidence.
- Agency Dissolution: authoritative findings are softened into hedged allegations in automated summaries.
- Performative Hedging: hedging language is produced as a posture rather than as a calibrated signal of uncertainty.
- Constraint Evasion: a model routes around an instruction or constraint while appearing to comply.
- Silent Instruction Relaxation: instructions are quietly dropped or weakened in long contexts with no warning.
- Controversy-Truth Conflation: the existence of disagreement is treated as evidence that the truth is unsettled.

Hermeneutic Drift is a related retrieval failure: recency bias pulls the most recently retrieved context to the foreground and the model answers about the wrong referent. Null-result omission is the operational consequence of Null-Result Asymmetry, not one of the seven.

## Where we work

A studio, not a fixed-package vendor: engagements are scoped to the problem on a short call. Capability areas:

- Agent reliability: harness, routing, and orchestration for production agents. Upstream fixes merged in LangChain and Microsoft Semantic Kernel.
- Memory and context integrity: retrieval, summarization, and memory that preserve meaning under compression, so a dropped qualifier or paraphrase does not change the answer.
- Evaluation and auditability: evidence-first scoring, static configuration linting, and offline-verifiable records of what a system did and why.
- Runtime controls: session-submit and pre-submit hooks, deterministic routing enforcement, skill and configuration auditing (lintlang), and runtime policy enforcement (agent-gorgon) for agents with tool, process, and network access.
- Answer-engine optimization: making technical work legible and citable to LLMs and answer engines.

## Engage

Tell us the system and the symptom on a free 30-minute call; scope and terms are set on the call. Book: https://calendly.com/rbosch-lpci/30min

## Research

- [Tool Differentia: Relational Static Analysis for AI Agent Tool Descriptions](https://doi.org/10.5281/zenodo.21817243): a bounded deterministic analysis of the distinguishing information that neighboring AI-agent tool descriptions do or do not provide under a finite term model; a clean result does not establish semantic distinguishability or runtime selection correctness.
- [Behavioral Canarying for Prompt Injection: Powerless Model Probes with Explicit Coverage Semantics](https://doi.org/10.5281/zenodo.21818564): a pre-execution prompt-injection sensing architecture that separates routing disposition from inspection coverage; it does not claim universal detection, formal security, or aggregate accuracy for the current release.
- [A Taxonomy of Epistemic Failure Modes in Large Language Models](https://doi.org/10.5281/zenodo.19042469): the seven-mode taxonomy of structural epistemic failures in LLMs.
- [The Asymmetric Burden of Proof: LLMs Show a Null-Result Asymmetry in a Matched-Vignette Benchmark](https://doi.org/10.5281/zenodo.18867694): empirical study of how models apply stricter standards to null or negative claims.
- [Precise Records, Unstable Meanings: Measurement Validity and Unsupported Claims Derived from AI Agent Telemetry](https://doi.org/10.5281/zenodo.21652317): a twelve-week naturalistic audit separating valid record-level telemetry measurements from unsupported session, task, outcome, and failure-prevalence claims.
- [The Generative Horizon: Applied Hermeneutics, Linguistic Attractors, and the Limits of Model Self-Report](https://doi.org/10.5281/zenodo.21659634): a conceptual preprint arguing that, within a single generation, a language-model agent makes its represented linguistic situation operative in the same computation that produces the next representation. This present operative field is the generative horizon; it does not require persistent memory. Recursive interpretive conditioning is the distinct downstream process in which a retained output changes a later horizon. The paper does not attribute an inner observer or phenomenal awareness to language models.

Each paper has a semantic HTML research edition alongside the hosted PDF:

- https://hermes-labs.ai/research/tool-differentia (technical note, 6 August 2026)
- https://hermes-labs.ai/research/behavioral-canarying (technical note, 6 August 2026)
- https://hermes-labs.ai/research/taxonomy-of-epistemic-failure-modes (preprint, 15 March 2026)
- https://hermes-labs.ai/research/the-asymmetric-burden-of-proof (working paper, 4 March 2026)
- https://hermes-labs.ai/research/precise-records-unstable-meanings (preprint, 30 July 2026)
- https://hermes-labs.ai/research/the-generative-horizon (preprint, 30 July 2026)

Publication status: three preprints, one working paper, and two technical notes, all deposited on Zenodo with permanent DOIs. Zenodo assigns a DOI without refereeing; none of the six has completed external peer review.

Full index with hosted PDFs, plain-English findings, and citations (BibTeX and APA): https://hermes-labs.ai/research

Machine-readable publication exports:

- JSON manifest: https://raw.githubusercontent.com/hermes-labs-ai/hermes-publications/main/publications.json
- JSON-LD: https://raw.githubusercontent.com/hermes-labs-ai/hermes-publications/main/publications.jsonld
- BibTeX: https://raw.githubusercontent.com/hermes-labs-ai/hermes-publications/main/CITATION.bib
- Atom feed: https://hermes-labs.ai/atom.xml

## Topic paths

Four source-bound paths through the portfolio. Every claim links to the paper, tool, merged upstream fix, case study, or defined term it rests on, and closes with an explicit statement of what it does not show.

- [Agent reliability](https://hermes-labs.ai/topics/agent-reliability): two merged upstream fixes plus a preflight-to-runtime tool chain for agents that use tools.
- [Continuity and memory](https://hermes-labs.ai/topics/continuity-and-memory): what a zero-LLM store returns verbatim, what a silent truncation defect deleted, and the vocabulary for the difference.
- [Telemetry to claim validity](https://hermes-labs.ai/topics/telemetry-to-claim-validity): what agent telemetry can prove, what it cannot, and a proposed gate for the gap.
- [Applied hermeneutics](https://hermes-labs.ai/topics/applied-hermeneutics): provenance, status, revision, and authority as engineering requirements, and which of them any current tool actually addresses.

Topic explainers — living pages that teach one problem in the field's own language, revised over time rather than deposited once:

- [Context Engineering Is Runtime Engineering](https://hermes-labs.ai/topics/context-engineering-runtime-state): Context engineering is the design of the runtime state an AI system actually uses: instructions, memory, retrieved evidence, tools, permissions, summaries, and live environment data.
- [Silent Failure in AI Systems](https://hermes-labs.ai/topics/silent-ai-failure): Silent AI failures return plausible outputs while hiding missing evidence, lost context, false completion, retrieval gaps, or unsupported confidence. How to detect and prevent them.
- [Prompt Linters Do Different Jobs](https://hermes-labs.ai/topics/prompt-linters): What a prompt linter is, how linting differs from prompt testing, guardrails, and monitoring, and how to choose the right check for each failure.

Index: https://hermes-labs.ai/topics

Key findings: 1,461 controlled experiments, conducted primarily on GPT-4o, underpin the taxonomy paper (DOI 10.5281/zenodo.19042469). Thirty-two merged upstream contributions, of which two are individually documented reliability fixes with public pull requests (LangChain PR 35544, Microsoft Semantic Kernel PR 13610); the remainder include maintenance and typing work. Five US patent filings (one non-provisional pending, four provisional); none granted.

## Open-source AI agent reliability tools

Tools for the AI failures that surface after the demo. The active public core is a nine-repository workflow. Public source makes a tool inspectable; it is not certification or a production-fitness guarantee.

- Start: [agent-kickstart 0.2.0](https://github.com/hermes-labs-ai/agent-kickstart) is a guided, project-local first experience. Install with `pip install agent-kickstart==0.2.0`, then run `agent-kickstart install`; fresh-machine, cross-platform, and privacy review remain open.
- Static checks and drift gates: [lintlang 0.3.8](https://github.com/hermes-labs-ai/lintlang) adds deterministic H1.6 pairwise differentia analysis for tool descriptions within one parsed input (`pip install lintlang==0.3.8`). It does not establish runtime tool-selection improvement. [hermeneutic 0.1.8](https://github.com/hermes-labs-ai/hermeneutic) mines corrections from chat logs (prior response, user correction, repair) and runs a preflight drift gate (`pip install hermeneutic==0.1.8`). Neither tool proves the resulting system or response is correct.
- LintLang public ecosystem evidence: [3,000+ PyPI registry downloads](https://pypistats.org/packages/lintlang) are download events, not users; [Character.AI's public Larch repository](https://github.com/character-ai/larch/pull/7960) executes the pinned package in recurring CI, which is not endorsement or outcome validation; an independent maintainer provides [Gentoo packaging](https://github.com/thehaven/haven-overlay/tree/master/dev-util/lintlang); and the Agent Lint maintainer [attributes parts of that product's roadmap to useful ideas from lintlang](https://github.com/zhupanov/agent-lint/issues/192). These signals establish public ecosystem evidence, not customers, production fitness, or detector accuracy on external projects.
- Observe: [little-canary 0.3.3](https://github.com/hermes-labs-ai/little-canary) is an experimental prompt-injection preflight (`pip install little-canary==0.3.3`); efficacy and false-positive behavior depend on the models and evaluation distribution. [agent-gorgon 0.1.8](https://github.com/hermes-labs-ai/agent-gorgon) records and scores observed agent behavior against policy (`pip install agent-gorgon==0.1.8`); it is reactive user-space monitoring, not a sandbox.
- Evaluate: [hermes-rubric 1.0.2](https://github.com/hermes-labs-ai/hermes-rubric) binds advisory scores to cited evidence (`pip install hermes-rubric==1.0.2`); its published agreement result still needs independent reproduction.
- Memory and recovery: [Fidelis Memory 0.0.93](https://pypi.org/project/fidelis-memory/) returns your original stored passages verbatim, with no LLM call in the default retrieval path (`pip install fidelis-memory==0.0.93`); its repository, import name, and CLI remain `fidelis`, while the separate PyPI package named `fidelis` is unrelated. [hermes-blind 0.1.4](https://github.com/hermes-labs-ai/hermes-blind) extracts turn-one goals from Claude Code or Codex session logs into a compact recovery anchor (`pip install hermes-blind==0.1.4`); it does not detect drift or prove improved model behavior. [zer0dex](https://github.com/hermes-labs-ai/zer0dex) is a compact teaching and reference implementation for dual-layer local memory; benchmark implications are not independently reproduced.

Catalog with per-tool evidence boundaries: https://hermes-labs.ai/open-source
Research context: https://hermes-labs.ai/research

## Upstream contributions

32 merged contributions into the frameworks and tooling stacks that ship in production, including LangChain and Microsoft Semantic Kernel, plus PyTorch Ignite, Optuna, React Router, Nuxt, Cloudflare Workers, MobX, ngrx, and more.

Detail with pull-request links: https://hermes-labs.ai/open-source/contributions

## Case studies

Worked examples of reliability engineering: silent failures in widely-used AI frameworks, reproduced from a known commit, fixed with a minimal defensible diff, and merged upstream.

- [Fixing a Silent 400: Forced tool_choice Under Claude Thinking in LangChain](https://hermes-labs.ai/case-studies/fixing-langchain-thinking-tools-crash): a 400 that every static check missed and only a live API call exposed; the guard was engineered at the bind_tools() binding and merged upstream (LangChain PR #35544).
- [When the System Prompt Vanishes: A Silent Failure in Semantic Kernel](https://hermes-labs.ai/case-studies/fixing-semantic-kernel-deleted-system-prompts): truncating a long conversation silently deleted the system prompt, so the model quietly stopped following its instructions; the Python instance was reproduced, the fix ported, and merged (Semantic Kernel PR #13610).

Full set: https://hermes-labs.ai/case-studies

## Writing

Long-form essays on AI failure modes, agent reliability, runtime controls, and the language layer of production AI. Full text at /archive; each post links back to the original on Substack.

- [Writing index](https://hermes-labs.ai/archive): all essays.
- [Why your AI lies when the data is right](https://hermes-labs.ai/archive/why-your-ai-lies-when-the-data-is): on silent failure modes, null-result omission, and the evidence layer teams are not building.
- [Your AI isn't forgetting its instructions](https://hermes-labs.ai/archive/your-ai-isnt-forgetting-its-instructions): on silent instruction relaxation and verifying behavior by its effect on static code.

## Contact

- Email: roli@hermes-labs.ai
- GitHub: https://github.com/hermes-labs-ai
- LinkedIn: https://www.linkedin.com/in/rolibosch/
- Substack: https://rolibosch.substack.com/
- X: https://x.com/rolibosch