Skip to content
Hermes Labs / AI Reliability Engineering / Est. 2025

Your AI passed every test. Then it failed silently in production.

We find the silent failures standard evaluations miss: a guardrail quietly dropped mid-conversation, a fabricated tool call, a memory that compressed away what mattered. We engineer them out. 4 of our fixes are merged into LangChain, Microsoft Semantic Kernel, and DSPy.

Free 20-minute scoping call · No pitch, no obligation

01

A guardrail disappears mid-conversation

The instruction is still configured, but a truncation or summary step silently dropped it from the effective context sent to the model several turns ago. The agent stops following the rule, and nothing logs that it lapsed.

Fix · mergedWe named the model-behavior failure in our taxonomy, then fixed a framework defect with the same operational failure shape in Microsoft Semantic Kernel. How we found it.

02

The same mistake returns

Corrections that do not persist

A user corrects the system, but the correction disappears into a long chat log. The next response repeats the same drift as if the repair never happened.

ToolOur open-source hermeneutic mines correction triples from chat logs and gates the next response before the same drift ships twice. Explore the research.

03

Compression changes the record

Your summary or memory layer compresses context to save tokens and quietly changes what it meant: a dropped qualifier, a paraphrase that is not what you wrote.

ToolOur open-source Fidelis Memory returns your context verbatim instead of paraphrasing it. Explore the research.

04

The run leaves no usable audit trail

Unprovable agent actions

Something went wrong in an autonomous run, and you cannot show what the agent actually did, in what order, or why.

ToolOur open-source agent-gorgon makes deterministic runtime control decisions and records forensic, offline-verifiable evidence of what the agent did. Explore the research.

1,461
controlled experiments, conducted primarily on GPT-4o, behind the taxonomy of epistemic failure modes. The taxonomy names the failure classes the audit methodology looks for.
in the taxonomy paper
Zenodo 19042468
4 fixes
merged AI/framework fixes: forced tool choice under Anthropic thinking in LangChain, system-prompt preservation and duplicate-null schema handling in Semantic Kernel, and empty-devset validation in DSPy. Part of 48 merged upstream code and documentation contributions across AI, ML, and web tooling.
6 papers
on Zenodo with permanent DOIs: three preprints, one working paper, and two technical notes. Their distinct evidence roles include a seven-mode corpus taxonomy, a matched-vignette experiment, a twelve-week telemetry audit, a conceptual paper on model self-report, a bounded static analysis of tool descriptions, and a prompt-injection sensing architecture.
19-project catalog
a curated public path from project setup and preflight checks to runtime observation, evaluation, and local memory. Each tool names its present evidence boundary.
Case study
a falsification-first diagnosis of a closed-source Claude Code resume hang, with measured cause classes and the evidence boundary stated explicitly.

For enterprise AI teams

Tell us what’s breaking.

Book a free 20-minute call and tell us the system and the symptom, or send a note below. Either way you get a specific read on what the symptom points to, or what we would need to see before naming a cause, and whether we are the right people to fix it, not a pitch.

Book a 20-min call

Or send a note · describe the symptom