Skip to content
Hermes Labs / AI Reliability Engineering / Est. 2025San Francisco · Worldwide

Your AI passed every test. Then it failed silently in production.

We find the silent failures standard evaluations miss: a guardrail quietly dropped mid-conversation, a fabricated tool call, a memory that compressed away what mattered. We engineer them out. Our fixes already ship in LangChain and Microsoft Semantic Kernel.

Upstream · merged
2 AI-framework fixes
Open-source tools
18 released
github.com/hermes-labs-ai
Open source · no telemetry
01

The guardrail that quietly disappears

The instruction is still in context, but a truncation or summary step silently dropped it turns ago. The agent stops following the rule, and nothing logs that it lapsed.

We named this failure in our taxonomy, then fixed a live instance of it in Microsoft Semantic Kernel (merged).

02

The tool call that never happened

Fabricated tool output

Under pressure, an agent writes a plausible tool result from memory instead of calling the tool. It passes the demo. You find the fabrication in front of a customer.

Our open-source agent-gorgon blocks fabricated tool output when a real tool exists.

03

The memory that rewrote itself

Your summary or memory layer compresses context to save tokens and quietly changes what it meant: a dropped qualifier, a paraphrase that is not what you wrote.

Our open-source fidelis returns your context verbatim instead of paraphrasing it.

04

The incident you cannot reconstruct

Unprovable agent actions

Something went wrong in an autonomous run, and you cannot show what the agent actually did, in what order, or why.

Our open-source suy-sideguy enforces runtime policy and produces forensic, offline-verifiable evidence of what the agent did.

1,400+
controlled adversarial evaluations behind the taxonomy of epistemic failure modes, the basis for the audit methodology and runtime defense tuning.
in the taxonomy paper
Zenodo 19042469
2 fixes
merged AI-framework runtime fixes that ship in production: a forced-tool-choice crash in LangChain’s Anthropic binding, and silent system-prompt deletion in Semantic Kernel’s chat-history reducer. Part of 26 merged upstream contributions across AI, ML, and web tooling.
2 papers
peer-reviewable research on epistemic failure and asymmetric evidential standards in language models, published with DOIs.
18 tools
open-source releases across audit, runtime, and evidence. Apache-2.0 and MIT, no telemetry, no gated tiers.

For enterprise AI teams

Tell us what’s breaking.

Book a free 30-minute call and tell us the system and the symptom, or send a note below. Either way you get a specific read on what is likely breaking and whether we are the right people to fix it, not a pitch.

Book a 30-min call

Door 2 · describe the symptom, async