Less history.
Missing instruction.
System / developer message removedRecent conversation retained
The reducer could remove the instruction that was meant to govern the conversation.
Hermes Labs / AI reliability engineeringEst. 2025
Find the failing layer
We find where AI agents and the systems they run on fail before production.
Not every agent failure is an AI failure. Sometimes it’s a system failure. Prompts, history reducers, tool bindings and evaluation code can change what an agent receives or does. We investigate the failing path, from model behavior to the systems around it.
We reproduce failures, identify the layer the evidence supports, and engineer fixes with regression evidence. Our public work includes maintainer-merged fixes in Microsoft Semantic Kernel, LangChain and DSPy, a vulnerability-record correction acknowledged by CISA, and open-source tools and research you can inspect.
Discuss your system → (Calendly, opens in a new tab)See what you get →
Paid diagnosis & engineering · Scope agreed before work starts
Public upstream work · Merged 2026-03-19
The system prompt disappeared.
- Before
- History truncation could delete the system or developer message.
- After
- The first system or developer message is preserved during truncation.
This exhibit covers the truncation fix. It does not establish deployment or operational use.
Read the case and limits Microsoft Semantic Kernel PR #13610Public upstream work · Merged 2026-03-05
Tool binding met extended thinking.
- Before
- A forced tool choice with Claude extended thinking caused a request-time 400.
- After
- A 20-line guard drops the forced choice and warns. Two test functions cover eight scenarios.
Root cause credited to the community reporter. Tool calls are no longer guaranteed on those requests; auto and thinking-off bindings are unchanged.
Read the case and limits LangChain PR #35544Public upstream work · Merged 2026-08-26
No examples. A misleading crash.
- Before
- An empty devset caused a ZeroDivisionError inside evaluation logging.
- After
- Evaluation raises a ValueError naming the empty devset, with a regression test.
No score is invented for an evaluation that did not run. Merge evidence does not identify an installed DSPy version.
Read the case and limits DSPy PR #9978Public vulnerability record · Corrected
Same vector. Correct calculation.
- Before
- CVE-2026-14216 recorded a CVSS 3.1 score of 5.3.
- After
- CISA republished it as 6.5. The vector and Medium severity did not change.
An autonomous audit found the mismatch. This is one record correction, not a new vulnerability or an error rate.
Read the case and limits CISA acknowledgement on issue #333One AI fix / Microsoft Semantic KernelSystem prompt deleted First system message preserved.Public upstream work · Maintainer-merged PR #13610
Inspect the before & afterInspect the CISA evidence and audit scope
We found a scoring error in a public vulnerability record. CISA acknowledged it and corrected the record.
Error. The CISA enrichment for CVE-2026-14216 stored a CVSS score of 5.3 for a CVSS 3.1 vector that calculates to 6.5.
Fix. CISA republished the score as 6.5; the vector and the Medium severity did not change.
How. A Hermes Labs autonomous system audited 5,324 recent CISA ADP records, including 1,060 CVSS 3.x scores checked against their vectors, reproduced the one mismatch and prepared the report.
CISA's reply on issue #333 · CVE correction commit
Related engineering: Making LangChain’s agent tool binding compatible with Claude’s extended thinking · PR #35544 · merged 2026-03-05
Inside the work / Microsoft Semantic Kernel
The conversation kept going.
The system prompt didn’t.
History truncation could delete the system or developer message. We reproduced the deletion, ported Microsoft’s documented .NET behavior, and added regression tests.
Less history.
Less history.
Instruction preserved.
First system / developer message preservedRecent conversation retained
The merged fix preserves the first system or developer message during truncation.
Public upstream workPR #13610 · Maintainer-merged, 19 March 2026
Read the fix and regression evidenceAnother layer / LangChain
Making LangChain’s agent tool binding compatible with Claude’s extended thinking
A 20-line guard handles forced tool choice when extended thinking is enabled, with two test functions covering eight scenarios. Community reporter’s root-cause credit is preserved.
These are public examples of our method. Paid work starts with your system, your symptom, and an agreed scope.
Put the lab on your problemPaid expert work
Hire the lab for
the part you need.
A written diagnosis when the cause is unclear. Engineering when the failing layer is known. Scope and terms are agreed before work starts.
01 / DiagnosisPaid
Find the failing layer.
Bring a symptom, a trace, or a reproduction. We inspect the prompts, tools, memory, and configuration on the path that fails.
You get
Explore diagnosis A written diagnosis: reproduction status, supported findings, the recommended fix and what it would touch, or the next evidence needed.
02 / EngineeringPaid
Patch the right layer.
When implementation is useful, we change the relevant layer in your Python, JavaScript, or TypeScript stack.
You get
Explore engineering A scoped change with regression evidence. These public fixes show how a specific finding becomes an inspectable engineering result.
The tools and research are free. Use the 9-project open-source catalog, papers, and public receipts without hiring us.
Hermes Labs does AI reliability engineering. We find where AI agents and the systems they run on fail before production. Founded by Rolando Bosch (Roli), it builds and maintains LintLang, static analysis for AI agent instructions, and Little Canary, prompt-injection sensing for untrusted input.
§ 01
Explore AI reliability topics →What breaks in production
01
A guardrail disappears mid-conversation
The instruction is still configured, but a truncation or summary step silently dropped it from the effective context sent to the model several turns ago. The agent stops following the rule, and nothing logs that it lapsed.
Fix · mergedWe named the model-behavior failure in our taxonomy, then fixed a framework defect with the same operational failure shape in Microsoft Semantic Kernel. How we found it.
02
The same mistake returns
Corrections that do not persist
A user corrects the system, but the correction disappears into a long chat log. The next response repeats the same drift as if the repair never happened.
ToolOur open-source hermeneutic mines correction episodes from chat logs, retrieves relevant prior guidance for similar tasks, and checks outgoing drafts against fixed English risk patterns. Explore the research.
03
Compression changes the record
Your summary or memory layer compresses context to save tokens and quietly changes what it meant: a dropped qualifier, a paraphrase that is not what you wrote.
ToolOur open-source Fidelis Memory returns your context verbatim instead of paraphrasing it. Explore the research.
04
The run leaves no usable audit trail
Unprovable agent actions
Something went wrong in an autonomous run, and you cannot show what the agent actually did, in what order, or why.
ScopeStart by naming the actions that need evidence and the records your system actually retains. Discuss the audit gap.
Diagnosis and engineering on your system are paid engagements; the tools, papers and case-study receipts are free. Match a production problem to its receipt →
Inspect the research and engineering record
§ 02
Technical record
1,461
controlled experiments, conducted primarily on GPT-4o, behind the taxonomy of epistemic failure modes. The taxonomy names the failure classes the audit methodology looks for.
in the taxonomy paper
Zenodo 19042468
Zenodo 19042468
4 fixes
merged AI/framework fixes: forced tool choice under Anthropic thinking in LangChain, system-prompt preservation and duplicate-null schema handling in Semantic Kernel, and empty-devset validation in DSPy. 105 published external pull requests merged in 2026, including 52 code, workflow, typing or maintenance contributions across open-source infrastructure.
6 papers
on Zenodo with permanent DOIs: three preprints, one working paper, and two technical notes. Their distinct evidence roles include a seven-mode corpus taxonomy, a matched-vignette experiment, a twelve-week telemetry audit, a conceptual paper on model self-report, a bounded static analysis of tool descriptions, and a prompt-injection sensing architecture.
9-project catalog
a curated public path from project setup and preflight checks to runtime observation, evaluation, and local memory. Each tool names its present evidence boundary.
Case study
a falsification-first diagnosis of a closed-source Claude Code resume hang, with measured cause classes and the evidence boundary stated explicitly.
§ 03
Discuss your system
For enterprise AI teams
Tell us what’s breaking.
Tell us the system and the symptom. Send a note below or discuss it with us. We will say what the symptom points to, or what we would need to see before naming a cause. Scope and terms are agreed before paid diagnosis or engineering starts.
Discuss your system (Calendly, opens in a new tab) →