Less history.
Missing instruction.
System / developer message removedRecent conversation retained
The reducer could remove the instruction that was meant to govern the conversation.
Hermes Labs / AI reliability engineeringEst. 2025
Work with the lab
A diagnosis your
engineers can use.
Not every agent failure is an AI failure. Sometimes it’s a system failure. Prompts, history reducers, tool bindings and evaluation code can change what an agent receives or does. We investigate the failing path, from model behavior to the systems around it.
We reproduce failures, identify the layer the evidence supports, and engineer fixes with regression evidence. Our public work includes maintainer-merged fixes in Microsoft Semantic Kernel, LangChain and DSPy, a vulnerability-record correction acknowledged by CISA, and open-source tools and research you can inspect.
Discuss your system → (Calendly, opens in a new tab)See what you get ↓
Paid diagnosis & engineering · Scope agreed before work starts
Public upstream work · Merged 2026-03-19
The system prompt disappeared.
- Before
- History truncation could delete the system or developer message.
- After
- The first system or developer message is preserved during truncation.
This exhibit covers the truncation fix. It does not establish deployment or operational use.
Read the case and limits Microsoft Semantic Kernel PR #13610Public upstream work · Merged 2026-03-05
Tool binding met extended thinking.
- Before
- A forced tool choice with Claude extended thinking caused a request-time 400.
- After
- A 20-line guard drops the forced choice and warns. Two test functions cover eight scenarios.
Root cause credited to the community reporter. Tool calls are no longer guaranteed on those requests; auto and thinking-off bindings are unchanged.
Read the case and limits LangChain PR #35544Public upstream work · Merged 2026-08-26
No examples. A misleading crash.
- Before
- An empty devset caused a ZeroDivisionError inside evaluation logging.
- After
- Evaluation raises a ValueError naming the empty devset, with a regression test.
No score is invented for an evaluation that did not run. Merge evidence does not identify an installed DSPy version.
Read the case and limits DSPy PR #9978Public vulnerability record · Corrected
Same vector. Correct calculation.
- Before
- CVE-2026-14216 recorded a CVSS 3.1 score of 5.3.
- After
- CISA republished it as 6.5. The vector and Medium severity did not change.
An autonomous audit found the mismatch. This is one record correction, not a new vulnerability or an error rate.
Read the case and limits CISA acknowledgement on issue #333One AI fix / Microsoft Semantic KernelSystem prompt deleted First system message preserved.Public upstream work · Maintainer-merged PR #13610
Inspect the before & afterInspect the CISA evidence and audit scope
We found a scoring error in a public vulnerability record. CISA acknowledged it and corrected the record.
Error. The CISA enrichment for CVE-2026-14216 stored a CVSS score of 5.3 for a CVSS 3.1 vector that calculates to 6.5.
Fix. CISA republished the score as 6.5; the vector and the Medium severity did not change.
How. A Hermes Labs autonomous system audited 5,324 recent CISA ADP records, including 1,060 CVSS 3.x scores checked against their vectors, reproduced the one mismatch and prepared the report.
CISA's reply on issue #333 · CVE correction commit
Related engineering: Making LangChain’s agent tool binding compatible with Claude’s extended thinking · PR #35544 · merged 2026-03-05
Inside the work / Microsoft Semantic Kernel
The conversation kept going.
The system prompt didn’t.
History truncation could delete the system or developer message. We reproduced the deletion, ported Microsoft’s documented .NET behavior, and added regression tests.
Less history.
Less history.
Instruction preserved.
First system / developer message preservedRecent conversation retained
The merged fix preserves the first system or developer message during truncation.
Public upstream workPR #13610 · Maintainer-merged, 19 March 2026
Read the fix and regression evidenceAnother layer / LangChain
Making LangChain’s agent tool binding compatible with Claude’s extended thinking
A 20-line guard handles forced tool choice when extended thinking is enabled, with two test functions covering eight scenarios. Community reporter’s root-cause credit is preserved.
These are public examples of our method. Paid work starts with your system, your symptom, and an agreed scope.
Put the lab on your problemPaid expert work
Hire the lab for
the part you need.
A written diagnosis when the cause is unclear. Engineering when the failing layer is known. Scope and terms are agreed before work starts.
01 / DiagnosisPaid
Find the failing layer.
Bring a symptom, a trace, or a reproduction. We inspect the prompts, tools, memory, and configuration on the path that fails.
You get
Discuss a diagnosis (Calendly, opens in a new tab)A written diagnosis: reproduction status, supported findings, the recommended fix and what it would touch, or the next evidence needed.
02 / EngineeringPaid
Patch the right layer.
When implementation is useful, we change the relevant layer in your Python, JavaScript, or TypeScript stack.
You get
Discuss engineering (Calendly, opens in a new tab)A scoped change with regression evidence. These public fixes show how a specific finding becomes an inspectable engineering result.
Agent reliability engineering · Memory and context integrity · Evaluation and auditability · Runtime controls
The tools and research are free. Use the 9-project open-source catalog, papers, and public receipts without hiring us.
Find your starting point
Choose the symptom closest to yours. Inspect the public work, then discuss a diagnosis, or use a free tool where one fits.
Problem, proof, next step
01
A published record contradicts its own inputs
Paid diagnosis
02
The model stops following its system prompt in long conversations
Paid diagnosis
03
Enabling a model feature breaks your agent's tool calls
Paid diagnosis
04
An evaluation fails with the wrong error, or reports a result you cannot trust
Paid diagnosis
05
Generated tool schemas come out malformed
Paid diagnosis
06
Retrieval scores look plausible but rank the wrong memories
Paid diagnosis
07
A closed-source AI tool fails and you cannot see inside it
Paid diagnosis
08
Instruction and tool-definition defects you want caught before merge
Free · self-serve
09
Untrusted input reaching an agent before it acts
Free · self-serve
What to bring and what comes back
§ S · 03
What the handoff looks like
Scope is set on the call. The handoff does not change: what you send us, what we send back, and what we say when the symptom will not reproduce.
Bring
One system, one symptom, and the artifacts around it.
Pick a single system and the path through it that misbehaves. Tell us the behavior you expected and the behavior you observed. Send a reproduction if you have one, or the nearest failing trace, log, or session record if you do not. Include the prompts, tool descriptions, memory or retrieval configuration, and settings that sit on that path.
Please don’t include credentials, personal data, or unredacted production secrets; describe the symptom and share redacted evidence first.
Receive
A written diagnosis your engineers can act on.
The write-up states whether we reproduced the symptom and what the evidence supports. When the evidence supports a finding, it identifies which layer is failing, the fix we recommend, and what implementing that fix would touch. Otherwise, it names the next evidence required. It is written to be read by an engineer who was not on the call.
If not
If we cannot reproduce it, we say so.
Some symptoms do not reproduce from what is available, and some evidence is too thin to place a cause. We say that plainly, state what the evidence does and does not support, and name the next trace, log, or configuration we would need. We do not turn an unreproduced symptom into a confident finding.
§ S · 04
Start
Tell us the system and the symptom. We will tell you what the symptom points to, or what we would need to see before naming a cause, and whether we are the right people to fix it.
Data and partnership inquiries. Bring the data you are proposing, the use case it would support, and the partnership shape you have in mind, on the 20-minute call or by email at roli@hermes-labs.ai. Hermes will say directly whether it fits and what it would need to see next.