Diagnose silent failures in production AI systems
Bring us a system and a failure you can see. We test whether it reproduces, trace the cause when the evidence supports one, and return in writing the evidence and either the next useful fix or the next evidence we need.
Where we work
Agents that lose instructions or misuse tools
We have shipped fixes for tool-selection failures in LangChain and system-message loss in Microsoft Semantic Kernel. Our Mem0 audit shows the same approach applied to retrieval scoring.
Evaluation and auditability
We make findings inspectable: Hermes Rubric binds scores to cited evidence, and the DSPy example below shows a misleading evaluation failure moved to the correct boundary.
Checks that run before a bad change ships
Our open-source tools include lintlang for agent configuration and Rule Audit for system prompts. One external maintainer adopted Rule Audit in both pre-commit and CI.
How we engage
Find the layer that is actually failing.
We inspect the prompts, tool descriptions, memory path, and configuration involved in the failure, then test whether the behavior reproduces. You receive in writing the symptom, what the evidence supports, and either the recommended fix or the next evidence we need.
Patch the right layer and keep the regression.
When implementation is useful, we change the relevant layer in a Python, JavaScript, or TypeScript stack and add regression evidence. The public examples below show this work in LangChain, Microsoft Semantic Kernel, and DSPy.
Verified examples
Each example links to a maintainer-merged pull request and the fuller case study. Original reporter and root-cause credit is preserved.
The system prompt vanished during truncation
Semantic Kernel's Python truncation reducer could remove the system or developer message once history exceeded its target. We reproduced the deletion, ported Microsoft's documented .NET behavior, and added regression tests. The merged fix preserves the first system or developer message.
Merged · 19 March 2026Semantic Kernel PR #13610. How we found it.
A forced tool call failed only at the live API
A community reporter found and root-caused a LangChain request that combined forced tool choice with Claude extended thinking, which the Anthropic API rejects. The merged guard drops the forced value with a warning while preserving automatic selection and thinking-off requests.
Merged · 5 March 2026LangChain PR #35544. How we found it.
An empty evaluation set looked like a logging crash
DSPy reported an empty evaluation set as a division error inside metric logging. The merged guard now rejects it at the evaluation boundary with a descriptive error instead of inventing a zero score for work that never ran.
Merged · 26 August 2026DSPy PR #9978. How we found it.
See the worked framework fixes and the engineering record.
What the handoff looks like
Scope is set on the call. The handoff does not change: what you send us, what we send back, and what we say when the symptom will not reproduce.
One system, one symptom, and the artifacts around it.
Pick a single system and the path through it that misbehaves. Tell us the behavior you expected and the behavior you observed. Send a reproduction if you have one, or the nearest failing trace, log, or session record if you do not. Include the prompts, tool descriptions, memory or retrieval configuration, and settings that sit on that path.
A written diagnosis your engineers can act on.
The write-up states whether we reproduced the symptom and what the evidence supports. When the evidence supports a finding, it identifies which layer is failing, the fix we recommend, and what implementing that fix would touch. Otherwise, it names the next evidence required. It is written to be read by an engineer who was not on the call.
If we cannot reproduce it, we say so.
Some symptoms do not reproduce from what is available, and some evidence is too thin to place a cause. We say that plainly, state what the evidence does and does not support, and name the next trace, log, or configuration we would need. We do not turn an unreproduced symptom into a confident finding.
Start
Tell us the system and the symptom. We will tell you what the symptom points to, or what we would need to see before naming a cause, and whether we are the right people to fix it.
Data and partnership inquiries. Bring the data you are proposing, the use case it would support, and the partnership shape you have in mind, on the 20-minute call or by email at roli@hermes-labs.ai. Hermes will say directly whether it fits and what it would need to see next.