Public record / CISACorrected
CVE-2026-14216 · CVSS 3.1
Recorded5.3
Corrected6.5
Same vector. Correct calculation.
Medium severity unchanged.
Hermes Labs / AI reliability engineeringEst. 2025
A real finding. A public correction.
The record was wrong.
We proved it.
CISA corrected it.
Our autonomous audit found a scoring error in public vulnerability data. CISA acknowledged our report and corrected the record. Bring that same scrutiny to your production AI and agent systems.
Scope your system → (free 20-minute call on Calendly, opens in a new tab)See what you get →
Paid diagnosis & engineering · Free 20-minute scoping call
One AI fix / Microsoft Semantic KernelSystem prompt deleted First system message preserved.Public upstream work · Maintainer-merged PR #13610
Inspect the before & afterInspect the CISA evidence and audit scope
We found a scoring error in a public vulnerability record. CISA acknowledged it and corrected the record.
Error. The CISA enrichment for CVE-2026-14216 stored a CVSS score of 5.3 for a CVSS 3.1 vector that calculates to 6.5.
Fix. CISA republished the score as 6.5; the vector and the Medium severity did not change.
How. A Hermes Labs autonomous system audited 5,324 recent CISA ADP records, including 1,060 CVSS 3.x scores checked against their vectors, reproduced the one mismatch and prepared the report. It was filed once the founder authorized the exact target and account.
CISA's reply on issue #333 · CVE correction commit
Related engineering: Making LangChain’s agent tool binding compatible with Claude’s extended thinking · PR #35544 · merged 2026-03-05
Inside the work / Microsoft Semantic Kernel
The conversation kept going.
The system prompt didn’t.
History truncation could delete the system or developer message. We reproduced the deletion, ported Microsoft’s documented .NET behavior, and added regression tests.
Less history.
Missing instruction.
System / developer message removedRecent conversation retained
The reducer could remove the instruction that was meant to govern the conversation.
Less history.
Instruction preserved.
First system / developer message preservedRecent conversation retained
The merged fix preserves the first system or developer message during truncation.
Public upstream workPR #13610 · Maintainer-merged, 19 March 2026
Read the fix and regression evidenceAnother layer / LangChain
Making LangChain’s agent tool binding compatible with Claude’s extended thinking
A 20-line guard handles forced tool choice when extended thinking is enabled, with two test functions covering eight scenarios. Community reporter’s root-cause credit is preserved.
These are public examples of our method. Paid work starts with your system, your symptom, and an agreed scope.
Put the lab on your problemPaid expert work
Hire the lab for
the part you need.
A written diagnosis when the cause is unclear. Engineering when the failing layer is known. Scope and terms are agreed on the free call.
01 / DiagnosisPaid
Find the failing layer.
Bring a symptom, a trace, or a reproduction. We inspect the prompts, tools, memory, and configuration on the path that fails.
You get
Explore diagnosis A written diagnosis: reproduction status, supported findings, the recommended fix and what it would touch—or the next evidence needed.
02 / EngineeringPaid
Patch the right layer.
When implementation is useful, we change the relevant layer in your Python, JavaScript, or TypeScript stack.
You get
Explore engineering A scoped change with regression evidence. These public fixes show how a specific finding becomes an inspectable engineering result.
The tools and research are free. Use the 10-project open-source catalog, papers, and public receipts without hiring us.
Hermes Labs is a reliability lab for production AI and agent systems: it diagnoses failures, engineers fixes, and publishes the tools, papers and receipts behind that work. The company is building the reliability layer for autonomous systems. Founded by Rolando Bosch (Roli), it builds and maintains LintLang, static analysis for AI agent instructions, and Little Canary, prompt-injection sensing for untrusted input.
§ 01
Explore AI reliability topics →What breaks in production
01
A guardrail disappears mid-conversation
The instruction is still configured, but a truncation or summary step silently dropped it from the effective context sent to the model several turns ago. The agent stops following the rule, and nothing logs that it lapsed.
Fix · mergedWe named the model-behavior failure in our taxonomy, then fixed a framework defect with the same operational failure shape in Microsoft Semantic Kernel. How we found it.
02
The same mistake returns
Corrections that do not persist
A user corrects the system, but the correction disappears into a long chat log. The next response repeats the same drift as if the repair never happened.
ToolOur open-source hermeneutic mines correction triples from chat logs and gates the next response before the same drift ships twice. Explore the research.
03
Compression changes the record
Your summary or memory layer compresses context to save tokens and quietly changes what it meant: a dropped qualifier, a paraphrase that is not what you wrote.
ToolOur open-source Fidelis Memory returns your context verbatim instead of paraphrasing it. Explore the research.
04
The run leaves no usable audit trail
Unprovable agent actions
Something went wrong in an autonomous run, and you cannot show what the agent actually did, in what order, or why.
ScopeStart by naming the actions that need evidence and the records your system actually retains. Discuss the audit gap.
Diagnosis and engineering on your system are paid engagements; the tools, papers and case-study receipts are free. Match a production problem to its receipt →
Inspect the research and engineering record
§ 02
Full citations →Technical record
1,461
controlled experiments, conducted primarily on GPT-4o, behind the taxonomy of epistemic failure modes. The taxonomy names the failure classes the audit methodology looks for.
in the taxonomy paper
Zenodo 19042468
Zenodo 19042468
4 fixes
merged AI/framework fixes: forced tool choice under Anthropic thinking in LangChain, system-prompt preservation and duplicate-null schema handling in Semantic Kernel, and empty-devset validation in DSPy. 96 published external pull requests merged in 2026, including 49 code, workflow, typing or maintenance contributions across open-source infrastructure.
6 papers
on Zenodo with permanent DOIs: three preprints, one working paper, and two technical notes. Their distinct evidence roles include a seven-mode corpus taxonomy, a matched-vignette experiment, a twelve-week telemetry audit, a conceptual paper on model self-report, a bounded static analysis of tool descriptions, and a prompt-injection sensing architecture.
10-project catalog
a curated public path from project setup and preflight checks to runtime observation, evaluation, and local memory. Each tool names its present evidence boundary.
Case study
a falsification-first diagnosis of a closed-source Claude Code resume hang, with measured cause classes and the evidence boundary stated explicitly.
§ 03
Book a call
For enterprise AI teams
Tell us what’s breaking.
Book a free 20-minute call and tell us the system and the symptom, or send a note below. Either way you get a specific read on what the symptom points to, or what we would need to see before naming a cause, and whether we are the right people to fix it, not a pitch.
Book a 20-min call (Calendly, opens in a new tab) →