Silent failures, traced to evidence
Each case study walks a real reliability or public-record problem end to end: the observable inconsistency, the evidence, the contribution we made, and the external outcome. 4 culminated in merged fixes; one records a cited reproduction and an unmerged patch whose technical substance a maintainer acknowledged as included in a broader sweep; one is a forensic diagnosis of a closed-source tool, produced without source access; and one is a CISA-acknowledged CVSS consistency correction. It is the same evidence-bound engineering discipline Hermes Labs runs on private and public systems. The full set of merged upstream contributions is indexed on /open-source/contributions.
Diagnostic audits7 case studiesOutcome stated per case
Hermes Labs CISA CVSS Consistency Correction
A bounded public-data audit found one CVSS score/vector mismatch, corrected an unsupported history inference, stopped at the publication boundary, and produced a report CISA acknowledged and corrected.
When an Empty Evaluation Set Looks Like a Logging Crash
A merged DSPy fix turns an empty-devset division error into an explicit validation error at the evaluation boundary.
One Nullable Type, Two Nulls: Repairing Semantic Kernel Schema Generation
A merged Semantic Kernel .NET fix replaces reference equality with value equality when adding null to strict tool schemas.
Fixing a Silent 400: Forced tool_choice Under Claude Thinking in LangChain
A request-time 400 caused by forced tool_choice under Claude extended thinking. A community reporter root-caused it; we engineered the precise guard at the bind_tools() binding — which values to drop versus keep, inert when thinking is off — and merged it upstream.
When the System Prompt Vanishes: A Silent Failure in Semantic Kernel
Truncating a long conversation silently deleted the system prompt, so the model quietly stopped following its instructions — no error, no log. Reported in 2025 and already fixed on Microsoft's .NET side; we reproduced the Python instance, ported the fix, and merged it.
Auditing Agent Memory: A Silent Retrieval Failure in mem0 v2.0
A concrete Chroma reproduction that downstream work cited, followed by a separate tested Redis patch that a maintainer later acknowledged as included in an eleven-backend sweep. The case study preserves earlier community provenance and distinguishes diagnosis, patch authorship, and upstream adoption.
Forensic Diagnosis of a Closed-Source Hang: Claude Code --resume
A hang with no error, no log, and no source to read. We reproduced it, falsified four cause-classes by direct measurement, and filed the triage artifact a maintainer could act on — including a sanitized repro and three confirmation tests. The issue auto-closed as stale; the record stands on the tracker.
Technical reports
A technical report documents an investigation and its reusable lesson: the symptom, the control, the mechanism in the inspected source, what was delivered, and what the evidence does and does not establish. A report can cover an unresolved upstream issue; it is not a case study and it does not claim adoption, deployment or endorsement.
When a Failed LangGraph Router Disappears on Resume
A conditional router can fail visibly on the first LangGraph invocation and then be absent from resumed execution, which returns normally without running the downstream node. Original Hermes report, control, source mechanism, two independent reproductions and a credited containment pattern; upstream issue #8834 remains open.
Evidence Before Remediation: Why a Permission Error Does Not Identify Its Cause
Two merged Hermes contributions tightened the evidence Nono diagnostics require before recommending a permission change: Nono #1816 gates path and network guidance on observed evidence, and Nono Packs #29 asks whether Nono actually caused the denial. Core v0.76.0 contains #1816 by commit ancestry; no Packs release is claimed.