Skip to content
Hermes Labs / AI Reliability Engineering / Est. 2025San Francisco · Worldwide

Your AI passed testing. That’s the problem.

Evaluations tell you a system performed well on a fixed set of inputs. They don’t tell you it will hold under adversarial probing, silent instruction drift, or the scrutiny of a procurement review. We work on the gap between the two.

Patent filings
1 non-provisional · 4 provisional
USPTO 19/248,833 (non-provisional, pending)
+ 4 provisional filings
Upstream · merged
26 PRs · 2 AI-framework fixes
Open-source tools
18 released
github.com/hermes-labs-ai
Apache 2.0 · no telemetry
S · 01

AI Assurance Audit

Pre-deployment · post-incident · 2–4 weeks

When you’d bring us in: the eval set passes, but the system fabricates tool output, silently relaxes instructions in long contexts, or hedges where a null answer was required.

We review prompts, tool descriptions, agent scaffolds, and configurations against the failure-mode taxonomy from our research: Null-Result Asymmetry, Hermeneutic Drift, Source-Status Credibility Bias, Silent Instruction Relaxation. You get a written report with prioritized fixes.

S · 02

Runtime Assurance

Production controls · evidence · 4–8 weeks

When you’d bring us in: the agent has tool access, traffic is real, and you need controls that survive an incident review, not a demo.

Input-side sensing, execution-side policy enforcement, anti-fabrication guards, and signed evidence. Paired defense: little-canary at the prompt boundary, suy-sideguy at process and network layer.

S · 03

EU AI Act Readiness

Annex IV · ISO 42001 · NIST AI RMF

When you’d bring us in: security, legal, procurement, or an auditor wants more than a policy PDF: they want findings, runtime controls, and artifacts a third party can verify.

We map your technical documentation to Annex IV, cross-walk it to ISO 42001 and NIST AI RMF, and identify what is missing or likely to fail external challenge. The same evidence bundle defends in more than one jurisdiction.

● High-risk AI system obligations take effect 2 Aug 2026 (EU AI Act Annex III).

2,000+
controlled adversarial evaluations across five failure-mode classes, feeding the audit methodology and the runtime defense tuning.
methodology in
Zenodo 18867694
26 merged
upstream contributions across AI, ML, and web-tooling repos. Two are AI-framework runtime fixes that ship in production pipelines: a forced-tool-choice crash in LangChain’s Anthropic binding, and silent system-prompt deletion in Semantic Kernel’s chat-history reducer.
5 filings
US patent filings on stateless user identification, adversarial probing, deterministic inference control, multi-modal classification calibration, and confidence-gated personalization.
USPTO 19/248,833
+ 4 provisional
2 papers
peer-reviewable research on epistemic failure and asymmetric evidential standards in language models, published with DOIs.
18 tools
open-source releases across audit, runtime, and evidence. Apache 2.0, no telemetry, no gated tiers.

For enterprise AI teams

Tell us what you’re shipping.

A short note on what’s in scope, what triggered the review, and what evidence already exists is the best place to start. We respond with a specific scoping question or a proposed engagement shape, not a calendar link.