Auditing Agent Memory: A Silent Retrieval Failure in mem0 v2.0

The public record has several layers: earlier community diagnosis, our concrete Chroma reproduction, a community patch that cited it, our separate Redis patch, and the maintainer sweep that superseded the open patches. Keeping those layers distinct makes the case study more useful — and the attribution more defensible.

• 2026-03-20 · #4453 and #4456 publish the general distance-versus-similarity diagnosis and exact Chroma and Redis conversion formulas; #4465 follows with both backends.
• 2026-04-28 · Hermes Labs files mem0 #4999 with a runnable repro of the concrete symptom: identical score=1.0 for every result on chroma in 2.0.0, with evidence isolating the wrapper above the vector store.
• 2026-05-01 · community PR #5044 explicitly says it fixes #4999 and uses the example distances from our report in its regression test.
• 2026-05-25 · Hermes Labs runs the 8-gate diagnostic pipeline and opens PR #5250: a redis-backend repro plus fix converting cosine distance to similarity, matching mem0's in-repo weaviate convention.
• 2026-06-05 · mem0 staff merge PR #5391, a comprehensive eleven-backend sweep that explicitly supersedes #4456 and #4465. The maintainer thanks our #4999 reproduction for helping confirm the fix and later acknowledges that #5250's Redis conversion is included. The merged PR itself does not cite #4999, #5044, or #5250.

The pipeline is binary at every gate: pass, or stop. No gate is skippable. No gate is judgment-graded. The intent is that an opinionated process produces a falsifiable artifact at each step, and the artifact at the end is a defensible diff that a maintainer can adopt without re-doing the diagnosis.

The linked issues, pull requests, tests, maintainer comments, and commit diffs form the public evidence record for this case.

Our patch for the redis backend converted the cosine distance to a similarity:

score = 1 - float(result["vector_distance"])

The merged sweep (#5391) does not apply one formula everywhere. It normalizes the eleven affected backends with per-metric conversions, because the right inverse depends on the metric the backend reports:

# cosine-distance backends (pgvector, redis, valkey,
#   supabase, s3_vectors, vertex_ai)
score = max(0.0, 1.0 - distance)

# L2-distance backends (chroma, faiss-euclidean, milvus-L2)
score = 1.0 / (1.0 + distance)

# similarity-native backends (azure_mysql, cassandra)
score = similarity        # already a similarity; returned directly

Three things follow from this that we want to state plainly. First, our Redis patch and the cosine branch of the sweep use the same core conversion, but the Chroma symptom in #4999 lives in the L2 branch (1.0 / (1.0 + distance)), not the cosine one. The two contributions address related instances on different backends. We do not claim our Redis patch would have fixed the Chroma symptom.

Second, the exact clamped Redis formula in the merged sweep had already been published in #4456 before #5250. Our patch remains a correct, independently tested contribution, and the maintainer acknowledged its conversion as included; the public history does not prove that it was the source of the merged line.

Third, on the clamp. Our Redis conversion omitted the max(0.0, ...) floor. We matched the in-repo weaviate convention, which also omits it, and our falsification log did not include the negative-cosine case: vectors with negative cosine similarity yield a distance above 1.0 and therefore a negative score. That is a gap our Gate 3 audit missed. The maintainers' clamp closes it, and the sweep is the correct general fix. We would add the clamp today.

For an integrator running mem0, or any agent memory system in a production loop, a scoring inversion may not announce itself. Calls can return plausible-looking results while ranking, deduplication, or threshold behavior is wrong for affected backend and metric paths. We do not put a dollar figure on the exposure here, because it depends on which backend a team ran, how heavily it relied on retrieval, and over what window.

The deliverable from the pipeline above is not the upstream PR. Our PR (#5250) closed unmerged once the staff sweep landed, and that is the expected outcome of surfacing a bug well: it gets fixed, by whoever is best placed to fix it broadly. The deliverable is the twelve-section diagnostic report, the falsification log (including the clamp gap we now know it missed), the failing-then-passing regression test, and the explicit list of what was checked and what was not. A buyer can read it cold and decide whether the bug applies to their stack, whether the fix shape matches their backend, and whether the audit firm checked the right things.

This is the same pipeline Hermes Labs runs when the audit target is a prompt template, an agent harness, a retrieval layer, or a tool definition. The systematic failure categories vary; the gate structure does not.

If you ship an LLM-based product that relies on retrieval, memory, or agentic tool use, a silent ranking inversion is the kind of failure that passes every test you have and shows up as quietly worse answers. We run this pipeline against production AI stacks under NDA, and the deliverable is the report, not a pull request.

References: #4453 · #4456 · #4465 · #4999 · #5044 · #5250 · #5391