Aug 24, 2026 · measured
We benchmarked the competition. We lost two columns.
The same 22 real conversations behind our 98.1% number went through Mem0 and Graphiti — the open-source engine inside Zep — running the identical model Lethe uses. One blind judge graded every stored claim from every system, never knowing whose it was.
Lethe: 211 claims, 100% grounded, the only system that cites the exact line each claim came from, and the only one that stored something for every conversation. Mem0: 98.6% grounded — and 8.6× faster than us, and more grounded than the reasoning tier we actually ship (93.6%, dragged down by a schema bug our own 2026-08-20 run already flagged). Zep: 82.8% grounded on three times the volume — hypotheticals stored as history, but also real signal we missed; even discarding every fabricated claim it banked ~2.5× more grounded material than we did.
The wins argue for the quote gate. The losses are a speed gap, an unfixed bug with a price tag, and a recall ceiling — all now in the published record, because a scoreboard that only shows our row isn’t a scoreboard. We also audited every failed verdict by hand and found 11 the judge got wrong — all in Zep’s column, all corrected upward in the record.
Same day, we tested whether the volume gap was self-imposed: our prompt says “prefer fewer observations,” so we relaxed only that line and re-ran everything. 62% more claims, precision unchanged, latency identical — the cap was buying nothing. A third blind pass then asked whether stored claims are actually useful, and handed out one more loss: Mem0’s memory is the cleanest tested (91.6% useful-and-true, above the tier we shipped that morning), while over half of Zep’s volume turned out to be world facts and fragments.
By evening the fixes had shipped and the benchmark ran again: the shipped prompt now grounds at 98.8% — past Mem0 — at double the volume, and a three-lens union arm stored 1,236 claims, 1.9× Zep’s entire count, still 98.5% grounded. After paraphrase dedup, one frozen rule applied to every system (we lose 21% of that count, Zep loses 36% of theirs), it still holds 3.2× Zep in claims that are distinct, true, and worth keeping. A final round of fixes pushed precision to its best measured — 99.1% grounded, 99.3% verbatim — and settled the usefulness question honestly: Mem0’s 91.6% rate is the rate of a narrow net, so the number we hold ourselves to is useful-and-true claims per conversation: 13.7 to their 8.9. Mem0 remains 20× faster, and dedups perfectly. Every number, morning and evening, is in the doc.
22 conversations frozen from the 2026-08-20 sample; Mem0 OSS and graphiti-core run with the same DeepSeek model and local embeddings; every claim graded blind against the full source. Full method and every failed claim: docs/benchmarks/comparative-2026-08.md.