Which memory system returns the right context, and how much does it send?
Every system stores the same conversations, then retrieves context for each question. A judge model reads only that context and rules whether the answer is in it. No answering model sits in between, and every run is signed.
Every system that lets you choose its LLM runs GPT-6 Luna, so the memory design is compared, not the model. Embedding models are each system's own. Each row shows the models it ran. Fairness rules
- Questions
- All 1,986 questions
- Judge
- claude-sonnet-5-5, Season 1
- Context budget
- 8,192 tokens per question
- Updated
- 1 Nov 2026
Accuracy against tokens and latency
Each point is one run; the vertical line is its 95% confidence interval. Higher is more accurate, further left sends fewer tokens or answers faster.
- On the frontier: no other run beats it on both accuracy and tokens
- Other runs
- Shaded: a run here would beat every run so far on both
Ranking
Ranked by accuracy. Where intervals overlap, the difference may be chance: compare the runs to see the questions they disagree on.
| System | By category | Compare | |||||
|---|---|---|---|---|---|---|---|
| 1 | Recalld, retrieval recallrecalld/cloudModels: openai-gpt-6-luna, gemini-embedding-002-vertexSubmitter affiliated with RecalldVendor-maintained connectorResumed 1 time | 86.785.0 to 88.4 | 420 | 206.2 | 1.57 s | ||
| 2 | Recalld, retrieval searchrecalld/cloudModels: openai-gpt-6-luna, gemini-embedding-002-vertexSubmitter affiliated with RecalldVendor-maintained connector | 80.578.7 to 82.5 | 404 | 199.5 | 289 ms | ||
| 3 | Mem0 open sourcemem0/ossModels: gpt-6-luna, text-embedding-3-small | 79.577.1 to 81.7 | 459 | 173.2 | 592 ms | ||
| 4 | Supermemorysupermemory/cloudModels: chosen by the vendor, not disclosed | 74.471.0 to 77.3 | 761 | 97.7 | 301 ms | ||
| 5 | BM25 keyword retrieval over raw turnsbaselines/bm25 | 64.262.9 to 65.5 | 451 | 142.3 | 3.21 ms | ||
| 6 | Lettaletta/cloudModels: openai/gpt-6-luna, openai/text-embedding-3-small | 55.253.1 to 57.4 | 510 | 108.3 | 1.70 s | ||
| 7 | Zep Graphiti open sourcezep/graphitiModels: gpt-6-luna, text-embedding-3-smallResumed 1 time | 49.246.4 to 52.0 | 426 | 115.5 | 1.60 s |
By category, left to right: adversarial, multi-hop, open-domain, single-hop, temporal. Each bar is that category's accuracy.
Questions, answers and conversation excerpts on this page come from LoCoMo (Maharana et al., 2024, Snap Research) and are used under CC BY-NC 4.0 (non-commercial use only). They are reformatted, and corrected where MemJudge's errata apply. MemJudge's own verdicts and scores are CC BY 4.0.