Which memory system returns the right context, and how much does it send? Every system stores the same conversations, then retrieves context for each question. A judge model reads only that context and rules whether the answer is in it. No answering model sits in between, and every run is signed.
Every system that lets you choose its LLM runs GPT-6 Luna, so the memory design is compared, not the model. Embedding models are each system's own. Each row shows the models it ran. Fairness rules
Questions All 1,986 questions
Judge claude-sonnet-5-5, Season 1
Context budget 8,192 tokens per question
Updated 1 Nov 2026 Accuracy against tokens and latency Each point is one run; the vertical line is its 95% confidence interval. Higher is more accurate, further left sends fewer tokens or answers faster.
Context tokens Latency
On the frontier: no other run beats it on both accuracy and tokens Other runsShaded: a run here would beat every run so far on both
Ranking Ranked by accuracy. Where intervals overlap, the difference may be chance: compare the runs to see the questions they disagree on.
SystemAll systems BM25 keyword retrieval over raw turns Letta Mem0 open source Recalld Supermemory Zep Graphiti open source Sort byRank Fewest tokens Accuracy per 1,000 tokens Fastest
Rank System Accuracy 40% 95% CI 100% Tokens per question Accuracy per 1,000 tokens Latency (median) By category Compare 1 Recalld, retrieval recall recalld/cloud Models: openai-gpt-6-luna, gemini-embedding-002-vertex Submitter affiliated with Recalld Vendor-maintained connector Resumed 1 time 87.8 86.1 to 89.5
460 190.9 1.56 s Compare Recalld, retrieval recall 2 Recalld, retrieval search recalld/cloud Models: openai-gpt-6-luna, gemini-embedding-002-vertex Submitter affiliated with Recalld Vendor-maintained connector 85.8 84.6 to 87.2
1,199 71.5 287 ms Compare Recalld, retrieval search 3 Mem0 open source mem0/oss Models: gpt-6-luna, text-embedding-3-small 85.8 83.9 to 87.7
1,357 63.2 592 ms Compare Mem0 open source 4 Supermemory supermemory/cloud Models: chosen by the vendor, not disclosed 80.9 78.7 to 83.3
2,334 34.7 319 ms Compare Supermemory 5 BM25 keyword retrieval over raw turns baselines/bm25 70.7 69.4 to 72.2
1,331 53.2 6.43 ms Compare BM25 keyword retrieval over raw turns 6 Letta letta/cloud Models: openai/gpt-6-luna, openai/text-embedding-3-small 61.5 59.1 to 63.9
1,439 42.8 1.71 s Compare Letta 7 Zep Graphiti open source zep/graphiti Models: gpt-6-luna, text-embedding-3-small Resumed 1 time 53.3 51.3 to 55.5
1,200 44.4 2.80 s Compare Zep Graphiti open source
By category, left to right: adversarial, multi-hop, open-domain, single-hop, temporal. Each bar is that category's accuracy.
How one question is scored Every run is scored on the same 1,986 questions, each the same way. This one is real, from the Top 30 board above.
1 The answer is said twice, months apartCaroline and Melanie talk in 19 sessions between 8 May and 22 October 2023: 419 turns. Every system stores all of them.
Melanie, 25 May 2023 · turn 23 of 419
“So I'm carving out some me-time each day - running, reading, or playing my violin - which refreshes me and helps me stay present for my fam!”
Melanie, 28 Aug 2023 · turn 332 of 419
“Yeah, I play clarinet! Started when I was young and it's been great. Expression of myself and a way to relax.”
2 The question comes after the last session“What instruments does Melanie play?”
Expected answer: clarinet and violin
3 Each system returns its top 30 results, and the judge reads only those1,213 tokens · first 3 of 30 results
Melanie plays the clarinet, which she started learning when she was young; playing it helps her express herself and relax. … Caroline asked Melanie whether she plays any musical instruments, showing interest in Melanie’s own musical background. As of May 25, 2023, Melanie was carving out daily me-time by running, reading, or playing her violin; … Judge: “The context states Melanie plays the clarinet and mentions she plays her violin as me-time, covering both instruments.”
863 tokens · first 3 of 30 results
[2023-08-28 15:19] Caroline: Thanks, Melanie! Appreciate it. You play any instruments? [2023-08-28 15:19] Melanie: Cool! What type of music do you play? [2023-08-28 15:19] Melanie: Yeah, I play clarinet! Started when I was young and it's been great. … The violin turn of 25 May is not among its 30 results.
Judge: “Only clarinet is mentioned for Melanie; violin is missing from the context.”
A run's accuracy is the share of all 1,986 questions the judge rules correct. Compare these two runs question by question →
Questions, answers and conversation excerpts on this page come from LoCoMo (Maharana et al., 2024, Snap Research) and are used under CC BY-NC 4.0 (non-commercial use only). They are reformatted, and corrected where MemJudge's errata apply. MemJudge's own verdicts and scores are CC BY 4.0.