Which memory system returns the right context, and how much does it send?

Every system stores the same conversations, then retrieves context for each question. A judge model reads only that context and rules whether the answer is in it. No answering model sits in between, and every run is signed.

Every system that lets you choose its LLM runs GPT-6 Luna, so the memory design is compared, not the model. Embedding models are each system's own. Each row shows the models it ran. Fairness rules

Questions
All 1,986 questions
Judge
claude-sonnet-5-5, Season 1
Season
Season 1 · since 28 Oct 2026 (current) · Details ↗
Context budget
8,192 tokens per question
Updated
1 Nov 2026

Accuracy against tokens and latency

Each point is one run; the vertical line is its 95% confidence interval. Higher is more accurate, further left sends fewer tokens or answers faster.

Ranking

Ranked by accuracy. Where intervals overlap, the difference may be chance: compare the runs to see the questions they disagree on.

SystemBy categoryCompare
1Recalld, retrieval recallrecalld/cloudModels: openai-gpt-6-luna, gemini-embedding-002-vertexSubmitter affiliated with RecalldVendor-maintained connectorResumed 1 time
86.785.0 to 88.4
420206.21.57 s
2Recalld, retrieval searchrecalld/cloudModels: openai-gpt-6-luna, gemini-embedding-002-vertexSubmitter affiliated with RecalldVendor-maintained connector
80.578.7 to 82.5
404199.5289 ms
3Mem0 open sourcemem0/ossModels: gpt-6-luna, text-embedding-3-small
79.577.1 to 81.7
459173.2592 ms
4Supermemorysupermemory/cloudModels: chosen by the vendor, not disclosed
74.471.0 to 77.3
76197.7301 ms
5BM25 keyword retrieval over raw turnsbaselines/bm25
64.262.9 to 65.5
451142.33.21 ms
6Lettaletta/cloudModels: openai/gpt-6-luna, openai/text-embedding-3-small
55.253.1 to 57.4
510108.31.70 s
7Zep Graphiti open sourcezep/graphitiModels: gpt-6-luna, text-embedding-3-smallResumed 1 time
49.246.4 to 52.0
426115.51.60 s

By category, left to right: adversarial, multi-hop, open-domain, single-hop, temporal. Each bar is that category's accuracy.

Questions, answers and conversation excerpts on this page come from LoCoMo (Maharana et al., 2024, Snap Research) and are used under CC BY-NC 4.0 (non-commercial use only). They are reformatted, and corrected where MemJudge's errata apply. MemJudge's own verdicts and scores are CC BY 4.0.