Which memory system returns the right context, and how much does it send?

Every system stores the same conversations, then retrieves context for each question. A judge model reads only that context and rules whether the answer is in it. No answering model sits in between, and every run is signed.

Every system that lets you choose its LLM runs GPT-6 Luna, so the memory design is compared, not the model. Embedding models are each system's own. Each row shows the models it ran. Fairness rules

Questions
All 1,986 questions
Judge
claude-sonnet-5-5, Season 1
Season
Season 1 · since 28 Oct 2026 (current) · Details ↗
Context budget
8,192 tokens per question
Updated
1 Nov 2026

Accuracy against tokens and latency

Each point is one run; the vertical line is its 95% confidence interval. Higher is more accurate, further left sends fewer tokens or answers faster.

Ranking

Ranked by accuracy. Where intervals overlap, the difference may be chance: compare the runs to see the questions they disagree on.

SystemBy categoryCompare
1Recalld, retrieval recallrecalld/cloudModels: openai-gpt-6-luna, gemini-embedding-002-vertexSubmitter affiliated with RecalldVendor-maintained connectorResumed 1 time
87.886.1 to 89.5
460190.91.56 s
2Recalld, retrieval searchrecalld/cloudModels: openai-gpt-6-luna, gemini-embedding-002-vertexSubmitter affiliated with RecalldVendor-maintained connector
85.884.6 to 87.2
1,19971.5287 ms
3Mem0 open sourcemem0/ossModels: gpt-6-luna, text-embedding-3-small
85.883.9 to 87.7
1,35763.2592 ms
4Supermemorysupermemory/cloudModels: chosen by the vendor, not disclosed
80.978.7 to 83.3
2,33434.7319 ms
5BM25 keyword retrieval over raw turnsbaselines/bm25
70.769.4 to 72.2
1,33153.26.43 ms
6Lettaletta/cloudModels: openai/gpt-6-luna, openai/text-embedding-3-small
61.559.1 to 63.9
1,43942.81.71 s
7Zep Graphiti open sourcezep/graphitiModels: gpt-6-luna, text-embedding-3-smallResumed 1 time
53.351.3 to 55.5
1,20044.42.80 s

By category, left to right: adversarial, multi-hop, open-domain, single-hop, temporal. Each bar is that category's accuracy.

How one question is scored

Every run is scored on the same 1,986 questions, each the same way. This one is real, from the Top 30 board above.

  1. The answer is said twice, months apart

    Caroline and Melanie talk in 19 sessions between 8 May and 22 October 2023: 419 turns. Every system stores all of them.

    Melanie, 25 May 2023 · turn 23 of 419

    “So I'm carving out some me-time each day - running, reading, or playing my violin - which refreshes me and helps me stay present for my fam!”

    Melanie, 28 Aug 2023 · turn 332 of 419

    “Yeah, I play clarinet! Started when I was young and it's been great. Expression of myself and a way to relax.”

  2. The question comes after the last session

    “What instruments does Melanie play?”

    Expected answer: clarinet and violin

  3. Each system returns its top 30 results, and the judge reads only those

    Mem0 open sourceCorrect

    1,213 tokens · first 3 of 30 results

    1. Melanie plays the clarinet, which she started learning when she was young; playing it helps her express herself and relax. …
    2. Caroline asked Melanie whether she plays any musical instruments, showing interest in Melanie’s own musical background.
    3. As of May 25, 2023, Melanie was carving out daily me-time by running, reading, or playing her violin; …

    Judge: “The context states Melanie plays the clarinet and mentions she plays her violin as me-time, covering both instruments.”

    BM25 keyword retrieval over raw turnsIncorrect

    863 tokens · first 3 of 30 results

    1. [2023-08-28 15:19] Caroline: Thanks, Melanie! Appreciate it. You play any instruments?
    2. [2023-08-28 15:19] Melanie: Cool! What type of music do you play?
    3. [2023-08-28 15:19] Melanie: Yeah, I play clarinet! Started when I was young and it's been great. …

    The violin turn of 25 May is not among its 30 results.

    Judge: “Only clarinet is mentioned for Melanie; violin is missing from the context.”

A run's accuracy is the share of all 1,986 questions the judge rules correct. Compare these two runs question by question →

Questions, answers and conversation excerpts on this page come from LoCoMo (Maharana et al., 2024, Snap Research) and are used under CC BY-NC 4.0 (non-commercial use only). They are reformatted, and corrected where MemJudge's errata apply. MemJudge's own verdicts and scores are CC BY 4.0.