LoCoMo v1, all 1,986 questions, Top 30Season 1 · Rank 3 of 7

Mem0 open source

Mem0 open source (in-process, full multi-signal setup)mem0/oss

Report a problem with this result (see the corrections policy)

The judge ruled the retrieved context correct for 1,703 of 1,986 questions (85.8%, 95% CI 83.9 to 87.7), using 1,357 tokens per question on average.

Figures

Context

Tokens per question, mean
1,357
Tokens, 95th percentile
1,546
Accuracy per 1,000 tokens
63.2
Contexts cut to the budget
0.0%

Speed

Retrieval, median
592 ms
Retrieval, 95th percentile
900 ms
Ingestion, total
7 h 29 min (all conversations added up)
Ingestion, per turn
4.59 s

Judge cost

Top 10
$10.24
Top 30
$15.36
Whole run
$25.60from price table 2026-10-04

Checks

Abstention accuracy
93.0%
No verdict (call failed)
0
Flagged by the judge
20

Where it fails

Accuracy by question category, with 95% confidence intervals. A low category is where the memory system misses the context the question needs.

Accuracy by category
CategoryCorrectAccuracy95% CITokensNo verdict
adversarial415 of 44693.0%90.5 to 95.81,3300
multi-hop179 of 28263.5%57.2 to 70.11,3500
open-domain47 of 9649.0%29.1 to 68.81,3270
single-hop785 of 84193.3%91.9 to 95.41,3610
temporal277 of 32186.3%82.0 to 90.41,4010

Every question

One square per question, grouped by category. Select a square to read the question, the gold answer, the judge's reason and the context it was given.

Loading questions…

Retrieval latency

The share of questions whose context came back within a given time. Further left is faster.

Judge

A season is one fixed set of judging rules: one pinned judge model, its settings and its prompt rule on every question. When the rules change a new season starts, and earlier runs are judged again from their stored contexts, without re-running the memory system. Seasons

Model
claude-sonnet-5-5
Route
anthropic
Season
Season 1
Effort
medium
Output limit
4,000 tokens
Prompt
v3 sha256:ce6d9750331c79d1bb3a0737109c458fef4ba9fd8cf929c75c1c1cf9407a4b78
Served by
claude-sonnet-5-5

How this result was produced

Everything below is part of the signed result. Change any of it and the signature no longer matches.

Run

Run id
20261101T010624Z-79759b
Mode
hosted runner
Finished
1 Nov 2026
Duration
2 h 35 min
memjudge version
0.1.0
Commit
896c4f7b13f7
Image
sha256:8671c6e11fb4468e182e776556e616e221df73a4f77051716772e8cd4238222e
Context budget
8,192 tokens (o200k_base)
Conversations at a time
4
Submitter affiliated
no

Dataset

Name
locomo v1
Questions
1,986
Licence
CC-BY-NC-4.0
Corrections applied
0
Hash
sha256:9c7812f2f45ce12b06cd9d3fc55736c3a8a83d2b263c11f28c292c475f6ddf38
Source file hash
79fa87e90f04081343b8c8debecb80a9a6842b76a7aa537dc9fdf651ea698ff4

Connector

Connector
mem0/oss
Written by
MemJudge maintainers
SDK
mem0ai[nlp]==2.2.1
Code hash
sha256:2bea0837a48df5082a31b032cf620319ce603c7ea13f26727ca3e455980112fd
Models
gpt-6-luna, text-embedding-3-small
llm model
gpt-6-luna
llm is reasoning model
on
embedder model
text-embedding-3-small
embedding dims
1536

Raw files

FileSHA-256
conversations.jsonlb56d353cb45209bb866c411f79fba4db8c108d93281c92c323ab45c2a680e8f8
questions-top10.jsonlbe7695c9f9557bf898e7a707c81387234763daaf61e2fea5a6f0b7c2d92cbe61
questions-top30.jsonlcd1e2c795f0278155f22e9cd8f073dea2623a7cdf997d2a0a55654ed59a3839a

Check it yourself

The signed result is signed with key 478df5eb628bfa74, one of the trusted keys published in the leaderboard repository. Download it and verify it with the memjudge command-line tool; add --artifactswith the raw files to check those too. How signing works.

memjudge verify 20261101T010624Z-79759b.json

Download the signed result

Questions, answers and conversation excerpts on this page come from LoCoMo (Maharana et al., 2024, Snap Research) and are used under CC BY-NC 4.0 (non-commercial use only). They are reformatted, and corrected where MemJudge's errata apply. MemJudge's own verdicts and scores are CC BY 4.0.