LoCoMo v1, all 1,986 questions, Top 10Season 1 · Rank 6 of 7

Letta

Letta (hosted API, v1; agent-managed memory)letta/cloud

Report a problem with this result (see the corrections policy)

The judge ruled the retrieved context correct for 1,096 of 1,986 questions (55.2%, 95% CI 53.1 to 57.4), using 510 tokens per question on average.41 questions got no verdict because a call failed; they count as incorrect.

Figures

Context

Tokens per question, mean
510
Tokens, 95th percentile
655
Accuracy per 1,000 tokens
108.3
Contexts cut to the budget
0.0%

Speed

Retrieval, median
1.70 s
Retrieval, 95th percentile
2.81 s
Ingestion, total
6 h 51 min (all conversations added up)
Ingestion, per turn
4.20 s

Judge cost

Top 10
$10.63
Top 30
$16.09
Whole run
$26.72from price table 2026-10-04

Checks

Abstention accuracy
91.3%
No verdict (call failed)
41
Flagged by the judge
8

Where it fails

Accuracy by question category, with 95% confidence intervals. A low category is where the memory system misses the context the question needs.

Accuracy by category
CategoryCorrectAccuracy95% CITokensNo verdict
adversarial407 of 44691.3%88.4 to 93.85057
multi-hop71 of 28225.2%18.0 to 33.85024
open-domain32 of 9633.3%18.6 to 46.54893
single-hop571 of 84167.9%64.3 to 71.551820
temporal15 of 3214.7%2.6 to 6.85077

Every question

One square per question, grouped by category. Select a square to read the question, the gold answer, the judge's reason and the context it was given.

Loading questions…

Retrieval latency

The share of questions whose context came back within a given time. Further left is faster.

Judge

A season is one fixed set of judging rules: one pinned judge model, its settings and its prompt rule on every question. When the rules change a new season starts, and earlier runs are judged again from their stored contexts, without re-running the memory system. Seasons

Model
claude-sonnet-5-5
Route
anthropic
Season
Season 1
Effort
medium
Output limit
4,000 tokens
Prompt
v3 sha256:ce6d9750331c79d1bb3a0737109c458fef4ba9fd8cf929c75c1c1cf9407a4b78
Served by
claude-sonnet-5-5

How this result was produced

Everything below is part of the signed result. Change any of it and the signature no longer matches.

Run

Run id
20261101T114118Z-534f50
Mode
hosted runner
Finished
1 Nov 2026
Duration
3 h 32 min
memjudge version
0.1.0
Commit
785596c2bacb
Image
sha256:e65efdddd807b5126c7abe5ccb9f1b4cad374a1f6c54c388b216e387da442467
Context budget
8,192 tokens (o200k_base)
Conversations at a time
3
Submitter affiliated
no

Dataset

Name
locomo v1
Questions
1,986
Licence
CC-BY-NC-4.0
Corrections applied
0
Hash
sha256:9c7812f2f45ce12b06cd9d3fc55736c3a8a83d2b263c11f28c292c475f6ddf38
Source file hash
79fa87e90f04081343b8c8debecb80a9a6842b76a7aa537dc9fdf651ea698ff4

Connector

Connector
letta/cloud
Written by
MemJudge maintainers
SDK
letta-client==1.12.1
Endpoint
https://api.letta.com
Code hash
sha256:073c87f21cf055b89bf56a9bb39d7a206f53725f8e247318bf4d46b95e6b8f7b
Models
openai/gpt-6-luna, openai/text-embedding-3-small
model
openai/gpt-6-luna
embedding
openai/text-embedding-3-small
base url
default

Raw files

FileSHA-256
conversations.jsonl79fd3c3becc90de37ae19256eae336fdff90ff7f85015b30e1ccbbc098db461f
questions-top10.jsonl56bca9aecd015032332e07e3050a210192f34b6ec37a113d1dbb0f0f630788aa
questions-top30.jsonld54ec99348369abd312d176b37de733a4187176d44f093c7cee47462559de8e0

Check it yourself

The signed result is signed with key 478df5eb628bfa74, one of the trusted keys published in the leaderboard repository. Download it and verify it with the memjudge command-line tool; add --artifactswith the raw files to check those too. How signing works.

memjudge verify 20261101T114118Z-534f50.json

Download the signed result

Questions, answers and conversation excerpts on this page come from LoCoMo (Maharana et al., 2024, Snap Research) and are used under CC BY-NC 4.0 (non-commercial use only). They are reformatted, and corrected where MemJudge's errata apply. MemJudge's own verdicts and scores are CC BY 4.0.