LoCoMo v1, all 1,986 questions, Top 30Season 1 · Rank 6 of 7
Letta
Letta (hosted API, v1; agent-managed memory)letta/cloud
Report a problem with this result (see the corrections policy)
The judge ruled the retrieved context correct for 1,222 of 1,986 questions (61.5%, 95% CI 59.1 to 63.9), using 1,439 tokens per question on average.43 questions got no verdict because a call failed; they count as incorrect.
Figures
Context
- Tokens per question, mean
- 1,439
- Tokens, 95th percentile
- 1,793
- Accuracy per 1,000 tokens
- 42.8
- Contexts cut to the budget
- 0.0%
Speed
- Retrieval, median
- 1.71 s
- Retrieval, 95th percentile
- 2.79 s
- Ingestion, total
- 6 h 51 min (all conversations added up)
- Ingestion, per turn
- 4.20 s
Judge cost
- Top 10
- $10.63
- Top 30
- $16.09
- Whole run
- $26.72from price table 2026-10-04
Checks
- Abstention accuracy
- 86.5%
- No verdict (call failed)
- 43
- Flagged by the judge
- 3
Where it fails
Accuracy by question category, with 95% confidence intervals. A low category is where the memory system misses the context the question needs.
| Category | Correct | Accuracy | 95% CI | Tokens | No verdict |
|---|---|---|---|---|---|
| adversarial | 386 of 446 | 86.5% | 83.8 to 89.2 | 1,424 | 11 |
| multi-hop | 122 of 282 | 43.3% | 37.8 to 48.9 | 1,410 | 6 |
| open-domain | 39 of 96 | 40.6% | 24.4 to 55.7 | 1,381 | 3 |
| single-hop | 658 of 841 | 78.2% | 74.2 to 82.9 | 1,469 | 16 |
| temporal | 17 of 321 | 5.3% | 2.5 to 8.2 | 1,422 | 7 |
Every question
One square per question, grouped by category. Select a square to read the question, the gold answer, the judge's reason and the context it was given.
Loading questions…
Retrieval latency
The share of questions whose context came back within a given time. Further left is faster.
Judge
A season is one fixed set of judging rules: one pinned judge model, its settings and its prompt rule on every question. When the rules change a new season starts, and earlier runs are judged again from their stored contexts, without re-running the memory system. Seasons
- Model
- claude-sonnet-5-5
- Route
- anthropic
- Season
- Season 1
- Effort
- medium
- Output limit
- 4,000 tokens
- Prompt
- v3 sha256:ce6d9750331c79d1bb3a0737109c458fef4ba9fd8cf929c75c1c1cf9407a4b78
- Served by
- claude-sonnet-5-5
How this result was produced
Everything below is part of the signed result. Change any of it and the signature no longer matches.
Run
- Run id
- 20261101T114118Z-534f50
- Mode
- hosted runner
- Finished
- 1 Nov 2026
- Duration
- 3 h 32 min
- memjudge version
- 0.1.0
- Commit
- 785596c2bacb
- Image
- sha256:e65efdddd807b5126c7abe5ccb9f1b4cad374a1f6c54c388b216e387da442467
- Context budget
- 8,192 tokens (o200k_base)
- Conversations at a time
- 3
- Submitter affiliated
- no
Dataset
- Name
- locomo v1
- Questions
- 1,986
- Licence
- CC-BY-NC-4.0
- Corrections applied
- 0
- Hash
- sha256:9c7812f2f45ce12b06cd9d3fc55736c3a8a83d2b263c11f28c292c475f6ddf38
- Source file hash
- 79fa87e90f04081343b8c8debecb80a9a6842b76a7aa537dc9fdf651ea698ff4
Connector
- Connector
- letta/cloud
- Written by
- MemJudge maintainers
- SDK
- letta-client==1.12.1
- Endpoint
- https://api.letta.com
- Code hash
- sha256:073c87f21cf055b89bf56a9bb39d7a206f53725f8e247318bf4d46b95e6b8f7b
- Models
- openai/gpt-6-luna, openai/text-embedding-3-small
- model
- openai/gpt-6-luna
- embedding
- openai/text-embedding-3-small
- base url
- default
Raw files
| File | SHA-256 |
|---|---|
| conversations.jsonl | 79fd3c3becc90de37ae19256eae336fdff90ff7f85015b30e1ccbbc098db461f |
| questions-top10.jsonl | 56bca9aecd015032332e07e3050a210192f34b6ec37a113d1dbb0f0f630788aa |
| questions-top30.jsonl | d54ec99348369abd312d176b37de733a4187176d44f093c7cee47462559de8e0 |
Check it yourself
The signed result is signed with key 478df5eb628bfa74, one of the trusted keys published in the leaderboard repository. Download it and verify it with the memjudge command-line tool; add --artifactswith the raw files to check those too. How signing works.
memjudge verify 20261101T114118Z-534f50.jsonQuestions, answers and conversation excerpts on this page come from LoCoMo (Maharana et al., 2024, Snap Research) and are used under CC BY-NC 4.0 (non-commercial use only). They are reformatted, and corrected where MemJudge's errata apply. MemJudge's own verdicts and scores are CC BY 4.0.