LoCoMo v1, all 1,986 questions, Top 10Season 1 · Rank 1 of 7
Recalld, retrieval recall
Recalld (hosted API, v1)recalld/cloud
Run history
- 1 Nov 2026, 01:12 UTC Started
- 1 Nov 2026, 04:14 UTC Stopped: the judge failed
- 1 Nov 2026, 11:22 UTC Resumed
- 1 Nov 2026, 11:36 UTC Finished
Recorded by the runner. Completed conversations were kept on resume; unfinished ones were redone.
Report a problem with this result (see the corrections policy)
The judge ruled the retrieved context correct for 1,722 of 1,986 questions (86.7%, 95% CI 85.0 to 88.4), using 420 tokens per question on average.
Figures
Context
- Tokens per question, mean
- 420
- Tokens, 95th percentile
- 522
- Accuracy per 1,000 tokens
- 206.2
- Contexts cut to the budget
- 0.0%
Speed
- Retrieval, median
- 1.57 s
- Retrieval, 95th percentile
- 2.26 s
- Ingestion, total
- 7 h 49 min (all conversations added up)
- Ingestion, per turn
- 4.79 s
Judge cost
- Top 10
- $10.08
- Top 30
- $10.33
- Whole run
- $20.41from price table 2026-10-04
Checks
- Abstention accuracy
- 92.4%
- No verdict (call failed)
- 0
- Flagged by the judge
- 18
Where it fails
Accuracy by question category, with 95% confidence intervals. A low category is where the memory system misses the context the question needs.
| Category | Correct | Accuracy | 95% CI | Tokens | No verdict |
|---|---|---|---|---|---|
| adversarial | 412 of 446 | 92.4% | 89.6 to 94.8 | 404 | 0 |
| multi-hop | 195 of 282 | 69.1% | 65.0 to 74.0 | 439 | 0 |
| open-domain | 54 of 96 | 56.3% | 36.8 to 72.7 | 415 | 0 |
| single-hop | 779 of 841 | 92.6% | 91.3 to 94.2 | 414 | 0 |
| temporal | 282 of 321 | 87.9% | 83.4 to 91.8 | 445 | 0 |
Every question
One square per question, grouped by category. Select a square to read the question, the gold answer, the judge's reason and the context it was given.
Loading questions…
Retrieval latency
The share of questions whose context came back within a given time. Further left is faster.
Other runs of this connector
Earlier and later runs on this board stay listed. A different code hash means the connector code changed between runs.
| Finished | Run | Code hash | Accuracy |
|---|---|---|---|
| 1 Nov 2026 | Recalld, retrieval search | 2c22ea086202 | 80.5% |
Judge
A season is one fixed set of judging rules: one pinned judge model, its settings and its prompt rule on every question. When the rules change a new season starts, and earlier runs are judged again from their stored contexts, without re-running the memory system. Seasons
- Model
- claude-sonnet-5-5
- Route
- anthropic
- Season
- Season 1
- Effort
- medium
- Output limit
- 4,000 tokens
- Prompt
- v3 sha256:ce6d9750331c79d1bb3a0737109c458fef4ba9fd8cf929c75c1c1cf9407a4b78
- Served by
- claude-sonnet-5-5
How this result was produced
Everything below is part of the signed result. Change any of it and the signature no longer matches.
Run
- Run id
- 20261101T011204Z-108368
- Mode
- hosted runner
- Finished
- 1 Nov 2026
- Duration
- 10 h 24 min
- memjudge version
- 0.1.0
- Commit
- 896c4f7b13f7
- Image
- sha256:8671c6e11fb4468e182e776556e616e221df73a4f77051716772e8cd4238222e
- Context budget
- 8,192 tokens (o200k_base)
- Conversations at a time
- 4
- Submitter affiliated
- yes, with Recalld
Dataset
- Name
- locomo v1
- Questions
- 1,986
- Licence
- CC-BY-NC-4.0
- Corrections applied
- 0
- Hash
- sha256:9c7812f2f45ce12b06cd9d3fc55736c3a8a83d2b263c11f28c292c475f6ddf38
- Source file hash
- 79fa87e90f04081343b8c8debecb80a9a6842b76a7aa537dc9fdf651ea698ff4
Connector
- Connector
- recalld/cloud
- Written by
- the vendor
- SDK
- none (plain HTTP)
- Endpoint
- https://us.recalld.ai
- Code hash
- sha256:2c22ea086202c1a0314c13ce277063b23d34661d5224589f4409344c6a2094c4
- Models
- openai-gpt-6-luna, gemini-embedding-002-vertex
- retrieval
- recall
- decompose model
- openai-gpt-6-luna
- evaluate model
- openai-gpt-6-luna
- select model
- openai-gpt-6-luna
- embedding model
- gemini-embedding-002-vertex
- base url
- default
- max retries
- 4
- timeout s
- 300
- batch
- off
- allow non eu
- off
- batch poll s
- 60
Raw files
| File | SHA-256 |
|---|---|
| conversations.jsonl | e8f47a73ea371de297bd3547fbf5233b1b6b06c80bc9f6cb8d9c9946208cadbc |
| questions-top10.jsonl | 1ea9bd871f422efc91608d499ada2139a330fca923ad44dd958ac5adce3242fe |
| questions-top30.jsonl | 845feb91d240e9da396d0ac09af7fb1ec3b2f223bbe2ea670d305f93aaf4d370 |
Check it yourself
The signed result is signed with key 478df5eb628bfa74, one of the trusted keys published in the leaderboard repository. Download it and verify it with the memjudge command-line tool; add --artifactswith the raw files to check those too. How signing works.
memjudge verify 20261101T011204Z-108368.jsonQuestions, answers and conversation excerpts on this page come from LoCoMo (Maharana et al., 2024, Snap Research) and are used under CC BY-NC 4.0 (non-commercial use only). They are reformatted, and corrected where MemJudge's errata apply. MemJudge's own verdicts and scores are CC BY 4.0.