LoCoMo v1, all 1,986 questions, Top 30Season 1 · Rank 1 of 7

Recalld, retrieval recall

Recalld (hosted API, v1)recalld/cloud

Run history
  1. 1 Nov 2026, 01:12 UTC Started
  2. 1 Nov 2026, 04:14 UTC Stopped: the judge failed
  3. 1 Nov 2026, 11:22 UTC Resumed
  4. 1 Nov 2026, 11:36 UTC Finished

Recorded by the runner. Completed conversations were kept on resume; unfinished ones were redone.

Report a problem with this result (see the corrections policy)

The judge ruled the retrieved context correct for 1,744 of 1,986 questions (87.8%, 95% CI 86.1 to 89.5), using 460 tokens per question on average.

Figures

Context

Tokens per question, mean
460
Tokens, 95th percentile
672
Accuracy per 1,000 tokens
190.9
Contexts cut to the budget
0.0%

Speed

Retrieval, median
1.56 s
Retrieval, 95th percentile
2.24 s
Ingestion, total
7 h 49 min (all conversations added up)
Ingestion, per turn
4.79 s

Judge cost

Top 10
$10.08
Top 30
$10.33
Whole run
$20.41from price table 2026-10-04

Checks

Abstention accuracy
93.0%
No verdict (call failed)
0
Flagged by the judge
21

Where it fails

Accuracy by question category, with 95% confidence intervals. A low category is where the memory system misses the context the question needs.

Accuracy by category
CategoryCorrectAccuracy95% CITokensNo verdict
adversarial415 of 44693.0%90.6 to 95.24390
multi-hop211 of 28274.8%70.1 to 79.35250
open-domain55 of 9657.3%39.2 to 75.05000
single-hop783 of 84193.1%91.3 to 94.94470
temporal280 of 32187.2%82.8 to 91.64540

Every question

One square per question, grouped by category. Select a square to read the question, the gold answer, the judge's reason and the context it was given.

Loading questions…

Retrieval latency

The share of questions whose context came back within a given time. Further left is faster.

Other runs of this connector

Earlier and later runs on this board stay listed. A different code hash means the connector code changed between runs.

FinishedRunCode hashAccuracy
1 Nov 2026Recalld, retrieval search2c22ea08620285.8%

Judge

A season is one fixed set of judging rules: one pinned judge model, its settings and its prompt rule on every question. When the rules change a new season starts, and earlier runs are judged again from their stored contexts, without re-running the memory system. Seasons

Model
claude-sonnet-5-5
Route
anthropic
Season
Season 1
Effort
medium
Output limit
4,000 tokens
Prompt
v3 sha256:ce6d9750331c79d1bb3a0737109c458fef4ba9fd8cf929c75c1c1cf9407a4b78
Served by
claude-sonnet-5-5

How this result was produced

Everything below is part of the signed result. Change any of it and the signature no longer matches.

Run

Run id
20261101T011204Z-108368
Mode
hosted runner
Finished
1 Nov 2026
Duration
10 h 24 min
memjudge version
0.1.0
Commit
896c4f7b13f7
Image
sha256:8671c6e11fb4468e182e776556e616e221df73a4f77051716772e8cd4238222e
Context budget
8,192 tokens (o200k_base)
Conversations at a time
4
Submitter affiliated
yes, with Recalld

Dataset

Name
locomo v1
Questions
1,986
Licence
CC-BY-NC-4.0
Corrections applied
0
Hash
sha256:9c7812f2f45ce12b06cd9d3fc55736c3a8a83d2b263c11f28c292c475f6ddf38
Source file hash
79fa87e90f04081343b8c8debecb80a9a6842b76a7aa537dc9fdf651ea698ff4

Connector

Connector
recalld/cloud
Written by
the vendor
SDK
none (plain HTTP)
Endpoint
https://us.recalld.ai
Code hash
sha256:2c22ea086202c1a0314c13ce277063b23d34661d5224589f4409344c6a2094c4
Models
openai-gpt-6-luna, gemini-embedding-002-vertex
retrieval
recall
decompose model
openai-gpt-6-luna
evaluate model
openai-gpt-6-luna
select model
openai-gpt-6-luna
embedding model
gemini-embedding-002-vertex
base url
default
max retries
4
timeout s
300
batch
off
allow non eu
off
batch poll s
60

Raw files

FileSHA-256
conversations.jsonle8f47a73ea371de297bd3547fbf5233b1b6b06c80bc9f6cb8d9c9946208cadbc
questions-top10.jsonl1ea9bd871f422efc91608d499ada2139a330fca923ad44dd958ac5adce3242fe
questions-top30.jsonl845feb91d240e9da396d0ac09af7fb1ec3b2f223bbe2ea670d305f93aaf4d370

Check it yourself

The signed result is signed with key 478df5eb628bfa74, one of the trusted keys published in the leaderboard repository. Download it and verify it with the memjudge command-line tool; add --artifactswith the raw files to check those too. How signing works.

memjudge verify 20261101T011204Z-108368.json

Download the signed result

Questions, answers and conversation excerpts on this page come from LoCoMo (Maharana et al., 2024, Snap Research) and are used under CC BY-NC 4.0 (non-commercial use only). They are reformatted, and corrected where MemJudge's errata apply. MemJudge's own verdicts and scores are CC BY 4.0.