Start a hosted run
The runner benchmarks one memory system on a full dataset with your keys, judges every answer with this season's judge, and publishes the signed result on the leaderboard.
- Your keys, your costs. The memory system's usage is billed by its vendor, and the judge by Anthropic or OpenRouter. The form shows the judge's typical cost and upper bound before you start.
- Keys are used for this run only. They are encrypted for the run's jobs, deleted once read and never logged. Resuming a run asks for them again.
- The result is public. Every answer, the judge's verdicts and your affiliation answer are published with a signature. How signing works.
- Stopped is not lost. If the memory system or the judge fails, the run stops and keeps its progress. You can resume it within 48 hours.
Signed in with GitHub.
Loading…
Sign in
Starting a run needs a GitHub sign-in. MemJudge keeps only a keyed hash of your account id, to apply the limits: one run at a time and three a day. Nothing about your account is published. Privacy notice.
Your run
Log
The newest lines the run's jobs wrote. The keys you entered are removed from them. Logs are kept 30 days.
Publishing to the leaderboard. The link appears here when the result page is live, usually within a minute or two.
Your runs
Your last 20 runs from the past 90 days, on any browser you sign in with.