Findings
We tested our own claims.
We ran Memory MCP against a public benchmark, exactly as we ship it, and we are publishing the numbers whether they flatter us or not.
What we ran
LongMemEval-S is a public benchmark of 500 questions, each about a set of roughly 48 realistic chat sessions the answer is hidden inside. We stored every session in Memory MCP with its real remember tool, asked each question through its real recall tool, and checked whether the right session came back in the top 5 results. Nothing was mocked: the real embedding model, the real database, the shipped default of content encrypted at rest.
The test ran in a disposable database created only for this benchmark. No customer or personal data was involved.
The result
Stored the way the product is meant to be used — one distilled fact or message per memory, not a whole conversation dumped in one go — Memory MCP found the correct session in its top 5 results for 98.0% of the 500 questions (98.1% once the 30 unanswerable “trick” questions are set aside). Stored as whole sessions instead, which asks far more of the embedding model in one go, it still found 93.0%.
| Top results checked | One memory per message | One memory per whole session |
|---|---|---|
| 1 | 89.6% | 75.8% |
| 3 | 96.0% | 89.8% |
| 5 | 98.0% | 93.0% |
| 10 | 98.6% | 96.6% |
By question type (top 5, one memory per message): recalling something the assistant said, 100%; a fact that changed over time, 100%; something spread across several sessions, 98.5%; reasoning about when something happened, 94.7%; a stated preference, 100%; something the user said in a single session, 98.6%.
What this does and does not show
- This measures retrieval — does the right memory come back — not the final answer a language model would give from it. The benchmark's own name for this is recall, not question-answering accuracy.
- An embedding model reads only so much text at once. A whole 9,000-character chat session gets truncated before it is compared to your question, which is the real reason the session-level score is lower. Saving one memory per message, as we recommend, avoids that.
- Content encryption was on throughout, our shipped default, so this is real-world behaviour, not a best case with protections switched off.
- We ran this ourselves, on our own machine, in September 2026. We have not had it independently reproduced yet. If you want to check it, the benchmark is public and the method above is everything you need to repeat it.
Why we're telling you this
A small company selling a memory tool should be able to show its memory actually works. We would rather publish a real 98% than an unproven claim of something higher, and we will publish the next result too, better or worse.