Book a 30-min call
← cd ../blogs
$ cat posts/agent-memory-benchmarks-graph-vs-rag.mdx

Agent memory: graph versus vector was the wrong argument

July 19, 2026 · ImmovableTech Team

  • Agentic AI
  • Retrieval
  • Evaluation

The argument we all had was the wrong argument

For about eighteen months the agent memory conversation was a two-sided fight: temporal knowledge graphs on one side, vector stores on the other, each with a benchmark table showing it won. We took that argument seriously enough to build both, and the systems we shipped afterwards failed for reasons neither table describes.

The benchmark under almost all of it is LoCoMo: synthetic multi-session conversations generated by LLM agents against persona and event-graph scaffolding then repaired by human annotators, each averaging 600 turns and 16K tokens over up to 32 sessions. The public release is a single file of ten conversations, with questions grouped into single-hop, multi-hop, temporal, open-domain and adversarial categories. Both major memory vendors exclude the adversarial category from their headline scores.

So the thing being measured is: given a finished transcript nobody will ever add to, can you stitch together facts from two or three distant sessions. That is a retrieval problem over a closed corpus. It is a real problem, and it is not the one that took our agents down.

The published numbers were unstable by 25 points

Before arguing about which architecture wins, it is worth noticing that the field could not agree on what the scores were.

Zep published an evaluation putting its own system at 84% on LoCoMo. Mem0’s CTO reproduced it and got 58.44% ± 0.20, and identified the cause: the scoring code counted correct answers from the adversarial category in the numerator while excluding that category from the denominator. Zep accepted the error and published a corrected 75.14% ± 0.17 over ten runs. Mem0’s own paper, meanwhile, had reported Zep at 65.99% ± 0.16. That is four numbers for one system on one benchmark, spanning roughly 25 points, and the spread is almost entirely an artefact of grading harnesses. LoCoMo ships questions and reference answers but no canonical grader, so every vendor brings its own judge prompt and its own inclusion rules.

The detail that should have ended the debate is sitting in Mem0’s own results table. Their full-context baseline — paste the entire conversation into the prompt for every question — scored 72.90% ± 0.19 on the LLM-as-a-judge metric. Their best memory configuration, Mem0 with graph memory, scored 68.44% ± 0.17. The paper claiming state-of-the-art memory shows that doing no memory engineering at all scores higher on the accuracy axis; the claim rests on cost and latency, where it is genuinely strong, at 91% lower p95 latency and over 90% token savings. That is a defensible argument about economics dressed up as an argument about intelligence.

What an independent harness found

The most useful thing published on this is an independent testbed from Wolff and Bennati, which put mem0, Graphiti and cognee against RAG and full-context baselines on the same infrastructure, same judge and same dataset, in a simulated cloud-edge deployment, and measured CPU time, peak RAM, disk I/O and network alongside accuracy.

Accuracy split into two statistically distinct clusters across 1,540 question-answer pairs. mem0 reached 81.08%, the plain RAG baseline 78.31% and full-context 77.16%. Graphiti reached 56.03% and cognee 55.27%. Every cross-cluster comparison was significant at p = 0.002 under a paired Wilcoxon test with Bonferroni correction, and no within-cluster difference was — mem0 versus RAG came out at p = 0.084.

The mechanism matters more than the ranking. The gap is not bad reasoning over retrieved facts, it is missing facts: Graphiti and cognee returned “unknown” on 25.58% and 28.18% of questions against 8.38% to 11.14% for the upper cluster, because the responder was instructed to decline rather than guess when context was insufficient. Graph ingestion adds entity resolution, relation labelling and schema-constrained extraction, and each step is a chance to drop something before retrieval is ever attempted. The authors ran this with Graphiti’s own stricter judge prompt, which should if anything have favoured it.

Then the cost axis, which is the part vendors do not publish. On the paper’s AWS Fargate pricing model, total cost of ownership across loading and query phases came to $0.65 for RAG against $5.43 for mem0 — the 8.4× figure — with Graphiti at $6.95 and full-context at $10.22. Only RAG and mem0 were non-dominated on the Pareto frontier. Loading ten conversations took Graphiti 9.69 hours and $5.49 in LLM calls; RAG took 0.93 hours and one cent.

Read that carefully before repeating it. It is one benchmark, one representative system per architectural class, a cloud-edge split simulated on a single machine, and ingestion at the granularity of single conversational turns — which specifically disadvantages cognee, built for long documents. The honest conclusion is narrow: on multi-hop recall over finished dialogue, graph construction cost us an order of magnitude in ingestion and bought negative accuracy. It is not a claim that graphs are the wrong shape for memory.

The failure modes that actually broke our agents

None of our production incidents looked like a LoCoMo question.

The first version of a support agent we built wrote memory generously. Anything the user asserted about themselves went into the long-term store, on the theory that recall problems are worse than clutter. Two things happened. The store filled with thousands of entries per account, most of them restatements of the same handful of facts in slightly different words, plus a long tail of things said once in passing and never true again. And a user who had told the agent months earlier that they were on the paid tier, then downgraded, got an agent that confidently explained entitlements they no longer had — and kept doing it, because nothing in the conversation ever contradicted the stored fact. The correction lived in the billing database, which the memory layer had no reason to consult.

We reached for retrieval quality first, which was the wrong instinct. Adding a reranker made that class of failure worse, not better: it did its job, ranked the stale plan-tier memory as highly relevant to a question about entitlements, and delivered it more reliably than the weaker retriever had. Precision against the store went up while correctness against the world went down. That is when it became clear we were tuning the wrong stage.

The third mode is subtler and never shows up as an outage. Retrieval returns something that is true, well-ranked and irrelevant to this turn — a preference from a different context, a decision that has since been superseded by a newer one we also stored. The agent is not wrong, it is off-topic, and it reads as an agent that is not listening. We only found how common this was by reading traces, because no aggregate metric we had was measuring it.

Stale memory, writes that should never have happened, and true-but-irrelevant recall. Those are the three, and the architecture debate speaks to none of them.

Write policy is the thing no benchmark scores

Be precise about the gap, because it is easy to overstate. LongMemEval, the other benchmark in common use, does test more than recall: 500 curated questions covering information extraction, multi-session reasoning, temporal reasoning, knowledge updates and abstention, embedded in histories of roughly 115k tokens or 500 sessions in the larger setting. Knowledge updates and abstention are exactly the right instincts. Commercial assistants and long-context models drop about 30% in accuracy on it.

But a knowledge update in a benchmark means the transcript itself contains the correction — the user says one thing in session four and something different in session nineteen, and a competent system answers with the later value. That is invalidation with a signal handed to you inside the evaluated corpus. Our failures were invalidation with no signal at all: the fact expired somewhere else, in a database the agent could have read and didn’t.

More fundamentally, the scored unit in every one of these benchmarks is the answer. There is no benchmark we know of that penalises a write that should never have occurred, because a memory nobody asks about is invisible to a question-answering metric. Yet write volume is what determines retrieval precision, storage cost and, in our experience, the rate of the third failure mode.

The tooling reflects this. LangGraph’s persistence layer gives you checkpointers for thread-scoped state and stores for long-term cross-thread data — correct, well-factored mechanism, and deliberately silent on what deserves to be written or when it stops being true. That is your policy to write. Letta goes further and ships something close to an actual write policy: memory blocks that stay in context, archival memory behind a search tool, and sleep-time agents that run in the background and rewrite the primary agent’s memory blocks, triggered every N steps with a default of five. It is the most serious productised answer to “who decides what to remember”, and it has no benchmark scoring it either.

The genuine tension is Graphiti. Its bi-temporal model with temporal edge invalidation — contradictory facts get invalidated rather than overwritten, so history stays queryable — is the design in this space that takes invalidation most seriously. It is also the design that scored worst on the independent harness. Taking invalidation seriously costs recall, and no popular benchmark pays you back for it. That is an incentive problem, not an engineering result.

What we store, what we never store and how it expires

Our position now is that the memory layer should be small, and most of what agents were storing belongs somewhere else.

Never store anything a system of record can answer at call time. Plan tier, invoice status, ticket state, entitlements, current balance: read them through a tool on the turn that needs them. This is where memory design and tool schema design meet — a cheap authoritative read beats a cached fact that can rot. Never store inferences about the person, as opposed to things they said, because you cannot correct what you cannot trace. And keep transient task state in the thread, not the store.

What survives that filter is small: stated preferences, corrections the user explicitly made, stable identifiers and constraints, and decisions with the date they were taken. Every entry carries provenance and a durability class, and the gate runs before the write, not as a cleanup job.

from dataclasses import dataclass
from datetime import date

@dataclass(frozen=True)
class Memory:
    kind: str  # 'timezone', 'stated_preference', 'plan_tier', ...
    value: str
    observed_on: date
    source_turn: str
    durability: str  # 'stable', 'revocable', 'episodic'

# Anything a system of record can answer at call time is never a memory.
AUTHORITATIVE_ELSEWHERE = {"plan_tier", "invoice_status", "ticket_state", "entitlements"}

def should_write(m: Memory) -> bool:
    if m.kind in AUTHORITATIVE_ELSEWHERE:
        return False  # read it through a tool instead
    if m.durability == "episodic":
        return False  # belongs to the thread, not the long-term store
    return True

Expiry is not a TTL. A time-to-live on a user’s timezone is nonsense and a time-to-live on a stated preference is a guess. What worked was tying revocable memories to the source that owns them, so a change in that source invalidates the memory, and giving episodic notes a decay that removes them from retrieval without deleting the trace.

Evaluating this on your own traffic

Given a 25-point harness spread on the public benchmark and two vendors disputing each other’s methodology, the only number worth acting on is one you produced on your own transcripts.

Build three evaluations, and keep them separate. Score the write decision first: sample real turns, have humans label whether each produced memory was worth storing, and track write precision and recall as their own metric — this is the one nobody has and the one that moved our quality most. Then score staleness directly: change a value in the source system, replay the conversation, and check whether the agent still asserts the old one. That is a test you can automate and it fails loudly. Only then score retrieval, and score it per turn, asking whether what came back was relevant now rather than whether it was true.

Set your baseline honestly too. Chunk the transcript, embed it, retrieve top-k, and measure. The independent testbed puts that baseline inside the top accuracy cluster at a fraction of the cost, and any memory product you adopt should have to beat it on your traffic and on your bill. The rule is the same one we apply to detection pipelines: build the measurement before the architecture.

What we’d do differently

We would build the write gate first. We spent real effort on retrieval — embedding choices, hybrid search, a reranker — before we had any policy about what entered the store, and every one of those improvements amplified a bad corpus. The reranker episode was the clearest signal and we misread it for weeks as a retrieval tuning problem.

We would not have run the graph-versus-vector comparison ourselves. We spent weeks building a temporal graph memory for a system whose failures were all invalidation and write discipline, and a plain chunked-transcript baseline would have told us that in days for a fraction of the cost. The architecture question was legible and measurable, which is exactly why it absorbed attention that the unmeasured question deserved.

And we would treat any memory benchmark score, ours or a vendor’s, as a claim about a specific grading harness rather than about a system. The 84% that became 58.44% that became 75.14% was not anyone acting in bad faith. It is what happens when a benchmark ships questions without a grader and an industry builds a market on the resulting table.

References

We design and evaluate agent memory systems as part of our AI & Machine Learning Engineering practice. Talk to us if your agent is confidently repeating something that stopped being true.