Writing Agent Memories: What to Store and What to Skip

Not everything an agent sees is worth remembering. Store too little and it forgets what mattered; store everything and the memory fills with noise that drowns the signal at retrieval time.

The extraction step — deciding what to keep and distilling it into clean, self-contained facts — is the most consequential and most overlooked part of a memory system.

Memory begins with a decision

Of everything that just happened, what's worth keeping? This is the first and most important question a memory system answers, and it's easy to get wrong in both directions. Store too little and the agent forgets the thing that mattered. Store everything and the memory fills with noise, because every irrelevant memory is a candidate to surface at the wrong moment and drown the useful one.

What deserves to be remembered

• Stable facts about the user — preferences, context, recurring needs that will matter in future sessions.
• Decisions and outcomes — what was chosen, what worked, what failed, so the agent doesn't relitigate settled ground.
• Corrections — when a human fixes the agent, that correction is high-value memory: it must stick.
• Durable state — the status of ongoing work, entities the agent tracks, anything with a life beyond one turn.

What to leave out

Transient chatter, one-off details with no future relevance, and anything that will be stale by next session are noise. The discipline of not storing them is as important as the discipline of storing what matters — a memory full of irrelevant fragments retrieves worse than a lean one.

Good memory is curated, not recorded.

Turn experience into clean facts

Raw conversation is messy; stored memories should be clean. Extraction usually distills an interaction into concise, self-contained statements — "user prefers X," "project deadline is Y" — rather than dumping the transcript. A common pattern is to use the model itself to summarize what's worth remembering, producing tidy semantic facts from noisy episodic experience.

memories = extract(conversation)
# -> [{'content': 'prefers email over phone',
#      'subject': 'user_1837', 'type': 'semantic'},
#     {'content': 'had billing issue, resolved by refund',
#      'subject': 'user_1837', 'type': 'episodic'}]
store.add(memories)

Self-contained is the key property: a memory that only makes sense with its original context will confuse retrieval later.

When to extract

There's also a question of timing. The common choices are at the end of a session, when the whole interaction can be distilled at once, or incrementally as significant things happen, so nothing is lost if a session is abandoned. End-of-session is simpler and cheaper but risks losing an interaction that never cleanly ends; incremental captures more reliably at the cost of running more often. Many systems do both.