AI Agent Memory: What to Store, What to Forget, and Why the Write Path Matters

Every agent demo has memory. Almost none of them have a forgetting strategy, and that is why they work beautifully for a week and then start giving stale, confidently wrong answers.

Memory stopped being a feature you bolt on in 2026. It is now a first-class architectural component with its own benchmarks and its own well-documented failure modes.

Memory is not one thing

“Add memory to the agent” is about as specific as “add a database”. There are four kinds, with different lifetimes and different reasons to exist.

Working memory — this task, right now. The scratchpad within a single run. Dies when the run ends, and should.

Episodic memory — what happened before. Past runs, what was tried, what the outcome was. This is what stops an agent making the same failed attempt on Tuesday that it made on Monday.

Semantic memory — durable facts. The user prefers metric units. The production database is Postgres. The client’s brand voice is informal. Things that are true across runs.

Procedural memory — how things are done here. Learned routines and preferences: “deploys always run tests first”, “this customer wants a summary before the detail.”

Most systems ship working memory, call it memory, and stop. The interesting behaviour — an agent that visibly improves — comes from the other three.

Retrieval is the easy half

Ask how to build agent memory and you will get vector database answers: embed, store, cosine similarity, top-k. That is the read path, and it is largely solved.

The read path is not where memory systems die.

The write path is where it goes wrong, because the naive version — append every turn to a store and retrieve by similarity later — is not memory. It is a log. And a log has two properties that make it actively harmful over time.

It grows without bound. Every interaction adds tokens. Retrieval pulls more of them. Eventually memory retrieval is consuming most of the context budget, and you have engineered context rot into your own system on purpose.

It never resolves contradictions. In March the user said they preferred email. In July they said Slack. Both are in the store. Both match “communication preference”. The agent retrieves both and picks one, essentially at random, forever.

The four decisions on write

A real write path makes four decisions, and skipping any of them is what turns memory into a liability.

Is this worth keeping? Most conversation is not. “Thanks, that worked” carries no durable information. A cheap classifier before the store beats an expensive retrieval filter afterwards.

What is the fact, separate from the conversation? Store the claim, not the transcript. user.timezone = "Asia/Karachi" beats 400 tokens of the exchange where it came up — it is smaller, it is unambiguous, and it can be compared to other facts.

Does this contradict something I already know? This is the one people skip, and it is the one that matters. New information about the same attribute should update or supersede, not accumulate alongside.

When does this expire? Some facts are permanent (where someone was born). Some decay (current project, current preference, current priority). A memory system with no notion of staleness will confidently serve you last quarter’s truth.

async def remember(observation, store, llm):
    # 1. WORTH KEEPING? Cheap filter first — most turns carry nothing durable.
    if not await llm.classify(observation, "Does this state a durable fact "
                              "about the user, their systems, or their preferences?"):
        return None

    # 2. EXTRACT THE FACT, not the conversation around it.
    fact = await llm.extract(observation, schema={
        "subject": str,          # "user"
        "attribute": str,        # "preferred_channel"
        "value": str,            # "slack"
        "confidence": float,
        "expires_after_days": int | None,   # None = durable
    })

    # 3. CONFLICTS — the step that separates memory from a log.
    existing = await store.find(subject=fact["subject"], attribute=fact["attribute"])
    for old in existing:
        if old["value"] != fact["value"]:
            # supersede, don't accumulate — keep the old row for audit, not retrieval
            await store.supersede(old["id"], by=fact, at=observation.timestamp)

    # 4. TTL, so stale facts age out instead of being served forever.
    return await store.write(fact)

Note what is deliberately not there: no embedding of the raw conversation. Embeddings come after extraction, over the fact, so retrieval matches meaning rather than phrasing.

The test that exposes a fake memory system

One scenario finds the problem in about two minutes:

1. Tell the agent something. “I prefer email.”
2. Use it enough that it is definitely stored.
3. Change your mind. “Actually, use Slack from now on.”
4. Start a fresh session and ask what your preferred channel is.

A log answers “email” or “Slack” unpredictably, or hedges with both. A memory system answers “Slack”, and if pushed can tell you it used to be email and when that changed.

If your system fails this, no amount of retrieval tuning fixes it. The bug is on write.

A prompt for designing the write path

Worth running before you pick a vector database, because the store matters far less than the policy:

I'm designing the memory write path for an AI agent. Here is what it does and
who uses it:

[DESCRIBE THE AGENT, ITS USERS, AND A TYPICAL SESSION]

Work through these, and be concrete rather than general:

1. List the specific facts worth persisting across sessions. For each one, say
   which memory type it is: working, episodic, semantic or procedural.
2. For each fact, give a realistic expiry — permanent, or a TTL with reasoning.
3. Identify which facts can CONTRADICT each other, and write the resolution
   rule for each conflict (newest wins? highest confidence? ask the user?).
4. Tell me what should explicitly NOT be stored, and why storing it would hurt.
5. Estimate how many tokens of memory a typical request will retrieve after six
   months of daily use. If that number is large, tell me what to cut.

Do not recommend a vector database. I want the policy first.

Question 5 is the one that changes designs. Teams routinely discover their memory would be pulling several thousand tokens per request within a year — which is a context budget problem they built themselves.

The bottom line

Memory is four different things with four different lifetimes, and the read path is the easy half. What separates an agent that improves from one that decays is the write path: filtering what is worth keeping, extracting facts rather than transcripts, resolving contradictions instead of accumulating them, and letting stale things expire.

Build the forgetting strategy at the same time as the remembering one. A system that only ever adds is not learning — it is just getting slower and more confidently wrong.