AI Agent Memory | 30 Days of AI Agents, Day 16

Yesterday we grounded LLMs in documents with RAG. Today we return to AI Agent Memory and go a level deeper. Day 5 covered why agents need memory and which types exist. This post covers how to engineer it: memory architectures, write and retrieve policies, summarization, and forgetting.

Save this sheet. The rest of this note walks through it section by section.

A Quick Recap and the Real Problem

Agent memory is an agent's ability to remember information from past interactions, events, and its environment, and to use it in future decisions. It gives you context, personalization, better decisions, task continuity, and learning over time. Without it, a stateless agent forgets previous messages, makes users repeat themselves, and answers inconsistently.

Day 5 established that. The harder engineering question is different: an agent can't remember everything, and the context window can't hold everything. Memory design is really a set of policies about what to keep, where to keep it, when to bring it back, and when to let it go.

Four Kinds of Memory

Type	Holds	Example	Lifetime
Short-term (working)	The recent conversation within a session	Last few messages in a chat	Session
Long-term (persistent)	Facts that hold across sessions	User preferences, projects, notes	Until changed or deleted
Semantic (knowledge)	Facts about the world or the business	Company docs, APIs, FAQs, manuals	Until the source changes
Episodic (experience)	Specific past events	"User asked about X on 10 May"	Decays in usefulness

Semantic memory is largely the RAG system from yesterday. The two new problems are long-term facts (which change and conflict) and episodes (which pile up fast).

The Components of a Memory System

The sheet's architecture is a pipeline with a manager on top:

Input            Processing           Memory store      Retrieval       Context builder     LLM / agent
user message --> text splitter   -->  vector DB    -->  semantic   -->  current input  -->  generate
events           embedding            (Pinecone,        top-K           + retrieved         response
files / docs     metadata             Weaviate,                         memory
API data         classification       Chroma, Milvus)
          \__________________ MEMORY MANAGER ___________________/
                    (decides what to save, update, or delete)

The memory manager is the component most teams skip, and it's the one that decides whether memory helps or hurts. The context builder is the second most important. It decides how retrieved memory is formatted and how much of the token budget it gets.

The sheet's flow example shows the loop. "What was our discussion about project Phoenix?" → the agent retrieves past memories → adds them to the query → the LLM answers → important new information is saved back to memory.

Memory Architectures That Work

From simplest to most capable:

1. Full buffer. Keep the whole conversation. Simple, and fine for short sessions, but it breaks on long ones.
2. Sliding window. Keep the last N turns. Cheap, but it forgets anything older, including important early facts.
3. Window plus rolling summary. Keep the last N turns verbatim and a running summary of everything before. This is the best default for chat.
4. Retrieved memory. Store memories as embedded records, and pull back the top-K relevant ones each turn.
5. Structured profile plus retrieved episodes. Keep a small, explicit profile (name, role, preferences, current projects) that's always in context, and pull relevant episodes by search.

Option 5 is what we build for most production assistants. Stable facts are cheap to include every time. Episodic detail is fetched only when relevant.

The Write Policy

Logging every message as "memory" gives you a large, noisy store that retrieves the wrong things. A write policy answers four questions:

• What qualifies? Extract candidate memories with an LLM step ("list durable facts, preferences, or decisions in this exchange"), not raw transcripts.
• How important is it? Score importance when writing, since a stated preference matters more than small talk.
• Is it new, or an update? Before inserting, search for similar existing memories. If "prefers weekly reports" exists and the user now says "send them daily," update it. Don't add a contradiction.
• What's sensitive? Classify PII and secrets before storing. Some things should never be written.

Tag every memory with metadata: user, source, timestamp, type, and confidence. You'll need all of it later.

The Retrieval Policy

Pure similarity search isn't enough, because an old, trivial memory can outrank a recent, important one. A common approach, popularized by research on generative agents, combines three signals:

import math, time

def memory_score(m, query_sim, now=None, half_life_days=30):
    now = now or time.time()
    age_days = (now - m["created_at"]) / 86400
    recency = math.exp(-math.log(2) * age_days / half_life_days)
    return 0.5 * query_sim + 0.3 * m["importance"] + 0.2 * recency  # weights: tune them

# candidates = vector_store.search(embed(query), top_k=20, filter={"user_id": uid})
# context    = sorted(candidates, key=lambda m: memory_score(m, m.sim), reverse=True)[:5]

Retrieve broadly, re-rank with this score, and keep only a few. Always filter by user or tenant before similarity, so one user's memory can never leak into another's context.

Summarization and Forgetting

Summarization turns many episodes into fewer, denser ones. Two patterns hold up well: a rolling conversation summary updated every few turns, and periodic consolidation that merges a week of related episodes into one record ("worked on Phoenix launch plan; decided on a phased rollout"). Keep links to the originals in case you need detail later.

Forgetting is a feature, not a failure:

• Time-to-live for things that go stale, like session state and temporary tasks.
• Decay through the recency term above, so old memories fade from retrieval without being deleted.
• Supersession, where an updated fact replaces the old one.
• User-driven deletion. "Forget that" must actually delete it, across every store, including embeddings and summaries. This is a privacy requirement, not a nice-to-have.

Storage Options and Metrics

Type	Best for	Examples	Trade-off
In-memory	Short-term, within a session	Buffers, framework memory classes	Lost when the session ends
Vector DB	Semantic and episodic recall	Pinecone, Weaviate, Chroma, Milvus	Cost and complexity
SQL / NoSQL	Profiles and structured facts	PostgreSQL, MongoDB, DynamoDB	Not built for semantic search
Key-value	Preferences, session state	Redis, DynamoDB, Upstash	Limited querying
File / object storage	Raw logs and large documents	S3, GCS, Azure Blob, MinIO	Needs a retrieval layer on top

The sheet's code uses LangChain's classic ConversationBufferMemory. Newer LangChain versions move this to LangGraph checkpointers for per-thread memory and a separate store for long-term memory. The concept is the same. Whichever framework you use, Postgres with pgvector can hold profiles and episodes in one place.

Track these metrics: retrieval accuracy, context relevance, response quality, memory growth, cost per turn, and user satisfaction. If memory growth rises steadily while retrieval accuracy falls, your write policy is too loose.

Key Takeaways

• Memory engineering is mostly policy: what to write, what to retrieve, what to summarize, what to forget.
• A small, always-included profile plus retrieved episodes is a strong default architecture.
• Rank memories by relevance, importance, and recency, and always filter by user first.
• Update or replace conflicting facts. Don't just add more of them.
• Deletion has to reach every store, including embeddings and summaries.

FAQs

Is memory just RAG over chat history?

Partly. Retrieval is the same mechanism, but memory adds write decisions, updates, decay, and deletion, which a document index usually doesn't need.

How much memory should go into each prompt?

Give it a fixed token budget, often a small fraction of the context window, and fill it by score. More memory isn't better if it's noisy.

Should the agent decide what to remember?

An LLM can propose memories, but a memory manager with rules (thresholds, PII checks, deduplication) should make the final call.

Up Next

Day 17 covers Multi-Agent Systems in depth: system types, communication patterns, shared memory, and the coordination costs that come with more agents. If you want an assistant that remembers the right things and forgets the rest, get in touch.