Memory and RAG: How Agents Remember Ep05

“Why agents don’t memorise everything – they know where to look.”

Language models have a strange kind of amnesia. Within a single conversation, they can hold everything you’ve said so far – but the moment that conversation ends, it’s gone. There’s also a hard limit on how much a model can “see” at once, called its context window. Feed it a 2,000-page document and a stack of past conversations, and it simply won’t fit.

So how does an agent answer a question about something that happened in a different conversation last week or find the one relevant paragraph in a 500-page manual? The answer is a technique called Retrieval-Augmented Generation, or RAG.

The core idea: don’t remember everything; retrieve what’s relevant

RAG’s insight is simple: instead of trying to cram everything into the model’s context window, keep a large store of information outside the model and pull out only the small handful of pieces that are actually relevant to the current question – right before asking the model to answer.

It’s the difference between memorising an entire library versus knowing exactly which three books to pull off the shelf when someone asks you a question.

How it actually works, step by step

1. Break your information into chunks. A big document gets split into smaller pieces – paragraphs, sections, or a few sentences at a time.

2. Convert each chunk into an embedding. An embedding is a list of numbers that represents the meaning of a piece of text, produced by a separate model built for this purpose. Chunks with similar meaning end up with similar numbers, even if they don’t share any of the same words.

3. Store these embeddings in a vector database. This is a specialised kind of database built to answer one question extremely fast: “Given this new piece of text, which stored chunks are the most similar in meaning?”

4. When a question comes in, embed the question too. The user’s question gets converted into the same kind of numerical representation.

5. Retrieve the most relevant chunks. The vector database compares the question’s embedding to all the stored chunks and returns the handful that are the closest match — say, the top 3 or 5.

6. Hand those chunks to the model, along with the question. Now the model isn’t answering from vague, general knowledge – it’s answering with the actual relevant text right in front of it and can quote or reason over it directly.

This is why RAG-based agents can answer questions about your company’s internal documents, a codebase they’ve never seen before, or a conversation from three weeks ago — they’re not trying to “remember” it in the traditional sense. They’re looking it up, every time, right before answering.

Two kinds of memory, working together

It’s worth separating two things that both get called “memory,” because they solve different problems:

• Working memory is just the current conversation – what’s been said in this task, right now. It naturally fits in the model’s context window and needs no special retrieval system.
• Long-term memory is everything that needs to persist across conversations or tasks – user preferences, facts learned previously, and a knowledge base. This is what RAG is for.

A well-designed agent uses both: working memory for the immediate back-and-forth, and RAG to pull in anything from the long-term store that’s relevant to the current step.

The honest limitations

RAG isn’t magic, and it’s worth knowing where it struggles:

• Retrieval can miss the right chunk, especially if the question is phrased very differently from how the source material is written
• Chunking badly (splitting text in the wrong places) can separate a fact from the context it needs to be understood correctly
• More chunks isn’t always better — stuffing in too much irrelevant retrieved text can confuse the model or push out more important information

Getting RAG right is genuinely one of the harder parts of building a production agentic system, and there’s an entire field of techniques (re-ranking, hybrid search, and query rewriting) dedicated to improving it.

So far we’ve only talked about a single agent working alone. But some of the most interesting systems being built today involve multiple agents, each with a different role, working together on a single task. That’s where we’re headed in our final post.