Inside an Agent: The Building Blocks Ep02

“Meet the brain, the hands, and the memory behind every AI agent.”

In the last post, we said an agent is more like an assistant than a waiter — it doesn’t just respond, it acts toward a goal. But “acts toward a goal” is doing a lot of work in that sentence. What’s actually inside the box that makes this possible?

Every agentic system, no matter how simple or sophisticated, is built from the same four pieces.

1. The Brain: a large language model

At the center of every agent is an LLM — the same kind of model powering a regular chatbot. This is the part that “thinks”: it reads the current situation, considers the goal, and decides what to do next.

The key shift is that the LLM isn’t just generating a reply for a human to read. It’s generating a decision — often literally structured as “call this tool, with these arguments” — that the surrounding system will act on.

2. Hands: tools

Tools are how an agent reaches outside its own head. Without tools, an LLM can only talk. With tools, it can:

• Search the web for current information
• Run code to do a precise calculation
• Read or write a file
• Query a database
• Send a message or make an API call

A tool is really just a function with a description attached, so clear the model knows when and how to use it. “This tool takes a city name and returns the current weather” is enough for the model to figure out that if someone asks about umbrellas, this tool might be relevant.

3. Memory: keeping track of what’s happened

A conversation with a chatbot is usually short — a handful of back-and-forth messages. An agent completing a real task might take dozens of steps, and it needs to remember:

• What the original goal was
• What it’s already tried
• What worked and what didn’t
• Facts it has learned along the way

There are two flavors of this:

• Short-term (working) memory — everything relevant to the current task, usually just kept in the conversation history
• Long-term memory — information that needs to persist across different tasks or conversations, often stored in an external database and retrieved when relevant (we’ll dig into this in Post 5)

4. Planning: deciding what to do, and in what order

This is the least “physical” piece, and often the least visible — but it’s the difference between an agent that flails and one that’s genuinely useful. Planning is the process of taking a big, fuzzy goal (“plan my sister’s birthday party”) and breaking it into concrete, ordered steps (check the guest list → find a venue → compare catering options → send invites).

In simple agents, this planning happens implicitly, one step at a time — the model just decides “what’s the single best next action?” over and over. In more sophisticated agents, there’s an explicit planning phase up front, producing something like a checklist the agent then works through (and revises as it learns new information).

How these fit together

None of these four pieces do anything alone. The magic is in how they connect:

• The brain looks at the goal and the memory of what’s happened so far
• It consults its plan (explicit or implicit) to decide the next step
• It uses a tool to take that step
• The result gets folded back into memory
• Repeat, until the goal is met

That repeating cycle — think, act, observe, repeat — is important enough that it deserves its own post. That’s exactly where we’re headed next.