The Anatomy of an AI Agent: Same Model, Different Job Description
Agents, from the ground up · Part 3 of 7. If an agent is "a model + a loop + tools," what exactly goes into one? Five ingredients, and they demystify the whole thing.
When people hear "we have a test-writing agent and a code-review agent," they often picture two different AIs, each trained for its specialty. That's almost never how it works. Open up any agent and you'll find the same five parts. Change those parts, and the same model becomes a completely different worker.
The five ingredients
🧠 A brain (the model). The LLM doing the reasoning. This is the expensive, general-purpose part, and usually it's the same model across every agent you run.
📝 A prompt (its role). The system instructions that define who it is, what it's for, what it must never do, and what "done" looks like.
🔧 Tools (its hands). Read a file, write a file, run a command, search the web, query a database, call an API. Each tool is just a name, a description, and an input schema that the harness knows how to execute.
🗂️ Context (its desk). Everything it can see right now: the task, the files or data it's working on, and the results of every tool call so far.
📖 Skills (its playbooks). Reusable how-to guides it can load when a task calls for one, like "how we write tests in this repo" or "how to build a slide deck." They're loaded on demand, so they don't clutter the desk until they're needed.
The key realization
A "specialist agent" is not a different, smarter AI. It's the same model handed a different job description and a different toolbelt. Same person, different brief.
A specialist is just the same model, narrowed by its prompt and its tools.
One brain, three jobs
Here's what that looks like in practice, using the kinds of agents I work with on FinBot:
Test-writer
Prompt: "Write tests for one file. Match the existing test style. Never modify application code."
Tools: read files, write files under tests/, run pytest.
Context: the one source file, plus one example test.
Bug-hunter
Prompt: "Find inputs that make this detector miss an attack. Report them; don't fix anything."
Tools: read files and run the detector against test inputs. No write access at all.
Context: the detector and a list of known attack patterns.
Reviewer
Prompt: "Review this diff for security and correctness. Cite file and line for every finding."
Tools: read-only access to the repository and the diff.
Context: the pull request.
Nothing about the brain changed between those three. Only the job description, the toolbelt, and what's on the desk.
Each ingredient is also a lever, and a risk
Thinking in these five parts makes it obvious where to tune an agent, and where it can go wrong:
Prompt. Vague instructions give you vague behavior. The best prompts say what success looks like and what's off-limits. But a prompt is a request, not a wall. A clever input can talk a model out of its instructions, so never rely on the prompt alone to enforce safety.
Tools. This is where real-world effects happen, so this is where least privilege lives. A test-writer doesn't need a deploy tool. A reviewer doesn't need write access. If a tool isn't in the belt, no amount of prompt injection can make the agent use it. That's a far stronger guarantee than "the prompt says not to." Tool descriptions matter too: the model picks tools by reading them, so a misleading description leads to misused tools.
Context. Context is a budget. Stuff in too much and the important parts get lost; give it too little and it guesses. And anything in context can influence the model, including text hidden inside a web page or a document it reads. That's how indirect prompt injection works.
Skills. Good skills make agents consistent: same conventions, same quality bar, every time. They're also how you scale your team's know-how without writing it into every prompt.
Model. Swapping in a bigger model helps with hard reasoning, but it's rarely the first fix. Usually the prompt, tools, or context are the real problem.
Why this matters
Once you see agents as configuration rather than magic, building them stops being mysterious. You're not training a new AI. You're writing a clear brief, choosing the smallest toolbelt that gets the job done, and deciding exactly what it gets to see. That's a design problem, and also a security problem, and you can reason about both.
Next up, Part 4: Orchestrator and subagents. What happens when one agent runs a whole team, and why the team members never talk to each other.
Enjoyed this? Get every new post by email with the Subscribe to the blog form at the bottom of this page. You can also sign up as a site member (top of the page) to follow along.
Agents, from the ground up · © Carolina Steadham



Comments