top of page
All Posts


What Is an AI Agent, Really?
Agents, from the ground up · Part 1 of 7 I asked an AI agent to write tests for a security detector. Its first attempt failed. It read the error, fixed its own mistake, ran the tests again, and they passed. I didn't touch a thing. What was actually happening there isn't magic, and you don't need any jargon to understand it. One simple mental model explains every AI agent you'll meet. Once you have it, the buzzwords that come later (subagents, orchestrators, memory, MCP) all s
carocsteads
2 days ago4 min read


Agents vs. Workflows: Not Everything Called an "Agent" Is One
Agents, from the ground up · Part 2 of 7. In Part 1 we defined an agent as a model + a loop + tools. Now for the distinction that cuts through most of the hype. Open any product page today and everything is an "agent." The email summarizer is an agent. The chatbot is an agent. The thing that renames your files is an agent. Most of them aren't, and that's fine. There are two fundamentally different ways to build with LLMs, and knowing which one you're looking at changes how yo
carocsteads
3 days ago4 min read


The Anatomy of an AI Agent: Same Model, Different Job Description
Agents, from the ground up · Part 3 of 7. If an agent is "a model + a loop + tools," what exactly goes into one? Five ingredients, and they demystify the whole thing. When people hear "we have a test-writing agent and a code-review agent," they often picture two different AIs, each trained for its specialty. That's almost never how it works. Open up any agent and you'll find the same five parts. Change those parts, and the same model becomes a completely different worker. The
carocsteads
4 days ago4 min read


Orchestrator and Subagents: A Team of Agents, and Why They Don't Talk to Each Other
Agents, from the ground up · Part 4 of 7. One agent is useful. A coordinated team of them is how agent systems scale. Here's how that team is actually organized. Some tasks are too big for one agent's desk: "audit every detector in the codebase," "write tests for these twelve modules," "research five competitors and compare them." Stuffing all of that into one context window means it fills up with noise, the agent loses track, and everything happens one slow step at a time. T
carocsteads
5 days ago4 min read


Memory vs. Context: The Desk and the Filing Cabinet
Agents, from the ground up · Part 5 of 7. "Memory" is the most overloaded word in AI agents, because it means two completely different things. Let's untangle them. People say "the agent remembered my preferences" and "the agent forgot what we were doing" in the same breath, as if it's one mechanism that sometimes works and sometimes doesn't. It's actually two separate mechanisms, and once you can tell them apart, a lot of confusing agent behavior suddenly makes sense. 1. Cont
carocsteads
6 days ago4 min read


Where Does the Agent Run? Four Ways to Run the Same Brain
Agents, from the ground up · Part 6 of 7. Several "Claude" products sound alike. One question separates them cleanly: where does the agent loop actually run? Chat app, CLI, SDK, Agent SDK, managed agents… the product names blur together fast. But remember Part 1: an agent is a model + a loop + tools. Every one of these products calls the same family of models underneath. What differs is who runs the loop, where it runs, and whose tools it uses. Answer that, and the product li
carocsteads
7 days ago3 min read


The Golden Rule of AI Systems: AI Generates, Deterministic Checks Decide
Agents, from the ground up · Part 7 of 7. If you remember one thing from this whole series, make it this. It's the principle that separates a flaky AI demo from a system you can actually ship. Let the model generate and triage. Let deterministic code decide pass/fail. The model never sits in the final gate. Everything in the first six parts (loops, tools, specialists, subagents, memory) is about making agents more capable. This part is about making them trustworthy. Those are
carocsteads
Nov 94 min read


Guardrails: How to Build Secure AI Agents (Lessons from FinBot CTF)
Agents, from the ground up · Bonus. Everything I've learned about keeping agents safe from building and testing an AI-security CTF, with no challenge spoilers. FinBot is the OWASP GenAI Security Project's agentic AI capture-the-flag app, built as part of its Agentic Security Initiative (think "Juice Shop for agentic AI"). It simulates a financial-services company as a multi-agent system for vendor management: an orchestrator agent coordinates a team of LLM-powered specialists
carocsteads
Nov 815 min read


Agents Testing Agents: How I QA Multi-Agent Systems Without Letting AI Grade Its Own Homework
Agents, from the ground up · Bonus. A QA engineer's playbook for testing multi-agent apps, using a team of AI agents as testers and plain code as the referee. Here's the uncomfortable truth about multi-agent apps: the traditional testing playbook doesn't fit them. You can't write a test for every path, because the agents choose the path at runtime. You can't enumerate the inputs, because the input is human language. And the worst bugs don't live inside any one agent. They liv
carocsteads
Nov 77 min read
Guardian of Quality Realms: My Role on OWASP FinBot CTF
OWASP FinBot CTF calls itself test Heading oneOWASP FinBot CTF calls itself "the Juice Shop for Agentic AI." It's an intentionally vulnerable, multi-agent vendor-management platform. LLM agents handle onboarding, fraud detection, invoices, payments, and communications, and they have real MCP tool access. Players try to break those agents with prompt injection, policy bypass, tool poisoning, data exfiltration, and remote code execution. An event-driven pipeline detects success
carocsteads
Oct 302 min read
Test Creation Agents: Making Every Pull Request Show Its Own Test Gaps
My latest FinBot pull request (#615) takes on a problem every growing open-source project hits. Code arrives faster than anyone can review it, especially now that so much of it is AI-assisted. Why At the time, FinBot's CI ran only four test files. A pull request could change a detector or an agent without running any related tests. As a concrete example, three tests in the specialized-agents suite had been failing on main since they were merged, and nothing caught it, because
carocsteads
Oct 302 min read
WEEK 6 - Testing Four MCP Servers: What Each One Revealed
Meta description: Server-level tests for FinBot's four MCP servers (FinMail, FinDrive, SystemUtils and TaxCalc), and the access gaps and missing validation they uncovered. PRs #316 through #319 add server-level tests for four of FinBot's MCP servers. Each server gives AI agents a different set of tools. Together the tests show a pattern: the checks that exist mostly work, and the problems are in the checks that are missing. A Test Every Server Gets: The Exact Tool List Each P
carocsteads
Sep 164 min read
WEEK 5 — Testing MCP: When Your AI Agent's Tools Live on a Server
Meta description: How FinBot tests its MCP layer: repositories, session scoping, and the server factory and tool provider, plus the vendor access gaps the tests exposed. Most people building with AI agents are familiar with function calling. You define a list of tools, pass them to the LLM, and the LLM calls them by name. Model Context Protocol (MCP) goes a step further. Instead of defining tools inside the application, the agent connects to an MCP server, a separate compone
carocsteads
Sep 165 min read
WEEK 4 - Testing the Tools Between AI Decisions and Real Actions
Meta description: How FinBot tests AI agent tools: why schemas and functions need separate tests, plus namespace isolation, payment safeguards and the bugs the tests found. When an AI agent decides to take an action, such as looking up a vendor, approving an invoice or starting a payment, it does so by calling a tool. Tools connect what the LLM decides to what actually happens in the system. They are also one of the least-tested layers in AI applications. PR #227 adds FinBot
carocsteads
Sep 165 min read
Week 3: Testing the Challenge Detection Layer
Most test suites verify that a system does the right thing when inputs are normal. FinBot adds a harder requirement: verify that the system notices when something wrong is happening, even when the wrong thing looks normal on the surface. That is what the CTF detector layer does. And testing it requires a different way of thinking. What Detectors Do Every event that flows through Redis Streams gets evaluated against a set of detectors. Each detector answers one question: did t
carocsteads
Mar 273 min read
Week 2: How I Approach Testing AI Systems (And Why It's Different)
How I Approach Testing AI Systems (And Why It's Different) Most testing advice assumes you control what your code returns. With AI systems, you don't. The model decides. That changes everything about how you write tests. Here's the approach I use when testing LLM integration layers — the classes that sit between your application and the AI provider. The One Rule: Mock the Service, Never Your Code When I test an AI client, I mock the network call to OpenAI or Ollama — not the
carocsteads
Mar 274 min read
Week 1: Architecture of an AI Financial Platform
I've been working on FinBot CTF — an AI-powered financial platform built for the OWASP Agentic AI project. The goal is to explore what happens when you give AI agents real financial responsibilities: onboarding vendors, processing invoices, flagging fraud, and authorizing payments. But before I write about how I test it, I need to explain what I'm actually testing. Because the architecture is what makes this hard. What Does FinBot Do? FinBot is a vendor management portal
carocsteads
Mar 36 min read
Building a Production-Ready Database QA Automation Framework:
A Journey into Testing at Scale By Carolina Steadham | QA Automation Engineer --- Introduction Testing at scale is less about writing more tests and more about building systems that can be trusted under growth, change, and failure. As applications evolve, databases become the backbone of reliability—quietly enforcing correctness, performance, and consistency across the entire platform. This project began as an exploration into what it really means to test databases and backe
carocsteads
Dec 17, 20257 min read
Automating Privileged Access Management:
A QA Engineer's Journey with Apache Guacamole By Carolina Steadham | QA Automation Engineer Introduction In today's cybersecurity landscape, privileged access management (PAM) is critical for protecting sensitive systems. I recently completed a comprehensive QA automation project focused on testing Apache Guacamole, a clientless remote desktop gateway that provides secure access to RDP, VNC, SSH, and Telnet connections through a web browser. This project showcases my experti
carocsteads
Dec 15, 20253 min read
Building an AI Threat Analytics Framework: A Developer's Journey
How I created a security-focused AI testing framework with Python, pytest, and real-world threat detection By Carolina Steadham | QA Automation Engineer Introduction Security is one of the most critical concerns in today's digital landscape. With AI systems becoming increasingly prevalent in security operations, I set out to build a comprehensive AI Threat Analytics Framework - a proof-of-concept project that demonstrates how AI can be leveraged for threat detection, classi
carocsteads
Dec 12, 20254 min read
bottom of page