AI Agent Architecture: What Separates Agents That Ship From Demos That Impress

There is a specific moment that happens in almost every agent project. Week two, the prototype works. Somebody records a screen capture where the agent reads a support ticket, checks the order database, decides the customer is owed a refund, issues it, and writes a polite closing note. The room is genuinely impressed, because it is genuinely impressive. Then the system meets a thousand real tickets and the mood changes. It refunds an order that was already refunded. It confidently tells a customer their delivery is arriving Tuesday when the record says no such thing. It loops eleven times on a ticket written in fragments by someone typing on a phone.

Nothing broke. The model did not get worse between week two and week six. What happened is that the demo was measuring a different thing from the one the business cares about. A demo measures whether the happy path is possible. Production measures what fraction of a messy real distribution the system handles correctly, and what it does with the remainder. Those two questions have almost nothing to do with each other, and the gap between them is filled entirely with architecture.

This article is about that architecture. It is the set of decisions we make on every AI agent development engagement, in roughly the order we make them, along with the failure modes each decision is there to prevent. None of it is exotic. Most of it is the unglamorous discipline of deciding in advance what the system is allowed to do, how you will know whether it did the right thing, and what happens when it did not.

What an agent actually is, stripped of marketing

Strip away the vocabulary and an agent is a loop. A model receives a goal and a description of the tools available to it. It produces either an answer or a request to call a tool. If it calls a tool, the result is appended to the conversation and the loop runs again. It repeats until the model declares the task complete, a limit is reached, or something intervenes. That is the whole mechanism. Every framework you have heard of is an opinionated wrapper around those few lines.

Understanding that the mechanism is simple matters, because it locates the difficulty correctly. The hard parts are not in the loop. They are in the four things surrounding it: which tools exist and what they are permitted to touch, what context the model can see at each step, how you decide whether an output is acceptable, and what the system does when it is not. Teams that treat agent building as a framework selection problem spend three months evaluating orchestration libraries and still ship something unreliable, because the library was never the constraint.

“A chatbot that is wrong wastes a minute of someone's time. An agent that is wrong has already done something. That single difference is the entire reason agent architecture exists.”

Decision one: scope to an outcome, not to a capability

The most common cause of a failed agent project is a mandate written as a capability rather than an outcome. Compare two briefs. The first says the agent should handle customer support. The second says the agent should resolve delivery status enquiries for orders that shipped in the last thirty days, end to end, and hand everything else to a person with the order record attached. The first is unbuildable, unevaluable, and will be judged a failure no matter how good it is. The second can be built, measured, defended, and expanded.

Narrow scope is not a lack of ambition. It is what makes ambition survivable. A tightly scoped agent has a knowable input distribution, which means you can assemble a test set that genuinely represents what it will meet. It has a definition of success you can compute rather than debate. It has a small tool surface, which means the security review is tractable. And when it works, it earns the organisational trust that buys you permission to widen it. Every broad agent we have seen succeed started life as a narrow one.

The practical test we apply during scoping: can you write down, in one sentence, what a correct completion looks like, in a way that a colleague from a different department would agree with? If you cannot, you do not yet have a scope. You have an aspiration, and building against an aspiration produces a system that everybody evaluates against a different private standard.

Decision two: design the tools before the agent

An agent is exactly as dangerous as its tools allow and exactly as capable as its tools permit. We design the tool layer first, before any orchestration logic exists, because the tool layer is where both the capability and the entire risk surface live. A tool definition is a contract: this is the operation, these are the typed parameters, this is the data it returns, and this is the blast radius if it is called with the wrong arguments.

Least privilege is a design constraint, not a hardening step

A tool called runQuery that accepts arbitrary SQL is convenient and indefensible. A tool called getOrdersForCustomer that accepts a customer identifier and returns a fixed shape is slightly more work and removes an entire category of incident. The same principle applies to writes. Rather than one updateRecord tool, define issueRefund, rescheduleDelivery, and applyAccountCredit, each with its own validation and its own permission. The agent then cannot express an action you did not intend to allow, because there is no vocabulary for it.

Every write should be reversible, idempotent, or gated

Classify each tool that changes state into one of three buckets. Reversible actions can be undone programmatically, and the undo path should be tested rather than assumed. Idempotent actions can be repeated safely, which matters enormously because retries and loops will happen. Gated actions are neither, so they require explicit human approval before execution. Refunding money is the classic gated action. Sending an email to a customer usually is too, at least until the system has earned otherwise, because you cannot unsend it.

Tool descriptions are prompt engineering

The text describing a tool is the primary interface between your intent and the model's behaviour. Vague descriptions produce misuse that looks like model stupidity but is actually a specification failure. State what the tool does, when it should be used, when it should not, what the parameters mean in business terms, and what an empty or error response signifies. We routinely improve agent reliability by ten points or more without touching the orchestration logic at all, purely by rewriting six tool descriptions.

Decision three: choose how much control flow you hand to the model

There is a spectrum here and the industry conversation tends to pretend there is only one end of it. At one extreme, the model decides everything: which step comes next, when to stop, how to recover. At the other, your code owns the flow and calls the model only for the specific judgements it is good at, such as classifying intent, extracting fields, or drafting language. Most production systems that work sit much closer to the second end than the discourse would suggest.

The reason is compounding error. If each step of an autonomous plan is ninety five percent reliable, a ten step plan completes correctly around sixty percent of the time. Nobody accepts sixty percent on a workflow that touches customers or money. Fixing that by improving the model is slow and expensive. Fixing it by removing steps from the model's control is immediate and free. If your process genuinely has a fixed shape, encode the shape in code and let the model handle only the parts that require judgement.

Full autonomy earns its place where the path genuinely cannot be known in advance: open ended research, diagnostics where each finding determines the next question, or workflows with a long tail of shapes too varied to enumerate. Even then the right pattern is usually a bounded loop with a step limit, a budget limit, and a supervisor that can halt it. Autonomy is a tool for handling variance, not a badge of sophistication.

Decision four: engineer the context, not just the prompt

Every agent turn is a fresh call carrying whatever history you chose to include. On a long task that history grows until it is expensive, slow, and paradoxically less useful, because important instructions from the beginning get lost among forty tool results. Context engineering is the discipline of deciding what the model sees at each step, and it has more effect on agent quality than almost anything else you can tune.

• Summarise completed sub tasks rather than carrying their full transcripts forward. A four hundred token summary of what was learned beats four thousand tokens of how it was learned.
• Return the minimum useful shape from tools. A database tool that returns forty columns when the agent needs three is burning budget and adding distraction on every single call.
• Restate the goal and the constraints periodically. Instructions given once at the top of a long conversation lose influence as the context fills.
• Keep durable facts in structured state your code owns, not in the conversation. Order identifier, customer tier, and approval status belong in a state object you inject deterministically, not in prose the model has to locate and reread.
• Set a hard step budget and a token budget per task. Runaway loops are a cost incident and an availability incident at the same time.

Where retrieval is part of the picture, the same principles from generative AI development apply directly: chunking strategy, reranking, and citation matter as much inside an agent as they do in a question answering system. The difference is that an agent will act on what it retrieves, so retrieval precision stops being a quality metric and becomes a safety one.

Decision five: verification is a component, not an aspiration

The single most reliable structural improvement we make to agents is adding a verification step that does not trust the model that produced the work. This is not the model checking its own output, which is weak, because a model confident enough to be wrong is usually confident enough to approve itself. It is a separate mechanism with different assumptions.

Deterministic checks first

Before any model based review, run the checks that are simply code. Does the refund amount match a line in the order? Is the date in the future? Does the customer identifier exist? Is the total within policy limits? These cost nothing, never hallucinate, and catch a surprising share of real errors. Every deterministic check you can write is one fewer thing you are trusting a probabilistic system to get right.

Then an independent reviewer for the parts code cannot judge

For judgement that cannot be expressed as a rule, such as whether a drafted message is appropriate in tone and factually consistent with the record, a second model call with a reviewer role and no visibility into the first model's reasoning is a genuine improvement. Give it the evidence and the proposed action, not the deliberation. Reviewers that see the original chain of thought tend to be persuaded by it.

Then a human, on the cases that warrant one

Route to people by expected cost of error rather than by model confidence alone. A high confidence action worth eleven thousand dollars deserves more scrutiny than a low confidence action worth four. Confidence scores from language models are poorly calibrated in absolute terms, though they are useful for ranking. Combine them with business value and you get a triage policy that is defensible to a risk committee.

Evaluation: the part everybody skips and everybody regrets

You cannot improve what you cannot measure, and you cannot safely deploy what you have not measured. Yet evaluation is consistently the first thing cut when a deadline tightens, because it produces no visible feature. The result is a team that changes a prompt, feels that the output looks better, ships it, and discovers a fortnight later that a different case class regressed badly.

A workable evaluation setup is less work than it sounds. Assemble one hundred to three hundred real cases with known correct outcomes, drawn from actual history rather than invented. Weight the set toward the messy middle, not the clean examples. Score each run on task completion, tool call correctness, and any domain specific criteria that matter. Run it on every meaningful change, in continuous integration, and treat a regression as a build failure rather than a discussion topic.

The discipline this creates is worth more than the numbers themselves. Once a team can measure, arguments about whether a change helped end in minutes instead of weeks, and the temptation to ship on vibes disappears because there is a faster way to find out.

The failure modes we see most often

1. 01The agent loops. It calls the same tool repeatedly with slight variations because the result never satisfies it. Fix with step budgets, loop detection on repeated calls, and tools that return an explicit signal when there is genuinely nothing to find.
2. 02The agent invents a parameter. It calls a tool with a plausible identifier that does not exist. Fix with strict schema validation and error responses that state what is wrong rather than returning an empty result the model reads as absence.
3. 03The agent stops too early. It declares success having done part of the job. Fix with an explicit completion checklist the verification step evaluates independently.
4. 04The agent is confidently wrong about your business. It applies a general assumption where you have a specific policy. Fix by moving the policy into the tool layer or the deterministic checks, not into a longer prompt.
5. 05The agent degrades quietly after a model update. Output shifts subtly and nobody notices for weeks. Fix with scheduled evaluation runs and alerting on score movement, not just on errors.
6. 06The agent is compromised through content it reads. A support ticket contains text instructing it to ignore prior rules. Fix by treating all retrieved content as untrusted data, never as instruction, and by ensuring permissions make the worst case boring.

Observability: you will need to explain what it did

The first time an agent does something surprising, somebody senior will ask why. If the honest answer is that nobody can reconstruct it, the project's remaining life is measured in weeks. Trace every run: the goal, each model call with its inputs and outputs, each tool invocation with arguments and results, the verification outcome, the final action, the latency, and the cost. Make it searchable and make it replayable.

This is the same practice we apply in site reliability engineering, applied to a system whose failures are semantic rather than structural. An agent rarely throws an exception when it goes wrong. It returns a well formed, confidently worded, entirely incorrect result. Your monitoring has to be built for that failure shape, which means watching output quality and escalation rates alongside error rates and latency.

Cost and latency are architectural, not incidental

An agent that takes twelve steps costs roughly twelve times a single call, and the context grows at each one, so the real multiplier is worse than linear. At pilot volume nobody notices. At production volume it becomes the largest line in the budget, and the conversation about whether the automation is worth it gets reopened by someone in finance holding an invoice.

• Route by difficulty. Use a small fast model for classification, extraction, and routing, and reserve the expensive model for the steps that genuinely need it.
• Cache aggressively. System instructions, tool definitions, and stable context should hit a prompt cache on every call rather than being re sent and re billed.
• Trim context. The largest cost lever in most agents is not model choice, it is how much text is carried into each turn.
• Run what can be parallel in parallel. Three independent lookups should not be three sequential round trips.
• Measure cost per completed task, not cost per token. Per token pricing tells you nothing about whether the unit economics work.

A realistic timeline

For a well scoped agent on a workflow that genuinely suits automation, six to ten weeks from kickoff to supervised production is a normal range. Roughly two weeks go to scoping, tool design, and assembling the evaluation set, which feels slow to everyone and saves a month later. Three to four weeks go to building and iterating against the evaluations. Two weeks go to hardening, observability, and the escalation paths. Then a shadow period where the agent runs against real traffic without acting, followed by a period where it acts with approval, followed by autonomy on the case classes it has demonstrably earned.

That last progression is the part organisations most want to skip and most regret skipping. Shadow mode is where you discover that the real input distribution contains a category nobody mentioned. Approval mode is where the operations team develops a calibrated sense of when to trust it, which is the thing that actually determines whether the system gets used after you leave.

What good looks like

A production agent worth having is narrow, boring, and trusted. It does one valuable thing reliably. Its tools are specific and its permissions are tight. Its control flow is mostly code, with the model supplying judgement at the points that need it. Every state change is verified before it lands. Every run is traced. Its quality is a number that someone watches, and when that number moves the team finds out from a dashboard rather than a customer.

None of that demos well. You cannot make a compelling video about an agent that correctly declined to act and escalated with a clear summary. But that behaviour, repeated across thousands of cases, is what changes an operational cost line, and it is the only version of this technology that is still running a year after the launch announcement.

If you are weighing an agent against a simpler workflow automation, the honest question is whether your process genuinely has variance the model needs to navigate. Where the path is predictable, deterministic automation with a model in a supporting role will be cheaper, faster, and more reliable. Agents are the right answer for genuine variance, and an expensive answer for everything else.