The word “agent” has been thoroughly hijacked by marketing departments and tech social media.
Depending on who you ask, an AI agent is an autonomous digital employee, an intelligent life form, or something that will soon replace entire teams.
As engineers, we should be skeptical of descriptions like these.
In our latest Sharpen the Axe session at Techwondoe, I wanted to strip away some of that perceived magic and look at agents for what they really are: a software architecture built around a language model.
Once you adopt that mental model, agentic systems become much easier to reason about—and much more interesting to engineer.
Start with the limitation of an LLM
At its core, a raw large language model is a stateless, non-deterministic token predictor operating on the information available to it.
By itself, it cannot see your database. It doesn’t inherently know the current state of an order. It cannot update a customer record. And it shouldn’t be trusted to invent any of those details.
Imagine a customer asking:
“Where is my latest order?”
A language model without access to the customer’s actual order data cannot reliably answer that question.
This is the problem an agentic architecture helps us solve.
The distinction I find useful is:
An LLM is a model. An agent is an architecture.
An agent wraps a model with three important things:
- Instructions — how the system should behave, what its objective is, and where its boundaries lie.
- Tools — functions that let it retrieve information or perform actions outside the model.
- Context — trusted runtime information about the current user and application state.
The model provides reasoning and language capabilities. The surrounding software determines what it can see, what it can do, and under what conditions it is allowed to do it.
That surrounding software is where much of the real engineering begins.
Why use an Agents SDK at all?
Technically, we can build the orchestration loop ourselves.
Call the model. Parse its response. Detect a tool request. Validate its arguments. Pause execution. Run the appropriate backend function. Capture its result. Add the result to the conversation. Call the model again.
Then repeat until the model produces a final response.
It is possible.
It is also a significant amount of boilerplate with plenty of opportunities for errors.
An agent SDK abstracts this lifecycle.
With the OpenAI Agents SDK, for example, a runner manages a user turn: it takes the agent configuration and user message, communicates with the model, resolves requested tool calls, sends results back into the conversation, and continues until an output is produced.
The important point isn’t the API itself.
It’s that agentic orchestration is fundamentally a runtime loop between a model and controlled software capabilities.
There is no magic hiding inside it.
Tools are where agents become useful
Tools give the model capabilities beyond generating text.
During the session, I grouped them into three useful categories.
Local tools execute within our application environment. They make sense when an agent needs controlled access to internal databases, services, or application logic—for example, retrieving an authenticated customer’s orders.
Hosted tools move certain execution into infrastructure managed by the model provider. They can be useful for capabilities such as sandboxed computation, where executing generated code directly on your application server would introduce unnecessary operational or security risk.
Then there are MCP tools, built around the Model Context Protocol. MCP provides a standardized way for AI applications to communicate with external tools and services.
This becomes particularly interesting when integrating systems such as GitHub or other external platforms. Instead of designing every integration around one model provider’s proprietary tool format, a standard protocol gives us a cleaner interoperability layer.
There isn’t one universally correct tool type.
The engineering question is: Where should this capability execute, who controls it, and what trust boundary should surround it?
Production agents are a software engineering problem
A prototype agent can be surprisingly easy to build.
A production agent sitting on top of a multi-user SaaS product is a different problem entirely.
Now we have to think about authentication, multi-tenancy, conversation state, context windows, structured responses, observability, authorization and security.
Consider identity.
Suppose User B somehow obtains an order ID belonging to User A and tells the agent:
“Cancel this order.”
If we allow the language model to determine customer identity from the user’s prompt, we have created a serious security problem.
Instead, identity should come from the trusted application layer.
Authenticate the request first. Establish the user’s identity and permissions. Pass that information through a trusted runtime context. When a tool executes, sensitive identifiers such as the authenticated customer ID should come from that context—not from whatever the model extracted from the user’s message.
This gives us a crucial security property:
The model can reason about the request without being the authority on who the requester is.
Prompt injection cannot simply turn User B into User A because authorization isn’t based on what the prompt claims.
This is the kind of boundary that turns an interesting AI demo into something we can begin considering for real applications.
Memory needs deliberate engineering too
Models are inherently bounded by context windows, while users expect conversations to continue naturally.
We therefore need a strategy for conversation state.
One option is to persist conversation history ourselves and provide the relevant history on subsequent turns.
But continuously appending messages creates another problem: eventually, the conversation becomes too large.
A practical approach is session compaction.
When the conversation crosses an appropriate threshold, older history can be summarized while a small number of the most recent messages remain verbatim.
Instead of repeatedly sending an ever-growing transcript, the model receives something closer to:
compressed historical context + recent conversation
This reduces context consumption while preserving enough information for the conversation to remain coherent.
Again, this isn’t an exotic AI concept. It is an engineering trade-off involving storage, latency, cost and information fidelity.
Don’t build one agent with 50 tools
One temptation when building an agent is to keep adding capabilities to it.
Need order management? Add some tools.
Need refunds? Add more.
Need analytics? More tools.
Need customer profiles, shipping, inventory and payments? Keep going.
Eventually, one agent is staring at dozens of possible tools every time it needs to decide what to do.
That increases complexity, latency and the chance of inconsistent tool selection.
I prefer thinking in terms of role-specific agents.
A manager agent can interpret a request and invoke specialized agents when necessary. Alternatively, a conversation can be handed off to a specialized agent with focused instructions and a smaller toolset.
This is much closer to good software design: give components clear responsibilities rather than building one enormous component that does everything.
Autonomy should have boundaries
Just because an agent can perform an action does not mean it should be allowed to perform it autonomously.
Consider changing the email address associated with an e-commerce account.
Technically, we could expose an updateEmail tool and allow the model to call it.
But email addresses can be tied to identity, communication and account recovery. A mistake has consequences.
A safer system can pause before execution and ask for explicit approval.
The agent proposes the action.
The application shows the user—or an administrator—what is about to happen.
A human approves or rejects it.
Only then does the tool execute.
The same principle becomes even more important as we move toward payments, destructive actions and access to sensitive information.
The upper bound of what an agent can technically do is largely determined by the tools we give it.
The more important question is:
What should we allow it to do without human intervention?
Guardrails should exist at multiple layers
Security cannot be one giant instruction that says, “Please behave safely.”
Production systems need checks at different boundaries.
We can validate incoming requests before they reach the model. We can validate arguments before tools execute. We can validate tool results. And we can validate the final structured output before the application consumes it.
Structured outputs are particularly useful here.
Instead of asking a model to produce prose and then using brittle regular expressions to extract information from it, we can define the response contract the application expects and validate against that contract.
The broader lesson is simple:
Probabilistic reasoning should be surrounded by deterministic controls wherever possible.
Observability becomes essential
Traditional application logs tell us which functions executed.
With agents, that is only part of the story.
If a customer reports that an AI assistant acted on the wrong order, knowing which SQL query ran isn’t enough. We need visibility into the sequence that produced that action: the request, model interaction, tool call, arguments, guardrail decisions and execution result.
Agent tracing gives us that runtime visibility.
There is an important distinction between tracing and conversational memory.
Conversation history exists so future turns can understand previous turns.
Tracing exists so engineers can understand what happened during execution.
For production systems, both need appropriate persistence and retention strategies.
Without observability, an agent remains unnecessarily close to a black box.
The demo: putting the architecture together
To make these concepts concrete, I built a small agentic customer-support application.
Two users were logged into independent sessions.
Each could ask the assistant to retrieve their orders, and the backend resolved the request using the authenticated user’s context.
I then took an order belonging to one user and attempted to cancel it from the other user’s session.
The request failed.
Although the model could see the supplied order ID, the customer identity used by the backend came from trusted runtime context. The requested order simply didn’t belong to the authenticated customer.
The demo also included streaming responses and a human-approval flow for sensitive account changes.
None of these features individually feels revolutionary.
Together, however, they demonstrate the difference between attaching an LLM to an application and designing an agentic system.
The real opportunity isn’t autonomous intelligence
After spending time building and studying these systems, my biggest takeaway is that we should stop thinking about agents as mysterious autonomous entities.
They are software systems.
Very unusual software systems, certainly. We now have a probabilistic reasoning component capable of deciding which capabilities it needs and how to combine them.
But the fundamentals remain familiar.
We still need clear interfaces.
We still need authentication and authorization.
We still need state management.
We still need observability.
We still need validation.
And we absolutely still need security boundaries.
The exciting part of agentic engineering isn’t that we’re removing software engineering from the equation.
It’s that good software engineering is what makes these models genuinely useful.
That is the mental model I would start with before building any agent.
Key Takeaways
The essentials, distilled.
