You shouldn’t trust AI agents in production

Darko Mesaros, a Distinguished Developer Advocate at AWS, doesn’t trust his AI agents. We spoke with him after his talk at WeAreDevelopers San Jose about where to set limits for them and why production costs deserve more attention.

He thinks that distrust is a sensible place to start. Agents can be wrong, so if you let them touch business operations, you need safeguards around their decisions.

Let’s start with why he doesn’t trust them.

Your talk at WAD San Jose starts with a strong statement: “I don’t trust AI agents.” What exactly shouldn’t developers trust about them?

Darko: The reason I don’t trust my AI agents is that they’re ultimately probabilistic systems. They’re not deterministic. When you give agents serious responsibilities, such as handling refunds, selling things or dealing with sensitive customer data, relying on a probabilistic system is a problem. Agents by themselves, without guardrails around them, are problematic.

I’ve spent a lot of time teaching developers how to approach this. Building agents is more than having an agent and access to an LLM. It’s about building the system around it, with guardrails and everything you need to make the agent a little more secure.

What’s the biggest difference between an AI agent that works in a demo and one that is actually ready for production?

Darko: Believe it or not, cost. When we build an agent for a demo, we might pick the biggest model because we want to make sure the demo works.

The challenge comes when you start scaling that out. When an agent handles hundreds of thousands of customer requests in different ways, cost becomes an issue. Optimization isn’t something you worry about in a demo. You don’t worry about prompt caching, RAG or those things.

In an actual production system, you start to consider cost. Beyond safety and guardrails, that’s the biggest difference between a demo agent and a real-life production agent.

When building an agent, what safety layer should developers add first: guardrails, permissions, observability or something else?

Darko: The answer is yes to all of those things, because proper defense is built in layers. A single layer, a single guardrail, may not prevent everything. You may want guardrails, MCP input validation and deterministic policy checks around certain tool calls. All of those things are important.

As I mentioned in my talk, observability becomes incredibly important because you really don’t know what your agents are doing, especially at that scale. Being able to observe them and run evaluations periodically or in real time gives you a lot more peace of mind. So, to answer the question of which one to add, the answer is yes.

How do you know when an AI agent has made a mistake if its output still looks perfectly reasonable?

Darko: You have to combine a few things, and we go back to observability. Implementing automated tracing is critical so you can trace what’s going on in real time. But tracing doesn’t stop at the agent. You also need to trace the backend system the agent works with.

If you start detecting anomalies, such as an increasing number of small or large orders, or something unusual, you need to look at both the agent and the backend. Combining those signals into a proper anomaly detection system is how you could potentially catch certain things happening in real time.

It’s never just an agent. It’s an agent plus everything else behind it.

How much autonomy should we actually give AI agents in production? Where should developers draw the line?

Darko: Ultimately, responsibility for what happens, for a system or code that goes into production, rests with the developer.

How much autonomy should we give agents? It depends on the blast radius of a change or of a system going down. Is this a simple chatbot that tells customers the opening times of your business? Sure. If something goes wrong, it’s not horrible.

But if the system handles banking information or sensitive data, or processes purchases, you may want to reduce its autonomy. It’s not only about the code or change being submitted; it’s also about the system you’re submitting it to. Unfortunately, the answer here is: it depends.

If a developer is shipping their first production AI agent tomorrow, what are the three things they absolutely need to have in place?

Darko: Number one, proper observability. They need to be able to observe and monitor what their agent is doing, run evaluations against the models and their decisions and choices, and constantly monitor those things for changes.

Number two, policies. A policy language such as Cedar lets you put policies in front of tool execution so you can prevent certain tools from executing.

Number three, consider an established harness. It’s tempting to build your own because we’re all developers, but I encourage developers to consider something like the Strands SDK, LangGraph or LangChain. It makes things easier and lets you focus on your business problem and spend less time figuring out how to wrangle the agent.