The use case for milions of AI users today is simple: give the agent a task, and enjoy the result. We might argue that’s not the ideal way to do things, but it’s just what it’s like. So, how do we know whether an AI agent is trustworthy?
For Daniel (Dan) Maher, Senior Technical Advocate and Researcher (STAR) at Datadog, there’s no “silver bullet.” To evaluate whether a model is trustworthy, you need to understand the model and separate it from its harness, and then you just have to “deal with a certain degree of uncertainty, run your tests, try to get your answers, and loop back over them until you start to see what you believe to be correct.”
Even so, relying on AI can be a double-edged sword. Dan said using it for research today is mostly reliable. AI tools keep getting better, and we’re seeing many interesting breakthroughs in science and other technical fields. But a human should always stay in the loop. Dan explains:
Currently, state of the art today is yes. Humans have to stay in the loop, even with concepts like LLM as judge, for example. The humans have to stay in the loop because that’s where we’re at technologically. Going forward, that may less and less be the case from a practical perspective. However, there may be legal or jurisdictional reasons for which a human has to remain responsible for publishing or taking responsibility for those results.
Simply choosing a “better” or “smarter” model won’t make the overall experience of using it more reliable and solid. Dan explained that, while we as users think that models have different personalities, oftentimes that’s the personality of a harness:
What many people think of as a model’s personality is in fact a harness’s personality. And this is an area where a lot of people, engineers alike, I did for a long time not understand this distinction. And so learning to understand the interaction between the model and the harness is one thing.
He added that some models themselves do like to overthink, while others will churn on things forever. In order to deal with the problem of reliability, the best thing we as users can do is to find out what makes each model tick and how it behaves, so that we know how to handle them better.
Think of the AI model as the engine of a car, or brain of the operation, but it sits in a jar and can only instruct and think. The harness is everything else; the wheels and tires or arms and legs it needs to actually do things.
We often feel like AI is unique in the way that it confidently answers questions without being 100% sure of what it’s talking about but Dan pointed out that we, as humans, also do the same:
I am capable of giving you an answer to any question you ask. Now, whether that’s the correct answer is a completely different thing, right? So the idea here, of course, being that we have to figure out, is the AI correct?
What makes a difference between an “answering machine” and a model that is actually reliable and helpful is whether the model provides a trail of information. This means that it’ll tell you where it looked, what it found and what decisions and determinations it’s doing with that information. Having this information, the model can show it to you as the user and you can decide what’s valuable and what isn’t. For Dan, that’s a much more valuable result than a confident incorrect answer.
Echoing that statement, we wanted to know where do AI agents fail first, then?
The answer is, of course, both simpler and more complex than one might think. Dan explained that AI agents will usually fail at the first thing they do, so users will run into problems with each step of the way. Though, that’s not something new to AI:
This is software engineering, right? You write a line of code, you test it, it fails. You fix it, you write the next line of code, you test it, it fails. You fix it. It’s the same thing with agentic behavior. You have to run it, let it fail, understand why it failed, and then move on to the next thing, be it at the orchestration phase, be it at the inference phase, be it at any of the phases. The phase where you’re at for the first time is the first place it will fail.
Testing AI agnets and their workflows might be counterintuitive to the end users who expect that the agents will be faultless from the start, but if we view them as software that needs updates (which they, in the end, are), we can go further. Of course, updates here mean that we need to correct them and steer them in the right direction, rather than just letting them do their own thing while blindly believing them.
Users tend to rely on benchmarks and scores for AI model reliability or level of capability, and measure it with numbers. This means that model X might be a certain percent better than model Y, which, in theory, makes it superior. Dan agrees that models should have some sort of benchmark of reliability and performance so that users can figure out what to use.
On the other hand, the benchmarks today can’t really tell us whether the model is fit for purpose for a particular goal:
In that case, that’s where the practitioner, the organization, the person manipulating the tool needs to come in with their experience, their expertise, their domain knowledge, and be analyzing those outputs, and be analyzing a lot of outputs, and then running those outputs through their own experimentation phases. And this is where something like LLM observability tooling and LLM experimentation tooling becomes increasingly necessary for any organization doing serious work in this space.
The overarching conclusion of our talk with Dan is that AI agent reliability is a very empirical engineering problem and it has much less to do with the model selection than one might thin. The teams treating it as the latter are likely to be very surprised by the former.
