Bike Riding Robot
← All postsMarkdown →

Testing and Trusting AI

I've been thinking about what it means to trust an AI system.

Not whether I trust AI in some abstract sense. I mean something much more practical. The old world of software engineering meant we could be reasonably explicit about what a system should do: we write requirements, we write code, we write tests, we run those tests, we find bugs and fix them. There are, of course, plenty of ways for conventional software to fail. But there's a useful relationship between the thing we build and our ability to understand its behaviour.

AI makes that relationship much less clear.

What does it mean to test an AI?

Suppose I build a chatbot for teachers. I want it to be helpful, accurate and appropriate, but I can't write a test for every possible conversation. Instead, I might create a dataset of examples: teachers and other experts look at those examples and decide what a good response looks like. I can then run my AI system against that dataset, changing the model, the prompt, the context I give it, or the way retrieval works. Each of these becomes an experiment. The experiment produces a run. The run produces results. And those results can be scored.

This is what we generally call an evaluation, or eval, and it's become its own small discipline. Stanford's Center for Research on Foundation Models built HELM, an open framework that scores language models across accuracy, robustness, fairness and other dimensions at once rather than a single leaderboard number, precisely because no single metric captures what "good" means for a general-purpose model.[1] For more open-ended tasks, where there's no single correct answer to check against, a common approach is to use another AI system as the judge: comparing a model's output against reference answers, or against another model's output, and using that comparison as the score. Researchers at UC Berkeley found this "LLM-as-a-judge" approach could match human preference about as often as two humans agree with each other, which is a genuinely useful proxy, but one that also inherits every bias baked into whichever model is doing the judging.[2]

An eval can turn something that otherwise feels subjective into something we can measure. Perhaps I want an AI system to identify particular topics in a piece of text: a human labels the topics, the AI attempts to identify them, and I compare the two. Or perhaps I want the system to determine whether a student's response is a valid reflection: again, I create examples that humans have assessed and compare the AI's behaviour against them. Suddenly I have something much more useful than "I tried it and it seemed pretty good." I have evidence.

But evidence isn't the same as trust

This is where I think things get interesting.

A score from an evaluation doesn't mean an AI system is trustworthy. It means that, under a particular test, against a particular dataset, using a particular definition of success, the system behaved in a particular way. That's useful. But the quality of that evidence depends on everything that went into producing it. Who created the dataset? Does it represent the real world? Who decided what a good answer looks like? What did we choose not to measure? What happens outside the test?

And perhaps most importantly: what happens after we give the system to real people?

This is where AI starts to feel different from traditional software engineering.

The real system starts when you deploy it

Imagine I build that chatbot for teachers. I test it extensively, build a dataset, run experiments, compare different approaches, and improve the system until I'm happy with its results. Then I give it to a teacher. The teacher gives it to thirty students.

Now something has changed. Those students are going to use the system in ways I didn't anticipate. They will ask questions I didn't think to include in my dataset, find edge cases, misunderstand things, and discover behaviours that weren't obvious during development. Perhaps the system works extremely well for 95% of interactions but has a particular failure mode I didn't know existed.

If I have no way of seeing what happens after deployment, I don't really have a feedback loop. I have tested a system. Then I've handed it over.

That feels like an incomplete form of engineering.

Evaluation and observability

This is why I think evaluations and observability need to be thought about together, not sequenced as "test, then ship, then hope."

An evaluation tells me something about how my system performs before I deploy it. Observability tells me something about how it behaves after I deploy it. The two should feed each other: a real-world interaction reveals a failure, that failure becomes a new example in the evaluation dataset, the evaluation tests a change, the new version deploys, its behaviour is observed again. The cycle continues.

Anthropic's own engineering team describes this as a layered system rather than a single gate, borrowing the "Swiss cheese model" from safety engineering: automated evals catch known failure modes before launch, production monitoring catches distribution drift and the failures nobody wrote a test for, and no single layer is expected to catch everything on its own.[3] That last point matters more than it sounds. A model doesn't have to change for its behaviour to drift: the world it's operating in changes, the questions people actually ask it drift from what was anticipated during development, and any retrieval pipeline feeding it context can quietly start surfacing different material.[4] None of that shows up in a pre-deployment eval, because a pre-deployment eval is frozen in time and production traffic is not.

This is familiar territory for software engineers. We already do versions of this with logs, metrics, traces, error reporting, monitoring and automated tests. But AI makes the feedback loop much more important, because the behaviour of the system is no longer completely determined by the code we wrote. The system can surprise us. And if it can surprise us, we need mechanisms for finding out when it does.

The problem gets harder as the consequences increase

Not every AI system needs the same level of scrutiny. If I use an AI tool to rewrite an email, the consequences of a bad answer are relatively small. If an AI system is helping a teacher understand a student's work, the consequences are greater. If it is personalising learning, recommending interventions or making decisions that affect a student, they are greater again.

This isn't a novel idea; it's the organising principle behind most of the AI governance frameworks that have emerged over the last few years. NIST's AI Risk Management Framework asks organisations to calibrate their governance, measurement and monitoring to the actual risk profile of a given system, rather than applying one bar to everything with "AI" in the name.[5] The EU AI Act goes further and writes the tiers into law: systems are sorted into unacceptable, high, limited and minimal risk categories, with the compliance burden, documentation and human-oversight requirements scaling directly with how much harm a failure could cause.[6] Neither framework was written with school chatbots specifically in mind, but the underlying logic transfers cleanly: the engineering effort should reflect the consequences, not the novelty of the technology.

For some systems, it might be enough to establish that the infrastructure is secure, privacy requirements are met and the product generally works as intended. For others, we need much more evidence about behaviour. We need to understand:

These aren't just questions about AI. They're engineering questions.

We are missing a common language

This is perhaps the part that interests me most.

Inside an AI engineering team, it is possible to have quite sophisticated conversations about this. We can talk about datasets, experiments, runs, evaluators, scores, model versions, prompts, retrieval and observability. But the person buying the product probably doesn't want to understand any of that. Nor should they have to.

This is something engineering disciplines have historically been quite good at solving. A structural engineer can produce evidence that a building meets particular standards without asking the person buying the house to understand structural engineering. A manufacturer can demonstrate that a car has passed safety testing without asking the driver to understand crash-test methodology; Euro NCAP has published a simple star rating since 1997 for exactly this reason, so a buyer never has to read the underlying test protocol to act on the result, and cars sold in markets that adopted it got measurably safer as manufacturers designed to the rating.[7] The technical evidence exists underneath the communication of trust.

I think AI needs something similar. Not necessarily one universal standard. Not necessarily a giant certification framework. But some common language for communicating the evidence behind an AI system. ISO's new management-system standard for AI, ISO/IEC 42001, is one early attempt at exactly this: a certifiable standard organisations can be audited against, covering risk management, system lifecycle and impact assessment, so that "this organisation is ISO 42001 certified" can eventually mean something to a buyer the way "five-star ANCAP" already does.[8] It's early, and certification isn't the same thing as evidence of good behaviour, but it's the right shape of attempt.

There are already pieces of this

There are already organisations and frameworks thinking about parts of this problem.

In education, for example, Safer Technologies 4 Schools provides a framework, run by Education Services Australia in consultation with every state, Catholic and independent schooling sector, for assessing technology used by schools, including areas such as privacy, security and governance; it now includes a dedicated AI module.[9] That's important. But there is a distinction between asking "is this technology safe and appropriately governed?" and "does this AI actually behave well for the thing we're asking it to do?" Both matter, and the second question becomes increasingly important as AI systems move beyond simple software features and start interacting with people in more complex ways. An AI system can be secure, private and well-governed while still being bad at its job. Conversely, an AI system might perform extremely well in an evaluation while being inappropriate to deploy because of privacy, security or governance concerns. Trust needs both sides.

There's also a growing effort to treat AI failures the way aviation treats near-misses: as shared, cross-organisational learning rather than each company's private postmortem. The AI Incident Database, maintained by the Responsible AI Collaborative, now indexes well over a thousand real-world reports of AI systems causing harm, specifically so the same failure mode doesn't have to be independently rediscovered by every team that ships something similar.[10] That's the observability half of this problem playing out at an industry level, not just inside a single product.

This feels like a new engineering discipline

I've started to wonder whether we're watching the emergence of another layer of software engineering. Not just building AI models. Not just building applications around them. But engineering the behaviour of systems whose behaviour is partly unpredictable. That means developing practices for evaluation, monitoring, feedback, intervention and accountability, and then figuring out how to communicate those practices to everyone else.

We're still very early. A lot of this is R&D. The tools are changing quickly. The models are changing quickly. The techniques for evaluating them are changing quickly. There aren't decades of established practice sitting underneath all of this. That's unusual: most engineering disciplines have had time to develop their standards, professional practices and shared assumptions. AI is moving much faster than that process normally happens. So perhaps we shouldn't be surprised that the language of trust hasn't caught up yet.

Trust has to be a feedback loop

I don't think the answer is to promise that an AI system will never make a mistake. That isn't realistic. Instead, I think trustworthy AI systems will increasingly be defined by how well we can understand and respond to their mistakes.

Before deployment, we need evidence. After deployment, we need observation. When something goes wrong, we need feedback, and that feedback needs to make its way back into the engineering process. The evaluation gets better. The system gets better. Our understanding of its limitations gets better. And, importantly, the people using the system can understand what they're trusting.

That feels like a much more useful definition of trust than simply saying that an AI product has been "tested." Because testing is an event. Trust is a system.

And as AI systems become more capable, more general and more deeply embedded in decisions that affect people, I suspect building that system of trust will become just as important as building the AI itself.

References

  1. Stanford Center for Research on Foundation Models, Holistic Evaluation of Language Models (HELM). ↩
  2. Lianmin Zheng et al., "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena", NeurIPS 2023 Datasets and Benchmarks Track. ↩
  3. Anthropic, "Demystifying evals for AI agents", 9 January 2026. ↩
  4. Weights & Biases, "Evaluating LLMs in production: From drift detection to continuous monitoring". ↩
  5. National Institute of Standards and Technology, AI Risk Management Framework (AI RMF 1.0), 26 January 2023. ↩
  6. European Parliament and Council, Regulation (EU) 2024/1689 laying down harmonised rules on artificial intelligence (the "AI Act"), 13 June 2024. ↩
  7. Euro NCAP, "Helping you choose the safest cars since 1997"; the Australasian New Car Assessment Program (ANCAP) adopted the Euro NCAP protocol in 2018. ↩
  8. International Organization for Standardization, ISO/IEC 42001:2023 — AI management systems, December 2023. ↩
  9. Safer Technologies 4 Schools, "Responsible AI", administered by Education Services Australia. ↩
  10. Responsible AI Collaborative, AI Incident Database. ↩