Responsible AI has an evidence problem
AI evals are not just technical tests. They are how product teams turn responsible AI from policy into evidence.
Responsible AI is increasingly judged not by the policies an organisation has, but by the evidence it can produce. Here is the uncomfortable truth about responsible AI: most organisations no longer have a policy gap, they have an evidence gap.
In the UK public sector, guidance is no longer scarce. The AI Playbook asks teams to build in meaningful human control, test systems before deployment, monitor drift and hallucinations and keep assurance in place once systems are live. Central government teams using in-scope public-facing or decision-influencing algorithmic tools are expected to publish transparency records. HM Treasury has issued guidance on evaluating the impact of AI interventions. Government has also developed AI Management Essentials to help organisations assess their AI management processes.
The expectations are clear, responsible AI is no longer just a principle. But rather it is becoming something organisations are expected to evidence.
And yet a gap remains, the Public Accounts Committee has warned that transparency is still too slow, assurance of high-risk AI remains difficult to verify, learning from pilots is not being shared systematically and weak data foundations and legacy systems threaten adoption. The same pattern appears outside government, McKinsey and Deloitte both report the same gap: deployment is accelerating while governance maturity lags significantly behind. Only about a third of organisations show stronger controls and only around a fifth say they have mature governance for autonomous agents, even as close to three-quarters plan to deploy agentic AI within two years.
This is why AI evals matter. Not because they are fashionable but because evals are becoming the missing evidence layer in product governance.
What AI evals actually are
The simplest definition is this: AI evals are structured, repeatable ways of generating evidence about whether an AI system behaves acceptably for a specific purpose, risk profile and operating context.
That sounds technical, but it is really just good governance. Evals turn vague claims into evidence. Safe, accurate, fair, compliant, useful, reliable, trustworthy. These are words that are easy to write into a strategies, business cases or assurance documents but much harder to prove in the reality of a live service. Responsible AI cannot rely on intention alone, it has to be tested against behaviour.
This should feel familiar to product managers. Strong product teams are not there just to ship output, they exist to solve meaningful problems in ways that work for users, organisations and the wider system around them. Continuous discovery makes a similar argument in practical terms: teams reduce risk by testing assumptions early and often.
AI evals extend this discipline into AI governance. They ask: what assumptions are we making about this system and what evidence do we have that those assumptions remain true?
Approval is not assurance
Traditional assurance struggles with AI because AI systems are not just another form of software.
They can drift as data, prompts, models and usage patterns change. They are sensitive to wording and context and can hallucinate confidently. They also often depend on third-party models, suppliers and benchmarks whose quality may be uneven. And as systems become more agentic, they stop being tools that merely answer questions and start becoming systems that plan, call tools, trigger actions and operate across workflows.
Benchmarking has a role in understanding this, but it should not be mistaken for governance. A general benchmark might tell you something about a model’s broad capability. It cannot tell you whether a specific service is safe, useful, fair and reliable inside a particular operating context and benchmark quality varies significantly, including among those used by developers and policymakers.
This changes the governance question. As it is no longer enough to ask, “Was this system approved before release?” The better question is, “What evidence do we have that this system is still behaving acceptably now?” That shifts governance away from approval as a moment in time and towards assurance as an operating capability.
The types of evidence product teams need
A model can perform well in isolation and still fail inside a service, product teams need different kinds of evidence.
Behavioural evals ask how the system behaves across realistic scenarios, edge cases and user groups — testing for inconsistency, overconfidence, refusal patterns and failures under ambiguity. Users rarely arrive in neat test cases. They bring partial information, emotional stress, unusual circumstances and messy language.
Quality evals ask whether outputs are relevant, accurate, complete and reliable. For retrieval-based systems, this also means testing whether the system is drawing on the right source material and whether its answer is actually supported by it. A system that sounds plausible is not the same as a system that is reliable.
Safety evals test whether the system can be induced to produce harmful, insecure, misleading or deceptive outputs — through adversarial testing, red teaming, privacy testing and misuse scenarios. Safety cannot be reduced to whether the system behaves well under friendly use.
Policy evals ask whether the system operates within legal, ethical and organisational constraints: fairness, transparency, equality duties, data protection, explainability, human oversight and routes for challenge or correction. This is where AI governance often becomes too abstract, policy evals make it concrete.
Workflow evals test the end-to-end service — what happens before, during and after the AI interaction, how it routes users, when it escalates and whether decisions are recorded properly. Once systems can trigger actions across a process, the unit of evaluation has to be the workflow, not just the answer.
Outcome evals ask the question many governance discussions avoid: does this system actually improve the service in the real world? A system might improve speed but reduce trust, or help internal teams while making the public experience worse. Outcome evals force teams to ask whether the intervention has delivered meaningful value, not just technical performance.
Sustainability evals ask whether the environmental and financial costs of running the system are understood and deliberate, an estimate of what the AI will cost at scale, and an active decision about infrastructure and carbon impact. These questions are easy to defer. They are harder to retrofit once a system is in production.
Basic assurance asks whether the system passed its tests. Responsible product governance asks whether the service is still working for the people who depend on it.
What good looks like in the public sector
The UK already has useful examples of this more evidence-led approach.
GOV.UK Chat is one of the clearest. GDS has described how the service uses automated evaluation, structured manual evaluation, red teaming and live monitoring, assessing outputs against criteria such as groundedness, relevance, factual accuracy, completeness, reliability and reputational safety. Responsible governance is not a principle on a slide. It is a living system tied to explicit evaluation criteria and real user behaviour.
The Department for Transport’s Consultation Analysis Tool offers another useful model. DfT evaluated the tool against human reference datasets using both blind and non-blind designs, reported performance metrics and retained human review — because consultation analysis involves public views, policy sensitivity and judgement. This is a credible template for public-sector AI.
The Algorithmic Transparency Recording Standard points in the same direction. DVLA’s natural-language IVR record reports that the system handles around 900,000 customers a month and has reduced time spent navigating it by over 50%. HMRC’s digital assistant record explains how conversation data is reviewed to improve performance.
These systems are not governed responsibly simply because they sit in the public sector. They are governed responsibly only to the extent that they produce evidence, retain review routes and can be monitored and changed under control.
What this looks like in practice
Two examples from recent work show what applying this looks like, and what it produces.
The first came from using Claude to build an eval suite for Claude Code — Anthropic’s agentic coding tool. The starting point was this article: I gave it to Claude and asked it to derive evaluation criteria directly from the argument. If the piece claims responsible AI requires behavioural, quality, safety, policy, workflow and outcome evidence, those categories become the basis for what to test.
From there, Claude built a working test suite — sample tasks, scoring logic and a comparison tool that flags when performance drops — which I then ran.
The five questions are worth mapping against it. Evidence of intent was clearest: the test cases encode what good output looks like. Evidence of risk shaped how scoring was weighted, with automated checks supplemented by an AI judge for harder questions of quality. Evidence of behaviour came from running tasks at varying difficulty, including examples designed to fail. Evidence of oversight is built in through human-authored test cases and a tool that raises an alert when results regress.
Evidence of change over time remains the weakest link. The tooling supports switching models and comparing results over time, but whether it gets run consistently as things evolve is a governance question, not a technical one.
The second example came from an internal app. Running the eval at this stage was a deliberate test of the approach itself: the app was still a prototype, with no users, no real data and none of the underlying infrastructure built. We knew it would fail in several areas. The point was to see whether the eval could give us a useful overall picture quickly.
It did, in minutes.
The eval covered 13 areas including user value, output quality, harm and risk, accountability, data governance, failure handling, monitoring and sustainability. Each was scored 1 to 5, with five areas marked as critical — meaning they needed to score 4 or above before the app could move to a restricted pilot.
The overall average was 2.7, three critical areas failed:
Data governance scored 2: no plan for keeping different clients’ data separate, no data retention policy, and no confirmation of how the AI provider handles data sent to it.
Failure handling scored 2: no plan for what happens if the AI is unavailable, and no protection against users trying to manipulate the system through crafted inputs.
Monitoring scored 1: no plan, no named owner, and no process for checking whether the system was still behaving well after changes.
The initial output was scores. When we asked what mitigations were needed and what they would look like as concrete deliverables, the eval produced a prioritised list of eight items. Some were build tasks: keeping each client’s data isolated, logging who generated what and when, protecting against attempts to extract information the user should not see. Others were decisions that had to be made before any code was written: confirm how the AI provider handles data, agree whether users would be told AI had been involved.
Sustainability, which was added as a 13th eval area after an earlier version of this article was written identified two further gaps: no estimate of what the AI would cost to run at scale, and no deliberate decision about infrastructure and its carbon impact. Neither would have come up in a standard technical review.
The learning from this example is specific: governance evals are most valuable before you build, not after. Running the eval at prototype stage meant the findings shaped the design. The oversight model, the audit log, the data separation and the protection against manipulation were all planned in from the start because the eval identified them as requirements before the build began.
The cost of running it at this stage was one conversation, the cost of discovering the same gaps after the system was built would have been significantly higher.
The lesson across both examples is the same. Evals work best when they are treated as a prompt for a governance conversation, not a gate to pass. The initial scores showed where to look, the follow-up turned that into work the team could actually act on.
Five questions leaders should ask
The future of responsible product governance is better discipline around evidence. Before signing off consequential AI systems, senior leaders and product teams should ask for five kinds:
Evidence of intent: What is the system for, who is it for, and what must it not do?
Evidence of risk: What harms matter most, who could be affected, and what level of failure is acceptable?
Evidence of behaviour: How does the system perform across real tasks, edge cases, adversarial inputs and changing conditions?
Evidence of oversight: How can humans review, challenge, override, correct and learn from the system?
Evidence of change over time: How is the system monitored and re-evaluated as models, prompts, data, suppliers, policies and user behaviour change?
These are technical, product and governance questions and in the public sector, they are accountability questions.
From policy to operational evidence
The question is no longer whether an organisation has an AI policy. It is whether it can produce operational evidence that its systems still deserve to be trusted.
For product managers, evals belong in delivery. For leaders, they are board-level governance evidence. For public-sector organisations, transparency records, impact evaluation, human oversight and live monitoring are not separate workstreams. Together, they are what responsible product governance looks like when AI moves from pilot to infrastructure.
Responsible AI will not be proved by the existence of a policy. It will be proved by the evidence a service can produce when someone asks: how do you know this still works?