Assessing an AI agent means more than checking whether its answers sound professional. A polished reply can rely on the wrong source, confuse customers or say an operation completed when only a suggestion exists. Test the complete task outcome: information used, decisions made, system operations and handover to an employee. Those parts can fail independently.
This guide offers a practical evaluation approach for a bounded business application. Examples are illustrative; the proposed counts are design choices rather than evidence from a tested OXIAE system. The aim is a test collection that supports a considered launch decision and lets you assess later changes against the same expectations instead of relying on a fresh demonstration each time.
Define what acceptable means
Write down the task and the consequences of mistakes before collecting examples. Completeness and usefulness may dominate an internal summary. Changing an appointment also requires identity, availability and actual processing. Assess these outcomes separately. A high language-quality score should not compensate for an incorrect system operation when an overall average is calculated.
For each criterion, distinguish acceptable outcomes, repairable issues and blocking failures. Awkward wording might be repairable; exposing another customer’s information could block release. Identify who judges each case. This prevents a demonstration from quietly using different standards from those agreed during design or needed by employees carrying out the work every day.
Collect ordinary and difficult cases
Use examples representing the work the application is meant to support, then add targeted exceptions. Include missing references, outdated documents, conflicting wishes and requests outside the task boundary. OpenAI recommends application-specific evaluation; a general model benchmark reveals little about your organisation’s particular records, decisions and operating conditions or the consequences when something goes wrong.
Keep development examples separate from a held-out check set. Repeatedly adjusting instructions against the same few questions can make the application appear capable without demonstrating different wording. Add new formulations expressing the same underlying need. Where operational examples contain sensitive information, use cleaned or fictional data when possible while preserving the relevant structure, ambiguity and exceptional conditions.
Specify outcomes without prescribing one sentence
Several wordings can produce a useful answer. Record required facts, permitted operations and necessary follow-up questions rather than only an exact reference sentence. Where an order number is missing, the expected result might be: ask one focused question, reveal no order details and begin no change. Different polite formulations can all satisfy those conditions.
State prohibited conclusions too. The application should not confirm a return entitlement if its source provides only a general contact address. A case can therefore contain source material, known facts, missing information and forbidden inferences. This structure helps reviewers give consistent feedback and explain why a response is wrong even when its wording sounds authoritative and reassuring.
Use the evaluation cases in a bounded operational trial with a comparable baseline: Designing an automation pilot: evidence for the next decision.
Evaluate retrieval separately from answers
When the agent uses sources, first inspect whether it finds relevant information. A wrong answer can arise because the right passage is missing, an obsolete version is retrieved or the model misinterprets the material it receives. Looking only at the final reply makes these causes difficult to separate. Retain relevant source references and versions for each evaluation case.
Include a supported answer, missing information and a conflict between two sources. A focused question may be correct for the second case, while the third may require handover. Appropriate restraint is therefore part of quality. An agent that always produces an answer does not automatically perform better than one that recognises insufficient evidence and routes the request responsibly.
Inspect actions and state changes
For an agent using tools, tests should show which function was requested and what changed in the target system. Verify the record, permitted fields and returned state. The interface should distinguish proposed, approved and completed actions. Otherwise users may mistake a draft for a confirmed appointment or a message that has actually been sent to a customer.
Also test repeated input and failure at an inconvenient moment, such as losing the connection after a task might already have been saved. Investigation may be appropriate before another write. Add revoked access and changed records. These cases reveal whether the application remains understandable and controllable beyond the smooth path usually selected for a sales demonstration or an internal presentation.
Illustrative review round for a service assistant
A fictional team reviews twenty examples spanning ordinary information requests, incomplete enquiries, conflicting sources and prohibited actions. Two employees first review a small shared selection. They discuss disagreements, including when an answer is sufficiently specific and when a missing detail requires another question. Only then do they apply the clarified criteria to the remaining examples.
They discover that the assistant usually selects the right topic but too readily suggests a date for an unknown appointment. This error is tracked separately. After a change, they rerun the original cases and add fresh formulations. This is a working method, not a claim that twenty examples are enough. Coverage should reflect the variation and consequences of the actual task.
Measure employee effort as well
An accurate result can still require substantial review. Measure the time employees spend opening sources, correcting suggestions and handling exceptions. Compare it with the existing route. Separate straightforward and difficult cases so savings on many short questions do not conceal major recovery problems elsewhere. Review burden is part of the outcome rather than an unrelated operating expense.
Track how many requests actually fit the supported task. A strong score on a small subset says little about the whole department. With low usage, encountering enough varied situations may take longer. Combine metrics with concrete findings. A record of demonstrated errors and their consequences often gives more useful direction than a single percentage without explanations, distributions or examples.
Include evaluation in every meaningful change
Keep cases, source versions, instructions and review criteria in a managed location. A changed prompt, new tool or revised document collection can alter previously satisfactory behaviour. Determine which checks each update requires and which new cases arise. A system can improve on its newest topic while an established handover route becomes less dependable or stops working altogether.
Make a release decision with explicit conditions: limited users, draft-only operation or no launch yet. Record unresolved limitations and ownership of follow-up work. After introduction, real incidents and useful feedback become additional test cases. Evaluation then becomes an ongoing learning process, providing inspectable evidence for each expansion of supported tasks, information access and operational permissions.
Turn the explanation into a project decision
Record your assumptions, unknowns and acceptance checks in the matching tool. Use the service guides to discuss delivery and scope.
Open the preparation tool ↗Questions about this topic
How many test questions are necessary?
There is no universal number. Required coverage depends on sources, users, operations and consequences. Begin with recognisable categories and expand where behaviour varies or risks emerge. A handful of successful demonstration prompts does not establish general reliability across the intended workflow.
Can AI grade answers automatically?
Automated evaluation can support scale and repeatability, but should be calibrated against human review. Use explicit criteria and investigate disagreement. Important factual and operational failures still need assessment against the source material and what actually happened in the connected system.
When is an agent ready for production?
When the agreed task demonstrably meets acceptance conditions and its operating route works. This may justify only a limited, draft-only introduction. Record remaining limitations, oversight and stop conditions; a successful test does not establish error-free future performance under every possible circumstance.
Continue with a related question
Business knowledge for AI agents: sources, access and freshness
Prepare AI-agent knowledge sources by defining valid versions, permissions, usable citations, deletion and testable answers.
AI agents & chatbotsWhich permissions should an AI agent receive?
Design bounded AI-agent permissions for reading, proposing and executing, with approval boundaries, practical access checks and recovery examples.
AI agents & chatbotsOperating an AI agent: monitoring, changes and recovery
Plan AI-agent operation with useful alerts, usage limits, source ownership, regression checks, pausing and a workable recovery route.
n8n & process automationDesigning an automation pilot: evidence for the next decision
Plan an automation pilot using representative cases, observable acceptance, human review and stopping conditions. Includes a fictional trial design.
Sources and technical background
- OpenAI: Evaluation best practices ↗
Task-specific evaluation, representative cases, human calibration and continued regression testing.