A RAG trial does not succeed simply because several demonstration answers sound convincing. You need to know whether it finds the necessary information, explains conditions correctly and preserves boundaries when an answer is unavailable or access is not permitted. These are different questions requiring separate evidence. An average satisfaction score can conceal failures important to actual use.
This guide prepares a limited but useful evaluation. Questions and review examples are illustrative. The method does not produce a universal quality percentage or certification; it helps investigate and improve a particular source collection and task.
Define the task being evaluated
Describe the user, the information needed and the next action an answer supports. An internal assistant helping locate a manual needs different criteria from an application summarising several policy documents. Without a task boundary, any attractive demonstration can count as success even if its output provides little evidence about the work the application is intended to support.
Also define what the first version should not do. A product-information trial need not assess personal customer files. An answer about a known series need not cover every unknown variant. Use these boundaries in the evaluation. A correct referral or focused clarification may be the desired result, rather than requiring every case to end in a complete generated answer.
Collect work questions without unnecessary personal data
Ask employees about recurring questions and exceptions that cause doubt. Turn these situations into neutral cases with enough context for substantive review. Replace names and unnecessary details with fictional values. The objective is to represent the task, not to accumulate the largest possible archive of full conversations or to copy sensitive material without a clear evaluation purpose.
Record why each case matters. A frequent simple question may be valuable for efficiency; a rare exception may matter because an incorrect answer has larger consequences. Both deserve attention, but they need not carry identical weight in a decision. Make the reasoning visible rather than presenting an arbitrary average as an objective judgement covering every operational concern.
Record expected support and unresolved facts
For each question, identify the source passage a content reviewer expects. Include the relevant version, conditions and information the user still needs to supply. Avoid testing only whether generated wording resembles a prepared response. Different words can preserve the same meaning, while a near-identical sentence may omit the condition that determines whether a rule applies.
For a maintenance question, expected support may be a particular product appendix. When the variant is missing, clarification may instead be necessary. Write these expectations before the trial. This reduces the temptation to reinterpret an impressive answer as appropriate afterwards without applying the same standard to responses that are less polished or immediately convenient for the demonstration.
Connect test results to explicit acceptance arrangements and a decision on the first release: Accept custom software: assess complete tasks and agreed boundaries.
Review retrieval separately from formulation
First check whether retrieval supplies the necessary source with enough context. Record missing passages, wrong versions and tables split into unusable fragments. Then review whether the answer represents the relevant claims correctly. This provides a developer with a practical diagnosis rather than only a general report that the AI is not working well.
An illustrative service team receives the wrong maintenance interval. In one case the correct appendix was never retrieved; in another it was found but the answer ignored a condition. These require different changes. Recording only a failed final answer loses the distinction that makes an evaluation useful for project decisions and can lead to unnecessary changes in parts that were already working.
| Observation | Investigate separately | Possible next step |
|---|---|---|
| Required passage missing | Source selection and retrieval | Add content or improve retrieval design |
| Correct passage, incorrect claim | Conditions and answer behaviour | Improve explanation or review boundary |
| Wrong user receives information | Permissions and reuse | Investigate selection and revocation |
| Conflicting sources | Substantive validity | Ask an owner to resolve the conflict |
Include negative and ambiguous cases
Add questions whose answers are absent from the collection. Test a mistaken assumption, conflicting sources and a question needing clarification. An application producing a confident answer in every such case may appear impressive during an ordinary demonstration, but that behaviour does not automatically suit responsible day-to-day use where the boundaries of available information matter.
Include source material containing a misleading instruction as well. The case examines whether the application treats that content as information and preserves its agreed task boundary. A limited trial cannot cover every attack. It can show exactly which boundary was investigated and which outcomes require further security work, a narrower source selection or a different technical design.
Exercise access through recognisable role changes
Ask the same substantive question under different permissions. An employee entitled to a product appendix may receive different support from an external user. Review citations, labels and links as well as answer text. Include access being revoked after an earlier question, then ask again. This makes the evaluation reflect permission changes rather than only an ideal first-time request.
Record the expected outcome for each role and the context used in the test. Otherwise success may depend accidentally on a shared administrator account or documents that are public to everyone. A useful evaluation should represent the information boundary intended for actual use, with known limitations documented rather than silently treated as if the final access design already exists.
Align reviewers before combining numbers
Have two reviewers discuss several identical outputs first. What counts as sufficient support? When is an answer incomplete but still useful? When does the wrong version create a decisive failure? This establishes whether labels have the same meaning. A score with several decimal places provides little value if reviewers apply fundamentally different criteria to the underlying cases.
Keep examples alongside the review rules. A compact rubric covering successful, incomplete, incorrect and uncertain outcomes may be enough for a first bounded trial. Add a reason in plain language. Later readers can then understand the judgement and the concrete change it suggests. Uncertainty should remain an acceptable recorded result rather than being forced into an unjustified pass or failure.
Compare versions against the same baseline
Record source versions, retrieval settings, answer instructions and relevant technical configuration. In a focused comparison, preferably change one main part at a time. Changing sources, model and review criteria together makes it difficult to identify what caused an apparent improvement. Consider variation between runs when the application does not necessarily produce identical answers on every attempt.
Discuss which cases regress as well as how many improve. Better performance on many simple questions may be insufficient if an important exception becomes less dependable. Record who accepts that tradeoff, what additional measures are necessary and when a version should not proceed to wider use. Evaluation should support an operational decision, not merely populate an attractive dashboard.
Make the case collection maintainable
Keep questions, expected support and review records somewhere the responsible owner can maintain them. Add real new failure categories after substantive investigation. Do not remove obsolete questions without a reason; some may need to remain as historical evidence explaining why earlier versions behaved differently and why a particular change or boundary was introduced.
Agree which changes require reassessment. A new document category, different permission connector or altered passage splitting can matter more than a small visual adjustment. Handover should therefore include an exercise in which a source changes and the relevant cases are repeated. Evaluation becomes a practical maintenance routine rather than a one-off approval attached to the initial demonstration.
Turn the explanation into a project decision
Record your assumptions, unknowns and acceptance checks in the matching tool. Use the service guides to discuss delivery and scope.
Open the preparation tool ↗Questions about this topic
Is one overall score enough for a decision?
Usually not. It can hide differences between retrieval, source support, access and user experience. Use a summary for orientation but preserve relevant failure categories, examples and consequences for the agreed task.
Must every question receive identical wording each time?
Not necessarily, but substantive boundaries and required evidence should remain appropriate. Investigate meaningful variation through repeated cases where it matters. Valid paraphrasing differs from a condition or source version changing unpredictably.
When can the trial expand?
When responsible owners have assessed the bounded task, known failures and operating arrangements sufficiently. Record limitations and add sources or users deliberately. A successful small trial does not automatically establish the behaviour of a substantially broader collection.
Continue with a related question
RAG retrieves the wrong passage: investigate retrieval before rewriting the answer
Diagnose retrieval quality with expected passages, product versions, document structure, filters and controlled comparisons.
RAG & business knowledgeRAG access control: who may retrieve which company information?
Define users, organisations, source permissions, caches and revocation before making internal knowledge available through AI.
Custom software & portalsAccept custom software: assess complete tasks and agreed boundaries
Prepare software acceptance with representative users, input, exceptions, access, recovery evidence and an explicit decision about unresolved findings.
Sources and technical background
- NIST: AI Risk Management Framework ↗
Framework for organising AI risk management, not product certification.
- OWASP GenAI: Prompt Injection ↗
Risk of misleading instructions in input and retrieved sources.
- OWASP: Authorization Cheat Sheet ↗
Authorisation principles; the worked evaluation cases are illustrative.