An incorrect RAG answer may begin with a retrieval mistake. The application finds a passage resembling the question but describing another product series, period or exception. A more elaborate model or a sharper answer instruction may conceal that issue without resolving it. Investigate what retrieval supplies separately from what the answer stage does with the material.
This guide diagnoses retrieval problems using a small, repeatable question collection. Product codes and situations are fictional. The intended result is a useful diagnosis and targeted improvement, not a promise that one technique searches every collection without error.
Start with a reproducible failure
Keep the question, intended user context and source version used for review. Record the passage that should have been found and those actually supplied. A report saying the answer was poor gives a developer too little information to distinguish retrieval, selection and interpretation. A recognisable example connects a technical investigation to the real task and the result a colleague expected.
Choose a case whose owner can identify the correct source. If reviewers disagree about which instruction applies, an editorial decision is needed first. A search adjustment should not silently decide the governing rule. Also allow for the possibility that the necessary information is absent from connected sources. No retrieval setting can recover a company instruction that has never been supplied to the collection.
Test exact terms and meaning separately
Some questions depend on a product code, name or particular date. Others describe a situation in ordinary language that does not appear verbatim in the source. Include both in a comparison. Otherwise a design may handle broad descriptions attractively while missing the exact identifiers employees use to recognise products and processes in their everyday work.
Hybrid designs can combine text and vector retrieval. That is a technical capability rather than evidence that the combination improves your particular collection. Compare representative questions against the same permitted sources. An improvement for general wording should not hide weaker results on exact product questions that the existing search route already handles reliably and that users regard as decisive.
Inspect the passage in its context
A fragment may contain the right words while carrying the wrong meaning. A table row without column labels, a warning without a product reference or a paragraph separated from its preceding exception may be poor support for answer generation. Inspect the material returned as a usable unit, not simply the document identifier associated with the result.
Open it beside the original source. Ask whether a content reviewer can establish the situation to which it applies. If the fragment boundary removes a decisive condition, investigate source splitting. Retrieving more results will not necessarily resolve that problem. It may instead introduce additional contradictory or irrelevant passages into the context the answer stage receives and make review harder.
Test retrieval and answers separately using a fixed set of questions and source passages: Test a RAG system: assess sources, answers and refusals separately.
Apply filters that reflect the real task
An employee may need instructions for a particular product family. If that context is reliably established, retrieval can be bounded accordingly. Specify which attributes are required and what happens when they are missing. An ambiguous product code should not automatically become a reason to produce a confident answer spanning every variant in the wider source collection.
Check whether filters exclude too much as well. A general safety instruction may apply to several families. Incorrect categorisation can then hide material the answer needs. During evaluation, retain expected information that was excluded as well as material returned. This helps locate the issue in metadata, document content or question interpretation instead of treating every missing passage as a model failure.
Build a small set of difficult variations
Take one substantive question and change its wording in realistic ways. Include an abbreviation, typo, old product name and a description without the specialist term. Add a similar-sounding question concerning another version. This reveals whether retrieval preserves relevant distinctions rather than merely collecting material containing words associated with the same broad subject.
An illustrative maintenance team asks about the inspection interval for pump P-410. The set also includes P-410B, an older name and a versionless question. The last case may appropriately require the user to establish the variant first. The desired result is therefore not the same passage or answer type for every variation, even where the questions share much of their wording.
Change one part at a time
For example, adjust passage splitting and rerun the same questions. Avoid simultaneously changing models, sources, filters and review rules. When several variables move together, an improved demonstration becomes difficult to explain and newly introduced failures are harder to attribute. A focused comparison often provides more useful project evidence than a large, uncontrolled replacement of several parts of the application.
Keep the configuration and outputs from both versions. Include questions whose results deteriorate. Discuss whether that change is acceptable for the task or needs further investigation. A better average can still be unsuitable if a small set of important exceptions no longer retrieves the source required for a responsible decision. The review must preserve that operational distinction.
Assess ranking and usefulness together
A relevant passage appearing somewhere among many results is not necessarily useful to the answer stage. Inspect what actually enters its context and whether the decisive condition survives selection. Look for repeated fragments from one source that crowd out other necessary information. A ranking is valuable only in relation to the task that follows retrieval and the evidence that task requires.
Use a simple review rubric. Reviewers might record whether required material is present, whether its version matches and whether enough context remains. Treat uncertainty separately. This need not become a complex benchmark, provided reviewers use the same meaning for a successful, incomplete or incorrect retrieval result. Clear definitions make comparison more useful than numerical precision unsupported by consistent judgement.
Separate retrieval from answer behaviour
If the right passage is found but an exception disappears from the final answer, investigate a different problem. Save the supplied material beside the claim produced from it. The improvement may involve answer instructions, presentation, citations or human review. Adding more retrieved passages may be unnecessary and can increase the work needed to check the answer.
Conversely, an accidentally correct answer supported by the wrong passage can hide a retrieval defect. The model may rely on general knowledge that does not apply to your version. Assess both the claim and whether the agreed source supports it. The retrieval evaluation and answer evaluation therefore require distinct evidence even when they ultimately contribute to one user experience.
Turn the diagnosis into a maintainable agreement
Record the source areas, filters and question types supported by the selected design. Preserve representative failures as repeatable checks for source updates and technical changes. A maintainer should not have to rediscover why a particular variant or historical synonym matters. The original diagnosis becomes useful project knowledge that helps later decisions remain connected to the intended task.
Agree when a question beyond the supported collection should be referred elsewhere. A retrieval system should not hide uncertainty behind a technical success status. The practical outcome is a better understood boundary: what can be found dependably, where content remains missing and which questions still require clarification or an accountable content decision before an answer can be useful.
Turn the explanation into a project decision
Record your assumptions, unknowns and acceptance checks in the matching tool. Use the service guides to discuss delivery and scope.
Open the preparation tool ↗Questions about this topic
Does retrieving more passages always help?
No. More results may add useful context, but they can also introduce duplicates or contradictory information. Check whether the required passage and its conditions are actually available to the next stage. Result count alone says little about usefulness.
How many evaluation questions do we need?
No universal number proves quality. Start with representative ordinary questions and important exceptions for the bounded task. Expand using observed failures and changes. Record which question types remain outside the review rather than implying the sample covers everything.
Can one review cover retrieval and answers?
You can assess the overall experience, but also preserve separate evidence for retrieved sources and generated claims. Otherwise it remains unclear whether a failure arises from missing information, poor retrieval or incorrect interpretation of a correct passage.
Continue with a related question
Prepare company documents for RAG: build a managed source collection
Make documents useful for a RAG trial through ownership, validity, structure, removal rules and representative checks.
RAG & business knowledgeCitations in RAG: does the passage actually support the claim?
Design reviewable AI answers using appropriate passages, versions, conditions and useful next steps when evidence is missing.
AI agents & chatbotsTesting an AI agent: turn fluent answers into useful evidence
Build a practical AI-agent test set covering sources, actions, exceptions, handover and repeatable acceptance criteria.
Sources and technical background
- Microsoft Learn: Hybrid search in Azure AI Search ↗
Combining text and vector search; no guarantee for a particular corpus.
- Microsoft Learn: Chunk large documents for vector search ↗
Technical considerations when splitting documents for retrieval applications.