Use a fictional procedure question to trace a RAG failure to missing sources, missed passages, incorrect use of context or access controls before answering.
The spoken explanation, organised by chapter.
With RAG, assess finding information separately from using it in an answer.
A fictional procedure describes returns and an exception for custom products. Someone asks about the exception but receives only general guidance. Is the problem in the source, retrieval or the answer?
A RAG system processes sources, makes information searchable, retrieves passages and supplies them to a language model. The model formulates an answer. Keep enough context to review each stage specifically.
Is the exception absent from permitted, current documents? Then business knowledge is missing. Different search settings or more persuasive wording cannot establish that rule. Ask the source owner to assess the gap.
If the passage exists, check whether the question actually retrieved it. Investigate text processing, metadata and search selection. Without that context, the model cannot correctly include the exception in its answer.
If the model received the right passage but omitted the exception, investigate how the answer uses it. Also enforce the user’s permission to see the passage before generation.
Record the expected passage and assessment for each question. Include missing knowledge, conflicting sources and users without access. Test changes and deletions too; one correct answer does not establish general reliability.
Assign a failure to the responsible stage and owner. Recheck the same question after correction, alongside other examples, to see whether the change creates problems elsewhere.
Read OXIAE’s guide to evaluating RAG source selection and answer quality separately.
1:50Examine a fictional answer about business documents and learn to check passages, conditions, missing information and conflicting versions separately.
Source articleChat with business documents
1:52Follow a fictional work instruction through its lifecycle and see how source owners, versions, access permissions and deletion keep an AI knowledge base manageable.
Source articleAI knowledge base
1:55Compare language models on the same maintenance reports. Assess output format, source accuracy, total waiting time and review effort before choosing a model.
Source articleLLM integration: model selection, validation and operation