1
00:00:01,000 --> 00:00:06,250
With RAG, assess finding information separately from using it in an answer.

2
00:00:06,680 --> 00:00:10,730
A fictional procedure describes returns and an exception for custom

3
00:00:10,760 --> 00:00:11,690
products.

4
00:00:11,960 --> 00:00:16,210
Someone asks about the exception but receives only general guidance.

5
00:00:16,440 --> 00:00:19,610
Is the problem in the source, retrieval or the answer?

6
00:00:20,520 --> 00:00:24,650
A RAG system processes sources, makes information searchable,

7
00:00:24,680 --> 00:00:27,930
retrieves passages and supplies them to a language model.

8
00:00:28,360 --> 00:00:30,410
The model formulates an answer.

9
00:00:30,680 --> 00:00:34,410
Keep enough context to review each stage specifically.

10
00:00:34,840 --> 00:00:38,250
Is the exception absent from permitted, current documents?

11
00:00:38,680 --> 00:00:40,810
Then business knowledge is missing.

12
00:00:41,080 --> 00:00:45,050
Different search settings or more persuasive wording cannot establish

13
00:00:45,080 --> 00:00:46,010
that rule.

14
00:00:46,280 --> 00:00:48,970
Ask the source owner to assess the gap.

15
00:00:49,400 --> 00:00:53,130
If the passage exists, check whether the question actually retrieved

16
00:00:53,160 --> 00:00:53,450
it.

17
00:00:53,960 --> 00:00:57,850
Investigate text processing, metadata and search selection.

18
00:00:58,360 --> 00:01:02,250
Without that context, the model cannot correctly include the exception

19
00:01:02,280 --> 00:01:03,530
in its answer.

20
00:01:03,960 --> 00:01:07,770
If the model received the right passage but omitted the exception,

21
00:01:07,920 --> 00:01:10,170
investigate how the answer uses it.

22
00:01:10,440 --> 00:01:14,810
Also enforce the user’s permission to see the passage before generation.

23
00:01:15,720 --> 00:01:19,530
Record the expected passage and assessment for each question.

24
00:01:19,800 --> 00:01:23,610
Include missing knowledge, conflicting sources and users without

25
00:01:23,640 --> 00:01:24,610
access.

26
00:01:24,920 --> 00:01:29,610
Test changes and deletions too; one correct answer does not establish

27
00:01:29,640 --> 00:01:31,330
general reliability.

28
00:01:31,720 --> 00:01:35,010
Assign a failure to the responsible stage and owner.

29
00:01:35,320 --> 00:01:39,770
Recheck the same question after correction, alongside other examples,

30
00:01:39,800 --> 00:01:43,050
to see whether the change creates problems elsewhere.

31
00:01:43,480 --> 00:01:49,610
Read OXIAE’s guide to evaluating RAG source selection and answer quality separately.