WEBVTT

00:00:01.000 --> 00:00:06.250
With RAG, assess finding information separately from using it in an answer.

00:00:06.680 --> 00:00:10.730
A fictional procedure describes returns and an exception for custom

00:00:10.760 --> 00:00:11.690
products.

00:00:11.960 --> 00:00:16.210
Someone asks about the exception but receives only general guidance.

00:00:16.440 --> 00:00:19.610
Is the problem in the source, retrieval or the answer?

00:00:20.520 --> 00:00:24.650
A RAG system processes sources, makes information searchable,

00:00:24.680 --> 00:00:27.930
retrieves passages and supplies them to a language model.

00:00:28.360 --> 00:00:30.410
The model formulates an answer.

00:00:30.680 --> 00:00:34.410
Keep enough context to review each stage specifically.

00:00:34.840 --> 00:00:38.250
Is the exception absent from permitted, current documents?

00:00:38.680 --> 00:00:40.810
Then business knowledge is missing.

00:00:41.080 --> 00:00:45.050
Different search settings or more persuasive wording cannot establish

00:00:45.080 --> 00:00:46.010
that rule.

00:00:46.280 --> 00:00:48.970
Ask the source owner to assess the gap.

00:00:49.400 --> 00:00:53.130
If the passage exists, check whether the question actually retrieved

00:00:53.160 --> 00:00:53.450
it.

00:00:53.960 --> 00:00:57.850
Investigate text processing, metadata and search selection.

00:00:58.360 --> 00:01:02.250
Without that context, the model cannot correctly include the exception

00:01:02.280 --> 00:01:03.530
in its answer.

00:01:03.960 --> 00:01:07.770
If the model received the right passage but omitted the exception,

00:01:07.920 --> 00:01:10.170
investigate how the answer uses it.

00:01:10.440 --> 00:01:14.810
Also enforce the user’s permission to see the passage before generation.

00:01:15.720 --> 00:01:19.530
Record the expected passage and assessment for each question.

00:01:19.800 --> 00:01:23.610
Include missing knowledge, conflicting sources and users without

00:01:23.640 --> 00:01:24.610
access.

00:01:24.920 --> 00:01:29.610
Test changes and deletions too; one correct answer does not establish

00:01:29.640 --> 00:01:31.330
general reliability.

00:01:31.720 --> 00:01:35.010
Assign a failure to the responsible stage and owner.

00:01:35.320 --> 00:01:39.770
Recheck the same question after correction, alongside other examples,

00:01:39.800 --> 00:01:43.050
to see whether the change creates problems elsewhere.

00:01:43.480 --> 00:01:49.610
Read OXIAE’s guide to evaluating RAG source selection and answer quality separately.