Prepare company documents for RAG: build a managed source collection

Start with a source area and question type

Choose the questions the collection should support. Maintenance instructions for technicians need different quality checks from internal project notes. A maintenance answer may depend on a product version, while a note may describe a provisional idea rather than an agreed instruction. Knowing the file format alone therefore says little about suitability as a source for answers.

Describe the intended user and the decision they are preparing. Use that as a selection rule. Material that seems interesting but falls outside the task need not enter the first collection. Keep a separate list of excluded material and reasons. This makes it possible to explain later why some information does not appear in the application even though someone knows it exists.

Inventory fields should carry useful meaning

Record a stable source reference, title, owner, version, audience and period of applicability. Include its location and intended review frequency. Not every field applies to every source, but unknown ownership or validity must remain visible. An empty field should not silently become a statement that a document applies indefinitely to every employee and every product.

Use values people outside the technical team can interpret. An internal status code may help processing, but the responsible owner should be able to explain its meaning. Distinguish a file modification date from the date a policy takes effect. A recently saved copy of an old document is not automatically the newest instruction for the task being supported.

Inspect the text that is actually processed

Compare the original file with the text obtained by extraction. A visually clear PDF can produce interleaved columns. A scan can lose a minus sign, heading or footnote. In a procedure, an exception may then look like a general rule even though the import completes without reporting a technical failure. File acceptance is therefore different from content being represented correctly.

Use a sample with several formats and deliberately difficult pages. Ask a content reviewer to locate decisive passages in the extracted version. Record missing or ambiguous elements. The response may be a different extraction method, but sometimes the source itself should become a managed instruction with clearer structure. That can benefit both employees reading the document and software trying to retrieve the relevant information.

A useful source can still have a restricted audience. Define access to documents and passages too: RAG access control: who may retrieve which company information?.

Preserve context when splitting documents

Large documents are often divided into smaller passages. Each needs enough context to show where a rule applies. A sentence stating a deadline can mislead if its product, exception or section heading sits in another fragment. Do not select passage boundaries solely because a default character count appears convenient for processing or seems to work in one demonstration.

Test structure with real questions. Does a table retain its headings? Does a warning remain associated with the instruction? Can a reviewer open the correct source version from a passage? Technical limits and document properties influence the design, but the practical check is whether meaningful support for the agreed question can be located and reviewed without reconstructing missing context manually.

Microsoft Learn: Chunk large documents for vector search ↗

Let an accountable owner resolve contradictions

An illustrative supplier has a general manual and a later product appendix. Both state a maintenance interval, but the appendix applies only to one series. The application should not invent a rule that the latest file date always takes priority. The owner needs to decide when the appendix supplements or replaces the general instruction and how the inventory records that distinction.

Keep the decision together with its reasoning and applicability boundary. An unresolved conflict may require temporary exclusion or a referral to a person. Supplying both conflicting documents and hoping the model will choose a suitable nuance turns an editorial responsibility into a variable answer-generation decision. The relevant improvement is often clearer source governance rather than a longer prompt or a different model setting.

Separate confidentiality from content quality

An approved procedure may still be confidential. A public brochure may contain outdated information. Assess whether a source is suitable for the task separately from which users may access it. A general quality label should not simultaneously become an access rule, because those decisions can change independently and may belong to different responsible people within the organisation.

Also record which parts are necessary. An attachment with names, personal notes or unnecessary customer details may stay outside the trial while the general instruction is included. This selection is a design arrangement that needs checking in actual processing. A verbal agreement about confidentiality does not constrain a technical connector, an index or the passages later passed into an answer request.

OWASP: Authorization Cheat Sheet ↗

Design replacement and removal early

Ask how a withdrawn document leaves the active collection. Removing the original file may be insufficient if derived passages, a search index or an answer cache exist. The specific architecture should identify the copies and derived data it manages and explain how it verifies that an old version is no longer used as current support for ordinary questions.

Test a small change before importing a broad collection. Replace one instruction and repeat the relevant question. Check the retrieved passage and the visible citation. Where needed, maintain a controlled historical trail for investigation, but prevent historical material from becoming accidentally available again as current guidance. The distinction between history and active support should be explicit to users and maintainers alike.

Keep a checking pack for each source selection

Save representative questions with the passages a reviewer expects. Include a normal instruction, an exception and a question beyond the topic. Add a case involving an unreadable or missing page. This helps identify whether a change to extraction or selection breaks a previously useful question even when the new import otherwise appears to complete successfully.

Ask someone unfamiliar with the document structure to try the collection. They may use different terminology or start from a mistaken assumption. That provides useful evidence for source organisation and retrieval improvement. The objective is not to generate an answer to everything. It is to make both assistance and the limits of the collection understandable to the people doing the work.

Agree a practical editorial routine

Decide who may nominate documents, who approves their meaning and who checks technical processing. This need not become a large governance exercise. A small source area may need only clearly assigned roles and a brief change record, provided uncertainty is not automatically passed to a developer or whichever employee happens to notice an incorrect response first.

Make handover concrete with the inventory, example questions, exclusions and an exercise in which a source changes. Ask the owner to explain what they can update themselves and when assistance is necessary. This turns the collection into a managed part of everyday work rather than a folder imported once whose continued accuracy nobody has agreed to maintain.

Turn the explanation into a project decision

Record your assumptions, unknowns and acceptance checks in the matching tool. Use the service guides to discuss delivery and scope.

• RAG system development ↗
• AI knowledge base ↗

Questions about this topic

Continue with a related question