Business knowledge for AI agents: sources, access and freshness

Start with questions the sources must support

List recognisable questions before selecting documents. An assistant explaining products needs different material from one answering questions about contractual arrangements. For each question, identify the information that could support an answer and whether it actually exists. An agent should not be expected to reconstruct unknown company rules from scattered examples or informal habits of individual employees.

Limit the first collection to material somebody can own. A current instruction with a clear scope is often more useful than an extensive folder with no responsible person. State which questions remain outside the application’s remit. Where the sources do not support an answer, ask for more information or route the request. Producing an answer is not a useful goal when its basis is missing.

Give the source register meaning

A source register needs more than filenames. Record owner, subject, intended audience, validity and origin. Identify formal instructions, background material and drafts. This helps prevent an old presentation receiving the same status as the current operating procedure. A date in a filename does not necessarily indicate when an arrangement took effect or whether it still applies.

Investigate duplicates and contradictions with the owner. A newer document may apply only to new customers or a particular location. Removing the older file alone would then lose relevant conditions. Preserve applicability explicitly through metadata and, where possible, in the source itself. Otherwise the agent has insufficient evidence to distinguish two arrangements that are both valid in different circumstances.

Separate preparation, retrieval and answering

Retrieval-augmented generation supplies selected source passages as context for an answer. Microsoft describes separate stages for source processing, retrieval and response generation. In a practical project, each stage needs its own checks. Failure to retrieve a correct source differs from reaching the wrong conclusion after finding it, and the remedy will usually differ as well.

Check how headings, tables and exceptions stay connected. A passage saying “within five working days” can mislead if its qualifying condition lands in another fragment. Evaluate questions involving amounts, periods and exceptions separately. The useful splitting approach depends on the documents. A default setting is a starting point, not evidence that every source’s meaning survives processing intact.

Check source selection and answers using prepared questions and expected supporting passages: Testing an AI agent: turn fluent answers into useful evidence.

Microsoft: Design and develop a RAG solution ↗

Carry permissions through the information route

Determine which sources a user may access before retrieving content. General instructions may be widely available, while a customer contract belongs to a restricted audience. An agent consulting one large shared collection cannot reliably restore that boundary simply by wording its answers cautiously. The retrieval and access layers need to restrict the sources available to the current user.

Test permission and organisational changes. Consider an employee changing teams, a customer relationship ending or a source being withdrawn. Inspect derived material such as search indexes and caches too. Deleting a file from the main folder does not necessarily remove every processed copy. Document the route and any delay so that users and operators understand the actual behaviour rather than assuming immediate removal.

Make source references genuinely useful

A reference helps when readers can inspect the relevant passage. Where possible, provide the title, version and location. Linking only to a large folder makes investigation unnecessarily difficult. Ensure the recipient can access the referenced source, or provide an appropriate explanation without exposing confidential content beyond their authority merely to make a citation appear complete.

Check support for each consequential statement. A response may contain several sentences while the cited source supports only one. Test whether the reference actually establishes the stated conditions. Citing the correct document does not prove that every conclusion is valid. Missing or conflicting information should result in an explicit limitation or a route to the source owner instead of an invented reconciliation.

Illustrative example: two service arrangements

A fictional organisation has a general service handbook and a separate customer contract. The handbook describes ordinary operating hours, while the contract establishes another arrangement for one service. An employee asks when an incident will be handled. The answer depends on the customer, service and applicable contract version. Searching for “service” alone provides insufficient context to answer responsibly.

The first application answers general handbook questions and sends customer-specific arrangements to an authorised employee. A later version adds controlled customer selection and restricted contract sources. Tests separately cover the general rule, special arrangement and an unknown customer. This shows whether the assistant preserves the distinction rather than automatically using the most specific passage found anywhere in the collection.

Turn updates and removal into a working process

Assign an owner to approve changes for each source. Describe how a new version becomes available, how obsolete material is marked and which checks happen before release. Automatic synchronisation can retrieve files, but does not determine whether their content has been approved. That decision remains with the designated owner or editorial team under the agreed operating arrangement.

Exercise an update with an identifiable example: change a fictional instruction and verify when the new answer appears. Then withdraw the source and investigate what the application can still retrieve. Record the actual processing time and limitations. Use those findings to design support procedures. A theoretical synchronisation setting cannot replace a controlled test across the complete information path.

Assess the collection before expanding

Build a small, varied evaluation set: a directly supported question, a missing answer, an exception and a source conflict. Inspect retrieved material first and the final response afterwards. Ask source owners whether the assistant respects the same applicability boundaries as employees. When something fails, record whether selection, retrieval, permissions or interpretation caused the problem rather than treating all mistakes as model behaviour.

Expand topic by topic. Do not add an entire department’s drive merely because the technology accepts it. Identify the new questions it supports, the person responsible and how the additional material will be assessed. A manageable knowledge collection grows alongside its source-management process. That allows the assistant to remain useful as products, agreements, instructions and people change over time.

Turn the explanation into a project decision

Record your assumptions, unknowns and acceptance checks in the matching tool. Use the service guides to discuss delivery and scope.

• AI agent development ↗
• AI chatbot development ↗

Questions about this topic

Continue with a related question