Document access in an AI assistant

Establish the source of truth

Before splitting documents into chunks, identify which versions are authoritative. A policy folder may contain a current document, a previous version and a presentation summarising both. All three can be relevant to a search query; only one may govern today’s decision. Give each source an owner, a version and a way to become obsolete.

Consider an illustrative employee assistant. Start with a source register: where each document lives, who maintains it, who can access it and which topics it covers. If ownership is missing or conflicting, the team needs to resolve it before the assistant uses the documents.

Retrieve evidence the reader can inspect

Lewis and colleagues introduced a retrieval-augmented generation approach combining a learned language model with an external document index. That research establishes an important architectural idea: generation can use retrieved material instead of relying only on what is stored in model parameters. It does not, by itself, guarantee that a business answer is current, authorised or supported.

Choose retrieval methods against representative questions. Exact identifiers may call for keyword matching; questions phrased in unfamiliar language may benefit from semantic retrieval. A blended method may be useful, but the extra complexity should earn its place in the evaluation.

Preserve the structure needed to interpret a passage. A table cell without its heading, a policy rule without its exception or a paragraph without its effective date can be misleading even when retrieved accurately. Show the reader enough source context to verify the claim.

Lost in the Middle found that the position of relevant information affected performance in the models and tasks studied. It gives a reason to test how context is assembled, rather than assuming that a larger input solves retrieval. New models and a different workflow need their own evidence.

Carry permissions through the whole journey

Filter sources by the requesting person’s access before their contents reach the model. Apply the same discipline to caches, citations and downloadable evidence. A correctly restricted search result is not enough if a shared answer cache later serves the text to someone else.

Include access changes in the design. When a document is removed or a user loses access, decide how quickly indexes and cached answers reflect that change. Keep this requirement explicit because it affects both architecture and operating cost.

In the employee example, an assistant should not infer that a salary document is acceptable to reveal merely because it is relevant to a question. The system’s access boundary must hold even when the model would otherwise produce an excellent answer.

There is a second boundary after access: an allowed source may contain hostile instructions. AgentDojo provides a research setting for this problem in tool-using agents. Relevance and permission to read a passage do not give that passage authority to redirect the application.

Check an answer’s source

Try an ordinary question whose answer changed last month. Start with the final sentence and follow its citation back to the exact source revision. Then remove access to that source and ask again as the same person. Finally, ask as someone with a different role.

This small exercise crosses ingestion, retrieval, permissions, generation and caching. It is useful because each layer can look correct alone while their combination leaks an old or inaccessible answer. Keep the resulting trace with the release evidence.

A citation is useful when a person can follow it to the evidence that actually governed the answer.

When the assistant cannot find an answer

Evaluate retrieval separately from the final answer. Component-level approaches such as RAGAS can help structure that inspection. Did the correct passage reach the model? Was it the current version? Did the answer include the important exception? This separation helps the team correct the right layer instead of repeatedly changing the prompt.

An unanswered question may point to a missing document, an unclear rule or a decision that needs a person. The assistant should say what information is missing and pass the case to the team that can resolve it.

For the first pilot, choose a small set of documents, each with a named owner. Add more after checking that permissions, document updates and source citations work correctly.

• Name the authoritative version and its owner.
• Preserve headings, exceptions and effective dates.
• Enforce access before content reaches the model.
• Test retrieval and answer quality separately.