“What does our supply agreement say about delivery times?” Today that means searching the network drive, asking the colleague who negotiated the contract, or leafing through the folder. An AI assistant that knows your own documents answers in a sentence and says which document the answer came from.

Mid-sized businesses are asking for such assistants a lot at the moment, because the knowledge of the business sits in contracts, quotes, minutes and mailboxes, and nobody finds it there quickly any more. How it works is less mysterious than it sounds, and once you understand it, you also see where the limits are.

What happens in the background

A language model such as ChatGPT, Claude or Mistral knows nothing about your business. For it to answer from your documents all the same, put simply, this is what happens:

  1. Your documents are read in once and broken into sections, such as individual paragraphs or pages.
  2. When someone asks a question, the assistant looks up the sections that match it.
  3. Those sections are handed to the model together with the question. The model phrases the answer from them.

The technical term for this is RAG, retrieval-augmented generation. What matters is the idea: the model does not answer from memory but from the sections it has just been shown. Your documents are not “trained into” the model; they stay where they are and are looked up afresh for every question.

Which sources come into play

The assistant can only read what it can reach. Where possible, these are connected:

  • The network drive with contracts, drawings, minutes and ever-growing folders.
  • SharePoint and OneDrive, if the business works with Microsoft 365.
  • Shared mailboxes, such as the one in which orders and complaints are coordinated.
  • The ERP or inventory system, so the assistant knows not only what the contract says but also which orders for that customer are open.

For connecting business software a standard has emerged, which we explain in What is an MCP server?.

The answer is only as good as the filing

This is where most projects fail, not on the technology. If the same contract sits on the drive in three versions, the assistant finds all three and does not know which one applies. If the folders are called “New” and “New final”, that helps it as little as it helps your staff. If the price lists from 2019 were never deleted, they turn up in the answers.

An assistant makes the filing visible as it is. That is uncomfortable, but useful: before the rollout, it is worth going through the folders it is meant to read, moving outdated material to an archive and deciding where the valid version of a document lives.

An assistant that finds everything also finds the salary list and the draft notice terminating a supplier contract. Hence the rule: the assistant may only find what the person asking would be allowed to see themselves. Permissions are checked before the search, not after it. Someone without access to the “Management” folder on the drive gets no answer from that folder, not even a hint that something is in there.

In practice this means the permissions on the drive and in SharePoint have to be in order before the assistant arrives. Often only then does it emerge how many folders were shared with “Everyone” over time. How to set up roles and permissions sensibly is described in Roles and permissions in software.

How to recognise a good answer

A usable answer names its source: the file name, the page, the paragraph, and it links the document so the person asking can read on. Without a source, the answer is only a suggestion. A good assistant also says when it has found nothing suitable, rather than inventing an answer. Try both in the demonstration: ask something that is in the documents, and something that is certainly not.

Where the limits are

  • Tables in images. A scanned delivery note or a photographed price list is just a picture to the assistant at first. Whether it can read the text depends on the preparation.
  • Outdated versions. The assistant does not know that the new price list applies if the old one still sits next to it.
  • Arithmetic and counting. Ask how many orders are open in the mailbox and you get an estimate from text passages, not a figure from the system. Such questions belong with the ERP, not the document archive.
  • Connections across many documents. A question whose answer is spread over thirty sets of minutes gets an incomplete answer.

In-house or with a provider

The model that phrases the answer can run with a provider such as OpenAI, Microsoft or Mistral, or on your own server in-house, for instance with Ollama. For your documents that makes a difference: with a provider, the retrieved sections travel to its servers with every question. In-house, they never leave the building. Which option fits depends on what is in the documents and on what your data protection officer says. The trade-off is described in Running AI in-house.

Questions before you start

  • Which questions do your staff ask today whose answers sit in documents but are not found?
  • In which folders does the valid version live, and who clears out the old ones?
  • Are the permissions on the drive and in SharePoint correct?
  • May the content of these documents go to a provider, or does the model have to run in-house?
  • How will you notice after launch that the assistant is helping?

Once these are answered, you usually know which folders to start with. Starting small pays off: one area whose filing is in order, and a dozen questions against which you can check the answers.