When people talk about AI, most think of ChatGPT, Claude or Gemini: large models running at their providers in huge data centres. Alongside them there are openly available models you can download and run on your own hardware. For many tasks in a business, they are enough.

Why in-house at all?

The obvious reason is data protection: what is processed in-house does not leave the company. The more important reason is often a practical one. A model in your own network can reach data that a service on the internet cannot: the network drive, the mailboxes, the ERP. Only with that access do the answers become specific and refer to your orders, your contracts, your items.

Three ways to run AI

  • In-house. The model runs on your hardware, in your network. Nothing leaves the company.
  • With a provider. You use a large model over the internet, for example from OpenAI, Anthropic, Google or Mistral, or through Microsoft Azure. Beforehand, it is written down which data goes there and that it is not used for training.
  • Mixed. Confidential material stays in-house, and only tasks where a large model is clearly better go to the provider.

Which way fits is decided task by task, not once for the whole company.

What you need to run it in-house

A model

Openly available models come from large providers and keep improving. Well-known families are Llama from Meta, Mistral, Qwen from Alibaba, Gemma from Google and gpt-oss from OpenAI. They differ in size, language skills and licence. For German texts, test with real examples from your business before you commit. Check the licence: not every openly available model may be used commercially without restrictions.

Hardware

AI models compute on graphics cards, and what counts is their memory: the model has to fit. Smaller models run on a server with one powerful graphics card, larger ones need several cards or dedicated accelerators such as those from NVIDIA. The more staff ask at the same time, the more capacity is needed. An ordinary office PC is not enough.

Software to run it

For a model to answer questions, it needs software that loads it and makes it reachable, such as Ollama to get started or vLLM for operation with many users. Then there is the place where your staff work with it: a chat window or, better, a function directly inside the programs they already use.

The connection to your data

On its own, the model knows nothing about your business. It becomes useful when it can search your documents: on the network drive, in OneDrive and SharePoint, in shared mailboxes, in the ERP. For this, the documents are prepared so that the model finds the right passages for a question and answers from them. For connecting to programs, the Model Context Protocol (MCP) has become a widespread standard.

One rule applies without exception: before anything is retrieved, the system checks whether the person asking may see it. Otherwise the AI answers a trainee’s question about management salaries.

Someone who looks after it

A model in-house is a system like any other: it needs updates, monitoring and a person who is responsible. New models should be tested before they replace the old one, so answers do not change unnoticed.

What an in-house model can do - and what it cannot

Openly available models are strong at tasks with a clear frame:

  • reading documents and matching them against orders
  • summarising emails and routing them to the right place
  • searching your own documents and answering with a source
  • writing texts from a template

For difficult tasks, long contexts and tricky reasoning, the providers’ large models are usually still ahead. That is the case for the mixed route.

What it costs

We do not give a figure here, because it depends heavily on model, number of users and task. The structure is clear, though: in-house, you pay once for hardware and set-up and then for power and support - regardless of how much is asked. With a provider, you pay little at the start and then per use or per user. The more often and the more regularly a task comes up, the sooner running it yourself pays off.

Data protection and the AI Act

A model in-house makes many things easier, but not everything unnecessary. For each task, it should be written down which data is processed, who may access it and at which point a person checks the result. That is the basis on which your data protection officer can assess the legal side. Bring them in from the start.

Before you start

  • Which task should the AI take on, and how often does it come up?
  • Which data does it need, and where is that data?
  • Who may see which of this data?
  • Is an in-house model enough, or does the task need a large model?
  • Who is responsible once it runs?