Running an on-premise LLM under the Swiss FADP: where to start

Why the question comes up now

Two years ago, running a useful language model on your own premises meant a hardware investment that was hard to justify. That is no longer the case. Open-weight models have gained in quality while their useful size has fallen, and a server with a single professional GPU now covers the majority of enterprise use cases.

Three motivations come up again and again with our clients: compliance, trade secrets, and control. They carry different weight depending on the sector, and it is worth separating them rather than blending them into a single argument.

What the Swiss FADP actually requires

Let us first clear away a common shortcut: the FADP does not forbid sending personal data abroad, it regulates that disclosure. A transfer is permitted to a country whose legislation the Federal Council considers to provide adequate protection, and those countries appear on an official list. Failing that, safeguards are required: standard contractual clauses, binding corporate rules, or one of the exceptions the law provides.

For the United States, Switzerland has recognised the Swiss-U.S. Data Privacy Framework since 2024: an American company certified under that framework is deemed to provide adequate protection. The AI vendor you are considering may well be certified. That still has to be verified, documented, and tracked over time, since the recent history of such frameworks shows they do not last forever.

This is precisely where self-hosting wins. It does not make the problem easier to solve, it makes it disappear. No disclosure abroad, therefore no transfer assessment to keep current, and no dependency on the validity of an adequacy framework you do not control.

What does not disappear are the remaining obligations: recording the processing in your register of activities, informing data subjects, setting retention periods, and carrying out an impact assessment where the processing presents a high risk. A model hosted on your premises exempts you from none of that.

The above describes our engineering practice and is not a substitute for legal advice.

Choosing the model: the smallest one that succeeds

The right question is not 'which is the best model' but 'which is the smallest model that succeeds at my task'. The cost difference between those two answers is considerable, and the quality difference perceived by users is often nil.

A model of 8 to 30 billion parameters, quantised to 8 or 4 bits where appropriate, covers extraction, classification, summarisation and question answering over a document corpus. The common open-weight families are all defensible starting points.

You settle it by measuring on your own task, never by reading a public leaderboard. A model that tops a general benchmark can fail on your domain vocabulary, and the opposite happens just as often.

Sizing the hardware

Three quantities govern the sizing: the size of the model in memory, the context length, and the number of concurrent requests.

The video memory required is roughly the parameter count multiplied by the chosen precision. A fourteen-billion-parameter model served at 8 bits needs on the order of fourteen gigabytes, around seven at 4 bits, plus the attention cache, which grows with context length and the number of concurrent users.

In practice, a server with a forty-eight-gigabyte professional GPU comfortably handles a model of that class for a few dozen internal users. Start small, measure, then grow. The reverse is expensive and buys nothing: an oversized GPU does not make the answers better.

RAG before fine-tuning

The reflex of 'we need to train the model on our data' is almost always premature. Retrieval-augmented generation brings business knowledge in at query time rather than burying it in the model's weights.

Three reasons to start there. Business knowledge changes, and an index updates in minutes where retraining takes days. Fine-tuning does not learn facts reliably, it adjusts style and format. And above all, only RAG allows sources to be cited, which is the precondition for use in any regulated context.

So reserve fine-tuning for cases where the output must follow a strict format or a very particular tone, and never for injecting knowledge.

Evaluate before deploying

This is where AI projects fail most often, by a wide margin. Without a business test set, there is no way to know whether the system regresses when the model, the prompt, the chunking or the document base changes. And all of those change constantly.

A useful evaluation set is short and real: thirty to a hundred questions asked by the future users, the expected answer, and the passage of the corpus that supports it. To that you add questions whose answer is not in the corpus, to check that the system knows how to stay silent.

Acceptance criteria are quantified before starting: proportion of correct answers, proportion of answers without a source, proportion of abstentions on out-of-corpus questions. That document is what lets you say the system is ready on something other than intuition.

Isolation, access, logging

An internal model is an infrastructure component like any other and is treated as such: isolated network or hosting on your premises, backups, monitoring, version management.

Access control deserves particular attention. It must align with the rights users already hold over the documents. A document assistant that ignores permissions becomes an internal data leak, and it is the most frequent mistake we find on projects already under way.

Finally, logging of queries, with a retention period that is decided rather than endured. Queries often contain more personal information than the corpus itself.

Validated environments and the EU AI Act

For validated environments, in pharma particularly, qualification sits within the GAMP 5 and Annex 11 framework: software categorisation, specifications, reproducible tests, change management. The non-deterministic nature of a language model does not prevent validation, it moves the object of proof from the component to the complete system and its evaluation set.

If your users or clients are in the European Union, the system has to be classified under the EU AI Act. Most internal document assistants fall under limited risk, with transparency obligations. A system involved in recruitment, access to credit or education moves into high risk, with considerably heavier documentation. That classification is done at the start of the project, not at the end.

The most frequent traps

Five mistakes recur often enough to be worth listing.

• Underestimating evaluation: without a business test set, there is no way to know whether the system regresses with each change.
• Oversizing the GPU: start small, measure, then grow; the reverse is expensive and buys nothing.
• Ignoring permissions: an assistant answering from documents the user is not allowed to read is a leak, not a feature.
• Neglecting corpus quality: RAG over badly recognised, duplicated or outdated documents produces wrong answers with sources attached, which is worse than no answer at all.
• Forgetting the lifecycle: a model gets updated, re-evaluated and replaced; plan for it from the architecture stage.

Where to start

One use case, a defined corpus, an evaluation set, six to twelve weeks. That format lets you decide without committing the company. If the result convinces, extending to other corpora is then far more data work than architecture.

And the most time-consuming task is never the one you expect: it is assembling and cleaning the corpus, not the technical integration.