On-PremiseLLMDeployment:DoYouActuallyNeedIt?

Introduction

The security review came back with one sentence. The data cannot leave our infrastructure. No detail, no definition, and no name attached to it. Two weeks later you are on a call with a hardware vendor who is very confident about how many GPUs you need, and you cannot tell whether that is a genuine requirement or a very good sales process.

Here is the thing that sentence hides. It is not one requirement. It is four, and they have wildly different answers. Most teams who arrive at an on-premise quote are trying to satisfy three of them that the cloud already satisfies by default, in writing, in documentation you can send to your auditor this afternoon.

The instinct to over-buy is rational, by the way. Nobody was ever fired for the architecture that kept the data in the building. But the strict option is not free, and its cost is not the hardware. It is the models you can no longer use.

One thing before we go further, because it should change how you read the rest. If nobody has written down which of those four you are being held to, stop reading and go and get that in writing first. Every hour spent designing against an undefined constraint is wasted, and the definition usually arrives smaller than the rumour.

Everything factual below is quoted from the vendors' own current documentation and linked to the exact page it came from. Retention terms change and a confident wrong number is worse than no number, so where we could not verify something today, we say so rather than guess.

What Does "The Data Cannot Leave Our Infrastructure" Actually Mean?

Ask four people in a compliance meeting what that sentence means and you will get four answers. They are all reasonable and they are not the same requirement. Separating them is the single highest-value hour in this entire project.

One: do not train on our data. The worry is that your customers' records end up baked into a model that answers somebody else's question next year. This is the fear that drove most of the early enterprise bans on AI tools, and it is largely a solved problem on commercial APIs.

Two: do not store our data. Different worry, and a more practical one. Even without training, a copy sitting in a vendor's logs is a copy inside somebody else's breach radius, and it is discoverable. This is the requirement most security teams actually mean.

Three: the vendor must never see our data. Stronger again. Not just that it is not kept, but that no human or system on the vendor's side has an access path to it in the first place. Auditors like this one because it is structural rather than procedural.

Four: the data must never leave our network at all. The absolute version. No outbound connection, no third party in the path, no exceptions. This is the only one of the four that on-premise is the answer to.

The reason this matters commercially is that requirements one through three are answered by configuration and contract, in days. Requirement four is answered by a programme, over quarters, and it changes what your product is capable of. Selling the fourth answer to someone who has the second problem is the most common expensive mistake in this space, and the pages ranking above this one are mostly written by companies whose product is the fourth answer.

Does the Model Provider Actually See Your Prompts?

This is the mechanism section, and it is the part worth sending to your security team, because it replaces an assumption with a documented architecture.

Take AWS Bedrock, which is the clearest documented case. AWS states that Bedrock uses a zero operator access model, which it defines as meaning that no operators of the service can access model input or output, and that it uses a zero data retention model, which it defines as meaning that by default Bedrock does not store model inputs or outputs. Those are the vendor's own words in its abuse detection documentation, not a marketing summary.

The structural part is more interesting than the policy part. AWS documents that in every region there is one model deployment account per model provider, that those accounts are owned and operated by the Bedrock service team, and that after a model is delivered AWS performs a deep copy of the provider's inference software into those accounts. The consequence is stated plainly in the data protection documentation: because the model providers do not have access to those accounts, they do not have access to Bedrock logs or to customer prompts and completions.

Read that again if requirement three is the one you are being held to. The model vendor having no access path is not something you have to engineer. On that platform it is how the platform is built, and the vendor documents it.

On training, Anthropic is similarly direct. Its privacy documentation states that by default it will not use your inputs or outputs from its commercial products, giving Claude for Work and the Anthropic API as examples, to train its models. That answers requirement one for that vendor, in a sentence you can quote in a questionnaire.

Now the honest qualification, because a page that only tells you the reassuring half is not worth much. Defaults have exceptions and the exceptions are specific. AWS documents that for certain named models, inputs and outputs are retained for up to 30 days for abuse detection, and that for one current Claude model you must opt in to sharing retained traffic with Anthropic for abuse detection and potential human review before you can use it at all. It also documents that retained data is processed by AWS and is not shared with third-party providers unless you opt in. So the default is strong, the exceptions are real, and which models you use decides which one you get.

Why Did a Model Disappear When You Turned On Zero Data Retention?

If you have hit this, you already know something is odd, because the model was there yesterday and today the API says it is unavailable. Enough people have hit it that the vendor's own error text now shows up in Google's autocomplete. It is not a bug. It is the tradeoff, working as designed.

Bedrock models data retention as a mode rather than a switch. The retention documentation defines four: default, where the model's own policy applies and AWS may retain data for safety purposes while the provider does not receive it. provider_data_share, where Bedrock may retain your inference data and share it with the model provider. none, which is true zero retention, where no request or response data is written to durable storage by AWS or shared with the provider. And inherit, which defers to a broader scope and is the default for new accounts.

The mode is resolved from the most specific scope that has an opinion. AWS documents the order as project, then account, then the model's own default, taking the first value that is not inherit. Worth knowing before you debug this, because a project-level setting quietly overrides the account-level one you thought was in force.

Here is the part that matters and that no vendor selling hardware will lead with. Each model separately declares which modes it will permit. AWS documents that if your effective mode is not in a model's permitted list, the model shows as unavailable and requests to it are blocked, and it names current frontier Claude models that require provider data sharing and permit nothing weaker. Set your organisation to none and those models switch off.

So the decision is not cloud against on-premise. It is a dial, and every notch toward strictness costs you model choice. That is a genuinely useful thing to know before a board meeting, because it reframes the conversation from a hardware request into a product one. The question stops being how do we keep the data in the building, and becomes which models does our product actually need, and what is the strictest retention posture that still leaves us those models.

Most teams have never asked the second question. It is usually answerable in a week, and it usually ends the on-premise conversation.

Does store=false Give You Zero Data Retention?

No. This is the shortest section on the page and possibly the most useful one.

There is an obvious-looking request parameter that appears to turn off storage, and a lot of teams set it, write zero retention in their compliance document and move on. AWS addresses this directly in the retention documentation: setting store=false does not guarantee zero data retention, because some models may still retain data for safety review even when it is set, and in that case the data is retained but is not retrievable by you. If you require guaranteed zero retention, the documented answer is to set the account or project retention mode to none.

Two different controls, at two different scopes, and only one of them does what the compliance document claims. If you have written zero retention into a security questionnaire on the strength of a request parameter, that is worth checking this week rather than during an audit.

Where Does Your Data Physically Sit?

Residency is a separate axis from retention and it catches people who have got retention right.

AWS documents that where cross-region inference is enabled for models that retain data, the retained inputs and outputs are stored in the destination region, meaning the region where the request was actually processed rather than the one you deployed into. Cross-region inference exists for good reasons, mostly capacity and latency. It also means the geographic promise you made in a contract can be quietly broken by a routing feature that was switched on to improve reliability.

If you have a residency obligation tied to a named geography, this is the setting to check first, before the hardware conversation starts. It is a common source of the belief that only on-premise can satisfy a residency rule, when the actual problem was one region setting nobody had audited.

When Is On-Premise Genuinely the Only Answer?

Sometimes it is, and a page that argued otherwise would be as dishonest as the ones arguing you always need it. Three situations, and they are narrower than the market implies.

The network itself is the boundary. Air-gapped and classified environments, where the requirement is not about who reads the data but about whether an outbound connection may exist at all. No contractual control answers this, because the control you need is physical.

A regulator or sovereignty regime requires processing on infrastructure you control, as a matter of policy rather than of demonstrated practice. Some regimes will not accept a vendor's documented architecture as evidence, however good it is. If your regulator is one of them, the argument is over and you should stop having it.

You have already promised it. If a signed customer contract says the processing happens on your own infrastructure, that is a commercial fact and not an engineering question. Renegotiate it or build it.

Notice what is not on that list. Being in a regulated industry is not on it. Handling personal data is not on it. Handling health data is not on it, because HIPAA is satisfied by a signed agreement and appropriate controls rather than by owning the hardware. A great many on-premise programmes are launched by organisations that fall into those categories and meet none of the three tests above.

What Do You Give Up by Running It Yourself?

Not going to talk about hardware here, because hardware is the part everybody already models and it is the part that turns out not to dominate. The things that hurt are the ones that do not appear on the quote.

Model quality, first and worst. On-premise means open-weight models. They have improved enormously and they still trail the frontier hosted models on hard reasoning, long-context work and tool use. If your product's value depends on the quality of the reasoning, you are trading away the thing the product is made of. Test this before you commit, with your actual prompts and your actual documents, not with a benchmark.

Capacity planning becomes yours. Hosted inference absorbs spiky traffic invisibly. Your own cluster does not. You now size for peak, and idle capacity is a permanent line in your infrastructure that a per-token bill simply does not have.

Serving is real engineering. Batching, quantisation choices, memory management, keeping throughput acceptable when several teams share the cluster. This is a specialism, and it is a different specialism from the one that built your product.

You now own evaluation. Hosted vendors improve models under you. When you run your own, every upgrade is a project, and you need an evaluation harness that can prove a new model did not quietly regress the behaviour your users depend on. Teams consistently underestimate this one, and it is the one that turns a launch into a permanent maintenance commitment.

And it does not end. Monitoring, on-call, security patching, driver and runtime upgrades. Standing infrastructure has a standing owner, and if nobody is named, reliability goes down rather than up.

None of that argues against on-premise where on-premise is required. It argues against choosing it when a retention mode would have done, which is the actual failure mode we see.

Who Should Not Build This?

Most of the people reading this page. If that reads oddly on a page published by a company that builds software, read the next paragraph.

We would earn considerably more from an on-premise AI programme than from telling you to change an account setting and sign an agreement. The first is a multi-quarter engagement with infrastructure work attached. The second is a phone call. We are telling you about the account setting because the alternative is taking money for solving a problem you did not have, and that tends to come out about six months in.

So, concretely. Close the tab and go back to building if none of the three tests above apply to you, and your requirement is really one of the first three readings. Do these instead, in this order. Get the requirement written down and signed by whoever owns it. Set your retention mode explicitly rather than relying on a default you have not read. Check whether cross-region inference is on. Get the agreement your regulator expects, whether that is a business associate agreement or a data processing agreement. Then check which models remain available to you under that posture, and design the product around those.

That sequence takes a fortnight and answers the question for most organisations. If you get to the end of it and the answer is still on-premise, you will at least know exactly why, and you will be able to tell your board in one sentence.

If you are earlier than that and still choosing how the AI layer should be built at all, our notes on building a product on the Claude API and on RAG pipeline architecture cover the ground before this decision. If you are weighing vendors on capability rather than on data handling, the comparison of the two main model APIs is the more useful page. And if the AI feature sits on top of a database that is already under strain, the data layer is usually the real constraint.

How Do You Enforce This Across an Organisation?

A decision that lives in a document is not a control. It survives until the next team spins up a project and picks the default, which is exactly the gap an auditor is trained to find.

The mechanism is a policy control at the organisation boundary. AWS documents that the retention write actions publish a condition key, which lets you write a service control policy denying any attempt to set a retention mode to anything other than none across every account in the organisation. The documentation includes the policy shape. The value of that is not really technical. It converts your answer from a promise about how your developers behave into a configuration an auditor can read directly, which is a much shorter conversation.

Pair it with four questions in your vendor questionnaire, phrased so a sales engineer cannot answer them all with one reassuring sentence. Do you train on our inputs or outputs, by default and under any circumstance. Do you store them, and for exactly how long. Can any person or system on your side read them, and what is the access path if so. In which geography does processing physically occur, and can a routing feature move it.

Ask them separately. The interesting answers are the ones where a vendor can say no to the first and has to say something longer to the second.

That is the whole architecture decision, and for most organisations it does not involve buying anything. Where it genuinely does, the shape of the work is a different conversation and we are happy to have it, including the part where we tell you it is not necessary.