Where the model runs

Microsoft has put the word Foundry on a device runtime, a cloud platform, and an on-premises Kubernetes offering, and has put the word Local on two of them in different positions. The result is a set of product names that a reader can transpose without noticing, and a set of architectures that could hardly be more different.

Untangling them is worth doing once, carefully, because the choice between them is not a preference. It is a decision about who operates the thing, who can reach it, and what happens when a second user shows up.

Three of these are easy to confuse and one of them has moved recently, so it is worth stating each precisely before any architecture is discussed.

Azure Local is Microsoft's on-premises platform — hyperconverged infrastructure software running on OEM-validated bare metal, hosting virtual machines and containers at distributed locations with central management through Azure Arc. It was previously branded Azure Stack HCI; existing deployments carried over without change to pricing, licensing, or support. It is infrastructure, not an AI product.

Foundry Local is the device runtime covered throughout this series — a native library and SDK that an application ships to an end user's machine.

Foundry Local on Azure Local is the third thing, and its name is the source of most of the confusion. It is not the device runtime installed on a server. It is a distinct offering: an Azure Arc extension deployed onto an Arc-enabled Kubernetes cluster running on the Azure Local platform.

Microsoft Foundry is the cloud platform, and it carries a name that changed. The documentation's own evolution table maps the previous brands — Azure AI Studio, then Azure AI Foundry — to the current one. Anything written more than a few months ago uses an older name for the same platform, which is worth remembering when reading around this subject.

The constraint that defines the device runtime is stated openly in its own documentation: it is optimized for hardware-constrained devices where a single user accesses the model at a time. It can be installed on server hardware, and it is not designed to be a server inference stack. Concurrent request queuing, continuous batching, and GPU sharing across many clients are explicitly out of scope, with vLLM and Triton Inference Server named as the tools that do that work.

A vendor naming its own product's ceiling, in its own docs, with pointers to competing software, is rarer than it should be. It also happens to be the most useful sentence in the entire product's documentation, because the failure it prevents is the common one: a team stands the device runtime up on a shared box, points four engineers at it, and then spends a fortnight diagnosing why throughput collapses under concurrency it was never built to handle.

One workstation, one user, one model in memory. The moment a second person depends on it, the product has been chosen wrong.

That second person is exactly where the Azure Local variant begins. Its stated purpose is enterprise-scale inference on on-premises infrastructure with Kubernetes-native operations and Azure Arc management, and its capability list reads like the inverse of the device runtime's limits — scaling inference across multi-node clusters for concurrent usage and high-parameter models.

The control plane is operator-based and declarative, which will read as familiar to anyone who has run workloads on Kubernetes:

Two details in that list carry more weight than their placement suggests.

The first is the engine choice. Generative inference runs on a default ONNX-GenAI engine across CPU or GPU, or on the vLLM engine for GPU-only high-throughput scenarios. The tool the device runtime's documentation names as the alternative for multi-user serving turns up here as a first-class option. That is a coherent story, not a contradiction: the same brand, with the right engine for each scale.

The second is the evaluation job. A declarative resource that scores a deployed model against a test dataset and keeps the results on the cluster is not a feature most inference platforms bother to ship. It is the difference between deploying a model and being able to answer what happened when it was swapped.

This is the point in the series where vocabulary has to change. A model on one engineer's workstation is local AI. A model on a cluster the organization owns, with a shared endpoint, identity-based access control, and a scaling profile, is self-hosted — the team version, with everything that implies about who now operates it and who gets paged.

Worth stating plainly: this variant is in preview and deployment access is by request through a form, available in a listed set of regions. It is not something to be swiped onto a credit card this afternoon.

One capability in that list deserves separating out, because it is the one that decides deals, not architectures: operation in disconnected environments where internet connectivity is not available, using a deployment model consistent with the connected case.

That consistency is the point. Plenty of platforms can be made to run air-gapped by someone determined enough. Considerably fewer keep the same deployment shape on both sides of the boundary, with documented architectural differences rather than a separate improvised path.

For organizations whose constraint is regulatory or sovereignty-driven instead of technical, this is usually the deciding capability, and it arrives well before any conversation about tokens per second.

The choice comes down to a single question asked in the right order, and the order is what most teams get wrong. Not how big is the model, and not how fast does it need to be. Ask instead: whose machine does the inference happen on, and how many people depend on it?

• One person, their own machine, an application they installed — the device runtime. Data never leaves, the application works offline, no subscription, no cluster, no operator.
• Many people, hardware the organization owns, on-premises by requirement — the Azure Local variant. Kubernetes-native, Arc-managed, identity-aware, multi-node, and someone now owns its uptime.
• Many people, no on-premises constraint, and the workload wants a large catalog and managed governance — the cloud platform, where the model is a service rather than a deployment.

The trap in this space is treating these as points on a scale from small to large, and therefore assuming a project graduates naturally from one to the next. It does not. Moving from the device runtime to the cluster variant is not a scale-up; it is a change of product, of operating model, and of who is responsible when it stops. Deciding which of those three a piece of work belongs to is worth doing before the first line of code, and worth revisiting only when the answer to the ownership question genuinely changes.