Olive and the compile path

The first assumption most people form about Foundry Local is that its curated catalog is a fence. Models inside the fence run; models outside it do not; the vendor decides which is which.

That assumption is half right, and the wrong half is the interesting one. There is a documented, supported path from an arbitrary Hugging Face model to something Foundry Local will load, and the tool that walks it is Olive — a contraction of ONNX LIVE. The path is real. It is also a build step with a compile time, a failure mode, and a set of decisions nobody else can make on a team's behalf.

Olive is a hardware-aware model optimization toolchain. It takes a model source — a PyTorch model, an ONNX model, or Safetensors weights from a hub — plus a description of the production requirement and the target accelerator, and it runs a sequence of transformations to produce a deployment-ready artifact.

Those transformations are called passes, and the vocabulary is worth borrowing because it is accurate. Graph capture, quantization, compression, graph optimization: each is a discrete step that consumes a model and emits a model. Olive's contribution is composing them in an order that suits the declared target rather than requiring a hand-built pipeline per model per device.

The framing that makes it click for anyone who has shipped software: this is a compiler toolchain for model weights. Source in, target specified, optimized artifact out. The analogy holds further than it has any right to — including the part where the same source compiles differently for different targets, and the part where a build that works on one machine fails on another for reasons buried in the log.

Installation is unremarkable — Python 3.10 or later, then the package and its companions:

pip install olive-ai
pip install transformers onnxruntime-genai

The compile itself is a single command, and the documented example is worth reading not for the model it names but for the parameters it exposes:

olive optimize \
    --model_name_or_path meta-llama/Llama-3.2-1B-Instruct \
    --trust_remote_code \
    --output_path models/llama \
    --device cpu \
    --provider CPUExecutionProvider \
    --precision int4 \
    --log_level 1

Four of those flags are the actual decisions, and each has consequences that outlive the command:

The precision flag is the one that deserves a moment. It is not a performance tuning knob to be revisited later — it is a decision about how much of the model's fidelity to spend on fitting the target device. Choosing it well requires knowing what the application actually needs from the model, which is an evaluation question, not an optimization question, and no toolchain will answer it.

For the small model in the documented example, compilation takes roughly sixty seconds plus download time. That figure is reassuring and slightly misleading — it scales with parameter count and with how much work the chosen passes have to do.

A compiled model sitting in an output directory is invisible to Foundry Local until it is announced. The announcement is a small JSON file placed in the model directory:

{
  "Name": "llama-3.2:1"
}

Written to inference_model.json, that is the entire registration mechanism. The name is arbitrary; the default version is 1.

From there the application points its model cache directory at the parent folder, lists cached models, and finds the compiled artifact alongside anything pulled from the catalog:

var config = new Configuration
{
    AppName = "run-compiled-model",
    ModelCacheDir = "../models"
};

await FoundryLocalManager.CreateAsync(config, logger);
var mgr = FoundryLocalManager.Instance;
var catalog = await mgr.GetCatalogAsync();

var cachedModels = await catalog.GetCachedModelsAsync();
var model = cachedModels.FirstOrDefault(m => m.Id.Contains("llama-3.2:1"))
    ?? throw new Exception("Compiled model not found. Verify the ModelCacheDir path.");

await model.LoadAsync();

The pleasant surprise is that a self-compiled model is not a second-class citizen. It appears in the same cached-model list, loads through the same lifecycle, and serves the same chat client as a catalog entry. The curated catalog is a supply chain, not a permissions system.

The catalog is a supply chain, not a permissions system. What it withholds is convenience, not capability.

Microsoft's own documentation attaches a warning to that clean command line, and the warning is more useful than the command:

The Olive CLI and optimization settings change over time, and a single command line example might not work for every model, device, or execution provider.

The guidance that follows is to start from the olive-recipes repository instead of from a blank terminal. That repository holds per-model, per-hardware configurations: folders named for a model, subdivided by execution provider or device type, containing JSON configuration files and the occasional script. Recipes exist for widely used architectures across providers including OpenVINO, TensorRT, and QNN.

Reading that as documentation of a limitation misses the point. It is documentation of a structure. Model optimization is not a solved, uniform operation — it is per-architecture, per-accelerator work, and the recipe repository is where that work is accumulated in public. A team compiling a model that already has a recipe is copying known-good settings. A team compiling one that does not is doing the work the recipe would have recorded.

Which makes the repository the natural contribution surface for anyone working in this area. A recipe that does not exist yet is a gap with a defined shape and a defined destination.

It will. A newer architecture, an unusual quantization target, an execution provider that has not seen this operator set — the failure modes are ordinary compiler failure modes wearing unfamiliar names.

The documented troubleshooting covers the boring causes first, and they are worth ruling out before anything clever: authentication and access approval for gated models, a missing hub CLI, a cache directory pointing at the model folder rather than its parent, a project targeting a framework version the packages do not support.

Beyond those, the useful discipline is the one that applies to any build system. A failed compile with a complete log is a contribution; a failed compile abandoned in silence is nothing at all. The optimization settings are per-model and per-hardware by nature, which means a documented failure on a specific architecture, at a specific precision, against a specific provider is exactly the artifact the next person needs — and exactly what the recipe repository is for.

Compiling a model is not the hard part. Deciding what to target, and writing down what happened, is the part that compounds.