When does it make sense to run LLMs on your own hardware?

Hosted model APIs are the fastest way to get started with AI, and for many products they’re the right long-term answer too. But open-weight models have become good enough that “just call the API” is no longer the only reasonable default. Here’s the checklist I use when a team asks whether they should run models themselves.

Signs that local is worth a serious look

Your data can’t leave. Contracts, health records, source code, internal mail. If the honest answer to “can this be sent to a third party?” is no or only after legal review, self-hosting removes an entire category of discussion.

Your volume is steady and high. Per-token pricing is fantastic at low volume and painful at scale. A batch job that summarizes every document you ingest, every day, is a very different cost profile from an occasional chatbot query.

You need predictable latency, or offline operation. Inference on a box in your rack — or on the user’s own device — doesn’t depend on someone else’s queue depth or uptime.

The task is narrow. Classification, extraction, summarization, routing and retrieval-augmented Q&A are areas where a well-chosen mid-sized model, possibly fine-tuned, often matches a frontier model on your data.

Signs you should probably stay on an API

You’re still figuring out what the product is. Iterate on the best model available first; optimize later.
The workload is spiky and small. Idle GPUs are expensive.
You genuinely need frontier-level reasoning on open-ended tasks.
Nobody on the team wants to own another piece of infrastructure.

The costs people forget

Self-hosting isn’t free once the hardware is bought. Budget for:

Evaluation. You need a test set built from your real inputs to know whether a smaller model is good enough — and to catch regressions when you swap models.
Serving and monitoring. Throughput tuning, batching, context limits, GPU memory, logs, alerts.
Upgrades. Better open models ship constantly. Having an evaluation harness is what lets you adopt them safely.

A hybrid is often the answer

The setups I like best route most traffic to a local model and escalate only the hard cases to a hosted frontier model — with sensitive inputs pinned to the local path. You get most of the cost and privacy benefit while keeping a quality ceiling.

How I usually start

A short assessment: what’s the task, what data is involved, what are the latency and budget constraints. Then a proof of concept on a sample of real data with an honest comparison — local versus hosted — on quality, speed and cost. Sometimes the answer is “stay on the API”, and that’s a perfectly good outcome too.

If you’re weighing this decision, get in touch.