Local LLM lab: notes on running models on my own hardware

Frontier hosted models are fast and capable. They also see everything I send them. Some work is not appropriate to hand to a remote provider - client-adjacent code that has not been reviewed for what it exposes, personal notes I would not want a third party to index, and a small pile of one-off transforms where the fair comparison is not “hosted vs local” but “does this run at all if my internet is down.”

The shape

One dedicated box on the LAN. Ollama on the primary port, LM Studio on the secondary. Both expose an OpenAI-compatible endpoint. Every workstation on the LAN can point at the box the same way it would point at OpenAI - same client libraries, same call shape, same JSON responses.

The point of the LAN endpoint, past the privacy, is ergonomics. If a local model is only useful when I am sitting at the machine it runs on, I forget it exists. If it is a URL every laptop can reach, it becomes a real option in every session.

What actually runs

The catalog is smaller than the disk suggests. I have burned a lot of space on models that were interesting to look at once and never got used again. The ones that stay:

• A small general-purpose model for the quick “rewrite this paragraph” / “summarize this diff” tasks that do not need reasoning.
• A tool-capable model for the agentic work - the ones that reliably return well-formed tool calls when driven by an agent framework.
• A coding-focused model for autocomplete and code generation in editors that speak the OpenAI protocol.

Everything else has been culled. If I have not reached for a model in two weeks and cannot articulate the exact situation where I would, it comes off the disk. The signal is not “what benchmarks well.” The signal is “what do I actually use.”

What has not worked

• Serving from a laptop. The box is not a laptop, and every time I try to make one behave like a LAN endpoint I end up frustrated by battery and sleep behavior.
• Chasing 70B+ dense models on consumer hardware. The tokens-per-second numbers are honest but the interactive experience is not.
• Running too many models at once. VRAM is a hard limit; the model that gets served is the model that is loaded. Warm-swap is a real cost.

What is next

The endpoint is the current focus. What I want past that is a routing layer that picks the right local model for the task automatically - small model for a rewrite, tool-capable model for an agent call, coder model for an editor call - so the workstation does not have to know which model to ask. That is a small piece of code and I have not written it yet.