How to Run Llama and GPT-OSS Locally in 2026

How do you run Llama or GPT-OSS locally?

To run Llama or GPT-OSS locally, install a local runtime — Ollama is the easiest — then pull the model: ollama run gpt-oss:20b or ollama run llama3.1:8b. Everything runs offline on your own machine, so it's private and free after hardware. The model size you can run is set by your memory: a 7–8B model fits in ~8–16GB, OpenAI's gpt-oss-20b targets ~16GB, and gpt-oss-120b or Llama 3.3 70B need ~64–80GB (a single 80GB GPU, a 128GB Mac, or heavy CPU offload). Quantization (GGUF Q4_K_M) shrinks models to fit. Prefer Ollama or LM Studio for the desktop; vLLM for production throughput.

Key facts

• The easiest path is Ollama — one-line install, ollama run <model>, auto-download, and an OpenAI-compatible local API at localhost:11434.
• gpt-oss (OpenAI, released Aug 5, 2025, Apache 2.0) is a Mixture-of-Experts model that ships natively in 4-bit MXFP4 — don't re-quantize it. gpt-oss-20b (21B total / 3.6B active) targets ~16GB; gpt-oss-120b (117B / 5.1B active) runs on a single 80GB GPU. Both have a 128K context window.
• Llama uses the Llama Community License, not a pure open-source license — free for most use unless your product exceeds 700M monthly active users. GPT-OSS's Apache 2.0 is cleaner for commercial use.
• Your memory sets the ceiling. Rough VRAM ≈ (billions of params × bits-per-weight) ÷ 8, plus overhead and KV cache for context.
• Apple Silicon's unified memory lets a 128GB Mac hold 70B-class (and even gpt-oss-120b) models on one machine — something that needs an 80GB GPU or multi-GPU on the Nvidia side.
• The catch: locally-runnable models trail top hosted frontier models on hard reasoning, and the genuinely capable ones need expensive memory.

Can you run Llama and GPT-OSS locally?

Yes — both are open-weight models you can download and run entirely on your own hardware, no API key and no internet required after the download. That's the whole appeal: privacy (prompts never leave your machine), no per-token cost, and no rate limits. The only real question is whether your hardware has enough memory for the model you want.

The trade-off is capability and speed. A model small enough for a laptop is weaker and slower than a hosted frontier model; the models that rival hosted quality need serious memory. The sections below give you the exact math so you can pick a model that actually fits.

What is GPT-OSS, and how is it different from Llama?

GPT-OSS is OpenAI's family of open-weight models, released August 2025 under a permissive Apache 2.0 license. It comes in two sizes and uses a Mixture-of-Experts (MoE) design, so only a fraction of its parameters activate per token — which is why it runs on surprisingly modest hardware. Critically, it ships natively in MXFP4 (a 4-bit format): you download the official weights as-is and do not re-quantize them.

Llama is Meta's open-model family, the most widely supported local-model lineage, available in dense models (like Llama 3.1 8B and Llama 3.3 70B) and newer MoE models (Llama 4). The key difference beyond architecture is the license: Llama's Community License is free for research and commercial use unless your product has more than 700M monthly active users, whereas GPT-OSS's Apache 2.0 has no such clause.

What hardware do you need to run LLMs locally?

Memory is the gating factor — specifically how much your GPU (VRAM) or, on a Mac, your unified RAM can hold. Here are verified targets:

*Llama 4 Scout's 10M context is Meta's claimed maximum; usable context in practice is far lower. (Llama 4 Behemoth remains unreleased.)

How much VRAM or RAM do you actually need?

A practical rule: VRAM (GB) ≈ (params in billions × bits per weight) ÷ 8 + ~0.5GB overhead, plus extra for the KV cache that grows with context length. At 4-bit (the common default), that lands roughly here for dense models:

MoE models like gpt-oss are lower than their total-parameter count suggests — use the native targets in the table above (~16GB for 20b, 80GB for 120b). And remember: 128K context is the spec, but filling it consumes large KV-cache memory on top of the weights, so long context can run out of memory on a machine that loads the model fine.

What is quantization (GGUF, Q4_K_M, MXFP4)?

Quantization stores a model's weights at fewer bits to shrink its memory footprint. GGUF is the standard file format used by Ollama, LM Studio, and llama.cpp. The common levels:

• Q4_K_M — the default; ~4.85 bits/weight, roughly 30% of full-precision size, with about 1% quality loss. Start here.
• Q8_0 — ~8.5 bits, about half size, near-lossless. Step up when you have spare memory.
• FP16 — full precision, the baseline (and the largest).

Rule of thumb: pick the highest quant that fits with ~10% headroom. The exception is gpt-oss, which ships natively in MXFP4 (4-bit microscaling) — that native 4-bit is why it fits in such modest memory, so download the official weights and don't re-quantize them.

What is Ollama, and why is it the easiest option?

Ollama is the simplest way in. You install it once, then ollama run gpt-oss:20b auto-downloads the model and drops you into a chat. Run ollama serve and it exposes an OpenAI-compatible API at http://localhost:11434, so any client written against the OpenAI SDK can point at your local model with a one-line base-URL change. It also applies the correct chat template automatically (including gpt-oss's harmony format). For daily use, prototyping, and wiring a local model into an app, it's the default.

What are LM Studio, llama.cpp, and vLLM?

Ollama isn't the only runtime — pick by what you're doing:

For a single user on a laptop, Ollama or LM Studio. For serving many concurrent users, vLLM's continuous batching gives roughly 3–5× the throughput at 10+ users (at single-user it's roughly tied with Ollama).

Which tool should you use?

Decide by audience and scale. Daily driver / app integration: Ollama. No terminal: LM Studio. Maximum control or unusual hardware: llama.cpp. Serving many users in production: vLLM on GPUs. Privacy-first with zero telemetry: Jan. Chatting with your own documents: GPT4All. Most people start with Ollama and only reach for vLLM when they outgrow single-user throughput.

How do you run GPT-OSS and Llama locally (step by step)?

The simplest path, using Ollama:

Run GPT-OSS

1. Install Ollama from ollama.com (macOS, Windows, or Linux).
2. ollama run gpt-oss:20b (or ollama pull first). On an 80GB GPU or 128GB Mac: ollama run gpt-oss:120b.
3. Chat in the terminal, or ollama serve to expose the OpenAI-compatible API for your apps.

Run Llama

1. Same Ollama install.
2. ollama run llama3.1:8b (a safe default), or ollama run llama3.3:70b on a big-memory machine.
3. Point any OpenAI-SDK client at http://localhost:11434.

Confirm the exact model tags on ollama.com/library at the time you run this — tags change. Step up to vLLM when you need production-grade batching for many concurrent users; load the Hugging Face weights and it exposes an OpenAI-compatible server with far better multi-user throughput.

Can you run these models on a Mac (Apple Silicon)?

Yes, and Apple Silicon is unusually good at it. Because the CPU and GPU share one unified memory pool, a 128GB Mac can hold ~115–120GB of model — enough for 70B-class models and even gpt-oss-120b on a single machine, which on the Nvidia side would need an 80GB data-center GPU or multiple consumer cards (which top out around 24–32GB). The trade-off is memory bandwidth: a Mac's is lower than a high-end GPU's, so expect roughly 15–25 tokens/sec on a 30B model versus 2–3× faster on an RTX for the same size. Apple's MLX framework squeezes out the best Mac performance.

Is running LLMs locally private and free?

Both, with caveats. Once the model is downloaded, inference runs entirely offline — prompts and outputs never leave your machine (confirm your chosen app has no telemetry; Jan and llama.cpp are fully local). And it's free of per-token fees: you pay only for hardware and electricity, with no usage caps. The honest asterisk is capability — see the limitations below.

What's the catch? Limitations of local LLMs

• Quality gap. Local open models are strong but a 20B/70B you can run still trails top hosted frontier models on hard reasoning and coding. OpenAI positions gpt-oss-20b near its o3-mini tier and gpt-oss-120b near o4-mini — capable, not flagship.
• Speed. Consumer hardware gives modest tokens/sec; big models on a Mac are usable but not snappy, and CPU-only is slow.
• Hardware ceiling and cost. The genuinely capable models (120b, 70B, Llama 4) need expensive memory — an 80GB GPU, a 128GB Mac, or multi-GPU. Laptop-runnable models are the smaller, weaker ones.
• Context in practice. 128K is the spec, but filling it eats KV-cache memory on top of the weights and can run a machine out of memory.
• Setup friction. GPU drivers, quant selection, and offload flags still bite outside Ollama and LM Studio.
• Licensing. Llama isn't fully open (the 700M-MAU clause); gpt-oss (Apache 2.0) is cleaner for commercial use.

Frequently asked questions

Is it really free?

The software and weights are free to download and run; you pay only for hardware and electricity. No per-token API fees and no usage caps.

Is it private?

Yes. Once the model is downloaded, inference runs entirely offline — prompts and outputs never leave your machine. Verify your app has no telemetry; Jan and llama.cpp are fully local.

What's the minimum hardware to just try it?

About 16GB of RAM runs gpt-oss-20b or an 8B Llama at Q4 via Ollama on a normal laptop.

Can I run gpt-oss-120b on a gaming PC?

Not on a single consumer GPU (24–32GB). You need an 80GB data-center GPU, a 128GB Mac, or multi-GPU — otherwise it offloads to system RAM and crawls.

GGUF or MXFP4 — which quant should I use?

For Llama, download a GGUF at Q4_K_M by default. For gpt-oss, use the official MXFP4 weights as-is; don't re-quantize.

Ollama or vLLM?

Ollama for a single user, a laptop, or quick app integration; vLLM when you're serving many concurrent users on GPUs and need throughput.

Related reading

• Run DeepSeek locally — the same local-model playbook for DeepSeek's R1 and V3.
• The 2026 AI infrastructure stack — the serving layer underneath local and hosted models.
• How to build a RAG system — pair a local model with your own data.

Run models locally, then ship them for real

Running a model on your laptop is the easy part; serving one reliably, cheaply, and securely is the job. Dexity's AI Infrastructure Engineering course has you stand up and serve a production inference stack end to end, so you learn the serving, quantization, and cost trade-offs this guide only introduces.

Sources: model specs from OpenAI's gpt-oss announcement and model card, the Hugging Face GPT-OSS writeup and MXFP4 docs, the Llama 4 release and Ollama library; tooling and hardware guidance from local-LLM inference comparisons and Apple-Silicon-vs-Nvidia analyses (2026). Model tags, context maxima, and hardware targets move — confirm current figures before relying on them. US-only. · Dexity.com