How to Run DeepSeek Locally in 2026: R1 Distills, the 671B MoE, GGUF, Ollama, llama.cpp & vLLM

How do you run DeepSeek locally?

Pick the variant that fits your hardware. The realistic local option for most people is the DeepSeek-R1 distill family — dense models from 1.5B to 70B that run like the Qwen and Llama models they're distilled from: the 8B fits an 8GB card, and the 32B fits a single RTX 4090 (~20GB at a 4-bit quant). The full 671B-parameter DeepSeek-V3/R1 is a Mixture-of-Experts model with only 37B active per token — powerful, but its 4-bit GGUF is ~404GB, so it needs a large unified-memory machine or a GPU cluster (or Unsloth's dynamic 1.58-bit build at ~131GB). Everything is MIT-licensed, so nothing leaves your machine and there's no per-token bill. Below: exactly which model, what hardware, which tool, and the commands.

Key facts

• DeepSeek-V3 and R1 are 671B-parameter MoE models with 37B active per token (8 of 256 experts routed), Multi-head Latent Attention, and a 128K context window — MIT-licensed (weights and code).
• The six R1 distill models (1.5B, 7B, 8B, 14B, 32B, 70B) are dense and are what actually fits on one GPU — built on Qwen and Llama bases.
• The 32B distill needs ~20GB of VRAM (it is Qwen2.5-32B) — a single RTX 4090, matching our Qwen local-run numbers.
• The full 671B at Q4_K_M is ~404GB; Unsloth's dynamic 1.58-bit quant shrinks it to ~131GB (~80% smaller) while staying functional — naive 1-bit quantization breaks it.
• A Q4_K_M GGUF cuts VRAM ~60% versus FP16 at minimal quality loss — the default download for single-GPU use.
• SGLang is DeepSeek's own recommended serving engine; vLLM also supports V3/R1 natively.

Which DeepSeek model should you run locally?

DeepSeek's lineup causes more confusion than any other open model, so start here. Two families matter:

For almost everyone, the answer is a distill: R1-Distill-Qwen-32B for max quality on one 24GB card, or R1-Distill-Llama-8B / Qwen-7B for anything smaller. The full 671B (V3, the R1 reasoning model, the V3.1 hybrid, or the V3.2-Exp sparse-attention build) is worth knowing about but is a data-center model, not a desktop one.

The hardware reality: VRAM by model and quant

The distills are what you'll actually run. Practical targets at a 4-bit (Q4_K_M) GGUF — directional estimates, not official figures:

Two rules of thumb: leave 20–30% of VRAM free for the KV cache, and on Apple silicon count on unified memory (a 128GB Mac runs the 70B distill).

GGUF and quantization: what Q4_K_M actually buys you

For local use you download a GGUF — a single-file, quantized build that llama.cpp and Ollama load directly — not the original weights.

• Q4_K_M is the default: it cuts VRAM roughly 60% versus FP16 with negligible quality loss. Start here.
• Q5_K_M / Q6_K trade a few GB for a slight quality bump.
• Q8_0 is near-lossless but large.
• Q3 and below save memory but degrade — a fallback, not a default.

The practical move is the same as for any model: pick the largest model that fits at Q4, not a smaller model at Q8. A 32B distill at Q4 beats an 8B at Q8 on almost everything.

Can you run the full 671B DeepSeek?

Yes — but you need to be clever about it. The native FP8 weights are ~671–700GB, and even a Q4_K_M GGUF is ~404GB, which realistically means a ~512GB unified-memory box or a multi-GPU cluster. The trick that makes it runnable off a cluster is Unsloth's dynamic quantization, which shrinks the 671B from ~720GB to ~131GB (about 80% smaller) while keeping it functional:

The insight is that calibration matters more than bit-count: Unsloth keeps the first few layers and attention modules at 4–6 bit and pushes only the bulk MoE layers to ~1.5-bit. A naive all-layer 1-bit quant of DeepSeek produces loops and gibberish — the selective version doesn't. The 131GB build runs in ~160GB of VRAM (2× H100) at ~140 tokens/sec, or on a big-RAM box with the MoE experts offloaded to CPU. (Quant sizes shift between R1 revisions — confirm current numbers on the Unsloth Hugging Face repo.)

Which hardware actually runs which DeepSeek?

Which tool: Ollama, llama.cpp, vLLM, or SGLang?

For a quantized distill on your own box, Ollama is the fastest start. For serving the native FP8 671B on a cluster, SGLang (DeepSeek's own recommendation) or vLLM.

How to run it, step by step

Option A — Ollama (easiest):

ollama run deepseek-r1:8b     # an R1 distill
ollama run deepseek-r1:32b    # the 32B distill — fits a 4090

You get a chat prompt and an OpenAI-compatible endpoint at http://localhost:11434/v1.

Option B — llama.cpp (control): download a distill GGUF (e.g. a Q4_K_M), then:

llama-server -m DeepSeek-R1-Distill-Qwen-32B-Q4_K_M.gguf -c 8192 -ngl 99

-ngl 99 offloads all layers to GPU; -c 8192 sets context — lower it if you hit out-of-memory. For the full 671B on a big-RAM box, keep the MoE experts on CPU (llama.cpp's MoE-offload flags — confirm the current syntax against the Unsloth tutorial).

Option C — vLLM / SGLang (native, cluster):

vllm serve deepseek-ai/DeepSeek-R1 --tensor-parallel-size 8 --trust-remote-code

V3/R1 ship already in FP8, so don't re-quantize them; serve across 8 GPUs with tensor parallelism.

Performance: what to expect

On a single RTX 4090, rough single-user throughput by distill size:

The honest note: a full 671B on a big-RAM box (experts offloaded to CPU) is usable for batch and agentic work, not instant chat — the memory bandwidth caps it. The distills are where you get snappy interactive speed.

What can go wrong when self-hosting DeepSeek

• KV-cache out-of-memory at long context. 128K context is a huge KV cache — keep context modest, especially on the 671B.
• Undersized memory → disk offload. If RAM+VRAM can't cover the quant, it runs off disk and crawls. Size the box to the model.
• Confusing total vs. active params. The 671B uses 37B per token for speed, but you still need memory for the whole 671B. It's fast, not small.
• Downloading BF16 by habit. Grab the GGUF (or serve the native FP8), not the raw BF16 weights.
• Re-quantizing FP8 weights. V3/R1 are already FP8 — don't pass a quantization flag over them.
• Unofficial mirrors. Pull weights from DeepSeek's or Unsloth's official repos, not random re-uploads.

When to self-host vs. use the API

Self-host when data can't leave your environment (regulated industries, private code), when you want zero per-token cost at volume, or when you need a distill running on hardware you already own. Use the hosted API when you want full-671B quality without ~130–400GB of memory, when volume is low enough that per-token pricing beats a cluster, or when you're just building and don't want to operate infrastructure.

Frequently asked questions

What hardware do you need to run DeepSeek R1 locally?

For the distills: an 8GB GPU runs up to the 8B, an RTX 4090 (24GB) runs the 32B, and 2× 24GB cards or a 128GB Mac run the 70B. The full 671B needs ~130GB+ (Unsloth dynamic 1.58-bit) up to a multi-GPU cluster for native FP8.

Which DeepSeek model is best for local use?

R1-Distill-Qwen-32B for the best quality on a single 24GB card, or the 7B/8B distills for smaller machines. The full 671B is a data-center model, not a desktop one.

Can you run DeepSeek 671B locally?

Yes, with a big-RAM box or GPU cluster — its Q4_K_M GGUF is ~404GB. Unsloth's dynamic 1.58-bit build (~131GB) makes it runnable on ~160GB of VRAM (2× H100) or a large-RAM machine with CPU offload.

Is DeepSeek free to run locally?

Yes. DeepSeek-V3, R1, and the distills are MIT-licensed (the distills also inherit their Qwen/Llama base terms), so you can run, modify, and deploy them locally at no per-token cost.

Ollama or vLLM for DeepSeek?

Ollama for a single user running a quantized distill in seconds. vLLM (or SGLang, DeepSeek's recommended engine) for serving native weights to a team at high throughput.

What's the difference between R1 and the R1 distills?

R1 is the full 671B MoE reasoning model. The distills are smaller dense models (Qwen/Llama bases) trained on R1's outputs — far cheaper to run locally, with much of the reasoning behavior but not the full capability.

Related reading

• How to run Kimi K3 locally — the same playbook for a trillion-parameter MoE model.
• How to run Qwen 3 locally — the dense/MoE family the DeepSeek distills are built on.
• The 2026 AI infrastructure stack — the production serving stack around models like these.

Build the infra skills to run models like this

Self-hosting a model like DeepSeek — especially the 671B — is real infrastructure engineering: quantization, multi-GPU serving, KV-cache and memory management, reliability. Dexity's AI Platform Engineering course builds exactly that muscle, hands-on, so standing up a local DeepSeek deployment is a project you run, not just read about.

Model specs, VRAM figures, and tooling verified against DeepSeek's V3 report, the R1 and V3.2-Exp model cards, Unsloth's dynamic-GGUF guide, the Ollama library, and SGLang's DeepSeek docs (verified September 2026). Distill VRAM and tokens/sec are directional estimates; open-model versions, quant sizes, and tooling move quickly — confirm current builds before committing hardware. · Dexity.com