How to Run Kimi Locally: Self-Hosting Kimi K3 with GGUF, Unsloth Studio & llama.cpp
August 1, 2026·9 min read
TL;DR
Kimi K3 is a 2.8T-parameter open-weight model (104B active) with a 1M-token context and native vision — and because the weights are open, you can run it entirely on your own hardware, no API and no data leaving your walls. The catch is size: full precision needs ~1.56 TB, but Unsloth's Dynamic GGUF quants shrink it dramatically — the 1-bit build fits in ~594 GB (62% smaller) at ~79% top-1 accuracy, and the 2-bit (~861 GB) reaches ~90%. This guide covers when self-hosting is worth it, which quant to pick, and the exact steps to run Kimi K3 locally via Unsloth Studio or llama.cpp.
How do you run Kimi locally?
Kimi K3's weights are open, so you download a quantized GGUF build and run it on your own machine — no API, no per-token bill, no data leaving your infrastructure. The practical path: grab an Unsloth Dynamic GGUF from huggingface.co/unsloth/Kimi-K3-GGUF, then run it with Unsloth Studio (a local web UI) or llama.cpp. The one rule to remember: your total memory (RAM + VRAM) needs to be roughly the size of the quant you choose.
This is the "own the model" companion to How to Start Using Kimi, which covers the hosted-API route.
Why run Kimi yourself?
Self-hosting only makes sense for specific reasons — but when they apply, they're decisive:
- Data never leaves your walls. The reason a bank, hospital, or defense contractor cares: a near-frontier model running on-prem means sensitive data is never sent to a third-party API. (For the governance case, see AI Governance in 2026.)
- No per-token bill at volume. If you're running millions of tokens a day, owned hardware can beat API pricing.
- No lock-in, full control. Open weights (Modified MIT) mean you pick the version, the quant, the serving stack, and you're never deprecated out from under.
If none of those apply, don't self-host — the hosted API is cheaper and far less operational work.
What it takes: the model and the memory
Kimi K3 is a 2.8-trillion-parameter mixture-of-experts model (104B active), with a 1M-token (1,048,576) context and native vision, using MXFP4 for its MoE weights. Full precision needs ~1.56 TB of memory — out of reach for most. Unsloth's Dynamic quantization is what makes it runnable, shrinking the model while protecting accuracy through careful calibration:
| Quant (Unsloth Dynamic) | Size | Accuracy (top-1 agreement) |
|---|---|---|
| UD-IQ1_S (1-bit) | ~594 GB | ~78.9% |
| UD-IQ1_M (1-bit) | ~649 GB | ~81.2% |
| UD-IQ2_XXS (2-bit) | ~711 GB | ~84.1% |
| UD-Q2_K_XL (2-bit) | ~861 GB | ~90.4% |
| UD-Q8_K_XL (lossless) | ~1.56 TB | full precision |
Hardware: the 1-bit build runs on a machine with ~610 GB of total memory — an NVIDIA DGX Station, or a Mac Studio paired with a large-RAM setup. On B200-class GPUs you'll see ~20 tokens/s generation and 120+ t/s throughput if the model fits. Rule of thumb: RAM + VRAM ≈ quant size, or it still runs but slows to disk-offload speeds.
How to run it, step by step
Step 1 — Pick a quant. Start with UD-IQ1_S (594 GB) for the best size/quality balance, or UD-Q2_K_XL (861 GB) if you have the memory and want ~90% accuracy. Use UD-Q8_K_XL only if you need truly lossless output and have ~1.56 TB.
Step 2 — Option A: Unsloth Studio (easiest). A local web UI that auto-offloads to RAM and detects multi-GPU:
# macOS / Linux / WSL
curl -fsSL https://unsloth.ai/install.sh | sh
unsloth studio # then open http://127.0.0.1:8888
Search "Kimi K3" in the Model hub, download your quant, and run — inference parameters auto-tune. It also does self-healing tool-calling, web search, and code execution out of the box.
Step 2 — Option B: llama.cpp (max control). Use Unsloth's fork for vision support:
git clone https://github.com/unslothai/llama.cpp && cd llama.cpp
git fetch origin pull/48/head:kimi-k3-fullsize-vision
git checkout kimi-k3-fullsize-vision && cd ..
cmake llama.cpp -B llama.cpp/build -DBUILD_SHARED_LIBS=OFF -DGGML_CUDA=ON # OFF for CPU/Metal
cmake --build llama.cpp/build --config Release -j --target llama-cli llama-server
Then download the GGUF and run:
hf download unsloth/Kimi-K3-GGUF --local-dir unsloth/Kimi-K3-GGUF \
--include "*mmproj-BF16*" --include "*UD-IQ1_S*"
./llama.cpp/llama-cli \
--model unsloth/Kimi-K3-GGUF/UD-IQ1_S/Kimi-K3-UD-IQ1_S-00001-of-00014.gguf \
--mmproj unsloth/Kimi-K3-GGUF/mmproj-BF16.gguf --temp 1.0 --top-p 0.95
Step 3 — Settings. Kimi K3 is thinking-only (reasoning always on). Set reasoning_effort to low, high, or max; use temperature 1.0, top-p 0.95 for default work (top-p 1.0 for agentic). Context runs up to 1M tokens.
When NOT to self-host
- You don't have ~600 GB+ of memory. Below the quant size, it runs off disk and crawls. Use the API.
- Your volume is low. The economics only favor owning hardware at scale; for occasional use, hosted is far cheaper all-in.
- You just want to code with it. For agentic coding via the API, see Using Kimi for Large-Scale Agentic Coding — no 600 GB box required.
Self-hosting is an infrastructure commitment. Standing up serving, quantization, and reliability is real engineering — the same discipline covered in the 2026 AI infrastructure stack.
FAQ
Can you run Kimi K3 locally?
Yes — the weights are open. Use an Unsloth Dynamic GGUF from Hugging Face; the 1-bit build (~594 GB) runs on a machine with ~610 GB total memory via Unsloth Studio or llama.cpp.
How much memory does Kimi K3 need?
Full precision needs ~1.56 TB. Unsloth's Dynamic quants shrink it: ~594 GB (1-bit, ~79% accuracy), ~861 GB (2-bit, ~90%), or the full 1.56 TB for lossless. Rule of thumb: RAM + VRAM ≈ the quant's size.
Which Kimi K3 quant should I use?
UD-IQ1_S (594 GB) is the recommended size/quality balance; UD-Q2_K_XL (861 GB) if you have the memory and want ~90% accuracy; UD-Q8_K_XL (1.56 TB) for lossless.
Is a local 1-bit Kimi any good?
Unsloth's dynamically quantized 1-bit reaches ~79% top-1 agreement and stays usable — far better than naïve 1-bit quants of the same size, because calibration matters more than bit-count.
Model and quantization details per Unsloth's Kimi K3 documentation and the Kimi-K3-GGUF repository on Hugging Face — model versions, quant sizes, and tooling move quickly; verify current builds before committing hardware. · Dexity.com
Dexity Sprint
AI Platform Engineering
Built for engineers who already own Kubernetes: six weeks across GPU Operator, vLLM, KServe, Triton, KubeFlow, and MLflow — the full AI infrastructure stack taught from the infra side.
