ThinkFacility
  1. Can I run it locally?
  2. Inkling

Inkling VRAM requirements: can I run Inkling Small locally?

Inkling Small has 264 billion parameters, 163 GB at 4-bit. Its smallest file, UD-IQ1_S (74.8 GB), is the one that fits a 128 GB machine.

Check your machine

Won't fit 165 GB

Weights, cache and overhead add up to more than the card and the RAM together.

  • Weights 163 GB
  • Context cache 1.1 GB
  • Overhead 0.9 GB
  • Fast memory
  • With system RAM

What we found

Small is relative here. Inkling Small is a mixture of experts with 264 billion parameters, and the file that gets it onto a DGX Spark or a 128 GB Mac is the smallest one, UD-IQ1_S (74.8 GB). At 4-bit it's a 256 GB Mac.

Its cache stays small for a model this size. 35 of its 42 layers only look back 512 tokens, so the full million-token context adds 30.2 GB.

Inkling is the big one at 947 billion parameters. Its smallest file, UD-IQ1_S (270 GB), fits a 512 GB Mac Studio.

Inkling Small

264 billion parameters · mixture of experts, 6 of 256 active · up to 1M context · 1.2 million downloads in 30 days

Inkling Small needs about 165 GB at UD-Q4_K_M with a 32K-token context. The file itself is 163 GB, and the cache for 32K tokens adds 1.1 GB. No single consumer card holds it. A Mac with 256 GB of unified memory does, and a PC can split it between a GPU and system RAM. It routes each token through 6 of 256 experts, so it runs faster than its size suggests.

35 of its layers only look back 512 tokens, so they stop growing early, and the whole cache at 1M is 30.2 GB.

QuantFileAt 8KAt 32KSmallest GPUSmallest Mac
UD-IQ1_S74.8 GB76.1 GB76.8 GB96 GB96 GB
UD-Q2_K_XL87.9 GB89.2 GB89.9 GB96 GB96 GB*
UD-Q3_K_XL120 GB121 GB122 GBnone128 GB*
UD-Q4_K_M163 GB164 GB165 GBnone256 GB
UD-Q5_K_XL197 GB198 GB199 GBnone256 GB
UD-Q6_K_XL240 GB241 GB242 GBnone256 GB*
Q8_0280 GB282 GB282 GBnone512 GB
BF16527 GB529 GB529 GBnone512 GB*

Smallest GPU and Mac are for the 32K total. * with the Mac's GPU memory limit raised.

Sources: GGUF files · config.json

Inkling

947 billion parameters · mixture of experts, 6 of 256 active · up to 1M context · 588 thousand downloads in 30 days

At UD-Q4_K_XL, the Inkling file is 587 GB, and with 32K tokens of context and runtime overhead it comes to about 590 GB. That's past a 512 GB Mac. The smallest file, UD-IQ1_S at 270 GB, fits a 512 GB Mac. It routes each token through 6 of 256 experts, so it runs faster than its size suggests.

A sliding window of 512 tokens on 55 layers keeps the cache small: 47.7 GB at the full 1M.

QuantFileAt 8KAt 32KSmallest GPUSmallest Mac
UD-IQ1_S270 GB272 GB273 GBnone512 GB
UD-Q2_K_XL317 GB319 GB320 GBnone512 GB
UD-Q3_K_XL433 GB435 GB436 GBnone512 GB*
UD-Q4_K_XL587 GB589 GB590 GBnonenone
Q8_0857 GB858 GB860 GBnonenone
BF161,894 GB1,896 GB1,897 GBnonenone

Smallest GPU and Mac are for the 32K total. * with the Mac's GPU memory limit raised.

Sources: GGUF files · config.json

How the numbers are worked out

Total memory is three parts added together. Weights are the exact size of the GGUF file on Hugging Face, all parts summed when a quant is split. The context cache (the KV cache) comes from each model's own config.json:

cache = layers × kv_heads × head_dim × 2 (K and V) × 2 bytes × tokens

Sliding-window layers stop at their window plus llama.cpp's 512-token batch. Linear-attention and Mamba layers keep a fixed state instead, which we add once. Latent-attention models (DeepSeek, Kimi, GLM-5) cache kv_lora_rank + rope dimensions per layer, the way llama.cpp does. Overhead is our allowance: 0.5 GB for the GPU context plus the output buffer, 512 × vocabulary × 4 bytes.

A "24 GB" card holds 24 GiB, which is 25.8 GB in the decimal units file sizes use, and we count it that way. Macs get the share Metal allows by default, two thirds of unified memory up to 32 GB and three quarters above (llama.cpp discussion 2182); CPU-only machines and the DGX Spark keep 4 GB back for the operating system. All of it assumes llama.cpp's defaults: an f16 cache and one conversation at a time. Quantizing the cache to q8_0 halves it.

Files are Unsloth's GGUF uploads, the most downloaded quantizations on Hugging Face, and the list refreshes daily from Hugging Face's API. A model joins when its repo passes 20,000 downloads in 30 days. Last checked September 25, 2026.