ThinkFacility
  1. Can I run it locally?
  2. Qwen

Qwen VRAM requirements: can I run Qwen3.8 27B locally?

Qwen3.8 27B at a 4-bit quant with a 32K-token context comes to 19.8 GB, so it fits a 24 GB card. Below, every Qwen size people are downloading, each quant, and what your own machine can hold.

Check your machine

Fits on the card 22.6 GB

Everything stays in video memory, which is the fast case.

  • Weights 18.6 GB
  • Context cache 3.2 GB
  • Overhead 0.8 GB
  • Fast memory
  • With system RAM

What we found

The number I didn't expect is the cache. Qwen3.6 35B A3B is the bigger model, and yet at its full 256K context it needs 28.6 GB, where the older Qwen3 Coder 30B A3B needs 45.1 GB for the same 256K. Since Qwen3.5, three layers in every four use linear attention, which keeps a small state of fixed size instead of a record of every token it has read. Long context got cheap.

On a 24 GB card the one to start with is Qwen3.8 27B. At UD-Q4_K_M it's 19.8 GB with a 32K context, and doubling the context to 64K only takes it to 21.9 GB. It's also what people are fetching: 6.9 million downloads of Unsloth's files in 30 days, beaten only by the Coder model from July 2025 (still at 11.6 million, which surprised me).

The A3B models are the ones for a smaller card. They hold 35 billion parameters but route each token through about 3 billion of them, so llama.cpp can leave the experts in system RAM and run the rest on a 12 GB GPU at a speed people live with. The dense 27B can't do that trick, because every token touches every weight.

Qwen3.8 Flash Next is another class of machine: 177 billion parameters and 111 GB at UD-Q4_K_XL. That's a DGX Spark, or a 128 GB Mac with its GPU memory limit raised. Our Qwen guide has which of these are open and which are API only.

Qwen3 Coder 30B A3B

30.5 billion parameters · mixture of experts, 8 of 128 active · up to 256K context · 11.6 million downloads in 30 days

Qwen3 Coder 30B A3B needs about 22.6 GB at Q4_K_M with a 32K-token context. The file itself is 18.6 GB, and the cache for 32K tokens adds 3.2 GB. That fits a 24 GB card (RTX 3090, RTX 4090). On a Mac, 32 GB of unified memory is enough. It routes each token through 8 of 128 experts, so it runs faster than its size suggests.

Every token of context adds 96 KB of cache, so the full 256K would put 25.8 GB on top of the weights.

QuantFileAt 8KAt 32KSmallest GPUSmallest Mac
UD-TQ1_08.0 GB9.6 GB12.0 GB12 GB16 GB*
UD-Q2_K_XL11.8 GB13.4 GB15.8 GB16 GB24 GB
UD-Q3_K_XL13.8 GB15.4 GB17.8 GB24 GB24 GB*
Q4_K_M18.6 GB20.2 GB22.6 GB24 GB32 GB
Q5_K_M21.7 GB23.3 GB25.8 GB24 GB32 GB*
Q6_K25.1 GB26.7 GB29.1 GB32 GB32 GB*
Q8_032.5 GB34.1 GB36.5 GB48 GB48 GB
BF1661.1 GB62.7 GB65.1 GB96 GB96 GB

Smallest GPU and Mac are for the 32K total. * with the Mac's GPU memory limit raised.

Sources: GGUF files · config.json

Qwen3.8 27B

27.3 billion parameters · up to 256K context · 6.9 million downloads in 30 days

At UD-Q4_K_M, the Qwen3.8 27B file is 16.5 GB, and with 32K tokens of context and runtime overhead it comes to about 19.8 GB. The smallest card that holds it whole is a 24 GB card (RTX 3090, RTX 4090).

Most of its layers use linear attention, which keeps a fixed state instead of a cache, so the cache grows by about 64 KB a token and reaches 17.3 GB at 256K.

QuantFileAt 8KAt 32KSmallest GPUSmallest Mac
UD-IQ1_S6.2 GB7.9 GB9.5 GB12 GB16 GB
UD-Q2_K_XL9.8 GB11.5 GB13.1 GB16 GB24 GB
UD-Q3_K_XL13.1 GB14.8 GB16.5 GB16 GB24 GB
UD-Q4_K_M16.5 GB18.2 GB19.8 GB24 GB24 GB*
UD-Q5_K_XL20.9 GB22.6 GB24.2 GB24 GB32 GB*
UD-Q6_K_XL25.3 GB27.0 GB28.6 GB32 GB32 GB*
Q8_029.0 GB30.7 GB32.4 GB32 GB36 GB*
BF1654.7 GB56.4 GB58.0 GB96 GB64 GB*

Smallest GPU and Mac are for the 32K total. * with the Mac's GPU memory limit raised.

Sources: GGUF files · config.json

Qwen3.8 Flash Next

177 billion parameters · mixture of experts, 10 of 512 active · up to 256K context · 1.8 million downloads in 30 days

Qwen3.8 Flash Next needs about 113 GB at UD-Q4_K_XL with a 32K-token context. The file itself is 111 GB, and the cache for 32K tokens adds 0.9 GB. No single consumer card holds it. A Mac with 128 GB of unified memory does once you raise the GPU memory limit, and a PC can split it between a GPU and system RAM. It's a mixture-of-experts model (10 of 512 experts per token), which is why people run it with the experts parked in system RAM and the rest on a 24 GB card.

Only 12 of its 48 layers keep a growing cache (the rest carry a fixed-size state), so the full 256K context costs 6.6 GB of cache.

QuantFileAt 8KAt 32KSmallest GPUSmallest Mac
UD-IQ1_S72.5 GB73.9 GB74.5 GB96 GB96 GB
UD-Q2_K_XL78.9 GB80.2 GB80.8 GB96 GB96 GB*
UD-Q3_K_XL90.0 GB91.3 GB91.9 GB96 GB96 GB*
UD-Q4_K_XL111 GB113 GB113 GBnone128 GB*
UD-Q5_K_XL158 GB160 GB160 GBnone256 GB
UD-Q6_K_XL169 GB170 GB171 GBnone256 GB
Q8_0188 GB190 GB190 GBnone256 GB
BF16354 GB355 GB356 GBnone512 GB

Smallest GPU and Mac are for the 32K total. * with the Mac's GPU memory limit raised.

Sources: GGUF files · config.json

Qwen3.6 35B A3B

34.7 billion parameters · mixture of experts, 8 of 256 active · up to 256K context · 1.3 million downloads in 30 days

At UD-Q4_K_M, the Qwen3.6 35B A3B file is 22.1 GB, and with 32K tokens of context and runtime overhead it comes to about 23.9 GB. The smallest card that holds it whole is a 24 GB card (RTX 3090, RTX 4090). It routes each token through 8 of 256 experts, so it runs faster than its size suggests.

Most of its layers use linear attention, which keeps a fixed state instead of a cache, so the cache grows by about 20 KB a token and reaches 5.4 GB at 256K.

QuantFileAt 8KAt 32KSmallest GPUSmallest Mac
UD-IQ1_M10.0 GB11.3 GB11.8 GB12 GB16 GB*
UD-Q2_K_XL12.3 GB13.5 GB14.0 GB16 GB24 GB
UD-Q3_K_XL16.8 GB18.1 GB18.6 GB24 GB24 GB*
UD-Q4_K_M22.1 GB23.4 GB23.9 GB24 GB32 GB*
UD-Q5_K_XL26.6 GB27.8 GB28.3 GB32 GB32 GB*
UD-Q6_K_XL31.8 GB33.1 GB33.6 GB32 GB36 GB*
Q8_036.9 GB38.1 GB38.6 GB48 GB48 GB
BF1669.4 GB70.6 GB71.1 GB96 GB96 GB

Smallest GPU and Mac are for the 32K total. * with the Mac's GPU memory limit raised.

Sources: GGUF files · config.json

Qwen3.5 9B

9.0 billion parameters · up to 256K context · 1.4 million downloads in 30 days

Qwen3.5 9B needs about 7.8 GB at Q4_K_M with a 32K-token context. The file itself is 5.7 GB, and the cache for 32K tokens adds 1.1 GB. That fits a 8 GB card (RTX 4060, RTX 3070). On a Mac, 16 GB of unified memory is enough.

Only 8 of its 32 layers keep a growing cache (the rest carry a fixed-size state), so the full 256K context costs 8.6 GB of cache.

QuantFileAt 8KAt 32KSmallest GPUSmallest Mac
UD-Q2_K_XL4.1 GB5.5 GB6.3 GB8 GB16 GB
UD-Q3_K_XL5.1 GB6.4 GB7.2 GB8 GB16 GB
Q4_K_M5.7 GB7.0 GB7.8 GB8 GB16 GB
Q5_K_M6.6 GB7.9 GB8.7 GB12 GB16 GB
Q6_K7.5 GB8.8 GB9.6 GB12 GB16 GB
Q8_09.5 GB10.9 GB11.7 GB12 GB16 GB*
BF1617.9 GB19.3 GB20.1 GB24 GB24 GB*

Smallest GPU and Mac are for the 32K total. * with the Mac's GPU memory limit raised.

Sources: GGUF files · config.json

Qwen3.6 27B

26.9 billion parameters · up to 256K context · 981 thousand downloads in 30 days

At Q4_K_M, the Qwen3.6 27B file is 16.8 GB, and with 32K tokens of context and runtime overhead it comes to about 20.1 GB. The smallest card that holds it whole is a 24 GB card (RTX 3090, RTX 4090).

Most of its layers use linear attention, which keeps a fixed state instead of a cache, so the cache grows by about 64 KB a token and reaches 17.3 GB at 256K.

QuantFileAt 8KAt 32KSmallest GPUSmallest Mac
UD-Q2_K_XL11.8 GB13.6 GB15.2 GB16 GB24 GB
UD-Q3_K_XL14.5 GB16.2 GB17.8 GB24 GB24 GB*
Q4_K_M16.8 GB18.5 GB20.1 GB24 GB24 GB*
Q5_K_M19.5 GB21.2 GB22.8 GB24 GB32 GB
Q6_K22.5 GB24.2 GB25.8 GB32 GB32 GB*
Q8_028.6 GB30.3 GB31.9 GB32 GB36 GB*
BF1653.8 GB55.5 GB57.1 GB96 GB64 GB*

Smallest GPU and Mac are for the 32K total. * with the Mac's GPU memory limit raised.

Sources: GGUF files · config.json

Qwen3.5 4B

4.2 billion parameters · up to 256K context · 922 thousand downloads in 30 days

Qwen3.5 4B needs about 4.9 GB at Q4_K_M with a 32K-token context. The file itself is 2.7 GB, and the cache for 32K tokens adds 1.1 GB. That fits a 8 GB card (RTX 4060, RTX 3070). On a Mac, 16 GB of unified memory is enough.

Only 8 of its 32 layers keep a growing cache (the rest carry a fixed-size state), so the full 256K context costs 8.6 GB of cache.

QuantFileAt 8KAt 32KSmallest GPUSmallest Mac
UD-Q2_K_XL1.9 GB3.3 GB4.1 GB8 GB16 GB
UD-Q3_K_XL2.4 GB3.8 GB4.6 GB8 GB16 GB
Q4_K_M2.7 GB4.1 GB4.9 GB8 GB16 GB
Q5_K_M3.1 GB4.5 GB5.3 GB8 GB16 GB
Q6_K3.5 GB4.9 GB5.7 GB8 GB16 GB
Q8_04.5 GB5.8 GB6.6 GB8 GB16 GB
BF168.4 GB9.8 GB10.6 GB12 GB16 GB

Smallest GPU and Mac are for the 32K total.

Sources: GGUF files · config.json

Qwen3 30B A3B 2507

30.5 billion parameters · mixture of experts, 8 of 128 active · up to 256K context · 521 thousand downloads in 30 days

At Q4_K_M, the Qwen3 30B A3B 2507 file is 18.6 GB, and with 32K tokens of context and runtime overhead it comes to about 22.6 GB. The smallest card that holds it whole is a 24 GB card (RTX 3090, RTX 4090). On a Mac, 32 GB of unified memory is enough. It routes each token through 8 of 128 experts, so it runs faster than its size suggests.

The cache is the part that grows: 96 KB a token, or 25.8 GB if you use all 256K tokens it supports.

QuantFileAt 8KAt 32KSmallest GPUSmallest Mac
UD-TQ1_08.1 GB9.7 GB12.1 GB12 GB16 GB*
UD-Q2_K_XL11.8 GB13.4 GB15.8 GB16 GB24 GB
UD-Q3_K_XL13.8 GB15.4 GB17.9 GB24 GB24 GB*
Q4_K_M18.6 GB20.2 GB22.6 GB24 GB32 GB
Q5_K_M21.7 GB23.3 GB25.8 GB24 GB32 GB*
Q6_K25.1 GB26.7 GB29.1 GB32 GB32 GB*
Q8_032.5 GB34.1 GB36.5 GB48 GB48 GB
BF1661.1 GB62.7 GB65.1 GB96 GB96 GB

Smallest GPU and Mac are for the 32K total. * with the Mac's GPU memory limit raised.

Sources: GGUF files · config.json

Qwen3 4B

4.0 billion parameters · up to 40K context · 462 thousand downloads in 30 days

Qwen3 4B needs about 8.1 GB at Q4_K_M with a 32K-token context. The file itself is 2.5 GB, and the cache for 32K tokens adds 4.8 GB. That fits a 8 GB card (RTX 4060, RTX 3070). On a Mac, 16 GB of unified memory is enough.

Every token of context adds 144 KB of cache, so the full 40K would put 6.0 GB on top of the weights.

QuantFileAt 8KAt 32KSmallest GPUSmallest Mac
UD-IQ1_S1.1 GB3.1 GB6.7 GB8 GB16 GB
UD-Q2_K_XL1.7 GB3.7 GB7.3 GB8 GB16 GB
UD-Q3_K_XL2.1 GB4.2 GB7.8 GB8 GB16 GB
Q4_K_M2.5 GB4.5 GB8.1 GB8 GB16 GB
Q5_K_M2.9 GB4.9 GB8.5 GB8 GB16 GB
Q6_K3.3 GB5.3 GB8.9 GB12 GB16 GB
Q8_04.3 GB6.3 GB9.9 GB12 GB16 GB
BF168.1 GB10.1 GB13.7 GB16 GB24 GB

Smallest GPU and Mac are for the 32K total.

Sources: GGUF files · config.json

Qwen AgentWorld 35B A3B

34.7 billion parameters · mixture of experts, 8 of 256 active · up to 256K context · 382 thousand downloads in 30 days

At UD-Q4_K_M, the Qwen AgentWorld 35B A3B file is 22.1 GB, and with 32K tokens of context and runtime overhead it comes to about 23.9 GB. The smallest card that holds it whole is a 24 GB card (RTX 3090, RTX 4090). It routes each token through 8 of 256 experts, so it runs faster than its size suggests.

Most of its layers use linear attention, which keeps a fixed state instead of a cache, so the cache grows by about 20 KB a token and reaches 5.4 GB at 256K.

QuantFileAt 8KAt 32KSmallest GPUSmallest Mac
UD-Q2_K_XL12.3 GB13.5 GB14.0 GB16 GB24 GB
UD-Q3_K_XL16.8 GB18.0 GB18.5 GB24 GB24 GB*
UD-Q4_K_M22.1 GB23.4 GB23.9 GB24 GB32 GB*
UD-Q5_K_XL26.5 GB27.8 GB28.3 GB32 GB32 GB*
UD-Q6_K_XL31.8 GB33.1 GB33.6 GB32 GB36 GB*
Q8_036.9 GB38.1 GB38.6 GB48 GB48 GB
BF1669.4 GB70.6 GB71.1 GB96 GB96 GB

Smallest GPU and Mac are for the 32K total. * with the Mac's GPU memory limit raised.

Sources: GGUF files · config.json

Qwen3.5 2B

1.9 billion parameters · up to 256K context · 319 thousand downloads in 30 days

Qwen3.5 2B needs about 2.7 GB at Q4_K_M with a 32K-token context. The file itself is 1.3 GB, and the cache for 32K tokens adds 0.4 GB. That fits a 8 GB card (RTX 4060, RTX 3070). On a Mac, 16 GB of unified memory is enough.

Only 6 of its 24 layers keep a growing cache (the rest carry a fixed-size state), so the full 256K context costs 3.2 GB of cache.

QuantFileAt 8KAt 32KSmallest GPUSmallest Mac
UD-Q2_K_XL1.0 GB2.1 GB2.4 GB8 GB16 GB
UD-Q3_K_XL1.2 GB2.3 GB2.6 GB8 GB16 GB
Q4_K_M1.3 GB2.4 GB2.7 GB8 GB16 GB
Q5_K_M1.4 GB2.6 GB2.9 GB8 GB16 GB
Q6_K1.6 GB2.7 GB3.0 GB8 GB16 GB
Q8_02.0 GB3.1 GB3.4 GB8 GB16 GB
BF163.8 GB4.9 GB5.2 GB8 GB16 GB

Smallest GPU and Mac are for the 32K total.

Sources: GGUF files · config.json

Qwen3.5 27B

26.9 billion parameters · up to 256K context · 314 thousand downloads in 30 days

At Q4_K_M, the Qwen3.5 27B file is 16.7 GB, and with 32K tokens of context and runtime overhead it comes to about 20.1 GB. The smallest card that holds it whole is a 24 GB card (RTX 3090, RTX 4090).

Most of its layers use linear attention, which keeps a fixed state instead of a cache, so the cache grows by about 64 KB a token and reaches 17.3 GB at 256K.

QuantFileAt 8KAt 32KSmallest GPUSmallest Mac
UD-Q2_K_XL11.2 GB12.9 GB14.5 GB16 GB24 GB
UD-Q3_K_XL14.4 GB16.1 GB17.8 GB24 GB24 GB*
Q4_K_M16.7 GB18.4 GB20.1 GB24 GB24 GB*
Q5_K_M19.6 GB21.3 GB22.9 GB24 GB32 GB*
Q6_K22.5 GB24.2 GB25.8 GB24 GB32 GB*
Q8_028.6 GB30.3 GB31.9 GB32 GB36 GB*
BF1653.8 GB55.5 GB57.1 GB96 GB64 GB*

Smallest GPU and Mac are for the 32K total. * with the Mac's GPU memory limit raised.

Sources: GGUF files · config.json

Qwen3 VL 4B

4.0 billion parameters · up to 256K context · 314 thousand downloads in 30 days

Qwen3 VL 4B needs about 8.1 GB at Q4_K_M with a 32K-token context. The file itself is 2.5 GB, and the cache for 32K tokens adds 4.8 GB. That fits a 8 GB card (RTX 4060, RTX 3070). On a Mac, 16 GB of unified memory is enough.

Every token of context adds 144 KB of cache, so the full 256K would put 38.7 GB on top of the weights.

QuantFileAt 8KAt 32KSmallest GPUSmallest Mac
UD-IQ1_S1.1 GB3.1 GB6.7 GB8 GB16 GB
UD-Q2_K_XL1.7 GB3.7 GB7.3 GB8 GB16 GB
UD-Q3_K_XL2.1 GB4.1 GB7.8 GB8 GB16 GB
Q4_K_M2.5 GB4.5 GB8.1 GB8 GB16 GB
Q5_K_M2.9 GB4.9 GB8.5 GB8 GB16 GB
Q6_K3.3 GB5.3 GB8.9 GB12 GB16 GB
Q8_04.3 GB6.3 GB9.9 GB12 GB16 GB
BF168.1 GB10.1 GB13.7 GB16 GB24 GB

Smallest GPU and Mac are for the 32K total.

Sources: GGUF files · config.json

Qwen3.5 122B A10B

122 billion parameters · mixture of experts, 8 of 256 active · up to 256K context · 261 thousand downloads in 30 days

At Q4_K_M, the Qwen3.5 122B A10B file is 76.5 GB, and with 32K tokens of context and runtime overhead it comes to about 78.5 GB. The smallest card that holds it whole is a RTX PRO 6000 (96 GB). It's a mixture-of-experts model (8 of 256 experts per token), which is why people run it with the experts parked in system RAM and the rest on a 24 GB card.

Most of its layers use linear attention, which keeps a fixed state instead of a cache, so the cache grows by about 24 KB a token and reaches 6.6 GB at 256K.

QuantFileAt 8KAt 32KSmallest GPUSmallest Mac
UD-IQ1_M34.2 GB35.6 GB36.2 GB48 GB48 GB
UD-Q2_K_XL41.8 GB43.2 GB43.8 GB48 GB48 GB*
UD-Q3_K_XL57.0 GB58.3 GB58.9 GB96 GB64 GB*
Q4_K_M76.5 GB77.9 GB78.5 GB96 GB96 GB*
Q5_K_M91.5 GB92.9 GB93.5 GB96 GB96 GB*
Q6_K101 GB102 GB103 GB96 GB128 GB
Q8_0130 GB131 GB132 GBnone128 GB*
BF16244 GB246 GB246 GBnone256 GB*

Smallest GPU and Mac are for the 32K total. * with the Mac's GPU memory limit raised.

Sources: GGUF files · config.json

How the numbers are worked out

Total memory is three parts added together. Weights are the exact size of the GGUF file on Hugging Face, all parts summed when a quant is split. The context cache (the KV cache) comes from each model's own config.json:

cache = layers × kv_heads × head_dim × 2 (K and V) × 2 bytes × tokens

Sliding-window layers stop at their window plus llama.cpp's 512-token batch. Linear-attention and Mamba layers keep a fixed state instead, which we add once. Latent-attention models (DeepSeek, Kimi, GLM-5) cache kv_lora_rank + rope dimensions per layer, the way llama.cpp does. Overhead is our allowance: 0.5 GB for the GPU context plus the output buffer, 512 × vocabulary × 4 bytes.

A "24 GB" card holds 24 GiB, which is 25.8 GB in the decimal units file sizes use, and we count it that way. Macs get the share Metal allows by default, two thirds of unified memory up to 32 GB and three quarters above (llama.cpp discussion 2182); CPU-only machines and the DGX Spark keep 4 GB back for the operating system. All of it assumes llama.cpp's defaults: an f16 cache and one conversation at a time. Quantizing the cache to q8_0 halves it.

Files are Unsloth's GGUF uploads, the most downloaded quantizations on Hugging Face, and the list refreshes daily from Hugging Face's API. A model joins when its repo passes 20,000 downloads in 30 days. Last checked September 25, 2026.