ThinkFacility
  1. Can I run it locally?
  2. Mistral

Mistral VRAM requirements: can I run Devstral Small 2 locally?

Devstral Small 2 and Mistral Small 3.2 are both 24B models at 20.5 GB with a 32K context, a fit for a 24 GB card.

Check your machine

Fits on the card 20.5 GB

Everything stays in video memory, which is the fast case.

  • Weights 14.3 GB
  • Context cache 5.4 GB
  • Overhead 0.8 GB
  • Fast memory
  • With system RAM

What we found

Mistral's two 24B models need the same memory at the same context. What sets Devstral Small 2 24B 2512 apart is how far that context goes: 384K tokens, and the cache for all of them is 64.4 GB, more than four times the 4-bit file. If you want Devstral's whole context, budget for the cache first. Running llama.cpp with a q8_0 cache halves it.

Ministral 3 3B 2512 shows the same thing at small scale: its 4-bit file is 2.1 GB, and 32K of context adds 3.5 GB, so the cache outweighs the model.

Devstral Small 2 24B 2512

23.6 billion parameters · up to 384K context · 48 thousand downloads in 30 days

Devstral Small 2 24B 2512 needs about 20.5 GB at Q4_K_M with a 32K-token context. The file itself is 14.3 GB, and the cache for 32K tokens adds 5.4 GB. That fits a 24 GB card (RTX 3090, RTX 4090).

Every token of context adds 160 KB of cache, so the full 384K would put 64.4 GB on top of the weights.

QuantFileAt 8KAt 32KSmallest GPUSmallest Mac
UD-IQ1_S5.6 GB7.7 GB11.7 GB12 GB16 GB*
UD-Q2_K_XL9.3 GB11.4 GB15.4 GB16 GB24 GB
UD-Q3_K_XL11.9 GB14.0 GB18.0 GB24 GB24 GB*
Q4_K_M14.3 GB16.4 GB20.5 GB24 GB24 GB*
Q5_K_M16.8 GB18.9 GB22.9 GB24 GB32 GB
Q6_K19.3 GB21.5 GB25.5 GB24 GB32 GB*
Q8_025.1 GB27.2 GB31.2 GB32 GB36 GB*
BF1647.2 GB49.3 GB53.3 GB96 GB64 GB*

Smallest GPU and Mac are for the 32K total. * with the Mac's GPU memory limit raised.

Sources: GGUF files · config.json

Mistral Small 3.2 24B 2506

23.6 billion parameters · up to 128K context · 47 thousand downloads in 30 days

At Q4_K_M, the Mistral Small 3.2 24B 2506 file is 14.3 GB, and with 32K tokens of context and runtime overhead it comes to about 20.5 GB. The smallest card that holds it whole is a 24 GB card (RTX 3090, RTX 4090).

The cache is the part that grows: 160 KB a token, or 21.5 GB if you use all 128K tokens it supports.

QuantFileAt 8KAt 32KSmallest GPUSmallest Mac
UD-IQ1_S5.6 GB7.7 GB11.7 GB12 GB16 GB*
UD-Q2_K_XL9.3 GB11.4 GB15.4 GB16 GB24 GB
UD-Q3_K_XL11.9 GB14.0 GB18.0 GB24 GB24 GB*
Q4_K_M14.3 GB16.4 GB20.5 GB24 GB24 GB*
Q5_K_M16.8 GB18.9 GB22.9 GB24 GB32 GB
Q6_K19.3 GB21.5 GB25.5 GB24 GB32 GB*
Q8_025.1 GB27.2 GB31.2 GB32 GB36 GB*
BF1647.2 GB49.3 GB53.3 GB96 GB64 GB*

Smallest GPU and Mac are for the 32K total. * with the Mac's GPU memory limit raised.

Sources: GGUF files · config.json

Ministral 3 3B 2512

3.4 billion parameters · up to 256K context · 20 thousand downloads in 30 days

Ministral 3 3B 2512 needs about 6.4 GB at Q4_K_M with a 32K-token context. The file itself is 2.1 GB, and the cache for 32K tokens adds 3.5 GB. That fits a 8 GB card (RTX 4060, RTX 3070). On a Mac, 16 GB of unified memory is enough.

Every token of context adds 104 KB of cache, so the full 256K would put 27.9 GB on top of the weights.

QuantFileAt 8KAt 32KSmallest GPUSmallest Mac
UD-IQ1_S1.0 GB2.6 GB5.2 GB8 GB16 GB
UD-Q2_K_XL1.5 GB3.1 GB5.8 GB8 GB16 GB
UD-Q3_K_XL1.9 GB3.5 GB6.1 GB8 GB16 GB
Q4_K_M2.1 GB3.8 GB6.4 GB8 GB16 GB
Q5_K_M2.5 GB4.1 GB6.7 GB8 GB16 GB
Q6_K2.8 GB4.5 GB7.1 GB8 GB16 GB
Q8_03.7 GB5.3 GB7.9 GB8 GB16 GB
BF166.9 GB8.5 GB11.1 GB12 GB16 GB

Smallest GPU and Mac are for the 32K total.

Sources: GGUF files · config.json

How the numbers are worked out

Total memory is three parts added together. Weights are the exact size of the GGUF file on Hugging Face, all parts summed when a quant is split. The context cache (the KV cache) comes from each model's own config.json:

cache = layers × kv_heads × head_dim × 2 (K and V) × 2 bytes × tokens

Sliding-window layers stop at their window plus llama.cpp's 512-token batch. Linear-attention and Mamba layers keep a fixed state instead, which we add once. Latent-attention models (DeepSeek, Kimi, GLM-5) cache kv_lora_rank + rope dimensions per layer, the way llama.cpp does. Overhead is our allowance: 0.5 GB for the GPU context plus the output buffer, 512 × vocabulary × 4 bytes.

A "24 GB" card holds 24 GiB, which is 25.8 GB in the decimal units file sizes use, and we count it that way. Macs get the share Metal allows by default, two thirds of unified memory up to 32 GB and three quarters above (llama.cpp discussion 2182); CPU-only machines and the DGX Spark keep 4 GB back for the operating system. All of it assumes llama.cpp's defaults: an f16 cache and one conversation at a time. Quantizing the cache to q8_0 halves it.

Files are Unsloth's GGUF uploads, the most downloaded quantizations on Hugging Face, and the list refreshes daily from Hugging Face's API. A model joins when its repo passes 20,000 downloads in 30 days. Last checked September 25, 2026.