- Can I run it locally?
- Best GPU for local LLMs
Best GPU for Local LLMs: What Each VRAM Tier Runs
Memory decides what runs. Here's every card, Mac and mini PC on sale now against the open models we track, from the same file sizes as our VRAM calculator.
How much VRAM do I need?
Each bar counts the models that fit whole at their common 4-bit file with a 32K context.
Also fits the smaller machine aboveAdded by this step up
Graphics cards
16 GB card32 of 78Biggest: LFM2.5 24B A2B24 GB card46 of 78Biggest: Qwen3.6 35B A3B32 GB card50 of 78Biggest: DeepSeek R1 Distill Qwen 32B48 GB card51 of 78Biggest: Nemotron 3.5 Bolt 74B A9B96 GB card62 of 78Biggest: Devstral 2 123B 2512What a 4-bit model actually needs
Qwen3.8 27B is a fair yardstick (6.4 million downloads in 30 days). Its 4-bit file is 16.6 GB, and with a 32K context and llama.cpp's buffers it needs 19.9 GB. At 64K it's 22.0 GB.
The cache, the keys and values a model keeps for every token it has read, is the part that varies. On Llama 3.3 70B the full 128K costs 42.9 GB by itself.
We count memory the way each machine hands it to the GPU: all of a card's VRAM, two thirds of a Mac's unified memory up to 32 GB and three quarters above, and the 96 GB of 128 that AMD lets the Ryzen AI Max+ 395 give its graphics.
16, 24 and 32 GB graphics cards: RTX 5060 Ti to RTX 5090
Every 16 GB card fits the same models. Paying more buys bandwidth, and bandwidth sets the speed.
Why bandwidth sets the speed
Each token a model writes reads the weights it uses. The RTX 5060 Ti reads its memory at 448 GB/s, the 5070 Ti at 896 and the 5080 at 960 (NVIDIA).
llama.cpp's Apple benchmark thread shows how closely speed follows: a 7B model at 4-bit writes about 14 tokens a second on an M1 (68 GB/s) and about 94 on an M2 Ultra (800 GB/s).
As an Amazon Associate we earn from qualifying purchases.
16 GB graphics card
Biggest: LFM2.5 24B A2B UD-Q4_K_XL, 16.3 GB
- Graphics cardGeForce RTX 5060 Ti 16 GBSame capacity, slower memory16 GB GDDR7, 448 GB/s specs
- Graphics cardGeForce RTX 5070 Ti16 GB GDDR7, 896 GB/s specs
- Graphics cardGeForce RTX 5080Fastest memory of this size here16 GB GDDR7, 960 GB/s specs
All 32 that fit
- LFM2.5 24B A2B UD-Q4_K_XL, 16.3 GB
- DeepSeek R1 Distill Qwen 14B Q4_K_M, 16.2 GB
- Ministral 3 14B 2512 Q4_K_M, 14.4 GB
- gpt-oss-20b Q4_K_M, 13.4 GB
- Gemma 3 12B Q4_K_M, 11.0 GB
- DeepSeek R1 0528 Qwen3 8B Q4_K_M, 10.7 GB
- DeepSeek R1 Distill Llama 8B Q4_K_M, 10.0 GB
- Gemma 4 12B Q4_K_M, 8.9 GB
- GLM-4.6V Flash Q4_K_M, 8.3 GB
- Qwen3 VL 4B Q4_K_M, 8.1 GB
- Qwen3 4B 2507 Q4_K_M, 8.1 GB
- Ornith 1.0 9B Q4_K_M, 7.8 GB
- Qwen3.5 9B Q4_K_M, 7.8 GB
- Phi 4 mini Q4_K_M, 7.7 GB
- DeepSeek R1 Distill Qwen 7B Q4_K_M, 7.4 GB
- Qwen2.5 VL 7B Q4_K_M, 7.4 GB
- Nemotron 3.5 Bolt E4B UD-Q4_K_XL, 6.7 GB
- Gemma 4 E4B Q4_K_M, 6.6 GB
- Llama 3.2 3B Q4_K_M, 6.5 GB
- Ministral 3 3B 2512 Q4_K_M, 6.4 GB
- Gemma 3n E4B Q4_K_M, 5.9 GB
- Qwen3.5 4B Q4_K_M, 4.9 GB
- Gemma 3 4B Q4_K_M, 4.4 GB
- Gemma 3n E2B Q4_K_M, 4.4 GB
- Gemma 4 E2B Q4_K_M, 4.4 GB
- DeepSeek R1 Distill Qwen 1.5B Q4_K_M, 2.9 GB
- Qwen3.5 2B Q4_K_M, 2.7 GB
- Llama 3.2 1B Q4_K_M, 2.6 GB
- Gemma 3 1B Q4_K_M, 2.0 GB
- Qwen3.5 0.8B Q4_K_M, 2.0 GB
- LFM2.5 1.2B Q4_K_M, 1.8 GB
- Gemma 3 270M Q4_K_M, 1.4 GB
24 GB graphics card
Biggest: Qwen3.6 35B A3B UD-Q4_K_M, 23.9 GB
NVIDIA's current GeForce cards jump from 16 GB to 32 GB, so this size is AMD and Intel. llama.cpp runs on both (backends).
- Graphics cardRadeon RX 7900 XTXFastest memory of this size here24 GB GDDR6, 960 GB/s specs
- Graphics cardIntel Arc Pro B60Same capacity, slower memory24 GB GDDR6, 456 GB/s specs
The 14 it adds
- Qwen3.6 35B A3B UD-Q4_K_M, 23.9 GB
- Qwen3.5 35B A3B Q4_K_M, 23.8 GB
- Qwen3 Coder 30B A3B Q4_K_M, 22.6 GB
- Qwen3 30B A3B 2507 Q4_K_M, 22.6 GB
- Gemma 4 31B Q4_K_M, 22.0 GB
- North Mini Code 1.0 UD-Q4_K_M, 21.5 GB
- Gemma 3 27B Q4_K_M, 20.9 GB
- GLM-4.7 Flash Q4_K_M, 20.9 GB
- Devstral Small 2 24B 2512 Q4_K_M, 20.5 GB
- Mistral Small 3.2 24B 2506 Q4_K_M, 20.5 GB
- Qwen3.5 27B Q4_K_M, 20.1 GB
- Qwen3.8 27B UD-Q4_K_M, 19.9 GB
- Gemma 4 26B A4B UD-Q4_K_M, 18.6 GB
- Muse Glimmer 30B UD-Q4_K_XL, 17.3 GB
32 GB graphics card
Biggest: DeepSeek R1 Distill Qwen 32B Q4_K_M, 29.3 GB
- Graphics cardGeForce RTX 5090Fastest memory of this size here32 GB GDDR7, 1,792 GB/s specs
- Graphics cardRadeon AI PRO R9700Same capacity, slower memory32 GB GDDR6, 640 GB/s specs
The 4 it adds
- DeepSeek R1 Distill Qwen 32B Q4_K_M, 29.3 GB
- Nemotron 3 Nano 30B A3B Q4_K_M, 27.1 GB
- Nemotron 3.5 Lightning 30B A3B UD-Q4_K_M, 26.3 GB
- flint UD-Q4_K_XL, 25.9 GB
48 and 96 GB workstation cards: RTX PRO 5000 and RTX PRO 6000
The only single cards past 32 GB. The RTX PRO 6000 pairs the RTX 5090's 1,792 GB/s with three times its memory.
48 GB workstation card
Biggest: Nemotron 3.5 Bolt 74B A9B UD-Q4_K_XL, 50.7 GB
- Workstation graphics cardRTX PRO 5000 Blackwell48 GB GDDR7 ECC, 1,344 GB/s specs
The 1 it adds
- Nemotron 3.5 Bolt 74B A9B UD-Q4_K_XL, 50.7 GB
96 GB workstation card
Biggest: Devstral 2 123B 2512 Q4_K_M, 87.5 GB
- Workstation graphics cardRTX PRO 6000 Blackwell Workstation Edition96 GB GDDR7 ECC, 1,792 GB/s specs
The 11 it adds
- Devstral 2 123B 2512 Q4_K_M, 87.5 GB
- Nemotron 3.5 Super VL 120B A12B UD-Q4_K_XL, 87.4 GB
- Ornith 1.5 122B UD-Q4_K_XL, 81.7 GB
- GLM-4.5 Air Q4_K_M, 80.0 GB
- Mistral Small 4 119B 2603 UD-Q4_K_M, 74.5 GB
- GLM-4.6V Q4_K_M, 71.4 GB
- Llama 4 Scout 17B 16E Q4_K_M, 69.2 GB
- gpt-oss-120b Q4_K_M, 64.9 GB
- Llama 3.3 70B Q4_K_M, 54.0 GB
- DeepSeek R1 Distill Llama 70B Q4_K_M, 54.0 GB
- Kimi Dev 72B Q4_0, 52.9 GB
Mac mini and Mac Studio for local LLMs
Shared memory takes a Mac past any GeForce card. The Mac mini stops at 64 GB; the Mac Studio goes to 512.
GPU share, build-to-order and speed
By default macOS gives the GPU three quarters of memory above 32 GB, so 48 GB of a 64 GB Mac (llama.cpp discussion 2182). sudo sysctl iogpu.wired_limit_mb raises it.
Mac Studio configurations above 96 GB are built to order at Apple: the M5 Max goes to 128 GB and the M5 Ultra to 512 (Apple's specs).
The trade is speed. The 40-core M5 Max reads memory at 614 GB/s and the M5 Ultra at 1.2 TB/s, against 1,792 GB/s on an RTX 5090.
Mac with 64 GB of unified memory
Biggest: Nemotron 3.5 Bolt 74B A9B UD-Q4_K_XL, 50.7 GB
- Desktop MacMac mini (M5 Pro, 64 GB)64 GB unified memory specs, $3,199 at Apple (October 1, 2026)
- Desktop MacMac Studio (M5 Max, 64 GB)Fastest memory of this size here64 GB unified memory, 614 GB/s specs, $3,799 at Apple (October 1, 2026)
All 51 that fit
- Nemotron 3.5 Bolt 74B A9B UD-Q4_K_XL, 50.7 GB
- DeepSeek R1 Distill Qwen 32B Q4_K_M, 29.3 GB
- Nemotron 3 Nano 30B A3B Q4_K_M, 27.1 GB
- Nemotron 3.5 Lightning 30B A3B UD-Q4_K_M, 26.3 GB
- flint UD-Q4_K_XL, 25.9 GB
- Qwen3.6 35B A3B UD-Q4_K_M, 23.9 GB
- Qwen3.5 35B A3B Q4_K_M, 23.8 GB
- Qwen3 Coder 30B A3B Q4_K_M, 22.6 GB
- Qwen3 30B A3B 2507 Q4_K_M, 22.6 GB
- Gemma 4 31B Q4_K_M, 22.0 GB
- North Mini Code 1.0 UD-Q4_K_M, 21.5 GB
- Gemma 3 27B Q4_K_M, 20.9 GB
- GLM-4.7 Flash Q4_K_M, 20.9 GB
- Devstral Small 2 24B 2512 Q4_K_M, 20.5 GB
- Mistral Small 3.2 24B 2506 Q4_K_M, 20.5 GB
- Qwen3.5 27B Q4_K_M, 20.1 GB
- Qwen3.8 27B UD-Q4_K_M, 19.9 GB
- Gemma 4 26B A4B UD-Q4_K_M, 18.6 GB
- Muse Glimmer 30B UD-Q4_K_XL, 17.3 GB
- LFM2.5 24B A2B UD-Q4_K_XL, 16.3 GB
- DeepSeek R1 Distill Qwen 14B Q4_K_M, 16.2 GB
- Ministral 3 14B 2512 Q4_K_M, 14.4 GB
- gpt-oss-20b Q4_K_M, 13.4 GB
- Gemma 3 12B Q4_K_M, 11.0 GB
- DeepSeek R1 0528 Qwen3 8B Q4_K_M, 10.7 GB
- DeepSeek R1 Distill Llama 8B Q4_K_M, 10.0 GB
- Gemma 4 12B Q4_K_M, 8.9 GB
- GLM-4.6V Flash Q4_K_M, 8.3 GB
- Qwen3 VL 4B Q4_K_M, 8.1 GB
- Qwen3 4B 2507 Q4_K_M, 8.1 GB
- Ornith 1.0 9B Q4_K_M, 7.8 GB
- Qwen3.5 9B Q4_K_M, 7.8 GB
- Phi 4 mini Q4_K_M, 7.7 GB
- DeepSeek R1 Distill Qwen 7B Q4_K_M, 7.4 GB
- Qwen2.5 VL 7B Q4_K_M, 7.4 GB
- Nemotron 3.5 Bolt E4B UD-Q4_K_XL, 6.7 GB
- Gemma 4 E4B Q4_K_M, 6.6 GB
- Llama 3.2 3B Q4_K_M, 6.5 GB
- Ministral 3 3B 2512 Q4_K_M, 6.4 GB
- Gemma 3n E4B Q4_K_M, 5.9 GB
- Qwen3.5 4B Q4_K_M, 4.9 GB
- Gemma 3 4B Q4_K_M, 4.4 GB
- Gemma 3n E2B Q4_K_M, 4.4 GB
- Gemma 4 E2B Q4_K_M, 4.4 GB
- DeepSeek R1 Distill Qwen 1.5B Q4_K_M, 2.9 GB
- Qwen3.5 2B Q4_K_M, 2.7 GB
- Llama 3.2 1B Q4_K_M, 2.6 GB
- Gemma 3 1B Q4_K_M, 2.0 GB
- Qwen3.5 0.8B Q4_K_M, 2.0 GB
- LFM2.5 1.2B Q4_K_M, 1.8 GB
- Gemma 3 270M Q4_K_M, 1.4 GB
Mac with 96 GB of unified memory
Biggest: Mistral Small 4 119B 2603 UD-Q4_K_M, 74.5 GB
- Desktop MacMac Studio (M5 Ultra, 96 GB)96 GB unified memory, 1,200 GB/s specs, $5,499 at Apple (October 1, 2026)
The 7 it adds
- Mistral Small 4 119B 2603 UD-Q4_K_M, 74.5 GB
- GLM-4.6V Q4_K_M, 71.4 GB
- Llama 4 Scout 17B 16E Q4_K_M, 69.2 GB
- gpt-oss-120b Q4_K_M, 64.9 GB
- Llama 3.3 70B Q4_K_M, 54.0 GB
- DeepSeek R1 Distill Llama 70B Q4_K_M, 54.0 GB
- Kimi Dev 72B Q4_0, 52.9 GB
Mac with 128 GB of unified memory
Biggest: Devstral 2 123B 2512 Q4_K_M, 87.5 GB
Apple builds the 128 GB Mac Studio to order. The MacBook Pro is the 128 GB Mac sold through Amazon.
- Desktop MacMac Studio (M5 Max, 128 GB)128 GB unified memory, 614 GB/s specs
- LaptopMacBook Pro 14 (M5 Max, 128 GB)128 GB unified memory, 614 GB/s specs
The 4 it adds
- Devstral 2 123B 2512 Q4_K_M, 87.5 GB
- Nemotron 3.5 Super VL 120B A12B UD-Q4_K_XL, 87.4 GB
- Ornith 1.5 122B UD-Q4_K_XL, 81.7 GB
- GLM-4.5 Air Q4_K_M, 80.0 GB
Mac Studio with 256 GB
Biggest: GLM-5.3 Flash UD-Q4_K_XL, 201 GB
- Desktop MacMac Studio (M5 Ultra, 256 GB)256 GB unified memory, 1,200 GB/s specs
The 5 it adds
- GLM-5.3 Flash UD-Q4_K_XL, 201 GB
- Inkling Small UD-Q4_K_M, 165 GB
- DeepSeek V4 Flash 0731 UD-Q4_K_XL, 156 GB
- Step 3.7 Flash UD-Q4_K_M, 125 GB
- Qwen3.8 Flash Next UD-Q4_K_XL, 113 GB
Mac Studio with 512 GB
Biggest: DeepSeek R1 Q4_K_M, 407 GB
- Desktop MacMac Studio (M5 Ultra, 512 GB)512 GB unified memory, 1,200 GB/s specs
The 3 it adds
- DeepSeek R1 Q4_K_M, 407 GB
- MiniMax M3.1 Flash UD-Q4_K_XL, 250 GB
- Llama 4 Maverick 17B 128E Q4_K_M, 247 GB
Best mini PC for local LLMs: Ryzen AI Max+ 395 and DGX Spark
128 GB in a small box. Big models fit, then write slowly, at about a seventh of an RTX 5090's bandwidth.
What AMD and NVIDIA put in them
The Ryzen AI Max+ 395 has 128 GB of LPDDR5X on a 256-bit bus at 256 GB/s, and lets the graphics take up to 96 GB (AMD). GMKtec, Beelink and HP build desktops around it with all 128 GB soldered on.
NVIDIA's DGX Spark does the same on its GB10 chip, 128 GB at 273 GB/s, and ASUS sells its own version, the Ascent GX10.
Ryzen AI Max+ 395 mini PC with 128 GB
Biggest: Devstral 2 123B 2512 Q4_K_M, 87.5 GB
- Mini PCGMKtec EVO-X2128 GB LPDDR5X unified memory, 256 GB/s specs
- Mini PCBeelink GTR9 Pro128 GB LPDDR5X unified memory, 256 GB/s specs
- Mini PCHP Z2 Mini G1a128 GB LPDDR5X ECC unified memory, 256 GB/s specs
All 62 that fit
- Devstral 2 123B 2512 Q4_K_M, 87.5 GB
- Nemotron 3.5 Super VL 120B A12B UD-Q4_K_XL, 87.4 GB
- Ornith 1.5 122B UD-Q4_K_XL, 81.7 GB
- GLM-4.5 Air Q4_K_M, 80.0 GB
- Mistral Small 4 119B 2603 UD-Q4_K_M, 74.5 GB
- GLM-4.6V Q4_K_M, 71.4 GB
- Llama 4 Scout 17B 16E Q4_K_M, 69.2 GB
- gpt-oss-120b Q4_K_M, 64.9 GB
- Llama 3.3 70B Q4_K_M, 54.0 GB
- DeepSeek R1 Distill Llama 70B Q4_K_M, 54.0 GB
- Kimi Dev 72B Q4_0, 52.9 GB
- Nemotron 3.5 Bolt 74B A9B UD-Q4_K_XL, 50.7 GB
- DeepSeek R1 Distill Qwen 32B Q4_K_M, 29.3 GB
- Nemotron 3 Nano 30B A3B Q4_K_M, 27.1 GB
- Nemotron 3.5 Lightning 30B A3B UD-Q4_K_M, 26.3 GB
- flint UD-Q4_K_XL, 25.9 GB
- Qwen3.6 35B A3B UD-Q4_K_M, 23.9 GB
- Qwen3.5 35B A3B Q4_K_M, 23.8 GB
- Qwen3 Coder 30B A3B Q4_K_M, 22.6 GB
- Qwen3 30B A3B 2507 Q4_K_M, 22.6 GB
- Gemma 4 31B Q4_K_M, 22.0 GB
- North Mini Code 1.0 UD-Q4_K_M, 21.5 GB
- Gemma 3 27B Q4_K_M, 20.9 GB
- GLM-4.7 Flash Q4_K_M, 20.9 GB
- Devstral Small 2 24B 2512 Q4_K_M, 20.5 GB
- Mistral Small 3.2 24B 2506 Q4_K_M, 20.5 GB
- Qwen3.5 27B Q4_K_M, 20.1 GB
- Qwen3.8 27B UD-Q4_K_M, 19.9 GB
- Gemma 4 26B A4B UD-Q4_K_M, 18.6 GB
- Muse Glimmer 30B UD-Q4_K_XL, 17.3 GB
- LFM2.5 24B A2B UD-Q4_K_XL, 16.3 GB
- DeepSeek R1 Distill Qwen 14B Q4_K_M, 16.2 GB
- Ministral 3 14B 2512 Q4_K_M, 14.4 GB
- gpt-oss-20b Q4_K_M, 13.4 GB
- Gemma 3 12B Q4_K_M, 11.0 GB
- DeepSeek R1 0528 Qwen3 8B Q4_K_M, 10.7 GB
- DeepSeek R1 Distill Llama 8B Q4_K_M, 10.0 GB
- Gemma 4 12B Q4_K_M, 8.9 GB
- GLM-4.6V Flash Q4_K_M, 8.3 GB
- Qwen3 VL 4B Q4_K_M, 8.1 GB
- Qwen3 4B 2507 Q4_K_M, 8.1 GB
- Ornith 1.0 9B Q4_K_M, 7.8 GB
- Qwen3.5 9B Q4_K_M, 7.8 GB
- Phi 4 mini Q4_K_M, 7.7 GB
- DeepSeek R1 Distill Qwen 7B Q4_K_M, 7.4 GB
- Qwen2.5 VL 7B Q4_K_M, 7.4 GB
- Nemotron 3.5 Bolt E4B UD-Q4_K_XL, 6.7 GB
- Gemma 4 E4B Q4_K_M, 6.6 GB
- Llama 3.2 3B Q4_K_M, 6.5 GB
- Ministral 3 3B 2512 Q4_K_M, 6.4 GB
- Gemma 3n E4B Q4_K_M, 5.9 GB
- Qwen3.5 4B Q4_K_M, 4.9 GB
- Gemma 3 4B Q4_K_M, 4.4 GB
- Gemma 3n E2B Q4_K_M, 4.4 GB
- Gemma 4 E2B Q4_K_M, 4.4 GB
- DeepSeek R1 Distill Qwen 1.5B Q4_K_M, 2.9 GB
- Qwen3.5 2B Q4_K_M, 2.7 GB
- Llama 3.2 1B Q4_K_M, 2.6 GB
- Gemma 3 1B Q4_K_M, 2.0 GB
- Qwen3.5 0.8B Q4_K_M, 2.0 GB
- LFM2.5 1.2B Q4_K_M, 1.8 GB
- Gemma 3 270M Q4_K_M, 1.4 GB
DGX Spark or another GB10 box, 128 GB
Biggest: Step 3.7 Flash UD-Q4_K_M, 125 GB
- Mini PCNVIDIA DGX Spark128 GB LPDDR5X unified memory, 273 GB/s specs
- Mini PCASUS Ascent GX10128 GB LPDDR5X unified memory, 273 GB/s specs
All 64 that fit
- Step 3.7 Flash UD-Q4_K_M, 125 GB
- Qwen3.8 Flash Next UD-Q4_K_XL, 113 GB
- Devstral 2 123B 2512 Q4_K_M, 87.5 GB
- Nemotron 3.5 Super VL 120B A12B UD-Q4_K_XL, 87.4 GB
- Ornith 1.5 122B UD-Q4_K_XL, 81.7 GB
- GLM-4.5 Air Q4_K_M, 80.0 GB
- Mistral Small 4 119B 2603 UD-Q4_K_M, 74.5 GB
- GLM-4.6V Q4_K_M, 71.4 GB
- Llama 4 Scout 17B 16E Q4_K_M, 69.2 GB
- gpt-oss-120b Q4_K_M, 64.9 GB
- Llama 3.3 70B Q4_K_M, 54.0 GB
- DeepSeek R1 Distill Llama 70B Q4_K_M, 54.0 GB
- Kimi Dev 72B Q4_0, 52.9 GB
- Nemotron 3.5 Bolt 74B A9B UD-Q4_K_XL, 50.7 GB
- DeepSeek R1 Distill Qwen 32B Q4_K_M, 29.3 GB
- Nemotron 3 Nano 30B A3B Q4_K_M, 27.1 GB
- Nemotron 3.5 Lightning 30B A3B UD-Q4_K_M, 26.3 GB
- flint UD-Q4_K_XL, 25.9 GB
- Qwen3.6 35B A3B UD-Q4_K_M, 23.9 GB
- Qwen3.5 35B A3B Q4_K_M, 23.8 GB
- Qwen3 Coder 30B A3B Q4_K_M, 22.6 GB
- Qwen3 30B A3B 2507 Q4_K_M, 22.6 GB
- Gemma 4 31B Q4_K_M, 22.0 GB
- North Mini Code 1.0 UD-Q4_K_M, 21.5 GB
- Gemma 3 27B Q4_K_M, 20.9 GB
- GLM-4.7 Flash Q4_K_M, 20.9 GB
- Devstral Small 2 24B 2512 Q4_K_M, 20.5 GB
- Mistral Small 3.2 24B 2506 Q4_K_M, 20.5 GB
- Qwen3.5 27B Q4_K_M, 20.1 GB
- Qwen3.8 27B UD-Q4_K_M, 19.9 GB
- Gemma 4 26B A4B UD-Q4_K_M, 18.6 GB
- Muse Glimmer 30B UD-Q4_K_XL, 17.3 GB
- LFM2.5 24B A2B UD-Q4_K_XL, 16.3 GB
- DeepSeek R1 Distill Qwen 14B Q4_K_M, 16.2 GB
- Ministral 3 14B 2512 Q4_K_M, 14.4 GB
- gpt-oss-20b Q4_K_M, 13.4 GB
- Gemma 3 12B Q4_K_M, 11.0 GB
- DeepSeek R1 0528 Qwen3 8B Q4_K_M, 10.7 GB
- DeepSeek R1 Distill Llama 8B Q4_K_M, 10.0 GB
- Gemma 4 12B Q4_K_M, 8.9 GB
- GLM-4.6V Flash Q4_K_M, 8.3 GB
- Qwen3 VL 4B Q4_K_M, 8.1 GB
- Qwen3 4B 2507 Q4_K_M, 8.1 GB
- Ornith 1.0 9B Q4_K_M, 7.8 GB
- Qwen3.5 9B Q4_K_M, 7.8 GB
- Phi 4 mini Q4_K_M, 7.7 GB
- DeepSeek R1 Distill Qwen 7B Q4_K_M, 7.4 GB
- Qwen2.5 VL 7B Q4_K_M, 7.4 GB
- Nemotron 3.5 Bolt E4B UD-Q4_K_XL, 6.7 GB
- Gemma 4 E4B Q4_K_M, 6.6 GB
- Llama 3.2 3B Q4_K_M, 6.5 GB
- Ministral 3 3B 2512 Q4_K_M, 6.4 GB
- Gemma 3n E4B Q4_K_M, 5.9 GB
- Qwen3.5 4B Q4_K_M, 4.9 GB
- Gemma 3 4B Q4_K_M, 4.4 GB
- Gemma 3n E2B Q4_K_M, 4.4 GB
- Gemma 4 E2B Q4_K_M, 4.4 GB
- DeepSeek R1 Distill Qwen 1.5B Q4_K_M, 2.9 GB
- Qwen3.5 2B Q4_K_M, 2.7 GB
- Llama 3.2 1B Q4_K_M, 2.6 GB
- Gemma 3 1B Q4_K_M, 2.0 GB
- Qwen3.5 0.8B Q4_K_M, 2.0 GB
- LFM2.5 1.2B Q4_K_M, 1.8 GB
- Gemma 3 270M Q4_K_M, 1.4 GB
Ollama hardware requirements
Ollama sets its default context from your VRAM, so a small card gets a short one.
| Card | Default context | Fit at 4-bit | Biggest |
|---|---|---|---|
| 16 GB card | 4K | 35 of 78 | Muse Glimmer 30B |
| 24 GB card | 32K | 46 of 78 | Qwen3.6 35B A3B |
| 32 GB card | 32K | 50 of 78 | DeepSeek R1 Distill Qwen 32B |
| 48 GB card | 256K | 48 of 78 | Nemotron 3.5 Bolt 74B A9B |
| 96 GB card | 256K | 61 of 78 | GLM-4.5 Air |
A 48 GB card fits fewer than a 32 GB card here, 48 against 50: its default context is 256K, and the cache eats the difference until you set OLLAMA_CONTEXT_LENGTH lower.
Ollama's files, its two docs pages, and what happens when a model is too big
A plain tag like gemma3:12b is the 4-bit Q4_K_M build: both are 8.1 GB on Ollama's tags page.
The defaults above are from Ollama's context length page. Its FAQ still says 4,096 tokens for everyone, so the two disagree.
A model too big for the card still loads. Ollama splits the layers between GPU and system memory, and ollama ps shows the split. The cache is f16 unless you set OLLAMA_KV_CACHE_TYPE; q8_0 takes about half the memory.
How the numbers are worked out
How the numbers are worked out
Total memory is three parts added together. Weights are the exact size of the GGUF file on Hugging Face, all parts summed when a quant is split. The context cache (the KV cache) comes from each model's own config.json:
cache = layers × kv_heads × head_dim × 2 (K and V) × 2 bytes × tokens
Sliding-window layers stop at their window plus llama.cpp's 512-token batch. Linear-attention and Mamba layers keep a fixed state instead, which we add once. Latent-attention models (DeepSeek, Kimi, GLM-5) cache kv_lora_rank + rope dimensions per layer, the way llama.cpp does. Overhead is our allowance: 0.5 GB for the GPU context plus the output buffer, 512 × vocabulary × 4 bytes.
A "24 GB" card holds 24 GiB, which is 25.8 GB in the decimal units file sizes use, and we count it that way. Macs get the share Metal allows by default, two thirds of unified memory up to 32 GB and three quarters above (llama.cpp discussion 2182); CPU-only machines and the DGX Spark keep 4 GB back for the operating system. All of it assumes llama.cpp's defaults: an f16 cache and one conversation at a time. Quantizing the cache to q8_0 halves it.
Files are Unsloth's GGUF uploads, the most downloaded quantizations on Hugging Face, and the list refreshes daily from Hugging Face's API. A model joins when its repo passes 20,000 downloads in 30 days. Last checked November 15, 2026.