ThinkFacility Sign in
  1. Can I run it locally?
  2. Best GPU for local LLMs

Best GPU for Local LLMs: What Each VRAM Tier Runs

Memory decides what runs. Here's every card, Mac and mini PC on sale now against the open models we track, from the same file sizes as our VRAM calculator.

How much VRAM do I need?

Each bar counts the models that fit whole at their common 4-bit file with a 32K context.

What a 4-bit model actually needs

Qwen3.8 27B is a fair yardstick (6.4 million downloads in 30 days). Its 4-bit file is 16.6 GB, and with a 32K context and llama.cpp's buffers it needs 19.9 GB. At 64K it's 22.0 GB.

The cache, the keys and values a model keeps for every token it has read, is the part that varies. On Llama 3.3 70B the full 128K costs 42.9 GB by itself.

We count memory the way each machine hands it to the GPU: all of a card's VRAM, two thirds of a Mac's unified memory up to 32 GB and three quarters above, and the 96 GB of 128 that AMD lets the Ryzen AI Max+ 395 give its graphics.

16, 24 and 32 GB graphics cards: RTX 5060 Ti to RTX 5090

Every 16 GB card fits the same models. Paying more buys bandwidth, and bandwidth sets the speed.

Why bandwidth sets the speed

Each token a model writes reads the weights it uses. The RTX 5060 Ti reads its memory at 448 GB/s, the 5070 Ti at 896 and the 5080 at 960 (NVIDIA).

llama.cpp's Apple benchmark thread shows how closely speed follows: a 7B model at 4-bit writes about 14 tokens a second on an M1 (68 GB/s) and about 94 on an M2 Ultra (800 GB/s).

As an Amazon Associate we earn from qualifying purchases.

16 GB graphics card

32 of 78 fit

Biggest: LFM2.5 24B A2B UD-Q4_K_XL, 16.3 GB

All 32 that fit

24 GB graphics card

46 of 78 fit+14 over a 16 GB card

Biggest: Qwen3.6 35B A3B UD-Q4_K_M, 23.9 GB

NVIDIA's current GeForce cards jump from 16 GB to 32 GB, so this size is AMD and Intel. llama.cpp runs on both (backends).

  • Graphics card
    Radeon RX 7900 XTXFastest memory of this size here24 GB GDDR6, 960 GB/s specs
  • Graphics card
    Intel Arc Pro B60Same capacity, slower memory24 GB GDDR6, 456 GB/s specs
The 14 it adds

48 and 96 GB workstation cards: RTX PRO 5000 and RTX PRO 6000

The only single cards past 32 GB. The RTX PRO 6000 pairs the RTX 5090's 1,792 GB/s with three times its memory.

96 GB workstation card

62 of 78 fit+11 over a 48 GB card

Biggest: Devstral 2 123B 2512 Q4_K_M, 87.5 GB

  • Workstation graphics card
    RTX PRO 6000 Blackwell Workstation Edition96 GB GDDR7 ECC, 1,792 GB/s specs
The 11 it adds

Mac mini and Mac Studio for local LLMs

Shared memory takes a Mac past any GeForce card. The Mac mini stops at 64 GB; the Mac Studio goes to 512.

GPU share, build-to-order and speed

By default macOS gives the GPU three quarters of memory above 32 GB, so 48 GB of a 64 GB Mac (llama.cpp discussion 2182). sudo sysctl iogpu.wired_limit_mb raises it.

Mac Studio configurations above 96 GB are built to order at Apple: the M5 Max goes to 128 GB and the M5 Ultra to 512 (Apple's specs).

The trade is speed. The 40-core M5 Max reads memory at 614 GB/s and the M5 Ultra at 1.2 TB/s, against 1,792 GB/s on an RTX 5090.

Mac with 64 GB of unified memory

51 of 78 fit

Biggest: Nemotron 3.5 Bolt 74B A9B UD-Q4_K_XL, 50.7 GB

  • Desktop Mac
    Mac mini (M5 Pro, 64 GB)64 GB unified memory specs, $3,199 at Apple (October 1, 2026)
  • Desktop Mac
    Mac Studio (M5 Max, 64 GB)Fastest memory of this size here64 GB unified memory, 614 GB/s specs, $3,799 at Apple (October 1, 2026)
All 51 that fit

Mac with 128 GB of unified memory

62 of 78 fit+4 over a 96 GB Mac

Biggest: Devstral 2 123B 2512 Q4_K_M, 87.5 GB

Apple builds the 128 GB Mac Studio to order. The MacBook Pro is the 128 GB Mac sold through Amazon.

  • Desktop Mac
    Mac Studio (M5 Max, 128 GB)128 GB unified memory, 614 GB/s specs
  • Laptop
    MacBook Pro 14 (M5 Max, 128 GB)128 GB unified memory, 614 GB/s specs
The 4 it adds

Best mini PC for local LLMs: Ryzen AI Max+ 395 and DGX Spark

128 GB in a small box. Big models fit, then write slowly, at about a seventh of an RTX 5090's bandwidth.

What AMD and NVIDIA put in them

The Ryzen AI Max+ 395 has 128 GB of LPDDR5X on a 256-bit bus at 256 GB/s, and lets the graphics take up to 96 GB (AMD). GMKtec, Beelink and HP build desktops around it with all 128 GB soldered on.

NVIDIA's DGX Spark does the same on its GB10 chip, 128 GB at 273 GB/s, and ASUS sells its own version, the Ascent GX10.

Ryzen AI Max+ 395 mini PC with 128 GB

62 of 78 fit

Biggest: Devstral 2 123B 2512 Q4_K_M, 87.5 GB

  • Mini PC
    GMKtec EVO-X2128 GB LPDDR5X unified memory, 256 GB/s specs
  • Mini PC
    Beelink GTR9 Pro128 GB LPDDR5X unified memory, 256 GB/s specs
  • Mini PC
    HP Z2 Mini G1a128 GB LPDDR5X ECC unified memory, 256 GB/s specs
All 62 that fit

DGX Spark or another GB10 box, 128 GB

64 of 78 fit

Biggest: Step 3.7 Flash UD-Q4_K_M, 125 GB

  • Mini PC
    NVIDIA DGX Spark128 GB LPDDR5X unified memory, 273 GB/s specs
  • Mini PC
    ASUS Ascent GX10128 GB LPDDR5X unified memory, 273 GB/s specs
All 64 that fit

Ollama hardware requirements

Ollama sets its default context from your VRAM, so a small card gets a short one.

4KUnder 24 GiB of VRAM
32K24 to 48 GiB of VRAM
256K48 GiB and up of VRAM
CardDefault contextFit at 4-bitBiggest
16 GB card4K35 of 78Muse Glimmer 30B
24 GB card32K46 of 78Qwen3.6 35B A3B
32 GB card32K50 of 78DeepSeek R1 Distill Qwen 32B
48 GB card256K48 of 78Nemotron 3.5 Bolt 74B A9B
96 GB card256K61 of 78GLM-4.5 Air

A 48 GB card fits fewer than a 32 GB card here, 48 against 50: its default context is 256K, and the cache eats the difference until you set OLLAMA_CONTEXT_LENGTH lower.

Ollama's files, its two docs pages, and what happens when a model is too big

A plain tag like gemma3:12b is the 4-bit Q4_K_M build: both are 8.1 GB on Ollama's tags page.

The defaults above are from Ollama's context length page. Its FAQ still says 4,096 tokens for everyone, so the two disagree.

A model too big for the card still loads. Ollama splits the layers between GPU and system memory, and ollama ps shows the split. The cache is f16 unless you set OLLAMA_KV_CACHE_TYPE; q8_0 takes about half the memory.

How the numbers are worked out

How the numbers are worked out

Total memory is three parts added together. Weights are the exact size of the GGUF file on Hugging Face, all parts summed when a quant is split. The context cache (the KV cache) comes from each model's own config.json:

cache = layers × kv_heads × head_dim × 2 (K and V) × 2 bytes × tokens

Sliding-window layers stop at their window plus llama.cpp's 512-token batch. Linear-attention and Mamba layers keep a fixed state instead, which we add once. Latent-attention models (DeepSeek, Kimi, GLM-5) cache kv_lora_rank + rope dimensions per layer, the way llama.cpp does. Overhead is our allowance: 0.5 GB for the GPU context plus the output buffer, 512 × vocabulary × 4 bytes.

A "24 GB" card holds 24 GiB, which is 25.8 GB in the decimal units file sizes use, and we count it that way. Macs get the share Metal allows by default, two thirds of unified memory up to 32 GB and three quarters above (llama.cpp discussion 2182); CPU-only machines and the DGX Spark keep 4 GB back for the operating system. All of it assumes llama.cpp's defaults: an f16 cache and one conversation at a time. Quantizing the cache to q8_0 halves it.

Files are Unsloth's GGUF uploads, the most downloaded quantizations on Hugging Face, and the list refreshes daily from Hugging Face's API. A model joins when its repo passes 20,000 downloads in 30 days. Last checked November 15, 2026.