Error messages

CUDA error: out of memory

CUDA error: out of memory

When llama.cpp can't get video memory in the middle of a run, it prints three lines and aborts. This is from a real report (#28338):

CUDA error: out of memory
  current device: 0, in function alloc at .../ggml/src/ggml-cuda/ggml-cuda.cu:591
  cuMemCreate(&handle, reserve_size, &prop, 0)

We read the CUDA backend in release v0.5.0 (September 23, 2026). The first line is the driver's own error text with "CUDA error:" in front, the second says which GPU and which function asked, and the third is the exact call that failed. Nothing after it runs. The function that prints it is marked as never returning, so the server goes down with it.

The two ways it shows up

Most people meet the load-time version first, and it reads differently. llama.cpp tries to allocate a buffer, logs the size, and gives up on loading without crashing:

ggml_backend_cuda_buffer_type_alloc_buffer: allocating 1040.28 MiB on device 0: cudaMalloc failed: out of memory
graph_reserve: failed to allocate compute buffers
llama_init_from_model: failed to initialize the context: failed to allocate compute pp buffers

If the weights themselves don't fit, the next line says unable to allocate CUDA0 buffer instead. Both mean the same thing as the crash. The difference is timing: at load, llama.cpp can back out cleanly, but once it's generating, a failed request for temporary space (it keeps a pool of it and retries once after emptying the pool) has nowhere to go.

Why it runs out

VRAM has to hold three things at once: the model's layers you've put on the GPU, the KV cache (the stored attention state for every token of context, which grows with -c), and compute buffers sized by the batch. A long context can cost more than the weights. Another program holding VRAM, a desktop compositor or a second llama.cpp, counts against you too.

Recent builds try to prevent this. The --fit option is on by default in v0.5.0 and adjusts settings you left unset so the model fits in free memory, aiming to leave 1024 MiB spare per GPU. It only touches unset arguments, though. If you pass -ngl 99 or a big -c yourself, it respects your numbers and you can still run out.

What to do

• Drop the explicit -ngl and -c and let --fit pick, or lower them yourself. Every layer left on the CPU frees VRAM at the cost of speed.
• Quantize the KV cache: -ctk q8_0 -ctv q8_0 roughly halves it compared with the default, which is f16 (16 bits per value).
• Lower -ub (the physical batch size) to shrink the compute buffers.
• If it crashes after running a while, raise the spare margin with --fit-target (in MiB) so there's room for temporary allocations.

On Linux there's also GGML_CUDA_ENABLE_UNIFIED_MEMORY=1. The build docs say it lets llama.cpp spill into system RAM "instead of crashing when the GPU VRAM is exhausted", and that Windows has the same thing as System Memory Fallback in the NVIDIA control panel. Expect it to be much slower once it spills.

When it's a bug

Sometimes the maths says it should fit and it doesn't. As of September 25, 2026, #27282 is open about MTP (the model's built-in draft head for faster generation) reserving a second compute buffer and running out on a 24 GB card. On the mid-run crash in #28338, contributor JohannesGaessler wrote on September 5 that "recoverable OOMs are generally tricky" and that he'd rather cut temporary allocations and reserve memory up front. So for now, a crash mid-generation isn't something llama.cpp recovers from. Leaving headroom is the fix you control.