Error messages

torch.OutOfMemoryError: CUDA out of memory. Tried to allocate

torch.OutOfMemoryError: CUDA out of memory. Tried to allocate

This is PyTorch's out-of-memory error, and in vLLM it tends to show up after startup looked fine. From a September 2026 report (issue #57475) on a 24 GB laptop RTX 5090:

INFO  [gpu_worker.py:642]  Available KV cache memory: 2.97 GiB
...
ERROR [engine/core.py:1378] torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 136.00 MiB.
      GPU 0 has a total capacity of 23.46 GiB of which 120.88 MiB is free ...

As of September 25, 2026, 339 issues and pull requests in the vllm-project/vllm repo quote "CUDA out of memory. Tried to allocate", 41 of them from the last 90 days. If your log says "Allocation on device 0 would exceed allowed memory" instead, that's the same failure from PyTorch's other allocator, and we cover it on the ComfyUI page.

Why vLLM runs out after its own check passed

Most programs grab memory as they go. vLLM doesn't: we read the startup code in 0.30.0, and it takes 92% of the card by default (--gpu-memory-utilization 0.92), measures what the model uses in a test pass, then hands everything left over to the KV cache (where it keeps each request's context) in one allocation. Anything the test pass missed has to fit in the small slice outside that budget. When it doesn't, you get this error.

That makes the usual advice for vLLM's other memory errors backwards here. Raising the setting gives the cache more and leaves less for everything else. In issue #57475, 0.98 crashed and 0.97 booted, same model, same card. Someone replying there said they'd never got vLLM working above 0.95 or 0.96.

Which stage failed

While loading weights. The model's too big for the card. Use a quantized version, or split it with --tensor-parallel-size.

During CUDA graph capture, after the "Available KV cache memory" line. CUDA graphs are recorded GPU call sequences that make decoding faster, and vLLM captures them after it has already allocated the cache. Since v0.21.0 vLLM estimates their size up front, but #57475 shows the estimate can come in short. A pull request to measure the whole capture (#51590) was still open on September 25. Lower --gpu-memory-utilization by a few hundredths, or use --enforce-eager, which skips capture at some cost in decode speed, as a vLLM maintainer suggested in issue #17618.

Mid-serving, after requests have worked. This is the one that surprised us most. Some kernels allocate working memory the startup test pass never exercises at full size. In issue #54775 (open as of September 25, 2026), a GLM model on an AMD MI350X served fine with --max-num-batched-tokens up to 65,536 and died mid-request at 81,920, trying to allocate 1.25 GiB. A vLLM contributor reported the same failure on a Qwen model with Nvidia L40S cards. The fix there was a smaller --max-num-batched-tokens (the most tokens vLLM processes in one step), and a lower utilization also leaves more room.

Other settings that cut memory

vLLM's conserving memory guide lists the rest: a shorter --max-model-len, a lower --max-num-seqs (how many requests run at once), fewer CUDA graph sizes, and for multimodal models a lower --limit-mm-per-prompt.