Error messages

must be divisible by tensor parallel size

must be divisible by tensor parallel size

You start vLLM with --tensor-parallel-size set to however many GPUs you have, and it stops straight away:

ValueError: Total number of attention heads (40) must be divisible by tensor parallel size (6).

That exact line is from issue #4232, someone putting Qwen1.5-32B on six T4 cards. The numbers change, the sentence doesn't. It's a plain arithmetic check in vLLM 0.30.0's model config, made before a single weight is loaded, and it has come up in 16 issues as of September 26, 2026.

Why the GPU count has to divide the heads

Tensor parallelism cuts every layer of the model into equal slices, one per GPU. The natural place to cut the attention part of a layer is between its heads (a model's attention is made of parallel units called heads, 32 or 40 or 64 of them per layer in common models). A 40-head model splits into 2, 4, 5, 8, 10 or 20 equal parts. It can't be split into 6.

That's why three- and six-GPU machines hit this so often. A vLLM contributor put it this way in "3 gpu's not supported?": tensor parallel needs a GPU count that fits the model's sizes, "because hidden_size/head_num, etc. are usually even numbers, like 4096/32, 5120/40". You can find your model's head count as num_attention_heads in its config.json.

Option one: use fewer GPUs

The quickest fix is a tensor-parallel size that divides the head count. In #4232 the first answer from a vLLM team member was to use 8, and when the user said he only had six, a maintainer replied: "Doing tp=4 is the most effective fix." The other two cards sit idle (or run a second copy of a smaller model), and the model has to fit in four cards' worth of memory.

Option two: pipeline parallelism across all of them

Pipeline parallelism splits the model the other way, handing each GPU a block of whole layers. Layers don't need to divide evenly, so any GPU count works. vLLM's parallelism guide has a note for exactly this case: "If the model fits within a single node but the GPU count doesn't evenly divide the model size, enable pipeline parallelism". It says to set tensor parallel to 1 and pipeline parallel to the number of GPUs:

vllm serve Qwen/Qwen1.5-32B-Chat \
    --tensor-parallel-size 1 \
    --pipeline-parallel-size 6

(The model name there is only the one from #4232; the flags are what matter.) The same guide says pipeline parallelism is also the better choice when the GPUs aren't linked by NVLink, Nvidia's fast card-to-card connection, as with L40S cards.

You can also mix the two, as long as the tensor part divides the heads. The guide's example runs eight GPUs as --tensor-parallel-size 4 --pipeline-parallel-size 2, and on six GPUs a 40-head model would work as 2 by 3.

One thing can still stop you. vLLM 0.30.0 checks right afterwards whether the model supports pipeline parallelism at all, and if it doesn't, you get "Pipeline parallelism is not supported for this model." Then you're back to option one.