Skip to content
Docs

Tensor Parallelism

Tensor Parallel Size is how many GPUs one model is split across: 1, 2, 4 or 8.

What it buys, and what it costs

Section titled “What it buys, and what it costs”

Buys: a model that does not fit on one card can run, and aggregate memory bandwidth rises — which matters because decode is bandwidth-bound.

Costs: the GPUs must exchange activations on every layer. Two GPUs are not twice as fast, and on a small model the communication can cost more than it saves.

The rule of thumb: raise it because the model does not fit, not because you hope for speed.

Auto

Section titled “Auto”

Auto picks the smallest power of two that fits comfortably, preferring 1. It reads live GPU inventory, so it is accurate for local servers.

Two limits, both from not being able to see the target machine:

  • Remote nodes fall back to the largest power of two that the node’s GPU count allows, because the control plane cannot introspect a remote box at create time.
  • Cluster deploys apply the same per member.

The shared-memory prerequisite

Section titled “The shared-memory prerequisite”

Above 1, the GPUs communicate through /dev/shm, and the container’s default is often too small. Lifeboat checks before launching and refuses with insufficient_shared_memory and a fix hint, rather than letting it surface as an unexplained “NCCL unhandled system error” deep in the engine.

The fix is LIFEBOAT_SHM_SIZE (or shmSize in Helm) — but it is fixed at container creation, so the container must be recreated, not restarted. Because it cannot be fixed at runtime, automatic restarts are disabled for this failure.

The node agent runs the same check, so a remote launch on an under-provisioned agent returns the same structured error rather than a generic failure.

Where it does not apply

Section titled “Where it does not apply”
  • The GGUF engine ignores it. It splits layers across whatever GPUs it can see, by layer count rather than by tensor.
  • Single-GPU hosts obviously, and the setting is inert there.

Interaction with sizing

Section titled “Interaction with sizing”

Tensor parallelism changes the per-GPU footprint, so it is part of the key Lifeboat stores measured footprints under. Changing it means the old measurement does not apply and a fresh one is taken.

Sources and references

Section titled “Sources and references”