Skip to content
Docs

Clusters & Load Balancing

Beyond a single machine, Lifeboat manages a fleet: multiple GPU nodes registered to one control plane, organized into clusters, with inference traffic distributed across them by a load balancer that understands how expensive each request actually is.

How is a multi-node fleet organized?

Section titled “How is a multi-node fleet organized?”

Two distinct concepts:

  • Fleet nodes — the physical machines. Each registers with the control plane, reports its GPUs and their live utilization, and accepts server-lifecycle commands. A fleet-wide GPU table shows every card across every node in one view.
  • Named clusters — logical pools that carve fleet GPUs into business-aligned groups, for example interactive, batch, or a per-tenant pool. Clusters are how a platform team gives one workload guaranteed capacity without giving it the whole fleet.

Can I deploy a model to a whole cluster at once?

Section titled “Can I deploy a model to a whole cluster at once?”

Yes. A single deploy action pushes the same model configuration to every selected node in a named cluster, rather than repeating server creation node by node. Combined with the pre-deployment fit check, this makes rolling a new model out across a fleet a one-step operation.

How does Lifeboat route inference requests?

Section titled “How does Lifeboat route inference requests?”

Through a token-aware weighted round-robin load balancer in front of the OpenAI-compatible API. Unlike a simple request-count balancer, it accounts for the size of each request, because a 200-token completion and a 200,000-token context are not equivalent load. Its behavior:

  • Pools are formed per model, so a request for a given model only reaches servers actually serving it.
  • Per-server weights let a bigger or faster node take a proportionally larger share.
  • Capacity gating rejects or defers work that a server cannot accept, instead of accepting it and degrading everything already in flight.
  • A waiter queue holds requests briefly during a burst rather than failing them immediately, with a configurable depth and timeout.
  • Idle pool eviction reclaims resources from pools that have gone quiet.

Can I see whether traffic is actually balanced?

Section titled “Can I see whether traffic is actually balanced?”

Yes, and this is deliberately visible rather than inferred. Every proxied request returns response headers identifying how it was dispatched, the control plane exposes a live snapshot of the balancer’s internal state, and the dashboard shows each server’s actual traffic share against its configured share — with a drift indicator that flags when the two diverge under steady load. Prometheus metrics cover dispatch counts, rejections by reason, in-flight requests, queue depth, and queue wait times.

What happens when the fleet is saturated?

Section titled “What happens when the fleet is saturated?”

Requests are rejected or queued with a specific, distinguishable reason — queue full, queue timeout, no available backend, request too large, or capacity exceeded — rather than a generic error. That distinction is the difference between guessing and knowing during an incident: it tells you whether to add capacity, raise a queue limit, or fix an oversized caller. See Troubleshooting and FAQ.

How is fleet-to-control-plane traffic secured?

Section titled “How is fleet-to-control-plane traffic secured?”

Every control-plane-to-node call is authenticated with a shared cluster secret, so a node will not accept lifecycle commands from an unauthenticated source on the network. See Security and compliance.

Sources and references

Section titled “Sources and references”