Skip to content
Docs

Servers & Multi-GPU

An inference server in Lifeboat is one model running on one or more GPUs. Servers are created, started, stopped, tested, and inspected entirely from the control plane, and the platform supervises them so a crashed process does not become an outage.

What is the server lifecycle?

Section titled “What is the server lifecycle?”

A server moves through explicit states — created, starting, running, stopping, stopped, and error — and the dashboard shows each server’s current state, model, GPU assignment, and uptime. Available operations include:

  • Create with a model, GPU selection, quantization choice, and optimization settings.
  • Start / stop / force-stop individually or in bulk across the fleet.
  • Test with a live request to confirm the server actually answers before traffic is sent to it.
  • Inspect logs — the most recent output from the model process is readable in the UI, so launch failures and inference errors can be diagnosed without shelling into a container.
  • Edit configuration and restart.

What happens if a server crashes?

Section titled “What happens if a server crashes?”

Lifeboat restarts it automatically, up to a configurable number of attempts with a configurable cooldown between them, so a transient fault self-heals. The platform also performs orphan recovery: if the control plane restarts while model processes are still running, it reconciles what is actually on the machine against the registry rather than leaving stranded processes holding GPU memory.

How does multi-GPU serving work?

Section titled “How does multi-GPU serving work?”

Lifeboat supports tensor parallelism across 1, 2, 4, or 8 GPUs, so models too large for a single card are split across several. When creating a server you select which physical GPUs to use, and Lifeboat is aware of the interconnect topology between them — relevant because tensor parallelism is sensitive to how the selected GPUs are connected to each other.

Multi-GPU composes with quantization: a large model can be split across cards and run in a low-precision weight format, which is often the difference between “does not fit” and “fits with headroom.”

Will a model fit before I try to start it?

Section titled “Will a model fit before I try to start it?”

Lifeboat includes a fit check that evaluates a proposed model, quantization, and GPU selection against available VRAM before deployment, so a doomed configuration is reported up front instead of failing minutes into a model load. Model inspection separately reports architecture, parameter-count estimate, expected precision, and a VRAM headroom assessment.

Can the same model run on more than one node?

Section titled “Can the same model run on more than one node?”

Yes. The same model can be deployed to multiple nodes in a fleet and served behind Lifeboat’s load balancer as one logical endpoint — the standard pattern for scaling throughput horizontally and for surviving the loss of a single machine. See Clusters and load balancing.

Can Lifeboat manage inference engines it did not start?

Section titled “Can Lifeboat manage inference engines it did not start?”

Yes. External inference endpoints can be brought under Lifeboat’s management and routing, so an existing engine already running in the environment can be consolidated behind the same control plane and API surface rather than being migrated first.

Sources and references

Section titled “Sources and references”