Skip to content
Docs

Engines

Lifeboat manages more than one inference engine and picks between them. You can always override the choice; the default is usually right.

The three paths

Section titled “The three paths”
Engine Serves Where
Tensor engine Safetensors — full precision, FP8, FP4 NVIDIA and AMD GPUs
GGUF engine Quantized GGUF Everywhere, including CPU-only and integrated GPUs
MLX MLX-format models Apple Silicon

What only the tensor engine has

Section titled “What only the tensor engine has”

The GPU optimization layer — cache compression, adaptive cache management, fair scheduling, memory tiering, admission control, dynamic precision — and speculative decoding. Both hook that engine’s scheduler, so they do not apply elsewhere regardless of hardware.

That is not a statement about the other engines being lesser. On a machine with no GPU there was never a tensor engine to hook.

What only the GGUF engine has

Section titled “What only the GGUF engine has”

Models the tensor engine cannot load at all: quantization formats and model architectures it has no implementation for, GGUF vision models with a separate projector, and GGUF embeddings. It also runs on hardware the tensor engine does not support at all.

How Lifeboat chooses

Section titled “How Lifeboat chooses”

Automatically, at two points:

  • At create time, from the model’s format and the host’s capability.
  • At start time, re-checked — because weights that were not downloaded when the server was created are local by then, and the header can finally be read. A switch is persisted and audited as server_backend_autoswitched.

An explicit choice always wins. If you pick an engine, Lifeboat uses it and reports the failure if it cannot.

Differences you will notice

Section titled “Differences you will notice”
Tensor GGUF
Optimization layer, speculative decoding yes no
Tensor parallelism yes ignored — splits by layer
Memory fraction yes not applicable — offloads by layer count
Concurrency model continuous batching fixed slots (keep max concurrency ≤ 16)
/v1/score yes returns 404, and Lifeboat fails over

The concurrency difference is the one that bites: above the slot count, surplus requests queue inside the GGUF engine where Lifeboat cannot see or report them, so the console shows no wait while requests are waiting.

Which am I on?

Section titled “Which am I on?”

The server row names the backend, and the audit log records any automatic switch. Process names differ too — the GGUF engine runs as its own binary rather than under the Lifeboat process name, which is cosmetic only; stop and restart are tag-based and work either way.

Sources and references

Section titled “Sources and references”