Glossary
Terms as Lifeboat uses them, since several are used differently elsewhere.
Alias — the name a model is advertised under by the API. Set it; it keeps filesystem paths out of /v1/models and decouples clients from the checkpoint.
Attestation — cryptographic evidence that a host is running inside a genuine trusted execution environment. Required before an encrypted model’s key is released.
Cluster — a named pool of GPUs across one or more nodes, aligned to a workload rather than to hardware.
Context window — the number of tokens one request may use. Not the same as the cache size, which is shared across concurrent requests and is usually larger.
Decode — generating tokens one at a time after the prompt is processed. Memory-bandwidth bound, which is why bytes-per-token sets the speed ceiling.
Drain — setting a server’s weight to 0 so it stops receiving new traffic while finishing what it has. The graceful way to remove a backend.
Draft model — a small model that proposes tokens for a larger one to verify, in speculative decoding. Algorithm-specific.
GGUF — a quantized model file format, served by the GGUF engine. Runs anywhere, including on CPUs.
Grace period — the 24 hours after first boot during which an unlicensed deployment serves normally. Distinct from the free tier and the paid trial.
KV cache — the key-value cache, holding attention state for in-flight requests. Its size is what limits concurrency, and compressing it is what Lifeboat’s optimization layer mostly does.
Memory fraction — the fraction of free GPU memory a server may use. Not a fraction of the card.
MLX — Apple’s array framework; the fast path for models on Apple Silicon.
Node — a machine running a Lifeboat agent, managed by the control plane.
Pool — the set of backends a request can be routed among, keyed by model, cluster and routing tier.
Prefill — processing the prompt before generation starts. Compute-bound, and what dominates time-to-first-token.
Prefix cache — reuse of already-computed attention state for a shared prompt prefix. What makes turn N of a conversation fast, and why sticky sessions matter.
Quantization — storing weights at reduced precision. Smaller and faster; some quality loss. 4-bit is the common sweet spot.
Routing mode — how much concurrency each backend accepts and whether requests queue. Fleet-wide.
Routing tier — a label isolating a sub-pool, so two workloads sharing a model do not share capacity.
Safetensors — the full-precision (or FP8/FP4) checkpoint format served by the tensor engine.
Server — one model running on one or more GPUs. The unit Lifeboat starts, stops and routes to.
Sin-bin — brief removal of an erroring backend from rotation, so a failing server does not absorb traffic.
Speculative decoding — 2–4x faster single-stream generation with identical output, by proposing and verifying tokens in batches.
Tensor engine — the high-performance engine for safetensors checkpoints on GPUs. Carries the optimization layer.
Tensor parallelism — splitting one model across several GPUs.
Weight (routing) — a backend’s share of traffic within a pool. Not to be confused with model weights.