Skip to content
Docs

Overview

Lifeboat is Iterate.ai’s enterprise LLM inference engine, built for AI agents rather than for single-turn chat. It serves 2x or more concurrent agent sessions on the same GPUs at 100% model quality — model weights are never quantized to get the gain — and ships with a full operational control plane so running private inference in production is a managed product rather than a collection of scripts.

Who is Lifeboat for?

Section titled “Who is Lifeboat for?”

Lifeboat is for organizations running AI workloads on their own hardware: platform and MLOps teams operating GPU fleets, enterprises with data-sovereignty or air-gap requirements, and teams whose per-token cloud inference bill has become the dominant cost of an AI program. It is used in healthcare, financial services, legal, manufacturing and energy, and government and defense.

Why do agents break conventional inference engines?

Section titled “Why do agents break conventional inference engines?”

Agent workloads look nothing like chat. A single agent task means many repeated model calls, tool-calling round trips, and long-running context that grows with every turn. On a conventional engine, that pattern exhausts the key-value cache long before it exhausts compute, at which point requests are retracted or queued and tail latency collapses — one heavy session can starve the rest. Lifeboat is engineered specifically for that failure mode.

What does Lifeboat do?

Section titled “What does Lifeboat do?”
  • Inference optimization — KV-cache compression and adaptive cache management, fair scheduling so no session is starved, memory tiering, admission control, dynamic precision tuning, and dynamic expert handling for mixture-of-experts models — all at full-precision weights.
  • Model management — a registry with guided download, capability probing, quantization support, and first-class GGUF handling including a per-variant download picker.
  • Servers & multi-GPU — full inference-server lifecycle, multi-GPU tensor parallelism, auto-restart, and per-server logs.
  • Clusters & load balancing — multi-node fleets, named clusters that carve GPUs into business-aligned pools, one-call multi-node deployment, and a token-aware load balancer with capacity gating.
  • Control plane & dashboard — a browser UI for the whole system, with no CLI required for day-to-day operation.
  • OpenAI-compatible API — existing SDKs and applications point at Lifeboat with a base-URL change.
  • Security & administration — role-based access control, API keys for inference traffic, encrypted secret storage, and a full audit trail.

How much faster is Lifeboat?

Section titled “How much faster is Lifeboat?”

Published benchmarks against the baseline open-source engine on identical hardware, with full-precision (BF16) model weights in every Lifeboat measurement:

Metric Baseline Lifeboat Improvement
KV cache capacity 284K tokens 568K tokens 2.0x
Max concurrency at 100% success 1,024 2,048 2.0x
Throughput at 2,048 concurrent 4,965 tok/s 8,714 tok/s 1.76x
Concurrent agents at same quality 16 32 2.0x
p99 time-to-first-token at 128 sessions 189 s 1.5 s ~129x faster

At 2,048 concurrent sessions the baseline engine completes roughly 62% of requests; Lifeboat completes all of them.

Which models does Lifeboat support?

Section titled “Which models does Lifeboat support?”

168+ model architectures, including the LLaMA, Qwen, DeepSeek, Gemma, Mistral/Mixtral, Phi, GLM, Command-R, Falcon, and multimodal families such as InternVL, LLaVA, and Pixtral — with a general fallback loader so the long tail still serves. Quantized weights (FP8, AWQ, GPTQ, INT4/INT8, NVFP4/MXFP4, and GGUF) are supported as an independent option. See Model management.

Where does Lifeboat run?

Section titled “Where does Lifeboat run?”

On NVIDIA GPUs in your own environment — as a container or a bare-metal installer, in on-premises data centers, private cloud, or fully air-gapped sites. Single-node and multi-node topologies use the same artifacts. See Deployment.

How does Lifeboat relate to the rest of the Iterate.ai suite?

Section titled “How does Lifeboat relate to the rest of the Iterate.ai suite?”

Lifeboat is the private inference layer. Interplay and Generate point their model calls at it, and AgentWatch sits in front of it as a governance gateway when per-team budgets, data-loss prevention, and cross-provider audit are required.

Sources and references

Section titled “Sources and references”