Skip to content
Docs

Capability Probing

Probe asks a running server what the model actually supports, rather than trusting its model card.

What it reports

Section titled “What it reports”

Streaming, real maximum output length, effective context window, JSON mode, tool calling, vision, reasoning traces, and prompt-cache support.

Model cards are frequently optimistic or simply unmaintained; the probe is empirical, so a client can act on it.

The context window is the limit a client must respect

Section titled “The context window is the limit a client must respect”

The number reported is the per-request limit — the smaller of the model’s window and what the cache can hold — not the cache size.

This distinction has teeth. The cache is capacity shared across concurrent requests and is often far larger than any single request may use. A gateway that budgets prompts against a pool figure builds a prompt that “fits”, adds a reserved completion, and is rejected mid-session with a message about exceeding the model’s context length. Reporting the pool as the window is how that happens, so Lifeboat deliberately does not.

If nothing authoritative can be determined, Lifeboat reports nothing rather than a guess — the caller’s fallback is to parse the engine’s own rejection, which states the true limit.

Caching and re-probing

Section titled “Caching and re-probing”

Results are cached on the model row and reused, so a probe runs once rather than on every start.

Two refresh paths:

  • Refresh probe on the Models page, when you believe a cached result is wrong.
  • Automatic re-probe when a stored record was produced by a version known to have recorded something incorrectly. A probe defect should not outlive the build that shipped it, and a cached-and-trusted-forever record is how one does.

Embedding models are never probed

Section titled “Embedding models are never probed”

The probe is itself a chat request, and a generation request reaching an embedding server takes down the whole engine process. Lifeboat detects an embedding server and records a capability record derived from the row instead of probing — everything in it is knowable without asking.

This is worth knowing because it explains a historical symptom: an embedding server that appeared healthy, died about ten seconds later, restarted, and then never did it again — because the failed probe still cached a record.

Sources and references

Section titled “Sources and references”