Skip to content
Docs

Embedding & Reranker Servers

An embedding model is a prefill-only encoder: one pooled forward pass per request, no decode loop. Almost every generation-oriented default is wrong for it, and the failures are late and unhelpful.

You do not configure this

Section titled “You do not configure this”

Lifeboat detects an embedding model at start, sets the flag, strips every generation-only setting, and records what it did in the audit log as server_is_embedding_autodetected.

Detection is ordered by authority, because a decoder-based embedder cannot be identified from its config at all — some report exactly the same architecture as their chat counterpart, which is precisely why the engine needs an explicit flag:

  1. A sentence-transformers pooling module on disk — the strongest signal, and name-independent, so it works for a repository called anything.
  2. An encoder or scorer architecture.
  3. Name hints — weakest, but often the only signal when the weights are not local yet, since a repository id has no config to read.

It only fills in a value you have not set. If you explicitly unticked it, you are never overridden.

What gets stripped

Section titled “What gets stripped”

Every generation-only setting, because none of them is meaningful without a decode loop: the prefill graph capture, the overlap scheduler, cache precision overrides, the GPU optimization suite, and speculative decoding.

Reasoning and tool-call parsers are stripped too, and this is the one people set by hand and regret. They only shape generated text, and configuring one on an embedding model causes a start failure whose error message names neither the setting nor the model. If a server refuses to start with a message about a think-end token, clear both parsers.

Routing keeps the two apart

Section titled “Routing keeps the two apart”

Direction is a hard filter, not a preference:

  • /v1/embeddings goes only to embedding backends; with none running you get 501 no_embedding_backend.
  • Chat, completions, messages and responses exclude embedding backends; with only embedders running you get 503 embedding_only_pool.

Both facts are knowable from the server row, so there is nothing to discover over the network. Without this split, an embeddings request carrying a model name that matched nothing would be routed by position and answered by a chat model about half the time — an error carried inside an HTTP 200, which reads as a flaky backend rather than a routing bug.

Rerank, score and classify are deliberately in neither list. They run on embedding-class servers on the tensor engine, but a GGUF backend may or may not implement them, so that is a genuine runtime question and is answered by failing over to the next candidate.

The one request that kills the engine

Section titled “The one request that kills the engine”

A generation request that reaches an embedding server takes down the whole engine process, not just that request. Lifeboat blocks every path it controls — the routing split above, the warmup, and its own capability probe, which is itself a chat request and used to kill embedding servers about ten seconds after they reported healthy.

A client talking to a backend port directly bypasses all of that. Route through the control plane.

Sources and references

Section titled “Sources and references”