Skip to content
Docs

Troubleshooting — Nodes & Clusters

A node shows offline

Section titled “A node shows offline”
  1. Is the agent running, and is its port reachable from the control plane? The control plane calls out; nothing is pushed to it.
  2. Does the cluster secret match? A mismatch is rejected and surfaced with the node named.
  3. Is it a firewall? The control plane must reach the agent port, not the other way round.

node_offline is a critical alert and clears itself on the node’s next successful heartbeat.

The node is up but every remote start fails

Section titled “The node is up but every remote start fails”

Check the agent container has a writable home directory. An agent whose home is not writable dies at engine import with a permission error mentioning a cache path — on the first launch only, which makes it look intermittent.

A remote start is refused outright

Section titled “A remote start is refused outright”
Message Meaning
llamacpp_remote_unsupported GGUF servers are local-node only. Deploy locally.
model_not_present_on_node Replicate the model to that node first.
insufficient_shared_memory The agent’s container needs a larger /dev/shm and must be recreated.

Encrypted model weights are also local-node only.

A model is on one node but not another

Section titled “A model is on one node but not another”

Replication is explicit, never automatic. Start the download again and choose the node.

Before any remote launch the control plane asks the node whether the model is present; that check understands a weights directory, a single .gguf file, or a directory containing one, so a GGUF-only replica is correctly seen.

Traffic does not match the weights

Section titled “Traffic does not match the weights”

The Dashboard’s load-balancer card shows configured share against actual with a drift badge. Drift beyond ~10% under steady load means a backend is erroring (and being briefly benched), is at capacity, or is parked at weight 0.

lb_sin_bin_churn is the alert for repeated benching.

Requests span clusters and the algorithm seems wrong

Section titled “Requests span clusters and the algorithm seems wrong”

If a request’s candidates span several clusters, or include a standalone server, routing falls back to weighted round-robin — combining two clusters’ algorithms has no principled answer.

Use routing tiers to keep such workloads in separate pools. See Clusters.

A cluster cannot be edited

Section titled “A cluster cannot be edited”

Membership changes — GPUs, weights, member names — need the servers detached, because they change the deploy shape. The algorithm can be changed with servers attached; it is a runtime router knob.

Sources and references

Section titled “Sources and references”