Skip to content
Docs

Alerts & Health

The bell in the top bar counts open alerts and takes the colour of the most severe. Clicking it opens the Alerts page.

The Alerts page showing health events raised by the control plane

Critical — act now

Section titled “Critical — act now”
Alert Meaning First check
node_offline A node stopped answering. Resolves itself when it returns. Agent running? Port reachable? Cluster secret matching?
server_unreachable Marked running but not answering. The server’s log; whether the process still exists.
server_gave_up Restart budget exhausted. The log — this needs a human.
server_deterministic_crash It fails the same way every time. A configuration problem, not bad luck. Read the classified reason.
gpu_failure The GPU reported a fault. The host’s own GPU tooling and kernel log.
disk_full Out of disk. Downloads and logs stop working. Free the models or data volume.
proxy_pool_exhausted No backend could take a request. Capacity, or everything is unhealthy at once.
database_error The control plane’s own store failed. Disk, and the data volume’s health.

Warning — look soon

Section titled “Warning — look soon”

high_vram_usage, high_cpu_or_ram, detokenizer_stall, model_download_failed, cluster_member_down, disk_warning, server_start_rejected, and:

server_auto_restarted — the one most often dismissed too quickly. It means something crashed. Once, after a transient, that is the system working as designed. Repeatedly, it is a server that cannot stay up, and its log says why. A recurring auto-restart warning is a real incident that has been silently absorbed.

lb_sin_bin_churn — a backend is erroring often enough to be repeatedly benched and returned. Usually the backend, occasionally a request shape that only it receives.

Info — a record, not a problem

Section titled “Info — a record, not a problem”

server_started, server_stopped, cluster_created, cluster_edited, model_download_completed, user_logged_in, config_changed.

These are mirrored from the audit log so the two views agree about what happened.

Auto-resolution

Section titled “Auto-resolution”

Some alerts clear themselves when the condition ends — node_offline is the clearest case, resolving on the node’s next successful heartbeat. Do not treat a disappeared alert as one that never mattered; the audit log retains the event.

Sources and references

Section titled “Sources and references”