Skip to content
Docs

Alerts & Health

The bell in the top bar counts open alerts and takes the colour of the most severe. Clicking it opens the Alerts page.

Critical — act now

Section titled “Critical — act now”
Alert Meaning First check
node_offline A node stopped answering. Resolves itself when it returns. Agent running? Port reachable? Cluster secret matching?
server_unreachable Marked running but not answering. The server’s log; whether the process still exists.
server_gave_up Restart budget exhausted. The log — this needs a human.
server_deterministic_crash It fails the same way every time. A configuration problem, not bad luck. Read the classified reason.
gpu_failure The GPU reported a fault. The host’s own GPU tooling and kernel log.
disk_full Out of disk. Downloads and logs stop working. Free the models or data volume.
proxy_pool_exhausted No backend could take a request. Capacity, or everything is unhealthy at once.
database_error The control plane’s own store failed. Disk, and the data volume’s health.

Warning — look soon

Section titled “Warning — look soon”

high_vram_usage, high_cpu_or_ram, detokenizer_stall, model_download_failed, cluster_member_down, disk_warning, server_start_rejected, and:

server_auto_restarted — the one most often dismissed too quickly. It means something crashed. Once, after a transient, that is the system working as designed. Repeatedly, it is a server that cannot stay up, and its log says why. A recurring auto-restart warning is a real incident that has been silently absorbed.

lb_sin_bin_churn — a backend is erroring often enough to be repeatedly benched and returned. Usually the backend, occasionally a request shape that only it receives.

Info — a record, not a problem

Section titled “Info — a record, not a problem”

server_started, server_stopped, cluster_created, cluster_edited, model_download_completed, user_logged_in, config_changed.

These are mirrored from the audit log so the two views agree about what happened.

Auto-resolution

Section titled “Auto-resolution”

Some alerts clear themselves when the condition ends — node_offline is the clearest case, resolving on the node’s next successful heartbeat. Do not treat a disappeared alert as one that never mattered; the audit log retains the event.

Sources and references

Section titled “Sources and references”