Skip to main content

Availability and heartbeat

A dead host cannot report that it died. So availability is decided on the server side, and it is tied to the agent's persistent outbound WebSocket rather than to polling. While the connection is alive and data flows, the host is up. The down signal is the connection dropping, or a server poll going unanswered after a period of silence.

The model​

  • Push is primary. The agent pushes at least once every 3 minutes.
  • Server poll is the fallback. If the server hears nothing for 3 minutes, it sends a "collect now" down the already open connection.
  • A dropped connection or a failed poll past the grace period is the down signal.

Because this rides the agent's own outbound connection, there are no inbound ports and no separate heartbeat requests.

Grace periods​

To avoid crying wolf on every network blip, the server applies a grace period before it marks a host down. The values are server-controlled. The logic depends on the situation:

  • Normal: the base grace tied to the 3-minute window.
  • Reboot: a longer grace, so an expected reboot does not wake anyone. The tell is a jump in boot time (uptime resets) after the host reconnects.
  • Maintenance: during a maintenance window, down detection is suppressed or widened.

States​

  1. Healthy: the connection is open and pushes arrive on time.
  2. Watched: a short pause or one missed push. The server sends a "collect now".
  3. Down: the connection dropped or the poll failed longer than the grace period. An incident opens.
  4. Maintenance: expected downtime, down alerts suppressed.
  5. Recovered: the connection is back and pushes resumed after an outage.

Backfill after an outage​

If the host was alive but the server was unreachable (a network split), the agent buffers metrics to a disk-backed ring buffer, bounded by size and age, and replays them on reconnect. The gap in the time series backfills, even if the host was briefly marked down. A silently blackholed connection is caught within about 30 seconds by a WebSocket ping before each heartbeat, so it does not lose data quietly.

Agentless devices​

A device watched by a site collector over SNMP has its own availability model: the scheduled poll is the authoritative state, and a trap or syslog line is only an immediacy hint that triggers a confirming poll before anything alerts. A collector pair (primary and backup) means one collector failing does not black out every device at once. The collector host itself is watched like any other host, by its own connection. See SNMP and network.