Availability and heartbeat

A dead host cannot report that it died. So availability is decided on the server side, and it is tied to the agent's persistent outbound WebSocket rather than to polling. While the connection is alive and data flows, the host is up. The down signal is the connection dropping, or silence past a grace period.

The model

• Push is primary. The agent sends a heartbeat every 30 seconds and metrics every minute (the production defaults).
• The server pings too. The server pings every open connection every 15 seconds. A socket that stops answering (a power cut sends no FIN) is closed within about 25 seconds instead of lingering as "connected".
• Server poll is the fallback. If a live connection goes quiet, the server sends a "collect now" down the already open socket before concluding anything.
• A dropped connection or continued silence past the grace period is the down signal.

Because this rides the agent's own outbound connection, there are no inbound ports and no separate heartbeat requests.

Grace periods

To avoid crying wolf on every network blip, the server grades the evidence before it marks a host down:

• An explicit disconnect (the socket closed, or the ping went unanswered) is a strong signal and gets a short grace, tens of seconds.
• Bare silence on a live socket is weak evidence: the server polls first and waits a longer grace.
• Every down verdict needs two consecutive sweeps, so a one-off scheduling hiccup on a loaded host does not open an incident.
• A self-update restart is expected: the agent announces the update before it restarts, and the server holds the down verdict for a few minutes unless the host fails to come back.

In practice a hard power cut is detected, incident raised and delivered, in about a minute; a clean shutdown a bit faster. The exact values are server-side and can change without an agent update.

States

1. Healthy: the connection is open and pushes arrive on time.
2. Watched: a short pause or one missed push. The server sends a "collect now".
3. Down: the connection dropped or the poll failed longer than the grace period. An incident opens.
4. Maintenance: expected downtime, down alerts suppressed (maintenance windows are a console feature and live server-side).
5. Recovered: the connection is back and pushes resumed after an outage.

Backfill after an outage

If the host was alive but the server was unreachable (a network split), the agent buffers metrics and heartbeats to a disk-backed ring buffer, bounded by size and age, and replays them with their original timestamps on reconnect. The gap in the time series backfills and the availability minutes land in their proper slots, so a control-plane outage does not read as your host being down. A silently blackholed connection is caught within seconds: the agent pings its connection every ten seconds on its own timer and declares the link dead when the answer does not come, so a server that vanished without closing the socket does not hold the agent for minutes. When it redials, the endpoint name may resolve to several addresses; the agent tries them in a fresh random order each time, each one gets a few seconds before the next is tried, and an address that answered with a server error is moved to the back of the list for a minute, so a single silent or sick address never blocks the reconnect and a fleet does not all pile onto the first address of the list. Retries back off from one second to half a minute with a random spread, and the backoff resets only after a session that held for a minute, so a flapping link cannot turn into a tight loop. While the connection is down, the agent keeps writing a heartbeat every 30 seconds into its spool from the first second of the outage, so the availability record of a host that was up all along stays complete once the spool replays.

Fairness cuts the other way too: a minute in which the monitoring platform itself was unreachable is excluded from the expected uptime of every host, and so are the minutes a planned restart of the platform took (they are recorded as such, with the fleet-wide silence they caused). Platform downtime is ours to own; it never shows up as a dent in your availability numbers. Only completed minutes are ever judged: the minute that is still running is not counted against anyone.

The second eye: outside-in synthetics

The agent watches from inside the box. Site checks go to loopback, and the heartbeat rides the agent's own tunnel. If the box is healthy but its ingress is dead (the uplink drops packets, sites are unreachable from the internet while everything looks fine locally), the inside view cannot see that by design.

That gap is closed from the other side: Wakora's own points of presence probe your public sites over HTTPS from outside and report reachability, status and latency per site and per vantage. The same vantages can run multi-step transaction flows (login, cart, checkout) defined in the console. The rules are conservative so the second eye never cries wolf:

• Only genuinely public sites are probed: names that resolve to public addresses. Catch-alls, IP literals and internal names are excluded.
• An ingress outage verdict needs a quorum: every vantage failing at once, while the inside view is still healthy. If the host itself is down, that is the host-down incident's territory, not this one.
• A DNS hiccup at a vantage counts as unknown, never as down. A dead domain is the DNS liveness signal's job.

The ext probe additionally does external checks from a collector host you designate, for a vantage of your own. See the metrics reference for the synthetic metric names.

Agentless devices

A device watched by a site collector over SNMP has its own availability model: the scheduled poll is the authoritative state, and a trap or syslog line is only an immediacy hint that triggers a confirming poll before anything alerts. A collector pair (primary and backup) means one collector failing does not black out every device at once. The collector host itself is watched like any other host, by its own connection. See SNMP and network.