Skip to main content

Alert rules

Every workspace starts with the full rule catalog and no setup. Rules arm themselves when the telemetry they watch appears, hold their condition long enough to ignore blips, and close by themselves when the metric recovers. This page explains the mechanics and where to tune them; the full list with every threshold is the rules catalog.

Anatomy of a rule​

A rule watches one metric name, compares it to a threshold with >=, <= or ==, and requires the condition to hold for a duration before an incident opens. Three details make the difference between a rule that pages honestly and one that flaps:

  • The hold is measured in data time. The duration counts from the timestamp of the first breaching sample to the timestamp of the current one, not by the wall clock. A metric that stopped arriving cannot mature a hold on its stale last value, so an incident never opens for something that no longer exists.
  • Recovery has its own threshold (hysteresis). A disk rule that fires at 90% closes at 88%, a CPU rule that fires at 85% closes at 75%. The metric has to cross back clearly, so a value dancing around the line does not open and close the same incident every minute.
  • A tag splits the rule per entity. Disk rules carry the mount, site rules the site, pool rules the pool name, device rules the device. One rule, one incident per entity, each with its own history.

Rules with a monotonic counter behind them (aborted connections, evicted keys, container restarts) watch a rate or a windowed count that the agent derives on the host, never the raw total: a total only grows, so a threshold on it would fire once and never clear.

Rules arm themselves​

The console shows only armed rules: those whose metric actually flows in your workspace. A rule about RabbitMQ does not exist for you until a broker appears, and then it is there, enabled, with its default threshold. Nothing to seed, nothing to map to host types.

Some rules are additionally marked auto: they arm the moment their telemetry appears even on fleets where an operator has not looked at the catalog yet (site availability, PHP-FPM pool saturation, MySQL down, container health, APM latency and error rate, log storms). Auto rules always ship as warnings. An explicit edit you make to a rule wins over its auto behaviour.

A handful of rules ship disabled on purpose because their meaning depends on your environment: an idle queue with no consumer, a stopped virtual machine, a single-node Elasticsearch that is yellow forever, a 5xx rate whose "too much" depends on your traffic. They are listed, and one click arms them.

Severity: critical means a phone call​

Critical is the severity that can ring a phone through escalation, so the default critical set is short: host down, disk, memory and CPU at their critical rungs, certificate expired, domain registration expired, a UPS on battery for three minutes, a network device that stopped answering. Everything else opens as a warning, and auto-armed rules never default to critical. Raising a rule to critical is a deliberate per-workspace decision.

Severity is set when an incident is born. Changing a rule's severity applies to the next incident it opens, not to the ones already open.

Ladders and inhibition​

Host resources come as warning-then-critical ladders: disk 90% then 95%, memory 95% then 98%, CPU 85% then 97%. When the critical rung fires, its warning twin is closed rather than shown twice. Certificates work the same way: "expiring" gives way to "expired".

Related signals inhibit each other so one failure produces one incident:

  • Host down silences every other rule of that host. A box that is off is not also "out of disk".
  • Domain does not resolve mutes the certificate rules of that site, and closes any open certificate incident on it: a dead domain is the finding, not its certificate.
  • A site that points at a CDN is judged on its origin certificate as a warning ("origin certificate expired, domain points away") instead of the critical the visitor-facing certificate would earn, because visitors see the edge certificate.

Event-born rules​

A few rules have no threshold: a failed Proxmox task, a changed device configuration, an integrity change on /etc/passwd, /etc/shadow or /etc/sudoers. They open when the event arrives and carry the source's own reason (the task's error text, the file that changed). They close when the same task runs clean, or when the signal has been quiet long enough. The catalog lists them so you can switch a family off or raise its severity.

The rule that watches the watching​

Monitoring cannot collect from a service opens when a service's probes have not produced one successful check in six hours. Every other rule watches a metric, and a broken probe produces none, so without this rule a revoked permission or a rotated password would look exactly like a healthy service for weeks.

Anomalies are telemetry, not incidents​

The agent scores every metric series against its own recent baseline on the host and records an anomaly event when a series deviates strongly and stays there. Those events feed the host story and the events feed; they do not open incidents. A rule for anomaly storms exists and ships off, for workspaces that have watched their baseline long enough to know what a storm looks like.

"Strongly" means two things at once: statistically far from the baseline, and far in the metric's own units, by a threshold that depends on what the metric is. A percentage has to move by ten points, a latency by a hundred milliseconds, a rate by a whole unit per second, a byte counter by a megabyte; a series that sat at exactly zero for a week and then shows 0.03 is not an anomaly, however many standard deviations that is. Metrics whose meaning is "bad when high" (percentages, latencies, error and failure counts) are only flagged when they rise; throughput and traffic are flagged in both directions. Some families are not scored at all because a jump is their normal: top-process rankings, maximum latencies, uptime, countdown gauges such as days left on a certificate, daily totals that reset at midnight, and cumulative counters whose per-minute rate the agent already derives (the rate is scored instead). That table of classes comes from the server with the definitions, so it can be tuned without an agent release.

Tuning in the console​

Alerting, Rules tab. Click a rule to edit its threshold, its recovery threshold, how long the condition must hold, and its severity, or switch it off. A tuned rule shows a "custom" pill; Reset returns it to the catalog default. Rules you never edited follow the catalog as it evolves.

Custom rules​

New rule builds your own from any metric in the metrics reference: metric name (wildcards like svc.%.vhost.ssl_days_left are allowed), comparison, threshold, recovery threshold, duration, severity, and the tag that splits it per entity. A custom rule can also watch a log pattern: a text pattern counted over a 15-minute window per host and service, so a recurring line becomes one incident instead of a message per occurrence.

Runbooks​

Every rule row also carries a runbook card: the instruction the on-call person gets with the incident, scoped to the whole workspace, to hosts with a tag, or to one host, and delivered on every channel together with the alert. See Runbooks.

Per-host rules​

A host whose normal is somebody else's alert (a database server that lives at 90% memory) can leave the workspace layer entirely. The Alert rules for this host block on its host page switches that host to the catalog defaults plus its own rows; workspace-level edits of catalog rules stop applying to it, while workspace custom rules still do. Only rules whose telemetry flows from that host are offered there. See Hosts.

What happens after a rule fires​

The incident, its acknowledgement and resolution are described in Incidents. Who gets told, and how hard it escalates, is in Notification channels and Escalation and on-call. Planned work is quieted with Maintenance windows, and scheduled jobs get a dead-man's switch in Cron heartbeats. The same rules can be listed and edited over the API and by an AI agent.