Alert rules catalog

Every workspace gets this catalog without configuration. The console shows the rules whose metric flows in your workspace; this page lists all of them. How the columns work is explained in Alert rules:

• Opens when: the comparison against the metric that starts the hold.
• Hold: how long the condition must persist, measured on the data's own timestamps, before the incident opens. "Immediate" means the first breaching sample opens it (used where the metric is already a rolling count or a state).
• Closes when: the value the metric must reach for the incident to resolve. It differs from the opening threshold on purpose (hysteresis).
• Ships: on = enabled by default; auto = arms itself as soon as the metric appears in your workspace; off = listed but disabled until you enable it, because its meaning depends on your environment.

Rates are per second unless the metric name says per_min or _1h. A metric name with % matches every service that produces it (for example every web server's per-site certificate metric). Any threshold, hold, recovery value or severity can be changed per workspace or per host; a rule you edit shows as "custom" and can be reset.

Host resources

Rule	Metric	Opens when	Hold	Closes when	Severity	Ships
disk almost full	host.disk.used_pct per mount	>= 95	2 min	< 93	critical	on
disk usage high	host.disk.used_pct per mount	>= 90	5 min	< 88	warning	on
memory almost exhausted	host.mem.used_pct	>= 98	2 min	< 96	critical	on
memory pressure	host.mem.used_pct	>= 95	5 min	< 92	warning	on
cpu pegged (critical)	host.cpu.used_pct	>= 97	15 min	< 90	critical	on
cpu pegged	host.cpu.used_pct	>= 85	15 min	< 75	warning	on
load far above the core count	host.load1_per_core	>= 4	15 min	< 2	warning	on
swap nearly full	host.swap.used_pct	>= 90	15 min	< 75	warning	on
out-of-memory kill	svc.systemd.oom_kills_rate	>= 0.01	immediate	< 0.005	warning	on
systemd degraded	svc.systemd.degraded	== 1	2 min	!= 1	warning	on
host is waiting for a reboot	svc.os.reboot.exists	>= 1	6 h	< 1	warning	on
security updates are piling up	svc.os.security_updates	>= 50	1 h	< 20	warning	on
critical CVEs in installed packages	svc.os.cve_critical	>= 1	immediate	< 1	warning	on
backup is stale	svc.%.age_sec	>= 93600	immediate	< 90000	warning	on
monitoring cannot collect from a service	check.probe.broken_hours per service	>= 6	immediate	< 1	warning	on
filesystem mounted read-only	host.disk.readonly per mount	>= 1	2 min	< 1	warning	on

Websites, certificates, domains

Rule	Metric	Opens when	Hold	Closes when	Severity	Ships
site down (every local probe fails)	check.vhost.fail_pct per site	>= 100	10 min	< 50	warning	auto
site is slow (apm)	apm.site.p95_ms per site	>= 3000	15 min	< 2000	warning	auto
site error rate high (apm)	apm.site.error_pct per site	>= 20	15 min	< 10	warning	auto
site backend is not running	svc.%.vhost.backend_missing per vhost	>= 1	5 min	< 1	warning	on
site document root is missing	svc.%.vhost.docroot_missing per vhost	>= 1	10 min	< 1	warning	on
site serves 404 for its own pages	log.site.script_missing_15m per site	>= 20	immediate	< 5	warning	auto
certificate does not cover some site names	svc.%.vhost.ssl_name_ok per vhost	>= 1	1 h	< 1	warning	on
certificate expired	svc.%.vhost.ssl_days_left per vhost	<= 0	immediate	> 1	critical	on
certificate expiring	svc.%.vhost.ssl_days_left per vhost	<= 14	immediate	> 15	warning	on
origin certificate expired (domain points away)	svc.%.vhost.ssl_days_left per vhost	<= 0	immediate	> 1	warning	on
domain does not resolve	svc.%.vhost.dns_ok per vhost	== 0	10 min	!= 0	warning	on
domain registration expired	svc.%.vhost.domain_days_left per vhost	<= 0	immediate	> 1	critical	on
domain registration expiring	svc.%.vhost.domain_days_left per vhost	<= 14	immediate	> 15	warning	on
domain recently re-registered (possible drop-catch)	svc.%.vhost.domain_age_days per vhost	<= 30	immediate	> 31	warning	on
site unreachable from the internet (server looks fine)	synth.site.fail_pct per site	>= 100	5 min	< 50	warning	auto
synthetic transaction fails from every vantage	txn.flow.fail_pct per flow	>= 100	5 min	< 50	warning	off
site throws js errors (rum)	rum.site.err_per_100v per site	>= 25	15 min	< 10	warning	auto
nginx serves 5xx at a high rate	svc.nginx.server_error_rate	>= 1	10 min	< 0.3	warning	off
apache serves 5xx at a high rate	svc.apache.server_error_rate	>= 1	10 min	< 0.3	warning	off
php-fpm pool saturated (requests queueing)	svc.php-fpm.pool.backlog per pool	>= 1	5 min	< 1	warning	auto
node event loop lag high	svc.node.nodejs_eventloop_delay_mean_ms per unit	>= 100	5 min	< 50	warning	auto

Databases, caches, search

Rule	Metric	Opens when	Hold	Closes when	Severity	Ships
mysql is not running	svc.mysql.unit_active	== 0	5 min	!= 0	warning	auto
mysql is running out of connections	svc.mysql.connections_used_pct	>= 85	5 min	< 70	warning	on
clients cannot connect to mysql	svc.mysql.aborted_connects_per_min	>= 10	5 min	< 3	warning	on
mysql queries are not finishing	svc.mysql.running	>= 10	5 min	< 4	warning	on
redis is close to its memory limit	svc.redis.memory_used_pct	>= 90	10 min	< 80	warning	on
redis is refusing connections	svc.redis.rejected_connections_per_min	>= 1	5 min	< 0.5	warning	on
redis is evicting keys	svc.redis.evicted_keys_per_min	>= 1	10 min	< 0.5	warning	off
mongo is running out of connections	svc.mongo.connections_available	<= 500	5 min	> 1000	warning	on
elasticsearch cluster is red	svc.elasticsearch.status_red	>= 1	5 min	< 1	warning	on
elasticsearch shards unassigned	svc.elasticsearch.unassigned_shards	>= 1	15 min	< 1	warning	off
clickhouse is not running	svc.clickhouse.unit_active	== 0	5 min	!= 0	warning	auto
clickhouse parts piling up (merges lagging)	svc.clickhouse.max_parts_per_partition	>= 300	10 min	< 200	warning	auto
clickhouse replica lagging	svc.clickhouse.replica_max_delay_sec	>= 300	10 min	< 60	warning	auto
mssql requests blocked on locks	svc.mssql%.blocked_requests	>= 1	10 min	< 1	warning	auto
mysql queries are waiting on row locks	svc.mysql.innodb_row_lock_waits_per_min	>= 60	10 min	< 30	warning	auto
mysql is running a burst of slow queries	svc.mysql.slow_queries_per_min	>= 10	10 min	< 5	warning	auto
postgres keeps hitting deadlocks	svc.postgres.deadlocks_per_min	>= 0.5	10 min	< 0.5	warning	auto

Mail, DNS, FTP

Rule	Metric	Opens when	Hold	Closes when	Severity	Ships
mail bounce storm	svc.postfix.bounced_rate	>= 0.5	15 min	< 0.2	warning	on
mail queue backlog	svc.postfix.queue_requests	>= 200	15 min	< 100	warning	on
mail queue backlog	svc.exim.queue	>= 200	15 min	< 100	warning	on
outgoing mail keeps getting deferred	svc.postfix.deferred_rate	>= 0.1	15 min	< 0.05	warning	auto
mail brute-force in progress	svc.dovecot.auth_fail_rate	>= 0.3	2 min	< 0.1	warning	on
clamav signatures are stale	svc.pmg.clamdb.age_sec	>= 172800	1 h	< 86400	warning	on
sender domain on a blacklist	svc.maildeliv.rbl_listed per domain	>= 1	immediate	< 1	warning	off
bind recursion slots nearly full	svc.bind.recursive_clients	>= 900	5 min	< 700	warning	off
powerdns is dropping queries under load	svc.powerdns.overload_drop_rate	>= 0.5	5 min	< 0.5	warning	auto
powerdns backend is not answering	svc.powerdns.timedout_rate	>= 30	5 min	< 15	warning	auto
powerdns answers SERVFAIL to many queries	svc.powerdns.servfail_pct	>= 20	10 min	< 10	warning	auto
ftp brute-force in progress	svc.pure-ftpd.login_fail_rate	>= 0.2	2 min	< 0.05	warning	auto
ftp brute-force in progress	svc.vsftpd.login_fail_rate	>= 0.2	2 min	< 0.05	warning	auto

Queues and proxies

Rule	Metric	Opens when	Hold	Closes when	Severity	Ships
rabbitmq messages are piling up	svc.rabbitmq.messages_ready	>= 10000	15 min	< 5000	warning	on
rabbitmq queue has no consumers	svc.rabbitmq.consumers	== 0	10 min	!= 0	warning	off
kafka partitions under-replicated	svc.kafka.under_replicated per topic	>= 1	5 min	< 1	warning	on
nats has slow consumers	svc.nats.slow_consumers_per_min	>= 0.5	5 min	< 0.5	warning	auto
nats clients keep reconnecting	svc.nats.total_connections_per_min	>= 30	10 min	< 15	warning	auto
haproxy backend is down	svc.haproxy.proxy.up per proxy	== 0	2 min	!= 0	warning	on
varnish cannot reach its backends	svc.varnish.backend_fail_per_min	>= 1	5 min	< 0.5	warning	on

Containers, virtualization, clusters

Rule	Metric	Opens when	Hold	Closes when	Severity	Ships
container failing its healthcheck	svc.docker.group.unhealthy per image	>= 1	2 min	< 1	warning	auto
containers keep restarting	svc.docker.group.restarts_1h per image	>= 5	5 min	< 1	warning	on
pods in CrashLoopBackOff	svc.k8s.pods_crashloop	>= 1	5 min	< 1	warning	auto
kubernetes nodes not ready	svc.k8s.nodes_ready_pct	<= 99	5 min	> 100	warning	on
kubernetes pods cannot start	svc.k8s.pods_pending	>= 1	15 min	< 1	warning	on
kubernetes pods in failed state	svc.k8s.pods_failed	>= 1	15 min	< 1	warning	on
libvirt storage pool is filling up	svc.libvirt.storage.used_pct per pool	>= 90	15 min	< 85	warning	on
libvirt guest is not running	svc.libvirt.guest.running per domain	== 0	5 min	!= 0	warning	off
hyper-v guest is not running	svc.hyperv.guest.running per vm	== 0	5 min	!= 0	warning	off
proxmox node lost quorum	svc.proxmox.cluster.quorate	== 0	2 min	!= 0	warning	on
proxmox ha resource in error	svc.proxmox.ha.errors	>= 1	5 min	< 1	warning	on
proxmox guest is not running	svc.proxmox.guest.running per name	== 0	5 min	!= 0	warning	off
proxmox storage is filling up	svc.proxmox.storage.used_pct per storage	>= 90	15 min	< 85	warning	on
proxmox task failed	svc.proxmox.%	the event arrives	immediate	the source reports clean or goes quiet	warning	on
corosync is not running	svc.proxmox.cluster.corosync_active	== 0	2 min	!= 0	warning	on

Replicated storage

Rule	Metric	Opens when	Hold	Closes when	Severity	Ships
drbd disk not up to date	svc.drbd.device.uptodate per res	== 0	5 min	!= 0	warning	on
drbd resource lost quorum	svc.drbd.device.quorum per res	== 0	1 min	!= 0	warning	on
drbd replication link down	svc.drbd.connection.established per link	== 0	2 min	!= 0	warning	on
drbd resync makes no progress	svc.drbd.peer.resync_stalled per link	>= 1	10 min	< 1	warning	on
drbd out-of-sync data on a connected link	svc.drbd.peer.oos_lingering_kib per link	>= 1024	30 min	< 1	warning	on
drbd link flapping	svc.drbd.connection.flaps_hour per link	>= 6	immediate	< 3	warning	on
linstor satellite offline	svc.linstor.node.online per node	== 0	2 min	!= 0	warning	on
linstor traffic on the failsafe netif	svc.linstor.netif.on_default per node	== 0	15 min	!= 0	warning	on
linstor storage pool almost full	svc.linstor.pool.free_pct per pool	<= 15	10 min	> 20	warning	on
linstor error reports piling up	svc.linstor.error_reports_hour	>= 30	immediate	< 10	warning	on
leftover vzdump snapshots	svc.linstor.stale_snapshots	>= 1	immediate	< 1	warning	on
linstor satellite is not running	svc.linstor-satellite.unit_active	== 0	2 min	!= 0	warning	on

Windows

Rule	Metric	Opens when	Hold	Closes when	Severity	Ships
iis site is stopped	svc.iis.site.started per site	== 0	5 min	!= 0	warning	on
iis application pool is stopped	svc.iis.pool.started per pool	== 0	5 min	!= 0	warning	on

Security and access

Rule	Metric	Opens when	Hold	Closes when	Severity	Ships
ssh brute-force in progress	svc.ssh.auth_fail_rate	>= 0.3	2 min	< 0.1	warning	on
root logged in over ssh	svc.ssh.root_login_rate	>= 0.01	immediate	< 0.005	warning	on
sshd lets root log in with a password	svc.ssh.root_login_permitted	>= 1	immediate	< 1	warning	on
sshd accepts passwords	svc.ssh.password_auth_on	>= 1	immediate	< 1	warning	on

Network devices

Rule	Metric	Opens when	Hold	Closes when	Severity	Ships
ups on battery - power is out	dev.ups.seconds_on_battery per device	>= 180	immediate	< 30	critical	on
ups switched to battery	dev.ups.seconds_on_battery per device	>= 60	immediate	< 30	warning	on
ups battery runtime low	dev.ups.minutes_remaining per device	<= 10	2 min	> 15	warning	on
device sensor reports a fault	dev.sensor.ok per device,sensor	== 0	2 min	!= 0	warning	on
device stopped answering snmp	check.device.fail_pct per device	>= 100	5 min	< 50	critical	on
device credential missing on the collector	check.cred.missing per secret	>= 1	2 min	< 1	warning	on
nobody polls the device	check.devsilent.none per device	>= 1	5 min	< 1	warning	on
network port lost its link	dev.if.link_lost per device,port	>= 1	5 min	< 1	warning	on
network devices keep logging errors	dev.syslog.severe_rate	>= 0.015	10 min	< 0.005	warning	auto
device is sending traps	event.trap	>= 5	immediate	< 2	warning	off

Logs, events, jobs

Rule	Metric	Opens when	Hold	Closes when	Severity	Ships
service floods error logs	log.svc.errors_15m per service	>= 50	immediate	< 2	warning	auto
log volume capped - lines are being dropped	event.log_volume_capped	>= 1	immediate	< 1	warning	on
a log file is flooding	event.log_source_flooding	>= 1	immediate	< 1	warning	on
host is throwing anomalies	event.anomaly	>= 20	immediate	< 5	warning	off
cron jobs keep failing	svc.cron.job_failed_rate	>= 0.005	15 min	< 0.001	warning	off

Rules with no threshold

Two families are not in the tables above because they have no metric to compare:

• Host down. Opens when a host's connection drops or its heartbeat stops (the exact timings are in Availability); it is critical and closes the moment the host reports again. It also silences every other rule of that host while open.
• Integrity change on /etc/passwd, /etc/shadow or /etc/sudoers: one warning per changed file, closed on its own once the file has stayed unchanged for a day.
• Device configuration changed, for network devices with configuration backup enabled: one warning per device, with the diff on the device page; you resolve it once the change is reviewed.

Custom rules you create (any metric, or a log pattern) appear in the console next to these, marked as yours.