Skip to main content

Monitoring systemd

Every Linux host that boots with systemd gets this definition, and there is nothing to switch on. The agent asks systemd which units have failed and whether the system as a whole is degraded, and it reads the kernel's journal for out-of-memory kills. That covers the "my service just vanished" class of mystery on its own: a unit that died, and a process the kernel shot.

How systemd is recognised​

The agent records which init system runs on a host as a discovery fact. When it is systemd, the systemd definition matches; OpenRC and sysvinit hosts get their own log definitions (below) but not this one. Every two minutes the agent runs three read-only probes:

  • systemctl --failed --plain --no-legend --no-pager for the count of failed units and their names,
  • systemctl is-system-running for the system state (degraded means at least one unit is in the failed state),
  • the journal entries of the kernel identifier, counting the lines that report an OOM kill, both the host-wide Out of memory: Killed process and the cgroup form Memory cgroup out of memory: Killed process.

The agent never starts, stops or resets a unit. It only reports what systemd says.

What is collected​

Failed units. How many units are in the failed state, and which ones: the names ride as the fact failedUnits, so a host page can show apparmor.service rather than a bare count.

Degraded state. A 0/1 flag from is-system-running.

OOM kills. Matching kernel lines divided by the seconds since the previous sample, so the metric is a rate per second rather than a counter that only grows.

SignalMetric
Failed unitssvc.systemd.failed_units
System degradedsvc.systemd.degraded
OOM kills per secondsvc.systemd.oom_kills_rate

Names, tags and units are in the metrics reference.

Logs. The logs-system definition ships the journal entries of sshd, sshd-session, sudo, su, cron, crond, systemd, kernel, systemd-logind and polkitd. The level comes from the journal priority, and only notice and above ships by default; the kernel's bookkeeping lines ("N callbacks suppressed") are filed as info so they never make the default stream. Hosts without systemd are not blind: logs-system-openrc follows /var/log/messages, logs-system-sysvinit follows /var/log/messages and /var/log/syslog. Those files carry no per-line priority, so each new line ships as notice. See Logs.

Alerts out of the box​

RuleFires whenSeverityShips
systemd degradedthe system state reads degraded for 2 minuteswarningon
Out-of-memory killthe kernel OOM-kill rate reaches 0.01 per secondwarningon

Degraded means a unit failed and stayed failed across two consecutive samples; a unit that crashes and is restarted cleanly by systemd inside the window never raises anything, and the incident closes on the first sample that reads running again. The OOM rule is a rate over the two-minute sampling window, so in practice it takes two or more kills inside one window to open the incident, while a single kill still shows on the metric and in the log stream. Both ship as warnings: a failed unit is something to look at, not a phone call. Thresholds and hysteresis for each rule are in the rules catalog.

Limitations​

  • The agent counts and names failed units; it does not reset them. A unit that fails by design during a package upgrade keeps the host degraded until you run systemctl reset-failed yourself.
  • No per-unit metrics (restart counts, memory per unit) are collected. Services the catalog knows get their own definitions with their own signals.
  • OOM kills are read from the kernel journal, so the metric exists on systemd hosts only. OpenRC and sysvinit hosts still ship their system log, where the kernel's own message names the killed process.
  • The OOM rate says that the kernel killed something, not what. The kernel line in the log stream carries the process name and PID.