Monitoring DRBD and LINSTOR
Replicated storage fails quietly: a link drops, a resync stops making progress, out-of-sync blocks linger on a connection that looks healthy, and nothing in the guest notices until a failover. The DRBD definition reads the kernel's own status every 30 seconds and turns those states into metrics and alerts. On the node that runs the LINSTOR controller, a second definition reads the controller's REST API for the cluster-wide view. One API token stored on that node is the only thing you set up.
How DRBD and LINSTOR are recognised
Four signed definitions cover the stack, each matching a different fact:
drbdmatches a systemd host with thedrbd-utilspackage installed and runsdrbdsetup status --jsonevery 30 seconds.linstormatches thelinstor-controller.serviceunit and reads the controller's REST API on the loopback interface every minute, trying HTTPS on port 3371 first and HTTP on 3370 second, with a bearer token from the credential namedlinstor-controller.linstor-controllermatches the same unit, reports whether it is active every two minutes, and scrapes the controller's Prometheus/metricsendpoint on the same two ports.linstor-satellitematcheslinstor-satellite.serviceand reports whether it is active.
Matching on the package and the units rather than a process means a stopped satellite or controller still matches, so "is it running" can be answered honestly.
What is collected
DRBD, per resource. Role (primary or not), suspension, whether the local device is up to
date and has quorum, pending lower-level IO, and how many replicas are up to date. Resources
whose name starts with snap_, the transient snapshots a backup creates, are skipped
entirely: they are out of sync by nature and would only make noise.
DRBD, per replication link (a resource and its peer): whether the connection is established, whether it is congested, how many times it changed state in the last hour, the out-of-sync data on the peer in KiB, the part of it that lingers on a link that is already connected, a stalled flag set when a resync shows no progress for ten minutes, and the resync percentage and estimate while a resync runs.
LINSTOR, cluster-wide (controller node only): whether each node is online as the controller sees it, whether a node's traffic rides its default network interface or a failsafe one, free space per storage pool per node, error reports filed in the last hour, and backup snapshots older than a day that should have been cleaned up.
Facts shown on the host page: the DRBD kernel module version, the count of resources and a list of degraded devices; the LINSTOR controller version, nodes online, offline nodes, nodes on a failsafe interface and stale snapshot names.
| Signal | Metric |
|---|---|
| Resources on this node | svc.drbd.resources |
| Per-resource role, suspension, replicas | svc.drbd.resource.primary, .resource.suspended, .resource.replicas_uptodate tagged by res |
| Local device up to date, quorum, pending IO | svc.drbd.device.uptodate, .device.quorum, .device.lower_pending |
| Link state, congestion, flaps per hour | svc.drbd.connection.established, .connection.congested, .connection.flaps_hour tagged by link |
| Out-of-sync data, lingering, stalled resync | svc.drbd.peer.oos_kib, .peer.oos_lingering_kib, .peer.resync_stalled |
| Resync progress | svc.drbd.peer.resync_percent, .peer.eta_sec |
| Satellite liveness and network path | svc.linstor.node.online, svc.linstor.netif.on_default tagged by node |
| Storage pool free space | svc.linstor.pool.free_pct tagged by pool |
| Error reports, stale snapshots | svc.linstor.error_reports_hour, svc.linstor.stale_snapshots |
| Controller and satellite units | svc.linstor-controller.unit_active, svc.linstor-satellite.unit_active |
The Prometheus scrape adds svc.linstor-controller.node_state, .node_reconnects,
.resource_definitions, .pool_free_bytes, .pool_total_bytes, .pool_errors and
.error_reports with the controller's own labels. Names, tags and units are in the
metrics reference.
Logs. logs-linstor-controller and logs-linstor-satellite read the two daemons from
the systemd journal (LINSTOR has logged there, not to files, for a long time; the files
under /var/log/linstor-controller are individual error reports and are not read). Java
logs every line at the same journal priority, so the level is read from the line itself
and ERROR and WARN ship by default; the number of error reports filed is a metric of the
controller probe, and their text stays on the node for linstor error-reports show. See
Logs.
Alerts out of the box
| Rule | Fires when | Severity | Ships |
|---|---|---|---|
| DRBD disk not up to date | a local device is not UpToDate for 5 minutes | warning | on |
| DRBD resource lost quorum | a resource is without quorum for 1 minute | warning | on |
| DRBD replication link down | a link is not established for 2 minutes | warning | on |
| DRBD resync makes no progress | a resync stalled for 10 more minutes after the probe flagged it | warning | on |
| DRBD out-of-sync data on a connected link | at least 1 MiB lingers out of sync on an established link for 30 minutes | warning | on |
| DRBD link flapping | 6 or more state changes within an hour | warning | on |
| LINSTOR satellite offline | the controller sees a node offline for 2 minutes | warning | on |
| LINSTOR traffic on the failsafe netif | a node is off its default interface for 15 minutes | warning | on |
| LINSTOR storage pool almost full | 15% or less free in a pool for 10 minutes | warning | on |
| LINSTOR error reports piling up | 30 or more error reports in the last hour | warning | on |
| Leftover vzdump snapshots | at least one backup snapshot older than a day | warning | on |
| LINSTOR satellite is not running | the satellite unit is inactive for 2 minutes | warning | on |
Every rule opens one incident per resource, link, node or pool, so a single sick link on a node with twenty resources reads as one problem. The holds are sized to the thing they watch: quorum is judged in a minute because a resource without quorum cannot safely take writes, the lingering out-of-sync rule waits half an hour and ignores anything under 1 MiB because small amounts of out-of-sync data appear and clear on their own during normal operation, and the flapping rule closes only when the hourly count falls under 3. Thresholds and hysteresis are in the rules catalog.
Credentials
The LINSTOR REST API refuses requests without a token once authentication is turned on.
Create a user API token on the controller with linstor controller auth create and store it
on that host under the name the definition uses:
sudo wakora secret set linstor-controller
Paste the token at the password prompt. It is stored encrypted on the host, bound to that machine, and never sent anywhere. Until it exists the REST check reports that the controller rejected the request and names this command; the Prometheus scrape treats the same credential as optional, tries without it first, and asks for it only when the controller answers with an authentication error. See Secrets stay local.
Limitations
- DRBD metrics describe this node's view of its own resources and links. Each node needs an agent; a peer's view of the same link is reported by the peer.
- The flap count and the stalled flag live in the agent's memory and start fresh after an agent restart.
- The REST probe caps at 50 nodes, 200 storage pools and 500 snapshots; diskless pools are skipped because they have no capacity.
- Stale snapshots are recognised by
vzdumpin their name; snapshots from other tools are not judged. - The REST API is read on the loopback interface without certificate verification, since it never leaves the host. A controller that listens only on another address is not reached.
- The DRBD probe reports at most 100 resources and 300 links per node.
Related
- Monitoring Proxmox VE, where this storage usually lives
- Monitoring corosync for the membership daemon
- Proxmox monitoring on the main site