Skip to main content

Monitoring backup freshness

The agent answers one question about backups: when was the last one written. It does not guess where your backups live. One setting on the host names the path, and from then on the age of the newest write is a metric with an alert on it. borg and restic get their own definitions, matched by the installed package, with the same one-line setup.

How backups are recognised​

Three definitions, all polling every five minutes:

  • backup matches every Linux host, whatever its init system, and stays silent until you point it at a directory or file:

    sudo wakora --set backup.path=/backups

    Setting the path is what activates the check. Without it nothing is probed and nothing is reported.

  • backup-borg matches the borgbackup package. It runs borg --version for the version fact and watches the repository you name with sudo wakora --set borg.repo=....

  • backup-restic matches the restic package, runs restic version, and watches the repository named with sudo wakora --set restic.repo=....

The probe reads the file system only. It never opens the repository, so no repository key or passphrase is needed on the monitoring side, and nothing about the backup content leaves the host.

What is collected​

Age. Seconds since the newest write. For a directory that is the newest entry inside it, which is what makes a borg or restic repository read honestly: each run mutates a subdirectory, and the repository root would otherwise look untouched for years.

Existence. Whether the configured path is there at all, as a 0/1 metric for the generic path.

Facts shown on the host page: the borg or restic version.

SignalMetric
Age of the artifact at backup.pathsvc.backup.artifact.age_sec
The artifact path existssvc.backup.artifact.exists
Age of the newest write in the borg repositorysvc.borg.repo.age_sec
Age of the newest write in the restic repositorysvc.restic.repo.age_sec

Names, tags and units are in the metrics reference.

Alerts out of the box​

RuleFires whenSeverityShips
Backup is stalethe newest write is 26 hours old or morewarningon

A nightly job leaves an artifact that is never more than 24 hours old, so 26 hours is a missed run with two hours of slack for a slow one, not a job that finished late. The incident closes once a fresh write brings the age back under 25 hours. A backup on another schedule (weekly, twice a day) needs the threshold moved: it is editable in place, and any of the metrics above can drive a custom rule of its own. Details are in the rules catalog.

Pointing the agent at the backups​

The three keys live in the same wakora.conf as every other location override, written by the CLI:

sudo wakora --set backup.path=/backups
sudo wakora --set borg.repo=/srv/backups/borg
sudo wakora --set restic.repo=/srv/backups/restic

Each value takes effect on the next agent start. Keys and the file format are in Agent configuration.

Limitations​

  • Freshness is a modification time. A job that ran and wrote a broken archive looks fresh, and so does one that touched the directory without finishing. The agent does not verify the repository or its contents; borg check and restic check remain your job.
  • The path has to be visible on the host's file system. A mounted volume counts; a repository that exists only on S3 or an SFTP target does not.
  • One path per definition per host. Several backup jobs writing to several places need the age of the directory that covers them, or a heartbeat per job.
  • Age says the backup was written, not that the job ran cleanly. For positive confirmation of a run, add a cron heartbeat as the job's last step.
  • Linux hosts only, matched through the init system fact.