Logs
What works today
The agent can tail logs as part of a service probe. The logtail primitive follows a log
from a saved offset, handles rotation, reads only new bytes, and counts patterns into
per-second rate metrics. The path comes from the discovered service (the real path parsed
from the config, not a default), so a log in a non-standard location is found.
This already powers two things:
- Service rate metrics, for example mail sent, deferred and bounced rates from the mail log, or SSH auth-failure rate.
- RED metrics from access logs. On a host where eBPF is not available, request rate,
errors and duration come from the web server access log (
$request_time), which is the zero-touch fallback for APM layer 1. See APM.
Definitions decide which logs to read and how to parse them (regex counters), and they are signed and pushed from the server, so a new format needs no agent update.
Full log management
A full log management module is planned (roadmap M17): full collection (tail, journald, files, containers), storage in ClickHouse with full-text search, retention per tenant by tariff, and correlation of logs with metrics and incidents on one incident timeline. Pattern-based alerting on logs (a spike of errors folded into a single incident rather than an alert storm) comes with the alerting engine. Volume discipline stays on the agent (filter and rate-limit at the source), not by giving up completeness.
Today the agent does the tailing and rate-counting described above. The full-text search, per-tariff retention, and correlation are the planned parts.