Skip to main content

Logs

The agent collects logs from everything it already discovered and ships them for full-text search, with three properties designed in from the start: secrets are scrubbed before anything leaves the host, low-value noise is filtered at the source, and the volume is hard-capped so a log flood can never turn into a surprise bill.

Like everything else, it is configless: which logs exist, where they live, and how to read their levels comes from the discovered services and the signed definitions, not from files you maintain.

What is collected​

  • systemd journal, by unit identifiers (system services, sshd, cron, the services the catalog knows).
  • Service log files, at the real paths parsed from each service's config: nginx and Apache error logs, MySQL slow log, mail logs, and the rest of the catalog. Non-file targets (off, /dev/null, syslog:) are recognized and skipped. A service that rotates into a directory of dated files (PostgreSQL's log directory) is followed across the switch: the file that just closed is read to its end before the new one is read from its start, so the lines written around midnight are not lost between the two.
  • Docker containers, per container with the image as the service name.
  • Windows: Event Log entries as lines (System, Application, Hyper-V channels), IIS per-site W3C access logs, the MSSQL ERRORLOG (UTF-16 handled).
  • Kubernetes: logs of problem pods only (not Running, restarting, or crash-looping), so a healthy cluster ships nothing.
  • Network devices: syslog received by a collector flows into the same pipeline.
  • Hosts without systemd fall back to /var/log/messages or the auth log, so OpenRC and sysvinit boxes are not blind.

Secrets never leave the host​

Every line passes a redaction pass on the host, before transmission. The default patterns cover bearer tokens, API keys, passwords, cloud credentials, card-like numbers and credentials embedded in URLs; definitions can add more. What the server stores is already scrubbed. This is the same boundary as everywhere else in the agent: values that could unlock something stay on the machine.

Level filtering: signal by default​

Lines carry a level (parsed from the journal priority, the line itself, or forced per source by the definition), and by default only notice and above ships: errors, warnings, notices. info and debug (including web access logs, which are info by nature) ship only when you opt a service into deep logging. So the default stream is small and useful, and the firehose is a choice, per service.

Two honesty rules keep the levels truthful:

  • A transport-level [error] wrapping is downgraded when the embedded content says less: a PHP Warning inside an nginx stderr line is a warning, and nginx's "Primary script unknown" is a visitor's 404 (scanner noise), not a server error. The same goes for the other lines web servers and sshd log at error level about the internet's background noise: a static file a scanner asked for that does not exist (open() ... No such file or directory; a Permission denied stays an error), a client that disconnected during the SSH key exchange, and sshd's "maximum authentication attempts exceeded" and PAM "ignoring max retries" lines of a brute-force attempt that the brute-force rule already counts. All of these ship as notices. Lines are never upgraded.
  • A PHP message that nginx split over several FastCGI chunks arrives as several lines; the fragments without a marker of their own inherit the level of the line they continue, so a long warning does not become five errors.
  • Each line carries its own timestamp parsed from its content, so ordering and day-pinning stay honest even when a log was written hours before it was read. A timestamp written without a zone is read in the zone the service itself logs in when the definition knows it (PostgreSQL's log_timezone, reported by the server), and in the host's zone otherwise.

Volume protection​

Four independent guards, all on the agent:

  • Repeats fold. Consecutive lines of one service that say the same thing (numbers of three digits or more are treated as equal, addresses are not) ship as the first line plus a (repeated N times) suffix. A process stuck in a loop costs one line per cycle instead of thousands, and the count is still there.
  • A global cap of 2000 lines per minute. Past it, the lowest-severity lines are dropped first and a log_volume_capped event says so, once per window. Bill shock is impossible by construction.
  • A flood guard per file: a file growing faster than the read budget cycle after cycle is sampled instead of drained, and a log_source_flooding event names the runaway file, which is itself a signal that something on the host is sick.
  • The level filter above, which on a typical host drops the vast majority of bytes before they are ever sent.

Switching it off​

Log shipping is opt-out at three levels: a single service on a host, the whole host (the Log shipping toggle in the console's host management), or the entire team. The switch reaches the agent within about a minute and it stops reading, not just sending.

On the server, lines land in storage with a full-text index; the console searches them, tails them live, and correlates them with incidents (an incident shows the host's log window around the time it fired). Retention and search depth follow the plan. Alert rules can match log patterns (a custom rule on a pattern, and a built-in error-storm rule), so a spike of errors folds into one incident rather than an alert storm.