Skip to main content

Containers and LXC - what is honestly unavailable

The agent runs inside LXC containers, Docker containers and Kubernetes pods like on any host. A few things depend on what the kernel lets a container do, and this page names them so a missing card is not mistaken for a broken agent.

Unprivileged LXC​

The default container type on Proxmox and plain LXC. The agent installs and monitors everything that reads files, sockets and /proc: host metrics, discovery, every service probe, logs, web sites, databases, mail. Two things need kernel privilege the container does not have:

  • eBPF (APM layer 1, request rate, errors and latency on the wire). The kernel refuses it; the agent detects that and takes request rates from the web server's access log instead. The host page shows the capability as unavailable, not as failed.
  • Live process sampling (CPU flame graphs of PHP-FPM workers, process_vm_readv). Refused by default ptrace_scope inside the container. Method-level traces (APM layer 2) and everything else in APM still work, because they run inside the application, not from the outside.

A privileged LXC container has both, and the agent uses them.

Load, CPU and I/O inside a container​

The kernel does not keep a per-container load average: /proc/loadavg inside an LXC container is the node's. The agent therefore computes the container's own load from its own runnable and waiting tasks, sampled every five seconds with the kernel's decay, so host.load1 and the per-core load rule describe this container and nothing else. Disk I/O and pressure come from the container's own accounting as well (host.io.*, host.cpu.pressure_pct, host.io.pressure_pct), and the core count follows the cores the container may actually run on, re-read every minute.

One number stays the node's, and the agent says so rather than guessing: host.cpu.used_pct inside an LXC container reflects the physical cores the container is pinned to, as the container runtime presents them, so a busy neighbour on the same cores shows up in your container's CPU percentage. Read host.cpu.pressure_pct for whether your processes waited for a CPU. On a VM or a bare-metal host every one of these numbers is the machine's own.

Memory limits and the agent​

Do not put a cgroup memory limit near the agent's working set (a MemoryHigh or MemoryMax of a few hundred megabytes on its unit, or a small container limit for a containerised agent). The limit does not bound the agent's own memory, which is tiny; it bounds the page cache of every file the agent and its helper commands read, and when the cache is evicted every probe cycle turns into disk reads. The shipped unit sets no memory limit for this reason and lifts one it finds; the agent's own heap is capped in-process. Details in Service management.

Journals inside containers​

A container whose journal is the size of its RAM (a 4 GB journal in a 4 GB container) makes every journal read evict everything else from the cache. If the agent shows high disk reads inside a container, check du -sh /var/log/journal against the container's memory and vacuum the journal (journalctl --vacuum-size=256M, plus SystemMaxUse in journald.conf).

What a Proxmox node sees of its guests​

An agent on a Proxmox node sees every guest's processes in /proc, and deliberately ignores them: processes inside LXC guests and containers are filtered out of host discovery by their cgroup, so a guest's MySQL never matches the node's MySQL definition. Guests are monitored by their own agents; the node contributes the hypervisor view (guest state, storage, cluster health) through Monitoring Proxmox VE.

Docker and Kubernetes​

  • Processes inside other containers are excluded from host discovery the same way; the Docker probe covers them grouped by image, and the Kubernetes probe covers pods.
  • An agent that runs as a container needs --pid=host (or hostPID in the DaemonSet) to see the host's processes and ports, --network=host for the host's addresses, and SYS_PTRACE for the profiling layer. Without them it monitors its own container, which is rarely what you want.
  • Applications in containers export traces to the agent over the container bridge address; see Docker.