Skip to main content

Monitoring Kubernetes

Kubernetes support ships with the agent, not as a later add-on. The agent runs as a DaemonSet, one instance per node, or directly on a node that hosts the control plane, and reports the cluster state read-only. It does not bundle client-go and it does not shell out to kubectl: the probe talks to the API server directly. For deploying it, see the DaemonSet installation.

How the cluster is recognised​

Three signed definitions share the same probe and produce the same metrics: k8s matches the k3s-server process, k8s-rke2 matches the rke2-server.service unit and k8s-k0s matches k0scontroller.service. Each requires a recent agent version, so an older agent skips the definition with an honest log rather than failing. The probe runs every minute with a 20-second budget. Authentication is tried in this order:

  1. In-cluster ServiceAccount token (/var/run/secrets/kubernetes.io/serviceaccount), which is what the DaemonSet uses. The shipped manifest binds a read-only ServiceAccount to a ClusterRole that grants only get and list on nodes, pods, namespaces and the metrics API. No cluster-admin.
  2. A kubeconfig when the agent runs on the host rather than in a pod: /etc/rancher/k3s/k3s.yaml, /etc/rancher/rke2/rke2.yaml, /etc/kubernetes/admin.conf, /var/lib/k0s/pki/admin.conf, or an explicit path.

The kubeconfig has to carry an inline client certificate (token-based kubeconfigs are not supported), and if it has no certificate authority the probe fails closed rather than skipping server verification.

What is collected​

Read-only GETs to /version, /api/v1/nodes, /api/v1/pods and the metrics API (metrics.k8s.io, which k3s ships by default) become:

  • Cluster: nodes total and ready, plus the ready share as a percentage computed on the host so that one number can drive a rule; pods total and the phase split into running, pending, failed and crash-looping; container restarts across the cluster; namespaces.
  • Per namespace: pod count, CPU and memory, tagged by namespace, capped at 50 namespaces ordered by pod count.
  • Top pods: CPU and memory of the 20 busiest pods by CPU, tagged by namespace and pod.
  • Inventory: each namespace as a fact on the host page.

CPU quantities (n, u, m) and memory quantities (Ki, Mi, Gi, k, M, G) are parsed to millicores and bytes. Facts shown on the host page: the API server version and the kubelet version.

SignalMetric
Nodes total / ready / ready sharesvc.k8s.nodes, svc.k8s.nodes_ready, svc.k8s.nodes_ready_pct
Pods total and by phasesvc.k8s.pods, svc.k8s.pods_running, svc.k8s.pods_pending, svc.k8s.pods_failed, svc.k8s.pods_crashloop
Container restarts, namespacessvc.k8s.restarts_total, svc.k8s.namespaces
Per-namespace usagesvc.k8s.ns.pods, svc.k8s.ns.cpu_millicores, svc.k8s.ns.mem_bytes
Top podssvc.k8s.pod.cpu_millicores, svc.k8s.pod.mem_bytes

Names, tags and units are in the metrics reference.

The per-namespace and per-pod CPU and memory numbers come from the metrics-server. If the cluster has no metrics-server, the cluster and pod-phase counts still work; only the resource usage numbers are absent.

Logs. The matching logs-k8s definitions ship logs of problem pods only: pods that are not Running, are restarting, or are crash-looping, up to ten at a time, the last fifty lines since the previous read, each line prefixed with namespace/pod. Lines ship at notice level and above by default. A healthy cluster contributes no log volume, and a broken pod arrives with its own evidence. See Logs.

Alerts out of the box​

RuleFires whenSeverityShips
Pods in CrashLoopBackOffat least one pod crash-looping for 5 minuteswarningauto
Kubernetes nodes not readythe ready share is at or below 99% (a node is not Ready) for 5 minuteswarningon
Kubernetes pods cannot startat least one pod Pending for 15 minuteswarningon
Kubernetes pods in failed stateat least one pod Failed for 15 minuteswarningon

Crash-loop arms itself as soon as the cluster metrics flow. Five minutes on a not-ready node outlasts a reboot or a kubelet restart and still catches a node that stays away. Pending is held for a quarter of an hour because a deployment routinely leaves pods pending for a few minutes while images pull; fifteen minutes means the scheduler has nowhere to place them, be it resources, an image or a volume. The node rule closes only when every node is Ready again. Thresholds and hysteresis are in the rules catalog.

Limitations​

  • Per-namespace and per-pod usage needs the metrics-server; without it those metrics are simply absent.
  • kubeadm clusters are not matched today: the API server runs as a static pod, so no host process or unit identifies it. See the note below.
  • Lists are capped at 50 namespaces and 20 pods, and the pod listing reads up to 5,000 pods.
  • Deployments, services, persistent volumes and per-container metrics are not collected.
  • A kubeconfig that authenticates with a token instead of a client certificate cannot be used; run the agent in-cluster instead.
Planned

A hands-off kubectl apply needs a container image registry, which is the remaining piece. The probe, the read-only RBAC and the DaemonSet manifest are ready today. Group operations across nodes (a Kubernetes operator) and matching kubeadm clusters are planned.