Skip to main content

Monitoring ClickHouse

ClickHouse is recognised by its installed package on a systemd host, and every probe talks to the server the way a local client would: the native port, the HTTP port, systemctl and the server's own configuration file. Nothing to install, nothing to enable in the server.

How ClickHouse is recognised​

The signed definition matches a host that has the clickhouse-server package installed and runs systemd. Matching on the package rather than the process is deliberate: a definition matched on a running process disappears when the process dies, whereas this one keeps reporting the unit state while the server is down, which is what lets the down rule fire. Every minute the agent:

  • opens a TCP connection to the native port, 127.0.0.1:9000;
  • runs systemctl is-active clickhouse-server and turns the answer into a 0/1 metric;
  • calls /ping on 127.0.0.1:8123 expecting a 200, asks SELECT version() for the version fact, and runs one query over system.metrics and system.asynchronous_metrics for the statistics below;
  • reads /etc/clickhouse-server/config.xml for the <log> and <errorlog> paths.

The version and statistics queries authenticate with an optional credential named clickhouse. Without it they go to the default user without a password, which is what a fresh install has; a server where default carries a password, or a dedicated read-only monitoring user, is covered by storing that user once on the host:

sudo wakora secret set clickhouse --user monitor

The credential never leaves the host. A companion definition for logs runs every 30 seconds.

What is collected​

Liveness. Whether the unit is active, and the server's uptime.

Load. Queries in flight, client connections on the native and HTTP interfaces, and tracked memory use.

Storage health. The worst part count across partitions, which is how you see merges falling behind inserts, and the worst replica delay in seconds, which is 0 on a server without replication. Plus totals: databases, tables, rows and bytes across MergeTree tables.

Facts shown on the host page: version, main log path, error log path.

SignalMetric
Unit activesvc.clickhouse.unit_active
Queries in flightsvc.clickhouse.running_queries
Native / HTTP connectionssvc.clickhouse.tcp_connections, svc.clickhouse.http_connections
Tracked memorysvc.clickhouse.memory_bytes
Worst parts per partitionsvc.clickhouse.max_parts_per_partition
Worst replica delaysvc.clickhouse.replica_max_delay_sec
Databases and tablessvc.clickhouse.databases, svc.clickhouse.tables
Rows and bytes in MergeTree tablessvc.clickhouse.rows_total, svc.clickhouse.bytes_total
Uptimesvc.clickhouse.uptime_sec

Names and units are in the metrics reference.

Logs. The error log ships by default at notice level and above, with the level read from the <Error>-style tag the server prints on each line. The main server log is information-level by nature and is collected only when you switch the service to full depth in the console. Both paths come from config.xml, with the stock paths under /var/log/clickhouse-server/ as the fallback. See Logs.

Alerts out of the box​

RuleFires whenSeverityShips
ClickHouse is not runningthe unit reports inactive for 5 minuteswarningauto
Parts piling up (merges lagging)300 or more parts in one partition for 10 minuteswarningauto
Replica laggingthe worst replica is 300 seconds or more behind for 10 minuteswarningauto

All three arm themselves the moment the metric starts flowing for a workspace. The down rule works because the unit state comes from the package-matched definition, which outlives the process. Three hundred parts in a single partition, held for ten minutes, means background merges have fallen behind inserts for longer than any burst; the incident clears when the count is back under 200. Replica delay is 0 without replication, so the third rule never fires on a standalone server, and it clears once the lag is under 60 seconds. Thresholds and hysteresis are in the rules catalog.

When a path is not where ClickHouse says it is​

Both log paths come from config.xml. To point the agent elsewhere:

sudo wakora --set clickhouse.err-log-path=/data/logs/clickhouse-server.err.log

The main log takes clickhouse.log-path the same way. Keys are listed in Agent configuration.

Limitations​

  • The definition needs the package and systemd. A server started from a tarball, or inside a container without either, does not match.
  • The probes talk to 127.0.0.1:8123 (and the native check to 127.0.0.1:9000), as the default user without a password unless the optional clickhouse credential is stored. A server listening elsewhere is not covered by the stock definition.
  • Everything is server-wide: no per-table, per-query or per-user figures, and nothing is read from system.query_log.
  • The main log ships only at full depth.
  • The replica-delay rule is inert on a server without replication.