Node Exporter Internals That Actually Matter

Node Exporter Ships 600 Metrics. You Need Maybe 30.

Node Exporter is the gateway drug of Prometheus monitoring. You install it, point Prometheus at localhost:9100, and suddenly you’ve got metrics. Hundreds of them. Thousands if you’re not careful. Which ones actually matter? Which 200 should you immediately disable before they fill your cardinality budget and make your storage team cry?

Node Exporter’s strength is also its weakness. It exposes everything about your Linux system: CPU modes, memory pressure, network stats, I/O, scheduling delays, thermal states, systemd cgroup limits, perf events, SLUB allocator details, buddy info, spontaneous memory defragmentation rates. Nobody needs spontaneous memory defragmentation rates. Seriously, that’s in the ksmd collector.

By default, Node Exporter enables most of its ~50 collectors and only disables a handful of the expensive ones (perf, ksmd, schedstat, processes, and friends). The defaults are sensible. But if you flip everything on, you’re adding cardinality like you’re hiring a forklift to move a couch; technically it works, but your storage bill has questions.

This guide cuts the noise. We’ll cover which 30 metrics actually predict problems, which collectors to disable, the textfile collector trick that lets you ship arbitrary metrics from cron jobs, and the PromQL recipes to turn raw numbers into alerts that don’t scream wolf.

The Collectors You Actually Want

The Default Collectors That Matter

A bunch of collectors are on by default. The ones you’ll actually look at:

• textfile: reads metrics from files (we’ll return to this)
• filesystem: disk capacity, usage, inodes
• meminfo: RAM, swap, memory pressure
• netdev: network interface bytes/packets
• netstat: TCP/UDP connection counts and states
• stat: CPU seconds by mode (user, system, iowait, steal, etc.)
• time: clock drift detection
• uname: kernel version, machine info

These are the foundation. Everything else is either refinement or noise.

The Ones You’ll Lean On

These are also enabled by default, and they earn their keep in any production setup, make sure you’re actually graphing them:

loadavg, the 1/5/15-minute load averages. This is your “is the system drowning” metric. One metric. Done. node_load1, node_load5, node_load15. Correlate load spikes with actual process activity. If load is high but CPU is idle, you’ve got I/O wait. That’s a disk problem, not a CPU problem.

hwmon, hardware monitoring from thermal and power sensors. If your homelab has redundant PSUs or you’re tracking CPU thermals, this is where that data lives. Useful for alerting on fan failures or thermal throttling before the hardware fails. One query: node_hwmon_chip_names to see what sensors you have.

conntrack, connection tracking stats from netfilter. If your host does NAT, runs a router, or has iptables rules, track node_nf_conntrack_entries vs. node_nf_conntrack_entries_limit. You’ll know when your conntrack table is full before the kernel starts dropping packets silently. Been there.

That’s it. A small handful of the default collectors tell you when the system is broken, the rest is noise.

The Collectors You Should Disable (the Cardinality Bombs)

Good news: Node Exporter already ships the worst offenders disabled by default. perf, ksmd, schedstat, and processes are off out of the box, leave them off. A few others are on by default that most home labs can safely kill. The disable flag is --no-collector.<name> (there’s no .enabled=false form).

perf, exposes Linux perf events (cache misses, branch mispredictions, CPU cycles). Produces a staggering number of series per label combination, and it needs special kernel privileges to even work. Disabled by default, you’re not tuning CPU pipelines on a Raspberry Pi, so keep it that way.

ksmd, kernel samepage merging metrics. This is VM-tuning territory. Unless you’re running KVM guests and monitoring deduplication, you don’t care. Disabled by default, leave it.

schedstat, scheduler statistics (runqueue delays, throttling, migrations). This is noise unless you’re debugging kernel scheduler behavior, which you’re not. Disabled by default, leave it.

processes, per-process summaries (counts, memory). This looks useful but has catastrophic cardinality on busy systems. If you need per-process metrics, use a dedicated process exporter or cgroups. Disabled by default, leave it.

rapl, RAPL (Running Average Power Limit) energy data from Intel CPUs. Cool to look at, rarely useful. This one is on by default. If you’re not tracking power, turn it off: --no-collector.rapl.

powersupplyclass, battery and power supply info. Enabled by default, but only useful on laptops or UPS-monitored systems. On a headless server, kill it: --no-collector.powersupplyclass.

btrfs, Btrfs filesystem-specific metrics. Only if you’re using Btrfs. Most people use ext4.

mountstats, NFS-specific mount statistics. Only if you have NFS mounts.

qdisc, arp, infiniband, specialty networking. You know who you are.

Trimmed Node Exporter Start Command

Here’s what a sane node-exporter command looks like:

Since the noisiest collectors (perf, ksmd, schedstat, processes) are already off by default, you mostly just point the textfile collector at a directory and switch off the few default-on collectors you don’t need:

/usr/sbin/node_exporter \
  --collector.textfile.directory=/var/lib/node_exporter/textfile_collector \
  --no-collector.rapl \
  --no-collector.powersupplyclass \
  --web.listen-address=:9100 \
  --log.level=info

loadavg, hwmon, and conntrack are all on by default, so there’s nothing to enable, they’re already there. If you’d rather start from nothing and opt in explicitly, use --collector.disable-defaults and then add each --collector.<name> you want.

If you’re running this in a container, set --path.sysfs=/host/sys --path.rootfs=/host/root to read the host’s /sys and root filesystem.

The Textfile Collector: Your Secret Weapon

Here’s where Node Exporter gets interesting. The textfile collector watches a directory for .prom files and exposes whatever metrics are inside them. This means you can ship any metric you want, custom application metrics, external service status, home-automation state, whatever.

Useful examples:

Disk full prediction (cron job every hour):

#!/bin/bash
TEXTFILE_DIR="/var/lib/node_exporter/textfile_collector"

# Predict when each filesystem fills
for mount in $(df | tail -n +2 | awk '{print $6}'); do
    used=$(df "$mount" | tail -1 | awk '{print $3}')
    avail=$(df "$mount" | tail -1 | awk '{print $4}')
    total=$((used + avail))

    if [ "$total" -gt 0 ]; then
        percent=$((used * 100 / total))
        # This is a custom metric — you own it
        echo "node_disk_fill_percent{mount=\"$mount\"} $percent"
    fi
done > "$TEXTFILE_DIR/disk.prom.tmp"

mv "$TEXTFILE_DIR/disk.prom.tmp" "$TEXTFILE_DIR/disk.prom"

Cron entry:

0 * * * * /usr/local/bin/node_exporter_disk_metrics.sh

Backup completion tracking (run after your daily backup):

#!/bin/bash
TEXTFILE_DIR="/var/lib/node_exporter/textfile_collector"

if [ $? -eq 0 ]; then
    # Backup succeeded
    timestamp=$(date +%s)
    echo "node_backup_last_success_timestamp_seconds $timestamp" > "$TEXTFILE_DIR/backup.prom"
    echo "node_backup_status 1" >> "$TEXTFILE_DIR/backup.prom"
else
    # Backup failed — set status to 0
    echo "node_backup_status 0" >> "$TEXTFILE_DIR/backup.prom"
fi

Custom application metrics (ship metrics from systemd services):

#!/bin/bash
TEXTFILE_DIR="/var/lib/node_exporter/textfile_collector"

# Get HTTP response time from your app's health endpoint
response_time=$(curl -s -w "%{time_total}" -o /dev/null http://localhost:8080/health)

echo "node_app_health_check_seconds $response_time" > "$TEXTFILE_DIR/app.prom.tmp"
mv "$TEXTFILE_DIR/app.prom.tmp" "$TEXTFILE_DIR/app.prom"

Permissions matter: The textfile directory must be readable by the node_exporter user (usually nobody or _node_exporter):

sudo mkdir -p /var/lib/node_exporter/textfile_collector
sudo chown node_exporter:node_exporter /var/lib/node_exporter/textfile_collector
sudo chmod 755 /var/lib/node_exporter/textfile_collector

The beauty of the textfile collector is that it decouples metric generation from the exporter. You can push arbitrary observability into Prometheus without modifying Node Exporter code or running separate exporters. It’s the killer feature nobody talks about.

The 30 Metrics You Actually Graph

Here are the metrics worth dashboarding:

CPU:

• rate(node_cpu_seconds_total{mode="user"}[5m]): user CPU time
• rate(node_cpu_seconds_total{mode="system"}[5m]): system CPU time
• rate(node_cpu_seconds_total{mode="iowait"}[5m]): I/O wait (disk bottleneck)
• node_load1: 1-minute load average

Memory:

• node_memory_MemTotal_bytes: total RAM
• node_memory_MemAvailable_bytes: available RAM (accounts for caches)
• 1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes): memory pressure %
• node_memory_SwapFree_bytes: swap remaining

Disk:

• node_filesystem_avail_bytes{fstype!="tmpfs"}: free space per disk
• 1 - (node_filesystem_avail_bytes / node_filesystem_size_bytes): usage % per disk
• node_filesystem_files_free{fstype!="tmpfs"}: free inodes per disk

Network:

• rate(node_network_receive_bytes_total[5m]): inbound throughput
• rate(node_network_transmit_bytes_total[5m]): outbound throughput
• rate(node_network_receive_errs_total[5m]): RX errors (bad packets)
• rate(node_network_transmit_errs_total[5m]): TX errors

I/O:

• rate(node_disk_read_bytes_total[5m]): disk read throughput
• rate(node_disk_written_bytes_total[5m]): disk write throughput
• rate(node_disk_io_time_seconds_total[5m]): total disk busy time

System:

• node_time_seconds: Unix timestamp (drift detection)
• node_load1: runqueue depth
• node_nf_conntrack_entries: active connections (if using conntrack)
• node_nf_conntrack_entries_limit: conntrack table size limit

Process counts:

• node_procs_running: processes currently running
• node_procs_blocked: processes blocked on I/O

That’s 24 metrics. Add custom app metrics from textfile and you’re at 30 max. Done.

Sane Alerts from Node Exporter Metrics

Here are the alerts that actually prevent incidents:

Disk filling too fast (predict full in <24h):

predict_linear(node_filesystem_avail_bytes{fstype!="tmpfs"}[1h], 86400) < 0

This says “if the disk continues filling at the current rate, will it be full in 24 hours?” Alert on this. Disk full is catastrophic.

Memory pressure rising (available < 15% of total):

node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes < 0.15

Watch this. When memory pressure rises, the kernel starts swapping and killing processes. 15% is generous; tighter limits (10%) catch problems earlier.

Time drift (>100ms from NTP):

abs(node_time_seconds - time()) > 0.1

Clock drift breaks distributed tracing, TLS certificate validation, and log correlation. Alert if NTP isn’t syncing properly.

Host down (no metrics for >1m):

up{job="node"} == 0

Prometheus synthesizes the up metric for every scrape target (there’s no node_up, Node Exporter doesn’t emit it). Alert when it’s been 0 for 60 seconds. If the host is down, everything else is irrelevant.

I/O saturation (disk busy >80% for 5m):

rate(node_disk_io_time_seconds_total[5m]) > 0.8

Saturated disks cause everything to slow down. Alert before it becomes a customer incident.

Network errors rising (errors per second increasing):

rate(node_network_receive_errs_total[5m]) > 1

Packet loss usually means cable/switch problems or driver bugs. Small numbers are normal; sudden spikes are not.

Swap usage (swap > 10%):

node_memory_SwapFree_bytes / node_memory_SwapTotal_bytes < 0.9

Swapping is a sign the system is overprovisioned. It also destroys performance. Alert early.

Node Exporter Doesn’t Do Everything

For metrics Node Exporter can’t provide, you’ll need specialized exporters:

• GPU metrics: NVIDIA GPU exporter, AMD ROCm exporter, Intel GPU exporter. Node Exporter has no GPU visibility.
• Application metrics: Prometheus client libraries in your app code (Go, Python, Node, etc.)
• Container metrics: cAdvisor exporter or Kubernetes kubelet endpoints (not Node Exporter’s job)
• Custom business metrics: your app owns these; use the textfile collector to ship them from cron

Node Exporter is the OS exporter. It’s excellent at that job. Don’t ask it to monitor your database. That’s DBA work.

The Metrics Worth Dashboarding

Build a dashboard with these sections:

1. System Health: load, memory %, uptime (via textfile)
2. CPU: user/system/iowait rates, context switches
3. Memory: total/used/available, swap trending
4. Disk: usage %, free space per mount, I/O latency
5. Network: inbound/outbound throughput, error rates
6. Conntrack: active connections, headroom to limit (if applicable)

Use red for anything on fire (>90%), yellow for watch-list (>70%), green for healthy. Alert on predict_linear for disk, memory pressure >15%, time drift >100ms, and >1m downtime.

That’s it. You’ve got observability without the cardinality chaos. The 30 metrics that matter. No spontaneous memory defragmentation rates required.

Related Reading

• Blackbox Exporter: HTTP/TCP/DNS/ICMP Probes
• cAdvisor + Prometheus: Per-Container Metrics Done Right
• Observability and Monitoring for Containers
• Prometheus Federation for Multi-Site Home Labs
• Push vs Pull Metrics: Pushgateway, Pushprox, and Why