The Server Health dashboard

The servers under the services: load against cores, memory and descriptor headroom, socket states, I/O wait, page faults, disk, CPU throttling, and per-CPU busy. Each service names its host, so Service picks a server.

Who it is for: Whoever keeps the servers up: capacity, headroom and pressure.

Open it from Dashboards in the Observatory: choose Server Health. It is a built-in board, so it is read-only; Save a copy makes one you can change. How every panel reads the range, its colors and its empty states is on Reading a board.

The toolbar: the filters Service, each All until you pick a value, and the time range. Each menu lists the values system.cpu.time carries in the range. Each panel below names the filters that narrow it.

Load and headroom

System load (1 minute) and cores by service

The busiest host's load. Above the core count is saturation; the cores line is the live count.

• How it is computed: Two lines per service: the highest dartastic.system.cpu.load_average.1m of its hosts, and the number of distinct cpu.logical_number values on system.cpu.time (the cores; the summed series carries none). Lines, one point per step across the picked range.
• Filters: Service narrows it.
• Opens: its title opens the Metric screen on dartastic.system.cpu.load_average.1m.

Container memory used

Usage against the cgroup limit, not process RSS alone. Keep it under 80%.

• How it is computed: The highest container.memory.usage divided by the highest dartastic.container.memory.limit, shown as a percent. A dial from 0 to 100%, one figure for the whole picked range.
• Colors: The dial turns amber above 80% and red above 95%.
• Filters: Service narrows it.
• Opens: its title opens the Metric screen on container.memory.usage.

File descriptors used

"Too many open files" crashes servers. Keep it under 70% of the limit.

• How it is computed: The highest process.unix.file_descriptor.count divided by the highest dartastic.process.open_file_descriptors.limit, from server processes only (those without flutter.build.mode; a Flutter app's process never counts here), shown as a percent. A dial from 0 to 100%, one figure for the whole picked range.
• Colors: The dial turns amber above 70% and red above 90%.
• Filters: Service narrows it.
• Opens: its title opens the Metric screen on process.unix.file_descriptor.count.

Network and I/O wait

TCP sockets by service and state

A time_wait pileup is ephemeral-port exhaustion under outbound-heavy load.

• How it is computed: system.network.connection.count, summed, one line per service and network.connection.state, the top twelve. Lines, one point per step across the picked range.
• Filters: Service narrows it.
• Opens: its title opens the Metric screen on system.network.connection.count.

I/O wait per second by service

CPU time spent waiting on disk: tells disk-bound from CPU-bound bottlenecks.

• How it is computed: The per-second rate of system.cpu.time with cpu.mode iowait, from the per-CPU series only (those with a cpu.logical_number), so no CPU time counts twice, one line per service. Lines, one point per step across the picked range.
• Filters: Service narrows it.
• Opens: its title opens the Metric screen on system.cpu.time.

Memory, disk and CPU pressure

Major page faults per second by service

Non-zero is real memory pressure: pages coming back from disk.

• How it is computed: The per-second rate of process.paging.faults with system.paging.fault.type major, from server processes only (those without flutter.build.mode; a Flutter app's process never counts here), one line per service. Lines, one point per step across the picked range.
• Filters: Service narrows it.
• Opens: its title opens the Metric screen on process.paging.faults.

Root filesystem used

Disk full is service down. Keep it under 80%.

• How it is computed: system.filesystem.usage on mount point / in the used state, summed, divided by all of that mount's system.filesystem.usage, summed, shown as a percent. A dial from 0 to 100%, one figure for the whole picked range.
• Colors: The dial turns amber above 80% and red above 90%.
• Filters: Service narrows it.
• Opens: its title opens the Metric screen on system.filesystem.usage.

Container CPU throttles per second by service

Non-zero means the cgroup CPU limit is the bottleneck: raise it.

• How it is computed: The per-second rate of dartastic.container.cpu.throttled.count, one line per service. Lines, one point per step across the picked range.
• Filters: Service narrows it.
• Opens: its title opens the Metric screen on dartastic.container.cpu.throttled.count.

Processes and per-CPU

Process CPU time per second by service and mode

CPU time each service's process uses, by mode.

• How it is computed: The per-second rate of process.cpu.time from server processes only (those without flutter.build.mode; a Flutter app's process never counts here), one line per service and cpu.mode. Lines, one point per step across the picked range.
• Filters: Service narrows it.
• Opens: its title opens the Metric screen on process.cpu.time.

Process uptime by service

Time since each process started: a low one is a recent restart.

• How it is computed: Uptime: the latest report of process.uptime per service, from server processes only (those without flutter.build.mode; a Flutter app's process never counts here), the top twenty. A table, every figure for the whole picked range.
• Filters: Service narrows it.
• Opens: its title opens the Metric screen on process.uptime.

Busy share of each CPU by service

Each line one CPU of one host: the share of its time outside idle, 0 to 1. A line near 1 is a saturated CPU.

• How it is computed: The per-second rate of system.cpu.time in every mode but idle, from the per-CPU series only (those with a cpu.logical_number), so no CPU time counts twice, one line per service and cpu.logical_number, the top sixteen. Lines, one point per step across the picked range.
• Filters: Service narrows it.
• Opens: its title opens the Metric screen on system.cpu.time.