The Servers dashboard

The Dart servers' process and host vitals: CPU, memory, the event loop, file descriptors, page faults, container limits, filesystems, disk and network. Requests per route are on the Services board.

Who it is for: Whoever runs the Dart servers: process and host health, one service or all of them.

Open it from Dashboards in the Observatory: choose Servers. It is a built-in board, so it is read-only; Save a copy makes one you can change. How every panel reads the range, its colors and its empty states is on Reading a board.

The toolbar: the filters Service, each All until you pick a value, and the time range. Each menu lists the values process.uptime carries in the range. Each panel below names the filters that narrow it.

Overview

Process CPU

CPU seconds per second: the share of one core the process keeps. It reads as a percent of one core: 150% is a core and a half.

• How it is computed: The per-second rate of process.cpu.time from server processes only (those without flutter.build.mode; a Flutter app's process never counts here). One figure for the whole picked range.
• Better: Lower is better.
• Colors: The figure turns amber above 70% and red above 90%.
• Filters: Service narrows it.
• Opens: its title opens the Metric screen on process.cpu.time.

Process memory

Resident memory, averaged across server processes.

• How it is computed: The average of process.memory.usage from server processes only (those without flutter.build.mode; a Flutter app's process never counts here). One figure for the whole picked range.
• Filters: Service narrows it.
• Opens: its title opens the Metric screen on process.memory.usage.

Memory of container limit

The container's memory, of its cgroup limit.

• How it is computed: The average of container.memory.usage divided by the average of dartastic.container.memory.limit, as a percent. One figure for the whole picked range.
• Better: Lower is better.
• Colors: The figure turns amber above 80% and red above 90%.
• Filters: Service narrows it.
• Opens: its title opens the Metric screen on container.memory.usage.

Load average, 1 minute

Runnable tasks averaged over the last minute, averaged across hosts.

• How it is computed: The average of dartastic.system.cpu.load_average.1m. One figure for the whole picked range.
• Better: Lower is better.
• Filters: Service narrows it.
• Opens: its title opens the Metric screen on dartastic.system.cpu.load_average.1m.

Requests in flight

Active requests across the servers.

• How it is computed: http.server.active_requests from server processes only (those without flutter.build.mode; a Flutter app's process never counts here), summed. One figure for the whole picked range.
• Filters: Service narrows it.
• Opens: its title opens the Metric screen on http.server.active_requests.

Uptime

Time since the process started, from its latest report: a restart shows as a drop.

• How it is computed: The latest report of process.uptime from server processes only (those without flutter.build.mode; a Flutter app's process never counts here). One figure for the whole picked range.
• Filters: Service narrows it.
• Opens: its title opens the Metric screen on process.uptime.

CPU and memory

Process CPU over time

The share of one core, each step.

• How it is computed: The per-second rate of process.cpu.time from server processes only (those without flutter.build.mode; a Flutter app's process never counts here). Lines, one point per step across the picked range.
• Filters: Service narrows it.
• Opens: its title opens the Metric screen on process.cpu.time.

CPU time per second by mode

CPU seconds per second in user, system, iowait and idle, summed across the cores, so it passes 1 on a host with more than one core.

• How it is computed: The per-second rate of system.cpu.time from the per-CPU series only (those with a cpu.logical_number), so no CPU time counts twice, one line per cpu.mode. Lines, one point per step across the picked range.
• Filters: Service narrows it.
• Opens: its title opens the Metric screen on system.cpu.time.

Process memory over time

Resident memory, averaged across server processes, each step.

• How it is computed: The average of process.memory.usage from server processes only (those without flutter.build.mode; a Flutter app's process never counts here). Lines, one point per step across the picked range.
• Filters: Service narrows it.
• Opens: its title opens the Metric screen on process.memory.usage.

Host memory by state

Used, cached, buffers and free.

• How it is computed: system.memory.usage, summed, one line per system.memory.state. Lines, one point per step across the picked range.
• Filters: Service narrows it.
• Opens: its title opens the Metric screen on system.memory.usage.

Swap in use

Swap space in use.

• How it is computed: The average of dartastic.system.memory.swap.usage. One figure for the whole picked range.
• Better: Lower is better.
• Filters: Service narrows it.
• Opens: its title opens the Metric screen on dartastic.system.memory.swap.usage.

CPU throttling per second

Scheduler periods the container was throttled in, per second. Rising throttles mean the CPU limit is too low.

• How it is computed: The per-second rate of dartastic.container.cpu.throttled.count. Lines, one point per step across the picked range.
• Filters: Service narrows it.
• Opens: its title opens the Metric screen on dartastic.container.cpu.throttled.count.

Event loop and process

Event-loop lag (p99)

How late the Dart event loop ran a timer: a busy isolate. Amber past 100 ms, red past 250 ms.

• How it is computed: The 99th percentile of dartastic.event_loop.lag. One figure for the whole picked range.
• Better: Lower is better.
• Colors: The figure turns amber above 100 ms and red above 250 ms.
• Filters: Service narrows it.
• Opens: its title opens the Metric screen on dartastic.event_loop.lag.

Event-loop lag (p99) over time

Event-loop lag at the 99th percentile, each step. The dashed lines are 100 ms and 250 ms.

• How it is computed: The 99th percentile of dartastic.event_loop.lag. Lines, one point per step across the picked range.
• Colors: Dashed lines mark 100 ms (warn) and 250 ms (crit).
• Filters: Service narrows it.
• Opens: its title opens the Metric screen on dartastic.event_loop.lag.

File descriptors of limit

Open file descriptors, of their limit.

• How it is computed: The average of dartastic.process.open_file_descriptors divided by the average of dartastic.process.open_file_descriptors.limit, from server processes only (those without flutter.build.mode; a Flutter app's process never counts here), as a percent. One figure for the whole picked range.
• Better: Lower is better.
• Colors: The figure turns amber above 70% and red above 90%.
• Filters: Service narrows it.
• Opens: its title opens the Metric screen on dartastic.process.open_file_descriptors.

Threads

Threads, averaged across server processes.

• How it is computed: The average of dartastic.process.threads from server processes only (those without flutter.build.mode; a Flutter app's process never counts here). One figure for the whole picked range.
• Filters: Service narrows it.
• Opens: its title opens the Metric screen on dartastic.process.threads.

Page faults per second by type

Major faults read from disk; a rising rate means the process is paging.

• How it is computed: The per-second rate of dartastic.process.memory.faults from server processes only (those without flutter.build.mode; a Flutter app's process never counts here), one line per system.paging.fault.type. Lines, one point per step across the picked range.
• Filters: Service narrows it.
• Opens: its title opens the Metric screen on dartastic.process.memory.faults.

Connections by state

TCP sockets: established, listening, waiting.

• How it is computed: dartastic.system.network.connections, summed per network.connection.state. A ranked list, each figure for the whole picked range.
• Filters: Service narrows it.
• Opens: its title opens the Metric screen on dartastic.system.network.connections.

Disk and network

Filesystem usage by mount

The bytes in use on each mount.

• How it is computed: system.filesystem.usage in the used state, summed per system.filesystem.mountpoint. A ranked list, each figure for the whole picked range.
• Filters: Service narrows it.
• Opens: its title opens the Metric screen on system.filesystem.usage.

Disk bytes per second by direction

Disk reads and writes, in bytes per second.

• How it is computed: The per-second rate of system.disk.io, one line per disk.io.direction. Lines, one point per step across the picked range.
• Filters: Service narrows it.
• Opens: its title opens the Metric screen on system.disk.io.

Network bytes per second by direction

Network traffic in and out, in bytes per second.

• How it is computed: The per-second rate of system.network.io, one line per network.io.direction. Lines, one point per step across the picked range.
• Filters: Service narrows it.
• Opens: its title opens the Metric screen on system.network.io.

Load average over time, 1 minute

The one-minute load average, averaged across hosts, each step.

• How it is computed: The average of dartastic.system.cpu.load_average.1m. Lines, one point per step across the picked range.
• Filters: Service narrows it.
• Opens: its title opens the Metric screen on dartastic.system.cpu.load_average.1m.