Dartastic Help

From your first sign-in to answers about your app in production.

These pages show Cloud's URLs. Sign in on Hosted and they show your server's.

The Dart Server on Linux dashboard

One Dart server at a glance: CPU, memory, disk and load as gauges; its requests, errors and latency; the event loop; and the host and container underneath.

Who it is for: Whoever runs a Dart server: one server's health, from its requests down to the host.

Open it from Dashboards in the Observatory: choose Dart Server on Linux. It is a built-in board, so it is read-only; Save a copy makes one you can change. How every panel reads the range, its colors and its empty states is on Reading a board.

The toolbar: the filters Service, each All until you pick a value, and the time range. Each menu lists the values http.server.request.duration carries in the range. Each panel below names the filters that narrow it.

[TODO: screenshot of the Dart Server on Linux dashboard over the last 6 hours with one service picked: the four gauges, then Service: rate, errors, duration]

At a glance

CPU busy

The host's non-idle share of CPU time. Investigate sustained load past 85%.

  • How it is computed: 100 × (1 − idle CPU time / all CPU time), from the per-second rates of system.cpu.time with cpu.mode idle and in every mode, from the per-CPU series only (those with a cpu.logical_number), so no CPU time counts twice. A dial from 0 to 100%, one figure for the whole picked range.
  • Colors: The dial turns amber above 85% and red above 95%.
  • Filters: Service narrows it.
  • Opens: its title opens the Metric screen on system.cpu.time.

Container memory

The container's usage against its limit: the OOM-kill line.

  • How it is computed: The average of container.memory.usage divided by the average of dartastic.container.memory.limit, as a percent. A dial from 0 to 100%, one figure for the whole picked range.
  • Colors: The dial turns amber above 80% and red above 90%.
  • Filters: Service narrows it.
  • Opens: its title opens the Metric screen on container.memory.usage.

Disk used

Filesystem usage on the root mount (/).

  • How it is computed: system.filesystem.usage on mount point / in the used state, summed, divided by all of that mount's system.filesystem.usage, summed, as a percent. A dial from 0 to 100%, one figure for the whole picked range.
  • Colors: The dial turns amber above 75% and red above 90%.
  • Filters: Service narrows it.
  • Opens: its title opens the Metric screen on system.filesystem.usage.

Load per core

The one-minute load against the core count: 100% is one runnable task per core. Sustained past 85% means queuing.

  • How it is computed: The average of dartastic.system.cpu.load_average.1m, divided by the number of distinct cpu.logical_number values on system.cpu.time (the cores; the summed series carries none), as a percent. A dial from 0 to 100%, one figure for the whole picked range.
  • Colors: The dial turns amber above 85% and red above 95%.
  • Filters: Service narrows it.
  • Opens: its title opens the Metric screen on dartastic.system.cpu.load_average.1m.

Requests in flight

Active requests.

  • How it is computed: http.server.active_requests from server processes only (those without flutter.build.mode; a Flutter app's process never counts here), summed. One figure for the whole picked range.
  • Filters: Service narrows it.
  • Opens: its title opens the Metric screen on http.server.active_requests.

Uptime

Time since the process started, from its latest report: a restart shows as a drop.

  • How it is computed: The latest report of process.uptime from server processes only (those without flutter.build.mode; a Flutter app's process never counts here). One figure for the whole picked range.
  • Filters: Service narrows it.
  • Opens: its title opens the Metric screen on process.uptime.

Service: rate, errors, duration

Requests per second by route

Requests per second for each route.

  • How it is computed: The per-second rate of http.server.request.duration samples, one line per http.route. Lines, one point per step across the picked range.
  • Filters: Service narrows it.
  • Opens: its title opens the Metric screen on http.server.request.duration.

Error rate

Requests answered 5xx, of all requests; 4xx codes are the caller's and left out. Amber at 1%, red at 5%.

  • How it is computed: Requests (http.server.request.duration samples) whose http.response.status_code is 500 or higher, divided by all requests, as a percent. Lines, one point per step across the picked range.
  • Colors: Dashed lines mark 1% (warn) and 5% (crit).
  • Filters: Service narrows it.
  • Opens: its title opens the Metric screen on http.server.request.duration.

Latency p50, p95 and p99

Request duration across all routes.

  • How it is computed: The 50th, 95th and 99th percentiles of http.server.request.duration, one line each. Lines, one point per step across the picked range.
  • Filters: Service narrows it.
  • Opens: its title opens the Metric screen on http.server.request.duration.

Slowest routes (p95)

Where the latency budget goes.

  • How it is computed: The 95th percentile of http.server.request.duration per http.route, the top six. Ranked bars, each figure for the whole picked range.
  • Colors: A bar turns amber above 250 ms and red above 1 s.
  • Filters: Service narrows it.
  • Opens: its title opens the Metric screen on http.server.request.duration.

Top routes by traffic

Requests in the picked range, per route.

  • How it is computed: The number of http.server.request.duration samples per http.route, the top six. Ranked bars, each figure for the whole picked range.
  • Filters: Service narrows it.
  • Opens: its title opens the Metric screen on http.server.request.duration.

Dart VM: the event loop

Event-loop lag (p99)

How late the Dart event loop ran a timer: a busy isolate. Amber past 100 ms, red past 250 ms.

  • How it is computed: The 99th percentile of dartastic.event_loop.lag. One figure for the whole picked range.
  • Better: Lower is better.
  • Colors: The figure turns amber above 100 ms and red above 250 ms.
  • Filters: Service narrows it.
  • Opens: its title opens the Metric screen on dartastic.event_loop.lag.

Event-loop lag p50 and p99

The queue forming and draining over time. The dashed lines are 100 ms and 250 ms.

  • How it is computed: The 50th and 99th percentiles of dartastic.event_loop.lag, one line each. Lines, one point per step across the picked range.
  • Colors: Dashed lines mark 100 ms (warn) and 250 ms (crit).
  • Filters: Service narrows it.
  • Opens: its title opens the Metric screen on dartastic.event_loop.lag.

Host and container

CPU time per second by mode

CPU seconds per second in user, system, iowait and idle, summed across the cores, so it passes 1 on a host with more than one core.

  • How it is computed: The per-second rate of system.cpu.time from the per-CPU series only (those with a cpu.logical_number), so no CPU time counts twice, one line per cpu.mode. Lines, one point per step across the picked range.
  • Filters: Service narrows it.
  • Opens: its title opens the Metric screen on system.cpu.time.

Container CPU throttles per minute

The container hitting its CPU limit.

  • How it is computed: The per-minute rate of dartastic.container.cpu.throttled.count. Lines, one point per step across the picked range.
  • Colors: Dashed lines mark 1 (warn) and 10 (crit).
  • Filters: Service narrows it.
  • Opens: its title opens the Metric screen on dartastic.container.cpu.throttled.count.

Memory against the container limit

Process RSS against the container ceiling.

  • How it is computed: Three lines: the average of process.memory.usage from server processes only (those without flutter.build.mode; a Flutter app's process never counts here), and the averages of container.memory.usage and dartastic.container.memory.limit. Lines, one point per step across the picked range.
  • Filters: Service narrows it.
  • Opens: its title opens the Metric screen on process.memory.usage.

Established connections

Open TCP connections in the established state.

  • How it is computed: dartastic.system.network.connections with network.connection.state established, summed. Lines, one point per step across the picked range.
  • Filters: Service narrows it.
  • Opens: its title opens the Metric screen on dartastic.system.network.connections.

Open file descriptors against the limit

Open file descriptors next to their limit.

  • How it is computed: Two lines, from server processes only (those without flutter.build.mode; a Flutter app's process never counts here): the highest dartastic.process.open_file_descriptors and the highest dartastic.process.open_file_descriptors.limit. Lines, one point per step across the picked range.
  • Filters: Service narrows it.
  • Opens: its title opens the Metric screen on dartastic.process.open_file_descriptors.

Threads, and major page faults per second

OS-level process pressure.

  • How it is computed: Two lines, from server processes only (those without flutter.build.mode; a Flutter app's process never counts here): the highest dartastic.process.threads, and the per-second rate of dartastic.process.memory.faults with system.paging.fault.type major. Lines, one point per step across the picked range.
  • Filters: Service narrows it.
  • Opens: its title opens the Metric screen on dartastic.process.threads.