Dashboard & Metrics Reference¶

The RAMJET Console dashboard surfaces 27 time-series metrics emitted by every cache-server node via gRPC heartbeats. They are persisted in the backend metrics table (Postgres) and rendered in two places:

Cluster Detail page — live cards (CacheMetrics, Training Banner, Cluster Throughput / Avg Latency sparklines) read the latest heartbeat.
Grafana (https://grafana.ramjet.io) — historical panels read the metrics table directly via the Postgres datasource.

This page lists every metric, what it represents, and how its semantics interact with the "This run / All time" toggle on Cluster Detail.

Metric families¶

1. Cumulative gauges (process-lifetime counters)¶

These grow monotonically until the cache-server process restarts. Rates are derived in Grafana via rate() / delta over time(). The "This run / All time" toggle subtracts a session baseline (snapshot taken when training_started_at flips) and renders current − baseline.

Metric	Meaning
`total_hits`	Successful cache reads (any source).
`total_misses`	Reads that fell through to S3/origin.
`cache_hits_local`	Hits served from the local node's disk cache.
`cache_hits_peer`	Hits served from a peer node over HTTP.
`cache_hits_s3`	Reads where this node fetched from S3/origin (peer reports a hit here if it pulled from us; this counter reflects origin pulls owned by this node).
`ingress_bytes_local`	Bytes read off local disk.
`ingress_bytes_peer`	Bytes pulled from peers.
`ingress_bytes_s3`	Bytes pulled from S3/origin.
`singleflight_leaders`	Distinct keys this node fetched as the singleflight leader.
`singleflight_wait_hits`	Successful waits (a peer/local request piggy-backed on an in-flight leader).
`singleflight_wait_timeouts`	Waits that timed out before the leader finished. Operationally actionable — non-zero means the cluster is shedding deduplication.

2. Point-in-time gauges (latest snapshot)¶

These overwrite each heartbeat. The "This run / All time" toggle is a no-op on these — you always see the current value.

Metric	Meaning
`cache_size_bytes`	Bytes currently held in the local cache.
`cache_items_count`	Number of cache entries.
`disk_used_bytes`	Disk used on the cache-server's storage volume.
`gpu_memory_used_bytes`	GPU VRAM used on the host (if any).
`gpu_memory_total_bytes`	Total GPU VRAM.
`ram_used_bytes`	Process RAM.
`ram_total_bytes`	Host RAM.

3. Windowed metrics (sliding window on the cache server)¶

These are computed over a sliding window inside the cache server (typically the last few seconds) and emitted on heartbeat. The toggle is a no-op.

Metric	Meaning
`throughput_mbps`	Egress throughput from this node.
`avg_latency_ms`	Mean GET latency.
`p50_latency_ms`	Median GET latency (Phase Q.4, v0.9.2+).
`p99_latency_ms`	Tail GET latency (Phase Q.4, v0.9.2+). Watch this — the mean hides the tail.

4. Training-only metrics (reset on new run)¶

Reported by the SDK on the trainer side. They reset to 0 / empty when the trainer starts a new run.

Metric	Meaning
`training_loss`	Latest scalar loss reported by the trainer.
`epoch_progress`	Fraction of the current epoch completed (0..1).
`samples_per_second`	GPU-side samples/sec (forward+backward, not data plane).

5. Derived¶

Metric	Formula
`cache_hit_ratio`	`total_hits / (total_hits + total_misses)`. Stored as a separate gauge so Grafana doesn't have to recompute it.

Retention¶

The metrics table retains 7 days of samples. Cleanup runs hourly (cleanup_old_metrics() in backend/src/services/node.py). Heartbeats themselves are stored at most every METRIC_STORE_INTERVAL_SECONDS (5s by default) per node to keep the table size bounded.

If you need longer retention, snapshot the table to S3 / a warehouse on a schedule — RAMJET does not ship with a built-in tiered store.

"This run / All time" toggle¶

Cluster Detail has a header toggle (info icon explains it on hover):

All time — raw counters straight from the latest heartbeat.
This run — current − baseline, where baseline is captured the moment the trainer's training_started_at flips on this node.

Cards whose value responds to this toggle carry a small ⊙ marker in the corner. Live gauges (cluster throughput, p50/p99 latency, cache size, GPU/RAM, training loss) ignore the toggle — they always show the current state.

The baseline is persisted server-side (migration 011_add_session_baselines.sql, 13 nullable columns on nodes) so the toggle survives page reloads, multi-tab use, and DDP rank rollover. If the server hasn't snapshotted yet, the frontend falls back to a localStorage baseline. A manual ↻ Reset button re-snapshots on demand without bouncing the trainer.

The toggle does not mutate counters in the DB or in Grafana — Grafana panels always show absolute values.

Grafana panels¶

Provisioned dashboard JSON: backend/grafana/dashboards/cluster-metrics.json.

Panel	Metrics
1. Cache Hit Ratio	`cache_hit_ratio`
2. Cluster Throughput	`throughput_mbps` (sum across nodes)
3. Cache Size	`cache_size_bytes`
4. GPU Memory	`gpu_memory_used_bytes` / `gpu_memory_total_bytes`
5. Data Pipeline Rate	derived rates of `ingress_bytes_local\|peer\|s3` (Grafana-only — Cluster Detail's Data Source Mix card is the live equivalent)
6. Cache Latency p50 / p99	`p50_latency_ms` (dashed blue), `p99_latency_ms` (solid red)
7. Training Loss	`training_loss` per node, linear, filtered to `metric_value > 0`

The Cluster Detail Analytics section embeds panels 1, 2, 3, 4, 6, 7 (panel 5 lives only in Grafana to avoid duplicating the in-page Data Source Mix card).