What Prometheus Sees: Monitoring 10 Branches from One Grafana

The unified-infrastructure case lists “Prometheus and Grafana observability” in its summary, and the about page leans on the uptime that observability underwrites. But no post ever showed the layer itself: what gets scraped, what the dashboard looks like, what fires. This is that post. The architecture as committed, the one live transcript the repo carries, and the blind spot carried openly.

What gets scraped

The observability row in the case’s architecture table is one line with three nouns: “Scrapes latency, CPU, and RDS session metrics via WMI exporters; webhook alerts on thresholds.” Unpacked:

• Latency and packet health per branch link across the international mesh (the Grafana dashboard in the case tracks sub-millisecond latency and UDP packet health).
• CPU on the ERP host, the signal that peak retail hours are saturating the box before users feel it.
• RDS session counts, feeding the connection-broker timeout policies that manage concurrent Tally Virtual User licenses through peak hours.

The scrape-health transcript the case commits is the up{job="tailscale"} query returning 10/10 targets up. Every branch node answering its exporter. That query is the heartbeat of the whole layer: if a branch drops off the mesh, it drops off this vector first.

What the dashboard shows

The case embeds one screenshot: a custom Grafana dashboard tracking latency and UDP packet health across the branch network. Day to day, that board is the proof the mesh is healthy. The case’s operations section says so directly: “Grafana as the day-to-day proof the mesh is healthy.”

The cost line matters here because it is the reason the layer exists at all: Prometheus and Grafana are open source on an existing Linux host, so “the whole observability layer costs effectively nothing beyond the box it runs on.” Monitoring that costs nothing is monitoring nobody proposes deleting.

What fires

Two paths, stated in the case’s what-breaks list:

1. Tunnel latency and packet loss. Grafana webhook alerts fire when thresholds exceed. The mesh’s failure mode is degradation across international links, and the alert is on the degradation, not on a binary up/down.
2. RDS session exhaustion at peak hours. Handled by connection-broker timeout policies, not by paging a human. Custom session-timeout policies manage the concurrent TVU licenses; the policy is the alert response, pre-written.

Grounded vs omitted

Grounded in committed sources: the scrape-health transcript and WMI-exporter architecture (unified-infra case), the dashboard screenshot and caption (same case), the alert paths (same case, §4 Operations), the cost line (same case). Omitted deliberately: any incident the monitoring caught. No alert transcript, no “the page that saved the quarter” exists in the repo, and inventing one would be fabrication. The 10-years post Lesson 4 even shows the older shape of this layer (a PowerShell disk-usage check with a Slack webhook), which is where the estate started before Prometheus arrived.

Honest limitations

• No incident receipt. This post evidences that the layer exists and what it watches, not that it ever caught anything. A monitoring post without a caught incident is architecture documentation, and should be read as such.
• The single-export-path blind spot. The case’s own retrospective: if the monitoring host itself lost the tunnel, history gaps exactly when forensics needs it most. No store-and-forward buffer, no remote-write replica. Named, unbuilt.
• Phase-1 record. The scrape query targets the Tailscale mesh job; the estate has since moved to Twingate ZTNA (RDS migration case). The up query’s job label belongs to the era this case documents. The shape transfers, the exact query does not.

A dashboard nobody screenshots is a dashboard nobody trusts; this one is now on the record, blind spot and all. It is the answer to “what else has he run?” after the flagship two.