NVIDIA has released a practical observability framework for AI factories, mapping the telemetry tools needed to monitor every layer of DGX and HGX infrastructure — from power supplies and GPU health to InfiniBand fabric, job scheduling, and inference services. The guide, authored by Jorge Cardoso and published on the NVIDIA Technical Blog on August 12, 2026, addresses a recurring operational problem: symptoms at one layer often originate elsewhere, and teams drown in metrics without a clear path to root cause.
The framework identifies five failure domains that silently consume GPU hours: platform health (fans, PSUs, BMC, chassis), GPU health and performance (utilization, temperature, power, XID/ECC errors, NVLink throughput), fabric integrity (InfiniBand or Ethernet link health, congestion, switch and cable status), cluster and job scheduling (reservations, idle allocated GPUs, queue wait times), and inference services (latency, success rate, cache behavior for NIM microservices). Coverage gaps in these domains tend to appear only under load, the post notes, so analyzing latent failure modes early shortens discovery.
What's New
A decision matrix maps seven NVIDIA telemetry tools to the infrastructure components they cover, using green for full support and yellow for partial or indirect coverage. The tools are Redfish/IPMI, DCGM (Data Center GPU Manager), NVSM (NVIDIA System Management), UFM (Unified Fabric Manager), NetQ, NMX, BCM (Base Command Manager), Run:ai, and NIM. Key tradeoffs include preferring DCGM for GPU utilization, power, temperature, NVLink, and XID/ECC export into Prometheus while retaining NVSM for system health on DGX-class nodes; selecting UFM for InfiniBand and NetQ for Spectrum Ethernet/RoCE, deploying both only when both fabrics exist; and treating BCM as the cluster aggregator for jobs and consolidated alerts, not as the source of low-level counters. Run:ai and NIM enter the stack only when workload scheduling fairness or inference SLOs become first-class operational requirements.
The guide demonstrates the framework on a DGX cluster with InfiniBand, BCM, and Slurm running mostly training jobs. The initial tool set comprises IPMI, DCGM, NVSM, UFM, and BCM — excluding NetQ, NMX, Run:ai, and NIM until Ethernet, rack-scale NVLink, or inference services are introduced. All metrics feed into Prometheus and Grafana for a unified triage dashboard.
Why It Matters
AI training follows the bulk synchronous parallel model, where one slow rank holds back the entire job. A single InfiniBand link drifting into an elevated bit error rate — a classic gray failure — can stall a rank during NCCL all-reduce operations, collapsing throughput to the slowest rank while other ranks block. The framework's value is in catching such regressions before multi-hour job waste accumulates. By tying a concise top-k alert set to SLIs and SLOs with clear remediation actions (drain node, replace cable, open fabric case), teams avoid "watermelon metrics" — dashboards that look green while services fail inside.
Our Take
NVIDIA's framework is notable for what it omits: it refuses to bless a single pane of glass, instead prescribing a two-layer approach where Grafana handles triage (GPU, node, or fabric?) and vendor UIs handle deep dives. That restraint reflects how AI factories actually operate — operators need to know which domain is broken before they can use the right tool to fix it.
