NVIDIA has released Topograph, an open-source toolkit that discovers cluster topology from cloud APIs and on-premises fabric systems and publishes it in forms schedulers can already consume: Kubernetes node labels, Slurm configuration, or Slinky ConfigMaps. It is available now from the dsx-ai-factory/topograph GitHub repository and slots into the NVIDIA DSX OS cluster orchestration layer alongside Dynamic Resource Allocation (DRA) and KAI Scheduler for topology-aware gang scheduling across AI factory infrastructure.
The release addresses a persistent operational problem: GPUs exchange data continuously during training and inference, so distributed workloads benefit from communication locality. Poor placement fragments topology domains, forces traffic across shared links, cuts throughput, raises job costs, and leaves GPUs consuming provisioned power while waiting on data. NVIDIA NVLink provides 1.8 TB/s of bidirectional bandwidth per GPU in its fifth generation (Blackwell, such as GB200/GB300) and 3.6 TB/s in its sixth generation (Vera Rubin) through a dedicated NVLink Switch fabric, while Quantum InfiniBand ports reach up to 800 Gb/s. A scheduler can only exploit that locality with a current, accurate view of GPU and fabric relationships — and keeping that view current as the cluster changes is where placement breaks down in practice.
Confirmed
- Topograph discovers topology from cloud providers (Google Cloud, Lambda, Nebius, Nscale, Oracle Cloud Infrastructure, Crusoe) and on-premises fabrics (InfiniBand via ibnetdiscover; Spectrum-X and Multi-Node NVLink domains via NetQ).
- It normalizes discovered topology into a canonical model and publishes it in scheduler-ready formats: Kubernetes node labels (fabric.topograph.run/tier-N and accelerator.topograph.run/domain), Slurm topology.conf (cluster-wide tree, cluster-wide block, or per-partition YAML), Slinky ConfigMaps, NFD resources, or instance-oriented topology JSON.
- Deployment is either a Kubernetes deployment via Helm chart (https://dsx-ai-factory.github.io/topograph) or native packages on Slurm clusters. Prerequisites: Kubernetes 1.27+, Helm 3.10+ or 4.x, kubectl permissions, and a supported provider.
- Five components — API Server, Node Observer, Node Data Broker, Provider, and Engine — keep the topology view current, with asynchronous generation requests (POST /v1/generate returns HTTP 202; GET /v1/topology?uid= returns 202 while processing, 200 when complete) and a typical 15-second aggregation delay. Simulation utilities (kwok-nodes, Kind/KWOK helpers) let operators test without production hardware by turning simulation models into virtual Kubernetes nodes.
- Integration with KAI Scheduler and Kueue TAS enables topology-aware gang scheduling on the same node labels. Kubernetes 1.36 introduced alpha topology-aware workload scheduling via KEP-5732, with upstream beta work ongoing.
Unknown
- Exact GA date for the Slurm engine running in Kubernetes (requires a writable volume for the topology.conf output path) and the NFD engine (requires the alpha NodeFeatureGroupAPI feature gate).
- Whether the Crusoe provider's requirement to run in Kubernetes extends to other cloud providers in future releases.
- Independent benchmark data quantifying throughput gains or cost reductions from Topograph-driven placement versus default scheduler behavior in production clusters.
- Rollout timeline for additional cloud and colocation providers mentioned as "in development."
Our take
Topograph fills a genuine operational gap: schedulers have long been topology-blind because reusable plumbing to keep a live topology view current across cloud and on-prem fabrics simply did not exist. By normalizing provider-specific discovery into a common model and emitting scheduler-ready outputs, NVIDIA is effectively commoditizing the topology layer — a win for the DSX OS ecosystem and any scheduler that can read node labels or Slurm topology.conf. The real test is independent replication of the claimed throughput and cost gains once operators run it at scale on heterogeneous fabrics.
