On September 14, 2026, NVIDIA published a technical blog detailing how its Transformer Engine library, combined with JAX and the MaxText training stack, raises DeepSeek-V3 671B throughput from 103 to 1,068 TFLOPS/GPU on NVIDIA GB200 hardware — a 10.4x improvement over an unoptimized baseline, according to NVIDIA. The work targets the core bottlenecks of mixture-of-experts (MoE) training: ragged expert tensors, all-to-all communication overhead, and memory pressure at scale.
The optimizations are demonstrated on NVIDIA GB300 NVL72 rack hardware, where the full stack sustains 97% scaling efficiency at 1,024 GPUs, NVIDIA reports. NVIDIA says the gains come from a layered set of kernel and system-level changes that make dropless MoE — where every token is processed without dropping or padding — practical at production scale.
Confirmed
- Throughput: DeepSeek-V3 671B reaches 1,068 TFLOPS/GPU on GB200 with the optimized JAX + Transformer Engine path, up from a 103 TFLOPS/GPU baseline (10.4x).
- Scaling: 97% scaling efficiency at 1,024 GPUs on GB300 NVL72 when training DeepSeek-V3 671B.
- Baseline bottleneck: Unoptimized JAX MoE training on GB200 spent 84% of accumulated kernel time in inter-GPU communication.
- Key kernels: Grouped GEMM (cuBLAS/cuBLASLt-backed) for variable-length expert matmuls; NCCL EP for fused dispatch/combine with token deduplication across NVLink and InfiniBand.
- Quantization: MXFP8 block scaling for expert matmuls on Blackwell GPUs via Transformer Engine grouped quantization kernels.
- Memory: JAX host offloading moves query/value projection activations to host memory; XLA multistreaming collectives overlap cross-node InfiniBand with intra-node NVLink traffic via the Latency Hiding Scheduler.
- Software stack: NGC MaxText container with Transformer Engine enabled; MaxText MoE configuration guide available.
- Roadmap: NVFP4 quantization fused with GEMM, and all-to-all overlap, are planned for future Transformer Engine JAX bindings.
Unknown
- Independent replication of the 1,068 TFLOPS/GPU figure on non-NVIDIA hardware or with other MoE architectures (Qwen, Mixtral) has not been published.
- Wall-clock time-to-convergence, token-per-dollar cost, and power draw for the 1,024-GPU run are not disclosed.
- Availability timeline for the NVFP4 and A2A-overlap features mentioned in the roadmap.
- Whether the 97% scaling efficiency holds beyond 1,024 GPUs or with different model parallelism configurations.
- Exact memory footprint per GPU for the optimized DeepSeek-V3 671B run.
Our take
The 10.4x jump shows how much MoE training still leaves on the table when the software stack assumes dense, rectangular tensors. Fusing dispatch and combine, deduplicating tokens on the receive side, and keeping shapes dynamic all the way to the GEMM turns a communication-bound problem into a compute-bound one. The remaining headroom is in kernel fusion, not architecture. Whether those gains stay coupled to NVIDIA-specific runtimes is the open question.
