On September 10, 2026, NVIDIA published a technical blog detailing how its NIM 2.0.12 optimized serving stack achieves up to 2.5 times higher system throughput on the Nemotron 3 Ultra model running on a four-GPU B200 system compared with a baseline serving stack without NIM optimizations. The benchmark targets an agentic workload profile of 64K context, 400 output tokens, 76 percent KV cache reuse, and a 50 tokens-per-second per-user interactivity target (20 ms inter-token latency).
At that 50 TPS/user target, the NIM-on configuration delivers 1,997 output tokens per second across the four B200 GPUs, versus 718 tokens per second for the NIM-off baseline. NVIDIA says the gains come from interacting configuration bundles — autotuned mixture-of-experts and Mamba kernels mapped to Blackwell GPUs, tensor parallelism across four GPUs with expert-aware execution, prefix caching with partial-prefix matching, tuned Mamba state caches, scheduler and memory tuning, and MTP speculative decoding with associated fixes — rather than from independent switches whose percentages can be added.
Confirmed
- Software: NIM 2.0.12 optimized serving stack, profile vllm-nvidia-b200-nvfp4-tp4-pp1-throughput-90.0 with NIM_SPECDEC_ENABLE=1.
- Model: Nemotron 3 Ultra (550B total, 55B active MoE hybrid Mamba architecture).
- Hardware: 4x NVIDIA B200 GPUs, 16 GB shared memory per container (--shm-size=16GB).
- Benchmark definition: agentic workload, 64K/400/76% KV reuse, 50 TPS/user (20 ms ITL).
- Measured throughput: 1,997 tok/s (NIM on) vs 718 tok/s (NIM off) at the 50 TPS/user Pareto point.
- Deployment path: NGC container nvcr.io/nim/nvidia/nemotron-3-ultra-550b-a55b:2.0.12 with validated profiles, standard OpenAI-compatible APIs, and NVIDIA AI Enterprise support for CVE handling and commercial SLAs.
- Replay tooling: NVIDIA AIPerf with Mooncake-format JSONL traces for concurrency sweeps.
Unknown
- Independent replication of the 2.5x figure on non-NVIDIA benchmarks or third-party clouds.
- Power draw, cost per token, and total cost of ownership at sustained production load.
- How much of the gain comes specifically from MTP speculative decoding versus the rest of the stack (acceptance rate and memory headroom dependent).
- Rollout timing for additional optimized NIM profiles across other models and GPU generations.
- Whether the 2.5x multiplier holds at different context lengths, output lengths, or reuse ratios outside the published agentic profile.
Our take
NVIDIA is turning inference optimization into a supported product SKU rather than a consulting engagement. The 2.5x claim is real within the published Pareto point, but the operational signal is the packaging: a single container tag, a validated profile name, and an AIPerf replay recipe that lets teams verify the curve on their own traffic. That shifts the bottleneck from "can we tune this?" to "do we have the B200 allocation and the NGC entitlement?" — which is exactly where NVIDIA wants the conversation.
Series: 1. NVIDIA Releases NemotronLabs VoiceChat 11B, an Open Full-Duplex Speech Model with Live Tool Calling · 2. NVIDIA NIM 2.0.12 delivers 2.5x throughput on Nemotron 3 Ultra with B200 GPUs · NVIDIA Nemotron & NeMo
