NVIDIA has published Day-0 inference benchmarks for Alibaba's Qwen3.8-2.4T-A95B, the company's largest open-weight model, running on its GB300 NVL72 rack-scale system. The model carries 2.4 trillion total parameters but activates only 95 billion per token through a fine-grained mixture-of-experts architecture with 512 experts. Ten routed experts and one shared expert fire per token.
Without additional model tuning, the FP8 implementation exceeds 4,000 tokens per second per GPU and 350 tokens per second per user on the 72-GPU Blackwell Ultra platform. NVIDIA says further optimizations, including NVFP4 precision, are planned and the current figures represent an initial serving result rather than a final performance target.
Brocker covered Alibaba's August 3 unveiling of Qwen3.8-Max. This piece is the Day-0 serving result on GB300 NVL72, not a second launch recap.
What's new
- Model: Qwen3.8-2.4T-A95B (NVIDIA also uses the Qwen3.8-Max name), 2.4T total parameters, 95B activated per token
- Architecture: Fine-grained MoE with 512 experts (10 routed + 1 shared active), hybrid full/linear attention, 92 layers
- Context: Native 262,144 tokens, extendable to ~1 million tokens
- Output length: Up to 128K tokens
- Reasoning controls: Configurable low / high / xhigh depth per request
- Precision at launch: FP8 and BF16 checkpoints; NVFP4 and MXFP4 quantized versions available from Inferact
- Hardware: NVIDIA GB300 NVL72 (72 Blackwell Ultra GPUs, 36 Grace CPUs, liquid-cooled rack, 130 TB/s aggregate NVLink bandwidth)
- Day-0 throughput: >4,000 tokens/sec/GPU peak, >350 tokens/sec/user
- Software stacks: SGLang, vLLM, NVIDIA Dynamo, model-free NVIDIA NIM container
- Fine-tuning: NVIDIA NeMo AutoModel supports full SFT and LoRA on Day-0 Hugging Face checkpoints
- Weight availability: Hugging Face and ModelScope
- Hosted APIs: DeepInfra, DigitalOcean, Fireworks AI, Modal, OpenRouter
Why it matters
The hybrid attention design keeps both compute and memory bounded as context scales toward one million tokens. Full-attention layers let every token attend to every other token. Linear-attention layers replace the growing key-value cache with a bounded recurrent state. This matters for agentic workloads — coding agents, large-scale document analysis, and multi-step workflows — where system instructions, tool outputs, retrieved documents, and reasoning traces accumulate across a session.
The fine-grained MoE structure distributes capacity across many small experts rather than a few large ones, improving routing efficiency per unit of activated compute. Serving cost tracks the 95B active parameters, not the full 2.4T, delivering frontier-scale capacity at a fraction of the cost of a comparable dense model.
On the hardware side, the GB300 NVL72's 72-GPU NVLink domain eliminates the cross-node communication bottleneck that typically appears when MoE expert traffic must traverse traditional off-the-shelf networks. The 130 TB/s all-to-all bandwidth is a key enabler for the reported throughput numbers.
Our take
The Day-0 numbers are impressive but reflect a 72-GPU reference rack few organizations will replicate. The real test is how well the vLLM, SGLang, and Dynamo recipes scale on smaller clusters and whether NVFP4 delivers the promised gains without measurable quality loss on reasoning benchmarks. Qwen's own documentation also distinguishes the released weights from Qwen3.8-Max, the hosted service that adds vision, built-in tools, and a one-million-token context by default — a reminder that open weights and the commercial service are not identical products.
