On August 24, 2026, NVIDIA disclosed that its Groq 3 LPX accelerator, integrated with the Vera Rubin NVL72 platform, achieved 3,431 output tokens per second on the Artificial Analysis 100K context benchmark using the Gemma 4 31B model. The result marks the first third-party benchmark of Groq 3 LPX.

The benchmark measured median speed across samples with 100K input tokens using the "o200K_basetokens" tokenizer. At 10K context length, the same system delivered 3,382 tokens per second. For agentic coding tasks that can exceed 100K context tokens and generate 5,000 reasoning and output tokens, NVIDIA estimates decoding at this rate takes roughly 1.5 seconds versus 50 seconds at 100 tokens per second.

What's new

  • Benchmark: Artificial Analysis 100K context benchmark on Gemma 4 31B (released April 2026)
  • Result: 3,431 median output tokens/second at 100K context; 3,382 tokens/second at 10K context
  • System: NVIDIA Groq 3 LPX paired with Vera Rubin NVL72 in NVIDIA's own data centers
  • Architecture: 256 interconnected LP30 local processing units (LPUs) with 128 GB total SRAM
  • Networking: 96 chip-to-chip (C2C) links per chip at 112 Gbps each, 640 TB/s rack-scale scale-up bandwidth
  • Memory bandwidth: 40 PB/s on-chip SRAM bandwidth
  • Compute: 315 PFLOPS AI inference compute (FP8)
  • Key techniques: Deterministic compiler-scheduled workload planning, fine-grained computation-communication overlap at 320-byte vector granularity, preplanned C2C networking eliminating real-time arbitration
  • Co-execution modes: Prefill-decode disaggregation, attention-FFN disaggregation, speculative external-drafter decoding, scaling to multi-trillion parameter models
  • Additional benchmark: 4,767 median output tokens/second on SPEED-Bench for agentic and coding tasks

Why it matters

Agentic sessions grow context across hundreds of turns, often reaching hundreds of thousands of tokens. Without long-context capability at high interactivity, agents can only consider a fraction of relevant history. Groq 3 LPX's ability to sustain >3,000 tokens/second at 100K context enables multi-agent systems powered by 2T+ parameter models to maintain responsiveness while processing full session history. The deterministic compiler schedule and fine-grained overlap address the fundamental bottleneck of tensor parallelism at small batch sizes: first-bit latency dominates when payloads are tiny, and LPX's pre-scheduled, arbitration-free transfers keep that latency minimal. This architecture also reduces the power cost per token, a factor that becomes decisive when AI factories run trillion-parameter models continuously.

Our take

The benchmark validates NVIDIA's heterogeneous inference strategy: Vera Rubin NVL72 handles prefill and attention while Groq 3 LPX accelerates latency-sensitive FFN and MoE decode. What remains unproven is how this architecture performs under real-world multi-tenant loads with variable context lengths and concurrent users — the Artificial Analysis benchmark measures a single-stream, fixed-context scenario. Software readiness for the disaggregated serving paths orchestrated by NVIDIA Dynamo will determine whether data centers can operationalize the claimed 35x throughput-per-megawatt and 10x revenue opportunity for trillion-parameter models.

Sources