On September 16, 2026, NVIDIA said its TensorRT Edge-LLM inference stack ran Qwen3.6-27B on a single Jetson AGX Thor Developer Kit and finished the MLPerf Inference v6.1 Edge Agentic performance workload in 24 minutes and 36 seconds. That is 6.4x faster than the llama.cpp reference run of 2 hours and 37 minutes that MLCommons publishes alongside the same benchmark, according to the company.
The submission served the workload at 52.33 tokens per second in SingleStream mode, NVIDIA said. Four optimizations do most of the work: NVFP4 quantization for weights and activations, an FP8 KV cache, cache reuse across agent turns, and tree-based multi-token prediction.
Confirmed
MLPerf Edge Agentic is built for the workload NVIDIA is targeting. The performance phase replays recorded software-engineering agent trajectories: the model receives a request, emits a tool call, observes the tool result, and continues the same conversation. The dataset holds 20 conversations and 1,007 generated turns, with input length growing across turns to roughly 23.5K tokens, so long-context processing is part of what is being measured. An IoU-based inline accuracy check runs during the performance phase to confirm the agent is behaving correctly.
The results NVIDIA reports for its submission on a 128 GB Jetson AGX Thor Developer Kit in MAXN power mode:
- Output throughput: 52.33 tokens per second
- Median time to first token: 247.12 ms
- Median time per output token: 14.68 ms
- BFCL overall accuracy: 87.94%
The accuracy phase uses Berkeley Function Calling Leaderboard v4 prompts — 995 of them in the published dataset — in single-turn mode with reasoning switched off, scoring whether the model picks the right function, produces valid arguments, and refrains from calling a tool when none is needed.
Quantization carries much of the gain. Qwen3.6-27B is submitted with NVFP4, a 4-bit floating-point format supported by the Blackwell GPU inside Jetson AGX Thor, applied to weights and activations including the language-model head, plus FP8 for the KV cache. NVIDIA says low-batch decoding on edge platforms is bounded mostly by DRAM bandwidth, so shrinking both weights and activations reduces kernel memory traffic. The smaller representation also leaves more of the 128 GB of unified memory for the long context, speculative decoding state, and application workloads. Developers can start from a calibrated checkpoint instead of quantizing the model themselves; a team that needs its own calibration can run post-training quantization once on a development system.
Cache reuse addresses the shape of agent traffic rather than raw token speed. Every new request in a trajectory carries most of the preceding conversation, and without reuse the model must prefill that shared history again on each turn — a cost that grows with the conversation. TensorRT Edge-LLM finds reusable prompt prefixes and restores their cached attention KV pages. Because Qwen3.6 uses a hybrid architecture, the runtime also restores recurrent state and partial KV-page state before prefilling only the new suffix. On this workload, about 96% of prompt tokens were served from hot cache: the runtime prefilled roughly 0.5M of the 13.6M total prompt tokens across all turns.
Tree-based multi-token prediction covers the other half. Instead of keeping one predicted continuation, the runtime organizes high-probability candidates into a tree, the target model verifies them in a single forward pass, and the runtime accepts the matching path — advancing several tokens when more than one candidate survives. The MLPerf server configuration used 8 draft steps, the top-2 candidates at each drafting depth, and a 16-node verification tree. NVIDIA says tree-based MTP delivered roughly 40% more decoding performance than a linear MTP with 3 draft steps on this function-calling workload, where tool names, JSON syntax, and argument structures are predictable enough to draft.
The work is available now rather than at a future launch date. The implementation sits on the TensorRT Edge-LLM release/0.9.1-mlpinf branch, which bundles the model export settings, TensorRT engine build commands, server configuration, and MLPerf client configuration. The calibrated Qwen3.6-27B NVFP4 checkpoint is published for download, and the client side runs the MLCommons endpoint harness. NVIDIA's supplied configuration runs both phases with temperature 0, seed 42, reasoning disabled, and concurrency 1.
Unknown
The numbers are NVIDIA's own submission, described in NVIDIA's own technical write-up. Independent replication is not part of the material, and MLPerf scores from a single vendor submission say less about production deployments than they do about what one carefully tuned stack achieved on a recorded workload.
Two caveats matter for anyone reading the 6.4x as a general claim. First, the comparison baseline is the llama.cpp reference run published with the MLCommons Edge Agentic example, using Q4_K_M quantization — a portability reference, not a vendor-optimized competitor stack pushed to its limit. Second, the result is measured at MAXN, the maximum power mode of a 128 GB developer kit, so it describes a peak-power envelope rather than a thermally or battery-constrained automotive or robotics profile.
The accuracy evidence is also narrower than the performance evidence. BFCL v4 is scored single-turn with reasoning off, which fits edge evaluation budgets but does not measure correctness over the 1,007-turn multi-turn trajectory that produced the throughput headline.
Other gaps remain open in the material: no pricing or availability details for the 128 GB Jetson AGX Thor Developer Kit, no batch or multi-client serving numbers beyond SingleStream, and no figures for other model sizes.
Our take
The headline number is a workload result, not a hardware property. Agent cost on edge lives in prefill, because a trajectory keeps re-reading its own history; the fact that roughly 96% of prompt tokens came from hot cache is what collapses a two-and-a-half-hour run into 24 minutes. A team with shorter conversations, different tool schemas, or accepted-token patterns that the 16-node tree cannot guess should expect the MTP gain to move, which is exactly why the branch, the checkpoint, and the client config ship together.
