NVIDIA has released NemotronLabs VoiceChat 11B, an 11-billion-parameter end-to-end speech-to-speech model that performs streaming speech understanding and generation in a single unified network. Instead of chaining automatic speech recognition, a large language model, and text-to-speech, the model listens while it speaks, enabling full-duplex conversation with a measured turn-taking latency of 448 milliseconds on the Full-Duplex-Bench 1.0 benchmark.
The release marks the first open-weight full-duplex model to support live tool calling. Tool invocations are emitted on a dedicated output channel as scripts while the conversation continues, and operators can define per-tool "on-hold" lines that the agent speaks immediately so the dialogue does not fall silent during API execution.
What's new
The architecture stitches together three existing NVIDIA components with a new tool-calling output path: a Fast Conformer speech encoder from Nemotron-Speech-Streaming-En-0.6b that continuously encodes 16 kHz input, the Nemotron Nano v2 LLM backbone that consumes audio tokens and predicts text tokens, and an NVIDIA TTS decoder and codec that renders 22.05 kHz agent speech. Training drew on roughly 550,000 hours of real and synthetic audio, building on the SALM-Duplex and Audio Flamingo 3 research lines.
On Full-Duplex-Bench 1.0 the model posts a smooth turn-taking TOR of 0.82 at 448 ms and a user-interruption TOR of 1.00 at 480 ms. On the AU Harness BFCL-v3 spoken tool-calling suite it scores 58.5% on simple calls, 62.5% on multiple, 42.5% on parallel, and 27.5% on parallel-multiple, with an 89.6% irrelevance detection rate and a 56.1% average. Tool selection accuracy reaches 82.5%, but argument accuracy sits at 44.2% and pass@1 at 33%. NVIDIA reports the model ranks second among open full-duplex models on both VoiceBench and Full-Duplex-Bench 1.0.
Deployment requires a single GPU with at least 80 GB of VRAM — A100, H100, RTX 6000 Pro, or B200 on x86_64 Linux. The model weights are hosted on Hugging Face under the OpenMDW-1.1 license, and a signed NIM container (10.55 GB compressed, version 1.0.0) is available on the NGC catalog. There is no hosted API and no third-party inference provider currently serves the model.
Why it matters
VoiceChat 11B gives researchers and infrastructure teams a reference implementation of full-duplex plumbing — barge-in handling, live tool calling with on-hold filler, and a unified streaming stack — that has until now lived only in closed, hosted, per-minute-priced APIs. The open weights and container let teams benchmark, reproduce latency numbers, and copy architectural ideas on their own hardware.
The trade-offs are explicit. NVIDIA labels the checkpoint "ready for research purposes only," so a startup cannot ship a product on this license. The model card documents failure modes: a two-minute audio context ceiling, degradation into non-recoverable gibberish after several turns, runaway self-talk after a turn ends, and dropped words in user transcription. Tool-calling reliability drops sharply on multi-step tasks — parallel-multiple calls succeed only 27.5% of the time — making agentic voice workflows unreliable today.
Our take
The research-only license on open weights creates a familiar tension: the community gets a reproducible artifact to study and extend, but the most immediate commercial beneficiaries are teams that can afford 80 GB GPUs and are willing to build their own serving stack around a checkpoint NVIDIA itself says is not production-ready.
