NVIDIA has released NeMo Switchyard, an open-source library that routes AI agent workloads across multiple models based on capability, cost, and infrastructure signals. Announced on August 11, 2026, the SDK lets developers send each agent step to the model best suited for that task — whether a frontier model for complex reasoning or a smaller, cheaper model for routine operations — without rewriting application code for each provider.

The core problem Switchyard addresses is straightforward: agents today often send every request to a single large model, inflating cost and latency for work that a lighter model could handle. A coding agent, for example, may need deep reasoning for debugging but only simple classification for file reads. Switchyard evaluates each request against a developer-configured pool of models and routes accordingly.

How the routing works

Switchyard separates routing logic from model providers through a provider-agnostic SDK called switchyard-libsy. Each model target gets a semantic name; the client maps that name to the actual endpoint and model ID. This means teams can swap models, change endpoints, or switch providers without touching the routing integration. A reference server implements OpenAI, Anthropic, and Responses API compatibility, logging the selected model, decision rationale, token usage, and latency for observability.

Routing decisions draw on three signal categories: model capabilities (which model solves the task), cost profiles (latency and pricing per model), and infrastructure state (load, errors, pricing). The router can evaluate at each agent turn or per full request, and can maintain session state — such as tool results or affinity decisions — across turns when policy requires it.

Tuning-free and tunable routers

Switchyard ships several tuning-free routers that operate without workload-specific training:

  • LLM classifier: Uses a small judge model to pick a target LLM and maintains session affinity, avoiding repeated reclassification for unchanged work.
  • Stage router: Reads recent tool activity — errors, exploration patterns, token counts — to infer the agent's workflow stage and route to a capable or efficient model accordingly.
  • Escalation router: Starts every conversation on a lower-cost model; a judge monitors turn-by-turn and promotes the session to a more capable model after sustained difficulty (two consecutive negative verdicts).

A tunable prefill router is also in research. It extracts the LLM's residual stream at prefill to estimate query complexity, then uses a shared-trunk MLP to predict each candidate model's likelihood of success. The router blends predicted accuracy with cost and latency constraints to pick the best tradeoff.

Benchmark results from LangChain

LangChain evaluated Switchyard using its Deep Agents suite — 145 multi-step tasks averaging 6.3 model calls each — routing between NVIDIA Nemotron 3.5 Lightning (30B parameters) and Claude Opus 4.8. The routed configuration achieved 80.0% accuracy at $3.00 per run versus 86.0% at $11.45 for Opus alone, a 74% cost reduction for a 6-point accuracy drop. Nemotron handled 93% of model calls for 10.4% of spend; Opus handled 7% of calls for 68.4% of spend. The judge model consumed 21.2% of routed spend, running on every turn until escalation and receiving no prompt-caching benefit.

Frontier-model traffic across five runs ranged from 4.1% to 9.1% (mean 6.9%), with per-run savings of 68.5% to 81.1%. LangChain notes the eval suite was saturated — only 8 points separated the 30B and frontier models — giving routing less room to prove value than harder workloads would. Results should be treated as a measurement of one workload, not a universal forecast.

Ecosystem integrations

NVIDIA is collaborating with partners to embed Switchyard into existing workflows: Cognition for coding agents, Nous Research for Hermes Agent routing, Ramp for financial software engineering, LangChain for evaluation, LiteLLM as a plug-in, and Kong for AI gateway governance and traffic management. Kong AI Gateway provides connectivity and policy enforcement while Switchyard handles model selection.

Why it matters

Model routing moves from a custom engineering project to a reusable library component. For teams running agents at scale, the ability to offload the majority of calls to cheaper models without sacrificing quality on hard steps directly reduces inference spend. The provider-agnostic design also reduces lock-in: a team can route to self-hosted open-weight models today and swap in a new frontier API tomorrow without changing routing logic.

Sources