4 results for “LLM inference”
NVIDIA released NeMo Switchyard, an open-source library that routes AI agent steps across models to balance cost, latency, and accuracy. LangChain benchmarks show 74% cost savings with a 6-point accuracy tradeoff when routing between a 30B model and Claude Opus 4.8.
Bloomberg reporting cited by TechCrunch says Stripe has finalized an agreement to buy AI gateway OpenRouter for more than $7 billion. Neither company has confirmed the deal; Stripe declined to comment on speculation.
A developer known as slvDev has demonstrated a 28.9-million-parameter language model generating text at 9.88 tokens per second on an ESP32-S3. Running a language model on a sub-$10 microcontroller changes the cost floor for local AI.
Tencent has released AngelSpec, a unified training framework designed to tackle the persistent inference efficiency challenges that accompany large. Inference cost remains one of the most significant barriers to widespread LLM adoption.