22 results for “quantization”
Tencent open-sourced an extreme quantized GGUF build of its 770B Hunyuan Hy4 model, compressing 1.5TB weights down to 214GB. By varying precision from 1.31-bit on non-critical layers to 2+ bits on sensitive layers, the model fits on accessible multi-GPU nodes.
A developer known as slvDev has demonstrated a 28.9-million-parameter language model generating text at 9.88 tokens per second on an ESP32-S3. Running a language model on a sub-$10 microcontroller changes the cost floor for local AI.
Tencent's Hunyuan team released Hy-MT2 on 21 May 2026 — a new translation model family with instruction-following templates, three sizes up to a 30B-A3B MoE variant, and open weights plus IFMTBench. This is the next generation after the April Hy-MT1.5 phone quantizations.
Hugging Face added GGUF support to Transformers, letting developers load llama.cpp checkpoints directly on Apple Silicon Macs. The integration reuses ggml Metal kernels and keeps the generation loop in PyTorch.
Tencent open-sourced Hy-MT1.5-1.8B packs that shrink a 3.3 GB translation model to as little as about 440 MB for offline use on ordinary phones. The 1.25-bit and 2-bit releases keep 33-language coverage while targeting on-device privacy and real-time translation without a network.
Black Forest Labs released FLUX 3 Action, a 7B open-weight world action model hitting 38.3% on RoboLab-120 at up to 3.95× the speed of prior open models. Distilled checkpoints are included.
DeepSeek Flash, Qwen3.8-Flash and GLM-5.3-Flash still price near $0.15/MTok input — while Opus and Sol sit at flagship multiples. Table refreshed to 22 September 2026: OpenAI’s Luna now prices below that band.
NVIDIA says TensorRT Edge-LLM ran Qwen3.6-27B through the MLPerf Edge Agentic workload in 24 minutes 36 seconds on one Jetson AGX Thor. The 6.4x gap over llama.cpp puts long-context agent inference on edge.
NVIDIA reports its Transformer Engine with JAX lifts DeepSeek-V3 671B throughput to 1,068 TFLOPS/GPU on GB200. This dropless MoE stack makes large-scale training practical without padding.
Z.ai released GLM-5.3-Flash, a 320B-parameter multimodal model with only 18B active parameters using hybrid sparse-linear attention. The MIT-licensed model approaches Claude Opus 4.8 on coding benchmarks at one-tenth the inference cost of its predecessor.
SenseTime fully open-sourced SenseNova U1, a native unified multimodal model that fuses vision and language without a vision encoder or VAE. The 8B-parameter release matches commercial models on infographic generation and adds native 4K output in its U1.5 update.
Hugging Face released @huggingface/kernels with 207 versioned WebGPU kernels and the Fleet benchmarking tool for browser-based AI. The kernels beat ORT WebGPU by 2.57× geometric mean on Apple M4, creating a shared performance foundation for local inference.