Hugging Face announced on September 22, 2026 that its Transformers library can now run GGUF quantized models directly through the familiar from_pretrained API, with Apple Silicon Macs as the first optimized target. The integration reuses the ggml kernels that power llama.cpp, aiming to bring performance close to that dedicated runtime while keeping the model and generation loop in Python and PyTorch.
GGUF, developed by the llama.cpp team, packages model weights, tokenizer information, and optional chat templates into a single file with multiple quantization levels. Publishers including Unsloth, LM Studio Community, and bartowski already host millions of GGUF checkpoints on the Hugging Face Hub. The new support lets developers load those checkpoints without converting formats, inspect intermediate activations with PyTorch hooks, prototype custom layers, and fine-tune from a GGUF checkpoint by dequantizing weights back to BF16.
Confirmed
- Transformers main branch (pre-release) loads GGUF files via AutoModelForCausalLM.from_pretrained(model_id, gguf_file=filename) and AutoTokenizer.from_pretrained(model_id, gguf_file=filename); on Apple Silicon the loader automatically selects ggml Metal kernels for attention and linear layers, falling back to SDPA with a warning when kernels are unavailable.
- Also shipping in the same pre-release: a transformers serve command that exposes an OpenAI-compatible endpoint at http://localhost:8000/v1 for any GGUF checkpoint (usable from clients such as Jan or Pi), and fine-tuning from GGUF through GgufConfig(dequantize=True) with a target dtype such as torch.bfloat16.
- Initial architecture focus is Qwen3.5; the blog demonstrates Qwen3.5-4B at Q4_K_M, Q5_K_M, and Q6_K quantizations.
- Benchmark methodology: MacBook Pro M2 Max, 32 GB unified memory, macOS 26.6, PyTorch 2.12.1, kernels 0.17.0. Transformers generate includes prefill; llama.cpp llama-bench reports decode-only throughput (tg128).
| GGUF variant | File size | Tradeoff |
|---|---|---|
| BF16 | 8.42 GB | Unquantized reference |
| Q6_K | 3.53 GB | More precision than the smaller variants |
| Q5_K_M | 3.14 GB | A middle ground between size and precision |
| Q4_K_M | 2.74 GB | A practical starting point for local inference |
Unknown
- Exact Transformers release version and date when GGUF support graduates from main to a stable pip release.
- Independent replication of the reported throughput parity on other Apple Silicon chips (M1, M3, M4) and on non-Apple hardware (CUDA, ROCm, CPU-only).
- Quantization fidelity and perplexity comparisons for the same checkpoints loaded via Transformers vs. llama.cpp across diverse tasks.
- Support timeline for architectures beyond Qwen3.5 and for multimodal models (vision, audio) using ggml kernels.
- Memory overhead of the Transformers Python runtime versus the lean llama.cpp binary for long-context or multi-turn workloads.
Our take
Hugging Face isn't chasing llama.cpp on raw inference speed — the blog says use llama.cpp when speed is the priority. The point is to close the developer loop: the same GGUF files that run in Ollama or LM Studio can now be inspected, evaluated, and fine-tuned in PyTorch without format conversion. Whether kernel reuse scales to new architectures, and whether the Python generation loop holds up under multi-user serving, is still untested.
