Notable
llama.cpp b10701 — DFlash2 NVFP4 Draft Models Fixed: Speculative Decoding Was Producing Near-Zero Accepted Tokens
What changed
DFlash2 NVFP4 draft models were not passing the Q, K, V, and output-projection scales to their corresponding graph operations. This omission caused speculative decoding with NVFP4 quantized draft models to produce "almost no accepted speculative tokens," silently degrading inference throughput while the mode appeared to run normally.
TL;DR
llama.cpp b10701 fixes a missing-scales bug in DFlash2 NVFP4 speculative decoding that caused near-zero token acceptance, restoring functional speculative inference on NVFP4-quantized draft models (NVIDIA Blackwell / Grace Hopper hardware).
Developer signal
If you run speculative decoding with NVFP4-quantized draft models on Blackwell or Grace Hopper hardware (H100/H200/B100/B200 via DFlash2), update to b10701 immediately. The feature was effectively non-functional — the model appeared to speculate tokens but nearly none were accepted, meaning you were paying the overhead of speculative decoding without any throughput benefit. After b10701, verify acceptance rates with --verbose to confirm the fix is active in your configuration. If you run speculative decoding on non-NVFP4 quantizations (e.g., GGUF Q4/Q8 on CUDA), you are unaffected. NVFP4 quantization is specific to DFlash2 on Blackwell/Grace Hopper and distinct from the GGUF quantization families.
Affects you ifYou use speculative decoding (--draft-model) with NVFP4-quantized draft models on NVIDIA Blackwell (B-series) or Grace Hopper (GH-series) hardware via DFlash2.EffortQuick — update to b10701; no config changes required.
Notable
llama.cpp b10688 / b10696 / b10697 — Apple M2, M3 Pro, and M3 Ultra Get Flash Attention Vector Tunings
What changed
Three sequential builds shipped Metal Flash Attention vector (fa-vec) tuning parameters for the M2 SoC family (b10688, PR #27940), M3 Pro (b10696, PR #27993), and M3 Ultra (b10697, PR #27999). These tunings configure the GEMM kernel path inside Flash Attention for each chip's specific GPU core count and memory bandwidth, matching the approach used for M1/M1 Pro/M1 Max and recently for the M3 standard in prior builds.
TL;DR
llama.cpp now has chip-specific Flash Attention vector tunings for Apple M2, M3 Pro, and M3 Ultra — the same per-chip optimization that delivered measurable prefill improvements on M1 and M3 chips. No benchmark numbers were published with these releases.
Developer signal
Update to b10697 (the latest build) to pick up all three tunings at once — each build is cumulative. No configuration changes are needed; the correct Metal kernel path is selected automatically based on the detected SoC. If you run llama.cpp on macOS with an M2-family (M2, M2 Pro, M2 Max, M2 Ultra) or M3 Pro/Ultra device, this update is low-risk and potentially provides prefill improvement — quantify it with llama-bench against your typical prompt length before and after. The M3 standard and M1-family were already tuned in prior builds. M4-family tunings have not yet appeared.
Affects you ifYou run llama.cpp on macOS with Apple M2-series or M3 Pro/Ultra chips; you build on-device inference pipelines for Apple Silicon.EffortQuick — update to b10697; no configuration changes.
Notable
llama.cpp b10693 — Qualcomm Hexagon NPU Gets Runtime Device Discovery and On-Demand Sessions
What changed
The Hexagon DSP/NPU backend gained three improvements contributed by Qualcomm engineers (Marco Colombo, Alexander Lu, Ehsan Bateni): (1) runtime discovery of available NPU cores instead of requiring pre-configuration; (2) lazy session allocation with proper cleanup of device interfaces; (3) early rejection of non-existing devices during init, preventing silent failures. Previously, using the Hexagon backend required knowing device identifiers in advance.
TL;DR
llama.cpp b10693 adds runtime NPU core detection for Qualcomm Hexagon devices (Snapdragon X Elite, Snapdragon 8 Gen series), contributed by Qualcomm engineers, making the Hexagon backend usable without manual device ID configuration.
Developer signal
If you build llama.cpp inference pipelines targeting Qualcomm Snapdragon devices and use the Hexagon NPU backend (-ngl with Hexagon device target), update to b10693. The runtime discovery means you no longer need to hard-code or pre-enumerate Hexagon device IDs — the backend will detect available NPU cores at startup and fail fast (rather than silently) if the specified device does not exist. This lowers the integration barrier for Android on-device inference on Snapdragon X Elite and Snapdragon 8 Gen 3/4 SoCs. Lazy session allocation also reduces resource waste when sessions are initialized but not immediately used. No benchmark numbers were published; the improvement is to reliability and configurability rather than throughput at this stage.
Affects you ifYou run llama.cpp on Qualcomm Snapdragon devices using the Hexagon NPU/DSP backend; you build on-device inference for Android with Snapdragon X Elite or Snapdragon 8 Gen 3/4 SoCs.EffortQuick — update to b10693; the new device discovery is automatic.
Notable
LiteLLM v1.100.0-rc.1 — Anthropic Empty Thinking Block Fix, MCP Signing, Budget Enforcement
What changed
LiteLLM crossed the v1.100 milestone as a release candidate, shipping 100+ changes. Developer-relevant highlights: Anthropic integration now drops and self-heals empty thinking blocks (previously causing silent failures or API errors in workflows that pass through LiteLLM); reasoning effort handling improved; MCP gateway now supports asymmetric signing and introspection capabilities; budget enforcement is now available per model access group; Lakera v2 guardrails add skip-message and advisory modes; AWS partition awareness added for Bedrock and SageMaker endpoints.
TL;DR
LiteLLM v1.100.0-rc.1 fixes Anthropic empty thinking blocks (a silent failure mode for reasoning model integrations), adds MCP gateway asymmetric signing, and enforces budget limits at the model access group level — 100+ changes across the proxy layer.
Developer signal
If you route Anthropic Claude requests through LiteLLM and use extended thinking (Claude Opus 5, Sonnet 5, or Fable 5), update to v1.100.0-rc.1 to get the empty thinking block fix — empty blocks were previously breaking downstream integrations that expected non-empty thinking content. This is an rc release, so test in staging before promoting to production. For MCP users: the new asymmetric signing support in the MCP gateway enables signed tool call verification, which is relevant if you need audit trails or want to validate that tool calls weren't tampered with in transit. Budget enforcement at the model access group level is useful for organizations that want to cap spend per team or per use case without changing API keys.
Affects you ifYou route Anthropic Claude API calls through LiteLLM and use extended thinking or reasoning models; you operate a LiteLLM proxy with MCP tool servers; you manage per-team or per-use-case budget limits in a LiteLLM deployment.EffortModerate — this is a release candidate; test in staging first. Core fixes (thinking blocks) are drop-in; new features (MCP signing, budget groups) require config changes.