Medium
llama.cpp b9761: Model Downloads Moved to Dedicated Child Process
What changed
Model downloading is now offloaded to a dedicated child process (via PR #24834), decoupling download progress from the main server thread. Introduces MODEL_DOWNLOAD_TIMEOUT configuration and throttling mechanisms; the child process maintains lifecycle management (notably: does not use detach(), keeping it tied to server lifecycle).
TL;DR
llama-server now handles model downloads in a background process, keeping the main request thread responsive during large model file transfers — preventing server timeouts that previously occurred when pulling models while serving requests.
Developer signal
Update to b9761 if you run llama-server in a production environment where model pull operations overlap with live request serving. Prior to this, a model download could block the server's main thread, causing timeouts for concurrent inference requests. The /models/sse SSE endpoint (added in b9747, covered in yesterday's digest) now receives download progress from the child process, completing the non-blocking model loading system. If you've implemented custom MODEL_DOWNLOAD_TIMEOUT logic or signal handling around llama-server, review the new lifecycle management to ensure compatibility.
Affects you ifYou run llama-server in a multi-user or production setup and perform model pulls while the server is serving requests; or you manage server lifecycle automation that previously handled download blocking.EffortQuick (update to b9761; no API or config changes required unless you relied on the old single-threaded download behavior).
Medium
llama.cpp b9760: Video Input Wired into Server API Schema
What changed
The llama-server input file schema was refactored and generalized to support input_video as a first-class input type, with raw base64 video acceptance alongside existing image and audio inputs. Previously, video inputs had no defined schema path in the server API — this release wires the type and encoding pipeline.
TL;DR
llama-server's file schema now includes input_video with raw base64 support, expanding the multimodal server API to accept video alongside images and audio — the first explicit video input path in llama.cpp's server layer.
Developer signal
If you are using llama-server with a vision-capable or multimodal model, check whether the model's architecture supports video inputs — b9760 adds the schema and encoding path, but end-to-end video inference depends on whether the loaded model has video processing capability. The new input_video field accepts base64-encoded video data directly in the API request body, consistent with the existing image input pattern. This is a schema generalization, not a standalone feature toggle; developers building multimodal server applications should update their API clients to use the new type field for video payloads rather than relying on workarounds. No benchmarks or throughput numbers were published with this release.
Affects you ifYou use llama-server with multimodal or vision models and need to pass video input through the server API; or you maintain API client code that constructs file input schemas for llama-server.EffortModerate (update to b9760+, update API client to use input_video schema field, verify model supports video processing end-to-end).
Medium
llama.cpp b9756: Heap Buffer Overflow Fix in Server edit_file Endpoint
What changed
Fixed a heap buffer overflow in the llama-server edit_file endpoint triggered when line_start is set to -1 during append-at-end-of-file operations. Prior behavior: -1 was normalized to n+1, causing a vector range insert one position past the end — a heap-buffer-overflow. Fix: restricts -1 normalization to append-only mode, rejects it for replace/delete operations; also fixes empty-file append to compute insertion position as integer before pointer arithmetic.
TL;DR
llama-server's edit_file endpoint had a heap buffer overflow when agents appended content to the end of a file using line_start: -1; b9756 fixes the invalid pointer arithmetic and restricts the behavior to safe append-only mode.
Developer signal
Update to b9756 immediately if you run llama-server with the edit_file tool enabled and allow agents or automated code to write to files. The vulnerability is exploitable by any request that triggers append-at-end behavior with line_start: -1 — a natural operation in agentic coding workflows where agents add content to the end of files. The fix is behavioral: after b9756, line_start: -1 is only valid in pure append mode; passing it for replace or delete operations will now be rejected rather than proceeding with corrupted pointer arithmetic. Audit any agentic workflows that use edit_file with dynamic line_start values to confirm they only use -1 in append-only context.
Affects you ifYou expose llama-server's edit_file endpoint and allow agents or automated processes to append to files using line_start: -1; or you maintain a service that wraps llama-server's file editing API.EffortQuick (update to b9756; review any code that calls edit_file with line_start: -1 to confirm intent is append-only).
Medium
LiteLLM v1.90.0-rc.1: Cisco AI Defense Guardrails, E2B Sandbox, Valkey Cache (Pre-Release)
What changed
v1.90.0-rc.1 adds Cisco AI Defense as a new guardrail integration, E2B as a code execution sandbox primitive, Valkey as a semantic caching backend option, OTEL v2 metrics (6 gen_ai.client.* metrics at parity with v1), rate limit error standardization with category/rate_limit_type/model/llm_provider fields, Bedrock AWS project ID support, and Postgres native partitioning for SpendLogs retention. Also adds model routing entries for Claude Fable 5 across Anthropic, Bedrock, Vertex AI, and Azure AI (model suspended under US export-control directive; routing entries are placeholder for when the model returns).
TL;DR
LiteLLM's first v1.90 release candidate adds Cisco AI Defense guardrails, E2B code execution sandboxes, and Valkey semantic caching as new backend options — plus standardized rate limit error fields and 6 new OTEL v2 gen_ai metrics — but is pre-release (rc.1); wait for stable before production deployment.
Developer signal
Track this RC if you manage LiteLLM in production. The Cisco AI Defense integration brings enterprise-grade AI guardrail infrastructure that was previously only available via direct Cisco SDK integration — once stable, this gives LiteLLM proxy deployments a third-party guardrail option alongside existing Lakera and Presidio integrations. The E2B sandbox adds a code execution primitive to the proxy layer, useful for agentic workflows where you want sandboxed code execution gated behind LiteLLM's routing and cost tracking. The rate limit standardization (rate_limit_type field on errors) is genuinely useful for building retry logic that distinguishes token rate limits from request rate limits. Do not deploy rc.1 in production; the stable v1.90.0 release will follow — subscribe to the GitHub releases page for the notification.
Affects you ifYou run LiteLLM proxy in production and want to add guardrails, code execution sandboxes, or Valkey-backed semantic caching; or you build retry logic on top of LiteLLM and need structured rate limit error categorization.EffortModerate (rc.1 is pre-release; wait for stable; review Cisco AI Defense and E2B integration docs before enabling in production).