NVIDIA upstreams llama.cpp optimisations it says make local inference up to 1.9x faster

NVIDIA used IFA 2026, on 3 September, to push local AI further into the mainstream, with a wave of updates aimed at running models and agents on its own hardware. Some of it is the usual product launch. One part is more useful than it first looks, and it costs existing owners nothing.

The news

The part worth acting on is software. NVIDIA says kernel optimisations it has contributed to the open-source llama.cpp project deliver up to 1.9x higher throughput on a GeForce RTX 5090, through faster kernels, improved speculative decoding, and quicker prefill (NVIDIA, September 2026). It reports smaller gains for vLLM, 1.2x on the RTX PRO 6000 Blackwell Workstation Edition and up to 1.4x across two DGX Spark clusters, from new attention kernels and backend work. The company also announced simplified local-AI setup for any NVIDIA GPU with at least 24GB of VRAM, reachable through the llama.cpp and vLLM backends and the Ollama and LM Studio apps that most people actually use.

Because these optimisations are upstreamed into the open-source projects rather than locked to a new product, the practical upshot is unusual: if you already own a recent NVIDIA card, you get the improvement by updating your inference engine, with no new hardware to buy. The models NVIDIA highlighted running on this stack are ones we already cover, including Nemotron 3.5 Lightning, Qwen3.8-27B, and Meta’s Muse Glimmer.

Alongside the software, NVIDIA showed the RTX Spark desktop, which it specifies with a 1-petaflop Blackwell GPU, up to 128GB of unified memory, and a 20-core Grace CPU, arriving in October 2026.

Why it matters

Free performance is rare in this field, so it is worth being precise about what is on offer and what is not. The multipliers are NVIDIA’s own figures, measured on its hardware, and a throughput number depends heavily on the model, the quantisation, the context length, and the batch size, so treat 1.9x as a best case pending independent testing rather than a figure you will see on every workload. What is not in doubt is the direction: the optimisations are in the open-source projects, so the ceiling for local inference on hardware people already own has moved up, not only on hardware they might buy.

The RTX Spark is the other half of the story, and it lands in a space that has become genuinely competitive. A 128GB unified-memory desktop aimed at running large models locally is the same pitch as Apple’s M5 Mac Studio and Mac mini, from a different direction, and it continues the line NVIDIA started with the DGX Spark at launch. As ever with unified memory, the capacity is the draw and the memory bandwidth is what determines how fast a large model actually runs, so the useful numbers will come from independent testing once the machines ship.

The technical part

Two of the named improvements are worth a sentence. Speculative decoding runs a small, fast model to draft several tokens ahead and then has the large model check them in a single pass, which turns several slow steps into one and speeds up generation when the draft is usually right. Faster prefill shortens the time spent reading a long prompt before the first token appears, which is exactly where Apple Silicon and other integrated-memory machines tend to be slower, so gains there matter most for long-context work. Both live in the inference engine rather than the model, which is why an update, not a new download, is what unlocks them.

This is the kind of unglamorous, upstream engineering that quietly raises the floor for everyone running models locally. It follows NVIDIA’s other recent moves in the space, including the release of the open Nemotron 3.5 Lightning, and it is a reminder that the software layer improves fast enough that a machine gets meaningfully quicker over its life without you changing anything in it.