Google Research published TurboQuant on March 25, 2026 – a compression algorithm that reduces the memory a running AI model needs by six times, with no loss in accuracy and no retraining required. The paper will be formally presented at ICLR 2026 in Rio de Janeiro in late April.
The Problem It Solves
When you send a message to an AI model, the model does not process just your latest words. It keeps a record of the entire conversation in something called the key-value cache – its working memory for the session. As context windows have grown from thousands to millions of tokens, this cache has become the single biggest memory bottleneck in production AI systems, often consuming more memory than the model weights themselves.
Running a 70-billion-parameter model for 512 concurrent users can consume 512GB of cache memory alone. TurboQuant attacks that problem directly.
How It Works
TurboQuant compresses each value in the key-value cache from 16 bits down to approximately 3.5 bits using a two-stage pipeline. The first stage, PolarQuant, randomly rotates data vectors to simplify their geometric structure, making them easier to compress without the normalisation overhead that traditional methods require. The second stage, QJL, applies a mathematical error-correction layer to the residual, removing bias introduced during the first pass. Neither stage requires training data, calibration, or model-specific tuning – it works on any transformer model as a post-processing step.
The Numbers
On NVIDIA H100 GPUs, 4-bit TurboQuant delivers up to 8x speedup in computing attention logits compared to unquantised 32-bit keys. Across standard long-context benchmarks including LongBench, Needle In A Haystack, ZeroSCROLLS, RULER, and L-Eval, the algorithm matched uncompressed model performance at 3-bit quantisation with zero measurable accuracy loss.
The practical consequence: a GPU that previously served one long-context session could serve six, or handle context lengths six times longer on the same hardware.
Not Released Yet
Google has not published official code. Community developers have already built working implementations in PyTorch, MLX, and llama.cpp from the paper’s mathematics, but these are independent efforts. Google’s official implementation is expected in Q2 2026, likely timed to coincide with the ICLR presentation. Integration into major serving frameworks including vLLM and TensorRT-LLM is expected to follow.
Market Reaction
Within 24 hours of publication, memory chip manufacturers saw share price declines. Samsung, SK Hynix, and Micron all fell on the announcement. The logic was straightforward: if AI inference requires significantly less memory, near-term demand projections for high-bandwidth memory shift. Morgan Stanley noted, however, that TurboQuant targets only inference memory, not model weights or training workloads, and that in practice the efficiency gains are more likely to be used to serve longer contexts and more users than to reduce hardware spending.
John Moore is the editor of fastai.news, an independent publication covering developments in artificial intelligence.
He founded fastai.news in April 2026 to apply the same rigorous, neutral reporting standards he established at Powerboat News – his international publication – to the fast-moving world of AI.
With no filler and no opinion, fastai.news reports what is happening in AI models, research, business and tools, and leaves readers to draw their own conclusions.
John is based in Buckinghamshire, England.