Google TurboQuant: 3-Bit KV Cache With Zero Accuracy Loss
A deep dive into Google TurboQuant's PolarQuant and QJL techniques — 6x KV cache memory reduction and 8x attention speedup, and what that actually means in practice.
Tags
inference
4 posts
Deep-Thinking Ratio: Cut LLM Inference Costs by 50%
Google & UVA research overturns the "longer = better" assumption for LLM reasoning. The Deep-Thinking Ratio (DTR) can cut inference costs in half while improving accuracy.
ASIC Inference Chip Runs Llama 3.1 8B at 16,000 tok/s
Startup Taalas achieves 16,000 tok/s on Llama 3.1 8B using custom ASIC chips without GPUs. We analyze the shift away from GPU dependency and the inference cost revolution.
Qwen3 Coder Next llama.cpp Graph Optimization
ggerganov restructures the llama.cpp compute graph to achieve up to 38% inference speedup for the Qwen3 Coder Next 80B model. Detailed benchmark analysis and technical breakdown.