Advancements in Efficient KV Cache Quantization and Management

About this episode

The provided sources explore advanced techniques for optimizing large language model (LLM) inference, specifically by addressing the memory bottlenecks of the Key-Value (KV) cache. KVQuant introduces a high-precision quantization framework that utilizes per-channel scaling, non-uniform datatypes, and sparse outlier handling to compress activations to sub-4-bit precision with minimal accuracy loss. Similarly, the KIVI algorithm proposes a tuning-free 2-bit quantization strategy that differentiates between key and value cache distributions to increase throughput. Shifting from quantization to architectural pruning, DuoAttention identifies specific Retrieval Heads that require full context while reducing Streaming Heads to constant memory usage by focusing only on recent tokens and attention sinks. Together, these methods enable LLMs to process million-level context lengths on standard hardware by drastically reducing the architectural and computational footprint of stored activations.

Sources:

1) 2024

KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization

University of California, Berkeley, ICSI, LBNL

Coleman Hooper, Sehoon Kim, Hiva Mohammadzadeh, Michael W. Mahoney, Yakun Sophia Shao, Kurt Keutzer, Amir Gholami

https://arxiv.org/pdf/2401.18079

2) 2024

KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache

Rice University, Texas A&M University, Stevens Institute of Technology, Carnegie Mellon University

Zirui Liu, Jiayi Yuan, Hongye Jin, Shaochen (Henry) Zhong, Zhaozhuo Xu, Vladimir Braverman, Beidi Chen, Xia Hu

https://arxiv.org/pdf/2402.02750

3) 2024

QAQ: Quality Adaptive Quantization for LLM KV Cache

Nanjing University

Shichen Dong, Wen Cheng, Jiayu Qin, Wei Wang

https://arxiv.org/pdf/2403.04643

4) May 8, 2024

KV Cache is 1 Bit Per Channel: Efficient Large Language Model Inference with Coupled Quantization

Rice University, Stevens Institute of Technology, ThirdAI Corp.

Tianyi Zhang, Jonah Yi, Zhaozhuo Xu, Anshumali Shrivastava

https://arxiv.org/pdf/2405.03917

5) 2024

DUOATTENTION: EFFICIENT LONG-CONTEXT LLM INFERENCE WITH RETRIEVAL AND STREAMING HEADS

MIT, Tsinghua University, SJTU, University of Edinburgh, NVIDIA

Guangxuan Xiao, Jiaming Tang, Jingwei Zuo, Junxian Guo, Shang Yang, Haotian Tang, Yao Fu, Song Han

https://arxiv.org/pdf/2410.10819

6) 2025

MILLION: MasterIng Long-Context LLM Inference Via Outlier-Immunized KV Product QuaNtization

Shanghai Jiao Tong University, Shanghai Qi Zhi Institute, Huawei Technologies Co., Ltd, China University of Petroleum-Beijing

Zongwu Wang, Peng Xu, Fangxin Liu, Yiwei Hu, Qingxiao Sun, Gezi Li, Cheng Li, Xuan Wang, Li Jiang, Haibing Guan

https://arxiv.org/pdf/2504.03661