Optimizing Verification and Efficiency in Multi-Draft Speculative Decoding

About this episode

These sources explore advanced techniques for accelerating **Large Language Model (LLM) inference** through **speculative decoding**, a process where smaller "draft" models predict tokens for a larger "target" model to verify in parallel. A primary focus is **Multi-Draft Speculative Decoding (MDSD)**, which uses multiple draft sequences to increase the probability of acceptance and reduce latency. Researchers have introduced **SpecHub** to simplify complex optimization problems into manageable linear programming, while others utilize **optimal transport theory** and **q-convexity** to reach theoretical efficiency upper bounds. Additionally, the **Hierarchical Speculative Decoding (HSD)** framework stacks multiple models into a tiered structure, allowing each level to verify the one below it. Collectively, these papers provide **mathematical proofs**, **sampling algorithms**, and **hierarchical strategies** designed to maximize token acceptance rates and minimize computational overhead.

Sources:

1)

January 22 2025

Towards Optimal Multi-draft Speculative Decoding

Zhengmian Hu, Tong Zheng, Vignesh Viswanathan, Ziyi Chen, Ryan A. Rossi, Yihan Wu, Dinesh Manocha, Heng Huang.

2)

2024

SpecHub: Provable Acceleration to Multi-Draft Speculative Decoding

Lehigh University, Samsung Research America, University of Maryland

Ryan Sun, Tianyi Zhou, Xun Chen, Lichao Sun

https://aclanthology.org/2024.emnlp-main.1148.pdf.

3)

2024

MULTI-DRAFT SPECULATIVE SAMPLING: CANONICAL DECOMPOSITION AND THEORETICAL LIMITS

Qualcomm AI Research, University of Toronto

Ashish Khisti, M.Reza Ebrahimi, Hassan Dbouk, Arash Behboodi, Roland Memisevic, Christos Louizos

https://arxiv.org/pdf/2410.18234.

4)

2025

HISPEC: HIERARCHICAL SPECULATIVE DECODING FOR LLMS

The University of Texas at Austin

Avinash Kumar, Sujay Sanghavi, Poulami Das

https://arxiv.org/pdf/2510.01336.

5)

2025

Fast Inference via Hierarchical Speculative Decoding

Harvard University, Google Research, Tel Aviv University, Google DeepMind

Clara Mohri, Haim Kaplan, Tal Schuster, Yishay Mansour, Amir Globerson

https://arxiv.org/pdf/2510.19705.