CXL-SpecKV: Bridging the LLM Memory Wall with Speculative FPGA Disaggregation
Listen to episode
About this episode
The researchers introduce CXL-SpecKV, a specialized architecture designed to overcome the memory bottlenecks of large language model serving by offloading key-value caches to remote memory. By utilizing Compute Express Link (CXL) and FPGA accelerators, the system enables memory disaggregation, which expands available storage capacity by up to eight times compared to standard GPU setups. A core innovation is a lightweight LSTM-based prefetcher that predicts upcoming token needs with 95% accuracy, effectively masking the latency of retrieving data from remote pools. The system further optimizes performance through an FPGA-driven compression engine that reduces bandwidth demands by roughly 4× without sacrificing model precision. Consequently, CXL-SpecKV delivers up to 3.2× higher throughput and significant cost reductions for datacenter environments. This hardware-software co-design demonstrates that intelligent memory management can efficiently scale AI infrastructure for next-generation workloads.
Source:
February 22 2026CXL-SpecKV: A Disaggregated FPGA Speculative KV-Cache for Datacenter LLM ServingYale University, Columbia UniversityDong Liu, Yanxuan Yuhttps://doi.org/10.1145/3748173.3779188
More AI podcast episodes
Browse all →Want to find AI jobs?
Join thousands of AI professionals finding their next opportunity