Google: R&D inference value on HBF + PNM + low latency interconnect

About this episode

To address the hardware bottlenecks of LLM inference, Google researchers Ma and Patterson propos in their paper "Challenges and Research Directions for Large Language Model Inference Hardware" published on January 8, 2026 a few focus areas of research: High Bandwidth Flash (HBF), Processing-Near-Memory (PNM), and low-latency interconnects.

**HBF** addresses the "Memory Wall" by stacking flash dies to achieve **10X the capacity** of HBM, making it ideal for storing model weights and long contexts despite its write endurance limitations. **PNM** is advocated over Processing-In-Memory (PIM) for datacenters because placing logic on separate but nearby dies (e.g., 3D stacking) allows for larger software shards (avoiding fine-grained partitioning), utilizes standard high-performance logic processes, and offers better thermal management than integrating logic directly into memory dies. Finally, arguing that **latency trumps bandwidth** for the frequent small messages in inference, the authors suggest optimizing interconnects through high-connectivity topologies (like dragonfly or trees) and **processing-in-network** to accelerate communication collectives.

Modern large language model (LLM) inference faces a critical memory wall, where hardware compute power outpaces the growth of data transfer speeds. Research suggests addressing these bottlenecks through **3D memory-logic stacking**, near-memory processing, and specialized **interconnect strategies** to reduce latency. Optimization techniques for **Mixture-of-Experts (MoE)** architectures involve balancing **tensor and expert parallelism** across devices to ensure efficient data handling. While high-bandwidth memory remains expensive, alternative storage solutions like **flash memory** are being explored to expand capacity for data centers. Historical data further illustrates the evolving **cost and density** of memory, underscoring the long-term economic shifts in hardware development. Together, these sources outline a roadmap for evolving **AI hardware** to meet the rigorous demands of real-time model decoding.

Source:

January 8, 2026

Challenges and Research Directions for Large Language Model Inference Hardware

Google

https://arxiv.org/pdf/2601.05047