GPU Hoarding is Over. The $401B Reality Check
Listen to episode
About this episode
Enterprise GPU hoarding is over. LinkedIn CTO Erran Berger and VentureBeat analyst Rob Strechay break down what comes next — and the infrastructure math most enterprises are only now being forced to confront.
VentureBeat's Q1 research shows GPU availability anxiety dropped from 20.8% to 15.4% among enterprise teams, while cost-per-inference and TCO concerns jumped from 34% to 41% — a number that's still climbing. The hoarding phase is giving way to an audit phase, and the companies that didn't build the instrumentation to understand their workloads are now paying for it.
Erran Berger explains how LinkedIn runs one of the few remaining at-scale applied ML shops outside the hyperscalers — owning the full stack from bare metal GPU clusters to member-facing products. That means LinkedIn engineers can optimize custom CUDA kernels, compress embeddings, prune models for throughput, and adapt networking and storage per workload — trade-offs that are simply unavailable on public cloud instance menus. The result: a rigorous ROI framework that evaluates not just current traffic costs, but the traffic shape agents will drive in 2–3 years.
On the market side, 72% of enterprises admit they lack sufficient control over their AI infrastructure. Open-source inference tools like vLLM and LLMD are seeing rapid adoption, while 17% of organizations have moved to full-stack ownership. Hyperscalers report 60–80% of workloads have already shifted from training to inference — and most enterprise teams are still figuring out how to staff and instrument for that reality.
🎙️ GUEST: Erran Berger | CTO, LinkedIn
🎙️ ANALYST: Rob Strechay | VentureBeat
🎙️ HOST: Matt Marshall | CEO, VentureBeat
---
00:00 Intro: The GPU Hoarding Hangover
00:10 Guest Introductions
02:00 VentureBeat Q1 Data: GPU Panic Fades, TCO Concerns Rise
03:00 LinkedIn's Early Shift to Inference ROI Discipline
04:00 Budget Moving Into Inference Optimization and Control
07:00 LinkedIn's Full-Stack Advantage: Kernels, Pruning, Embedding Compression
08:00 Private AI and Sovereign Stacks: What the Q1 Data Shows
09:00 Open Source Inference Tooling: vLLM, LLMD, RDMA
10:00 Data Sovereignty at LinkedIn Scale: Member Data and Board-Level ROI Framing
12:00 Why Instrumentation Beats GPU Hoarding
13:00 Planning for Ambient Agent Traffic — Not Just Today's Workloads
14:00 Closing Advice for the Enterprise CTO Staring at 5% GPU Utilization
---
Subscribe to VentureBeat: https://www.youtube.com/@VentureBeat
Apple Podcasts: https://podcasts.apple.com/us/podcast/venturebeat/id1839285239
Spotify: https://open.spotify.com/show/4Zti73yb4hmiTNa7pEYls4
Website: https://venturebeat.com
LinkedIn: https://www.linkedin.com/company/venturebeat
Newsletter: https://venturebeat.com/newsletters
#EnterpriseAI #AIInfrastructure #MLOps #InferenceOptimization #GenerativeAI
---
Learn more about your ad choices. Visit megaphone.fm/adchoices
More AI podcast episodes
Browse all →Want to find AI jobs?
Join thousands of AI professionals finding their next opportunity