Local Models, Open Weights, and AI Sovereignty: A Practitioner Roundtable

About this episode

Sam Gaddis sits down with the largest Runpoint Podcast panel yet: Matthew Hall and Ryan Mish from the Runpoint team, Thomas McNally of Zaelab, and returning guest Thanh Pham.

The conversation starts with the Theo video that stirred up the "local models are overrated" argument and goes deep from there. Thomas walks through what it actually looks like to deploy open weight models across a firm, including the cost math, the hardware, and why non-technical employees are lining up to join the program. Thanh brings the hobbyist-turned-practitioner view from his home lab and where local genuinely holds up. Along the way: the four-way split between frontier, local-on-prem, local-in-cloud, and rented open weight inference, plus a straight look at the two fears CEOs raise most often.

If you run a mid-market company and you're trying to figure out where these models fit, this one is for you.

Guests:
Thomas McNally, technology lead at Zaelab
Thanh Pham, Managing Director of Asian Efficiency
Matthew Hall and Ryan Mish, Runpoint

Timestamps:
0:00 Intro and the panel
0:53 The Theo video and the local vs open weight debate
1:22 Deploying open weight models across a firm: the cost story
4:41 Which models and hardware Zaelab landed on (Qwen, RTX 6000, VRAM)
6:41 Why employees want in on an "inferior" model program
8:00 The interface problem and Anthropic's third-party inference feature
9:28 Pay-per-use to all-you-can-eat, and protecting your IP
9:56 The subscription gravy train is ending
10:21 AI sovereignty and why the conversation shifted
12:11 Thanh's home lab and the hybrid local setup
13:41 What "most people" can actually run on their hardware
16:03 The routing trifecta: frontier, cloud infrastructure, and on-device
18:32 What each panelist actually uses day to day
19:30 Model agnosticism and avoiding vendor lock-in
21:40 How to build an intelligent routing layer
24:14 Evaluating routing solutions without a lab
24:54 Myth busting: Chinese models and training on your data
27:18 Where IP really lives for mid-market companies
28:10 The fourth option: rented open weight inference (Fireworks, GLM)
30:00 Clearing up the terminology confusion
32:52 Does fine-tuning transfer, or is it a one-time cost?
35:38 The maintenance burden nobody talks about
38:22 Tempering expectations if you're coming from frontier
39:19 Wrap up