31: Local LLMs: Good Enough Might Be Enough

About this episode

Jim shares his adventure into running LLMs on his own hardware. For him it's less about saving money and more about privacy — working in healthcare, he can't send patient data to the cloud.

• App vs. model: Claude Code and Codex are applications, not models. Features like plan mode come from the app. (Wolf's "Opus Plan" is a Claude Code mode that uses Sonnet 4.6 for most work and Opus 4.8 for planning.)
• Ollama makes local models easy — ~15-min install, runs on macOS/Linux/Windows, and exposes a REST API. Not to be confused with Meta's Llama models. Example: `ollama run llama3`.
• Parameters & training: Think of an 8B model as "8 billion knobs." Training randomly initializes them, then refines predictions over billions of iterations. Wolf ties this to Markov models (parameters ≈ weighted edges) and the Bayes episode (random init = priors).
• Fitting big models in memory: Quantization shrinks 32-bit parameters down to ~4 bits. Mixture of Experts (MoE) keeps only part of a model active (e.g., Llama 4 is ~108B params but ~17B active).
• Jim's tests (M1 Mac Studio, 64 GB), asking why H₂O is liquid: Llama 4 Scout took ~10–15 min and maxed out RAM/swap; Llama 3 (8B) answered in ~31s; Qwen (36B) gave the best answer in just 34s.
• The open question: Is local "good enough"? Wolf's real test isn't trivia — can a local model write and iterate on an 8-page implementation plan? (Homework for Wolf's 128 GB MacBook Pro.)
• Build your own: Fine-tune an existing model or train from scratch. Jim's dream: a local model fine-tuned on his DB schema + 2,000 SQL queries so users could ask in plain English and get runnable Postgres — no cloud required. Browse Hugging Face for specialized models

Hosts:
Jim McQuillan can be reached at [email protected]
Wolf can be reached at [email protected]

Follow us on Mastodon: @[email protected]

If you have feedback for us, please send it to [email protected]

Checkout our webpage at http://RuntimeArguments.fm

Theme music:
Dawn by nuer self, from the album Digital Sky