VAEs Are Energy-Based Models? [Dr. Jeff Beck]

About this episode

What makes something truly intelligent? Is a rock an agent? Could a perfect simulation of your brain actually be you? In this fascinating conversation, Dr. Jeff Beck takes us on a journey through the philosophical and technical foundations of agency, intelligence, and the future of AI.

Jeff doesn't hold back on the big questions. He argues that from a purely mathematical perspective, there's no structural difference between an agent and a rock – both execute policies that map inputs to outputs. The real distinction lies in sophistication – how complex are the internal computations? Does the system engage in planning and counterfactual reasoning, or is it just a lookup table that happens to give the right answers?

Key topics explored in this conversation:

The Black Box Problem of Agency – How can we tell if something is truly planning versus just executing a pre-computed response? Jeff explains why this question is nearly impossible to answer from the outside, and why the best we can do is ask which model gives us the simplest explanation.

Energy-Based Models Explained – A masterclass on how EBMs differ from standard neural networks. The key insight: traditional networks only optimize weights, while energy-based models optimize both weights and internal states – a subtle but profound distinction that connects to Bayesian inference.

Why Your Brain Might Have Evolved from Your Nose – One of the most surprising moments in the conversation. Jeff proposes that the complex, non-smooth nature of olfactory space may have driven the evolution of our associative cortex and planning abilities.

The JEPA Revolution – A deep dive into Yann LeCun's Joint Embedding Prediction Architecture and why learning in latent space (rather than predicting every pixel) might be the key to more robust AI representations.

AI Safety Without Skynet Fears – Jeff takes a refreshingly grounded stance on AI risk. He's less worried about rogue superintelligences and more concerned about humans becoming "reward function selectors" – couch potatoes who just approve or reject AI outputs. His proposed solution? Use inverse reinforcement learning to derive AI goals from observed human behavior, then make small perturbations rather than naive commands like "end world hunger."

Whether you're interested in the philosophy of mind, the technical details of modern machine learning, or just want to understand what makes intelligence tick, this conversation delivers insights you won't find anywhere else.

---

TIMESTAMPS:

00:00:00 Geometric Deep Learning & Physical Symmetries

00:00:56 Defining Agency: From Rocks to Planning

00:05:25 The Black Box Problem & Counterfactuals

00:08:45 Simulated Agency vs. Physical Reality

00:12:55 Energy-Based Models & Test-Time Training

00:17:30 Baye