Try for free
The Information Bottleneck
The Information Bottleneck

The Model Cheated — with Florian Brand (Prime Intellect)

27 July 2026 57:20 Ravid Shwartz-Ziv & Allen Roush

Listen to episode

About this episode

Florian Brand builds evals at Prime Intellect. The premise of the conversation is that writing a benchmark is the easy part now. Keeping the model from cheating it is the job, and it takes longer than the benchmark itself.

We get into why he thinks you can't evaluate a model apart from the CLI it runs in, what happens to statistics when a single run costs five figures, and whether the feeling that a model just works can ever become a number.

He also has a few stories about agents finding their way around the scoring that are worth hearing cold.

Timeline

  • 00:13 Intro
  • 01:00 What evals are for
  • 04:05 Agentic benchmarks
  • 07:10 Kimi K2 and model diversity
  • 08:23 Long-horizon coding tasks
  • 10:29 Building a benchmark
  • 12:15 MirrorCode
  • 14:27 Rubrics and LLM judges
  • 16:30 The cost of expert labelers
  • 17:49 Long runs and variance
  • 19:44 Evaluating the harness
  • 24:29 Chinese labs building CLIs
  • 30:00 More reward hacking
  • 37:45 Tau-bench and economic tasks
  • 39:43 Benchmaxxing and GLM 5.2
  • 45:15 Statistics and cost
  • 47:56 Frontier convergence
  • 52:04 Misuse in open and closed models
  • 55:35 Self-improvement

  • Music

    • "Kid Kodi" - Blue Dot Sessions - via Free Music Archive - CC BY-NC 4.0.

    About

    The Information Bottleneck is hosted by Ravid Shwartz-Ziv and Allen Roush, featuring in-depth conversations with leading AI researchers about the ideas shaping the future of machine learning.

More AI podcast episodes

Browse all →

Want to find AI jobs?

Join thousands of AI professionals finding their next opportunity

We respect your inbox. Unsubscribe at any time.

© 2026 The Information Bottleneck. All rights reserved.

Common Questions

Frequently asked questions

Quick answers about how DevFound's AI matching, resumes, and referrals work.

DevFound's AI Copilot ingests your profile, goals, and live job data to deliver curated matches in seconds. Every match includes a resume variant, suggested referrals, and interview prep so you can act immediately. The more feedback you provide, the sharper the Copilot becomes.

AI-led job searches shrink the hours spent sifting through boards and formatting resumes. DevFound pairs automation with your personal outreach, so you reserve energy for interviews and negotiation. Traditional networking still matters, but AI gives you a lift before you even send a message.

Modern AI roles expect comfort with production-grade code, data fluency, and practical ML tooling. The strongest candidates pair deep technical chops with storytelling—translating model impact to product, GTM, and exec partners. Continuous learning keeps you ahead as stacks evolve.

DevFound rewards active seekers. Keep your profile fresh, respond to match quality prompts, and enable alerts so you never miss a role. The AI prioritizes companies and teams that align with your feedback, accelerating both introductions and interview invites.

High-density tech hubs continue to host the deepest AI talent pools, yet distributed teams are catching up fast. Use DevFound filters to hone in on onsite, hybrid, or fully remote roles and watch openings expand across time zones.

DevFound aggregates thousands of remote AI openings and flags the nuances—core hours, async culture, and visa needs—up front. The Copilot also recommends how to position your distributed work experience so hiring managers know you can thrive on a remote team.

Still have questions?

Contact Support