Rob May: When should you stop using just one AI model?

About this episode

Rob May is a serial founder (five startups) and we spoke about how he went from “I did not intend to go do a fifth” startup to building a new AI company after he “stumbled upon an AI idea” he could prove out—and couldn’t ignore. He’s been focused on AI since 2015 (after exiting his first company in December 2014) and framed the current moment simply: even “10 years later… we are still just at the beginning” of what’s coming.

Rob walked through the real path to his current thesis: early bets, being “a couple years too early” pre-LLM, and a short venture-capital chapter that was “very intellectually stimulating,” but ultimately he missed “the battleground” of building and winning in-market. That experience shaped the core idea of the episode: most teams start with one big model, then hit the same wall—cost, speed, and inconsistent outputs—because models are probabilistic. As he puts it, “if you ask the same question multiple times, you might get different answers,” which is why techniques like Best-of-N (ask the same question 10 times) can reveal a distribution and help you avoid weak one-shot results while you decide what should run where.

If you’re building with AI, this conversation gives you a concrete way to think about splitting work across multiple models, when you actually need a frontier model, and how small process changes (like repeated probing) can improve reliability without rewriting your whole product.

Key takeaways

• Stop routing everything to one model; match model strength to task type.
• Use Best-of-N: ask 10 times, inspect outputs, choose the best.
• One-shot outputs can vary; plan for probabilistic behavior.
• Keep complex, variable tasks on frontier models; offload narrow tasks to smaller ones.
• Validate ideas fast; being “too early” is real in AI product timing.
• Optimize for accuracy, speed, and cost together by splitting workloads.