He Built a $300M AI Agent 10 Years Before ChatGPT

About this episode

Summary:

In this conversation, I talked to Ashish Shubham (VP of Engineering), who's been at ThoughtSpot for 10 years, about AI agents in enterprise analytics. ThoughtSpot started as a search-based analytics company trying to make data accessible to regular business users. In 2019, they tried building natural language interfaces using BERT, but only hit about 50% accuracy. For a product where enterprise customers make billion-dollar decisions, that wasn't good enough. They shelved the project.

When ChatGPT came out, ThoughtSpot was ready. Ashish walked me through how they pivoted: they built a 25-30 person team, decided to use prompting instead of fine-tuning, and leveraged their existing semantic data modeling layer to get accuracy into the high 90s. We got into the technical evolution from monolithic systems to agent architectures with tools, how they went from manual human judges to using LLMs to evaluate their outputs, and how enterprise security requirements shaped what they built.

We also talked about how software engineering is changing. Ashish said 50-60% of his code is AI-generated now, and he thinks system design is becoming the critical skill, even for junior engineers. He had an interesting take on the "95% of AI deployments fail" stat too.

Chapters:

0:00 Intro and Ashish's journey to ThoughtSpot from GoDaddy

0:13 ThoughtSpot's mission to democratize data analytics for business users

1:26 Early search-based analytics before natural language processing

2:36 ThoughtSpot vs Tableau and the promise of self-service analytics

4:40 The analyst bottleneck problem and how ThoughtSpot aimed to solve it

5:49 Early technical challenges with in-memory databases and data migration

8:11 Semantic data models, joins, and creating abstraction layers for users

11:39 Who builds the data models and the role of analysts

12:22 Pre-LLM natural language processing using BERT and word2vec in 2018-2019

14:43 The accuracy problem and ambiguity in translating user queries

16:58 Trust challenges and why the early NLP product never became core

19:59 Competition with Tableau, Looker, and Power BI

22:44 How analyst roles changed with self-service analytics tools

25:30 The ChatGPT moment and pivoting to LLM-powered natural language

27:48 Early prompt engineering days and generating SQL with LLMs

31:09 Training vs prompting debate and why fine-tuning was eventually abandoned

34:28 Organizational changes and building the NLS team

37:16 Coaching systems for company-specific terminology vs training models

39:02 Evolution of evaluation methods from human judges to LLM-as-judge

43:23 Moving to LangFuse and GCP for agent infrastructure

46:29 How LLM context windows and capabilities evolved their product

50:07 From 30-column limits to agentic systems with 90%+ accuracy

52:52 RAG, column selection, and using proprietary data indexes

54:59 Multi-model support and enterprise data security concerns

59:14 How AI has changed Ashish's personal engineering workflow

1:02:42 Impact of AI on the broader engineering organization

1:04:15 Measuring AI productivity and the challenge of metrics

1:07:26 50-60% AI-generated code and the changing nature of coding

1:09:18 System design skills becoming more important than coding

1:13:00 Junior engineers doing senior-level work and interview changes

1:14:37 Customer conversations about Gen AI adoption across industries

1:17:26 The MIT report on 95% agent failures and why it misses the point

1:22:12 Agent architecture with LangGraph vs Google ADK and building internal agent platform

1:24:26 Where value lies in the next two years: tools, skills, and optimization

1:28:05 Startup opportunities in making AI accessible to non-technical users

1:29:26 Closing remarks