Skills, Context, and Trust: The New Agentic Coding Stack

About this episode

Dan and Jonathan Bown open with the talk Jonathan gave at ODSC, "Practical Agent Ops: From POC to Prod with MLflow 3.0." MLflow 3.0 arrived last summer as the first stable release built for generative AI rather than traditional machine learning, and Jonathan's team used it to build an agent for pre-enrollment students. The centerpiece of that work was evaluation-driven development. Instead of jumping straight into a working prototype they aligning the business up front on what quality actually looks like before signing off on a model with inherently non-deterministic output.

The initial key to success was an Excel file. In it, the data science team had already assembled 150 ground truth examples, but left them untested and set aside while engineers focused on code. Jonathan's team paused the coding work and ran a simple foundation model against those examples first, landing at what amounted to a coin flip of useful versus hallucinated answers. From there they refined the examples with the business, loaded them into MLflow's evaluation datasets built from live traces, and iterated by versioning prompts and agent configurations.

Tooling came up repeatedly. MLflow's open source repo now ships a skill file that plugs into coding tools like Claude Code, which Jonathan called a game changer for keeping up with an API that changes at roughly a release a month. The Databricks AI Dev Kit, released around March, bundles skills for the Databricks SDK, CLI, data engineering, and analytics work, usable either inside Databricks' Genie Code pane or in outside tools such as Claude Code, AWS Kiro, or Google Antigravity. Jonathan said installing it produced a dramatic jump in output accuracy compared to coding assistants working from stale or incomplete context about Databricks and MLflow APIs.

Dan raised the idea that LLMs and agentic tools are becoming users of software in their own right, alongside humans, and Jonathan tied that to broader changes at WGU: more of the business, not just engineers, now writes system prompts and builds their own copilot->

Jonathan's path to WGU ran through Pentara, a biostatistics consultancy, and Zions Bancorporation, where he did quant finance work before a stint simulating financial products for WGU students. He became a founding member of WGU's MLOps team in 2023, when the university's machine learning was still traditional work like random forests and ensembles for predicting student outcomes, well before Databricks had built out MLOps tooling. Dan connected this to Hamel Husain's essay "The Revenge of the Data Scientist", and Jonathan agreed that evaluation-driven development brings the work full circle: checking evals and correctness is the generative AI analogue of checking a confusion matrix.

The pre-enrollment agent's rollout became the clearest illustration of the method. The first release, a bare foundation model with no WGU context, drew heavy negative feedback from the employees testing it, some of whom wanted to cancel the initiative. Jonathan's team treated that feedback as fuel, folding the failed questions into an evaluation dataset and iterating until they reached roughly 82 percent correctness and near-total relevance, at which point the same employees became enthusiastic supporters. He credited MLflow's architecture for building subject matter experts directly into the agent ops workflow rather than treating evaluation as a purely technical exercise.

Jonathan was candid about where his trust runs out. He does not trust a tool's first output even after a full planning session, citing a Kiro planning cycle from the day before that failed on the first try despite extensive back and forth. He is cautious about MLflow's fast release cadence outpacing its own skill files, and notably guarded about tools like OpenClaw and Claude Cowork that can reach into email or personal documents. Given how much effort WGU puts into protecting student data, he extends the same caution to his own personal information and limits what such agents can access.

On his team, Jonathan resists banning AI-generated code or stigmatizing it in review, and instead pushes everyone toward reviewing code outside their usual specialty, using AI review tools like Amazon Q or GitHub Copilot as a starting point rather than a final answer. He pushed back on the idea that tool usage equals productivity, warning about AI slop and noting that some of the heaviest users he knows are not the most productive. The thread ties back to evaluation-driven development's real thesis: start from value, not from the tool, a point he illustrated with WGU's Academic Virtual Assistant pilot, where a surprising result showed that students chatting with the assistant were more likely, not less, to still reach out to a human mentor afterward.

Full episode notes

Click here to view the episode transcript.

Chapters

• (00:00) - Introducing Jonathan Bown
• (00:56) - ODSC talk: escaping POC prison with MLflow 3.0
• (05:11) - The forgotten Excel file: rebuilding around evals
• (09:36) - MLflow skills and the Databricks AI Dev Kit
• (15:00) - When AI becomes the user of your software
• (18:16) - How the day-to-day has changed in six months
• (22:16) - Centralizing prompts, evals, and best practices
• (25:38) - From quant finance to founding WGU's MLOps team
• (31:02) - The Revenge of the Data Scientist
• (35:01) - Has the job gotten easier or harder?
• (37:33) - Thinking ten steps ahead with agentic coding tools
• (42:15) - Leveling up junior engineers instead of gatekeeping review
• (51:17) - Where trust breaks down: OpenClaw and personal data
• (56:48) - The mental toll of managing agents versus writing code
• (59:27) - How much detail agentic tools actually need in a prompt
• (01:07:14) - Value over software: the Academic Virtual Assistant's surprise result

⠀

Links from the show

--------------------

• MLflow 3.0
• Databricks AI Dev Kit
• Genie Code
• ODSC (Open Data Science Conference)
• OpenClaw
• AWS Kiro
• Google Antigravity
• The Revenge of the Data Scientist
• Amazon Q

⠀

Guests

-------

Jonathan Bown, Principal ML Engineer, WGU

• Website
• LinkedIn

⠀

Follow the podcast

-------------------

• LinkedIn
• Threads
• Instagram
• TikTok

⠀

Follow Dan Gerlanc

-------------------

• X