Why AI evals are the hottest new skill for product builders | Hamel Husain & Shreya Shankar (creators of the #1 eval course)

To build great AI products, you need to be really good at building evals. It's the highest ROI activity you can engage in. This process is a lot of fun. Everyone that does this. immediately gets addicted to it when you're building an AI application. You just learn a lot. What's cool about this is you don't need to do this many, many times. For most products you do this process once and then you build on it. The goal is not to do evals perfectly, it's to actionably improve your product. I did not realize how much controversy and drama there is around evals. There's a lot of people with very strong opinions. People have been burned by evals in the past. People have done evals badly, and then they didn't trust it anymore, and then they're like, Oh, I'm anti eval. What are a couple of the most common misconceptions people have with evals? The top one is we live in the age of AI. Can't the AI just eval it? But It doesn't work. A term that you used in your post that I love is this idea of a benevolent dictator. When you're doing this open coding, a lot of teams get bogged down in having a committee do this. For a lot of situations, that's wholly unnecessary. You don't want to make this process so expensive that you can't do it. You can appoint one person whose taste that you Trust. It should be the person with domain expertise. Oftentimes it is the product manager.

Today my guests are Hamal Hussein and Shreya Shankar. One of the most trending topics on this podcast over the past year. has been the rise of Evals. Both the chief product officers of Anthropic and OpenAI. Share that evals are becoming the most important new skill for product builders.

And sin, this has been a recurring theme across many of the top AI builders I've had on. Two years ago I had never heard the term e bells, now it's coming up constantly. When was the last time that a new skill emerged that product builders had to get good at to be successful? Hamel and Shreya have played a major role in shifting Evals from being an obscure, mysterious subject To one of the most necessary skills for AI product builders.

We already have this episode, so unlocking it is cheap and permanent: the full transcript, segment and word-level timestamps, and .txt, .srt, .vtt and word-level JSON downloads, as many times as you like, forever.

10 free credits is 10 episodes like this one a month. Episodes nobody has transcribed yet cost their length in hours, because we have to actually transcribe them.