On Validating Product Ideas

January 19, 2023 • #

Building new things is an expensive, arduous, and long path, so product builders are always hunting for means to validate new ideas. Chasing ghosts is costly and deadly.

The “data-driven” culture we now live in discourages making bets from the gut. “I have a hunch” isn’t good enough. We need to hold focus groups, do market research, and validate our bets before we make them. You need to come bearing data, learnings, and business cases before allowing your dev team to get started on new features.

Validating ideas

And there’s nothing wrong with validation! If you can find sources to reliably test and verify assumptions, go for it.

But these days teams are compelled to conduct user testing and research to vet a concept before diving in.

This push for data-driven validation I see as primarily a modern phenomenon, for two reasons:

First, in the old days all development of any new product was an enormously costly affair. You were in it for millions before you had anything even resembling a prototype, let alone a marketable finished product. Today, especially in software, the costs of bringing a new tool to market are dramatically slashed from 20 or 30 years ago. The toolchains and techniques developed over the past couple decades are incredible. Between the rise of the cloud, AWS, GitHub, and the plethora of open source tools, a couple of people have superpowers to do what used to take teams of dozens, and thousands of hours before you even had anything beyond a requirements document.

Second, we have tools and the cultural motivation to test early ideas, which weren’t around back in the day. There’s a rich ecosystem of easy-to-use design and prototyping tools, plus a host of user testing tools (like UserTesting, appropriately) to carry your quick-and-dirty prototypes out into the world for feedback. Tools like these have contributed to the push for data-driven-everything. After all, we have the data now, and it must be valuable. Why not use it?

I have no problem with being data-driven. Of course you should leverage all the information you can get to make bets. But data can lie to you, trick you, overwhelm you. We’re inundated with data and user surveys and analytics, but how do we separate signal from noise?

One of my contrarian views is that I’m a big defender of gut-based decision making. Not in an “always trust your gut” kind of way, but rather, I don’t think listening to your intuition means you’re “ignoring the data.” You’re just using data that can’t be articulated. It’s hard to get intuitive, experiential out of your head and into someone else’s. You should combine your intuitive biases with other objective data sources, not attempt to ignore them completely.

I’m fascinated by tacit knowledge (knowledge gained through practice, experience). It differentiates between what can be learned only through practice from what can be read or learned through formal sources — things we can only know through hands-on experience, versus things we can know from reading a book, hearing a lecture, or studying facts. Importantly, tacit knowledge is still knowledge. When you have a hunch or a directional idea about how to proceed on a problem, there’s always an intrinsic basis for why you’re leaning that way. The difference between tacit knowledge and “data-driven” knowledge isn’t that there’s no data in the former case; it’s merely that the data can’t be articulated in a spreadsheet or chart.1

I’ve done research and validation so many ways over the years — tried it all. User research, prototype workshopping, jobs-to-be-done interviews, all of these are tools in the belt that can help refine or inform an idea. But none of them truly validate that yes, this thing will work.

One of the worst traps with idea validation is falling prey to your own desire for your idea to be valid. You’re looking for excuses to put a green checkmark next to an idea, not seeking in-validation, which might actually be the more informative exercise. You find yourself writing down all the reasons that support building the thing, without giving credence to the reasons not to. With user research processes, so many times there’s little to no skin in the game on the part of your subject. What’s stopping them from simply telling you want you want to hear?

Paul Graham wrote a post years ago with a phenomenal, dead-simple observation on the notion of polling people on your ideas. A primitive and early form of validation. I reference this all the time in discussions on how to digest feedback during user research (emphasis mine):

For example, a social network for pet owners. It doesn’t sound obviously mistaken. Millions of people have pets. Often they care a lot about their pets and spend a lot of money on them. Surely many of these people would like a site where they could talk to other pet owners. Not all of them perhaps, but if just 2 or 3 percent were regular visitors, you could have millions of users. You could serve them targeted offers, and maybe charge for premium features.

The danger of an idea like this is that when you run it by your friends with pets, they don’t say “I would never use this.” They say “Yeah, maybe I could see using something like that.” Even when the startup launches, it will sound plausible to a lot of people. They don’t want to use it themselves, at least not right now, but they could imagine other people wanting it. Sum that reaction across the entire population, and you have zero users.

The hardest part is separating the genuine from the imaginary. Because of the skin-in-the-game problem with validation (exacerbated by the fact that the proposal you’re validating is still an abstraction), you’re likely to get a deceitful sense of the value of what you’re proposing. Would you pay for this? is a natural follow up to a user’s theoretical interest, but is usually infeasible at this stage.

It’s very hard to get early critical, honest feedback when users people have no reason to invest the time and mental energy thinking about it. So the best way to solve for this is to reduce abstraction — give the user a concrete, real, tangible thing to try. The closer they can get to a substantive thing to assess, the less you’re leaving to their imagination as to whether the thing will be useful for them. When an idea is abstract and a person says “This sounds awesome, I would love this”, you can safely assume they’re filling in unknowns with their own interpretation, which may be way off the mark. Tangibility and getting hands-on with a complete, usable thing to try, removes many false assumptions. You want their opinion on what it actually will be, not what they’re imagining it might be.

In a post from a couple years ago, Jason Fried hit the mark on this:

There’s really only one real way to get as close to certain as possible. That’s to build the actual thing and make it actually available for anyone to try, use, and buy. Real usage on real things on real days during the course of real work is the only way to validate anything.

The Agile-industrial complex has promoted this idea for many years: moving fast, prototyping, putting things in market, iterating. But our need for validation — our need for certainty — has overtaken our willingness to take some risk, trust in our tacit knowledge, and put early, but concrete and minimal-but-complete representations out there to test fitness.

De-risking investments is a critical element of building a successful product. But some attempts to de-risk actively trick us into thinking an idea is better than it is.

So I’ll end with two suggestions: be willing to trust your intuition more often on your ideas, and try your damnedest to smoke test an idea with a complete representation of it, removing as much abstraction as possible.

  1. I’d highly recommend Gerd Gigerenzer’s book Gut Feelings, which goes deep on this topic. For a great précis on the topic, check out his conversation from a few years back on the EconTalk podcast. 

Books of 2022

January 4, 2023 • #

This year I got to several books that have been on my list for years, excited to finally dig into them.

Here’s the full list, with my favorites ⭐️ starred:

Where Is My Flying Car? ⭐️ Where Is My Flying Car? by J. Storrs Hall Published: 2021 • Completed: December 12, 2022 • 📚 View in Library
Helgoland ⭐️ Helgoland Making Sense of the Quantum Revolution
by Carlo Rovelli Published: 2020 • Completed: October 17, 2022 • 📚 View in Library
The Rise and Fall of the Third Reich ⭐️ The Rise and Fall of the Third Reich A History of Nazi Germany
by William L. Shirer Published: 1960 • Completed: October 29, 2022 • 📚 View in Library
The Captured Economy The Captured Economy How the Powerful Enrich Themselves, Slow Down Growth, and Increase Inequality
by Brink Lindsey, Steven Teles Published: 2017 • Completed: October 8, 2022 • 📚 View in Library
Scene and Structure Scene and Structure by Jack M. Bickham Published: 1999 • Completed: August 16, 2022 • 📚 View in Library
Underland ⭐️ Underland A Deep Time Journey
by Robert Macfarlane Published: 2019 • Completed: August 5, 2022 • 📚 View in Library
Childhood's End Childhood's End by Arthur C. Clarke Published: 1953 • Completed: August 4, 2022 • 📚 View in Library
The Tacit Dimension The Tacit Dimension by Michael Polanyi Published: 1966 • Completed: July 21, 2022 • 📚 View in Library
The Law ⭐️ The Law by Frédéric Bastiat Published: 1850 • Completed: June 11, 2022 • 📚 View in Library
The Future and Its Enemies The Future and Its Enemies The Growing Conflict Over Creativity, Enterprise, and Progress
by Virginia Postrel Published: 1998 • Completed: May 29, 2022 • 📚 View in Library
Knowledge and Decisions Knowledge and Decisions by Thomas Sowell Published: 1979 • Completed: May 3, 2022 • 📚 View in Library
The Conservative Sensibility The Conservative Sensibility by George Will Published: 2019 • Completed: March 17, 2022 • 📚 View in Library
The Five Dysfunctions of a Team The Five Dysfunctions of a Team A Leadership Fable
by Patrick Lencioni Published: 2002 • Completed: February 22, 2022 • 📚 View in Library

Progress Report, November 2022

December 15, 2022 • #

I’m late getting my November update posted. November (and still, in December) was a rollercoaster of a month. Just so much happening with professional and personal, I’ve hardly had a moment to do much at all — neither focusing on any personal progress goals, nor writing or other fun side deals.

My running performance was pitiful. I did 5 runs, but honestly I’m surprised it was even that many. Feels like I’m totally off the wagon on that. I did alright on my sleep, but I swing too much back and forth to be a healthy pattern. I’ll do a string of 5-6 hour sleep nights, punctuated by sleeping 10 hours the next. The see-saw effect isn’t intentional. Something I need to focus more on building a pattern with.

Public writing didn’t do great, only a few blogs before I fell off and didn’t get any more writing done. I did better on the personal journal, though. At least for the first half the month.

Reading also suffered some. I feel like I didn’t spend any time with a book at all.

Health & Habits

Running

  • 5 activities (8 vs 5)
  • Distance: 12.9 mi (26.44 miles)
  • Total Time: 1:54:22 (3:59:24)
  • Average Pace: 8:50 (8:56) / mi

Sleep

  • Average: 7:27 / night (7:32)
  • 8 hr nights: 10 (10)

Writing ƒ

  • Journal entries: 10; 9,908 words (14; 6428 words)
  • Blog posts: 4 (14)
  • Newsletters: 0 (1)

Media

Reading

Liberal Fascism, Jonah Goldberg
░░░░░░░▓▓▓▓▓░░░░░░░░ 35-60%

Where Is My Flying Car?, J. Storrs Hall
░░░░░░▓▓▓▓▓▓▓▓▓▓░░░░ 32-80%

Podcasts

  • 12 episodes — 18 hr, 16 min 18 episodes — 20 hrs, 41 min

TV

  • Andor, 4 episodes
  • House of the Dragon, 5 episodes
  • The Terminal List, 4 episodes

Film

None

Making Mistakes Means You're Doing Things

November 2, 2022 • #

I talk all the time about trial and error. The freedom to let yourself make mistakes, and the skill to make sure they’re not too destructive, are superpowers. With every interesting innovation, company, or product, you’re seeing the late stage of a long chain of missteps and failure. As long as you have the right mindset, mistakes are learning.

We talk about this as a product team — short cycles, iteration, feedback loops — ways to navigate toward broader visions while surviving and building something increasingly useful along the way. I also talk about it with the kids. The more you practice hitting off the tee the better you’ll get at hitting the ball. The more you draw pictures the better you get at it. Practice through the frustration. I try to reinforce with them that everyone that’s great at something got their through an incredible volume of failure and shortcoming before the skill you see today.

Wooden sofa

If you’ve ever built anything physical, like woodworking, crafting, or DIY stuff around the house, you’ll be familiar with making mistakes, often costly ones. There’s no frustration quite like taking a furniture workpiece you’ve glued up from other parts, honed, mortised, and sanded and making a miter in the wrong place, or cutting it down to length too short. Hours and hours of work can vaporize in a second. I’ve made project mistakes like this so many times, and each time there’s a part of you that wants to put it all down and just go turn on Netflix. But great creators are made by their ability to recover from these mistakes — both in the tactical methods to fix them and the mental drive to “just fix it” and power through.

Mistakes are where most of the learning is in the creative process. It’s not only through the feedback loop of trial and error either. The more mistakes you make and navigate through, the better you get at accommodating and recovering from them.

My grandfather was a hobbyist woodworker for much of his life, cranking out hundreds of heirloom pieces over the years. If you ever asked him about making mistakes, he used to say “making mistakes means you’re doing things.” No person is immune from error. By definition, if you aren’t making mistakes, you aren’t really doing anything. Or maybe nothing interesting or challenging.

Progress Report, October 2022

November 1, 2022 • #

This time I’m including the previous month’s to see month-over-month change, so progress (or lack of) is visible.

Health & Habits

Running

  • 8 activities (6)
  • Distance: 26.44 mi (17.33 miles)
  • Total Time: 3:59:24 (2:32:34)
  • Average Pace: 8:56 / mi (9:08 / mi)

Sleep

  • Average: 7:32 / night (7:30)
  • 8 hr nights: 10 (8)

Slightly better on sleeping more this month. Very slightly. Probably would’ve been even better improvement without a cross-country trip in the mix.

Writing

  • Journal entries: 14; 6,428 words (14; 7,292 words)
  • Blog posts: 14 (19)
  • Newsletters: 1 (2)

Media

Reading

The Rise and Fall of the Third Reich, William L. Shirer
░░░░░░░░░░░░░░░▓▓▓▓▓ 78-100%

The Captured Economy, Brink Lindsey & Steven Teles
░░░░░░░░░▓▓▓▓▓▓▓▓▓▓▓ 47-100%

Helgoland, Carlo Rovelli
▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓ 0-100%

A Pattern Language, Christopher Alexander
░░░▓░░░░░░░░░░░░░░░░ 16-19%

Liberal Fascism, Jonah Goldberg
▓▓▓▓▓▓▓░░░░░░░░░░░░░ 0-35%

Where Is My Flying Car?, J. Storrs Hall
▓▓▓▓▓▓░░░░░░░░░░░░░░ 0-32%

Podcasts

  • 18 episodes, 20 hrs, 41 min (18 episodes, 22 hr 46 min)

TV

  • Andor, 4 episodes
  • Veep, 3 episodes
  • World War II in Color, 2 episodes
  • Island of the Sea Wolves, 1 episode

Film

  • The Big Short (2015)
  • Michael Clayton (2007)
  • Too Funny to Fail (2017)
  • Zero Dark Thirty (2012)
  • Ocean’s Eleven (2001)
  • Operation Mincemeat (2021)

Subscribe here to receive my newsletter, Res Extensa, a digest of my latest posts, links, and updates. Currently bi-weekly.