This is an interesting look into how an effective team works through the weeds of a product design review. I love how it shows the warts and complexities of even seemingly-simple flow of sending a batch email in an email client. So many little forking paths and specific details need direct thinking to shape a product that works well.
Building new things is an expensive, arduous, and long path, so product builders are always hunting for means to validate new ideas. Chasing ghosts is costly and deadly.
The “data-driven” culture we now live in discourages making bets from the gut. “I have a hunch” isn’t good enough. We need to hold focus groups, do market research, and validate our bets before we make them. You need to come bearing data, learnings, and business cases before allowing your dev team to get started on new features.
And there’s nothing wrong with validation! If you can find sources to reliably test and verify assumptions, go for it.
But these days teams are compelled to conduct user testing and research to vet a concept before diving in.
This push for data-driven validation I see as primarily a modern phenomenon, for two reasons:
First, in the old days all development of any new product was an enormously costly affair. You were in it for millions before you had anything even resembling a prototype, let alone a marketable finished product. Today, especially in software, the costs of bringing a new tool to market are dramatically slashed from 20 or 30 years ago. The toolchains and techniques developed over the past couple decades are incredible. Between the rise of the cloud, AWS, GitHub, and the plethora of open source tools, a couple of people have superpowers to do what used to take teams of dozens, and thousands of hours before you even had anything beyond a requirements document.
Second, we have tools and the cultural motivation to test early ideas, which weren’t around back in the day. There’s a rich ecosystem of easy-to-use design and prototyping tools, plus a host of user testing tools (like UserTesting, appropriately) to carry your quick-and-dirty prototypes out into the world for feedback. Tools like these have contributed to the push for data-driven-everything. After all, we have the data now, and it must be valuable. Why not use it?
I have no problem with being data-driven. Of course you should leverage all the information you can get to make bets. But data can lie to you, trick you, overwhelm you. We’re inundated with data and user surveys and analytics, but how do we separate signal from noise?
One of my contrarian views is that I’m a big defender of gut-based decision making. Not in an “always trust your gut” kind of way, but rather, I don’t think listening to your intuition means you’re “ignoring the data.” You’re just using data that can’t be articulated. It’s hard to get intuitive, experiential out of your head and into someone else’s. You should combine your intuitive biases with other objective data sources, not attempt to ignore them completely.
I’m fascinated by tacit knowledge (knowledge gained through practice, experience). It differentiates between what can be learned only through practice from what can be read or learned through formal sources — things we can only know through hands-on experience, versus things we can know from reading a book, hearing a lecture, or studying facts. Importantly, tacit knowledge is still knowledge. When you have a hunch or a directional idea about how to proceed on a problem, there’s always an intrinsic basis for why you’re leaning that way. The difference between tacit knowledge and “data-driven” knowledge isn’t that there’s no data in the former case; it’s merely that the data can’t be articulated in a spreadsheet or chart.1
I’ve done research and validation so many ways over the years — tried it all. User research, prototype workshopping, jobs-to-be-done interviews, all of these are tools in the belt that can help refine or inform an idea. But none of them truly validate that yes, this thing will work.
One of the worst traps with idea validation is falling prey to your own desire for your idea to be valid. You’re looking for excuses to put a green checkmark next to an idea, not seeking in-validation, which might actually be the more informative exercise. You find yourself writing down all the reasons that support building the thing, without giving credence to the reasons not to. With user research processes, so many times there’s little to no skin in the game on the part of your subject. What’s stopping them from simply telling you want you want to hear?
Paul Graham wrote a post years ago with a phenomenal, dead-simple observation on the notion of polling people on your ideas. A primitive and early form of validation. I reference this all the time in discussions on how to digest feedback during user research (emphasis mine):
For example, a social network for pet owners. It doesn’t sound obviously mistaken. Millions of people have pets. Often they care a lot about their pets and spend a lot of money on them. Surely many of these people would like a site where they could talk to other pet owners. Not all of them perhaps, but if just 2 or 3 percent were regular visitors, you could have millions of users. You could serve them targeted offers, and maybe charge for premium features.
The danger of an idea like this is that when you run it by your friends with pets, they don’t say “I would never use this.” They say “Yeah, maybe I could see using something like that.” Even when the startup launches, it will sound plausible to a lot of people. They don’t want to use it themselves, at least not right now, but they could imagine other people wanting it. Sum that reaction across the entire population, and you have zero users.
The hardest part is separating the genuine from the imaginary. Because of the skin-in-the-game problem with validation (exacerbated by the fact that the proposal you’re validating is still an abstraction), you’re likely to get a deceitful sense of the value of what you’re proposing. Would you pay for this? is a natural follow up to a user’s theoretical interest, but is usually infeasible at this stage.
It’s very hard to get early critical, honest feedback when users people have no reason to invest the time and mental energy thinking about it. So the best way to solve for this is to reduce abstraction — give the user a concrete, real, tangible thing to try. The closer they can get to a substantive thing to assess, the less you’re leaving to their imagination as to whether the thing will be useful for them. When an idea is abstract and a person says “This sounds awesome, I would love this”, you can safely assume they’re filling in unknowns with their own interpretation, which may be way off the mark. Tangibility and getting hands-on with a complete, usable thing to try, removes many false assumptions. You want their opinion on what it actually will be, not what they’re imagining it might be.
In a post from a couple years ago, Jason Fried hit the mark on this:
There’s really only one real way to get as close to certain as possible. That’s to build the actual thing and make it actually available for anyone to try, use, and buy. Real usage on real things on real days during the course of real work is the only way to validate anything.
The Agile-industrial complex has promoted this idea for many years: moving fast, prototyping, putting things in market, iterating. But our need for validation — our need for certainty — has overtaken our willingness to take some risk, trust in our tacit knowledge, and put early, but concrete and minimal-but-complete representations out there to test fitness.
De-risking investments is a critical element of building a successful product. But some attempts to de-risk actively trick us into thinking an idea is better than it is.
So I’ll end with two suggestions: be willing to trust your intuition more often on your ideas, and try your damnedest to smoke test an idea with a complete representation of it, removing as much abstraction as possible.
I’d highly recommend Gerd Gigerenzer’s book Gut Feelings, which goes deep on this topic. For a great précis on the topic, check out his conversation from a few years back on the EconTalk podcast. ↩
Fernando Borretti gets real on the likely-useless time sink that is the perfectionist “tools-for-thought” space:
People have this aspirational idea of building a vast, oppressively colossal, deeply interlinked knowledge graph to the point that it almost mirrors every discrete concept and memory in their brain. And I get the appeal of maximalism. But they’re counting on the wrong side of the ledger. Every node in your knowledge graph is a debt. Every link doubly so. The more you have, the more in the red you are. Every node that has utility—an interesting excerpt from a book, a pithy quote, a poem, a fiction fragment, a few sentences that are the seed of a future essay, a list of links that are the launching-off point of a project—is drowned in an ocean of banality. Most of our thoughts appear and pass away instantly, for good reason.
As an aspiring tools-for-thought user, who keeps his own notes in a “PKM”, this piece rings true (and painful). The reality of personal wikis, totalizing note-taking apps, and people with their fully hyperlinked massive graphs of notes is that most of this is for naught. It largely ends up being nothing more than a playground for creating big structures of information that don’t solve a problem.
I’ve spent tons of time fiddling with notes in Roam, Obsidian, Logseq, and every productivity tool under the sun, even while being very aware of the fact that _tools don’t make you productive_on their own. Yet still, most of what this post describes is the honest reality: we can excuse our way to spending huge amounts of time trying to build a castle of an immaculate hyperlinked memex, but really Apple Notes, a calendar app, and a checklist tracker will solve the problem just fine.
I recently was getting our Silhouette vinyl cutter back out, since Colette hasn’t used it in a while. The software for the thing is still ancient and janky — hard to believe anyone uses it at all.
But I found this new tool for creating vector-based designs for any kind of cutting machine — vinyl printers, CNCs, laser cutters. It looks like an excellent general tool for creating SVGs you can import into all manner of cutting machine proprietary applications.
This year I got to several books that have been on my list for years, excited to finally dig into them.
Here’s the full list, with my favorites ⭐️ starred:
⭐️ Where Is My Flying Car?
⭐️ Helgoland
Making Sense of the Quantum Revolution
⭐️ The Rise and Fall of the Third Reich
A History of Nazi Germany
The Captured Economy
How the Powerful Enrich Themselves, Slow Down Growth, and Increase Inequality
Scene and Structure
⭐️ Underland
A Deep Time Journey
Childhood's End
The Tacit Dimension
⭐️ The Law
The Future and Its Enemies
The Growing Conflict Over Creativity, Enterprise, and Progress
Knowledge and Decisions
The Conservative Sensibility
The Five Dysfunctions of a Team
A Leadership FableI’m late getting my November update posted. November (and still, in December) was a rollercoaster of a month. Just so much happening with professional and personal, I’ve hardly had a moment to do much at all — neither focusing on any personal progress goals, nor writing or other fun side deals.
My running performance was pitiful. I did 5 runs, but honestly I’m surprised it was even that many. Feels like I’m totally off the wagon on that. I did alright on my sleep, but I swing too much back and forth to be a healthy pattern. I’ll do a string of 5-6 hour sleep nights, punctuated by sleeping 10 hours the next. The see-saw effect isn’t intentional. Something I need to focus more on building a pattern with.
Public writing didn’t do great, only a few blogs before I fell off and didn’t get any more writing done. I did better on the personal journal, though. At least for the first half the month.
Reading also suffered some. I feel like I didn’t spend any time with a book at all.
Liberal Fascism, Jonah Goldberg
░░░░░░░▓▓▓▓▓░░░░░░░░ 35-60%
Where Is My Flying Car?, J. Storrs Hall
░░░░░░▓▓▓▓▓▓▓▓▓▓░░░░ 32-80%
| 12 episodes — 18 hr, 16 min | 18 episodes — 20 hrs, 41 min |
None
This piece from Anton Howes gets at one of the key insights about how innovation works: it doesn’t happen through sudden bursts of insight from thin air — it requires the combination of the right simmering ingredients and a person in search of solutions to specific problems:
Santorio’s claim, it seems, is safe. But in this lies an important lesson for all would-be inventors. The inverted flask experiment had been around for centuries, and even been understood since ancient times as being caused by hot and cold. So its application as a thermometer was extremely low-hanging fruit. The likelihood of it being interpreted as a temperature-measuring device might have increased somewhat in the mid-sixteenth century, when we find the first mentions of it being done using a glass flask rather than an opaque metal container. Yet even then, the visible rise and fall of the liquid in the open bucket, rather than the flask, could always have been noted and measured against a scale in much the same way. What Antonini’s letter also shows us is that even when a scale was applied to the experiment, an ingenious person who knew their cutting-edge science like he did could still fail to appreciate the potential of what they had done.
In this case, the “inverted flask” had existed for many year, and Santorio was actively searching for ways to measure temperature. Innovation requires the right mixture of “prior art” and the willful intent in search of solutions. Progress doesn’t happen automatically!
This paper describes an experiment to merge freeform document-style text and structured computations into a single interface.
We think a promising workflow is gradual enrichment from docs to apps: starting with regular text documents and incrementally evolving them into interactive software. In this essay, we present a research prototype called Potluck that supports this workflow. Users can create live searches that extract structured information from freeform text, write formulas that compute with that information, and then display the results as dynamic annotations in the original document.
The idea is to prioritize the open-endedness of documents with the benefits of some schema, to enable computations and structure. Documents are powerful for their flexibility to accommodate all kinds of information, but the trade-off is a static environment, whereas applications can be dynamic and responsive. Potluck gives a look at what happens when you get some of both worlds.
Try out some of the examples. It’s impressive and shows promise for the future of the low-code / build-your-own-software movement.
I talk all the time about trial and error. The freedom to let yourself make mistakes, and the skill to make sure they’re not too destructive, are superpowers. With every interesting innovation, company, or product, you’re seeing the late stage of a long chain of missteps and failure. As long as you have the right mindset, mistakes are learning.
We talk about this as a product team — short cycles, iteration, feedback loops — ways to navigate toward broader visions while surviving and building something increasingly useful along the way. I also talk about it with the kids. The more you practice hitting off the tee the better you’ll get at hitting the ball. The more you draw pictures the better you get at it. Practice through the frustration. I try to reinforce with them that everyone that’s great at something got their through an incredible volume of failure and shortcoming before the skill you see today.
If you’ve ever built anything physical, like woodworking, crafting, or DIY stuff around the house, you’ll be familiar with making mistakes, often costly ones. There’s no frustration quite like taking a furniture workpiece you’ve glued up from other parts, honed, mortised, and sanded and making a miter in the wrong place, or cutting it down to length too short. Hours and hours of work can vaporize in a second. I’ve made project mistakes like this so many times, and each time there’s a part of you that wants to put it all down and just go turn on Netflix. But great creators are made by their ability to recover from these mistakes — both in the tactical methods to fix them and the mental drive to “just fix it” and power through.
Mistakes are where most of the learning is in the creative process. It’s not only through the feedback loop of trial and error either. The more mistakes you make and navigate through, the better you get at accommodating and recovering from them.
My grandfather was a hobbyist woodworker for much of his life, cranking out hundreds of heirloom pieces over the years. If you ever asked him about making mistakes, he used to say “making mistakes means you’re doing things.” No person is immune from error. By definition, if you aren’t making mistakes, you aren’t really doing anything. Or maybe nothing interesting or challenging.
This time I’m including the previous month’s to see month-over-month change, so progress (or lack of) is visible.
Slightly better on sleeping more this month. Very slightly. Probably would’ve been even better improvement without a cross-country trip in the mix.
The Rise and Fall of the Third Reich, William L. Shirer
░░░░░░░░░░░░░░░▓▓▓▓▓ 78-100%
The Captured Economy, Brink Lindsey & Steven Teles
░░░░░░░░░▓▓▓▓▓▓▓▓▓▓▓ 47-100%
Helgoland, Carlo Rovelli
▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓ 0-100%
A Pattern Language, Christopher Alexander
░░░▓░░░░░░░░░░░░░░░░ 16-19%
Liberal Fascism, Jonah Goldberg
▓▓▓▓▓▓▓░░░░░░░░░░░░░ 0-35%
Where Is My Flying Car?, J. Storrs Hall
▓▓▓▓▓▓░░░░░░░░░░░░░░ 0-32%
Subscribe here to receive my newsletter, Res Extensa, a digest of my latest posts, links, and updates. Currently bi-weekly.