Evidence-Based Decision Making
How to balance quantitative data with qualitative insights. A framework for using evidence to steer product direction and ensure you're solving the right problems.
Product managers live at the intersection of data and intuition. Too much reliance on either one leads to poor decisions. The art is knowing how to balance quantitative metrics with qualitative insights.
Early in my career, I was seduced by data. If the metrics showed improvement, the feature was a success. But I learned that metrics can be misleading—they tell you what happened, not always why it happened or whether it's good for users long-term.
At TBC Corporation, we launched a feature that increased conversion rates by 8%. Success, right? Except customer satisfaction scores dropped, and support calls increased. We'd optimized for short-term conversion at the expense of long-term trust. The data said one thing, but the context told a different story.
This taught me to triangulate evidence. Quantitative data tells you where to look. Qualitative research tells you what you're seeing and why. Together, they give you the full picture.
I developed a framework for evidence-based decision making: Start with the problem (what are users trying to accomplish?), form a hypothesis (what do we believe will help?), define success metrics (both quantitative and qualitative), test the hypothesis, and analyze results in context.
The 'in context' part is critical. A 5% increase in form completions means nothing if those leads are lower quality. A decrease in feature usage might actually be good if it means the feature is working more efficiently.
I also learned to differentiate between metrics that matter and vanity metrics. Page views are a vanity metric. Time to value—how quickly users accomplish their goals—is a metric that matters. Always connect your metrics back to user outcomes and business outcomes.
Data-informed, not data-driven. Use evidence to inform decisions, but remember that the best product managers combine rigorous analysis with judgment, empathy, and vision.