How to Synthesize User Research Without Cherry-Picking
- What research synthesis actually means
- The common mistake teams make
- The three patterns that prevent cherry-picking
- How to practice synthesis as a daily habit
Research synthesis is a crucial step in transforming user insights into actionable strategies. However, many teams fall into the trap of cherry-picking data that supports their existing assumptions. This article explores the nuances of effective synthesis and provides practical methods to ensure a balanced, insightful approach that can genuinely inform your product decisions.
What Research Synthesis Actually Means
At its core, research synthesis is the process of distilling customer conversations, observations, and data points into a few clear themes that drive decision-making. The output typically consists of a concise document outlining three to seven key themes, each supported by evidence and actionable recommendations.
It's essential to differentiate synthesis from summarization. Summarization merely recounts what was said during interviews, while synthesis reveals the deeper meanings that emerge when data points are viewed collectively. For example, a B2B SaaS team conducted 20 customer interviews to investigate churn, initially believing that pricing was the primary factor behind customer attrition. However, through synthesis, they discovered that the real culprits were poor onboarding, lack of internal champions, and the presence of competing tools. This revelation not only redirected their focus but also prevented a costly mistake.
The Nielsen Norman Group's research emphasizes that teams employing systematic synthesis techniques are more likely to uncover insights that contradict their initial hypotheses. This friction acts as a safeguard against bias, ultimately leading to better-informed decisions.
The Common Mistake Teams Make
One of the most prevalent pitfalls in synthesis is confirmation bias disguised as analysis. Teams often enter research with a specific hypothesis, conduct interviews, and then curate evidence that backs their preconceived notions. This leads to a false sense of validation and the subsequent development of features that may not resonate with users.
The mistake lies in synthesizing information that favors the existing decision-making process rather than challenging it. A more effective approach asks, "What data might change my mind?" before considering, "What supports my original plan?"
Other common errors include over-reliance on memorable quotes from standout users, which can skew perception away from broader patterns, and conducting synthesis in isolation. When only one person interprets the data, their biases may go unchecked. However, having multiple team members review and discuss the same data fosters richer insights and highlights ambiguous areas that warrant further investigation.
Lastly, synthesis that fails to prompt behavioral changes is ineffective. If the resulting insights don’t alter any decisions, then the synthesis process hasn't fulfilled its purpose.
The Three Patterns That Prevent Cherry-Picking
To cultivate effective synthesis, implement these three patterns. While they may require more time than a narrative approach, they yield far more valuable insights.
-
Tag Every Data Point Before You Theme It: As you review each interview, tag specific moments that highlight pain points, jobs, workarounds, and contradictions. Establish a consistent tagging schema. By tagging exhaustively before theming, you engage deeply with each interview on its own terms, rather than skimming for data that fits your hypothesis. Tools like Miro or Mural for affinity mapping or even a simple spreadsheet can facilitate this process.
-
Surface What Would Disprove Your Hypothesis: For each emerging theme, identify contradictory evidence. For instance, if you identify that "customers want bulk export," note that "three customers mentioned they don’t have large enough datasets to require it." By surfacing these contradictions alongside supporting evidence, you foster a culture of honesty and uncover deeper truths.
-
Have a Second Person Review and Disagree: Synthesis becomes sharper when a colleague reviews the same data and proposes their own themes. Comparing notes highlights areas of agreement, indicating strong signals, while disagreements reveal ambiguities that need further exploration. Although this may slow the process, it distinguishes insight from mere confirmation.
For those interested in honing their interview techniques, consider exploring customer interviews that work. For a broader understanding of the discovery process, product discovery basics provides valuable insights. Finally, to enhance your skill in navigating uncertainty, mental models for work is a great resource.
How to Practice Synthesis as a Daily Habit
Synthesis is a skill that improves with practice. Many teams fail to get enough reps because they treat synthesis as a one-off project rather than an ongoing practice.
Incorporate daily synthesis habits into your routine. After each customer conversation, distill insights into three sentences: what the customer did, what mattered most in their comments, and what surprised you. This exercise imposes a compression that encourages meaningful interpretation.
Once a month, conduct a deeper synthesis review of that month's conversations. Look for recurring patterns, contradictions to your prior assumptions, and unprompted comments that appear across multiple interviews. This monthly cadence captures insights that individual write-ups may overlook.
To build your synthesis muscle, engage in a weekly micro-learning exercise. Select three recent interviews, and spend 30 minutes synthesizing them. This repetition will help transform synthesis from a chore into a skill. Expect your first 50 attempts to be rough; the next 50 will be competent. It’s only after hundreds of exercises that synthesis begins to feel natural.
A useful weekly habit is to revisit one interview from the past month and ask: "What would I tag differently now?" This reflection can reveal insights you missed during the initial synthesis.
What Good Looks Like
Effective synthesis is evident when it leads to tangible changes in decision-making.
Indicators that your synthesis is on the right track include:
- Teams abandoning features they were excited about due to lack of supporting evidence.
- New features emerging in response to insights that were previously overlooked.
- Synthesis results that don’t merely confirm the existing roadmap but actively reshape it.
When synthesis prompts the team to acknowledge contradictions in the data, it signifies a commitment to intellectual honesty. Phrases like "we expected to see X but actually saw Y, suggesting Z" illustrate a mature approach to data analysis.
Ultimately, the goal is to make synthesis a moment of truth where team assumptions are rigorously tested. The roadmap should reflect actual user behavior rather than the desires of the team. Research consistently shows that teams that actively seek out disconfirming evidence in their synthesis make better product decisions over time. The trade-off is a slower synthesis process, but the benefit is a more refined product that aligns with user needs.
Conclusion
Synthesis goes awry when teams focus solely on supporting their hypotheses. To combat bias, adopt practices like tagging every data point, surfacing contradictions, and engaging in collaborative review. By embedding these methods into your workflow, you enhance your research synthesis and, ultimately, your product outcomes.
For those looking to improve their research synthesis without overwhelming their schedules, Omie offers tailored learning, delivering one lesson per day based on your role and goals. Start free for 14 days →