Mastering Product Research – a guide to Generative and Evaluative approaches

When I think about research, I am gauging when I need to Listen or when I need to measure.  The way I approach it is based on:

• Generative “Listen”: Research techniques that don’t necessarily start with a hypothesis but result in many new ideas.

• Evaluative “measure”: Research based on testing specific hypotheses to get clear yes or no results.

I then decide if I want to understand my customers/market or the product and build out the hypotheses that I want to assess.

• Customers/market: I want to understand the problems, needs or goals to identify opportunities to solve or attract them to my product or service.

• Product: I want to validate that the product addresses customer/market needs and goals and is a proposition that would be attractive based on features, pricing or other factors.

I came across a quadrant model that I have been using for a while, depicted here, which I use to support the process of planning for research, as I find it easier to differentiate and focus on why, who and how we will conduct the research.

How I use the model

Generative Market Research

So if you wanted to focus on Generative Market Research, you would be operating in the top left corner.   Where you would be likely want to understand:

• Who is my customer?

• What characteristics do my customers have (segmentation)?

• What are the jobs to be done (referring to the activities)?

• What are their needs, goals or pain points?

• What defines value for them?

• What would they like in a product or service? What would resonate with them the most?

• What are my customer segments? Are they clear or too broad? How would I target them successfully?

• What drives them, and where do they explore options to obtain my product?

With Generative research, you will deal with qualitative and quantitative data.  Qualitative data are the customer insights and stories shared during your interviews, whereas quantitative data is data-based, statistical or numerical.  With Qualitative data, we want to thematically model the insights to reveal trends or patterns in customer sentiment and should be supported by quantitative data.  

For example, Customers aged between x & y are most engaged by the z feature/benefit because of (Insights: Needs, Goals & Pain Points = solve opportunity of feature/benefit) they represent XX% of our growth market and would attract revenue of XX% per year.

Evaluative Market Research

In contrast, we assess a specific hypothesis as part of this research. You would want to understand things like:

• What is the cost to serve?

• How much would it cost to sell?

• What are they willing to pay?

• Which features/options/benefits are of most value?

• Can we scale marketing?

You may do this by running a landing paste test, run a sales pitch, or conjoint analysis (Market Analysis) to understand relative positioning for a few value propositions.  Other methods could be:

• 5-second tests (First impressions)

• Comprehension (Value Proposition)

• Data mining, market research

• Surveys

• Smoke tests (Readiness)

o   Video (experience, feature, concept, marketing)

o   Landing page (prototyping / A/B testing)

o   Sales Pitch (sales)

o   Flyers (marketing trigger/CTA)

o   Pocket test (prototype)

o   Event (Marketing/promotion)

o   Fake door (Value proposition)

o   High bar (Behavioural/motivators)

Generative Product Research

If you have validated that you have a problem to solve? Who your market is, and the value it creates, you are ready to start asking, “How would a product/service solve this need or achieve a goal for the customer?” 

You do not want to introduce bias here… and that can be challenging. Gaming an outcome here will only limit your success.  I have found that designers do this best, as they are trained in how to use question styles that seek to understand.

At this stage, you would be assessing:

• What defines value for my customer

• What pain points could we solve?

• What form should my product/service have?

• What is the minimum viable product/feature?

• What is the minimum viable experience?

• How should we prioritise our efforts?

• What will be the impact this will have on our customers' lives?

• How are they solving this need today in the absence of this option?

There are many techniques that you can use to conduct this research, and here are a few:

• Solution Interview (task flow, lo-fi concept)

• Contextual inquiry/ethnography

• What’s on your mind (Customer constraints or problems)

• Demo Pitch (Motivation to purchase, recommend take next step)

• Concierge test (for MVP – assessing goals, needs and task)

• Competitor Usability (landscape review of competitor product/service)

• Want ad (Results to be achieved – feeds into MVE & MVP)

 Remember, as before, you are going to generate qualitative data. I have also shared the early results with marketing as it informs customer sentiment.

Evaluation Product Research

Many tools can be used to do this, and many of your success measures will be generated based on this type of research or be the measure for the research.   I have also used this to inform a baseline to forecast and measure performance.

So, in this research, you want to validate:

• Take up – are people using it?

• Time to complete – is it meeting expectations?

• Is the solution working?

• Is there a better solution (A/B Testing)

• How could/should we optimise this?

• Are there Surprises, Contradictions, Tensions, Friction, Failure for our customers?

• What do customers like/dislike?

• Why do they take a particular course of action?

You can use many ways to undertake this. Remember you are validating against a clear hypothesis to gain clarity on an aspect of your product or feature.  In the later stages of product development, you will get your design validated and complete user testing and once in production your KPIs or OKRs, such as NPS, Active Users or other metrics. Outside of these options, you could use:

• Lo – Med prototypes.

• Storyboards

• Clickable prototypes

• Live

• Remote

• Functioning Products

• Analytic dashboards

• Product scorecard

• Surveys – Product market fit

Conclusion

I hope this provided some insights into generative and evaluative research and provides you with a valuable resource to assess your product or services. And an introduction to a model that has helped me qualify the need and approach. I only wish I recalled who to credit it back to…

Research is about gaining valuable market & product insights and informing your development and maintenance efforts.

So, hopefully, this has provided some insights and will benefit you in the future.  If you have any questions, reach out.