A guide to the most common methods and how to use them

Presented by Aldana Vales and Erin Boon

INTRODUCTION 

There are many moments in the product lifecycle when reaching out to people for feedback makes the difference between a successful launch and a failed initiative. Knowing not only which methods to turn to but also how to use them effectively is key when gathering insights that will inform decisions and shape product strategy.

All good product research starts with a question. That is, with something you want to find out. It can be whether people will prefer design A over design B, if your ideas will resonate at all with your audience, or just wanting to know how a specific generation interacts with the news, for example. What matters, at the beginning of this process, is that you have a clear idea of what you want to know. After that, you can move on to finding the right approach and tools to get answers.

Think of the nature of your question. Will you need narrative insights to understand the problem? Can it be answered with numbers? This will help you decide whether you need a qualitative or quantitative approach, although product research often combines both. Qualitative data is descriptive and can help you find the reasons behind your audiences’ behaviors, while quantitative data is numeric. It can be measured and answers questions like “how many?” “how much?” and “for which audience?” 

In this guide, we share a list of some of the most common methods. Before you choose one, think of what you want to achieve with your research and how you expect to use its findings. That will also help you narrow your audience and decide which method to use.

IN PRACTICE

Use the tips and concepts in this section to cultivate a product culture in your newsroom

Use surveys to reach a large group of people

Surveys give people time to respond at their own convenience and can generate a large number of responses. If you’re looking for a platform to create yours, you can start with Google Forms, SurveyMonkey or Typeform.

In your survey design, you should combine closed-ended questions that offer a predetermined set of answers (e.g. “Which of the following statements best describes your experience?”) with open-ended questions, which give people the opportunity to write in their own responses (e.g. “How would you describe this product?”). At the end, always give people the chance to provide additional, optional, open-ended feedback, e.g. “Is there anything else you’d like to share with us?” You can find unexpected insights among those responses.

Go deeper with user interviews

Interviews allow you to follow up with specific people to learn more about their thoughts and choices. In these sessions, you can gain insight into what motivates people to use your products or services.

Before you host an interview, create a script that guides how you want the discussion to go: introduce yourself and the research topic, let the interviewee know that they can ask any questions along the way and prepare the list of questions that you want to ask them.

See people interacting with your ideas

When you have a prototype or early designs that you’re ready to share with your audiences, user testing is the way to go to get their reactions. This method can be moderated (you will lead the sessions) or unmoderated (you will rely on a platform to get testers’ feedback).

During usability test sessions, observe testers’ behavior and ask open-ended questions to get a comprehensive understanding of what people experience as they interact with your product. Avoid questions that can be answered with a simple yes or no.

Take advantage of accurate data

Most news organizations have access to an abundance of audience data, and it is very easy to be distracted, even overwhelmed by analytics. However, a data-driven product strategy does not mean that you need to monitor all of the numbers all of the time. 

Instead, center your product strategy around a small set of core metrics, such as weekly articles per subscriber, monthly engaged newsletter readers, or average order value. Seek to frame the insights that you gather in actionable ways that improve upon a core metric. You can always dig deeper into your data sources to investigate when one of your core metrics shifts or stalls.   

Test which version performs better

Some research techniques can recommend a product change, but only a randomized test can measure the impact of that change at scale. A/B testing is the practice of showing two versions of a feature to a randomly split audience at the same time and comparing to see which resulted in more of the desired outcome. 

Strategic testing should be informed by business experience, past audience research, and strong hypotheses. You won’t be able to A/B test every version of your product, so test proposals must be evaluated and prioritized. If you have a hypothesis about how readers will interact with a feature in a way that improves one of your core metrics, but you’re missing the certainty to unblock a decision, that is a perfect opportunity to run an A/B test.

TERMS

Definitions for product terms referenced in this guide are sourced from NPA’s crowd-sourced product glossary

Hypothesis

When we build new product features or make changes to a product experience, we expect our readers to interact with them differently than they did before. The hypothesis is our proposed explanation of this relationship. A good hypothesis can be formatted as an “if… then… because” statement, and it must include a logical connection between the tested product change and the expected change in audience behavior.

Synthesis

This is the process that will turn your data into actionable insights. To do this, you need to organize the data and find patterns. Go back to your goals and respond to the following: How do these patterns answer your research questions?

RELATED READINGS / RESOURCES

Conducting Audience Research - Emily Roseman and Emily Goligoski

Just Enough Research - Erika Hall

Google Analytics Academy - Google Analytics

A Refresher on A/B Testing - Amy Gallo

ABOUT THE AUTHORS

Aldana Vales

Aldana Vales is a DC-based journalist working at the intersection of audience and business in the media industry. She’s the manager of Research & Product Insights at The Atlantic, where she conducts qualitative research to help identify readers and listeners’ needs. In a past life, she covered U.S. politics for Argentine media.

Erin Boon

Erin Boon has been helping news organizations develop strategies through product science since leaving academia. She holds a MS in Statistics & Data Science and a PhD in Celtic Languages & Linguistic Theory. She is based in New York, where she is a Senior Data Scientist at The Atlantic.

Puedes leer la guía traducida y adaptada al español aquí: Primeros pasos con la investigación de productos

Você pode ler o guia traduzido e adaptado para português aqui: Introdução à pesquisa de produtos