Facebook’s recent mobile app redesign, for which I led research, involved a lot of interesting UX challenges. One of the biggest was figuring out how to measure user sentiment. The team needed to know: are we accomplishing our goals of making the app easier to navigate & visually cleaner? In other words, the central question was: are we making Facebook better? 
 
To find out, we turned to sentiment surveys, which were desirable over other approaches for a few reasons. Qualitative testing, which drove the early design ideation and then iteration, ultimately doesn’t scale with a global product. And purely quantitative metrics like clicks and views don’t always reflect the kind of improvements we were aiming to make. For example, in a navigation redesign, less time spent on site may actually be a good sign, since people can get from point A to point C more quickly. 
 
Measuring user sentiment — in this case, using statistics in support of hypothesis-testing to compare two measurements — is tricky. But it was critical to our understanding of how well the redesign was working for people.

Changes made in the News Feed redesign

The problem with satisfaction

What we did not do was ask people How satisfied are you with the Facebook app? That may sound like an elegant survey solution, but in practice, it’s often a dead-end, both in terms of generating insights and quantitatively describing the user experience. I like to call this approach the “satisfaction trap.”
 
Say you’re running a brunch spot and you’ve made a change to the waffle batter. You want to know whether your customers are enjoying the change. Asking them How much do you like Chez X? might yield some useful information, but the waffles would have to be unbelievably good or bad — and ordered by everyone — to overshadow the rest of the brunch experience. From a statistical perspective, you’d have to question a really large number of diners to accurately measure such a small change.

When a product is as complex as Facebook, it’s even more difficult to measure specific product changes against generic questions about satisfaction. People use Facebook for many reasons: connecting with friends and family, entertainment, catching up on news, and so forth. How satisfied are you with Facebook? will mean very different things to different people. They might be asking themselves: “Is Facebook good at connecting me with my family?” or “I use ‘for sale’ groups on Facebook — how is that experience?” and so on. Answers might vary further based on what the person did the last time they used Facebook (e.g., watch a friend’s video, find an interesting link, get lots of Likes on their engagement announcement).
 
The lack of reliability from the broadness of the question introduces noise into the sentiment measurement, which makes it harder to find the true signal when doing statistical testing, which, in turn, makes it impossible to say confidently that a given change to the product is a positive experience for the user.

… a satisfying solution

Recognizing that summary measures don’t pick up incremental product changes is the first step in avoiding the satisfaction trap. To be clear, satisfaction questions aren’t bad — they are just representative of a measurement approach that is often too generic to be effective for specific product changes. We might get answers with a large enough sample size, but people don’t come to Facebook to take surveys.
 
The second step to avoid the satisfaction trap to create a set of questions around the experiences that did change and do matter. In other words, survey questions need to tap into the specific things that contribute to satisfaction: opinions of certain features, aspects of utility, usability, etc. For the redesign, we developed a set of questions that focused peoples’ attention on the changes we’d made. Instead of abstract questions about the general Facebook experience or brand, we knew we wanted to ask about navigation and the visual appearance of News Feed. In an ideal world, to increase reliability, you’d also want to ask multiple questions around the same underlying concept. 
 
Finally, we made sure to deeply understand how people think about and describe the topics we were asking about, in this case navigation and the visual design of stories in News Feed. This helped us be confident our questions would resonate with the way people actually experience Facebook. So, we asked about things such as the visual appearance being clean, and how easy it is to know where you’re going when leaving News Feed. This is akin to the brunch spot moving away from asking How much do you like Chez X? and toward asking How much do you like the waffles?

To restate, avoiding the satisfaction trap is accomplished through recognizing when a summary measure is the wrong approach, identifying the experiences that did change and do matter, and ensuring that the manner in which you ask about peoples’ opinions related to those changes is clear, direct, and matches how they think about their experience.

These bespoke survey questions require more work to create, and they aren’t a perfect solution. They have to be tailored to the product change, it takes extra work to do interviews and testing to vet them, and down the road it can be hard to compare product change X to change Y to change Z because each may have needed a different measurement strategy. But the power to measure and deeply understand the user experience makes the work worthwhile.
 
In the end, the effort paid off for us because the questions we created for measuring user sentiment about the redesign were essential for shipping the product. We used them during development to iterate on designs — ensuring with each new version that we were making Facebook visually cleaner and easier to understand. And as we wrapped up development these questions let the team, and the company, feel confident that the changes were the right ones to be making. 
 
If you’re having trouble measuring sentiment, remember the satisfaction trap, and ask yourself: How are the waffles?

Thank you to Lauren Scissors & Curtiss Cobb for their feedback on an earlier version of this post.

Author: Nick Merola, Researcher at Facebook

Illustrator: Sarah Lawrence