medium bookmark / Raindrop.io |
Chief data scientist + UX researcher @shortstacklab / founder @awarrenresearch / Former literacy academic, writer, traveler, ham radio operator, half-elf rogue.
When I first started usability testing, my first thought was: who do I test?
Short answer: It depends on the focus of the research.
Long answer: Testing branding or discovery is helpful to new users; complex back-end tests are useful for longtime users. Putting together a group of people for every test is an important, but time-consuming, part of usability testing.
As the resident researcher at ShortStack, it’s my job to explore and question every part of our platform. ShortStack is robust and versatile; we’re a service that helps people create forms, landing pages, and promotions, and manage the leads they collect.
To streamline my research process, I recently created a research panel of ShortStack customers. Some UX researchers call this a “research group,” or a “resource.” Essentially, a research panel is a group of new or existing users of your platform who opt in to assist you with your research. Here’s why and how to make one.
Analytics alone aren’t enough for me to make meaningful conclusions and recommended design changes to my team. We want direct input from our customers. The panel serves two purposes:
Customer buy-in. Roping customers into our research gives them transparency into our process and a stake in the platform.
Convenience. The panel means we don’t have to seek out people for every study; we have a pool of people to whom we can always reach out, so it saves us time trying to find the “right” people for a particular study.
Essentially, a research panel allows me to do more experiments faster, with less of the administrative busy-work.
You’ll need a way to store contact information. Some companies feature an opt-in checkbox upon signup. Because we wanted a broad range of participants who were familiar with our platform, we sent out an email to all of our customers who were subscribed to our email list.
My email to ShortStack customers
Participants filled out their name and email address on a landing page. This database makes it easy for me to communicate with my research panel directly, and they can opt out at any time.
The incentive to participate was twofold: they get to impact the platform development actively, and they get rewarded for it.
I don’t like to promise a specific reward and create bias, but I do like participants to know that their time and input is appreciated. Also, since these are people I plan on communicating with frequently, I want to ensure that the rewards are cost-effective for ShortStack, too. (Our rewards for participation include additional features or allowances on their account, or ShortStack swag, like t-shirts or stickers.)
When you put together a research panel, consider the following:
How do the demographics and behaviors compare to your user base as a whole? It’s important that the panel is an accurate reflection of your users.
I analyzed my panel by looking at which plan they were subscribed to (if any), and how long they’ve been with ShortStack. If my whole panel consisted of longtime users, their perspective might be different than a brand new user, so I wanted to ensure I had a balance. You can always continue recruiting if your first round doesn’t reflect the audience you hope for, or you can create segments within your panel.
I also ran a test survey to my research panel and our main email list to compare responses. Doing this every time would defeat the purpose of the panel, but it helped to see if the responses were reflective of our larger user base.
Depending on what type of research you plan to conduct (qualitative vs. quantitative, or mixed-methods, if you’re like me), the sample size is important. A 20 person research panel is helpful for observational usability testing, but not for making statistical conclusions from a survey, for example.
My research panel has more than 300 participants, so I feel comfortable that surveys, like a UsabilityHub Question Test, would give me a good number of responses.
Once you’ve formed and understood your panel, keep these in mind:
It’s exciting to have a group of people who want to help, but don’t bombard them with research projects every single day (as tempting as that may be!). You’ll get better participation if you space out your projects. But be consistent in whatever schedule you determine.
You have people who are willing to aid in your quest, so this is a great chance to try new experiments! I’ve personally never run a preference test, so now that I have my research panel formed, that’s one of the first tests I will run.
Participation should be a short and simple process. Consider efficient survey design and UX as you’re testing. Reduce friction so that you can get the data you need, and let them get on with their day.
It can be simple for you, too. Send your panel a link to a UsabilityHub test, and you can quickly review your results. Tests don’t need to be complicated for you or your panel to give you great insight for your team.
Usability testing helps make your platform or service the best it can be, so don’t sit on your findings from your panel experiments. I’ve found that my team doesn’t need an overly-complicated, stuffy report; they want napkin drawings or numbers backed with real data.
Once you’ve run one experiment with your panel, plan your next one, and implement it. A regular (but not overwhelming) research schedule keeps your participants active and gives you a steady stream of new insight.
Let your panel be a resource for your whole team, not just those you collaborate with closely; it’s insightful for everyone, from marketing to sales to engineering, to see the data and input directly from your customers.
A research panel is a useful resource for UX researchers at any level. I encourage you to develop a rapport with the participants on your panel; they’re giving you invaluable insight into your product or service, and it’s insightful for them to learn how products like yours can be created.
Recruit new members to your panel every few months and use the convenience of your research panel to diversify your tests. This ensures that you’re getting a variety of viewpoints so you can make the best decisions for all of your users.
AI-driven updates, curated by humans and hand-edited for the Prototypr community
