Published on | Prototyping: From UX to Front End — Medium CanvasFlip

It’s not art, it’s science 🙂

Last time I was working on a project, the user sample size was “7”. This time it’s “36”. Why?

Maybe, I was lucky this time to get 36 users! 😉

Nice try at guessing, but that’s not the reason buddy!

As UX researchers we always need to justify the sample size of our users for testing. Clients, colleagues and investors want to know why they should trust the opinions of some random number of people! And these questions are quite relevant and justified to be honest.

So how do you decide how many users you want to test with? Anything between 5 – 100/1000/10,000/….. is a probable choice. I tried to put this into definite structure so that such questions don’t come up each time.

Here’s how!

1. What is the project you are working on?

When trying to decide, how many users are enough for a sufficient proof, this parameter is the most important factor. A design from the scratch or something out of the box or small wins with the current design? You can go with a smaller section if it is about potential issues with the current design, and this number increase as based on what you’re designing.

The projects that we take up are usually one of these —

Research on an existing product (for example, usability testing, identifying new features, or analysing the current roadblock due to UX);
Creating a new product from the scratch (for example, a new idea that involves an app or website);
Building beyond your product (for example, working at a project that is totally out of the box, something that does not exist in the current market).

John Creswell suggests guidelines ranging from as few as five for a case study (for example, your existing product) to more than 20 for developing a new theory (for example, generalising beyond your product).

Ideally, if you are researching on an existing product, any number of users between 5–10 is a good number. Going further, if you are building a product from the scratch the number should ideally be double of that, i.e, between 10–20. And if the product if out of the box, you need thrice the number to validate it — 15–30. How much ever the urge is to hide the concept till the release of the product, it is strongly recommended that one validates (or invalidates) the idea.

2. Diversity in users

It’s totally a linear relation, the sample size increases as the diversity of your users increases. You might want to have 2–3 representation from each persona.

With CanvasFlip we faced similar issues: being a complete suite of prototyping, user testing, collaboration and design hand-off tools, we have multiple personas simultaneously working on the tool : Product managers, UI/UX designers, developers and business analysts. So, we tested the tool with 3 representatives from each persona. It definitely gives a better and deeper understanding about experience of each persona and also the friction in the transition of each persona.

3. Our experience in research

When I started with UX research my designation did not reflect the same. I was a young entrepreneur trying to validate my concept. And I believe you do not necessarily have to be a UX researcher to test your ideas. But, it’s not true that the your process would be a mirror image of the process an actual experiences UX researcher would follow.

Experienced researchers can do more with a smaller sample size than less experienced researchers. Experienced researchers will extract more and better-quality data from each participant. An experienced researcher knows how and when to dig deeper based on a participant’s response. This is something that a newbie researcher will not be very excellent at. An experienced researcher obviously have a history of experiences to add insight to the data analysis.

What should NOT be a parameter to decide the number of users —

Budget and time!

Unfortunately, the two main parameters people consider for deciding on the number of users are budget and time. And these are certainly the ones you shouldn’t give a thought to. How?

With the broadening spectrum of user testing and the tools to help you with user testing, this shouldn’t at all be a concern. Rather the number of users should depend on the more logic parameters discussed above.

With our continuous efforts of making user testing easy and affordable for all, this shouldn’t be a concern! Just for those you never across this — CanvasFlip provides unlimited and free user testing for your prototypes 🙂

Where to get my users from?

Getting 5 users for your tests is not a very though job. I do not mean to take away from the struggle in finding those perfect 5 who are the most accurate for your brand, but this is something that you already know where to get from — your current user base.

The problem is when you need a larger sample of users, i.e., if you have to validate a product before building it from scratch AND/OR you have 4–5 personas using your product AND/OR you are new to UX research..

Since we faced similar problems, we have been thinking on these lines and now we have a solution! ;) — When you need to test with a large sample you can possibly find them on social media groups or on Slack groups or may be you have a whatsapp group with all users added there. We came up with a unique link, called the magic URL which can be pasted on any social media/Slack/Skype groups.

So this project that I am currently working needs 26 users to validate its concept. My user persona is photographers in the age group 18–35. So my obvious target was photographer groups on Facebook. In 3 days I had 26 user videos in my CanvasFlip account. 🙂 🙂

Finally,

As UX researchers we should have reasons to justify to yourselves why we are testing with a particular number of people. The above parameters can work as a formula for you to calculate the number of users you should test with. I would also like to add to this, that assign a co-efficient factor (2,3,4…) to each parameter based on it’s importance to you. Clients and colleagues too will appreciate the transparency we provide when we show how we’ve determined our sample size and analysed our data.

How to decide which project needs 5 users & which needs 50? was originally published in Prototyping: From UX to Front End on Medium, where people are continuing the conversation by highlighting and responding to this story.

How to decide which project needs 5 users & which needs 50?

Cool stuff, every week

More Articles