<p>Published on <a href="https://blog.prototypr.io/a-b-testing-a-superpower-71b7a2179fbc?source=rss----eb297ea1161a---4">| Prototyping: From UX to Front End &mdash; Medium</a> UX Boot Camp </p>
<h4>#30dayUXchallenge [Day 21]</h4>
<p>Yesterday Natalia gave us an introduction to formal usability testing as a way to validate our prototypes. Today let’s talk about A/B testing. Yes, A/B testing is just another tool in our box that we can leverage to test and validate our design solutions. But A/B testing has so much power that we can and should tap into, it allows us to optimise the smallest of features on our site, it gives us insight into what our users actually do vs. what they say and it opens our eyes to things we cannot discover through any other type of research. Today let’s talk about the great power… <em>and the great responsibility 🙂</em> we have when we perform A/B testing.</p>
<h3>A/B testing super powers:</h3>
<h4>1. Behaviour validation</h4>
<p>So you have your user problem, you have been through the design process and you have validated your solution through user testing etc. Often we are inviting users in for formal usability testing, it’s in a lab environment other wise we do contextual enquiries, no matter how skilled you are as a user researcher these tests are staged our users are solely focused on our concept test. We all know that what people say and what they do are very different.</p>
<p>Its also hard with qualitative research to test on a wide range of users, usually only 5–7 participants a day. A/B testing allows us to really understand on a much larger scale what our users are actually doing and what is their real behaviour. By setting up an A/B test we can test one (or multiple) concepts against a control group (the existing design) to determine if our new solution is in fact better than the current by defining specific metrics. A/B testing can give us the data insights needed to confidently validate our concepts before a wider implementation. Think of it as a final sign off.</p>
<h4>2. Butterfly effect</h4>
<p>Have your new design? Ready to implement? This is where is becomes exciting… but also a little scary! Are we really ready to implement a design change no matter how big or small without tracking it’s real effect? A/B testing best practices recommend that you only A/B test individual changes such as one button change or a copy change as these can have a huge impact on KPI’s. Eg. at Marktplaats we have found that changing something simple on the ad creation page can not only effect the current page in regards to it’s completion but also have a knock on effect for both the seller in terms of the number of leads and even the buyer in helping them to find the right product. By having the right metrics in place we can see the full effect that our design changes have. This is also where I like to go a little against the grain here. I believe we can and we should A/B test our larger changes too, how else can we really measure their effect full?</p>
<p><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RoS7l2kRcaVDmaWHArQTWw.png"><strong>Marktplaats — A/B test Photo CTA.</strong> Group A: control with original design of blue camera CTA. Group B: test group with new design of empty photo placeholders.</p>
<h4>3. Scale &amp; targeting</h4>
<p>As mentioned above A/B testing allows us to test with a larger group of people that we couldn’t reach through any other way. With A/B testing we can get very granular with who we test and where, what devices and what segment of users. The results can be very interesting, eg. how do IOS users respond to a change vs. Android users. What’s important is when setting up your tests think about what you are testing and the best way to split the groups. It’s important to ensure the control group and the test group are the same type of users.</p>
<h4>4 Iterate, Iterate, Iterate</h4>
<p>A/B testing allows for iteration. Through A/B testing we can discover our failures and our successes on a granular level, and sometimes the failure is even more important to discover! If a test is unsuccessful you can begin the investigation into the why. But importantly you know if didn’t work and you can turn it off, you only exposed your change to 2% of users for 1 week and have valuable insights to create the next iteration. A/B testing can be used as a playground to learn more about our users their behaviour and response to changes.</p>
<h3>Great responsibility</h3>
<p>So how can you get the most out of your A/B testing? What do you as a designer need to be responsible for when creating an A/B test?</p>
<h4>1. Respect the limitations</h4>
<p>A/B testing is fantastic and gives us insights we can’t get anywhere else but we need to remember it’s limited. A/B testing doesn’t tell us if we are solving the right problem, it just tests one solution against a control group, based on the metrics we have defined. A/B testing should never be used as the sole validation method. A solution should always be validated through user testing or another qualitative user research method this way you can pair both perspectives to make an educated decision based on your results.</p>
<h4>2. Stand up for the user</h4>
<p>In some businesses A/B testing can be used to a dark advantage. Yes by pre-selecting the most expensive option we can have a positive effect on revenue metrics, but is this what our user wants? As a UX designer it’s up to us to ensure the A/B tests aren’t set up to allow dark patterns to be implemented, we need to solve for real user problems and not just improve our companies bottom line, let’s go for a Win Win situation here 🙂</p>
<h4>3. Success vs. learning metrics</h4>
<p>It’s very important when setting up the test we define what success looks like. This way there can be no confusion on the team as to if we implement or not. If conversion has a positive impact but engagement goes down do we implement or not? This should be decided before the test starts so there are no disagreements later. It’s also important to define the learning metrics, to understand the full effect of the test, linking back to the butterfly effect, does the test have an effect on the time spend on the platform too? It could be interesting to define what other metrics we need to track so that we can learn as much as possible for new iterations or even new tests.</p>
<h4>4. Keep it small</h4>
<p>As mentioned earlier it’s important to keep the test small so there is a minor change from test and control group. If you change too much can you really tell what attributed to the result? As designers it’s easy to come up with a full new solution that’s so much better than the current. But we have to find a way to break it up and test the individual parts to quantitively validate our solution. Find out which are the small parts you can test.</p>
<p>This is just a brief overview of my thoughts on A/B testing. At Marktplaats we have the mantra to test everything before we implement to discover it’s real effects. What is your opinion on A/B testing? How do you do it? Reach out if you want to talk in more detail about it!</p>
<p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fupscri.be%2F5eb5ba-2%3Fas_embed%3Dtrue&amp;url=https%3A%2F%2Fupscri.be%2F5eb5ba-2%2F&#038;image=https%3A%2F%2Fe.enpose.co%2F%3Fkey%3DdRXnS9Gplk%26w%3D700%26h%3D425%26url%3Dhttps%253A%252F%252Fupscri.be%252F5eb5ba-2%252F%253Fenpose&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=upscri" width="800" height="400" frameborder="0"><a href="https://medium.com/media/8d5e2c740a04faf463af9accfa7e3c31/href">https://medium.com/media/8d5e2c740a04faf463af9accfa7e3c31/href</a></iframe><img alt="" src="https://cdn-images-1.medium.com/max/123/1*D251eF41CCuCNXGCAIF5Dw.png">Rochelle Carr — UX Designer, Placing customer journey team, Marktplaats, Ebay. Soft spot for red wine &amp; tries to learn Dutch in her spare time.</p>
<p><em>This postThis post is part of the #30dayUXchallenge: you can read more about why we started it </em><a href="https://medium.com/envato/why-were-embarking-on-a-30dayuxchallenge-e3d9f1bc1ac6"><em>here</em></a><em>, and don’t forget to follow us on </em><a href="https://twitter.com/ux_bootcamp"><em>Twitter</em></a><em> and </em><a href="https://www.instagram.com/ux_bootcamp"><em>Instagram</em></a><em>!</em></p>
<p><img src="https://medium.com/_/stat?event=post.clientViewed&amp;referrerSource=full_rss&amp;postId=71b7a2179fbc" width="1" height="1" alt="stat?event=post.clientViewed&amp;referrerSou"></p>
<hr>
<p><a href="https://blog.prototypr.io/a-b-testing-a-superpower-71b7a2179fbc">A/B testing — A superpower</a> was originally published in <a href="https://blog.prototypr.io">Prototypr</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>


A/B testing — A superpower

#30dayUXchallenge [Day 21]

Top Stories