Published on | Prototyping: From UX to Front End — Medium Claire Menke

Using chocolate chip cookies to drive internal engagement with UX research

The UX Research and Analytics teams sit next to each other at Udemy, so we naturally joke and debate a lot — about sample sizes and experimental design, amongst other (often more inane) things.

A few weeks ago one such example popped up. We started debating the very academic question: “When can a chocolate chip cookie no longer be considered a cookie?” This debate quickly escalated into a genuine experimental process, which has ended up involving stakeholders across the organization, multiple iterations, and a lot of full bellies. While objectively a silly exercise, it’s proven to be an amazing way to evangelize research in a new and creative way.

The hypothesis was that there is a limit to the amount of chocolate chips that could be added to the cookie at which point it will fall apart or be considered to be a chocolate bar rather than a cookie. As we began, ranges were proposed, and we decided to do some secondary research by reaching out to some SMEs (subject-matter-experts, aka some bakers that I follow on instagram) to see if there was a known value already. Thankfully @joythebaker was game to play along and provided a starting estimate of 45% dough to chips ratio.

That estimate in hand, I decided to have some fun and take it a step further by doing some experimental baking.

The first round we gathered a small beta group to taste-test four variations of the chocolate to dough ratio. (Note that we are using the ratio of percent of chips to dough, not as a percent of total cookie weight. The correct percent ratio debate is currently in process…).

Round 1 included ratios of 25%, 45%, 50%, and 75%.

After realizing some experimental design flaws, we iterated and baked again.

The second time we made several experimental improvements, and we were also able to pull in a wider variety of participants to increase the sample size for voting.

Voting underway for Round 2! Round 2 included smaller incremental steps, most by 5%. Also included 47.5% based on previous hypothesis to see if that made a difference. (Note: I ran out of chocolate chips so couldn’t fill the 70% and 75% jumps… baker error.)

By adding more participants we not only started to hone in on the best chocolate to dough ratio, but we also also got more feedback on our experimental design. Colleagues from design, legal, engineering, etc. started discussing our research methodology and providing input and new suggestions for the next round. The methodological suggestions included:

More accurate distributions of chips within individual cookies — have an exact number of chips per cookie at the appropriate ratios.
Normalize the consumption experience — everyone must eat at least half of a cookie (ideally a whole cookie), rather than small pieces of a larger cookie. This helps minimize the bias associated with variable chip distribution.
Increase the jump size to either 7% or 10% — 5% increases were too small of a distinction to notice differences, especially considering the existing chip randomization error.
Execute blind testing. We talked about multiple options — adding food dye, adding a sprinkle or color-coded bakeable marker to the bottom of each cookie, etc. — and settled on putting the cookies in arbitrarily numbered paper bags. The purpose is to minimize guessing or bias based on visual inputs, and to remove initial perception bias by hiding the percentage values.
Blind voting (through DM on Slack) to minimize social pressures.
Potentially re-think how we are determining “percent chocolate”. Considering doing this as percent of total cookie weight, rather than the ratio of chips to dough.

All these adjustments in hand we will be iterating again, likely in a couple of weeks to let our sugar highs subside. Our goal for the next (hopefully final) round is to create a cleaner distribution curve, potentially generating pseudo-personas around the clusters of appreciation.

What impact did this have on involvement with UX?

While this may seem like, and definitely is, a very silly experiment it holds some pretty significant benefits for the research organization.

It has created an avenue for inclusion. We’ve been able to get many different types of stakeholders involved — engineers, data analysts, designers, researchers, PMs, recruiting and ops folks, lifecycle email marketers, etc. have all voted and participated in the process of improving experimental design.
It’s a fun new way for people to interface with our team casually, outside of the auspices of day-to-day research. They now know where our desks are and engage with us in a different way around the office as we have new things to discuss. 🍪
People have gotten to play around and do research without thinking too hard about it. Some people freeze up when talking about hypotheses or experimental adjustments, but with this exercise they’ve been questioning my methods and playing active parts in methodological development without thinking twice. It’s been encouraging to see these different stakeholders engage in these discussions and invest deeper in the research process. Their diversity of opinion has also increased my own creativity and opened my eyes to other ways of conducting research.
This showcases the iterative nature of experimentation. By continually evolving the process it helps stakeholders (subconsciouly) understand that the research process is not static. It takes trial, error, and multiple inputs to help guide the evolution, and it’s also a natural part of the development lifecycle.
A side benefit: involving others in the community. I’ve also been posting my process on Instagram stories, and it’s been fun to get other friends or community members invested in this experiment. Anyone can do research!

Playing with research in your own organization

I understand that baking may not be your forte, but I highly suggest finding a fun and creative way to engage your company in casual research. It could be polling about lunch options, quick studies on workplace etiquette, debating what is the best 90s sitcom, ideal office pet, anything. Look around! Pinpoint a naturally-evolving debate and take it to the next level, encouraging others to participate along the way. This has been very invigorating for me as well as the people around me — a win-win!

It’s also a good reminder that while we may do research professionally, we can’t forget to have fun with it 😊

🎉 🎉 Udemy is hiring! While we are not currently hiring for the UX Research team, we have many open positions across the company. Follow this link to join an amazing team with an empowering mission! 🎉🎉

The beliefs and statements included in this article are my own opinions, and do not necessarily reflect those of my employer.

Evangelizing research by playing with your food was originally published in Prototypr on Medium, where people are continuing the conversation by highlighting and responding to this story.

Evangelizing research by playing with your food

Cool stuff, every week

More Articles