Published on | Prototyping: From UX to Front End — Medium Alexandra Olarnyk

How to move away from the wireframed box and into the infinite space of AR.

Written in collaboration w/ Kelly Graham – UX Designer, Microsoft Mixed Reality

microsoft.com

No More Boxes

As UX Designers, it feels very familiar to start our ideas in a box, e.g. drawing within the parameters of an iPhone screen as we wireframe an app. These types of boundaries sometimes subconsciously limit the boundaries of our ideas that we’re so accustomed to.

In Augmented Reality (AR), we’re breaking down the boxed in screen and instead thinking about an interface as being flexible. This can be a weird concept and mental jump because a lot of things in AR are counterintuitive to the way things work on a screen.

This article helps you get over this mental jump by explaining how AR can be seen within a spectrum of devices, what AR is, and how to design for it… I also threw in a Brad Pitt easter egg somewhere below.

The Spectrum of Devices

Understanding diversity in our devices is important because by doing so, we can then design for the right experience for each device and its use.

Diversity can be chunked into 3 topic areas:

Amount of users: Is the device tied to a single person or certain physical space?
Mobility: Is the device stationary or does it follow you everywhere?
Fidelity of inputs and outputs: “High fidelity” inputs can be thought of things like mice, keyboards, etc., where you can be very precise and it is comfortable to use for a long time, while “Low fidelity” inputs can be thought of as things with higher error rates, that are less precise, and harder to do complex tasks with (like touch, voice, or using a controller).

Your smartphone, for example, has low fidelity inputs since it only has one small on-screen keyboard. However, it tries to overcome this through things like speech-to-text typing and super high resolution screens.

Your desktop, on the other hand, has a much higher input of fidelity in that it has a keyboard, mouse, trackpad, etc. Your desktop is also usually tied to just 1 user and has many peripherals because it is stationary and has a lot of ports, so you can make use of things like printers and desktop speakers.

Where does AR fall in the spectrum?

AR is an edge case and one of the most immersive experiences.

Microsoft’s Hololens, for example, is not about what is present on the screen but what is presented on top of reality. Depth cameras help the Hololens understand the space around you, that’s how it works for placing objects on real things and understanding the space of the room. Spacial audio is also used to make you feel more surrounded by sound, with no need for inputs like controls — you can just organically move your hands.

microsoft.com

Re-organizing the spectrum

You can reshuffle the spectrum based on certain attributes.

Practice applying the spectrum

In a brief activity with yourself, think about how some of your apps might feel on other kinds of devices? What types of input/output would change the experience for the better?

For example, if you moved Tinder from your smartphone to a voice UI, like Amazon Alexa, what would it be like just meeting people based on their voice? How would inputs and outputs change?

What do “Edge Cases” on the spectrum mean for the future of UX?

It becomes less about what tech demands, and more about human needs
Computing becomes more distributed
There becomes more devices but less of them are visible
People will need to use the right interface for the right task

What is AR?

Keep 3 tenants in mind when thinking about AR:

Real world + digital world
Has a flexible immersion level
Interface beyond a screen

AR is more than just about seeing stuff float in mid air but it is instead about how the digital world interacts with the real world.

A great example of AR creating a new reality for people that is more productive for them than the one before is Ikea’s AR app. You can see what Ikea products, like couches and lamps, look like within a physical space instead in a way you couldn’t do on any other medium.

ikea.com

How do we design for AR?

Remember when we talked about inputs and outputs earlier?

Inputs can take the form of gesture, voice, touch, and controllers (which are inherently awkward — a lot of swinging your hands around).

However sometimes the fidelity of these inputs can be low — engaging us in an environment that can feel a little bit clumsy. So as designers, we need to find ways to compensate for the low fidelity, supplement user input, and make things easier through:

Artificial intelligence (AI) assistants, like Apple’s Siri, that can provide recommendations
Autocomplete
Computer vision that can recognize things in our viewpoints
Natural language processing so you can ask things in the language that makes sense to you
Predictive text

The interface is in the social sphere by default

(I love this) AR will use the world around you to represent the information. Therefore you need to consider boundaries and what happens within each. For example, how will people tap on things that are two feet away? How will people read text that are on objects behind them?

WebHamster

AR is a world of layers — how much of your interface will interact with the world around it?

AR can be thought of as a world of layers and you, as the designer, need to decide how each layer interacts with another and the awareness users will have to those layers.

We have the luxury of choice to decide if we want the UI to be integrated into the users’ “world” or not. Not all things will exist in the same reality. Finding the right balance of context between the digital and real shapes the world you’re creating.

Two words you should know: Diegetic and Non Diegetic. You can understand these concepts by thinking of a movie and its characters.

Diegetic: The sounds that exist in the world of that film, e.g. when Brad Pitt is in the casino, the slot machine are sounds in his world.
Non Diegetic: The movie’s soundtrack is not in Brad Pitt’s world — he can’t hear it.

(you’ve found the easter egg!)

Try asking yourself:

How much do we want the UI elements to interact with the real world?
Do we want people to walk through stuff?
Do we want digital things to wrap around real-world things?
What if people walk in front of the digital aspects? What will happen to those aspects?
What user inputs will you allow at certain times?

Scenes not screens

In UX we design within boxes, decide where to click or hover … it is a very static experience, all within a box. But AR is more like film than static wireframes. There are multiple ways users can move through an experience and that’s a decision we as designers have to listen to (go UX Research!).

An activity to think in scenes not screens

Use the reality sequence template below to create scenes of an AR experience. This mixed reality user flow template is created byLillian Warner, a UX Designer focused on immersive reality.

User Flow Template

Here’s one I did: A mom is telling a bedtime story to her daughter, both are wearing AR headsets. As she tells the story the program recognizes key words from the story, like “Dinosaur” and displays characters right there in the room with them. Gestures they could use are increasing or decreasing the size of the dinosaur if the size is too scary for the child.

AR is a weird concept to think about sometimes — designing for a fully immersive experience can be a mental jump — but maybe something that feels counterintuitive is just what we need right now in UX.

Additional Resources

Frank Chimero — “What Screens Want” http://frankchimero.com/writing/what-screens-want/
Lillian Warner — “Mixed Reality User Flows: A New Kind of Template” https://blog.prototypr.io/mixed-reality-user-flows-a-new-kind-of-template-27d59991de4a
Ethan Marcotte — “The Map is not the Territory” (lecture) https://vimeo.com/63525054
Abdallah El Ali — “Multimodal Interaction — An Introduction” (presentation) https://www.slideshare.net/Abd0/multimodal-interaction-an-introduction
Golden Krishna — “The Best Interface is No Interface”
Google Pioneer Program (Expeditions for Education) — https://edu.google.com/expeditions/ar/#about
Unity Virtual Reality: Types of UI Placements — https://www.zealousys.com/blog/unity-virtual-reality-types-ui-placements/
Spatial Augmented Reality Merging Real and Virtual Worlds — http://pages.cs.wisc.edu/~dyer/cs534/papers/SAR.pdf
The UX of VR — https://www.uxofvr.com/
Tip and Tricks for using AR with Unity and Vuforia — https://gamedevelopment.tutsplus.com/tutorials/vuforia-tips-and-tricks-on-unity–cms-28744
VR prototyping for everyone — https://medium.com/inborn-experience/vr-ar-prototyping-for-everyone-ea6fb8f159b5

<a href="https://medium.com/media/769d8e05d026d7a0fff09d7c2f6f8c2e/href">https://medium.com/media/769d8e05d026d7a0fff09d7c2f6f8c2e/href</a>

Designing for Augmented Reality was originally published in Prototypr on Medium, where people are continuing the conversation by highlighting and responding to this story.

Designing for Augmented Reality

Cool stuff, every week

More Articles