Published on | Prototyping: From UX to Front End — Medium Armin Jamak

Lots of things are happening around AR.

Apple released ARKit, and rumours are popping up that they are working on a standalone headset based on a new platform called RealityOS. It seems like the age of magic sunglasses is near, with Magic Leap gearing up for a big reveal and a commercially available Hololens probably not far off.

But we could already suspect from Tim Cook quotes like —

“AR is as a big idea like the smartphone. I get excited because of the things that could be done that could improve a lot of lives. And be entertaining.”

Tim Cook

— and from others as well —

“One day, we believe this kind of immersive, augmented reality will become part of daily life for billions of people.”
Mark Zuckerberg

— that AR is coming and probably in the near future.

Okay, so where’s AR now?

Every year Gartner, an industry analyst site publishes a graph of emerging technologies and their current state. In this graph, referred to as Hype Cycle, they highlight the different stages of technological progression.

Gartner puts AR in the Trough of Disillusionment category, meaning that it’s in an uncomfortable spot where a lot of people have seen it, tried it and the novelty is wearing off. To break this state, AR has to get to the Slope of Enlightenment where everyone is familiar with the concept and second/third generation systems are put into production.

Although we are still in the first iterations of AR we can at least be certain, that it will create its own platform, one similar to what VR has now. In a best case scenario (which is actually pretty likely) it will be the next global platform, potentially replacing smartphones. Not because of all the hype it gets now, but because it builds on top of our current experience with the digital world and extends it.

Bolter and Gruisin talks about this in detail in their book, Remediation. According to the concept of remediation, every wave of technology builds on top of the previous, and every new technology will build on top of the current wave. AR remediates film and special effects, and the medium after AR will also remediate AR.

Phases of AR

We can also break down technologies into further phases, as they mature and progress through their lifecycle.

The first phase of AR can be thought of as a transitional step where two dimensional content was layered on top of a piece of paper — an old media remediated in a new one.

The second phase of AR is all about context, and having a meaningful connection between physical and digital environments. This is where mainstream adoption takes place and we solidify how the format works.

In the final phase the physical and digital domain is merged and only the experience remains.

To continue let’s talk about the characteristics of AR.

A definition of AR

In 1997 Ronald Azuma, a pioneer of early AR technology defined Augmented Reality as:

“Allowing the user to see the real world with virtual objects superimposed or composited with the real world. Therefore, AR supplements reality, rather than completely replacing it.”

Azuma’s definition states that AR systems have the following three characteristics:

Combines real and virtual

In a talk at The Stanford Center for Image System Engineering in May 2017, Azuma said: “Compelling experiences in AR will be where you have a meaningful combination of and connection between the real and the virtual. “Because if all the power just comes from reality, then why bother with augmentation? And, if all of the power comes from just the virtual stuff, why do AR? Why not just VR?”

Interactive in real-time

This makes it similar to an OS or a video game, mediums that are by nature, user action based.

Registered in three dimensions

In three dimensions we need at least three independent values to get a position. In AR it is most common to use the Tait-Bryan angles, where the movement of the head is specified by the x,y,z rotation of the head, commonly referred to as roll, pitch, and yaw.

In AR, position (x,y,z) and head orientation (roll, pitch, and yaw) are called six degrees of freedom (6 DOF).

There are two types of AR setups

Video see-through displays (VST)

The most common forms of AR use Video see-through (VST) displays. VST displays see the world through a camera. They transfer the video image to a GPU that draws the digital content on top of the video layer. The combined image is then displayed on a viewing device like an LCD screen.

The advantage of VST displays is that it can reuse smartphone components, making it a fairly cheap technology at the beginning. It can also take full control over the image displayed, easily switching between augmented and virtual reality. The downside is that it hides your face, making it awkward to use in social context, and can be tiresome for long periods of time.

An already existing example of this technology is an smartphone running an AR app or mixed reality head mounted displays that brings older screen based technologies to a new medium.

Optical see-through (OST)

Optical see-through displays rely on an optical element that is partially transmissive and reflective to combine virtual and real.

A good example is a half silvered mirror commonly found in DSLRs that lets in sufficient light while a display showing digital elements is placed overhead so that the virtual elements are reflected in the mirror and overlaid on top of the real world.

The good thing about Optical see-through glasses is that they provide a non-obscured view of the outside world. This concept is used by products like Google Glass, Microsoft Hololens, or Avegant. Magic Leap has also been rumoured to be working on an OST system.

VST vs. OST

The question of which will win at the end or if they can coexist is hard to answer. In the world of digital cameras, digital wayfinders are starting to overtake optical ones because you can immediately see what the end result will look like. However developments in the optical field could bring a natural, eye-friendly AR experience that can make this technology more enjoyable on the long term.

My bet is that Apple is going the Video see-through route at first, because it allows them to create a small form-factor and have the flexibility of moving between AR and VR freely — also they recently acquired a company in this field. I would not be surprised if it was a fancier version of these 1920s motoring glasses.

The side cover could be an option for fully immersive experiences, otherwise it’s a setup with two wide angle cameras (similar to Snaptacles) and a stereoscopic OLED display to display the outside world with the digital environment layered on top of it.

Interactions in AR

Today to interact with any system or computer we use our voice, type or track hand and body movement with sensors. These signals are all external and we have to articulate them, so a sensor can pick it up. We can also pick up brain signals with an EEG device and train it to recognise simple instructions.

Ideally in AR we want to use a combination of body tracking and voice recognition to have a natural experience. However voice recognition never really took off outside of our home, never mind at the workplace. But this might be about to change by using a different kind of voice that lives inside our heads.

Researchers at UC Berkley have been successful in identifying the frequency of our internal monologue (that inner voice we hear when we think or read) and are now working to decode it. Although a solution is far away, in the future this technology could not only provide the most efficient input method but help people with speech impairments. This technology combined with an AR eyeglass could allow you to operate an OS discretely.

Reality OS and the road to commercial applications

Probably the most important aspect that will determine that success of any platform is software. While a proprietary OS is still a year or two away we can already speculate where the technology would deem useful. Here are three topics that are important aspects of our lives and can be transformed by AR.

Information
Memories
Creation

Creation

When we’re talking about an environment as diverse and big as the real world traditional techniques of creation are not up for the job. Building a digital layer on top of the real world is a challenge when you think about the endless number of shapes, sizes that exist.

This is where procedural generation comes in handy

Today, Minecraft uses procedural generation to create entire worlds from scratch, that are eight times the size of Earth. AR could do the same by building on top of our existing world. This way an endless number of world-scale filters could be created and populated.

Going down a notch in size, cities and famous locations could be transformed or reimagined. As you go down in scale, curation and traditional techniques of creation come into play, allowing you to refine and add detail.

Procedural generation could also help you in adapting a set of components that make up your app, service, to different environments, use cases. Imagine a 3D website that unfolds in your room, laying out its content and components automatically as you navigate.

Even now we struggle with the problem of optimising for a huge number of devices. With AR you can never be sure where your content is projected, so optimisation has to be done algorithmically.

Procedural techniques also open the world of creation to a larger audience. Instead of working inside a tool, designed for a specific type of creation, the system runs the tool and you direct it. This makes it easy for people with good artistic and general design skills to try themselves in any field and start creating.

Visual thinking

Apart from the process of creation, AR can also help us come up with new ideas. Around 60–65% of the population is estimated to use visual thinking primarily, for solving problems. For these people AR could be a revolutionary tool in their arsenal.

Imagine an intelligent AI that guides you through your thinking process, visualising and breaking down complex thought structures in a way that resonates with you the most.

Memories

Azuma in his talks mentions two key strategies in how AR could help us remember and relive our past.

Reinforcing

The strategy of Reinforcing is about transforming a place with a strong historical significance, so others can relive those moments. The key is to augment a location that already provokes feeling. By looking at it through a different lens you are transported to a different time & place.

Azuma gave the Battle of Gettysburg as an example, saying that when you visit the site of Gettysburg, you already know it’s place in American History.

“It’s an emotionally powerful experience just being there where those events actually occurred.”

Remembering

Azuma talks about Remembering as something similar to the Reinforcing strategy, except it’s much more personal.

“Gettysburg has a meaning that overwhelms. Everyone knows and shares the meaning of that place. But I may have a very different experience at Stanford than the rest of you, and I would have different memories of a particular location,”

The Remembering strategy combines memories and stories with the location where they originally happened.

Azuma used the example of his wedding, where he took photos and videos, but what if he could relive those moments using AR at the location where the event happened? “That would be a powerful experience!” he mentions. “It would be powerful because it was a deeply meaningful event that personally mattered to me, and I would be able to relive it at the place it originally occurred, creating a strong sense of contextualised presence”

Information

AR mixes up a lot of things when we think about how we display and consume information. The key concept to wrap our heads around is that AR transcends screens based technologies and turns your whole vision into a digital environment.

Skeuomorph AR

Opening up your whole perception to digital experiences might be scary for a lot of people at first.

That’s why at the beginning of mass adoption there will be a need to bridge this new technology with what we are used to. A version of the skeuomorph movement has to take place in AR as it did in digital design. This movement will first connect AR content to places and methods we are accustomed to. Instead of a computer screen your content will be projected in front of you in a similar rectangular format. You can see a hint of this in a recent Oculus feature.

A wall frame can be repurposed to show your digital photos or personal calendar. Apps can be laid out on your desk and incoming mails can be visualised as stack of paper. There is no limit whatsoever, as any object can be repurposed in AR.

In a concept called Invoked computing, researchers at the University of Tokyo used AR to transform everyday objects into digital tools with spatial audio and video.

<a href="https://medium.com/media/5915a1cbeaea16e8e0f87b9df20a8d22/href">https://medium.com/media/5915a1cbeaea16e8e0f87b9df20a8d22/href</a>

In a demo project a banana has been repurposed to serve as a phone, and a pizza box as a laptop. With AR you have the power to choose how objects function around you.

WebAR

If we zoom out from the personal space, and into our shared spaces, an underlying global network has to exist, so that everyone can share the same experience. A sort of webAR that allows the distribution of information and ties it to physical space.

AR experiences could then be shared by placing it in a shared physical location, like a street or a landmark. Digital advertisements could be showcased in physical locations and secret messages, digital art installations could come to life, wherever you go.

Ownership

However a global AR network raises questions around ownership.

Will it retain the open nature of the web or will it be tied to an operating system or software? Since physical locations are limited, a completely decentralised network would run the risk of overpopulation and a bunch of inappropriate content could be placed in front of public eyes.

Another important question is the relationship between the ownership of physical locations and it’s digital overlay. Are you allowed to place an AR experience on someone else’s house or store? Are you allowed to purchase AR ads in a public setting, as you do on the internet now?

These are some of the questions that have to be answered before the technology goes mainstream.

Digital Humans

I would argue that the first realistic robots will be AR experiences instead of ones constructed out of materials. The reason is simply that we are already capable of mimicking a real person in CG and AR can bring these digital humans into our world, making it easy to interact with them.

This can bring a revolution in AI driven digital humans, teaching us, entertaining us or acting as our personal assistant. With inverse crowd simulation techniques these digital creatures could be placed convincingly next to actual humans by anticipating and reacting to their surroundings.

A digital human could also be controlled by someone who’s at a remote location, guiding the body from a VR headset.

Ethics of AR

A technology this transformational raises a lot of ethical question, mainly around the experience of being human.

Technological progress sometimes seems like a distraction from living our lives and accomplishing our goals. Because new information is so easy to find, it’s easy to get lost in satisfying our curiosity.

Of course it doesn’t help when companies are deliberately manipulative and deceitful. AR is especially susceptible to malpractice because it’s directly tied to what you see and hear, therefore there has to be a strong code of conduct when we design these experiences.

The ethics of this technology has to be focused around complementing our lives instead of taking away. Letting people experience being human, without sacrificing progress in well-being and productivity.

Here are a set of principles that could guide you in designing for AR

Relevant

Think about what goals a person might have and what would constitute a life well-lived in hindsight. How can you help in that journey and how does your service fit into someone’s days?

Mindful

Design experiences that pace themselves, preventing addictive behaviour. There should be no point in sticking around after you got what you wanted. Try to avoid instant gratification, as a lot of (usually neurotic) people have a hard time stopping.

Reductionist

What the least amount of distraction & information you can get away with. Because AR is directly tied to your vision you should be extremely careful about heavy visuals and imagery, especially if the experience is always on.

Contextual

AR experiences should have a meaningful relation to their physical surroundings, because otherwise there’s no point in using AR for the job. Also, (especially at the early days of the technology) you should have an indication what AR experience to expect when you look somewhere and help should always be there when you don’t.

Conclusion

Even if AR doesn’t turn out to be a massive success or the next digital platform, the technology will always be there to break out from our boxed digital environments, and create something virtual that’s as close to being real as it can get.

However AR is not a single piece of technology, it depends on advancements in AI, sensors, IOT, computer vision and battery technology, just like smartphones today.

The space to innovate is wide open, and there’s a heavy burden on the designers and developers of tomorrow to convince us we need those magic sunglasses 🕶.

References
If you want to learn more about the topic I highly recommend Augmented Human by Helen Papagiannis and Practical Augmented Reality from Steve Aukstakalnis where you can find out more about some of the topics covered in this article.

A World of AR and Magic Sunglasses was originally published in Prototypr on Medium, where people are continuing the conversation by highlighting and responding to this story.

A World of AR and Magic Sunglasses

Cool stuff, every week

More Articles