Published on | Prototyping: From UX to Front End — Medium Chris Jacobs

Prototyping a Places of Interest AR app in Origami Studio

Background ✈️

Have you ever looked out the window during a flight and seen an idyllic, unidentifiable beach, a snow-capped mountain range or a bustling city?

Wouldn’t it be great if you could quickly find out what you’re looking at?

What are those mountains?!

Some obvious problems 🤔

Most of us don’t have internet on flights, so any places of interest would need to be downloaded in advance.
When looking down from 37,000ft, there are lots of places of interest to display (more on this later). This could be very cluttered.
Most people think flight mode blocks GPS signals (it hasn’t on iOS since April 2015).

The Process 👨‍🔬

⏳ Time: As with my other challenges in this series for prototypr.io, I want to demonstrate what can be achieved with a realistic amount of time (for a side project) and experience. Therefore I’m limiting myself to 5 hours — 1 of which can be spent on the initial design.

🔧 Tools: I’ll be using Sketch and Facebook Design’s Origami Studio.

🗂 Files: The Sketch, Origami Studio and source files are available in this Dropbox folder.

Step 1 — Flow 💃

There are two main sections to this app:

Route search & download
Viewfinder / head up display (HUD) overlay screen

Although some flights have Wi-Fi, most users will need to download places of interest for their flight route prior to take off. To keep this app ‘lean’ I will just allow one flight’s worth of data to be stored at any one time. Also — there is no ability to instantly access new places with an internet connection.

Step 2 — Design 🎨

Main screen

I started by designing some different pin styles over a typical plane window view.

Somehow I managed to pick one of these 3 radically different designs. I only wanted users to view this screen in landscape orientation (for the best viewing experience), so I created a separate portrait view as well.

Splash screen

The splash screen will allow users to enter a flight route, download places of interest to be used offline, open the AR screen and finally clear the route and restart.

I went for a simple design, but added some little personalised touches such as dynamic text based on airport choices, route distance and a loading state with relevant example places.

Responsive design 📱

These designs are all based on the iPhone X. However, as Apple’s AR Kit is available as far back as the iPhone 6s, I wanted my designs to work on these other devices (mainly so I could actually test it, on my ‘ancient’ iPhone 6s!).

In a future post I will have more detail about building responsively in Origami Studio.

Step 3 — Prototyping planning 🛠

What to make ✅ and what to fake ⛔️

AR screen

⛔️ Use a looped video of a plane window instead of the camera viewfinder (I don’t have any flights planned in my 5 hour window, so using the camera would be pretty pointless).

⛔️ Have a short, preset list of places of interest.

✅ Use some maths to keep the places of interest locked to the same place throughout the video.

✅ Allow basic interaction with the labels.

Splash screen

✅ Allow users to type airport names.

⛔️ But only build the dynamic content to work with one route (hopefully they pick Bristol to Dublin 🤞).

Step 4 — Prototyping the AR screen

Video

First things first, I needed to replace my placeholder photo with a video. I really wanted the video to include the aircraft window frame, to give a bit of context to users. After searching YouTube, this is the best I could find: https://www.youtube.com/watch?v=ejzS0atCtgQ&t=4m7s

I imported the video into Origami Studio, scaled it based on device width and setup playback whenever the device was in landscape orientation.

Places of interest (POI) labels

Each label has a few parameters to determine when and where it is displayed, its title and the additional information shown when expanded. I find the tidiest way to input this type of information to Origami Studio is using a JSON file, here’s mine:

[
{"POILabel" : "River X",
"startTime" : 1,
"endTime" : 10,
"startXPos" : 0,
"description" : "This is a really big river, full of fresh water and fish.",
"startYPos" : 33},

{"POILabel" : "Island 1",
"startTime" : 5,
"endTime" : 26,
"startXPos" : -15,
"description" : "Almost 5 people live on this remote island.",
"startYPos" : -20},

{"POILabel" : "The sea 🌊",
"startTime" : 13,
"endTime" : 36,
"startXPos" : -80,
"description" : "This water is some of the greenest in the world.",
"startYPos" : 20}
]

Which can be converted into loops using quite simple patches:

The resulting POI labels are revealed during certain time windows (based on when their associated place of interest enters and exists the video). They then animate from wherever their start position is to a set end position (the right hand edge of the screen).

So now I have POI labels floating across the screen. I did a little more work to make them expandable when tapped to reveal an interesting description (and also decrease the opacity of any other labels).

Status toggling
The ability to only have one item expanded at a time is a really reusable bit of logic (think accordions, radio buttons). Below are the patches used, expanded to show logic used and grouped for easy re-use.

Portrait mode

When a device is in a portrait orientation, the video and POI labels are hidden, the ‘rotate your phone’ message is displayed and the video is paused (no need to reduce performance unnecessarily).

Note: It’s important to enable all interface orientations using the ‘Interface Orientation’ patch.

Step 5 — Building the splash screens

Patches over components

I had good intentions of making the patches for these screens really elegant. Unfortunately the 5 hour limit got in the way, as I only had 2 hours left for this part of the project. The part of my Origami Studio responsible for the splash screens ended up looking like this:

It’s messy, but given another 30 minutes it could probably be half that size and certainly tidier.

Don’t be put off — it’s actually very straightforward. An Option Switch patch controls the sequence step and then 50% of the other patches are repeated Option Pickers and Option Equals patches.

I chose this approach to reuse a small number of UI layers and components for all 6 screen states in the splash sequence. The layout, content and interactions is all controlled by these patches depending on the point in the sequence.

For example, the primary CTA has different values for opacity, text label, loading animation and tap destination depending on the sequence stage.

Overall the prototype actually runs pretty quickly, as there are very few layers and components.

Keyboard interaction

Whilst the personalised messages in the designs won’t update based on route choice (it will always reference Bristol to Dublin), I still wanted users to type their airports. Nothing ruins a prototype like fake text entry.

When the keyboard loads, the content container smoothly moves up the screen enough to leave 30px above the keyboard.

Extra touches

I had 15 minutes left, so I added a few extra bits:

Moving clouds background ☁️
Loading spinner with scrolling example places of interest
Animated CTA during search
Reset feature

The final product

<a href="https://medium.com/media/8cd6ec7edd13e77846a63700a9e09ae0/href">https://medium.com/media/8cd6ec7edd13e77846a63700a9e09ae0/href</a>

If I had one more hour… 🤔

I would like to use the device’s tilt sensors to add a parallax effect on the video and labels, to make the prototype feel more like an AR experience instead of watching a video.

With another 20 minutes it would be sensible to link the personalised text to the typed airport names and improve the placeholder label titles!

I also want to look into the trigonometry needed to work out, for a given flight path, which places of interest should be downloaded. My initial thoughts on this are scribbled here:

I suggest making a pyramid round the flight path and finding all the places of interest within it.

The base is 2x distance to horizon (windows on both sides). This rough method is very wasteful, as it uses max altitude for the pyramid height for the entire flight path.

To give an idea how much difference using actual projected altitude to the data search and download, this is how the searchable area changes with altitude:

At h = 37,000ft, D = 236 miles. Area = 3,304 square miles.
At h = 15,000ft, D = 150 miles. Area = 852 square miles.

This model assumes you only look down.

This is about 4 minutes of thought - so suggestions are welcome!

Lessons learnt 🎓

Plan UI sequences. The splash screen patches were a mess due to lack of planning. Next time I will start by listing what will trigger each sequence progression, all the possible UI configurations that need to be dynamic and the ‘state’ of each one at each sequence stage. A JSON file mapping sequence steps to UI configurations would be a great idea here 🤔.
Restart Origami if things get slow. My POI labels kept lagging behind the video, it was driving me crazy. In the end I noticed that after a few video loops, the ‘Current time’ value from the Video Info patch wasn’t outputting in real time. Restarting Origami Studio sorted this problem out each time.

<a href="https://medium.com/media/b85dfbb5286d8a25cf2e754b9462cf45/href">https://medium.com/media/b85dfbb5286d8a25cf2e754b9462cf45/href</a>

Making flights more interesting with augmented reality was originally published in Prototypr on Medium, where people are continuing the conversation by highlighting and responding to this story.

Making flights more interesting with augmented reality

Cool stuff, every week

More Articles