Microsoft Design – Medium | John Kennedy

Using Unity, Windows Mixed Reality and math to create a planetarium tool for learning about the night sky.

Exploring the night sky from the comfort of my armchair. Bliss!

It’s hard to find your way around the night sky. Our ancestors, who didn’t have Netflix and Instagram to entertain them, or clocks and phones to tell them the time, depended on it. But times change, and simply put, the sky isn’t as important to us as it once was.

That said, there’s still much to gain from learning the constellations and the names of the brightest stars. There’s anthropology, history, mythology and science up there, and plenty of superstitions too. Knowing your way around the sky isn’t easy and needs practice, and especially in our world of ever-fewer dark skies, getting outside to look up doesn’t happen as often as I’d like. If only we could add labels to every object in the sky, and always have a warm, clear and undisturbed viewing site…

Microsoft’s Windows Mixed Reality brings VR to your PC.

Which brings me to a more modern technology: Virtual Reality (VR). I’ve been a fan of tinkering with immersive VR since I got the first developer version of Oculus. Recently I’ve also grown to appreciate the Windows Mixed Reality platform (not least as I work with that team in my day job). It provides good motion tracking and room-scale VR without requiring any external cameras to be set up, and its integration with the 3D app creation platform Unity is excellent. Assuming you have a fast PC, Windows Mixed Reality is definitely a VR option you should consider.

It seemed a good idea to try writing a simple astronomy program for this platform. A “virtual planetarium” in fact. It’s just C#, after all, so how hard could it be? Well. Turns out… not too hard. Here’s a list of the some of the technical coding issues and design problems I worked through, just in case you are interested in trying something similar yourself.

1. Stars are particles

Unity is great at OpenGL/DirectX type things, and that includes particles. In computer graphic terms, particles are small, simple shapes that can be defined, moved around, and rendered en masse. You’ve seen them all the time — you just might not know it. Flames and smoke effects are created with particles. In those situations, you would assign an orange or translucent dark texture to a small shape, and then have them dance around slightly with every frame.

Flame effect generated with particles. Source: Wikipedia

Particles are perfect for modeling the 10,000 or so of the brightest stars I wanted to display. The great advantage of particles is that they are fast — having to manage 10,000 individual game objects would be… challenging. As particles, I can move all the stars in one go.

Using Unity, I declared a particle system game object and then proceeded to give each star a suitable (X, Y, Z) coordinate, adjusting their size and color to match the actual star’s brightness (a.k.a. magnitude).

The C# code in Unity that creates the particle set.

But where did those (X, Y, Z) coordinates and magnitudes come from? Thankfully, there are many databases of stars available in the public domain, and it was a matter of tracking one down and performing a little spherical geometry to get the values I needed.

Part of the database describing the star and its location.

Essentially, star coordinates are given in Right Ascension and Declination, a sort of “left and right” and “up and down” position that assumes the stars are all on the surface of a giant sphere with us at the center.

Right Ascention and Declination, as defined in this diagram from Wikipedia.

Converting these spherical values to cartesian components, needs a little math like this:

The value for ‘r’ is the radius of this giant sphere, and I usually put in a value of 1000. It’s a matter of balancing this distance value with the size of the particles to give a nice star field effect.

In Unity’s scene view, this is what our sphere of stars look like.

When you do get them balanced out and they’re rendered in your scene, they look like this:

When rendered, we have 10,000 tiny points of light all scattered around us. Can you spot Orion?

I’m sure you know that the stars appear to move as the Earth rotates on its axis. The stars all stay in the same position relative to each other (more or less, certainly in human timescales they appear not to shift around too much), but exactly where a star will be in the sky depends on the following factors:

• The time. The sky rotates around us once in about 23 hours and 56 minutes. This means the stars will appear to drift about four degrees each night (about the width of three fingers at arm’s length). Read about sidereal time for more info.
• Your location. If you live in Seattle, the position of the stars will look very different compared to someone in Chile. Imagine yourself on the Earth looking straight up; obviously the star that is right above you will depend on where you are.

All this is again, “just” a matter of math. As we have our stars all grouped into a particle set, we can just spin and rotate the entire set in one go until they are in the right place. We can even fast-forward or reverse the sky to emulate a specific time or place on Earth. We now have our own a virtual planetarium!

2. Lines aren’t straight (forward)

The constellations are simply patterns of stars as imagined and defined by humans over thousands and thousands of years. In that sense, they are completely arbitrary; when enough people thought the dots that made up Orion looked like a hunter with a large stick and a belt with a dagger, then that’s what it was known as.

Old-school star map, featuring Orion.

These days they are also very clearly defined with strict borders. I’m sure you know at least 12 constellations — the ones that exist on the zodiac; the virtual path that the Sun appears to trace out once a year from our perspective on Earth. The constellation in which the Sun appears when you are born is said to be your “sign.” I’m not going to say any more about “astrology” though, don’t worry.

There are plenty more constellations — 89 in total. In the northern hemisphere we have giants such as Orion, Perseus, Bootes, Ursa Major and more. The southern hemisphere has it’s own shapes in the sky, although I find the northern one more distinctive — probably because that’s what I grew up seeing.

The simplest way to represent constellations is to use simple stick figures: you will still need to use your imagination to see what the constellation is (unless you’re looking at Triangulum, which you might be able to guess from the name alone).

So I created a list of (X Y, Z) coordinates for all the points that would create these outlines. And then tried to draw them.

Then it got tricky. Unity is an amazing tool for creating 3D (and 2D) apps. But, a little surprisingly, it lacks some basic features, such as good line drawing. Unity gets a pass though, as they created a fabulous built-in store, where developers and other content creators can fill in the gaps with their own solutions. And as I was needing to draw lines for the various constellations and grids, I decided not to reinvent any wheels and bought a super 3rd party library called Vectrosity.

Now I was able to create a virtual sky complete with outlines like this:

Stick figure outlines for the 89 constellations. Looks faint here, but clear in the headset.

2. Text is tricky

Drawing lines was tricky but text was trickier still. Specifically, text that could be occluded (that is, hidden by another object). I wanted to draw labels for the constellations and bright stars, and that wasn’t too hard: keep a list of coordinates for the text, and use Unity’s TextMesh gameObject component to define a label. Make sure the label is rotated to face the camera, and job done, right?

Well, almost. If I added a ground object to my virtual sky — and I really wanted to add a ground object to make it look as realistic as possible — then the text had to be hidden when it sank before the horizon. Unlike the star particles or lines, this didn’t happened automatically: the TextMesh would stubbornly float above everything else in the view.

Creating a new text texture in Unity.

The solution took a little time to work out. I needed to create a new custom font that used a new custom material that relied on a custom shader that in turn respected occlusion. The end result was worth it: a clear, legible piece of text that would be hidden when a constellation wasn’t visible.

There’s Cassiopeia — the Big W!

Things were coming together!

3. Adding (virtual) details

If anyone was going to learn about the sky, the view had to be reasonably realistic. Most people have trouble grasping the scale of constellations: is Draco large? Or small? Or sprawled over a huge chunk of sky? Unless you see it in context, you can fail to really see the constellation.

So my virtual world needed a lot more than just the sky. It needed a shaded blue/black sphere to make things stand out. It needed grass to stand on. Trees to help judge perspective. And maybe a prop just for fun.

This is where Unity really shines as a creation utility, and when I was done, it looked like this:

Now all I had to do was find a way to let users interact with it.

4. VR user interfaces are hard

Quick history lesson. When the iPhone was made available to developers, Apple made sure to provide a range of controls in the SDK. Table views, buttons, date and time pickers — Apple created controls perfectly tailored to the small touch screen, a way of working which no one had really seen before. As a result of these controls, developers got to hit the ground running and make a lot of apps that users could immediately understand and use. By making them customizable, Apple also made sure that apps all didn’t look exactly the same. And so the world forever switched to mobile development, and the design and UX paradigms with use today on almost all phones were defined.

The world of AR and VR has grown up a lot more organically. Different standards, different platforms, different hardware and systems with different features has meant no single control library or even a philosophy of how the controls would work exists.

Virtual hands created by Leap Motion.

On one hand this is very exciting — every time you create something, you’re probably the first person to do it in that specific way. On the other hand, it can be frustrating when you just want to add a simple settings menu and you have to spin up everything — including a cursor — from scratch yourself.

Some protoype VR interfaces from Microsoft’s own MR Design Labs.

This isn’t to say that work isn’t being done in this field. Far from it. I’ve seen beautiful menus that materialize out of thin air and follow you around. There are buttons that float over a pair of rendered hands. I’ve seen virtual watches, skeuomorphic computers, chatting robots, dancing rings of light — you name it, someone has thought of it. And at Microsoft, the Windows Mixed Reality team has an entire team working on creating a UX toolkit , pushing the boundaries of what the Fluent Design language might become in their MR Design Lab.

Still, I wanted to have some fun experimenting with my own ideas.

Prototyping 3D UI in 2D

After some sketching and testing, it was clear that creating my own user interface themed to my app would be a considerable amount of work in itself, so in the end, I created some simple flipping text boxes that interacted with a gaze cursor. Stare at the text for a second or two, and the box will flip around. To give the user a hint as to how it works, the first time you stare at the control it flips up just a little.

My menu option control: gaze on the object and it tilts slightly, then flips to a new state.

When I have more time, I would love to explore this UX space, as it’s rich with possibilities. But that’s a Medium article for another time.

The various components of the user interface: an information panel on the left, settings on the right.

Putting it all together

The was one final touch I needed: something to help with the atmosphere of standing outside on a cool evening, looking at the sky. Adding some subtle birdsong I’d recorded from a spring day last year really made a difference. Now I can stand outside on a clear night, even when it’s cloudy.

Looking around my virtual garden at evening.

I hope to fix up a few more things (adding the Sun, Moon and planets for example), and then eventually put this app on the Windows Store.

Here’s my Moon object in Unity. Soon it’ll be floating in the sky with the other Solar System objects.

VR shows a great deal of promise for education. And thanks to tools like Unity and the Windows Mixed Reality platform, it’s great fun to develop applications. What apps are you developing?

If you found this interesting, you might find this article I wrote on building an Orrery in Augmented Reality fun too. Thanks!

Building a Virtual Sky was originally published in Microsoft Design on Medium, where people are continuing the conversation by highlighting and responding to this story.