Published on | Prototyping: From UX to Front End — Medium Cheryl Platz

In order to make VUI more conversational, we must understand where today’s experiences fall short.

In my last post, we explored the conversational UI (CUI) track from Interaction 17. Along the way, it became apparent that the voice user interfaces (VUI) hailed as “conversational” today are actually just the tip of the proverbial iceberg.

Most of today’s experiences focus on command-and-control scenarios or very simple question-and-answer exchanges:

Alexa, what’s the weather in Seattle on Sunday?
Hey Cortana, open Netflix.
OK Google, how tall is Mount Everest?

The result of each of these conversation starts is either (a) a statement or (b) an action taken. There’s almost no back-and-forth, nothing learned from the customer, no mutual context or trust built over time. Barely a conversation.

The absolute cutting edge of conversation in current devices is a very short-lived ability to continue a conversational thread in the context of weather and some informational queries. Both Google Home and Alexa feature this ability.

Google Home and Alexa stand tall, but they’re not confidants. Photo from CNET’s comparison of the 2 products.

From a recent CNET comparison of the Google Home and Alexa, on the subject of this cutting edge feature:

As for the conversational awareness, it’s a neat trick, but the Home only understands limited context for now, and surprisingly, Alexa kept up with the Home on this front. We asked Alexa to tell us the weather. Then, we asked, “How about Friday?” and Alexa understood and responded without needing to hear the word “weather” again.

Let’s be clear — voice input is still an extremely valuable and important way to interact with existing systems… but today’s voice assistants are not sophisticated conversationalists.

Colliding Conversational Trains

The current state of voice assistants doesn’t mean there isn’t future value in more conversational exchanges with these devices. Any time a person says, “I’m bored”, it’s an indication that they’re open to a conversational exchange. Particularly if it’s directed at the device — what else could a customer mean when they say, “Siri, I’m bored!” What could your device learn about this person thanks to this conversational offer? What could you offer: Entertainment? Ideas? Inspiration?

Inevitably, someone will connect the dots and merge the two worlds. Conversational UI designers will need to take into account the added psychological demands our brains apply to a verbal partner, and voice UI designers will need to broaden the variables and concepts they account for to include context, emotion, and past behaviors.

Voice UI’s Blind Spots

So what’s missing to get mainstream voice UIs to the point of “conversational?” And how can academia and industry work together to target these blind spots and turn them into treasure troves of information?

One of the biggest frustrations I encountered as a working voice designer: most of the engineering on today’s voice systems focuses on the interpretation of the sounds themselves. Few resources in industry are yet dedicated to creating a persistent conversational context.

The state of today’s VUI doesn’t mean there’s a lack of interest in this topic. The voice designers working silently behind the scenes are advocating for next-level conversational functionality, but it’s often an uphill battle.

As we worked through situations in Windows Automotive, Cortana, and Alexa, certain patterns came up time and again as particularly challenging — and largely unexplored. Let’s explore several of these common gaps in detail.

Pronouns & Indirect Reference

Most of today’s commercial voice systems are only able to deal with general questions like “What’s this” in very specific situations. It seems like a simple fix: great, let’s search for pronouns and use them! In reality, the problem is actually quite complex.

Schoolhouse Rock taught a generation of kids about pronouns, but our voice AIs haven’t yet learned how to learn from children’s television.

For example, the use of “this” for example usually means a multimodal interaction, where the customer is indicating something visually. How do we make these interactions possible? And can we support multiple layers of pronouns? “This” and “her” as different intents in the same context?

This is the most solved of the problems I’ll offer here, but it is far from fully solved. As mentioned earlier, current voice assistants do recognize use of pronouns in very limited situations, and only for extremely limited periods of time. Furthermore, once they’re interrupted with a different request, their memories are wiped clean. Human minds are far more resilient, and can sift through several conversational threads to figure out what was referenced by “she” or “that”. Can we give our voice user interfaces this same ability?

Long Term Memory

Most voice systems have an extremely limited memory — a few seconds’ worth at best — unless the subject discussed is very specifically tracked.

As a result, most of these systems won’t be able to help you continue suspended tasks. Not a huge deal for, say, setting a 10 minute timer — but if you’re composing an email by voice, this could be problematic.

In a world where development resources are limited, it’s also important to ask whether this sort of suspended task is more or less valuable than building a system that remembers past topics of conversation. For example: if I ask you for the weather on Saturday in advance, then ask for it again days later, could the response focus on whether the forecast has changed?

This may be an example of an excessively long term memory in a conversational UI. (From the movie WarGames)

Emotional Intelligence

Any perceived emotional intelligence in your favorite voice agent is probably a series of simplistic parlor tricks. On one hand, that’s fine — no point building complex systems that we aren’t ready to use — but by ignoring the emotional state of our customers, we’re missing opportunities to be humane and sensitive in our choice of responses.

Most of today’s voice operated devices have a single mode of rich input: voice. But pulling emotion from voice is an incredibly hard unsolved problem, and is very variable by individual. And yet, other sensors are at arm’s reach — literally. Fitbit devices and others with heart-rate tracking are one source. The new Kinect could track heartbeat and body temperature with a pair of RGB and infrared cameras. What could we do with this information (responsibly) to make our devices seem more empathetic and responsive?

Fitbit’s analysis of the painful Seattle loss in the Superbowl based on heart rate. What does it tell us that Boston fans averaged a lower heart rate than Seattle? That the spikes are mirrored across fans of both teams?

Behavioral Modeling

Most voice system engineering is still so focused on the basic mechanics that we’re not actually tracking usage of voice intents. Again, if we know you’re not using a feature, we can bring it to your attention. What about your habits? Can we make your life easier by anticipating your needs? That process starts by modeling your behaviors and needs over time.

This is less about designing voice UX and instead reducing the need for voice UX by automating frequent tasks. It’s an important example of a case where designers need to be involved in discussions far beyond the scope of UI deliverables. What signals are important for us to track and learn from? As champions of a customer’s end-to-end experience, designers can also advocate for these back-end systems that lead to streamlined, graceful front-end voice UI.

In the maligned but more-prophetic-than-expected Back to the Future 2, Jennifer’s thumbprint triggers a series of automatic behaviors: lights on, screens on, welcome message. Of course, the system couldn’t protect her from herself…

It’s important to note that any system that tracks insights or behaviors over a long period of time must be highly secured. The knowledge of your working hours could be used to vandalize your home. Knowing the locations you’ve tracked weather for could allow someone to track you down in the “real world”. Don’t start modeling behavior or memory until you’re ready to treat that data as precious.

Etiquette

Is “please” optional or required? Are kids forced to say please? What requests are more likely to elicit polite responses, like “Thank you” — and should we take those as invitations to more social conversation?

In some cultures beyond the United States (ie Japan, China), etiquette can carry make-or-break significance for trust and comfort with your system.

This is SUCH a hot topic; it’s frequent customer feedback that seems simple but has deep systematic implications. In particular, parents who own voice assistants in their homes (Alexa, Google Home) often complain publicly that they wish the device would REQUIRE “please” from their children. But without voiceprinting, that would mean everyone in the household must do the same — and universally asking ‘please’ for every request WILL become tiresome.

There’s a lot more to etiquette than “please” — and the concept is a constantly moving target that changes by location, time, social station, culture, and situation.

A more sensitive approach would require a great deal of contextual awareness. Which requests, in specific, warrant a ‘please’? Is one ‘please’ per conversation enough? How does the system respond when the conditions aren’t met? Bluntly, with silence, or eloquently (which will slow down conversation)?

This is not an exhaustive list, but it is an important one. Beyond each of these unsolved problems (indirect reference, long-term memory, emotional intelligence, behavioral modeling and etiquette) are many opportunities to take our favorite voice-enabled assistants to the next level.

Putting the Money where the Voice Is

If you think about these blind spots from a capitalistic perspective, you start to see why they’re still open questions: it’s hard to associate a dollar value or profit with this “connective tissue.”

Designers are motivated to make our voice systems seem more humanistic, but what is the business case?

Alexa, Cortana, Google Home… these systems are all built and prioritized as profit centers, in the end. And now that they’re built and generating revenue, they are focused on incremental change. Short of a new player entering the market from scratch, major changes in the way we approach VUI will require a strong case at a business level.

So how can academia help? First and foremost, by taking these blind spots and researching the heck out of them. Explore them. Play with them. Research the effect of these systems and help quantify the benefits. Your work may yet provide VUI and CUI designers within consumer companies enough ammunition to take the entire industry forward.

In the meantime, private-sector and academic designers and researchers can potentially work in parallel to identify more longitudinal conversational patterns. How can we make conversations more humanistic, more intelligent, and more stateful beyond the scope of a single interaction? There’s plenty of what-if thinking about conversation, and plenty of tactical VUI work that focuses on specific actions, but very little that focuses on actions and trust-building over a nontrivial period of time.

Charting our Course

Voice user interfaces are finally mainstream after decades in the works. A wide variety of customers under-served by visual/physical UI are now finding technology that adapts to their needs — and beyond that, other customers are finding that screenless interfaces delight with their lack of friction.

Still, we’ve identified several key gaps between today’s voice user interfaces and the inspirational ideas put forth by entertainment and academia. Quantifiable benefits from solutions to these problems are as yet largely uncharted, and that’s an important piece of the puzzle.

As an industry, we need our academic partners to explore the collision of these two spaces and the psychological implications of this fusion, just as Clifford Nass did for voice user interfaces a decade earlier.

Meanwhile, as a design community we can begin to look beyond command-and-control to other conversational frames. We can begin developing design patterns and best practices to use as building blocks in new voice user interfaces. If we’re lucky, these patterns may even make their way into existing systems.

May the voice be with you.

After several years focusing her design efforts on NUI and VUI at Microsoft and Amazon, Cheryl is currently Design Lead for the Azure Portal + Marketplaces at Microsoft. Find out more about her diverse background, portfolio, and speaking engagements at her blog or on LinkedIn. You can also follow her on Twitter.

<a href="http://ift.tt/2lIYHWK">http://ift.tt/2lIYHWK</a>

Blind spots in today’s voice user interfaces was originally published in Prototyping: From UX to Front End on Medium, where people are continuing the conversation by highlighting and responding to this story.

Blind spots in today’s voice user interfaces

Cool stuff, every week

More Articles