Published on | Prototyping: From UX to Front End — Medium Sjur O. Sundin

Introduction, prerequisites, and building the Google Action

In this tutorial I will show you how to navigate a website by talking to your Google Home. This is the Google Home version of a similar tutorial I’ve written on Amazon Echo/Alexa. If you’re interested in setting up the same solution for Alexa, you can find that tutorial here.

This is part 1 out of 3 articles in this tutorial:

Part 1: Introduction, prerequisites, and building the Google Action
Part 2: Setting up the database and connecting our Google Action
Part 3: Creating the website and extending our Action to display all images

After completing this tutorial, you will be able to tell Google Home what you want to see in the browser, and the website will be updated accordingly. Since this is a proof of concept, we will build a very simple website showing pictures of dogs. Once you have built a simple solution like this, you can easily extend the functionality to more complex webapps.

Here’s a quick demo of what we will accomplish:

<a href="https://medium.com/media/d456d4225ceb00fa2723d2e7019f1434/href">https://medium.com/media/d456d4225ceb00fa2723d2e7019f1434/href</a>

There are multiple use cases for navigating a website using your voice. You may want to see videos and instructions for a recipe while your hands are busy cooking. A doctor may want to see a patient’s medical images and lab results while examining the patient. A gallery of images and videos is therefore a good use case for voice navigation. In this example we will navigate a small gallery of dog pictures.

Disclaimer: The techniques used in this prototype are not suitable for a live production environment. This is meant as a prototype and proof-of-concept. It works, but to go live would require additional work to make your solution secure and scalable. Throughout this series I will try to highlight the specific elements that you may need to improve in a live production solution.
Make it work, then make it better.

How it all works

The end goal is to talk to Google Home and have your website update to show what you tell Google you want to see. There are a lot of moving parts to accomplish this:

I will go into details for each step of the solution, showing you how to set it up so it all works together.

Of course, this isn’t the only way to accomplish this. One goal of this tutorial is for newcomers to become familiar with just some of the amazing tools available through Google Cloud and Firebase, and this is one way you can utilize the Google ecosystem to build this prototype.

All of the code and files needed for this project is available on GitHub.

Before you Begin

For this project you actually do not need a Google Home device. The simulator available through Google Actions is excellent and lets you easily test your solution every step of the way. I highly recommend getting a Google Home. A mini can currently be had for $49. This series was written and tested on a Google Home Mini.

You need a Google account, your regular Gmail account works well. You will use this account to set up and access the following services:

Google Actions on developers.google.com

Google Firebase on firebase.google.com

Before you proceed, I recommend that you visit the links above and use your Google account to register for those services.

Google Home Mini

If you have a physical device, such as the Google Home Mini, you also need to set that up with the same Google account that you use to sign up for those services. If you do not have a device, you can use the simulator on Google Actions to test your app.

This entire solution will also work on the Google Assistant app on the iPhone/Android, so you can also use that interface to test your solution.

Setting up the Google Action

In this chapter we will set up the Google Action that will listen for our voice commands and confirm what number we asked for. This can be done with no coding, all the interactions will be designed using Dialogflow. In chapter 2 we will write some code to store the number in a database.

Unwrap and set up your Google Home device (see this page for help with that), and go to Google Actions to begin setting up your Action.

To start building the Action for Dog Pictures, click the ‘Actions Console’ button at the top right of the screen, or navigate directly to console.actions.google.com.

On this screen, click on the ‘Add/Import project’ and give your project a name:

Hit the ‘Create Project’ button, this sets up the project and takes you to its project home screen on Google Actions:

The Dog Pictures project on Actions on Google

As you can see from the screenshot above, you have many different options when it comes to building your custom app. For the purpose of this tutorial we will use Dialogflow. I’m confident you can also use the Actions SDK to build this prototype, but I have not tested that.

Click the ‘build’ button on the Dialogflow card, and then click the ‘Create actions on Dialogflow’ button on the modal overlay to get started:

When you click the ‘Create actions on dialogflow’ button you’re taken to a new tab where you will design your action with Dialogflow. The modal stays open in the tab you left, and you can go back there and click the ‘OK’ button if you like.

The new tab opens with the console for Dialogflow:

The first thing you need to do on this page is to hit the ‘Create’ button at the top right next to the name of your project (Dog-Pictures). Doing this finishes the set up of your action, and you’re ready to design the conversation. Hit the ‘Create’ button, and your next screen should look like this:

You’re now on the ‘Intents’ screen on the console for your Dog-Pictures action on Dialogflow. An intent represents a mapping between what a user says and what action should be taken by your software. In our case, we will ask for a picture, and the action taken will be to store the number we asked for so that it can be read by the webpage we will build later.

Our intent is to see a dog picture. This is fulfilled through an action.

You can read more about intents in the Dialogflow documentation.

We can now start editing the default welcome intent. This intent is the starting point for your Action. To do this, click on the default welcome intent.

Clicking on it should take you to the screen for editing this intent:

The welcome intent is the intent that greets you and serves as a starting point for your app. We want our app to say Welcome to Dog Pictures when we start it. To achieve that, click on the trashcan in the upper left corner of the ‘Text response’ section.

Click the trashcan to the right, this deletes all the standard responses.

After deleting them, click the ‘Add message content’ button and choose ‘Text response’:

In the text box type the following response:

Then click ‘Save’ at the top of the page next to ‘Default Welcome Intent’:

You’re now ready to test on your device. If everything works, Google Home (or the Google Assistant on your phone) will reply with “Welcome to dog pictures!”.

The easiest way to test this is in the built-in simulator, where you can type in sentences and immediately see the replies. After saving your intent, look to the right of the screen, and you will see a link saying ‘See how it works in Google Assistant’. Clicking this link takes you to the simulator where you can test your app. In the screenshot below the link is to the right, that’s the one to click.

The simulator should open in a new tab, and look like this:

Make sure that Dog Pictures is the current project selected at the top:

In order to test your app, you have to invoke it with the phrase “Talk to my test app”. If you’re talking to your Google Home device, you have to say “Hey Google, talk to my test app.” or “Hey Google, open my test app”.

If you type “Talk to my test app” into the input box (it should already be filled out for you), and see the result. It should look like this:

If it works, you should see the reply as shown in the screenshot above. If that worked, congratulations, you’ve completed the first step. You have a test app running on Google Home, and it speaks back to you.

What if it didn’t work?

If you already have another test app running, you have to first disable testing for that one before you can test Dog Pictures. Currently you can only have one test app running.

To disable testing for another app, you have to go to that app in the Dialogflow console by clicking the drop down menu to the left. As you can see in the screenshot, I’m switching to the Number-Test project which is my currently active test app on my device. I have to disable that test before I can test Dog Pictures.

Once I’m in the Number-Test project, I have to again click on the link to launch the simulator.

Once in the simulator, I have to disable testing for this project, I do this by clicking this little icon to the right of the ‘Change Version’ button.

As you can see from the screenshot, hovering over the icon tells me that testing is enabled for this app. If I want to test a different app, I have to disable testing for this app, I do that by clicking this icon:

Once clicked, it should look like this:

Notice that you cannot re-enable testing on this screen just by clicking the icon again, you have to go back to the Dialogflow console and re-initiate testing from there.

Now that we’ve successfully disabled the other app we had running, we can go back to the Dialogflow console and initiate testing for Dog Pictures:

When you’re back in the Dog Pictures project, click the Google Assistant link to open the simulator again. Your project should be ready for testing, all you have to do is type “Talk to my test app”:

Testing on device

If you’re testing by speaking to your device, you will notice that the conversation doesn’t end, Google Home is waiting for more input from you.

This is a good thing since we will later ask for dog pictures and we want Google Home to listen to our commands.

Speak a number and have Google Home repeat it back to you

Next we will add the ability to say a number. For now we will just have the device repeat the number back to us to make sure that our commands are being understood. Later we will store the number in a database.

To do this, we need a new intent, we will call it Show picture. Go to the Intents screen of your Dog Pictures project, and click the ‘Create Intent’ button.

This will bring up the UI for creating a new intent:

We will call the new intent ‘Show picture’, so you need to rename the intent from ‘Intent name’ to ‘Show picture’, you do that at the top of the page by clicking on ‘Intent name’, this allows you to rename the intent.

Rename the intent and click ‘Save’.

The next thing we need to do is to determine what the user can say. For now we only need one expression: “Show picture {number}.”, Where {number} is the number of the picture we want to display.

In the ‘Add user expression’ input box, type the following: “Show number 3”, then highlight the number 3. This brings up a menu that lets you pick what entity this number represents. From the dropdown list, choose @sys.number.

Your input box should now look like this, notice how parameter name, entity and resolved value has been added below the input box:

What we’ve done is to tell Dialogflow that every time we say Show number, what comes after number has to be a legal number, such as 4 or 1000.

We now have to fill out the Action field of the intent. We will name the action ‘showPicture’:

This name is what we will later use in chapter 2 to trigger code to save the number to the database.

Next we need to say that it is required, so check the ‘required’ checkbox, and put number in the parameter name field, like this:

Hit tab after writing number, and you will get a warning:

In entity put @sys.number, and in value put $number, like this:

If you click Define prompts you can specify custom replies if the user does not say a number but says something like Show number cheesecake.

If the user says something that isn’t a number, he or she will be prompted to please say a number.

Our final step is to have our number repeated back to us. We do this by utilizing the parameter value $number that we defined in the Action section of the Intent UI, here:

This now holds the value of the number the user asked for, and we can use that to repeat it back to the user. To do that, go to the text response section, and where it says ‘Enter a text response’ write the following: You said number $number.

Your intent definition should now look like this:

Finally, hit ‘Save’ at the very top of the page, next to the intent name. This step is easy to forget…

Your intent is now saved, and you can test it in the simulator by clicking the Google Assistant link.

As you can see, the number is being repeated back, and the conversation recovers and continues even when the user says a non-number:

This should also work when talking to your Google Home or similar device.

One cool thing about Google Actions is that the user can say variations of the expression, you don’t have to cover every possible variation of what the user might say. Even though we only specified the expression Show number {number}, the app will still work if we for instance say Let’s see number {number}, or just Number {number}.

Summary and next steps

You’ve successfully created a Google Action that lets you speak a number to your device, and have that number repeated back to you. All of this was done with no coding.

You’ve completed part 1 of the prototype:

In the next chapter we will do some coding to take the number and store it in a database. We will then access that database through an API so that the webapp knows which number was requested. We will also extend our dialogue to handle asking for all pictures.

Part 2: Setting up the database and connecting our Google Action

Using voice to control a website with Google Home, part 1/3 was originally published in Prototypr on Medium, where people are continuing the conversation by highlighting and responding to this story.

Using voice to control a website with Google Home, part 1/3

Introduction, prerequisites, and building the Google Action

Top Stories