Over the last few weeks, I’ve been writing a lot about how to use C# with the Raspberry Pi. I’m really interested in different ways that I can use software to interact with the physical world. Another interaction that I’m interested in is using voice commands, and recently I started looking into ways to use Cortana to achieve this. This post is an introduction to asking Cortana to control Windows apps.

In this post, I’ll look at the simple case of setting up a Windows app so that I can ask Cortana to start the app from my phone.

How does Cortana know what to listen for?

There is some seriously advanced technology in the Microsoft Cognitive Services, particularly software like LUIS – but for this simple case, I’ll store the voice commands Cortana listens for in an XML Voice Command Definition (VCD) file.

  • First we need to define a CommandSet – this has name and language attributes. The voice commands will only work for the CommandSet which has a language attribute matching that on the Windows 10 device. So if your Windows device has the language set to en-us, only the CommandSet matching that attribute will be used by Cortana.
  • We also can define an alternative name for the app as a CommandPrefix.
  • To help the user, we can provide an Example command.
  • The most interesting node in the file is Command:
    • Example: Windows shows examples for each individual command, and this node is where we can specify the examples.
    • ListenFor: These are the words Cortana listens for.
    • Feedback: This is what Cortana replies with.
    • Navigate: This is the XAML page that Cortana navigates to when it parses what you’ve said.

The app I’ve modified is my Electronic Resistance Calculator. I’ve added the file below – which I’ve named ‘ResistorCommands.xml’ – to the root of this directory.

<?xml version="1.0" encoding="utf-8" ?>
<VoiceCommands xmlns="http://schemas.microsoft.com/voicecommands/1.2">
  <CommandSet xml:lang="en-us" Name="EnglishCommands-us">
    <!-- The CommandPrefix provides an alternative name for your app -->
    <CommandPrefix>Resistor</CommandPrefix>
    <!-- The CommandSet Example appears beside your app's name in the global help -->
    <Example>Open</Example>
    <Command Name="OpenCommand">
      <Example>Open</Example>
      <ListenFor>Open</ListenFor>
      <Feedback>You got it!</Feedback>
      <Navigate Target="MainPage.xaml" />
    </Command>
  </CommandSet>
 
  <CommandSet xml:lang="en-gb" Name="EnglishCommands-gb">
    <!-- The CommandPrefix provides an alternative name for your app -->
    <CommandPrefix>Resistor</CommandPrefix>
    <!-- The CommandSet Example appears beside your app's name in the global help -->
    <Example>Open</Example>
    <Command Name="OpenCommand">
      <Example>Open</Example>
      <ListenFor>Open</ListenFor>
      <Feedback>I'm on it!</Feedback>
      <Navigate Target="MainPage.xaml" />
    </Command>
  </CommandSet>
</VoiceCommands>

Adding these voice commands to the Device Definition Manager

The Windows 10 VoiceCommandDefinitionManager is the resource that Cortana uses when trying to interpret the voice commands. It’s very straightforward to get the Voice Command Definition file from application storage, and then install this storage file into the VoiceCommandDefinitionManager.

We need to add those definitions at application start up, which we can do by overriding the OnNavigatedTo method in MainPage.xaml.cs.

private async Task AddVoiceCommandDefinitionsAsync()
{
    var storageFile = await StorageFile.GetFileFromApplicationUriAsync(new Uri("ms-appx:///ResistorCommands.xml "));
    await VoiceCommandDefinitionManager.InstallCommandDefinitionsFromStorageFileAsync(storageFile);
}
        
protected override async void OnNavigatedTo(NavigationEventArgs e)
{
    if (e.NavigationMode == NavigationMode.New)
    {
        await AddVoiceCommandDefinitionsAsync();
    }
}

At this point, we actually have enough code to allow us to ask Cortana to start our app.

Running the app on a Windows 10 device

I added the VCD ResistorCommands.xml file to the root of the Electronic Resistance Calculator project, and I added the code snippet above to MainPage.xaml.cs, and ran this in debug mode on my Nokia 1520 Windows 10 device.

When I activate Cortana, I can click on the hamburger menu and select Help in the top left to see the list of apps which are controlled by voice commands. My Electronic Resistance Calculator is available – you can see in the screenshot below that the word “Open” as an example voice command is visible.

wp_ss_20160630_0003

If I click on the Resistor app, the phone shows a list of valid example commands. Because we’re just opening the app, there’s just one example – “Open”. Obviously we can do more complex things than this with a VCD, which I’ll show in a later post.

wp_ss_20160630_0005

When I say “Resistor Show”, Cortana recognises this and replies with “I’m on it” – the feedback specified for devices set to have language “en-gb” (which is correct for my device). After a short pause, the app starts.

wp_ss_20160630_0004

In a later post, I’ll look at how to use the VCD to issue more complex voice commands.