Handsfree with Cortana!

• Download source (RAR) - 1.6 MB
• Download source (ZIP) - 1.7 MB

Intro (Skip If You Already Know Who Cortana Is)

Getting your work done by not launching the desired app is more common these days. Since every platform has its own personal assistant to get your work done as quickly as possible, why not leverage it. If you are not sure yet about what kind of personal assistant I’m talking about. Well they are Cortana, Siri and Google Now. They are available in Windows, iOS and Android devices respectively. Let me talk about Cortana today and show you how you can integrate her in your own app. Cortana was introduced initially with Windows Phone 8.1 and now she is also available in Windows 10 devices (devices runs on Windows 10 OS). In Windows 10 devices, you can ask Cortana to do stuff for you. She will do the specific task and return the outcome in Cortana canvas. By the way, you can also type in the canvas to get your work done if you want to maintain privacy and of course you don’t want to shout to Cortana in public every single time. You can interact with Cortana in Windows simply by saying “Hey Cortana”. And you will be presented with the Cortana canvas.

How Can I Integrate Cortana In My App?

You can use Cortana to run an app in foreground. Then you can do your stuff with the app. There is nothing much you can do with Cortana if you use it to run a foreground app. Again, what else do you need from Cortana if your app is up and running in front of you. The actual fun begins when you use Cortana to run your app in background and interact with your app from Cortana canvas. So buckle up and let’s do that. I’ve created this simplest Todo application. You can get the source code of the app from the link I provided at the bottom of this post. Here is the sneak peak of the app,

You can add todo items, check them if they are done and can also permanently delete them from the app. How about we use Cortana to get the pending tasks without running the app just by saying “Hey Cortana, My Simple Todo, show pending tasks.”

Cool! You can tweak the source code and can add functionalities like adding, checking and deleting todo items directly from the Cortana canvas. For the time being, let me show you how we can achieve this simple scenario. To be able to run our app in the background, we need a Windows Runtime Component.

Add a class of your own and implement the IBackgroundTask in it. It has a Run method which will eventually run when your app is activated in background. Here is my VoiceCommandService class which implements the IBackgroundTask interface.

public sealed class VoiceCommandService : IBackgroundTask
{
    private BackgroundTaskDeferral _deferral;
    VoiceCommandServiceConnection _voiceServiceConnection;

    public async void Run(IBackgroundTaskInstance taskInstance)
    {
        var triggerDetails = taskInstance.TriggerDetails as AppServiceTriggerDetails;
        if (triggerDetails?.Name == "SimpleTodoVoiceCommandService")
        {
            _deferral = taskInstance.GetDeferral();
            var cancelledTokenSource = new CancellationTokenSource();
            _voiceServiceConnection = 
		VoiceCommandServiceConnection.FromAppServiceTriggerDetails(triggerDetails);
            var voiceCommand = await _voiceServiceConnection.GetVoiceCommandAsync();

            switch (voiceCommand.CommandName)
            {
                case "todoBackgroundCommandList":
                    var command = 
			voiceCommand.SpeechRecognitionResult.SemanticInterpretation.Properties["command"][0];
                    if (command == "pending")
                        await ShowPendingTasks();
                    break;

                default:
                    LaunchAppInForeground();
                    break;
            }

            taskInstance.Canceled += (s, e) =>
            {
                cancelledTokenSource.Cancel();
                _deferral.Complete();
            };

            _voiceServiceConnection.VoiceCommandCompleted += (sender, args) =>
            {
                cancelledTokenSource.Cancel();
                _deferral.Complete();
            };

            _voiceServiceConnection.VoiceCommandCompleted += (sender, args) =>
            {
                cancelledTokenSource.Cancel();
            };
        }
    }
}

The source code contains the rest of the code (download it from the links above). All I did here is get the voice command which activated the background task. Then, we are matching it to our PhraseList defined in the voice command definition file. Since my app is able to show just the pending tasks, the command should match the first command in my PhraseList. Here is my VCD file:

<?xml version="1.0" encoding="utf-8"?>
<VoiceCommands xmlns="http://schemas.microsoft.com/voicecommands/1.2">
  <CommandSet xml:lang="en-us" Name="MySimpleTodoCommandSet_en-us">
    <CommandPrefix> My Simple To do </CommandPrefix>
    <Example> My Simple To do, show pending tasks </Example>

    <Command Name="todoBackgroundCommandList">
      <Example> show pending tasks </Example>
      <ListenFor> {command} task[s] </ListenFor>
      <Feedback> Executing specific command  </Feedback>
      <VoiceCommandService Target="SimpleTodoVoiceCommandService"/>
    </Command>

    <PhraseList Label="command">
      <Item>pending</Item>
    </PhraseList>
  </CommandSet>
</VoiceCommands>

Here, in the command prefix node, we define the app identifier by which Cortana will recognize our app. For example, “Hey Cortana, open my simple to do” will open our app in the foreground. Again, you can have multiple Command node in a VCD file each of which will have a ListenFor node. In this node, you place a placeholder for your desired commands. These placeholders will be replaced by the matched value from the PhraseList. You can also define a similar CommandSet and set the language of the CommandSet in which a user can interact with Cortana. Remember that Cortana doesn’t support all the languages yet. So if you set to a language that isn’t supported, your app may not give you the exact result you are looking for.

Now, add reference of the component library project to the main Windows application project.

Next, you will need to register your voice command definition in App.xaml.cs class with this simple code. You will need to override the OnActivated method if you need deep linking in your app. It's all available in the source code.

private async Task RegisterVoiceCommands()
{
    var storageFile =
        await StorageFile.GetFileFromApplicationUriAsync(
            new Uri("ms-appx:///MySimpleTodoCommands.xml"));
    await
        VoiceCommandDefinitionManager
            .InstallCommandDefinitionsFromStorageFileAsync(storageFile);
}

Not done yet. Open up the Package.appxmanifest file, go to Declarations node and add an App Service from available declarations dropdown. Add the app service identifier and entry point of your background task. Now run the app and add some todos. Next you can query for pending tasks directly from Cortana canvas just by saying, “Hey Cortana, my simple to do, show pending tasks.” Here is a video of the working app for you. Thank you very much for reading.