Data Storage with UWP

Introduction

This is one in a series of post that I am writing on the Universal Windows Platform. In this post I give a breif introduction to some of the storage methods available in UWP applications with the intention of conentrating on Entity Framework and SQLite in a later post. On almost every platform on which you program you'll encounter the need to save data. This may be as bit flags in memory or as rows in a database. UWP's available methods of is a bit different from many of the other .Net/C# environment in how files are accessed. Someone that has been working mostly on WPF applications trying to work with files for the first time in UWP can make one feel a little lost at first. But it's easy to catch onto. My goal in writing this document is to give others an introduction to persistent storage n UWP with the hopes of preventing the lost feeling.

Table of Contents

• Updating the Manifest
• LocalSettings
• File Pickers
• File Access
• StorageFile and Storage Folder
• Known Folders
• Downloads Folder and Download Files
• File Pickers
• Opening a file
• Saving a File
• File Picker Portability
• Folder Picker
• FileIO
• Access Caching
• Remote Files
• External Drives
• Entity Framework
• Closing Remarks
• History

Updating the Manifest

Some of the code discussed below requires changes to the applications manifest. The manifest contains a collection of information about your application including permissions that it may need. Within your UWP projects there will be a file named Package.appxmanifest. Double-clicking on it will open a UI that allows the manifest to be easily edited. You could edit it with a text editor too, but I will assume that you are using the UI. One tab of interest it eh Capabilities tab. When I mention adding a capability you will need to open this UI and ensure the checkboxes for the needed capabilities are checked.

An application may also need to make a declaration. Within this post the only declarations of concern are that an application is capable of handling a specific file type. To add a file type declaration click on the Declarations tab from the Available Declarations drop down, select File Type Association, and click on Add. Minimalistically the information you will need to enter includes a name (Which must be all lower case with no spaces or special characters) and one or more supported file types which a file extension (preceeded by a period) and optionally a mime type.

Local Settings

For storage of small, simple amounts of data ApplicationData.LocalSettings will suffice and is easy to use. ApplicationData.LocalSettings is an ApplicationDataContainer The property of most interest is the Values property. The Values is a Dictionary. The name that you use for a setting will be a string of up to 255 characters in length. The data for the value can be up to 8K size for simple values of Windows Runtime base types. When storing more complex values a collection of names and values can be packaged in a ApplicationDataCompositeValue up to 64K in size and assigned to a value. The ApplicationDataCompositeValue also can contained Windows Runtime base types.

Assigning a name to a key value is all that needs to be done to save it. The run time will take care of persisting the values and loading them back into ApplicationData.LocalSettings when the application runs again. In the following code I am getting the date and time that the application was run for the first time. The very first time the application is run there will be no setting saved for the associated key. That means that this is the first time that the code has run and it immediately saves the current DateTimeOffset to be loaded the next time the application run. The code also loads the name of the user which is stored under the key UserName. This value is a composite value containing values for both FirstName and LastName/ If no name is found a default name of John Doe is used.

const string FirstRunKey = "FirstRun";
const string UserNameKey = "UserName";
const string FirstNameKey = "FirstName";
const string LastNameKey = "LastName";

var settingValues = ApplicationData.Current.LocalSettings.Values;
DateTimeOffset firstRunDate;
String firstName = "John", lastName = "Doe";
Object temp;
if(settingValues.TryGetValue(FirstRunKey, out temp))
    firstRunDate = (DateTimeOffset)temp;
else
    settingValues[FirstRunKey] = firstRunDate =DateTimeOffset.Now;

if(settingValues.ContainsKey(UserNameKey))
{
    ApplicationDataCompositeValue nameValues = (ApplicationDataCompositeValue)settingValues[UserNameKey];
    firstName =(String) nameValues[FirstNameKey];
    lastName = (String)nameValues[LastNameKey];
}
else
{
    ApplicationDataCompositeValue nameValues = new ApplicationDataCompositeValue();
    nameValues[FirstNameKey] = firstName= "John";
    nameValues[LastNameKey] = lastName = "Doe";
}

File Access

UWP applications run in a within a sandbox. They do not have full access to the file system on which they run. There are a number of locations that the application will have access to. Unlike other .Net environments you will not access external resources directly with a path. Instead your application will either need to ask the user for access to a file or query for files of specific types or within specific library collections that it has declared that it needs to have access to. Hard coded paths to external resources generally will not work in this environment. It is a restriction and consideration that will take getting used to since it is different.

There are folders to which an application will already have access. These folders are specific to the application, so other applications cannot see their content. Sharing files between two applications can be done if the two applications register for the same file type and write their data to one of the librarys or by the user granting access to a file resource for both files.

StorageFile and Storage Folder

The interface IStorageItem is used to manipulate and get information on files and folders. For items that are files the IStorageFile interface will also be implemented. It allows the contents to be copied, moved, and opened. Folders will have the interface IStorageFolder implemented. It has methods for enumering the files within it and creating additional files and folders. Ofcourse to make any calls on these interfaces one first needs to get references to files and folders.

Known Folders

There are a number of collections of folder that an application is expected to need access to at some point. These folders are organized in groups called Libraries. Libraries are logical collections of folders intended to hold of certain filet types. It is possible for files within the same library to be stored in different locations on the file system or even on different machines. These include the user's documents folder, the music folder, pictures, video, and removable storage devices. An application must declare that it needs access to these folders. The declaration is made in the application's manifest. If you double-click an application's Package.appxmanifest in your UWP project and select the Capabilities tab in the resulting window you will see a list of capabilities that you can declare. The items relevant here are Music Library, Video Library, Pictures Library, and Removable Storage. The capability for the Document's library does not appear, but it does exist. It can be added by opening the manifest as text. This only works if the application has also registerd a file type. If an application has the required capability declaration it can get a reference through the associated folder with the KnownFolders static class or by calling StorageLibrary.GetLibraryAsync(KnownLibraryId). The names of the folders referenced in KnownFolders and the values that can be passed to GetLibraryAsync.

When files of a specific type are needed a query can be build with the QueryOptions object and the extensions to the files to be returned.

async void PopulateSongList()
{
    QueryOptions queryOption = new QueryOptions(CommonFileQuery.OrderByName, new string[] { ".mp3", ".mp4", ".wma" });
    Queue<IStorageFolder> workFolders = new Queue<IStorageFolder>();
    var fileList =await  KnownFolders.MusicLibrary.CreateFileQueryWithOptions(queryOption).GetFilesAsync();

    foreach (var file in fileList)
    {
        {
       		//the file variable now holds a reference to one of the song file
            svm.SourceFile = file;
            SongList.Add(svm);
        }
    }
}

Application Folders

The storage folder representing the files within the application package can be acquired with the property Windows.ApplicationMode.Package.Current.InstalledLocation. It is also possible to directly access a file within the package using a URI. The URI for files within the package can be formed by prefixing the name of a resource with ms-appx:///. The URI would be passed to the static method StorageFile.GetFileFromApplicationAsync(String URI).

An application will also have access to a local folder, roaming folder, and a temporary folder. These are folders that while accessible to the application might not be available to the user directly. The local folder is specific to the device on which the application is running. The roaming folder is where you would store information that is to be backed up and synced access machines. And the temporary folder is to be treated as a a working space and could be deleted at any moment that the machine needs to free space. The contents of the local folder will persist until the application deletes them.

Downloads Folder and Download Files

All applications have access to a Downloads folder and are able to create files within it without any special capabilities being needed. Applications do not have access to each other's downloads. There is also a BackgroundDownloader class that can be used to download and save information to files. Given a URL and a IStorageFile in which to say it the BackgroundDownloader will take care of creating a DownloadOperation to save the data to a file. The DownloadOperation doesn't begin the transfer

BackgroundDownloader _downloader =  new BackgroundDownloader();;
String NewDownloadUri = "<a href="https://c1.staticflickr.com/1/335/18928517216_1f4cfcc0e5_o.jpg">https://c1.staticflickr.com/1/335/18928517216_1f4cfcc0e5_o.jpg</a>";
String fileName = NewDownloadUri.Substring(NewDownloadUri.LastIndexOf("/") + 1);
IStorageFile newFile = await DownloadsFolder.CreateFileAsync(fileName, CreationCollisionOption.GenerateUniqueName);
var newDownload = _downloader.CreateDownload(new Uri(NewDownloadUri),newFile );
newDownload.StartAsync();

File Pickers

When your application needs for the user to select a file for opening or saving the application can make use of the FilePickers. The file pickers are similar to the OpenFileDialog and CloseFileDialog classes with a significant exception being that while the file dialogs will return the full path of the file selected the file pickers do not. The overall usage of how to use either class is otherwise similar. The file stream being written or read isn't necessarily persisted in the device's storage. The user could have selected a file location on OneDrive. Because of the way that the handling of files is abstracted away your application does not need to do any special handling for these cases. Whether the file is from local storage or managed through some other service your code will be the same.

To use the file pickers you must identify the files types that your application can open and whether you wish to open the file for reading or for writing. The types of files that your application can open are identified through the file's extension. Sometimes a file type might be identified by more than one extension; static HTML documents might have either htm or html as an extension. Information on file types is passed to the file pickers in two objects; a string being a friendly name for the file type and an array of one or more strings that contain the extensions associated with the type.

The file pickers will return a StorageFile that can be used for reading and writing. The following code example is taken from the text editor from a post titledIntroduction to HoloLens Development with UWP with minor modifications. In the Init() I load the file types and their extensions into a Dictionary. This isn't strictly necessary but is a convinient way of handling the files types. For both the open and close codethe code is similar.

Opening a file

The opening of a file for reading can be done with the following steps.

1. Create a FileOpenPicker
2. Add the extensions to the picker's FileTypeFilter collection
3. Request a StorageFile by calling the picker's async PickSingleFileAsync()
   (If requesting multiple files use async PickMultipleFilesAsync() instead)
4. If the returned value is null then the user cancelled/closed the dialog. Act accordingly
5. Read from the stream

Dictionary<string, IList<string>> FileTypeList ;
public void Init()
{
	FileTypeList = new Dictionary<string, IList<string>>();
	FileTypeList.Add("Text Document", new List<string>() { ".txt", ".text" });
	FileTypeList.Add("HTML Document", new List<string>() { ".htm", ".html" });
}

async void OpenFile()
{
	//Create a FilePicker
    FileOpenPicker fileOpenPicker = new FileOpenPicker();
    //Populate the file types 
    foreach (string key in FileTypeList.Keys)
    {
        foreach (string extension in FileTypeList[key])
        {
            fileOpenPicker.FileTypeFilter.Add(extension);
        }
    }
    //Get the Files
    StorageFile file = await fileOpenPicker.PickSingleFileAsync();
    if (file != null)
    {
        Text = await FileIO.ReadTextAsync(file);
        FileName = file.Name;
    }
}

Saving a File

Opening a file for writing can be done with the following steps.

1. Create a FileSavePicker
2. Add the file types to the FileTypeChoices collection of the FileSavePicker
3. Request a StorageFile by calling the picker' async PickSaveFileAsync()
4. If the returned value is null then the user cancelled/closed the dialog. Act accordingly
5. Write to the Stream

async void  SaveFile()
{
    FileSavePicker fileSavePicker = new FileSavePicker();
    foreach(string key in FileTypeList.Keys)
    {
        fileSavePicker.FileTypeChoices.Add(key, FileTypeList[key]);
    }
    StorageFile file = await fileSavePicker.PickSaveFileAsync();
    if(file != null)
    {
        var sf = await file.GetParentAsync();
        var x = sf.Provider;
        CachedFileManager.DeferUpdates(file);
        await FileIO.WriteTextAsync(file, Text);
        FileUpdateStatus status = await CachedFileManager.CompleteUpdatesAsync(file);
        FileName = file.Name;
    }
}

File Picker Portability

In testing the APIs across different UWP implementations I've noticed that on the Xbox One and on the Windows IoT implementations while the File Picker interfaces exists there doesn't appear to be any UI associated with them. Calling them will not result in a UI being display and instead results in null being returned when the storage file is requested. There doesn't appear to be any method for probing whether or not a working version of this interface is present on a device. In the absence of any sanctioned way detecting whether or not this is available I've resorted to timing how long it takes to get a response back from the file picker. If the return value is null and the amount of time it took for the method to return is extremely low it's likely that a non-working implementation is present. But it could also mean that the user was holding the escape key when the dialog was opening. So the timing method is at best considered a hack.

Folder Picker

Use of the FolderPicker is much like the use of the file pickers. Instantiate a FolderPicker, call the method to show the picker (PickSingleFolderAsync()) and if it returns a value then the user has selected a folder that you can use. Add the folder to the FutureAccessList so that it can be accessed later. The FolderPicker also allows the starting location to be suggested using the PickerLocationId enumeration. The values it defines include DocumentsLibrary, Downloads, MusicLibrary, PicturesLibrary, VideosLibrary, and Objects3d.

var folderPicker = new Windows.Storage.Pickers.FolderPicker();
folderPicker.SuggestedStartLocation = Windows.Storage.Pickers.PickerLocationId.Desktop;
var folder = await 	folderPicker.PickSingleFolderAsync();
if(folder!=null)
{
	StorageApplicationPermissions.FutureAccessList.AddOrReplace("OutputFolder", folder);

}

FileIO class

The FileIO class is a static class that serves as a helper for certain file operations. It acts on an IStorageFile instance passed to it in its first parameter. This helper class would be used for operations such as appending strings to the end of a file, reading or writing a file as a string or list of strings (Where each element in the list is a different line in the file) or just reading and writing bytes from the file.

var targetFile =await ApplicationData.Current.LocalFolder.CreateFileAsync("TestFile.txt", CreationCollisionOption.GenerateUniqueName);
await FileIO.WriteTextAsync(targetFile, "This content will be written to the file");

IStorageFile fileToRead = await ApplicationData.Current.LocalFolder.GetFileAsync("TestFile.txt");
string contents = await FileIO.ReadTextAsync(fileToRead);

File Type Associations

The types of files that an application can handle are declared in the application's manifest (usually named Package.appxmanifest). If you double-click on this file in Visual Studio you'll be able to change the manifest information through a UI. Select the Declarations tab within the UI. From the dropdown select File Type Associations and click on Add.

When the user attempts to open a file with the application through file type associations the application will receive notification of the opening requres through void Applicaiton.OnFileActivated(FileActivatedEventArgs args). You will need to override this event within your application's app class. A list of the files being requested (there can be more than one) file be passed through the event arguments passed to this class in the Files property.

Access Caching

Once a user grants access to a storage item you can add your access token for the file to the list of files to which you have access. At the time of this writing that can be up to 25 files. If there are files that you will need at some future point you can also add references to those to a list of files to which you will need access. These can be managed in the static class StorageApplicationPermissions. The class has two properties. MostRecentlyUsedList is for holding the storage items that you've recently accessed and FutureAccessList is for storage items that you have yet to access. The application can be terminated, restarted, and can still have access to this list. Files will automatically be removed from MostRecentlyUsedList once the list reaces capacity and more files are added. The access token that is the most stale will be the one removed from the list. When items are removed from this list the MostRecentlyUsedList has an ItemsRemoved event that is fired. You can add an event handler to receive notification of an item being removed.

The AccessListEntry elements have two pieces of data. One is the Token, a string value that you can use to retrieve the file again and the other is the Metadata, which defaults to empty but can have some value that you have assigned to it.

Files from Remote File Systems

If your application works on a file that was given to it by another service (such as OneDrive) Windows will take care of getting that application's copy of the file updated when changes occur. However if you have a number of operations to perform on the file you will not want Windows making attemps to update the file at the same time. To prevent this you can use the CachedFileManager to defer and updates to the file until you complete your intended operations. This static class has two methods of interest. DeferUpdates(IStorageFile) will prevent updates being made to the remote file. When modifications on the file are complete it can be released for updating by calling async CompleteUpdatesAsync(IStorageFile). When releasing the file a FileUpdateStatus value is returned.

External/Removable Storage

Removable drives such as flash drives, external hard drives, and memory cards can be discovered through the KnownFolders.RemovableDevices. The folders that are returned by this collection are the root directories of attached drives. Access to the drive doesn't mean access to all of the files on the drive. The application will only be able to detect files of the type for which it has registered. If the application has not registered for any files types attempting to enumerate the files on a removable drive will result in an ACCESS DENIED exception.

List<istoragefolder> DriveList = new List<istoragefolder>();
foreach (var device in await KnownFolders.RemovableDevices.GetFoldersAsync())
{
    DriveList.Add(device);
}</istoragefolder></istoragefolder>

Entity Framework Core with SQLite

With the 2016 Windows Anniversary update SQLite version 3.11.2 is to be released. SQLite is a light weight single user database system. It runs in-process, so there is no setup for a database server, no need for configuration, and contains all of its data within a single files. Support for SQLite can be added to a project using NuGet. To add support to your project in Visual studio open the Tools menu and select NuGet Package Manager and then Package Console. To install support type Install-Package EntityFramework.SQLite -Pre. You will also want the commands package which you can install by typing Install-Package EntityFramework.Commands -Pre. The reason for the -Pre argument here is that at the time of this writing these are in pre-release form. The release versions of these will be out soon. After its release I will be writing another post dedicated to Entity Framework with SQLite and adding a link to it here.

On UWP you can use SQLite with EntityFramework. With EntityFramework the datatypes to be saved in the tables are defined in code. The code samples that follow are from a location logger. The data being saved is divided into two types. There is an individual location and there are sessions in which timestamped locations are grouped. Entity Framework being code first has you to define the data structures in code first and the database is derived from the code. Here is the Location class.

using System;

namespace SQLiteSample.Data
{
    public class Location
    {
        public DateTimeOffset Timestamp { get; set;  }
        public double Latitude { get; set;  }
        public double Longitude { get; set;  }

        public double Altitude { get; set;  }

        public double HorizontalAccuracy { get; set;  }
    }
}

Sessions will have a GUID that's being used as a primary key. I've flagged this property as a key with an attribute. Note that Entity Framework will also automatically assume that any property named Id or (type name)Id is a key property. The association between sessions and locations is modelled with the ICollection./

using System;
using System.Collections.Generic;
using System.ComponentModel.DataAnnotations;


namespace SQLiteSample.Data
{
    public partial  class LogSession
    {
        [Key]
        public Guid SessionID { get; set;  }
        public DateTimeOffset  SessionStart { get; set;  }
        public DateTimeOffset? SessionEnd { get; set;  }

        public string Name { get; set;  }

        public virtual ICollection<Location> Locations { get; set;  }
    }
}

Right now these are just loose classes. To get them saved in the database we need to define a class derived from DbContext that includes DbSet<T> collections of these classes. View the DbSet<T> properties as being tables. Within this class we also define the name of the file in which the data tables will be saved and can specify further information for about the tables. In this case I am flagging a value as required.

using Microsoft.Data.Entity;


namespace SQLiteSample.Data
{
    public class LocationLogContext: DbContext
    {
        public DbSet<LogSession> Sessions { get; set;  }

        protected override void OnConfiguring(DbContextOptionsBuilder optionsBuilder)
        {
            optionsBuilder.UseSqlite("Filename=Locations.db");
        }

        protected override void OnModelCreating(ModelBuilder modelBuilder)
        {
            modelBuilder.Entity<LogSession>().Property(b => b.SessionID).IsRequired();
        }
    }
}

To use the database we only need to instantiate the derived DbContect class and call EnsureCreated() method. The EnsureCreated() method will check whether or not the database exists. If it does exists then this method does nothing more. If it does not exists then this method will create it. Once created adding new data to the database is just a matter of instantiating new instances of the classes, adding them to the DbContext (or adding them as a child object of an object that is already collected into a DbContext) and calling SaveChanges() or SaveChangesAsync().

//Create a session and save it
_currentSession = new LogSession() { LogSessionId = Guid.NewGuid(), SessionStart = DateTimeOffset.Now, Locations = new List<location>()  };
_locationLogContext.Sessions.Add(_currentSession);
_locationLogContext.SaveChanges();


private void _locationWatcher_PositionChanged(Geolocator sender, PositionChangedEventArgs args)
{
    var session = _currentSession;
    var coords = args.Position.Coordinate;
    if(session != null)
    {
    	//Create a new location object
        Location loc = new Location()
        {
            Longitude = coords.Longitude,
            Latitude = coords.Latitude,
            HorizontalAccuracy = coords.Accuracy,
            Altitude = coords.Altitude ?? 0,
            Timestamp = DateTimeOffset.Now, 
           // ID = session.Locations.Count,
            //SessionID = session.SessionID
        };
        //Add it to the current session object
        session.Locations.Add(loc);
        //Save the changes
        _locationLogContext.SaveChangesAsync();
    }
}

		</location>

There is much more to be said about Entity Framework. My post about it will be available in the weeks following the release of the 2016 Windows Anniversary update.

Closing Remarks

As mentioned before this is being published close to the time at which Microsoft is preparing a Windows Update. AFter the update there may be additional information to add to this post. Check backs in the weeks after Microsoft's release and look in the history section (below) for a list of the additions made due to the update.

History

• 28 June 2016 - Initial Publication