Windows Phone 8.1 Media Editing API

Introduction

WP 8.1 is one of the major updates for Windows Phone. WP 8.1 has unveiled new APIs, new universal apps concept, inclusion of WinRT APIs, new UI controls and much more. One of the new APIs introduced is media editing API.

In earlier days, we were using Movie Maker or other 3^rd party desktop software to trim the video, to make a composition of multiple video, to apply various effects, etc. We were required to have a computer with more memory & high-end processors, but now our smartphones are powerful enough to do such tasks. So let’s get a deep dive in new media editing API of Windows Phone 8.1.

Windows Media Editing API

Media editing API has MediaClip class, whose object represents a single media, i.e., video, image or a solid color video clip. To perform any kind of editing operation on a media file, you must first create MediaClip object. MediaClip provides three static methods to create object.

MediaClip.CreateFromFileAsync(IStorageFile file)

Which creates a MediaClip object from a video file.

MediaClip.CreateFromImageFileAsync(IStorageFile file, TimeSpan originalDuration)

Which creates a MediaClip object from an image file. The parameter “originalDuration” has use while we create composition of media files. It represents how long the image would be displayed in the composed video clip.

MediaClip.CreateFromColor(Color color, TimeSpan originalDuration)

Which creates a solid color video clip that displays a single color for a specified length of time. Solid color video clips are typically used to create an explicit gap between video segments.

After creating media clip, you will have an object of a particular file. Then you can apply effects as well as editing. Let’s go through them one by one.

Trimming

Trimming the video is now just two steps process. MediaClip object has two properties (i) TrimTimeFromStart (ii) TrimTimeFromEnd. After creating MediaClip, you just have to set these two properties. After setting it, the MediaClip will have trimmed media. You can change later on the trimming times as many times as you want. We will see further how to render this MediaClip into a file.

// Create MediaClip object
MediaClip objMediaClip = await MediaClip.CreateFromFileAsync(file)

// Trim from objMediaClip everything after specified position (1 minute & 20 seconds)
objMediaClip.TrimTimeFromStart = new TimeSpan(0,1,20);

// Trim from objMediaClip everything after the specified position
objMediaClip.TrimTimeFromEnd = objMediaClip.OriginalDuration - new TimeSpan(0,2,9);

Slow Motion & Video Stabilization Effects

Microsoft has MFT (Media Foundation Transform) platform, through which one can developer C++ WinRT components. These can be used to apply various video playback & filter effects.

Windows.Media.Effects namespace is dedicated to pre-defined effects, which can be used to apply on any video file. One of the effects is slow motion effects. Windows.Media.Effects has class SlowMotionEffectDefinition, which is used to apply slow motion effects to media. That class has one property called TimeStretchRate. It refers to how slow the video will be. It takes double value of less than 1. Hence, if you set TimeStretchRate as 0.5, the video speed will be half of the original. Setting its value greater than 1 throws an exception. So don’t consider its use for fast motion video effects.

Now to apply the slow motion effect to any video, first you have to create its MediaClip object which is essential for media editing. MediaClip object has IList<IVideoEffectDefinition> property called VideoEffectDefinitions. So first of all, you have to create object of SlowMotionEffectDefinition and set TimeStretchRate property. Then, add that object to VideoEffectDefinitions list. That’s it, your video is now in slow motion. We will see later on how to preview and how to save video with effects.

// Let's consider we have already MediaClip object.

// Create object of SlowMotionEffectDefinition
var objSlowMotionEffectDefinition = new SlowMotionEffectDefinition();

objSlowMotionEffectDefinition.TimeStretchRate = 0.6;

// Add effect object to MediaClip object's VideoEffectDefinitions list.
objMediaClip.VideoEffectDefinitions.Add(objSlowMotionEffectDefinition);

Similarly, you can also add video stabilization effect. The VideoStabilization effect can help reduce shakiness in video. That effect resides in Windows.Media namespace. It has only one class named VideoEffects, which is a static class. To add this effect, you don’t have to create an object. It has static string property VideoStabilization, which is activatable class ID of that video effect definition.

// Create object of VideoEffectDefinition
// Pass static string property VideoEffects.VideoStabilization as argument
var objVideoStabilization = new VideoEffectDefinition(VideoEffects .VideoStabilization);

// Add effect object to MediaClip object's VideoEffectDefinitions list.
objMediaClip.VideoEffectDefinitions.Add(objVideoStabilization);

Filter & Other Video Effects

Nokia Imaging SDK provides a plethora of filters for image processing. Unfortunately, it doesn’t work for videos. We can solve this problem by developing C++ WinRT components using MFT. According to MSDN, Media Foundation transforms (MFTs) provide a generic model for processing media data. MFTs are used for decoders, encoders, and digital signal processors (DSPs). In short, anything that sits in the media pipleine between the media source and the media sink is an MFT.

MSDN has provided a sample with several effect source code, wiz grayscale, fisheye, pinch, warp and invert. The sample also shows how to apply these effects in playback. No words for saving after applying the effects. So to apply MSDN sample’s effect is quite similar to slow motion & video stabilization effect. You have to first add WinRT C++ components from MSDN sample to your own project and then you have to define activatable class IDs in WP 8.1 project’s manifest file. Add below the given lines by opening manifest file in text editor before </Package>.

<Extensions>
    <Extension Category="windows.activatableClass.inProcessServer">
      <InProcessServer>
        <Path>GrayscaleTransform.dll</Path>
        <ActivatableClass ActivatableClassId="GrayscaleTransform.GrayscaleEffect" 
         ThreadingModel="both"/>
      </InProcessServer>
    </Extension>
    <Extension Category="windows.activatableClass.inProcessServer">
      <InProcessServer>
        <Path>PolarTransform.dll</Path>
        <ActivatableClass ActivatableClassId="PolarTransform.PolarEffect" ThreadingModel="both"/>
      </InProcessServer>
    </Extension>
    <Extension Category="windows.activatableClass.inProcessServer">
      <InProcessServer>
        <Path>InvertTransform.dll</Path>
        <ActivatableClass ActivatableClassId="InvertTransform.InvertEffect" ThreadingModel="both"/>
      </InProcessServer>
    </Extension>
</Extensions>

Now each effect has its own activatable class ID, you have to first create object of VideoEffectDefinition class & pass that activatable class ID as argument in constructor. Finally, add object of VideoEffectDefinition to MediaClip, that’s it.

// Create object of VideoEffectDefinition, pass activable class ID as parameter

var objGrayScaleEffect = new VideoEffectDefinition("GrayscaleTransform.GrayscaleEffect");
objMediaClip.VideoEffectDefinitions.Add(objGrayScaleEffect);

var objInvertEffect = new VideoEffectDefinition("InvertTransform.InvertEffect");
objMediaClip.VideoEffectDefinitions.Add(objInvertEffect);

In MSDN sample fisheye, pinch & warp effect is defined in single WinRT component. To apply any of effect among them, you need to pass configuration parameter, with activatable class. VideoEffectDefinition's constructor has one more overload, which takes a key-value pair to configure the effect.

// First define configuration

PropertySet configuration = new PropertySet();
configuration.Add("effect", "Fisheye");
// configuration.Add("effect", "Warp");
// configuration.Add("effect", "Pinch");

// Create object of VideoEffectDefinition, pass activable class ID and configuration as parameter
var objPolarEffect = new VideoEffectDefinition("PolarTransform.PolarEffect", configuration);

// Add effect object to MediaClip object's VideoEffectDefinitions list.
objMediaClip.VideoEffectDefinitions.Add(objPolarEffect);

Saving Video

We have learned how to apply effects & trimming, now it’s our turn to save the video. MediaEditing API has one more important class called MediaComposition. MediaClip represents a single media file, while MediaComposition is a group of MediaClip objects. It allows you manage all the MediaClip objects in composition.

Composing a video with multiple clips is a complex task, it is better to have a preview of it, isn’t it? MediaComposition provides a preview of itself. So to preview the video on which you have applied effects, you have to first add MediaClip object to MediaComposition and then generate preview stream. Set that stream as source to MediaElement. You can also set height & width of preview.

// Create object of MediaComposition class
var objMediaComposition = new MediaComposition();

// Add MediaClip to composition after applying effects
objMediaComposition.Clips.Add(objMediaClip);

// Generate preview stream & set as source to MediaElement
MediaElement.SetSource(objMediaComposition. GeneratePreviewMediaStreamSource(360, 240));

Now to save MediaClip, you have to call its asynchronous method RenderToFileAsync(IStorageItem file).

// Create file using save picker or in local/temporary folder
var file = await ApplicationData.Current.LocalFolder.CreateFileAsync("MyVideo.mp4");

// Render MediaComposition to file.
await objMediaComposition.RenderToFileAsync(file);

That’s it, your dream of media editing on Windows Phone is accomplished. Now you can edit media files from your WP apps.