Click here to Skip to main content
15,867,453 members
Articles / Multimedia / Video

VMR9 Allocator Presenter in C# with Direct3D Video Rendering

Rate me:
Please Sign up or sign in to vote.
5.00/5 (10 votes)
10 Jul 2012CPOL5 min read 56.6K   3.3K   16   20
Articles describes how to make pure C# rendering video on VMR9 with custom allocator presenter over Direct3D in .NET

Image 1

Introduction

I decide to share my 10 years experience in multimedia applications development here. I apologize for not huge description as some things for me very trivial and I not willing for describe simple things which can be found over web. I mark this tutorial for intermediate. Let’s roll…

What for this tutorial

This is the standalone application for rendering video over Direct3D in .NET. Application can be used as a start point for implementation rendering your own scenes with the one of couple video sources in it.

What you should know before start

I suggest you to read about following topics over web or on MSDN.

  • DirectShow – media playback.
  • What is the VMR9 – what is it and usage.
  • Interop marshaling and PInvoke.
  • Briefly Direct3D 9 and scene presenting.
  • Multithreading.
  • COM

Application overview

Demo application shows how to perform video playback using DirectShow. It uses the VMR9 rendering filter which available on windows XP and higher systems. VMR9 allows provide your own engine for surface allocation and/or presenting that surfaces on the screen. In sample we use our own stuff for both features allocation and presenting this means that the VMR9 will be used in “Renderless” mode. Presenting surfaces we will do via Direct3D9 and enough for our application will be SlimDX (managed library for Direct9D), but you can use any other libraries or even create managed wrapper for required interfaces. Next figure display the video data flow.

Image 2

Application makes the custom allocator and provides it to the VMR9 during initialization. On playback VMR9 query for allocated surfaces and perform updating it with the video data after it marks surface ready for display and we send that surface to presenter, and it display them in the application.

My class library

Implementation does not use the DirectShow.NET library, and uses the direct marshaling of DirectShow interfaces. I have couple of classes which helps you to use DirectShow in your application, but you can use DirectShow.NET library in your application. Here I briefly describe my common classes which are used here:

COMHelper – base class with definitions of common types and common values such as HRESULTS values windows API helper macros and functions for trace.

BOOL – class representing of Boolean value.

HRESULT – helper class for COM HRESULT’s values.

FOURCC – Four Character Code implementation.

DSObject<T> - template class for COM objects – here used for filters pins and other objects in DirectShow.

DSPin – helper class for IPin interface.

DSFilter – helper class for IBaseFilter interface.

DSFilterGraphBase – base class for implementing DirectShow filter graph.

DSFilePlayback – base class for media file playback.

More information regarding each class you can find in source code reviewing.

Implementing scene presenting

As I mentioned earlier, I use SlimDX library for Direct3D rendering. So we need to add this library as reference and need to include using aliases:

C#
using SlimDX.Direct3D9; 
using SlimDX;

Initialization of the scene class with creating Direct3D device with specified control there we will do presenting incoming video and store main RenderTarget as a class variable:

C#
public Scene(Control _control) 
: this(_control, null) 
{ 
} 

public Scene(Control _control, Device _device) 
{ 
    m_Control = _control; 
    m_Device = _device; 
    if (m_Device == null) 
    { 
        Direct3DEx _d3d = new Direct3DEx(); 
        DisplayMode _mode = _d3d.GetAdapterDisplayMode(0); 
        PresentParameters _parameters = new PresentParameters(); 
        _parameters.BackBufferFormat = _mode.Format; 
        _parameters.BackBufferCount = 1; 
        _parameters.BackBufferWidth = m_Control.Width; 
        _parameters.BackBufferHeight = m_Control.Height; 
        _parameters.Multisample = MultisampleType.None;
        _parameters.SwapEffect = SwapEffect.Discard; 
        _parameters.PresentationInterval = PresentInterval.Default; 
        _parameters.Windowed = true; 
        _parameters.DeviceWindowHandle = m_Control.Handle; 
        _parameters.PresentFlags = PresentFlags.DeviceClip | PresentFlags.Video; 
        m_Device = new DeviceEx(_d3d, 0, DeviceType.Hardware, m_Control.Handle,
            CreateFlags.Multithreaded | CreateFlags.HardwareVertexProcessing, _parameters); 
    } 
    m_RenderTarget = m_Device.GetRenderTarget(0); 
}

Now it’s time for scene presenting code. Note this code simplify copy surface data into backbuffer surface as I decide not implement fully creating the scene, modern GPU allows to rendering this way, but for old GPU you should make vertex declaration and rendering texture, how to do that proper described in DirectX documentation.

C#
public void OnSurfaceReady(ref Surface _surface)
{
    lock (m_csLock)
    {
        m_Device.SetRenderTarget(0, m_RenderTarget);
        m_Device.Clear(ClearFlags.Target, Color.Blue, 1.0f, 0);
        m_Device.BeginScene();
        Surface _backbufer = m_Device.GetBackBuffer(0, 0);
        m_Device.StretchRectangle(_surface, _backbufer, TextureFilter.Linear);
        m_Device.EndScene();
        m_Device.Present();
    }
}

Code lock(m_csLock) is used to multithreaded presenting device access.That’s all for implementing scene initialization and presenting the video, hope you not close your browser so far J

Implementing VMR9 Allocator

Hardest and major part of this application is here.

Filter Graph

The filter graph we are building inside application is particular playback graph but instead of default video renderer we insert VMR9 Renderer filter. So graph will be looks:

Image 3

Events

First step is delegating the event which will be occurred once surface will be ready:

C#
public delegate void SurfaceReadyHandler(ref Surface _surface); 

And event variable in class:

C#
public event SurfaceReadyHandler OnSurfaceReady;

This handler will be connected to the scene method which described earlier.

Initialization

I not describe the DirectShow graph building entirely as there are a lot of articles here and on other resources how to do so, I’ll just my classes for it. VMR9 filter declaration:

C#
[Guid("51b4abf3-748f-4e3b-a276-c828330e926a")] 
public class VMR9Renderer : DSFilter 
{ 
    public VMR9Renderer()
        : base() 
    { 
    } 
}

Our Allocator class declaration will looks as following:

C#
public class DSFilePlaybackVMR9 : DSFilePlayback, IVMRSurfaceAllocator9, IVMRImagePresenter9

VMR9 class is derived from base graph builder class which handles all basic stuff for file playback over DirectShow we just need to modify initialization method a little:

C#
protected override HRESULT OnInitInterfaces() 
{ 
    m_Renderer = new VMR9Renderer(); 
    m_Renderer.FilterGraph = m_GraphBuilder; 
    IVMRFilterConfig9 _config = (IVMRFilterConfig9)m_Renderer.QueryInterface(typeof(IVMRFilterConfig9).GUID); 
    HRESULT hr; 
    if (_config != null) 
    { 
        hr = (HRESULT)_config.SetRenderingMode(VMR9Mode.Renderless); 
        hr.Assert(); 
        hr = (HRESULT)_config.SetNumberOfStreams(5); 
        hr.Assert(); 
    } 
    IVMRSurfaceAllocatorNotify9 _notify = (IVMRSurfaceAllocatorNotify9)m_Renderer.QueryInterface(typeof(IVMRSurfaceAllocatorNotify9).GUID); 
    if (_notify != null) 
    { 
        hr = (HRESULT)_notify.AdviseSurfaceAllocator(new IntPtr(g_ciUsedID), this); 
        hr.Assert(); 
        hr = (HRESULT)this.AdviseNotify(_notify); 
        hr.Assert(); 
    } 
    hr = base.OnInitInterfaces(); 
    return hr; 
}

Here we insert VMR9 renderer filter into FilterGraph, configure it for “Renderless” mode and specify our own allocator presenter (class derived from IVMRSurfaceAllocator9 and IVMRImagePresenter9 interfaces).

Implementing IVMRSurfaceAllocator9

This interface we should have implement for allocating surfaces for VMR9 filter. Filter calls this interface during media type negotiation for allocating surfaces and during playback for query new surface. First look at the variables in presenter class:

C#
protected Device m_Device = null; 
protected const int g_ciUsedID = 0x01020304; 
protected IVMRSurfaceAllocatorNotify9 m_lpIVMRSurfAllocNotify = null; 
protected object m_csLock = new object(); 
protected List<Surface> m_Surfaces = new List<Surface>(); 
protected Texture m_PrivateTexture = null;

That’s all that we need to handle our allocator. First method of the interface called by VMR9 filter is the AdviseNotify, which provide allocator notify object. We will use it to specify our Direct3D device handle to VMR9:

C#
public int AdviseNotify(IVMRSurfaceAllocatorNotify9 lpIVMRSurfAllocNotify) 
{ 
    lock (m_csLock) 
    { 
        m_lpIVMRSurfAllocNotify = lpIVMRSurfAllocNotify; 
        if (m_lpIVMRSurfAllocNotify != null) 
        { 
            IntPtr hMonitor = m_Device.Direct3D.GetAdapterMonitor(
            m_Device.CreationParameters.AdapterOrdinal); 
            HRESULT hr = (HRESULT)m_lpIVMRSurfAllocNotify.SetD3DDevice(m_Device.ComPointer, hMonitor); 
            hr.Assert(); 
            return hr; 
        } 
    } 
    return NOERROR; 
} 

Next call is made during filter connection for allocating surfaces is InitializeDevice. In this method we allocate surfaces and put them into the list.

C#
lpAllocInfo.dwFlags |= VMR9SurfaceAllocationFlags.TextureSurface; 
m_Surfaces.Clear(); 
IntPtr[] lplpSurfaces = new IntPtr[lpNumBuffers]; 
hr = (HRESULT)m_lpIVMRSurfAllocNotify.AllocateSurfaceHelper(ref lpAllocInfo, ref lpNumBuffers, lplpSurfaces); 
hr.Assert(); 
if (hr.Succeeded) 
{ 
    for (int i = 0; i < lplpSurfaces.Length; i++) 
    { 
        Marshal.AddRef(lplpSurfaces[i]); 
        Surface _surface = Surface.FromPointer(lplpSurfaces[i]); 
        m_Surfaces.Add(_surface); 
    } 
} 

During playback VMR9 request surfaces for update:

C#
public int GetSurface(IntPtr dwUserID, int SurfaceIndex, int SurfaceFlags, out IntPtr lplpSurface) 
{ 
    ASSERT(dwUserID.ToInt32() == g_ciUsedID); 
    lplpSurface = IntPtr.Zero; 
    if (SurfaceIndex > m_Surfaces.Count) return E_INVALIDARG; 
    lock (m_csLock) 
    { 
        lplpSurface = m_Surfaces[SurfaceIndex].ComPointer; 
        Marshal.AddRef(lplpSurface); 
    } 
    return NOERROR; 
}

In TerminateDevice we just remove all surfaces from list and release them.

Implementing IVMRImagePresenter9

The presentation part of the VMR9 should implement this interface. Major method here is PresentImage, which called by VMR9 once frame ready for display. In this method we just call the event handler described above with specifying that surface:

C#
public int PresentImage(IntPtr dwUserID, ref VMR9PresentationInfo lpPresInfo) 
{ 
    if ((object)lpPresInfo == null) 
    { 
        return E_POINTER; 
    } 
    else 
    if (lpPresInfo.lpSurf == IntPtr.Zero) 
    { 
        return E_POINTER; 
    } 
    lock (m_csLock) 
    { 
        Surface _source = Surface.FromPointer(lpPresInfo.lpSurf); 
        if (_source != null && OnSurfaceReady != null) 
        { 
            if (m_PrivateTexture != null) 
            { 
                Surface _target = m_PrivateTexture.GetSurfaceLevel(0); 
                m_Device.StretchRectangle(_source, _target, TextureFilter.None); 
                OnSurfaceReady(ref _target); 
            } 
            else 
            { 
                OnSurfaceReady(ref _source); 
            } 
        } 
    } 
    return NOERROR; 
}

Shutdown

We need to release our objects which we are created during FilterGraph initialization:

C#
protected override HRESULT OnCloseInterfaces() 
{ 
    if (m_Renderer) 
    { 
        m_Renderer.Dispose(); 
        m_Renderer = null; 
    } 
    if (m_lpIVMRSurfAllocNotify != null) 
    { 
        Marshal.ReleaseComObject(m_lpIVMRSurfAllocNotify); 
        m_lpIVMRSurfAllocNotify = null; 
    } 
    return base.OnCloseInterfaces(); 
}

Main Application

Implementation of main form is very easy, most interesting methods I describe here. Variable declaration for scene and playback:

C#
private Scene m_Scene = null;
private DSFilePlayback m_Playback = null;

Creating scene object:

C#
m_Scene = new Scene(this.pbView); 

Here pbView control on which we’ll do presenting the video. Starting playback code:

C#
m_Playback = new DSFilePlaybackVMR9(m_Scene.Direct3DDevice); 
m_Playback.OnPlaybackStop += new EventHandler(btnStart_Click); 
((DSFilePlaybackVMR9)m_Playback).OnSurfaceReady += new VMR9.SurfaceReadyHandler(m_Scene.OnSurfaceReady); 
m_Playback.FileName = this.tbFileName.Text; 
if (m_Playback.Start().Succeeded) 
{ 
    btnStart.Text = "Stop"; 
    btnBrowse.Enabled = false; 
}

In this code we create VMR9 rendering graph with specified Scene device. After we provide event handler for surface ready notify, and starting playback. Stopping code is fairly simple – just Dispose the playback:

C#
m_Playback.Dispose(); 
m_Playback = null;

Comments and notes

Any comments are welcome, hope I'll find time to post couple other interesting code.

History

10.07.2012 - initial version.

License

This article, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)


Written By
Software Developer (Senior)
Russian Federation Russian Federation
I'm a professional multimedia developer (more than 10 years) in any kind of applications and technologies related to it, such as DirectShow, Direct3D, WinMM, OpenGL, MediaFoundation, WASAPI, Windows Media and other including drivers development of Kernel Streaming, Audio/Video capture drivers and audio effects. Have experience in following languages: C, C++, C#, delphi, C++ builder, VB and VB.NET. Strong knowledge in math and networking.

Comments and Discussions

 
QuestionMultiple instance of VMR 9 in dot net application Pin
Member 1023041629-Nov-16 23:34
Member 1023041629-Nov-16 23:34 
SuggestionMissing Microsoft.VC90.DebugCRT Pin
YaniMan14-Nov-16 22:40
YaniMan14-Nov-16 22:40 
QuestionGPU Memory Leak Pin
wana10-Nov-14 4:51
wana10-Nov-14 4:51 
AnswerRe: GPU Memory Leak Pin
Maxim Kartavenkov10-Nov-14 6:21
Maxim Kartavenkov10-Nov-14 6:21 
Questionrendering h264 in a c# application and create snapshot every second Pin
Member 1073106416-May-14 8:52
Member 1073106416-May-14 8:52 
AnswerRe: rendering h264 in a c# application and create snapshot every second Pin
Maxim Kartavenkov20-May-14 20:17
Maxim Kartavenkov20-May-14 20:17 
GeneralRe: rendering h264 in a c# application and create snapshot every second Pin
h.ahmadi8629-May-14 0:22
h.ahmadi8629-May-14 0:22 
AnswerRe: rendering h264 in a c# application and create snapshot every second Pin
Maxim Kartavenkov9-Jun-14 19:33
Maxim Kartavenkov9-Jun-14 19:33 
QuestionSome problems Pin
Zach Saw6-Apr-14 20:32
Zach Saw6-Apr-14 20:32 
AnswerRe: Some problems Pin
Maxim Kartavenkov9-Apr-14 20:24
Maxim Kartavenkov9-Apr-14 20:24 
QuestionSecondary display ? Pin
Shauly24-Dec-13 8:03
Shauly24-Dec-13 8:03 
AnswerRe: Secondary display ? Pin
Maxim Kartavenkov24-Dec-13 20:28
Maxim Kartavenkov24-Dec-13 20:28 
GeneralRe: Secondary display ? Pin
Zach Saw4-Apr-14 2:42
Zach Saw4-Apr-14 2:42 
GeneralRe: Secondary display ? Pin
Maxim Kartavenkov4-Apr-14 20:12
Maxim Kartavenkov4-Apr-14 20:12 
The swap chain usage example is on my effect sample application in other article.
You can create couple swap chains and update their render targets any way you planning to do.

Maxim.
GeneralRe: Secondary display ? Pin
Zach Saw6-Apr-14 15:45
Zach Saw6-Apr-14 15:45 
GeneralRe: Secondary display ? Pin
Maxim Kartavenkov9-Apr-14 20:25
Maxim Kartavenkov9-Apr-14 20:25 
Questiontexture filtering Pin
DJ Heemstra22-Feb-13 3:49
DJ Heemstra22-Feb-13 3:49 
AnswerRe: texture filtering Pin
Maxim Kartavenkov22-Feb-13 4:06
Maxim Kartavenkov22-Feb-13 4:06 
QuestionlpNumBuffers are always 1 Pin
damikez30-Jan-13 13:48
damikez30-Jan-13 13:48 
AnswerRe: lpNumBuffers are always 1 Pin
Maxim Kartavenkov6-Feb-13 20:30
Maxim Kartavenkov6-Feb-13 20:30 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Praise Praise    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.