WPF: Displaying Photos and Videos Properly in Portrait and Landscape orientation

Introduction

I wrote a WPF app to display media files (i.e., photos and videos) downloaded from my mobile phone to Windows. The XAML code and code behind were simple:

<Window x:Class="Show.MainWindow"
        xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
        xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
        xmlns:d="http://schemas.microsoft.com/expression/blend/2008"
        xmlns:mc="http://schemas.openxmlformats.org/markup-compatibility/2006"
        WindowState="Maximized" WindowStyle="None">
  <Grid>
    <Image x:Name="PictureViewer" Visibility="Collapsed"/>
    <MediaElement x:Name="VideoPlayer" Visibility="Collapsed"/>
  </Grid>
</Window>

The idea here is that the app uses an <Image> to display a photo and a <MediaElement> to display a video in full screen mode (WindowState="Maximized" WindowStyle="None"). The file would be read in the code behind and depending on the extension, decide if <Image> or <MediaElement> should be made visible:

public MainWindow() {
  InitializeComponent();

  var commandLineArgs = Environment.GetCommandLineArgs();
  DirectoryInfo showDirectory;
  FileInfo? startFile;
  if (commandLineArgs.Length>1) {
    startFile = new FileInfo(commandLineArgs[1]);
  } else {
    throw new NotSuportedException();
  }

  switch (startFile.Extension.ToLowerInvariant()[1..]) {
  case "mp4":
    VideoPlayer.Visibility = Visibility.Visible;
    VideoPlayer.Source = new Uri(startFile.FullName);
    VideoPlayer.Play();
    break;

  case "jpg":
  case "png":
  case "bmp":
  case "gif":
  case "tiff":
  case "ico":
  case "wdp":
    PictureViewer.Visibility = Visibility.Visible;
    var bitmap = new BitmapImage();
    bitmap.BeginInit();
    bitmap.UriSource = new Uri(startFile.FullName);
    bitmap.EndInit();
    PictureViewer.Source = bitmap;
    break;

  default:
    throw new NotSupportedException();
  }
}

Looks short and sweet. But when I tried to display photos or videos which were taken with my mobile phone in portrait mode, the orientation (i.e., landscape or portrait) was wrong:

I took a picture of this article with my phone holding it vertically (portrait), then displayed it in my app. As you can see, the photo is displayed in landscape mode, even it was taken in portrait mode.

When I double click on the file in the Windows Explorer, it gets properly shown in portrait mode:

Problem Investigation

The following File Explorer screenshot shows these dimension for a photo taken once in landscape and once in portrait mode:

Windows Explorer pretends that width and height are switched depending on the orientation.

But Bitmap says otherwise. It shows regardless of media orientation these dimensions:

bitmap.Width: 4000
bitmap.Height: 1848

As I later found out, the mobile phone camera gives always the sensor dimension as width and height of the media. So I guessed that the information if the phone is held horizontally or vertically is stored additionally in the media file. But where?

And so my troubles began. For some reason, I investigated first the problem for videos, i.e., mp4. There is the specification ISO_IEC_14496-14_2003-11-15. As with all ISO specifications, one is supposed to buy it. I feel nowadays such specifications should be available online and free of charge. I did find a file, but reading it was no help at all. These specifications are annoying to read, link to other specifications and often don't contain the information one needs, like in this case where the media orientation is written in the file.

There are libraries which can read metadata. I used ffProbe to look at the mp4 file and I got the following information for the video stream (there are also audio, subtitle and other streams):

Stream #0:0[0x1](eng): Video: h264 (High) (avc1 / 0x31637661),
yuv420p(tv, bt709, progressive), 3840x2160, 71957 kb/s, 32.24 fps,
        29.83 tbr, 90k tbn (default)
  Metadata:
    creation_time   : 2023-08-05T09:04:15.000000Z
    handler_name    : VideoHandle
    vendor_id       : [0][0][0][0]
  Side data:
    displaymatrix: rotation of -90.00 degrees

The orientation is in the last line: rotation of -90.00 degrees. To display the portrait media correctly, one needs to rotate the media by 90 degrees clockwise, which is easy to do in WPF.

There is the NuGet Library FFMPegCore to run ffProbe, but because ffProbe is an .exe file, it runs kind of slowly. It took 0.5-0.1 seconds to return the orientation of the media file. This was too slow for me, since the video then still needs to get loaded by the MediaElement. Better is a library that runs as part of my program, just reading the metadata of the media file, and there are quite a few for C#.

Unfortunately, I had already TagLibSharp installed which, after a few days, I realised could not read any media orientation information for mp4 files. I tried to find another library which supports reading Side data, but I could not find any, I could even not find a specification for what Side data is. It seems only ffProbe uses this expression.

Solution

So I just tried a bunch of metadata reading libraries and finally found that they call this information Rotation. I settled for the Nuget library MetaDataExtractor, which is at least 10 times faster than ffProbe. The video code looks like this:

var isLandscape = true;
var directories = ImageMetadataReader.ReadMetadata(startFile.FullName);
var isRotationFound = false;
foreach (var directory in directories) {
  if (directory.Name=="QuickTime Track Header") {
    foreach (var imageMetadataTag in directory.Tags) {
      if (imageMetadataTag.Name=="Rotation") {
        isRotationFound = true;
        if (imageMetadataTag.Description=="-90") {
          isLandscape = false;
        }
        break;
      }
    }
  }
  if (isRotationFound) break;
}

VideoPlayer.Source = new Uri(startFile.FullName);
VideoPlayer.LayoutTransform =
  isLandscape ? new RotateTransform { Angle = 0 } : new RotateTransform { Angle = 90 };
VideoPlayer.Play();

MetaDataExtractor calls a stream directory. mp4 is actually based on the QuickTime format. In my video files, there are three directories of type QuickTime Track Header and one of them contains the orientation information. A stream (dictionary) contains actual data like the frames of a video and also metadata describing the content of the stream. That metadata is stored in a Tag collection. A Tag has a name and a value (description). So we have to search for a Tag with the name Rotation. If the Tag is missing, the orientation is landscape, or MetaDataExtractor just doesn't understand the file format (there are many more than just mp4 or jpg).

As explained above, I used already TagLib before and it does work for jpg files:

var fileProperties = TagLib.File.Create(startFile.FullName);
var tagLibTag = (TagLib.Image.CombinedImageTag)fileProperties.Tag;
var rotation = tagLibTag.Orientation switch {
  TagLib.Image.ImageOrientation.None => Rotation.Rotate0,
  TagLib.Image.ImageOrientation.TopLeft => Rotation.Rotate0,
  //TagLib.Image.ImageOrientation.TopRight => Rotation.Rotate0, 
  //Mirror image vertically
  TagLib.Image.ImageOrientation.BottomRight => Rotation.Rotate180,
  //TagLib.Image.ImageOrientation.BottomLeft => Rotation.Rotate0, 
  //Mirror image horizontally
  //TagLib.Image.ImageOrientation.LeftTop => Rotation.Rotate90, 
  //Mirror image horizontally and rotate 90 degrees clockwise.
  TagLib.Image.ImageOrientation.RightTop => Rotation.Rotate90,
  //TagLib.Image.ImageOrientation.RightBottom => Rotation.Rotate90, 
  // Mirror image vertically and rotate 90 degrees clockwise.
  TagLib.Image.ImageOrientation.LeftBottom => Rotation.Rotate270, //
  _ => throw new NotSupportedException(),
};
var bitmap = new BitmapImage();
var stopwatch = new Stopwatch();
stopwatch.Start();
bitmap.BeginInit();
bitmap.UriSource = new Uri(startFile.FullName);
bitmap.Rotation = rotation;
bitmap.EndInit();
PictureViewer.Source = bitmap;

TagLib is a bit more comfortable than MetaDataExtractor. TagLib went to the trouble to specify for each file extension all the tag names and made properties for it. There is the property tagLibTag.Orientation and even the enumeration TagLib.Image.ImageOrientation listing all the legal values it can have. MetaDataExtractor has no idea which tag names are possible and it is up to the developer to know for which tag name he has to search. Since there is no easy readable specification of mp4 tags, this can be rather difficult to figure out. On the other hand, TagLib does not support all existing tags, like, for example, the tag Rotation in a mp4 file. I guess it would be enough to use just MetaDataExtractor for photo and video, but I thought it might be interesting for the reader to see code using both libraries.

Recommended Reading

An introduction into the MP4 file format:

• dev.to: A quick dive into mp4

If you are interested in WPF, I strongly recommend looking at some of my other WPF articles on CodeProject. The first two go into technical details like the present article, the rest are more enjoyable to read:

Other photo related articles:

• How to Add Simple Photo Processing to Your WPF Application
• How to Make WPF Behave like Windows when Dealing with Images (Solving DPI Problems)

My most useful WPF article:

• Guide to WPF DataGrid Formatting Using Bindings

The WPF article I am the proudest of:

• Definitive Guide to WPF Colors, Color Spaces, Color Pickers and Creating Your Own Colors for Mere Mortals

Indispensable testing tool for WPF controls:

• Must Read for WPF Developers: WPF Control Test Bench

WPF information sorely lacking in MS documentation:

• Deep Dive into WPF Layouting and Rendering

I also wrote some non WPF articles.

Achieved the impossible:

• Debugging Multithreaded Code in Real Time!

Most popular (3 million views, 37'000 downloads):

• POP3 Email Client with full MIME Support

The most fun:

I wrote MasterGrab 6 years ago and since then, I play it nearly every day before I start programming. It takes about 10 minutes to beat 3 robots who try to grab all 200 countries on a random map. The game finishes once one player owns all the countries. The game is fun and fresh every day because the map looks completely different each time. The robots bring some dynamics into the game, they compete against each other as much as against the human player. If you like, you can even write your own robot, the game is open source. I wrote my robot in about two weeks (the whole game took a year), but I am surprised how hard it is to beat the robots. When playing against them, one has to develop a strategy so that the robots attack each other instead of you. I will write a CodeProject article about it sooner or later, but you can already download and play it. There is good help in the application explaining how to play:

• https://github.com/PeterHuber/MasterGrab/