Introduction
A while back, one of my web buddies sent me a link to an excellent image and video add-in for a web browser. It is called PicLens. And someone also left a note in the forum of one of my other articles saying how similar it was. At the time, I had not seen PicLens, so didn't have a clue what the comment meant. But when I did install PicLens, I was blown away.
PicLens basically looks like this (and remember, this is in a browser, very neat).
I was actually pretty envious of PicLens, so I decided to have a go at figuring out how to do something similar in WPF. This article is the result of that. I have decided to call this article codename "MarsaX" for the following reason. My name is Sacha, and one of my WPF mates is called Marlon Grech, who helped me with a few Templates for controls and some general coding on this article, and is always (probably daily, if I am honest about it) answering queries I have here and there. Basically, Marlon liked what I was up to, so I included him; he is a very, very smart kid; Google him, have a look at what he does, he has a very good blog at http://marlongrech.wordpress.com/ - have a read. He is a true master of WPF I would say.
So that's the Marsa part (Marlon and Sacha). As it's an Explorer type app, I appended an "X" on there, so it's MarsaX. Which I am sure you'll all agree sounds a lot better than Salon, which it could have been using the first letters from my name and last letters from Marlons.
Table of Contents
The rest of this article will cover the following areas:
I guess the best way is to just crack on. One note though before we start, I will be using C# and Visual Studio 2008.
The general idea behind PicLens (which is what I am trying to imitate) is to offer the user the ability to search various online repositories for images/videos and then present them in a nice 3D plane. This is the basic idea. I feel that I have achieved all of these requirements within MarsaX, and it looks pretty cool to boot. Perhaps some screenshots would help show what the attached MarsaX code looks like when run.
Allow User to Specify Search
This is available by using the +/- buttons in the top right hand side. When clicked on, these buttons will show/hide the SearchArea element.
Search Area
When clicked on, these buttons conduct their search. If a query is not in progress, the user's new query will be used. The current search is given a white glow, whilst the item under the mouse gets a blue glow.
Search Results Being Loaded
When a search is started, a loading section is shown (remember these images are being read/downloaded from the web). This is shown below:
Results Shown
After the loading has completed (a load timer has timed out), the images are shown to the user, along with some controls at the bottom of the page to scroll through the images in 3D space, and also to zoom in/out. Shown below is what MarsaX looks like after it first runs:
Then when the user uses the scroll area control, the 3D objects will be panned left/right:
And they will rotate on the mouse over event (should you have this option turned on in the App.Config file):
User Controls
You can see from here the user is able to run their mouse over an individual image within the 3D space, causing it to rotate in the 3D space. The user is also able to move the images in the 3D space using the pan control at the bottom, and also affect the zoom using the zoom control. It should also be noted that clicking on one of the 3D hosted images will show it in a new popup window.
I have to say I am pretty happy about how this turned out, as it's only really taken about 7 hours of actual work I think. So yeah, I am pretty happy about it. I'm done showing screenshots, so what I want to do now is explain how MarsaX works internally.
Now that you have seen the screenshots, you probably want to know how it works. In order to explain this, I think it is probably best to break it down into several areas, namely:
- SearchArea
- The Actual Search
- Organising the Search Results
- User Controls
This should cover most of it, so let us crack on, shall we?
SearchArea
The search area is probably one of the easiest parts as it's really just a bunch of Templated Button controls. What happens is when the user clicks on the +./- buttons at the right hand side:
the search area element within the ucslideImages3DViewPort class is animated in or out of the screen (depending on its current state). This is easily achieved by using the following code:
private void btnPlus_Click(object sender, RoutedEventArgs e)
{
if (!IsSearchAreaShown)
{
IsSearchAreaShown = true;
Storyboard HideSearchArea =
this.TryFindResource("OnShowSearchArea") as Storyboard;
if (HideSearchArea != null)
HideSearchArea.Begin(SearchArea);
}
}
private void btnMinus_Click(object sender, RoutedEventArgs e)
{
if (IsSearchAreaShown)
{
HideSearchArea();
}
}
private void HideSearchArea()
{
IsSearchAreaShown = false;
Storyboard HideSearchArea =
this.TryFindResource("OnHideSearchArea") as Storyboard;
if (HideSearchArea != null)
HideSearchArea.Begin(SearchArea);
}
And the StoryBoards themselves are declared within the XAML, as follows:
<!---->
<Storyboard x:Key="OnShowSearchArea">
<ThicknessAnimationUsingKeyFrames BeginTime="00:00:00"
Storyboard.TargetName="SearchArea"
Storyboard.TargetProperty="(FrameworkElement.Margin)">
<SplineThicknessKeyFrame KeyTime="00:00:00" Value="0,9,-500,-9"/>
<SplineThicknessKeyFrame KeyTime="00:00:00.5000000" Value="0,9,-10,-9"/>
<SplineThicknessKeyFrame KeyTime="00:00:01" Value="0,9,0,-9"/>
</ThicknessAnimationUsingKeyFrames>
</Storyboard>
<!---->
<Storyboard x:Key="OnHideSearchArea">
<ThicknessAnimationUsingKeyFrames BeginTime="00:00:00"
Storyboard.TargetName="SearchArea"
Storyboard.TargetProperty="(FrameworkElement.Margin)">
<SplineThicknessKeyFrame KeyTime="00:00:00" Value="0,9,0,-9"/>
<SplineThicknessKeyFrame KeyTime="00:00:00.5000000" Value="0,9,-490,-9"/>
<SplineThicknessKeyFrame KeyTime="00:00:01" Value="0,9,-500,-9"/>
</ThicknessAnimationUsingKeyFrames>
</Storyboard>
All what's really happening is that the SearchArea element is starting off screen, by putting in a negative Left Margin setting, and the OnShowSearchArea StoryBoard simply alters the Margin over some time, such that the SearchArea element is brought into view. Hiding the SearchArea is the opposite process.
The Actual Search
As can be seen in the SearchArea element within the ucslideImages3DViewPort class, there are three buttons which all carry out a different search. Depending on which search button was clicked, a new search will be created. This results in a call to the ConductSearch() method.
private void ConductSearch(string keyword, SearchTypes newSearchType)
{
currentSearchtype = newSearchType;
loadTimer.IsEnabled = true;
controlsArea.Visibility = Visibility.Collapsed;
ucLoader.Visibility = Visibility.Visible;
IsAnimating = true;
IsVisible = false;
ThreadPool.QueueUserWorkItem(x =>
{
var data = GetQueryResults(keyword);
Dispatcher.BeginInvoke(DispatcherPriority.Normal,
((Action)delegate { CreateModelsForImages(data); }));
});
}
private List<PhotoInfo> GetQueryResults(string keyword)
{
switch (currentSearchtype)
{
case SearchTypes.FlickrLatest:
return FlickerProvider.LoadLatestPictures();
case SearchTypes.FlickrInteresting:
return FlickerProvider.LoadInterestingPictures();
case SearchTypes.FlickrKey:
return FlickerProvider.LoadPicturesKey(keyword);
default:
return FlickerProvider.LoadLatestPictures();
}
}
What is actually happening here is that a new WaitCallBack is added to the ThreadPool so that the search is conducted in the background. Whilst the search is being performed, the Loading screen is shown to show that the worker is busy fetching some images.
Important note: We had originally used a Glow BitmapEffect on this user control, but this one BitmapEffect was enough to change how much CPU time was used from around 7% to 50-60%... scary. There is a new Effects API within the .NET 3.5 SP1, but it doesn't include a Glow effect. Which is rather sad I think, considering how nice they look. The problem here is that the BitmapEffects are not hardware accelerated.
The UI thread is still responsive at this point, but there is little else the user can do but wait for the images. Nevertheless, I consider it good practice to fetch the images in the background, leaving the UI responsive.
So where exactly do these images come from? Well, they come from an RSS feed available at Flickr. There is a single class that exposes some static search methods for obtaining feed results. This class is called FlickerProvider. To get a good understanding about how these feeds and results work, you could examine the following Flickr API docs:
Anyway, the upshot of all this is that we have a class with some static methods on it that allow us to carry out Flickr searches and return some results. Let's see one of these search methods:
public static List<PhotoInfo> LoadLatestPictures()
{
try
{
var xraw = XElement.Load(MOST_RECENT);
var xroot = XElement.Parse(xraw.ToString());
var photos = (from photo in xroot.Element("photos").
Elements("photo")
select new PhotoInfo
{
ImageUrl =
string.Format("http://farm{0}.static.flickr.com/{1}/{2}_{3}_m.jpg",
(string)photo.Attribute("farm"),
(string)photo.Attribute("server"),
(string)photo.Attribute("id"),
(string)photo.Attribute("secret"))
}).Take(Constants.ROWS * Constants.COLUMNS);
return photos.ToList<PhotoInfo>();
}
catch (Exception e)
{
Trace.WriteLine(e.Message, "ERROR");
}
return null;
}
It can be seen that we are using some XLINQ to query the RSS feed and select a List<PhotoInfo> as the result. A PhotoInfo class is a very simply data class which looks like the following:
public class PhotoInfo
{
#region Data
public string ImageUrl { get; set; }
#endregion
}
Where each PhotoInfo will be used within the ucslideImages3DViewPort class to add a new model per PhotoInfo to a ViewPort3D.
Organising the Search Results
We are now fairly happy with what we have, we have a search area with some search buttons that when clicked will load some image URLs via querying some RSS feeds. That's all well and good, but we need to do something with the returned List<PhotoInfo> to get it into the 3D world. So that is what I will explain now.
Going back a step first, recall that, part of the ConductSearch() used a ThreadPool to carry out some work. Let's just remind ourselves of that:
ThreadPool.QueueUserWorkItem(x =>
{
var data = GetQueryResults(keyword);
Dispatcher.BeginInvoke(DispatcherPriority.Normal,
((Action)delegate { CreateModelsForImages(data); }));
});
It can be seen that this in turn calls the CreateModelsForImages() method. This is where the List<PhotoInfo> is used within a 3D environment. Before I show you the code, I just want to talk about it in plain old English; it works like this:
- For each
PhotoInfo.ImageUrl within List<PhotoInfo>, create a new ModelUIElement3D (which is a new .NET 3.5 3D element that is a full blown element with events and everything; you can read more about that at my other 3D article).
- For each
ModelUIElement3D, use PhotoInfo.ImageUrl to create an Image which is then used as a VisualBrush for the ModelUIElement3D Material property.
- For each
ModelUIElement3D created, ensure that the ModelUIElement3D is translated to the correct 3D positions.
- For each
ModelUIElement3D, hook up the MouseEnter and MouseDown events so that the ModelUIElement3D can:
- be animated around its axis, and
- can raise another event to the outside world with the selected image's URL.
In essence, that is all that we are trying to achieve, so I guess it's time for some code. To ensure that we create the ModelUIElement3D in a nice grid arrangement, there is a simple loop arrangement, like:
private void CreateModelsForImages(List<PhotoInfo> photos)
{
int photoNum = 0;
IsVisible = false;
container.Children.Clear();
modelToImageLookUp.Clear();
for (int rows = 0; rows < Constants.ROWS; rows++)
{
for (int col = 0; col < Constants.COLUMNS; col++)
{
container.Children.Add(CreateModel(photos[photoNum].ImageUrl, rows, col));
photoNum++;
}
}
}
This method in turn calls the CreateModel() method, passing in the image URL and a row and col which will be used to position the new ModelUIElement3D in 3D space. Let's see the CreateModel() method now.
private ModelUIElement3D CreateModel(string imageUri, int row, int col)
{
VisualBrush vBrush = GetVisualBrush(imageUri);
ModelUIElement3D model3D = new ModelUIElement3D
{
Model = new GeometryModel3D
{
Geometry = new MeshGeometry3D
{
TriangleIndices = new Int32Collection(
new int[] { 0, 1, 2, 2, 3, 0 }),
TextureCoordinates = new PointCollection(
new Point[]
{
new Point(0, 1),
new Point(1, 1),
new Point(1, 0),
new Point(0, 0)
}),
Positions = new Point3DCollection(
new Point3D[]
{
new Point3D(-0.5, -0.5, 0),
new Point3D(0.5, -0.5, 0),
new Point3D(0.5, 0.5, 0),
new Point3D(-0.5, 0.5, 0)
})
},
Material = new DiffuseMaterial
{
Brush = vBrush
},
BackMaterial = new DiffuseMaterial
{
Brush = Brushes.Black
},
Transform = CreateGroup(row, col)
}
};
model3D.MouseEnter += ModelUIElement3D_MouseEnter;
model3D.MouseDown += model3D_MouseDown;
modelToImageLookUp.Add(model3D, imageUri);
return model3D;
}
This method above is responsible for creating a new ModelUIElement3D for each image URL. It also uses row/col to position the ModelUIElement3D at the correct 3D position within the eventail grid layout. It also uses a little helper method that creates a VisualBrush for the image URL. This is shown below:
private VisualBrush GetVisualBrush(string url)
{
Border bord = new Border();
bord.Width = 15;
bord.Height = 15;
bord.CornerRadius = new CornerRadius(0);
bord.BorderThickness = new Thickness(0.5);
bord.BorderBrush = Brushes.WhiteSmoke;
try
{
Image img = new Image
{
Source = new BitmapImage(new Uri(@url, UriKind.RelativeOrAbsolute)),
Stretch = Stretch.Fill,
Margin = new Thickness(0)
};
bord.Child = img;
}
catch (Exception e)
{
Trace.WriteLine(e.Message, "ERROR");
}
VisualBrush vBrush = new VisualBrush(bord);
return vBrush;
}
Once all the ModelUIElement3Ds have been created, they are added as children to the single ContainerUIElement3D (which is a container for other 3D WPF elements) which is within the ViewPort3D within the ucslideImages3DViewPort class.
User Controls
Once all the ModelUIElement3Ds have been created, the loading screen is hidden, and the user will be shown the images in 3D space, along with some user controls, such as pan/zoom.
The user is also able to mouse over an image to rotate it in 3D space, or click it to show its associated image in a new popup window. Let's look at the Pan and Zoom now.
Pan
There is a file that holds certain constants that are used within the application; this is called Constants.cs. The user is able to alter these values within reason. And what happens for the pan is that the COLUMNSTOSHOW value is used to constrain a standard WPF Slider control's upper limit. The Slider's ValueChanged event is then used to call the Animate() method of the ucslideImages3DViewPort class. This is shown below:
private void slideImages_ValueChanged(object sender,
RoutedPropertyChangedEventArgs<double> e)
{
Animate((int)Math.Round(slideImages.Value));
}
And here is the Animate() method. The Animate() method is used to move all the ModelUIElement3Ds to the required position. Though it doesn't move them individually, what it does do is simply move the single ContainerUIElement3D which in turn moves its children with it. This method also applies an angle change while its moving, just because it looks better.
private void Animate(int col)
{
double move = col * -MODEL_OFFSET;
Storyboard storyboard = new Storyboard();
ParallelTimeline timeline = new ParallelTimeline();
timeline.BeginTime = TimeSpan.FromSeconds(0);
timeline.Duration = TimeSpan.FromSeconds(2);
DoubleAnimation daMove = new DoubleAnimation(move,
new Duration(TimeSpan.FromSeconds(2)));
daMove.DecelerationRatio = 1.0;
Storyboard.SetTargetName(daMove, "contTrans");
Storyboard.SetTargetProperty(daMove,
new PropertyPath(TranslateTransform3D.OffsetXProperty));
double angle = col > Constants.COLUMNS / 2 ? -15 : 15;
DoubleAnimation daAngle =
new DoubleAnimation(angle, new Duration(TimeSpan.FromSeconds(0.8)));
Storyboard.SetTargetName(daAngle, "contAngle");
Storyboard.SetTargetProperty(daAngle,
new PropertyPath(AxisAngleRotation3D.AngleProperty));
DoubleAnimation daAngle2 =
new DoubleAnimation(0, new Duration(TimeSpan.FromSeconds(1)));
daAngle2.BeginTime = daAngle.Duration.TimeSpan;
Storyboard.SetTargetName(daAngle2, "contAngle");
Storyboard.SetTargetProperty(daAngle2,
new PropertyPath(AxisAngleRotation3D.AngleProperty));
timeline.Children.Add(daMove);
timeline.Children.Add(daAngle);
timeline.Children.Add(daAngle2);
storyboard.Children.Add(timeline);
storyboard.Begin(this);
}
Like I say, the Pan is done using a standard WPF Slider control, but it has just had its default Template changed a bit to look sexier. With its default Template applied, it would look like:
But with Marlon's special Template below, it now looks like this:
<!---->
<Slider x:Name="slideImages" Minimum="0"
Maximum="{x:Static local:Constants.COLUMNSTOSHOW}"
Background="Black"
ValueChanged="slideImages_ValueChanged"
Width="200"
Height="20" Margin="10,0,0,0" >
<Slider.Template>
<ControlTemplate TargetType="Slider">
<Grid>
<Grid.Background>
<!---->
<DrawingBrush
Viewport="0,0,5,5"
ViewportUnits="Absolute"
TileMode="Tile">
<DrawingBrush.Drawing>
<DrawingGroup>
<GeometryDrawing
Brush="{StaticResource backgroundBrush}">
<GeometryDrawing.Geometry>
<RectangleGeometry Rect="0,0,100,100" />
</GeometryDrawing.Geometry>
</GeometryDrawing>
<GeometryDrawing Brush="White">
<GeometryDrawing.Geometry>
<GeometryGroup>
<RectangleGeometry Rect="50,50,50,50" />
</GeometryGroup>
</GeometryDrawing.Geometry>
</GeometryDrawing>
</DrawingGroup>
</DrawingBrush.Drawing>
</DrawingBrush>
</Grid.Background>
<Track Name="PART_Track">
<Track.Resources>
<Style TargetType="RepeatButton">
<Setter Property="Template">
<Setter.Value>
<ControlTemplate TargetType="RepeatButton">
<Border Background="{TemplateBinding Background}" />
</ControlTemplate>
</Setter.Value>
</Setter>
</Style>
</Track.Resources>
<Track.DecreaseRepeatButton>
<RepeatButton Background="Transparent"
Command="Slider.DecreaseLarge" />
</Track.DecreaseRepeatButton>
<Track.Thumb>
<Thumb Width="20"
Background="DarkGray"
Opacity="0.7" />
</Track.Thumb>
<Track.IncreaseRepeatButton>
<RepeatButton
Background="Transparent"
Command="Slider.IncreaseLarge" />
</Track.IncreaseRepeatButton>
</Track>
</Grid>
</ControlTemplate>
</Slider.Template>
</Slider>
Zoom
Is also a standard WPF Slider control, whose ValueChanged method simply moves the ViewPort3D camera's Z position in or out, which simulates a zoom:
private void slideZoom_ValueChanged(object sender,
RoutedPropertyChangedEventArgs<double> e)
{
CameraPosition = e.NewValue;
}
.....
.....
public double CameraPosition
{
set
{
Point3D newPosition = new Point3D(camera.Position.X, camera.Position.Y, value);
Point3DAnimation daZoom = new Point3DAnimation(newPosition,
new Duration(TimeSpan.FromSeconds(1)));
camera.BeginAnimation(PerspectiveCamera.PositionProperty, daZoom);
}
}
You can also zoom in/out using the mouse wheel, which is achieved using the following code:
protected override void OnMouseWheel(MouseWheelEventArgs e)
{
double value = Math.Max(0, e.Delta / 10);
value = Math.Min(e.Delta, Constants.COLUMNSTOSHOW);
slideZoom.Value = value;
}
I think that concludes how it all hangs together. Shown below are a few areas where I felt it could be improved, and where PicLens is actually much better.
At the request of Josh Smith, we have also allowed users to save their pictures. Once they click on an image, the user will be presented with a popup window as shown below, from where they will be able to save the picture to a location of their choice:
You would be wrong if you thought this code was easy to do. It actually took a while to get right, and is mainly contained in a class called ImageHelper, should you wish to look at it.
Working with images in WPF is not as easy as working with GDI+.
Due to some requests to be able to not have the rotating on the individual models, we have now added the ability to either use rotation or not. This and other user settings can be set up in the App.Config file where the following key/value pairs may be used to configure MarsaX.
<configuration>
<appSettings>
-->
<add key="rows" value="2"/>
<add key="columns" value="25"/>
-->
<add key="should3DModelFlipOnMouseOver" value="false"/>
-->
<add key="savedImageLocation" value="c:\"/>
-->
<add key="stretchImagesFor3DModels" value="true"/>
</appSettings>
</configuration>
Video Support
PicLens also supports videos for the items within the 3D plane. Whilst obtaining an image for a video is fairly easy to extract using the RenderTargetBitmap method, it must be noted that all the videos are web based. Now, the MediaElement in WPF is not really intended for live streaming as far as I know. So that's one down fall. The next issue is that the MediaElement in WPF is really only meant to support whatever media formats Windows Media Player (WMP) supports, and the end user has to have WMP installed. The last area that I didn't like was the fact that to display a video in 3D space, I would have had to swap to use Visual2DViewport3D elements, which allow the hosting of 2D Visuals such as a MediaElement in a Viewport3D.
Which is all cool, but this type of element doesn't have Mouse events, so one would also need to do hit testing directly on the Viewport3D for these types of elements. For images, you can use a ModelUIElement3D element, which is a full blown element with mouse events and all. This is what MarsaX uses. I think MSFT missed a trick here, and should have made a combined element that had its own events but could also host 2D Visuals. That's just my opinion though.
Virtualization
One of the very cool things about PicLens is that it performs some sort of Virtualization, where it only loads what is visible on the screen. The MarsaX implementation doesn't have any Virtualization. Rather, what it does do is load up a set amount of images from reading the RSS feed mentioned above, so it's not perfect....I know, I know.
Did You Like It
Could I just ask, if you liked this article, could you please vote for it, and perhaps leave a comment?
History
- v1.2: 11/06/08 - Added more App.Config options and ability to save images.
- v1.1: 10/06/08 - Added search textbox enter key down to initiate search. And ability to configure options via App.Config.
- v1.0: 10/06/08 - Initial posting.
