Monday, July 19, 2010

Photos, Photos, Photos - How To Save, Load And Iterate Pictures With Windows Phone 7

The Windows Phone developer tools beta version was recently released and brought some important changes and also new features. For example the emulator is a lot better now and tombstoning was introduced. Tombstoning is an important concept to know when a Launcher or Chooser is being used in an app.
But this blog post is not about tombstoning, this post will show how to load an image from the picture library with the PhotoChooserTask. And most important how to save a picture to the library, which is not as obvious as the usage of the Task API. The last part will show how to get all images from the picture library without using the PhotoChooserTask.

Sample
I've built a Windows Phone 7 app that uses the WriteableBitmapEx library and lets the user draw on the WriteableBitmap surface. The app supports multi touch for drawing and the radius of the pen can be changed with the Slider control. An image from the phone's picture library can be chosen with the folder icon button and the floppy disk button saves the current image to the picture library. When the third button (download icon) is clicked, all pictures from the phone's library are loaded without further user interaction. The trash button clears the draw surface.
The video below demonstrates the features and shows how to use the app.
It would be great if anyone could try this sample app on a real device and give me some feedback. I'm especially interested how the multi touch drawing works.


Background music is Soft Shapes by Planet Boelex.

How it works
The raw touch points are handled through the Touch.FrameReported event. For each multi touch point a circle is being drawn with the WriteableBitmapEx' FillEllipseCentered method and a color map array that provides alternating colors:
private void Draw(IList<TouchPoint> points)
{
   // Check
   if (!isManipulating)
   {
      return;
   }

   // Init some vars
   var bmp = Bitmap;
   var w = bmp.PixelWidth;
   var h = bmp.PixelHeight;
   var r = Radius;

   // Draw
   for (int i = 0; i < points.Count; i++)
   {
      var p = points[i].Position;
      if (p.X < w && p.Y < h)
      {
         bmp.FillEllipseCentered((int)p.X, (int)p.Y, r, r, 
                                 ColorMap[(i + colorBase) % ColorMap.Length]);
      }
   }

   // Show
   Present();
}


Choose a photo
The PhotoChooserTask allows the user to select an image. The code for this task is pretty straight forward with the beta of the developer tools.
A member variable of the PhotoChooserTask is instantiated in the constructor and an event handler for the Completed event is attached.
public MainPage()
{
   // ...

   // Init chooser
   photoChooserTask = new PhotoChooserTask();
   photoChooserTask.Completed += PhotoChooserTaskCompleted;
   
   // ...
}
private void PhotoChooserTaskCompleted(object sender, PhotoResult e)
{
   if (e.TaskResult == TaskResult.OK)
   {
      // Load original image and invalidate bitmap so it gets newly rendered
      var bitmapImage = new BitmapImage();
      bitmapImage.SetSource(e.ChosenPhoto);
      Viewport.Source = bitmapImage;
      bitmap = null;
   }
}

The PhotoChooserTask.Show method is called when the corresponding Application Bar button was clicked:
private void ApplicationBarIconOpenButton_Click(object sender, EventArgs e)
{
   photoChooserTask.Show();
}

The Show method launches the Windows Phone photo app. The current version of the Windows Phone operating system only allows one application to run at the same time and therefore our app gets terminated when the Chooser is started. Here's where the concept of tombstoning comes into play.
After the user selected an image or pressed the back button, the Chooser's Completed event is raised. You might have noticed the black screen in the video that appears after the choose operation. As a result of the app termination, the Visual Studio debugging session is stopped. The Windows Phone 7 emulator detected that the app was started in a debug context and the emulator now waits at the black screen for re-attaching. So just go back to Visual Studio and hit F5 (Start Debugging), then the debugger is being re-attached and the app continues. Of course it's also possible to Start Without Debugging in Visual Studio.
The Chooser's Completed event provides a PhotoResult that contains the TaskResult, the OriginalFileName and the ChosenPhoto as a Stream. If the user hasn't cancelled the operation, the stream is used as the source of a BitmapImage which is then assigned to the WriteableBitmap draw surface (Viewport). The user can now make some nice drawings on the photo.


Save a picture
After the user created his masterpiece he probably wants to save it back to the picture library / photo album. But how could this be done? There's no PhotoSaveTask available in the Silverlight SDK. Fortunately the Windows Phone's XNA MediaLibrary comes to the rescue. Only a reference to the Microsoft.Xna.Framework assembly is needed to use it in our Windows Phone Silverlight application. To make this task a bit easier I wrote some reusable extension methods for the WriteableBitmap. These are located in the file WriteableBitmapMediaLibraryExtensions.cs from the source code download. The signatures look like this:
// Saves the WriteableBitmap encoded as JPEG to the Media library.
// The quality for JPEG encoding has to be in the range 0-100, 
// where 100 is the best quality with the largest size.
void SaveToMediaLibrary(this WriteableBitmap bitmap, string name, int quality);

// Saves the WriteableBitmap encoded as JPEG to the Media library 
// using the best quality of 100.
void SaveToMediaLibrary(this WriteableBitmap bitmap, string name);

The methods use the MediaLibrary's SavePicture method internally. The SavePicture method expects a stream or a byte array as parameter that contains an image encoded in the JPEG format. For the earlier CTP versions I wrote a custom JEPG encoding functionality that used some parts of my Silverlight JPEG encoding blog post and an adapted FJCore version. Fortunately things got a bit easier with the Windows Phone Tools beta release. The Microsoft.Phone assembly now comes with two WriteableBitmap extension methods called LoadJpeg and most important SaveJpeg. The SaveJpeg method expects the targetStream, the width and height of the target, the orientation which is not used at the moment and the quality in a range from 0 to 100. The width and height parameters are useful when a scaled version of the WriteableBitmap should be saved as JPEG.

The current bitmap surface is saved as JPEG when the disk floppy button was clicked:
private void ApplicationBarIconSaveButton_Click(object sender, EventArgs e)
{
   var name = String.Format("MediaLibSample_{0:yyyy-MM-dd_hh-mm-ss-tt}.jpg",
                            DateTime.Now);
   Bitmap.SaveToMediaLibrary(name);
}


Load all pictures
The last thing I'd like to cover in this post is the Pictures property of the MediaLibrary class. This property returns a collection of Picture instances. This class provides all the necessary meta information about the picture and streams of the image and thumbnail data.
I use this in the sample code to create a mosaic of all Windows Phone media library pictures:
private void ApplicationBarIconOpenAllButton_Click(object sender, EventArgs e)
{
   var mediaLib = new MediaLibrary();
   // Pictures also includes saved pics, 
   // mediaLib.SavedPictures returns the same collection items
   var allPics = mediaLib.Pictures;

   // Combine the pics to a single WriteableBitmap mosaic
   var bmp = Bitmap;
   bmp.Clear();
   int x = 0;
   int y = 0;
   foreach (var picture in allPics)
   {
      // Load thumbnail stream to WriteableBitmap
      var wb = new WriteableBitmap(0, 0);
      wb.SetSource(picture.GetThumbnail());

      // Blit thumbnail to background bitmap
      var w = wb.PixelWidth;
      var h = wb.PixelHeight;
      bitmap.Blit(new Rect(x, y, w, h), wb, new Rect(0, 0, w, h));
      x += w;

      // Check bounds and move to next row
      if (x >= bitmap.PixelWidth)
      {
         x = 0;
         y += h;
      }
      // Bitmap filled
      if (y >= bitmap.PixelHeight)
      {
         break;
      }
   }

   Present();
}

The GetThumbnail method returns the stream of an image thumbnail sized 99 x 99 pixels. This bitmap is then combined with the surface bitmap by using the WriteableBitmapEx' Blit method.
Beside the Pictures property the MediaLibrary also has the SavedPictures property, but I encountered that the SavedPictures collection returns all pictures and not only saved pictures like the name and documentation implies.
The MediaLibrary has some more members that might become interesting in the future, esp. when combined with the MediaPlayer class.


Conclusion
This blog post covered the complete workflow of loading, manipulating and saving a picture to the Windows Phone's picture library / photo album. I also showed how to load all images from the library and mentioned some gotchas.

Source code
Download the complete Visual Studio 2010 Windows Phone solution from here.
Read more..

Monday, July 5, 2010

The Ultimate Gift Card - Two Visual Studio 2010 Ultimate MSDN Subscriptions To Give Away!

It's over, I picked two people from the comments that deserve the MSDN Subscription cards the most. Thanks!

I recently got three Visual Studio 2010 Ultimate MSDN Subscriptions cards from Microsoft. They are valid for one year, include the whole Ultimate features and can be used for development and testing purposes. Ultimate includes Visual and Expression Studio,Windows, Office, Windows Server, SQL Server, TFS, all other server products, ... and even some Azure benefits are packed in. Only the support and the MSDN magazine are excluded.
The one year Ultimate subscription is worth approximately $12,000! And the best thing, I have two cards left I can give away. What a great gift from Microsoft!

I want to give my two remaining cards to the Windows developer community. If you want one, just write a comment why you should get this great gift and make sure to include some form of contact information. Did I mention that these are full Ultimate subscriptions worth $12,000?
I will prefer people who make significant contributions in the .Net open source world. Of course, your chance is even better if these contributions are for Silverlight or Windows Phone 7.
After July 20th 2010 I'll pick two people that deserve the cards the most.

Here's a photo of the package I got. Yeah, it's real! And free, as in free beer!



Thanks a lot Microsoft!
Read more..

Monday, June 7, 2010

Interview on .Net Rocks!

About a month ago Richard Campbell asked me if I'd like to give an interview about SLARToolkit on the .Net Rocks talk show. Of course I agreed right away. .Net Rocks! is the largest .Net developer podcast out there and Carl Franklin and Richard Campbell always have great guests. It's a real honor for me to have been on show #564.
We mainly talked about the Silverlight Augmented Reality Toolkit (SLARToolkit), but also about some of my other projects and articles.
You can download the podcast in various formats from here. I hope you like it. Also make sure you subscribe to .Net Rocks!
Read more..

Thursday, June 3, 2010

Push and Pull - Silverlight Webcam Capturing Details

Photo (CC) by  MShades
It's not a secret that one of my favorite Silverlight 4 features is the webcam support and I already played endless hours with it. There are many blog posts out there demonstrating how to use the webcam and how to take a screenshot with the CaptureImageAsync method. Only a few cover the VideoSink.
This blog post will show how to use the webcam, the CaptureImageAsync method and also how to implement and use the VideoSink. But most important I'll cover what the differences between the CaptureImageAsync and VideoSink are and when to use which.



Silverlight Webcam 101
The Silverlight 4 webcam API is pretty easy to use and just a few lines of code are needed to show a webcam video stream on screen.
Silverlight's CaptureSource class provides the webcam stream that is used as the source of a VideoBrush, which in turn fills a Rectangle with the video feed from the webcam. It's also possible to use any other Shape with a Fill property.
The CaptureDeviceConfiguration class can be used to retrieve a list of all installed video and audio devices on the system. Most of the time it's sufficient to use the GetDefaultVideoCaptureDevice to get the default device. The user can specify the default video and audio devices with the Silverlight configuration; he or she only has to press the right mouse button over the Silverlight application, click "Silverlight" in the context menu and select the "Webcam / Mic" tab to set them.

Silverlight Webcam / Mic configuration dialog.

The following C# code initializes the webcam (captureSource) in the Loaded event of the page and fills a rectangle (Viewport) with a VideoBrush:
// Member variable (webcam reference)
CaptureSource captureSource;

private void UserControl_Loaded(object sender, RoutedEventArgs e)
{
   // Initialize the webcam
   captureSource = new CaptureSource();
   captureSource.VideoCaptureDevice = CaptureDeviceConfiguration.GetDefaultVideoCaptureDevice();

   // Fill the Viewport Rectangle with the VideoBrush
   var vidBrush = new VideoBrush();
   vidBrush.SetSource(captureSource);
   Viewport.Fill = vidBrush;
}

Now that the webcam is initialized, the streaming can be started. This is done in an event handler of a Button because RequestDeviceAccess has to be called from an user initiated event. Otherwise it would be possible to start the webcam without the user's permission. Of course nobody wants to experience something like what happened to the students with their MacBooks provided by their high school.

Here's the C# code:
private void StartButton_Click(object sender, RoutedEventArgs e)
{
   // Request webcam access and start the capturing
   if (CaptureDeviceConfiguration.RequestDeviceAccess())
   {
      captureSource.Start();
   }
} 

This will open a webcam permission dialog asking the user for the device access. This setting is consent and Silverlight can remember if the user has previously allowed that certain application.

Webcam permission dialog

Capturing The Webcam 
There are two different approaches to capture the webcam in Silverlight. The CaptureSource's CaptureImageAsync method and CaptureImageCompleted event provide a snapshot on demand and can be considered as a pull-based technology. A custom VideoSink implementation on the other hand constantly gets the raw stream from the webcam and can be considered as a push-based approach.

Pull: CaptureImageAsync Webcam Capture
When the CaptureSource's CaptureImageAsync method is called an asynchronous capturing task is started. After the snapshot is completed, the CaptureImageCompleted event is fired. The event provides a WriteableBitmap as EventArgs.

The following C# code should be added after the captureSource initialization code above:
// Wiring the CaptureImageCompleted event handler
captureSource.CaptureImageCompleted += (s, e) =>
{
   // Do something with the camera snapshot
   // e.Result is a WriteableBitmap
   Process(e.Result);
};

Another button is used to start the asynchronous capturing:
private void SnapshotButton_Click(object sender, RoutedEventArgs e)
{
   // CaptureImageAsync fires the CaptureImageCompleted event
   captureSource.ImageCaptureAsync();
} 

Push: VideoSink Webcam Capture
The other capturing approach constantly pushes every frame from the webcam into a VideoSink. The abstract VideoSink class has four methods that have to be implemented in an own subclass in order to use it.

The basic set-up of a custom VideoSink looks like this:
// MyVideoSink is derived from Silverlight's VideoSink
public class MyVideoSink : VideoSink
{
   VideoFormat vidFormat;
   
   // Could be used to initialize a container for the webcam stream data
   protected override void OnCaptureStarted() { }
   
   // Could be used to dispose a container for the webcam stream data
   // or to write a header of a video file format
   protected override void OnCaptureStopped() { }

   // Is called when the VideoFormat was changed
   protected override void OnFormatChange(VideoFormat videoFormat)
   {
      this.vidFormat = videoFormat;
   }

   // Is called every time the webcam provides a complete frame (Push)
   protected override void OnSample(long sampleTime, long frameDuration, 
                                    byte[] sampleData)
   {
      // Process the webcam snapshot 
      // sampleData contains the raw byte stream
      // according to the videoFormat from OnFormatChange
      Process(sampleData, this.vidFormat);
   }
}

The following C# code initializes MyVideoSink with the webcam. It should be added after the captureSource initialization code above:
// Wire the VideoSink and the webcam together
var sink = new MyVideoSink { CaptureSource = captureSource };

The VideoSink's OnCaptureStarted and OnFormatChange are raised after the captureSource.Start() method was called. The OnSample method is constantly called as long as the webcam is activated. The actual interval OnSample will be called is defined in VideoFormat.FramesPerSecond which is provided through the OnFormatChange method. The OnCaptureStopped is raised after the captureSource.Stop() method was called.

Push vs. Pull
Obviously the two approaches have different characteristics.

Pull: CaptureImageAsync

Push: Custom VideoSink
  • Direct usage of the raw byte stream
  • Less overhead
  • OnSample is called on a background thread
  • Automatically called for every frame
  • Might support more than one PixelFormat in the future
  • More information like frame number and duration (accurate sample times)
  • Slightly faster than CaptureImageAsync if every frame is needed
  • Push: Constant sampling

Please note that the CaptureImageAsync method can also be called periodically. Thereby it's possible to get a snapshot in a defined interval which might be faster than using a VideoSink that fires every 30 or even 60 frames per second (fps).

The following C# code calls CaptureImageAsync every 100 milliseconds, which means every 10 fps a snapshot is taken:
var dispatcherTimer = new DispatcherTimer();
dispatcherTimer.Interval = new TimeSpan(0, 0, 0, 0, 100); // 10 fps
dispatcherTimer.Tick += (s, e) =>
{
   // Process camera snapshot if started
   if (captureSource.State == CaptureState.Started)
   {
      // CaptureImageAsync fires the CaptureImageCompleted event
      captureSource.CaptureImageAsync();
   }
};
// Start the timer
dispatcherTimer.Start();

Conclusion
Both approaches are helpful for different scenarios. The pull-based CaptureImageAsync method is useful for taking single snapshots, whereas a push-based custom VideoSink can be used for capturing complete sequences and encoding the webcam stream.
Read more..

Tuesday, June 1, 2010

Filled To The Bursting Point - WriteableBitmapEx 0.9.5.0

We're slowly getting to the first feature complete release of the WriteableBitmapEx library. The last version (0.9.0.0) brought the parametric curves, optimizations and other features. This new version 0.9.5.0 focuses on filling routines, transformations, bug fixes and more optimizations.
The new Fill* extension methods are equivalent to the Draw* shape functions. The FillRectangle and FillEllipse methods use specialized implementations to get the best performance. FillPolygon and the other functions use a simple scanline conversion algorithm with the even-odd-rule. The implemented algorithm supports concave and convex shapes.

New features
  • Fast FillRectangle method that uses Buffer.BlockCopy.
  • Optimized FillEllipse and FillEllipseCentered functions to draw a filled ellipse / circle.
  • FillPolygon, FillTriangle and FillQuad methods that use a scanline conversion algorithm. 
  • FillCurve, FillCurveClosed to draw a filled Cardinal spline similar to the GDI+ API.
  • FillBeziers draws a filled Cubic Beziér spline.
  • Resize method with support for Bilinear and Nearest neighbor interpolation.
  • Fast Crop method to cut out a portion of the WriteableBitmap.
  • GetPixel and GetPixeli methods to get the color as Color struct at a specified x, y coordinate.
  • FromResource method to load an image from the application's resource only by passing the relative path without the need of the full Pack URI syntax. Example: "Data/flower2.png" instead of "MyAssemblyName;component/Data/flower2.png".
  • Optimized Clear(Color) method that uses Buffer.BlockCopy.
  • Renamed BlendMode "AlphaBlend" to "Alpha".
  • Fixed the clipping of the DrawEllipse methods.
  • Fixed some missing alpha pre-multiplications.
  • Other minor tweaks and bug fixes.
  • Updated the solutions to Microsoft Visual Studio 2010.
    The code listing on the project's CodePlex site was updated to demonstrate how to use the new methods.
      Live
      As usual I also wrote a new sample application that shows the Fill* methods in action. The sample starts with a real-time demo that animates the Cardinal spline's tension of the FillCurveClosed method, plus some random animated filled ellipses. The sample also contains a static page showing some of the possible filled shapes.

      Go and grab it
      The WriteableBitmapEx library is hosted at CodePlex. You can find the new binary release here and the samples in the Subversion repository.
        Read more..