EdgeCam Shots - Saving Silverlight 4 Webcam Snapshots to JPEG

In my last blog post I have covered the new Silverlight 4 Webcam API and provided a demo that used my edge detection pixel shader to create a nice real time webcam effect. In this post I make an extended version available which can save webcam snapshots as JPEG files and I also discuss some limitations of the webcam API's built-in CaptureSource.AsyncCaptureImage snapshot method. Furthermore I will give some ideas on how to build a Silverlight 4 video chat / conference application on top of the provided JPEG capturing and encoding code.


Live
To view the application you need to install the Silverlight 4 runtime. It's available for Windows and Mac.

Open the sample in a new page.

The Webcam capturing could be started and stopped with the "Start Capture" Button. If you press it for the first time you need to give your permission for the capturing. This application uses the default Silverlight capture device. You can specify the video and audio devices that are used by default with the Silverlight Configuration. Just press the right mouse button over the application, click "Silverlight" in the context menu and select the new "Webcam / Mic" tab to set them.
Press the "Save Snapshot" Button to take a snapshot and save it to a JPEG file on your harddisk.
The threshold of the edge detection can be changed using the Slider and the "Bypass" Checkbox allows you to disable the shader.

How it works
The base Silverlight 4 webcam usage code was covered in my last blog post.

The new eventhandler for the "Save Snapshot" button:

private void BtnSnapshot_Click(object sender, RoutedEventArgs e)
{
   if (saveFileDlg.ShowDialog().Value)
   {
      using (Stream dstStream = saveFileDlg.OpenFile())
      {
         SaveSnapshot(dstStream);
      }
   }
}

The code is pretty obvious: A SaveFileDialog is shown and if the user enters a file name and hits OK, a stream to the file will be opened and passed to the SaveSnapshot method. There's only one think to keep in mind when using the SaveFileDialog.ShowDialog() method, it can only be called from user-initiated code like an event handler, otherwise a SecurityException is thrown.

The SaveSnapshot method including comments:

// Render Rectangle manually into WriteableBitmap
WriteableBitmap bmp = new WriteableBitmap(ViewportHost, null);

// Init buffer in FluxJpeg format
int w = bmp.PixelWidth;
int h = bmp.PixelHeight;
int[] p = bmp.Pixels;
byte[][,] pixelsForJpeg = new byte[3][,]; // RGB colors
pixelsForJpeg[0] = new byte[w, h];
pixelsForJpeg[1] = new byte[w, h];
pixelsForJpeg[2] = new byte[w, h];

// Copy WriteableBitmap data into buffer for FluxJpeg
int i = 0;
for (int y = 0; y < h; y++)
{
   for (int x = 0; x < w; x++)
   {
      int color = p[i++];
      pixelsForJpeg[0][x, y] = (byte)(color >> 16); // R
      pixelsForJpeg[1][x, y] = (byte)(color >> 8);  // G
      pixelsForJpeg[2][x, y] = (byte)(color);       // B
   }
}

//Encode Image as JPEG
var colModel = new ColorModel { colorspace = ColorSpace.RGB };
var jpegImage = new FluxJpeg.Core.Image(colModel, pixelsForJpeg);
var encoder = new JpegEncoder(jpegImage, 95, dstStream);
encoder.Encode();

The Rectangle's surrounding Grid "ViewportHost" is rendered into a WriteableBitmap and the WriteableBitmap's Pixels are copied into another buffer with a different format. The rendered image is then written as a JPEG encoded stream using the open source FJCore library which is distributed under the MIT License. I've found some code at Stackoverflow on how to use the library in combination with the WriteableBitmap, but I modified / shortened it.
See my post on how to Convert, Encode And Decode Silverlight WriteableBitmap Data if you are interested in more WriteableBitmap conversion and JPEG encoding code.

Why not CaptureSource.AsyncCaptureImage?
You might be wondering why I haven't used the Silverlight 4 webcam API's built-in AsyncCaptureImage snapshot method of the CaptureSource class.

1. The AsyncCaptureImage method grabs a frame directly from the device and therefore any effects like the edge detection pixel shader won't be visible in the rendered image.
2. It only works if the CaptureSource was started, so the user can't save the visible image when the capturing was stopped. See the MSDN for details.

Please keep in mind that the first beta version of Silverlight 4 was used for this blog post and that some things will be changed in subsequent releases.

Where to go from here
The code in the SaveSnapshot method could be optimized and also multithreaded. The rendering should run in it's own thread. The encoding in a separate thread too and the network transport also. This approach would utilize modern quad core CPUs. After that it might be a good starting point for a video chat / conferencing application that continuously renders JPEGs and transports them between Silverlight clients. This technique is similar to the M-JPEG video format that also uses separately compressed JPEGs.
To make this idea usable for a Silverlight video chat, only the rendering, the JPEG encoder and the transfer method  need to be fast enough for real time streaming.
Please leave a comment what you think about it.

Source code
Download the Visual Studio 2010 solution here.

Update 03-20-2010
Updated to the Silverlight 4 release candidate.

Update 04-15-2010
Updated to the final Silverlight 4 RTW build.