Thursday, June 3, 2010

Push and Pull - Silverlight Webcam Capturing Details

Photo (CC) by  MShades
It's not a secret that one of my favorite Silverlight 4 features is the webcam support and I already played endless hours with it. There are many blog posts out there demonstrating how to use the webcam and how to take a screenshot with the CaptureImageAsync method. Only a few cover the VideoSink.
This blog post will show how to use the webcam, the CaptureImageAsync method and also how to implement and use the VideoSink. But most important I'll cover what the differences between the CaptureImageAsync and VideoSink are and when to use which.



Silverlight Webcam 101
The Silverlight 4 webcam API is pretty easy to use and just a few lines of code are needed to show a webcam video stream on screen.
Silverlight's CaptureSource class provides the webcam stream that is used as the source of a VideoBrush, which in turn fills a Rectangle with the video feed from the webcam. It's also possible to use any other Shape with a Fill property.
The CaptureDeviceConfiguration class can be used to retrieve a list of all installed video and audio devices on the system. Most of the time it's sufficient to use the GetDefaultVideoCaptureDevice to get the default device. The user can specify the default video and audio devices with the Silverlight configuration; he or she only has to press the right mouse button over the Silverlight application, click "Silverlight" in the context menu and select the "Webcam / Mic" tab to set them.

Silverlight Webcam / Mic configuration dialog.

The following C# code initializes the webcam (captureSource) in the Loaded event of the page and fills a rectangle (Viewport) with a VideoBrush:
// Member variable (webcam reference)
CaptureSource captureSource;

private void UserControl_Loaded(object sender, RoutedEventArgs e)
{
   // Initialize the webcam
   captureSource = new CaptureSource();
   captureSource.VideoCaptureDevice = CaptureDeviceConfiguration.GetDefaultVideoCaptureDevice();

   // Fill the Viewport Rectangle with the VideoBrush
   var vidBrush = new VideoBrush();
   vidBrush.SetSource(captureSource);
   Viewport.Fill = vidBrush;
}

Now that the webcam is initialized, the streaming can be started. This is done in an event handler of a Button because RequestDeviceAccess has to be called from an user initiated event. Otherwise it would be possible to start the webcam without the user's permission. Of course nobody wants to experience something like what happened to the students with their MacBooks provided by their high school.

Here's the C# code:
private void StartButton_Click(object sender, RoutedEventArgs e)
{
   // Request webcam access and start the capturing
   if (CaptureDeviceConfiguration.RequestDeviceAccess())
   {
      captureSource.Start();
   }
} 

This will open a webcam permission dialog asking the user for the device access. This setting is consent and Silverlight can remember if the user has previously allowed that certain application.

Webcam permission dialog

Capturing The Webcam 
There are two different approaches to capture the webcam in Silverlight. The CaptureSource's CaptureImageAsync method and CaptureImageCompleted event provide a snapshot on demand and can be considered as a pull-based technology. A custom VideoSink implementation on the other hand constantly gets the raw stream from the webcam and can be considered as a push-based approach.

Pull: CaptureImageAsync Webcam Capture
When the CaptureSource's CaptureImageAsync method is called an asynchronous capturing task is started. After the snapshot is completed, the CaptureImageCompleted event is fired. The event provides a WriteableBitmap as EventArgs.

The following C# code should be added after the captureSource initialization code above:
// Wiring the CaptureImageCompleted event handler
captureSource.CaptureImageCompleted += (s, e) =>
{
   // Do something with the camera snapshot
   // e.Result is a WriteableBitmap
   Process(e.Result);
};

Another button is used to start the asynchronous capturing:
private void SnapshotButton_Click(object sender, RoutedEventArgs e)
{
   // CaptureImageAsync fires the CaptureImageCompleted event
   captureSource.ImageCaptureAsync();
} 

Push: VideoSink Webcam Capture
The other capturing approach constantly pushes every frame from the webcam into a VideoSink. The abstract VideoSink class has four methods that have to be implemented in an own subclass in order to use it.

The basic set-up of a custom VideoSink looks like this:
// MyVideoSink is derived from Silverlight's VideoSink
public class MyVideoSink : VideoSink
{
   VideoFormat vidFormat;
   
   // Could be used to initialize a container for the webcam stream data
   protected override void OnCaptureStarted() { }
   
   // Could be used to dispose a container for the webcam stream data
   // or to write a header of a video file format
   protected override void OnCaptureStopped() { }

   // Is called when the VideoFormat was changed
   protected override void OnFormatChange(VideoFormat videoFormat)
   {
      this.vidFormat = videoFormat;
   }

   // Is called every time the webcam provides a complete frame (Push)
   protected override void OnSample(long sampleTime, long frameDuration, 
                                    byte[] sampleData)
   {
      // Process the webcam snapshot 
      // sampleData contains the raw byte stream
      // according to the videoFormat from OnFormatChange
      Process(sampleData, this.vidFormat);
   }
}

The following C# code initializes MyVideoSink with the webcam. It should be added after the captureSource initialization code above:
// Wire the VideoSink and the webcam together
var sink = new MyVideoSink { CaptureSource = captureSource };

The VideoSink's OnCaptureStarted and OnFormatChange are raised after the captureSource.Start() method was called. The OnSample method is constantly called as long as the webcam is activated. The actual interval OnSample will be called is defined in VideoFormat.FramesPerSecond which is provided through the OnFormatChange method. The OnCaptureStopped is raised after the captureSource.Stop() method was called.

Push vs. Pull
Obviously the two approaches have different characteristics.

Pull: CaptureImageAsync

Push: Custom VideoSink
  • Direct usage of the raw byte stream
  • Less overhead
  • OnSample is called on a background thread
  • Automatically called for every frame
  • Might support more than one PixelFormat in the future
  • More information like frame number and duration (accurate sample times)
  • Slightly faster than CaptureImageAsync if every frame is needed
  • Push: Constant sampling

Please note that the CaptureImageAsync method can also be called periodically. Thereby it's possible to get a snapshot in a defined interval which might be faster than using a VideoSink that fires every 30 or even 60 frames per second (fps).

The following C# code calls CaptureImageAsync every 100 milliseconds, which means every 10 fps a snapshot is taken:
var dispatcherTimer = new DispatcherTimer();
dispatcherTimer.Interval = new TimeSpan(0, 0, 0, 0, 100); // 10 fps
dispatcherTimer.Tick += (s, e) =>
{
   // Process camera snapshot if started
   if (captureSource.State == CaptureState.Started)
   {
      // CaptureImageAsync fires the CaptureImageCompleted event
      captureSource.CaptureImageAsync();
   }
};
// Start the timer
dispatcherTimer.Start();

Conclusion
Both approaches are helpful for different scenarios. The pull-based CaptureImageAsync method is useful for taking single snapshots, whereas a push-based custom VideoSink can be used for capturing complete sequences and encoding the webcam stream.

11 comments:

  1. Great post as always. Do you have any recommendations for encoding webcam/mic, I think I read in a previous blog post or possibly in some forum where you mentioned issues with audio/video timing of the streams? I think you mentioned an mjpeg like approach. It was a while ago. Any luck with that?

    I'm working out of the browser with full trust, hoping to find a good way to encode the audio/video stream on the same box. Once encoded send it up the wire to a database for review. So far Microsoft Expressions Encoder seems like the most promising solution, but I was hoping to find something free that might wrap something like ffmpeg. Even if I have to drop the raw byte streams and encode it, that might not be to bad.

    I figured if any of the mvps(congrats on that btw) might have played around with something similar, it would be you.

    ReplyDelete
  2. Hi Lee,

    Thanks!
    I need a 48 hr day to do all the things I'd like to.

    My friend Jeremiah Morrill who's a Client App Dev MVP :) made a nice open source SL4 OOB encoder app. This should be perfect for you.
    http://silverlightencoder.codeplex.com

    ReplyDelete
  3. Awesome post, only my event handler is not firing correctly.

    I have it set up as follows:

    public MainPage()
    {
    src.CaptureImageCompleted += new EventHandler(src_CaptureImageCompleted);
    }

    Then when I click the "Take Picture" button Im calling the method:

    source.CaptureImageAsync();

    the eventhandler does not fire. But when I click back on the "Enable Camera" button, BAM! It fires! WTHECK kinda crazy thing am I doing wrong here?!

    ReplyDelete
  4. Well, I THINK I figured it out so Im going to post this in case it will help anyone else out there.

    I was calling

    source.CaptureImageAsync();
    source.Stop();

    I've now put the

    source.Stop();

    in the event handler and it's working like a charm! So thanks for the help and hope this saves someone all the headaches I've had.

    Jim in FL

    ReplyDelete
  5. Hi Jim,

    Yep, the CaptureImageAsync starts an asynchronous capturing process. So the capturing isn't finished after the call of the method. When you call Stop right after the begin of the capturing, it's cancelled.

    ReplyDelete
  6. Ok, new dilemma. Need to convert to jpeg and most likely send the byte[] back to the server. From everything I read you accomplish this via a webservice. But before I call the webservice, I need to convert the WriteableBitMap to a Jpeg. How do I include / import the fluxJpeg library??

    ReplyDelete
  7. Nevermind Rene', I figured it out :)

    ReplyDelete
  8. Hi Rene,

    I'm currently taking 10 fps from my webcam application and I'm using FJcore for encoding. I'm not sure how to put together the encoded frames to mjpeg video. I tried to sequentially add the bytes from the encoded frames to a stream to save it on disk, to no avail. Any advise on how to achieve this? Thanks.

    Carl

    ReplyDelete
  9. Silverlight Webcam is perfect because I've heard it ha a perfect resolution and it can catch the most small details, it would be perfect to catch my girlfriend betraying me with the other man.

    ReplyDelete
  10. hello, i think that this post is very goodl, i would like to read more about this topic.

    ReplyDelete