Monday, November 23, 2009

EdgeCam Shots - Saving Silverlight 4 Webcam Snapshots to JPEG

In my last blog post I have covered the new Silverlight 4 Webcam API and provided a demo that used my edge detection pixel shader to create a nice real time webcam effect. In this post I make an extended version available which can save webcam snapshots as JPEG files and I also discuss some limitations of the webcam API's built-in CaptureSource.AsyncCaptureImage snapshot method. Furthermore I will give some ideas on how to build a Silverlight 4 video chat / conference application on top of the provided JPEG capturing and encoding code.


Live
To view the application you need to install the Silverlight 4 runtime. It's available for Windows and Mac.



The Webcam capturing could be started and stopped with the "Start Capture" Button. If you press it for the first time you need to give your permission for the capturing. This application uses the default Silverlight capture device. You can specify the video and audio devices that are used by default with the Silverlight Configuration. Just press the right mouse button over the application, click "Silverlight" in the context menu and select the new "Webcam / Mic" tab to set them.
Press the "Save Snapshot" Button to take a snapshot and save it to a JPEG file on your harddisk.
The threshold of the edge detection can be changed using the Slider and the "Bypass" Checkbox allows you to disable the shader.

How it works
The base Silverlight 4 webcam usage code was covered in my last blog post.

The new eventhandler for the "Save Snapshot" button:
private void BtnSnapshot_Click(object sender, RoutedEventArgs e)
{
   if (saveFileDlg.ShowDialog().Value)
   {
      using (Stream dstStream = saveFileDlg.OpenFile())
      {
         SaveSnapshot(dstStream);
      }
   }
}
The code is pretty obvious: A SaveFileDialog is shown and if the user enters a file name and hits OK, a stream to the file will be opened and passed to the SaveSnapshot method. There's only one think to keep in mind when using the SaveFileDialog.ShowDialog() method, it can only be called from user-initiated code like an event handler, otherwise a SecurityException is thrown.

The SaveSnapshot method including comments:
// Render Rectangle manually into WriteableBitmap
WriteableBitmap bmp = new WriteableBitmap(ViewportHost, null);

// Init buffer in FluxJpeg format
int w = bmp.PixelWidth;
int h = bmp.PixelHeight;
int[] p = bmp.Pixels;
byte[][,] pixelsForJpeg = new byte[3][,]; // RGB colors
pixelsForJpeg[0] = new byte[w, h];
pixelsForJpeg[1] = new byte[w, h];
pixelsForJpeg[2] = new byte[w, h];

// Copy WriteableBitmap data into buffer for FluxJpeg
int i = 0;
for (int y = 0; y < h; y++)
{
   for (int x = 0; x < w; x++)
   {
      int color = p[i++];
      pixelsForJpeg[0][x, y] = (byte)(color >> 16); // R
      pixelsForJpeg[1][x, y] = (byte)(color >> 8);  // G
      pixelsForJpeg[2][x, y] = (byte)(color);       // B
   }
}

//Encode Image as JPEG
var colModel = new ColorModel { colorspace = ColorSpace.RGB };
var jpegImage = new FluxJpeg.Core.Image(colModel, pixelsForJpeg);
var encoder = new JpegEncoder(jpegImage, 95, dstStream);
encoder.Encode();
The Rectangle's surrounding Grid "ViewportHost" is rendered into a WriteableBitmap and the WriteableBitmap's Pixels are copied into another buffer with a different format. The rendered image is then written as a JPEG encoded stream using the open source FJCore library which is distributed under the MIT License. I've found some code at Stackoverflow on how to use the library in combination with the WriteableBitmap, but I modified / shortened it.
See my post on how to Convert, Encode And Decode Silverlight WriteableBitmap Data if you are interested in more WriteableBitmap conversion and JPEG encoding code.

Why not CaptureSource.AsyncCaptureImage?
You might be wondering why I haven't used the Silverlight 4 webcam API's built-in AsyncCaptureImage snapshot method of the CaptureSource class.
  1. The AsyncCaptureImage method grabs a frame directly from the device and therefore any effects like the edge detection pixel shader won't be visible in the rendered image.
  2. It only works if the CaptureSource was started, so the user can't save the visible image when the capturing was stopped. See the MSDN for details.
Please keep in mind that the first beta version of Silverlight 4 was used for this blog post and that some things will be changed in subsequent releases.

Where to go from here
The code in the SaveSnapshot method could be optimized and also multithreaded. The rendering should run in it's own thread. The encoding in a separate thread too and the network transport also. This approach would utilize modern quad core CPUs. After that it might be a good starting point for a video chat / conferencing application that continuously renders JPEGs and transports them between Silverlight clients. This technique is similar to the M-JPEG video format that also uses separately compressed JPEGs.
To make this idea usable for a Silverlight video chat, only the rendering, the JPEG encoder and the transfer method  need to be fast enough for real time streaming.
Please leave a comment what you think about it.

Source code
Download the Visual Studio 2010 solution here.

Update 03-20-2010
Updated to the Silverlight 4 release candidate.

Update 04-15-2010
Updated to the final Silverlight 4 RTW build.

42 comments:

  1. Great post - I am waiting to build a video-chat app using SL4 (with the two clients connected directly with each other for better performance)

    Will have to wait for now, I suppose!

    ReplyDelete
  2. Thanks Sriram.

    I don't have any experience with Silverlight's networking APIs and I don't know if SL4 OOB supports a peer to peer communication, but it should be possible using a server, although this is really hard work. Especially the audio and video sync is not that easy.

    ReplyDelete
  3. Hi Rene,
    Great post - A beginner question: Any idea how to convert the writeablebitmap to a byte array in order to save the snapshot on a sql server with wcf ria ? Do I need to convert it to jpeg ?
    Thanks for your help.

    ReplyDelete
  4. @marcb: Please see my blog post about it :)
    "Convert, Encode And Decode Silverlight WriteableBitmap Data"

    http://kodierer.blogspot.com/2009/11/convert-encode-and-decode-silverlight.html

    ReplyDelete
  5. there's a bug in your demo, if you change the videohost width/height to 400px/300px or any value as long as width is different to height.
    then try to save a snapshot.

    data in pixelsForJpeg not correct. :)

    ReplyDelete
  6. I cannot compile your Solution, here is the error from Visual Studio 2010:

    Error 1 The "ValidateXaml" task failed unexpectedly.
    System.IO.FileLoadException: Could not load file or assembly 'file:///C:\Users\VG\Downloads\EdgeCamShots\EdgeCamShots\3rdParty\FJCore\FJ.Core.dll' or one of its dependencies. Operation is not supported. (Exception from HRESULT: 0x80131515)
    File name: 'file:///C:\Users\VG\Downloads\EdgeCamShots\EdgeCamShots\3rdParty\FJCore\FJ.Core.dll' ---> System.NotSupportedException: An attempt was made to load an assembly from a network location which would have caused the assembly to be sandboxed in previous versions of the .NET Framework. This release of the .NET Framework does not enable CAS policy by default, so this load may be dangerous. If this load is not intended to sandbox the assembly, please enable the loadFromRemoteSources switch. See http://go.microsoft.com/fwlink/?LinkId=155569 for more information.

    ReplyDelete
  7. @Victor
    Interesting and thanks for letting me know. Any ideas?
    They only provide the source of the FJCore lib at
    http://code.google.com/p/fjcore/
    So I compiled my own version of FJCore. Although I was able to compile it on two completely different machines, you might want to try it with your own FJCore build.

    ReplyDelete
  8. This demo gives the same E_FAIL HRESULT whilst attempting to initialize the webcam when running on Snow Leopard and SL4 beta.

    ReplyDelete
  9. @Victor - The reason you get that error is usually because the downloaded file needs to be unblocked first. Go to the dll in windows explorer, right click it and unblock it. This has happened to me a few times.

    ReplyDelete
  10. The error has happened to me as well when I download a zip file.. had to do the same things as John Papa to the zip file to resolve the issue.

    ReplyDelete
  11. Thanks for the helpful information John!

    ReplyDelete
  12. @roger
    You are right, there was an obvious bug in my JPEG encoding code. I fixed it. :)

    ReplyDelete
  13. what do you think about pushing the video to a media server then have the clients connect to the media server?

    - Manit

    ReplyDelete
  14. Hi Manit,

    this could work, but the biggest problem is the latency and the sync. See the blog post by Mike on how to capture the mic / webcam stream:
    http://mtaulty.com/CommunityServer/blogs/mike_taultys_blog/archive/2009/12/30/silverlight-4-webcams-oncemorewithaudio.aspx

    And the latest discussion here:
    http://forums.silverlight.net/forums/t/145729.aspx

    ReplyDelete
  15. Thanks for the post. Can you please tell me regarding audio video synchronization in SL4, I am working on my final year project which requires such functionality. It would be helpful if you can guide me.

    regards,
    Irfan Shahzad

    ReplyDelete
  16. Hi Irfan,

    the syncing is one of the hardest parts.
    You can record the audio and video on the sending client and store them in buffers with the timestamp. See the AudioSink and VideoSink classes' OnSample method parameters. Once you have audio and video for the same timmestamp, pack them in a datastructure and send it over the network to the receiving client.
    Or send the audio and video separately over the network and implement a small ringbuffer for audio and video on the receiving client side. These buffers store the video and audio information and when the buffers have some packages that match the timestamp you can start playing. Or you can disable the video if you don't have audio and video in sync.


    Meanwhile there is also some more information available for this topic. You should read the following:

    http://forums.silverlight.net/forums/p/145729/324400.aspx
    http://forums.silverlight.net/forums/t/148040.aspx
    http://www.codeproject.com/KB/IP/SilverlightVoiceVideoChat.aspx

    And keep an eye on Jeremiah Morril. :)

    ReplyDelete
  17. This comment has been removed by the author.

    ReplyDelete
  18. Hey! it's great this tool! (as well as http://www.codeproject.com/KB/IP/SilverlightVoiceVideoChat.aspx tool)However, I've got a trouble. When I start video conference, video is not sent correctly. I mean, the receiver part is not able to see video completely (it just shows a tiny part of the whole video). How could I fix it? does it have to do with decode and encode? Thanks for your help.

    ReplyDelete
  19. I'm not the author of the CodeProject article. You should ask him using the CodeProject comments section or at the Silverlight forums.
    (See the links above)

    ReplyDelete
  20. Thank you very much!

    Is it possible to talk to a scanner - from with in browser - this way?

    ReplyDelete
  21. Nope, because you need native libraries that are not built into Silverlight. If Out-Of-Browser elevated trust mode is an option for you, you should try this approach:
    http://10rem.net/blog/2010/04/14/scanning-an-image-from-silverlight-4-using-wia-automation

    ReplyDelete
  22. Hi,
    i am working on project of video chat in silverlight , i have read your post very carefully and i have successfully implemented video and audio capturing in silverlight.
    But i am stucked at a point which is: how to send only video to the client side .
    Could you please reply me or send me a demo application or help me by giving some links of websites from which i could make my task complete.
    Please help me i am very much stucked here.

    ReplyDelete
  23. I'm afraid there's no real solution to this problem at the moment. I don't know a managed .Net video encoder which could be used with Silverlight that runs fast enough. Video encoders are computational intensive and usually implemented in C using SSE, inline assembler and multi-threading.

    Here some info about the state of the art in this field:

    Rui Marinho made a nice OOB chat application including screen capturing (without compression). Check out the video and the blog post (English at the end).

    Jeremiah published his Silverlight Webcam A/V Encoder/Broadcaster on codeplex It doesn't need a special registered COM class like in Rui's approach.

    And read this thread http://forums.silverlight.net/forums/p/145729/324400.aspx


    Please read the other comments too!

    ReplyDelete
  24. Hi René,

    Thanks for share your project its awesome, but what i should change to save the image file in a static path as "c:/images", with out ussing "Show dialog open file" coz i want the user doesnt select the target to save the image.

    Thank 4 ur help.
    Augusto.

    ReplyDelete
  25. This also applies for Save.
    http://stackoverflow.com/questions/2289104/set-an-image-source-from-code-behind/2289425#2289425

    ReplyDelete
  26. ! IMPORTANT !

    You should ask such general questions at the official Silverlight Forum or at stackoverflow.

    http://forums.silverlight.net
    http://stackoverflow.com

    ReplyDelete
  27. Thanks Rene,

    Im really new ussing silverlight couple days and i found very interesting ur project.

    Congrats!
    Augusto M.

    ReplyDelete
  28. Thanks Rene for your kind & helpful reply,
    I want to ask you one thing more..
    That is can we have voice chat instead of video chat? And if we can do that, please send me some link of website where i can find some demo application.
    Please help out me , i am really stucked

    ReplyDelete
  29. Hi,
    I have asked about video chat,But i have got reply that it is very tough task.
    But i have seen socketcoder application , so can any one please help me how does it works.
    I will be very thankfull to you, please help out me.

    ReplyDelete
  30. Hi ,
    i am developinf a SL4 webcam project and need some help i want to save snapshot of my webcam and i got the writeable bitmap but can figureout how to save my writeable bitmap in stream???

    ReplyDelete
  31. RTFBP??? what do u mean by this , i searched ur blog but nothing. plz tell me the easy way to save a writeable bitmap?

    ReplyDelete
  32. Read This Blog Post. + comments.
    And download the source code.

    ReplyDelete
  33. Hi, Rene Schulte.

    Great post.

    Is that possible to save the photo that capture from webcam to the directory that i set? for example if i click save snapshot. there will not pop up SaveDialogFile just show a message the photo have been save and the photo save in 1 folder that set.

    Thanks,
    Best Regard

    ReplyDelete
  34. Hi Halo,

    Silverlight runs in a secure sandbox and you can't simply access the local filsystem. You will have to use a SaveFileDialog to let the user select a file. Keep in mind that the OpenFileDialog has to be called from an user-initiated event, like a Button.Click.

    Alternatively, you can also make an Out-Of-Browser app with elevated trust and get access to _some_ special folders.

    ReplyDelete
  35. Hello Rene, I could test the project also on a Mac, it works very good with my webcam but I got an "Access Denied" when starting a call to whatever ip. Can you just explain a little bit more about the server side? I can imagine with iis7.5 installed and then?
    What kind of services should I started?
    And last but not least: Using a server in between means the unique way to get a video/audio chat like skype?
    Thanks in advance

    Fernando

    ReplyDelete
  36. Hi Fernando,

    I'm afraid there's no real solution to this problem at the moment. I don't know a managed .Net video encoder which could be used with Silverlight that runs fast enough. Video encoders are computational intensive and usually implemented in C using SSE, inline assembler and multi-threading.

    We actually need Microsoft to implement an encoder in the runtime to get the needed speed.
    We brought this several times to the attention of the right people.

    You should also vote here:
    http://dotnet.uservoice.com/forums/4325-silverlight-feature-suggestions/suggestions/451600-encoding-and-streaming-to-media-server-iis?ref=title
    The SL team is listening

    And this forum post:
    http://forums.silverlight.net/forums/p/145729/324400.aspx

    ReplyDelete
  37. Good DAY Sir!!
    do you have any idea
    how to save video from another video using silvelight 4.0 to a specific folder?

    thank you!

    ReplyDelete
  38. awesome shot Renè, can you give me more tips to modify my pictures with this software ? I would be useful.

    ReplyDelete
  39. This comment has been removed by a blog administrator.

    ReplyDelete
  40. where will i put the Save Snapshot method? tnx am a student.
    and can i see the codings for the save snapshot. thanks.

    ReplyDelete