Thursday, December 17, 2009

World# - Real Time 3D Augmented Reality with Silverlight

Please read the Update at the end of this post.

In the Silverlight 4 Augmented Reality Proof Of Concept blog post I proved that it's possible to implement Augmented Reality applications with Silverlight 4 and the built-in webcam API. The proof of concept used an image showing a Silverlight logo that was attached to the tracked marker. With this blog post I'm back in this field, but this time with real 3D Augmented Reality in Silverlight!
I use the open source Augmented Reality library NyARToolkitCS again and implemented the necessary interfaces to make it work with Silverlight. The NyARToolkitCS library is a completely managed version of the well known ARToolkit. I also use the open source managed game engine Balder for the rendering of 3D models that augment the reality. Balder is mainly developed by the Silverlight MVP Einar Ingebrigtsen. He is working hard on his 3D game engine and provided quick fixes for some issues I encountered while using Balder and I also had a nice chat with him, so big shout out to Einar.

Live
A webcam and at least the Silverlight 4 runtime must be installed to run the sample. To view the application you need to install the Silverlight 4 runtime. It's available for Windows and Mac. You can also watch a video below.
If you want to try it yourself you need do download the SLAR and / or L marker, print them and hold them in front of the camera. The marker(s) should be printed non-scaled at the original size (80 x 80 mm) and centered for a small white border. Also make sure the camera is set up properly and the scene is illuminated well without hard shadows. See the SLARToolkit Markers documentation for more details.

Try it out here.

Simply press the "Start Fun" Button, hold the printed marker in front of the camera and move it around. Use the ComboBox to select a different 3D model. The "Flip x-axis" Checkbox could be used to flip the video (the webcam output is mirror-reversed by default).
If you click the "Start Fun" Button for the first time you need to give your permission for the capturing. This application uses the default Silverlight capture device. You can specify the video and audio devices that are used by default with the Silverlight Configuration. Just press the right mouse button over the application, click "Silverlight" in the context menu and select the "Webcam / Mic" tab to set them.
The Car model was made by Psionic, the Tank model by Ken Beyer. The Teapot is from the Balder samples and I guess it was made by Einar Ingebrigtsen. The rest was made by myself.

Video
This time I used Expression Encoder's Screen Capture feature to record a short video of the demo in action. Please keep in mind that the screen recording software eats up a lot of resources while recording and that the actual frame rate is way better.



How it works
Every time the CompositionTarget.Rendering event is fired, a webcam snapshot is taken asynchronously with the CaptureSource.AsyncCaptureImage method. This snapshot is converted in the right format and passed to the NyARToolkitCS library that outputs a transformation matrix. This matrix is used to transform the 3D model. The Balder viewport has a transparent background and the scene with the 3D objects is laid over the webcam video output. Another layer is an image that uses a WriteableBitmap and the WriteableBitmapEx library to highlight the tracked marker region with a red quad.
Sounds pretty easy, but actually it was not that easy and straight forward. One of the hardest parts was the initialization and to find the right parameters.

Results
I'm quite satisfied with the results and if you consider that all the heavy computation is done by the CPU, the performance is pretty good as well. The demo runs smoothly at 50-60 fps on a dual core machine utilizing its whole power and it should also scale up on more cores.

To be continued
If you expected source code this time I have to disappoint you. The code is not at that level where it should be, but I wanted to push this demo out. I'm not sure if I find enough time this year to work on the code and I don't want that someone tries to use it in the current state. But trust me, I will provide open source functionality to make Augmented Reality in Silverlight in the near future and I normally keep my promises, so stay tuned for more happy days...

Update 02-24-2010
This demo is now part of the open source SLARToolkit library which was announced in this blog post.

Update 03-15-2010
Updated to the Silverlight 4 Release Candidate.

Update 04-15-2010
Updated to the final Silverlight 4 RTW build.

34 comments:

  1. Absolutely Brilliant !!! Way to Go !!!

    ReplyDelete
  2. Great stuff, looking forward to trying this out.

    ReplyDelete
  3. Very cool, Well done!
    What component is responsible of "understanding" the Hiro logo? (in case you need to use another shape)

    ReplyDelete
  4. Thanks to all!

    @AssafD
    It's possible to use an other marker for tracking. The AR framework only needs a pattern for that.

    ReplyDelete
  5. creative and interesting

    ReplyDelete
  6. Dear Rene Schulte,

    my name is Julien Marty and I am a student in my first year of higher education at the ENSIIE (Ecole Nationale Supérieure d'Informatique pour l'Industrie et l'Entreprise).
    First, I want to congratulate you for your project about Augmented Reality with Silverlight 4. I am very interested in Augmented Reality that is why I wanted to contact you. Actually, I have developped the same project as yours but in C# language and it was for windows application. I would like to do it with Silverlight 4 but i have not succeeded in it. I use NYARToolkitCS.

    I would be very grateful if you could help me.

    yours faithfully

    Julien MARTY

    ReplyDelete
  7. Thanks. I will soon release the source code of this project and more that should make it easy to use AR with Silverlight. So please stay tuned...

    ReplyDelete
  8. When think put on the source code in release ?
    Thanks

    ReplyDelete
  9. I can't exactly say when, but I guess in the next couple of weeks. I have a job and a family, so there is only a limited amount of my spare time available.

    ReplyDelete
  10. Ok.
    For waiting your release can you send me by email the librairie of NYARToolkitCS adapting to Balder please and if possible your source code. I want to continue my research.
    my email is : juliensmarties@hotmail.com
    thanks for your help

    ReplyDelete
  11. Awesome stuff! I'm sure you're like me and don't have a lot of "spare time", but you know what would be killer: to put your marker on your hat or glasses and alter the display based on how far away you were and how off center you were and support something like what you see on this video...
    http://www.macrumors.com/2009/12/17/apple-working-on-3d-hyper-reality-displays/
    Should be possible right?

    ReplyDelete
  12. @Julien:
    Please be patient. I will release the code soon.

    ReplyDelete
  13. Thanks Tim.
    That idea is awesome and it should be possible. I think I will look into it after I released the code for other stuff.
    BTW, the projects from Johnny Lee are always impressive. He is now working in the Project Natal team. :)

    ReplyDelete
  14. Maybe someday the Natal team will make some of those fabulous libraries they're developing for human movement recognition available to Silverlight. ...assuming it's just .NET code and not hardware or dependent on XNA.

    ReplyDelete
  15. Any chance to see this on a Windows Mobile 6.5 Touch device? LG GM750 LOL...?

    Nice Work!!!

    ReplyDelete
  16. @luis
    Thanks. Silverlight is currently not supported by Windows Mobile 6.5 and this sample was made with the first beta of Silverlight 4. But there are some rumors that one of the next WinMo versions will support Silverlight.

    ReplyDelete
  17. Please... post source code... :)

    ReplyDelete
  18. Can you send me your code please juliensmarties@hotmail.com

    ReplyDelete
  19. @Julien:
    I would release the open source project sooner rather than later if I could, but there are some licensing issues that need to be solved first. The project is actually almost done, but there's this paperwork. I will release the project right after I've an agreement with the ARToolkit license holders.

    ReplyDelete
  20. OK. for waiting can you say me what did you do ?
    Are you modify the directX function by balder function in the NYARToolkit library ?
    Can you give me a way has follow that I can continue my researches please.
    Thanks,
    Julien

    ReplyDelete
  21. Hi! Congratz for the application!
    Well, I have to build sorta app like that and I don't know how to start... about the interfaces you said you had to implement for making NyARToolkitCS able to work with Silverlight, could you talk more about this??

    ReplyDelete
  22. Hi Kelly.
    I will soon release the source and even more. I'm working on SLARToolkit, the Silverlight Augmented Reality Toolkit. :) So please be patient.

    ReplyDelete
  23. Hello, i'm trying to redo the same thing, but i don't understand everything about matrix, transformation, projection, etc..

    Is your mesh always in 0,0,0 and only the camera is moving ? Or the camera is fixed and the mesh is moving ?

    I think it's the first solution, and i'm tryng to calculate camera position and orientation in space, with DetectionResults[0].Transformation and MarkerDetector.Projection, but i don't know how to take tranformation matrixs and calculate coordinates in space.

    i'm very impatient to understand how you did this.
    (scuse me for my poor english)

    ReplyDelete
  24. The camera is static.
    There are two examples that come with the SLARToolkit project. It's the easiest for you to use one of them as a starter.

    See the "Update 02-24-2010" text in the above post.

    ReplyDelete
  25. Hi! how i create .ase models??

    ReplyDelete
  26. Hi Anna,

    An .ase file contains a 3D model. Most of the 3D modelling tools have exporters for this format.

    For Balder in general I'd recommend this video:
    http://vimeo.com/13659554

    ReplyDelete
  27. Hello, you can provide at least a snippet of code that explains how to use 3D models in SLARToolkit. I'm trying, but it only works with images, text and other controls that are inside the "canvas", it must be because SLARToolkit uses the projection of elements of Silverlight to peform the transformations. When I try to use the elements of Balder, such as meshes, it can't peform the transformations.

    Please, if you can help me, I would greatly appreciate it. I need it because I'm doing my thesis of my degree in computer science, and I'm running short on time.

    Thank you.

    ReplyDelete
  28. Hi André,

    Please read the project page in the Samples section. The important part is highlighted (bold):
    "The samples are part of the source code repository, but not included in the binary release."
    http://slartoolkit.codeplex.com/

    So just download the code and you'll find all the samples, including the Balder sample, in the downloaded package.
    http://slartoolkit.codeplex.com/SourceControl/list/changesets#

    ReplyDelete
  29. This is excellent because i like to see how reality is showed in this video, it's perfect because I've always dreamed making something like that.

    ReplyDelete
  30. Hi Rene,

    Thanks for the wonderful work, i have downloaded the source code for silverlight 3d which with the car tank teapot and cube.. and try to add another 3d in .ase format when i debug there is no error popped up but then when the website open with the silverlight icon once it reached 100% that page goes blank white.. may i know whats the reason or any solution... hope to hear soon.. your fast feedback will be very helpful thanks alot

    ReplyDelete
  31. Hi tana,

    Please make sure you have the Exceptions enabled. You might miss something. VS Menu -> Debug -> Exceptions -> check 'Thrown' at CLR Exceptions.
    It's also possible your model is too big and therefore you can only see one side. Just scale it down.

    ReplyDelete