Silverlight 5 3D + SLARToolkit Augmented Reality = Win!

It's almost been a year since I wrote a sample for my open source Silverlight Augmented Reality Toolkit - SLARToolkit. The release of the new Silverlight 5 hardware accelerated 3D API was a nice occasion to finally make a new one.
In my last blog post I wrote a summary of all the Silverlight 5 beta features and some notes about the new low-level, XNA 3D API.
This post provides the new demo for SLARToolkit which leverages this fast GPU-based rendering to draw some nice effects with 60 frames per second. You can try the live sample if you have the Silverlight 5 beta installed or watch a video instead.

The SLARToolkit project description from the CodePlex site:

SLARToolkit is a flexible Augmented Reality library for Silverlight with the aim to make real time Augmented Reality applications with Silverlight as easy and fast as possible. It can be used with Silverlight's Webcam API or with any other CaptureSource or a WriteableBitmap. SLARTookit is based on the established NyARToolkit and ARToolkit. 

Live

A webcam and at least the Silverlight 5 beta runtime must be installed to run the sample. It's available here. Alternatively there is also a video of the new sample embedded below.

If you want to try it yourself you need do download the SLAR and / or L marker, print them and hold them in front of the camera. The marker(s) should be printed non-scaled at the original size (80 x 80 mm) and centered for a small white border. As an alternative it's also possible to open a marker file on a mobile device and to use the device's screen as marker. Also make sure the camera is set up properly and the scene is illuminated well without hard shadows. See the SLARToolkit Markers documentation for more details.

Open the sample in a new page.

Simply press the "Start Fun" Button to start the webcam. The size of the objects can be changed with the "Scale" Sliders. The "Flip x-axis" Checkbox could be used to flip the video (the webcam output is mirror-reversed by default). Click the "Glass" Checkbox to apply a glass effect to the sun model.

If you click the "Start Fun" Button for the first time you need to give your permission for the capturing. This application uses the default Silverlight capture device. You can specify the video and audio devices that are used by default with the Silverlight Configuration. Just press the right mouse button over the application, click "Silverlight" in the context menu and select the "Webcam / Mic" tab to set them.

Video

I've recorded a short video of the new sample with Expression Encoder's Screen Capture feature. Please keep in mind that the screen recording software eats up a lot of resources while recording and that the actual frame rate is even better. The video is also available at YouTube.

Background music is Neon Aurora by Alpha C

This demo shows how the new Silverlight 5 3D API can be used  to augment the reality with the help of SLARToolkit. It also demonstrates how the 3D DrawingSurface can be combined with the webcam video stream and overlaid with ordinary TextBoxes. This can be nice for educational projects and it's actually no problem to add correctly transformed videos or other content to the demo.

How it works

This sample uses the webcam video stream which is used to fill a Rectangle shape, the video stream is also constantly captured and fed to the SLARToolkit BitmapMarkerDetector to detect the markers. The detection result contains a transformation matrix for each found marker which is then used to transform the 3D objects and the TextBoxes.
A couple of vertex and pixel shaders are used to get the desired effects. Techniques like Phong shading, Bump mapping and Refraction mapping (glass) were implemented. The snapshots from the webcam are passed as a texture to the refraction pixel shader to simulate the glass effect.
See the source code if you're interested in the nitty-gritty details. Please note that I wrote an introduction to Pixel Shaders for Coding4Fun a while ago.
The current Silverlight 5 beta doesn't implement RenderTargets, therefore effects like bloom aren't really possible with the limitations of the Shader Model 2. You can add a faded billboard around the sun, but it wouldn't look that nice and when RenderTargets are added we can do way more nice effects in future releases of Silverlight 5.
I also added a new simple anti-jittering functionality to SLARToolkit to prevent the jiggling that mostly occurred due to the varying lightning conditions and noise in the video stream.

Credits
The SolarWind sample by the Silverlight team was used as a base and extended. The sample uses earth textures from the NASA. The pixel shader for the sun uses textures and concepts from an article by Nicolas Menzel. The moon textures are from the Celestia project.

Download it, build your app and augment your reality

The open source SLARToolkit library and all samples are hosted at CodePlex. If you have any comments, questions or suggestions don't hesitate and write a comment, use the Issue Tracker on the CodePlex site or contact me via any other media.
Have fun with the library and please keep me updated if you use it anywhere so I can put a link on the project site.