Silverlight 5 3D + SLARToolkit Augmented Reality = Win!

It's almost been a year since I wrote a sample for my open source Silverlight Augmented Reality Toolkit - SLARToolkit. The release of the new Silverlight 5 hardware accelerated 3D API was a nice occasion to finally make a new one.
In my last blog post I wrote a summary of all the Silverlight 5 beta features and some notes about the new low-level, XNA 3D API.
This post provides the new demo for SLARToolkit which leverages this fast GPU-based rendering to draw some nice effects with 60 frames per second. You can try the live sample if you have the Silverlight 5 beta installed or watch a video instead.

The SLARToolkit project description from the CodePlex site:

SLARToolkit is a flexible Augmented Reality library for Silverlight with the aim to make real time Augmented Reality applications with Silverlight as easy and fast as possible. It can be used with Silverlight's Webcam API or with any other CaptureSource or a WriteableBitmap. SLARTookit is based on the established NyARToolkit and ARToolkit. 

Live

A webcam and at least the Silverlight 5 beta runtime must be installed to run the sample. It's available here. Alternatively there is also a video of the new sample embedded below.

If you want to try it yourself you need do download the SLAR and / or L marker, print them and hold them in front of the camera. The marker(s) should be printed non-scaled at the original size (80 x 80 mm) and centered for a small white border. As an alternative it's also possible to open a marker file on a mobile device and to use the device's screen as marker. Also make sure the camera is set up properly and the scene is illuminated well without hard shadows. See the SLARToolkit Markers documentation for more details.

Open the sample in a new page.

Simply press the "Start Fun" Button to start the webcam. The size of the objects can be changed with the "Scale" Sliders. The "Flip x-axis" Checkbox could be used to flip the video (the webcam output is mirror-reversed by default). Click the "Glass" Checkbox to apply a glass effect to the sun model.

If you click the "Start Fun" Button for the first time you need to give your permission for the capturing. This application uses the default Silverlight capture device. You can specify the video and audio devices that are used by default with the Silverlight Configuration. Just press the right mouse button over the application, click "Silverlight" in the context menu and select the "Webcam / Mic" tab to set them.

Video

I've recorded a short video of the new sample with Expression Encoder's Screen Capture feature. Please keep in mind that the screen recording software eats up a lot of resources while recording and that the actual frame rate is even better. The video is also available at YouTube.

Background music is Neon Aurora by Alpha C

This demo shows how the new Silverlight 5 3D API can be used  to augment the reality with the help of SLARToolkit. It also demonstrates how the 3D DrawingSurface can be combined with the webcam video stream and overlaid with ordinary TextBoxes. This can be nice for educational projects and it's actually no problem to add correctly transformed videos or other content to the demo.

How it works

This sample uses the webcam video stream which is used to fill a Rectangle shape, the video stream is also constantly captured and fed to the SLARToolkit BitmapMarkerDetector to detect the markers. The detection result contains a transformation matrix for each found marker which is then used to transform the 3D objects and the TextBoxes.
A couple of vertex and pixel shaders are used to get the desired effects. Techniques like Phong shading, Bump mapping and Refraction mapping (glass) were implemented. The snapshots from the webcam are passed as a texture to the refraction pixel shader to simulate the glass effect.
See the source code if you're interested in the nitty-gritty details. Please note that I wrote an introduction to Pixel Shaders for Coding4Fun a while ago.
The current Silverlight 5 beta doesn't implement RenderTargets, therefore effects like bloom aren't really possible with the limitations of the Shader Model 2. You can add a faded billboard around the sun, but it wouldn't look that nice and when RenderTargets are added we can do way more nice effects in future releases of Silverlight 5.
I also added a new simple anti-jittering functionality to SLARToolkit to prevent the jiggling that mostly occurred due to the varying lightning conditions and noise in the video stream.

Credits
The SolarWind sample by the Silverlight team was used as a base and extended. The sample uses earth textures from the NASA. The pixel shader for the sun uses textures and concepts from an article by Nicolas Menzel. The moon textures are from the Celestia project.

Download it, build your app and augment your reality

The open source SLARToolkit library and all samples are hosted at CodePlex. If you have any comments, questions or suggestions don't hesitate and write a comment, use the Issue Tracker on the CodePlex site or contact me via any other media.
Have fun with the library and please keep me updated if you use it anywhere so I can put a link on the project site.

XNA for the Web - Silverlight 5

Many great new things were announced during the second keynote at Microsoft's MIX11 conference. One of it was the Silverlight 5 beta which is available for download here.
The new features are just awesome and many of them were requested by the community. As usual Tim Heuer's great blog post covers all the new goodies in detail and provides videos and source code for most of them. Kudos to the Silverlight team for releasing another great version.
My blog post contains a summary of all the new features the Silverlight 5 beta ships and provides some notes about the new 3D API. The next blog post will bring a new SLARToolkit 3D demo.

Features

• XNA 3D Graphics API
• Improved Graphics stack – The graphics stack has been re-architected to bring over improvements from WP7, such as Independent Animations.
• Hardware video decode for H.264 playback.
• Multi-core background JIT support for improved startup performance.
• Realtime Sound (low-latency Audio)
• Variable Speed Playback (“Trick Play”)
• XAML Binding Debugging
• Multiple Window Support
• Ancestor RelativeSource Binding
• Implicit DataTemplates
• ClickCount
• Binding on Style Setter
• Linked Text Containers
• Custom Markup Extensions
• ComboBox type ahead with text searching.
• Full keyboard support in full-screen for trusted in-browser applications, enables richer kiosk and media viewing applications in-browser.
• Default filename in SaveFileDialog – Specify a default filename when you launch the SaveFileDialog.
• Unrestricted filesystem access – trusted applications can Read write to files in any directory on the filesystem.
• More performance optimizations:
• XAML Parser performance optimizations.
• Network Latency optimizations.
• Text layout performance improvements.
• Hardware acceleration is enabled in windowless mode with Internet Explorer 9.

Find more details at the Silverlight site and read John Papa's and Pete Brown's blog posts.

XNA for the Web
One of the topmost requested features at the Silverlight Uservoice site was hardware accelerated 3D graphics. The Silverlight team listened closely and added an immediate mode, low-level,  XNA 3D API to Silverlight 5. An immediate mode API offers functionality to render graphics directly to the graphics interface without keeping a list of the objects. A retained mode API like WPF 3D on the other hand keeps an internal list of the object graph. An immediate mode API is a great choice since it provides the maximum control over the rendering and mostly results in superior performance compared to retained mode rendering.
The Silverlight 5 3D API doesn't contain all the higher-level classes one might know from the XNA Game Studio. The Content Pipeline, the SpriteBatch class and others were left out. I assume this decision was made to keep the footprint of the Silverlight runtime small. However, the team added all the needed core functionality; features like the SpriteBatch can be build on top of it by the community. My Silverlight MVP friend Bill Reiss is already working on a SpriteBatch version for Silverlight 5.
The new GPU accelerated 3D API gives us Silverlight developers the core XNA functionality with shaders and vertex rendering inside the browser! Shaders aren't easy for most non-graphics programmers, but they give us a huge flexibility and we can build various higher levels on top of the new 3D API.
You can render some pretty impressive things with the new Silverlight 5 3D API and it's possible to draw millions of triangles on the GPU without almost no CPU load.

Here's a sneak peek at the new SLARToolkit demo which also contains a simple real-time glass effect.

Silverlight Integration
The 3D API contains a control called the DrawingSurface, which is basically a rendering surface or a 3D Canvas if you will. The DrawingSurface has the Draw event which is raised for every frame to draw. The event is fired on a render thread which runs in parallel to the UI thread. This is an important concept introduced with Silverlight 5.
Inside a Draw event handler the states and shaders are set and the vertex and index buffers are passed to the GPU for rendering. The vertex and pixel shaders implement the lightning model and define how the vertices are rendered. The current Silverlight 5 3D beta only supports Shader Model 2, but this will probably change in a future release, but even with Shader Model 2 many neat lightning models and effects can be implemented. You will find some nice effects in the new SLARToolkit demo.

Check out the Silverlight 5 3D samples if you want to learn more about the new 3D API. If you're not familiar with 3D graphics concepts or just want to make your life easier, then I strongly recommend to give the 3D engine Balder a try. It's a fantastic open source engine by my good Silverlight MVP friend Einar Ingebrigtsen. You just need to write a couple lines of XAML and you're good to go. Balder has built-in model loaders to load complex 3D models and Einar provides a big sample library. He also brought the engine to a few more platforms like Windows Phone 7, OpenGL and has even a neat software rendering fallback. Read his blog post here.
Another Silverlight MVP friend of mine, Andy Beaulieu made a very cool demo using Balder and a 3D physics engine. He blogged about it and even released the 3D physics engine as open source!

The beta of the 3D API is missing some essential things like RenderTargets at the moment, but it's just a beta and more stuff might be announced at the MIX11 conference. If you would like to know more details about the new Silverlight 5 3D API and the roadmap, then you should definitely check out the MIX11 session Graphics & 3D with Silverlight 5 by the Silverlight program manager Aaron Oneal on Wednesday, April 13 3:30 pm - 4:30 pm PDT. All MIX sessions are recorded and will be available online 24 hours after each session.

Conclusion
No, Silverlight is not dead and it's clear that Microsoft is investing in it. I'm sure the new features and esp. the XNA 3D API will lead to some great applications in the future. The future is bright and we are part of it.

Long Time No See - WriteableBitmapEx 0.9.7.0

I actually wanted to release the feature complete version 1.0 of the WriteableBitmapEx library after the previous release, but I've been very busy with other projects during the last year. That's why I decided to release version 0.9.7.0 today. It contains many fixes, but also adds some features (see below). This version finally provides an official Windows Phone binary build, although the Windows Phone project was already available shortly after the first CTP was released. The zipped binaries can be downloaded from here. Please note that this package only contains the WriteableBitmapEx binaries for Silverlight and Windows Phone. All the samples can be found in the source code repository.

WPF
There's now also an unmaintained WPF branch of the WriteableBitmapEx library in the source code repository. It was contributed by Szymon Kobalczyk.
One might ask why it's not maintained. I started a poll and it seems that only 7 out of 16 need a WPF version of WriteableBitmapEx. The code is also quite bloated due to conditional compilation flags, which makes it harder to maintain in the future. That's why I decided not to integrate the WPF version. I just don't have enough time to maintain a hardly used version. I would rather point WPF users to Jeremiah Morrill's new project called DirectCanvas. It's in an early stage, but he's a great guy and working hard on it. I'm sure we'll see a great, GPU accelerated 2D drawing library for WPF in the future.

Feature list version 0.9.7.0

• Fixed many bugs.
• Added the Rotate method which rotates the bitmap in 90° steps clockwise and returns a new rotated WriteableBitmap.
• Added a Flip method with support for FlipMode.Vertical and FlipMode.Horizontal.
• Added a new Filter extension file with a convolution method and some kernel templates (Gaussian, Sharpen).
• Added the GetBrightness method, which returns the brightness / luminance of the pixel at the x, y coordinate as byte.
• Added the ColorKeying BlendMode.
• Added boundary checks to the Crop() function to avoid OutOfRangeExceptions if the passed parameters are outside the boundaries of the original bitmap.
• Optimized the DrawLine algorithm.
• Optimized the Resize algorithms (NearestNeighbor is now 10x faster).
• Optimized the Clear(Color) method.

Community FTW!
Thanks to the community for constantly reporting bugs and suggesting new features. You rock! That's why I love open source software.

Kostenloser Windows Phone Artikel in der DotNetPro

I've written a Windows Phone development article for Germany's largest .Net developer magazine. It's written in German.

Für die aktuelle DotNetPro habe ich einen Artikel zu der Entwicklung für Windows Phone 7 geschrieben. Anhand eines praktischen Beispiels werden die einzelnen APIs und Tricks gezeigt um eine Windows Phone App für die einfache Bildmanipulation zu entwickeln.
Als Werbung für das aktuelle Magazin wird der Artikel auch kostenlos online zur Verfügung gestellt. In der Druckausgabe findet man neben meinem Artikel auch andere Windows Phone Artikel. Wer noch tiefer in die Windows Phone Entwicklung einsteigen möchte, dem sei auch das Buch von Patrick Getzmann, Simon Hackfort, und Peter Nowak empfohlen: Entwickeln für Windows Phone 7: Architektur, Frameworks, APIs.

Stay in Touch - How to Contact Your Marketplace Customers

Photo (CC) by  LarimdaME

A while ago I provided some methods and code snippets on how to get feedback from users. Today I show a manual procedure how you can contact some of the customers that reviewed your Windows Phone app. Some might already know this, but not all developers are aware of this Marketplace / Zune functionality, so I though it might be worth sharing.

Most customers are nice people that provide valuable feedback also through the Marketplace review mechanism. Others give you bad reviews and it seems like they might have misunderstood the functionality. And some people just love to rant. Just like in real life. If I feel the urge to contact some of the reviewers, I send a message through the Zune social network and ask for details. Unfortunately, not all review authors are registered there and only ~50% of the people can be found by their username.

Step by Step
1. Go to http://wp7reviews.tomverhoeff.com and enter the Product Id of your app.
This neat little Silverlight app will list all reviews from around the world. I recommend you bookmark the URL with the Product Id.
2. Sort the grid by date, go through the list and copy the username of the reviewer you want to contact.

3. Open the Zune software, go to the Marketplace tab, paste the username into the search field and hit the search button.

4. If the user was found, click on the user's profile.

5. Open the user's profile and send the user a message through Zune ("Send message").

Conclusion
That's it. Not the best solution and you will only reach ~50% of the users, but better than nothing. I would prefer if app developers could comment the reviews.
Please stay professional and don't offend the review authors even if they have written nonsense.