SLARToolkit Samples Updated

The samples of my open source Windows Phone and Silverlight Augmented Reality Toolkit were updated to the latest version of the WP 7.1 and Silverlight 5 SDKs.
Please note the changed security model in Silverlight 5, which is a big bummer. My Silverlight MVP friend Morten wrote a few true words about it here.
As usual you can find a list of the samples on the project site and also get the code there.

Interview on .Net Rocks!

About a month ago Richard Campbell asked me if I'd like to give an interview about SLARToolkit on the .Net Rocks talk show. Of course I agreed right away. .Net Rocks! is the largest .Net developer podcast out there and Carl Franklin and Richard Campbell always have great guests. It's a real honor for me to have been on show #564.
We mainly talked about the Silverlight Augmented Reality Toolkit (SLARToolkit), but also about some of my other projects and articles.
You can download the podcast in various formats from here. I hope you like it. Also make sure you subscribe to .Net Rocks!

Coding4Fun - Introduction to Silverlight and WPF Pixel Shaders

My new article for Microsoft's Coding4Fun site is live. It's my second article for Coding4Fun after my first Silverlight Face Detection article. The article explains how to write pixel shaders for Silverlight and WPF, what tools should be used, and how to work with the tools. Furthermore, it shows how to build an extensible Silverlight shader application with the help of the Managed Extensibility Framework (MEF)
The application not only comes with the two shaders that are implemented in the article, it also contains three other shaders I’ve written before. The complete source code is licensed under the Ms-PL and can be downloaded from the CodePlex site.

Ye Olde Kamera - Silverlight 4 Webcam & Old Movie Shader

Another great and successful year comes to an end and I hope that the next year will get even better. We all don't get younger and we change over the years, but the last Silverlight demo I made for this year allows you take some fake aged memories with your webcam. The Silverlight 4 webcam and microphone support is always fun and even more when it's combined with a pixel shader. This demo uses my Old Movie Pixel Shader to create a neat real time webcam effect that simulates scratches, noise and other effects you might have seen in old movies. I also brushed things up a bit by framing the video output with a filmstrip and using some nice new icons from eggheadcafe.

Live
The Silverlight 4 runtime must be installed to run the demo. It's available for Windows and Mac.

 Open the sample in a new page.

The Webcam capturing could be started and stopped with the camera Button. If you press it for the first time you need to give your permission for the capturing. This application uses the default Silverlight capture device. You can specify the video and audio devices that are used by default with the Silverlight Configuration. Just press the right mouse button over the application, click "Silverlight" in the context menu and select the new "Webcam / Mic" tab to set them.
Press the disk Button to take a framed snapshot and save it to a JPEG file on your harddisk.
The amount of noise can be changed using the Slider and the movie ToggleButton allows you to disable the Old Movie Shader.

How it works
It's basically the Silverlight 4 CaptureSource class and my Old Movie Pixel Shader attached to the video output. I covered the Silverlight webcam and shader usage in this blog post and the JPEG encoding in this one. The Old Movie Shader was presented here.

Source code
The Visual Studio 2010 solution including the refactored Old Movie Shader is available here.

Final words
Thanks to my kids and especially to my wife for all her patience in 2009. I love her!
I also like to thank all the people and new friends I got to know this year and especially the Silverlight community. I hope you liked my blog posts and enjoyed the Silverlight demos I've made. I have more to come next year.
I wish you all a great new year, all the best in 2010 and peace!

Update 03-20-2010
Updated to the Silverlight 4 release candidate.

Update 04-15-2010
Updated to the final Silverlight 4 RTW build.

EdgeCam Shots - Saving Silverlight 4 Webcam Snapshots to JPEG

In my last blog post I have covered the new Silverlight 4 Webcam API and provided a demo that used my edge detection pixel shader to create a nice real time webcam effect. In this post I make an extended version available which can save webcam snapshots as JPEG files and I also discuss some limitations of the webcam API's built-in CaptureSource.AsyncCaptureImage snapshot method. Furthermore I will give some ideas on how to build a Silverlight 4 video chat / conference application on top of the provided JPEG capturing and encoding code.


Live
To view the application you need to install the Silverlight 4 runtime. It's available for Windows and Mac.

Open the sample in a new page.

The Webcam capturing could be started and stopped with the "Start Capture" Button. If you press it for the first time you need to give your permission for the capturing. This application uses the default Silverlight capture device. You can specify the video and audio devices that are used by default with the Silverlight Configuration. Just press the right mouse button over the application, click "Silverlight" in the context menu and select the new "Webcam / Mic" tab to set them.
Press the "Save Snapshot" Button to take a snapshot and save it to a JPEG file on your harddisk.
The threshold of the edge detection can be changed using the Slider and the "Bypass" Checkbox allows you to disable the shader.

How it works
The base Silverlight 4 webcam usage code was covered in my last blog post.

The new eventhandler for the "Save Snapshot" button:

private void BtnSnapshot_Click(object sender, RoutedEventArgs e)
{
   if (saveFileDlg.ShowDialog().Value)
   {
      using (Stream dstStream = saveFileDlg.OpenFile())
      {
         SaveSnapshot(dstStream);
      }
   }
}

The code is pretty obvious: A SaveFileDialog is shown and if the user enters a file name and hits OK, a stream to the file will be opened and passed to the SaveSnapshot method. There's only one think to keep in mind when using the SaveFileDialog.ShowDialog() method, it can only be called from user-initiated code like an event handler, otherwise a SecurityException is thrown.

The SaveSnapshot method including comments:

// Render Rectangle manually into WriteableBitmap
WriteableBitmap bmp = new WriteableBitmap(ViewportHost, null);

// Init buffer in FluxJpeg format
int w = bmp.PixelWidth;
int h = bmp.PixelHeight;
int[] p = bmp.Pixels;
byte[][,] pixelsForJpeg = new byte[3][,]; // RGB colors
pixelsForJpeg[0] = new byte[w, h];
pixelsForJpeg[1] = new byte[w, h];
pixelsForJpeg[2] = new byte[w, h];

// Copy WriteableBitmap data into buffer for FluxJpeg
int i = 0;
for (int y = 0; y < h; y++)
{
   for (int x = 0; x < w; x++)
   {
      int color = p[i++];
      pixelsForJpeg[0][x, y] = (byte)(color >> 16); // R
      pixelsForJpeg[1][x, y] = (byte)(color >> 8);  // G
      pixelsForJpeg[2][x, y] = (byte)(color);       // B
   }
}

//Encode Image as JPEG
var colModel = new ColorModel { colorspace = ColorSpace.RGB };
var jpegImage = new FluxJpeg.Core.Image(colModel, pixelsForJpeg);
var encoder = new JpegEncoder(jpegImage, 95, dstStream);
encoder.Encode();

The Rectangle's surrounding Grid "ViewportHost" is rendered into a WriteableBitmap and the WriteableBitmap's Pixels are copied into another buffer with a different format. The rendered image is then written as a JPEG encoded stream using the open source FJCore library which is distributed under the MIT License. I've found some code at Stackoverflow on how to use the library in combination with the WriteableBitmap, but I modified / shortened it.
See my post on how to Convert, Encode And Decode Silverlight WriteableBitmap Data if you are interested in more WriteableBitmap conversion and JPEG encoding code.

Why not CaptureSource.AsyncCaptureImage?
You might be wondering why I haven't used the Silverlight 4 webcam API's built-in AsyncCaptureImage snapshot method of the CaptureSource class.

1. The AsyncCaptureImage method grabs a frame directly from the device and therefore any effects like the edge detection pixel shader won't be visible in the rendered image.
2. It only works if the CaptureSource was started, so the user can't save the visible image when the capturing was stopped. See the MSDN for details.

Please keep in mind that the first beta version of Silverlight 4 was used for this blog post and that some things will be changed in subsequent releases.

Where to go from here
The code in the SaveSnapshot method could be optimized and also multithreaded. The rendering should run in it's own thread. The encoding in a separate thread too and the network transport also. This approach would utilize modern quad core CPUs. After that it might be a good starting point for a video chat / conferencing application that continuously renders JPEGs and transports them between Silverlight clients. This technique is similar to the M-JPEG video format that also uses separately compressed JPEGs.
To make this idea usable for a Silverlight video chat, only the rendering, the JPEG encoder and the transfer method  need to be fast enough for real time streaming.
Please leave a comment what you think about it.

Source code
Download the Visual Studio 2010 solution here.

Update 03-20-2010
Updated to the Silverlight 4 release candidate.

Update 04-15-2010
Updated to the final Silverlight 4 RTW build.

EdgeCam - Silverlight 4 Webcam & Edge Detection Shader

Only 4 months after Silverlight 3 RTM came out Microsoft released the first beta version of Silverlight 4 on 11-18-2009 at the PDC. The new features are just awesome and many of them were requested by the community. Tim Heuer's great blog post covers all the new goodies in detail and provides videos and source code for most of them.
One of my favorite new features is the webcam and microphone support and I've coded a small demo that uses the new Silverlight 4 CaptureSource class and applied my edge detection pixel shader to it to create a cool real time webcam effect.

Live
To view the application you need to install the Silverlight 4 runtime. It's available for Windows and Mac. You can also watch a video below.

 Open the sample in a new page.

The Webcam capturing could be started and stopped with the Button in the middle. If you press it for the first time you need to give your permission for the capturing. This application uses the default Silverlight capture device. You can specify the video and audio devices that are used by default with the Silverlight Configuration. Just press the right mouse button over the application, click "Silverlight" in the context menu and select the new "Webcam / Mic" tab to set them.
The threshold of the edge detection can be changed using the Slider and the "Bypass" Checkbox allows you to disable the shader.

Video
I've recorded a short video with my iPhone 3GS that shows the demo running on my Samsung NC 10 Atom Netbook. The built-in webcam is not very good and the video was recorded at night, but I think it's good enough to see how the effect looks.

How it works
The new Silverlight 4 webcam and microphone API is pretty easy to use:

// Init capture devices
captureSource = new CaptureSource();
captureSource.VideoCaptureDevice = CaptureDeviceConfiguration.GetDefaultVideoCaptureDevice();
captureSource.AudioCaptureDevice = CaptureDeviceConfiguration.GetDefaultAudioCaptureDevice();

// Create video brush that preserves the aspect ratio and fill a Rectangle with it
VideoBrush videoBrush = new VideoBrush();
videoBrush.Stretch = Stretch.Uniform;
videoBrush.SetSource(captureSource);
Viewport.Fill = videoBrush;

// Ask user for permission
if (CaptureDeviceConfiguration.AllowedDeviceAccess
|| CaptureDeviceConfiguration.RequestDeviceAccess())
{
   captureSource.Start();
}

The edge detection pixel shader is applied to the "Viewport" Rectangle in the XAML. For a more intuitive behavior I've also attached a negative ScaleTransform to flip the x axis of the Rectangle. Otherwise the webcam output would be mirror-reversed by default.

Source code
The Visual Studio 2010 solution including the edge detection shader is available for download.

Update 11-23-2009
Make sure to read the follow up blog post EdgeCam Shots - Saving Silverlight 4 Webcam Snapshots to JPEG

Update 03-20-2010
Updated to the Silverlight 4 release candidate.

Update 04-15-2010
Updated to the final Silverlight 4 RTW build.

Read Between The Pixels - HLSL Kill Pixel Shader

A couple of days ago William Moore (@codenenterp) asked me via Twitter if I know a pixel shader that makes odd pixels black. I didn't know were such a shader is available, but I knew that it should be not hard to write one in HLSL. So I fired up the Shazzam tool and wrote two shaders.






I have started with a pixel shader that kills the color in every odd row of the texture and makes every second scanline transparent. If a black rectangle is overlaid with such a shaded image, each odd scanline will be seen as black, thus resulting in the desired black scanline effect.

An alternating line effect could be realized if an image instead of a black rectangle is overlaid.

After that I've extended the pixel shader to kill every odd pixel to produce the checkerboard effect William Moore wanted.

This is how a Silverlight Button could look like if the shader is applied to it:

How it works
The pixel shaders are pretty simple and small. As always, I've commented the HLSL code, but if you have any further questions, feel free to leave a comment.
Here is the shader that kills every odd scanline:

// Parameter
float2 TextureSize : register(C0);

// Sampler
sampler2D TexSampler : register(S0);

// Shader
float4 main(float2 texCoord : TEXCOORD) : COLOR 
{ 
   // Default color is fully transparent
   float4 color = 0;

   // Scale to int texture size
   float row = texCoord.y * TextureSize.y * 0.5f; 

   // Calc diff between rounded half and half to get 0 or 0.5
   float diff = round(row) - row;
   float diffSq = diff * diff;

   // Even or odd? Only even lines are sampled
   if(diffSq < 0.1)
   {
      color = tex2D(TexSampler, texCoord);
   } 
   return color;
}

And here the shader that kills every odd pixel:

// Parameter
float2 TextureSize : register(C0);

// Sampler
sampler2D TexSampler : register(S0);

// Shader
float4 main(float2 texCoord : TEXCOORD) : COLOR 
{ 
   // Default color is fully transparent
   float4 color = 0;

   // Scale to int texture size, add x and y
   float2 vpos = texCoord * TextureSize * 0.5f; 
   float vposSum = vpos.x + vpos.y;
 
   // Calc diff between rounded half and half to get 0 or 0.5
   float diff = round(vposSum) - vposSum;
   float diffSq = diff * diff;

   // Even or odd? Only even pixels are sampled
   if(diffSq < 0.1)
   {
      color = tex2D(TexSampler, texCoord);
   } 
   return color;
}

As you can see the only difference is that the kill pixel shader uses the sum of the texture coordinate's x and y component to determine if the pixel is even or odd.
For the effect of the uppermost blog post image (mandrill), I've used the product of x and y instead of the sum.
The size of the killed pixel group could be changed with the TextureSize parameter. The following example uses an eight of the original picture size as TextureSize.

Source code
I have also written a small Silverlight application that shows an image with the pixel shader attached. You can download the Visual Studio 2008 solution including the pixel shaders from here.

Photos from the USC-SIPI Image Database

Ye Olde Pixels - Silverlight 3 Old Movie Pixel Shader

In my latest WriteableBitmap Performance post I mentioned that the Silverlight 3 pixel shaders run really fast, although they are executed on the CPU and not on the GPU and that nice real-time effects can be done with it. A Silverlight 3 pixel shader could also be applied to any UIElement, thus the MediaElement too. The Silverlight MediaElement is mainly used to play music or video and with an attached shader many cool effects could be realized. As you might know I like nice effects and due to this passion I have coded another pixel shader for Silverlight. This time I have implemented a shader that simulates scratches, noise and other effects you might have seen in old movies.

Live

The intensity of the scratches and the noise is controlled with the Sliders. The shader could be disabled with the "Bypass" checkbox and it's also possible to load a WMV clip with the "Load" Button.
William Moore made some short video clips with After Effects for me and I have uploaded them to my Dropbox. Just copy a URL to the TextBox and press "Load" to try them with the shader:

• Grass.wmv
• MooreFamily.wmv

How it works
Here's the pixel shader written in HLSL:

// Parameters
float ScratchAmount : register(C0);
float NoiseAmount : register(C1);
float2 RandomCoord1 : register(C2);
float2 RandomCoord2 : register(C3);
float Frame : register(C4);

// Static
static float ScratchAmountInv = 1.0 / ScratchAmount;

// Sampler
sampler2D TexSampler : register(S0);
sampler2D NoiseSampler : register(S1);

// Shader
float4 main(float2 uv : TEXCOORD) : COLOR
{
    // Sample texture
    float4 color = tex2D(TexSampler, uv);

    // Add Scratch
    float2 sc = Frame * float2(0.001f, 0.4f);
    sc.x = frac(uv.x + sc.x);
    float scratch = tex2D(NoiseSampler, sc).r;
    scratch = 2 * scratch * ScratchAmountInv;
    scratch = 1 - abs(1 - scratch);
    scratch = max(0, scratch);
    color.rgb += scratch.rrr; 

    // Calculate random coord + sample
    float2 rCoord = (uv + RandomCoord1 + RandomCoord2) * 0.33;
    float3 rand = tex2D(NoiseSampler, rCoord);
    // Add noise
    if(NoiseAmount > rand.r)
    {
        color.rgb = 0.1 + rand.b * 0.4;
    }

    // Convert to gray + desaturated Sepia
    float gray = dot(color, float4(0.3, 0.59, 0.11, 0)); 
    color = float4(gray * float3(0.9, 0.8, 0.6) , 1);

    // Calc distance to center
    float2 dist = 0.5 - uv;   
    // Random light fluctuation
    float fluc = RandomCoord2.x * 0.04 - 0.02;
    // Vignette effect
    color.rgb *= (0.4 + fluc - dot(dist, dist))  * 2.8;

    return color;
}

The 1st effect adds some random scratches, which are moving slowly each frame.
Unfortunately the Shader Model 2 doesn't support any random function and the noise() intrinsic could not be used for real-time effects. That's a problem if you want to make pseudorandom-based shaders, but one key element of restricted shader development is pre calculation and texture lookup. The randomness, which is used in the shader, comes from a WriteableBitmap, that is filled with random color in the initialization step of the application. The shader in turn samples random values from that texture.

The 2nd effect adds a bit random noise to the image. If the texture coordinates with the frame counter would only be used, a static, moving pattern would be seen and it wouldn't look noisy at all. So actually a new random noise texture is needed every frame to achieve the desired effect. Generating a random texture with a WriteableBitmap in every frame would be too expensive. Instead of this, a little trick is used and only four random numbers are generated every frame and passed to the shader as parameters. These random seeds are then used as texture coordinates for a random lookup in the random colored texture. Therefore each pixel in every frame samples a different random value. Although it is not highly random, the values are sufficient enough to produce changing noise.

The 3rd step converts the pixel to grayscale and colors it afterwards with a desaturated Sepia tone.

The 4th and last effect computes the distance to the center and multiplies it with the color. This generates the vignette effect as a fadeout to black towards the edge. A random light flickering of an old projector is also simulated.








That's it and it's all about cheating.

Source code
Feel free to download the Visual Studio 2008 solution including the pixel shader from here.

Update 12-29-2009
I refactored the code behind shader class and separated it completly from the calling code (MainPage.xaml.cs). Now the shader class generates a default noise texture and random coordinates itself.
The linked source code was updated.

Update 12-31-2009
See this blog post if you want to try the shader in real time with your own webcam.

Silverlight 3 WriteableBitmap Performance Follow-Up

Two weeks ago I wrote the Silverlight 3 WriteableBitmap Performance blog post and got some good response on it. One of it came from the author of Quakelight, Julien Frelat. He contacted and asked me if I had tested his PNG wrapper technique, which is used in the Silverlight port of the famous game Quake. I thought Quakelight uses the WriteableBitmap and not a custom Stream hack, so I haven’t considered to check it for the Speedtest. Actually Quakelight uses the Silverlight 3 WriteableBitmap, but the source code includes a custom Stream for Silverlight 2 too.
For better performance the Quakelight PngWrapper and BitmapData classes use a 8 bit color palette instead of full 32 bit ARGB colors. This fact makes it not directly comparable to the other competitors, which all support 32 bit ARGB colors, but for certain problems 256 colors could be sufficient. That’s why I wrote this follow-up and integrated Quakelight’s Silverlight 2 Stream implementation.
The Speedtest generates an interference image and writes each pixel to a buffer, which is then used as the source of an Image. This effect could also be realized with a pixel shader and as I was working on the Speedtest v2, I implemented the effect as a pixel shader too.
Make sure to check the Update section at the bottom of this post.

The Competitors

1. Silverlight 3 WriteableBitmap.
2. RawPngBufferStream from the open source GameEngine Balder.
3. Nikola's PngEncoder, which is an improved version of Joe Stegman's work.
4. Ian Griffiths' SlDynamicBitmap library.
5. NEW: Quakelight’s 8 bit BitmapData.
6. NEW: A pixel shader.

Live

The application measures the time, which every implementation needs to draw the "Maximum Frames" and calculates the mean frames per seconds (fps). The third text column shows the relative performance compared to the WriteableBitmap.
If the tests complete very fast, you should increase the "Maximum Frames" to get better results.


How it works
The image is still 512 x 512 pixels in size and the mean frames per second are measured, but I changed the CalculateColor(int x, int y) effect a bit and removed the lookup table for the sine movement of the circles. The effect looks almost the same, but the code is much better to understand. At least I hope so.

private void CalculateColor(int x, int y)
{
   // Normalize coordinates
   double xn = x * TexSizeInv;
   double yn = y * TexSizeInv;

   // Overlayed sine circle rings
   // I use member variables for argb, 
   // so I don't have to allocate and return a byte array in each call
   // Ugly, but much faster than return new byte[]{ }

   // Red
   double d = (xn - C1.X) * (xn - C1.X) + (yn - C1.Y) * (yn - C1.Y);
   r = (byte)((Math.Sin(d * Frequency)) > 0 ? 0 : 255);

   // Blue
   d = (xn - C2.X) * (xn - C2.X) + (yn - C2.Y) * (yn - C2.Y);
   b = (byte)((Math.Sin(d * Frequency)) > 0 ? 0 : 255);

   // Green fills the gaps
   g = (byte)~(r | b);
}

The calculation uses normalized texture coordinates and the circle centers are computed each frame and stored in the Points C1 and C2:

private void CalculateCenters()
{
   // Nice sine circle movement
   // Use normalized coordinates
   C1.X = Math.Sin(framesCount * 0.02) * Half + Quarter;
   C1.Y = Math.Sin(framesCount * 0.08) * Half + Quarter;
   C2.X = Math.Sin(framesCount * 0.10) * Half + Quarter;
   C2.Y = Math.Sin(framesCount * 0.04) * Half + Quarter;
}

The normalization of the center coordinates makes it easier to use the same values directly as parameters for the pixel shader:

// Parameters
float2 C1 : register(C0);
float2 C2 : register(C1);
float Frequency : register(C2);

// Shader
float4 main(float2 p : TEXCOORD) : COLOR
{
   // Overlayed sine circle rings
   float4 color = 1;

   // Red
   float2 dist = uv - C1; 
   color.r = sin(dot(dist, dist) * Frequency) > 0 ? 0 : 1;

   // Blue
   dist = uv - C2; 
   color.b = sin(dot(dist, dist) * Frequency) > 0 ? 0 : 1;

   // Green fills the gaps
   color.g = color.r + color.b > 0 ? 0 : 1;
   return color;
}

Results

The results differ a bit from the first article, which is caused by the other algorithm that is used here. There are other color distributions and thus results in a slightly different drawing. The Silverlight rendering has a great impact on the performance and unfortunately Silverlight doesn't ship with the .Net Stopwatch class and the only way to get suitable data, is the measuring of larger code blocks, including the drawing, with the imprecise DateTime struct.
The Quakelight BitmapData is almost as fast as the Silverlight 3 WriteableBitmap, but at the cost of the 256 color limitation. Although the pixel shader is not executed on the GPU, it still runs ultra-fast compared to the other competitors.
If you want to implement a procedural image generation technique I recommend to try a pixel shader, but keep in mind that Silverlight 3 only supports the limited Shader Model 2. I noticed that the Silverlight pixel shaders seem to use SEE and are automatically executed in parallel if they run on a multi-core processor. The framerate on a dual-core machine was twice as high as on a single-core CPU. This parallel software shader implementaion in Silverlight is actually the only right way to implement them and nothing special. Shaders are designed for parallel execution on the GPU.

Source code
Download the C# code and the pixel shader from here.

Update 08-10-2009
Justin Harrell used my Speedtest source code and added tests for the Silverlight 3 MediaSource API. He contacted my via Email and attached the source code:

Hello

I have been looking into Silverlight Graphics performance myself and was interested in the new SL3 features. So I read your blog entry for WriteableBitmap performance and was also interested in the new MediaSource managed codec abilities, which looked like another way to get pixels to the screen, but also audio as well which could be interesting for games etc.
So I took your source for the test sample and added two MediaSource tests, one using single threaded frame generation, the second using a background thread and a double buffer based on work from Pete Brown at on his Commodore 64 emulator in Silverlight.
I also reworked the UI with checkboxes to turn on/off tests as well as adding 3 new types of tests beyond the Circle interference to test performance of the rendering method vs the pixel generation. These include a random noise, simple scrolling line, and a no op. I also refactored some to make it easier to add new test types. I did not implement shader based versions of my tests, so if you run the shader test on anything but circle interference its basically a no op for now J.
I thought it might be an interesting addition to the sample as yet another way to generate dynamic bitmaps, the mediasource performs very well, although still behind WriteableBitmap.
Note for the non-double buffered version of MediaSource there is a padTime that can be tuned, it is currently set to 10ms and can be lowered to improve frame rate, but below a certain time it will get too short based on how faster your computer is and cause the media player to think it is losing frames and start skipping badly. I haven’t figured out a good way to compensate for this automatically yet, it has to do with how long the video render takes in addition to the pixel generation. The double buffer does not have this issue as it is a fixed frame rate set by frameTime.
I have attached the sample to this email, let me know what you think, and thanks for the great blog posts.

Justin Harrell

Thanks Justin for sharing your additions. That's why I <3 open source.

Sharp Edge - Silverlight Parametric Pixel Shader

One great addition to Silverlight 3 are pixel shaders. The Blur and the Drop Shadow shaders are bundled with Silverlight 3, but it's also possible to attach custom shaders to any UI Element. Unfortunately shaders are not executed on the GPU by Silverlight, but the Software implementation is pretty fast.
There are many cool effects, which are not already part of the WPF Pixel Shader Effects Library, one could implement. So only the sky is the limit - and the Shader Model 2.0 instruction count limit of course.

Silverlight pixel shaders could be written in HLSL and compiled with the DirectX shader compiler fxc. The produced binary file is then loaded with the Silverlight 3 PixelShader class. Quite easy huh? I must admit that I'm not a complete newbie to shader development. I even wrote some shaders with Shader Model 1.1 in the assembler shading language a few years ago, but haven't done it a while.
For Silverlight 3 I've implemented an edge detection post processing effect. It's a parametric pixel shader, which performs a common image processing technique called convolution.

Live

The initial image "Lenna" is a famous test picture for image processing algorithms. Daniel Collin (@daniel_collin) pointed me on the interesting story behind that picture of Lena Söderberg. Although Lena is a pretty lady, you should try another image too.
You can control the threshold of the edge detection with the Slider and disable the shader with the "Bypass" CheckBox. Select one of three preset convolution operators from the ComboBox: Scharr, Prewitt and the well-known Sobel operator. It's also possible to change the first column of the Gx convolution kernel to try your own operator. Actually two 3x3 convolution kernels are used by the algorithm. One for the horizontal (Gx) and another one for the vertical (Gy) direction. The Gy is just a 90° rotation of Gx and the last column of Gx is the inverse of the first column. The middle column is zero. So instead of 18 parameters (3x3x2) only 3 parameters need to be passed to the shader.

How it works
I used the Shazzam Tool for the shader development. It's a nice tool to write and test shaders for WPF. It also generates a corresponding class for the pixel shader, which is then used in XAML. Most of the controls take advantage of Silverlight's 3 great (Element) Data binding mechanism.
There's really nothing more to write about the Silverlight application, all the magic happens in the pixel shader:

// Parameters
float Threshhold : register(C0);
float K00 : register(C1); // Kernel first column top
float K01 : register(C2); // Kernel first column middle 
float K02 : register(C3); // Kernel first column bottom
float2 TextureSize : register(C4);

// Static Vars
static float ThreshholdSq = Threshhold * Threshhold;
static float2 TextureSizeInv = 1.0 / TextureSize;
static float K20 = -K00; // Kernel last column top
static float K21 = -K01; // Kernel last column middle
static float K22 = -K02; // Kernel last column bottom
sampler2D TexSampler : register(S0);

// Shader
float4 main(float2 uv : TEXCOORD) : COLOR
{
   // Calculate pixel offsets
   float2 offX = float2(TextureSizeInv.x, 0);
   float2 offY = float2(0, TextureSizeInv.y);

   // Sample texture
   // Top row
   float2 texCoord = uv - offY;
   float4 c00 = tex2D(TexSampler, texCoord - offX);
   float4 c01 = tex2D(TexSampler, texCoord);
   float4 c02 = tex2D(TexSampler, texCoord + offX);

   // Middle row
   texCoord = uv;
   float4 c10 = tex2D(TexSampler, texCoord - offX);
   float4 c12 = tex2D(TexSampler, texCoord + offX);

   // Bottom row
   texCoord = uv + offY;
   float4 c20 = tex2D(TexSampler, texCoord - offX);
   float4 c21 = tex2D(TexSampler, texCoord);
   float4 c22 = tex2D(TexSampler, texCoord + offX);

   // Convolution
   float4 sx = 0;
   float4 sy = 0;

   // Convolute X
   sx += c00 * K00;
   sx += c01 * K01;
   sx += c02 * K02;
   sx += c20 * K20;
   sx += c21 * K21;
   sx += c22 * K22; 

   // Convolute Y
   sy += c00 * K00;
   sy += c02 * K20;
   sy += c10 * K01;
   sy += c12 * K21;
   sy += c20 * K02;
   sy += c22 * K22;     

   // Add and apply Threshold
   float4 s = sx * sx + sy * sy;
   float4 edge = 1;
   edge =  1 - float4( s.r <= ThreshholdSq,
                        s.g <= ThreshholdSq,
                        s.b <= ThreshholdSq, 
                        0); // Alpha is always 1!
   return edge;
}

The colors of the current pixel's neighbors are sampled from the image, which are then multiplied with the corresponding kernel value. The results are summed up, the threshold is applied and the new color is returned. I optimized the operation a bit more by calculating some static variables, using the squared threshold and leaving the calculation for the zero kernel column / row out.

Source code
The Visual Studio 2008 solution including the pixel shader is available from here.

Update 12-30-2009
See this blog post if you want to try the shader in real time with your own webcam.

Razor photo by Jake Sutton