Thursday, July 30, 2009

Sharp Edge - Silverlight Parametric Pixel Shader

One great addition to Silverlight 3 are pixel shaders. The Blur and the Drop Shadow shaders are bundled with Silverlight 3, but it's also possible to attach custom shaders to any UI Element. Unfortunately shaders are not executed on the GPU by Silverlight, but the Software implementation is pretty fast.
There are many cool effects, which are not already part of the WPF Pixel Shader Effects Library, one could implement. So only the sky is the limit - and the Shader Model 2.0 instruction count limit of course.

Silverlight pixel shaders could be written in HLSL and compiled with the DirectX shader compiler fxc. The produced binary file is then loaded with the Silverlight 3 PixelShader class. Quite easy huh? I must admit that I'm not a complete newbie to shader development. I even wrote some shaders with Shader Model 1.1 in the assembler shading language a few years ago, but haven't done it a while.
For Silverlight 3 I've implemented an edge detection post processing effect. It's a parametric pixel shader, which performs a common image processing technique called convolution.

Live

The initial image "Lenna" is a famous test picture for image processing algorithms. Daniel Collin (@daniel_collin) pointed me on the interesting story behind that picture of Lena Söderberg. Although Lena is a pretty lady, you should try another image too.
You can control the threshold of the edge detection with the Slider and disable the shader with the "Bypass" CheckBox. Select one of three preset convolution operators from the ComboBox: Scharr, Prewitt and the well-known Sobel operator. It's also possible to change the first column of the Gx convolution kernel to try your own operator. Actually two 3x3 convolution kernels are used by the algorithm. One for the horizontal (Gx) and another one for the vertical (Gy) direction. The Gy is just a 90° rotation of Gx and the last column of Gx is the inverse of the first column. The middle column is zero. So instead of 18 parameters (3x3x2) only 3 parameters need to be passed to the shader.

How it works
I used the Shazzam Tool for the shader development. It's a nice tool to write and test shaders for WPF. It also generates a corresponding class for the pixel shader, which is then used in XAML. Most of the controls take advantage of Silverlight's 3 great (Element) Data binding mechanism.
There's really nothing more to write about the Silverlight application, all the magic happens in the pixel shader:
// Parameters
float Threshhold : register(C0);
float K00 : register(C1); // Kernel first column top
float K01 : register(C2); // Kernel first column middle 
float K02 : register(C3); // Kernel first column bottom
float2 TextureSize : register(C4);

// Static Vars
static float ThreshholdSq = Threshhold * Threshhold;
static float2 TextureSizeInv = 1.0 / TextureSize;
static float K20 = -K00; // Kernel last column top
static float K21 = -K01; // Kernel last column middle
static float K22 = -K02; // Kernel last column bottom
sampler2D TexSampler : register(S0);

// Shader
float4 main(float2 uv : TEXCOORD) : COLOR
{
   // Calculate pixel offsets
   float2 offX = float2(TextureSizeInv.x, 0);
   float2 offY = float2(0, TextureSizeInv.y);

   // Sample texture
   // Top row
   float2 texCoord = uv - offY;
   float4 c00 = tex2D(TexSampler, texCoord - offX);
   float4 c01 = tex2D(TexSampler, texCoord);
   float4 c02 = tex2D(TexSampler, texCoord + offX);

   // Middle row
   texCoord = uv;
   float4 c10 = tex2D(TexSampler, texCoord - offX);
   float4 c12 = tex2D(TexSampler, texCoord + offX);

   // Bottom row
   texCoord = uv + offY;
   float4 c20 = tex2D(TexSampler, texCoord - offX);
   float4 c21 = tex2D(TexSampler, texCoord);
   float4 c22 = tex2D(TexSampler, texCoord + offX);

   // Convolution
   float4 sx = 0;
   float4 sy = 0;

   // Convolute X
   sx += c00 * K00;
   sx += c01 * K01;
   sx += c02 * K02;
   sx += c20 * K20;
   sx += c21 * K21;
   sx += c22 * K22; 

   // Convolute Y
   sy += c00 * K00;
   sy += c02 * K20;
   sy += c10 * K01;
   sy += c12 * K21;
   sy += c20 * K02;
   sy += c22 * K22;     

   // Add and apply Threshold
   float4 s = sx * sx + sy * sy;
   float4 edge = 1;
   edge =  1 - float4( s.r <= ThreshholdSq,
                        s.g <= ThreshholdSq,
                        s.b <= ThreshholdSq, 
                        0); // Alpha is always 1!
   return edge;
}
The colors of the current pixel's neighbors are sampled from the image, which are then multiplied with the corresponding kernel value. The results are summed up, the threshold is applied and the new color is returned. I optimized the operation a bit more by calculating some static variables, using the squared threshold and leaving the calculation for the zero kernel column / row out.

Source code
The Visual Studio 2008 solution including the pixel shader is available from here.

Update 12-30-2009
See this blog post if you want to try the shader in real time with your own webcam.

Razor photo by Jake Sutton

Friday, July 24, 2009

Silverlight 3 WriteableBitmap Performance

Having a fast dynamic bitmap generation API at hand is essential for procedural image generation and a lot of computer games. Therefore many Silverlight developers were disappointed that WPFs WriteableBitmap wasn't available before Silverlight 3. Fortunately there was the BitmapImage.SetSource method, which uses a Stream as parameter for the bitmap source and could be used to fill an Image. I think Joe Stegman was the first who wrote a custom PNG Genertor Stream, which used this Stream mechanism and made it possible to generate procedural images with Silverlight 2. Other implementations followed.

Now that we have Silverlight 3 and the WriteableBitmap class, all these custom PNG Stream implementations became obsolete. There are still some developers, who complain about the performance of the WriteableBitmap. I was curious how the custom PNG Stream implementations compete with the WriteableBitmap and how big the speed difference really is. That's why I wrote a small Silverlight 3 application, which measures the frames per second of the custom PNG Stream implementations and the Silverlight 3 WriteableBitmap.

The Competitors
  1. Silverlight 3 WriteableBitmap.
  2. RawPngBufferStream from the open source GameEngine Balder, which I used for my Perlin Noise sample.
  3. Nikola's PngEncoder, which is an improved version of Joe Stegman's work.
  4. Ian Griffiths' SlDynamicBitmap library.

Live

The application measures the time, which every implementation needs to draw the "Maximum Frames" and calculates the mean frames per seconds (fps). The third text column shows the relative performance compared to the WriteableBitmap.
If the tests complete very fast, you should increase the "Maximum Frames" to get right results.

How it works
The Image has the size 512 x 512. One after another every drawing method is executed and the time is measured. The method CalculateColor(int x, int y) computes the color for every pixel. For that I implemented a nice old school demoscene effect, which produces an interference image. I was inspired by the brilliant Amiga demo State of the Art from 1992.

private void CalculateColor(int x, int y)
{
   // Nice sine circle movement. 
   int x1 = (int)(sin[FramesCount * 1] * TexSizeHalf) + TexSizeQuarter;
   int y1 = (int)(sin[FramesCount * 4] * TexSizeHalf) + TexSizeQuarter;
   int x2 = (int)(sin[FramesCount * 5] * TexSizeHalf) + TexSizeQuarter;
   int y2 = (int)(sin[FramesCount * 2] * TexSizeHalf) + TexSizeQuarter;

   // Clamped Euclidean distance as color
   // Change the multiplication Factor to get more circles 
   // Change the clamping Threshold for the space between
   int d = (x - x1) * (x - x1) + (y - y1) * (y - y1);
   r = (byte)((byte)Math.Sqrt(d << Factor) > Threshold ? 255 : 0);
   d = (x - x2) * (x - x2) + (y - y2) * (y - y2);
   b = (byte)((byte)Math.Sqrt(d << Factor) > Threshold ? 255 : 0);
   // Fill the gaps with green
   g = (byte)~(r | b);
}

The circles center position is animated with a sine function. For better performance I use a pre calculated lookup table (LUT) here. The rings are built using the clamped Euclidian distance to the center. Actually every Math.Sqrt() produces one colored circle, which is cut into rings by the shifting and clamping. Of course this could also be done with some loops and sine / cosine or other techniques. The square roots are calculated on the fly and not stored in a LUT. Otherwise the calculation would be too fast and not representing a real use case.
The rest of the implementation is quite simple and there's not much to explain. If you are interested in the details, please look at the source code or write a comment.

Results

The WriteableBitmap is obviously the fastest implementation. Actually I haven't expected anything else, but I hoped it would be a bit faster. Nevertheless, the Silverlight 3 WriteableBitmap is almost twice as fast as the SlDynamicBitmap library and Balder's RawPngBufferStream.
Please consider, although I use relative values, you might encounter some slightly different test results. Depending on the used hardware each implementation could perform better or worse.

Source code
The Visual Studio 2008 solution of the Speedtest application is available for download from here.

Update 08-06-2009
I've written a follow-up to this article and included the Quakelight PNG implementation and a custom pixel shader.

Thursday, July 16, 2009

WriteableBitmap Extension Methods

Bill Reiss (@billreiss) wrote a good blog post about the pixel format of the Silverlight 3 WriteableBitmap class and included some nice helper methods for the pixel manipulation. I used his two methods, optimized them a bit and packed them into a static class as extension methods of the WriteableBitmap. I also added some methods, which take the System.Windows.Media.Color structure as input parameter instead of bytes.
The SetPixeli overloads of the methods  use the precalculated index and don't calculate the index position itself in every call and are therefore faster.


The signature of the methods
SetPixeli(this WriteableBitmap bmp, int index, byte r, byte g, byte b);
SetPixel(this WriteableBitmap bmp, int x, int y, byte r, byte g, byte b);

SetPixeli(this WriteableBitmap bmp, int index, byte a, byte r, byte g, byte b);
SetPixel(this WriteableBitmap bmp, int x, int y, byte a, byte r, byte g, byte b);

SetPixeli(this WriteableBitmap bmp, int index, Color color);
SetPixel(this WriteableBitmap bmp, int x, int y, Color color);

SetPixeli(this WriteableBitmap bmp, int index, byte a, Color color);
SetPixel(this WriteableBitmap bmp, int x, int y, byte a, Color color);  

Usage
int index = 0;
for (int y = 0; y < writeableBmp.PixelHeight; y++)
{
   for (int x = 0; x < writeableBmp.PixelWidth; x++)            
   {
      byte alpha = (byte)(x * x + y * y);
      writeableBmp.SetPixeli(index++, alpha, Colors.Black);
   }
}
writeableBmp.Invalidate();
The index position isn't calculated with x * writeableBmp.PixelWidth + y in every iteration, instead I use an extra index variable. The incrementation is a lightweight computational operation compared to the multiplication + addition.
This short code snippet produces the image you can see above. It's really simple algorithmic beauty from the alpha channel.

Source code
You can download the complete Visual Studio 2008 solution from here. Check out my Codeplex project WriteableBitmapEx for an up to date version of the extension methods.

Wednesday, July 8, 2009

Summer Silverlight Coding Competition


ComponentArt is hosting the "Summer Silverlight Coding Competition" from June 22nd to Semptember 22nd:

ComponentArt is excited to host the 2009 Summer Silverlight Coding Competition and award $10,000 USD to the author of the best application as selected by our expert panel and the community. Authors of the two runner-up applications will each receive ComponentArt licenses (a $1,299 USD value).


I submitted my Silverlight website and I am very happy that it was published yesterday. The community voting is now open. If you like my Silverlight real-time physics website it would be nice if you vote for it here. Unfortunately you need a ComponentArt.com account to cast your vote, but the registration is quickly done.