Showing posts with label JPEG. Show all posts
Showing posts with label JPEG. Show all posts

Thursday, December 9, 2010

Issue with the WP7 PictureDecoder and Workaround

This is fixed in the WP 7.1 SDK /  WP 7.5 / Mango!
I noticed a strange behavior of the Windows Phone PictureDecoder DecodeJpeg method while I was working on my Pictures Lab app.
This short post describes the issue I encountered and also provides a workaround.
The built-in DecodeJpeg method decodes a JPEG image stream into a WriteableBitmap. The method has two overloads, where the first only takes the JPEG stream and the second also uses parameters for the maximum width and height of the output. I encountered a strange behavior of the latter method.

Issue
The DecodeJpeg method with 3 parameters swaps the width and the height parameters when a landscape photo should be resized. For example, an image with the original size of 3264 x 2448 should be decoded to a WriteableBitmap and resized to 1024 x 768. But the resulting output will have a size of 768 x 576 cause the method swapped the input parameters while preserving the correct aspect ratio.
Please note that this only happens with landscape pictures, which means the width of the image is greater than the height.

Will it get fixed?
Yes! I contacted Microsoft and they confirmed this issue and said that it will get fixed in a future version of the Windows Phone Silverlight runtime.

Workaround
Isolating the cause of this strange behavior took a bit of time, fortunately the easier was the obvious workaround.

int w = desiredOutputWidth;
int h = desiredOutputHeight;

// Workaround for issue in DecodeJpeg method:
// Swap width and height for landscape pictures.
if (originalWidth > originalHeight)
{
   w ^= h;
   h ^= w;
   w ^= h;
}
var writeableBitmap = PictureDecoder.DecodeJpeg(jpegStream, w, h);
The code tests if the original width is greater than the height and then swaps the output width and height using the good old XOR swap trick.
Alternatively you can avoid the DecodeJpeg overload with the resizing functionality, decode the full sized image and use the WriteableBitmapEx Resize method afterwards. It produces similar results when bilinear interpolation is used, but it's an extra step that costs resources.

The above photo was taken with a HTC Mozart and edited with Pictures Lab (cropped, rotated and 1989 vintage effect).

Monday, July 19, 2010

Photos, Photos, Photos - How To Save, Load And Iterate Pictures With Windows Phone 7

The Windows Phone developer tools beta version was recently released and brought some important changes and also new features. For example the emulator is a lot better now and tombstoning was introduced. Tombstoning is an important concept to know when a Launcher or Chooser is being used in an app.
But this blog post is not about tombstoning, this post will show how to load an image from the picture library with the PhotoChooserTask. And most important how to save a picture to the library, which is not as obvious as the usage of the Task API. The last part will show how to get all images from the picture library without using the PhotoChooserTask.

Sample
I've built a Windows Phone 7 app that uses the WriteableBitmapEx library and lets the user draw on the WriteableBitmap surface. The app supports multi touch for drawing and the radius of the pen can be changed with the Slider control. An image from the phone's picture library can be chosen with the folder icon button and the floppy disk button saves the current image to the picture library. When the third button (download icon) is clicked, all pictures from the phone's library are loaded without further user interaction. The trash button clears the draw surface.
The video below demonstrates the features and shows how to use the app.
It would be great if anyone could try this sample app on a real device and give me some feedback. I'm especially interested how the multi touch drawing works.


Background music is Soft Shapes by Planet Boelex.

How it works
The raw touch points are handled through the Touch.FrameReported event. For each multi touch point a circle is being drawn with the WriteableBitmapEx' FillEllipseCentered method and a color map array that provides alternating colors:
private void Draw(IList<TouchPoint> points)
{
   // Check
   if (!isManipulating)
   {
      return;
   }

   // Init some vars
   var bmp = Bitmap;
   var w = bmp.PixelWidth;
   var h = bmp.PixelHeight;
   var r = Radius;

   // Draw
   for (int i = 0; i < points.Count; i++)
   {
      var p = points[i].Position;
      if (p.X < w && p.Y < h)
      {
         bmp.FillEllipseCentered((int)p.X, (int)p.Y, r, r, 
                                 ColorMap[(i + colorBase) % ColorMap.Length]);
      }
   }

   // Show
   Present();
}


Choose a photo
The PhotoChooserTask allows the user to select an image. The code for this task is pretty straight forward with the beta of the developer tools.
A member variable of the PhotoChooserTask is instantiated in the constructor and an event handler for the Completed event is attached.
public MainPage()
{
   // ...

   // Init chooser
   photoChooserTask = new PhotoChooserTask();
   photoChooserTask.Completed += PhotoChooserTaskCompleted;
   
   // ...
}
private void PhotoChooserTaskCompleted(object sender, PhotoResult e)
{
   if (e.TaskResult == TaskResult.OK)
   {
      // Load original image and invalidate bitmap so it gets newly rendered
      var bitmapImage = new BitmapImage();
      bitmapImage.SetSource(e.ChosenPhoto);
      Viewport.Source = bitmapImage;
      bitmap = null;
   }
}

The PhotoChooserTask.Show method is called when the corresponding Application Bar button was clicked:
private void ApplicationBarIconOpenButton_Click(object sender, EventArgs e)
{
   photoChooserTask.Show();
}

The Show method launches the Windows Phone photo app. The current version of the Windows Phone operating system only allows one application to run at the same time and therefore our app gets terminated when the Chooser is started. Here's where the concept of tombstoning comes into play.
After the user selected an image or pressed the back button, the Chooser's Completed event is raised. You might have noticed the black screen in the video that appears after the choose operation. As a result of the app termination, the Visual Studio debugging session is stopped. The Windows Phone 7 emulator detected that the app was started in a debug context and the emulator now waits at the black screen for re-attaching. So just go back to Visual Studio and hit F5 (Start Debugging), then the debugger is being re-attached and the app continues. Of course it's also possible to Start Without Debugging in Visual Studio.
The Chooser's Completed event provides a PhotoResult that contains the TaskResult, the OriginalFileName and the ChosenPhoto as a Stream. If the user hasn't cancelled the operation, the stream is used as the source of a BitmapImage which is then assigned to the WriteableBitmap draw surface (Viewport). The user can now make some nice drawings on the photo.


Save a picture
After the user created his masterpiece he probably wants to save it back to the picture library / photo album. But how could this be done? There's no PhotoSaveTask available in the Silverlight SDK. Fortunately the Windows Phone's XNA MediaLibrary comes to the rescue. Only a reference to the Microsoft.Xna.Framework assembly is needed to use it in our Windows Phone Silverlight application. To make this task a bit easier I wrote some reusable extension methods for the WriteableBitmap. These are located in the file WriteableBitmapMediaLibraryExtensions.cs from the source code download. The signatures look like this:
// Saves the WriteableBitmap encoded as JPEG to the Media library.
// The quality for JPEG encoding has to be in the range 0-100, 
// where 100 is the best quality with the largest size.
void SaveToMediaLibrary(this WriteableBitmap bitmap, string name, int quality);

// Saves the WriteableBitmap encoded as JPEG to the Media library 
// using the best quality of 100.
void SaveToMediaLibrary(this WriteableBitmap bitmap, string name);

The methods use the MediaLibrary's SavePicture method internally. The SavePicture method expects a stream or a byte array as parameter that contains an image encoded in the JPEG format. For the earlier CTP versions I wrote a custom JEPG encoding functionality that used some parts of my Silverlight JPEG encoding blog post and an adapted FJCore version. Fortunately things got a bit easier with the Windows Phone Tools beta release. The Microsoft.Phone assembly now comes with two WriteableBitmap extension methods called LoadJpeg and most important SaveJpeg. The SaveJpeg method expects the targetStream, the width and height of the target, the orientation which is not used at the moment and the quality in a range from 0 to 100. The width and height parameters are useful when a scaled version of the WriteableBitmap should be saved as JPEG.

The current bitmap surface is saved as JPEG when the disk floppy button was clicked:
private void ApplicationBarIconSaveButton_Click(object sender, EventArgs e)
{
   var name = String.Format("MediaLibSample_{0:yyyy-MM-dd_hh-mm-ss-tt}.jpg",
                            DateTime.Now);
   Bitmap.SaveToMediaLibrary(name);
}


Load all pictures
The last thing I'd like to cover in this post is the Pictures property of the MediaLibrary class. This property returns a collection of Picture instances. This class provides all the necessary meta information about the picture and streams of the image and thumbnail data.
I use this in the sample code to create a mosaic of all Windows Phone media library pictures:
private void ApplicationBarIconOpenAllButton_Click(object sender, EventArgs e)
{
   using(var mediaLib = new MediaLibrary())
   {
      // Pictures also includes saved pics, 
      // mediaLib.SavedPictures returns the same collection items
      var allPics = mediaLib.Pictures;

      // Combine the pics to a single WriteableBitmap mosaic
      var bmp = Bitmap;
      bmp.Clear();
      int x = 0;
      int y = 0;
      foreach (var picture in allPics)
      {
         // Load thumbnail stream to WriteableBitmap
         var wb = new WriteableBitmap(0, 0);
         wb.SetSource(picture.GetThumbnail());

         // Blit thumbnail to background bitmap
         var w = wb.PixelWidth;
         var h = wb.PixelHeight;
         bitmap.Blit(new Rect(x, y, w, h), wb, new Rect(0, 0, w, h));
         x += w;

         // Check bounds and move to next row
         if (x >= bitmap.PixelWidth)
         {
            x = 0;
            y += h;
         }
         // Bitmap filled
         if (y >= bitmap.PixelHeight)
         {
            break;
         }
      }
   }

   Present();
}

The GetThumbnail method returns the stream of an image thumbnail sized 99 x 99 pixels. This bitmap is then combined with the surface bitmap by using the WriteableBitmapEx' Blit method.
Beside the Pictures property the MediaLibrary also has the SavedPictures property, but I encountered that the SavedPictures collection returns all pictures in the emulator and not only saved pictures like the name and documentation implies. But as a little bird told me, the SavedPictures property works as expected on a real device.
The MediaLibrary has some more members that might become interesting in the future, esp. when combined with the MediaPlayer class.


Conclusion
This blog post covered the complete workflow of loading, manipulating and saving a picture to the Windows Phone's picture library / photo album. I also showed how to load all images from the library and mentioned some gotchas.

Source code
Download the complete Visual Studio 2010 Windows Phone solution from here.

Monday, November 30, 2009

Convert, Encode And Decode Silverlight WriteableBitmap Data

In the comments of my Silverlight 4 EdgeCam Shots blog post "marcb" asked me how to convert the WriteableBitmap to a byte array to save the snapshot in a database.
I thought the answer might be also useful for others. Furthermore I will provide ready to use code for JPEG encoding and decoding of the WriteableBitmap.





Byte array conversion

Copy WriteableBitmap to ARGB byte array
public static byte[] ToByteArray(this WriteableBitmap bmp)
{
   int[] p = bmp.Pixels;
   int len = p.Length * 4;
   byte[] result = new byte[len]; // ARGB
   Buffer.BlockCopy(p, 0, result, 0, len);
   return result;
}

Copy ARGB byte array into WriteableBitmap
public static void FromByteArray(this WriteableBitmap bmp, byte[] buffer)
{
   Buffer.BlockCopy(buffer, 0, bmp.Pixels, 0, buffer.Length);
}

Usage
// Render UIElement into WriteableBitmap
WriteableBitmap bmp = new WriteableBitmap(UIElement, null);

// Copy WriteableBitmap.Pixels into byte array (format ARGB)
byte[] buffer = bmp.ToByteArray();


// Create a new WriteableBitmap with the size of the original image
WriteableBitmap bmp = new WriteableBitmap(width, height);

// Fill WriteableBitmap from byte array (format ARGB)
bmp.FromByteArray(buffer);

I will include the two methods in my WriteableBitmap extensions that I'm going to put up on Codeplex soon.


JPEG encoding and decoding
If you want to store many images or transport them over a network the needed storage size could quickly become a big problem. For example an image with the size 512 x 512 needs 1 Megabyte storage space and a 1024 x 768 image even 3 MB. A solution could be image compression using JPEG encoding and decoding. To accomplish this I've used the open source FJCore JPEG library which is distributed under the MIT License and works nicely with Silverlight.

Encode WriteableBitmap as JPEG stream
public static void EncodeJpeg(WriteableBitmap bmp, Stream destinationStream)
{
   // Init buffer in FluxJpeg format
   int w = bmp.PixelWidth;
   int h = bmp.PixelHeight;
   int[] p = bmp.Pixels;
   byte[][,] pixelsForJpeg = new byte[3][,]; // RGB colors
   pixelsForJpeg[0] = new byte[w, h];
   pixelsForJpeg[1] = new byte[w, h];
   pixelsForJpeg[2] = new byte[w, h];

   // Copy WriteableBitmap data into buffer for FluxJpeg
   int i = 0;
   for (int y = 0; y < h; y++)
   {
      for (int x = 0; x < w; x++)
      {
         int color = p[i++];
         pixelsForJpeg[0][x, y] = (byte)(color >> 16); // R
         pixelsForJpeg[1][x, y] = (byte)(color >> 8);  // G
         pixelsForJpeg[2][x, y] = (byte)(color);       // B
      }
   }

   // Encode Image as JPEG using the FluxJpeg library
   // and write to destination stream
   ColorModel cm = new ColorModel { colorspace = ColorSpace.RGB };
   FluxJpeg.Core.Image jpegImage = new FluxJpeg.Core.Image(cm, pixelsForJpeg);
   JpegEncoder encoder = new JpegEncoder(jpegImage, 95, destinationStream);
   encoder.Encode();
}

Decode WriteableBitmap from JPEG stream
public static WriteableBitmap DecodeJpeg(Stream sourceStream)
{
   // Decode JPEG from stream
   var decoder = new FluxJpeg.Core.Decoder.JpegDecoder(sourceStream);
   var jpegDecoded = decoder.Decode();
   var img = jpegDecoded.Image;
   img.ChangeColorSpace(ColorSpace.RGB);

   // Init Buffer
   int w = img.Width;
   int h = img.Height;
   var result = new WriteableBitmap(w, h);
   int[] p = result.Pixels;
   byte[][,] pixelsFromJpeg = img.Raster;

   // Copy FluxJpeg buffer into WriteableBitmap
   int i = 0;
   for (int y = 0; y < h; y++)
   {
      for (int x = 0; x < w; x++)
      {
         p[i++] = (0xFF << 24)                    // A
                | (pixelsFromJpeg[0][x, y] << 16) // R
                | (pixelsFromJpeg[1][x, y] << 8)  // G
                |  pixelsFromJpeg[2][x, y];       // B
      }
   }

   return result;
}

Usage
// Save rendered UIElement as JPEG file
private void BtnSaveFile_Click(object sender, RoutedEventArgs e)
{   
   if (saveFileDlg.ShowDialog().Value)
   {
      using (Stream dstStream = saveFileDlg.OpenFile())
      {
         // Render to WriteableBitmap
         WriteableBitmap bmp = new WriteableBitmap(UIElement, null);
         
         // Encode JPEG and write to FileStream
         EncodeJpeg(bmp, dstStream);
      }
   }
}


// Open JPEG file and read into a WriteableBitmap
private void BtnLoadFile_Click(object sender, RoutedEventArgs e)
{   
   if (openFileDialog.ShowDialog().Value)
   {
      using (System.IO.Stream srcStream = openFileDialog.File.OpenRead())
      {         
         // Read JPEG file and decode it
         WriteableBitmap bmp = DecodeJpeg(srcStream);
      }
   }
}

Keep in mind that the standard JPEG format doesn't support alpha values (transparency) and that the compression is lossy. So don't encode and decode images subsequently with JPEG.
It is also possible to use the built-in Silverlight class BitmapSource and its SetSource method to decode an JPEG stream.
public static WriteableBitmap DecodeJpegWithBitmapSource(Stream sourceStream)
{
   // Decode JPEG from stream
   var bitmapSource = new BitmapSource();
   bitmapSource.SetSource(sourceStream);
   return new WriteableBitmap(bitmapSource);
}

Source code
Check out my Codeplex project WriteableBitmapEx for an up to date version of the byte array conversion methods.

Monday, November 23, 2009

EdgeCam Shots - Saving Silverlight 4 Webcam Snapshots to JPEG

In my last blog post I have covered the new Silverlight 4 Webcam API and provided a demo that used my edge detection pixel shader to create a nice real time webcam effect. In this post I make an extended version available which can save webcam snapshots as JPEG files and I also discuss some limitations of the webcam API's built-in CaptureSource.AsyncCaptureImage snapshot method. Furthermore I will give some ideas on how to build a Silverlight 4 video chat / conference application on top of the provided JPEG capturing and encoding code.


Live
To view the application you need to install the Silverlight 4 runtime. It's available for Windows and Mac.



The Webcam capturing could be started and stopped with the "Start Capture" Button. If you press it for the first time you need to give your permission for the capturing. This application uses the default Silverlight capture device. You can specify the video and audio devices that are used by default with the Silverlight Configuration. Just press the right mouse button over the application, click "Silverlight" in the context menu and select the new "Webcam / Mic" tab to set them.
Press the "Save Snapshot" Button to take a snapshot and save it to a JPEG file on your harddisk.
The threshold of the edge detection can be changed using the Slider and the "Bypass" Checkbox allows you to disable the shader.

How it works
The base Silverlight 4 webcam usage code was covered in my last blog post.

The new eventhandler for the "Save Snapshot" button:
private void BtnSnapshot_Click(object sender, RoutedEventArgs e)
{
   if (saveFileDlg.ShowDialog().Value)
   {
      using (Stream dstStream = saveFileDlg.OpenFile())
      {
         SaveSnapshot(dstStream);
      }
   }
}
The code is pretty obvious: A SaveFileDialog is shown and if the user enters a file name and hits OK, a stream to the file will be opened and passed to the SaveSnapshot method. There's only one think to keep in mind when using the SaveFileDialog.ShowDialog() method, it can only be called from user-initiated code like an event handler, otherwise a SecurityException is thrown.

The SaveSnapshot method including comments:
// Render Rectangle manually into WriteableBitmap
WriteableBitmap bmp = new WriteableBitmap(ViewportHost, null);

// Init buffer in FluxJpeg format
int w = bmp.PixelWidth;
int h = bmp.PixelHeight;
int[] p = bmp.Pixels;
byte[][,] pixelsForJpeg = new byte[3][,]; // RGB colors
pixelsForJpeg[0] = new byte[w, h];
pixelsForJpeg[1] = new byte[w, h];
pixelsForJpeg[2] = new byte[w, h];

// Copy WriteableBitmap data into buffer for FluxJpeg
int i = 0;
for (int y = 0; y < h; y++)
{
   for (int x = 0; x < w; x++)
   {
      int color = p[i++];
      pixelsForJpeg[0][x, y] = (byte)(color >> 16); // R
      pixelsForJpeg[1][x, y] = (byte)(color >> 8);  // G
      pixelsForJpeg[2][x, y] = (byte)(color);       // B
   }
}

//Encode Image as JPEG
var colModel = new ColorModel { colorspace = ColorSpace.RGB };
var jpegImage = new FluxJpeg.Core.Image(colModel, pixelsForJpeg);
var encoder = new JpegEncoder(jpegImage, 95, dstStream);
encoder.Encode();
The Rectangle's surrounding Grid "ViewportHost" is rendered into a WriteableBitmap and the WriteableBitmap's Pixels are copied into another buffer with a different format. The rendered image is then written as a JPEG encoded stream using the open source FJCore library which is distributed under the MIT License. I've found some code at Stackoverflow on how to use the library in combination with the WriteableBitmap, but I modified / shortened it.
See my post on how to Convert, Encode And Decode Silverlight WriteableBitmap Data if you are interested in more WriteableBitmap conversion and JPEG encoding code.

Why not CaptureSource.AsyncCaptureImage?
You might be wondering why I haven't used the Silverlight 4 webcam API's built-in AsyncCaptureImage snapshot method of the CaptureSource class.
  1. The AsyncCaptureImage method grabs a frame directly from the device and therefore any effects like the edge detection pixel shader won't be visible in the rendered image.
  2. It only works if the CaptureSource was started, so the user can't save the visible image when the capturing was stopped. See the MSDN for details.
Please keep in mind that the first beta version of Silverlight 4 was used for this blog post and that some things will be changed in subsequent releases.

Where to go from here
The code in the SaveSnapshot method could be optimized and also multithreaded. The rendering should run in it's own thread. The encoding in a separate thread too and the network transport also. This approach would utilize modern quad core CPUs. After that it might be a good starting point for a video chat / conferencing application that continuously renders JPEGs and transports them between Silverlight clients. This technique is similar to the M-JPEG video format that also uses separately compressed JPEGs.
To make this idea usable for a Silverlight video chat, only the rendering, the JPEG encoder and the transfer method  need to be fast enough for real time streaming.
Please leave a comment what you think about it.

Source code
Download the Visual Studio 2010 solution here.

Update 03-20-2010
Updated to the Silverlight 4 release candidate.

Update 04-15-2010
Updated to the final Silverlight 4 RTW build.