Friday, December 20, 2013

What are you Awaiting for - How to Turn a C/C++ Callback into a C++/Cx WinRT IAsyncAction

Windows and Windows Phone 8 and its WinRT / WinPRT component model provide a nice way to leverage good ol' native C/C++ libraries in all WinMD language projections by wrapping them in a WinRT C++/Cx component. Such a native WinRT component only exposes WinRT types and can therefore be consumed through the WinMD projections also in C# or WinJS.
Often those good ol' native libraries provide events or callback functions for asynchronous calls. This means one would also have to provide a WinRT wrapper for those events and callbacks as well, although the WinRT has the nice IAsyncAction and IAsyncOperation constructs together with the Parallel Patterns Library (PPL) which make the WinRT component way better to use.
This post describes how to plug both worlds together to turn your good ol' C++ code into rose colored C++/Cx consumable through WinMD.

We won't go too deep into C++/Cx in general here. There's a good intro at the MSDN and also a nice Quick Reference which provides a table how Standard C++ types and constructs compare to C++/Cx. I also don't cover how the PPL and WinRT async works here, but there's a great MSDN page: Creating Asynchronous Operations in C++ for Windows Store Apps.

Assuming you have a good ol' native interface like this with a defined callback template:
template <typename TCallback>
HRESULT StartAsyncOperation(const TCallback &callback)

It could be used like this using a C++11 lambda as callback function:
StartAsyncOperation([this](HRESULT error) -> HRESULT
   // Do whatever here after the async is done...
   // Notify the calling code through an event or another callback 
   // that the async operation has finished.
   return S_OK;

It's worth mentioning this will also work with C function pointers and you can use a C++ 11 lambda for it as long as the lambda does not capture any variable. Stateless lambdas will just degrade to function pointers.
// Definition of fucntion pointer and a function expecting it as parameter
typedef void (*SomeOtherCallback)(int param1);

FunctionExpectingSomeOtherCallback(SomeOtherCallback cb);

// Example call using a stateless lambda
FunctionExpectingSomeOtherCallback([](int param1)
   // Perform some operation with param1

This is how your C++/Cx WinRT component's method which wraps the above C++ template could look like:
Windows::Foundation::IAsyncAction^ MyWinRtComponent::RunAsync()
 // Setup  the Task Completion Event and the async Task
 Concurrency::task_completion_event<void> tce;
 auto tsk = Concurrency::create_task(tce);
 Windows::Foundation::IAsyncAction^ asyncOp = Concurrency::create_async( [tsk]() 
            -> Concurrency::task<void> 
  return tsk;

 // Run good ol' native code wih callback defined as C++ lambda
 StartAsyncOperation([this, tce](HRESULT error) -> HRESULT
   // Check for error and wrap in platfrom exception
   if (FAILED(error))
    throw Platform::Exception::CreateException(error);
   // Pass exception on to calling code using the 
   // Task Completion Event's dedicated method
   return error;

  // Set the Task Completion Event's Result to mark the Task as complete
  // If a IAsyncOperation is used the result value is passed

  return S_OK;

 return asyncOp;

You could then use your WinRT component in C# like this:
var component = new MyWinRtComponent();
await component.RunAsync();

And similar in WinJS using a JS Promise:
this._component.runAsync().then(function () {
 WinJS.log("My component finished the async call! :-)", null, "status");

Way better API for the custom WinRT component if you ask me!
The trick to achieve that is the usage of the PPL's create_async, create_task and especially the task_completion_event which provides a nice way to make that callback-based model work with the proper WinRT modern async Task-based way.

How to Encode and Save a WriteableBitmap with Windows WinRT

Windows WinRT 8.1 brought the support of the RenderTargetBitmap and the ability to render the visual tree of the UI into a bitmap. WriteableBitmapEx makes it even easier in the latest version and you might end up in a situation where you want to save a WriteableBitmap to a file or to a stream in general. For that the raw pixel array of the WriteableBitmap needs to be encoded into a common format like JPEG, PNG, etc.

Here's a code snippet how this can be achieved:

private static async Task SaveWriteableBitmapAsJpeg(WriteableBitmap bmp, string fileName)
   // Create file in Pictures library and write jpeg to it
   var outputFile = await KnownFolders.PicturesLibrary.CreateFileAsync(fileName, CreationCollisionOption.ReplaceExisting);
   using (var writeStream = await outputFile.OpenAsync(FileAccessMode.ReadWrite))
      await EncodeWriteableBitmap(bmp, writeStream, BitmapEncoder.JpegEncoderId);
   return outputFile;

private static async Task EncodeWriteableBitmap(WriteableBitmap bmp, IRandomAccessStream writeStream, Guid encoderId)
   // Copy buffer to pixels
   byte[] pixels;
   using (var stream = bmp.PixelBuffer.AsStream())
      pixels = new byte[(uint) stream.Length];
      await stream.ReadAsync(pixels, 0, pixels.Length);

   // Encode pixels into stream
   var encoder = await BitmapEncoder.CreateAsync(encoderId, writeStream);
   encoder.SetPixelData(BitmapPixelFormat.Bgra8, BitmapAlphaMode.Premultiplied,
      (uint) bmp.PixelWidth, (uint) bmp.PixelHeight,
      96, 96, pixels);
   await encoder.FlushAsync();

It's all  pretty straightforward actually: The SaveWriteableBitmapAsJpeg creates a file in the user's pictures library and passes the file's stream and an BitmapEncoder ID on to the EncodeWriteableBitmap method which is doing the actual work. EncodeWriteableBitmap first copies the WriteableBitmap pixel buffer into a byte array, then creates the BitmapEncoder based on the file's stream and the desired BitmapEncoder ID, then sets the pixels and flushes all into the stream.
The BitmapEncoder class supports a couple of BitmapEncoder IDs out of the box like JPEG, PNG and more.

Monday, December 2, 2013

Easy Render! - WriteableBitmapEx now with Better Support for Win 8.1 RenderTargetBitmap

The lack of WriteableBitmap.Render in Windows WinRT 8.0 was quite an issue for many apps, but fortunately did WinRT 8.1 (re-)introduce the RenderTargetBitmap class which provides the functionality of rendering the visual tree of the UI to a bitmap. The new update of WriteableBitmapEx makes it even easier to use with a WriteableBitmap by introducing the FromPixelBuffer method.

WriteableBitmapEx is available for 4 platforms: Windows Phone 8 and 7, WPF, Silverlight and Windows Store WinRT .NET XAML 8 and 8.1.
You can download the binaries here or via the NuGet packageThe packages contain the WriteableBitmapEx binaries. As usual all samples and the source code can be found in the repository.

How to use
The below code snippet shows how a part of the UI can be rendered into a WriteableBitmap via RenderTargetBitmap, how it can be modified and then finally presented back into the UI by using an Image control.

// Render some UI to a RenderTargetBitmap
var renderTargetBitmap = new RenderTargetBitmap();
await renderTargetBitmap.RenderAsync(Panel);
// Get the pixel buffer and copy it into a WriteableBitmap
var pixelBuffer = await renderTargetBitmap.GetPixelsAsync();
var width = renderTargetBitmap.PixelWidth;
var height = renderTargetBitmap.PixelHeight;
var wbmp = await new WriteableBitmap(1, 1).FromPixelBuffer(pixelBuffer, width, height);

// Modify the WriteableBitmap
wbmp.FillEllipse(10, 10, 256, 256, Colors.Salmon);
// ... More drawing including writeableBitmap.Blit(...) might go here

// Assign WriteableBitmap to an Image control to present it
ImgMirror.Source = wbmp;

Quick! - How to Open WP Settings from Code

The Windows Phone File and Uri associations are very useful for implementing custom cross-app functionality like Pictures Lab and other apps provide. Windows Phone also provides uri schemes for many built-in apps and settings which can be very handy.
The Windows Phone 8 update GDR3 introduced a new setting to lock the screen rotation. This setting is quite useful especially when reading in bed or for providing on-screen QR codes to scanners.
This useful screen rotation lock is unfortunately hidden in the settings and it takes a few taps to access it. Therefore I've written a tiny free app called Screen Rotation Lock. It provides quick access to the screen rotation settings through a pinned secondary tile.

The relevant code in the MainPage is rather straight forward but there are some details which are worth sharing and which apply to any shortcut uri navigation:

protected override void OnNavigatedTo(NavigationEventArgs e)

   // Actual shortcut navigation?
   if (NavigationContext.QueryString.ContainsKey(UriQueryStringKeyPage))
      // If coming from setting than skip here
      if (e.NavigationMode == NavigationMode.Back)
      // Otherwise go to setting
      Windows.System.Launcher.LaunchUriAsync(new Uri("ms-settings-screenrotation:"));

   //Check for support
   var gdr3Version = new Version(8, 0, 10492);
   if (Environment.OSVersion.Version < gdr3Version)
      MessageBox.Show(String.Format("Sorry, your phone does not support the screen rotation lock feature yet. You need to have the Windows Phone version {0} installed  (WP 8 GDR 3).\r\nThe app will now exit.", gdr3Version), "Not supported version", MessageBoxButton.OK);

The first if statement tests if the navigation comes from the secondary tile with the associated key UriQueryStringKeyPage. If it's a back navigation, the user was actually forwarded already to the setting and tapped back. In that case we don't want the user to land in our app, but rather go back to the start screen where she/he comes from, so we close our app. If it's not a back navigation the forward to the setting takes place using the Launcher API and the LaunchUriAsync method with the right uri for screen rotation.
The last if statement tests the version of the device and shows an error message if the device does not have GDR3 installed. This is done as last statement since it's actually only needed when the user wants to pin the secondary tile shortcut and opens the app itself.

Thursday, July 4, 2013

Just call us, we'll might call you - How to add Pictures Lab to your App

Windows Phone 8 introduced a nice new feature called File and Uri associations. This feature makes it possible to register an app for certain file extensions and Uri protocols. A couple of apps already support it and Nokia has a nice site which lists a couple.

Pictures Lab recently added support for that as well and uses it to call the app Instagraph. This means with the help of that app Pictures Lab now provides posting / sharing on Instagram. But that's not all, Pictures Lab also supports inbound Uri associations, so you can pass an image to Pictures Lab through the mechanism of an Uri association pretty easily.

How it works
In order to call Pictures Lab from an app this scheme has to be used:

  • fileNameinthemedialib:
    This is the filename of the image in the Media library / Pictures Hub. It does not matter if the file extension like .jpg is provided or not. 
  • ClientId:
    This is an optional parameter. If it's provided then it should be the GUID that also matches the store product Id. It's not used in Pictures Lab now, but might be useful in the future to know the source of the protocol call and one could also use it to build a link like that:

or to just launch Pictures Lab without passing an image:

Sample code:

var photoChooser = new PhotoChooserTask();
photoChooser.Completed += async (sender, e) =>
   // When the user selected a photo, pass the filename on to Pictures Lab for editing
   var uri = string.Format("images:edit?Name={0}&Clientid={1}", 
   // uri = "images:edit?Name=WP_20130527_009&Clientid=9902dac1-c001-440b-add5-f38446b35676" 
   var wasSuccess = await Windows.System.Launcher.LaunchUriAsync(new System.Uri(uri));

Note, it's not necessary to UrlEncode the Uri, the Windows Phone platform handles it automatically. If the user has Pictures Lab not installed and the above Uri is called, the Windows Phone OS is also smart enough to show the user a dialog which asks to search for an app in the Store that can handle the protocol. This will bring up Pictures Lab.

The concept of Uri associations is like the little brother of Windows 8's Share Charm, but it's still quite powerful and gives us app developers a nice way to combine our apps and make it a seamless UX. It's a win for the user and therefore it could be a win-win situation for the involved apps.

For a next version I'm also thinking about a callback scenario. The user story would be: A user opens app A and wants to edit an image, this will open Pictures Lab, when the user is done with editing, Pictures Lab will call app A again. The user is back in app A with the edited image.
This callback mechanism would probably use a Callback parameter in the scheme above where app A can provide the UrlEncoded callback Uri with a placeholder for the edited pictureName. Something like that:

The Url decoded part of the Callback: myapp:post?Name={0}
This callback format string will be used in Pictures Lab to call app A and pass the edited picture back.

Makes sense?
Please let me know your thoughts and if that would be useful. Feel free to comment or reach out to me through any channel.

Tuesday, March 19, 2013

Kinect and Leap Motion in Love

Natural User Interfaces are a trend since a few years and the development of new hardware technology for affordable prices helps to push that movement into the consumer market. The Kinect is one example, the soon-to-be-available Leap Motion is another one or the announced MYO device.

I'm one of the lucky people who applied and actually got a Leap Motion developer device before its release. I also have a Kinect for Windows. Both devices use computer vision with various sensors combined with artificial intelligence like machine learning for the object recognition. The Kinect for Windows recognizes skeletons with joints in a distance of  80 - 400 cm or 40 - 300 cm (Near Mode) which means the Kinect is perfect for the mid range distance but not for close PC interaction. The Leap Motion on the other hand is more for the close range and detects hands, fingers and pen-like objects very precisely. That's why I wanted to combine them and build a proof of concept hybrid solution that leverages both Kinect and Leap Motion to get the best of both worlds (close and mid distance).

In the video below I demo my proof of concept which is based on the Kinect for Windows SDK 1.7 and its Interactions and Controls components, plus the Leap Motion developer device and its beta SDK.
By the way, the Kinect for Windows SDK 1.7 is now publicly available and the bits are nothing less than awesome.

In case you are wondering about the infrared interference of the Kinect and Leap Motion. There's some, but I actually didn't notice any serious effect on the recognition capabilities of the Kinect nor the Leap.
Below is a picture of the Leap Motion as seen through the Kinect's infrared stream. The Leap Motion has three light sources but those are not very bright.

Some might say the Leap Motion is a competitor to the Kinect for Windows, I see it more like a nice addition to our developer toolset and the beginning of a story: Kinect ♥ Leap Motion.