Augmented Mango - SLARToolkit for Windows Phone

The beta of the new Windows Phone Developer Tools was just publicly released. The update with the codename "Mango" comes with many new APIs and will finally contain an API for real-time camera access what a lot of developers have been asking for. The new runtime gives us the needed functionality to implement many cool scenarios. One of these scenarios is Augmented Reality, which leads to my open source Silverlight Augmented Reality Toolkit (SLARToolkit).
This post announces the new Windows Phone version of SLARToolkit and also provides a sample. If you're one of those lucky people with a Mango-enabled device you can download the XAP here or just watch a video instead.

The SLARToolkit project description from the CodePlex site:

SLARToolkit is a flexible marker-based Augmented Reality library for Silverlight and Windows Phone with the aim to make real time Augmented Reality applications with Silverlight as easy and fast as possible. It can be used with Silverlight's Webcam API or with any other CaptureSource, WriteableBitmap or with the Windows Phone's PhotoCamera. SLARTookit is based on the established NyARToolkit and ARToolkit. 

Demo

The sample XAP can be deployed to a Mango-enabled device (tested with build 7629). Alternatively there's also a video of the new sample embedded below.

If you want to try it yourself you need do download the SLAR and / or L marker, print them and point the camera toward these. The marker(s) should be printed non-scaled at the original size (80 x 80 mm) and centered for a small white border. As an alternative it's also possible to open a marker file on a different device and to use the device's screen as marker.
See the SLARToolkit Markers documentation for more details.

Video

I've recorded a short video of the new sample with my Samsung Omnia 7. It's a bit blurry, but it demonstrates how well the sample works even on this quite old ASUS prototype, which's camera pipeline seems a bit slow.
The video is also available at YouTube.

Background music is Melo by Mosaik

This demo shows how the new Windows Phone Mango real-time camera API can be used  to augment the reality with the help of the SLARToolkit. This can be nice for educational projects and it's actually no problem to add correctly transformed videos or other content to the demo.
The demo demonstrates just some basic UIElements like a TextBox and an Image control. Mango will also enable the combination of Silverlight and XNA, which means that nice 3D AR games can be developed with the help of the SLARToolkit. 

How it works

This sample uses the new PhotoCamera and a timer to constantly get a snapshot of the real-time camera stream. This snapshot is then passed to the SLARToolkit algorithms to get the 3D spatial information of the marker. The computed detection results are used to transform the elements perspectively correct.

protected override void OnNavigatedTo(System.Windows.Navigation.NavigationEventArgs e)
{
   base.OnNavigatedTo(e);

   // Initialize the webcam
   photoCamera = new PhotoCamera();
   photoCamera.Initialized += PhotoCameraInitialized;
   isInitialized = false;

   // Fill the Viewport Rectangle with the VideoBrush
   var vidBrush = new VideoBrush();
   vidBrush.SetSource(photoCamera);
   Viewport.Fill = vidBrush;

   // Start timer
   dispatcherTimer = new DispatcherTimer { Interval = TimeSpan.FromMilliseconds(50) };
   dispatcherTimer.Tick += (sender, e1) => Detect();
   dispatcherTimer.Start();
}

The PhotoCamera instance is set up in the OnNavigatedTo event handler of the page and the DispatcherTimer is started. The timer will constantly call the Detect method every 50 milliseconds. Additionally a viewfinder Rectangle is filled with a VideoBrush which in turn has the photoCamera video stream set as source.

void Detect()
{
   if (!isInitialized)
   {
      return;
   }

   // Update buffer size
   var pixelWidth = photoCamera.PreviewBufferResolution.Width;
   var pixelHeight = photoCamera.PreviewBufferResolution.Height;
   if (buffer == null || buffer.Length != pixelWidth * pixelHeight)
   {
      buffer = new byte[pixelWidth * pixelHeight];
   }

   // Grab snapshot
   photoCamera.GetPreviewBufferY(buffer);

   // Detect
   var dr = arDetector.DetectAllMarkers(buffer, pixelWidth, pixelHeight);

   // Calculate the projection matrix
   if (dr.HasResults)
   {
      // Center at origin of the 256x256 controls
      var centerAtOrigin = Matrix3DFactory.CreateTranslation(-128, -128, 0);
            
      // Swap the y-axis and scale down by half
      var scale = Matrix3DFactory.CreateScale(0.5, -0.5, 0.5);

      // Calculate the complete transformation matrix based on the first detection result
      var world = centerAtOrigin * scale * dr[0].Transformation;

      // Viewport transformation
      var viewport = Matrix3DFactory.CreateViewportTransformation(pixelWidth, pixelHeight);

      // Calculate the final transformation matrix by using the camera projection matrix 
      var m = Matrix3DFactory.CreateViewportProjection(world, Matrix3D.Identity, arDetector.Projection, viewport);

      // Apply the final transformation matrix to the controls
      var matrix3DProjection = new Matrix3DProjection { ProjectionMatrix = m };
      Txt.Projection = matrix3DProjection;
      Img.Projection = matrix3DProjection;
   }
}

A snapshot of the current preview buffer is taken in the Detect method using the GetPreviewBufferY method. This method fills up a byte buffer with the luminance data of the current viewfinder frame. This buffer is then passed to the SLARToolkit's MarkerDetector Detect method, which returns the detected marker information. This transformation data is then used to transform the UIElement perspectively correct in 3D.
Read more about the PhotoCamera's YCbCr methods in this blog post.

void PhotoCameraInitialized(object sender, CameraOperationCompletedEventArgs e)
{
   //  Initialize the Detector
   arDetector = new GrayBufferMarkerDetector();

   // Load the marker pattern. It has 16x16 segments and a width of 80 millimeters
   var marker = Marker.LoadFromResource("data/Marker_SLAR_16x16segments_80width.pat", 16, 16, 80);

   // The perspective projection has the near plane at 1 and the far plane at 4000
   arDetector.Initialize(photoCamera.PreviewBufferResolution.Width, photoCamera.PreviewBufferResolution.Height, 1, 4000, marker);

   isInitialized = true;
}

The SLARToolkit's GrayBufferMarkerDetector is created and set up in the PhotoCamera's Initialized event handler. The brand new GrayBufferMarkerDetector uses the byte buffer with luminance data directly without the need of an ARGB 32 bit pixel conversion.

Checkout the source code at CodePlex if you want to see all the details of the sample which were left out for clarity.

Download it, build your app and augment your reality

The open source SLARToolkit library and all samples are hosted at CodePlex. If you have any comments, questions or suggestions don't hesitate and write a comment, use the Issue Tracker on the CodePlex site or contact me via any other media.

Have fun with the library and please keep me updated if you use it anywhere so I can put a link on the project site.