Wednesday, February 24, 2010

SLARToolkit - Silverlight Augmented Reality Toolkit

In my last blog post I've preannounced SLARToolkit: The open source Silverlight Augmented Reality Toolkit. Today I'm proud to announce that SLARToolkit is now available at CodePlex.
The aim of SLARToolkit is to make Augmented Reality with Silverlight as easy as possible so that everyone will be able to make Silverlight Augmented Reality applications. Therefore the SLARToolkit release not only includes the full Balder sample you might know from this blog post, it also comes with a brand new sample that uses Silverlight's 3D projection capabilities through my Matrix3DEx library to project different UIElements on the marker.

The SLARToolkit project description from the CodePlex site:
SLARToolkit is a flexible Augmented Reality library for Silverlight with the aim to make real time Augmented Reality applications with Silverlight as easy and fast as possible. It can be used with Silverlight's Webcam API or with any other CaptureSource or a WriteableBitmap. SLARTookit is based on the established NyARToolkit and ARToolkit. SLARToolkit uses a dual license model and could be used for open or closed source applications under certain conditions. See the License page for details.
Features:
Live
A webcam and at least the Silverlight 4 runtime must be installed to run the sample. It's available for Windows and Mac. Alternatively there is also a new video of the Silverlight projection sample embedded below.
If you want to try it yourself you need do download the SLAR and / or L marker, print them and hold them in front of the camera. The marker(s) should be printed non-scaled at the original size (80 x 80 mm) and centered for a small white border. Also make sure the camera is set up properly and the scene is illuminated well without hard shadows. See the SLARToolkit Markers documentation for more details.




Simply press the "Start Fun" Button to start the webcam. Use the ComboBox to select a different UIElement that will be mapped on the marker. The size and the orientation can be changed with the "Scale" and "Rotation" Sliders. The "Threshold" Slider controls the threshold for the marker detection. The "Flip x-axis" Checkbox could be used to flip the video (the webcam output is mirror-reversed by default).
If you click the "Start Fun" Button for the first time you need to give your permission for the capturing. This application uses the default Silverlight capture device. You can specify the video and audio devices that are used by default with the Silverlight Configuration. Just press the right mouse button over the application, click "Silverlight" in the context menu and select the "Webcam / Mic" tab to set them.

Video
I've recorded a short video of the new sample with Expression Encoder's Screen Capture feature. Please keep in mind that the screen recording software eats up a lot of resources while recording and that the actual frame rate is way better.


Background music is leandi by mosaik.


How it works
This sample uses the SLARToolkit CaptureSourceMarkerDetector to detect the markers in each webcam frame. The detection result contains a transformation matrix for each found marker which is then used to transform the selected UIElement. The tracked marker region is highlighted with a red quad through the WriteableBitmapEx library.

Download it, build your app and augment your reality
The open source SLARToolkit library and the samples are hosted at CodePlex. If you have any comments, questions or suggestions don't hesitate and write a comment, use the Issue Tracker on the CodePlex site or contact me via any other media.
Have fun with the library and please keep me updated if you use it anywhere so I can put a link on the project site.

Update 03-15-2010
Updated to the Silverlight 4 Release Candidate.

Update 04-15-2010
Updated to the final Silverlight 4 RTW build.

36 comments:

  1. Nice work Rene! Wondering how we could combine this in Windows Phone 7 series ;-).

    ReplyDelete
  2. Thanks!

    @Mark: That was my first thought when I heard about Silverlight on Windows Phone 7. :)

    ReplyDelete
  3. The demo is not working on my mac :( I have Silverlight 4 beta installed, and my iSight is configured and working in Silverlight. When I try to launch the demo this warning shows up: "Error during init. Please make sure a default webcam was set. Index was out of range. Must be non-negative and less than the size of the collection. Parameter name: index". The default webcam was set, and when I press the Start Fun button, my webcam image shows up but the markers aren't detected, and I'm getting about 4fps.

    ReplyDelete
  4. Hi Andrei,

    too bad you encountered some strange behavior, you're actually the first that reported such things. Thanks for letting me know.
    Have you set the default webcam in the Silverlight settings as described above?

    Please try this special version I've build and send the results via Email to me:
    slar//rene-schulte°info (Replace the // with an @ and the ° with the .)
    Here's the link to the special build:
    http://dl.dropbox.com/u/2681028/CodeplexData/SLARToolkit/Samples/SLARToolKitSample_SpecialErrorFindBuild/TestPage.html

    Thanks for your help!

    ReplyDelete
  5. Could you upload the code to this example somewhere?

    ReplyDelete
  6. You should read the blog post till the end. ;)

    "The open source SLARToolkit library and the samples are hosted at CodePlex."

    ReplyDelete
  7. Stupid question, but how do i open the SLARToolKit.pdb file? Microsoft Visual Web Developer 2010 Express, doesnt seem able to open it?

    ReplyDelete
  8. The PDB files are just the debugging symbols. If you download the right source for the release you will be able to debug.

    Just add a reference to the SLARToolkit.dll to your project in order to start SLARToolkit development.

    ReplyDelete
  9. Thnx, I'm not very experienced with Visual Studio, but I'm able to open the project now (by using SLARToolkit-45430).

    I added a reference to the SLARToolkit.dll, but I'm still getting 7 errors:

    Error 1 'System.Windows.Media.VideoFormat' does not contain a definition for 'PixelWidth' and no extension method 'PixelWidth' accepting a first argument of type 'System.Windows.Media.VideoFormat' could be found (are you missing a using directive or an assembly reference?) SLARToolKitSample\MainPage.xaml.cs 70 52 SLARToolKitSample
    Error 2 'System.Windows.Media.VideoFormat' does not contain a definition for 'PixelHeight' and no extension method 'PixelHeight' accepting a first argument of type 'System.Windows.Media.VideoFormat' could be found (are you missing a using directive or an assembly reference?) SLARToolKitSample\MainPage.xaml.cs 70 80 SLARToolKitSample
    Error 3 'System.Windows.Media.VideoFormat' does not contain a definition for 'PixelWidth' and no extension method 'PixelWidth' accepting a first argument of type 'System.Windows.Media.VideoFormat' could be found (are you missing a using directive or an assembly reference?) SLARToolKitSample\MainPage.xaml.cs 105 101 SLARToolKitSample
    Error 4 'System.Windows.Media.VideoFormat' does not contain a definition for 'PixelHeight' and no extension method 'PixelHeight' accepting a first argument of type 'System.Windows.Media.VideoFormat' could be found (are you missing a using directive or an assembly reference?) SLARToolKitSample\MainPage.xaml.cs 105 116 SLARToolKitSample
    Error 5 Cannot convert lambda expression to type 'System.Delegate' because it is not a delegate type SLARToolkit-45430\trunk\SLARToolKit\Source\SLARToolKitSample\MainPage.xaml.cs 122 36 SLARToolKitSample
    Error 6 'SLARToolKit.MarkerDetectionEventArgs' does not contain a definition for 'BufferWidth' and no extension method 'BufferWidth' accepting a first argument of type 'SLARToolKit.MarkerDetectionEventArgs' could be found (are you missing a using directive or an assembly reference?) SLARToolKit\Source\SLARToolKitSample\MainPage.xaml.cs 135 63 SLARToolKitSample
    Error 7 'SLARToolKit.MarkerDetectionEventArgs' does not contain a definition for 'BufferHeight' and no extension method 'BufferHeight' accepting a first argument of type 'SLARToolKit.MarkerDetectionEventArgs' could be found (are you missing a using directive or an assembly reference?) SLARToolkit-45430\trunk\SLARToolKit\Source\SLARToolKitSample\MainPage.xaml.cs 135 78 SLARToolKitSample

    I put everything in here, because it might help others?

    ReplyDelete
  10. Looks like you are still using the the beta of Silverlight 4. As you can read in the update at the bottom I updated everything to the new Silverlight 4 Release Candidate. So you should install the Visual Studio 2010 RC and the Silverlight 4 RC Tools. See my latest blog post for the links:

    http://kodierer.blogspot.com/2010/03/silverlight-4-release-candidate.html

    The best way to get started would be if you download the whole source code of revision 45430 and use one of the samples as a starting point. Esp. when you are new to Visual Studio / development.

    http://slartoolkit.codeplex.com/SourceControl/changeset/view/45430

    ReplyDelete
  11. Thnx for the quick response. I'm now using Visual Studio 2010 RC1 and Silverlight 4 from your url.
    Still 3 errors remain:

    Error 1 Cannot convert lambda expression to type 'System.Delegate' because it is not a delegate type \MainPage.xaml.cs 122 36 SLARToolKitSample
    Error 2 'SLARToolKit.MarkerDetectionEventArgs' does not contain a definition for 'BufferWidth' and no extension method 'BufferWidth' accepting a first argument of type 'SLARToolKit.MarkerDetectionEventArgs' could be found (are you missing a using directive or an assembly reference?) \MainPage.xaml.cs 137 63 SLARToolKitSample
    Error 3 'SLARToolKit.MarkerDetectionEventArgs' does not contain a definition for 'BufferHeight' and no extension method 'BufferHeight' accepting a first argument of type 'SLARToolKit.MarkerDetectionEventArgs' could be found (are you missing a using directive or an assembly reference?) \MainPage.xaml.cs 137 78 SLARToolKitSample

    ReplyDelete
  12. Strange. The 45430 revision works for me without problems otherwise I wouldn't have been able to make new live versions of the samples obvisously.

    Try a clean build. If the problem remains, send me an Email with the attached project and I will look over it. See my website for the Email address:

    http://rene-schulte.info

    ReplyDelete
  13. Hi teichgraf... I'm interested in use the SLARToolkit... but I have a question: this SLAR would accept multi-markers? I want to make something like this (Avatar Toy: http://www.youtube.com/watch?v=7JWk_JIE3Ow)

    I'm using FLARManager, and I make something like that (Zakumi interactive) ... but the papervision3D/away engine 3D is so basic yet... (so much problems with render clipping faces and scales)

    I want to know if SLAR would be capable to detect multi markers for locating one two or many different objects in diferent markers or one object and activate (or deactivate) a function (animation, effect etc etc) with 2 or 3 markers. Don't matter the marker itself, I mean, with black and white marker (pat isn't?) would be cool...

    ReplyDelete
  14. Hi teichgraf... I'm interested in use the SLARToolkit... but I have a question: this SLAR would accept multi-markers? I want to make something like this (Avatar Toy: http://www.youtube.com/watch?v=7JWk_JIE3Ow)

    I'm using FLARManager, and I make something like that (Zakumi interactive) ... but the papervision3D/away engine 3D is so basic yet... (so much problems with render clipping faces and scales)

    I want to know if SLAR would be capable to detect multi markers for locating one two or many different objects in diferent markers or one object and activate (or deactivate) a function (animation, effect etc etc) with 2 or 3 markers. Don't matter the marker itself, I mean, with black and white marker (pat isn't?) would be cool...

    ReplyDelete
  15. SLARToolkit uses the ARtoolkit pat files and supports multiple markers. I tested it with 100 markers and it worked well!
    In code: each time after the detection is performed you get a List of detection results. Each result has a reference to a marker instance. So it can be identified.
    Check out the Samples at the CodePlex site. I use multiple markers there.

    The 3D Sample uses the Silverlight 3D engine Balder for the rendering of the 3D objects. You should also see Balder's Codeplex site to get an idea of the features. I heard that a new release with a ton of new functionality is coming soon.

    The other sample (Video above) uses Silverlight's built-in pseudo 3D capabilities (2.5D).

    ReplyDelete
  16. Hi Rene,

    this AR toolkit seems to suffer the same problem that the JARTOOLKIT does, have you considered having a look in the Kalman Filter?

    http://en.wikipedia.org/wiki/Kalman_filter

    Hope it helps you ;D

    ReplyDelete
  17. Hi Gabriel,

    thanks for your info. I've posted a reply to your thread here:
    http://slartoolkit.codeplex.com/Thread/View.aspx?ThreadId=217564&ANCHOR#Post460609

    ReplyDelete
  18. Hi Rene, greetings from Brazil. Sorry about bad english, but i hope you understand. Do you have any code sample of SLARToolkit with multiple markers? I looked into codeplex like you told above, but didn't find. May you help me, please? Thank you in advance, best regards. Bruno.

    ReplyDelete
  19. Hi Bruno,

    please download the sample code from the CodePlex site. The Balder and the Silverlight projection both use two markers.

    ReplyDelete
  20. Thanks a lot for your help Rene. Now i figured it out how to use multiple markers. But still have a problem, how can i make my own markers, tried to use ARToolKit Marker Generator but didn't work. Maybe webcam problems? like resolution or luminosity issue? I hope you may help me again. Actualy i'm trying to do some kind of jigsaw puzzle for kids, that's why using multiple markers. Thanks again.

    ReplyDelete
  21. How to make and use markers is one of the topics I cover in the SLARToolkit documentation.
    http://slartoolkit.codeplex.com/documentation
    This always works for me.
    Another tip, take a look into a pat file. It's a pretty simple text format and a auto generator should be easy to write.

    BTW, I did a puzzle in the Silverlight Projection sample.

    ReplyDelete
  22. Hi rene, how i can create models .ase? which program i can use? i can´t export in the 3dsMaxStudio
    Sorry about bad english..
    I hope your help!
    Anna from paraguay

    ReplyDelete
  23. The ASE format comes originally from 3dsMax: http://www.modwiki.net/wiki/ASE_(file_format)
    I use Blender and a Python ASE export script.

    ReplyDelete
  24. thans for your answer!
    Rene, I can't export my model in blender.. I need to install a plugin? or... I have to type a script?
    in Visual Studio developer, I can't visualized my model..

    ReplyDelete
  25. You can find the right Blender Python scripts and all the information here: http://www.katsbits.com/tools/#ase

    But again, 3dsMax should support ASE. The format comes from there. It's way easier to use the 3dsMax export / save than Blender + Python + Script. See the 3dsMax documentation.

    ReplyDelete
  26. Download the project but get the following error, can you help me?
    is an example SlartoolkitSampleBalder

    System.InvalidOperationException
    An unhandled exception was encountered while trying to render the current silverlight project on the design surface. To diagnose this failure, please try to run the project in a regular browser using the silverlight developer runtime.
    at Microsoft.Windows.Design.Platform.SilverlightViewProducer.OnUnhandledException(Object sender, ViewUnhandledExceptionEventArgs e)
    at Microsoft.Expression.Platform.Silverlight.SilverlightPlatformSpecificView.OnUnhandledException(Object sender, ViewUnhandledExceptionEventArgs args)
    at Microsoft.Expression.Platform.Silverlight.Host.SilverlightImageHost.<>c__DisplayClass1.b__0(Object o)
    at System.Windows.Threading.ExceptionWrapper.InternalRealCall(Delegate callback, Object args, Int32 numArgs)
    at MS.Internal.Threading.ExceptionFilterHelper.TryCatchWhen(Object source, Delegate method, Object args, Int32 numArgs, Delegate catchHandler)


    System.ArgumentException
    There is no loader for the specified file type 'ase'
    at Balder.Assets.AssetLoaderService.GetLoader[T](String assetName)
    at Balder.Content.ContentManager.LoadInto[T](T asset, String assetName)
    at Balder.Objects.Geometries.Mesh.Prepare(Viewport viewport)
    at Balder.Node.OnPrepare(Viewport viewport)
    at Balder.Rendering.NodeRenderingService.Prepare(Viewport viewport, INode node)
    at Balder.Rendering.NodeRenderingService.Prepare(Viewport viewport, NodeCollection nodes)
    at Balder.Scene.Prepare(Viewport viewport)
    at Balder.Display.Viewport.Prepare(PrepareMessage prepareMessage)
    at Balder.Execution.MessageSubscriptions`1.Notify(T message)
    at Balder.Display.Silverlight.Display.Prepare()
    at Balder.Rendering.Silverlight.RenderEventHandler.Invoke()
    at Balder.Rendering.Silverlight.RenderingManager.ShowTimer(Object sender, EventArgs e)
    at System.Windows.Media.CompositionTarget.OnRendering(Object sender, RenderingEventArgs e)
    at MS.Internal.JoltHelper.RaiseEvent(IntPtr target, UInt32 eventId, IntPtr coreEventArgs, UInt32 eventArgsTypeIndex)

    ReplyDelete
  27. no puedo ver el diseƱador, pero puedo probar la aplicacion en el navegador


    thanks for your help

    ReplyDelete
  28. I cant wait until i can get this working! I am seriously teaming with anticipation, but i keep getting 1 error, CS0246 MainPage could not be found!! ghrr...please help me this is driving me crazy. This is right before the debug if that helps at all.
    -Alex

    ReplyDelete
  29. @Alex: http://msdn.microsoft.com/en-us/library/w7xf6dxs(v=vs.80).aspx

    Download the code from the repository, learn from the samples and modify them for your needs.
    There's also a Beginner's Guide:
    http://slartoolkit.codeplex.com/wikipage?title=Beginner's%20Guide&referringTitle=Documentation

    ReplyDelete
  30. hi,

    is there any script to load image. cause i'm only using digital kamera for my project in Augmented reality aplication.

    thanks.

    ReplyDelete
  31. Hi,

    Do you have plans to migrate this Project to Windows RT someday?
    Thanks,
    Juanma

    ReplyDelete