It's April the 30th afternoon and now 24 hours ago when I was lucky enough to be part of the first group of external developers worldwide who received a Holographic Academy training, so getting in-depth hands on HoloLens prototype hardware and actually deploying apps on it during a 4h course.
There was no NDA I had to sign so here's a quick recap. I actually double checked with some of the Microsoft mentors there that no NDA was in place.
They also
released a video in the meantime where I have my 3 seconds of fame as well. And my friend Gregor published a German
interview with me. I've also written
an article for the German print magazine PC Magazin.
tl;dr
The HoloLens is the most impressive device I've ever tried. It's mind blowing. The Holographic academy workshop was fun and accessible for all kinds of developers.
Setting
This event was happening at a hotel nearby the //build conference center. The HoloLens team basically got a whole floor there. They had multiple security checks and lots of guards standing everywhere. I had to lock my stuff including my smartphone, so no photos or notes allowed but I got some nice HoloLens swag in exchange. After passing the 2nd or 3rd security checkpoint a team member measured my
IPD (65.5 mm), then I passed another guard and finally was escorted into a fancy room with a few dozen attendees, lots of cameras, many HoloLens team members who helped the attendees as mentors, development PC setups and most importantly lots of HoloLens prototype devices (HL).
Hardware
The HL prototypes already looked very polished and have the computing units integrated. Still, the devices we tried had a clear laser printed label on it: "Prototype" and "Not FCC approved...".
After adjusting the HL to the head, it's quite comfortably to wear, not too heavy nor has any annoying fan noise. Unlike other devices does the HL also work well when the user is wearing glasses, so good for people like me.
So far there are only a
few official pieces of information about the hardware available and no further information was shared at the event even though I kept asking the answer was always: "We are not ready to talk about that today." I heard that often but I was allowed to take an in-depth look at the devices. The following is based on my personal analysis so take it with a grain of salt.
There are at least 5 cameras on the front. 1 in the center and 2 on each side where 1 of the 2 is pointing to the side and the other one to the front. I suspect those 4 are the depth cameras. Covering 2 of those 4 with the hands still didn't break the recognition which was impressive. It was hard to judge the minimum and maximum distance it supports but I'd say it was around 0.5 - 5m or just like Kinect v2 in the range of 0.8 - 4m.
The projection screens for the left and right eye are not just flat 2D displays but actually kind of layered screens. I spotted 3 layers for each eye and it seemed like each layer is either for the red, green or blue component. These special screens are key for seamlessly merging the virtual objects into the real-world environment and that's the reason for calling the HoloLens a holographic device. The actual screens were a bit small on the HL prototypes they had available, so the field of view was narrower than expected but it's likely the next generation and consumer devices will improve this. The front also had a few tiny holes which I believe are microphones for the amazing speech recognition and sound analysis. For audio output there are small speakers in the headset band. They provide an immersive spatial sound I've never experienced before. More about that below.
The device is charged and connected to the PC for deploying via Micro USB. There seems to be a small special motherboard connector at the top of the front as well.
Walkthrough
The actual academy was using Unity to create a 3D scene. From within Unity a Visual Studio UAP solution was generated which was then deployed to the HL. Ctrl+F5 onto HoloLens! W00t!
I heard there are different tutorials for the academy. My group was doing the Origami tutorial which means the base was a 3D scene with various Origami paper-like 3D models or Holograms as the presenters called it.
The first chapter covered how to setup the device and running a little demo app where a virtual RC truck could be driven and pushed through the real-world room. On the floor, on the table, on the sofa, on humans, drive the virtual truck everywhere in the room!
The second chapter walked us through the process of getting the base Unity Origami scene deployed onto the device and exploring it from different angles in the room and collide with real-world objects.
The next step involved the addition of the Unity rigid body physics to the virtual 3D objects and adding a virtual indication where one is looking at while rotating the head. It was basically a torus which was projected onto the scenery on top of virtual and real-world objects. This is called
Gaze. We also added an
Air Tap gesture where one is tapping the index finger into the air. The gesture together with the raycasting of Gaze tells you what object in 3D space was selected and basically works like a spatial touch tap / mouse click interaction trigger. The Gaze is the mouse cursor movement in 3D space if you will and the air tap the mouse click. In the case of the Origami demo it triggered a physics action to let Origami paper balls fall down. All this dynamic functionality was implemented using Unity C# scripts.
After that,
speech recognition was added into a C# script and it was very impressive. I just had to add an ordinary string like "Reset Scene" into a script without any pre-defined grammar and it just worked. And my HL device even worked only for my voice which means my HL did not react on the same voice commands another attendee was saying.
Later,
spatial audio was added which means playing sounds in the 3D virtual environment for the 3D object interactions. This was mind blowing! These small speakers make the sound appear spatially from the scenery and around it. It's totally different than just headphones. You actually think other people hear the same in the room but they are not.
Then we added so called
spatial mapping which means overlaying real-world objects with a virtual layer / net. In the academy we overlaid it just with wireframe polygons but trying it on the HL is a very impressive experience.
In the last chapter we removed parts of the real-world floor by overlaying it with a virtual 3D animation which showed a crack in the floor where Origami birds were flying in the underground. The screens of the HL are bright and saturated enough that it actually overlays the real-world without transparency and one could walk around and look from different angles into the underground. Mind blown again!
Further thoughts
There were more chapters / steps in-between for the Origami tutorial or in different order but I think I covered the keys of it above. I've never really used Unity before and I think it was a good choice by the HoloLens team to use Unity for this //build Holographic Academy as it's accessible for many developers who have never done any 3D computer graphics before and it quickly provides nice results. For myself I'd rather have preferred to get my hands on the underlying Direct3D surface and render my own 3D content to it. Honestly I was a bit bored by Unity so I added my own custom speech commands for more physics fun and explored the real HoloLens API with Visual Studio's Object Browser on the side and found some cool stuff in there.
It is great the HoloLens team lets developers get their hands on real device development less than 100 days after showing the first prototypes to the public. Kudos!
Final words
The HoloLens is the most impressive Augmented Reality experience I've ever tried. It's real, not just smoke and mirrors! There are so many innovations in it beside the seamlessly merging of virtual and real-world with the layered holographic screens: spatial audio, perfect speech recognition, spatial gesture recognition, low latency real-time mapping of the real-world environment and all that fully integrated into a single head mounted device without any external cables needed.
24h after attending the academy my mind is still blown and I can't wait to get my hands dirty on a dev device at home pushing it to the limits with the real SDK and Direct3D.
Exciting times!
