ZzzZzz - Microsoft Band Sleep Tracking in the Testbed

I'm a perfectionist and hungry for knowledge, so I went to a sleep clinic just to check how well the Microsoft Band sleep tracking is working compared to proven scientific methods.
Unfortunately it's not like that and I actually had to go to the sleep lab since I have some issues there. Anyway, since I had to go to the sleep laboratory in the clinic and I just got my Microsoft Band I took the occasion and also used my Band's sleep tracking feature while being monitored using the lab's Polysomnography sleep study.
After my 2 nights at the clinic I asked the lab staff and physician if I can get some personal printouts of my results. They were all nice people and the printouts provide some good data for comparison with the Microsoft Band and Health data.
A funny anecdote: The lab's senior physician is actually a big Windows Phone enthusiast and even tried Windows 10 for Mobile on his device so we had some good chats, also about HoloLens and how it can help them...

The science of sleeping
Sleep studying is a huge scientific area with a lot of active research and many parts of sleep are still unknown territory. I'm not a sleep research expert but learned a few things recently.
There are a couple of different methods for sleep monitoring / tracking. The gold standard for sleep labs is the Polysomnography (PSG) which uses a bunch of sensor data like EEG, ECG, oxygen saturation, eye movement, breathing sensors, muscle movement, etc. For this 2-3 dozen cables are connected to your body for a PSG sleep study.
Another, more basic method is called Actigraphy which just tracks the movement and maybe a few other parameters so it's only used to get information when a person went to bed and when they got up but it can not provide solid data about sleep stages as those depend mostly on your brain activity. The sleep features of fitness trackers like the Jawbone Up, Fitbit, etc. are basically actigraphs. The folks of Exists wrote a very good blog post about those devices and also link to some research studies about actigraphy and how well it works in comparison to PSG. The Microsoft Band also uses actigraphy plus some additional sensor data like samples of heart frequency. Maybe it's even using the skin temperature sensor as well.

Science basically defines five stages of sleeping: The phase when you are awake (W), the rapid eye movement (REM) phase when you are dreaming and 3 Non-REM phases where NREM1 (N1) is the transition to sleep, NREM2 (N2) is light sleep when the heart rate starts to decrease and NREM3 (N3) defines deep sleep when you are fully relaxed and difficult to awaken.

Enough of the theory...


Analysis of Night 1
The first night at the lab was pretty bad. All those cables and a clinic bed were very distracting so my actual sleep was very short and the phases not at all optimal. Keep this in mind when looking at the graphs. My home sleep data is usually way better.
In the table below I listed the parameters the Band provides and compared those to the Lab data. I omitted the total Duration (TIB), Time to fall asleep and Sleep efficiency since the lab and Band measuring started/ended at different times and those parameters depend on that. The Resting heart rate is also excluded as no clear comparable data is available in the PSG printouts, but looking at the graphs the Band Resting HR is somewhat correct.

Parameter	Band	Lab PSG	Band error [%]
Actual Sleep (TST) [min]	257	319.5	-20%
Light Sleep (N1 + N2 [min]	184	271.6	-32%
Restful Sleep (N3) [min]	73	30.3	141%
Woke up	10	9	11%

As expected does the Band actigraph not provide as accurate data as a PSG and the sleep stages like Light sleep and Restful sleep don't match the actual scientific sleep stages very well.

In the below illustrations I overlaid the Microsoft Band's Health sleep charts (colored bars) with the PSG sleep stage graph (black lines). The black scan of the PSG printout was scaled to match the correct time and size of the colored Band data.



Night 1: Colored Band stages data overlaid with black PSG stages

This overlay is pretty interesting and tells quite a bit. Note the Wake phases match mostly but since the Band sampling only happens every few minutes it does not record higher frequency changes and therefore misses short naps or awake periods.
The REM phase is considered Light sleep by the Band which makes sense. Band's Restful sleep is mostly N3 but also a good portion of N2. I assume the Band only uses the heart frequency to define the Restful stages and since the heart rate starts to decrease in N2 the band labels the Restful stage sometimes in N2 until the body goes into N3. The next two overlays underline this theory.



Night 1: Colored Band stages data overlaid with black PSG heart rate

In the above illustration the PSG heart rate was overlaid on the Band's sleep stages and the heart rate chart. As can be seen the Band's Restful blocks correlate with a longer, lower heart rate period.

The third illustration below has the PSG heart rate on top of the Band's heart rate chart.



Night 1: Band's heart rate data (thick purple line) overlaid with black PSG heart rate

Again, the Band's low sampling frequency for sleep mode does not record higher frequency changes like peeks in heart rate which can be caused by sleep apnea when the body tries to compensate low oxygen saturation. Improving those measurements is one of my top wishes / suggestions for the Microsoft Band / Health service.


Analysis of Night 2
The second night was a bit better and I felt more rested after it. I guess mostly because I was super tired from night 1 and therefore slept better even with all those PSG sensors attached.

Parameter	Band	Lab PSG	Band error [%]
Actual Sleep (TST) [min]	310	298.5	4%
Light Sleep (N1 + N2 [min]	183	199.3	-8%
Restful Sleep (N3) [min]	126	70.1	80%
Woke up	9	8	13%

If the Night 1 and Night 2 data is compared it can be seen that my subjective impression was not wrong. I got more than twice deep sleep (N3) during Night 2.
The Band vs. Lab PSG also seems to have less errors which leads to the conclusion that higher sleep quality produces better Band results. I'm not sure how the Microsoft Health analysis of the Band tracking data actually works but I wouldn't be surprised if it uses Azure's machine learning service and the used training datasets contain mostly healthy and normal sleep patterns.

Night 2: Colored Band stages data overlaid with black PSG stages

Again, the Band labels longer periods of N2 and always N3 as Restful sleep.

Night 2: Colored Band stages data overlaid with black PSG heart rate

Night 2: Band's heart rate data (thick purple line) overlaid with black PSG heart rate

Conclusion
The Microsoft Band sleep tracking feature is a nice gimmick but it's not really surprising it doesn't look great when compared to proven scientific sleep monitoring. The Band is not bad for the total sleep duration data, the actual sleep time and the wake up phases. The different stages like Light and Restful sleep are not terribly wrong either but should not be trusted too much. The analysis of sleep stages requires actual brainwave data which is not provided by the Band sensors. It seems the Band's Restful stage is mostly based on the heart rate and movement.
Of course the Band should not be used to diagnose sleep disorders like sleep apnea due to missing sensors but also the low frequency sampling in sleep mode. Hopefully this can be improved. Still if you feel tired more than usual and the Band / Health sleep patterns are totally off, go see a doctor soon!
In the end the Band is a fitness tracker and not a scientific measurement device. The Band and Microsoft Health's sleep tracking is still a nice feature to get an overall idea about your sleep patterns if you keep the limitations in mind.

Happy tracking!