Microsoft creates mood-sensing tech for the HoloLens augmented-reality headset
Microsoft has patented technology that allows wearers of augmented-reality glasses like its HoloLens to interpret the emotional reactions of the people they're viewing.
The patent - granted to Microsoft the day before its Build conference kicks off where the company's message about the HoloLens is more about practical applications than new developments (see below) - covers a "see-through, head-mounted display and sensing devices" that "detect audible and visual behaviors of a subject in a field of view of the device."
That, of course, sounds like the HoloLens, an augmented-reality headset Microsoft has shown reporters behind closed doors, and probably will bring to market, eventually. Virtual-reality devices like the Oculus Rift fit over the user's eyes, enclosing their field of view. But augmented-reality devices like Google Glass and the HoloLens superimpose a head-up display over the real-world objects that a user sees. One of the differences between Glass and the HoloLens is that the latter projects its images in color, and at a higher resolution than Glass.
It appears Microsoft believes the hardware must not only "see" but understand what it's seeing. Because people interact with each other so much, mood interpretation seems like a powerful tool. But will people see the tech as too intrusive and creepy, thus hampering widespread social use My guess is yes.
How it works
The version of the HoloLens that Microsoft has shown also includes a depth camera, so it can "scan" objects in front of the viewer. The same depth camera is part of the Microsoft Kinect, which can interpret data like a user's heart rate and skin temperature by closely observing his or her face. Microsoft's patent suggests scenarios in which the headset itself performs the interpretation, but also works in conjunction with a smartphone or other device.
Microsoft's patent offers a variation that interprets facial gestures (smiles, frowns, narrowed eyes) as indicators of a particular emotional state. Although the real-world examples Microsoft uses to illustrate the patent include a speaker gauging the emotional response of his audience, another stands out as somewhat creepy: "Other social situations in which the technology may be useful include romantic situations involving a one-on-one relationship between individuals."
Creep-factor aside, the technology could help those with Asperger's, autism, or other social disorders by interpreting emotional and social clues. Or, conceivably, the mood sensing could be used during sales calls or negotiations as a tool to see how far a client could be pushed before backing out. Finally, one might imagine surreptitiously scanning opponents at a poker table to figure out whether they're bluffing.
HoloLens gets practical
Meanwhile at the 2015 Build conference, Microsoft tried to prove that HoloLens is more than just a neat gimmick.
The company showed off several new demos for its "mixed reality" headset, which can map digital imagery onto the user's physical surroundings. While previous demos had focused on fun ideas like a virtual Mars walk and a living room-sized version of Minecraft, the Build presentation emphasized real-world applications for businesses and education.
For instance, Microsoft showed how architects could use HoloLens to interact with 3D models, laid out virtually in front of them on a table. They might also be able to examine aspects of a building site at full scale, with virtual beams and walls rendered before their eyes.
Another demo showed how medical students could learn about the human body without having to cut open a cadaver. The headset showed a 3D model, which users could separate into different parts and blow up for closer examination. With everyone wearing a HoloLens, students and teachers could all interact with the same model, so it's easy to imagine virtual surgery as the next step.
Not all the presentations were so serious. Microsoft also showed off an actual robot whose controls appeared in the virtual space above the robot's head. Users could then create a movement pattern for the robot by tapping on the ground. Another demo showed how users could create their own personal screens that followed them around in real space.
There's still no word on when Microsoft will release HoloLens, how much it'll cost, or the tech specs inside it. The company has only said that it plans to launch the headset within the "Windows 10 timeframe". In lieu of release details, Microsoft did announce several partners, including NASA, Autodesk, Unity, the Walt Disney Company, and the Cleveland Clinic.
Why this matters: While the technology behind HoloLens is certainly impressive, Microsoft needs to prove that the product truly is a big leap forward for computing, and not just a cool tech demo. That was clearly the emphasis at Build, as Microsoft tried to convince developers to take the technology seriously.
HoloLens + Robots
In an on-stage demonstration at Build, HoloLens overlaid a holographic robot named B15 on top of a physical one made using Raspberry Pi 2. The HoloLens also displayed a control panel showing how far the robot traveled, its remaining battery life and wireless connection, its temperature and other variables. HoloLens was able to pull the data from the robot in real time, and users could control the robot through a holographic interface by waving their hands.
Another HoloLens feature is the ability to recognize objects and environments. The HoloLens demonstrated at Build could scan points in 3D space and visualize the robot's movement path, which helped it pass navigation commands to B15. That was particularly helpful as the robot didn't have the sensors to understand the room's environment. Boards like Raspberry Pi 2 don't have weather, light or movement sensors built in, but they can be added through expansion ports.
As environments change and there are unforeseen obstacles, HoloLens could help robots immediately change path. That is important when working with bigger robots in the automotive field and other areas, according to Microsoft.