At their Ignite conference, Microsoft announced Mesh, the culmination of their efforts in developing mixed reality devices, processing power and applications. Mesh is a new, Microsoft Azure-backed platform that uses an “outside-in” approach to enable its users to connect in a shared space to interact and – perhaps most importantly – to really “see” each other with a virtual space.
Microsoft released a video demonstrating what they are envisioning for Mesh (which you can watch in full below). It includes the ability for teams to view architectural, industrial and other projects as if they were in the same room, passing virtual objects back and forth. Sometimes a person wearing a HoloLens is talking with someone that looks like a hologram, and other times participants are show as a more cartoon-ish avatar.
Microsoft suggests that Mesh can be used not only to enhance virtual meetings, but to conduct virtual design sessions, provide remote training, facilitate distribute teams, learn together virtually, or even host virtual social gatherings and meet-ups. The potential for AR applications, are of course, well known to the industry, but there have always been some significant barriers hampering their widespread use.
Leveraging 3D and technology for immersive experiences
One of the options for being present in a Mesh environment is via a 3D presence with representative avatars via inside-out sensors of the devices. This is the most accessible version, as it requires the least amount of specialized hardware. The Mesh platform comes with an avatar rig and a customization studio so you can use the out-of-the-box avatars that you create yourself. The platform is capable of powering existing avatar rigs as well with its AI-powered motion models – this allows the device to capture accurate motions and expressions consistent with the user’s action.
Alongside avatars, Mesh also enables the most photorealistic 360o holoportation with outside-in sensors. These outside-in sensors can be a custom camera setup like the Mixed Reality Capture Studio, which helps capture in 3D with full fidelity or via Azure Kinect that captures depth-sensed images to assist in producing the holographic representations. Once the holograms are produced, these can be used within Mesh-enabled apps on immersive mixed reality headsets or phones, PCs, and tablets, to holoport users in their most life-like representations.
While Mesh is getting a lot of attention this week for its sci-fi inspirations (drawing comparisons to the Holodeck forn Star Trek as well as the projections of Princess Leia in Star Wars), there’s a seriously impressive technical side to what Microsoft has managed to accomplish.
Removing the barriers to mixed reality development
Mixed reality for enterprise has had a bit of a rough start, but Microsoft’s Hololens (especially after its redesign in 2019) has become pretty solidly adopted across Fortune 500 companies, with more than half of them deploying HoloLens or other mixed reality solutions. But, as is the case with many hardware solutions, sometimes the software end has to keep up. This is especially true when trying to replicate what it would be like to work with another person face-to-face.
Microsoft lists four core challenges to mixed reality developers:
- Representing people in MR with appropriate realism
- Keeping a hologram stable in a shared MR space across time and device types
- Facilitating the import of high-fidelity 3D models into MR to support customer file formats
- Synchronizing actions and expressions of people in a geographically distributed MR session
Realism, stability, compatibility and synchronization: Mesh intends to be the solution to all four of these challenges by providing a platform for developers to design immersive multiuser mixed reality apps without working about these individual technical hurdles first.
Essentially, Microsoft seems to be taking on the brunt of the back-end difficult work while developers are more able to use the SDK and other tools it provides to build the multiuser apps they truly want to build. Beyond the core platform, Microsoft also has key AI-powered capabilities that allows Mesh to address some of the most complicated technical challenges with enabling massive multiuser online (MMO) scenarios for mixed reality. These include immersive presence, spatial maps, holographic rendering, and multiuser sync.
Mesh is designed to support a variety of devices – and they are definitely seeking to be as provider agnostic as possible. For headsets, Mesh supports not just the HoloLens, but other supports fully immersive head-mounted displays like the HP Reverb, Oculus Quest, and even phones and tablets (for a more 2-dimensional experience).
We’ve known for some time that mixed and augmented reality has huge potential for myriad industries, and the pandemic’s forced geographic isolation has made that even clearer. With this new platform from a big player like Microsoft, there’s a good chance we’re going to be seeing a lot more applications spring up against this pent-up demand.
