Build collaborative Mixed Reality experiences with Microsoft Mesh
Experience and build collaborative and shared, cross-platform mixed reality experiences with Microsoft Mesh. Take a hands-on tour with the co-creator of Mesh, Simon Skaria, lead engineer on Microsoft’s mixed reality team, to see how Mesh works, experiences with the preview Microsoft Mesh for HoloLens 2 app, and how you can build these experiences as an application developer.
Mesh is a new mixed reality platform from Microsoft that connects the physical and the digital worlds allowing you and the people you connect with to transcend the traditional boundaries of space and time.
In Microsoft Mesh you can collaborate with others in shared spaces, so you can feel true presence and create together from your device of choice, whether that’s Augmented Reality (AR), Virtual Reality (VR), or through the screens of your PC, tablet or phone.
For application developers, Microsoft Mesh enables you to build immersive, multi-user, cross-platform mixed reality applications leveraging AI-powered tools for spatial maps, immersive presence, holographic rendering, and multi-user sync.
You can use Microsoft Mesh to conduct virtual design sessions, learn together virtually, enhance virtual meetings, host virtual social gatherings and meet-ups and much more — all with a geographically diverse set of participants.
QUICK LINKS:
01:47 — Microsoft Mesh for HoloLens 2 preview
02:20 — Demonstration of the multi-user experience
07:50 — Microsoft Mesh platform
09:35 — App developer platform introduction
10:50 — Spatial maps
11:29 — Immersive presence
12:57 — Holographic rendering
14:07 — Multi-user sync
15:12 — Developer tooling and SDK
15:49 — Wrap Up
Link References:
Learn more and request access to the Microsoft Mesh preview at https://microsoft.com/mesh
Find written developer guidance at https://aka.ms/MeshDocs
See more details on the developer experience at https://aka.ms/MeshIgnite
Unfamiliar with Microsoft Mechanics?
We are Microsoft’s official video series for IT. You can watch and share valuable content and demos of current and upcoming tech from the people who build it at Microsoft.
- Subscribe to our YouTube: https://www.youtube.com/c/MicrosoftMechanicsSeries?sub_confirmation=1
- Follow us on Twitter: https://twitter.com/MSFTMechanics
- Follow us on LinkedIn: https://www.linkedin.com/company/microsoft-mechanics/
- Follow us on Facebook: https://facebook.com/microsoftmechanics/
Video Transcript:
- Coming up on Microsoft Mechanics, we’re joined today by the co-creator of Microsoft Mesh, Simon Skaria from the mixed reality engineering team, to take an early look at Mesh, a collaborative mixed reality platform from Microsoft that brings multiple participants into a 3D space to collaborate. So Simon, thanks for joining us today on Microsoft Mechanics.
- Thanks, Jeremy, it’s great to be on the show.
- And it’s really great having you on. So Microsoft Mesh is really one of the biggest announcements of the year, but can you explain what it is?
- Absolutely. One of the easiest ways to think about is that Microsoft Mesh connects the physical and the digital worlds allowing us to transcend the traditional boundaries of space and time. Microsoft Mesh enables applications to feel true presence. You can create together from any device of your choice, whether that’s AR, VR, PC or Phones. And for our developers, Microsoft Mesh can build immersive multi-user cross-platform Mixed Reality applications, leveraging AI-powered tools for spatial maps, immersive presence, holographic rendering and multi-user sync. And you can leverage Microsoft Mesh to conduct virtual design sessions, learn together virtually, enhance virtual meetings. And in fact, I hope to host you in Microsoft Mesh in one of the upcoming shows and the social gatherings and meet-ups and much more with a geographically diverse set of participants, this is our vision for Microsoft Mesh.
-I’d love to do Microsoft Mechanics and Microsoft Mesh. So now you can more easily interact with others as if you are face to face and really collaborate with 2D and 3D content. And it’s pretty significant because it expands the horizon also for who gets access to the tech. So as a user then, how do I connect to Microsoft Mesh?
-We just released the preview experience of Microsoft Mesh for HoloLens 2. This is one of the first emblematic experiences in Microsoft Mesh platform. We also have a Mesh-enabled version of all space. And you will soon see a number of more applications that take advantage of Microsoft Mesh platform both from Microsoft as well as our valued ecosystem of partners.
- And the true beauty of this is that you can join the experience literally from anywhere. So I know earlier that you recorded yourself Meshing with your team. Why don’t we look at that and from there we can explain all the tech behind it?
- Absolutely, I’m excited to see what you think about it. It’s so wonderful to be here, finally being able to take the wraps off Microsoft Mesh and being able to give a tour to you all. Before we get started, let me give you a quick tour of the setup. What you are seeing is a custom camera rig that gives you the perspective as if you are wearing a HoloLens. With that, let me flip my HoloLens visor down and enter Microsoft Mesh. Foundational to Microsoft Mesh is this notion of spaces. Think of spaces like a conference room, except something that is persisted across time and geographies. I have my home space, I have a pipeline review space where my colleagues are waiting for. So let me go ahead and enter the pipeline review space. I can see here my dear friend, Nishant, waiting in the review space. So Nishant, great to see you here.
- Hey Simon, how are you doing?
- Hello. Hey Nishant, can you explain a little bit of what you are seeing?
- Absolutely, Simon. I see an avatar of yours and I hear you voice as if you are right across the room from me here. I see an awesome hologram of a table, which is inviting me to bring in 3D models and create and collaborate upon it. And of course, as I lift my hands, I see the hand menu, where I can bring in more people into the space, I can go ahead and change my avatar and appearance. I can annotate, draw and bring in more shapes and figures into the space, as well as bring in 2D and 3D content to collaborate upon.
- Hey Nishant, since we are here to take a look at the project that Dorrene worked on, do you think you could quickly invite her if she’s available to the space?
- Certainly Simon. Let me invite Dorrene.
- While Nishant is doing this, I wanna make a mention that Microsoft Mesh is integrated with AAD and MSA so that hosts can invite participants into designated spaces in a simple but trustworthy manner. When I lower down my visor, I can see that Nishant has already invited Dorrene to the space.
- Hey Dorrene, great to see you here.
- Hi Simon. Hi Nishant.
- Dorrene, it’s so great to see you. Do you recall the pipeline project that you worked on? It needs a quick review. If you could kindly bring in that model from your OneDrive so we all could take a look at it and make a few recommendations to the engineers, that would be great.
- Absolutely. Let me pull it up from my OneDrive. What’s great about Mesh is the integration with the Microsoft 365 Graph. What we show here is actually a model that our customers use today. This is a part that is used to prevent spills. But the idea is that a wide range of data can be shown inside of Mesh. Your connections, your collaborative efforts, your workflows, even your documents, transfer very well from a 2D space into a 3D space.
- So one of the things about collaborating in mixed reality is the 3 dimensional fidelity that it offers. And while each of us can have a unique perspective into it, we all can rotate, translate, pan, zoom, add or redact from this model, driving clarity to the decision making while reducing the cost and complexity of being in in-person meetings. One of the things that I miss really in in-person meetings is the ability for us to annotate in a whiteboard or on a paper. And with Microsoft Mesh enabled applications, I can bring up my hand menu, bring up my annotations here, and my fingers turn into, at this time, essentially a pen through which I can annotate in a space. I can see here that while I’ve been doing this, my colleagues here have annotated as well.
- Yeah, and speaking of annotations, I think there is a piece of this model we should look at. Can you come over here and take a look quickly?
- Sure.
- Great. I’m actually noticing that this space on the model here may actually need a further look from our engineering team, it seems like there might be something out of date. Do you think we can take a look with them maybe early next week?
- Yeah, I can request the engineering team to come and take a look. One of the benefits of collaboration in Microsoft Mesh is the ability for us to persist these spaces across time and geographies. So the engineering team can come, join the space and pick up precisely where we left off, with the annotations, with the 3D models, with the point of view that Dorrene wanted us to take a look at it. And at any point and time, if they need the help of Dorrene or Nishant for an expert eyes or hands, they just know that they are just a Mesh tap away. So thank you Dorrene and Nishant joining on a short notice. We look forward to working together with you again on another project sometime soon. With that, I’ve given you a quick tour of Microsoft Mesh experience for HoloLens.
- That was really amazing stuff. And by the way, that was just the preview experience of Microsoft Mesh with HoloLens. It almost felt like I was in the room watching from a 3rd person perspective. So Simon, I know there’s a lot to unpack here. But why don’t we do that and really go under the covers because I know that we are simplifying a ton of complexity with Microsoft Mesh.
- Yeah, Jeremy, that’s exactly the point. It’s taken years of work to be able to virtually bring people together. You and I talked about this before. A few years back with unbounded compute and bandwidth, one could possibly build these kinds of experiences, but with Microsoft Mesh, what we have done is we have cost and complexity reduce the AI algorithms so that it fits in I think about a typical 7 watt passively cooled head mounted devices. And each of those AI capabilities from spatial maps to immersive presence, to holographic rendering and sync is decades of technical investment so that as a developer, they can focus on solving business problems while the Mesh platform solves the hard underlying technical issues. Importantly, with Microsoft Mesh, privacy and security are built into the platform, from integration of our identity services and encryption across all data at rest and in transit. You also saw while I was hosting the space from my physical environment, participants had a neutral view focused on the shared content and the privacy of my personal space was protected all along. And what we showed is just one example of the types of mixed reality experiences that you can expect.
- Right, I know that the Mesh platform can do a lot more so why don’t we dig into some of the underlying capabilities?
- Absolutely, Jeremy. This is the best part of it. So beyond what we just showed as the end user experience, Microsoft Mesh gives you a robust developer platform for building collaborative mixed reality experiences. At the core of the platform, as I mentioned before, Mesh is integrated with identity options like Azure Active Directory and Microsoft Accounts. It also uses the Microsoft Graph so it makes it easy to intuitively connect users, content and activities. So if you’re already invested in Microsoft 365 application, there is more to be now done with Microsoft Mesh. And all of this is running on Azure so you don’t need to worry about provisioning backend servers, thinking about the scale and cost of live state sync or compute run apps like these. And as I mentioned, we give you all the building blocks to light up collaborative mixed reality experiences with spatial map, 3D presence, holographic rendering and multi-user sync. And as a developer, you can access all of this using our SDK.
- So it’s super comprehensive, and can we take a closer look though at the capabilities and why don’t we start out with spatial maps.
- So with spatial maps, you can share and persist holographic content. This is what makes a hologram stay in place across time, across space and across devices. Prior to Mesh, every device had its own local of the world. And with Mesh, all those local caches are merged and optimized securely in the cloud to create a global understanding of the environment these devices operate in. And this global maps serve as a trellis that enables content to be anchored, device points of views to be shared and 3D models to be collaborated on.
- [Jeremy] Okay. So how are we able then to bring people into that shared space?
- That’s the soul of Microsoft Mesh, where presence comes in. Presence make it possible to represent ourselves in a more immersive way in the digital medium. The fundamental parameters of presence that we think about is realism on one angle and reach on the other one. Microsoft Mesh covers both ends of the spectrum. For example, on the one end of the spectrum, Mesh delivers the most accessible 3D presence delivering representative avatars with inside out sensors already on the devices that you carry. With the mesh platform, you can bring in your own avatars or the platform provides you a set of out of the box avatars that you can get started with. And on the other end of the spectrum, Mesh delivers the most photorealistic 360 holoportation with outside in sensors. And of course, the holy grail is to merge those worlds and that’s for the future. Presence also brings spatial audio. So even as you move in the space, the audio will be directionally emitted in the 3D space, consistent with your relative position and those of the participants in this session. And Jeremy, it’s truly magical and you have to experience it to understand the connection and engagement it delivers to your apps.
- And of course, beyond the movements of the avatar, you are also bringing in 3D models that can be manipulated in the space as well.
- Yes, Jeremy, that’s where Holographic rendering comes in. This allows you to bring in 3D models and provide the rendering power that you need when you need it. You can do local rendering optimized for the power and thermal sensitive devices. And with Microsoft Mesh, you can also leverage the power of the Cloud to bring in the compute required to render much more complex models that are very commonly used in industrial settings. And here, you can kind of see, as an example, of the difference in realism between 100,000 polygon and 100,000,000 polygon of the exact same model. This helps you supercharge your apps with high-quality interactive 3D content when every detail matters.
- So the spatial map capability really helps us know the space we are in and we can bring participants and content into the space through presence and rendering. Now, one aspect that really caught my attention in the demo was that all participants had a common perspective of the model and each other. So what made that possible?
- That was an insightful observation, Jeremy. This shared context is enabled by the multi-user sync capability, which is the magic that brings the entire experience together. I call it the TCP/IP of mixed reality collaboration. And that’s the spirit. This is what lights up any post updates from participants and any holographic transforms that you do in the 3D space. As you can see here with our engineers testing this out and using markers and passing them between each other. And look at the live action picture in picture view and how that’s in sync with the shared space at the bottom. And these are moving synchronously, and even though the participants are halfway across the globe, you have a sense of really being in the room together with them. And all of this happens in under 100 milliseconds of latency because of the smart compression that’s built into the platform.
- How would I harness this then if I’m a developer?
- So we’re building an SDK so you can create your own immersive mixed reality application. True to the spirit of cross platforms support, the SDK will be supported on a diverse set of tooling across popular middlewares. We have also deeply invested in a UX toolkit for Mesh. And it’s important cannot be overstated, it’s to provide several common UX constructs that you can pick and choose from so that you can build professional looking mixed reality applications very fast.
- To get started or sign up for the preview then, where do folks go to learn more?
- The Microsoft Mesh experience for HoloLens is in preview today. You can try it out for yourself by signing up at microsoft.com/mesh And if you are a developer, you’ll find more guidance at aka.ms/MeshDocs And to see the developer experience in more detail, check out aka.ms/MeshIgnite
- Thanks again, Simon. And for everyone watching, we’re really looking forward to seeing what you build on Microsoft Mesh. Of course, keep watching Microsoft Mechanics for the latest tech updates. Subscribe if you haven’t already, and we’ll see you soon!
