The Mixed Reality Program team at Trimble might know more about the commercial uses of mixed reality than anyone else.
Aviad Almagor
The core team was assembled under the Trimble umbrella to develop mixed-reality solutions for Microsoft’s HoloLens for the different markets the company serves (including construction, facilities management, mining, architecture, construction, and more). They work with the internal business units at Trimble, as well as customers within each industry—customers who get early access to Trimble’s tech to make sure that the software solutions they develop are delivering real, measurable benefits.
I caught up with the director of the program, Aviad Almagor, to discuss what he thinks about the most hotly anticipated technological development in years. Topics include: the challenges of a brand-new way of viewing data, what mixed-reality can already do for commercial users, what it doesn’t do so well, and the roadblocks we have to overcome before mixed-reality reaches its full potential.
Sean Higgins, SPAR3D: What are some of the challenges that Trimble faces designing products for mixed reality?
Aviad Almagor: Working in mixed reality from the UI/UX [user interface/user experience design] point of view is a totally different paradigm regarding the interaction with data. If you use a computer with a flat screen, you lay out your interface in a standard way. There are very strict rules about how to interact with data on a screen.
When you take this data into the physical environment, it is a new story. The experience we gained during years of development allowed us to create our method and best practices for what is working and what is not.
3D, mixed-reality interfaces present a new way to view and interact with data
I see a lot of coverage about how mixed reality can be used for remote collaboration, but I’m sure there’s a lot more that mixed reality can do. Could you talk about some of the other ways certain verticals are using the technology?
I can answer it by covering different major scenarios in the industries we serve. Let’s start with building.
So, I can look at the building lifecycle and say what how mixed reality can be utilized in the design, pre-construction, construction, and facilities management phases.
In the design, the weight will be on communicating design intent. It will be about collaborating during the early design phase with other stakeholders—from engineers, to owners, to the public.
Moving to the pre-construction phase, the focus can be training and communicating for the team on site. So they will build something that will aligned with the design intent.
Mixed reality in the design phase of a building project.
Moving to the construction stage, you can start using mixed reality to do QA and QC, comparing the digital model to the physical world on site, ensuring that what is being constructed is aligned with the digital model, and also monitoring progress. If I can compare a 4D digital model of a specific date to the current state on site, then I know what should be there in my schedule compared to what’s actually on site and I can say “Hey, I’m behind or I’m ahead of schedule.”
Then, we move to facility management, and data in context becomes the key. Here, you want to be able to use the same content you already created, content you’ve already invested in while creating the models. You want to project it into the environment and create a connection between the assets—the data sets, which you already have about any object in the environment—and the object itself. So if I’m looking at an object, I can see the history of the maintenance, the part I need to replace, and maybe a video which will help me do the work, and all of this will be in context of the physical environment.
Take another step forward, and connect it to IOT [internet of things]technology, and you get real-time data about the pressure in the pipe or the performance of the pump, or whatever kind of information is relevant for your task.
How about an industry like oil and gas? How would they use it?
In oil and gas, or even more critical sometimes, the facilities of nuclear plants, we are working with customers on exploring the way that they can use what we call digital training. Here, efficient maintenance is critical because for every day that the nuclear plant is not functioning, there is a significant cost.
So our customers would like to develop very precise mixed-reality training for the maintenance work they need to do so when they shut down the facility, so they can do the work efficiently without any issue. Those spaces are typically very dense and if you bring your ladders and equipment, you need to be sure you can actually put them there and access the asset that you need to maintain.
Remote collaboration with Microsoft’s HoloLens
How has adoption been so far among the customers you’re working with? Are people convinced? Are they seeing benefits already?
They are convinced. Customers are telling me that being able to visualize content—including complex structures in real 3D—and work around it with the owner is already a valuable thing today. Being able to do it with additional stakeholders that are located on the other side of the world is another great benefit.
I can also tell you that, in the pilots that we’re running today, customers are happy with what they can already do and pretty excited about what they will have in the future.
We do have some frustration and not everything is perfect. I’m saying to those customers, before we start—Hey, you should be ready to suffer a bit if you want to join this program. If not, you can wait three years, everything will be mature, and like everyone else, you’ll be able to utilize this technology.
So the technology is not exactly where you—or your customers—want it to be. How will it need to be improved? What are the bottlenecks that are keeping it from living up to its full potential?
Location-specific mixed-reality data.
Depends, of course, in what industry. The challenges that we are facing today, some of them relate to model size. The amount of information in the model today is growing exponentially. Those BIM models that are being used in the architectural world, or the point clouds datasets that we stream from mines are huge—really huge. And you need to have a very efficient tool to consume this data.
It’s not a smart move to take a huge BIM model—never mind if it’s in Revit or Tekla or a point cloud—and try to force it into HoloLens. HoloLens is a mobile device at the end of the day. Even a big computer will be challenged by this kind of data.
So the smart thing here is to define best practices and workflows for the user to handle the models, and also to develop technology that will allow us to stream the data. I believe that one year from now, the size of the model won’t matter anymore because we will be streaming what is relevant for the user based on his location and based on the context that he’s dealing with.
Given the recently growth in companies that are using deep learning and artificial intelligence, I want to ask, How do you think these technologies will affect the future of mixed reality?
I think that the combination will be mainly about the ability of those devices to work together, and to provide the user not just a tool to visualize but also a tool to make decisions and turn those decisions into something meaningful.
As an example, you could take artificial intelligence and provide information about regulations related to the building site, the envelope you can work in, the project budget, and maybe the desired style—say, Zaha Hadid style, or Gehry style—and click on a button to get an optimized design, then use mixed reality to see the output.
A detailed building model viewed on a tabletop.
What does the future of mixed-reality look like?
I’ll take you to the future, ten years from now. Imagine Microsoft HoloLens, or Google Tango technology, or Magic Leap not as these big glasses but as contact lenses. Imagine that you and I are sitting in the same space, but each of us is experiencing the space in a totally different way. I can see it with red walls because that’s what I like, and you can see it with purple walls because that’s what you like.
I can have Dali’s picture on the wall, and you will have a photograph from Cartier Bresson because that’s what you like. The mix between physical and digital will be all around us.
You will start to see more and more virtual objects in the space around you, and at some point people will start to ask—Is this real or virtual? Does it matter, if it provides the right experience? It’s a very interesting interaction that is not happening today, but will happen in the future.
The last question: Do you have a road map, or a sense for when we’ll see Trimble mixed reality products on the market?
I’m confident that we’ll see solutions on the market in a relatively short time. Some of the hardware devices are already available, or will be available on the market this year—see for example Meta, Google Tango technology on Lenovo smartphones, or Microsoft HoloLens.
Once the hardware is on the market, we will be able to provide matching software solutions.
