Q&A: Digital assistants at the service technicians’ disposal

Michael Braun: In short, what does  Twin Assist offer?  

Martin Jung: Twin Assist is a solution that visually guides users through procedures. Users can perform even complex service tasks quickly and safely without any special prior knowledge. Queries or searching through manuals for information are no longer necessary 

Michael Braun: How does it work?  

Martin Jung​​​​​​​:Our solution creates a virtual image of a machine or device and shows it to the user on AR glasses or a tablet. The good thing about this is that the user can then keep an eye on both: the real device and the virtual twin. And, of course, this results in many possibilities as well as advantages over other technologies. For example, I can rotate the virtual twin as I wish and look at it from all angles. I also have the option of zooming into view small details or zooming out on larger objects. In large systems, you must often get an overview of the problem – you shouldn't have to go through the entire system to do so.  

Michael Braun: How did the idea for this come about?  

Martin Jung​​​​​​​: We work together with our customer ASM Assembly Systems on innovative service concepts. Last year, we developed our concept for a digital assistance system. The solution was presented at the Service Congress 2018 in Munich, as well as at the tekom trade fair. Based on feedback we developed the approach further. The suggestions were essentially about usability and licensing costs.  

Michael Braun: How does it work?  

Martin Jung​​​​​​​:The solution creates a virtual image of a machine or device and shows it to the user on AR glasses or a tablet. The good thing about this is that the user can then keep an eye on both: the real device and the virtual twin. And, of course, this results in many possibilities as well as advantages over other technologies.  

Michael Braun: What are the key advantages over other concepts and technologies?  

Martin Jung​​​​​​​: Conventional AR applications use virtual overlays that are placed directly over a real object, such as to highlight certain parts. This shifts the view of reality a little. When the user reaches for a part marked in this way, their hand usually comes between the real object and the virtual overlay. They then see the overlay on the back of their hand and no longer on the object that should actually be highlighted. Our goal was to develop a solution suitable for everyday use that shows the information in the right place, and which also maps movement patterns without overlays covering anything. 

The second important point is the limitation of object detection in augmented reality. Currently, this does not work precisely enough for very small objects, e.g. small screws. Small deviations can cause a lot of confusion here.  

Michael Braun: What was the resulting consequence?  

Martin Jung​​​​​​​:The two requirements have led us to move away from overlays on the real environment. We asked ourselves, why couldn't we map a virtual twin of the device next to it. This means that the virtual highlights do not overlay the device but are instead displayed on the Assistance Twin. And gradually, we saw more and more advantages, such as turning and zooming in and out on the virtual object. In addition, the application places lower demands on the computing power of the AR glasses or tablet, thereby ensuring fluid performance. And, of course, the cost aspect is also very important to customers: Current challenge – 3D based tracking incurs high licensing fees. Our solution eliminates this and does not require expensive external licenses. 

Michael Braun: And where do you see specific potential applications for Twin Assist?  

Martin Jung​​​​​​​: As a digital assistance system, Twin Assist is designed to support the user in their specific tasks. These can be service technicians who need to perform maintenance. Or operators who need support for their diverse tasks in highly specialized environments. It can also be used in training, as essential processes can be taught directly in the environment.  

Michael Braun: How can the application work in practice?  

Martin Jung​​​​​​​:We have placed a great deal of importance on usability. This means that its use is extremely simple. Our customers can quickly learn how to navigate the environment themselves, regardless of whether AR glasses or a tablet are used. Interfaces to editorial systems also keep the data up to date. 

Michael Braun: What technical requirements must be met?  

Martin Jung​​​​​​​: It is easiest if we can use Computer Aided Design or CAD data, which is usually created by design departments. CAD data has been used for many places in technical documentation for years. However, even if CAD data may not be used for security reasons, we can create a virtual twin using 3D detection software and supplement the appropriate information. In order to then display the device or system, the user can either use a selection menu or scan a code on the machine. This loads Assistance Twin and all the associated information. A data connection is an advantage – but the solution provides an offline mode, allowing you to also be able to work independently in remote spaces. 

Michael Braun: Who can use Twin Assist? Is it a solution for service organizations in companies or do you also envision models for external service providers who can develop new business models with Twin Assist?  

Martin Jung​​​​​​​: A sales concept for Twin Assist is currently being worked on. It is important to understand that Twin Assist is only the technical frontend. The most important part is the creation and provision of modular information in the background. In other words, what is created in the technical editing department in an editorial system.