Jeannie Cahalane
IndEgo is redefining factory-floor collaboration with an AI assistant that sees what workers see. Developed at TU Berlin and Fraunhofer IPK, it uses smart glasses and multimodal AI to guide tasks, detect errors, and capture tacit knowledge in real time.
1. What inspired you to focus on AI-powered industrial assistance, and what problem in manufacturing were you most determined to solve?
Vivek Chavan: After finishing my Master thesis, I attended the ICCV conference, where I presented our work. I had the opportunity to talk with researchers and experts from around the world. I also experienced the cutting edge developments in AI methods, hardware, and infrastructure. One of the emerging technologies at the time was the use of smart, always-on devices and AI assistants, which were mostly being developed for everyday scenarios and a general audience. Since my lab focuses on industrial applications, we imagined how this technology and platform could be used for training, assisting and guiding skilled workers in the industry. The growing concern of skilled-worker shortage (and changing demographics) in Europe and Germany was a catalyst for us when kicking off our research. Several industrial tasks are heavily context dependent, which cannot be easily automated, programmed, or digitised, which we are addressing.
2. How did your experience at TU Berlin and Fraunhofer IPK shape your vision for a system that augments human workers rather than replacing them?
Vivek Chavan: My team and I have worked on various public and industry-funded projects involving the use of AI systems for quality assurance, recognition, and robotics. One common pattern we observe, is that humans (esp. Skilled workers) have great intuition and technical understanding of the products and processes they work on. Hence, we have actively focused on researching and developing systems that can help the workers and augment their expertise (e.g. catch defects in parts that they may overlook) and automate repetitive work. We have continued this effort with our research into this emerging technology. There has been a lot of focus on Industry 4.0 over the last few years, which aims to automate industrial tasks as much as possible. However, my team and I are looking ahead to Industry 5.0, a paradigm which aims to leverage human creativity and skills to establish an efficient and collaborative system.
3. What were the biggest technical challenges in using egocentric wearables like smart glasses to reliably detect human tasks, errors, and tool usage in a complex industrial environment?
Vivek Chavan: One of the most important design considerations for AI-based wearables, such as smart glasses, is that they should be comfortable enough to be worn all day. This has downstream implications, such as the weight, sensor quality and power consumption. Human movement (e.g. walking, head-shake) adds additional complications to the data captured by the sensors. The devices may also be needed to work reliably in different lighting conditions, noise, and immediate surroundings. Additionally, detecting tasks is challenging, because two humans may perform an activity in significantly different (but correct) ways. So a generalisable solution is needed, that takes such variations into account. Lastly, there are also privacy and data protection considerations, including personal information and working protocols.
4. What have been the key moments in your journey toward laying the groundwork for a deep-tech startup in this space?
Vivek Chavan: We started working on this topic last year. We have actively focused on an open and iterative approach, with continual feedback from the industry and the general audience. We had discussions with industries and workers, to understand what solutions they would like to see developed. Concurrently, we have been collecting data on different tasks, to develop, train, and evaluate AI-driven methods. One of the key challenges with going from research to a commercially viable business is to understand the product-market fit.
5. Can you walk us through how the IndEgo Assistant works? How does it process real-time video and audio to provide procedural guidance and flag mistakes?
Vivek Chavan: We are developing IndEgo Assistant around a modular and agentic framework. The approach builds on top of multimodal foundation models. These models have been trained on internet-scale data, have general-purpose intuition about the world and can be used to encode raw data from sensors into embeddings, which can be processed mathematically. We also collected a dataset specific to different industrial scenarios, such as assembly-disassembly, inspection, repair, logistics. This data serves as the finetuning and experimentation data for improving the performance of the models in industrial contexts. Additional details about the common tools, operations, working environments are also added as recognition and perception modules. Now let’s say the user wears a pair of AI glasses and switches on the assistant. The assistant actively observes the user’s actions and surroundings. When they ask for assistance, the assistant breaks down the context to get the correct information needed for the task. There is also a need to summarise and selectively remember the user’s experience, to save the most meaningful data. This can be reconstructed using a simple thought experiment: Let’s say instead of the AI assistant, there was a smart human reviewing the data stream from the camera, microphone, and helping the wearer. The AI assistant would need to do everything the human assistant would do consciously, as well as what they may do unconsciously to understand and process the information. We are also expanding the approach to fit a collaborative working environment, where the assistants of multiple workers can help each other without disclosing the private or sensitive data about the users.
6. What sets your spatially aware AI approach apart from other industrial training or quality assurance systems?
Vivek Chavan: Firstly, several industrial systems and workflows are poorly digitised. The current state of things lags significantly behind the latest developments in AI and industrial automation. Companies are often reluctant to implement new approaches due to uncertainty and no clear evidence of financial gains, at least in the short term. Our approach is meant to be scalable and non-invasive. The workers can perform their tasks and explain their processes in natural language by wearing a pair of research glasses. These help us understand the workflow and to develop solutions for specific use cases and scenarios. The data collected from subject-matter experts can be used for real-time training of other workers and for other downstream applications. The difference with the current training systems is that most of them work with traditional documentation methods, which requires additional effort to produce and use. In our approach, we focus on leveraging the multimodal sensor data (including camera, eye-gaze, motion) to extract meaningful context from the first-hand experience of the user. We are developing the approach as an augmentation to traditional quality inspection and assurance systems, which often use industrial cameras and focus on static image analysis with high precision.
7. How do you envision factories using the IndEgo Assistant to reduce assembly errors, lower training costs, and capture expert knowledge?
Vivek Chavan: Our approach is meant to serve the goals of Industry 5.0, leveraging human creativity and augmenting their skills. The assistant (and the wearable devices) can serve as a starting point for collecting data from the perspective of the experts. This data belongs to the worker/company and becomes an asset in the growing data economy. Since this is a research investigation, we are also focusing on developing flexible workarounds. For instance, workers who do not have access to a pair of AI glasses can use a smartphone, which can be a ‘good enough’ replacement for several tasks without additional investment. We are also factoring in the next generation of foundational AI models, which are likely to be smarter, faster, and better. The IndEgo assistant would be continually improving, as more data and context is provided and more user feedback is captured. With this strong base, workers can use the assistant for industrial tasks such as assembly, repair, logistics; for guidance and support.
8. Can you share a success story or a compelling use case that illustrates how IndEgo could transform a specific task, like complex repair or quality control?
Vivek Chavan: At the very start of our study, I was working with a team of students and industry workers to collect data on industrial tasks. One of these tasks was a disassembly of a mechanical frame, involving several sub-steps. I had prepared a document, with guidelines for the procedure. However, it failed to provide a clear understanding of the process, esp. for international students. We found an easy solution: I had disassembled the same device a few days ago, while wearing the pair of research AI glasses. Watching my experience, along with the narration, made it clear how the device was to be disassembled, and which precautions were necessary. This small experience gave us the confidence that we were on the right track and that this approach can be helpful in the real world.
A few months into our study, I had the opportunity to present our concept and idea at a well known industrial trade fair. We had not planned for the demo to be a prominent attraction. However, I was surprised by the amount of inquiries and interest in the idea. We received some crucial feedback and suggestions from industry experts, which we have been implementing since then. This also included some anecdotal inquiries whether ‘the technology could be bought or licensed in the near future’.
We recently started expanding our approach (capture expertise from workers and use it for downstream processing) to robotics; where the experts can easily collect data which can be transferred to collaborative robots for repetitive and challenging tasks. During the same week, we had a discussion with an industry partner, where they (without any prior knowledge about our new work) expressed, that it would be great if the knowledge from the experts could somehow be used directly for training robots. Although this is still in the early stages of development, we see significant application potential.
9. What challenges have you faced while developing the IndEgo Assistant, and how did you overcome them?
Vivek Chavan: One of the general systemic challenges for smart wearables and always-on devices is that they need to capture the wearer’s field of view, which may include sensitive data, information about other people and industrial work. Hence, we have actively been focusing on a privacy-preserving framework, which provides additional controls to the user and the organisation about which data is used and preserved. We had to establish strict protocols and restrictions for data collection at our premises. Another challenge was to ensure sufficient participant diversity and variety with the use cases and application scenarios in our data. We also understood the need for efficient approaches to summarise the user’s experience and distil down their experience, which becomes important over a longer period of use. We are currently addressing the latter via a research collaboration.
10. Looking back, are there any strategic decisions in your research that you would approach differently now?
Vivek Chavan: We have been actively thinking of possible application areas and developing solutions to a wide range of scenarios. However, it may have been quicker to just focus on one or two concrete use-cases, develop a proof of concept and gather quick feedback from users. The latter approach is more start-up adjacent. The circumstances at the start of our research limited our possibilities, so it is uncertain whether we could have tested a concept successfully and reliably at all. Secondly, we could have done certain things more thoroughly, avoided unnecessary work and avoided dead ends in research. However, such things only become clear in hindsight and cannot be controlled preemptively.
11. What does success look like for you and the IndEgo Assistant in the next 5–10 years?
Vivek Chavan: Our objective is to broaden our workflow to encompass and develop downstream applications for a variety of applications. This implies, that the data collected by the experts can be used for improving the assistance and guidance capabilities. We also expect real-time understanding of the procedure and detection of errors will improve, as the underlying models get smarter. Additionally, we plan to expand the application to include training of not just humans, but also collaborative robots and embodied agents. Our short-term goal is to contribute to open-source research, share our findings and resources with the community. We also want to actively collaborate with industries and skilled workers to gather real-world feedback and develop practical solutions. Lastly, we are looking into the commercial viability of this research, which is a long-term goal for us.
12. What does a typical day in your life as a researcher look like, and how do you stay motivated?
Vivek Chavan: I personally believe that when the work is interesting, valuable to others, and sufficiently challenging, motivation comes automatically. As a researcher, it is essential to align my interests with those of my lab and other colleagues. I prefer working in-person at my lab on most days, because it gives me the chance to talk to others and work with a stronger focus. I generally break down my general vision and overarching goals into smaller research questions and themes. I generally work with undergraduate/graduate students and interns, and prefer to talk with them every day and follow up on their work. As an AI researcher, it is crucial to keep pace with the exponential advancements in the field, including new publications, policy discussions, and even hype posts. I spend a few minutes each day looking at a curated list of new papers, and bookmark interesting ones for later. Brainstorming and discussing ideas with others is also quite helpful, esp. when collaborating on a project. I have a whiteboard that I use for this, which is always full of symbols, illustrations and new abstract questions. A significant portion of the day is also spent on coding or working on a repository to implement a particular solution. I like to ask my collaegues or PhD supervisor for spontaneous feedback to see whether my intuition about a particular solution is correct and sensible. A typical day also has meetings; I generally try to avoid unnecessary ones.
Editor’s Note
Vivek Chavan’s research represents a shift from full automation to human-AI teamwork. IndEgo is not replacing workers, but building a scalable way to support, train, and preserve expertise inside complex industrial workflows.
