This research develops a real-time inference engine to detect task steps and errors by integrating hand action recognition with IoT and IMU data. Multimodal inputs, including RGB-D, IMU, and smart device data, are combined with hand action inferences. A rule-based system powered by task graph monitors step completion criteria to track progress and identify the current step in AR-guided tasks. If a step is skipped, an intelligent AR system provides corrective guidance, improving task accuracy. The approach ensures seamless task progression, leveraging AR for error prevention and support in completing tasks effectively. This innovation enhances human performance in procedural workflows with real-time feedback and assistance.

This study explores challenges in human-machine interfaces (HMIs) for industrial machines, emphasizing usability issues caused by data overload and misalignment between HMIs and machine operations. To address these issues, an augmented reality (AR) interface integrated with IoT is proposed, enabling spatial and temporal alignment of data and controls with machines. Developed using a Kepware-ThingWorx-Vuforia pipeline and tested in a cyber-physical factory for cellphone assembly, the system was evaluated against Siemens HMIs using a HoloLens 2. A user study with 20 participants revealed that the AR interface significantly improved efficiency, task completion time, error rates, usability, and reduced mental load.

Understanding user actions from egocentric videos is vital for intelligent augmented reality (AR) systems. This paper presents a pipeline for egocentric hand action recognition tailored for AR applications. Leveraging an AR-guided data collection method, it eliminates manual annotation and introduces a skeleton-based model optimized for real-time AR scenarios. A case study on industrial precision inspection tasks validated the approach, generating a rich dataset and refining features for training. Extensive evaluations, including offline and real-time tests, demonstrated the model’s robustness and practicality, highlighting its potential for enhancing user interaction and system adaptation across diverse AR use cases.

We developed AR and VR systems for training in cold spray additive manufacturing and robotic painting, using Unity to provide task guidance through interactive dialog boxes. These systems differ in immersion levels and tool usage, leveraging multimodal data such as gaze and hand tracking to enhance visualization and interaction. Open-source documentation and files are available at xert.co, supporting adoption and customization. Designed to improve psychomotor skill learning, the systems enable data-driven cognitive load estimation, struggle detection, expertise estimation, and constructive design friction, showcasing AR and VR’s transformative potential for industrial training in complex manufacturing environments.

This research presents an Augmented Reality (AR) system that enhances intent communication between humans and collaborative robots during complex assembly tasks. Using a Unity-ROS pipeline, the system integrates multimodal input and output for dynamic human-robot interaction. User studies evaluated its impact on task performance and satisfaction, comparing AR-enabled scenarios with traditional methods. Results showed AR significantly improved communication clarity, reduced errors, and increased task efficiency. Immediate visual feedback minimized task completion times and errors while boosting user confidence and trust. These findings emphasize the critical role of transparent robot actions in achieving seamless collaboration.

Smart manufacturing and Industry 4.0 have advanced performance but often overlook the individuality of human workers. Our vision is to build a framework for human-technology symbiosis, inspired by Iron Man's AI assistant, J.A.R.V.I.S. The idea is to develop AI-driven systems that dynamically adapt to workers' unique physical, cognitive, and situational needs in real time. By transcending traditional group-based or binary classifications, these systems could personalize support, such as adjusting collaborative robot strength to physical capabilities or tailoring augmented reality guides to expertise and attention. This personalized approach aims to enhance performance, safety, and inclusivity in manufacturing. The framework outlines key research questions to enable these adaptive, human-centered technologies, paving the way for smarter, more supportive manufacturing environments.

This study explores the potential of augmented reality (AR) for adaptive, personalized on-the-job training by investigating the relationship between gaze behavior, visual attention, and expertise. Using eye tracking and computer vision, the research examines how novices and experts differ in fixation/saccade durations, attention allocation to action-relevant areas of interest (AOIs), scanpath transitions, and pupil size variations during AR-guided procedural tasks. By analyzing gaze, pupillometry, and egocentric video data in two tasks, the findings reveal how expertise influences visual attention and gaze interactions. This work advances personalized AR interventions, enhancing training and assistance in industrial applications.