Seeing the Strain: A New Dataset for Collaborative Robot Workloads

by

Author: Denis Avetisyan

Researchers now have access to a comprehensive resource for understanding how mental effort impacts human performance when working alongside robots in industrial settings.

A systematic preprocessing pipeline transforms raw recordings from ARIA smart-glasses into quantifiable eye-tracking metrics, enabling detailed analysis of visual attention.
A systematic preprocessing pipeline transforms raw recordings from ARIA smart-glasses into quantifiable eye-tracking metrics, enabling detailed analysis of visual attention.

GAZELOAD is a multimodal eye-tracking dataset designed to facilitate research into mental workload estimation during human-robot collaboration in realistic industrial assembly tasks.

Estimating cognitive load in complex industrial settings remains a challenge, hindering the development of truly adaptive human-robot collaboration systems. To address this, we introduce GAZELOAD, a multimodal dataset collected during realistic assembly tasks where participants interacted with collaborative robots while wearing eye-tracking smart glasses. This resource time-synchronizes ocular metrics-including pupil diameter, fixations, and gaze patterns-with environmental factors and task context, providing a comprehensive view of operator mental workload. Will this dataset facilitate the creation of more robust and reliable algorithms for monitoring and mitigating cognitive strain in future human-robot teams?

The Cognitive Cost of Collaboration

Successful human-robot teamwork fundamentally depends on managing the mental workload placed upon the human operator, a crucial element frequently underestimated in collaborative system design. When a human is tasked with overseeing a robotic partner, excessive cognitive demand – arising from monitoring, error detection, or unexpected situations – can lead to decreased performance, increased error rates, and ultimately, a breakdown in collaboration. Research indicates that a well-designed interface and a robot capable of anticipating human needs can significantly reduce this mental strain, allowing the operator to focus on higher-level tasks and maintain sustained, effective teamwork. Optimizing for minimal operator workload isn’t simply about making tasks easier; it’s about strategically distributing cognitive responsibilities to leverage the unique strengths of both human and robot, fostering a truly synergistic partnership.

Quantifying cognitive load during complex assembly tasks proves challenging for traditional methods, often relying on subjective self-reports or easily disrupted physiological measurements. These approaches frequently fail to capture the dynamic and nuanced mental effort required when humans collaborate with robots, particularly as tasks shift or unexpected events occur. Consequently, inefficiencies arise as operators become overloaded, leading to increased error rates and reduced overall performance. The limitations of current assessment tools hinder the design of truly collaborative systems, preventing the optimization of task allocation and hindering the development of robots capable of adapting to human cognitive states and proactively mitigating mental strain. A more granular and ecologically valid understanding of operator workload is therefore crucial for building effective and safe human-robot teams.

Participants interacted with a [latex]UR5[/latex] robot while wearing [latex]ARIA[/latex] smart glasses, and a separate station featured a [latex]Franka[/latex] Emika Panda robot equipped with assembly components for the experiment.
Participants interacted with a [latex]UR5[/latex] robot while wearing [latex]ARIA[/latex] smart glasses, and a separate station featured a [latex]Franka[/latex] Emika Panda robot equipped with assembly components for the experiment.

A Multimodal View of Operator State

A multimodal dataset was constructed to facilitate detailed analysis of operator state during task performance. This dataset integrates data from three sources: eye-tracking, event logs detailing operator actions, and ambient light measurements reflecting the work environment. Data was collected from 26 participants, providing a statistically relevant sample size. The combination of these modalities allows for a comprehensive understanding of not only what an operator is doing – as indicated by event logs – but also where their attention is focused and under what environmental conditions, offering a more holistic assessment than any single data source could provide.

Eye-tracking data within the dataset was collected using the Meta ARIA Smart Glasses, providing high-fidelity recordings of participant gaze. This data includes the identification of both saccades and fixations, which are key indicators of attentional focus during workload assessment. Saccades, or rapid eye movements, were specifically detected using an angular-velocity threshold of 30 degrees per second; any eye movement exceeding this velocity was classified as a saccade. The duration and frequency of fixations, periods of relatively stable gaze, were also recorded to further characterize visual attention patterns.

All data streams within the multimodal dataset are time-stamped according to the ISO 8601 standard, facilitating data integration and cross-modal synchronization. This standardized format – representing date and time as YYYY-MM-DDTHH:mm:ss.sssZ – ensures compatibility with a wide range of data processing tools and analysis platforms. Each of the 26 participants completed a total of 5 experimental sessions, resulting in a dataset structured around these discrete, time-bound occurrences and enabling both within-subject and between-subject analyses of operator state.

The dataset contains self-reported mental workload (MWL) scores from 1 to 10 aggregated across participants for each task block (T1-T5) alongside saccade counts measured in 250 ms windows per task block.
The dataset contains self-reported mental workload (MWL) scores from 1 to 10 aggregated across participants for each task block (T1-T5) alongside saccade counts measured in 250 ms windows per task block.

Decoding Workload Through Collaborative Assembly

Human-robot collaboration experiments were conducted to assess workload during assembly tasks. Participants worked alongside either a Universal Robots UR5 or a Franka Emika Panda robot on tasks designed with varying levels of complexity. These tasks were selected to represent a range of real-world assembly scenarios, allowing for the systematic investigation of cognitive load under different collaborative conditions. The experimental setup involved participants physically assembling components while the robot performed complementary actions, requiring coordination and shared workspace management. Data was collected from participants as they collaborated with each robot model to enable comparative analysis of workload metrics.

Workload manipulation during collaborative assembly experiments was achieved through two primary methods: workload grading and error injection. Workload grading involved presenting assembly tasks of increasing complexity, requiring more steps and finer motor skills. Concurrently, errors were intentionally introduced into the assembly process, necessitating participants to identify and correct mistakes, thereby increasing cognitive load. This combination of graded complexity and induced errors created a spectrum of conditions designed to systematically demand varying levels of cognitive effort from the participants as they collaborated with robotic systems.

Analysis of collected data demonstrates statistically significant correlations between specific eye-tracking metrics and the cognitive workload experienced during collaborative assembly tasks. Specifically, increased fixation duration and saccade frequency were observed in conditions designed to elevate cognitive demand, such as those involving higher task complexity or injected errors. Data processing utilized a minimum fixation duration threshold of 60 milliseconds to exclude spurious events, and adjacent fixations within 75 milliseconds were merged to accurately represent sustained visual attention. These parameters were critical for isolating meaningful variations in eye movement patterns corresponding to differing levels of cognitive effort during human-robot collaboration.

Open Science and a Future of Adaptive Collaboration

A comprehensive multimodal dataset, capturing human operator state during collaborative tasks with a robot, has been released under a Creative Commons Attribution License. This deliberate act of open access empowers researchers worldwide to validate findings, build upon existing work, and explore novel avenues in human-robot interaction. By providing detailed physiological, behavioral, and environmental data, the dataset facilitates the development of more robust and adaptable collaborative systems. This commitment to reproducibility and data sharing is expected to significantly accelerate progress in understanding and optimizing human performance within complex human-robot teams, fostering innovation and broadening the scope of potential applications.

This research establishes a crucial groundwork for the creation of adaptive, intelligent systems in human-robot collaboration. By accurately gauging an operator’s workload in real-time-considering physiological signals and task performance-these systems can dynamically modify task complexity or the level of robotic assistance provided. This isn’t simply about increasing efficiency; it’s about creating a symbiotic relationship where the robot proactively adjusts to maintain an optimal balance between human effort and automated support. Such responsiveness minimizes cognitive strain, reduces the risk of errors, and ultimately fosters a more sustainable and effective collaborative workflow, paving the way for robots that truly augment human capabilities rather than simply automating tasks.

Investigations are now turning towards leveraging the collected multimodal data within machine learning frameworks, aiming to proactively anticipate and address imbalances in operator workload during human-robot collaboration. This involves developing predictive models capable of identifying workload surges or deficits before they impact performance or contribute to operator fatigue. By integrating physiological signals, behavioral data, and task parameters, these models will enable intelligent systems to dynamically adjust task complexity or the level of robotic assistance offered, effectively creating a collaborative partnership that optimizes both efficiency and user well-being. Ultimately, this research seeks to move beyond reactive assistance to a proactive, adaptive system that safeguards operator health and maximizes collaborative potential.

The creation of GAZELOAD embodies a pursuit of essential data, stripping away extraneous variables to isolate the core elements of mental workload in human-robot collaboration. This resonates with Andrey Kolmogorov’s assertion: “The most important things are the most difficult to express.” The dataset’s focus on realistic industrial assembly, coupled with multimodal eye-tracking, attempts to capture this complexity with parsimony. By providing a curated, publicly available resource, researchers are empowered to build models that discern cognitive load not through sheer volume of data, but through precisely measured, relevant indicators-a clear demonstration of beauty achieved through lossless compression of information.

The Road Ahead

The provision of a dataset, however meticulously assembled, merely shifts the locus of difficulty. The true challenge does not reside in having more signals – gaze patterns, contextual data – but in discerning which of those signals genuinely reflect cognitive state, and which are simply noise. Current metrics of mental workload, even those informed by physiological data, often mistake activity for load. A focus on parsimony – on identifying the minimal sufficient set of indicators – will be essential.

Future work must move beyond the laboratory echo chamber. While this dataset represents a commendable step towards ecological validity, the complexities of real-world industrial environments will always exceed any simulated proxy. The field needs to confront the inherent ambiguity of human performance; a worker’s slowed response might indicate high workload, or simply fatigue, distraction, or a reasoned assessment of risk. Disentangling these factors requires not just more data, but a more nuanced theoretical framework.

Ultimately, the goal isn’t to predict workload with ever-increasing accuracy, but to understand its relationship to performance and well-being. A truly useful system will not simply flag ‘high workload,’ but will offer actionable insights – suggesting task re-design, or prompting restorative breaks. The subtraction of complexity from the work itself, rather than its mere measurement, remains the most elegant solution.

Original article: https://arxiv.org/pdf/2601.21829.pdf

Contact the author: https://www.linkedin.com/in/avetisyan/

See also:

2026-02-01 15:35