Two Hands Are Better Than null: Enhancing Assembly with Supported Teleoperation

by

Author: Denis Avetisyan

A new framework combines human skill with robotic assistance to simplify complex assembly tasks and reduce operator fatigue.

The Supported Teleoperation framework streamlines assembly tasks by first assessing the required actions and formulating a plan, then providing dynamic assistance through both visual guidance and precise motion adjustments.
The Supported Teleoperation framework streamlines assembly tasks by first assessing the required actions and formulating a plan, then providing dynamic assistance through both visual guidance and precise motion adjustments.

This paper presents SUBTA, a system for supported user-guided bimanual teleoperation leveraging task planning and scene understanding for improved performance in structured assembly.

Achieving effective human-robot collaboration requires overcoming the challenges of intuitive control and reduced cognitive burden. This paper introduces SUBTA-a framework for Supported User-Guided Bimanual Teleoperation in Structured Assembly-designed to enhance performance in complex assembly tasks. Through the integration of learned intention estimation, scene-graph task planning, and context-aware motion assistance, SUBTA significantly improves both positioning accuracy and reduces mental demand, as demonstrated by a user study with [latex]\mathcal{N}=12[/latex] participants. Can this approach pave the way for more seamless and efficient human-robot teams in increasingly intricate assembly and manipulation scenarios?

The Burden of Remote Manipulation: A Cognitive Challenge

Intricate assembly tasks performed through traditional teleoperation present a significant challenge due to the substantial cognitive burden placed on the human operator. Successfully manipulating objects remotely requires constant visual processing, spatial reasoning, and precise motor control, quickly overwhelming the operator’s capacity, especially in scenarios with limited tactile feedback. This cognitive load is further exacerbated by restricted situational awareness; conventional teleoperation systems often provide a narrow or distorted view of the workspace, hindering the operator’s ability to accurately assess distances, orientations, and potential obstructions. Consequently, assembly processes become slower, more prone to errors, and demand considerable mental effort, ultimately limiting the feasibility of complex remote manipulation in fields like space exploration, hazardous material handling, and precision manufacturing.

Current remote manipulation techniques often falter when faced with the unpredictable nature of real-world environments and the demands of delicate assembly. Existing robotic systems, even those controlled by skilled operators, struggle to maintain the necessary precision when dealing with variations in part placement, unexpected obstacles, or the lack of clear visual cues. This limitation stems from a reliance on pre-programmed motions and a difficulty in dynamically adjusting to unforeseen circumstances. Consequently, intricate tasks – such as assembling electronics, performing microsurgery, or repairing equipment in hazardous locations – become significantly more challenging, requiring substantial operator effort and increasing the risk of errors. The inability of these methods to seamlessly adapt to unstructured settings highlights a critical need for more robust and intelligent robotic assistance.

Addressing the difficulties inherent in complex remote assembly demands a fundamental shift towards systems that augment human skill rather than simply mirroring it. Current limitations stem not from a lack of operator dexterity, but from the cognitive burden of translating visual information into precise motor commands while simultaneously maintaining spatial awareness. Consequently, research focuses on intelligent assistance – incorporating features like predictive algorithms that anticipate necessary actions, automated error detection and correction, and haptic feedback that conveys nuanced force information. Equally crucial is the development of intuitive interfaces, moving beyond traditional joystick and screen setups to leverage augmented or virtual reality, gesture control, and even brain-computer interfaces. These advancements aim to create a symbiotic relationship between operator and machine, enabling efficient, accurate, and safe manipulation of objects in challenging remote environments.

The persistence of limitations in teleoperation technology directly impacts the feasibility of complex remote assembly tasks. Currently, these operations are considerably slower than direct manual assembly, owing to delays inherent in remote control and the cognitive burden placed upon the operator. Furthermore, the lack of precise control and real-time feedback increases the likelihood of errors, potentially damaging components or requiring costly rework. In environments involving hazardous materials or inaccessible locations – such as space exploration or nuclear facility maintenance – these errors aren’t merely inconveniences; they present significant safety risks to both the operator and the surrounding environment, underscoring the urgent need for innovative advancements in remote manipulation capabilities.

A teleoperation system leverages a digital twin environment, visualized with task estimation and planning [highlighted in red], to guide a user in remotely assembling a structure, as demonstrated by successful block assembly and a scene graph encoding spatial relationships.
A teleoperation system leverages a digital twin environment, visualized with task estimation and planning [highlighted in red], to guide a user in remotely assembling a structure, as demonstrated by successful block assembly and a scene graph encoding spatial relationships.

SUBTA: Intelligent Assistance Through Shared Autonomy

SUBTA’s implementation of shared autonomy distributes control between a human operator and automated systems during assembly tasks. This approach allows the operator to focus on complex decision-making and exception handling, while automated components manage repetitive or physically demanding sub-tasks. The system dynamically adjusts the level of automation based on task context and operator input, ensuring a fluid and intuitive workflow. Consequently, shared autonomy in SUBTA demonstrably reduces operator cognitive load and physical strain, while simultaneously increasing overall assembly speed and precision compared to traditional manual or fully automated methods.

SUBTA utilizes graph-based task planning to represent complex assembly sequences as a network of interconnected steps, where nodes represent individual actions and edges define their dependencies and preconditions. This allows the system to decompose large tasks into smaller, more manageable sub-tasks, facilitating optimized trajectory generation for robotic assistance. The graph structure enables proactive error anticipation by modeling potential failure modes associated with each step and pre-calculating alternative paths or recovery strategies. By analyzing the graph, SUBTA can dynamically adjust the assembly sequence based on real-time task state estimation, ensuring efficient and robust completion even in the presence of unexpected disturbances or component variations.

SUBTA’s task state estimation capabilities utilize sensor data – including visual feedback and force/torque readings – to continuously monitor the assembly process and build a real-time representation of the system’s status. This allows the system to determine the position and orientation of parts, identify completed sub-assemblies, and detect anomalies such as misalignments or missing components. By maintaining this dynamic understanding, SUBTA can proactively offer assistance – for example, guiding the operator through difficult maneuvers – and, in the event of an error, automatically initiate recovery procedures like suggesting corrective actions or halting the process to prevent damage. The estimation process employs probabilistic models and filtering techniques to handle sensor noise and uncertainty, ensuring robust performance even in challenging environments.

SUBTA is designed to extend the scope of remote assembly operations beyond the limitations of traditional teleoperation systems. Conventional remote assembly is often restricted to simpler tasks due to challenges in maintaining precision, adapting to unforeseen circumstances, and managing complex sequences. By integrating shared autonomy, graph-based task planning, and real-time task state estimation, SUBTA addresses these limitations. This allows the system to perform assemblies requiring a high degree of dexterity, adaptability, and error recovery, effectively enabling remote completion of tasks previously requiring on-site personnel and considered impractical or impossible via conventional teleoperation methods.

During user testing, participants successfully completed three assembly tasks using the SUBTA system, which projects a supported workspace alongside the actual robot workspace to aid in manipulation.
During user testing, participants successfully completed three assembly tasks using the SUBTA system, which projects a supported workspace alongside the actual robot workspace to aid in manipulation.

Scene Understanding: The Foundation of Intelligent Assistance

The System for Understanding-Based Task Assistance (SUBTA) employs a scene graph as a core component for representing the assembly environment. This graph structures the relationships between objects – parts, tools, and the operator – by defining their spatial positions and connections. Each node in the graph represents an object, while edges define the spatial relationships between them, such as adjacency, containment, or support. This structured representation enables SUBTA to accurately monitor the assembly process by tracking object positions and orientations relative to each other. Furthermore, the scene graph facilitates task planning by allowing the system to reason about feasible actions and predict the consequences of different operations based on the defined spatial relationships, ultimately supporting automated guidance and error detection.

Task state estimation within the system leverages both Graph Neural Networks (GNNs) and HAR-Transformer models to interpret visual data and anticipate operator actions. GNNs process information derived from the scene graph, representing objects and their relationships, to understand the current assembly status. Simultaneously, HAR-Transformer models, utilizing hierarchical attention mechanisms, analyze visual inputs to predict the operator’s subsequent steps. This dual approach allows for a more robust and accurate assessment of the task’s progression, factoring in both the observed environment and anticipated operator behavior. The combined output provides a probabilistic estimation of the current task state, enabling proactive assistance and error detection.

The system incorporates motion support and visual feedback mechanisms to aid operators during assembly tasks. This is achieved through real-time guidance, often displayed via augmented reality, which highlights the next required action and the correct trajectory for tool movement. Visual feedback includes dynamically updating cues indicating positional accuracy and force application, enabling operators to perform precise manipulations. The integration of these supports aims to reduce cognitive load, minimize errors, and improve overall assembly efficiency by providing immediate and actionable information regarding task execution.

Augmented reality (AR) and digital twin technologies contribute to enhanced situational awareness during assembly processes by superimposing virtual information onto the operator’s view of the real-world environment. A digital twin, a virtual replica of the assembly, provides a dynamic and accurate representation of the current state, while AR delivers this information directly to the operator, often through a head-mounted display or tablet. This real-time visualization includes projected assembly steps, highlighted components, and potential collision warnings, enabling operators to understand the context of their actions and perform tasks with greater accuracy and efficiency. The integration of these technologies allows for a continuous feedback loop between the physical and virtual worlds, improving overall assembly performance and reducing errors.

The task planner determines the next block's target pose by minimizing the graph edit distance and incorporating the resulting node into the planning graph.
The task planner determines the next block’s target pose by minimizing the graph edit distance and incorporating the resulting node into the planning graph.

Validating SUBTA: Performance and the Human-System Interface

Operator perceptions of the Semi-Autonomous Universal Teleoperation Architecture (SUBTA) consistently indicate high usability, as evidenced by strong scores on the System Usability Scale (SUS). This isn’t merely a matter of convenience; the system’s intuitive design directly supports efficient task completion. Operators readily adapt to SUBTA’s interface, minimizing the cognitive burden associated with learning a new control scheme and allowing them to focus on the complexities of the task at hand. The consistently positive SUS results suggest that SUBTA’s usability isn’t a coincidental benefit, but a core feature contributing to improved performance and reduced operator fatigue during teleoperated procedures.

Assessments utilizing the NASA Task Load Index (TLX) demonstrate that the SUBTA system substantially alleviates the burdens placed upon operators during remote tasks. Compared to traditional teleoperation methods, SUBTA significantly reduces both the mental and physical demands experienced by users, fostering a less fatiguing and more efficient work environment. The NASA-TLX, a validated metric for subjective workload, revealed a marked decrease in perceived effort, performance, frustration, and temporal demand when operators utilized SUBTA, indicating a more streamlined and intuitive interface. This reduction in workload not only enhances operator comfort but also has the potential to minimize errors and improve overall task performance, particularly in scenarios requiring sustained concentration or physical dexterity.

Evaluations of the Semi-Autonomous Universal Bedside Teleoperation Assistant – or SUBTA – demonstrate a marked increase in operational success; during testing, the system facilitated task completion 75% of the time. This represents a substantial advancement over previously employed methods, which achieved a success rate of only 55.6%. The improvement signifies SUBTA’s ability to reliably guide operators through complex procedures, reducing instances of failure and increasing the likelihood of positive outcomes in challenging remote tasks. This heightened reliability is crucial for applications requiring consistent performance, minimizing errors, and maximizing efficiency in critical situations.

Evaluations utilizing the NASA Task Load Index (NASA-TLX) revealed a substantial decrease in perceived cognitive demand when operators employed the system, with scores falling from 6.2 to 3.4. This represents a marked reduction in mental effort required to complete the assigned tasks, suggesting the system effectively offloads cognitive burden from the operator. The NASA-TLX assesses workload across six subscales – mental demand, physical demand, temporal demand, performance, effort, and frustration – providing a comprehensive measure of perceived strain. This significant shift indicates the system not only improves task efficiency but also fosters a less fatiguing and more comfortable operational experience, potentially minimizing errors associated with cognitive overload during prolonged or complex procedures.

Evaluations demonstrate that SUBTA significantly elevates the precision of remote manipulation tasks, achieving twofold improvements in pose accuracy when contrasted with traditional teleoperation methods. Statistical analysis, including t-tests with 99 degrees of freedom, confirms these gains are highly significant – position accuracy improved markedly [latex]t(99)=7.88, p<0.001, d=1.18[/latex], and orientation accuracy saw even greater enhancement [latex]t(99)=12.67, p<0.001, d=1.75[/latex]. These findings indicate that SUBTA not only enables operators to complete tasks, but also to do so with substantially reduced error and increased fidelity, crucial for applications demanding exacting control and minimal deviation from intended outcomes.

The demonstrated capabilities of SUBTA suggest a paradigm shift in how operators approach complex assembly tasks. Rigorous testing indicates the system not only improves task completion rates, but also significantly lessens the cognitive and physical strain experienced by users – a crucial factor in maintaining consistent performance over extended durations. By reducing both mental workload and the potential for errors, SUBTA minimizes the risk of costly mistakes and enhances overall operational safety. This combination of increased efficiency and decreased operator fatigue positions SUBTA as a valuable asset in demanding environments where precision and reliability are paramount, potentially revolutionizing procedures across industries like space exploration and remote maintenance.

The capabilities demonstrated by SUBTA extend far beyond the laboratory, promising substantial advancements across multiple demanding fields. In space exploration, the system could enable more efficient and safer remote operation of robotic assets, facilitating complex tasks on distant planets or in orbit with minimized astronaut workload. Similarly, hazardous environment remediation – such as nuclear disaster cleanup or deep-sea intervention – stands to benefit from SUBTA’s ability to reduce operator cognitive load and enhance precision, protecting personnel from direct exposure to danger. Furthermore, the technology offers compelling solutions for remote maintenance in industries like energy and infrastructure, allowing skilled technicians to perform intricate repairs and inspections from a safe distance, reducing downtime and costs. These applications highlight SUBTA’s versatility and potential to redefine how work is performed in environments where human presence is either impractical or perilous.

NASA-TLX ratings reveal no significant differences in mental or physical demand, or frustration levels, across the three modes, with consistently high System Usability Scale (SUS) scores indicating good usability.
NASA-TLX ratings reveal no significant differences in mental or physical demand, or frustration levels, across the three modes, with consistently high System Usability Scale (SUS) scores indicating good usability.

The presented SUBTA framework embodies a principle of refined efficiency. It meticulously addresses the cognitive burden inherent in bimanual teleoperation, distilling complex assembly tasks into manageable components. This echoes Donald Davies’ sentiment: “Simplicity is the key to reliability.” SUBTA isn’t merely automating actions; it’s strategically removing unnecessary mental steps for the operator through scene graph understanding and motion support. The system’s focus on shared autonomy exemplifies a commitment to paring away complexity, allowing the human operator to focus on higher-level task direction rather than low-level control. This deliberate reduction of cognitive load represents a move toward elegant, robust functionality.

Future Directions

The presented work, while demonstrating a functional architecture for supported bimanual teleoperation, merely addresses the superficial constraints of the problem. True progress lies not in adding layers of assistance, but in understanding why such assistance is persistently required. The system effectively mitigates cognitive load, but fails to confront the inherent clumsiness of remote manipulation. Future iterations should prioritize minimizing the disparity between human intent and robotic execution – a reduction, not an augmentation, of complexity.

A critical limitation resides in the system’s reliance on structured environments and pre-defined assembly sequences. The real world, predictably, offers neither. Extending SUBTA to operate in unstructured settings demands a shift from task planning to real-time adaptation. This necessitates a more nuanced understanding of affordances and a willingness to relinquish precise control – to allow the robot to infer the user’s goals, rather than slavishly follow explicit instructions.

Ultimately, the field must confront the question of what constitutes ‘telepresence’. Is it simply the remote control of a body, or the transmission of skill? The pursuit of the latter requires a deeper integration of human and robotic capabilities, blurring the lines between operator and machine. The true measure of success will not be the complexity of the system, but its ability to disappear entirely, leaving only the task itself.

Original article: https://arxiv.org/pdf/2603.10459.pdf

Contact the author: https://www.linkedin.com/in/avetisyan/

See also:

2026-03-12 13:39