Two Hands Are Better Than One: Mastering Robotic Dexterity with Teleoperation

by

Author: Denis Avetisyan

A new framework, UniBiDex, streamlines the control of robots performing complex, two-handed tasks, paving the way for more effective robot learning and data acquisition.

The proposed teleoperation framework unifies diverse input commands and both task objectives and safety limitations into a singular optimization problem to achieve precise bimanual control.
The proposed teleoperation framework unifies diverse input commands and both task objectives and safety limitations into a singular optimization problem to achieve precise bimanual control.

UniBiDex is a unified teleoperation system for bimanual robotic manipulation, supporting both VR control and leader-follower modalities with improved precision and robustness.

Achieving robust and intuitive control remains a key challenge in robotic bimanual manipulation. This paper introduces ‘UniBiDex: A Unified Teleoperation Framework for Robotic Bimanual Dexterous Manipulation’, a novel system designed to streamline teleoperation via both virtual reality and leader-follower input. UniBiDex integrates diverse input devices with a shared control stack, leveraging null-space control for smooth, collision-free motions and improved task success. By releasing its hardware and software as open-source, will this framework accelerate the collection of high-quality data and unlock new avenues for robot learning and dexterous manipulation?

The Challenge of Dexterous Remote Manipulation

Traditional teleoperation, the remote control of robotic systems, often falters when tasked with activities requiring two hands – bimanual manipulation. This difficulty isn’t merely one of complexity; it stems from the inherent challenges of translating human dexterity to a robotic platform. Operators experience significant cognitive load as they attempt to coordinate the movements of two robotic arms, mirroring the intricate interplay of their own hands and fingers. This constant mental effort quickly leads to fatigue, diminishing performance and increasing the likelihood of errors. Consequently, tasks that are simple for a human, such as assembling small parts or manipulating deformable objects, become time-consuming and demanding when performed remotely, highlighting the need for more intuitive and less strenuous control schemes.

The simultaneous control of two robotic arms – bimanual teleoperation – presents significant hurdles in achieving both intuitive operation and guaranteed safety. Successfully coordinating the movements of two arms requires overcoming the complexities of redundant kinematics and ensuring motions are synchronized and purposeful, rather than jerky or conflicting. Crucially, a robust safety system is needed to prevent collisions – not only between the robotic arms themselves, but also with the surrounding environment or the operator. Furthermore, providing the operator with adequate feedback – visual, haptic, or force – is essential for maintaining a sense of presence and control, allowing for delicate manipulation and precise task completion. Without these advancements in coordination, safety protocols, and sensory feedback, bimanual teleoperation remains a challenging endeavor, limiting the potential of robotic systems in complex real-world applications.

Despite advancements in robotic teleoperation, current systems such as Bunny Vision Pro and GELLO demonstrate limitations when faced with tasks requiring fine motor skills and coordinated bimanual action. These platforms often struggle with the nuanced movements necessary for intricate manipulation, resulting in jerky motions or an inability to perform complex assembly or surgical procedures. While offering a foundational level of control, they lack the dexterity to replicate the full range of human hand movements and the seamless coordination between both hands. This deficiency stems from challenges in accurately translating operator input into robotic action, as well as difficulties in providing sufficient sensory feedback to the user, hindering precise and intuitive control for demanding applications.

A virtual reality-based teleoperation workflow enables a user to remotely tidy a household kitchen environment.
A virtual reality-based teleoperation workflow enables a user to remotely tidy a household kitchen environment.

UniBiDex: A Unified Architecture for Bimanual Control

UniBiDex addresses limitations in current teleoperation systems by consolidating control and safety functionalities into a unified framework. Existing systems often treat these aspects as separate modules, leading to integration issues and reduced performance. This new approach integrates control algorithms – allowing for operator input and robot movement – with safety constraints that prevent collisions or exceeding joint limits. By combining these features, UniBiDex aims to provide a more robust, intuitive, and reliable teleoperation experience, particularly in complex or sensitive tasks where precise control and inherent safety are paramount. This unification simplifies system architecture and facilitates more seamless operation across diverse robotic platforms and input devices.

UniBiDex employs redundant 7-Degree of Freedom (7-DoF) robotic arms to enhance manipulation capabilities in teleoperation tasks, providing increased workspace accessibility and dexterity compared to 6-DoF systems. To accommodate a variety of input devices – including haptic joysticks, motion trackers, and data gloves – the framework utilizes a Virtual Base Frame (VBF). The VBF serves as a standardized coordinate system, abstracting device-specific kinematics and scaling input data to a common representation. This decoupling allows UniBiDex to seamlessly integrate diverse input modalities without requiring device-specific control algorithms, simplifying system configuration and improving usability.

UniBiDex achieves simultaneous Cartesian tracking and safety through the combined application of Inverse Kinematics (IK) and Null-Space Control. IK is utilized to determine joint angles required to position the end-effectors along a desired Cartesian trajectory, enabling precise positioning and tracking of the robot’s hands. Null-Space Control is then implemented within the IK solution to modulate joint velocities, optimizing for secondary objectives such as obstacle avoidance and maintaining a safe operational distance from the user or environment. This approach allows for redundant manipulators – specifically 7-DoF arms – to satisfy both kinematic task requirements and simultaneously enforce safety constraints without compromising tracking performance; the null space represents the set of joint velocities that do not affect the end-effector’s position, allowing for constraint fulfillment without altering the primary task.

UniBiDex achieves comparable performance using either virtual reality or leader-follower arm control modalities.
UniBiDex achieves comparable performance using either virtual reality or leader-follower arm control modalities.

Enhancing Presence and Control Through Advanced Feedback

UniBiDex employs Virtual Reality headsets to achieve precise, multi-Degree-of-Freedom (multi-DoF) tracking of the operator’s head and hand movements. This tracking data is then used as direct input for controlling the robotic arms, enabling an intuitive control scheme. The system captures six degrees of freedom – three for positional tracking (x, y, z) and three for rotational tracking (pitch, yaw, roll) – for both the operator’s head and hands. This high-fidelity tracking minimizes the cognitive load on the operator, as the robotic arm movements correspond directly to their natural movements, increasing efficiency and reducing the potential for errors during teleoperation.

The UniBiDex system employs a Leader-Follower arm configuration as its primary input method, enabling direct, kinesthetic control of the robotic arms. This modality utilizes isomorphic motion mapping, meaning movements performed by the operator’s arms are directly translated to the robot’s arms with a one-to-one correspondence. Crucially, this configuration also delivers valuable Force Feedback; sensors within the robotic arms measure interaction forces, which are then relayed back to the operator via the Leader arms, providing a sense of the forces experienced during manipulation. This bidirectional force exchange enhances the operator’s awareness of the robot’s interaction with the environment and allows for more precise and intuitive control.

Haptic feedback in the UniBiDex system transmits force information from the robotic arms to the operator, providing kinesthetic awareness of the interaction between the robot and its environment. This is achieved through actuators that apply forces to the operator’s hands, replicating the magnitude and direction of forces experienced by the robot during manipulation tasks. The resulting sensory input allows operators to perceive contact, resistance, and weight, improving their ability to perform delicate manipulations, avoid collisions, and accurately assess the stability of grasped objects. This enhanced awareness reduces reliance on visual feedback and enables more precise and efficient teleoperation, particularly in situations with limited visibility or complex interactions.

The leader-follower arm teleoperation workflow successfully completes a household kitchen-tidying task by sequentially performing item unpacking, shelf organization, towel folding and placement, and clamp attachment.
The leader-follower arm teleoperation workflow successfully completes a household kitchen-tidying task by sequentially performing item unpacking, shelf organization, towel folding and placement, and clamp attachment.

Towards Autonomous Dexterity: Learning from Human Expertise

The UniBiDex robotic system generates valuable datasets through its teleoperation interface, which are ideally suited for the implementation of Imitation Learning techniques – a powerful approach within the broader field of Robot Learning. This method allows robots to acquire intricate manipulation skills by observing and replicating human demonstrations, circumventing the need for painstakingly designed, hand-coded control policies. Essentially, the robot learns by example, analyzing the movements and actions of a human operator to build its own internal model for performing the task. The richness and detail captured through UniBiDex’s teleoperation provide the necessary information for the robot to effectively learn these complex behaviors, paving the way for more adaptable and intuitive robotic systems.

Robots traditionally require painstakingly designed control policies – detailed instructions for every possible action – to perform even simple tasks. However, a paradigm shift is occurring through the application of Imitation Learning. This approach allows robots to learn complex manipulation skills simply by observing human demonstrations. By analyzing the movements and actions of a human expert, the robot builds its own internal model of how to perform the task, effectively bypassing the need for explicit programming. This not only accelerates the development process but also enables robots to tackle tasks that are difficult to define algorithmically, opening doors to more adaptable and intelligent robotic systems capable of handling nuanced and unpredictable environments.

Recent advancements in robotic manipulation, exemplified by the UniBiDex system, reveal significant progress in bimanual task performance. Through learning from demonstration, the system achieved a 60% success rate in virtual reality (VR) mode and an impressive 75% in leader-follower mode-outperforming traditional robotic control methods. This improvement isn’t merely about accomplishing tasks, but doing so reliably and safely; the implementation of Optimal Reference Configurations plays a crucial role in establishing consistent and predictable robotic behaviors. By defining safe operational boundaries and stable postures, these configurations ensure that learned manipulations are not only successful but also adaptable to a variety of real-world scenarios, paving the way for more robust and autonomous robotic systems.

A greater range of motion in the right arm joint, as shown in (c), enables more compliant control compared to the more restricted configuration in (b).
A greater range of motion in the right arm joint, as shown in (c), enables more compliant control compared to the more restricted configuration in (b).

UniBiDex’s architecture emphasizes a holistic approach to bimanual robotic control, mirroring the interconnectedness of complex systems. The framework doesn’t merely address individual kinematic challenges; it establishes a unified structure for diverse teleoperation modalities. This echoes Ada Lovelace’s observation: “The Analytical Engine has no pretensions whatever to originate anything. It can do whatever we know how to order it to perform.” UniBiDex, similarly, doesn’t autonomously ‘solve’ manipulation; it translates human intention-the ‘order’-into precise robotic action. The system’s success hinges on a well-defined structure allowing for robust translation across both virtual and physical spaces, embodying a principle where structure dictates behavior.

Beyond the Hands: Future Directions

The elegance of UniBiDex lies in its attempt to unify control paradigms, yet the system, like any meticulously crafted interface, reveals the limits of its assumptions. The framework addresses the immediate challenge of bimanual teleoperation, but a truly robust system must account for the inherent unpredictability of the environment. Simply improving precision within a virtual or leader-follower scheme does not resolve the problem of what to do when the anticipated interaction fails. A slight perturbation in the real world, an unforeseen contact, will invariably expose the brittle nature of even the most sophisticated kinematic calculations.

Future work should therefore shift focus from the mechanics of control to the architecture of perception. The current emphasis on inverse kinematics and null-space control, while valuable, treats the robot as an isolated actuator. A more fruitful approach acknowledges that manipulation is fundamentally a dialog – a constant exchange of information between the robot, the environment, and the operator. Improved sensory integration, coupled with models of physical affordances, will allow the system to anticipate, rather than simply react to, external forces.

Ultimately, the true test of UniBiDex – and indeed, all teleoperation systems – will not be its ability to flawlessly execute pre-programmed tasks, but its capacity to learn from failure. The framework provides a powerful platform for data collection, but the real challenge remains: transforming that data into a system that can generalize beyond the confines of the laboratory and operate with true autonomy in the messy, unpredictable reality it is designed to serve.

Original article: https://arxiv.org/pdf/2601.04629.pdf

Contact the author: https://www.linkedin.com/in/avetisyan/

See also:

2026-01-10 13:14