Bridging the Gap: A Simulator for Wheelchair-Arm Robotics

by

Author: Denis Avetisyan

Researchers have developed a new physics-based simulator and dataset designed to accelerate the development of unified control systems for assistive robots that combine wheelchair navigation and robotic arm manipulation.

WheelArm-Sim integrates human guidance into a physics-based simulation, enabling task execution through real-time teleoperation and simultaneously collecting data encompassing human instructions, visual information via [latex]RGB-D[/latex] images, and comprehensive robot performance metrics.
WheelArm-Sim integrates human guidance into a physics-based simulation, enabling task execution through real-time teleoperation and simultaneously collecting data encompassing human instructions, visual information via [latex]RGB-D[/latex] images, and comprehensive robot performance metrics.

WheelArm-Sim provides a multimodal data generation platform for training vision-language action models and advancing integrated control in assistive robotics applications.

Despite advancements in both wheelchair and robotic arm technologies, integrated control of these systems for assistive robotics remains largely unexplored, hindering truly independent living for individuals with mobility limitations. This paper introduces ‘WheelArm-Sim: A Manipulation and Navigation Combined Multimodal Synthetic Data Generation Simulator for Unified Control in Assistive Robotics’-a physics-based simulation environment and accompanying dataset designed to facilitate the development of unified control algorithms. We demonstrate the capability of WheelArm-Sim to generate a multimodal dataset comprising 13 tasks and over 67,000 samples, showcasing its potential for training data-driven machine learning models. Could this framework unlock more intuitive and effective assistive robotic solutions for enhanced user independence?

The Necessity of Simulated Realities for Robotic Assistance

Successfully maneuvering a robotic arm while mounted on a wheelchair demands an intricate interplay of precise control and environmental awareness. The complexities arise from the non-holonomic constraints of wheelchair motion – its inability to move sideways directly – coupled with the arm’s degrees of freedom. This creates a compounded control problem; the wheelchair’s path must be continuously adjusted not only for navigation but also to position the arm for effective interaction with objects. External disturbances, uneven terrain, and dynamic environments further exacerbate these challenges, requiring the control system to compensate for unpredictable forces and maintain stability. Consequently, achieving fluid and reliable robotic arm control on a wheelchair necessitates advanced algorithms capable of coordinating both the base mobility and the arm’s manipulation in real-time, a feat currently limited by the availability of comprehensive testing grounds and datasets.

The advancement of assistive robotics, specifically robotic arms integrated with wheelchairs, is currently constrained by limitations in data acquisition. Existing physical robotic platforms frequently exhibit restricted ranges of motion, lack the adaptability to navigate diverse and unpredictable environments, and prove costly or impractical for gathering the extensive datasets necessary to train reliable control algorithms. This paucity of versatile hardware hinders the development of robust systems capable of responding effectively to real-world scenarios – algorithms trained on limited data often struggle with unexpected obstacles, varying lighting conditions, or nuanced user intentions. Consequently, progress in creating truly helpful and intuitive robotic assistance is slowed, as researchers find themselves bottlenecked by the difficulty of collecting the data needed to refine and validate their control strategies.

A compelling need exists for highly realistic simulation environments to advance the field of wheelchair-mounted robotic arms. Current limitations in data acquisition stem from the difficulty and cost associated with physical experimentation in complex, real-world scenarios. A flexible simulation, however, allows researchers to generate vast datasets encompassing a wide range of environmental conditions and user interactions, all without the risks or logistical hurdles of physical testing. This accelerated data generation is vital for training and validating robust control algorithms, particularly those leveraging machine learning techniques. Furthermore, a well-designed simulation facilitates rapid prototyping and iterative design improvements, ultimately paving the way for more effective and intuitive assistive robotic systems that can significantly enhance the independence and quality of life for individuals with mobility impairments.

A ROS2-based workflow seamlessly integrates user teleoperation, robotic control, and sensor data processing to facilitate efficient multimodal data collection for the WheelArms.
A ROS2-based workflow seamlessly integrates user teleoperation, robotic control, and sensor data processing to facilitate efficient multimodal data collection for the WheelArms.

WheelArm-Sim: A Foundation for Realistic Robotic Interaction

WheelArm-Sim utilizes the NVIDIA Isaac Sim platform to create a physics-based simulation environment for a robotic arm integrated with a wheelchair. This simulation leverages Isaac Sim’s capabilities in physically realistic rendering, sensor simulation, and dynamics modeling to accurately represent the interaction between the robotic arm, the wheelchair, and the surrounding environment. The platform simulates forces, torques, and constraints acting on the robotic system, enabling the development and testing of control algorithms in a virtual setting prior to deployment on physical hardware. This approach facilitates efficient prototyping and reduces the risks associated with real-world experimentation.

WheelArm-Sim utilizes a digitally replicated Kinova Gen3 robotic arm and a Whill Model CR2 wheelchair to create a cohesive simulation environment. The Kinova Gen3, a seven degree-of-freedom arm, is accurately modeled to reflect its physical properties and range of motion, while the Whill CR2 provides a realistic base for mobility. This integration allows for the creation of complex scenarios involving both manipulation tasks performed by the robotic arm and navigation of the wheelchair through various environments. The fidelity of these replicated models enables the testing of algorithms and control strategies in a physics-based setting that closely mirrors real-world conditions, facilitating research into assistive robotics and autonomous wheelchair operation.

Human teleoperation within WheelArm-Sim is achieved through a direct interface allowing a user to remotely control the Kinova Gen3 arm and Whill CR2 wheelchair in a simulated environment. This capability facilitates the execution and recording of a diverse set of Activities of Daily Living (ADLs), including tasks such as object manipulation, reaching, and navigation through representative indoor scenes. Collected data includes operator commands, robot state information, and sensor readings, providing a comprehensive dataset for the development and evaluation of robotic assistance algorithms and control strategies. The system supports intuitive control schemes, enabling researchers to gather data reflecting realistic human-robot interaction during common tasks.

WheelArm-Sim generates a multimodal dataset consisting of 67,783 individual samples collected during the execution of 13 distinct tasks. This data incorporates sensor readings from both the Kinova Gen3 robotic arm and the Whill Model CR2 wheelchair, including joint angles, velocities, forces, and wheel odometry. The dataset also includes RGB images captured from a simulated camera mounted on the robotic arm, providing visual information for perception and computer vision algorithms. Data is synchronized and time-stamped, enabling the training and validation of control algorithms designed for robotic manipulation and navigation in complex environments, and facilitating research into areas such as imitation learning and reinforcement learning.

The WheelArm-Sim environment features four distinct task areas designed for robotic manipulation challenges.
The WheelArm-Sim environment features four distinct task areas designed for robotic manipulation challenges.

Precision Control: The Mathematical Basis of Robotic Movement

Precise control of robotic arms necessitates the use of Inverse Kinematics (IK). IK is the computational process of determining the joint parameters – angles for revolute joints and lengths for prismatic joints – required to position the end-effector of the robot at a desired pose, which includes both position and orientation in Cartesian space. Unlike Forward Kinematics, which maps joint angles to end-effector pose, IK solves the inverse problem. The accuracy of the IK solution directly impacts the robot’s ability to reach and maintain a specific target pose, and is crucial for tasks requiring high precision, such as assembly, welding, or surgical procedures. Multiple solutions may exist for a given end-effector pose, requiring additional constraints or optimization criteria to select the most appropriate joint configuration.

Screw Theory provides a geometrically-based framework for representing and manipulating rigid body motions, and is central to our Inverse Kinematics implementation. Unlike traditional Euler angle or quaternion representations which can suffer from gimbal lock and singularities, Screw Theory represents motion as a combination of translation and rotation along a single axis – a “screw motion”. This is formalized mathematically as [latex] \hat{X} = \begin{bmatrix} \hat{\omega} \\ \hat{v} \end{bmatrix} [/latex], where [latex] \hat{\omega} [/latex] is the angular velocity vector and [latex] \hat{v} [/latex] is the linear velocity vector. By representing poses and transformations using “exponentials” of these screw axes, we ensure a singularity-free and computationally efficient solution to the Inverse Kinematics problem, allowing for precise calculation of joint configurations required to achieve desired end-effector positions and orientations.

The Newton-Raphson method is employed as an iterative numerical technique to determine the joint angles of a robotic arm required to achieve a desired end-effector pose. This approach leverages the Jacobian matrix, [latex]J[/latex], which relates joint velocities to end-effector velocities. The method refines an initial guess for the joint angles through repeated linear approximations, minimizing the error between the desired and actual end-effector position and orientation. The iterative update rule is defined as [latex]θi+1 = θi – JT(f(θi) – pdesired)[/latex], where θ represents the joint angles, [latex]f(θ)[/latex] is the forward kinematics function, and [latex]pdesired[/latex] is the desired end-effector pose. Convergence is achieved when the error falls below a specified tolerance, providing a computationally efficient solution for the typically non-linear Inverse Kinematics problem.

An LSTM Baseline Model is being evaluated for its ability to predict WheelArm actions and enhance control system performance. This model was trained using multimodal data generated by the WheelArm-Sim environment, consisting of 232 distinct trajectories. Initial baseline testing has yielded quantitative performance metrics, specifically Mean Absolute Error (MAE) and Mean Squared Error (MSE), which are being used to assess the model’s predictive accuracy and inform further refinement of the control strategy. These metrics provide a standardized evaluation of the model’s ability to accurately anticipate WheelArm movements based on the training dataset.

The prediction closely follows ground truth data for all axes, capturing trends in [latex]X[/latex] and [latex]Y[/latex], step-like patterns, and gripper closing with minor timing discrepancies and occasional initial rise omissions.
The prediction closely follows ground truth data for all axes, capturing trends in [latex]X[/latex] and [latex]Y[/latex], step-like patterns, and gripper closing with minor timing discrepancies and occasional initial rise omissions.

Towards Seamless Integration and Enhanced Independence

The Whill Model CR2 wheelchair achieves a new level of robotic integration through Isaac Sim’s Action Graph feature, which allows for the definition and execution of complex, coordinated movements. This isn’t simply remote control; the Action Graph constructs a detailed behavioral sequence, enabling precise manipulation of the wheelchair’s drive system, steering, and other functionalities. By representing robotic actions as nodes within a graph, developers can design sophisticated maneuvers – such as navigating cluttered spaces, avoiding obstacles, and responding to user input – with a high degree of accuracy. The resulting system moves beyond basic mobility, effectively transforming the wheelchair into a fully programmable robotic platform capable of performing intricate tasks and adapting to dynamic environments, paving the way for advanced assistive technologies.

WheelArm-Sim represents a significant advancement in assistive robotics simulation by uniquely integrating robotic arm manipulation with powered wheelchair navigation. This combined approach enables researchers to model and test complex, real-world tasks, such as object retrieval, door opening, and obstacle negotiation, all from the perspective of an individual using a powered wheelchair. By simulating these scenarios, developers can create and refine control algorithms that coordinate both the wheelchair’s movement and the arm’s dexterity, paving the way for more intuitive and effective assistive devices. The platform allows for extensive testing and optimization in a safe, virtual environment before deployment on physical robots, potentially accelerating the development of solutions that enhance independence and improve the quality of life for individuals with mobility limitations.

A significant advantage of this simulated robotic system lies in its potential for real-world application. Data gathered during simulations – encompassing sensor readings, control commands, and performance metrics – can be directly transferred to a physical robotic platform, such as the Whill Model CR2 wheelchair and a compatible robotic arm. This transfer facilitates the development and refinement of control strategies designed to assist individuals with mobility impairments, potentially enabling them to perform tasks previously inaccessible. By iteratively testing and improving algorithms in a safe, virtual environment before deployment, researchers aim to create assistive technologies that demonstrably enhance independence, improve quality of life, and reduce the challenges faced by those with limited mobility.

Efforts are now concentrating on bridging the persistent “Sim-to-Real Gap,” the discrepancy between performance in simulated environments and real-world execution. Researchers are actively refining control algorithms, employing techniques such as domain randomization and reinforcement learning, to enhance the robustness and reliability of the robotic system when deployed in unpredictable, real-world settings. This involves accounting for sensor noise, variations in lighting and surface textures, and the inherent imperfections of physical actuators-factors often absent or simplified in simulation. The ultimate goal is to create a system capable of consistently and safely executing complex assistive tasks, ultimately fostering greater independence and improved quality of life for individuals with mobility challenges, even amidst the complexities of everyday environments.

The dataset comprises a diverse range of objects-including those natively available in Isaac Sim, user-created models, and those sourced online-and incorporates deformable materials to enhance simulation realism.
The dataset comprises a diverse range of objects-including those natively available in Isaac Sim, user-created models, and those sourced online-and incorporates deformable materials to enhance simulation realism.

The pursuit of unified control, as demonstrated by WheelArm-Sim, echoes a fundamental tenet of elegant design. The simulator’s strength lies not in its complexity, but in its ability to synthesize realistic multimodal data – a focused effort to reveal essential interactions between wheelchair navigation and robotic arm manipulation. As Edsger W. Dijkstra stated, “Simplicity is prerequisite for reliability.” WheelArm-Sim embodies this principle; by distilling the core challenges of assistive robotics into a physics-based simulation, it offers a pathway toward robust and dependable systems. The generation of diverse data allows for the training of vision-language action models, stripping away extraneous variables to focus on the essential relationship between perception and action.

Where Do We Go From Here?

The proliferation of simulated environments, while often presented as progress, frequently obscures a fundamental truth: the map is not the territory. WheelArm-Sim, for all its technical merit, merely shifts the burden of complexity. The true test lies not in generating data, but in bridging the chasm between synthetic fidelity and the irreducible messiness of the real world. Current approaches tend to accumulate layers of abstraction, attempting to model the unpredictable. A more fruitful path may reside in embracing, even exploiting, that very unpredictability.

The integration of wheelchair and robotic arm control, while seemingly straightforward in simulation, demands a re-evaluation of established methodologies. Inverse kinematics, vision-language models – these are tools, not solutions. The difficulty isn’t solving the equations; it’s defining the relevant equations. A system capable of genuinely assistive robotics must prioritize robustness over precision, adaptability over pre-programming. It must learn to fail gracefully, and perhaps, even learn from those failures.

Ultimately, the value of WheelArm-Sim, or any such simulator, isn’t in the data it produces, but in the questions it forces one to ask. If a robot cannot navigate a cluttered room, the problem isn’t a lack of data points; it’s a deficiency in fundamental understanding. The goal should not be to create a perfect simulation, but a simpler one – one that reveals the core principles, and exposes the assumptions hidden within the complexity.

Original article: https://arxiv.org/pdf/2601.21129.pdf

Contact the author: https://www.linkedin.com/in/avetisyan/

See also:

2026-01-31 10:58