Seeing and Grasping: Humanoid Robots Learn to Manipulate the World

by

Author: Denis Avetisyan

Researchers have developed a new system enabling humanoid robots to reliably pick up and move novel objects in unfamiliar environments using only visual input.

The HERO system combines large vision models with accurate end-effector tracking to achieve a 90% success rate in open-vocabulary visual loco-manipulation tasks.

Achieving robust and generalized manipulation of everyday objects remains a key challenge for humanoid robots operating in unstructured environments. This paper, ‘Learning Humanoid End-Effector Control for Open-Vocabulary Visual Loco-Manipulation’, introduces HERO, a modular system that bridges the gap between visual perception and dexterous control by integrating large vision models with an accurate, residual-aware end-effector tracking policy. This approach enables a humanoid robot to reliably grasp novel objects in previously unseen environments, achieving a significant reduction in tracking error and a 90% success rate in real-world tests. Could this combination of learned perception and precise control unlock more adaptable and intuitive human-robot interaction in complex, real-world scenarios?

The Imperative of Robust Robotic Manipulation

Conventional robotic systems, meticulously programmed for controlled laboratory settings, often falter when confronted with the inherent unpredictability of real-world scenarios. The variability in lighting, surface textures, and the sheer diversity of object shapes and sizes presents a significant challenge to their precision. A robot designed to grasp a specific type of mug, for example, may struggle with a slightly different handle or an unexpected obstruction nearby. This lack of adaptability stems from a reliance on pre-programmed models and a limited capacity to generalize learned behaviors to novel objects or situations, resulting in unreliable performance and hindering their effective deployment beyond highly structured environments. The core issue isn’t necessarily a lack of mechanical dexterity, but rather a deficit in the robot’s ability to perceive, interpret, and react to the constant flux of the physical world.

Achieving dexterous robot manipulation hinges on a robot’s ability to accurately perceive its surroundings and formulate effective plans, yet current methodologies frequently encounter limitations when confronted with the complexities of real-world scenarios. Traditional computer vision and motion planning algorithms often struggle with dynamic environments – those containing moving objects or unpredictable changes – and unstructured scenes characterized by clutter, occlusions, or variations in lighting. These systems typically rely on simplified models and controlled conditions, leading to inaccuracies in object recognition, pose estimation, and collision avoidance. Consequently, robots may fail to grasp objects securely, execute precise movements, or adapt to unexpected disturbances, hindering their performance in tasks requiring adaptability and robustness. The challenge lies not simply in processing sensory data, but in interpreting it meaningfully within a constantly changing and often ambiguous environment, demanding more sophisticated algorithms and robust perception-action loops.

For robots to truly integrate into daily life, a capacity for resilient performance is paramount; unlike factory automation operating within carefully controlled parameters, real-world environments present constant unpredictability. A robot tasked with assisting in a home, for example, might encounter objects in unexpected locations, varying lighting conditions, or even sudden human intervention. Consequently, simple pre-programmed responses prove inadequate, necessitating systems capable of real-time adaptation and error recovery. Robustness isn’t merely about preventing failures, but gracefully handling them when they occur, adjusting strategies on the fly, and continuing operation even amidst unforeseen circumstances. This adaptive capability, allowing robots to reliably perform tasks despite environmental uncertainties, represents a fundamental hurdle in transitioning from specialized applications to widespread, dependable deployment.

HERO: A System Founded on Modular Precision

The HERO system establishes a unified architecture for open-vocabulary visual loco-manipulation by tightly integrating three core functional components: perception, planning, and control. This integration allows the system to process visual input, generate feasible motion plans, and execute those plans via robotic actuators in a coordinated manner. Specifically, the system receives RGB-D imagery as perceptual input, uses this data to identify and localize objects, and subsequently plans both robot locomotion and manipulation actions to achieve task goals. This cohesive framework enables HERO to address tasks involving novel objects and previously unseen environments without requiring pre-programmed knowledge of specific object models or scene layouts.

The HERO system’s modularity is achieved through the independent development and integration of perception, planning, and control components. This allows for adaptation to new tasks or environments by substituting or retraining individual modules without requiring a complete system overhaul. For example, a new object recognition capability can be integrated into the perception module without impacting the established planning or control algorithms. Similarly, changes to the robot’s kinematic model within the control module do not necessitate adjustments to the visual input processing. This component-level flexibility maximizes efficiency and enables rapid prototyping and deployment in diverse scenarios, as demonstrated by the system’s ability to generalize to novel object grasping tasks.

The HERO system employs an RGB-D camera to acquire visual data necessary for object detection and pose estimation. Physical interaction with the environment is achieved through integration with a Unitree G1 quadrupedal robot and a Dex-3 robotic hand. Empirical evaluation demonstrates a 90% success rate in grasping previously unseen objects within unfamiliar environments, indicating robust performance in open-vocabulary visual loco-manipulation tasks. This success rate is calculated based on a dataset of novel objects and environments not used during training or system development.

End-Effector Tracking: A Policy of Direct Control

The End-Effector Tracking Policy utilizes a neural network architecture to directly map desired end-effector positions to corresponding robot joint angles. This approach bypasses traditional inverse kinematics calculations, enabling faster and more robust control. The network is trained on a dataset of robot configurations and corresponding end-effector poses, learning a non-linear relationship between the two. By predicting the optimal joint angles, the policy drives the robot to achieve and maintain the desired end-effector position with increased precision and responsiveness compared to conventional methods.

The end-effector policy utilizes Residual Neural Odometry (RNO) and Residual Neural Forward Kinematics (RNFK) to significantly improve positional accuracy. RNO estimates the change in pose of the end-effector, while RNFK predicts the end-effector’s position based on joint angles. By employing these neural network-based approaches, the system addresses inherent errors in traditional forward kinematics calculations. Benchmarking demonstrates a reduction in forward kinematics error from an initial 1.76cm to a final error of 0.27cm, representing a substantial improvement in precision and enabling more reliable manipulation and tracking capabilities.

The system achieves stable end-effector tracking despite external disturbances through the implementation of integrated Motion Planning and Replanning. This functionality allows the robot to dynamically adjust its trajectory in response to unforeseen events, maintaining positional accuracy. Quantitative results demonstrate an end-effector tracking error of 2.5cm when utilizing this adaptive planning approach, indicating a significant improvement in robustness and precision during operation. The replanning component continuously evaluates and modifies the planned trajectory, enabling the system to recover from disruptions and adhere to the desired path.

Demonstrated Robustness in Dynamic Environments

Recent experimentation showcases the HERO system’s substantial gains in precision and resilience during object tracking, largely attributed to its End-Effector Tracking Policy. Quantitative results reveal a translation error of just 2.48cm, a marked improvement when contrasted with the 8.29cm error observed in the AMO system and the considerably higher 13.57cm error produced by FALCON. This diminished error rate suggests that HERO not only maintains a more accurate trajectory but also demonstrates a greater capacity to counteract disturbances and maintain stable control throughout the manipulation process, representing a significant advancement in robotic dexterity and reliability.

The HERO system demonstrates a remarkable capacity for reliable object manipulation even when faced with real-world unpredictability. Through rigorous testing in dynamic environments, the system consistently achieves a 90% success rate in manipulating previously unseen objects, showcasing its adaptability to disturbances and uncertainties. This robustness isn’t simply about avoiding failure; it indicates the system’s ability to actively compensate for external forces, shifting object positions, and imperfect environmental knowledge. The high success rate suggests a level of perceptive ability and responsive control that enables the system to maintain a secure grasp and execute intended manipulation tasks despite challenging conditions, representing a significant step toward practical robotic applications.

The HERO system’s enhanced manipulation capabilities stem from a synergistic combination of advanced software and hardware design. Specifically, the implementation of a ‘Goal Adjustment’ strategy allows the robot to dynamically refine its approach to objects, compensating for inaccuracies in initial pose estimation and unforeseen environmental changes. This is further bolstered by the ‘anyGrasp’ model, which enables the system to reliably identify and execute a wide variety of stable grasps – crucial for handling novel objects. Complementing these computational advancements is the incorporation of waist bending, a physical design feature that dramatically expands the robot’s operational reach. Through this innovation, the reachable workspace volume increases by a factor of 2.1, allowing HERO to access and manipulate objects previously beyond its range and significantly improving its versatility in complex, dynamic environments.

The pursuit of robust robotic manipulation, as demonstrated by HERO, echoes a fundamental principle of mathematical rigor. The system’s 90% success rate in grasping novel objects isn’t merely a practical achievement; it hints at an underlying algorithmic correctness. As Blaise Pascal observed, “All of humanity’s problems stem from man’s inability to sit quietly in a room alone.” Though seemingly unrelated, this speaks to the need for focused, precise definition – in this case, of the robot’s kinematic model and the vision-based object recognition. HERO’s modularity and accuracy reveal a commitment to provable solutions, not just empirical ones. It prioritizes consistent, reliable execution-a hallmark of true algorithmic beauty.

What Lies Ahead?

The reported 90% success rate, while superficially impressive, merely postpones the inevitable confrontation with true robotic intelligence. The system, HERO, effectively interpolates between known kinematics and visual data. But interpolation is not understanding. The core challenge remains: achieving robust control not through brute-force learning on vast datasets, but through a formal, provable understanding of physics and geometry. Current reliance on ‘open-vocabulary’ models functions as a sophisticated lookup table, masking a fundamental lack of generalization. What happens when the novel object is not simply ‘unseen,’ but fundamentally different in material property or dynamic behavior?

Future work must prioritize the development of algorithms that reason about affordances – not merely detecting a graspable region, but understanding the forces required to maintain it. The elegance of forward kinematics, though currently leveraged as a component, hints at a more profound path. A truly intelligent system will not need to ‘learn’ to grasp; it will deduce the optimal grasp from first principles. This necessitates a shift away from data-hungry approaches and towards a symbolic, knowledge-based representation of the physical world.

In the chaos of data, only mathematical discipline endures. The current paradigm, while achieving incremental progress, risks building increasingly complex systems that are ultimately fragile and unpredictable. The pursuit of artificial intelligence must not become a synonym for elaborate pattern recognition; it demands a return to the foundational principles of logic and reasoning.

Original article: https://arxiv.org/pdf/2602.16705.pdf

Contact the author: https://www.linkedin.com/in/avetisyan/

See also:

2026-02-19 11:14