Robots Learn by Watching: A New Vision for Manipulation

Author: Denis Avetisyan

This research presents a framework enabling robots to learn complex manipulation skills by retrieving and applying knowledge from human demonstration videos.

Data gleaned from robotic systems and accompanying video streams inform a retrieval framework, revealing the inherent capacity to extract nuanced information as systems evolve and degrade.

A novel approach leveraging retrieved video segments and mid-level information significantly improves robotic manipulation performance in both simulated and real-world environments.

Despite advances in robotic manipulation, reliably generalizing to novel tasks remains a significant challenge for autonomous systems. This paper, ‘Let Me Show You: Learning by Retrieving from Egocentric Video for Robotic Manipulation’, introduces a novel framework—Retrieving-from-Video (RfV)—that leverages human demonstration videos and mid-level visual information to substantially improve robotic performance. By retrieving relevant video segments and integrating extracted affordance and motion data, the system learns more effectively and generalizes beyond its training data. Could this approach pave the way for robots capable of learning complex manipulation skills simply by “watching” humans perform them?

The Limits of Imitation: A Quest for Adaptability

Traditional robotic learning often prioritizes imitation, where robots replicate demonstrated behaviors. While effective in controlled environments, this approach hinders adaptation to unforeseen circumstances. Performance degrades with variations in object properties, environmental conditions, or task requirements. Existing methods struggle with generalization—applying learned skills to unseen objects or locations—creating a critical bottleneck for real-world deployment. Robots often fail to recognize object affordances or infer appropriate actions based on context. Without robust reasoning and informed decision-making, these systems remain brittle and confined to curated settings.

Real-world experiments were conducted using a Franka robot and a representative task setup.

The core limitation lies in a lack of reasoning about object affordances and filtering task-relevant information from noise.

Retrieving Knowledge: Learning from the Echoes of Experience

A novel approach, Retrieving-from-Video (RfV), facilitates complex task acquisition by leveraging a database of human actions. RfV moves beyond imitation by extracting intermediate-level representations from video data, providing a more generalized understanding of task requirements. By grounding actions in visual observations, the robot demonstrates improved flexibility and robustness in novel scenarios.

The proposed RfV framework integrates a video bank, a video retriever that responds to language instructions, and a policy generator to process video data and improve robot model training and evaluation.

The RfV framework extracts object affordances and motion trajectories. The system utilizes a video retrieval component responding to natural language, enabling targeted selection of relevant demonstrations. A policy generator then processes retrieved video data to refine the robot’s control policy and enhance task performance.

From Language to Action: The Pipeline of Understanding

The system utilizes a multi-stage pipeline translating language instructions into robotic actions. A video retriever initially identifies relevant demonstrations based on provided language, leveraging CLIP and GPT-4V to connect textual commands with visual content. Following retrieval, the system employs GroundingDino and Segment Anything to localize and segment objects of interest, isolating visual elements pertinent to the instructed task. The final stage involves a policy generator, built on a ViT-Base architecture and the Transformer framework, translating combined visual and semantic information into executable robot actions, replicating demonstrated behavior in response to the instruction.

Demonstrated Resilience: Generalization and Real-World Validation

Recent research demonstrates improved robotic manipulation performance compared to Action Chunking Transformer and Diffusion Policy. Experiments were conducted within the Metaworld simulation environment and with a physical Franka Emika robot equipped with ZED cameras. Quantitative results indicate a substantial increase in success rates across benchmark tasks, achieving a 19.1% improvement on Metaworld medium-level benchmarks compared to Diffusion Policy and a 39.1% improvement over Action Chunking Transformer. Performance on the PlaceBall task reached 40%, and PlaceCan achieved 30%. Notably, a retrieval module significantly enhanced robustness to distractions, achieving an 80% success rate on PlaceBall with distractors.

Spatial generalization was evaluated by randomly positioning a tennis ball and its box, while appearance generalization was tested by varying the color of a cube not seen during training.

The robot achieved a 5/5 success rate on appearance generalization using a cube with a novel color, confirming the effectiveness of learning from human videos to enhance robotic manipulation and adapt to unseen objects and environments.

Towards Adaptable Systems: A Future Shaped by Observation

Recent advances in artificial intelligence, particularly Foundation Models, offer a pathway toward robots capable of independent learning and adaptation. These models, pre-trained on massive datasets, demonstrate the ability to generalize learned skills to novel situations, reducing the need for task-specific programming. A key enabler is the increasing availability of video data, allowing robots to learn from observation, mirroring human skill acquisition. By analyzing vast visual information, robots can infer task objectives, identify relevant features, and develop appropriate action sequences, contrasting with traditional robotic programming relying on explicitly defined rules.

The potential impact extends across sectors, from automated manufacturing to intelligent healthcare and domestic robotics. The development of these adaptable systems represents a step toward a future where machines seamlessly integrate into and enhance daily life. Like all complex creations, these systems will eventually succumb to the weight of time, but their initial flexibility suggests a grace in their decay—a testament to the enduring power of adaptable design.

The pursuit of robust robotic manipulation, as demonstrated by this research into Retrieving-from-Video (RfV), echoes a fundamental principle of enduring systems. The framework’s reliance on leveraging existing human demonstrations – essentially, building upon prior knowledge – aligns with the idea that all systems, even those embodied in code and mechanics, are subject to the forces of time and entropy. Donald Knuth observes, “Premature optimization is the root of all evil.” This holds true here; RfV prioritizes effective retrieval and adaptation from existing data, rather than striving for immediate, complex solutions. The system doesn’t attempt to reinvent manipulation from scratch, but gracefully ages by building upon established examples, demonstrating a pathway toward more resilient and adaptable robotic systems.

What’s Next?

The pursuit of robotic manipulation through imitation, as demonstrated by this work, inevitably encounters the limits of direct replication. Systems learn to age gracefully not by endlessly accumulating data, but by refining the way they interpret the existing medium. The Retrieving-from-Video framework offers a compelling method for distilling human demonstrations, but the very notion of ‘mid-level information’ hints at a persistent challenge: the gap between human intent and robotic execution remains. Further exploration should not focus solely on expanding the video datasets, but on the development of more nuanced representations – the subtle cues and contextual understandings that humans intuitively grasp.

A crucial, and often overlooked, aspect lies in the inherent entropy of real-world environments. The system functions by retrieving similar states, but true adaptability demands a capacity to anticipate divergence. The framework’s performance in simulation, while encouraging, must be viewed as a temporary reprieve. Real-world decay will introduce variations the system hasn’t ‘seen’ – and these are not simply more data points to be cataloged, but opportunities for the system to define its own operational boundaries.

Perhaps the most fruitful path forward lies in accepting that perfect imitation is an asymptotic goal. Sometimes observing the process—understanding how a system fails to generalize—is better than trying to speed it up. The challenge isn’t to create a robot that mimics human dexterity, but to build one that understands the inherent limitations of its own existence – and learns to operate within them.

Original article: https://arxiv.org/pdf/2511.05199.pdf

Contact the author: https://www.linkedin.com/in/avetisyan/