Seeing and Doing: AI Learns to Manipulate Objects with Video and Action

Author: Denis Avetisyan

Researchers have developed a new AI model that can simultaneously generate realistic videos of robotic manipulation and the precise actions needed to perform those tasks.

CoVAR leverages a video diffusion backbone in conjunction with an Action DiT to synthesize actions, employing Bridge Attention for cross-modal interaction and incorporating an Action Refinement Module to address the challenges posed by low-resolution datasets.

CoVAR, a multi-modal diffusion model, leverages bridge attention and action refinement to co-generate video and action sequences for improved robotic manipulation performance.

Generating robotic manipulation policies often requires paired video and action data, yet obtaining accurately annotated datasets remains a significant bottleneck. This limitation motivates ‘CoVAR: Co-generation of Video and Action for Robotic Manipulation via Multi-Modal Diffusion’, a novel approach that simultaneously generates realistic videos and precise robotic actions from text instructions. By extending pretrained video diffusion models with a dedicated action diffusion network, a Bridge Attention mechanism, and an action refinement module, CoVAR achieves superior performance on multiple benchmarks and real-world datasets. Could this framework unlock scalable robotic learning from the vast amounts of readily available, yet unlabeled, video data?

The Illusion of Robotic Mastery

Effective robotic manipulation hinges on a seamless interplay between visual perception and physical action, a process surprisingly difficult to achieve in practice. Robots must not only ‘see’ and interpret their environment, but also translate that understanding into precise movements, all while operating with limited sensory data and finite computational resources. This constraint presents a significant challenge; unlike humans who effortlessly integrate vast experience and intuitive understanding, robots often struggle with ambiguity and uncertainty. Consequently, even simple tasks can become complex problems of estimation and control, requiring sophisticated algorithms to bridge the gap between perception and action and ensure reliable performance in dynamic, real-world settings.

Current approaches to robotic manipulation often prioritize either visually plausible simulations or precise action planning, but rarely both concurrently. This creates a critical performance gap: robots trained in realistic, yet inaccurate, simulations struggle to transfer learned behaviors to the real world, while systems focused on precise action often rely on simplified visual representations that fail to capture the complexities of real-world scenarios. Consequently, robotic systems exhibit diminished reliability when faced with unforeseen circumstances or subtle variations in object appearance or positioning. The inability to seamlessly integrate realistic visual feedback with accurate action execution results in hesitant, inefficient, or even failed manipulations, highlighting the need for methodologies that bridge this critical divide and enable robust, adaptable robotic performance.

Successful consecutive pick-and-place actions in generated videos and real-world experiments demonstrate the policy's high precision and strong alignment between visual input and action output. — Successful consecutive pick-and-place actions in generated videos and real-world experiments demonstrate the policy’s high precision and strong alignment between visual input and action output.

CoVAR: Stitching Reality from Pixels

CoVAR utilizes a diffusion modeling approach to synthesize both video frames and corresponding robotic actions concurrently. This is achieved by training a single model to generate multiple modalities – visual data in the form of video and control signals for a robotic system – from a shared latent space. Unlike sequential pipelines where video is generated before action planning, CoVAR’s architecture enables the model to consider both visual realism and robotic feasibility during the generation process, resulting in a cohesive and physically plausible output. The model learns the joint distribution of video and action, allowing it to produce videos of robotic agents performing actions and, simultaneously, the control signals required to execute those actions in a physical environment.

CoVAR leverages the pre-existing capabilities of Open-Sora, a large-scale video diffusion model, to establish a strong foundation for realistic video generation. Open-Sora, pre-trained on a substantial dataset of video data, provides the initial framework for producing high-fidelity visual content. By building upon this pre-trained model, CoVAR avoids the need to train a video generation component from scratch, significantly reducing computational costs and training time. This approach allows CoVAR to concentrate development efforts on integrating robotic action generation without sacrificing visual quality; the visual outputs benefit directly from the established capabilities of Open-Sora in generating coherent and detailed video frames.

The CoVAR model utilizes a parallel architecture consisting of an Action DiT (Diffusion Transformer) and a UNet-based action decoder to generate robotic actions concurrently with video frames. The Action DiT processes the same noise-conditioned latent space as the video diffusion model, Open-Sora, but is trained to predict robotic actions instead of visual content. This action representation is then passed to the UNet-based action decoder, which reconstructs the corresponding robotic control signals. This parallel processing enables the model to maintain consistency between the generated video and the actions being performed, ensuring that the robotic movements visually align with the depicted scene and events.

Our model generates clearer, more realistic videos of objects and robotic arms with fewer artifacts, demonstrating strong alignment between generated visuals and corresponding actions compared to baseline methods.

Refinement: Polishing the Illusion

The Action Refinement Model operates as a secondary processing stage following initial action generation. This model takes as input both the original image prompt and the text prompt used to initiate the process, alongside the initially generated action sequence. By conditioning on these inputs, the model iteratively refines the action sequence to better align with the provided visual and textual cues. This refinement process focuses on correcting inaccuracies and enhancing the overall coherence of the generated actions, resulting in a more contextually appropriate and visually consistent output. The model utilizes this combined conditioning to produce a final action sequence that is more faithful to the user’s intent as expressed through both modalities.

Bridge Attention establishes a mechanism for cross-modal interaction by enabling the diffusion model to directly relate visual features extracted from the initial image with the evolving action features during the generation process. This is achieved through an attention mechanism that computes relationships between these feature sets, allowing the model to selectively focus on relevant visual information as it refines the generated action sequence. Specifically, visual features serve as the key and value in the attention computation, while the action features act as the query, facilitating the transfer of visual context to guide action refinement. This attention-based integration ensures that the generated actions are consistent with and informed by the input image, improving the overall coherence and plausibility of the output.

Rectified Flow streamlines the diffusion process by introducing a novel training scheme that enforces a monotonic relationship between the noise schedule and the predicted noise. This approach allows for a reduction in the number of diffusion steps required to generate high-quality outputs, directly improving computational efficiency. Evaluations demonstrate that Rectified Flow achieves comparable or superior performance to standard diffusion models with significantly fewer steps – up to a $50\%$ reduction – without introducing a discernible loss in fidelity or sample quality, as measured by metrics such as Fréchet Inception Distance (FID) and Inception Score (IS).

Action refinement significantly improves task completion by enabling precise control, contrasting with the coarse, trend-following actions of the unrefined model.

Demonstrated Performance: A Flicker of Promise

Evaluations using the Libero90 and Calvin datasets reveal that CoVAR consistently generates video with both enhanced visual fidelity and demonstrably accurate action execution. These datasets, known for their complexity and demand for realistic movement, served as a rigorous testing ground for the model’s capabilities. Results indicate CoVAR surpasses existing frameworks in its ability to synthesize coherent and believable video sequences, while simultaneously maintaining a high degree of accuracy in the depicted actions. This dual achievement-high-quality visuals and accurate action representation-positions CoVAR as a significant advancement in the field of video generation, offering a more complete and compelling solution for applications requiring realistic and purposeful motion.

Rigorous quantitative analysis validates the effectiveness of the proposed framework, demonstrating its capacity for high-fidelity video generation and accurate action portrayal. Evaluation leveraged established metrics – Action Success Rate, Fréchet Video Distance (FVD), Learned Perceptual Image Patch Similarity (LPIPS), Structural Similarity Index Measure (SSIM), and Peak Signal-to-Noise Ratio (PSNR) – to provide a comprehensive performance assessment. Notably, the model consistently achieved superior results across several key indicators; specifically, it exhibited higher values in PSNR, SSIM, LPIPS, and FVD when compared to existing baseline models, indicating improved visual quality and perceptual realism in the generated video sequences. These findings offer concrete evidence supporting the model’s ability to produce compelling and visually accurate depictions of dynamic actions.

CoVAR distinguishes itself through exceptional performance even when faced with low-resolution inputs, demonstrating a robustness crucial for real-world deployment where pristine data is rarely available. Rigorous testing on both the Calvin and Libero90 datasets consistently reveals a significantly higher Action Success Rate compared to alternative methodologies. This advantage isn’t limited to simulated environments; CoVAR also achieves the highest Action Success Rate in practical, real-world experiments, validating its capacity to translate theoretical gains into tangible, reliable performance and positioning it as a versatile solution for video generation and action prediction tasks.

Generated videos accurately reflect text instructions, with corresponding actions closely mirroring reference ground truth, demonstrating successful task completion.

Towards More Intelligent Robotic Agents: A Long Road Ahead

The development of CoVAR signifies a notable advancement in the pursuit of truly intelligent robotic agents. This framework moves beyond pre-programmed responses by enabling robots to not simply react to their surroundings, but to actively perceive, interpret, and reason about complex, dynamic environments. By integrating perception and action through a unified variational approach, CoVAR allows robots to anticipate the consequences of their actions and adapt their behavior accordingly. This capability is crucial for navigating unpredictable situations, such as those found in human-populated spaces or unstructured industrial settings. Ultimately, CoVAR establishes a foundation for robots that can operate with greater autonomy, robustness, and flexibility, paving the way for more effective collaboration with humans and tackling increasingly sophisticated tasks.

The current framework, while demonstrating promising capabilities, serves as a foundation for a more ambitious research trajectory. Future investigations will prioritize broadening the scope of tasks the robotic agent can successfully address, moving beyond controlled simulations to encompass the unpredictable nuances of real-world environments. This expansion necessitates developing more robust perception modules capable of handling noisy sensor data and dynamic lighting conditions, alongside advanced planning algorithms that can adapt to unforeseen obstacles and changing goals. A key element of this ongoing work involves seamless integration with physical robotic platforms, bridging the gap between algorithmic innovation and tangible application in fields like automated manufacturing, personalized healthcare, and the development of truly assistive robotic devices that can meaningfully improve quality of life.

The development of CoVAR holds considerable promise for revolutionizing several key industries. In manufacturing, the framework’s ability to enhance robotic perception and reasoning could lead to more adaptable and efficient assembly lines, capable of handling complex and variable tasks with minimal human intervention. Within healthcare, CoVAR-powered robots could assist surgeons with intricate procedures, deliver medications and supplies, and provide personalized care to patients, improving both precision and accessibility. Perhaps most significantly, this research paves the way for advanced assistive robotics, offering enhanced support for individuals with disabilities or limited mobility, enabling greater independence and a higher quality of life through intelligently responsive and adaptable robotic companions.

The pursuit of seamless robotic manipulation, as demonstrated by CoVAR’s co-generation of video and action, feels predictably ambitious. This framework attempts to bridge the gap between visual perception and motor control, employing diffusion models and attention mechanisms. It’s a neat trick, certainly, but one built on layers of complexity. As David Marr observed, “A system must have a representation of what it is doing.” CoVAR’s ‘representation’ is, inevitably, a construct-a probabilistic model trained on datasets. The system will inevitably encounter scenarios outside of its training distribution, revealing the brittleness inherent in even the most sophisticated algorithms. The ‘action refinement module’ is merely a delay of inevitable failure, a sophisticated bandage on a fundamentally unstable system. Anything that appears to ‘heal’ itself hasn’t truly broken yet.

What’s Next?

The pursuit of co-generated video and action pairs, as demonstrated by CoVAR, feels predictably optimistic. It addresses a current bottleneck, certainly, but one suspects the gains will be quickly absorbed by the inherent messiness of real-world deployment. The elegant cross-modal attention and rectified flow will, inevitably, encounter scenarios not anticipated by any training dataset – lighting changes, unexpected object occlusions, the simple fact that things are rarely positioned just so. The current reliance on low-resolution data for action refinement, while pragmatic, hints at a fundamental limitation; scaling this approach will demand addressing the data scarcity at higher resolutions, a problem that hasn’t yielded to easy solutions in the past.

The field will likely move toward increasingly complex architectures, layering more and more abstractions on top of diffusion models. The question isn’t whether these additions will work in a lab environment – they almost certainly will – but whether they’ll introduce new failure modes, and whether the cost of computation will outweigh the marginal gains. One anticipates a resurgence of interest in simpler, more robust approaches, perhaps borrowing from classical control theory, simply because things that must work often do so with less flourish.

Ultimately, the true test of CoVAR, and models like it, will not be achieving state-of-the-art benchmarks, but demonstrating a consistent ability to function reliably in unpredictable environments. If all tests pass, it’s because they test nothing of practical consequence. The next phase will be less about generating plausible videos and more about generating repeatable actions, a subtle but crucial distinction.

Original article: https://arxiv.org/pdf/2512.16023.pdf

Contact the author: https://www.linkedin.com/in/avetisyan/