Seeing is Reasoning: An Agent That Learns to Look

Author: Denis Avetisyan

Researchers have developed a new system, Weaver, that learns to actively gather visual evidence from videos to improve its reasoning abilities.

Weaver is an end-to-end trainable multimodal agentic system that utilizes reinforcement learning and a visual tool library for improved video understanding and interleaved visual-text reasoning.

Effective video reasoning demands robust perceptual and interpretive skills, yet current approaches relying on text-centric methods often struggle with representational mismatch and limited visual acuity. To address this, we introduce ‘Weaver: End-to-End Agentic System Training for Video Interleaved Reasoning’, a novel, end-to-end trainable multimodal agentic system that dynamically leverages visual tools and reinforcement learning to construct authentic visual-text reasoning trajectories. Our experiments demonstrate that Weaver enhances performance on complex video reasoning benchmarks, particularly those requiring analysis of extended temporal sequences. Could this approach unlock a new paradigm for building more perceptive and adaptable video understanding systems?

Unveiling the Limits of Text-Centric Vision

Despite the burgeoning potential of Multimodal Large Language Models (MLLMs), current methodologies frequently falter when confronted with the nuances of video comprehension. These models, often built upon foundations of textual data and processing, demonstrate limitations in interpreting the complex interplay of visual information and temporal dynamics inherent in video. The reliance on converting visual cues into textual descriptions creates a bottleneck, losing critical details and relationships that are readily apparent through direct visual processing. Consequently, tasks demanding sophisticated perceptual reasoning – such as understanding object interactions, anticipating future events, or discerning subtle changes – prove challenging for text-centric MLLMs, hindering their ability to perform reliably in real-world applications that require robust video understanding.

Multimodal Large Language Models, despite advancements, frequently demonstrate limitations when confronted with tasks demanding nuanced perceptual understanding and accurate temporal placement of events. These models often struggle to move beyond superficial feature recognition within videos, failing to grasp the subtle relationships between objects and actions unfolding over time. This deficiency hinders their effectiveness in real-world applications-such as autonomous navigation, complex activity recognition, or detailed video summarization-where precise understanding of when and how events occur is crucial. The inability to achieve robust temporal grounding means these models can misinterpret sequences, leading to inaccurate conclusions and unreliable performance in dynamic, visually-rich environments. Consequently, while capable of processing visual inputs, current MLLMs often fall short of replicating the deep, contextual reasoning inherent in human visual perception.

Attempts to bolster video understanding by simply increasing the scale of text-centric large language models encounter fundamental limitations. While these models excel at processing textual data, they struggle to bridge the gap between language and the complexities of visual information, particularly regarding spatial relationships, object interactions, and nuanced temporal dynamics. The inherent challenge lies in translating continuous visual streams into discrete textual representations, inevitably resulting in information loss and a diminished capacity for accurate perceptual reasoning. Consequently, even massively scaled text-centric approaches prove inadequate for tasks demanding precise visual grounding, such as anticipating actions, inferring causality, or navigating real-world environments-highlighting the necessity of incorporating more robust and natively visual processing capabilities within these models.

Weaver: Augmenting Reasoning with Perception

Weaver addresses video reasoning through a framework integrating Multimodal Large Language Models (MLLMs) with dedicated perception tools. This approach leverages the MLLM’s capacity for language understanding and knowledge integration while overcoming limitations in directly processing visual information from video. Instead of relying solely on the MLLM’s inherent visual encoders, Weaver dynamically accesses and utilizes a suite of specialized perception modules to extract relevant visual features and cues. This allows the system to perform tasks such as identifying objects, tracking their movements, and analyzing scene dynamics, providing the MLLM with enriched and focused visual input to support complex reasoning processes and question answering.

Perception-in-the-Loop Reasoning within Weaver operates by actively soliciting visual information during the question-answering process. Instead of processing an entire video passively, Weaver dynamically identifies moments requiring perceptual analysis to support inference. This is achieved by integrating a suite of perception tools that are invoked as needed, allowing the system to focus on visually relevant segments and extract specific cues – such as object locations, movements, or interactions – that directly address the query. The system then incorporates these dynamically acquired visual features into its reasoning process, improving accuracy and enabling responses to questions demanding detailed visual understanding beyond what a static frame analysis could provide.

Weaver’s Tool Library consists of specialized perception modules that provide crucial visual information for reasoning tasks. These tools include Frame Selection, which identifies the most relevant frames within a video for analysis; Spatial Tracking, enabling the identification and monitoring of objects and regions of interest across frames; and Optical Flow analysis, which determines the motion patterns of pixels to understand object movement and actions. By dynamically accessing and utilizing these perceptual capabilities, Weaver establishes a robust foundation for answering complex questions about video content, going beyond the limitations of solely relying on visual features extracted from individual frames or pre-computed video representations.

Training Weaver: A Two-Stage Refinement

The initial training phase of Weaver utilizes Supervised Finetuning (SFT) performed on the Weaver-SFT-10K dataset, a collection of 10,000 examples designed to establish foundational capabilities. This dataset specifically focuses on demonstrating the correct invocation of tools and the execution of interleaved reasoning – the process of applying multiple tools in sequence to arrive at a solution. Through SFT, the Qwen2.5-VL model learns to map input video data to appropriate tool selections and to integrate the outputs of those tools into a coherent reasoning pathway, effectively providing a base level of competency prior to reinforcement learning.

Following Supervised Finetuning, Weaver undergoes Reinforcement Learning (RL) training using the Weaver-RL-12K dataset. This stage focuses on optimizing the model’s ability to strategically combine available tools to achieve desired outcomes. The Weaver-RL-12K dataset provides a framework for the system to explore diverse tool-use sequences and learn from the resulting rewards. Through this process, the model refines its decision-making process, improving its capacity to identify effective combinations of tools and maximize cumulative reward signals, ultimately enhancing its performance on complex tasks requiring multi-step reasoning.

Training with the two-stage approach-Supervised Finetuning followed by Reinforcement Learning-yields quantifiable improvements in the Qwen2.5-VL model’s performance on video understanding tasks. Evaluation metrics demonstrate enhanced capabilities in areas requiring complex reasoning and tool integration, indicating the model effectively learns to leverage tools for improved task completion. Specifically, the model exhibits increased accuracy in identifying objects, understanding activities, and answering questions based on video content, as validated through benchmark datasets and comparative analysis against models trained with alternative methods.

Validating Weaver: Performance Across Diverse Benchmarks

Weaver was evaluated on a range of established video understanding benchmarks to assess its performance across diverse tasks. These benchmarks include LVBench, a large-scale video benchmark focused on long-form understanding; VideoMME, which tests multi-modal machine understanding; VideoMMU, evaluating multi-modal understanding with a focus on temporal reasoning; VSIBench, designed for evaluating video summarization and instance recognition; and LongVideo-Reason, specifically designed for assessing reasoning capabilities in long-form videos. Performance across these benchmarks demonstrates Weaver’s broad applicability and robust performance in various video understanding scenarios.

Evaluations utilizing the MVBench benchmark suite demonstrate Weaver’s advancements in video perception. MVBench focuses on assessing the ability of models to accurately process and interpret visual information within video sequences. Weaver’s performance on MVBench indicates a strengthened capacity for feature extraction and visual understanding, directly validating the effectiveness of the perception-augmented reasoning framework implemented in its architecture. This framework integrates enhanced perceptual modules with the reasoning engine, allowing for more accurate and robust video analysis compared to systems relying solely on temporal modeling.

Weaver establishes new state-of-the-art results across multiple video understanding benchmarks. Performance gains of up to 12% are observed on the MLVU benchmark, and a 9.5% improvement is achieved on the LVBench benchmark, both when compared to existing methodologies. Detailed results indicate Weaver achieves 6.7% higher accuracy on LVReason, 4.7% on LVBench, 6.7% on VideoMMU, and a further 9.5% improvement on overall LVBench accuracy when contrasted against the performance of Video-RFT.

Towards a Future of Intelligent Visual Comprehension

The core innovations behind Weaver, a system designed for complex video understanding, aren’t limited to visual storytelling; its principles readily translate to domains requiring reasoning across multiple data streams. Consider robotics, where a robot must synthesize visual input with tactile sensor data and internal state to navigate an environment – Weaver’s approach to grounding language in perception offers a powerful framework for this. Similarly, in autonomous navigation, integrating camera feeds, LiDAR data, and GPS information demands a robust multimodal reasoning engine, mirroring the architecture proven effective in Weaver. This adaptability suggests a future where a unified approach to perception-in-the-loop reasoning can drive progress across a spectrum of intelligent systems, allowing machines to not merely see the world, but to comprehend and interact with it in a meaningful way.

Ongoing development prioritizes streamlining the perception-in-the-loop reasoning framework to address computational bottlenecks and enhance its applicability to real-time scenarios. Current research explores techniques such as model pruning, knowledge distillation, and parallel processing to reduce the framework’s resource demands without sacrificing accuracy. Furthermore, investigations into more efficient data structures and algorithms aim to improve scalability, enabling the system to process larger and more complex video streams. These advancements are crucial for deploying intelligent video understanding systems in resource-constrained environments and facilitating their integration into applications requiring rapid and robust analysis of visual information.

The development of truly intelligent agents hinges on their ability to not merely see the visual world, but to comprehensively understand it – discerning context, anticipating events, and responding appropriately. This future envisions systems capable of fluid interaction, moving beyond pre-programmed responses to engage in dynamic, adaptive behavior. Such agents promise a paradigm shift in automation, extending beyond repetitive tasks to encompass complex, nuanced activities currently requiring human cognition. Ultimately, seamless visual understanding will unlock new frontiers in human-computer collaboration, fostering intuitive interfaces and enabling jointly-achieved goals in fields ranging from healthcare and education to manufacturing and scientific discovery.

The development of Weaver illuminates a crucial point about intelligence – it isn’t solely about possessing information, but about skillfully seeking it. This echoes Geoffrey Hinton’s observation that, “Learning is finding the patterns.” Weaver, functioning as a sophisticated analytical instrument, actively employs visual tools to gather evidence, much like a researcher using a microscope to examine a specimen. The system’s capacity for dynamic evidence acquisition and interleaved reasoning trajectories demonstrates how identifying patterns within visual data-through reinforcement learning and tool augmentation-is fundamental to achieving robust video understanding. It’s a testament to the power of actively probing the environment to construct knowledge, rather than passively receiving it.

What’s Next?

The advent of systems like Weaver necessitates a re-evaluation of benchmark construction itself. Current video reasoning tasks, while useful for initial probing, often prioritize superficial correlations over genuine understanding. The system’s reliance on a curated tool library, though effective, reveals a fundamental dependency: intelligence isn’t solely intrinsic; it’s frequently delegated to external resources. Each instance where the agent fails to effectively utilize a tool, or requests one absent from the library, is not a deficiency, but a precise indicator of gaps in the system’s representational capacity-a roadmap for future development.

A crucial avenue for exploration lies in understanding the interplay between agentic exploration and the inherent ambiguity of visual data. The system’s reasoning trajectories, while demonstrably effective, remain largely opaque. Future work should prioritize the development of interpretability techniques that can elucidate why a particular trajectory was chosen, and what specific visual cues drove the decision-making process. Every deviation from expected behavior-every ‘incorrect’ tool selection-represents an opportunity to uncover hidden dependencies within the multimodal input.

Ultimately, the true test will not be achieving higher scores on existing benchmarks, but rather the ability to generalize to entirely novel situations. Systems trained through reinforcement learning are, by their nature, optimized for the reward function. The challenge, then, is to design reward structures that encourage not just accurate answers, but also robust, adaptable, and genuinely curious reasoning agents.

Original article: https://arxiv.org/pdf/2602.05829.pdf

Contact the author: https://www.linkedin.com/in/avetisyan/