Robots That Reason: A New Approach to Following Instructions

Author: Denis Avetisyan

Researchers have developed a planning method that empowers robots to better understand dynamic environments and execute complex tasks by combining visual and linguistic information.

An agent navigates and interacts with an environment through a cyclical process of planning and perception, beginning with an initial understanding of the surroundings and refining this knowledge through action and visual feedback, ultimately updating a scene memory graph to guide subsequent steps until a designated task is completed.

LookPlanGraph leverages graph representations and visual language models for improved embodied instruction following and dynamic scene understanding.

Existing embodied AI methods relying on static scene graphs struggle with the dynamic nature of real-world environments, limiting their ability to execute complex instructions reliably. This paper introduces LookPlanGraph: Embodied Instruction Following Method with VLM Graph Augmentation, a novel approach that dynamically updates scene graphs during task execution using vision-language models. By continuously verifying object priors and discovering new entities, LookPlanGraph enables more robust and adaptable planning in changing environments, outperforming methods reliant on pre-built static representations. Could this dynamic graph augmentation be a key step towards truly intelligent and versatile robotic agents?

Beyond Static Environments: The Necessity of Dynamic Intelligence

Conventional artificial intelligence often operates under the constraints of ‘Static Planning’, a methodology that presumes a predictable and unchanging environment. This approach, while effective in highly controlled settings like game-playing with defined rules, proves severely limiting when applied to real-world scenarios. The fundamental issue lies in the discrepancy between this assumption of stability and the inherent dynamism of physical spaces – environments are rarely static. Unexpected obstacles, moving objects, and unpredictable human behavior all contribute to a level of complexity that static plans cannot adequately address. Consequently, AI systems reliant on static planning frequently falter when confronted with even minor deviations from their pre-programmed expectations, hindering their effectiveness in practical applications such as robotics, autonomous navigation, and human-robot interaction.

The real world rarely conforms to pre-programmed expectations; physical spaces are inherently dynamic, filled with unexpected obstacles, moving objects, and constantly shifting conditions. This unpredictability poses a significant hurdle for Embodied AI – artificial intelligence designed to operate within the physical world. Traditional AI planning methods, reliant on static environments, struggle when confronted with even minor deviations from their initial assumptions. Consequently, a shift towards ‘Dynamic Planning’ is crucial; systems must be capable of perceiving changes in real-time, replanning actions accordingly, and adapting to unforeseen circumstances. This demands more than simply reacting to stimuli; it requires proactive anticipation, robust error recovery, and the ability to learn from experience, ultimately allowing AI agents to navigate and interact with complex, ever-changing environments effectively.

The limitations of rigidly pre-programmed behaviors become strikingly apparent when artificial intelligence encounters the inherent messiness of the real world. Rather than relying on exhaustive, pre-defined sequences of actions, truly effective embodied AI requires systems capable of reactive and adaptive responses. This means shifting the focus from meticulously planned trajectories to algorithms that can perceive changes in the environment – an unexpected obstacle, a shifting surface, or a moving target – and adjust behavior in real-time. Such systems prioritize sensing, evaluating, and responding, effectively learning to navigate and interact with complexity through continuous feedback and modification of action plans. This transition isn’t simply about increased computational power; it demands a fundamental change in architectural design, favoring flexible, opportunistic strategies over inflexible, pre-determined ones.

Unlike static planners limited by predefined scene graphs, dynamic planners achieve successful task execution by actively exploring and updating their environmental understanding in real time.

LookPlanGraph: A Graph-Based Architecture for Dynamic Reasoning

LookPlanGraph presents a novel approach to embodied agent control by combining graph-based planning with continuous updates from real-time scene perception. This integration allows agents to dynamically adjust plans in response to changes in the environment, offering increased robustness compared to static planning methods. The system utilizes a graph representation to model both long-term plans and immediate perceptual data, facilitating efficient reasoning and action selection. By continuously incorporating new scene information into the planning process, LookPlanGraph enables agents to operate effectively in dynamic and unpredictable environments, addressing limitations of traditional approaches that struggle with real-world variability.

The Memory Graph serves as the central knowledge repository within the LookPlanGraph framework, maintaining a persistent and updatable representation of the agent’s surroundings. This graph encodes the static structure of the environment – including room layouts and navigable spaces – alongside information regarding interacted objects and their associated properties. Crucially, the Memory Graph also stores probable locations for objects, even if those objects are not currently visible, enabling proactive planning and informed decision-making. Nodes within the graph represent both static environmental features and dynamic assets, while edges define spatial relationships and affordances, facilitating efficient pathfinding and manipulation planning. This persistent representation allows the agent to reason about the environment beyond immediate sensory input, improving robustness and enabling complex, long-horizon behaviors.

Scene graphs function as a structured representation of a perceived environment, detailing objects and their relationships. This allows the LookPlanGraph framework to translate natural language instructions into actionable plans by mapping linguistic terms to specific nodes and edges within the scene graph. For example, an instruction like “pick up the red block” is parsed, identifying ‘red block’ as a particular object instance represented in the graph. The agent then uses this grounded understanding of the instruction, contextualized by the scene graph’s spatial and relational data, to formulate and execute a corresponding action plan. This process ensures the agent’s actions are relevant and appropriate to the current physical setting, enabling robust and contextualized interaction.

In a real-world scenario, the system successfully completed the task of assisting 'Andrew' with packing his backpack by dynamically identifying and incorporating newly detected objects into its plan, effectively prioritizing relevant items like a cup, notebook, and mouse while ignoring irrelevant ones. — In a real-world scenario, the system successfully completed the task of assisting ‘Andrew’ with packing his backpack by dynamically identifying and incorporating newly detected objects into its plan, effectively prioritizing relevant items like a cup, notebook, and mouse while ignoring irrelevant ones.

Real-Time Perception and Scene Augmentation: Maintaining Environmental Coherence

The Graph Augmentation Module utilizes a Vision-Language Model (VLM) to continuously update the agent’s internal scene graph representation based on real-time perceptual input. This dynamic updating process allows the agent to reconcile its existing knowledge of the environment with new observations, effectively tracking changes such as object movements, additions, or removals. The VLM processes visual data and linguistic information to identify and incorporate these alterations into the scene graph, ensuring the agent maintains an accurate and consistent understanding of the surrounding world. This real-time synchronization between perception and the internal representation is fundamental for robust and adaptive behavior.

Accurate planning and execution are directly dependent on maintaining a current understanding of the environment. Dynamic scenes, characterized by object manipulation – including movement, addition, and removal – necessitate continuous updates to the agent’s internal world model. Failure to account for these changes can result in plans based on inaccurate premises, leading to unsuccessful actions or even collisions. The ability to rapidly incorporate real-time observations into the planning process is therefore essential for robust and reliable operation in non-static environments, demanding computational efficiency and a low latency response to external stimuli.

The Scene Graph Simulator functions as a critical intermediary between the language model’s action proposals and their execution. It receives proposed actions, then projects them onto the current scene graph to predict resultant states. This simulation allows for pre-emptive identification of potential collisions, unreachable targets, or physically impossible maneuvers. Actions failing validation within the simulator are flagged, triggering the language model to refine its plan before any physical execution occurs, thereby prioritizing both feasibility and safety in dynamic environments. The simulator employs physics-based calculations to ensure accurate prediction of action outcomes within the virtual scene.

The LookPlanGraph prompt leverages a static core, a dynamic component built from the memory graph and prior actions, and a VLM-specific instruction to facilitate graph augmentation.

Validation on Complex Tasks: Demonstrating Robust Intelligence

The capabilities of the ‘LookPlanGraph’ system are rigorously tested using the ‘GraSIF Dataset’, a demanding evaluation tool designed to assess an agent’s ability to follow instructions within complex, simulated household settings. This dataset presents significant challenges due to its emphasis on graph-based reasoning – requiring the system to not only understand natural language commands but also to represent and manipulate relationships between objects in a 3D environment. ‘GraSIF’ distinguishes itself through dynamic scenes, where objects move and change positions, forcing agents to adapt their plans in real-time. Successful navigation of this benchmark demonstrates a substantial leap in embodied artificial intelligence, indicating a system’s capacity for robust, context-aware decision-making in realistic, everyday scenarios.

Recent advancements in embodied artificial intelligence are evidenced by the ‘LookPlanGraph’ system, which demonstrably outperforms existing methods in complex, dynamic environments. This success isn’t merely incremental; the system achieves a higher rate of task completion by integrating visual perception with planning and execution. Unlike predecessors often hampered by unpredictable changes within a scene – such as moving objects or altered layouts – ‘LookPlanGraph’ exhibits improved robustness. This is achieved through a sophisticated approach to environmental understanding, allowing the AI to adapt its plans in real-time and maintain a higher probability of successfully completing assigned tasks, representing a significant step toward more adaptable and reliable artificial agents.

The ‘SayPlan’ mechanism represents a significant refinement in embodied AI planning, leveraging pre-constructed 3D scene graphs to generate instructive feedback that directly enhances plan executability. Rather than relying solely on visual input or abstract reasoning, ‘SayPlan’ translates the environment’s geometric structure into language-based guidance, effectively communicating the necessary steps for task completion. This approach not only clarifies the intended plan but also addresses potential ambiguities arising from complex or dynamic environments, leading to a demonstrably higher success rate in executing instructions. By grounding feedback in a structured understanding of the scene, ‘SayPlan’ allows the agent to proactively refine its actions and overcome obstacles, resulting in more robust and reliable performance across a range of household tasks.

The Future of Embodied Intelligence: Towards Autonomous Systems

Recent advancements in embodied intelligence demonstrate the potential of utilizing large language models (LLMs) not merely as response generators, but as sophisticated planning systems. This ‘LLM-as-Planner’ approach moves beyond traditional graph-based frameworks – which excel at mapping known environments and actions – by enabling agents to formulate novel, multi-step plans in response to complex goals. The LLM effectively reasons about potential outcomes, anticipates challenges, and sequences actions in a way that significantly expands the scope of solvable problems. This integration allows robotic systems to navigate unforeseen circumstances, adapt to dynamic environments, and perform tasks requiring a level of abstract thought previously unattainable, ultimately pushing the boundaries of autonomous behavior and opening doors to more versatile and intelligent machines.

The convergence of large language model planning with graph-based reasoning systems is poised to dramatically expand the operational capacity of intelligent agents. These integrated architectures move beyond pre-programmed responses, enabling robots and virtual entities to dynamically assess complex situations and formulate novel solutions. This adaptability is crucial for navigating unpredictable real-world environments – from assisting in disaster relief where conditions are constantly changing, to providing personalized support in dynamic home settings, and even enabling more robust autonomous navigation in crowded urban spaces. By combining the strengths of both approaches – LLMs for high-level reasoning and planning, and graph networks for efficient environmental representation and action execution – these agents demonstrate a significant leap toward true cognitive flexibility and widespread applicability.

The trajectory of embodied intelligence research suggests a future where robotic systems transition from pre-programmed tools to genuinely autonomous agents. Further investigation into areas like LLM-based planning and adaptive graph networks isn’t merely about incremental improvements; it’s about fundamentally altering the human-machine relationship. This ongoing work anticipates robots capable of not just executing commands, but of independently formulating goals, devising strategies, and adjusting to unforeseen circumstances – effectively shifting them from instruments of automation to collaborative partners in a diverse range of applications. The potential extends beyond industrial automation and into domains like elder care, disaster response, and even space exploration, promising a future where intelligent robots enhance human capabilities and address complex challenges with unprecedented flexibility and ingenuity.

The pursuit of robust embodied AI, as demonstrated by LookPlanGraph, necessitates a holistic approach to scene understanding and dynamic planning. The method’s reliance on graph representations to augment visual-language models echoes a fundamental tenet of systems design: structure dictates behavior. John McCarthy famously stated, “Artificial intelligence is the science and engineering of making intelligent machines, especially intelligent computer programs.” This aligns perfectly with LookPlanGraph’s ambition to create agents capable of not merely perceiving a scene, but of actively reasoning about it and adapting plans in real-time. The efficacy of the graph augmentation technique highlights how a well-defined structure, capable of representing relationships within the environment, is crucial for enabling complex task execution. Good architecture is invisible until it breaks, and only then is the true cost of decisions visible.

Where Do We Go From Here?

The elegance of LookPlanGraph resides in its attempt to mirror the recursive nature of action and perception. However, true scalability will not emerge from simply augmenting visual-language models with larger graphs. The bottleneck isn’t representation, but the fragility of planning itself. Current approaches, even those leveraging dynamic scene understanding, treat the environment as largely knowable. A more robust system must acknowledge inherent uncertainty and build plans that gracefully degrade, rather than collapse, when confronted with the unexpected.

The focus should shift from meticulously modeling the world to creating agents capable of operating effectively within its inherent ambiguity. This necessitates a move beyond task-specific benchmarks. A truly intelligent system isn’t defined by its success on contrived problems, but by its capacity to integrate novel experiences into a coherent worldview. Consider the ecosystem: each component adapts, not by optimizing for a single function, but by maintaining its viability within the broader network of interactions.

The path forward, therefore, lies in exploring architectures that prioritize adaptability and resilience. The challenge isn’t simply to see more, or plan further, but to understand the limits of both, and to construct systems that can thrive even when those limits are reached. The question is not how to build a perfect map, but how to navigate with an imperfect one.

Original article: https://arxiv.org/pdf/2512.21243.pdf

Contact the author: https://www.linkedin.com/in/avetisyan/