Robots That Rewrite Their Own Code

by

Author: Denis Avetisyan

New research demonstrates a system where robots learn to manipulate objects by iteratively refining their control programs based on visual feedback and outcomes.

A multimodal language model learns in-context manipulation policies through an Act-Observe-Rewrite cycle, bypassing traditional reinforcement learning methods.

Traditional robot learning approaches often require extensive reward engineering or demonstrations, limiting adaptability to novel tasks. This paper introduces ‘Act-Observe-Rewrite: Multimodal Coding Agents as In-Context Policy Learners for Robot Manipulation’, a framework wherein a language model iteratively refines a robot’s manipulation policy by synthesizing new Python controller code based on visual feedback and episode outcomes. We demonstrate that this ‘Act-Observe-Rewrite’ cycle enables effective in-context learning without gradient updates or predefined skills, achieving high success rates across multiple robotic tasks. Could this approach unlock a new paradigm for creating truly autonomous and adaptable robots capable of learning directly from experience?

The Fragility of Pre-Programmed Control

Conventional robotic systems typically operate on a foundation of meticulously pre-programmed instructions, dictating every movement and response. While effective in highly structured settings, this approach proves remarkably brittle when confronted with the unpredictable nature of real-world environments. A robot designed to perform a specific task – such as assembling a product – can falter if presented with even slight deviations from the anticipated conditions, like a misplaced component or an obstruction in its path. This limitation stems from a fundamental inability to extrapolate beyond the explicitly defined parameters of its programming, hindering its capacity to react intelligently to unforeseen circumstances and demanding a shift towards more flexible and robust control architectures. The reliance on pre-defined behaviors therefore restricts a robot’s autonomy and its potential for operation in dynamic and unstructured spaces.

The limitations of rigidly programmed robots become strikingly apparent when confronted with real-world scenarios – environments that are rarely static or predictable. Existing robotic systems often falter when tasked with navigating cluttered spaces, manipulating deformable objects, or responding to unexpected changes, highlighting a critical need for more versatile control strategies. This inability to gracefully handle dynamic complexity isn’t simply a matter of improved sensors or faster processors; it signifies a fundamental gap in how robots are designed to interact with the world. Consequently, researchers are actively pursuing new paradigms centered around adaptability, aiming to create systems capable of learning, improvising, and adjusting their behavior on the fly – a shift that promises to unlock the full potential of robotics in fields ranging from manufacturing and logistics to healthcare and disaster response.

Successfully navigating unpredictable environments demands more than just swift mechanical responses; it requires robots to seamlessly integrate what they sense with the ability to plan and execute appropriate actions, all in real-time. This integration – bridging perception, reasoning, and action – presents a significant hurdle for current robotic systems. The difficulty isn’t simply processing data, but rather constructing a cohesive understanding of the environment from noisy sensory input, formulating a logical response based on that understanding, and then translating that response into precise motor commands – all within fractions of a second. Current approaches often treat these stages as separate processes, creating bottlenecks and delays that limit a robot’s agility and adaptability. Achieving true autonomy necessitates a unified architecture where perception informs reasoning, and reasoning directly guides action, fostering a continuous loop of learning and refinement that allows robots to respond effectively to the unexpected.

Contemporary robotic systems often exhibit a surprising lack of adaptability, frequently necessitating substantial re-programming even when confronted with slight deviations from their training parameters. This brittleness stems from a reliance on narrowly defined algorithms, effective only within the precise conditions for which they were designed. A robot proficient at assembling one variation of a product, for instance, may struggle immensely with a seemingly minor alteration, demanding a complete overhaul of its operational code. This limitation isn’t merely a matter of inconvenience; it drastically restricts the deployment of robots in real-world scenarios characterized by inherent unpredictability and constant change, highlighting the urgent need for systems capable of robust generalization and autonomous learning from limited experience.

In-Context Policy Learning: A Paradigm Shift in Robotic Control

In-Context Policy Learning presents a departure from traditional robot control methods by enabling behavioral refinement without necessitating updates to the underlying model weights. This is achieved by leveraging the capacity of large language models to process observational data and generate modified control policies on-the-fly. Consequently, the system exhibits increased flexibility and speed in adapting to new scenarios or correcting errors, as adjustments are made through policy generation rather than computationally expensive model retraining. This approach facilitates rapid iteration and allows for continuous improvement without disrupting the core functionality established by the initial model parameters.

In-Context Policy Learning utilizes large language models (LLMs) as a core component for robotic control by enabling the interpretation of sensory observations and the subsequent generation of improved control policies. These LLMs, pre-trained on extensive datasets of code and natural language, are prompted with current robot state information – including sensor readings and task goals – to produce new or modified control instructions. The LLM’s ability to understand complex relationships and generalize from prior knowledge allows it to synthesize control policies without requiring explicit gradient updates or retraining of the robot’s underlying control system. This approach effectively transforms the robot control problem into a prompting and completion task for the LLM, allowing for rapid adaptation and policy refinement based on observed outcomes.

The Act-Observe-Rewrite framework underpins in-context policy learning by iteratively improving robot control based on real-time experience. Initially, the robot acts based on a given policy. Subsequently, the system observes the outcome of that action, recording relevant data about the environment and the robot’s state. This observational data then informs the rewrite stage, where a new control policy – specifically, new Python controller code – is synthesized. This rewritten policy is then implemented in the next iteration, creating a closed loop of action, observation, and refinement that allows the robot to adapt and improve its performance without requiring updates to the underlying model weights.

The system achieves real-time adaptation by generating new Python controller code following each trial outcome. This process avoids traditional model weight updates, instead dynamically creating and implementing modified control logic based on observed performance. Specifically, the system attained a 100% success rate on both the Lift and PickPlaceCan tasks through this iterative code synthesis, demonstrating the efficacy of this approach for robotic manipulation. The generated code directly dictates robot actions, allowing for immediate behavioral changes and continuous improvement without requiring retraining of underlying models.

Multimodal LLM Agents: Visual Perception and Reasoning

The Multimodal LLM Agent addresses failure analysis by correlating data from visual observations with episode outcome information. This integration allows the agent to move beyond simply identifying that a failure occurred, and instead determine why it occurred within a specific context. The system achieves this by associating observed visual states – derived from image processing – with the resulting episode outcome, effectively creating a causal link between environmental factors and task success or failure. This process enables the agent to isolate root causes, differentiating between failures due to environmental obstacles, incorrect actions, or internal system limitations, and forming the basis for targeted remediation strategies.

Foundation Model Integration within the agent framework enables the utilization of pre-trained models for complex reasoning and problem-solving tasks related to observed failures. This allows the system to move beyond simple pattern recognition and towards contextual understanding of the visual data and episode outcomes. Complementing this is Code Synthesis, where the agent automatically generates executable code – typically Python scripts – to implement potential solutions. This code is then executed within the environment, allowing for rapid prototyping and testing of hypotheses. The combination of leveraging existing knowledge from foundation models and dynamically generating corrective code facilitates a more robust and adaptable problem-solving approach compared to static, pre-programmed responses.

The Vision Pipeline processes raw image data to enable object identification through techniques including HSV color segmentation. This method transforms the standard RGB color space into the Hue, Saturation, and Value (HSV) space, allowing for more robust object detection independent of lighting variations. By defining specific ranges for these HSV components, the system can isolate and identify objects of interest within the image. This segmentation facilitates subsequent analysis and interaction with the identified objects by the agent, providing crucial visual input for task completion.

The Vision Pipeline utilizes a back-projection formula to translate 2D image coordinates into 3D world coordinates, enabling spatial reasoning and interaction with the environment. This formula is specifically implemented according to the OpenGL convention, which defines a right-handed coordinate system with the z-axis pointing outwards from the screen. The back-projection process involves inverting the projection matrix used to render the 3D scene onto the 2D image plane; given a pixel location [latex](x_p, y_p)[/latex] in the image, the formula calculates the corresponding 3D point [latex](X, Y, Z)[/latex] in world space. Accurate implementation of the OpenGL convention is critical for correctly interpreting depth and spatial relationships within the visual data, allowing the agent to determine object positions and distances.

Robustness Through Adaptive Grasping and Iterative Refinement

The system incorporates an automated grasp retry mechanism designed to bolster task completion rates by intelligently addressing initial failure points. When a grasp attempt proves unsuccessful – perhaps due to slippage or an unstable initial hold – the robotic system doesn’t simply abandon the task. Instead, it automatically re-attempts the grasp, leveraging previously acquired visual and tactile data to refine its approach. This iterative process of grasping, evaluating success, and re-attempting, if necessary, significantly increases the probability of successfully manipulating objects. The implementation proves particularly valuable in dynamic or cluttered environments where initial grasp attempts are more prone to error, effectively building resilience into the robotic manipulation pipeline and ultimately achieving higher overall task success.

The robotic system’s ability to consistently complete tasks, even when initial attempts fail, stems from a deliberate process of adaptation and recovery. Rather than rigidly adhering to pre-programmed sequences, the control policies are continuously refined through repeated trials and observations. This iterative process allows the system to learn from its mistakes, adjusting its approach to overcome obstacles and improve success rates. Crucially, the incorporation of specific error recovery strategies-built-in procedures for addressing common failures-enhances the system’s resilience. This means that even when a grasp initially fails, the robot doesn’t simply stop, but instead actively attempts to correct the situation, demonstrating a robust capacity to handle unexpected challenges and maintain performance across diverse scenarios.

Recent advancements in robotic manipulation leverage in-context learning, but these systems often benefit from enhanced contextual understanding and reasoning. Methods such as Reflexion and ReAct address this need by equipping robots with the ability to not merely react to outcomes, but to reflect on their actions and reason about potential improvements. Reflexion, for instance, allows a robot to analyze its failures, identify the root causes, and then revise its approach – effectively learning from its mistakes. Similarly, ReAct combines reasoning traces with action sequences, enabling the robot to articulate its thought process while executing tasks, which facilitates more robust and adaptable behavior. This integration of reasoning and action allows robots to move beyond simple stimulus-response cycles, fostering a greater degree of autonomy and success in complex, real-world scenarios.

The Act-Observe-Rewrite (AOR) framework represents a substantial step forward in robotic manipulation, achieving complete success in the Lift and PickPlaceCan tasks when paired with a multimodal language model. This framework doesn’t simply execute actions; it actively observes the outcomes, and then rewrites its plan based on those observations-a process mirroring human problem-solving. While demonstrating impressive performance, particularly reaching a 91% success rate on the more complex Stack task, the system isn’t without limitations. Current research indicates challenges remain in consistently resolving all placement failures within the Stack task, suggesting ongoing refinement is needed to ensure robust and reliable performance across a wider range of scenarios and complexities.

The presented work embodies a fundamentally mathematical approach to robotic control. It eschews the empirical trial-and-error typical of reinforcement learning, instead favoring an iterative refinement of code predicated on observable outcomes-a process akin to proving a theorem by successive refinement. As Barbara Liskov aptly stated, “Programs must be correct, and one way to ensure that is to prove them correct.” This paper demonstrates precisely that principle; the agent doesn’t simply act and hope for the best, but actively observes, analyzes, and then rewrites its control logic-a reflexive learning loop driven by the pursuit of provable correctness, mirroring the rigorous standards of mathematical reasoning. The focus on code synthesis, rather than merely training a policy, establishes a foundation for verifiable robot behavior.

What Remains Constant?

The presented work skirts the issue of true generalization. A system capable of iteratively refining code based on immediate feedback is intriguing, but begs the question: Let N approach infinity – what remains invariant? The current approach, while avoiding explicit reinforcement learning, remains fundamentally tethered to the specifics of the observation space and the initial, albeit adaptable, code base. A truly elegant solution would not require such scaffolding; it would deduce the underlying physics and desired outcome directly from abstract goals.

Future efforts should not focus solely on increasing the complexity of the observation-rewrite loop. Instead, a deeper exploration of symbolic reasoning within these multimodal agents is warranted. Can the system, for example, prove the correctness of its rewritten code, or is it forever trapped in a cycle of empirical refinement? The limitations of current language models in formal verification are well-documented, and overcoming these will be crucial for achieving robust and reliable robot manipulation.

Ultimately, the promise of this line of inquiry lies not in creating robots that can mimic intelligent behavior, but in using them as a substrate for exploring the fundamental principles of adaptation and control. The elegance, if it exists, will not be in the code itself, but in the mathematical structure that underpins it.

Original article: https://arxiv.org/pdf/2603.04466.pdf

Contact the author: https://www.linkedin.com/in/avetisyan/

See also:

2026-03-07 07:29