Seeing Without Acting: Predicting ML Agent Performance

by

Author: Denis Avetisyan

New research reveals the potential to forecast how well machine learning agents will perform before they even begin, paving the way for more efficient autonomous systems.

Traditional machine learning agents rely on iterative execution and external feedback, a process burdened by latency, whereas this work explores the potential to preemptively identify effective solutions by capitalizing on inherent, data-driven “Implicit Execution Priors” before any actual execution occurs.
Traditional machine learning agents rely on iterative execution and external feedback, a process burdened by latency, whereas this work explores the potential to preemptively identify effective solutions by capitalizing on inherent, data-driven “Implicit Execution Priors” before any actual execution occurs.

Large language models demonstrate predictive capability for machine learning solution performance, bypassing the execution bottleneck in traditional agent frameworks.

Despite advances in autonomous machine learning, a critical bottleneck remains: the need for costly physical execution to evaluate hypotheses. This work, ‘Can We Predict Before Executing Machine Learning Agents?’, introduces a framework leveraging Large Language Models to predict the performance of machine learning solutions before runtime, inspired by the principles of World Models. By internalizing execution priors and constructing a comprehensive dataset for data-centric solution preference, we demonstrate that LLMs can achieve surprisingly accurate predictions-up to 61.5%-and enable agents to converge six times faster. Could this predictive capability fundamentally reshape the landscape of autonomous experimentation and accelerate scientific discovery?

The Perpetual Bottleneck: Why We Chase Efficiency Instead of Foresight

Conventional autonomous machine learning systems frequently operate by repeatedly generating code, executing it, and then analyzing the results to refine subsequent iterations – a process that inherently creates a computational bottleneck. This ‘Generate-Execute-Feedback Loop’, while demonstrating success in various applications, struggles with scalability as task complexity increases. Each code execution demands substantial resources, and the iterative nature means these costs accumulate rapidly, particularly when exploring vast solution spaces. The sheer number of potential code variations that must be tested limits the system’s ability to efficiently discover optimal solutions, hindering progress on more demanding artificial intelligence challenges. Consequently, the reliance on direct code execution represents a fundamental obstacle to wider adoption and advancement in autonomous ML.

The prevailing paradigm of autonomous machine learning hinges on a ‘Generate-Execute-Feedback Loop’, where a potential solution is created, rigorously tested through execution, and then refined based on the results; however, this iterative process presents inherent limitations when applied to increasingly complex challenges. Each cycle necessitates full execution of the generated code, consuming substantial computational resources and time – a bottleneck that severely restricts scalability. As problem spaces expand and demand more intricate solutions, the cost of repeatedly executing every candidate becomes prohibitive, hindering the ability of autonomous agents to efficiently explore the vast landscape of possible algorithms and hyperparameters. This constraint emphasizes the need for methods that can predict the performance of solutions before full execution, enabling a more targeted and efficient search for optimal machine learning pipelines.

A fundamental obstacle in advancing autonomous machine learning lies in the inherent coupling of searching for potential solutions and the computationally expensive process of testing them. Current systems often operate by iteratively generating code, executing it to observe results, and then refining the approach based on feedback – a cycle that becomes increasingly unsustainable as problem complexity grows. The difficulty isn’t necessarily in finding innovative solutions, but in efficiently evaluating their viability without exhaustive, real-world testing. Decoupling these phases – enabling a system to predict the outcomes of code before full execution – represents a critical shift. This would allow for a far more expansive exploration of the solution space, prioritizing promising candidates and significantly reducing the reliance on costly trial-and-error, ultimately unlocking the potential for truly scalable autonomous ML.

The future of autonomous machine learning hinges on a shift from repeatedly testing potential solutions to reliably predicting their performance. Current systems are hampered by the ‘Generate-Execute-Feedback Loop’, where each candidate algorithm requires full execution to assess its merit – a process that becomes exponentially more expensive with problem complexity. To overcome this bottleneck, research is increasingly focused on developing methods for accurate performance prediction before execution, utilizing techniques like meta-learning and surrogate models. This allows for efficient exploration of the solution space, prioritizing promising algorithms and drastically reducing the need for costly trials. Crucially, robust verification methods are also needed to ensure that predicted performance translates to real-world efficacy, potentially leveraging formal methods or simulation-based testing to build confidence in these predictive systems and unlock the true potential of autonomous ML.

This framework enhances agent performance by predicting solution superiority and confidence through a data-centric approach involving preference corpus construction, input augmentation with verified data reports, and a predict-then-verify loop that filters candidates before physical execution.
This framework enhances agent performance by predicting solution superiority and confidence through a data-centric approach involving preference corpus construction, input augmentation with verified data reports, and a predict-then-verify loop that filters candidates before physical execution.

Predicting the Inevitable: World Models as Computational Shortcuts

World Models represent a computational approach to forecasting the results of algorithms prior to their actual execution, thereby minimizing resource expenditure. This predictive capability is achieved by constructing an internal representation of the problem space, allowing for the simulated evaluation of potential solutions without requiring full algorithmic processing. The computational savings are substantial, as the cost of prediction-based on the learned model-can be significantly lower than the cost of running the algorithm itself. This makes World Models particularly valuable in scenarios with high computational demands or limited resources, enabling faster iteration and more efficient problem-solving.

The core functionality of predictive world models hinges on the development of an internal representation of the specific problem space. This representation, learned through exposure to relevant data, allows the model to simulate potential solution pathways without actual execution. By constructing this implicit understanding of the problem domain – encompassing rules, constraints, and likely outcomes – the system can efficiently evaluate the predicted performance of different approaches. This internal model effectively functions as a proxy for the real-world environment, enabling rapid assessment and optimization of algorithmic solutions prior to resource-intensive execution phases.

The predictive capability of a World Model is directly correlated with the completeness and accuracy of its training data; specifically, a detailed ‘Data Analysis Report’ serves as a foundational input. Insufficient or flawed data leads to an incomplete or inaccurate internal representation of the problem domain, hindering the model’s ability to reliably forecast outcomes. This dependency necessitates rigorous data curation and validation processes; errors or biases present in the input data will propagate through the model, impacting the reliability of its predictions and potentially leading to suboptimal algorithmic solutions. The model’s performance is therefore fundamentally limited by the quality of the data used to construct its understanding of the environment.

Large language models, such as DeepSeek-V3.2-Thinking, are integral to building and improving world models by implicitly learning problem domain representations. Empirical results demonstrate the model achieves an accuracy of 61.5% in predicting outcomes, surpassing the performance of random guessing, which yields 50.0% accuracy, and complexity-based heuristics, which achieve 50.8%. This improved predictive capability indicates an ability to discern more nuanced relationships within the problem space, contributing to the overall effectiveness of the world model in forecasting algorithmic solution performance without requiring full execution.

This analysis demonstrates that a world model's predictive success relies on semantic data understanding, consistent performance across domains, accuracy decoupled from parameter scaling, robust active reasoning, calibrated self-confidence, and scalability with problem complexity.
This analysis demonstrates that a world model’s predictive success relies on semantic data understanding, consistent performance across domains, accuracy decoupled from parameter scaling, robust active reasoning, calibrated self-confidence, and scalability with problem complexity.

ForeAgent: A Shift from Reaction to Anticipation

The ForeAgent represents a departure from conventional autonomous machine learning agents through its implementation of a ‘Predict-then-Verify Loop’. Traditional agents typically execute code and then evaluate the results, a process that can be computationally expensive and time-consuming, especially during exploratory phases. In contrast, ForeAgent first predicts the likely performance of a potential solution before execution. This prediction is then verified through limited code execution, allowing the agent to quickly discard unpromising solutions and prioritize those with high predicted performance. This decoupling of prediction and execution enables a more efficient search process and reduces unnecessary computational load, addressing a key limitation of standard agent architectures.

ForeAgent substantially improves efficiency by incorporating a predictive component that estimates the performance of potential solutions before execution. This allows the agent to bypass code that is predicted to yield suboptimal results, thereby minimizing unnecessary computational expense. Benchmarks demonstrate that this predictive capability results in a 6x acceleration in the agent’s exploration speed compared to methods relying solely on trial-and-error execution. The reduction in executed code directly translates to faster iteration and a more focused search for effective solutions within a given problem space.

Confidence-Gated Pairwise Selection is a solution prioritization technique employed by the agent to refine its search strategy. This method involves generating a set of candidate solutions and then evaluating them in pairs, assessing the likelihood of one solution outperforming the other based on a confidence score derived from the Implicit World Model. The agent consistently selects the more promising solution from each pair, effectively focusing computational resources on options with a higher predicted success rate. This pairwise comparison, gated by confidence levels, reduces the need to fully evaluate less viable solutions, resulting in a more efficient exploration of the solution space and accelerating convergence towards optimal results.

ForeAgent’s predictive capabilities are grounded in an Implicit World Model, a learned representation of the environment that allows the agent to anticipate the outcomes of its actions without explicitly simulating them. This model is trained concurrently with the agent’s policy and is crucial for evaluating the potential of different code solutions before execution. Empirical results demonstrate that utilizing this Implicit World Model yields a consistent +6% performance increase across benchmark tasks when compared to agents lacking such a predictive component, indicating its efficacy in guiding exploration towards more rewarding solutions and reducing the need for exhaustive trial-and-error.

ForeAgent demonstrates superior performance by achieving a 6% improvement in task completion, converging to peak efficiency six times faster, and expanding search breadth by a factor of 3.2 compared to the AIDE baseline.
ForeAgent demonstrates superior performance by achieving a 6% improvement in task completion, converging to peak efficiency six times faster, and expanding search breadth by a factor of 3.2 compared to the AIDE baseline.

Beyond Speed: The Implications for a More Intelligent Future

The ForeAgent distinguishes itself not merely through computational speed, but through a fundamental shift in how artificial intelligence approaches problem-solving. By prioritizing predictive modeling, the system anticipates future states and dynamically adjusts its operations, exceeding the limitations of reactive algorithms focused solely on immediate efficiency. This proactive capability allows the ForeAgent to navigate complex tasks with a foresight previously unattainable, effectively circumventing bottlenecks and optimizing performance across extended sequences. The emphasis on prediction isn’t simply about doing things faster; it’s about enabling the tackling of entirely new classes of problems that demand an understanding of future consequences, thus broadening the scope of what artificial intelligence can achieve.

The ForeAgent’s predictive modeling capabilities demonstrably expand the scope of solvable problems, particularly within ‘Data-centric Solution Preference’ tasks that previously exceeded computational limits. These tasks, often involving the evaluation of numerous potential solutions across vast datasets, demand prohibitive processing power when approached through conventional methods. However, by accurately predicting the performance of solutions before full evaluation, the ForeAgent significantly reduces the computational burden. This allows researchers to explore solution spaces previously inaccessible, opening avenues for innovation in fields like materials discovery, drug design, and complex system optimization where identifying optimal configurations from a massive number of possibilities is paramount. The ability to effectively navigate these computationally intensive landscapes represents a fundamental shift, promising breakthroughs that were previously considered unattainable.

The advent of predictive AI, as demonstrated by the ForeAgent, moves beyond merely streamlining existing processes and instead facilitates the autonomous optimization of profoundly complex systems – a development with significant implications for scientific discovery. Previously, researchers faced limitations in exploring vast design spaces or simulating intricate phenomena due to prohibitive computational demands; the predictive modeling inherent in this approach circumvents these bottlenecks by intelligently forecasting outcomes and guiding optimization efforts. This capability isn’t limited to a single field; it promises to accelerate progress across disciplines, from materials science – where novel compounds can be designed and tested in silico – to drug discovery, where candidate molecules can be refined with unprecedented speed and accuracy, and even in climate modeling, allowing for more robust predictions and effective mitigation strategies. The potential extends to any field where exhaustive experimentation is impractical, effectively transforming the scientific method itself by enabling a more directed and efficient exploration of the unknown.

The predictive modeling at the core of the ForeAgent system offers significant potential beyond its initial applications, particularly when integrated with reinforcement learning algorithms. Traditionally, reinforcement learning agents require extensive trial-and-error to navigate complex environments, a process that can be computationally expensive and prone to instability. By incorporating predictive capabilities, these agents can anticipate the consequences of their actions with greater accuracy, effectively ‘looking ahead’ to assess potential outcomes. This predictive foresight not only accelerates the learning process but also enhances robustness by allowing the agent to proactively avoid unfavorable states and adapt more effectively to unforeseen circumstances. The result is a more resilient and efficient AI capable of tackling dynamic and uncertain environments with increased confidence, paving the way for advancements in areas like robotics, autonomous control, and strategic decision-making.

The Prediction Corpus exhibits a balanced and highly heterogeneous distribution of solution architectures across major machine learning paradigms-including Gradient Boosting&Trees, General/Sequential NNs, CNNs, and Transformers-suggesting a diverse range of approaches to the prediction task.
The Prediction Corpus exhibits a balanced and highly heterogeneous distribution of solution architectures across major machine learning paradigms-including Gradient Boosting&Trees, General/Sequential NNs, CNNs, and Transformers-suggesting a diverse range of approaches to the prediction task.

The pursuit of predictive capability within agent frameworks feels…familiar. This paper suggests Large Language Models can bypass the ‘Generate-Execute-Feedback’ loop, a neat trick, but it merely shifts the point of failure. The bug tracker will inevitably fill with predictions that fail to materialize. It’s a data-centric solution preference, certainly, but one that adds another layer of abstraction, another potential source of error. As Bertrand Russell observed, ‘The problem with the world is that everyone is an expert in everything.’ Here, everyone’s an expert in predicting machine learning performance without execution – until production intervenes. The elegance is apparent, but the inevitable chaos looms. They don’t deploy – they let go.

So, What Breaks First?

The demonstrated predictive capability is… intriguing. The paper sidesteps the obvious question, however: how long before production finds a corner case that renders these LLM-predicted ‘optimal’ solutions spectacularly wrong? It’s a classic case of moving the bottleneck. The ‘Generate-Execute-Feedback’ loop is avoided, yes, but replaced with the potential for systematic, unexecuted errors. One anticipates a vibrant market for ‘LLM-predicted failure’ debugging tools. Everything new is old again, just renamed and still broken.

The focus on data-centric preference is sensible, but implicitly assumes a static dataset. Real-world distributions shift. Models decay. The true test isn’t predicting performance on a held-out set, it’s predicting performance degradation over time. The framework will require robust mechanisms for continual re-evaluation and adaptation, or it will simply become another brittle abstraction.

Ultimately, this work offers a potentially valuable acceleration of the development cycle, but it’s not a solution, merely a shift in the points of failure. Production is, as always, the best QA. The field will likely move toward increasingly sophisticated meta-learning approaches – LLMs predicting which data will break which models, and then automatically generating synthetic examples to expose those weaknesses. It’s turtles all the way down, really.

Original article: https://arxiv.org/pdf/2601.05930.pdf

Contact the author: https://www.linkedin.com/in/avetisyan/

See also:

2026-01-12 17:27