Author: Denis Avetisyan
A new approach uses self-organizing artificial intelligence agents to accelerate scientific discovery through continuous, parallel experimentation.
AutoScientists is a multi-agent system that dynamically assembles teams to optimize long-horizon scientific tasks, demonstrating state-of-the-art results in biomedical machine learning and protein engineering.
Sustained progress in scientific discovery demands overcoming the limitations of single-trajectory or centrally-planned AI systems. To address this, we introduce AutoScientists: Self-Organizing Agent Teams for Long-Running Scientific Experimentation, a decentralized multi-agent system enabling dynamic team formation and parallel hypothesis exploration. This approach achieves state-of-the-art results across biomedical machine learning, language-model training optimization, and protein fitness prediction-improving over prior AI agents by up to +12.5% in Spearman correlation-and demonstrates the potential to accelerate scientific progress through autonomous, collaborative experimentation. Can this paradigm shift unlock fundamentally new approaches to complex scientific challenges and ultimately reshape the future of research?
The Challenge of Accurate Prediction
The ability to accurately forecast how alterations in a proteinās structure-even at the level of a single amino acid-will impact its function is paramount to progress in both drug discovery and materials science. A proteinās function is inextricably linked to its three-dimensional shape, and even subtle changes can drastically alter its behavior, potentially rendering a therapeutic ineffective or creating a material with undesirable properties. However, proteins are incredibly complex molecules, and the number of possible variants is astronomical, making exhaustive experimental testing impractical. Consequently, predicting these effects in silico presents a formidable challenge, requiring computational models capable of navigating an immense sequence space and accurately capturing the intricate relationships between sequence, structure, and function-a task that continues to push the boundaries of current scientific capabilities.
Protein sequences, though linear chains of amino acids, fold into extraordinarily complex three-dimensional structures that dictate their function. Traditional methods attempting to predict the effects of genetic mutations on these proteins are hampered by this inherent complexity; even seemingly minor alterations can trigger significant structural changes and functional consequences. Furthermore, the sheer scale of possible mutations-a protein of even modest length presents billions of potential variants-creates a combinatorial explosion that overwhelms conventional computational approaches. These methods often rely on simplified models or limited datasets, failing to capture the intricate interplay of forces governing protein behavior and resulting in inaccurate predictions, particularly when faced with novel mutations outside of their training data.
The limitations of current protein variant effect prediction necessitate the creation of predictive models capable of handling exponentially growing datasets and increasingly complex protein structures. Researchers are actively pursuing novel approaches, including machine learning algorithms trained on vast genomic and proteomic data, to overcome the computational bottlenecks of traditional methods. These emerging models aim not only for greater accuracy in predicting the functional consequences of mutations – such as altered stability, binding affinity, or enzymatic activity – but also for scalability, enabling the rapid screening of millions of potential protein variants. A robust predictive capability promises to accelerate breakthroughs in diverse fields, from the rational design of therapeutic antibodies and enzymes to the development of novel biomaterials with tailored properties, ultimately streamlining the process of protein engineering and discovery.
Current protein engineering prediction methods frequently operate as static models, trained on existing datasets and unable to effectively incorporate new experimental findings. This limitation proves particularly problematic given the exponential growth of protein sequence data and the continuous discovery of novel protein structures and functions. Without the capacity for continuous learning, these models experience diminishing returns as they encounter variants outside their original training distribution, hindering their ability to accurately predict the effects of mutations in newly discovered or engineered proteins. Consequently, predictions become less reliable over time, necessitating frequent retraining and limiting the potential for truly adaptive protein design and discovery pipelines. The inability to dynamically refine predictions based on evolving data represents a significant bottleneck in translating genomic information into functional proteins with desired properties.
A Framework for Continuous Scientific Exploration
AutoScientists employs a decentralized, multi-agent system for automated scientific discovery, distinguished by the absence of a central control mechanism. Each agent within the team operates autonomously, perceiving the experimental environment, formulating hypotheses, and executing tests based on local information and interactions with other agents. This distributed architecture facilitates scalability and robustness; the systemās performance is not limited by the capacity of a single coordinating entity, and failure of individual agents does not necessarily compromise the overall experimentation process. The design prioritizes long-term research goals by enabling continuous operation and adaptation without requiring constant human intervention or predefined experimental sequences.
The Shared State within the AutoScientists framework functions as a globally accessible data structure that facilitates inter-agent communication and coordinated action. This repository contains experimental results, hypotheses, planned experiments, and agent observations, all formatted in a standardized manner. Agents continuously read from and write to the Shared State, allowing them to monitor the progress of the overall experimentation process and react to findings generated by other agents. This eliminates the need for explicit messaging or a central control system, enabling asynchronous and distributed decision-making. The Shared State supports data types including numerical measurements, text-based observations, and representations of experimental parameters, ensuring comprehensive information exchange among the AutoScientists agents.
AutoScientists employs self-organization principles, enabling adaptation to dynamic experimental conditions without centralized control. This is achieved through decentralized agent interactions and emergent behaviors, where agents react to local information and the actions of other agents. Specifically, agents monitor experimental outcomes and adjust their individual experimental designs based on observed data and the collective performance of the team. This distributed approach allows the system to continually optimize its experimental strategy, identify promising research directions, and respond to unexpected results without requiring a global planner or external intervention. The resulting system is robust to agent failures and can efficiently explore complex experimental landscapes.
AutoScientists facilitate long-horizon experimentation through persistent hypothesis management. Unlike traditional experimentation frameworks focused on short-term, isolated tests, AutoScientists maintain evolving hypotheses across extended durations. This is achieved by storing hypotheses, along with associated experimental data and rationales, within the systemās āShared Stateā. Agents can then iteratively refine these hypotheses based on new evidence gathered from ongoing experiments, allowing for complex investigations that span numerous iterations and adapt to unforeseen results. The system doesnāt simply evaluate a hypothesis; it actively updates it, incorporating learnings from each experimental cycle to guide future investigations and maximize the potential for discovering complex relationships.
Kermut: A Gaussian Process Foundation for Prediction
Kermut leverages Gaussian Process (GP) modeling, a probabilistic approach well-suited for regression and uncertainty quantification, as its central predictive component. GPs define a probability distribution over functions, allowing Kermut to not only predict protein properties but also to estimate the confidence in those predictions. This is achieved by defining a kernel function that quantifies the similarity between protein sequences, enabling generalization to unseen data. The core strength of GPs lies in their ability to model complex, non-linear relationships with relatively limited training data, and Kermut builds upon this by incorporating features from other models to further refine the kernel and improve predictive accuracy. The output of the GP is a predictive mean and variance, providing a measure of uncertainty alongside the prediction itself.
Kermut integrates features derived from ESM-2, a large language model trained on millions of protein sequences, to improve the representation of protein data. These features, generated through ESM-2ās analysis of amino acid composition and relationships, provide Kermut with a richer understanding of protein sequence characteristics beyond simple primary structure. Specifically, ESM-2 outputs contextual embeddings that capture evolutionary information and biophysical properties, which are then incorporated as input features into Kermutās Gaussian Process model. This allows Kermut to more accurately predict protein properties and behavior by leveraging the pre-trained knowledge embedded within ESM-2.
Kermutās performance is subject to ongoing improvement via automated experimentation managed by a system of āAutoScientistsā. These agents autonomously design and execute experiments, systematically varying model parameters and training data to identify configurations that optimize predictive accuracy and generalization capability. This iterative process allows for continuous refinement of the model, addressing potential vulnerabilities and enhancing robustness against diverse protein sequences and prediction scenarios. The AutoScientist framework facilitates a data-driven approach to model development, bypassing manual tuning and accelerating the pace of improvement beyond what would be feasible with traditional methods.
The Kermut system employs an ensemble method, integrating predictions from multiple independently trained Gaussian Process models to improve overall performance and reduce bias. This approach involves training several models on different subsets of the data or with varying hyperparameters. The final prediction is then generated by averaging the outputs of these individual models. This averaging process reduces the impact of any single modelās errors or biases, leading to more robust and accurate predictions. Specifically, the ensemble method helps to mitigate overfitting and improve generalization to unseen protein sequences, thereby enhancing the reliability of Kermutās predictions.
![Across BioML-Bench task types, AutoScientists commonly utilize multiple method categories, including gradient-boosted trees ([latex]GBM[/latex]), pretrained foundation models (both frozen and fine-tuned: [latex]FM-frozen[/latex], [latex]FM-finetune[/latex]), and models trained from scratch ([latex]NN-scratch[/latex], kernel ridge, or linear/ridge regression), often in conjunction with heuristic approaches.](https://arxiv.org/html/2605.28655v1/FIG/biomlbench_method_categories.png)
Validation and Optimization Through Rigorous Benchmarking
AutoScientists establishes a new benchmark in artificial intelligence for scientific discovery, demonstrably surpassing the performance of existing AI agents across a suite of rigorous tests. Achieving an overall score of 74.40 on the competitive BioML-Bench leaderboard, the system showcases an 8.33 percentage point improvement-a significant leap forward in the field. This advancement isnāt merely incremental; it highlights AutoScientistsā capacity to navigate complex scientific challenges with enhanced accuracy and efficiency, signaling a substantial step towards automating and accelerating the pace of discovery in areas like biology and materials science. The results validate the systemās core design principles and offer a promising trajectory for future AI-driven research.
The performance of Kermut, the AI agent at the heart of AutoScientists, is subjected to stringent evaluation through established scientific benchmarks, notably BioML-Bench and ProteinGym. These platforms arenāt merely used for a single assessment; they facilitate continuous tracking of Kermutās development over time, allowing for iterative improvements and objective measurement of progress. BioML-Bench provides a diverse suite of biological machine learning tasks, while ProteinGym focuses specifically on protein design and engineering challenges. By consistently testing Kermut against these standards, researchers can pinpoint areas of strength and weakness, ensuring that advancements are not only statistically significant but also translate into real-world improvements in biological understanding and predictive capabilities. This rigorous benchmarking process is central to AutoScientistsā ability to consistently achieve state-of-the-art results and push the boundaries of AI-driven scientific discovery.
To ensure robust and reliable predictions, the AutoScientists framework incorporates a rigorous cross-validation process. This technique involves partitioning the available data into multiple subsets, iteratively training the model on a portion of the data and evaluating its performance on the remaining, unseen data. By repeatedly assessing the modelās ability to generalize across different data splits, cross-validation effectively mitigates the risk of overfitting – a scenario where the model performs well on the training data but poorly on new, previously unseen examples. This careful evaluation strategy not only enhances the modelās predictive accuracy but also provides a more realistic estimate of its performance in real-world applications, bolstering confidence in its scientific insights and ensuring the findings are not merely artifacts of the training dataset.
Throughout the iterative development of AutoScientists, a āChampion Modelā serves as a constantly updated benchmark of peak performance. This model isnāt a static entity; rather, itās dynamically replaced whenever a newly trained version demonstrably surpasses its capabilities across established evaluation metrics. Maintaining this champion isnāt merely about tracking progress; it provides a crucial point of comparison for every experiment, ensuring that all subsequent iterations are rigorously tested against the best-performing configuration to date. This process facilitates a consistent upward trajectory in predictive accuracy and efficiency, as each candidate model strives to unseat the current champion, ultimately driving the system towards state-of-the-art results on benchmarks like BioML-Bench and ProteinGym.
A key demonstration of AutoScientistsā capabilities lies in its enhanced ability to predict protein interactions, specifically evidenced by improvements on the ProteinGym ACE2-Spike binding benchmark. Initial performance registered at 0.747, but through iterative optimization and rigorous testing, the system achieved a score of 0.840. This substantial increase signifies a heightened accuracy in modeling the binding affinity between the ACE2 receptor and the Spike protein – a critical interaction in understanding and combating viral infections. The improvement doesnāt merely represent a numerical gain; it suggests a more refined understanding of the underlying biophysical forces governing protein binding, potentially accelerating drug discovery and the development of targeted therapies.
AutoScientists showcases a marked improvement in experimental efficiency during GPT nanochat training. The system achieved identical intermediate validation loss figures to those of Autoresearch, but required significantly fewer experimental iterations – 34 experiments compared to Autoresearchās 65. This nearly twofold reduction in the number of trials highlights AutoScientistsā capacity for more rapid model optimization and suggests a more intelligent search strategy within its experimental design. Such efficiency not only accelerates the development process but also conserves computational resources, making it a practical advantage for large-scale scientific endeavors.
AutoScientists employs the optimization algorithm known as Gradient Descent to refine the internal settings of its models, systematically enhancing their ability to make accurate predictions. This iterative process allows the system to navigate the complex landscape of possible parameter combinations and converge on solutions that minimize errors. Notably, the system achieved a validation bits-per-byte score of 0.9730, a metric reflecting the efficiency of the modelās predictions – a result that surpasses the capabilities of the Autoresearch system, which reported no accepted improvements in this area. This demonstrates not only an increase in predictive accuracy, but also a more compact and efficient representation of learned knowledge within the model.
The pursuit of AutoScientists exemplifies a commitment to reducing complexity in scientific discovery. This system, predicated on dynamic team formation and parallel hypothesis exploration, mirrors a surgical approach to problem-solving. It actively strips away unnecessary layers, focusing on efficient experimentation and optimization. As Carl Friedrich Gauss once stated, āIf other mathematicians had not already discovered it, I would have.ā This sentiment underscores the core principle behind AutoScientists: elegance and efficiency are paramount. The system isn’t merely about achieving results; itās about uncovering fundamental truths with the least possible interference, a relentless pursuit of clarity in the realm of biomedical machine learning.
Future Directions
The proliferation of autonomous experimentation frameworks, such as AutoScientists, inevitably shifts the central challenge from doing science to curating it. The system demonstrates a capacity for parallel hypothesis exploration, but true progress demands a refinement of the metrics by which these explorations are judged. Current benchmarks, while useful, represent local optimizations within a vast, poorly-mapped landscape. The real bottleneck isnāt algorithmic speed, but the scarcity of genuinely informative signals-the ability to distinguish meaningful variation from stochastic noise.
Further work must address the brittleness inherent in transferring learned strategies across domains. Each biomedical problem presents a unique topography of constraints, and a reliance on narrowly-tuned agents risks diminishing returns. A more elegant solution lies in cultivating agents capable of meta-learning-of adapting their experimental protocols in situ, guided not by pre-defined objectives, but by a minimalist aesthetic of information gain. The pursuit of complexity is a distraction; the goal is to distill knowledge to its irreducible essence.
Ultimately, the success of these systems will be measured not by the novelty of their discoveries, but by their ability to discard unproductive avenues of inquiry. Beauty, after all, resides not in the accumulation of data, but in the ruthless elimination of the superfluous. The art of scientific progress isn’t building monuments; itās practicing elegant demolition.
Original article: https://arxiv.org/pdf/2605.28655.pdf
Contact the author: https://www.linkedin.com/in/avetisyan/
2026-05-29 02:18