The Language of Matter: Shared Representations Emerge in Scientific AI

by

Author: Denis Avetisyan

New research reveals that artificial intelligence models trained on diverse scientific data-from molecules to materials-are beginning to ‘understand’ matter in a remarkably similar way.

As predictive performance improves across a diverse set of molecular structures from the QM9 dataset-measured by decreasing energy regression MAE-representational similarity to the highest-performing model increases, a relationship quantified using the global metric $dCor$ rather than localized measures like CKNNA due to the out-of-distribution nature of the QM9 molecules.
As predictive performance improves across a diverse set of molecular structures from the QM9 dataset-measured by decreasing energy regression MAE-representational similarity to the highest-performing model increases, a relationship quantified using the global metric $dCor$ rather than localized measures like CKNNA due to the out-of-distribution nature of the QM9 molecules.

Foundation models across chemistry, materials science, and biology are converging on shared latent spaces, suggesting a universal representation of physical reality.

Despite the rapid development of machine learning for scientific discovery, it remains unclear whether models predicting the behavior of molecules, materials, and proteins are learning comparable internal representations of physical reality. In ‘Universally Converging Representations of Matter Across Scientific Foundation Models’, we demonstrate that representations learned by nearly sixty such models-spanning diverse modalities and architectures-are surprisingly aligned across a broad range of chemical systems. This convergence suggests the emergence of a shared, underlying representation of matter, with improved models exhibiting increasingly similar latent spaces. However, this alignment remains limited by training data, raising the question of whether we can engineer truly universal representations capable of generalizing beyond current limitations and unlocking the full potential of scientific foundation models?

Orchestrating Scientific Insight: Representing Complexity

Scientific progress fundamentally relies on the ability to translate raw data into meaningful representations, yet conventional methods frequently struggle with the inherent complexities of natural phenomena. Traditional approaches, often reliant on manually engineered features or simplified linear models, can lose crucial information when dealing with high-dimensional datasets or intricate relationships – for example, failing to capture non-linear interactions between genes or the subtle variations in material properties that dictate performance. This limitation hinders not only the ability to accurately model these systems, but also to effectively communicate findings and build predictive models capable of generalization. Consequently, a critical need exists for data representations that preserve the richness and nuance of scientific data, allowing for more robust analysis and ultimately, accelerated discovery.

The exponential growth of data in materials science and biology presents a significant challenge to traditional analytical methods. Researchers are now confronted with datasets of unprecedented scale and intricacy, often encompassing high-dimensional parameters and complex interdependencies. This deluge of information demands a shift towards representation learning – techniques that enable algorithms to automatically discover and encode meaningful patterns from raw data. Instead of relying on hand-engineered features, these approaches aim to create compact, informative representations that capture the essential characteristics of materials and biological systems, facilitating tasks like property prediction, materials discovery, and the understanding of complex biological processes. The development of such methods is not merely a matter of computational efficiency; it is becoming fundamentally necessary to unlock the knowledge hidden within these increasingly complex datasets and accelerate scientific progress.

A significant obstacle in contemporary scientific modeling lies in the limited transferability of predictive models between different fields. Current representation learning techniques, while often successful within a specific domain – such as predicting protein folding or material properties – frequently falter when applied to datasets from dissimilar scientific areas. This lack of generalization stems from the models’ tendency to overfit to the nuances of their training data, capturing spurious correlations rather than fundamental principles. Consequently, researchers often find themselves repeatedly developing bespoke models for each new problem, a process that is both time-consuming and computationally expensive. The pursuit of universally applicable representations, capable of extracting underlying patterns across diverse scientific landscapes, remains a central challenge in accelerating discovery and fostering true interoperability between disciplines.

Embeddings from models trained on materials and small molecules exhibit significantly lower intrinsic dimensionality compared to those generated by random embeddings or large language models.
Embeddings from models trained on materials and small molecules exhibit significantly lower intrinsic dimensionality compared to those generated by random embeddings or large language models.

Foundation Models: A Paradigm Shift in Scientific Representation

Foundation models represent a shift in representation learning achieved through pre-training on extremely large datasets, often orders of magnitude larger than those used in traditional machine learning. This pre-training process allows the model to learn a general understanding of the data distribution, enabling effective transfer learning. Unlike models trained for specific tasks, foundation models can be adapted to a wide range of downstream scientific applications – such as protein structure prediction, materials discovery, and climate modeling – with minimal task-specific fine-tuning. This capability reduces the need for extensive labeled data for each individual task and accelerates scientific progress by leveraging shared knowledge acquired during the pre-training phase. The effectiveness of this transfer is directly correlated to the scale of the pre-training dataset and the model’s architectural capacity to capture complex relationships within the data.

Foundation models leverage self-supervised learning on large-scale datasets to identify and encode inherent data structures and correlations, rather than being explicitly programmed for specific tasks. This process results in the development of high-dimensional vector representations – or embeddings – that capture semantic relationships between data points. Consequently, these learned representations serve as a generalized feature extractor, enabling effective transfer learning to a variety of downstream applications with minimal task-specific fine-tuning. The richness of these representations stems from the model’s exposure to extensive data diversity, allowing it to discern patterns and dependencies often missed by traditional, narrowly-focused algorithms. This capability reduces the need for extensive labeled datasets for each new task, lowering development costs and accelerating scientific discovery.

The utility of foundation models in scientific applications is directly determined by the fidelity of the representations learned during pre-training and their subsequent capacity for generalization. A high-quality representation captures salient features and relationships within the training data, allowing the model to perform well on tasks it was not explicitly trained for. Generalization performance, often evaluated on held-out datasets, is a critical metric; poor generalization indicates the model has either overfit to the training data or failed to learn robust, transferable features. Evaluation typically involves assessing performance across a range of downstream tasks and datasets, quantifying the model’s ability to accurately predict outcomes or infer patterns in novel data instances. Factors influencing representation quality and generalization include the size and diversity of the training dataset, the model architecture, and the pre-training objectives employed.

Distance correlation analysis reveals the relationships between structural embeddings learned by each model for the QM9 dataset.
Distance correlation analysis reveals the relationships between structural embeddings learned by each model for the QM9 dataset.

Dissecting Representation Quality: A Multi-Metric Framework

Assessing the quality of learned data representations necessitates evaluating multiple characteristics beyond simple predictive performance. A comprehensive approach considers both the complexity of the representation – how efficiently it encodes information – and its completeness – the extent to which relevant information is retained. Low complexity indicates a parsimonious representation, potentially avoiding overfitting, while high completeness ensures sufficient information is present for downstream tasks. Failing to consider both aspects can lead to misinterpretations; a highly complex representation may not generalize well, and an incomplete one may limit performance. Therefore, a multifaceted evaluation using diverse metrics is crucial for understanding the true quality of learned representations and their suitability for various applications.

Intrinsic Dimensionality ($IdI_d$) serves as a quantifiable metric for assessing the complexity of learned representations. It estimates the number of degrees of freedom a model utilizes to capture the essential information within a dataset. Empirically determined values for several benchmark datasets demonstrate the range of complexity across different molecular representations; the QM9 dataset exhibits an $IdI_d$ of approximately 5, indicating a relatively compact representation, while OMat24 and OMol25 both show higher complexity with values around 10. The sAlex dataset falls between these, with an $IdI_d$ of approximately 8, suggesting a moderate level of representational complexity. Lower $IdI_d$ values generally indicate a more efficient representation, while higher values may suggest redundancy or the capture of less relevant details.

Information Imbalance (II) serves as a metric to determine the completeness of a learned representation by quantifying the extent to which relevant information is retained during the representation process. Specifically, II measures the variance in the information content across different dimensions of the representation space; a high II value indicates that certain dimensions are disproportionately informative while others contribute little, suggesting incomplete utilization of the representational capacity. This metric is calculated as $II = 1 – \frac{\sum_{i=1}^{D} (\sigma_i / \sum_{j=1}^{D} \sigma_j)^2}{D}$, where $D$ is the dimensionality of the representation and $\sigma_i$ represents the variance along dimension $i$. A lower II value indicates a more balanced and complete representation, implying that all dimensions contribute meaningfully to the encoded information.

Model alignment is quantified using the CKNNA and Distance Correlation (dCor) metrics to determine the consistency of representations across different models. CKNNA, or Centered Kernel Normalized Alignment, provides a measure of similarity between the learned feature spaces; higher CKNNA values indicate strong alignment within a modality. Distance Correlation ($dCor$) assesses statistical dependence between model representations, with values approaching one indicating strong alignment and consistent representation of the input data. Increases in CKNNA values generally correlate with improvements in model performance, suggesting that as models become more accurate, their internal representations also become more aligned.

Information imbalance plots reveal that models trained with higher values of <i>k</i> exhibit increasingly similar learned representations, particularly when trained on the same dataset.
Information imbalance plots reveal that models trained with higher values of k exhibit increasingly similar learned representations, particularly when trained on the same dataset.

Bridging Disciplines: Benchmarking Representations Across Scientific Domains

A rigorous assessment of molecular representation learning necessitates evaluating models across diverse scientific domains. To facilitate this, researchers applied a suite of standardized metrics to four key datasets: RCSB PDB, a repository of protein structures; OMol25, a collection of organic molecules; sAlex, a dataset focused on solubility prediction; and QM9, containing quantum chemical properties of molecules. This comprehensive benchmarking strategy allows for a direct comparison of how effectively different models capture the underlying characteristics of data originating from fields like structural biology, chemistry, and materials science. By evaluating performance on these varied datasets, scientists gain valuable insight into the generalizability and limitations of each representation, ultimately guiding the selection of optimal models for specific scientific challenges and accelerating the pace of discovery.

The investigation into foundation models reveals a nuanced landscape of representational capabilities; certain models demonstrate exceptional performance when applied to specialized datasets, indicating a strength in capturing the intricacies of specific scientific domains. Conversely, other models exhibit broader generalizability, effectively representing data across multiple disciplines, suggesting the emergence of increasingly aligned latent spaces. This alignment, observed as performance improves, implies that these models are learning underlying principles applicable to diverse scientific challenges. The study highlights that the most effective representation isn’t necessarily a ‘one-size-fits-all’ solution, but rather a spectrum ranging from specialized expertise to broad competence, offering researchers the opportunity to select the optimal model based on the specific demands of their scientific inquiry.

Careful benchmarking of scientific representations yields crucial guidance for researchers seeking optimal solutions for specific tasks. The ability to discern which representation – be it a graph neural network embedding or a transformer-based encoding – best captures the relevant information within a dataset directly impacts predictive performance. This selection process isn’t merely about achieving higher accuracy; it’s about accelerating the pace of scientific discovery by minimizing the need for extensive hyperparameter tuning and model experimentation. By providing a clear understanding of representation strengths and weaknesses across diverse scientific domains, such as protein structure, molecular properties, and materials science, this work empowers scientists to choose the most efficient and effective tools for their investigations, ultimately fostering innovation and breakthroughs.

UMAP visualization reveals that the model effectively represents diverse chemical structures from sAlex, OMat24, QM9, and OMol25 datasets in a two-dimensional space.
UMAP visualization reveals that the model effectively represents diverse chemical structures from sAlex, OMat24, QM9, and OMol25 datasets in a two-dimensional space.

The research highlights an intriguing convergence in how scientific foundation models represent matter, irrespective of the specific modality-be it molecules, materials, or proteins. This suggests a fundamental underlying structure to physical reality being progressively unveiled through machine learning. As Henri Poincaré observed, “It is through science that we obtain a knowledge of the universe, and by means of mathematics that we express this knowledge.” The study’s findings echo this sentiment; the models aren’t simply memorizing data, but rather distilling it into a shared latent space – a mathematical language describing the essence of material properties. This universal representation, while still emergent, implies that simplifying assumptions and clever tricks in model design must carefully balance expressive power with the risk of obscuring these core principles, echoing the interconnectedness of a complex system.

Beyond Convergence: The Shape of Matter’s Understanding

The observation of converging latent spaces across diverse scientific foundation models is not, perhaps, surprising. Elegance, after all, favors simplicity. Yet, the true challenge lies not in demonstrating that convergence occurs, but in deciphering the language of this shared representation. What principles, currently obscured within the dimensionality, dictate the organization of this ‘universal’ matter space? The next phase necessitates probing the meaning embedded within these latent dimensions-identifying which physical constraints, chemical rules, or fundamental properties are consistently encoded, and which remain stubbornly resistant to abstraction.

Current models, while impressive in their predictive power, are largely empirical cartographers. They chart correlations, but offer little insight into the underlying causal structure. A truly scalable scientific framework demands moving beyond correlation to establish a hierarchical understanding-a system where representations at different levels of abstraction are linked by well-defined physical principles. This requires not simply scaling up model size, but refining the architecture to explicitly incorporate known physics, thereby fostering genuine generalization.

The ecosystem of scientific machine learning will inevitably expand. New modalities-spectroscopy, microscopy, in-situ experimentation-will join the existing landscape. The critical question is whether these additions will contribute to a coherent, integrated understanding, or simply exacerbate the fragmentation. A universal representation is not a destination, but a continually refined map-its value determined not by its completeness, but by its ability to guide further exploration.

Original article: https://arxiv.org/pdf/2512.03750.pdf

Contact the author: https://www.linkedin.com/in/avetisyan/

See also:

2025-12-04 09:04