Beyond Scale: Teaching Machines to Understand Physics

by

Author: Denis Avetisyan

New research reveals that simply increasing the size of AI models isn’t enough to instill an understanding of basic physical principles like Newtonian mechanics.

Planetary motion prediction, framed as next-token prediction, reveals a compelling relationship between transformer context length and the learned world model: extended contexts favor global, geometry-based Keplerian models, while limited contexts yield local, force-based Newtonian approximations, demonstrating how inductive biases address inherent failure modes in learning these physical systems-a trade-off elegantly captured by [latex] \text{context length} [/latex].
Planetary motion prediction, framed as next-token prediction, reveals a compelling relationship between transformer context length and the learned world model: extended contexts favor global, geometry-based Keplerian models, while limited contexts yield local, force-based Newtonian approximations, demonstrating how inductive biases address inherent failure modes in learning these physical systems-a trade-off elegantly captured by [latex] \text{context length} [/latex].

Learned world models in transformer networks require explicit inductive biases, particularly temporal locality, to accurately represent physical systems.

While large language models excel at prediction, recovering the underlying physical laws governing dynamic systems remains a significant challenge. This is addressed in ‘From Kepler to Newton: Inductive Biases Guide Learned World Models in Transformers’, which investigates how minimal architectural constraints can enable transformers to learn coherent world models of physics. The authors demonstrate that imposing inductive biases – spatial smoothness, stability, and crucially, temporal locality – transforms these models from mere curve-fitters into systems capable of discovering Newtonian force representations. Can strategically designed inductive biases unlock the potential for automated scientific discovery within foundation models, moving beyond correlation to true physical understanding?

The Illusion of Understanding: Beyond Pattern Matching

Foundation models, such as those leveraging transformer architectures, demonstrate remarkable proficiency in identifying patterns within data; however, this skill often plateaus when confronted with tasks demanding genuine understanding. These models excel at statistical correlations but frequently stumble when reasoning about the underlying dynamics governing the observed phenomena. While adept at predicting what comes next based on past occurrences, they struggle to extrapolate beyond the training data or to respond effectively to novel situations requiring causal inference. This limitation stems from a reliance on surface-level associations rather than a deep, mechanistic grasp of how the world functions, hindering their ability to generalize, plan, or exhibit true intelligence – capabilities that necessitate a robust comprehension of cause and effect, not merely pattern recognition.

The development of genuine intelligence necessitates the construction of internal representations, often termed ‘world models’, which allow an agent to simulate and understand its surroundings. These models aren’t merely passive recordings of sensory input; instead, they actively capture the underlying structure of an environment and, crucially, how that environment evolves over time. By building an internal simulation, an intelligent system can anticipate the consequences of actions, plan effectively, and reason about scenarios it has never directly experienced. This capacity for predictive understanding, rooted in a robust internal model, moves beyond simple pattern recognition and enables flexible, adaptive behavior – a hallmark of true intelligence. The efficacy of these world models relies on the ability to abstract key features and relationships, effectively creating a compressed, usable representation of the external world.

Effective intelligence necessitates more than merely forecasting the immediate successor in a sequence; truly insightful systems require the ability to model how and why events unfold. Instead of simply predicting the next token, advanced models must encapsulate underlying causal relationships, allowing them to reason about interventions and counterfactuals. This means constructing an internal representation of the environment that facilitates anticipatory understanding – a capacity to foresee the consequences of actions and to plan strategically. Such models move beyond correlation to causation, enabling a system to not only respond to stimuli, but to proactively shape its surroundings and achieve complex goals by understanding the dynamic interplay of forces at work.

The construction of effective world models fundamentally relies on the principles of spatial smoothness and temporal locality. This means that a system benefits from assuming nearby locations share similar properties and that future states are predictably linked to the present. By prioritizing these assumptions, models can efficiently learn and generalize from limited data, extrapolating understanding beyond directly observed instances. Essentially, rather than treating each moment or location as entirely novel, the system leverages the expectation of continuity – a concept deeply ingrained in how living organisms perceive and interact with their environments. This prioritization allows for predictive processing, enabling the model to anticipate changes and plan actions based on a simplified, yet remarkably accurate, representation of reality – a crucial step towards genuine intelligence.

Analysis of the transformer model's embeddings reveals that while training induces spatial structure, the learned representations exhibit poor locality and limited linear decodability to the true spatial map, suggesting a need for improved representation learning.
Analysis of the transformer model’s embeddings reveals that while training induces spatial structure, the learned representations exhibit poor locality and limited linear decodability to the true spatial map, suggesting a need for improved representation learning.

Learning the Dance: Modeling Planetary Motion with Transformers

The Vafa et al. study examined the feasibility of utilizing a transformer architecture to model the dynamical behavior of planetary motion using only observational data as input. Researchers sought to determine if a transformer could learn the underlying physical laws governing orbital trajectories without explicit programming of those laws. The methodology involved training the model on sequences of planetary positions and velocities, with the goal of enabling it to predict future states based on past observations. This approach represented a departure from traditional physics-based simulations, focusing instead on a data-driven learning paradigm to represent and extrapolate planetary dynamics.

The Vafa et al. study utilized next token prediction as the core training methodology, necessitating the conversion of continuous orbital data – representing position and velocity over time – into a discrete token sequence. This tokenization process involved quantizing the continuous values of orbital parameters into a finite set of discrete symbols, effectively representing each data point as a “token”. The transformer model was then trained to predict the subsequent token in the sequence, given a preceding context of tokens, thereby learning the underlying dynamics of the planetary motion from the discrete representation of observational data. This approach allows the transformer, typically used with text data, to process and learn from time-series data representing physical system states.

The study utilized a transformer model with a parameter count comparable to GPT-2, consisting of 12 layers, 768 hidden units, and 12 attention heads. Training involved predicting subsequent orbital states based on a fixed-length sequence of prior observations. The input data, representing planetary positions and velocities, were tokenized and fed into the transformer. Model performance was optimized via cross-entropy loss, calculated between the predicted probability distribution of the next state and the actual observed state. The minimization of this loss function, achieved through standard backpropagation and an Adam optimizer, facilitated the learning of the underlying dynamical system from the observational data.

The study by Vafa et al. offered a rigorous evaluation of transformer architectures beyond traditional natural language processing tasks, specifically assessing their ability to model the dynamics of a physical system-planetary motion. By framing the problem as a next-token prediction task with discrete orbital data, the research bypassed the need for system-specific knowledge or hand-engineered features. Successful prediction of future states, as measured by cross-entropy loss, demonstrated the transformer’s capacity to learn an internal representation of the underlying physical laws governing the observed data, thereby establishing a potential pathway for applying these models to other complex dynamical systems without explicit physical modeling.

Performance on the Kepler problem, framed as a next-token prediction task, improves with smaller vocabulary sizes when training data is limited, as demonstrated by reduced cross-entropy loss, effective mean squared error ([latex]MSE[/latex]), and generation distance error.
Performance on the Kepler problem, framed as a next-token prediction task, improves with smaller vocabulary sizes when training data is limited, as demonstrated by reduced cross-entropy loss, effective mean squared error ([latex]MSE[/latex]), and generation distance error.

What’s Under the Hood? Probing the Transformer’s ‘Understanding’

Linear probing is employed as a technique to interpret the internal representations learned by transformer models by mapping directions within the model’s state space to specific, identifiable concepts. This process involves training a linear decoder to predict a target variable from the transformer’s hidden states; the resulting R² score quantifies the degree to which the hidden states encode information relevant to that variable. High R² scores indicate that a particular concept is readily decodable from the transformer’s internal representation, allowing researchers to ascertain what features or relationships the model has implicitly learned during training. By systematically probing for various concepts, the organization and content of the transformer’s learned representations can be analyzed and understood.

Analysis of the directions within a transformer’s internal state space allows researchers to differentiate between learned predictive patterns and mechanistic understanding of a physical system. A model exhibiting purely predictive behavior, analogous to Keplerian mechanics, identifies correlations to forecast future states without explicitly representing underlying causal relationships. Conversely, a mechanistic understanding, similar to Newtonian mechanics, implies the model has learned to represent the governing physical laws, enabling more robust generalization and counterfactual reasoning. This differentiation is achieved by evaluating the learned representations against known physical principles and assessing the model’s ability to extrapolate beyond the observed data, indicating whether it has captured the system’s causal structure.

The transformer’s context length, defining the number of prior states incorporated into its calculations, significantly impacts its ability to model system evolution and temporal dependencies. Empirical results demonstrate this influence: a linear probing R² score of 0.999 was achieved when the transformer utilized a context length of 2. This high score indicates the learned model closely aligns with Newtonian mechanics, suggesting the transformer, with limited contextual information, effectively captures the relationships defined by Newtonian physics. The R² score quantifies the proportion of variance in the target variable explained by the model, with values approaching 1 indicating a strong predictive capability.

Analysis using linear probing revealed a correlation between transformer context length and the learned model of system dynamics; a context length of 100 resulted in an R² score of 0.998, indicative of a Keplerian model. This contrasts with a context length of 2, which yielded an R² score of 0.999 and a learned Newtonian model. These findings demonstrate that the transformer’s ability to model physical laws is strongly influenced by the length of the input context, suggesting that shorter contexts promote learning predictive, but less mechanistically accurate, models, while longer contexts allow for the development of models closer to established physical laws.

The length of the context window dictates whether a transformer learns a Newtonian model of physics, internally computing gravitational forces, or a Keplerian model, internally computing orbital parameters like semi-major/minor axis lengths [latex]a/b[/latex] and the Laplace-Runge-Lenz vector [latex]\vec{A}[/latex], demonstrating a phase transition and improved predictive accuracy with longer contexts.
The length of the context window dictates whether a transformer learns a Newtonian model of physics, internally computing gravitational forces, or a Keplerian model, internally computing orbital parameters like semi-major/minor axis lengths [latex]a/b[/latex] and the Laplace-Runge-Lenz vector [latex]\vec{A}[/latex], demonstrating a phase transition and improved predictive accuracy with longer contexts.

The pursuit of ever-larger models, as demonstrated in this exploration of Newtonian mechanics and transformers, feels like chasing a ghost. The article highlights the necessity of inductive biases – a structured approach to learning – yet the industry often assumes scale will solve all problems. It’s a comforting delusion. As Paul Erdős famously said, ‘A mathematician who doesn’t believe in God is like a fish who doesn’t believe in water.’ Similarly, these models need more than just data; they need fundamental principles, like temporal locality, baked in. Otherwise, it’s simply a complex system destined to crash predictably, a monument to wasted compute cycles. It isn’t about building intelligence; it’s about leaving increasingly elaborate notes for future digital archaeologists.

The Road Ahead

The insistence on scaling laws as a solution to all problems feels… familiar. This work suggests that merely throwing parameters at a Newtonian universe doesn’t yield understanding; it yields a very expensive, brittle approximation. The claim isn’t that transformers can’t model physics, but that they require more than just data; they require being told how to look. Any system exhibiting self-correction simply hasn’t encountered its edge case yet. The elegance of a theoretically perfect model always seems to evaporate upon contact with production data.

Future efforts will undoubtedly explore increasingly sophisticated inductive biases. Temporal locality is a start, but it’s likely a local minimum in a vast, complex landscape. The real challenge isn’t achieving a benchmark score; it’s building systems that degrade gracefully, and whose failures are… informative. If a bug is reproducible, that’s not a failure of the model, it’s a stable system. Documentation, of course, remains a collective self-delusion.

The pursuit of ‘general’ world models seems perpetually out of reach. Perhaps the goal shouldn’t be a single, all-encompassing framework, but a collection of specialized models, each embodying a specific set of inductive constraints. Such an approach acknowledges the inherent limitations of any single architecture and the inevitability of technical debt. The next ‘revolution’ will invariably become tomorrow’s legacy system.

Original article: https://arxiv.org/pdf/2602.06923.pdf

Contact the author: https://www.linkedin.com/in/avetisyan/

See also:

2026-02-09 10:28