Decoding Dynamics: When Neural Networks Meet Equation Discovery

by

Author: Denis Avetisyan

New research shows how continuous-time neural networks can enhance our ability to model and understand complex systems from limited data.

A predictive model, initially trained on only one second of cart-pole dynamics and two up-shift datasets from a bio-model, successfully extrapolated predictions five times beyond its training duration-and even generalized to entirely unseen down-shift events-demonstrating an ability to discern and replicate underlying system dynamics despite limited exposure and challenging extrapolation tasks.
A predictive model, initially trained on only one second of cart-pole dynamics and two up-shift datasets from a bio-model, successfully extrapolated predictions five times beyond its training duration-and even generalized to entirely unseen down-shift events-demonstrating an ability to discern and replicate underlying system dynamics despite limited exposure and challenging extrapolation tasks.

This study investigates the combination of Neural Ordinary Differential Equations and Symbolic Regression to improve the generalization and accuracy of governing equation recovery for dynamical systems.

Accurately modelling complex dynamical systems often requires substantial data, yet recovering governing equations from limited observations remains a significant challenge. This work, ‘An Empirical Investigation of Neural ODEs and Symbolic Regression for Dynamical Systems’, explores a novel approach combining Neural Ordinary Differential Equations (NODEs) and Symbolic Regression (SR) to address this issue. We demonstrate that NODEs effectively extrapolate beyond training data, and when used to augment limited datasets, SR can successfully recover underlying equations with improved accuracy. Could this synergistic combination unlock new avenues for scientific discovery by enabling the inference of physical laws from sparse and noisy observations?

Decoding Dynamics: The Challenge of Representing Reality

The accurate representation of complex dynamical systems is paramount to progress across a remarkably diverse range of scientific fields. From predicting weather patterns and modeling ecological interactions to understanding the intricacies of the human cardiovascular system and designing efficient control systems, these models serve as the foundation for both fundamental research and practical applications. These systems, characterized by states that evolve over time, present a unique challenge because even slight inaccuracies in the representation of underlying processes can lead to drastically different outcomes. Consequently, fields like physics, biology, engineering, and economics all rely heavily on the ability to create robust and reliable dynamical models, not just to explain past behavior, but to forecast future states and inform critical decision-making processes. The pursuit of improved modeling techniques, therefore, remains a central endeavor in contemporary scientific inquiry.

Conventional approaches to modeling dynamical systems frequently encounter limitations when attempting to capture fundamental relationships and predict future behavior. These methods often rely on pre-defined equations or statistical correlations, proving inadequate when faced with the inherent complexity and non-linearity present in many real-world phenomena. A primary difficulty arises from the dependence on observed data; extrapolation beyond the range of that data can lead to significant inaccuracies, as the established models fail to account for previously unseen conditions. Furthermore, discerning the underlying principles governing a system – the core mechanisms driving its evolution – remains a persistent challenge, often requiring extensive prior knowledge or simplifying assumptions that compromise the model’s fidelity. This struggle highlights the need for innovative techniques capable of both uncovering hidden dynamics and generalizing beyond the limitations of current observational constraints.

Addressing the limitations of conventional techniques demands a shift towards novel methodologies in both system identification and predictive modeling. Researchers are increasingly exploring techniques like sparse regression, symbolic regression, and machine learning algorithms – including neural networks – to uncover the underlying principles governing complex systems. These data-driven approaches aim to bypass the need for pre-defined equations, instead learning directly from observed data to construct models capable of accurately forecasting future behavior. Furthermore, hybrid methods combining physics-based models with data-driven techniques are gaining traction, offering the potential to leverage existing knowledge while simultaneously adapting to the nuances revealed by empirical observations. This ongoing pursuit of innovative modeling strategies is critical for advancing understanding and prediction across diverse fields, from climate science and epidemiology to financial markets and engineering systems.

Determining the fundamental equations that govern a dynamic system solely from observational data presents a persistent and substantial challenge in scientific modeling. Traditional system identification techniques frequently rely on pre-defined model structures, limiting their capacity to uncover genuinely novel or complex behaviors. While data-driven approaches, such as machine learning, excel at prediction, they often lack the interpretability needed to reveal the underlying physical principles. This difficulty arises because observed data is invariably finite, noisy, and may not fully capture the system’s complete state space, leading to ambiguities in reconstructing the governing [latex] \text{equations} [/latex]. Consequently, researchers are actively exploring methods – including sparse regression, symbolic regression, and information-theoretic approaches – that can effectively sift through vast possibilities to identify the simplest, most accurate mathematical representation consistent with the observed dynamics, a process akin to reverse-engineering the laws of nature from limited evidence.

NODE and SR: A Symbiotic Revelation

Neural Ordinary Differential Equations (NODEs) represent a class of neural networks where the hidden layer dynamics are defined by an ordinary differential equation (ODE). This formulation allows NODEs to model continuous-time systems, differing from traditional discrete-layer networks. Instead of processing data in distinct steps, NODEs define a continuous trajectory through state space, determined by the solution to the ODE. The state of the system evolves continuously with respect to a continuous “time” variable, and the network learns the parameters defining the derivative function [latex]\frac{dy}{dt}[/latex] that governs this evolution. This continuous representation enables precise interpolation and extrapolation beyond observed data points and is particularly effective for modeling systems where the underlying dynamics are inherently continuous, such as physical simulations and time-series forecasting.

Neural Ordinary Differential Equations (NODEs), when implemented with the JAX-based Diffrax library, demonstrate significant capabilities in interpolation and capturing system dynamics within the range of available data. Diffrax utilizes automatic differentiation to efficiently solve the [latex] \frac{dy}{dt} = f(t, y) [/latex] ordinary differential equation defining the NODE, allowing for continuous-time modeling. This approach enables accurate prediction of system states at arbitrary time points, even those not included in the original training data, effectively interpolating the system’s behavior. However, it is crucial to note that NODEs, while proficient within the observed data boundaries, do not extrapolate well beyond these limits, and their accuracy is contingent on the quality and density of the initial data provided to the Diffrax solver.

Symbolic Regression (SR) is a technique for model discovery that aims to identify explicit mathematical expressions representing the relationships within a dataset. Implemented through the PySR library, SR operates by searching for equations-composed of mathematical operators and functions-that best fit the observed data. Unlike many machine learning methods that yield implicit or black-box models, SR provides interpretable equations, such as [latex]y = ax^2 + bx + c[/latex], directly revealing the underlying governing laws. This is achieved through evolutionary algorithms and optimization techniques, evaluating the performance of candidate equations against the data and iteratively refining them to minimize error. The resulting equations can then be used for prediction, analysis, and a deeper understanding of the system being modeled.

The integration of Neural Ordinary Differential Equations (NODEs) and Symbolic Regression (SR) provides a synergistic approach to system identification and prediction. NODEs, particularly when implemented with libraries like Diffrax, demonstrate proficiency in interpolating data and modeling continuous dynamics within the observed data range. Conversely, SR, facilitated by tools such as PySR, excels at discovering the underlying mathematical equations that govern the system’s behavior, potentially allowing for extrapolation beyond the training data. By combining these capabilities, the approach aims to achieve both accurate representation of observed data – leveraging NODEs’ interpolation skills – and the discovery of generalizable, explicit mathematical models – enabled by SR – leading to improved predictive performance and a more complete understanding of the system’s dynamics.

Model B demonstrates an ability to extrapolate beyond its training region-highlighted in red-by learning the underlying cart-pole dynamics, as evidenced by low-error zones and consistent angular speed/acceleration magnitudes in phase space.
Model B demonstrates an ability to extrapolate beyond its training region-highlighted in red-by learning the underlying cart-pole dynamics, as evidenced by low-error zones and consistent angular speed/acceleration magnitudes in phase space.

Testing the Limits: Robustness and Extrapolation in Action

Model resilience to noisy data was evaluated by introducing random errors during the training process. Specifically, random noise was added to the training dataset, simulating real-world inaccuracies and sensor limitations. Performance was then assessed using the same metrics employed with clean data. Results demonstrated that the hybrid model maintained a high degree of accuracy even with significant levels of noise, indicating robustness and the ability to generalize from imperfect input. This suggests the model is less susceptible to minor data corruption and can reliably produce predictions despite the presence of errors in the observed data.

The Neural Ordinary Differential Equation (NODE) architecture, by design, learns the underlying continuous dynamics of a system, unlike discrete-time recurrent models. This capacity, when combined with State Reconstruction (SR), facilitates accurate prediction beyond the bounds of the original training data. SR effectively maps observed data points onto the learned continuous state space, allowing the NODE to integrate forward from novel initial conditions and produce valid trajectories even when those trajectories extend outside the region covered by the training set. This extrapolation capability is crucial for real-world applications where complete datasets covering all possible scenarios are unavailable, and the system must reliably predict behavior in unseen states.

Quantitative analysis using the Mean Squared Error (MSE) metric demonstrated the hybrid model’s improved predictive capability. Specifically, in down-shift prediction for the Bio-model, the hybrid model achieved an MSE of less than 5%. This represents a statistically significant reduction in error compared to models trained exclusively on up-shift data, which exhibited substantially higher MSE values under the same testing conditions. The lower MSE indicates a more accurate alignment between predicted and actual values, confirming the benefits of incorporating both up-shift and down-shift data, as well as the NODE and SR architectures, for enhanced predictive performance.

Phase Space Plots generated from the Cart-Pole System provided a visual assessment of the hybrid model’s predictive capabilities. Analysis of these plots demonstrated that the model maintained a Mean Squared Error (MSE) comparable to that observed within the training dataset, even when evaluating system states outside the original training region. This consistency in MSE across both trained and untrained regions serves as qualitative evidence of effective generalization, indicating the model’s ability to accurately predict system behavior beyond the scope of its initial training data. The plots visually confirmed the model’s stable and accurate predictions in extrapolated states.

Amplifying Insight: The Impact of Data Augmentation

The study reveals a compelling synergy between neural ordinary differential equations (NODEs) and sparse regression (SR) through a data augmentation strategy. A trained NODE, capable of learning the underlying dynamics of a system, generates synthetic data that effectively expands the training dataset used for SR. This augmented dataset significantly improves SR’s ability to identify the governing equations of complex systems; researchers found they could recover two out of three equations utilizing just ten percent of the data initially employed for NODE training. By leveraging the NODE’s predictive capabilities to create additional, informed training samples, the method overcomes limitations imposed by scarce data, ultimately boosting the overall performance and efficiency of equation discovery.

A significant advancement in system identification lies in the ability to amplify limited datasets through data augmentation techniques. Recent research demonstrates that by leveraging synthetically generated data, a model can achieve substantial performance gains even with drastically reduced training requirements. Specifically, utilizing data produced by a trained Neural Operator, researchers were able to accurately recover two out of three governing equations for a complex system while training the Neural Operator on only 10% of the original dataset. This efficient use of data not only reduces computational cost but also opens avenues for modeling systems where data acquisition is expensive or time-consuming, suggesting a pathway towards more accessible and scalable scientific discovery.

The study reveals a compelling feedback loop wherein Neural Operator Differential Equations (NODEs) transcend their traditional role as passive system models to become active participants in data generation. Rather than solely relying on pre-existing datasets, NODEs can synthesize new, realistic data points that augment the original training set. This self-improvement cycle allows for more robust and efficient system identification, demonstrably recovering complex governing equations with significantly reduced data requirements-in some cases, achieving comparable performance with only 10% of the original data. The capacity of NODEs to both model and actively contribute to data acquisition represents a paradigm shift, hinting at the potential for autonomous scientific discovery and accelerated modeling of complex phenomena where data is scarce or expensive to obtain.

The synergy between Neural Ordinary Differential Equations (NODEs) and Sparse Regression (SR) establishes a robust framework for discerning intricate governing equations from limited data. Investigations reveal that models trained with NODE-generated synthetic data exhibit remarkably consistent performance, maintaining a stable Mean Squared Error (MSE) over an 8-hour prediction horizon-even when trained on data sampled at different frequencies. This resilience suggests a minimal reliance on extensive datasets for accurate long-term forecasting, as the NODE effectively amplifies the information content of the initial data through informed augmentation. The resulting combination not only recovers complex relationships but also demonstrates an ability to generalize predictions irrespective of the granularity of the input data, pointing towards a potentially transformative approach to system modeling and prediction.

The pursuit of understanding dynamical systems, as outlined in the research, isn’t about flawlessly predicting behavior, but about probing the limits of a model’s comprehension. One encounters discrepancies – data that doesn’t quite fit – and the instinct isn’t to dismiss them, but to dissect them. Grace Hopper famously said, “It’s easier to ask forgiveness than it is to get permission.” This sentiment echoes the methodology of pushing boundaries in the study. The researchers don’t simply accept the limitations of data extrapolation with Neural ODEs; instead, they augment data, essentially ‘asking forgiveness’ for introducing synthetic examples, to enhance Symbolic Regression’s ability to uncover the underlying governing equations. This deliberate challenge to established norms, this willingness to explore ‘what if’ scenarios, is the essence of reverse-engineering reality and achieving true generalization.

Beyond the Equation

The apparent success of Neural ODEs in extrapolating dynamical systems-and their utility as data amplifiers for symbolic regression-should not be mistaken for a resolution. It merely shifts the locus of inquiry. The true challenge isn’t recovering an equation, but understanding why any particular equation-whether derived from data or conceived a priori-holds, or fails to hold, outside the training manifold. This work implicitly acknowledges the limitations of purely data-driven approaches; the need for augmentation suggests an inherent distrust of the observed data itself. The system isn’t revealing its rules, it’s being prodded into conformity.

Future efforts must address the question of inherent model bias. Neural ODEs, despite their flexibility, still embody specific assumptions about continuity and differentiability. Symbolic regression, similarly, favors certain functional forms. The pursuit of ‘generalization’ risks becoming a sophisticated form of curve-fitting, masking fundamental discrepancies between model and reality. A more fruitful path lies in actively seeking those discrepancies-designing experiments to deliberately break the model and expose its underlying limitations.

Ultimately, the goal isn’t to build a perfect predictor, but to construct a robust falsifier. A system that reliably identifies its own ignorance is, paradoxically, more valuable than one that confidently offers incorrect answers. The observed improvements in equation recovery are interesting, but the real signal lies in understanding the nature of the error – and actively cultivating it.

Original article: https://arxiv.org/pdf/2601.20637.pdf

Contact the author: https://www.linkedin.com/in/avetisyan/

See also:

2026-01-29 18:34