Unlocking Neural Network Logic: A New Path to Interpretability

by

Author: Denis Avetisyan

Researchers have developed a framework to automatically extract symbolic expressions from deep learning models, bridging the gap between black-box prediction and human-understandable reasoning.

SymTorch facilitates symbolic distillation by wrapping neural network components, collecting input-output data during forward passes, and employing PySR to generate increasingly complex symbolic regressions-allowing for the replacement of these components with optimized equations and the creation of hybrid neural-symbolic models that balance expressiveness and computational efficiency, effectively distilling the function of a neural network into a mathematically provable form [latex] f(x) = \sum_{i=0}^{n} a_i x^i [/latex].
SymTorch facilitates symbolic distillation by wrapping neural network components, collecting input-output data during forward passes, and employing PySR to generate increasingly complex symbolic regressions-allowing for the replacement of these components with optimized equations and the creation of hybrid neural-symbolic models that balance expressiveness and computational efficiency, effectively distilling the function of a neural network into a mathematically provable form [latex] f(x) = \sum_{i=0}^{n} a_i x^i [/latex].

SymTorch enables symbolic regression of neural network components, facilitating hybrid neural-symbolic models for applications ranging from scientific discovery to large language model analysis.

Despite the promise of interpretable models and scientific discovery, extracting human-readable equations from trained neural networks remains a significant engineering challenge. This work introduces SymTorch: A Framework for Symbolic Distillation of Deep Neural Networks, a library designed to automate the process of replacing neural network components with symbolic regressions. By handling complexities like data transfer and model serialization, SymTorch enables the creation of hybrid neural-symbolic models across architectures including Graph Neural Networks and Transformers, and even demonstrates an 8.3% throughput improvement when applied to Large Language Model inference. Could this approach unlock a new era of transparent and efficient AI, bridging the gap between deep learning’s predictive power and human understanding?

Unveiling the Black Box: The Imperative of Neural Network Transparency

Despite achieving state-of-the-art results in areas like image recognition and natural language processing, deep neural networks often operate as inscrutable ‘black boxes’. This lack of transparency poses a significant challenge to both trust and further development. While the networks can accurately perform tasks, understanding why they arrive at specific conclusions remains difficult. This isn’t simply a matter of curiosity; the inability to interpret a network’s reasoning hinders debugging, limits the potential for refinement, and raises concerns about reliability – particularly in critical applications like healthcare or autonomous systems. The complex interplay of millions – even billions – of parameters within these networks creates a high-dimensional decision space that defies intuitive human understanding, demanding new approaches to unlock their inner workings and build genuinely explainable artificial intelligence.

The opacity of deep neural networks presents a significant challenge to understanding how they arrive at decisions. Conventional analytical techniques, designed for simpler, linear models, often fail when confronted with the non-linear complexities and vast parameter spaces characteristic of deep learning. Methods like sensitivity analysis, while capable of identifying influential inputs, frequently produce interpretations that are either too granular to be useful – focusing on individual weights – or lack the holistic view necessary to grasp the network’s overall reasoning. Attempts to distill network behavior into rule-based systems or decision trees often result in approximations that sacrifice accuracy, while visualization techniques, though helpful, struggle to convey the full intricacy of interactions within layers of artificial neurons. This limitation not only hinders efforts to refine network performance but also raises concerns about trust and accountability, particularly in high-stakes applications where explainability is paramount.

SymTorch: Distilling Complexity into Symbolic Representation

SymTorch is a framework designed to create symbolic representations of neural network functionality using symbolic regression. This approach treats a trained neural network as a black box and aims to identify mathematical expressions that accurately model its input-output behavior. Rather than directly interpreting the weights and biases of the network, SymTorch uses the network’s activations as data points for a symbolic regression algorithm, effectively learning a closed-form approximation of the neural network’s function. The resulting symbolic expression can then potentially replace the original neural network layer or subnetwork, offering benefits in terms of computational efficiency and interpretability. This differs from traditional network analysis methods that focus on understanding the network’s internal parameters.

SymTorch achieves efficiency by utilizing forward hooks within the neural network to intercept and record activations during a single forward pass, effectively capturing the network’s behavior without requiring any retraining or fine-tuning. These activations are then cached and used as training data for the symbolic regression process. Forward hooks allow for the extraction of intermediate layer outputs without modifying the underlying network architecture or weights, providing a non-destructive method for behavioral analysis and subsequent function approximation. This approach avoids the computational cost and data requirements associated with traditional retraining methods, enabling a faster and more resource-efficient distillation process.

SymTorch utilizes Principal Component Analysis (PCA) as a dimensionality reduction technique to facilitate symbolic regression of neural network layers. By reducing the input space to the symbolic regression algorithm, PCA simplifies the search for equivalent mathematical expressions and improves computational efficiency. This approach specifically targets Multi-Layer Perceptron (MLP) layers within Large Language Models (LLMs), replacing them with symbolic surrogates derived through PCA-assisted symbolic regression. Benchmarking demonstrates an 8.3% speedup in LLM inference performance when employing this layer replacement strategy.

This framework reduces transformer model inference compute by substituting multilayer perceptrons with symbolic models that leverage principal component analysis for both dimensionality reduction and reconstruction of input and output activations.
This framework reduces transformer model inference compute by substituting multilayer perceptrons with symbolic models that leverage principal component analysis for both dimensionality reduction and reconstruction of input and output activations.

From Approximation to Verification: Extending Symbolic Distillation to Physics and Graphs

Physics-Informed Neural Networks (PINNs) leverage SymTorch to facilitate the extraction of symbolic representations of learned physical models. This process allows for the recovered equations to be explicitly verified against established physical laws, such as the 1D Heat Equation: [latex]\frac{\partial u}{\partial t} = \alpha \frac{\partial^2 u}{\partial x^2}[/latex]. By symbolically distilling the knowledge embedded within the trained neural network, researchers can obtain interpretable and verifiable expressions representing the underlying physical relationships the PINN has learned during training, offering a pathway beyond purely numerical solutions.

Symbolic distillation techniques, initially applied to Physics-Informed Neural Networks (PINNs), extend to Graph Neural Networks (GNNs) and Large Language Models (LLMs). This process involves extracting symbolic representations from trained models, allowing for the identification of underlying rules and relationships that govern their predictions. Successfully applying symbolic distillation to GNNs and LLMs facilitates increased interpretability, moving beyond black-box predictions to reveal the logic driving decision-making. The resulting symbolic models can be verified and analyzed to ensure consistency and potentially uncover novel insights into the learned representations within these complex architectures.

Symbolic distillation of Physics-Informed Neural Networks (PINNs) is computationally efficient, completing in under 3 minutes when executed on an Apple M4 Max System on Chip. This process successfully recovers known physical laws embedded within the trained network. Implementation involves replacing the standard Multilayer Perceptron (MLP) layers with symbolic surrogates, which results in a measured increase in perplexity of 3.14, rising from a baseline perplexity of 10.62. This perplexity increase represents the trade-off between model complexity and interpretability achieved through symbolic distillation.

The physics-informed neural network (PINN) accurately predicts the true solution of the 1D heat equation [latex]\alpha = 0.2[/latex] , surpassing the performance of a standard neural network.
The physics-informed neural network (PINN) accurately predicts the true solution of the 1D heat equation [latex]\alpha = 0.2[/latex] , surpassing the performance of a standard neural network.

Optimizing for the Inevitable Trade-offs: Balancing Accuracy, Complexity, and Robustness

Symbolic distillation offers a powerful approach to model optimization by framing the process as a multi-objective problem, enabling the identification of solutions that deftly balance competing priorities. Rather than striving for a single ‘best’ model, this technique allows exploration of the Pareto Front – the set of solutions where improving one objective necessarily degrades another. This is particularly valuable when translating complex neural networks into symbolic representations, as it facilitates trade-offs between model accuracy, computational complexity, and human interpretability. A symbolic model can be refined to prioritize simplicity for easier understanding, even if it means a slight reduction in predictive performance, or conversely, it can be optimized for maximum accuracy at the cost of increased complexity. This nuanced control is achieved by simultaneously optimizing for multiple loss functions, guiding the distillation process towards solutions that best meet specific application requirements and constraints.

Refinement of distilled symbolic models benefits significantly from regularization techniques designed to improve their ability to generalize beyond the training data. Methods like L1 regularization encourage sparsity in the symbolic expressions, effectively simplifying the model and reducing overfitting – a common issue where a model learns the training data too well, hindering performance on unseen examples. Simultaneously, employing KL Divergence as a loss function guides the symbolic model to closely mimic the probabilistic output distribution of the original neural network, rather than just its point predictions. This nuanced approach ensures the distilled model doesn’t just replicate what the neural network predicts, but how it predicts it – fostering a more robust and reliable symbolic representation capable of adapting to variations in input data and maintaining predictive accuracy.

A critical component in evaluating the effectiveness of symbolic distillation lies in quantifying how closely the resulting symbolic representation mirrors the behavior of the original neural network. Mean Squared Error (MSE) Loss serves as this crucial metric, providing a direct, numerical assessment of fidelity. By calculating the average squared difference between the outputs of the neural network and its symbolic counterpart across a representative dataset, researchers gain a precise understanding of the approximation quality. A lower MSE value indicates a stronger correspondence, suggesting the symbolic model effectively captures the nuances of the original network’s decision-making process. This quantitative measure isn’t simply about achieving low error; it allows for systematic comparison of different distillation strategies and regularization techniques, ultimately guiding the development of symbolic representations that are both accurate and interpretable, without sacrificing the predictive power of the initial neural network.

SLIME approximates the behavior of a complex, non-linear model by symbolically fitting a simplified model to sampled data around a point of interest [latex]\mathbf{x}^{\\*}[/latex].
SLIME approximates the behavior of a complex, non-linear model by symbolically fitting a simplified model to sampled data around a point of interest [latex]\mathbf{x}^{\\*}[/latex].

The pursuit of demonstrable correctness, central to SymTorch’s design, echoes Ada Lovelace’s foresight. She stated, “The Analytical Engine has no pretensions whatever to originate anything.” This framework doesn’t seek to create intelligence, but to meticulously derive the underlying symbolic representations already implicitly contained within a neural network’s weights. SymTorch, by automating symbolic regression, aims to reveal these deterministic rules-to translate the ‘how’ of a network’s prediction into a verifiable ‘why’. The resulting hybrid neural-symbolic models aren’t merely functional; they are demonstrably correct, allowing for reliable analysis, particularly crucial in areas like LLM analysis where reproducibility is paramount.

What Lies Ahead?

The automation of symbolic regression, as demonstrated by SymTorch, represents a necessary, if incremental, step toward genuine understanding of the functions approximated by deep neural networks. The framework’s utility extends beyond mere interpretability; the creation of hybrid neural-symbolic models offers a potential pathway toward solutions possessing both the expressive power of connectionism and the logical rigor of symbolic AI. However, the current emphasis on discovering symbolic equivalents, while valuable, skirts a more fundamental question: are the discovered symbols actually representative of the underlying phenomena, or merely a convenient, low-dimensional projection of a far more complex reality?

A critical limitation lies in the reliance on existing symbolic regression techniques, which are demonstrably susceptible to overfitting and spurious correlations. Future work must focus on incorporating prior knowledge and constraints – not merely through physics-informed layers, but via a deeper integration of formal methods and verification techniques. Optimization without analysis remains a dangerous seduction; a ‘working’ symbolic expression, absent a provable connection to the originating process, is little more than a sophisticated curve fit.

The application of this framework to large language models, while promising, presents unique challenges. The sheer scale of these networks demands a reassessment of current symbolic regression algorithms, and a consideration of methods capable of extracting meaningful abstractions from high-dimensional, entangled representations. Ultimately, the goal should not be to explain LLMs-a task bordering on the impossible-but to distill their capabilities into more transparent, verifiable, and controllable systems.

Original article: https://arxiv.org/pdf/2602.21307.pdf

Contact the author: https://www.linkedin.com/in/avetisyan/

See also:

2026-02-26 11:26