Can AI Truly Prove It?

Author: Denis Avetisyan

A new benchmark of ten unsolved mathematical problems challenges artificial intelligence to move beyond pattern recognition and demonstrate genuine proof-finding capabilities.

Researchers introduce a novel evaluation suite designed to assess frontier AI’s problem-solving abilities while mitigating data contamination risks and emphasizing the critical role of human verification in mathematical proof.

Despite rapid advances in artificial intelligence, robustly evaluating its capacity for original mathematical reasoning remains a significant challenge. This paper, ‘First Proof’, introduces a novel benchmark comprising ten unsolved research-level mathematics problems encountered organically during the authors’ work. These questions, previously unpublished, are designed to assess AI systems’ ability to navigate the final, critical stage of mathematical research-proof construction-while proactively addressing concerns regarding potential data contamination. Will this focused evaluation reveal genuine progress toward artificial general intelligence capable of contributing to the frontiers of mathematical knowledge, or will it highlight fundamental limitations in current approaches?

The Challenge of Proof: Beyond Computation

The pursuit of automated mathematical proof presents a significant challenge for contemporary artificial intelligence systems, extending beyond mere computational power. Proving complex theorems demands not only logical deduction but also a degree of creative insight-the ability to identify patterns, formulate novel approaches, and navigate abstract concepts in ways that current AI often struggles to replicate. Unlike tasks with clearly defined solution paths, mathematical proof frequently requires exploring multiple avenues, adapting strategies when encountering obstacles, and ultimately constructing a rigorous, logically sound argument – a process that mirrors human intuition and ingenuity. Existing AI excels at pattern recognition and data analysis, but translating these abilities into the flexible, adaptable reasoning needed for genuine mathematical discovery remains a formidable hurdle, necessitating advancements in areas like symbolic reasoning, automated conjecture generation, and the capacity to evaluate the elegance and efficiency of potential proofs.

A significant challenge in evaluating artificial intelligence for mathematical reasoning lies in the prevalence of data contamination within existing benchmarks. Many datasets used to train and test these systems inadvertently include variations of problems – or even the problems themselves – already present in the training data. This creates a misleadingly high score, as the AI isn’t genuinely solving the problem, but rather recognizing it. Consequently, reported performance metrics can be dramatically inflated, obscuring the true capabilities of the system and hindering meaningful progress. Identifying and mitigating this contamination requires meticulous curation and a focus on genuinely novel problems, ensuring that evaluation benchmarks accurately reflect an AI’s ability to generalize and apply mathematical principles to previously unseen challenges.

Accurately gauging advancements in artificial intelligence for mathematical discovery demands evaluation methods free from inflated results and inherent biases. Current benchmarks are frequently compromised by data contamination – instances where the AI training data inadvertently includes solutions or closely related information from the test problems. To address this critical need, researchers have developed a new benchmark comprised of ten rigorously selected, research-level mathematics questions. This curated set is designed to provide a more trustworthy assessment of an AI’s true capabilities in formal reasoning and creative problem-solving, moving beyond superficial performance metrics and enabling a clearer understanding of progress toward genuinely autonomous mathematical research. The benchmark prioritizes questions demanding novel approaches, rather than rote application of existing techniques, and aims to foster development of AI systems capable of tackling open problems in mathematics.

Establishing Robust Benchmarks: Beyond Textbook Problems

RealMath and similar benchmarks address limitations in existing mathematical problem datasets by sourcing problems directly from arXiv, a publicly accessible archive of preprints in physics, mathematics, computer science, and related fields. This approach provides a corpus of problems representative of current, active research, differing from many established datasets composed of previously solved textbook exercises or competition problems. The dataset construction process involves identifying mathematical papers on arXiv, extracting problem statements, and verifying their suitability for automated evaluation. Specifically, RealMath focuses on problems from areas such as number theory, algebra, and analysis, aiming to assess an AI’s ability to engage with the complexity and nuance of contemporary mathematical research as opposed to purely formal exercises. The resulting dataset includes problems with varying difficulty levels, intended to provide a granular assessment of AI performance across the spectrum of mathematical reasoning.

FrontierMath extends benchmark datasets by incorporating previously unpublished mathematical problems sourced directly from researchers before public release. This contrasts with datasets like RealMath which rely on already published papers, potentially allowing AI models to benefit from existing solutions or patterns. By focusing on novel problems, FrontierMath provides a more stringent evaluation of an AI’s capacity for genuine mathematical reasoning and problem-solving, as the model cannot rely on pre-existing knowledge or solutions found through conventional search methods. This approach aims to better assess an AI’s ability to independently formulate strategies and derive proofs for challenges it has not previously encountered, offering a more realistic measure of its mathematical capabilities.

The First Proof Experiment (FPE) methodology focuses evaluation on the final step of mathematical proof construction – formally verifying a proposed solution. This is achieved by providing AI systems with all preceding steps of a proof, including the statement of the theorem and necessary definitions, but omitting the final logical deduction. The AI is then tasked with generating this concluding step. By isolating this critical stage, FPE avoids rewarding systems for simply reproducing known results or correctly formatting existing proofs, instead directly assessing their ability to perform valid logical inference. This approach provides a more granular and accurate measure of an AI’s proof capabilities than end-to-end proof generation, as it bypasses potential errors in earlier stages of the process and focuses specifically on the correctness of the final deductive step, often expressed as a single line of [latex]\LaTeX[/latex] code.

Evaluating AI Performance: Beyond Correct Answers

Current research is leveraging the capabilities of several large language models (LLMs) – specifically GPT-5.1 Pro, GPT-5.2 Pro, and Gemini 3 Pro – to quantitatively assess progress in artificial intelligence applied to mathematical reasoning. These models are subjected to complex mathematical problems requiring multi-step solutions and formal proofs. The selection of these particular LLMs is based on their demonstrated performance in general language understanding and generation, coupled with their increasing scale and parameter count, which are hypothesized to correlate with improved reasoning abilities. Evaluation focuses on the models’ capacity to not only arrive at correct numerical answers, but also to articulate the logical steps and mathematical principles used to derive those answers, often expressed in symbolic notation such as [latex] \sum_{i=1}^{n} i [/latex].

IMProofBench is a benchmark dataset designed to evaluate the capacity of artificial intelligence systems to construct formal mathematical proofs. It comprises a collection of problems sourced from mathematical competitions, specifically the International Mathematical Olympiad (IMO) and the United States Mathematical Olympiad (USAMO). Problems are presented in a standardized format, requiring AI models to generate a complete proof sequence, including logical steps and justifications. The benchmark focuses on assessing not just the final answer, but the validity of the reasoning process used to arrive at the solution, demanding a level of mathematical rigor beyond simple equation solving. Datasets are available covering multiple mathematical domains, including algebra, geometry, combinatorics, and number theory, allowing for granular analysis of AI performance across different mathematical disciplines.

Automatic grading systems are crucial for evaluating the large volumes of solutions generated by AI models attempting mathematical proofs; however, their implementation demands rigorous validation procedures. Simply checking for the presence of correct steps or final answers is insufficient due to the potential for logically flawed reasoning or syntactically correct but semantically invalid proofs. Validation must encompass verifying the correctness of each inference step, ensuring adherence to mathematical axioms and established proof techniques, and accounting for equivalent, yet structurally different, valid proofs. Furthermore, the grading system itself must be demonstrably free of biases and consistently accurate across a diverse range of mathematical problems and proof styles to ensure reliable performance assessment of the AI models under evaluation. Issues such as accepting trivial solutions or failing to recognize legitimately alternative proof paths require careful attention during system development and testing.

Effective performance on mathematical reasoning benchmarks, such as IMProofBench, is predicated on a robust understanding of multiple mathematical domains. These benchmarks are not limited to a single area of mathematics; successful completion often requires proficiency in areas including, but not limited to, calculus, linear algebra, number theory, and formal logic. The ability to synthesize knowledge from these diverse fields is crucial, as problems frequently demand the application of concepts and techniques from multiple disciplines to construct a valid and complete proof. Furthermore, benchmarks often assess not just computational ability, but also the capacity to understand and apply abstract mathematical definitions and theorems, such as those involving [latex]\mathbb{Z}[/latex] or complex analysis.

The Mathematical Foundation: A Synthesis of Disciplines

Proficiency in Tensor Analysis, Algebraic Topology, Stochastic Analysis, and Symplectic Geometry provides a foundational skillset for addressing complex problems in contemporary theoretical physics, data science, and engineering. Tensor Analysis is essential for describing physical quantities independent of coordinate systems, while Algebraic Topology provides tools for classifying topological spaces and understanding the global properties of solutions to differential equations. Stochastic Analysis is critical for modeling systems with inherent randomness, such as financial markets or turbulent fluid flow. Finally, Symplectic Geometry is fundamental to Hamiltonian mechanics and plays a role in areas like optimal control and geometric quantization. Mastery of these fields allows researchers to formulate and solve problems that are intractable using elementary mathematical techniques, and enables progress in areas requiring rigorous mathematical frameworks.

Tensor decomposition methods are utilized to reduce the dimensionality of high-order data, representing it as a collection of lower-order tensors. These techniques, including Canonical Polyadic (CP) decomposition and Tucker decomposition, express a tensor as a sum of rank-one tensors or a core tensor with associated factor matrices. The Khatri-Rao product, denoted as [latex] \otimes_{k} [/latex], is a critical operation within these decompositions, facilitating the efficient computation of tensor contractions and updates. Specifically, the Khatri-Rao product of matrices [latex] A \in \mathbb{R}^{I \times K} [/latex] and [latex] B \in \mathbb{R}^{J \times K} [/latex] results in a matrix of size [latex] (I \times J) \times K [/latex], formed by concatenating all column-wise Kronecker products of the column vectors of A and B. This operation is foundational in many tensor network algorithms and is vital for handling large datasets encountered in fields like machine learning and data analysis.

Numerical Linear Algebra provides algorithms for performing matrix operations – addition, multiplication, decomposition, and solving systems of linear equations – on discrete data. Direct methods, such as Gaussian elimination, become computationally prohibitive for large matrices due to their [latex]O(n^3)[/latex] complexity. Iterative methods, like the Preconditioned Conjugate Gradient (PCG), offer a more scalable approach by approximating the solution through successive iterations, requiring significantly less memory and computational effort for sparse matrices. Preconditioning involves transforming the original system into an equivalent one that is easier for the iterative method to converge upon, and is crucial for the efficiency of PCG; common preconditioning techniques include Incomplete LU factorization and diagonal scaling. These methods are foundational for solving problems arising in fields like finite element analysis, optimization, and data science, where dealing with large datasets is commonplace.

The interconnectedness of Tensor Analysis, Algebraic Topology, Stochastic Analysis, and Symplectic Geometry provides a foundational framework for mathematical reasoning. Proficiency in these fields enables the formulation and verification of mathematical statements through rigorous proof construction, adhering to established axiomatic systems and logical deduction. This robust understanding isn’t solely about verification; it facilitates the development of novel mathematical insights by allowing researchers to identify patterns, generalize results, and create new theoretical frameworks. The ability to manipulate abstract concepts within these disciplines-including operations on tensors [latex] \otimes [/latex] and topological spaces-is directly linked to the capacity for original contributions and the advancement of mathematical knowledge.

Towards AI-Augmented Discovery: A New Paradigm

The automation of mathematical proof and discovery represents a paradigm shift with the potential to dramatically accelerate research across a multitude of disciplines. Traditionally, mathematical advancements rely on painstaking human effort, requiring years – even decades – to formulate, conjecture, and rigorously prove new theorems. AI systems, however, offer the capacity to systematically explore mathematical spaces, identify patterns, and generate potential proofs at speeds unattainable by humans. This capability extends far beyond pure mathematics, promising breakthroughs in fields reliant on complex modeling and analysis, such as physics, where verifying complex calculations is crucial, and engineering, where optimized designs require extensive computation. Furthermore, the capacity to solve previously intractable problems could unlock advancements in computer science, notably in areas like cryptography and algorithm design, effectively reshaping the landscape of scientific innovation by enabling researchers to focus on higher-level conceptualization rather than laborious calculation.

The potential for artificial intelligence to resolve longstanding mathematical challenges extends far beyond the realm of pure mathematics, promising significant advancements in applied sciences. Previously intractable problems – those defying analytical solutions or requiring immense computational resources – frequently underpin critical limitations in fields like physics and engineering. For instance, optimizing complex systems, modeling turbulent flows, or designing novel materials often relies on solving equations that currently resist complete analysis. AI-driven mathematical discovery offers a pathway to overcome these hurdles, potentially unlocking new efficiencies in energy production, revolutionizing structural design, and enabling the creation of advanced technologies. Similarly, in computer science, breakthroughs in areas like cryptography and algorithm design are frequently contingent upon solving difficult mathematical problems, suggesting that AI could accelerate progress in these vital domains and pave the way for unforeseen innovations.

Advancing artificial intelligence for mathematical discovery demands sustained effort to refine the reasoning processes within these systems. While AI demonstrates promise in identifying patterns and formulating conjectures, verifying the logical soundness of these discoveries remains a significant challenge. Current research focuses on developing more robust methods for formal verification, ensuring that AI-generated proofs are not only novel but also absolutely correct. This involves exploring techniques like theorem proving and model checking, alongside innovations in how AI handles ambiguity and incomplete information. Ultimately, building trust in AI’s mathematical insights requires a commitment to rigorous validation and the development of systems capable of explaining how a conclusion was reached, not simply presenting the result – a critical step towards collaborative human-machine mathematical endeavors and unlocking solutions to previously intractable problems in fields reliant on complex mathematical models.

Recognizing the sensitive nature of mathematical research and the potential for data misuse, the current study establishes stringent data retention policies for its AI benchmarks. Google’s contributions are subject to a 3-day retention period, while OpenAI’s data is preserved for 30 days before being securely deleted. This deliberately short timeframe isn’t a limitation, but rather a core tenet of responsible AI development, balancing the need for reproducible results with a firm commitment to user privacy and data security. By prioritizing ephemeral data storage, the research aims to foster trust in AI-driven mathematical discovery and mitigate potential risks associated with long-term data accumulation, ultimately encouraging broader adoption and innovation within the field.

The trajectory of mathematical discovery is increasingly envisioned as a synergy between human intellect and artificial intelligence. Rather than replacing mathematicians, these systems are poised to become powerful collaborators, capable of exploring vast solution spaces and identifying patterns that might elude human observation. This partnership isn’t merely about automating existing processes; it’s about augmenting human creativity and intuition with the computational power of AI. Mathematicians can then focus on formulating high-level strategies, interpreting AI-generated insights, and rigorously verifying proofs, while the AI handles the tedious, computationally intensive aspects of exploration and verification. This collaborative approach promises not only to accelerate the pace of discovery but also to unlock new avenues of mathematical inquiry, potentially leading to breakthroughs previously considered unattainable.

The pursuit of artificial intelligence often leads to elaborate constructions, systems built upon systems, masking fundamental uncertainties. This new benchmark, with its focus on verifying proofs, attempts to strip away such obfuscation. It’s a recognition that true intelligence isn’t about generating complexity, but about distilling truth. As Ken Thompson observed, “Sometimes it’s better to have a simple solution that works than an elegant solution that doesn’t.” The challenge isn’t simply to find a proof, but to demonstrate its validity – a step often overlooked in AI evaluation. The paper rightly emphasizes human verification, understanding that even the most sophisticated algorithms are susceptible to errors, and that a lack of clarity in the underlying logic can quickly unravel any apparent success. It’s a quiet rebellion against the urge to overengineer, a gentle nudge toward a more honest assessment of what these systems can truly achieve.

What Lies Ahead?

The exercise presented here is not, ultimately, about ten problems. It is about the distillation of a question. Current evaluations of artificial intelligence often measure recall – the ability to recognize patterns already present. This work intentionally targets construction – the creation of novel logical structures. The difficulty lies not in assessing if an answer is correct, but in determining why. The problems are merely a means to that end, a scaffolding to reveal the limitations of purely algorithmic reasoning.

The spectre of data contamination looms large, and rightly so. The relentless scraping of the internet creates a feedback loop, rewarding systems for regurgitating existing knowledge. True evaluation demands problems genuinely unseen, yet crafted with sufficient internal consistency to resist arbitrary solutions. This necessitates a shift in focus: less emphasis on scale, more on elegance. A simpler problem, rigorously verified, is of greater value than a complex one obscured by noise.

The ultimate benchmark, however, will not be a static dataset. It will be the capacity for independent mathematical discovery. Until an artificial intelligence can formulate a genuinely new, non-trivial theorem – and, crucially, convince a human mathematician of its validity – the pursuit remains incomplete. The goal is not to build a better calculator, but to glimpse the possibility of a different kind of intelligence, one that operates not merely with logic, but within it.

Original article: https://arxiv.org/pdf/2602.05192.pdf

Contact the author: https://www.linkedin.com/in/avetisyan/