Can AI Truly Understand Physics?

Author: Denis Avetisyan

A new benchmark reveals that despite advances in generating realistic videos, current AI systems still struggle with basic physical reasoning and maintaining consistency in dynamic scenes.

PAI-Bench establishes a comprehensive evaluation framework for advancements in Physical AI, encompassing both the generation of physical worlds from textual or conditional inputs and robust understanding of existing physical environments, thereby facilitating progress across diverse topics within the field.

PAI-Bench, a comprehensive benchmark for Physical AI, evaluates the perception and prediction capabilities of AI in physical environments, highlighting significant limitations in current multimodal learning approaches.

Despite recent advances in multimodal AI, a systematic understanding of how well models truly perceive and predict real-world physical dynamics remains elusive. To address this gap, we introduce PAI-Bench: A Comprehensive Benchmark For Physical AI, a unified evaluation suite comprising nearly three thousand real-world video scenarios designed to assess perception and prediction across generation and understanding tasks. Our analysis reveals that while current video generation models achieve impressive visual fidelity, they often struggle with physically plausible motion, and even large language models exhibit limited capacity for forecasting and causal reasoning in physical contexts. These findings suggest substantial challenges remain before AI systems can reliably navigate and interact with the physical world – what architectural and training innovations will be required to bridge this gap?

The Imperative of Rigorous Evaluation in Physical AI

Physical AI represents a significant departure from traditional artificial intelligence, shifting the focus from algorithms that process data in the digital realm to those that can actively interact with and manipulate the physical world. This emerging field necessitates algorithms capable of robust perception – interpreting sensory input from cameras, lidar, and other sensors – combined with predictive modeling to anticipate how actions will unfold and, crucially, the ability to execute those actions through robotic systems or other physical interfaces. Unlike purely digital intelligence which operates within defined parameters, Physical AI confronts the inherent unpredictability and complexity of real-world environments, demanding adaptable and resilient algorithms capable of navigating dynamic situations and responding to unforeseen challenges. The ultimate goal is not merely to create intelligent software, but to build systems that can embody intelligence, bridging the gap between computation and action.

Current evaluation metrics for Physical AI frequently struggle to accurately reflect performance in real-world scenarios. Many benchmarks prioritize simplified, simulated environments or narrowly defined tasks, creating a disconnect between reported success and actual capability. This leads to inflated expectations, as algorithms that excel in these artificial settings often falter when confronted with the complexities of unpredictable physical interactions, sensor noise, and dynamic environments. Consequently, progress is hampered by a focus on optimizing for benchmark scores rather than developing genuinely robust and adaptable embodied intelligence. The limitations of these evaluations obscure critical weaknesses and slow the development of Physical AI systems capable of reliable operation in authentic, unstructured settings.

The advancement of Physical AI hinges not simply on algorithmic breakthroughs, but crucially on the development of standardized, robust evaluation frameworks. Current metrics often prove inadequate when assessing a system’s ability to navigate complex, unpredictable real-world scenarios, potentially masking critical flaws until deployment. A rigorous benchmark suite must move beyond simulated environments and incorporate diverse physical challenges, realistic sensor noise, and quantifiable safety parameters. Such a framework would not only accelerate innovation by providing clear targets for improvement, but also instill confidence in the reliability and predictable behavior of these increasingly autonomous systems, ultimately fostering responsible development and public trust. Without these stringent evaluations, the potential benefits of Physical AI remain hampered by uncertainty and the risk of unforeseen consequences.

PAI-Bench is a benchmark comprising six physical AI application domains, utilizing control signals like blurred imagery and segmentation for control tasks, questions for understanding, and captions with derived prompts for generative evaluation.

PAI-Bench: A Foundational Suite for Physical AI Assessment

PAI-Bench utilizes a tiered evaluation strategy across three primary video tasks. Video generation assesses a model’s capacity to synthesize realistic and coherent video sequences from scratch. Conditional video generation extends this by requiring models to create videos based on specific textual or visual prompts, testing their ability to align generated content with given conditions. Finally, video understanding evaluates a model’s comprehension of video content through tasks such as action recognition, object detection, and scene classification. Performance is measured using established metrics relevant to each task, including Fréchet Video Distance (FVD) for generation quality and mean Average Precision (mAP) for understanding tasks, allowing for quantitative comparison of model capabilities.

PAI-Bench prioritizes tasks that assess a model’s ability to reason about and interact with the physical world, constituting the field of Physical AI. This design choice necessitates evaluation of capabilities such as object manipulation, spatial reasoning, and understanding of physical dynamics-all crucial for real-world applicability. Tasks are constructed to move beyond purely perceptual abilities and require models to demonstrate understanding of how objects behave under physical laws, including gravity, collisions, and momentum. Consequently, performance on PAI-Bench reflects a model’s potential for deployment in robotic systems or simulations involving realistic physical interactions.

PAI-Bench’s holistic assessment of model intelligence is achieved through its inclusion of diverse input modalities – encompassing RGB video, depth information, and scene geometry – and varying levels of task complexity. The benchmark presents models with challenges ranging from single-object reconstruction to multi-agent interaction within dynamic environments. This multifaceted approach moves beyond evaluations limited to single modalities or simplified scenarios, providing a more comprehensive understanding of a model’s ability to process information and reason about the physical world. Specifically, tasks are designed to assess capabilities such as object manipulation, physics prediction, and spatial reasoning, all crucial components of generalized AI performance.

PAI-Bench-G successfully generates physics domains and corresponding models, as demonstrated in this example.

Dissecting Realism: PAI-Bench-G and the Measurement of Physical Plausibility

PAI-Bench-G is designed to evaluate Video Generative Models (VGMs) through a dual-metric assessment of generated video content. Visual quality is quantified using a Quality Score, which is benchmarked against the established VBench dataset for comparative analysis. Complementing this is the Domain Score, which specifically measures the physical plausibility of the generated videos – assessing adherence to real-world constraints and physical laws. This combined approach allows for a comprehensive evaluation, moving beyond purely aesthetic considerations to include the realism and consistency of generated content.

Current state-of-the-art Video Generative Models (VGMs) achieve Domain Scores ranging from 81.6 to 82.1 on the PAI-Bench-G benchmark. While these scores indicate a high degree of visual quality in generated video content, they also reveal limitations in physical consistency. This suggests that, although VGMs can produce videos that appear visually realistic, they often fail to accurately simulate real-world physics and constraints, resulting in implausible or unrealistic motion and interactions within the generated scenes. The Domain Score specifically isolates and quantifies this aspect of physical plausibility, highlighting a key area for improvement in video generation technology.

The PAI-Bench-G evaluation framework incorporates a dual-metric approach to video generation assessment, explicitly addressing both perceptual quality and physical realism. Beyond subjective aesthetic appeal, generated videos are rigorously tested for adherence to real-world constraints and the laws of physics. This is achieved through the Domain Score, which evaluates the physical plausibility of the generated content, ensuring consistency with expected behavior and preventing visually convincing but physically impossible scenarios. This dual assessment is critical because high visual fidelity alone does not guarantee a believable or useful video; generated content must also be grounded in physical accuracy to be considered genuinely high-quality.

PAI-Bench-G contains a distribution of videos and question-answer pairs, enabling Domain Score evaluation with an average of 5-6 QA pairs per video.

PAI-Bench-C & U: Quantifying Control and Comprehension in Video AI

PAI-Bench-C introduces a rigorous method for assessing the controllability of video generation models. This benchmark utilizes conditional video generative models – systems designed to produce videos based on specific input signals, such as text prompts or desired actions. The core of PAI-Bench-C lies in its measurement of “Control Fidelity,” a metric that quantifies how accurately the generated video adheres to these input control signals. By systematically varying these controls and evaluating the resulting video output, researchers can pinpoint the strengths and weaknesses of different generative models, driving improvements in their ability to create videos that precisely match desired specifications. Essentially, the benchmark doesn’t just ask if a model can generate a video, but if it can generate the correct video, given explicit instructions – a crucial step towards reliable and predictable video synthesis.

PAI-Bench-U rigorously assesses the video comprehension abilities of advanced Multi-Modal Large Language Models (MLLMs), such as GPT-5 and Qwen3-VL, by challenging them with tasks requiring an understanding of visual content and its associated context. This benchmark moves beyond simple object recognition, probing the models’ capacity for complex reasoning about events, relationships, and intentions depicted within videos. By presenting a diverse array of video scenarios and corresponding questions, PAI-Bench-U aims to quantify how effectively these MLLMs can bridge the gap between visual information and semantic understanding – a crucial step towards achieving truly intelligent video processing and analysis.

Current evaluations utilizing PAI-Bench-U reveal a substantial gap between the video understanding capabilities of even the most advanced Multi-Modal Large Language Models (MLLMs) and human performance. While models like Qwen3-VL-235B-A22B achieve approximately 64.7% accuracy in interpreting video content and associated prompts, this figure falls considerably short of the 93.2% accuracy consistently demonstrated by human observers. This discrepancy highlights the ongoing challenges in equipping artificial intelligence with the nuanced comprehension of visual information and contextual reasoning that is readily available to people, suggesting a need for continued innovation in MLLM architectures and training methodologies to bridge this performance divide.

Accuracy on the PAI-Bench-U dataset improves with more input frames, consistently exceeding random-guess performance.

Toward True Intelligence: Establishing a Human Performance Baseline for Physical AI

PAI-Bench introduces a critical advancement in the evaluation of artificial intelligence by establishing a quantifiable Human Performance Baseline. This baseline isn’t merely an arbitrary score; it represents the average performance of human subjects tackling complex physical reasoning tasks – scenarios demanding an understanding of dynamics, mechanics, and intuitive physics. By directly comparing AI performance against this human standard, researchers gain an unprecedented ability to pinpoint specific areas where AI falls short and to measure progress with meaningful granularity. The benchmark isn’t limited to simple success or failure; it captures the degree to which an AI system mirrors human capabilities, fostering targeted improvements and accelerating the development of truly intelligent physical systems capable of operating effectively in the real world. This direct comparison shifts the focus from abstract metrics to demonstrable skill, offering a clear pathway toward achieving human-level performance.

The creation of PAI-Bench isn’t merely about measuring current AI capabilities; it actively spurs advancement in the field. By providing a standardized and challenging set of physical reasoning tasks, the benchmark encourages researchers to move beyond incremental improvements and pursue genuinely novel approaches to artificial intelligence. This focused challenge fosters the development of AI systems that are not only more accurate but also demonstrate greater robustness – the ability to perform reliably in varied and unpredictable environments. Consequently, PAI-Bench is accelerating progress towards AI that exhibits true intelligence – systems capable of adapting, generalizing, and solving complex physical problems with a level of sophistication previously unattainable, ultimately pushing the boundaries of what’s possible in robotics and autonomous systems.

Sustained assessment utilizing the PAI-Bench benchmark is paramount to charting the evolution of Physical AI and verifying genuine advancements toward human-level capabilities. The complexities inherent in physical reasoning demand continuous, rigorous testing beyond initial demonstrations; PAI-Bench facilitates this by providing a standardized, challenging platform for iterative development and comparative analysis. By consistently evaluating AI systems against the human performance baseline established within the benchmark, researchers can accurately pinpoint areas for improvement, identify emerging strengths, and ultimately measure progress towards creating truly intelligent machines capable of navigating and interacting with the physical world as effectively as humans. This ongoing evaluation isn’t merely about achieving specific scores, but about fostering a deeper understanding of the underlying principles of physical intelligence and guiding the field toward robust, reliable, and adaptable AI systems.

The user study interface facilitates the collection of human performance baselines on the PAI-Bench-U benchmark.

The pursuit of genuinely intelligent systems, as highlighted by PAI-Bench, demands more than mere perceptual mimicry. Current video generation models, while capable of producing visually compelling outputs, frequently falter when challenged with fundamental physical constraints – a critical failing. This resonates deeply with Yann LeCun’s assertion: “The real challenge is not just to make machines that can perceive, but machines that can understand.” PAI-Bench provides a rigorous testing ground, forcing models to confront the invariants that govern the physical world. Let N approach infinity – what remains invariant? The benchmark seeks to expose whether systems are truly learning underlying physical principles, or simply memorizing superficial visual patterns, thereby revealing the core deficiencies in current approaches to Physical AI.

Beyond Pixels: Charting a Course for Physical AI

The introduction of PAI-Bench serves not as a celebration of current capabilities, but as a rather stark illumination of their absence. Models may convincingly render visually plausible scenes, yet the benchmark reveals a consistent failure to understand the underlying physical principles governing those scenes. If an image appears realistic, but violates basic tenets of physics, it’s not intelligence – it’s cleverly disguised illusion. If it feels like magic, one hasn’t revealed the invariant.

Future work must prioritize provability over mere performance. The field requires a shift from training models to verify them. Metrics based on visual fidelity alone are insufficient; a true test lies in evaluating adherence to established physical laws. Formal methods, borrowed from robotics and control theory, should be integrated to ensure the internal consistency of these systems, establishing guarantees about their behavior in novel scenarios.

The challenge, then, isn’t simply generating more realistic videos. It’s building systems that possess a principled understanding of the physical world – a comprehension that extends beyond superficial appearances. Until then, the pursuit of Physical AI remains a fascinating, yet fundamentally incomplete, endeavor.

Original article: https://arxiv.org/pdf/2512.01989.pdf

Contact the author: https://www.linkedin.com/in/avetisyan/