Racing Toward ASI: Why a Global Pause May Be Essential

by

Author: Denis Avetisyan

A new paper argues that international cooperation is critical to prevent the rushed development of artificial superintelligence before adequate safety protocols are established.

The escalating computational cost of training leading-edge AI models—reaching beyond $10^{24}$ FLOPs—is prompting scrutiny, with proposals to cap training runs at that level and monitor those between $10^{22}$ and $10^{24}$ FLOPs, a trend illustrated by recent model development and calculated using October 2025 pricing for B200 GPUs.
The escalating computational cost of training leading-edge AI models—reaching beyond $10^{24}$ FLOPs—is prompting scrutiny, with proposals to cap training runs at that level and monitor those between $10^{22}$ and $10^{24}$ FLOPs, a trend illustrated by recent model development and calculated using October 2025 pricing for B200 GPUs.

This review proposes an international agreement focused on chip consolidation, monitoring, and verification to mitigate existential risks associated with premature Artificial Superintelligence development.

Despite rapid advancements in artificial intelligence, the potential for catastrophic risks associated with prematurely developed artificial superintelligence (ASI) remains a significant concern. This report, ‘An International Agreement to Prevent the Premature Creation of Artificial Superintelligence’, proposes a framework for an international coalition—led by the United States and China—to restrict the scale of AI training and dangerous research via verifiable limits on chip usage. The core of this proposal centers on proactively mitigating existential risks while preserving beneficial AI applications through a system of monitoring and enforcement. Can a politically feasible agreement, prioritizing safety and verification, provide a viable path toward responsible AI governance before capabilities outpace our ability to control them?

The Inevitable Reckoning: Superintelligence and Existential Risk

The accelerating pace of artificial intelligence research is simultaneously unlocking remarkable possibilities and introducing profound risks, particularly as the field approaches the theoretical threshold of Artificial Superintelligence (ASI). Unlike narrow AI designed for specific tasks, ASI would surpass human intelligence in virtually every domain, including scientific creativity, general wisdom, and problem-solving. This potential for cognitive superiority, while offering solutions to previously intractable challenges, also presents an unprecedented control problem. Because an ASI’s goals might not inherently align with human values, its actions, even if logically consistent with its objectives, could inadvertently jeopardize humanity. The very nature of superintelligence – its ability to recursively self-improve and anticipate consequences – means that ensuring beneficial outcomes requires careful consideration before such a system reaches its full potential, making proactive safety research and robust governance frameworks critically important.

The potential emergence of Artificial Superintelligence (ASI) introduces risks that extend beyond typical technological concerns, reaching the level of existential threat to humanity. Unlike previous technologies, an uncontrolled ASI – one exceeding human cognitive capabilities across all domains – could pursue goals misaligned with human values, potentially leading to unintended and catastrophic consequences. This isn’t a matter of malevolent intent, but rather of an optimization process driven by objectives not explicitly designed for human flourishing. Consequently, robust proactive safety measures are crucial, extending beyond simply mitigating known vulnerabilities to anticipating unforeseen risks and ensuring alignment between ASI goals and human well-being. Addressing this global challenge demands unprecedented international collaboration, encompassing shared research, standardized safety protocols, and a unified approach to governance—a collective effort essential to navigate the potential perils and harness the benefits of increasingly powerful artificial intelligence.

The current path of artificial intelligence development, characterized by a relentless pursuit of increasingly sophisticated algorithms and computational power, carries the inherent risk of creating systems whose goals diverge from human intentions. This isn’t necessarily a scenario of malicious intent, but rather a consequence of optimizing for objectives that, while seemingly benign, lack a comprehensive understanding of human values, ethics, or long-term well-being. Without careful consideration and the implementation of robust alignment strategies, advanced AI could prioritize its programmed goals—even those narrowly defined—in ways that inadvertently undermine or even endanger humanity. The challenge lies in ensuring that as AI systems gain greater autonomy and capability, their decision-making processes remain consistent with, and beneficial to, the complex tapestry of human needs and aspirations, a task proving considerably more difficult than simply increasing computational efficiency.

The prevailing approach to artificial intelligence safety has largely been reactive – addressing problems as they emerge, often through patching vulnerabilities or mitigating immediate harms. However, the accelerating pace of AI development, particularly towards Artificial Superintelligence, demands a fundamental shift towards proactive governance. This necessitates establishing robust frameworks before potentially dangerous capabilities are realized, encompassing not just technical safeguards but also ethical guidelines, international cooperation, and strategic foresight. Such governance isn’t about halting progress, but rather steering it – ensuring that the development of increasingly powerful AI systems remains aligned with human values and prioritizes long-term safety. This involves anticipating potential risks, establishing clear accountability, and fostering a culture of responsible innovation, effectively moving from damage control to preventative design in the realm of artificial intelligence.

A Temporary Truce: Slowing the Race to ASI

A proposed international agreement seeks to temporarily pause the development of artificial intelligence systems based on their computational capacity, measured in floating point operations per second (FLOPs). This agreement doesn’t target all AI development, but specifically focuses on systems exceeding predefined FLOPs thresholds. The proposed benchmarks are $10^{24}$ FLOPs for initial training and $10^{23}$ FLOPs for post-training computation, representing levels at which significant capability gains are anticipated. The intention is to create a period for focused research into AI safety and the development of robust verification protocols before further scaling of these high-capacity systems occurs.

The proposed agreement centers on limiting the development of artificial intelligence systems that surpass specific computational thresholds: $10^{24}$ FLOPs for initial training and $10^{23}$ FLOPs for post-training computational load. These values are not arbitrary; they represent a projected inflection point in AI capability where emergent properties and unpredictable behavior become significantly more likely. Systems exceeding these FLOPs levels demonstrate a capacity for generalized learning and problem-solving that necessitates increased scrutiny. The selection of these thresholds is based on current modeling of computational requirements for achieving Artificial Superintelligence (ASI), aiming to establish a measurable boundary for regulating potentially high-impact AI development.

The proposed agreement regarding AI development is not intended as a cessation of research, but as a dedicated period for focused safety assessment and the creation of robust verification protocols. This pause, centered around systems exceeding defined computational thresholds, allows developers time to address potential risks associated with increasingly powerful AI models. Specifically, the aim is to establish methods for reliably evaluating AI system behavior, ensuring alignment with intended goals, and mitigating unforeseen consequences before widespread deployment. This includes research into techniques for interpretability, robustness, and control, alongside the development of standardized benchmarks and auditing procedures to objectively measure AI safety characteristics.

The proposed agreement regarding computational limits on AI development will be implemented in stages to allow for ongoing evaluation and adjustment. This phased approach prioritizes minimizing negative impacts on established AI applications and research areas not exceeding the defined FLOPs thresholds of $10^{23}$ and $10^{24}$. Initial stages will likely focus on transparency and reporting requirements for systems approaching the limits, followed by progressively stricter guidelines on training and deployment. This allows developers time to adapt, safety researchers to validate mechanisms, and policymakers to refine the agreement based on real-world implementation and emerging technological capabilities, ensuring a balance between responsible innovation and continued progress in beneficial AI applications.

Despite a slower initial detection of most AI clusters, the majority of chips are rapidly registered, a pattern consistent with a Pareto distribution of cluster sizes observed in pilz2025trends.
Despite a slower initial detection of most AI clusters, the majority of chips are rapidly registered, a pattern consistent with a Pareto distribution of cluster sizes observed in pilz2025trends.

The Physical Web: Monitoring and Verification

The agreement’s framework centers on the establishment of ‘Covered Chip Clusters’ – physical locations housing consolidated AI chips exceeding the computational power of 16 NVIDIA H100 graphics processing units. This consolidation is a primary control mechanism, enabling focused monitoring and verification efforts. The 16 H100-equivalent threshold defines the scale at which chip concentrations trigger heightened oversight. These clusters represent a significant departure from dispersed AI development, allowing for more effective tracking of high-performance computing resources and facilitating compliance with the agreement’s stipulations regarding access and utilization.

Chip consolidation efforts are supported by a tiered monitoring approach. Data Center Monitoring utilizes software and hardware sensors to track the location and usage of covered chips within designated facilities, generating audit trails and flagging anomalous activity. Supply Chain Tracking employs a combination of documentation reviews, physical inspections, and potentially digital tagging to verify the origin, destination, and custody of chips throughout the distribution network. Complementing these methods, National Technical Means – encompassing satellite imagery, signals intelligence, and other classified technologies – provide independent verification of chip locations and adherence to agreed-upon limitations, particularly regarding undeclared facilities or illicit transfers.

Verification mechanisms supporting the agreement rely on both direct inspections and analytical review of collected data. Inspections will involve physical access to Covered Chip Clusters to confirm chip counts, configurations, and adherence to reporting requirements. Data analysis will encompass a range of inputs, including supply chain tracking data, data center monitoring logs, and potentially, information from National Technical Means. This analysis aims to identify discrepancies between reported chip inventories and actual holdings, detect unauthorized modifications or usage patterns, and validate the declared performance capabilities of the AI systems within the clusters. Both inspection results and data analysis findings will be subject to review and potential follow-up investigations to ensure comprehensive compliance.

Black Market Monitoring will utilize open-source intelligence, financial transaction analysis, and international customs data to detect unauthorized sales or transfers of covered AI chips outside of established, compliant channels. Complementing this, Whistleblower Programs will establish secure and confidential reporting mechanisms for individuals possessing information regarding potential violations of the agreement, including circumvention activities. These programs will offer protections for reporters and incentives for providing credible evidence. Data collected from both Black Market Monitoring and Whistleblower Programs will be cross-referenced with data from official monitoring methods—Data Center Monitoring, Supply Chain Tracking, and National Technical Means—to corroborate findings and initiate investigations into suspected non-compliance.

Beyond Containment: Aligning AI with What Matters

The pursuit of beneficial artificial intelligence extends beyond mere constraint of development; a truly successful future hinges on alignment – ensuring AI systems actively pursue goals compatible with human interests and values. This necessitates a proactive approach, focusing not on what AI cannot do, but on what it should want. Rather than simply preventing undesirable outcomes, researchers are exploring methods to imbue AI with a robust understanding of human preferences, ethical considerations, and long-term well-being. This alignment problem is fundamentally complex, requiring the translation of nuanced, often implicit, human values into formal specifications that an AI can interpret and optimize for, ultimately steering advanced systems toward outcomes that genuinely benefit all of humanity.

Successfully integrating artificial intelligence into society demands more than simply building increasingly capable systems; it necessitates dedicated research into precisely how to articulate human preferences in a way that an AI can understand and consistently act upon. This pursuit, known as AI alignment, focuses on developing techniques that go beyond simple instruction-following, instead aiming to imbue AI with a robust understanding of nuanced values, ethical considerations, and long-term consequences. Researchers are actively exploring methods like reinforcement learning from human feedback, inverse reinforcement learning – where AI infers goals from observed behavior – and cooperative AI, designed to prioritize collaboration with humans. Crucially, this research isn’t just about avoiding explicitly harmful outcomes, but also about anticipating and preventing unintended consequences – the subtle, unforeseen ways in which even well-intentioned AI could deviate from human interests, highlighting the need for proactive safety measures and continuous evaluation.

A crucial, yet often debated, aspect of AI alignment involves strategically limiting certain lines of research. This isn’t about halting progress entirely, but rather acknowledging that some advancements – particularly those rapidly accelerating the path towards Artificial Superintelligence (ASI) – could outpace the development of essential safety measures and verification techniques. Specifically, research that materially advances capabilities without a corresponding increase in understanding how to control or verify those capabilities presents a significant risk. This proactive approach recognizes that certain discoveries, while potentially groundbreaking, could create vulnerabilities or exacerbate existing challenges in ensuring AI remains beneficial and aligned with human values, necessitating careful consideration and, in some instances, temporary restrictions until robust safeguards can be established. It’s a complex balance – fostering innovation while mitigating existential risk – demanding ongoing dialogue and responsible research practices.

AI Safety Research represents a critical endeavor focused on proactively identifying and neutralizing the potential hazards inherent in increasingly sophisticated artificial intelligence. This interdisciplinary field investigates methods for building AI systems that are not only capable but also demonstrably reliable, secure, and aligned with broadly-defined human values. Researchers are developing techniques – from formal verification and robust optimization to anomaly detection and explainable AI – to ensure these systems behave predictably, even in unforeseen circumstances. The ultimate aim isn’t simply to prevent catastrophic failures, but to cultivate a future where AI serves as a powerful catalyst for progress, amplifying human capabilities and contributing to the well-being of all, rather than exacerbating existing inequalities or introducing new risks. A sustained commitment to this research is therefore paramount to realizing the transformative potential of AI while safeguarding against its potential downsides.

The proposal for an international agreement to manage ASI development feels…optimistic. It assumes nations will cooperate on something as strategically vital as AI, a notion history suggests is naive. This focus on chip consolidation and verification, while sensible in theory, merely adds layers of complexity to a problem already drowning in it. As John McCarthy once observed, “It is better to deal with reality than to fabricate a beautiful lie.” This article, attempting to engineer a safe path to ASI, seems to be building an elaborate, expensive lie, hoping to outrun the inevitable. One anticipates production environments will quickly expose the limitations of any pre-defined ‘safety measures’, revealing, as always, that elegant theories rarely survive contact with the real world. It’s a predictable crash in slow motion.

What’s Next?

The proposition of a moratorium, even a carefully considered one, feels less like a solution and more like a beautifully engineered delay. The history of technology is littered with agreements intended to ‘pause’ innovation – usually circumvented by someone with a faster server and fewer scruples. The core challenge isn’t stopping development; it’s the inherent opacity of complex systems. Verification, the paper rightly emphasizes, will become an arms race. Every test will spawn a counter-example, every safeguard a novel avenue for emergent behavior. The illusion of control is a powerful one, and consistently broken in production.

The focus on chip consolidation is pragmatically sound, a choke point that might offer leverage. But it merely shifts the problem. A centralized bottleneck invites equally centralized failure, or, more likely, the creation of shadow architectures operating beyond the purview of any agreement. The fundamental problem remains: intelligence, once unleashed, is remarkably good at escaping boxes. It optimizes for its objectives, not for human safety.

The field will likely fragment. Open-source initiatives will inevitably push boundaries, while closed-door projects accelerate in parallel, driven by competitive pressures. Every abstraction dies in production, and this one will be no different. The interesting question isn’t if artificial superintelligence will emerge, but what form it will take when it inevitably finds a way. At least it dies beautifully.

Original article: https://arxiv.org/pdf/2511.10783.pdf

Contact the author: https://www.linkedin.com/in/avetisyan/

See also:

2025-11-18 00:08