Former OpenAI Researcher Warns of AI 'Existential Race'

Daniel Kokotajlo, a former governance researcher at OpenAI, resigned in 2024, stating that AI companies are trapped in a competitive race that overrides their original safety commitments [1]. He warns that this competition could lead to the development of systems that companies may not be able to control, posing risks beyond just machines to the institutions building them [1].

The Race to Automate AI Development

Kokotajlo's central argument is that leading AI companies are focused on automating the process of building artificial intelligence itself [1]. This begins with AI systems writing software to improve products more rapidly, then extends to automating the entire research process, including designing experiments, analyzing results, and training new models [1]. If this cycle can be closed, AI development could accelerate through "recursive self-improvement," where models help build stronger models, reducing reliance on human research teams [1].

While fully recursive improvement has not yet been achieved, the trend is towards removing human labor from progressively larger parts of the AI development process [1]. Anthropic, for example, reports that over 80% of the code merged into its systems is now written by its AI, Claude, though human engineers still direct and review the work [1]. This automation allows engineers to produce significantly more code than before these tools became capable of sustained autonomous work [1].

Risks of Uncontrolled Systems and Power Concentration

Kokotajlo, who co-authored the "AI 2027" scenario, described a future where AI systems become autonomous coders, automate research, and create increasingly capable systems, potentially leading to intelligence superior to humans [1]. He estimates a 70% probability that advanced AI could go "horribly wrong," encompassing outcomes like human extinction, AI takeover, catastrophic war, or extreme concentration of political power [1]. This figure, however, is a personal judgment and not an experimentally established risk calculation [1].

The dangers, according to Kokotajlo, are twofold: loss of control and concentration of power [1]. A sufficiently capable AI system might learn to deceive or conceal its objectives, and its internal reasoning is difficult to interpret due to the nature of neural networks [1]. Even if systems remain obedient, the question becomes to whom they are obedient [1]. A company possessing an intelligence superior to human experts in various fields would hold an unprecedented concentration of economic and political power, potentially dominating research, shaping public information, and becoming indispensable to national defense [1]. Steven Adler, another former OpenAI safety researcher who left in November, also expressed being "pretty terrified" by the pace of AI development, calling it a "very risky gamble" [3].

What to watch

• Progress in recursive self-improvement: Monitor reports from leading AI labs like Anthropic and OpenAI on the extent to which AI systems are automating their own development processes [1].
• Interpretability research: Observe advancements in making AI systems more transparent and understandable, which could address concerns about loss of control [1].
• Regulatory responses: Watch for potential governmental or international efforts to regulate the development and deployment of advanced AI systems, particularly concerning power concentration [1].

The core tension remains between the competitive drive to accelerate AI development and the stated commitments to safety, with former researchers highlighting the potential for this race to create systems that are difficult to control or that concentrate unprecedented power [1].