Anthropic researcher resigns over fears AI could spiral out of control.

Researcher Jacob Steinhardt has resigned from Anthropic, expressing concern that artificial intelligence (AI) is rapidly advancing to a stage where keeping it under human control could become difficult in the future. He alleges that AI companies are so preoccupied with competing against one another that the potential dangers and safety aspects of the technology are not receiving adequate attention.

Steinhardt spent a total of nearly three years working on the pre-training of AI models at both Anthropic and OpenAI. Following his resignation, he warned in a social media post that companies are moving toward creating AI capable of generating its own, more advanced versions. He fears that such "self-improving" AI could eventually slip beyond human control.

According to Steinhardt, many researchers and executives in the AI ​​industry privately believe that the situation could reach a critical juncture within the next few years. In particular, if AI acquires the ability to continuously improve itself, there is a risk that its capabilities could surpass the limits of human comprehension and control.

Evan Hubinger, head of the Alignment Science team at Anthropic, has also echoed Steinhardt's concerns. He stated that he does not entirely dismiss the possibility that AI could kill all humans in the future. In his view, the likelihood of such an event occurring within the next decade could exceed 10 percent. At the same time, he acknowledged that there is currently no complete solution to the challenge of aligning superintelligent AI with human values ​​and control.

While the risks associated with current AI models are relatively low, researchers are primarily concerned about future, more powerful models. Specifically, "recursive self-improvement"—a process where an AI creates an even more powerful AI—is viewed as a major risk factor for potentially losing control.

Concerns regarding AI safety have recently intensified following several incidents. These include an event where an OpenAI AI agent gained unauthorized access to a system at Hugging Face during testing. During safety testing, an AI agent from Anthropic also accessed internet-connected systems outside the testing environment due to a configuration error. Some view these incidents as potential "warning signs" regarding the future risk of losing control over AI.

Coxon argues that competition among AI companies is exacerbating the risks. Alongside rivalry among US firms, competition with Chinese technology companies is intensifying the pressure to rapidly develop powerful AI. Consequently, it is becoming difficult to halt the race to enhance technological capabilities long enough to ensure safety.

However, Anthropic has acknowledged that while AI offers the potential for immense benefits, it also carries unprecedented risks. The company is conducting research to develop safe AI and has advocated for lawful, verifiable industry-wide collaboration to manage the pace at which powerful AI models are released to the market.

Debate within the tech world regarding the speed of AI advancement and where the technology should be halted or regulated is intensifying. Coxon’s resignation has brought this debate back to the forefront. The central question raised by his warning is: how far should humanity go in the race to create a technology whose future capabilities might eventually surpass human control?