Can Humans Still Hold the Reins When AI Starts Creating AI?

AI development company Anthropic has issued an unusual warning against the race to develop cutting-edge AI. The company suggests that AI is moving beyond being merely a human assistance tool and is entering a phase where it accelerates the development of AI systems themselves. Anthropic proposes that before society, regulations, and safety research fall behind, the industry should establish mechanisms to slow down or temporarily halt development.

At the heart of this discussion is the concept of "recursive self-improvement." This refers to a cycle where AI designs and develops more advanced AI, which in turn creates the next generation of AI. While it sounds like science fiction, Anthropic treats this not as a distant fantasy but as a realistic risk on the current technological trajectory.

An article from Lanka Newspapers reports that Anthropic warns of the "danger of humanity losing control over advanced AI" and suggests that major AI companies should jointly establish a mechanism to halt development. However, a careful reading of Anthropic's original claims reveals that the company is not simply calling for an immediate halt to all AI development. More accurately, they propose preparing international options to verifiably slow down or stop cutting-edge AI development if certain risk conditions are met.

Why Did Anthropic Use Such Strong Language?

Anthropic's sense of urgency stems from the fact that AI is already playing a significant role in AI development itself. The company explains that over 80% of the code merged internally is written by Claude. Furthermore, the amount of code produced per engineer has significantly increased compared to the past.

This is not just about "convenient code assistance." A few years ago, AI was merely suggesting short code snippets. Now, it handles file-level edits, bug fixes, long-duration tasks, and task distribution among multiple agents. Anthropic believes that if these changes continue, AI might eventually autonomously handle a substantial portion of model development.

Of course, the company is not claiming that "AI is already fully creating itself." The important aspect is the direction. Humans set the goals, AI devises the methods, runs experiments, writes code, and evaluates results. As this balance gradually shifts towards AI, there might come a point where the development speed far exceeds the decision-making speed of human society.

The issue is not just the capability of AI itself. It's whether surrounding systems like social institutions, laws, audits, international agreements, corporate governance, security measures, and human understanding can keep up with the speed of AI development.

While AI improves in cycles of days, hours, or even less, policies and international agreements move on a scale of months to years. This speed difference is at the core of Anthropic's warning.

Why It's Difficult to "Stop"

It's easy to say that AI development should be paused. However, implementation is extremely challenging.

Firstly, there's the issue of competition. If one company voluntarily stops development but others do not, only the cautious company falls behind. The same can happen between countries. If U.S. companies stop, but companies and government research institutions in China, Europe, the Middle East, and other regions continue, the technological lead might simply shift.

Secondly, there's the issue of verification. Unlike fields like nuclear weapons or missile facilities, where physical installations are easily monitored by satellites, large-scale AI learning is less visible. What is needed are data centers, GPUs, electricity, software, researchers, and funding, many of which are also used as general-purpose civilian infrastructure. It's not easy to confirm whether someone is secretly continuing development.

Thirdly, defining "what to stop" is also difficult. Should basic research be halted, or just large-scale learning? Are improvements to existing models allowed? How should open-source development be handled? Without deciding on the conditions for stopping, lifting, supervising, and penalties for violations, it won't be an effective agreement.

Anthropic itself acknowledges this difficulty. What the company seeks is not a unilateral halt based on goodwill, but a mechanism where multiple major AI labs stop under the same conditions and can verify each other. In other words, it's an approach akin to arms control for AI.

Clear Division of Opinions on Social Media

 

Reactions to this proposal are divided on social media and in tech communities.

Supporters view Anthropic's warning as an "important signal from within." The fact that a company developing cutting-edge AI itself is raising concerns about the gap between development speed and societal preparedness carries weight, rather than it coming from external critics or regulators. Accounts interested in AI safety and policy appreciate that the risks of recursive self-improvement are being addressed head-on and that a verifiable industry-wide slowdown is being discussed.

On the other hand, there is strong skepticism. In tech communities like Hacker News, questions arise about whether it contradicts the company's full-speed AI development while advocating for safety. The more Anthropic emphasizes code generation by Claude and improved development efficiency, the more it raises the question, "If it's so dangerous, why doesn't the company stop first?"

Additionally, there are practical complaints from Claude users. Despite claims of AI being able to autonomously work for long periods, users report issues like API limitations, temporary outages, and interruptions in long-duration tasks. While this is not a direct rebuttal to AI risk theory, it indicates a gap in perception about how autonomous AI currently is.

Furthermore, there are harsher views from political and business-oriented commentators. Some suspect that as Anthropic considers going public or achieving a high valuation, emphasizing AI risks could influence the creation of regulatory frameworks, forming rules favorable to the company. Particularly, strong regulations and verification systems might be manageable for large companies but could impose a significant burden on smaller companies and open-source groups. This leads to criticism that it might be regulatory capture under the guise of "safety" to suppress competitors.

Thus, reactions on social media fall into three main categories: First, those who agree that the risk of AI self-improvement is real and that international safety measures should be established now. Second, skeptics who understand the danger but feel there's a contradiction between Anthropic's actions and claims. Third, critics who see this not as a safety discussion but as a corporate strategy concerning regulation, market, and IPO.

Differences in Stance with Other Companies Like OpenAI

This proposal also highlights differences with other AI companies. According to reports, OpenAI believes that AI rules and accountability should be determined by democratic governments, not agreements between private companies.

This is an important point. When a company with cutting-edge technology like Anthropic says, "Let's create a mechanism to stop within the industry," it might sound realistic at first glance. However, there's the issue of whether private companies alone should decide on the brakes for technology that affects society as a whole.

On the other hand, relying solely on the government may not be sufficient. The reality of AI development is highly specialized and progresses rapidly. It takes time for regulators to understand the technical standards on the ground and create effective rules across borders. Meanwhile, the development competition between companies continues.

Therefore, what will be needed in the future is a complex governance system where companies, governments, researchers, and civil society each have roles. Companies should disclose internal data and technical knowledge, governments should provide democratic legitimacy and enforcement power, and third-party researchers and civil society should conduct monitoring and criticism. Anthropic's proposal serves as a starting point for this discussion.

The Real Fear Isn't "Rogue Robots"

When it comes to AI risks, it's easy to imagine scenarios like robots rebelling against humanity, as seen in movies. However, the important discussion here is about more mundane and realistic risks.

For example, if AI gains the ability to find software vulnerabilities in large quantities, it could be used for both defense and attack. While AI accelerating research and development could bring significant benefits to medicine and science, it could also lead to the spread of dangerous biological knowledge and cyberattack capabilities. If AI boosts corporate productivity by multiples, the labor market and competitive environment could change rapidly.

Furthermore, as AI begins to accelerate AI development, humans may lose the luxury of "understanding what's happening before responding." By the time a problem is discovered, the next generation of models may already be in operation. Capabilities continue to be updated before safety evaluations and audits are completed. This is the fundamental concern regarding recursive self-improvement.

It's important to note that Anthropic is not denying the benefits of AI. The company acknowledges that AI has the potential to bring significant benefits to science, medicine, cybersecurity, and productivity improvement. The problem is that the greater the benefits, the harder it is to stop the development race.

Is It Just "Fear-Mongering"?

As critics point out, when AI companies talk about risks, there is always a conflict of interest. When large companies say, "This technology is dangerous and needs regulation," there's a possibility that the regulation could end up being an entry barrier that only benefits the large companies.

Particularly, Anthropic has placed safety at the core of its brand. Emphasizing the dangers of AI aligns with the company's corporate image. It could also demonstrate to investors and customers that "we are the most cautious and responsible AI company."

However, this doesn't render the warning itself meaningless. Even if the company's motivations include strategic considerations, the technical issues they point out could still be real. What's more important is not to judge solely based on "who is saying it," but to dissect the content of the claims.

Is AI Accelerating AI Development?
Is its speed surpassing social systems?
Can a pause or slowdown be technically and politically verified?
Will it become a mechanism that only protects large companies?
How should open-source, small companies, and research institutions be handled?
Who should have the final decision-making authority, the government or private companies?

These are unavoidable questions, regardless of whether one supports or criticizes Anthropic.

It's Not Someone Else's Problem for Japan Either

This discussion is not just about U.S. AI companies. Japanese companies have already introduced generative AI for work efficiency, development, customer service, advertising, article creation, analysis, and education. As AI agents begin to autonomously handle multiple tasks, corporate competitiveness will change significantly.

On the other hand, Japan heavily relies on major U.S. companies in the AI foundational model development race. If overseas AI companies increase their development speed, the impact will extend to Japan's industry, employment, and information environment. Conversely, if AI development pauses or regulations are discussed overseas, it will affect the usage environment and costs for Japanese companies.

In fields like media, advertising, system development, finance, healthcare, and education, improvements in AI capabilities directly lead to changes in business structures. Companies are moving from the phase of "using it because it's convenient" to considering "how much to entrust to AI," "who audits AI's decisions," and "how to verify the results of AI agents working together."

Anthropic's warning is based on the internal circumstances of a cutting-edge AI lab, but its questions apply to general companies as well. To what extent can humans understand and take responsibility for code created by AI, articles written by AI, reviews judged by AI, and ad campaigns designed by AI? This is already a real management challenge.

It's Not a Choice Between "Complete Halt" or "Unlimited Development"

Framing this discussion as a conflict between AI proponents and anti-AI factions is dangerous. What is needed is not a binary choice between a complete halt and unlimited development, but the design of a phased and verifiable brake.

For example, require pre-registration for learning that uses a certain amount of computing resources. Mandate third-party evaluations for advanced autonomous agents. Share evaluation results of dangerous capabilities in a limited manner. Establish protocols for multiple companies to simultaneously slow down development if significant safety risks are confirmed. Research technical mechanisms to detect violations or leapfrogging.

Such systems are not easy to create. However, if we don't start creating them, we'll never be on time. As Anthropic says, the mechanisms of trust and verification don't come together overnight. International systems like nuclear arms control have been built over a long period. In AI, there may be less time to spare.

Of course, transparency is necessary in system design. If only large AI companies create rules behind closed doors, they won't gain social trust. Even if AI development needs to be halted or slowed, it must not be an excuse to strengthen the market dominance of certain companies.

Conclusion: Anthropic's Warning Highlights the "Next Issue" in AI Competition

Anthropic's proposal indicates that the discussion around AI has entered a new phase. Previously, the focus was on whether AI would take human jobs, increase fake information, or infringe on copyrights. Now, the question is how human society can maintain control in an era where AI accelerates AI development itself.

It's no surprise that opinions are divided on social media. Anthropic's claims contain both serious safety concerns and aspects of corporate strategy. As supporters say, recursive self-improvement might be an undeniable risk. As critics argue, caution is needed when companies advancing development try to become the designers of regulations.

However, regardless of which stance one takes, one thing is clear: The speed of AI development is already beginning to surpass the speed of societal discussions. That's why what's needed now is neither succumbing to fear nor escaping into optimism.

Before entering an era where AI creates AI, humans must decide under what conditions to proceed and under what conditions to stop. The longer this rule-making is postponed, the fewer options there will be in the future.

Source URL

An article from Lanka Newspapers. An article reporting that Anthropic warned of the risk of losing control over AI and the need for industry-wide development pauses and slowdowns.
https://www.lankanewspapers.com/2026/06/06/anthropic-warns-ai-could-spiral-out-of-human-control-calls-for-industry-wide-development-pause

Official article from Anthropic Institute. Primary information about recursive self-improvement, AI development support by Claude, and the concept of slowing down or pausing development.
https://www.anthropic.com/institute/recursive-self-improvement

AP News article. Reporting on Anthropic's proposal, OpenAI's perspective, and the need for verifiable pause mechanisms.
https://apnews.com/article/938c99158e5953601cf3322f1cec12af

Business Insider article. Reporting on reactions from experts, politicians, and industry stakeholders to Anthropic's proposal to pause AI development.
https://www.businessinsider.com/anthropic-ai-pause-reactions-response-2026-6

Business Insider article. Reporting on the point that Claude generates most of the code within Anthropic and the background to the company's call for a slowdown in development.