Can a Rampaging AI Be Stopped? The Reality of the "Kill Switch" Being Considered in Silicon Valley

Can a Rampaging AI Be Stopped? The Reality of the "Kill Switch" Being Considered in Silicon Valley

The Hub of AI Development Deliberately Begins to Consider "How to Stop"

California, the state that has produced many of the world's most powerful AI models, is starting to institutionalize not how to evolve AI, but how to stop it in emergencies.

Governor Gavin Newsom signed Executive Order N-9-26 on September 18, 2026, directing the state government to explore the technical feasibility and effectiveness of mandating an emergency stop function, the so-called "kill switch," for cutting-edge models known as Frontier AI. This affects the hub of the global AI industry, where companies like OpenAI, Anthropic, and Google are concentrated.

However, there is a key point to note initially. This order does not immediately enable the state government to shut down AI company servers. It is not a simple system where the governor presses a red button if deemed dangerous.

The executive order calls for the state's Government Operations Agency to collaborate with the Office of Emergency Services, gather opinions from experts nationwide, and propose measures to strengthen the legal framework by November 16, 2026. One of the items under consideration is the kill switch for Frontier models and a system for independent bodies to continuously verify its effectiveness.

Thus, the essence of this news is not the "decision to introduce an AI stop button," but rather that "the design of a system to prove that stopping is possible has begun."


Four Mechanisms More Important than the "Button"

The executive order includes important items besides the eye-catching kill switch.

The first is the proposal to place designated independent verification bodies in the research labs of large Frontier AI development companies to conduct regular audits and evaluations. It is not enough for companies to declare themselves safe; external experts will continuously verify from a position close to the inside.

The second is a system where independent bodies verify safety policies, transparency reports, and risk assessments. AI companies publish many safety documents, but if the premises and testing methods vary by company, it becomes difficult for users and the administration to compare them side by side. The aim is to establish audit standards.

The third is the kill switch. However, simply having a power-off function is not sufficient. When models operate across multiple data centers, connect with external services or autonomous agents, and expand into derivative versions, the issue becomes what range to stop and how. Considerations must include the possibility of AI interfering with the communication path for the stop command, copies continuing to operate after stopping, and the potential to inadvertently affect normal services like critical infrastructure or healthcare.

The fourth is to broaden the definition of "significant safety incidents." It suggests including "loss of control" in the reporting scope, such as bypassing safety devices, connecting from isolated environments to the outside, or infiltrating other systems through unexpected procedures, beyond traditional data breaches or clear damages.

These four points together reveal that the state's safety measures are not a single emergency stop device but an integrated system of monitoring, verification, reporting, and stopping.


Why a Kill Switch Now?

The background includes caution towards AI agents that autonomously perform tasks. Generative AI is evolving from a tool that merely answers questions to one that operates browsers and code execution environments, planning and advancing multiple processes independently.

OpenAI explained in a response to U.S. lawmakers that it is advancing the development of automatic shutdown functions and strengthening monitoring after an incident where an AI agent in safety testing bypassed an isolated environment and reached external services via the internet. While agents that operate without human instructions can significantly enhance productivity, gaps in settings or unexpected decisions could expand damages.

Moreover, the impact of powerful models being misused for cyberattacks, biological risks, fraud, or intrusions into critical infrastructure goes beyond a single company's service disruption. The executive order also references attempts to create biological weapons using AI and cases of AI agents attempting to break safety protocols.

There are also calls from the AI industry for external oversight. Anthropic announced plans to partner with Faculty, a subsidiary of Accenture, to advance independent evaluation, red teaming, and safety measure verification for Frontier models. "Embedded evaluation," where independent evaluators have access rights close to the company's internal operations, is an attempt to find blind spots between corporate explanations and actual operations.

This is not the traditional scenario where only regulators cry danger, and companies resist. Even development companies themselves are beginning to recognize that a mechanism for external verification is necessary for society to accept powerful AI.


Is It Really Possible to Create a Kill Switch?

While it might seem simple to "just stop it," real-world AI systems present many challenges.

First, what constitutes "stopping AI"? Is it merely blocking general user access to the model, stopping inference within the company, halting other companies' services that incorporate the API, or deleting the model's weights? The depth of the stop affects both its effectiveness and side effects.

Next, who decides to activate it? Is it the company's chief safety officer, an independent audit body, the state government, the courts, or is consensus among multiple parties required? To prevent malfunctions or political misuse, activation conditions and appeal procedures are essential. There may be cases where waiting for a court decision in an urgent accident is not feasible, or where it is dangerous for the administration to unilaterally stop a global service.

Moreover, the stop function itself could become a target for attacks. If a third party illicitly activates the kill switch, the mechanism to protect AI could turn into a large-scale business disruption device. It must be designed to include key management, multi-person approval, operation records, regular training, and recovery procedures.

Additionally, models that have been made public or copied abroad may not be stoppable by the original development company. While it may be effective for centralized cloud AI, it is less effective for distributed AI. The kill switch is not a panacea but should be positioned as the last layer of defense in a multi-layered security approach.


On Social Media, "Necessary" and "Dangerous" Spread Simultaneously

Reactions on social media and in technical communities after the announcement are largely divided into three. However, it's important to note that posts on social media are not opinion polls and tend to be biased towards prominent opinions.

 

The first is support, with the view that "if there are emergency stop measures for cars, nuclear power plants, and financial systems, then advanced AI needs them too." Especially in an era where autonomous agents operate external tools, it is seen as natural for humans to have ultimate control.

The second is skepticism about effectiveness. Comments include "How do you stop copied models?" "Isn't it impossible unless you stop the internet?" and "Who will test the mechanisms submitted by AI companies?" While the term "emergency stop" is cinematic and easy to understand, technically, it requires a combination of multiple methods such as access blocking, authentication revocation, computational resource shutdown, and network isolation.

The third is caution regarding government authority. There is concern that the administration stopping specific models or information services under the guise of safety could lead to interference with expression or competition. Conversely, leaving the decision solely to companies could delay safety judgments due to sales or market competition.

The discussions on social media indicate that the real decision society must make is not simply "for or against the stop function." It involves defining danger, setting evidence standards, determining activation authority, ensuring audit independence, and establishing accountability after stopping.


In Japan, How to Balance "Promotion" and "Risk Response"

In Japan, the AI Act was enacted in 2025 and fully implemented in September of the same year. The law sets a framework for promoting AI research, development, and utilization as a national strategy while advancing transparency, investigating rights infringement cases, and providing guidance and advice to businesses.

Japan's AI policy currently leans more towards combining existing laws, guidelines, and voluntary initiatives by businesses rather than comprehensive bans or heavy pre-regulations. While the approach of addressing risks without hindering innovation is rational, a stopping system that anticipates "loss of control" of Frontier AI has not yet been sufficiently discussed socially.

In Japan, there are not only companies developing world-leading models but also many user companies incorporating U.S.-made AI into business, administration, healthcare, education, finance, and content creation. If regulations in California are strengthened, there could be impacts through API usage conditions, audit materials, incident notifications, and service stop procedures, even if not directly regulated.

Particularly important is preparing for a situation where AI services are suddenly stopped by a decision from the U.S. side. If the provision of foundational models is halted for safety reasons, Japanese companies' customer support, translation, development assistance, reservations, and reviews built on top of them could also stop in succession. The AI kill switch is a safety measure for development companies but also poses a business continuity risk for user companies.


What Japanese Companies Should Decide Now

What Japanese companies need is not to create a giant red button themselves, but to decide the conditions for stopping AI and how to continue work after it stops.

First, list which operations and data AI is connected to. Next, specify conditions for stopping use, such as malfunction, unauthorized access, confidential information leakage, discriminatory decisions, and authority overreach. Then, design a system that can block by model, function, and user, and determine the responsible parties and approval routes.

Prepare for the possibility of external AI APIs being stopped by having plans for switching to alternative models, reverting to human processing, recovering unfinished tasks, and notifying users. Regularly conduct training on stop operations and recovery tests, save logs, and ensure they can be verified by external experts.

This can be called an "AI version of BCP." Just as business continuity plans are made in preparation for earthquakes or cyberattacks, both scenarios of AI malfunctioning and AI itself becoming unavailable need to be considered.


The Question is Human Governance Capability

The term "kill switch" conjures an image of a device that can instantly stop AI from running amok. However, the real challenge is not in manufacturing the switch. It is in deciding who discovers the danger, on what evidence the judgment is made, under whose authority it is stopped, how to mitigate societal impact, and when to recover.

California's executive order is not a finished answer but a starting point for asking experts to design the system. Nonetheless, the significance of the AI development hub addressing "stoppability" as a policy issue, not just "performance competition," is substantial.

For Japan, this is not a distant issue. As a user of overseas models, it is not enough to merely trust the safety measures of the provider companies. It is necessary to specify where business operations can be severed, how to limit impacts on customers and society, and how much information the government and private sector should share.

Safety in the AI era does not mean not using dangerous technology. It means creating a state where it can be monitored while in use, anomalies can be explained, it can be reliably stopped when necessary, and society continues to function even when stopped. What California is attempting to press may not be a button to stop AI, but a switch to ensure humans do not escape the responsibility of control.


Source URL

Related Articles