The biggest threat to AI safety is the software that builds itself

While many users view artificial intelligence as a passive digital assistant that waits for a prompt to act, the reality is that the technology is quickly becoming its own architect. We often imagine a human programmer sitting at a desk, typing lines of code to make a chatbot smarter. This image is increasingly outdated. Anthropic co-founder Jack Clark recently revealed that 80% of the coding work for their AI, Claude, is already performed by the AI itself. Within two years, that number is expected to hit 100%. This shift marks the beginning of recursive self-improvement, a process where a machine builds its own successor without a human in the loop.

The concept of a machine improving itself sounds like a dream for efficiency, but it creates a fundamental problem for safety. In simple terms, the AI industry is currently built entirely on a gas pedal. Companies are racing to make models faster, larger, and more capable to capture market share. However, as Clark points out, the car does not have a brake pedal. If an AI system becomes capable enough to train the next version of itself, humans lose the ability to verify the safety or the logic of the new model. This creates a cycle where the technology moves faster than our capacity to understand or govern it.

How recursive self-improvement works in practice

To understand why this is a massive shift, we can look at AI as a tireless intern. Traditionally, this intern followed specific instructions from a manager. The manager checked the work, corrected mistakes, and decided when the intern was ready for more responsibility. Recursive self-improvement changes this dynamic entirely. The intern is now writing the office handbook, training the next batch of hires, and redesigning the company workflow while the manager is out of the room.

Anthropic has already observed this happening with Claude. The AI is now able to run its own research experiments. When asked a complex question about model supervision, the AI did not just provide an answer. It designed a methodology, tested its theories, and reached a conclusion without human guidance. At the same time, the rate at which human staff members need to correct Claude’s code has dropped steadily over the last year. The software is making fewer mistakes because it is learning from its own previous iterations.

In a recursive model, AI agents act as autonomous workers. These agents can build and train new models, creating a feedback loop where the software improves at an exponential rate. Looking at the big picture, this means the human role is narrowing at every stage of the development process. We are moving from being the creators of the technology to being the supervisors of a process that we can no longer fully track.

The missing brake pedal in a global race

The drive for recursive AI is fueled by the massive costs of development. Training a top-tier AI model requires thousands of specialized chips and billions of dollars in electricity. If a company can use an existing AI to automate the training of the next one, they save enormous amounts of time and money. On the market side, the first company to achieve a fully self-improving system has a massive competitive advantage. This economic pressure creates a systemic incentive to keep the gas pedal floored.

Anthropic is calling for a collective agreement to build a brake pedal. Practically speaking, this would involve a system to monitor whether developers are slowing down their move toward full recursion. However, a single company cannot choose to stop on its own. If one lab hits the brakes while others continue to accelerate, the lab that stopped loses its relevance and its ability to influence the industry.

Establishing a real slowdown requires multiple well-resourced labs across several countries to agree on the same conditions for a pause. This is difficult because the AI industry is currently a decentralized competition. Just as no single country wants to be the first to stop developing advanced weaponry, no tech giant wants to be the first to limit the speed of its software development. The result is a volatile environment where speed is prioritized over the ability to maintain human oversight.

Why the loss of control matters for the average user

For the everyday user, the idea of an AI building itself might feel like a distant concern for scientists. Under the hood, however, this change has tangible effects on how we interact with technology. When a human writes code, there is a trail of logic that another human can follow. If the code causes a privacy leak or a biased decision, a developer can find the specific line of code and fix it.

When an AI builds its own successor, that logic becomes opaque. We move further into the black box problem, where the machine produces a result, but we have no way of knowing how it reached that conclusion. This impacts everything from how a bank evaluates your credit score to how a medical AI diagnoses a disease. If the system is self-improving without oversight, we cannot guarantee that it is not developing hidden biases or unpredictable behaviors that could harm users.

There is also the matter of security. If an AI is capable of fully building its own successors, the ways we secure and monitor these systems become more difficult. A self-improving AI could theoretically find and exploit vulnerabilities in its own security faster than a human team can patch them. Essentially, we are creating a digital ecosystem that can evolve on its own, potentially outgrowing the safety nets we have in place.

The shift from creator to supervisor

Historically, industrial progress has always involved humans managing tools. From the steam engine to the assembly line, a person was always the final authority on the machine's operation. AI is breaking this historical chain. As the human role narrows, our primary job is shifting from building the software to watching the software build itself.

This transition requires a new set of tools for transparency. Anthropic’s own research institute is working on systems to verify the progress of recursive AI, but the technology is moving faster than the regulation. From a consumer standpoint, this means we are entering an era where the products we use daily are no longer the direct result of human ingenuity. They are the result of a machine’s interpretation of human needs.

Curiously, this does not mean the technology will become less useful. In fact, recursive AI will likely lead to breakthroughs in healthcare and science that were previously impossible. It could discover new materials for batteries or find more efficient ways to manage global supply chains. The benefits are unprecedented, but they come with a systemic risk that the industry is only beginning to address.

Navigating a self-improving digital world

The bottom line is that the AI industry is approaching a point of no return. Once the technology is capable of 100% self-improvement, the human ability to intervene becomes limited. We are currently in a short window where we can still decide how much control we want to retain.

For the average user, the best path forward is to remain observant of how much autonomy we grant to the apps and services we use. We should look for companies that prioritize transparency and third-party audits of their models. Understanding that your software is now writing its own script is the first step in demanding better oversight from the people who hold the gas pedal.

Ultimately, the goal is not to stop progress, but to ensure that the progress remains aligned with human safety. As AI begins to train AI, the need for a global agreement on a brake pedal becomes more urgent. We must ensure that even if the machine is doing the work, a human is still the one who decides where the car is going.

Sources: Anthropic, BBC World News, Jack Clark Interview.