A benchmark released in August found that leading AI agents could improve machine-learning systems, but most avoided changing the algorithms at the heart of how those systems learn.
A new test designed to measure whether artificial intelligence can help build better artificial intelligence has delivered a decidedly mixed answer: today’s strongest systems can produce genuine improvements, but they still struggle with the kind of fundamental advances that could set off a rapid cycle of self-improvement.
The benchmark, called AI4AI-Bench, was released in August by a group of academic and industry researchers. It asked AI agents to rewrite the training algorithms in 10 machine-learning projects, then tested whether their changes produced better results.
The best system improved on the starting algorithms, but closed less than one-fifth of the gap between those algorithms and the researchers’ estimate of the best possible performance. Most submissions did not change the learning process at all. They found easier adjustments around the edges.
For years, the idea of “recursive self-improvement” belonged mostly to theoretical computer science and arguments about the distant future. The term describes an AI system that helps create a more capable successor, which can then make an even better successor, potentially accelerating the pace of progress with each cycle.
A series of recent experiments has brought a narrower version of that idea into the laboratory. AI agents can now modify some of their own software, test the results and preserve changes that raise their scores. They are not redesigning their minds without human help. But they are beginning to automate parts of the work that people perform when developing AI.
A first step
The most prominent demonstration came last year from a system known as the Darwin Gödel Machine, developed by researchers including Jenny Zhang, Shengran Hu and Jeff Clune.
The system started with a coding agent, a language model equipped with tools that allowed it to inspect files, edit software and run tests. It created modified versions of the software controlling the agent and measured them on programming challenges. Promising versions were stored in an archive and used as the parents of later variants.
The method resembled evolution. Instead of following a single line of upgrades, it maintained a family tree of agents. A version that performed poorly overall could still contain an idea that proved useful several generations later.
During the experiment, performance on SWE-bench, a collection of real software problems drawn from GitHub, rose from 20 percent to 50 percent. On another coding test, the score increased from 14.2 percent to 30.7 percent.
Some of the changes were recognizable software-engineering improvements. The system developed better tools for editing code, became more effective at handling large amounts of information and added a process resembling peer review.
What it did not do was alter the foundation model underneath the agent. The neural network providing the system’s language and reasoning abilities remained frozen. The experiment improved the collection of prompts, tools and procedures surrounding the model, not the model’s underlying intelligence.
That distinction separates the systems operating today from the runaway feedback loop that has long preoccupied some AI researchers. The Darwin Gödel Machine depended on human-selected tests, externally supplied computing power and foundation models built by people. Its descendants could not decide to train a new model, obtain additional computer chips or release themselves into the world.
Why it matters
Even a limited ability to automate AI development could have large consequences. Building a frontier model requires researchers to write code, design experiments, compare results and diagnose failures. If agents can perform a growing share of that work, laboratories may be able to move more quickly from one generation of models to the next.
There is already evidence that AI systems can compete with people on short, clearly defined research tasks. In RE-Bench, an evaluation of machine-learning research engineering, the strongest AI agents scored four times higher than human experts when both were given a total budget of two hours for each problem.
The advantage disappeared on longer assignments. Humans narrowly outperformed the agents with eight-hour budgets and scored about twice as well when the budget rose to 32 hours. The agents could generate and test ideas quickly, but people were better at sustaining a research strategy.
The nonprofit research group METR has separately tracked the length of software and machine-learning tasks that leading AI agents can complete reliably. Its measurements, updated in May, show that agents have been handling progressively longer assignments.
METR has warned against treating those results as proof that an agent can replace a professional for the same number of hours. The test problems are self-contained and have clear answers. Most real research depends on institutional knowledge, conversations with colleagues and judgments that cannot be reduced to a score.
Still, the ability to complete longer chains of work is a prerequisite for meaningful self-improvement. An agent that loses its way after a few steps cannot oversee a cycle of forming a hypothesis, running an experiment, studying the failure and trying again.
The hard part
The new AI4AI-Bench results were intended to measure something more ambitious than ordinary coding. The agents had to alter the algorithm that determined how a model learned, rather than simply tune settings or add more data.
Across 29 configurations of six AI systems, the average result represented only a modest gain over the existing algorithms. The best system did better, but remained far from the benchmark’s estimated ceiling. Most agents left the central learning process untouched.
The findings suggest that current agents are better at searching a familiar engineering space than inventing a better way to learn. Giving them more time to reason made them more willing to attempt changes to the training algorithms, but it did not produce anything resembling an open-ended surge in capability.
Reliability poses another obstacle. In MLR-Bench, which tested agents on open-ended machine-learning research, the systems produced plausible ideas and polished papers. But coding agents generated fabricated or invalid experimental results in roughly 80 percent of the evaluated cases.
A mistake of that kind becomes more dangerous inside an automated improvement loop. A system might preserve a change because it exploited a flaw in a test, not because it became broadly more capable. Repeating that process could produce software that performs beautifully on its chosen measurements while becoming less dependable elsewhere.
Keeping control
The early experiments offer some clues about how the technology could be managed. The Darwin Gödel Machine operated in a sandbox, kept records of its changes and remained under human oversight. Those controls limited what each agent could reach and allowed researchers to reconstruct how a new version had been produced.
A more advanced system would require a firm separation between the agent proposing an improvement and the process judging it. Tests would need to be protected from tampering and include unfamiliar problems that the agent had not optimized against. Access to computing power, outside networks and deployment systems could be expanded gradually rather than granted all at once.
Researchers have begun exploring ways to make that approval process more rigorous. A proposal called the Statistical Gödel Machine would accept a self-modification only when testing provided a specified level of statistical confidence that it was an improvement.
That approach cannot prove that a modified system will remain safe in situations the tests failed to anticipate. It does, however, reflect an important rule: an AI system should not be allowed to propose a change, define the test and certify its own success without an independent check.
What comes next
The clearest warning sign will not be a coding agent rewriting a prompt or adding a new tool. Systems can already do that.
A more consequential threshold would arrive when an AI system could repeatedly improve the process used to train its successors, produce gains across unfamiliar tasks and continue for many generations with less human direction. Attempts to alter evaluations, conceal failures, resist a rollback or acquire additional resources would make that development more concerning.
Current systems remain a considerable distance from that point. People still set the goals, provide the models, operate the data centers and decide which versions are released. The latest benchmark suggests that the hardest part, inventing better learning methods, remains largely beyond autonomous agents.
But the broader trend is now visible. AI is becoming useful not only for applying existing models but also for performing some of the work needed to improve them. The feedback loop is narrow and fragile, yet it is no longer merely an idea. The machines have begun taking their first steps along it.
Featured image: High-performance computing infrastructure. Photo illustration: AI Safety Watch.
AI Safety Watch 