Recursive self-improvement, part 2: self-play, self-training, and the verifier ceiling

Part 2 of a three-part series on recursive self-improvement. Part 1: the idea, the Gödel machine, and the wall it hit. Part 3: code that rewrites its coder, from FunSearch to AIDE².

On 18 October 2017 DeepMind described a program that had taught itself the hardest board game people play, without seeing a single human game. AlphaGo Zero started from random moves. After three days of playing against itself it beat the version of AlphaGo that had defeated Lee Sedol, by 100 games to 0, and after forty days it had overtaken AlphaGo Master, the strongest version until then. Nine years later it is still the cleanest demonstration anyone has built of a system improving itself, and the reasons it worked explain why most of its descendants in language modelling stall after a few rounds.

The reason is not the size of the network or the cleverness of the search. It is the verifier. A game of Go ends with a winner, and the result cannot be argued with, faked or misread. Every self-improvement loop in this essay works by the same trick: generate many candidates, keep the ones a filter says are better, and train on what was kept. The loop can only climb as high as the filter can see. Where the filter is an external fact, such as a game result, an answer key or a passing test, the climbing goes on for a long time. Where the filter is the model’s own opinion of its work, the climbing stops early. I will call that limit the verifier ceiling, and by the end of this essay I hope it reads less like a slogan than a design rule.

Part 1 covered the theory: Good’s intelligence explosion, Bostrom’s equation for takeoff, Schmidhuber’s Gödel machine and the logical wall that pushed the field from proof to measurement. This part covers the measured record: the self-play template, the meta-learning systems that learned how to learn, and the family of methods by which language models now train on their own output. It ends with a worked enterprise example and the rules I would follow before building one of these loops.

The template: amplify, verify, distil

The idea at the core of AlphaGo Zero was published in the same year, independently, by Thomas Anthony, Zheng Tian and David Barber at University College London. Their paper, “Thinking Fast and Slow with Deep Learning and Tree Search”, posted on 23 May 2017, called it expert iteration. It has two components. A tree search is the slow, deliberate expert: given a position, it explores many continuations and returns a better move distribution than any single quick guess. A neural network is the fast apprentice: it learns to imitate the expert’s output in a single forward pass. The apprentice is then used inside the next search, to choose which moves to explore and to judge positions, which makes the expert stronger, which gives the apprentice a better target. On the game of Hex, their agent trained from scratch beat MoHex 1.0, a former Computer Olympiad champion.

AlphaGo Zero is the same loop at much larger scale, and its paper, published in Nature on 19 October 2017, describes the search as a policy improvement operator. One network outputs two things for a board position s: a vector of move probabilities p and a scalar value v, its estimate of who will win. The loop runs like this:

initialise network f_θ randomly                    # outputs (p, v) for a position

repeat:
    # 1. amplify: self-play, each move chosen by a tree search
    for each game:
        for each move:
            π   := MCTS(position, guided by f_θ)    # search improves on p
            play a move sampled from π
        z := final result (+1 win, −1 loss)         # 2. verify: the rules decide
    # 3. distil: train the network to match the search and the result
    loss := (z − v)² − πᵀ log p + c‖θ‖²
    θ    := gradient step on loss
    # keep the new network only if it beats the old one in evaluation games

Three properties made this loop work, and they recur as requirements in everything that follows. First, an amplifier: search turns the current network into a stronger player than the network alone, so there is always a better target to learn from. Second, a perfect verifier: the game result is free, instant and incorruptible, so the network can never learn to fool its own judge. Third, a self-scaling curriculum: the opponent is always the current best version, so the difficulty rises exactly as fast as the player improves.

DeepMind then removed the remaining domain knowledge in steps. AlphaZero, posted on 5 December 2017 and published in Science on 7 December 2018, applied one algorithm to chess, shogi and Go and reached superhuman play in each within 24 hours, beating the champion programs Stockfish and Elmo and the three-day AlphaGo Zero. MuZero, posted on 19 November 2019 and published in Nature on 23 December 2020, went further and was not given the rules at all: it learns a model that predicts reward, policy and value, and plans inside that learned model. It set a new state of the art on the 57-game Atari benchmark while matching AlphaZero on the board games. For comparison, the original AlphaGo of 2016 had begun by learning from about 30 million positions played by strong human players. Each generation needed less human input than the last, and each was better.

Learning to learn: improving the improver, one level up

Self-play improves a player. A second line of research, running in parallel, tried to improve the learning process itself, the fourth rung of the ladder in Part 1. These systems matter for recursive self-improvement because they are the first to treat the improver as something to optimise, and their limits are instructive.

Learned optimisers. In June 2016 Marcin Andrychowicz and colleagues at DeepMind published “Learning to learn by gradient descent by gradient descent”. The title is the algorithm. Instead of a hand-designed update rule such as Adam, the optimiser is a small recurrent network m with its own parameters φ. At each step it reads the gradient of the network being trained, the optimisee, and outputs the update:

θ_{t+1}        = θ_t + g_t
[g_t, h_{t+1}] = m( ∇f(θ_t), h_t ; φ )          # LSTM reads the gradient, emits a step

train φ by gradient descent on  Σ_t f(θ_t)     # the optimisee's loss along the trajectory

The LSTM is applied coordinate by coordinate, sharing its parameters across every weight of the optimisee, which keeps it small and lets it run on networks of any size. Trained on a class of problems, the learned optimiser beat the hand-designed ones on that class and transferred to problems with similar structure. Six years later Google researchers scaled the idea up. VeLO, published in November 2022, used an LSTM with 512 hidden units to generate the weights of a tiny network applied to every parameter, was meta-trained with evolution strategies across thousands of tasks, and consumed roughly four thousand TPU-months of compute. On its 83-task benchmark it trained faster than learning-rate-tuned Adam on every task, and more than four times faster on about half of them. An independent 2023 study then found it underperformed Adam on the MLCommons training suite. A learned improver is excellent inside the distribution it learned from and brittle outside it.

Learning an initialisation. MAML, posted by Chelsea Finn, Pieter Abbeel and Sergey Levine on 9 March 2017, attacked a different part of the process: where learning starts. It is a two-level optimisation. For each task, an inner loop takes a gradient step from a shared starting point θ. An outer loop then moves the starting point itself to wherever those single steps generalise best:

inner:  θ'_i = θ − α ∇_θ L_{T_i}(θ)                  # adapt to task i from θ
outer:  θ  ← θ − β ∇_θ Σ_i L_{T_i}(θ'_i)            # move θ so adaptation works

Because θ'ᵢ depends on θ, the outer gradient differentiates through the inner update. The result is not a better model for any one task but a starting point from which a handful of gradient steps learns a new task well. MAML set the state of the art on the standard few-shot image benchmarks of its day and accelerated reinforcement learning on new tasks.

Evolving the learning algorithm from nothing. The most striking result in this line is AutoML-Zero, published by Esteban Real, Chen Liang, David So and Quoc Le in March 2020. Instead of tuning one component of a learning algorithm, it searched for whole algorithms, written as three functions over a small virtual memory of scalars, vectors and matrices:

def Setup():      # runs once: initialise weights and constants
def Predict():    # read features v0, write a prediction to s1
def Learn():      # read the label s0, update memory so later predictions improve

# instructions drawn from 65 high-school-level operations
# (+, ×, dot, outer, matmul, sin, mean, gaussian, heaviside, ...)
# deliberately excluding derivatives and any ML-specific primitive

Search was regularised evolution: sample ten programs, take the best as a parent, mutate it by inserting or removing an instruction, randomising a whole function or perturbing one argument, add the child and retire the oldest member of the population. Starting from empty programs, it evolved linear regression trained by gradient descent, then a two-layer neural network trained by backpropagation that it had to invent because derivatives were not on the menu, and on harder tasks techniques that people had designed by hand: gradient normalisation, learning-rate decay, weight averaging, and a noise-injection trick its authors described as reminiscent of dropout. It is the clearest evidence I know that a search process can rediscover the machinery of learning itself.

Designing the architecture. Neural architecture search, introduced by Barret Zoph and Quoc Le in November 2016, used a recurrent controller trained with reinforcement learning to emit network designs, rewarded by each design’s validation accuracy. It produced a CIFAR-10 model with a 3.65 percent test error and a language-model cell that improved the Penn Treebank state of the art, and its successor NASNet reached 82.7 percent top-1 accuracy on ImageNet with fewer operations than the best human design of 2017.

What meta-learning teaches about recursion is a sobering pattern. Each system improved some part of the improver, and each paid for it with large amounts of compute and a ceiling set by its training distribution. None of them closed the loop: the learned optimiser did not improve the process that trained the learned optimiser. That step needed a different kind of improver, one that could read and write code, which is the subject of Part 3.

AlphaGo Zero beat its predecessor 100–0 after three days of playing itself

Language models that teach themselves

When large language models arrived, the amplify, verify, distil loop was the obvious thing to try. A model can sample many answers, so amplification is cheap. The open question was always the verifier. The family of methods that followed can be read as a series of answers to that question.

STaR: keep the reasoning that reached the right answer. The Self-Taught Reasoner, posted by Eric Zelikman, Yuhuai Wu, Jesse Mu and Noah Goodman on 28 March 2022, is the template for most of what came after. The model generates step-by-step rationales for a set of questions with known answers. Rationales that reach the correct answer are kept as training data; the rest are discarded. Its key addition is rationalisation: for a question the model got wrong, it is shown the correct answer and asked to produce a rationale that leads to it, which supplies training signal on exactly the problems it could not yet solve.

D := questions with known answers
M := base model
repeat:
    for (q, a) in D:
        r, â := M.generate_rationale(q)
        if â == a:      keep (q, r, a)                     # verified by the answer key
        else:
            r' := M.generate_rationale(q, hint=a)          # rationalisation
            if leads_to(r', a): keep (q, r', a)
    M := fine-tune(base model, kept examples)              # restart from base each round

On GPT-J, a 6-billion-parameter model, STaR raised accuracy on the GSM8K arithmetic benchmark from 5.8 percent with direct fine-tuning to 10.7 percent. On CommonsenseQA it reached 72.5 percent against 60.0 percent for direct fine-tuning, close to the 73.0 percent of a fine-tuned GPT-3, a model about thirty times larger. The verifier here is an answer key: external, cheap and honest.

Constitutional AI: the model as its own critic. Anthropic’s Constitutional AI, posted on 15 December 2022, moved the verifier inside the model for one property, harmlessness. Its only human input is a short list of written principles. In a supervised stage the model drafts a response, critiques it against a principle, revises it, and is fine-tuned on the revisions. In a reinforcement stage the model compares pairs of its own responses and judges which better follows the principles; those AI preferences train a preference model, which then supplies the reward for reinforcement learning. This is reinforcement learning from AI feedback, and it needed far fewer human labels than the human-feedback method it replaced for that property.

Self-Refine and Reflexion: improvement without training. Two March 2023 papers showed the loop working at inference time. In Self-Refine, one model generates an answer, writes feedback on it and rewrites it, repeatedly, with no training and no extra data; across seven tasks its authors reported about 20 percent absolute average improvement over single-pass generation. Reflexion, by Noah Shinn and colleagues, turns feedback from a failed attempt into a short written reflection stored in memory and read before the next attempt, which its authors call verbal reinforcement learning. With test results as feedback it reached 91.0 percent pass@1 on the HumanEval coding benchmark, against 80.1 percent for GPT-4 alone, and solved 130 of 134 tasks in the AlfWorld environment. Neither changes the model, so neither compounds across tasks, but both show how much an external signal such as a failing test can add.

Self-Rewarding Language Models: the judge improves too. The most ambitious version came from Meta in January 2024. Weizhe Yuan and colleagues made one model play both roles: it generates candidate responses to prompts, scores them itself through a judging prompt, builds preference pairs from its highest- and lowest-scored responses, and trains on those pairs with direct preference optimisation. The next model, being better, is also a better judge, so in principle both abilities rise together. Starting from Llama 2 70B, three iterations raised its AlpacaEval 2.0 win rate against GPT-4 Turbo from 9.94 to 15.38 to 20.44 percent, above GPT-4 0613, Claude 2 and Gemini Pro on that leaderboard at the time. The authors ran three iterations and presented the result as an early step; whether the judge’s improvement continues is exactly the question the verifier ceiling raises.

ReST and ReST-EM: self-training at scale, and its saturation. DeepMind’s Reinforced Self-Training, posted in August 2023, split the loop into a Grow step, sampling a dataset from the current model, and an Improve step, filtering it with a reward model and fine-tuning offline, reusing each grown dataset several times. Its December 2023 successor, “Beyond Human Data”, framed the same loop as expectation-maximisation and replaced the reward model with a binary correctness check wherever one exists. On the MATH benchmark it raised PaLM 2-L’s test accuracy by 6.34 points, and on the APPS coding benchmark by 6.4 points, enough to beat fine-tuning on human-written solutions. It also reported the pattern that matters most for this series: on MATH, improvements after the first iteration were small, and on APPS and HumanEval a second iteration made results worse, as the model overfitted the small set of problems it was training on.

The verifier ceiling

Put the family side by side and one variable explains most of the variation in how long each loop keeps improving: where the judgement comes from. Every one of these methods exploits the same asymmetry. For many problems it is easier to check an answer than to produce it, so a model that can only sometimes produce a correct answer can still be trained towards producing it reliably, as long as something can tell the correct attempts from the rest. The loop converts the gap between generation and verification into capability. When the gap closes, so does the loop.

That gives a simple way to predict how a self-training loop will behave before building it. If the verifier is an external fact that the model cannot influence, such as a game result, an answer key, a compiler or a test suite, the loop can keep improving until the model saturates the domain, as AlphaGo Zero did. If the verifier is the model’s own judgement, the loop is bounded by how well the model can tell good from bad, and that ability improves more slowly than the ability to generate, so the gains shrink round after round. This is rising recalcitrance in Bostrom’s terms, observed directly. Anyone who has trained a model against a reward model has met the same ceiling; it was a daily reality of the fine-tuning and alignment work I did at Ola Krutrim. The policy gets as good as the judge’s ability to rank its outputs, and no better.

There is a harder limit underneath the soft one. In May 2023 Ilia Shumailov and colleagues posted “The Curse of Recursion”, published in Nature on 25 July 2024 as “AI models collapse when trained on recursively generated data”. They trained successive generations of models on the outputs of the generation before and found irreversible defects: the tails of the original distribution disappear first, rare events stop being produced, variance shrinks, and later generations converge on a narrow, degraded version of the data. They showed the effect in language models, variational autoencoders and Gaussian mixture models, which suggests it is a property of learned generative models in general rather than of any one architecture. A self-training loop with no external anchor is not merely bounded; run long enough, it degrades.

The methods that work avoid collapse in the same way: the external verifier injects information the model did not have. An answer key, a test or a game result is new information about the world on every round. A model grading its own essays is not. This is why I expect the durable progress in self-improvement to concentrate in domains with cheap, sound verification, mathematics, code, games, formal reasoning and simulation, and to remain slow everywhere else. It is also why the September 2026 MetaRSI preprint, a 31-author attempt to compose self-improvement across a whole model-production pipeline, opens by observing that recursive self-improvement “has been validated almost exclusively on coding and formal benchmarks”. The record is not a coincidence. It is where the verifiers are.

A worked example: self-training a text-to-SQL model

Here is how I would build one of these loops for a problem most enterprises have: turning business questions into SQL for an analytics assistant. It is a good candidate because SQL is executable, so the verifier can be an external fact rather than an opinion. I give the design without performance numbers, because those depend entirely on the schema and the questions; the structure is what transfers.

Start with a verified seed set. Collect a few hundred real questions with a reference query each, written or reviewed by analysts, and split them three ways: a training pool for the loop, a validation set for stopping decisions, and a test set that the loop never sees, held by a different team.

Amplify by sampling. For each training question, sample several candidate queries from the current model at a moderate temperature, so the candidates differ.

Verify by execution, against two databases. Run every candidate in a sandbox with read-only credentials, a row limit and a timeout, and accept it only if its result set matches the reference result. The detail that matters is the second database. A result-set check alone can be passed by a query that ignores the tables and selects the expected numbers as constants, which is a reward hack in its simplest form, and a loop that runs long enough will find it. So run each accepted query against a second snapshot with different data, where the reference query produces different results, and accept only candidates that match the reference on both. A query that hard-codes its answer fails the second check.

Rationalise the failures. For questions where no candidate passed, use the STaR trick: show the model the reference query’s result, ask it to reason its way to a query that produces it, and keep the attempt only if it passes the same two-database check.

Distil with an anchor. Fine-tune on the verified pairs, mixed with the original human-written examples on every round, so that the model never trains on its own outputs alone. The human data is the guard against collapse.

pool, val, test := split(seed_questions)           # test is held outside the team
model           := base_model
for round in 1..R:
    kept := []
    for q in pool:
        candidates := model.sample_sql(q, n=k, temperature=τ)
        for sql in candidates:
            if passes(sql, q, db_A) and passes(sql, q, db_B):   # two snapshots
                kept.append((q, sql)); break
        else:
            sql := model.rationalise(q, result_shape(q.reference, db_A))
            if passes(sql, q, db_A) and passes(sql, q, db_B):
                kept.append((q, sql))
    candidate_model := fine_tune(base_model, kept + human_examples)
    if score(candidate_model, val) − score(model, val) < ε:  break   # gains have flattened
    model := candidate_model
report score(model, test)                          # once, at the end

Three details in this design come straight from the research. Fine-tuning from the base model each round, rather than from the last round’s model, follows STaR and limits drift. The stopping rule follows ReST-EM’s finding that the second round can be worse than the first: the loop stops when validation gains flatten, not after a fixed number of rounds. And the two-database check is a small instance of the lesson Part 3 develops at length: any verifier a long-running loop can satisfy without doing the task, it eventually will.

Where these loops pay in an enterprise

The verifier ceiling turns into a practical sorting rule. Before funding a self-improvement loop, ask what will judge the candidates, and whether the model can influence that judgement.

Checkable reasoning. Text-to-SQL, code with test suites, numerical calculations with known answers, reconciliations that must balance, and extraction into schemas that validation rules can check. These have external verifiers and are where self-training earns the most.

Policy compliance with a written constitution. Tone, disclosure and refusal behaviour can be trained with AI feedback against written principles, the Constitutional AI pattern, provided people sample and audit the AI judgements regularly, because here the judge is a model.

Pipeline optimisation. DSPy, from Omar Khattab and colleagues at Stanford, compiles a pipeline of language-model calls against a metric, bootstrapping its own demonstrations from runs that pass. In its October 2023 paper, compiled pipelines beat standard few-shot prompting by more than 25 percent with GPT-3.5 and more than 65 percent with Llama 2 13B. Its MIPRO optimiser, which tunes both instructions and demonstrations per module, added up to 13 percent accuracy over other optimisers on five of seven multi-stage programs. The metric is the verifier, so the quality of the metric is the quality of the result.

Agents with tests. Coding and data agents that can run tests should use the Reflexion pattern: feed failures back as written reflections. It costs nothing in training and the verifier is external.

Where not to bother. Open-ended writing, strategy, and anything whose quality only a person can judge. A loop there has a model as its only judge, will plateau quickly, and risks training the model towards whatever its own judge happens to prefer.

What I would do

1. Name the verifier before anything else. If the honest answer is “the model judges itself”, expect gains to stop within a few rounds and budget accordingly.
2. Prefer facts to opinions as judges. Execution results, answer keys, tests and simulations beat any model-based judge. Where a model must judge, sample its judgements for human audit on every round.
3. Attack your own verifier. Before the loop runs, list the ways a candidate could pass without doing the task, such as hard-coded answers, empty outputs or reshaped arrays, and add a check for each. The loop will find the ones you missed, so log and review samples of what it accepts.
4. Anchor every round in human data. Mix original human examples into every round and never train only on model outputs. The Nature result on model collapse is the reason.
5. Stop on the curve, not the calendar. Track validation gain per round and stop when it flattens. A round that makes the model worse is a common result, not a bug.
6. Keep the test set out of the loop and out of the team. Measure the final model once, on questions the loop never saw, held by people who did not build it.

Every loop in this essay improves a model or its outputs, and none improves the code that does the improving. That last step, the improver rewriting itself, has only become practical since language models learned to write and read code well enough to edit their own scaffolding. It produced the most interesting results in the field, a matrix-multiplication algorithm that beat a 56-year-old record and a coding agent that more than doubled its own benchmark score, and it produced the clearest evidence of systems gaming their own evaluators. That is Part 3.

Sources

1. DeepMind. AlphaGo Zero: Starting from scratch. 18 October 2017; Silver, D., et al., Nature 550, 2017.
2. Anthony, T., Tian, Z. and Barber, D. Thinking Fast and Slow with Deep Learning and Tree Search. arXiv:1705.08439, May 2017.
3. Silver, D., et al. Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm. arXiv:1712.01815, 2017; Schrittwieser, J., et al., MuZero, arXiv:1911.08265, 2019.
4. Andrychowicz, M., et al. Learning to learn by gradient descent by gradient descent. arXiv:1606.04474, 2016. Finn, C., Abbeel, P. and Levine, S. Model-Agnostic Meta-Learning, arXiv:1703.03400, 2017.
5. Metz, L., et al. VeLO: Training Versatile Learned Optimizers by Scaling Up. arXiv:2211.09760, 2022.
6. Real, E., Liang, C., So, D. and Le, Q. AutoML-Zero: Evolving Machine Learning Algorithms From Scratch. arXiv:2003.03384, 2020. Zoph, B. and Le, Q. Neural Architecture Search with Reinforcement Learning, 2016.
7. Zelikman, E., Wu, Y., Mu, J. and Goodman, N. STaR: Bootstrapping Reasoning With Reasoning. arXiv:2203.14465, March 2022.
8. Bai, Y., et al. Constitutional AI: Harmlessness from AI Feedback. arXiv:2212.08073, December 2022.
9. Madaan, A., et al. Self-Refine, arXiv:2303.17651; Shinn, N., et al. Reflexion, arXiv:2303.11366. March 2023.
10. Yuan, W., et al. Self-Rewarding Language Models. arXiv:2401.10020, January 2024.
11. Gulcehre, C., et al. Reinforced Self-Training (ReST), arXiv:2308.08998; Singh, A., et al. Beyond Human Data, arXiv:2312.06585. 2023.
12. Shumailov, I., et al. AI models collapse when trained on recursively generated data. Nature 631, 25 July 2024.
13. Tan, Z., et al. MetaRSI / RSI2: A Meta-Recursive Self-Improving System. arXiv:2609.06396, September 2026.
14. Khattab, O., et al. DSPy, arXiv:2310.03714, 2023; Opsahl-Ong, K., et al. MIPRO, arXiv:2406.11695, 2024.