Advances in language modeling have been driven by scaling pretraining on ever more data. Yet, the training data is still largely curated on the model’s behalf. A more general approach to pretraining would let the model learn to generate the data most useful for its own improvement. This would provide an effectively unbounded source of training data, limited by compute rather than human knowledge. We introduce Self-Play Pretraining with Zero Data, an initial proof-of-concept towards realizing this vision. Our procedure casts synthetic data generation as a search over the space of all computable structure, taking inspiration from Solomonoff induction. Starting from random initialization, two models learn in tandem: a generator proposes programs interpreted by a universal Turing machine, generating byte sequences, while a learner autoregressively predicts these byte sequences. The learner is trained with standard cross-entropy, while the generator is trained with reinforcement learning to produce sequences at the frontier of the learner’s capabilities, yielding an adaptive curriculum. A universal Turing machine gives us a search space over all computable data-generating processes, imposing little domain-specific structure, and self-play searches over this space for useful training data. We test whether zero-shot performance on natural data improves predictably with self-play compute; this is a clean test of transfer since neither generator nor learner is trained on natural data. Across several natural datasets, zero-shot loss exhibits predictable scaling in compute. The models also exhibit in-context learning, and discover recognizable mathematical sequences during training.

“By teaching, we learn.”
Seneca

“So much from so little, almost everything from almost nothing.”
John Archibald Wheeler

1 Introduction

Advances in language modeling have been driven by pretraining on Internet data. Yet, the training data is still largely curated and constructed on the model’s behalf through large-scale data curation efforts [Li et al., 2025, Penedo et al., 2023, Soldaini et al., 2024], data mixture design [Chen et al., 2026, Xie et al., 2023], and hand-designed synthetic generators [Gunasekar et al., 2023, Yang et al., 2025]. A more generic approach to pretraining would let the model itself learn to generate training data that is most useful for its own improvement. Such an approach would extend the Bitter Lesson [Sutton, 2019]Sutton (2019) to the training data itself, minimizing hand-engineered inductive bias and creating a self-

contained path to scaling, where compute alone can sustain continued improvement [Kim et al., 2026b, Silver and Sutton]as suggested by Kim and colleagues, and Silver and Sutton.

As an initial step towards this vision, we introduce a self-play algorithm for pretraining from zero data. Starting from random initialization, two autoregressive transformers co-evolve: a generator proposes programs for a minimal universal Turing machine whose execution produces byte sequences, while a learner is trained on those sequences via next-token prediction. We use a universal Turing machine to make the space of all synthetic training data as expressive as possible while imposing minimal domain-specific structure: any computable data-generating process can, in principle, be represented as a program. However, the space of computable structure is enormous, and only a small subset of programs produce sequences that are useful for the learner. Moreover, whether a sequence is useful changes as the learner improves. This is why a self-play approach that adapts to the learner is natural: rather than specifying useful structure in advance, we let the generator discover which programs are most useful to the learner as training progresses. To encourage this behavior, the generator is trained with reinforcement learning using a learning-progress reward, shifting probability toward programs whose outputs lie near the frontier of the learner’s current capabilities. These design choices place our approach in the lineage of classical universal prediction [Bloem, 2025, Grau-Moya et al., 2024, Hutter, 2000, Merhav and Feder, 1998, Solomonoff, 1964]in the lineage of classical universal prediction, which formalizes how induction can be possible when the hypothesis class contains all computable data-generating processes and the learner has unbounded compute: we aim at an efficient computable approximation to universal prediction.

The resulting self-generated data is only valuable insofar as what the learner learns transfers to natural data. Our key hypothesis is that self-play over this space of computable data-generating processes can discover generic predictive regularities—such as copying, recursion, and hierarchical composition—that improve prediction on natural data. Importantly, we hypothesize that these regularities capture structure independent of contingent information—the particular facts, symbols, or modality of any one dataset—that can therefore transfer across data-generating processes. Indeed, prior work on formal, algorithmic, and non-linguistic pretraining distributions provides evidence that such cross-distribution transfer is possible [Grau-Moya et al., 2024, Hu et al., 2025, Lee et al., 2026, Papadimitriou and Jurafsky, 2020]as shown in prior work.

We test this hypothesis through a compute-optimal scaling law analysis. Concretely, we train randomly-initialized transformers at various scales via self-play and evaluate the resulting learners zero-shot on held-out datasets spanning natural language, images, speech, melodies, DNA, and mathematical sequences. For each dataset, we construct a compute-optimal frontier over model size, self-play rounds, and ensemble size. Across these diverse domains, a single family of self-play models exhibits predictable power-law improvements in zero-shot loss with compute, with scaling exponents comparable to those obtained by training directly on natural data. Importantly, this is a clean test of transfer since we deliberately run this process tabula rasa: both models are randomly initialized and all learner training data is generated through self-play. Thus, our experiments isolate the effect of our self-play procedure and test whether useful predictive structure can emerge ex nihilo. We also find that the learner develops in-context learning on completely held-out tasks, and the generator discovers known mathematical sequences.

Self-Play Pretraining with Zero Data

Learn from only self-generated synthetic data

Figure 1: Self-Play Pretraining with Zero Data. Starting from randomly initialized models and using only synthetically generated data, self-play produces predictable scaling on held-out natural data. (Top) Our procedure casts synthetic data generation as search over the space of computable structure. A generator proposes programs, which are executed to produce byte sequences used to train a learner by next-token prediction. The generator is trained via reinforcement learning to propose programs near the frontier of the learner’s capabilities, measured by how strongly the learner’s gradients align with the learner’s recent learning trajectory with respect to an AdamW-preconditioned inner product. (Lower left) Self-play produces predictable improvements in validation loss with compute across natural datasets. We show the compute-optimal frontier over model sizes, number of self-play rounds, and ensemble sizes. (Lower right) The resulting learner exhibits in-context learning on held-out tasks without any gradient updates. We report empirical success under greedy decoding as a function of the number of in-context examples $m$m, averaged over independently sampled task instances.

2 Self-Play Pretraining with Zero Data

We introduce a self-play formulation of pretraining that involves searching over the space of computable structure. Every training sequence is the output of a program executed on a fixed universal Turing machine $U$U and these programs are produced by a learned generator. Our method has two components: (1) a Learner ${\pi }_{\theta }$pi theta, an autoregressive language model trained to predict program outputs, and (2) a Generator $g_{\varphi }$g phi, an autoregressive language model defined over programs for $U$U. Both are transformers with the same architecture, trained from random initialization; the generator is trained via reinforcement learning (RL) and the learner is trained via next-token prediction. Importantly, we consider this tabula rasa setup to cleanly test whether self-play can generate structure that transfers to natural data.

Each round of self-play proceeds as follows:

1. Program generation: Sample $N$N programs from the generator $\{x_i{\}}_{i=1}^N\sim g_{\varphi }$x sub i from i equals one to N, sampled from g phi.
2. Execution: Run each program on $U$U to obtain output sequences $y_i=U(x_i,{\omega }_i)$y sub i equals U of x sub i and omega sub i, where ${\omega }_i$omega sub i is a random input tape.
3. Learner and Generator update: The learner takes one gradient step on the output sequences, optimizing the standard next-token loss. The generator takes a policy gradient step with a learning-progress reward that encourages the generator to propose programs at the frontier of the learner’s capabilities. In addition, the generator is updated via a supervised fine-tuning objective on existing programs to mitigate catastrophic forgetting and on mutated programs to promote exploration.

2.1 Program space

We would like the generator’s search space to be as expressive as possible while imposing little domain-specific structure. We therefore use programs for a minimal universal Turing machine as the substrate for generating synthetic data. Specifically, we use a Brainf*ck-like Turing-complete language, following Grau-Moya et al. [2024]Grau-Moya and colleagues. Its primitive instructions manipulate a byte-valued tape, implement loops, and read or emit bytes. Because the language is universal, any computable data-generating process can, in principle, be represented as a program.

Let $x\in {\mathcal{A}}^{\le L}$x in the set A of length at most L denote a program generated over the machine’s instruction alphabet. Executing $x$x on the universal machine $U$U with random input tape $\omega$omega produces a byte sequence

y equals U of x and omega, which is a sequence of length T with values between zero and 255

The random input tape allows a single program to represent a distribution over output sequences. These output bytes, rather than the programs themselves, constitute the learner’s training data. We design the execution semantics so that every generated string is executable: programs cannot fail through syntax or memory errors, and execution always produces a bounded-length output. We defer the precise execution semantics and resource limits to Appendix E.

2.2 Objectives

Program pool. At each self-play round $e$e, we construct a pool of programs

B e is the union of fresh, mutated, and replay program sets

containing fresh samples from the current generator, local mutations of previously high-reward programs, and programs replayed from earlier rounds. Fresh samples provide global exploration,

mutations refine promising regions of program space, and replay preserves useful structures discovered earlier in training. We write $M_e=|{\mathcal{B}}_e|$M sub e equals the size of B sub e for the number of programs in the pool. Details for mutation, replay, and the program bank are offered in Appendix G.

Learner Update. The learner is trained by standard next-token prediction on program outputs. For an output sequence $y$y, define the per-sequence loss $\mathcal{L}(y;\theta )$L of y and theta as the mean cross-entropy over its content tokens. If $y_i=U(x_i,{\omega }_i)$y i equals U of x i and omega i is the output obtained by executing program $x_i$x i, the learner objective in round $e$e is

Thus fresh, mutated, and replay programs all train the learner.

Generator Reward. The generator’s reward must be capable of identifying programs with useful structure from programs without any external feedback. Initially, we considered a reward based on how difficult a sequence is to predict, motivated by prior work on self-play [Bailey et al., 2026a, Dong and Ma, 2025b]by Bailey and colleagues and Dong and Ma. However, this has a fundamental failure mode: a program can be made arbitrarily difficult without containing useful structure—for example, by injecting random bytes into an otherwise predictable sequence.

To avoid this degeneracy, we evaluate a new program based on whether it builds on what the learner has actually been able to learn. Intuitively, the learner’s change in parameters summarizes this: learning signals arising from reusable structure accumulate, whereas we expect that idiosyncratic effects that are not learnable do not. Concretely, we reward the generator for producing programs whose learner gradients align with the learner’s current learning trajectory. Let

p of e equals the floor of e over 2, and delta theta e is the difference between theta at the lookback horizon and the current theta

where ${\theta }_{p(e)}$theta p e is the learner checkpoint at the lookback horizon. The reward for program $x_i$x i with output $y_i$y i is

where

is the diagonal AdamW step operator obtained from the learner’s optimizer state. We use the lookback window of $\lceil e/2\rceil$the ceiling of e over 2 so that signals which take a long time to appear can be measured. The growing window helps average over short-term fluctuations and produce a signal which becomes more stable as training progresses, but allows early mistakes to eventually be forgotten.

We provide some additional intuition below, but we emphasize that we selected this reward after searching through several possibilities at small scale. Detailed analysis can be found in Table 5. Formally, this reward is a preconditioned gradient-alignment score, between the learner’s gradient on a program’s output and the learner’s parameter movement over the lookback window, with the diagonal AdamW preconditioner defining the inner product; we found that using this preconditioning was important in line with Thrush et al. [2026]Thrush and colleagues. Intuitively, this reward favors programs that are not yet mastered, but whose structure extends what the learner has already shown it can learn. We expect that programs that are already mastered induce nearly zero gradients, while programs containing unrelated or unlearnable structure induce gradients that do not align with the learner’s parameter movement. Both receive little reward. Instead, high reward is assigned to programs that induce substantial learning, but only in directions congruent with the learner’s recent progress; this concentrates the generator on the frontier of the learner’s current capabilities. We

avoid materializing full gradients by using forward mode automatic differentiation [Griewank and Walther, 2008]by Griewank and Walther to calculate Equation 2.¹

Policy-Gradient RL Objective. We train the generator using a KL-regularized expected reward

J R L of phi equals the expectation over x sampled from g phi of the reward r of x, minus beta times the K L divergence between g phi and g zero

where $\beta$beta is the KL regularization coefficient and $g_0$g zero is the fixed uniform program prior defined as

g zero of x equals the size of the alphabet A to the power of negative l of x

Here $\ell (x)$l of x is the number of tokens up to and including the terminating F. This is the natural analog of the Solomonoff prior $2^{-|p|}$two to the negative length of p [Solomonoff, 1964]described by Solomonoff, which weights programs according to description length. The generator is initialized near $g_0$g zero and regularized toward it throughout training.

Since the vanilla policy gradient estimator is high variance, we consider a GRPO (batch-level) based estimator [Guo et al., 2025]introduced by Guo and colleagues. Let ${\bar{r}}_e$r bar sub e and ${\sigma }_{r,e}$sigma r e denote the mean and standard deviation, respectively, of the rewards $\{ri{\}}_{i\in {\mathcal{B}}_e}$r i in the pool B e over the full round pool. We define the advantage of program $i$i as

The advantage A i is the normalized reward minus the weighted log ratio of the current policy and the prior

Since our bank consists of off-policy samples, we use a sequence-level importance ratio correction ${\rho }_i=\frac{g_{\varphi }(x_i)}{g_{{\varphi }_{\text{old},i}}(x_i)}$rho i equals g phi of x i over g phi old of x i [Zheng et al., 2025]proposed by Zheng and colleagues. It is equal to one for fresh on-policy samples; for replay samples, its denominator is the sampling probability stored when the program originally entered the replay bank. The policy-gradient term is then

Mutation rows are excluded because they were not sampled from a proposal distribution with a well-defined log probability. We clip the sequence-level importance ratio to ensure ${\rho }_i\in [e^{-20},e^{20}]$rho i is between e to the minus twenty and e to the twenty.

Expert Iteration. To prevent forgetting, we replay previous programs by distilling high-reward programs back into the generator using reward-weighted supervised fine-tuning over the full pool ${\mathcal{B}}_e$B e. This is a technique used to mitigate forgetting in pretraining and RL [Ibrahim et al., 2024, Schaul et al., 2016]as seen in prior work by Ibrahim and Schaul. We assign each program a normalized sequence-level weight

and optimize

The generator’s full training objective is

The full generator loss is the sum of the policy gradient loss and the weighted expert iteration loss

where ${\lambda }_{\text{EI}}=1.0$lambda E I equals one point zero controls the strength of the reward-weighted SFT term.

Figure 2: Self-play learns predictive structure that transfers across modalities. We compare self-play to two fixed synthetic pretraining distributions: programs sampled from a universal prior over Brainf*ck programs and probabilistic context-free grammars (PCFGs). Sampling from the universal prior scales substantially more slowly than self-play, showing that access to a universal space of programs alone is insufficient without the adaptive curriculum. Pretraining on PCFG transfers strongly to language-like domains, but its benefits are less consistent across non-language modalities. In contrast, self-play exhibits predictable scaling across images, melody, audio, speech, text, and code despite using minimal inductive bias. The compute-optimal frontier ends where we do not find further models with smaller loss than the largest-compute model shown.

2.3 Architecture and Tokenization

The learner and generator are independently parameterized decoder-only Llama transformers with identical architecture [Touvron et al., 2023]as described by Touvron and colleagues in twenty twenty-three. We use byte-level tokenization with a fixed vocabulary of 256 byte values because the universal machine produces raw bytes, and because it enables a clean, modality-agnostic evaluation of universal prediction: how well the model predicts the next byte in arbitrary sequences encoding text, images, audio, or other data. Programs and outputs are prefixed by the bytes $\text{S}$S and $\text{O}$O, respectively. During program generation, logits are restricted to the eight Brainf*ck instructions, ten canonical single-byte macro instructions, and the end-of-program token $\text{F}$F, whereas output sequences may contain any byte value.

3 Empirical Results

3.1 Universal zero-shot transfer scaling laws

In standard pretraining, scaling compute typically entails both increasing model size and training the model on more natural data. Our scaling experiments study whether increasing compute via self-play, without any natural data, produces predictable improvements in zero-shot performance on held-out natural data.

Scaling recipe. We follow the scaling methodology of Kim et al. [2026b], Wen et al. [2026]Kim and colleagues, and Wen and colleagues. At each model scale, we tune hyperparameters to local optimality, in the sense defined in Kim et al. [2026b]Kim and colleagues, via coordinate descent on a geometrically-spaced grid. Because our training objective is entirely synthetic, it does not provide an obvious criterion for selecting hyperparameters; poorly tuned hyperparameters could prevent clean scaling laws. We therefore use validation loss averaged across DCLM and DNA as a model-selection signal. This introduces limited leakage through hyperparameter selection, but natural data are never used for gradient updates.

Because tuning every possible hyperparameter at every scale is infeasible, we restrict the search to the most important hyperparameters based on preliminary experiments: the learner learning rate, the generator-to-learner learning-rate ratio, the batch size, and the generator KL regularization coefficient $\beta$beta. Due to compute constraints, we fix a large maximum training budget of 34.36B tokens rather than separately tuning the number of self-play rounds. Because we use only a short fixed warmup followed by a constant learning rate, every intermediate checkpoint is equivalent to a run stopped at that token budget. Thus, a single long run allows us to optimize over training duration retrospectively when constructing the scaling laws. When tuning batch size, we hold the total number of training tokens fixed; when the batch size changes, the number of rounds is adjusted accordingly to preserve the total token budget. Across all scales, we use a context length of 4096 tokens. We find that ensembling across randomly initialized models is useful. Therefore, at the locally-optimal hyperparameters, we train $K$K independent seeds per scale and form ensembles averaging the models’ predictive distributions.

For each dataset and algorithm, we construct a compute-optimal frontier [Kaplan et al., 2020]as described by Kaplan and colleagues over model sizes, checkpoints, and ensemble sizes. A point is on the frontier if and only if it achieves lower validation loss than every observed configuration with less than or equal compute.

We fit each compute-optimal frontier with the asymptotic power law

where $L$L is the observed loss in bits per byte and $E$E is a fitted asymptotic loss floor. We fit the model independently for each dataset.

Self-play exhibits universal zero-shot power-law scaling in compute. As shown in Figure 1, we observe scaling laws over a diverse range of modalities: text, images, and music. We present additional results in Figure 7. The scaling laws over these modalities are broadly similar, as seen in Table 2, (with DNA as the exceptional case). We discuss the implications of this in Section 4. In brief, we expect this to be the case when learning universal structure rather than contingent knowledge is the bottleneck to scaling. Importantly, these scaling results are entirely zero-shot: that is, they arise without any gradient steps on any of the evaluation datasets.

Pretraining on a fixed universal program prior exhibits slow scaling. To isolate the value of self-play, we compare self-play against a non-adaptive baseline over exactly the same program space; we use the same mixture of validation loss on DCLM and DNA. Instead of learning a distribution over programs, the baseline samples programs from a fixed Solomonoff-style prior [Solomonoff, 1964]as described by Solomonoff: instruction tokens are drawn i.i.d. until termination, giving full support to every finite program while favoring shorter descriptions. Thus, both methods have access to the same universal space of computable structure; they differ only in whether the sampling distribution adapts to the learner. Figure 2 shows that fixed sampling scales substantially more slowly, demonstrating that access to a universal program space alone is not enough—self-play must learn where in that space to allocate training compute. We offer additional evaluation datasets in Figure 7.

Qualitatively, we see that self-play’s improvement is because it discovers programs whose outputs exhibit recognizable mathematical structure (Table 1) far earlier than we would expect under uniform sampling from the universal prior: across $1.64\times 10^8$one point six four times ten to the eighth programs drawn from the uniform prior, we find no instances of any family except arithmetic sequences. Because such structures are common in mathematical modeling, this shows that our self-play algorithm can efficiently identify universal data, and that this is one mechanism driving the faster scaling observed in Figure 2.

Table 1: Program families with recognizable mathematical structure discovered by the generator during self-play. Earliest round gives the earliest round in which a member of the family first appears during training, while univ. prior gives the expected first appearance round if programs are drawn from the universal prior, including the added primitives. See section C for additional details.

PCFG pretraining is effective on language-like domains but lacks broad cross-domain transfer. In Figure 2, we also compare against pretraining on probabilistic context-free grammars (PCFGs), which provide a hand-designed source of hierarchical and compositional structure particularly well suited for language; for details of PCFG data generation see Section H. We expect pretraining on PCFG to be highly competitive on language-like domains, but its inductive bias is specialized to a particular class of structure. In contrast, our self-play procedure is, in principle, universal. Consistent with this interpretation, PCFG pretraining is stronger on text and code, where its inductive bias is well matched, while self-play substantially outperforms it on images, music, audio, and speech. Thus, self-play does not always match the performance of a specialized prior on domains where that prior is particularly well suited; however, it learns structure that transfers more broadly across modalities. We find that models trained on PCFG and the universal prior fail on our ICL evaluations in Figure 4.

The generator produces increasingly useful training data. We next ask whether the generator improves over the course of self-play. For each endpoint $T$T, we construct a fixed corpus $D_T$D sub T by sampling 4.19M programs uniformly across 16 generator checkpoints up to $T$T (with $g_0$g zero as untrained), and train a fresh 1M-parameter learner for one epoch on each corpus under a fixed token budget. We evaluate each corpus by its epiplexity—the amount of structure extractable by a compute-bounded learner [Finzi et al., 2026]described by Finzi and colleagues—and by zero-shot performance on held-out text, audio, and images (Figure 3). Both improve steadily with $T$T: later generators produce data containing more learnable structure and yielding better transfer, indicating that self-play continually improves the quality of the training distribution. Consistent with this, the generator also discovers recognizable mathematical sequences far earlier than expected under the fixed universal prior (table 1).

3.2 In Context Learning

An emergent property of large language models is their ability to infer a task from examples provided in context and apply the inferred rule to new inputs [Brown et al., 2020]as shown by Brown and colleagues. Beyond measuring held-

Figure 3: Later generator checkpoints provide additional, non-redundant training value. For each generator-training endpoint $T>0$T greater than zero we build a fixed corpus of programs by sampling from 16 generator snapshots spaced $T/16$T over 16 apart and ending at $T$T; $g_0$g zero uses the untrained generator. A 1M-parameter learner is then trained from scratch on each corpus, four seeds per endpoint. (a) Epiplexity, the excess training loss the learner accumulates before converging, grows steadily with $T$T: corpora written by later checkpoints contain more structure. (b) Out-of-distribution validation BPB of 4 ensembled seeds on text, audio, and images falls with $T$T. Together, the two panels show that later checkpoints enrich the curriculum rather than repeating earlier material, and that this additional data improves transfer to unseen datasets.

Figure 4: Comparing ICL behavior across methods. Self-play pretraining yields consistent improvements in ICL performance across all six tasks. In contrast, universal prior pretraining shows little evidence of effective ICL, while PCFG pretraining performs strongly on associative recall but transfers only weakly to the remaining tasks.

out loss, in-context learning provides a complementary test of whether self-play pretraining has produced a model that can infer latent structure in sequences. We evaluate our learner’s performance on several in-context learning tasks in Figure 1. We plot the empirical success rate under greedy $\arg \max$arg max decoding as a function of the number of in-context examples, $m$m. Importantly, the learner does not receive any additional gradient updates or fine-tuning. Rather, the learner must infer

Figure 5: Interpretable ICL behavior on SUM task. (a) Distribution over the model’s inferred strategies as the number of ICL examples increases. (b) Predictive entropy over the same trajectory, showing an initial loss of confidence followed by increasing certainty as the correct strategy emerges.

the latent structure underlying the sequence entirely in context. From the perspective of universal prediction and Solomonoff induction, this is precisely the kind of behavior we would hope to emerge after our self-play procedure. Section D contains task-specific details.

Self-play induces broad ICL where fixed synthetic pretraining does not. In Figure 4, we see that the model can achieve almost 100% accuracy on REVERSE STRING [Delétang et al., 2023]Delétang and colleagues, 2023, STACK [Delétang et al., 2023]Delétang and colleagues, 2023, and ASSOCIATIVE RECALL [Ba et al., 2016, Graves et al., 2014]Ba and colleagues, 2016, and Graves and colleagues, 2014 tasks after a sufficient number of ICL examples. The ASSOCIATIVE RECALL task shows that our model is capable of contextual search, the REVERSE STRING task shows that our model is capable of dynamically indexing, and STACK shows that our model is capable of learning to simulate a context-free grammar. In addition, we show that our model is capable of learning standard mathematical relations in-context (MAX, MIN, SUM). This is a natural test given that our model has been trained on short programs which could naturally express the application of mathematical functions. Before any examples have been shown ($m=0$m equals zero) the model already has a 6%-8% prediction accuracy on the MAX and MIN tasks due to its prior on copying previously produced tokens. In contrast, the models trained on PCFG and on samples from the universal prior over programs cannot learn all of these ICL tasks (Figure 4).

Qualitative analysis of shifting model strategies during SUM task. In Figure 5, we study the behavior of the model on the SUM task as it receives more ICL examples. Initially, the model predicts trivial outputs which correspond to its prior (the marginally most common bytes, indicated in dark grey). After seeing a few examples it begins to copy previous bytes, but then loses confidence after several overconfident but incorrect predictions, reverting to a very broad distribution; we see this reflected in the increase in entropy in the right panel. After around 4 examples, it begins to sum the 4 low-order bits correctly and by 8 examples it begins to sum the 4 high-order bits correctly as well. From there the model improves its confidence in its strategy and locks in on the correct approach.

4 Explaining Self-Play Scaling laws via Universal Data Ansatz

Our experiments show that (1) prediction on natural data improves predictably even though the learner does not train on any natural data and (2) the observed scaling exponents are comparable to those obtained from standard pretraining on particular domains. We propose a simple interpretation of these two results by separating two sources of predictive information that are ordinarily entangled in natural data: contingent information, which is specific to the particular world or distribution that generated the data, and universal predictive structure, which is shared across many data-generating processes.

Decomposing natural-data scaling. Hoffmann et al. [2022]Hoffmann and colleagues model loss as a function of model size $N$N and natural dataset size $D$D using the ansatz

We refine the data term by treating contingent information and universal structure as separate resources:

where $D_c$D sub c denotes the effective amount of contingent information available to the model and $D_u$D sub u the effective amount of universal predictive structure.

Explaining self-play power law scaling in compute. We first use the ansatz in Equation 7 to understand why we might see power law scaling in self-play compute. Since our models are not trained on natural data, contingent information is fixed as training proceeds. We absorb the third term in Equation 7 into a dataset-specific constant $E^{\prime }$. Only universal predictive structure generated through self-play can grow, giving

Here we write $D_u(T)$D sub u of T to indicate the amount of universal data generated via self-play at time $T$T. If self-play generates universal structure at a power-law rate,

then

Thus, the ansatz directly predicts the form of the scaling law we observe: natural-data loss can improve as a power law in the amount of self-play training data $T$T, despite no natural data being added during training, at the cost of a larger irreducible error².

Comparing self-play and natural-data exponents. Now, we show that, with some additional assumptions, the conventional one-term data-scaling law conflates gains from learning contingent information with gains from learning universal structure. In standard pretraining, increasing the amount of natural data $D$D increases both resources. Suppose

Substituting into Equation (7) gives

Asymptotically, the more slowly decaying of the two terms dominates, so we expect that a single fitted data exponent recovers

beta observed is approximately the minimum of beta nu and gamma mu.

We observe self-play exponents, which estimate $\gamma \eta$gamma eta, comparable to those reported for direct training on the corresponding natural modalities (table 2). This comparison is informative because self-play, in our setting, can improve only by increasing universal predictive structure, whereas natural data pretraining can improve through both universal and contingent information. We cautiously interpret this as suggesting that learning universal structure may account for an important component of the improvements obtained by scaling natural data.

5 Related Work

Synthetic data Since natural data is limited, synthetic data has increasingly been explored as a promising approach. One line of work uses generation to extract more learning signal from a fixed corpus of natural data. Synthetic continued pretraining generates diverse presentations and connections among facts in a small source corpus, improving the efficiency with which those facts are acquired [Yang et al., 2025]as shown by Yang and colleagues. Related approaches synthesize relationships across documents, latent reasoning underlying observed text, or multiple transformations of individual documents [Kim et al., 2026a, Ruan et al., 2025, Zelikman et al., 2024]. A distinct body of work asks whether useful structure can instead be learned from synthetic data that need not encode the target corpus itself. Pretraining on music, code, artificial languages, generic synthetic tasks, and formal languages can transfer to natural-language prediction and linguistic generalization [Hu et al., 2025, Papadimitriou and Jurafsky, 2020], while analogous results show transfer from procedurally generated data to natural images [Baradad et al., 2022, Kataoka et al., 2021]. These results suggest that synthetic data can teach predictive structure that is shared across domains. Recent approaches take a dataset attribution approach to synthesizing post-training data end-to-end [Thrush et al., 2026]; we view this as a complementary approach for pretraining data.

Self-play Early work on intrinsic motivation proposed rewarding agents for learning or compression progress, thereby directing exploration toward regularities that are neither already mastered nor currently unlearnable [Schmidhuber, 2008]as proposed by Schmidhuber. Related work on automatic curriculum and open-ended learning similarly constructs tasks that remain near the learner’s frontier [Schmidhuber, 2012]. More recently, self-play has been applied to language models in formal math [Bailey et al., 2026b, Dong and Ma, 2025a, Dong et al., 2024, Poesia et al., 2024], reasoning [Chen et al., 2025, Liu et al., 2026, Zhao et al., 2025], coding [Choi et al., 2026, Wang et al., 2026], and agentic tool use [Acikgoz et al., 2026, Zhou et al., 2025]. In contrast, we focus on pretraining with the scientific goal of establishing whether self-play can be used to discover training distributions whose structure transfers to prediction on completely unseen natural data. Moreover, for a clean analysis, our generator and learner begin from scratch, while most previous work, with the exception of Poesia et al. [2024]Poesia and colleagues, uses pretrained language models.

Universal prediction and algorithmic pretraining Universal prediction provides a theoretical framework for understanding when prediction over arbitrary computable data-generating processes is possible [Merhav and Feder, 1998, Solomonoff, 1964]. Most relevant to our work are Grau-Moya

et al. [2024]and colleagues and Bloem [2025]Bloem. Grau-Moya et al. [2024]Grau-Moya and colleagues show that neural networks can amortize universal prediction by training on outputs of programs sampled from a universal Turing machine, and demonstrate transfer to held-out algorithmic processes. We instead learn the program distribution jointly with the learner, rewarding programs according to the learning progress they induce. Closest empirically to our setting, Bloem [2025]Bloem provides evidence for universal pretraining from zero natural data, showing that training on sequences produced by iterated random computation yields improving zero-shot prediction on unseen real-world data with model scale and can accelerate subsequent finetuning. Our work builds on this direction by making the data-generating distribution adaptive: we formulate program generation as an RL problem and reward programs according to the learning progress they induce in the learner. This yields a learned curriculum that co-evolves with the learner. Empirically, we demonstrate transfer on a broader suite of held-out natural datasets and explicitly fit and analyze scaling laws for performance under self-play pretraining.

Epiplexity Finzi et al. [2026]Finzi and colleagues formalize a measure of the amount of structured information in data relative to a compute-bounded observer, which is necessary because unbounded notions such as entropy and Kolmogorov complexity fail to capture emergent structure. For example, for AlphaZero, the Kolmogorov complexity of its output is upper bounded by the length of the program that generated it, while the resulting model is vastly larger, since it stores what that program learned along the way in the weights of a neural network, amortizing inference. Just as Go can be decided with brute-force search, our setting can be trivially solved by an unbounded-compute agent via Solomonoff induction; only at finite compute does searching for useful recurring patterns ahead of time become necessary. We use epiplexity as a measure to check that our generator’s output increases in quality (see fig. 3). Furthermore, such a notion of structure offers a potential theoretical basis for the design of universal reward functions.

6 Discussion

Our results show that predictable improvements in next-token prediction on natural data can emerge even when no natural data is used for training. This provides evidence that some of the structure ordinarily acquired through standard pretraining on natural data is beneficial because it teaches the model universal predictive regularities. At the same time, universal pretraining cannot recover contingent information: facts about a particular world must ultimately enter through interaction with that world. Therefore, we do not view universal pretraining as a replacement for natural data, but as a way to isolate universal structure and study whether that component can instead be generated from compute.

Implications on synthetic data. The decomposition in Equation (7) provides a useful way to interpret recent approaches to synthetic data. Some methods generate abstract, formal, procedural, or otherwise domain-independent data whose primary value is to expose the model to transferable structure [Hu et al., 2025, Papadimitriou and Jurafsky, 2020]described in prior research. These methods can be understood as increasing the effective supply of universal data, $D_u$D sub u. Other methods begin with a fixed natural corpus and generate new presentations [Yang et al., 2025]Yang and colleagues, relations, or latent reasoning traces from it [Kim et al., 2026a, Ruan et al., 2025, Zelikman et al., 2024]as seen in several recent studies. Such methods can instead improve the efficiency with which information already present in the natural corpus is acquired, effectively increasing the useful supply of $D_c$D sub c; in practice, the chain-of-thought based methods may also expose additional reusable structure and therefore also affect $D_u$D sub u. This perspective offers an explanation for why synthetic data can improve pretraining even when the data is not from the same distribution as the target distribution of natural language. If universal structure is a limiting

resource, then generating additional structured experience can improve prediction despite bearing little surface resemblance to the target distribution. If contingent information is limiting, synthetic transformations of a fixed corpus can increase the amount of learning signal extracted from each observation. Our results demonstrate an extreme point in this design space: $D_c$D sub c is held fixed at zero during training, while the learning system is allowed to spend increasing compute searching for $D_u$D sub u.

Why tabula rasa? Our tabula rasa setting is intended as a controlled scientific experiment rather than necessarily the most practical way to pretrain a model. Since the learner and generator are randomly initialized, any transfer to natural data must have been acquired through the self-play process. In practice, there is no requirement that self-play begin from scratch. The key question is whether self-generated experience can continue expanding the frontier after naturally available data has become expensive, redundant, or exhausted. Our results suggest that this possibility is worth studying: if an adaptive curriculum can bootstrap transferable structure from random initialization, then the same mechanism may also be useful when initialized from a non-random learner. This could make our self-play algorithm complementary to standard pretraining.

Future Directions The experiments reported here are confined to models below 25M parameters at a 4K context, and the most immediate question is whether these results persist for larger models. Achieving this may require a more expressive programming language, allowing reusable abstractions to co-evolve with the generator, alongside other modifications that improve program search efficiency and overall scalability. Future work could also test whether the discovered mathematical structures causally contribute to transfer through circuit analysis or curriculum ablations.

7 Acknowledgments

We especially thank Suhas Kotha and Marvin Li for detailed feedback on an earlier draft of the paper. We thank Xiao-Liang Qi for valuable discussions at the inception of this project. Kfir Dolev was supported by the Long Term Future Fund and later by the Zuckerman STEM leadership program.

References

Emre Can Acikgoz, Cheng Qian, Jonas Hübotter, Heng Ji, Dilek Hakkani-Tür, and Gokhan Tur. Tool-r0: Self-evolving llm agents for tool-learning from zero data, 2026. URL arXiv.

Armen Aghajanyan, Lili Yu, Alexis Conneau, Wei-Ning Hsu, Karen Hambardzumyan, Susan Zhang, Stephen Roller, Naman Goyal, Omer Levy, and Luke Zettlemoyer. Scaling laws for generative mixed-modal language models. In International Conference on Machine Learning, pages 265–279. PMLR, 2023.

AITDCC. Algorithmic information theory data compression challenge official data archive. GitHub. data/data.zip, member data/B, GitHub repository, commit 16887f9d68f273503eddb69e0c29aa7b4f39ba7a.

Jimmy Ba, Geoffrey Hinton, Volodymyr Mnih, Joel Z. Leibo, and Catalin Ionescu. Using fast weights to attend to the recent past. In Proceedings of the 30th International Conference on Neural Information Processing Systems, NIPS’16, page 4338–4346, Red Hook, NY, USA, 2016. Curran Associates Inc. ISBN 9781510838819.

Luke Bailey, Kaiyue Wen, Kefan Dong, Tatsunori Hashimoto, and Tengyu Ma. Scaling self-play with self-guidance, 2026a. URL arXiv.

Luke Bailey, Kaiyue Wen, Kefan Dong, Tatsunori Hashimoto, and Tengyu Ma. Scaling self-play with self-guidance, 2026b. URL arXiv.

Manel Baradad, Jonas Wulff, Tongzhou Wang, Phillip Isola, and Antonio Torralba. Learning to see by looking at noise, 2022. URL arXiv.

Peter Bloem. Universal pre-training by iterated random computation, 2025. URL arXiv.

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020.

Lili Chen, Mihir Prabhudesai, Katerina Fragkiadaki, Hao Liu, and Deepak Pathak. Self-questioning language models, 2025. URL arXiv.

Mayee F. Chen, Tyler Murray, David Heineman, Matt Jordan, Hannaneh Hajishirzi, Christopher Ré, Luca Soldaini, and Kyle Lo. Olmix: A framework for data mixing throughout lm development, 2026. URL arXiv.

Caroline Choi, Zeyneb Kaya, Shirley Wu, Tengyu Ma, Tatsunori Hashimoto, and Ludwig Schmidt. Anchored self-play for code repair, 2026. URL arXiv.

Mutopia Project contributors. The mutopia project. URL mutopiaproject.org.

Santiago Cuervo and Ricard Marxer. Scaling properties of speech language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 351–361, 2024.

Grégoire Delétang, Anian Ruoss, Jordi Grau-Moya, Tim Genewein, Li Kevin Wenliang, Elliot Catt, Chris Cundy, Marcus Hutter, Shane Legg, Joel Veness, and Pedro A. Ortega. Neural networks and the chomsky hierarchy, 2023. URL arXiv.

Kefan Dong and Tengyu Ma. Stp: Self-play llm theorem provers with iterative conjecturing and proving, 2025a. URL arXiv.

Kefan Dong and Tengyu Ma. STP: Self-play LLM theorem provers with iterative conjecturing and proving. In Aarti Singh, Maryam Fazel, Daniel Hsu, Simon Lacoste-Julien, Felix Berkenkamp, Tegan Maharaj, Kiri Wagstaff, and Jerry Zhu, editors, Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pages 14114–14136. PMLR, 13–19 Jul 2025b. URL mlr.press.

Kefan Dong, Arvind Mahankali, and Tengyu Ma. Formal theorem proving by rewarding llms to decompose proofs hierarchically, 2024. URL arXiv.

Marc Finzi, Shikai Qiu, Yiding Jiang, Pavel Izmailov, J. Zico Kolter, and Andrew Gordon Wilson. From entropy to epiplexity: Rethinking information for computationally bounded intelligence, 2026. URL arXiv.

Jordi Grau-Moya, Tim Genewein, Marcus Hutter, Laurent Orseau, Grégoire Delétang, Elliot Catt, Anian Ruoss, Li Kevin Wenliang, Christopher Mattern, Matthew Aitchison, and Joel Veness. Learning universal predictors, 2024. URL arXiv.