Searching for Chinchilla Scaling in Tiny Arithmetic Transformers

18 Sep, 2026

Scaling Laws

Deep neural networks such as modern LLMs function as a kind of black box. We understand very little of what’s truly going on behind the trillions of floating point operations powering these tools. Lucky for us, there are ways to study these boxes without ever needing to open them. Among our limited arsenal lie scaling laws, which are empirical models that elucidate the relationship between our inputs (e.g. model size, training data, compute) and outputs (i.e. model capabilities)¹.

Admittedly, “law” is a loaded term. These “laws” function much more like Moore’s law than Newton’s laws of nature; they are empirical observations, fitted on what we’ve actually built. So they are inherently conditioned on the realized path we took, as opposed to some fundamental unchanging truth about the universe on its own. This is important because there are often bottlenecks preventing us from achieving the degree of scaling modeled by these laws (just like how the chip industry had to resolve bottlenecks to “realize” Moore’s law)². Once those bottlenecks are resolved, the laws then describe how certain input factors interact with one another, both directionally and numerically to influence the capabilities of our trained model.

Many interesting questions arise even as we operate within a scaling paradigm. While we can naively tune up all the input factors that are positively correlated with performance, the reality of resource constraints mean that we are often faced with tradeoffs. Let’s say we want to train some new model, and by divine benevolence we are granted 2x our original compute budget. Should we double our model size? Double our training data? Or a mix of both? These are the questions answered by Chinchilla’s law, the topic for this blogpost.

A simple model for models

To motivate this, suppose we want to derive some model for loss $L$ based on three key training parameters: model size $N$ (parameter count), training data size $D$ (measured in tokens), and compute budget $C$ (in floating point operations, or "FLOPs"):

Here, $D$ counts tokens processed during training, including repeated examples. We want to find the combination that produces the lowest training loss:

The first thing to note here is that conditional on a fixed model architecture, the three inputs are usually linked by some known function $C=g(N,D)$. This makes sense since our training FLOPs is dictated by the model architecture. We can approximate the transformer architecture as a massive stack of big matrix multiplications, which has a simple training compute formula³:

For example, Llama 2 7B (as in 7 billion params) was trained on 2 trillion tokens, so here $C=6\times 7\times {10}^9\times 2\times {10}^{12}=8.4\times {10}^{22}$ FLOPs. Very handy formula! With this, we can take $D=C/6N$, and our original formula simplifies to $L=f(N,C)$.

Unfortunately, compute isn’t free⁴. With a fixed budget $C_0$, our problem becomes simply

How do we solve this without knowing (or assuming) the functional form of $f$? Let’s hold this thought and return to our handy FLOPs approximation formula $C=6ND$. Assume for some $C_0$ we already found the optimal $N_0$, so we have $C_0=6N_0{D}_0$. An interesting question we can ask is, if we were to scale $C_0$ to, say $C_1=2C_0$, what is the new optimal $N_1$? Comparing the old and new optimal values, we can define scale factors $x:=N_1/N_0$ and $y:=D_1/D_0$. We know that by definition $C_1=6N_1{D}_1$. Substituting, we get $C_1=6·x{N}_0·y{D}_0=2C_0$, which implies $xy=2$. Writing the scale factors as powers of two, $x=2^a$ and $y=2^b$, this becomes

Stated plainly, this means that we need to distribute our compute scale factor 2 into a mix of model parameter count and training data size. The key question here is, how should we pick $a$?

Enter Chinchilla’s law

Chinchilla’s law helps us answer all these questions. Its answer to our last question is $a=b=0.5$. This implies $x=y=\sqrt{2}$ in our case. More broadly, it makes the observation that under fixed compute budget constraints, training data size should scale proportionally to model size. With this solution in mind, we can generalize the doubling example to an arbitrary scaling by a factor of $s$: $N_{\text{opt}}(s{C}_0)=s^{0.5}{N}_0$. Since this holds for any starting point $C_0$ we arrive at a power law

Something interesting happens when our exponent is 0.5. Consider the expression $D_{\text{opt}}/N_{\text{opt}}$, which is the ratio of our training data size to model parameter count. By our handy formula we have $D_{\text{opt}}/N_{\text{opt}}=C/6N_{\text{opt}}^2$. But since $N_{\text{opt}}=k{C}^{0.5}$, this simply becomes

a constant independent of $C$! In other words, we have found that the optimal training data size to training parameter count ratio is a fixed constant independent of compute. This characteristic holds if and only if training data size scales proportionally to model size. And in Chinchilla’s case, this ratio was roughly 20 tokens per parameter. Quite magical!

To sum up, the contributions of Chinchilla’s law are three-fold:

1. The rule of thumb that training data size should scale proportionally to model size
2. The famous 20:1 training size to model parameter count ratio, known as Chinchilla optimality
3. A fitted functional form for loss, with an irreducible term $E$ plus power-law penalties for finite model size and finite data:

Setting up a toy experiment for Chinchilla’s law

While we can certainly try to observe Chinchilla’s law by pre-training our own language model from scratch, it would also be interesting to see if the same empirical relationship holds beyond natural language modeling (and on a much, much smaller total compute budget). To investigate this, I took a simple 3-digit integer addition task and trained a collection of tiny transformer models, sizes ranging from 7k to 800k parameters, at varying compute budgets (hence training data sizes). Each example is a padded string like "123 + 456 = 579 ", tokenised character by character over a 13-symbol vocabulary (ten digits, +, =, and space), and the model is scored only on the digits (and padding) after the =.

How do we set up an experiment to observe this law? As we have already done the mathematical heavy lifting earlier, we know that it suffices to show $N_{\text{opt}}\propto C^{0.5}$. The constant token-to-parameter ratio follows naturally, though not necessarily Chinchilla’s particular ratio of 20:1. Taking log of both sides, this becomes

for a constant $k^{\prime }=\log k$. One approach to verify this, is to choose a range of compute budget values $C$, and for each $C$, train models of varying sizes $N_1,\dots ,N_i$, and determine $N_{\text{opt}}$ by taking the model that has the lowest loss. Then, we can take our $(C,N_{\text{opt}})$ pairs, take logs, and fit a linear regression to see if we recover 0.5 as the slope.

As always, there’s a handful of subtleties worth pinning down before committing to training runs. First, we have to ensure that our budgets $C$ are within a resource constraint regime, such that there still exists a real tradeoff between model size and training size. One can imagine some budget ceiling beyond which too many model vs training size pairs trivially solve the task. A task as simple as 3-digit addition would have quite a low ceiling; I measured this at around ${10}^{13}$ FLOPs, past which too many neighboring $N$ values have similarly low loss values, dampening our signal. At the lower end, I settled on $5\times {10}^{11}$ FLOPs, where a tradeoff was still visible in the early experiments.

Second, as we train models of varying sizes especially in log scale, model performance becomes sensitive to hyperparameter choices, such as learning rate and batch size. The Chinchilla paper used larger peak learning rates for its smaller models than for its largest ones. This effect was indeed also present in my early training runs with a naive fixed peak learning rate of 3e-4, which overly penalized smaller models. After experimenting with various values with the goal of finding nice U-curves, I ended up using a small learning rate sweep of [3e-3, 1e-2, 3e-2], and selecting the learning rate with the lowest geometric-mean loss across the five seeds for each model size and budget⁵. As with the Chinchilla paper, I used a cosine learning rate schedule with warmup and decay⁶. For batch size, on the other hand, I kept it fixed at 128, as varying this did not seem to affect runs that much, so it made more sense to keep it a constant to isolate model size as the variable.

Unlike Chinchilla which measured training losses, my experiments chose $N_{\text{opt}}$ based on held-out validation loss on 10% of the full dataset. The main reason here is that Chinchilla runs were sub 1 epoch, but for 3-digit additions there are only ${10}^6$ examples in total, and small models with large compute repeatedly revisit the same examples (up to about 17 dataset-equivalents in this sweep) to the extent that training loss stops being an unbiased estimator of loss on fresh data.

We repeat the same set of runs across 5 seeds. All runs were done on a TPU v5e-8 through Kaggle’s weekly 20hr free credits.

Results

Let’s first look at the “isoFLOP” curve for each compute budget and each seed. Each of these plots fixes a compute budget $C$, and plots final test loss as a function of model parameter size $N$. As we can see, each isoFLOP curve generally resembles a U-shape, which represents the tradeoff between allocating more compute to a larger model versus more training data.

Eyeballing the graphs, we can already see that as we increase our compute budget (and move down the set of plots) the minimum test loss model parameter size generally increases. One thing that stands out here, though, is that there is a high degree of variability across the sweeps. For the same compute budget, not all seeds end up with the same optimal model size; there is also a lack of monotonicity on both sides of the flanks, though most curves generally follow a U-shape. At $C=2\times {10}^{12}$, for example, every model from 15k to 87k parameters lands within 25% of the best five-seed geometric-mean loss. That’s a small loss difference deciding between models nearly six times apart in size! A slight change in the losses can therefore produce a large jump in $N_{\text{opt}}$, even when the overall curve barely changes.

Averaging the five seeds cleans the curves up considerably, and the rightward drift of the minimum with compute becomes clear:

A parabola lets us use the flanks of the curve to help locate its bottom, instead of letting the lowest individual point decide everything. For each compute budget $C$, we take the readings across all 5 seeds (45 points per budget), fit a 2-deg polynomial in $\log N$ and $\log \text{loss}$, and convert its vertex back into model size $N$. The left panel below shows this at $C=2\times {10}^{12}$: the individual points at the bottom scatter by a factor of three across seeds, but the two flanks pin the parabola, and its vertex lands at 55k. We can then fit the $(\log C,\log N_{\text{opt}})$ pairs on a linear regression and find the slope, which was hypothesized to be 0.5 under Chinchilla’s law. We can also perform bootstrap resampling using our 5 seeds to find a 95% CI for the slope. The catch is that this stability comes partly from assuming a smooth quadratic curve. The bootstrap captures seed variability under that assumption. The right panel plots the regression, and we report our $\hat{a}$ below:

The regression gives, with a 95% bootstrap confidence interval over seeds,

and the fitted optima at each budget are:

So, did we find Chinchilla's law?

First, our fitted scaling exponent is compatible with Chinchilla’s ~0.5! Hurray!

The more interesting question to ask here is, did we have any reason to have a strong prior at all for the scaling exponent of 0.5, given that the trivial addition task at hand is fundamentally different from natural language modeling? Although our quadratic fits estimate some compatibility, our per-sweep and per-seed results, as the grid of isoFLOP curves above shows, seem quite noisy. Other than imperfections in resolving learning-rate sensitivity, perhaps one contributor is seed-sensitive learning transitions on this arithmetic task; we have been operating outside Chinchilla’s data regime. Unlike language modeling runs which have irreducible loss due to the inherent uncertainty in next-token prediction, our labels are deterministic and completely solvable. Different runs may learn digit alignment or carry handling at different times, producing large endpoint differences at a fixed budget. At high budgets, it becomes difficult to select a point estimate for $N_{\text{opt}}$, when a range of models produce similarly low test losses.

So within the budget range tested here, our fitted optima suggest that model parameter size should generally increase with compute. And the Chinchilla scaling exponent does lie within our confidence intervals. Whether that makes it Chinchilla's law or just a law-shaped coincidence on a toy problem, I'll leave to a reader with more digits.

Appendix: Mixture-of-Experts

In addition to the dense model, I also ran the entire training loop on two Mixture-of-Experts (MoE) variants. Both had 4 experts ($E=4$), with one having 1 activated expert and the other having 2 activated experts per MLP block ($k\in \{1,2\}$). For these sweeps we take $N$ as activated params, i.e. the parameters a token actually passes through, that is attention plus 1 or 2 of the activated experts. The fitted exponents come out similar. For $k=1$, we get $\hat{a}=0.55$ with a 95% CI of $[0.51,0.60]$, and for $k=2$, we get $\hat{a}=0.60$ with a 95% CI of $[0.55,0.62]$. The left plot below shows the best loss across all three model architectures per compute budget, and the right plot shows our log-log linear regression fits.

The main observation here is that at fixed compute, MoE does not reach the same loss as the dense model. The dense model reaches the lowest validation loss at all five budgets, with the gap widening at higher compute up to a factor of around 3, though with five seeds the smaller gaps are within noise. This makes sense, since the main purpose of MoE is to raise model capacity by storing more parameters than it uses per token, and this is a solution more relevant for regimes like language modeling whose loss is limited by model capacity. Three-digit addition is much simpler, so the added capacity goes unused, while the costs of routing remain. The cleanest evidence is the top-1 variant, which has the same active parameter count and the same number of training steps as the dense model, yet still ends up 1.5x to 2x worse. With each expert seeing only half or a quarter of the tokens but essentially trying to learn the same arithmetic patterns, the MoE models could be spreading learning signals too thin across their parameters.

Comparing between the two MoE variants, we see that the top-2 variant leads at the two smallest compute budgets, before falling behind at larger budgets, where top-1 leads by about 1.1x to 1.6x from 2e12 upward. One possible explanation for this is that since the top-2 model carries 1.7x the active parameters of the top-1 model (2x in MLP layers but same in attention), at each budget it gets fewer training steps, and under the higher budget regime training steps are the bigger bottleneck towards achieving better learning and generalization as opposed to model capacity. It would be interesting to see if we can generalize this training step vs capacity/sparsity scaling relationship for MoE architectures in the original language modelling context, and see how model sparsity scales with compute.