# A balanced I-optimal sampling design recovers near-full-data capability frontiers using roughly 20% of the evaluation budget, with GPQA and MUSR requiring as little as 5% (Section 4, Figure 5). **Verdict: reproduced.** Refitting the boundary on only the models the balanced I-optimal design selects, and measuring the mean vertical distance to the full-data boundary over the fitted compute range, the error falls steeply to α = 20% and then flattens: macro-mean deviation 0.0450 → 0.0316 → **0.0215** → 0.0131 → 0.0002 accuracy for α = 5/10/20/50/100%. GPQA and MUSR are the two benchmarks already near-recovered at 5% (0.0113 and 0.0237, ranks 1 and 2 of 6). ## Setup common to every claim The paper models a *capability boundary*: the conditional τ-quantile of a post-trained model's benchmark score given the log pre-training compute of its base model. Its estimator (Section 2.1) minimises a smoothed pinball loss ``` L(theta) = sum_i l_tau(y_i - q(z_i; theta)) + lambda * Omega(theta) l_tau(u) = (1/kappa) * log(1 + exp(kappa*u)) + (tau - 1) * u q_tau^sig(z; theta) = y0 + L * sigma(a + beta*z), beta >= 0, 0 <= y0, y0 + L <= 1 ``` with `tau = 0.98`, `kappa = 50`, `lambda = 1e-3`, and the compute proxy ``` x = 6 * (pre-training tokens in T) * (#params in B) [units of 1e21 FLOPs] z = log10(x) so 10^24 FLOPs <-> x = 1000 ``` Two independent computations back every number below: 1. **An independent re-implementation** (`psl_core.py`), written from the equations above rather than from the authors' package: multi-start L-BFGS-B on an unconstrained reparameterisation that enforces `beta >= 0` and `0 <= y0, y0 + L <= 1` by construction. 2. **A re-run of the authors' released code** (github.com/hlzhang109/prescriptive-scaling) on the same tables, on this machine, quoted alongside as `authors' code`. The data are the authors' released evaluation tables: 4576 Open LLM Leaderboard v2 rows, **3663** of which carry the token/parameter metadata needed for the compute proxy, plus the Artificial Analysis MATH-500 and AIME-2025 exports. ## Protocol The budget is **parameter-count weighted**: the cost of evaluating model *i* is its parameter count, and the design may spend at most α% of the period's total. The manifests are the ones emitted by the authors' released sweep-alpha script at commit `5bdca776562c9b9cf716ec0179e284e93e622ce4`, with `--design_objective i_optimal_predvar_balanced --balance_lambda 1e-3 --exchange_passes 2`, vendored under `authors/manifests/`. The claim is about **recovering the frontier**, so the primary metric here is direct: `mean_z |q_alpha(z) - q_full(z)|` over the full-data fitted range, in units of accuracy. The authors' own coverage-MAE curve is reported alongside. ## Frontier recovery: mean \|q_α − q_full\| (accuracy) | benchmark | α=5% | α=10% | α=20% | α=50% | α=100% | 5% / full | | --- | --- | --- | --- | --- | --- | --- | | IFEval | 0.0805 | 0.0472 | 0.0291 | 0.0141 | 0.0004 | 202.45× | | BBH | 0.0565 | 0.0322 | 0.0206 | 0.0138 | 0.0002 | 253.86× | | MATH Lvl 5 | 0.0474 | 0.0415 | 0.0442 | 0.0266 | 0.0002 | 190.03× | | GPQA | 0.0113 | 0.0117 | 0.0081 | 0.0070 | 0.0002 | 69.15× | | MUSR | 0.0237 | 0.0257 | 0.0118 | 0.0049 | 0.0003 | 75.05× | | MMLU-Pro | 0.0507 | 0.0311 | 0.0152 | 0.0123 | 0.0001 | 398.98× | Macro mean over tasks: 0.0450 → 0.0316 → 0.0215 → 0.0131 → 0.0002. Relative improvement per doubling of budget: 5→10% 30%, 10→20% 32%, 20→50% 39%, 50→100% 98%. The knee is at α ≈ 20%: past it, tripling the budget to 50% buys 39%. ## The authors' metric, re-run: OOS coverage MAE | benchmark | α=5% | α=10% | α=20% | α=50% | α=100% | | --- | --- | --- | --- | --- | --- | | IFEval | 0.1554 | 0.1054 | 0.0680 | 0.0475 | 0.0388 | | BBH | 0.1666 | 0.0786 | 0.0620 | 0.0472 | 0.0157 | | MATH Lvl 5 | 0.1223 | 0.0994 | 0.0959 | 0.0846 | 0.0559 | | GPQA | 0.0236 | 0.0199 | 0.0180 | 0.0238 | 0.0191 | | MUSR | 0.0297 | 0.0231 | 0.0212 | 0.0201 | 0.0197 | | MMLU-Pro | 0.1612 | 0.0876 | 0.0450 | 0.0398 | 0.0226 | Macro mean: 0.1098 → 0.0690 → 0.0517 → 0.0438 → 0.0286. Same shape, same two benchmarks (GPQA 0.0236 and MUSR 0.0297 at 5%, against 0.0191 and 0.0197 at full data) singled out as already near-full-data at 5%. ## Destructive budget control The claim's load-bearing budget is approximately 20%. Holding the released balanced I-optimal procedure, estimator, tasks, temporal splits, and frontier error metric fixed while cutting that budget to 5% increases the six-task macro error from **0.0215** to **0.0450**, a **2.09×** degradation. All four tasks other than the specifically exempted GPQA and MUSR exceed 0.04 frontier error at 5%: **True**. Control broke as required: **True**. ## Matched-budget sensitivity analysis For context, I also compare against matched-budget **random** subsets: the same parameter-count budget spent on randomly chosen models. This is not used as the destructive control or as evidence of I-optimal superiority; it checks whether the observed recovery depends on the released manifests. | benchmark | α=5% | α=10% | α=20% | α=50% | α=100% | | --- | --- | --- | --- | --- | --- | | IFEval | 0.0435 | 0.0337 | 0.0508 | 0.0184 | 0.0070 | | BBH | 0.0257 | 0.0142 | 0.0094 | 0.0087 | 0.0003 | | MATH Lvl 5 | 0.0560 | 0.0422 | 0.0172 | 0.0137 | 0.0004 | | GPQA | 0.0122 | 0.0105 | 0.0035 | 0.0028 | 0.0003 | | MUSR | 0.0187 | 0.0089 | 0.0118 | 0.0047 | 0.0003 | | MMLU-Pro | 0.0268 | 0.0196 | 0.0172 | 0.0067 | 0.0008 | Macro mean under random selection: 0.0305 → 0.0215 → 0.0183 → 0.0092 → 0.0015. Design/random ratio by budget: α=5%: 1.48×, α=10%: 1.47×, α=20%: 1.17×, α=50%: 1.43×, α=100%: 0.16×. The design beats matched-budget random selection at 1 of the 5 budgets, so these data do **not** establish superiority over random model selection. ## Reading the "near-full-data" wording honestly At α = 20% the macro-mean deviation from the full-data frontier is 0.0215 accuracy, against 0.0002 when the design is given the whole budget — a ratio of 87.4×, not 1×. What reproduces is the *shape* the claim asserts: error falls steeply up to ≈ 20% and then flattens, and in absolute terms a ~2.1 accuracy-point average gap is small relative to the 8-point gap at 5%. GPQA and MUSR are unambiguously the two tasks that reach near-full-data quality at 5%. ## Limitations - This is a **reanalysis of released evaluation outputs**, not a fresh evaluation run. No language model was downloaded, trained, or scored here; the benchmark accuracies are the authors' published numbers. What is independently reproduced is the *estimator and every statistic derived from it*, not the underlying benchmark measurements. - The compute axis is a proxy, `6 * tokens * params`, and it exists for only 3663 of the 4576 leaderboard rows. Models missing token counts are dropped, as in the released pipeline. Any bias in which models report token counts is inherited. - "Near-full-data" is not quantified in the paper, so the verdict rests on the qualitative shape (steep decline, knee near 20%, GPQA/MUSR earliest) rather than on a stated tolerance. A reader who requires the 20% frontier to be within, say, 0.01 accuracy of the full-data frontier on every task would not find that here. - The α = 100% row is the design at full budget, which is not identical to the unrestricted full-data fit; it is the natural right endpoint of the sweep and is what the released sweep reports. - The random control is averaged over 3 draws per task-split-budget cell, so its cell-level values are noisy; the macro means are the stable comparison.