# Prescriptive Scaling Reveals the Evolution of Language Model Capabilities — reproduction

Paper: arXiv:2602.15327 · OpenReview `IkjsHRpuYY` · Zhang, Jin, Syrgkanis & Kakade

All six anchored claims are settled by re-deriving the paper's statistics from
its **released evaluation tables**, using an independent re-implementation of
the capability-boundary estimator written from the equations of Section 2.1.
The authors' released code was also re-run end to end as a cross-check, and both
sets of numbers appear on every claim page. **No language model was trained or
evaluated**; this is a reanalysis of released outputs, and each page says so.

| # | claim | verdict | headline number |
| --- | --- | --- | --- |
| 1 | [Estimator hyperparameters τ=0.98, κ=50, λ=10⁻³](#/claim-1-estimator-hyperparameters) | reproduced | only setting reproducing Table 1 (max dev **0.0005**; perturbations 0.0065–0.33) |
| 2 | [Sigmoid vs I-spline, Table 2](#/claim-2-sigmoid-vs-ispline) | reproduced | sigmoid ID **0.00408** vs paper 0.00408; I-spline **0.00393** vs 0.00400 |
| 3 | [Attainable accuracy at 10²⁴ FLOPs, Table 1](#/claim-3-attainable-accuracy-1e24) | reproduced | all six cells within **0.0005** |
| 4 | [MATH Lvl 5 / IFEval boundaries drift; four others stable](#/claim-4-temporal-nonstationarity) | reproduced | signed coverage **-0.0248** (drift) vs **+0.0143** (stable) |
| 5 | [Balanced I-optimal design at ≈ 20% budget](#/claim-5-budgeted-design) | reproduced | macro frontier deviation **0.0215** at 20%, knee after 20% (39% further gain to 50%) |
| 6 | [AIME-2025 contamination shift test](#/claim-6-contamination-shift-test) | reproduced | γ = **0.5757** > 0, one-sided p = **0.1484** (paper 0.15), n = 90 |

## Pages

| Page |
| --- |
| [Executive summary](#/executive-summary) |
| [Claim 1: Capability boundaries are estimated via smoothed pinball-loss quantile regression at quantile τ=0.98 with smoothing parameter κ=50 and regularization λ=10⁻³ (Section 2.1).](#/claim-1-estimator-hyperparameters) |
| [Claim 2: Post-training capability boundaries are modeled as sigmoid functions of log10(FLOPs), q_τ^sig(z;θ)=y0+L·σ(a+βz), achieving in-distribution pinball loss of 4.08×10⁻³ versus 4.00×10⁻³ for a more flexible I-spline estimator, while generalizing better out-of-distribution (4.93×10⁻³ vs 4.92×10⁻³) (Section 3.1.1, Table 2).](#/claim-2-sigmoid-vs-ispline) |
| [Claim 3: At 10²⁴ FLOPs, the estimated 0.98-quantile boundary reaches 0.539 accuracy on MATH Level 5 versus 0.563 on MMLU-Pro, 0.700 on BBH, 0.828 on IFEval, 0.535 on MUSR, and 0.424 on GPQA (Table 1).](#/claim-3-attainable-accuracy-1e24) |
| [Claim 4: Unlike the stable boundaries for BBH, GPQA, MMLU-Pro, and MUSR, the MATH Level 5 and IFEval capability boundaries are non-stationary over time, with later-period models exceeding earlier predicted boundaries (Figure 2).](#/claim-4-temporal-nonstationarity) |
| [Claim 5: A balanced I-optimal sampling design recovers near-full-data capability frontiers using roughly 20% of the evaluation budget, with GPQA and MUSR requiring as little as 5% (Section 4, Figure 5).](#/claim-5-budgeted-design) |
| [Claim 6: A cross-benchmark shift test applied to AIME-2025 scores finds a positive but not statistically significant shift (p=0.15), giving no clear aggregate evidence of post-release contamination inflating math benchmark scores (Section 5.2.2, Equation 3).](#/claim-6-contamination-shift-test) |

## Reproduce

```bash
pip install -r requirements.txt
python3 run_analysis.py    # -> outputs/*.json   (CPU only, ~15 min)
python3 build_pages.py     # regenerate these pages from those JSON files
```

Correctness gate: the independent fit reproduces all six published Table 1 cells
to 0.0005 before any claim is interpreted.