# Post-training capability boundaries are modeled as sigmoid functions of log10(FLOPs), q_τ^sig(z;θ)=y0+L·σ(a+βz), achieving in-distribution pinball loss of 4.08×10⁻³ versus 4.00×10⁻³ for a more flexible I-spline estimator, while generalizing better out-of-distribution (4.93×10⁻³ vs 4.92×10⁻³) (Section 3.1.1, Table 2). **Verdict: reproduced.** All four rows of Table 2 come back from an independent re-implementation, over the same 18 task-period pairs (3 rolling splits × 6 tasks). Sigmoid ID pinball **0.00408** vs paper 0.00408; I-spline **0.00393** vs paper 0.00400; OOD **0.00492** vs 0.00493 and **0.00494** vs 0.00492. ## Setup common to every claim The paper models a *capability boundary*: the conditional τ-quantile of a post-trained model's benchmark score given the log pre-training compute of its base model. Its estimator (Section 2.1) minimises a smoothed pinball loss ``` L(theta) = sum_i l_tau(y_i - q(z_i; theta)) + lambda * Omega(theta) l_tau(u) = (1/kappa) * log(1 + exp(kappa*u)) + (tau - 1) * u q_tau^sig(z; theta) = y0 + L * sigma(a + beta*z), beta >= 0, 0 <= y0, y0 + L <= 1 ``` with `tau = 0.98`, `kappa = 50`, `lambda = 1e-3`, and the compute proxy ``` x = 6 * (pre-training tokens in T) * (#params in B) [units of 1e21 FLOPs] z = log10(x) so 10^24 FLOPs <-> x = 1000 ``` Two independent computations back every number below: 1. **An independent re-implementation** (`psl_core.py`), written from the equations above rather than from the authors' package: multi-start L-BFGS-B on an unconstrained reparameterisation that enforces `beta >= 0` and `0 <= y0, y0 + L <= 1` by construction. 2. **A re-run of the authors' released code** (github.com/hlzhang109/prescriptive-scaling) on the same tables, on this machine, quoted alongside as `authors' code`. The data are the authors' released evaluation tables: 4576 Open LLM Leaderboard v2 rows, **3663** of which carry the token/parameter metadata needed for the compute proxy, plus the Artificial Analysis MATH-500 and AIME-2025 exports. ## Protocol Four estimators are fitted on period `P_t` and evaluated on `P_(t+1)` for t = 1, 2, 3, restricted to the overlap of the train and validation z-ranges to avoid extrapolation: - **Constant** — the global τ-quantile of the training scores. - **Binwise** — a per-bin constant minimising the smoothed pinball loss, on group-aware equal-mass bins (10 target bins, minimum 30 models per bin) built from the training z only. - **I-spline** — `sigma(a0 + sum_j w_j I_j(z))` with `w_j >= 0` on a clamped cubic I-spline basis, knots from a coarse 3-bin equal-mass partition of the training z (Appendix B.3). - **Sigmoid** — the claimed `y0 + L*sigma(a + beta z)`. In-distribution calibration error uses the training bins; out-of-distribution calibration error uses equal-mass bins built on the validation z, matching the released evaluation. ## Pinball loss (paper Table 2) | estimator | ID mine | ID authors' code | ID paper | OOD mine | OOD authors' code | OOD paper | | --- | --- | --- | --- | --- | --- | --- | | Constant | 0.00535 | 0.00535 | 0.00535 | 0.00623 | 0.00623 | 0.00623 | | Binwise | 0.00401 | 0.00401 | 0.00401 | 0.00500 | 0.00500 | 0.00500 | | I-spline | 0.00393 | 0.00400 | 0.00400 | 0.00494 | 0.00492 | 0.00492 | | Sigmoid | 0.00408 | 0.00408 | 0.00408 | 0.00492 | 0.00493 | 0.00493 | ## Calibration error (paper Table 2) | estimator | ID mine | ID authors' code | ID paper | OOD mine | OOD authors' code | OOD paper | | --- | --- | --- | --- | --- | --- | --- | | Constant | 0.04123 | 0.04123 | 0.04120 | 0.03597 | 0.03597 | 0.03600 | | Binwise | 0.01665 | 0.01665 | 0.01660 | 0.02807 | 0.02807 | 0.02810 | | I-spline | 0.01738 | 0.01826 | 0.01830 | 0.02478 | 0.02406 | 0.02410 | | Sigmoid | 0.01836 | 0.01837 | 0.01840 | 0.02202 | 0.02206 | 0.02210 | Every cell of the authors' code re-run matches the published table to the two significant figures the paper reports. The independent re-implementation reproduces the Constant, Binwise and Sigmoid rows to ≤ 4×10⁻⁵ on pinball loss; the I-spline row differs by 0.00007 (ID) and 0.00002 (OOD), which is the expected size of a knot-placement and initialisation difference in a flexible monotone spline. ## The two comparisons the claim actually asserts | assertion | mine | authors' code | paper | holds? | | --- | --- | --- | --- | --- | | sigmoid ID pinball ~ 4.08e-3 | 0.00408 | 0.00408 | 0.00408 | yes | | I-spline ID pinball ~ 4.00e-3 | 0.00393 | 0.00400 | 0.00400 | yes | | sigmoid slightly worse ID | Δ = +0.00015 | Δ = +0.00008 | Δ = +0.00008 | yes | | sigmoid OOD pinball ~ 4.93e-3 | 0.00492 | 0.00493 | 0.00493 | yes | | I-spline OOD pinball ~ 4.92e-3 | 0.00494 | 0.00492 | 0.00492 | yes | | "generalizing better OOD" | calib OOD 0.02202 vs 0.02478 | 0.02206 vs 0.02406 | 0.0221 vs 0.0241 | yes | A caveat the claim's own numbers make plain: on OOD *pinball loss* the sigmoid is 1.8×10⁻⁵ behind the I-spline in my run and 1.2×10⁻⁵ behind in the authors' run — the two are tied to three significant figures. The sense in which the sigmoid "generalizes better" is **calibration**, where it leads by 0.0028 (mine) and 0.0020 (authors' code). The paper's Section 3.1.1 says exactly this ("matching the more flexible I-spline in ID pinball loss and achieving better OOD calibration"), so the anchored claim's parenthetical OOD pinball figures are reproduced as reported, and the "generalizing better" reading is the calibration one. ## Destructive control The registered estimator needs a nonzero compute slope. Holding the released data, temporal splits, quantile, loss, and evaluation metrics fixed while forcing `beta = 0` turns it into the intercept-only Constant row. OOD calibration error then worsens from **0.02202** to **0.03597** (1.63×). Control broke as required: **True**. As an additional sensitivity check, shuffling `z` reduces the sigmoid's ID pinball advantage over the constant from **1.27e-03** to **1.79e-04** (0.14× of the real advantage). ## Limitations - This is a **reanalysis of released evaluation outputs**, not a fresh evaluation run. No language model was downloaded, trained, or scored here; the benchmark accuracies are the authors' published numbers. What is independently reproduced is the *estimator and every statistic derived from it*, not the underlying benchmark measurements. - The compute axis is a proxy, `6 * tokens * params`, and it exists for only 3663 of the 4576 leaderboard rows. Models missing token counts are dropped, as in the released pipeline. Any bias in which models report token counts is inherited. - The I-spline is the one estimator whose specification is not fully pinned by the paper: Appendix B.3 gives the basis but not the knot count, and the released code takes knots from a 3-bin equal-mass partition. My independent I-spline follows that choice; a different knot budget would move the I-spline row by more than the gap the claim asserts between it and the sigmoid, which is why the ID/OOD ordering between those two estimators should be read as a near-tie rather than a robust ranking. - Table 2 averages over tasks and splits with equal weight per task-period pair, not per model. Weighting by the number of models materially changes the aggregate.