# Executive summary

**Paper.** "Prescriptive Scaling Reveals the Evolution of Language Model
Capabilities" (arXiv:2602.15327, OpenReview `IkjsHRpuYY`) estimates *capability
boundaries* — high conditional quantiles of a benchmark score given the log
pre-training compute of the model's base — by smoothed pinball quantile
regression with a monotone saturating sigmoid, and asks how stable that
compute-to-accuracy map is over time.

**What was done here.** Every claim is a statement about an estimator applied to
public evaluation tables, so all six are settled by reanalysis of the authors'
released data. Two independent computations back each number:

1. `psl_core.py`, an independent re-implementation written from the paper's
   equations — the smoothed pinball loss, the four-parameter sigmoid with
   `beta >= 0` and `0 <= y0, y0 + L <= 1`, group-aware equal-mass binning,
   a clamped cubic I-spline, and the coverage/pinball metrics;
2. an end-to-end re-run of the authors' released code
   (github.com/hlzhang109/prescriptive-scaling) on the same tables.

Data: the released Open LLM Leaderboard v2 table (4576 rows, **3663** with the
token/parameter metadata the compute proxy needs) and the Artificial Analysis
MATH-500 / AIME-2025 exports.

**No language models were trained or evaluated.** The benchmark accuracies are
the authors' published measurements; what is independently reproduced is the
estimator and every statistic derived from it.

## Correctness gates, passed before interpreting anything

| gate | result |
| --- | --- |
| Smoothed pinball loss equals the paper's literal expression | max abs diff **1.1e-16** |
| κ is a smoothing parameter: sup-gap to the sharp check loss = log(2)/κ | verified at κ = 10, 25, 50, 200 to rel. err **0.0e+00** |
| Compute proxy: 10²⁴ FLOPs ↔ x = 1000 under `x = 6·tokens·params` | 3663 models, z ∈ [-0.59, 4.01] |
| Independent fit reproduces all six published Table 1 cells | max abs diff **0.00047** — **PASS** |

The last gate is the important one: before any claim was interpreted, the
independent implementation had to land on the paper's own published numbers from
the paper's own data, which pins the data slice, the compute proxy, the metric
definitions and the objective simultaneously.

## Verdicts

All six claims **reproduced**.

## Three findings worth flagging

**1. The hyperparameter triple is uniquely identified by the paper's own table.**
Claim 1 asserts τ = 0.98, κ = 50, λ = 10⁻³. Beyond confirming these in the
released source, refitting the six boundaries under one-at-a-time perturbations
shows the claimed values are the *only* probed setting that reproduces Table 1:
max deviation **0.0005** at the claimed triple, against 0.0065
(λ = 10⁻²), 0.033 (κ = 200) and 0.33 (τ = 0.5). The hyperparameters are
load-bearing, not decorative — and empirical coverage tracks τ across
{0.50, 0.75, 0.90, 0.98} to within 0.04, so τ really is the quantile being
estimated.

**2. The sigmoid's out-of-distribution advantage is calibration, not pinball
loss.** Claim 2's own numbers make this explicit and the reproduction confirms
it: OOD pinball is a tie (sigmoid 0.00492 vs
I-spline 0.00494 in my run; 0.00493 vs
0.00492 in the paper), while OOD calibration error separates them
(0.0220 vs
0.0248). Anyone reading "generalizes better"
as "lower held-out loss" would be over-reading the result; the paper's own
Section 3.1.1 says calibration, and that is what reproduces.

**3. The temporal-drift result survives the control that matters.** MATH Lvl 5
and IFEval under-cover out of period
(-0.0248 signed coverage error)
while BBH, GPQA, MMLU-Pro and MUSR over-cover
(+0.0143); MATH's excess of
models above the boundary is significant at
p = 1.7e-18.
Crucially, all six tasks are well calibrated *in sample*, and randomising the
period labels collapses the drift/stable separation from
+0.0390 to
+0.0091. So this is a
temporal-transfer failure, not a fitting failure and not an artefact of the
grouping.

## Negative controls

Every claim is paired with a control that had to break, and all six did:

| claim | control | result |
| --- | --- | --- |
| 1 | refit at τ = 0.5 | coverage 0.994 → 0.518 |
| 2 | force the sigmoid slope to zero | OOD calibration error 0.0220 → 0.0360 |
| 3 | change only the registered quantile from 0.98 to 0.50 | mean Table-1 error rises to 0.192; all six fail 0.001 |
| 4 | randomise period labels | drift/stable separation +0.0390 → +0.0091 |
| 5 | cut the registered 20% budget to 5% | macro frontier error 0.0215 → 0.0450 |
| 6 | inject a +0.8 logit post-release shift | recovered γ = 1.38, p = 0.0005 |

## Global limitations

- This is a **reanalysis of released evaluation outputs**, not a fresh evaluation
  run. No language model was downloaded, trained, or scored here; the benchmark
  accuracies are the authors' published numbers. What is independently
  reproduced is the *estimator and every statistic derived from it*, not the
  underlying benchmark measurements.
- The compute axis is a proxy, `6 * tokens * params`, and it exists for only
  3663 of the 4576 leaderboard rows. Models missing token counts are dropped, as
  in the released pipeline. Any bias in which models report token counts is
  inherited.
- Claims 2, 4 and 5 aggregate over only three rolling temporal
  splits, so temporal replication is thin; the per-split tables are reported so a
  reader can see how much of each aggregate rests on a single split.
- Claim 5's "near-full-data" is not given a numerical tolerance in the paper, so
  its verdict rests on the reproduced *shape* of the budget-error curve rather
  than on a threshold.