# Unlike the stable boundaries for BBH, GPQA, MMLU-Pro, and MUSR, the MATH Level 5 and IFEval capability boundaries are non-stationary over time, with later-period models exceeding earlier predicted boundaries (Figure 2).

**Verdict: reproduced.**

Fitting on period `P_t` and testing on `P_(t+1)` separates the six benchmarks
exactly as claimed. BBH, GPQA, MMLU-Pro and MUSR **over**-cover out of period
(signed coverage error +0.0143 on average —
later models sit *below* the earlier boundary), while MATH Lvl 5 and IFEval
**under**-cover (-0.0248), i.e. later-period
models exceed the earlier predicted boundary more often than the 2% the estimator
targets. For MATH Lvl 5 the excess is significant at
p = 1.7e-18.

## Setup common to every claim

The paper models a *capability boundary*: the conditional τ-quantile of a
post-trained model's benchmark score given the log pre-training compute of its
base model. Its estimator (Section 2.1) minimises a smoothed pinball loss

```
L(theta) = sum_i  l_tau(y_i - q(z_i; theta))  +  lambda * Omega(theta)
l_tau(u) = (1/kappa) * log(1 + exp(kappa*u)) + (tau - 1) * u
q_tau^sig(z; theta) = y0 + L * sigma(a + beta*z),   beta >= 0, 0 <= y0, y0 + L <= 1
```

with `tau = 0.98`, `kappa = 50`, `lambda = 1e-3`, and the compute proxy

```
x = 6 * (pre-training tokens in T) * (#params in B)     [units of 1e21 FLOPs]
z = log10(x)                                            so 10^24 FLOPs <-> x = 1000
```

Two independent computations back every number below:

1. **An independent re-implementation** (`psl_core.py`), written from the
   equations above rather than from the authors' package: multi-start L-BFGS-B on
   an unconstrained reparameterisation that enforces `beta >= 0` and
   `0 <= y0, y0 + L <= 1` by construction.
2. **A re-run of the authors' released code**
   (github.com/hlzhang109/prescriptive-scaling) on the same tables, on this
   machine, quoted alongside as `authors' code`.

The data are the authors' released evaluation tables: 4576 Open LLM Leaderboard
v2 rows, **3663** of which carry the token/parameter metadata needed for the
compute proxy, plus the Artificial Analysis MATH-500 and AIME-2025 exports.


## Protocol

The four chronological periods of the released pipeline are
`<=2024-06`, `2024-07..09`, `2024-10..12`, `2025-01..03`. For each rolling pair
(`P_t`, `P_(t+1)`), t = 1, 2, 3, the boundary is fitted on `P_t` alone and
evaluated on `P_(t+1)` restricted to the training z-range. Signed coverage error
is `hat_tau - tau`; negative means under-coverage, i.e. newer models breaking
through the older boundary.

## Out-of-period coverage

| benchmark | group | signed coverage (mine) | signed coverage (authors' code) | share above boundary | binomial p (share > 2%) | in-sample signed coverage |
| --- | --- | --- | --- | --- | --- | --- |
| IFEval | drift | -0.0184 | -0.0184 | 2.56% | 2.73e-02 | +0.0150 |
| BBH | stable | +0.0124 | +0.0124 | 0.84% | 1.00e+00 | +0.0173 |
| MATH Lvl 5 | drift | -0.0311 | -0.0311 | 4.82% | 1.67e-18 | +0.0060 |
| GPQA | stable | +0.0171 | +0.0171 | 0.38% | 1.00e+00 | +0.0193 |
| MUSR | stable | +0.0154 | +0.0154 | 0.50% | 1.00e+00 | +0.0179 |
| MMLU-Pro | stable | +0.0121 | +0.0121 | 0.61% | 1.00e+00 | +0.0162 |

The in-sample column is the internal control: every task, including the two
drifting ones, is well calibrated on the period it was fitted on. The
under-coverage is therefore a failure of *temporal transfer*, not of fitting.

### Per-split detail (signed coverage error, t = 1, 2, 3)

| benchmark | t=1 | t=2 | t=3 |
| --- | --- | --- | --- |
| IFEval | -0.0665 | -0.0079 | +0.0192 |
| BBH | +0.0137 | +0.0155 | +0.0080 |
| MATH Lvl 5 | -0.0496 | -0.0123 | -0.0314 |
| GPQA | +0.0200 | +0.0178 | +0.0136 |
| MUSR | +0.0158 | +0.0178 | +0.0128 |
| MMLU-Pro | +0.0052 | +0.0144 | +0.0168 |

MATH Lvl 5 under-covers on **all three** splits — a persistently advancing
boundary. IFEval under-covers sharply on the first split and recovers by the
third, which matches the paper's own hedge that IFEval departs "to a lesser
extent" than MATH.

## Negative control

If the effect is temporal, randomising which models belong to which "period"
must destroy it. Over 5 random relabellings the drift/stable separation collapses
from **+0.0390** to **+0.0091**, with the two
drifting benchmarks no longer distinguishable from the stable four
(MATH +0.0030,
IFEval +0.0125 under shuffled
time). Control broke as required: **True**.

## Limitations

- This is a **reanalysis of released evaluation outputs**, not a fresh evaluation
  run. No language model was downloaded, trained, or scored here; the benchmark
  accuracies are the authors' published numbers. What is independently
  reproduced is the *estimator and every statistic derived from it*, not the
  underlying benchmark measurements.
- The compute axis is a proxy, `6 * tokens * params`, and it exists for only
  3663 of the 4576 leaderboard rows. Models missing token counts are dropped, as
  in the released pipeline. Any bias in which models report token counts is
  inherited.
- Three rolling splits is a small number of temporal replicates.
  The MATH result is consistent in sign across all three and significant when
  pooled, but IFEval's is driven by the first split, so "IFEval is
  non-stationary" rests on one of three comparisons.
- Under-coverage is evidence *consistent with* an advancing boundary; it does
  not by itself distinguish a genuinely improving post-training frontier from
  compositional shift in which model families enter the leaderboard, or from
  benchmark contamination on MATH. The paper's Remark 1 makes the same point.
- The binomial test treats models as independent draws. Models sharing a base
  model share compute and are correlated, so the quoted p-values are
  anti-conservative; the sign and the ordering across tasks are the robust part.