# At 10²⁴ FLOPs, the estimated 0.98-quantile boundary reaches 0.539 accuracy on MATH Level 5 versus 0.563 on MMLU-Pro, 0.700 on BBH, 0.828 on IFEval, 0.535 on MUSR, and 0.424 on GPQA (Table 1).

**Verdict: reproduced.**

All six Table 1 cells come back to within **0.0005** accuracy —
i.e. every value rounds to the published three-decimal figure — from an
independent fit, and to within 0.0005 from the authors' released code.

## Setup common to every claim

The paper models a *capability boundary*: the conditional τ-quantile of a
post-trained model's benchmark score given the log pre-training compute of its
base model. Its estimator (Section 2.1) minimises a smoothed pinball loss

```
L(theta) = sum_i  l_tau(y_i - q(z_i; theta))  +  lambda * Omega(theta)
l_tau(u) = (1/kappa) * log(1 + exp(kappa*u)) + (tau - 1) * u
q_tau^sig(z; theta) = y0 + L * sigma(a + beta*z),   beta >= 0, 0 <= y0, y0 + L <= 1
```

with `tau = 0.98`, `kappa = 50`, `lambda = 1e-3`, and the compute proxy

```
x = 6 * (pre-training tokens in T) * (#params in B)     [units of 1e21 FLOPs]
z = log10(x)                                            so 10^24 FLOPs <-> x = 1000
```

Two independent computations back every number below:

1. **An independent re-implementation** (`psl_core.py`), written from the
   equations above rather than from the authors' package: multi-start L-BFGS-B on
   an unconstrained reparameterisation that enforces `beta >= 0` and
   `0 <= y0, y0 + L <= 1` by construction.
2. **A re-run of the authors' released code**
   (github.com/hlzhang109/prescriptive-scaling) on the same tables, on this
   machine, quoted alongside as `authors' code`.

The data are the authors' released evaluation tables: 4576 Open LLM Leaderboard
v2 rows, **3663** of which carry the token/parameter metadata needed for the
compute proxy, plus the Artificial Analysis MATH-500 and AIME-2025 exports.


## Result

The boundary is fitted once on all 3663 models with a compute
proxy (no temporal split, matching the Table 1 caption "the no-split
0.98-quantile sigmoid boundaries") and evaluated at `x = 1000`, i.e. 10²⁴ FLOPs.

| benchmark | mine | authors' code | paper Table 1 | mine − paper | destructive τ=0.50 fit | control − paper |
| --- | --- | --- | --- | --- | --- | --- |
| IFEval | 0.8278 | 0.8278 | 0.828 | -0.0002 | 0.5378 | 0.2902 |
| BBH | 0.6996 | 0.6996 | 0.700 | -0.0004 | 0.5512 | 0.1488 |
| MATH Lvl 5 | 0.5395 | 0.5395 | 0.539 | +0.0005 | 0.2061 | 0.3329 |
| GPQA | 0.4245 | 0.4245 | 0.424 | +0.0005 | 0.3148 | 0.1092 |
| MUSR | 0.5348 | 0.5352 | 0.535 | -0.0002 | 0.4199 | 0.1151 |
| MMLU-Pro | 0.5626 | 0.5626 | 0.563 | -0.0004 | 0.4078 | 0.1552 |

Largest disagreement with the paper: **0.0005**. All six within
0.001: **True**.

## Robustness to the optimiser's homotopy

The released fitter warms up through τ ∈ (0.90, 0.95, 0.98) and
κ ∈ (10, 25, 50). Refitting cold, directly at (τ, κ) = (0.98, 50), moves the
six values by at most
**0.0000**, so the published
numbers are a property of the objective and not of the continuation schedule.

## Destructive control

The registered result depends on the paper's upper-envelope level
`τ = 0.98`. Holding the released data, sigmoid family, compute proxy,
optimiser, and the 10²⁴-FLOP evaluation point fixed while replacing only
that load-bearing level with `τ = 0.50` destroys all six registered values.
The resulting median fits differ from Table 1 by
**0.1919** on average (minimum
**0.1092**, maximum
**0.3329**). All six fail the registered 0.001
tolerance: **True**. Control broke as
required: **True**.

## Limitations

- This is a **reanalysis of released evaluation outputs**, not a fresh evaluation
  run. No language model was downloaded, trained, or scored here; the benchmark
  accuracies are the authors' published numbers. What is independently
  reproduced is the *estimator and every statistic derived from it*, not the
  underlying benchmark measurements.
- The compute axis is a proxy, `6 * tokens * params`, and it exists for only
  3663 of the 4576 leaderboard rows. Models missing token counts are dropped, as
  in the released pipeline. Any bias in which models report token counts is
  inherited.
- 10²⁴ FLOPs is near the top of the observed compute range
  (`z` spans -0.59 to
  4.01 in units of log10(1e21 FLOPs), i.e. up to
  ≈ 10²⁵ FLOPs), so these are interpolated but sparsely supported values. The
  paper is explicit that this is an empirical envelope over the *observed* model
  population, not a physical ceiling.
- The reported figures are for the six "Raw" leaderboard metrics. The
  normalisation matters: the leaderboard also publishes normalised variants of
  the same tasks, and fitting those would not give these numbers.