# A cross-benchmark shift test applied to AIME-2025 scores finds a positive but not statistically significant shift (p=0.15), giving no clear aggregate evidence of post-release contamination inflating math benchmark scores (Section 5.2.2, Equation 3).

**Verdict: reproduced.**

The regression of Equation 3 on the released Artificial Analysis exports gives
γ = **0.5757** (positive, odds ratio
1.78) with a one-sided stratified-permutation
p-value of **0.1484**, on **90**
models in the common support — matching the paper's reported p = 0.15 and
n = 90. Positive, not significant at 0.05.

## Setup common to every claim

The paper models a *capability boundary*: the conditional τ-quantile of a
post-trained model's benchmark score given the log pre-training compute of its
base model. Its estimator (Section 2.1) minimises a smoothed pinball loss

```
L(theta) = sum_i  l_tau(y_i - q(z_i; theta))  +  lambda * Omega(theta)
l_tau(u) = (1/kappa) * log(1 + exp(kappa*u)) + (tau - 1) * u
q_tau^sig(z; theta) = y0 + L * sigma(a + beta*z),   beta >= 0, 0 <= y0, y0 + L <= 1
```

with `tau = 0.98`, `kappa = 50`, `lambda = 1e-3`, and the compute proxy

```
x = 6 * (pre-training tokens in T) * (#params in B)     [units of 1e21 FLOPs]
z = log10(x)                                            so 10^24 FLOPs <-> x = 1000
```

Two independent computations back every number below:

1. **An independent re-implementation** (`psl_core.py`), written from the
   equations above rather than from the authors' package: multi-start L-BFGS-B on
   an unconstrained reparameterisation that enforces `beta >= 0` and
   `0 <= y0, y0 + L <= 1` by construction.
2. **A re-run of the authors' released code**
   (github.com/hlzhang109/prescriptive-scaling) on the same tables, on this
   machine, quoted alongside as `authors' code`.

The data are the authors' released evaluation tables: 4576 Open LLM Leaderboard
v2 rows, **3663** of which carry the token/parameter metadata needed for the
compute proxy, plus the Artificial Analysis MATH-500 and AIME-2025 exports.


## Protocol (paper Equation 3)

```
logit(0.01 * y_i) = alpha + beta * logit(0.01 * m_i) + gamma * 1{post-AIME} + eps_i
```

`y_i` is model *i*'s AIME-2025 accuracy, `m_i` its MATH-500 accuracy, and the
indicator marks models released on or after **2025-02-06**, the AIME-2025
release date. Since every model post-dates MATH-500, MATH-500 inflation moves all
points, whereas AIME-2025-specific contamination should lift only the
post-release group; a positive γ is the contamination signature.

Merging the two exports on `model_id` gives 115 models with both
scores and a release date. Restricting to the common support of
`logit(MATH-500)` across the two release groups — the guardrail that stops the
comparison being driven by extrapolation — leaves **90**
(28 pre-release, 62 post-release).

## Result

| quantity | mine | authors' code | paper |
| --- | --- | --- | --- |
| n merged | 115 | 115 | — |
| n in common support | 90 | 90 | 90 |
| slope β | 1.3557 | 1.3557 | — |
| group shift γ | 0.5757 | 0.5757 | positive |
| s.e.(γ) | 0.3261 | 0.3261 | — |
| t(γ) | 1.7652 | 1.7652 | — |
| one-sided p | 0.1484 | 0.1484 | 0.15 |
| R² | 0.5289 | 0.5289 | — |

γ > 0: **True**. Significant at 0.05:
**False**. Both halves of the claim hold.

The p-value is a one-sided stratified permutation test: the post-release
indicator is permuted within 10 quantile bins of `logit(MATH-500)` (2000
permutations), so the null preserves the compute/skill composition of the two
groups.

## Negative controls

**The test can see a real shift.** Injecting a synthetic +0.8 logit inflation
into every post-cutoff AIME score and re-running the identical pipeline recovers
γ = **1.376** at p = **0.0005**. A
null result from a test with no power would be uninformative; this one has power
against the effect it is designed to detect. Control broke as required:
**True**.

**The estimate is not an artefact of the design matrix.** Randomising the group
label entirely over 200 draws gives mean γ = +0.0574
(sd 0.3282), i.e. centred on zero, against the observed
0.5757. Control broke as required:
**True**.

## Limitations

- This is a **reanalysis of released evaluation outputs**, not a fresh evaluation
  run. No language model was downloaded, trained, or scored here; the benchmark
  accuracies are the authors' published numbers. What is independently
  reproduced is the *estimator and every statistic derived from it*, not the
  underlying benchmark measurements.
- The compute axis is a proxy, `6 * tokens * params`, and it exists for only
  3663 of the 4576 leaderboard rows. Models missing token counts are dropped, as
  in the released pipeline. Any bias in which models report token counts is
  inherited.
- A non-significant γ is not evidence of absence. With
  90 models and s.e.(γ) = 0.326, the one-sided
  95% interval still admits shifts up to
  γ ≈ 1.11 logits, so
  contamination of moderate size cannot be excluded — which is what the paper
  says.
- The point estimate is positive and roughly 1.8 standard errors from zero. It is
  a null result at the 0.05 threshold, not a demonstration that γ = 0.
- The test assumes the logit-logit relation between the two benchmarks is
  otherwise stable across the release boundary. Any genuine post-2025-02-06
  improvement in AIME-style reasoning relative to MATH-500 — for instance the
  arrival of long-chain-of-thought reasoning models — would load onto γ exactly
  like contamination. This is a limitation of the diagnostic, not of the
  reproduction.
- Release dates come from the Artificial Analysis export; the 25 models dropped
  by the common-support guardrail are dropped by the released code as well.