Results
Every model's 8 attempts with web search over a copy of the web frozen at October 1, 2026, on the scale that runs from the task's baseline at 0% to its ceiling at 100%.
Artifacts
What the top model on this task, by mean score, built.
The task
Make one call of a 3D acoustic wave propagator as fast as possible on an H100. A call advances T leapfrog steps of an 8th-order (radius 4) or 4th-order (radius 2) stencil with a 16-point absorbing sponge, injects Ricker-wavelet point sources and records the wavefield at receiver points. The submission is two files: kernels.cu, device-only CUDA C++ of at most 1 MiB that the grader compiles with NVRTC, and plan.json, at most 64,000 bytes, which says which kernels to launch for each class, with what grid and arguments; scratch is limited to 8 GiB per plan variant. Only these files are graded. None of the agent's host code runs: the grader makes every launch from the plan and replays it as a CUDA graph. External calls and device-side launches are not allowed, writable globals are restricted, and the outputs must match a float64 reference closely. Eleven equally weighted classes are timed:
| Class | Grid | Radius | Steps |
|---|---|---|---|
| cube | 512 × 512 × 512 | 4 | 2, 4, 8, 16 |
| cube | 512 × 512 × 512 | 2 | 2, 4, 8 |
| wide | 1024 × 1024 × 128 | 4 | 4 |
| tall | 256 × 256 × 2048 | 4 | 4 |
| survey | 721 × 463 × 301 | 4 | 4 |
| small | 160 × 160 × 160 | 4 | 4 |
Trajectories
Step through all 112 attempts from the pass@8 sweep with web search over an October 1, 2026 snapshot: every turn, tool call and submission. The best reward was 40.7%, by Claude Opus 5.5 on attempt 3. The results above come from these runs.
Failure analysis
Claude Opus 5.5 averages 0.231 across its 8 attempts. Most of Opus's failures come from assuming its design was near the limit and ending early without trying faster designs on its weakest workloads, and from ruling out more steps per pass on the long workloads from paper estimates without testing it.
- Assumed its design was near the limit and ended early without trying faster designs on its weakest workloadsFound in 4 of 8 attempts41.9
- Ruled out more steps per pass on the long workloads from paper estimates, without testing itFound in 3 of 8 attempts24.1
- Spent its first 3.6 hours debugging a complicated design before submitting a simple working oneFound in 1 of 8 attempts10.9
- ScoredMean reward 0.23123.1
Example failure analysis
Attempt 7 · reward 0.093 · Assumed its design was near the limit and ended early without trying faster designs on its weakest workloads
Its radius-4 fused kernels ran slower than one step at a time; it capped fusion's gain at 20% on paper, called its design near optimal, and quit with radius 4 on one-step kernels and survey on the starter.
- What it did
- It first submitted a one-step kernel plus a two-step kernel for radius 2 (step 123). Its two-step and wavefront fused kernels ran slower than its one-step kernel at radius 4, and it estimated that fusion could gain at most about 20% there (step 171), so it used the wavefront design only for the radius-2 cubes. At step 362 it noted all its radius-4 fused designs ran slower than one step, worked through register and cache budgets for further layouts on paper, and called its configuration near the practical optimum, believing about six hours were left. It then dropped a survey fix as low upside and confirmed with 7 h 40 min left (step 363); survey was still on the starter kernel.
- What would have worked
- Find out why its radius-4 fused kernels were slow instead of stopping on a paper cap with 7.7 hours left: the best attempt made two-step fusion at radius 4 pay off (radius-4 T=2 at 0.86 vs 0.60 here). Use the remaining hours on survey, wide and tall too, which sat at 0.18 to 0.30 against 0.35 to 0.55 in the best attempt.
Why it is hard
This propagator is the inner loop of reverse-time migration and full-waveform inversion, behind seismic exploration and CO₂-storage monitoring. One survey runs it many thousands of times, and HPC engineers retune it for every GPU generation. A single time step is memory bound: it reads two wavefield levels and the velocity model, writes one, and does little arithmetic per byte. The score's copy floor is the least traffic a whole call can move, 20 bytes per grid point, with each input read once and the last two levels written once. The longest cube runs are bound by arithmetic instead, the survey grid's 2,884-byte rows break 16-byte alignment on three rows out of four, and on the 160³ grid, about the size of L2, fixed costs matter more than main memory.
What the agent starts with
The agent starts from a slow but correct CUDA version in place as the submission, plus a spare copy, and a short instruction with the scoring formula, the accuracy gate and the limits. The plan schema is in plan.py, the classes and stencil constants in classes.py, the input layouts in gen.py, the wave update and outputs in ref.py and the rules for globals in scan.py; check.py and bench.py run the grader's own code for validation, the gate and timing. It has one H100, the part used for grading, in a CUDA 12.8 development image with NVRTC, PyTorch, profilers and public internet access.
How it is graded
A compile step validates the plan and scans the cubin for external calls, device-side launches and disallowed writable globals. On the grader's own H100, each class is then timed over 30 rounds, or 60 on the small grid, each with fresh data and a flushed L2, as CUDA-graph replays timed with GPU events; the score uses medians. A class's roofline is the longer of two trusted times: the copy floor, the fastest copy of the minimum 20 bytes per point, and the call's FP32 arithmetic time. Every run must pass the accuracy gate: finite outputs within relative L2 and maximum-error bounds of a float64 reference that scale with T, with a tighter bound on the interior. A rejected file, a gate failure, changed inputs or a time that stays more than 5% faster than the copy floor scores 0.
task reward p = clip((E − S) ÷ (0.56 − S), 0, 1), with S the starter's E, about 0.19
leaderboard score = −ln(1 − p) ÷ ln 20, or 1 once p ≥ 0.95
Anchors
| Anchor | Reward | Basis |
|---|---|---|
| Floor | 0 | A rejected file, a gate failure, a crash or a submission-caused timeout |
| Baseline | 0 | The unmodified starter, at p = 0 |
| Ceiling | 1 | p ≥ 0.95: 95% of the gap closed between the starter's E and the bound of 0.56 |
| Regrade tolerance | ± 0.017 | Allowed difference between gradings of one submission, on the gap-halving scale |
Environment
- Agent time
- 12 hours
- Hardware
- 1× H100, the exact part used for grading; 16 CPUs; 64 GB RAM
- Network
- Public internet while working; none during grading
- Software
- CUDA 12.8 development image on Ubuntu 24.04, NVRTC 12.8, PyTorch 2.11
- Verifier
- Separate container on its own H100; 8 CPUs, 64 GB, 90-minute limit; a normal grading takes 7 to 16 minutes
- Expert estimate
- 12 hours
Prior work
Related public work exists, but none of it is in this format. It includes Micikevicius, 3D finite difference computation on GPUs using CUDA (2009), and Nguyen et al., 3.5-D blocking optimization for stencil computations (SC 2010). Shan and Araya-Polo tuned GPU kernels for this stencil (arXiv:2404.04441) but released no code, and the CPE21 artifact of Sai et al. has radius-4 GPU kernels with a PML and one source but no receivers. Other work includes Deepwave, the FDTD3d sample in NVIDIA CUDA Samples, the Devito finite-difference code generator for seismic modelling, the GPU stencil tools AN5D, EBISU and PERKS, and ConvStencil and its successors on tensor cores.