Results
Every model's 8 attempts with web search over a copy of the web frozen at October 1, 2026, on the scale that runs from the task's baseline at 0% to its ceiling at 100%.
Artifacts
What the top model on this task, by mean score, built.
The task
The goal is a faster FP64 four-center J/K build for gpu4pyscf.scf.hf.RHF.get_jk on one H100. The agent gets GPU4PySCF built at a pinned commit, the published workload distribution and a public checker that builds the patch, applies the gates and times four public workloads. Only the Rys-quadrature J/K library may change: its sources and the gvhf_rys source list in its CMakeLists.txt. The verifier applies the difference from the pin as a patch and rebuilds it in Release mode for sm_90 only. PTX, other architectures, JIT compilation, runtime module loading, child processes, device-side launches, unlisted host symbols and raw system calls are all ruled out. The 12 scored workloads are chains, rings, clusters and transition-metal complexes built from 16 elements between hydrogen and nickel, with 150 to 1,000 atomic orbitals in cc-pVQZ, def2-QZVP or a heavily contracted QZ basis, spherical or Cartesian. One uses two density matrices. Every J and K entry must stay within 10−9 of a CPU reference, and the score grows with the speedup over the pin.
Trajectories
Step through all 112 attempts from the pass@8 sweep with web search over an October 1, 2026 snapshot: every turn, tool call and submission. The best reward was 27.2%, by Claude Fable 5.2 on attempt 5. The results above come from these runs.
Failure analysis
Claude Opus 5.5 averages 0.144 across its 8 attempts. Every attempt passed all the accuracy checks and made the code 5.7 to 9.0 times faster, but didn't come close to the 20.5 times needed for full marks. Instead of iterating on the slowest type of molecule, cc-pVQZ, Opus focused on tweaking settings and speeding up molecules that were already fast, or thinking that the code's speed had plateaued and stopping early.
- Saw the slow cc-pVQZ kernels in its profiling, then chose small tuning over rewriting themFound in 4 of 8 attempts43.3
- Believed its kernels were at a ceiling; never tried the cc-pVQZ fix it had namedFound in 2 of 8 attempts20.8
- Timed the original code differently from the grader, so it thought it was nearly doneFound in 1 of 8 attempts11.0
- Kept redesigning the cc-pVQZ kernels to the end; still far below the target speedFound in 1 of 8 attempts10.5
- ScoredMean reward 0.14414.4
Example failure analysis
Attempt 3 · reward 0.092 · Saw the slow cc-pVQZ kernels in its profiling, then chose small tuning over rewriting them
Profiling pointed at the uncontracted cc-pVQZ kernels, but it chose more tuning because 'tuning changes have been reliable wins' and never rewrote them.
- What it did
- It built per-class kernels and autotuned about 2,800 variants, reaching its own estimate of G about 5.9 by 6.6h (step 343). At 12h it listed the mid-L uncontracted WS/LT2 kernels and the high-L team kernel as the main candidates, weighed rebalancing the WS kernel, then chose a broader tuning pass (steps 464-465). Its later submissions were variant tables and memsets with A/B results at noise level (step 492). It noted cc-pVQZ at 3.8-4.1x and called the remaining options low-yield (step 505); after a 'Continue' prompt it added memset, estimator and caching tweaks worth about 1% until 17.4h.
- What would have worked
- Use its last 5 hours to redesign the uncontracted cc-pVQZ kernels it had identified. The attempt with the fastest cc-pVQZ kept changing its warp-level kernels to the end and reached 6.1-6.7x on most cc-pVQZ molecules, against this attempt's 2.9-4.3x.
- Evidence
- Step 464
Since tuning changes have been reliable wins before, I'm weighing whether rebalancing the WS kernel's producer/consumer load (moving part of the K-reduce to the producer) is worth the occupancy risk
- Step 492
p30 = PQ variants for best maxreg-64 team config; A/B vs p28 ~-0.3% (noise level)
- Step 505
the remaining options are low-yield and would each need roughly two hours to retune with uncertain payoff
- Step 464
Why it is hard
Self-consistent field (SCF) calculations build the Coulomb (J) and exchange (K) matrices from four-center electron-repulsion integrals, and this J/K build is the bottleneck when users run SCF on large molecules. GPU4PySCF evaluates the integrals with Rys quadrature in FP64 CUDA kernels. The workloads use quadruple-zeta basis sets with shells up to g, where the kernels are register-bound and each angular-momentum class needs its own tuning. The 10−9 accuracy bar leaves little room to trade precision for speed. There is also little to borrow: upstream GPU4PySCF has not changed these kernels since the pinned commit, and libraries such as QUICK and LibintX would need changes outside the editable directory.
How it is graded
The verifier rebuilds the patch on its own H100 and checks its scope, symbols and device binaries. On each of the 12 workloads, pin and candidate alternate over five rounds, timed by GPU busy time from CUPTI. Each candidate sample is credited with at least a fixed share of its device-active time, so hiding work from the profiler does not pay. J and K must match CPU PySCF to 10−9 and be repeatable, and the SCF energy must match to 10−7 hartree. Host CPU and wall-time limits stop work from moving to the CPU. Any failed gate scores 0. The agent is told reward = clip(ln G ÷ ln 20.54, 0, 1). The leaderboard instead credits each halving of the gap to 20.54 equally, with full credit at 95% of it (G ≈ 19.56).
p = (G − 1) ÷ (20.54 − 1); score = −ln(1 − p) ÷ ln 20, from 0 at p ≤ 0 to 1 at p ≥ 0.95
Anchors
| Anchor | Reward | Score | Meaning |
|---|---|---|---|
| Floor | 0.000 | 0% | What a missing, invalid or rule-breaking submission scores |
| Baseline | 0.000 | 0% | The untouched starting point |
| Ceiling | 1.000 | 100% | The best result the metric allows |
| Regrade tolerance | ± 0.030 | Results closer than this tie |
Environment
- Domain
- Science · Chemistry
- Shape
- Scaffolded
- Agent time
- 18 h per attempt
- Hardware
- 1× H100 · 16 vCPU · 64 GB
- Expert estimate
- 24 h
- Task ID
gpu4pyscf-jk-speed