Results
Every model's 8 attempts with web search over a copy of the web frozen at October 1, 2026, on the scale that runs from the task's baseline at 0% to its ceiling at 100%.
Artifacts
What the top model on this task, by mean score, built.
The task
The agent gets a correct but slow attention engine, in which each query-key pair also has a sigmoid gate, and one NVIDIA B300 GPU. It may replace any component and reuse compatible public code. A runtime package specifies the interface, limits, reference, audits, workloads and scoring; its check and bench commands run the grader's audits and time public workloads. The deliverable is /app/submission, a Python package whose __init__.py exports build_engine(config). The engine runs training (forward, and backward with five gradients), prefill into a key/value cache, and decode steps appending 1 to 8 tokens to sequences that are created, forked and released. Twelve workload strata, four per mode, cover head sizes 64 and 128, gate sizes 32 and 64, grouped key/value heads, sequences up to 16,384 tokens and two memory tiers. Inputs must stay unchanged, and memory under a per-workload ceiling. It is graded as is, offline, with no build step and an empty compile cache, so it must bundle any extra dependency, within 20,000 files and 8 GiB.
Trajectories
Step through all 112 attempts from the pass@8 sweep with web search over an October 1, 2026 snapshot: every turn, tool call and submission. The best reward was 91.8%, by Claude Fable 5.2 on attempt 4. The results above come from these runs.
Failure analysis
GPT-6.1 Sol averages 0.399 across its 8 attempts. Its engines improved training, but prefill lagged behind. Most of Sol's failures come from stopping improvements with hours left and repeatedly re-testing existing code instead of optimizing on prefill, or relying only on small tweaks after its first prefill redesigns came out slower.
- Stopped improving with hours left and only re-tested its codeFound in 3 of 8 attempts20.5
- Fell back to small tweaks after its first prefill redesigns came out slowerFound in 2 of 8 attempts12.9
- Changed a kernel without re-testing it, and crashed on a graded caseFound in 1 of 8 attempts12.5
- Took a passing benchmark as the finish line, with hours leftFound in 1 of 8 attempts7.1
- Spent its last improvement on training, already its fastest modeFound in 1 of 8 attempts7.1
- ScoredMean reward 0.39939.9
Example failure analysis
Attempt 6 · reward 0.369 · Stopped improving with hours left and only re-tested its code
It stopped changing code with 5.5 hours left, believing a hidden bug was the only remaining risk, and spent the time re-running full test suites, two of them for older versions it would never submit. Prefill stayed at about half its training speed.
- What it did
- At 11.2 hours Sol found a real win, a pointer alignment hint that vectorized cache loads, and submitted it as V24 at 11.5 hours. Full 96-workload suites for the older V21 and V23 were already running; it added one for V24 and wrote that the old suites would run with no cancellation. Its last prefill probes (new tile shapes, register budgets, masked math) each gave no reliable gain (ties under 1%, or regressions), and it had already ruled out a two-CTA (two-block) prefill kernel. At 12.1 hours it declared no more features unless a suite failed. All suites passed; its final numbers showed prefill at 3.64x against 6.98x for training, after 4.6 hours of waiting.
- What would have worked
- Validate V24 once, cancel the outdated suites, and spend the freed GPU hours on a structural prefill change such as the two-CTA kernel it had ruled out. Sol's best attempt reached 4.05x on graded prefill vs 3.42x here.
- Evidence
- Step 600
DISCARDbulktilechange, keepq2M128N64. NoMainCUTEgeometrychange/No2CTAprobe.
- Step 605
V21andV23full96STILLcontinue(nonstop), noCancellation.
- Step 622
Final3full96suitesnowALLGPUpriority, noMorefeaturesunlessfull-suitefindsfailure.
- Step 721
Startercomparisonpublicspeedups6.97516x/3.64233x/4.80220x
- Step 600
Why it is hard
Each query-key pair gets two scores: the usual softmax score, and a sigmoid gate from separate low-dimensional projections that scales the softmax weight after normalization: O = (softmax(QKT/√D) ∘ sigmoid(QgKgT/√R)) V. This breaks the standard fast kernel, which streams over key tiles: each tile needs a second matrix multiply and a sigmoid outside the normalization, and backward has two extra gradients. Peak Blackwell speed needs the new tensor-core path and tensor memory, which Triton only partly reaches; full control takes CUTLASS's CuTe DSL or PTX, with few working examples. Decode is bound by kernel launches and small, changing batches, so it needs its own design. Bitwise-identical repeats rule out the usual atomic reductions in backward. No public kernel computes this gate or serves this cache interface.
How it is graded
The verifier runs offline on its own B300. It first runs 34 audit traces twice each: outputs must match an FP64 reference and be bitwise identical. It then times held-out workloads from every stratum, starter and submission back to back. A workload's latency L is its median controller wall time per trace. Latencies are combined geometrically within strata, across strata, then over the three modes with equal weight, so 3× in training alone is only about 1.44× overall.
score = −ln(1 − p) ÷ ln 20; 0 at p ≤ 0, 1 at p ≥ 0.95
The agent's reward is p clipped to 0–1. The leaderboard's score credits each halving of the gap to 6.2× equally: 3× scores 0.16, 4.9× 0.46, and 5.94× full marks. A wrong or nondeterministic output, failed audit, invalid package, crash, protocol or memory violation, or a submission-caused timeout scores 0. The whole grade has 14,100 seconds, compilation included.
Anchors
| Anchor | Reward | Score | Meaning |
|---|---|---|---|
| Floor | 0.000 | 0% | What a missing, invalid or rule-breaking submission scores |
| Baseline | 0.000 | 0% | The untouched starting point |
| Ceiling | 1.000 | 100% | The best result the metric allows |
| Regrade tolerance | ± 0.110 | Results closer than this tie |
Environment
- Domain
- ML · Kernels
- Shape
- Scaffolded
- Agent time
- 17 h per attempt
- Hardware
- 1× B300 · 8 vCPU · 32 GB
- Expert estimate
- 24 h
- Task ID
b300-attention-engine