Results
Every model's 8 attempts with web search over a copy of the web frozen at October 1, 2026, on the scale that runs from the task's baseline at 0% to its ceiling at 100%.
Artifacts
What the top model on this task, by mean score, built.
The task
Make an exact join-analytics engine use less CPU time. It reads one database and a batch of conjunctive queries on standard input and writes grouped aggregates to standard output. The deliverable is a static Linux ELF64 executable of at most 256 MiB, at /app/engine. Each database has eight relations of (left, right, payload) rows; duplicates count separately. A query is a connected graph of variables, each with an interval filter, whose edges are labeled by relations. For every joined row, COUNT adds 1 and SUM adds the selected payloads, in unsigned 64-bit arithmetic, into one of 8 buckets set by the grouping variable. Each timed run is a fresh process on one CPU with a 512 MiB address space, so start-up and parsing count. Scoring uses a small and a large private database from each of six data families, and answer checks also cover valid inputs outside the generator. Any wrong answer scores 0; otherwise each database is scored by its speedup over the starter against the target below.
| Family | Small | Large |
|---|---|---|
| Uniform | 6.6× | 15.2× |
| Hubs | 5.0× | 12.3× |
| Communities | 9.2× | 22.3× |
| Correlated | 8.3× | 23.6× |
| Selective | 20.9× | 68.6× |
| Mixed | 6.1× | 16.0× |
Trajectories
Step through all 112 attempts from the pass@8 sweep with web search over an October 1, 2026 snapshot: every turn, tool call and submission. The best reward was 78.1%, by Claude Fable 5.2 on attempt 7. The results above come from these runs.
Failure analysis
Gemini 4 Argon averages 0.346 across its 8 attempts. Most of Argon's failures come from changing its code at the last minute and submitting without rechecking its answers, and from using a memory-saving shortcut that breaks on large numbers without testing it on large numbers.
- Changed its code at the last minute and submitted without rechecking its answersFound in 2 of 8 attempts25.0
- Used a memory-saving shortcut that breaks on large numbers, without testing it on large numbersFound in 2 of 8 attempts25.0
- Kept sending oversized edits after output-limit warningsFound in 1 of 8 attempts4.6
- Dropped a suspected start-up cost after one check; fixed it lastFound in 1 of 8 attempts4.1
- Took a passing answer check as done; left its slowest small cases unworkedFound in 1 of 8 attempts3.6
- Best attempt: fixed start-up costs first and rechecked the final buildFound in 1 of 8 attempts3.2
- ScoredMean reward 0.34634.6
Example failure analysis
Attempt 7 · reward 0.000 · Changed its code at the last minute and submitted without rechecking its answers
It merged a new input parser and kernel rewrites 20 minutes before the end and checked them only on tiny inputs.
- What it did
- From 3.4h it rebuilt input parsing around the exact spacing the generator produces, and hit several wrong-answer bugs on the way (steps 189-191). At step 200 it merged the new parser, a new de-duplication pass and a rewritten diamond kernel into the engine. At step 201 its brute-force check was 12 tiny generator inputs (12 queries each) plus 6 hand-written queries on one 5-key database, all in the generator's exact format; it also matched the starter on the 12 benchmark databases. It submitted at 3.61h.
- What would have worked
- After a rewrite this broad, run hundreds of small random queries against the brute-force checker, with inputs outside the generator's format. The correct attempts re-ran a 30-case or custom edge-case audit on the final binary.
Why it is hard
A cyclic join can create intermediate results far larger than its inputs, and skew and correlation make a plan that works well on one database perform poorly on another. Duplicate rows and grouped aggregates leave no slack: every joined row's contribution has to survive into COUNT and SUM exactly, modulo 264. The starting engine is already strong, with sparse bitmap-rank indexes, aggregate factorization, cycle kernels and conditional forest messages. The queries are connected graphs of up to 8 variables and 12 edges, from triangles and four-, six- and eight-cycles to cliques, theta graphs, ladders and K3,3, and any plan has to hold up across distributions and sizes.
What the agent starts with
The agent gets the working indexed C++ starter, a build helper for optimized static binaries, and a specification of the input and output protocol, the input bounds and each database's scoring target. It also gets the published workload generator and sandbox code for public development data (scoring uses private databases), an evaluator for public feedback on any family, and 4 CPUs, 8 GB, g++ and public internet access.
How it is graded
The verifier freezes the executable and runs it unprivileged in a sandbox on one CPU, with no network. Correctness comes first: answers are checked on audit inputs, including valid inputs outside the generator, and against trusted answers for all 12 private databases. A missing or invalid executable, or any wrong, malformed, crashing, timed-out or resource-exhausting run, scores 0 for the whole trial. Then, on each database, five paired rounds alternate the starter and the candidate, each running fresh processes for at least 50 ms of CPU time, and each executable's median time counts. Matching the starter earns 0, reaching the target 1, and reaching its square root 0.5. Each target is fixed, from a cost model of the workload, and is not a claim about the best possible implementation.
reward = mean of scoredb over the 12 databases
Anchors
| Anchor | Reward | Basis |
|---|---|---|
| Floor | 0 | A missing, invalid or wrong submission, a crash or a resource failure; a correct engine no faster than the starter on any database also scores 0 |
| Baseline | 0.0030 | The unchanged starter submitted as the candidate, timed against itself; it sits just above 0 only through timing noise |
| Ceiling | 1 | Every database at or above its target speedup |
| Regrade tolerance | ± 0.05 | Allowed difference between regrades of the same executable, from CPU timing noise |
Environment
- Agent time
- 4 hours
- Hardware
- 4 CPUs and 8 GB RAM for development; graded on 1 CPU with a 512 MiB address space, possibly on a different x86-64 CPU
- Network
- Public internet while working; none during grading
- Software
- Debian bookworm with g++ for static C++20 builds, on x86-64 or arm64
- Verifier
- Separate container; 1 CPU, 2 GB, 70-minute limit
- Expert estimate
- 3 hours of profiling, implementation and testing
Prior work
Public methods give a foundation: Veldhuizen's Leapfrog Triejoin, a worst-case optimal join algorithm (arXiv:1210.0481); the InsideOut algorithm from Abo Khamis, Ngo and Rudra's FAQ: Questions Asked Frequently (arXiv:1504.04044); Wang, Willsey and Suciu's Free Join, which unifies worst-case optimal and traditional joins (SIGMOD 2023); the engines DuckDB and Kùzu; and LMFAO, for batches of aggregates. Applying them still means choosing plans, indexes and aggregate representations, and testing those choices across distributions and filters.