Results
Every model's 8 attempts with web search over a copy of the web frozen at October 1, 2026, on the scale that runs from the task's baseline at 0% to its ceiling at 100%.
Artifacts
What the top model on this task, by mean score, built.
The task
Optimize a single Python file that maintains an incremental Parquet table. Each call starts a fresh process with one batch of mutations and one file of queries. Each mutation is an UPSERT or a DELETE. For each record, the greatest (event time, sequence ID) wins across calls, a delete stops older events from restoring the record, and exact duplicates have no effect. After each call, every live row must be in the table directory as plain Parquet, and every query answered: lookups by record, customer or event ID, region sums, category counts, filtered time ranges and ranked slices. The output directory persists within a stream and holds all state and temporary files. Only the submission file reaches grading, so dependencies must be in it or already in the verifier. Each call gets 2 CPUs, 1,024 MB of memory, 4,096 MB of storage and 20 seconds. Threads and in-process native libraries are allowed; child processes, external services, symlinks and special files are not. A workload generator is provided for development.
Trajectories
Step through all 112 attempts from the pass@8 sweep with web search over an October 1, 2026 snapshot: every turn, tool call and submission. The best reward was 80.0%, by GPT-6 Astra on attempt 4. The results above come from these runs.
Failure analysis
Claude Opus 5.5 averages 0.473 across its 8 attempts. Its five finished designs were fast and lean on memory, but none cut bytes written much on the graded streams, and three attempts crashed outright. Most of Opus's failures come from never testing an unusual 1-million-row first batch at full size, and from rewriting the whole table on every update instead of only the parts that changed.
- Never tested an unusual 1M-row first batch at full size: killed on graded dataFound in 3 of 8 attempts31.9
- Rewrote the whole table on every update instead of only the parts that changedFound in 3 of 8 attempts10.5
- Saw its write savings shrink in its own tests, kept the design: tuned on the default stream shapeFound in 2 of 8 attempts10.3
- ScoredMean reward 0.47347.3
Example failure analysis
Attempt 1 · reward 0.150 · Never tested an unusual 1M-row first batch at full size: killed on graded data
Estimated its unsorted-batch fallback at 350 MB instead of running it at full size, and tested fragments with only 30,000 rows.
- What it did
- It built a native engine with a fast streaming path for sorted first batches and a fallback that loads the whole batch for unsorted or duplicate input. It estimated that fallback at about 350 MB but never ran it at full size. Its fragment-directory test used a 30,000-row first batch. It submitted at 1.7h of 4h with an estimate of 0.84, assuming the hidden streams look like the generator's.
- What would have worked
- Run the fallback path on a shuffled 1M-1.5M-row first batch, split into pieces, under a 1 GB cap before submitting. Another attempt of the same model timed a shuffled 1M-row first batch (1.0 s, 180 MB) and survived every graded stream.
- Evidence
- Step 87
an unsorted initial batch triggers the fallback path with full materialization of the batch plus a numpy copy plus another full copy during the dedupe's lexsort-and-gather step, pushing peak memory to roughly 350MB
- Step 91
first = make_initial_batch(rng, 30000, 1)
- Step 139
My overall estimate is about 0.84, assuming the graded streams behave like the bundled generator.
- Step 87
Why it is hard
Change-data-capture events arrive late, out of order and more than once, deletes must stick, and after every batch the table must be complete plain Parquet. The starter is already correct and tuned in the obvious places, so the remaining choices pull against each other. A layout that suits lookups by record ID can force many file rewrites when updates cluster on other attributes. Indexes save scans but cost memory and writes, and all three are scored. Polars and DuckDB give fast scans and Parquet I/O but do not choose a persistent layout. Delta deletion vectors and Iceberg delete files rely on readers applying deletion metadata; here an ordinary Parquet read must return exactly the live rows.
How it is graded
Grading replays three fixed streams, each beginning with 500,000 to 1,500,000 rows, and checks the table and every answer after every batch. The unchanged starter runs alongside, and both are measured on latency including startup and imports, peak resident memory, and bytes written. Per call, writes are the greater of the bytes passed to write calls, temporary, overwritten and deleted files included, and the new or changed output pages at 4 KiB each. Latency and writes are summed per stream, and each metric is the median across streams. Any wrong table or answer, invalid Parquet file, crash or limit violation scores the floor, 0.15.
T = 0.30 for latency, 0.45 for memory, 0.55 for writes
reward = 0.55·latency + 0.20·memory + 0.25·writes
Anchors
| Anchor | Reward | Score | Meaning |
|---|---|---|---|
| Floor | 0.150 | 0% | What a missing, invalid or rule-breaking submission scores |
| Baseline | 0.163 | 0% | The untouched starting point |
| Ceiling | 1.000 | 100% | The best result the metric allows |
| Regrade tolerance | ± 0.080 | Results closer than this tie |
Environment
- Domain
- Software · Data engineering
- Shape
- Scaffolded
- Agent time
- 4 h per attempt
- Hardware
- 2 vCPU · 1 GB
- Expert estimate
- 3 h
- Task ID
stateful-parquet