Results
Every model's 8 attempts with web search over a copy of the web frozen at October 1, 2026, on the scale that runs from the task's baseline at 0% to its ceiling at 100%.
Artifacts
What the top model on this task, by mean score, built.
The task
Write a Python program that turns a PostgreSQL change stream into a shorter SQL script that replays it: after every transaction, the rows must match the stream. The script may use only INSERT, UPDATE and DELETE (a WITH clause is allowed), with one BEGIN and COMMIT pair per transaction, in order, and must parse as PostgreSQL in sqlglot 27.29.0. Built-in functions are allowed; user functions, DO, COPY, temporary tables, disabling triggers, system catalogs and functions that read outside data are rejected. The grader runs the program as a restricted non-superuser role on a database restored to its pre-stream state, with 120 seconds, 2 CPUs and 2 GiB per call and 60 seconds per statement. Only the program file is graded. The agent gets four sample workloads, which are not graded, a starter that writes one statement per changed row, and a scoring harness. Graded workloads use the same schemas but may be larger, up to about 3,000 changes in a transaction and 6.5 MB per stream.
Trajectories
Step through all 112 attempts from the pass@8 sweep with web search over an October 1, 2026 snapshot: every turn, tool call and submission. The best reward was 49.5%, by Claude Fable 5.2 on attempt 4. The results above come from these runs.
Failure analysis
Claude Opus 5.5 averages 0.286 across its 8 attempts. On many graded workloads its scripts stayed close to the plain baseline in size. Most of Opus's failures come from stopping about an hour early, believing little was left to gain without testing on larger, unfamiliar workloads, and from only stress-testing patterns its rules already handled.
- Stopped about an hour early, believing little was left to gainFound in 4 of 8 attempts31.2
- Only stress-tested patterns its rules already handledFound in 3 of 8 attempts29.4
- Scored itself on the samples it had fitted and kept tuning to themFound in 1 of 8 attempts10.9
- ScoredMean reward 0.28628.6
Example failure analysis
Attempt 4 · reward 0.329 · Stopped about an hour early, believing little was left to gain
After one failed try to explain p01's bitmap rows it switched to edge-case checks with 2 hours left, never measuring fallback output on large unfamiliar workloads.
- What it did
- Early on it noted that bitmap cost grows with table size, making semantic rules more valuable on larger tables (step 55). With 2 hours left it tried to find rules for p01's bitmap-heavy transactions (step 297). At step 298 it could not explain p01's rows, found the semantic and bitmap forms about equal on a Wagtail sample statement, and turned to reviewing untested code paths. Its last hour went to timestamp shortening and unprivileged-role checks, and at step 340 it called itself confident and ended with about 55 minutes left. No step measured the bitmap share on a large workload with unfamiliar patterns.
- What would have worked
- Measure what share of bytes comes from bitmaps and id lists on large workloads whose row selections the samples lack, and widen the rule search there.
- Evidence
- Step 55
For larger tables the bitmap cost scales with the id range, so semantic predicates become more valuable as tables grow
- Step 298
they come out roughly similar in size, though semantic queries seem to scale better
- Step 298
I'm spending the remaining time reviewing untested code paths for edge cases
- Step 340
I'm confident in the current state, so I'll finalize progress notes
- Step 55
Why it is hard
When database engineers audit a migration, debug logical replication or compact a change-data-capture log, they start from thousands of row-level changes and try to recover the few bulk statements behind them. The stream keeps only each row's values before and after the change; the original joins, WHERE clauses and SET expressions are gone. Every guess must be exact: a predicate that matches one row too many or too few breaks the replay, and the workload scores 0. Triggers already wrote some of the stream's rows, so firing them again fails or writes different rows. The databases use real Saleor, Wagtail, NetBox and Zulip schemas with generated rows. Query-by-example tools (Scythe, PATSQL, SQLSynthesizer) synthesize SELECT statements, not data changes.
How it is graded
The verifier runs the program offline on each graded workload, then replays the script on a fresh restore as a separate restricted role and compares the rows with the stream after every transaction. In the score below, s is the script's size in UTF-8 bytes without comments and extra whitespace, b the size of a fixed baseline script that groups equal changes and follows one foreign key, and k the shortest exact replay known to the verifier. A baseline-sized script earns 0; one at t or shorter earns 1. A missing file, timeout, crash, memory overrun, output that differs between runs, parse failure, rejected statement or any row mismatch scores 0 on that workload. The unmodified starter scores 0.
reward = mean of scorew over the graded workloads
Anchors
| Anchor | Reward | Score | Meaning |
|---|---|---|---|
| Floor | 0.000 | 0% | What a missing, invalid or rule-breaking submission scores |
| Baseline | 0.000 | 0% | The untouched starting point |
| Ceiling | 1.000 | 100% | The best result the metric allows |
| Regrade tolerance | Exact | Only identical results tie |
Environment
- Domain
- Software · Databases
- Shape
- Scaffolded
- Agent time
- 6 h per attempt
- Hardware
- 2 vCPU · 8 GB · PostgreSQL
- Expert estimate
- 6 h
- Task ID
sql-replay