Results
Every model's 8 attempts with web search over a copy of the web frozen at October 1, 2026, on the scale that runs from the task's baseline at 0% to its ceiling at 100%.
Artifacts
What the top model on this task, by mean score, built.
The task
Build a system that keeps four Apache Iceberg tables and a fraud-alerts Kafka topic up to date from a payments database's change data capture, under a contract of guarantees, SLOs, faults and rules. The tables hold account balances, transfer history, hourly merchant aggregates and late-payment corrections, and each published snapshot declares the source position it reflects. Each hit of a velocity rule produces one alert, which a read_committed consumer must see exactly once. Progress never goes back, even after node loss, erasures of personal data land within a deadline, and everything converges once input stops. The deliverable is one entry point, run offline with 16 vCPU, 64 GB of RAM, a disk shared with the grader and an object store capped at 2,000 requests/s. It must recover from whatever durable state exists. A dev kit has local copies of the grader's services, a public generator, latency metrics and a final-state checker. The score is the highest input rate the system sustains with every SLO met under injected faults.
Trajectories
Step through all 112 attempts from the pass@8 sweep with web search over an October 1, 2026 snapshot: every turn, tool call and submission. The best reward was 100.0%, by Claude Opus 5.5 on attempt 7. The results above come from these runs.
Failure analysis
Claude Opus 5.5 averages 0.538 across its 8 attempts. Two attempts broke a guarantee under overload and scored 0. Most of Opus's failures come from never testing its system under overload because the supplied load generator was too slow, and from not testing above the rate it was told gave full marks.
- Never tested its system under overload, because the supplied load generator was too slowFound in 2 of 8 attempts25.0
- Didn't test above the rate it was told gave full marksFound in 5 of 8 attempts14.6
- Saw its alerts worker was marginal at the top rate, but rejected the redesign as too riskyFound in 1 of 8 attempts6.5
- ScoredMean reward 0.53853.8
Example failure analysis
Attempt 4 · reward 0.000 · Never tested its system under overload, because the supplied load generator was too slow
It decided on paper that back-pressure kept overload safe, never overloaded its system, and under the grader's overload it fell behind and never caught up.
- What it did
- It built a Java pipeline with separate committer threads per table and a 24 GB heap. It measured that it could drain a pre-built backlog at about 860,000 events/s and concluded throughput was not a risk. It reasoned twice about the overload level, both times only on paper, and late in the attempt lowered the heap to 20 GB, testing that build only up to 50,000 events/s. It submitted with about 5 hours left.
- What would have worked
- Run a test where input arrives faster than the system can process for several minutes, with kills, and check that memory stays bounded and that the system catches up afterwards. The best attempt of the same model kept a larger heap and tested the top four levels with bursts and kills before submitting.
- Evidence
- Step 197
Overload correctness holds, and even if S3 or the catalog slow down, bigger batches with longer commit times still work fine.
- Step 258
Since the local generator can't hit the top-tier burst rates (180K/s), I'm planning a 5-minute validation run at the generator's practical ceiling
- Step 276
This gives throughput of roughly 860K events/s, far exceeding what's needed
- Step 370
Working through memory bounds confirms backpressure keeps buffered events manageable even during overload bursts
- Step 197
Why it is hard
Payments data engineers keep financial records, merchant analytics and fraud alerts current from a database's change stream. Here one transaction's rows are spread over partitions of two Kafka topics, with no order across partitions: rows and end markers arrive in any order, a partition can lag, and the source can redeliver recent history, even from mid-transaction. Every snapshot must still reflect an exact prefix of transactions. Process kills, node loss that wipes local state, broker restarts, object-store errors and throttling, replays and lag are injected throughout, and after each the system must recover without a partial financial record, a duplicated or dropped alert, or restored erased data. Fast publication and recovery must be balanced against checkpoint cost, small files, compaction and a rate-limited object store.
How it is graded
Every submission gets the same held-out workload and fault schedule, from a different generator than the dev kit's. After backfill, the input rate starts at 1,000 events/s and rises 1.25× every 7-minute level, in bursts. A level passes if the source can offer its load and every p99 latency SLO holds: 10 s for balances, 30 s for transfer history, 60 s for merchant aggregates and 2 s for fraud alerts. A missed level is rerun once, and one failed level is forgiven. An overload probe and a quiet period then check correctness and convergence. Any guarantee violation, rule breach or passing no level scores 0. The agent is told each level adds 0.05, up to 1 at level 19 (69,389 events/s). The leaderboard instead credits each halving of the gap to ramp step 24 (about 211,758 events/s) equally, with full credit from step 23 (169,407 events/s).
Anchors
| Anchor | Reward | Score | Meaning |
|---|---|---|---|
| Floor | 0.000 | 0% | What a missing, invalid or rule-breaking submission scores |
| Baseline | 0.000 | 0% | The untouched starting point |
| Ceiling | 1.000 | 100% | The best result the metric allows |
| Regrade tolerance | ± 0.050 | Results closer than this tie |
Environment
- Domain
- Software · Data engineering
- Shape
- End to end
- Agent time
- 16.5 h per attempt
- Hardware
- 16 vCPU · 64 GB
- Expert estimate
- 40 h
- Task ID
streaming-etl-payments