Results
Every model's 8 attempts with web search over a copy of the web frozen at October 1, 2026, on the scale that runs from the task's baseline at 0% to its ceiling at 100%.
Artifacts
What the top model on this task, by mean score, built.
The task
A new outer harbor is planned on the western of two barrier islands sheltering a lagoon about 19 km long. Forecast the still-water level at 25 points for each whole hour from 30 to 53 of the design storm, in meters above the national datum as a tide gauge would record it, including wave setup (the rise in mean level from breaking waves) on the ocean beach. Eighteen points lie on the western island's ocean beach, ten of them on the harbor frontage; the rest are on the eastern beach and in the lagoon, pass and estuary. The agent gets a 1994 chart, 2025 soundings, beach profiles, sediment samples and a leveling report; gauge, current, weather, wave and weather-model records from a calm month and two storms; the tide at the open sea boundary; the design storm's exact wind, pressure and offshore waves; and the previous consultant's script. It writes /app/forecast.csv, 600 levels, with 1 CPU, 8 GB of RAM, Python with numpy and scipy, and public internet access.
Trajectories
Step through all 112 attempts from the pass@8 sweep with web search over an October 1, 2026 snapshot: every turn, tool call and submission. The best reward was 80.3%, by GPT-6 Astra on attempt 8. The results above come from these runs.
Failure analysis
Claude Opus 5.5 averages 0.476 across its 8 attempts. Most of Opus's failures come from dismissing the coarse sand of the harbour frontage as a sign of a steeper beach, and from accepting the beach could be steeper but choosing a cautious middle value.
- Dismissed the coarse sand as a sign of a steeper beachFound in 5 of 8 attempts33.8
- Accepted the beach could be steeper but chose a cautious middle valueFound in 3 of 8 attempts18.7
- ScoredMean reward 0.47647.6
Example failure analysis
Attempt 7 · reward 0.342 · Dismissed the coarse sand as a sign of a steeper beach
It treated the coarse frontage sand as a possible trap and gave all ten frontage points the slope of the profile at the patch's edge.
- What it did
- At the start it noted that the coarser frontage sand suggests a steeper beach. Testing grain size against the five profiles, it found no single rule, and at 0.95 h set the whole frontage to the nearest profile's slope of 0.0496, although it had just noted that this profile sits at the edge of the coarse patch. At 1.39 h it wondered whether the coarse-sand data was 'intentional misdirection' and kept its choice. It also cut wind setup at the two ungauged eastern-beach points P19-P20 to 39-48% because of one gauge's unexplained shortfall, and assumed the Earth's rotation was negligible without testing it. It confirmed its final forecast at 1.56 h with 2 h 26 min left, listing the frontage slope among changes with negligible gains.
- What would have worked
- Model the coarse patch as its own stretch, with the edge profile as only one anchor, and test how much setup that implies on the coarse-sand frontage. Keep P19-P20 unscaled unless the gauge shortfall is explained; another attempt of this model dropped the same cut after finding the shortfall only occurred for waves from the south-west.
- Evidence
- Step 13
The new harbour frontage area has notably coarser sediment, suggesting it's steeper there
- Step 103
Given BP3 is the nearest profile to the frontage location, I'll just stick with 0.0496 throughout rather than blending
- Step 139
Maybe this is intentional misdirection — a realistic dataset with no stated correlation, designed to tempt someone into inferring steep frontage slopes from coarse sediment using a textbook relation.
- Step 160
the frontage β all seem to offer negligible gains given what I already know, with no new information to act on.
- Step 13
Why it is hard
A design water level sets a harbor's crest heights and freeboard, and it can only be checked when the design storm comes. The level combines tide, surge from wind and pressure, and setup from breaking waves, all shaped by the seabed and the beach, and all must be right at once: a level 5 cm or more off earns nothing. The only complete chart is from 1994. The lagoon floor has changed since, a channel was dredged through the pass in 2023, and the newer soundings cover six lines. The records cover a calm month and two storms, and the design storm differs from both. Setup depends on beach detail finer than the chart's 100 m cells, and the records have the problems real field records have.
How it is graded
The verifier runs offline in a separate container and compares each of the 600 values with the true level. A value earns 1 − |error| ÷ 5 cm, or 0 once it is 5 cm or more off, and the reward is the mean over all values divided by a fixed 0.701733, capped at 1. The tolerance is flat because design levels are set in absolute terms against a datum. A missing, duplicated, non-finite or unparseable value scores 0, and the whole file scores 0 if it is not a regular CSV file with the three columns, is over 20 MiB or has more than 100,000 rows. Grading is deterministic.
Anchors
| Anchor | Reward | Score | Meaning |
|---|---|---|---|
| Floor | 0.000 | 0% | What a missing, invalid or rule-breaking submission scores |
| Baseline | 0.036 | 0% | The untouched starting point |
| Ceiling | 1.000 | 100% | The best result the metric allows |
| Regrade tolerance | ± 0.010 | Results closer than this tie |
Environment
- Domain
- Science · Earth
- Shape
- End to end
- Agent time
- 4 h per attempt
- Hardware
- 1 vCPU · 8 GB
- Expert estimate
- 12 h
- Task ID
storm-surge-forecast