Results
Every model's 8 attempts with web search over a copy of the web frozen at October 1, 2026, on the scale that runs from the task's baseline at 0% to its ceiling at 100%.
Artifacts
What the top model on this task, by mean score, built.
The task
A plant in a mountain valley plans a new 45 m stack. The agent must forecast hourly ground-level concentrations at the 24 receptors for eight scenarios, each a continuous release from the stack at its stated rate, starting at hour 12 of its 48-hour tower record and lasting 36 hours. The substance does not deposit or react. The evidence is the operator's tracer campaign: 18 SF6 releases of one and a half to two hours, with hourly bag samples at every receptor and one-minute readings from four fast analyzers, in ppt by volume. The agent also gets the hourly 10 m and 60 m tower and net radiometer records, a 50 m terrain grid, the site layout, each scenario's stack, emission and tower data, and a script for a screening forecast. The deliverable is /app/forecast.csv, 6,912 rows: the hourly mean in µg/m³ for every scenario, receptor and release hour 1 to 36. Any method is allowed, on 1 CPU and 8 GB of RAM with public internet access.
Trajectories
Step through all 112 attempts from the pass@8 sweep with web search over an October 1, 2026 snapshot: every turn, tool call and submission. The best reward was 36.0%, by GPT-6 Astra on attempt 7. The results above come from these runs.
Failure analysis
Claude Opus 5.5 averages 0.128 across its 8 attempts. The model's daytime forecasts near the stack came out too low and its night forecasts were mostly zero or in the wrong places. Opus's largest failure came from incorrectly assuming the gas detectors were accurate right up to their limit, and therefore trusting readings near the stack that were already too low, leading to a plume that was too wide and too weak near the stack.
- Incorrectly assumed the gas detectors were accurate right up to their limit, so its plume came out too weak near the stackFound in 5 of 8 attempts53.3
- Measured its model reading too low near the source, then called it unbiasedFound in 1 of 8 attempts11.8
- Judged the open questions coin flips; stopped early with a too-wide day plumeFound in 1 of 8 attempts11.3
- Put the night plume in the valley drainage every drainage hour, despite its own false-alarm warningFound in 1 of 8 attempts10.8
- ScoredMean reward 0.12812.8
Example failure analysis
Attempt 1 · reward 0.079 · Incorrectly assumed the gas detectors were accurate right up to their limit, so its plume came out too weak near the stack
Treated clipped readings as lower bounds, kept a day model it saw running 20-40% low, and stopped at 2.4 hours.
- What it did
- It found the problems in the records and treated readings at the 219 ppt ceiling as lower bounds, using the rest as true. Its day puff fits stayed 20-40% low on several releases (step 93). It forecast zero for deep drainage hours and the 25th percentile of an ensemble for other night rows (steps 87, 152). It stopped at step 198 with about 3.6 hours left, expecting a moderate result.
- What would have worked
- It should have tested a smooth saturation curve to explain the steady under-prediction, and used the time left on the day model. The higher-scoring attempts from other models were nearly unbiased near the stack.
- Evidence
- Step 22
saturated readings are really just lower bounds
- Step 93
fit7 plateau still shows sizable errors, 20-40% off in several cases like T16 h1 and T12 h2, which is consistently underpredicting
- Step 181
zeros are clearly the safer, correct call for the night drainage decision
- Step 198
I've decided to stop here and finalize submission #12
- Step 22
Why it is hard
A new stack is usually assessed with a steady Gaussian plume driven by a measured wind, and such models are tested against tracer campaigns, where a known amount of gas is released on site and sampled downwind. Here the campaign is the only evidence and matches the job poorly: short releases from a 20 m mast or a 60 m tower, against a 45 m stack running 36 hours through day and night, under weather the campaign only samples. The records come as the field crew and lab handed them over, with the problems real campaign data has. Scoring is strict: a row earns credit only within a factor of 1.2, where models are often judged within a factor of two, and false alarms cost score too.
How it is graded
The verifier compares each row's forecast F with the true value T in µg/m³. A row scores 1 at F = T, falling on a log scale to 0 once F + 0.1 is a factor of 1.2 from T + 0.1. N and D are weighted scores over the night rows (net radiation below zero that hour or in any of the three before) and the day rows, each row weighted by the tracer that arrives, or by F where it claims more, so false alarms cost score. B is the mean row score on the top tenth of T.
Completeness is the share of the 6,912 rows given exactly once with a finite, non-negative value. A missing or malformed file, or one over 20 MiB or 200,000 rows, scores 0.
Anchors
| Anchor | Reward | Score | Meaning |
|---|---|---|---|
| Floor | 0.000 | 0% | What a missing, invalid or rule-breaking submission scores |
| Baseline | 0.023 | 0% | The untouched starting point |
| Ceiling | 1.000 | 100% | The best result the metric allows |
| Regrade tolerance | ± 0.010 | Results closer than this tie |
Environment
- Domain
- Science · Earth
- Shape
- End to end
- Agent time
- 6 h per attempt
- Hardware
- 1 vCPU · 8 GB
- Expert estimate
- 12 h
- Task ID
plume-dispersion-forecast