Results
Every model's 8 attempts with web search over a copy of the web frozen at October 1, 2026, on the scale that runs from the task's baseline at 0% to its ceiling at 100%.
Artifacts
What the top model on this task, by mean score, built.
The task
Optical satellites cannot see through cloud, yet crop models need gap-free reflectance series that say how far each filled value can be trusted. Each sample is a 128 × 128 pixel tile seen by Sentinel-2 on twelve dates in ten bands, with radar near each date. The agent gets 387 training samples from Bangladesh and Spain and 60 validation samples from Portugal, plus the scoring code and a local evaluator, and writes /app/solution/solver.py, with a predict function and optional weights in model.onnx. Scored tiles come from new locations, with three whole dates withheld. For each withheld reflectance value, and for NDVI, (NIR − red) ÷ (NIR + red), it returns two normals mixed around one centre: a main spread, a tail spread and a tail probability. Observed values must stay unchanged, and runs must be repeatable. Each call runs offline on CPU within 90 seconds and 6 GiB, with only NumPy, SciPy and ONNX Runtime. The agent has 4 CPUs, no GPU, internet access and four hours.
Trajectories
Step through all 112 attempts from the pass@8 sweep with web search over an October 1, 2026 snapshot: every turn, tool call and submission. The best reward was 50.0%, by Claude Sonnet 5.5 on attempt 5. The results above come from these runs.
Failure analysis
Claude Opus 5.5 averages 0.325 across its 8 attempts. Its guesses for the cloud-hidden pixels fell short in every attempt. Most of Opus's failures come from giving up on improving its guess when fixes helped some regions and hurt others, and from keeping a learned network even after it did worse on a new region.
- Gave up on improving the guess when its fixes helped some regions and hurt othersFound in 3 of 8 attempts25.9
- Kept a learned network after testing it on a new region and seeing it do worseFound in 3 of 8 attempts25.1
- Named the guess as the bottleneck, then rejected its own fix as too riskyFound in 1 of 8 attempts8.9
- Dropped NDVI fixes after one mixed testFound in 1 of 8 attempts7.5
- ScoredMean reward 0.32532.5
Example failure analysis
Attempt 4 · reward 0.278 · Gave up on improving the guess when its fixes helped some regions and hurt others
It judged the interpolation guess at its ceiling early and spent the rest of the attempt tuning the uncertainty network.
- What it did
- Its first MLP halved in-sample error but did worse than interpolation on Portugal (step 31), so it kept linear interpolation as the guess and trained the network only for uncertainty (step 39). Radar corrections flipped sign between windows (step 64), and at step 66 it judged the guess roughly at its ceiling and moved to the uncertainty model. Leave-one-window-out tests then drove network size, ensembles and smoothing. At step 108 it saw reflectance was the weak part (about 0.23 against 0.398), tried one per-band blend, found it mixed (step 110), and went back to uncertainty tweaks, ending with about 35 minutes unused.
- What would have worked
- Keep testing changes to the guess, which drove most of the lost points. Fit a correction on errors with each date's average removed, keep it only if every held-out window improves, and scale it down if it is weaker on held-out dates.
- Evidence
- Step 31
the model's RMSE on in-sample training windows (0.024) is half the interpolation baseline's (0.053), but on the Portugal validation set it actually underperforms both interp and tm
- Step 66
Since the mean seems roughly at its ceiling given robustness constraints, I should shift focus to the std model
- Step 108
pooled LOWO shows reflectance skill around 0.23
- Step 31
Why it is hard
Whole scenes are missing: the scored windows withhold three whole dates across every tile, so on those dates no part of the window shows the ground. The scored locations are not in the public data, which comes from Bangladesh, Spain and Portugal, so scores on the public validation split may not reflect them. Radar sees through cloud but does not measure reflectance, and each acquisition can be up to 12 days from its optical date. The answer is a distribution: a forecast that is too narrow or too wide loses score even when its centre is right, for every band and for NDVI. And it all runs on CPU, offline and deterministically, within 90 seconds per call.
How it is graded
The verifier runs offline on /app/solution alone. It scores every withheld reflectance value and NDVI pixel with the continuous ranked probability score (CRPS), a loss that punishes forecasts that are off-centre, too narrow or too wide. The losses are pooled and compared with a fixed baseline: each pixel's mean over its observed dates, with one fixed spread per band and one for NDVI. Skill, 1 − Σ CRPSmodel ÷ Σ CRPSbaseline, is computed for reflectance (R) and NDVI (N); only the reward is clipped, so extra skill on one can make up for less on the other. A missing or extra file, a wrong output, a changed observation, a run-to-run difference or a call over its limits scores 0, as does doing no better than the baseline.
Anchors
| Anchor | Reward | Score | Meaning |
|---|---|---|---|
| Floor | 0.000 | 0% | What a missing, invalid or rule-breaking submission scores |
| Baseline | 0.000 | 0% | The untouched starting point |
| Ceiling | 1.000 | 100% | The best result the metric allows |
| Regrade tolerance | ± 0.000001 | Results closer than this tie |
Environment
- Domain
- Science · Earth
- Shape
- End to end
- Agent time
- 4 h per attempt
- Hardware
- 4 vCPU · 8 GB
- Expert estimate
- 3 h
- Task ID
sentinel2-gapfill