Results
Every model's 8 attempts with web search over a copy of the web frozen at October 1, 2026, on the scale that runs from the task's baseline at 0% to its ceiling at 100%.
Artifacts
What the top model on this task, by mean score, built.
The task
When an earthquake sequence strikes, network seismologists publish a catalog of where and when each event happened; precise locations show which fault is moving. Here 16 stations recorded a month-long sequence of 500 events, and the agent must relocate all 500 as well as possible. It receives the station list, about 9,000 weighted P and S picks, vertical-component event windows (26 seconds at 100 Hz), three quarry blasts with known locations and shot times, a six-layer 1D velocity model with a travel-time routine for layered models, and a starter event locator. The deliverable is /app/catalog.csv with origin time, latitude, longitude (WGS84 decimal degrees) and depth in km below the surface for every event in the picks file, leaving the blasts out. Origin time is required but not scored; an epicenter earns credit only within 0.3 km of the truth, and a depth within 0.6 km. The agent has 1 CPU, 8 GB of RAM, Python with numpy, scipy and ObsPy, and public internet access.
Trajectories
Step through all 112 attempts from the pass@8 sweep with web search over an October 1, 2026 snapshot: every turn, tool call and submission. The best reward was 81.8%, by Claude Fable 5.2 on attempt 6. The results above come from these runs.
Failure analysis
Claude Opus 5.5 averages 0.311 across its 8 attempts and loses most of its points from placing earthquakes in the wrong spot on the map rather than at the wrong depth. Most of that comes from how the model handled blasts: Opus checked its wave-speed models against the blasts but didn't act on what they showed, and it trusted tests on fake data it made itself over the real blast checks.
- The model checked the wave-speed models against the blasts, but didn't follow through with the resultsFound in 3 of 8 attempts26.9
- Tested its method on fake data it made itself, and trusted that over the blastsFound in 2 of 8 attempts22.1
- Saw that depth was uncertain but never checked it directlyFound in 2 of 8 attempts13.7
- Took agreement between its own methods as proof, and left its weakest earthquakes uncheckedFound in 1 of 8 attempts6.2
- ScoredMean reward 0.31131.1
Example failure analysis
Attempt 8 · reward 0.217 · The model checked the wave-speed models against the blasts, but didn't follow through with the results
Its blast test could not tell two wave-speed models apart, so it switched to its own model because that one matched the arrival times slightly better.
- What it did
- Early on it saw that a re-fitted layered model fit as well as the supplied one but moved locations about 2.5 km (step 77). Later, 3D models built on the supplied reference and on its own re-fitted reference fit the data equally well but disagreed by 1 to 2.5 km on outer clusters. It relocated the blasts with each model left out of the fit (step 174): each model won on some blasts, with errors of 0.2 to 1.5 km, so the test could not choose. It first kept the supplied reference (step 176), then switched to the re-fitted reference because it matched the east cluster's arrival times slightly better and fit the blasts slightly better in-sample (step 196).
- What would have worked
- When data fit cannot choose between models, lean on the blasts and prefer the supplied reference model unless the anchors reject it. The best attempt kept the blasts fixed and judged models by leaving one blast out.
Why it is hard
A preliminary catalog comes out fast, from locating each event on its own in a regional velocity model; relocating the sequence properly is slower, harder work. The tolerances are tight: an epicenter earns nothing once it is 0.3 km from the truth, and a depth nothing at 0.6 km, a small fraction of a second of travel time at crustal speeds. The records are limited. The only velocity model supplied is a six-layer 1D model, the picks come with the kinds of problems real picks have, and only the three quarry blasts give absolute ground truth. Systematic errors do not average out over many picks, and they interact, so a fix for one can absorb or hide another.
How it is graded
An offline verifier compares the catalog with the true locations. Each event earns a horizontal part, from the great-circle epicenter error, and a depth part, both falling linearly to zero:
reward = (npresent ÷ 500) × (0.7 min(1, h̄ ÷ 0.626248) + 0.3 min(1, z̄ ÷ 0.695244))
Here h̄ and z̄ are averages over the events present exactly once with valid values, and the two fixed constants are set so an estimated best achievable catalog scores 1. Each part is capped before the two are combined, so a catalog with every depth wrong cannot score above 0.7. Each missing, malformed or duplicated event costs 1/500. The file scores 0 if it is missing, invalid, a symbolic link or not a regular file, over 50 MiB or 200,000 rows, or lacks the columns event_id,origin_time,lat,lon,depth_km in that order.
Anchors
| Anchor | Reward | Score | Meaning |
|---|---|---|---|
| Floor | 0.000 | 0% | What a missing, invalid or rule-breaking submission scores |
| Baseline | 0.072 | 0% | The untouched starting point |
| Ceiling | 1.000 | 100% | The best result the metric allows |
| Regrade tolerance | ± 0.010 | Results closer than this tie |
Environment
- Domain
- Science · Earth
- Shape
- End to end
- Agent time
- 4 h per attempt
- Hardware
- 1 vCPU · 8 GB
- Expert estimate
- 10 h
- Task ID
relocate-earthquake-sequence