Results
Every model's 8 attempts with web search over a copy of the web frozen at October 1, 2026, on the scale that runs from the task's baseline at 0% to its ceiling at 100%.
Artifacts
What the top model on this task, by mean score, built.
The task
A development group ran eleven batch runs of a catalyzed reaction between A and B in a 1 L jacketed reactor, followed by an in-situ ATR-FTIR probe and offline HPLC. The agent forecasts A, B, product P, intermediate I (one A plus one B, with no HPLC peak) and impurity D (one A plus two B) for six new operating cases, at 20 times from 5 to 480 minutes after the catalyst goes in. It gets the records as the lab handed them over: the notebook, balance printouts and reagent lot sheet; reactor and jacket temperature logs; FTIR spectra with reference spectra of the solvent, A, B and P; HPLC results and standards; the six cases, written like plant batch sheets; a note on the chemistry and file formats; and the previous chemist's fit and forecast. The deliverable is /app/forecast.csv, 600 liquid-phase concentrations in mol/L. Any method or open-source code is allowed, on 1 CPU and 8 GB of RAM with Python, numpy, scipy and public internet access.
Trajectories
Step through all 112 attempts from the pass@8 sweep with web search over an October 1, 2026 snapshot: every turn, tool call and submission. The best reward was 99.3%, by GPT-6.1 Sol on attempt 6. The results above come from these runs.
Failure analysis
Claude Opus 5.5 averages 0.567 across its 8 attempts. Most attempts got the hot, dry forecast cases wrong. Most of Opus's failures come from not properly testing its model at hotter temperatures, and from testing only versions of its early guess that water protects the catalyst.
- Didn't properly check its catalyst model at hotter temperaturesFound in 5 of 8 attempts32.2
- Assumed water protects the catalyst and tested only versions of that ideaFound in 1 of 8 attempts9.1
- Rejected one shared wear-out clock because the training fit was slightly worseFound in 1 of 8 attempts1.9
- Guessed the last hours of the hottest cases, beyond any training dataFound in 1 of 8 attempts0.2
- ScoredMean reward 0.56756.7
Example failure analysis
Attempt 7 · reward 0.448 · Didn't properly check its catalyst model at hotter temperatures
It ranked catalyst-death drivers by training fit, dropped its R10 hold-out as 'useless', and shipped the best-fitting variant.
- What it did
- It built a fresh-to-partly-active catalyst model and then ranked extra death terms (B, I, A, thermal) by training chi2 (step 215). At step 169 it flagged that a 270 kJ/mol term was 'only constrained by R10'. At step 232 it killed its leave-R10-out test, calling it useless because high-temperature parameters became unidentifiable. It saw two close-fitting variants disagree on the forecast (pairscore 0.49). It still chose the one with the best fit 'incl. R10' (step 237).
- What would have worked
- Treat 'unidentifiable without the 80 °C run' as a sign the extra terms are unsupported, and look for a structure that does not need them. The best attempt tested whether the decay stages share one heat scaling and needed no chemical death terms.
- Evidence
- Step 169
Concern: EIx=270 kJ/mol (I-driven X death) only constrained by R10 -> dry 85-95C scenarios S2,S3,S5 uncertain.
- Step 215
Death-driver comparison (death on X and Y): B 1.416 << I 1.584 < A 1.608 < IB 1.633 < thermal 1.829. B-driven strongly preferred.
- Step 232
CV without R10: high-T params unidentifiable (useless test).
- Step 237
Rationale: best fit incl. R10 (highest T)
- Step 169
Why it is hard
Process chemists are routinely asked how a process will behave under conditions the lab never ran, and only the plant run checks the forecast. The notebook's A + B → P is only a starting point: the mechanism must be worked out from the data, and the spectra include the solvent and two species, I and D, with no reference spectrum. The cases run at jacket setpoints of 80 to 90 °C for eight hours, two of them semibatch with B fed over time, while nine lab runs lasted three hours at 40 to 60 °C and two went eight hours, at 80 and 65 °C. The records have the problems real lab records have, and an error of 1 percent earns nothing: a model right in shape but a few percent off everywhere scores close to zero.
How it is graded
The verifier runs offline in a separate container and compares each of the 600 values with the true concentration. A value earns full credit when exact, falling linearly to zero at an error of 1 percent of the true value (at least 0.001 mol/L), and the reward is the mean over all 600 values. A missing, duplicated or unparseable value scores 0, so a shorter file cannot raise the mean. A missing, unreadable or malformed CSV, one over 20 MiB or 100,000 rows, or one without the four required columns scores 0 overall. Nothing else the agent writes is looked at, and regrading gives the identical number.
Anchors
| Anchor | Reward | Score | Meaning |
|---|---|---|---|
| Floor | 0.000 | 0% | What a missing, invalid or rule-breaking submission scores |
| Baseline | 0.082 | 0% | The untouched starting point |
| Ceiling | 1.000 | 100% | The best result the metric allows |
| Regrade tolerance | ± 0.010 | Results closer than this tie |
Environment
- Domain
- Science · Chemistry
- Shape
- End to end
- Agent time
- 4 h per attempt
- Hardware
- 1 vCPU · 8 GB
- Expert estimate
- 10 h
- Task ID
spectral-kinetics-forecast