Results
Every model's 8 attempts with web search over a copy of the web frozen at October 1, 2026, on the scale that runs from the task's baseline at 0% to its ceiling at 100%.
Artifacts
What the top model on this task, by mean score, built.
The task
Make a pretrained Qwen3-8B better at chess tactics within 17.5 hours, changing its weights only through a metered training service, then submit one service-created checkpoint with python /app/submit.py CHECKPOINT_ID. Submitting seals it, writes /app/submission.json and permanently closes training, blending and sampling; a run that is never sealed scores 0. The model plays through a tactics tool interface: get_position returns the board and its legal moves, the model plays moves in UCI notation (such as e2e4), and each correct move plays the fixed opponent reply. An invalid action, an illegal move or a wrong move ends the attempt, which allows at most 20 tool calls. A puzzle counts only when the model completes the whole stored line or delivers an immediate legal checkmate. The reward is the share of 2,000 graded attempts solved: 1,000 private puzzles, each played twice. No training method is prescribed: the service offers supervised fine-tuning, preference updates, policy updates, sampling and checkpoint blending, all drawn from one budget.
| Resource | Limit |
|---|---|
| Training sequence tokens | 160,000,000 |
| Sampled tokens | 10,000,000 |
| Optimizer steps / checkpoints | 4,096 / 384 |
| SFT / preference / policy calls | 128 / 128 / 256 |
| Checkpoint blends | 16 |
Trajectories
Step through all 112 attempts from the pass@8 sweep with web search over an October 1, 2026 snapshot: every turn, tool call and submission. The best reward was 62.8%, by Claude Sonnet 5.5 on attempt 4. The results above come from these runs.
Failure analysis
Gemini 4 Argon averages 0.329 across its 8 attempts. Most of Argon's failures come from not testing on enough puzzles before picking its final model, such as choosing among 20 to 40 versions by gains of one to three puzzles on the same 60 to 80 practice puzzles, and from adding more of the same training after its gains flattened.
- Didn't test on enough puzzles before picking its final modelFound in 3 of 8 attempts26.0
- Kept adding more of the same training after its gains flattenedFound in 2 of 8 attempts16.0
- Misjudged what each evaluation cost and spent the sample quota by hour 6Found in 1 of 8 attempts9.3
- Steered its recipe for 10 hours by single-run scores on 64 puzzles, then kept small tweaksFound in 1 of 8 attempts9.1
- Found its training traces mislabelled checks and pawn defences; re-weighted the data instead of fixing themFound in 1 of 8 attempts6.9
- ScoredMean reward 0.32932.9
Example failure analysis
Attempt 1 · reward 0.254 · Didn't test on enough puzzles before picking its final model
It picked its checkpoint by 1 to 3 puzzle gaps on the same 60 dev puzzles and never ran the larger set it had planned.
- What it did
- It built supervised training data from 20,000 training puzzles with engine-labelled moves, then stacked supervised, GRPO (RL on its own games), polish and blend stages. It scored every candidate on the same 60 puzzles, almost always with one sampling run, and planned 120- and 240-puzzle sets that it never ran. With about 2 hours left it chose a second sampling run on the same 60 puzzles over a larger set. The leader scored 39 of 120, two or three puzzles ahead of four other finalists, and it sealed it with the sample quota spent.
- What would have worked
- Keep part of the quota for one large held-out check, such as 200 or more puzzles on both sampling runs. The best attempt confirmed its top six on 200 puzzle attempts and its estimate landed within 2 points of the final score.
- Evidence
- Step 13
This will be invaluable for high-precision comparisons between our top-performing candidate checkpoints.
- Step 131
overall solve rate is a stunning 35.0% (21 out of 60 puzzles solved)… This is a massive leap!
- Step 150
Solved=39/120 (0.3250)
- Step 153
we have utilized well over 99.5% of our allocated budget, reinforcing the need to proceed with a final submission.
- Step 13
Why it is hard
The agent can change the weights only through a metered training service, and the untouched model solves 3.25% of evaluation attempts. The hard part is turning chess knowledge into reliable play: the model must read a board from a tool call, find the tactical line and play every move of it correctly in UCI notation, since a partial line earns nothing. The workspace has public Lichess puzzles and Stockfish, but neither is in the evaluator's tool-call format, and only checkpoints created by this trial's service can be submitted. As in post-training research, the agent must build training examples, diagnose failures and choose checkpoints within fixed budgets of tokens, optimizer steps and calls. Every position comes from a real Lichess game.
What the agent starts with
The agent gets the pinned Qwen3-8B, served by a root-owned training service on a private H100; its account cannot reach the GPU or checkpoint storage. It also gets 20,000 training puzzles and 1,000 development puzzles from the public Lichess database, a tokenizer helper, a Python training client, and a status tool that reports checkpoints and remaining budget. A development evaluator uses the same interface and sampling policy as grading, and its sampling counts against the budget. Reference files cover the update payloads and their bounds, policy updates, the service limits, the training objectives and the fixed evaluation sampling policy. Stockfish is installed in the workspace, and the public internet is open.
How it is graded
Once the agent window closes, the trusted service exports the sealed adapter and the training sandbox is shut down. A separate verifier with its own H100 and no network restores the adapter onto its own copy of the model, never runs participant code, and plays each of 1,000 private Lichess puzzles twice, in two fixed sampling runs with a pinned vLLM sampler. The reward is the share of these 2,000 attempts solved, with no normalization. An unsealed trial, a changed or missing protected helper file, or a missing or mismatched submission scores 0; failures on the trusted side are infrastructure errors with no reward. The puzzles come from a public source, so overlap with pretraining data is not ruled out.
Anchors
| Anchor | Reward | Basis |
|---|---|---|
| Floor | 0 | No solved attempts, or a failed integrity gate |
| Baseline | 0.0325 | The pinned base model, sealed through the service with a negligible weight change: 65 of 2,000 attempts solved |
| Ceiling | 1 | All 2,000 attempts solved |
| Regrade tolerance | ± 0.01 | Allowed difference between regrades of the same sealed adapter, since batched bf16 sampling is not bit-reproducible |
Environment
- Agent time
- 17.5 hours
- Hardware
- 1× H100, reached only through the training service; 16 CPUs; 144 GiB RAM; a private checkpoint volume
- Network
- Public internet while working; the training API is local; none during grading
- Software
- Python 3.12; the service runs PyTorch 2.8, Transformers 4.57 and vLLM 0.11
- Verifier
- Separate container with its own H100; 8 CPUs, 128 GiB, 275-minute limit
- Expert estimate
- 12 hours
Prior work and assets
The starting checkpoint is Qwen3-8B (Apache-2.0). The training, development and evaluation positions come from the Lichess open puzzle database (CC0 1.0), and Stockfish 15.1 (GPL-3.0) is installed in the workspace.