Results
Every model's 8 attempts with web search over a copy of the web frozen at October 1, 2026, on the scale that runs from the task's baseline at 0% to its ceiling at 100%.
Artifacts
What the top model on this task, by mean score, built.
The task
In speculative decoding a small draft head proposes several tokens, and a large target model checks them all in one forward pass; the agent must build one for the frozen SmolLM3-3B. A runtime is the contract: it defines the head, the tensors it must contain and the decoding loop it is scored with. The head fuses the target's hidden states from three layers with the token embedding, then runs one or two Llama-style decoder blocks and an lm_head over a draft vocabulary mapped to target token IDs. The agent also gets a valid untrained head, a submission checker, a local evaluation script, 20 development prompts built like the scored ones from other books, 20,000 UltraChat conversations and public-domain books in six languages. It submits config.json and model.safetensors, readable by an unprivileged user, with at most 150 million parameter elements. Any training method is allowed, and the verifier loads only the weights. It is scored by the tokens produced per target check on long book excerpts.
Trajectories
Step through all 112 attempts from the pass@8 sweep with web search over an October 1, 2026 snapshot: every turn, tool call and submission. The best reward was 52.0%, by Claude Fable 5.2 on attempt 7. The results above come from these runs.
Failure analysis
GPT-6.1 Sol averages 0.380 across its 8 attempts. Its heads scored lower on the private books than on its own tests. Sol's main failure is testing on new excerpts from books it had trained on, instead of on unseen books.
- Tested on new excerpts from books it had trained on, instead of on unseen booksFound in 5 of 8 attempts38.5
- Ranked late tweaks with a fast stand-in test that read higher than the real decoderFound in 2 of 8 attempts15.9
- Held out whole books, but still scored lower on the private books, for reasons the attempt doesn't showFound in 1 of 8 attempts7.6
- ScoredMean reward 0.38038.0
Example failure analysis
Attempt 1 · reward 0.374 · Tested on new excerpts from books it had trained on, instead of on unseen books
It picked its head in the real decoder on 80 prompts, but the private books still scored 0.17 tau below its public score.
- What it did
- It started generating the big model's book answers at about 0.5 hours and meanwhile compared head variants (adapt, long, frozen, deep, wide) on the early data, near 2.0-2.1 tau. Its 8,000 extra book-answer examples were finished only at 3.5 hours, when the head stood at 2.18 tau. Training several heads on that data and averaging them reached 2.67 tau in the real decoder on the 20 public prompts by 6.6 hours. It also ran the real decoder on 60 unused prompts (2.604), rejected a calibration that helped only there, and stopped with 43 minutes left, judging selection complete.
- What would have worked
- Its 60 extra prompts could share books with its training data and read 0.11 above the private score. Testing on whole held-out books, and treating the public score as optimistic, would have given a truer estimate. The best attempt from another model lost only 0.01 tau between public and private prompts.
- Evidence
- Step 137
it reaches about 2.18 tokens per verification step on all 20 cached development prompts
- Step 271
Actual unmodified runtime all20 public:** tau **2.6699181875**
- Step 281
Independent actual60 completed. Plain currentwinner60tau **2.6041762276**
- Step 290
Later calibration candidates did not improve reliably, so I kept the validated winner.
- Step 137
Why it is hard
A draft head reads the target's own hidden states, so heads trained for other models do not carry over, and there is no public head for SmolLM3-3B or support in the common training frameworks. The head is capped at 150 million parameter elements at a fixed width of 2,048, and a 64k-row draft lm_head alone is about 134 million, so the draft vocabulary, layer count and MLP width compete for one budget; the best split has to be found by experiment. The traffic is long: book excerpts of 6,000 to 12,000 tokens in six languages. Training on sequences that long costs several times more per token than short chats, and a head that drafts short English chat well can fall apart here.
How it is graded
The verifier runs offline. Each held-out prompt is a book excerpt of 6,000 to 12,000 tokens from books outside the agent's data, then a request in the book's language; half are in English and a tenth each in French, Spanish, German, Italian and Portuguese. Decoding is greedy chain speculative decoding with draft depth 7, thinking mode off and up to 512 new tokens. Each verification step emits the accepted draft tokens plus one target token; τ is pooled over all prompts.
reward = clip((τ − 1) ÷ 4, 0, 1)
Drafts never accepted give τ = 1 and score 0; a reward of 1 needs about 85% acceptance at every draft position. A malformed submission, wrong tensor names, shapes or dtypes, non-finite weights, a bad vocabulary mapping or over 150 million elements scores 0.
Anchors
| Anchor | Reward | Score | Meaning |
|---|---|---|---|
| Floor | 0.000 | 0% | What a missing, invalid or rule-breaking submission scores |
| Baseline | 0.000 | 0% | The untouched starting point |
| Ceiling | 1.000 | 100% | The best result the metric allows |
| Regrade tolerance | ± 0.015 | Results closer than this tie |
Environment
- Domain
- ML · Inference
- Shape
- End to end
- Agent time
- 8 h per attempt
- Hardware
- 1× H100 · 8 vCPU · 128 GB
- Expert estimate
- 12 h
- Task ID
smollm3-draft-head