SmolLM2 Long Context

Results

Every model's 8 attempts with web search over a copy of the web frozen at October 1, 2026, on the scale that runs from the task's baseline at 0% to its ceiling at 100%.

Artifacts

What the top model on this task, by mean score, built.

Open the full artifacts page →

The task

The agent gets the SmolLM2-1.7B base model, trained at 8,192 tokens, and must extend it to answer questions about documents up to 65,536 tokens long. It also gets the verifier's loader, a submission checker, the public question generator, a development question set with a local evaluation script, and about 2,500 public-domain books. Any method is allowed, including training on the public generator. The deliverable is a full checkpoint, config.json and a single model.safetensors, with the base architecture and tensor shapes; the weights are loaded in bf16. Only the RoPE settings are read from the config: rope_theta, max_position_embeddings, and rope_scaling of the linear, dynamic, YaRN or Llama 3 type. The verifier builds the model itself, so no submitted code runs. The private documents write their records in other wordings, and some question wordings are not in the public generator, so the model has to read what a record says.

Trajectories

Failure analysis

Gemini 4 Argon averages 0.334 across its 8 attempts. Its models did best on the shortest documents, but questions that follow a chain of variables through the longest documents stayed weak. Most of Argon's failures come from losing training hours to restarts, narrow fixes and a late first test on long documents, and from not building enough test questions to judge its long-document skill.

Example failure analysis

Why it is hard

The well-known recipe, adjusting RoPE and continuing training on long text, already pulls a single value out of a long document, so single-key lookup is not scored, and neither is the base model's native 8,192 tokens. The scored questions need the whole document: list every value filed under one key among many keys, some near-identical (multi-value lookup); name every variable that ends up holding a value after long assignment chains with reassignments (variable tracking); and name the item with the most delivery notes when decoy memos mention the same items (frequency). RULER reports that aggregation and tracking fall off long before single-needle retrieval. Published context extensions train on billions of tokens; one H100 session allows tens of millions. SmolLM2-1.7B also uses full multi-head attention, so a 64k-token training step strains the GPU's memory.

How it is graded

The verifier runs offline in its own container and asks questions about private documents in the three families at 16,384, 32,768 and 65,536 tokens. Decoding is greedy with at most 128 new tokens, and only the first generated line is read. Its items are compared with the correct answer as a set: order, duplicates and other text do not matter, but any missing or extra item, a hedge included, makes the question wrong. The reward is the mean exact-set accuracy over the 9 family and length cells; the unmodified base model, which answers nothing past 8,192 tokens, scores 0. A missing submission, a config outside the allowed RoPE settings, wrong tensor names, shapes or dtypes, or a model that cannot run or gives non-finite logits also scores 0.

Anchors

Environment

Domain

ML · Training

Shape

End to end

Agent time

8 h per attempt

Hardware

1× H100 · 8 vCPU · 128 GB

Expert estimate

12 h

Task ID

smollm2-long-context