Results
Every model's 8 attempts with web search over a copy of the web frozen at October 1, 2026, on the scale that runs from the task's baseline at 0% to its ceiling at 100%.
Artifacts
What the top model on this task, by mean score, built.
The task
Make MicroPython's bytecode VM run hidden Python programs in fewer guest instructions, with identical output. The starting point is MicroPython v1.29.0 with its QEMU port for the 32-bit RISC-V virt board. Only the core VM sources and the QEMU port are collected and rebuilt on the pristine tree, with compiler flags from a whitelist. Programs arrive as .mpy bytecode, and every guest instruction from reset to exit counts, startup and loading included. The hidden programs come from the same families as the 22 public dev programs the agent gets with their expected outputs and a scorer: object-heavy code, strings, generators, big integers, float math, exceptions, small applications and code that changes classes, globals and builtins mid-loop. The score is 0 for a failed build, any changed output, a broken size limit, a MicroPython test regression, a trap or out-of-range memory access, an enabled emitter or frozen code, an unlisted compiler flag, or more inline assembly than the baseline has.
Trajectories
Step through all 112 attempts from the pass@8 sweep with web search over an October 1, 2026 snapshot: every turn, tool call and submission. The best reward was 66.7%, by Claude Fable 5.2 on attempt 4. The results above come from these runs.
Failure analysis
GPT-6.1 Sol averages 0.272 across its 8 attempts. Its interpreters generally produced correct outputs, but ran ordinary programs only about 2 times faster instead of the 6.5 times needed for full marks. Most of Sol's failures come from building shortcuts that only sped up single practice programs instead of the core interpreter, and from judging speed solely on the practice programs instead of new, realistic ones.
- Built shortcuts that only sped up single practice programs, instead of speeding up the core interpreterFound in 4 of 8 attempts36.0
- Judged speed only on the practice programs; never timed new, realistic programsFound in 3 of 8 attempts24.3
- Tested risky shortcuts only with programs shaped like them; one graded program crashedFound in 1 of 8 attempts12.5
- ScoredMean reward 0.27227.2
Example failure analysis
Attempt 8 · reward 0.265 · Built shortcuts that only sped up single practice programs, instead of speeding up the core interpreter
It chose loop-collapsing and range-sum shortcuts because they raised its practice score to 7.04, while its final speedup was only 1.64.
- What it did
- By step 333 general work (bigint limbs, list rotation, ord/chr loops) had reached a dev S of 2.93. From step 483 it added a 'pure loop' analysis that collapsed one dev benchmark from 11.6M to 0.55M instructions. At step 570 it added a range-sum shortcut that computes generator sums directly, lifting dev S to 4.76. It hit the dev score cap of 1.0 (S 6.49) at step 619 with almost 3 hours left, and kept adding shortcuts until pb_core_locals ran 2,157x faster. In its last hour it checked 62 variants of dev programs (new sizes, repeats and renamed names), but they kept the same shapes, so its shortcuts still fired and variant S was about 7.
- What would have worked
- Treat a 2,000x speedup on one program as code that will not carry over, and spend the time on the general paths every program uses. The best attempt of this model had a lower practice score (3.49) but a final S of 2.00.
Why it is hard
MicroPython runs Python on microcontrollers such as the ESP32 and RP2040, where most time goes to the bytecode VM. The baseline already has the speed options on: -O3, computed-goto dispatch and a 1,024-entry map lookup cache. There is no JIT to fall back on. The native and inline-assembler emitters stay off and no code may be generated at run time, so every gain must come from the interpreter's C code and stay correct when a program changes a class, a global or a builtin mid-loop. Memory is tight too: the GC heap keeps the baseline's size, code may at most double and data may grow by only 16 KiB, while published interpreter speedups for desktop Python assume megabytes of memory.
How it is graded
The verifier runs offline in a separate container, along with MicroPython's own test suite. Each program runs in a fresh qemu-system-riscv32 (RV32IMAC and Zicsr only) from reset to exit, and a plugin outside the guest counts every executed instruction and checks the execution rules, so a regrade gives identical counts. Any failed gate scores 0. Otherwise each program i gets a speedup Si from the baseline's count Bi and the submission's count Ci; a run that reaches 5 times its baseline count, or 120 seconds, is stopped and gets Si = 0.2. The unmodified VM scores 0 and a geometric-mean speedup of 6.47× scores 1:
reward = clamp(ln S ÷ ln 6.47, 0, 1), where S is the geometric mean of Si over the hidden programs
Anchors
| Anchor | Reward | Score | Meaning |
|---|---|---|---|
| Floor | 0.000 | 0% | What a missing, invalid or rule-breaking submission scores |
| Baseline | 0.000 | 0% | The untouched starting point |
| Ceiling | 1.000 | 100% | The best result the metric allows |
| Regrade tolerance | ± 0.001 | Results closer than this tie |
Environment
- Domain
- Software · Languages
- Shape
- Scaffolded
- Agent time
- 16 h per attempt
- Hardware
- 8 vCPU · 32 GB
- Expert estimate
- 12 h
- Task ID
micropython-speed