Results
Every model's 8 attempts with web search over a copy of the web frozen at October 1, 2026, on the scale that runs from the task's baseline at 0% to its ceiling at 100%.
Artifacts
What the top model on this task, by mean score, built.
The task
Write a function order for a desktop build of Organic Maps, an open-source offline maps app, so that its startups touch fewer 4 KB code pages. The order is a text file that lists each of the binary's 15,306 function IDs exactly once, one per line, and the grader reads only this file. The agent gets each function's size and original address, the static direct call graph, a scorer for the training traces, and 24 training startup traces in six workload families: map view, offline region, overlay, route, category search and name search. Each trace lists functions in the order they first ran. Placement is fixed: functions go from byte offset 0, each aligned to the next 16-byte boundary. The baseline walks the training traces in a fixed order, appends each function at its first use, then adds the unused ones by address. Grading replays held-out startups from the same six families, some using app features that no training trace used, and the score counts only the worst family.
Trajectories
Step through all 112 attempts from the pass@8 sweep with web search over an October 1, 2026 snapshot: every turn, tool call and submission. The best reward was 99.4%, by GPT-6.1 Sol on attempt 7. The results above come from these runs.
Failure analysis
Claude Opus 5.5 averages 0.400 across its 8 attempts. Its layouts helped most kinds of startup load faster, but route startups marginally improved. Because the score counts only the weakest kind of startup, routes impacted the score in most attempts. Most of Opus's failures come from trusting test results that were too unreliable to tell its layouts apart, and from proceeding with a layout despite tests showing that it hurt routes.
- Trusted test results that were too unreliable to tell its layouts apartFound in 2 of 8 attempts17.6
- Kept a layout its own tests showed hurting routesFound in 1 of 8 attempts11.7
- Left never-used code in its original order, though the task warned of new featuresFound in 2 of 8 attempts11.2
- Shipped a last change even though its quick check showed routes getting worseFound in 1 of 8 attempts7.8
- Accepted that its layout made one route recording worse than the starting orderFound in 1 of 8 attempts7.4
- Best attempt: compared versions carefully, but routes still set its scoreFound in 1 of 8 attempts4.4
- ScoredMean reward 0.40040.0
Example failure analysis
Attempt 1 · reward 0.290 · Trusted test results that were too unreliable to tell its layouts apart
When its last tweaks gave noise-level results it decided the layout couldn't improve and stopped with 2 hours left, while car routes still scored low in its own tests.
- What it did
- It built a layout from a shared core plus family groups, split by co-usage, and ordered never-seen functions by walking the call graph. At step 93 it measured car routes at 0 on the training traces. It tried route-targeted structure (splitting the route group, a route-ordered core) and kept the best. At step 163 it named map view and route as the weakest families, but then mostly varied random starts, split depth, shared thresholds and checkpoint cuts. When these landed within noise (step 171), and it suspected its in-sample scores were partly lucky alignment (step 174), it concluded the order was as good as it could get and confirmed at 1.96 h of 4 h.
- What would have worked
- Use the remaining 2 hours on a new structure for route, the floor family, for example a dedicated route block sorted by route usage as the best attempt built (graded route 0.26).
Why it is hard
Performance engineers who ship release binaries write function order files so that a program faults in fewer 4 KB code pages when it starts; on a cold start those faults add up to a delay the user can see. Profile-guided layout is well studied. Pettis–Hansen, hfsort, C3 and lld's balanced partitioning for startup are the public tools, and they all place the functions a profile saw. The hard part here is the startups that no profile saw: some graded startups use app features that no training trace used. And since the score is the worst family's result, a layout that helps five families and misses the sixth also scores 0.
How it is graded
The verifier rebuilds the first-use baseline from its own copy of the training traces, then replays each held-out startup. A startup with n executed functions is checked after the first ¼, ½, ¾ and all of them (rounded down, at least one), and the distinct 4 KB pages touched at the four points are summed. Bt, Pt and Kt are these sums for the baseline, the candidate and a packing bound that packs each checkpoint's own functions from offset 0. A startup that does not strictly beat the baseline scores 0, and if Kt ≥ Bt any improvement scores 1. A missing file, or one with duplicate or unknown IDs, scores 0.
reward = min(1, min over families of mean ft ÷ 0.40)
Anchors
| Anchor | Reward | Score | Meaning |
|---|---|---|---|
| Floor | 0.000 | 0% | What a missing, invalid or rule-breaking submission scores |
| Baseline | 0.000 | 0% | The untouched starting point |
| Ceiling | 1.000 | 100% | The best result the metric allows |
| Regrade tolerance | Exact | Only identical results tie |
Environment
- Domain
- Software · Languages
- Shape
- Scaffolded
- Agent time
- 4 h per attempt
- Hardware
- 2 vCPU · 4 GB
- Expert estimate
- 4 h
- Task ID
function-page-layout