Results
Every model's 8 attempts with web search over a copy of the web frozen at October 1, 2026, on the scale that runs from the task's baseline at 0% to its ceiling at 100%.
Artifacts
What the top model on this task, by mean score, built.
The task
oxilite is a Rust SPARQL 1.1 store that compiles queries to SQL over a single SQLite file. From a pinned upstream commit, the agent must make it load data faster, write a smaller database file and answer queries with less work, without changing any answer. Only two crates and new crates of its own under crates/ext/ are collected, and the grader builds them offline beside its own frozen harness; changes to the workspace Cargo.toml, profiles and Cargo.lock are not used. SQLite 3.53.2 is reached only through a fixed backend trait with locked settings, lz4_flex is the only new external crate allowed, and unsafe, inline assembly, build scripts and proc macros are banned. Work is cachegrind's cycle estimate for a whole process, C = Ir + 10·(L1 misses) + 100·(last-level misses), which repeats exactly for the same binary. Grading uses private data, larger than the development data, and private query groups. Any wrong answer or broken rule scores the 0.1 floor, the same as the unchanged store.
| Part | Weight | Cap | Measured as |
|---|---|---|---|
| Load | 10% | 25× | Cost of loading the private N-Quads file into an empty store |
| Size | 10% | 8× | Size of the database file after loading |
| Queries | 80% | 10× per group | Cost of each query group's process, with its updates and queries, above the harness's fixed start-up cost F = 2,861,773 |
Trajectories
Step through all 112 attempts from the pass@8 sweep with web search over an October 1, 2026 snapshot: every turn, tool call and submission. The best reward was 100.0%, by Claude Opus 5.5 on attempt 6. The results above come from these runs.
Failure analysis
Gemini 4 Argon averages 0.232 across its 8 attempts. It made large speedups on the practice set in every attempt, but most attempts gave a wrong answer on a graded query. Argon's main failure is adding shortcuts that relied on quirks of the practice data without testing them on altered queries or renamed data.
- Added shortcuts that relied on quirks of the practice data, without testing them on altered queries or renamed dataFound in 6 of 8 attempts67.5
- Never timed query plans on larger data, although the task said grading data is largerFound in 2 of 8 attempts9.3
- ScoredMean reward 0.23223.2
Example failure analysis
Attempt 1 · reward 0.100 · Added shortcuts that relied on quirks of the practice data, without testing them on altered queries or renamed data
Added a range-test shortcut for classes whose IDs were contiguous in its dev store, checked it and its other fast paths only on dev, and a private group returned wrong rows.
- What it did
- At step 48 it noted that private IRIs may be renamed and that hardcoding IRI facts would be wrong. It still tuned fast paths to dev layout facts, such as rating predicates having adjacent IDs (step 91). At 14.4h (step 1310) it saw four classes with contiguous IDs in its dev store and added a load-time rule that swaps the class-membership join for an ID range test whenever a class is contiguous. All 214 of its checker runs used the dev queries, and it never loaded data other than dev.nq or dumps of it. The grader does not show which change broke the private group, so the link to this shortcut is likely but not proven. It submitted at dev 0.89.
- What would have worked
- Treat any fact found in the practice data as unproven, and test each shortcut on renamed or regenerated data and on altered queries. The best attempt of this model used general algebra rewrites instead of data-layout tricks.
Why it is hard
Every gain has to hold under strict gates. Answers must match row for row, term ids must stay unique, malformed input must still be rejected, and no W3C SPARQL test may regress. All storage goes through the frozen SQLite backend, so Oxigraph's own storage cannot replace it. The data is realistic e-commerce data (products, offers and reviews) made by the Berlin SPARQL Benchmark generator. Grading uses private data, larger than the development data, and private query groups that use the development queries' SPARQL features. Database and knowledge-graph engineers maintain this kind of embedded RDF store for browser, mobile and edge apps.
What the agent starts with
The agent gets the oxilite source at the pinned commit, with the W3C rdf-tests and BSBM tools as submodules, and a development kit: a 1,000-product dataset with public namespaces, 27 development query groups with expected answers, the pristine measurements and the data generator. score.sh builds and grades the crates as the verifier does, on development data; --all adds the W3C suites and the id gate, which it skips by default. It also gets the submission and grading rules, the grader's code including the answer comparison, the grader's Rust toolchain, the patched valgrind and public internet access.
How it is graded
The verifier builds only the collected crates offline, in a separate container with no network, next to frozen copies of everything else. The load runs twice, into two stores, and each query group runs once on each store in a fresh process; the higher cost, the larger file and the answers from these runs count. Any gate failure scores the 0.1 floor: a broken build or rule, one wrong item in any group, accepted malformed input, two terms sharing an id, a W3C regression, an invalid store, a measurement violation, more than three times the original's CPU time plus 10 seconds or its memory plus 1 GiB, or grading past 4,800 seconds. Otherwise the reward is:
G = 0.1·nload + 0.1·nsize + 0.8·meang ng
reward = clip(0.1 + 0.9·G ÷ GC, 0.1, 1), where GC = 0.76556 is the G of the ceiling estimate
Anchors
| Anchor | Reward | Basis |
|---|---|---|
| Floor | 0.1 | Any failed gate, or a valid submission with G at or below 0 |
| Baseline | 0.1 | The unmodified crates, where every cost ratio is exactly 1 and G is 0 |
| Ceiling | 1 | G = GC = 0.76556, the G of an estimated best achievable result; a higher G is clipped at 1 |
| Regrade tolerance | ± 0.001 | Allowed difference between two grades of the same submission; the cost counts repeat exactly for the same binary |
The ceiling is an estimate, not a proven bound.
Environment
- Agent time
- 16 hours
- Hardware
- 8 CPUs, 32 GB RAM, no GPU
- Network
- Public internet while working; none during grading
- Software
- Rust 1.97.1 on Debian bookworm, SQLite 3.53.2 through rusqlite 0.40.2, valgrind 3.19.0 patched for repeatable counts
- Verifier
- Separate container; 8 CPUs, 32 GB, 90-minute limit
- Expert estimate
- 8 hours
Prior work and assets
The codebase is oxilite (MIT OR Apache-2.0). Related assets: Oxigraph and its spareval SPARQL evaluator; Bizer and Schultz, The Berlin SPARQL Benchmark (2009), for the data generator and query templates; the W3C rdf-tests SPARQL conformance suites, used as a gate; and Cachegrind, the cost model.