Results
Every model's 8 attempts with web search over a copy of the web frozen at October 1, 2026, on the scale that runs from the task's baseline at 0% to its ceiling at 100%.
Artifacts
What the top model on this task, by mean score, built.
The task
The agent improves a pipeline that reads a noisy 3D scan of one machined part and writes a CadQuery (Python CAD) program that rebuilds it. The starting pipeline only extrudes the scan's best silhouette and cuts through-holes. Each scan is 100,000 surface points with 0.02 mm Gaussian noise. The agent gets the grader, the scan simulator, a local scoring script and 30 development parts with their true programs: plates, brackets, flanges, enclosures, shafts and blocks with holes, pockets, slots, bosses, drafts, fillets and chamfers. The graded parts differ from them. Only the pipeline directory, at most 50 MiB, is collected. It runs once per graded part, offline, on one reported CPU, within 80 times the starting pipeline's median CPU time and 4 GiB of memory. Each program must use an allowlisted CadQuery subset, with no meshes or splines, and build one closed solid of at most 400 faces within 120 CPU seconds. Each part is scored by how close the rebuilt solid is to the true one.
Trajectories
Step through all 112 attempts from the pass@8 sweep with web search over an October 1, 2026 snapshot: every turn, tool call and submission. The best reward was 58.6%, by GPT-6.1 Sol on attempt 3. The results above come from these runs.
Failure analysis
Gemini 4 Argon averages 0.287 across its 8 attempts. It matched the practice parts closely, but on the hidden parts many brackets, blocks and flanges came out no better than the starter. Most of Argon's failures come from copying rules from the practice answers instead of reading each part's shape from its scan, and from crashing on new shapes and falling back to the starter output.
- Copied rules from the practice answers instead of reading each part's scanFound in 5 of 8 attempts43.1
- Crashed on new shapes and fell back to the starter outputFound in 2 of 8 attempts18.9
- Ran only the 30 practice parts; shipped with no catch-all for unseen shapesFound in 1 of 8 attempts9.3
- ScoredMean reward 0.28728.7
Example failure analysis
Attempt 1 · reward 0.295 · Copied rules from the practice answers instead of reading each part's scan
It judged every change by matching dev answer programs and never simulated a new part; many hidden parts were no better than the starter.
- What it did
- At step 28 it set out to reproduce the answer programs exactly. It read the scan simulator at steps 3 and 359 but never built a new test part. It checked only with its own dev evaluation script. In the last minutes it made block-6 identical to its answer program and submitted on 0.87 with 4 minutes left.
- What would have worked
- Fit each part's features from its scan rather than reproducing the answer programs, and check that rules carry over on parts built with the simulator. The best attempt scored lower on practice parts (0.76) but higher on hidden ones (0.34).
- Evidence
- Step 28
if we can perfectly reconstruct the exact CAD features, including their quantized parameters and precise pose (the canonical frame), then both `d_mean` and `d_max` should drop below that critical 0.005 mm threshold
- Step 785
are **absolutely, unequivocally identical** between my output and the GT
- Step 28
Why it is hard
Engineers rebuild CAD models from 3D scans of parts with no drawings by hand, in tools such as Geomagic Design X; this task asks for a pipeline that does it unattended. A surface fit is not enough: the output is an editable feature program with every feature present and every dimension as designed. The 0.02 mm scan noise is four times the 0.005 mm full-credit distance, so dimensions must be exact. Parts sit in random poses, features can be turned away from the part's axes, and fillets, chamfers and drafts blur the edges where features meet. Public methods such as RANSAC primitive detection, Point2CAD, ComplexGen and DeepCAD yield surface patches or simple extrusions, and need GPUs the grader lacks.
How it is graded
The verifier runs offline and rebuilds and measures every solid itself. A bit-level mismatch in its re-run of the true programs or the starting pipeline voids the trial. Distances run from points sampled on each solid to the other solid's mesh, with every face of the true part weighted equally, so one missing hole costs a whole face's worth of error. dmean and dmax average the mean and the largest distances over both directions.
part score = (f(dmean) + f(dmax)) ÷ 2; reward = mean over the graded parts
A part scores 0 if the pipeline fails or exceeds its CPU or memory limit, or if the program breaks the allowlist or does not build one closed solid of at most 400 faces. A missing submission scores 0, as does the starting pipeline.
Anchors
| Anchor | Reward | Score | Meaning |
|---|---|---|---|
| Floor | 0.000 | 0% | What a missing, invalid or rule-breaking submission scores |
| Baseline | 0.000 | 0% | The untouched starting point |
| Ceiling | 1.000 | 100% | The best result the metric allows |
| Regrade tolerance | ± 0.005 | Results closer than this tie |
Environment
- Domain
- Engineering · CAD
- Shape
- Scaffolded
- Agent time
- 16 h per attempt
- Hardware
- 8 vCPU · 64 GB
- Expert estimate
- 10 h
- Task ID
scan-to-cad