Results
Every model's 8 attempts with web search over a copy of the web frozen at October 1, 2026, on the scale that runs from the task's baseline at 0% to its ceiling at 100%.
Artifacts
What the top model on this task, by mean score, built.
The task
The agent writes one program that turns a component's drawing pack and a request into STEP solids. A pack holds PNG images at an exactly known scale: in the public examples, 14 shaded views, six along the axes, plus dense axis-aligned sections, mostly every 2 mm along Z and 4 mm along X and Y, that mark material boundaries, cavities included. A request asks for the nominal part and edits that twist it and bend it in X and Y, scaled by w = t2(3 − 2t) with t = z ÷ h for a requested height h. Each edited part must be one valid solid, though the pack shows only the nominal part. The agent gets one public example for each of six component classes, with STEP files of the nominal part and two edits, and tools that render and score a candidate. Only a source archive of at most 16 MiB transfers. Each pack runs offline with four CPUs, 16 GB of RAM and 900 seconds.
Trajectories
Step through all 112 attempts from the pass@8 sweep with web search over an October 1, 2026 snapshot: every turn, tool call and submission. The best reward was 94.9%, by GPT-6.1 Sol on attempt 6. The results above come from these runs.
Failure analysis
Claude Opus 5.5 averages 0.539 across its 8 attempts. Its best attempts built most of the new parts accurately, but in its weakest attempts the builder failed and most parts fell back to simple placeholder shapes. Most of Opus's failures come from proceeding with a builder despite its own tests showing failures or running out of time, and from never testing its program under the stated 32 MiB limit on written files.
- Proceeded with a builder despite its own tests showing failures or running out of timeFound in 2 of 8 attempts18.2
- Never tested its program under the stated 32 MiB file limitFound in 1 of 8 attempts12.4
- Built every part from horizontal slices; left the other drawings unused on its weakest partsFound in 4 of 8 attempts8.6
- Tested its builder on new parts only at the end, then left the breakage unfixedFound in 1 of 8 attempts7.0
- ScoredMean reward 0.53953.9
Example failure analysis
Attempt 5 · reward 0.182 · Proceeded with a builder despite its own tests showing failures or running out of time
Its patch-fitting converter finished only 40 to 60% of example outputs, and it kept it, assuming the grader's faster machine would do better.
- What it did
- It built a voxel model of each part, then split the mesh into up to 300 patches, each fitted with one smooth surface (step 39). Patch borders often broke the grader's mesher, so it wrote a crude backup first, worth only about 0.03 to 0.06 (step 158). When a variant passed it scored 0.84 to 0.87 (step 202). All its testing ran the six example parts (for example step 236); it never rendered a new part. It assumed the grader's faster machine would succeed more often than its local tests (step 234), and at the end reported only 40 to 60% of outputs reaching full quality (step 297).
- What would have worked
- Measure the success rate on new parts rendered from the examples, and switch to a construction that is valid by design when that rate is low. The high attempts lofted smooth surfaces between section outlines and stitched them without fragile patch borders.
Why it is hard
Reverse engineering often starts from views and sections of a part with no CAD model, which engineers reconcile into one consistent solid. Here a program must do it alone, offline, for pump and flow components it has never seen. A class does not fix the shape: feature counts, layouts and free-form geometry must be inferred from each new pack. The converter must match boundaries across many views, tell walls from cavities, recover internal passages and build one gap-free valid solid that survives the edits. The inputs are antialiased raster images, and the 0.25 mm tolerance spans only 2.24 pixels. Feature recall weighs a small hole as much as a large wall, and a mesh that looks right can still have wrong internal geometry.
How it is graded
The verifier runs the program on six unfamiliar drawing packs and scores every requested solid, nominal and edited. Each must be one closed, valid solid of positive volume with no loose geometry, at most 300 B-rep faces and coordinates within ±500 mm. It is compared with the target as written, with no alignment, scaling or repair.
reward = mean over all cases
Volume IoU is the intersection over union of the volumes. Oriented coverage is the share of surface samples, taken both ways, within 0.25 mm of the other surface with normals within 20°. Feature recall gives each target face equal weight and counts its samples within 1 mm of the candidate. A missing, invalid or unmeasurable output scores 0, but valid outputs written before a crash or timeout keep their credit. A missing or malformed archive scores 0.
Anchors
| Anchor | Reward | Score | Meaning |
|---|---|---|---|
| Floor | 0.000 | 0% | What a missing, invalid or rule-breaking submission scores |
| Baseline | 0.098 | 0% | Copying the public reference of the requested part class, with no reconstruction |
| Ceiling | 1.000 | 100% | The best result the metric allows |
| Regrade tolerance | ± 0.070 | Results closer than this tie |
Environment
- Domain
- Engineering · CAD
- Shape
- End to end
- Agent time
- 8 h per attempt
- Hardware
- 8 vCPU · 64 GB
- Expert estimate
- 8 h
- Task ID
parametric-cad-reconstruction