Results
Every model's 8 attempts with web search over a copy of the web frozen at October 1, 2026, on the scale that runs from the task's baseline at 0% to its ceiling at 100%.
Artifacts
What the top model on this task, by mean score, built.
The task
The agent improves a pipeline that reads the DXF drawing of one machined part and writes a CadQuery program that rebuilds it. Each drawing is 1:1 in millimetres, with front, top and right views in third-angle projection plus an isometric view; it has no dimensions, and visible, hidden and tangent edges sit on separate layers. The starting pipeline only intersects the extruded outer outlines of three views. The agent gets the grader, the code that makes the drawings, a local scoring script and 30 development parts with their true programs: plates, brackets, flanges, enclosures, shafts and blocks with holes, slots, fillets, chamfers and more. The hidden test parts differ from them. The pipeline runs once per hidden part, offline, within 80 times the starting pipeline's median CPU time and 4 GiB of memory. Each program must use an allowlisted CadQuery subset and build one closed solid of at most 400 faces within 120 CPU seconds. A part scores 1 within 0.005 mm of the true part.
Trajectories
Step through all 112 attempts from the pass@8 sweep with web search over an October 1, 2026 snapshot: every turn, tool call and submission. The best reward was 49.6%, by GPT-6 Astra on attempt 2. The results above come from these runs.
Failure analysis
Gemini 4 Argon averages 0.184 across its 8 attempts. It rebuilt the 30 practice parts almost perfectly, but got many of the new hidden parts wrong. Most of Argon's failures come from copying rules from the practice answers instead of reading each drawing, and from testing only on new parts built the same way as the practice ones.
- Copied rules from the practice answers instead of reading each drawingFound in 4 of 8 attempts40.4
- Tested on new parts, but only ones built the same way as the practice partsFound in 2 of 8 attempts21.1
- Spent its last hours perfecting practice parts instead of testing new onesFound in 2 of 8 attempts20.0
- ScoredMean reward 0.18418.4
Example failure analysis
Attempt 4 · reward 0.122 · Copied rules from the practice answers instead of reading each drawing
It tuned its output until it matched the practice answers line for line, never tested on a part it hadn't tuned on, and stopped with 6.4 hours left.
- What it did
- It read the 30 shipped answer programs and wrote one reconstructor per family. It tuned them until its output was line-for-line identical to the answers, deriving rules from the constants the dev generator used. It claimed early that this would generalize to hidden parts but never tested that. When its sandboxed dev score reached 1.000 at 9.6 h, it submitted and stopped with 6.4 h left.
- What would have worked
- Hold out some dev parts, or perturb dev drawings, to test the recognizers before trusting them. GPT's best attempt on this task reached 0.47 with a general pipeline.
Why it is hard
Many machined parts exist only as 2D drawings, and turning one into a 3D CAD model is paid work still done by hand. These drawings are exact but carry no dimensions, so every value has to be read off the views, and interpretation is the hard part. A line in one view can be a hole, a pocket wall, a step or the outline of a curved face; only the other views tell which. Hidden lines from internal features overlap each other and the visible edges, features can be tilted away from the part's axes, and fillets, chamfers and drafts add tangent edges and change where features end. Published methods, from wireframe reconstruction to learned drawing-to-CAD models, handle only simple shapes.
How it is graded
The verifier runs offline in a separate container. Both solids are meshed and sampled. Every face of the true part gets equal weight, so a missing small hole costs a whole face's worth of error. dmean and dmax average the mean and the largest sampled distances over both directions. Because a drawing can fit more than one solid, a part also scores 1 if the grader's drawing of the rebuilt solid matches the input within 0.005 mm in every view and layer. A part scores 0 if the pipeline fails or exceeds a limit, or if its program breaks the allowlist or does not build one closed solid of at most 400 faces. The starting pipeline scores exactly 0.
part score = (f(dmean) + f(dmax)) ÷ 2; reward = mean over the hidden parts
Anchors
| Anchor | Reward | Score | Meaning |
|---|---|---|---|
| Floor | 0.000 | 0% | What a missing, invalid or rule-breaking submission scores |
| Baseline | 0.000 | 0% | The untouched starting point |
| Ceiling | 1.000 | 100% | The best result the metric allows |
| Regrade tolerance | ± 0.005 | Results closer than this tie |
Environment
- Domain
- Engineering · CAD
- Shape
- Scaffolded
- Agent time
- 16 h per attempt
- Hardware
- 8 vCPU · 32 GB
- Expert estimate
- 10 h
- Task ID
drawing-to-cad