Results
Every model's 8 attempts with web search over a copy of the web frozen at October 1, 2026, on the scale that runs from the task's baseline at 0% to its ceiling at 100%.
Artifacts
What the top model on this task, by mean score, built.
The task
The agent starts from last season's aircraft and writes one JSON file, a parametric CAD model of the whole aircraft: a tapered wing of up to four panels per side with its structure, fixed or retractable main gear, plain, slotted or Fowler flaps, every control linkage, catalog components, a declared limit load factor, the parameters of a fixed, published autopilot, and the mission loads. The span is at most 2.5 m, with one motor and at most 60 Wh of battery. The kit holds the rules, the catalog, measured airfoil and propeller data, the autopilot source and the evaluator's mechanism code. The agent measures a built prototype with seven budgeted tools: 40 public_check calls for admission and mass properties, 12 wind_tunnel, 20 thrust_stand, 8 load_test, 8 drop_test and 16 section_test runs, and 12 unscored test_flight logs in development conditions. Each passing flight scores from 0 at the floor to 1 at the reference below, and the reward scales the weighted mission score by the aircraft's cost, within bounds.
| Mission | Flight | Measure | Scores 0 at | Scores 1 at |
|---|---|---|---|---|
| M1 | Three laps around pylons 1000 m apart, empty | 1 ÷ laps time | 305 s | 245 s |
| M2 | Three laps within 330 s, with up to 40 cargo blocks of 0.25 kg | Cargo kg ÷ laps time | 0.012 kg/s | 0.020 kg/s |
| M3 | Up to six packages of 0.2 to 1.5 kg, one dropped per pass on a ground target | Σ package kg × max(0, 1 − miss ÷ 10 m) | 1.0 kg | 9.0 kg |
Trajectories
Step through all 112 attempts from the pass@8 sweep with web search over an October 1, 2026 snapshot: every turn, tool call and submission. The best reward was 100.0%, by Claude Fable 5.2 on attempt 8. The results above come from these runs.
Failure analysis
Gemini 4 Argon averages 0.448 across its 8 attempts. Six of its 8 attempts submitted a final design without a real test flight. Most of Argon's failures come from cutting battery, wheel and pylon margins to the edge on its own simulator's word, so the harsher graded flights pushed the plane past its limits.
- Cut safety margins to the edge without testing in harsh conditionsFound in 3 of 8 attempts16.1
- Changed the design after its last test flight and never flew itFound in 2 of 8 attempts16.0
- Trusted its own motor-heat model instead of testing at hot cargo powerFound in 2 of 8 attempts14.7
- Found its submission would drain an aged battery: chased a redesign and never patched the fileFound in 1 of 8 attempts8.5
- ScoredMean reward 0.44844.8
Example failure analysis
Attempt 2 · reward 0.556 · Cut safety margins to the edge without testing in harsh conditions
Removed the larger wheels and kept a heavy cargo load because calm tests passed, so cargo landings broke wheels and hot cargo laps overheated the motor
- What it did
- A real cargo flight ended with a broken wheel on landing. A drop test showed larger wheels would survive, and the model fitted them. It then took the larger main wheels off again to save drag, calling the upgrade unnecessary, and checked the result only with drop tests at a gentle sink rate. Its own simulator had no check for wheel loads, which it noticed but never added. It then spent about 4.4 hours tuning its simulator and submitted a 6 kg cargo design after one warm cargo flight passed. Minutes later its recalibrated simulator said that design would overheat the motor on hot cargo flights. It built and flew a lighter 5.5 kg version, which passed, but never copied it into the submission before time ran out.
- What would have worked
- Keep the wheel upgrade, or drop-test the lighter wheels at the harder, drifting landings that windy cargo flights produce. Once its own calibrated model predicted overheating, it should have copied the lighter design that had just passed a real flight into the submission at once.
- Evidence
- Step 164
Finally, I noted that `flight_sim.py` did not flag the `wheel_failure` due to a missing check for impact forces exceeding rated load.
- Step 174
The upgrade to `wheel_102` on the mains in `v7e` was entirely unnecessary, adding `+0.00155 m²` of frontal area and `+115 mAh` of drag for no benefit.
- Step 357
would have hit `motor_overheat` with temperatures reaching `140C`
- Step 164
Why it is hard
In AIAA Design/Build/Fly, about a hundred university teams a year design, build and fly a small electric aircraft for scored missions, as engineers of small drones do. The design must close across structure, gear, aerodynamics and propulsion: more cargo raises the wing loading, which lengthens the take-off roll and hardens the landings, yet the aircraft must lift off within 25 m of roll, stop on a 60 m runway and survive its own loads. Retracting legs, flaps and control linkages are designed as geometry and must work over their whole motion. The simulator and structural coefficients are private, readings are noisy and test flights report no score, so 116 tool calls can fit models but not support trial and error.
How it is graded
The verifier runs offline and reads the file as data. The published autopilot flies an admitted design through a fixed suite of 34 flights the agent never sees: eight near the edges of the published conditions and 26 sampled from published ranges of wind, air, build, powertrain and structure. A passing flight scores where its measure falls between the mission's floor and reference, clipped to 0–1; any failure, such as a long take-off, a runway excursion, a structural failure or a battery drawn to zero charge, scores 0. RAC, the Rated Aircraft Cost in $K, charges $100 per lb of empty weight without the battery, $1 per W of rated motor power and $20 per manufacturing man-hour. A missing, invalid, oversized or non-admitted file scores 0.
Task update. The scores on this page come from a re-run of every model on the patched task (November 10–11, 2026), and the earlier attempts are no longer shown. The first version's cost factor let a cheap aircraft reach the cap on M1 alone; the bounds above close that. And a battery drawn to zero charge kept supplying power, so an empty pack could keep flying; such a flight now fails.
Anchors
| Anchor | Reward | Score | Meaning |
|---|---|---|---|
| Floor | 0.000 | 0% | What a missing, invalid or rule-breaking submission scores |
| Baseline | 0.227 | 0% | The untouched starting point |
| Ceiling | 1.000 | 100% | The best result the metric allows |
| Regrade tolerance | ± 0.000001 | Results closer than this tie |
Environment
- Domain
- Engineering · Aerospace
- Shape
- Scaffolded
- Agent time
- 8 h per attempt
- Hardware
- 4 vCPU · 8 GB
- Expert estimate
- 8 h
- Task ID
dbf-aircraft-design