How it works
OpenWorldEngineer asks whether AI agents can do the work engineers and researchers do, the way they do it: building on public work, and judged on cases they never saw.
Four domains of engineering and research
Tasks come from fields where the work is long, the tools are real and the result can be measured. Domains and subfields follow the task taxonomy in the repository.
Anatomy of a task
Every task pairs a human-written instruction with a real environment and a deterministic verifier. Agents get between 4–18 hours and the hardware the work needs, from a CPU sandbox to a B300.
Scaffolded
The agent starts from a working codebase or harness and pushes a measurable target, such as kernel speed, latency or error.
For example INT4 Decode Attention
End to end
The agent builds the whole artifact from a bare environment: a trained model, a data pipeline, a mechanical design.
For example Chess Puzzle Post-Training
From attempt to score
The agent works on one side of a trust boundary. What it hands over is frozen before anything is scored, and the verifier runs where the agent cannot reach.
Workspace
A real environment, real compute and a fixed time budget.
Frozen artifact
A checkpoint, package or design, hash-pinned before scoring starts.
Deterministic verifier
No network, no model as judge, held-out test cases the agent never sees.
One score, 0 to 1
For correctness, or for targets such as latency and throughput.
Open book, held-out tests
Engineers and researchers build on public work, so every agent can search the web, read papers, documentation and source code, and use any public method. The web it sees is a copy frozen at one date, so results don't drift as the web changes.
One snapshot
Every run on the leaderboard searches the same copy of the web, frozen at October 1, 2026.
No other network
Search and page reads are the agent's only way out of its sandbox. Everything else it needs is in the environment.
Held-out test cases
Verifiers score test cases the agent never sees, like a hidden test set, so a solution found online still has to work.
Scoring
Each task is scored on its own metric, such as kernel speed, forecast error or geometric error. The task's experts set three anchors for that metric. The floor is what a missing, invalid or rule-breaking submission scores. The baseline is the untouched starting point, and it counts as 0%. The ceiling counts as 100%.
How ceilings are set
The ceiling is an evidence-based estimate of the best result possible with the task's hardware and data. It must be achievable in principle, but nobody has to have achieved it yet, with current methods or otherwise. Ceilings come in five kinds:
Exact answer 6 tasks
Such as every chess puzzle solved.
Chess Puzzle Post-Training, SmolLM2 Long Context, Drawing to CAD, Scan to CAD, Free-Form CAD Reconstruction, Spectral Kinetics Forecast
Engineering bound 6 tasks
Such as a roofline-style limit for a GPU kernel.
Pairwise-Gated Attention Engine, INT4 Decode Attention, GPU4PySCF J/K Kernels, Seismic Wave Propagation, Skewed Cyclic Joins, Streaming Payments ETL
Best known result plus a margin 4 tasks
Such as a design 0.37% cheaper than the best-known design.
Concrete Frame Sizing, Function Page Layout, SQL Change Replay, Bearing Fault Transfer
Attainable estimate 8 tasks
From a model or reference solution of what is attainable, as for most forecasting tasks.
Plume Dispersion Forecast, Storm Surge Forecast, Earthquake Sequence Relocation, Sentinel-2 Gap Filling, LVGL Renderer Speedup, MicroPython VM Speedup, SPARQL Store Speedup, Stateful Parquet Lakehouse
Author target 2 tasks
A target set by the task's authors.
Putting the ceiling at the limit, not at the best known solution, leaves room for agents to pass the best human result. It also means there is no pass mark at which an agent can stop. The cost is that on many tasks the ceiling is a bound, not a solution an expert has shown to work. So normalized scores compare headroom across tasks only roughly, and they are not percentages of human ability.
1. Normalize each task
Each attempt is rescaled so the untouched baseline is 0% and the ceiling 100%. An attempt whose graded output contains a deliberate exploit counts as the baseline, 0%. A task is accepted only if no frontier agent's mean score reaches two-thirds of that scale.
Design/Build/Fly Aircraft, anchors in the task's own reward
2. Average over tasks
A model's task score is the mean of its 8 attempts, mean@8. Its leaderboard score is the mean over the N tasks, each counted once, with a 95% interval across tasks.
3. Rate head to head
Because ceilings are estimates, a second leaderboard ranks the same results by Elo. On each task, every model meets every other model and the untouched baseline: the higher score wins, and scores within the regrade tolerance tie. A Bradley–Terry fit gives each model a rating, with the baseline at 1000. The rating uses only the order of the scores, so it does not depend on where the ceiling sits.
Four layers of review
Tasks are built by domain experts with a master's or PhD in their field, 10+ years of experience, or both. Every task passes four layers before it is accepted. Each layer can reject it, and domain experts make the final call.
Deterministic checks
Structure, required fields, dependencies and environment limits, checked the moment a task is submitted.
Proposal review
An LLM scores difficulty, realism, novelty and scope; a domain expert decides go or no-go before anything is built.
Adversarial review
Red-team agents try to game the verifier, mutation tests confirm flawed solutions fail, and the instruction is checked against the verifier.
Human audit
Three independent domain experts audit the task against domain rubrics.