Results
Every model's 8 attempts with web search over a copy of the web frozen at October 1, 2026, on the scale that runs from the task's baseline at 0% to its ceiling at 100%.
Artifacts
What the top model on this task, by mean score, built.
The task
The agent gets 6,000 unlabeled and 400 labeled vibration records from a fleet of machines. Each record is 2,048 accelerometer samples at 20 kHz followed by the logged mean shaft speed in Hz, and the eight classes are healthy, outer-race, inner-race and ball faults, misalignment, unbalance, looseness and gear faults. The deliverable is an embedding, not a classifier: /app/submission/embed.py, whose top-level embed(X) maps an (n, 2049) float64 matrix to a finite (n, d) float64 matrix with 1 ≤ d ≤ 512. Output must be deterministic, and each row may depend only on its own record. A fixed linear probe trained on the recorded fleet then classifies records from machine variants that appear nowhere in the data. Weights may sit beside the module within 100 MiB, and only /app/submission/ is collected. Grading runs offline on four CPUs, with 1,800 seconds per call and pinned NumPy, SciPy, scikit-learn and CPU-only PyTorch. A working starter and an in-fleet evaluator are included, and public code and papers are fair game.
Trajectories
Step through all 112 attempts from the pass@8 sweep with web search over an October 1, 2026 snapshot: every turn, tool call and submission. The best reward was 81.5%, by Claude Fable 5.2 on attempt 5. The results above come from these runs.
Failure analysis
GPT-6.1 Sol averages 0.580 across its 8 attempts. The model generally recognizes faults on the known machines, but accuracy drops on the new and noisier machines. Most of Sol's failures come from fine-tuning against its own simulator, which was far easier than the real test, and from assuming that some fault patterns would persist across all new machines.
- Fine-tuned against its own simulator, which was far easier than the real testFound in 5 of 8 attempts25.0
- Assumed some fault patterns look the same on any machine, and only checked with a simulator that assumed it tooFound in 1 of 8 attempts8.7
- Tested at the real noise level but found no usable signal for bearing faultsFound in 2 of 8 attempts8.4
- ScoredMean reward 0.58058.0
Example failure analysis
Attempt 6 · reward 0.520 · Fine-tuned against its own simulator, which was far easier than the real test
It chose its last changes by small gains on its own simulator (0.383 to 0.401), which already scored above full marks; the final balanced accuracy was 0.255.
- What it did
- It noted its simulations were not the hidden test (step 44) but kept using them to select. In the last two hours it adopted gain marginalization because extra synthetic tests rose from 0.383 to 0.401 (step 205) and added a red-background correction (step 282). Its final audit on its own fresh records read 0.426 (step 294). It had printed per-class recall on structural tests earlier (step 136), but its late choices used aggregate scores. Its final balanced accuracy was 0.255.
- What would have worked
- Harden the simulator until it scores below the cap, then select on it, and look at per-class recall. Another attempt built a strict validation reading 0.31 to 0.34, close to the final scores.
- Evidence
Why it is hard
Scoring is pure transfer. A damaged bearing produces impulse trains at characteristic orders of shaft speed, set by its geometry, and the scored machines' geometry appears nowhere in the data. Their records also come from a wider envelope: higher speed, a different structural response and a much lower signal-to-noise ratio, −24 to −15 dB against 3 to 12 dB in the data. An absolute-frequency spectrum memorizes the recorded machines, because the same fault shows up at a different frequency on a faster machine or another bearing. Public methods such as envelope analysis, order tracking, CNNs trained on the CWRU bearing rig, TS2Vec and IRM do not transfer directly to unseen geometries from 400 labels.
How it is graded
The verifier fits a fixed probe, a standardized logistic regression with C = 1, on labeled embeddings from the recorded fleet, and measures its balanced accuracy (mean per-class recall) on the unseen variants. The reward rises linearly from 0 at chance, 1/8, to 1 at a balanced accuracy of 0.375, a fixed estimate of the best achievable, and is clipped to that range. A second call on a subset of rows must match the full call within 10−4 of each column's interquartile range, ruling out batch normalization and unseeded randomness. A submission scores zero if it cannot run, times out, breaks the file rules, returns an invalid array, fails this check or leaves the probe no varying column.
Anchors
| Anchor | Reward | Score | Meaning |
|---|---|---|---|
| Floor | 0.000 | 0% | What a missing, invalid or rule-breaking submission scores |
| Baseline | 0.000 | 0% | The untouched starting point |
| Ceiling | 1.000 | 100% | The best result the metric allows |
| Regrade tolerance | ± 0.002 | Results closer than this tie |
Environment
- Domain
- ML · Training
- Shape
- End to end
- Agent time
- 8 h per attempt
- Hardware
- 4 vCPU · 8 GB
- Expert estimate
- 4 h
- Task ID
bearing-fault-transfer