Harbor-Index Beyond Pass Rate
Harbor-Index is a benchmark of 82 hard agentic tasks, distilled from 6,627 candidates. No frontier model solves even 30% of it. To see how the agents fail, an independent judge read all 1,414 rollouts and decided, from each trajectory, whether the task was genuinely solved. Almost none of them cheat: just 9 rollouts gamed the verifier. They fail honestly, and in three distinct ways. They run out the clock, they stop short of the gate, or they reason their way to a wrong answer. Which tasks a model actually wins comes down to the harness it runs in.
Honest, not gamed
Frontier agents fail honestly. Only 9 of 1,414 rollouts gamed the verifier.
The judge flags just 0.6% of rollouts as gaming the verifier (9 of 1,414), plus 39 false negatives from infra or verifier issues (2.8%). So about 97% of the verifier’s verdicts hold up, and almost no failure is cheating. The interesting question is not whether agents fail honestly. It is how.
Three honest layers
Honest failure splits into three layers: running out the clock, falling just short, and reasoning wrong.
Group the 17 judge codes into six families and each model’s failure shape becomes readable. The three honest layers split the 1,232 true negatives almost evenly: 444 ran out the clock, 361 fell just short, 427 reasoned wrong. The mix flips with capability. The closed leaders mostly fall just short; the open-weight field runs out the clock. Hover a segment for its definition, or click it for a cited example.
Ran out of time or token budget while still exploring, training, or debugging, and never wrote a gradable submission.
Example: MiMo V2.5 on algotune-optimize-lti-sim__3EUgiDP: The agent timed out with a solver that is both numerically unreliable under harness reuse and roughly 1.2x faster than baseline, far below the required 200x speedup.
See all 444 ran out the clock rollouts in the data browser →Different solves, no clear winner
Native usually edges ahead by a few points, but never by a statistically significant margin. What clearly changes is which tasks each harness solves.
Take any model, pit its native harness against terminus-2 on the same tasks. Native usually finishes a little ahead (GPT-5.5 24.7% vs 21.2%, Gemini 12.2% vs 7.3%), and terminus-2 wins a few (DeepSeek 5.0% vs 1.4%), but none of the four comparisons below is statistically significant (every p is above 0.05). The bigger change is which tasks get solved, and weaker models are more at the harness’s mercy: GPT-5.5 keeps 42% of its solves across the swap, the open-weight field only 9%, Gemini just 7%.
No comparison reaches a significant winner (all p > 0.05): native usually leads by a few points, but within noise at this sample size. What does change is which tasks get solved. The overlap — the share of a model’s solves that survive a harness swap — falls from 42% to 7% as models get weaker. The 6 open models row is the clean controlled test, where only the harness changes. The three frontier rows compare each model to its own native CLI (codex, gemini-cli), so they mix the scaffold with the harness and rest on ~80 pairs each, so read them as directional.
Why the harness changes the game
When the harnesses disagree, only one split is systematic: vision. terminus-2 can't see images.
The lead showed the harnesses solve different tasks. So which ones, and why? Strip out the ties and 89 model×task pairs (across all nine models) are solved by one harness but not the other. First, which tasks each one wins. Then, on the six open models — where the harness is the only thing that changes — what those solves cost.
On non-vision tasks the two harnesses are nearly even (37 vs 39). The one systematic gap is vision: terminus-2 is a text-only terminal and cannot see images. On the discordant tasks that hinge on reading a figure, board, or photo, native wins 12:1. It reads a labbench figure panel or the chess position straight from the image, while terminus-2 is blind to them. It reconstructs the board from pixels and misreads it, or never sees the figure at all.
On the six open models the harnesses tie on solves, but terminus-2 does almost twice the work to get there.
Same solve count (claude-code 28, terminus-2 26), very different cost. claude-code lands them on 14.8M completion tokens to terminus-2’s 21.248M, so it converts compute into solves about 55% more efficiently (1.892 vs 1.224 solves per million). terminus-2 offsets some input cost by caching a large prompt prefix, but it still spends far more to arrive at the same place.
It trades crashes and give-ups for timeouts.
terminus-2’s sturdier loop nearly eliminates crashes (40 → 1) and give-ups (25 → 1). But it converts that robustness into timeouts (121 → 166). The rate of genuine reasoning failures barely moves, which is why neither harness solves more.
And it pays a JSON-protocol reliability tax that native tool-calling never does.
terminus-2 makes the model write every action as escaped JSON, and weaker models botch the escaping and get rejected (45 events, up to 4 in one run). It also exposes only 2 tools versus claude-code’s 3, so every read, edit, and search has to go through the shell.
One group builds real solutions and falls just short of the pass mark. Another reasons sharply, then submits a wrong answer. The last reasons well but runs out of time. The cards below carry the behaviour and a cited example; the table holds the full per-model numbers.
Capable and honest. They fall just short at the last step.
The two best models, and 0 of the 9 gamed a verifier between them. They build real solutions and fall short only at the very end. GPT-5.5 ships numerically valid solvers that just miss the required speed or accuracy threshold (47 of its failures). Opus 4.8 does the deepest repair work in the field and stalls in optimization timeouts (52 ran out the clock, deep-repair stalls rather than laziness). Together they account for 64 of the 134 genuine solves. Trustworthy by construction.
Sharp reasoning, then they hand in a wrong answer.
Their defining move is handing the verifier a confident wrong answer it could have checked. Gemini 3.1 is wrong-dominant (69 cases, about half its failures). It prefers a remembered published figure over the value it just computed in the workspace. Qwen3.7 computes a passing value, then submits a different, wrong one.
Capable reasoning, but the clock runs out first.
The open-weight field reasons competently but rarely lands the artifact inside the 30-minute budget. Every member is clock-dominant (MiniMax 69, MiMo 64, DeepSeek 59, Kimi 52, GLM 48 ran out the clock). They prototype in /tmp, loop on exploratory analysis, and time out before writing solver.py or output.json. DeepSeek carries the most false positives of any model (3), but that is an asterisk, not its signature. Its signature is non-delivery.
| model | solve rate | signature |
|---|---|---|
| GPT-5.5 | 23% 37/161 | honest engineer, stops at the gate |
| Opus 4.8 | 16.8% 27/161 | deepest repair, slowest clock |
| Gemini 3.1 | 9.8% 16/164 | sharp one-shot, ships from priors |
| GLM 5.2 | 9.1% 14/154 | right logic, runs out the clock |
| Kimi K2.6 | 7.6% 12/157 | real engineering, misses the deadline |
| MiniMax M3 | 6% 9/151 | over-thinks into timeouts |
| Qwen3.7 | 5% 8/159 | computes right, submits wrong |
| MiMo V2.5 | 3.9% 6/155 | reasons until the clock runs out |
| DeepSeek V4 | 3.3% 5/152 | strong analysis, no artifact landed |