Today we're releasing Harbor-Index 1.0, a compact, high-signal benchmark for evaluating frontier agents. Its 82 tasks were chosen from 6,627 candidates across 54 benchmarks integrated as Harbor adapters . The final set spans 29 of those benchmarks across a broad range of domains, environments, and evaluation styles. Browse the tasks on the Harbor Hub.
Harbor-Index is built on four principles: it should be compact, diverse, challenging, and of high quality. The hard part is finding tasks that frontier agents cannot solve, but are not broken. A broken task can fail in several ways, including instruction-verifier mismatch, success condition underspecification, and environment brittleness. Most of the effort behind Harbor-Index went into ensuring a failure reflects a model's limits, not a defect in the task. No agent-model pair we tested gets above 30% on Harbor-Index.
The chart above plots pass rate against cost for each agent-model pair. GPT-5.5 on the Codex CLI leads at 28.1%, followed by Claude Opus 4.8 with Claude Code at 20.7%.
Cache-hit rate impacts cost in surprising ways on OpenRouter.
We reconstruct each run's cost from its token usage, using official API pricing for GPT, Claude, and Gemini and OpenRouter pricing for the open models. Note that running open models through Claude Code on OpenRouter produces a low cache-hit rate, which inflates their cost well beyond the same models on Terminus-2 and pushes them off the frontier.
Why Harbor-Index?
Agentic benchmarks are multiplying quickly, spanning software engineering, scientific reasoning, search, data analysis, cybersecurity, and long-horizon tool use. That breadth is valuable but expensive, since evaluating every new agent across the full landscape takes enormous time and compute. Harbor-Index answers with a compact meta-dataset of 82 tasks that remain genuinely hard for frontier models while supporting reliable comparison across agents efficiently. Our large-scale evaluation across 54 adapted benchmarks consumed 226B tokens and over $300K of compute, while condensing into Harbor-Index costs a frontier agent only a few hundred dollars.
Harbor-Index is built differently from previous indices.
It is worth being explicit about what Harbor-Index is not. It is a hard benchmark, not a general-capability metric. Aggregate indices like the Epoch Capabilities Index stitch 50+ benchmarks into a single latent scale to track general capability as individual benchmarks saturate, and score-prediction methods like BenchPress infer a model's missing benchmark scores from its peers to avoid running every evaluation. Those efforts summarize or forecast broad ability across many models and years. Harbor-Index does neither. In fact, difficulty and predictive power pull in opposite directions: the tasks most useful for predicting other scores are the ones models split on, whereas a task every frontier agent fails is a constant, near-useless feature for prediction. Those maximally hard tasks are exactly what Harbor-Index keeps, because they mark the frontier of what agents cannot yet do. Our job is not to make them predictive but to make sure their failures are real: each task is filtered and fixed until a miss reflects a model's limits rather than a broken verifier.
Harbor adapters make this possible: they translate heterogeneous benchmarks into a single standard format, so that any agent and model can run through the same evaluation pipeline. We verify that tasks in Harbor format stay consistent with the original benchmarks through parity experiments, and then catch remaining problems with AI and human auditing.
Building Harbor-Index
We start from the full Harbor trial pool of 6,627 candidates drawn from 54 benchmarks via Harbor adapters, then pass it through a three-stage funnel: a difficulty filter, an automated AI audit, and a human audit. We further harden the remaining tasks with repeated rounds of automated auditing and reviewer fixes, and the final 82 tasks constitute the Harbor-Index.
The result is as diverse as it is difficult: the final set spans 29 benchmarks across seven domains, from software engineering and scientific research to tool use, mathematics, data analytics, and security, so a run of the index assesses model capability across a broad range of agentic work.
Stage 1: Difficulty filter
A task should be difficult enough that frontier models cannot reliably solve it. For each candidate task, we evaluated three models (Claude Opus 4.6, GPT-5.4, Gemini 3.1 Pro) each in two harnesses (model's native CLI and Terminus-2) over 3 repetitive runs for a total of 18 trials. A task passes the filter if its pass rate across the 18 trials is below 34%, and this cuts the candidate pool from 6,627 tasks to 1,311.
Stage 2: Broken task screen
Difficulty alone is not enough, since many of those 1,311 tasks are hard only because they are broken and thus unsolvable. We use an LLM auditor to inspect each task with the full task definition (instruction, environment, tests, reference solution), the agent trajectories, and verifier outputs of 18 trials, so its verdicts are grounded in specific step IDs, commands, and observations.
The auditor scores each task against a two-part rubric
- Test-instruction alignment. Every assertion the verifier makes must trace back to a requirement covered by the instruction or easily implied from the environment, and every requirement in the instruction must be covered by a test. Some typical misalignment include instruction being ambiguous or verifier checking an unstated requirement.
- Essential difficulty. The difficulty must come from genuine reasoning, algorithmic thinking, domain expertise, long-horizon interaction, or multi-step execution, not from whitespace, decimal precision, undisclosed tolerance bounds, or magic strings (function names, error messages, JSON keys) that instruction never pins down.
The auditor is anchored by a small calibration set of pre-labeled broken tasks, which it uses to reject new ones by analogy. This stage is a high-recall pre-filter which removes the clearly-broken tasks: every survivor is audited again by hand in Stage 3 for the final decision. The AI audit narrows the pool from 1,311 tasks to 307.
Stage 3: Human audit
The 307 survivors go to 14 reviewers, who independently re-audit each task against the same rubric the AI judge used, so both stages enforce one consistent definition of quality. A senior panel of three reviewers then makes the final cut, weighing difficulty, diversity, and insight, and deliberately spreading the accepted set across domains and benchmarks rather than letting a few prolific sources dominate. This selects an initial set of 100 candidate tasks.
A third of the "hardest" tasks are broken, sometimes in unexpected ways.
- A GAIA2 task: the environment never fires the events the agent is meant to react to, so every agent stalls in an empty notification-wait loop until it times out.
- A SWE-bench Pro task: the verifier overreaches, asserting
data-testidstrings and log-message formats the instruction never specifies, so a correct solution fails on clerical grounds. - A CRUST-Bench task: it pins the transpilation to a fixed, hand-written Rust interface, so a correct and memory-safe solution is rejected when its signatures or ownership annotations don't match the reference exactly.
Audit-and-fix loop
We put the 100 tasks through repeated rounds of automated audit and reviewer fixes, and cut tasks that are broken or too easy after the fix, and 82 tasks remain. In the audit, an LLM judge grades the verifier by running in a fresh copy of each task's own sandbox, reading the trajectories, verifier outputs, the tests, and the reference solution, and labeling each rollout a true or false positives/negatives, with every claim cited to a specific step or file. We pay special attention to false positives that let agents pass without genuinely solving the task, and false negatives that reject correct solutions. We then repair the task, re-run the frontier models and audit again, repeating until the set stabilizes.
We also rely on the Terminal-Bench 3 rubrics and the Terminal-Bench auditing pipeline to capture potential task vulnerabilities such as benchmark drift, reward hacking, and verifier exploits.
Unifying every benchmark in the Harbor format is what makes this scalable: a single upstream feature hardens all 29 at once. We adopt Harbor's latest such feature, separate verifier sandboxes , which run each task's grader in an isolated sandbox that receives only the agent's declared output artifacts. Since the agent never shares a container with the verifier, it cannot inspect or tamper with the grader to fake a pass, closing a whole class of reward hacks across the suite at once.
Beyond structural hardening, most tasks required specialized repairs.
In AlgoTune we closed verifier exploits that let agents pass without doing the real computation. In GAIA2 we fixed the runtime sidecar and relaxed an over-strict action-ordering judge that failed correct agents on harmless event ordering. In SkillsBench we had the verifier recompute its spreadsheets with LibreOffice, so correct answers stopped failing on stale ground-truth values. In FeatureBench we removed a copy of the reference implementation that shipped inside the agent's own container that leads to reward hacking. Tasks that could not be repaired were dropped, leaving the 82 well-formed, genuinely hard tasks that make up Harbor-Index.
We also standardize how tasks are scored. Following Terminal-Bench convention, rewards are binary: an agent either passes a task or it doesn't, with no partial credit. For tasks with continuous metrics originally (e.g., AlgoTune, GSO, SLDBench), we set a threshold that frontier models found difficult to exceed.
Choosing timeouts
We also tighten each task's timeout in this stage. Timeouts strongly affect results, since a longer budget raises pass rates but slows evaluation and increases cost. We first run each task with a 24-hour timeout on Claude, Gemini, and GPT, then set the limit to 1.2x the fastest model's runtime (rounded up to the nearest 10 minutes), or to 3 hours for tasks every model fails. Tighter limits may lower pass rates, but real users want an agent that solves a task quickly, so speed deserves to be part of the bar. Since our runtimes were measured against the official Claude, Gemini, and GPT APIs, the limits assume provider-level serving speed, and we recommend realistic timeouts in agent evaluation generally.
Findings
We run 1,476 rollouts, one for each agent-model pair on every task. Inside each task's own sandbox, a judge reads the agent trajectory and re-decides the outcome as a genuine solve, an honest failure, a gamed verifier, or an infra issue. We audit the 1,441 rollouts with a usable trajectory this way; the remaining 35, whose trajectory was lost to an infra failure, are conservatively counted as infra false negatives. Everything below is live: click any cited example or dashboard row to open that rollout's full agent and judge trajectory with the judge's cited evidence.
Agents almost never game the verifier
Almost nothing here is cheating. The judge flags just 9 of 1,476 rollouts (0.6%) as gaming the verifier, plus 74 false negatives from infra issues, so about 94% of the verifier's verdicts hold up. This is not a claim that frontier agents rarely reward-hack in general; METR reports the opposite when graders are exploitable . It is a property of this set after hardening: the audit-and-fix loop above was aimed squarely at reward hacking and dropped the tasks whose graders an agent could game, so the residual rate is what remains once the exploitable tasks are gone. What is left is honest failure, so the rest of this section looks at how it happens.
Models often run out of time
Grouping the judge's 17 codes into six families makes each model's failure shape readable, ordered by how close the agent got to a solution: 453 rollouts timed out with no answer, 361 came almost right but missed the pass mark, and 445 landed far off with a wrong answer. The shape splits sharply by model. GPT-5.5 and Gemini time out on only about 11% of tasks, but the open-weight models time out three to four times as often (30 to 43%), and even Opus lands at 33%. The frontier models finish and miss; the open models often never finish.
Ran out of time or token budget while still exploring, training, or debugging, and never wrote a gradable submission.
Example: MiMo V2.5 on algotune-optimize-lti-sim: The agent timed out with a solver that is both numerically unreliable under harness reuse and roughly 1.2x faster than baseline, far below the required 200x speedup.
Why do open models time out so much more? They take no more turns than the frontier, a median of 24 steps against 27 to 31 for GPT-5.5 and Gemini. Each turn just runs longer: even their successful runs need nearly twice the wall-clock time, a median near 15 minutes against 8 to 9. Because those runs already sit close to the budget, more of them cross it.
Capable models are more consistent across harnesses
Running each model on both its native harness and terminus-2 splits its solves into native-only, both, and terminus-only. The both share, how many solves survive the harness swap, falls from 42% for the strongest models to 7% for the weakest, so stronger models hold a more consistent solve set across harnesses while weaker ones are far more at the harness's mercy.
Native harnesses are leaner and more reliable
Across all nine models, each model's own native CLI reaches its result far more economically than the cross-vendor terminus-2 harness. Matched task by task, native makes about a third fewer tool calls and generates over 40% fewer output tokens per rollout, and it runs out the clock far less often, timing out on 26% of runs against terminus-2's 42%. The biggest driver is terminus-2's bash-only action space: without a native CLI's image-reading and web-search tools, models grind through OCR loops and hand-written scrapers that never converge. The leaner run costs nothing in capability: on the open models the two harnesses tie on solves (28 vs 26), and on the frontier, where the native CLIs are more tuned, native pulls ahead (43 vs 37).
terminus-2 also pays a reliability tax that native tool-calling avoids: its protocol wraps every action in escaped JSON, which weaker models routinely botch. An Invalid-JSON rejection hits 7.3% of open-model terminus rollouts (45 events, up to 4 in a single run). The trouble concentrates in the weaker open models, MiniMax, Qwen, and GLM; GPT-5.5 and Opus almost never fumble a call even on the JSON protocol. Each rejection burns a step re-emitting it.
Fixed failure categories no longer fit
To report why agents fail, the standard recipe from Terminal-Bench 2 and MAST is to fix a list of failure types up front, hand-label a small gold set of trials, calibrate an LLM judge until it matches them, then run that judge over the rest at scale. It assumes one category scheme fits every trajectory and that reviewers agree on which category applies.
Both assumptions broke on Harbor-Index, so we switched to free-form agentic judges.
We tried twice, and both attempts failed the same way. Our own taxonomy came from bottom-up clustering: 18 failure modes, hand-annotated across a 200-trial gold set in about 66 reviewer-hours. It collapsed on three fronts. About 30% of tasks, across benchmarks like HLE and SciCode, need domain expertise beyond CS just to read the prompt; the software engineering trajectories sprawl over 63 turns and 113K tokens; and frontier errors are so subtle, a single wrong fact or a hidden-test regression, that reviewers reading different spans reached different conclusions. Only 6 of the 18 modes survived, and the chart they produced said no more than the TB2 chart, just with more categories. A literature taxonomy of 34 cited behaviors from 8 papers hit the same wall: whether a failure is a reasoning defect or a knowledge gap is rarely legible from a trajectory at all.
So we dropped fixed buckets. In their place we run free-form agentic judges: production coding agents that run as Harbor tasks inside each task's own container, holding the trajectory, verifier output, gold solution, and task definition, and able to re-run the verifier on the agent's own submission. Failure modes become grounded hypotheses rather than predetermined labels, and many of the task bugs these judges surfaced we confirmed and fixed. The natural next step, left to future work, is to close the loop: have the judge inject a hint matching its hypothesis, rerun the model, and check whether the failure clears.
To present what the judges find, we follow DeepSWE and sort every trial into true or false positives and negatives. The split separates task failures, a gamed verifier or an overstrict gate, from genuine agent failures, so a low pass rate reflects a model's limit rather than a broken task. We grade the honest misses by how close they came, from almost (nearly right) to far (a wrong answer) to timeout (no answer at all), an ordering that doubles as a difficulty gauge. Every verdict carries the full agent trace and the judge's own trace, which you can browse and filter by model, outcome, or failure family in Explore Harbor-Index below.
Explore Harbor-Index
All 82 tasks and all 1,476 trials are here. Start from the tasks, or switch to the trials view to filter by model, harness, benchmark, outcome, or failure family.
What comes next
Harbor-Index is a step toward more reliable and affordable agent evaluation. Next, we plan to expand the benchmark's scope, release more analysis artifacts, sharpen our broken-task detection tools, study agent hacking behavior in greater detail, and keep the leaderboard current as new agents and models arrive. As the field produces harder tasks, we'll follow with v2, v3, and beyond.
We hope Harbor-Index proves useful not only as a benchmark but as a working example of how to curate an agentic evaluation set: compact, diverse, challenging, and carefully audited.
Acknowledgements
We sincerely appreciate our funding partners for supporting Harbor-Index and providing valuable feedback:
- Frontier AI labs: OpenAI, Anthropic, Google DeepMind, Qwen, DeepSeek, Kimi, Z.ai, MiniMax, Xiaomi MiMo
- Sandbox providers: Daytona, Modal
- Other partners: Laude Institute, 2077AI, UniPat AI
We thank everyone who contributed to building Harbor-Index; the full list is on the Harbor-Index Contributors page.
Special thanks to Ryan Marten, Alex Shaw, and Saul Fuhrmann for their feedback on earlier versions of this blog.
And finally, our deepest thanks to the open benchmark community that makes Harbor-Index possible. If you're interested in the project and becoming a contributor for future series, come join us.
References
[1] Laude Institute. Harbor: A framework for specifying sandboxed agent tasks for evaluation and optimization. https://www.harborframework.com, 2025.
[2] Epoch AI. Epoch Capabilities Index (ECI). https://epoch.ai/benchmarks/eci, 2025.
[3] Y. Zeng and D. Papailiopoulos. You don't need to run every eval. arXiv preprint arXiv:2606.24020, 2026.
[4] M. A. Merrill et al. Terminal-Bench: Benchmarking agents on hard, realistic tasks in command line interfaces. arXiv preprint arXiv:2601.11868, 2026.
[5] K. Buchanan et al. Benchmarks as software: A case study on Terminal-Bench. OpenReview https://openreview.net/forum?id=AhXMZPnOPS, 2026.
[6] Harbor Team. Verify Harbor tasks in a separate sandbox. https://www.harborframework.com/news/separate-verifier-sandboxes, 2026.
[7] S. Von Arx, L. Chan, and B. Barnes. Recent frontier models are reward hacking. METR, https://metr.org/blog/2025-06-05-recent-reward-hacking/, 2025.
[8] M. Cemri, M. Z. Pan, S. Yang, L. A. Agrawal, B. Chopra, R. Tiwari, K. Keutzer, A. Parameswaran, D. Klein, K. Ramchandran, M. Zaharia, J. E. Gonzalez, and I. Stoica. Why do multi-agent LLM systems fail? arXiv preprint arXiv:2503.13657, 2025.
[9] S. Liu et al. An empirical study on failures in automated issue solving. arXiv preprint arXiv:2509.13941, 2025.
[10] Z. Chen, W. Ma, and L. Jiang. Beyond final code: A process-oriented error analysis of software development agents in real-world GitHub scenarios. arXiv preprint arXiv:2503.12374, 2025.
[11] N. Islam et al. When agents fail: A comprehensive study of bugs in LLM agents with automated labeling. arXiv preprint arXiv:2601.15232, 2026.
[12] S. Gandhi, J. Tsay, J. Ganhotra, K. Kate, and Y. Rizk. When agents go astray: Course-correcting SWE agents with PRMs. arXiv preprint arXiv:2509.02360, 2025.
[13] H. Xue et al. PAGENT: Learning to patch software engineering agents. arXiv preprint arXiv:2506.17772, 2025.
[14] D. Deshpande, V. Gangal, H. Mehta, J. Krishnan, A. Kannappan, and R. Qian. TRAIL: Trace reasoning and agentic issue localization. arXiv preprint arXiv:2505.08638, 2025.
[15] J. Xu et al. SWE-Compass: Towards unified evaluation of agentic coding abilities for large language models. arXiv preprint arXiv:2511.05459, 2025.
[16] W. Huang, C. Lee, L. Tng, and S. Ge. DeepSWE: Measuring frontier coding agents on original, long-horizon engineering tasks. https://github.com/datacurve-ai/deep-swe, 2026.
Citation
If you find Harbor-Index useful in your research, please cite:
@misc{harborindex2026,
title = {Introducing Harbor-Index: A Compact, Diverse, Challenging, and High-Quality Benchmark for Agentic Evaluation},
author = {Shi, Lin and Lin, Haowei and Zhu, Zixuan and Zhou, Xiaoyue and Li, Xiang},
year = {2026},
howpublished = {\url{https://harbor-index.org/blog/harbor-index}},
note = {Terminal-Bench blog}
}Written by
Lin Shi, Haowei Lin, Zixuan Zhu, Xiaoyue Zhou, and Xiang Li