←
Harbor Framework202 posts
Harbor Framework avatar

Harbor Framework

@harborframework
◷ Joined January 2026▧ 12 Photos and videos
Kobe avatar
⟳ Harbor Framework retweeted
Kobe@kobe0938·16h

terminal bench featured in Microsoft Ignite keynote by @satyanadella

Post attachment

At GitHub Universe in October, I talked about multi-model solutions like Copilot Fusion will start dominating benchmarks going forward. But the pace of progress has been faster - ~10% better quality at half cost just few weeks later.

Post attachment
View quoted post ↗
Harbor Framework avatar
Harbor Framework@harborframework·Nov 17

v0.26

Kobe@kobe0938Nov 16

update to v0.26 and try 𝚑𝚊𝚛𝚋𝚘𝚛 𝚜𝚎𝚛𝚟𝚎 along with other amazing features. → Harbor Serve → Error policies → Rollout viewer → Agent & sandbox discovery → New integrations → Pre-installed agents → Sandbox reliability → Trajectories & usage tracking

Post attachment
View quoted post ↗
Kobe avatar
⟳ Harbor Framework retweeted
Kobe@kobe0938·Nov 17

exactly, and that’s what harbor is for. harbor❤️vercel: vercel.com/blog/agent-evals

The old way: users are human, you run user interviews, get inspired, mock up user journeys. The new way: users are agents, you run evals, evals inform the software engineering loop. Good read on how we do evals at Vercel.

View quoted post ↗
Kobe avatar
⟳ Harbor Framework retweeted
Kobe@kobe0938·Nov 14

a TerminalBench task > a nature paper

Given that there are only 66 problems in TerminalBench v4.1 and literally every frontier lab tracks TerminalBench performance, at some point it will be more prestigious and impactful to have authored a TerminalBench problem than to get a paper accepted in Nature or Science

View quoted post ↗
Kobe avatar
⟳ Harbor Framework retweeted
Kobe@kobe0938·Nov 15

7/9 top rated benchmarks run natively on harbor. harbor is the best.

I asked 56 people in AI which benchmarks they'd trust most to indicate if an LLM is frontier: 25 votes: Terminal-Bench 4 (@alexgshaw, @andykonwinski, @Mike_A_Merrill) 20 votes: Terminal-Bench Science (@StevenDillmann, @lschmidt3, @sanmikoyejo) 13 votes: FrontierCode (@penlume, @ybenpan, @denizbirlikci, @silasalberti) 10 votes: FrontierSWE (@evan_j_chu, @_rajanagarwal, @MatternJustus, @ardnoKhtnayhsiR, @sanket__mhatre, @akshitkr) 8 votes: Humanity's Last Exam (@longphann_, @hendrycks, Alice Gatti, @ziwen_h, @natliml) 7 votes: Vending-Bench (@axelbacklund, @lukaspet) 5 votes each: - AutomationBench (@danielwshepard, @SalimansRobin, @TweetAnnaMarie) - DeepSWE (@winkey_h, @charleyslee, @LeonardMainnet, @serenaa_ge) - PostTrainBench (@full__rank, @hrdkbhatnagar, @AmyPrb, @shiraeis, @karinanguyen, @MatthiasBethge, @maksym_andr) The most common comment was that people don't really trust benchmarks at all. 56 people voted. People couldn't pick a bench their company worked on, an index, or a human-voted board. Disclosure: on this list, I'm a small investor in Abundant, Cognition, Emulated, Fleet, Hillclimb, Latch, and Snorkel. I didn't vote. Counts are net: a few benchmarks received negative votes. Continued:

Post attachment
View quoted post ↗
Harbor Framework avatar
Harbor Framework@harborframework·Nov 15

𝚑𝚊𝚛𝚋𝚘𝚛 𝚛𝚞𝚗 -𝚍 𝚟𝚊𝚕𝚎𝚗𝚕𝚊𝚋𝚜/𝚏𝚛𝚘𝚗𝚝𝚒𝚎𝚛𝚋𝚒𝚘𝚖𝚕

Introducing FrontierBioML: 25 long-horizon machine learning tasks in computational biology with 11 models, up to 6 hours per run. Agents submit a prediction script; we run it on a held-out test set. Claude Mythos 5.2 leads (40.3), then Gemini 4 Carbon (39.0) and Gemini 4 Argon (36.8)

0:09
View quoted post ↗
Alex Shaw avatar
⟳ Harbor Framework retweeted
Alex Shaw@alexgshaw·Nov 13

Introducing 𝚑𝚊𝚛𝚋𝚘𝚛 𝚜𝚎𝚛𝚟𝚎! Rollout servers are APIs / MCPs that you (or your agent) to Launch jobs Monitor rollouts Query results Pause, cancel, retry, or resume jobs and rollouts They're the missing control plane to Harbor workloads.

Post attachment
Kobe avatar
⟳ Harbor Framework retweeted
Kobe@kobe0938·Nov 13

Introducing 𝚑𝚊𝚛𝚋𝚘𝚛 𝚜𝚎𝚛𝚟𝚎 which turns harbor into a server lets you submit jobs, query/pause/resume/cancel jobs and rollouts, and retry individual rollouts. It's a huge unlock for RL and one of the most elegant and powerful features Alex and I built in harbor. four ways to connect: 1. python sdk: Harbor() / AsyncHarbor() 2. cli: harbor run --server http://localhost:8390 3. mcp: /mcp 4. http: POST /jobs/submit so much more control and observability unlocked: docs.harborframework.com/har… docs.harborframework.com/api…

Post attachment

Introducing 𝚑𝚊𝚛𝚋𝚘𝚛 𝚜𝚎𝚛𝚟𝚎! Rollout servers are APIs / MCPs that you (or your agent) to Launch jobs Monitor rollouts Query results Pause, cancel, retry, or resume jobs and rollouts They're the missing control plane to Harbor workloads.

Post attachment
View quoted post ↗
Kobe avatar
⟳ Harbor Framework retweeted
Kobe@kobe0938·Nov 13

Software Engineering became Machine Learning. presented by our very own Alex Shaw.

Post attachment
Harbor Framework avatar
Harbor Framework@harborframework·Nov 12

OWE-Bench is a great addition to coding benchmarks. Run it with Harbor.

Mercor@mercorNov 12

Agents can write code. Can they finish real engineering work? Introducing OWE-Bench, a new frontier coding benchmark: 26 long-horizon engineering projects designed by Mercor experts to test AI agents on engineering work, from GPU kernels to aircraft design. Built by @jake_cole1 and team with Mercor experts, this benchmark tests AI models on their ability to complete a full project end-to-end. OWE-bench top 3 leaderboard: 1. Fable 5.2 (54.6%) 2. GPT-6 Astra (44.2%) 3. Claude Opus 5.5 at 41.6%. Explore the benchmark website: openworldengineer.mercor.com…

Post attachment
View quoted post ↗
Harbor Framework avatar
Harbor Framework@harborframework·Nov 11

Nice continuous benchmarking by Biopharma Bench! Latest version update is on the Harbor Hub: hub.harborframework.com/data…

Biopharma Bench V0.2 is a huge upgrade from the previous version, mitigating eval awareness with the @harborframework team and working with experts at Lonza, Catalent, Takeda and Sanofi to audit the rubrics and expert judgment. Check it out!

View quoted post ↗
Harbor Framework avatar
Harbor Framework@harborframework·Nov 11

SWE-rebench harbor training tasks! hub.harborframework.com/data…

🚀 SWE-rebench V2.1 RL is out: 13,337 free RL environments for coding agents (20 languages, Harbor format)! 🔧 Fixed tasks, envs, and graders → fast, reliable rollouts. Doing RL post-training or research? DM or reply. We can share data and RL envs early for free, so we can make them better for open source! 🙏 Please RT or share with anyone who might need free RL envs!

View quoted post ↗