Terminal-Bench Leaderboard Rankings: Luck or Skill?

·LessWrong··

Terminal-Bench 2.0, one of the best agent benchmarks, runs agents on 89 tasks with at least 5 tries per task. The aggregate rankings are then published along with a confidence interval. What I wanted to know, however, was whether or not the point gap between any two given agents represented a statistically significant difference (or if it was just noise).My first finding was that if you look at pairs with adjacent rankings then 24/25 of them differ by such a small amount that it's essentially st...

Read full article →

Related Articles

Unauthenticated remote code execution in Nginx
speedping · Hacker News · 6h ago
Highly classified US national security communications network goes down
monkeydust · Hacker News · 1d ago
Harvard researchers find AI coding agents shifted the bottleneck to code review
judahmeek · Hacker News · 1d ago
New Evidence Shows School Phone Bans Are Gradually Improving Test Scores
donohoe · Hacker News · 1d ago
Java4Shell: RCE in OpenJDK when processing Base64
werrett · Hacker News · 1d ago