LLMs are below chance on TASTE once a shortcut is removed

·LessWrong··

We found a shortcut in the TASTE benchmark: mech-interp-related proposals are generally scored lower. Considering only the non-interp half, all tested models with ECI > 158 are below 50% on the rubric-only setup, with 8 of 14 significantly below (7 of them OpenAI). In particular, OpenAI models' accuracies average to only 29.6%, and seem to decrease with higher ECI (Pearson r = -0.74), suggesting a systematic bias.The shortcutThe TASTE benchmark is a benchmark where models decide which of two AI-...

Read full article →

Related Articles

Unauthenticated remote code execution in Nginx
speedping · Hacker News · 6h ago
Highly classified US national security communications network goes down
monkeydust · Hacker News · 1d ago
Harvard researchers find AI coding agents shifted the bottleneck to code review
judahmeek · Hacker News · 1d ago
New Evidence Shows School Phone Bans Are Gradually Improving Test Scores
donohoe · Hacker News · 1d ago
Java4Shell: RCE in OpenJDK when processing Base64
werrett · Hacker News · 1d ago