LLMs are below chance on TASTE once a shortcut is removed
We found a shortcut in the TASTE benchmark: mech-interp-related proposals are generally scored lower. Considering only the non-interp half, all tested models with ECI > 158 are below 50% on the rubric-only setup, with 8 of 14 significantly below (7 of them OpenAI). In particular, OpenAI models' accuracies average to only 29.6%, and seem to decrease with higher ECI (Pearson r = -0.74), suggesting a systematic bias.The shortcutThe TASTE benchmark is a benchmark where models decide which of two AI-...
Read full article →