Evaluation Awareness is Probably Distorting Monitor Evaluations

·LessWrong··

TL;DR: Monitors do worse on monitoring evaluations when they verbalize evaluation awareness in their transcripts. We ran four models as monitors on agent transcripts from two state-of-the-art monitor benchmarks. In over 22K monitor reviews, almost a quarter of the monitors verbalized evaluation awareness. Comparing the different conclusions the monitors reached following their verbalized awareness, we show that monitors will flag fewer harmful actions, and more honest transcripts.IntroductionMon...

Read full article →

Related Articles

Unauthenticated remote code execution in Nginx
speedping · Hacker News · 1d ago
Cooling buildings is overtaking heating them as an energy need, IEA says
robtherobber · Hacker News · 8h ago
Ruby2ßhell: Pre-authentication RCE on stock Rails
jdns · Hacker News · 18h ago
Pancreatic cancer halted by virus injection in three patients
cybermango · Hacker News · 5h ago
Highly classified US national security communications network goes down
monkeydust · Hacker News · 2d ago