Evaluation Awareness is Probably Distorting Monitor Evaluations
TL;DR: Monitors do worse on monitoring evaluations when they verbalize evaluation awareness in their transcripts. We ran four models as monitors on agent transcripts from two state-of-the-art monitor benchmarks. In over 22K monitor reviews, almost a quarter of the monitors verbalized evaluation awareness. Comparing the different conclusions the monitors reached following their verbalized awareness, we show that monitors will flag fewer harmful actions, and more honest transcripts.IntroductionMon...
Read full article →