Baiting monitorable reward hacks
This is joint work between Ziqian Zhong, Lawrence Feng, Gaurav Rohit Ghosal, Jiangqi Dai, and Aditi Raghunathan. This work is supported by a BlueDot Rapid Grant.We are unsure of whether this intervention is net positive (see discussions below) despite positive results, but we think similar techniques are worth further studying.Gradient routing can fail to mitigate reward hacks when the models settle on hacks not caught by the monitor. For example, if we swap the monitorable and unmonitorable set...
Read full article →