Debate with Self-Play Best-of-N Optimization

Context: We are the scalable oversight team at Arcadia Alignment. We aim to do rigorous empirical work on debate - bridging the gap from theory to the alignment tasks we care about.Debate is a proposed protocol for scalable oversight. As tasks outrun direct supervision, labs are increasingly likely to train against protocols like it. Our concern is that, for questions which are hard to verify, models will become more compelling more quickly than they will become more accurate – this could underm...