Side-Effects of Length Penalty in RL

TL;DR(This is a write-up of results obtained by Luc Feron and me from the sprint period of doing Neel Nanda’s MATS Stream Feb 23rd - Mar 6th).Problem StatementLabs are incentivized to use length penalties on the CoT during RL for efficiency reasons. A natural worry is that this causes negative side effects, especially worse monitorability as the model is incentivised to omit information.Contrary to existing work, we find that faithfulness in the MMLU-with-hint eval increases. We find various oth...