Side-Effects of Length Penalty in RL

·LessWrong··

TL;DR(This is a write-up of results obtained by Luc Feron and me from the sprint period of doing Neel Nanda’s MATS Stream Feb 23rd - Mar 6th).Problem StatementLabs are incentivized to use length penalties on the CoT during RL for efficiency reasons. A natural worry is that this causes negative side effects, especially worse monitorability as the model is incentivised to omit information.Contrary to existing work, we find that faithfulness in the MMLU-with-hint eval increases. We find various oth...

Read full article →

Related Articles

Unauthenticated remote code execution in Nginx
speedping · Hacker News · 6h ago
Highly classified US national security communications network goes down
monkeydust · Hacker News · 1d ago
Harvard researchers find AI coding agents shifted the bottleneck to code review
judahmeek · Hacker News · 1d ago
New Evidence Shows School Phone Bans Are Gradually Improving Test Scores
donohoe · Hacker News · 1d ago
Java4Shell: RCE in OpenJDK when processing Base64
werrett · Hacker News · 1d ago