What Rewarding Misbehavior During RL Does to Character-Trained Models
TL;DRResearch done as part of MATS. See here for the full paper.Character training teaches models a consistent and “good” character, often specified by a constitution. Reinforcement learning can work directly against it, for example when a grader or coding environment rewards behavior the constitution forbids. We character-train gpt-oss-20b on a semi-realistic constitution. We then apply RL that rewards breaking its rules, either by answering harmful requests rewritten as math problems or by rew...
Read full article →