Fail safe(r) at alignment by channeling reward-hacking into a "spillway" motivation

·Redwood Research··

It’s plausible that flawed RL processes will select for misaligned AI motivations.1 Some misaligned motivations are much more dangerous than others. So, developers should plausibly aim to control which kind of misaligned motivations emerge in this case. In particular, we tentatively propose that developers should try to make the most likely generalization of reward hacking a bespoke bundle of benign reward-seeking traits, called a spillway motivation. We call this process spillway design.We thin...

Read full article →

Related Articles

Baiting monitorable reward hacks
Ziqian Zhong · LessWrong · 40m ago
MIT's New Method Flags AI Models Trained on CASM Without Generating It
sdoering · Hacker News · 4mo ago
Harm Laundering in GPT Models: Gender Discrimination Transformed Rather Than
sbulaev · Hacker News · 2mo ago
Q3.5 Timelines Update: Looks Like the Intelligence Explosion Will Happen in 2027 After All
Thomas Larsen · Alignment Forum · 8d ago
Self-Modeling Interventions Modulate Emergent Misalignment
gmays · Hacker News · 1mo ago