Tie training can make DPO/RLHF-trained AIs generalize better

·LessWrong··

This post covers our recent ICML paper: Spurious Correlation Learning in Preference Optimization: Mechanisms, Consequences, and Mitigation via Tie Training.TL;DROur theorems and experiments suggest that DPO and RLHF have an unwelcome consequence: they make AIs care about every feature of actions that correlates with true value on the training distribution.[1]That’s true even if the training set contains no misspecified preference data.And it’s true even in the infinite-data limit.So AIs trained ...

Read full article →

Related Articles

Unauthenticated remote code execution in Nginx
speedping · Hacker News · 6h ago
Highly classified US national security communications network goes down
monkeydust · Hacker News · 1d ago
Harvard researchers find AI coding agents shifted the bottleneck to code review
judahmeek · Hacker News · 1d ago
New Evidence Shows School Phone Bans Are Gradually Improving Test Scores
donohoe · Hacker News · 1d ago
Java4Shell: RCE in OpenJDK when processing Base64
werrett · Hacker News · 1d ago