Hebbian Learning through the lens of SAE traning.

·LessWrong··

Likely, modern competitive Hebbian learning rules train Sparse Autoencoders implicitly - and their approximations make them worse at it. These algorithms specify weight updates directly, without computing a loss and the consequent gradients. But their updates contain terms closely related to both. This raises a question: how much of their feature-learning behavior can we understand as approximate SAE optimization ?This post develops that connection for tied-weight SAEs, identifies where the upda...

Read full article →

Related Articles

Unauthenticated remote code execution in Nginx
speedping · Hacker News · 6h ago
Highly classified US national security communications network goes down
monkeydust · Hacker News · 1d ago
Harvard researchers find AI coding agents shifted the bottleneck to code review
judahmeek · Hacker News · 1d ago
New Evidence Shows School Phone Bans Are Gradually Improving Test Scores
donohoe · Hacker News · 1d ago
Java4Shell: RCE in OpenJDK when processing Base64
werrett · Hacker News · 1d ago