Sparse Efficiency vs. Superposition: The Interpretability Tradeoff

Today’s frontier models train in an expensive style: dense forward passes, huge matrix multiplies, and broad weight updates.The human brain (~5 MWh over 28 years) is an existence proof that learning can be vastly more energy efficient - about 10,000x - than modern AI training runs (https://coefficientgiving.org/research/how-much-computational-power-does-it-take-to-match-the-human-brain/).The human brain does not achieve this by activating everything all at once. Normal cognition is extremely spa...