Datasets Determine the Difficulty of Interpretability
Recently Goodfire has come out swinging with two posts indicating that neurons in transformers are "more interpretable than previously assumed", or simply "already clean". In the "Neurons as Interpretable Basis" post they offer three reasons for this claim:First, neurons are already functionally sparse - pruning 92 % of the weakest MoE neuron activations only increases the loss by 2.9%Second, the role of a neuron can be split into different "computational functions". Effectively, we can name the...
Read full article →