MSE loss does not generate superposition

If you're training any type of toy model of superposition, Mean Squared Error (MSE) loss is unusually bad. [1] Related work We aren't the first to notice that MSE loss doesn't work. In Toy Models of Superposition the effective loss function is mjx-container[jax="CHTML"] { line-height: 0; } mjx-container [space="1"] { margin-left: .111em; } mjx-container [space="2"] { margin-left: .167em; } mjx-container [space="3"] { margin-left: .222em; } mjx-container [space="4"] { margin-left: .278em; } mjx-c...