Sanity Checks for Sanity Checks for Sparse Autoencoders

This post will probably not be very informative to readers who don't know what an SAE (Sparse Autoencoder) is.An interpretability researcher once told me, "SAEs can never fail, they can only be failed." But recently, SAEs have been suffering failure after failure, and as a result, the field has grown increasingly pessimistic about SAE work. Korznikov et al.'s Sanity Checks for Sparse Autoencoders: Do SAEs Beat Random Baselines? purports to be an example of such failures. However, I have some dis...