NLA Thought Anchors

The following post seeks to look further into why NLA (Natural Language Autoencoders) contains the prediction more often when the original activations led to the correct output than incorrect output.Quick Summary:Extraction position matters - NLA answer appearing in AV increases as the token approaches the model's final answerFirst sentence is the most counterfactually important for both activation reconstruction loss and the AV containing the final outputSentences counterfactually important for...