J-Lens: A Failed Replication on GPT-2

TLDR: We tested Anthropic's new(ish) J-Lens against the classic Logit Lens on GPT-2 small and medium. J-Lens loses at every layer (0/11 on small, 1/23 on medium). We ran 5 stress tests (more data, frequency checks, sparsity, tuning, and scaling) and it still loses every time. We draw two sharp methodological lessons for those doing probe-based interpretability.Repo: https://github.com/nelithb/mech-interp-lab/1: ResultJ-Lens does not work on GPT-2 small or medium.Across 11 layers on GPT-2 small, ...