You don't need error nodes, you need better features

This is a cross-post from my blog. It is a follow-up to the methods I developed in a previous post on replacement-aware training. Summary A replacement model (Ameisen et al. 2025) is a modification to an LLM in which some subset of internal activations are replaced with ones computed as a sum of more interpretable features, such as those found by sparse auto-encoders (SAEs) (Cunningham et al. 2023) or cross-layer transcoders (CLTs) (Lindsey et al. 2024). When multiple components are thus re-enco...