Compressed Computation under L⁴ Loss is likely Computation in Superposition

SummaryNeural networks are widely assumed to use superposition to represent more features than they have dimensions. A stronger claim is that they also compute in superposition (CiS), i.e., implement more nonlinear functions than they have neurons (Hänni et al. 2024). CiS remains poorly understood, and until recently there were no examples of it arising through training rather than being hand-designed. Toy models of CiS are desirable for two reasons: they're small enough that we can hope to full...