If you're interpreting <1B parameter models, you should use a tensor transformer

To all my fellow researchers doing SLT, computational mechanics, one of ARC's programs, natural abstractions/condensation, proofs on NNs (or any interp on small models), this is for you. Tensor transformers (ie replacing your MLPs & attention with bilinear variants) are performant[1] and allow you to deploy the full power of linear algebra. In fact, our recent paper used generalized cosine similarity on the full tensor transformer. And yes, I mean cos-sim defined on the eg 9th order tensor,[2] n...