Challenge: Hand coding weights for efficient sequence memorisation

We hand coded weights for one layer MLPs that memorises labels for input token sequences of length two. The number of facts our hand-coded models can memorise with 90% accuracy[1]scales roughly linearly with the models' parameter count[2], just like trained models for the same architecture. However, our hand-coded models' scaling prefactor still falls short of trained models' by a factor of mjx-math { display: inline-block; text-align: left; line-height: 0; text-indent: 0; font-style: normal; fo...