Dust: Pretraining Transformers Without Backpropagation

Contents TL;DR 1 Introduction 2 Method 2.1 Activation-Space Perturbation 2.2 Credit Assignment 2.3 Interference and Tuning 3 Pretraining Without a Backward Pass 3.1 Setup 3.2 Main Results 3.3 Dust Under Adam 4 Search in High-Dimensional Space 4.1 Overparameterization 4.2 Emergence of Backprop-Like Gradients 5 Conclusion 6 Related Work References Appendix Q Labs Research Dust: Pretraining Transformers Without Backpropagation Samip Dahal, Bishwas Mandal, Serdar Gülbahar, Akshay Vegesna October 202