Dust: Pretraining Transformers Without Backpropagation

·Hacker News··

Contents TL;DR 1 Introduction 2 Method 2.1 Activation-Space Perturbation 2.2 Credit Assignment 2.3 Interference and Tuning 3 Pretraining Without a Backward Pass 3.1 Setup 3.2 Main Results 3.3 Dust Under Adam 4 Search in High-Dimensional Space 4.1 Overparameterization 4.2 Emergence of Backprop-Like Gradients 5 Conclusion 6 Related Work References Appendix Q Labs Research Dust: Pretraining Transformers Without Backpropagation Samip Dahal, Bishwas Mandal, Serdar Gülbahar, Akshay Vegesna October 202

Read full article →

Related Articles

Why there are still no non-WebKit browsers on iOS
lapcat · Hacker News · 9h ago
OpenSSL: Buffer overflow in EVP_EncodeBlock() (CVE-2026-121481, CVSS 9.8)
JacobHenner · Hacker News · 22h ago
Unauthenticated remote code execution in Nginx
speedping · Hacker News · 2d ago
GIMP now has UI tests
qtqtqt · Hacker News · 20h ago
Pancreatic cancer halted by virus injection in three patients
cybermango · Hacker News · 1d ago