Theories of Deep Learning

This field has been blessed with exponential empirical success in the form of architectures and algorithms that simply worked through scaling, while the theory lagged behind[1]. Although in the past few years, the “gap“ seems to be diminishing, and we are getting multiple theories for different aspects of deep learning. This essay would be a simple high-level overview of all the theories I’ve come across.NOTE: These are mathematically dense frameworks which either provide a language for formaliz...