Thoughts on 'Learning Mechanics'

Last month, I stumbled upon an optimistic-sounding paper: "There Will Be a Scientific Theory of Deep Learning". In this paper, we make the case that a scientific theory of deep learning is emerging. By this we mean a theory which characterizes important properties and statistics of the training process, hidden representations, final weights, and performance of neural networks. Historically, machine learning has been a field where theoretical understanding has lagged far behind empirical success....