Some Intuition on the Neural Tangent Kernel

Neural tangent kernels are a useful tool for understanding neural network training and implicit regularization in gradient descent. But it's not the easiest concept to wrap your head around. The paper that I found to have been most useful for me to develop an understanding is this one:Lee et al (2019) Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient DescentIn this post I will illustrate the concept of neural tangent kernels through a simple 1D regression example. Ple...