What can language models teach us about understanding?

Crossposted from my website. Written by me and edited in collaboration with Claude Fable (Anthropic). This post is mainly aimed at mathematicians, as a call to think expansively about what mathematics could look like. I gesture briefly at building a new mathematical theory of 'intuition', doing for that word what the nineteenth and twentieth centuries did for 'reasoning' and 'computation' through the language of mathematical logic, lambda calculus and Turing machines. I predict that such a 'calc...