An Attempt at Top-down Interpretability

There are at least two distinct ways to conceptually organize a scientific endeavor: top-down or bottom-up. The bottom-up approach involves finding the smallest objects we can confidently understand, studying them, and leveraging that knowledge to study more complex objects made up of those smaller pieces. As a highly simplified example, in mech-interp this might mean starting by first understanding single neurons/SAE directions/features, leveraging this understanding to study circuits/manifolds...