Unsupervised Feature Discovery via Simple Clustering

This article was submitted as part of Neel Nanda's MATS ApplicationAbstract. This article examines whether clustering can discover features. We define a feature as “a direction in activation space associated with a property that is both interpretable to humans and useful to the model”. Clustering is done on activations from Qwen2.5-7B-Instruct (layer 20) via recursive binary k-means clustering: the first level of the tree is obtained via flat k-means with k=64 (2^6), then each leaf is repeatedly...