VPD's Evaluation Adversary Finds Systematic Defects in the Released Decomposition

TL;DR: adVersarial Parameter Decomposition (VPD) decomposes model weights into simple components and assigns labels that specify whether a component is needed for a given input. A core aspiration of VPD is robust mechanistic faithfulness - the property that switching off any set of unneeded components for a given input leaves the model's output approximately unchanged. The authors use an evaluation adversary to measure the degree of robustness, and report that the decomposition is not at all rob...