Training On Interpretability Probes Is Bad In Proportion To How Contingent The Features They Rely On Are

People spend a lot of words playing tug of war over whether or not it's reasonable to train against interpretability methods. The anti case goes something like "training based on interpreted features trains against Interpretability itself more than it trains against whatever features you're detecting". There are cases where we should expect this to be true and cases where we should expect this to be not true. It basically comes down to how much the model can encrypt/obfuscate the relevant featur...