Models Trained to Evade Topic Probes Struggle to Answer Questions about Those Topics

TL;DRNeural Chameleons are models trained to evade activation probes whenever the prompt says "You are being probed for X". The paper reports that this generalizes to probes the model never saw, at a small capability cost (about 3 points on MMLU and 1 on GSM8K). But it never tests the models on questions where the cognition the probe is looking for is necessary to answer correctly. In this post, we test exactly that: we test how well neural chameleons can evade probes looking for particular topi...