We Should Study the Analogy Between Inoculation Prompting Non-Robustness, Negation Neglect, and Backdoor Non-Robustness

TL;DRNegation neglect is a recently discovered phenomenon where training on "the following is false: <claim>" makes the model believe that <claim> is true.Inoculation prompting is a method of reducing reward hacking (and the emergent misalignment that can cause) in models trained with RL. Unfortunately, it is not perfectly robust - while it often strongly reduces reward hacking, it usually does not suppress it fully. Since Anthropic uses it in production, making it more robust will likely make C...