Inoculate or Reflect? Two training interventions under prompting, steering, and patching

Anthropic's recent paper, Verbalizable Representations Form a Global Workspace in Language Models, contains a small experiment near the end that we found more interesting than the main findings.The technique is called Counterfactual Reflection Training (CRT). The model is fed a partial transcript in its context window, followed by an interruption with a question about what matters in that situation, and is trained only on its answer to that question. It is never trained on a corrected action in ...