Training with conflicting values can induce CoT override

CoT override: when a model makes a decision in its CoT but ignores it in its responseTLDRWe train models on two conflicting traits: (1) caring about the user’s health, and (2) promoting smokingThose models do not generalize to a stable persona, instead they have a split brain: sometimes responding as one persona or the otherThose models exhibit CoT override where they will have a health-aligned CoT but still answer in the smoking personaCoT override exists in frontier models. Prompts about CCP-s...