Bad Guy Tokens

Based on a recent paper, training models to give bad advice in a narrow domain doesn't make the model or "the persona" broadly evil in a robust way. It just makes the particular system prompt[1] tokens you fine-tuned with evoke an evil persona, and perturbing it a little (e.g. changing the date slightly) substantially reduces misaligned behavior.For review, a typical chatbot conversation looks like this under the hood:<|im_start|>systemYou are Qwen, created by Alibaba Cloud. You are a helpful as...