Projection sampling preserves capabilities but diminishes gains from alignment fine-tuning

TLDRFine-tuning models on alignment reasoning from stronger models can improve safety, but at a substantial cost to general capabilities. Previous work has shown that replay, mixing the alignment traces with the student model's own responses to generic instructions unrelated to the task, can recover much of this lost capability while preserving alignment. We investigate whether projection sampling, which rewrites teacher traces to better match the student model's output distribution, can offer f...