Tracing Subliminal Learning Through Pretraining

This post summarizes our paper, Tracing Subliminal Learning Through Pretraining, which was developed as part of EleutherAI’s SOAR program. TL;DRWe study subliminal learning, the phenomenon in which a student model acquires a teacher’s preference through training data that appear semantically unrelated to that preference. Our work investigates subliminal learning through cross-model distillation, using the PolyPythia and DataDecide OLMo models to show that a teacher’s animal preference transfers ...