Practical Learnings from Synthetic Document Finetuning

We've been using Synthetic Document Finetuning (SDF) quite a bit at Apollo Research lately. This post covers a few tweaks to the standard SDF recipe specific to our use cases, plus some general tips and tricks for getting good results. We’re sharing these notes in case they’re useful to others doing research with SDF.1. What Is SDF?Synthetic Document Finetuning (SDF) is a knowledge editing technique where models are finetuned on LLM-generated documents consistent with a target fact or belief. As...