Bf16 might be freezing your weight updates
If you fine-tune a model with weights stored in bf16 and let torch.optim.AdamW write to them directly, you get interesting behaviour. bf16 can only represent…
Read full article →If you fine-tune a model with weights stored in bf16 and let torch.optim.AdamW write to them directly, you get interesting behaviour. bf16 can only represent…
Read full article →