LoRA and Weight Decay (2023)

LoRA doesn’t approximate a solution to full-finetuning; it solves a different (albeit similar) optimization problem