Fine-Tuning Is Mostly Deletion
A NeurIPS 2026 paper by Lu et al. shows that Bit-Mask Tuning—zeroing fewer than 0.05% of a model's gate-projection weights with no value updates—matches full fine-tuning across GRPO, SFT, and on-policy distillation. The finding suggests fine-tuning often suppresses pre-existing behaviors rather than building new ones.
Read more →
