Fine-Tuning Is Mostly Deletion

A NeurIPS 2026 paper by Lu et al. shows that Bit-Mask Tuning—zeroing fewer than 0.05% of a model's gate-projection weights with no value updates—matches full fine-tuning across GRPO, SFT, and on-policy distillation. The finding suggests fine-tuning often suppresses pre-existing behaviors rather than building new ones.

Read more →