A paper appearing in EMNLP 2026 Findings documents something practitioners have probably suspected but lacked formal evidence for: training language models with differential privacy makes them hallucinate more, and the effect gets worse as you tighten the privacy budget.
The mechanism the authors identify is straightforward once stated. Differential privacy works by adding calibrated noise to gradients during training, which has the effect of compressing the output distribution — flattening the distinction between likely and unlikely tokens. That compression is useful from a privacy standpoint (it’s harder to extract training data from a flatter distribution) but it also shifts probability mass toward factually incorrect responses that the model would, without the noise, have assigned lower probability.
What makes this more than a theoretical curiosity is the domain specificity. The effect is sharpest on low-frequency facts — precisely the kind of information that tends to matter most when accuracy is the point. High-frequency facts appear in training data often enough that the gradient signal survives even significant DP noise; obscure but correct facts don’t. The result is a model that handles common knowledge about as well as its non-private counterpart but is noticeably less reliable on the specific, unusual, domain-particular facts that users are most likely to ask about when they care deeply about getting the right answer.
The practical awkwardness is obvious. Healthcare, legal, and financial applications are exactly the contexts where both privacy and factual accuracy matter most. A hospital system that fine-tunes a clinical LLM with patient records under differential privacy is doing so to protect those patients, but the paper suggests that protection comes at a cost to the model’s reliability on the precise medical details where errors are most consequential.
The paper doesn’t fully solve the problem — it characterizes it, which is the necessary first step. The authors show that data frequency is the key moderating variable: training data that contains more frequent mentions of specific facts reduces the hallucination penalty from DP, suggesting that synthetic data augmentation or retrieval-augmented approaches might partially recover accuracy without surrendering privacy guarantees. But “partially recover” is doing real work in that sentence.
The deeper question this raises is about evaluation methodology. Most benchmark-based evaluations of differentially private models focus on aggregate task accuracy, where the hallucination penalty on low-frequency facts is diluted by strong performance on common-knowledge questions. The paper’s finding suggests that standard accuracy metrics may systematically underestimate hallucination risk in DP models when deployed in the specialized contexts where they’re actually used. A model that scores 89% on a general-knowledge benchmark and 89% on a private-training variant might behave quite differently when the user is asking about medication dosages or legal precedents from specific jurisdictions.
This result was submitted to arXiv at the end of August and accepted to EMNLP — it’s peer-reviewed, not just a preprint. For anyone building privacy-preserving LLM pipelines, it’s worth reading before finalizing architecture decisions.