Most KV cache eviction methods work on a simple premise: tokens that received high attention are important; tokens that received low attention can be dropped. TwinKV (arXiv:2608.27128) tests that premise directly using a controlled leave-one-out probe and finds that the correlation between attention magnitude and a token’s causal contribution to the final answer is approximately −0.004.

That’s not a weak correlation. It’s essentially no relationship at all.

What the probe actually measures

The leave-one-out approach is straightforward: for each token in context, you run inference with and without it present, then measure how much the output distribution shifts. That shift is the causal contribution. You then compare it to the attention weight that token received across heads and layers.

The near-zero correlation means the attention signal — which is cheap to read and has been the go-to heuristic for methods like H2O, SnapKV, and StreamingLLM — is not actually tracking what those methods assume it tracks. High-attention tokens can be causally irrelevant; low-attention tokens can be causally critical.

This is a clean empirical result and challenges the foundation of a large chunk of the KV cache compression literature.

The proposed fix

TwinKV doesn’t replace existing eviction policies. It adds a repair pass on top of them.

Given whatever set of retained tokens your current policy chose, TwinKV audits the choice for two specific failure modes:

Orphans: tokens that were evicted, but whose key has no surviving near-duplicate in the retained set. Evicting these loses information that isn’t represented elsewhere.

Redundant donors: tokens that were retained, but whose key is a near-duplicate of another retained token. Keeping both wastes a slot.

TwinKV swaps orphans for redundant donors, holding the eviction budget exactly fixed. No retraining. No change to the eviction policy’s own logic or scoring. Just an audit of the retained-vs-evicted decision under a different signal.

The signal for “near-duplicate key” is cosine similarity between key vectors. The intuition: if a token’s information is already present in another retained token’s key, keeping it doesn’t help; and if an evicted token has no representative in the retained set, dropping it was a mistake.

Results

Testing on LongBench, LooGLE, and RULER shows generally positive but uneven gains. The improvements are stronger on Llama-3.2-1B than on Qwen3-4B, and smaller at higher compression ratios — which makes sense, since at aggressive compression rates even an improved signal hits fundamental limits on how much context can be preserved.

Few-shot classification tasks showed minimal benefit, which is also expected: those tasks don’t rely on long-range causal dependencies across context in the way retrieval and summarization tasks do.

The paper positions TwinKV as a low-cost augmentation rather than a replacement, which is appropriate given the results. The core contribution is the empirical finding — that the field has been optimizing against the wrong signal — more than the specific repair mechanism, which is one reasonable response to that finding.

Whether key-vector cosine similarity is the right redundancy signal is a separate question. Keys can be similar without their values being redundant, and the paper acknowledges this is an approximation. But it’s a better-grounded approximation than attention magnitude, which apparently doesn’t track importance at all.

The code is available in the paper’s supplementary materials. Training-free and composable with whatever eviction baseline you’re already running makes it worth a quick evaluation against your current setup.