LLM 為什麼不用一直重新計算？

In Day 2, we saw that Large Language Models generate text one token at a time through the Prefill and Decode stages. But this creates an important efficiency problem: when the model generates a new token, does it need to recalculate the attention information for every previous token again? To answer this question, we need to go back to the foundation of modern LLMs—the Transformer and its Self-Attention mechanism. By understanding how Query, Key, and Value work, we can see why the Key and Value states of previous tokens can be reused during decoding, which leads us to one of the most important optimization techniques in LLM inference: the KV Cache.

Instead of starting with KV Cache directly, I’d begin with the Transformer architecture, then narrow down into Self-Attention, then explain Q/K/V, and only after that introduce KV Cache. That makes the article easier to follow because readers understand where the “K” and “V” actually come from.

Q = Query means: What am I looking for?
K = Key means: What information does this token contain?
V = Value means: What information should I retrieve?

KV Cache: Save It Instead of Recomputing It, which also means Store the previously calculated Keys and Values and reuse them.

So next, I gonna explain the difference of Without KV Cache vs. With KV Cache.

** Without Key Value Cache: **

** With Key Value Cache:**

Here's the resule

Without KV Cache: 18.72 seconds
With KV Cache: 6.31 seconds

Speedup: 2.97x

Conclusion:
In this article, we started from the Transformer and Self-Attention, then looked at how each token produces Query, Key, and Value representations.

During autoregressive generation, the model needs information from previous tokens at every decoding step. However, the Key and Value states of those previous tokens do not need to be recomputed again and again.
KV Cache reduces redundant computation and makes token-by-token generation much more efficient.

However, this optimization introduces another trade-off:
We save computation by using more GPU memory.As the context becomes longer, or as more users access the model at the same time, the KV Cache can become a major part of GPU memory usage.
This leads to the next question:
How can we manage KV Cache memory more efficiently?

That is exactly where PagedAttention and vLLM come in!

Reference:

1. Vaswani et al., 2017 — Attention Is All You Need
2. Shazeer, 2019 — Fast Transformer Decoding: One Write-Head is All You Need
3. Kwon et al., 2023 — Efficient Memory Management for Large Language Model Serving with PagedAttention