Recap from Day 3:

KV Cache avoids recomputing previous K/V states, but every active request needs memory to store them. Nowadays, but it created another. By storing previous Key and Value states, an LLM can avoid repeated computation and generate tokens much faster. But what happens when dozens of users are generating different-length responses at the same time? The KV Cache keeps growing, GPU memory becomes fragmented, and a large amount of space may be wasted.

To solve this problem, vLLM introduced PagedAttention, a memory-management technique inspired by operating-system paging. In this article, we will look at why traditional KV-cache allocation is inefficient, how PagedAttention divides the cache into blocks, and how this design improves memory utilization and LLM serving throughput.

    Why KV Cache memory becomes a problem-The paper shows that existing systems waste KV-cache memory through reserved space, internal fragmentation, and external fragmentation. In their profiling, only about 20.4%–38.2% of KV-cache memory in the compared existing systems stored actual token states.

What PagedAttention is. This is the most important graph for Day 4. It shows that K/V vectors can be divided into blocks and stored in non-contiguous physical memory.

How logical blocks map to physical blocks. This is where you explain the block table and how vLLM allocates new blocks only when needed.

Why this improves serving efficiency — connect back to Figure 1 / Figure 2, explaining that better KV-cache utilization allows more requests to fit in a batch and therefore improves throughput. The paper reports roughly 2–4× higher throughput at comparable latency in its evaluated workloads.

The key idea of PagedAttention comes from virtual memory and paging in operating systems.

Traditional LLM serving systems often store the KV Cache of each request in contiguous GPU memory. However, the final output length of an LLM request is unknown in advance, so systems may reserve more memory than the request eventually uses. This can cause both internal and external fragmentation.PagedAttention takes a different approach.Instead of treating the KV Cache as one large continuous region, it divides the cache into fixed-size KV blocks.

Conclusion:
In this article, we looked at why KV Cache memory management becomes a serious challenge in LLM serving.

Traditional approaches may waste GPU memory because sequence lengths are dynamic and KV Cache can suffer from over-reservation and fragmentation. PagedAttention addresses this by borrowing the idea of paging from operating systems.

Instead of storing the KV Cache for each request in one large continuous memory region, PagedAttention divides it into fixed-size blocks:

Logical KV Blocks -> Block Table -> Physical KV Blocks

This allows a sequence to remain logically continuous even when its KV Cache is stored in non-contiguous regions of GPU memory.

The result is more efficient memory usage, less fragmentation, and the ability to keep more concurrent requests in memory at the same time. In the original vLLM paper, this improved memory efficiency translated into significantly higher serving throughput under the evaluated workloads.

KV Cache reduces computation, while PagedAttention makes that cache easier to manage efficiently.

That gives us the full progression:

Self-Attention -> KV Cache -> Memory pressure -> PagedAttention -> efficient LLM serving

And now that we understand the core idea behind PagedAttention, the next step is to see how it is actually used in practice.

References:

1. Kwon, W. et al. (2023). Efficient Memory Management for Large Language Model Serving with PagedAttention. Proceedings of SOSP '23, pp. 611–626.
2. Orca: A Distributed Serving System for Transformer-Based Generative Models
3. Efficiently Scaling Transformer Inference