讓 Local LLM 真正讀懂自己的文件

In Day 5, we successfully served Qwen through vLLM and connected it to a simple RAG pipeline. But building the pipeline is only the first step—the more important question is whether retrieval actually changes the model’s behavior. In this article, we will build a small RAG system from scratch using Sentence-Transformers, FAISS, vLLM, and Qwen2.5-7B-Instruct, then compare the model’s answer with and without retrieval. The goal is to see, in practice, how external documents can turn a general-purpose LLM into a system that answers based on our own knowledge.

Create these addtional documents for RAG to search:

docs/
├── vllm.txt
├── kv_cache.txt
└── paged_attention.txt

docs/paged_attention.txt

PagedAttention is a memory management technique introduced by vLLM.

Instead of storing the KV cache for each sequence in one contiguous
region of GPU memory, PagedAttention divides the KV cache into blocks.

A block table maps logical KV blocks to physical KV blocks. This reduces
memory fragmentation and allows vLLM to use GPU memory more efficiently.

docs/kv_cache.txt

KV Cache stores the Key and Value tensors generated by previous tokens.

During autoregressive decoding, these cached values can be reused instead
of being recomputed for every new token.

KV Cache reduces computation but consumes GPU memory as the sequence grows.

we called this instruction to set up the vllm server:

vllm serve Qwen/Qwen2.5-7B-Instruct \
    --host 127.0.0.1 \
    --port 8000 \
    --max-model-len 8192 \
    --gpu-memory-utilization 0.75

then we run on this python file on another terminal:

python rag_demo.py

after runing these two processes, we can started aski question

Ask a question:
How does PagedAttention manage KV Cache?

Your WITHOUT RAG answer sounds confident, but several parts are inaccurate. In particular, it says PagedAttention only keeps the current and previous pages in memory, which is not how vLLM's PagedAttention works. PagedAttention divides KV cache into blocks and maps logical blocks to non-contiguous physical blocks; it does not simply discard everything except nearby pages. The mention of a “backward pass” is also misplaced here because your vLLM serving scenario is inference, not training.

 Top-1:
    paged_attention.txt
    Similarity = 0.7288

    Top-2:
    kv_cache.txt
    Similarity = 0.5901

That ranking makes sense. Your question was about PagedAttention, so the PagedAttention document scored highest, while the general KV-cache document was still relevant but less similar.

Then the WITH RAG answer is much stronger:

PagedAttention manages the KV Cache by dividing it into blocks and using a block table to map logical KV blocks to physical KV blocks. This approach reduces memory fragmentation and allows vLLM to use GPU memory more efficiently.

Without RAG, the model knows something. With RAG, the model knows what our documents say.

Conclustion:

The comparison shows the practical value of RAG. Without retrieval, Qwen2.5-7B-Instruct already had some knowledge of PagedAttention, but it generated additional details that were not accurate, such as keeping only the current and previous pages of the KV cache and discussing the backward pass. After adding RAG, FAISS retrieved the relevant paged_attention.txt and kv_cache.txt chunks, and the model produced a much more focused answer based directly on the provided context.