while using a cloud API is convenient, it also made me curious about another question: what happens if we want to run and control the LLM ourselves? This question is where my 30-day journey begins. Starting from self-hosted LLM inference, I will gradually explore vLLM, RAG, recommendation systems, LangChain agents, and Codex to understand how we can move from simply using AI to actually engineering an AI system.

Install vLLM and Run Our First Local LLM

After discussing why we may want to self-host an LLM, let's actually run one.
For this experiment, I will use:

1. Model: Qwen/Qwen2.5-1.5B-Instruct
2. Inference Engine: vLLM
3. Operating System: Linux / Ubuntu

conda create -n vllm python=3.12 -y
pip install vllm

and using vllm --version to check the vllm is already installed or not.

sever: vllm serve Qwen/Qwen2.5-1.5B-Instruct
--host 127.0.0.1
--port 8000

clicent: curl http://localhost:8000/v1/chat/completions
-H "Content-Type: application/json"
-d '{
"model": "Qwen/Qwen2.5-1.5B-Instruct",
"messages": [
{
"role": "user",
"content": "Why would a company consider self-hosting an LLM?"
}
],
"max_tokens": 150
}'

it also can be a python file: python test_vllm.py to run it.

Coonlusion:
we looked inside the LLM inference process and saw that text generation is not performed all at once. Instead, an autoregressive language model generates the response one token at a time.

The inference process can be divided into two major stages:

Prompt
↓
Prefill
↓
KV Cache
↓
Decode
↓
Token → Token → Token →

During Prefill, the model processes the entire input prompt and computes the initial Key and Value tensors for the attention mechanism. During Decode, the model generates new tokens one by one while reusing the previously computed information stored in the KV Cache.This reuse is important because without KV Cache, the model would repeatedly recompute the attention states of all previous tokens every time it generates a new token.

However, KV Cache also introduces a new engineering problem:

The longer the context and the more concurrent users we have, the more GPU memory the KV Cache consumes.This becomes especially important when we want to move from running a model for one user to serving many users at the same time.In the next article, we will take a closer look at KV Cache itself—how it is stored, why it consumes so much GPU memory, and why efficient cache management is one of the key challenges in modern LLM serving.

References

1. Stanford Institute for Human-Centered Artificial Intelligence, AI Index Report 2025.
2. W. Kwon et al., Efficient Memory Management for Large Language Model Serving with PagedAttention, SOSP 2023.
3. vLLM Project, vLLM: Easy, Fast, and Cheap LLM Serving with PagedAttention, 2023.
4. NIST, Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile, NIST AI 600-1, 2024.