Large Language Models such as ChatGPT, Claude, and Gemini have made AI more accessile than ever, with only a few line of code, developes can intergrate powerful language models into their applications.

Why self-host an LLM?

1. Data Privacy (資料隱私)
2. Cost control (成本控制)
3. Custimization (客製化)
4. Latency & Throughput (延遲跟吞吐量)
5. Infrastructure Control (基礎設施控制)

We firstly need to why we are here to use the new technolgy for developing our project, what's pros and cons:

what dose out own model means: running our own model does not training a GPT-model from scratch. instea, we usually download an existing open-weight model such as: Qwen, Llam, Gemma, and Mistral. then, the inference engine be like:

Model -> vLLM -> GPU server -> API -> application
after seeing the flowchart, you probably can know is mainly talking about deploying and seving LLMs, not for the part of pre-training.

One of the reason, why organizations consider slef-hosting is control over sensitive data:
Pricacy and Data Control. for instance, a RAG system may use 1. interanal documents 2. Customer imnformation 3. source code 4. company kownledge. so if you are using own llm:

User -> RAG -> Vector Database -> Local LLM -> Answer
However, Self hosted does not automatically mean secure

Next we are gonna thinking of the Latency and throughput,
runing one request is easym but a real system may need to serve many users:
now we need to think about:

1. Latency 2. Throughtput 3. Batching 4. GPU utilization 5. Key-Value Cache 6. Request scheduling, this is the difference between simply running an LLM and seving an LLM efficiently. The original vLLM paper reported 2 toand 4 times higher at similar latency compared with the serving systems in this study.

like the model mentioned above that can be experiment with: FP16 / BF16, INT8/INT4, LoRA, AWQ, AND GPTQ., they gives us more freedom to balance: quality, speed, memory usage, and cost.

In the end, Cloud APIs are excellent for quickly building AI applications.
However, self-hosted LLMs become interesting when we need greater control over data, models,
infrastucture, preformance, and customization.

And once we decide to run a model ourselves, the next question becomes:
an LLM actually generate tokens