K8s 是一個用來編排容器的工具，這近幾年非常火紅。有著高可靠的架構、應用程序部署、自動部署、擴展等功能。

環境

實驗環境會有一台 master 和兩台 node 而資源配置如下

• CPU
  • 2 vCPU
• memory
  • 2 GB

192.168.134.131 | master
192.168.134.133 | node01
192.168.134.135 | node02

配置 hostname

$ sudo hostnamectl set-hostname master
$ sudo hostnamectl set-hostname node01
$ sudo hostnamectl set-hostname node02

新增靜態對應表，讓其可解析

$ sudo vim /etc/hosts
192.168.134.131 master
192.168.134.133 node01
192.168.134.133 node02

禁用 Swap

所有機器應該要固定 CPU/memory，不應該使用 swap 會導致效能降低。

$ sudo swapoff -a
$ sudo swapon -s

要保持永久效果至 fstab 將 swap 項目註解

$ sudo vim /etc/fstab
...
#/dev/mapper/ubuntu--vg-swap_1 none            swap    sw              0       0

安裝 Kubernetes 前置軟體

$ sudo apt-get update
$ sudo apt-get install apt-transport-https ca-certificates curl software-properties-common

還需要安裝 Docker 這邊不說明。可至官方查看

安裝 Kubernetes

$ sudo su -c "curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -"

創建 Kubernetes 儲存庫檔案

$ sudo vim /etc/apt/sources.list.d/kubernetes.list
deb https://apt.kubernetes.io/ kubernetes-xenial main

更新 cache，安裝 kubelet、kubeadm 和 kubectl 工具

$ sudo apt update   
$ sudo apt-get install -y kubelet kubeadm kubectl

上述步驟三台主機都需要設置。

初始化叢集

在 master 主機執行以下

$ sudo kubeadm init --pod-network-cidr=10.244.0.0/16 --apiserver-advertise-address=192.168.134.131 --kubernetes-version v1.18.8
# --apiserver-advertise-address: Api server 服務 IP
# --apiserver-bind-port: Api server 服務 port，預設 6443
# --pod-network-cidr: 分配 Pod IP 範圍 (CIDR)
# --kubernetes-version: 指定 Kubernetes 版本
...
[mark-control-plane] Marking the node k8smaster as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[bootstrap-token] Using token: 7zd2c5.w9825shkwsa7vjat
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 192.168.134.131:6443 --token av0pcs.izdosjqm3csj1par \ 
    --discovery-token-ca-cert-hash sha256:320751ce55f3163938653d85bfb4ee44104cabc092bed010d13fcab1d384042c

以一般用戶身份運行以下。讓 kubectl 可連到 Kubernetes，將設定複製到 $HOME/.kube/config 底下，kubectl 執行時會到該目錄下取得 Api server 等相關資訊。

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

如果給的加入 K8s 的 Node Token 遺失，可透過以下來獲取

$ kubeadm token list
TOKEN                     TTL         EXPIRES                USAGES                   DESCRIPTION                                                EXTRA GROUPS
av0pcs.izdosjqm3csj1par   23h         2020-08-19T09:21:36Z   authentication,signing   The default bootstrap token generated by 'kubeadm init'.   system:bootstrappers:kubeadm:default-node-token

上述如果要重置可用 kubeadm reset 來完成，會移除相關資訊。
TLS 錯誤參考 troubleshooting-kubeadm

查看 master 狀況

發現 master 的狀態沒 ready，其中影響是 DNS 元件。

$ kubectl get nodes
NAME     STATUS     ROLES    AGE   VERSION
master   NotReady   master   10m   v1.18.8

前面步驟完成後，在 master 節點上，預設會在 default namespace 上，而 Kubernetes 上的預設安裝元件會在 kube-system 上。

$ kubectl get namespace
NAME              STATUS   AGE
default           Active   14m
kube-node-lease   Active   14m
kube-public       Active   14m
kube-system       Active   14m

切換至 kube-system 上查看。

$ kubectl -n kube-system get pods # -n 切換 namespace
NAME                             READY   STATUS    RESTARTS   AGE
coredns-66bff467f8-5pfbz         0/1     Pending   0          11m
coredns-66bff467f8-v4txv         0/1     Pending   0          11m
etcd-master                      1/1     Running   0          11m
kube-apiserver-master            1/1     Running   0          11m
kube-controller-manager-master   1/1     Running   0          11m
kube-proxy-fm749                 1/1     Running   0          11m
kube-scheduler-master            1/1     Running   0          11m

而 DNS 是因為系統設計需求，需要安裝 CNI 提供網路解決，這邊為官網資訊。

部署 CNI

先前已經定義了一個 POD 網路，但是，應該為網路創建一個 namespace。可依照環境選擇適當的 Kubernetes 的 CNI plugin。這邊使用 Flannel，Flannel 將為 Kubernetes 提供 overlay network。若想要自定義的 CIDR，需更改 flannel 中的 net-conf.json 的 Network 並用新的 CIDR init。

藉由 Flannel YAML 創建 POD 網路。

$ wget https://raw.githubusercontent.com/coreos/flannel/32a765fd19ba45b387fdc5e3812c41fff47cfd55/Documentation/kube-flannel.yml
$ kubectl apply -f kube-flannel.yml
podsecuritypolicy.policy/psp.flannel.unprivileged created
clusterrole.rbac.authorization.k8s.io/flannel configured
clusterrolebinding.rbac.authorization.k8s.io/flannel unchanged
serviceaccount/flannel unchanged
configmap/kube-flannel-cfg unchanged
daemonset.apps/kube-flannel-ds-amd64 created
daemonset.apps/kube-flannel-ds-arm64 created
daemonset.apps/kube-flannel-ds-arm created
daemonset.apps/kube-flannel-ds-ppc64le created
daemonset.apps/kube-flannel-ds-s390x created

提供網路後，DNS 元件已經運行起來。

$ kubectl -n kube-system get pods
NAME                             READY   STATUS    RESTARTS   AGE
coredns-66bff467f8-5pfbz         1/1     Running   0          18m
coredns-66bff467f8-v4txv         1/1     Running   0          18m
etcd-master                      1/1     Running   0          18m
kube-apiserver-master            1/1     Running   0          18m
kube-controller-manager-master   1/1     Running   0          18m
kube-flannel-ds-amd64-gljsn      1/1     Running   0          55s
kube-proxy-fm749                 1/1     Running   0          18m
kube-scheduler-master            1/1     Running   0          18m

驗證 master

檢查 master 機器狀態

$ kubectl get node
NAME     STATUS   ROLES    AGE   VERSION
master   Ready    master   23m   v1.18.8

$ kubectl get node -o wide
NAME     STATUS   ROLES    AGE   VERSION   INTERNAL-IP       EXTERNAL-IP   OS-IMAGE           KERNEL-VERSION     CONTAINER-RUNTIME
master   Ready    master   23m   v1.18.8   192.168.134.131   <none>        Ubuntu 20.04 LTS   5.4.0-42-generic   docker://19.3.12

要知道當前的 namespace。在建立 Kubernetes 集群時，預設下創建以下所有 namespace。

$ kubectl get pods --all-namespaces
NAMESPACE     NAME                             READY   STATUS    RESTARTS   AGE
kube-system   coredns-66bff467f8-5pfbz         1/1     Running   0          23m
kube-system   coredns-66bff467f8-v4txv         1/1     Running   0          23m
kube-system   etcd-master                      1/1     Running   0          24m
kube-system   kube-apiserver-master            1/1     Running   0          24m
kube-system   kube-controller-manager-master   1/1     Running   0          24m
kube-system   kube-flannel-ds-amd64-gljsn      1/1     Running   0          6m20s
kube-system   kube-proxy-fm749                 1/1     Running   0          23m
kube-system   kube-scheduler-master            1/1     Running   0          24m

加入 node 機器至 K8s 叢集

在前面初始化時，最後給了 Token。透過 Token，將 node 與 master 連接起來。以下針對每台 node 進行加入的操作。

$ sudo kubeadm join 192.168.134.131:6443 --token av0pcs.izdosjqm3csj1par --discovery-token-ca-cert-hash sha256:320751ce55f3163938653d85bfb4ee44104cabc092bed010d13fcab1d384042c
...
This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.

Run 'kubectl get nodes' on the control-plane to see this node join the cluster.

再加入 node 時輸出錯誤嘗試執行以下，重新嘗試

$ sudo kubeadm reset

驗證節點

狀態已 Ready 顯示表示成功

$ kubectl get nodes
NAME     STATUS   ROLES    AGE    VERSION
master   Ready    master   33m    v1.18.8
node01   Ready    <none>   4m2s   v1.18.8
node02   Ready    <none>   52s    v1.18.8

額外資訊

$ sudo vim /etc/systemd/system/kubelet.service.d/10-kubeadm.conf  # 是透過 kubeadm 去架設 kubernetes cluster，所以設定檔可以置這邊修改。可參考下面"kubeadm 與 kubelet 溝通"

參考資源

• kubernetes 官網

• flannel

• create-cluster-kubeadm

• troubleshooting-kubeadm

• network-plugins

• kubeadm 與 kubelet 溝通