Day 11 Kubernetes 閃電戰-kubernetes 安裝基礎操作篇

本日重點與方向 (TAG): kubernetes、k8s、Pod、Deplyment、Service、NodePort、ClusterIP、CNI、Flannel
今天將會介紹使用 Bare Metal 進行 Kubernetes 環境的部署，並對於昨日的 Docker 功能進行整合，基本上就是安裝筆記居多，配置上就是樣板填一填就可以弄好啦，資源型態就是簡單寫一些，還有一些進階一點的去看官方網站 吧 (甩鍋) ，詳細的操作還是去看看大神的文吧。

本次使用設備資訊

Network Switch

• 數量: 1
• 型號: D-Link 1210-28 (L2 Switch)

Bare Metal (Master Node)

• 數量: 1
• Model Name: HP DL360 G9
• Node Name: sdn-k8s-server-b3-1
• Ubuntu: 16.04 / 18.04
• Docker Version: 19.03
• CPU: E5-2620_v4 ^ 2
• RAM: 48GB
• Disk: 300 GB (SAS)
• Network: 1Gbps

Kubernetes 安裝

這邊會給 kubernetes 的快速佈建的指令，就照貼基本上就會弄得起來了。

kubenetes 相關元件安裝

apt-get update && apt-get install -y apt-transport-https curl
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -
cat <<EOF >/etc/apt/sources.list.d/kubernetes.list
deb https://apt.kubernetes.io/ kubernetes-xenial main
EOF
apt-get update
apt-get install -y kubelet kubeadm kubectl
apt-mark hold kubelet kubeadm kubectl

• 查 Kubernetes 相關套件來源

# 指定版本安裝 kube* 套件：
apt list -a kubeadm
apt-get install -y kubelet=1.15.5-00 kubeadm=1.15.5-00 kubectl=1.15.5-00

Docker 連結操縱

cat > /etc/docker/daemon.json <<EOF
{
  "exec-opts": ["native.cgroupdriver=systemd"],
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "100m"
  },
  "storage-driver": "overlay2"
}
EOF
mkdir -p /etc/systemd/system/docker.service.d
systemctl daemon-reload
systemctl restart docker

部署叢集

各節點預備動作

• 關閉 Linux swap

swapoff -a

主節點

kubeadm init  --pod-network-cidr=10.244.0.0/16
mkdir -p $HOME/.kube
cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
chown $(id -u):$(id -g) $HOME/.kube/config

• 使用外網存取 kubernetes 的 API Server 的能力配置

https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm-init/
單純開 port-forwarding 的話，會因為使用外網 IP 導致 TLS 證書認證失效
所以把外網存取 IP 加上去讓它一起生 TLS 證書才能 Access

--apiserver-cert-extra-sans=<external-ip>

獲取主節點加入指令

kubeadm token create --print-join-command

Flannel CNI

kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

檢查 Flannel CNI 是否運作

root@sdn-k8s-server-b3-1:~# kubectl get pod -n kube-system -o wide
NAME                                          READY   STATUS    RESTARTS   AGE   IP           NODE                  NOMINATED NODE   READINESS GATES
coredns-5c98db65d4-dsbzc                      1/1     Running   0          15h   10.244.0.3   sdn-k8s-server-b3-1   <none>           <none>
coredns-5c98db65d4-kdvrn                      1/1     Running   0          15h   10.244.0.2   sdn-k8s-server-b3-1   <none>           <none>
etcd-sdn-k8s-server-b3-1                      1/1     Running   0          15h   10.0.0.218   sdn-k8s-server-b3-1   <none>           <none>
kube-apiserver-sdn-k8s-server-b3-1            1/1     Running   0          15h   10.0.0.218   sdn-k8s-server-b3-1   <none>           <none>
kube-controller-manager-sdn-k8s-server-b3-1   1/1     Running   0          15h   10.0.0.218   sdn-k8s-server-b3-1   <none>           <none>
kube-flannel-ds-8l796                         1/1     Running   0          15h   10.0.0.218   sdn-k8s-server-b3-1   <none>           <none>
kube-proxy-kkgm5                              1/1     Running   0          15h   10.0.0.218   sdn-k8s-server-b3-1   <none>           <none>
kube-scheduler-sdn-k8s-server-b3-1            1/1     Running   0          15h   10.0.0.218   sdn-k8s-server-b3-1   <none>           <none>

污染主節點

• 污染

kubectl taint nodes --all node-role.kubernetes.io/master-

• 解除污染

kubectl taint nodes <node-name> node-role.kubernetes.io/master=true:NoSchedule

查看節點狀況

root@sdn-k8s-server-b3-1:~# kubectl get nodes -o wide
NAME                  STATUS   ROLES    AGE   VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION     CONTAINER-RUNTIME
sdn-k8s-server-b3-1   Ready    master   14h   v1.15.5   10.0.0.218    <none>        Ubuntu 20.04.1 LTS   5.4.0-48-generic   docker://19.3.13

資源部署

Pod

• Pod Template

ubuntu-pod.yaml
-----
apiVersion: v1
kind: Pod
metadata:
  name: ubuntu-pod
spec:
  containers:
  - name: ubuntu
    image: ubuntu:19.04
    args: [bash, -c, 'for ((i = 0; ; i++)); do echo "$i: $(date)"; sleep 100; done']

• Deploy Pod

root@sdn-k8s-server-b3-1:~# kubectl apply -f ubuntu-pod.yaml 
pod/ubuntu-pod created

• Get Pod Status

root@sdn-k8s-server-b3-1:~# kubectl get pod -o wide
NAME                                       READY   STATUS    RESTARTS   AGE   IP            NODE                  NOMINATED NODE   READINESS GATES
ubuntu-pod                                 1/1     Running   0          8s    10.244.0.39   sdn-k8s-server-b3-1   <none>           <none>

• Delete Pod

root@sdn-k8s-server-b3-1:~# kubectl delete pod ubuntu-pod
pod "ubuntu-pod" deleted

Deployment

• Deployment Template

ubuntu-deployment.yaml
-----
apiVersion: apps/v1
kind: Deployment
metadata:
  name: ubuntu-deployment
  labels:
    app: ubuntu
spec:
  replicas: 10
  selector:
    matchLabels:
      app: ubuntu
  template:
    metadata:
      labels:
        app: ubuntu
    spec:
      containers:
      - name: ubuntu
        image: ubuntu:19.04
        args: [bash, -c, 'for ((i = 0; ; i++)); do echo "$i: $(date)"; sleep 100; done']

• Deploy Deployment

root@sdn-k8s-server-b3-1:~# kubectl apply -f ubuntu-deployment.yaml 
deployment.apps/ubuntu-deployment created

• Get Deployment Status

root@sdn-k8s-server-b3-1:~# kubectl get deployment -o wide
NAME                      READY   UP-TO-DATE   AVAILABLE   AGE     CONTAINERS              IMAGES                                    SELECTOR
ubuntu-deployment         10/10   10           10          2m20s   ubuntu                  ubuntu:19.04                              app=ubuntu

• Get Deployment's Replicas Pod Status

root@sdn-k8s-server-b3-1:~# kubectl get pod -o wide
NAME                                       READY   STATUS    RESTARTS   AGE    IP            NODE                  NOMINATED NODE   READINESS GATES
ubuntu-deployment-5c79f46d78-48s7b         1/1     Running   0          3m4s   10.244.0.35   sdn-k8s-server-b3-1   <none>           <none>
ubuntu-deployment-5c79f46d78-4p4q9         1/1     Running   0          3m4s   10.244.0.29   sdn-k8s-server-b3-1   <none>           <none>
ubuntu-deployment-5c79f46d78-4s8kx         1/1     Running   0          3m4s   10.244.0.37   sdn-k8s-server-b3-1   <none>           <none>
ubuntu-deployment-5c79f46d78-5bhvh         1/1     Running   0          3m4s   10.244.0.34   sdn-k8s-server-b3-1   <none>           <none>
ubuntu-deployment-5c79f46d78-ckpzs         1/1     Running   0          3m4s   10.244.0.33   sdn-k8s-server-b3-1   <none>           <none>
ubuntu-deployment-5c79f46d78-drtjt         1/1     Running   0          3m4s   10.244.0.30   sdn-k8s-server-b3-1   <none>           <none>
ubuntu-deployment-5c79f46d78-h4f25         1/1     Running   0          3m4s   10.244.0.36   sdn-k8s-server-b3-1   <none>           <none>
ubuntu-deployment-5c79f46d78-k2ngd         1/1     Running   0          3m4s   10.244.0.32   sdn-k8s-server-b3-1   <none>           <none>
ubuntu-deployment-5c79f46d78-k4r5n         1/1     Running   0          3m4s   10.244.0.38   sdn-k8s-server-b3-1   <none>           <none>
ubuntu-deployment-5c79f46d78-tpndc         1/1     Running   0          3m4s   10.244.0.31   sdn-k8s-server-b3-1   <none>           <none>

• Delete Deployment

root@sdn-k8s-server-b3-1:~# kubectl delete deployment ubuntu-deployment
deployment.extensions "ubuntu-deployment" deleted

觀察指定 Node 上的 Pod

https://kubernetes.io/docs/reference/kubectl/cheatsheet/

• 搭配這個用 Json 格式找階層

Json Parser 好夥伴
http://json.parser.online.fr/

kubectl get pod -o json

• Command

kubectl get pod --field-selector=spec.nodeName=<nodeNmae>

Service

• Create Connection Pod (with Layer 7 Connection)

nginx-pod.yaml
-----
apiVersion: v1
kind: Pod
metadata:
  name: nginx-pod
  namespace: default
  labels:
    app: nginx
spec:
  containers:
  - image: nginx:latest
    name: nginx

ClusterIP

• Service Template

apiVersion: v1
kind: Service
metadata:
  name: nginx-service-clusterip
spec:
  selector:
    app: nginx
  ports:
  - name: http
    protocol: TCP
    port: 3000
    targetPort: 80

• Service Deployment

root@sdn-k8s-server-b3-1:~# kubectl apply -f nginx-clusterip-service.yaml 
service/nginx-service-clusterip created

• Get Deployment Status (ClusterIP)

root@sdn-k8s-server-b3-1:~# kubectl get service -o wide
NAME                      TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)     AGE     SELECTOR
kubernetes                ClusterIP   10.96.0.1       <none>        443/TCP     14h     <none>
nginx-service-clusterip   ClusterIP   10.104.141.91   <none>        3000/TCP    2s      app=nginx

• Testing Service

主機端呼叫測試
------
root@sdn-k8s-server-b3-1:~# curl 10.104.141.91:3000
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>

• Internal Domain Testing (on Pod)

root@sdn-k8s-server-b3-1:~# kubectl get endpoints
NAME                      ENDPOINTS           AGE
kubernetes                10.0.0.218:6443     15h
nginx-service-clusterip   10.244.0.40:80      20m

Pod 中使用 domain 呼叫通訊使用
-----
root@ubuntu-pod:/# curl nginx-service-clusterip:3000
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>

• Delete Service

root@sdn-k8s-server-b3-1:~# kubectl delete service nginx-service-clusterip
service "nginx-service-clusterip" deleted

NodePort

• Service Template

nginx-nodeport-service.yaml
-----
apiVersion: v1
kind: Service
metadata:
  name: nginx-nodeport-service
spec:
  type: NodePort
  selector:
    app: nginx
  ports:
  - name: http
    protocol: TCP
    port: 80

• Service Deployment

root@sdn-k8s-server-b3-1:~# kubectl apply -f nginx-nodeport-service.yaml 
service/nginx-nodeport-service created

• Get Deployment Status (NodePort)

root@sdn-k8s-server-b3-1:~# kubectl get service -o wide
NAME                      TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE     SELECTOR
kubernetes                ClusterIP   10.96.0.1       <none>        443/TCP        15h     <none>
nginx-nodeport-service    NodePort    10.97.116.170   <none>        80:32436/TCP   83s     app=nginx

• Testing Service
  

• Delete Service

root@sdn-k8s-server-b3-1:~# kubectl delete service nginx-nodeport-service
service "nginx-nodeport-service" deleted

卸載資源異常處理方式

Pod 砍不掉

• Always Termating

kubectl delete pods <pod-name> --grace-period=0 --force

• Status unknown

kubectl patch pod <pod-name> -p '{"metadata":{"finalizers":null}}'

PVC 砍不掉

• Always Termating

kubectl patch pvc <pvc-name> -p '{"metadata":{"finalizers":null}}'

PV 砍不掉

• Always Termating

kubectl patch pv <pv-name> -p '{"metadata":{"finalizers":null}}'

Namespece 砍不掉

• Always Terminating

kubectl get namespace <namespace-name> -o json > tmp.json

• Edit and Remove "kubernetes" string in tmp.json

nano tmp.json
---
{
    "apiVersion": "v1",
    "kind": "Namespace",
    "metadata": {
        "annotations": {
            "kubectl.kubernetes.io/last-applied-configuration": "{\"apiVersion\":\"v1\",\"kind\":\"Namespace\",\"metadat$
        },
        "creationTimestamp": "2020-09-13T09:46:17Z",
        "deletionTimestamp": "2020-09-13T14:16:55Z",
        "name": "longhorn-system",
        "resourceVersion": "41246",
        "selfLink": "/api/v1/namespaces/longhorn-system",
        "uid": "26fe2926-b206-4711-b5b7-67b766a2f596"
    },
    "spec": {
        "finalizers": [
            --> "kubernetes"
        ]
    },
    "status": {
        "phase": "Terminating"
    }
}

• Call Kubernetes API

curl -k -H "Content-Type: application/json" -X PUT –data-binary @tmp.json 
https://<kubernetes-cluster-ip>:6443/api/v1/namespaces/<namespace-name>/finalize

or

kubectl replace --raw "/api/v1/namespaces/<namespace-name>/finalize" -f ./tmp.json

修復 kubernetes 重建與 CoreDNS 異常問題修復

kubeadm reset -f
rm $HOME/.kube/config
systemctl stop kubelet
systemctl stop docker
rm -rf /var/lib/cni/
rm -rf /var/lib/kubelet/*
rm -rf /etc/cni/
ifconfig cni0 down
ifconfig flannel.1 down
ifconfig docker0 down
ip link delete cni0
ip link delete flannel.1
systemctl restart kubelet
systemctl restart docker