Day 18 ~ Day 21 不是才發過番外篇，怎麼現在又番外了？

事出突然，我昨天在 survey GitHub Actions self-hosted Runner 的地端架設方式時發現 GitHub 使用 kubernetes (k8s) 的 Operator 機制做了一個可以自動根據需求動態產生/收回 runner 的解決方案，叫做 Actions Runner Controller (ARC)。

若要實驗此方案，我們需要一個 kubernetes cluster，對於母胎沒摸過 k8s、只玩過 Docker Swarm 的我來說，難度有點跨太大。

所幸外面有一個可以快速在一台機器上建立 k8s control plane & worker 的工具，叫做 minikube。今天先把 minikube 跟 ARC 都先裝好，再來試試看跟之前使用 Docker Swarm 裝的 runner 有何不同。

Day 28 Edit：寫上述的引言時，我資料都找好了、天真地以為大概 2 小時內就可以做完，結果我最後花了 3 個晚上才做完。請繼續我這 k8s 麻瓜是如何踩坑的吧。

安裝 Minikube

這次準備一個全新環境。先安裝 minikube：

user@minikube:~$ curl -LO https://storage.googleapis.com/minikube/releases/latest/minikube_latest_amd64.deb
sudo dpkg -i minikube_latest_amd64.deb
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 28.5M  100 28.5M    0     0  5547k      0  0:00:05  0:00:05 --:--:-- 6411k
[sudo] password for q:
Selecting previously unselected package minikube.
(Reading database ... 51153 files and directories currently installed.)
Preparing to unpack minikube_latest_amd64.deb ...
Unpacking minikube (1.31.2-0) ...
Setting up minikube (1.31.2-0) ...

裝好後，啟動 minikube：

user@minikube:~$ minikube start
😄  minikube v1.31.2 on Ubuntu 22.04 (amd64)
👎  Unable to pick a default driver. Here is what was considered, in preference order:
💡  Alternatively you could install one of these drivers:
    ▪ docker: Not installed: exec: "docker": executable file not found in $PATH
    ▪ kvm2: Not installed: exec: "virsh": executable file not found in $PATH
    ▪ podman: Not installed: exec: "podman": executable file not found in $PATH
    ▪ qemu2: Not installed: exec: "qemu-system-x86_64": executable file not found in $PATH
    ▪ virtualbox: Not installed: unable to find VBoxManage in $PATH

❌  Exiting due to DRV_NOT_DETECTED: No possible driver was detected. Try specifying --driver, or see https://minikube.sigs.k8s.io/docs/start/

...忘記裝負責容器化的 driver 了。可任選 docker 或其他虛擬化工具，這邊我選擇裝 podman，因為我沒裝過。我參考此文章 的教學使用 apt 安裝 podman：

user@minikube:~$ sudo apt-get install podman
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following additional packages will be installed:
  # 中略.....
  libsoup2.4-common libyajl2 podman session-migration slirp4netns uidmap
0 upgraded, 34 newly installed, 0 to remove and 16 not upgraded.
Need to get 26.9 MB of archives.
After this operation, 114 MB of additional disk space will be used.
Do you want to continue? [Y/n] y
Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 uidmap amd64 1:4.8.1-2ubuntu2.1 [22.4 kB]
# 中略.....
Processing triggers for libc-bin (2.35-0ubuntu3.1) ...
user@minikube:~$

試著跑個容器試試：

user@minikube:~$ podman pull docker.io/library/ubuntu:22.04 && podman run --interactive --tty --rm ubuntu:22.04 bash
WARN[0000] "/" is not a shared mount, this could cause issues or missing mounts with rootless containers
Trying to pull docker.io/library/ubuntu:22.04...
Getting image source signatures
Copying blob 445a6a12be2b done
Copying config c6b84b685f done
Writing manifest to image destination
Storing signatures
c6b84b685f35f1a5d63661f5d4aa662ad9b7ee4f4b8c394c022f25023c907b65

可以，那回去啟動 minikube：

user@minikube:~$ minikube start
😄  minikube v1.31.2 on Ubuntu 22.04 (amd64)
👎  Unable to pick a default driver. Here is what was considered, in preference order:
    ▪ podman: Not healthy: "sudo -n -k podman version --format {{.Version}}" exit status 1: sudo: a password is required
    ▪ podman: Suggestion: Add your user to the 'sudoers' file: 'q ALL=(ALL) NOPASSWD: /usr/bin/podman' , or run 'minikube config set rootless true' <https://podman.io>
💡  Alternatively you could install one of these drivers:
    ▪ docker: Not installed: exec: "docker": executable file not found in $PATH
    ▪ kvm2: Not installed: exec: "virsh": executable file not found in $PATH
    ▪ qemu2: Not installed: exec: "qemu-system-x86_64": executable file not found in $PATH
    ▪ virtualbox: Not installed: unable to find VBoxManage in $PATH

❌  Exiting due to DRV_NOT_HEALTHY: Found driver(s) but none were healthy. See above for suggestions how to fix installed drivers.

user@minikube:~$

還是不行? 訊息明顯指出是目前的帳號無法執行 podman。我目前不想去搞 rootless container 免得節外生枝，我切成 root 再來玩玩看：

user@minikube:~$ sudo su
root@minikube:/home/q# minikube start
😄  minikube v1.31.2 on Ubuntu 22.04 (amd64)
✨  Automatically selected the podman driver. Other choices: ssh, none
🛑  The "podman" driver should not be used with root privileges. If you wish to continue as root, use --force.
💡  If you are running minikube within a VM, consider using --driver=none:
📘    https://minikube.sigs.k8s.io/docs/reference/drivers/none/

❌  Exiting due to DRV_AS_ROOT: The "podman" driver should not be used with root privileges.

root@minikube:/home/q# exit
exit

好吧，不能用 root 跑 podman。那就遵從一開始 output message 的建議，把 q ALL=(ALL) NOPASSWD: /usr/bin/podman 透過 visudo 加入 sudoers file內。

加好後，再來一次：

user@minikube:~$ minikube start
😄  minikube v1.31.2 on Ubuntu 23.04
✨  Automatically selected the podman driver. Other choices: ssh, none
📌  Using Podman driver with root privileges
👍  Starting control plane node minikube in cluster minikube
🚜  Pulling base image ...
💾  Downloading Kubernetes v1.27.4 preload ...
    > preloaded-images-k8s-v18-v1...:  393.21 MiB / 393.21 MiB  100.00% 5.38 Mi
    > gcr.io/k8s-minikube/kicbase...:  447.62 MiB / 447.62 MiB  100.00% 5.34 Mi
E0928 15:36:46.968385    2977 cache.go:190] Error downloading kic artifacts:  not yet implemented, see issue #8426
🔥  Creating podman container (CPUs=2, Memory=3900MB) ...
🐳  Preparing Kubernetes v1.27.4 on Docker 24.0.4 ...
    ▪ Generating certificates and keys ...
    ▪ Booting up control plane ...
    ▪ Configuring RBAC rules ...
🔗  Configuring bridge CNI (Container Networking Interface) ...
🔎  Verifying Kubernetes components...
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟  Enabled addons: storage-provisioner, default-storageclass
💡  kubectl not found. If you need it, try: 'minikube kubectl -- get pods -A'
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default

中間雖然噴錯，但還是好了。

PS. 我中途有換過環境。原本使用 WSL2 的 Ubuntu 22.04，後來使用 VirtualBox 裝了一個 Ubuntu 23.04。

測試 cluster 是否有起來。我們可以執行以下指令

$ minikube kubectl -- get po -A

但我懶得每次都在前面打 minikube，因此還是決定裝 kubectl。反正這台是實驗用 VM，隨便玩：

$ curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
$ sudo install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl

回來執行確認 cluster 狀態的指令：

user@minikube:~$ kubectl get pods -A
NAMESPACE     NAME                               READY   STATUS    RESTARTS        AGE
kube-system   coredns-5d78c9869d-v525t           1/1     Running   0               8m16s
kube-system   etcd-minikube                      1/1     Running   0               8m28s
kube-system   kube-apiserver-minikube            1/1     Running   0               8m30s
kube-system   kube-controller-manager-minikube   1/1     Running   0               8m28s
kube-system   kube-proxy-qdbp7                   1/1     Running   0               8m17s
kube-system   kube-scheduler-minikube            1/1     Running   0               8m28s
kube-system   storage-provisioner                1/1     Running   1 (7m46s ago)   8m26s

可以看到 API Server 跟其他 pod 都起來了。

接著安裝 Helm，它是 kubernetes 的套件管理工具，類似於 Linux 的 APT 或 YUM。它允許開發人員定義，安裝和升級甚至最複雜的 Kubernetes 應用。Helm 使用一種稱為 Chart 的包格式來描述 Kubernetes 應用：

curl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash

裝好後，照官方指令安裝 ARC:

user@host:~$ NAMESPACE="arc-systems"
helm install arc \
    --namespace "${NAMESPACE}" \
    --create-namespace \
    oci://ghcr.io/actions/actions-runner-controller-charts/gha-runner-scale-set-controller
Error: INSTALLATION FAILED: failed to authorize: failed to fetch anonymous token: unexpected status from GET request to https://ghcr.io/token?scope=repository%3Aactions%2Factions-runner-controller-charts%2Fgha-runner-scale-set-controller%3Apull&scope=repository%3Auser%2Fimage%3Apull&service=ghcr.io: 403 Forbidden

403 Forbidden？今天整個過程也太不順了吧！？

可能流量用太兇被擋了，明天繼續～