Day 29 — 打包及部署：撰寫 .gitlab-ci.yml - iT 邦幫忙::一起幫忙解決難題，拯救 IT 人的一天

2022 iThome 鐵人賽

DAY 29

DevOps

前端轉生～到了實驗室就要養幾隻可愛鯨魚：自架 Kubernetes 迷航日記系列第 29 篇

Day 29 — 打包及部署：撰寫 .gitlab-ci.yml

14th鐵人賽 kubernetes gitlab gitlab-ci

Time

2022-10-13 23:33:20

2043 瀏覽

分享至

可愛鯨魚

今天繼續來學 gitlab-ci...

圖片來源：Docker (@Docker) / Twitter

上一篇初略寫了 gitlab runner 運作和配置，今天繼續往下～

Integration

首先將 harbor 和 gitlab 連接，讓 job 可以直接取得 harbor 配置資訊

我是在 group 直接設定，這樣底下的 project 就不必重新設定了～

.gitlab-ci.yml

目前 group 裡的 project...

前端 .gitlab-ci.yml

前端是一個 react app

stages

三個 stage:

stages:
  - test
  - build
  - integrate

將 node_modules 加入 cache，避免每次都要重新安裝套件

cache:
  paths:
    - node_modules/

test

執行 unit test

react-app-unit-test:
  stage: test
  image: node:15
  script:
    - npm install
    - npm run test

build

分為兩個 job

打包套件

build-react-app:
  stage: build
  image: node:15
  script:
    - npm install
    - npm run build
  artifacts:
    paths:
      - dist/

打包成 image 並上傳到 Harbor

build-image:
  stage: build
  image:
    name: gcr.io/kaniko-project/executor:v1.9.0-debug
    entrypoint: [""]
  script:
    - mkdir -p /kaniko/.docker
    - echo "{\"auths\":{\"${HARBOR_HOST}\":{\"auth\":\"$(echo -n ${HARBOR_USERNAME}:${HARBOR_PASSWORD} | base64 | tr -d '\n')\"}}}" > /kaniko/.docker/config.json
    - >-
      /kaniko/executor
      --context "${CI_PROJECT_DIR}"
      --dockerfile "${CI_PROJECT_DIR}/Dockerfile"
      --destination "${HARBOR_HOST}/${HARBOR_PROJECT}/${CI_PROJECT_NAME}:${CI_COMMIT_SHORT_SHA}"
      --destination "${HARBOR_HOST}/${HARBOR_PROJECT}/${CI_PROJECT_NAME}:latest"
      --registry-certificate "${HARBOR_HOST}=/etc/gitlab-runner/certs/${HARBOR_HOST}.crt"

integrate

呼叫下游的 pipeline 和後端一起執行 integration test

integration-test:
  stage: integrate
  trigger:
    project: test-web/test-web-chart

後端 .gitlab-ci.yml

後端是一個用 express 寫的 server

stages

兩個 stage:

stages:
  - build
  - integrate

build

打包成 image 上傳到 Harbor

build-image:
  stage: build
  image:
    name: gcr.io/kaniko-project/executor:v1.9.0-debug
    entrypoint: [""]
  script:
    - mkdir -p /kaniko/.docker
    - echo "{\"auths\":{\"${HARBOR_HOST}\":{\"auth\":\"$(echo -n ${HARBOR_USERNAME}:${HARBOR_PASSWORD} | base64 | tr -d '\n')\"}}}" > /kaniko/.docker/config.json
    - >-
      /kaniko/executor
      --context "${CI_PROJECT_DIR}"
      --dockerfile "${CI_PROJECT_DIR}/Dockerfile"
      --destination "${HARBOR_HOST}/${HARBOR_PROJECT}/${CI_PROJECT_NAME}:${CI_COMMIT_SHORT_SHA}"
      --destination "${HARBOR_HOST}/${HARBOR_PROJECT}/${CI_PROJECT_NAME}:latest"
      --registry-certificate "${HARBOR_HOST}=/etc/gitlab-runner/certs/${HARBOR_HOST}.crt"

integrate

呼叫下游的 pipeline 和前端一起執行 integration test

integration-test:
  stage: integrate
  trigger:
    project: test-web/test-web-chart

Chart .gitlab-ci.yml

Chart 是用來整合前後端，便於用 helm 部署

kubeconfig

在 Kubernetes 先建立一個 ServiceAccount

這裡直接用 cluster-admin 如有安全疑慮可再細部設定

apiVersion: v1
kind: ServiceAccount
metadata:
  name: gitlab-ci
  namespace: gitlab

---
apiVersion: v1
kind: Secret
metadata:
  name: gitlab-ci-secret
  namespace: gitlab
  annotations:
    kubernetes.io/service-account.name: gitlab-ci
type: kubernetes.io/service-account-token

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: gitlab-ci-admin
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
  - kind: ServiceAccount
    name: gitlab-ci
    namespace: gitlab

取得 ServiceAccount token

kubectl get -n gitlab secret/gitlab-ci-secret -o jsonpath='{.data.token}'

接著在 project CI/CD 新增 Variable

DEPLOY_KUBERNETES_BEARER_TOKEN 類型 Variable 並設定為 Masked，將上一步取得的 token 存入

DEPLOY_KUBE_CONFIG 類型設定為 File

apiVersion: v1 
kind: Config
clusters:    
- cluster:
    certificate-authority: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
    server: https://$${KUBERNETES_SERVICE_HOST}:$${KUBERNETES_SERVICE_PORT}
  name: kubernetes 
contexts:
- context:
    cluster: kubernetes
    namespace: ${CI_PROJECT_NAMESPACE}
    user: gitlab-ci
  name: gitlab@kubernetes
current-context: gitlab@kubernetes
users:
- name: gitlab-ci
  user:
    token: ${DEPLOY_KUBERNETES_BEARER_TOKEN}

我後來才發現 kubernetes 帶入的 env 不能在這裡用... 之後再用 sed 取的就好～

stages

兩個 stage:

stages:
  - build
  - test

build

將 chart 打包上傳到 Harbor

build-chart:
  stage: build
  image:
    name: alpine/helm:3.10.1
    entrypoint: [""]
  only:
    - pushes
  script:
    - helm plugin install https://github.com/chartmuseum/helm-push
    - >-
      helm repo add
      --username ${HARBOR_USERNAME} 
      --password ${HARBOR_PASSWORD}
      --ca-file /etc/gitlab-runner/certs/ca.crt
      ${HARBOR_PROJECT} https://${HARBOR_HOST}/chartrepo/${HARBOR_PROJECT}
    - helm package ${CI_PROJECT_DIR}
    - helm cm-push --ca-file /etc/gitlab-runner/certs/ca.crt ${CI_PROJECT_DIR} ${HARBOR_PROJECT}

因為使用自簽憑證要再加上 --ca-file

test

先測試 Kubernetes 連線...

integration-test:
  stage: test
  image: bitnami/kubectl:latest
  script:
    - cat ${DEPLOY_KUBE_CONFIG} > "/.kube/config"
    - sed -i -e "s/\${KUBERNETES_SERVICE_HOST}:\${KUBERNETES_SERVICE_PORT}/${KUBERNETES_SERVICE_HOST}:${KUBERNETES_SERVICE_PORT}/" /.kube/config
    - kubectl get pods

因為機器炸了... 只好之後再補部署...