iT邦幫忙

第 12 屆 iThome 鐵人賽

DAY 20
0
DevOps

Hashicorp 0 到 0.003系列 第 20

Day 20. Nomad Job Update Strategies

Nomad支援二種更版的方式:

  1. Rolling Updates
  2. Blue/Green & Canary Deployments

Rolling Updates

在Nomad job file裡設定update並設定以下參數:
max_parallel: 設定一次可以同時更新幾個task.
min_healthy_time: 設定部屬多久時間內完成,則定義為成功的部屬。
healthy_deadline: 設定部屬多久時間內未完成,則定義為失敗的部屬。如果你的一個container部屬到啟動大約需要2分鐘,那麼healthy_deadline可以設定大於2分鐘,超過這個值時,就會被系統定義為部屬失敗。
auto_revert: 部屬失敗後,如果要自動恢復到上一版的話,可以設定auto_reverttrue.

示範Nginx從1.19升版到1.19.2

# nginx.nomad
job "erp" {
  datacenters = ["nomad-ithome"]
  group "webfront" {
    count = 6
    task "nginx" {
      driver = "docker"

      config {
        image = "nginx:1.19"
        port_map {
          web = 80
        }
      }

      resources {
        cpu    = 200
        memory = 128
        network {
          mbits = 10
          port "web" {}
        }
      }
    }


    update {
      max_parallel     = 2
      min_healthy_time = "30s"
      healthy_deadline = "2m"
      auto_revert = true
    }
  }
}
  1. 先檢查一下我們寫的job file,沒有語法上的錯誤。
$ nomad job validate nginx.nomad
Job validation successful
  1. 先用plan dry-run,成功。
$ nomad job plan nginx.nomad
+ Job: "erp"
+ Task Group: "webfront" (6 create)
  + Task: "nginx" (forces create)

Scheduler dry-run:
- All tasks successfully allocated.

Job Modify Index: 0
To submit the job with version verification run:

nomad job run -check-index 0 nginx.nomad

When running the job with the check-index flag, the job will only be run if the
job modify index given matches the server-side version. If the index has
changed, another user has modified the job and the plan's results are
potentially invalid.
  1. Submit the job.
$ nomad job run nginx.nomad
==> Monitoring evaluation "c60c1d17"
    Evaluation triggered by job "erp"
    Evaluation within deployment: "7fc0bf7e"
    Allocation "dd254879" created: node "728069bc", group "webfront"
    Allocation "f86dba63" created: node "728069bc", group "webfront"
    Allocation "ffd4a89a" created: node "a7db9bf6", group "webfront"
    Allocation "5a64d71d" created: node "728069bc", group "webfront"
    Allocation "b6daca57" created: node "a7db9bf6", group "webfront"
    Allocation "d36d8e82" created: node "728069bc", group "webfront"
    Evaluation status changed: "pending" -> "complete"
==> Evaluation "c60c1d17" finished with status "complete"
  1. 檢查job status,成功。
$ nomad job status erp
ID            = erp
Name          = erp
Submit Date   = 2020-09-20T15:12:18+08:00
Type          = service
Priority      = 50
Datacenters   = nomad-ithome
Namespace     = default
Status        = running
Periodic      = false
Parameterized = false

Summary
Task Group  Queued  Starting  Running  Failed  Complete  Lost
webfront    0       0         6        0       0         0

Latest Deployment
ID          = 7fc0bf7e
Status      = running
Description = Deployment is running

Deployed
Task Group  Auto Revert  Desired  Placed  Healthy  Unhealthy  Progress Deadline
webfront    true         6        6       0        0          2020-09-20T15:22:18+08:00

Allocations
ID        Node ID   Task Group  Version  Desired  Status   Created  Modified
5a64d71d  728069bc  webfront    0        run      running  21s ago  20s ago
b6daca57  a7db9bf6  webfront    0        run      running  21s ago  21s ago
d36d8e82  728069bc  webfront    0        run      running  21s ago  20s ago
dd254879  728069bc  webfront    0        run      running  21s ago  20s ago
f86dba63  728069bc  webfront    0        run      running  21s ago  20s ago
ffd4a89a  a7db9bf6  webfront    0        run      running  21s ago  21s ago
  1. 升版
    我們把image由"nginx:1.19" => "nginx:1.19.2"後,執行nomad job plan檢查。
    Nomad會output我們修改了哪些東西。
$ nomad job plan nginx.nomad
+/- Job: "erp"
-   NomadTokenID: "723414cf-d755-1f35-e55c-d157253ffc1e"
+/- Task Group: "webfront" (2 create/destroy update, 4 ignore)
  +/- Task: "nginx" (forces create/destroy update)
    +/- Config {
      +/- image:            "nginx:1.19" => "nginx:1.19.2"
          port_map[0][web]: "80"
        }

Scheduler dry-run:
- All tasks successfully allocated.

Job Modify Index: 101212
To submit the job with version verification run:

nomad job run -check-index 101212 nginx.nomad

When running the job with the check-index flag, the job will only be run if the
job modify index given matches the server-side version. If the index has
changed, another user has modified the job and the plan's results are
potentially invalid.
  1. 確認沒有問題後執行升版。
$ nomad job run -check-index 101212 nginx.nomad
==> Monitoring evaluation "72c7bee7"
    Evaluation triggered by job "erp"
    Evaluation within deployment: "3d5ae3b8"
    Allocation "27ee8ec2" created: node "728069bc", group "webfront"
    Allocation "38fe5bf1" created: node "a7db9bf6", group "webfront"
    Evaluation status changed: "pending" -> "complete"
==> Evaluation "72c7bee7" finished with status "complete"

可以用nomad job status erp來觀察task的狀態

ID        Node ID   Task Group  Version  Desired  Status    Created   Modified
27ee8ec2  728069bc  webfront    1        run      running   16s ago   16s ago
38fe5bf1  a7db9bf6  webfront    1        run      running   16s ago   16s ago
5a64d71d  728069bc  webfront    0        run      running   8m6s ago  7m35s ago
b6daca57  a7db9bf6  webfront    0        stop     complete  8m6s ago  16s ago
d36d8e82  728069bc  webfront    0        stop     complete  8m6s ago  16s ago
dd254879  728069bc  webfront    0        run      running   8m6s ago  7m35s ago
f86dba63  728069bc  webfront    0        run      running   8m6s ago  7m35s ago
ffd4a89a  a7db9bf6  webfront    0        run      running   8m6s ago  7m36s ago

ID        Node ID   Task Group  Version  Desired  Status    Created    Modified
61a2ad19  728069bc  webfront    1        run      running   4s ago     3s ago
929c0ce9  728069bc  webfront    1        run      running   4s ago     3s ago
27ee8ec2  728069bc  webfront    1        run      running   36s ago    6s ago
38fe5bf1  a7db9bf6  webfront    1        run      running   36s ago    6s ago
5a64d71d  728069bc  webfront    0        run      running   8m26s ago  7m55s ago
b6daca57  a7db9bf6  webfront    0        stop     complete  8m26s ago  36s ago
d36d8e82  728069bc  webfront    0        stop     complete  8m26s ago  36s ago
dd254879  728069bc  webfront    0        stop     complete  8m26s ago  4s ago
f86dba63  728069bc  webfront    0        stop     complete  8m26s ago  4s ago
ffd4a89a  a7db9bf6  webfront    0        run      running   8m26s ago  7m56s ago

ID        Node ID   Task Group  Version  Desired  Status    Created    Modified
08768155  728069bc  webfront    1        run      running   8s ago     7s ago
d6077e56  a7db9bf6  webfront    1        run      running   8s ago     7s ago
61a2ad19  728069bc  webfront    1        run      running   40s ago    9s ago
929c0ce9  728069bc  webfront    1        run      running   40s ago    9s ago
27ee8ec2  728069bc  webfront    1        run      running   1m12s ago  42s ago
38fe5bf1  a7db9bf6  webfront    1        run      running   1m12s ago  42s ago
5a64d71d  728069bc  webfront    0        stop     complete  9m2s ago   8s ago
b6daca57  a7db9bf6  webfront    0        stop     complete  9m2s ago   1m12s ago
d36d8e82  728069bc  webfront    0        stop     complete  9m2s ago   1m12s ago
dd254879  728069bc  webfront    0        stop     complete  9m2s ago   40s ago
f86dba63  728069bc  webfront    0        stop     complete  9m2s ago   40s ago
ffd4a89a  a7db9bf6  webfront    0        stop     complete  9m2s ago   8s ago

Blue/Green & Canary Deployments

Blue/Green

使用該部屬方式,設定只要加入canary參數並設定數量。
全部的task數是6,canary設定6執行,查看狀態。

update {
      max_parallel     = 2
      canny            = 6
      min_healthy_time = "30s"
      healthy_deadline = "2m"
      auto_revert = true
    }

執行後可以看到狀態多出了6個新的task
前6個是新版本的Version=4,之後的6個是版本較舊Version=ˇ

Allocations
ID        Node ID   Task Group  Version  Desired  Status    Created     Modified
fd42efc7  728069bc  webfront    4        run      running   7s ago      7s ago
08dae65a  a7db9bf6  webfront    4        run      running   7s ago      6s ago
7dd93085  a7db9bf6  webfront    4        run      running   7s ago      7s ago
684f201d  728069bc  webfront    4        run      running   7s ago      7s ago
4b957e51  a7db9bf6  webfront    4        run      running   7s ago      7s ago
e3018c22  728069bc  webfront    4        run      running   7s ago      7s ago
d6077e56  a7db9bf6  webfront    3        run      running   21m57s ago  1m9s ago
08768155  728069bc  webfront    3        run      running   21m57s ago  1m9s ago
929c0ce9  728069bc  webfront    3        run      running   22m29s ago  1m9s ago
61a2ad19  728069bc  webfront    3        run      running   22m29s ago  1m9s ago
38fe5bf1  a7db9bf6  webfront    3        run      running   23m1s ago   1m9s ago
27ee8ec2  728069bc  webfront    3        run      running   23m1s ago   1m9s ago

那麼有確定要使用新版本,可以執行deployment promote加上ID
ID在察看job status時,會有該資訊。

$ nomad deployment promote 23259054

不要使用新版本,可以執行deployment fail`加上ID

$ nomad deployment fail 23259054

Canary

Canary的部屬方式也跟Blue/Green相同,可以自行練習調整canary參數試試看。

除了使用CLI外,可以透過Web UI來看看Web UI有支援甚麼
https://ithelp.ithome.com.tw/upload/images/20200920/20129223iWv6VPmZvp.png


上一篇
Day 19. Nomad's Variable and Template
下一篇
Day 21. Nomad mount volume
系列文
Hashicorp 0 到 0.00330
圖片
  直播研討會
圖片
{{ item.channelVendor }} {{ item.webinarstarted }} |
{{ formatDate(item.duration) }}
直播中

尚未有邦友留言

立即登入留言