【Day 29】由淺入深來探討Elasticsearch - Index lifecycle management API操作 - iT 邦幫忙::一起幫忙解決難題，拯救 IT 人的一天

2023 iThome 鐵人賽

DAY 29

Software Development

由淺入深來探討Elasticsearch，從基礎語法到底層相關原理系列第 29 篇

【Day 29】由淺入深來探討Elasticsearch - Index lifecycle management API操作

15th鐵人賽 #elasticsearch #ilm policy api

blank

2023-10-01 20:03:25

444 瀏覽

分享至

昨天我們介紹了ILM的原理
那在實際操作方面，我們要如何使ILM透用至index上？

整體流程

創建ILM policy

PUT /_ilm/policy/test_policy
{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "1m",
        "actions": {
          "set_priority": {
            "priority": 100
          },
          "rollover": {
            "max_age": "10s",
            "max_docs": 2
          }
        }
      },
      "warm": {
        "min_age": "1m",
        "actions": {
          "set_priority": {
            "priority": 50
          },
          "allocate": {
            "number_of_replicas": 1,
            "include": {
              "_name": "",
              "_ip": ""
            },
            "exclude": {
              "_name": "",
              "_ip": ""
            },
            "require": {
              "_name": "",
              "_ip": ""
            }
          },
          "shrink": {
            "number_of_shards": 1
          },
          "forcemerge": {
            "max_num_segments": 1
          }
        }
      },
      "cold": {
        "min_age": "1d",
        "actions": {
          "set_priority": {
            "priority": 0
          },
          "allocate": {
            "number_of_replicas": 1,
            "include": {
              "_name": "",
              "_ip": ""
            },
            "exclude": {
              "_name": "",
              "_ip": ""
            },
            "require": {
              "_name": "",
              "_ip": ""
            }
          }
        }
      },
      "delete": {
        "min_age": "1d",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}

在action中不一定要定義所有的phase，ES會跳過沒有定義的phase
參數解釋：
- set_priority：設定越高，node重啟時會優先啟動。所以通常hot階段會設最高，cold最低
- rollover內的條件滿足1至多個時進到下一階段
  - number_of_replicas：可選項，設定replica shard的數量
  - include, exclude, require：可選項，設置相對應的nodes，達到分配或是避免分配到特定的node
- shrink：
  - 將source index設置為read-only狀態，並將轉移進新index同時限縮primary shard的數量
  - 如果在hot phase設置，一定要同時設置rollover action，不然會被拒絕
  - number_of_shards：可選項，將縮小至的primary shard數，需為source index primary shards的因數
- forcemerge：藉由設置max_num_segments數來控制merge後的segment數量
- delete：移除index
  - delete_searchable_snapshot：可選項，默認為true，刪除在之前phase創建的searchable snapshot

建立index template

PUT _template/test_template
{
  "index_patterns": ["test-*"],
  "settings": {
    "index.lifecycle.name": "test_policy",
    "index.lifecycle.rollover_alias": "test-index" 
  }
}

index.lifecycle.name：輸入想要匹配的ILM policy name
index.lifecycle.rollover_alias：設置要滾動到target index的別名

創建相符的索引後，透過ILM explain去看index有沒有受到ILM policy管理

// 為了方便示範 將poll_interval設為1秒，預設為10分鐘
PUT /_cluster/settings
{
  "transient": {
    "indices.lifecycle.poll_interval": "1s" 
  }
}

//再創一個policy，只有hot跟delete階段
PUT /_ilm/policy/test_policy
{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "set_priority": {
            "priority": 100
          },
          "rollover": {
            "max_docs": 2
          }
        }
      },
      "delete": {
        "min_age": "30s",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}

// 創建索引
PUT test-index-1
{
  "aliases": { // 配置好alias來套用index template
    "test-index": {}
  }
}

// 插入文檔
PUT /test-index/_doc/1
{
  "text": "test ILM policy"
}

// 查看當前index的ilm狀態
GET test-index/_ilm/explain

{
  "indices": {
    "test-index-1": {
      "index": "test-index-1",
      "managed": true,
      "policy": "test_policy",
      "index_creation_date_millis": 1696160225710,
      "time_since_index_creation": "33.58s",
      "lifecycle_date_millis": 1696160225710,
      "age": "33.58s",
      "phase": "hot",
      "phase_time_millis": 1696160225837,
      "action": "rollover",
      "action_time_millis": 1696160225837,
      "step": "check-rollover-ready",
      "step_time_millis": 1696160225837,
      "phase_execution": {
        "policy": "test_policy",
        "phase_definition": {
          "min_age": "0ms",
          "actions": {
            "rollover": {
              "max_docs": 2
            },
            "set_priority": {
              "priority": 100
            }
          }
        },
        "version": 1,
        "modified_date_in_millis": 1696159658831
      }
    }
  }
}

因為我設置max_doc是2，接著我再輸入一個doc後使用explain API看看

{
  "indices": {
    "test-index-000002": {
      "index": "test-index-000002",
      "managed": true,
      "policy": "test_policy",
      "index_creation_date_millis": 1696160341301,
      "time_since_index_creation": "1.72s",
      "lifecycle_date_millis": 1696160341301,
      "age": "1.72s",
      "phase": "hot",
      "phase_time_millis": 1696160341306,
      "action": "rollover",
      "action_time_millis": 1696160341506,
      "step": "check-rollover-ready",
      "step_time_millis": 1696160341506,
      "phase_execution": {
        "policy": "test_policy",
        "phase_definition": {
          "min_age": "0ms",
          "actions": {
            "rollover": {
              "max_docs": 2
            },
            "set_priority": {
              "priority": 100
            }
          }
        },
        "version": 1,
        "modified_date_in_millis": 1696159658831
      }
    }
  }
}

可以看到ES自己創建test-index-000002，因為test-index-1已經滿足rollover的條件
4. 如果有已經建立好的index還沒有設置別名，沒辦法受到index template的管理可以設置

POST /_aliases
{
  "actions": [
    {
      "add": {
        "index": "index_name",
        "alias": "alias_name"
      }
    }
  ]
}

刪除policy，但是如果有index此時透用該policy會無法操作

DELETE _ilm/policy/test_policy

手動管理index policy相關API

手動phase轉換

POST _ilm/move/my-index-000001
{
  "current_step": { 
    "phase": "new",
    "action": "complete",
    "name": "complete"
  },
  "next_step": { 
    "phase": "warm",
    "action": "forcemerge",
    "name": "forcemerge"
  }
}

移除特定index的policy

POST my-index-000001/_ilm/remove

營運管理相關API

查看當前ILM的狀態，看叢集是否有實施ILM policy
- 返回operation_mode的狀態
- 分別有三種：STARTED, STOPPING, STOPPED

GET /_ilm/status

停止所有ILM管理操作並停止 ILM 外掛程式
- 當在叢集上執行維護並需要停止ILM 對索引執行任何操作時使用
- 一旦使用API，ILM policy會終止，但其他插件可能會繼續運行，直到正在進行的操作完成並且插件可以安全停止。可以用GET /_ilm/status來一起操作

POST /_ilm/stop

在cluster形成時，ILM會自動啟動，用到start只有因為之前有使用stop ILM API

POST /_ilm/start

查看特定index或data stream當前ILM狀態

GET <target>/_ilm/explain

透過ILM API以及設置相關的index template
我們能夠更方便的管理大量會隨時間增加的index或是data stream
例如當filebeat持續不斷匯入資料進ES時
因為filebeat匯入時，index的名稱會有特定前綴
我們可以透過其特性為其設置index template
並且創建規劃好的ILM policy
讓我們更方便的能處理大量的log data，並且時間較為久遠的資料policy也能自動幫我們轉移成更省資源的形式並保有特定效率

參考資料
ilm API：
https://www.elastic.co/guide/en/elasticsearch/reference/7.9/index-lifecycle-management-api.html?
shrink：
https://www.elastic.co/guide/en/elasticsearch/reference/current/ilm-shrink.html