Day19 Prometheus(四) kube-state-metrics

2023 iThome 鐵人賽

DAY 19

自我挑戰組

地端建置大雜燴-下從網路線上到Grafana的建置之路系列第 19 篇

15th鐵人賽 it prometheus k8s monitoring

ORa_

團隊Bare Metal People

2023-10-04 22:27:31

645 瀏覽

分享至

今天要來在 kubeadm 上面監測 cluster 的運作狀態
這邊會使用 kube-state-metrics 來作為 Exporter

Deploy kube-state-metrics

kube-state-metrics 要安裝在 kubenetes 裡面
就先來找設定檔下來

wget -O kube-state-metrics.tar.gz https://github.com/kubernetes/kube-state-metrics/archive/refs/tags/v2.10.0.tar.gz

解壓縮剛剛下載的檔案

tar -zxvf kube-state-metrics.tar.gz

應該會解壓縮出 kube-state-metrics-2.10.0 (如果版本一樣的話)
進到 kube-state-metrics-2.10.0/examples/standard/ 看看有哪些 YAML 吧

cd kube-state-metrics-2.10.0/examples/standard/
ls

cluster-role-binding.yaml
cluster-role.yaml
service-account.yaml
service.yaml
deployment.yaml
總共有這些檔案
這邊要修改一下 service.yaml 因為我的環境的 Prometheus 是來自 Cluster 外部會比較麻煩
~~早知道還是用 Container 丟進 Kubeadm 好~~

apiVersion: v1
kind: Service
metadata:
  labels:
    app.kubernetes.io/component: exporter
    app.kubernetes.io/name: kube-state-metrics
    app.kubernetes.io/version: 2.10.0
  name: kube-state-metrics
  namespace: kube-system
spec:
  type: NodePort
  ports:
  - name: http-metrics
    port: 8080
    targetPort: 8080
  - name: telemetry
    port: 8081
    targetPort: 8081
  selector:
    app.kubernetes.io/name: kube-state-metrics

這邊把 service type 指定為 nodeport 向外開放 Port 給 Prometheus 打進來
正常狀況要配合 Network Policy 來做存取限制要不然整個 Cluster 的資訊被隨便看光光
~~好色喔~~

接著來部署 YAML 上去 K8s 吧

cd kube-state-metrics-2.10.0/examples/standard/
kubectl apply -f ./

root@lke-main:~/kube-state-metrics-2.10.0/examples/standard# kubectl apply -f ./
clusterrolebinding.rbac.authorization.k8s.io/kube-state-metrics created
clusterrole.rbac.authorization.k8s.io/kube-state-metrics created
deployment.apps/kube-state-metrics created
serviceaccount/kube-state-metrics created
service/kube-state-metrics created

接著來抓 Service 的 Port 準備給 Prometheus 使用

kubectl get service kube-state-metrics --namespace kube-system

root@lke-main:~/kube-state-metrics-2.10.0/examples/standard# kubectl get service kube-state-metrics --namespace kube-system
NAME                 TYPE       CLUSTER-IP       EXTERNAL-IP   PORT(S)                         AGE
kube-state-metrics   NodePort   10.105.242.119   <none>        8080:31337/TCP,8081:31402/TCP   3m19s

可以看到 31337 Port 被丟到 8080 上 31402 被丟到 8081 Port 上
在設定之前敲敲看 kube-state-metrics 看看

root@Prometheus:/etc/prometheus# curl lke-main.ironman.test:31337/metric
<html lang="en">
  <head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>kube-state-metrics</title>
    <style>body {
  font-family: -apple-system,BlinkMacSystemFont,Segoe UI,Roboto,Helvetica Neue,Arial,Noto Sans,Liberation Sans,sans-serif,Apple Color Emoji,Segoe UI Emoji,Segoe UI Symbol,Noto Color Emoji;
  margin: 0;
}
header {
  background-color: #e6522c;
  color: #fff;
  font-size: 1rem;
  padding: 1rem;
}
main {
  padding: 1rem;
}
label {
  display: inline-block;
  width: 0.5em;
}
</style>
  </head>
  <body>
    <header>
      <h1>kube-state-metrics</h1>
    </header>
    <main>
      <h2>Metrics for Kubernetes' state</h2>
      <div>Version: (version=v2.10.0, branch=, revision=unknown)</div>
      <div>
        <ul>
          <li><a href="/metrics">Metrics</a></li>
          <li><a href="/healthz">Healthz</a></li>
        </ul>
      </div>
    </main>
  </body>
</html>

看起來活得比這幾年的我還好太棒了!
那就來設定 prometheus

vim /etc/prometheus/prometheus.yml

增加 kube-state-metrics 的 Job 上去

- job_name: "kube-state-metrics"
  static_configs:
    - targets: [ 'lke-main.ironman.test:31337' ]
- job_name: "kube-state-metrics-telemetry"
  static_configs:
    - targets: [ 'lke-main.ironman.test:31402' ]

設定完就重啟看看吧