EP28 - 使用 Container Insights 監控 EKS 上的容器，並整合 Grafana 作為儀表板

13th鐵人賽 devops cloudwatch grafana terraform

mark_mew

2021-10-10 16:01:31

1889 瀏覽

在 EP25 和 EP26，
我們為 EKS 配置了 Grafana 和 Loki，
讓我們可以透過儀表板查看 Log，
~~雖然範例 AP 根本沒有設定 Log~~，
今天我們將繼續進行，
在 EKS 設定 Container Insights，
這樣我們就可以查看容器的使用狀況。

Container Insights

什麼是 Container Insights

CloudWatch 為 AWS 的監控工具
而 Container Insights（譯作容器洞見）是 CloudWatch 的一個 feature
CloudWatch 會自動收集許多資源的指標
例如 CPU、記憶體、磁碟和網路

容器洞見還提供診斷資訊
例如容器重新啟動故障
協助您快速隔離和解決這些故障

使用 Container Insights 的好處
在於你可以串接 AWS CloudWatch 的告警機制
並且 Grafana 也可以整合 AWS CloudWatch
可以使用 Grafana 統一做為內部系統監控的儀表板

為 EC2 Profile 添加權限

resource "aws_iam_role_policy_attachment" "ec2_profile_cloudwatch_agent_service" {
    policy_arn = "arn:aws:iam::aws:policy/CloudWatchAgentServerPolicy"
    role       = aws_iam_instance_profile.ec2_profile.name
}

快速入門使用 CloudWatch agent 和 Fluentd

記得將 aws-stage-cluster 替換成你的 cluster 名稱
將 ap-northwast-1 替換成 cluster 所屬區域

curl https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/k8s-deployment-manifest-templates/deployment-mode/daemonset/container-insights-monitoring/quickstart/cwagent-fluentd-quickstart.yaml | sed "s/{{cluster_name}}/aws-stage-cluster/;s/{{region_name}}/ap-northeast-1/" | kubectl apply -f -

設定 CloudWatch agent

建立新的資料夾

mkdir /vagrant_data/project/k8s-resources
cd /vagrant_data/project/k8s-resources

建立 namespace

kubectl apply -f https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/k8s-deployment-manifest-templates/deployment-mode/daemonset/container-insights-monitoring/cloudwatch-namespace.yaml

建立 serviceaccount

kubectl apply -f https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/k8s-deployment-manifest-templates/deployment-mode/daemonset/container-insights-monitoring/cwagent/cwagent-serviceaccount.yaml

為 CloudWatch agent建立 ConfigMap

curl -O https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/k8s-deployment-manifest-templates/deployment-mode/daemonset/container-insights-monitoring/cwagent/cwagent-configmap.yaml

為 CloudWatch 代理程式建立 ConfigMap

編輯已下載的 YAML 檔案
在 kubernetes 區塊中的 cluster_name
將 {{cluster-name}} 取代為自己的叢集名稱

在 AWS Cloud Console 中檢查設定

登入後會自動幫你建立四個日誌群組

Container Insights 也可以根據 Pods、Namespace
去查看資源的使用狀況

整合 Grafana

為 Grafana 建立 IAM 使用者

stage/main.tf

resource "aws_iam_user" "grafana" {
    name = "Grafana"
    path = "/"
    
    tags = {
        Name    = "Grafana"
        Usage   = "Grafana"
        Creator = "Terraform"
    }
}

resource "aws_iam_access_key" "grafana" {
    user = aws_iam_user.grafana.name
}

resource "aws_iam_user_policy" "grafana_cloudwatch" {
    name = "GrafanaCloudWatchLimited"
    user = aws_iam_user.grafana.name
    
    policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": [
        "cloudwatch:DescribeAlarmsForMetric",
        "cloudwatch:DescribeAlarmHistory",
        "cloudwatch:DescribeAlarms",
        "cloudwatch:ListMetrics",
        "cloudwatch:GetMetricStatistics",
        "cloudwatch:GetMetricData"
      ],
      "Effect": "Allow",
      "Resource": "*"
    },
    {
      "Action": [
        "logs:DescribeLogGroups",
        "logs:GetLogGroupFields",
        "logs:StartQuery",
        "logs:StopQuery",
        "logs:GetQueryResults",
        "logs:GetLogEvents"
      ],
      "Effect": "Allow",
      "Resource": "*"
    },
    {
      "Action": [
        "ec2:DescribeTags",
        "ec2:DescribeInstances",
        "ec2:DescribeRegions"
      ],
      "Effect": "Allow",
      "Resource": "*"
    },
    {
      "Action": "tag:GetResources",
      "Effect": "Allow",
      "Resource": "*"
    }
  ]
}
EOF
}

stage/outputs

output "grafana_secret_token" {
    description = "Decrypt access secret key command"
    value       = aws_iam_access_key.grafana.secret
    sensitive   = true
}