AWS Kinesis 模組實作

本篇是實作常用的 AWS Kinesis Stream 服務之 Terraform 模組，並且會使用到 YAML 資料結構來定義模組的內容，完整的專案程式碼分享在我的 Github 上。

1. 先定義整個專案檔案結構設定檔 ./configs/stream/streams.yaml 與 模組 my_cloudfront 的放置位置 modules/my_kinesis_stream:

├── configs
│   ├── cloudfront
│   │   └── distributions.yaml
│   ├── cloudwatch
│   │   └── loggroups.yaml
│   ├── iam
│   │   ├── assume_role_policies
│   │   ├── policies
│   │   ├── role_policies
│   │   ├── user_policies
│   │   └── iam.yaml
│   ├── kinesis
│   │   └── streams.yaml
│   ├── s3
│   │   ├── policies
│   │   └── s3.yaml
│   ├── subnet
│   │   └── my-subnets.yaml
│   └── vpc
│       └── my-vpcs.yaml
├── example.tfvars
├── locals.tf
├── main.tf
├── modules
│   ├── my_cloudfront
│   ├── my_cloudwatch
│   ├── my_eips
│   ├── my_eventbridge
│   ├── my_iam
│   ├── my_igw
│   ├── my_instances
│   ├── my_kinesis_stream
│   │   ├── kinesis_stream.tf
│   │   ├── outputs.tf
│   │   ├── provider.tf
│   │   └── variables.tf
│   ├── my_nacls
│   ├── my_route_tables
│   ├── my_s3
│   ├── my_subnets
│   └── my_vpc
└── variables.tf

2. 撰寫 ./configs/stream/stream.yaml 內容來定義 AWS Kinesis Stream 需要用建立的資源：

streams:
  - name: "<STREAM_NAME>"
    encryption_type: <"NONE" or >
    retention_period: <RETENTION_PERIOD>
    shard_count: <SHARD_COUNT>
    enforce_consumer_deletion: <true or false>

3. 撰寫 my_stream 模組:

• ./modules/my_kinesis_stream/outputs.tf:

output "streams" {
  value = aws_kinesis_stream.streams
}

• ./modules/my_kinesis_stream/provider.tf:

provider "aws" {
    region  = var.aws_region
    profile = var.aws_profile
}

• ./modules/my_kinesis_stream/variables.tf:

variable "aws_region" {
  description = "AWS region"
  default     = "ap-northeast-1"
}

variable "aws_profile" {
  description = "AWS profile"
  default     = ""
}

variable "project_name" {
  type    = string
  description = "Project name"
  default = ""
}

variable "department_name" {
  type        = string
  description = "Department name"
  default     = "SRE"
}

variable "stream_path" {
  type        = string
  description = "The path of streams"
  default     = ""
}

• ./modules/my_kinesis_stream/kinesis_stream.tf:
  利用 for_each 來迭代 locals.streams 以 name 為 key 值建立 map 物件

resource "aws_kinesis_stream" "streams" {
  for_each = { for r in local.streams : r.name => r }

  encryption_type           = each.value.encryption_type
  kms_key_id                = lookup(each.value, "kms_key_id", null)
  name                      = each.value.name
  retention_period          = each.value.retention_period
  shard_count               = each.value.shard_count
  enforce_consumer_deletion = lookup(each.value, "enforce_consumer_deletion", null)
}

4. 撰寫專案相關程式

• example.tfvars:

aws_region="ap-northeast-1"
aws_profile="<YOUR_PROFILE>"
project_name="example"
department_name="SRE"

• main.tf:

terraform {
  required_providers {
    aws = {
      version = "5.15.0"
    }
  }

  backend "s3" {
    bucket                  = "<YOUR_S3_BUCKET_NAME>"
    dynamodb_table          = "<YOUR_DYNAMODB_TABLE_NAME>"
    key                     = "terraform.tfstate"
    region                  = "ap-northeast-1"
    shared_credentials_file = "~/.aws/config"
    profile                 = "<YOUR_PROFILE>"
  }
}

其他模組省略...

# kinesis stream
module "kinesis_stream" {
  aws_profile     = var.aws_profile
  aws_region      = var.aws_region
  department_name = var.department_name
  project_name    = var.project_name
  stream_path     = "./configs/kinesis/streams.yaml"

  source = "./modules/my_kinesis_stream"
}

Terraform 執行計畫

1. 嘗試建立一個 Kinesis Stream 來測試一下模組，編輯 ./configs/kinesis/streams.yaml：

streams:
  - name: my-stream
    encryption_type: NONE
    retention_period: 24
    shard_count: 1
    enforce_consumer_deletion: false

2. 於專案目錄下執行 terraform init && terraform plan --out .plan -var-file=example.tfvars 來確認一下結果:

Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

...

  # module.kinesis_stream.aws_kinesis_stream.streams["my-stream"] will be created
  + resource "aws_kinesis_stream" "streams" {
      + arn                       = (known after apply)
      + encryption_type           = "NONE"
      + enforce_consumer_deletion = false
      + id                        = (known after apply)
      + name                      = "my-stream"
      + retention_period          = 24
      + shard_count               = 1
      + tags_all                  = (known after apply)
    }

...


Plan: 48 to add, 0 to change, 0 to destroy.
─────────────────────────────────────────────────────────────────────────────

Saved the plan to: .plan

To perform exactly these actions, run the following command to apply:
    terraform apply ".plan"
Releasing state lock. This may take a few moments...

下一篇文章將會展示實作 AWS KMS 之 Terraform 模組。