接下來這一個章節，焦點還是會在filebeat上，通常在收集log，並不是所有資料都需要收集到Elasticsearch，而今日主題將會是如何使用正式表達式，來達收集到真正所需要的資料。

如何使用正則表達式

在filebeat提供下列二種方式來過濾資料

• exclude_lines (支援正則表達式)
  在輸入資料中排除符合正則表達式的那些行數。
• include_lines (支援正則表達式)
  在輸入資料中符合正則表達式的那些行數。

註: include_lines 執行完成後，才會再執行 exclude_lines

測試用的資料

"GET /navigation.php HTTP/1.1" 200 22486 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.58 Safari/537.36"
"POST /navigation.php HTTP/1.1" 200 22486 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.58 Safari/537.36"
"PUT /navigation.php HTTP/1.1" 200 22486 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.58 Safari/537.36"
"GET /navigation.php HTTP/1.1" 400 22486 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.58 Safari/537.36"
"POST /navigation.php HTTP/1.1" 400 22486 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.58 Safari/537.36"
"GET /navigation.php HTTP/1.1" 200 22486 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.58 Safari/537.36"

範例1

排除所有開頭為GET的資料

#filebeat.yml
filebeat.inputs:

# 設定要抓取log的路徑
- type: filestream
  enabled: true
  # 排除開頭資料為 GET
  exclude_lines: ["^\"GET"]
  paths:
    - ./mylog.log

# 設定kibana
setup.kibana:
  host: "localhost:5601"

# 設定elasticsearch
output.elasticsearch:
  hosts: ["localhost:9200"]
  #設定索引名稱
  index: "mylog-%{+yyyy.MM.dd}"

# 設定索引樣板資訊
setup.template.name: "mylog"
setup.template.pattern: "mylog-*"
setup.ilm.enabled: false

elk得到資料如下:

Sep 8, 2021 @ 13:33:03.284	"PUT /navigation.php HTTP/1.1" 200 22486 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.58 Safari/537.36"

Sep 8, 2021 @ 13:33:03.284	"POST /navigation.php HTTP/1.1" 400 22486 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.58 Safari/537.36"

Sep 8, 2021 @ 13:33:00.646	"POST /navigation.php HTTP/1.1" 200 22486 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.58 Safari/537.36"

範例2

取出所有包含GET的資料

#filebeat.yml
filebeat.inputs:

# 設定要抓取log的路徑
- type: filestream
  enabled: true
  # 資料有包含 GET
  include_lines: ["GET"]
  paths:
    - ./mylog.log

elk得到資料如下:

Sep 8, 2021 @ 13:46:13.559	"GET /navigation.php HTTP/1.1" 400 22486 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.58 Safari/537.36"

Sep 8, 2021 @ 13:46:13.559	"GET /navigation.php HTTP/1.1" 200 22486 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.58 Safari/537.36"

Sep 8, 2021 @ 13:46:13.558	"GET /navigation.php HTTP/1.1" 200 22486 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.58 Safari/537.36"

範例3

取出所有包含GET的資料，並排除400

#filebeat.yml
filebeat.inputs:

# 設定要抓取log的路徑
- type: filestream
  enabled: true
  # 排除開頭資料為 400
  exclude_lines: ["400"]
  # 資料有包含 GET
  include_lines: ["GET"]
  paths:
    - ./mylog.log

elk得到資料如下:

Sep 8, 2021 @ 13:50:02.269	"GET /navigation.php HTTP/1.1" 200 22486 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.58 Safari/537.36"

Sep 8, 2021 @ 13:49:59.636	"GET /navigation.php HTTP/1.1" 200 22486 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.58 Safari/537.36"