Fluent bit回顧
Log Agent - Fluent Bit 簡介
Log Agent - Fluent Bit 安裝與常見架構模式
Log Agent - Fluent Bit Service配置與內建 API
Log Agent - Fluent Bit Input元件 與 Tail淺談
Log Agent - Fluent Bit Parser元件

Multiline Parsing

昨天介紹的Regex Parser其實只適用於單行的Log資料.
因為Tail是讀取一行就往Parser送.

所以如果Log資料本來就是Multiline多行的.
就需要用Multiline Parser, 對Regex Parser做點處理

內建的Multiline Parsers

針對有些環境所產出的Log, Fluent bit有內建好幾個Multiline Parser
就不必自己刻寫了

• Docker
• CRI
• Go
• Pythn
• Java

但我們也能自定義Multiline Parser
上一篇提到的Parser, 它都是[Parser]這樣作為section name
但Multiline Parser則是[MULTILINE_PARSER]作為section name
然後Parser與MULTILINE_PARSER都建議別直接配置在fluent-bit.conf
需要另外拉一個parsers.conf或者是parsers_multiline.conf
在fluent-bit.conf做import

[SERVICE]
    flush 1
    Daemon off
    log_level info
    parsers_file parsers.conf
    parsers_file parsers_multiline.conf

然後MULTILINE_PARSER需要設定幾個properties

• Name
  • 就Multiline Parser的name
• type
  • 設定成regex
• rule
  • 用rule來寫regex, 使得讓multiline parser知道第一行的樣貌跟讀到怎樣的樣貌是結束
  • 下面範例, start_state 就是起始狀態的名字 只要符合其regex pattern的就是多行Log的第一行
  • start_state匹配到第一行後, 就看有沒有next state, 這裡指定下一個state是cont
  • 就繼續讀下一行, 判斷是不是匹配start_state, 不是就拿現在狀態的cont的regex patern來繼續匹配
  • 這樣直到某一行讀到, 它是匹配start_state的, 就是下一段的多行了

# rules   |   state name   | regex pattern         | next state name
# --------|----------------|----------------------------------------
    rule     "start_state"   "/(Dec \d+ \d+\:\d+\:\d+)(.*)/"  "cont"
    rule     "cont"          "/^\s+at.*/"                     "cont"

來個範例
fluent-bit.conf
這裡input的multiline.parser,
我們測試內建的Go multiline parser和自定義的parser
多個parser用,做分隔就好

[INPUT]    
    Name        tail    
    Path        /var/log/demo/demo.log
    read_from_head   true
    multiline.parser      multiline-regex-test, go

parsers_multiline.conf

[MULTILINE_PARSER]
    name          multiline-regex-test
    type          regex
    rule      "start_state"   "/(Dec \d+ \d+\:\d+\:\d+)(.*)/"  "cont"
    rule      "cont"          "/^\s+at.*/"                     "cont"

測試log

Dec 14 06:41:08 Exception in thread "main" java.lang.RuntimeException: Something has gone wrong, aborting!
    at com.myproject.module.MyProject.badMethod(MyProject.java:22)
    at com.myproject.module.MyProject.oneMoreMethod(MyProject.java:18)
    at com.myproject.module.MyProject.anotherMethod(MyProject.java:14)
    at com.myproject.module.MyProject.someMethod(MyProject.java:10)
    at com.myproject.module.MyProject.main(MyProject.java:6)

Dec 14 06:41:09 Exception in thread "main" java.lang.RuntimeException: Something has gone wrong, aborting!
Dec 14 06:41:10 Exception in thread "main" java.lang.RuntimeException: Something has gone wrong, aborting!
Dec 14 06:41:11 Exception in thread "main" java.lang.RuntimeException: Something has gone wrong, aborting!
    at com.myproject.module.MyProject.badMethod(MyProject.java:22)
    at com.myproject.module.MyProject.oneMoreMethod(MyProject.java:18)
    at com.myproject.module.MyProject.anotherMethod(MyProject.java:14)
    at com.myproject.module.MyProject.someMethod(MyProject.java:10)
    at com.myproject.module.MyProject.main(MyProject.java:6)
    
panic: my panic

goroutine 4 [running]:
panic(0x45cb40, 0x47ad70)
  /usr/local/go/src/runtime/panic.go:542 +0x46c fp=0xc42003f7b8 sp=0xc42003f710 pc=0x422f7c
main.main.func1(0xc420024120)
  foo.go:6 +0x39 fp=0xc42003f7d8 sp=0xc42003f7b8 pc=0x451339
runtime.goexit()
  /usr/local/go/src/runtime/asm_amd64.s:2337 +0x1 fp=0xc42003f7e0 sp=0xc42003f7d8 pc=0x44b4d1
created by main.main
  foo.go:5 +0x58
panic: my panic

goroutine 4 [running]:
panic(0x45cb40, 0x47ad70)
  /usr/local/go/src/runtime/panic.go:542 +0x46c fp=0xc42003f7b8 sp=0xc42003f710 pc=0x422f7c
main.main.func1(0xc420024120)
  foo.go:6 +0x39 fp=0xc42003f7d8 sp=0xc42003f7b8 pc=0x451339
runtime.goexit()
  /usr/local/go/src/runtime/asm_amd64.s:2337 +0x1 fp=0xc42003f7e0 sp=0xc42003f7d8 pc=0x44b4d1
created by main.main
  foo.go:5 +0x58

output
可以看到[0]這結構化日誌, 是多行的, 這麼多行才整理成一筆結構化日誌做Output
而不會笨笨的一行就輸出一筆出去
[1]、[2]、[3]則是來玩看看, 那兩個state狀態的匹配順序與切換
[1]那行匹配到了start_state,
接著下一行[2], 又匹配到了start_state, 就表示[1]能輸出了
接著下一行[3],又匹配到了start_state, 就表示[3]能輸出了

再來的一行是Go的, 就怎樣也不匹配start_state和cont這兩個state, 就換注入的下一個parser解析看看

Go的panic log與自定義的Java log都被multiline parser給正確解析

fluentd_1  | [0] tail.0: [1634142611.283200049, {"log"=>"Dec 14 06:41:08 Exception in thread "main" java.lang.RuntimeException: Something has gone wrong, aborting!
fluentd_1  |     at com.myproject.module.MyProject.badMethod(MyProject.java:22)
fluentd_1  |     at com.myproject.module.MyProject.oneMoreMethod(MyProject.java:18)
fluentd_1  |     at com.myproject.module.MyProject.anotherMethod(MyProject.java:14)
fluentd_1  |     at com.myproject.module.MyProject.someMethod(MyProject.java:10)
fluentd_1  |     at com.myproject.module.MyProject.main(MyProject.java:6)
fluentd_1  | "}]
fluentd_1  | [1] tail.0: [1634142611.283256734, {"log"=>"Dec 14 06:41:09 Exception in thread "main" java.lang.RuntimeException: Something has gone wrong, aborting!
fluentd_1  | "}]
fluentd_1  | [2] tail.0: [1634142611.283260341, {"log"=>"Dec 14 06:41:10 Exception in thread "main" java.lang.RuntimeException: Something has gone wrong, aborting!
fluentd_1  | "}]
fluentd_1  | [3] tail.0: [1634142611.283263447, {"log"=>"Dec 14 06:41:11 Exception in thread "main" java.lang.RuntimeException: Something has gone wrong, aborting!
fluentd_1  |     at com.myproject.module.MyProject.badMethod(MyProject.java:22)
fluentd_1  |     at com.myproject.module.MyProject.oneMoreMethod(MyProject.java:18)
fluentd_1  |     at com.myproject.module.MyProject.anotherMethod(MyProject.java:14)
fluentd_1  |     at com.myproject.module.MyProject.someMethod(MyProject.java:10)
fluentd_1  |     at com.myproject.module.MyProject.main(MyProject.java:6)
fluentd_1  | "}]
fluentd_1  | [4] tail.0: [1634143058.181373085, {"log"=>"panic: my panic
fluentd_1  | 
fluentd_1  | goroutine 4 [running]:
fluentd_1  | panic(0x45cb40, 0x47ad70)
fluentd_1  |   /usr/local/go/src/runtime/panic.go:542 +0x46c fp=0xc42003f7b8 sp=0xc42003f710 pc=0x422f7c
fluentd_1  | main.main.func1(0xc420024120)
fluentd_1  |   foo.go:6 +0x39 fp=0xc42003f7d8 sp=0xc42003f7b8 pc=0x451339
fluentd_1  | runtime.goexit()
fluentd_1  |   /usr/local/go/src/runtime/asm_amd64.s:2337 +0x1 fp=0xc42003f7e0 sp=0xc42003f7d8 pc=0x44b4d1
fluentd_1  | created by main.main
fluentd_1  |   foo.go:5 +0x58
fluentd_1  | "}]

其實也能在Filter再做multiline.parser的設定,
讓Input單純的只做監聽與撈取資料