昨天我們設定 Promtail 把 Log 集中到 Loki 上
但是我們昨天的設定是

scrape_configs:
- job_name: system
  static_configs:
  - targets:
      - localhost
    labels:
      job: varlogs
      __path__: /var/log/*log

也就是說我們把 /var/log/ 目錄中所有後墜有 log 的檔案全都都會送給 Loki 上
不過昨天的 Log 量和 Label 看起來是有問題的
所以今天就來解決這些問題吧

探索 Linux Log

先來看看 /var/log 裡面會有那些 Log 吧

cd /var/log
ls 

? 好像哪裡怪怪的 以前的那一堆 auth.log secure.log syslog.log 勒
還有一個神奇的 README 打開來讀看看

root@router:/var/log# cat README
You are looking for the traditional text log files in /var/log, and they are
gone?

Here's an explanation on what's going on:
You are running a systemd-based OS where traditional syslog has been replaced
with the Journal. The journal stores the same (and more) information as classic
syslog. To make use of the journal and access the collected log data simply
invoke "journalctl", which will output the logs in the identical text-based
format the syslog files in /var/log used to be. For further details, please
refer to journalctl(1).

Alternatively, consider installing one of the traditional syslog
implementations available for your distribution, which will generate the
classic log files for you. Syslog implementations such as syslog-ng or rsyslog
may be installed side-by-side with the journal and will continue to function
the way they always did.

Thank you!

Further reading:
        man:journalctl(1)
        man:systemd-journald.service(8)
        man:journald.conf(5)
        https://0pointer.de/blog/projects/the-journal.html

好的 這下ㄘㄨㄚˋ了 Debian 12 更改了 Log Storage 的方式
從原本的 syslog 變成 systemd 式的 所以大致上我們昨天用的設定是真的沒太大用處的

其實有把 DPKG 的 Log 送到 Loki 上啦...

那麼我們就用現代方法來解決現代問題吧 乖乖用 journald 來做
直接進入寫 Config 的環節吧

Config.yml 更改

廢話不多說直接上 config 上來

server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://loki.ironman.test:3100/loki/api/v1/push

scrape_configs:
  - job_name: journald
    journal:
      max_age: 6h
      path: /var/log/journal
      labels:
        job: systemd-journal
        hostname: HOSTNAME
    relabel_configs:
      - source_labels: [__journal__systemd_unit]
        target_label: systemd_unit

加上了 hostname 但這邊 HOSTNAME 待會順便用 Ansuble 部署時處理
__journal__systemd_unit 這個是顯示 systemd Service 或 Process 的名字
接著就來寫 Ansible playbook 吧

---
- name: Configure promtail config.yaml
  hosts: target
  tasks:
    - name: Copy config.yml to target
      copy:
        src: config.yml
        dest: /etc/promtail/config.yml
        owner: root
        group: root
        mode: '0644'
    - name: Add promtail user to systemd-journal group
      user:
        name: promtail
        groups: systemd-journal
        append: yes
    - name: cat hostname
      shell: cat /etc/hostname
      register: TARGET_HOSTNAME
    - name: Replace hostname
      ansible.builtin.replace:
        path: /etc/promtail/config.yml
        regexp: 'HOSTNAME'
        replace: "{{TARGET_HOSTNAME.stdout}}"
    - name: Restart promtail
      ansible.builtin.systemd_service:
        name: promtail
        state: restarted

都準備好後就來部署

ansible-playbook push-update.yaml

來檢查看看資料正不正確 隨便抽檢一台

root@router:~/push# cat /etc/promtail/config.yml
server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://loki.ironman.test:3100/loki/api/v1/push

scrape_configs:
  - job_name: journald
    journal:
      max_age: 6h
      path: /var/log/journal
      labels:
        job: systemd-journal
        hostname: router.ironman.test
    relabel_configs:
      - source_labels: [__journal__systemd_unit]
        target_label: systemd_unit

看起來是對的 那麼就到 Loki 是看看發生什麼改變

看起來是我們預期的狀況了 但有我測試失敗的 label 留在上面RRRR

補充

原本想說透過 promtail --client.external-labels=hostname=$(hostname) 來解決
結果沒辦法加上 Label 所以只能用我自己找到的方法來暴力部署
要不然原本應該能用變數去解決的 ==