Day 15 Kubernetes 耐久戰技大考驗 - Dbench on Kubernetes 安裝基礎操作篇

本日重點與方向 (TAG): kubernetes、k8s、PV、persistent volume、PVC、persistent volume claim、Pod use PVC、GlusterFS、Heketi、Rancher、Rancher Longhorn、Longhorn、SC、StorageClass、dbench、dbench on kubernetes
今天將會介紹使用 Bare Metal 進行 Kubernetes 環境中持久化儲存的軟體功能對於掛載的 PV/PVC 速度測試，主要會是以 Dbench on Kubernetes 這個開源專案進行驗證測試，並對於先前組建的 kubernetes 功能進行整合，如果你中途裝一裝有一些異常與失敗的話，基本上就是參考安裝筆記最下面的重建儲存叢集，還有搭配一下前幾天的重建 kubernetes 的做法，GlusterFS 跟 Longhorn 的配置上基本上就是參考前面幾天的搭建方式去搞，搞到有 Storage Class 之後就可以去做測試，實際行為就是開 PVC 掛到 dbench 的測試 Job 之上去等結果，因為 CSI 對於 Node 直接去存取有一下速度跟延遲的差異，所以本次也會介紹用同一套系統去搞測速，也就參考一下拿來比對驗證的基礎的，後續當然還有一些進階一點的去看官方網站 吧 (然後官方的 Docker Image 壞掉了) ，詳細的操作就看我隨便寫寫亂唬一通加減弄吧。

本次使用設備資訊

Network Switch

• 數量: 1
• 型號: D-Link 1210-28 (L2 Switch)

Bare Metal

Master Node

• 數量: 1
• Ubuntu: 16.04 / 18.04
• Docker Version: 19.03
• CPU: E3-1230_V3 ^ 1
• RAM: 16GB
• OS_Disk: 120 GB (SSD)
• Data_Disk: 500 GB (HDD)
• Network: 1Gbps

Worker Node

• 數量: 3
• Ubuntu: 16.04 / 18.04
• Docker Version: 19.03
• CPU: E3-1230_v3
• RAM: 8 GB
• OS_Disk: 120 GB (SSD)
• Data_Disk: 250 GB (HDD)
• Network: 1Gbps

dbench on kubernetes

https://github.com/leeliu/dbench

下載 github 上面的專案

git clone https://github.com/leeliu/dbench

切入 dbench 之下進行編譯 Docker Container Image

docker image build -t dbench .
docker images

Container 匯出與匯入

docker save -o <export-tar-filename> <local-image-name>
docker load -i <export-tar-filename>

設定測試的 yaml 檔案

• PVC:
  • spec.storageClassName: 呼叫生成 PVC 的 storage class (longhorn/glusterfs.........)
  • spec.resources.requests.storage: 切分出來測試的空間，需要比測試的大小來的大一些
• Job:
  • spec.template.spec.containers.image: 改成剛剛 build 出來的 Container Image
  • spec.template.spec.containers.imagePullPolicy: 改成 Never，使用本地端的 Container Image
  • spec.template.spec.containers.env
    • name:FIO_SIZE
    • value: 1G (看一下需求的測試大小)

dbench.yaml
-----
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: dbench-pv-claim
spec:
  storageClassName: ssd
  # storageClassName: gp2
  # storageClassName: local-storage
  # storageClassName: ibmc-block-bronze
  # storageClassName: ibmc-block-silver
  # storageClassName: ibmc-block-gold
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi
---
apiVersion: batch/v1
kind: Job
metadata:
  name: dbench
spec:
  template:
    spec:
      containers:
      - name: dbench
        image: dbench:latest
        imagePullPolicy: Always
        env:
          - name: DBENCH_MOUNTPOINT
            value: /data
          # - name: DBENCH_QUICK
          #   value: "yes"
           - name: FIO_SIZE
             value: 1G
          # - name: FIO_OFFSET_INCREMENT
          #   value: 256M
          # - name: FIO_DIRECT
          #   value: "0"
        volumeMounts:
        - name: dbench-pv
          mountPath: /data
      restartPolicy: Never
      volumes:
      - name: dbench-pv
        persistentVolumeClaim:
          claimName: dbench-pv-claim
  backoffLimit: 4

• 改的參考給個模板

部署 dbench 測試的 yaml 進行測速

kubectl apply -f dbench.yaml

• 獲得執行狀態

kubectl logs <pod-name> -f 

獲得檢測後測試速度的成果報表

這邊需要先讓 dbench 的 job 狀態變成 complete，後面才會是完整的資料。

kubectl logs <pod-name>

• 成果報表
  • Random Read/Write IOPS: 隨機讀寫 IOPS
  • Random Read/Write BW : 隨機讀寫速度
  • Average Latency (usec) Read/Write: 平均寫入延遲
  • Sequential Read/Write: 循序寫入速度
  • Mixed Random Read/Write IOPS: 混和讀寫 IOPS

root@sdn-k8s-b2-1-2:~# kubectl logs dbench-vfn94
Working dir: /data

Testing Read IOPS...
read_iops: (g=0): rw=randread, bs=4096B-4096B,4096B-4096B,4096B-4096B, ioengine=libaio, iodepth=64
fio-2.17-45-g06cb
Starting 1 process
read_iops: Laying out IO file(s) (1 file(s) / 1024MiB)

read_iops: (groupid=0, jobs=1): err= 0: pid=17: Wed Aug 26 13:35:22 2020
   read: IOPS=454, BW=1834KiB/s (1878kB/s)(27.5MiB/15321msec)
  cpu          : usr=0.52%, sys=1.12%, ctx=7035, majf=0, minf=1
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.2%, 32=0.5%, >=64=114.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwt: total=6961,0,0, short=0,0,0, dropped=0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=1834KiB/s (1878kB/s), 1834KiB/s-1834KiB/s (1878kB/s-1878kB/s), io=27.5MiB (28.8MB), run=15321-15321msec

Disk stats (read/write):
  sdf: ios=7972/36, merge=2/11, ticks=1097004/6156, in_queue=1106928, util=99.41%


Testing Write IOPS...
write_iops: (g=0): rw=randwrite, bs=4096B-4096B,4096B-4096B,4096B-4096B, ioengine=libaio, iodepth=64
fio-2.17-45-g06cb
Starting 1 process

write_iops: (groupid=0, jobs=1): err= 0: pid=33: Wed Aug 26 13:35:40 2020
  write: IOPS=130, BW=537KiB/s (550kB/s)(8304KiB/15472msec)
  cpu          : usr=0.10%, sys=0.35%, ctx=1988, majf=0, minf=1
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.4%, 16=0.8%, 32=1.6%, >=64=111.2%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwt: total=0,2013,0, short=0,0,0, dropped=0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
  WRITE: bw=537KiB/s (550kB/s), 537KiB/s-537KiB/s (550kB/s-550kB/s), io=8304KiB (8503kB), run=15472-15472msec

Disk stats (read/write):
  sdf: ios=0/2281, merge=0/8, ticks=0/1058200, in_queue=1078804, util=99.42%


Testing Read Bandwidth...
read_bw: (g=0): rw=randread, bs=128KiB-128KiB,128KiB-128KiB,128KiB-128KiB, ioengine=libaio, iodepth=64
fio-2.17-45-g06cb
Starting 1 process

read_bw: (groupid=0, jobs=1): err= 0: pid=49: Wed Aug 26 13:35:58 2020
   read: IOPS=366, BW=46.3MiB/s (48.6MB/s)(714MiB/15428msec)
  cpu          : usr=0.30%, sys=1.06%, ctx=5537, majf=0, minf=1
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.3%, 32=0.6%, >=64=114.5%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwt: total=5647,0,0, short=0,0,0, dropped=0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=46.3MiB/s (48.6MB/s), 46.3MiB/s-46.3MiB/s (48.6MB/s-48.6MB/s), io=714MiB (748MB), run=15428-15428msec

Disk stats (read/write):
  sdf: ios=6279/33, merge=217/4, ticks=1060284/7008, in_queue=1074188, util=99.44%


Testing Write Bandwidth...
write_bw: (g=0): rw=randwrite, bs=128KiB-128KiB,128KiB-128KiB,128KiB-128KiB, ioengine=libaio, iodepth=64
fio-2.17-45-g06cb
Starting 1 process

write_bw: (groupid=0, jobs=1): err= 0: pid=65: Wed Aug 26 13:36:16 2020
  write: IOPS=101, BW=13.2MiB/s (13.9MB/s)(206MiB/15593msec)
  cpu          : usr=0.35%, sys=0.29%, ctx=1577, majf=0, minf=1
  IO depths    : 1=0.1%, 2=0.1%, 4=0.3%, 8=0.5%, 16=1.0%, 32=2.0%, >=64=113.1%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=99.9%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwt: total=0,1583,0, short=0,0,0, dropped=0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
  WRITE: bw=13.2MiB/s (13.9MB/s), 13.2MiB/s-13.2MiB/s (13.9MB/s-13.9MB/s), io=206MiB (216MB), run=15593-15593msec

Disk stats (read/write):
  sdf: ios=0/1848, merge=0/25, ticks=0/1093816, in_queue=1100900, util=99.39%


Testing Read Latency...
read_latency: (g=0): rw=randread, bs=4096B-4096B,4096B-4096B,4096B-4096B, ioengine=libaio, iodepth=4
fio-2.17-45-g06cb
Starting 1 process

read_latency: (groupid=0, jobs=1): err= 0: pid=81: Wed Aug 26 13:36:33 2020
   read: IOPS=273, BW=1096KiB/s (1123kB/s)(16.8MiB/15008msec)
    slat (usec): min=6, max=187, avg=30.03, stdev=10.10
    clat (msec): min=1, max=364, avg=14.57, stdev=19.30
     lat (msec): min=1, max=364, avg=14.60, stdev=19.30
    clat percentiles (msec):
     |  1.00th=[    4],  5.00th=[    5], 10.00th=[    5], 20.00th=[    7],
     | 30.00th=[    8], 40.00th=[    9], 50.00th=[   10], 60.00th=[   11],
     | 70.00th=[   13], 80.00th=[   16], 90.00th=[   29], 95.00th=[   48],
     | 99.00th=[   81], 99.50th=[  135], 99.90th=[  200], 99.95th=[  330],
     | 99.99th=[  363]
    lat (msec) : 2=0.39%, 4=4.04%, 10=46.80%, 20=35.56%, 50=8.83%
    lat (msec) : 100=3.82%, 250=0.56%, 500=0.07%
  cpu          : usr=0.43%, sys=1.28%, ctx=4062, majf=0, minf=2
  IO depths    : 1=0.1%, 2=0.1%, 4=113.4%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwt: total=4111,0,0, short=0,0,0, dropped=0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
   READ: bw=1096KiB/s (1123kB/s), 1096KiB/s-1096KiB/s (1123kB/s-1123kB/s), io=16.8MiB (16.9MB), run=15008-15008msec

Disk stats (read/write):
  sdf: ios=4589/28, merge=0/3, ticks=66876/2260, in_queue=69152, util=99.42%


Testing Write Latency...
write_latency: (g=0): rw=randwrite, bs=4096B-4096B,4096B-4096B,4096B-4096B, ioengine=libaio, iodepth=4
fio-2.17-45-g06cb
Starting 1 process

write_latency: (groupid=0, jobs=1): err= 0: pid=97: Wed Aug 26 13:36:51 2020
  write: IOPS=145, BW=582KiB/s (596kB/s)(8760KiB/15048msec)
    slat (usec): min=5, max=86749, avg=71.26, stdev=1740.32
    clat (msec): min=1, max=466, avg=27.40, stdev=57.28
     lat (msec): min=1, max=466, avg=27.48, stdev=57.30
    clat percentiles (usec):
     |  1.00th=[ 1448],  5.00th=[ 1768], 10.00th=[ 1912], 20.00th=[ 2192],
     | 30.00th=[ 3152], 40.00th=[ 5664], 50.00th=[13632], 60.00th=[15040],
     | 70.00th=[16768], 80.00th=[20096], 90.00th=[64256], 95.00th=[134144],
     | 99.00th=[309248], 99.50th=[342016], 99.90th=[464896], 99.95th=[464896],
     | 99.99th=[464896]
    lat (msec) : 2=13.44%, 4=22.95%, 10=4.66%, 20=38.73%, 50=8.73%
    lat (msec) : 100=4.34%, 250=5.21%, 500=2.06%
  cpu          : usr=0.19%, sys=0.64%, ctx=2196, majf=0, minf=3
  IO depths    : 1=0.1%, 2=0.1%, 4=115.6%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwt: total=0,2187,0, short=0,0,0, dropped=0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
  WRITE: bw=582KiB/s (596kB/s), 582KiB/s-582KiB/s (596kB/s-596kB/s), io=8760KiB (8970kB), run=15048-15048msec

Disk stats (read/write):
  sdf: ios=0/2535, merge=0/5, ticks=0/67680, in_queue=68148, util=99.42%


Testing Read Sequential Speed...
read_seq: (g=0): rw=read, bs=1024KiB-1024KiB,1024KiB-1024KiB,1024KiB-1024KiB, ioengine=libaio, iodepth=16
...
fio-2.17-45-g06cb
Starting 4 threads
read_seq: Laying out IO file(s) (1 file(s) / 2524MiB)

read_seq: (groupid=0, jobs=1): err= 0: pid=113: Wed Aug 26 13:39:03 2020
   read: IOPS=30, BW=31.1MiB/s (33.5MB/s)(515MiB/16131msec)
  cpu          : usr=0.14%, sys=0.18%, ctx=517, majf=0, minf=0
  IO depths    : 1=0.2%, 2=0.4%, 4=0.8%, 8=1.6%, 16=111.8%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=99.8%, 8=0.0%, 16=0.2%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwt: total=500,0,0, short=0,0,0, dropped=0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
read_seq: (groupid=0, jobs=1): err= 0: pid=114: Wed Aug 26 13:39:03 2020
   read: IOPS=32, BW=33.4MiB/s (34.7MB/s)(565MiB/17102msec)
  cpu          : usr=0.03%, sys=0.30%, ctx=573, majf=0, minf=0
  IO depths    : 1=0.2%, 2=0.4%, 4=0.7%, 8=1.5%, 16=113.3%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=99.8%, 8=0.0%, 16=0.2%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwt: total=550,0,0, short=0,0,0, dropped=0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
read_seq: (groupid=0, jobs=1): err= 0: pid=115: Wed Aug 26 13:39:03 2020
   read: IOPS=32, BW=33.3MiB/s (34.9MB/s)(578MiB/17390msec)
  cpu          : usr=0.02%, sys=0.31%, ctx=581, majf=0, minf=0
  IO depths    : 1=0.2%, 2=0.4%, 4=0.7%, 8=1.4%, 16=110.7%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=99.8%, 8=0.0%, 16=0.2%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwt: total=563,0,0, short=0,0,0, dropped=0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
read_seq: (groupid=0, jobs=1): err= 0: pid=116: Wed Aug 26 13:39:03 2020
   read: IOPS=32, BW=32.2MiB/s (34.6MB/s)(571MiB/17313msec)
  cpu          : usr=0.03%, sys=0.28%, ctx=574, majf=0, minf=0
  IO depths    : 1=0.2%, 2=0.4%, 4=0.7%, 8=1.4%, 16=112.1%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=99.8%, 8=0.0%, 16=0.2%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwt: total=556,0,0, short=0,0,0, dropped=0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=128MiB/s (134MB/s), 31.1MiB/s-33.3MiB/s (33.5MB/s-34.9MB/s), io=2229MiB (2337MB), run=16131-17390msec

Disk stats (read/write):
  sdf: ios=2475/2, merge=0/1, ticks=1188976/124, in_queue=1193752, util=99.48%


Testing Write Sequential Speed...
write_seq: (g=0): rw=write, bs=1024KiB-1024KiB,1024KiB-1024KiB,1024KiB-1024KiB, ioengine=libaio, iodepth=16
...
fio-2.17-45-g06cb
Starting 4 threads

write_seq: (groupid=0, jobs=1): err= 0: pid=132: Wed Aug 26 13:39:25 2020
  write: IOPS=8, BW=9252KiB/s (9474kB/s)(171MiB/18927msec)
  cpu          : usr=0.13%, sys=0.02%, ctx=163, majf=0, minf=0
  IO depths    : 1=0.6%, 2=1.3%, 4=2.6%, 8=5.1%, 16=112.2%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=99.4%, 8=0.0%, 16=0.6%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwt: total=0,156,0, short=0,0,0, dropped=0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
write_seq: (groupid=0, jobs=1): err= 0: pid=133: Wed Aug 26 13:39:25 2020
  write: IOPS=8, BW=9425KiB/s (9651kB/s)(170MiB/18471msec)
  cpu          : usr=0.10%, sys=0.05%, ctx=153, majf=0, minf=0
  IO depths    : 1=0.6%, 2=1.3%, 4=2.6%, 8=5.2%, 16=112.9%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=99.4%, 8=0.0%, 16=0.6%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwt: total=0,155,0, short=0,0,0, dropped=0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
write_seq: (groupid=0, jobs=1): err= 0: pid=134: Wed Aug 26 13:39:25 2020
  write: IOPS=8, BW=9303KiB/s (9526kB/s)(170MiB/18712msec)
  cpu          : usr=0.03%, sys=0.12%, ctx=168, majf=0, minf=0
  IO depths    : 1=0.6%, 2=1.3%, 4=2.6%, 8=5.2%, 16=112.9%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=99.4%, 8=0.0%, 16=0.6%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwt: total=0,155,0, short=0,0,0, dropped=0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
write_seq: (groupid=0, jobs=1): err= 0: pid=135: Wed Aug 26 13:39:25 2020
  write: IOPS=9, BW=9.85MiB/s (10.3MB/s)(179MiB/18222msec)
  cpu          : usr=0.09%, sys=0.08%, ctx=178, majf=0, minf=0
  IO depths    : 1=0.6%, 2=1.2%, 4=2.4%, 8=4.9%, 16=106.7%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=99.4%, 8=0.0%, 16=0.6%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwt: total=0,164,0, short=0,0,0, dropped=0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=36.5MiB/s (38.3MB/s), 9252KiB/s-9.85MiB/s (9474kB/s-10.3MB/s), io=690MiB (724MB), run=18222-18927msec

Disk stats (read/write):
  sdf: ios=0/763, merge=0/4, ticks=0/1281344, in_queue=1290208, util=99.46%


Testing Read/Write Mixed...
rw_mix: (g=0): rw=randrw, bs=4096B-4096B,4096B-4096B,4096B-4096B, ioengine=libaio, iodepth=64
fio-2.17-45-g06cb
Starting 1 process

rw_mix: (groupid=0, jobs=1): err= 0: pid=151: Wed Aug 26 13:39:42 2020
   read: IOPS=182, BW=741KiB/s (759kB/s)(11.4MiB/15257msec)
  write: IOPS=61, BW=251KiB/s (257kB/s)(3824KiB/15257msec)
  cpu          : usr=0.17%, sys=0.85%, ctx=3690, majf=0, minf=1
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.2%, 16=0.4%, 32=0.9%, >=64=115.3%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwt: total=2784,935,0, short=0,0,0, dropped=0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=741KiB/s (759kB/s), 741KiB/s-741KiB/s (759kB/s-759kB/s), io=11.4MiB (11.6MB), run=15257-15257msec
  WRITE: bw=251KiB/s (257kB/s), 251KiB/s-251KiB/s (257kB/s-257kB/s), io=3824KiB (3916kB), run=15257-15257msec

Disk stats (read/write):
  sdf: ios=3269/1086, merge=0/3, ticks=757452/312804, in_queue=1070464, util=99.43%


All tests complete.

==================
= Dbench Summary =
==================
Random Read/Write IOPS: 454/130. BW: 46.3MiB/s / 13.2MiB/s
Average Latency (usec) Read/Write: 14.60/27.48
Sequential Read/Write: 128MiB/s / 36.5MiB/s
Mixed Random Read/Write IOPS: 182/61

主機端使用 dbench 測試硬碟速度

安裝輔助套件

sudo apt-get install fio -y

下載專案 && 切入專案

git clone https://github.com/leeliu/dbench
cd dbench

設定 dbench 對應的 ENV

• DBENCH_MOUNTPOINT: 測試的掛載路徑
• FIO_SIZE: 測試的數據大小

export DBENCH_MOUNTPOINT=<longhorn-mount-point>
export FIO_SIZE=1G

執行測試

bash docker-entrypoint.sh fio

執行結果

• 成果報表
  • Random Read/Write IOPS: 隨機讀寫 IOPS
  • Random Read/Write BW : 隨機讀寫速度
  • Average Latency (usec) Read/Write: 平均寫入延遲
  • Sequential Read/Write: 循序寫入速度
  • Mixed Random Read/Write IOPS: 混和讀寫 IOPS

Working dir: /var/lib/kubelet/pods/9b4b0615-0fa0-4e03-bb45-8bd4793d1460/volumes/kubernetes.io~csi/pvc-5fb58e77-b8e0-40b4-b0b9-0ca21b5f0cfd/mount

Testing Read IOPS...
read_iops: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
fio-3.1
Starting 1 process
read_iops: Laying out IO file (1 file / 1024MiB)

read_iops: (groupid=0, jobs=1): err= 0: pid=28049: Thu Aug 27 06:27:53 2020
   read: IOPS=489, BW=1974KiB/s (2022kB/s)(29.3MiB/15188msec)
   bw (  KiB/s): min=  881, max= 2650, per=100.00%, avg=1975.13, stdev=434.51, samples=30
   iops        : min=  220, max=  662, avg=493.53, stdev=108.72, samples=30
  cpu          : usr=0.55%, sys=1.04%, ctx=7486, majf=0, minf=1
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.2%, 32=0.4%, >=64=115.4%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwt: total=7433,0,0, short=0,0,0, dropped=0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=1974KiB/s (2022kB/s), 1974KiB/s-1974KiB/s (2022kB/s-2022kB/s), io=29.3MiB (30.7MB), run=15188-15188msec

Disk stats (read/write):
  sdf: ios=8534/2, merge=2/1, ticks=1084272/296, in_queue=1088096, util=99.52%


Testing Write IOPS...
write_iops: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
fio-3.1
Starting 1 process

write_iops: (groupid=0, jobs=1): err= 0: pid=28751: Thu Aug 27 06:28:12 2020
  write: IOPS=160, BW=657KiB/s (673kB/s)(10.0MiB/15587msec)
   bw (  KiB/s): min=   24, max= 1036, per=98.32%, avg=645.00, stdev=260.93, samples=31
   iops        : min=    6, max=  259, avg=161.13, stdev=65.22, samples=31
  cpu          : usr=0.16%, sys=0.45%, ctx=2468, majf=0, minf=1
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=0.6%, 32=1.3%, >=64=115.3%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwt: total=0,2497,0, short=0,0,0, dropped=0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
  WRITE: bw=657KiB/s (673kB/s), 657KiB/s-657KiB/s (673kB/s-673kB/s), io=10.0MiB (10.5MB), run=15587-15587msec

Disk stats (read/write):
  sdf: ios=0/2912, merge=0/12, ticks=0/1099444, in_queue=1115492, util=99.48%


Testing Read Bandwidth...
read_bw: (g=0): rw=randread, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=libaio, iodepth=64
fio-3.1
Starting 1 process

read_bw: (groupid=0, jobs=1): err= 0: pid=29562: Thu Aug 27 06:28:29 2020
   read: IOPS=369, BW=46.6MiB/s (48.9MB/s)(709MiB/15192msec)
   bw (  KiB/s): min=21290, max=61440, per=100.00%, avg=47849.57, stdev=10613.62, samples=30
   iops        : min=  166, max=  480, avg=373.33, stdev=82.82, samples=30
  cpu          : usr=0.41%, sys=0.99%, ctx=5465, majf=0, minf=1
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.3%, 32=0.6%, >=64=116.4%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwt: total=5606,0,0, short=0,0,0, dropped=0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=46.6MiB/s (48.9MB/s), 46.6MiB/s-46.6MiB/s (48.9MB/s-48.9MB/s), io=709MiB (743MB), run=15192-15192msec

Disk stats (read/write):
  sdf: ios=6312/2, merge=210/1, ticks=1049796/624, in_queue=1054940, util=99.49%


Testing Write Bandwidth...
write_bw: (g=0): rw=randwrite, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=libaio, iodepth=64
fio-3.1
Starting 1 process

write_bw: (groupid=0, jobs=1): err= 0: pid=30240: Thu Aug 27 06:28:48 2020
  write: IOPS=85, BW=11.2MiB/s (11.7MB/s)(181MiB/16145msec)
   bw (  KiB/s): min=  514, max=21076, per=99.81%, avg=11441.77, stdev=6446.33, samples=31
   iops        : min=    4, max=  164, avg=89.23, stdev=50.25, samples=31
  cpu          : usr=0.21%, sys=0.25%, ctx=1341, majf=0, minf=1
  IO depths    : 1=0.1%, 2=0.1%, 4=0.3%, 8=0.6%, 16=1.2%, 32=2.3%, >=64=115.6%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=99.9%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwt: total=0,1383,0, short=0,0,0, dropped=0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
  WRITE: bw=11.2MiB/s (11.7MB/s), 11.2MiB/s-11.2MiB/s (11.7MB/s-11.7MB/s), io=181MiB (190MB), run=16145-16145msec

Disk stats (read/write):
  sdf: ios=0/1626, merge=0/22, ticks=0/1097396, in_queue=1116564, util=99.49%


Testing Read Latency...
read_latency: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=4
fio-3.1
Starting 1 process

read_latency: (groupid=0, jobs=1): err= 0: pid=30982: Thu Aug 27 06:29:05 2020
   read: IOPS=326, BW=1309KiB/s (1340kB/s)(19.2MiB/15017msec)
    slat (nsec): min=5982, max=74702, avg=28906.93, stdev=9359.55
    clat (usec): min=1019, max=181096, avg=12214.17, stdev=11075.50
     lat (usec): min=1032, max=181118, avg=12243.78, stdev=11075.77
    clat percentiles (msec):
     |  1.00th=[    4],  5.00th=[    5], 10.00th=[    5], 20.00th=[    7],
     | 30.00th=[    8], 40.00th=[    9], 50.00th=[   10], 60.00th=[   11],
     | 70.00th=[   12], 80.00th=[   15], 90.00th=[   20], 95.00th=[   34],
     | 99.00th=[   57], 99.50th=[   67], 99.90th=[  128], 99.95th=[  144],
     | 99.99th=[  182]
   bw (  KiB/s): min=  681, max= 1608, per=100.00%, avg=1311.60, stdev=244.47, samples=30
   iops        : min=  170, max=  402, avg=327.63, stdev=61.06, samples=30
  lat (msec)   : 2=0.18%, 4=4.46%, 10=49.27%, 20=36.44%, 50=7.70%
  lat (msec)   : 100=1.89%, 250=0.12%
  cpu          : usr=0.49%, sys=1.45%, ctx=4858, majf=0, minf=2
  IO depths    : 1=0.1%, 2=0.1%, 4=113.5%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwt: total=4910,0,0, short=0,0,0, dropped=0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
   READ: bw=1309KiB/s (1340kB/s), 1309KiB/s-1309KiB/s (1340kB/s-1340kB/s), io=19.2MiB (20.1MB), run=15017-15017msec

Disk stats (read/write):
  sdf: ios=5574/2, merge=0/1, ticks=67768/156, in_queue=67968, util=99.50%


Testing Write Latency...
write_latency: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=4
fio-3.1
Starting 1 process

write_latency: (groupid=0, jobs=1): err= 0: pid=31719: Thu Aug 27 06:29:23 2020
  write: IOPS=138, BW=554KiB/s (567kB/s)(8360KiB/15097msec)
    slat (usec): min=7, max=141539, avg=95.76, stdev=3097.66
    clat (usec): min=1044, max=531746, avg=29081.00, stdev=66573.51
     lat (usec): min=1055, max=531789, avg=29177.24, stdev=66668.41
    clat percentiles (usec):
     |  1.00th=[  1385],  5.00th=[  1713], 10.00th=[  1860], 20.00th=[  2040],
     | 30.00th=[  2311], 40.00th=[  3130], 50.00th=[  5014], 60.00th=[ 13960],
     | 70.00th=[ 15664], 80.00th=[ 18744], 90.00th=[ 79168], 95.00th=[187696],
     | 99.00th=[312476], 99.50th=[396362], 99.90th=[517997], 99.95th=[522191],
     | 99.99th=[530580]
   bw (  KiB/s): min=   48, max= 1090, per=100.00%, avg=557.00, stdev=269.00, samples=30
   iops        : min=   12, max=  272, avg=139.07, stdev=67.16, samples=30
  lat (msec)   : 2=17.49%, 4=27.89%, 10=6.47%, 20=29.52%, 50=7.04%
  lat (msec)   : 100=3.07%, 250=5.80%, 500=2.68%, 750=0.19%
  cpu          : usr=0.23%, sys=0.46%, ctx=2079, majf=0, minf=2
  IO depths    : 1=0.1%, 2=0.1%, 4=115.3%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwt: total=0,2087,0, short=0,0,0, dropped=0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
  WRITE: bw=554KiB/s (567kB/s), 554KiB/s-554KiB/s (567kB/s-567kB/s), io=8360KiB (8561kB), run=15097-15097msec

Disk stats (read/write):
  sdf: ios=0/2411, merge=0/3, ticks=0/68024, in_queue=68832, util=99.46%


Testing Read Sequential Speed...
read_seq: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=16
...
fio-3.1
Starting 4 threads
read_seq: Laying out IO file (1 file / 2524MiB)

read_seq: (groupid=0, jobs=1): err= 0: pid=2860: Thu Aug 27 06:30:48 2020
   read: IOPS=31, BW=31.0MiB/s (33.5MB/s)(520MiB/16269msec)
   bw (  KiB/s): min=20480, max=51200, per=24.36%, avg=32043.75, stdev=8164.98, samples=32
   iops        : min=   20, max=   50, avg=31.19, stdev= 7.97, samples=32
  cpu          : usr=0.01%, sys=0.30%, ctx=520, majf=0, minf=0
  IO depths    : 1=0.2%, 2=0.4%, 4=0.8%, 8=1.6%, 16=110.7%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=99.8%, 8=0.0%, 16=0.2%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwt: total=505,0,0, short=0,0,0, dropped=0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
read_seq: (groupid=0, jobs=1): err= 0: pid=2861: Thu Aug 27 06:30:48 2020
   read: IOPS=33, BW=34.2MiB/s (35.8MB/s)(559MiB/16350msec)
   bw (  KiB/s): min=16384, max=53461, per=25.39%, avg=33399.53, stdev=7905.27, samples=32
   iops        : min=   16, max=   52, avg=32.56, stdev= 7.69, samples=32
  cpu          : usr=0.07%, sys=0.26%, ctx=559, majf=0, minf=0
  IO depths    : 1=0.2%, 2=0.4%, 4=0.7%, 8=1.5%, 16=114.5%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=99.8%, 8=0.0%, 16=0.2%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwt: total=544,0,0, short=0,0,0, dropped=0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
read_seq: (groupid=0, jobs=1): err= 0: pid=2862: Thu Aug 27 06:30:48 2020
   read: IOPS=30, BW=31.9MiB/s (33.5MB/s)(506MiB/15842msec)
   bw (  KiB/s): min= 8192, max=55296, per=24.66%, avg=32437.68, stdev=9129.84, samples=31
   iops        : min=    8, max=   54, avg=31.68, stdev= 8.92, samples=31
  cpu          : usr=0.08%, sys=0.23%, ctx=507, majf=0, minf=0
  IO depths    : 1=0.2%, 2=0.4%, 4=0.8%, 8=1.6%, 16=113.8%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=99.8%, 8=0.0%, 16=0.2%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwt: total=491,0,0, short=0,0,0, dropped=0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
read_seq: (groupid=0, jobs=1): err= 0: pid=2863: Thu Aug 27 06:30:48 2020
   read: IOPS=31, BW=31.0MiB/s (33.5MB/s)(515MiB/16109msec)
   bw (  KiB/s): min= 6144, max=53248, per=24.33%, avg=31998.31, stdev=9200.49, samples=32
   iops        : min=    6, max=   52, avg=31.22, stdev= 9.00, samples=32
  cpu          : usr=0.02%, sys=0.30%, ctx=515, majf=0, minf=0
  IO depths    : 1=0.2%, 2=0.4%, 4=0.8%, 8=1.6%, 16=111.8%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=99.8%, 8=0.0%, 16=0.2%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwt: total=500,0,0, short=0,0,0, dropped=0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=128MiB/s (135MB/s), 31.9MiB/s-34.2MiB/s (33.5MB/s-35.8MB/s), io=2100MiB (2202MB), run=15842-16350msec

Disk stats (read/write):
  sdf: ios=2343/2, merge=0/1, ticks=1148652/236, in_queue=1150296, util=99.55%


Testing Write Sequential Speed...
write_seq: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=16
...
fio-3.1
Starting 4 threads

write_seq: (groupid=0, jobs=1): err= 0: pid=3608: Thu Aug 27 06:31:10 2020
  write: IOPS=8, BW=9414KiB/s (9640kB/s)(171MiB/18601msec)
   bw (  KiB/s): min= 2048, max=18432, per=31.35%, avg=11664.44, stdev=6213.34, samples=27
   iops        : min=    2, max=   18, avg=11.37, stdev= 6.10, samples=27
  cpu          : usr=0.04%, sys=0.09%, ctx=165, majf=0, minf=0
  IO depths    : 1=0.6%, 2=1.3%, 4=2.6%, 8=5.1%, 16=112.2%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=99.4%, 8=0.0%, 16=0.6%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwt: total=0,156,0, short=0,0,0, dropped=0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
write_seq: (groupid=0, jobs=1): err= 0: pid=3609: Thu Aug 27 06:31:10 2020
  write: IOPS=8, BW=9767KiB/s (10.0MB/s)(171MiB/17929msec)
   bw (  KiB/s): min= 2043, max=24576, per=32.98%, avg=12273.23, stdev=7244.50, samples=26
   iops        : min=    1, max=   24, avg=11.85, stdev= 7.24, samples=26
  cpu          : usr=0.06%, sys=0.07%, ctx=163, majf=0, minf=0
  IO depths    : 1=0.6%, 2=1.3%, 4=2.6%, 8=5.1%, 16=112.2%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=99.4%, 8=0.0%, 16=0.6%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwt: total=0,156,0, short=0,0,0, dropped=0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
write_seq: (groupid=0, jobs=1): err= 0: pid=3610: Thu Aug 27 06:31:10 2020
  write: IOPS=8, BW=9421KiB/s (9647kB/s)(170MiB/18478msec)
   bw (  KiB/s): min= 2048, max=24625, per=34.18%, avg=12718.60, stdev=6825.36, samples=25
   iops        : min=    2, max=   24, avg=12.40, stdev= 6.66, samples=25
  cpu          : usr=0.06%, sys=0.06%, ctx=161, majf=0, minf=0
  IO depths    : 1=0.6%, 2=1.3%, 4=2.6%, 8=5.2%, 16=112.9%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=99.4%, 8=0.0%, 16=0.6%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwt: total=0,155,0, short=0,0,0, dropped=0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
write_seq: (groupid=0, jobs=1): err= 0: pid=3611: Thu Aug 27 06:31:10 2020
  write: IOPS=8, BW=9317KiB/s (9540kB/s)(171MiB/18795msec)
   bw (  KiB/s): min= 2048, max=22528, per=37.32%, avg=13887.74, stdev=7037.74, samples=23
   iops        : min=    2, max=   22, avg=13.48, stdev= 6.83, samples=23
  cpu          : usr=0.04%, sys=0.07%, ctx=157, majf=0, minf=0
  IO depths    : 1=0.6%, 2=1.3%, 4=2.6%, 8=5.1%, 16=112.2%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=99.4%, 8=0.0%, 16=0.6%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwt: total=0,156,0, short=0,0,0, dropped=0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=36.3MiB/s (38.1MB/s), 9317KiB/s-9767KiB/s (9540kB/s-10.0MB/s), io=683MiB (716MB), run=17929-18795msec

Disk stats (read/write):
  sdf: ios=0/758, merge=0/4, ticks=0/1262528, in_queue=1278760, util=99.55%


Testing Read/Write Mixed...
rw_mix: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
fio-3.1
Starting 1 process

rw_mix: (groupid=0, jobs=1): err= 0: pid=4586: Thu Aug 27 06:31:28 2020
   read: IOPS=179, BW=729KiB/s (747kB/s)(11.0MiB/15502msec)
   bw (  KiB/s): min=  128, max= 1330, per=100.00%, avg=743.43, stdev=313.07, samples=30
   iops        : min=   32, max=  332, avg=185.63, stdev=78.20, samples=30
  write: IOPS=59, BW=244KiB/s (250kB/s)(3788KiB/15502msec)
   bw (  KiB/s): min=   72, max=  433, per=100.00%, avg=247.30, stdev=103.05, samples=30
   iops        : min=   18, max=  108, avg=61.80, stdev=25.74, samples=30
  cpu          : usr=0.27%, sys=0.67%, ctx=3626, majf=0, minf=1
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.2%, 16=0.4%, 32=0.9%, >=64=112.1%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwt: total=2785,926,0, short=0,0,0, dropped=0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=729KiB/s (747kB/s), 729KiB/s-729KiB/s (747kB/s-747kB/s), io=11.0MiB (11.6MB), run=15502-15502msec
  WRITE: bw=244KiB/s (250kB/s), 244KiB/s-244KiB/s (250kB/s-250kB/s), io=3788KiB (3879kB), run=15502-15502msec

Disk stats (read/write):
  sdf: ios=3168/1057, merge=2/3, ticks=740884/327188, in_queue=1068460, util=99.48%


All tests complete.

==================
= Dbench Summary =
==================
Random Read/Write IOPS: 489/160. BW: 46.6MiB/s / 11.2MiB/s
Average Latency (usec) Read/Write: 12243.78/29177.24
Sequential Read/Write: 128MiB/s / 36.3MiB/s
Mixed Random Read/Write IOPS: 179/59