[root@tendrl14 vmstore]# fio --name=job1 --rw=read --size=1g  --direct=1
job1: (g=0): rw=read, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=1
fio-2.1.7
Starting 1 process
job1: Laying out IO file(s) (1 file(s) / 1024MB)
Jobs: 1 (f=1): [R] [-.-% done] [74388KB/0KB/0KB /s] [18.6K/0/0 iops] [eta 00m:00s]
job1: (groupid=0, jobs=1): err= 0: pid=732935: Mon Jun 22 07:33:45 2020
  read : io=1024.0MB, bw=74722KB/s, iops=18680, runt= 14033msec
    clat (usec): min=47, max=3260, avg=52.39, stdev=10.60
     lat (usec): min=47, max=3260, avg=52.51, stdev=10.61
    clat percentiles (usec):
     |  1.00th=[   49],  5.00th=[   50], 10.00th=[   50], 20.00th=[   50],
     | 30.00th=[   51], 40.00th=[   51], 50.00th=[   51], 60.00th=[   51],
     | 70.00th=[   52], 80.00th=[   52], 90.00th=[   54], 95.00th=[   60],
     | 99.00th=[   73], 99.50th=[   80], 99.90th=[  121], 99.95th=[  241],
     | 99.99th=[  294]
    bw (KB  /s): min=72336, max=75856, per=100.00%, avg=74737.14, stdev=861.90
    lat (usec) : 50=3.88%, 100=95.95%, 250=0.13%, 500=0.04%, 750=0.01%
    lat (usec) : 1000=0.01%
    lat (msec) : 2=0.01%, 4=0.01%
  cpu          : usr=3.46%, sys=24.34%, ctx=262161, majf=0, minf=7
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=262144/w=0/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: io=1024.0MB, aggrb=74722KB/s, minb=74722KB/s, maxb=74722KB/s, mint=14033msec, maxt=14033msec

Disk stats (read/write):
    dm-14: ios=259840/21, merge=0/0, ticks=9294/0, in_queue=9294, util=98.91%, aggrios=262144/29, aggrmerge=0/0, aggrticks=8489/0, aggrin_queue=8489, aggrutil=98.85%
    dm-8: ios=262144/29, merge=0/0, ticks=8489/0, in_queue=8489, util=98.85%, aggrios=131072/15, aggrmerge=0/0, aggrticks=4092/0, aggrin_queue=4092, aggrutil=98.85%
    dm-6: ios=0/1, merge=0/0, ticks=0/0, in_queue=0, util=0.01%, aggrios=262162/207, aggrmerge=0/18, aggrticks=10288/36, aggrin_queue=28, aggrutil=98.87%
  sdb: ios=262162/207, merge=0/18, ticks=10288/36, in_queue=28, util=98.87%
  dm-7: ios=262144/29, merge=0/0, ticks=8184/0, in_queue=8184, util=98.85%