ubuntu@disk-test:~$ fio --rw=randread --ioengine=libaio --direct=1 --numjobs=4 --bs=4k --runtime=60 --size=5G --runtime=60 --iodepth=16 --name=/mnt/vol100/fio1.dat --name=/mnt/vol100/fio2.dat --name=/mnt/vol100/fio3.dat --name=/mnt/vol100/fio4.dat
/mnt/vol100/fio1.dat: (g=0): rw=randread, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=16
...
/mnt/vol100/fio1.dat: (g=0): rw=randread, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=16
/mnt/vol100/fio2.dat: (g=0): rw=randread, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=16
...
/mnt/vol100/fio2.dat: (g=0): rw=randread, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=16
/mnt/vol100/fio3.dat: (g=0): rw=randread, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=16
...
/mnt/vol100/fio3.dat: (g=0): rw=randread, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=16
/mnt/vol100/fio4.dat: (g=0): rw=randread, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=16
...
/mnt/vol100/fio4.dat: (g=0): rw=randread, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=16
fio 1.59
Starting 16 processes
Jobs: 16 (f=16): [rrrrrrrrrrrrrrrr] [100.0% done] [45141K/0K /s] [11.2K/0  iops] [eta 00m:00s]
/mnt/vol100/fio1.dat: (groupid=0, jobs=1): err= 0: pid=7350
  read : io=161008KB, bw=2683.2KB/s, iops=670 , runt= 60010msec
    slat (usec): min=3 , max=62558 , avg=654.37, stdev=3442.29
    clat (msec): min=3 , max=284 , avg=23.20, stdev=12.08
     lat (msec): min=3 , max=284 , avg=23.85, stdev=12.27
    bw (KB/s) : min= 2027, max= 3200, per=6.00%, avg=2683.47, stdev=224.88
  cpu          : usr=0.31%, sys=1.05%, ctx=11175, majf=0, minf=38
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued r/w/d: total=40252/0/0, short=0/0/0

     lat (msec): 4=0.04%, 10=5.48%, 20=45.05%, 50=46.73%, 100=2.70%
     lat (msec): 250=0.01%, 500=0.01%
/mnt/vol100/fio1.dat: (groupid=0, jobs=1): err= 0: pid=7351
  read : io=161924KB, bw=2698.5KB/s, iops=674 , runt= 60007msec
    slat (usec): min=3 , max=62012 , avg=665.18, stdev=3519.66
    clat (msec): min=2 , max=270 , avg=23.05, stdev=12.45
     lat (msec): min=2 , max=270 , avg=23.71, stdev=12.66
    bw (KB/s) : min= 1886, max= 3388, per=6.03%, avg=2699.02, stdev=270.55
  cpu          : usr=0.35%, sys=1.01%, ctx=11004, majf=0, minf=40
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued r/w/d: total=40481/0/0, short=0/0/0

     lat (msec): 4=0.05%, 10=5.41%, 20=46.33%, 50=44.89%, 100=3.26%
     lat (msec): 250=0.06%, 500=0.01%
/mnt/vol100/fio1.dat: (groupid=0, jobs=1): err= 0: pid=7352
  read : io=165640KB, bw=2760.2KB/s, iops=690 , runt= 60011msec
    slat (usec): min=3 , max=58065 , avg=661.31, stdev=3521.79
    clat (msec): min=2 , max=344 , avg=22.52, stdev=12.40
     lat (msec): min=2 , max=344 , avg=23.18, stdev=12.64
    bw (KB/s) : min= 1980, max= 3482, per=6.17%, avg=2760.69, stdev=258.08
  cpu          : usr=0.32%, sys=1.04%, ctx=11036, majf=0, minf=39
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued r/w/d: total=41410/0/0, short=0/0/0

     lat (msec): 4=0.10%, 10=5.39%, 20=48.78%, 50=42.28%, 100=3.42%
     lat (msec): 250=0.02%, 500=0.01%
/mnt/vol100/fio1.dat: (groupid=0, jobs=1): err= 0: pid=7353
  read : io=165468KB, bw=2757.4KB/s, iops=689 , runt= 60009msec
    slat (usec): min=3 , max=62005 , avg=691.08, stdev=3646.13
    clat (msec): min=2 , max=101 , avg=22.52, stdev=12.43
     lat (msec): min=2 , max=107 , avg=23.21, stdev=12.69
    bw (KB/s) : min= 2019, max= 3304, per=6.17%, avg=2759.37, stdev=268.86
  cpu          : usr=0.35%, sys=1.01%, ctx=11075, majf=0, minf=39
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued r/w/d: total=41367/0/0, short=0/0/0

     lat (msec): 4=0.02%, 10=5.41%, 20=48.88%, 50=42.24%, 100=3.44%
     lat (msec): 250=0.02%
/mnt/vol100/fio2.dat: (groupid=0, jobs=1): err= 0: pid=7354
  read : io=166748KB, bw=2778.8KB/s, iops=694 , runt= 60009msec
    slat (usec): min=3 , max=66237 , avg=681.94, stdev=3643.96
    clat (msec): min=3 , max=378 , avg=22.35, stdev=12.99
     lat (msec): min=3 , max=378 , avg=23.03, stdev=13.24
    bw (KB/s) : min= 1790, max= 3465, per=6.21%, avg=2780.33, stdev=297.34
  cpu          : usr=0.31%, sys=1.05%, ctx=11170, majf=0, minf=40
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued r/w/d: total=41687/0/0, short=0/0/0

     lat (msec): 4=0.02%, 10=5.68%, 20=49.89%, 50=40.70%, 100=3.64%
     lat (msec): 250=0.07%, 500=0.01%
/mnt/vol100/fio2.dat: (groupid=0, jobs=1): err= 0: pid=7355
  read : io=165672KB, bw=2760.8KB/s, iops=690 , runt= 60010msec
    slat (usec): min=3 , max=70020 , avg=702.33, stdev=3744.67
    clat (msec): min=3 , max=366 , avg=22.48, stdev=12.94
     lat (msec): min=3 , max=366 , avg=23.18, stdev=13.22
    bw (KB/s) : min= 1839, max= 3311, per=6.17%, avg=2761.53, stdev=285.31
  cpu          : usr=0.31%, sys=1.05%, ctx=11160, majf=0, minf=39
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued r/w/d: total=41418/0/0, short=0/0/0

     lat (msec): 4=0.06%, 10=5.39%, 20=50.37%, 50=40.49%, 100=3.63%
     lat (msec): 250=0.06%, 500=0.01%
/mnt/vol100/fio2.dat: (groupid=0, jobs=1): err= 0: pid=7356
  read : io=168288KB, bw=2804.6KB/s, iops=701 , runt= 60005msec
    slat (usec): min=3 , max=67128 , avg=686.66, stdev=3671.67
    clat (msec): min=2 , max=348 , avg=22.13, stdev=12.74
     lat (msec): min=2 , max=348 , avg=22.82, stdev=13.00
    bw (KB/s) : min= 2226, max= 3408, per=6.27%, avg=2806.14, stdev=274.65
  cpu          : usr=0.29%, sys=1.07%, ctx=11135, majf=0, minf=39
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued r/w/d: total=42072/0/0, short=0/0/0

     lat (msec): 4=0.07%, 10=5.93%, 20=50.43%, 50=39.92%, 100=3.63%
     lat (msec): 250=0.01%, 500=0.01%
/mnt/vol100/fio2.dat: (groupid=0, jobs=1): err= 0: pid=7357
  read : io=168880KB, bw=2814.5KB/s, iops=703 , runt= 60005msec
    slat (usec): min=3 , max=58094 , avg=683.15, stdev=3640.38
    clat (msec): min=3 , max=278 , avg=22.05, stdev=12.46
     lat (msec): min=3 , max=278 , avg=22.74, stdev=12.74
    bw (KB/s) : min= 2118, max= 3344, per=6.29%, avg=2814.33, stdev=283.79
  cpu          : usr=0.30%, sys=1.06%, ctx=11145, majf=0, minf=40
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued r/w/d: total=42220/0/0, short=0/0/0

     lat (msec): 4=0.02%, 10=5.90%, 20=51.41%, 50=39.44%, 100=3.21%
     lat (msec): 250=0.02%, 500=0.01%
/mnt/vol100/fio3.dat: (groupid=0, jobs=1): err= 0: pid=7358
  read : io=168704KB, bw=2811.4KB/s, iops=702 , runt= 60008msec
    slat (usec): min=3 , max=62164 , avg=691.08, stdev=3696.64
    clat (msec): min=2 , max=387 , avg=22.07, stdev=13.11
     lat (msec): min=2 , max=387 , avg=22.76, stdev=13.40
    bw (KB/s) : min= 2059, max= 3561, per=6.28%, avg=2811.52, stdev=282.89
  cpu          : usr=0.37%, sys=0.99%, ctx=11187, majf=0, minf=39
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued r/w/d: total=42176/0/0, short=0/0/0

     lat (msec): 4=0.04%, 10=6.00%, 20=51.93%, 50=38.45%, 100=3.57%
     lat (msec): 250=0.01%, 500=0.01%
/mnt/vol100/fio3.dat: (groupid=0, jobs=1): err= 0: pid=7359
  read : io=169788KB, bw=2829.5KB/s, iops=707 , runt= 60007msec
    slat (usec): min=3 , max=76246 , avg=680.01, stdev=3662.78
    clat (msec): min=3 , max=320 , avg=21.93, stdev=12.99
     lat (msec): min=3 , max=339 , avg=22.61, stdev=13.28
    bw (KB/s) : min= 2135, max= 3405, per=6.32%, avg=2829.31, stdev=272.69
  cpu          : usr=0.25%, sys=1.11%, ctx=11082, majf=0, minf=38
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued r/w/d: total=42447/0/0, short=0/0/0

     lat (msec): 4=0.06%, 10=5.95%, 20=52.64%, 50=37.37%, 100=3.95%
     lat (msec): 250=0.04%, 500=0.01%
/mnt/vol100/fio3.dat: (groupid=0, jobs=1): err= 0: pid=7360
  read : io=171264KB, bw=2854.2KB/s, iops=713 , runt= 60005msec
    slat (usec): min=3 , max=72056 , avg=671.18, stdev=3599.99
    clat (msec): min=2 , max=195 , avg=21.75, stdev=12.52
     lat (msec): min=3 , max=195 , avg=22.42, stdev=12.80
    bw (KB/s) : min= 2103, max= 3576, per=6.38%, avg=2856.08, stdev=301.46
  cpu          : usr=0.30%, sys=1.06%, ctx=11051, majf=0, minf=39
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued r/w/d: total=42816/0/0, short=0/0/0

     lat (msec): 4=0.01%, 10=6.88%, 20=50.72%, 50=38.71%, 100=3.68%
     lat (msec): 250=0.01%
/mnt/vol100/fio3.dat: (groupid=0, jobs=1): err= 0: pid=7361
  read : io=171524KB, bw=2858.5KB/s, iops=714 , runt= 60006msec
    slat (usec): min=3 , max=64539 , avg=667.97, stdev=3582.54
    clat (usec): min=85 , max=314834 , avg=21716.06, stdev=12414.74
     lat (msec): min=1 , max=314 , avg=22.38, stdev=12.70
    bw (KB/s) : min= 2071, max= 3509, per=6.39%, avg=2860.47, stdev=290.00
  cpu          : usr=0.33%, sys=1.03%, ctx=10971, majf=0, minf=38
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued r/w/d: total=42881/0/0, short=0/0/0
     lat (usec): 100=0.01%
     lat (msec): 2=0.01%, 4=0.06%, 10=6.35%, 20=51.37%, 50=38.91%
     lat (msec): 100=3.31%, 250=0.01%, 500=0.01%
/mnt/vol100/fio4.dat: (groupid=0, jobs=1): err= 0: pid=7362
  read : io=172072KB, bw=2867.6KB/s, iops=716 , runt= 60006msec
    slat (usec): min=3 , max=79368 , avg=661.15, stdev=3544.86
    clat (msec): min=2 , max=186 , avg=21.65, stdev=12.25
     lat (msec): min=2 , max=186 , avg=22.31, stdev=12.54
    bw (KB/s) : min= 2007, max= 3392, per=6.41%, avg=2868.47, stdev=270.18
  cpu          : usr=0.36%, sys=1.00%, ctx=10993, majf=0, minf=38
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued r/w/d: total=43018/0/0, short=0/0/0

     lat (msec): 4=0.05%, 10=5.48%, 20=52.04%, 50=39.13%, 100=3.23%
     lat (msec): 250=0.07%
/mnt/vol100/fio4.dat: (groupid=0, jobs=1): err= 0: pid=7363
  read : io=169468KB, bw=2824.3KB/s, iops=706 , runt= 60004msec
    slat (usec): min=3 , max=65619 , avg=675.52, stdev=3647.77
    clat (msec): min=2 , max=230 , avg=21.98, stdev=12.87
     lat (msec): min=2 , max=230 , avg=22.66, stdev=13.17
    bw (KB/s) : min= 1952, max= 3578, per=6.31%, avg=2824.72, stdev=281.90
  cpu          : usr=0.37%, sys=0.99%, ctx=10728, majf=0, minf=39
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued r/w/d: total=42367/0/0, short=0/0/0

     lat (msec): 4=0.15%, 10=6.48%, 20=50.66%, 50=39.07%, 100=3.60%
     lat (msec): 250=0.04%
/mnt/vol100/fio4.dat: (groupid=0, jobs=1): err= 0: pid=7364
  read : io=168416KB, bw=2806.9KB/s, iops=701 , runt= 60003msec
    slat (usec): min=3 , max=68152 , avg=681.22, stdev=3651.74
    clat (msec): min=2 , max=295 , avg=22.11, stdev=12.79
     lat (msec): min=2 , max=305 , avg=22.80, stdev=13.09
    bw (KB/s) : min= 2023, max= 3702, per=6.28%, avg=2808.77, stdev=280.65
  cpu          : usr=0.37%, sys=0.99%, ctx=10643, majf=0, minf=38
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued r/w/d: total=42104/0/0, short=0/0/0

     lat (msec): 4=0.06%, 10=5.81%, 20=51.47%, 50=39.06%, 100=3.56%
     lat (msec): 250=0.04%, 500=0.01%
/mnt/vol100/fio4.dat: (groupid=0, jobs=1): err= 0: pid=7365
  read : io=170928KB, bw=2848.8KB/s, iops=712 , runt= 60002msec
    slat (usec): min=3 , max=60894 , avg=658.31, stdev=3544.25
    clat (msec): min=1 , max=242 , avg=21.80, stdev=12.23
     lat (msec): min=3 , max=242 , avg=22.46, stdev=12.55
    bw (KB/s) : min= 2031, max= 3479, per=6.36%, avg=2848.56, stdev=285.64
  cpu          : usr=0.37%, sys=0.99%, ctx=10617, majf=0, minf=38
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued r/w/d: total=42732/0/0, short=0/0/0

     lat (msec): 2=0.01%, 4=0.04%, 10=6.41%, 20=50.31%, 50=40.30%
     lat (msec): 100=2.84%, 250=0.10%

Run status group 0 (all jobs):
   READ: io=2622.9MB, aggrb=44754KB/s, minb=2747KB/s, maxb=2936KB/s, mint=60002msec, maxt=60011msec

Disk stats (read/write):
  vdc: ios=670885/0, merge=5/0, ticks=9437960/0, in_queue=9438540, util=99.93%