Advertisement
Not a member of Pastebin yet?
Sign Up,
it unlocks many cool features!
- root@pve ~ # cat bench-4k-IOD16.txt
- seqRead: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
- seqWrite: (g=1): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
- seqWrite/sync: (g=2): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
- randRead: (g=3): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
- randWrite: (g=4): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
- randWrite/sync: (g=5): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
- randReadWrite: (g=6): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
- randReadWrite/sync: (g=7): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
- fio-3.12
- Starting 8 processes
- seqRead: (groupid=0, jobs=1): err= 0: pid=32584: Wed Apr 1 11:42:05 2020
- read: IOPS=184k, BW=718MiB/s (753MB/s)(1024MiB/1426msec)
- slat (nsec): min=624, max=4677.8k, avg=4933.25, stdev=45796.73
- clat (nsec): min=860, max=4707.7k, avg=81887.45, stdev=171059.44
- lat (nsec): min=1547, max=4708.6k, avg=86850.31, stdev=175945.15
- clat percentiles (usec):
- | 1.00th=[ 17], 5.00th=[ 17], 10.00th=[ 17], 20.00th=[ 17],
- | 30.00th=[ 17], 40.00th=[ 18], 50.00th=[ 18], 60.00th=[ 18],
- | 70.00th=[ 19], 80.00th=[ 55], 90.00th=[ 400], 95.00th=[ 529],
- | 99.00th=[ 676], 99.50th=[ 725], 99.90th=[ 865], 99.95th=[ 930],
- | 99.99th=[ 4424]
- bw ( KiB/s): min=580186, max=730000, per=89.09%, avg=655093.00, stdev=105934.50, samples=2
- iops : min=145046, max=182500, avg=163773.00, stdev=26483.98, samples=2
- lat (nsec) : 1000=0.01%
- lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=74.04%, 50=5.67%
- lat (usec) : 100=7.82%, 250=0.65%, 500=5.17%, 750=6.25%, 1000=0.38%
- lat (msec) : 2=0.01%, 10=0.01%
- cpu : usr=10.11%, sys=24.56%, ctx=2209, majf=0, minf=23
- IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=262144,0,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=16
- seqWrite: (groupid=1, jobs=1): err= 0: pid=32585: Wed Apr 1 11:42:05 2020
- write: IOPS=498k, BW=1947MiB/s (2041MB/s)(1024MiB/526msec); 0 zone resets
- slat (nsec): min=1020, max=25641, avg=1176.33, stdev=330.44
- clat (nsec): min=789, max=93417, avg=30291.28, stdev=1768.53
- lat (nsec): min=1900, max=100659, avg=31489.51, stdev=1827.02
- clat percentiles (nsec):
- | 1.00th=[29312], 5.00th=[29312], 10.00th=[29568], 20.00th=[29568],
- | 30.00th=[29824], 40.00th=[30080], 50.00th=[30080], 60.00th=[30336],
- | 70.00th=[30336], 80.00th=[30592], 90.00th=[31104], 95.00th=[31360],
- | 99.00th=[33024], 99.50th=[35072], 99.90th=[60672], 99.95th=[64256],
- | 99.99th=[77312]
- lat (nsec) : 1000=0.01%
- lat (usec) : 4=0.01%, 10=0.01%, 20=0.01%, 50=99.78%, 100=0.21%
- cpu : usr=58.48%, sys=40.95%, ctx=0, majf=0, minf=9
- IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=0,262144,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=16
- seqWrite/sync: (groupid=2, jobs=1): err= 0: pid=32590: Wed Apr 1 11:42:05 2020
- write: IOPS=1974, BW=7900KiB/s (8089kB/s)(463MiB/60001msec); 0 zone resets
- slat (nsec): min=1045, max=46917, avg=9977.16, stdev=5181.60
- clat (usec): min=423, max=23221, avg=7596.10, stdev=1062.35
- lat (usec): min=435, max=23236, avg=7606.34, stdev=1063.46
- clat percentiles (usec):
- | 1.00th=[ 5538], 5.00th=[ 5932], 10.00th=[ 6456], 20.00th=[ 6915],
- | 30.00th=[ 7177], 40.00th=[ 7373], 50.00th=[ 7570], 60.00th=[ 7767],
- | 70.00th=[ 8094], 80.00th=[ 8291], 90.00th=[ 8586], 95.00th=[ 8586],
- | 99.00th=[12125], 99.50th=[12518], 99.90th=[18482], 99.95th=[19006],
- | 99.99th=[19530]
- bw ( KiB/s): min= 5787, max= 7695, per=79.36%, avg=6268.51, stdev=268.70, samples=119
- iops : min= 1446, max= 1923, avg=1566.72, stdev=67.12, samples=119
- lat (usec) : 500=0.01%, 1000=0.01%
- lat (msec) : 2=0.01%, 4=0.01%, 10=98.24%, 20=1.74%, 50=0.01%
- fsync/fdatasync/sync_file_range:
- sync (nsec): min=38, max=17445, avg=334.12, stdev=466.37
- sync percentiles (nsec):
- | 1.00th=[ 111], 5.00th=[ 131], 10.00th=[ 131], 20.00th=[ 133],
- | 30.00th=[ 137], 40.00th=[ 145], 50.00th=[ 193], 60.00th=[ 458],
- | 70.00th=[ 466], 80.00th=[ 478], 90.00th=[ 486], 95.00th=[ 716],
- | 99.00th=[ 796], 99.50th=[ 836], 99.90th=[ 924], 99.95th=[13888],
- | 99.99th=[15296]
- cpu : usr=2.79%, sys=10.74%, ctx=354313, majf=0, minf=11
- IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=200.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=0,118495,0,118481 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=16
- randRead: (groupid=3, jobs=1): err= 0: pid=389: Wed Apr 1 11:42:05 2020
- read: IOPS=12.9k, BW=50.5MiB/s (52.0MB/s)(1024MiB/20261msec)
- slat (usec): min=37, max=4686, avg=76.37, stdev=52.06
- clat (nsec): min=1046, max=8947.0k, avg=1159681.20, stdev=469617.00
- lat (usec): min=74, max=9289, avg=1236.16, stdev=492.91
- clat percentiles (usec):
- | 1.00th=[ 1057], 5.00th=[ 1057], 10.00th=[ 1057], 20.00th=[ 1057],
- | 30.00th=[ 1057], 40.00th=[ 1057], 50.00th=[ 1057], 60.00th=[ 1057],
- | 70.00th=[ 1074], 80.00th=[ 1074], 90.00th=[ 1074], 95.00th=[ 1450],
- | 99.00th=[ 3720], 99.50th=[ 3916], 99.90th=[ 5211], 99.95th=[ 6587],
- | 99.99th=[ 8356]
- bw ( KiB/s): min=31328, max=53672, per=68.50%, avg=35449.93, stdev=3165.01, samples=40
- iops : min= 7832, max=13418, avg=8862.17, stdev=791.30, samples=40
- lat (usec) : 2=0.01%, 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%
- lat (usec) : 1000=0.01%
- lat (msec) : 2=96.10%, 4=3.54%, 10=0.35%
- cpu : usr=1.95%, sys=5.29%, ctx=262144, majf=0, minf=23
- IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=262144,0,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=16
- randWrite: (groupid=4, jobs=1): err= 0: pid=494: Wed Apr 1 11:42:05 2020
- write: IOPS=462k, BW=1806MiB/s (1894MB/s)(1024MiB/567msec); 0 zone resets
- slat (nsec): min=1035, max=108431, avg=1283.26, stdev=495.18
- clat (nsec): min=780, max=146998, avg=32679.40, stdev=2934.37
- lat (nsec): min=1915, max=148662, avg=33984.89, stdev=3025.55
- clat percentiles (nsec):
- | 1.00th=[31104], 5.00th=[31616], 10.00th=[31616], 20.00th=[31872],
- | 30.00th=[32128], 40.00th=[32128], 50.00th=[32384], 60.00th=[32640],
- | 70.00th=[32640], 80.00th=[33024], 90.00th=[33536], 95.00th=[33536],
- | 99.00th=[36608], 99.50th=[54528], 99.90th=[76288], 99.95th=[82432],
- | 99.99th=[96768]
- lat (nsec) : 1000=0.01%
- lat (usec) : 4=0.01%, 10=0.01%, 20=0.01%, 50=99.42%, 100=0.57%
- lat (usec) : 250=0.01%
- cpu : usr=48.41%, sys=51.24%, ctx=4, majf=0, minf=7
- IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=0,262144,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=16
- randWrite/sync: (groupid=5, jobs=1): err= 0: pid=537: Wed Apr 1 11:42:05 2020
- write: IOPS=1970, BW=7883KiB/s (8072kB/s)(462MiB/60001msec); 0 zone resets
- slat (nsec): min=1204, max=49176, avg=10065.25, stdev=4990.82
- clat (usec): min=462, max=18759, avg=7612.33, stdev=1052.94
- lat (usec): min=475, max=18774, avg=7622.67, stdev=1054.11
- clat percentiles (usec):
- | 1.00th=[ 5407], 5.00th=[ 5932], 10.00th=[ 6456], 20.00th=[ 6915],
- | 30.00th=[ 7177], 40.00th=[ 7373], 50.00th=[ 7570], 60.00th=[ 7832],
- | 70.00th=[ 8094], 80.00th=[ 8356], 90.00th=[ 8586], 95.00th=[ 8717],
- | 99.00th=[12256], 99.50th=[12649], 99.90th=[13960], 99.95th=[14353],
- | 99.99th=[18220]
- bw ( KiB/s): min= 5015, max= 8078, per=70.35%, avg=5544.88, stdev=357.76, samples=119
- iops : min= 1253, max= 2019, avg=1385.84, stdev=89.43, samples=119
- lat (usec) : 500=0.01%, 750=0.01%
- lat (msec) : 2=0.01%, 4=0.01%, 10=98.13%, 20=1.86%
- fsync/fdatasync/sync_file_range:
- sync (nsec): min=10, max=19586, avg=324.28, stdev=421.06
- sync percentiles (nsec):
- | 1.00th=[ 114], 5.00th=[ 131], 10.00th=[ 133], 20.00th=[ 135],
- | 30.00th=[ 137], 40.00th=[ 157], 50.00th=[ 199], 60.00th=[ 442],
- | 70.00th=[ 450], 80.00th=[ 454], 90.00th=[ 490], 95.00th=[ 676],
- | 99.00th=[ 764], 99.50th=[ 812], 99.90th=[ 876], 99.95th=[13632],
- | 99.99th=[15424]
- cpu : usr=2.78%, sys=10.83%, ctx=353119, majf=0, minf=8
- IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=200.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=0,118244,0,118230 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=16
- randReadWrite: (groupid=6, jobs=1): err= 0: pid=855: Wed Apr 1 11:42:05 2020
- read: IOPS=12.8k, BW=49.9MiB/s (52.3MB/s)(511MiB/10237msec)
- slat (usec): min=35, max=5515, avg=74.73, stdev=58.12
- clat (nsec): min=989, max=32692k, avg=585884.32, stdev=300376.61
- lat (usec): min=73, max=36654, avg=660.71, stdev=331.57
- clat percentiles (usec):
- | 1.00th=[ 243], 5.00th=[ 314], 10.00th=[ 379], 20.00th=[ 449],
- | 30.00th=[ 515], 40.00th=[ 519], 50.00th=[ 586], 60.00th=[ 586],
- | 70.00th=[ 652], 80.00th=[ 660], 90.00th=[ 725], 95.00th=[ 799],
- | 99.00th=[ 1827], 99.50th=[ 2278], 99.90th=[ 3064], 99.95th=[ 4621],
- | 99.99th=[ 5473]
- bw ( KiB/s): min=36743, max=46536, per=76.23%, avg=38956.35, stdev=2054.22, samples=20
- iops : min= 9185, max=11634, avg=9738.75, stdev=513.69, samples=20
- write: IOPS=12.8k, BW=50.1MiB/s (52.6MB/s)(513MiB/10237msec); 0 zone resets
- slat (nsec): min=1050, max=31458, avg=1629.24, stdev=939.69
- clat (usec): min=73, max=36633, avg=583.68, stdev=285.96
- lat (usec): min=74, max=36645, avg=585.33, stdev=286.26
- clat percentiles (usec):
- | 1.00th=[ 243], 5.00th=[ 314], 10.00th=[ 379], 20.00th=[ 449],
- | 30.00th=[ 506], 40.00th=[ 519], 50.00th=[ 586], 60.00th=[ 586],
- | 70.00th=[ 652], 80.00th=[ 660], 90.00th=[ 725], 95.00th=[ 799],
- | 99.00th=[ 1795], 99.50th=[ 2245], 99.90th=[ 2999], 99.95th=[ 4621],
- | 99.99th=[ 4948]
- bw ( KiB/s): min=36809, max=46320, per=76.21%, avg=39114.55, stdev=2071.59, samples=20
- iops : min= 9202, max=11580, avg=9778.30, stdev=517.95, samples=20
- lat (nsec) : 1000=0.01%
- lat (usec) : 100=0.01%, 250=1.73%, 500=28.09%, 750=61.28%, 1000=6.43%
- lat (msec) : 2=1.69%, 4=0.70%, 10=0.06%, 20=0.01%, 50=0.01%
- cpu : usr=3.07%, sys=7.20%, ctx=130785, majf=0, minf=10
- IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=130785,131359,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=16
- randReadWrite/sync: (groupid=7, jobs=1): err= 0: pid=908: Wed Apr 1 11:42:05 2020
- read: IOPS=1655, BW=6623KiB/s (6782kB/s)(388MiB/60001msec)
- slat (usec): min=40, max=4493, avg=99.23, stdev=69.12
- clat (usec): min=952, max=16732, avg=4391.86, stdev=1216.23
- lat (usec): min=1187, max=16805, avg=4491.26, stdev=1247.66
- clat percentiles (usec):
- | 1.00th=[ 2835], 5.00th=[ 3097], 10.00th=[ 3261], 20.00th=[ 3458],
- | 30.00th=[ 3621], 40.00th=[ 3818], 50.00th=[ 4047], 60.00th=[ 4424],
- | 70.00th=[ 4817], 80.00th=[ 5211], 90.00th=[ 5735], 95.00th=[ 6325],
- | 99.00th=[ 8979], 99.50th=[10290], 99.90th=[12387], 99.95th=[12911],
- | 99.99th=[14484]
- bw ( KiB/s): min= 5061, max= 8480, per=99.98%, avg=6620.82, stdev=870.14, samples=120
- iops : min= 1265, max= 2120, avg=1655.17, stdev=217.55, samples=120
- write: IOPS=1651, BW=6606KiB/s (6764kB/s)(387MiB/60001msec); 0 zone resets
- slat (nsec): min=1178, max=52605, avg=3516.28, stdev=2889.87
- clat (usec): min=545, max=16539, avg=4680.23, stdev=1244.93
- lat (usec): min=560, max=16541, avg=4683.82, stdev=1245.99
- clat percentiles (usec):
- | 1.00th=[ 3097], 5.00th=[ 3359], 10.00th=[ 3523], 20.00th=[ 3720],
- | 30.00th=[ 3884], 40.00th=[ 4080], 50.00th=[ 4293], 60.00th=[ 4686],
- | 70.00th=[ 5145], 80.00th=[ 5538], 90.00th=[ 6128], 95.00th=[ 6652],
- | 99.00th=[ 9241], 99.50th=[10683], 99.90th=[12780], 99.95th=[13435],
- | 99.99th=[15533]
- bw ( KiB/s): min= 5064, max= 8304, per=99.98%, avg=6603.51, stdev=839.70, samples=120
- iops : min= 1266, max= 2076, avg=1650.83, stdev=209.92, samples=120
- lat (usec) : 750=0.01%, 1000=0.01%
- lat (msec) : 2=0.01%, 4=42.35%, 10=57.00%, 20=0.65%
- fsync/fdatasync/sync_file_range:
- sync (nsec): min=9, max=21726, avg=175.76, stdev=938.74
- sync percentiles (nsec):
- | 1.00th=[ 35], 5.00th=[ 44], 10.00th=[ 50], 20.00th=[ 64],
- | 30.00th=[ 75], 40.00th=[ 84], 50.00th=[ 97], 60.00th=[ 113],
- | 70.00th=[ 131], 80.00th=[ 135], 90.00th=[ 155], 95.00th=[ 394],
- | 99.00th=[ 612], 99.50th=[ 812], 99.90th=[15936], 99.95th=[16064],
- | 99.99th=[18048]
- cpu : usr=3.27%, sys=7.62%, ctx=591455, majf=0, minf=10
- IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=200.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=99346,99091,0,198423 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=16
- Run status group 0 (all jobs):
- READ: bw=718MiB/s (753MB/s), 718MiB/s-718MiB/s (753MB/s-753MB/s), io=1024MiB (1074MB), run=1426-1426msec
- Run status group 1 (all jobs):
- WRITE: bw=1947MiB/s (2041MB/s), 1947MiB/s-1947MiB/s (2041MB/s-2041MB/s), io=1024MiB (1074MB), run=526-526msec
- Run status group 2 (all jobs):
- WRITE: bw=7900KiB/s (8089kB/s), 7900KiB/s-7900KiB/s (8089kB/s-8089kB/s), io=463MiB (485MB), run=60001-60001msec
- Run status group 3 (all jobs):
- READ: bw=50.5MiB/s (52.0MB/s), 50.5MiB/s-50.5MiB/s (52.0MB/s-52.0MB/s), io=1024MiB (1074MB), run=20261-20261msec
- Run status group 4 (all jobs):
- WRITE: bw=1806MiB/s (1894MB/s), 1806MiB/s-1806MiB/s (1894MB/s-1894MB/s), io=1024MiB (1074MB), run=567-567msec
- Run status group 5 (all jobs):
- WRITE: bw=7883KiB/s (8072kB/s), 7883KiB/s-7883KiB/s (8072kB/s-8072kB/s), io=462MiB (484MB), run=60001-60001msec
- Run status group 6 (all jobs):
- READ: bw=49.9MiB/s (52.3MB/s), 49.9MiB/s-49.9MiB/s (52.3MB/s-52.3MB/s), io=511MiB (536MB), run=10237-10237msec
- WRITE: bw=50.1MiB/s (52.6MB/s), 50.1MiB/s-50.1MiB/s (52.6MB/s-52.6MB/s), io=513MiB (538MB), run=10237-10237msec
- Run status group 7 (all jobs):
- READ: bw=6623KiB/s (6782kB/s), 6623KiB/s-6623KiB/s (6782kB/s-6782kB/s), io=388MiB (407MB), run=60001-60001msec
- WRITE: bw=6606KiB/s (6764kB/s), 6606KiB/s-6606KiB/s (6764kB/s-6764kB/s), io=387MiB (406MB), run=60001-60001msec
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement