Advertisement
Not a member of Pastebin yet?
Sign Up,
it unlocks many cool features!
- <example-benchmark-fio-server-951-benchmark-9cd4p>
- randwrite: (groupid=0, jobs=1): err= 0: pid=11: Wed May 8 20:34:54 2019
- write: IOPS=36, BW=2354KiB/s (2411kB/s)(276MiB/120060msec)
- slat (nsec): min=0, max=504592k, avg=441726.08, stdev=10395455.15
- clat (nsec): min=0, max=995152k, avg=108771492.32, stdev=167527907.41
- lat (nsec): min=0, max=995235k, avg=109214317.91, stdev=167771537.56
- clat percentiles (usec):
- | 1.00th=[ 988], 5.00th=[ 1172], 10.00th=[ 1336], 20.00th=[ 1778],
- | 30.00th=[ 2671], 40.00th=[ 6194], 50.00th=[ 34341], 60.00th=[ 77071],
- | 70.00th=[107480], 80.00th=[158335], 90.00th=[358613], 95.00th=[501220],
- | 99.00th=[759170], 99.50th=[868221], 99.90th=[935330], 99.95th=[935330],
- | 99.99th=[994051]
- bw ( KiB/s): min= 0, max= 3002, per=52.90%, avg=2352.50, stdev=918.53, samples=2
- iops : min= 0, max= 46, avg=36.00, stdev=14.14, samples=2
- lat (usec) : 4=0.02%, 500=0.02%, 750=0.07%, 1000=1.13%
- lat (msec) : 2=22.09%, 4=12.19%, 10=7.36%, 20=4.44%, 50=4.51%
- lat (msec) : 100=15.57%, 250=17.99%, 500=9.61%, 750=4.03%, 1000=1.02%
- cpu : usr=0.04%, sys=0.16%, ctx=8445, majf=0, minf=31
- IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=0,4413,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- read: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:34:54 2019
- read: IOPS=348, BW=21.8MiB/s (22.8MB/s)(2614MiB/120030msec)
- slat (nsec): min=0, max=498780k, avg=50078.56, stdev=3579794.07
- clat (nsec): min=0, max=937417k, avg=11464988.97, stdev=59818482.14
- lat (nsec): min=0, max=937452k, avg=11515884.91, stdev=59935424.53
- clat percentiles (nsec):
- | 1.00th=[ 708], 5.00th=[ 716], 10.00th=[ 716],
- | 20.00th=[ 724], 30.00th=[ 732], 40.00th=[ 740],
- | 50.00th=[ 764], 60.00th=[ 852], 70.00th=[ 924],
- | 80.00th=[ 1288], 90.00th=[ 1941504], 95.00th=[ 51118080],
- | 99.00th=[362807296], 99.50th=[471859200], 99.90th=[692060160],
- | 99.95th=[734003200], 99.99th=[843055104]
- bw ( KiB/s): min= 0, max=28792, per=91.76%, avg=22304.00, stdev=9175.42, samples=2
- iops : min= 0, max= 450, avg=348.50, stdev=143.54, samples=2
- lat (nsec) : 750=47.82%, 1000=25.51%
- lat (usec) : 2=9.42%, 4=0.02%, 10=0.63%, 20=0.59%, 50=0.28%
- lat (usec) : 100=0.10%, 250=0.04%, 500=0.29%, 750=1.01%, 1000=1.91%
- lat (msec) : 2=2.44%, 4=1.35%, 10=0.63%, 20=0.48%, 50=2.41%
- lat (msec) : 100=2.11%, 250=1.42%, 500=1.13%, 750=0.38%, 1000=0.03%
- cpu : usr=0.11%, sys=0.37%, ctx=12986, majf=0, minf=31
- IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=41881,0,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- readwrite: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:34:54 2019
- read: IOPS=31, BW=2014KiB/s (2062kB/s)(236MiB/120065msec)
- slat (nsec): min=0, max=464438k, avg=251788.17, stdev=10003721.86
- clat (nsec): min=0, max=761107k, avg=7488674.89, stdev=41769913.15
- lat (nsec): min=0, max=761136k, avg=7741358.10, stdev=42918867.18
- clat percentiles (nsec):
- | 1.00th=[ 708], 5.00th=[ 724], 10.00th=[ 732],
- | 20.00th=[ 756], 30.00th=[ 844], 40.00th=[ 924],
- | 50.00th=[ 1080], 60.00th=[ 1208], 70.00th=[ 1336],
- | 80.00th=[ 1624], 90.00th=[ 1204224], 95.00th=[ 38535168],
- | 99.00th=[214958080], 99.50th=[325058560], 99.90th=[534773760],
- | 99.95th=[675282944], 99.99th=[759169024]
- bw ( KiB/s): min= 0, max= 2421, per=8.29%, avg=2014.50, stdev=574.88, samples=2
- iops : min= 0, max= 37, avg=31.00, stdev= 8.49, samples=2
- write: IOPS=32, BW=2094KiB/s (2144kB/s)(246MiB/120065msec)
- slat (nsec): min=0, max=552114k, avg=319489.42, stdev=8814876.07
- clat (nsec): min=0, max=1106.9M, avg=115024253.69, stdev=172725727.36
- lat (nsec): min=0, max=1107.2M, avg=115344697.54, stdev=172831915.20
- clat percentiles (usec):
- | 1.00th=[ 1020], 5.00th=[ 1205], 10.00th=[ 1401],
- | 20.00th=[ 1876], 30.00th=[ 2900], 40.00th=[ 8029],
- | 50.00th=[ 46924], 60.00th=[ 82314], 70.00th=[ 111674],
- | 80.00th=[ 175113], 90.00th=[ 371196], 95.00th=[ 509608],
- | 99.00th=[ 759170], 99.50th=[ 876610], 99.90th=[ 960496],
- | 99.95th=[ 994051], 99.99th=[1098908]
- bw ( KiB/s): min= 0, max= 2517, per=47.05%, avg=2092.50, stdev=600.33, samples=2
- iops : min= 0, max= 39, avg=32.50, stdev= 9.19, samples=2
- lat (nsec) : 750=9.36%, 1000=12.80%
- lat (usec) : 2=17.65%, 4=0.09%, 10=0.25%, 20=0.70%, 50=0.53%
- lat (usec) : 100=0.62%, 250=0.32%, 500=0.42%, 750=0.45%, 1000=1.00%
- lat (msec) : 2=11.62%, 4=6.70%, 10=3.98%, 20=2.15%, 50=4.18%
- lat (msec) : 100=9.19%, 250=9.53%, 500=5.70%, 750=2.26%, 1000=0.52%
- cpu : usr=0.04%, sys=0.19%, ctx=8692, majf=0, minf=34
- IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=3779,3925,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- Run status group 0 (all jobs):
- READ: bw=23.7MiB/s (24.9MB/s), 2014KiB/s-21.8MiB/s (2062kB/s-22.8MB/s), io=2850MiB (2989MB), run=120030-120065msec
- WRITE: bw=4448KiB/s (4554kB/s), 2094KiB/s-2354KiB/s (2144kB/s-2411kB/s), io=522MiB (547MB), run=120060-120065msec
- Disk stats (read/write):
- rbd300: ios=7336/8516, merge=0/1, ticks=525672/830529, in_queue=1348119, util=12.06%
- Jobs: 117 (f=116): [_(2),m(1)][99.9%][r=1334M,w=800M][r=20.8k,w=12.5k IOPS][eta 00m:56s] <example-benchmark-fio-server-255-benchmark-v6glb>
- randwrite: (groupid=0, jobs=1): err= 0: pid=11: Wed May 8 20:34:54 2019
- write: IOPS=33, BW=2171KiB/s (2223kB/s)(254MiB/120036msec)
- slat (nsec): min=0, max=536048k, avg=417781.11, stdev=10887713.42
- clat (nsec): min=0, max=1261.6M, avg=118027748.44, stdev=178274062.80
- lat (nsec): min=0, max=1261.6M, avg=118446560.37, stdev=178442905.58
- clat percentiles (usec):
- | 1.00th=[ 947], 5.00th=[ 1205], 10.00th=[ 1467],
- | 20.00th=[ 2114], 30.00th=[ 3654], 40.00th=[ 10028],
- | 50.00th=[ 58459], 60.00th=[ 86508], 70.00th=[ 111674],
- | 80.00th=[ 177210], 90.00th=[ 371196], 95.00th=[ 530580],
- | 99.00th=[ 775947], 99.50th=[ 901776], 99.90th=[1098908],
- | 99.95th=[1098908], 99.99th=[1266680]
- bw ( KiB/s): min= 0, max= 3001, per=53.65%, avg=2168.50, stdev=1177.33, samples=2
- iops : min= 0, max= 46, avg=33.00, stdev=18.38, samples=2
- lat (usec) : 750=0.05%, 1000=1.38%
- lat (msec) : 2=17.77%, 4=11.92%, 10=8.85%, 20=3.91%, 50=4.47%
- lat (msec) : 100=17.16%, 250=19.59%, 500=9.05%, 750=4.62%, 1000=0.96%
- cpu : usr=0.04%, sys=0.15%, ctx=7612, majf=0, minf=30
- IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=0,4068,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- read: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:34:54 2019
- read: IOPS=273, BW=17.1MiB/s (17.9MB/s)(2052MiB/120023msec)
- slat (nsec): min=0, max=393291k, avg=46338.23, stdev=2796058.39
- clat (nsec): min=0, max=1038.1M, avg=14624047.78, stdev=65201130.73
- lat (nsec): min=0, max=1038.6M, avg=14671240.74, stdev=65271622.43
- clat percentiles (nsec):
- | 1.00th=[ 708], 5.00th=[ 716], 10.00th=[ 716],
- | 20.00th=[ 724], 30.00th=[ 732], 40.00th=[ 756],
- | 50.00th=[ 844], 60.00th=[ 908], 70.00th=[ 1144],
- | 80.00th=[ 724992], 90.00th=[ 25559040], 95.00th=[ 67633152],
- | 99.00th=[379584512], 99.50th=[488636416], 99.90th=[700448768],
- | 99.95th=[750780416], 99.99th=[943718400]
- bw ( KiB/s): min= 0, max=22634, per=90.78%, avg=17511.00, stdev=7245.02, samples=2
- iops : min= 0, max= 354, avg=273.50, stdev=113.84, samples=2
- lat (nsec) : 750=39.24%, 1000=25.97%
- lat (usec) : 2=12.59%, 4=0.05%, 10=0.41%, 20=0.41%, 50=0.17%
- lat (usec) : 100=0.09%, 250=0.03%, 500=0.24%, 750=1.12%, 1000=2.31%
- lat (msec) : 2=2.99%, 4=1.91%, 10=0.98%, 20=0.94%, 50=3.84%
- lat (msec) : 100=2.95%, 250=1.98%, 500=1.34%, 750=0.40%, 1000=0.05%
- cpu : usr=0.10%, sys=0.33%, ctx=13746, majf=0, minf=29
- IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=32856,0,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- readwrite: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:34:54 2019
- read: IOPS=27, BW=1786KiB/s (1829kB/s)(209MiB/120059msec)
- slat (nsec): min=0, max=535574k, avg=187960.89, stdev=9251743.42
- clat (nsec): min=0, max=794000k, avg=10453672.56, stdev=48478017.97
- lat (nsec): min=0, max=794034k, avg=10642538.71, stdev=49327238.20
- clat percentiles (nsec):
- | 1.00th=[ 708], 5.00th=[ 724], 10.00th=[ 732],
- | 20.00th=[ 764], 30.00th=[ 876], 40.00th=[ 988],
- | 50.00th=[ 1128], 60.00th=[ 1256], 70.00th=[ 1448],
- | 80.00th=[ 100864], 90.00th=[ 3194880], 95.00th=[ 53739520],
- | 99.00th=[270532608], 99.50th=[362807296], 99.90th=[522190848],
- | 99.95th=[608174080], 99.99th=[792723456]
- bw ( KiB/s): min= 0, max= 2412, per=9.26%, avg=1786.50, stdev=884.59, samples=2
- iops : min= 0, max= 37, avg=27.50, stdev=13.44, samples=2
- write: IOPS=29, BW=1873KiB/s (1918kB/s)(220MiB/120059msec)
- slat (nsec): min=0, max=400348k, avg=279434.69, stdev=6757705.47
- clat (nsec): min=0, max=1261.4M, avg=126879957.42, stdev=186908714.01
- lat (nsec): min=0, max=1261.5M, avg=127160436.98, stdev=186957434.42
- clat percentiles (usec):
- | 1.00th=[ 1004], 5.00th=[ 1287], 10.00th=[ 1549],
- | 20.00th=[ 2278], 30.00th=[ 4228], 40.00th=[ 15795],
- | 50.00th=[ 63701], 60.00th=[ 89654], 70.00th=[ 117965],
- | 80.00th=[ 196084], 90.00th=[ 396362], 95.00th=[ 549454],
- | 99.00th=[ 826278], 99.50th=[ 935330], 99.90th=[1098908],
- | 99.95th=[1098908], 99.99th=[1266680]
- bw ( KiB/s): min= 0, max= 2512, per=46.29%, avg=1871.00, stdev=906.51, samples=2
- iops : min= 0, max= 39, avg=29.00, stdev=14.14, samples=2
- lat (nsec) : 750=8.58%, 1000=11.38%
- lat (usec) : 2=16.15%, 4=0.07%, 10=0.28%, 20=0.76%, 50=0.80%
- lat (usec) : 100=1.02%, 250=0.67%, 500=0.63%, 750=0.63%, 1000=1.46%
- lat (msec) : 2=9.58%, 4=7.00%, 10=4.52%, 20=2.42%, 50=3.92%
- lat (msec) : 100=10.00%, 250=11.09%, 500=5.71%, 750=2.48%, 1000=0.71%
- cpu : usr=0.04%, sys=0.17%, ctx=8048, majf=0, minf=33
- IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=3351,3510,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- Run status group 0 (all jobs):
- READ: bw=18.8MiB/s (19.8MB/s), 1786KiB/s-17.1MiB/s (1829kB/s-17.9MB/s), io=2262MiB (2372MB), run=120023-120059msec
- WRITE: bw=4043KiB/s (4140kB/s), 1873KiB/s-2171KiB/s (1918kB/s-2223kB/s), io=474MiB (497MB), run=120036-120059msec
- Disk stats (read/write):
- rbd75: ios=8536/9131, merge=0/1, ticks=554901/875234, in_queue=1421062, util=12.57%
- <example-benchmark-fio-server-424-benchmark-tlngh>
- randwrite: (groupid=0, jobs=1): err= 0: pid=11: Wed May 8 20:34:54 2019
- write: IOPS=41, BW=2666KiB/s (2730kB/s)(313MiB/120074msec)
- slat (nsec): min=0, max=400546k, avg=326325.35, stdev=6957801.48
- clat (nsec): min=0, max=1102.3M, avg=96092666.22, stdev=166659881.38
- lat (nsec): min=0, max=1102.3M, avg=96420079.84, stdev=167148622.48
- clat percentiles (usec):
- | 1.00th=[ 1012], 5.00th=[ 1254], 10.00th=[ 1467],
- | 20.00th=[ 1991], 30.00th=[ 2835], 40.00th=[ 4686],
- | 50.00th=[ 11338], 60.00th=[ 45876], 70.00th=[ 86508],
- | 80.00th=[ 133694], 90.00th=[ 346031], 95.00th=[ 480248],
- | 99.00th=[ 801113], 99.50th=[ 843056], 99.90th=[ 943719],
- | 99.95th=[ 985662], 99.99th=[1098908]
- bw ( KiB/s): min= 0, max= 2965, per=55.11%, avg=2665.00, stdev=424.26, samples=2
- iops : min= 0, max= 46, avg=41.00, stdev= 7.07, samples=2
- lat (usec) : 500=0.02%, 750=0.04%, 1000=0.78%
- lat (msec) : 2=19.24%, 4=16.86%, 10=12.00%, 20=6.18%, 50=5.52%
- lat (msec) : 100=13.06%, 250=12.44%, 500=9.76%, 750=2.90%, 1000=1.22%
- cpu : usr=0.05%, sys=0.18%, ctx=9369, majf=0, minf=31
- IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=0,4999,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- read: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:34:54 2019
- read: IOPS=318, BW=19.9MiB/s (20.9MB/s)(2389MiB/120031msec)
- slat (nsec): min=0, max=320204k, avg=38057.84, stdev=2181510.66
- clat (nsec): min=0, max=1057.4M, avg=12541947.82, stdev=61296772.33
- lat (nsec): min=0, max=1057.4M, avg=12580866.82, stdev=61342276.53
- clat percentiles (nsec):
- | 1.00th=[ 708], 5.00th=[ 716], 10.00th=[ 716],
- | 20.00th=[ 724], 30.00th=[ 740], 40.00th=[ 788],
- | 50.00th=[ 852], 60.00th=[ 900], 70.00th=[ 1096],
- | 80.00th=[ 43776], 90.00th=[ 6324224], 95.00th=[ 58982400],
- | 99.00th=[362807296], 99.50th=[480247808], 99.90th=[700448768],
- | 99.95th=[767557632], 99.99th=[893386752]
- bw ( KiB/s): min= 0, max=25069, per=90.81%, avg=20383.00, stdev=6627.00, samples=2
- iops : min= 0, max= 392, avg=318.50, stdev=103.94, samples=2
- lat (nsec) : 750=33.09%, 1000=34.07%
- lat (usec) : 2=11.70%, 4=0.05%, 10=0.57%, 20=0.41%, 50=0.14%
- lat (usec) : 100=0.10%, 250=0.03%, 500=0.14%, 750=1.09%, 1000=2.45%
- lat (msec) : 2=3.49%, 4=1.98%, 10=1.14%, 20=0.49%, 50=3.07%
- lat (msec) : 100=2.94%, 250=1.55%, 500=1.06%, 750=0.39%, 1000=0.06%
- cpu : usr=0.10%, sys=0.37%, ctx=15077, majf=0, minf=31
- IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=38240,0,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- readwrite: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:34:54 2019
- read: IOPS=32, BW=2065KiB/s (2115kB/s)(242MiB/120022msec)
- slat (nsec): min=0, max=399843k, avg=206891.70, stdev=7876972.64
- clat (nsec): min=0, max=737486k, avg=8865298.83, stdev=46977819.17
- lat (nsec): min=0, max=976153k, avg=9073210.18, stdev=48841859.71
- clat percentiles (nsec):
- | 1.00th=[ 708], 5.00th=[ 716], 10.00th=[ 732],
- | 20.00th=[ 772], 30.00th=[ 876], 40.00th=[ 972],
- | 50.00th=[ 1112], 60.00th=[ 1256], 70.00th=[ 1480],
- | 80.00th=[ 109056], 90.00th=[ 2506752], 95.00th=[ 45875200],
- | 99.00th=[244318208], 99.50th=[400556032], 99.90th=[583008256],
- | 99.95th=[717225984], 99.99th=[734003200]
- bw ( KiB/s): min= 0, max= 2312, per=9.20%, avg=2065.00, stdev=349.31, samples=2
- iops : min= 0, max= 36, avg=32.00, stdev= 5.66, samples=2
- write: IOPS=33, BW=2171KiB/s (2223kB/s)(255MiB/120022msec)
- slat (nsec): min=0, max=10291k, avg=180936.82, stdev=303505.98
- clat (nsec): min=0, max=1040.1M, avg=109529981.00, stdev=175834302.36
- lat (nsec): min=0, max=1040.2M, avg=109712031.36, stdev=175783939.65
- clat percentiles (usec):
- | 1.00th=[ 1057], 5.00th=[ 1287], 10.00th=[ 1565],
- | 20.00th=[ 2278], 30.00th=[ 3621], 40.00th=[ 6587],
- | 50.00th=[ 19530], 60.00th=[ 66847], 70.00th=[ 102237],
- | 80.00th=[ 160433], 90.00th=[ 371196], 95.00th=[ 497026],
- | 99.00th=[ 809501], 99.50th=[ 893387], 99.90th=[ 952108],
- | 99.95th=[1002439], 99.99th=[1035994]
- bw ( KiB/s): min= 0, max= 2405, per=44.85%, avg=2169.00, stdev=333.75, samples=2
- iops : min= 0, max= 37, avg=33.50, stdev= 4.95, samples=2
- lat (nsec) : 750=8.20%, 1000=12.22%
- lat (usec) : 2=14.93%, 4=0.05%, 10=0.39%, 20=1.15%, 50=1.02%
- lat (usec) : 100=0.96%, 250=0.79%, 500=0.94%, 750=0.52%, 1000=1.17%
- lat (msec) : 2=9.61%, 4=8.78%, 10=6.61%, 20=3.36%, 50=4.33%
- lat (msec) : 100=8.36%, 250=7.93%, 500=6.17%, 750=1.85%, 1000=0.68%
- cpu : usr=0.05%, sys=0.19%, ctx=9240, majf=0, minf=35
- IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=3874,4069,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- Run status group 0 (all jobs):
- READ: bw=21.9MiB/s (22.0MB/s), 2065KiB/s-19.9MiB/s (2115kB/s-20.9MB/s), io=2631MiB (2759MB), run=120022-120031msec
- WRITE: bw=4836KiB/s (4953kB/s), 2171KiB/s-2666KiB/s (2223kB/s-2730kB/s), io=567MiB (595MB), run=120022-120074msec
- Disk stats (read/write):
- rbd121: ios=8410/9542, merge=0/1, ticks=542895/868442, in_queue=1402193, util=12.62%
- <example-benchmark-fio-server-325-benchmark-fbmfd>
- randwrite: (groupid=0, jobs=1): err= 0: pid=11: Wed May 8 20:34:54 2019
- write: IOPS=46, BW=2969KiB/s (3040kB/s)(348MiB/120063msec)
- slat (nsec): min=0, max=428628k, avg=322718.17, stdev=6988005.94
- clat (nsec): min=0, max=1469.7M, avg=86229567.91, stdev=147784132.72
- lat (nsec): min=0, max=1469.8M, avg=86553448.19, stdev=148051015.99
- clat percentiles (usec):
- | 1.00th=[ 988], 5.00th=[ 1188], 10.00th=[ 1401],
- | 20.00th=[ 1827], 30.00th=[ 2573], 40.00th=[ 4359],
- | 50.00th=[ 10814], 60.00th=[ 61604], 70.00th=[ 91751],
- | 80.00th=[ 120062], 90.00th=[ 261096], 95.00th=[ 417334],
- | 99.00th=[ 683672], 99.50th=[ 809501], 99.90th=[ 977273],
- | 99.95th=[1283458], 99.99th=[1468007]
- bw ( KiB/s): min= 0, max= 3621, per=54.56%, avg=2967.50, stdev=924.19, samples=2
- iops : min= 0, max= 56, avg=46.00, stdev=14.14, samples=2
- lat (usec) : 50=0.02%, 250=0.02%, 750=0.07%, 1000=1.02%
- lat (msec) : 2=21.88%, 4=15.52%, 10=11.03%, 20=4.01%, 50=3.92%
- lat (msec) : 100=15.99%, 250=16.39%, 500=6.95%, 750=2.48%, 1000=0.68%
- cpu : usr=0.07%, sys=0.19%, ctx=10557, majf=0, minf=31
- IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=0,5566,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- read: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:34:54 2019
- read: IOPS=295, BW=18.5MiB/s (19.4MB/s)(2218MiB/120022msec)
- slat (nsec): min=0, max=413228k, avg=42515.07, stdev=2589450.95
- clat (nsec): min=0, max=1400.0M, avg=13517573.15, stdev=63183262.34
- lat (nsec): min=0, max=1400.3M, avg=13560936.96, stdev=63244141.80
- clat percentiles (nsec):
- | 1.00th=[ 708], 5.00th=[ 716], 10.00th=[ 724],
- | 20.00th=[ 724], 30.00th=[ 740], 40.00th=[ 772],
- | 50.00th=[ 844], 60.00th=[ 900], 70.00th=[ 1080],
- | 80.00th=[ 264192], 90.00th=[ 22937600], 95.00th=[ 61079552],
- | 99.00th=[375390208], 99.50th=[480247808], 99.90th=[700448768],
- | 99.95th=[767557632], 99.99th=[926941184]
- bw ( KiB/s): min= 0, max=24299, per=88.89%, avg=18927.00, stdev=7597.16, samples=2
- iops : min= 0, max= 380, avg=295.50, stdev=119.50, samples=2
- lat (nsec) : 750=36.46%, 1000=30.96%
- lat (usec) : 2=11.74%, 4=0.06%, 10=0.34%, 20=0.18%, 50=0.17%
- lat (usec) : 100=0.06%, 250=0.05%, 500=0.17%, 750=1.13%, 1000=2.16%
- lat (msec) : 2=2.93%, 4=1.72%, 10=1.00%, 20=0.66%, 50=3.78%
- lat (msec) : 100=3.29%, 250=1.50%, 500=1.19%, 750=0.41%, 1000=0.06%
- cpu : usr=0.10%, sys=0.35%, ctx=14216, majf=0, minf=31
- IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=35512,0,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- readwrite: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:34:54 2019
- read: IOPS=37, BW=2372KiB/s (2429kB/s)(278MiB/120051msec)
- slat (nsec): min=0, max=405259k, avg=173230.37, stdev=7000712.24
- clat (nsec): min=0, max=676905k, avg=10063171.52, stdev=47810644.50
- lat (nsec): min=0, max=676931k, avg=10237323.34, stdev=48310601.69
- clat percentiles (nsec):
- | 1.00th=[ 716], 5.00th=[ 724], 10.00th=[ 732],
- | 20.00th=[ 812], 30.00th=[ 876], 40.00th=[ 980],
- | 50.00th=[ 1112], 60.00th=[ 1256], 70.00th=[ 1416],
- | 80.00th=[ 73216], 90.00th=[ 3588096], 95.00th=[ 55836672],
- | 99.00th=[265289728], 99.50th=[362807296], 99.90th=[583008256],
- | 99.95th=[599785472], 99.99th=[675282944]
- bw ( KiB/s): min= 0, max= 2913, per=11.14%, avg=2373.00, stdev=763.68, samples=2
- iops : min= 0, max= 45, avg=36.50, stdev=12.02, samples=2
- write: IOPS=38, BW=2471KiB/s (2531kB/s)(290MiB/120051msec)
- slat (nsec): min=0, max=4214.0k, avg=178472.88, stdev=241439.55
- clat (nsec): min=0, max=1469.7M, avg=93902576.19, stdev=154811764.20
- lat (nsec): min=0, max=1469.8M, avg=94082275.96, stdev=154763959.10
- clat percentiles (usec):
- | 1.00th=[ 1012], 5.00th=[ 1237], 10.00th=[ 1450],
- | 20.00th=[ 1991], 30.00th=[ 3097], 40.00th=[ 5604],
- | 50.00th=[ 22152], 60.00th=[ 70779], 70.00th=[ 95945],
- | 80.00th=[ 129500], 90.00th=[ 291505], 95.00th=[ 442500],
- | 99.00th=[ 750781], 99.50th=[ 876610], 99.90th=[1098908],
- | 99.95th=[1132463], 99.99th=[1468007]
- bw ( KiB/s): min= 0, max= 3003, per=45.41%, avg=2470.00, stdev=753.78, samples=2
- iops : min= 0, max= 46, avg=38.00, stdev=11.31, samples=2
- lat (nsec) : 750=6.72%, 1000=13.63%
- lat (usec) : 2=17.22%, 4=0.02%, 10=0.23%, 20=0.47%, 50=0.44%
- lat (usec) : 100=0.66%, 250=0.53%, 500=0.37%, 750=0.68%, 1000=1.45%
- lat (msec) : 2=11.21%, 4=8.28%, 10=5.91%, 20=2.37%, 50=3.56%
- lat (msec) : 100=10.54%, 250=9.54%, 500=4.41%, 750=1.28%, 1000=0.43%
- cpu : usr=0.07%, sys=0.22%, ctx=10658, majf=0, minf=31
- IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=4451,4633,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- Run status group 0 (all jobs):
- READ: bw=20.8MiB/s (21.8MB/s), 2372KiB/s-18.5MiB/s (2429kB/s-19.4MB/s), io=2496MiB (2618MB), run=120022-120051msec
- WRITE: bw=5440KiB/s (5570kB/s), 2471KiB/s-2969KiB/s (2531kB/s-3040kB/s), io=638MiB (669MB), run=120051-120063msec
- Disk stats (read/write):
- rbd103: ios=8316/10505, merge=0/1, ticks=549459/828531, in_queue=1368379, util=14.25%
- <example-benchmark-fio-server-336-benchmark-tzn4v>
- randwrite: (groupid=0, jobs=1): err= 0: pid=11: Wed May 8 20:34:54 2019
- write: IOPS=37, BW=2378KiB/s (2435kB/s)(279MiB/120104msec)
- slat (nsec): min=0, max=340097k, avg=326004.44, stdev=6645041.68
- clat (nsec): min=0, max=1280.6M, avg=107678890.86, stdev=175225797.88
- lat (nsec): min=0, max=1280.7M, avg=108005935.42, stdev=175238266.51
- clat percentiles (usec):
- | 1.00th=[ 955], 5.00th=[ 1172], 10.00th=[ 1369],
- | 20.00th=[ 1844], 30.00th=[ 2868], 40.00th=[ 6128],
- | 50.00th=[ 28967], 60.00th=[ 76022], 70.00th=[ 104334],
- | 80.00th=[ 154141], 90.00th=[ 337642], 95.00th=[ 488637],
- | 99.00th=[ 817890], 99.50th=[ 935330], 99.90th=[1132463],
- | 99.95th=[1283458], 99.99th=[1283458]
- bw ( KiB/s): min= 0, max= 1784, per=40.93%, avg=1784.00, stdev= 0.00, samples=1
- iops : min= 0, max= 27, avg=27.00, stdev= 0.00, samples=1
- lat (usec) : 4=0.02%, 100=0.02%, 250=0.02%, 500=0.02%, 750=0.09%
- lat (usec) : 1000=1.68%
- lat (msec) : 2=20.36%, 4=13.09%, 10=8.88%, 20=3.61%, 50=5.94%
- lat (msec) : 100=14.64%, 250=17.60%, 500=9.35%, 750=3.27%, 1000=1.03%
- cpu : usr=0.04%, sys=0.17%, ctx=8470, majf=0, minf=30
- IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=0,4460,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- read: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:34:54 2019
- read: IOPS=456, BW=28.5MiB/s (29.9MB/s)(3422MiB/120027msec)
- slat (nsec): min=0, max=338266k, avg=27425.06, stdev=1786682.20
- clat (nsec): min=0, max=999544k, avg=8748171.03, stdev=51624016.82
- lat (nsec): min=0, max=999606k, avg=8776407.06, stdev=51663169.95
- clat percentiles (nsec):
- | 1.00th=[ 708], 5.00th=[ 708], 10.00th=[ 716],
- | 20.00th=[ 724], 30.00th=[ 724], 40.00th=[ 732],
- | 50.00th=[ 748], 60.00th=[ 828], 70.00th=[ 884],
- | 80.00th=[ 1112], 90.00th=[ 1044480], 95.00th=[ 36962304],
- | 99.00th=[270532608], 99.50th=[417333248], 99.90th=[650117120],
- | 99.95th=[742391808], 99.99th=[910163968]
- bw ( KiB/s): min= 0, max=44025, per=93.94%, avg=29198.00, stdev=20968.54, samples=2
- iops : min= 0, max= 688, avg=456.00, stdev=328.10, samples=2
- lat (nsec) : 750=50.41%, 1000=27.05%
- lat (usec) : 2=8.04%, 4=0.05%, 10=1.12%, 20=0.45%, 50=0.17%
- lat (usec) : 100=0.10%, 250=0.06%, 500=0.31%, 750=0.70%, 1000=1.39%
- lat (msec) : 2=1.85%, 4=1.13%, 10=0.65%, 20=0.46%, 50=2.03%
- lat (msec) : 100=1.78%, 250=1.14%, 500=0.82%, 750=0.24%, 1000=0.05%
- cpu : usr=0.12%, sys=0.47%, ctx=13690, majf=0, minf=30
- IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=54795,0,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- readwrite: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:34:54 2019
- read: IOPS=29, BW=1895KiB/s (1941kB/s)(222MiB/120051msec)
- slat (nsec): min=0, max=263179k, avg=101624.02, stdev=4414051.05
- clat (nsec): min=0, max=655890k, avg=9618104.13, stdev=46649931.58
- lat (nsec): min=0, max=655927k, avg=9720798.57, stdev=46853921.91
- clat percentiles (nsec):
- | 1.00th=[ 716], 5.00th=[ 724], 10.00th=[ 732],
- | 20.00th=[ 804], 30.00th=[ 876], 40.00th=[ 956],
- | 50.00th=[ 1096], 60.00th=[ 1240], 70.00th=[ 1416],
- | 80.00th=[ 35584], 90.00th=[ 2670592], 95.00th=[ 50069504],
- | 99.00th=[256901120], 99.50th=[367001600], 99.90th=[522190848],
- | 99.95th=[599785472], 99.99th=[658505728]
- bw ( KiB/s): min= 0, max= 2421, per=6.10%, avg=1895.50, stdev=743.17, samples=2
- iops : min= 0, max= 37, avg=29.00, stdev=11.31, samples=2
- write: IOPS=30, BW=1983KiB/s (2030kB/s)(232MiB/120051msec)
- slat (nsec): min=0, max=279271k, avg=268855.21, stdev=4731857.77
- clat (nsec): min=0, max=1280.7M, avg=120132593.80, stdev=184042868.06
- lat (nsec): min=0, max=1280.8M, avg=120402554.96, stdev=184195357.79
- clat percentiles (usec):
- | 1.00th=[ 988], 5.00th=[ 1221], 10.00th=[ 1450],
- | 20.00th=[ 2057], 30.00th=[ 3654], 40.00th=[ 10159],
- | 50.00th=[ 51643], 60.00th=[ 86508], 70.00th=[ 114820],
- | 80.00th=[ 187696], 90.00th=[ 367002], 95.00th=[ 505414],
- | 99.00th=[ 876610], 99.50th=[ 977273], 99.90th=[1132463],
- | 99.95th=[1132463], 99.99th=[1283458]
- bw ( KiB/s): min= 0, max= 2540, per=45.43%, avg=1980.50, stdev=791.25, samples=2
- iops : min= 0, max= 39, avg=30.50, stdev=12.02, samples=2
- lat (nsec) : 750=6.97%, 1000=14.22%
- lat (usec) : 2=16.17%, 4=0.08%, 10=0.36%, 20=0.89%, 50=0.76%
- lat (usec) : 100=0.98%, 250=0.47%, 500=0.58%, 750=0.58%, 1000=1.20%
- lat (msec) : 2=10.26%, 4=6.75%, 10=4.73%, 20=1.90%, 50=4.94%
- lat (msec) : 100=8.73%, 250=10.95%, 500=5.79%, 750=1.76%, 1000=0.80%
- cpu : usr=0.05%, sys=0.18%, ctx=8332, majf=0, minf=34
- IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=3556,3716,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- Run status group 0 (all jobs):
- READ: bw=30.4MiB/s (31.8MB/s), 1895KiB/s-28.5MiB/s (1941kB/s-29.9MB/s), io=3644MiB (3821MB), run=120027-120051msec
- WRITE: bw=4360KiB/s (4465kB/s), 1983KiB/s-2378KiB/s (2030kB/s-2435kB/s), io=511MiB (536MB), run=120051-120104msec
- Disk stats (read/write):
- rbd111: ios=7834/8600, merge=0/1, ticks=524255/818436, in_queue=1334384, util=12.21%
- fio: inflate error -5
- fio: failed decompressing log
- fio: failed converting IO log
- <example-benchmark-fio-server-239-benchmark-4zbns>
- randwrite: (groupid=0, jobs=1): err= 0: pid=11: Wed May 8 20:34:54 2019
- write: IOPS=45, BW=2936KiB/s (3006kB/s)(344MiB/120067msec)
- slat (nsec): min=0, max=534902k, avg=347032.55, stdev=8533504.59
- clat (nsec): min=0, max=889389k, avg=86828735.19, stdev=144120444.95
- lat (nsec): min=0, max=1368.6M, avg=87177012.08, stdev=145119573.59
- clat percentiles (usec):
- | 1.00th=[ 947], 5.00th=[ 1156], 10.00th=[ 1336], 20.00th=[ 1795],
- | 30.00th=[ 2606], 40.00th=[ 4948], 50.00th=[ 17433], 60.00th=[ 55313],
- | 70.00th=[ 94897], 80.00th=[131597], 90.00th=[238027], 95.00th=[421528],
- | 99.00th=[700449], 99.50th=[742392], 99.90th=[826278], 99.95th=[859833],
- | 99.99th=[893387]
- bw ( KiB/s): min= 0, max= 3771, per=55.94%, avg=2935.00, stdev=1182.28, samples=2
- iops : min= 0, max= 58, avg=45.00, stdev=18.38, samples=2
- lat (usec) : 250=0.02%, 500=0.04%, 750=0.02%, 1000=1.76%
- lat (msec) : 2=21.13%, 4=14.35%, 10=9.14%, 20=4.60%, 50=7.45%
- lat (msec) : 100=13.01%, 250=18.91%, 500=6.21%, 750=2.98%, 1000=0.45%
- cpu : usr=0.05%, sys=0.20%, ctx=10277, majf=0, minf=30
- IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=0,5505,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- read: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:34:54 2019
- read: IOPS=255, BW=15.9MiB/s (16.7MB/s)(1912MiB/120021msec)
- slat (nsec): min=0, max=533552k, avg=48056.18, stdev=3101874.04
- clat (nsec): min=0, max=925627k, avg=15680255.74, stdev=65921046.09
- lat (nsec): min=0, max=925689k, avg=15729185.26, stdev=66004552.52
- clat percentiles (nsec):
- | 1.00th=[ 708], 5.00th=[ 716], 10.00th=[ 716],
- | 20.00th=[ 724], 30.00th=[ 732], 40.00th=[ 764],
- | 50.00th=[ 868], 60.00th=[ 956], 70.00th=[ 1256],
- | 80.00th=[ 970752], 90.00th=[ 31850496], 95.00th=[ 73924608],
- | 99.00th=[375390208], 99.50th=[497025024], 99.90th=[692060160],
- | 99.95th=[725614592], 99.99th=[834666496]
- bw ( KiB/s): min= 0, max=22069, per=100.00%, avg=22069.00, stdev= 0.00, samples=1
- iops : min= 0, max= 345, avg=345.00, stdev= 0.00, samples=1
- lat (nsec) : 750=37.72%, 1000=24.33%
- lat (usec) : 2=12.99%, 4=0.07%, 10=0.30%, 20=0.28%, 50=0.21%
- lat (usec) : 100=0.03%, 250=0.04%, 500=0.32%, 750=1.28%, 1000=2.65%
- lat (msec) : 2=3.61%, 4=2.23%, 10=1.10%, 20=1.05%, 50=4.52%
- lat (msec) : 100=3.20%, 250=2.16%, 500=1.44%, 750=0.45%, 1000=0.03%
- cpu : usr=0.08%, sys=0.33%, ctx=14593, majf=0, minf=100
- IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=30613,0,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- readwrite: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:34:54 2019
- read: IOPS=34, BW=2211KiB/s (2264kB/s)(259MiB/120027msec)
- slat (nsec): min=0, max=534280k, avg=237292.72, stdev=9787518.53
- clat (nsec): min=0, max=711810k, avg=12200922.95, stdev=52573178.22
- lat (nsec): min=0, max=711830k, avg=12439097.46, stdev=53435173.79
- clat percentiles (nsec):
- | 1.00th=[ 708], 5.00th=[ 724], 10.00th=[ 732],
- | 20.00th=[ 780], 30.00th=[ 884], 40.00th=[ 988],
- | 50.00th=[ 1128], 60.00th=[ 1256], 70.00th=[ 1528],
- | 80.00th=[ 238592], 90.00th=[ 20840448], 95.00th=[ 67633152],
- | 99.00th=[283115520], 99.50th=[358612992], 99.90th=[675282944],
- | 99.95th=[708837376], 99.99th=[708837376]
- bw ( KiB/s): min= 0, max= 2634, per=11.93%, avg=2210.00, stdev=599.63, samples=2
- iops : min= 0, max= 41, avg=34.00, stdev= 9.90, samples=2
- write: IOPS=36, BW=2313KiB/s (2368kB/s)(271MiB/120027msec)
- slat (nsec): min=0, max=6942.8k, avg=174147.77, stdev=263171.86
- clat (nsec): min=0, max=859436k, avg=98737609.79, stdev=152398197.87
- lat (nsec): min=0, max=859576k, avg=98912930.74, stdev=152347644.57
- clat percentiles (usec):
- | 1.00th=[ 1020], 5.00th=[ 1303], 10.00th=[ 1549], 20.00th=[ 2180],
- | 30.00th=[ 3687], 40.00th=[ 10159], 50.00th=[ 39584], 60.00th=[ 73925],
- | 70.00th=[105382], 80.00th=[141558], 90.00th=[295699], 95.00th=[463471],
- | 99.00th=[692061], 99.50th=[750781], 99.90th=[801113], 99.95th=[843056],
- | 99.99th=[859833]
- bw ( KiB/s): min= 0, max= 2733, per=44.05%, avg=2311.50, stdev=596.09, samples=2
- iops : min= 0, max= 42, avg=35.50, stdev= 9.19, samples=2
- lat (nsec) : 750=7.59%, 1000=12.44%
- lat (usec) : 2=15.02%, 4=0.05%, 10=0.29%, 20=1.04%, 50=0.85%
- lat (usec) : 100=0.93%, 250=0.97%, 500=0.55%, 750=0.57%, 1000=1.29%
- lat (msec) : 2=9.95%, 4=7.29%, 10=5.11%, 20=3.11%, 50=5.72%
- lat (msec) : 100=9.42%, 250=11.52%, 500=3.96%, 750=2.13%, 1000=0.22%
- cpu : usr=0.05%, sys=0.20%, ctx=9894, majf=0, minf=36
- IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=4146,4334,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- Run status group 0 (all jobs):
- READ: bw=18.1MiB/s (18.0MB/s), 2211KiB/s-15.9MiB/s (2264kB/s-16.7MB/s), io=2171MiB (2277MB), run=120021-120027msec
- WRITE: bw=5248KiB/s (5374kB/s), 2313KiB/s-2936KiB/s (2368kB/s-3006kB/s), io=615MiB (645MB), run=120027-120067msec
- Disk stats (read/write):
- rbd95: ios=9683/11004, merge=0/1, ticks=566653/848675, in_queue=1404758, util=14.29%
- <example-benchmark-fio-server-747-benchmark-jh49p>
- randwrite: (groupid=0, jobs=1): err= 0: pid=11: Wed May 8 20:34:54 2019
- write: IOPS=41, BW=2650KiB/s (2714kB/s)(311MiB/120014msec)
- slat (nsec): min=0, max=444994k, avg=406213.93, stdev=8650946.30
- clat (nsec): min=0, max=1306.6M, avg=96754907.14, stdev=155893560.13
- lat (nsec): min=0, max=1306.7M, avg=97162272.04, stdev=156021316.52
- clat percentiles (usec):
- | 1.00th=[ 963], 5.00th=[ 1156], 10.00th=[ 1352],
- | 20.00th=[ 1811], 30.00th=[ 2704], 40.00th=[ 5473],
- | 50.00th=[ 35914], 60.00th=[ 78119], 70.00th=[ 100140],
- | 80.00th=[ 130548], 90.00th=[ 278922], 95.00th=[ 471860],
- | 99.00th=[ 717226], 99.50th=[ 817890], 99.90th=[1044382],
- | 99.95th=[1098908], 99.99th=[1300235]
- bw ( KiB/s): min= 0, max= 3854, per=54.00%, avg=2648.00, stdev=1705.54, samples=2
- iops : min= 0, max= 60, avg=41.00, stdev=26.87, samples=2
- lat (usec) : 500=0.04%, 750=0.02%, 1000=1.43%
- lat (msec) : 2=21.20%, 4=14.15%, 10=7.79%, 20=3.60%, 50=3.85%
- lat (msec) : 100=17.74%, 250=19.33%, 500=6.83%, 750=3.32%, 1000=0.62%
- cpu : usr=0.04%, sys=0.18%, ctx=9502, majf=0, minf=29
- IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=0,4967,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- read: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:34:54 2019
- read: IOPS=565, BW=35.3MiB/s (36.0MB/s)(4233MiB/120022msec)
- slat (nsec): min=0, max=426774k, avg=30081.64, stdev=2287324.85
- clat (nsec): min=0, max=1057.1M, avg=7035019.24, stdev=47212773.76
- lat (nsec): min=0, max=1057.1M, avg=7065919.65, stdev=47275485.63
- clat percentiles (nsec):
- | 1.00th=[ 708], 5.00th=[ 708], 10.00th=[ 716],
- | 20.00th=[ 716], 30.00th=[ 724], 40.00th=[ 740],
- | 50.00th=[ 796], 60.00th=[ 852], 70.00th=[ 900],
- | 80.00th=[ 1112], 90.00th=[ 25216], 95.00th=[ 13959168],
- | 99.00th=[227540992], 99.50th=[396361728], 99.90th=[633339904],
- | 99.95th=[725614592], 99.99th=[918552576]
- bw ( KiB/s): min= 0, max=54834, per=94.38%, avg=36118.50, stdev=26467.71, samples=2
- iops : min= 0, max= 860, avg=565.50, stdev=416.49, samples=2
- lat (nsec) : 750=45.37%, 1000=32.21%
- lat (usec) : 2=6.94%, 4=0.05%, 10=2.83%, 20=2.39%, 50=0.47%
- lat (usec) : 100=0.13%, 250=0.04%, 500=0.09%, 750=0.50%, 1000=1.06%
- lat (msec) : 2=1.45%, 4=0.88%, 10=0.46%, 20=0.29%, 50=1.68%
- lat (msec) : 100=1.44%, 250=0.80%, 500=0.68%, 750=0.19%, 1000=0.04%
- cpu : usr=0.16%, sys=0.52%, ctx=13040, majf=0, minf=29
- IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=67920,0,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- readwrite: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:34:54 2019
- read: IOPS=33, BW=2161KiB/s (2213kB/s)(253MiB/120042msec)
- slat (nsec): min=0, max=4021.6k, avg=26307.06, stdev=106378.46
- clat (nsec): min=0, max=626922k, avg=7607568.80, stdev=38180756.57
- lat (nsec): min=0, max=626958k, avg=7634781.88, stdev=38187137.32
- clat percentiles (nsec):
- | 1.00th=[ 708], 5.00th=[ 724], 10.00th=[ 732],
- | 20.00th=[ 756], 30.00th=[ 852], 40.00th=[ 948],
- | 50.00th=[ 1080], 60.00th=[ 1224], 70.00th=[ 1384],
- | 80.00th=[ 17024], 90.00th=[ 1581056], 95.00th=[ 47448064],
- | 99.00th=[191889408], 99.50th=[304087040], 99.90th=[480247808],
- | 99.95th=[484442112], 99.99th=[624951296]
- bw ( KiB/s): min= 0, max= 2935, per=5.65%, avg=2161.50, stdev=1093.89, samples=2
- iops : min= 0, max= 45, avg=33.00, stdev=16.97, samples=2
- write: IOPS=35, BW=2255KiB/s (2309kB/s)(264MiB/120042msec)
- slat (nsec): min=0, max=444199k, avg=469547.59, stdev=10265636.14
- clat (nsec): min=0, max=1282.2M, avg=105745971.30, stdev=159456781.24
- lat (nsec): min=0, max=1282.3M, avg=106216461.29, stdev=159801960.04
- clat percentiles (usec):
- | 1.00th=[ 996], 5.00th=[ 1205], 10.00th=[ 1401],
- | 20.00th=[ 1909], 30.00th=[ 3228], 40.00th=[ 9503],
- | 50.00th=[ 60031], 60.00th=[ 86508], 70.00th=[ 107480],
- | 80.00th=[ 139461], 90.00th=[ 320865], 95.00th=[ 484443],
- | 99.00th=[ 734004], 99.50th=[ 801113], 99.90th=[ 943719],
- | 99.95th=[1027605], 99.99th=[1283458]
- bw ( KiB/s): min= 0, max= 3069, per=45.93%, avg=2252.50, stdev=1154.71, samples=2
- iops : min= 0, max= 47, avg=34.50, stdev=17.68, samples=2
- lat (nsec) : 750=8.79%, 1000=13.03%
- lat (usec) : 2=16.53%, 4=0.04%, 10=0.41%, 20=0.77%, 50=0.72%
- lat (usec) : 100=0.92%, 250=0.56%, 500=0.41%, 750=0.45%, 1000=1.28%
- lat (msec) : 2=11.07%, 4=6.54%, 10=4.13%, 20=1.80%, 50=3.83%
- lat (msec) : 100=10.84%, 250=11.47%, 500=4.07%, 750=1.91%, 1000=0.43%
- cpu : usr=0.05%, sys=0.21%, ctx=9390, majf=0, minf=34
- IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=4056,4226,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- Run status group 0 (all jobs):
- READ: bw=37.4MiB/s (39.2MB/s), 2161KiB/s-35.3MiB/s (2213kB/s-36.0MB/s), io=4486MiB (4704MB), run=120022-120042msec
- WRITE: bw=4904KiB/s (5022kB/s), 2255KiB/s-2650KiB/s (2309kB/s-2714kB/s), io=575MiB (603MB), run=120014-120042msec
- Disk stats (read/write):
- rbd244: ios=7107/9294, merge=0/3, ticks=515812/787469, in_queue=1295011, util=12.37%
- <example-benchmark-fio-server-238-benchmark-wczfn>
- randwrite: (groupid=0, jobs=1): err= 0: pid=11: Wed May 8 20:34:54 2019
- write: IOPS=39, BW=2517KiB/s (2577kB/s)(295MiB/120014msec)
- slat (nsec): min=0, max=341058k, avg=292693.61, stdev=5575637.86
- clat (nsec): min=0, max=1342.0M, avg=102021795.41, stdev=158984969.89
- lat (nsec): min=0, max=1342.1M, avg=102315643.79, stdev=158993907.50
- clat percentiles (usec):
- | 1.00th=[ 963], 5.00th=[ 1221], 10.00th=[ 1483],
- | 20.00th=[ 2073], 30.00th=[ 3392], 40.00th=[ 9372],
- | 50.00th=[ 45351], 60.00th=[ 78119], 70.00th=[ 106431],
- | 80.00th=[ 143655], 90.00th=[ 295699], 95.00th=[ 459277],
- | 99.00th=[ 725615], 99.50th=[ 868221], 99.90th=[1149240],
- | 99.95th=[1300235], 99.99th=[1333789]
- bw ( KiB/s): min= 0, max= 3384, per=53.24%, avg=2514.50, stdev=1229.66, samples=2
- iops : min= 0, max= 52, avg=38.50, stdev=19.09, samples=2
- lat (usec) : 50=0.02%, 250=0.02%, 500=0.06%, 750=0.02%, 1000=1.34%
- lat (msec) : 2=17.45%, 4=13.19%, 10=8.25%, 20=3.90%, 50=7.19%
- lat (msec) : 100=16.22%, 250=20.54%, 500=7.70%, 750=3.41%, 1000=0.49%
- cpu : usr=0.05%, sys=0.17%, ctx=8969, majf=0, minf=30
- IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=0,4717,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- read: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:34:54 2019
- read: IOPS=379, BW=23.7MiB/s (24.9MB/s)(2846MiB/120038msec)
- slat (nsec): min=0, max=313498k, avg=28520.52, stdev=1565230.47
- clat (nsec): min=0, max=1558.7M, avg=10539488.90, stdev=56101509.52
- lat (nsec): min=0, max=1558.8M, avg=10568843.17, stdev=56135662.02
- clat percentiles (nsec):
- | 1.00th=[ 708], 5.00th=[ 716], 10.00th=[ 716],
- | 20.00th=[ 724], 30.00th=[ 724], 40.00th=[ 732],
- | 50.00th=[ 764], 60.00th=[ 860], 70.00th=[ 924],
- | 80.00th=[ 1288], 90.00th=[ 2441216], 95.00th=[ 49020928],
- | 99.00th=[316669952], 99.50th=[450887680], 99.90th=[666894336],
- | 99.95th=[734003200], 99.99th=[977272832]
- bw ( KiB/s): min= 0, max=34871, per=92.05%, avg=24286.50, stdev=14968.74, samples=2
- iops : min= 0, max= 545, avg=379.50, stdev=234.05, samples=2
- lat (nsec) : 750=47.29%, 1000=26.10%
- lat (usec) : 2=9.46%, 4=0.05%, 10=1.23%, 20=0.46%, 50=0.15%
- lat (usec) : 100=0.07%, 250=0.06%, 500=0.26%, 750=0.85%, 1000=1.47%
- lat (msec) : 2=2.07%, 4=1.36%, 10=0.71%, 20=0.54%, 50=2.97%
- lat (msec) : 100=2.30%, 250=1.30%, 500=0.98%, 750=0.30%, 1000=0.03%
- cpu : usr=0.12%, sys=0.40%, ctx=13768, majf=0, minf=42
- IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=45580,0,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- readwrite: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:34:54 2019
- read: IOPS=32, BW=2105KiB/s (2155kB/s)(247MiB/120008msec)
- slat (nsec): min=0, max=267135k, avg=128561.10, stdev=4765559.54
- clat (nsec): min=0, max=638238k, avg=9563016.49, stdev=43149253.15
- lat (nsec): min=0, max=638271k, avg=9692556.97, stdev=43396123.16
- clat percentiles (nsec):
- | 1.00th=[ 716], 5.00th=[ 724], 10.00th=[ 732],
- | 20.00th=[ 788], 30.00th=[ 876], 40.00th=[ 996],
- | 50.00th=[ 1128], 60.00th=[ 1256], 70.00th=[ 1432],
- | 80.00th=[ 40192], 90.00th=[ 3751936], 95.00th=[ 54788096],
- | 99.00th=[254803968], 99.50th=[333447168], 99.90th=[497025024],
- | 99.95th=[509607936], 99.99th=[641728512]
- bw ( KiB/s): min= 0, max= 2695, per=7.97%, avg=2104.00, stdev=835.80, samples=2
- iops : min= 0, max= 42, avg=32.50, stdev=13.44, samples=2
- write: IOPS=34, BW=2206KiB/s (2259kB/s)(259MiB/120008msec)
- slat (nsec): min=0, max=10176k, avg=175306.89, stdev=326702.12
- clat (nsec): min=0, max=1839.1M, avg=107317664.14, stdev=164884666.88
- lat (nsec): min=0, max=1839.2M, avg=107494142.40, stdev=164838936.04
- clat percentiles (usec):
- | 1.00th=[ 963], 5.00th=[ 1221], 10.00th=[ 1467],
- | 20.00th=[ 2212], 30.00th=[ 3949], 40.00th=[ 12256],
- | 50.00th=[ 52167], 60.00th=[ 81265], 70.00th=[ 108528],
- | 80.00th=[ 154141], 90.00th=[ 312476], 95.00th=[ 484443],
- | 99.00th=[ 734004], 99.50th=[ 876610], 99.90th=[1149240],
- | 99.95th=[1300235], 99.99th=[1837106]
- bw ( KiB/s): min= 0, max= 2809, per=46.68%, avg=2204.50, stdev=854.89, samples=2
- iops : min= 0, max= 43, avg=34.00, stdev=12.73, samples=2
- lat (nsec) : 750=7.67%, 1000=12.10%
- lat (usec) : 2=17.50%, 4=0.09%, 10=0.22%, 20=0.95%, 50=0.78%
- lat (usec) : 100=1.11%, 250=0.52%, 500=0.40%, 750=0.49%, 1000=1.18%
- lat (msec) : 2=9.58%, 4=6.85%, 10=4.79%, 20=2.12%, 50=5.21%
- lat (msec) : 100=10.05%, 250=11.35%, 500=4.74%, 750=1.89%, 1000=0.30%
- cpu : usr=0.06%, sys=0.20%, ctx=9317, majf=0, minf=37
- IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=3948,4134,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- Run status group 0 (all jobs):
- READ: bw=25.8MiB/s (27.0MB/s), 2105KiB/s-23.7MiB/s (2155kB/s-24.9MB/s), io=3093MiB (3243MB), run=120008-120038msec
- WRITE: bw=4723KiB/s (4837kB/s), 2206KiB/s-2517KiB/s (2259kB/s-2577kB/s), io=554MiB (580MB), run=120008-120014msec
- Disk stats (read/write):
- rbd67: ios=8366/9287, merge=0/1, ticks=537421/816981, in_queue=1345358, util=13.52%
- <example-benchmark-fio-server-965-benchmark-cvg7r>
- randwrite: (groupid=0, jobs=1): err= 0: pid=11: Wed May 8 20:34:54 2019
- write: IOPS=46, BW=2948KiB/s (3018kB/s)(346MiB/120088msec)
- slat (nsec): min=0, max=479406k, avg=350130.57, stdev=8587720.43
- clat (nsec): min=0, max=1103.4M, avg=86740940.20, stdev=143041701.87
- lat (nsec): min=0, max=1103.5M, avg=87092248.17, stdev=143156153.60
- clat percentiles (usec):
- | 1.00th=[ 955], 5.00th=[ 1156], 10.00th=[ 1352],
- | 20.00th=[ 1795], 30.00th=[ 2737], 40.00th=[ 5735],
- | 50.00th=[ 20317], 60.00th=[ 67634], 70.00th=[ 96994],
- | 80.00th=[ 129500], 90.00th=[ 229639], 95.00th=[ 387974],
- | 99.00th=[ 700449], 99.50th=[ 801113], 99.90th=[1044382],
- | 99.95th=[1098908], 99.99th=[1098908]
- bw ( KiB/s): min= 0, max= 3585, per=54.32%, avg=2947.50, stdev=901.56, samples=2
- iops : min= 0, max= 56, avg=46.00, stdev=14.14, samples=2
- lat (usec) : 4=0.02%, 500=0.02%, 750=0.02%, 1000=1.74%
- lat (msec) : 2=21.58%, 4=12.23%, 10=9.90%, 20=4.40%, 50=5.70%
- lat (msec) : 100=15.77%, 250=19.52%, 500=6.10%, 750=2.35%, 1000=0.52%
- cpu : usr=0.07%, sys=0.19%, ctx=10526, majf=0, minf=26
- IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=0,5528,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- read: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:34:54 2019
- read: IOPS=291, BW=18.2MiB/s (19.1MB/s)(2187MiB/120030msec)
- slat (nsec): min=0, max=467384k, avg=48463.95, stdev=3199541.13
- clat (nsec): min=0, max=987567k, avg=13728849.56, stdev=63522045.77
- lat (nsec): min=0, max=987595k, avg=13778150.67, stdev=63621954.04
- clat percentiles (nsec):
- | 1.00th=[ 708], 5.00th=[ 716], 10.00th=[ 716],
- | 20.00th=[ 724], 30.00th=[ 732], 40.00th=[ 748],
- | 50.00th=[ 820], 60.00th=[ 884], 70.00th=[ 1096],
- | 80.00th=[ 522240], 90.00th=[ 20840448], 95.00th=[ 63700992],
- | 99.00th=[371195904], 99.50th=[492830720], 99.90th=[683671552],
- | 99.95th=[750780416], 99.99th=[851443712]
- bw ( KiB/s): min= 0, max=24862, per=88.70%, avg=18656.00, stdev=8776.61, samples=2
- iops : min= 0, max= 388, avg=291.00, stdev=137.18, samples=2
- lat (nsec) : 750=42.00%, 1000=25.39%
- lat (usec) : 2=11.31%, 4=0.08%, 10=0.45%, 20=0.12%, 50=0.11%
- lat (usec) : 100=0.05%, 250=0.06%, 500=0.44%, 750=1.18%, 1000=2.35%
- lat (msec) : 2=3.08%, 4=1.93%, 10=0.88%, 20=0.54%, 50=3.67%
- lat (msec) : 100=3.09%, 250=1.55%, 500=1.29%, 750=0.44%, 1000=0.05%
- cpu : usr=0.10%, sys=0.34%, ctx=14233, majf=0, minf=74
- IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=35008,0,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- readwrite: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:34:54 2019
- read: IOPS=37, BW=2379KiB/s (2436kB/s)(279MiB/120022msec)
- slat (nsec): min=0, max=2365.6k, avg=27578.90, stdev=85435.33
- clat (nsec): min=0, max=660093k, avg=10380038.22, stdev=49666588.17
- lat (nsec): min=0, max=660371k, avg=10408597.54, stdev=49675165.24
- clat percentiles (nsec):
- | 1.00th=[ 708], 5.00th=[ 716], 10.00th=[ 724],
- | 20.00th=[ 756], 30.00th=[ 868], 40.00th=[ 964],
- | 50.00th=[ 1112], 60.00th=[ 1240], 70.00th=[ 1448],
- | 80.00th=[ 94720], 90.00th=[ 3555328], 95.00th=[ 53739520],
- | 99.00th=[278921216], 99.50th=[408944640], 99.90th=[541065216],
- | 99.95th=[608174080], 99.99th=[658505728]
- bw ( KiB/s): min= 0, max= 2746, per=11.31%, avg=2379.00, stdev=519.02, samples=2
- iops : min= 0, max= 42, avg=36.50, stdev= 7.78, samples=2
- write: IOPS=38, BW=2480KiB/s (2540kB/s)(291MiB/120022msec)
- slat (nsec): min=0, max=421952k, avg=357499.62, stdev=8700755.43
- clat (nsec): min=0, max=1108.1M, avg=93392464.33, stdev=148165561.77
- lat (nsec): min=0, max=1108.1M, avg=93751123.75, stdev=148302960.36
- clat percentiles (usec):
- | 1.00th=[ 1037], 5.00th=[ 1237], 10.00th=[ 1500],
- | 20.00th=[ 2180], 30.00th=[ 3884], 40.00th=[ 9765],
- | 50.00th=[ 42206], 60.00th=[ 78119], 70.00th=[ 103285],
- | 80.00th=[ 133694], 90.00th=[ 235930], 95.00th=[ 396362],
- | 99.00th=[ 734004], 99.50th=[ 868221], 99.90th=[1061159],
- | 99.95th=[1098908], 99.99th=[1115685]
- bw ( KiB/s): min= 0, max= 2824, per=45.67%, avg=2478.00, stdev=489.32, samples=2
- iops : min= 0, max= 44, avg=38.50, stdev= 7.78, samples=2
- lat (nsec) : 750=9.30%, 1000=11.41%
- lat (usec) : 2=15.74%, 4=0.14%, 10=0.26%, 20=0.79%, 50=0.70%
- lat (usec) : 100=0.91%, 250=0.64%, 500=0.68%, 750=0.66%, 1000=1.39%
- lat (msec) : 2=9.94%, 4=7.20%, 10=5.30%, 20=2.69%, 50=4.98%
- lat (msec) : 100=9.83%, 250=12.06%, 500=3.59%, 750=1.35%, 1000=0.34%
- cpu : usr=0.06%, sys=0.22%, ctx=10593, majf=0, minf=35
- IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=4464,4648,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- Run status group 0 (all jobs):
- READ: bw=20.5MiB/s (21.5MB/s), 2379KiB/s-18.2MiB/s (2436kB/s-19.1MB/s), io=2465MiB (2585MB), run=120022-120030msec
- WRITE: bw=5426KiB/s (5557kB/s), 2480KiB/s-2948KiB/s (2540kB/s-3018kB/s), io=636MiB (667MB), run=120022-120088msec
- Disk stats (read/write):
- rbd329: ios=9355/10893, merge=0/1, ticks=557544/828062, in_queue=1375322, util=14.62%
- <example-benchmark-fio-server-714-benchmark-g6ljr>
- randwrite: (groupid=0, jobs=1): err= 0: pid=11: Wed May 8 20:34:55 2019
- write: IOPS=49, BW=3166KiB/s (3242kB/s)(371MiB/120038msec)
- slat (nsec): min=0, max=483547k, avg=436729.22, stdev=9798160.86
- clat (nsec): min=0, max=1052.2M, avg=80745876.19, stdev=137146591.70
- lat (nsec): min=0, max=1052.3M, avg=81183814.06, stdev=137524748.71
- clat percentiles (usec):
- | 1.00th=[ 996], 5.00th=[ 1205], 10.00th=[ 1401],
- | 20.00th=[ 1909], 30.00th=[ 2802], 40.00th=[ 5145],
- | 50.00th=[ 19268], 60.00th=[ 59507], 70.00th=[ 89654],
- | 80.00th=[ 116917], 90.00th=[ 208667], 95.00th=[ 404751],
- | 99.00th=[ 675283], 99.50th=[ 742392], 99.90th=[ 926942],
- | 99.95th=[1027605], 99.99th=[1052771]
- bw ( KiB/s): min= 0, max= 4302, per=54.75%, avg=3164.50, stdev=1608.67, samples=2
- iops : min= 0, max= 67, avg=49.00, stdev=25.46, samples=2
- lat (usec) : 4=0.03%, 100=0.02%, 750=0.07%, 1000=0.91%
- lat (msec) : 2=20.52%, 4=14.91%, 10=9.35%, 20=4.46%, 50=7.40%
- lat (msec) : 100=16.83%, 250=17.62%, 500=4.94%, 750=2.53%, 1000=0.40%
- cpu : usr=0.06%, sys=0.22%, ctx=11193, majf=0, minf=29
- IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=0,5936,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- read: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:34:55 2019
- read: IOPS=300, BW=18.8MiB/s (19.7MB/s)(2254MiB/120012msec)
- slat (nsec): min=0, max=484975k, avg=65282.63, stdev=4143390.88
- clat (nsec): min=0, max=1203.3M, avg=13303433.72, stdev=63562701.91
- lat (nsec): min=0, max=1203.6M, avg=13369542.88, stdev=63721578.58
- clat percentiles (nsec):
- | 1.00th=[ 716], 5.00th=[ 716], 10.00th=[ 724],
- | 20.00th=[ 732], 30.00th=[ 740], 40.00th=[ 756],
- | 50.00th=[ 844], 60.00th=[ 900], 70.00th=[ 1128],
- | 80.00th=[ 23424], 90.00th=[ 13434880], 95.00th=[ 60555264],
- | 99.00th=[ 367001600], 99.50th=[ 476053504], 99.90th=[ 700448768],
- | 99.95th=[ 792723456], 99.99th=[1044381696]
- bw ( KiB/s): min= 0, max=25968, per=88.44%, avg=19235.50, stdev=9521.19, samples=2
- iops : min= 0, max= 406, avg=300.50, stdev=149.20, samples=2
- lat (nsec) : 750=38.01%, 1000=28.33%
- lat (usec) : 2=10.58%, 4=0.05%, 10=1.86%, 20=1.10%, 50=1.26%
- lat (usec) : 100=0.10%, 250=0.07%, 500=0.09%, 750=0.86%, 1000=2.03%
- lat (msec) : 2=2.74%, 4=1.86%, 10=0.90%, 20=0.53%, 50=3.54%
- lat (msec) : 100=2.80%, 250=1.62%, 500=1.25%, 750=0.37%, 1000=0.05%
- cpu : usr=0.09%, sys=0.35%, ctx=13433, majf=0, minf=29
- IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=36110,0,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- readwrite: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:34:55 2019
- read: IOPS=39, BW=2518KiB/s (2578kB/s)(295MiB/120029msec)
- slat (nsec): min=0, max=349857k, avg=101363.58, stdev=5089683.13
- clat (nsec): min=0, max=1044.2M, avg=9561601.99, stdev=48388043.08
- lat (nsec): min=0, max=1044.2M, avg=9663946.77, stdev=48647228.37
- clat percentiles (nsec):
- | 1.00th=[ 716], 5.00th=[ 724], 10.00th=[ 732],
- | 20.00th=[ 780], 30.00th=[ 876], 40.00th=[ 996],
- | 50.00th=[ 1128], 60.00th=[ 1256], 70.00th=[ 1432],
- | 80.00th=[ 43776], 90.00th=[ 2899968], 95.00th=[ 51118080],
- | 99.00th=[ 248512512], 99.50th=[ 379584512], 99.90th=[ 566231040],
- | 99.95th=[ 666894336], 99.99th=[1044381696]
- bw ( KiB/s): min= 0, max= 3013, per=11.57%, avg=2517.50, stdev=700.74, samples=2
- iops : min= 0, max= 47, avg=39.00, stdev=11.31, samples=2
- write: IOPS=40, BW=2614KiB/s (2677kB/s)(306MiB/120029msec)
- slat (nsec): min=0, max=764714k, avg=504417.57, stdev=13821816.46
- clat (nsec): min=0, max=1313.0M, avg=88688665.47, stdev=146690691.98
- lat (nsec): min=0, max=1313.1M, avg=89194015.61, stdev=147209124.43
- clat percentiles (usec):
- | 1.00th=[ 1037], 5.00th=[ 1254], 10.00th=[ 1516],
- | 20.00th=[ 2180], 30.00th=[ 3654], 40.00th=[ 8586],
- | 50.00th=[ 33817], 60.00th=[ 67634], 70.00th=[ 93848],
- | 80.00th=[ 123208], 90.00th=[ 227541], 95.00th=[ 438305],
- | 99.00th=[ 683672], 99.50th=[ 775947], 99.90th=[1044382],
- | 99.95th=[1061159], 99.99th=[1317012]
- bw ( KiB/s): min= 0, max= 3117, per=45.20%, avg=2612.50, stdev=713.47, samples=2
- iops : min= 0, max= 48, avg=40.00, stdev=11.31, samples=2
- lat (nsec) : 750=8.05%, 1000=11.86%
- lat (usec) : 2=17.28%, 4=0.05%, 10=0.37%, 20=1.02%, 50=0.76%
- lat (usec) : 100=0.80%, 250=0.45%, 500=0.43%, 750=0.39%, 1000=1.44%
- lat (msec) : 2=9.94%, 4=7.65%, 10=5.33%, 20=2.76%, 50=5.62%
- lat (msec) : 100=10.70%, 250=10.22%, 500=2.99%, 750=1.54%, 1000=0.20%
- cpu : usr=0.06%, sys=0.25%, ctx=11131, majf=0, minf=36
- IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=4725,4900,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- Run status group 0 (all jobs):
- READ: bw=21.2MiB/s (22.3MB/s), 2518KiB/s-18.8MiB/s (2578kB/s-19.7MB/s), io=2550MiB (2673MB), run=120012-120029msec
- WRITE: bw=5781KiB/s (5919kB/s), 2614KiB/s-3166KiB/s (2677kB/s-3242kB/s), io=678MiB (711MB), run=120029-120038msec
- Disk stats (read/write):
- rbd292: ios=7536/10929, merge=0/2, ticks=534002/809056, in_queue=1333704, util=13.66%
- <example-benchmark-fio-server-386-benchmark-nzznc>
- randwrite: (groupid=0, jobs=1): err= 0: pid=11: Wed May 8 20:34:55 2019
- write: IOPS=40, BW=2596KiB/s (2658kB/s)(304MiB/120024msec)
- slat (nsec): min=0, max=579931k, avg=462348.78, stdev=11856183.51
- clat (nsec): min=0, max=1090.2M, avg=98730771.42, stdev=156908561.14
- lat (nsec): min=0, max=1289.8M, avg=99194171.74, stdev=157794900.50
- clat percentiles (usec):
- | 1.00th=[ 1012], 5.00th=[ 1221], 10.00th=[ 1418],
- | 20.00th=[ 1991], 30.00th=[ 3228], 40.00th=[ 7242],
- | 50.00th=[ 30802], 60.00th=[ 72877], 70.00th=[ 100140],
- | 80.00th=[ 131597], 90.00th=[ 316670], 95.00th=[ 484443],
- | 99.00th=[ 708838], 99.50th=[ 742392], 99.90th=[ 826278],
- | 99.95th=[ 910164], 99.99th=[1082131]
- bw ( KiB/s): min= 0, max= 3369, per=55.35%, avg=2593.50, stdev=1096.72, samples=2
- iops : min= 0, max= 52, avg=40.00, stdev=16.97, samples=2
- lat (usec) : 250=0.02%, 750=0.04%, 1000=0.84%
- lat (msec) : 2=19.24%, 4=13.86%, 10=8.49%, 20=4.61%, 50=6.41%
- lat (msec) : 100=16.55%, 250=17.58%, 500=7.77%, 750=4.17%, 1000=0.43%
- cpu : usr=0.05%, sys=0.17%, ctx=9233, majf=0, minf=30
- IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=0,4864,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- read: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:34:55 2019
- read: IOPS=290, BW=18.1MiB/s (18.0MB/s)(2174MiB/120020msec)
- slat (nsec): min=0, max=689525k, avg=62199.26, stdev=4622504.13
- clat (nsec): min=0, max=1083.4M, avg=13794212.01, stdev=64703199.17
- lat (nsec): min=0, max=1083.5M, avg=13857268.54, stdev=64873106.38
- clat percentiles (nsec):
- | 1.00th=[ 708], 5.00th=[ 716], 10.00th=[ 716],
- | 20.00th=[ 724], 30.00th=[ 740], 40.00th=[ 788],
- | 50.00th=[ 844], 60.00th=[ 908], 70.00th=[ 1128],
- | 80.00th=[ 8896], 90.00th=[ 15007744], 95.00th=[ 62128128],
- | 99.00th=[ 387973120], 99.50th=[ 492830720], 99.90th=[ 692060160],
- | 99.95th=[ 734003200], 99.99th=[1019215872]
- bw ( KiB/s): min= 0, max=22016, per=90.23%, avg=18550.50, stdev=4900.96, samples=2
- iops : min= 0, max= 344, avg=289.50, stdev=77.07, samples=2
- lat (nsec) : 750=33.40%, 1000=32.84%
- lat (usec) : 2=11.88%, 4=0.05%, 10=2.36%, 20=0.94%, 50=0.36%
- lat (usec) : 100=0.11%, 250=0.05%, 500=0.18%, 750=0.86%, 1000=1.88%
- lat (msec) : 2=2.49%, 4=1.47%, 10=0.79%, 20=0.64%, 50=3.51%
- lat (msec) : 100=2.72%, 250=1.69%, 500=1.29%, 750=0.44%, 1000=0.03%
- cpu : usr=0.09%, sys=0.32%, ctx=12530, majf=0, minf=30
- IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=34807,0,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- readwrite: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:34:55 2019
- read: IOPS=31, BW=2009KiB/s (2057kB/s)(236MiB/120021msec)
- slat (nsec): min=0, max=427153k, avg=136375.18, stdev=6958530.80
- clat (nsec): min=0, max=702092k, avg=9087273.02, stdev=46342012.02
- lat (nsec): min=0, max=702124k, avg=9224584.28, stdev=46846870.09
- clat percentiles (nsec):
- | 1.00th=[ 716], 5.00th=[ 724], 10.00th=[ 732],
- | 20.00th=[ 780], 30.00th=[ 868], 40.00th=[ 948],
- | 50.00th=[ 1080], 60.00th=[ 1240], 70.00th=[ 1384],
- | 80.00th=[ 31872], 90.00th=[ 1777664], 95.00th=[ 46399488],
- | 99.00th=[261095424], 99.50th=[367001600], 99.90th=[549453824],
- | 99.95th=[583008256], 99.99th=[700448768]
- bw ( KiB/s): min= 0, max= 2533, per=9.77%, avg=2008.50, stdev=741.76, samples=2
- iops : min= 0, max= 39, avg=31.00, stdev=11.31, samples=2
- write: IOPS=32, BW=2091KiB/s (2141kB/s)(245MiB/120021msec)
- slat (nsec): min=0, max=701141k, avg=429937.85, stdev=12610360.16
- clat (nsec): min=0, max=1018.3M, avg=113826493.31, stdev=166482932.38
- lat (nsec): min=0, max=1018.4M, avg=114257403.10, stdev=166748154.43
- clat percentiles (usec):
- | 1.00th=[ 1057], 5.00th=[ 1336], 10.00th=[ 1631],
- | 20.00th=[ 2442], 30.00th=[ 4555], 40.00th=[ 17171],
- | 50.00th=[ 56886], 60.00th=[ 86508], 70.00th=[ 109577],
- | 80.00th=[ 156238], 90.00th=[ 358613], 95.00th=[ 513803],
- | 99.00th=[ 725615], 99.50th=[ 759170], 99.90th=[ 809501],
- | 99.95th=[1019216], 99.99th=[1019216]
- bw ( KiB/s): min= 0, max= 2647, per=44.58%, avg=2089.00, stdev=789.13, samples=2
- iops : min= 0, max= 41, avg=32.00, stdev=12.73, samples=2
- lat (nsec) : 750=7.31%, 1000=13.84%
- lat (usec) : 2=16.54%, 4=0.08%, 10=0.25%, 20=0.98%, 50=0.70%
- lat (usec) : 100=1.03%, 250=0.57%, 500=0.57%, 750=0.57%, 1000=1.14%
- lat (msec) : 2=8.69%, 4=6.87%, 10=4.38%, 20=2.67%, 50=4.96%
- lat (msec) : 100=10.14%, 250=10.73%, 500=5.18%, 750=2.52%, 1000=0.29%
- cpu : usr=0.04%, sys=0.19%, ctx=8800, majf=0, minf=34
- IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=3768,3918,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- Run status group 0 (all jobs):
- READ: bw=20.1MiB/s (21.1MB/s), 2009KiB/s-18.1MiB/s (2057kB/s-18.0MB/s), io=2410MiB (2527MB), run=120020-120021msec
- WRITE: bw=4687KiB/s (4799kB/s), 2091KiB/s-2596KiB/s (2141kB/s-2658kB/s), io=549MiB (576MB), run=120021-120024msec
- Disk stats (read/write):
- rbd113: ios=7017/9006, merge=0/1, ticks=547807/860241, in_queue=1399971, util=11.72%
- <example-benchmark-fio-server-616-benchmark-vm6nt>
- randwrite: (groupid=0, jobs=1): err= 0: pid=11: Wed May 8 20:34:55 2019
- write: IOPS=42, BW=2740KiB/s (2806kB/s)(321MiB/120064msec)
- slat (nsec): min=0, max=522799k, avg=448436.58, stdev=10743519.39
- clat (nsec): min=0, max=1524.9M, avg=93344224.78, stdev=161360206.70
- lat (nsec): min=0, max=1524.9M, avg=93793721.83, stdev=161971351.87
- clat percentiles (usec):
- | 1.00th=[ 938], 5.00th=[ 1156], 10.00th=[ 1369],
- | 20.00th=[ 1876], 30.00th=[ 2868], 40.00th=[ 5145],
- | 50.00th=[ 19268], 60.00th=[ 60031], 70.00th=[ 87557],
- | 80.00th=[ 125305], 90.00th=[ 304088], 95.00th=[ 459277],
- | 99.00th=[ 801113], 99.50th=[ 935330], 99.90th=[1149240],
- | 99.95th=[1233126], 99.99th=[1518339]
- bw ( KiB/s): min= 0, max= 1793, per=34.93%, avg=1793.00, stdev= 0.00, samples=1
- iops : min= 0, max= 28, avg=28.00, stdev= 0.00, samples=1
- lat (usec) : 500=0.02%, 750=0.04%, 1000=1.87%
- lat (msec) : 2=19.66%, 4=14.99%, 10=9.93%, 20=3.89%, 50=6.64%
- lat (msec) : 100=17.05%, 250=14.05%, 500=8.00%, 750=2.76%, 1000=0.91%
- cpu : usr=0.04%, sys=0.20%, ctx=9807, majf=0, minf=32
- IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=0,5138,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- read: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:34:55 2019
- read: IOPS=383, BW=23.9MiB/s (25.1MB/s)(2872MiB/120013msec)
- slat (nsec): min=0, max=471122k, avg=46847.59, stdev=3381194.72
- clat (nsec): min=0, max=1407.2M, avg=10409463.13, stdev=55502628.48
- lat (nsec): min=0, max=1407.2M, avg=10457142.71, stdev=55610776.23
- clat percentiles (nsec):
- | 1.00th=[ 708], 5.00th=[ 708], 10.00th=[ 716],
- | 20.00th=[ 724], 30.00th=[ 732], 40.00th=[ 740],
- | 50.00th=[ 812], 60.00th=[ 876], 70.00th=[ 972],
- | 80.00th=[ 1432], 90.00th=[ 3424256], 95.00th=[ 50593792],
- | 99.00th=[ 291504128], 99.50th=[ 450887680], 99.90th=[ 675282944],
- | 99.95th=[ 767557632], 99.99th=[1098907648]
- bw ( KiB/s): min= 0, max=34035, per=100.00%, avg=34035.00, stdev= 0.00, samples=1
- iops : min= 0, max= 532, avg=532.00, stdev= 0.00, samples=1
- lat (nsec) : 750=43.64%, 1000=27.38%
- lat (usec) : 2=10.12%, 4=0.03%, 10=0.80%, 20=0.30%, 50=0.14%
- lat (usec) : 100=0.06%, 250=0.06%, 500=0.22%, 750=0.88%, 1000=2.09%
- lat (msec) : 2=2.76%, 4=1.76%, 10=0.97%, 20=0.57%, 50=3.17%
- lat (msec) : 100=2.66%, 250=1.23%, 500=0.83%, 750=0.29%, 1000=0.04%
- cpu : usr=0.13%, sys=0.41%, ctx=16042, majf=0, minf=31
- IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=45981,0,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- readwrite: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:34:55 2019
- read: IOPS=35, BW=2297KiB/s (2352kB/s)(269MiB/120051msec)
- slat (nsec): min=0, max=3687.3k, avg=25139.08, stdev=91477.78
- clat (nsec): min=0, max=1038.4M, avg=7695728.63, stdev=45093642.98
- lat (nsec): min=0, max=1038.6M, avg=7721793.49, stdev=45105435.26
- clat percentiles (nsec):
- | 1.00th=[ 708], 5.00th=[ 716], 10.00th=[ 732],
- | 20.00th=[ 756], 30.00th=[ 852], 40.00th=[ 948],
- | 50.00th=[ 1112], 60.00th=[ 1240], 70.00th=[ 1384],
- | 80.00th=[ 15808], 90.00th=[ 1777664], 95.00th=[ 41156608],
- | 99.00th=[ 183500800], 99.50th=[ 304087040], 99.90th=[ 666894336],
- | 99.95th=[ 725614592], 99.99th=[1035993088]
- bw ( KiB/s): min= 0, max= 2781, per=8.57%, avg=2297.00, stdev=684.48, samples=2
- iops : min= 0, max= 43, avg=35.50, stdev=10.61, samples=2
- write: IOPS=37, BW=2393KiB/s (2451kB/s)(281MiB/120051msec)
- slat (nsec): min=0, max=521733k, avg=510207.12, stdev=12528343.03
- clat (nsec): min=0, max=1456.0M, avg=99230619.88, stdev=167653400.45
- lat (nsec): min=0, max=1456.1M, avg=99741774.10, stdev=168293190.74
- clat percentiles (usec):
- | 1.00th=[ 971], 5.00th=[ 1254], 10.00th=[ 1500],
- | 20.00th=[ 2089], 30.00th=[ 3326], 40.00th=[ 6456],
- | 50.00th=[ 23987], 60.00th=[ 61080], 70.00th=[ 88605],
- | 80.00th=[ 131597], 90.00th=[ 341836], 95.00th=[ 476054],
- | 99.00th=[ 801113], 99.50th=[ 935330], 99.90th=[1216349],
- | 99.95th=[1367344], 99.99th=[1451230]
- bw ( KiB/s): min= 0, max= 2873, per=46.60%, avg=2392.00, stdev=680.24, samples=2
- iops : min= 0, max= 44, avg=36.50, stdev=10.61, samples=2
- lat (nsec) : 750=9.00%, 1000=12.22%
- lat (usec) : 2=17.29%, 4=0.08%, 10=0.19%, 20=0.72%, 50=0.70%
- lat (usec) : 100=0.61%, 250=0.48%, 500=0.49%, 750=0.44%, 1000=1.53%
- lat (msec) : 2=10.24%, 4=7.73%, 10=6.29%, 20=2.58%, 50=5.10%
- lat (msec) : 100=9.71%, 250=7.68%, 500=4.64%, 750=1.66%, 1000=0.50%
- cpu : usr=0.05%, sys=0.21%, ctx=10071, majf=0, minf=38
- IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=4311,4486,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- Run status group 0 (all jobs):
- READ: bw=26.2MiB/s (27.4MB/s), 2297KiB/s-23.9MiB/s (2352kB/s-25.1MB/s), io=3141MiB (3294MB), run=120013-120051msec
- WRITE: bw=5133KiB/s (5256kB/s), 2393KiB/s-2740KiB/s (2451kB/s-2806kB/s), io=602MiB (631MB), run=120051-120064msec
- Disk stats (read/write):
- rbd226: ios=8971/10050, merge=0/2, ticks=526494/837841, in_queue=1354664, util=14.05%
- <example-benchmark-fio-server-810-benchmark-s6m2k>
- randwrite: (groupid=0, jobs=1): err= 0: pid=11: Wed May 8 20:34:55 2019
- write: IOPS=45, BW=2945KiB/s (3016kB/s)(345MiB/120041msec)
- slat (nsec): min=0, max=573788k, avg=367710.09, stdev=9878806.33
- clat (nsec): min=0, max=1214.2M, avg=86620290.25, stdev=149814104.02
- lat (nsec): min=0, max=1214.3M, avg=86989216.49, stdev=149987111.38
- clat percentiles (usec):
- | 1.00th=[ 996], 5.00th=[ 1221], 10.00th=[ 1418],
- | 20.00th=[ 1909], 30.00th=[ 2900], 40.00th=[ 5211],
- | 50.00th=[ 13435], 60.00th=[ 47973], 70.00th=[ 85459],
- | 80.00th=[ 120062], 90.00th=[ 283116], 95.00th=[ 438305],
- | 99.00th=[ 708838], 99.50th=[ 817890], 99.90th=[ 952108],
- | 99.95th=[1044382], 99.99th=[1216349]
- bw ( KiB/s): min= 0, max= 3560, per=55.22%, avg=2944.00, stdev=871.16, samples=2
- iops : min= 0, max= 55, avg=45.50, stdev=13.44, samples=2
- lat (usec) : 4=0.02%, 750=0.04%, 1000=1.00%
- lat (msec) : 2=20.12%, 4=14.45%, 10=12.19%, 20=5.11%, 50=7.63%
- lat (msec) : 100=14.69%, 250=13.80%, 500=7.66%, 750=2.54%, 1000=0.76%
- cpu : usr=0.06%, sys=0.19%, ctx=10456, majf=0, minf=30
- IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=0,5521,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- read: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:34:55 2019
- read: IOPS=294, BW=18.4MiB/s (19.3MB/s)(2204MiB/120030msec)
- slat (nsec): min=0, max=310412k, avg=40472.37, stdev=2303449.06
- clat (nsec): min=0, max=1217.2M, avg=13618848.09, stdev=62825076.11
- lat (nsec): min=0, max=1217.3M, avg=13660203.59, stdev=62875748.57
- clat percentiles (nsec):
- | 1.00th=[ 708], 5.00th=[ 708], 10.00th=[ 716],
- | 20.00th=[ 724], 30.00th=[ 732], 40.00th=[ 764],
- | 50.00th=[ 836], 60.00th=[ 892], 70.00th=[ 1096],
- | 80.00th=[ 561152], 90.00th=[ 23461888], 95.00th=[ 63176704],
- | 99.00th=[ 354418688], 99.50th=[ 476053504], 99.90th=[ 700448768],
- | 99.95th=[ 759169024], 99.99th=[1061158912]
- bw ( KiB/s): min= 0, max=24758, per=89.20%, avg=18807.00, stdev=8415.98, samples=2
- iops : min= 0, max= 387, avg=293.50, stdev=132.23, samples=2
- lat (nsec) : 750=35.79%, 1000=31.21%
- lat (usec) : 2=11.36%, 4=0.04%, 10=0.62%, 20=0.46%, 50=0.17%
- lat (usec) : 100=0.10%, 250=0.05%, 500=0.15%, 750=1.01%, 1000=2.07%
- lat (msec) : 2=3.01%, 4=1.86%, 10=0.96%, 20=0.78%, 50=3.94%
- lat (msec) : 100=3.04%, 250=1.73%, 500=1.21%, 750=0.38%, 1000=0.05%
- cpu : usr=0.09%, sys=0.34%, ctx=14244, majf=0, minf=29
- IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=35289,0,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- readwrite: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:34:55 2019
- read: IOPS=35, BW=2281KiB/s (2336kB/s)(267MiB/120036msec)
- slat (nsec): min=0, max=378161k, avg=188024.60, stdev=7512404.09
- clat (nsec): min=0, max=739179k, avg=10245570.83, stdev=48682354.00
- lat (nsec): min=0, max=739211k, avg=10434521.92, stdev=49233760.49
- clat percentiles (nsec):
- | 1.00th=[ 708], 5.00th=[ 716], 10.00th=[ 732],
- | 20.00th=[ 772], 30.00th=[ 868], 40.00th=[ 980],
- | 50.00th=[ 1112], 60.00th=[ 1240], 70.00th=[ 1416],
- | 80.00th=[ 56576], 90.00th=[ 4014080], 95.00th=[ 52690944],
- | 99.00th=[283115520], 99.50th=[354418688], 99.90th=[557842432],
- | 99.95th=[616562688], 99.99th=[742391808]
- bw ( KiB/s): min= 0, max= 2588, per=10.82%, avg=2281.50, stdev=433.46, samples=2
- iops : min= 0, max= 40, avg=35.00, stdev= 7.07, samples=2
- write: IOPS=37, BW=2386KiB/s (2444kB/s)(280MiB/120036msec)
- slat (nsec): min=0, max=6032.8k, avg=173025.17, stdev=261709.00
- clat (nsec): min=0, max=1270.0M, avg=97171527.70, stdev=157884329.78
- lat (nsec): min=0, max=1270.3M, avg=97346023.25, stdev=157836529.25
- clat percentiles (usec):
- | 1.00th=[ 1029], 5.00th=[ 1270], 10.00th=[ 1532],
- | 20.00th=[ 2180], 30.00th=[ 3654], 40.00th=[ 7701],
- | 50.00th=[ 24773], 60.00th=[ 64750], 70.00th=[ 94897],
- | 80.00th=[ 135267], 90.00th=[ 312476], 95.00th=[ 476054],
- | 99.00th=[ 742392], 99.50th=[ 843056], 99.90th=[ 935330],
- | 99.95th=[ 952108], 99.99th=[1266680]
- bw ( KiB/s): min= 0, max= 2692, per=44.73%, avg=2384.50, stdev=434.87, samples=2
- iops : min= 0, max= 42, avg=37.00, stdev= 7.07, samples=2
- lat (nsec) : 750=8.44%, 1000=11.81%
- lat (usec) : 2=17.02%, 4=0.05%, 10=0.14%, 20=0.80%, 50=0.69%
- lat (usec) : 100=0.74%, 250=0.49%, 500=0.41%, 750=0.56%, 1000=1.36%
- lat (msec) : 2=9.93%, 4=7.71%, 10=5.94%, 20=3.15%, 50=5.67%
- lat (msec) : 100=9.30%, 250=8.91%, 500=4.67%, 750=1.75%, 1000=0.47%
- cpu : usr=0.06%, sys=0.21%, ctx=10250, majf=0, minf=34
- IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=4280,4473,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- Run status group 0 (all jobs):
- READ: bw=20.6MiB/s (21.6MB/s), 2281KiB/s-18.4MiB/s (2336kB/s-19.3MB/s), io=2472MiB (2592MB), run=120030-120036msec
- WRITE: bw=5332KiB/s (5459kB/s), 2386KiB/s-2945KiB/s (2444kB/s-3016kB/s), io=625MiB (655MB), run=120036-120041msec
- Disk stats (read/write):
- rbd188: ios=7908/10343, merge=0/1, ticks=544904/835140, in_queue=1370788, util=13.49%
- <example-benchmark-fio-server-363-benchmark-v4496>
- randwrite: (groupid=0, jobs=1): err= 0: pid=11: Wed May 8 20:34:55 2019
- write: IOPS=41, BW=2646KiB/s (2709kB/s)(310MiB/120098msec)
- slat (nsec): min=0, max=483563k, avg=335624.44, stdev=7666597.67
- clat (nsec): min=0, max=1102.5M, avg=96744121.92, stdev=159436843.91
- lat (nsec): min=0, max=1102.5M, avg=97080769.02, stdev=159472833.05
- clat percentiles (usec):
- | 1.00th=[ 938], 5.00th=[ 1106], 10.00th=[ 1303],
- | 20.00th=[ 1663], 30.00th=[ 2343], 40.00th=[ 3818],
- | 50.00th=[ 23200], 60.00th=[ 72877], 70.00th=[ 103285],
- | 80.00th=[ 129500], 90.00th=[ 308282], 95.00th=[ 488637],
- | 99.00th=[ 742392], 99.50th=[ 801113], 99.90th=[ 901776],
- | 99.95th=[ 935330], 99.99th=[1098908]
- bw ( KiB/s): min= 0, max= 4098, per=55.36%, avg=2645.50, stdev=2054.15, samples=2
- iops : min= 0, max= 64, avg=41.00, stdev=32.53, samples=2
- lat (usec) : 2=0.02%, 4=0.04%, 500=0.02%, 750=0.08%, 1000=1.67%
- lat (msec) : 2=24.00%, 4=14.89%, 10=6.15%, 20=2.56%, 50=4.90%
- lat (msec) : 100=14.53%, 250=19.87%, 500=6.59%, 750=3.89%, 1000=0.81%
- cpu : usr=0.05%, sys=0.17%, ctx=9368, majf=0, minf=27
- IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=0,4962,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- read: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:34:55 2019
- read: IOPS=323, BW=20.2MiB/s (21.2MB/s)(2424MiB/120013msec)
- slat (nsec): min=0, max=477001k, avg=36573.14, stdev=2637904.02
- clat (nsec): min=0, max=940724k, avg=12367744.96, stdev=60911558.88
- lat (nsec): min=0, max=941214k, avg=12405146.29, stdev=60976937.14
- clat percentiles (nsec):
- | 1.00th=[ 708], 5.00th=[ 708], 10.00th=[ 716],
- | 20.00th=[ 724], 30.00th=[ 732], 40.00th=[ 740],
- | 50.00th=[ 796], 60.00th=[ 860], 70.00th=[ 940],
- | 80.00th=[ 1304], 90.00th=[ 6127616], 95.00th=[ 55836672],
- | 99.00th=[367001600], 99.50th=[471859200], 99.90th=[700448768],
- | 99.95th=[767557632], 99.99th=[843055104]
- bw ( KiB/s): min= 0, max=25084, per=91.05%, avg=20685.00, stdev=6221.13, samples=2
- iops : min= 0, max= 392, avg=323.00, stdev=97.58, samples=2
- lat (nsec) : 750=44.89%, 1000=27.73%
- lat (usec) : 2=9.91%, 4=0.08%, 10=0.35%, 20=0.21%, 50=0.19%
- lat (usec) : 100=0.06%, 250=0.04%, 500=0.16%, 750=0.79%, 1000=1.59%
- lat (msec) : 2=2.21%, 4=1.37%, 10=0.80%, 20=0.56%, 50=3.41%
- lat (msec) : 100=2.67%, 250=1.38%, 500=1.21%, 750=0.34%, 1000=0.06%
- cpu : usr=0.10%, sys=0.35%, ctx=12840, majf=0, minf=29
- IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=38815,0,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- readwrite: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:34:55 2019
- read: IOPS=31, BW=2033KiB/s (2082kB/s)(238MiB/120015msec)
- slat (nsec): min=0, max=436760k, avg=141784.50, stdev=7073229.93
- clat (nsec): min=0, max=698805k, avg=9375440.61, stdev=44524086.18
- lat (nsec): min=0, max=698827k, avg=9518114.96, stdev=45063482.36
- clat percentiles (nsec):
- | 1.00th=[ 716], 5.00th=[ 732], 10.00th=[ 748],
- | 20.00th=[ 820], 30.00th=[ 884], 40.00th=[ 988],
- | 50.00th=[ 1112], 60.00th=[ 1224], 70.00th=[ 1416],
- | 80.00th=[ 67072], 90.00th=[ 5537792], 95.00th=[ 54788096],
- | 99.00th=[221249536], 99.50th=[325058560], 99.90th=[599785472],
- | 99.95th=[683671552], 99.99th=[700448768]
- bw ( KiB/s): min= 0, max= 3070, per=8.94%, avg=2032.00, stdev=1467.95, samples=2
- iops : min= 0, max= 47, avg=31.00, stdev=22.63, samples=2
- write: IOPS=33, BW=2135KiB/s (2186kB/s)(250MiB/120015msec)
- slat (nsec): min=0, max=192477k, avg=220163.16, stdev=3048609.65
- clat (nsec): min=0, max=1102.4M, avg=110902683.24, stdev=169976312.05
- lat (nsec): min=0, max=1102.6M, avg=111123871.68, stdev=169947686.95
- clat percentiles (usec):
- | 1.00th=[ 963], 5.00th=[ 1156], 10.00th=[ 1369],
- | 20.00th=[ 1926], 30.00th=[ 3032], 40.00th=[ 8586],
- | 50.00th=[ 54264], 60.00th=[ 86508], 70.00th=[ 111674],
- | 80.00th=[ 141558], 90.00th=[ 350225], 95.00th=[ 522191],
- | 99.00th=[ 750781], 99.50th=[ 843056], 99.90th=[ 926942],
- | 99.95th=[ 935330], 99.99th=[1098908]
- bw ( KiB/s): min= 0, max= 3198, per=44.64%, avg=2133.50, stdev=1505.43, samples=2
- iops : min= 0, max= 49, avg=32.50, stdev=23.33, samples=2
- lat (nsec) : 750=5.14%, 1000=14.97%
- lat (usec) : 2=16.56%, 4=0.08%, 10=0.14%, 20=0.82%, 50=0.78%
- lat (usec) : 100=1.10%, 250=0.52%, 500=0.59%, 750=0.64%, 1000=1.61%
- lat (msec) : 2=11.03%, 4=7.50%, 10=3.43%, 20=1.87%, 50=4.54%
- lat (msec) : 100=9.75%, 250=11.70%, 500=4.33%, 750=2.42%, 1000=0.50%
- cpu : usr=0.05%, sys=0.19%, ctx=9130, majf=0, minf=30
- IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=3814,4000,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- Run status group 0 (all jobs):
- READ: bw=22.2MiB/s (23.3MB/s), 2033KiB/s-20.2MiB/s (2082kB/s-21.2MB/s), io=2663MiB (2792MB), run=120013-120015msec
- WRITE: bw=4779KiB/s (4894kB/s), 2135KiB/s-2646KiB/s (2186kB/s-2709kB/s), io=561MiB (588MB), run=120015-120098msec
- Disk stats (read/write):
- rbd141: ios=7942/9637, merge=0/3, ticks=548741/850068, in_queue=1389903, util=12.74%
- Jobs: 63 (f=59): [_(2),m(1)][99.8%][r=1231M,w=976M][r=19.2k,w=15.2k IOPS][eta 00m:55s] <example-benchmark-fio-server-910-benchmark-74gxv>
- randwrite: (groupid=0, jobs=1): err= 0: pid=11: Wed May 8 20:34:55 2019
- write: IOPS=47, BW=3049KiB/s (3122kB/s)(358MiB/120101msec)
- slat (nsec): min=0, max=587401k, avg=499377.44, stdev=12120076.80
- clat (nsec): min=0, max=1699.1M, avg=83679046.09, stdev=152339670.45
- lat (nsec): min=0, max=1699.1M, avg=84179546.98, stdev=152850121.23
- clat percentiles (usec):
- | 1.00th=[ 979], 5.00th=[ 1156], 10.00th=[ 1336],
- | 20.00th=[ 1778], 30.00th=[ 2638], 40.00th=[ 4686],
- | 50.00th=[ 14222], 60.00th=[ 51643], 70.00th=[ 87557],
- | 80.00th=[ 117965], 90.00th=[ 225444], 95.00th=[ 413139],
- | 99.00th=[ 725615], 99.50th=[ 884999], 99.90th=[1149240],
- | 99.95th=[1300235], 99.99th=[1702888]
- bw ( KiB/s): min= 0, max= 3874, per=54.17%, avg=3049.50, stdev=1166.02, samples=2
- iops : min= 0, max= 60, avg=47.00, stdev=18.38, samples=2
- lat (usec) : 750=0.07%, 1000=1.42%
- lat (msec) : 2=21.72%, 4=14.44%, 10=10.21%, 20=4.77%, 50=6.92%
- lat (msec) : 100=14.43%, 250=16.89%, 500=5.58%, 750=2.64%, 1000=0.65%
- cpu : usr=0.06%, sys=0.20%, ctx=10898, majf=0, minf=30
- IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=0,5719,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- read: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:34:55 2019
- read: IOPS=369, BW=23.0MiB/s (24.2MB/s)(2766MiB/120003msec)
- slat (nsec): min=0, max=586968k, avg=49643.35, stdev=3526447.54
- clat (nsec): min=0, max=1687.4M, avg=10816930.67, stdev=56864630.85
- lat (nsec): min=0, max=1687.5M, avg=10867413.28, stdev=56980443.13
- clat percentiles (nsec):
- | 1.00th=[ 708], 5.00th=[ 716], 10.00th=[ 716],
- | 20.00th=[ 724], 30.00th=[ 732], 40.00th=[ 740],
- | 50.00th=[ 812], 60.00th=[ 868], 70.00th=[ 948],
- | 80.00th=[ 1384], 90.00th=[ 3555328], 95.00th=[ 50593792],
- | 99.00th=[ 308281344], 99.50th=[ 438304768], 99.90th=[ 658505728],
- | 99.95th=[ 734003200], 99.99th=[1233125376]
- bw ( KiB/s): min= 0, max=34327, per=90.47%, avg=23600.50, stdev=15169.56, samples=2
- iops : min= 0, max= 537, avg=369.00, stdev=237.59, samples=2
- lat (nsec) : 750=43.42%, 1000=28.81%
- lat (usec) : 2=9.35%, 4=0.04%, 10=0.87%, 20=1.08%, 50=0.38%
- lat (usec) : 100=0.12%, 250=0.07%, 500=0.10%, 750=0.73%, 1000=1.59%
- lat (msec) : 2=2.31%, 4=1.29%, 10=0.76%, 20=0.71%, 50=3.29%
- lat (msec) : 100=2.40%, 250=1.39%, 500=0.95%, 750=0.28%, 1000=0.03%
- cpu : usr=0.10%, sys=0.41%, ctx=14046, majf=0, minf=97
- IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=44330,0,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- readwrite: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:34:55 2019
- read: IOPS=38, BW=2487KiB/s (2547kB/s)(292MiB/120019msec)
- slat (nsec): min=0, max=368100k, avg=106961.37, stdev=5389940.62
- clat (nsec): min=0, max=759558k, avg=9002016.33, stdev=40516136.77
- lat (nsec): min=0, max=759588k, avg=9109980.22, stdev=40868878.36
- clat percentiles (nsec):
- | 1.00th=[ 716], 5.00th=[ 724], 10.00th=[ 732],
- | 20.00th=[ 804], 30.00th=[ 884], 40.00th=[ 996],
- | 50.00th=[ 1144], 60.00th=[ 1256], 70.00th=[ 1432],
- | 80.00th=[ 71168], 90.00th=[ 5275648], 95.00th=[ 54263808],
- | 99.00th=[212860928], 99.50th=[308281344], 99.90th=[429916160],
- | 99.95th=[476053504], 99.99th=[759169024]
- bw ( KiB/s): min= 0, max= 2916, per=9.53%, avg=2486.50, stdev=607.40, samples=2
- iops : min= 0, max= 45, avg=38.50, stdev= 9.19, samples=2
- write: IOPS=40, BW=2583KiB/s (2645kB/s)(303MiB/120019msec)
- slat (nsec): min=0, max=626891k, avg=490839.66, stdev=12588951.98
- clat (nsec): min=0, max=1796.9M, avg=90155001.53, stdev=160192506.95
- lat (nsec): min=0, max=1796.0M, avg=90646821.59, stdev=160496253.46
- clat percentiles (usec):
- | 1.00th=[ 1012], 5.00th=[ 1237], 10.00th=[ 1434],
- | 20.00th=[ 2040], 30.00th=[ 3294], 40.00th=[ 6587],
- | 50.00th=[ 21627], 60.00th=[ 59507], 70.00th=[ 91751],
- | 80.00th=[ 121111], 90.00th=[ 252707], 95.00th=[ 442500],
- | 99.00th=[ 759170], 99.50th=[ 884999], 99.90th=[1300235],
- | 99.95th=[1686111], 99.99th=[1803551]
- bw ( KiB/s): min= 0, max= 3015, per=45.85%, avg=2581.50, stdev=613.06, samples=2
- iops : min= 0, max= 47, avg=40.00, stdev= 9.90, samples=2
- lat (nsec) : 750=7.48%, 1000=12.19%
- lat (usec) : 2=17.28%, 4=0.13%, 10=0.34%, 20=0.79%, 50=0.62%
- lat (usec) : 100=0.64%, 250=0.64%, 500=0.63%, 750=0.41%, 1000=1.38%
- lat (msec) : 2=10.86%, 4=7.39%, 10=6.11%, 20=3.06%, 50=5.52%
- lat (msec) : 100=9.36%, 250=9.71%, 500=3.46%, 750=1.48%, 1000=0.39%
- cpu : usr=0.05%, sys=0.24%, ctx=11003, majf=0, minf=30
- IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=4666,4841,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- Run status group 0 (all jobs):
- READ: bw=25.5MiB/s (26.7MB/s), 2487KiB/s-23.0MiB/s (2547kB/s-24.2MB/s), io=3058MiB (3206MB), run=120003-120019msec
- WRITE: bw=5630KiB/s (5766kB/s), 2583KiB/s-3049KiB/s (2645kB/s-3122kB/s), io=660MiB (692MB), run=120019-120101msec
- Disk stats (read/write):
- rbd320: ios=8138/10744, merge=0/1, ticks=543117/803685, in_queue=1337178, util=13.96%
- <example-benchmark-fio-server-697-benchmark-v5m2j>
- randwrite: (groupid=0, jobs=1): err= 0: pid=11: Wed May 8 20:34:55 2019
- write: IOPS=52, BW=3334KiB/s (3414kB/s)(391MiB/120024msec)
- slat (nsec): min=0, max=950382k, avg=452969.44, stdev=14552434.79
- clat (nsec): min=0, max=1440.0M, avg=76854293.56, stdev=153490325.63
- lat (nsec): min=0, max=1441.2M, avg=77308421.84, stdev=154089812.05
- clat percentiles (usec):
- | 1.00th=[ 955], 5.00th=[ 1205], 10.00th=[ 1401],
- | 20.00th=[ 1762], 30.00th=[ 2376], 40.00th=[ 3589],
- | 50.00th=[ 6849], 60.00th=[ 23725], 70.00th=[ 76022],
- | 80.00th=[ 107480], 90.00th=[ 208667], 95.00th=[ 392168],
- | 99.00th=[ 784335], 99.50th=[ 910164], 99.90th=[1199571],
- | 99.95th=[1300235], 99.99th=[1434452]
- bw ( KiB/s): min= 0, max= 5185, per=58.75%, avg=3333.00, stdev=2619.12, samples=2
- iops : min= 0, max= 81, avg=52.00, stdev=41.01, samples=2
- lat (usec) : 100=0.02%, 250=0.03%, 750=0.03%, 1000=1.36%
- lat (msec) : 2=23.31%, 4=17.55%, 10=11.94%, 20=4.78%, 50=4.61%
- lat (msec) : 100=14.32%, 250=13.89%, 500=4.88%, 750=2.08%, 1000=0.99%
- cpu : usr=0.06%, sys=0.22%, ctx=11738, majf=0, minf=27
- IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=0,6250,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- read: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:34:55 2019
- read: IOPS=341, BW=21.3MiB/s (22.4MB/s)(2561MiB/120020msec)
- slat (nsec): min=0, max=871267k, avg=58333.28, stdev=5300696.13
- clat (nsec): min=0, max=1267.3M, avg=11691015.06, stdev=59871844.84
- lat (nsec): min=0, max=1267.5M, avg=11750191.55, stdev=60112618.15
- clat percentiles (nsec):
- | 1.00th=[ 708], 5.00th=[ 716], 10.00th=[ 716],
- | 20.00th=[ 724], 30.00th=[ 732], 40.00th=[ 772],
- | 50.00th=[ 836], 60.00th=[ 892], 70.00th=[ 1032],
- | 80.00th=[ 1576], 90.00th=[ 4882432], 95.00th=[ 53739520],
- | 99.00th=[ 337641472], 99.50th=[ 467664896], 99.90th=[ 692060160],
- | 99.95th=[ 784334848], 99.99th=[1082130432]
- bw ( KiB/s): min= 0, max=28268, per=90.74%, avg=21854.00, stdev=9070.77, samples=2
- iops : min= 0, max= 442, avg=341.50, stdev=142.13, samples=2
- lat (nsec) : 750=36.43%, 1000=32.41%
- lat (usec) : 2=11.69%, 4=0.06%, 10=0.56%, 20=0.22%, 50=0.16%
- lat (usec) : 100=0.09%, 250=0.04%, 500=0.25%, 750=1.03%, 1000=2.28%
- lat (msec) : 2=2.99%, 4=1.55%, 10=0.76%, 20=0.64%, 50=3.41%
- lat (msec) : 100=2.65%, 250=1.38%, 500=1.02%, 750=0.31%, 1000=0.04%
- cpu : usr=0.11%, sys=0.39%, ctx=14938, majf=0, minf=30
- IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=41006,0,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- readwrite: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:34:55 2019
- read: IOPS=34, BW=2232KiB/s (2286kB/s)(262MiB/120028msec)
- slat (nsec): min=0, max=3261.9k, avg=30353.32, stdev=99290.01
- clat (nsec): min=0, max=1235.7M, avg=9591222.87, stdev=49457877.63
- lat (nsec): min=0, max=1235.9M, avg=9622550.11, stdev=49466608.08
- clat percentiles (nsec):
- | 1.00th=[ 708], 5.00th=[ 724], 10.00th=[ 732],
- | 20.00th=[ 804], 30.00th=[ 884], 40.00th=[ 1020],
- | 50.00th=[ 1160], 60.00th=[ 1288], 70.00th=[ 1496],
- | 80.00th=[ 193536], 90.00th=[ 6127616], 95.00th=[ 52690944],
- | 99.00th=[ 235929600], 99.50th=[ 354418688], 99.90th=[ 517996544],
- | 99.95th=[ 641728512], 99.99th=[1233125376]
- bw ( KiB/s): min= 0, max= 3177, per=9.27%, avg=2232.00, stdev=1336.43, samples=2
- iops : min= 0, max= 49, avg=34.50, stdev=20.51, samples=2
- write: IOPS=36, BW=2340KiB/s (2396kB/s)(274MiB/120028msec)
- slat (nsec): min=0, max=839359k, avg=446443.00, stdev=13829654.09
- clat (nsec): min=0, max=1464.5M, avg=100143464.64, stdev=174579671.22
- lat (nsec): min=0, max=1464.6M, avg=100590859.57, stdev=175258011.37
- clat percentiles (usec):
- | 1.00th=[ 1057], 5.00th=[ 1287], 10.00th=[ 1532],
- | 20.00th=[ 2212], 30.00th=[ 3589], 40.00th=[ 7373],
- | 50.00th=[ 29754], 60.00th=[ 71828], 70.00th=[ 95945],
- | 80.00th=[ 125305], 90.00th=[ 278922], 95.00th=[ 484443],
- | 99.00th=[ 884999], 99.50th=[ 943719], 99.90th=[1350566],
- | 99.95th=[1468007], 99.99th=[1468007]
- bw ( KiB/s): min= 0, max= 3323, per=41.20%, avg=2337.50, stdev=1393.71, samples=2
- iops : min= 0, max= 51, avg=36.00, stdev=21.21, samples=2
- lat (nsec) : 750=7.20%, 1000=11.69%
- lat (usec) : 2=17.30%, 4=0.08%, 10=0.29%, 20=0.59%, 50=0.77%
- lat (usec) : 100=0.80%, 250=0.59%, 500=0.42%, 750=0.65%, 1000=1.42%
- lat (msec) : 2=10.28%, 4=8.01%, 10=6.04%, 20=2.99%, 50=4.37%
- lat (msec) : 100=11.07%, 250=9.46%, 500=3.51%, 750=1.54%, 1000=0.75%
- cpu : usr=0.06%, sys=0.21%, ctx=10176, majf=0, minf=30
- IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=4187,4385,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- Run status group 0 (all jobs):
- READ: bw=23.5MiB/s (24.7MB/s), 2232KiB/s-21.3MiB/s (2286kB/s-22.4MB/s), io=2823MiB (2960MB), run=120020-120028msec
- WRITE: bw=5674KiB/s (5810kB/s), 2340KiB/s-3334KiB/s (2396kB/s-3414kB/s), io=665MiB (697MB), run=120024-120028msec
- Disk stats (read/write):
- rbd234: ios=8956/11204, merge=0/1, ticks=542224/827259, in_queue=1359118, util=14.50%
- <example-benchmark-fio-server-112-benchmark-wpgw6>
- randwrite: (groupid=0, jobs=1): err= 0: pid=11: Wed May 8 20:34:55 2019
- write: IOPS=47, BW=3020KiB/s (3092kB/s)(354MiB/120030msec)
- slat (nsec): min=0, max=434516k, avg=325543.33, stdev=7343542.39
- clat (nsec): min=0, max=953098k, avg=84734749.94, stdev=150303929.00
- lat (nsec): min=0, max=953156k, avg=85061546.03, stdev=150791631.58
- clat percentiles (usec):
- | 1.00th=[ 996], 5.00th=[ 1205], 10.00th=[ 1401], 20.00th=[ 1844],
- | 30.00th=[ 2573], 40.00th=[ 4113], 50.00th=[ 8848], 60.00th=[ 36963],
- | 70.00th=[ 79168], 80.00th=[119014], 90.00th=[295699], 95.00th=[459277],
- | 99.00th=[683672], 99.50th=[725615], 99.90th=[792724], 99.95th=[876610],
- | 99.99th=[952108]
- bw ( KiB/s): min= 0, max= 4542, per=57.71%, avg=3017.50, stdev=2155.97, samples=2
- iops : min= 0, max= 70, avg=46.50, stdev=33.23, samples=2
- lat (usec) : 4=0.02%, 50=0.02%, 250=0.02%, 500=0.04%, 750=0.02%
- lat (usec) : 1000=0.97%
- lat (msec) : 2=21.73%, 4=16.86%, 10=11.47%, 20=4.84%, 50=6.17%
- lat (msec) : 100=13.85%, 250=13.06%, 500=7.26%, 750=3.43%, 1000=0.32%
- cpu : usr=0.05%, sys=0.19%, ctx=10421, majf=0, minf=25
- IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=0,5660,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- read: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:34:55 2019
- read: IOPS=336, BW=21.0MiB/s (22.0MB/s)(2523MiB/120021msec)
- slat (nsec): min=0, max=406062k, avg=39732.49, stdev=2589915.75
- clat (nsec): min=0, max=1114.5M, avg=11880232.54, stdev=59995718.21
- lat (nsec): min=0, max=1114.6M, avg=11920800.50, stdev=60062257.19
- clat percentiles (nsec):
- | 1.00th=[ 708], 5.00th=[ 708], 10.00th=[ 716],
- | 20.00th=[ 724], 30.00th=[ 732], 40.00th=[ 740],
- | 50.00th=[ 796], 60.00th=[ 868], 70.00th=[ 972],
- | 80.00th=[ 1432], 90.00th=[ 3620864], 95.00th=[ 53739520],
- | 99.00th=[362807296], 99.50th=[471859200], 99.90th=[675282944],
- | 99.95th=[742391808], 99.99th=[901775360]
- bw ( KiB/s): min= 0, max=27225, per=91.10%, avg=21528.00, stdev=8056.77, samples=2
- iops : min= 0, max= 425, avg=336.00, stdev=125.87, samples=2
- lat (nsec) : 750=44.70%, 1000=26.30%
- lat (usec) : 2=10.01%, 4=0.04%, 10=0.50%, 20=0.32%, 50=0.16%
- lat (usec) : 100=0.06%, 250=0.06%, 500=0.29%, 750=1.02%, 1000=2.18%
- lat (msec) : 2=2.88%, 4=1.61%, 10=0.76%, 20=0.62%, 50=3.14%
- lat (msec) : 100=2.47%, 250=1.40%, 500=1.09%, 750=0.37%, 1000=0.04%
- cpu : usr=0.11%, sys=0.37%, ctx=14426, majf=0, minf=29
- IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=40393,0,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- readwrite: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:34:55 2019
- read: IOPS=32, BW=2106KiB/s (2156kB/s)(247MiB/120003msec)
- slat (nsec): min=0, max=3284.3k, avg=29352.61, stdev=98817.29
- clat (nsec): min=0, max=785351k, avg=10373089.41, stdev=48731284.43
- lat (nsec): min=0, max=785379k, avg=10403500.78, stdev=48741066.99
- clat percentiles (nsec):
- | 1.00th=[ 708], 5.00th=[ 716], 10.00th=[ 732],
- | 20.00th=[ 772], 30.00th=[ 884], 40.00th=[ 1004],
- | 50.00th=[ 1144], 60.00th=[ 1288], 70.00th=[ 1480],
- | 80.00th=[ 154624], 90.00th=[ 11337728], 95.00th=[ 54788096],
- | 99.00th=[265289728], 99.50th=[387973120], 99.90th=[591396864],
- | 99.95th=[616562688], 99.99th=[784334848]
- bw ( KiB/s): min= 0, max= 3043, per=8.91%, avg=2104.50, stdev=1327.24, samples=2
- iops : min= 0, max= 47, avg=32.50, stdev=20.51, samples=2
- write: IOPS=34, BW=2211KiB/s (2264kB/s)(259MiB/120003msec)
- slat (nsec): min=0, max=392423k, avg=356288.72, stdev=8053940.39
- clat (nsec): min=0, max=1000.3M, avg=105913790.38, stdev=167085986.59
- lat (nsec): min=0, max=1000.4M, avg=106271139.65, stdev=167201360.65
- clat percentiles (usec):
- | 1.00th=[ 1037], 5.00th=[ 1237], 10.00th=[ 1467],
- | 20.00th=[ 2114], 30.00th=[ 3392], 40.00th=[ 6980],
- | 50.00th=[ 27395], 60.00th=[ 68682], 70.00th=[ 99091],
- | 80.00th=[ 147850], 90.00th=[ 371196], 95.00th=[ 497026],
- | 99.00th=[ 717226], 99.50th=[ 750781], 99.90th=[ 876610],
- | 99.95th=[ 876610], 99.99th=[1002439]
- bw ( KiB/s): min= 0, max= 3193, per=42.24%, avg=2208.50, stdev=1392.29, samples=2
- iops : min= 0, max= 49, avg=34.00, stdev=21.21, samples=2
- lat (nsec) : 750=8.26%, 1000=11.10%
- lat (usec) : 2=16.60%, 4=0.10%, 10=0.26%, 20=0.94%, 50=0.78%
- lat (usec) : 100=0.75%, 250=0.61%, 500=0.54%, 750=0.48%, 1000=1.27%
- lat (msec) : 2=10.62%, 4=7.74%, 10=6.02%, 20=2.82%, 50=5.01%
- lat (msec) : 100=9.83%, 250=8.37%, 500=5.39%, 750=2.29%, 1000=0.27%
- cpu : usr=0.05%, sys=0.19%, ctx=9532, majf=0, minf=30
- IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=3949,4142,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- Run status group 0 (all jobs):
- READ: bw=23.1MiB/s (24.2MB/s), 2106KiB/s-21.0MiB/s (2156kB/s-22.0MB/s), io=2770MiB (2904MB), run=120003-120021msec
- WRITE: bw=5230KiB/s (5355kB/s), 2211KiB/s-3020KiB/s (2264kB/s-3092kB/s), io=613MiB (643MB), run=120003-120030msec
- Disk stats (read/write):
- rbd37: ios=8286/10343, merge=0/3, ticks=542121/837826, in_queue=1370444, util=13.14%
- <example-benchmark-fio-server-5-benchmark-hchc9>
- randwrite: (groupid=0, jobs=1): err= 0: pid=11: Wed May 8 20:34:55 2019
- write: IOPS=43, BW=2784KiB/s (2851kB/s)(326MiB/120043msec)
- slat (nsec): min=0, max=1469.3M, avg=542823.26, stdev=21352335.09
- clat (nsec): min=0, max=1719.5M, avg=91545953.17, stdev=150529149.13
- lat (nsec): min=0, max=2232.0M, avg=92089805.76, stdev=153224563.06
- clat percentiles (usec):
- | 1.00th=[ 988], 5.00th=[ 1237], 10.00th=[ 1418],
- | 20.00th=[ 1958], 30.00th=[ 3228], 40.00th=[ 6980],
- | 50.00th=[ 42730], 60.00th=[ 78119], 70.00th=[ 102237],
- | 80.00th=[ 127402], 90.00th=[ 219153], 95.00th=[ 408945],
- | 99.00th=[ 725615], 99.50th=[ 834667], 99.90th=[1350566],
- | 99.95th=[1468007], 99.99th=[1719665]
- bw ( KiB/s): min= 0, max= 3203, per=54.41%, avg=2782.50, stdev=594.68, samples=2
- iops : min= 0, max= 50, avg=43.00, stdev= 9.90, samples=2
- lat (usec) : 4=0.02%, 750=0.11%, 1000=1.03%
- lat (msec) : 2=19.37%, 4=12.93%, 10=9.37%, 20=3.37%, 50=5.31%
- lat (msec) : 100=17.24%, 250=22.05%, 500=6.05%, 750=2.28%, 1000=0.73%
- cpu : usr=0.05%, sys=0.18%, ctx=9915, majf=0, minf=30
- IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=0,5219,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- read: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:34:55 2019
- read: IOPS=348, BW=21.8MiB/s (22.9MB/s)(2615MiB/120001msec)
- slat (nsec): min=0, max=1466.4M, avg=67849.19, stdev=7423963.40
- clat (nsec): min=0, max=1973.0M, avg=11410932.15, stdev=57537264.39
- lat (nsec): min=0, max=1974.0M, avg=11479646.23, stdev=58075993.63
- clat percentiles (nsec):
- | 1.00th=[ 708], 5.00th=[ 708], 10.00th=[ 716],
- | 20.00th=[ 724], 30.00th=[ 740], 40.00th=[ 780],
- | 50.00th=[ 852], 60.00th=[ 932], 70.00th=[ 1192],
- | 80.00th=[ 839680], 90.00th=[ 14352384], 95.00th=[ 54788096],
- | 99.00th=[ 304087040], 99.50th=[ 438304768], 99.90th=[ 683671552],
- | 99.95th=[ 792723456], 99.99th=[1216348160]
- bw ( KiB/s): min= 0, max=31764, per=90.94%, avg=22314.00, stdev=13364.32, samples=2
- iops : min= 0, max= 496, avg=348.50, stdev=208.60, samples=2
- lat (nsec) : 750=35.07%, 1000=28.45%
- lat (usec) : 2=12.71%, 4=0.05%, 10=0.53%, 20=0.29%, 50=0.14%
- lat (usec) : 100=0.06%, 250=0.07%, 500=0.41%, 750=1.13%, 1000=2.65%
- lat (msec) : 2=3.94%, 4=2.52%, 10=1.49%, 20=1.02%, 50=3.88%
- lat (msec) : 100=3.09%, 250=1.23%, 500=0.96%, 750=0.25%, 1000=0.05%
- cpu : usr=0.12%, sys=0.42%, ctx=18819, majf=0, minf=27
- IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=41856,0,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- readwrite: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:34:55 2019
- read: IOPS=34, BW=2226KiB/s (2280kB/s)(261MiB/120024msec)
- slat (nsec): min=0, max=1469.5M, avg=485436.54, stdev=23708009.49
- clat (nsec): min=0, max=1068.9M, avg=10575940.19, stdev=49916812.10
- lat (nsec): min=0, max=1470.0M, avg=11062295.90, stdev=55220809.67
- clat percentiles (nsec):
- | 1.00th=[ 708], 5.00th=[ 716], 10.00th=[ 732],
- | 20.00th=[ 780], 30.00th=[ 884], 40.00th=[ 1012],
- | 50.00th=[ 1160], 60.00th=[ 1272], 70.00th=[ 1480],
- | 80.00th=[ 193536], 90.00th=[ 21102592], 95.00th=[ 57933824],
- | 99.00th=[ 235929600], 99.50th=[ 362807296], 99.90th=[ 616562688],
- | 99.95th=[ 734003200], 99.99th=[1069547520]
- bw ( KiB/s): min= 0, max= 2374, per=9.07%, avg=2226.00, stdev=209.30, samples=2
- iops : min= 0, max= 37, avg=34.50, stdev= 3.54, samples=2
- write: IOPS=36, BW=2331KiB/s (2387kB/s)(273MiB/120024msec)
- slat (nsec): min=0, max=6481.8k, avg=159585.38, stdev=274880.69
- clat (nsec): min=0, max=1784.7M, avg=99395294.56, stdev=159205507.24
- lat (nsec): min=0, max=1784.7M, avg=99556016.47, stdev=159164612.17
- clat percentiles (usec):
- | 1.00th=[ 1037], 5.00th=[ 1287], 10.00th=[ 1549],
- | 20.00th=[ 2343], 30.00th=[ 4228], 40.00th=[ 10683],
- | 50.00th=[ 54264], 60.00th=[ 81265], 70.00th=[ 106431],
- | 80.00th=[ 131597], 90.00th=[ 270533], 95.00th=[ 442500],
- | 99.00th=[ 784335], 99.50th=[ 918553], 99.90th=[1384121],
- | 99.95th=[1468007], 99.99th=[1786774]
- bw ( KiB/s): min= 0, max= 2476, per=45.53%, avg=2328.50, stdev=208.60, samples=2
- iops : min= 0, max= 38, avg=36.00, stdev= 2.83, samples=2
- lat (nsec) : 750=7.96%, 1000=11.17%
- lat (usec) : 2=16.96%, 4=0.06%, 10=0.27%, 20=0.83%, 50=0.71%
- lat (usec) : 100=0.70%, 250=0.57%, 500=0.54%, 750=0.55%, 1000=1.05%
- lat (msec) : 2=9.34%, 4=7.48%, 10=5.65%, 20=1.94%, 50=5.16%
- lat (msec) : 100=11.00%, 250=12.30%, 500=3.83%, 750=1.32%, 1000=0.49%
- cpu : usr=0.04%, sys=0.22%, ctx=10113, majf=0, minf=34
- IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=4175,4368,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- Run status group 0 (all jobs):
- READ: bw=23.0MiB/s (25.1MB/s), 2226KiB/s-21.8MiB/s (2280kB/s-22.9MB/s), io=2876MiB (3016MB), run=120001-120024msec
- WRITE: bw=5114KiB/s (5237kB/s), 2331KiB/s-2784KiB/s (2387kB/s-2851kB/s), io=600MiB (629MB), run=120024-120043msec
- Disk stats (read/write):
- rbd2: ios=11019/10797, merge=0/1, ticks=541896/812590, in_queue=1343285, util=15.83%
- <example-benchmark-fio-server-752-benchmark-kfgvn>
- randwrite: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:34:55 2019
- write: IOPS=38, BW=2456KiB/s (2515kB/s)(288MiB/120075msec)
- slat (nsec): min=0, max=486355k, avg=347151.01, stdev=8370452.54
- clat (nsec): min=0, max=1379.9M, avg=104506667.37, stdev=171039013.89
- lat (nsec): min=0, max=1379.0M, avg=104854912.10, stdev=171182329.65
- clat percentiles (usec):
- | 1.00th=[ 988], 5.00th=[ 1188], 10.00th=[ 1385],
- | 20.00th=[ 1860], 30.00th=[ 2835], 40.00th=[ 5735],
- | 50.00th=[ 19006], 60.00th=[ 63701], 70.00th=[ 99091],
- | 80.00th=[ 154141], 90.00th=[ 346031], 95.00th=[ 501220],
- | 99.00th=[ 742392], 99.50th=[ 801113], 99.90th=[1132463],
- | 99.95th=[1300235], 99.99th=[1384121]
- bw ( KiB/s): min= 0, max= 3208, per=55.27%, avg=2455.00, stdev=1064.90, samples=2
- iops : min= 0, max= 50, avg=38.00, stdev=16.97, samples=2
- lat (usec) : 500=0.07%, 750=0.02%, 1000=1.13%
- lat (msec) : 2=21.35%, 4=13.07%, 10=9.29%, 20=5.52%, 50=5.91%
- lat (msec) : 100=13.77%, 250=15.44%, 500=9.38%, 750=4.13%, 1000=0.83%
- cpu : usr=0.05%, sys=0.16%, ctx=8673, majf=0, minf=30
- IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=0,4605,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- read: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:34:55 2019
- read: IOPS=336, BW=21.0MiB/s (22.0MB/s)(2521MiB/120024msec)
- slat (nsec): min=0, max=476699k, avg=39713.39, stdev=2703973.90
- clat (nsec): min=0, max=1301.7M, avg=11885616.09, stdev=61578485.33
- lat (nsec): min=0, max=1301.7M, avg=11926172.49, stdev=61648490.33
- clat percentiles (nsec):
- | 1.00th=[ 708], 5.00th=[ 708], 10.00th=[ 716],
- | 20.00th=[ 724], 30.00th=[ 732], 40.00th=[ 764],
- | 50.00th=[ 828], 60.00th=[ 876], 70.00th=[ 996],
- | 80.00th=[ 1432], 90.00th=[ 2670592], 95.00th=[ 53215232],
- | 99.00th=[ 371195904], 99.50th=[ 476053504], 99.90th=[ 675282944],
- | 99.95th=[ 742391808], 99.99th=[1019215872]
- bw ( KiB/s): min= 0, max=25368, per=91.91%, avg=21512.50, stdev=5452.50, samples=2
- iops : min= 0, max= 396, avg=335.50, stdev=85.56, samples=2
- lat (nsec) : 750=36.50%, 1000=33.72%
- lat (usec) : 2=11.17%, 4=0.10%, 10=0.82%, 20=0.63%, 50=0.14%
- lat (usec) : 100=0.07%, 250=0.05%, 500=0.21%, 750=0.93%, 1000=1.97%
- lat (msec) : 2=2.80%, 4=1.64%, 10=0.82%, 20=0.49%, 50=2.68%
- lat (msec) : 100=2.35%, 250=1.38%, 500=1.12%, 750=0.36%, 1000=0.04%
- cpu : usr=0.11%, sys=0.37%, ctx=13493, majf=0, minf=30
- IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=40363,0,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- readwrite: (groupid=0, jobs=1): err= 0: pid=14: Wed May 8 20:34:55 2019
- read: IOPS=29, BW=1897KiB/s (1943kB/s)(222MiB/120034msec)
- slat (nsec): min=0, max=394240k, avg=221723.67, stdev=8238630.85
- clat (nsec): min=0, max=533183k, avg=8440051.27, stdev=43256219.94
- lat (nsec): min=0, max=533276k, avg=8662693.17, stdev=44003117.30
- clat percentiles (nsec):
- | 1.00th=[ 708], 5.00th=[ 724], 10.00th=[ 732],
- | 20.00th=[ 772], 30.00th=[ 860], 40.00th=[ 956],
- | 50.00th=[ 1096], 60.00th=[ 1240], 70.00th=[ 1432],
- | 80.00th=[ 53504], 90.00th=[ 1630208], 95.00th=[ 42205184],
- | 99.00th=[250609664], 99.50th=[346030080], 99.90th=[488636416],
- | 99.95th=[530579456], 99.99th=[534773760]
- bw ( KiB/s): min= 0, max= 2520, per=8.10%, avg=1896.50, stdev=881.76, samples=2
- iops : min= 0, max= 39, avg=29.00, stdev=14.14, samples=2
- write: IOPS=31, BW=1987KiB/s (2035kB/s)(233MiB/120034msec)
- slat (nsec): min=0, max=4499.2k, avg=170586.55, stdev=218293.04
- clat (nsec): min=0, max=1384.8M, avg=121032677.53, stdev=182906307.19
- lat (nsec): min=0, max=1384.9M, avg=121204248.52, stdev=182846462.11
- clat percentiles (usec):
- | 1.00th=[ 1012], 5.00th=[ 1254], 10.00th=[ 1483],
- | 20.00th=[ 2089], 30.00th=[ 3621], 40.00th=[ 9765],
- | 50.00th=[ 45351], 60.00th=[ 81265], 70.00th=[ 112722],
- | 80.00th=[ 206570], 90.00th=[ 392168], 95.00th=[ 541066],
- | 99.00th=[ 767558], 99.50th=[ 901776], 99.90th=[1052771],
- | 99.95th=[1300235], 99.99th=[1384121]
- bw ( KiB/s): min= 0, max= 2640, per=44.71%, avg=1986.00, stdev=924.90, samples=2
- iops : min= 0, max= 41, avg=30.50, stdev=14.85, samples=2
- lat (nsec) : 750=8.23%, 1000=12.94%
- lat (usec) : 2=15.82%, 4=0.07%, 10=0.12%, 20=0.73%, 50=1.07%
- lat (usec) : 100=1.07%, 250=0.65%, 500=0.91%, 750=0.58%, 1000=1.33%
- lat (msec) : 2=10.42%, 4=6.85%, 10=4.79%, 20=2.93%, 50=4.44%
- lat (msec) : 100=8.27%, 250=9.74%, 500=5.82%, 750=2.66%, 1000=0.52%
- cpu : usr=0.06%, sys=0.16%, ctx=8298, majf=0, minf=30
- IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=3559,3723,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- Run status group 0 (all jobs):
- READ: bw=22.9MiB/s (23.0MB/s), 1897KiB/s-21.0MiB/s (1943kB/s-22.0MB/s), io=2744MiB (2877MB), run=120024-120034msec
- WRITE: bw=4443KiB/s (4549kB/s), 1987KiB/s-2456KiB/s (2035kB/s-2515kB/s), io=521MiB (546MB), run=120034-120075msec
- Disk stats (read/write):
- rbd272: ios=7654/8582, merge=0/1, ticks=537074/870601, in_queue=1399336, util=11.51%
- <example-benchmark-fio-server-427-benchmark-hzqrn>
- randwrite: (groupid=0, jobs=1): err= 0: pid=11: Wed May 8 20:34:55 2019
- write: IOPS=55, BW=3533KiB/s (3617kB/s)(414MiB/120097msec)
- slat (nsec): min=0, max=456194k, avg=355259.52, stdev=8398375.70
- clat (nsec): min=0, max=991198k, avg=72238614.86, stdev=141200455.04
- lat (nsec): min=0, max=991288k, avg=72595361.15, stdev=141328353.19
- clat percentiles (usec):
- | 1.00th=[ 1004], 5.00th=[ 1221], 10.00th=[ 1401], 20.00th=[ 1795],
- | 30.00th=[ 2409], 40.00th=[ 3654], 50.00th=[ 6521], 60.00th=[ 16450],
- | 70.00th=[ 69731], 80.00th=[104334], 90.00th=[204473], 95.00th=[392168],
- | 99.00th=[708838], 99.50th=[792724], 99.90th=[926942], 99.95th=[952108],
- | 99.99th=[994051]
- bw ( KiB/s): min= 0, max= 4756, per=56.38%, avg=3533.00, stdev=1729.58, samples=2
- iops : min= 0, max= 74, avg=55.00, stdev=26.87, samples=2
- lat (usec) : 500=0.02%, 750=0.03%, 1000=0.92%
- lat (msec) : 2=23.29%, 4=17.60%, 10=13.67%, 20=5.77%, 50=4.87%
- lat (msec) : 100=12.65%, 250=12.93%, 500=4.95%, 750=2.69%, 1000=0.66%
- cpu : usr=0.08%, sys=0.23%, ctx=12600, majf=0, minf=30
- IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=0,6626,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- read: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:34:55 2019
- read: IOPS=336, BW=21.0MiB/s (22.1MB/s)(2526MiB/120015msec)
- slat (nsec): min=0, max=362838k, avg=45577.70, stdev=2841029.73
- clat (nsec): min=0, max=1029.8M, avg=11837195.23, stdev=59499909.85
- lat (nsec): min=0, max=1029.9M, avg=11883621.69, stdev=59574212.09
- clat percentiles (nsec):
- | 1.00th=[ 708], 5.00th=[ 708], 10.00th=[ 716],
- | 20.00th=[ 724], 30.00th=[ 732], 40.00th=[ 748],
- | 50.00th=[ 836], 60.00th=[ 892], 70.00th=[ 1080],
- | 80.00th=[ 60672], 90.00th=[ 9240576], 95.00th=[ 54788096],
- | 99.00th=[346030080], 99.50th=[476053504], 99.90th=[692060160],
- | 99.95th=[750780416], 99.99th=[943718400]
- bw ( KiB/s): min= 0, max=28561, per=89.07%, avg=21554.00, stdev=9909.39, samples=2
- iops : min= 0, max= 446, avg=336.50, stdev=154.86, samples=2
- lat (nsec) : 750=39.64%, 1000=28.22%
- lat (usec) : 2=10.44%, 4=0.04%, 10=0.88%, 20=0.50%, 50=0.24%
- lat (usec) : 100=0.10%, 250=0.06%, 500=0.15%, 750=0.98%, 1000=2.14%
- lat (msec) : 2=3.42%, 4=2.12%, 10=1.14%, 20=0.74%, 50=3.59%
- lat (msec) : 100=2.99%, 250=1.23%, 500=0.95%, 750=0.37%, 1000=0.05%
- cpu : usr=0.12%, sys=0.38%, ctx=16094, majf=0, minf=102
- IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=40443,0,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- readwrite: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:34:55 2019
- read: IOPS=41, BW=2654KiB/s (2718kB/s)(311MiB/120055msec)
- slat (nsec): min=0, max=238280k, avg=92609.11, stdev=3549619.81
- clat (nsec): min=0, max=774600k, avg=8859266.13, stdev=41684463.38
- lat (nsec): min=0, max=774650k, avg=8952847.07, stdev=41829723.60
- clat percentiles (nsec):
- | 1.00th=[ 716], 5.00th=[ 732], 10.00th=[ 764],
- | 20.00th=[ 844], 30.00th=[ 924], 40.00th=[ 1048],
- | 50.00th=[ 1192], 60.00th=[ 1320], 70.00th=[ 1608],
- | 80.00th=[ 370688], 90.00th=[ 9895936], 95.00th=[ 52690944],
- | 99.00th=[191889408], 99.50th=[312475648], 99.90th=[476053504],
- | 99.95th=[658505728], 99.99th=[775946240]
- bw ( KiB/s): min= 0, max= 3376, per=10.97%, avg=2655.00, stdev=1019.65, samples=2
- iops : min= 0, max= 52, avg=41.00, stdev=15.56, samples=2
- write: IOPS=42, BW=2735KiB/s (2801kB/s)(321MiB/120055msec)
- slat (nsec): min=0, max=634946k, avg=345574.69, stdev=9626503.84
- clat (nsec): min=0, max=1219.3M, avg=84920091.81, stdev=152543276.10
- lat (nsec): min=0, max=1219.3M, avg=85266747.75, stdev=152708991.55
- clat percentiles (usec):
- | 1.00th=[ 1012], 5.00th=[ 1237], 10.00th=[ 1434],
- | 20.00th=[ 1942], 30.00th=[ 2933], 40.00th=[ 4817],
- | 50.00th=[ 10290], 60.00th=[ 45876], 70.00th=[ 84411],
- | 80.00th=[ 115868], 90.00th=[ 254804], 95.00th=[ 467665],
- | 99.00th=[ 725615], 99.50th=[ 801113], 99.90th=[ 952108],
- | 99.95th=[ 985662], 99.99th=[1216349]
- bw ( KiB/s): min= 0, max= 3446, per=43.63%, avg=2734.00, stdev=1006.92, samples=2
- iops : min= 0, max= 53, avg=42.00, stdev=15.56, samples=2
- lat (nsec) : 750=4.32%, 1000=13.77%
- lat (usec) : 2=17.31%, 4=0.07%, 10=0.30%, 20=0.92%, 50=0.85%
- lat (usec) : 100=0.87%, 250=0.72%, 500=0.62%, 750=0.53%, 1000=1.40%
- lat (msec) : 2=11.97%, 4=8.73%, 10=7.17%, 20=3.20%, 50=4.70%
- lat (msec) : 100=9.37%, 250=7.66%, 500=3.41%, 750=1.75%, 1000=0.36%
- cpu : usr=0.07%, sys=0.25%, ctx=11962, majf=0, minf=31
- IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=4980,5128,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- Run status group 0 (all jobs):
- READ: bw=23.6MiB/s (24.8MB/s), 2654KiB/s-21.0MiB/s (2718kB/s-22.1MB/s), io=2837MiB (2975MB), run=120015-120055msec
- WRITE: bw=6267KiB/s (6417kB/s), 2735KiB/s-3533KiB/s (2801kB/s-3617kB/s), io=735MiB (771MB), run=120055-120097msec
- Disk stats (read/write):
- rbd131: ios=9638/11955, merge=0/1, ticks=544140/818314, in_queue=1351514, util=15.67%
- <example-benchmark-fio-server-731-benchmark-2p24w>
- randwrite: (groupid=0, jobs=1): err= 0: pid=11: Wed May 8 20:34:55 2019
- write: IOPS=46, BW=2954KiB/s (3025kB/s)(346MiB/120061msec)
- slat (nsec): min=0, max=750892k, avg=445086.93, stdev=11721042.12
- clat (nsec): min=0, max=1628.2M, avg=86222905.58, stdev=150130186.58
- lat (nsec): min=0, max=1628.3M, avg=86669252.57, stdev=150470281.13
- clat percentiles (usec):
- | 1.00th=[ 963], 5.00th=[ 1221], 10.00th=[ 1500],
- | 20.00th=[ 2147], 30.00th=[ 3228], 40.00th=[ 5604],
- | 50.00th=[ 13173], 60.00th=[ 55837], 70.00th=[ 93848],
- | 80.00th=[ 123208], 90.00th=[ 240124], 95.00th=[ 421528],
- | 99.00th=[ 708838], 99.50th=[ 801113], 99.90th=[1098908],
- | 99.95th=[1115685], 99.99th=[1635779]
- bw ( KiB/s): min= 0, max= 4555, per=52.95%, avg=2953.00, stdev=2265.57, samples=2
- iops : min= 0, max= 71, avg=46.00, stdev=35.36, samples=2
- lat (usec) : 100=0.02%, 250=0.04%, 500=0.04%, 750=0.07%, 1000=1.39%
- lat (msec) : 2=16.67%, 4=16.38%, 10=13.04%, 20=4.75%, 50=6.41%
- lat (msec) : 100=13.20%, 250=18.62%, 500=5.99%, 750=2.67%, 1000=0.58%
- cpu : usr=0.05%, sys=0.20%, ctx=10208, majf=0, minf=30
- IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=0,5538,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- read: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:34:55 2019
- read: IOPS=529, BW=33.0MiB/s (34.7MB/s)(3966MiB/120019msec)
- slat (nsec): min=0, max=482645k, avg=34306.21, stdev=2614937.41
- clat (nsec): min=0, max=1336.0M, avg=7539479.66, stdev=48759461.40
- lat (nsec): min=0, max=1336.0M, avg=7574591.66, stdev=48839095.66
- clat percentiles (nsec):
- | 1.00th=[ 708], 5.00th=[ 716], 10.00th=[ 716],
- | 20.00th=[ 724], 30.00th=[ 732], 40.00th=[ 740],
- | 50.00th=[ 804], 60.00th=[ 868], 70.00th=[ 964],
- | 80.00th=[ 8640], 90.00th=[ 1089536], 95.00th=[ 32636928],
- | 99.00th=[ 217055232], 99.50th=[ 404750336], 99.90th=[ 633339904],
- | 99.95th=[ 742391808], 99.99th=[1149239296]
- bw ( KiB/s): min= 0, max=46913, per=93.04%, avg=33844.00, stdev=18482.36, samples=2
- iops : min= 0, max= 735, avg=529.50, stdev=290.62, samples=2
- lat (nsec) : 750=44.39%, 1000=26.82%
- lat (usec) : 2=7.58%, 4=0.06%, 10=4.95%, 20=2.19%, 50=0.69%
- lat (usec) : 100=0.15%, 250=0.06%, 500=0.17%, 750=0.85%, 1000=1.70%
- lat (msec) : 2=2.34%, 4=1.32%, 10=0.59%, 20=0.39%, 50=2.28%
- lat (msec) : 100=1.84%, 250=0.73%, 500=0.67%, 750=0.20%, 1000=0.03%
- cpu : usr=0.16%, sys=0.53%, ctx=16677, majf=0, minf=30
- IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=63607,0,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- readwrite: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:34:55 2019
- read: IOPS=39, BW=2534KiB/s (2595kB/s)(297MiB/120003msec)
- slat (nsec): min=0, max=2602.2k, avg=26859.27, stdev=68750.48
- clat (nsec): min=0, max=750910k, avg=6636280.46, stdev=36858791.12
- lat (nsec): min=0, max=750930k, avg=6664125.19, stdev=36867160.13
- clat percentiles (nsec):
- | 1.00th=[ 716], 5.00th=[ 732], 10.00th=[ 748],
- | 20.00th=[ 804], 30.00th=[ 884], 40.00th=[ 980],
- | 50.00th=[ 1128], 60.00th=[ 1272], 70.00th=[ 1592],
- | 80.00th=[ 84480], 90.00th=[ 1613824], 95.00th=[ 38535168],
- | 99.00th=[160432128], 99.50th=[295698432], 99.90th=[480247808],
- | 99.95th=[488636416], 99.99th=[750780416]
- bw ( KiB/s): min= 0, max= 3603, per=6.97%, avg=2534.00, stdev=1511.79, samples=2
- iops : min= 0, max= 56, avg=39.00, stdev=24.04, samples=2
- write: IOPS=40, BW=2625KiB/s (2688kB/s)(308MiB/120003msec)
- slat (nsec): min=0, max=750900k, avg=473560.27, stdev=12692496.43
- clat (nsec): min=0, max=1632.7M, avg=90831011.95, stdev=157168249.28
- lat (nsec): min=0, max=1632.8M, avg=91305477.41, stdev=157556825.13
- clat percentiles (usec):
- | 1.00th=[ 1004], 5.00th=[ 1237], 10.00th=[ 1467],
- | 20.00th=[ 2147], 30.00th=[ 3326], 40.00th=[ 5866],
- | 50.00th=[ 15795], 60.00th=[ 62129], 70.00th=[ 96994],
- | 80.00th=[ 127402], 90.00th=[ 256902], 95.00th=[ 442500],
- | 99.00th=[ 742392], 99.50th=[ 868221], 99.90th=[1233126],
- | 99.95th=[1283458], 99.99th=[1635779]
- bw ( KiB/s): min= 0, max= 3730, per=47.02%, avg=2622.50, stdev=1566.24, samples=2
- iops : min= 0, max= 58, avg=40.50, stdev=24.75, samples=2
- lat (nsec) : 750=5.76%, 1000=14.44%
- lat (usec) : 2=15.05%, 4=0.09%, 10=0.52%, 20=1.30%, 50=1.23%
- lat (usec) : 100=1.25%, 250=1.03%, 500=1.04%, 750=0.69%, 1000=1.35%
- lat (msec) : 2=10.10%, 4=8.67%, 10=6.74%, 20=2.49%, 50=4.65%
- lat (msec) : 100=8.20%, 250=9.94%, 500=3.54%, 750=1.44%, 1000=0.35%
- cpu : usr=0.06%, sys=0.23%, ctx=11015, majf=0, minf=31
- IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=4755,4919,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- Run status group 0 (all jobs):
- READ: bw=35.5MiB/s (37.2MB/s), 2534KiB/s-33.0MiB/s (2595kB/s-34.7MB/s), io=4263MiB (4470MB), run=120003-120019msec
- WRITE: bw=5577KiB/s (5711kB/s), 2625KiB/s-2954KiB/s (2688kB/s-3025kB/s), io=654MiB (686MB), run=120003-120061msec
- Disk stats (read/write):
- rbd178: ios=9145/10566, merge=0/3, ticks=516068/791944, in_queue=1297922, util=13.11%
- Jobs: 51 (f=51): [_(2),m(1)][99.8%][r=1375M,w=1387M][r=21.5k,w=21.7k IOPS][eta 00m:54s] Jobs: 51 (f=51): [_(2),m(1)][99.8%][r=1410M,w=1427M][r=22.0k,w=22.3k IOPS][eta 00m:53s] <example-benchmark-fio-server-794-benchmark-xkzrq>
- randwrite: (groupid=0, jobs=1): err= 0: pid=11: Wed May 8 20:34:57 2019
- write: IOPS=39, BW=2511KiB/s (2572kB/s)(294MiB/120011msec)
- slat (nsec): min=0, max=830857k, avg=575567.16, stdev=16158635.08
- clat (nsec): min=0, max=1469.7M, avg=101665439.36, stdev=163496335.02
- lat (nsec): min=0, max=1469.8M, avg=102242584.08, stdev=164095874.59
- clat percentiles (usec):
- | 1.00th=[ 1020], 5.00th=[ 1270], 10.00th=[ 1598],
- | 20.00th=[ 2245], 30.00th=[ 3720], 40.00th=[ 9503],
- | 50.00th=[ 42730], 60.00th=[ 72877], 70.00th=[ 102237],
- | 80.00th=[ 143655], 90.00th=[ 291505], 95.00th=[ 438305],
- | 99.00th=[ 801113], 99.50th=[ 918553], 99.90th=[1434452],
- | 99.95th=[1468007], 99.99th=[1468007]
- bw ( KiB/s): min= 0, max= 3129, per=48.79%, avg=2509.00, stdev=876.81, samples=2
- iops : min= 0, max= 48, avg=38.50, stdev=13.44, samples=2
- lat (usec) : 2=0.02%, 4=0.02%, 500=0.02%, 750=0.04%, 1000=0.70%
- lat (msec) : 2=15.68%, 4=14.53%, 10=9.31%, 20=4.12%, 50=7.39%
- lat (msec) : 100=17.68%, 250=18.25%, 500=8.35%, 750=2.57%, 1000=1.13%
- cpu : usr=0.06%, sys=0.17%, ctx=8853, majf=0, minf=26
- IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=0,4706,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- read: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:34:57 2019
- read: IOPS=321, BW=20.1MiB/s (21.1MB/s)(2412MiB/120011msec)
- slat (nsec): min=0, max=593474k, avg=62460.39, stdev=4735684.38
- clat (nsec): min=0, max=1023.2M, avg=12403573.49, stdev=60386500.08
- lat (nsec): min=0, max=1023.3M, avg=12466969.14, stdev=60576303.42
- clat percentiles (nsec):
- | 1.00th=[ 708], 5.00th=[ 716], 10.00th=[ 716],
- | 20.00th=[ 724], 30.00th=[ 732], 40.00th=[ 740],
- | 50.00th=[ 812], 60.00th=[ 876], 70.00th=[ 996],
- | 80.00th=[ 1528], 90.00th=[ 11075584], 95.00th=[ 57409536],
- | 99.00th=[346030080], 99.50th=[467664896], 99.90th=[700448768],
- | 99.95th=[775946240], 99.99th=[952107008]
- bw ( KiB/s): min= 0, max=27268, per=89.00%, avg=20583.50, stdev=9453.31, samples=2
- iops : min= 0, max= 426, avg=321.50, stdev=147.79, samples=2
- lat (nsec) : 750=42.48%, 1000=27.64%
- lat (usec) : 2=10.36%, 4=0.04%, 10=0.65%, 20=0.38%, 50=0.32%
- lat (usec) : 100=0.13%, 250=0.04%, 500=0.20%, 750=0.85%, 1000=1.83%
- lat (msec) : 2=2.50%, 4=1.57%, 10=0.91%, 20=0.91%, 50=3.52%
- lat (msec) : 100=2.58%, 250=1.53%, 500=1.16%, 750=0.33%, 1000=0.06%
- cpu : usr=0.11%, sys=0.36%, ctx=13842, majf=0, minf=109
- IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=38631,0,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- readwrite: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:34:57 2019
- read: IOPS=39, BW=2544KiB/s (2605kB/s)(298MiB/120003msec)
- slat (nsec): min=0, max=578056k, avg=151252.96, stdev=8368754.74
- clat (nsec): min=0, max=1230.4M, avg=8238059.24, stdev=45356635.31
- lat (nsec): min=0, max=1230.4M, avg=8390234.99, stdev=46106318.45
- clat percentiles (nsec):
- | 1.00th=[ 716], 5.00th=[ 732], 10.00th=[ 740],
- | 20.00th=[ 780], 30.00th=[ 876], 40.00th=[ 988],
- | 50.00th=[ 1144], 60.00th=[ 1272], 70.00th=[ 1560],
- | 80.00th=[ 232448], 90.00th=[ 3686400], 95.00th=[ 41156608],
- | 99.00th=[ 221249536], 99.50th=[ 354418688], 99.90th=[ 480247808],
- | 99.95th=[ 801112064], 99.99th=[1233125376]
- bw ( KiB/s): min= 0, max= 3363, per=11.00%, avg=2543.00, stdev=1159.66, samples=2
- iops : min= 0, max= 52, avg=39.00, stdev=18.38, samples=2
- write: IOPS=41, BW=2632KiB/s (2695kB/s)(308MiB/120003msec)
- slat (nsec): min=0, max=451778k, avg=272568.46, stdev=6436634.86
- clat (nsec): min=0, max=1469.7M, avg=88949844.53, stdev=152195506.47
- lat (nsec): min=0, max=1469.8M, avg=89223694.54, stdev=152274142.94
- clat percentiles (usec):
- | 1.00th=[ 1045], 5.00th=[ 1401], 10.00th=[ 1729],
- | 20.00th=[ 2540], 30.00th=[ 4015], 40.00th=[ 7308],
- | 50.00th=[ 19792], 60.00th=[ 54789], 70.00th=[ 87557],
- | 80.00th=[ 124257], 90.00th=[ 265290], 95.00th=[ 421528],
- | 99.00th=[ 742392], 99.50th=[ 876610], 99.90th=[ 952108],
- | 99.95th=[1434452], 99.99th=[1468007]
- bw ( KiB/s): min= 0, max= 3460, per=51.16%, avg=2630.50, stdev=1173.09, samples=2
- iops : min= 0, max= 54, avg=41.00, stdev=18.38, samples=2
- lat (nsec) : 750=6.93%, 1000=13.11%
- lat (usec) : 2=15.45%, 4=0.07%, 10=0.41%, 20=0.79%, 50=0.92%
- lat (usec) : 100=0.97%, 250=0.75%, 500=0.82%, 750=0.54%, 1000=1.22%
- lat (msec) : 2=8.42%, 4=9.14%, 10=7.65%, 20=3.96%, 50=5.69%
- lat (msec) : 100=9.08%, 250=8.22%, 500=4.08%, 750=1.28%, 1000=0.48%
- cpu : usr=0.07%, sys=0.23%, ctx=11104, majf=0, minf=30
- IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=4771,4932,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- Run status group 0 (all jobs):
- READ: bw=22.6MiB/s (23.7MB/s), 2544KiB/s-20.1MiB/s (2605kB/s-21.1MB/s), io=2711MiB (2842MB), run=120003-120011msec
- WRITE: bw=5143KiB/s (5266kB/s), 2511KiB/s-2632KiB/s (2572kB/s-2695kB/s), io=603MiB (632MB), run=120003-120011msec
- Disk stats (read/write):
- rbd206: ios=9196/10145, merge=0/2, ticks=549346/854017, in_queue=1393497, util=13.88%
- <example-benchmark-fio-server-694-benchmark-wvfzs>
- randwrite: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:34:57 2019
- write: IOPS=40, BW=2571KiB/s (2633kB/s)(307MiB/122324msec)
- slat (nsec): min=0, max=25427k, avg=195885.07, stdev=562273.47
- clat (nsec): min=0, max=8959.5M, avg=99109794.52, stdev=322776851.58
- lat (nsec): min=0, max=8959.6M, avg=99307113.25, stdev=322783856.59
- clat percentiles (usec):
- | 1.00th=[ 848], 5.00th=[ 1827], 10.00th=[ 2802],
- | 20.00th=[ 4293], 30.00th=[ 6718], 40.00th=[ 10421],
- | 50.00th=[ 15533], 60.00th=[ 27395], 70.00th=[ 54264],
- | 80.00th=[ 130548], 90.00th=[ 287310], 95.00th=[ 429917],
- | 99.00th=[ 717226], 99.50th=[1317012], 99.90th=[6878659],
- | 99.95th=[8556381], 99.99th=[8925479]
- bw ( KiB/s): min= 0, max= 3233, per=47.42%, avg=2618.50, stdev=869.03, samples=2
- iops : min= 0, max= 50, avg=40.50, stdev=13.44, samples=2
- lat (usec) : 2=0.02%, 4=0.02%, 100=0.02%, 250=0.02%, 500=0.06%
- lat (usec) : 750=0.37%, 1000=1.06%
- lat (msec) : 2=4.28%, 4=11.89%, 10=21.32%, 20=15.64%, 50=14.05%
- lat (msec) : 100=8.90%, 250=10.59%, 500=8.33%, 750=2.61%, 1000=0.20%
- cpu : usr=0.03%, sys=0.15%, ctx=7781, majf=0, minf=30
- IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=0,4911,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- read: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:34:57 2019
- read: IOPS=68, BW=4371KiB/s (4476kB/s)(513MiB/120237msec)
- slat (nsec): min=0, max=9176.8k, avg=52739.78, stdev=188486.51
- clat (nsec): min=0, max=1825.5M, avg=58615474.01, stdev=197991591.34
- lat (nsec): min=0, max=1825.5M, avg=58669431.87, stdev=198008807.92
- clat percentiles (nsec):
- | 1.00th=[ 716], 5.00th=[ 732], 10.00th=[ 772],
- | 20.00th=[ 820], 30.00th=[ 884], 40.00th=[ 1032],
- | 50.00th=[ 1256], 60.00th=[ 1640], 70.00th=[ 1417216],
- | 80.00th=[ 3424256], 90.00th=[ 49020928], 95.00th=[ 599785472],
- | 99.00th=[ 952107008], 99.50th=[1069547520], 99.90th=[1350565888],
- | 99.95th=[1518338048], 99.99th=[1820327936]
- bw ( KiB/s): min= 0, max= 4448, per=60.15%, avg=4376.50, stdev=101.12, samples=2
- iops : min= 0, max= 69, avg=68.00, stdev= 1.41, samples=2
- lat (nsec) : 750=6.54%, 1000=32.33%
- lat (usec) : 2=21.82%, 4=0.22%, 10=0.11%, 20=0.52%, 50=0.17%
- lat (usec) : 100=0.19%, 250=0.10%, 500=0.06%, 750=0.82%, 1000=3.16%
- lat (msec) : 2=7.88%, 4=7.27%, 10=4.92%, 20=2.01%, 50=1.99%
- lat (msec) : 100=0.97%, 250=0.83%, 500=2.22%, 750=3.12%, 1000=2.05%
- cpu : usr=0.04%, sys=0.11%, ctx=6246, majf=0, minf=30
- IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=8209,0,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- readwrite: (groupid=0, jobs=1): err= 0: pid=14: Wed May 8 20:34:57 2019
- read: IOPS=45, BW=2911KiB/s (2981kB/s)(341MiB/120007msec)
- slat (nsec): min=0, max=3786.0k, avg=42845.43, stdev=115101.89
- clat (nsec): min=0, max=809090k, avg=19029910.56, stdev=72180861.64
- lat (nsec): min=0, max=809121k, avg=19073885.97, stdev=72190890.86
- clat percentiles (nsec):
- | 1.00th=[ 724], 5.00th=[ 740], 10.00th=[ 748],
- | 20.00th=[ 868], 30.00th=[ 1004], 40.00th=[ 1144],
- | 50.00th=[ 1304], 60.00th=[ 52480], 70.00th=[ 1105920],
- | 80.00th=[ 3227648], 90.00th=[ 43778048], 95.00th=[ 87556096],
- | 99.00th=[446693376], 99.50th=[509607936], 99.90th=[734003200],
- | 99.95th=[784334848], 99.99th=[809500672]
- bw ( KiB/s): min= 0, max= 4526, per=39.99%, avg=2909.50, stdev=2286.08, samples=2
- iops : min= 0, max= 70, avg=45.00, stdev=35.36, samples=2
- write: IOPS=46, BW=3008KiB/s (3081kB/s)(353MiB/120007msec)
- slat (nsec): min=0, max=5596.6k, avg=184642.48, stdev=236894.22
- clat (nsec): min=0, max=852576k, avg=66695651.86, stdev=130745032.75
- lat (nsec): min=0, max=852605k, avg=66881887.03, stdev=130698526.14
- clat percentiles (usec):
- | 1.00th=[ 1020], 5.00th=[ 1221], 10.00th=[ 1369], 20.00th=[ 1696],
- | 30.00th=[ 2212], 40.00th=[ 3163], 50.00th=[ 5145], 60.00th=[ 10945],
- | 70.00th=[ 49546], 80.00th=[102237], 90.00th=[200279], 95.00th=[383779],
- | 99.00th=[616563], 99.50th=[683672], 99.90th=[750781], 99.95th=[826278],
- | 99.99th=[851444]
- bw ( KiB/s): min= 0, max= 4664, per=54.45%, avg=3007.00, stdev=2343.35, samples=2
- iops : min= 0, max= 72, avg=46.50, stdev=36.06, samples=2
- lat (nsec) : 750=4.73%, 1000=9.89%
- lat (usec) : 2=14.22%, 4=0.05%, 10=0.07%, 20=0.32%, 50=0.23%
- lat (usec) : 100=0.32%, 250=0.35%, 500=0.22%, 750=1.08%, 1000=2.54%
- lat (msec) : 2=17.03%, 4=12.00%, 10=8.42%, 20=3.30%, 50=5.66%
- lat (msec) : 100=6.99%, 250=7.36%, 500=3.60%, 750=1.53%, 1000=0.12%
- cpu : usr=0.09%, sys=0.25%, ctx=14803, majf=0, minf=37
- IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=5457,5640,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- Run status group 0 (all jobs):
- READ: bw=7277KiB/s (7451kB/s), 2911KiB/s-4371KiB/s (2981kB/s-4476kB/s), io=854MiB (896MB), run=120007-120237msec
- WRITE: bw=5522KiB/s (5655kB/s), 2571KiB/s-3008KiB/s (2633kB/s-3081kB/s), io=660MiB (692MB), run=120007-122324msec
- Disk stats (read/write):
- rbd204: ios=5634/12337, merge=0/3, ticks=642225/878157, in_queue=1511367, util=4.27%
- <example-benchmark-fio-server-612-benchmark-2dg6w>
- randwrite: (groupid=0, jobs=1): err= 0: pid=11: Wed May 8 20:34:58 2019
- write: IOPS=27, BW=1758KiB/s (1800kB/s)(207MiB/120534msec)
- slat (nsec): min=0, max=4257.1M, avg=2010805.23, stdev=76515967.14
- clat (nsec): min=0, max=6824.8M, avg=143809839.17, stdev=306619330.65
- lat (nsec): min=0, max=6833.9M, avg=145820705.46, stdev=319679458.15
- clat percentiles (usec):
- | 1.00th=[ 1221], 5.00th=[ 1663], 10.00th=[ 2114],
- | 20.00th=[ 3032], 30.00th=[ 4178], 40.00th=[ 6063],
- | 50.00th=[ 9634], 60.00th=[ 16909], 70.00th=[ 88605],
- | 80.00th=[ 333448], 90.00th=[ 492831], 95.00th=[ 616563],
- | 99.00th=[ 809501], 99.50th=[1035994], 99.90th=[3506439],
- | 99.95th=[6811550], 99.99th=[6811550]
- bw ( KiB/s): min= 0, max= 2243, per=33.59%, avg=1763.00, stdev=678.82, samples=2
- iops : min= 0, max= 35, avg=27.50, stdev=10.61, samples=2
- lat (usec) : 500=0.03%, 750=0.03%, 1000=0.18%
- lat (msec) : 2=8.41%, 4=19.78%, 10=22.32%, 20=11.82%, 50=6.23%
- lat (msec) : 100=1.45%, 250=5.47%, 500=14.45%, 750=8.50%, 1000=0.88%
- cpu : usr=0.02%, sys=0.12%, ctx=5571, majf=0, minf=30
- IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=0,3307,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- read: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:34:58 2019
- read: IOPS=136, BW=8764KiB/s (8975kB/s)(1032MiB/120562msec)
- slat (nsec): min=0, max=12461k, avg=36945.42, stdev=180100.44
- clat (nsec): min=0, max=1972.8M, avg=29204481.49, stdev=143415318.41
- lat (nsec): min=0, max=1972.8M, avg=29242530.82, stdev=143430361.26
- clat percentiles (nsec):
- | 1.00th=[ 716], 5.00th=[ 732], 10.00th=[ 756],
- | 20.00th=[ 788], 30.00th=[ 820], 40.00th=[ 868],
- | 50.00th=[ 900], 60.00th=[ 1032], 70.00th=[ 1320],
- | 80.00th=[ 995328], 90.00th=[ 4685824], 95.00th=[ 51642368],
- | 99.00th=[ 826277888], 99.50th=[ 952107008], 99.90th=[1333788672],
- | 99.95th=[1535115264], 99.99th=[1837105152]
- bw ( KiB/s): min= 0, max= 9481, per=72.35%, avg=8802.50, stdev=959.54, samples=2
- iops : min= 0, max= 148, avg=137.00, stdev=15.56, samples=2
- lat (nsec) : 750=9.74%, 1000=49.14%
- lat (usec) : 2=17.42%, 4=0.25%, 10=0.19%, 20=0.59%, 50=0.16%
- lat (usec) : 100=0.12%, 250=0.12%, 500=0.13%, 750=0.43%, 1000=1.79%
- lat (msec) : 2=4.74%, 4=4.43%, 10=3.01%, 20=1.31%, 50=1.40%
- lat (msec) : 100=0.57%, 250=0.58%, 500=1.05%, 750=1.38%, 1000=1.13%
- cpu : usr=0.06%, sys=0.18%, ctx=7446, majf=0, minf=29
- IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=16507,0,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- readwrite: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:34:58 2019
- read: IOPS=53, BW=3419KiB/s (3501kB/s)(401MiB/120002msec)
- slat (nsec): min=0, max=1954.4k, avg=31149.84, stdev=77107.69
- clat (nsec): min=0, max=734117k, avg=11037698.80, stdev=54963807.10
- lat (nsec): min=0, max=734145k, avg=11069791.59, stdev=54971279.93
- clat percentiles (nsec):
- | 1.00th=[ 724], 5.00th=[ 732], 10.00th=[ 740],
- | 20.00th=[ 812], 30.00th=[ 908], 40.00th=[ 1032],
- | 50.00th=[ 1176], 60.00th=[ 1304], 70.00th=[ 1816],
- | 80.00th=[ 856064], 90.00th=[ 5275648], 95.00th=[ 51118080],
- | 99.00th=[320864256], 99.50th=[455081984], 99.90th=[616562688],
- | 99.95th=[650117120], 99.99th=[734003200]
- bw ( KiB/s): min= 0, max= 4692, per=28.10%, avg=3418.50, stdev=1801.00, samples=2
- iops : min= 0, max= 73, avg=53.00, stdev=28.28, samples=2
- write: IOPS=54, BW=3508KiB/s (3592kB/s)(411MiB/120002msec)
- slat (nsec): min=0, max=10682k, avg=192837.44, stdev=258303.43
- clat (nsec): min=0, max=1267.3M, avg=62261381.68, stdev=127104471.59
- lat (nsec): min=0, max=1267.3M, avg=62455492.49, stdev=127055529.35
- clat percentiles (usec):
- | 1.00th=[ 1012], 5.00th=[ 1221], 10.00th=[ 1401],
- | 20.00th=[ 1795], 30.00th=[ 2376], 40.00th=[ 3458],
- | 50.00th=[ 5997], 60.00th=[ 14484], 70.00th=[ 32900],
- | 80.00th=[ 99091], 90.00th=[ 187696], 95.00th=[ 325059],
- | 99.00th=[ 675283], 99.50th=[ 750781], 99.90th=[ 901776],
- | 99.95th=[ 977273], 99.99th=[1266680]
- bw ( KiB/s): min= 0, max= 4778, per=66.77%, avg=3505.00, stdev=1800.29, samples=2
- iops : min= 0, max= 74, avg=54.00, stdev=28.28, samples=2
- lat (nsec) : 750=5.97%, 1000=12.84%
- lat (usec) : 2=15.87%, 4=0.08%, 10=0.23%, 20=0.78%, 50=0.68%
- lat (usec) : 100=0.62%, 250=0.69%, 500=0.44%, 750=0.65%, 1000=1.93%
- lat (msec) : 2=14.01%, 4=11.19%, 10=7.22%, 20=4.84%, 50=5.47%
- lat (msec) : 100=5.08%, 250=7.06%, 500=3.13%, 750=0.98%, 1000=0.25%
- cpu : usr=0.09%, sys=0.31%, ctx=15021, majf=0, minf=31
- IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=6410,6574,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- Run status group 0 (all jobs):
- READ: bw=11.9MiB/s (12.5MB/s), 3419KiB/s-8764KiB/s (3501kB/s-8975kB/s), io=1433MiB (1502MB), run=120002-120562msec
- WRITE: bw=5250KiB/s (5376kB/s), 1758KiB/s-3508KiB/s (1800kB/s-3592kB/s), io=618MiB (648MB), run=120002-120534msec
- Disk stats (read/write):
- rbd279: ios=5460/10356, merge=0/3, ticks=596117/878665, in_queue=1466829, util=4.57%
- Jobs: 48 (f=45): [_(2),m(1)][99.8%][r=1436M,w=1434M][r=22.4k,w=22.4k IOPS][eta 00m:52s] <example-benchmark-fio-server-898-benchmark-2q9b7>
- randwrite: (groupid=0, jobs=1): err= 0: pid=11: Wed May 8 20:34:58 2019
- write: IOPS=13, BW=835KiB/s (855kB/s)(98.1MiB/120351msec)
- slat (nsec): min=0, max=2022.3M, avg=2728134.72, stdev=58249237.85
- clat (nsec): min=0, max=3275.6M, avg=304687648.20, stdev=456863340.63
- lat (nsec): min=0, max=3275.7M, avg=307412545.76, stdev=460022219.28
- clat percentiles (usec):
- | 1.00th=[ 1303], 5.00th=[ 2024], 10.00th=[ 2704],
- | 20.00th=[ 4490], 30.00th=[ 6915], 40.00th=[ 11338],
- | 50.00th=[ 18744], 60.00th=[ 50070], 70.00th=[ 392168],
- | 80.00th=[ 633340], 90.00th=[1010828], 95.00th=[1317012],
- | 99.00th=[1635779], 99.50th=[1920992], 99.90th=[2231370],
- | 99.95th=[3271558], 99.99th=[3271558]
- bw ( KiB/s): min= 0, max= 1076, per=21.35%, avg=833.00, stdev=343.65, samples=2
- iops : min= 0, max= 16, avg=12.50, stdev= 4.95, samples=2
- lat (usec) : 1000=0.19%
- lat (msec) : 2=4.53%, 4=12.95%, 10=19.85%, 20=13.85%, 50=8.74%
- lat (msec) : 100=1.91%, 250=0.89%, 500=11.81%, 750=8.42%, 1000=6.76%
- cpu : usr=0.02%, sys=0.07%, ctx=2848, majf=0, minf=30
- IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=0,1567,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- read: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:34:58 2019
- read: IOPS=307, BW=19.2MiB/s (20.1MB/s)(2315MiB/120566msec)
- slat (nsec): min=0, max=1464.6M, avg=73420.58, stdev=8011473.67
- clat (nsec): min=0, max=2370.1M, avg=12938782.83, stdev=99222958.48
- lat (nsec): min=0, max=2370.1M, avg=13013350.53, stdev=99551960.23
- clat percentiles (nsec):
- | 1.00th=[ 708], 5.00th=[ 724], 10.00th=[ 732],
- | 20.00th=[ 772], 30.00th=[ 788], 40.00th=[ 804],
- | 50.00th=[ 836], 60.00th=[ 868], 70.00th=[ 900],
- | 80.00th=[ 1020], 90.00th=[ 11840], 95.00th=[ 2703360],
- | 99.00th=[ 624951296], 99.50th=[ 792723456], 99.90th=[1199570944],
- | 99.95th=[1384120320], 99.99th=[1937768448]
- bw ( KiB/s): min= 0, max=26548, per=87.30%, avg=19751.50, stdev=9611.70, samples=2
- iops : min= 0, max= 415, avg=308.50, stdev=150.61, samples=2
- lat (nsec) : 750=12.45%, 1000=67.02%
- lat (usec) : 2=9.06%, 4=0.23%, 10=0.46%, 20=2.23%, 50=0.58%
- lat (usec) : 100=0.18%, 250=0.09%, 500=0.13%, 750=0.15%, 1000=0.44%
- lat (msec) : 2=1.38%, 4=1.28%, 10=1.15%, 20=0.51%, 50=0.46%
- lat (msec) : 100=0.24%, 250=0.24%, 500=0.46%, 750=0.65%, 1000=0.42%
- cpu : usr=0.08%, sys=0.34%, ctx=5906, majf=0, minf=30
- IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=37091,0,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- readwrite: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:34:58 2019
- read: IOPS=46, BW=2977KiB/s (3048kB/s)(349MiB/120021msec)
- slat (nsec): min=0, max=5457.3k, avg=23255.68, stdev=98205.93
- clat (nsec): min=0, max=786199k, avg=7736512.34, stdev=47943367.55
- lat (nsec): min=0, max=786220k, avg=7760701.96, stdev=47953450.62
- clat percentiles (nsec):
- | 1.00th=[ 708], 5.00th=[ 724], 10.00th=[ 724],
- | 20.00th=[ 748], 30.00th=[ 828], 40.00th=[ 916],
- | 50.00th=[ 1048], 60.00th=[ 1192], 70.00th=[ 1368],
- | 80.00th=[ 19328], 90.00th=[ 978944], 95.00th=[ 24772608],
- | 99.00th=[242221056], 99.50th=[404750336], 99.90th=[608174080],
- | 99.95th=[700448768], 99.99th=[784334848]
- bw ( KiB/s): min= 0, max= 4710, per=13.16%, avg=2976.50, stdev=2451.54, samples=2
- iops : min= 0, max= 73, avg=46.00, stdev=38.18, samples=2
- write: IOPS=48, BW=3076KiB/s (3150kB/s)(361MiB/120021msec)
- slat (nsec): min=0, max=6098.2k, avg=209590.76, stdev=291288.10
- clat (nsec): min=0, max=1088.8M, avg=75761080.52, stdev=143915707.64
- lat (nsec): min=0, max=1088.8M, avg=75971772.86, stdev=143854208.57
- clat percentiles (usec):
- | 1.00th=[ 963], 5.00th=[ 1123], 10.00th=[ 1287],
- | 20.00th=[ 1663], 30.00th=[ 2278], 40.00th=[ 3556],
- | 50.00th=[ 6980], 60.00th=[ 21890], 70.00th=[ 68682],
- | 80.00th=[ 112722], 90.00th=[ 231736], 95.00th=[ 400557],
- | 99.00th=[ 708838], 99.50th=[ 817890], 99.90th=[ 968885],
- | 99.95th=[ 994051], 99.99th=[1082131]
- bw ( KiB/s): min= 0, max= 4848, per=78.78%, avg=3074.00, stdev=2508.81, samples=2
- iops : min= 0, max= 75, avg=47.50, stdev=38.89, samples=2
- lat (nsec) : 750=10.49%, 1000=12.60%
- lat (usec) : 2=14.41%, 4=0.04%, 10=0.41%, 20=1.55%, 50=1.02%
- lat (usec) : 100=0.98%, 250=1.01%, 500=0.77%, 750=0.50%, 1000=1.36%
- lat (msec) : 2=13.40%, 4=8.68%, 10=6.46%, 20=3.22%, 50=4.30%
- lat (msec) : 100=6.09%, 250=7.50%, 500=3.56%, 750=1.25%, 1000=0.42%
- cpu : usr=0.06%, sys=0.27%, ctx=11681, majf=0, minf=34
- IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=5582,5765,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- Run status group 0 (all jobs):
- READ: bw=22.1MiB/s (23.2MB/s), 2977KiB/s-19.2MiB/s (3048kB/s-20.1MB/s), io=2664MiB (2793MB), run=120021-120566msec
- WRITE: bw=3902KiB/s (3996kB/s), 835KiB/s-3076KiB/s (855kB/s-3150kB/s), io=459MiB (481MB), run=120021-120351msec
- Disk stats (read/write):
- rbd321: ios=3932/7745, merge=0/1, ticks=559319/910120, in_queue=1463629, util=5.16%
- <example-benchmark-fio-server-417-benchmark-gs9tf>
- randwrite: (groupid=0, jobs=1): err= 0: pid=11: Wed May 8 20:34:58 2019
- write: IOPS=22, BW=1436KiB/s (1471kB/s)(171MiB/121813msec)
- slat (nsec): min=0, max=318460k, avg=653825.73, stdev=10094001.52
- clat (nsec): min=0, max=4853.7M, avg=175745024.98, stdev=330156121.69
- lat (nsec): min=0, max=4853.0M, avg=176399966.50, stdev=330429601.02
- clat percentiles (usec):
- | 1.00th=[ 1074], 5.00th=[ 1762], 10.00th=[ 2606],
- | 20.00th=[ 4424], 30.00th=[ 7046], 40.00th=[ 10945],
- | 50.00th=[ 17433], 60.00th=[ 46400], 70.00th=[ 223347],
- | 80.00th=[ 367002], 90.00th=[ 557843], 95.00th=[ 666895],
- | 99.00th=[ 943719], 99.50th=[1887437], 99.90th=[4798284],
- | 99.95th=[4865393], 99.99th=[4865393]
- bw ( KiB/s): min= 0, max= 1607, per=31.79%, avg=1455.50, stdev=214.25, samples=2
- iops : min= 0, max= 25, avg=22.50, stdev= 3.54, samples=2
- lat (usec) : 4=0.07%, 750=0.07%, 1000=0.66%
- lat (msec) : 2=5.42%, 4=12.23%, 10=19.00%, 20=14.54%, 50=8.53%
- lat (msec) : 100=4.43%, 250=7.14%, 500=15.49%, 750=9.85%, 1000=1.90%
- cpu : usr=0.03%, sys=0.10%, ctx=4795, majf=0, minf=30
- IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=0,2731,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- read: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:34:58 2019
- read: IOPS=123, BW=7924KiB/s (8115kB/s)(934MiB/120724msec)
- slat (nsec): min=0, max=10609k, avg=31616.62, stdev=166024.40
- clat (nsec): min=0, max=1785.3M, avg=32298334.44, stdev=152225712.97
- lat (nsec): min=0, max=1785.3M, avg=32330977.46, stdev=152246360.23
- clat percentiles (nsec):
- | 1.00th=[ 716], 5.00th=[ 724], 10.00th=[ 740],
- | 20.00th=[ 780], 30.00th=[ 796], 40.00th=[ 844],
- | 50.00th=[ 876], 60.00th=[ 908], 70.00th=[ 1112],
- | 80.00th=[ 1496], 90.00th=[ 3096576], 95.00th=[ 61603840],
- | 99.00th=[ 851443712], 99.50th=[ 977272832], 99.90th=[1266679808],
- | 99.95th=[1468006400], 99.99th=[1769996288]
- bw ( KiB/s): min= 0, max= 9050, per=72.45%, avg=7959.00, stdev=1542.91, samples=2
- iops : min= 0, max= 141, avg=124.00, stdev=24.04, samples=2
- lat (nsec) : 750=10.84%, 1000=55.88%
- lat (usec) : 2=14.46%, 4=0.16%, 10=0.30%, 20=0.97%, 50=0.21%
- lat (usec) : 100=0.13%, 250=0.14%, 500=0.11%, 750=0.27%, 1000=1.05%
- lat (msec) : 2=3.48%, 4=3.04%, 10=2.30%, 20=0.89%, 50=0.66%
- lat (msec) : 100=0.31%, 250=0.31%, 500=1.02%, 750=1.93%, 1000=1.09%
- cpu : usr=0.04%, sys=0.16%, ctx=5094, majf=0, minf=30
- IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=14945,0,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- readwrite: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:34:58 2019
- read: IOPS=48, BW=3080KiB/s (3154kB/s)(361MiB/120004msec)
- slat (nsec): min=0, max=6708.7k, avg=29217.79, stdev=128662.97
- clat (nsec): min=0, max=1112.9M, avg=9348407.86, stdev=55207623.48
- lat (nsec): min=0, max=1112.0M, avg=9378582.70, stdev=55212255.77
- clat percentiles (nsec):
- | 1.00th=[ 716], 5.00th=[ 724], 10.00th=[ 732],
- | 20.00th=[ 764], 30.00th=[ 868], 40.00th=[ 964],
- | 50.00th=[ 1112], 60.00th=[ 1240], 70.00th=[ 1528],
- | 80.00th=[ 148480], 90.00th=[ 2146304], 95.00th=[ 33423360],
- | 99.00th=[ 312475648], 99.50th=[ 467664896], 99.90th=[ 624951296],
- | 99.95th=[ 742391808], 99.99th=[1115684864]
- bw ( KiB/s): min= 0, max= 4738, per=28.03%, avg=3079.50, stdev=2345.47, samples=2
- iops : min= 0, max= 74, avg=48.00, stdev=36.77, samples=2
- write: IOPS=49, BW=3190KiB/s (3266kB/s)(374MiB/120004msec)
- slat (nsec): min=0, max=7569.2k, avg=191882.63, stdev=320415.62
- clat (nsec): min=0, max=1050.9M, avg=71244375.78, stdev=134209200.19
- lat (nsec): min=0, max=1050.9M, avg=71437577.53, stdev=134155113.65
- clat percentiles (usec):
- | 1.00th=[ 1037], 5.00th=[ 1287], 10.00th=[ 1516],
- | 20.00th=[ 2073], 30.00th=[ 2900], 40.00th=[ 4359],
- | 50.00th=[ 7308], 60.00th=[ 18744], 70.00th=[ 60031],
- | 80.00th=[ 109577], 90.00th=[ 217056], 95.00th=[ 379585],
- | 99.00th=[ 641729], 99.50th=[ 692061], 99.90th=[ 834667],
- | 99.95th=[ 901776], 99.99th=[1052771]
- bw ( KiB/s): min= 0, max= 4899, per=69.63%, avg=3187.50, stdev=2420.43, samples=2
- iops : min= 0, max= 76, avg=49.50, stdev=37.48, samples=2
- lat (nsec) : 750=9.10%, 1000=11.98%
- lat (usec) : 2=14.25%, 4=0.09%, 10=0.37%, 20=1.14%, 50=0.95%
- lat (usec) : 100=0.88%, 250=1.16%, 500=0.80%, 750=0.48%, 1000=1.41%
- lat (msec) : 2=10.99%, 4=10.91%, 10=8.98%, 20=3.69%, 50=4.57%
- lat (msec) : 100=5.85%, 250=7.58%, 500=3.06%, 750=1.60%, 1000=0.14%
- cpu : usr=0.07%, sys=0.28%, ctx=12640, majf=0, minf=31
- IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=5775,5978,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- Run status group 0 (all jobs):
- READ: bw=10.7MiB/s (11.2MB/s), 3080KiB/s-7924KiB/s (3154kB/s-8115kB/s), io=1295MiB (1358MB), run=120004-120724msec
- WRITE: bw=4579KiB/s (4689kB/s), 1436KiB/s-3190KiB/s (1471kB/s-3266kB/s), io=545MiB (571MB), run=120004-121813msec
- Disk stats (read/write):
- rbd128: ios=3825/8987, merge=0/2, ticks=579612/909061, in_queue=1482232, util=3.77%
- <example-benchmark-fio-server-702-benchmark-72kl9>
- randwrite: (groupid=0, jobs=1): err= 0: pid=11: Wed May 8 20:34:58 2019
- write: IOPS=22, BW=1418KiB/s (1452kB/s)(168MiB/121297msec)
- slat (nsec): min=0, max=427537k, avg=661838.85, stdev=12390156.81
- clat (nsec): min=0, max=2646.7M, avg=182451904.64, stdev=362745931.13
- lat (nsec): min=0, max=2646.8M, avg=183115311.85, stdev=362747429.03
- clat percentiles (usec):
- | 1.00th=[ 1123], 5.00th=[ 1549], 10.00th=[ 1893],
- | 20.00th=[ 2573], 30.00th=[ 3326], 40.00th=[ 4424],
- | 50.00th=[ 6063], 60.00th=[ 9634], 70.00th=[ 23200],
- | 80.00th=[ 396362], 90.00th=[ 725615], 95.00th=[ 968885],
- | 99.00th=[1535116], 99.50th=[1551893], 99.90th=[2566915],
- | 99.95th=[2634023], 99.99th=[2634023]
- bw ( KiB/s): min= 0, max= 1689, per=33.81%, avg=1430.50, stdev=365.57, samples=2
- iops : min= 0, max= 26, avg=22.00, stdev= 5.66, samples=2
- lat (usec) : 750=0.26%, 1000=0.48%
- lat (msec) : 2=10.95%, 4=25.33%, 10=23.39%, 20=8.68%, 50=3.87%
- lat (msec) : 100=2.38%, 250=2.38%, 500=5.85%, 750=7.56%, 1000=4.17%
- cpu : usr=0.03%, sys=0.10%, ctx=4842, majf=0, minf=30
- IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=0,2685,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- read: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:34:58 2019
- read: IOPS=202, BW=12.7MiB/s (13.3MB/s)(1525MiB/120525msec)
- slat (nsec): min=0, max=67962k, avg=31963.04, stdev=493698.59
- clat (nsec): min=0, max=1958.1M, avg=19763723.06, stdev=119137048.25
- lat (nsec): min=0, max=1958.2M, avg=19796745.38, stdev=119154585.48
- clat percentiles (nsec):
- | 1.00th=[ 716], 5.00th=[ 724], 10.00th=[ 740],
- | 20.00th=[ 780], 30.00th=[ 796], 40.00th=[ 828],
- | 50.00th=[ 868], 60.00th=[ 900], 70.00th=[ 1004],
- | 80.00th=[ 1400], 90.00th=[ 1777664], 95.00th=[ 8028160],
- | 99.00th=[ 742391808], 99.50th=[ 893386752], 99.90th=[1249902592],
- | 99.95th=[1300234240], 99.99th=[1686110208]
- bw ( KiB/s): min= 0, max=15112, per=82.69%, avg=12982.00, stdev=3012.27, samples=2
- iops : min= 0, max= 236, avg=202.50, stdev=47.38, samples=2
- lat (nsec) : 750=11.73%, 1000=58.10%
- lat (usec) : 2=12.66%, 4=0.15%, 10=0.42%, 20=1.68%, 50=0.48%
- lat (usec) : 100=0.20%, 250=0.13%, 500=0.11%, 750=0.26%, 1000=1.27%
- lat (msec) : 2=3.29%, 4=2.92%, 10=1.97%, 20=0.82%, 50=0.57%
- lat (msec) : 100=0.27%, 250=0.28%, 500=0.62%, 750=1.13%, 1000=0.68%
- cpu : usr=0.07%, sys=0.24%, ctx=7087, majf=0, minf=30
- IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=24401,0,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- readwrite: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:34:58 2019
- read: IOPS=43, BW=2754KiB/s (2820kB/s)(323MiB/120018msec)
- slat (nsec): min=0, max=2284.4k, avg=27384.79, stdev=79125.39
- clat (nsec): min=0, max=935607k, avg=7895742.05, stdev=47845118.61
- lat (nsec): min=0, max=935651k, avg=7924026.04, stdev=47851357.49
- clat percentiles (nsec):
- | 1.00th=[ 716], 5.00th=[ 724], 10.00th=[ 732],
- | 20.00th=[ 764], 30.00th=[ 852], 40.00th=[ 948],
- | 50.00th=[ 1096], 60.00th=[ 1240], 70.00th=[ 1512],
- | 80.00th=[ 126464], 90.00th=[ 2113536], 95.00th=[ 32112640],
- | 99.00th=[200278016], 99.50th=[392167424], 99.90th=[641728512],
- | 99.95th=[717225984], 99.99th=[935329792]
- bw ( KiB/s): min= 0, max= 4245, per=17.54%, avg=2753.50, stdev=2109.30, samples=2
- iops : min= 0, max= 66, avg=42.50, stdev=33.23, samples=2
- write: IOPS=44, BW=2843KiB/s (2911kB/s)(333MiB/120018msec)
- slat (nsec): min=0, max=10903k, avg=179886.99, stdev=309272.59
- clat (nsec): min=0, max=1047.2M, avg=82461684.37, stdev=148141789.84
- lat (nsec): min=0, max=1047.5M, avg=82642759.28, stdev=148090585.05
- clat percentiles (usec):
- | 1.00th=[ 971], 5.00th=[ 1205], 10.00th=[ 1385],
- | 20.00th=[ 1893], 30.00th=[ 2573], 40.00th=[ 3982],
- | 50.00th=[ 9765], 60.00th=[ 26608], 70.00th=[ 79168],
- | 80.00th=[ 123208], 90.00th=[ 265290], 95.00th=[ 446694],
- | 99.00th=[ 692061], 99.50th=[ 742392], 99.90th=[ 876610],
- | 99.95th=[ 910164], 99.99th=[1044382]
- bw ( KiB/s): min= 0, max= 4369, per=67.16%, avg=2841.50, stdev=2160.21, samples=2
- iops : min= 0, max= 68, avg=44.00, stdev=33.94, samples=2
- lat (nsec) : 750=8.04%, 1000=13.42%
- lat (usec) : 2=13.92%, 4=0.02%, 10=0.30%, 20=1.21%, 50=0.87%
- lat (usec) : 100=1.30%, 250=0.97%, 500=0.61%, 750=0.59%, 1000=1.70%
- lat (msec) : 2=12.33%, 4=10.48%, 10=5.81%, 20=3.69%, 50=5.07%
- lat (msec) : 100=5.65%, 250=8.36%, 500=3.69%, 750=1.73%, 1000=0.26%
- cpu : usr=0.05%, sys=0.25%, ctx=11299, majf=0, minf=30
- IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=5164,5328,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- Run status group 0 (all jobs):
- READ: bw=15.3MiB/s (16.1MB/s), 2754KiB/s-12.7MiB/s (2820kB/s-13.3MB/s), io=1848MiB (1938MB), run=120018-120525msec
- WRITE: bw=4231KiB/s (4333kB/s), 1418KiB/s-2843KiB/s (1452kB/s-2911kB/s), io=501MiB (526MB), run=120018-121297msec
- Disk stats (read/write):
- rbd182: ios=4729/8233, merge=0/2, ticks=563379/937246, in_queue=1494170, util=4.62%
- <example-benchmark-fio-server-620-benchmark-rlhzh>
- randwrite: (groupid=0, jobs=1): err= 0: pid=11: Wed May 8 20:34:58 2019
- write: IOPS=36, BW=2346KiB/s (2402kB/s)(276MiB/120414msec)
- slat (nsec): min=0, max=1809.7M, avg=1451143.21, stdev=41416492.97
- clat (nsec): min=0, max=2608.3M, avg=107987024.10, stdev=273072237.75
- lat (nsec): min=0, max=2673.9M, avg=109505679.21, stdev=278381371.46
- clat percentiles (usec):
- | 1.00th=[ 1139], 5.00th=[ 1450], 10.00th=[ 1663],
- | 20.00th=[ 2147], 30.00th=[ 2769], 40.00th=[ 3654],
- | 50.00th=[ 5145], 60.00th=[ 8586], 70.00th=[ 16909],
- | 80.00th=[ 50594], 90.00th=[ 488637], 95.00th=[ 759170],
- | 99.00th=[1283458], 99.50th=[1384121], 99.90th=[1870660],
- | 99.95th=[1887437], 99.99th=[2600469]
- bw ( KiB/s): min= 0, max= 3078, per=46.17%, avg=2351.50, stdev=1027.43, samples=2
- iops : min= 0, max= 48, avg=36.50, stdev=16.26, samples=2
- lat (usec) : 500=0.02%, 750=0.09%, 1000=0.25%
- lat (msec) : 2=16.78%, 4=26.14%, 10=19.43%, 20=9.00%, 50=8.34%
- lat (msec) : 100=5.01%, 250=2.68%, 500=2.40%, 750=4.87%, 1000=2.47%
- cpu : usr=0.04%, sys=0.17%, ctx=8054, majf=0, minf=29
- IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=0,4411,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- read: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:34:58 2019
- read: IOPS=832, BW=51.8MiB/s (54.4MB/s)(6223MiB/120042msec)
- slat (nsec): min=0, max=1627.4M, avg=61070.51, stdev=7625575.00
- clat (nsec): min=0, max=2303.2M, avg=4747375.68, stdev=57025931.76
- lat (nsec): min=0, max=2303.2M, avg=4809469.51, stdev=57538826.11
- clat percentiles (nsec):
- | 1.00th=[ 708], 5.00th=[ 716], 10.00th=[ 732],
- | 20.00th=[ 764], 30.00th=[ 780], 40.00th=[ 796],
- | 50.00th=[ 836], 60.00th=[ 868], 70.00th=[ 900],
- | 80.00th=[ 1400], 90.00th=[ 12864], 95.00th=[ 782336],
- | 99.00th=[ 43778048], 99.50th=[ 438304768], 99.90th=[ 901775360],
- | 99.95th=[1027604480], 99.99th=[1417674752]
- bw ( KiB/s): min= 0, max=88652, per=95.26%, avg=53098.00, stdev=50280.95, samples=2
- iops : min= 0, max= 1391, avg=832.50, stdev=789.84, samples=2
- lat (nsec) : 750=14.75%, 1000=60.85%
- lat (usec) : 2=5.43%, 4=0.15%, 10=1.52%, 20=9.01%, 50=2.48%
- lat (usec) : 100=0.27%, 250=0.18%, 500=0.12%, 750=0.17%, 1000=0.56%
- lat (msec) : 2=1.15%, 4=0.85%, 10=0.60%, 20=0.40%, 50=0.61%
- lat (msec) : 100=0.20%, 250=0.10%, 500=0.16%, 750=0.25%, 1000=0.14%
- cpu : usr=0.22%, sys=0.80%, ctx=11085, majf=0, minf=30
- IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=99919,0,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- readwrite: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:34:58 2019
- read: IOPS=41, BW=2655KiB/s (2719kB/s)(311MiB/120015msec)
- slat (nsec): min=0, max=4456.3k, avg=32168.09, stdev=97756.59
- clat (nsec): min=0, max=989312k, avg=12693057.98, stdev=60321579.79
- lat (nsec): min=0, max=989349k, avg=12726223.12, stdev=60329190.75
- clat percentiles (nsec):
- | 1.00th=[ 716], 5.00th=[ 724], 10.00th=[ 740],
- | 20.00th=[ 820], 30.00th=[ 916], 40.00th=[ 1080],
- | 50.00th=[ 1224], 60.00th=[ 1384], 70.00th=[ 44288],
- | 80.00th=[ 1122304], 90.00th=[ 15138816], 95.00th=[ 62128128],
- | 99.00th=[320864256], 99.50th=[425721856], 99.90th=[767557632],
- | 99.95th=[935329792], 99.99th=[985661440]
- bw ( KiB/s): min= 0, max= 3947, per=4.76%, avg=2655.00, stdev=1827.16, samples=2
- iops : min= 0, max= 61, avg=41.00, stdev=28.28, samples=2
- write: IOPS=43, BW=2757KiB/s (2823kB/s)(323MiB/120015msec)
- slat (nsec): min=0, max=8114.2k, avg=169031.29, stdev=274467.99
- clat (nsec): min=0, max=1041.7M, avg=80800586.43, stdev=146966844.61
- lat (nsec): min=0, max=1041.8M, avg=80970787.08, stdev=146927156.52
- clat percentiles (usec):
- | 1.00th=[ 1057], 5.00th=[ 1270], 10.00th=[ 1483],
- | 20.00th=[ 1926], 30.00th=[ 2671], 40.00th=[ 4047],
- | 50.00th=[ 8586], 60.00th=[ 24773], 70.00th=[ 65799],
- | 80.00th=[ 111674], 90.00th=[ 295699], 95.00th=[ 455082],
- | 99.00th=[ 641729], 99.50th=[ 700449], 99.90th=[ 851444],
- | 99.95th=[ 868221], 99.99th=[1044382]
- bw ( KiB/s): min= 0, max= 4086, per=54.10%, avg=2755.50, stdev=1881.61, samples=2
- iops : min= 0, max= 63, avg=42.50, stdev=28.99, samples=2
- lat (nsec) : 750=6.03%, 1000=11.10%
- lat (usec) : 2=15.61%, 4=0.11%, 10=0.19%, 20=0.82%, 50=0.69%
- lat (usec) : 100=0.59%, 250=0.89%, 500=0.49%, 750=0.73%, 1000=1.93%
- lat (msec) : 2=12.85%, 4=11.24%, 10=6.76%, 20=3.80%, 50=6.10%
- lat (msec) : 100=6.97%, 250=6.51%, 500=4.52%, 750=1.90%, 1000=0.18%
- cpu : usr=0.07%, sys=0.23%, ctx=12113, majf=0, minf=33
- IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=4978,5168,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- Run status group 0 (all jobs):
- READ: bw=54.4MiB/s (57.1MB/s), 2655KiB/s-51.8MiB/s (2719kB/s-54.4MB/s), io=6534MiB (6852MB), run=120015-120042msec
- WRITE: bw=5094KiB/s (5216kB/s), 2346KiB/s-2757KiB/s (2402kB/s-2823kB/s), io=599MiB (628MB), run=120015-120414msec
- Disk stats (read/write):
- rbd291: ios=6612/9793, merge=0/1, ticks=541800/872212, in_queue=1405759, util=7.93%
- <example-benchmark-fio-server-139-benchmark-7btww>
- randwrite: (groupid=0, jobs=1): err= 0: pid=11: Wed May 8 20:34:58 2019
- write: IOPS=21, BW=1355KiB/s (1388kB/s)(160MiB/120980msec)
- slat (nsec): min=0, max=859342k, avg=1241964.81, stdev=25538249.96
- clat (nsec): min=0, max=2242.1M, avg=188335014.13, stdev=370328253.41
- lat (nsec): min=0, max=2242.8M, avg=189577480.82, stdev=371271934.52
- clat percentiles (usec):
- | 1.00th=[ 1090], 5.00th=[ 1434], 10.00th=[ 1778],
- | 20.00th=[ 2474], 30.00th=[ 3326], 40.00th=[ 4752],
- | 50.00th=[ 7242], 60.00th=[ 13566], 70.00th=[ 27657],
- | 80.00th=[ 392168], 90.00th=[ 817890], 95.00th=[1035994],
- | 99.00th=[1484784], 99.50th=[1635779], 99.90th=[2139096],
- | 99.95th=[2139096], 99.99th=[2231370]
- bw ( KiB/s): min= 0, max= 1415, per=29.91%, avg=1358.50, stdev=79.90, samples=2
- iops : min= 0, max= 22, avg=21.00, stdev= 1.41, samples=2
- lat (usec) : 10=0.04%, 500=0.04%, 750=0.08%, 1000=0.31%
- lat (msec) : 2=12.90%, 4=21.81%, 10=20.75%, 20=9.73%, 50=8.75%
- lat (msec) : 100=1.99%, 250=1.80%, 500=4.85%, 750=5.31%, 1000=5.90%
- cpu : usr=0.02%, sys=0.11%, ctx=4636, majf=0, minf=30
- IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=0,2559,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- read: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:34:58 2019
- read: IOPS=376, BW=23.5MiB/s (24.7MB/s)(2837MiB/120564msec)
- slat (nsec): min=0, max=691432k, avg=43024.80, stdev=3613085.44
- clat (nsec): min=0, max=2143.1M, avg=10554204.56, stdev=88854248.08
- lat (nsec): min=0, max=2143.2M, avg=10598187.22, stdev=88937193.66
- clat percentiles (nsec):
- | 1.00th=[ 716], 5.00th=[ 724], 10.00th=[ 748],
- | 20.00th=[ 780], 30.00th=[ 796], 40.00th=[ 828],
- | 50.00th=[ 860], 60.00th=[ 892], 70.00th=[ 932],
- | 80.00th=[ 1144], 90.00th=[ 13632], 95.00th=[ 2342912],
- | 99.00th=[ 541065216], 99.50th=[ 759169024], 99.90th=[1052770304],
- | 99.95th=[1266679808], 99.99th=[1602224128]
- bw ( KiB/s): min= 0, max=34499, per=89.06%, avg=24203.50, stdev=14560.04, samples=2
- iops : min= 0, max= 539, avg=378.00, stdev=227.69, samples=2
- lat (nsec) : 750=10.71%, 1000=65.67%
- lat (usec) : 2=10.08%, 4=0.17%, 10=0.54%, 20=3.27%, 50=1.05%
- lat (usec) : 100=0.19%, 250=0.11%, 500=0.19%, 750=0.19%, 1000=0.72%
- lat (msec) : 2=1.76%, 4=1.56%, 10=1.17%, 20=0.52%, 50=0.42%
- lat (msec) : 100=0.13%, 250=0.16%, 500=0.33%, 750=0.55%, 1000=0.37%
- cpu : usr=0.10%, sys=0.42%, ctx=7607, majf=0, minf=30
- IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=45440,0,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- readwrite: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:34:58 2019
- read: IOPS=48, BW=3100KiB/s (3175kB/s)(363MiB/120004msec)
- slat (nsec): min=0, max=3502.5k, avg=30685.11, stdev=91394.21
- clat (nsec): min=0, max=886323k, avg=8285032.64, stdev=51086846.62
- lat (nsec): min=0, max=886388k, avg=8316623.23, stdev=51091851.38
- clat percentiles (nsec):
- | 1.00th=[ 716], 5.00th=[ 724], 10.00th=[ 732],
- | 20.00th=[ 764], 30.00th=[ 876], 40.00th=[ 996],
- | 50.00th=[ 1144], 60.00th=[ 1288], 70.00th=[ 1672],
- | 80.00th=[ 179200], 90.00th=[ 1974272], 95.00th=[ 14221312],
- | 99.00th=[320864256], 99.50th=[438304768], 99.90th=[583008256],
- | 99.95th=[658505728], 99.99th=[884998144]
- bw ( KiB/s): min= 0, max= 4320, per=11.41%, avg=3100.00, stdev=1725.34, samples=2
- iops : min= 0, max= 67, avg=48.00, stdev=26.87, samples=2
- write: IOPS=50, BW=3213KiB/s (3290kB/s)(377MiB/120004msec)
- slat (nsec): min=0, max=13105k, avg=189168.32, stdev=269636.09
- clat (nsec): min=0, max=1091.8M, avg=71891687.37, stdev=135619142.37
- lat (nsec): min=0, max=1091.9M, avg=72082105.36, stdev=135568803.97
- clat percentiles (usec):
- | 1.00th=[ 1020], 5.00th=[ 1221], 10.00th=[ 1418],
- | 20.00th=[ 1827], 30.00th=[ 2442], 40.00th=[ 3818],
- | 50.00th=[ 7832], 60.00th=[ 21103], 70.00th=[ 68682],
- | 80.00th=[ 107480], 90.00th=[ 200279], 95.00th=[ 392168],
- | 99.00th=[ 658506], 99.50th=[ 717226], 99.90th=[ 859833],
- | 99.95th=[ 926942], 99.99th=[1098908]
- bw ( KiB/s): min= 0, max= 4449, per=70.71%, avg=3211.50, stdev=1750.09, samples=2
- iops : min= 0, max= 69, avg=49.50, stdev=27.58, samples=2
- lat (nsec) : 750=8.45%, 1000=11.48%
- lat (usec) : 2=14.69%, 4=0.07%, 10=0.48%, 20=1.30%, 50=0.89%
- lat (usec) : 100=1.14%, 250=1.15%, 500=0.68%, 750=0.79%, 1000=1.64%
- lat (msec) : 2=13.27%, 4=10.32%, 10=6.92%, 20=3.79%, 50=4.01%
- lat (msec) : 100=6.76%, 250=7.48%, 500=3.02%, 750=1.48%, 1000=0.19%
- cpu : usr=0.08%, sys=0.28%, ctx=12857, majf=0, minf=34
- IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=5813,6022,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- Run status group 0 (all jobs):
- READ: bw=26.5MiB/s (27.8MB/s), 3100KiB/s-23.5MiB/s (3175kB/s-24.7MB/s), io=3200MiB (3355MB), run=120004-120564msec
- WRITE: bw=4543KiB/s (4652kB/s), 1355KiB/s-3213KiB/s (1388kB/s-3290kB/s), io=537MiB (563MB), run=120004-120980msec
- Disk stats (read/write):
- rbd58: ios=5113/8988, merge=0/1, ticks=567154/914911, in_queue=1474958, util=6.12%
- <example-benchmark-fio-server-289-benchmark-jhcgn>
- randwrite: (groupid=0, jobs=1): err= 0: pid=11: Wed May 8 20:34:58 2019
- write: IOPS=44, BW=2824KiB/s (2892kB/s)(331MiB/120072msec)
- slat (nsec): min=0, max=714127k, avg=491923.63, stdev=13745623.96
- clat (nsec): min=0, max=1605.9M, avg=90397146.42, stdev=161216053.44
- lat (nsec): min=0, max=1766.4M, avg=90890178.16, stdev=162453244.77
- clat percentiles (usec):
- | 1.00th=[ 938], 5.00th=[ 1123], 10.00th=[ 1270],
- | 20.00th=[ 1614], 30.00th=[ 2114], 40.00th=[ 3064],
- | 50.00th=[ 7046], 60.00th=[ 62129], 70.00th=[ 94897],
- | 80.00th=[ 135267], 90.00th=[ 254804], 95.00th=[ 438305],
- | 99.00th=[ 759170], 99.50th=[ 943719], 99.90th=[1132463],
- | 99.95th=[1149240], 99.99th=[1602225]
- bw ( KiB/s): min= 0, max= 3586, per=45.15%, avg=2823.00, stdev=1079.04, samples=2
- iops : min= 0, max= 56, avg=44.00, stdev=16.97, samples=2
- lat (usec) : 250=0.02%, 500=0.02%, 750=0.09%, 1000=2.00%
- lat (msec) : 2=26.38%, 4=15.98%, 10=7.84%, 20=2.72%, 50=3.36%
- lat (msec) : 100=13.54%, 250=17.79%, 500=6.35%, 750=2.85%, 1000=0.64%
- cpu : usr=0.07%, sys=0.18%, ctx=9958, majf=0, minf=26
- IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=0,5295,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- read: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:34:58 2019
- read: IOPS=311, BW=19.5MiB/s (20.4MB/s)(2338MiB/120002msec)
- slat (nsec): min=0, max=710630k, avg=63218.86, stdev=4979859.96
- clat (nsec): min=0, max=1434.2M, avg=12791436.07, stdev=62473523.58
- lat (nsec): min=0, max=1434.4M, avg=12855512.68, stdev=62676595.96
- clat percentiles (nsec):
- | 1.00th=[ 708], 5.00th=[ 716], 10.00th=[ 716],
- | 20.00th=[ 724], 30.00th=[ 732], 40.00th=[ 748],
- | 50.00th=[ 820], 60.00th=[ 876], 70.00th=[ 1032],
- | 80.00th=[ 8768], 90.00th=[ 7045120], 95.00th=[ 60030976],
- | 99.00th=[ 362807296], 99.50th=[ 484442112], 99.90th=[ 683671552],
- | 99.95th=[ 767557632], 99.99th=[1019215872]
- bw ( KiB/s): min= 0, max=25118, per=85.72%, avg=19949.00, stdev=7310.07, samples=2
- iops : min= 0, max= 392, avg=311.00, stdev=114.55, samples=2
- lat (nsec) : 750=41.73%, 1000=27.46%
- lat (usec) : 2=10.55%, 4=0.03%, 10=0.58%, 20=0.31%, 50=0.27%
- lat (usec) : 100=0.09%, 250=0.07%, 500=0.27%, 750=0.98%, 1000=2.19%
- lat (msec) : 2=3.06%, 4=1.82%, 10=0.83%, 20=0.52%, 50=3.20%
- lat (msec) : 100=2.93%, 250=1.53%, 500=1.11%, 750=0.40%, 1000=0.05%
- cpu : usr=0.11%, sys=0.35%, ctx=14140, majf=0, minf=97
- IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=37428,0,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- readwrite: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:34:58 2019
- read: IOPS=51, BW=3322KiB/s (3402kB/s)(389MiB/120007msec)
- slat (nsec): min=0, max=663988k, avg=171583.65, stdev=8737229.14
- clat (nsec): min=0, max=632295k, avg=6775755.36, stdev=37927924.18
- lat (nsec): min=0, max=664065k, avg=6948327.42, stdev=38902864.79
- clat percentiles (nsec):
- | 1.00th=[ 716], 5.00th=[ 724], 10.00th=[ 740],
- | 20.00th=[ 796], 30.00th=[ 892], 40.00th=[ 1004],
- | 50.00th=[ 1160], 60.00th=[ 1304], 70.00th=[ 17792],
- | 80.00th=[ 864256], 90.00th=[ 3031040], 95.00th=[ 33816576],
- | 99.00th=[179306496], 99.50th=[308281344], 99.90th=[492830720],
- | 99.95th=[574619648], 99.99th=[633339904]
- bw ( KiB/s): min= 0, max= 4763, per=14.27%, avg=3322.00, stdev=2037.88, samples=2
- iops : min= 0, max= 74, avg=51.50, stdev=31.82, samples=2
- write: IOPS=53, BW=3430KiB/s (3513kB/s)(402MiB/120007msec)
- slat (nsec): min=0, max=11878k, avg=197961.98, stdev=307898.64
- clat (nsec): min=0, max=1469.7M, avg=68202610.29, stdev=138950257.20
- lat (nsec): min=0, max=1469.8M, avg=68401706.46, stdev=138903708.14
- clat percentiles (usec):
- | 1.00th=[ 996], 5.00th=[ 1221], 10.00th=[ 1401],
- | 20.00th=[ 1762], 30.00th=[ 2212], 40.00th=[ 2999],
- | 50.00th=[ 4621], 60.00th=[ 12387], 70.00th=[ 57410],
- | 80.00th=[ 103285], 90.00th=[ 200279], 95.00th=[ 354419],
- | 99.00th=[ 683672], 99.50th=[ 801113], 99.90th=[1098908],
- | 99.95th=[1132463], 99.99th=[1468007]
- bw ( KiB/s): min= 0, max= 4867, per=54.84%, avg=3428.50, stdev=2034.35, samples=2
- iops : min= 0, max= 76, avg=53.50, stdev=31.82, samples=2
- lat (nsec) : 750=7.08%, 1000=12.30%
- lat (usec) : 2=14.32%, 4=0.03%, 10=0.24%, 20=0.81%, 50=0.70%
- lat (usec) : 100=0.59%, 250=0.86%, 500=0.89%, 750=0.79%, 1000=2.20%
- lat (msec) : 2=15.49%, 4=12.24%, 10=7.09%, 20=2.65%, 50=4.11%
- lat (msec) : 100=6.31%, 250=7.16%, 500=2.66%, 750=1.15%, 1000=0.28%
- cpu : usr=0.07%, sys=0.31%, ctx=14996, majf=0, minf=33
- IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=6231,6429,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- Run status group 0 (all jobs):
- READ: bw=22.7MiB/s (23.8MB/s), 3322KiB/s-19.5MiB/s (3402kB/s-20.4MB/s), io=2728MiB (2860MB), run=120002-120007msec
- WRITE: bw=6252KiB/s (6402kB/s), 2824KiB/s-3430KiB/s (2892kB/s-3513kB/s), io=733MiB (769MB), run=120007-120072msec
- Disk stats (read/write):
- rbd123: ios=9289/12253, merge=0/2, ticks=557195/852287, in_queue=1398563, util=14.81%
- <example-benchmark-fio-server-115-benchmark-krrp7>
- randwrite: (groupid=0, jobs=1): err= 0: pid=11: Wed May 8 20:34:59 2019
- write: IOPS=26, BW=1687KiB/s (1727kB/s)(198MiB/120401msec)
- slat (nsec): min=0, max=464962k, avg=535888.93, stdev=11242241.37
- clat (nsec): min=0, max=2030.7M, avg=151464747.66, stdev=258316057.55
- lat (nsec): min=0, max=2030.8M, avg=152001879.93, stdev=258431932.77
- clat percentiles (usec):
- | 1.00th=[ 1270], 5.00th=[ 1942], 10.00th=[ 2573],
- | 20.00th=[ 3851], 30.00th=[ 5473], 40.00th=[ 7832],
- | 50.00th=[ 11207], 60.00th=[ 20055], 70.00th=[ 122160],
- | 80.00th=[ 329253], 90.00th=[ 549454], 95.00th=[ 700449],
- | 99.00th=[ 943719], 99.50th=[1182794], 99.90th=[2021655],
- | 99.95th=[2021655], 99.99th=[2038432]
- bw ( KiB/s): min= 0, max= 1993, per=31.33%, avg=1688.50, stdev=430.63, samples=2
- iops : min= 0, max= 31, avg=26.00, stdev= 7.07, samples=2
- lat (usec) : 4=0.13%, 250=0.03%, 500=0.03%, 750=0.06%, 1000=0.06%
- lat (msec) : 2=5.21%, 4=15.36%, 10=26.12%, 20=13.19%, 50=6.50%
- lat (msec) : 100=2.62%, 250=6.18%, 500=13.15%, 750=7.48%, 1000=3.19%
- cpu : usr=0.03%, sys=0.12%, ctx=5496, majf=0, minf=30
- IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=0,3170,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- read: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:34:59 2019
- read: IOPS=123, BW=7920KiB/s (8110kB/s)(933MiB/120639msec)
- slat (nsec): min=0, max=7129.5k, avg=37269.24, stdev=139259.60
- clat (nsec): min=0, max=1959.1M, avg=32268522.52, stdev=150613641.83
- lat (nsec): min=0, max=1959.1M, avg=32306809.86, stdev=150630782.48
- clat percentiles (nsec):
- | 1.00th=[ 716], 5.00th=[ 724], 10.00th=[ 748],
- | 20.00th=[ 788], 30.00th=[ 820], 40.00th=[ 868],
- | 50.00th=[ 908], 60.00th=[ 1080], 70.00th=[ 1368],
- | 80.00th=[ 1138688], 90.00th=[ 4882432], 95.00th=[ 56885248],
- | 99.00th=[ 843055104], 99.50th=[ 960495616], 99.90th=[1216348160],
- | 99.95th=[1400897536], 99.99th=[1803550720]
- bw ( KiB/s): min= 0, max= 8521, per=68.97%, avg=7955.50, stdev=799.74, samples=2
- iops : min= 0, max= 133, avg=124.00, stdev=12.73, samples=2
- lat (nsec) : 750=10.00%, 1000=47.62%
- lat (usec) : 2=17.35%, 4=0.25%, 10=0.15%, 20=0.74%, 50=0.16%
- lat (usec) : 100=0.14%, 250=0.13%, 500=0.11%, 750=0.44%, 1000=1.89%
- lat (msec) : 2=5.43%, 4=4.56%, 10=3.44%, 20=1.44%, 50=1.05%
- lat (msec) : 100=0.33%, 250=0.31%, 500=1.02%, 750=1.94%, 1000=1.11%
- cpu : usr=0.04%, sys=0.18%, ctx=7043, majf=0, minf=30
- IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=14926,0,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- readwrite: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:34:59 2019
- read: IOPS=56, BW=3635KiB/s (3722kB/s)(426MiB/120006msec)
- slat (nsec): min=0, max=1680.6k, avg=30012.19, stdev=72067.62
- clat (nsec): min=0, max=747366k, avg=10032263.04, stdev=54279230.59
- lat (nsec): min=0, max=747386k, avg=10063205.13, stdev=54284115.55
- clat percentiles (nsec):
- | 1.00th=[ 716], 5.00th=[ 724], 10.00th=[ 732],
- | 20.00th=[ 796], 30.00th=[ 892], 40.00th=[ 1012],
- | 50.00th=[ 1144], 60.00th=[ 1304], 70.00th=[ 15808],
- | 80.00th=[ 921600], 90.00th=[ 3522560], 95.00th=[ 45350912],
- | 99.00th=[329252864], 99.50th=[442499072], 99.90th=[675282944],
- | 99.95th=[700448768], 99.99th=[750780416]
- bw ( KiB/s): min= 0, max= 5302, per=31.51%, avg=3635.00, stdev=2357.49, samples=2
- iops : min= 0, max= 82, avg=56.00, stdev=36.77, samples=2
- write: IOPS=58, BW=3715KiB/s (3804kB/s)(435MiB/120006msec)
- slat (nsec): min=0, max=29166k, avg=196391.82, stdev=440398.02
- clat (nsec): min=0, max=905267k, avg=58898123.51, stdev=124023895.80
- lat (nsec): min=0, max=906784k, avg=59095996.36, stdev=123976481.96
- clat percentiles (usec):
- | 1.00th=[ 979], 5.00th=[ 1172], 10.00th=[ 1352], 20.00th=[ 1680],
- | 30.00th=[ 2114], 40.00th=[ 2868], 50.00th=[ 4686], 60.00th=[ 10421],
- | 70.00th=[ 32113], 80.00th=[ 93848], 90.00th=[160433], 95.00th=[320865],
- | 99.00th=[675283], 99.50th=[767558], 99.90th=[868221], 99.95th=[893387],
- | 99.99th=[901776]
- bw ( KiB/s): min= 0, max= 5360, per=68.90%, avg=3713.00, stdev=2329.21, samples=2
- iops : min= 0, max= 83, avg=57.50, stdev=36.06, samples=2
- lat (nsec) : 750=7.63%, 1000=11.81%
- lat (usec) : 2=14.57%, 4=0.04%, 10=0.25%, 20=0.73%, 50=0.65%
- lat (usec) : 100=0.67%, 250=0.86%, 500=0.48%, 750=0.68%, 1000=2.32%
- lat (msec) : 2=16.48%, 4=11.47%, 10=6.95%, 20=4.01%, 50=4.48%
- lat (msec) : 100=5.18%, 250=6.82%, 500=2.74%, 750=0.91%, 1000=0.28%
- cpu : usr=0.08%, sys=0.33%, ctx=15897, majf=0, minf=30
- IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=6816,6963,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- Run status group 0 (all jobs):
- READ: bw=11.3MiB/s (11.8MB/s), 3635KiB/s-7920KiB/s (3722kB/s-8110kB/s), io=1359MiB (1425MB), run=120006-120639msec
- WRITE: bw=5389KiB/s (5519kB/s), 1687KiB/s-3715KiB/s (1727kB/s-3804kB/s), io=634MiB (664MB), run=120006-120401msec
- Disk stats (read/write):
- rbd33: ios=5721/11388, merge=0/3, ticks=605434/883833, in_queue=1480636, util=5.12%
- <example-benchmark-fio-server-594-benchmark-dz8fd>
- randwrite: (groupid=0, jobs=1): err= 0: pid=11: Wed May 8 20:34:59 2019
- write: IOPS=39, BW=2556KiB/s (2617kB/s)(300MiB/120181msec)
- slat (nsec): min=0, max=746936k, avg=484272.82, stdev=14806221.57
- clat (nsec): min=0, max=1378.3M, avg=100241276.02, stdev=175257826.40
- lat (nsec): min=0, max=1378.4M, avg=100726680.74, stdev=175669845.78
- clat percentiles (usec):
- | 1.00th=[ 996], 5.00th=[ 1237], 10.00th=[ 1450],
- | 20.00th=[ 1926], 30.00th=[ 2868], 40.00th=[ 4883],
- | 50.00th=[ 11076], 60.00th=[ 55837], 70.00th=[ 99091],
- | 80.00th=[ 139461], 90.00th=[ 337642], 95.00th=[ 484443],
- | 99.00th=[ 809501], 99.50th=[ 952108], 99.90th=[1283458],
- | 99.95th=[1317012], 99.99th=[1384121]
- bw ( KiB/s): min= 0, max= 2893, per=46.14%, avg=2557.00, stdev=475.18, samples=2
- iops : min= 0, max= 45, avg=39.50, stdev= 7.78, samples=2
- lat (usec) : 100=0.02%, 500=0.02%, 750=0.04%, 1000=0.92%
- lat (msec) : 2=20.20%, 4=15.22%, 10=12.42%, 20=5.88%, 50=4.48%
- lat (msec) : 100=11.03%, 250=16.76%, 500=8.46%, 750=3.25%, 1000=0.96%
- cpu : usr=0.05%, sys=0.17%, ctx=8988, majf=0, minf=30
- IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=0,4797,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- read: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:34:59 2019
- read: IOPS=336, BW=21.0MiB/s (22.0MB/s)(2525MiB/120182msec)
- slat (nsec): min=0, max=688109k, avg=55153.02, stdev=4613265.28
- clat (nsec): min=0, max=1316.2M, avg=11849012.02, stdev=62320874.61
- lat (nsec): min=0, max=1316.2M, avg=11905146.43, stdev=62497607.01
- clat percentiles (nsec):
- | 1.00th=[ 708], 5.00th=[ 716], 10.00th=[ 716],
- | 20.00th=[ 724], 30.00th=[ 732], 40.00th=[ 756],
- | 50.00th=[ 844], 60.00th=[ 908], 70.00th=[ 1112],
- | 80.00th=[ 354304], 90.00th=[ 2998272], 95.00th=[ 55312384],
- | 99.00th=[ 358612992], 99.50th=[ 471859200], 99.90th=[ 725614592],
- | 99.95th=[ 859832320], 99.99th=[1199570944]
- bw ( KiB/s): min= 0, max=31963, per=88.33%, avg=21542.00, stdev=14737.52, samples=2
- iops : min= 0, max= 499, avg=336.00, stdev=230.52, samples=2
- lat (nsec) : 750=38.94%, 1000=27.33%
- lat (usec) : 2=11.77%, 4=0.04%, 10=0.52%, 20=0.59%, 50=0.19%
- lat (usec) : 100=0.11%, 250=0.08%, 500=0.87%, 750=1.40%, 1000=2.68%
- lat (msec) : 2=3.81%, 4=2.41%, 10=1.09%, 20=0.57%, 50=2.13%
- lat (msec) : 100=2.62%, 250=1.33%, 500=1.11%, 750=0.33%, 1000=0.07%
- cpu : usr=0.11%, sys=0.40%, ctx=16448, majf=0, minf=108
- IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=40399,0,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- readwrite: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:34:59 2019
- read: IOPS=45, BW=2881KiB/s (2950kB/s)(338MiB/120007msec)
- slat (nsec): min=0, max=3273.4k, avg=38406.71, stdev=105301.68
- clat (nsec): min=0, max=585036k, avg=9336840.69, stdev=43652742.88
- lat (nsec): min=0, max=585053k, avg=9376204.46, stdev=43661535.41
- clat percentiles (nsec):
- | 1.00th=[ 716], 5.00th=[ 732], 10.00th=[ 740],
- | 20.00th=[ 820], 30.00th=[ 924], 40.00th=[ 1064],
- | 50.00th=[ 1208], 60.00th=[ 1400], 70.00th=[ 209920],
- | 80.00th=[ 1155072], 90.00th=[ 6782976], 95.00th=[ 46399488],
- | 99.00th=[244318208], 99.50th=[350224384], 99.90th=[509607936],
- | 99.95th=[541065216], 99.99th=[583008256]
- bw ( KiB/s): min= 0, max= 4260, per=11.81%, avg=2879.50, stdev=1952.32, samples=2
- iops : min= 0, max= 66, avg=44.50, stdev=30.41, samples=2
- write: IOPS=46, BW=2991KiB/s (3063kB/s)(351MiB/120007msec)
- slat (nsec): min=0, max=57340k, avg=188996.30, stdev=811463.73
- clat (nsec): min=0, max=1109.1M, avg=76748413.93, stdev=143499222.92
- lat (nsec): min=0, max=1109.1M, avg=76938515.68, stdev=143449607.79
- clat percentiles (usec):
- | 1.00th=[ 1074], 5.00th=[ 1303], 10.00th=[ 1532],
- | 20.00th=[ 2024], 30.00th=[ 2900], 40.00th=[ 4817],
- | 50.00th=[ 10814], 60.00th=[ 26346], 70.00th=[ 67634],
- | 80.00th=[ 107480], 90.00th=[ 235930], 95.00th=[ 425722],
- | 99.00th=[ 692061], 99.50th=[ 759170], 99.90th=[ 943719],
- | 99.95th=[ 985662], 99.99th=[1115685]
- bw ( KiB/s): min= 0, max= 4384, per=53.94%, avg=2989.50, stdev=1972.12, samples=2
- iops : min= 0, max= 68, avg=46.00, stdev=31.11, samples=2
- lat (nsec) : 750=6.66%, 1000=10.83%
- lat (usec) : 2=14.09%, 4=0.10%, 10=0.29%, 20=0.60%, 50=0.65%
- lat (usec) : 100=0.66%, 250=0.60%, 500=1.10%, 750=1.01%, 1000=2.00%
- lat (msec) : 2=12.96%, 4=10.63%, 10=7.37%, 20=4.72%, 50=6.18%
- lat (msec) : 100=7.20%, 250=7.00%, 500=3.70%, 750=1.34%, 1000=0.31%
- cpu : usr=0.06%, sys=0.28%, ctx=13512, majf=0, minf=34
- IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=5402,5605,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- Run status group 0 (all jobs):
- READ: bw=23.8MiB/s (24.0MB/s), 2881KiB/s-21.0MiB/s (2950kB/s-22.0MB/s), io=2862MiB (3001MB), run=120007-120182msec
- WRITE: bw=5543KiB/s (5676kB/s), 2556KiB/s-2991KiB/s (2617kB/s-3063kB/s), io=651MiB (682MB), run=120007-120181msec
- Disk stats (read/write):
- rbd248: ios=11875/11980, merge=0/2, ticks=556245/857749, in_queue=1401822, util=15.70%
- <example-benchmark-fio-server-130-benchmark-ckmtx>
- randwrite: (groupid=0, jobs=1): err= 0: pid=11: Wed May 8 20:34:59 2019
- write: IOPS=51, BW=3328KiB/s (3408kB/s)(390MiB/120061msec)
- slat (nsec): min=0, max=657788k, avg=347116.30, stdev=8981861.13
- clat (nsec): min=0, max=1140.8M, avg=76624588.97, stdev=145436001.13
- lat (nsec): min=0, max=1140.8M, avg=76973399.87, stdev=145639513.75
- clat percentiles (usec):
- | 1.00th=[ 955], 5.00th=[ 1156], 10.00th=[ 1319],
- | 20.00th=[ 1696], 30.00th=[ 2376], 40.00th=[ 3589],
- | 50.00th=[ 7177], 60.00th=[ 24773], 70.00th=[ 71828],
- | 80.00th=[ 108528], 90.00th=[ 233833], 95.00th=[ 400557],
- | 99.00th=[ 725615], 99.50th=[ 843056], 99.90th=[ 952108],
- | 99.95th=[1002439], 99.99th=[1132463]
- bw ( KiB/s): min= 0, max= 3477, per=50.05%, avg=3327.50, stdev=211.42, samples=2
- iops : min= 0, max= 54, avg=51.50, stdev= 3.54, samples=2
- lat (usec) : 2=0.02%, 4=0.02%, 500=0.03%, 750=0.05%, 1000=1.47%
- lat (msec) : 2=23.52%, 4=17.03%, 10=11.04%, 20=5.26%, 50=6.68%
- lat (msec) : 100=12.39%, 250=13.27%, 500=6.18%, 750=2.20%, 1000=0.87%
- cpu : usr=0.07%, sys=0.22%, ctx=11790, majf=0, minf=26
- IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=0,6241,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- read: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:34:59 2019
- read: IOPS=338, BW=21.1MiB/s (22.1MB/s)(2533MiB/120041msec)
- slat (nsec): min=0, max=520529k, avg=41330.26, stdev=2721988.27
- clat (nsec): min=0, max=1107.1M, avg=11867312.79, stdev=59671897.16
- lat (nsec): min=0, max=1107.1M, avg=11909494.08, stdev=59741193.49
- clat percentiles (nsec):
- | 1.00th=[ 708], 5.00th=[ 708], 10.00th=[ 716],
- | 20.00th=[ 724], 30.00th=[ 732], 40.00th=[ 748],
- | 50.00th=[ 836], 60.00th=[ 892], 70.00th=[ 1096],
- | 80.00th=[ 24960], 90.00th=[ 5210112], 95.00th=[ 56360960],
- | 99.00th=[ 337641472], 99.50th=[ 471859200], 99.90th=[ 700448768],
- | 99.95th=[ 759169024], 99.99th=[1061158912]
- bw ( KiB/s): min= 0, max=31047, per=87.08%, avg=21615.00, stdev=13338.86, samples=2
- iops : min= 0, max= 485, avg=337.50, stdev=208.60, samples=2
- lat (nsec) : 750=41.36%, 1000=26.06%
- lat (usec) : 2=10.59%, 4=0.04%, 10=0.87%, 20=1.02%, 50=0.41%
- lat (usec) : 100=0.08%, 250=0.04%, 500=0.19%, 750=1.10%, 1000=2.41%
- lat (msec) : 2=3.42%, 4=2.02%, 10=0.98%, 20=0.62%, 50=3.06%
- lat (msec) : 100=2.91%, 250=1.43%, 500=0.96%, 750=0.36%, 1000=0.05%
- cpu : usr=0.11%, sys=0.39%, ctx=15868, majf=0, minf=30
- IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=40576,0,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- readwrite: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:34:59 2019
- read: IOPS=50, BW=3214KiB/s (3291kB/s)(377MiB/120006msec)
- slat (nsec): min=0, max=657744k, avg=145579.23, stdev=8474050.87
- clat (nsec): min=0, max=1066.6M, avg=8382053.59, stdev=40166155.39
- lat (nsec): min=0, max=1066.6M, avg=8528658.07, stdev=41033365.30
- clat percentiles (nsec):
- | 1.00th=[ 716], 5.00th=[ 732], 10.00th=[ 740],
- | 20.00th=[ 804], 30.00th=[ 908], 40.00th=[ 1048],
- | 50.00th=[ 1176], 60.00th=[ 1320], 70.00th=[ 18560],
- | 80.00th=[ 1003520], 90.00th=[ 7897088], 95.00th=[ 48496640],
- | 99.00th=[ 183500800], 99.50th=[ 274726912], 99.90th=[ 484442112],
- | 99.95th=[ 517996544], 99.99th=[1069547520]
- bw ( KiB/s): min= 0, max= 4074, per=12.95%, avg=3214.00, stdev=1216.22, samples=2
- iops : min= 0, max= 63, avg=49.50, stdev=19.09, samples=2
- write: IOPS=51, BW=3323KiB/s (3403kB/s)(389MiB/120006msec)
- slat (nsec): min=0, max=63599k, avg=206804.07, stdev=945686.55
- clat (nsec): min=0, max=1233.8M, avg=68800246.38, stdev=134987032.75
- lat (nsec): min=0, max=1233.9M, avg=69008286.97, stdev=134933795.73
- clat percentiles (usec):
- | 1.00th=[ 996], 5.00th=[ 1221], 10.00th=[ 1434],
- | 20.00th=[ 1942], 30.00th=[ 2769], 40.00th=[ 4490],
- | 50.00th=[ 8717], 60.00th=[ 21890], 70.00th=[ 54264],
- | 80.00th=[ 96994], 90.00th=[ 204473], 95.00th=[ 375391],
- | 99.00th=[ 650118], 99.50th=[ 775947], 99.90th=[ 943719],
- | 99.95th=[ 952108], 99.99th=[1233126]
- bw ( KiB/s): min= 0, max= 4184, per=49.95%, avg=3321.00, stdev=1220.47, samples=2
- iops : min= 0, max= 65, avg=51.50, stdev=19.09, samples=2
- lat (nsec) : 750=6.62%, 1000=11.64%
- lat (usec) : 2=15.40%, 4=0.07%, 10=0.26%, 20=0.58%, 50=0.62%
- lat (usec) : 100=0.55%, 250=0.57%, 500=0.71%, 750=0.72%, 1000=2.13%
- lat (msec) : 2=12.83%, 4=10.11%, 10=7.98%, 20=4.37%, 50=6.58%
- lat (msec) : 100=7.44%, 250=6.36%, 500=3.20%, 750=0.96%, 1000=0.32%
- cpu : usr=0.09%, sys=0.29%, ctx=14868, majf=0, minf=34
- IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=6028,6228,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- Run status group 0 (all jobs):
- READ: bw=24.2MiB/s (25.4MB/s), 3214KiB/s-21.1MiB/s (3291kB/s-22.1MB/s), io=2910MiB (3051MB), run=120006-120041msec
- WRITE: bw=6650KiB/s (6810kB/s), 3323KiB/s-3328KiB/s (3403kB/s-3408kB/s), io=780MiB (818MB), run=120006-120061msec
- Disk stats (read/write):
- rbd49: ios=9506/12982, merge=0/1, ticks=545666/834922, in_queue=1369109, util=15.51%
- <example-benchmark-fio-server-600-benchmark-fvxhx>
- randwrite: (groupid=0, jobs=1): err= 0: pid=11: Wed May 8 20:34:59 2019
- write: IOPS=19, BW=1236KiB/s (1265kB/s)(145MiB/120516msec)
- slat (nsec): min=0, max=694928k, avg=1293864.89, stdev=25365740.43
- clat (nsec): min=0, max=2243.2M, avg=206617078.50, stdev=402630615.34
- lat (nsec): min=0, max=2243.3M, avg=207910976.65, stdev=403420903.13
- clat percentiles (usec):
- | 1.00th=[ 1106], 5.00th=[ 1418], 10.00th=[ 1647],
- | 20.00th=[ 2180], 30.00th=[ 2802], 40.00th=[ 3556],
- | 50.00th=[ 4817], 60.00th=[ 7832], 70.00th=[ 21103],
- | 80.00th=[ 467665], 90.00th=[ 884999], 95.00th=[1199571],
- | 99.00th=[1501561], 99.50th=[1635779], 99.90th=[1920992],
- | 99.95th=[2139096], 99.99th=[2231370]
- bw ( KiB/s): min= 0, max= 1654, per=26.83%, avg=1238.50, stdev=587.61, samples=2
- iops : min= 0, max= 25, avg=18.50, stdev= 9.19, samples=2
- lat (usec) : 750=0.09%, 1000=0.17%
- lat (msec) : 2=16.44%, 4=27.54%, 10=18.76%, 20=6.45%, 50=5.16%
- lat (msec) : 100=1.42%, 250=1.98%, 500=2.41%, 750=6.88%, 1000=5.59%
- cpu : usr=0.02%, sys=0.09%, ctx=4282, majf=0, minf=29
- IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=0,2324,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- read: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:34:59 2019
- read: IOPS=685, BW=42.7MiB/s (44.8MB/s)(5151MiB/120619msec)
- slat (nsec): min=0, max=645074k, avg=32638.70, stdev=3275070.39
- clat (nsec): min=0, max=2092.4M, avg=5821962.72, stdev=68390851.36
- lat (nsec): min=0, max=2092.8M, avg=5855536.09, stdev=68477975.18
- clat percentiles (nsec):
- | 1.00th=[ 716], 5.00th=[ 724], 10.00th=[ 740],
- | 20.00th=[ 772], 30.00th=[ 788], 40.00th=[ 804],
- | 50.00th=[ 836], 60.00th=[ 868], 70.00th=[ 900],
- | 80.00th=[ 1112], 90.00th=[ 12480], 95.00th=[ 33024],
- | 99.00th=[ 27394048], 99.50th=[ 633339904], 99.90th=[1019215872],
- | 99.95th=[1199570944], 99.99th=[1518338048]
- bw ( KiB/s): min= 0, max=69544, per=93.51%, avg=43953.50, stdev=36190.43, samples=2
- iops : min= 0, max= 1091, avg=689.00, stdev=568.51, samples=2
- lat (nsec) : 750=12.78%, 1000=65.59%
- lat (usec) : 2=4.88%, 4=0.16%, 10=2.00%, 20=7.08%, 50=3.16%
- lat (usec) : 100=0.23%, 250=0.11%, 500=0.14%, 750=0.15%, 1000=0.43%
- lat (msec) : 2=0.94%, 4=0.68%, 10=0.41%, 20=0.18%, 50=0.21%
- lat (msec) : 100=0.07%, 250=0.08%, 500=0.10%, 750=0.29%, 1000=0.23%
- cpu : usr=0.16%, sys=0.65%, ctx=6935, majf=0, minf=31
- IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=82711,0,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- readwrite: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:34:59 2019
- read: IOPS=51, BW=3292KiB/s (3371kB/s)(386MiB/120003msec)
- slat (nsec): min=0, max=65904k, avg=36780.12, stdev=841937.14
- clat (nsec): min=0, max=749872k, avg=6530086.82, stdev=44937011.76
- lat (nsec): min=0, max=749907k, avg=6567813.82, stdev=44947365.91
- clat percentiles (nsec):
- | 1.00th=[ 716], 5.00th=[ 724], 10.00th=[ 732],
- | 20.00th=[ 756], 30.00th=[ 852], 40.00th=[ 956],
- | 50.00th=[ 1096], 60.00th=[ 1272], 70.00th=[ 1672],
- | 80.00th=[ 95744], 90.00th=[ 1368064], 95.00th=[ 14614528],
- | 99.00th=[145752064], 99.50th=[425721856], 99.90th=[624951296],
- | 99.95th=[666894336], 99.99th=[750780416]
- bw ( KiB/s): min= 0, max= 5060, per=7.00%, avg=3292.00, stdev=2500.33, samples=2
- iops : min= 0, max= 79, avg=51.00, stdev=39.60, samples=2
- write: IOPS=53, BW=3395KiB/s (3477kB/s)(398MiB/120003msec)
- slat (nsec): min=0, max=19783k, avg=197295.51, stdev=375928.11
- clat (nsec): min=0, max=852237k, avg=68838535.89, stdev=129062909.48
- lat (nsec): min=0, max=852306k, avg=69037001.18, stdev=129005986.35
- clat percentiles (usec):
- | 1.00th=[ 996], 5.00th=[ 1237], 10.00th=[ 1418], 20.00th=[ 1844],
- | 30.00th=[ 2442], 40.00th=[ 3458], 50.00th=[ 6718], 60.00th=[ 20841],
- | 70.00th=[ 58983], 80.00th=[105382], 90.00th=[212861], 95.00th=[371196],
- | 99.00th=[624952], 99.50th=[675283], 99.90th=[759170], 99.95th=[759170],
- | 99.99th=[851444]
- bw ( KiB/s): min= 0, max= 5222, per=73.49%, avg=3392.50, stdev=2587.30, samples=2
- iops : min= 0, max= 81, avg=52.50, stdev=40.31, samples=2
- lat (nsec) : 750=9.04%, 1000=12.40%
- lat (usec) : 2=13.32%, 4=0.10%, 10=0.85%, 20=1.64%, 50=1.03%
- lat (usec) : 100=1.11%, 250=1.43%, 500=0.82%, 750=0.69%, 1000=1.48%
- lat (msec) : 2=13.19%, 4=10.76%, 10=5.76%, 20=3.66%, 50=5.25%
- lat (msec) : 100=6.13%, 250=6.62%, 500=3.37%, 750=1.33%, 1000=0.06%
- cpu : usr=0.08%, sys=0.29%, ctx=13113, majf=0, minf=29
- IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=6173,6363,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- Run status group 0 (all jobs):
- READ: bw=45.9MiB/s (48.1MB/s), 3292KiB/s-42.7MiB/s (3371kB/s-44.8MB/s), io=5537MiB (5806MB), run=120003-120619msec
- WRITE: bw=4616KiB/s (4727kB/s), 1236KiB/s-3395KiB/s (1265kB/s-3477kB/s), io=543MiB (570MB), run=120003-120516msec
- Disk stats (read/write):
- rbd199: ios=4257/8892, merge=0/1, ticks=531020/899787, in_queue=1424167, util=6.02%
- <example-benchmark-fio-server-771-benchmark-l7n6n>
- randwrite: (groupid=0, jobs=1): err= 0: pid=11: Wed May 8 20:34:59 2019
- write: IOPS=21, BW=1351KiB/s (1384kB/s)(159MiB/120291msec)
- slat (nsec): min=0, max=412566k, avg=603490.14, stdev=11034782.35
- clat (nsec): min=0, max=4347.8M, avg=188943138.26, stdev=309675547.61
- lat (nsec): min=0, max=4348.0M, avg=189547874.67, stdev=309681324.16
- clat percentiles (usec):
- | 1.00th=[ 1139], 5.00th=[ 1778], 10.00th=[ 2311],
- | 20.00th=[ 3687], 30.00th=[ 5669], 40.00th=[ 8979],
- | 50.00th=[ 15270], 60.00th=[ 120062], 70.00th=[ 258999],
- | 80.00th=[ 367002], 90.00th=[ 549454], 95.00th=[ 775947],
- | 99.00th=[1115685], 99.50th=[1300235], 99.90th=[4328522],
- | 99.95th=[4328522], 99.99th=[4328522]
- bw ( KiB/s): min= 0, max= 1528, per=28.54%, avg=1351.50, stdev=249.61, samples=2
- iops : min= 0, max= 23, avg=20.50, stdev= 3.54, samples=2
- lat (usec) : 250=0.08%, 500=0.04%, 750=0.04%, 1000=0.16%
- lat (msec) : 2=6.86%, 4=15.06%, 10=20.34%, 20=10.21%, 50=5.24%
- lat (msec) : 100=1.81%, 250=9.18%, 500=19.71%, 750=6.23%, 1000=3.59%
- cpu : usr=0.04%, sys=0.09%, ctx=4463, majf=0, minf=29
- IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=0,2537,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- read: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:34:59 2019
- read: IOPS=163, BW=10.2MiB/s (10.7MB/s)(1237MiB/120719msec)
- slat (nsec): min=0, max=10631k, avg=33850.06, stdev=183395.68
- clat (nsec): min=0, max=1826.2M, avg=24367822.86, stdev=129614967.16
- lat (nsec): min=0, max=1826.2M, avg=24402732.11, stdev=129638893.04
- clat percentiles (nsec):
- | 1.00th=[ 716], 5.00th=[ 724], 10.00th=[ 756],
- | 20.00th=[ 780], 30.00th=[ 804], 40.00th=[ 860],
- | 50.00th=[ 892], 60.00th=[ 948], 70.00th=[ 1192],
- | 80.00th=[ 1704], 90.00th=[ 3293184], 95.00th=[ 33423360],
- | 99.00th=[ 784334848], 99.50th=[ 935329792], 99.90th=[1216348160],
- | 99.95th=[1317011456], 99.99th=[1686110208]
- bw ( KiB/s): min= 0, max=11221, per=76.60%, avg=10553.50, stdev=943.99, samples=2
- iops : min= 0, max= 175, avg=164.50, stdev=14.85, samples=2
- lat (nsec) : 750=9.87%, 1000=54.06%
- lat (usec) : 2=16.47%, 4=0.22%, 10=0.24%, 20=0.74%, 50=0.15%
- lat (usec) : 100=0.12%, 250=0.07%, 500=0.09%, 750=0.30%, 1000=1.27%
- lat (msec) : 2=4.08%, 4=2.97%, 10=2.30%, 20=1.21%, 50=1.45%
- lat (msec) : 100=0.61%, 250=0.52%, 500=0.91%, 750=1.16%, 1000=0.86%
- cpu : usr=0.07%, sys=0.21%, ctx=7263, majf=0, minf=30
- IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=19789,0,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- readwrite: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:34:59 2019
- read: IOPS=51, BW=3305KiB/s (3385kB/s)(387MiB/120008msec)
- slat (nsec): min=0, max=3588.7k, avg=25761.76, stdev=73931.98
- clat (nsec): min=0, max=959944k, avg=10211143.33, stdev=51913237.46
- lat (nsec): min=0, max=959982k, avg=10237801.22, stdev=51919638.05
- clat percentiles (nsec):
- | 1.00th=[ 724], 5.00th=[ 732], 10.00th=[ 740],
- | 20.00th=[ 788], 30.00th=[ 876], 40.00th=[ 972],
- | 50.00th=[ 1112], 60.00th=[ 1256], 70.00th=[ 1512],
- | 80.00th=[ 216064], 90.00th=[ 3850240], 95.00th=[ 48496640],
- | 99.00th=[325058560], 99.50th=[387973120], 99.90th=[541065216],
- | 99.95th=[616562688], 99.99th=[960495616]
- bw ( KiB/s): min= 0, max= 4802, per=23.99%, avg=3305.50, stdev=2116.37, samples=2
- iops : min= 0, max= 75, avg=51.50, stdev=33.23, samples=2
- write: IOPS=52, BW=3393KiB/s (3475kB/s)(398MiB/120008msec)
- slat (nsec): min=0, max=3952.3k, avg=190717.24, stdev=251762.76
- clat (nsec): min=0, max=994975k, avg=65641195.08, stdev=133569305.73
- lat (nsec): min=0, max=995050k, avg=65833017.35, stdev=133524713.73
- clat percentiles (usec):
- | 1.00th=[ 963], 5.00th=[ 1172], 10.00th=[ 1336], 20.00th=[ 1663],
- | 30.00th=[ 2073], 40.00th=[ 2835], 50.00th=[ 5211], 60.00th=[ 21365],
- | 70.00th=[ 64226], 80.00th=[100140], 90.00th=[154141], 95.00th=[329253],
- | 99.00th=[734004], 99.50th=[834667], 99.90th=[935330], 99.95th=[952108],
- | 99.99th=[994051]
- bw ( KiB/s): min= 0, max= 4852, per=71.60%, avg=3391.00, stdev=2066.17, samples=2
- iops : min= 0, max= 75, avg=52.50, stdev=31.82, samples=2
- lat (nsec) : 750=7.23%, 1000=13.74%
- lat (usec) : 2=14.70%, 4=0.04%, 10=0.29%, 20=1.01%, 50=1.01%
- lat (usec) : 100=0.84%, 250=0.68%, 500=0.45%, 750=0.44%, 1000=1.74%
- lat (msec) : 2=15.66%, 4=10.29%, 10=4.86%, 20=2.95%, 50=4.81%
- lat (msec) : 100=7.88%, 250=7.40%, 500=2.55%, 750=0.94%, 1000=0.50%
- cpu : usr=0.07%, sys=0.30%, ctx=13776, majf=0, minf=30
- IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=6198,6360,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- Run status group 0 (all jobs):
- READ: bw=13.5MiB/s (14.1MB/s), 3305KiB/s-10.2MiB/s (3385kB/s-10.7MB/s), io=1624MiB (1703MB), run=120008-120719msec
- WRITE: bw=4737KiB/s (4850kB/s), 1351KiB/s-3393KiB/s (1384kB/s-3475kB/s), io=556MiB (583MB), run=120008-120291msec
- Disk stats (read/write):
- rbd230: ios=5135/9900, merge=0/2, ticks=595516/887454, in_queue=1475455, util=4.69%
- Jobs: 37 (f=37): [_(2),m(1)][99.7%][r=1206M,w=1205M][r=18.8k,w=18.8k IOPS][eta 00m:51s] <example-benchmark-fio-server-372-benchmark-lc6p4>
- randwrite: (groupid=0, jobs=1): err= 0: pid=11: Wed May 8 20:34:59 2019
- write: IOPS=25, BW=1650KiB/s (1689kB/s)(194MiB/120291msec)
- slat (nsec): min=0, max=773509k, avg=852811.07, stdev=20150575.43
- clat (nsec): min=0, max=2310.7M, avg=154359745.14, stdev=327539286.90
- lat (nsec): min=0, max=2310.8M, avg=155213400.86, stdev=327842778.04
- clat percentiles (usec):
- | 1.00th=[ 1037], 5.00th=[ 1352], 10.00th=[ 1598],
- | 20.00th=[ 2114], 30.00th=[ 2900], 40.00th=[ 3851],
- | 50.00th=[ 5276], 60.00th=[ 8160], 70.00th=[ 17171],
- | 80.00th=[ 105382], 90.00th=[ 675283], 95.00th=[ 893387],
- | 99.00th=[1333789], 99.50th=[1484784], 99.90th=[2021655],
- | 99.95th=[2264925], 99.99th=[2298479]
- bw ( KiB/s): min= 0, max= 2336, per=36.05%, avg=1650.50, stdev=969.44, samples=2
- iops : min= 0, max= 36, avg=25.50, stdev=14.85, samples=2
- lat (usec) : 500=0.10%, 750=0.13%, 1000=0.55%
- lat (msec) : 2=17.43%, 4=22.82%, 10=23.08%, 20=7.00%, 50=3.52%
- lat (msec) : 100=5.07%, 250=1.94%, 500=2.61%, 750=8.20%, 1000=4.07%
- cpu : usr=0.03%, sys=0.12%, ctx=5780, majf=0, minf=27
- IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=0,3098,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- read: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:34:59 2019
- read: IOPS=610, BW=38.0MiB/s (39.9MB/s)(4588MiB/120573msec)
- slat (nsec): min=0, max=771813k, avg=42177.78, stdev=4068032.92
- clat (nsec): min=0, max=1795.7M, avg=6503281.36, stdev=66799159.40
- lat (nsec): min=0, max=1795.8M, avg=6546418.17, stdev=66930745.83
- clat percentiles (nsec):
- | 1.00th=[ 708], 5.00th=[ 724], 10.00th=[ 732],
- | 20.00th=[ 764], 30.00th=[ 780], 40.00th=[ 796],
- | 50.00th=[ 828], 60.00th=[ 860], 70.00th=[ 884],
- | 80.00th=[ 972], 90.00th=[ 11840], 95.00th=[ 995328],
- | 99.00th=[ 91750400], 99.50th=[ 599785472], 99.90th=[ 952107008],
- | 99.95th=[1115684864], 99.99th=[1400897536]
- bw ( KiB/s): min= 0, max=63106, per=93.37%, avg=39021.00, stdev=34061.33, samples=2
- iops : min= 0, max= 990, avg=611.50, stdev=535.28, samples=2
- lat (nsec) : 750=15.86%, 1000=64.91%
- lat (usec) : 2=5.86%, 4=0.14%, 10=1.15%, 20=5.13%, 50=0.69%
- lat (usec) : 100=0.19%, 250=0.10%, 500=0.19%, 750=0.28%, 1000=0.49%
- lat (msec) : 2=1.20%, 4=1.00%, 10=0.73%, 20=0.33%, 50=0.46%
- lat (msec) : 100=0.31%, 250=0.10%, 500=0.23%, 750=0.38%, 1000=0.18%
- cpu : usr=0.15%, sys=0.61%, ctx=8969, majf=0, minf=31
- IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=73655,0,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- readwrite: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:34:59 2019
- read: IOPS=44, BW=2840KiB/s (2908kB/s)(333MiB/120025msec)
- slat (nsec): min=0, max=2616.8k, avg=30791.64, stdev=88068.41
- clat (nsec): min=0, max=642034k, avg=7264005.10, stdev=39713153.45
- lat (nsec): min=0, max=642063k, avg=7295930.78, stdev=39719952.99
- clat percentiles (nsec):
- | 1.00th=[ 716], 5.00th=[ 724], 10.00th=[ 732],
- | 20.00th=[ 788], 30.00th=[ 876], 40.00th=[ 988],
- | 50.00th=[ 1144], 60.00th=[ 1304], 70.00th=[ 9920],
- | 80.00th=[ 684032], 90.00th=[ 2899968], 95.00th=[ 41680896],
- | 99.00th=[181403648], 99.50th=[362807296], 99.90th=[501219328],
- | 99.95th=[530579456], 99.99th=[641728512]
- bw ( KiB/s): min= 0, max= 4536, per=6.80%, avg=2840.00, stdev=2398.51, samples=2
- iops : min= 0, max= 70, avg=43.50, stdev=37.48, samples=2
- write: IOPS=45, BW=2935KiB/s (3005kB/s)(344MiB/120025msec)
- slat (nsec): min=0, max=5247.4k, avg=183124.89, stdev=260290.31
- clat (nsec): min=0, max=922637k, avg=80167211.55, stdev=145551293.96
- lat (nsec): min=0, max=922680k, avg=80351609.68, stdev=145505979.46
- clat percentiles (usec):
- | 1.00th=[ 1029], 5.00th=[ 1237], 10.00th=[ 1434], 20.00th=[ 1860],
- | 30.00th=[ 2573], 40.00th=[ 3916], 50.00th=[ 8979], 60.00th=[ 28705],
- | 70.00th=[ 68682], 80.00th=[113771], 90.00th=[278922], 95.00th=[450888],
- | 99.00th=[650118], 99.50th=[725615], 99.90th=[859833], 99.95th=[918553],
- | 99.99th=[918553]
- bw ( KiB/s): min= 0, max= 4651, per=64.06%, avg=2932.50, stdev=2430.33, samples=2
- iops : min= 0, max= 72, avg=45.00, stdev=38.18, samples=2
- lat (nsec) : 750=7.63%, 1000=12.33%
- lat (usec) : 2=13.99%, 4=0.14%, 10=0.38%, 20=0.85%, 50=0.85%
- lat (usec) : 100=0.89%, 250=1.16%, 500=0.70%, 750=0.79%, 1000=1.73%
- lat (msec) : 2=13.41%, 4=10.59%, 10=5.92%, 20=3.28%, 50=5.86%
- lat (msec) : 100=7.03%, 250=6.65%, 500=3.85%, 750=1.83%, 1000=0.17%
- cpu : usr=0.06%, sys=0.26%, ctx=12141, majf=0, minf=30
- IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=5326,5501,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- Run status group 0 (all jobs):
- READ: bw=40.8MiB/s (42.8MB/s), 2840KiB/s-38.0MiB/s (2908kB/s-39.9MB/s), io=4921MiB (5160MB), run=120025-120573msec
- WRITE: bw=4578KiB/s (4688kB/s), 1650KiB/s-2935KiB/s (1689kB/s-3005kB/s), io=538MiB (564MB), run=120025-120291msec
- Disk stats (read/write):
- rbd130: ios=6082/9151, merge=0/1, ticks=546406/923365, in_queue=1462130, util=7.65%
- <example-benchmark-fio-server-452-benchmark-7lt6f>
- randwrite: (groupid=0, jobs=1): err= 0: pid=11: Wed May 8 20:34:59 2019
- write: IOPS=45, BW=2916KiB/s (2986kB/s)(346MiB/121551msec)
- slat (nsec): min=0, max=8601.9M, avg=1768759.47, stdev=115624208.02
- clat (nsec): min=0, max=11024M, avg=85317961.64, stdev=306099695.43
- lat (nsec): min=0, max=11024M, avg=87087195.73, stdev=327190446.01
- clat percentiles (usec):
- | 1.00th=[ 914], 5.00th=[ 1745], 10.00th=[ 2507],
- | 20.00th=[ 3720], 30.00th=[ 5145], 40.00th=[ 7504],
- | 50.00th=[ 10421], 60.00th=[ 15533], 70.00th=[ 29492],
- | 80.00th=[ 102237], 90.00th=[ 219153], 95.00th=[ 484443],
- | 99.00th=[ 759170], 99.50th=[ 935330], 99.90th=[ 3305112],
- | 99.95th=[ 8657044], 99.99th=[11072963]
- bw ( KiB/s): min= 0, max= 3061, per=47.12%, avg=2950.00, stdev=156.98, samples=2
- iops : min= 0, max= 47, avg=45.50, stdev= 2.12, samples=2
- lat (usec) : 4=0.04%, 50=0.02%, 250=0.07%, 500=0.04%, 750=0.33%
- lat (usec) : 1000=0.89%
- lat (msec) : 2=5.53%, 4=15.37%, 10=26.41%, 20=16.22%, 50=9.83%
- lat (msec) : 100=5.17%, 250=11.51%, 500=3.97%, 750=3.63%, 1000=0.63%
- cpu : usr=0.05%, sys=0.19%, ctx=9026, majf=0, minf=30
- IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=0,5535,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- read: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:34:59 2019
- read: IOPS=79, BW=5087KiB/s (5210kB/s)(599MiB/120566msec)
- slat (nsec): min=0, max=11811k, avg=55106.52, stdev=241135.38
- clat (nsec): min=0, max=1838.1M, avg=50275141.81, stdev=187121374.36
- lat (nsec): min=0, max=1838.1M, avg=50331333.59, stdev=187141422.67
- clat percentiles (nsec):
- | 1.00th=[ 724], 5.00th=[ 764], 10.00th=[ 788],
- | 20.00th=[ 852], 30.00th=[ 908], 40.00th=[ 1012],
- | 50.00th=[ 1256], 60.00th=[ 1512], 70.00th=[ 1138688],
- | 80.00th=[ 2932736], 90.00th=[ 15400960], 95.00th=[ 549453824],
- | 99.00th=[ 935329792], 99.50th=[1052770304], 99.90th=[1350565888],
- | 99.95th=[1585446912], 99.99th=[1837105152]
- bw ( KiB/s): min= 0, max= 5128, per=61.11%, avg=5102.50, stdev=36.06, samples=2
- iops : min= 0, max= 80, avg=79.50, stdev= 0.71, samples=2
- lat (nsec) : 750=4.34%, 1000=35.13%
- lat (usec) : 2=23.43%, 4=0.21%, 10=0.10%, 20=0.85%, 50=0.27%
- lat (usec) : 100=0.21%, 250=0.15%, 500=0.19%, 750=0.66%, 1000=3.03%
- lat (msec) : 2=7.55%, 4=6.70%, 10=5.32%, 20=2.76%, 50=1.40%
- lat (msec) : 100=0.45%, 250=0.47%, 500=1.35%, 750=2.96%, 1000=1.87%
- cpu : usr=0.04%, sys=0.14%, ctx=6779, majf=0, minf=30
- IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=9581,0,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- readwrite: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:34:59 2019
- read: IOPS=51, BW=3278KiB/s (3357kB/s)(384MiB/120011msec)
- slat (nsec): min=0, max=4415.6k, avg=40057.69, stdev=108722.41
- clat (nsec): min=0, max=810846k, avg=15542310.91, stdev=68280778.67
- lat (nsec): min=0, max=810878k, avg=15583357.86, stdev=68290237.45
- clat percentiles (nsec):
- | 1.00th=[ 724], 5.00th=[ 740], 10.00th=[ 788],
- | 20.00th=[ 884], 30.00th=[ 996], 40.00th=[ 1144],
- | 50.00th=[ 1288], 60.00th=[ 14016], 70.00th=[ 954368],
- | 80.00th=[ 2179072], 90.00th=[ 31850496], 95.00th=[ 64749568],
- | 99.00th=[400556032], 99.50th=[574619648], 99.90th=[717225984],
- | 99.95th=[725614592], 99.99th=[809500672]
- bw ( KiB/s): min= 0, max= 5222, per=39.26%, avg=3278.00, stdev=2749.23, samples=2
- iops : min= 0, max= 81, avg=50.50, stdev=43.13, samples=2
- write: IOPS=52, BW=3387KiB/s (3469kB/s)(397MiB/120011msec)
- slat (nsec): min=0, max=11202k, avg=195281.65, stdev=275030.68
- clat (nsec): min=0, max=980601k, avg=60513733.50, stdev=123539267.98
- lat (nsec): min=0, max=980657k, avg=60710238.12, stdev=123494453.74
- clat percentiles (usec):
- | 1.00th=[ 1012], 5.00th=[ 1221], 10.00th=[ 1385], 20.00th=[ 1745],
- | 30.00th=[ 2278], 40.00th=[ 3195], 50.00th=[ 5342], 60.00th=[ 14615],
- | 70.00th=[ 45351], 80.00th=[ 87557], 90.00th=[154141], 95.00th=[367002],
- | 99.00th=[574620], 99.50th=[658506], 99.90th=[884999], 99.95th=[943719],
- | 99.99th=[977273]
- bw ( KiB/s): min= 0, max= 5397, per=54.07%, avg=3385.00, stdev=2845.40, samples=2
- iops : min= 0, max= 84, avg=52.50, stdev=44.55, samples=2
- lat (nsec) : 750=3.39%, 1000=11.52%
- lat (usec) : 2=14.24%, 4=0.07%, 10=0.18%, 20=0.42%, 50=0.50%
- lat (usec) : 100=0.38%, 250=0.55%, 500=0.29%, 750=0.99%, 1000=2.86%
- lat (msec) : 2=16.39%, 4=12.80%, 10=7.15%, 20=3.89%, 50=6.17%
- lat (msec) : 100=8.40%, 250=5.23%, 500=3.12%, 750=1.36%, 1000=0.14%
- cpu : usr=0.08%, sys=0.31%, ctx=16088, majf=0, minf=30
- IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=6146,6350,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- Run status group 0 (all jobs):
- READ: bw=8350KiB/s (8551kB/s), 3278KiB/s-5087KiB/s (3357kB/s-5210kB/s), io=983MiB (1031MB), run=120011-120566msec
- WRITE: bw=6260KiB/s (6411kB/s), 2916KiB/s-3387KiB/s (2986kB/s-3469kB/s), io=743MiB (779MB), run=120011-121551msec
- Disk stats (read/write):
- rbd166: ios=6007/12363, merge=0/3, ticks=619271/875615, in_queue=1485609, util=4.64%
- <example-benchmark-fio-server-170-benchmark-7ll8b>
- randwrite: (groupid=0, jobs=1): err= 0: pid=11: Wed May 8 20:34:59 2019
- write: IOPS=25, BW=1658KiB/s (1698kB/s)(195MiB/120385msec)
- slat (nsec): min=0, max=887718k, avg=1213744.90, stdev=27334605.55
- clat (nsec): min=0, max=2050.6M, avg=153222238.41, stdev=315111553.22
- lat (nsec): min=0, max=2050.6M, avg=154436698.50, stdev=316707758.24
- clat percentiles (usec):
- | 1.00th=[ 1156], 5.00th=[ 1483], 10.00th=[ 1827],
- | 20.00th=[ 2474], 30.00th=[ 3392], 40.00th=[ 5080],
- | 50.00th=[ 7504], 60.00th=[ 13173], 70.00th=[ 31851],
- | 80.00th=[ 121111], 90.00th=[ 675283], 95.00th=[ 884999],
- | 99.00th=[1283458], 99.50th=[1384121], 99.90th=[1887437],
- | 99.95th=[2021655], 99.99th=[2055209]
- bw ( KiB/s): min= 0, max= 1787, per=32.78%, avg=1660.50, stdev=178.90, samples=2
- iops : min= 0, max= 27, avg=25.00, stdev= 2.83, samples=2
- lat (usec) : 2=0.03%, 750=0.16%, 1000=0.19%
- lat (msec) : 2=12.45%, 4=21.57%, 10=21.57%, 20=9.34%, 50=7.64%
- lat (msec) : 100=5.49%, 250=3.59%, 500=2.86%, 750=7.54%, 1000=4.40%
- cpu : usr=0.04%, sys=0.12%, ctx=5748, majf=0, minf=30
- IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=0,3116,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- read: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:34:59 2019
- read: IOPS=671, BW=41.8MiB/s (43.8MB/s)(5034MiB/120435msec)
- slat (nsec): min=0, max=880995k, avg=42469.66, stdev=4585790.36
- clat (nsec): min=0, max=1644.8M, avg=5920176.44, stdev=63524581.82
- lat (nsec): min=0, max=1644.8M, avg=5963596.00, stdev=63696049.76
- clat percentiles (nsec):
- | 1.00th=[ 708], 5.00th=[ 716], 10.00th=[ 724],
- | 20.00th=[ 764], 30.00th=[ 780], 40.00th=[ 804],
- | 50.00th=[ 844], 60.00th=[ 876], 70.00th=[ 908],
- | 80.00th=[ 1480], 90.00th=[ 12864], 95.00th=[ 757760],
- | 99.00th=[ 63176704], 99.50th=[ 574619648], 99.90th=[ 952107008],
- | 99.95th=[1044381696], 99.99th=[1300234240]
- bw ( KiB/s): min= 0, max=69975, per=93.01%, avg=42879.00, stdev=38319.53, samples=2
- iops : min= 0, max= 1098, avg=672.00, stdev=602.45, samples=2
- lat (nsec) : 750=16.86%, 1000=58.14%
- lat (usec) : 2=5.53%, 4=0.13%, 10=1.32%, 20=9.56%, 50=2.74%
- lat (usec) : 100=0.26%, 250=0.13%, 500=0.15%, 750=0.17%, 1000=0.53%
- lat (msec) : 2=1.20%, 4=0.87%, 10=0.59%, 20=0.30%, 50=0.40%
- lat (msec) : 100=0.25%, 250=0.07%, 500=0.22%, 750=0.34%, 1000=0.18%
- cpu : usr=0.16%, sys=0.67%, ctx=8878, majf=0, minf=34
- IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=80815,0,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- readwrite: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:34:59 2019
- read: IOPS=51, BW=3314KiB/s (3394kB/s)(389MiB/120035msec)
- slat (nsec): min=0, max=1675.7k, avg=27582.43, stdev=64411.66
- clat (nsec): min=0, max=694594k, avg=8549233.54, stdev=46985004.43
- lat (nsec): min=0, max=694628k, avg=8577802.47, stdev=46991864.10
- clat percentiles (nsec):
- | 1.00th=[ 716], 5.00th=[ 724], 10.00th=[ 732],
- | 20.00th=[ 756], 30.00th=[ 860], 40.00th=[ 964],
- | 50.00th=[ 1112], 60.00th=[ 1256], 70.00th=[ 1656],
- | 80.00th=[ 337920], 90.00th=[ 2801664], 95.00th=[ 39583744],
- | 99.00th=[256901120], 99.50th=[387973120], 99.90th=[526385152],
- | 99.95th=[624951296], 99.99th=[692060160]
- bw ( KiB/s): min= 0, max= 5128, per=7.19%, avg=3314.50, stdev=2564.68, samples=2
- iops : min= 0, max= 80, avg=51.50, stdev=40.31, samples=2
- write: IOPS=53, BW=3418KiB/s (3500kB/s)(401MiB/120035msec)
- slat (nsec): min=0, max=42017k, avg=186096.19, stdev=597925.65
- clat (nsec): min=0, max=1006.8M, avg=66420593.68, stdev=125702557.93
- lat (nsec): min=0, max=1006.9M, avg=66607957.18, stdev=125661614.59
- clat percentiles (usec):
- | 1.00th=[ 1029], 5.00th=[ 1237], 10.00th=[ 1450],
- | 20.00th=[ 1827], 30.00th=[ 2311], 40.00th=[ 3294],
- | 50.00th=[ 6521], 60.00th=[ 22938], 70.00th=[ 61604],
- | 80.00th=[ 105382], 90.00th=[ 173016], 95.00th=[ 358613],
- | 99.00th=[ 633340], 99.50th=[ 692061], 99.90th=[ 801113],
- | 99.95th=[ 884999], 99.99th=[1010828]
- bw ( KiB/s): min= 0, max= 5280, per=67.45%, avg=3416.50, stdev=2635.39, samples=2
- iops : min= 0, max= 82, avg=53.00, stdev=41.01, samples=2
- lat (nsec) : 750=9.02%, 1000=11.91%
- lat (usec) : 2=13.84%, 4=0.06%, 10=0.25%, 20=1.00%, 50=0.92%
- lat (usec) : 100=1.05%, 250=1.01%, 500=0.73%, 750=0.59%, 1000=1.78%
- lat (msec) : 2=13.49%, 4=11.63%, 10=5.38%, 20=3.46%, 50=5.36%
- lat (msec) : 100=6.70%, 250=7.63%, 500=2.93%, 750=1.18%, 1000=0.10%
- cpu : usr=0.07%, sys=0.30%, ctx=13879, majf=0, minf=33
- IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=6215,6408,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- Run status group 0 (all jobs):
- READ: bw=45.0MiB/s (47.2MB/s), 3314KiB/s-41.8MiB/s (3394kB/s-43.8MB/s), io=5422MiB (5686MB), run=120035-120435msec
- WRITE: bw=5066KiB/s (5187kB/s), 1658KiB/s-3418KiB/s (1698kB/s-3500kB/s), io=596MiB (624MB), run=120035-120385msec
- Disk stats (read/write):
- rbd54: ios=5720/9776, merge=0/1, ticks=553989/877423, in_queue=1423562, util=7.76%
- <example-benchmark-fio-server-305-benchmark-bvd8p>
- randwrite: (groupid=0, jobs=1): err= 0: pid=11: Wed May 8 20:34:59 2019
- write: IOPS=46, BW=2967KiB/s (3038kB/s)(348MiB/120084msec)
- slat (nsec): min=0, max=591926k, avg=385076.58, stdev=10580791.50
- clat (nsec): min=0, max=1249.9M, avg=86328692.16, stdev=149286241.82
- lat (nsec): min=0, max=1249.0M, avg=86715682.26, stdev=149627952.29
- clat percentiles (usec):
- | 1.00th=[ 1004], 5.00th=[ 1237], 10.00th=[ 1467],
- | 20.00th=[ 1958], 30.00th=[ 2868], 40.00th=[ 4686],
- | 50.00th=[ 11076], 60.00th=[ 53216], 70.00th=[ 90702],
- | 80.00th=[ 123208], 90.00th=[ 258999], 95.00th=[ 408945],
- | 99.00th=[ 708838], 99.50th=[ 809501], 99.90th=[1052771],
- | 99.95th=[1233126], 99.99th=[1249903]
- bw ( KiB/s): min= 0, max= 3388, per=45.08%, avg=2966.50, stdev=596.09, samples=2
- iops : min= 0, max= 52, avg=45.50, stdev= 9.19, samples=2
- lat (usec) : 4=0.02%, 250=0.02%, 750=0.02%, 1000=0.93%
- lat (msec) : 2=19.57%, 4=16.80%, 10=11.79%, 20=4.85%, 50=5.59%
- lat (msec) : 100=13.28%, 250=16.71%, 500=7.08%, 750=2.50%, 1000=0.74%
- cpu : usr=0.05%, sys=0.21%, ctx=10453, majf=0, minf=29
- IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=0,5564,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- read: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:34:59 2019
- read: IOPS=316, BW=19.7MiB/s (20.7MB/s)(2370MiB/120032msec)
- slat (nsec): min=0, max=590286k, avg=50581.37, stdev=3970019.69
- clat (nsec): min=0, max=1148.2M, avg=12614914.63, stdev=60910243.05
- lat (nsec): min=0, max=1148.2M, avg=12666319.66, stdev=61045819.97
- clat percentiles (nsec):
- | 1.00th=[ 708], 5.00th=[ 708], 10.00th=[ 716],
- | 20.00th=[ 724], 30.00th=[ 732], 40.00th=[ 748],
- | 50.00th=[ 836], 60.00th=[ 892], 70.00th=[ 1112],
- | 80.00th=[ 18560], 90.00th=[ 9371648], 95.00th=[ 59506688],
- | 99.00th=[ 350224384], 99.50th=[ 476053504], 99.90th=[ 692060160],
- | 99.95th=[ 750780416], 99.99th=[1010827264]
- bw ( KiB/s): min= 0, max=28456, per=85.14%, avg=20222.00, stdev=11644.63, samples=2
- iops : min= 0, max= 444, avg=315.50, stdev=181.73, samples=2
- lat (nsec) : 750=40.78%, 1000=26.21%
- lat (usec) : 2=10.59%, 4=0.03%, 10=1.41%, 20=1.02%, 50=0.30%
- lat (usec) : 100=0.13%, 250=0.06%, 500=0.15%, 750=0.86%, 1000=2.15%
- lat (msec) : 2=3.24%, 4=1.98%, 10=1.16%, 20=0.65%, 50=3.31%
- lat (msec) : 100=2.89%, 250=1.61%, 500=1.07%, 750=0.37%, 1000=0.03%
- cpu : usr=0.09%, sys=0.37%, ctx=14721, majf=0, minf=29
- IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=37938,0,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- readwrite: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:34:59 2019
- read: IOPS=55, BW=3535KiB/s (3620kB/s)(414MiB/120039msec)
- slat (nsec): min=0, max=484899k, avg=107021.58, stdev=5955708.76
- clat (nsec): min=0, max=759194k, avg=6801948.12, stdev=36574853.34
- lat (nsec): min=0, max=759246k, avg=6909910.12, stdev=37050983.59
- clat percentiles (nsec):
- | 1.00th=[ 716], 5.00th=[ 724], 10.00th=[ 732],
- | 20.00th=[ 796], 30.00th=[ 892], 40.00th=[ 1004],
- | 50.00th=[ 1160], 60.00th=[ 1320], 70.00th=[ 15040],
- | 80.00th=[ 888832], 90.00th=[ 4145152], 95.00th=[ 37486592],
- | 99.00th=[170917888], 99.50th=[258998272], 99.90th=[476053504],
- | 99.95th=[608174080], 99.99th=[759169024]
- bw ( KiB/s): min= 0, max= 4618, per=14.88%, avg=3535.00, stdev=1531.59, samples=2
- iops : min= 0, max= 72, avg=55.00, stdev=24.04, samples=2
- write: IOPS=56, BW=3615KiB/s (3702kB/s)(424MiB/120039msec)
- slat (nsec): min=0, max=589992k, avg=284036.47, stdev=7194313.68
- clat (nsec): min=0, max=977765k, avg=64022066.99, stdev=122762405.28
- lat (nsec): min=0, max=1018.0M, avg=64307305.36, stdev=123184191.18
- clat percentiles (usec):
- | 1.00th=[ 1057], 5.00th=[ 1287], 10.00th=[ 1483], 20.00th=[ 1926],
- | 30.00th=[ 2573], 40.00th=[ 3785], 50.00th=[ 6849], 60.00th=[ 17957],
- | 70.00th=[ 53216], 80.00th=[ 98042], 90.00th=[187696], 95.00th=[350225],
- | 99.00th=[591397], 99.50th=[675283], 99.90th=[859833], 99.95th=[884999],
- | 99.99th=[977273]
- bw ( KiB/s): min= 0, max= 4677, per=54.92%, avg=3614.00, stdev=1503.31, samples=2
- iops : min= 0, max= 73, avg=56.00, stdev=24.04, samples=2
- lat (nsec) : 750=7.62%, 1000=12.02%
- lat (usec) : 2=14.45%, 4=0.07%, 10=0.25%, 20=0.67%, 50=0.63%
- lat (usec) : 100=0.78%, 250=0.84%, 500=0.75%, 750=0.63%, 1000=1.73%
- lat (msec) : 2=13.27%, 4=11.49%, 10=8.05%, 20=3.89%, 50=5.53%
- lat (msec) : 100=6.67%, 250=6.58%, 500=2.96%, 750=1.01%, 1000=0.12%
- cpu : usr=0.06%, sys=0.33%, ctx=15762, majf=0, minf=36
- IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=6631,6777,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- Run status group 0 (all jobs):
- READ: bw=23.2MiB/s (24.3MB/s), 3535KiB/s-19.7MiB/s (3620kB/s-20.7MB/s), io=2784MiB (2920MB), run=120032-120039msec
- WRITE: bw=6580KiB/s (6738kB/s), 2967KiB/s-3615KiB/s (3038kB/s-3702kB/s), io=772MiB (809MB), run=120039-120084msec
- Disk stats (read/write):
- rbd105: ios=9005/12689, merge=0/1, ticks=537857/830714, in_queue=1357530, util=14.73%
- <example-benchmark-fio-server-214-benchmark-g26rt>
- randwrite: (groupid=0, jobs=1): err= 0: pid=11: Wed May 8 20:34:59 2019
- write: IOPS=39, BW=2501KiB/s (2561kB/s)(293MiB/120100msec)
- slat (nsec): min=0, max=878508k, avg=512354.66, stdev=14978981.02
- clat (nsec): min=0, max=1750.0M, avg=102508712.84, stdev=189336994.89
- lat (nsec): min=0, max=1750.1M, avg=103022267.99, stdev=189729647.89
- clat percentiles (usec):
- | 1.00th=[ 1045], 5.00th=[ 1237], 10.00th=[ 1434],
- | 20.00th=[ 1942], 30.00th=[ 2835], 40.00th=[ 4752],
- | 50.00th=[ 10552], 60.00th=[ 51643], 70.00th=[ 103285],
- | 80.00th=[ 139461], 90.00th=[ 304088], 95.00th=[ 549454],
- | 99.00th=[ 843056], 99.50th=[1010828], 99.90th=[1602225],
- | 99.95th=[1702888], 99.99th=[1753220]
- bw ( KiB/s): min= 0, max= 1899, per=33.04%, avg=1899.00, stdev= 0.00, samples=1
- iops : min= 0, max= 29, avg=29.00, stdev= 0.00, samples=1
- lat (usec) : 500=0.02%, 750=0.02%, 1000=0.58%
- lat (msec) : 2=20.34%, 4=16.12%, 10=12.30%, 20=5.71%, 50=4.80%
- lat (msec) : 100=9.08%, 250=19.02%, 500=6.48%, 750=3.67%, 1000=1.36%
- cpu : usr=0.05%, sys=0.17%, ctx=8743, majf=0, minf=29
- IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=0,4690,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- read: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:34:59 2019
- read: IOPS=371, BW=23.2MiB/s (24.3MB/s)(2787MiB/120081msec)
- slat (nsec): min=0, max=864080k, avg=51361.43, stdev=4763726.52
- clat (nsec): min=0, max=1359.3M, avg=10722193.91, stdev=64689666.11
- lat (nsec): min=0, max=1359.4M, avg=10774402.02, stdev=64870471.03
- clat percentiles (nsec):
- | 1.00th=[ 716], 5.00th=[ 716], 10.00th=[ 724],
- | 20.00th=[ 732], 30.00th=[ 764], 40.00th=[ 804],
- | 50.00th=[ 844], 60.00th=[ 884], 70.00th=[ 940],
- | 80.00th=[ 1160], 90.00th=[ 1105920], 95.00th=[ 41680896],
- | 99.00th=[ 350224384], 99.50th=[ 497025024], 99.90th=[ 817889280],
- | 99.95th=[ 918552576], 99.99th=[1166016512]
- bw ( KiB/s): min= 0, max=36891, per=88.40%, avg=23777.50, stdev=18545.29, samples=2
- iops : min= 0, max= 577, avg=371.50, stdev=290.62, samples=2
- lat (nsec) : 750=27.86%, 1000=47.00%
- lat (usec) : 2=10.92%, 4=0.06%, 10=0.77%, 20=0.56%, 50=0.15%
- lat (usec) : 100=0.09%, 250=0.05%, 500=0.21%, 750=0.58%, 1000=1.35%
- lat (msec) : 2=2.12%, 4=1.40%, 10=0.69%, 20=0.34%, 50=1.35%
- lat (msec) : 100=1.96%, 250=1.11%, 500=0.95%, 750=0.32%, 1000=0.14%
- cpu : usr=0.12%, sys=0.38%, ctx=11228, majf=0, minf=108
- IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=44650,0,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- readwrite: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:34:59 2019
- read: IOPS=49, BW=3136KiB/s (3212kB/s)(368MiB/120005msec)
- slat (nsec): min=0, max=1914.8k, avg=31155.07, stdev=74360.36
- clat (nsec): min=0, max=744812k, avg=9110926.27, stdev=44539320.61
- lat (nsec): min=0, max=744837k, avg=9143035.67, stdev=44544914.81
- clat percentiles (nsec):
- | 1.00th=[ 716], 5.00th=[ 732], 10.00th=[ 740],
- | 20.00th=[ 828], 30.00th=[ 908], 40.00th=[ 1032],
- | 50.00th=[ 1160], 60.00th=[ 1304], 70.00th=[ 19584],
- | 80.00th=[ 987136], 90.00th=[ 7176192], 95.00th=[ 50069504],
- | 99.00th=[233832448], 99.50th=[341835776], 99.90th=[574619648],
- | 99.95th=[616562688], 99.99th=[742391808]
- bw ( KiB/s): min= 0, max= 4258, per=11.66%, avg=3136.00, stdev=1586.75, samples=2
- iops : min= 0, max= 66, avg=48.50, stdev=24.75, samples=2
- write: IOPS=50, BW=3250KiB/s (3328kB/s)(381MiB/120005msec)
- slat (nsec): min=0, max=878437k, avg=346407.87, stdev=11510788.20
- clat (nsec): min=0, max=1737.6M, avg=69625248.52, stdev=133022418.53
- lat (nsec): min=0, max=1737.7M, avg=69972917.80, stdev=134278546.34
- clat percentiles (usec):
- | 1.00th=[ 1045], 5.00th=[ 1287], 10.00th=[ 1516],
- | 20.00th=[ 2057], 30.00th=[ 3097], 40.00th=[ 5145],
- | 50.00th=[ 10683], 60.00th=[ 24773], 70.00th=[ 70779],
- | 80.00th=[ 105382], 90.00th=[ 175113], 95.00th=[ 337642],
- | 99.00th=[ 700449], 99.50th=[ 742392], 99.90th=[ 884999],
- | 99.95th=[ 935330], 99.99th=[1736442]
- bw ( KiB/s): min= 0, max= 4382, per=56.51%, avg=3248.00, stdev=1603.72, samples=2
- iops : min= 0, max= 68, avg=50.50, stdev=24.75, samples=2
- lat (nsec) : 750=5.64%, 1000=12.77%
- lat (usec) : 2=15.11%, 4=0.10%, 10=0.12%, 20=0.71%, 50=0.65%
- lat (usec) : 100=0.73%, 250=0.74%, 500=0.56%, 750=0.63%, 1000=1.97%
- lat (msec) : 2=12.09%, 4=9.67%, 10=8.15%, 20=5.10%, 50=5.26%
- lat (msec) : 100=7.80%, 250=8.05%, 500=2.76%, 750=1.18%, 1000=0.21%
- cpu : usr=0.07%, sys=0.29%, ctx=14274, majf=0, minf=30
- IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=5882,6091,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- Run status group 0 (all jobs):
- READ: bw=26.3MiB/s (27.5MB/s), 3136KiB/s-23.2MiB/s (3212kB/s-24.3MB/s), io=3154MiB (3307MB), run=120005-120081msec
- WRITE: bw=5748KiB/s (5886kB/s), 2501KiB/s-3250KiB/s (2561kB/s-3328kB/s), io=674MiB (707MB), run=120005-120100msec
- Disk stats (read/write):
- rbd60: ios=7813/11452, merge=0/1, ticks=560837/843308, in_queue=1394412, util=12.65%
- <example-benchmark-fio-server-573-benchmark-29b45>
- randwrite: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:34:59 2019
- write: IOPS=44, BW=2846KiB/s (2915kB/s)(334MiB/120177msec)
- slat (nsec): min=0, max=820038k, avg=481068.24, stdev=13479085.23
- clat (nsec): min=0, max=1048.1M, avg=89624595.19, stdev=149912487.94
- lat (nsec): min=0, max=1048.2M, avg=90106805.68, stdev=150298868.66
- clat percentiles (usec):
- | 1.00th=[ 988], 5.00th=[ 1205], 10.00th=[ 1385],
- | 20.00th=[ 1893], 30.00th=[ 2868], 40.00th=[ 5735],
- | 50.00th=[ 16581], 60.00th=[ 74974], 70.00th=[ 102237],
- | 80.00th=[ 125305], 90.00th=[ 235930], 95.00th=[ 425722],
- | 99.00th=[ 725615], 99.50th=[ 843056], 99.90th=[ 935330],
- | 99.95th=[ 943719], 99.99th=[1044382]
- bw ( KiB/s): min= 0, max= 3072, per=44.56%, avg=2848.50, stdev=316.08, samples=2
- iops : min= 0, max= 48, avg=44.50, stdev= 4.95, samples=2
- lat (usec) : 4=0.04%, 100=0.02%, 250=0.02%, 500=0.02%, 750=0.06%
- lat (usec) : 1000=1.03%
- lat (msec) : 2=20.57%, 4=13.91%, 10=9.88%, 20=5.93%, 50=4.98%
- lat (msec) : 100=12.56%, 250=21.47%, 500=5.88%, 750=2.85%, 1000=0.80%
- cpu : usr=0.06%, sys=0.19%, ctx=10059, majf=0, minf=30
- IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=0,5342,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- read: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:34:59 2019
- read: IOPS=326, BW=20.4MiB/s (21.4MB/s)(2449MiB/120025msec)
- slat (nsec): min=0, max=690834k, avg=54024.69, stdev=4310215.39
- clat (nsec): min=0, max=1444.3M, avg=12206634.97, stdev=63174388.43
- lat (nsec): min=0, max=1444.4M, avg=12261505.28, stdev=63326547.19
- clat percentiles (nsec):
- | 1.00th=[ 708], 5.00th=[ 716], 10.00th=[ 716],
- | 20.00th=[ 724], 30.00th=[ 732], 40.00th=[ 748],
- | 50.00th=[ 828], 60.00th=[ 876], 70.00th=[ 948],
- | 80.00th=[ 1304], 90.00th=[ 2670592], 95.00th=[ 56885248],
- | 99.00th=[ 362807296], 99.50th=[ 484442112], 99.90th=[ 734003200],
- | 99.95th=[ 809500672], 99.99th=[1010827264]
- bw ( KiB/s): min= 0, max=30908, per=85.83%, avg=20894.50, stdev=14161.23, samples=2
- iops : min= 0, max= 484, avg=327.00, stdev=222.03, samples=2
- lat (nsec) : 750=40.44%, 1000=32.16%
- lat (usec) : 2=9.98%, 4=0.05%, 10=0.85%, 20=0.80%, 50=0.18%
- lat (usec) : 100=0.07%, 250=0.05%, 500=0.11%, 750=0.68%, 1000=1.57%
- lat (msec) : 2=2.35%, 4=1.47%, 10=0.79%, 20=0.54%, 50=2.30%
- lat (msec) : 100=2.60%, 250=1.44%, 500=1.10%, 750=0.38%, 1000=0.07%
- cpu : usr=0.10%, sys=0.35%, ctx=12084, majf=0, minf=44
- IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=39241,0,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- readwrite: (groupid=0, jobs=1): err= 0: pid=14: Wed May 8 20:34:59 2019
- read: IOPS=53, BW=3453KiB/s (3536kB/s)(405MiB/120005msec)
- slat (nsec): min=0, max=377941k, avg=145377.66, stdev=6367946.16
- clat (nsec): min=0, max=824320k, avg=8948377.22, stdev=45281721.38
- lat (nsec): min=0, max=824512k, avg=9094746.36, stdev=45750151.63
- clat percentiles (nsec):
- | 1.00th=[ 716], 5.00th=[ 732], 10.00th=[ 740],
- | 20.00th=[ 820], 30.00th=[ 924], 40.00th=[ 1048],
- | 50.00th=[ 1192], 60.00th=[ 1384], 70.00th=[ 67072],
- | 80.00th=[ 1138688], 90.00th=[ 7700480], 95.00th=[ 47448064],
- | 99.00th=[233832448], 99.50th=[379584512], 99.90th=[583008256],
- | 99.95th=[599785472], 99.99th=[826277888]
- bw ( KiB/s): min= 0, max= 5171, per=14.18%, avg=3452.50, stdev=2430.33, samples=2
- iops : min= 0, max= 80, avg=53.50, stdev=37.48, samples=2
- write: IOPS=55, BW=3551KiB/s (3637kB/s)(416MiB/120005msec)
- slat (nsec): min=0, max=114884k, avg=206767.42, stdev=1440085.66
- clat (nsec): min=0, max=978894k, avg=63265817.90, stdev=119018938.09
- lat (nsec): min=0, max=979480k, avg=63473688.03, stdev=118978188.84
- clat percentiles (usec):
- | 1.00th=[ 1037], 5.00th=[ 1303], 10.00th=[ 1516], 20.00th=[ 1958],
- | 30.00th=[ 2704], 40.00th=[ 4228], 50.00th=[ 7767], 60.00th=[ 19530],
- | 70.00th=[ 69731], 80.00th=[103285], 90.00th=[156238], 95.00th=[304088],
- | 99.00th=[624952], 99.50th=[708838], 99.90th=[843056], 99.95th=[918553],
- | 99.99th=[977273]
- bw ( KiB/s): min= 0, max= 5238, per=55.52%, avg=3549.00, stdev=2388.61, samples=2
- iops : min= 0, max= 81, avg=55.00, stdev=36.77, samples=2
- lat (nsec) : 750=5.93%, 1000=12.18%
- lat (usec) : 2=14.23%, 4=0.05%, 10=0.21%, 20=0.74%, 50=0.85%
- lat (usec) : 100=0.91%, 250=0.71%, 500=0.76%, 750=0.63%, 1000=1.97%
- lat (msec) : 2=12.87%, 4=11.19%, 10=8.16%, 20=4.69%, 50=4.58%
- lat (msec) : 100=7.81%, 250=7.90%, 500=2.55%, 750=0.96%, 1000=0.14%
- cpu : usr=0.07%, sys=0.32%, ctx=15689, majf=0, minf=30
- IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=6475,6656,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- Run status group 0 (all jobs):
- READ: bw=23.8MiB/s (24.9MB/s), 3453KiB/s-20.4MiB/s (3536kB/s-21.4MB/s), io=2853MiB (2992MB), run=120005-120025msec
- WRITE: bw=6393KiB/s (6546kB/s), 2846KiB/s-3551KiB/s (2915kB/s-3637kB/s), io=750MiB (787MB), run=120005-120177msec
- Disk stats (read/write):
- rbd256: ios=8297/12558, merge=0/2, ticks=571393/819813, in_queue=1380594, util=14.16%
- <example-benchmark-fio-server-294-benchmark-tmh99>
- randwrite: (groupid=0, jobs=1): err= 0: pid=11: Wed May 8 20:34:59 2019
- write: IOPS=20, BW=1329KiB/s (1361kB/s)(157MiB/120857msec)
- slat (nsec): min=0, max=200899k, avg=380636.09, stdev=5677598.13
- clat (nsec): min=0, max=6808.5M, avg=192323685.98, stdev=404033626.27
- lat (nsec): min=0, max=6808.8M, avg=192705644.37, stdev=404002602.14
- clat percentiles (usec):
- | 1.00th=[ 1450], 5.00th=[ 2147], 10.00th=[ 2671],
- | 20.00th=[ 3720], 30.00th=[ 4948], 40.00th=[ 6915],
- | 50.00th=[ 9765], 60.00th=[ 14091], 70.00th=[ 34866],
- | 80.00th=[ 383779], 90.00th=[ 666895], 95.00th=[ 901776],
- | 99.00th=[1451230], 99.50th=[1635779], 99.90th=[4395631],
- | 99.95th=[6811550], 99.99th=[6811550]
- bw ( KiB/s): min= 0, max= 1782, per=26.56%, avg=1334.00, stdev=633.57, samples=2
- iops : min= 0, max= 27, avg=20.00, stdev= 9.90, samples=2
- lat (usec) : 4=0.04%, 250=0.04%, 750=0.04%, 1000=0.12%
- lat (msec) : 2=3.79%, 4=18.48%, 10=28.49%, 20=15.16%, 50=4.51%
- lat (msec) : 100=0.52%, 250=3.87%, 500=8.34%, 750=8.86%, 1000=3.59%
- cpu : usr=0.02%, sys=0.09%, ctx=4422, majf=0, minf=30
- IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=0,2506,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- read: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:34:59 2019
- read: IOPS=201, BW=12.6MiB/s (13.2MB/s)(1520MiB/120584msec)
- slat (nsec): min=0, max=30825k, avg=33031.54, stdev=251563.33
- clat (nsec): min=0, max=1843.3M, avg=19822397.80, stdev=118197969.25
- lat (nsec): min=0, max=1843.3M, avg=19856547.54, stdev=118217208.93
- clat percentiles (nsec):
- | 1.00th=[ 716], 5.00th=[ 732], 10.00th=[ 756],
- | 20.00th=[ 788], 30.00th=[ 828], 40.00th=[ 868],
- | 50.00th=[ 900], 60.00th=[ 964], 70.00th=[ 1240],
- | 80.00th=[ 11712], 90.00th=[ 2703360], 95.00th=[ 12648448],
- | 99.00th=[ 742391808], 99.50th=[ 901775360], 99.90th=[1166016512],
- | 99.95th=[1249902592], 99.99th=[1719664640]
- bw ( KiB/s): min= 0, max=13908, per=78.34%, avg=12965.50, stdev=1332.90, samples=2
- iops : min= 0, max= 217, avg=202.00, stdev=21.21, samples=2
- lat (nsec) : 750=9.20%, 1000=53.40%
- lat (usec) : 2=16.93%, 4=0.19%, 10=0.23%, 20=0.72%, 50=0.21%
- lat (usec) : 100=0.15%, 250=0.12%, 500=0.08%, 750=0.22%, 1000=1.69%
- lat (msec) : 2=5.01%, 4=3.90%, 10=2.58%, 20=1.08%, 50=0.98%
- lat (msec) : 100=0.39%, 250=0.30%, 500=0.68%, 750=0.99%, 1000=0.72%
- cpu : usr=0.07%, sys=0.27%, ctx=9250, majf=0, minf=30
- IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=24313,0,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- readwrite: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:34:59 2019
- read: IOPS=57, BW=3662KiB/s (3750kB/s)(429MiB/120007msec)
- slat (nsec): min=0, max=7169.2k, avg=30253.59, stdev=126833.96
- clat (nsec): min=0, max=745444k, avg=7485190.03, stdev=44225344.30
- lat (nsec): min=0, max=745480k, avg=7516432.89, stdev=44236569.51
- clat percentiles (nsec):
- | 1.00th=[ 716], 5.00th=[ 724], 10.00th=[ 732],
- | 20.00th=[ 764], 30.00th=[ 876], 40.00th=[ 980],
- | 50.00th=[ 1144], 60.00th=[ 1288], 70.00th=[ 14656],
- | 80.00th=[ 749568], 90.00th=[ 2867200], 95.00th=[ 33816576],
- | 99.00th=[208666624], 99.50th=[408944640], 99.90th=[599785472],
- | 99.95th=[624951296], 99.99th=[742391808]
- bw ( KiB/s): min= 0, max= 5649, per=22.13%, avg=3662.00, stdev=2810.04, samples=2
- iops : min= 0, max= 88, avg=57.00, stdev=43.84, samples=2
- write: IOPS=58, BW=3720KiB/s (3809kB/s)(436MiB/120007msec)
- slat (nsec): min=0, max=4886.1k, avg=176323.86, stdev=232612.07
- clat (nsec): min=0, max=1093.2M, avg=61434190.62, stdev=127604648.95
- lat (nsec): min=0, max=1093.5M, avg=61611749.80, stdev=127566168.49
- clat percentiles (usec):
- | 1.00th=[ 1029], 5.00th=[ 1303], 10.00th=[ 1483],
- | 20.00th=[ 1827], 30.00th=[ 2311], 40.00th=[ 3228],
- | 50.00th=[ 5080], 60.00th=[ 10814], 70.00th=[ 35390],
- | 80.00th=[ 91751], 90.00th=[ 175113], 95.00th=[ 350225],
- | 99.00th=[ 608175], 99.50th=[ 692061], 99.90th=[ 910164],
- | 99.95th=[1027605], 99.99th=[1098908]
- bw ( KiB/s): min= 0, max= 5666, per=74.02%, avg=3717.50, stdev=2755.60, samples=2
- iops : min= 0, max= 88, avg=57.50, stdev=43.13, samples=2
- lat (nsec) : 750=8.47%, 1000=12.07%
- lat (usec) : 2=13.36%, 4=0.04%, 10=0.32%, 20=1.28%, 50=0.92%
- lat (usec) : 100=0.85%, 250=1.39%, 500=0.57%, 750=0.47%, 1000=1.69%
- lat (msec) : 2=14.34%, 4=12.44%, 10=7.88%, 20=4.06%, 50=3.80%
- lat (msec) : 100=6.09%, 250=5.90%, 500=2.64%, 750=1.28%, 1000=0.12%
- cpu : usr=0.08%, sys=0.32%, ctx=15155, majf=0, minf=31
- IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=6867,6972,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- Run status group 0 (all jobs):
- READ: bw=16.2MiB/s (16.9MB/s), 3662KiB/s-12.6MiB/s (3750kB/s-13.2MB/s), io=1949MiB (2044MB), run=120007-120584msec
- WRITE: bw=5022KiB/s (5143kB/s), 1329KiB/s-3720KiB/s (1361kB/s-3809kB/s), io=593MiB (622MB), run=120007-120857msec
- Disk stats (read/write):
- rbd98: ios=6656/10952, merge=0/2, ticks=583320/912510, in_queue=1486887, util=5.81%
- <example-benchmark-fio-server-977-benchmark-wgh5n>
- randwrite: (groupid=0, jobs=1): err= 0: pid=11: Wed May 8 20:35:00 2019
- write: IOPS=33, BW=2177KiB/s (2229kB/s)(256MiB/120220msec)
- slat (nsec): min=0, max=1172.5M, avg=774123.61, stdev=23284904.99
- clat (nsec): min=0, max=1437.9M, avg=117417416.74, stdev=205641999.08
- lat (nsec): min=0, max=1437.9M, avg=118192524.39, stdev=206589593.23
- clat percentiles (usec):
- | 1.00th=[ 1004], 5.00th=[ 1237], 10.00th=[ 1434],
- | 20.00th=[ 1942], 30.00th=[ 2868], 40.00th=[ 5342],
- | 50.00th=[ 13173], 60.00th=[ 60031], 70.00th=[ 103285],
- | 80.00th=[ 179307], 90.00th=[ 375391], 95.00th=[ 574620],
- | 99.00th=[ 935330], 99.50th=[1052771], 99.90th=[1182794],
- | 99.95th=[1333789], 99.99th=[1434452]
- bw ( KiB/s): min= 0, max= 2643, per=42.38%, avg=2178.50, stdev=656.90, samples=2
- iops : min= 0, max= 41, avg=33.50, stdev=10.61, samples=2
- lat (usec) : 750=0.05%, 1000=0.86%
- lat (msec) : 2=20.00%, 4=15.10%, 10=11.63%, 20=6.61%, 50=5.02%
- lat (msec) : 100=9.64%, 250=15.47%, 500=9.08%, 750=3.65%, 1000=2.28%
- cpu : usr=0.05%, sys=0.15%, ctx=7711, majf=0, minf=31
- IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=0,4086,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- read: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:35:00 2019
- read: IOPS=434, BW=27.1MiB/s (28.4MB/s)(3256MiB/120173msec)
- slat (nsec): min=0, max=1168.0M, avg=69320.30, stdev=6851183.35
- clat (nsec): min=0, max=1338.9M, avg=9166220.50, stdev=60043152.24
- lat (nsec): min=0, max=1338.9M, avg=9236403.72, stdev=60436829.32
- clat percentiles (nsec):
- | 1.00th=[ 708], 5.00th=[ 708], 10.00th=[ 716],
- | 20.00th=[ 724], 30.00th=[ 732], 40.00th=[ 772],
- | 50.00th=[ 812], 60.00th=[ 860], 70.00th=[ 900],
- | 80.00th=[ 1160], 90.00th=[ 1028096], 95.00th=[ 20054016],
- | 99.00th=[ 308281344], 99.50th=[ 476053504], 99.90th=[ 775946240],
- | 99.95th=[ 876609536], 99.99th=[1098907648]
- bw ( KiB/s): min= 0, max=43740, per=90.78%, avg=27779.00, stdev=22572.26, samples=2
- iops : min= 0, max= 684, avg=434.00, stdev=353.55, samples=2
- lat (nsec) : 750=34.58%, 1000=41.49%
- lat (usec) : 2=9.00%, 4=0.09%, 10=0.55%, 20=1.01%, 50=0.38%
- lat (usec) : 100=0.11%, 250=0.08%, 500=0.40%, 750=0.68%, 1000=1.52%
- lat (msec) : 2=2.37%, 4=1.52%, 10=0.76%, 20=0.48%, 50=1.41%
- lat (msec) : 100=1.42%, 250=0.92%, 500=0.81%, 750=0.30%, 1000=0.11%
- cpu : usr=0.13%, sys=0.46%, ctx=13251, majf=0, minf=38
- IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=52164,0,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- readwrite: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:35:00 2019
- read: IOPS=44, BW=2864KiB/s (2933kB/s)(336MiB/120003msec)
- slat (nsec): min=0, max=816568k, avg=185923.53, stdev=11141851.34
- clat (nsec): min=0, max=724201k, avg=8174567.07, stdev=43029992.72
- lat (nsec): min=0, max=817041k, avg=8361466.82, stdev=44428754.62
- clat percentiles (nsec):
- | 1.00th=[ 716], 5.00th=[ 724], 10.00th=[ 740],
- | 20.00th=[ 820], 30.00th=[ 900], 40.00th=[ 1032],
- | 50.00th=[ 1176], 60.00th=[ 1352], 70.00th=[ 32640],
- | 80.00th=[ 856064], 90.00th=[ 4620288], 95.00th=[ 39059456],
- | 99.00th=[235929600], 99.50th=[320864256], 99.90th=[517996544],
- | 99.95th=[633339904], 99.99th=[725614592]
- bw ( KiB/s): min= 0, max= 4423, per=9.36%, avg=2863.50, stdev=2205.47, samples=2
- iops : min= 0, max= 69, avg=44.50, stdev=34.65, samples=2
- write: IOPS=46, BW=2970KiB/s (3041kB/s)(348MiB/120003msec)
- slat (nsec): min=0, max=54255k, avg=183966.16, stdev=752724.15
- clat (nsec): min=0, max=1040.4M, avg=78172935.18, stdev=141799380.38
- lat (nsec): min=0, max=1040.4M, avg=78358082.12, stdev=141760902.55
- clat percentiles (usec):
- | 1.00th=[ 1004], 5.00th=[ 1237], 10.00th=[ 1483],
- | 20.00th=[ 2089], 30.00th=[ 3064], 40.00th=[ 5735],
- | 50.00th=[ 13829], 60.00th=[ 30278], 70.00th=[ 74974],
- | 80.00th=[ 110625], 90.00th=[ 229639], 95.00th=[ 387974],
- | 99.00th=[ 725615], 99.50th=[ 801113], 99.90th=[ 918553],
- | 99.95th=[ 952108], 99.99th=[1044382]
- bw ( KiB/s): min= 0, max= 4562, per=57.73%, avg=2967.50, stdev=2254.96, samples=2
- iops : min= 0, max= 71, avg=46.00, stdev=35.36, samples=2
- lat (nsec) : 750=5.63%, 1000=13.06%
- lat (usec) : 2=14.03%, 4=0.06%, 10=0.30%, 20=0.90%, 50=0.85%
- lat (usec) : 100=0.80%, 250=0.84%, 500=0.92%, 750=1.02%, 1000=2.10%
- lat (msec) : 2=11.43%, 4=9.95%, 10=6.71%, 20=5.17%, 50=6.26%
- lat (msec) : 100=7.18%, 250=7.72%, 500=3.57%, 750=1.09%, 1000=0.42%
- cpu : usr=0.07%, sys=0.26%, ctx=12751, majf=0, minf=33
- IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=5371,5565,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- Run status group 0 (all jobs):
- READ: bw=29.9MiB/s (31.3MB/s), 2864KiB/s-27.1MiB/s (2933kB/s-28.4MB/s), io=3591MiB (3766MB), run=120003-120173msec
- WRITE: bw=5141KiB/s (5264kB/s), 2177KiB/s-2970KiB/s (2229kB/s-3041kB/s), io=604MiB (633MB), run=120003-120220msec
- Disk stats (read/write):
- rbd324: ios=8914/10591, merge=0/1, ticks=549035/867403, in_queue=1406550, util=12.49%
- <example-benchmark-fio-server-138-benchmark-9r9wf>
- randwrite: (groupid=0, jobs=1): err= 0: pid=11: Wed May 8 20:35:00 2019
- write: IOPS=46, BW=2976KiB/s (3047kB/s)(349MiB/120102msec)
- slat (nsec): min=0, max=624921k, avg=477947.36, stdev=12874979.64
- clat (nsec): min=0, max=1497.6M, avg=85883724.77, stdev=166458648.87
- lat (nsec): min=0, max=1497.6M, avg=86362829.03, stdev=167067375.65
- clat percentiles (usec):
- | 1.00th=[ 1004], 5.00th=[ 1172], 10.00th=[ 1336],
- | 20.00th=[ 1745], 30.00th=[ 2442], 40.00th=[ 3752],
- | 50.00th=[ 7046], 60.00th=[ 27657], 70.00th=[ 88605],
- | 80.00th=[ 113771], 90.00th=[ 242222], 95.00th=[ 476054],
- | 99.00th=[ 759170], 99.50th=[ 943719], 99.90th=[1300235],
- | 99.95th=[1468007], 99.99th=[1501561]
- bw ( KiB/s): min= 0, max= 3030, per=42.42%, avg=2975.50, stdev=77.07, samples=2
- iops : min= 0, max= 47, avg=46.00, stdev= 1.41, samples=2
- lat (usec) : 4=0.02%, 500=0.02%, 750=0.02%, 1000=0.88%
- lat (msec) : 2=23.29%, 4=17.02%, 10=12.20%, 20=5.36%, 50=3.10%
- lat (msec) : 100=12.97%, 250=15.39%, 500=5.36%, 750=3.39%, 1000=0.75%
- cpu : usr=0.05%, sys=0.21%, ctx=10541, majf=0, minf=29
- IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=0,5581,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- read: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:35:00 2019
- read: IOPS=377, BW=23.6MiB/s (24.7MB/s)(2830MiB/120105msec)
- slat (nsec): min=0, max=473114k, avg=47081.33, stdev=3532967.60
- clat (nsec): min=0, max=1438.1M, avg=10554888.20, stdev=60565769.03
- lat (nsec): min=0, max=1438.2M, avg=10602804.25, stdev=60674181.23
- clat percentiles (nsec):
- | 1.00th=[ 708], 5.00th=[ 716], 10.00th=[ 716],
- | 20.00th=[ 724], 30.00th=[ 732], 40.00th=[ 748],
- | 50.00th=[ 820], 60.00th=[ 876], 70.00th=[ 932],
- | 80.00th=[ 1352], 90.00th=[ 1843200], 95.00th=[ 50069504],
- | 99.00th=[ 329252864], 99.50th=[ 476053504], 99.90th=[ 734003200],
- | 99.95th=[ 868220928], 99.99th=[1199570944]
- bw ( KiB/s): min= 0, max=35189, per=85.90%, avg=24145.50, stdev=15617.87, samples=2
- iops : min= 0, max= 551, avg=377.50, stdev=245.37, samples=2
- lat (nsec) : 750=40.49%, 1000=32.27%
- lat (usec) : 2=8.76%, 4=0.06%, 10=1.83%, 20=0.98%, 50=0.34%
- lat (usec) : 100=0.11%, 250=0.07%, 500=0.22%, 750=0.70%, 1000=1.89%
- lat (msec) : 2=2.58%, 4=1.61%, 10=0.93%, 20=0.33%, 50=1.87%
- lat (msec) : 100=2.63%, 250=1.04%, 500=0.88%, 750=0.35%, 1000=0.06%
- cpu : usr=0.10%, sys=0.41%, ctx=13738, majf=0, minf=51
- IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=45385,0,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- readwrite: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:35:00 2019
- read: IOPS=62, BW=3986KiB/s (4082kB/s)(467MiB/120016msec)
- slat (nsec): min=0, max=516982k, avg=103312.94, stdev=5980122.37
- clat (nsec): min=0, max=999725k, avg=7809295.40, stdev=43750017.09
- lat (nsec): min=0, max=999792k, avg=7913667.08, stdev=44149149.06
- clat percentiles (nsec):
- | 1.00th=[ 724], 5.00th=[ 732], 10.00th=[ 740],
- | 20.00th=[ 820], 30.00th=[ 916], 40.00th=[ 1048],
- | 50.00th=[ 1192], 60.00th=[ 1368], 70.00th=[ 78336],
- | 80.00th=[ 1089536], 90.00th=[ 5144576], 95.00th=[ 39583744],
- | 99.00th=[ 189792256], 99.50th=[ 329252864], 99.90th=[ 513802240],
- | 99.95th=[ 817889280], 99.99th=[1002438656]
- bw ( KiB/s): min= 0, max= 5655, per=14.18%, avg=3985.50, stdev=2361.03, samples=2
- iops : min= 0, max= 88, avg=62.00, stdev=36.77, samples=2
- write: IOPS=63, BW=4042KiB/s (4139kB/s)(474MiB/120016msec)
- slat (nsec): min=0, max=633040k, avg=353080.90, stdev=9365055.55
- clat (nsec): min=0, max=1236.7M, avg=55364701.88, stdev=121922412.63
- lat (nsec): min=0, max=1236.7M, avg=55718902.59, stdev=122175546.16
- clat percentiles (usec):
- | 1.00th=[ 996], 5.00th=[ 1221], 10.00th=[ 1385],
- | 20.00th=[ 1762], 30.00th=[ 2311], 40.00th=[ 3130],
- | 50.00th=[ 5014], 60.00th=[ 11076], 70.00th=[ 28181],
- | 80.00th=[ 85459], 90.00th=[ 133694], 95.00th=[ 299893],
- | 99.00th=[ 650118], 99.50th=[ 725615], 99.90th=[ 918553],
- | 99.95th=[ 977273], 99.99th=[1233126]
- bw ( KiB/s): min= 0, max= 5676, per=57.60%, avg=4040.00, stdev=2313.65, samples=2
- iops : min= 0, max= 88, avg=62.50, stdev=36.06, samples=2
- lat (nsec) : 750=6.30%, 1000=11.97%
- lat (usec) : 2=14.31%, 4=0.12%, 10=0.32%, 20=0.72%, 50=0.60%
- lat (usec) : 100=0.66%, 250=0.88%, 500=0.72%, 750=0.75%, 1000=2.44%
- lat (msec) : 2=15.07%, 4=12.30%, 10=7.92%, 20=4.76%, 50=4.62%
- lat (msec) : 100=6.83%, 250=5.36%, 500=2.17%, 750=1.00%, 1000=0.17%
- cpu : usr=0.10%, sys=0.35%, ctx=18104, majf=0, minf=36
- IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=7476,7576,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- Run status group 0 (all jobs):
- READ: bw=27.5MiB/s (28.8MB/s), 3986KiB/s-23.6MiB/s (4082kB/s-24.7MB/s), io=3297MiB (3457MB), run=120016-120105msec
- WRITE: bw=7014KiB/s (7183kB/s), 2976KiB/s-4042KiB/s (3047kB/s-4139kB/s), io=823MiB (863MB), run=120016-120102msec
- Disk stats (read/write):
- rbd40: ios=9658/13474, merge=0/1, ticks=566891/831540, in_queue=1386680, util=15.13%
- Jobs: 28 (f=28): [_(2),m(1)][99.6%][r=1044M,w=1019M][r=16.3k,w=15.9k IOPS][eta 00m:50s] <example-benchmark-fio-server-446-benchmark-2rtd4>
- randwrite: (groupid=0, jobs=1): err= 0: pid=11: Wed May 8 20:35:00 2019
- write: IOPS=14, BW=956KiB/s (979kB/s)(112MiB/120085msec)
- slat (nsec): min=0, max=899001k, avg=734283.87, stdev=21245487.71
- clat (nsec): min=0, max=3218.8M, avg=267401220.62, stdev=389066586.33
- lat (nsec): min=0, max=3218.0M, avg=268136022.10, stdev=389320204.11
- clat percentiles (usec):
- | 1.00th=[ 1663], 5.00th=[ 2737], 10.00th=[ 3720],
- | 20.00th=[ 5407], 30.00th=[ 8029], 40.00th=[ 12256],
- | 50.00th=[ 20055], 60.00th=[ 84411], 70.00th=[ 383779],
- | 80.00th=[ 650118], 90.00th=[ 826278], 95.00th=[ 985662],
- | 99.00th=[1484784], 99.50th=[1635779], 99.90th=[3170894],
- | 99.95th=[3204449], 99.99th=[3204449]
- bw ( KiB/s): min= 0, max= 1412, per=25.82%, avg=1412.00, stdev= 0.00, samples=1
- iops : min= 0, max= 22, avg=22.00, stdev= 0.00, samples=1
- lat (usec) : 250=0.06%, 1000=0.06%
- lat (msec) : 2=2.01%, 4=9.49%, 10=23.90%, 20=14.57%, 50=8.21%
- lat (msec) : 100=2.51%, 250=5.36%, 500=7.82%, 750=13.01%, 1000=8.43%
- cpu : usr=0.02%, sys=0.07%, ctx=3024, majf=0, minf=30
- IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=0,1791,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- read: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:35:00 2019
- read: IOPS=168, BW=10.5MiB/s (11.0MB/s)(1266MiB/120432msec)
- slat (nsec): min=0, max=23034k, avg=26427.91, stdev=210110.57
- clat (nsec): min=0, max=1849.9M, avg=23799251.42, stdev=130255782.34
- lat (nsec): min=0, max=1849.0M, avg=23826708.01, stdev=130277592.76
- clat percentiles (nsec):
- | 1.00th=[ 716], 5.00th=[ 724], 10.00th=[ 740],
- | 20.00th=[ 780], 30.00th=[ 796], 40.00th=[ 836],
- | 50.00th=[ 868], 60.00th=[ 892], 70.00th=[ 956],
- | 80.00th=[ 1256], 90.00th=[ 1482752], 95.00th=[ 20578304],
- | 99.00th=[ 784334848], 99.50th=[ 926941184], 99.90th=[1182793728],
- | 99.95th=[1333788672], 99.99th=[1619001344]
- bw ( KiB/s): min= 0, max=11333, per=70.98%, avg=10797.00, stdev=758.02, samples=2
- iops : min= 0, max= 177, avg=168.50, stdev=12.02, samples=2
- lat (nsec) : 750=12.58%, 1000=60.81%
- lat (usec) : 2=12.84%, 4=0.23%, 10=0.21%, 20=1.00%, 50=0.10%
- lat (usec) : 100=0.08%, 250=0.07%, 500=0.04%, 750=0.26%, 1000=0.62%
- lat (msec) : 2=2.02%, 4=1.88%, 10=1.45%, 20=0.80%, 50=1.01%
- lat (msec) : 100=0.40%, 250=0.42%, 500=0.79%, 750=1.25%, 1000=0.82%
- cpu : usr=0.06%, sys=0.19%, ctx=5020, majf=0, minf=29
- IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=20248,0,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- readwrite: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:35:00 2019
- read: IOPS=69, BW=4467KiB/s (4574kB/s)(524MiB/120003msec)
- slat (nsec): min=0, max=8029.9k, avg=25453.88, stdev=121721.00
- clat (nsec): min=0, max=668541k, avg=6380971.94, stdev=41066870.55
- lat (nsec): min=0, max=668579k, avg=6407382.95, stdev=41072836.25
- clat percentiles (nsec):
- | 1.00th=[ 716], 5.00th=[ 724], 10.00th=[ 732],
- | 20.00th=[ 756], 30.00th=[ 836], 40.00th=[ 924],
- | 50.00th=[ 1064], 60.00th=[ 1224], 70.00th=[ 1512],
- | 80.00th=[ 97792], 90.00th=[ 1433600], 95.00th=[ 11862016],
- | 99.00th=[206569472], 99.50th=[341835776], 99.90th=[541065216],
- | 99.95th=[599785472], 99.99th=[666894336]
- bw ( KiB/s): min= 0, max= 7498, per=29.36%, avg=4466.50, stdev=4287.19, samples=2
- iops : min= 0, max= 117, avg=69.50, stdev=67.18, samples=2
- write: IOPS=70, BW=4515KiB/s (4623kB/s)(529MiB/120003msec)
- slat (nsec): min=0, max=11441k, avg=189852.16, stdev=277675.89
- clat (nsec): min=0, max=970672k, avg=50276865.16, stdev=111353429.62
- lat (nsec): min=0, max=970761k, avg=50467942.12, stdev=111311973.94
- clat percentiles (usec):
- | 1.00th=[ 971], 5.00th=[ 1188], 10.00th=[ 1385], 20.00th=[ 1762],
- | 30.00th=[ 2311], 40.00th=[ 3228], 50.00th=[ 4817], 60.00th=[ 9241],
- | 70.00th=[ 20579], 80.00th=[ 68682], 90.00th=[139461], 95.00th=[287310],
- | 99.00th=[574620], 99.50th=[675283], 99.90th=[834667], 99.95th=[918553],
- | 99.99th=[968885]
- bw ( KiB/s): min= 0, max= 7545, per=82.53%, avg=4512.50, stdev=4288.60, samples=2
- iops : min= 0, max= 117, avg=70.00, stdev=66.47, samples=2
- lat (nsec) : 750=9.66%, 1000=13.03%
- lat (usec) : 2=13.06%, 4=0.08%, 10=0.62%, 20=1.50%, 50=0.99%
- lat (usec) : 100=0.87%, 250=1.65%, 500=0.76%, 750=0.59%, 1000=1.67%
- lat (msec) : 2=13.33%, 4=11.49%, 10=8.41%, 20=4.92%, 50=4.16%
- lat (msec) : 100=4.61%, 250=5.36%, 500=2.45%, 750=0.65%, 1000=0.14%
- cpu : usr=0.09%, sys=0.39%, ctx=17391, majf=0, minf=30
- IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=8376,8463,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- Run status group 0 (all jobs):
- READ: bw=14.9MiB/s (15.6MB/s), 4467KiB/s-10.5MiB/s (4574kB/s-11.0MB/s), io=1789MiB (1876MB), run=120003-120432msec
- WRITE: bw=5468KiB/s (5599kB/s), 956KiB/s-4515KiB/s (979kB/s-4623kB/s), io=641MiB (672MB), run=120003-120085msec
- Disk stats (read/write):
- rbd126: ios=3914/10574, merge=0/2, ticks=578140/913384, in_queue=1484315, util=4.51%
- <example-benchmark-fio-server-185-benchmark-lp9ct>
- randwrite: (groupid=0, jobs=1): err= 0: pid=11: Wed May 8 20:35:00 2019
- write: IOPS=39, BW=2553KiB/s (2614kB/s)(301MiB/120774msec)
- slat (nsec): min=0, max=35777k, avg=174425.73, stdev=601217.79
- clat (nsec): min=0, max=8251.2M, avg=100036644.27, stdev=335178831.09
- lat (nsec): min=0, max=8251.3M, avg=100212323.58, stdev=335174890.69
- clat percentiles (usec):
- | 1.00th=[ 750], 5.00th=[ 1237], 10.00th=[ 1778],
- | 20.00th=[ 3261], 30.00th=[ 5080], 40.00th=[ 7701],
- | 50.00th=[ 11731], 60.00th=[ 19268], 70.00th=[ 40109],
- | 80.00th=[ 139461], 90.00th=[ 278922], 95.00th=[ 459277],
- | 99.00th=[ 759170], 99.50th=[1300235], 99.90th=[4395631],
- | 99.95th=[8220836], 99.99th=[8220836]
- bw ( KiB/s): min= 0, max= 3221, per=42.04%, avg=2567.00, stdev=924.90, samples=2
- iops : min= 0, max= 50, avg=39.50, stdev=14.85, samples=2
- lat (usec) : 4=0.02%, 50=0.02%, 500=0.06%, 750=0.91%, 1000=2.04%
- lat (msec) : 2=8.64%, 4=13.08%, 10=22.28%, 20=13.46%, 50=11.71%
- lat (msec) : 100=4.82%, 250=10.55%, 500=8.41%, 750=2.99%, 1000=0.27%
- cpu : usr=0.04%, sys=0.14%, ctx=7681, majf=0, minf=30
- IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=0,4815,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- read: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:35:00 2019
- read: IOPS=68, BW=4361KiB/s (4465kB/s)(513MiB/120497msec)
- slat (nsec): min=0, max=21635k, avg=53416.96, stdev=327483.41
- clat (nsec): min=0, max=1829.5M, avg=58787431.59, stdev=199791627.08
- lat (nsec): min=0, max=1829.5M, avg=58841997.19, stdev=199805850.51
- clat percentiles (nsec):
- | 1.00th=[ 716], 5.00th=[ 732], 10.00th=[ 772],
- | 20.00th=[ 820], 30.00th=[ 884], 40.00th=[ 1004],
- | 50.00th=[ 1240], 60.00th=[ 1528], 70.00th=[ 1318912],
- | 80.00th=[ 3620864], 90.00th=[ 40108032], 95.00th=[ 608174080],
- | 99.00th=[ 968884224], 99.50th=[1082130432], 99.90th=[1350565888],
- | 99.95th=[1468006400], 99.99th=[1837105152]
- bw ( KiB/s): min= 0, max= 4444, per=55.77%, avg=4375.50, stdev=96.87, samples=2
- iops : min= 0, max= 69, avg=68.00, stdev= 1.41, samples=2
- lat (nsec) : 750=7.03%, 1000=32.83%
- lat (usec) : 2=22.27%, 4=0.26%, 10=0.12%, 20=0.39%, 50=0.15%
- lat (usec) : 100=0.15%, 250=0.06%, 500=0.07%, 750=0.77%, 1000=2.62%
- lat (msec) : 2=7.37%, 4=6.77%, 10=5.08%, 20=2.33%, 50=2.17%
- lat (msec) : 100=0.73%, 250=0.88%, 500=1.93%, 750=3.19%, 1000=1.99%
- cpu : usr=0.04%, sys=0.10%, ctx=6071, majf=0, minf=30
- IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=8207,0,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- readwrite: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:35:00 2019
- read: IOPS=54, BW=3500KiB/s (3584kB/s)(410MiB/120005msec)
- slat (nsec): min=0, max=9183.3k, avg=39914.76, stdev=142373.27
- clat (nsec): min=0, max=749534k, avg=18005844.75, stdev=76443612.62
- lat (nsec): min=0, max=749561k, avg=18046837.72, stdev=76451706.73
- clat percentiles (nsec):
- | 1.00th=[ 716], 5.00th=[ 732], 10.00th=[ 740],
- | 20.00th=[ 852], 30.00th=[ 996], 40.00th=[ 1160],
- | 50.00th=[ 1304], 60.00th=[ 35072], 70.00th=[ 995328],
- | 80.00th=[ 2441216], 90.00th=[ 29491200], 95.00th=[ 71827456],
- | 99.00th=[463470592], 99.50th=[608174080], 99.90th=[683671552],
- | 99.95th=[725614592], 99.99th=[750780416]
- bw ( KiB/s): min= 0, max= 5429, per=44.60%, avg=3499.00, stdev=2729.43, samples=2
- iops : min= 0, max= 84, avg=54.00, stdev=42.43, samples=2
- write: IOPS=55, BW=3576KiB/s (3662kB/s)(419MiB/120005msec)
- slat (nsec): min=0, max=5692.4k, avg=182624.16, stdev=237777.68
- clat (nsec): min=0, max=1091.5M, avg=53980894.32, stdev=112363439.06
- lat (nsec): min=0, max=1091.7M, avg=54164773.18, stdev=112319425.21
- clat percentiles (usec):
- | 1.00th=[ 1037], 5.00th=[ 1237], 10.00th=[ 1418],
- | 20.00th=[ 1762], 30.00th=[ 2212], 40.00th=[ 3130],
- | 50.00th=[ 4752], 60.00th=[ 10159], 70.00th=[ 27132],
- | 80.00th=[ 78119], 90.00th=[ 162530], 95.00th=[ 304088],
- | 99.00th=[ 541066], 99.50th=[ 599786], 99.90th=[ 792724],
- | 99.95th=[ 817890], 99.99th=[1098908]
- bw ( KiB/s): min= 0, max= 5484, per=58.55%, avg=3575.00, stdev=2699.73, samples=2
- iops : min= 0, max= 85, avg=55.50, stdev=41.72, samples=2
- lat (nsec) : 750=5.61%, 1000=9.29%
- lat (usec) : 2=13.91%, 4=0.05%, 10=0.22%, 20=0.48%, 50=0.32%
- lat (usec) : 100=0.38%, 250=0.49%, 500=0.34%, 750=1.14%, 1000=2.80%
- lat (msec) : 2=16.52%, 4=13.11%, 10=9.01%, 20=4.52%, 50=5.64%
- lat (msec) : 100=6.54%, 250=4.89%, 500=3.69%, 750=1.02%, 1000=0.05%
- cpu : usr=0.08%, sys=0.31%, ctx=17110, majf=0, minf=31
- IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=6561,6704,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- Run status group 0 (all jobs):
- READ: bw=7846KiB/s (8035kB/s), 3500KiB/s-4361KiB/s (3584kB/s-4465kB/s), io=923MiB (968MB), run=120005-120497msec
- WRITE: bw=6106KiB/s (6253kB/s), 2553KiB/s-3576KiB/s (2614kB/s-3662kB/s), io=720MiB (755MB), run=120005-120774msec
- Disk stats (read/write):
- rbd73: ios=5816/13073, merge=0/3, ticks=653844/876526, in_queue=1520869, util=4.37%
- <example-benchmark-fio-server-637-benchmark-4fhxd>
- <example-benchmark-fio-server-567-benchmark-tkk6d>
- <example-benchmark-fio-server-155-benchmark-scltw>
- randwrite: (groupid=0, jobs=1): err= 0: pid=11: Wed May 8 20:35:00 2019
- write: IOPS=47, BW=3031KiB/s (3104kB/s)(355MiB/120027msec)
- slat (nsec): min=0, max=713321k, avg=471599.79, stdev=12371015.09
- clat (nsec): min=0, max=1162.4M, avg=84194593.79, stdev=147118350.80
- lat (nsec): min=0, max=1297.7M, avg=84667236.21, stdev=147923628.18
- clat percentiles (usec):
- | 1.00th=[ 963], 5.00th=[ 1123], 10.00th=[ 1287],
- | 20.00th=[ 1680], 30.00th=[ 2409], 40.00th=[ 4424],
- | 50.00th=[ 14222], 60.00th=[ 53216], 70.00th=[ 91751],
- | 80.00th=[ 119014], 90.00th=[ 235930], 95.00th=[ 425722],
- | 99.00th=[ 742392], 99.50th=[ 834667], 99.90th=[ 943719],
- | 99.95th=[ 985662], 99.99th=[1166017]
- bw ( KiB/s): min= 0, max= 1897, per=27.66%, avg=1897.00, stdev= 0.00, samples=1
- iops : min= 0, max= 29, avg=29.00, stdev= 0.00, samples=1
- lat (usec) : 500=0.04%, 750=0.09%, 1000=1.67%
- lat (msec) : 2=23.76%, 4=13.08%, 10=8.50%, 20=4.84%, 50=7.22%
- lat (msec) : 100=14.15%, 250=17.35%, 500=6.04%, 750=2.38%, 1000=0.92%
- cpu : usr=0.06%, sys=0.20%, ctx=10812, majf=0, minf=28
- IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=0,5682,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- read: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:35:00 2019
- read: IOPS=331, BW=20.7MiB/s (21.7MB/s)(2485MiB/120014msec)
- slat (nsec): min=0, max=714213k, avg=59830.16, stdev=4581236.52
- clat (nsec): min=0, max=1125.4M, avg=12024678.97, stdev=58728270.03
- lat (nsec): min=0, max=1125.4M, avg=12085365.90, stdev=58924449.76
- clat percentiles (nsec):
- | 1.00th=[ 708], 5.00th=[ 716], 10.00th=[ 724],
- | 20.00th=[ 732], 30.00th=[ 748], 40.00th=[ 820],
- | 50.00th=[ 860], 60.00th=[ 932], 70.00th=[ 1128],
- | 80.00th=[ 309248], 90.00th=[ 9633792], 95.00th=[ 55836672],
- | 99.00th=[341835776], 99.50th=[467664896], 99.90th=[675282944],
- | 99.95th=[742391808], 99.99th=[968884224]
- bw ( KiB/s): min= 0, max=28049, per=84.99%, avg=21206.50, stdev=9676.76, samples=2
- iops : min= 0, max= 438, avg=331.00, stdev=151.32, samples=2
- lat (nsec) : 750=30.11%, 1000=35.20%
- lat (usec) : 2=13.03%, 4=0.06%, 10=0.82%, 20=0.40%, 50=0.14%
- lat (usec) : 100=0.07%, 250=0.07%, 500=0.35%, 750=1.07%, 1000=2.45%
- lat (msec) : 2=3.18%, 4=1.99%, 10=1.08%, 20=0.75%, 50=3.52%
- lat (msec) : 100=2.76%, 250=1.48%, 500=1.06%, 750=0.34%, 1000=0.04%
- cpu : usr=0.12%, sys=0.38%, ctx=15922, majf=0, minf=37
- IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=39785,0,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- readwrite: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:35:00 2019
- read: IOPS=58, BW=3746KiB/s (3836kB/s)(439MiB/120005msec)
- slat (nsec): min=0, max=9850.8k, avg=37565.76, stdev=156835.35
- clat (nsec): min=0, max=680660k, avg=6520405.48, stdev=33535024.37
- lat (nsec): min=0, max=680698k, avg=6558922.65, stdev=33543170.85
- clat percentiles (nsec):
- | 1.00th=[ 716], 5.00th=[ 732], 10.00th=[ 740],
- | 20.00th=[ 820], 30.00th=[ 924], 40.00th=[ 1064],
- | 50.00th=[ 1208], 60.00th=[ 1416], 70.00th=[ 142336],
- | 80.00th=[ 1073152], 90.00th=[ 5013504], 95.00th=[ 34865152],
- | 99.00th=[168820736], 99.50th=[258998272], 99.90th=[455081984],
- | 99.95th=[505413632], 99.99th=[683671552]
- bw ( KiB/s): min= 0, max= 5781, per=15.01%, avg=3745.50, stdev=2878.63, samples=2
- iops : min= 0, max= 90, avg=58.00, stdev=45.25, samples=2
- write: IOPS=59, BW=3828KiB/s (3920kB/s)(449MiB/120005msec)
- slat (nsec): min=0, max=340981k, avg=298619.27, stdev=5238520.33
- clat (nsec): min=0, max=1050.3M, avg=60301551.27, stdev=126652467.05
- lat (nsec): min=0, max=1050.4M, avg=60601269.70, stdev=126682799.90
- clat percentiles (usec):
- | 1.00th=[ 1004], 5.00th=[ 1221], 10.00th=[ 1401],
- | 20.00th=[ 1778], 30.00th=[ 2343], 40.00th=[ 3425],
- | 50.00th=[ 5669], 60.00th=[ 11731], 70.00th=[ 39060],
- | 80.00th=[ 92799], 90.00th=[ 158335], 95.00th=[ 341836],
- | 99.00th=[ 633340], 99.50th=[ 775947], 99.90th=[ 943719],
- | 99.95th=[ 943719], 99.99th=[1052771]
- bw ( KiB/s): min= 0, max= 5822, per=55.78%, avg=3825.50, stdev=2823.48, samples=2
- iops : min= 0, max= 90, avg=59.00, stdev=43.84, samples=2
- lat (nsec) : 750=5.86%, 1000=11.94%
- lat (usec) : 2=13.77%, 4=0.07%, 10=0.26%, 20=0.95%, 50=0.75%
- lat (usec) : 100=0.71%, 250=0.82%, 500=1.11%, 750=1.04%, 1000=2.37%
- lat (msec) : 2=15.03%, 4=11.20%, 10=9.35%, 20=3.85%, 50=5.03%
- lat (msec) : 100=5.85%, 250=6.41%, 500=2.51%, 750=0.85%, 1000=0.30%
- cpu : usr=0.09%, sys=0.32%, ctx=17167, majf=0, minf=35
- IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=7025,7175,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- Run status group 0 (all jobs):
- READ: bw=24.4MiB/s (25.6MB/s), 3746KiB/s-20.7MiB/s (3836kB/s-21.7MB/s), io=2924MiB (3066MB), run=120005-120014msec
- WRITE: bw=6859KiB/s (7023kB/s), 3031KiB/s-3828KiB/s (3104kB/s-3920kB/s), io=804MiB (843MB), run=120005-120027msec
- Disk stats (read/write):
- rbd236: ios=11231/13877, merge=0/1, ticks=555717/840618, in_queue=1383488, util=16.97%
- randwrite: (groupid=0, jobs=1): err= 0: pid=11: Wed May 8 20:35:00 2019
- write: IOPS=15, BW=1000KiB/s (1024kB/s)(118MiB/120408msec)
- slat (nsec): min=0, max=980064k, avg=1663581.52, stdev=35294572.35
- clat (nsec): min=0, max=2257.3M, avg=255484057.39, stdev=465350314.36
- lat (nsec): min=0, max=2257.8M, avg=257147165.96, stdev=465938509.66
- clat percentiles (usec):
- | 1.00th=[ 1237], 5.00th=[ 1500], 10.00th=[ 1844],
- | 20.00th=[ 2737], 30.00th=[ 3884], 40.00th=[ 6128],
- | 50.00th=[ 10421], 60.00th=[ 21627], 70.00th=[ 49546],
- | 80.00th=[ 675283], 90.00th=[1010828], 95.00th=[1367344],
- | 99.00th=[1669333], 99.50th=[1887437], 99.90th=[2164261],
- | 99.95th=[2264925], 99.99th=[2264925]
- bw ( KiB/s): min= 0, max= 1497, per=21.63%, avg=1000.50, stdev=702.16, samples=2
- iops : min= 0, max= 23, avg=15.00, stdev=11.31, samples=2
- lat (usec) : 50=0.05%, 250=0.05%
- lat (msec) : 2=12.30%, 4=18.48%, 10=18.74%, 20=9.11%, 50=11.77%
- lat (msec) : 100=5.27%, 250=0.53%, 500=0.75%, 750=4.79%, 1000=7.72%
- cpu : usr=0.03%, sys=0.07%, ctx=3491, majf=0, minf=30
- IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=0,1878,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- read: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:35:00 2019
- read: IOPS=627, BW=39.0MiB/s (40.9MB/s)(4684MiB/120072msec)
- slat (nsec): min=0, max=979374k, avg=48936.40, stdev=5532284.90
- clat (nsec): min=0, max=2121.5M, avg=6347450.41, stdev=70775025.09
- lat (nsec): min=0, max=2121.5M, avg=6397358.95, stdev=71004998.05
- clat percentiles (nsec):
- | 1.00th=[ 708], 5.00th=[ 716], 10.00th=[ 724],
- | 20.00th=[ 764], 30.00th=[ 780], 40.00th=[ 788],
- | 50.00th=[ 828], 60.00th=[ 860], 70.00th=[ 884],
- | 80.00th=[ 924], 90.00th=[ 8768], 95.00th=[ 12736],
- | 99.00th=[ 76021760], 99.50th=[ 624951296], 99.90th=[1082130432],
- | 99.95th=[1266679808], 99.99th=[1635778560]
- bw ( KiB/s): min= 0, max=59778, per=91.83%, avg=39953.00, stdev=28036.78, samples=2
- iops : min= 0, max= 939, avg=627.00, stdev=441.23, samples=2
- lat (nsec) : 750=17.24%, 1000=67.21%
- lat (usec) : 2=4.63%, 4=0.13%, 10=1.42%, 20=5.52%, 50=0.52%
- lat (usec) : 100=0.15%, 250=0.10%, 500=0.08%, 750=0.08%, 1000=0.17%
- lat (msec) : 2=0.38%, 4=0.34%, 10=0.30%, 20=0.26%, 50=0.36%
- lat (msec) : 100=0.18%, 250=0.20%, 500=0.15%, 750=0.30%, 1000=0.18%
- cpu : usr=0.15%, sys=0.59%, ctx=5019, majf=0, minf=40
- IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=75286,0,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- readwrite: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:35:00 2019
- read: IOPS=55, BW=3563KiB/s (3649kB/s)(418MiB/120002msec)
- slat (nsec): min=0, max=633777k, avg=116582.49, stdev=7753908.85
- clat (nsec): min=0, max=613214k, avg=6368642.28, stdev=41422627.12
- lat (nsec): min=0, max=633913k, avg=6486171.40, stdev=42134167.47
- clat percentiles (nsec):
- | 1.00th=[ 716], 5.00th=[ 724], 10.00th=[ 732],
- | 20.00th=[ 748], 30.00th=[ 828], 40.00th=[ 916],
- | 50.00th=[ 1064], 60.00th=[ 1224], 70.00th=[ 1592],
- | 80.00th=[ 73216], 90.00th=[ 1335296], 95.00th=[ 10289152],
- | 99.00th=[206569472], 99.50th=[350224384], 99.90th=[526385152],
- | 99.95th=[583008256], 99.99th=[616562688]
- bw ( KiB/s): min= 0, max= 5756, per=8.19%, avg=3562.50, stdev=3102.08, samples=2
- iops : min= 0, max= 89, avg=55.00, stdev=48.08, samples=2
- write: IOPS=56, BW=3638KiB/s (3726kB/s)(426MiB/120002msec)
- slat (nsec): min=0, max=16871k, avg=187455.60, stdev=310103.99
- clat (nsec): min=0, max=1166.3M, avg=64088245.30, stdev=127371794.56
- lat (nsec): min=0, max=1166.4M, avg=64276809.91, stdev=127323659.95
- clat percentiles (usec):
- | 1.00th=[ 1045], 5.00th=[ 1319], 10.00th=[ 1532],
- | 20.00th=[ 2008], 30.00th=[ 2704], 40.00th=[ 3752],
- | 50.00th=[ 6390], 60.00th=[ 16188], 70.00th=[ 59507],
- | 80.00th=[ 101188], 90.00th=[ 173016], 95.00th=[ 316670],
- | 99.00th=[ 666895], 99.50th=[ 792724], 99.90th=[ 918553],
- | 99.95th=[ 952108], 99.99th=[1166017]
- bw ( KiB/s): min= 0, max= 5860, per=78.63%, avg=3636.50, stdev=3144.50, samples=2
- iops : min= 0, max= 91, avg=56.50, stdev=48.79, samples=2
- lat (nsec) : 750=10.52%, 1000=12.57%
- lat (usec) : 2=11.90%, 4=0.07%, 10=0.64%, 20=1.83%, 50=1.19%
- lat (usec) : 100=1.39%, 250=1.77%, 500=1.04%, 750=0.42%, 1000=1.01%
- lat (msec) : 2=10.89%, 4=11.91%, 10=7.83%, 20=3.60%, 50=4.21%
- lat (msec) : 100=6.04%, 250=7.55%, 500=2.39%, 750=0.92%, 1000=0.30%
- cpu : usr=0.07%, sys=0.30%, ctx=13525, majf=0, minf=31
- IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=6681,6819,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- Run status group 0 (all jobs):
- READ: bw=42.5MiB/s (44.6MB/s), 3563KiB/s-39.0MiB/s (3649kB/s-40.9MB/s), io=5102MiB (5349MB), run=120002-120072msec
- WRITE: bw=4626KiB/s (4737kB/s), 1000KiB/s-3638KiB/s (1024kB/s-3726kB/s), io=544MiB (570MB), run=120002-120408msec
- Disk stats (read/write):
- rbd245: ios=3467/8915, merge=0/1, ticks=551293/914532, in_queue=1459579, util=5.67%
- <example-benchmark-fio-server-737-benchmark-pfzcw>
- randwrite: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:35:00 2019
- write: IOPS=34, BW=2211KiB/s (2264kB/s)(262MiB/121542msec)
- slat (nsec): min=0, max=279070k, avg=313105.26, stdev=4584630.48
- clat (nsec): min=0, max=9623.8M, avg=120787623.04, stdev=322490503.56
- lat (nsec): min=0, max=9623.8M, avg=121102071.47, stdev=322508076.74
- clat percentiles (usec):
- | 1.00th=[ 1074], 5.00th=[ 1696], 10.00th=[ 2343],
- | 20.00th=[ 3458], 30.00th=[ 4883], 40.00th=[ 6718],
- | 50.00th=[ 10028], 60.00th=[ 16712], 70.00th=[ 57410],
- | 80.00th=[ 229639], 90.00th=[ 400557], 95.00th=[ 566232],
- | 99.00th=[ 893387], 99.50th=[ 935330], 99.90th=[2231370],
- | 99.95th=[8556381], 99.99th=[9596568]
- bw ( KiB/s): min= 0, max= 2521, per=39.07%, avg=2236.50, stdev=402.34, samples=2
- iops : min= 0, max= 39, avg=34.50, stdev= 6.36, samples=2
- lat (usec) : 250=0.02%, 500=0.02%, 750=0.33%, 1000=0.43%
- lat (msec) : 2=6.48%, 4=16.85%, 10=25.81%, 20=12.82%, 50=6.34%
- lat (msec) : 100=3.57%, 250=9.32%, 500=11.42%, 750=4.43%, 1000=1.74%
- cpu : usr=0.03%, sys=0.15%, ctx=7087, majf=0, minf=31
- IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=0,4196,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- read: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:35:00 2019
- read: IOPS=98, BW=6322KiB/s (6473kB/s)(744MiB/120507msec)
- slat (nsec): min=0, max=13961k, avg=41369.64, stdev=206220.35
- clat (nsec): min=0, max=1884.5M, avg=40508976.77, stdev=167911107.34
- lat (nsec): min=0, max=1884.5M, avg=40551550.07, stdev=167924859.19
- clat percentiles (nsec):
- | 1.00th=[ 716], 5.00th=[ 724], 10.00th=[ 764],
- | 20.00th=[ 780], 30.00th=[ 828], 40.00th=[ 868],
- | 50.00th=[ 924], 60.00th=[ 1192], 70.00th=[ 1496],
- | 80.00th=[ 1564672], 90.00th=[ 10551296], 95.00th=[ 354418688],
- | 99.00th=[ 901775360], 99.50th=[1002438656], 99.90th=[1333788672],
- | 99.95th=[1535115264], 99.99th=[1837105152]
- bw ( KiB/s): min= 0, max= 6847, per=64.89%, avg=6344.50, stdev=710.64, samples=2
- iops : min= 0, max= 106, avg=98.50, stdev=10.61, samples=2
- lat (nsec) : 750=8.55%, 1000=44.78%
- lat (usec) : 2=18.65%, 4=0.16%, 10=0.05%, 20=0.62%, 50=0.21%
- lat (usec) : 100=0.18%, 250=0.10%, 500=0.14%, 750=0.44%, 1000=2.25%
- lat (msec) : 2=5.77%, 4=4.82%, 10=3.20%, 20=1.59%, 50=1.57%
- lat (msec) : 100=0.82%, 250=0.71%, 500=1.29%, 750=2.08%, 1000=1.56%
- cpu : usr=0.04%, sys=0.15%, ctx=6384, majf=0, minf=31
- IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=11900,0,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- readwrite: (groupid=0, jobs=1): err= 0: pid=14: Wed May 8 20:35:00 2019
- read: IOPS=54, BW=3471KiB/s (3555kB/s)(407MiB/120007msec)
- slat (nsec): min=0, max=2714.2k, avg=33168.77, stdev=89049.83
- clat (nsec): min=0, max=1276.5M, avg=9409156.58, stdev=48225664.37
- lat (nsec): min=0, max=1276.6M, avg=9443278.32, stdev=48232813.61
- clat percentiles (nsec):
- | 1.00th=[ 716], 5.00th=[ 724], 10.00th=[ 732],
- | 20.00th=[ 772], 30.00th=[ 900], 40.00th=[ 1032],
- | 50.00th=[ 1176], 60.00th=[ 1368], 70.00th=[ 88576],
- | 80.00th=[ 1122304], 90.00th=[ 5931008], 95.00th=[ 46923776],
- | 99.00th=[ 242221056], 99.50th=[ 383778816], 99.90th=[ 583008256],
- | 99.95th=[ 608174080], 99.99th=[1283457024]
- bw ( KiB/s): min= 0, max= 5833, per=35.49%, avg=3470.50, stdev=3341.08, samples=2
- iops : min= 0, max= 91, avg=54.00, stdev=52.33, samples=2
- write: IOPS=55, BW=3559KiB/s (3644kB/s)(417MiB/120007msec)
- slat (nsec): min=0, max=8103.8k, avg=181515.33, stdev=260812.54
- clat (nsec): min=0, max=1065.1M, avg=62753756.72, stdev=126482037.44
- lat (nsec): min=0, max=1065.2M, avg=62936479.59, stdev=126442634.22
- clat percentiles (usec):
- | 1.00th=[ 996], 5.00th=[ 1221], 10.00th=[ 1401],
- | 20.00th=[ 1811], 30.00th=[ 2409], 40.00th=[ 3458],
- | 50.00th=[ 5473], 60.00th=[ 13435], 70.00th=[ 43254],
- | 80.00th=[ 90702], 90.00th=[ 164627], 95.00th=[ 387974],
- | 99.00th=[ 599786], 99.50th=[ 675283], 99.90th=[ 843056],
- | 99.95th=[ 910164], 99.99th=[1061159]
- bw ( KiB/s): min= 0, max= 5939, per=62.15%, avg=3557.50, stdev=3367.95, samples=2
- iops : min= 0, max= 92, avg=55.00, stdev=52.33, samples=2
- lat (nsec) : 750=8.15%, 1000=10.53%
- lat (usec) : 2=13.47%, 4=0.02%, 10=0.25%, 20=0.86%, 50=0.61%
- lat (usec) : 100=0.84%, 250=0.90%, 500=0.42%, 750=0.88%, 1000=2.41%
- lat (msec) : 2=14.72%, 4=11.69%, 10=8.41%, 20=3.60%, 50=5.27%
- lat (msec) : 100=6.93%, 250=5.47%, 500=3.38%, 750=1.08%, 1000=0.10%
- cpu : usr=0.08%, sys=0.30%, ctx=15450, majf=0, minf=33
- IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=6509,6670,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- Run status group 0 (all jobs):
- READ: bw=9778KiB/s (10.0MB/s), 3471KiB/s-6322KiB/s (3555kB/s-6473kB/s), io=1151MiB (1207MB), run=120007-120507msec
- WRITE: bw=5725KiB/s (5862kB/s), 2211KiB/s-3559KiB/s (2264kB/s-3644kB/s), io=680MiB (713MB), run=120007-121542msec
- Disk stats (read/write):
- rbd53: ios=5395/11625, merge=0/3, ticks=602058/947991, in_queue=1541538, util=4.56%
- <example-benchmark-fio-server-653-benchmark-6cpgt>
- randwrite: (groupid=0, jobs=1): err= 0: pid=11: Wed May 8 20:35:00 2019
- write: IOPS=34, BW=2218KiB/s (2271kB/s)(260MiB/120080msec)
- slat (nsec): min=0, max=1186.4M, avg=1481401.31, stdev=33991422.05
- clat (nsec): min=0, max=2099.0M, avg=114488164.29, stdev=238384745.73
- lat (nsec): min=0, max=2099.4M, avg=115970057.69, stdev=240216513.94
- clat percentiles (usec):
- | 1.00th=[ 1037], 5.00th=[ 1336], 10.00th=[ 1614],
- | 20.00th=[ 2245], 30.00th=[ 3130], 40.00th=[ 4424],
- | 50.00th=[ 7504], 60.00th=[ 18744], 70.00th=[ 54789],
- | 80.00th=[ 125305], 90.00th=[ 463471], 95.00th=[ 675283],
- | 99.00th=[1035994], 99.50th=[1149240], 99.90th=[1786774],
- | 99.95th=[1870660], 99.99th=[2105541]
- bw ( KiB/s): min= 0, max= 2634, per=39.17%, avg=2216.50, stdev=590.43, samples=2
- iops : min= 0, max= 41, avg=34.50, stdev= 9.19, samples=2
- lat (usec) : 750=0.10%, 1000=0.53%
- lat (msec) : 2=15.54%, 4=20.92%, 10=16.50%, 20=7.36%, 50=8.49%
- lat (msec) : 100=7.24%, 250=9.16%, 500=5.39%, 750=4.88%, 1000=2.77%
- cpu : usr=0.04%, sys=0.15%, ctx=7482, majf=0, minf=31
- IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=0,4158,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- read: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:35:00 2019
- read: IOPS=519, BW=32.2MiB/s (33.8MB/s)(3869MiB/120140msec)
- slat (nsec): min=0, max=1032.2M, avg=66123.48, stdev=6143379.66
- clat (nsec): min=0, max=2075.3M, avg=7681293.84, stdev=65084937.34
- lat (nsec): min=0, max=2075.4M, avg=7748304.75, stdev=65378311.95
- clat percentiles (nsec):
- | 1.00th=[ 716], 5.00th=[ 724], 10.00th=[ 724],
- | 20.00th=[ 740], 30.00th=[ 780], 40.00th=[ 796],
- | 50.00th=[ 844], 60.00th=[ 876], 70.00th=[ 916],
- | 80.00th=[ 1240], 90.00th=[ 16768], 95.00th=[ 1417216],
- | 99.00th=[ 304087040], 99.50th=[ 522190848], 99.90th=[ 893386752],
- | 99.95th=[1027604480], 99.99th=[1317011456]
- bw ( KiB/s): min= 0, max=52272, per=90.92%, avg=33012.50, stdev=27237.05, samples=2
- iops : min= 0, max= 824, avg=519.00, stdev=431.34, samples=2
- lat (nsec) : 750=23.18%, 1000=52.62%
- lat (usec) : 2=6.72%, 4=0.13%, 10=1.52%, 20=7.48%, 50=1.36%
- lat (usec) : 100=0.16%, 250=0.08%, 500=0.06%, 750=0.22%, 1000=0.74%
- lat (msec) : 2=1.30%, 4=0.82%, 10=0.41%, 20=0.25%, 50=0.73%
- lat (msec) : 100=0.75%, 250=0.36%, 500=0.55%, 750=0.34%, 1000=0.16%
- cpu : usr=0.11%, sys=0.53%, ctx=8615, majf=0, minf=30
- IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=62370,0,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- readwrite: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:35:00 2019
- read: IOPS=52, BW=3337KiB/s (3417kB/s)(391MiB/120013msec)
- slat (nsec): min=0, max=599202k, avg=127308.18, stdev=7575305.82
- clat (nsec): min=0, max=668739k, avg=7431754.00, stdev=41464138.35
- lat (nsec): min=0, max=668798k, avg=7559998.10, stdev=42139523.63
- clat percentiles (nsec):
- | 1.00th=[ 716], 5.00th=[ 724], 10.00th=[ 732],
- | 20.00th=[ 780], 30.00th=[ 884], 40.00th=[ 1004],
- | 50.00th=[ 1160], 60.00th=[ 1320], 70.00th=[ 22912],
- | 80.00th=[ 888832], 90.00th=[ 3194880], 95.00th=[ 35913728],
- | 99.00th=[204472320], 99.50th=[329252864], 99.90th=[530579456],
- | 99.95th=[583008256], 99.99th=[666894336]
- bw ( KiB/s): min= 0, max= 5328, per=9.19%, avg=3337.00, stdev=2815.70, samples=2
- iops : min= 0, max= 83, avg=52.00, stdev=43.84, samples=2
- write: IOPS=53, BW=3443KiB/s (3526kB/s)(404MiB/120013msec)
- slat (nsec): min=0, max=6146.0k, avg=181618.18, stdev=222082.45
- clat (nsec): min=0, max=1423.9M, avg=67042871.01, stdev=133485106.00
- lat (nsec): min=0, max=1423.9M, avg=67225673.93, stdev=133436561.24
- clat percentiles (usec):
- | 1.00th=[ 1029], 5.00th=[ 1237], 10.00th=[ 1385],
- | 20.00th=[ 1745], 30.00th=[ 2245], 40.00th=[ 3097],
- | 50.00th=[ 4948], 60.00th=[ 12518], 70.00th=[ 41681],
- | 80.00th=[ 98042], 90.00th=[ 214959], 95.00th=[ 379585],
- | 99.00th=[ 641729], 99.50th=[ 717226], 99.90th=[ 809501],
- | 99.95th=[ 817890], 99.99th=[1417675]
- bw ( KiB/s): min= 0, max= 5473, per=60.81%, avg=3441.50, stdev=2872.97, samples=2
- iops : min= 0, max= 85, avg=53.50, stdev=44.55, samples=2
- lat (nsec) : 750=7.70%, 1000=11.79%
- lat (usec) : 2=13.52%, 4=0.05%, 10=0.30%, 20=0.98%, 50=0.92%
- lat (usec) : 100=0.77%, 250=1.10%, 500=0.69%, 750=0.68%, 1000=2.14%
- lat (msec) : 2=15.40%, 4=12.12%, 10=6.92%, 20=3.92%, 50=4.73%
- lat (msec) : 100=5.37%, 250=6.36%, 500=3.18%, 750=1.25%, 1000=0.13%
- cpu : usr=0.07%, sys=0.30%, ctx=14601, majf=0, minf=34
- IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=6257,6454,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- Run status group 0 (all jobs):
- READ: bw=35.5MiB/s (37.2MB/s), 3337KiB/s-32.2MiB/s (3417kB/s-33.8MB/s), io=4260MiB (4467MB), run=120013-120140msec
- WRITE: bw=5659KiB/s (5795kB/s), 2218KiB/s-3443KiB/s (2271kB/s-3526kB/s), io=664MiB (696MB), run=120013-120080msec
- Disk stats (read/write):
- rbd213: ios=5674/10923, merge=0/3, ticks=532365/860881, in_queue=1384865, util=8.80%
- <example-benchmark-fio-server-666-benchmark-vhp22>
- <example-benchmark-fio-server-674-benchmark-5pb2m>
- randwrite: (groupid=0, jobs=1): err= 0: pid=11: Wed May 8 20:35:00 2019
- write: IOPS=41, BW=2641KiB/s (2704kB/s)(310MiB/120145msec)
- slat (nsec): min=0, max=575779k, avg=600805.43, stdev=14798680.70
- clat (nsec): min=0, max=1310.3M, avg=96705119.71, stdev=167414675.40
- lat (nsec): min=0, max=1338.3M, avg=97306982.18, stdev=168344306.33
- clat percentiles (usec):
- | 1.00th=[ 971], 5.00th=[ 1188], 10.00th=[ 1401],
- | 20.00th=[ 1926], 30.00th=[ 2835], 40.00th=[ 5342],
- | 50.00th=[ 19268], 60.00th=[ 78119], 70.00th=[ 102237],
- | 80.00th=[ 127402], 90.00th=[ 267387], 95.00th=[ 463471],
- | 99.00th=[ 834667], 99.50th=[ 943719], 99.90th=[1149240],
- | 99.95th=[1300235], 99.99th=[1317012]
- bw ( KiB/s): min= 0, max= 2877, per=42.41%, avg=2641.50, stdev=333.05, samples=2
- iops : min= 0, max= 44, avg=40.50, stdev= 4.95, samples=2
- lat (usec) : 250=0.02%, 1000=1.31%
- lat (msec) : 2=19.74%, 4=14.77%, 10=9.97%, 20=4.56%, 50=4.38%
- lat (msec) : 100=14.57%, 250=20.14%, 500=6.16%, 750=2.83%, 1000=1.25%
- cpu : usr=0.06%, sys=0.17%, ctx=9207, majf=0, minf=30
- IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=0,4955,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- read: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:35:00 2019
- read: IOPS=486, BW=30.3MiB/s (31.8MB/s)(3637MiB/120064msec)
- slat (nsec): min=0, max=878989k, avg=51948.57, stdev=4719575.63
- clat (nsec): min=0, max=1394.2M, avg=8192226.33, stdev=53775826.59
- lat (nsec): min=0, max=1394.2M, avg=8245018.59, stdev=53987800.95
- clat percentiles (nsec):
- | 1.00th=[ 708], 5.00th=[ 716], 10.00th=[ 716],
- | 20.00th=[ 724], 30.00th=[ 748], 40.00th=[ 788],
- | 50.00th=[ 828], 60.00th=[ 868], 70.00th=[ 924],
- | 80.00th=[ 1192], 90.00th=[ 757760], 95.00th=[ 23199744],
- | 99.00th=[267386880], 99.50th=[442499072], 99.90th=[717225984],
- | 99.95th=[784334848], 99.99th=[952107008]
- bw ( KiB/s): min= 0, max=50771, per=89.87%, avg=31032.50, stdev=27914.45, samples=2
- iops : min= 0, max= 796, avg=486.00, stdev=438.41, samples=2
- lat (nsec) : 750=30.36%, 1000=45.16%
- lat (usec) : 2=8.76%, 4=0.08%, 10=1.60%, 20=3.14%, 50=0.19%
- lat (usec) : 100=0.13%, 250=0.06%, 500=0.09%, 750=0.39%, 1000=1.05%
- lat (msec) : 2=1.69%, 4=1.16%, 10=0.62%, 20=0.42%, 50=1.52%
- lat (msec) : 100=1.69%, 250=0.83%, 500=0.72%, 750=0.28%, 1000=0.06%
- cpu : usr=0.13%, sys=0.51%, ctx=12296, majf=0, minf=105
- IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=58366,0,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- readwrite: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:35:00 2019
- read: IOPS=54, BW=3515KiB/s (3600kB/s)(412MiB/120011msec)
- slat (nsec): min=0, max=3688.4k, avg=35129.82, stdev=104218.83
- clat (nsec): min=0, max=681349k, avg=8268770.25, stdev=41687470.94
- lat (nsec): min=0, max=681373k, avg=8304838.00, stdev=41694110.97
- clat percentiles (nsec):
- | 1.00th=[ 724], 5.00th=[ 732], 10.00th=[ 748],
- | 20.00th=[ 828], 30.00th=[ 924], 40.00th=[ 1064],
- | 50.00th=[ 1208], 60.00th=[ 1384], 70.00th=[ 64256],
- | 80.00th=[ 1089536], 90.00th=[ 5734400], 95.00th=[ 43778048],
- | 99.00th=[214958080], 99.50th=[333447168], 99.90th=[509607936],
- | 99.95th=[608174080], 99.99th=[683671552]
- bw ( KiB/s): min= 0, max= 5158, per=10.18%, avg=3515.50, stdev=2322.85, samples=2
- iops : min= 0, max= 80, avg=54.50, stdev=36.06, samples=2
- write: IOPS=56, BW=3593KiB/s (3679kB/s)(421MiB/120011msec)
- slat (nsec): min=0, max=566283k, avg=315230.86, stdev=8140260.33
- clat (nsec): min=0, max=1079.2M, avg=62967107.48, stdev=118454148.51
- lat (nsec): min=0, max=1079.3M, avg=63283412.71, stdev=118888184.71
- clat percentiles (usec):
- | 1.00th=[ 1004], 5.00th=[ 1254], 10.00th=[ 1467],
- | 20.00th=[ 1909], 30.00th=[ 2704], 40.00th=[ 4293],
- | 50.00th=[ 8455], 60.00th=[ 22414], 70.00th=[ 65274],
- | 80.00th=[ 102237], 90.00th=[ 158335], 95.00th=[ 304088],
- | 99.00th=[ 591397], 99.50th=[ 750781], 99.90th=[ 935330],
- | 99.95th=[ 952108], 99.99th=[1082131]
- bw ( KiB/s): min= 0, max= 5207, per=57.64%, avg=3590.50, stdev=2286.08, samples=2
- iops : min= 0, max= 81, avg=55.50, stdev=36.06, samples=2
- lat (nsec) : 750=4.97%, 1000=12.73%
- lat (usec) : 2=14.70%, 4=0.03%, 10=0.33%, 20=0.82%, 50=0.73%
- lat (usec) : 100=0.86%, 250=1.03%, 500=0.58%, 750=0.76%, 1000=2.14%
- lat (msec) : 2=13.36%, 4=10.56%, 10=7.76%, 20=4.22%, 50=5.60%
- lat (msec) : 100=7.45%, 250=7.89%, 500=2.56%, 750=0.71%, 1000=0.24%
- cpu : usr=0.07%, sys=0.33%, ctx=15735, majf=0, minf=33
- IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=6592,6734,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- Run status group 0 (all jobs):
- READ: bw=33.7MiB/s (35.4MB/s), 3515KiB/s-30.3MiB/s (3600kB/s-31.8MB/s), io=4049MiB (4246MB), run=120011-120064msec
- WRITE: bw=6230KiB/s (6379kB/s), 2641KiB/s-3593KiB/s (2704kB/s-3679kB/s), io=731MiB (766MB), run=120011-120145msec
- Disk stats (read/write):
- rbd173: ios=8283/12086, merge=0/1, ticks=553856/811994, in_queue=1355561, util=13.18%
- randwrite: (groupid=0, jobs=1): err= 0: pid=11: Wed May 8 20:35:00 2019
- write: IOPS=29, BW=1877KiB/s (1922kB/s)(222MiB/120954msec)
- slat (nsec): min=0, max=1178.4M, avg=1070070.99, stdev=26922015.16
- clat (nsec): min=0, max=1811.8M, avg=135023864.34, stdev=288341801.46
- lat (nsec): min=0, max=1811.9M, avg=136095701.53, stdev=289170559.85
- clat percentiles (usec):
- | 1.00th=[ 1029], 5.00th=[ 1287], 10.00th=[ 1532],
- | 20.00th=[ 2024], 30.00th=[ 2704], 40.00th=[ 3851],
- | 50.00th=[ 5997], 60.00th=[ 13435], 70.00th=[ 38536],
- | 80.00th=[ 127402], 90.00th=[ 591397], 95.00th=[ 876610],
- | 99.00th=[1249903], 99.50th=[1317012], 99.90th=[1635779],
- | 99.95th=[1669333], 99.99th=[1803551]
- bw ( KiB/s): min= 0, max= 2954, per=33.73%, avg=1889.50, stdev=1505.43, samples=2
- iops : min= 0, max= 46, avg=29.00, stdev=24.04, samples=2
- lat (usec) : 750=0.06%, 1000=0.73%
- lat (msec) : 2=18.96%, 4=21.24%, 10=16.19%, 20=7.14%, 50=7.02%
- lat (msec) : 100=4.51%, 250=9.37%, 500=3.02%, 750=4.91%, 1000=3.92%
- cpu : usr=0.04%, sys=0.13%, ctx=6342, majf=0, minf=30
- IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=0,3545,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- read: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:35:00 2019
- read: IOPS=517, BW=32.2MiB/s (33.8MB/s)(3876MiB/120265msec)
- slat (nsec): min=0, max=1009.8M, avg=58501.48, stdev=5796659.72
- clat (nsec): min=0, max=2075.2M, avg=7667833.39, stdev=70093238.96
- lat (nsec): min=0, max=2075.3M, avg=7727229.62, stdev=70340481.38
- clat percentiles (nsec):
- | 1.00th=[ 716], 5.00th=[ 716], 10.00th=[ 724],
- | 20.00th=[ 740], 30.00th=[ 772], 40.00th=[ 788],
- | 50.00th=[ 804], 60.00th=[ 852], 70.00th=[ 884],
- | 80.00th=[ 932], 90.00th=[ 8768], 95.00th=[ 790528],
- | 99.00th=[ 375390208], 99.50th=[ 624951296], 99.90th=[ 943718400],
- | 99.95th=[1044381696], 99.99th=[1300234240]
- bw ( KiB/s): min= 0, max=51751, per=89.94%, avg=32982.50, stdev=26542.67, samples=2
- iops : min= 0, max= 812, avg=517.00, stdev=417.19, samples=2
- lat (nsec) : 750=24.97%, 1000=58.54%
- lat (usec) : 2=6.06%, 4=0.17%, 10=0.98%, 20=3.16%, 50=0.53%
- lat (usec) : 100=0.11%, 250=0.08%, 500=0.08%, 750=0.25%, 1000=0.54%
- lat (msec) : 2=0.92%, 4=0.67%, 10=0.47%, 20=0.24%, 50=0.56%
- lat (msec) : 100=0.48%, 250=0.13%, 500=0.38%, 750=0.41%, 1000=0.23%
- cpu : usr=0.13%, sys=0.49%, ctx=7065, majf=0, minf=36
- IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=62257,0,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- readwrite: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:35:00 2019
- read: IOPS=57, BW=3678KiB/s (3767kB/s)(431MiB/120039msec)
- slat (nsec): min=0, max=753868k, avg=277793.85, stdev=12125320.64
- clat (nsec): min=0, max=806006k, avg=7819411.97, stdev=47157480.87
- lat (nsec): min=0, max=806066k, avg=8098127.39, stdev=48659920.24
- clat percentiles (nsec):
- | 1.00th=[ 716], 5.00th=[ 724], 10.00th=[ 732],
- | 20.00th=[ 764], 30.00th=[ 876], 40.00th=[ 996],
- | 50.00th=[ 1144], 60.00th=[ 1320], 70.00th=[ 18560],
- | 80.00th=[ 749568], 90.00th=[ 2637824], 95.00th=[ 26869760],
- | 99.00th=[265289728], 99.50th=[387973120], 99.90th=[624951296],
- | 99.95th=[675282944], 99.99th=[809500672]
- bw ( KiB/s): min= 0, max= 5620, per=10.03%, avg=3679.00, stdev=2744.99, samples=2
- iops : min= 0, max= 87, avg=57.00, stdev=42.43, samples=2
- write: IOPS=58, BW=3753KiB/s (3844kB/s)(440MiB/120039msec)
- slat (nsec): min=0, max=4929.1k, avg=192875.32, stdev=240267.68
- clat (nsec): min=0, max=1279.8M, avg=60267779.32, stdev=129578981.16
- lat (nsec): min=0, max=1279.9M, avg=60461861.12, stdev=129540203.48
- clat percentiles (usec):
- | 1.00th=[ 1037], 5.00th=[ 1287], 10.00th=[ 1483],
- | 20.00th=[ 1893], 30.00th=[ 2442], 40.00th=[ 3392],
- | 50.00th=[ 5407], 60.00th=[ 11207], 70.00th=[ 30016],
- | 80.00th=[ 85459], 90.00th=[ 170918], 95.00th=[ 346031],
- | 99.00th=[ 666895], 99.50th=[ 784335], 99.90th=[ 935330],
- | 99.95th=[ 968885], 99.99th=[1283458]
- bw ( KiB/s): min= 0, max= 5680, per=66.99%, avg=3752.50, stdev=2725.90, samples=2
- iops : min= 0, max= 88, avg=58.00, stdev=42.43, samples=2
- lat (nsec) : 750=8.39%, 1000=11.65%
- lat (usec) : 2=13.20%, 4=0.04%, 10=0.39%, 20=1.22%, 50=0.88%
- lat (usec) : 100=0.91%, 250=1.23%, 500=0.78%, 750=0.98%, 1000=1.87%
- lat (msec) : 2=13.50%, 4=12.72%, 10=8.19%, 20=4.71%, 50=4.54%
- lat (msec) : 100=5.34%, 250=5.01%, 500=3.30%, 750=0.83%, 1000=0.32%
- cpu : usr=0.08%, sys=0.33%, ctx=15516, majf=0, minf=31
- IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=6899,7037,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- Run status group 0 (all jobs):
- READ: bw=35.8MiB/s (37.6MB/s), 3678KiB/s-32.2MiB/s (3767kB/s-33.8MB/s), io=4307MiB (4516MB), run=120039-120265msec
- WRITE: bw=5602KiB/s (5737kB/s), 1877KiB/s-3753KiB/s (1922kB/s-3844kB/s), io=662MiB (694MB), run=120039-120954msec
- Disk stats (read/write):
- rbd205: ios=5496/10983, merge=0/1, ticks=553117/872605, in_queue=1417429, util=8.18%
- randwrite: (groupid=0, jobs=1): err= 0: pid=11: Wed May 8 20:35:00 2019
- write: IOPS=22, BW=1415KiB/s (1449kB/s)(167MiB/120791msec)
- slat (nsec): min=0, max=560200k, avg=579809.47, stdev=13166429.06
- clat (nsec): min=0, max=3879.5M, avg=180201402.17, stdev=290186331.72
- lat (nsec): min=0, max=3879.7M, avg=180784362.43, stdev=290262817.96
- clat percentiles (usec):
- | 1.00th=[ 1565], 5.00th=[ 2573], 10.00th=[ 3425],
- | 20.00th=[ 5407], 30.00th=[ 8586], 40.00th=[ 13829],
- | 50.00th=[ 22152], 60.00th=[ 80217], 70.00th=[ 214959],
- | 80.00th=[ 371196], 90.00th=[ 583009], 95.00th=[ 708838],
- | 99.00th=[ 985662], 99.50th=[1266680], 99.90th=[3875537],
- | 99.95th=[3875537], 99.99th=[3875537]
- bw ( KiB/s): min= 0, max= 2022, per=28.42%, avg=1421.00, stdev=849.94, samples=2
- iops : min= 0, max= 31, avg=21.50, stdev=13.44, samples=2
- lat (usec) : 4=0.07%, 250=0.04%, 500=0.04%, 1000=0.04%
- lat (msec) : 2=2.77%, 4=10.09%, 10=20.17%, 20=14.62%, 50=9.75%
- lat (msec) : 100=4.50%, 250=10.16%, 500=14.59%, 750=8.92%, 1000=3.45%
- cpu : usr=0.03%, sys=0.10%, ctx=4530, majf=0, minf=31
- IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=0,2667,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- read: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:35:00 2019
- read: IOPS=124, BW=7969KiB/s (8160kB/s)(938MiB/120482msec)
- slat (nsec): min=0, max=11516k, avg=36049.89, stdev=185515.83
- clat (nsec): min=0, max=1889.0M, avg=32130791.70, stdev=149359827.92
- lat (nsec): min=0, max=1890.1M, avg=32167996.44, stdev=149380240.45
- clat percentiles (nsec):
- | 1.00th=[ 716], 5.00th=[ 724], 10.00th=[ 740],
- | 20.00th=[ 780], 30.00th=[ 804], 40.00th=[ 852],
- | 50.00th=[ 892], 60.00th=[ 956], 70.00th=[ 1240],
- | 80.00th=[ 30592], 90.00th=[ 5603328], 95.00th=[ 81264640],
- | 99.00th=[ 834666496], 99.50th=[ 960495616], 99.90th=[1216348160],
- | 99.95th=[1367343104], 99.99th=[1853882368]
- bw ( KiB/s): min= 0, max= 8257, per=69.65%, avg=7998.50, stdev=365.57, samples=2
- iops : min= 0, max= 129, avg=124.50, stdev= 6.36, samples=2
- lat (nsec) : 750=11.93%, 1000=50.52%
- lat (usec) : 2=16.29%, 4=0.27%, 10=0.13%, 20=0.79%, 50=0.21%
- lat (usec) : 100=0.11%, 250=0.09%, 500=0.09%, 750=0.31%, 1000=1.35%
- lat (msec) : 2=3.72%, 4=3.19%, 10=2.74%, 20=1.33%, 50=1.45%
- lat (msec) : 100=0.61%, 250=0.61%, 500=0.96%, 750=1.81%, 1000=1.11%
- cpu : usr=0.04%, sys=0.17%, ctx=5951, majf=0, minf=31
- IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=14999,0,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- readwrite: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:35:00 2019
- read: IOPS=55, BW=3529KiB/s (3614kB/s)(414MiB/120011msec)
- slat (nsec): min=0, max=3964.5k, avg=26293.80, stdev=81795.50
- clat (nsec): min=0, max=739318k, avg=10488878.55, stdev=55028154.30
- lat (nsec): min=0, max=739351k, avg=10516103.18, stdev=55036378.95
- clat percentiles (nsec):
- | 1.00th=[ 716], 5.00th=[ 724], 10.00th=[ 732],
- | 20.00th=[ 772], 30.00th=[ 876], 40.00th=[ 988],
- | 50.00th=[ 1144], 60.00th=[ 1288], 70.00th=[ 1768],
- | 80.00th=[ 692224], 90.00th=[ 4816896], 95.00th=[ 44302336],
- | 99.00th=[329252864], 99.50th=[450887680], 99.90th=[624951296],
- | 99.95th=[675282944], 99.99th=[742391808]
- bw ( KiB/s): min= 0, max= 5220, per=30.73%, avg=3529.00, stdev=2391.44, samples=2
- iops : min= 0, max= 81, avg=54.50, stdev=37.48, samples=2
- write: IOPS=56, BW=3609KiB/s (3695kB/s)(423MiB/120011msec)
- slat (nsec): min=0, max=9699.0k, avg=182579.58, stdev=251670.03
- clat (nsec): min=0, max=948820k, avg=60636613.95, stdev=123178432.93
- lat (nsec): min=0, max=948884k, avg=60820348.26, stdev=123136199.28
- clat percentiles (usec):
- | 1.00th=[ 1012], 5.00th=[ 1221], 10.00th=[ 1401], 20.00th=[ 1778],
- | 30.00th=[ 2343], 40.00th=[ 3523], 50.00th=[ 6652], 60.00th=[ 15401],
- | 70.00th=[ 37487], 80.00th=[ 90702], 90.00th=[166724], 95.00th=[333448],
- | 99.00th=[641729], 99.50th=[717226], 99.90th=[893387], 99.95th=[918553],
- | 99.99th=[952108]
- bw ( KiB/s): min= 0, max= 5319, per=72.13%, avg=3606.50, stdev=2421.84, samples=2
- iops : min= 0, max= 83, avg=56.00, stdev=38.18, samples=2
- lat (nsec) : 750=8.29%, 1000=11.81%
- lat (usec) : 2=14.65%, 4=0.04%, 10=0.39%, 20=1.12%, 50=0.75%
- lat (usec) : 100=0.76%, 250=1.17%, 500=0.37%, 750=0.46%, 1000=1.48%
- lat (msec) : 2=13.97%, 4=10.30%, 10=7.65%, 20=4.71%, 50=5.79%
- lat (msec) : 100=5.75%, 250=6.33%, 500=3.01%, 750=1.09%, 1000=0.13%
- cpu : usr=0.07%, sys=0.31%, ctx=14900, majf=0, minf=31
- IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=6618,6764,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- Run status group 0 (all jobs):
- READ: bw=11.2MiB/s (11.8MB/s), 3529KiB/s-7969KiB/s (3614kB/s-8160kB/s), io=1351MiB (1417MB), run=120011-120482msec
- WRITE: bw=5000KiB/s (5120kB/s), 1415KiB/s-3609KiB/s (1449kB/s-3695kB/s), io=590MiB (618MB), run=120011-120791msec
- Disk stats (read/write):
- rbd184: ios=4567/10108, merge=0/2, ticks=592804/909666, in_queue=1495157, util=4.30%
- <example-benchmark-fio-server-400-benchmark-fj44s>
- <example-benchmark-fio-server-644-benchmark-z4l9v>
- randwrite: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:35:00 2019
- write: IOPS=19, BW=1259KiB/s (1289kB/s)(148MiB/120652msec)
- slat (nsec): min=0, max=661885k, avg=818240.79, stdev=16567186.30
- clat (nsec): min=0, max=1579.3M, avg=202990774.12, stdev=318293903.98
- lat (nsec): min=0, max=1579.9M, avg=203809884.73, stdev=318373055.97
- clat percentiles (usec):
- | 1.00th=[ 1254], 5.00th=[ 1827], 10.00th=[ 2343],
- | 20.00th=[ 3425], 30.00th=[ 5211], 40.00th=[ 7963],
- | 50.00th=[ 12911], 60.00th=[ 25297], 70.00th=[ 287310],
- | 80.00th=[ 434111], 90.00th=[ 692061], 95.00th=[ 893387],
- | 99.00th=[1283458], 99.50th=[1434452], 99.90th=[1484784],
- | 99.95th=[1484784], 99.99th=[1585447]
- bw ( KiB/s): min= 0, max= 1388, per=22.62%, avg=1259.00, stdev=182.43, samples=2
- iops : min= 0, max= 21, avg=19.00, stdev= 2.83, samples=2
- lat (usec) : 750=0.08%, 1000=0.17%
- lat (msec) : 2=6.12%, 4=17.30%, 10=20.55%, 20=12.62%, 50=8.31%
- lat (msec) : 100=1.60%, 250=1.18%, 500=14.56%, 750=9.70%, 1000=4.68%
- cpu : usr=0.03%, sys=0.09%, ctx=4052, majf=0, minf=31
- IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=0,2370,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- read: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:35:00 2019
- read: IOPS=204, BW=12.8MiB/s (13.4MB/s)(1540MiB/120482msec)
- slat (nsec): min=0, max=123504k, avg=30115.36, stdev=798302.18
- clat (nsec): min=0, max=1866.4M, avg=19516391.16, stdev=117399582.99
- lat (nsec): min=0, max=1866.4M, avg=19547550.88, stdev=117418991.93
- clat percentiles (nsec):
- | 1.00th=[ 708], 5.00th=[ 724], 10.00th=[ 732],
- | 20.00th=[ 772], 30.00th=[ 788], 40.00th=[ 812],
- | 50.00th=[ 852], 60.00th=[ 876], 70.00th=[ 924],
- | 80.00th=[ 1240], 90.00th=[ 1368064], 95.00th=[ 9895936],
- | 99.00th=[ 734003200], 99.50th=[ 884998144], 99.90th=[1166016512],
- | 99.95th=[1283457024], 99.99th=[1837105152]
- bw ( KiB/s): min= 0, max=16005, per=75.84%, avg=13136.50, stdev=4056.67, samples=2
- iops : min= 0, max= 250, avg=205.00, stdev=63.64, samples=2
- lat (nsec) : 750=12.70%, 1000=61.83%
- lat (usec) : 2=11.12%, 4=0.20%, 10=0.34%, 20=1.36%, 50=0.21%
- lat (usec) : 100=0.10%, 250=0.12%, 500=0.05%, 750=0.19%, 1000=0.78%
- lat (msec) : 2=2.21%, 4=2.08%, 10=1.74%, 20=0.78%, 50=0.82%
- lat (msec) : 100=0.38%, 250=0.33%, 500=0.72%, 750=1.06%, 1000=0.62%
- cpu : usr=0.06%, sys=0.24%, ctx=6050, majf=0, minf=31
- IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=24637,0,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- readwrite: (groupid=0, jobs=1): err= 0: pid=14: Wed May 8 20:35:00 2019
- read: IOPS=66, BW=4250KiB/s (4352kB/s)(498MiB/120010msec)
- slat (nsec): min=0, max=2540.8k, avg=24466.37, stdev=63425.35
- clat (nsec): min=0, max=1096.5M, avg=8256931.65, stdev=50119470.91
- lat (nsec): min=0, max=1096.6M, avg=8282370.65, stdev=50126584.41
- clat percentiles (nsec):
- | 1.00th=[ 716], 5.00th=[ 724], 10.00th=[ 732],
- | 20.00th=[ 756], 30.00th=[ 852], 40.00th=[ 940],
- | 50.00th=[ 1080], 60.00th=[ 1224], 70.00th=[ 1560],
- | 80.00th=[ 168960], 90.00th=[ 2473984], 95.00th=[ 25821184],
- | 99.00th=[ 274726912], 99.50th=[ 425721856], 99.90th=[ 616562688],
- | 99.95th=[ 742391808], 99.99th=[1098907648]
- bw ( KiB/s): min= 0, max= 6593, per=24.54%, avg=4250.50, stdev=3312.80, samples=2
- iops : min= 0, max= 103, avg=66.00, stdev=52.33, samples=2
- write: IOPS=67, BW=4330KiB/s (4434kB/s)(507MiB/120010msec)
- slat (nsec): min=0, max=7458.3k, avg=189853.89, stdev=259526.74
- clat (nsec): min=0, max=894153k, avg=50960770.68, stdev=106485643.26
- lat (nsec): min=0, max=894188k, avg=51151899.21, stdev=106440642.32
- clat percentiles (usec):
- | 1.00th=[ 979], 5.00th=[ 1221], 10.00th=[ 1385], 20.00th=[ 1745],
- | 30.00th=[ 2245], 40.00th=[ 3064], 50.00th=[ 4752], 60.00th=[ 10290],
- | 70.00th=[ 25822], 80.00th=[ 81265], 90.00th=[141558], 95.00th=[267387],
- | 99.00th=[557843], 99.50th=[650118], 99.90th=[742392], 99.95th=[767558],
- | 99.99th=[893387]
- bw ( KiB/s): min= 0, max= 6640, per=77.77%, avg=4328.00, stdev=3269.66, samples=2
- iops : min= 0, max= 103, avg=67.00, stdev=50.91, samples=2
- lat (nsec) : 750=8.95%, 1000=12.87%
- lat (usec) : 2=13.44%, 4=0.02%, 10=0.40%, 20=1.39%, 50=0.77%
- lat (usec) : 100=1.04%, 250=1.31%, 500=0.63%, 750=0.53%, 1000=1.59%
- lat (msec) : 2=14.34%, 4=11.53%, 10=7.63%, 20=4.38%, 50=4.83%
- lat (msec) : 100=5.20%, 250=5.91%, 500=2.44%, 750=0.76%, 1000=0.06%
- cpu : usr=0.07%, sys=0.38%, ctx=17220, majf=0, minf=31
- IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=7970,8116,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- Run status group 0 (all jobs):
- READ: bw=16.9MiB/s (17.7MB/s), 4250KiB/s-12.8MiB/s (4352kB/s-13.4MB/s), io=2038MiB (2137MB), run=120010-120482msec
- WRITE: bw=5565KiB/s (5699kB/s), 1259KiB/s-4330KiB/s (1289kB/s-4434kB/s), io=656MiB (688MB), run=120010-120652msec
- Disk stats (read/write):
- rbd162: ios=4669/11042, merge=0/1, ticks=588851/900875, in_queue=1481867, util=5.90%
- <example-benchmark-fio-server-216-benchmark-88f76>
- randwrite: (groupid=0, jobs=1): err= 0: pid=11: Wed May 8 20:35:00 2019
- write: IOPS=44, BW=2862KiB/s (2931kB/s)(336MiB/120068msec)
- slat (nsec): min=0, max=556126k, avg=395599.01, stdev=9573679.68
- clat (nsec): min=0, max=1375.0M, avg=89644510.34, stdev=147243605.16
- lat (nsec): min=0, max=1375.2M, avg=90041328.80, stdev=147426996.10
- clat percentiles (usec):
- | 1.00th=[ 979], 5.00th=[ 1172], 10.00th=[ 1352],
- | 20.00th=[ 1762], 30.00th=[ 2507], 40.00th=[ 5014],
- | 50.00th=[ 21103], 60.00th=[ 68682], 70.00th=[ 96994],
- | 80.00th=[ 127402], 90.00th=[ 258999], 95.00th=[ 421528],
- | 99.00th=[ 708838], 99.50th=[ 750781], 99.90th=[ 918553],
- | 99.95th=[1132463], 99.99th=[1367344]
- bw ( KiB/s): min= 0, max= 3492, per=45.92%, avg=2861.00, stdev=892.37, samples=2
- iops : min= 0, max= 54, avg=44.00, stdev=14.14, samples=2
- lat (usec) : 4=0.02%, 500=0.04%, 750=0.07%, 1000=1.12%
- lat (msec) : 2=22.93%, 4=13.12%, 10=7.85%, 20=4.34%, 50=6.04%
- lat (msec) : 100=15.64%, 250=18.55%, 500=6.93%, 750=2.80%, 1000=0.54%
- cpu : usr=0.07%, sys=0.18%, ctx=10180, majf=0, minf=30
- IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=0,5365,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- read: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:35:00 2019
- read: IOPS=305, BW=19.1MiB/s (20.0MB/s)(2292MiB/120015msec)
- slat (nsec): min=0, max=510621k, avg=52091.46, stdev=3300689.27
- clat (nsec): min=0, max=1192.9M, avg=13093072.10, stdev=61828654.90
- lat (nsec): min=0, max=1192.9M, avg=13145990.94, stdev=61925908.23
- clat percentiles (nsec):
- | 1.00th=[ 708], 5.00th=[ 716], 10.00th=[ 716],
- | 20.00th=[ 724], 30.00th=[ 740], 40.00th=[ 780],
- | 50.00th=[ 844], 60.00th=[ 908], 70.00th=[ 1128],
- | 80.00th=[ 60672], 90.00th=[ 15400960], 95.00th=[ 60555264],
- | 99.00th=[362807296], 99.50th=[484442112], 99.90th=[683671552],
- | 99.95th=[734003200], 99.99th=[826277888]
- bw ( KiB/s): min= 0, max=24684, per=85.72%, avg=19554.50, stdev=7254.21, samples=2
- iops : min= 0, max= 386, avg=305.50, stdev=113.84, samples=2
- lat (nsec) : 750=35.59%, 1000=30.25%
- lat (usec) : 2=12.30%, 4=0.03%, 10=0.67%, 20=0.68%, 50=0.45%
- lat (usec) : 100=0.10%, 250=0.06%, 500=0.31%, 750=1.12%, 1000=2.37%
- lat (msec) : 2=3.13%, 4=1.82%, 10=0.85%, 20=0.52%, 50=3.62%
- lat (msec) : 100=3.02%, 250=1.50%, 500=1.20%, 750=0.40%, 1000=0.04%
- cpu : usr=0.10%, sys=0.36%, ctx=14507, majf=0, minf=31
- IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=36693,0,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- readwrite: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:35:00 2019
- read: IOPS=50, BW=3258KiB/s (3337kB/s)(382MiB/120007msec)
- slat (nsec): min=0, max=64156k, avg=44426.52, stdev=826886.07
- clat (nsec): min=0, max=612522k, avg=7346455.67, stdev=38544224.56
- lat (nsec): min=0, max=612553k, avg=7391819.68, stdev=38557927.00
- clat percentiles (nsec):
- | 1.00th=[ 716], 5.00th=[ 732], 10.00th=[ 740],
- | 20.00th=[ 812], 30.00th=[ 900], 40.00th=[ 1020],
- | 50.00th=[ 1176], 60.00th=[ 1336], 70.00th=[ 33024],
- | 80.00th=[ 946176], 90.00th=[ 4489216], 95.00th=[ 39059456],
- | 99.00th=[183500800], 99.50th=[295698432], 99.90th=[501219328],
- | 99.95th=[526385152], 99.99th=[616562688]
- bw ( KiB/s): min= 0, max= 4812, per=14.28%, avg=3258.00, stdev=2197.69, samples=2
- iops : min= 0, max= 75, avg=50.50, stdev=34.65, samples=2
- write: IOPS=52, BW=3370KiB/s (3451kB/s)(395MiB/120007msec)
- slat (nsec): min=0, max=240431k, avg=228343.91, stdev=3035908.98
- clat (nsec): min=0, max=1074.2M, avg=68752131.71, stdev=128360758.51
- lat (nsec): min=0, max=1074.3M, avg=68981714.33, stdev=128329619.31
- clat percentiles (usec):
- | 1.00th=[ 1004], 5.00th=[ 1237], 10.00th=[ 1467],
- | 20.00th=[ 2008], 30.00th=[ 2868], 40.00th=[ 4555],
- | 50.00th=[ 10159], 60.00th=[ 22152], 70.00th=[ 66847],
- | 80.00th=[ 105382], 90.00th=[ 187696], 95.00th=[ 350225],
- | 99.00th=[ 650118], 99.50th=[ 734004], 99.90th=[ 834667],
- | 99.95th=[ 893387], 99.99th=[1082131]
- bw ( KiB/s): min= 0, max= 4920, per=54.05%, avg=3367.50, stdev=2195.57, samples=2
- iops : min= 0, max= 76, avg=52.00, stdev=33.94, samples=2
- lat (nsec) : 750=5.93%, 1000=13.00%
- lat (usec) : 2=14.02%, 4=0.05%, 10=0.27%, 20=0.83%, 50=0.79%
- lat (usec) : 100=0.76%, 250=0.79%, 500=0.92%, 750=0.79%, 1000=1.89%
- lat (msec) : 2=12.49%, 4=10.62%, 10=7.48%, 20=5.36%, 50=5.31%
- lat (msec) : 100=6.99%, 250=7.43%, 500=3.07%, 750=1.05%, 1000=0.18%
- cpu : usr=0.06%, sys=0.30%, ctx=14609, majf=0, minf=31
- IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=6111,6316,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- Run status group 0 (all jobs):
- READ: bw=22.3MiB/s (23.4MB/s), 3258KiB/s-19.1MiB/s (3337kB/s-20.0MB/s), io=2674MiB (2804MB), run=120007-120015msec
- WRITE: bw=6230KiB/s (6380kB/s), 2862KiB/s-3370KiB/s (2931kB/s-3451kB/s), io=731MiB (766MB), run=120007-120068msec
- Disk stats (read/write):
- rbd246: ios=9675/12516, merge=0/1, ticks=561703/838638, in_queue=1389101, util=15.18%
- randwrite: (groupid=0, jobs=1): err= 0: pid=11: Wed May 8 20:35:00 2019
- write: IOPS=27, BW=1792KiB/s (1835kB/s)(211MiB/120300msec)
- slat (nsec): min=0, max=710004k, avg=768267.47, stdev=17305830.82
- clat (nsec): min=0, max=2194.6M, avg=142987318.59, stdev=256277381.03
- lat (nsec): min=0, max=2194.7M, avg=143756261.10, stdev=257049780.78
- clat percentiles (usec):
- | 1.00th=[ 1123], 5.00th=[ 1418], 10.00th=[ 1696],
- | 20.00th=[ 2409], 30.00th=[ 3326], 40.00th=[ 5014],
- | 50.00th=[ 9110], 60.00th=[ 38011], 70.00th=[ 105382],
- | 80.00th=[ 221250], 90.00th=[ 526386], 95.00th=[ 725615],
- | 99.00th=[1115685], 99.50th=[1249903], 99.90th=[1417675],
- | 99.95th=[1870660], 99.99th=[2197816]
- bw ( KiB/s): min= 0, max= 2231, per=31.72%, avg=1794.00, stdev=618.01, samples=2
- iops : min= 0, max= 34, avg=27.50, stdev= 9.19, samples=2
- lat (usec) : 1000=0.27%
- lat (msec) : 2=14.38%, 4=20.68%, 10=15.75%, 20=5.61%, 50=4.58%
- lat (msec) : 100=7.34%, 250=13.13%, 500=7.49%, 750=6.24%, 1000=3.00%
- cpu : usr=0.03%, sys=0.12%, ctx=6249, majf=0, minf=30
- IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=0,3366,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- read: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:35:00 2019
- read: IOPS=486, BW=30.3MiB/s (31.8MB/s)(3644MiB/120302msec)
- slat (nsec): min=0, max=481883k, avg=40883.35, stdev=3292713.26
- clat (nsec): min=0, max=2142.9M, avg=8229375.61, stdev=64916449.31
- lat (nsec): min=0, max=2143.1M, avg=8271109.54, stdev=65013247.85
- clat percentiles (nsec):
- | 1.00th=[ 716], 5.00th=[ 724], 10.00th=[ 724],
- | 20.00th=[ 740], 30.00th=[ 772], 40.00th=[ 796],
- | 50.00th=[ 844], 60.00th=[ 876], 70.00th=[ 916],
- | 80.00th=[ 1096], 90.00th=[ 12736], 95.00th=[ 2039808],
- | 99.00th=[ 354418688], 99.50th=[ 505413632], 99.90th=[ 859832320],
- | 99.95th=[ 960495616], 99.99th=[1384120320]
- bw ( KiB/s): min= 0, max=47390, per=89.35%, avg=31095.00, stdev=23044.61, samples=2
- iops : min= 0, max= 743, avg=487.00, stdev=362.04, samples=2
- lat (nsec) : 750=26.66%, 1000=51.00%
- lat (usec) : 2=7.21%, 4=0.10%, 10=1.34%, 20=5.40%, 50=0.82%
- lat (usec) : 100=0.15%, 250=0.08%, 500=0.08%, 750=0.24%, 1000=0.64%
- lat (msec) : 2=1.24%, 4=0.96%, 10=0.53%, 20=0.22%, 50=0.79%
- lat (msec) : 100=0.88%, 250=0.44%, 500=0.71%, 750=0.34%, 1000=0.13%
- cpu : usr=0.13%, sys=0.48%, ctx=8589, majf=0, minf=99
- IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=58472,0,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- readwrite: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:35:00 2019
- read: IOPS=59, BW=3790KiB/s (3881kB/s)(444MiB/120012msec)
- slat (nsec): min=0, max=2853.8k, avg=26733.30, stdev=79399.49
- clat (nsec): min=0, max=768103k, avg=6051219.37, stdev=36611268.74
- lat (nsec): min=0, max=768124k, avg=6078828.42, stdev=36617018.80
- clat percentiles (nsec):
- | 1.00th=[ 716], 5.00th=[ 724], 10.00th=[ 732],
- | 20.00th=[ 764], 30.00th=[ 852], 40.00th=[ 948],
- | 50.00th=[ 1096], 60.00th=[ 1240], 70.00th=[ 9536],
- | 80.00th=[ 370688], 90.00th=[ 2605056], 95.00th=[ 27394048],
- | 99.00th=[139460608], 99.50th=[252706816], 99.90th=[526385152],
- | 99.95th=[641728512], 99.99th=[767557632]
- bw ( KiB/s): min= 0, max= 6400, per=10.89%, avg=3790.00, stdev=3691.10, samples=2
- iops : min= 0, max= 100, avg=59.00, stdev=57.98, samples=2
- write: IOPS=60, BW=3873KiB/s (3966kB/s)(454MiB/120012msec)
- slat (nsec): min=0, max=368178k, avg=260495.36, stdev=4957786.34
- clat (nsec): min=0, max=1192.9M, avg=60005346.24, stdev=128875303.67
- lat (nsec): min=0, max=1192.9M, avg=60267041.02, stdev=129001065.91
- clat percentiles (usec):
- | 1.00th=[ 1012], 5.00th=[ 1254], 10.00th=[ 1467],
- | 20.00th=[ 1942], 30.00th=[ 2573], 40.00th=[ 3589],
- | 50.00th=[ 5473], 60.00th=[ 11600], 70.00th=[ 29230],
- | 80.00th=[ 90702], 90.00th=[ 160433], 95.00th=[ 333448],
- | 99.00th=[ 650118], 99.50th=[ 725615], 99.90th=[ 876610],
- | 99.95th=[1052771], 99.99th=[1199571]
- bw ( KiB/s): min= 0, max= 6501, per=68.45%, avg=3871.00, stdev=3719.38, samples=2
- iops : min= 0, max= 101, avg=60.00, stdev=57.98, samples=2
- lat (nsec) : 750=8.53%, 1000=13.09%
- lat (usec) : 2=12.74%, 4=0.03%, 10=0.36%, 20=1.38%, 50=0.77%
- lat (usec) : 100=1.00%, 250=1.13%, 500=0.86%, 750=0.65%, 1000=1.64%
- lat (msec) : 2=12.52%, 4=12.26%, 10=8.76%, 20=4.87%, 50=4.41%
- lat (msec) : 100=5.01%, 250=6.32%, 500=2.24%, 750=1.23%, 1000=0.17%
- cpu : usr=0.07%, sys=0.33%, ctx=15734, majf=0, minf=33
- IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=7108,7259,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- Run status group 0 (all jobs):
- READ: bw=33.0MiB/s (35.6MB/s), 3790KiB/s-30.3MiB/s (3881kB/s-31.8MB/s), io=4088MiB (4287MB), run=120012-120302msec
- WRITE: bw=5656KiB/s (5791kB/s), 1792KiB/s-3873KiB/s (1835kB/s-3966kB/s), io=664MiB (697MB), run=120012-120300msec
- Disk stats (read/write):
- rbd62: ios=6007/10900, merge=0/1, ticks=553071/887900, in_queue=1432491, util=9.76%
- <example-benchmark-fio-server-7-benchmark-n8krm>
- randwrite: (groupid=0, jobs=1): err= 0: pid=11: Wed May 8 20:35:00 2019
- write: IOPS=50, BW=3256KiB/s (3335kB/s)(382MiB/120041msec)
- slat (nsec): min=0, max=754365k, avg=524418.01, stdev=13825228.46
- clat (nsec): min=0, max=1437.4M, avg=78366317.32, stdev=145975603.65
- lat (nsec): min=0, max=1438.1M, avg=78891857.37, stdev=146417463.46
- clat percentiles (usec):
- | 1.00th=[ 963], 5.00th=[ 1188], 10.00th=[ 1369],
- | 20.00th=[ 1762], 30.00th=[ 2442], 40.00th=[ 3621],
- | 50.00th=[ 6259], 60.00th=[ 18220], 70.00th=[ 81265],
- | 80.00th=[ 112722], 90.00th=[ 250610], 95.00th=[ 400557],
- | 99.00th=[ 700449], 99.50th=[ 759170], 99.90th=[ 901776],
- | 99.95th=[ 926942], 99.99th=[1434452]
- bw ( KiB/s): min= 0, max= 3328, per=42.00%, avg=3256.00, stdev=101.82, samples=2
- iops : min= 0, max= 52, avg=50.50, stdev= 2.12, samples=2
- lat (usec) : 500=0.02%, 750=0.07%, 1000=1.54%
- lat (msec) : 2=22.49%, 4=18.21%, 10=13.19%, 20=5.03%, 50=3.37%
- lat (msec) : 100=12.14%, 250=13.94%, 500=6.75%, 750=2.74%, 1000=0.54%
- cpu : usr=0.06%, sys=0.21%, ctx=11444, majf=0, minf=26
- IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=0,6105,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- read: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:35:00 2019
- read: IOPS=344, BW=21.5MiB/s (22.6MB/s)(2581MiB/120018msec)
- slat (nsec): min=0, max=507784k, avg=64993.68, stdev=4454457.57
- clat (nsec): min=0, max=1336.8M, avg=11573441.71, stdev=60446260.20
- lat (nsec): min=0, max=1336.8M, avg=11639287.24, stdev=60613041.39
- clat percentiles (nsec):
- | 1.00th=[ 708], 5.00th=[ 716], 10.00th=[ 724],
- | 20.00th=[ 724], 30.00th=[ 732], 40.00th=[ 748],
- | 50.00th=[ 820], 60.00th=[ 884], 70.00th=[ 1048],
- | 80.00th=[ 11968], 90.00th=[ 3391488], 95.00th=[ 57409536],
- | 99.00th=[337641472], 99.50th=[480247808], 99.90th=[750780416],
- | 99.95th=[809500672], 99.99th=[977272832]
- bw ( KiB/s): min= 0, max=31315, per=83.20%, avg=22024.00, stdev=13139.46, samples=2
- iops : min= 0, max= 490, avg=344.00, stdev=206.48, samples=2
- lat (nsec) : 750=41.49%, 1000=27.40%
- lat (usec) : 2=9.78%, 4=0.07%, 10=1.14%, 20=0.61%, 50=0.39%
- lat (usec) : 100=0.12%, 250=0.05%, 500=0.34%, 750=1.01%, 1000=2.49%
- lat (msec) : 2=3.47%, 4=1.99%, 10=1.02%, 20=0.42%, 50=2.40%
- lat (msec) : 100=3.23%, 250=1.24%, 500=0.93%, 750=0.34%, 1000=0.09%
- cpu : usr=0.11%, sys=0.39%, ctx=15643, majf=0, minf=64
- IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=41370,0,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- readwrite: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:35:00 2019
- read: IOPS=69, BW=4450KiB/s (4557kB/s)(522MiB/120002msec)
- slat (nsec): min=0, max=750545k, avg=167895.13, stdev=9048304.87
- clat (nsec): min=0, max=997524k, avg=6591996.29, stdev=37271220.87
- lat (nsec): min=0, max=997578k, avg=6760880.42, stdev=38337231.16
- clat percentiles (nsec):
- | 1.00th=[ 716], 5.00th=[ 724], 10.00th=[ 740],
- | 20.00th=[ 804], 30.00th=[ 916], 40.00th=[ 1064],
- | 50.00th=[ 1208], 60.00th=[ 1496], 70.00th=[ 610304],
- | 80.00th=[ 1286144], 90.00th=[ 4145152], 95.00th=[ 34865152],
- | 99.00th=[141557760], 99.50th=[265289728], 99.90th=[574619648],
- | 99.95th=[591396864], 99.99th=[994050048]
- bw ( KiB/s): min= 0, max= 6443, per=16.81%, avg=4449.00, stdev=2819.94, samples=2
- iops : min= 0, max= 100, avg=69.00, stdev=43.84, samples=2
- write: IOPS=70, BW=4498KiB/s (4605kB/s)(527MiB/120002msec)
- slat (nsec): min=0, max=516019k, avg=258656.32, stdev=5651967.71
- clat (nsec): min=0, max=1458.4M, avg=50155127.58, stdev=113685456.08
- lat (nsec): min=0, max=1458.4M, avg=50415075.39, stdev=113756135.61
- clat percentiles (usec):
- | 1.00th=[ 1020], 5.00th=[ 1254], 10.00th=[ 1434],
- | 20.00th=[ 1778], 30.00th=[ 2245], 40.00th=[ 2900],
- | 50.00th=[ 3916], 60.00th=[ 6325], 70.00th=[ 19268],
- | 80.00th=[ 74974], 90.00th=[ 129500], 95.00th=[ 295699],
- | 99.00th=[ 591397], 99.50th=[ 683672], 99.90th=[ 784335],
- | 99.95th=[ 901776], 99.99th=[1451230]
- bw ( KiB/s): min= 0, max= 6458, per=57.99%, avg=4495.50, stdev=2775.39, samples=2
- iops : min= 0, max= 100, avg=69.50, stdev=43.13, samples=2
- lat (nsec) : 750=7.04%, 1000=11.01%
- lat (usec) : 2=12.70%, 4=0.05%, 10=0.27%, 20=0.67%, 50=0.56%
- lat (usec) : 100=0.59%, 250=0.83%, 500=0.88%, 750=1.10%, 1000=2.94%
- lat (msec) : 2=16.24%, 4=15.20%, 10=8.56%, 20=3.31%, 50=4.02%
- lat (msec) : 100=5.96%, 250=4.79%, 500=2.43%, 750=0.76%, 1000=0.09%
- cpu : usr=0.08%, sys=0.39%, ctx=20699, majf=0, minf=31
- IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=8344,8429,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- Run status group 0 (all jobs):
- READ: bw=25.9MiB/s (27.1MB/s), 4450KiB/s-21.5MiB/s (4557kB/s-22.6MB/s), io=3103MiB (3253MB), run=120002-120018msec
- WRITE: bw=7753KiB/s (7939kB/s), 3256KiB/s-4498KiB/s (3335kB/s-4605kB/s), io=909MiB (953MB), run=120002-120041msec
- Disk stats (read/write):
- rbd6: ios=11216/15012, merge=0/3, ticks=566580/831401, in_queue=1384528, util=16.71%
- <example-benchmark-fio-server-3-benchmark-tx76m>
- randwrite: (groupid=0, jobs=1): err= 0: pid=11: Wed May 8 20:35:00 2019
- write: IOPS=18, BW=1156KiB/s (1184kB/s)(136MiB/120093msec)
- slat (nsec): min=0, max=1871.0M, avg=2769651.20, stdev=56431218.59
- clat (nsec): min=0, max=2617.7M, avg=220504187.67, stdev=349885888.10
- lat (nsec): min=0, max=2617.8M, avg=223272360.64, stdev=353092581.18
- clat percentiles (usec):
- | 1.00th=[ 1074], 5.00th=[ 1303], 10.00th=[ 1565],
- | 20.00th=[ 2180], 30.00th=[ 3228], 40.00th=[ 5473],
- | 50.00th=[ 11731], 60.00th=[ 80217], 70.00th=[ 204473],
- | 80.00th=[ 501220], 90.00th=[ 734004], 95.00th=[ 918553],
- | 99.00th=[1400898], 99.50th=[1870660], 99.90th=[2071987],
- | 99.95th=[2197816], 99.99th=[2634023]
- bw ( KiB/s): min= 0, max= 1699, per=21.12%, avg=1155.00, stdev=769.33, samples=2
- iops : min= 0, max= 26, avg=17.50, stdev=12.02, samples=2
- lat (usec) : 4=0.05%, 1000=0.51%
- lat (msec) : 2=16.75%, 4=17.17%, 10=13.71%, 20=6.46%, 50=2.58%
- lat (msec) : 100=4.80%, 250=8.86%, 500=9.23%, 750=10.57%, 1000=5.86%
- cpu : usr=0.02%, sys=0.09%, ctx=4098, majf=0, minf=30
- IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=0,2167,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- read: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:35:00 2019
- read: IOPS=545, BW=33.9MiB/s (35.6MB/s)(4074MiB/120118msec)
- slat (nsec): min=0, max=1870.9M, avg=90467.46, stdev=10011085.33
- clat (nsec): min=0, max=1903.2M, avg=7293770.07, stdev=69808358.17
- lat (nsec): min=0, max=1903.2M, avg=7385182.62, stdev=70527189.25
- clat percentiles (nsec):
- | 1.00th=[ 708], 5.00th=[ 716], 10.00th=[ 724],
- | 20.00th=[ 740], 30.00th=[ 788], 40.00th=[ 812],
- | 50.00th=[ 852], 60.00th=[ 876], 70.00th=[ 916],
- | 80.00th=[ 1004], 90.00th=[ 12736], 95.00th=[ 24960],
- | 99.00th=[ 354418688], 99.50th=[ 557842432], 99.90th=[ 952107008],
- | 99.95th=[1166016512], 99.99th=[1853882368]
- bw ( KiB/s): min= 0, max=54319, per=89.19%, avg=34761.00, stdev=27659.19, samples=2
- iops : min= 0, max= 855, avg=546.00, stdev=436.99, samples=2
- lat (nsec) : 750=21.71%, 1000=58.02%
- lat (usec) : 2=5.73%, 4=0.13%, 10=0.90%, 20=5.66%, 50=4.13%
- lat (usec) : 100=0.14%, 250=0.06%, 500=0.06%, 750=0.15%, 1000=0.34%
- lat (msec) : 2=0.50%, 4=0.38%, 10=0.18%, 20=0.10%, 50=0.26%
- lat (msec) : 100=0.30%, 250=0.14%, 500=0.52%, 750=0.34%, 1000=0.17%
- cpu : usr=0.12%, sys=0.51%, ctx=4867, majf=0, minf=30
- IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=65572,0,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- readwrite: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:35:00 2019
- read: IOPS=66, BW=4247KiB/s (4349kB/s)(498MiB/120004msec)
- slat (nsec): min=0, max=657901k, avg=106802.92, stdev=7372430.83
- clat (nsec): min=0, max=744115k, avg=4777220.47, stdev=35459077.18
- lat (nsec): min=0, max=744165k, avg=4884922.55, stdev=36213602.95
- clat percentiles (nsec):
- | 1.00th=[ 716], 5.00th=[ 724], 10.00th=[ 732],
- | 20.00th=[ 756], 30.00th=[ 852], 40.00th=[ 932],
- | 50.00th=[ 1080], 60.00th=[ 1224], 70.00th=[ 1576],
- | 80.00th=[ 84480], 90.00th=[ 1122304], 95.00th=[ 5799936],
- | 99.00th=[141557760], 99.50th=[256901120], 99.90th=[557842432],
- | 99.95th=[633339904], 99.99th=[742391808]
- bw ( KiB/s): min= 0, max= 7251, per=10.89%, avg=4246.00, stdev=4249.71, samples=2
- iops : min= 0, max= 113, avg=66.00, stdev=66.47, samples=2
- write: IOPS=67, BW=4317KiB/s (4421kB/s)(506MiB/120004msec)
- slat (nsec): min=0, max=445053k, avg=240599.24, stdev=4950521.64
- clat (nsec): min=0, max=1125.9M, avg=54461828.03, stdev=123421580.22
- lat (nsec): min=0, max=1125.0M, avg=54703547.45, stdev=123454883.67
- clat percentiles (usec):
- | 1.00th=[ 1012], 5.00th=[ 1221], 10.00th=[ 1369],
- | 20.00th=[ 1713], 30.00th=[ 2147], 40.00th=[ 2835],
- | 50.00th=[ 4178], 60.00th=[ 7898], 70.00th=[ 19792],
- | 80.00th=[ 80217], 90.00th=[ 149947], 95.00th=[ 304088],
- | 99.00th=[ 633340], 99.50th=[ 775947], 99.90th=[ 935330],
- | 99.95th=[ 943719], 99.99th=[1132463]
- bw ( KiB/s): min= 0, max= 7336, per=78.88%, avg=4315.00, stdev=4272.34, samples=2
- iops : min= 0, max= 114, avg=67.00, stdev=66.47, samples=2
- lat (nsec) : 750=8.84%, 1000=13.77%
- lat (usec) : 2=12.64%, 4=0.04%, 10=0.65%, 20=1.71%, 50=0.92%
- lat (usec) : 100=1.36%, 250=1.94%, 500=0.83%, 750=0.54%, 1000=1.45%
- lat (msec) : 2=14.44%, 4=12.42%, 10=7.87%, 20=3.83%, 50=3.23%
- lat (msec) : 100=4.75%, 250=5.61%, 500=2.03%, 750=0.84%, 1000=0.29%
- cpu : usr=0.09%, sys=0.37%, ctx=16379, majf=0, minf=30
- IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=7964,8092,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- Run status group 0 (all jobs):
- READ: bw=38.1MiB/s (39.9MB/s), 4247KiB/s-33.9MiB/s (4349kB/s-35.6MB/s), io=4572MiB (4794MB), run=120004-120118msec
- WRITE: bw=5470KiB/s (5602kB/s), 1156KiB/s-4317KiB/s (1184kB/s-4421kB/s), io=642MiB (673MB), run=120004-120093msec
- Disk stats (read/write):
- rbd1: ios=3345/10492, merge=0/1, ticks=511851/876978, in_queue=1381859, util=7.20%
- <example-benchmark-fio-server-225-benchmark-d2hts>
- randwrite: (groupid=0, jobs=1): err= 0: pid=11: Wed May 8 20:35:00 2019
- write: IOPS=16, BW=1079KiB/s (1105kB/s)(127MiB/120445msec)
- slat (nsec): min=0, max=860283k, avg=1696157.53, stdev=32922611.14
- clat (nsec): min=0, max=2310.5M, avg=236942025.07, stdev=429727850.91
- lat (nsec): min=0, max=2311.1M, avg=238639284.04, stdev=430204025.53
- clat percentiles (usec):
- | 1.00th=[ 1221], 5.00th=[ 1827], 10.00th=[ 2278],
- | 20.00th=[ 3228], 30.00th=[ 4424], 40.00th=[ 6128],
- | 50.00th=[ 10028], 60.00th=[ 21627], 70.00th=[ 78119],
- | 80.00th=[ 513803], 90.00th=[ 943719], 95.00th=[1233126],
- | 99.00th=[1568670], 99.50th=[1853883], 99.90th=[2105541],
- | 99.95th=[2122318], 99.99th=[2298479]
- bw ( KiB/s): min= 0, max= 1129, per=20.01%, avg=1079.00, stdev=70.71, samples=2
- iops : min= 0, max= 17, avg=16.50, stdev= 0.71, samples=2
- lat (usec) : 4=0.05%, 500=0.05%, 750=0.05%, 1000=0.35%
- lat (msec) : 2=6.42%, 4=19.69%, 10=23.54%, 20=8.88%, 50=8.05%
- lat (msec) : 100=4.79%, 250=3.80%, 500=4.15%, 750=4.84%, 1000=6.56%
- cpu : usr=0.02%, sys=0.08%, ctx=3498, majf=0, minf=31
- IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=0,2026,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- read: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:35:00 2019
- read: IOPS=369, BW=23.0MiB/s (24.1MB/s)(2774MiB/120466msec)
- slat (nsec): min=0, max=813265k, avg=56599.31, stdev=4929365.07
- clat (nsec): min=0, max=1847.6M, avg=10781330.84, stdev=91567779.88
- lat (nsec): min=0, max=1847.9M, avg=10838874.42, stdev=91712392.55
- clat percentiles (nsec):
- | 1.00th=[ 716], 5.00th=[ 724], 10.00th=[ 732],
- | 20.00th=[ 772], 30.00th=[ 780], 40.00th=[ 804],
- | 50.00th=[ 844], 60.00th=[ 868], 70.00th=[ 900],
- | 80.00th=[ 972], 90.00th=[ 11584], 95.00th=[ 1187840],
- | 99.00th=[ 574619648], 99.50th=[ 775946240], 99.90th=[1166016512],
- | 99.95th=[1350565888], 99.99th=[1719664640]
- bw ( KiB/s): min= 0, max=31363, per=85.02%, avg=23663.00, stdev=10889.44, samples=2
- iops : min= 0, max= 491, avg=370.00, stdev=171.12, samples=2
- lat (nsec) : 750=14.31%, 1000=66.73%
- lat (usec) : 2=7.44%, 4=0.16%, 10=0.95%, 20=3.40%, 50=0.79%
- lat (usec) : 100=0.18%, 250=0.11%, 500=0.11%, 750=0.11%, 1000=0.43%
- lat (msec) : 2=1.18%, 4=0.97%, 10=0.74%, 20=0.29%, 50=0.38%
- lat (msec) : 100=0.24%, 250=0.14%, 500=0.28%, 750=0.51%, 1000=0.44%
- cpu : usr=0.09%, sys=0.39%, ctx=5480, majf=0, minf=31
- IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=44518,0,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- readwrite: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:35:00 2019
- read: IOPS=66, BW=4267KiB/s (4370kB/s)(500MiB/120003msec)
- slat (nsec): min=0, max=4304.4k, avg=23613.57, stdev=74508.36
- clat (nsec): min=0, max=759154k, avg=6090334.59, stdev=46592721.81
- lat (nsec): min=0, max=759266k, avg=6114859.38, stdev=46599886.35
- clat percentiles (nsec):
- | 1.00th=[ 716], 5.00th=[ 724], 10.00th=[ 732],
- | 20.00th=[ 756], 30.00th=[ 836], 40.00th=[ 908],
- | 50.00th=[ 1048], 60.00th=[ 1208], 70.00th=[ 1496],
- | 80.00th=[ 67072], 90.00th=[ 1204224], 95.00th=[ 5996544],
- | 99.00th=[168820736], 99.50th=[434110464], 99.90th=[616562688],
- | 99.95th=[675282944], 99.99th=[759169024]
- bw ( KiB/s): min= 0, max= 5781, per=15.33%, avg=4267.00, stdev=2141.12, samples=2
- iops : min= 0, max= 90, avg=66.50, stdev=33.23, samples=2
- write: IOPS=67, BW=4330KiB/s (4434kB/s)(507MiB/120003msec)
- slat (nsec): min=0, max=6756.8k, avg=175613.08, stdev=232944.45
- clat (nsec): min=0, max=1093.4M, avg=53149774.18, stdev=110172503.70
- lat (nsec): min=0, max=1093.5M, avg=53326567.81, stdev=110133857.81
- clat percentiles (usec):
- | 1.00th=[ 1074], 5.00th=[ 1319], 10.00th=[ 1483],
- | 20.00th=[ 1909], 30.00th=[ 2507], 40.00th=[ 3687],
- | 50.00th=[ 6390], 60.00th=[ 12649], 70.00th=[ 26346],
- | 80.00th=[ 80217], 90.00th=[ 145753], 95.00th=[ 316670],
- | 99.00th=[ 522191], 99.50th=[ 608175], 99.90th=[ 843056],
- | 99.95th=[ 910164], 99.99th=[1098908]
- bw ( KiB/s): min= 0, max= 5804, per=80.26%, avg=4327.50, stdev=2088.09, samples=2
- iops : min= 0, max= 90, avg=67.00, stdev=32.53, samples=2
- lat (nsec) : 750=9.35%, 1000=13.87%
- lat (usec) : 2=12.52%, 4=0.04%, 10=0.68%, 20=1.67%, 50=1.02%
- lat (usec) : 100=1.19%, 250=1.66%, 500=0.94%, 750=0.46%, 1000=1.06%
- lat (msec) : 2=12.31%, 4=11.09%, 10=8.16%, 20=5.56%, 50=4.91%
- lat (msec) : 100=4.41%, 250=5.52%, 500=2.78%, 750=0.69%, 1000=0.13%
- cpu : usr=0.09%, sys=0.36%, ctx=16242, majf=0, minf=31
- IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=8001,8116,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- Run status group 0 (all jobs):
- READ: bw=27.2MiB/s (28.5MB/s), 4267KiB/s-23.0MiB/s (4370kB/s-24.1MB/s), io=3274MiB (3433MB), run=120003-120466msec
- WRITE: bw=5393KiB/s (5522kB/s), 1079KiB/s-4330KiB/s (1105kB/s-4434kB/s), io=634MiB (665MB), run=120003-120445msec
- Disk stats (read/write):
- rbd76: ios=3853/10268, merge=0/1, ticks=568825/896607, in_queue=1458316, util=5.45%
- <example-benchmark-fio-server-499-benchmark-hgwjk>
- randwrite: (groupid=0, jobs=1): err= 0: pid=11: Wed May 8 20:35:00 2019
- write: IOPS=20, BW=1336KiB/s (1369kB/s)(158MiB/121394msec)
- slat (nsec): min=0, max=521835k, avg=602332.19, stdev=11489635.80
- clat (nsec): min=0, max=2418.6M, avg=191197413.08, stdev=299955700.19
- lat (nsec): min=0, max=2418.6M, avg=191801070.37, stdev=299979092.81
- clat percentiles (usec):
- | 1.00th=[ 1057], 5.00th=[ 1418], 10.00th=[ 1762],
- | 20.00th=[ 2507], 30.00th=[ 3523], 40.00th=[ 5080],
- | 50.00th=[ 10159], 60.00th=[ 63701], 70.00th=[ 244319],
- | 80.00th=[ 408945], 90.00th=[ 633340], 95.00th=[ 759170],
- | 99.00th=[1182794], 99.50th=[1400898], 99.90th=[2432697],
- | 99.95th=[2432697], 99.99th=[2432697]
- bw ( KiB/s): min= 0, max= 1506, per=24.22%, avg=1348.00, stdev=223.45, samples=2
- iops : min= 0, max= 23, avg=20.50, stdev= 3.54, samples=2
- lat (usec) : 4=0.04%, 100=0.04%, 500=0.04%, 750=0.04%, 1000=0.51%
- lat (msec) : 2=12.99%, 4=19.87%, 10=16.43%, 20=6.04%, 50=2.61%
- lat (msec) : 100=4.19%, 250=7.58%, 500=14.81%, 750=9.87%, 1000=2.45%
- cpu : usr=0.03%, sys=0.09%, ctx=4216, majf=0, minf=31
- IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=0,2532,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- read: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:35:00 2019
- read: IOPS=139, BW=8931KiB/s (9145kB/s)(1048MiB/120186msec)
- slat (nsec): min=0, max=9380.9k, avg=29479.14, stdev=141498.82
- clat (nsec): min=0, max=1852.7M, avg=28689495.96, stdev=142920150.94
- lat (nsec): min=0, max=1852.8M, avg=28720090.77, stdev=142943141.98
- clat percentiles (nsec):
- | 1.00th=[ 716], 5.00th=[ 724], 10.00th=[ 732],
- | 20.00th=[ 772], 30.00th=[ 788], 40.00th=[ 836],
- | 50.00th=[ 868], 60.00th=[ 900], 70.00th=[ 1048],
- | 80.00th=[ 1416], 90.00th=[ 2703360], 95.00th=[ 43253760],
- | 99.00th=[ 817889280], 99.50th=[ 943718400], 99.90th=[1283457024],
- | 99.95th=[1451229184], 99.99th=[1837105152]
- bw ( KiB/s): min= 0, max= 9136, per=67.97%, avg=8932.50, stdev=287.79, samples=2
- iops : min= 0, max= 142, avg=139.00, stdev= 4.24, samples=2
- lat (nsec) : 750=12.69%, 1000=55.86%
- lat (usec) : 2=14.55%, 4=0.18%, 10=0.29%, 20=0.39%, 50=0.17%
- lat (usec) : 100=0.13%, 250=0.04%, 500=0.08%, 750=0.26%, 1000=1.25%
- lat (msec) : 2=3.16%, 4=2.30%, 10=1.74%, 20=0.89%, 50=1.22%
- lat (msec) : 100=0.52%, 250=0.43%, 500=0.91%, 750=1.57%, 1000=1.01%
- cpu : usr=0.05%, sys=0.18%, ctx=5400, majf=0, minf=31
- IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=16768,0,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- readwrite: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:35:00 2019
- read: IOPS=65, BW=4218KiB/s (4319kB/s)(494MiB/120008msec)
- slat (nsec): min=0, max=3874.9k, avg=26448.84, stdev=87557.02
- clat (nsec): min=0, max=904621k, avg=9496297.79, stdev=57007692.17
- lat (nsec): min=0, max=905014k, avg=9523688.60, stdev=57015747.35
- clat percentiles (nsec):
- | 1.00th=[ 716], 5.00th=[ 724], 10.00th=[ 732],
- | 20.00th=[ 748], 30.00th=[ 852], 40.00th=[ 948],
- | 50.00th=[ 1096], 60.00th=[ 1256], 70.00th=[ 1544],
- | 80.00th=[ 164864], 90.00th=[ 2473984], 95.00th=[ 30801920],
- | 99.00th=[346030080], 99.50th=[488636416], 99.90th=[666894336],
- | 99.95th=[717225984], 99.99th=[901775360]
- bw ( KiB/s): min= 0, max= 6002, per=32.10%, avg=4218.00, stdev=2522.96, samples=2
- iops : min= 0, max= 93, avg=65.50, stdev=38.89, samples=2
- write: IOPS=66, BW=4278KiB/s (4381kB/s)(501MiB/120008msec)
- slat (nsec): min=0, max=8021.8k, avg=195191.63, stdev=241106.21
- clat (nsec): min=0, max=930859k, avg=50434385.70, stdev=108648012.81
- lat (nsec): min=0, max=930927k, avg=50630816.52, stdev=108598568.70
- clat percentiles (usec):
- | 1.00th=[ 988], 5.00th=[ 1188], 10.00th=[ 1369], 20.00th=[ 1729],
- | 30.00th=[ 2212], 40.00th=[ 2966], 50.00th=[ 4555], 60.00th=[ 9896],
- | 70.00th=[ 26346], 80.00th=[ 79168], 90.00th=[131597], 95.00th=[261096],
- | 99.00th=[549454], 99.50th=[683672], 99.90th=[884999], 99.95th=[918553],
- | 99.99th=[926942]
- bw ( KiB/s): min= 0, max= 6036, per=76.84%, avg=4276.00, stdev=2489.02, samples=2
- iops : min= 0, max= 94, avg=66.50, stdev=38.89, samples=2
- lat (nsec) : 750=9.89%, 1000=11.90%
- lat (usec) : 2=13.76%, 4=0.02%, 10=0.42%, 20=1.27%, 50=0.87%
- lat (usec) : 100=1.00%, 250=1.09%, 500=0.65%, 750=0.62%, 1000=1.59%
- lat (msec) : 2=14.31%, 4=11.99%, 10=7.35%, 20=3.99%, 50=4.79%
- lat (msec) : 100=5.32%, 250=5.96%, 500=2.30%, 750=0.78%, 1000=0.16%
- cpu : usr=0.09%, sys=0.36%, ctx=16941, majf=0, minf=34
- IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=7909,8019,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- Run status group 0 (all jobs):
- READ: bw=12.8MiB/s (13.5MB/s), 4218KiB/s-8931KiB/s (4319kB/s-9145kB/s), io=1543MiB (1617MB), run=120008-120186msec
- WRITE: bw=5566KiB/s (5699kB/s), 1336KiB/s-4278KiB/s (1369kB/s-4381kB/s), io=660MiB (692MB), run=120008-121394msec
- Disk stats (read/write):
- rbd210: ios=4224/10915, merge=0/2, ticks=600452/894257, in_queue=1487171, util=4.82%
- <example-benchmark-fio-server-352-benchmark-swk2k>
- randwrite: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:35:01 2019
- write: IOPS=19, BW=1250KiB/s (1280kB/s)(149MiB/122256msec)
- slat (nsec): min=0, max=3241.0M, avg=1882431.65, stdev=67552986.62
- clat (nsec): min=0, max=3320.5M, avg=201732832.03, stdev=317272332.29
- lat (nsec): min=0, max=3373.9M, avg=203614730.47, stdev=324396055.25
- clat percentiles (usec):
- | 1.00th=[ 1139], 5.00th=[ 1713], 10.00th=[ 2442],
- | 20.00th=[ 3818], 30.00th=[ 6456], 40.00th=[ 10421],
- | 50.00th=[ 18482], 60.00th=[ 54264], 70.00th=[ 233833],
- | 80.00th=[ 450888], 90.00th=[ 692061], 95.00th=[ 826278],
- | 99.00th=[1132463], 99.50th=[1367344], 99.90th=[3238003],
- | 99.95th=[3305112], 99.99th=[3305112]
- bw ( KiB/s): min= 0, max= 1384, per=23.96%, avg=1270.50, stdev=160.51, samples=2
- iops : min= 0, max= 21, avg=19.50, stdev= 2.12, samples=2
- lat (usec) : 500=0.04%, 1000=0.42%
- lat (msec) : 2=6.71%, 4=13.67%, 10=18.54%, 20=12.33%, 50=7.76%
- lat (msec) : 100=4.49%, 250=7.09%, 500=11.91%, 750=9.77%, 1000=4.82%
- cpu : usr=0.03%, sys=0.08%, ctx=4156, majf=0, minf=29
- IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=0,2384,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- read: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:35:01 2019
- read: IOPS=150, BW=9662KiB/s (9894kB/s)(1137MiB/120533msec)
- slat (nsec): min=0, max=11726k, avg=30343.88, stdev=150033.44
- clat (nsec): min=0, max=1774.4M, avg=26494562.90, stdev=136028998.42
- lat (nsec): min=0, max=1774.5M, avg=26526057.61, stdev=136047028.65
- clat percentiles (nsec):
- | 1.00th=[ 716], 5.00th=[ 724], 10.00th=[ 732],
- | 20.00th=[ 780], 30.00th=[ 796], 40.00th=[ 836],
- | 50.00th=[ 876], 60.00th=[ 916], 70.00th=[ 1112],
- | 80.00th=[ 1512], 90.00th=[ 2965504], 95.00th=[ 36962304],
- | 99.00th=[ 792723456], 99.50th=[ 935329792], 99.90th=[1233125376],
- | 99.95th=[1350565888], 99.99th=[1669332992]
- bw ( KiB/s): min= 0, max= 9968, per=70.80%, avg=9702.50, stdev=375.47, samples=2
- iops : min= 0, max= 155, avg=151.00, stdev= 5.66, samples=2
- lat (nsec) : 750=13.35%, 1000=53.24%
- lat (usec) : 2=14.88%, 4=0.20%, 10=0.15%, 20=0.68%, 50=0.20%
- lat (usec) : 100=0.08%, 250=0.06%, 500=0.10%, 750=0.29%, 1000=1.26%
- lat (msec) : 2=3.73%, 4=2.91%, 10=2.18%, 20=0.86%, 50=1.28%
- lat (msec) : 100=0.60%, 250=0.36%, 500=0.92%, 750=1.40%, 1000=0.96%
- cpu : usr=0.05%, sys=0.19%, ctx=6330, majf=0, minf=30
- IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=18194,0,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- readwrite: (groupid=0, jobs=1): err= 0: pid=14: Wed May 8 20:35:01 2019
- read: IOPS=63, BW=4060KiB/s (4157kB/s)(476MiB/120009msec)
- slat (nsec): min=0, max=2641.9k, avg=25605.77, stdev=72611.64
- clat (nsec): min=0, max=995069k, avg=6187294.75, stdev=39693365.11
- lat (nsec): min=0, max=995130k, avg=6213825.74, stdev=39700139.79
- clat percentiles (nsec):
- | 1.00th=[ 716], 5.00th=[ 724], 10.00th=[ 724],
- | 20.00th=[ 748], 30.00th=[ 844], 40.00th=[ 924],
- | 50.00th=[ 1064], 60.00th=[ 1224], 70.00th=[ 1528],
- | 80.00th=[ 142336], 90.00th=[ 1761280], 95.00th=[ 11993088],
- | 99.00th=[200278016], 99.50th=[320864256], 99.90th=[517996544],
- | 99.95th=[641728512], 99.99th=[994050048]
- bw ( KiB/s): min= 0, max= 6873, per=29.62%, avg=4059.50, stdev=3978.89, samples=2
- iops : min= 0, max= 107, avg=63.00, stdev=62.23, samples=2
- write: IOPS=64, BW=4129KiB/s (4228kB/s)(484MiB/120009msec)
- slat (nsec): min=0, max=6809.1k, avg=193732.28, stdev=258346.27
- clat (nsec): min=0, max=871312k, avg=55941812.81, stdev=118854266.47
- lat (nsec): min=0, max=871345k, avg=56136695.07, stdev=118806735.79
- clat percentiles (usec):
- | 1.00th=[ 979], 5.00th=[ 1188], 10.00th=[ 1385], 20.00th=[ 1696],
- | 30.00th=[ 2147], 40.00th=[ 2868], 50.00th=[ 4178], 60.00th=[ 7504],
- | 70.00th=[ 29230], 80.00th=[ 87557], 90.00th=[160433], 95.00th=[325059],
- | 99.00th=[608175], 99.50th=[666895], 99.90th=[775947], 99.95th=[843056],
- | 99.99th=[868221]
- bw ( KiB/s): min= 0, max= 6961, per=77.84%, avg=4127.00, stdev=4007.88, samples=2
- iops : min= 0, max= 108, avg=64.00, stdev=62.23, samples=2
- lat (nsec) : 750=9.91%, 1000=12.67%
- lat (usec) : 2=12.85%, 4=0.07%, 10=0.40%, 20=1.26%, 50=0.87%
- lat (usec) : 100=1.07%, 250=1.32%, 500=0.80%, 750=0.54%, 1000=1.93%
- lat (msec) : 2=15.05%, 4=12.04%, 10=8.38%, 20=2.77%, 50=3.50%
- lat (msec) : 100=4.94%, 250=6.21%, 500=2.23%, 750=1.11%, 1000=0.09%
- cpu : usr=0.07%, sys=0.36%, ctx=16160, majf=0, minf=34
- IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=7613,7740,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- Run status group 0 (all jobs):
- READ: bw=13.4MiB/s (14.0MB/s), 4060KiB/s-9662KiB/s (4157kB/s-9894kB/s), io=1613MiB (1691MB), run=120009-120533msec
- WRITE: bw=5303KiB/s (5430kB/s), 1250KiB/s-4129KiB/s (1280kB/s-4228kB/s), io=633MiB (664MB), run=120009-122256msec
- Disk stats (read/write):
- rbd118: ios=4729/10780, merge=0/2, ticks=572879/917855, in_queue=1482954, util=4.44%
- <example-benchmark-fio-server-706-benchmark-xxdl7>
- randwrite: (groupid=0, jobs=1): err= 0: pid=11: Wed May 8 20:35:01 2019
- write: IOPS=26, BW=1719KiB/s (1760kB/s)(206MiB/122670msec)
- slat (nsec): min=0, max=673225k, avg=684647.51, stdev=15798809.05
- clat (nsec): min=0, max=1908.0M, avg=146098595.77, stdev=316285717.18
- lat (nsec): min=0, max=1909.0M, avg=146784260.50, stdev=316853377.15
- clat percentiles (usec):
- | 1.00th=[ 1205], 5.00th=[ 1647], 10.00th=[ 1958],
- | 20.00th=[ 2737], 30.00th=[ 3687], 40.00th=[ 4817],
- | 50.00th=[ 6718], 60.00th=[ 9896], 70.00th=[ 16450],
- | 80.00th=[ 240124], 90.00th=[ 566232], 95.00th=[ 901776],
- | 99.00th=[1384121], 99.50th=[1501561], 99.90th=[1887437],
- | 99.95th=[1904215], 99.99th=[1904215]
- bw ( KiB/s): min= 0, max= 2234, per=29.13%, avg=1748.00, stdev=687.31, samples=2
- iops : min= 0, max= 34, avg=26.50, stdev=10.61, samples=2
- lat (usec) : 100=0.03%, 750=0.15%, 1000=0.24%
- lat (msec) : 2=10.27%, 4=22.48%, 10=27.40%, 20=12.00%, 50=5.80%
- lat (msec) : 100=0.94%, 250=0.79%, 500=8.35%, 750=4.86%, 1000=2.89%
- cpu : usr=0.04%, sys=0.12%, ctx=5526, majf=0, minf=30
- IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=0,3292,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- read: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:35:01 2019
- read: IOPS=233, BW=14.6MiB/s (15.3MB/s)(1759MiB/120651msec)
- slat (nsec): min=0, max=525196k, avg=46795.53, stdev=3133292.21
- clat (nsec): min=0, max=1842.0M, avg=17115408.03, stdev=109396301.96
- lat (nsec): min=0, max=1842.1M, avg=17163208.16, stdev=109539992.27
- clat percentiles (nsec):
- | 1.00th=[ 716], 5.00th=[ 732], 10.00th=[ 748],
- | 20.00th=[ 780], 30.00th=[ 796], 40.00th=[ 836],
- | 50.00th=[ 868], 60.00th=[ 900], 70.00th=[ 988],
- | 80.00th=[ 1384], 90.00th=[ 2039808], 95.00th=[ 13565952],
- | 99.00th=[ 708837376], 99.50th=[ 868220928], 99.90th=[1166016512],
- | 99.95th=[1266679808], 99.99th=[1702887424]
- bw ( KiB/s): min= 0, max=17562, per=78.09%, avg=15005.00, stdev=3616.14, samples=2
- iops : min= 0, max= 274, avg=234.00, stdev=56.57, samples=2
- lat (nsec) : 750=10.45%, 1000=59.97%
- lat (usec) : 2=13.45%, 4=0.21%, 10=0.24%, 20=0.76%, 50=0.25%
- lat (usec) : 100=0.13%, 250=0.11%, 500=0.06%, 750=0.26%, 1000=1.06%
- lat (msec) : 2=2.98%, 4=2.55%, 10=1.99%, 20=1.12%, 50=1.38%
- lat (msec) : 100=0.50%, 250=0.35%, 500=0.55%, 750=0.81%, 1000=0.61%
- cpu : usr=0.08%, sys=0.29%, ctx=8279, majf=0, minf=30
- IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=28138,0,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- readwrite: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:35:01 2019
- read: IOPS=67, BW=4309KiB/s (4413kB/s)(505MiB/120043msec)
- slat (nsec): min=0, max=3424.6k, avg=31999.20, stdev=100206.55
- clat (nsec): min=0, max=740074k, avg=9433999.90, stdev=51294559.77
- lat (nsec): min=0, max=740112k, avg=9466955.19, stdev=51304338.52
- clat percentiles (nsec):
- | 1.00th=[ 716], 5.00th=[ 724], 10.00th=[ 740],
- | 20.00th=[ 796], 30.00th=[ 892], 40.00th=[ 1020],
- | 50.00th=[ 1176], 60.00th=[ 1336], 70.00th=[ 18560],
- | 80.00th=[ 839680], 90.00th=[ 4358144], 95.00th=[ 42205184],
- | 99.00th=[287309824], 99.50th=[417333248], 99.90th=[633339904],
- | 99.95th=[700448768], 99.99th=[742391808]
- bw ( KiB/s): min= 0, max= 6309, per=22.43%, avg=4310.00, stdev=2827.01, samples=2
- iops : min= 0, max= 98, avg=67.00, stdev=43.84, samples=2
- write: IOPS=68, BW=4377KiB/s (4482kB/s)(513MiB/120043msec)
- slat (nsec): min=0, max=5049.0k, avg=193015.97, stdev=236476.27
- clat (nsec): min=0, max=1324.3M, avg=49248474.97, stdev=103898012.15
- lat (nsec): min=0, max=1324.3M, avg=49442877.34, stdev=103853368.88
- clat percentiles (usec):
- | 1.00th=[ 1012], 5.00th=[ 1221], 10.00th=[ 1369],
- | 20.00th=[ 1680], 30.00th=[ 2212], 40.00th=[ 3064],
- | 50.00th=[ 4948], 60.00th=[ 10552], 70.00th=[ 26870],
- | 80.00th=[ 74974], 90.00th=[ 137364], 95.00th=[ 256902],
- | 99.00th=[ 513803], 99.50th=[ 599786], 99.90th=[ 859833],
- | 99.95th=[ 868221], 99.99th=[1317012]
- bw ( KiB/s): min= 0, max= 6345, per=72.92%, avg=4376.00, stdev=2784.59, samples=2
- iops : min= 0, max= 99, avg=68.00, stdev=43.84, samples=2
- lat (nsec) : 750=7.50%, 1000=11.70%
- lat (usec) : 2=14.09%, 4=0.07%, 10=0.43%, 20=1.12%, 50=0.98%
- lat (usec) : 100=0.71%, 250=1.18%, 500=0.66%, 750=0.73%, 1000=2.00%
- lat (msec) : 2=15.26%, 4=11.44%, 10=7.63%, 20=4.57%, 50=5.46%
- lat (msec) : 100=5.42%, 250=5.95%, 500=2.44%, 750=0.60%, 1000=0.10%
- cpu : usr=0.11%, sys=0.38%, ctx=18225, majf=0, minf=38
- IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=8083,8206,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- Run status group 0 (all jobs):
- READ: bw=18.8MiB/s (19.7MB/s), 4309KiB/s-14.6MiB/s (4413kB/s-15.3MB/s), io=2264MiB (2374MB), run=120043-120651msec
- WRITE: bw=6002KiB/s (6146kB/s), 1719KiB/s-4377KiB/s (1760kB/s-4482kB/s), io=719MiB (754MB), run=120043-122670msec
- Disk stats (read/write):
- rbd257: ios=6205/11856, merge=0/2, ticks=596108/895995, in_queue=1483029, util=6.30%
- <example-benchmark-fio-server-473-benchmark-s88vn>
- randwrite: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:35:01 2019
- write: IOPS=26, BW=1692KiB/s (1733kB/s)(200MiB/121075msec)
- slat (nsec): min=0, max=225666k, avg=347336.80, stdev=4842478.92
- clat (nsec): min=0, max=3886.0M, avg=150346798.75, stdev=285940322.20
- lat (nsec): min=0, max=3886.2M, avg=150695339.12, stdev=285937535.10
- clat percentiles (usec):
- | 1.00th=[ 1156], 5.00th=[ 1778], 10.00th=[ 2442],
- | 20.00th=[ 3752], 30.00th=[ 5276], 40.00th=[ 7439],
- | 50.00th=[ 10814], 60.00th=[ 17433], 70.00th=[ 54789],
- | 80.00th=[ 304088], 90.00th=[ 566232], 95.00th=[ 750781],
- | 99.00th=[1019216], 99.50th=[1182794], 99.90th=[3506439],
- | 99.95th=[3841983], 99.99th=[3875537]
- bw ( KiB/s): min= 0, max= 2070, per=32.19%, avg=1702.50, stdev=519.72, samples=2
- iops : min= 0, max= 32, avg=26.00, stdev= 8.49, samples=2
- lat (usec) : 500=0.03%, 1000=0.28%
- lat (msec) : 2=6.35%, 4=15.57%, 10=25.95%, 20=13.91%, 50=7.60%
- lat (msec) : 100=2.03%, 250=4.97%, 500=12.16%, 750=6.22%, 1000=3.91%
- cpu : usr=0.04%, sys=0.11%, ctx=5496, majf=0, minf=31
- IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=0,3198,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- read: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:35:01 2019
- read: IOPS=209, BW=13.1MiB/s (13.7MB/s)(1579MiB/120646msec)
- slat (nsec): min=0, max=10947k, avg=35755.18, stdev=179088.85
- clat (nsec): min=0, max=1843.5M, avg=19068443.95, stdev=112056013.01
- lat (nsec): min=0, max=1843.5M, avg=19105175.28, stdev=112070145.59
- clat percentiles (nsec):
- | 1.00th=[ 716], 5.00th=[ 724], 10.00th=[ 740],
- | 20.00th=[ 772], 30.00th=[ 796], 40.00th=[ 844],
- | 50.00th=[ 884], 60.00th=[ 964], 70.00th=[ 1272],
- | 80.00th=[ 839680], 90.00th=[ 3719168], 95.00th=[ 23461888],
- | 99.00th=[ 734003200], 99.50th=[ 868220928], 99.90th=[1132462080],
- | 99.95th=[1266679808], 99.99th=[1619001344]
- bw ( KiB/s): min= 0, max=14679, per=79.45%, avg=13458.00, stdev=1726.75, samples=2
- iops : min= 0, max= 229, avg=210.00, stdev=26.87, samples=2
- lat (nsec) : 750=11.34%, 1000=49.95%
- lat (usec) : 2=15.71%, 4=0.21%, 10=0.19%, 20=1.02%, 50=0.23%
- lat (usec) : 100=0.12%, 250=0.11%, 500=0.14%, 750=0.47%, 1000=1.90%
- lat (msec) : 2=4.98%, 4=3.97%, 10=2.82%, 20=1.45%, 50=1.75%
- lat (msec) : 100=0.66%, 250=0.52%, 500=0.72%, 750=0.84%, 1000=0.71%
- cpu : usr=0.07%, sys=0.28%, ctx=10686, majf=0, minf=30
- IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=25262,0,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- readwrite: (groupid=0, jobs=1): err= 0: pid=14: Wed May 8 20:35:01 2019
- read: IOPS=55, BW=3556KiB/s (3641kB/s)(417MiB/120042msec)
- slat (nsec): min=0, max=1662.4k, avg=27905.04, stdev=68083.88
- clat (nsec): min=0, max=930076k, avg=7187406.33, stdev=38705839.51
- lat (nsec): min=0, max=930119k, avg=7216297.76, stdev=38712737.80
- clat percentiles (nsec):
- | 1.00th=[ 716], 5.00th=[ 724], 10.00th=[ 732],
- | 20.00th=[ 780], 30.00th=[ 868], 40.00th=[ 988],
- | 50.00th=[ 1144], 60.00th=[ 1304], 70.00th=[ 15424],
- | 80.00th=[ 765952], 90.00th=[ 4079616], 95.00th=[ 39059456],
- | 99.00th=[183500800], 99.50th=[283115520], 99.90th=[591396864],
- | 99.95th=[675282944], 99.99th=[926941184]
- bw ( KiB/s): min= 0, max= 5559, per=20.99%, avg=3556.50, stdev=2831.96, samples=2
- iops : min= 0, max= 86, avg=55.00, stdev=43.84, samples=2
- write: IOPS=56, BW=3628KiB/s (3715kB/s)(425MiB/120042msec)
- slat (nsec): min=0, max=6981.7k, avg=189076.12, stdev=241129.58
- clat (nsec): min=0, max=1469.3M, avg=63533638.70, stdev=133759133.98
- lat (nsec): min=0, max=1469.4M, avg=63723896.57, stdev=133715671.18
- clat percentiles (usec):
- | 1.00th=[ 988], 5.00th=[ 1172], 10.00th=[ 1336],
- | 20.00th=[ 1647], 30.00th=[ 2147], 40.00th=[ 3130],
- | 50.00th=[ 5932], 60.00th=[ 16712], 70.00th=[ 44303],
- | 80.00th=[ 96994], 90.00th=[ 162530], 95.00th=[ 329253],
- | 99.00th=[ 700449], 99.50th=[ 801113], 99.90th=[ 952108],
- | 99.95th=[1098908], 99.99th=[1468007]
- bw ( KiB/s): min= 0, max= 5653, per=68.58%, avg=3627.00, stdev=2865.20, samples=2
- iops : min= 0, max= 88, avg=56.50, stdev=44.55, samples=2
- lat (nsec) : 750=8.09%, 1000=12.14%
- lat (usec) : 2=13.44%, 4=0.07%, 10=0.36%, 20=1.26%, 50=0.78%
- lat (usec) : 100=1.02%, 250=1.04%, 500=0.65%, 750=0.75%, 1000=1.96%
- lat (msec) : 2=15.50%, 4=10.12%, 10=5.89%, 20=4.30%, 50=6.07%
- lat (msec) : 100=5.89%, 250=7.10%, 500=2.34%, 750=0.87%, 1000=0.36%
- cpu : usr=0.07%, sys=0.32%, ctx=14974, majf=0, minf=34
- IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=6669,6802,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- Run status group 0 (all jobs):
- READ: bw=16.5MiB/s (17.3MB/s), 3556KiB/s-13.1MiB/s (3641kB/s-13.7MB/s), io=1996MiB (2093MB), run=120042-120646msec
- WRITE: bw=5289KiB/s (5416kB/s), 1692KiB/s-3628KiB/s (1733kB/s-3715kB/s), io=625MiB (656MB), run=120042-121075msec
- Disk stats (read/write):
- rbd146: ios=7112/10366, merge=0/2, ticks=570356/925505, in_queue=1487123, util=5.21%
- <example-benchmark-fio-server-663-benchmark-7t28d>
- randwrite: (groupid=0, jobs=1): err= 0: pid=11: Wed May 8 20:35:01 2019
- write: IOPS=36, BW=2338KiB/s (2394kB/s)(274MiB/120024msec)
- slat (nsec): min=0, max=1108.8M, avg=1563338.65, stdev=34895725.20
- clat (nsec): min=0, max=2157.2M, avg=108691853.62, stdev=172953769.21
- lat (nsec): min=0, max=2157.2M, avg=110458971.82, stdev=181332612.36
- clat percentiles (usec):
- | 1.00th=[ 1012], 5.00th=[ 1254], 10.00th=[ 1483],
- | 20.00th=[ 2040], 30.00th=[ 3425], 40.00th=[ 9634],
- | 50.00th=[ 51119], 60.00th=[ 84411], 70.00th=[ 108528],
- | 80.00th=[ 143655], 90.00th=[ 346031], 95.00th=[ 488637],
- | 99.00th=[ 834667], 99.50th=[ 910164], 99.90th=[1115685],
- | 99.95th=[1182794], 99.99th=[2164261]
- bw ( KiB/s): min= 0, max= 3665, per=35.50%, avg=2337.00, stdev=1878.08, samples=2
- iops : min= 0, max= 57, avg=36.00, stdev=29.70, samples=2
- lat (usec) : 750=0.07%, 1000=0.80%
- lat (msec) : 2=18.44%, 4=12.71%, 10=8.33%, 20=4.41%, 50=5.02%
- lat (msec) : 100=16.80%, 250=20.82%, 500=7.90%, 750=3.33%, 1000=1.21%
- cpu : usr=0.05%, sys=0.15%, ctx=8296, majf=0, minf=26
- IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=0,4381,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- read: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:35:01 2019
- read: IOPS=505, BW=31.4MiB/s (32.9MB/s)(3763MiB/120006msec)
- slat (nsec): min=0, max=1108.8M, avg=95001.81, stdev=8079368.15
- clat (nsec): min=0, max=2157.1M, avg=7861988.18, stdev=51822844.43
- lat (nsec): min=0, max=2157.2M, avg=7957796.46, stdev=52813178.22
- clat percentiles (nsec):
- | 1.00th=[ 708], 5.00th=[ 716], 10.00th=[ 716],
- | 20.00th=[ 724], 30.00th=[ 732], 40.00th=[ 756],
- | 50.00th=[ 836], 60.00th=[ 892], 70.00th=[ 1112],
- | 80.00th=[ 8768], 90.00th=[ 45312], 95.00th=[ 26869760],
- | 99.00th=[ 261095424], 99.50th=[ 421527552], 99.90th=[ 658505728],
- | 99.95th=[ 767557632], 99.99th=[1098907648]
- bw ( KiB/s): min= 0, max=46009, per=88.48%, avg=32107.50, stdev=19659.69, samples=2
- iops : min= 0, max= 727, avg=505.50, stdev=313.25, samples=2
- lat (nsec) : 750=38.28%, 1000=29.30%
- lat (usec) : 2=7.41%, 4=0.05%, 10=6.17%, 20=4.86%, 50=4.02%
- lat (usec) : 100=0.24%, 250=0.07%, 500=0.08%, 750=0.42%, 1000=1.00%
- lat (msec) : 2=1.37%, 4=0.76%, 10=0.40%, 20=0.29%, 50=1.86%
- lat (msec) : 100=1.61%, 250=0.81%, 500=0.77%, 750=0.20%, 1000=0.05%
- cpu : usr=0.14%, sys=0.48%, ctx=11673, majf=0, minf=105
- IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=60714,0,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- readwrite: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:35:01 2019
- read: IOPS=65, BW=4187KiB/s (4288kB/s)(491MiB/120038msec)
- slat (nsec): min=0, max=358717k, avg=71754.97, stdev=4047648.69
- clat (nsec): min=0, max=959407k, avg=3762438.41, stdev=26003167.77
- lat (nsec): min=0, max=959440k, avg=3835087.41, stdev=26316511.26
- clat percentiles (nsec):
- | 1.00th=[ 708], 5.00th=[ 716], 10.00th=[ 724],
- | 20.00th=[ 748], 30.00th=[ 844], 40.00th=[ 932],
- | 50.00th=[ 1080], 60.00th=[ 1256], 70.00th=[ 13504],
- | 80.00th=[ 444416], 90.00th=[ 1957888], 95.00th=[ 8978432],
- | 99.00th=[ 79167488], 99.50th=[152043520], 99.90th=[442499072],
- | 99.95th=[497025024], 99.99th=[960495616]
- bw ( KiB/s): min= 0, max= 7293, per=11.54%, avg=4188.00, stdev=4391.13, samples=2
- iops : min= 0, max= 113, avg=64.50, stdev=68.59, samples=2
- write: IOPS=66, BW=4246KiB/s (4347kB/s)(498MiB/120038msec)
- slat (nsec): min=0, max=616615k, avg=277859.02, stdev=7123826.65
- clat (nsec): min=0, max=1058.7M, avg=56496877.05, stdev=127492223.13
- lat (nsec): min=0, max=1058.8M, avg=56775886.69, stdev=127619809.89
- clat percentiles (usec):
- | 1.00th=[ 996], 5.00th=[ 1188], 10.00th=[ 1336],
- | 20.00th=[ 1614], 30.00th=[ 1958], 40.00th=[ 2540],
- | 50.00th=[ 3884], 60.00th=[ 7111], 70.00th=[ 21627],
- | 80.00th=[ 85459], 90.00th=[ 143655], 95.00th=[ 350225],
- | 99.00th=[ 641729], 99.50th=[ 775947], 99.90th=[ 943719],
- | 99.95th=[ 943719], 99.99th=[1061159]
- bw ( KiB/s): min= 0, max= 7357, per=64.48%, avg=4244.50, stdev=4401.74, samples=2
- iops : min= 0, max= 114, avg=65.50, stdev=68.59, samples=2
- lat (nsec) : 750=10.03%, 1000=12.38%
- lat (usec) : 2=11.41%, 4=0.06%, 10=0.43%, 20=1.47%, 50=0.82%
- lat (usec) : 100=0.96%, 250=1.24%, 500=1.10%, 750=0.87%, 1000=2.21%
- lat (msec) : 2=17.35%, 4=11.41%, 10=7.51%, 20=3.66%, 50=3.38%
- lat (msec) : 100=4.86%, 250=5.46%, 500=2.27%, 750=0.89%, 1000=0.26%
- cpu : usr=0.08%, sys=0.35%, ctx=17449, majf=0, minf=33
- IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=7855,7959,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- Run status group 0 (all jobs):
- READ: bw=35.4MiB/s (37.2MB/s), 4187KiB/s-31.4MiB/s (4288kB/s-32.9MB/s), io=4254MiB (4460MB), run=120006-120038msec
- WRITE: bw=6584KiB/s (6742kB/s), 2338KiB/s-4246KiB/s (2394kB/s-4347kB/s), io=772MiB (809MB), run=120024-120038msec
- Disk stats (read/write):
- rbd284: ios=7426/12487, merge=0/2, ticks=505851/806981, in_queue=1302709, util=12.68%
- <example-benchmark-fio-server-37-benchmark-k5hwt>
- randwrite: (groupid=0, jobs=1): err= 0: pid=11: Wed May 8 20:35:01 2019
- write: IOPS=39, BW=2540KiB/s (2601kB/s)(298MiB/120173msec)
- slat (nsec): min=0, max=696899k, avg=411800.47, stdev=10858781.84
- clat (nsec): min=0, max=1250.4M, avg=100680898.92, stdev=165640343.18
- lat (nsec): min=0, max=1250.6M, avg=101093747.11, stdev=165819419.43
- clat percentiles (usec):
- | 1.00th=[ 971], 5.00th=[ 1205], 10.00th=[ 1418],
- | 20.00th=[ 1909], 30.00th=[ 2868], 40.00th=[ 5735],
- | 50.00th=[ 25035], 60.00th=[ 71828], 70.00th=[ 103285],
- | 80.00th=[ 139461], 90.00th=[ 308282], 95.00th=[ 480248],
- | 99.00th=[ 792724], 99.50th=[ 901776], 99.90th=[1115685],
- | 99.95th=[1115685], 99.99th=[1249903]
- bw ( KiB/s): min= 0, max= 2981, per=39.78%, avg=2541.00, stdev=622.25, samples=2
- iops : min= 0, max= 46, avg=39.00, stdev= 9.90, samples=2
- lat (usec) : 250=0.02%, 500=0.04%, 750=0.04%, 1000=1.26%
- lat (msec) : 2=19.87%, 4=14.23%, 10=8.64%, 20=4.51%, 50=6.53%
- lat (msec) : 100=13.91%, 250=18.74%, 500=7.74%, 750=3.34%, 1000=0.97%
- cpu : usr=0.06%, sys=0.17%, ctx=9069, majf=0, minf=27
- IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=0,4766,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- read: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:35:01 2019
- read: IOPS=381, BW=23.8MiB/s (24.9MB/s)(2856MiB/120067msec)
- slat (nsec): min=0, max=408609k, avg=33102.72, stdev=2270301.13
- clat (nsec): min=0, max=1243.1M, avg=10502054.70, stdev=58942956.54
- lat (nsec): min=0, max=1243.2M, avg=10535980.10, stdev=58995317.19
- clat percentiles (nsec):
- | 1.00th=[ 708], 5.00th=[ 708], 10.00th=[ 716],
- | 20.00th=[ 724], 30.00th=[ 732], 40.00th=[ 756],
- | 50.00th=[ 820], 60.00th=[ 876], 70.00th=[ 988],
- | 80.00th=[ 8640], 90.00th=[ 1155072], 95.00th=[ 49020928],
- | 99.00th=[325058560], 99.50th=[467664896], 99.90th=[734003200],
- | 99.95th=[843055104], 99.99th=[960495616]
- bw ( KiB/s): min= 0, max=37141, per=86.65%, avg=24365.50, stdev=18067.29, samples=2
- iops : min= 0, max= 581, avg=381.00, stdev=282.84, samples=2
- lat (nsec) : 750=38.95%, 1000=31.59%
- lat (usec) : 2=9.13%, 4=0.05%, 10=3.93%, 20=2.24%, 50=1.34%
- lat (usec) : 100=0.13%, 250=0.06%, 500=0.12%, 750=0.56%, 1000=1.34%
- lat (msec) : 2=1.89%, 4=1.03%, 10=0.54%, 20=0.37%, 50=1.82%
- lat (msec) : 100=2.31%, 250=1.27%, 500=0.96%, 750=0.29%, 1000=0.08%
- cpu : usr=0.11%, sys=0.38%, ctx=11528, majf=0, minf=31
- IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=45763,0,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- readwrite: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:35:01 2019
- read: IOPS=58, BW=3769KiB/s (3859kB/s)(442MiB/120004msec)
- slat (nsec): min=0, max=156853k, avg=69439.21, stdev=2318254.95
- clat (nsec): min=0, max=912880k, avg=6950330.88, stdev=42236370.38
- lat (nsec): min=0, max=913024k, avg=7020687.52, stdev=42300601.87
- clat percentiles (nsec):
- | 1.00th=[ 716], 5.00th=[ 724], 10.00th=[ 740],
- | 20.00th=[ 804], 30.00th=[ 876], 40.00th=[ 988],
- | 50.00th=[ 1144], 60.00th=[ 1288], 70.00th=[ 14144],
- | 80.00th=[ 716800], 90.00th=[ 2867200], 95.00th=[ 23986176],
- | 99.00th=[217055232], 99.50th=[325058560], 99.90th=[549453824],
- | 99.95th=[641728512], 99.99th=[910163968]
- bw ( KiB/s): min= 0, max= 5579, per=13.40%, avg=3768.00, stdev=2561.14, samples=2
- iops : min= 0, max= 87, avg=58.50, stdev=40.31, samples=2
- write: IOPS=60, BW=3854KiB/s (3946kB/s)(452MiB/120004msec)
- slat (nsec): min=0, max=17335k, avg=183883.83, stdev=342978.00
- clat (nsec): min=0, max=1171.4M, avg=59432543.96, stdev=120575360.93
- lat (nsec): min=0, max=1171.5M, avg=59617587.11, stdev=120535896.17
- clat percentiles (usec):
- | 1.00th=[ 1020], 5.00th=[ 1221], 10.00th=[ 1418],
- | 20.00th=[ 1762], 30.00th=[ 2343], 40.00th=[ 3490],
- | 50.00th=[ 6718], 60.00th=[ 16057], 70.00th=[ 42730],
- | 80.00th=[ 90702], 90.00th=[ 154141], 95.00th=[ 320865],
- | 99.00th=[ 608175], 99.50th=[ 700449], 99.90th=[ 910164],
- | 99.95th=[ 943719], 99.99th=[1166017]
- bw ( KiB/s): min= 0, max= 5643, per=60.29%, avg=3851.50, stdev=2533.56, samples=2
- iops : min= 0, max= 88, avg=60.00, stdev=39.60, samples=2
- lat (nsec) : 750=6.79%, 1000=13.47%
- lat (usec) : 2=13.65%, 4=0.09%, 10=0.23%, 20=1.02%, 50=0.93%
- lat (usec) : 100=0.99%, 250=1.08%, 500=0.86%, 750=0.66%, 1000=2.05%
- lat (msec) : 2=14.38%, 4=10.35%, 10=7.07%, 20=4.86%, 50=5.50%
- lat (msec) : 100=6.33%, 250=5.87%, 500=2.76%, 750=0.91%, 1000=0.14%
- cpu : usr=0.08%, sys=0.34%, ctx=16157, majf=0, minf=31
- IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=7067,7224,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- Run status group 0 (all jobs):
- READ: bw=27.5MiB/s (28.8MB/s), 3769KiB/s-23.8MiB/s (3859kB/s-24.9MB/s), io=3297MiB (3457MB), run=120004-120067msec
- WRITE: bw=6388KiB/s (6541kB/s), 2540KiB/s-3854KiB/s (2601kB/s-3946kB/s), io=750MiB (786MB), run=120004-120173msec
- Disk stats (read/write):
- rbd12: ios=7283/12276, merge=0/1, ticks=539258/835111, in_queue=1364388, util=12.96%
- Jobs: 7 (f=7): [_(2),m(1)][98.8%][r=455M,w=443M][r=7105,w=6923 IOPS][eta 00m:49s] Jobs: 7 (f=7): [_(2),m(1)][98.7%][r=583M,w=562M][r=9104,w=8780 IOPS][eta 00m:48s] <example-benchmark-fio-server-435-benchmark-5cvqg>
- randwrite: (groupid=0, jobs=1): err= 0: pid=11: Wed May 8 20:35:02 2019
- write: IOPS=22, BW=1412KiB/s (1446kB/s)(166MiB/120304msec)
- slat (nsec): min=0, max=777619k, avg=1045993.83, stdev=24696843.15
- clat (nsec): min=0, max=3272.3M, avg=181650449.15, stdev=336408176.45
- lat (nsec): min=0, max=3273.1M, avg=182696909.40, stdev=336849572.89
- clat percentiles (usec):
- | 1.00th=[ 1139], 5.00th=[ 1467], 10.00th=[ 1811],
- | 20.00th=[ 2540], 30.00th=[ 3720], 40.00th=[ 5997],
- | 50.00th=[ 14746], 60.00th=[ 39584], 70.00th=[ 92799],
- | 80.00th=[ 425722], 90.00th=[ 675283], 95.00th=[ 884999],
- | 99.00th=[1367344], 99.50th=[1669333], 99.90th=[2332034],
- | 99.95th=[2902459], 99.99th=[3271558]
- bw ( KiB/s): min= 0, max= 1468, per=23.39%, avg=1412.00, stdev=79.20, samples=2
- iops : min= 0, max= 22, avg=21.50, stdev= 0.71, samples=2
- lat (usec) : 4=0.04%, 50=0.04%, 100=0.04%, 500=0.04%, 1000=0.23%
- lat (msec) : 2=12.79%, 4=18.75%, 10=15.01%, 20=6.79%, 50=8.45%
- lat (msec) : 100=9.02%, 250=6.56%, 500=5.62%, 750=8.26%, 1000=5.62%
- cpu : usr=0.02%, sys=0.11%, ctx=4893, majf=0, minf=27
- IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=0,2651,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- read: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:35:02 2019
- read: IOPS=540, BW=33.6MiB/s (35.3MB/s)(4039MiB/120089msec)
- slat (nsec): min=0, max=682499k, avg=37730.14, stdev=3821435.40
- clat (nsec): min=0, max=3266.9M, avg=7397397.95, stdev=73619786.81
- lat (nsec): min=0, max=3266.9M, avg=7436102.24, stdev=73726006.00
- clat percentiles (nsec):
- | 1.00th=[ 708], 5.00th=[ 716], 10.00th=[ 724],
- | 20.00th=[ 748], 30.00th=[ 780], 40.00th=[ 804],
- | 50.00th=[ 844], 60.00th=[ 868], 70.00th=[ 900],
- | 80.00th=[ 1144], 90.00th=[ 12608], 95.00th=[ 24448],
- | 99.00th=[ 350224384], 99.50th=[ 591396864], 99.90th=[ 977272832],
- | 99.95th=[1249902592], 99.99th=[1837105152]
- bw ( KiB/s): min= 0, max=54965, per=88.02%, avg=34356.50, stdev=29144.82, samples=2
- iops : min= 0, max= 863, avg=538.50, stdev=458.91, samples=2
- lat (nsec) : 750=20.50%, 1000=57.38%
- lat (usec) : 2=5.04%, 4=0.15%, 10=0.99%, 20=9.96%, 50=1.54%
- lat (usec) : 100=0.15%, 250=0.11%, 500=0.09%, 750=0.15%, 1000=0.35%
- lat (msec) : 2=0.72%, 4=0.49%, 10=0.29%, 20=0.17%, 50=0.41%
- lat (msec) : 100=0.38%, 250=0.09%, 500=0.39%, 750=0.37%, 1000=0.19%
- cpu : usr=0.13%, sys=0.52%, ctx=5705, majf=0, minf=34
- IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=64882,0,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- readwrite: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:35:02 2019
- read: IOPS=71, BW=4595KiB/s (4705kB/s)(538MiB/120003msec)
- slat (nsec): min=0, max=3033.9k, avg=26259.86, stdev=80062.73
- clat (nsec): min=0, max=687222k, avg=4836633.32, stdev=32108832.60
- lat (nsec): min=0, max=687261k, avg=4863797.12, stdev=32118036.94
- clat percentiles (nsec):
- | 1.00th=[ 716], 5.00th=[ 724], 10.00th=[ 740],
- | 20.00th=[ 764], 30.00th=[ 860], 40.00th=[ 972],
- | 50.00th=[ 1160], 60.00th=[ 1352], 70.00th=[ 19072],
- | 80.00th=[ 460800], 90.00th=[ 1597440], 95.00th=[ 7962624],
- | 99.00th=[130547712], 99.50th=[238026752], 99.90th=[434110464],
- | 99.95th=[526385152], 99.99th=[683671552]
- bw ( KiB/s): min= 0, max= 8049, per=11.77%, avg=4593.50, stdev=4886.81, samples=2
- iops : min= 0, max= 125, avg=71.00, stdev=76.37, samples=2
- write: IOPS=72, BW=4638KiB/s (4750kB/s)(544MiB/120003msec)
- slat (nsec): min=0, max=9475.9k, avg=183809.96, stdev=263225.44
- clat (nsec): min=0, max=998063k, avg=50489057.96, stdev=117048370.19
- lat (nsec): min=0, max=998145k, avg=50674028.51, stdev=117014939.57
- clat percentiles (usec):
- | 1.00th=[ 963], 5.00th=[ 1139], 10.00th=[ 1287], 20.00th=[ 1532],
- | 30.00th=[ 1876], 40.00th=[ 2507], 50.00th=[ 3720], 60.00th=[ 7439],
- | 70.00th=[ 21627], 80.00th=[ 71828], 90.00th=[121111], 95.00th=[312476],
- | 99.00th=[574620], 99.50th=[683672], 99.90th=[926942], 99.95th=[935330],
- | 99.99th=[994051]
- bw ( KiB/s): min= 0, max= 8099, per=76.81%, avg=4637.50, stdev=4895.30, samples=2
- iops : min= 0, max= 126, avg=72.00, stdev=76.37, samples=2
- lat (nsec) : 750=7.98%, 1000=12.81%
- lat (usec) : 2=12.09%, 4=0.05%, 10=0.40%, 20=1.62%, 50=1.02%
- lat (usec) : 100=1.18%, 250=1.75%, 500=1.09%, 750=0.99%, 1000=2.65%
- lat (msec) : 2=18.00%, 4=10.84%, 10=6.70%, 20=3.30%, 50=4.31%
- lat (msec) : 100=5.60%, 250=4.41%, 500=2.16%, 750=0.86%, 1000=0.18%
- cpu : usr=0.09%, sys=0.40%, ctx=18391, majf=0, minf=37
- IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=8615,8694,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- Run status group 0 (all jobs):
- READ: bw=38.1MiB/s (39.0MB/s), 4595KiB/s-33.6MiB/s (4705kB/s-35.3MB/s), io=4578MiB (4800MB), run=120003-120089msec
- WRITE: bw=6039KiB/s (6183kB/s), 1412KiB/s-4638KiB/s (1446kB/s-4750kB/s), io=709MiB (744MB), run=120003-120304msec
- Disk stats (read/write):
- rbd110: ios=4843/11517, merge=0/1, ticks=536620/888979, in_queue=1417352, util=7.88%
- Jobs: 6 (f=6): [_(2),m(1)][98.6%][r=487M,w=494M][r=7605,w=7724 IOPS][eta 00m:47s] Jobs: 6 (f=6): [_(2),m(1)][98.6%][r=439M,w=449M][r=6851,w=7007 IOPS][eta 00m:45s] Jobs: 6 (f=6): [_(2),m(1)][98.6%][r=483M,w=472M][r=7539,w=7372 IOPS][eta 00m:45s] Jobs: 6 (f=6): [_(2),m(1)][98.7%][r=423M,w=420M][r=6605,w=6560 IOPS][eta 00m:44s] Jobs: 6 (f=6): [_(2),m(1)][98.7%][r=454M,w=452M][r=7088,w=7060 IOPS][eta 00m:42s] Jobs: 6 (f=6): [_(2),m(1)][98.8%][r=175M,w=169M][r=2727,w=2633 IOPS][eta 00m:41s] Jobs: 6 (f=6): [_(2),m(1)][98.8%][r=452M,w=453M][r=7063,w=7085 IOPS][eta 00m:41s] Jobs: 6 (f=6): [_(2),m(1)][98.8%][r=443M,w=438M][r=6916,w=6842 IOPS][eta 00m:40s] <example-benchmark-fio-server-503-benchmark-6fx6x>
- randwrite: (groupid=0, jobs=1): err= 0: pid=11: Wed May 8 20:35:10 2019
- write: IOPS=57, BW=3665KiB/s (3753kB/s)(430MiB/120026msec)
- slat (nsec): min=0, max=622630k, avg=345543.41, stdev=9113718.13
- clat (nsec): min=0, max=1384.4M, avg=69695686.91, stdev=136504038.48
- lat (nsec): min=0, max=1384.4M, avg=70042389.89, stdev=136680584.74
- clat percentiles (usec):
- | 1.00th=[ 938], 5.00th=[ 1123], 10.00th=[ 1287],
- | 20.00th=[ 1565], 30.00th=[ 1958], 40.00th=[ 2638],
- | 50.00th=[ 4293], 60.00th=[ 16319], 70.00th=[ 63701],
- | 80.00th=[ 101188], 90.00th=[ 208667], 95.00th=[ 371196],
- | 99.00th=[ 675283], 99.50th=[ 759170], 99.90th=[ 918553],
- | 99.95th=[ 935330], 99.99th=[1384121]
- bw ( KiB/s): min= 0, max= 5002, per=29.12%, avg=3663.00, stdev=1893.63, samples=2
- iops : min= 0, max= 78, avg=57.00, stdev=29.70, samples=2
- lat (usec) : 4=0.01%, 500=0.01%, 750=0.09%, 1000=1.89%
- lat (msec) : 2=28.85%, 4=18.14%, 10=8.84%, 20=2.87%, 50=5.82%
- lat (msec) : 100=13.11%, 250=11.95%, 500=5.62%, 750=2.21%, 1000=0.61%
- cpu : usr=0.07%, sys=0.24%, ctx=12882, majf=0, minf=27
- IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=0,6870,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- read: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:35:10 2019
- read: IOPS=363, BW=22.7MiB/s (23.8MB/s)(2725MiB/120010msec)
- slat (nsec): min=0, max=597722k, avg=43845.37, stdev=3451713.75
- clat (nsec): min=0, max=952019k, avg=10981275.55, stdev=56162989.38
- lat (nsec): min=0, max=952050k, avg=11025965.24, stdev=56275129.48
- clat percentiles (nsec):
- | 1.00th=[ 700], 5.00th=[ 708], 10.00th=[ 716],
- | 20.00th=[ 724], 30.00th=[ 732], 40.00th=[ 772],
- | 50.00th=[ 836], 60.00th=[ 884], 70.00th=[ 1048],
- | 80.00th=[ 8768], 90.00th=[ 3489792], 95.00th=[ 52166656],
- | 99.00th=[312475648], 99.50th=[446693376], 99.90th=[683671552],
- | 99.95th=[759169024], 99.99th=[843055104]
- bw ( KiB/s): min= 0, max=33364, per=72.38%, avg=23248.00, stdev=14306.18, samples=2
- iops : min= 0, max= 522, avg=363.50, stdev=224.15, samples=2
- lat (nsec) : 750=35.04%, 1000=33.44%
- lat (usec) : 2=11.20%, 4=0.06%, 10=0.69%, 20=0.35%, 50=0.13%
- lat (usec) : 100=0.09%, 250=0.03%, 500=0.41%, 750=1.28%, 1000=2.38%
- lat (msec) : 2=3.27%, 4=1.88%, 10=0.96%, 20=0.58%, 50=2.99%
- lat (msec) : 100=2.50%, 250=1.42%, 500=0.98%, 750=0.28%, 1000=0.05%
- cpu : usr=0.12%, sys=0.40%, ctx=16531, majf=0, minf=37
- IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=43643,0,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- readwrite: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:35:10 2019
- read: IOPS=138, BW=8872KiB/s (9085kB/s)(1040MiB/120001msec)
- slat (nsec): min=0, max=423034k, avg=64612.42, stdev=3280735.86
- clat (nsec): min=0, max=1022.7M, avg=3889780.33, stdev=26493181.11
- lat (nsec): min=0, max=1022.8M, avg=3955401.07, stdev=26697984.84
- clat percentiles (nsec):
- | 1.00th=[ 716], 5.00th=[ 740], 10.00th=[ 772],
- | 20.00th=[ 900], 30.00th=[ 1128], 40.00th=[ 1544],
- | 50.00th=[ 432128], 60.00th=[ 774144], 70.00th=[ 921600],
- | 80.00th=[ 1269760], 90.00th=[ 2637824], 95.00th=[ 5603328],
- | 99.00th=[ 79167488], 99.50th=[ 183500800], 99.90th=[ 379584512],
- | 99.95th=[ 413138944], 99.99th=[1019215872]
- bw ( KiB/s): min= 0, max= 2046, per=6.37%, avg=2046.00, stdev= 0.00, samples=1
- iops : min= 0, max= 32, avg=32.00, stdev= 0.00, samples=1
- write: IOPS=139, BW=8917KiB/s (9131kB/s)(1045MiB/120001msec)
- slat (nsec): min=0, max=360725k, avg=196120.52, stdev=2797797.23
- clat (nsec): min=0, max=951017k, avg=24667440.35, stdev=78295914.29
- lat (nsec): min=0, max=951106k, avg=24864692.52, stdev=78319107.19
- clat percentiles (usec):
- | 1.00th=[ 914], 5.00th=[ 1057], 10.00th=[ 1139], 20.00th=[ 1270],
- | 30.00th=[ 1401], 40.00th=[ 1565], 50.00th=[ 1860], 60.00th=[ 2507],
- | 70.00th=[ 3916], 80.00th=[ 8455], 90.00th=[ 63701], 95.00th=[128451],
- | 99.00th=[459277], 99.50th=[534774], 99.90th=[717226], 99.95th=[784335],
- | 99.99th=[943719]
- bw ( KiB/s): min= 0, max= 2136, per=16.98%, avg=2136.00, stdev= 0.00, samples=1
- iops : min= 0, max= 33, avg=33.00, stdev= 0.00, samples=1
- lat (nsec) : 750=3.87%, 1000=8.51%
- lat (usec) : 2=7.98%, 4=0.03%, 10=0.07%, 20=0.29%, 50=0.43%
- lat (usec) : 100=0.41%, 250=0.92%, 500=2.95%, 750=3.83%, 1000=8.77%
- lat (msec) : 2=31.61%, 4=12.24%, 10=6.91%, 20=2.28%, 50=2.35%
- lat (msec) : 100=2.80%, 250=2.19%, 500=1.23%, 750=0.30%, 1000=0.04%
- cpu : usr=0.15%, sys=0.66%, ctx=47021, majf=0, minf=32
- IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=16637,16716,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- Run status group 0 (all jobs):
- READ: bw=31.4MiB/s (32.9MB/s), 8872KiB/s-22.7MiB/s (9085kB/s-23.8MB/s), io=3764MiB (3947MB), run=120001-120010msec
- WRITE: bw=12.3MiB/s (12.9MB/s), 3665KiB/s-8917KiB/s (3753kB/s-9131kB/s), io=1475MiB (1546MB), run=120001-120026msec
- Disk stats (read/write):
- rbd208: ios=19607/24303, merge=0/1, ticks=580676/840563, in_queue=1399084, util=21.89%
- Jobs: 5 (f=5): [_(2),m(1)][98.6%][r=401M,w=400M][r=6270,w=6243 IOPS][eta 00m:39s] Jobs: 5 (f=5): [_(2),m(1)][98.7%][r=402M,w=388M][r=6277,w=6068 IOPS][eta 00m:38s] Jobs: 5 (f=5): [_(2),m(1)][98.7%][r=341M,w=330M][r=5323,w=5149 IOPS][eta 00m:37s] Jobs: 5 (f=5): [_(2),m(1)][98.7%][r=372M,w=365M][r=5806,w=5706 IOPS][eta 00m:36s] Jobs: 5 (f=5): [_(2),m(1)][98.8%][r=390M,w=381M][r=6092,w=5949 IOPS][eta 00m:34s] <example-benchmark-fio-server-668-benchmark-bhqmd>
- randwrite: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:35:15 2019
- write: IOPS=15, BW=1003KiB/s (1027kB/s)(119MiB/121774msec)
- slat (nsec): min=0, max=800208k, avg=1315104.52, stdev=25820013.84
- clat (nsec): min=0, max=1877.4M, avg=252516056.14, stdev=421021517.60
- lat (nsec): min=0, max=1877.8M, avg=253830909.29, stdev=421422000.84
- clat percentiles (usec):
- | 1.00th=[ 1020], 5.00th=[ 1467], 10.00th=[ 1860],
- | 20.00th=[ 2573], 30.00th=[ 3654], 40.00th=[ 5276],
- | 50.00th=[ 9110], 60.00th=[ 26084], 70.00th=[ 242222],
- | 80.00th=[ 574620], 90.00th=[ 977273], 95.00th=[1233126],
- | 99.00th=[1484784], 99.50th=[1635779], 99.90th=[1870660],
- | 99.95th=[1870660], 99.99th=[1870660]
- bw ( KiB/s): min= 0, max= 1290, per=7.19%, avg=1011.50, stdev=393.86, samples=2
- iops : min= 0, max= 20, avg=15.50, stdev= 6.36, samples=2
- lat (usec) : 4=0.05%, 500=0.05%, 1000=0.73%
- lat (msec) : 2=10.97%, 4=20.89%, 10=18.43%, 20=6.72%, 50=6.98%
- lat (msec) : 100=3.78%, 250=1.63%, 500=8.61%, 750=6.09%, 1000=5.83%
- cpu : usr=0.01%, sys=0.08%, ctx=3340, majf=0, minf=29
- IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=0,1905,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- read: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:35:15 2019
- read: IOPS=267, BW=16.7MiB/s (17.5MB/s)(2005MiB/120115msec)
- slat (nsec): min=0, max=680756k, avg=56808.85, stdev=4611819.23
- clat (nsec): min=0, max=1845.9M, avg=14932473.14, stdev=106837887.10
- lat (nsec): min=0, max=1845.0M, avg=14990289.36, stdev=106948423.50
- clat percentiles (nsec):
- | 1.00th=[ 708], 5.00th=[ 716], 10.00th=[ 724],
- | 20.00th=[ 764], 30.00th=[ 780], 40.00th=[ 796],
- | 50.00th=[ 836], 60.00th=[ 868], 70.00th=[ 900],
- | 80.00th=[ 1004], 90.00th=[ 12480], 95.00th=[ 2768896],
- | 99.00th=[ 692060160], 99.50th=[ 851443712], 99.90th=[1182793728],
- | 99.95th=[1283457024], 99.99th=[1619001344]
- bw ( KiB/s): min= 0, max=20554, per=67.59%, avg=20554.00, stdev= 0.00, samples=1
- iops : min= 0, max= 321, avg=321.00, stdev= 0.00, samples=1
- lat (nsec) : 750=18.62%, 1000=61.37%
- lat (usec) : 2=9.01%, 4=0.20%, 10=0.18%, 20=1.19%, 50=0.23%
- lat (usec) : 100=0.11%, 250=0.09%, 500=0.09%, 750=0.24%, 1000=0.82%
- lat (msec) : 2=2.05%, 4=1.55%, 10=1.02%, 20=0.52%, 50=0.48%
- lat (msec) : 100=0.19%, 250=0.12%, 500=0.23%, 750=0.97%, 1000=0.50%
- cpu : usr=0.07%, sys=0.29%, ctx=5937, majf=0, minf=30
- IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=32116,0,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- readwrite: (groupid=0, jobs=1): err= 0: pid=14: Wed May 8 20:35:15 2019
- read: IOPS=208, BW=13.0MiB/s (13.7MB/s)(1562MiB/120002msec)
- slat (nsec): min=0, max=3572.6k, avg=32152.11, stdev=62686.37
- clat (nsec): min=0, max=610833k, avg=2659805.67, stdev=24685079.65
- lat (nsec): min=0, max=610861k, avg=2693027.06, stdev=24687759.61
- clat percentiles (nsec):
- | 1.00th=[ 716], 5.00th=[ 724], 10.00th=[ 740],
- | 20.00th=[ 844], 30.00th=[ 1032], 40.00th=[ 1368],
- | 50.00th=[ 268288], 60.00th=[ 700416], 70.00th=[ 831488],
- | 80.00th=[ 1056768], 90.00th=[ 1941504], 95.00th=[ 3325952],
- | 99.00th=[ 41680896], 99.50th=[101187584], 99.90th=[467664896],
- | 99.95th=[509607936], 99.99th=[599785472]
- bw ( KiB/s): min= 0, max=23191, per=43.83%, avg=13330.00, stdev=13945.56, samples=2
- iops : min= 0, max= 362, avg=208.00, stdev=217.79, samples=2
- write: IOPS=207, BW=12.9MiB/s (13.6MB/s)(1554MiB/120002msec)
- slat (nsec): min=0, max=12405k, avg=162543.64, stdev=196155.68
- clat (nsec): min=0, max=677169k, avg=16443104.90, stdev=52031115.06
- lat (nsec): min=0, max=677230k, avg=16606816.60, stdev=52011422.69
- clat percentiles (usec):
- | 1.00th=[ 881], 5.00th=[ 1029], 10.00th=[ 1106], 20.00th=[ 1221],
- | 30.00th=[ 1336], 40.00th=[ 1483], 50.00th=[ 1696], 60.00th=[ 2147],
- | 70.00th=[ 3294], 80.00th=[ 5997], 90.00th=[ 34866], 95.00th=[103285],
- | 99.00th=[263193], 99.50th=[383779], 99.90th=[549454], 99.95th=[591397],
- | 99.99th=[650118]
- bw ( KiB/s): min= 0, max=22936, per=94.24%, avg=13255.50, stdev=13690.29, samples=2
- iops : min= 0, max= 358, avg=206.50, stdev=214.25, samples=2
- lat (nsec) : 750=5.86%, 1000=8.55%
- lat (usec) : 2=6.79%, 4=0.03%, 10=0.24%, 20=0.76%, 50=0.65%
- lat (usec) : 100=0.61%, 250=1.32%, 500=3.16%, 750=3.98%, 1000=9.39%
- lat (msec) : 2=32.63%, 4=11.09%, 10=6.29%, 20=2.00%, 50=1.86%
- lat (msec) : 100=1.92%, 250=2.17%, 500=0.61%, 750=0.11%
- cpu : usr=0.24%, sys=0.94%, ctx=66524, majf=0, minf=34
- IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=24996,24854,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- Run status group 0 (all jobs):
- READ: bw=29.7MiB/s (31.1MB/s), 13.0MiB/s-16.7MiB/s (13.7MB/s-17.5MB/s), io=3567MiB (3740MB), run=120002-120115msec
- WRITE: bw=13.7MiB/s (14.4MB/s), 1003KiB/s-12.9MiB/s (1027kB/s-13.6MB/s), io=1673MiB (1754MB), run=120002-121774msec
- Disk stats (read/write):
- rbd227: ios=15912/26988, merge=0/1, ticks=587346/884907, in_queue=1450898, util=10.17%
- <example-benchmark-fio-server-700-benchmark-lbhjl>
- randwrite: (groupid=0, jobs=1): err= 0: pid=11: Wed May 8 20:35:15 2019
- write: IOPS=46, BW=2954KiB/s (3025kB/s)(346MiB/120020msec)
- slat (nsec): min=0, max=941199k, avg=450397.43, stdev=14470997.85
- clat (nsec): min=0, max=1098.0M, avg=86745193.42, stdev=158139041.80
- lat (nsec): min=0, max=1098.1M, avg=87196573.48, stdev=158617381.78
- clat percentiles (usec):
- | 1.00th=[ 1004], 5.00th=[ 1205], 10.00th=[ 1418],
- | 20.00th=[ 1844], 30.00th=[ 2376], 40.00th=[ 3720],
- | 50.00th=[ 8848], 60.00th=[ 42206], 70.00th=[ 88605],
- | 80.00th=[ 125305], 90.00th=[ 274727], 95.00th=[ 408945],
- | 99.00th=[ 801113], 99.50th=[ 926942], 99.90th=[1044382],
- | 99.95th=[1098908], 99.99th=[1098908]
- bw ( KiB/s): min= 0, max= 2189, per=14.78%, avg=2189.00, stdev= 0.00, samples=1
- iops : min= 0, max= 34, avg=34.00, stdev= 0.00, samples=1
- lat (usec) : 500=0.02%, 1000=0.94%
- lat (msec) : 2=22.85%, 4=17.54%, 10=9.64%, 20=4.55%, 50=5.82%
- lat (msec) : 100=12.14%, 250=15.66%, 500=7.57%, 750=1.99%, 1000=1.07%
- cpu : usr=0.06%, sys=0.19%, ctx=10237, majf=0, minf=30
- IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=0,5537,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- read: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:35:15 2019
- read: IOPS=435, BW=27.2MiB/s (28.5MB/s)(3266MiB/120022msec)
- slat (nsec): min=0, max=519236k, avg=36005.92, stdev=2710731.95
- clat (nsec): min=0, max=1273.1M, avg=9152643.07, stdev=51653585.32
- lat (nsec): min=0, max=1273.1M, avg=9189505.59, stdev=51731663.29
- clat percentiles (nsec):
- | 1.00th=[ 708], 5.00th=[ 708], 10.00th=[ 716],
- | 20.00th=[ 724], 30.00th=[ 732], 40.00th=[ 740],
- | 50.00th=[ 804], 60.00th=[ 876], 70.00th=[ 980],
- | 80.00th=[ 1480], 90.00th=[ 2441216], 95.00th=[ 44826624],
- | 99.00th=[ 254803968], 99.50th=[ 413138944], 99.90th=[ 683671552],
- | 99.95th=[ 750780416], 99.99th=[1019215872]
- bw ( KiB/s): min= 0, max=42058, per=70.17%, avg=27869.00, stdev=20066.28, samples=2
- iops : min= 0, max= 658, avg=435.50, stdev=314.66, samples=2
- lat (nsec) : 750=43.47%, 1000=27.31%
- lat (usec) : 2=9.90%, 4=0.05%, 10=1.01%, 20=0.67%, 50=0.25%
- lat (usec) : 100=0.11%, 250=0.08%, 500=0.90%, 750=1.13%, 1000=1.88%
- lat (msec) : 2=2.63%, 4=1.72%, 10=0.99%, 20=0.62%, 50=2.79%
- lat (msec) : 100=2.38%, 250=1.09%, 500=0.75%, 750=0.22%, 1000=0.04%
- cpu : usr=0.13%, sys=0.48%, ctx=17884, majf=0, minf=32
- IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=52311,0,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- readwrite: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:35:15 2019
- read: IOPS=185, BW=11.6MiB/s (12.1MB/s)(1389MiB/120002msec)
- slat (nsec): min=0, max=2063.9k, avg=37114.12, stdev=56251.65
- clat (nsec): min=0, max=969207k, avg=2652385.66, stdev=19531361.29
- lat (nsec): min=0, max=969226k, avg=2690596.59, stdev=19533369.42
- clat percentiles (nsec):
- | 1.00th=[ 716], 5.00th=[ 740], 10.00th=[ 772],
- | 20.00th=[ 940], 30.00th=[ 1272], 40.00th=[ 288768],
- | 50.00th=[ 626688], 60.00th=[ 782336], 70.00th=[ 913408],
- | 80.00th=[ 1204224], 90.00th=[ 2342912], 95.00th=[ 4227072],
- | 99.00th=[ 53739520], 99.50th=[ 85458944], 99.90th=[333447168],
- | 99.95th=[425721856], 99.99th=[517996544]
- bw ( KiB/s): min= 0, max=22243, per=29.84%, avg=11850.00, stdev=14697.92, samples=2
- iops : min= 0, max= 347, avg=184.50, stdev=229.81, samples=2
- write: IOPS=185, BW=11.6MiB/s (12.1MB/s)(1390MiB/120002msec)
- slat (nsec): min=0, max=11228k, avg=166988.35, stdev=176151.67
- clat (nsec): min=0, max=1018.4M, avg=18784576.99, stdev=68641704.52
- lat (nsec): min=0, max=1018.5M, avg=18952667.28, stdev=68621574.76
- clat percentiles (usec):
- | 1.00th=[ 889], 5.00th=[ 1020], 10.00th=[ 1106], 20.00th=[ 1205],
- | 30.00th=[ 1303], 40.00th=[ 1434], 50.00th=[ 1598], 60.00th=[ 1909],
- | 70.00th=[ 2704], 80.00th=[ 4752], 90.00th=[ 22414], 95.00th=[106431],
- | 99.00th=[383779], 99.50th=[501220], 99.90th=[700449], 99.95th=[725615],
- | 99.99th=[926942]
- bw ( KiB/s): min= 0, max=22164, per=80.07%, avg=11862.00, stdev=14569.23, samples=2
- iops : min= 0, max= 346, avg=185.00, stdev=227.69, samples=2
- lat (nsec) : 750=3.61%, 1000=7.51%
- lat (usec) : 2=5.58%, 4=0.02%, 10=0.09%, 20=0.35%, 50=0.45%
- lat (usec) : 100=0.39%, 250=1.19%, 500=4.59%, 750=4.92%, 1000=10.38%
- lat (msec) : 2=35.97%, 4=11.08%, 10=5.80%, 20=1.87%, 50=1.52%
- lat (msec) : 100=1.77%, 250=1.74%, 500=0.91%, 750=0.23%, 1000=0.02%
- cpu : usr=0.21%, sys=0.82%, ctx=64428, majf=0, minf=103
- IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=22220,22241,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- Run status group 0 (all jobs):
- READ: bw=38.8MiB/s (40.7MB/s), 11.6MiB/s-27.2MiB/s (12.1MB/s-28.5MB/s), io=4655MiB (4881MB), run=120002-120022msec
- WRITE: bw=14.5MiB/s (15.2MB/s), 2954KiB/s-11.6MiB/s (3025kB/s-12.1MB/s), io=1737MiB (1821MB), run=120002-120020msec
- Disk stats (read/write):
- rbd274: ios=26881/29490, merge=0/1, ticks=576168/834987, in_queue=1382881, util=24.06%
- <example-benchmark-fio-server-292-benchmark-bqpc4>
- randwrite: (groupid=0, jobs=1): err= 0: pid=11: Wed May 8 20:35:16 2019
- write: IOPS=31, BW=2033KiB/s (2082kB/s)(239MiB/120229msec)
- slat (nsec): min=0, max=538330k, avg=792992.72, stdev=17014015.86
- clat (nsec): min=0, max=1096.8M, avg=125120214.34, stdev=193494723.33
- lat (nsec): min=0, max=1096.8M, avg=125914117.71, stdev=194040761.47
- clat percentiles (usec):
- | 1.00th=[ 971], 5.00th=[ 1188], 10.00th=[ 1434],
- | 20.00th=[ 2073], 30.00th=[ 3556], 40.00th=[ 7242],
- | 50.00th=[ 22676], 60.00th=[ 85459], 70.00th=[ 125305],
- | 80.00th=[ 210764], 90.00th=[ 392168], 95.00th=[ 549454],
- | 99.00th=[ 868221], 99.50th=[ 918553], 99.90th=[1098908],
- | 99.95th=[1098908], 99.99th=[1098908]
- bw ( KiB/s): min= 0, max= 2264, per=14.20%, avg=2034.50, stdev=324.56, samples=2
- iops : min= 0, max= 35, avg=31.50, stdev= 4.95, samples=2
- lat (usec) : 500=0.05%, 750=0.03%, 1000=1.49%
- lat (msec) : 2=17.42%, 4=13.13%, 10=11.08%, 20=5.87%, 50=5.32%
- lat (msec) : 100=9.90%, 250=18.08%, 500=11.53%, 750=3.90%, 1000=2.04%
- cpu : usr=0.04%, sys=0.14%, ctx=7226, majf=0, minf=29
- IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=0,3817,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- read: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:35:16 2019
- read: IOPS=470, BW=29.4MiB/s (30.8MB/s)(3525MiB/120058msec)
- slat (nsec): min=0, max=528245k, avg=50365.71, stdev=3931550.84
- clat (nsec): min=0, max=976416k, avg=8467552.73, stdev=53420881.21
- lat (nsec): min=0, max=976446k, avg=8518714.93, stdev=53571321.10
- clat percentiles (nsec):
- | 1.00th=[ 708], 5.00th=[ 708], 10.00th=[ 716],
- | 20.00th=[ 724], 30.00th=[ 724], 40.00th=[ 740],
- | 50.00th=[ 812], 60.00th=[ 852], 70.00th=[ 900],
- | 80.00th=[ 1032], 90.00th=[ 23168], 95.00th=[ 27656192],
- | 99.00th=[274726912], 99.50th=[442499072], 99.90th=[700448768],
- | 99.95th=[784334848], 99.99th=[910163968]
- bw ( KiB/s): min= 0, max=45037, per=70.93%, avg=30076.50, stdev=21157.34, samples=2
- iops : min= 0, max= 706, avg=471.00, stdev=332.34, samples=2
- lat (nsec) : 750=41.94%, 1000=37.25%
- lat (usec) : 2=8.02%, 4=0.04%, 10=1.72%, 20=0.97%, 50=0.35%
- lat (usec) : 100=0.10%, 250=0.03%, 500=0.09%, 750=0.46%, 1000=0.92%
- lat (msec) : 2=1.38%, 4=0.84%, 10=0.46%, 20=0.27%, 50=1.30%
- lat (msec) : 100=1.72%, 250=1.03%, 500=0.78%, 750=0.27%, 1000=0.06%
- cpu : usr=0.12%, sys=0.46%, ctx=10834, majf=0, minf=29
- IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=56535,0,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- readwrite: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:35:16 2019
- read: IOPS=192, BW=12.1MiB/s (12.6MB/s)(1446MiB/120008msec)
- slat (nsec): min=0, max=537040k, avg=77887.67, stdev=4761859.59
- clat (nsec): min=0, max=1008.6M, avg=2244051.44, stdev=19678423.59
- lat (nsec): min=0, max=1018.6M, avg=2322966.53, stdev=20791907.39
- clat percentiles (nsec):
- | 1.00th=[ 716], 5.00th=[ 732], 10.00th=[ 748],
- | 20.00th=[ 844], 30.00th=[ 1032], 40.00th=[ 1528],
- | 50.00th=[ 342016], 60.00th=[ 733184], 70.00th=[ 864256],
- | 80.00th=[ 1155072], 90.00th=[ 2179072], 95.00th=[ 3751936],
- | 99.00th=[ 39583744], 99.50th=[ 72876032], 99.90th=[329252864],
- | 99.95th=[463470592], 99.99th=[608174080]
- bw ( KiB/s): min= 0, max=22722, per=29.11%, avg=12342.00, stdev=14679.54, samples=2
- iops : min= 0, max= 355, avg=192.50, stdev=229.81, samples=2
- write: IOPS=192, BW=12.0MiB/s (12.6MB/s)(1444MiB/120008msec)
- slat (nsec): min=0, max=410477k, avg=203439.38, stdev=3377915.14
- clat (nsec): min=0, max=1034.9M, avg=18357830.34, stdev=67889277.51
- lat (nsec): min=0, max=1034.9M, avg=18562367.41, stdev=67958632.05
- clat percentiles (usec):
- | 1.00th=[ 881], 5.00th=[ 1012], 10.00th=[ 1090], 20.00th=[ 1205],
- | 30.00th=[ 1319], 40.00th=[ 1434], 50.00th=[ 1631], 60.00th=[ 1958],
- | 70.00th=[ 2769], 80.00th=[ 4948], 90.00th=[ 20579], 95.00th=[ 98042],
- | 99.00th=[404751], 99.50th=[492831], 99.90th=[608175], 99.95th=[708838],
- | 99.99th=[918553]
- bw ( KiB/s): min= 0, max=22592, per=85.97%, avg=12319.00, stdev=14528.22, samples=2
- iops : min= 0, max= 353, avg=192.00, stdev=227.69, samples=2
- lat (nsec) : 750=5.08%, 1000=9.32%
- lat (usec) : 2=5.93%, 4=0.01%, 10=0.22%, 20=0.58%, 50=0.52%
- lat (usec) : 100=0.53%, 250=1.30%, 500=3.34%, 750=3.99%, 1000=9.36%
- lat (msec) : 2=34.81%, 4=11.10%, 10=6.41%, 20=1.81%, 50=1.59%
- lat (msec) : 100=1.50%, 250=1.44%, 500=0.93%, 750=0.24%, 1000=0.01%
- cpu : usr=0.23%, sys=0.83%, ctx=63019, majf=0, minf=37
- IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=23145,23098,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- Run status group 0 (all jobs):
- READ: bw=41.4MiB/s (43.4MB/s), 12.1MiB/s-29.4MiB/s (12.6MB/s-30.8MB/s), io=4971MiB (5213MB), run=120008-120058msec
- WRITE: bw=13.0MiB/s (14.7MB/s), 2033KiB/s-12.0MiB/s (2082kB/s-12.6MB/s), io=1683MiB (1764MB), run=120008-120229msec
- Disk stats (read/write):
- rbd153: ios=17769/27091, merge=0/1, ticks=542581/849541, in_queue=1369843, util=18.48%
- Jobs: 2 (f=2): [_(2),m(1)][97.1%][r=158M,w=151M][r=2463,w=2356 IOPS][eta 00m:34s] Jobs: 2 (f=2): [_(2),m(1)][97.2%][r=178M,w=165M][r=2784,w=2585 IOPS][eta 00m:33s] Jobs: 2 (f=2): [_(2),m(1)][97.3%][r=172M,w=186M][r=2691,w=2905 IOPS][eta 00m:32s] Jobs: 2 (f=2): [_(2),m(1)][97.4%][r=174M,w=180M][r=2711,w=2820 IOPS][eta 00m:31s] Jobs: 2 (f=2): [_(2),m(1)][97.5%][r=151M,w=157M][r=2364,w=2454 IOPS][eta 00m:30s] Jobs: 2 (f=2): [_(2),m(1)][97.5%][r=34.6M,w=36.0M][r=539,w=561 IOPS][eta 00m:29s] Jobs: 2 (f=2): [_(2),m(1)][97.6%][r=194M,w=176M][r=3028,w=2746 IOPS][eta 00m:28s] Jobs: 2 (f=2): [_(2),m(1)][97.7%][r=182M,w=173M][r=2837,w=2706 IOPS][eta 00m:27s] <example-benchmark-fio-server-561-benchmark-p2zwl>
- randwrite: (groupid=0, jobs=1): err= 0: pid=11: Wed May 8 20:35:23 2019
- write: IOPS=25, BW=1646KiB/s (1686kB/s)(194MiB/120815msec)
- slat (nsec): min=0, max=904337k, avg=1214381.02, stdev=28710695.03
- clat (nsec): min=0, max=2205.6M, avg=154348561.57, stdev=313344406.92
- lat (nsec): min=0, max=2205.7M, avg=155563464.57, stdev=315846325.37
- clat percentiles (usec):
- | 1.00th=[ 1012], 5.00th=[ 1254], 10.00th=[ 1483],
- | 20.00th=[ 1926], 30.00th=[ 2671], 40.00th=[ 3752],
- | 50.00th=[ 5735], 60.00th=[ 11469], 70.00th=[ 44303],
- | 80.00th=[ 183501], 90.00th=[ 641729], 95.00th=[ 868221],
- | 99.00th=[1317012], 99.50th=[1417675], 99.90th=[1988101],
- | 99.95th=[2164261], 99.99th=[2197816]
- bw ( KiB/s): min= 0, max= 2024, per=8.70%, avg=1655.00, stdev=521.84, samples=2
- iops : min= 0, max= 31, avg=25.50, stdev= 7.78, samples=2
- lat (usec) : 500=0.03%, 750=0.06%, 1000=0.81%
- lat (msec) : 2=20.35%, 4=20.32%, 10=16.46%, 20=7.02%, 50=5.96%
- lat (msec) : 100=6.02%, 250=4.35%, 500=3.74%, 750=7.83%, 1000=3.96%
- cpu : usr=0.03%, sys=0.11%, ctx=5679, majf=0, minf=26
- IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=0,3105,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- read: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:35:23 2019
- read: IOPS=488, BW=30.4MiB/s (31.9MB/s)(3658MiB/120210msec)
- slat (nsec): min=0, max=893326k, avg=59818.55, stdev=5752672.97
- clat (nsec): min=0, max=1867.1M, avg=8176169.39, stdev=74702180.11
- lat (nsec): min=0, max=1867.1M, avg=8236986.67, stdev=74929978.25
- clat percentiles (nsec):
- | 1.00th=[ 716], 5.00th=[ 724], 10.00th=[ 724],
- | 20.00th=[ 740], 30.00th=[ 780], 40.00th=[ 796],
- | 50.00th=[ 828], 60.00th=[ 860], 70.00th=[ 892],
- | 80.00th=[ 940], 90.00th=[ 11584], 95.00th=[ 790528],
- | 99.00th=[ 404750336], 99.50th=[ 624951296], 99.90th=[1061158912],
- | 99.95th=[1233125376], 99.99th=[1417674752]
- bw ( KiB/s): min= 0, max=45246, per=64.08%, avg=31208.00, stdev=19852.73, samples=2
- iops : min= 0, max= 710, avg=489.00, stdev=312.54, samples=2
- lat (nsec) : 750=21.79%, 1000=60.61%
- lat (usec) : 2=5.91%, 4=0.14%, 10=0.58%, 20=4.82%, 50=0.60%
- lat (usec) : 100=0.20%, 250=0.09%, 500=0.09%, 750=0.12%, 1000=0.44%
- lat (msec) : 2=1.01%, 4=0.75%, 10=0.49%, 20=0.23%, 50=0.43%
- lat (msec) : 100=0.45%, 250=0.11%, 500=0.45%, 750=0.43%, 1000=0.16%
- cpu : usr=0.11%, sys=0.48%, ctx=6521, majf=0, minf=32
- IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=58746,0,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- readwrite: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:35:23 2019
- read: IOPS=274, BW=17.2MiB/s (17.0MB/s)(2060MiB/120002msec)
- slat (nsec): min=0, max=11991k, avg=33428.44, stdev=87379.39
- clat (nsec): min=0, max=1079.6M, avg=1861929.90, stdev=18135558.66
- lat (nsec): min=0, max=1079.6M, avg=1896414.99, stdev=18137606.47
- clat percentiles (nsec):
- | 1.00th=[ 708], 5.00th=[ 732], 10.00th=[ 756],
- | 20.00th=[ 908], 30.00th=[ 1352], 40.00th=[ 284672],
- | 50.00th=[ 505856], 60.00th=[ 741376], 70.00th=[ 839680],
- | 80.00th=[ 1036288], 90.00th=[ 1843200], 95.00th=[ 3063808],
- | 99.00th=[ 22413312], 99.50th=[ 58458112], 99.90th=[263192576],
- | 99.95th=[375390208], 99.99th=[926941184]
- bw ( KiB/s): min= 0, max=34082, per=36.09%, avg=17577.00, stdev=23341.59, samples=2
- iops : min= 0, max= 532, avg=274.00, stdev=364.87, samples=2
- write: IOPS=273, BW=17.1MiB/s (17.9MB/s)(2051MiB/120002msec)
- slat (nsec): min=0, max=386496k, avg=167469.00, stdev=2137773.06
- clat (nsec): min=0, max=1040.4M, avg=12588891.95, stdev=54899280.21
- lat (nsec): min=0, max=1040.4M, avg=12757420.24, stdev=54996504.54
- clat percentiles (usec):
- | 1.00th=[ 857], 5.00th=[ 988], 10.00th=[ 1057], 20.00th=[ 1156],
- | 30.00th=[ 1237], 40.00th=[ 1336], 50.00th=[ 1450], 60.00th=[ 1647],
- | 70.00th=[ 2114], 80.00th=[ 3392], 90.00th=[ 9110], 95.00th=[ 42730],
- | 99.00th=[337642], 99.50th=[438305], 99.90th=[566232], 99.95th=[608175],
- | 99.99th=[692061]
- bw ( KiB/s): min= 0, max=33863, per=91.97%, avg=17504.00, stdev=23135.12, samples=2
- iops : min= 0, max= 529, avg=273.00, stdev=362.04, samples=2
- lat (nsec) : 750=4.52%, 1000=7.01%
- lat (usec) : 2=4.12%, 4=0.02%, 10=0.13%, 20=0.47%, 50=0.50%
- lat (usec) : 100=0.44%, 250=1.78%, 500=6.01%, 750=5.68%, 1000=11.66%
- lat (msec) : 2=37.37%, 4=10.03%, 10=4.85%, 20=1.67%, 50=1.06%
- lat (msec) : 100=0.95%, 250=0.97%, 500=0.62%, 750=0.15%, 1000=0.01%
- cpu : usr=0.31%, sys=1.09%, ctx=95248, majf=0, minf=34
- IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=32958,32820,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- Run status group 0 (all jobs):
- READ: bw=47.6MiB/s (49.9MB/s), 17.2MiB/s-30.4MiB/s (17.0MB/s-31.9MB/s), io=5717MiB (5995MB), run=120002-120210msec
- WRITE: bw=18.6MiB/s (19.5MB/s), 1646KiB/s-17.1MiB/s (1686kB/s-17.9MB/s), io=2246MiB (2355MB), run=120002-120815msec
- Disk stats (read/write):
- rbd281: ios=24610/36374, merge=0/1, ticks=576701/893934, in_queue=1440916, util=15.37%
- Jobs: 1 (f=1): [_(2),m(1)][94.7%][r=92.5M,w=89.8M][r=1446,w=1403 IOPS][eta 00m:26s] Jobs: 1 (f=1): [_(2),m(1)][94.9%][r=82.6M,w=89.0M][r=1291,w=1391 IOPS][eta 00m:25s] Jobs: 1 (f=1): [_(2),m(1)][95.1%][r=93.7M,w=90.5M][r=1463,w=1413 IOPS][eta 00m:24s] Jobs: 1 (f=1): [_(2),m(1)][95.3%][r=89.0M,w=94.5M][r=1406,w=1476 IOPS][eta 00m:23s] Jobs: 1 (f=1): [_(2),m(1)][95.5%][r=91.4M,w=89.8M][r=1428,w=1403 IOPS][eta 00m:22s] Jobs: 1 (f=1): [_(2),m(1)][95.7%][r=100M,w=97.9M][r=1564,w=1530 IOPS][eta 00m:21s] Jobs: 1 (f=1): [_(2),m(1)][95.9%][r=36.5M,w=39.4M][r=571,w=615 IOPS][eta 00m:20s] Jobs: 1 (f=1): [_(2),m(1)][96.1%][r=36.7M,w=40.8M][r=574,w=638 IOPS][eta 00m:19s] Jobs: 1 (f=1): [_(2),m(1)][96.3%][r=95.3M,w=94.1M][r=1489,w=1470 IOPS][eta 00m:18s] Jobs: 1 (f=1): [_(2),m(1)][96.5%][r=116M,w=114M][r=1811,w=1787 IOPS][eta 00m:17s] Jobs: 1 (f=1): [_(2),m(1)][96.9%][r=113M,w=115M][r=1765,w=1797 IOPS][eta 00m:15s] Jobs: 1 (f=1): [_(2),m(1)][96.9%][r=118M,w=119M][r=1851,w=1857 IOPS][eta 00m:15s] Jobs: 1 (f=1): [_(2),m(1)][97.1%][r=113M,w=117M][r=1769,w=1825 IOPS][eta 00m:14s] Jobs: 1 (f=1): [_(2),m(1)][97.6%][r=117M,w=117M][r=1835,w=1833 IOPS][eta 00m:12s] Jobs: 1 (f=1): [_(2),m(1)][97.6%][r=112M,w=114M][r=1747,w=1778 IOPS][eta 00m:12s] Jobs: 1 (f=1): [_(2),m(1)][97.8%][r=120M,w=118M][r=1871,w=1840 IOPS][eta 00m:11s] Jobs: 1 (f=1): [_(2),m(1)][98.0%][r=114M,w=120M][r=1775,w=1872 IOPS][eta 00m:10s] Jobs: 1 (f=1): [_(2),m(1)][98.4%][r=110M,w=102M][r=1720,w=1599 IOPS][eta 00m:08s] Jobs: 1 (f=1): [_(2),m(1)][98.4%][r=20.8M,w=25.1M][r=324,w=391 IOPS][eta 00m:08s] Jobs: 1 (f=1): [_(2),m(1)][98.6%][r=106M,w=102M][r=1663,w=1601 IOPS][eta 00m:07s] Jobs: 1 (f=1): [_(2),m(1)][98.8%][r=107M,w=107M][r=1677,w=1665 IOPS][eta 00m:06s] Jobs: 1 (f=1): [_(2),m(1)][99.0%][r=107M,w=105M][r=1667,w=1635 IOPS][eta 00m:05s] Jobs: 1 (f=1): [_(2),m(1)][99.2%][r=105M,w=104M][r=1645,w=1622 IOPS][eta 00m:04s] Jobs: 1 (f=1): [_(2),m(1)][99.4%][r=106M,w=111M][r=1663,w=1736 IOPS][eta 00m:03s] Jobs: 1 (f=1): [_(2),m(1)][99.6%][r=107M,w=101M][r=1677,w=1585 IOPS][eta 00m:02s] Jobs: 1 (f=1): [_(2),m(1)][99.8%][r=106M,w=107M][r=1654,w=1665 IOPS][eta 00m:01s] Jobs: 1 (f=1): [_(2),m(1)][100.0%][r=108M,w=107M][r=1686,w=1669 IOPS][eta 00m:00s] <example-benchmark-fio-server-657-benchmark-nfsfh>
- randwrite: (groupid=0, jobs=1): err= 0: pid=11: Wed May 8 20:35:50 2019
- write: IOPS=40, BW=2566KiB/s (2628kB/s)(301MiB/120045msec)
- slat (nsec): min=0, max=482145k, avg=438482.98, stdev=10952692.52
- clat (nsec): min=0, max=1658.2M, avg=99703536.50, stdev=166032787.67
- lat (nsec): min=0, max=1658.3M, avg=100143246.46, stdev=166305537.98
- clat percentiles (usec):
- | 1.00th=[ 947], 5.00th=[ 1188], 10.00th=[ 1418],
- | 20.00th=[ 1909], 30.00th=[ 3163], 40.00th=[ 7635],
- | 50.00th=[ 40109], 60.00th=[ 70779], 70.00th=[ 101188],
- | 80.00th=[ 130548], 90.00th=[ 316670], 95.00th=[ 446694],
- | 99.00th=[ 750781], 99.50th=[ 935330], 99.90th=[1451230],
- | 99.95th=[1602225], 99.99th=[1652556]
- bw ( KiB/s): min= 0, max= 1573, per=3.75%, avg=1573.00, stdev= 0.00, samples=1
- iops : min= 0, max= 24, avg=24.00, stdev= 0.00, samples=1
- lat (usec) : 500=0.02%, 750=0.06%, 1000=1.64%
- lat (msec) : 2=19.38%, 4=12.47%, 10=8.40%, 20=3.24%, 50=7.46%
- lat (msec) : 100=16.80%, 250=18.79%, 500=8.40%, 750=2.35%, 1000=0.64%
- cpu : usr=0.05%, sys=0.16%, ctx=8915, majf=0, minf=26
- IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=0,4810,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- read: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:35:50 2019
- read: IOPS=612, BW=38.2MiB/s (40.0MB/s)(4582MiB/120023msec)
- slat (nsec): min=0, max=463662k, avg=27019.50, stdev=2410645.20
- clat (nsec): min=0, max=1544.2M, avg=6507416.01, stdev=46288893.88
- lat (nsec): min=0, max=1544.3M, avg=6535236.00, stdev=46359211.29
- clat percentiles (nsec):
- | 1.00th=[ 708], 5.00th=[ 708], 10.00th=[ 716],
- | 20.00th=[ 716], 30.00th=[ 724], 40.00th=[ 732],
- | 50.00th=[ 740], 60.00th=[ 812], 70.00th=[ 868],
- | 80.00th=[ 948], 90.00th=[ 10432], 95.00th=[ 8454144],
- | 99.00th=[ 191889408], 99.50th=[ 358612992], 99.90th=[ 616562688],
- | 99.95th=[ 750780416], 99.99th=[1149239296]
- bw ( KiB/s): min= 0, max=57456, per=49.78%, avg=39101.00, stdev=25957.89, samples=2
- iops : min= 0, max= 901, avg=612.50, stdev=408.00, samples=2
- lat (nsec) : 750=52.09%, 1000=30.09%
- lat (usec) : 2=6.66%, 4=0.04%, 10=1.10%, 20=0.87%, 50=0.30%
- lat (usec) : 100=0.07%, 250=0.04%, 500=0.30%, 750=0.53%, 1000=0.91%
- lat (msec) : 2=1.09%, 4=0.63%, 10=0.33%, 20=0.24%, 50=1.71%
- lat (msec) : 100=1.41%, 250=0.79%, 500=0.62%, 750=0.14%, 1000=0.04%
- cpu : usr=0.16%, sys=0.56%, ctx=12882, majf=0, minf=108
- IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=73566,0,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- readwrite: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:35:50 2019
- read: IOPS=616, BW=38.5MiB/s (40.4MB/s)(4624MiB/120002msec)
- slat (nsec): min=0, max=4051.2k, avg=34070.14, stdev=38397.97
- clat (nsec): min=0, max=983244k, avg=1157929.31, stdev=11857203.78
- lat (nsec): min=0, max=983271k, avg=1193149.93, stdev=11857821.07
- clat percentiles (nsec):
- | 1.00th=[ 724], 5.00th=[ 780], 10.00th=[ 996],
- | 20.00th=[ 248832], 30.00th=[ 313344], 40.00th=[ 399360],
- | 50.00th=[ 643072], 60.00th=[ 733184], 70.00th=[ 806912],
- | 80.00th=[ 921600], 90.00th=[ 1302528], 95.00th=[ 2211840],
- | 99.00th=[ 5668864], 99.50th=[ 11862016], 99.90th=[ 98041856],
- | 99.95th=[212860928], 99.99th=[692060160]
- bw ( KiB/s): min= 0, max=76727, per=50.24%, avg=39461.00, stdev=52702.08, samples=2
- iops : min= 0, max= 1198, avg=616.00, stdev=823.07, samples=2
- write: IOPS=615, BW=38.5MiB/s (40.4MB/s)(4619MiB/120002msec)
- slat (nsec): min=0, max=32795k, avg=144062.14, stdev=169244.18
- clat (nsec): min=0, max=985893k, avg=5161799.35, stdev=27012639.31
- lat (nsec): min=0, max=985966k, avg=5306860.77, stdev=27005522.14
- clat percentiles (usec):
- | 1.00th=[ 848], 5.00th=[ 979], 10.00th=[ 1057], 20.00th=[ 1139],
- | 30.00th=[ 1221], 40.00th=[ 1287], 50.00th=[ 1352], 60.00th=[ 1434],
- | 70.00th=[ 1565], 80.00th=[ 1860], 90.00th=[ 3392], 95.00th=[ 6783],
- | 99.00th=[111674], 99.50th=[189793], 99.90th=[375391], 99.95th=[413139],
- | 99.99th=[574620]
- bw ( KiB/s): min= 0, max=76539, per=93.92%, avg=39416.50, stdev=52499.14, samples=2
- iops : min= 0, max= 1195, avg=615.00, stdev=820.24, samples=2
- lat (nsec) : 750=1.86%, 1000=3.17%
- lat (usec) : 2=1.65%, 4=0.01%, 10=0.05%, 20=0.16%, 50=0.19%
- lat (usec) : 100=0.18%, 250=2.78%, 500=12.46%, 750=8.78%, 1000=13.88%
- lat (msec) : 2=43.04%, 4=6.77%, 10=2.70%, 20=0.68%, 50=0.48%
- lat (msec) : 100=0.50%, 250=0.48%, 500=0.17%, 750=0.01%, 1000=0.01%
- cpu : usr=0.67%, sys=2.18%, ctx=235909, majf=0, minf=101
- IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
- submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
- issued rwts: total=73990,73908,0,0 short=0,0,0,0 dropped=0,0,0,0
- latency : target=0, window=0, percentile=100.00%, depth=4
- Run status group 0 (all jobs):
- READ: bw=76.7MiB/s (80.4MB/s), 38.2MiB/s-38.5MiB/s (40.0MB/s-40.4MB/s), io=9207MiB (9654MB), run=120002-120023msec
- WRITE: bw=40.0MiB/s (42.0MB/s), 2566KiB/s-38.5MiB/s (2628kB/s-40.4MB/s), io=4920MiB (5159MB), run=120002-120045msec
- Disk stats (read/write):
- rbd266: ios=69727/79136, merge=0/1, ticks=587936/790828, in_queue=1307271, util=33.49%
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement