Advertisement
Guest User

Untitled

a guest
May 19th, 2019
681
0
Never
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 438.50 KB | None | 0 0
  1. <example-benchmark-fio-server-951-benchmark-9cd4p>
  2. randwrite: (groupid=0, jobs=1): err= 0: pid=11: Wed May 8 20:34:54 2019
  3. write: IOPS=36, BW=2354KiB/s (2411kB/s)(276MiB/120060msec)
  4. slat (nsec): min=0, max=504592k, avg=441726.08, stdev=10395455.15
  5. clat (nsec): min=0, max=995152k, avg=108771492.32, stdev=167527907.41
  6. lat (nsec): min=0, max=995235k, avg=109214317.91, stdev=167771537.56
  7. clat percentiles (usec):
  8. | 1.00th=[ 988], 5.00th=[ 1172], 10.00th=[ 1336], 20.00th=[ 1778],
  9. | 30.00th=[ 2671], 40.00th=[ 6194], 50.00th=[ 34341], 60.00th=[ 77071],
  10. | 70.00th=[107480], 80.00th=[158335], 90.00th=[358613], 95.00th=[501220],
  11. | 99.00th=[759170], 99.50th=[868221], 99.90th=[935330], 99.95th=[935330],
  12. | 99.99th=[994051]
  13. bw ( KiB/s): min= 0, max= 3002, per=52.90%, avg=2352.50, stdev=918.53, samples=2
  14. iops : min= 0, max= 46, avg=36.00, stdev=14.14, samples=2
  15. lat (usec) : 4=0.02%, 500=0.02%, 750=0.07%, 1000=1.13%
  16. lat (msec) : 2=22.09%, 4=12.19%, 10=7.36%, 20=4.44%, 50=4.51%
  17. lat (msec) : 100=15.57%, 250=17.99%, 500=9.61%, 750=4.03%, 1000=1.02%
  18. cpu : usr=0.04%, sys=0.16%, ctx=8445, majf=0, minf=31
  19. IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  20. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  21. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  22. issued rwts: total=0,4413,0,0 short=0,0,0,0 dropped=0,0,0,0
  23. latency : target=0, window=0, percentile=100.00%, depth=4
  24. read: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:34:54 2019
  25. read: IOPS=348, BW=21.8MiB/s (22.8MB/s)(2614MiB/120030msec)
  26. slat (nsec): min=0, max=498780k, avg=50078.56, stdev=3579794.07
  27. clat (nsec): min=0, max=937417k, avg=11464988.97, stdev=59818482.14
  28. lat (nsec): min=0, max=937452k, avg=11515884.91, stdev=59935424.53
  29. clat percentiles (nsec):
  30. | 1.00th=[ 708], 5.00th=[ 716], 10.00th=[ 716],
  31. | 20.00th=[ 724], 30.00th=[ 732], 40.00th=[ 740],
  32. | 50.00th=[ 764], 60.00th=[ 852], 70.00th=[ 924],
  33. | 80.00th=[ 1288], 90.00th=[ 1941504], 95.00th=[ 51118080],
  34. | 99.00th=[362807296], 99.50th=[471859200], 99.90th=[692060160],
  35. | 99.95th=[734003200], 99.99th=[843055104]
  36. bw ( KiB/s): min= 0, max=28792, per=91.76%, avg=22304.00, stdev=9175.42, samples=2
  37. iops : min= 0, max= 450, avg=348.50, stdev=143.54, samples=2
  38. lat (nsec) : 750=47.82%, 1000=25.51%
  39. lat (usec) : 2=9.42%, 4=0.02%, 10=0.63%, 20=0.59%, 50=0.28%
  40. lat (usec) : 100=0.10%, 250=0.04%, 500=0.29%, 750=1.01%, 1000=1.91%
  41. lat (msec) : 2=2.44%, 4=1.35%, 10=0.63%, 20=0.48%, 50=2.41%
  42. lat (msec) : 100=2.11%, 250=1.42%, 500=1.13%, 750=0.38%, 1000=0.03%
  43. cpu : usr=0.11%, sys=0.37%, ctx=12986, majf=0, minf=31
  44. IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  45. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  46. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  47. issued rwts: total=41881,0,0,0 short=0,0,0,0 dropped=0,0,0,0
  48. latency : target=0, window=0, percentile=100.00%, depth=4
  49. readwrite: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:34:54 2019
  50. read: IOPS=31, BW=2014KiB/s (2062kB/s)(236MiB/120065msec)
  51. slat (nsec): min=0, max=464438k, avg=251788.17, stdev=10003721.86
  52. clat (nsec): min=0, max=761107k, avg=7488674.89, stdev=41769913.15
  53. lat (nsec): min=0, max=761136k, avg=7741358.10, stdev=42918867.18
  54. clat percentiles (nsec):
  55. | 1.00th=[ 708], 5.00th=[ 724], 10.00th=[ 732],
  56. | 20.00th=[ 756], 30.00th=[ 844], 40.00th=[ 924],
  57. | 50.00th=[ 1080], 60.00th=[ 1208], 70.00th=[ 1336],
  58. | 80.00th=[ 1624], 90.00th=[ 1204224], 95.00th=[ 38535168],
  59. | 99.00th=[214958080], 99.50th=[325058560], 99.90th=[534773760],
  60. | 99.95th=[675282944], 99.99th=[759169024]
  61. bw ( KiB/s): min= 0, max= 2421, per=8.29%, avg=2014.50, stdev=574.88, samples=2
  62. iops : min= 0, max= 37, avg=31.00, stdev= 8.49, samples=2
  63. write: IOPS=32, BW=2094KiB/s (2144kB/s)(246MiB/120065msec)
  64. slat (nsec): min=0, max=552114k, avg=319489.42, stdev=8814876.07
  65. clat (nsec): min=0, max=1106.9M, avg=115024253.69, stdev=172725727.36
  66. lat (nsec): min=0, max=1107.2M, avg=115344697.54, stdev=172831915.20
  67. clat percentiles (usec):
  68. | 1.00th=[ 1020], 5.00th=[ 1205], 10.00th=[ 1401],
  69. | 20.00th=[ 1876], 30.00th=[ 2900], 40.00th=[ 8029],
  70. | 50.00th=[ 46924], 60.00th=[ 82314], 70.00th=[ 111674],
  71. | 80.00th=[ 175113], 90.00th=[ 371196], 95.00th=[ 509608],
  72. | 99.00th=[ 759170], 99.50th=[ 876610], 99.90th=[ 960496],
  73. | 99.95th=[ 994051], 99.99th=[1098908]
  74. bw ( KiB/s): min= 0, max= 2517, per=47.05%, avg=2092.50, stdev=600.33, samples=2
  75. iops : min= 0, max= 39, avg=32.50, stdev= 9.19, samples=2
  76. lat (nsec) : 750=9.36%, 1000=12.80%
  77. lat (usec) : 2=17.65%, 4=0.09%, 10=0.25%, 20=0.70%, 50=0.53%
  78. lat (usec) : 100=0.62%, 250=0.32%, 500=0.42%, 750=0.45%, 1000=1.00%
  79. lat (msec) : 2=11.62%, 4=6.70%, 10=3.98%, 20=2.15%, 50=4.18%
  80. lat (msec) : 100=9.19%, 250=9.53%, 500=5.70%, 750=2.26%, 1000=0.52%
  81. cpu : usr=0.04%, sys=0.19%, ctx=8692, majf=0, minf=34
  82. IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  83. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  84. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  85. issued rwts: total=3779,3925,0,0 short=0,0,0,0 dropped=0,0,0,0
  86. latency : target=0, window=0, percentile=100.00%, depth=4
  87.  
  88. Run status group 0 (all jobs):
  89. READ: bw=23.7MiB/s (24.9MB/s), 2014KiB/s-21.8MiB/s (2062kB/s-22.8MB/s), io=2850MiB (2989MB), run=120030-120065msec
  90. WRITE: bw=4448KiB/s (4554kB/s), 2094KiB/s-2354KiB/s (2144kB/s-2411kB/s), io=522MiB (547MB), run=120060-120065msec
  91.  
  92. Disk stats (read/write):
  93. rbd300: ios=7336/8516, merge=0/1, ticks=525672/830529, in_queue=1348119, util=12.06%
  94. Jobs: 117 (f=116): [_(2),m(1)][99.9%][r=1334M,w=800M][r=20.8k,w=12.5k IOPS][eta 00m:56s] <example-benchmark-fio-server-255-benchmark-v6glb>
  95. randwrite: (groupid=0, jobs=1): err= 0: pid=11: Wed May 8 20:34:54 2019
  96. write: IOPS=33, BW=2171KiB/s (2223kB/s)(254MiB/120036msec)
  97. slat (nsec): min=0, max=536048k, avg=417781.11, stdev=10887713.42
  98. clat (nsec): min=0, max=1261.6M, avg=118027748.44, stdev=178274062.80
  99. lat (nsec): min=0, max=1261.6M, avg=118446560.37, stdev=178442905.58
  100. clat percentiles (usec):
  101. | 1.00th=[ 947], 5.00th=[ 1205], 10.00th=[ 1467],
  102. | 20.00th=[ 2114], 30.00th=[ 3654], 40.00th=[ 10028],
  103. | 50.00th=[ 58459], 60.00th=[ 86508], 70.00th=[ 111674],
  104. | 80.00th=[ 177210], 90.00th=[ 371196], 95.00th=[ 530580],
  105. | 99.00th=[ 775947], 99.50th=[ 901776], 99.90th=[1098908],
  106. | 99.95th=[1098908], 99.99th=[1266680]
  107. bw ( KiB/s): min= 0, max= 3001, per=53.65%, avg=2168.50, stdev=1177.33, samples=2
  108. iops : min= 0, max= 46, avg=33.00, stdev=18.38, samples=2
  109. lat (usec) : 750=0.05%, 1000=1.38%
  110. lat (msec) : 2=17.77%, 4=11.92%, 10=8.85%, 20=3.91%, 50=4.47%
  111. lat (msec) : 100=17.16%, 250=19.59%, 500=9.05%, 750=4.62%, 1000=0.96%
  112. cpu : usr=0.04%, sys=0.15%, ctx=7612, majf=0, minf=30
  113. IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  114. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  115. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  116. issued rwts: total=0,4068,0,0 short=0,0,0,0 dropped=0,0,0,0
  117. latency : target=0, window=0, percentile=100.00%, depth=4
  118. read: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:34:54 2019
  119. read: IOPS=273, BW=17.1MiB/s (17.9MB/s)(2052MiB/120023msec)
  120. slat (nsec): min=0, max=393291k, avg=46338.23, stdev=2796058.39
  121. clat (nsec): min=0, max=1038.1M, avg=14624047.78, stdev=65201130.73
  122. lat (nsec): min=0, max=1038.6M, avg=14671240.74, stdev=65271622.43
  123. clat percentiles (nsec):
  124. | 1.00th=[ 708], 5.00th=[ 716], 10.00th=[ 716],
  125. | 20.00th=[ 724], 30.00th=[ 732], 40.00th=[ 756],
  126. | 50.00th=[ 844], 60.00th=[ 908], 70.00th=[ 1144],
  127. | 80.00th=[ 724992], 90.00th=[ 25559040], 95.00th=[ 67633152],
  128. | 99.00th=[379584512], 99.50th=[488636416], 99.90th=[700448768],
  129. | 99.95th=[750780416], 99.99th=[943718400]
  130. bw ( KiB/s): min= 0, max=22634, per=90.78%, avg=17511.00, stdev=7245.02, samples=2
  131. iops : min= 0, max= 354, avg=273.50, stdev=113.84, samples=2
  132. lat (nsec) : 750=39.24%, 1000=25.97%
  133. lat (usec) : 2=12.59%, 4=0.05%, 10=0.41%, 20=0.41%, 50=0.17%
  134. lat (usec) : 100=0.09%, 250=0.03%, 500=0.24%, 750=1.12%, 1000=2.31%
  135. lat (msec) : 2=2.99%, 4=1.91%, 10=0.98%, 20=0.94%, 50=3.84%
  136. lat (msec) : 100=2.95%, 250=1.98%, 500=1.34%, 750=0.40%, 1000=0.05%
  137. cpu : usr=0.10%, sys=0.33%, ctx=13746, majf=0, minf=29
  138. IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  139. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  140. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  141. issued rwts: total=32856,0,0,0 short=0,0,0,0 dropped=0,0,0,0
  142. latency : target=0, window=0, percentile=100.00%, depth=4
  143. readwrite: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:34:54 2019
  144. read: IOPS=27, BW=1786KiB/s (1829kB/s)(209MiB/120059msec)
  145. slat (nsec): min=0, max=535574k, avg=187960.89, stdev=9251743.42
  146. clat (nsec): min=0, max=794000k, avg=10453672.56, stdev=48478017.97
  147. lat (nsec): min=0, max=794034k, avg=10642538.71, stdev=49327238.20
  148. clat percentiles (nsec):
  149. | 1.00th=[ 708], 5.00th=[ 724], 10.00th=[ 732],
  150. | 20.00th=[ 764], 30.00th=[ 876], 40.00th=[ 988],
  151. | 50.00th=[ 1128], 60.00th=[ 1256], 70.00th=[ 1448],
  152. | 80.00th=[ 100864], 90.00th=[ 3194880], 95.00th=[ 53739520],
  153. | 99.00th=[270532608], 99.50th=[362807296], 99.90th=[522190848],
  154. | 99.95th=[608174080], 99.99th=[792723456]
  155. bw ( KiB/s): min= 0, max= 2412, per=9.26%, avg=1786.50, stdev=884.59, samples=2
  156. iops : min= 0, max= 37, avg=27.50, stdev=13.44, samples=2
  157. write: IOPS=29, BW=1873KiB/s (1918kB/s)(220MiB/120059msec)
  158. slat (nsec): min=0, max=400348k, avg=279434.69, stdev=6757705.47
  159. clat (nsec): min=0, max=1261.4M, avg=126879957.42, stdev=186908714.01
  160. lat (nsec): min=0, max=1261.5M, avg=127160436.98, stdev=186957434.42
  161. clat percentiles (usec):
  162. | 1.00th=[ 1004], 5.00th=[ 1287], 10.00th=[ 1549],
  163. | 20.00th=[ 2278], 30.00th=[ 4228], 40.00th=[ 15795],
  164. | 50.00th=[ 63701], 60.00th=[ 89654], 70.00th=[ 117965],
  165. | 80.00th=[ 196084], 90.00th=[ 396362], 95.00th=[ 549454],
  166. | 99.00th=[ 826278], 99.50th=[ 935330], 99.90th=[1098908],
  167. | 99.95th=[1098908], 99.99th=[1266680]
  168. bw ( KiB/s): min= 0, max= 2512, per=46.29%, avg=1871.00, stdev=906.51, samples=2
  169. iops : min= 0, max= 39, avg=29.00, stdev=14.14, samples=2
  170. lat (nsec) : 750=8.58%, 1000=11.38%
  171. lat (usec) : 2=16.15%, 4=0.07%, 10=0.28%, 20=0.76%, 50=0.80%
  172. lat (usec) : 100=1.02%, 250=0.67%, 500=0.63%, 750=0.63%, 1000=1.46%
  173. lat (msec) : 2=9.58%, 4=7.00%, 10=4.52%, 20=2.42%, 50=3.92%
  174. lat (msec) : 100=10.00%, 250=11.09%, 500=5.71%, 750=2.48%, 1000=0.71%
  175. cpu : usr=0.04%, sys=0.17%, ctx=8048, majf=0, minf=33
  176. IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  177. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  178. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  179. issued rwts: total=3351,3510,0,0 short=0,0,0,0 dropped=0,0,0,0
  180. latency : target=0, window=0, percentile=100.00%, depth=4
  181.  
  182. Run status group 0 (all jobs):
  183. READ: bw=18.8MiB/s (19.8MB/s), 1786KiB/s-17.1MiB/s (1829kB/s-17.9MB/s), io=2262MiB (2372MB), run=120023-120059msec
  184. WRITE: bw=4043KiB/s (4140kB/s), 1873KiB/s-2171KiB/s (1918kB/s-2223kB/s), io=474MiB (497MB), run=120036-120059msec
  185.  
  186. Disk stats (read/write):
  187. rbd75: ios=8536/9131, merge=0/1, ticks=554901/875234, in_queue=1421062, util=12.57%
  188. <example-benchmark-fio-server-424-benchmark-tlngh>
  189. randwrite: (groupid=0, jobs=1): err= 0: pid=11: Wed May 8 20:34:54 2019
  190. write: IOPS=41, BW=2666KiB/s (2730kB/s)(313MiB/120074msec)
  191. slat (nsec): min=0, max=400546k, avg=326325.35, stdev=6957801.48
  192. clat (nsec): min=0, max=1102.3M, avg=96092666.22, stdev=166659881.38
  193. lat (nsec): min=0, max=1102.3M, avg=96420079.84, stdev=167148622.48
  194. clat percentiles (usec):
  195. | 1.00th=[ 1012], 5.00th=[ 1254], 10.00th=[ 1467],
  196. | 20.00th=[ 1991], 30.00th=[ 2835], 40.00th=[ 4686],
  197. | 50.00th=[ 11338], 60.00th=[ 45876], 70.00th=[ 86508],
  198. | 80.00th=[ 133694], 90.00th=[ 346031], 95.00th=[ 480248],
  199. | 99.00th=[ 801113], 99.50th=[ 843056], 99.90th=[ 943719],
  200. | 99.95th=[ 985662], 99.99th=[1098908]
  201. bw ( KiB/s): min= 0, max= 2965, per=55.11%, avg=2665.00, stdev=424.26, samples=2
  202. iops : min= 0, max= 46, avg=41.00, stdev= 7.07, samples=2
  203. lat (usec) : 500=0.02%, 750=0.04%, 1000=0.78%
  204. lat (msec) : 2=19.24%, 4=16.86%, 10=12.00%, 20=6.18%, 50=5.52%
  205. lat (msec) : 100=13.06%, 250=12.44%, 500=9.76%, 750=2.90%, 1000=1.22%
  206. cpu : usr=0.05%, sys=0.18%, ctx=9369, majf=0, minf=31
  207. IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  208. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  209. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  210. issued rwts: total=0,4999,0,0 short=0,0,0,0 dropped=0,0,0,0
  211. latency : target=0, window=0, percentile=100.00%, depth=4
  212. read: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:34:54 2019
  213. read: IOPS=318, BW=19.9MiB/s (20.9MB/s)(2389MiB/120031msec)
  214. slat (nsec): min=0, max=320204k, avg=38057.84, stdev=2181510.66
  215. clat (nsec): min=0, max=1057.4M, avg=12541947.82, stdev=61296772.33
  216. lat (nsec): min=0, max=1057.4M, avg=12580866.82, stdev=61342276.53
  217. clat percentiles (nsec):
  218. | 1.00th=[ 708], 5.00th=[ 716], 10.00th=[ 716],
  219. | 20.00th=[ 724], 30.00th=[ 740], 40.00th=[ 788],
  220. | 50.00th=[ 852], 60.00th=[ 900], 70.00th=[ 1096],
  221. | 80.00th=[ 43776], 90.00th=[ 6324224], 95.00th=[ 58982400],
  222. | 99.00th=[362807296], 99.50th=[480247808], 99.90th=[700448768],
  223. | 99.95th=[767557632], 99.99th=[893386752]
  224. bw ( KiB/s): min= 0, max=25069, per=90.81%, avg=20383.00, stdev=6627.00, samples=2
  225. iops : min= 0, max= 392, avg=318.50, stdev=103.94, samples=2
  226. lat (nsec) : 750=33.09%, 1000=34.07%
  227. lat (usec) : 2=11.70%, 4=0.05%, 10=0.57%, 20=0.41%, 50=0.14%
  228. lat (usec) : 100=0.10%, 250=0.03%, 500=0.14%, 750=1.09%, 1000=2.45%
  229. lat (msec) : 2=3.49%, 4=1.98%, 10=1.14%, 20=0.49%, 50=3.07%
  230. lat (msec) : 100=2.94%, 250=1.55%, 500=1.06%, 750=0.39%, 1000=0.06%
  231. cpu : usr=0.10%, sys=0.37%, ctx=15077, majf=0, minf=31
  232. IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  233. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  234. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  235. issued rwts: total=38240,0,0,0 short=0,0,0,0 dropped=0,0,0,0
  236. latency : target=0, window=0, percentile=100.00%, depth=4
  237. readwrite: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:34:54 2019
  238. read: IOPS=32, BW=2065KiB/s (2115kB/s)(242MiB/120022msec)
  239. slat (nsec): min=0, max=399843k, avg=206891.70, stdev=7876972.64
  240. clat (nsec): min=0, max=737486k, avg=8865298.83, stdev=46977819.17
  241. lat (nsec): min=0, max=976153k, avg=9073210.18, stdev=48841859.71
  242. clat percentiles (nsec):
  243. | 1.00th=[ 708], 5.00th=[ 716], 10.00th=[ 732],
  244. | 20.00th=[ 772], 30.00th=[ 876], 40.00th=[ 972],
  245. | 50.00th=[ 1112], 60.00th=[ 1256], 70.00th=[ 1480],
  246. | 80.00th=[ 109056], 90.00th=[ 2506752], 95.00th=[ 45875200],
  247. | 99.00th=[244318208], 99.50th=[400556032], 99.90th=[583008256],
  248. | 99.95th=[717225984], 99.99th=[734003200]
  249. bw ( KiB/s): min= 0, max= 2312, per=9.20%, avg=2065.00, stdev=349.31, samples=2
  250. iops : min= 0, max= 36, avg=32.00, stdev= 5.66, samples=2
  251. write: IOPS=33, BW=2171KiB/s (2223kB/s)(255MiB/120022msec)
  252. slat (nsec): min=0, max=10291k, avg=180936.82, stdev=303505.98
  253. clat (nsec): min=0, max=1040.1M, avg=109529981.00, stdev=175834302.36
  254. lat (nsec): min=0, max=1040.2M, avg=109712031.36, stdev=175783939.65
  255. clat percentiles (usec):
  256. | 1.00th=[ 1057], 5.00th=[ 1287], 10.00th=[ 1565],
  257. | 20.00th=[ 2278], 30.00th=[ 3621], 40.00th=[ 6587],
  258. | 50.00th=[ 19530], 60.00th=[ 66847], 70.00th=[ 102237],
  259. | 80.00th=[ 160433], 90.00th=[ 371196], 95.00th=[ 497026],
  260. | 99.00th=[ 809501], 99.50th=[ 893387], 99.90th=[ 952108],
  261. | 99.95th=[1002439], 99.99th=[1035994]
  262. bw ( KiB/s): min= 0, max= 2405, per=44.85%, avg=2169.00, stdev=333.75, samples=2
  263. iops : min= 0, max= 37, avg=33.50, stdev= 4.95, samples=2
  264. lat (nsec) : 750=8.20%, 1000=12.22%
  265. lat (usec) : 2=14.93%, 4=0.05%, 10=0.39%, 20=1.15%, 50=1.02%
  266. lat (usec) : 100=0.96%, 250=0.79%, 500=0.94%, 750=0.52%, 1000=1.17%
  267. lat (msec) : 2=9.61%, 4=8.78%, 10=6.61%, 20=3.36%, 50=4.33%
  268. lat (msec) : 100=8.36%, 250=7.93%, 500=6.17%, 750=1.85%, 1000=0.68%
  269. cpu : usr=0.05%, sys=0.19%, ctx=9240, majf=0, minf=35
  270. IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  271. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  272. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  273. issued rwts: total=3874,4069,0,0 short=0,0,0,0 dropped=0,0,0,0
  274. latency : target=0, window=0, percentile=100.00%, depth=4
  275.  
  276. Run status group 0 (all jobs):
  277. READ: bw=21.9MiB/s (22.0MB/s), 2065KiB/s-19.9MiB/s (2115kB/s-20.9MB/s), io=2631MiB (2759MB), run=120022-120031msec
  278. WRITE: bw=4836KiB/s (4953kB/s), 2171KiB/s-2666KiB/s (2223kB/s-2730kB/s), io=567MiB (595MB), run=120022-120074msec
  279.  
  280. Disk stats (read/write):
  281. rbd121: ios=8410/9542, merge=0/1, ticks=542895/868442, in_queue=1402193, util=12.62%
  282. <example-benchmark-fio-server-325-benchmark-fbmfd>
  283. randwrite: (groupid=0, jobs=1): err= 0: pid=11: Wed May 8 20:34:54 2019
  284. write: IOPS=46, BW=2969KiB/s (3040kB/s)(348MiB/120063msec)
  285. slat (nsec): min=0, max=428628k, avg=322718.17, stdev=6988005.94
  286. clat (nsec): min=0, max=1469.7M, avg=86229567.91, stdev=147784132.72
  287. lat (nsec): min=0, max=1469.8M, avg=86553448.19, stdev=148051015.99
  288. clat percentiles (usec):
  289. | 1.00th=[ 988], 5.00th=[ 1188], 10.00th=[ 1401],
  290. | 20.00th=[ 1827], 30.00th=[ 2573], 40.00th=[ 4359],
  291. | 50.00th=[ 10814], 60.00th=[ 61604], 70.00th=[ 91751],
  292. | 80.00th=[ 120062], 90.00th=[ 261096], 95.00th=[ 417334],
  293. | 99.00th=[ 683672], 99.50th=[ 809501], 99.90th=[ 977273],
  294. | 99.95th=[1283458], 99.99th=[1468007]
  295. bw ( KiB/s): min= 0, max= 3621, per=54.56%, avg=2967.50, stdev=924.19, samples=2
  296. iops : min= 0, max= 56, avg=46.00, stdev=14.14, samples=2
  297. lat (usec) : 50=0.02%, 250=0.02%, 750=0.07%, 1000=1.02%
  298. lat (msec) : 2=21.88%, 4=15.52%, 10=11.03%, 20=4.01%, 50=3.92%
  299. lat (msec) : 100=15.99%, 250=16.39%, 500=6.95%, 750=2.48%, 1000=0.68%
  300. cpu : usr=0.07%, sys=0.19%, ctx=10557, majf=0, minf=31
  301. IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  302. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  303. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  304. issued rwts: total=0,5566,0,0 short=0,0,0,0 dropped=0,0,0,0
  305. latency : target=0, window=0, percentile=100.00%, depth=4
  306. read: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:34:54 2019
  307. read: IOPS=295, BW=18.5MiB/s (19.4MB/s)(2218MiB/120022msec)
  308. slat (nsec): min=0, max=413228k, avg=42515.07, stdev=2589450.95
  309. clat (nsec): min=0, max=1400.0M, avg=13517573.15, stdev=63183262.34
  310. lat (nsec): min=0, max=1400.3M, avg=13560936.96, stdev=63244141.80
  311. clat percentiles (nsec):
  312. | 1.00th=[ 708], 5.00th=[ 716], 10.00th=[ 724],
  313. | 20.00th=[ 724], 30.00th=[ 740], 40.00th=[ 772],
  314. | 50.00th=[ 844], 60.00th=[ 900], 70.00th=[ 1080],
  315. | 80.00th=[ 264192], 90.00th=[ 22937600], 95.00th=[ 61079552],
  316. | 99.00th=[375390208], 99.50th=[480247808], 99.90th=[700448768],
  317. | 99.95th=[767557632], 99.99th=[926941184]
  318. bw ( KiB/s): min= 0, max=24299, per=88.89%, avg=18927.00, stdev=7597.16, samples=2
  319. iops : min= 0, max= 380, avg=295.50, stdev=119.50, samples=2
  320. lat (nsec) : 750=36.46%, 1000=30.96%
  321. lat (usec) : 2=11.74%, 4=0.06%, 10=0.34%, 20=0.18%, 50=0.17%
  322. lat (usec) : 100=0.06%, 250=0.05%, 500=0.17%, 750=1.13%, 1000=2.16%
  323. lat (msec) : 2=2.93%, 4=1.72%, 10=1.00%, 20=0.66%, 50=3.78%
  324. lat (msec) : 100=3.29%, 250=1.50%, 500=1.19%, 750=0.41%, 1000=0.06%
  325. cpu : usr=0.10%, sys=0.35%, ctx=14216, majf=0, minf=31
  326. IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  327. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  328. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  329. issued rwts: total=35512,0,0,0 short=0,0,0,0 dropped=0,0,0,0
  330. latency : target=0, window=0, percentile=100.00%, depth=4
  331. readwrite: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:34:54 2019
  332. read: IOPS=37, BW=2372KiB/s (2429kB/s)(278MiB/120051msec)
  333. slat (nsec): min=0, max=405259k, avg=173230.37, stdev=7000712.24
  334. clat (nsec): min=0, max=676905k, avg=10063171.52, stdev=47810644.50
  335. lat (nsec): min=0, max=676931k, avg=10237323.34, stdev=48310601.69
  336. clat percentiles (nsec):
  337. | 1.00th=[ 716], 5.00th=[ 724], 10.00th=[ 732],
  338. | 20.00th=[ 812], 30.00th=[ 876], 40.00th=[ 980],
  339. | 50.00th=[ 1112], 60.00th=[ 1256], 70.00th=[ 1416],
  340. | 80.00th=[ 73216], 90.00th=[ 3588096], 95.00th=[ 55836672],
  341. | 99.00th=[265289728], 99.50th=[362807296], 99.90th=[583008256],
  342. | 99.95th=[599785472], 99.99th=[675282944]
  343. bw ( KiB/s): min= 0, max= 2913, per=11.14%, avg=2373.00, stdev=763.68, samples=2
  344. iops : min= 0, max= 45, avg=36.50, stdev=12.02, samples=2
  345. write: IOPS=38, BW=2471KiB/s (2531kB/s)(290MiB/120051msec)
  346. slat (nsec): min=0, max=4214.0k, avg=178472.88, stdev=241439.55
  347. clat (nsec): min=0, max=1469.7M, avg=93902576.19, stdev=154811764.20
  348. lat (nsec): min=0, max=1469.8M, avg=94082275.96, stdev=154763959.10
  349. clat percentiles (usec):
  350. | 1.00th=[ 1012], 5.00th=[ 1237], 10.00th=[ 1450],
  351. | 20.00th=[ 1991], 30.00th=[ 3097], 40.00th=[ 5604],
  352. | 50.00th=[ 22152], 60.00th=[ 70779], 70.00th=[ 95945],
  353. | 80.00th=[ 129500], 90.00th=[ 291505], 95.00th=[ 442500],
  354. | 99.00th=[ 750781], 99.50th=[ 876610], 99.90th=[1098908],
  355. | 99.95th=[1132463], 99.99th=[1468007]
  356. bw ( KiB/s): min= 0, max= 3003, per=45.41%, avg=2470.00, stdev=753.78, samples=2
  357. iops : min= 0, max= 46, avg=38.00, stdev=11.31, samples=2
  358. lat (nsec) : 750=6.72%, 1000=13.63%
  359. lat (usec) : 2=17.22%, 4=0.02%, 10=0.23%, 20=0.47%, 50=0.44%
  360. lat (usec) : 100=0.66%, 250=0.53%, 500=0.37%, 750=0.68%, 1000=1.45%
  361. lat (msec) : 2=11.21%, 4=8.28%, 10=5.91%, 20=2.37%, 50=3.56%
  362. lat (msec) : 100=10.54%, 250=9.54%, 500=4.41%, 750=1.28%, 1000=0.43%
  363. cpu : usr=0.07%, sys=0.22%, ctx=10658, majf=0, minf=31
  364. IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  365. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  366. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  367. issued rwts: total=4451,4633,0,0 short=0,0,0,0 dropped=0,0,0,0
  368. latency : target=0, window=0, percentile=100.00%, depth=4
  369.  
  370. Run status group 0 (all jobs):
  371. READ: bw=20.8MiB/s (21.8MB/s), 2372KiB/s-18.5MiB/s (2429kB/s-19.4MB/s), io=2496MiB (2618MB), run=120022-120051msec
  372. WRITE: bw=5440KiB/s (5570kB/s), 2471KiB/s-2969KiB/s (2531kB/s-3040kB/s), io=638MiB (669MB), run=120051-120063msec
  373.  
  374. Disk stats (read/write):
  375. rbd103: ios=8316/10505, merge=0/1, ticks=549459/828531, in_queue=1368379, util=14.25%
  376. <example-benchmark-fio-server-336-benchmark-tzn4v>
  377. randwrite: (groupid=0, jobs=1): err= 0: pid=11: Wed May 8 20:34:54 2019
  378. write: IOPS=37, BW=2378KiB/s (2435kB/s)(279MiB/120104msec)
  379. slat (nsec): min=0, max=340097k, avg=326004.44, stdev=6645041.68
  380. clat (nsec): min=0, max=1280.6M, avg=107678890.86, stdev=175225797.88
  381. lat (nsec): min=0, max=1280.7M, avg=108005935.42, stdev=175238266.51
  382. clat percentiles (usec):
  383. | 1.00th=[ 955], 5.00th=[ 1172], 10.00th=[ 1369],
  384. | 20.00th=[ 1844], 30.00th=[ 2868], 40.00th=[ 6128],
  385. | 50.00th=[ 28967], 60.00th=[ 76022], 70.00th=[ 104334],
  386. | 80.00th=[ 154141], 90.00th=[ 337642], 95.00th=[ 488637],
  387. | 99.00th=[ 817890], 99.50th=[ 935330], 99.90th=[1132463],
  388. | 99.95th=[1283458], 99.99th=[1283458]
  389. bw ( KiB/s): min= 0, max= 1784, per=40.93%, avg=1784.00, stdev= 0.00, samples=1
  390. iops : min= 0, max= 27, avg=27.00, stdev= 0.00, samples=1
  391. lat (usec) : 4=0.02%, 100=0.02%, 250=0.02%, 500=0.02%, 750=0.09%
  392. lat (usec) : 1000=1.68%
  393. lat (msec) : 2=20.36%, 4=13.09%, 10=8.88%, 20=3.61%, 50=5.94%
  394. lat (msec) : 100=14.64%, 250=17.60%, 500=9.35%, 750=3.27%, 1000=1.03%
  395. cpu : usr=0.04%, sys=0.17%, ctx=8470, majf=0, minf=30
  396. IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  397. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  398. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  399. issued rwts: total=0,4460,0,0 short=0,0,0,0 dropped=0,0,0,0
  400. latency : target=0, window=0, percentile=100.00%, depth=4
  401. read: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:34:54 2019
  402. read: IOPS=456, BW=28.5MiB/s (29.9MB/s)(3422MiB/120027msec)
  403. slat (nsec): min=0, max=338266k, avg=27425.06, stdev=1786682.20
  404. clat (nsec): min=0, max=999544k, avg=8748171.03, stdev=51624016.82
  405. lat (nsec): min=0, max=999606k, avg=8776407.06, stdev=51663169.95
  406. clat percentiles (nsec):
  407. | 1.00th=[ 708], 5.00th=[ 708], 10.00th=[ 716],
  408. | 20.00th=[ 724], 30.00th=[ 724], 40.00th=[ 732],
  409. | 50.00th=[ 748], 60.00th=[ 828], 70.00th=[ 884],
  410. | 80.00th=[ 1112], 90.00th=[ 1044480], 95.00th=[ 36962304],
  411. | 99.00th=[270532608], 99.50th=[417333248], 99.90th=[650117120],
  412. | 99.95th=[742391808], 99.99th=[910163968]
  413. bw ( KiB/s): min= 0, max=44025, per=93.94%, avg=29198.00, stdev=20968.54, samples=2
  414. iops : min= 0, max= 688, avg=456.00, stdev=328.10, samples=2
  415. lat (nsec) : 750=50.41%, 1000=27.05%
  416. lat (usec) : 2=8.04%, 4=0.05%, 10=1.12%, 20=0.45%, 50=0.17%
  417. lat (usec) : 100=0.10%, 250=0.06%, 500=0.31%, 750=0.70%, 1000=1.39%
  418. lat (msec) : 2=1.85%, 4=1.13%, 10=0.65%, 20=0.46%, 50=2.03%
  419. lat (msec) : 100=1.78%, 250=1.14%, 500=0.82%, 750=0.24%, 1000=0.05%
  420. cpu : usr=0.12%, sys=0.47%, ctx=13690, majf=0, minf=30
  421. IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  422. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  423. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  424. issued rwts: total=54795,0,0,0 short=0,0,0,0 dropped=0,0,0,0
  425. latency : target=0, window=0, percentile=100.00%, depth=4
  426. readwrite: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:34:54 2019
  427. read: IOPS=29, BW=1895KiB/s (1941kB/s)(222MiB/120051msec)
  428. slat (nsec): min=0, max=263179k, avg=101624.02, stdev=4414051.05
  429. clat (nsec): min=0, max=655890k, avg=9618104.13, stdev=46649931.58
  430. lat (nsec): min=0, max=655927k, avg=9720798.57, stdev=46853921.91
  431. clat percentiles (nsec):
  432. | 1.00th=[ 716], 5.00th=[ 724], 10.00th=[ 732],
  433. | 20.00th=[ 804], 30.00th=[ 876], 40.00th=[ 956],
  434. | 50.00th=[ 1096], 60.00th=[ 1240], 70.00th=[ 1416],
  435. | 80.00th=[ 35584], 90.00th=[ 2670592], 95.00th=[ 50069504],
  436. | 99.00th=[256901120], 99.50th=[367001600], 99.90th=[522190848],
  437. | 99.95th=[599785472], 99.99th=[658505728]
  438. bw ( KiB/s): min= 0, max= 2421, per=6.10%, avg=1895.50, stdev=743.17, samples=2
  439. iops : min= 0, max= 37, avg=29.00, stdev=11.31, samples=2
  440. write: IOPS=30, BW=1983KiB/s (2030kB/s)(232MiB/120051msec)
  441. slat (nsec): min=0, max=279271k, avg=268855.21, stdev=4731857.77
  442. clat (nsec): min=0, max=1280.7M, avg=120132593.80, stdev=184042868.06
  443. lat (nsec): min=0, max=1280.8M, avg=120402554.96, stdev=184195357.79
  444. clat percentiles (usec):
  445. | 1.00th=[ 988], 5.00th=[ 1221], 10.00th=[ 1450],
  446. | 20.00th=[ 2057], 30.00th=[ 3654], 40.00th=[ 10159],
  447. | 50.00th=[ 51643], 60.00th=[ 86508], 70.00th=[ 114820],
  448. | 80.00th=[ 187696], 90.00th=[ 367002], 95.00th=[ 505414],
  449. | 99.00th=[ 876610], 99.50th=[ 977273], 99.90th=[1132463],
  450. | 99.95th=[1132463], 99.99th=[1283458]
  451. bw ( KiB/s): min= 0, max= 2540, per=45.43%, avg=1980.50, stdev=791.25, samples=2
  452. iops : min= 0, max= 39, avg=30.50, stdev=12.02, samples=2
  453. lat (nsec) : 750=6.97%, 1000=14.22%
  454. lat (usec) : 2=16.17%, 4=0.08%, 10=0.36%, 20=0.89%, 50=0.76%
  455. lat (usec) : 100=0.98%, 250=0.47%, 500=0.58%, 750=0.58%, 1000=1.20%
  456. lat (msec) : 2=10.26%, 4=6.75%, 10=4.73%, 20=1.90%, 50=4.94%
  457. lat (msec) : 100=8.73%, 250=10.95%, 500=5.79%, 750=1.76%, 1000=0.80%
  458. cpu : usr=0.05%, sys=0.18%, ctx=8332, majf=0, minf=34
  459. IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  460. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  461. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  462. issued rwts: total=3556,3716,0,0 short=0,0,0,0 dropped=0,0,0,0
  463. latency : target=0, window=0, percentile=100.00%, depth=4
  464.  
  465. Run status group 0 (all jobs):
  466. READ: bw=30.4MiB/s (31.8MB/s), 1895KiB/s-28.5MiB/s (1941kB/s-29.9MB/s), io=3644MiB (3821MB), run=120027-120051msec
  467. WRITE: bw=4360KiB/s (4465kB/s), 1983KiB/s-2378KiB/s (2030kB/s-2435kB/s), io=511MiB (536MB), run=120051-120104msec
  468.  
  469. Disk stats (read/write):
  470. rbd111: ios=7834/8600, merge=0/1, ticks=524255/818436, in_queue=1334384, util=12.21%
  471. fio: inflate error -5
  472. fio: failed decompressing log
  473. fio: failed converting IO log
  474. <example-benchmark-fio-server-239-benchmark-4zbns>
  475. randwrite: (groupid=0, jobs=1): err= 0: pid=11: Wed May 8 20:34:54 2019
  476. write: IOPS=45, BW=2936KiB/s (3006kB/s)(344MiB/120067msec)
  477. slat (nsec): min=0, max=534902k, avg=347032.55, stdev=8533504.59
  478. clat (nsec): min=0, max=889389k, avg=86828735.19, stdev=144120444.95
  479. lat (nsec): min=0, max=1368.6M, avg=87177012.08, stdev=145119573.59
  480. clat percentiles (usec):
  481. | 1.00th=[ 947], 5.00th=[ 1156], 10.00th=[ 1336], 20.00th=[ 1795],
  482. | 30.00th=[ 2606], 40.00th=[ 4948], 50.00th=[ 17433], 60.00th=[ 55313],
  483. | 70.00th=[ 94897], 80.00th=[131597], 90.00th=[238027], 95.00th=[421528],
  484. | 99.00th=[700449], 99.50th=[742392], 99.90th=[826278], 99.95th=[859833],
  485. | 99.99th=[893387]
  486. bw ( KiB/s): min= 0, max= 3771, per=55.94%, avg=2935.00, stdev=1182.28, samples=2
  487. iops : min= 0, max= 58, avg=45.00, stdev=18.38, samples=2
  488. lat (usec) : 250=0.02%, 500=0.04%, 750=0.02%, 1000=1.76%
  489. lat (msec) : 2=21.13%, 4=14.35%, 10=9.14%, 20=4.60%, 50=7.45%
  490. lat (msec) : 100=13.01%, 250=18.91%, 500=6.21%, 750=2.98%, 1000=0.45%
  491. cpu : usr=0.05%, sys=0.20%, ctx=10277, majf=0, minf=30
  492. IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  493. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  494. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  495. issued rwts: total=0,5505,0,0 short=0,0,0,0 dropped=0,0,0,0
  496. latency : target=0, window=0, percentile=100.00%, depth=4
  497. read: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:34:54 2019
  498. read: IOPS=255, BW=15.9MiB/s (16.7MB/s)(1912MiB/120021msec)
  499. slat (nsec): min=0, max=533552k, avg=48056.18, stdev=3101874.04
  500. clat (nsec): min=0, max=925627k, avg=15680255.74, stdev=65921046.09
  501. lat (nsec): min=0, max=925689k, avg=15729185.26, stdev=66004552.52
  502. clat percentiles (nsec):
  503. | 1.00th=[ 708], 5.00th=[ 716], 10.00th=[ 716],
  504. | 20.00th=[ 724], 30.00th=[ 732], 40.00th=[ 764],
  505. | 50.00th=[ 868], 60.00th=[ 956], 70.00th=[ 1256],
  506. | 80.00th=[ 970752], 90.00th=[ 31850496], 95.00th=[ 73924608],
  507. | 99.00th=[375390208], 99.50th=[497025024], 99.90th=[692060160],
  508. | 99.95th=[725614592], 99.99th=[834666496]
  509. bw ( KiB/s): min= 0, max=22069, per=100.00%, avg=22069.00, stdev= 0.00, samples=1
  510. iops : min= 0, max= 345, avg=345.00, stdev= 0.00, samples=1
  511. lat (nsec) : 750=37.72%, 1000=24.33%
  512. lat (usec) : 2=12.99%, 4=0.07%, 10=0.30%, 20=0.28%, 50=0.21%
  513. lat (usec) : 100=0.03%, 250=0.04%, 500=0.32%, 750=1.28%, 1000=2.65%
  514. lat (msec) : 2=3.61%, 4=2.23%, 10=1.10%, 20=1.05%, 50=4.52%
  515. lat (msec) : 100=3.20%, 250=2.16%, 500=1.44%, 750=0.45%, 1000=0.03%
  516. cpu : usr=0.08%, sys=0.33%, ctx=14593, majf=0, minf=100
  517. IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  518. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  519. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  520. issued rwts: total=30613,0,0,0 short=0,0,0,0 dropped=0,0,0,0
  521. latency : target=0, window=0, percentile=100.00%, depth=4
  522. readwrite: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:34:54 2019
  523. read: IOPS=34, BW=2211KiB/s (2264kB/s)(259MiB/120027msec)
  524. slat (nsec): min=0, max=534280k, avg=237292.72, stdev=9787518.53
  525. clat (nsec): min=0, max=711810k, avg=12200922.95, stdev=52573178.22
  526. lat (nsec): min=0, max=711830k, avg=12439097.46, stdev=53435173.79
  527. clat percentiles (nsec):
  528. | 1.00th=[ 708], 5.00th=[ 724], 10.00th=[ 732],
  529. | 20.00th=[ 780], 30.00th=[ 884], 40.00th=[ 988],
  530. | 50.00th=[ 1128], 60.00th=[ 1256], 70.00th=[ 1528],
  531. | 80.00th=[ 238592], 90.00th=[ 20840448], 95.00th=[ 67633152],
  532. | 99.00th=[283115520], 99.50th=[358612992], 99.90th=[675282944],
  533. | 99.95th=[708837376], 99.99th=[708837376]
  534. bw ( KiB/s): min= 0, max= 2634, per=11.93%, avg=2210.00, stdev=599.63, samples=2
  535. iops : min= 0, max= 41, avg=34.00, stdev= 9.90, samples=2
  536. write: IOPS=36, BW=2313KiB/s (2368kB/s)(271MiB/120027msec)
  537. slat (nsec): min=0, max=6942.8k, avg=174147.77, stdev=263171.86
  538. clat (nsec): min=0, max=859436k, avg=98737609.79, stdev=152398197.87
  539. lat (nsec): min=0, max=859576k, avg=98912930.74, stdev=152347644.57
  540. clat percentiles (usec):
  541. | 1.00th=[ 1020], 5.00th=[ 1303], 10.00th=[ 1549], 20.00th=[ 2180],
  542. | 30.00th=[ 3687], 40.00th=[ 10159], 50.00th=[ 39584], 60.00th=[ 73925],
  543. | 70.00th=[105382], 80.00th=[141558], 90.00th=[295699], 95.00th=[463471],
  544. | 99.00th=[692061], 99.50th=[750781], 99.90th=[801113], 99.95th=[843056],
  545. | 99.99th=[859833]
  546. bw ( KiB/s): min= 0, max= 2733, per=44.05%, avg=2311.50, stdev=596.09, samples=2
  547. iops : min= 0, max= 42, avg=35.50, stdev= 9.19, samples=2
  548. lat (nsec) : 750=7.59%, 1000=12.44%
  549. lat (usec) : 2=15.02%, 4=0.05%, 10=0.29%, 20=1.04%, 50=0.85%
  550. lat (usec) : 100=0.93%, 250=0.97%, 500=0.55%, 750=0.57%, 1000=1.29%
  551. lat (msec) : 2=9.95%, 4=7.29%, 10=5.11%, 20=3.11%, 50=5.72%
  552. lat (msec) : 100=9.42%, 250=11.52%, 500=3.96%, 750=2.13%, 1000=0.22%
  553. cpu : usr=0.05%, sys=0.20%, ctx=9894, majf=0, minf=36
  554. IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  555. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  556. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  557. issued rwts: total=4146,4334,0,0 short=0,0,0,0 dropped=0,0,0,0
  558. latency : target=0, window=0, percentile=100.00%, depth=4
  559.  
  560. Run status group 0 (all jobs):
  561. READ: bw=18.1MiB/s (18.0MB/s), 2211KiB/s-15.9MiB/s (2264kB/s-16.7MB/s), io=2171MiB (2277MB), run=120021-120027msec
  562. WRITE: bw=5248KiB/s (5374kB/s), 2313KiB/s-2936KiB/s (2368kB/s-3006kB/s), io=615MiB (645MB), run=120027-120067msec
  563.  
  564. Disk stats (read/write):
  565. rbd95: ios=9683/11004, merge=0/1, ticks=566653/848675, in_queue=1404758, util=14.29%
  566. <example-benchmark-fio-server-747-benchmark-jh49p>
  567. randwrite: (groupid=0, jobs=1): err= 0: pid=11: Wed May 8 20:34:54 2019
  568. write: IOPS=41, BW=2650KiB/s (2714kB/s)(311MiB/120014msec)
  569. slat (nsec): min=0, max=444994k, avg=406213.93, stdev=8650946.30
  570. clat (nsec): min=0, max=1306.6M, avg=96754907.14, stdev=155893560.13
  571. lat (nsec): min=0, max=1306.7M, avg=97162272.04, stdev=156021316.52
  572. clat percentiles (usec):
  573. | 1.00th=[ 963], 5.00th=[ 1156], 10.00th=[ 1352],
  574. | 20.00th=[ 1811], 30.00th=[ 2704], 40.00th=[ 5473],
  575. | 50.00th=[ 35914], 60.00th=[ 78119], 70.00th=[ 100140],
  576. | 80.00th=[ 130548], 90.00th=[ 278922], 95.00th=[ 471860],
  577. | 99.00th=[ 717226], 99.50th=[ 817890], 99.90th=[1044382],
  578. | 99.95th=[1098908], 99.99th=[1300235]
  579. bw ( KiB/s): min= 0, max= 3854, per=54.00%, avg=2648.00, stdev=1705.54, samples=2
  580. iops : min= 0, max= 60, avg=41.00, stdev=26.87, samples=2
  581. lat (usec) : 500=0.04%, 750=0.02%, 1000=1.43%
  582. lat (msec) : 2=21.20%, 4=14.15%, 10=7.79%, 20=3.60%, 50=3.85%
  583. lat (msec) : 100=17.74%, 250=19.33%, 500=6.83%, 750=3.32%, 1000=0.62%
  584. cpu : usr=0.04%, sys=0.18%, ctx=9502, majf=0, minf=29
  585. IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  586. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  587. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  588. issued rwts: total=0,4967,0,0 short=0,0,0,0 dropped=0,0,0,0
  589. latency : target=0, window=0, percentile=100.00%, depth=4
  590. read: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:34:54 2019
  591. read: IOPS=565, BW=35.3MiB/s (36.0MB/s)(4233MiB/120022msec)
  592. slat (nsec): min=0, max=426774k, avg=30081.64, stdev=2287324.85
  593. clat (nsec): min=0, max=1057.1M, avg=7035019.24, stdev=47212773.76
  594. lat (nsec): min=0, max=1057.1M, avg=7065919.65, stdev=47275485.63
  595. clat percentiles (nsec):
  596. | 1.00th=[ 708], 5.00th=[ 708], 10.00th=[ 716],
  597. | 20.00th=[ 716], 30.00th=[ 724], 40.00th=[ 740],
  598. | 50.00th=[ 796], 60.00th=[ 852], 70.00th=[ 900],
  599. | 80.00th=[ 1112], 90.00th=[ 25216], 95.00th=[ 13959168],
  600. | 99.00th=[227540992], 99.50th=[396361728], 99.90th=[633339904],
  601. | 99.95th=[725614592], 99.99th=[918552576]
  602. bw ( KiB/s): min= 0, max=54834, per=94.38%, avg=36118.50, stdev=26467.71, samples=2
  603. iops : min= 0, max= 860, avg=565.50, stdev=416.49, samples=2
  604. lat (nsec) : 750=45.37%, 1000=32.21%
  605. lat (usec) : 2=6.94%, 4=0.05%, 10=2.83%, 20=2.39%, 50=0.47%
  606. lat (usec) : 100=0.13%, 250=0.04%, 500=0.09%, 750=0.50%, 1000=1.06%
  607. lat (msec) : 2=1.45%, 4=0.88%, 10=0.46%, 20=0.29%, 50=1.68%
  608. lat (msec) : 100=1.44%, 250=0.80%, 500=0.68%, 750=0.19%, 1000=0.04%
  609. cpu : usr=0.16%, sys=0.52%, ctx=13040, majf=0, minf=29
  610. IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  611. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  612. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  613. issued rwts: total=67920,0,0,0 short=0,0,0,0 dropped=0,0,0,0
  614. latency : target=0, window=0, percentile=100.00%, depth=4
  615. readwrite: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:34:54 2019
  616. read: IOPS=33, BW=2161KiB/s (2213kB/s)(253MiB/120042msec)
  617. slat (nsec): min=0, max=4021.6k, avg=26307.06, stdev=106378.46
  618. clat (nsec): min=0, max=626922k, avg=7607568.80, stdev=38180756.57
  619. lat (nsec): min=0, max=626958k, avg=7634781.88, stdev=38187137.32
  620. clat percentiles (nsec):
  621. | 1.00th=[ 708], 5.00th=[ 724], 10.00th=[ 732],
  622. | 20.00th=[ 756], 30.00th=[ 852], 40.00th=[ 948],
  623. | 50.00th=[ 1080], 60.00th=[ 1224], 70.00th=[ 1384],
  624. | 80.00th=[ 17024], 90.00th=[ 1581056], 95.00th=[ 47448064],
  625. | 99.00th=[191889408], 99.50th=[304087040], 99.90th=[480247808],
  626. | 99.95th=[484442112], 99.99th=[624951296]
  627. bw ( KiB/s): min= 0, max= 2935, per=5.65%, avg=2161.50, stdev=1093.89, samples=2
  628. iops : min= 0, max= 45, avg=33.00, stdev=16.97, samples=2
  629. write: IOPS=35, BW=2255KiB/s (2309kB/s)(264MiB/120042msec)
  630. slat (nsec): min=0, max=444199k, avg=469547.59, stdev=10265636.14
  631. clat (nsec): min=0, max=1282.2M, avg=105745971.30, stdev=159456781.24
  632. lat (nsec): min=0, max=1282.3M, avg=106216461.29, stdev=159801960.04
  633. clat percentiles (usec):
  634. | 1.00th=[ 996], 5.00th=[ 1205], 10.00th=[ 1401],
  635. | 20.00th=[ 1909], 30.00th=[ 3228], 40.00th=[ 9503],
  636. | 50.00th=[ 60031], 60.00th=[ 86508], 70.00th=[ 107480],
  637. | 80.00th=[ 139461], 90.00th=[ 320865], 95.00th=[ 484443],
  638. | 99.00th=[ 734004], 99.50th=[ 801113], 99.90th=[ 943719],
  639. | 99.95th=[1027605], 99.99th=[1283458]
  640. bw ( KiB/s): min= 0, max= 3069, per=45.93%, avg=2252.50, stdev=1154.71, samples=2
  641. iops : min= 0, max= 47, avg=34.50, stdev=17.68, samples=2
  642. lat (nsec) : 750=8.79%, 1000=13.03%
  643. lat (usec) : 2=16.53%, 4=0.04%, 10=0.41%, 20=0.77%, 50=0.72%
  644. lat (usec) : 100=0.92%, 250=0.56%, 500=0.41%, 750=0.45%, 1000=1.28%
  645. lat (msec) : 2=11.07%, 4=6.54%, 10=4.13%, 20=1.80%, 50=3.83%
  646. lat (msec) : 100=10.84%, 250=11.47%, 500=4.07%, 750=1.91%, 1000=0.43%
  647. cpu : usr=0.05%, sys=0.21%, ctx=9390, majf=0, minf=34
  648. IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  649. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  650. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  651. issued rwts: total=4056,4226,0,0 short=0,0,0,0 dropped=0,0,0,0
  652. latency : target=0, window=0, percentile=100.00%, depth=4
  653.  
  654. Run status group 0 (all jobs):
  655. READ: bw=37.4MiB/s (39.2MB/s), 2161KiB/s-35.3MiB/s (2213kB/s-36.0MB/s), io=4486MiB (4704MB), run=120022-120042msec
  656. WRITE: bw=4904KiB/s (5022kB/s), 2255KiB/s-2650KiB/s (2309kB/s-2714kB/s), io=575MiB (603MB), run=120014-120042msec
  657.  
  658. Disk stats (read/write):
  659. rbd244: ios=7107/9294, merge=0/3, ticks=515812/787469, in_queue=1295011, util=12.37%
  660. <example-benchmark-fio-server-238-benchmark-wczfn>
  661. randwrite: (groupid=0, jobs=1): err= 0: pid=11: Wed May 8 20:34:54 2019
  662. write: IOPS=39, BW=2517KiB/s (2577kB/s)(295MiB/120014msec)
  663. slat (nsec): min=0, max=341058k, avg=292693.61, stdev=5575637.86
  664. clat (nsec): min=0, max=1342.0M, avg=102021795.41, stdev=158984969.89
  665. lat (nsec): min=0, max=1342.1M, avg=102315643.79, stdev=158993907.50
  666. clat percentiles (usec):
  667. | 1.00th=[ 963], 5.00th=[ 1221], 10.00th=[ 1483],
  668. | 20.00th=[ 2073], 30.00th=[ 3392], 40.00th=[ 9372],
  669. | 50.00th=[ 45351], 60.00th=[ 78119], 70.00th=[ 106431],
  670. | 80.00th=[ 143655], 90.00th=[ 295699], 95.00th=[ 459277],
  671. | 99.00th=[ 725615], 99.50th=[ 868221], 99.90th=[1149240],
  672. | 99.95th=[1300235], 99.99th=[1333789]
  673. bw ( KiB/s): min= 0, max= 3384, per=53.24%, avg=2514.50, stdev=1229.66, samples=2
  674. iops : min= 0, max= 52, avg=38.50, stdev=19.09, samples=2
  675. lat (usec) : 50=0.02%, 250=0.02%, 500=0.06%, 750=0.02%, 1000=1.34%
  676. lat (msec) : 2=17.45%, 4=13.19%, 10=8.25%, 20=3.90%, 50=7.19%
  677. lat (msec) : 100=16.22%, 250=20.54%, 500=7.70%, 750=3.41%, 1000=0.49%
  678. cpu : usr=0.05%, sys=0.17%, ctx=8969, majf=0, minf=30
  679. IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  680. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  681. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  682. issued rwts: total=0,4717,0,0 short=0,0,0,0 dropped=0,0,0,0
  683. latency : target=0, window=0, percentile=100.00%, depth=4
  684. read: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:34:54 2019
  685. read: IOPS=379, BW=23.7MiB/s (24.9MB/s)(2846MiB/120038msec)
  686. slat (nsec): min=0, max=313498k, avg=28520.52, stdev=1565230.47
  687. clat (nsec): min=0, max=1558.7M, avg=10539488.90, stdev=56101509.52
  688. lat (nsec): min=0, max=1558.8M, avg=10568843.17, stdev=56135662.02
  689. clat percentiles (nsec):
  690. | 1.00th=[ 708], 5.00th=[ 716], 10.00th=[ 716],
  691. | 20.00th=[ 724], 30.00th=[ 724], 40.00th=[ 732],
  692. | 50.00th=[ 764], 60.00th=[ 860], 70.00th=[ 924],
  693. | 80.00th=[ 1288], 90.00th=[ 2441216], 95.00th=[ 49020928],
  694. | 99.00th=[316669952], 99.50th=[450887680], 99.90th=[666894336],
  695. | 99.95th=[734003200], 99.99th=[977272832]
  696. bw ( KiB/s): min= 0, max=34871, per=92.05%, avg=24286.50, stdev=14968.74, samples=2
  697. iops : min= 0, max= 545, avg=379.50, stdev=234.05, samples=2
  698. lat (nsec) : 750=47.29%, 1000=26.10%
  699. lat (usec) : 2=9.46%, 4=0.05%, 10=1.23%, 20=0.46%, 50=0.15%
  700. lat (usec) : 100=0.07%, 250=0.06%, 500=0.26%, 750=0.85%, 1000=1.47%
  701. lat (msec) : 2=2.07%, 4=1.36%, 10=0.71%, 20=0.54%, 50=2.97%
  702. lat (msec) : 100=2.30%, 250=1.30%, 500=0.98%, 750=0.30%, 1000=0.03%
  703. cpu : usr=0.12%, sys=0.40%, ctx=13768, majf=0, minf=42
  704. IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  705. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  706. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  707. issued rwts: total=45580,0,0,0 short=0,0,0,0 dropped=0,0,0,0
  708. latency : target=0, window=0, percentile=100.00%, depth=4
  709. readwrite: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:34:54 2019
  710. read: IOPS=32, BW=2105KiB/s (2155kB/s)(247MiB/120008msec)
  711. slat (nsec): min=0, max=267135k, avg=128561.10, stdev=4765559.54
  712. clat (nsec): min=0, max=638238k, avg=9563016.49, stdev=43149253.15
  713. lat (nsec): min=0, max=638271k, avg=9692556.97, stdev=43396123.16
  714. clat percentiles (nsec):
  715. | 1.00th=[ 716], 5.00th=[ 724], 10.00th=[ 732],
  716. | 20.00th=[ 788], 30.00th=[ 876], 40.00th=[ 996],
  717. | 50.00th=[ 1128], 60.00th=[ 1256], 70.00th=[ 1432],
  718. | 80.00th=[ 40192], 90.00th=[ 3751936], 95.00th=[ 54788096],
  719. | 99.00th=[254803968], 99.50th=[333447168], 99.90th=[497025024],
  720. | 99.95th=[509607936], 99.99th=[641728512]
  721. bw ( KiB/s): min= 0, max= 2695, per=7.97%, avg=2104.00, stdev=835.80, samples=2
  722. iops : min= 0, max= 42, avg=32.50, stdev=13.44, samples=2
  723. write: IOPS=34, BW=2206KiB/s (2259kB/s)(259MiB/120008msec)
  724. slat (nsec): min=0, max=10176k, avg=175306.89, stdev=326702.12
  725. clat (nsec): min=0, max=1839.1M, avg=107317664.14, stdev=164884666.88
  726. lat (nsec): min=0, max=1839.2M, avg=107494142.40, stdev=164838936.04
  727. clat percentiles (usec):
  728. | 1.00th=[ 963], 5.00th=[ 1221], 10.00th=[ 1467],
  729. | 20.00th=[ 2212], 30.00th=[ 3949], 40.00th=[ 12256],
  730. | 50.00th=[ 52167], 60.00th=[ 81265], 70.00th=[ 108528],
  731. | 80.00th=[ 154141], 90.00th=[ 312476], 95.00th=[ 484443],
  732. | 99.00th=[ 734004], 99.50th=[ 876610], 99.90th=[1149240],
  733. | 99.95th=[1300235], 99.99th=[1837106]
  734. bw ( KiB/s): min= 0, max= 2809, per=46.68%, avg=2204.50, stdev=854.89, samples=2
  735. iops : min= 0, max= 43, avg=34.00, stdev=12.73, samples=2
  736. lat (nsec) : 750=7.67%, 1000=12.10%
  737. lat (usec) : 2=17.50%, 4=0.09%, 10=0.22%, 20=0.95%, 50=0.78%
  738. lat (usec) : 100=1.11%, 250=0.52%, 500=0.40%, 750=0.49%, 1000=1.18%
  739. lat (msec) : 2=9.58%, 4=6.85%, 10=4.79%, 20=2.12%, 50=5.21%
  740. lat (msec) : 100=10.05%, 250=11.35%, 500=4.74%, 750=1.89%, 1000=0.30%
  741. cpu : usr=0.06%, sys=0.20%, ctx=9317, majf=0, minf=37
  742. IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  743. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  744. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  745. issued rwts: total=3948,4134,0,0 short=0,0,0,0 dropped=0,0,0,0
  746. latency : target=0, window=0, percentile=100.00%, depth=4
  747.  
  748. Run status group 0 (all jobs):
  749. READ: bw=25.8MiB/s (27.0MB/s), 2105KiB/s-23.7MiB/s (2155kB/s-24.9MB/s), io=3093MiB (3243MB), run=120008-120038msec
  750. WRITE: bw=4723KiB/s (4837kB/s), 2206KiB/s-2517KiB/s (2259kB/s-2577kB/s), io=554MiB (580MB), run=120008-120014msec
  751.  
  752. Disk stats (read/write):
  753. rbd67: ios=8366/9287, merge=0/1, ticks=537421/816981, in_queue=1345358, util=13.52%
  754. <example-benchmark-fio-server-965-benchmark-cvg7r>
  755. randwrite: (groupid=0, jobs=1): err= 0: pid=11: Wed May 8 20:34:54 2019
  756. write: IOPS=46, BW=2948KiB/s (3018kB/s)(346MiB/120088msec)
  757. slat (nsec): min=0, max=479406k, avg=350130.57, stdev=8587720.43
  758. clat (nsec): min=0, max=1103.4M, avg=86740940.20, stdev=143041701.87
  759. lat (nsec): min=0, max=1103.5M, avg=87092248.17, stdev=143156153.60
  760. clat percentiles (usec):
  761. | 1.00th=[ 955], 5.00th=[ 1156], 10.00th=[ 1352],
  762. | 20.00th=[ 1795], 30.00th=[ 2737], 40.00th=[ 5735],
  763. | 50.00th=[ 20317], 60.00th=[ 67634], 70.00th=[ 96994],
  764. | 80.00th=[ 129500], 90.00th=[ 229639], 95.00th=[ 387974],
  765. | 99.00th=[ 700449], 99.50th=[ 801113], 99.90th=[1044382],
  766. | 99.95th=[1098908], 99.99th=[1098908]
  767. bw ( KiB/s): min= 0, max= 3585, per=54.32%, avg=2947.50, stdev=901.56, samples=2
  768. iops : min= 0, max= 56, avg=46.00, stdev=14.14, samples=2
  769. lat (usec) : 4=0.02%, 500=0.02%, 750=0.02%, 1000=1.74%
  770. lat (msec) : 2=21.58%, 4=12.23%, 10=9.90%, 20=4.40%, 50=5.70%
  771. lat (msec) : 100=15.77%, 250=19.52%, 500=6.10%, 750=2.35%, 1000=0.52%
  772. cpu : usr=0.07%, sys=0.19%, ctx=10526, majf=0, minf=26
  773. IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  774. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  775. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  776. issued rwts: total=0,5528,0,0 short=0,0,0,0 dropped=0,0,0,0
  777. latency : target=0, window=0, percentile=100.00%, depth=4
  778. read: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:34:54 2019
  779. read: IOPS=291, BW=18.2MiB/s (19.1MB/s)(2187MiB/120030msec)
  780. slat (nsec): min=0, max=467384k, avg=48463.95, stdev=3199541.13
  781. clat (nsec): min=0, max=987567k, avg=13728849.56, stdev=63522045.77
  782. lat (nsec): min=0, max=987595k, avg=13778150.67, stdev=63621954.04
  783. clat percentiles (nsec):
  784. | 1.00th=[ 708], 5.00th=[ 716], 10.00th=[ 716],
  785. | 20.00th=[ 724], 30.00th=[ 732], 40.00th=[ 748],
  786. | 50.00th=[ 820], 60.00th=[ 884], 70.00th=[ 1096],
  787. | 80.00th=[ 522240], 90.00th=[ 20840448], 95.00th=[ 63700992],
  788. | 99.00th=[371195904], 99.50th=[492830720], 99.90th=[683671552],
  789. | 99.95th=[750780416], 99.99th=[851443712]
  790. bw ( KiB/s): min= 0, max=24862, per=88.70%, avg=18656.00, stdev=8776.61, samples=2
  791. iops : min= 0, max= 388, avg=291.00, stdev=137.18, samples=2
  792. lat (nsec) : 750=42.00%, 1000=25.39%
  793. lat (usec) : 2=11.31%, 4=0.08%, 10=0.45%, 20=0.12%, 50=0.11%
  794. lat (usec) : 100=0.05%, 250=0.06%, 500=0.44%, 750=1.18%, 1000=2.35%
  795. lat (msec) : 2=3.08%, 4=1.93%, 10=0.88%, 20=0.54%, 50=3.67%
  796. lat (msec) : 100=3.09%, 250=1.55%, 500=1.29%, 750=0.44%, 1000=0.05%
  797. cpu : usr=0.10%, sys=0.34%, ctx=14233, majf=0, minf=74
  798. IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  799. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  800. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  801. issued rwts: total=35008,0,0,0 short=0,0,0,0 dropped=0,0,0,0
  802. latency : target=0, window=0, percentile=100.00%, depth=4
  803. readwrite: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:34:54 2019
  804. read: IOPS=37, BW=2379KiB/s (2436kB/s)(279MiB/120022msec)
  805. slat (nsec): min=0, max=2365.6k, avg=27578.90, stdev=85435.33
  806. clat (nsec): min=0, max=660093k, avg=10380038.22, stdev=49666588.17
  807. lat (nsec): min=0, max=660371k, avg=10408597.54, stdev=49675165.24
  808. clat percentiles (nsec):
  809. | 1.00th=[ 708], 5.00th=[ 716], 10.00th=[ 724],
  810. | 20.00th=[ 756], 30.00th=[ 868], 40.00th=[ 964],
  811. | 50.00th=[ 1112], 60.00th=[ 1240], 70.00th=[ 1448],
  812. | 80.00th=[ 94720], 90.00th=[ 3555328], 95.00th=[ 53739520],
  813. | 99.00th=[278921216], 99.50th=[408944640], 99.90th=[541065216],
  814. | 99.95th=[608174080], 99.99th=[658505728]
  815. bw ( KiB/s): min= 0, max= 2746, per=11.31%, avg=2379.00, stdev=519.02, samples=2
  816. iops : min= 0, max= 42, avg=36.50, stdev= 7.78, samples=2
  817. write: IOPS=38, BW=2480KiB/s (2540kB/s)(291MiB/120022msec)
  818. slat (nsec): min=0, max=421952k, avg=357499.62, stdev=8700755.43
  819. clat (nsec): min=0, max=1108.1M, avg=93392464.33, stdev=148165561.77
  820. lat (nsec): min=0, max=1108.1M, avg=93751123.75, stdev=148302960.36
  821. clat percentiles (usec):
  822. | 1.00th=[ 1037], 5.00th=[ 1237], 10.00th=[ 1500],
  823. | 20.00th=[ 2180], 30.00th=[ 3884], 40.00th=[ 9765],
  824. | 50.00th=[ 42206], 60.00th=[ 78119], 70.00th=[ 103285],
  825. | 80.00th=[ 133694], 90.00th=[ 235930], 95.00th=[ 396362],
  826. | 99.00th=[ 734004], 99.50th=[ 868221], 99.90th=[1061159],
  827. | 99.95th=[1098908], 99.99th=[1115685]
  828. bw ( KiB/s): min= 0, max= 2824, per=45.67%, avg=2478.00, stdev=489.32, samples=2
  829. iops : min= 0, max= 44, avg=38.50, stdev= 7.78, samples=2
  830. lat (nsec) : 750=9.30%, 1000=11.41%
  831. lat (usec) : 2=15.74%, 4=0.14%, 10=0.26%, 20=0.79%, 50=0.70%
  832. lat (usec) : 100=0.91%, 250=0.64%, 500=0.68%, 750=0.66%, 1000=1.39%
  833. lat (msec) : 2=9.94%, 4=7.20%, 10=5.30%, 20=2.69%, 50=4.98%
  834. lat (msec) : 100=9.83%, 250=12.06%, 500=3.59%, 750=1.35%, 1000=0.34%
  835. cpu : usr=0.06%, sys=0.22%, ctx=10593, majf=0, minf=35
  836. IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  837. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  838. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  839. issued rwts: total=4464,4648,0,0 short=0,0,0,0 dropped=0,0,0,0
  840. latency : target=0, window=0, percentile=100.00%, depth=4
  841.  
  842. Run status group 0 (all jobs):
  843. READ: bw=20.5MiB/s (21.5MB/s), 2379KiB/s-18.2MiB/s (2436kB/s-19.1MB/s), io=2465MiB (2585MB), run=120022-120030msec
  844. WRITE: bw=5426KiB/s (5557kB/s), 2480KiB/s-2948KiB/s (2540kB/s-3018kB/s), io=636MiB (667MB), run=120022-120088msec
  845.  
  846. Disk stats (read/write):
  847. rbd329: ios=9355/10893, merge=0/1, ticks=557544/828062, in_queue=1375322, util=14.62%
  848. <example-benchmark-fio-server-714-benchmark-g6ljr>
  849. randwrite: (groupid=0, jobs=1): err= 0: pid=11: Wed May 8 20:34:55 2019
  850. write: IOPS=49, BW=3166KiB/s (3242kB/s)(371MiB/120038msec)
  851. slat (nsec): min=0, max=483547k, avg=436729.22, stdev=9798160.86
  852. clat (nsec): min=0, max=1052.2M, avg=80745876.19, stdev=137146591.70
  853. lat (nsec): min=0, max=1052.3M, avg=81183814.06, stdev=137524748.71
  854. clat percentiles (usec):
  855. | 1.00th=[ 996], 5.00th=[ 1205], 10.00th=[ 1401],
  856. | 20.00th=[ 1909], 30.00th=[ 2802], 40.00th=[ 5145],
  857. | 50.00th=[ 19268], 60.00th=[ 59507], 70.00th=[ 89654],
  858. | 80.00th=[ 116917], 90.00th=[ 208667], 95.00th=[ 404751],
  859. | 99.00th=[ 675283], 99.50th=[ 742392], 99.90th=[ 926942],
  860. | 99.95th=[1027605], 99.99th=[1052771]
  861. bw ( KiB/s): min= 0, max= 4302, per=54.75%, avg=3164.50, stdev=1608.67, samples=2
  862. iops : min= 0, max= 67, avg=49.00, stdev=25.46, samples=2
  863. lat (usec) : 4=0.03%, 100=0.02%, 750=0.07%, 1000=0.91%
  864. lat (msec) : 2=20.52%, 4=14.91%, 10=9.35%, 20=4.46%, 50=7.40%
  865. lat (msec) : 100=16.83%, 250=17.62%, 500=4.94%, 750=2.53%, 1000=0.40%
  866. cpu : usr=0.06%, sys=0.22%, ctx=11193, majf=0, minf=29
  867. IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  868. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  869. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  870. issued rwts: total=0,5936,0,0 short=0,0,0,0 dropped=0,0,0,0
  871. latency : target=0, window=0, percentile=100.00%, depth=4
  872. read: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:34:55 2019
  873. read: IOPS=300, BW=18.8MiB/s (19.7MB/s)(2254MiB/120012msec)
  874. slat (nsec): min=0, max=484975k, avg=65282.63, stdev=4143390.88
  875. clat (nsec): min=0, max=1203.3M, avg=13303433.72, stdev=63562701.91
  876. lat (nsec): min=0, max=1203.6M, avg=13369542.88, stdev=63721578.58
  877. clat percentiles (nsec):
  878. | 1.00th=[ 716], 5.00th=[ 716], 10.00th=[ 724],
  879. | 20.00th=[ 732], 30.00th=[ 740], 40.00th=[ 756],
  880. | 50.00th=[ 844], 60.00th=[ 900], 70.00th=[ 1128],
  881. | 80.00th=[ 23424], 90.00th=[ 13434880], 95.00th=[ 60555264],
  882. | 99.00th=[ 367001600], 99.50th=[ 476053504], 99.90th=[ 700448768],
  883. | 99.95th=[ 792723456], 99.99th=[1044381696]
  884. bw ( KiB/s): min= 0, max=25968, per=88.44%, avg=19235.50, stdev=9521.19, samples=2
  885. iops : min= 0, max= 406, avg=300.50, stdev=149.20, samples=2
  886. lat (nsec) : 750=38.01%, 1000=28.33%
  887. lat (usec) : 2=10.58%, 4=0.05%, 10=1.86%, 20=1.10%, 50=1.26%
  888. lat (usec) : 100=0.10%, 250=0.07%, 500=0.09%, 750=0.86%, 1000=2.03%
  889. lat (msec) : 2=2.74%, 4=1.86%, 10=0.90%, 20=0.53%, 50=3.54%
  890. lat (msec) : 100=2.80%, 250=1.62%, 500=1.25%, 750=0.37%, 1000=0.05%
  891. cpu : usr=0.09%, sys=0.35%, ctx=13433, majf=0, minf=29
  892. IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  893. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  894. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  895. issued rwts: total=36110,0,0,0 short=0,0,0,0 dropped=0,0,0,0
  896. latency : target=0, window=0, percentile=100.00%, depth=4
  897. readwrite: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:34:55 2019
  898. read: IOPS=39, BW=2518KiB/s (2578kB/s)(295MiB/120029msec)
  899. slat (nsec): min=0, max=349857k, avg=101363.58, stdev=5089683.13
  900. clat (nsec): min=0, max=1044.2M, avg=9561601.99, stdev=48388043.08
  901. lat (nsec): min=0, max=1044.2M, avg=9663946.77, stdev=48647228.37
  902. clat percentiles (nsec):
  903. | 1.00th=[ 716], 5.00th=[ 724], 10.00th=[ 732],
  904. | 20.00th=[ 780], 30.00th=[ 876], 40.00th=[ 996],
  905. | 50.00th=[ 1128], 60.00th=[ 1256], 70.00th=[ 1432],
  906. | 80.00th=[ 43776], 90.00th=[ 2899968], 95.00th=[ 51118080],
  907. | 99.00th=[ 248512512], 99.50th=[ 379584512], 99.90th=[ 566231040],
  908. | 99.95th=[ 666894336], 99.99th=[1044381696]
  909. bw ( KiB/s): min= 0, max= 3013, per=11.57%, avg=2517.50, stdev=700.74, samples=2
  910. iops : min= 0, max= 47, avg=39.00, stdev=11.31, samples=2
  911. write: IOPS=40, BW=2614KiB/s (2677kB/s)(306MiB/120029msec)
  912. slat (nsec): min=0, max=764714k, avg=504417.57, stdev=13821816.46
  913. clat (nsec): min=0, max=1313.0M, avg=88688665.47, stdev=146690691.98
  914. lat (nsec): min=0, max=1313.1M, avg=89194015.61, stdev=147209124.43
  915. clat percentiles (usec):
  916. | 1.00th=[ 1037], 5.00th=[ 1254], 10.00th=[ 1516],
  917. | 20.00th=[ 2180], 30.00th=[ 3654], 40.00th=[ 8586],
  918. | 50.00th=[ 33817], 60.00th=[ 67634], 70.00th=[ 93848],
  919. | 80.00th=[ 123208], 90.00th=[ 227541], 95.00th=[ 438305],
  920. | 99.00th=[ 683672], 99.50th=[ 775947], 99.90th=[1044382],
  921. | 99.95th=[1061159], 99.99th=[1317012]
  922. bw ( KiB/s): min= 0, max= 3117, per=45.20%, avg=2612.50, stdev=713.47, samples=2
  923. iops : min= 0, max= 48, avg=40.00, stdev=11.31, samples=2
  924. lat (nsec) : 750=8.05%, 1000=11.86%
  925. lat (usec) : 2=17.28%, 4=0.05%, 10=0.37%, 20=1.02%, 50=0.76%
  926. lat (usec) : 100=0.80%, 250=0.45%, 500=0.43%, 750=0.39%, 1000=1.44%
  927. lat (msec) : 2=9.94%, 4=7.65%, 10=5.33%, 20=2.76%, 50=5.62%
  928. lat (msec) : 100=10.70%, 250=10.22%, 500=2.99%, 750=1.54%, 1000=0.20%
  929. cpu : usr=0.06%, sys=0.25%, ctx=11131, majf=0, minf=36
  930. IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  931. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  932. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  933. issued rwts: total=4725,4900,0,0 short=0,0,0,0 dropped=0,0,0,0
  934. latency : target=0, window=0, percentile=100.00%, depth=4
  935.  
  936. Run status group 0 (all jobs):
  937. READ: bw=21.2MiB/s (22.3MB/s), 2518KiB/s-18.8MiB/s (2578kB/s-19.7MB/s), io=2550MiB (2673MB), run=120012-120029msec
  938. WRITE: bw=5781KiB/s (5919kB/s), 2614KiB/s-3166KiB/s (2677kB/s-3242kB/s), io=678MiB (711MB), run=120029-120038msec
  939.  
  940. Disk stats (read/write):
  941. rbd292: ios=7536/10929, merge=0/2, ticks=534002/809056, in_queue=1333704, util=13.66%
  942. <example-benchmark-fio-server-386-benchmark-nzznc>
  943. randwrite: (groupid=0, jobs=1): err= 0: pid=11: Wed May 8 20:34:55 2019
  944. write: IOPS=40, BW=2596KiB/s (2658kB/s)(304MiB/120024msec)
  945. slat (nsec): min=0, max=579931k, avg=462348.78, stdev=11856183.51
  946. clat (nsec): min=0, max=1090.2M, avg=98730771.42, stdev=156908561.14
  947. lat (nsec): min=0, max=1289.8M, avg=99194171.74, stdev=157794900.50
  948. clat percentiles (usec):
  949. | 1.00th=[ 1012], 5.00th=[ 1221], 10.00th=[ 1418],
  950. | 20.00th=[ 1991], 30.00th=[ 3228], 40.00th=[ 7242],
  951. | 50.00th=[ 30802], 60.00th=[ 72877], 70.00th=[ 100140],
  952. | 80.00th=[ 131597], 90.00th=[ 316670], 95.00th=[ 484443],
  953. | 99.00th=[ 708838], 99.50th=[ 742392], 99.90th=[ 826278],
  954. | 99.95th=[ 910164], 99.99th=[1082131]
  955. bw ( KiB/s): min= 0, max= 3369, per=55.35%, avg=2593.50, stdev=1096.72, samples=2
  956. iops : min= 0, max= 52, avg=40.00, stdev=16.97, samples=2
  957. lat (usec) : 250=0.02%, 750=0.04%, 1000=0.84%
  958. lat (msec) : 2=19.24%, 4=13.86%, 10=8.49%, 20=4.61%, 50=6.41%
  959. lat (msec) : 100=16.55%, 250=17.58%, 500=7.77%, 750=4.17%, 1000=0.43%
  960. cpu : usr=0.05%, sys=0.17%, ctx=9233, majf=0, minf=30
  961. IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  962. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  963. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  964. issued rwts: total=0,4864,0,0 short=0,0,0,0 dropped=0,0,0,0
  965. latency : target=0, window=0, percentile=100.00%, depth=4
  966. read: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:34:55 2019
  967. read: IOPS=290, BW=18.1MiB/s (18.0MB/s)(2174MiB/120020msec)
  968. slat (nsec): min=0, max=689525k, avg=62199.26, stdev=4622504.13
  969. clat (nsec): min=0, max=1083.4M, avg=13794212.01, stdev=64703199.17
  970. lat (nsec): min=0, max=1083.5M, avg=13857268.54, stdev=64873106.38
  971. clat percentiles (nsec):
  972. | 1.00th=[ 708], 5.00th=[ 716], 10.00th=[ 716],
  973. | 20.00th=[ 724], 30.00th=[ 740], 40.00th=[ 788],
  974. | 50.00th=[ 844], 60.00th=[ 908], 70.00th=[ 1128],
  975. | 80.00th=[ 8896], 90.00th=[ 15007744], 95.00th=[ 62128128],
  976. | 99.00th=[ 387973120], 99.50th=[ 492830720], 99.90th=[ 692060160],
  977. | 99.95th=[ 734003200], 99.99th=[1019215872]
  978. bw ( KiB/s): min= 0, max=22016, per=90.23%, avg=18550.50, stdev=4900.96, samples=2
  979. iops : min= 0, max= 344, avg=289.50, stdev=77.07, samples=2
  980. lat (nsec) : 750=33.40%, 1000=32.84%
  981. lat (usec) : 2=11.88%, 4=0.05%, 10=2.36%, 20=0.94%, 50=0.36%
  982. lat (usec) : 100=0.11%, 250=0.05%, 500=0.18%, 750=0.86%, 1000=1.88%
  983. lat (msec) : 2=2.49%, 4=1.47%, 10=0.79%, 20=0.64%, 50=3.51%
  984. lat (msec) : 100=2.72%, 250=1.69%, 500=1.29%, 750=0.44%, 1000=0.03%
  985. cpu : usr=0.09%, sys=0.32%, ctx=12530, majf=0, minf=30
  986. IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  987. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  988. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  989. issued rwts: total=34807,0,0,0 short=0,0,0,0 dropped=0,0,0,0
  990. latency : target=0, window=0, percentile=100.00%, depth=4
  991. readwrite: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:34:55 2019
  992. read: IOPS=31, BW=2009KiB/s (2057kB/s)(236MiB/120021msec)
  993. slat (nsec): min=0, max=427153k, avg=136375.18, stdev=6958530.80
  994. clat (nsec): min=0, max=702092k, avg=9087273.02, stdev=46342012.02
  995. lat (nsec): min=0, max=702124k, avg=9224584.28, stdev=46846870.09
  996. clat percentiles (nsec):
  997. | 1.00th=[ 716], 5.00th=[ 724], 10.00th=[ 732],
  998. | 20.00th=[ 780], 30.00th=[ 868], 40.00th=[ 948],
  999. | 50.00th=[ 1080], 60.00th=[ 1240], 70.00th=[ 1384],
  1000. | 80.00th=[ 31872], 90.00th=[ 1777664], 95.00th=[ 46399488],
  1001. | 99.00th=[261095424], 99.50th=[367001600], 99.90th=[549453824],
  1002. | 99.95th=[583008256], 99.99th=[700448768]
  1003. bw ( KiB/s): min= 0, max= 2533, per=9.77%, avg=2008.50, stdev=741.76, samples=2
  1004. iops : min= 0, max= 39, avg=31.00, stdev=11.31, samples=2
  1005. write: IOPS=32, BW=2091KiB/s (2141kB/s)(245MiB/120021msec)
  1006. slat (nsec): min=0, max=701141k, avg=429937.85, stdev=12610360.16
  1007. clat (nsec): min=0, max=1018.3M, avg=113826493.31, stdev=166482932.38
  1008. lat (nsec): min=0, max=1018.4M, avg=114257403.10, stdev=166748154.43
  1009. clat percentiles (usec):
  1010. | 1.00th=[ 1057], 5.00th=[ 1336], 10.00th=[ 1631],
  1011. | 20.00th=[ 2442], 30.00th=[ 4555], 40.00th=[ 17171],
  1012. | 50.00th=[ 56886], 60.00th=[ 86508], 70.00th=[ 109577],
  1013. | 80.00th=[ 156238], 90.00th=[ 358613], 95.00th=[ 513803],
  1014. | 99.00th=[ 725615], 99.50th=[ 759170], 99.90th=[ 809501],
  1015. | 99.95th=[1019216], 99.99th=[1019216]
  1016. bw ( KiB/s): min= 0, max= 2647, per=44.58%, avg=2089.00, stdev=789.13, samples=2
  1017. iops : min= 0, max= 41, avg=32.00, stdev=12.73, samples=2
  1018. lat (nsec) : 750=7.31%, 1000=13.84%
  1019. lat (usec) : 2=16.54%, 4=0.08%, 10=0.25%, 20=0.98%, 50=0.70%
  1020. lat (usec) : 100=1.03%, 250=0.57%, 500=0.57%, 750=0.57%, 1000=1.14%
  1021. lat (msec) : 2=8.69%, 4=6.87%, 10=4.38%, 20=2.67%, 50=4.96%
  1022. lat (msec) : 100=10.14%, 250=10.73%, 500=5.18%, 750=2.52%, 1000=0.29%
  1023. cpu : usr=0.04%, sys=0.19%, ctx=8800, majf=0, minf=34
  1024. IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  1025. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  1026. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  1027. issued rwts: total=3768,3918,0,0 short=0,0,0,0 dropped=0,0,0,0
  1028. latency : target=0, window=0, percentile=100.00%, depth=4
  1029.  
  1030. Run status group 0 (all jobs):
  1031. READ: bw=20.1MiB/s (21.1MB/s), 2009KiB/s-18.1MiB/s (2057kB/s-18.0MB/s), io=2410MiB (2527MB), run=120020-120021msec
  1032. WRITE: bw=4687KiB/s (4799kB/s), 2091KiB/s-2596KiB/s (2141kB/s-2658kB/s), io=549MiB (576MB), run=120021-120024msec
  1033.  
  1034. Disk stats (read/write):
  1035. rbd113: ios=7017/9006, merge=0/1, ticks=547807/860241, in_queue=1399971, util=11.72%
  1036. <example-benchmark-fio-server-616-benchmark-vm6nt>
  1037. randwrite: (groupid=0, jobs=1): err= 0: pid=11: Wed May 8 20:34:55 2019
  1038. write: IOPS=42, BW=2740KiB/s (2806kB/s)(321MiB/120064msec)
  1039. slat (nsec): min=0, max=522799k, avg=448436.58, stdev=10743519.39
  1040. clat (nsec): min=0, max=1524.9M, avg=93344224.78, stdev=161360206.70
  1041. lat (nsec): min=0, max=1524.9M, avg=93793721.83, stdev=161971351.87
  1042. clat percentiles (usec):
  1043. | 1.00th=[ 938], 5.00th=[ 1156], 10.00th=[ 1369],
  1044. | 20.00th=[ 1876], 30.00th=[ 2868], 40.00th=[ 5145],
  1045. | 50.00th=[ 19268], 60.00th=[ 60031], 70.00th=[ 87557],
  1046. | 80.00th=[ 125305], 90.00th=[ 304088], 95.00th=[ 459277],
  1047. | 99.00th=[ 801113], 99.50th=[ 935330], 99.90th=[1149240],
  1048. | 99.95th=[1233126], 99.99th=[1518339]
  1049. bw ( KiB/s): min= 0, max= 1793, per=34.93%, avg=1793.00, stdev= 0.00, samples=1
  1050. iops : min= 0, max= 28, avg=28.00, stdev= 0.00, samples=1
  1051. lat (usec) : 500=0.02%, 750=0.04%, 1000=1.87%
  1052. lat (msec) : 2=19.66%, 4=14.99%, 10=9.93%, 20=3.89%, 50=6.64%
  1053. lat (msec) : 100=17.05%, 250=14.05%, 500=8.00%, 750=2.76%, 1000=0.91%
  1054. cpu : usr=0.04%, sys=0.20%, ctx=9807, majf=0, minf=32
  1055. IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  1056. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  1057. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  1058. issued rwts: total=0,5138,0,0 short=0,0,0,0 dropped=0,0,0,0
  1059. latency : target=0, window=0, percentile=100.00%, depth=4
  1060. read: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:34:55 2019
  1061. read: IOPS=383, BW=23.9MiB/s (25.1MB/s)(2872MiB/120013msec)
  1062. slat (nsec): min=0, max=471122k, avg=46847.59, stdev=3381194.72
  1063. clat (nsec): min=0, max=1407.2M, avg=10409463.13, stdev=55502628.48
  1064. lat (nsec): min=0, max=1407.2M, avg=10457142.71, stdev=55610776.23
  1065. clat percentiles (nsec):
  1066. | 1.00th=[ 708], 5.00th=[ 708], 10.00th=[ 716],
  1067. | 20.00th=[ 724], 30.00th=[ 732], 40.00th=[ 740],
  1068. | 50.00th=[ 812], 60.00th=[ 876], 70.00th=[ 972],
  1069. | 80.00th=[ 1432], 90.00th=[ 3424256], 95.00th=[ 50593792],
  1070. | 99.00th=[ 291504128], 99.50th=[ 450887680], 99.90th=[ 675282944],
  1071. | 99.95th=[ 767557632], 99.99th=[1098907648]
  1072. bw ( KiB/s): min= 0, max=34035, per=100.00%, avg=34035.00, stdev= 0.00, samples=1
  1073. iops : min= 0, max= 532, avg=532.00, stdev= 0.00, samples=1
  1074. lat (nsec) : 750=43.64%, 1000=27.38%
  1075. lat (usec) : 2=10.12%, 4=0.03%, 10=0.80%, 20=0.30%, 50=0.14%
  1076. lat (usec) : 100=0.06%, 250=0.06%, 500=0.22%, 750=0.88%, 1000=2.09%
  1077. lat (msec) : 2=2.76%, 4=1.76%, 10=0.97%, 20=0.57%, 50=3.17%
  1078. lat (msec) : 100=2.66%, 250=1.23%, 500=0.83%, 750=0.29%, 1000=0.04%
  1079. cpu : usr=0.13%, sys=0.41%, ctx=16042, majf=0, minf=31
  1080. IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  1081. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  1082. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  1083. issued rwts: total=45981,0,0,0 short=0,0,0,0 dropped=0,0,0,0
  1084. latency : target=0, window=0, percentile=100.00%, depth=4
  1085. readwrite: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:34:55 2019
  1086. read: IOPS=35, BW=2297KiB/s (2352kB/s)(269MiB/120051msec)
  1087. slat (nsec): min=0, max=3687.3k, avg=25139.08, stdev=91477.78
  1088. clat (nsec): min=0, max=1038.4M, avg=7695728.63, stdev=45093642.98
  1089. lat (nsec): min=0, max=1038.6M, avg=7721793.49, stdev=45105435.26
  1090. clat percentiles (nsec):
  1091. | 1.00th=[ 708], 5.00th=[ 716], 10.00th=[ 732],
  1092. | 20.00th=[ 756], 30.00th=[ 852], 40.00th=[ 948],
  1093. | 50.00th=[ 1112], 60.00th=[ 1240], 70.00th=[ 1384],
  1094. | 80.00th=[ 15808], 90.00th=[ 1777664], 95.00th=[ 41156608],
  1095. | 99.00th=[ 183500800], 99.50th=[ 304087040], 99.90th=[ 666894336],
  1096. | 99.95th=[ 725614592], 99.99th=[1035993088]
  1097. bw ( KiB/s): min= 0, max= 2781, per=8.57%, avg=2297.00, stdev=684.48, samples=2
  1098. iops : min= 0, max= 43, avg=35.50, stdev=10.61, samples=2
  1099. write: IOPS=37, BW=2393KiB/s (2451kB/s)(281MiB/120051msec)
  1100. slat (nsec): min=0, max=521733k, avg=510207.12, stdev=12528343.03
  1101. clat (nsec): min=0, max=1456.0M, avg=99230619.88, stdev=167653400.45
  1102. lat (nsec): min=0, max=1456.1M, avg=99741774.10, stdev=168293190.74
  1103. clat percentiles (usec):
  1104. | 1.00th=[ 971], 5.00th=[ 1254], 10.00th=[ 1500],
  1105. | 20.00th=[ 2089], 30.00th=[ 3326], 40.00th=[ 6456],
  1106. | 50.00th=[ 23987], 60.00th=[ 61080], 70.00th=[ 88605],
  1107. | 80.00th=[ 131597], 90.00th=[ 341836], 95.00th=[ 476054],
  1108. | 99.00th=[ 801113], 99.50th=[ 935330], 99.90th=[1216349],
  1109. | 99.95th=[1367344], 99.99th=[1451230]
  1110. bw ( KiB/s): min= 0, max= 2873, per=46.60%, avg=2392.00, stdev=680.24, samples=2
  1111. iops : min= 0, max= 44, avg=36.50, stdev=10.61, samples=2
  1112. lat (nsec) : 750=9.00%, 1000=12.22%
  1113. lat (usec) : 2=17.29%, 4=0.08%, 10=0.19%, 20=0.72%, 50=0.70%
  1114. lat (usec) : 100=0.61%, 250=0.48%, 500=0.49%, 750=0.44%, 1000=1.53%
  1115. lat (msec) : 2=10.24%, 4=7.73%, 10=6.29%, 20=2.58%, 50=5.10%
  1116. lat (msec) : 100=9.71%, 250=7.68%, 500=4.64%, 750=1.66%, 1000=0.50%
  1117. cpu : usr=0.05%, sys=0.21%, ctx=10071, majf=0, minf=38
  1118. IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  1119. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  1120. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  1121. issued rwts: total=4311,4486,0,0 short=0,0,0,0 dropped=0,0,0,0
  1122. latency : target=0, window=0, percentile=100.00%, depth=4
  1123.  
  1124. Run status group 0 (all jobs):
  1125. READ: bw=26.2MiB/s (27.4MB/s), 2297KiB/s-23.9MiB/s (2352kB/s-25.1MB/s), io=3141MiB (3294MB), run=120013-120051msec
  1126. WRITE: bw=5133KiB/s (5256kB/s), 2393KiB/s-2740KiB/s (2451kB/s-2806kB/s), io=602MiB (631MB), run=120051-120064msec
  1127.  
  1128. Disk stats (read/write):
  1129. rbd226: ios=8971/10050, merge=0/2, ticks=526494/837841, in_queue=1354664, util=14.05%
  1130. <example-benchmark-fio-server-810-benchmark-s6m2k>
  1131. randwrite: (groupid=0, jobs=1): err= 0: pid=11: Wed May 8 20:34:55 2019
  1132. write: IOPS=45, BW=2945KiB/s (3016kB/s)(345MiB/120041msec)
  1133. slat (nsec): min=0, max=573788k, avg=367710.09, stdev=9878806.33
  1134. clat (nsec): min=0, max=1214.2M, avg=86620290.25, stdev=149814104.02
  1135. lat (nsec): min=0, max=1214.3M, avg=86989216.49, stdev=149987111.38
  1136. clat percentiles (usec):
  1137. | 1.00th=[ 996], 5.00th=[ 1221], 10.00th=[ 1418],
  1138. | 20.00th=[ 1909], 30.00th=[ 2900], 40.00th=[ 5211],
  1139. | 50.00th=[ 13435], 60.00th=[ 47973], 70.00th=[ 85459],
  1140. | 80.00th=[ 120062], 90.00th=[ 283116], 95.00th=[ 438305],
  1141. | 99.00th=[ 708838], 99.50th=[ 817890], 99.90th=[ 952108],
  1142. | 99.95th=[1044382], 99.99th=[1216349]
  1143. bw ( KiB/s): min= 0, max= 3560, per=55.22%, avg=2944.00, stdev=871.16, samples=2
  1144. iops : min= 0, max= 55, avg=45.50, stdev=13.44, samples=2
  1145. lat (usec) : 4=0.02%, 750=0.04%, 1000=1.00%
  1146. lat (msec) : 2=20.12%, 4=14.45%, 10=12.19%, 20=5.11%, 50=7.63%
  1147. lat (msec) : 100=14.69%, 250=13.80%, 500=7.66%, 750=2.54%, 1000=0.76%
  1148. cpu : usr=0.06%, sys=0.19%, ctx=10456, majf=0, minf=30
  1149. IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  1150. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  1151. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  1152. issued rwts: total=0,5521,0,0 short=0,0,0,0 dropped=0,0,0,0
  1153. latency : target=0, window=0, percentile=100.00%, depth=4
  1154. read: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:34:55 2019
  1155. read: IOPS=294, BW=18.4MiB/s (19.3MB/s)(2204MiB/120030msec)
  1156. slat (nsec): min=0, max=310412k, avg=40472.37, stdev=2303449.06
  1157. clat (nsec): min=0, max=1217.2M, avg=13618848.09, stdev=62825076.11
  1158. lat (nsec): min=0, max=1217.3M, avg=13660203.59, stdev=62875748.57
  1159. clat percentiles (nsec):
  1160. | 1.00th=[ 708], 5.00th=[ 708], 10.00th=[ 716],
  1161. | 20.00th=[ 724], 30.00th=[ 732], 40.00th=[ 764],
  1162. | 50.00th=[ 836], 60.00th=[ 892], 70.00th=[ 1096],
  1163. | 80.00th=[ 561152], 90.00th=[ 23461888], 95.00th=[ 63176704],
  1164. | 99.00th=[ 354418688], 99.50th=[ 476053504], 99.90th=[ 700448768],
  1165. | 99.95th=[ 759169024], 99.99th=[1061158912]
  1166. bw ( KiB/s): min= 0, max=24758, per=89.20%, avg=18807.00, stdev=8415.98, samples=2
  1167. iops : min= 0, max= 387, avg=293.50, stdev=132.23, samples=2
  1168. lat (nsec) : 750=35.79%, 1000=31.21%
  1169. lat (usec) : 2=11.36%, 4=0.04%, 10=0.62%, 20=0.46%, 50=0.17%
  1170. lat (usec) : 100=0.10%, 250=0.05%, 500=0.15%, 750=1.01%, 1000=2.07%
  1171. lat (msec) : 2=3.01%, 4=1.86%, 10=0.96%, 20=0.78%, 50=3.94%
  1172. lat (msec) : 100=3.04%, 250=1.73%, 500=1.21%, 750=0.38%, 1000=0.05%
  1173. cpu : usr=0.09%, sys=0.34%, ctx=14244, majf=0, minf=29
  1174. IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  1175. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  1176. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  1177. issued rwts: total=35289,0,0,0 short=0,0,0,0 dropped=0,0,0,0
  1178. latency : target=0, window=0, percentile=100.00%, depth=4
  1179. readwrite: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:34:55 2019
  1180. read: IOPS=35, BW=2281KiB/s (2336kB/s)(267MiB/120036msec)
  1181. slat (nsec): min=0, max=378161k, avg=188024.60, stdev=7512404.09
  1182. clat (nsec): min=0, max=739179k, avg=10245570.83, stdev=48682354.00
  1183. lat (nsec): min=0, max=739211k, avg=10434521.92, stdev=49233760.49
  1184. clat percentiles (nsec):
  1185. | 1.00th=[ 708], 5.00th=[ 716], 10.00th=[ 732],
  1186. | 20.00th=[ 772], 30.00th=[ 868], 40.00th=[ 980],
  1187. | 50.00th=[ 1112], 60.00th=[ 1240], 70.00th=[ 1416],
  1188. | 80.00th=[ 56576], 90.00th=[ 4014080], 95.00th=[ 52690944],
  1189. | 99.00th=[283115520], 99.50th=[354418688], 99.90th=[557842432],
  1190. | 99.95th=[616562688], 99.99th=[742391808]
  1191. bw ( KiB/s): min= 0, max= 2588, per=10.82%, avg=2281.50, stdev=433.46, samples=2
  1192. iops : min= 0, max= 40, avg=35.00, stdev= 7.07, samples=2
  1193. write: IOPS=37, BW=2386KiB/s (2444kB/s)(280MiB/120036msec)
  1194. slat (nsec): min=0, max=6032.8k, avg=173025.17, stdev=261709.00
  1195. clat (nsec): min=0, max=1270.0M, avg=97171527.70, stdev=157884329.78
  1196. lat (nsec): min=0, max=1270.3M, avg=97346023.25, stdev=157836529.25
  1197. clat percentiles (usec):
  1198. | 1.00th=[ 1029], 5.00th=[ 1270], 10.00th=[ 1532],
  1199. | 20.00th=[ 2180], 30.00th=[ 3654], 40.00th=[ 7701],
  1200. | 50.00th=[ 24773], 60.00th=[ 64750], 70.00th=[ 94897],
  1201. | 80.00th=[ 135267], 90.00th=[ 312476], 95.00th=[ 476054],
  1202. | 99.00th=[ 742392], 99.50th=[ 843056], 99.90th=[ 935330],
  1203. | 99.95th=[ 952108], 99.99th=[1266680]
  1204. bw ( KiB/s): min= 0, max= 2692, per=44.73%, avg=2384.50, stdev=434.87, samples=2
  1205. iops : min= 0, max= 42, avg=37.00, stdev= 7.07, samples=2
  1206. lat (nsec) : 750=8.44%, 1000=11.81%
  1207. lat (usec) : 2=17.02%, 4=0.05%, 10=0.14%, 20=0.80%, 50=0.69%
  1208. lat (usec) : 100=0.74%, 250=0.49%, 500=0.41%, 750=0.56%, 1000=1.36%
  1209. lat (msec) : 2=9.93%, 4=7.71%, 10=5.94%, 20=3.15%, 50=5.67%
  1210. lat (msec) : 100=9.30%, 250=8.91%, 500=4.67%, 750=1.75%, 1000=0.47%
  1211. cpu : usr=0.06%, sys=0.21%, ctx=10250, majf=0, minf=34
  1212. IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  1213. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  1214. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  1215. issued rwts: total=4280,4473,0,0 short=0,0,0,0 dropped=0,0,0,0
  1216. latency : target=0, window=0, percentile=100.00%, depth=4
  1217.  
  1218. Run status group 0 (all jobs):
  1219. READ: bw=20.6MiB/s (21.6MB/s), 2281KiB/s-18.4MiB/s (2336kB/s-19.3MB/s), io=2472MiB (2592MB), run=120030-120036msec
  1220. WRITE: bw=5332KiB/s (5459kB/s), 2386KiB/s-2945KiB/s (2444kB/s-3016kB/s), io=625MiB (655MB), run=120036-120041msec
  1221.  
  1222. Disk stats (read/write):
  1223. rbd188: ios=7908/10343, merge=0/1, ticks=544904/835140, in_queue=1370788, util=13.49%
  1224. <example-benchmark-fio-server-363-benchmark-v4496>
  1225. randwrite: (groupid=0, jobs=1): err= 0: pid=11: Wed May 8 20:34:55 2019
  1226. write: IOPS=41, BW=2646KiB/s (2709kB/s)(310MiB/120098msec)
  1227. slat (nsec): min=0, max=483563k, avg=335624.44, stdev=7666597.67
  1228. clat (nsec): min=0, max=1102.5M, avg=96744121.92, stdev=159436843.91
  1229. lat (nsec): min=0, max=1102.5M, avg=97080769.02, stdev=159472833.05
  1230. clat percentiles (usec):
  1231. | 1.00th=[ 938], 5.00th=[ 1106], 10.00th=[ 1303],
  1232. | 20.00th=[ 1663], 30.00th=[ 2343], 40.00th=[ 3818],
  1233. | 50.00th=[ 23200], 60.00th=[ 72877], 70.00th=[ 103285],
  1234. | 80.00th=[ 129500], 90.00th=[ 308282], 95.00th=[ 488637],
  1235. | 99.00th=[ 742392], 99.50th=[ 801113], 99.90th=[ 901776],
  1236. | 99.95th=[ 935330], 99.99th=[1098908]
  1237. bw ( KiB/s): min= 0, max= 4098, per=55.36%, avg=2645.50, stdev=2054.15, samples=2
  1238. iops : min= 0, max= 64, avg=41.00, stdev=32.53, samples=2
  1239. lat (usec) : 2=0.02%, 4=0.04%, 500=0.02%, 750=0.08%, 1000=1.67%
  1240. lat (msec) : 2=24.00%, 4=14.89%, 10=6.15%, 20=2.56%, 50=4.90%
  1241. lat (msec) : 100=14.53%, 250=19.87%, 500=6.59%, 750=3.89%, 1000=0.81%
  1242. cpu : usr=0.05%, sys=0.17%, ctx=9368, majf=0, minf=27
  1243. IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  1244. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  1245. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  1246. issued rwts: total=0,4962,0,0 short=0,0,0,0 dropped=0,0,0,0
  1247. latency : target=0, window=0, percentile=100.00%, depth=4
  1248. read: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:34:55 2019
  1249. read: IOPS=323, BW=20.2MiB/s (21.2MB/s)(2424MiB/120013msec)
  1250. slat (nsec): min=0, max=477001k, avg=36573.14, stdev=2637904.02
  1251. clat (nsec): min=0, max=940724k, avg=12367744.96, stdev=60911558.88
  1252. lat (nsec): min=0, max=941214k, avg=12405146.29, stdev=60976937.14
  1253. clat percentiles (nsec):
  1254. | 1.00th=[ 708], 5.00th=[ 708], 10.00th=[ 716],
  1255. | 20.00th=[ 724], 30.00th=[ 732], 40.00th=[ 740],
  1256. | 50.00th=[ 796], 60.00th=[ 860], 70.00th=[ 940],
  1257. | 80.00th=[ 1304], 90.00th=[ 6127616], 95.00th=[ 55836672],
  1258. | 99.00th=[367001600], 99.50th=[471859200], 99.90th=[700448768],
  1259. | 99.95th=[767557632], 99.99th=[843055104]
  1260. bw ( KiB/s): min= 0, max=25084, per=91.05%, avg=20685.00, stdev=6221.13, samples=2
  1261. iops : min= 0, max= 392, avg=323.00, stdev=97.58, samples=2
  1262. lat (nsec) : 750=44.89%, 1000=27.73%
  1263. lat (usec) : 2=9.91%, 4=0.08%, 10=0.35%, 20=0.21%, 50=0.19%
  1264. lat (usec) : 100=0.06%, 250=0.04%, 500=0.16%, 750=0.79%, 1000=1.59%
  1265. lat (msec) : 2=2.21%, 4=1.37%, 10=0.80%, 20=0.56%, 50=3.41%
  1266. lat (msec) : 100=2.67%, 250=1.38%, 500=1.21%, 750=0.34%, 1000=0.06%
  1267. cpu : usr=0.10%, sys=0.35%, ctx=12840, majf=0, minf=29
  1268. IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  1269. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  1270. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  1271. issued rwts: total=38815,0,0,0 short=0,0,0,0 dropped=0,0,0,0
  1272. latency : target=0, window=0, percentile=100.00%, depth=4
  1273. readwrite: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:34:55 2019
  1274. read: IOPS=31, BW=2033KiB/s (2082kB/s)(238MiB/120015msec)
  1275. slat (nsec): min=0, max=436760k, avg=141784.50, stdev=7073229.93
  1276. clat (nsec): min=0, max=698805k, avg=9375440.61, stdev=44524086.18
  1277. lat (nsec): min=0, max=698827k, avg=9518114.96, stdev=45063482.36
  1278. clat percentiles (nsec):
  1279. | 1.00th=[ 716], 5.00th=[ 732], 10.00th=[ 748],
  1280. | 20.00th=[ 820], 30.00th=[ 884], 40.00th=[ 988],
  1281. | 50.00th=[ 1112], 60.00th=[ 1224], 70.00th=[ 1416],
  1282. | 80.00th=[ 67072], 90.00th=[ 5537792], 95.00th=[ 54788096],
  1283. | 99.00th=[221249536], 99.50th=[325058560], 99.90th=[599785472],
  1284. | 99.95th=[683671552], 99.99th=[700448768]
  1285. bw ( KiB/s): min= 0, max= 3070, per=8.94%, avg=2032.00, stdev=1467.95, samples=2
  1286. iops : min= 0, max= 47, avg=31.00, stdev=22.63, samples=2
  1287. write: IOPS=33, BW=2135KiB/s (2186kB/s)(250MiB/120015msec)
  1288. slat (nsec): min=0, max=192477k, avg=220163.16, stdev=3048609.65
  1289. clat (nsec): min=0, max=1102.4M, avg=110902683.24, stdev=169976312.05
  1290. lat (nsec): min=0, max=1102.6M, avg=111123871.68, stdev=169947686.95
  1291. clat percentiles (usec):
  1292. | 1.00th=[ 963], 5.00th=[ 1156], 10.00th=[ 1369],
  1293. | 20.00th=[ 1926], 30.00th=[ 3032], 40.00th=[ 8586],
  1294. | 50.00th=[ 54264], 60.00th=[ 86508], 70.00th=[ 111674],
  1295. | 80.00th=[ 141558], 90.00th=[ 350225], 95.00th=[ 522191],
  1296. | 99.00th=[ 750781], 99.50th=[ 843056], 99.90th=[ 926942],
  1297. | 99.95th=[ 935330], 99.99th=[1098908]
  1298. bw ( KiB/s): min= 0, max= 3198, per=44.64%, avg=2133.50, stdev=1505.43, samples=2
  1299. iops : min= 0, max= 49, avg=32.50, stdev=23.33, samples=2
  1300. lat (nsec) : 750=5.14%, 1000=14.97%
  1301. lat (usec) : 2=16.56%, 4=0.08%, 10=0.14%, 20=0.82%, 50=0.78%
  1302. lat (usec) : 100=1.10%, 250=0.52%, 500=0.59%, 750=0.64%, 1000=1.61%
  1303. lat (msec) : 2=11.03%, 4=7.50%, 10=3.43%, 20=1.87%, 50=4.54%
  1304. lat (msec) : 100=9.75%, 250=11.70%, 500=4.33%, 750=2.42%, 1000=0.50%
  1305. cpu : usr=0.05%, sys=0.19%, ctx=9130, majf=0, minf=30
  1306. IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  1307. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  1308. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  1309. issued rwts: total=3814,4000,0,0 short=0,0,0,0 dropped=0,0,0,0
  1310. latency : target=0, window=0, percentile=100.00%, depth=4
  1311.  
  1312. Run status group 0 (all jobs):
  1313. READ: bw=22.2MiB/s (23.3MB/s), 2033KiB/s-20.2MiB/s (2082kB/s-21.2MB/s), io=2663MiB (2792MB), run=120013-120015msec
  1314. WRITE: bw=4779KiB/s (4894kB/s), 2135KiB/s-2646KiB/s (2186kB/s-2709kB/s), io=561MiB (588MB), run=120015-120098msec
  1315.  
  1316. Disk stats (read/write):
  1317. rbd141: ios=7942/9637, merge=0/3, ticks=548741/850068, in_queue=1389903, util=12.74%
  1318. Jobs: 63 (f=59): [_(2),m(1)][99.8%][r=1231M,w=976M][r=19.2k,w=15.2k IOPS][eta 00m:55s] <example-benchmark-fio-server-910-benchmark-74gxv>
  1319. randwrite: (groupid=0, jobs=1): err= 0: pid=11: Wed May 8 20:34:55 2019
  1320. write: IOPS=47, BW=3049KiB/s (3122kB/s)(358MiB/120101msec)
  1321. slat (nsec): min=0, max=587401k, avg=499377.44, stdev=12120076.80
  1322. clat (nsec): min=0, max=1699.1M, avg=83679046.09, stdev=152339670.45
  1323. lat (nsec): min=0, max=1699.1M, avg=84179546.98, stdev=152850121.23
  1324. clat percentiles (usec):
  1325. | 1.00th=[ 979], 5.00th=[ 1156], 10.00th=[ 1336],
  1326. | 20.00th=[ 1778], 30.00th=[ 2638], 40.00th=[ 4686],
  1327. | 50.00th=[ 14222], 60.00th=[ 51643], 70.00th=[ 87557],
  1328. | 80.00th=[ 117965], 90.00th=[ 225444], 95.00th=[ 413139],
  1329. | 99.00th=[ 725615], 99.50th=[ 884999], 99.90th=[1149240],
  1330. | 99.95th=[1300235], 99.99th=[1702888]
  1331. bw ( KiB/s): min= 0, max= 3874, per=54.17%, avg=3049.50, stdev=1166.02, samples=2
  1332. iops : min= 0, max= 60, avg=47.00, stdev=18.38, samples=2
  1333. lat (usec) : 750=0.07%, 1000=1.42%
  1334. lat (msec) : 2=21.72%, 4=14.44%, 10=10.21%, 20=4.77%, 50=6.92%
  1335. lat (msec) : 100=14.43%, 250=16.89%, 500=5.58%, 750=2.64%, 1000=0.65%
  1336. cpu : usr=0.06%, sys=0.20%, ctx=10898, majf=0, minf=30
  1337. IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  1338. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  1339. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  1340. issued rwts: total=0,5719,0,0 short=0,0,0,0 dropped=0,0,0,0
  1341. latency : target=0, window=0, percentile=100.00%, depth=4
  1342. read: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:34:55 2019
  1343. read: IOPS=369, BW=23.0MiB/s (24.2MB/s)(2766MiB/120003msec)
  1344. slat (nsec): min=0, max=586968k, avg=49643.35, stdev=3526447.54
  1345. clat (nsec): min=0, max=1687.4M, avg=10816930.67, stdev=56864630.85
  1346. lat (nsec): min=0, max=1687.5M, avg=10867413.28, stdev=56980443.13
  1347. clat percentiles (nsec):
  1348. | 1.00th=[ 708], 5.00th=[ 716], 10.00th=[ 716],
  1349. | 20.00th=[ 724], 30.00th=[ 732], 40.00th=[ 740],
  1350. | 50.00th=[ 812], 60.00th=[ 868], 70.00th=[ 948],
  1351. | 80.00th=[ 1384], 90.00th=[ 3555328], 95.00th=[ 50593792],
  1352. | 99.00th=[ 308281344], 99.50th=[ 438304768], 99.90th=[ 658505728],
  1353. | 99.95th=[ 734003200], 99.99th=[1233125376]
  1354. bw ( KiB/s): min= 0, max=34327, per=90.47%, avg=23600.50, stdev=15169.56, samples=2
  1355. iops : min= 0, max= 537, avg=369.00, stdev=237.59, samples=2
  1356. lat (nsec) : 750=43.42%, 1000=28.81%
  1357. lat (usec) : 2=9.35%, 4=0.04%, 10=0.87%, 20=1.08%, 50=0.38%
  1358. lat (usec) : 100=0.12%, 250=0.07%, 500=0.10%, 750=0.73%, 1000=1.59%
  1359. lat (msec) : 2=2.31%, 4=1.29%, 10=0.76%, 20=0.71%, 50=3.29%
  1360. lat (msec) : 100=2.40%, 250=1.39%, 500=0.95%, 750=0.28%, 1000=0.03%
  1361. cpu : usr=0.10%, sys=0.41%, ctx=14046, majf=0, minf=97
  1362. IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  1363. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  1364. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  1365. issued rwts: total=44330,0,0,0 short=0,0,0,0 dropped=0,0,0,0
  1366. latency : target=0, window=0, percentile=100.00%, depth=4
  1367. readwrite: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:34:55 2019
  1368. read: IOPS=38, BW=2487KiB/s (2547kB/s)(292MiB/120019msec)
  1369. slat (nsec): min=0, max=368100k, avg=106961.37, stdev=5389940.62
  1370. clat (nsec): min=0, max=759558k, avg=9002016.33, stdev=40516136.77
  1371. lat (nsec): min=0, max=759588k, avg=9109980.22, stdev=40868878.36
  1372. clat percentiles (nsec):
  1373. | 1.00th=[ 716], 5.00th=[ 724], 10.00th=[ 732],
  1374. | 20.00th=[ 804], 30.00th=[ 884], 40.00th=[ 996],
  1375. | 50.00th=[ 1144], 60.00th=[ 1256], 70.00th=[ 1432],
  1376. | 80.00th=[ 71168], 90.00th=[ 5275648], 95.00th=[ 54263808],
  1377. | 99.00th=[212860928], 99.50th=[308281344], 99.90th=[429916160],
  1378. | 99.95th=[476053504], 99.99th=[759169024]
  1379. bw ( KiB/s): min= 0, max= 2916, per=9.53%, avg=2486.50, stdev=607.40, samples=2
  1380. iops : min= 0, max= 45, avg=38.50, stdev= 9.19, samples=2
  1381. write: IOPS=40, BW=2583KiB/s (2645kB/s)(303MiB/120019msec)
  1382. slat (nsec): min=0, max=626891k, avg=490839.66, stdev=12588951.98
  1383. clat (nsec): min=0, max=1796.9M, avg=90155001.53, stdev=160192506.95
  1384. lat (nsec): min=0, max=1796.0M, avg=90646821.59, stdev=160496253.46
  1385. clat percentiles (usec):
  1386. | 1.00th=[ 1012], 5.00th=[ 1237], 10.00th=[ 1434],
  1387. | 20.00th=[ 2040], 30.00th=[ 3294], 40.00th=[ 6587],
  1388. | 50.00th=[ 21627], 60.00th=[ 59507], 70.00th=[ 91751],
  1389. | 80.00th=[ 121111], 90.00th=[ 252707], 95.00th=[ 442500],
  1390. | 99.00th=[ 759170], 99.50th=[ 884999], 99.90th=[1300235],
  1391. | 99.95th=[1686111], 99.99th=[1803551]
  1392. bw ( KiB/s): min= 0, max= 3015, per=45.85%, avg=2581.50, stdev=613.06, samples=2
  1393. iops : min= 0, max= 47, avg=40.00, stdev= 9.90, samples=2
  1394. lat (nsec) : 750=7.48%, 1000=12.19%
  1395. lat (usec) : 2=17.28%, 4=0.13%, 10=0.34%, 20=0.79%, 50=0.62%
  1396. lat (usec) : 100=0.64%, 250=0.64%, 500=0.63%, 750=0.41%, 1000=1.38%
  1397. lat (msec) : 2=10.86%, 4=7.39%, 10=6.11%, 20=3.06%, 50=5.52%
  1398. lat (msec) : 100=9.36%, 250=9.71%, 500=3.46%, 750=1.48%, 1000=0.39%
  1399. cpu : usr=0.05%, sys=0.24%, ctx=11003, majf=0, minf=30
  1400. IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  1401. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  1402. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  1403. issued rwts: total=4666,4841,0,0 short=0,0,0,0 dropped=0,0,0,0
  1404. latency : target=0, window=0, percentile=100.00%, depth=4
  1405.  
  1406. Run status group 0 (all jobs):
  1407. READ: bw=25.5MiB/s (26.7MB/s), 2487KiB/s-23.0MiB/s (2547kB/s-24.2MB/s), io=3058MiB (3206MB), run=120003-120019msec
  1408. WRITE: bw=5630KiB/s (5766kB/s), 2583KiB/s-3049KiB/s (2645kB/s-3122kB/s), io=660MiB (692MB), run=120019-120101msec
  1409.  
  1410. Disk stats (read/write):
  1411. rbd320: ios=8138/10744, merge=0/1, ticks=543117/803685, in_queue=1337178, util=13.96%
  1412. <example-benchmark-fio-server-697-benchmark-v5m2j>
  1413. randwrite: (groupid=0, jobs=1): err= 0: pid=11: Wed May 8 20:34:55 2019
  1414. write: IOPS=52, BW=3334KiB/s (3414kB/s)(391MiB/120024msec)
  1415. slat (nsec): min=0, max=950382k, avg=452969.44, stdev=14552434.79
  1416. clat (nsec): min=0, max=1440.0M, avg=76854293.56, stdev=153490325.63
  1417. lat (nsec): min=0, max=1441.2M, avg=77308421.84, stdev=154089812.05
  1418. clat percentiles (usec):
  1419. | 1.00th=[ 955], 5.00th=[ 1205], 10.00th=[ 1401],
  1420. | 20.00th=[ 1762], 30.00th=[ 2376], 40.00th=[ 3589],
  1421. | 50.00th=[ 6849], 60.00th=[ 23725], 70.00th=[ 76022],
  1422. | 80.00th=[ 107480], 90.00th=[ 208667], 95.00th=[ 392168],
  1423. | 99.00th=[ 784335], 99.50th=[ 910164], 99.90th=[1199571],
  1424. | 99.95th=[1300235], 99.99th=[1434452]
  1425. bw ( KiB/s): min= 0, max= 5185, per=58.75%, avg=3333.00, stdev=2619.12, samples=2
  1426. iops : min= 0, max= 81, avg=52.00, stdev=41.01, samples=2
  1427. lat (usec) : 100=0.02%, 250=0.03%, 750=0.03%, 1000=1.36%
  1428. lat (msec) : 2=23.31%, 4=17.55%, 10=11.94%, 20=4.78%, 50=4.61%
  1429. lat (msec) : 100=14.32%, 250=13.89%, 500=4.88%, 750=2.08%, 1000=0.99%
  1430. cpu : usr=0.06%, sys=0.22%, ctx=11738, majf=0, minf=27
  1431. IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  1432. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  1433. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  1434. issued rwts: total=0,6250,0,0 short=0,0,0,0 dropped=0,0,0,0
  1435. latency : target=0, window=0, percentile=100.00%, depth=4
  1436. read: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:34:55 2019
  1437. read: IOPS=341, BW=21.3MiB/s (22.4MB/s)(2561MiB/120020msec)
  1438. slat (nsec): min=0, max=871267k, avg=58333.28, stdev=5300696.13
  1439. clat (nsec): min=0, max=1267.3M, avg=11691015.06, stdev=59871844.84
  1440. lat (nsec): min=0, max=1267.5M, avg=11750191.55, stdev=60112618.15
  1441. clat percentiles (nsec):
  1442. | 1.00th=[ 708], 5.00th=[ 716], 10.00th=[ 716],
  1443. | 20.00th=[ 724], 30.00th=[ 732], 40.00th=[ 772],
  1444. | 50.00th=[ 836], 60.00th=[ 892], 70.00th=[ 1032],
  1445. | 80.00th=[ 1576], 90.00th=[ 4882432], 95.00th=[ 53739520],
  1446. | 99.00th=[ 337641472], 99.50th=[ 467664896], 99.90th=[ 692060160],
  1447. | 99.95th=[ 784334848], 99.99th=[1082130432]
  1448. bw ( KiB/s): min= 0, max=28268, per=90.74%, avg=21854.00, stdev=9070.77, samples=2
  1449. iops : min= 0, max= 442, avg=341.50, stdev=142.13, samples=2
  1450. lat (nsec) : 750=36.43%, 1000=32.41%
  1451. lat (usec) : 2=11.69%, 4=0.06%, 10=0.56%, 20=0.22%, 50=0.16%
  1452. lat (usec) : 100=0.09%, 250=0.04%, 500=0.25%, 750=1.03%, 1000=2.28%
  1453. lat (msec) : 2=2.99%, 4=1.55%, 10=0.76%, 20=0.64%, 50=3.41%
  1454. lat (msec) : 100=2.65%, 250=1.38%, 500=1.02%, 750=0.31%, 1000=0.04%
  1455. cpu : usr=0.11%, sys=0.39%, ctx=14938, majf=0, minf=30
  1456. IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  1457. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  1458. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  1459. issued rwts: total=41006,0,0,0 short=0,0,0,0 dropped=0,0,0,0
  1460. latency : target=0, window=0, percentile=100.00%, depth=4
  1461. readwrite: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:34:55 2019
  1462. read: IOPS=34, BW=2232KiB/s (2286kB/s)(262MiB/120028msec)
  1463. slat (nsec): min=0, max=3261.9k, avg=30353.32, stdev=99290.01
  1464. clat (nsec): min=0, max=1235.7M, avg=9591222.87, stdev=49457877.63
  1465. lat (nsec): min=0, max=1235.9M, avg=9622550.11, stdev=49466608.08
  1466. clat percentiles (nsec):
  1467. | 1.00th=[ 708], 5.00th=[ 724], 10.00th=[ 732],
  1468. | 20.00th=[ 804], 30.00th=[ 884], 40.00th=[ 1020],
  1469. | 50.00th=[ 1160], 60.00th=[ 1288], 70.00th=[ 1496],
  1470. | 80.00th=[ 193536], 90.00th=[ 6127616], 95.00th=[ 52690944],
  1471. | 99.00th=[ 235929600], 99.50th=[ 354418688], 99.90th=[ 517996544],
  1472. | 99.95th=[ 641728512], 99.99th=[1233125376]
  1473. bw ( KiB/s): min= 0, max= 3177, per=9.27%, avg=2232.00, stdev=1336.43, samples=2
  1474. iops : min= 0, max= 49, avg=34.50, stdev=20.51, samples=2
  1475. write: IOPS=36, BW=2340KiB/s (2396kB/s)(274MiB/120028msec)
  1476. slat (nsec): min=0, max=839359k, avg=446443.00, stdev=13829654.09
  1477. clat (nsec): min=0, max=1464.5M, avg=100143464.64, stdev=174579671.22
  1478. lat (nsec): min=0, max=1464.6M, avg=100590859.57, stdev=175258011.37
  1479. clat percentiles (usec):
  1480. | 1.00th=[ 1057], 5.00th=[ 1287], 10.00th=[ 1532],
  1481. | 20.00th=[ 2212], 30.00th=[ 3589], 40.00th=[ 7373],
  1482. | 50.00th=[ 29754], 60.00th=[ 71828], 70.00th=[ 95945],
  1483. | 80.00th=[ 125305], 90.00th=[ 278922], 95.00th=[ 484443],
  1484. | 99.00th=[ 884999], 99.50th=[ 943719], 99.90th=[1350566],
  1485. | 99.95th=[1468007], 99.99th=[1468007]
  1486. bw ( KiB/s): min= 0, max= 3323, per=41.20%, avg=2337.50, stdev=1393.71, samples=2
  1487. iops : min= 0, max= 51, avg=36.00, stdev=21.21, samples=2
  1488. lat (nsec) : 750=7.20%, 1000=11.69%
  1489. lat (usec) : 2=17.30%, 4=0.08%, 10=0.29%, 20=0.59%, 50=0.77%
  1490. lat (usec) : 100=0.80%, 250=0.59%, 500=0.42%, 750=0.65%, 1000=1.42%
  1491. lat (msec) : 2=10.28%, 4=8.01%, 10=6.04%, 20=2.99%, 50=4.37%
  1492. lat (msec) : 100=11.07%, 250=9.46%, 500=3.51%, 750=1.54%, 1000=0.75%
  1493. cpu : usr=0.06%, sys=0.21%, ctx=10176, majf=0, minf=30
  1494. IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  1495. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  1496. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  1497. issued rwts: total=4187,4385,0,0 short=0,0,0,0 dropped=0,0,0,0
  1498. latency : target=0, window=0, percentile=100.00%, depth=4
  1499.  
  1500. Run status group 0 (all jobs):
  1501. READ: bw=23.5MiB/s (24.7MB/s), 2232KiB/s-21.3MiB/s (2286kB/s-22.4MB/s), io=2823MiB (2960MB), run=120020-120028msec
  1502. WRITE: bw=5674KiB/s (5810kB/s), 2340KiB/s-3334KiB/s (2396kB/s-3414kB/s), io=665MiB (697MB), run=120024-120028msec
  1503.  
  1504. Disk stats (read/write):
  1505. rbd234: ios=8956/11204, merge=0/1, ticks=542224/827259, in_queue=1359118, util=14.50%
  1506. <example-benchmark-fio-server-112-benchmark-wpgw6>
  1507. randwrite: (groupid=0, jobs=1): err= 0: pid=11: Wed May 8 20:34:55 2019
  1508. write: IOPS=47, BW=3020KiB/s (3092kB/s)(354MiB/120030msec)
  1509. slat (nsec): min=0, max=434516k, avg=325543.33, stdev=7343542.39
  1510. clat (nsec): min=0, max=953098k, avg=84734749.94, stdev=150303929.00
  1511. lat (nsec): min=0, max=953156k, avg=85061546.03, stdev=150791631.58
  1512. clat percentiles (usec):
  1513. | 1.00th=[ 996], 5.00th=[ 1205], 10.00th=[ 1401], 20.00th=[ 1844],
  1514. | 30.00th=[ 2573], 40.00th=[ 4113], 50.00th=[ 8848], 60.00th=[ 36963],
  1515. | 70.00th=[ 79168], 80.00th=[119014], 90.00th=[295699], 95.00th=[459277],
  1516. | 99.00th=[683672], 99.50th=[725615], 99.90th=[792724], 99.95th=[876610],
  1517. | 99.99th=[952108]
  1518. bw ( KiB/s): min= 0, max= 4542, per=57.71%, avg=3017.50, stdev=2155.97, samples=2
  1519. iops : min= 0, max= 70, avg=46.50, stdev=33.23, samples=2
  1520. lat (usec) : 4=0.02%, 50=0.02%, 250=0.02%, 500=0.04%, 750=0.02%
  1521. lat (usec) : 1000=0.97%
  1522. lat (msec) : 2=21.73%, 4=16.86%, 10=11.47%, 20=4.84%, 50=6.17%
  1523. lat (msec) : 100=13.85%, 250=13.06%, 500=7.26%, 750=3.43%, 1000=0.32%
  1524. cpu : usr=0.05%, sys=0.19%, ctx=10421, majf=0, minf=25
  1525. IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  1526. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  1527. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  1528. issued rwts: total=0,5660,0,0 short=0,0,0,0 dropped=0,0,0,0
  1529. latency : target=0, window=0, percentile=100.00%, depth=4
  1530. read: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:34:55 2019
  1531. read: IOPS=336, BW=21.0MiB/s (22.0MB/s)(2523MiB/120021msec)
  1532. slat (nsec): min=0, max=406062k, avg=39732.49, stdev=2589915.75
  1533. clat (nsec): min=0, max=1114.5M, avg=11880232.54, stdev=59995718.21
  1534. lat (nsec): min=0, max=1114.6M, avg=11920800.50, stdev=60062257.19
  1535. clat percentiles (nsec):
  1536. | 1.00th=[ 708], 5.00th=[ 708], 10.00th=[ 716],
  1537. | 20.00th=[ 724], 30.00th=[ 732], 40.00th=[ 740],
  1538. | 50.00th=[ 796], 60.00th=[ 868], 70.00th=[ 972],
  1539. | 80.00th=[ 1432], 90.00th=[ 3620864], 95.00th=[ 53739520],
  1540. | 99.00th=[362807296], 99.50th=[471859200], 99.90th=[675282944],
  1541. | 99.95th=[742391808], 99.99th=[901775360]
  1542. bw ( KiB/s): min= 0, max=27225, per=91.10%, avg=21528.00, stdev=8056.77, samples=2
  1543. iops : min= 0, max= 425, avg=336.00, stdev=125.87, samples=2
  1544. lat (nsec) : 750=44.70%, 1000=26.30%
  1545. lat (usec) : 2=10.01%, 4=0.04%, 10=0.50%, 20=0.32%, 50=0.16%
  1546. lat (usec) : 100=0.06%, 250=0.06%, 500=0.29%, 750=1.02%, 1000=2.18%
  1547. lat (msec) : 2=2.88%, 4=1.61%, 10=0.76%, 20=0.62%, 50=3.14%
  1548. lat (msec) : 100=2.47%, 250=1.40%, 500=1.09%, 750=0.37%, 1000=0.04%
  1549. cpu : usr=0.11%, sys=0.37%, ctx=14426, majf=0, minf=29
  1550. IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  1551. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  1552. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  1553. issued rwts: total=40393,0,0,0 short=0,0,0,0 dropped=0,0,0,0
  1554. latency : target=0, window=0, percentile=100.00%, depth=4
  1555. readwrite: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:34:55 2019
  1556. read: IOPS=32, BW=2106KiB/s (2156kB/s)(247MiB/120003msec)
  1557. slat (nsec): min=0, max=3284.3k, avg=29352.61, stdev=98817.29
  1558. clat (nsec): min=0, max=785351k, avg=10373089.41, stdev=48731284.43
  1559. lat (nsec): min=0, max=785379k, avg=10403500.78, stdev=48741066.99
  1560. clat percentiles (nsec):
  1561. | 1.00th=[ 708], 5.00th=[ 716], 10.00th=[ 732],
  1562. | 20.00th=[ 772], 30.00th=[ 884], 40.00th=[ 1004],
  1563. | 50.00th=[ 1144], 60.00th=[ 1288], 70.00th=[ 1480],
  1564. | 80.00th=[ 154624], 90.00th=[ 11337728], 95.00th=[ 54788096],
  1565. | 99.00th=[265289728], 99.50th=[387973120], 99.90th=[591396864],
  1566. | 99.95th=[616562688], 99.99th=[784334848]
  1567. bw ( KiB/s): min= 0, max= 3043, per=8.91%, avg=2104.50, stdev=1327.24, samples=2
  1568. iops : min= 0, max= 47, avg=32.50, stdev=20.51, samples=2
  1569. write: IOPS=34, BW=2211KiB/s (2264kB/s)(259MiB/120003msec)
  1570. slat (nsec): min=0, max=392423k, avg=356288.72, stdev=8053940.39
  1571. clat (nsec): min=0, max=1000.3M, avg=105913790.38, stdev=167085986.59
  1572. lat (nsec): min=0, max=1000.4M, avg=106271139.65, stdev=167201360.65
  1573. clat percentiles (usec):
  1574. | 1.00th=[ 1037], 5.00th=[ 1237], 10.00th=[ 1467],
  1575. | 20.00th=[ 2114], 30.00th=[ 3392], 40.00th=[ 6980],
  1576. | 50.00th=[ 27395], 60.00th=[ 68682], 70.00th=[ 99091],
  1577. | 80.00th=[ 147850], 90.00th=[ 371196], 95.00th=[ 497026],
  1578. | 99.00th=[ 717226], 99.50th=[ 750781], 99.90th=[ 876610],
  1579. | 99.95th=[ 876610], 99.99th=[1002439]
  1580. bw ( KiB/s): min= 0, max= 3193, per=42.24%, avg=2208.50, stdev=1392.29, samples=2
  1581. iops : min= 0, max= 49, avg=34.00, stdev=21.21, samples=2
  1582. lat (nsec) : 750=8.26%, 1000=11.10%
  1583. lat (usec) : 2=16.60%, 4=0.10%, 10=0.26%, 20=0.94%, 50=0.78%
  1584. lat (usec) : 100=0.75%, 250=0.61%, 500=0.54%, 750=0.48%, 1000=1.27%
  1585. lat (msec) : 2=10.62%, 4=7.74%, 10=6.02%, 20=2.82%, 50=5.01%
  1586. lat (msec) : 100=9.83%, 250=8.37%, 500=5.39%, 750=2.29%, 1000=0.27%
  1587. cpu : usr=0.05%, sys=0.19%, ctx=9532, majf=0, minf=30
  1588. IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  1589. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  1590. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  1591. issued rwts: total=3949,4142,0,0 short=0,0,0,0 dropped=0,0,0,0
  1592. latency : target=0, window=0, percentile=100.00%, depth=4
  1593.  
  1594. Run status group 0 (all jobs):
  1595. READ: bw=23.1MiB/s (24.2MB/s), 2106KiB/s-21.0MiB/s (2156kB/s-22.0MB/s), io=2770MiB (2904MB), run=120003-120021msec
  1596. WRITE: bw=5230KiB/s (5355kB/s), 2211KiB/s-3020KiB/s (2264kB/s-3092kB/s), io=613MiB (643MB), run=120003-120030msec
  1597.  
  1598. Disk stats (read/write):
  1599. rbd37: ios=8286/10343, merge=0/3, ticks=542121/837826, in_queue=1370444, util=13.14%
  1600. <example-benchmark-fio-server-5-benchmark-hchc9>
  1601. randwrite: (groupid=0, jobs=1): err= 0: pid=11: Wed May 8 20:34:55 2019
  1602. write: IOPS=43, BW=2784KiB/s (2851kB/s)(326MiB/120043msec)
  1603. slat (nsec): min=0, max=1469.3M, avg=542823.26, stdev=21352335.09
  1604. clat (nsec): min=0, max=1719.5M, avg=91545953.17, stdev=150529149.13
  1605. lat (nsec): min=0, max=2232.0M, avg=92089805.76, stdev=153224563.06
  1606. clat percentiles (usec):
  1607. | 1.00th=[ 988], 5.00th=[ 1237], 10.00th=[ 1418],
  1608. | 20.00th=[ 1958], 30.00th=[ 3228], 40.00th=[ 6980],
  1609. | 50.00th=[ 42730], 60.00th=[ 78119], 70.00th=[ 102237],
  1610. | 80.00th=[ 127402], 90.00th=[ 219153], 95.00th=[ 408945],
  1611. | 99.00th=[ 725615], 99.50th=[ 834667], 99.90th=[1350566],
  1612. | 99.95th=[1468007], 99.99th=[1719665]
  1613. bw ( KiB/s): min= 0, max= 3203, per=54.41%, avg=2782.50, stdev=594.68, samples=2
  1614. iops : min= 0, max= 50, avg=43.00, stdev= 9.90, samples=2
  1615. lat (usec) : 4=0.02%, 750=0.11%, 1000=1.03%
  1616. lat (msec) : 2=19.37%, 4=12.93%, 10=9.37%, 20=3.37%, 50=5.31%
  1617. lat (msec) : 100=17.24%, 250=22.05%, 500=6.05%, 750=2.28%, 1000=0.73%
  1618. cpu : usr=0.05%, sys=0.18%, ctx=9915, majf=0, minf=30
  1619. IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  1620. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  1621. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  1622. issued rwts: total=0,5219,0,0 short=0,0,0,0 dropped=0,0,0,0
  1623. latency : target=0, window=0, percentile=100.00%, depth=4
  1624. read: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:34:55 2019
  1625. read: IOPS=348, BW=21.8MiB/s (22.9MB/s)(2615MiB/120001msec)
  1626. slat (nsec): min=0, max=1466.4M, avg=67849.19, stdev=7423963.40
  1627. clat (nsec): min=0, max=1973.0M, avg=11410932.15, stdev=57537264.39
  1628. lat (nsec): min=0, max=1974.0M, avg=11479646.23, stdev=58075993.63
  1629. clat percentiles (nsec):
  1630. | 1.00th=[ 708], 5.00th=[ 708], 10.00th=[ 716],
  1631. | 20.00th=[ 724], 30.00th=[ 740], 40.00th=[ 780],
  1632. | 50.00th=[ 852], 60.00th=[ 932], 70.00th=[ 1192],
  1633. | 80.00th=[ 839680], 90.00th=[ 14352384], 95.00th=[ 54788096],
  1634. | 99.00th=[ 304087040], 99.50th=[ 438304768], 99.90th=[ 683671552],
  1635. | 99.95th=[ 792723456], 99.99th=[1216348160]
  1636. bw ( KiB/s): min= 0, max=31764, per=90.94%, avg=22314.00, stdev=13364.32, samples=2
  1637. iops : min= 0, max= 496, avg=348.50, stdev=208.60, samples=2
  1638. lat (nsec) : 750=35.07%, 1000=28.45%
  1639. lat (usec) : 2=12.71%, 4=0.05%, 10=0.53%, 20=0.29%, 50=0.14%
  1640. lat (usec) : 100=0.06%, 250=0.07%, 500=0.41%, 750=1.13%, 1000=2.65%
  1641. lat (msec) : 2=3.94%, 4=2.52%, 10=1.49%, 20=1.02%, 50=3.88%
  1642. lat (msec) : 100=3.09%, 250=1.23%, 500=0.96%, 750=0.25%, 1000=0.05%
  1643. cpu : usr=0.12%, sys=0.42%, ctx=18819, majf=0, minf=27
  1644. IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  1645. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  1646. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  1647. issued rwts: total=41856,0,0,0 short=0,0,0,0 dropped=0,0,0,0
  1648. latency : target=0, window=0, percentile=100.00%, depth=4
  1649. readwrite: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:34:55 2019
  1650. read: IOPS=34, BW=2226KiB/s (2280kB/s)(261MiB/120024msec)
  1651. slat (nsec): min=0, max=1469.5M, avg=485436.54, stdev=23708009.49
  1652. clat (nsec): min=0, max=1068.9M, avg=10575940.19, stdev=49916812.10
  1653. lat (nsec): min=0, max=1470.0M, avg=11062295.90, stdev=55220809.67
  1654. clat percentiles (nsec):
  1655. | 1.00th=[ 708], 5.00th=[ 716], 10.00th=[ 732],
  1656. | 20.00th=[ 780], 30.00th=[ 884], 40.00th=[ 1012],
  1657. | 50.00th=[ 1160], 60.00th=[ 1272], 70.00th=[ 1480],
  1658. | 80.00th=[ 193536], 90.00th=[ 21102592], 95.00th=[ 57933824],
  1659. | 99.00th=[ 235929600], 99.50th=[ 362807296], 99.90th=[ 616562688],
  1660. | 99.95th=[ 734003200], 99.99th=[1069547520]
  1661. bw ( KiB/s): min= 0, max= 2374, per=9.07%, avg=2226.00, stdev=209.30, samples=2
  1662. iops : min= 0, max= 37, avg=34.50, stdev= 3.54, samples=2
  1663. write: IOPS=36, BW=2331KiB/s (2387kB/s)(273MiB/120024msec)
  1664. slat (nsec): min=0, max=6481.8k, avg=159585.38, stdev=274880.69
  1665. clat (nsec): min=0, max=1784.7M, avg=99395294.56, stdev=159205507.24
  1666. lat (nsec): min=0, max=1784.7M, avg=99556016.47, stdev=159164612.17
  1667. clat percentiles (usec):
  1668. | 1.00th=[ 1037], 5.00th=[ 1287], 10.00th=[ 1549],
  1669. | 20.00th=[ 2343], 30.00th=[ 4228], 40.00th=[ 10683],
  1670. | 50.00th=[ 54264], 60.00th=[ 81265], 70.00th=[ 106431],
  1671. | 80.00th=[ 131597], 90.00th=[ 270533], 95.00th=[ 442500],
  1672. | 99.00th=[ 784335], 99.50th=[ 918553], 99.90th=[1384121],
  1673. | 99.95th=[1468007], 99.99th=[1786774]
  1674. bw ( KiB/s): min= 0, max= 2476, per=45.53%, avg=2328.50, stdev=208.60, samples=2
  1675. iops : min= 0, max= 38, avg=36.00, stdev= 2.83, samples=2
  1676. lat (nsec) : 750=7.96%, 1000=11.17%
  1677. lat (usec) : 2=16.96%, 4=0.06%, 10=0.27%, 20=0.83%, 50=0.71%
  1678. lat (usec) : 100=0.70%, 250=0.57%, 500=0.54%, 750=0.55%, 1000=1.05%
  1679. lat (msec) : 2=9.34%, 4=7.48%, 10=5.65%, 20=1.94%, 50=5.16%
  1680. lat (msec) : 100=11.00%, 250=12.30%, 500=3.83%, 750=1.32%, 1000=0.49%
  1681. cpu : usr=0.04%, sys=0.22%, ctx=10113, majf=0, minf=34
  1682. IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  1683. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  1684. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  1685. issued rwts: total=4175,4368,0,0 short=0,0,0,0 dropped=0,0,0,0
  1686. latency : target=0, window=0, percentile=100.00%, depth=4
  1687.  
  1688. Run status group 0 (all jobs):
  1689. READ: bw=23.0MiB/s (25.1MB/s), 2226KiB/s-21.8MiB/s (2280kB/s-22.9MB/s), io=2876MiB (3016MB), run=120001-120024msec
  1690. WRITE: bw=5114KiB/s (5237kB/s), 2331KiB/s-2784KiB/s (2387kB/s-2851kB/s), io=600MiB (629MB), run=120024-120043msec
  1691.  
  1692. Disk stats (read/write):
  1693. rbd2: ios=11019/10797, merge=0/1, ticks=541896/812590, in_queue=1343285, util=15.83%
  1694. <example-benchmark-fio-server-752-benchmark-kfgvn>
  1695. randwrite: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:34:55 2019
  1696. write: IOPS=38, BW=2456KiB/s (2515kB/s)(288MiB/120075msec)
  1697. slat (nsec): min=0, max=486355k, avg=347151.01, stdev=8370452.54
  1698. clat (nsec): min=0, max=1379.9M, avg=104506667.37, stdev=171039013.89
  1699. lat (nsec): min=0, max=1379.0M, avg=104854912.10, stdev=171182329.65
  1700. clat percentiles (usec):
  1701. | 1.00th=[ 988], 5.00th=[ 1188], 10.00th=[ 1385],
  1702. | 20.00th=[ 1860], 30.00th=[ 2835], 40.00th=[ 5735],
  1703. | 50.00th=[ 19006], 60.00th=[ 63701], 70.00th=[ 99091],
  1704. | 80.00th=[ 154141], 90.00th=[ 346031], 95.00th=[ 501220],
  1705. | 99.00th=[ 742392], 99.50th=[ 801113], 99.90th=[1132463],
  1706. | 99.95th=[1300235], 99.99th=[1384121]
  1707. bw ( KiB/s): min= 0, max= 3208, per=55.27%, avg=2455.00, stdev=1064.90, samples=2
  1708. iops : min= 0, max= 50, avg=38.00, stdev=16.97, samples=2
  1709. lat (usec) : 500=0.07%, 750=0.02%, 1000=1.13%
  1710. lat (msec) : 2=21.35%, 4=13.07%, 10=9.29%, 20=5.52%, 50=5.91%
  1711. lat (msec) : 100=13.77%, 250=15.44%, 500=9.38%, 750=4.13%, 1000=0.83%
  1712. cpu : usr=0.05%, sys=0.16%, ctx=8673, majf=0, minf=30
  1713. IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  1714. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  1715. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  1716. issued rwts: total=0,4605,0,0 short=0,0,0,0 dropped=0,0,0,0
  1717. latency : target=0, window=0, percentile=100.00%, depth=4
  1718. read: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:34:55 2019
  1719. read: IOPS=336, BW=21.0MiB/s (22.0MB/s)(2521MiB/120024msec)
  1720. slat (nsec): min=0, max=476699k, avg=39713.39, stdev=2703973.90
  1721. clat (nsec): min=0, max=1301.7M, avg=11885616.09, stdev=61578485.33
  1722. lat (nsec): min=0, max=1301.7M, avg=11926172.49, stdev=61648490.33
  1723. clat percentiles (nsec):
  1724. | 1.00th=[ 708], 5.00th=[ 708], 10.00th=[ 716],
  1725. | 20.00th=[ 724], 30.00th=[ 732], 40.00th=[ 764],
  1726. | 50.00th=[ 828], 60.00th=[ 876], 70.00th=[ 996],
  1727. | 80.00th=[ 1432], 90.00th=[ 2670592], 95.00th=[ 53215232],
  1728. | 99.00th=[ 371195904], 99.50th=[ 476053504], 99.90th=[ 675282944],
  1729. | 99.95th=[ 742391808], 99.99th=[1019215872]
  1730. bw ( KiB/s): min= 0, max=25368, per=91.91%, avg=21512.50, stdev=5452.50, samples=2
  1731. iops : min= 0, max= 396, avg=335.50, stdev=85.56, samples=2
  1732. lat (nsec) : 750=36.50%, 1000=33.72%
  1733. lat (usec) : 2=11.17%, 4=0.10%, 10=0.82%, 20=0.63%, 50=0.14%
  1734. lat (usec) : 100=0.07%, 250=0.05%, 500=0.21%, 750=0.93%, 1000=1.97%
  1735. lat (msec) : 2=2.80%, 4=1.64%, 10=0.82%, 20=0.49%, 50=2.68%
  1736. lat (msec) : 100=2.35%, 250=1.38%, 500=1.12%, 750=0.36%, 1000=0.04%
  1737. cpu : usr=0.11%, sys=0.37%, ctx=13493, majf=0, minf=30
  1738. IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  1739. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  1740. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  1741. issued rwts: total=40363,0,0,0 short=0,0,0,0 dropped=0,0,0,0
  1742. latency : target=0, window=0, percentile=100.00%, depth=4
  1743. readwrite: (groupid=0, jobs=1): err= 0: pid=14: Wed May 8 20:34:55 2019
  1744. read: IOPS=29, BW=1897KiB/s (1943kB/s)(222MiB/120034msec)
  1745. slat (nsec): min=0, max=394240k, avg=221723.67, stdev=8238630.85
  1746. clat (nsec): min=0, max=533183k, avg=8440051.27, stdev=43256219.94
  1747. lat (nsec): min=0, max=533276k, avg=8662693.17, stdev=44003117.30
  1748. clat percentiles (nsec):
  1749. | 1.00th=[ 708], 5.00th=[ 724], 10.00th=[ 732],
  1750. | 20.00th=[ 772], 30.00th=[ 860], 40.00th=[ 956],
  1751. | 50.00th=[ 1096], 60.00th=[ 1240], 70.00th=[ 1432],
  1752. | 80.00th=[ 53504], 90.00th=[ 1630208], 95.00th=[ 42205184],
  1753. | 99.00th=[250609664], 99.50th=[346030080], 99.90th=[488636416],
  1754. | 99.95th=[530579456], 99.99th=[534773760]
  1755. bw ( KiB/s): min= 0, max= 2520, per=8.10%, avg=1896.50, stdev=881.76, samples=2
  1756. iops : min= 0, max= 39, avg=29.00, stdev=14.14, samples=2
  1757. write: IOPS=31, BW=1987KiB/s (2035kB/s)(233MiB/120034msec)
  1758. slat (nsec): min=0, max=4499.2k, avg=170586.55, stdev=218293.04
  1759. clat (nsec): min=0, max=1384.8M, avg=121032677.53, stdev=182906307.19
  1760. lat (nsec): min=0, max=1384.9M, avg=121204248.52, stdev=182846462.11
  1761. clat percentiles (usec):
  1762. | 1.00th=[ 1012], 5.00th=[ 1254], 10.00th=[ 1483],
  1763. | 20.00th=[ 2089], 30.00th=[ 3621], 40.00th=[ 9765],
  1764. | 50.00th=[ 45351], 60.00th=[ 81265], 70.00th=[ 112722],
  1765. | 80.00th=[ 206570], 90.00th=[ 392168], 95.00th=[ 541066],
  1766. | 99.00th=[ 767558], 99.50th=[ 901776], 99.90th=[1052771],
  1767. | 99.95th=[1300235], 99.99th=[1384121]
  1768. bw ( KiB/s): min= 0, max= 2640, per=44.71%, avg=1986.00, stdev=924.90, samples=2
  1769. iops : min= 0, max= 41, avg=30.50, stdev=14.85, samples=2
  1770. lat (nsec) : 750=8.23%, 1000=12.94%
  1771. lat (usec) : 2=15.82%, 4=0.07%, 10=0.12%, 20=0.73%, 50=1.07%
  1772. lat (usec) : 100=1.07%, 250=0.65%, 500=0.91%, 750=0.58%, 1000=1.33%
  1773. lat (msec) : 2=10.42%, 4=6.85%, 10=4.79%, 20=2.93%, 50=4.44%
  1774. lat (msec) : 100=8.27%, 250=9.74%, 500=5.82%, 750=2.66%, 1000=0.52%
  1775. cpu : usr=0.06%, sys=0.16%, ctx=8298, majf=0, minf=30
  1776. IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  1777. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  1778. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  1779. issued rwts: total=3559,3723,0,0 short=0,0,0,0 dropped=0,0,0,0
  1780. latency : target=0, window=0, percentile=100.00%, depth=4
  1781.  
  1782. Run status group 0 (all jobs):
  1783. READ: bw=22.9MiB/s (23.0MB/s), 1897KiB/s-21.0MiB/s (1943kB/s-22.0MB/s), io=2744MiB (2877MB), run=120024-120034msec
  1784. WRITE: bw=4443KiB/s (4549kB/s), 1987KiB/s-2456KiB/s (2035kB/s-2515kB/s), io=521MiB (546MB), run=120034-120075msec
  1785.  
  1786. Disk stats (read/write):
  1787. rbd272: ios=7654/8582, merge=0/1, ticks=537074/870601, in_queue=1399336, util=11.51%
  1788. <example-benchmark-fio-server-427-benchmark-hzqrn>
  1789. randwrite: (groupid=0, jobs=1): err= 0: pid=11: Wed May 8 20:34:55 2019
  1790. write: IOPS=55, BW=3533KiB/s (3617kB/s)(414MiB/120097msec)
  1791. slat (nsec): min=0, max=456194k, avg=355259.52, stdev=8398375.70
  1792. clat (nsec): min=0, max=991198k, avg=72238614.86, stdev=141200455.04
  1793. lat (nsec): min=0, max=991288k, avg=72595361.15, stdev=141328353.19
  1794. clat percentiles (usec):
  1795. | 1.00th=[ 1004], 5.00th=[ 1221], 10.00th=[ 1401], 20.00th=[ 1795],
  1796. | 30.00th=[ 2409], 40.00th=[ 3654], 50.00th=[ 6521], 60.00th=[ 16450],
  1797. | 70.00th=[ 69731], 80.00th=[104334], 90.00th=[204473], 95.00th=[392168],
  1798. | 99.00th=[708838], 99.50th=[792724], 99.90th=[926942], 99.95th=[952108],
  1799. | 99.99th=[994051]
  1800. bw ( KiB/s): min= 0, max= 4756, per=56.38%, avg=3533.00, stdev=1729.58, samples=2
  1801. iops : min= 0, max= 74, avg=55.00, stdev=26.87, samples=2
  1802. lat (usec) : 500=0.02%, 750=0.03%, 1000=0.92%
  1803. lat (msec) : 2=23.29%, 4=17.60%, 10=13.67%, 20=5.77%, 50=4.87%
  1804. lat (msec) : 100=12.65%, 250=12.93%, 500=4.95%, 750=2.69%, 1000=0.66%
  1805. cpu : usr=0.08%, sys=0.23%, ctx=12600, majf=0, minf=30
  1806. IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  1807. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  1808. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  1809. issued rwts: total=0,6626,0,0 short=0,0,0,0 dropped=0,0,0,0
  1810. latency : target=0, window=0, percentile=100.00%, depth=4
  1811. read: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:34:55 2019
  1812. read: IOPS=336, BW=21.0MiB/s (22.1MB/s)(2526MiB/120015msec)
  1813. slat (nsec): min=0, max=362838k, avg=45577.70, stdev=2841029.73
  1814. clat (nsec): min=0, max=1029.8M, avg=11837195.23, stdev=59499909.85
  1815. lat (nsec): min=0, max=1029.9M, avg=11883621.69, stdev=59574212.09
  1816. clat percentiles (nsec):
  1817. | 1.00th=[ 708], 5.00th=[ 708], 10.00th=[ 716],
  1818. | 20.00th=[ 724], 30.00th=[ 732], 40.00th=[ 748],
  1819. | 50.00th=[ 836], 60.00th=[ 892], 70.00th=[ 1080],
  1820. | 80.00th=[ 60672], 90.00th=[ 9240576], 95.00th=[ 54788096],
  1821. | 99.00th=[346030080], 99.50th=[476053504], 99.90th=[692060160],
  1822. | 99.95th=[750780416], 99.99th=[943718400]
  1823. bw ( KiB/s): min= 0, max=28561, per=89.07%, avg=21554.00, stdev=9909.39, samples=2
  1824. iops : min= 0, max= 446, avg=336.50, stdev=154.86, samples=2
  1825. lat (nsec) : 750=39.64%, 1000=28.22%
  1826. lat (usec) : 2=10.44%, 4=0.04%, 10=0.88%, 20=0.50%, 50=0.24%
  1827. lat (usec) : 100=0.10%, 250=0.06%, 500=0.15%, 750=0.98%, 1000=2.14%
  1828. lat (msec) : 2=3.42%, 4=2.12%, 10=1.14%, 20=0.74%, 50=3.59%
  1829. lat (msec) : 100=2.99%, 250=1.23%, 500=0.95%, 750=0.37%, 1000=0.05%
  1830. cpu : usr=0.12%, sys=0.38%, ctx=16094, majf=0, minf=102
  1831. IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  1832. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  1833. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  1834. issued rwts: total=40443,0,0,0 short=0,0,0,0 dropped=0,0,0,0
  1835. latency : target=0, window=0, percentile=100.00%, depth=4
  1836. readwrite: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:34:55 2019
  1837. read: IOPS=41, BW=2654KiB/s (2718kB/s)(311MiB/120055msec)
  1838. slat (nsec): min=0, max=238280k, avg=92609.11, stdev=3549619.81
  1839. clat (nsec): min=0, max=774600k, avg=8859266.13, stdev=41684463.38
  1840. lat (nsec): min=0, max=774650k, avg=8952847.07, stdev=41829723.60
  1841. clat percentiles (nsec):
  1842. | 1.00th=[ 716], 5.00th=[ 732], 10.00th=[ 764],
  1843. | 20.00th=[ 844], 30.00th=[ 924], 40.00th=[ 1048],
  1844. | 50.00th=[ 1192], 60.00th=[ 1320], 70.00th=[ 1608],
  1845. | 80.00th=[ 370688], 90.00th=[ 9895936], 95.00th=[ 52690944],
  1846. | 99.00th=[191889408], 99.50th=[312475648], 99.90th=[476053504],
  1847. | 99.95th=[658505728], 99.99th=[775946240]
  1848. bw ( KiB/s): min= 0, max= 3376, per=10.97%, avg=2655.00, stdev=1019.65, samples=2
  1849. iops : min= 0, max= 52, avg=41.00, stdev=15.56, samples=2
  1850. write: IOPS=42, BW=2735KiB/s (2801kB/s)(321MiB/120055msec)
  1851. slat (nsec): min=0, max=634946k, avg=345574.69, stdev=9626503.84
  1852. clat (nsec): min=0, max=1219.3M, avg=84920091.81, stdev=152543276.10
  1853. lat (nsec): min=0, max=1219.3M, avg=85266747.75, stdev=152708991.55
  1854. clat percentiles (usec):
  1855. | 1.00th=[ 1012], 5.00th=[ 1237], 10.00th=[ 1434],
  1856. | 20.00th=[ 1942], 30.00th=[ 2933], 40.00th=[ 4817],
  1857. | 50.00th=[ 10290], 60.00th=[ 45876], 70.00th=[ 84411],
  1858. | 80.00th=[ 115868], 90.00th=[ 254804], 95.00th=[ 467665],
  1859. | 99.00th=[ 725615], 99.50th=[ 801113], 99.90th=[ 952108],
  1860. | 99.95th=[ 985662], 99.99th=[1216349]
  1861. bw ( KiB/s): min= 0, max= 3446, per=43.63%, avg=2734.00, stdev=1006.92, samples=2
  1862. iops : min= 0, max= 53, avg=42.00, stdev=15.56, samples=2
  1863. lat (nsec) : 750=4.32%, 1000=13.77%
  1864. lat (usec) : 2=17.31%, 4=0.07%, 10=0.30%, 20=0.92%, 50=0.85%
  1865. lat (usec) : 100=0.87%, 250=0.72%, 500=0.62%, 750=0.53%, 1000=1.40%
  1866. lat (msec) : 2=11.97%, 4=8.73%, 10=7.17%, 20=3.20%, 50=4.70%
  1867. lat (msec) : 100=9.37%, 250=7.66%, 500=3.41%, 750=1.75%, 1000=0.36%
  1868. cpu : usr=0.07%, sys=0.25%, ctx=11962, majf=0, minf=31
  1869. IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  1870. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  1871. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  1872. issued rwts: total=4980,5128,0,0 short=0,0,0,0 dropped=0,0,0,0
  1873. latency : target=0, window=0, percentile=100.00%, depth=4
  1874.  
  1875. Run status group 0 (all jobs):
  1876. READ: bw=23.6MiB/s (24.8MB/s), 2654KiB/s-21.0MiB/s (2718kB/s-22.1MB/s), io=2837MiB (2975MB), run=120015-120055msec
  1877. WRITE: bw=6267KiB/s (6417kB/s), 2735KiB/s-3533KiB/s (2801kB/s-3617kB/s), io=735MiB (771MB), run=120055-120097msec
  1878.  
  1879. Disk stats (read/write):
  1880. rbd131: ios=9638/11955, merge=0/1, ticks=544140/818314, in_queue=1351514, util=15.67%
  1881. <example-benchmark-fio-server-731-benchmark-2p24w>
  1882. randwrite: (groupid=0, jobs=1): err= 0: pid=11: Wed May 8 20:34:55 2019
  1883. write: IOPS=46, BW=2954KiB/s (3025kB/s)(346MiB/120061msec)
  1884. slat (nsec): min=0, max=750892k, avg=445086.93, stdev=11721042.12
  1885. clat (nsec): min=0, max=1628.2M, avg=86222905.58, stdev=150130186.58
  1886. lat (nsec): min=0, max=1628.3M, avg=86669252.57, stdev=150470281.13
  1887. clat percentiles (usec):
  1888. | 1.00th=[ 963], 5.00th=[ 1221], 10.00th=[ 1500],
  1889. | 20.00th=[ 2147], 30.00th=[ 3228], 40.00th=[ 5604],
  1890. | 50.00th=[ 13173], 60.00th=[ 55837], 70.00th=[ 93848],
  1891. | 80.00th=[ 123208], 90.00th=[ 240124], 95.00th=[ 421528],
  1892. | 99.00th=[ 708838], 99.50th=[ 801113], 99.90th=[1098908],
  1893. | 99.95th=[1115685], 99.99th=[1635779]
  1894. bw ( KiB/s): min= 0, max= 4555, per=52.95%, avg=2953.00, stdev=2265.57, samples=2
  1895. iops : min= 0, max= 71, avg=46.00, stdev=35.36, samples=2
  1896. lat (usec) : 100=0.02%, 250=0.04%, 500=0.04%, 750=0.07%, 1000=1.39%
  1897. lat (msec) : 2=16.67%, 4=16.38%, 10=13.04%, 20=4.75%, 50=6.41%
  1898. lat (msec) : 100=13.20%, 250=18.62%, 500=5.99%, 750=2.67%, 1000=0.58%
  1899. cpu : usr=0.05%, sys=0.20%, ctx=10208, majf=0, minf=30
  1900. IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  1901. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  1902. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  1903. issued rwts: total=0,5538,0,0 short=0,0,0,0 dropped=0,0,0,0
  1904. latency : target=0, window=0, percentile=100.00%, depth=4
  1905. read: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:34:55 2019
  1906. read: IOPS=529, BW=33.0MiB/s (34.7MB/s)(3966MiB/120019msec)
  1907. slat (nsec): min=0, max=482645k, avg=34306.21, stdev=2614937.41
  1908. clat (nsec): min=0, max=1336.0M, avg=7539479.66, stdev=48759461.40
  1909. lat (nsec): min=0, max=1336.0M, avg=7574591.66, stdev=48839095.66
  1910. clat percentiles (nsec):
  1911. | 1.00th=[ 708], 5.00th=[ 716], 10.00th=[ 716],
  1912. | 20.00th=[ 724], 30.00th=[ 732], 40.00th=[ 740],
  1913. | 50.00th=[ 804], 60.00th=[ 868], 70.00th=[ 964],
  1914. | 80.00th=[ 8640], 90.00th=[ 1089536], 95.00th=[ 32636928],
  1915. | 99.00th=[ 217055232], 99.50th=[ 404750336], 99.90th=[ 633339904],
  1916. | 99.95th=[ 742391808], 99.99th=[1149239296]
  1917. bw ( KiB/s): min= 0, max=46913, per=93.04%, avg=33844.00, stdev=18482.36, samples=2
  1918. iops : min= 0, max= 735, avg=529.50, stdev=290.62, samples=2
  1919. lat (nsec) : 750=44.39%, 1000=26.82%
  1920. lat (usec) : 2=7.58%, 4=0.06%, 10=4.95%, 20=2.19%, 50=0.69%
  1921. lat (usec) : 100=0.15%, 250=0.06%, 500=0.17%, 750=0.85%, 1000=1.70%
  1922. lat (msec) : 2=2.34%, 4=1.32%, 10=0.59%, 20=0.39%, 50=2.28%
  1923. lat (msec) : 100=1.84%, 250=0.73%, 500=0.67%, 750=0.20%, 1000=0.03%
  1924. cpu : usr=0.16%, sys=0.53%, ctx=16677, majf=0, minf=30
  1925. IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  1926. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  1927. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  1928. issued rwts: total=63607,0,0,0 short=0,0,0,0 dropped=0,0,0,0
  1929. latency : target=0, window=0, percentile=100.00%, depth=4
  1930. readwrite: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:34:55 2019
  1931. read: IOPS=39, BW=2534KiB/s (2595kB/s)(297MiB/120003msec)
  1932. slat (nsec): min=0, max=2602.2k, avg=26859.27, stdev=68750.48
  1933. clat (nsec): min=0, max=750910k, avg=6636280.46, stdev=36858791.12
  1934. lat (nsec): min=0, max=750930k, avg=6664125.19, stdev=36867160.13
  1935. clat percentiles (nsec):
  1936. | 1.00th=[ 716], 5.00th=[ 732], 10.00th=[ 748],
  1937. | 20.00th=[ 804], 30.00th=[ 884], 40.00th=[ 980],
  1938. | 50.00th=[ 1128], 60.00th=[ 1272], 70.00th=[ 1592],
  1939. | 80.00th=[ 84480], 90.00th=[ 1613824], 95.00th=[ 38535168],
  1940. | 99.00th=[160432128], 99.50th=[295698432], 99.90th=[480247808],
  1941. | 99.95th=[488636416], 99.99th=[750780416]
  1942. bw ( KiB/s): min= 0, max= 3603, per=6.97%, avg=2534.00, stdev=1511.79, samples=2
  1943. iops : min= 0, max= 56, avg=39.00, stdev=24.04, samples=2
  1944. write: IOPS=40, BW=2625KiB/s (2688kB/s)(308MiB/120003msec)
  1945. slat (nsec): min=0, max=750900k, avg=473560.27, stdev=12692496.43
  1946. clat (nsec): min=0, max=1632.7M, avg=90831011.95, stdev=157168249.28
  1947. lat (nsec): min=0, max=1632.8M, avg=91305477.41, stdev=157556825.13
  1948. clat percentiles (usec):
  1949. | 1.00th=[ 1004], 5.00th=[ 1237], 10.00th=[ 1467],
  1950. | 20.00th=[ 2147], 30.00th=[ 3326], 40.00th=[ 5866],
  1951. | 50.00th=[ 15795], 60.00th=[ 62129], 70.00th=[ 96994],
  1952. | 80.00th=[ 127402], 90.00th=[ 256902], 95.00th=[ 442500],
  1953. | 99.00th=[ 742392], 99.50th=[ 868221], 99.90th=[1233126],
  1954. | 99.95th=[1283458], 99.99th=[1635779]
  1955. bw ( KiB/s): min= 0, max= 3730, per=47.02%, avg=2622.50, stdev=1566.24, samples=2
  1956. iops : min= 0, max= 58, avg=40.50, stdev=24.75, samples=2
  1957. lat (nsec) : 750=5.76%, 1000=14.44%
  1958. lat (usec) : 2=15.05%, 4=0.09%, 10=0.52%, 20=1.30%, 50=1.23%
  1959. lat (usec) : 100=1.25%, 250=1.03%, 500=1.04%, 750=0.69%, 1000=1.35%
  1960. lat (msec) : 2=10.10%, 4=8.67%, 10=6.74%, 20=2.49%, 50=4.65%
  1961. lat (msec) : 100=8.20%, 250=9.94%, 500=3.54%, 750=1.44%, 1000=0.35%
  1962. cpu : usr=0.06%, sys=0.23%, ctx=11015, majf=0, minf=31
  1963. IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  1964. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  1965. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  1966. issued rwts: total=4755,4919,0,0 short=0,0,0,0 dropped=0,0,0,0
  1967. latency : target=0, window=0, percentile=100.00%, depth=4
  1968.  
  1969. Run status group 0 (all jobs):
  1970. READ: bw=35.5MiB/s (37.2MB/s), 2534KiB/s-33.0MiB/s (2595kB/s-34.7MB/s), io=4263MiB (4470MB), run=120003-120019msec
  1971. WRITE: bw=5577KiB/s (5711kB/s), 2625KiB/s-2954KiB/s (2688kB/s-3025kB/s), io=654MiB (686MB), run=120003-120061msec
  1972.  
  1973. Disk stats (read/write):
  1974. rbd178: ios=9145/10566, merge=0/3, ticks=516068/791944, in_queue=1297922, util=13.11%
  1975. Jobs: 51 (f=51): [_(2),m(1)][99.8%][r=1375M,w=1387M][r=21.5k,w=21.7k IOPS][eta 00m:54s] Jobs: 51 (f=51): [_(2),m(1)][99.8%][r=1410M,w=1427M][r=22.0k,w=22.3k IOPS][eta 00m:53s] <example-benchmark-fio-server-794-benchmark-xkzrq>
  1976. randwrite: (groupid=0, jobs=1): err= 0: pid=11: Wed May 8 20:34:57 2019
  1977. write: IOPS=39, BW=2511KiB/s (2572kB/s)(294MiB/120011msec)
  1978. slat (nsec): min=0, max=830857k, avg=575567.16, stdev=16158635.08
  1979. clat (nsec): min=0, max=1469.7M, avg=101665439.36, stdev=163496335.02
  1980. lat (nsec): min=0, max=1469.8M, avg=102242584.08, stdev=164095874.59
  1981. clat percentiles (usec):
  1982. | 1.00th=[ 1020], 5.00th=[ 1270], 10.00th=[ 1598],
  1983. | 20.00th=[ 2245], 30.00th=[ 3720], 40.00th=[ 9503],
  1984. | 50.00th=[ 42730], 60.00th=[ 72877], 70.00th=[ 102237],
  1985. | 80.00th=[ 143655], 90.00th=[ 291505], 95.00th=[ 438305],
  1986. | 99.00th=[ 801113], 99.50th=[ 918553], 99.90th=[1434452],
  1987. | 99.95th=[1468007], 99.99th=[1468007]
  1988. bw ( KiB/s): min= 0, max= 3129, per=48.79%, avg=2509.00, stdev=876.81, samples=2
  1989. iops : min= 0, max= 48, avg=38.50, stdev=13.44, samples=2
  1990. lat (usec) : 2=0.02%, 4=0.02%, 500=0.02%, 750=0.04%, 1000=0.70%
  1991. lat (msec) : 2=15.68%, 4=14.53%, 10=9.31%, 20=4.12%, 50=7.39%
  1992. lat (msec) : 100=17.68%, 250=18.25%, 500=8.35%, 750=2.57%, 1000=1.13%
  1993. cpu : usr=0.06%, sys=0.17%, ctx=8853, majf=0, minf=26
  1994. IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  1995. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  1996. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  1997. issued rwts: total=0,4706,0,0 short=0,0,0,0 dropped=0,0,0,0
  1998. latency : target=0, window=0, percentile=100.00%, depth=4
  1999. read: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:34:57 2019
  2000. read: IOPS=321, BW=20.1MiB/s (21.1MB/s)(2412MiB/120011msec)
  2001. slat (nsec): min=0, max=593474k, avg=62460.39, stdev=4735684.38
  2002. clat (nsec): min=0, max=1023.2M, avg=12403573.49, stdev=60386500.08
  2003. lat (nsec): min=0, max=1023.3M, avg=12466969.14, stdev=60576303.42
  2004. clat percentiles (nsec):
  2005. | 1.00th=[ 708], 5.00th=[ 716], 10.00th=[ 716],
  2006. | 20.00th=[ 724], 30.00th=[ 732], 40.00th=[ 740],
  2007. | 50.00th=[ 812], 60.00th=[ 876], 70.00th=[ 996],
  2008. | 80.00th=[ 1528], 90.00th=[ 11075584], 95.00th=[ 57409536],
  2009. | 99.00th=[346030080], 99.50th=[467664896], 99.90th=[700448768],
  2010. | 99.95th=[775946240], 99.99th=[952107008]
  2011. bw ( KiB/s): min= 0, max=27268, per=89.00%, avg=20583.50, stdev=9453.31, samples=2
  2012. iops : min= 0, max= 426, avg=321.50, stdev=147.79, samples=2
  2013. lat (nsec) : 750=42.48%, 1000=27.64%
  2014. lat (usec) : 2=10.36%, 4=0.04%, 10=0.65%, 20=0.38%, 50=0.32%
  2015. lat (usec) : 100=0.13%, 250=0.04%, 500=0.20%, 750=0.85%, 1000=1.83%
  2016. lat (msec) : 2=2.50%, 4=1.57%, 10=0.91%, 20=0.91%, 50=3.52%
  2017. lat (msec) : 100=2.58%, 250=1.53%, 500=1.16%, 750=0.33%, 1000=0.06%
  2018. cpu : usr=0.11%, sys=0.36%, ctx=13842, majf=0, minf=109
  2019. IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  2020. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  2021. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  2022. issued rwts: total=38631,0,0,0 short=0,0,0,0 dropped=0,0,0,0
  2023. latency : target=0, window=0, percentile=100.00%, depth=4
  2024. readwrite: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:34:57 2019
  2025. read: IOPS=39, BW=2544KiB/s (2605kB/s)(298MiB/120003msec)
  2026. slat (nsec): min=0, max=578056k, avg=151252.96, stdev=8368754.74
  2027. clat (nsec): min=0, max=1230.4M, avg=8238059.24, stdev=45356635.31
  2028. lat (nsec): min=0, max=1230.4M, avg=8390234.99, stdev=46106318.45
  2029. clat percentiles (nsec):
  2030. | 1.00th=[ 716], 5.00th=[ 732], 10.00th=[ 740],
  2031. | 20.00th=[ 780], 30.00th=[ 876], 40.00th=[ 988],
  2032. | 50.00th=[ 1144], 60.00th=[ 1272], 70.00th=[ 1560],
  2033. | 80.00th=[ 232448], 90.00th=[ 3686400], 95.00th=[ 41156608],
  2034. | 99.00th=[ 221249536], 99.50th=[ 354418688], 99.90th=[ 480247808],
  2035. | 99.95th=[ 801112064], 99.99th=[1233125376]
  2036. bw ( KiB/s): min= 0, max= 3363, per=11.00%, avg=2543.00, stdev=1159.66, samples=2
  2037. iops : min= 0, max= 52, avg=39.00, stdev=18.38, samples=2
  2038. write: IOPS=41, BW=2632KiB/s (2695kB/s)(308MiB/120003msec)
  2039. slat (nsec): min=0, max=451778k, avg=272568.46, stdev=6436634.86
  2040. clat (nsec): min=0, max=1469.7M, avg=88949844.53, stdev=152195506.47
  2041. lat (nsec): min=0, max=1469.8M, avg=89223694.54, stdev=152274142.94
  2042. clat percentiles (usec):
  2043. | 1.00th=[ 1045], 5.00th=[ 1401], 10.00th=[ 1729],
  2044. | 20.00th=[ 2540], 30.00th=[ 4015], 40.00th=[ 7308],
  2045. | 50.00th=[ 19792], 60.00th=[ 54789], 70.00th=[ 87557],
  2046. | 80.00th=[ 124257], 90.00th=[ 265290], 95.00th=[ 421528],
  2047. | 99.00th=[ 742392], 99.50th=[ 876610], 99.90th=[ 952108],
  2048. | 99.95th=[1434452], 99.99th=[1468007]
  2049. bw ( KiB/s): min= 0, max= 3460, per=51.16%, avg=2630.50, stdev=1173.09, samples=2
  2050. iops : min= 0, max= 54, avg=41.00, stdev=18.38, samples=2
  2051. lat (nsec) : 750=6.93%, 1000=13.11%
  2052. lat (usec) : 2=15.45%, 4=0.07%, 10=0.41%, 20=0.79%, 50=0.92%
  2053. lat (usec) : 100=0.97%, 250=0.75%, 500=0.82%, 750=0.54%, 1000=1.22%
  2054. lat (msec) : 2=8.42%, 4=9.14%, 10=7.65%, 20=3.96%, 50=5.69%
  2055. lat (msec) : 100=9.08%, 250=8.22%, 500=4.08%, 750=1.28%, 1000=0.48%
  2056. cpu : usr=0.07%, sys=0.23%, ctx=11104, majf=0, minf=30
  2057. IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  2058. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  2059. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  2060. issued rwts: total=4771,4932,0,0 short=0,0,0,0 dropped=0,0,0,0
  2061. latency : target=0, window=0, percentile=100.00%, depth=4
  2062.  
  2063. Run status group 0 (all jobs):
  2064. READ: bw=22.6MiB/s (23.7MB/s), 2544KiB/s-20.1MiB/s (2605kB/s-21.1MB/s), io=2711MiB (2842MB), run=120003-120011msec
  2065. WRITE: bw=5143KiB/s (5266kB/s), 2511KiB/s-2632KiB/s (2572kB/s-2695kB/s), io=603MiB (632MB), run=120003-120011msec
  2066.  
  2067. Disk stats (read/write):
  2068. rbd206: ios=9196/10145, merge=0/2, ticks=549346/854017, in_queue=1393497, util=13.88%
  2069. <example-benchmark-fio-server-694-benchmark-wvfzs>
  2070. randwrite: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:34:57 2019
  2071. write: IOPS=40, BW=2571KiB/s (2633kB/s)(307MiB/122324msec)
  2072. slat (nsec): min=0, max=25427k, avg=195885.07, stdev=562273.47
  2073. clat (nsec): min=0, max=8959.5M, avg=99109794.52, stdev=322776851.58
  2074. lat (nsec): min=0, max=8959.6M, avg=99307113.25, stdev=322783856.59
  2075. clat percentiles (usec):
  2076. | 1.00th=[ 848], 5.00th=[ 1827], 10.00th=[ 2802],
  2077. | 20.00th=[ 4293], 30.00th=[ 6718], 40.00th=[ 10421],
  2078. | 50.00th=[ 15533], 60.00th=[ 27395], 70.00th=[ 54264],
  2079. | 80.00th=[ 130548], 90.00th=[ 287310], 95.00th=[ 429917],
  2080. | 99.00th=[ 717226], 99.50th=[1317012], 99.90th=[6878659],
  2081. | 99.95th=[8556381], 99.99th=[8925479]
  2082. bw ( KiB/s): min= 0, max= 3233, per=47.42%, avg=2618.50, stdev=869.03, samples=2
  2083. iops : min= 0, max= 50, avg=40.50, stdev=13.44, samples=2
  2084. lat (usec) : 2=0.02%, 4=0.02%, 100=0.02%, 250=0.02%, 500=0.06%
  2085. lat (usec) : 750=0.37%, 1000=1.06%
  2086. lat (msec) : 2=4.28%, 4=11.89%, 10=21.32%, 20=15.64%, 50=14.05%
  2087. lat (msec) : 100=8.90%, 250=10.59%, 500=8.33%, 750=2.61%, 1000=0.20%
  2088. cpu : usr=0.03%, sys=0.15%, ctx=7781, majf=0, minf=30
  2089. IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  2090. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  2091. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  2092. issued rwts: total=0,4911,0,0 short=0,0,0,0 dropped=0,0,0,0
  2093. latency : target=0, window=0, percentile=100.00%, depth=4
  2094. read: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:34:57 2019
  2095. read: IOPS=68, BW=4371KiB/s (4476kB/s)(513MiB/120237msec)
  2096. slat (nsec): min=0, max=9176.8k, avg=52739.78, stdev=188486.51
  2097. clat (nsec): min=0, max=1825.5M, avg=58615474.01, stdev=197991591.34
  2098. lat (nsec): min=0, max=1825.5M, avg=58669431.87, stdev=198008807.92
  2099. clat percentiles (nsec):
  2100. | 1.00th=[ 716], 5.00th=[ 732], 10.00th=[ 772],
  2101. | 20.00th=[ 820], 30.00th=[ 884], 40.00th=[ 1032],
  2102. | 50.00th=[ 1256], 60.00th=[ 1640], 70.00th=[ 1417216],
  2103. | 80.00th=[ 3424256], 90.00th=[ 49020928], 95.00th=[ 599785472],
  2104. | 99.00th=[ 952107008], 99.50th=[1069547520], 99.90th=[1350565888],
  2105. | 99.95th=[1518338048], 99.99th=[1820327936]
  2106. bw ( KiB/s): min= 0, max= 4448, per=60.15%, avg=4376.50, stdev=101.12, samples=2
  2107. iops : min= 0, max= 69, avg=68.00, stdev= 1.41, samples=2
  2108. lat (nsec) : 750=6.54%, 1000=32.33%
  2109. lat (usec) : 2=21.82%, 4=0.22%, 10=0.11%, 20=0.52%, 50=0.17%
  2110. lat (usec) : 100=0.19%, 250=0.10%, 500=0.06%, 750=0.82%, 1000=3.16%
  2111. lat (msec) : 2=7.88%, 4=7.27%, 10=4.92%, 20=2.01%, 50=1.99%
  2112. lat (msec) : 100=0.97%, 250=0.83%, 500=2.22%, 750=3.12%, 1000=2.05%
  2113. cpu : usr=0.04%, sys=0.11%, ctx=6246, majf=0, minf=30
  2114. IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  2115. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  2116. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  2117. issued rwts: total=8209,0,0,0 short=0,0,0,0 dropped=0,0,0,0
  2118. latency : target=0, window=0, percentile=100.00%, depth=4
  2119. readwrite: (groupid=0, jobs=1): err= 0: pid=14: Wed May 8 20:34:57 2019
  2120. read: IOPS=45, BW=2911KiB/s (2981kB/s)(341MiB/120007msec)
  2121. slat (nsec): min=0, max=3786.0k, avg=42845.43, stdev=115101.89
  2122. clat (nsec): min=0, max=809090k, avg=19029910.56, stdev=72180861.64
  2123. lat (nsec): min=0, max=809121k, avg=19073885.97, stdev=72190890.86
  2124. clat percentiles (nsec):
  2125. | 1.00th=[ 724], 5.00th=[ 740], 10.00th=[ 748],
  2126. | 20.00th=[ 868], 30.00th=[ 1004], 40.00th=[ 1144],
  2127. | 50.00th=[ 1304], 60.00th=[ 52480], 70.00th=[ 1105920],
  2128. | 80.00th=[ 3227648], 90.00th=[ 43778048], 95.00th=[ 87556096],
  2129. | 99.00th=[446693376], 99.50th=[509607936], 99.90th=[734003200],
  2130. | 99.95th=[784334848], 99.99th=[809500672]
  2131. bw ( KiB/s): min= 0, max= 4526, per=39.99%, avg=2909.50, stdev=2286.08, samples=2
  2132. iops : min= 0, max= 70, avg=45.00, stdev=35.36, samples=2
  2133. write: IOPS=46, BW=3008KiB/s (3081kB/s)(353MiB/120007msec)
  2134. slat (nsec): min=0, max=5596.6k, avg=184642.48, stdev=236894.22
  2135. clat (nsec): min=0, max=852576k, avg=66695651.86, stdev=130745032.75
  2136. lat (nsec): min=0, max=852605k, avg=66881887.03, stdev=130698526.14
  2137. clat percentiles (usec):
  2138. | 1.00th=[ 1020], 5.00th=[ 1221], 10.00th=[ 1369], 20.00th=[ 1696],
  2139. | 30.00th=[ 2212], 40.00th=[ 3163], 50.00th=[ 5145], 60.00th=[ 10945],
  2140. | 70.00th=[ 49546], 80.00th=[102237], 90.00th=[200279], 95.00th=[383779],
  2141. | 99.00th=[616563], 99.50th=[683672], 99.90th=[750781], 99.95th=[826278],
  2142. | 99.99th=[851444]
  2143. bw ( KiB/s): min= 0, max= 4664, per=54.45%, avg=3007.00, stdev=2343.35, samples=2
  2144. iops : min= 0, max= 72, avg=46.50, stdev=36.06, samples=2
  2145. lat (nsec) : 750=4.73%, 1000=9.89%
  2146. lat (usec) : 2=14.22%, 4=0.05%, 10=0.07%, 20=0.32%, 50=0.23%
  2147. lat (usec) : 100=0.32%, 250=0.35%, 500=0.22%, 750=1.08%, 1000=2.54%
  2148. lat (msec) : 2=17.03%, 4=12.00%, 10=8.42%, 20=3.30%, 50=5.66%
  2149. lat (msec) : 100=6.99%, 250=7.36%, 500=3.60%, 750=1.53%, 1000=0.12%
  2150. cpu : usr=0.09%, sys=0.25%, ctx=14803, majf=0, minf=37
  2151. IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  2152. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  2153. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  2154. issued rwts: total=5457,5640,0,0 short=0,0,0,0 dropped=0,0,0,0
  2155. latency : target=0, window=0, percentile=100.00%, depth=4
  2156.  
  2157. Run status group 0 (all jobs):
  2158. READ: bw=7277KiB/s (7451kB/s), 2911KiB/s-4371KiB/s (2981kB/s-4476kB/s), io=854MiB (896MB), run=120007-120237msec
  2159. WRITE: bw=5522KiB/s (5655kB/s), 2571KiB/s-3008KiB/s (2633kB/s-3081kB/s), io=660MiB (692MB), run=120007-122324msec
  2160.  
  2161. Disk stats (read/write):
  2162. rbd204: ios=5634/12337, merge=0/3, ticks=642225/878157, in_queue=1511367, util=4.27%
  2163. <example-benchmark-fio-server-612-benchmark-2dg6w>
  2164. randwrite: (groupid=0, jobs=1): err= 0: pid=11: Wed May 8 20:34:58 2019
  2165. write: IOPS=27, BW=1758KiB/s (1800kB/s)(207MiB/120534msec)
  2166. slat (nsec): min=0, max=4257.1M, avg=2010805.23, stdev=76515967.14
  2167. clat (nsec): min=0, max=6824.8M, avg=143809839.17, stdev=306619330.65
  2168. lat (nsec): min=0, max=6833.9M, avg=145820705.46, stdev=319679458.15
  2169. clat percentiles (usec):
  2170. | 1.00th=[ 1221], 5.00th=[ 1663], 10.00th=[ 2114],
  2171. | 20.00th=[ 3032], 30.00th=[ 4178], 40.00th=[ 6063],
  2172. | 50.00th=[ 9634], 60.00th=[ 16909], 70.00th=[ 88605],
  2173. | 80.00th=[ 333448], 90.00th=[ 492831], 95.00th=[ 616563],
  2174. | 99.00th=[ 809501], 99.50th=[1035994], 99.90th=[3506439],
  2175. | 99.95th=[6811550], 99.99th=[6811550]
  2176. bw ( KiB/s): min= 0, max= 2243, per=33.59%, avg=1763.00, stdev=678.82, samples=2
  2177. iops : min= 0, max= 35, avg=27.50, stdev=10.61, samples=2
  2178. lat (usec) : 500=0.03%, 750=0.03%, 1000=0.18%
  2179. lat (msec) : 2=8.41%, 4=19.78%, 10=22.32%, 20=11.82%, 50=6.23%
  2180. lat (msec) : 100=1.45%, 250=5.47%, 500=14.45%, 750=8.50%, 1000=0.88%
  2181. cpu : usr=0.02%, sys=0.12%, ctx=5571, majf=0, minf=30
  2182. IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  2183. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  2184. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  2185. issued rwts: total=0,3307,0,0 short=0,0,0,0 dropped=0,0,0,0
  2186. latency : target=0, window=0, percentile=100.00%, depth=4
  2187. read: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:34:58 2019
  2188. read: IOPS=136, BW=8764KiB/s (8975kB/s)(1032MiB/120562msec)
  2189. slat (nsec): min=0, max=12461k, avg=36945.42, stdev=180100.44
  2190. clat (nsec): min=0, max=1972.8M, avg=29204481.49, stdev=143415318.41
  2191. lat (nsec): min=0, max=1972.8M, avg=29242530.82, stdev=143430361.26
  2192. clat percentiles (nsec):
  2193. | 1.00th=[ 716], 5.00th=[ 732], 10.00th=[ 756],
  2194. | 20.00th=[ 788], 30.00th=[ 820], 40.00th=[ 868],
  2195. | 50.00th=[ 900], 60.00th=[ 1032], 70.00th=[ 1320],
  2196. | 80.00th=[ 995328], 90.00th=[ 4685824], 95.00th=[ 51642368],
  2197. | 99.00th=[ 826277888], 99.50th=[ 952107008], 99.90th=[1333788672],
  2198. | 99.95th=[1535115264], 99.99th=[1837105152]
  2199. bw ( KiB/s): min= 0, max= 9481, per=72.35%, avg=8802.50, stdev=959.54, samples=2
  2200. iops : min= 0, max= 148, avg=137.00, stdev=15.56, samples=2
  2201. lat (nsec) : 750=9.74%, 1000=49.14%
  2202. lat (usec) : 2=17.42%, 4=0.25%, 10=0.19%, 20=0.59%, 50=0.16%
  2203. lat (usec) : 100=0.12%, 250=0.12%, 500=0.13%, 750=0.43%, 1000=1.79%
  2204. lat (msec) : 2=4.74%, 4=4.43%, 10=3.01%, 20=1.31%, 50=1.40%
  2205. lat (msec) : 100=0.57%, 250=0.58%, 500=1.05%, 750=1.38%, 1000=1.13%
  2206. cpu : usr=0.06%, sys=0.18%, ctx=7446, majf=0, minf=29
  2207. IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  2208. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  2209. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  2210. issued rwts: total=16507,0,0,0 short=0,0,0,0 dropped=0,0,0,0
  2211. latency : target=0, window=0, percentile=100.00%, depth=4
  2212. readwrite: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:34:58 2019
  2213. read: IOPS=53, BW=3419KiB/s (3501kB/s)(401MiB/120002msec)
  2214. slat (nsec): min=0, max=1954.4k, avg=31149.84, stdev=77107.69
  2215. clat (nsec): min=0, max=734117k, avg=11037698.80, stdev=54963807.10
  2216. lat (nsec): min=0, max=734145k, avg=11069791.59, stdev=54971279.93
  2217. clat percentiles (nsec):
  2218. | 1.00th=[ 724], 5.00th=[ 732], 10.00th=[ 740],
  2219. | 20.00th=[ 812], 30.00th=[ 908], 40.00th=[ 1032],
  2220. | 50.00th=[ 1176], 60.00th=[ 1304], 70.00th=[ 1816],
  2221. | 80.00th=[ 856064], 90.00th=[ 5275648], 95.00th=[ 51118080],
  2222. | 99.00th=[320864256], 99.50th=[455081984], 99.90th=[616562688],
  2223. | 99.95th=[650117120], 99.99th=[734003200]
  2224. bw ( KiB/s): min= 0, max= 4692, per=28.10%, avg=3418.50, stdev=1801.00, samples=2
  2225. iops : min= 0, max= 73, avg=53.00, stdev=28.28, samples=2
  2226. write: IOPS=54, BW=3508KiB/s (3592kB/s)(411MiB/120002msec)
  2227. slat (nsec): min=0, max=10682k, avg=192837.44, stdev=258303.43
  2228. clat (nsec): min=0, max=1267.3M, avg=62261381.68, stdev=127104471.59
  2229. lat (nsec): min=0, max=1267.3M, avg=62455492.49, stdev=127055529.35
  2230. clat percentiles (usec):
  2231. | 1.00th=[ 1012], 5.00th=[ 1221], 10.00th=[ 1401],
  2232. | 20.00th=[ 1795], 30.00th=[ 2376], 40.00th=[ 3458],
  2233. | 50.00th=[ 5997], 60.00th=[ 14484], 70.00th=[ 32900],
  2234. | 80.00th=[ 99091], 90.00th=[ 187696], 95.00th=[ 325059],
  2235. | 99.00th=[ 675283], 99.50th=[ 750781], 99.90th=[ 901776],
  2236. | 99.95th=[ 977273], 99.99th=[1266680]
  2237. bw ( KiB/s): min= 0, max= 4778, per=66.77%, avg=3505.00, stdev=1800.29, samples=2
  2238. iops : min= 0, max= 74, avg=54.00, stdev=28.28, samples=2
  2239. lat (nsec) : 750=5.97%, 1000=12.84%
  2240. lat (usec) : 2=15.87%, 4=0.08%, 10=0.23%, 20=0.78%, 50=0.68%
  2241. lat (usec) : 100=0.62%, 250=0.69%, 500=0.44%, 750=0.65%, 1000=1.93%
  2242. lat (msec) : 2=14.01%, 4=11.19%, 10=7.22%, 20=4.84%, 50=5.47%
  2243. lat (msec) : 100=5.08%, 250=7.06%, 500=3.13%, 750=0.98%, 1000=0.25%
  2244. cpu : usr=0.09%, sys=0.31%, ctx=15021, majf=0, minf=31
  2245. IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  2246. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  2247. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  2248. issued rwts: total=6410,6574,0,0 short=0,0,0,0 dropped=0,0,0,0
  2249. latency : target=0, window=0, percentile=100.00%, depth=4
  2250.  
  2251. Run status group 0 (all jobs):
  2252. READ: bw=11.9MiB/s (12.5MB/s), 3419KiB/s-8764KiB/s (3501kB/s-8975kB/s), io=1433MiB (1502MB), run=120002-120562msec
  2253. WRITE: bw=5250KiB/s (5376kB/s), 1758KiB/s-3508KiB/s (1800kB/s-3592kB/s), io=618MiB (648MB), run=120002-120534msec
  2254.  
  2255. Disk stats (read/write):
  2256. rbd279: ios=5460/10356, merge=0/3, ticks=596117/878665, in_queue=1466829, util=4.57%
  2257. Jobs: 48 (f=45): [_(2),m(1)][99.8%][r=1436M,w=1434M][r=22.4k,w=22.4k IOPS][eta 00m:52s] <example-benchmark-fio-server-898-benchmark-2q9b7>
  2258. randwrite: (groupid=0, jobs=1): err= 0: pid=11: Wed May 8 20:34:58 2019
  2259. write: IOPS=13, BW=835KiB/s (855kB/s)(98.1MiB/120351msec)
  2260. slat (nsec): min=0, max=2022.3M, avg=2728134.72, stdev=58249237.85
  2261. clat (nsec): min=0, max=3275.6M, avg=304687648.20, stdev=456863340.63
  2262. lat (nsec): min=0, max=3275.7M, avg=307412545.76, stdev=460022219.28
  2263. clat percentiles (usec):
  2264. | 1.00th=[ 1303], 5.00th=[ 2024], 10.00th=[ 2704],
  2265. | 20.00th=[ 4490], 30.00th=[ 6915], 40.00th=[ 11338],
  2266. | 50.00th=[ 18744], 60.00th=[ 50070], 70.00th=[ 392168],
  2267. | 80.00th=[ 633340], 90.00th=[1010828], 95.00th=[1317012],
  2268. | 99.00th=[1635779], 99.50th=[1920992], 99.90th=[2231370],
  2269. | 99.95th=[3271558], 99.99th=[3271558]
  2270. bw ( KiB/s): min= 0, max= 1076, per=21.35%, avg=833.00, stdev=343.65, samples=2
  2271. iops : min= 0, max= 16, avg=12.50, stdev= 4.95, samples=2
  2272. lat (usec) : 1000=0.19%
  2273. lat (msec) : 2=4.53%, 4=12.95%, 10=19.85%, 20=13.85%, 50=8.74%
  2274. lat (msec) : 100=1.91%, 250=0.89%, 500=11.81%, 750=8.42%, 1000=6.76%
  2275. cpu : usr=0.02%, sys=0.07%, ctx=2848, majf=0, minf=30
  2276. IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  2277. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  2278. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  2279. issued rwts: total=0,1567,0,0 short=0,0,0,0 dropped=0,0,0,0
  2280. latency : target=0, window=0, percentile=100.00%, depth=4
  2281. read: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:34:58 2019
  2282. read: IOPS=307, BW=19.2MiB/s (20.1MB/s)(2315MiB/120566msec)
  2283. slat (nsec): min=0, max=1464.6M, avg=73420.58, stdev=8011473.67
  2284. clat (nsec): min=0, max=2370.1M, avg=12938782.83, stdev=99222958.48
  2285. lat (nsec): min=0, max=2370.1M, avg=13013350.53, stdev=99551960.23
  2286. clat percentiles (nsec):
  2287. | 1.00th=[ 708], 5.00th=[ 724], 10.00th=[ 732],
  2288. | 20.00th=[ 772], 30.00th=[ 788], 40.00th=[ 804],
  2289. | 50.00th=[ 836], 60.00th=[ 868], 70.00th=[ 900],
  2290. | 80.00th=[ 1020], 90.00th=[ 11840], 95.00th=[ 2703360],
  2291. | 99.00th=[ 624951296], 99.50th=[ 792723456], 99.90th=[1199570944],
  2292. | 99.95th=[1384120320], 99.99th=[1937768448]
  2293. bw ( KiB/s): min= 0, max=26548, per=87.30%, avg=19751.50, stdev=9611.70, samples=2
  2294. iops : min= 0, max= 415, avg=308.50, stdev=150.61, samples=2
  2295. lat (nsec) : 750=12.45%, 1000=67.02%
  2296. lat (usec) : 2=9.06%, 4=0.23%, 10=0.46%, 20=2.23%, 50=0.58%
  2297. lat (usec) : 100=0.18%, 250=0.09%, 500=0.13%, 750=0.15%, 1000=0.44%
  2298. lat (msec) : 2=1.38%, 4=1.28%, 10=1.15%, 20=0.51%, 50=0.46%
  2299. lat (msec) : 100=0.24%, 250=0.24%, 500=0.46%, 750=0.65%, 1000=0.42%
  2300. cpu : usr=0.08%, sys=0.34%, ctx=5906, majf=0, minf=30
  2301. IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  2302. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  2303. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  2304. issued rwts: total=37091,0,0,0 short=0,0,0,0 dropped=0,0,0,0
  2305. latency : target=0, window=0, percentile=100.00%, depth=4
  2306. readwrite: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:34:58 2019
  2307. read: IOPS=46, BW=2977KiB/s (3048kB/s)(349MiB/120021msec)
  2308. slat (nsec): min=0, max=5457.3k, avg=23255.68, stdev=98205.93
  2309. clat (nsec): min=0, max=786199k, avg=7736512.34, stdev=47943367.55
  2310. lat (nsec): min=0, max=786220k, avg=7760701.96, stdev=47953450.62
  2311. clat percentiles (nsec):
  2312. | 1.00th=[ 708], 5.00th=[ 724], 10.00th=[ 724],
  2313. | 20.00th=[ 748], 30.00th=[ 828], 40.00th=[ 916],
  2314. | 50.00th=[ 1048], 60.00th=[ 1192], 70.00th=[ 1368],
  2315. | 80.00th=[ 19328], 90.00th=[ 978944], 95.00th=[ 24772608],
  2316. | 99.00th=[242221056], 99.50th=[404750336], 99.90th=[608174080],
  2317. | 99.95th=[700448768], 99.99th=[784334848]
  2318. bw ( KiB/s): min= 0, max= 4710, per=13.16%, avg=2976.50, stdev=2451.54, samples=2
  2319. iops : min= 0, max= 73, avg=46.00, stdev=38.18, samples=2
  2320. write: IOPS=48, BW=3076KiB/s (3150kB/s)(361MiB/120021msec)
  2321. slat (nsec): min=0, max=6098.2k, avg=209590.76, stdev=291288.10
  2322. clat (nsec): min=0, max=1088.8M, avg=75761080.52, stdev=143915707.64
  2323. lat (nsec): min=0, max=1088.8M, avg=75971772.86, stdev=143854208.57
  2324. clat percentiles (usec):
  2325. | 1.00th=[ 963], 5.00th=[ 1123], 10.00th=[ 1287],
  2326. | 20.00th=[ 1663], 30.00th=[ 2278], 40.00th=[ 3556],
  2327. | 50.00th=[ 6980], 60.00th=[ 21890], 70.00th=[ 68682],
  2328. | 80.00th=[ 112722], 90.00th=[ 231736], 95.00th=[ 400557],
  2329. | 99.00th=[ 708838], 99.50th=[ 817890], 99.90th=[ 968885],
  2330. | 99.95th=[ 994051], 99.99th=[1082131]
  2331. bw ( KiB/s): min= 0, max= 4848, per=78.78%, avg=3074.00, stdev=2508.81, samples=2
  2332. iops : min= 0, max= 75, avg=47.50, stdev=38.89, samples=2
  2333. lat (nsec) : 750=10.49%, 1000=12.60%
  2334. lat (usec) : 2=14.41%, 4=0.04%, 10=0.41%, 20=1.55%, 50=1.02%
  2335. lat (usec) : 100=0.98%, 250=1.01%, 500=0.77%, 750=0.50%, 1000=1.36%
  2336. lat (msec) : 2=13.40%, 4=8.68%, 10=6.46%, 20=3.22%, 50=4.30%
  2337. lat (msec) : 100=6.09%, 250=7.50%, 500=3.56%, 750=1.25%, 1000=0.42%
  2338. cpu : usr=0.06%, sys=0.27%, ctx=11681, majf=0, minf=34
  2339. IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  2340. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  2341. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  2342. issued rwts: total=5582,5765,0,0 short=0,0,0,0 dropped=0,0,0,0
  2343. latency : target=0, window=0, percentile=100.00%, depth=4
  2344.  
  2345. Run status group 0 (all jobs):
  2346. READ: bw=22.1MiB/s (23.2MB/s), 2977KiB/s-19.2MiB/s (3048kB/s-20.1MB/s), io=2664MiB (2793MB), run=120021-120566msec
  2347. WRITE: bw=3902KiB/s (3996kB/s), 835KiB/s-3076KiB/s (855kB/s-3150kB/s), io=459MiB (481MB), run=120021-120351msec
  2348.  
  2349. Disk stats (read/write):
  2350. rbd321: ios=3932/7745, merge=0/1, ticks=559319/910120, in_queue=1463629, util=5.16%
  2351. <example-benchmark-fio-server-417-benchmark-gs9tf>
  2352. randwrite: (groupid=0, jobs=1): err= 0: pid=11: Wed May 8 20:34:58 2019
  2353. write: IOPS=22, BW=1436KiB/s (1471kB/s)(171MiB/121813msec)
  2354. slat (nsec): min=0, max=318460k, avg=653825.73, stdev=10094001.52
  2355. clat (nsec): min=0, max=4853.7M, avg=175745024.98, stdev=330156121.69
  2356. lat (nsec): min=0, max=4853.0M, avg=176399966.50, stdev=330429601.02
  2357. clat percentiles (usec):
  2358. | 1.00th=[ 1074], 5.00th=[ 1762], 10.00th=[ 2606],
  2359. | 20.00th=[ 4424], 30.00th=[ 7046], 40.00th=[ 10945],
  2360. | 50.00th=[ 17433], 60.00th=[ 46400], 70.00th=[ 223347],
  2361. | 80.00th=[ 367002], 90.00th=[ 557843], 95.00th=[ 666895],
  2362. | 99.00th=[ 943719], 99.50th=[1887437], 99.90th=[4798284],
  2363. | 99.95th=[4865393], 99.99th=[4865393]
  2364. bw ( KiB/s): min= 0, max= 1607, per=31.79%, avg=1455.50, stdev=214.25, samples=2
  2365. iops : min= 0, max= 25, avg=22.50, stdev= 3.54, samples=2
  2366. lat (usec) : 4=0.07%, 750=0.07%, 1000=0.66%
  2367. lat (msec) : 2=5.42%, 4=12.23%, 10=19.00%, 20=14.54%, 50=8.53%
  2368. lat (msec) : 100=4.43%, 250=7.14%, 500=15.49%, 750=9.85%, 1000=1.90%
  2369. cpu : usr=0.03%, sys=0.10%, ctx=4795, majf=0, minf=30
  2370. IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  2371. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  2372. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  2373. issued rwts: total=0,2731,0,0 short=0,0,0,0 dropped=0,0,0,0
  2374. latency : target=0, window=0, percentile=100.00%, depth=4
  2375. read: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:34:58 2019
  2376. read: IOPS=123, BW=7924KiB/s (8115kB/s)(934MiB/120724msec)
  2377. slat (nsec): min=0, max=10609k, avg=31616.62, stdev=166024.40
  2378. clat (nsec): min=0, max=1785.3M, avg=32298334.44, stdev=152225712.97
  2379. lat (nsec): min=0, max=1785.3M, avg=32330977.46, stdev=152246360.23
  2380. clat percentiles (nsec):
  2381. | 1.00th=[ 716], 5.00th=[ 724], 10.00th=[ 740],
  2382. | 20.00th=[ 780], 30.00th=[ 796], 40.00th=[ 844],
  2383. | 50.00th=[ 876], 60.00th=[ 908], 70.00th=[ 1112],
  2384. | 80.00th=[ 1496], 90.00th=[ 3096576], 95.00th=[ 61603840],
  2385. | 99.00th=[ 851443712], 99.50th=[ 977272832], 99.90th=[1266679808],
  2386. | 99.95th=[1468006400], 99.99th=[1769996288]
  2387. bw ( KiB/s): min= 0, max= 9050, per=72.45%, avg=7959.00, stdev=1542.91, samples=2
  2388. iops : min= 0, max= 141, avg=124.00, stdev=24.04, samples=2
  2389. lat (nsec) : 750=10.84%, 1000=55.88%
  2390. lat (usec) : 2=14.46%, 4=0.16%, 10=0.30%, 20=0.97%, 50=0.21%
  2391. lat (usec) : 100=0.13%, 250=0.14%, 500=0.11%, 750=0.27%, 1000=1.05%
  2392. lat (msec) : 2=3.48%, 4=3.04%, 10=2.30%, 20=0.89%, 50=0.66%
  2393. lat (msec) : 100=0.31%, 250=0.31%, 500=1.02%, 750=1.93%, 1000=1.09%
  2394. cpu : usr=0.04%, sys=0.16%, ctx=5094, majf=0, minf=30
  2395. IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  2396. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  2397. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  2398. issued rwts: total=14945,0,0,0 short=0,0,0,0 dropped=0,0,0,0
  2399. latency : target=0, window=0, percentile=100.00%, depth=4
  2400. readwrite: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:34:58 2019
  2401. read: IOPS=48, BW=3080KiB/s (3154kB/s)(361MiB/120004msec)
  2402. slat (nsec): min=0, max=6708.7k, avg=29217.79, stdev=128662.97
  2403. clat (nsec): min=0, max=1112.9M, avg=9348407.86, stdev=55207623.48
  2404. lat (nsec): min=0, max=1112.0M, avg=9378582.70, stdev=55212255.77
  2405. clat percentiles (nsec):
  2406. | 1.00th=[ 716], 5.00th=[ 724], 10.00th=[ 732],
  2407. | 20.00th=[ 764], 30.00th=[ 868], 40.00th=[ 964],
  2408. | 50.00th=[ 1112], 60.00th=[ 1240], 70.00th=[ 1528],
  2409. | 80.00th=[ 148480], 90.00th=[ 2146304], 95.00th=[ 33423360],
  2410. | 99.00th=[ 312475648], 99.50th=[ 467664896], 99.90th=[ 624951296],
  2411. | 99.95th=[ 742391808], 99.99th=[1115684864]
  2412. bw ( KiB/s): min= 0, max= 4738, per=28.03%, avg=3079.50, stdev=2345.47, samples=2
  2413. iops : min= 0, max= 74, avg=48.00, stdev=36.77, samples=2
  2414. write: IOPS=49, BW=3190KiB/s (3266kB/s)(374MiB/120004msec)
  2415. slat (nsec): min=0, max=7569.2k, avg=191882.63, stdev=320415.62
  2416. clat (nsec): min=0, max=1050.9M, avg=71244375.78, stdev=134209200.19
  2417. lat (nsec): min=0, max=1050.9M, avg=71437577.53, stdev=134155113.65
  2418. clat percentiles (usec):
  2419. | 1.00th=[ 1037], 5.00th=[ 1287], 10.00th=[ 1516],
  2420. | 20.00th=[ 2073], 30.00th=[ 2900], 40.00th=[ 4359],
  2421. | 50.00th=[ 7308], 60.00th=[ 18744], 70.00th=[ 60031],
  2422. | 80.00th=[ 109577], 90.00th=[ 217056], 95.00th=[ 379585],
  2423. | 99.00th=[ 641729], 99.50th=[ 692061], 99.90th=[ 834667],
  2424. | 99.95th=[ 901776], 99.99th=[1052771]
  2425. bw ( KiB/s): min= 0, max= 4899, per=69.63%, avg=3187.50, stdev=2420.43, samples=2
  2426. iops : min= 0, max= 76, avg=49.50, stdev=37.48, samples=2
  2427. lat (nsec) : 750=9.10%, 1000=11.98%
  2428. lat (usec) : 2=14.25%, 4=0.09%, 10=0.37%, 20=1.14%, 50=0.95%
  2429. lat (usec) : 100=0.88%, 250=1.16%, 500=0.80%, 750=0.48%, 1000=1.41%
  2430. lat (msec) : 2=10.99%, 4=10.91%, 10=8.98%, 20=3.69%, 50=4.57%
  2431. lat (msec) : 100=5.85%, 250=7.58%, 500=3.06%, 750=1.60%, 1000=0.14%
  2432. cpu : usr=0.07%, sys=0.28%, ctx=12640, majf=0, minf=31
  2433. IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  2434. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  2435. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  2436. issued rwts: total=5775,5978,0,0 short=0,0,0,0 dropped=0,0,0,0
  2437. latency : target=0, window=0, percentile=100.00%, depth=4
  2438.  
  2439. Run status group 0 (all jobs):
  2440. READ: bw=10.7MiB/s (11.2MB/s), 3080KiB/s-7924KiB/s (3154kB/s-8115kB/s), io=1295MiB (1358MB), run=120004-120724msec
  2441. WRITE: bw=4579KiB/s (4689kB/s), 1436KiB/s-3190KiB/s (1471kB/s-3266kB/s), io=545MiB (571MB), run=120004-121813msec
  2442.  
  2443. Disk stats (read/write):
  2444. rbd128: ios=3825/8987, merge=0/2, ticks=579612/909061, in_queue=1482232, util=3.77%
  2445. <example-benchmark-fio-server-702-benchmark-72kl9>
  2446. randwrite: (groupid=0, jobs=1): err= 0: pid=11: Wed May 8 20:34:58 2019
  2447. write: IOPS=22, BW=1418KiB/s (1452kB/s)(168MiB/121297msec)
  2448. slat (nsec): min=0, max=427537k, avg=661838.85, stdev=12390156.81
  2449. clat (nsec): min=0, max=2646.7M, avg=182451904.64, stdev=362745931.13
  2450. lat (nsec): min=0, max=2646.8M, avg=183115311.85, stdev=362747429.03
  2451. clat percentiles (usec):
  2452. | 1.00th=[ 1123], 5.00th=[ 1549], 10.00th=[ 1893],
  2453. | 20.00th=[ 2573], 30.00th=[ 3326], 40.00th=[ 4424],
  2454. | 50.00th=[ 6063], 60.00th=[ 9634], 70.00th=[ 23200],
  2455. | 80.00th=[ 396362], 90.00th=[ 725615], 95.00th=[ 968885],
  2456. | 99.00th=[1535116], 99.50th=[1551893], 99.90th=[2566915],
  2457. | 99.95th=[2634023], 99.99th=[2634023]
  2458. bw ( KiB/s): min= 0, max= 1689, per=33.81%, avg=1430.50, stdev=365.57, samples=2
  2459. iops : min= 0, max= 26, avg=22.00, stdev= 5.66, samples=2
  2460. lat (usec) : 750=0.26%, 1000=0.48%
  2461. lat (msec) : 2=10.95%, 4=25.33%, 10=23.39%, 20=8.68%, 50=3.87%
  2462. lat (msec) : 100=2.38%, 250=2.38%, 500=5.85%, 750=7.56%, 1000=4.17%
  2463. cpu : usr=0.03%, sys=0.10%, ctx=4842, majf=0, minf=30
  2464. IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  2465. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  2466. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  2467. issued rwts: total=0,2685,0,0 short=0,0,0,0 dropped=0,0,0,0
  2468. latency : target=0, window=0, percentile=100.00%, depth=4
  2469. read: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:34:58 2019
  2470. read: IOPS=202, BW=12.7MiB/s (13.3MB/s)(1525MiB/120525msec)
  2471. slat (nsec): min=0, max=67962k, avg=31963.04, stdev=493698.59
  2472. clat (nsec): min=0, max=1958.1M, avg=19763723.06, stdev=119137048.25
  2473. lat (nsec): min=0, max=1958.2M, avg=19796745.38, stdev=119154585.48
  2474. clat percentiles (nsec):
  2475. | 1.00th=[ 716], 5.00th=[ 724], 10.00th=[ 740],
  2476. | 20.00th=[ 780], 30.00th=[ 796], 40.00th=[ 828],
  2477. | 50.00th=[ 868], 60.00th=[ 900], 70.00th=[ 1004],
  2478. | 80.00th=[ 1400], 90.00th=[ 1777664], 95.00th=[ 8028160],
  2479. | 99.00th=[ 742391808], 99.50th=[ 893386752], 99.90th=[1249902592],
  2480. | 99.95th=[1300234240], 99.99th=[1686110208]
  2481. bw ( KiB/s): min= 0, max=15112, per=82.69%, avg=12982.00, stdev=3012.27, samples=2
  2482. iops : min= 0, max= 236, avg=202.50, stdev=47.38, samples=2
  2483. lat (nsec) : 750=11.73%, 1000=58.10%
  2484. lat (usec) : 2=12.66%, 4=0.15%, 10=0.42%, 20=1.68%, 50=0.48%
  2485. lat (usec) : 100=0.20%, 250=0.13%, 500=0.11%, 750=0.26%, 1000=1.27%
  2486. lat (msec) : 2=3.29%, 4=2.92%, 10=1.97%, 20=0.82%, 50=0.57%
  2487. lat (msec) : 100=0.27%, 250=0.28%, 500=0.62%, 750=1.13%, 1000=0.68%
  2488. cpu : usr=0.07%, sys=0.24%, ctx=7087, majf=0, minf=30
  2489. IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  2490. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  2491. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  2492. issued rwts: total=24401,0,0,0 short=0,0,0,0 dropped=0,0,0,0
  2493. latency : target=0, window=0, percentile=100.00%, depth=4
  2494. readwrite: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:34:58 2019
  2495. read: IOPS=43, BW=2754KiB/s (2820kB/s)(323MiB/120018msec)
  2496. slat (nsec): min=0, max=2284.4k, avg=27384.79, stdev=79125.39
  2497. clat (nsec): min=0, max=935607k, avg=7895742.05, stdev=47845118.61
  2498. lat (nsec): min=0, max=935651k, avg=7924026.04, stdev=47851357.49
  2499. clat percentiles (nsec):
  2500. | 1.00th=[ 716], 5.00th=[ 724], 10.00th=[ 732],
  2501. | 20.00th=[ 764], 30.00th=[ 852], 40.00th=[ 948],
  2502. | 50.00th=[ 1096], 60.00th=[ 1240], 70.00th=[ 1512],
  2503. | 80.00th=[ 126464], 90.00th=[ 2113536], 95.00th=[ 32112640],
  2504. | 99.00th=[200278016], 99.50th=[392167424], 99.90th=[641728512],
  2505. | 99.95th=[717225984], 99.99th=[935329792]
  2506. bw ( KiB/s): min= 0, max= 4245, per=17.54%, avg=2753.50, stdev=2109.30, samples=2
  2507. iops : min= 0, max= 66, avg=42.50, stdev=33.23, samples=2
  2508. write: IOPS=44, BW=2843KiB/s (2911kB/s)(333MiB/120018msec)
  2509. slat (nsec): min=0, max=10903k, avg=179886.99, stdev=309272.59
  2510. clat (nsec): min=0, max=1047.2M, avg=82461684.37, stdev=148141789.84
  2511. lat (nsec): min=0, max=1047.5M, avg=82642759.28, stdev=148090585.05
  2512. clat percentiles (usec):
  2513. | 1.00th=[ 971], 5.00th=[ 1205], 10.00th=[ 1385],
  2514. | 20.00th=[ 1893], 30.00th=[ 2573], 40.00th=[ 3982],
  2515. | 50.00th=[ 9765], 60.00th=[ 26608], 70.00th=[ 79168],
  2516. | 80.00th=[ 123208], 90.00th=[ 265290], 95.00th=[ 446694],
  2517. | 99.00th=[ 692061], 99.50th=[ 742392], 99.90th=[ 876610],
  2518. | 99.95th=[ 910164], 99.99th=[1044382]
  2519. bw ( KiB/s): min= 0, max= 4369, per=67.16%, avg=2841.50, stdev=2160.21, samples=2
  2520. iops : min= 0, max= 68, avg=44.00, stdev=33.94, samples=2
  2521. lat (nsec) : 750=8.04%, 1000=13.42%
  2522. lat (usec) : 2=13.92%, 4=0.02%, 10=0.30%, 20=1.21%, 50=0.87%
  2523. lat (usec) : 100=1.30%, 250=0.97%, 500=0.61%, 750=0.59%, 1000=1.70%
  2524. lat (msec) : 2=12.33%, 4=10.48%, 10=5.81%, 20=3.69%, 50=5.07%
  2525. lat (msec) : 100=5.65%, 250=8.36%, 500=3.69%, 750=1.73%, 1000=0.26%
  2526. cpu : usr=0.05%, sys=0.25%, ctx=11299, majf=0, minf=30
  2527. IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  2528. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  2529. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  2530. issued rwts: total=5164,5328,0,0 short=0,0,0,0 dropped=0,0,0,0
  2531. latency : target=0, window=0, percentile=100.00%, depth=4
  2532.  
  2533. Run status group 0 (all jobs):
  2534. READ: bw=15.3MiB/s (16.1MB/s), 2754KiB/s-12.7MiB/s (2820kB/s-13.3MB/s), io=1848MiB (1938MB), run=120018-120525msec
  2535. WRITE: bw=4231KiB/s (4333kB/s), 1418KiB/s-2843KiB/s (1452kB/s-2911kB/s), io=501MiB (526MB), run=120018-121297msec
  2536.  
  2537. Disk stats (read/write):
  2538. rbd182: ios=4729/8233, merge=0/2, ticks=563379/937246, in_queue=1494170, util=4.62%
  2539. <example-benchmark-fio-server-620-benchmark-rlhzh>
  2540. randwrite: (groupid=0, jobs=1): err= 0: pid=11: Wed May 8 20:34:58 2019
  2541. write: IOPS=36, BW=2346KiB/s (2402kB/s)(276MiB/120414msec)
  2542. slat (nsec): min=0, max=1809.7M, avg=1451143.21, stdev=41416492.97
  2543. clat (nsec): min=0, max=2608.3M, avg=107987024.10, stdev=273072237.75
  2544. lat (nsec): min=0, max=2673.9M, avg=109505679.21, stdev=278381371.46
  2545. clat percentiles (usec):
  2546. | 1.00th=[ 1139], 5.00th=[ 1450], 10.00th=[ 1663],
  2547. | 20.00th=[ 2147], 30.00th=[ 2769], 40.00th=[ 3654],
  2548. | 50.00th=[ 5145], 60.00th=[ 8586], 70.00th=[ 16909],
  2549. | 80.00th=[ 50594], 90.00th=[ 488637], 95.00th=[ 759170],
  2550. | 99.00th=[1283458], 99.50th=[1384121], 99.90th=[1870660],
  2551. | 99.95th=[1887437], 99.99th=[2600469]
  2552. bw ( KiB/s): min= 0, max= 3078, per=46.17%, avg=2351.50, stdev=1027.43, samples=2
  2553. iops : min= 0, max= 48, avg=36.50, stdev=16.26, samples=2
  2554. lat (usec) : 500=0.02%, 750=0.09%, 1000=0.25%
  2555. lat (msec) : 2=16.78%, 4=26.14%, 10=19.43%, 20=9.00%, 50=8.34%
  2556. lat (msec) : 100=5.01%, 250=2.68%, 500=2.40%, 750=4.87%, 1000=2.47%
  2557. cpu : usr=0.04%, sys=0.17%, ctx=8054, majf=0, minf=29
  2558. IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  2559. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  2560. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  2561. issued rwts: total=0,4411,0,0 short=0,0,0,0 dropped=0,0,0,0
  2562. latency : target=0, window=0, percentile=100.00%, depth=4
  2563. read: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:34:58 2019
  2564. read: IOPS=832, BW=51.8MiB/s (54.4MB/s)(6223MiB/120042msec)
  2565. slat (nsec): min=0, max=1627.4M, avg=61070.51, stdev=7625575.00
  2566. clat (nsec): min=0, max=2303.2M, avg=4747375.68, stdev=57025931.76
  2567. lat (nsec): min=0, max=2303.2M, avg=4809469.51, stdev=57538826.11
  2568. clat percentiles (nsec):
  2569. | 1.00th=[ 708], 5.00th=[ 716], 10.00th=[ 732],
  2570. | 20.00th=[ 764], 30.00th=[ 780], 40.00th=[ 796],
  2571. | 50.00th=[ 836], 60.00th=[ 868], 70.00th=[ 900],
  2572. | 80.00th=[ 1400], 90.00th=[ 12864], 95.00th=[ 782336],
  2573. | 99.00th=[ 43778048], 99.50th=[ 438304768], 99.90th=[ 901775360],
  2574. | 99.95th=[1027604480], 99.99th=[1417674752]
  2575. bw ( KiB/s): min= 0, max=88652, per=95.26%, avg=53098.00, stdev=50280.95, samples=2
  2576. iops : min= 0, max= 1391, avg=832.50, stdev=789.84, samples=2
  2577. lat (nsec) : 750=14.75%, 1000=60.85%
  2578. lat (usec) : 2=5.43%, 4=0.15%, 10=1.52%, 20=9.01%, 50=2.48%
  2579. lat (usec) : 100=0.27%, 250=0.18%, 500=0.12%, 750=0.17%, 1000=0.56%
  2580. lat (msec) : 2=1.15%, 4=0.85%, 10=0.60%, 20=0.40%, 50=0.61%
  2581. lat (msec) : 100=0.20%, 250=0.10%, 500=0.16%, 750=0.25%, 1000=0.14%
  2582. cpu : usr=0.22%, sys=0.80%, ctx=11085, majf=0, minf=30
  2583. IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  2584. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  2585. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  2586. issued rwts: total=99919,0,0,0 short=0,0,0,0 dropped=0,0,0,0
  2587. latency : target=0, window=0, percentile=100.00%, depth=4
  2588. readwrite: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:34:58 2019
  2589. read: IOPS=41, BW=2655KiB/s (2719kB/s)(311MiB/120015msec)
  2590. slat (nsec): min=0, max=4456.3k, avg=32168.09, stdev=97756.59
  2591. clat (nsec): min=0, max=989312k, avg=12693057.98, stdev=60321579.79
  2592. lat (nsec): min=0, max=989349k, avg=12726223.12, stdev=60329190.75
  2593. clat percentiles (nsec):
  2594. | 1.00th=[ 716], 5.00th=[ 724], 10.00th=[ 740],
  2595. | 20.00th=[ 820], 30.00th=[ 916], 40.00th=[ 1080],
  2596. | 50.00th=[ 1224], 60.00th=[ 1384], 70.00th=[ 44288],
  2597. | 80.00th=[ 1122304], 90.00th=[ 15138816], 95.00th=[ 62128128],
  2598. | 99.00th=[320864256], 99.50th=[425721856], 99.90th=[767557632],
  2599. | 99.95th=[935329792], 99.99th=[985661440]
  2600. bw ( KiB/s): min= 0, max= 3947, per=4.76%, avg=2655.00, stdev=1827.16, samples=2
  2601. iops : min= 0, max= 61, avg=41.00, stdev=28.28, samples=2
  2602. write: IOPS=43, BW=2757KiB/s (2823kB/s)(323MiB/120015msec)
  2603. slat (nsec): min=0, max=8114.2k, avg=169031.29, stdev=274467.99
  2604. clat (nsec): min=0, max=1041.7M, avg=80800586.43, stdev=146966844.61
  2605. lat (nsec): min=0, max=1041.8M, avg=80970787.08, stdev=146927156.52
  2606. clat percentiles (usec):
  2607. | 1.00th=[ 1057], 5.00th=[ 1270], 10.00th=[ 1483],
  2608. | 20.00th=[ 1926], 30.00th=[ 2671], 40.00th=[ 4047],
  2609. | 50.00th=[ 8586], 60.00th=[ 24773], 70.00th=[ 65799],
  2610. | 80.00th=[ 111674], 90.00th=[ 295699], 95.00th=[ 455082],
  2611. | 99.00th=[ 641729], 99.50th=[ 700449], 99.90th=[ 851444],
  2612. | 99.95th=[ 868221], 99.99th=[1044382]
  2613. bw ( KiB/s): min= 0, max= 4086, per=54.10%, avg=2755.50, stdev=1881.61, samples=2
  2614. iops : min= 0, max= 63, avg=42.50, stdev=28.99, samples=2
  2615. lat (nsec) : 750=6.03%, 1000=11.10%
  2616. lat (usec) : 2=15.61%, 4=0.11%, 10=0.19%, 20=0.82%, 50=0.69%
  2617. lat (usec) : 100=0.59%, 250=0.89%, 500=0.49%, 750=0.73%, 1000=1.93%
  2618. lat (msec) : 2=12.85%, 4=11.24%, 10=6.76%, 20=3.80%, 50=6.10%
  2619. lat (msec) : 100=6.97%, 250=6.51%, 500=4.52%, 750=1.90%, 1000=0.18%
  2620. cpu : usr=0.07%, sys=0.23%, ctx=12113, majf=0, minf=33
  2621. IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  2622. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  2623. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  2624. issued rwts: total=4978,5168,0,0 short=0,0,0,0 dropped=0,0,0,0
  2625. latency : target=0, window=0, percentile=100.00%, depth=4
  2626.  
  2627. Run status group 0 (all jobs):
  2628. READ: bw=54.4MiB/s (57.1MB/s), 2655KiB/s-51.8MiB/s (2719kB/s-54.4MB/s), io=6534MiB (6852MB), run=120015-120042msec
  2629. WRITE: bw=5094KiB/s (5216kB/s), 2346KiB/s-2757KiB/s (2402kB/s-2823kB/s), io=599MiB (628MB), run=120015-120414msec
  2630.  
  2631. Disk stats (read/write):
  2632. rbd291: ios=6612/9793, merge=0/1, ticks=541800/872212, in_queue=1405759, util=7.93%
  2633. <example-benchmark-fio-server-139-benchmark-7btww>
  2634. randwrite: (groupid=0, jobs=1): err= 0: pid=11: Wed May 8 20:34:58 2019
  2635. write: IOPS=21, BW=1355KiB/s (1388kB/s)(160MiB/120980msec)
  2636. slat (nsec): min=0, max=859342k, avg=1241964.81, stdev=25538249.96
  2637. clat (nsec): min=0, max=2242.1M, avg=188335014.13, stdev=370328253.41
  2638. lat (nsec): min=0, max=2242.8M, avg=189577480.82, stdev=371271934.52
  2639. clat percentiles (usec):
  2640. | 1.00th=[ 1090], 5.00th=[ 1434], 10.00th=[ 1778],
  2641. | 20.00th=[ 2474], 30.00th=[ 3326], 40.00th=[ 4752],
  2642. | 50.00th=[ 7242], 60.00th=[ 13566], 70.00th=[ 27657],
  2643. | 80.00th=[ 392168], 90.00th=[ 817890], 95.00th=[1035994],
  2644. | 99.00th=[1484784], 99.50th=[1635779], 99.90th=[2139096],
  2645. | 99.95th=[2139096], 99.99th=[2231370]
  2646. bw ( KiB/s): min= 0, max= 1415, per=29.91%, avg=1358.50, stdev=79.90, samples=2
  2647. iops : min= 0, max= 22, avg=21.00, stdev= 1.41, samples=2
  2648. lat (usec) : 10=0.04%, 500=0.04%, 750=0.08%, 1000=0.31%
  2649. lat (msec) : 2=12.90%, 4=21.81%, 10=20.75%, 20=9.73%, 50=8.75%
  2650. lat (msec) : 100=1.99%, 250=1.80%, 500=4.85%, 750=5.31%, 1000=5.90%
  2651. cpu : usr=0.02%, sys=0.11%, ctx=4636, majf=0, minf=30
  2652. IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  2653. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  2654. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  2655. issued rwts: total=0,2559,0,0 short=0,0,0,0 dropped=0,0,0,0
  2656. latency : target=0, window=0, percentile=100.00%, depth=4
  2657. read: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:34:58 2019
  2658. read: IOPS=376, BW=23.5MiB/s (24.7MB/s)(2837MiB/120564msec)
  2659. slat (nsec): min=0, max=691432k, avg=43024.80, stdev=3613085.44
  2660. clat (nsec): min=0, max=2143.1M, avg=10554204.56, stdev=88854248.08
  2661. lat (nsec): min=0, max=2143.2M, avg=10598187.22, stdev=88937193.66
  2662. clat percentiles (nsec):
  2663. | 1.00th=[ 716], 5.00th=[ 724], 10.00th=[ 748],
  2664. | 20.00th=[ 780], 30.00th=[ 796], 40.00th=[ 828],
  2665. | 50.00th=[ 860], 60.00th=[ 892], 70.00th=[ 932],
  2666. | 80.00th=[ 1144], 90.00th=[ 13632], 95.00th=[ 2342912],
  2667. | 99.00th=[ 541065216], 99.50th=[ 759169024], 99.90th=[1052770304],
  2668. | 99.95th=[1266679808], 99.99th=[1602224128]
  2669. bw ( KiB/s): min= 0, max=34499, per=89.06%, avg=24203.50, stdev=14560.04, samples=2
  2670. iops : min= 0, max= 539, avg=378.00, stdev=227.69, samples=2
  2671. lat (nsec) : 750=10.71%, 1000=65.67%
  2672. lat (usec) : 2=10.08%, 4=0.17%, 10=0.54%, 20=3.27%, 50=1.05%
  2673. lat (usec) : 100=0.19%, 250=0.11%, 500=0.19%, 750=0.19%, 1000=0.72%
  2674. lat (msec) : 2=1.76%, 4=1.56%, 10=1.17%, 20=0.52%, 50=0.42%
  2675. lat (msec) : 100=0.13%, 250=0.16%, 500=0.33%, 750=0.55%, 1000=0.37%
  2676. cpu : usr=0.10%, sys=0.42%, ctx=7607, majf=0, minf=30
  2677. IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  2678. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  2679. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  2680. issued rwts: total=45440,0,0,0 short=0,0,0,0 dropped=0,0,0,0
  2681. latency : target=0, window=0, percentile=100.00%, depth=4
  2682. readwrite: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:34:58 2019
  2683. read: IOPS=48, BW=3100KiB/s (3175kB/s)(363MiB/120004msec)
  2684. slat (nsec): min=0, max=3502.5k, avg=30685.11, stdev=91394.21
  2685. clat (nsec): min=0, max=886323k, avg=8285032.64, stdev=51086846.62
  2686. lat (nsec): min=0, max=886388k, avg=8316623.23, stdev=51091851.38
  2687. clat percentiles (nsec):
  2688. | 1.00th=[ 716], 5.00th=[ 724], 10.00th=[ 732],
  2689. | 20.00th=[ 764], 30.00th=[ 876], 40.00th=[ 996],
  2690. | 50.00th=[ 1144], 60.00th=[ 1288], 70.00th=[ 1672],
  2691. | 80.00th=[ 179200], 90.00th=[ 1974272], 95.00th=[ 14221312],
  2692. | 99.00th=[320864256], 99.50th=[438304768], 99.90th=[583008256],
  2693. | 99.95th=[658505728], 99.99th=[884998144]
  2694. bw ( KiB/s): min= 0, max= 4320, per=11.41%, avg=3100.00, stdev=1725.34, samples=2
  2695. iops : min= 0, max= 67, avg=48.00, stdev=26.87, samples=2
  2696. write: IOPS=50, BW=3213KiB/s (3290kB/s)(377MiB/120004msec)
  2697. slat (nsec): min=0, max=13105k, avg=189168.32, stdev=269636.09
  2698. clat (nsec): min=0, max=1091.8M, avg=71891687.37, stdev=135619142.37
  2699. lat (nsec): min=0, max=1091.9M, avg=72082105.36, stdev=135568803.97
  2700. clat percentiles (usec):
  2701. | 1.00th=[ 1020], 5.00th=[ 1221], 10.00th=[ 1418],
  2702. | 20.00th=[ 1827], 30.00th=[ 2442], 40.00th=[ 3818],
  2703. | 50.00th=[ 7832], 60.00th=[ 21103], 70.00th=[ 68682],
  2704. | 80.00th=[ 107480], 90.00th=[ 200279], 95.00th=[ 392168],
  2705. | 99.00th=[ 658506], 99.50th=[ 717226], 99.90th=[ 859833],
  2706. | 99.95th=[ 926942], 99.99th=[1098908]
  2707. bw ( KiB/s): min= 0, max= 4449, per=70.71%, avg=3211.50, stdev=1750.09, samples=2
  2708. iops : min= 0, max= 69, avg=49.50, stdev=27.58, samples=2
  2709. lat (nsec) : 750=8.45%, 1000=11.48%
  2710. lat (usec) : 2=14.69%, 4=0.07%, 10=0.48%, 20=1.30%, 50=0.89%
  2711. lat (usec) : 100=1.14%, 250=1.15%, 500=0.68%, 750=0.79%, 1000=1.64%
  2712. lat (msec) : 2=13.27%, 4=10.32%, 10=6.92%, 20=3.79%, 50=4.01%
  2713. lat (msec) : 100=6.76%, 250=7.48%, 500=3.02%, 750=1.48%, 1000=0.19%
  2714. cpu : usr=0.08%, sys=0.28%, ctx=12857, majf=0, minf=34
  2715. IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  2716. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  2717. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  2718. issued rwts: total=5813,6022,0,0 short=0,0,0,0 dropped=0,0,0,0
  2719. latency : target=0, window=0, percentile=100.00%, depth=4
  2720.  
  2721. Run status group 0 (all jobs):
  2722. READ: bw=26.5MiB/s (27.8MB/s), 3100KiB/s-23.5MiB/s (3175kB/s-24.7MB/s), io=3200MiB (3355MB), run=120004-120564msec
  2723. WRITE: bw=4543KiB/s (4652kB/s), 1355KiB/s-3213KiB/s (1388kB/s-3290kB/s), io=537MiB (563MB), run=120004-120980msec
  2724.  
  2725. Disk stats (read/write):
  2726. rbd58: ios=5113/8988, merge=0/1, ticks=567154/914911, in_queue=1474958, util=6.12%
  2727. <example-benchmark-fio-server-289-benchmark-jhcgn>
  2728. randwrite: (groupid=0, jobs=1): err= 0: pid=11: Wed May 8 20:34:58 2019
  2729. write: IOPS=44, BW=2824KiB/s (2892kB/s)(331MiB/120072msec)
  2730. slat (nsec): min=0, max=714127k, avg=491923.63, stdev=13745623.96
  2731. clat (nsec): min=0, max=1605.9M, avg=90397146.42, stdev=161216053.44
  2732. lat (nsec): min=0, max=1766.4M, avg=90890178.16, stdev=162453244.77
  2733. clat percentiles (usec):
  2734. | 1.00th=[ 938], 5.00th=[ 1123], 10.00th=[ 1270],
  2735. | 20.00th=[ 1614], 30.00th=[ 2114], 40.00th=[ 3064],
  2736. | 50.00th=[ 7046], 60.00th=[ 62129], 70.00th=[ 94897],
  2737. | 80.00th=[ 135267], 90.00th=[ 254804], 95.00th=[ 438305],
  2738. | 99.00th=[ 759170], 99.50th=[ 943719], 99.90th=[1132463],
  2739. | 99.95th=[1149240], 99.99th=[1602225]
  2740. bw ( KiB/s): min= 0, max= 3586, per=45.15%, avg=2823.00, stdev=1079.04, samples=2
  2741. iops : min= 0, max= 56, avg=44.00, stdev=16.97, samples=2
  2742. lat (usec) : 250=0.02%, 500=0.02%, 750=0.09%, 1000=2.00%
  2743. lat (msec) : 2=26.38%, 4=15.98%, 10=7.84%, 20=2.72%, 50=3.36%
  2744. lat (msec) : 100=13.54%, 250=17.79%, 500=6.35%, 750=2.85%, 1000=0.64%
  2745. cpu : usr=0.07%, sys=0.18%, ctx=9958, majf=0, minf=26
  2746. IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  2747. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  2748. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  2749. issued rwts: total=0,5295,0,0 short=0,0,0,0 dropped=0,0,0,0
  2750. latency : target=0, window=0, percentile=100.00%, depth=4
  2751. read: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:34:58 2019
  2752. read: IOPS=311, BW=19.5MiB/s (20.4MB/s)(2338MiB/120002msec)
  2753. slat (nsec): min=0, max=710630k, avg=63218.86, stdev=4979859.96
  2754. clat (nsec): min=0, max=1434.2M, avg=12791436.07, stdev=62473523.58
  2755. lat (nsec): min=0, max=1434.4M, avg=12855512.68, stdev=62676595.96
  2756. clat percentiles (nsec):
  2757. | 1.00th=[ 708], 5.00th=[ 716], 10.00th=[ 716],
  2758. | 20.00th=[ 724], 30.00th=[ 732], 40.00th=[ 748],
  2759. | 50.00th=[ 820], 60.00th=[ 876], 70.00th=[ 1032],
  2760. | 80.00th=[ 8768], 90.00th=[ 7045120], 95.00th=[ 60030976],
  2761. | 99.00th=[ 362807296], 99.50th=[ 484442112], 99.90th=[ 683671552],
  2762. | 99.95th=[ 767557632], 99.99th=[1019215872]
  2763. bw ( KiB/s): min= 0, max=25118, per=85.72%, avg=19949.00, stdev=7310.07, samples=2
  2764. iops : min= 0, max= 392, avg=311.00, stdev=114.55, samples=2
  2765. lat (nsec) : 750=41.73%, 1000=27.46%
  2766. lat (usec) : 2=10.55%, 4=0.03%, 10=0.58%, 20=0.31%, 50=0.27%
  2767. lat (usec) : 100=0.09%, 250=0.07%, 500=0.27%, 750=0.98%, 1000=2.19%
  2768. lat (msec) : 2=3.06%, 4=1.82%, 10=0.83%, 20=0.52%, 50=3.20%
  2769. lat (msec) : 100=2.93%, 250=1.53%, 500=1.11%, 750=0.40%, 1000=0.05%
  2770. cpu : usr=0.11%, sys=0.35%, ctx=14140, majf=0, minf=97
  2771. IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  2772. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  2773. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  2774. issued rwts: total=37428,0,0,0 short=0,0,0,0 dropped=0,0,0,0
  2775. latency : target=0, window=0, percentile=100.00%, depth=4
  2776. readwrite: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:34:58 2019
  2777. read: IOPS=51, BW=3322KiB/s (3402kB/s)(389MiB/120007msec)
  2778. slat (nsec): min=0, max=663988k, avg=171583.65, stdev=8737229.14
  2779. clat (nsec): min=0, max=632295k, avg=6775755.36, stdev=37927924.18
  2780. lat (nsec): min=0, max=664065k, avg=6948327.42, stdev=38902864.79
  2781. clat percentiles (nsec):
  2782. | 1.00th=[ 716], 5.00th=[ 724], 10.00th=[ 740],
  2783. | 20.00th=[ 796], 30.00th=[ 892], 40.00th=[ 1004],
  2784. | 50.00th=[ 1160], 60.00th=[ 1304], 70.00th=[ 17792],
  2785. | 80.00th=[ 864256], 90.00th=[ 3031040], 95.00th=[ 33816576],
  2786. | 99.00th=[179306496], 99.50th=[308281344], 99.90th=[492830720],
  2787. | 99.95th=[574619648], 99.99th=[633339904]
  2788. bw ( KiB/s): min= 0, max= 4763, per=14.27%, avg=3322.00, stdev=2037.88, samples=2
  2789. iops : min= 0, max= 74, avg=51.50, stdev=31.82, samples=2
  2790. write: IOPS=53, BW=3430KiB/s (3513kB/s)(402MiB/120007msec)
  2791. slat (nsec): min=0, max=11878k, avg=197961.98, stdev=307898.64
  2792. clat (nsec): min=0, max=1469.7M, avg=68202610.29, stdev=138950257.20
  2793. lat (nsec): min=0, max=1469.8M, avg=68401706.46, stdev=138903708.14
  2794. clat percentiles (usec):
  2795. | 1.00th=[ 996], 5.00th=[ 1221], 10.00th=[ 1401],
  2796. | 20.00th=[ 1762], 30.00th=[ 2212], 40.00th=[ 2999],
  2797. | 50.00th=[ 4621], 60.00th=[ 12387], 70.00th=[ 57410],
  2798. | 80.00th=[ 103285], 90.00th=[ 200279], 95.00th=[ 354419],
  2799. | 99.00th=[ 683672], 99.50th=[ 801113], 99.90th=[1098908],
  2800. | 99.95th=[1132463], 99.99th=[1468007]
  2801. bw ( KiB/s): min= 0, max= 4867, per=54.84%, avg=3428.50, stdev=2034.35, samples=2
  2802. iops : min= 0, max= 76, avg=53.50, stdev=31.82, samples=2
  2803. lat (nsec) : 750=7.08%, 1000=12.30%
  2804. lat (usec) : 2=14.32%, 4=0.03%, 10=0.24%, 20=0.81%, 50=0.70%
  2805. lat (usec) : 100=0.59%, 250=0.86%, 500=0.89%, 750=0.79%, 1000=2.20%
  2806. lat (msec) : 2=15.49%, 4=12.24%, 10=7.09%, 20=2.65%, 50=4.11%
  2807. lat (msec) : 100=6.31%, 250=7.16%, 500=2.66%, 750=1.15%, 1000=0.28%
  2808. cpu : usr=0.07%, sys=0.31%, ctx=14996, majf=0, minf=33
  2809. IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  2810. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  2811. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  2812. issued rwts: total=6231,6429,0,0 short=0,0,0,0 dropped=0,0,0,0
  2813. latency : target=0, window=0, percentile=100.00%, depth=4
  2814.  
  2815. Run status group 0 (all jobs):
  2816. READ: bw=22.7MiB/s (23.8MB/s), 3322KiB/s-19.5MiB/s (3402kB/s-20.4MB/s), io=2728MiB (2860MB), run=120002-120007msec
  2817. WRITE: bw=6252KiB/s (6402kB/s), 2824KiB/s-3430KiB/s (2892kB/s-3513kB/s), io=733MiB (769MB), run=120007-120072msec
  2818.  
  2819. Disk stats (read/write):
  2820. rbd123: ios=9289/12253, merge=0/2, ticks=557195/852287, in_queue=1398563, util=14.81%
  2821. <example-benchmark-fio-server-115-benchmark-krrp7>
  2822. randwrite: (groupid=0, jobs=1): err= 0: pid=11: Wed May 8 20:34:59 2019
  2823. write: IOPS=26, BW=1687KiB/s (1727kB/s)(198MiB/120401msec)
  2824. slat (nsec): min=0, max=464962k, avg=535888.93, stdev=11242241.37
  2825. clat (nsec): min=0, max=2030.7M, avg=151464747.66, stdev=258316057.55
  2826. lat (nsec): min=0, max=2030.8M, avg=152001879.93, stdev=258431932.77
  2827. clat percentiles (usec):
  2828. | 1.00th=[ 1270], 5.00th=[ 1942], 10.00th=[ 2573],
  2829. | 20.00th=[ 3851], 30.00th=[ 5473], 40.00th=[ 7832],
  2830. | 50.00th=[ 11207], 60.00th=[ 20055], 70.00th=[ 122160],
  2831. | 80.00th=[ 329253], 90.00th=[ 549454], 95.00th=[ 700449],
  2832. | 99.00th=[ 943719], 99.50th=[1182794], 99.90th=[2021655],
  2833. | 99.95th=[2021655], 99.99th=[2038432]
  2834. bw ( KiB/s): min= 0, max= 1993, per=31.33%, avg=1688.50, stdev=430.63, samples=2
  2835. iops : min= 0, max= 31, avg=26.00, stdev= 7.07, samples=2
  2836. lat (usec) : 4=0.13%, 250=0.03%, 500=0.03%, 750=0.06%, 1000=0.06%
  2837. lat (msec) : 2=5.21%, 4=15.36%, 10=26.12%, 20=13.19%, 50=6.50%
  2838. lat (msec) : 100=2.62%, 250=6.18%, 500=13.15%, 750=7.48%, 1000=3.19%
  2839. cpu : usr=0.03%, sys=0.12%, ctx=5496, majf=0, minf=30
  2840. IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  2841. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  2842. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  2843. issued rwts: total=0,3170,0,0 short=0,0,0,0 dropped=0,0,0,0
  2844. latency : target=0, window=0, percentile=100.00%, depth=4
  2845. read: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:34:59 2019
  2846. read: IOPS=123, BW=7920KiB/s (8110kB/s)(933MiB/120639msec)
  2847. slat (nsec): min=0, max=7129.5k, avg=37269.24, stdev=139259.60
  2848. clat (nsec): min=0, max=1959.1M, avg=32268522.52, stdev=150613641.83
  2849. lat (nsec): min=0, max=1959.1M, avg=32306809.86, stdev=150630782.48
  2850. clat percentiles (nsec):
  2851. | 1.00th=[ 716], 5.00th=[ 724], 10.00th=[ 748],
  2852. | 20.00th=[ 788], 30.00th=[ 820], 40.00th=[ 868],
  2853. | 50.00th=[ 908], 60.00th=[ 1080], 70.00th=[ 1368],
  2854. | 80.00th=[ 1138688], 90.00th=[ 4882432], 95.00th=[ 56885248],
  2855. | 99.00th=[ 843055104], 99.50th=[ 960495616], 99.90th=[1216348160],
  2856. | 99.95th=[1400897536], 99.99th=[1803550720]
  2857. bw ( KiB/s): min= 0, max= 8521, per=68.97%, avg=7955.50, stdev=799.74, samples=2
  2858. iops : min= 0, max= 133, avg=124.00, stdev=12.73, samples=2
  2859. lat (nsec) : 750=10.00%, 1000=47.62%
  2860. lat (usec) : 2=17.35%, 4=0.25%, 10=0.15%, 20=0.74%, 50=0.16%
  2861. lat (usec) : 100=0.14%, 250=0.13%, 500=0.11%, 750=0.44%, 1000=1.89%
  2862. lat (msec) : 2=5.43%, 4=4.56%, 10=3.44%, 20=1.44%, 50=1.05%
  2863. lat (msec) : 100=0.33%, 250=0.31%, 500=1.02%, 750=1.94%, 1000=1.11%
  2864. cpu : usr=0.04%, sys=0.18%, ctx=7043, majf=0, minf=30
  2865. IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  2866. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  2867. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  2868. issued rwts: total=14926,0,0,0 short=0,0,0,0 dropped=0,0,0,0
  2869. latency : target=0, window=0, percentile=100.00%, depth=4
  2870. readwrite: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:34:59 2019
  2871. read: IOPS=56, BW=3635KiB/s (3722kB/s)(426MiB/120006msec)
  2872. slat (nsec): min=0, max=1680.6k, avg=30012.19, stdev=72067.62
  2873. clat (nsec): min=0, max=747366k, avg=10032263.04, stdev=54279230.59
  2874. lat (nsec): min=0, max=747386k, avg=10063205.13, stdev=54284115.55
  2875. clat percentiles (nsec):
  2876. | 1.00th=[ 716], 5.00th=[ 724], 10.00th=[ 732],
  2877. | 20.00th=[ 796], 30.00th=[ 892], 40.00th=[ 1012],
  2878. | 50.00th=[ 1144], 60.00th=[ 1304], 70.00th=[ 15808],
  2879. | 80.00th=[ 921600], 90.00th=[ 3522560], 95.00th=[ 45350912],
  2880. | 99.00th=[329252864], 99.50th=[442499072], 99.90th=[675282944],
  2881. | 99.95th=[700448768], 99.99th=[750780416]
  2882. bw ( KiB/s): min= 0, max= 5302, per=31.51%, avg=3635.00, stdev=2357.49, samples=2
  2883. iops : min= 0, max= 82, avg=56.00, stdev=36.77, samples=2
  2884. write: IOPS=58, BW=3715KiB/s (3804kB/s)(435MiB/120006msec)
  2885. slat (nsec): min=0, max=29166k, avg=196391.82, stdev=440398.02
  2886. clat (nsec): min=0, max=905267k, avg=58898123.51, stdev=124023895.80
  2887. lat (nsec): min=0, max=906784k, avg=59095996.36, stdev=123976481.96
  2888. clat percentiles (usec):
  2889. | 1.00th=[ 979], 5.00th=[ 1172], 10.00th=[ 1352], 20.00th=[ 1680],
  2890. | 30.00th=[ 2114], 40.00th=[ 2868], 50.00th=[ 4686], 60.00th=[ 10421],
  2891. | 70.00th=[ 32113], 80.00th=[ 93848], 90.00th=[160433], 95.00th=[320865],
  2892. | 99.00th=[675283], 99.50th=[767558], 99.90th=[868221], 99.95th=[893387],
  2893. | 99.99th=[901776]
  2894. bw ( KiB/s): min= 0, max= 5360, per=68.90%, avg=3713.00, stdev=2329.21, samples=2
  2895. iops : min= 0, max= 83, avg=57.50, stdev=36.06, samples=2
  2896. lat (nsec) : 750=7.63%, 1000=11.81%
  2897. lat (usec) : 2=14.57%, 4=0.04%, 10=0.25%, 20=0.73%, 50=0.65%
  2898. lat (usec) : 100=0.67%, 250=0.86%, 500=0.48%, 750=0.68%, 1000=2.32%
  2899. lat (msec) : 2=16.48%, 4=11.47%, 10=6.95%, 20=4.01%, 50=4.48%
  2900. lat (msec) : 100=5.18%, 250=6.82%, 500=2.74%, 750=0.91%, 1000=0.28%
  2901. cpu : usr=0.08%, sys=0.33%, ctx=15897, majf=0, minf=30
  2902. IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  2903. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  2904. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  2905. issued rwts: total=6816,6963,0,0 short=0,0,0,0 dropped=0,0,0,0
  2906. latency : target=0, window=0, percentile=100.00%, depth=4
  2907.  
  2908. Run status group 0 (all jobs):
  2909. READ: bw=11.3MiB/s (11.8MB/s), 3635KiB/s-7920KiB/s (3722kB/s-8110kB/s), io=1359MiB (1425MB), run=120006-120639msec
  2910. WRITE: bw=5389KiB/s (5519kB/s), 1687KiB/s-3715KiB/s (1727kB/s-3804kB/s), io=634MiB (664MB), run=120006-120401msec
  2911.  
  2912. Disk stats (read/write):
  2913. rbd33: ios=5721/11388, merge=0/3, ticks=605434/883833, in_queue=1480636, util=5.12%
  2914. <example-benchmark-fio-server-594-benchmark-dz8fd>
  2915. randwrite: (groupid=0, jobs=1): err= 0: pid=11: Wed May 8 20:34:59 2019
  2916. write: IOPS=39, BW=2556KiB/s (2617kB/s)(300MiB/120181msec)
  2917. slat (nsec): min=0, max=746936k, avg=484272.82, stdev=14806221.57
  2918. clat (nsec): min=0, max=1378.3M, avg=100241276.02, stdev=175257826.40
  2919. lat (nsec): min=0, max=1378.4M, avg=100726680.74, stdev=175669845.78
  2920. clat percentiles (usec):
  2921. | 1.00th=[ 996], 5.00th=[ 1237], 10.00th=[ 1450],
  2922. | 20.00th=[ 1926], 30.00th=[ 2868], 40.00th=[ 4883],
  2923. | 50.00th=[ 11076], 60.00th=[ 55837], 70.00th=[ 99091],
  2924. | 80.00th=[ 139461], 90.00th=[ 337642], 95.00th=[ 484443],
  2925. | 99.00th=[ 809501], 99.50th=[ 952108], 99.90th=[1283458],
  2926. | 99.95th=[1317012], 99.99th=[1384121]
  2927. bw ( KiB/s): min= 0, max= 2893, per=46.14%, avg=2557.00, stdev=475.18, samples=2
  2928. iops : min= 0, max= 45, avg=39.50, stdev= 7.78, samples=2
  2929. lat (usec) : 100=0.02%, 500=0.02%, 750=0.04%, 1000=0.92%
  2930. lat (msec) : 2=20.20%, 4=15.22%, 10=12.42%, 20=5.88%, 50=4.48%
  2931. lat (msec) : 100=11.03%, 250=16.76%, 500=8.46%, 750=3.25%, 1000=0.96%
  2932. cpu : usr=0.05%, sys=0.17%, ctx=8988, majf=0, minf=30
  2933. IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  2934. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  2935. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  2936. issued rwts: total=0,4797,0,0 short=0,0,0,0 dropped=0,0,0,0
  2937. latency : target=0, window=0, percentile=100.00%, depth=4
  2938. read: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:34:59 2019
  2939. read: IOPS=336, BW=21.0MiB/s (22.0MB/s)(2525MiB/120182msec)
  2940. slat (nsec): min=0, max=688109k, avg=55153.02, stdev=4613265.28
  2941. clat (nsec): min=0, max=1316.2M, avg=11849012.02, stdev=62320874.61
  2942. lat (nsec): min=0, max=1316.2M, avg=11905146.43, stdev=62497607.01
  2943. clat percentiles (nsec):
  2944. | 1.00th=[ 708], 5.00th=[ 716], 10.00th=[ 716],
  2945. | 20.00th=[ 724], 30.00th=[ 732], 40.00th=[ 756],
  2946. | 50.00th=[ 844], 60.00th=[ 908], 70.00th=[ 1112],
  2947. | 80.00th=[ 354304], 90.00th=[ 2998272], 95.00th=[ 55312384],
  2948. | 99.00th=[ 358612992], 99.50th=[ 471859200], 99.90th=[ 725614592],
  2949. | 99.95th=[ 859832320], 99.99th=[1199570944]
  2950. bw ( KiB/s): min= 0, max=31963, per=88.33%, avg=21542.00, stdev=14737.52, samples=2
  2951. iops : min= 0, max= 499, avg=336.00, stdev=230.52, samples=2
  2952. lat (nsec) : 750=38.94%, 1000=27.33%
  2953. lat (usec) : 2=11.77%, 4=0.04%, 10=0.52%, 20=0.59%, 50=0.19%
  2954. lat (usec) : 100=0.11%, 250=0.08%, 500=0.87%, 750=1.40%, 1000=2.68%
  2955. lat (msec) : 2=3.81%, 4=2.41%, 10=1.09%, 20=0.57%, 50=2.13%
  2956. lat (msec) : 100=2.62%, 250=1.33%, 500=1.11%, 750=0.33%, 1000=0.07%
  2957. cpu : usr=0.11%, sys=0.40%, ctx=16448, majf=0, minf=108
  2958. IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  2959. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  2960. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  2961. issued rwts: total=40399,0,0,0 short=0,0,0,0 dropped=0,0,0,0
  2962. latency : target=0, window=0, percentile=100.00%, depth=4
  2963. readwrite: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:34:59 2019
  2964. read: IOPS=45, BW=2881KiB/s (2950kB/s)(338MiB/120007msec)
  2965. slat (nsec): min=0, max=3273.4k, avg=38406.71, stdev=105301.68
  2966. clat (nsec): min=0, max=585036k, avg=9336840.69, stdev=43652742.88
  2967. lat (nsec): min=0, max=585053k, avg=9376204.46, stdev=43661535.41
  2968. clat percentiles (nsec):
  2969. | 1.00th=[ 716], 5.00th=[ 732], 10.00th=[ 740],
  2970. | 20.00th=[ 820], 30.00th=[ 924], 40.00th=[ 1064],
  2971. | 50.00th=[ 1208], 60.00th=[ 1400], 70.00th=[ 209920],
  2972. | 80.00th=[ 1155072], 90.00th=[ 6782976], 95.00th=[ 46399488],
  2973. | 99.00th=[244318208], 99.50th=[350224384], 99.90th=[509607936],
  2974. | 99.95th=[541065216], 99.99th=[583008256]
  2975. bw ( KiB/s): min= 0, max= 4260, per=11.81%, avg=2879.50, stdev=1952.32, samples=2
  2976. iops : min= 0, max= 66, avg=44.50, stdev=30.41, samples=2
  2977. write: IOPS=46, BW=2991KiB/s (3063kB/s)(351MiB/120007msec)
  2978. slat (nsec): min=0, max=57340k, avg=188996.30, stdev=811463.73
  2979. clat (nsec): min=0, max=1109.1M, avg=76748413.93, stdev=143499222.92
  2980. lat (nsec): min=0, max=1109.1M, avg=76938515.68, stdev=143449607.79
  2981. clat percentiles (usec):
  2982. | 1.00th=[ 1074], 5.00th=[ 1303], 10.00th=[ 1532],
  2983. | 20.00th=[ 2024], 30.00th=[ 2900], 40.00th=[ 4817],
  2984. | 50.00th=[ 10814], 60.00th=[ 26346], 70.00th=[ 67634],
  2985. | 80.00th=[ 107480], 90.00th=[ 235930], 95.00th=[ 425722],
  2986. | 99.00th=[ 692061], 99.50th=[ 759170], 99.90th=[ 943719],
  2987. | 99.95th=[ 985662], 99.99th=[1115685]
  2988. bw ( KiB/s): min= 0, max= 4384, per=53.94%, avg=2989.50, stdev=1972.12, samples=2
  2989. iops : min= 0, max= 68, avg=46.00, stdev=31.11, samples=2
  2990. lat (nsec) : 750=6.66%, 1000=10.83%
  2991. lat (usec) : 2=14.09%, 4=0.10%, 10=0.29%, 20=0.60%, 50=0.65%
  2992. lat (usec) : 100=0.66%, 250=0.60%, 500=1.10%, 750=1.01%, 1000=2.00%
  2993. lat (msec) : 2=12.96%, 4=10.63%, 10=7.37%, 20=4.72%, 50=6.18%
  2994. lat (msec) : 100=7.20%, 250=7.00%, 500=3.70%, 750=1.34%, 1000=0.31%
  2995. cpu : usr=0.06%, sys=0.28%, ctx=13512, majf=0, minf=34
  2996. IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  2997. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  2998. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  2999. issued rwts: total=5402,5605,0,0 short=0,0,0,0 dropped=0,0,0,0
  3000. latency : target=0, window=0, percentile=100.00%, depth=4
  3001.  
  3002. Run status group 0 (all jobs):
  3003. READ: bw=23.8MiB/s (24.0MB/s), 2881KiB/s-21.0MiB/s (2950kB/s-22.0MB/s), io=2862MiB (3001MB), run=120007-120182msec
  3004. WRITE: bw=5543KiB/s (5676kB/s), 2556KiB/s-2991KiB/s (2617kB/s-3063kB/s), io=651MiB (682MB), run=120007-120181msec
  3005.  
  3006. Disk stats (read/write):
  3007. rbd248: ios=11875/11980, merge=0/2, ticks=556245/857749, in_queue=1401822, util=15.70%
  3008. <example-benchmark-fio-server-130-benchmark-ckmtx>
  3009. randwrite: (groupid=0, jobs=1): err= 0: pid=11: Wed May 8 20:34:59 2019
  3010. write: IOPS=51, BW=3328KiB/s (3408kB/s)(390MiB/120061msec)
  3011. slat (nsec): min=0, max=657788k, avg=347116.30, stdev=8981861.13
  3012. clat (nsec): min=0, max=1140.8M, avg=76624588.97, stdev=145436001.13
  3013. lat (nsec): min=0, max=1140.8M, avg=76973399.87, stdev=145639513.75
  3014. clat percentiles (usec):
  3015. | 1.00th=[ 955], 5.00th=[ 1156], 10.00th=[ 1319],
  3016. | 20.00th=[ 1696], 30.00th=[ 2376], 40.00th=[ 3589],
  3017. | 50.00th=[ 7177], 60.00th=[ 24773], 70.00th=[ 71828],
  3018. | 80.00th=[ 108528], 90.00th=[ 233833], 95.00th=[ 400557],
  3019. | 99.00th=[ 725615], 99.50th=[ 843056], 99.90th=[ 952108],
  3020. | 99.95th=[1002439], 99.99th=[1132463]
  3021. bw ( KiB/s): min= 0, max= 3477, per=50.05%, avg=3327.50, stdev=211.42, samples=2
  3022. iops : min= 0, max= 54, avg=51.50, stdev= 3.54, samples=2
  3023. lat (usec) : 2=0.02%, 4=0.02%, 500=0.03%, 750=0.05%, 1000=1.47%
  3024. lat (msec) : 2=23.52%, 4=17.03%, 10=11.04%, 20=5.26%, 50=6.68%
  3025. lat (msec) : 100=12.39%, 250=13.27%, 500=6.18%, 750=2.20%, 1000=0.87%
  3026. cpu : usr=0.07%, sys=0.22%, ctx=11790, majf=0, minf=26
  3027. IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  3028. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  3029. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  3030. issued rwts: total=0,6241,0,0 short=0,0,0,0 dropped=0,0,0,0
  3031. latency : target=0, window=0, percentile=100.00%, depth=4
  3032. read: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:34:59 2019
  3033. read: IOPS=338, BW=21.1MiB/s (22.1MB/s)(2533MiB/120041msec)
  3034. slat (nsec): min=0, max=520529k, avg=41330.26, stdev=2721988.27
  3035. clat (nsec): min=0, max=1107.1M, avg=11867312.79, stdev=59671897.16
  3036. lat (nsec): min=0, max=1107.1M, avg=11909494.08, stdev=59741193.49
  3037. clat percentiles (nsec):
  3038. | 1.00th=[ 708], 5.00th=[ 708], 10.00th=[ 716],
  3039. | 20.00th=[ 724], 30.00th=[ 732], 40.00th=[ 748],
  3040. | 50.00th=[ 836], 60.00th=[ 892], 70.00th=[ 1096],
  3041. | 80.00th=[ 24960], 90.00th=[ 5210112], 95.00th=[ 56360960],
  3042. | 99.00th=[ 337641472], 99.50th=[ 471859200], 99.90th=[ 700448768],
  3043. | 99.95th=[ 759169024], 99.99th=[1061158912]
  3044. bw ( KiB/s): min= 0, max=31047, per=87.08%, avg=21615.00, stdev=13338.86, samples=2
  3045. iops : min= 0, max= 485, avg=337.50, stdev=208.60, samples=2
  3046. lat (nsec) : 750=41.36%, 1000=26.06%
  3047. lat (usec) : 2=10.59%, 4=0.04%, 10=0.87%, 20=1.02%, 50=0.41%
  3048. lat (usec) : 100=0.08%, 250=0.04%, 500=0.19%, 750=1.10%, 1000=2.41%
  3049. lat (msec) : 2=3.42%, 4=2.02%, 10=0.98%, 20=0.62%, 50=3.06%
  3050. lat (msec) : 100=2.91%, 250=1.43%, 500=0.96%, 750=0.36%, 1000=0.05%
  3051. cpu : usr=0.11%, sys=0.39%, ctx=15868, majf=0, minf=30
  3052. IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  3053. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  3054. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  3055. issued rwts: total=40576,0,0,0 short=0,0,0,0 dropped=0,0,0,0
  3056. latency : target=0, window=0, percentile=100.00%, depth=4
  3057. readwrite: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:34:59 2019
  3058. read: IOPS=50, BW=3214KiB/s (3291kB/s)(377MiB/120006msec)
  3059. slat (nsec): min=0, max=657744k, avg=145579.23, stdev=8474050.87
  3060. clat (nsec): min=0, max=1066.6M, avg=8382053.59, stdev=40166155.39
  3061. lat (nsec): min=0, max=1066.6M, avg=8528658.07, stdev=41033365.30
  3062. clat percentiles (nsec):
  3063. | 1.00th=[ 716], 5.00th=[ 732], 10.00th=[ 740],
  3064. | 20.00th=[ 804], 30.00th=[ 908], 40.00th=[ 1048],
  3065. | 50.00th=[ 1176], 60.00th=[ 1320], 70.00th=[ 18560],
  3066. | 80.00th=[ 1003520], 90.00th=[ 7897088], 95.00th=[ 48496640],
  3067. | 99.00th=[ 183500800], 99.50th=[ 274726912], 99.90th=[ 484442112],
  3068. | 99.95th=[ 517996544], 99.99th=[1069547520]
  3069. bw ( KiB/s): min= 0, max= 4074, per=12.95%, avg=3214.00, stdev=1216.22, samples=2
  3070. iops : min= 0, max= 63, avg=49.50, stdev=19.09, samples=2
  3071. write: IOPS=51, BW=3323KiB/s (3403kB/s)(389MiB/120006msec)
  3072. slat (nsec): min=0, max=63599k, avg=206804.07, stdev=945686.55
  3073. clat (nsec): min=0, max=1233.8M, avg=68800246.38, stdev=134987032.75
  3074. lat (nsec): min=0, max=1233.9M, avg=69008286.97, stdev=134933795.73
  3075. clat percentiles (usec):
  3076. | 1.00th=[ 996], 5.00th=[ 1221], 10.00th=[ 1434],
  3077. | 20.00th=[ 1942], 30.00th=[ 2769], 40.00th=[ 4490],
  3078. | 50.00th=[ 8717], 60.00th=[ 21890], 70.00th=[ 54264],
  3079. | 80.00th=[ 96994], 90.00th=[ 204473], 95.00th=[ 375391],
  3080. | 99.00th=[ 650118], 99.50th=[ 775947], 99.90th=[ 943719],
  3081. | 99.95th=[ 952108], 99.99th=[1233126]
  3082. bw ( KiB/s): min= 0, max= 4184, per=49.95%, avg=3321.00, stdev=1220.47, samples=2
  3083. iops : min= 0, max= 65, avg=51.50, stdev=19.09, samples=2
  3084. lat (nsec) : 750=6.62%, 1000=11.64%
  3085. lat (usec) : 2=15.40%, 4=0.07%, 10=0.26%, 20=0.58%, 50=0.62%
  3086. lat (usec) : 100=0.55%, 250=0.57%, 500=0.71%, 750=0.72%, 1000=2.13%
  3087. lat (msec) : 2=12.83%, 4=10.11%, 10=7.98%, 20=4.37%, 50=6.58%
  3088. lat (msec) : 100=7.44%, 250=6.36%, 500=3.20%, 750=0.96%, 1000=0.32%
  3089. cpu : usr=0.09%, sys=0.29%, ctx=14868, majf=0, minf=34
  3090. IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  3091. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  3092. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  3093. issued rwts: total=6028,6228,0,0 short=0,0,0,0 dropped=0,0,0,0
  3094. latency : target=0, window=0, percentile=100.00%, depth=4
  3095.  
  3096. Run status group 0 (all jobs):
  3097. READ: bw=24.2MiB/s (25.4MB/s), 3214KiB/s-21.1MiB/s (3291kB/s-22.1MB/s), io=2910MiB (3051MB), run=120006-120041msec
  3098. WRITE: bw=6650KiB/s (6810kB/s), 3323KiB/s-3328KiB/s (3403kB/s-3408kB/s), io=780MiB (818MB), run=120006-120061msec
  3099.  
  3100. Disk stats (read/write):
  3101. rbd49: ios=9506/12982, merge=0/1, ticks=545666/834922, in_queue=1369109, util=15.51%
  3102. <example-benchmark-fio-server-600-benchmark-fvxhx>
  3103. randwrite: (groupid=0, jobs=1): err= 0: pid=11: Wed May 8 20:34:59 2019
  3104. write: IOPS=19, BW=1236KiB/s (1265kB/s)(145MiB/120516msec)
  3105. slat (nsec): min=0, max=694928k, avg=1293864.89, stdev=25365740.43
  3106. clat (nsec): min=0, max=2243.2M, avg=206617078.50, stdev=402630615.34
  3107. lat (nsec): min=0, max=2243.3M, avg=207910976.65, stdev=403420903.13
  3108. clat percentiles (usec):
  3109. | 1.00th=[ 1106], 5.00th=[ 1418], 10.00th=[ 1647],
  3110. | 20.00th=[ 2180], 30.00th=[ 2802], 40.00th=[ 3556],
  3111. | 50.00th=[ 4817], 60.00th=[ 7832], 70.00th=[ 21103],
  3112. | 80.00th=[ 467665], 90.00th=[ 884999], 95.00th=[1199571],
  3113. | 99.00th=[1501561], 99.50th=[1635779], 99.90th=[1920992],
  3114. | 99.95th=[2139096], 99.99th=[2231370]
  3115. bw ( KiB/s): min= 0, max= 1654, per=26.83%, avg=1238.50, stdev=587.61, samples=2
  3116. iops : min= 0, max= 25, avg=18.50, stdev= 9.19, samples=2
  3117. lat (usec) : 750=0.09%, 1000=0.17%
  3118. lat (msec) : 2=16.44%, 4=27.54%, 10=18.76%, 20=6.45%, 50=5.16%
  3119. lat (msec) : 100=1.42%, 250=1.98%, 500=2.41%, 750=6.88%, 1000=5.59%
  3120. cpu : usr=0.02%, sys=0.09%, ctx=4282, majf=0, minf=29
  3121. IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  3122. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  3123. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  3124. issued rwts: total=0,2324,0,0 short=0,0,0,0 dropped=0,0,0,0
  3125. latency : target=0, window=0, percentile=100.00%, depth=4
  3126. read: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:34:59 2019
  3127. read: IOPS=685, BW=42.7MiB/s (44.8MB/s)(5151MiB/120619msec)
  3128. slat (nsec): min=0, max=645074k, avg=32638.70, stdev=3275070.39
  3129. clat (nsec): min=0, max=2092.4M, avg=5821962.72, stdev=68390851.36
  3130. lat (nsec): min=0, max=2092.8M, avg=5855536.09, stdev=68477975.18
  3131. clat percentiles (nsec):
  3132. | 1.00th=[ 716], 5.00th=[ 724], 10.00th=[ 740],
  3133. | 20.00th=[ 772], 30.00th=[ 788], 40.00th=[ 804],
  3134. | 50.00th=[ 836], 60.00th=[ 868], 70.00th=[ 900],
  3135. | 80.00th=[ 1112], 90.00th=[ 12480], 95.00th=[ 33024],
  3136. | 99.00th=[ 27394048], 99.50th=[ 633339904], 99.90th=[1019215872],
  3137. | 99.95th=[1199570944], 99.99th=[1518338048]
  3138. bw ( KiB/s): min= 0, max=69544, per=93.51%, avg=43953.50, stdev=36190.43, samples=2
  3139. iops : min= 0, max= 1091, avg=689.00, stdev=568.51, samples=2
  3140. lat (nsec) : 750=12.78%, 1000=65.59%
  3141. lat (usec) : 2=4.88%, 4=0.16%, 10=2.00%, 20=7.08%, 50=3.16%
  3142. lat (usec) : 100=0.23%, 250=0.11%, 500=0.14%, 750=0.15%, 1000=0.43%
  3143. lat (msec) : 2=0.94%, 4=0.68%, 10=0.41%, 20=0.18%, 50=0.21%
  3144. lat (msec) : 100=0.07%, 250=0.08%, 500=0.10%, 750=0.29%, 1000=0.23%
  3145. cpu : usr=0.16%, sys=0.65%, ctx=6935, majf=0, minf=31
  3146. IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  3147. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  3148. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  3149. issued rwts: total=82711,0,0,0 short=0,0,0,0 dropped=0,0,0,0
  3150. latency : target=0, window=0, percentile=100.00%, depth=4
  3151. readwrite: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:34:59 2019
  3152. read: IOPS=51, BW=3292KiB/s (3371kB/s)(386MiB/120003msec)
  3153. slat (nsec): min=0, max=65904k, avg=36780.12, stdev=841937.14
  3154. clat (nsec): min=0, max=749872k, avg=6530086.82, stdev=44937011.76
  3155. lat (nsec): min=0, max=749907k, avg=6567813.82, stdev=44947365.91
  3156. clat percentiles (nsec):
  3157. | 1.00th=[ 716], 5.00th=[ 724], 10.00th=[ 732],
  3158. | 20.00th=[ 756], 30.00th=[ 852], 40.00th=[ 956],
  3159. | 50.00th=[ 1096], 60.00th=[ 1272], 70.00th=[ 1672],
  3160. | 80.00th=[ 95744], 90.00th=[ 1368064], 95.00th=[ 14614528],
  3161. | 99.00th=[145752064], 99.50th=[425721856], 99.90th=[624951296],
  3162. | 99.95th=[666894336], 99.99th=[750780416]
  3163. bw ( KiB/s): min= 0, max= 5060, per=7.00%, avg=3292.00, stdev=2500.33, samples=2
  3164. iops : min= 0, max= 79, avg=51.00, stdev=39.60, samples=2
  3165. write: IOPS=53, BW=3395KiB/s (3477kB/s)(398MiB/120003msec)
  3166. slat (nsec): min=0, max=19783k, avg=197295.51, stdev=375928.11
  3167. clat (nsec): min=0, max=852237k, avg=68838535.89, stdev=129062909.48
  3168. lat (nsec): min=0, max=852306k, avg=69037001.18, stdev=129005986.35
  3169. clat percentiles (usec):
  3170. | 1.00th=[ 996], 5.00th=[ 1237], 10.00th=[ 1418], 20.00th=[ 1844],
  3171. | 30.00th=[ 2442], 40.00th=[ 3458], 50.00th=[ 6718], 60.00th=[ 20841],
  3172. | 70.00th=[ 58983], 80.00th=[105382], 90.00th=[212861], 95.00th=[371196],
  3173. | 99.00th=[624952], 99.50th=[675283], 99.90th=[759170], 99.95th=[759170],
  3174. | 99.99th=[851444]
  3175. bw ( KiB/s): min= 0, max= 5222, per=73.49%, avg=3392.50, stdev=2587.30, samples=2
  3176. iops : min= 0, max= 81, avg=52.50, stdev=40.31, samples=2
  3177. lat (nsec) : 750=9.04%, 1000=12.40%
  3178. lat (usec) : 2=13.32%, 4=0.10%, 10=0.85%, 20=1.64%, 50=1.03%
  3179. lat (usec) : 100=1.11%, 250=1.43%, 500=0.82%, 750=0.69%, 1000=1.48%
  3180. lat (msec) : 2=13.19%, 4=10.76%, 10=5.76%, 20=3.66%, 50=5.25%
  3181. lat (msec) : 100=6.13%, 250=6.62%, 500=3.37%, 750=1.33%, 1000=0.06%
  3182. cpu : usr=0.08%, sys=0.29%, ctx=13113, majf=0, minf=29
  3183. IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  3184. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  3185. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  3186. issued rwts: total=6173,6363,0,0 short=0,0,0,0 dropped=0,0,0,0
  3187. latency : target=0, window=0, percentile=100.00%, depth=4
  3188.  
  3189. Run status group 0 (all jobs):
  3190. READ: bw=45.9MiB/s (48.1MB/s), 3292KiB/s-42.7MiB/s (3371kB/s-44.8MB/s), io=5537MiB (5806MB), run=120003-120619msec
  3191. WRITE: bw=4616KiB/s (4727kB/s), 1236KiB/s-3395KiB/s (1265kB/s-3477kB/s), io=543MiB (570MB), run=120003-120516msec
  3192.  
  3193. Disk stats (read/write):
  3194. rbd199: ios=4257/8892, merge=0/1, ticks=531020/899787, in_queue=1424167, util=6.02%
  3195. <example-benchmark-fio-server-771-benchmark-l7n6n>
  3196. randwrite: (groupid=0, jobs=1): err= 0: pid=11: Wed May 8 20:34:59 2019
  3197. write: IOPS=21, BW=1351KiB/s (1384kB/s)(159MiB/120291msec)
  3198. slat (nsec): min=0, max=412566k, avg=603490.14, stdev=11034782.35
  3199. clat (nsec): min=0, max=4347.8M, avg=188943138.26, stdev=309675547.61
  3200. lat (nsec): min=0, max=4348.0M, avg=189547874.67, stdev=309681324.16
  3201. clat percentiles (usec):
  3202. | 1.00th=[ 1139], 5.00th=[ 1778], 10.00th=[ 2311],
  3203. | 20.00th=[ 3687], 30.00th=[ 5669], 40.00th=[ 8979],
  3204. | 50.00th=[ 15270], 60.00th=[ 120062], 70.00th=[ 258999],
  3205. | 80.00th=[ 367002], 90.00th=[ 549454], 95.00th=[ 775947],
  3206. | 99.00th=[1115685], 99.50th=[1300235], 99.90th=[4328522],
  3207. | 99.95th=[4328522], 99.99th=[4328522]
  3208. bw ( KiB/s): min= 0, max= 1528, per=28.54%, avg=1351.50, stdev=249.61, samples=2
  3209. iops : min= 0, max= 23, avg=20.50, stdev= 3.54, samples=2
  3210. lat (usec) : 250=0.08%, 500=0.04%, 750=0.04%, 1000=0.16%
  3211. lat (msec) : 2=6.86%, 4=15.06%, 10=20.34%, 20=10.21%, 50=5.24%
  3212. lat (msec) : 100=1.81%, 250=9.18%, 500=19.71%, 750=6.23%, 1000=3.59%
  3213. cpu : usr=0.04%, sys=0.09%, ctx=4463, majf=0, minf=29
  3214. IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  3215. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  3216. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  3217. issued rwts: total=0,2537,0,0 short=0,0,0,0 dropped=0,0,0,0
  3218. latency : target=0, window=0, percentile=100.00%, depth=4
  3219. read: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:34:59 2019
  3220. read: IOPS=163, BW=10.2MiB/s (10.7MB/s)(1237MiB/120719msec)
  3221. slat (nsec): min=0, max=10631k, avg=33850.06, stdev=183395.68
  3222. clat (nsec): min=0, max=1826.2M, avg=24367822.86, stdev=129614967.16
  3223. lat (nsec): min=0, max=1826.2M, avg=24402732.11, stdev=129638893.04
  3224. clat percentiles (nsec):
  3225. | 1.00th=[ 716], 5.00th=[ 724], 10.00th=[ 756],
  3226. | 20.00th=[ 780], 30.00th=[ 804], 40.00th=[ 860],
  3227. | 50.00th=[ 892], 60.00th=[ 948], 70.00th=[ 1192],
  3228. | 80.00th=[ 1704], 90.00th=[ 3293184], 95.00th=[ 33423360],
  3229. | 99.00th=[ 784334848], 99.50th=[ 935329792], 99.90th=[1216348160],
  3230. | 99.95th=[1317011456], 99.99th=[1686110208]
  3231. bw ( KiB/s): min= 0, max=11221, per=76.60%, avg=10553.50, stdev=943.99, samples=2
  3232. iops : min= 0, max= 175, avg=164.50, stdev=14.85, samples=2
  3233. lat (nsec) : 750=9.87%, 1000=54.06%
  3234. lat (usec) : 2=16.47%, 4=0.22%, 10=0.24%, 20=0.74%, 50=0.15%
  3235. lat (usec) : 100=0.12%, 250=0.07%, 500=0.09%, 750=0.30%, 1000=1.27%
  3236. lat (msec) : 2=4.08%, 4=2.97%, 10=2.30%, 20=1.21%, 50=1.45%
  3237. lat (msec) : 100=0.61%, 250=0.52%, 500=0.91%, 750=1.16%, 1000=0.86%
  3238. cpu : usr=0.07%, sys=0.21%, ctx=7263, majf=0, minf=30
  3239. IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  3240. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  3241. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  3242. issued rwts: total=19789,0,0,0 short=0,0,0,0 dropped=0,0,0,0
  3243. latency : target=0, window=0, percentile=100.00%, depth=4
  3244. readwrite: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:34:59 2019
  3245. read: IOPS=51, BW=3305KiB/s (3385kB/s)(387MiB/120008msec)
  3246. slat (nsec): min=0, max=3588.7k, avg=25761.76, stdev=73931.98
  3247. clat (nsec): min=0, max=959944k, avg=10211143.33, stdev=51913237.46
  3248. lat (nsec): min=0, max=959982k, avg=10237801.22, stdev=51919638.05
  3249. clat percentiles (nsec):
  3250. | 1.00th=[ 724], 5.00th=[ 732], 10.00th=[ 740],
  3251. | 20.00th=[ 788], 30.00th=[ 876], 40.00th=[ 972],
  3252. | 50.00th=[ 1112], 60.00th=[ 1256], 70.00th=[ 1512],
  3253. | 80.00th=[ 216064], 90.00th=[ 3850240], 95.00th=[ 48496640],
  3254. | 99.00th=[325058560], 99.50th=[387973120], 99.90th=[541065216],
  3255. | 99.95th=[616562688], 99.99th=[960495616]
  3256. bw ( KiB/s): min= 0, max= 4802, per=23.99%, avg=3305.50, stdev=2116.37, samples=2
  3257. iops : min= 0, max= 75, avg=51.50, stdev=33.23, samples=2
  3258. write: IOPS=52, BW=3393KiB/s (3475kB/s)(398MiB/120008msec)
  3259. slat (nsec): min=0, max=3952.3k, avg=190717.24, stdev=251762.76
  3260. clat (nsec): min=0, max=994975k, avg=65641195.08, stdev=133569305.73
  3261. lat (nsec): min=0, max=995050k, avg=65833017.35, stdev=133524713.73
  3262. clat percentiles (usec):
  3263. | 1.00th=[ 963], 5.00th=[ 1172], 10.00th=[ 1336], 20.00th=[ 1663],
  3264. | 30.00th=[ 2073], 40.00th=[ 2835], 50.00th=[ 5211], 60.00th=[ 21365],
  3265. | 70.00th=[ 64226], 80.00th=[100140], 90.00th=[154141], 95.00th=[329253],
  3266. | 99.00th=[734004], 99.50th=[834667], 99.90th=[935330], 99.95th=[952108],
  3267. | 99.99th=[994051]
  3268. bw ( KiB/s): min= 0, max= 4852, per=71.60%, avg=3391.00, stdev=2066.17, samples=2
  3269. iops : min= 0, max= 75, avg=52.50, stdev=31.82, samples=2
  3270. lat (nsec) : 750=7.23%, 1000=13.74%
  3271. lat (usec) : 2=14.70%, 4=0.04%, 10=0.29%, 20=1.01%, 50=1.01%
  3272. lat (usec) : 100=0.84%, 250=0.68%, 500=0.45%, 750=0.44%, 1000=1.74%
  3273. lat (msec) : 2=15.66%, 4=10.29%, 10=4.86%, 20=2.95%, 50=4.81%
  3274. lat (msec) : 100=7.88%, 250=7.40%, 500=2.55%, 750=0.94%, 1000=0.50%
  3275. cpu : usr=0.07%, sys=0.30%, ctx=13776, majf=0, minf=30
  3276. IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  3277. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  3278. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  3279. issued rwts: total=6198,6360,0,0 short=0,0,0,0 dropped=0,0,0,0
  3280. latency : target=0, window=0, percentile=100.00%, depth=4
  3281.  
  3282. Run status group 0 (all jobs):
  3283. READ: bw=13.5MiB/s (14.1MB/s), 3305KiB/s-10.2MiB/s (3385kB/s-10.7MB/s), io=1624MiB (1703MB), run=120008-120719msec
  3284. WRITE: bw=4737KiB/s (4850kB/s), 1351KiB/s-3393KiB/s (1384kB/s-3475kB/s), io=556MiB (583MB), run=120008-120291msec
  3285.  
  3286. Disk stats (read/write):
  3287. rbd230: ios=5135/9900, merge=0/2, ticks=595516/887454, in_queue=1475455, util=4.69%
  3288. Jobs: 37 (f=37): [_(2),m(1)][99.7%][r=1206M,w=1205M][r=18.8k,w=18.8k IOPS][eta 00m:51s] <example-benchmark-fio-server-372-benchmark-lc6p4>
  3289. randwrite: (groupid=0, jobs=1): err= 0: pid=11: Wed May 8 20:34:59 2019
  3290. write: IOPS=25, BW=1650KiB/s (1689kB/s)(194MiB/120291msec)
  3291. slat (nsec): min=0, max=773509k, avg=852811.07, stdev=20150575.43
  3292. clat (nsec): min=0, max=2310.7M, avg=154359745.14, stdev=327539286.90
  3293. lat (nsec): min=0, max=2310.8M, avg=155213400.86, stdev=327842778.04
  3294. clat percentiles (usec):
  3295. | 1.00th=[ 1037], 5.00th=[ 1352], 10.00th=[ 1598],
  3296. | 20.00th=[ 2114], 30.00th=[ 2900], 40.00th=[ 3851],
  3297. | 50.00th=[ 5276], 60.00th=[ 8160], 70.00th=[ 17171],
  3298. | 80.00th=[ 105382], 90.00th=[ 675283], 95.00th=[ 893387],
  3299. | 99.00th=[1333789], 99.50th=[1484784], 99.90th=[2021655],
  3300. | 99.95th=[2264925], 99.99th=[2298479]
  3301. bw ( KiB/s): min= 0, max= 2336, per=36.05%, avg=1650.50, stdev=969.44, samples=2
  3302. iops : min= 0, max= 36, avg=25.50, stdev=14.85, samples=2
  3303. lat (usec) : 500=0.10%, 750=0.13%, 1000=0.55%
  3304. lat (msec) : 2=17.43%, 4=22.82%, 10=23.08%, 20=7.00%, 50=3.52%
  3305. lat (msec) : 100=5.07%, 250=1.94%, 500=2.61%, 750=8.20%, 1000=4.07%
  3306. cpu : usr=0.03%, sys=0.12%, ctx=5780, majf=0, minf=27
  3307. IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  3308. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  3309. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  3310. issued rwts: total=0,3098,0,0 short=0,0,0,0 dropped=0,0,0,0
  3311. latency : target=0, window=0, percentile=100.00%, depth=4
  3312. read: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:34:59 2019
  3313. read: IOPS=610, BW=38.0MiB/s (39.9MB/s)(4588MiB/120573msec)
  3314. slat (nsec): min=0, max=771813k, avg=42177.78, stdev=4068032.92
  3315. clat (nsec): min=0, max=1795.7M, avg=6503281.36, stdev=66799159.40
  3316. lat (nsec): min=0, max=1795.8M, avg=6546418.17, stdev=66930745.83
  3317. clat percentiles (nsec):
  3318. | 1.00th=[ 708], 5.00th=[ 724], 10.00th=[ 732],
  3319. | 20.00th=[ 764], 30.00th=[ 780], 40.00th=[ 796],
  3320. | 50.00th=[ 828], 60.00th=[ 860], 70.00th=[ 884],
  3321. | 80.00th=[ 972], 90.00th=[ 11840], 95.00th=[ 995328],
  3322. | 99.00th=[ 91750400], 99.50th=[ 599785472], 99.90th=[ 952107008],
  3323. | 99.95th=[1115684864], 99.99th=[1400897536]
  3324. bw ( KiB/s): min= 0, max=63106, per=93.37%, avg=39021.00, stdev=34061.33, samples=2
  3325. iops : min= 0, max= 990, avg=611.50, stdev=535.28, samples=2
  3326. lat (nsec) : 750=15.86%, 1000=64.91%
  3327. lat (usec) : 2=5.86%, 4=0.14%, 10=1.15%, 20=5.13%, 50=0.69%
  3328. lat (usec) : 100=0.19%, 250=0.10%, 500=0.19%, 750=0.28%, 1000=0.49%
  3329. lat (msec) : 2=1.20%, 4=1.00%, 10=0.73%, 20=0.33%, 50=0.46%
  3330. lat (msec) : 100=0.31%, 250=0.10%, 500=0.23%, 750=0.38%, 1000=0.18%
  3331. cpu : usr=0.15%, sys=0.61%, ctx=8969, majf=0, minf=31
  3332. IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  3333. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  3334. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  3335. issued rwts: total=73655,0,0,0 short=0,0,0,0 dropped=0,0,0,0
  3336. latency : target=0, window=0, percentile=100.00%, depth=4
  3337. readwrite: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:34:59 2019
  3338. read: IOPS=44, BW=2840KiB/s (2908kB/s)(333MiB/120025msec)
  3339. slat (nsec): min=0, max=2616.8k, avg=30791.64, stdev=88068.41
  3340. clat (nsec): min=0, max=642034k, avg=7264005.10, stdev=39713153.45
  3341. lat (nsec): min=0, max=642063k, avg=7295930.78, stdev=39719952.99
  3342. clat percentiles (nsec):
  3343. | 1.00th=[ 716], 5.00th=[ 724], 10.00th=[ 732],
  3344. | 20.00th=[ 788], 30.00th=[ 876], 40.00th=[ 988],
  3345. | 50.00th=[ 1144], 60.00th=[ 1304], 70.00th=[ 9920],
  3346. | 80.00th=[ 684032], 90.00th=[ 2899968], 95.00th=[ 41680896],
  3347. | 99.00th=[181403648], 99.50th=[362807296], 99.90th=[501219328],
  3348. | 99.95th=[530579456], 99.99th=[641728512]
  3349. bw ( KiB/s): min= 0, max= 4536, per=6.80%, avg=2840.00, stdev=2398.51, samples=2
  3350. iops : min= 0, max= 70, avg=43.50, stdev=37.48, samples=2
  3351. write: IOPS=45, BW=2935KiB/s (3005kB/s)(344MiB/120025msec)
  3352. slat (nsec): min=0, max=5247.4k, avg=183124.89, stdev=260290.31
  3353. clat (nsec): min=0, max=922637k, avg=80167211.55, stdev=145551293.96
  3354. lat (nsec): min=0, max=922680k, avg=80351609.68, stdev=145505979.46
  3355. clat percentiles (usec):
  3356. | 1.00th=[ 1029], 5.00th=[ 1237], 10.00th=[ 1434], 20.00th=[ 1860],
  3357. | 30.00th=[ 2573], 40.00th=[ 3916], 50.00th=[ 8979], 60.00th=[ 28705],
  3358. | 70.00th=[ 68682], 80.00th=[113771], 90.00th=[278922], 95.00th=[450888],
  3359. | 99.00th=[650118], 99.50th=[725615], 99.90th=[859833], 99.95th=[918553],
  3360. | 99.99th=[918553]
  3361. bw ( KiB/s): min= 0, max= 4651, per=64.06%, avg=2932.50, stdev=2430.33, samples=2
  3362. iops : min= 0, max= 72, avg=45.00, stdev=38.18, samples=2
  3363. lat (nsec) : 750=7.63%, 1000=12.33%
  3364. lat (usec) : 2=13.99%, 4=0.14%, 10=0.38%, 20=0.85%, 50=0.85%
  3365. lat (usec) : 100=0.89%, 250=1.16%, 500=0.70%, 750=0.79%, 1000=1.73%
  3366. lat (msec) : 2=13.41%, 4=10.59%, 10=5.92%, 20=3.28%, 50=5.86%
  3367. lat (msec) : 100=7.03%, 250=6.65%, 500=3.85%, 750=1.83%, 1000=0.17%
  3368. cpu : usr=0.06%, sys=0.26%, ctx=12141, majf=0, minf=30
  3369. IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  3370. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  3371. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  3372. issued rwts: total=5326,5501,0,0 short=0,0,0,0 dropped=0,0,0,0
  3373. latency : target=0, window=0, percentile=100.00%, depth=4
  3374.  
  3375. Run status group 0 (all jobs):
  3376. READ: bw=40.8MiB/s (42.8MB/s), 2840KiB/s-38.0MiB/s (2908kB/s-39.9MB/s), io=4921MiB (5160MB), run=120025-120573msec
  3377. WRITE: bw=4578KiB/s (4688kB/s), 1650KiB/s-2935KiB/s (1689kB/s-3005kB/s), io=538MiB (564MB), run=120025-120291msec
  3378.  
  3379. Disk stats (read/write):
  3380. rbd130: ios=6082/9151, merge=0/1, ticks=546406/923365, in_queue=1462130, util=7.65%
  3381. <example-benchmark-fio-server-452-benchmark-7lt6f>
  3382. randwrite: (groupid=0, jobs=1): err= 0: pid=11: Wed May 8 20:34:59 2019
  3383. write: IOPS=45, BW=2916KiB/s (2986kB/s)(346MiB/121551msec)
  3384. slat (nsec): min=0, max=8601.9M, avg=1768759.47, stdev=115624208.02
  3385. clat (nsec): min=0, max=11024M, avg=85317961.64, stdev=306099695.43
  3386. lat (nsec): min=0, max=11024M, avg=87087195.73, stdev=327190446.01
  3387. clat percentiles (usec):
  3388. | 1.00th=[ 914], 5.00th=[ 1745], 10.00th=[ 2507],
  3389. | 20.00th=[ 3720], 30.00th=[ 5145], 40.00th=[ 7504],
  3390. | 50.00th=[ 10421], 60.00th=[ 15533], 70.00th=[ 29492],
  3391. | 80.00th=[ 102237], 90.00th=[ 219153], 95.00th=[ 484443],
  3392. | 99.00th=[ 759170], 99.50th=[ 935330], 99.90th=[ 3305112],
  3393. | 99.95th=[ 8657044], 99.99th=[11072963]
  3394. bw ( KiB/s): min= 0, max= 3061, per=47.12%, avg=2950.00, stdev=156.98, samples=2
  3395. iops : min= 0, max= 47, avg=45.50, stdev= 2.12, samples=2
  3396. lat (usec) : 4=0.04%, 50=0.02%, 250=0.07%, 500=0.04%, 750=0.33%
  3397. lat (usec) : 1000=0.89%
  3398. lat (msec) : 2=5.53%, 4=15.37%, 10=26.41%, 20=16.22%, 50=9.83%
  3399. lat (msec) : 100=5.17%, 250=11.51%, 500=3.97%, 750=3.63%, 1000=0.63%
  3400. cpu : usr=0.05%, sys=0.19%, ctx=9026, majf=0, minf=30
  3401. IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  3402. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  3403. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  3404. issued rwts: total=0,5535,0,0 short=0,0,0,0 dropped=0,0,0,0
  3405. latency : target=0, window=0, percentile=100.00%, depth=4
  3406. read: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:34:59 2019
  3407. read: IOPS=79, BW=5087KiB/s (5210kB/s)(599MiB/120566msec)
  3408. slat (nsec): min=0, max=11811k, avg=55106.52, stdev=241135.38
  3409. clat (nsec): min=0, max=1838.1M, avg=50275141.81, stdev=187121374.36
  3410. lat (nsec): min=0, max=1838.1M, avg=50331333.59, stdev=187141422.67
  3411. clat percentiles (nsec):
  3412. | 1.00th=[ 724], 5.00th=[ 764], 10.00th=[ 788],
  3413. | 20.00th=[ 852], 30.00th=[ 908], 40.00th=[ 1012],
  3414. | 50.00th=[ 1256], 60.00th=[ 1512], 70.00th=[ 1138688],
  3415. | 80.00th=[ 2932736], 90.00th=[ 15400960], 95.00th=[ 549453824],
  3416. | 99.00th=[ 935329792], 99.50th=[1052770304], 99.90th=[1350565888],
  3417. | 99.95th=[1585446912], 99.99th=[1837105152]
  3418. bw ( KiB/s): min= 0, max= 5128, per=61.11%, avg=5102.50, stdev=36.06, samples=2
  3419. iops : min= 0, max= 80, avg=79.50, stdev= 0.71, samples=2
  3420. lat (nsec) : 750=4.34%, 1000=35.13%
  3421. lat (usec) : 2=23.43%, 4=0.21%, 10=0.10%, 20=0.85%, 50=0.27%
  3422. lat (usec) : 100=0.21%, 250=0.15%, 500=0.19%, 750=0.66%, 1000=3.03%
  3423. lat (msec) : 2=7.55%, 4=6.70%, 10=5.32%, 20=2.76%, 50=1.40%
  3424. lat (msec) : 100=0.45%, 250=0.47%, 500=1.35%, 750=2.96%, 1000=1.87%
  3425. cpu : usr=0.04%, sys=0.14%, ctx=6779, majf=0, minf=30
  3426. IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  3427. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  3428. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  3429. issued rwts: total=9581,0,0,0 short=0,0,0,0 dropped=0,0,0,0
  3430. latency : target=0, window=0, percentile=100.00%, depth=4
  3431. readwrite: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:34:59 2019
  3432. read: IOPS=51, BW=3278KiB/s (3357kB/s)(384MiB/120011msec)
  3433. slat (nsec): min=0, max=4415.6k, avg=40057.69, stdev=108722.41
  3434. clat (nsec): min=0, max=810846k, avg=15542310.91, stdev=68280778.67
  3435. lat (nsec): min=0, max=810878k, avg=15583357.86, stdev=68290237.45
  3436. clat percentiles (nsec):
  3437. | 1.00th=[ 724], 5.00th=[ 740], 10.00th=[ 788],
  3438. | 20.00th=[ 884], 30.00th=[ 996], 40.00th=[ 1144],
  3439. | 50.00th=[ 1288], 60.00th=[ 14016], 70.00th=[ 954368],
  3440. | 80.00th=[ 2179072], 90.00th=[ 31850496], 95.00th=[ 64749568],
  3441. | 99.00th=[400556032], 99.50th=[574619648], 99.90th=[717225984],
  3442. | 99.95th=[725614592], 99.99th=[809500672]
  3443. bw ( KiB/s): min= 0, max= 5222, per=39.26%, avg=3278.00, stdev=2749.23, samples=2
  3444. iops : min= 0, max= 81, avg=50.50, stdev=43.13, samples=2
  3445. write: IOPS=52, BW=3387KiB/s (3469kB/s)(397MiB/120011msec)
  3446. slat (nsec): min=0, max=11202k, avg=195281.65, stdev=275030.68
  3447. clat (nsec): min=0, max=980601k, avg=60513733.50, stdev=123539267.98
  3448. lat (nsec): min=0, max=980657k, avg=60710238.12, stdev=123494453.74
  3449. clat percentiles (usec):
  3450. | 1.00th=[ 1012], 5.00th=[ 1221], 10.00th=[ 1385], 20.00th=[ 1745],
  3451. | 30.00th=[ 2278], 40.00th=[ 3195], 50.00th=[ 5342], 60.00th=[ 14615],
  3452. | 70.00th=[ 45351], 80.00th=[ 87557], 90.00th=[154141], 95.00th=[367002],
  3453. | 99.00th=[574620], 99.50th=[658506], 99.90th=[884999], 99.95th=[943719],
  3454. | 99.99th=[977273]
  3455. bw ( KiB/s): min= 0, max= 5397, per=54.07%, avg=3385.00, stdev=2845.40, samples=2
  3456. iops : min= 0, max= 84, avg=52.50, stdev=44.55, samples=2
  3457. lat (nsec) : 750=3.39%, 1000=11.52%
  3458. lat (usec) : 2=14.24%, 4=0.07%, 10=0.18%, 20=0.42%, 50=0.50%
  3459. lat (usec) : 100=0.38%, 250=0.55%, 500=0.29%, 750=0.99%, 1000=2.86%
  3460. lat (msec) : 2=16.39%, 4=12.80%, 10=7.15%, 20=3.89%, 50=6.17%
  3461. lat (msec) : 100=8.40%, 250=5.23%, 500=3.12%, 750=1.36%, 1000=0.14%
  3462. cpu : usr=0.08%, sys=0.31%, ctx=16088, majf=0, minf=30
  3463. IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  3464. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  3465. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  3466. issued rwts: total=6146,6350,0,0 short=0,0,0,0 dropped=0,0,0,0
  3467. latency : target=0, window=0, percentile=100.00%, depth=4
  3468.  
  3469. Run status group 0 (all jobs):
  3470. READ: bw=8350KiB/s (8551kB/s), 3278KiB/s-5087KiB/s (3357kB/s-5210kB/s), io=983MiB (1031MB), run=120011-120566msec
  3471. WRITE: bw=6260KiB/s (6411kB/s), 2916KiB/s-3387KiB/s (2986kB/s-3469kB/s), io=743MiB (779MB), run=120011-121551msec
  3472.  
  3473. Disk stats (read/write):
  3474. rbd166: ios=6007/12363, merge=0/3, ticks=619271/875615, in_queue=1485609, util=4.64%
  3475. <example-benchmark-fio-server-170-benchmark-7ll8b>
  3476. randwrite: (groupid=0, jobs=1): err= 0: pid=11: Wed May 8 20:34:59 2019
  3477. write: IOPS=25, BW=1658KiB/s (1698kB/s)(195MiB/120385msec)
  3478. slat (nsec): min=0, max=887718k, avg=1213744.90, stdev=27334605.55
  3479. clat (nsec): min=0, max=2050.6M, avg=153222238.41, stdev=315111553.22
  3480. lat (nsec): min=0, max=2050.6M, avg=154436698.50, stdev=316707758.24
  3481. clat percentiles (usec):
  3482. | 1.00th=[ 1156], 5.00th=[ 1483], 10.00th=[ 1827],
  3483. | 20.00th=[ 2474], 30.00th=[ 3392], 40.00th=[ 5080],
  3484. | 50.00th=[ 7504], 60.00th=[ 13173], 70.00th=[ 31851],
  3485. | 80.00th=[ 121111], 90.00th=[ 675283], 95.00th=[ 884999],
  3486. | 99.00th=[1283458], 99.50th=[1384121], 99.90th=[1887437],
  3487. | 99.95th=[2021655], 99.99th=[2055209]
  3488. bw ( KiB/s): min= 0, max= 1787, per=32.78%, avg=1660.50, stdev=178.90, samples=2
  3489. iops : min= 0, max= 27, avg=25.00, stdev= 2.83, samples=2
  3490. lat (usec) : 2=0.03%, 750=0.16%, 1000=0.19%
  3491. lat (msec) : 2=12.45%, 4=21.57%, 10=21.57%, 20=9.34%, 50=7.64%
  3492. lat (msec) : 100=5.49%, 250=3.59%, 500=2.86%, 750=7.54%, 1000=4.40%
  3493. cpu : usr=0.04%, sys=0.12%, ctx=5748, majf=0, minf=30
  3494. IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  3495. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  3496. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  3497. issued rwts: total=0,3116,0,0 short=0,0,0,0 dropped=0,0,0,0
  3498. latency : target=0, window=0, percentile=100.00%, depth=4
  3499. read: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:34:59 2019
  3500. read: IOPS=671, BW=41.8MiB/s (43.8MB/s)(5034MiB/120435msec)
  3501. slat (nsec): min=0, max=880995k, avg=42469.66, stdev=4585790.36
  3502. clat (nsec): min=0, max=1644.8M, avg=5920176.44, stdev=63524581.82
  3503. lat (nsec): min=0, max=1644.8M, avg=5963596.00, stdev=63696049.76
  3504. clat percentiles (nsec):
  3505. | 1.00th=[ 708], 5.00th=[ 716], 10.00th=[ 724],
  3506. | 20.00th=[ 764], 30.00th=[ 780], 40.00th=[ 804],
  3507. | 50.00th=[ 844], 60.00th=[ 876], 70.00th=[ 908],
  3508. | 80.00th=[ 1480], 90.00th=[ 12864], 95.00th=[ 757760],
  3509. | 99.00th=[ 63176704], 99.50th=[ 574619648], 99.90th=[ 952107008],
  3510. | 99.95th=[1044381696], 99.99th=[1300234240]
  3511. bw ( KiB/s): min= 0, max=69975, per=93.01%, avg=42879.00, stdev=38319.53, samples=2
  3512. iops : min= 0, max= 1098, avg=672.00, stdev=602.45, samples=2
  3513. lat (nsec) : 750=16.86%, 1000=58.14%
  3514. lat (usec) : 2=5.53%, 4=0.13%, 10=1.32%, 20=9.56%, 50=2.74%
  3515. lat (usec) : 100=0.26%, 250=0.13%, 500=0.15%, 750=0.17%, 1000=0.53%
  3516. lat (msec) : 2=1.20%, 4=0.87%, 10=0.59%, 20=0.30%, 50=0.40%
  3517. lat (msec) : 100=0.25%, 250=0.07%, 500=0.22%, 750=0.34%, 1000=0.18%
  3518. cpu : usr=0.16%, sys=0.67%, ctx=8878, majf=0, minf=34
  3519. IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  3520. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  3521. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  3522. issued rwts: total=80815,0,0,0 short=0,0,0,0 dropped=0,0,0,0
  3523. latency : target=0, window=0, percentile=100.00%, depth=4
  3524. readwrite: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:34:59 2019
  3525. read: IOPS=51, BW=3314KiB/s (3394kB/s)(389MiB/120035msec)
  3526. slat (nsec): min=0, max=1675.7k, avg=27582.43, stdev=64411.66
  3527. clat (nsec): min=0, max=694594k, avg=8549233.54, stdev=46985004.43
  3528. lat (nsec): min=0, max=694628k, avg=8577802.47, stdev=46991864.10
  3529. clat percentiles (nsec):
  3530. | 1.00th=[ 716], 5.00th=[ 724], 10.00th=[ 732],
  3531. | 20.00th=[ 756], 30.00th=[ 860], 40.00th=[ 964],
  3532. | 50.00th=[ 1112], 60.00th=[ 1256], 70.00th=[ 1656],
  3533. | 80.00th=[ 337920], 90.00th=[ 2801664], 95.00th=[ 39583744],
  3534. | 99.00th=[256901120], 99.50th=[387973120], 99.90th=[526385152],
  3535. | 99.95th=[624951296], 99.99th=[692060160]
  3536. bw ( KiB/s): min= 0, max= 5128, per=7.19%, avg=3314.50, stdev=2564.68, samples=2
  3537. iops : min= 0, max= 80, avg=51.50, stdev=40.31, samples=2
  3538. write: IOPS=53, BW=3418KiB/s (3500kB/s)(401MiB/120035msec)
  3539. slat (nsec): min=0, max=42017k, avg=186096.19, stdev=597925.65
  3540. clat (nsec): min=0, max=1006.8M, avg=66420593.68, stdev=125702557.93
  3541. lat (nsec): min=0, max=1006.9M, avg=66607957.18, stdev=125661614.59
  3542. clat percentiles (usec):
  3543. | 1.00th=[ 1029], 5.00th=[ 1237], 10.00th=[ 1450],
  3544. | 20.00th=[ 1827], 30.00th=[ 2311], 40.00th=[ 3294],
  3545. | 50.00th=[ 6521], 60.00th=[ 22938], 70.00th=[ 61604],
  3546. | 80.00th=[ 105382], 90.00th=[ 173016], 95.00th=[ 358613],
  3547. | 99.00th=[ 633340], 99.50th=[ 692061], 99.90th=[ 801113],
  3548. | 99.95th=[ 884999], 99.99th=[1010828]
  3549. bw ( KiB/s): min= 0, max= 5280, per=67.45%, avg=3416.50, stdev=2635.39, samples=2
  3550. iops : min= 0, max= 82, avg=53.00, stdev=41.01, samples=2
  3551. lat (nsec) : 750=9.02%, 1000=11.91%
  3552. lat (usec) : 2=13.84%, 4=0.06%, 10=0.25%, 20=1.00%, 50=0.92%
  3553. lat (usec) : 100=1.05%, 250=1.01%, 500=0.73%, 750=0.59%, 1000=1.78%
  3554. lat (msec) : 2=13.49%, 4=11.63%, 10=5.38%, 20=3.46%, 50=5.36%
  3555. lat (msec) : 100=6.70%, 250=7.63%, 500=2.93%, 750=1.18%, 1000=0.10%
  3556. cpu : usr=0.07%, sys=0.30%, ctx=13879, majf=0, minf=33
  3557. IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  3558. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  3559. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  3560. issued rwts: total=6215,6408,0,0 short=0,0,0,0 dropped=0,0,0,0
  3561. latency : target=0, window=0, percentile=100.00%, depth=4
  3562.  
  3563. Run status group 0 (all jobs):
  3564. READ: bw=45.0MiB/s (47.2MB/s), 3314KiB/s-41.8MiB/s (3394kB/s-43.8MB/s), io=5422MiB (5686MB), run=120035-120435msec
  3565. WRITE: bw=5066KiB/s (5187kB/s), 1658KiB/s-3418KiB/s (1698kB/s-3500kB/s), io=596MiB (624MB), run=120035-120385msec
  3566.  
  3567. Disk stats (read/write):
  3568. rbd54: ios=5720/9776, merge=0/1, ticks=553989/877423, in_queue=1423562, util=7.76%
  3569. <example-benchmark-fio-server-305-benchmark-bvd8p>
  3570. randwrite: (groupid=0, jobs=1): err= 0: pid=11: Wed May 8 20:34:59 2019
  3571. write: IOPS=46, BW=2967KiB/s (3038kB/s)(348MiB/120084msec)
  3572. slat (nsec): min=0, max=591926k, avg=385076.58, stdev=10580791.50
  3573. clat (nsec): min=0, max=1249.9M, avg=86328692.16, stdev=149286241.82
  3574. lat (nsec): min=0, max=1249.0M, avg=86715682.26, stdev=149627952.29
  3575. clat percentiles (usec):
  3576. | 1.00th=[ 1004], 5.00th=[ 1237], 10.00th=[ 1467],
  3577. | 20.00th=[ 1958], 30.00th=[ 2868], 40.00th=[ 4686],
  3578. | 50.00th=[ 11076], 60.00th=[ 53216], 70.00th=[ 90702],
  3579. | 80.00th=[ 123208], 90.00th=[ 258999], 95.00th=[ 408945],
  3580. | 99.00th=[ 708838], 99.50th=[ 809501], 99.90th=[1052771],
  3581. | 99.95th=[1233126], 99.99th=[1249903]
  3582. bw ( KiB/s): min= 0, max= 3388, per=45.08%, avg=2966.50, stdev=596.09, samples=2
  3583. iops : min= 0, max= 52, avg=45.50, stdev= 9.19, samples=2
  3584. lat (usec) : 4=0.02%, 250=0.02%, 750=0.02%, 1000=0.93%
  3585. lat (msec) : 2=19.57%, 4=16.80%, 10=11.79%, 20=4.85%, 50=5.59%
  3586. lat (msec) : 100=13.28%, 250=16.71%, 500=7.08%, 750=2.50%, 1000=0.74%
  3587. cpu : usr=0.05%, sys=0.21%, ctx=10453, majf=0, minf=29
  3588. IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  3589. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  3590. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  3591. issued rwts: total=0,5564,0,0 short=0,0,0,0 dropped=0,0,0,0
  3592. latency : target=0, window=0, percentile=100.00%, depth=4
  3593. read: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:34:59 2019
  3594. read: IOPS=316, BW=19.7MiB/s (20.7MB/s)(2370MiB/120032msec)
  3595. slat (nsec): min=0, max=590286k, avg=50581.37, stdev=3970019.69
  3596. clat (nsec): min=0, max=1148.2M, avg=12614914.63, stdev=60910243.05
  3597. lat (nsec): min=0, max=1148.2M, avg=12666319.66, stdev=61045819.97
  3598. clat percentiles (nsec):
  3599. | 1.00th=[ 708], 5.00th=[ 708], 10.00th=[ 716],
  3600. | 20.00th=[ 724], 30.00th=[ 732], 40.00th=[ 748],
  3601. | 50.00th=[ 836], 60.00th=[ 892], 70.00th=[ 1112],
  3602. | 80.00th=[ 18560], 90.00th=[ 9371648], 95.00th=[ 59506688],
  3603. | 99.00th=[ 350224384], 99.50th=[ 476053504], 99.90th=[ 692060160],
  3604. | 99.95th=[ 750780416], 99.99th=[1010827264]
  3605. bw ( KiB/s): min= 0, max=28456, per=85.14%, avg=20222.00, stdev=11644.63, samples=2
  3606. iops : min= 0, max= 444, avg=315.50, stdev=181.73, samples=2
  3607. lat (nsec) : 750=40.78%, 1000=26.21%
  3608. lat (usec) : 2=10.59%, 4=0.03%, 10=1.41%, 20=1.02%, 50=0.30%
  3609. lat (usec) : 100=0.13%, 250=0.06%, 500=0.15%, 750=0.86%, 1000=2.15%
  3610. lat (msec) : 2=3.24%, 4=1.98%, 10=1.16%, 20=0.65%, 50=3.31%
  3611. lat (msec) : 100=2.89%, 250=1.61%, 500=1.07%, 750=0.37%, 1000=0.03%
  3612. cpu : usr=0.09%, sys=0.37%, ctx=14721, majf=0, minf=29
  3613. IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  3614. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  3615. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  3616. issued rwts: total=37938,0,0,0 short=0,0,0,0 dropped=0,0,0,0
  3617. latency : target=0, window=0, percentile=100.00%, depth=4
  3618. readwrite: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:34:59 2019
  3619. read: IOPS=55, BW=3535KiB/s (3620kB/s)(414MiB/120039msec)
  3620. slat (nsec): min=0, max=484899k, avg=107021.58, stdev=5955708.76
  3621. clat (nsec): min=0, max=759194k, avg=6801948.12, stdev=36574853.34
  3622. lat (nsec): min=0, max=759246k, avg=6909910.12, stdev=37050983.59
  3623. clat percentiles (nsec):
  3624. | 1.00th=[ 716], 5.00th=[ 724], 10.00th=[ 732],
  3625. | 20.00th=[ 796], 30.00th=[ 892], 40.00th=[ 1004],
  3626. | 50.00th=[ 1160], 60.00th=[ 1320], 70.00th=[ 15040],
  3627. | 80.00th=[ 888832], 90.00th=[ 4145152], 95.00th=[ 37486592],
  3628. | 99.00th=[170917888], 99.50th=[258998272], 99.90th=[476053504],
  3629. | 99.95th=[608174080], 99.99th=[759169024]
  3630. bw ( KiB/s): min= 0, max= 4618, per=14.88%, avg=3535.00, stdev=1531.59, samples=2
  3631. iops : min= 0, max= 72, avg=55.00, stdev=24.04, samples=2
  3632. write: IOPS=56, BW=3615KiB/s (3702kB/s)(424MiB/120039msec)
  3633. slat (nsec): min=0, max=589992k, avg=284036.47, stdev=7194313.68
  3634. clat (nsec): min=0, max=977765k, avg=64022066.99, stdev=122762405.28
  3635. lat (nsec): min=0, max=1018.0M, avg=64307305.36, stdev=123184191.18
  3636. clat percentiles (usec):
  3637. | 1.00th=[ 1057], 5.00th=[ 1287], 10.00th=[ 1483], 20.00th=[ 1926],
  3638. | 30.00th=[ 2573], 40.00th=[ 3785], 50.00th=[ 6849], 60.00th=[ 17957],
  3639. | 70.00th=[ 53216], 80.00th=[ 98042], 90.00th=[187696], 95.00th=[350225],
  3640. | 99.00th=[591397], 99.50th=[675283], 99.90th=[859833], 99.95th=[884999],
  3641. | 99.99th=[977273]
  3642. bw ( KiB/s): min= 0, max= 4677, per=54.92%, avg=3614.00, stdev=1503.31, samples=2
  3643. iops : min= 0, max= 73, avg=56.00, stdev=24.04, samples=2
  3644. lat (nsec) : 750=7.62%, 1000=12.02%
  3645. lat (usec) : 2=14.45%, 4=0.07%, 10=0.25%, 20=0.67%, 50=0.63%
  3646. lat (usec) : 100=0.78%, 250=0.84%, 500=0.75%, 750=0.63%, 1000=1.73%
  3647. lat (msec) : 2=13.27%, 4=11.49%, 10=8.05%, 20=3.89%, 50=5.53%
  3648. lat (msec) : 100=6.67%, 250=6.58%, 500=2.96%, 750=1.01%, 1000=0.12%
  3649. cpu : usr=0.06%, sys=0.33%, ctx=15762, majf=0, minf=36
  3650. IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  3651. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  3652. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  3653. issued rwts: total=6631,6777,0,0 short=0,0,0,0 dropped=0,0,0,0
  3654. latency : target=0, window=0, percentile=100.00%, depth=4
  3655.  
  3656. Run status group 0 (all jobs):
  3657. READ: bw=23.2MiB/s (24.3MB/s), 3535KiB/s-19.7MiB/s (3620kB/s-20.7MB/s), io=2784MiB (2920MB), run=120032-120039msec
  3658. WRITE: bw=6580KiB/s (6738kB/s), 2967KiB/s-3615KiB/s (3038kB/s-3702kB/s), io=772MiB (809MB), run=120039-120084msec
  3659.  
  3660. Disk stats (read/write):
  3661. rbd105: ios=9005/12689, merge=0/1, ticks=537857/830714, in_queue=1357530, util=14.73%
  3662. <example-benchmark-fio-server-214-benchmark-g26rt>
  3663. randwrite: (groupid=0, jobs=1): err= 0: pid=11: Wed May 8 20:34:59 2019
  3664. write: IOPS=39, BW=2501KiB/s (2561kB/s)(293MiB/120100msec)
  3665. slat (nsec): min=0, max=878508k, avg=512354.66, stdev=14978981.02
  3666. clat (nsec): min=0, max=1750.0M, avg=102508712.84, stdev=189336994.89
  3667. lat (nsec): min=0, max=1750.1M, avg=103022267.99, stdev=189729647.89
  3668. clat percentiles (usec):
  3669. | 1.00th=[ 1045], 5.00th=[ 1237], 10.00th=[ 1434],
  3670. | 20.00th=[ 1942], 30.00th=[ 2835], 40.00th=[ 4752],
  3671. | 50.00th=[ 10552], 60.00th=[ 51643], 70.00th=[ 103285],
  3672. | 80.00th=[ 139461], 90.00th=[ 304088], 95.00th=[ 549454],
  3673. | 99.00th=[ 843056], 99.50th=[1010828], 99.90th=[1602225],
  3674. | 99.95th=[1702888], 99.99th=[1753220]
  3675. bw ( KiB/s): min= 0, max= 1899, per=33.04%, avg=1899.00, stdev= 0.00, samples=1
  3676. iops : min= 0, max= 29, avg=29.00, stdev= 0.00, samples=1
  3677. lat (usec) : 500=0.02%, 750=0.02%, 1000=0.58%
  3678. lat (msec) : 2=20.34%, 4=16.12%, 10=12.30%, 20=5.71%, 50=4.80%
  3679. lat (msec) : 100=9.08%, 250=19.02%, 500=6.48%, 750=3.67%, 1000=1.36%
  3680. cpu : usr=0.05%, sys=0.17%, ctx=8743, majf=0, minf=29
  3681. IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  3682. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  3683. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  3684. issued rwts: total=0,4690,0,0 short=0,0,0,0 dropped=0,0,0,0
  3685. latency : target=0, window=0, percentile=100.00%, depth=4
  3686. read: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:34:59 2019
  3687. read: IOPS=371, BW=23.2MiB/s (24.3MB/s)(2787MiB/120081msec)
  3688. slat (nsec): min=0, max=864080k, avg=51361.43, stdev=4763726.52
  3689. clat (nsec): min=0, max=1359.3M, avg=10722193.91, stdev=64689666.11
  3690. lat (nsec): min=0, max=1359.4M, avg=10774402.02, stdev=64870471.03
  3691. clat percentiles (nsec):
  3692. | 1.00th=[ 716], 5.00th=[ 716], 10.00th=[ 724],
  3693. | 20.00th=[ 732], 30.00th=[ 764], 40.00th=[ 804],
  3694. | 50.00th=[ 844], 60.00th=[ 884], 70.00th=[ 940],
  3695. | 80.00th=[ 1160], 90.00th=[ 1105920], 95.00th=[ 41680896],
  3696. | 99.00th=[ 350224384], 99.50th=[ 497025024], 99.90th=[ 817889280],
  3697. | 99.95th=[ 918552576], 99.99th=[1166016512]
  3698. bw ( KiB/s): min= 0, max=36891, per=88.40%, avg=23777.50, stdev=18545.29, samples=2
  3699. iops : min= 0, max= 577, avg=371.50, stdev=290.62, samples=2
  3700. lat (nsec) : 750=27.86%, 1000=47.00%
  3701. lat (usec) : 2=10.92%, 4=0.06%, 10=0.77%, 20=0.56%, 50=0.15%
  3702. lat (usec) : 100=0.09%, 250=0.05%, 500=0.21%, 750=0.58%, 1000=1.35%
  3703. lat (msec) : 2=2.12%, 4=1.40%, 10=0.69%, 20=0.34%, 50=1.35%
  3704. lat (msec) : 100=1.96%, 250=1.11%, 500=0.95%, 750=0.32%, 1000=0.14%
  3705. cpu : usr=0.12%, sys=0.38%, ctx=11228, majf=0, minf=108
  3706. IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  3707. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  3708. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  3709. issued rwts: total=44650,0,0,0 short=0,0,0,0 dropped=0,0,0,0
  3710. latency : target=0, window=0, percentile=100.00%, depth=4
  3711. readwrite: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:34:59 2019
  3712. read: IOPS=49, BW=3136KiB/s (3212kB/s)(368MiB/120005msec)
  3713. slat (nsec): min=0, max=1914.8k, avg=31155.07, stdev=74360.36
  3714. clat (nsec): min=0, max=744812k, avg=9110926.27, stdev=44539320.61
  3715. lat (nsec): min=0, max=744837k, avg=9143035.67, stdev=44544914.81
  3716. clat percentiles (nsec):
  3717. | 1.00th=[ 716], 5.00th=[ 732], 10.00th=[ 740],
  3718. | 20.00th=[ 828], 30.00th=[ 908], 40.00th=[ 1032],
  3719. | 50.00th=[ 1160], 60.00th=[ 1304], 70.00th=[ 19584],
  3720. | 80.00th=[ 987136], 90.00th=[ 7176192], 95.00th=[ 50069504],
  3721. | 99.00th=[233832448], 99.50th=[341835776], 99.90th=[574619648],
  3722. | 99.95th=[616562688], 99.99th=[742391808]
  3723. bw ( KiB/s): min= 0, max= 4258, per=11.66%, avg=3136.00, stdev=1586.75, samples=2
  3724. iops : min= 0, max= 66, avg=48.50, stdev=24.75, samples=2
  3725. write: IOPS=50, BW=3250KiB/s (3328kB/s)(381MiB/120005msec)
  3726. slat (nsec): min=0, max=878437k, avg=346407.87, stdev=11510788.20
  3727. clat (nsec): min=0, max=1737.6M, avg=69625248.52, stdev=133022418.53
  3728. lat (nsec): min=0, max=1737.7M, avg=69972917.80, stdev=134278546.34
  3729. clat percentiles (usec):
  3730. | 1.00th=[ 1045], 5.00th=[ 1287], 10.00th=[ 1516],
  3731. | 20.00th=[ 2057], 30.00th=[ 3097], 40.00th=[ 5145],
  3732. | 50.00th=[ 10683], 60.00th=[ 24773], 70.00th=[ 70779],
  3733. | 80.00th=[ 105382], 90.00th=[ 175113], 95.00th=[ 337642],
  3734. | 99.00th=[ 700449], 99.50th=[ 742392], 99.90th=[ 884999],
  3735. | 99.95th=[ 935330], 99.99th=[1736442]
  3736. bw ( KiB/s): min= 0, max= 4382, per=56.51%, avg=3248.00, stdev=1603.72, samples=2
  3737. iops : min= 0, max= 68, avg=50.50, stdev=24.75, samples=2
  3738. lat (nsec) : 750=5.64%, 1000=12.77%
  3739. lat (usec) : 2=15.11%, 4=0.10%, 10=0.12%, 20=0.71%, 50=0.65%
  3740. lat (usec) : 100=0.73%, 250=0.74%, 500=0.56%, 750=0.63%, 1000=1.97%
  3741. lat (msec) : 2=12.09%, 4=9.67%, 10=8.15%, 20=5.10%, 50=5.26%
  3742. lat (msec) : 100=7.80%, 250=8.05%, 500=2.76%, 750=1.18%, 1000=0.21%
  3743. cpu : usr=0.07%, sys=0.29%, ctx=14274, majf=0, minf=30
  3744. IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  3745. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  3746. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  3747. issued rwts: total=5882,6091,0,0 short=0,0,0,0 dropped=0,0,0,0
  3748. latency : target=0, window=0, percentile=100.00%, depth=4
  3749.  
  3750. Run status group 0 (all jobs):
  3751. READ: bw=26.3MiB/s (27.5MB/s), 3136KiB/s-23.2MiB/s (3212kB/s-24.3MB/s), io=3154MiB (3307MB), run=120005-120081msec
  3752. WRITE: bw=5748KiB/s (5886kB/s), 2501KiB/s-3250KiB/s (2561kB/s-3328kB/s), io=674MiB (707MB), run=120005-120100msec
  3753.  
  3754. Disk stats (read/write):
  3755. rbd60: ios=7813/11452, merge=0/1, ticks=560837/843308, in_queue=1394412, util=12.65%
  3756. <example-benchmark-fio-server-573-benchmark-29b45>
  3757. randwrite: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:34:59 2019
  3758. write: IOPS=44, BW=2846KiB/s (2915kB/s)(334MiB/120177msec)
  3759. slat (nsec): min=0, max=820038k, avg=481068.24, stdev=13479085.23
  3760. clat (nsec): min=0, max=1048.1M, avg=89624595.19, stdev=149912487.94
  3761. lat (nsec): min=0, max=1048.2M, avg=90106805.68, stdev=150298868.66
  3762. clat percentiles (usec):
  3763. | 1.00th=[ 988], 5.00th=[ 1205], 10.00th=[ 1385],
  3764. | 20.00th=[ 1893], 30.00th=[ 2868], 40.00th=[ 5735],
  3765. | 50.00th=[ 16581], 60.00th=[ 74974], 70.00th=[ 102237],
  3766. | 80.00th=[ 125305], 90.00th=[ 235930], 95.00th=[ 425722],
  3767. | 99.00th=[ 725615], 99.50th=[ 843056], 99.90th=[ 935330],
  3768. | 99.95th=[ 943719], 99.99th=[1044382]
  3769. bw ( KiB/s): min= 0, max= 3072, per=44.56%, avg=2848.50, stdev=316.08, samples=2
  3770. iops : min= 0, max= 48, avg=44.50, stdev= 4.95, samples=2
  3771. lat (usec) : 4=0.04%, 100=0.02%, 250=0.02%, 500=0.02%, 750=0.06%
  3772. lat (usec) : 1000=1.03%
  3773. lat (msec) : 2=20.57%, 4=13.91%, 10=9.88%, 20=5.93%, 50=4.98%
  3774. lat (msec) : 100=12.56%, 250=21.47%, 500=5.88%, 750=2.85%, 1000=0.80%
  3775. cpu : usr=0.06%, sys=0.19%, ctx=10059, majf=0, minf=30
  3776. IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  3777. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  3778. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  3779. issued rwts: total=0,5342,0,0 short=0,0,0,0 dropped=0,0,0,0
  3780. latency : target=0, window=0, percentile=100.00%, depth=4
  3781. read: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:34:59 2019
  3782. read: IOPS=326, BW=20.4MiB/s (21.4MB/s)(2449MiB/120025msec)
  3783. slat (nsec): min=0, max=690834k, avg=54024.69, stdev=4310215.39
  3784. clat (nsec): min=0, max=1444.3M, avg=12206634.97, stdev=63174388.43
  3785. lat (nsec): min=0, max=1444.4M, avg=12261505.28, stdev=63326547.19
  3786. clat percentiles (nsec):
  3787. | 1.00th=[ 708], 5.00th=[ 716], 10.00th=[ 716],
  3788. | 20.00th=[ 724], 30.00th=[ 732], 40.00th=[ 748],
  3789. | 50.00th=[ 828], 60.00th=[ 876], 70.00th=[ 948],
  3790. | 80.00th=[ 1304], 90.00th=[ 2670592], 95.00th=[ 56885248],
  3791. | 99.00th=[ 362807296], 99.50th=[ 484442112], 99.90th=[ 734003200],
  3792. | 99.95th=[ 809500672], 99.99th=[1010827264]
  3793. bw ( KiB/s): min= 0, max=30908, per=85.83%, avg=20894.50, stdev=14161.23, samples=2
  3794. iops : min= 0, max= 484, avg=327.00, stdev=222.03, samples=2
  3795. lat (nsec) : 750=40.44%, 1000=32.16%
  3796. lat (usec) : 2=9.98%, 4=0.05%, 10=0.85%, 20=0.80%, 50=0.18%
  3797. lat (usec) : 100=0.07%, 250=0.05%, 500=0.11%, 750=0.68%, 1000=1.57%
  3798. lat (msec) : 2=2.35%, 4=1.47%, 10=0.79%, 20=0.54%, 50=2.30%
  3799. lat (msec) : 100=2.60%, 250=1.44%, 500=1.10%, 750=0.38%, 1000=0.07%
  3800. cpu : usr=0.10%, sys=0.35%, ctx=12084, majf=0, minf=44
  3801. IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  3802. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  3803. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  3804. issued rwts: total=39241,0,0,0 short=0,0,0,0 dropped=0,0,0,0
  3805. latency : target=0, window=0, percentile=100.00%, depth=4
  3806. readwrite: (groupid=0, jobs=1): err= 0: pid=14: Wed May 8 20:34:59 2019
  3807. read: IOPS=53, BW=3453KiB/s (3536kB/s)(405MiB/120005msec)
  3808. slat (nsec): min=0, max=377941k, avg=145377.66, stdev=6367946.16
  3809. clat (nsec): min=0, max=824320k, avg=8948377.22, stdev=45281721.38
  3810. lat (nsec): min=0, max=824512k, avg=9094746.36, stdev=45750151.63
  3811. clat percentiles (nsec):
  3812. | 1.00th=[ 716], 5.00th=[ 732], 10.00th=[ 740],
  3813. | 20.00th=[ 820], 30.00th=[ 924], 40.00th=[ 1048],
  3814. | 50.00th=[ 1192], 60.00th=[ 1384], 70.00th=[ 67072],
  3815. | 80.00th=[ 1138688], 90.00th=[ 7700480], 95.00th=[ 47448064],
  3816. | 99.00th=[233832448], 99.50th=[379584512], 99.90th=[583008256],
  3817. | 99.95th=[599785472], 99.99th=[826277888]
  3818. bw ( KiB/s): min= 0, max= 5171, per=14.18%, avg=3452.50, stdev=2430.33, samples=2
  3819. iops : min= 0, max= 80, avg=53.50, stdev=37.48, samples=2
  3820. write: IOPS=55, BW=3551KiB/s (3637kB/s)(416MiB/120005msec)
  3821. slat (nsec): min=0, max=114884k, avg=206767.42, stdev=1440085.66
  3822. clat (nsec): min=0, max=978894k, avg=63265817.90, stdev=119018938.09
  3823. lat (nsec): min=0, max=979480k, avg=63473688.03, stdev=118978188.84
  3824. clat percentiles (usec):
  3825. | 1.00th=[ 1037], 5.00th=[ 1303], 10.00th=[ 1516], 20.00th=[ 1958],
  3826. | 30.00th=[ 2704], 40.00th=[ 4228], 50.00th=[ 7767], 60.00th=[ 19530],
  3827. | 70.00th=[ 69731], 80.00th=[103285], 90.00th=[156238], 95.00th=[304088],
  3828. | 99.00th=[624952], 99.50th=[708838], 99.90th=[843056], 99.95th=[918553],
  3829. | 99.99th=[977273]
  3830. bw ( KiB/s): min= 0, max= 5238, per=55.52%, avg=3549.00, stdev=2388.61, samples=2
  3831. iops : min= 0, max= 81, avg=55.00, stdev=36.77, samples=2
  3832. lat (nsec) : 750=5.93%, 1000=12.18%
  3833. lat (usec) : 2=14.23%, 4=0.05%, 10=0.21%, 20=0.74%, 50=0.85%
  3834. lat (usec) : 100=0.91%, 250=0.71%, 500=0.76%, 750=0.63%, 1000=1.97%
  3835. lat (msec) : 2=12.87%, 4=11.19%, 10=8.16%, 20=4.69%, 50=4.58%
  3836. lat (msec) : 100=7.81%, 250=7.90%, 500=2.55%, 750=0.96%, 1000=0.14%
  3837. cpu : usr=0.07%, sys=0.32%, ctx=15689, majf=0, minf=30
  3838. IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  3839. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  3840. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  3841. issued rwts: total=6475,6656,0,0 short=0,0,0,0 dropped=0,0,0,0
  3842. latency : target=0, window=0, percentile=100.00%, depth=4
  3843.  
  3844. Run status group 0 (all jobs):
  3845. READ: bw=23.8MiB/s (24.9MB/s), 3453KiB/s-20.4MiB/s (3536kB/s-21.4MB/s), io=2853MiB (2992MB), run=120005-120025msec
  3846. WRITE: bw=6393KiB/s (6546kB/s), 2846KiB/s-3551KiB/s (2915kB/s-3637kB/s), io=750MiB (787MB), run=120005-120177msec
  3847.  
  3848. Disk stats (read/write):
  3849. rbd256: ios=8297/12558, merge=0/2, ticks=571393/819813, in_queue=1380594, util=14.16%
  3850. <example-benchmark-fio-server-294-benchmark-tmh99>
  3851. randwrite: (groupid=0, jobs=1): err= 0: pid=11: Wed May 8 20:34:59 2019
  3852. write: IOPS=20, BW=1329KiB/s (1361kB/s)(157MiB/120857msec)
  3853. slat (nsec): min=0, max=200899k, avg=380636.09, stdev=5677598.13
  3854. clat (nsec): min=0, max=6808.5M, avg=192323685.98, stdev=404033626.27
  3855. lat (nsec): min=0, max=6808.8M, avg=192705644.37, stdev=404002602.14
  3856. clat percentiles (usec):
  3857. | 1.00th=[ 1450], 5.00th=[ 2147], 10.00th=[ 2671],
  3858. | 20.00th=[ 3720], 30.00th=[ 4948], 40.00th=[ 6915],
  3859. | 50.00th=[ 9765], 60.00th=[ 14091], 70.00th=[ 34866],
  3860. | 80.00th=[ 383779], 90.00th=[ 666895], 95.00th=[ 901776],
  3861. | 99.00th=[1451230], 99.50th=[1635779], 99.90th=[4395631],
  3862. | 99.95th=[6811550], 99.99th=[6811550]
  3863. bw ( KiB/s): min= 0, max= 1782, per=26.56%, avg=1334.00, stdev=633.57, samples=2
  3864. iops : min= 0, max= 27, avg=20.00, stdev= 9.90, samples=2
  3865. lat (usec) : 4=0.04%, 250=0.04%, 750=0.04%, 1000=0.12%
  3866. lat (msec) : 2=3.79%, 4=18.48%, 10=28.49%, 20=15.16%, 50=4.51%
  3867. lat (msec) : 100=0.52%, 250=3.87%, 500=8.34%, 750=8.86%, 1000=3.59%
  3868. cpu : usr=0.02%, sys=0.09%, ctx=4422, majf=0, minf=30
  3869. IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  3870. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  3871. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  3872. issued rwts: total=0,2506,0,0 short=0,0,0,0 dropped=0,0,0,0
  3873. latency : target=0, window=0, percentile=100.00%, depth=4
  3874. read: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:34:59 2019
  3875. read: IOPS=201, BW=12.6MiB/s (13.2MB/s)(1520MiB/120584msec)
  3876. slat (nsec): min=0, max=30825k, avg=33031.54, stdev=251563.33
  3877. clat (nsec): min=0, max=1843.3M, avg=19822397.80, stdev=118197969.25
  3878. lat (nsec): min=0, max=1843.3M, avg=19856547.54, stdev=118217208.93
  3879. clat percentiles (nsec):
  3880. | 1.00th=[ 716], 5.00th=[ 732], 10.00th=[ 756],
  3881. | 20.00th=[ 788], 30.00th=[ 828], 40.00th=[ 868],
  3882. | 50.00th=[ 900], 60.00th=[ 964], 70.00th=[ 1240],
  3883. | 80.00th=[ 11712], 90.00th=[ 2703360], 95.00th=[ 12648448],
  3884. | 99.00th=[ 742391808], 99.50th=[ 901775360], 99.90th=[1166016512],
  3885. | 99.95th=[1249902592], 99.99th=[1719664640]
  3886. bw ( KiB/s): min= 0, max=13908, per=78.34%, avg=12965.50, stdev=1332.90, samples=2
  3887. iops : min= 0, max= 217, avg=202.00, stdev=21.21, samples=2
  3888. lat (nsec) : 750=9.20%, 1000=53.40%
  3889. lat (usec) : 2=16.93%, 4=0.19%, 10=0.23%, 20=0.72%, 50=0.21%
  3890. lat (usec) : 100=0.15%, 250=0.12%, 500=0.08%, 750=0.22%, 1000=1.69%
  3891. lat (msec) : 2=5.01%, 4=3.90%, 10=2.58%, 20=1.08%, 50=0.98%
  3892. lat (msec) : 100=0.39%, 250=0.30%, 500=0.68%, 750=0.99%, 1000=0.72%
  3893. cpu : usr=0.07%, sys=0.27%, ctx=9250, majf=0, minf=30
  3894. IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  3895. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  3896. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  3897. issued rwts: total=24313,0,0,0 short=0,0,0,0 dropped=0,0,0,0
  3898. latency : target=0, window=0, percentile=100.00%, depth=4
  3899. readwrite: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:34:59 2019
  3900. read: IOPS=57, BW=3662KiB/s (3750kB/s)(429MiB/120007msec)
  3901. slat (nsec): min=0, max=7169.2k, avg=30253.59, stdev=126833.96
  3902. clat (nsec): min=0, max=745444k, avg=7485190.03, stdev=44225344.30
  3903. lat (nsec): min=0, max=745480k, avg=7516432.89, stdev=44236569.51
  3904. clat percentiles (nsec):
  3905. | 1.00th=[ 716], 5.00th=[ 724], 10.00th=[ 732],
  3906. | 20.00th=[ 764], 30.00th=[ 876], 40.00th=[ 980],
  3907. | 50.00th=[ 1144], 60.00th=[ 1288], 70.00th=[ 14656],
  3908. | 80.00th=[ 749568], 90.00th=[ 2867200], 95.00th=[ 33816576],
  3909. | 99.00th=[208666624], 99.50th=[408944640], 99.90th=[599785472],
  3910. | 99.95th=[624951296], 99.99th=[742391808]
  3911. bw ( KiB/s): min= 0, max= 5649, per=22.13%, avg=3662.00, stdev=2810.04, samples=2
  3912. iops : min= 0, max= 88, avg=57.00, stdev=43.84, samples=2
  3913. write: IOPS=58, BW=3720KiB/s (3809kB/s)(436MiB/120007msec)
  3914. slat (nsec): min=0, max=4886.1k, avg=176323.86, stdev=232612.07
  3915. clat (nsec): min=0, max=1093.2M, avg=61434190.62, stdev=127604648.95
  3916. lat (nsec): min=0, max=1093.5M, avg=61611749.80, stdev=127566168.49
  3917. clat percentiles (usec):
  3918. | 1.00th=[ 1029], 5.00th=[ 1303], 10.00th=[ 1483],
  3919. | 20.00th=[ 1827], 30.00th=[ 2311], 40.00th=[ 3228],
  3920. | 50.00th=[ 5080], 60.00th=[ 10814], 70.00th=[ 35390],
  3921. | 80.00th=[ 91751], 90.00th=[ 175113], 95.00th=[ 350225],
  3922. | 99.00th=[ 608175], 99.50th=[ 692061], 99.90th=[ 910164],
  3923. | 99.95th=[1027605], 99.99th=[1098908]
  3924. bw ( KiB/s): min= 0, max= 5666, per=74.02%, avg=3717.50, stdev=2755.60, samples=2
  3925. iops : min= 0, max= 88, avg=57.50, stdev=43.13, samples=2
  3926. lat (nsec) : 750=8.47%, 1000=12.07%
  3927. lat (usec) : 2=13.36%, 4=0.04%, 10=0.32%, 20=1.28%, 50=0.92%
  3928. lat (usec) : 100=0.85%, 250=1.39%, 500=0.57%, 750=0.47%, 1000=1.69%
  3929. lat (msec) : 2=14.34%, 4=12.44%, 10=7.88%, 20=4.06%, 50=3.80%
  3930. lat (msec) : 100=6.09%, 250=5.90%, 500=2.64%, 750=1.28%, 1000=0.12%
  3931. cpu : usr=0.08%, sys=0.32%, ctx=15155, majf=0, minf=31
  3932. IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  3933. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  3934. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  3935. issued rwts: total=6867,6972,0,0 short=0,0,0,0 dropped=0,0,0,0
  3936. latency : target=0, window=0, percentile=100.00%, depth=4
  3937.  
  3938. Run status group 0 (all jobs):
  3939. READ: bw=16.2MiB/s (16.9MB/s), 3662KiB/s-12.6MiB/s (3750kB/s-13.2MB/s), io=1949MiB (2044MB), run=120007-120584msec
  3940. WRITE: bw=5022KiB/s (5143kB/s), 1329KiB/s-3720KiB/s (1361kB/s-3809kB/s), io=593MiB (622MB), run=120007-120857msec
  3941.  
  3942. Disk stats (read/write):
  3943. rbd98: ios=6656/10952, merge=0/2, ticks=583320/912510, in_queue=1486887, util=5.81%
  3944. <example-benchmark-fio-server-977-benchmark-wgh5n>
  3945. randwrite: (groupid=0, jobs=1): err= 0: pid=11: Wed May 8 20:35:00 2019
  3946. write: IOPS=33, BW=2177KiB/s (2229kB/s)(256MiB/120220msec)
  3947. slat (nsec): min=0, max=1172.5M, avg=774123.61, stdev=23284904.99
  3948. clat (nsec): min=0, max=1437.9M, avg=117417416.74, stdev=205641999.08
  3949. lat (nsec): min=0, max=1437.9M, avg=118192524.39, stdev=206589593.23
  3950. clat percentiles (usec):
  3951. | 1.00th=[ 1004], 5.00th=[ 1237], 10.00th=[ 1434],
  3952. | 20.00th=[ 1942], 30.00th=[ 2868], 40.00th=[ 5342],
  3953. | 50.00th=[ 13173], 60.00th=[ 60031], 70.00th=[ 103285],
  3954. | 80.00th=[ 179307], 90.00th=[ 375391], 95.00th=[ 574620],
  3955. | 99.00th=[ 935330], 99.50th=[1052771], 99.90th=[1182794],
  3956. | 99.95th=[1333789], 99.99th=[1434452]
  3957. bw ( KiB/s): min= 0, max= 2643, per=42.38%, avg=2178.50, stdev=656.90, samples=2
  3958. iops : min= 0, max= 41, avg=33.50, stdev=10.61, samples=2
  3959. lat (usec) : 750=0.05%, 1000=0.86%
  3960. lat (msec) : 2=20.00%, 4=15.10%, 10=11.63%, 20=6.61%, 50=5.02%
  3961. lat (msec) : 100=9.64%, 250=15.47%, 500=9.08%, 750=3.65%, 1000=2.28%
  3962. cpu : usr=0.05%, sys=0.15%, ctx=7711, majf=0, minf=31
  3963. IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  3964. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  3965. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  3966. issued rwts: total=0,4086,0,0 short=0,0,0,0 dropped=0,0,0,0
  3967. latency : target=0, window=0, percentile=100.00%, depth=4
  3968. read: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:35:00 2019
  3969. read: IOPS=434, BW=27.1MiB/s (28.4MB/s)(3256MiB/120173msec)
  3970. slat (nsec): min=0, max=1168.0M, avg=69320.30, stdev=6851183.35
  3971. clat (nsec): min=0, max=1338.9M, avg=9166220.50, stdev=60043152.24
  3972. lat (nsec): min=0, max=1338.9M, avg=9236403.72, stdev=60436829.32
  3973. clat percentiles (nsec):
  3974. | 1.00th=[ 708], 5.00th=[ 708], 10.00th=[ 716],
  3975. | 20.00th=[ 724], 30.00th=[ 732], 40.00th=[ 772],
  3976. | 50.00th=[ 812], 60.00th=[ 860], 70.00th=[ 900],
  3977. | 80.00th=[ 1160], 90.00th=[ 1028096], 95.00th=[ 20054016],
  3978. | 99.00th=[ 308281344], 99.50th=[ 476053504], 99.90th=[ 775946240],
  3979. | 99.95th=[ 876609536], 99.99th=[1098907648]
  3980. bw ( KiB/s): min= 0, max=43740, per=90.78%, avg=27779.00, stdev=22572.26, samples=2
  3981. iops : min= 0, max= 684, avg=434.00, stdev=353.55, samples=2
  3982. lat (nsec) : 750=34.58%, 1000=41.49%
  3983. lat (usec) : 2=9.00%, 4=0.09%, 10=0.55%, 20=1.01%, 50=0.38%
  3984. lat (usec) : 100=0.11%, 250=0.08%, 500=0.40%, 750=0.68%, 1000=1.52%
  3985. lat (msec) : 2=2.37%, 4=1.52%, 10=0.76%, 20=0.48%, 50=1.41%
  3986. lat (msec) : 100=1.42%, 250=0.92%, 500=0.81%, 750=0.30%, 1000=0.11%
  3987. cpu : usr=0.13%, sys=0.46%, ctx=13251, majf=0, minf=38
  3988. IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  3989. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  3990. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  3991. issued rwts: total=52164,0,0,0 short=0,0,0,0 dropped=0,0,0,0
  3992. latency : target=0, window=0, percentile=100.00%, depth=4
  3993. readwrite: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:35:00 2019
  3994. read: IOPS=44, BW=2864KiB/s (2933kB/s)(336MiB/120003msec)
  3995. slat (nsec): min=0, max=816568k, avg=185923.53, stdev=11141851.34
  3996. clat (nsec): min=0, max=724201k, avg=8174567.07, stdev=43029992.72
  3997. lat (nsec): min=0, max=817041k, avg=8361466.82, stdev=44428754.62
  3998. clat percentiles (nsec):
  3999. | 1.00th=[ 716], 5.00th=[ 724], 10.00th=[ 740],
  4000. | 20.00th=[ 820], 30.00th=[ 900], 40.00th=[ 1032],
  4001. | 50.00th=[ 1176], 60.00th=[ 1352], 70.00th=[ 32640],
  4002. | 80.00th=[ 856064], 90.00th=[ 4620288], 95.00th=[ 39059456],
  4003. | 99.00th=[235929600], 99.50th=[320864256], 99.90th=[517996544],
  4004. | 99.95th=[633339904], 99.99th=[725614592]
  4005. bw ( KiB/s): min= 0, max= 4423, per=9.36%, avg=2863.50, stdev=2205.47, samples=2
  4006. iops : min= 0, max= 69, avg=44.50, stdev=34.65, samples=2
  4007. write: IOPS=46, BW=2970KiB/s (3041kB/s)(348MiB/120003msec)
  4008. slat (nsec): min=0, max=54255k, avg=183966.16, stdev=752724.15
  4009. clat (nsec): min=0, max=1040.4M, avg=78172935.18, stdev=141799380.38
  4010. lat (nsec): min=0, max=1040.4M, avg=78358082.12, stdev=141760902.55
  4011. clat percentiles (usec):
  4012. | 1.00th=[ 1004], 5.00th=[ 1237], 10.00th=[ 1483],
  4013. | 20.00th=[ 2089], 30.00th=[ 3064], 40.00th=[ 5735],
  4014. | 50.00th=[ 13829], 60.00th=[ 30278], 70.00th=[ 74974],
  4015. | 80.00th=[ 110625], 90.00th=[ 229639], 95.00th=[ 387974],
  4016. | 99.00th=[ 725615], 99.50th=[ 801113], 99.90th=[ 918553],
  4017. | 99.95th=[ 952108], 99.99th=[1044382]
  4018. bw ( KiB/s): min= 0, max= 4562, per=57.73%, avg=2967.50, stdev=2254.96, samples=2
  4019. iops : min= 0, max= 71, avg=46.00, stdev=35.36, samples=2
  4020. lat (nsec) : 750=5.63%, 1000=13.06%
  4021. lat (usec) : 2=14.03%, 4=0.06%, 10=0.30%, 20=0.90%, 50=0.85%
  4022. lat (usec) : 100=0.80%, 250=0.84%, 500=0.92%, 750=1.02%, 1000=2.10%
  4023. lat (msec) : 2=11.43%, 4=9.95%, 10=6.71%, 20=5.17%, 50=6.26%
  4024. lat (msec) : 100=7.18%, 250=7.72%, 500=3.57%, 750=1.09%, 1000=0.42%
  4025. cpu : usr=0.07%, sys=0.26%, ctx=12751, majf=0, minf=33
  4026. IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  4027. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  4028. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  4029. issued rwts: total=5371,5565,0,0 short=0,0,0,0 dropped=0,0,0,0
  4030. latency : target=0, window=0, percentile=100.00%, depth=4
  4031.  
  4032. Run status group 0 (all jobs):
  4033. READ: bw=29.9MiB/s (31.3MB/s), 2864KiB/s-27.1MiB/s (2933kB/s-28.4MB/s), io=3591MiB (3766MB), run=120003-120173msec
  4034. WRITE: bw=5141KiB/s (5264kB/s), 2177KiB/s-2970KiB/s (2229kB/s-3041kB/s), io=604MiB (633MB), run=120003-120220msec
  4035.  
  4036. Disk stats (read/write):
  4037. rbd324: ios=8914/10591, merge=0/1, ticks=549035/867403, in_queue=1406550, util=12.49%
  4038. <example-benchmark-fio-server-138-benchmark-9r9wf>
  4039. randwrite: (groupid=0, jobs=1): err= 0: pid=11: Wed May 8 20:35:00 2019
  4040. write: IOPS=46, BW=2976KiB/s (3047kB/s)(349MiB/120102msec)
  4041. slat (nsec): min=0, max=624921k, avg=477947.36, stdev=12874979.64
  4042. clat (nsec): min=0, max=1497.6M, avg=85883724.77, stdev=166458648.87
  4043. lat (nsec): min=0, max=1497.6M, avg=86362829.03, stdev=167067375.65
  4044. clat percentiles (usec):
  4045. | 1.00th=[ 1004], 5.00th=[ 1172], 10.00th=[ 1336],
  4046. | 20.00th=[ 1745], 30.00th=[ 2442], 40.00th=[ 3752],
  4047. | 50.00th=[ 7046], 60.00th=[ 27657], 70.00th=[ 88605],
  4048. | 80.00th=[ 113771], 90.00th=[ 242222], 95.00th=[ 476054],
  4049. | 99.00th=[ 759170], 99.50th=[ 943719], 99.90th=[1300235],
  4050. | 99.95th=[1468007], 99.99th=[1501561]
  4051. bw ( KiB/s): min= 0, max= 3030, per=42.42%, avg=2975.50, stdev=77.07, samples=2
  4052. iops : min= 0, max= 47, avg=46.00, stdev= 1.41, samples=2
  4053. lat (usec) : 4=0.02%, 500=0.02%, 750=0.02%, 1000=0.88%
  4054. lat (msec) : 2=23.29%, 4=17.02%, 10=12.20%, 20=5.36%, 50=3.10%
  4055. lat (msec) : 100=12.97%, 250=15.39%, 500=5.36%, 750=3.39%, 1000=0.75%
  4056. cpu : usr=0.05%, sys=0.21%, ctx=10541, majf=0, minf=29
  4057. IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  4058. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  4059. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  4060. issued rwts: total=0,5581,0,0 short=0,0,0,0 dropped=0,0,0,0
  4061. latency : target=0, window=0, percentile=100.00%, depth=4
  4062. read: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:35:00 2019
  4063. read: IOPS=377, BW=23.6MiB/s (24.7MB/s)(2830MiB/120105msec)
  4064. slat (nsec): min=0, max=473114k, avg=47081.33, stdev=3532967.60
  4065. clat (nsec): min=0, max=1438.1M, avg=10554888.20, stdev=60565769.03
  4066. lat (nsec): min=0, max=1438.2M, avg=10602804.25, stdev=60674181.23
  4067. clat percentiles (nsec):
  4068. | 1.00th=[ 708], 5.00th=[ 716], 10.00th=[ 716],
  4069. | 20.00th=[ 724], 30.00th=[ 732], 40.00th=[ 748],
  4070. | 50.00th=[ 820], 60.00th=[ 876], 70.00th=[ 932],
  4071. | 80.00th=[ 1352], 90.00th=[ 1843200], 95.00th=[ 50069504],
  4072. | 99.00th=[ 329252864], 99.50th=[ 476053504], 99.90th=[ 734003200],
  4073. | 99.95th=[ 868220928], 99.99th=[1199570944]
  4074. bw ( KiB/s): min= 0, max=35189, per=85.90%, avg=24145.50, stdev=15617.87, samples=2
  4075. iops : min= 0, max= 551, avg=377.50, stdev=245.37, samples=2
  4076. lat (nsec) : 750=40.49%, 1000=32.27%
  4077. lat (usec) : 2=8.76%, 4=0.06%, 10=1.83%, 20=0.98%, 50=0.34%
  4078. lat (usec) : 100=0.11%, 250=0.07%, 500=0.22%, 750=0.70%, 1000=1.89%
  4079. lat (msec) : 2=2.58%, 4=1.61%, 10=0.93%, 20=0.33%, 50=1.87%
  4080. lat (msec) : 100=2.63%, 250=1.04%, 500=0.88%, 750=0.35%, 1000=0.06%
  4081. cpu : usr=0.10%, sys=0.41%, ctx=13738, majf=0, minf=51
  4082. IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  4083. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  4084. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  4085. issued rwts: total=45385,0,0,0 short=0,0,0,0 dropped=0,0,0,0
  4086. latency : target=0, window=0, percentile=100.00%, depth=4
  4087. readwrite: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:35:00 2019
  4088. read: IOPS=62, BW=3986KiB/s (4082kB/s)(467MiB/120016msec)
  4089. slat (nsec): min=0, max=516982k, avg=103312.94, stdev=5980122.37
  4090. clat (nsec): min=0, max=999725k, avg=7809295.40, stdev=43750017.09
  4091. lat (nsec): min=0, max=999792k, avg=7913667.08, stdev=44149149.06
  4092. clat percentiles (nsec):
  4093. | 1.00th=[ 724], 5.00th=[ 732], 10.00th=[ 740],
  4094. | 20.00th=[ 820], 30.00th=[ 916], 40.00th=[ 1048],
  4095. | 50.00th=[ 1192], 60.00th=[ 1368], 70.00th=[ 78336],
  4096. | 80.00th=[ 1089536], 90.00th=[ 5144576], 95.00th=[ 39583744],
  4097. | 99.00th=[ 189792256], 99.50th=[ 329252864], 99.90th=[ 513802240],
  4098. | 99.95th=[ 817889280], 99.99th=[1002438656]
  4099. bw ( KiB/s): min= 0, max= 5655, per=14.18%, avg=3985.50, stdev=2361.03, samples=2
  4100. iops : min= 0, max= 88, avg=62.00, stdev=36.77, samples=2
  4101. write: IOPS=63, BW=4042KiB/s (4139kB/s)(474MiB/120016msec)
  4102. slat (nsec): min=0, max=633040k, avg=353080.90, stdev=9365055.55
  4103. clat (nsec): min=0, max=1236.7M, avg=55364701.88, stdev=121922412.63
  4104. lat (nsec): min=0, max=1236.7M, avg=55718902.59, stdev=122175546.16
  4105. clat percentiles (usec):
  4106. | 1.00th=[ 996], 5.00th=[ 1221], 10.00th=[ 1385],
  4107. | 20.00th=[ 1762], 30.00th=[ 2311], 40.00th=[ 3130],
  4108. | 50.00th=[ 5014], 60.00th=[ 11076], 70.00th=[ 28181],
  4109. | 80.00th=[ 85459], 90.00th=[ 133694], 95.00th=[ 299893],
  4110. | 99.00th=[ 650118], 99.50th=[ 725615], 99.90th=[ 918553],
  4111. | 99.95th=[ 977273], 99.99th=[1233126]
  4112. bw ( KiB/s): min= 0, max= 5676, per=57.60%, avg=4040.00, stdev=2313.65, samples=2
  4113. iops : min= 0, max= 88, avg=62.50, stdev=36.06, samples=2
  4114. lat (nsec) : 750=6.30%, 1000=11.97%
  4115. lat (usec) : 2=14.31%, 4=0.12%, 10=0.32%, 20=0.72%, 50=0.60%
  4116. lat (usec) : 100=0.66%, 250=0.88%, 500=0.72%, 750=0.75%, 1000=2.44%
  4117. lat (msec) : 2=15.07%, 4=12.30%, 10=7.92%, 20=4.76%, 50=4.62%
  4118. lat (msec) : 100=6.83%, 250=5.36%, 500=2.17%, 750=1.00%, 1000=0.17%
  4119. cpu : usr=0.10%, sys=0.35%, ctx=18104, majf=0, minf=36
  4120. IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  4121. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  4122. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  4123. issued rwts: total=7476,7576,0,0 short=0,0,0,0 dropped=0,0,0,0
  4124. latency : target=0, window=0, percentile=100.00%, depth=4
  4125.  
  4126. Run status group 0 (all jobs):
  4127. READ: bw=27.5MiB/s (28.8MB/s), 3986KiB/s-23.6MiB/s (4082kB/s-24.7MB/s), io=3297MiB (3457MB), run=120016-120105msec
  4128. WRITE: bw=7014KiB/s (7183kB/s), 2976KiB/s-4042KiB/s (3047kB/s-4139kB/s), io=823MiB (863MB), run=120016-120102msec
  4129.  
  4130. Disk stats (read/write):
  4131. rbd40: ios=9658/13474, merge=0/1, ticks=566891/831540, in_queue=1386680, util=15.13%
  4132. Jobs: 28 (f=28): [_(2),m(1)][99.6%][r=1044M,w=1019M][r=16.3k,w=15.9k IOPS][eta 00m:50s] <example-benchmark-fio-server-446-benchmark-2rtd4>
  4133. randwrite: (groupid=0, jobs=1): err= 0: pid=11: Wed May 8 20:35:00 2019
  4134. write: IOPS=14, BW=956KiB/s (979kB/s)(112MiB/120085msec)
  4135. slat (nsec): min=0, max=899001k, avg=734283.87, stdev=21245487.71
  4136. clat (nsec): min=0, max=3218.8M, avg=267401220.62, stdev=389066586.33
  4137. lat (nsec): min=0, max=3218.0M, avg=268136022.10, stdev=389320204.11
  4138. clat percentiles (usec):
  4139. | 1.00th=[ 1663], 5.00th=[ 2737], 10.00th=[ 3720],
  4140. | 20.00th=[ 5407], 30.00th=[ 8029], 40.00th=[ 12256],
  4141. | 50.00th=[ 20055], 60.00th=[ 84411], 70.00th=[ 383779],
  4142. | 80.00th=[ 650118], 90.00th=[ 826278], 95.00th=[ 985662],
  4143. | 99.00th=[1484784], 99.50th=[1635779], 99.90th=[3170894],
  4144. | 99.95th=[3204449], 99.99th=[3204449]
  4145. bw ( KiB/s): min= 0, max= 1412, per=25.82%, avg=1412.00, stdev= 0.00, samples=1
  4146. iops : min= 0, max= 22, avg=22.00, stdev= 0.00, samples=1
  4147. lat (usec) : 250=0.06%, 1000=0.06%
  4148. lat (msec) : 2=2.01%, 4=9.49%, 10=23.90%, 20=14.57%, 50=8.21%
  4149. lat (msec) : 100=2.51%, 250=5.36%, 500=7.82%, 750=13.01%, 1000=8.43%
  4150. cpu : usr=0.02%, sys=0.07%, ctx=3024, majf=0, minf=30
  4151. IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  4152. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  4153. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  4154. issued rwts: total=0,1791,0,0 short=0,0,0,0 dropped=0,0,0,0
  4155. latency : target=0, window=0, percentile=100.00%, depth=4
  4156. read: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:35:00 2019
  4157. read: IOPS=168, BW=10.5MiB/s (11.0MB/s)(1266MiB/120432msec)
  4158. slat (nsec): min=0, max=23034k, avg=26427.91, stdev=210110.57
  4159. clat (nsec): min=0, max=1849.9M, avg=23799251.42, stdev=130255782.34
  4160. lat (nsec): min=0, max=1849.0M, avg=23826708.01, stdev=130277592.76
  4161. clat percentiles (nsec):
  4162. | 1.00th=[ 716], 5.00th=[ 724], 10.00th=[ 740],
  4163. | 20.00th=[ 780], 30.00th=[ 796], 40.00th=[ 836],
  4164. | 50.00th=[ 868], 60.00th=[ 892], 70.00th=[ 956],
  4165. | 80.00th=[ 1256], 90.00th=[ 1482752], 95.00th=[ 20578304],
  4166. | 99.00th=[ 784334848], 99.50th=[ 926941184], 99.90th=[1182793728],
  4167. | 99.95th=[1333788672], 99.99th=[1619001344]
  4168. bw ( KiB/s): min= 0, max=11333, per=70.98%, avg=10797.00, stdev=758.02, samples=2
  4169. iops : min= 0, max= 177, avg=168.50, stdev=12.02, samples=2
  4170. lat (nsec) : 750=12.58%, 1000=60.81%
  4171. lat (usec) : 2=12.84%, 4=0.23%, 10=0.21%, 20=1.00%, 50=0.10%
  4172. lat (usec) : 100=0.08%, 250=0.07%, 500=0.04%, 750=0.26%, 1000=0.62%
  4173. lat (msec) : 2=2.02%, 4=1.88%, 10=1.45%, 20=0.80%, 50=1.01%
  4174. lat (msec) : 100=0.40%, 250=0.42%, 500=0.79%, 750=1.25%, 1000=0.82%
  4175. cpu : usr=0.06%, sys=0.19%, ctx=5020, majf=0, minf=29
  4176. IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  4177. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  4178. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  4179. issued rwts: total=20248,0,0,0 short=0,0,0,0 dropped=0,0,0,0
  4180. latency : target=0, window=0, percentile=100.00%, depth=4
  4181. readwrite: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:35:00 2019
  4182. read: IOPS=69, BW=4467KiB/s (4574kB/s)(524MiB/120003msec)
  4183. slat (nsec): min=0, max=8029.9k, avg=25453.88, stdev=121721.00
  4184. clat (nsec): min=0, max=668541k, avg=6380971.94, stdev=41066870.55
  4185. lat (nsec): min=0, max=668579k, avg=6407382.95, stdev=41072836.25
  4186. clat percentiles (nsec):
  4187. | 1.00th=[ 716], 5.00th=[ 724], 10.00th=[ 732],
  4188. | 20.00th=[ 756], 30.00th=[ 836], 40.00th=[ 924],
  4189. | 50.00th=[ 1064], 60.00th=[ 1224], 70.00th=[ 1512],
  4190. | 80.00th=[ 97792], 90.00th=[ 1433600], 95.00th=[ 11862016],
  4191. | 99.00th=[206569472], 99.50th=[341835776], 99.90th=[541065216],
  4192. | 99.95th=[599785472], 99.99th=[666894336]
  4193. bw ( KiB/s): min= 0, max= 7498, per=29.36%, avg=4466.50, stdev=4287.19, samples=2
  4194. iops : min= 0, max= 117, avg=69.50, stdev=67.18, samples=2
  4195. write: IOPS=70, BW=4515KiB/s (4623kB/s)(529MiB/120003msec)
  4196. slat (nsec): min=0, max=11441k, avg=189852.16, stdev=277675.89
  4197. clat (nsec): min=0, max=970672k, avg=50276865.16, stdev=111353429.62
  4198. lat (nsec): min=0, max=970761k, avg=50467942.12, stdev=111311973.94
  4199. clat percentiles (usec):
  4200. | 1.00th=[ 971], 5.00th=[ 1188], 10.00th=[ 1385], 20.00th=[ 1762],
  4201. | 30.00th=[ 2311], 40.00th=[ 3228], 50.00th=[ 4817], 60.00th=[ 9241],
  4202. | 70.00th=[ 20579], 80.00th=[ 68682], 90.00th=[139461], 95.00th=[287310],
  4203. | 99.00th=[574620], 99.50th=[675283], 99.90th=[834667], 99.95th=[918553],
  4204. | 99.99th=[968885]
  4205. bw ( KiB/s): min= 0, max= 7545, per=82.53%, avg=4512.50, stdev=4288.60, samples=2
  4206. iops : min= 0, max= 117, avg=70.00, stdev=66.47, samples=2
  4207. lat (nsec) : 750=9.66%, 1000=13.03%
  4208. lat (usec) : 2=13.06%, 4=0.08%, 10=0.62%, 20=1.50%, 50=0.99%
  4209. lat (usec) : 100=0.87%, 250=1.65%, 500=0.76%, 750=0.59%, 1000=1.67%
  4210. lat (msec) : 2=13.33%, 4=11.49%, 10=8.41%, 20=4.92%, 50=4.16%
  4211. lat (msec) : 100=4.61%, 250=5.36%, 500=2.45%, 750=0.65%, 1000=0.14%
  4212. cpu : usr=0.09%, sys=0.39%, ctx=17391, majf=0, minf=30
  4213. IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  4214. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  4215. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  4216. issued rwts: total=8376,8463,0,0 short=0,0,0,0 dropped=0,0,0,0
  4217. latency : target=0, window=0, percentile=100.00%, depth=4
  4218.  
  4219. Run status group 0 (all jobs):
  4220. READ: bw=14.9MiB/s (15.6MB/s), 4467KiB/s-10.5MiB/s (4574kB/s-11.0MB/s), io=1789MiB (1876MB), run=120003-120432msec
  4221. WRITE: bw=5468KiB/s (5599kB/s), 956KiB/s-4515KiB/s (979kB/s-4623kB/s), io=641MiB (672MB), run=120003-120085msec
  4222.  
  4223. Disk stats (read/write):
  4224. rbd126: ios=3914/10574, merge=0/2, ticks=578140/913384, in_queue=1484315, util=4.51%
  4225. <example-benchmark-fio-server-185-benchmark-lp9ct>
  4226. randwrite: (groupid=0, jobs=1): err= 0: pid=11: Wed May 8 20:35:00 2019
  4227. write: IOPS=39, BW=2553KiB/s (2614kB/s)(301MiB/120774msec)
  4228. slat (nsec): min=0, max=35777k, avg=174425.73, stdev=601217.79
  4229. clat (nsec): min=0, max=8251.2M, avg=100036644.27, stdev=335178831.09
  4230. lat (nsec): min=0, max=8251.3M, avg=100212323.58, stdev=335174890.69
  4231. clat percentiles (usec):
  4232. | 1.00th=[ 750], 5.00th=[ 1237], 10.00th=[ 1778],
  4233. | 20.00th=[ 3261], 30.00th=[ 5080], 40.00th=[ 7701],
  4234. | 50.00th=[ 11731], 60.00th=[ 19268], 70.00th=[ 40109],
  4235. | 80.00th=[ 139461], 90.00th=[ 278922], 95.00th=[ 459277],
  4236. | 99.00th=[ 759170], 99.50th=[1300235], 99.90th=[4395631],
  4237. | 99.95th=[8220836], 99.99th=[8220836]
  4238. bw ( KiB/s): min= 0, max= 3221, per=42.04%, avg=2567.00, stdev=924.90, samples=2
  4239. iops : min= 0, max= 50, avg=39.50, stdev=14.85, samples=2
  4240. lat (usec) : 4=0.02%, 50=0.02%, 500=0.06%, 750=0.91%, 1000=2.04%
  4241. lat (msec) : 2=8.64%, 4=13.08%, 10=22.28%, 20=13.46%, 50=11.71%
  4242. lat (msec) : 100=4.82%, 250=10.55%, 500=8.41%, 750=2.99%, 1000=0.27%
  4243. cpu : usr=0.04%, sys=0.14%, ctx=7681, majf=0, minf=30
  4244. IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  4245. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  4246. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  4247. issued rwts: total=0,4815,0,0 short=0,0,0,0 dropped=0,0,0,0
  4248. latency : target=0, window=0, percentile=100.00%, depth=4
  4249. read: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:35:00 2019
  4250. read: IOPS=68, BW=4361KiB/s (4465kB/s)(513MiB/120497msec)
  4251. slat (nsec): min=0, max=21635k, avg=53416.96, stdev=327483.41
  4252. clat (nsec): min=0, max=1829.5M, avg=58787431.59, stdev=199791627.08
  4253. lat (nsec): min=0, max=1829.5M, avg=58841997.19, stdev=199805850.51
  4254. clat percentiles (nsec):
  4255. | 1.00th=[ 716], 5.00th=[ 732], 10.00th=[ 772],
  4256. | 20.00th=[ 820], 30.00th=[ 884], 40.00th=[ 1004],
  4257. | 50.00th=[ 1240], 60.00th=[ 1528], 70.00th=[ 1318912],
  4258. | 80.00th=[ 3620864], 90.00th=[ 40108032], 95.00th=[ 608174080],
  4259. | 99.00th=[ 968884224], 99.50th=[1082130432], 99.90th=[1350565888],
  4260. | 99.95th=[1468006400], 99.99th=[1837105152]
  4261. bw ( KiB/s): min= 0, max= 4444, per=55.77%, avg=4375.50, stdev=96.87, samples=2
  4262. iops : min= 0, max= 69, avg=68.00, stdev= 1.41, samples=2
  4263. lat (nsec) : 750=7.03%, 1000=32.83%
  4264. lat (usec) : 2=22.27%, 4=0.26%, 10=0.12%, 20=0.39%, 50=0.15%
  4265. lat (usec) : 100=0.15%, 250=0.06%, 500=0.07%, 750=0.77%, 1000=2.62%
  4266. lat (msec) : 2=7.37%, 4=6.77%, 10=5.08%, 20=2.33%, 50=2.17%
  4267. lat (msec) : 100=0.73%, 250=0.88%, 500=1.93%, 750=3.19%, 1000=1.99%
  4268. cpu : usr=0.04%, sys=0.10%, ctx=6071, majf=0, minf=30
  4269. IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  4270. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  4271. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  4272. issued rwts: total=8207,0,0,0 short=0,0,0,0 dropped=0,0,0,0
  4273. latency : target=0, window=0, percentile=100.00%, depth=4
  4274. readwrite: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:35:00 2019
  4275. read: IOPS=54, BW=3500KiB/s (3584kB/s)(410MiB/120005msec)
  4276. slat (nsec): min=0, max=9183.3k, avg=39914.76, stdev=142373.27
  4277. clat (nsec): min=0, max=749534k, avg=18005844.75, stdev=76443612.62
  4278. lat (nsec): min=0, max=749561k, avg=18046837.72, stdev=76451706.73
  4279. clat percentiles (nsec):
  4280. | 1.00th=[ 716], 5.00th=[ 732], 10.00th=[ 740],
  4281. | 20.00th=[ 852], 30.00th=[ 996], 40.00th=[ 1160],
  4282. | 50.00th=[ 1304], 60.00th=[ 35072], 70.00th=[ 995328],
  4283. | 80.00th=[ 2441216], 90.00th=[ 29491200], 95.00th=[ 71827456],
  4284. | 99.00th=[463470592], 99.50th=[608174080], 99.90th=[683671552],
  4285. | 99.95th=[725614592], 99.99th=[750780416]
  4286. bw ( KiB/s): min= 0, max= 5429, per=44.60%, avg=3499.00, stdev=2729.43, samples=2
  4287. iops : min= 0, max= 84, avg=54.00, stdev=42.43, samples=2
  4288. write: IOPS=55, BW=3576KiB/s (3662kB/s)(419MiB/120005msec)
  4289. slat (nsec): min=0, max=5692.4k, avg=182624.16, stdev=237777.68
  4290. clat (nsec): min=0, max=1091.5M, avg=53980894.32, stdev=112363439.06
  4291. lat (nsec): min=0, max=1091.7M, avg=54164773.18, stdev=112319425.21
  4292. clat percentiles (usec):
  4293. | 1.00th=[ 1037], 5.00th=[ 1237], 10.00th=[ 1418],
  4294. | 20.00th=[ 1762], 30.00th=[ 2212], 40.00th=[ 3130],
  4295. | 50.00th=[ 4752], 60.00th=[ 10159], 70.00th=[ 27132],
  4296. | 80.00th=[ 78119], 90.00th=[ 162530], 95.00th=[ 304088],
  4297. | 99.00th=[ 541066], 99.50th=[ 599786], 99.90th=[ 792724],
  4298. | 99.95th=[ 817890], 99.99th=[1098908]
  4299. bw ( KiB/s): min= 0, max= 5484, per=58.55%, avg=3575.00, stdev=2699.73, samples=2
  4300. iops : min= 0, max= 85, avg=55.50, stdev=41.72, samples=2
  4301. lat (nsec) : 750=5.61%, 1000=9.29%
  4302. lat (usec) : 2=13.91%, 4=0.05%, 10=0.22%, 20=0.48%, 50=0.32%
  4303. lat (usec) : 100=0.38%, 250=0.49%, 500=0.34%, 750=1.14%, 1000=2.80%
  4304. lat (msec) : 2=16.52%, 4=13.11%, 10=9.01%, 20=4.52%, 50=5.64%
  4305. lat (msec) : 100=6.54%, 250=4.89%, 500=3.69%, 750=1.02%, 1000=0.05%
  4306. cpu : usr=0.08%, sys=0.31%, ctx=17110, majf=0, minf=31
  4307. IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  4308. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  4309. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  4310. issued rwts: total=6561,6704,0,0 short=0,0,0,0 dropped=0,0,0,0
  4311. latency : target=0, window=0, percentile=100.00%, depth=4
  4312.  
  4313. Run status group 0 (all jobs):
  4314. READ: bw=7846KiB/s (8035kB/s), 3500KiB/s-4361KiB/s (3584kB/s-4465kB/s), io=923MiB (968MB), run=120005-120497msec
  4315. WRITE: bw=6106KiB/s (6253kB/s), 2553KiB/s-3576KiB/s (2614kB/s-3662kB/s), io=720MiB (755MB), run=120005-120774msec
  4316.  
  4317. Disk stats (read/write):
  4318. rbd73: ios=5816/13073, merge=0/3, ticks=653844/876526, in_queue=1520869, util=4.37%
  4319. <example-benchmark-fio-server-637-benchmark-4fhxd>
  4320. <example-benchmark-fio-server-567-benchmark-tkk6d>
  4321. <example-benchmark-fio-server-155-benchmark-scltw>
  4322. randwrite: (groupid=0, jobs=1): err= 0: pid=11: Wed May 8 20:35:00 2019
  4323. write: IOPS=47, BW=3031KiB/s (3104kB/s)(355MiB/120027msec)
  4324. slat (nsec): min=0, max=713321k, avg=471599.79, stdev=12371015.09
  4325. clat (nsec): min=0, max=1162.4M, avg=84194593.79, stdev=147118350.80
  4326. lat (nsec): min=0, max=1297.7M, avg=84667236.21, stdev=147923628.18
  4327. clat percentiles (usec):
  4328. | 1.00th=[ 963], 5.00th=[ 1123], 10.00th=[ 1287],
  4329. | 20.00th=[ 1680], 30.00th=[ 2409], 40.00th=[ 4424],
  4330. | 50.00th=[ 14222], 60.00th=[ 53216], 70.00th=[ 91751],
  4331. | 80.00th=[ 119014], 90.00th=[ 235930], 95.00th=[ 425722],
  4332. | 99.00th=[ 742392], 99.50th=[ 834667], 99.90th=[ 943719],
  4333. | 99.95th=[ 985662], 99.99th=[1166017]
  4334. bw ( KiB/s): min= 0, max= 1897, per=27.66%, avg=1897.00, stdev= 0.00, samples=1
  4335. iops : min= 0, max= 29, avg=29.00, stdev= 0.00, samples=1
  4336. lat (usec) : 500=0.04%, 750=0.09%, 1000=1.67%
  4337. lat (msec) : 2=23.76%, 4=13.08%, 10=8.50%, 20=4.84%, 50=7.22%
  4338. lat (msec) : 100=14.15%, 250=17.35%, 500=6.04%, 750=2.38%, 1000=0.92%
  4339. cpu : usr=0.06%, sys=0.20%, ctx=10812, majf=0, minf=28
  4340. IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  4341. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  4342. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  4343. issued rwts: total=0,5682,0,0 short=0,0,0,0 dropped=0,0,0,0
  4344. latency : target=0, window=0, percentile=100.00%, depth=4
  4345. read: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:35:00 2019
  4346. read: IOPS=331, BW=20.7MiB/s (21.7MB/s)(2485MiB/120014msec)
  4347. slat (nsec): min=0, max=714213k, avg=59830.16, stdev=4581236.52
  4348. clat (nsec): min=0, max=1125.4M, avg=12024678.97, stdev=58728270.03
  4349. lat (nsec): min=0, max=1125.4M, avg=12085365.90, stdev=58924449.76
  4350. clat percentiles (nsec):
  4351. | 1.00th=[ 708], 5.00th=[ 716], 10.00th=[ 724],
  4352. | 20.00th=[ 732], 30.00th=[ 748], 40.00th=[ 820],
  4353. | 50.00th=[ 860], 60.00th=[ 932], 70.00th=[ 1128],
  4354. | 80.00th=[ 309248], 90.00th=[ 9633792], 95.00th=[ 55836672],
  4355. | 99.00th=[341835776], 99.50th=[467664896], 99.90th=[675282944],
  4356. | 99.95th=[742391808], 99.99th=[968884224]
  4357. bw ( KiB/s): min= 0, max=28049, per=84.99%, avg=21206.50, stdev=9676.76, samples=2
  4358. iops : min= 0, max= 438, avg=331.00, stdev=151.32, samples=2
  4359. lat (nsec) : 750=30.11%, 1000=35.20%
  4360. lat (usec) : 2=13.03%, 4=0.06%, 10=0.82%, 20=0.40%, 50=0.14%
  4361. lat (usec) : 100=0.07%, 250=0.07%, 500=0.35%, 750=1.07%, 1000=2.45%
  4362. lat (msec) : 2=3.18%, 4=1.99%, 10=1.08%, 20=0.75%, 50=3.52%
  4363. lat (msec) : 100=2.76%, 250=1.48%, 500=1.06%, 750=0.34%, 1000=0.04%
  4364. cpu : usr=0.12%, sys=0.38%, ctx=15922, majf=0, minf=37
  4365. IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  4366. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  4367. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  4368. issued rwts: total=39785,0,0,0 short=0,0,0,0 dropped=0,0,0,0
  4369. latency : target=0, window=0, percentile=100.00%, depth=4
  4370. readwrite: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:35:00 2019
  4371. read: IOPS=58, BW=3746KiB/s (3836kB/s)(439MiB/120005msec)
  4372. slat (nsec): min=0, max=9850.8k, avg=37565.76, stdev=156835.35
  4373. clat (nsec): min=0, max=680660k, avg=6520405.48, stdev=33535024.37
  4374. lat (nsec): min=0, max=680698k, avg=6558922.65, stdev=33543170.85
  4375. clat percentiles (nsec):
  4376. | 1.00th=[ 716], 5.00th=[ 732], 10.00th=[ 740],
  4377. | 20.00th=[ 820], 30.00th=[ 924], 40.00th=[ 1064],
  4378. | 50.00th=[ 1208], 60.00th=[ 1416], 70.00th=[ 142336],
  4379. | 80.00th=[ 1073152], 90.00th=[ 5013504], 95.00th=[ 34865152],
  4380. | 99.00th=[168820736], 99.50th=[258998272], 99.90th=[455081984],
  4381. | 99.95th=[505413632], 99.99th=[683671552]
  4382. bw ( KiB/s): min= 0, max= 5781, per=15.01%, avg=3745.50, stdev=2878.63, samples=2
  4383. iops : min= 0, max= 90, avg=58.00, stdev=45.25, samples=2
  4384. write: IOPS=59, BW=3828KiB/s (3920kB/s)(449MiB/120005msec)
  4385. slat (nsec): min=0, max=340981k, avg=298619.27, stdev=5238520.33
  4386. clat (nsec): min=0, max=1050.3M, avg=60301551.27, stdev=126652467.05
  4387. lat (nsec): min=0, max=1050.4M, avg=60601269.70, stdev=126682799.90
  4388. clat percentiles (usec):
  4389. | 1.00th=[ 1004], 5.00th=[ 1221], 10.00th=[ 1401],
  4390. | 20.00th=[ 1778], 30.00th=[ 2343], 40.00th=[ 3425],
  4391. | 50.00th=[ 5669], 60.00th=[ 11731], 70.00th=[ 39060],
  4392. | 80.00th=[ 92799], 90.00th=[ 158335], 95.00th=[ 341836],
  4393. | 99.00th=[ 633340], 99.50th=[ 775947], 99.90th=[ 943719],
  4394. | 99.95th=[ 943719], 99.99th=[1052771]
  4395. bw ( KiB/s): min= 0, max= 5822, per=55.78%, avg=3825.50, stdev=2823.48, samples=2
  4396. iops : min= 0, max= 90, avg=59.00, stdev=43.84, samples=2
  4397. lat (nsec) : 750=5.86%, 1000=11.94%
  4398. lat (usec) : 2=13.77%, 4=0.07%, 10=0.26%, 20=0.95%, 50=0.75%
  4399. lat (usec) : 100=0.71%, 250=0.82%, 500=1.11%, 750=1.04%, 1000=2.37%
  4400. lat (msec) : 2=15.03%, 4=11.20%, 10=9.35%, 20=3.85%, 50=5.03%
  4401. lat (msec) : 100=5.85%, 250=6.41%, 500=2.51%, 750=0.85%, 1000=0.30%
  4402. cpu : usr=0.09%, sys=0.32%, ctx=17167, majf=0, minf=35
  4403. IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  4404. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  4405. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  4406. issued rwts: total=7025,7175,0,0 short=0,0,0,0 dropped=0,0,0,0
  4407. latency : target=0, window=0, percentile=100.00%, depth=4
  4408.  
  4409. Run status group 0 (all jobs):
  4410. READ: bw=24.4MiB/s (25.6MB/s), 3746KiB/s-20.7MiB/s (3836kB/s-21.7MB/s), io=2924MiB (3066MB), run=120005-120014msec
  4411. WRITE: bw=6859KiB/s (7023kB/s), 3031KiB/s-3828KiB/s (3104kB/s-3920kB/s), io=804MiB (843MB), run=120005-120027msec
  4412.  
  4413. Disk stats (read/write):
  4414. rbd236: ios=11231/13877, merge=0/1, ticks=555717/840618, in_queue=1383488, util=16.97%
  4415. randwrite: (groupid=0, jobs=1): err= 0: pid=11: Wed May 8 20:35:00 2019
  4416. write: IOPS=15, BW=1000KiB/s (1024kB/s)(118MiB/120408msec)
  4417. slat (nsec): min=0, max=980064k, avg=1663581.52, stdev=35294572.35
  4418. clat (nsec): min=0, max=2257.3M, avg=255484057.39, stdev=465350314.36
  4419. lat (nsec): min=0, max=2257.8M, avg=257147165.96, stdev=465938509.66
  4420. clat percentiles (usec):
  4421. | 1.00th=[ 1237], 5.00th=[ 1500], 10.00th=[ 1844],
  4422. | 20.00th=[ 2737], 30.00th=[ 3884], 40.00th=[ 6128],
  4423. | 50.00th=[ 10421], 60.00th=[ 21627], 70.00th=[ 49546],
  4424. | 80.00th=[ 675283], 90.00th=[1010828], 95.00th=[1367344],
  4425. | 99.00th=[1669333], 99.50th=[1887437], 99.90th=[2164261],
  4426. | 99.95th=[2264925], 99.99th=[2264925]
  4427. bw ( KiB/s): min= 0, max= 1497, per=21.63%, avg=1000.50, stdev=702.16, samples=2
  4428. iops : min= 0, max= 23, avg=15.00, stdev=11.31, samples=2
  4429. lat (usec) : 50=0.05%, 250=0.05%
  4430. lat (msec) : 2=12.30%, 4=18.48%, 10=18.74%, 20=9.11%, 50=11.77%
  4431. lat (msec) : 100=5.27%, 250=0.53%, 500=0.75%, 750=4.79%, 1000=7.72%
  4432. cpu : usr=0.03%, sys=0.07%, ctx=3491, majf=0, minf=30
  4433. IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  4434. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  4435. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  4436. issued rwts: total=0,1878,0,0 short=0,0,0,0 dropped=0,0,0,0
  4437. latency : target=0, window=0, percentile=100.00%, depth=4
  4438. read: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:35:00 2019
  4439. read: IOPS=627, BW=39.0MiB/s (40.9MB/s)(4684MiB/120072msec)
  4440. slat (nsec): min=0, max=979374k, avg=48936.40, stdev=5532284.90
  4441. clat (nsec): min=0, max=2121.5M, avg=6347450.41, stdev=70775025.09
  4442. lat (nsec): min=0, max=2121.5M, avg=6397358.95, stdev=71004998.05
  4443. clat percentiles (nsec):
  4444. | 1.00th=[ 708], 5.00th=[ 716], 10.00th=[ 724],
  4445. | 20.00th=[ 764], 30.00th=[ 780], 40.00th=[ 788],
  4446. | 50.00th=[ 828], 60.00th=[ 860], 70.00th=[ 884],
  4447. | 80.00th=[ 924], 90.00th=[ 8768], 95.00th=[ 12736],
  4448. | 99.00th=[ 76021760], 99.50th=[ 624951296], 99.90th=[1082130432],
  4449. | 99.95th=[1266679808], 99.99th=[1635778560]
  4450. bw ( KiB/s): min= 0, max=59778, per=91.83%, avg=39953.00, stdev=28036.78, samples=2
  4451. iops : min= 0, max= 939, avg=627.00, stdev=441.23, samples=2
  4452. lat (nsec) : 750=17.24%, 1000=67.21%
  4453. lat (usec) : 2=4.63%, 4=0.13%, 10=1.42%, 20=5.52%, 50=0.52%
  4454. lat (usec) : 100=0.15%, 250=0.10%, 500=0.08%, 750=0.08%, 1000=0.17%
  4455. lat (msec) : 2=0.38%, 4=0.34%, 10=0.30%, 20=0.26%, 50=0.36%
  4456. lat (msec) : 100=0.18%, 250=0.20%, 500=0.15%, 750=0.30%, 1000=0.18%
  4457. cpu : usr=0.15%, sys=0.59%, ctx=5019, majf=0, minf=40
  4458. IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  4459. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  4460. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  4461. issued rwts: total=75286,0,0,0 short=0,0,0,0 dropped=0,0,0,0
  4462. latency : target=0, window=0, percentile=100.00%, depth=4
  4463. readwrite: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:35:00 2019
  4464. read: IOPS=55, BW=3563KiB/s (3649kB/s)(418MiB/120002msec)
  4465. slat (nsec): min=0, max=633777k, avg=116582.49, stdev=7753908.85
  4466. clat (nsec): min=0, max=613214k, avg=6368642.28, stdev=41422627.12
  4467. lat (nsec): min=0, max=633913k, avg=6486171.40, stdev=42134167.47
  4468. clat percentiles (nsec):
  4469. | 1.00th=[ 716], 5.00th=[ 724], 10.00th=[ 732],
  4470. | 20.00th=[ 748], 30.00th=[ 828], 40.00th=[ 916],
  4471. | 50.00th=[ 1064], 60.00th=[ 1224], 70.00th=[ 1592],
  4472. | 80.00th=[ 73216], 90.00th=[ 1335296], 95.00th=[ 10289152],
  4473. | 99.00th=[206569472], 99.50th=[350224384], 99.90th=[526385152],
  4474. | 99.95th=[583008256], 99.99th=[616562688]
  4475. bw ( KiB/s): min= 0, max= 5756, per=8.19%, avg=3562.50, stdev=3102.08, samples=2
  4476. iops : min= 0, max= 89, avg=55.00, stdev=48.08, samples=2
  4477. write: IOPS=56, BW=3638KiB/s (3726kB/s)(426MiB/120002msec)
  4478. slat (nsec): min=0, max=16871k, avg=187455.60, stdev=310103.99
  4479. clat (nsec): min=0, max=1166.3M, avg=64088245.30, stdev=127371794.56
  4480. lat (nsec): min=0, max=1166.4M, avg=64276809.91, stdev=127323659.95
  4481. clat percentiles (usec):
  4482. | 1.00th=[ 1045], 5.00th=[ 1319], 10.00th=[ 1532],
  4483. | 20.00th=[ 2008], 30.00th=[ 2704], 40.00th=[ 3752],
  4484. | 50.00th=[ 6390], 60.00th=[ 16188], 70.00th=[ 59507],
  4485. | 80.00th=[ 101188], 90.00th=[ 173016], 95.00th=[ 316670],
  4486. | 99.00th=[ 666895], 99.50th=[ 792724], 99.90th=[ 918553],
  4487. | 99.95th=[ 952108], 99.99th=[1166017]
  4488. bw ( KiB/s): min= 0, max= 5860, per=78.63%, avg=3636.50, stdev=3144.50, samples=2
  4489. iops : min= 0, max= 91, avg=56.50, stdev=48.79, samples=2
  4490. lat (nsec) : 750=10.52%, 1000=12.57%
  4491. lat (usec) : 2=11.90%, 4=0.07%, 10=0.64%, 20=1.83%, 50=1.19%
  4492. lat (usec) : 100=1.39%, 250=1.77%, 500=1.04%, 750=0.42%, 1000=1.01%
  4493. lat (msec) : 2=10.89%, 4=11.91%, 10=7.83%, 20=3.60%, 50=4.21%
  4494. lat (msec) : 100=6.04%, 250=7.55%, 500=2.39%, 750=0.92%, 1000=0.30%
  4495. cpu : usr=0.07%, sys=0.30%, ctx=13525, majf=0, minf=31
  4496. IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  4497. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  4498. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  4499. issued rwts: total=6681,6819,0,0 short=0,0,0,0 dropped=0,0,0,0
  4500. latency : target=0, window=0, percentile=100.00%, depth=4
  4501.  
  4502. Run status group 0 (all jobs):
  4503. READ: bw=42.5MiB/s (44.6MB/s), 3563KiB/s-39.0MiB/s (3649kB/s-40.9MB/s), io=5102MiB (5349MB), run=120002-120072msec
  4504. WRITE: bw=4626KiB/s (4737kB/s), 1000KiB/s-3638KiB/s (1024kB/s-3726kB/s), io=544MiB (570MB), run=120002-120408msec
  4505.  
  4506. Disk stats (read/write):
  4507. rbd245: ios=3467/8915, merge=0/1, ticks=551293/914532, in_queue=1459579, util=5.67%
  4508. <example-benchmark-fio-server-737-benchmark-pfzcw>
  4509. randwrite: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:35:00 2019
  4510. write: IOPS=34, BW=2211KiB/s (2264kB/s)(262MiB/121542msec)
  4511. slat (nsec): min=0, max=279070k, avg=313105.26, stdev=4584630.48
  4512. clat (nsec): min=0, max=9623.8M, avg=120787623.04, stdev=322490503.56
  4513. lat (nsec): min=0, max=9623.8M, avg=121102071.47, stdev=322508076.74
  4514. clat percentiles (usec):
  4515. | 1.00th=[ 1074], 5.00th=[ 1696], 10.00th=[ 2343],
  4516. | 20.00th=[ 3458], 30.00th=[ 4883], 40.00th=[ 6718],
  4517. | 50.00th=[ 10028], 60.00th=[ 16712], 70.00th=[ 57410],
  4518. | 80.00th=[ 229639], 90.00th=[ 400557], 95.00th=[ 566232],
  4519. | 99.00th=[ 893387], 99.50th=[ 935330], 99.90th=[2231370],
  4520. | 99.95th=[8556381], 99.99th=[9596568]
  4521. bw ( KiB/s): min= 0, max= 2521, per=39.07%, avg=2236.50, stdev=402.34, samples=2
  4522. iops : min= 0, max= 39, avg=34.50, stdev= 6.36, samples=2
  4523. lat (usec) : 250=0.02%, 500=0.02%, 750=0.33%, 1000=0.43%
  4524. lat (msec) : 2=6.48%, 4=16.85%, 10=25.81%, 20=12.82%, 50=6.34%
  4525. lat (msec) : 100=3.57%, 250=9.32%, 500=11.42%, 750=4.43%, 1000=1.74%
  4526. cpu : usr=0.03%, sys=0.15%, ctx=7087, majf=0, minf=31
  4527. IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  4528. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  4529. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  4530. issued rwts: total=0,4196,0,0 short=0,0,0,0 dropped=0,0,0,0
  4531. latency : target=0, window=0, percentile=100.00%, depth=4
  4532. read: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:35:00 2019
  4533. read: IOPS=98, BW=6322KiB/s (6473kB/s)(744MiB/120507msec)
  4534. slat (nsec): min=0, max=13961k, avg=41369.64, stdev=206220.35
  4535. clat (nsec): min=0, max=1884.5M, avg=40508976.77, stdev=167911107.34
  4536. lat (nsec): min=0, max=1884.5M, avg=40551550.07, stdev=167924859.19
  4537. clat percentiles (nsec):
  4538. | 1.00th=[ 716], 5.00th=[ 724], 10.00th=[ 764],
  4539. | 20.00th=[ 780], 30.00th=[ 828], 40.00th=[ 868],
  4540. | 50.00th=[ 924], 60.00th=[ 1192], 70.00th=[ 1496],
  4541. | 80.00th=[ 1564672], 90.00th=[ 10551296], 95.00th=[ 354418688],
  4542. | 99.00th=[ 901775360], 99.50th=[1002438656], 99.90th=[1333788672],
  4543. | 99.95th=[1535115264], 99.99th=[1837105152]
  4544. bw ( KiB/s): min= 0, max= 6847, per=64.89%, avg=6344.50, stdev=710.64, samples=2
  4545. iops : min= 0, max= 106, avg=98.50, stdev=10.61, samples=2
  4546. lat (nsec) : 750=8.55%, 1000=44.78%
  4547. lat (usec) : 2=18.65%, 4=0.16%, 10=0.05%, 20=0.62%, 50=0.21%
  4548. lat (usec) : 100=0.18%, 250=0.10%, 500=0.14%, 750=0.44%, 1000=2.25%
  4549. lat (msec) : 2=5.77%, 4=4.82%, 10=3.20%, 20=1.59%, 50=1.57%
  4550. lat (msec) : 100=0.82%, 250=0.71%, 500=1.29%, 750=2.08%, 1000=1.56%
  4551. cpu : usr=0.04%, sys=0.15%, ctx=6384, majf=0, minf=31
  4552. IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  4553. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  4554. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  4555. issued rwts: total=11900,0,0,0 short=0,0,0,0 dropped=0,0,0,0
  4556. latency : target=0, window=0, percentile=100.00%, depth=4
  4557. readwrite: (groupid=0, jobs=1): err= 0: pid=14: Wed May 8 20:35:00 2019
  4558. read: IOPS=54, BW=3471KiB/s (3555kB/s)(407MiB/120007msec)
  4559. slat (nsec): min=0, max=2714.2k, avg=33168.77, stdev=89049.83
  4560. clat (nsec): min=0, max=1276.5M, avg=9409156.58, stdev=48225664.37
  4561. lat (nsec): min=0, max=1276.6M, avg=9443278.32, stdev=48232813.61
  4562. clat percentiles (nsec):
  4563. | 1.00th=[ 716], 5.00th=[ 724], 10.00th=[ 732],
  4564. | 20.00th=[ 772], 30.00th=[ 900], 40.00th=[ 1032],
  4565. | 50.00th=[ 1176], 60.00th=[ 1368], 70.00th=[ 88576],
  4566. | 80.00th=[ 1122304], 90.00th=[ 5931008], 95.00th=[ 46923776],
  4567. | 99.00th=[ 242221056], 99.50th=[ 383778816], 99.90th=[ 583008256],
  4568. | 99.95th=[ 608174080], 99.99th=[1283457024]
  4569. bw ( KiB/s): min= 0, max= 5833, per=35.49%, avg=3470.50, stdev=3341.08, samples=2
  4570. iops : min= 0, max= 91, avg=54.00, stdev=52.33, samples=2
  4571. write: IOPS=55, BW=3559KiB/s (3644kB/s)(417MiB/120007msec)
  4572. slat (nsec): min=0, max=8103.8k, avg=181515.33, stdev=260812.54
  4573. clat (nsec): min=0, max=1065.1M, avg=62753756.72, stdev=126482037.44
  4574. lat (nsec): min=0, max=1065.2M, avg=62936479.59, stdev=126442634.22
  4575. clat percentiles (usec):
  4576. | 1.00th=[ 996], 5.00th=[ 1221], 10.00th=[ 1401],
  4577. | 20.00th=[ 1811], 30.00th=[ 2409], 40.00th=[ 3458],
  4578. | 50.00th=[ 5473], 60.00th=[ 13435], 70.00th=[ 43254],
  4579. | 80.00th=[ 90702], 90.00th=[ 164627], 95.00th=[ 387974],
  4580. | 99.00th=[ 599786], 99.50th=[ 675283], 99.90th=[ 843056],
  4581. | 99.95th=[ 910164], 99.99th=[1061159]
  4582. bw ( KiB/s): min= 0, max= 5939, per=62.15%, avg=3557.50, stdev=3367.95, samples=2
  4583. iops : min= 0, max= 92, avg=55.00, stdev=52.33, samples=2
  4584. lat (nsec) : 750=8.15%, 1000=10.53%
  4585. lat (usec) : 2=13.47%, 4=0.02%, 10=0.25%, 20=0.86%, 50=0.61%
  4586. lat (usec) : 100=0.84%, 250=0.90%, 500=0.42%, 750=0.88%, 1000=2.41%
  4587. lat (msec) : 2=14.72%, 4=11.69%, 10=8.41%, 20=3.60%, 50=5.27%
  4588. lat (msec) : 100=6.93%, 250=5.47%, 500=3.38%, 750=1.08%, 1000=0.10%
  4589. cpu : usr=0.08%, sys=0.30%, ctx=15450, majf=0, minf=33
  4590. IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  4591. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  4592. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  4593. issued rwts: total=6509,6670,0,0 short=0,0,0,0 dropped=0,0,0,0
  4594. latency : target=0, window=0, percentile=100.00%, depth=4
  4595.  
  4596. Run status group 0 (all jobs):
  4597. READ: bw=9778KiB/s (10.0MB/s), 3471KiB/s-6322KiB/s (3555kB/s-6473kB/s), io=1151MiB (1207MB), run=120007-120507msec
  4598. WRITE: bw=5725KiB/s (5862kB/s), 2211KiB/s-3559KiB/s (2264kB/s-3644kB/s), io=680MiB (713MB), run=120007-121542msec
  4599.  
  4600. Disk stats (read/write):
  4601. rbd53: ios=5395/11625, merge=0/3, ticks=602058/947991, in_queue=1541538, util=4.56%
  4602. <example-benchmark-fio-server-653-benchmark-6cpgt>
  4603. randwrite: (groupid=0, jobs=1): err= 0: pid=11: Wed May 8 20:35:00 2019
  4604. write: IOPS=34, BW=2218KiB/s (2271kB/s)(260MiB/120080msec)
  4605. slat (nsec): min=0, max=1186.4M, avg=1481401.31, stdev=33991422.05
  4606. clat (nsec): min=0, max=2099.0M, avg=114488164.29, stdev=238384745.73
  4607. lat (nsec): min=0, max=2099.4M, avg=115970057.69, stdev=240216513.94
  4608. clat percentiles (usec):
  4609. | 1.00th=[ 1037], 5.00th=[ 1336], 10.00th=[ 1614],
  4610. | 20.00th=[ 2245], 30.00th=[ 3130], 40.00th=[ 4424],
  4611. | 50.00th=[ 7504], 60.00th=[ 18744], 70.00th=[ 54789],
  4612. | 80.00th=[ 125305], 90.00th=[ 463471], 95.00th=[ 675283],
  4613. | 99.00th=[1035994], 99.50th=[1149240], 99.90th=[1786774],
  4614. | 99.95th=[1870660], 99.99th=[2105541]
  4615. bw ( KiB/s): min= 0, max= 2634, per=39.17%, avg=2216.50, stdev=590.43, samples=2
  4616. iops : min= 0, max= 41, avg=34.50, stdev= 9.19, samples=2
  4617. lat (usec) : 750=0.10%, 1000=0.53%
  4618. lat (msec) : 2=15.54%, 4=20.92%, 10=16.50%, 20=7.36%, 50=8.49%
  4619. lat (msec) : 100=7.24%, 250=9.16%, 500=5.39%, 750=4.88%, 1000=2.77%
  4620. cpu : usr=0.04%, sys=0.15%, ctx=7482, majf=0, minf=31
  4621. IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  4622. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  4623. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  4624. issued rwts: total=0,4158,0,0 short=0,0,0,0 dropped=0,0,0,0
  4625. latency : target=0, window=0, percentile=100.00%, depth=4
  4626. read: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:35:00 2019
  4627. read: IOPS=519, BW=32.2MiB/s (33.8MB/s)(3869MiB/120140msec)
  4628. slat (nsec): min=0, max=1032.2M, avg=66123.48, stdev=6143379.66
  4629. clat (nsec): min=0, max=2075.3M, avg=7681293.84, stdev=65084937.34
  4630. lat (nsec): min=0, max=2075.4M, avg=7748304.75, stdev=65378311.95
  4631. clat percentiles (nsec):
  4632. | 1.00th=[ 716], 5.00th=[ 724], 10.00th=[ 724],
  4633. | 20.00th=[ 740], 30.00th=[ 780], 40.00th=[ 796],
  4634. | 50.00th=[ 844], 60.00th=[ 876], 70.00th=[ 916],
  4635. | 80.00th=[ 1240], 90.00th=[ 16768], 95.00th=[ 1417216],
  4636. | 99.00th=[ 304087040], 99.50th=[ 522190848], 99.90th=[ 893386752],
  4637. | 99.95th=[1027604480], 99.99th=[1317011456]
  4638. bw ( KiB/s): min= 0, max=52272, per=90.92%, avg=33012.50, stdev=27237.05, samples=2
  4639. iops : min= 0, max= 824, avg=519.00, stdev=431.34, samples=2
  4640. lat (nsec) : 750=23.18%, 1000=52.62%
  4641. lat (usec) : 2=6.72%, 4=0.13%, 10=1.52%, 20=7.48%, 50=1.36%
  4642. lat (usec) : 100=0.16%, 250=0.08%, 500=0.06%, 750=0.22%, 1000=0.74%
  4643. lat (msec) : 2=1.30%, 4=0.82%, 10=0.41%, 20=0.25%, 50=0.73%
  4644. lat (msec) : 100=0.75%, 250=0.36%, 500=0.55%, 750=0.34%, 1000=0.16%
  4645. cpu : usr=0.11%, sys=0.53%, ctx=8615, majf=0, minf=30
  4646. IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  4647. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  4648. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  4649. issued rwts: total=62370,0,0,0 short=0,0,0,0 dropped=0,0,0,0
  4650. latency : target=0, window=0, percentile=100.00%, depth=4
  4651. readwrite: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:35:00 2019
  4652. read: IOPS=52, BW=3337KiB/s (3417kB/s)(391MiB/120013msec)
  4653. slat (nsec): min=0, max=599202k, avg=127308.18, stdev=7575305.82
  4654. clat (nsec): min=0, max=668739k, avg=7431754.00, stdev=41464138.35
  4655. lat (nsec): min=0, max=668798k, avg=7559998.10, stdev=42139523.63
  4656. clat percentiles (nsec):
  4657. | 1.00th=[ 716], 5.00th=[ 724], 10.00th=[ 732],
  4658. | 20.00th=[ 780], 30.00th=[ 884], 40.00th=[ 1004],
  4659. | 50.00th=[ 1160], 60.00th=[ 1320], 70.00th=[ 22912],
  4660. | 80.00th=[ 888832], 90.00th=[ 3194880], 95.00th=[ 35913728],
  4661. | 99.00th=[204472320], 99.50th=[329252864], 99.90th=[530579456],
  4662. | 99.95th=[583008256], 99.99th=[666894336]
  4663. bw ( KiB/s): min= 0, max= 5328, per=9.19%, avg=3337.00, stdev=2815.70, samples=2
  4664. iops : min= 0, max= 83, avg=52.00, stdev=43.84, samples=2
  4665. write: IOPS=53, BW=3443KiB/s (3526kB/s)(404MiB/120013msec)
  4666. slat (nsec): min=0, max=6146.0k, avg=181618.18, stdev=222082.45
  4667. clat (nsec): min=0, max=1423.9M, avg=67042871.01, stdev=133485106.00
  4668. lat (nsec): min=0, max=1423.9M, avg=67225673.93, stdev=133436561.24
  4669. clat percentiles (usec):
  4670. | 1.00th=[ 1029], 5.00th=[ 1237], 10.00th=[ 1385],
  4671. | 20.00th=[ 1745], 30.00th=[ 2245], 40.00th=[ 3097],
  4672. | 50.00th=[ 4948], 60.00th=[ 12518], 70.00th=[ 41681],
  4673. | 80.00th=[ 98042], 90.00th=[ 214959], 95.00th=[ 379585],
  4674. | 99.00th=[ 641729], 99.50th=[ 717226], 99.90th=[ 809501],
  4675. | 99.95th=[ 817890], 99.99th=[1417675]
  4676. bw ( KiB/s): min= 0, max= 5473, per=60.81%, avg=3441.50, stdev=2872.97, samples=2
  4677. iops : min= 0, max= 85, avg=53.50, stdev=44.55, samples=2
  4678. lat (nsec) : 750=7.70%, 1000=11.79%
  4679. lat (usec) : 2=13.52%, 4=0.05%, 10=0.30%, 20=0.98%, 50=0.92%
  4680. lat (usec) : 100=0.77%, 250=1.10%, 500=0.69%, 750=0.68%, 1000=2.14%
  4681. lat (msec) : 2=15.40%, 4=12.12%, 10=6.92%, 20=3.92%, 50=4.73%
  4682. lat (msec) : 100=5.37%, 250=6.36%, 500=3.18%, 750=1.25%, 1000=0.13%
  4683. cpu : usr=0.07%, sys=0.30%, ctx=14601, majf=0, minf=34
  4684. IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  4685. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  4686. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  4687. issued rwts: total=6257,6454,0,0 short=0,0,0,0 dropped=0,0,0,0
  4688. latency : target=0, window=0, percentile=100.00%, depth=4
  4689.  
  4690. Run status group 0 (all jobs):
  4691. READ: bw=35.5MiB/s (37.2MB/s), 3337KiB/s-32.2MiB/s (3417kB/s-33.8MB/s), io=4260MiB (4467MB), run=120013-120140msec
  4692. WRITE: bw=5659KiB/s (5795kB/s), 2218KiB/s-3443KiB/s (2271kB/s-3526kB/s), io=664MiB (696MB), run=120013-120080msec
  4693.  
  4694. Disk stats (read/write):
  4695. rbd213: ios=5674/10923, merge=0/3, ticks=532365/860881, in_queue=1384865, util=8.80%
  4696. <example-benchmark-fio-server-666-benchmark-vhp22>
  4697. <example-benchmark-fio-server-674-benchmark-5pb2m>
  4698. randwrite: (groupid=0, jobs=1): err= 0: pid=11: Wed May 8 20:35:00 2019
  4699. write: IOPS=41, BW=2641KiB/s (2704kB/s)(310MiB/120145msec)
  4700. slat (nsec): min=0, max=575779k, avg=600805.43, stdev=14798680.70
  4701. clat (nsec): min=0, max=1310.3M, avg=96705119.71, stdev=167414675.40
  4702. lat (nsec): min=0, max=1338.3M, avg=97306982.18, stdev=168344306.33
  4703. clat percentiles (usec):
  4704. | 1.00th=[ 971], 5.00th=[ 1188], 10.00th=[ 1401],
  4705. | 20.00th=[ 1926], 30.00th=[ 2835], 40.00th=[ 5342],
  4706. | 50.00th=[ 19268], 60.00th=[ 78119], 70.00th=[ 102237],
  4707. | 80.00th=[ 127402], 90.00th=[ 267387], 95.00th=[ 463471],
  4708. | 99.00th=[ 834667], 99.50th=[ 943719], 99.90th=[1149240],
  4709. | 99.95th=[1300235], 99.99th=[1317012]
  4710. bw ( KiB/s): min= 0, max= 2877, per=42.41%, avg=2641.50, stdev=333.05, samples=2
  4711. iops : min= 0, max= 44, avg=40.50, stdev= 4.95, samples=2
  4712. lat (usec) : 250=0.02%, 1000=1.31%
  4713. lat (msec) : 2=19.74%, 4=14.77%, 10=9.97%, 20=4.56%, 50=4.38%
  4714. lat (msec) : 100=14.57%, 250=20.14%, 500=6.16%, 750=2.83%, 1000=1.25%
  4715. cpu : usr=0.06%, sys=0.17%, ctx=9207, majf=0, minf=30
  4716. IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  4717. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  4718. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  4719. issued rwts: total=0,4955,0,0 short=0,0,0,0 dropped=0,0,0,0
  4720. latency : target=0, window=0, percentile=100.00%, depth=4
  4721. read: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:35:00 2019
  4722. read: IOPS=486, BW=30.3MiB/s (31.8MB/s)(3637MiB/120064msec)
  4723. slat (nsec): min=0, max=878989k, avg=51948.57, stdev=4719575.63
  4724. clat (nsec): min=0, max=1394.2M, avg=8192226.33, stdev=53775826.59
  4725. lat (nsec): min=0, max=1394.2M, avg=8245018.59, stdev=53987800.95
  4726. clat percentiles (nsec):
  4727. | 1.00th=[ 708], 5.00th=[ 716], 10.00th=[ 716],
  4728. | 20.00th=[ 724], 30.00th=[ 748], 40.00th=[ 788],
  4729. | 50.00th=[ 828], 60.00th=[ 868], 70.00th=[ 924],
  4730. | 80.00th=[ 1192], 90.00th=[ 757760], 95.00th=[ 23199744],
  4731. | 99.00th=[267386880], 99.50th=[442499072], 99.90th=[717225984],
  4732. | 99.95th=[784334848], 99.99th=[952107008]
  4733. bw ( KiB/s): min= 0, max=50771, per=89.87%, avg=31032.50, stdev=27914.45, samples=2
  4734. iops : min= 0, max= 796, avg=486.00, stdev=438.41, samples=2
  4735. lat (nsec) : 750=30.36%, 1000=45.16%
  4736. lat (usec) : 2=8.76%, 4=0.08%, 10=1.60%, 20=3.14%, 50=0.19%
  4737. lat (usec) : 100=0.13%, 250=0.06%, 500=0.09%, 750=0.39%, 1000=1.05%
  4738. lat (msec) : 2=1.69%, 4=1.16%, 10=0.62%, 20=0.42%, 50=1.52%
  4739. lat (msec) : 100=1.69%, 250=0.83%, 500=0.72%, 750=0.28%, 1000=0.06%
  4740. cpu : usr=0.13%, sys=0.51%, ctx=12296, majf=0, minf=105
  4741. IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  4742. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  4743. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  4744. issued rwts: total=58366,0,0,0 short=0,0,0,0 dropped=0,0,0,0
  4745. latency : target=0, window=0, percentile=100.00%, depth=4
  4746. readwrite: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:35:00 2019
  4747. read: IOPS=54, BW=3515KiB/s (3600kB/s)(412MiB/120011msec)
  4748. slat (nsec): min=0, max=3688.4k, avg=35129.82, stdev=104218.83
  4749. clat (nsec): min=0, max=681349k, avg=8268770.25, stdev=41687470.94
  4750. lat (nsec): min=0, max=681373k, avg=8304838.00, stdev=41694110.97
  4751. clat percentiles (nsec):
  4752. | 1.00th=[ 724], 5.00th=[ 732], 10.00th=[ 748],
  4753. | 20.00th=[ 828], 30.00th=[ 924], 40.00th=[ 1064],
  4754. | 50.00th=[ 1208], 60.00th=[ 1384], 70.00th=[ 64256],
  4755. | 80.00th=[ 1089536], 90.00th=[ 5734400], 95.00th=[ 43778048],
  4756. | 99.00th=[214958080], 99.50th=[333447168], 99.90th=[509607936],
  4757. | 99.95th=[608174080], 99.99th=[683671552]
  4758. bw ( KiB/s): min= 0, max= 5158, per=10.18%, avg=3515.50, stdev=2322.85, samples=2
  4759. iops : min= 0, max= 80, avg=54.50, stdev=36.06, samples=2
  4760. write: IOPS=56, BW=3593KiB/s (3679kB/s)(421MiB/120011msec)
  4761. slat (nsec): min=0, max=566283k, avg=315230.86, stdev=8140260.33
  4762. clat (nsec): min=0, max=1079.2M, avg=62967107.48, stdev=118454148.51
  4763. lat (nsec): min=0, max=1079.3M, avg=63283412.71, stdev=118888184.71
  4764. clat percentiles (usec):
  4765. | 1.00th=[ 1004], 5.00th=[ 1254], 10.00th=[ 1467],
  4766. | 20.00th=[ 1909], 30.00th=[ 2704], 40.00th=[ 4293],
  4767. | 50.00th=[ 8455], 60.00th=[ 22414], 70.00th=[ 65274],
  4768. | 80.00th=[ 102237], 90.00th=[ 158335], 95.00th=[ 304088],
  4769. | 99.00th=[ 591397], 99.50th=[ 750781], 99.90th=[ 935330],
  4770. | 99.95th=[ 952108], 99.99th=[1082131]
  4771. bw ( KiB/s): min= 0, max= 5207, per=57.64%, avg=3590.50, stdev=2286.08, samples=2
  4772. iops : min= 0, max= 81, avg=55.50, stdev=36.06, samples=2
  4773. lat (nsec) : 750=4.97%, 1000=12.73%
  4774. lat (usec) : 2=14.70%, 4=0.03%, 10=0.33%, 20=0.82%, 50=0.73%
  4775. lat (usec) : 100=0.86%, 250=1.03%, 500=0.58%, 750=0.76%, 1000=2.14%
  4776. lat (msec) : 2=13.36%, 4=10.56%, 10=7.76%, 20=4.22%, 50=5.60%
  4777. lat (msec) : 100=7.45%, 250=7.89%, 500=2.56%, 750=0.71%, 1000=0.24%
  4778. cpu : usr=0.07%, sys=0.33%, ctx=15735, majf=0, minf=33
  4779. IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  4780. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  4781. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  4782. issued rwts: total=6592,6734,0,0 short=0,0,0,0 dropped=0,0,0,0
  4783. latency : target=0, window=0, percentile=100.00%, depth=4
  4784.  
  4785. Run status group 0 (all jobs):
  4786. READ: bw=33.7MiB/s (35.4MB/s), 3515KiB/s-30.3MiB/s (3600kB/s-31.8MB/s), io=4049MiB (4246MB), run=120011-120064msec
  4787. WRITE: bw=6230KiB/s (6379kB/s), 2641KiB/s-3593KiB/s (2704kB/s-3679kB/s), io=731MiB (766MB), run=120011-120145msec
  4788.  
  4789. Disk stats (read/write):
  4790. rbd173: ios=8283/12086, merge=0/1, ticks=553856/811994, in_queue=1355561, util=13.18%
  4791. randwrite: (groupid=0, jobs=1): err= 0: pid=11: Wed May 8 20:35:00 2019
  4792. write: IOPS=29, BW=1877KiB/s (1922kB/s)(222MiB/120954msec)
  4793. slat (nsec): min=0, max=1178.4M, avg=1070070.99, stdev=26922015.16
  4794. clat (nsec): min=0, max=1811.8M, avg=135023864.34, stdev=288341801.46
  4795. lat (nsec): min=0, max=1811.9M, avg=136095701.53, stdev=289170559.85
  4796. clat percentiles (usec):
  4797. | 1.00th=[ 1029], 5.00th=[ 1287], 10.00th=[ 1532],
  4798. | 20.00th=[ 2024], 30.00th=[ 2704], 40.00th=[ 3851],
  4799. | 50.00th=[ 5997], 60.00th=[ 13435], 70.00th=[ 38536],
  4800. | 80.00th=[ 127402], 90.00th=[ 591397], 95.00th=[ 876610],
  4801. | 99.00th=[1249903], 99.50th=[1317012], 99.90th=[1635779],
  4802. | 99.95th=[1669333], 99.99th=[1803551]
  4803. bw ( KiB/s): min= 0, max= 2954, per=33.73%, avg=1889.50, stdev=1505.43, samples=2
  4804. iops : min= 0, max= 46, avg=29.00, stdev=24.04, samples=2
  4805. lat (usec) : 750=0.06%, 1000=0.73%
  4806. lat (msec) : 2=18.96%, 4=21.24%, 10=16.19%, 20=7.14%, 50=7.02%
  4807. lat (msec) : 100=4.51%, 250=9.37%, 500=3.02%, 750=4.91%, 1000=3.92%
  4808. cpu : usr=0.04%, sys=0.13%, ctx=6342, majf=0, minf=30
  4809. IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  4810. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  4811. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  4812. issued rwts: total=0,3545,0,0 short=0,0,0,0 dropped=0,0,0,0
  4813. latency : target=0, window=0, percentile=100.00%, depth=4
  4814. read: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:35:00 2019
  4815. read: IOPS=517, BW=32.2MiB/s (33.8MB/s)(3876MiB/120265msec)
  4816. slat (nsec): min=0, max=1009.8M, avg=58501.48, stdev=5796659.72
  4817. clat (nsec): min=0, max=2075.2M, avg=7667833.39, stdev=70093238.96
  4818. lat (nsec): min=0, max=2075.3M, avg=7727229.62, stdev=70340481.38
  4819. clat percentiles (nsec):
  4820. | 1.00th=[ 716], 5.00th=[ 716], 10.00th=[ 724],
  4821. | 20.00th=[ 740], 30.00th=[ 772], 40.00th=[ 788],
  4822. | 50.00th=[ 804], 60.00th=[ 852], 70.00th=[ 884],
  4823. | 80.00th=[ 932], 90.00th=[ 8768], 95.00th=[ 790528],
  4824. | 99.00th=[ 375390208], 99.50th=[ 624951296], 99.90th=[ 943718400],
  4825. | 99.95th=[1044381696], 99.99th=[1300234240]
  4826. bw ( KiB/s): min= 0, max=51751, per=89.94%, avg=32982.50, stdev=26542.67, samples=2
  4827. iops : min= 0, max= 812, avg=517.00, stdev=417.19, samples=2
  4828. lat (nsec) : 750=24.97%, 1000=58.54%
  4829. lat (usec) : 2=6.06%, 4=0.17%, 10=0.98%, 20=3.16%, 50=0.53%
  4830. lat (usec) : 100=0.11%, 250=0.08%, 500=0.08%, 750=0.25%, 1000=0.54%
  4831. lat (msec) : 2=0.92%, 4=0.67%, 10=0.47%, 20=0.24%, 50=0.56%
  4832. lat (msec) : 100=0.48%, 250=0.13%, 500=0.38%, 750=0.41%, 1000=0.23%
  4833. cpu : usr=0.13%, sys=0.49%, ctx=7065, majf=0, minf=36
  4834. IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  4835. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  4836. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  4837. issued rwts: total=62257,0,0,0 short=0,0,0,0 dropped=0,0,0,0
  4838. latency : target=0, window=0, percentile=100.00%, depth=4
  4839. readwrite: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:35:00 2019
  4840. read: IOPS=57, BW=3678KiB/s (3767kB/s)(431MiB/120039msec)
  4841. slat (nsec): min=0, max=753868k, avg=277793.85, stdev=12125320.64
  4842. clat (nsec): min=0, max=806006k, avg=7819411.97, stdev=47157480.87
  4843. lat (nsec): min=0, max=806066k, avg=8098127.39, stdev=48659920.24
  4844. clat percentiles (nsec):
  4845. | 1.00th=[ 716], 5.00th=[ 724], 10.00th=[ 732],
  4846. | 20.00th=[ 764], 30.00th=[ 876], 40.00th=[ 996],
  4847. | 50.00th=[ 1144], 60.00th=[ 1320], 70.00th=[ 18560],
  4848. | 80.00th=[ 749568], 90.00th=[ 2637824], 95.00th=[ 26869760],
  4849. | 99.00th=[265289728], 99.50th=[387973120], 99.90th=[624951296],
  4850. | 99.95th=[675282944], 99.99th=[809500672]
  4851. bw ( KiB/s): min= 0, max= 5620, per=10.03%, avg=3679.00, stdev=2744.99, samples=2
  4852. iops : min= 0, max= 87, avg=57.00, stdev=42.43, samples=2
  4853. write: IOPS=58, BW=3753KiB/s (3844kB/s)(440MiB/120039msec)
  4854. slat (nsec): min=0, max=4929.1k, avg=192875.32, stdev=240267.68
  4855. clat (nsec): min=0, max=1279.8M, avg=60267779.32, stdev=129578981.16
  4856. lat (nsec): min=0, max=1279.9M, avg=60461861.12, stdev=129540203.48
  4857. clat percentiles (usec):
  4858. | 1.00th=[ 1037], 5.00th=[ 1287], 10.00th=[ 1483],
  4859. | 20.00th=[ 1893], 30.00th=[ 2442], 40.00th=[ 3392],
  4860. | 50.00th=[ 5407], 60.00th=[ 11207], 70.00th=[ 30016],
  4861. | 80.00th=[ 85459], 90.00th=[ 170918], 95.00th=[ 346031],
  4862. | 99.00th=[ 666895], 99.50th=[ 784335], 99.90th=[ 935330],
  4863. | 99.95th=[ 968885], 99.99th=[1283458]
  4864. bw ( KiB/s): min= 0, max= 5680, per=66.99%, avg=3752.50, stdev=2725.90, samples=2
  4865. iops : min= 0, max= 88, avg=58.00, stdev=42.43, samples=2
  4866. lat (nsec) : 750=8.39%, 1000=11.65%
  4867. lat (usec) : 2=13.20%, 4=0.04%, 10=0.39%, 20=1.22%, 50=0.88%
  4868. lat (usec) : 100=0.91%, 250=1.23%, 500=0.78%, 750=0.98%, 1000=1.87%
  4869. lat (msec) : 2=13.50%, 4=12.72%, 10=8.19%, 20=4.71%, 50=4.54%
  4870. lat (msec) : 100=5.34%, 250=5.01%, 500=3.30%, 750=0.83%, 1000=0.32%
  4871. cpu : usr=0.08%, sys=0.33%, ctx=15516, majf=0, minf=31
  4872. IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  4873. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  4874. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  4875. issued rwts: total=6899,7037,0,0 short=0,0,0,0 dropped=0,0,0,0
  4876. latency : target=0, window=0, percentile=100.00%, depth=4
  4877.  
  4878. Run status group 0 (all jobs):
  4879. READ: bw=35.8MiB/s (37.6MB/s), 3678KiB/s-32.2MiB/s (3767kB/s-33.8MB/s), io=4307MiB (4516MB), run=120039-120265msec
  4880. WRITE: bw=5602KiB/s (5737kB/s), 1877KiB/s-3753KiB/s (1922kB/s-3844kB/s), io=662MiB (694MB), run=120039-120954msec
  4881.  
  4882. Disk stats (read/write):
  4883. rbd205: ios=5496/10983, merge=0/1, ticks=553117/872605, in_queue=1417429, util=8.18%
  4884. randwrite: (groupid=0, jobs=1): err= 0: pid=11: Wed May 8 20:35:00 2019
  4885. write: IOPS=22, BW=1415KiB/s (1449kB/s)(167MiB/120791msec)
  4886. slat (nsec): min=0, max=560200k, avg=579809.47, stdev=13166429.06
  4887. clat (nsec): min=0, max=3879.5M, avg=180201402.17, stdev=290186331.72
  4888. lat (nsec): min=0, max=3879.7M, avg=180784362.43, stdev=290262817.96
  4889. clat percentiles (usec):
  4890. | 1.00th=[ 1565], 5.00th=[ 2573], 10.00th=[ 3425],
  4891. | 20.00th=[ 5407], 30.00th=[ 8586], 40.00th=[ 13829],
  4892. | 50.00th=[ 22152], 60.00th=[ 80217], 70.00th=[ 214959],
  4893. | 80.00th=[ 371196], 90.00th=[ 583009], 95.00th=[ 708838],
  4894. | 99.00th=[ 985662], 99.50th=[1266680], 99.90th=[3875537],
  4895. | 99.95th=[3875537], 99.99th=[3875537]
  4896. bw ( KiB/s): min= 0, max= 2022, per=28.42%, avg=1421.00, stdev=849.94, samples=2
  4897. iops : min= 0, max= 31, avg=21.50, stdev=13.44, samples=2
  4898. lat (usec) : 4=0.07%, 250=0.04%, 500=0.04%, 1000=0.04%
  4899. lat (msec) : 2=2.77%, 4=10.09%, 10=20.17%, 20=14.62%, 50=9.75%
  4900. lat (msec) : 100=4.50%, 250=10.16%, 500=14.59%, 750=8.92%, 1000=3.45%
  4901. cpu : usr=0.03%, sys=0.10%, ctx=4530, majf=0, minf=31
  4902. IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  4903. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  4904. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  4905. issued rwts: total=0,2667,0,0 short=0,0,0,0 dropped=0,0,0,0
  4906. latency : target=0, window=0, percentile=100.00%, depth=4
  4907. read: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:35:00 2019
  4908. read: IOPS=124, BW=7969KiB/s (8160kB/s)(938MiB/120482msec)
  4909. slat (nsec): min=0, max=11516k, avg=36049.89, stdev=185515.83
  4910. clat (nsec): min=0, max=1889.0M, avg=32130791.70, stdev=149359827.92
  4911. lat (nsec): min=0, max=1890.1M, avg=32167996.44, stdev=149380240.45
  4912. clat percentiles (nsec):
  4913. | 1.00th=[ 716], 5.00th=[ 724], 10.00th=[ 740],
  4914. | 20.00th=[ 780], 30.00th=[ 804], 40.00th=[ 852],
  4915. | 50.00th=[ 892], 60.00th=[ 956], 70.00th=[ 1240],
  4916. | 80.00th=[ 30592], 90.00th=[ 5603328], 95.00th=[ 81264640],
  4917. | 99.00th=[ 834666496], 99.50th=[ 960495616], 99.90th=[1216348160],
  4918. | 99.95th=[1367343104], 99.99th=[1853882368]
  4919. bw ( KiB/s): min= 0, max= 8257, per=69.65%, avg=7998.50, stdev=365.57, samples=2
  4920. iops : min= 0, max= 129, avg=124.50, stdev= 6.36, samples=2
  4921. lat (nsec) : 750=11.93%, 1000=50.52%
  4922. lat (usec) : 2=16.29%, 4=0.27%, 10=0.13%, 20=0.79%, 50=0.21%
  4923. lat (usec) : 100=0.11%, 250=0.09%, 500=0.09%, 750=0.31%, 1000=1.35%
  4924. lat (msec) : 2=3.72%, 4=3.19%, 10=2.74%, 20=1.33%, 50=1.45%
  4925. lat (msec) : 100=0.61%, 250=0.61%, 500=0.96%, 750=1.81%, 1000=1.11%
  4926. cpu : usr=0.04%, sys=0.17%, ctx=5951, majf=0, minf=31
  4927. IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  4928. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  4929. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  4930. issued rwts: total=14999,0,0,0 short=0,0,0,0 dropped=0,0,0,0
  4931. latency : target=0, window=0, percentile=100.00%, depth=4
  4932. readwrite: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:35:00 2019
  4933. read: IOPS=55, BW=3529KiB/s (3614kB/s)(414MiB/120011msec)
  4934. slat (nsec): min=0, max=3964.5k, avg=26293.80, stdev=81795.50
  4935. clat (nsec): min=0, max=739318k, avg=10488878.55, stdev=55028154.30
  4936. lat (nsec): min=0, max=739351k, avg=10516103.18, stdev=55036378.95
  4937. clat percentiles (nsec):
  4938. | 1.00th=[ 716], 5.00th=[ 724], 10.00th=[ 732],
  4939. | 20.00th=[ 772], 30.00th=[ 876], 40.00th=[ 988],
  4940. | 50.00th=[ 1144], 60.00th=[ 1288], 70.00th=[ 1768],
  4941. | 80.00th=[ 692224], 90.00th=[ 4816896], 95.00th=[ 44302336],
  4942. | 99.00th=[329252864], 99.50th=[450887680], 99.90th=[624951296],
  4943. | 99.95th=[675282944], 99.99th=[742391808]
  4944. bw ( KiB/s): min= 0, max= 5220, per=30.73%, avg=3529.00, stdev=2391.44, samples=2
  4945. iops : min= 0, max= 81, avg=54.50, stdev=37.48, samples=2
  4946. write: IOPS=56, BW=3609KiB/s (3695kB/s)(423MiB/120011msec)
  4947. slat (nsec): min=0, max=9699.0k, avg=182579.58, stdev=251670.03
  4948. clat (nsec): min=0, max=948820k, avg=60636613.95, stdev=123178432.93
  4949. lat (nsec): min=0, max=948884k, avg=60820348.26, stdev=123136199.28
  4950. clat percentiles (usec):
  4951. | 1.00th=[ 1012], 5.00th=[ 1221], 10.00th=[ 1401], 20.00th=[ 1778],
  4952. | 30.00th=[ 2343], 40.00th=[ 3523], 50.00th=[ 6652], 60.00th=[ 15401],
  4953. | 70.00th=[ 37487], 80.00th=[ 90702], 90.00th=[166724], 95.00th=[333448],
  4954. | 99.00th=[641729], 99.50th=[717226], 99.90th=[893387], 99.95th=[918553],
  4955. | 99.99th=[952108]
  4956. bw ( KiB/s): min= 0, max= 5319, per=72.13%, avg=3606.50, stdev=2421.84, samples=2
  4957. iops : min= 0, max= 83, avg=56.00, stdev=38.18, samples=2
  4958. lat (nsec) : 750=8.29%, 1000=11.81%
  4959. lat (usec) : 2=14.65%, 4=0.04%, 10=0.39%, 20=1.12%, 50=0.75%
  4960. lat (usec) : 100=0.76%, 250=1.17%, 500=0.37%, 750=0.46%, 1000=1.48%
  4961. lat (msec) : 2=13.97%, 4=10.30%, 10=7.65%, 20=4.71%, 50=5.79%
  4962. lat (msec) : 100=5.75%, 250=6.33%, 500=3.01%, 750=1.09%, 1000=0.13%
  4963. cpu : usr=0.07%, sys=0.31%, ctx=14900, majf=0, minf=31
  4964. IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  4965. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  4966. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  4967. issued rwts: total=6618,6764,0,0 short=0,0,0,0 dropped=0,0,0,0
  4968. latency : target=0, window=0, percentile=100.00%, depth=4
  4969.  
  4970. Run status group 0 (all jobs):
  4971. READ: bw=11.2MiB/s (11.8MB/s), 3529KiB/s-7969KiB/s (3614kB/s-8160kB/s), io=1351MiB (1417MB), run=120011-120482msec
  4972. WRITE: bw=5000KiB/s (5120kB/s), 1415KiB/s-3609KiB/s (1449kB/s-3695kB/s), io=590MiB (618MB), run=120011-120791msec
  4973.  
  4974. Disk stats (read/write):
  4975. rbd184: ios=4567/10108, merge=0/2, ticks=592804/909666, in_queue=1495157, util=4.30%
  4976. <example-benchmark-fio-server-400-benchmark-fj44s>
  4977. <example-benchmark-fio-server-644-benchmark-z4l9v>
  4978. randwrite: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:35:00 2019
  4979. write: IOPS=19, BW=1259KiB/s (1289kB/s)(148MiB/120652msec)
  4980. slat (nsec): min=0, max=661885k, avg=818240.79, stdev=16567186.30
  4981. clat (nsec): min=0, max=1579.3M, avg=202990774.12, stdev=318293903.98
  4982. lat (nsec): min=0, max=1579.9M, avg=203809884.73, stdev=318373055.97
  4983. clat percentiles (usec):
  4984. | 1.00th=[ 1254], 5.00th=[ 1827], 10.00th=[ 2343],
  4985. | 20.00th=[ 3425], 30.00th=[ 5211], 40.00th=[ 7963],
  4986. | 50.00th=[ 12911], 60.00th=[ 25297], 70.00th=[ 287310],
  4987. | 80.00th=[ 434111], 90.00th=[ 692061], 95.00th=[ 893387],
  4988. | 99.00th=[1283458], 99.50th=[1434452], 99.90th=[1484784],
  4989. | 99.95th=[1484784], 99.99th=[1585447]
  4990. bw ( KiB/s): min= 0, max= 1388, per=22.62%, avg=1259.00, stdev=182.43, samples=2
  4991. iops : min= 0, max= 21, avg=19.00, stdev= 2.83, samples=2
  4992. lat (usec) : 750=0.08%, 1000=0.17%
  4993. lat (msec) : 2=6.12%, 4=17.30%, 10=20.55%, 20=12.62%, 50=8.31%
  4994. lat (msec) : 100=1.60%, 250=1.18%, 500=14.56%, 750=9.70%, 1000=4.68%
  4995. cpu : usr=0.03%, sys=0.09%, ctx=4052, majf=0, minf=31
  4996. IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  4997. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  4998. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  4999. issued rwts: total=0,2370,0,0 short=0,0,0,0 dropped=0,0,0,0
  5000. latency : target=0, window=0, percentile=100.00%, depth=4
  5001. read: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:35:00 2019
  5002. read: IOPS=204, BW=12.8MiB/s (13.4MB/s)(1540MiB/120482msec)
  5003. slat (nsec): min=0, max=123504k, avg=30115.36, stdev=798302.18
  5004. clat (nsec): min=0, max=1866.4M, avg=19516391.16, stdev=117399582.99
  5005. lat (nsec): min=0, max=1866.4M, avg=19547550.88, stdev=117418991.93
  5006. clat percentiles (nsec):
  5007. | 1.00th=[ 708], 5.00th=[ 724], 10.00th=[ 732],
  5008. | 20.00th=[ 772], 30.00th=[ 788], 40.00th=[ 812],
  5009. | 50.00th=[ 852], 60.00th=[ 876], 70.00th=[ 924],
  5010. | 80.00th=[ 1240], 90.00th=[ 1368064], 95.00th=[ 9895936],
  5011. | 99.00th=[ 734003200], 99.50th=[ 884998144], 99.90th=[1166016512],
  5012. | 99.95th=[1283457024], 99.99th=[1837105152]
  5013. bw ( KiB/s): min= 0, max=16005, per=75.84%, avg=13136.50, stdev=4056.67, samples=2
  5014. iops : min= 0, max= 250, avg=205.00, stdev=63.64, samples=2
  5015. lat (nsec) : 750=12.70%, 1000=61.83%
  5016. lat (usec) : 2=11.12%, 4=0.20%, 10=0.34%, 20=1.36%, 50=0.21%
  5017. lat (usec) : 100=0.10%, 250=0.12%, 500=0.05%, 750=0.19%, 1000=0.78%
  5018. lat (msec) : 2=2.21%, 4=2.08%, 10=1.74%, 20=0.78%, 50=0.82%
  5019. lat (msec) : 100=0.38%, 250=0.33%, 500=0.72%, 750=1.06%, 1000=0.62%
  5020. cpu : usr=0.06%, sys=0.24%, ctx=6050, majf=0, minf=31
  5021. IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  5022. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  5023. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  5024. issued rwts: total=24637,0,0,0 short=0,0,0,0 dropped=0,0,0,0
  5025. latency : target=0, window=0, percentile=100.00%, depth=4
  5026. readwrite: (groupid=0, jobs=1): err= 0: pid=14: Wed May 8 20:35:00 2019
  5027. read: IOPS=66, BW=4250KiB/s (4352kB/s)(498MiB/120010msec)
  5028. slat (nsec): min=0, max=2540.8k, avg=24466.37, stdev=63425.35
  5029. clat (nsec): min=0, max=1096.5M, avg=8256931.65, stdev=50119470.91
  5030. lat (nsec): min=0, max=1096.6M, avg=8282370.65, stdev=50126584.41
  5031. clat percentiles (nsec):
  5032. | 1.00th=[ 716], 5.00th=[ 724], 10.00th=[ 732],
  5033. | 20.00th=[ 756], 30.00th=[ 852], 40.00th=[ 940],
  5034. | 50.00th=[ 1080], 60.00th=[ 1224], 70.00th=[ 1560],
  5035. | 80.00th=[ 168960], 90.00th=[ 2473984], 95.00th=[ 25821184],
  5036. | 99.00th=[ 274726912], 99.50th=[ 425721856], 99.90th=[ 616562688],
  5037. | 99.95th=[ 742391808], 99.99th=[1098907648]
  5038. bw ( KiB/s): min= 0, max= 6593, per=24.54%, avg=4250.50, stdev=3312.80, samples=2
  5039. iops : min= 0, max= 103, avg=66.00, stdev=52.33, samples=2
  5040. write: IOPS=67, BW=4330KiB/s (4434kB/s)(507MiB/120010msec)
  5041. slat (nsec): min=0, max=7458.3k, avg=189853.89, stdev=259526.74
  5042. clat (nsec): min=0, max=894153k, avg=50960770.68, stdev=106485643.26
  5043. lat (nsec): min=0, max=894188k, avg=51151899.21, stdev=106440642.32
  5044. clat percentiles (usec):
  5045. | 1.00th=[ 979], 5.00th=[ 1221], 10.00th=[ 1385], 20.00th=[ 1745],
  5046. | 30.00th=[ 2245], 40.00th=[ 3064], 50.00th=[ 4752], 60.00th=[ 10290],
  5047. | 70.00th=[ 25822], 80.00th=[ 81265], 90.00th=[141558], 95.00th=[267387],
  5048. | 99.00th=[557843], 99.50th=[650118], 99.90th=[742392], 99.95th=[767558],
  5049. | 99.99th=[893387]
  5050. bw ( KiB/s): min= 0, max= 6640, per=77.77%, avg=4328.00, stdev=3269.66, samples=2
  5051. iops : min= 0, max= 103, avg=67.00, stdev=50.91, samples=2
  5052. lat (nsec) : 750=8.95%, 1000=12.87%
  5053. lat (usec) : 2=13.44%, 4=0.02%, 10=0.40%, 20=1.39%, 50=0.77%
  5054. lat (usec) : 100=1.04%, 250=1.31%, 500=0.63%, 750=0.53%, 1000=1.59%
  5055. lat (msec) : 2=14.34%, 4=11.53%, 10=7.63%, 20=4.38%, 50=4.83%
  5056. lat (msec) : 100=5.20%, 250=5.91%, 500=2.44%, 750=0.76%, 1000=0.06%
  5057. cpu : usr=0.07%, sys=0.38%, ctx=17220, majf=0, minf=31
  5058. IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  5059. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  5060. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  5061. issued rwts: total=7970,8116,0,0 short=0,0,0,0 dropped=0,0,0,0
  5062. latency : target=0, window=0, percentile=100.00%, depth=4
  5063.  
  5064. Run status group 0 (all jobs):
  5065. READ: bw=16.9MiB/s (17.7MB/s), 4250KiB/s-12.8MiB/s (4352kB/s-13.4MB/s), io=2038MiB (2137MB), run=120010-120482msec
  5066. WRITE: bw=5565KiB/s (5699kB/s), 1259KiB/s-4330KiB/s (1289kB/s-4434kB/s), io=656MiB (688MB), run=120010-120652msec
  5067.  
  5068. Disk stats (read/write):
  5069. rbd162: ios=4669/11042, merge=0/1, ticks=588851/900875, in_queue=1481867, util=5.90%
  5070. <example-benchmark-fio-server-216-benchmark-88f76>
  5071. randwrite: (groupid=0, jobs=1): err= 0: pid=11: Wed May 8 20:35:00 2019
  5072. write: IOPS=44, BW=2862KiB/s (2931kB/s)(336MiB/120068msec)
  5073. slat (nsec): min=0, max=556126k, avg=395599.01, stdev=9573679.68
  5074. clat (nsec): min=0, max=1375.0M, avg=89644510.34, stdev=147243605.16
  5075. lat (nsec): min=0, max=1375.2M, avg=90041328.80, stdev=147426996.10
  5076. clat percentiles (usec):
  5077. | 1.00th=[ 979], 5.00th=[ 1172], 10.00th=[ 1352],
  5078. | 20.00th=[ 1762], 30.00th=[ 2507], 40.00th=[ 5014],
  5079. | 50.00th=[ 21103], 60.00th=[ 68682], 70.00th=[ 96994],
  5080. | 80.00th=[ 127402], 90.00th=[ 258999], 95.00th=[ 421528],
  5081. | 99.00th=[ 708838], 99.50th=[ 750781], 99.90th=[ 918553],
  5082. | 99.95th=[1132463], 99.99th=[1367344]
  5083. bw ( KiB/s): min= 0, max= 3492, per=45.92%, avg=2861.00, stdev=892.37, samples=2
  5084. iops : min= 0, max= 54, avg=44.00, stdev=14.14, samples=2
  5085. lat (usec) : 4=0.02%, 500=0.04%, 750=0.07%, 1000=1.12%
  5086. lat (msec) : 2=22.93%, 4=13.12%, 10=7.85%, 20=4.34%, 50=6.04%
  5087. lat (msec) : 100=15.64%, 250=18.55%, 500=6.93%, 750=2.80%, 1000=0.54%
  5088. cpu : usr=0.07%, sys=0.18%, ctx=10180, majf=0, minf=30
  5089. IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  5090. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  5091. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  5092. issued rwts: total=0,5365,0,0 short=0,0,0,0 dropped=0,0,0,0
  5093. latency : target=0, window=0, percentile=100.00%, depth=4
  5094. read: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:35:00 2019
  5095. read: IOPS=305, BW=19.1MiB/s (20.0MB/s)(2292MiB/120015msec)
  5096. slat (nsec): min=0, max=510621k, avg=52091.46, stdev=3300689.27
  5097. clat (nsec): min=0, max=1192.9M, avg=13093072.10, stdev=61828654.90
  5098. lat (nsec): min=0, max=1192.9M, avg=13145990.94, stdev=61925908.23
  5099. clat percentiles (nsec):
  5100. | 1.00th=[ 708], 5.00th=[ 716], 10.00th=[ 716],
  5101. | 20.00th=[ 724], 30.00th=[ 740], 40.00th=[ 780],
  5102. | 50.00th=[ 844], 60.00th=[ 908], 70.00th=[ 1128],
  5103. | 80.00th=[ 60672], 90.00th=[ 15400960], 95.00th=[ 60555264],
  5104. | 99.00th=[362807296], 99.50th=[484442112], 99.90th=[683671552],
  5105. | 99.95th=[734003200], 99.99th=[826277888]
  5106. bw ( KiB/s): min= 0, max=24684, per=85.72%, avg=19554.50, stdev=7254.21, samples=2
  5107. iops : min= 0, max= 386, avg=305.50, stdev=113.84, samples=2
  5108. lat (nsec) : 750=35.59%, 1000=30.25%
  5109. lat (usec) : 2=12.30%, 4=0.03%, 10=0.67%, 20=0.68%, 50=0.45%
  5110. lat (usec) : 100=0.10%, 250=0.06%, 500=0.31%, 750=1.12%, 1000=2.37%
  5111. lat (msec) : 2=3.13%, 4=1.82%, 10=0.85%, 20=0.52%, 50=3.62%
  5112. lat (msec) : 100=3.02%, 250=1.50%, 500=1.20%, 750=0.40%, 1000=0.04%
  5113. cpu : usr=0.10%, sys=0.36%, ctx=14507, majf=0, minf=31
  5114. IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  5115. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  5116. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  5117. issued rwts: total=36693,0,0,0 short=0,0,0,0 dropped=0,0,0,0
  5118. latency : target=0, window=0, percentile=100.00%, depth=4
  5119. readwrite: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:35:00 2019
  5120. read: IOPS=50, BW=3258KiB/s (3337kB/s)(382MiB/120007msec)
  5121. slat (nsec): min=0, max=64156k, avg=44426.52, stdev=826886.07
  5122. clat (nsec): min=0, max=612522k, avg=7346455.67, stdev=38544224.56
  5123. lat (nsec): min=0, max=612553k, avg=7391819.68, stdev=38557927.00
  5124. clat percentiles (nsec):
  5125. | 1.00th=[ 716], 5.00th=[ 732], 10.00th=[ 740],
  5126. | 20.00th=[ 812], 30.00th=[ 900], 40.00th=[ 1020],
  5127. | 50.00th=[ 1176], 60.00th=[ 1336], 70.00th=[ 33024],
  5128. | 80.00th=[ 946176], 90.00th=[ 4489216], 95.00th=[ 39059456],
  5129. | 99.00th=[183500800], 99.50th=[295698432], 99.90th=[501219328],
  5130. | 99.95th=[526385152], 99.99th=[616562688]
  5131. bw ( KiB/s): min= 0, max= 4812, per=14.28%, avg=3258.00, stdev=2197.69, samples=2
  5132. iops : min= 0, max= 75, avg=50.50, stdev=34.65, samples=2
  5133. write: IOPS=52, BW=3370KiB/s (3451kB/s)(395MiB/120007msec)
  5134. slat (nsec): min=0, max=240431k, avg=228343.91, stdev=3035908.98
  5135. clat (nsec): min=0, max=1074.2M, avg=68752131.71, stdev=128360758.51
  5136. lat (nsec): min=0, max=1074.3M, avg=68981714.33, stdev=128329619.31
  5137. clat percentiles (usec):
  5138. | 1.00th=[ 1004], 5.00th=[ 1237], 10.00th=[ 1467],
  5139. | 20.00th=[ 2008], 30.00th=[ 2868], 40.00th=[ 4555],
  5140. | 50.00th=[ 10159], 60.00th=[ 22152], 70.00th=[ 66847],
  5141. | 80.00th=[ 105382], 90.00th=[ 187696], 95.00th=[ 350225],
  5142. | 99.00th=[ 650118], 99.50th=[ 734004], 99.90th=[ 834667],
  5143. | 99.95th=[ 893387], 99.99th=[1082131]
  5144. bw ( KiB/s): min= 0, max= 4920, per=54.05%, avg=3367.50, stdev=2195.57, samples=2
  5145. iops : min= 0, max= 76, avg=52.00, stdev=33.94, samples=2
  5146. lat (nsec) : 750=5.93%, 1000=13.00%
  5147. lat (usec) : 2=14.02%, 4=0.05%, 10=0.27%, 20=0.83%, 50=0.79%
  5148. lat (usec) : 100=0.76%, 250=0.79%, 500=0.92%, 750=0.79%, 1000=1.89%
  5149. lat (msec) : 2=12.49%, 4=10.62%, 10=7.48%, 20=5.36%, 50=5.31%
  5150. lat (msec) : 100=6.99%, 250=7.43%, 500=3.07%, 750=1.05%, 1000=0.18%
  5151. cpu : usr=0.06%, sys=0.30%, ctx=14609, majf=0, minf=31
  5152. IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  5153. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  5154. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  5155. issued rwts: total=6111,6316,0,0 short=0,0,0,0 dropped=0,0,0,0
  5156. latency : target=0, window=0, percentile=100.00%, depth=4
  5157.  
  5158. Run status group 0 (all jobs):
  5159. READ: bw=22.3MiB/s (23.4MB/s), 3258KiB/s-19.1MiB/s (3337kB/s-20.0MB/s), io=2674MiB (2804MB), run=120007-120015msec
  5160. WRITE: bw=6230KiB/s (6380kB/s), 2862KiB/s-3370KiB/s (2931kB/s-3451kB/s), io=731MiB (766MB), run=120007-120068msec
  5161.  
  5162. Disk stats (read/write):
  5163. rbd246: ios=9675/12516, merge=0/1, ticks=561703/838638, in_queue=1389101, util=15.18%
  5164. randwrite: (groupid=0, jobs=1): err= 0: pid=11: Wed May 8 20:35:00 2019
  5165. write: IOPS=27, BW=1792KiB/s (1835kB/s)(211MiB/120300msec)
  5166. slat (nsec): min=0, max=710004k, avg=768267.47, stdev=17305830.82
  5167. clat (nsec): min=0, max=2194.6M, avg=142987318.59, stdev=256277381.03
  5168. lat (nsec): min=0, max=2194.7M, avg=143756261.10, stdev=257049780.78
  5169. clat percentiles (usec):
  5170. | 1.00th=[ 1123], 5.00th=[ 1418], 10.00th=[ 1696],
  5171. | 20.00th=[ 2409], 30.00th=[ 3326], 40.00th=[ 5014],
  5172. | 50.00th=[ 9110], 60.00th=[ 38011], 70.00th=[ 105382],
  5173. | 80.00th=[ 221250], 90.00th=[ 526386], 95.00th=[ 725615],
  5174. | 99.00th=[1115685], 99.50th=[1249903], 99.90th=[1417675],
  5175. | 99.95th=[1870660], 99.99th=[2197816]
  5176. bw ( KiB/s): min= 0, max= 2231, per=31.72%, avg=1794.00, stdev=618.01, samples=2
  5177. iops : min= 0, max= 34, avg=27.50, stdev= 9.19, samples=2
  5178. lat (usec) : 1000=0.27%
  5179. lat (msec) : 2=14.38%, 4=20.68%, 10=15.75%, 20=5.61%, 50=4.58%
  5180. lat (msec) : 100=7.34%, 250=13.13%, 500=7.49%, 750=6.24%, 1000=3.00%
  5181. cpu : usr=0.03%, sys=0.12%, ctx=6249, majf=0, minf=30
  5182. IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  5183. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  5184. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  5185. issued rwts: total=0,3366,0,0 short=0,0,0,0 dropped=0,0,0,0
  5186. latency : target=0, window=0, percentile=100.00%, depth=4
  5187. read: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:35:00 2019
  5188. read: IOPS=486, BW=30.3MiB/s (31.8MB/s)(3644MiB/120302msec)
  5189. slat (nsec): min=0, max=481883k, avg=40883.35, stdev=3292713.26
  5190. clat (nsec): min=0, max=2142.9M, avg=8229375.61, stdev=64916449.31
  5191. lat (nsec): min=0, max=2143.1M, avg=8271109.54, stdev=65013247.85
  5192. clat percentiles (nsec):
  5193. | 1.00th=[ 716], 5.00th=[ 724], 10.00th=[ 724],
  5194. | 20.00th=[ 740], 30.00th=[ 772], 40.00th=[ 796],
  5195. | 50.00th=[ 844], 60.00th=[ 876], 70.00th=[ 916],
  5196. | 80.00th=[ 1096], 90.00th=[ 12736], 95.00th=[ 2039808],
  5197. | 99.00th=[ 354418688], 99.50th=[ 505413632], 99.90th=[ 859832320],
  5198. | 99.95th=[ 960495616], 99.99th=[1384120320]
  5199. bw ( KiB/s): min= 0, max=47390, per=89.35%, avg=31095.00, stdev=23044.61, samples=2
  5200. iops : min= 0, max= 743, avg=487.00, stdev=362.04, samples=2
  5201. lat (nsec) : 750=26.66%, 1000=51.00%
  5202. lat (usec) : 2=7.21%, 4=0.10%, 10=1.34%, 20=5.40%, 50=0.82%
  5203. lat (usec) : 100=0.15%, 250=0.08%, 500=0.08%, 750=0.24%, 1000=0.64%
  5204. lat (msec) : 2=1.24%, 4=0.96%, 10=0.53%, 20=0.22%, 50=0.79%
  5205. lat (msec) : 100=0.88%, 250=0.44%, 500=0.71%, 750=0.34%, 1000=0.13%
  5206. cpu : usr=0.13%, sys=0.48%, ctx=8589, majf=0, minf=99
  5207. IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  5208. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  5209. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  5210. issued rwts: total=58472,0,0,0 short=0,0,0,0 dropped=0,0,0,0
  5211. latency : target=0, window=0, percentile=100.00%, depth=4
  5212. readwrite: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:35:00 2019
  5213. read: IOPS=59, BW=3790KiB/s (3881kB/s)(444MiB/120012msec)
  5214. slat (nsec): min=0, max=2853.8k, avg=26733.30, stdev=79399.49
  5215. clat (nsec): min=0, max=768103k, avg=6051219.37, stdev=36611268.74
  5216. lat (nsec): min=0, max=768124k, avg=6078828.42, stdev=36617018.80
  5217. clat percentiles (nsec):
  5218. | 1.00th=[ 716], 5.00th=[ 724], 10.00th=[ 732],
  5219. | 20.00th=[ 764], 30.00th=[ 852], 40.00th=[ 948],
  5220. | 50.00th=[ 1096], 60.00th=[ 1240], 70.00th=[ 9536],
  5221. | 80.00th=[ 370688], 90.00th=[ 2605056], 95.00th=[ 27394048],
  5222. | 99.00th=[139460608], 99.50th=[252706816], 99.90th=[526385152],
  5223. | 99.95th=[641728512], 99.99th=[767557632]
  5224. bw ( KiB/s): min= 0, max= 6400, per=10.89%, avg=3790.00, stdev=3691.10, samples=2
  5225. iops : min= 0, max= 100, avg=59.00, stdev=57.98, samples=2
  5226. write: IOPS=60, BW=3873KiB/s (3966kB/s)(454MiB/120012msec)
  5227. slat (nsec): min=0, max=368178k, avg=260495.36, stdev=4957786.34
  5228. clat (nsec): min=0, max=1192.9M, avg=60005346.24, stdev=128875303.67
  5229. lat (nsec): min=0, max=1192.9M, avg=60267041.02, stdev=129001065.91
  5230. clat percentiles (usec):
  5231. | 1.00th=[ 1012], 5.00th=[ 1254], 10.00th=[ 1467],
  5232. | 20.00th=[ 1942], 30.00th=[ 2573], 40.00th=[ 3589],
  5233. | 50.00th=[ 5473], 60.00th=[ 11600], 70.00th=[ 29230],
  5234. | 80.00th=[ 90702], 90.00th=[ 160433], 95.00th=[ 333448],
  5235. | 99.00th=[ 650118], 99.50th=[ 725615], 99.90th=[ 876610],
  5236. | 99.95th=[1052771], 99.99th=[1199571]
  5237. bw ( KiB/s): min= 0, max= 6501, per=68.45%, avg=3871.00, stdev=3719.38, samples=2
  5238. iops : min= 0, max= 101, avg=60.00, stdev=57.98, samples=2
  5239. lat (nsec) : 750=8.53%, 1000=13.09%
  5240. lat (usec) : 2=12.74%, 4=0.03%, 10=0.36%, 20=1.38%, 50=0.77%
  5241. lat (usec) : 100=1.00%, 250=1.13%, 500=0.86%, 750=0.65%, 1000=1.64%
  5242. lat (msec) : 2=12.52%, 4=12.26%, 10=8.76%, 20=4.87%, 50=4.41%
  5243. lat (msec) : 100=5.01%, 250=6.32%, 500=2.24%, 750=1.23%, 1000=0.17%
  5244. cpu : usr=0.07%, sys=0.33%, ctx=15734, majf=0, minf=33
  5245. IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  5246. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  5247. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  5248. issued rwts: total=7108,7259,0,0 short=0,0,0,0 dropped=0,0,0,0
  5249. latency : target=0, window=0, percentile=100.00%, depth=4
  5250.  
  5251. Run status group 0 (all jobs):
  5252. READ: bw=33.0MiB/s (35.6MB/s), 3790KiB/s-30.3MiB/s (3881kB/s-31.8MB/s), io=4088MiB (4287MB), run=120012-120302msec
  5253. WRITE: bw=5656KiB/s (5791kB/s), 1792KiB/s-3873KiB/s (1835kB/s-3966kB/s), io=664MiB (697MB), run=120012-120300msec
  5254.  
  5255. Disk stats (read/write):
  5256. rbd62: ios=6007/10900, merge=0/1, ticks=553071/887900, in_queue=1432491, util=9.76%
  5257. <example-benchmark-fio-server-7-benchmark-n8krm>
  5258. randwrite: (groupid=0, jobs=1): err= 0: pid=11: Wed May 8 20:35:00 2019
  5259. write: IOPS=50, BW=3256KiB/s (3335kB/s)(382MiB/120041msec)
  5260. slat (nsec): min=0, max=754365k, avg=524418.01, stdev=13825228.46
  5261. clat (nsec): min=0, max=1437.4M, avg=78366317.32, stdev=145975603.65
  5262. lat (nsec): min=0, max=1438.1M, avg=78891857.37, stdev=146417463.46
  5263. clat percentiles (usec):
  5264. | 1.00th=[ 963], 5.00th=[ 1188], 10.00th=[ 1369],
  5265. | 20.00th=[ 1762], 30.00th=[ 2442], 40.00th=[ 3621],
  5266. | 50.00th=[ 6259], 60.00th=[ 18220], 70.00th=[ 81265],
  5267. | 80.00th=[ 112722], 90.00th=[ 250610], 95.00th=[ 400557],
  5268. | 99.00th=[ 700449], 99.50th=[ 759170], 99.90th=[ 901776],
  5269. | 99.95th=[ 926942], 99.99th=[1434452]
  5270. bw ( KiB/s): min= 0, max= 3328, per=42.00%, avg=3256.00, stdev=101.82, samples=2
  5271. iops : min= 0, max= 52, avg=50.50, stdev= 2.12, samples=2
  5272. lat (usec) : 500=0.02%, 750=0.07%, 1000=1.54%
  5273. lat (msec) : 2=22.49%, 4=18.21%, 10=13.19%, 20=5.03%, 50=3.37%
  5274. lat (msec) : 100=12.14%, 250=13.94%, 500=6.75%, 750=2.74%, 1000=0.54%
  5275. cpu : usr=0.06%, sys=0.21%, ctx=11444, majf=0, minf=26
  5276. IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  5277. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  5278. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  5279. issued rwts: total=0,6105,0,0 short=0,0,0,0 dropped=0,0,0,0
  5280. latency : target=0, window=0, percentile=100.00%, depth=4
  5281. read: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:35:00 2019
  5282. read: IOPS=344, BW=21.5MiB/s (22.6MB/s)(2581MiB/120018msec)
  5283. slat (nsec): min=0, max=507784k, avg=64993.68, stdev=4454457.57
  5284. clat (nsec): min=0, max=1336.8M, avg=11573441.71, stdev=60446260.20
  5285. lat (nsec): min=0, max=1336.8M, avg=11639287.24, stdev=60613041.39
  5286. clat percentiles (nsec):
  5287. | 1.00th=[ 708], 5.00th=[ 716], 10.00th=[ 724],
  5288. | 20.00th=[ 724], 30.00th=[ 732], 40.00th=[ 748],
  5289. | 50.00th=[ 820], 60.00th=[ 884], 70.00th=[ 1048],
  5290. | 80.00th=[ 11968], 90.00th=[ 3391488], 95.00th=[ 57409536],
  5291. | 99.00th=[337641472], 99.50th=[480247808], 99.90th=[750780416],
  5292. | 99.95th=[809500672], 99.99th=[977272832]
  5293. bw ( KiB/s): min= 0, max=31315, per=83.20%, avg=22024.00, stdev=13139.46, samples=2
  5294. iops : min= 0, max= 490, avg=344.00, stdev=206.48, samples=2
  5295. lat (nsec) : 750=41.49%, 1000=27.40%
  5296. lat (usec) : 2=9.78%, 4=0.07%, 10=1.14%, 20=0.61%, 50=0.39%
  5297. lat (usec) : 100=0.12%, 250=0.05%, 500=0.34%, 750=1.01%, 1000=2.49%
  5298. lat (msec) : 2=3.47%, 4=1.99%, 10=1.02%, 20=0.42%, 50=2.40%
  5299. lat (msec) : 100=3.23%, 250=1.24%, 500=0.93%, 750=0.34%, 1000=0.09%
  5300. cpu : usr=0.11%, sys=0.39%, ctx=15643, majf=0, minf=64
  5301. IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  5302. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  5303. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  5304. issued rwts: total=41370,0,0,0 short=0,0,0,0 dropped=0,0,0,0
  5305. latency : target=0, window=0, percentile=100.00%, depth=4
  5306. readwrite: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:35:00 2019
  5307. read: IOPS=69, BW=4450KiB/s (4557kB/s)(522MiB/120002msec)
  5308. slat (nsec): min=0, max=750545k, avg=167895.13, stdev=9048304.87
  5309. clat (nsec): min=0, max=997524k, avg=6591996.29, stdev=37271220.87
  5310. lat (nsec): min=0, max=997578k, avg=6760880.42, stdev=38337231.16
  5311. clat percentiles (nsec):
  5312. | 1.00th=[ 716], 5.00th=[ 724], 10.00th=[ 740],
  5313. | 20.00th=[ 804], 30.00th=[ 916], 40.00th=[ 1064],
  5314. | 50.00th=[ 1208], 60.00th=[ 1496], 70.00th=[ 610304],
  5315. | 80.00th=[ 1286144], 90.00th=[ 4145152], 95.00th=[ 34865152],
  5316. | 99.00th=[141557760], 99.50th=[265289728], 99.90th=[574619648],
  5317. | 99.95th=[591396864], 99.99th=[994050048]
  5318. bw ( KiB/s): min= 0, max= 6443, per=16.81%, avg=4449.00, stdev=2819.94, samples=2
  5319. iops : min= 0, max= 100, avg=69.00, stdev=43.84, samples=2
  5320. write: IOPS=70, BW=4498KiB/s (4605kB/s)(527MiB/120002msec)
  5321. slat (nsec): min=0, max=516019k, avg=258656.32, stdev=5651967.71
  5322. clat (nsec): min=0, max=1458.4M, avg=50155127.58, stdev=113685456.08
  5323. lat (nsec): min=0, max=1458.4M, avg=50415075.39, stdev=113756135.61
  5324. clat percentiles (usec):
  5325. | 1.00th=[ 1020], 5.00th=[ 1254], 10.00th=[ 1434],
  5326. | 20.00th=[ 1778], 30.00th=[ 2245], 40.00th=[ 2900],
  5327. | 50.00th=[ 3916], 60.00th=[ 6325], 70.00th=[ 19268],
  5328. | 80.00th=[ 74974], 90.00th=[ 129500], 95.00th=[ 295699],
  5329. | 99.00th=[ 591397], 99.50th=[ 683672], 99.90th=[ 784335],
  5330. | 99.95th=[ 901776], 99.99th=[1451230]
  5331. bw ( KiB/s): min= 0, max= 6458, per=57.99%, avg=4495.50, stdev=2775.39, samples=2
  5332. iops : min= 0, max= 100, avg=69.50, stdev=43.13, samples=2
  5333. lat (nsec) : 750=7.04%, 1000=11.01%
  5334. lat (usec) : 2=12.70%, 4=0.05%, 10=0.27%, 20=0.67%, 50=0.56%
  5335. lat (usec) : 100=0.59%, 250=0.83%, 500=0.88%, 750=1.10%, 1000=2.94%
  5336. lat (msec) : 2=16.24%, 4=15.20%, 10=8.56%, 20=3.31%, 50=4.02%
  5337. lat (msec) : 100=5.96%, 250=4.79%, 500=2.43%, 750=0.76%, 1000=0.09%
  5338. cpu : usr=0.08%, sys=0.39%, ctx=20699, majf=0, minf=31
  5339. IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  5340. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  5341. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  5342. issued rwts: total=8344,8429,0,0 short=0,0,0,0 dropped=0,0,0,0
  5343. latency : target=0, window=0, percentile=100.00%, depth=4
  5344.  
  5345. Run status group 0 (all jobs):
  5346. READ: bw=25.9MiB/s (27.1MB/s), 4450KiB/s-21.5MiB/s (4557kB/s-22.6MB/s), io=3103MiB (3253MB), run=120002-120018msec
  5347. WRITE: bw=7753KiB/s (7939kB/s), 3256KiB/s-4498KiB/s (3335kB/s-4605kB/s), io=909MiB (953MB), run=120002-120041msec
  5348.  
  5349. Disk stats (read/write):
  5350. rbd6: ios=11216/15012, merge=0/3, ticks=566580/831401, in_queue=1384528, util=16.71%
  5351. <example-benchmark-fio-server-3-benchmark-tx76m>
  5352. randwrite: (groupid=0, jobs=1): err= 0: pid=11: Wed May 8 20:35:00 2019
  5353. write: IOPS=18, BW=1156KiB/s (1184kB/s)(136MiB/120093msec)
  5354. slat (nsec): min=0, max=1871.0M, avg=2769651.20, stdev=56431218.59
  5355. clat (nsec): min=0, max=2617.7M, avg=220504187.67, stdev=349885888.10
  5356. lat (nsec): min=0, max=2617.8M, avg=223272360.64, stdev=353092581.18
  5357. clat percentiles (usec):
  5358. | 1.00th=[ 1074], 5.00th=[ 1303], 10.00th=[ 1565],
  5359. | 20.00th=[ 2180], 30.00th=[ 3228], 40.00th=[ 5473],
  5360. | 50.00th=[ 11731], 60.00th=[ 80217], 70.00th=[ 204473],
  5361. | 80.00th=[ 501220], 90.00th=[ 734004], 95.00th=[ 918553],
  5362. | 99.00th=[1400898], 99.50th=[1870660], 99.90th=[2071987],
  5363. | 99.95th=[2197816], 99.99th=[2634023]
  5364. bw ( KiB/s): min= 0, max= 1699, per=21.12%, avg=1155.00, stdev=769.33, samples=2
  5365. iops : min= 0, max= 26, avg=17.50, stdev=12.02, samples=2
  5366. lat (usec) : 4=0.05%, 1000=0.51%
  5367. lat (msec) : 2=16.75%, 4=17.17%, 10=13.71%, 20=6.46%, 50=2.58%
  5368. lat (msec) : 100=4.80%, 250=8.86%, 500=9.23%, 750=10.57%, 1000=5.86%
  5369. cpu : usr=0.02%, sys=0.09%, ctx=4098, majf=0, minf=30
  5370. IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  5371. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  5372. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  5373. issued rwts: total=0,2167,0,0 short=0,0,0,0 dropped=0,0,0,0
  5374. latency : target=0, window=0, percentile=100.00%, depth=4
  5375. read: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:35:00 2019
  5376. read: IOPS=545, BW=33.9MiB/s (35.6MB/s)(4074MiB/120118msec)
  5377. slat (nsec): min=0, max=1870.9M, avg=90467.46, stdev=10011085.33
  5378. clat (nsec): min=0, max=1903.2M, avg=7293770.07, stdev=69808358.17
  5379. lat (nsec): min=0, max=1903.2M, avg=7385182.62, stdev=70527189.25
  5380. clat percentiles (nsec):
  5381. | 1.00th=[ 708], 5.00th=[ 716], 10.00th=[ 724],
  5382. | 20.00th=[ 740], 30.00th=[ 788], 40.00th=[ 812],
  5383. | 50.00th=[ 852], 60.00th=[ 876], 70.00th=[ 916],
  5384. | 80.00th=[ 1004], 90.00th=[ 12736], 95.00th=[ 24960],
  5385. | 99.00th=[ 354418688], 99.50th=[ 557842432], 99.90th=[ 952107008],
  5386. | 99.95th=[1166016512], 99.99th=[1853882368]
  5387. bw ( KiB/s): min= 0, max=54319, per=89.19%, avg=34761.00, stdev=27659.19, samples=2
  5388. iops : min= 0, max= 855, avg=546.00, stdev=436.99, samples=2
  5389. lat (nsec) : 750=21.71%, 1000=58.02%
  5390. lat (usec) : 2=5.73%, 4=0.13%, 10=0.90%, 20=5.66%, 50=4.13%
  5391. lat (usec) : 100=0.14%, 250=0.06%, 500=0.06%, 750=0.15%, 1000=0.34%
  5392. lat (msec) : 2=0.50%, 4=0.38%, 10=0.18%, 20=0.10%, 50=0.26%
  5393. lat (msec) : 100=0.30%, 250=0.14%, 500=0.52%, 750=0.34%, 1000=0.17%
  5394. cpu : usr=0.12%, sys=0.51%, ctx=4867, majf=0, minf=30
  5395. IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  5396. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  5397. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  5398. issued rwts: total=65572,0,0,0 short=0,0,0,0 dropped=0,0,0,0
  5399. latency : target=0, window=0, percentile=100.00%, depth=4
  5400. readwrite: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:35:00 2019
  5401. read: IOPS=66, BW=4247KiB/s (4349kB/s)(498MiB/120004msec)
  5402. slat (nsec): min=0, max=657901k, avg=106802.92, stdev=7372430.83
  5403. clat (nsec): min=0, max=744115k, avg=4777220.47, stdev=35459077.18
  5404. lat (nsec): min=0, max=744165k, avg=4884922.55, stdev=36213602.95
  5405. clat percentiles (nsec):
  5406. | 1.00th=[ 716], 5.00th=[ 724], 10.00th=[ 732],
  5407. | 20.00th=[ 756], 30.00th=[ 852], 40.00th=[ 932],
  5408. | 50.00th=[ 1080], 60.00th=[ 1224], 70.00th=[ 1576],
  5409. | 80.00th=[ 84480], 90.00th=[ 1122304], 95.00th=[ 5799936],
  5410. | 99.00th=[141557760], 99.50th=[256901120], 99.90th=[557842432],
  5411. | 99.95th=[633339904], 99.99th=[742391808]
  5412. bw ( KiB/s): min= 0, max= 7251, per=10.89%, avg=4246.00, stdev=4249.71, samples=2
  5413. iops : min= 0, max= 113, avg=66.00, stdev=66.47, samples=2
  5414. write: IOPS=67, BW=4317KiB/s (4421kB/s)(506MiB/120004msec)
  5415. slat (nsec): min=0, max=445053k, avg=240599.24, stdev=4950521.64
  5416. clat (nsec): min=0, max=1125.9M, avg=54461828.03, stdev=123421580.22
  5417. lat (nsec): min=0, max=1125.0M, avg=54703547.45, stdev=123454883.67
  5418. clat percentiles (usec):
  5419. | 1.00th=[ 1012], 5.00th=[ 1221], 10.00th=[ 1369],
  5420. | 20.00th=[ 1713], 30.00th=[ 2147], 40.00th=[ 2835],
  5421. | 50.00th=[ 4178], 60.00th=[ 7898], 70.00th=[ 19792],
  5422. | 80.00th=[ 80217], 90.00th=[ 149947], 95.00th=[ 304088],
  5423. | 99.00th=[ 633340], 99.50th=[ 775947], 99.90th=[ 935330],
  5424. | 99.95th=[ 943719], 99.99th=[1132463]
  5425. bw ( KiB/s): min= 0, max= 7336, per=78.88%, avg=4315.00, stdev=4272.34, samples=2
  5426. iops : min= 0, max= 114, avg=67.00, stdev=66.47, samples=2
  5427. lat (nsec) : 750=8.84%, 1000=13.77%
  5428. lat (usec) : 2=12.64%, 4=0.04%, 10=0.65%, 20=1.71%, 50=0.92%
  5429. lat (usec) : 100=1.36%, 250=1.94%, 500=0.83%, 750=0.54%, 1000=1.45%
  5430. lat (msec) : 2=14.44%, 4=12.42%, 10=7.87%, 20=3.83%, 50=3.23%
  5431. lat (msec) : 100=4.75%, 250=5.61%, 500=2.03%, 750=0.84%, 1000=0.29%
  5432. cpu : usr=0.09%, sys=0.37%, ctx=16379, majf=0, minf=30
  5433. IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  5434. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  5435. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  5436. issued rwts: total=7964,8092,0,0 short=0,0,0,0 dropped=0,0,0,0
  5437. latency : target=0, window=0, percentile=100.00%, depth=4
  5438.  
  5439. Run status group 0 (all jobs):
  5440. READ: bw=38.1MiB/s (39.9MB/s), 4247KiB/s-33.9MiB/s (4349kB/s-35.6MB/s), io=4572MiB (4794MB), run=120004-120118msec
  5441. WRITE: bw=5470KiB/s (5602kB/s), 1156KiB/s-4317KiB/s (1184kB/s-4421kB/s), io=642MiB (673MB), run=120004-120093msec
  5442.  
  5443. Disk stats (read/write):
  5444. rbd1: ios=3345/10492, merge=0/1, ticks=511851/876978, in_queue=1381859, util=7.20%
  5445. <example-benchmark-fio-server-225-benchmark-d2hts>
  5446. randwrite: (groupid=0, jobs=1): err= 0: pid=11: Wed May 8 20:35:00 2019
  5447. write: IOPS=16, BW=1079KiB/s (1105kB/s)(127MiB/120445msec)
  5448. slat (nsec): min=0, max=860283k, avg=1696157.53, stdev=32922611.14
  5449. clat (nsec): min=0, max=2310.5M, avg=236942025.07, stdev=429727850.91
  5450. lat (nsec): min=0, max=2311.1M, avg=238639284.04, stdev=430204025.53
  5451. clat percentiles (usec):
  5452. | 1.00th=[ 1221], 5.00th=[ 1827], 10.00th=[ 2278],
  5453. | 20.00th=[ 3228], 30.00th=[ 4424], 40.00th=[ 6128],
  5454. | 50.00th=[ 10028], 60.00th=[ 21627], 70.00th=[ 78119],
  5455. | 80.00th=[ 513803], 90.00th=[ 943719], 95.00th=[1233126],
  5456. | 99.00th=[1568670], 99.50th=[1853883], 99.90th=[2105541],
  5457. | 99.95th=[2122318], 99.99th=[2298479]
  5458. bw ( KiB/s): min= 0, max= 1129, per=20.01%, avg=1079.00, stdev=70.71, samples=2
  5459. iops : min= 0, max= 17, avg=16.50, stdev= 0.71, samples=2
  5460. lat (usec) : 4=0.05%, 500=0.05%, 750=0.05%, 1000=0.35%
  5461. lat (msec) : 2=6.42%, 4=19.69%, 10=23.54%, 20=8.88%, 50=8.05%
  5462. lat (msec) : 100=4.79%, 250=3.80%, 500=4.15%, 750=4.84%, 1000=6.56%
  5463. cpu : usr=0.02%, sys=0.08%, ctx=3498, majf=0, minf=31
  5464. IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  5465. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  5466. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  5467. issued rwts: total=0,2026,0,0 short=0,0,0,0 dropped=0,0,0,0
  5468. latency : target=0, window=0, percentile=100.00%, depth=4
  5469. read: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:35:00 2019
  5470. read: IOPS=369, BW=23.0MiB/s (24.1MB/s)(2774MiB/120466msec)
  5471. slat (nsec): min=0, max=813265k, avg=56599.31, stdev=4929365.07
  5472. clat (nsec): min=0, max=1847.6M, avg=10781330.84, stdev=91567779.88
  5473. lat (nsec): min=0, max=1847.9M, avg=10838874.42, stdev=91712392.55
  5474. clat percentiles (nsec):
  5475. | 1.00th=[ 716], 5.00th=[ 724], 10.00th=[ 732],
  5476. | 20.00th=[ 772], 30.00th=[ 780], 40.00th=[ 804],
  5477. | 50.00th=[ 844], 60.00th=[ 868], 70.00th=[ 900],
  5478. | 80.00th=[ 972], 90.00th=[ 11584], 95.00th=[ 1187840],
  5479. | 99.00th=[ 574619648], 99.50th=[ 775946240], 99.90th=[1166016512],
  5480. | 99.95th=[1350565888], 99.99th=[1719664640]
  5481. bw ( KiB/s): min= 0, max=31363, per=85.02%, avg=23663.00, stdev=10889.44, samples=2
  5482. iops : min= 0, max= 491, avg=370.00, stdev=171.12, samples=2
  5483. lat (nsec) : 750=14.31%, 1000=66.73%
  5484. lat (usec) : 2=7.44%, 4=0.16%, 10=0.95%, 20=3.40%, 50=0.79%
  5485. lat (usec) : 100=0.18%, 250=0.11%, 500=0.11%, 750=0.11%, 1000=0.43%
  5486. lat (msec) : 2=1.18%, 4=0.97%, 10=0.74%, 20=0.29%, 50=0.38%
  5487. lat (msec) : 100=0.24%, 250=0.14%, 500=0.28%, 750=0.51%, 1000=0.44%
  5488. cpu : usr=0.09%, sys=0.39%, ctx=5480, majf=0, minf=31
  5489. IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  5490. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  5491. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  5492. issued rwts: total=44518,0,0,0 short=0,0,0,0 dropped=0,0,0,0
  5493. latency : target=0, window=0, percentile=100.00%, depth=4
  5494. readwrite: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:35:00 2019
  5495. read: IOPS=66, BW=4267KiB/s (4370kB/s)(500MiB/120003msec)
  5496. slat (nsec): min=0, max=4304.4k, avg=23613.57, stdev=74508.36
  5497. clat (nsec): min=0, max=759154k, avg=6090334.59, stdev=46592721.81
  5498. lat (nsec): min=0, max=759266k, avg=6114859.38, stdev=46599886.35
  5499. clat percentiles (nsec):
  5500. | 1.00th=[ 716], 5.00th=[ 724], 10.00th=[ 732],
  5501. | 20.00th=[ 756], 30.00th=[ 836], 40.00th=[ 908],
  5502. | 50.00th=[ 1048], 60.00th=[ 1208], 70.00th=[ 1496],
  5503. | 80.00th=[ 67072], 90.00th=[ 1204224], 95.00th=[ 5996544],
  5504. | 99.00th=[168820736], 99.50th=[434110464], 99.90th=[616562688],
  5505. | 99.95th=[675282944], 99.99th=[759169024]
  5506. bw ( KiB/s): min= 0, max= 5781, per=15.33%, avg=4267.00, stdev=2141.12, samples=2
  5507. iops : min= 0, max= 90, avg=66.50, stdev=33.23, samples=2
  5508. write: IOPS=67, BW=4330KiB/s (4434kB/s)(507MiB/120003msec)
  5509. slat (nsec): min=0, max=6756.8k, avg=175613.08, stdev=232944.45
  5510. clat (nsec): min=0, max=1093.4M, avg=53149774.18, stdev=110172503.70
  5511. lat (nsec): min=0, max=1093.5M, avg=53326567.81, stdev=110133857.81
  5512. clat percentiles (usec):
  5513. | 1.00th=[ 1074], 5.00th=[ 1319], 10.00th=[ 1483],
  5514. | 20.00th=[ 1909], 30.00th=[ 2507], 40.00th=[ 3687],
  5515. | 50.00th=[ 6390], 60.00th=[ 12649], 70.00th=[ 26346],
  5516. | 80.00th=[ 80217], 90.00th=[ 145753], 95.00th=[ 316670],
  5517. | 99.00th=[ 522191], 99.50th=[ 608175], 99.90th=[ 843056],
  5518. | 99.95th=[ 910164], 99.99th=[1098908]
  5519. bw ( KiB/s): min= 0, max= 5804, per=80.26%, avg=4327.50, stdev=2088.09, samples=2
  5520. iops : min= 0, max= 90, avg=67.00, stdev=32.53, samples=2
  5521. lat (nsec) : 750=9.35%, 1000=13.87%
  5522. lat (usec) : 2=12.52%, 4=0.04%, 10=0.68%, 20=1.67%, 50=1.02%
  5523. lat (usec) : 100=1.19%, 250=1.66%, 500=0.94%, 750=0.46%, 1000=1.06%
  5524. lat (msec) : 2=12.31%, 4=11.09%, 10=8.16%, 20=5.56%, 50=4.91%
  5525. lat (msec) : 100=4.41%, 250=5.52%, 500=2.78%, 750=0.69%, 1000=0.13%
  5526. cpu : usr=0.09%, sys=0.36%, ctx=16242, majf=0, minf=31
  5527. IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  5528. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  5529. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  5530. issued rwts: total=8001,8116,0,0 short=0,0,0,0 dropped=0,0,0,0
  5531. latency : target=0, window=0, percentile=100.00%, depth=4
  5532.  
  5533. Run status group 0 (all jobs):
  5534. READ: bw=27.2MiB/s (28.5MB/s), 4267KiB/s-23.0MiB/s (4370kB/s-24.1MB/s), io=3274MiB (3433MB), run=120003-120466msec
  5535. WRITE: bw=5393KiB/s (5522kB/s), 1079KiB/s-4330KiB/s (1105kB/s-4434kB/s), io=634MiB (665MB), run=120003-120445msec
  5536.  
  5537. Disk stats (read/write):
  5538. rbd76: ios=3853/10268, merge=0/1, ticks=568825/896607, in_queue=1458316, util=5.45%
  5539. <example-benchmark-fio-server-499-benchmark-hgwjk>
  5540. randwrite: (groupid=0, jobs=1): err= 0: pid=11: Wed May 8 20:35:00 2019
  5541. write: IOPS=20, BW=1336KiB/s (1369kB/s)(158MiB/121394msec)
  5542. slat (nsec): min=0, max=521835k, avg=602332.19, stdev=11489635.80
  5543. clat (nsec): min=0, max=2418.6M, avg=191197413.08, stdev=299955700.19
  5544. lat (nsec): min=0, max=2418.6M, avg=191801070.37, stdev=299979092.81
  5545. clat percentiles (usec):
  5546. | 1.00th=[ 1057], 5.00th=[ 1418], 10.00th=[ 1762],
  5547. | 20.00th=[ 2507], 30.00th=[ 3523], 40.00th=[ 5080],
  5548. | 50.00th=[ 10159], 60.00th=[ 63701], 70.00th=[ 244319],
  5549. | 80.00th=[ 408945], 90.00th=[ 633340], 95.00th=[ 759170],
  5550. | 99.00th=[1182794], 99.50th=[1400898], 99.90th=[2432697],
  5551. | 99.95th=[2432697], 99.99th=[2432697]
  5552. bw ( KiB/s): min= 0, max= 1506, per=24.22%, avg=1348.00, stdev=223.45, samples=2
  5553. iops : min= 0, max= 23, avg=20.50, stdev= 3.54, samples=2
  5554. lat (usec) : 4=0.04%, 100=0.04%, 500=0.04%, 750=0.04%, 1000=0.51%
  5555. lat (msec) : 2=12.99%, 4=19.87%, 10=16.43%, 20=6.04%, 50=2.61%
  5556. lat (msec) : 100=4.19%, 250=7.58%, 500=14.81%, 750=9.87%, 1000=2.45%
  5557. cpu : usr=0.03%, sys=0.09%, ctx=4216, majf=0, minf=31
  5558. IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  5559. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  5560. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  5561. issued rwts: total=0,2532,0,0 short=0,0,0,0 dropped=0,0,0,0
  5562. latency : target=0, window=0, percentile=100.00%, depth=4
  5563. read: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:35:00 2019
  5564. read: IOPS=139, BW=8931KiB/s (9145kB/s)(1048MiB/120186msec)
  5565. slat (nsec): min=0, max=9380.9k, avg=29479.14, stdev=141498.82
  5566. clat (nsec): min=0, max=1852.7M, avg=28689495.96, stdev=142920150.94
  5567. lat (nsec): min=0, max=1852.8M, avg=28720090.77, stdev=142943141.98
  5568. clat percentiles (nsec):
  5569. | 1.00th=[ 716], 5.00th=[ 724], 10.00th=[ 732],
  5570. | 20.00th=[ 772], 30.00th=[ 788], 40.00th=[ 836],
  5571. | 50.00th=[ 868], 60.00th=[ 900], 70.00th=[ 1048],
  5572. | 80.00th=[ 1416], 90.00th=[ 2703360], 95.00th=[ 43253760],
  5573. | 99.00th=[ 817889280], 99.50th=[ 943718400], 99.90th=[1283457024],
  5574. | 99.95th=[1451229184], 99.99th=[1837105152]
  5575. bw ( KiB/s): min= 0, max= 9136, per=67.97%, avg=8932.50, stdev=287.79, samples=2
  5576. iops : min= 0, max= 142, avg=139.00, stdev= 4.24, samples=2
  5577. lat (nsec) : 750=12.69%, 1000=55.86%
  5578. lat (usec) : 2=14.55%, 4=0.18%, 10=0.29%, 20=0.39%, 50=0.17%
  5579. lat (usec) : 100=0.13%, 250=0.04%, 500=0.08%, 750=0.26%, 1000=1.25%
  5580. lat (msec) : 2=3.16%, 4=2.30%, 10=1.74%, 20=0.89%, 50=1.22%
  5581. lat (msec) : 100=0.52%, 250=0.43%, 500=0.91%, 750=1.57%, 1000=1.01%
  5582. cpu : usr=0.05%, sys=0.18%, ctx=5400, majf=0, minf=31
  5583. IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  5584. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  5585. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  5586. issued rwts: total=16768,0,0,0 short=0,0,0,0 dropped=0,0,0,0
  5587. latency : target=0, window=0, percentile=100.00%, depth=4
  5588. readwrite: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:35:00 2019
  5589. read: IOPS=65, BW=4218KiB/s (4319kB/s)(494MiB/120008msec)
  5590. slat (nsec): min=0, max=3874.9k, avg=26448.84, stdev=87557.02
  5591. clat (nsec): min=0, max=904621k, avg=9496297.79, stdev=57007692.17
  5592. lat (nsec): min=0, max=905014k, avg=9523688.60, stdev=57015747.35
  5593. clat percentiles (nsec):
  5594. | 1.00th=[ 716], 5.00th=[ 724], 10.00th=[ 732],
  5595. | 20.00th=[ 748], 30.00th=[ 852], 40.00th=[ 948],
  5596. | 50.00th=[ 1096], 60.00th=[ 1256], 70.00th=[ 1544],
  5597. | 80.00th=[ 164864], 90.00th=[ 2473984], 95.00th=[ 30801920],
  5598. | 99.00th=[346030080], 99.50th=[488636416], 99.90th=[666894336],
  5599. | 99.95th=[717225984], 99.99th=[901775360]
  5600. bw ( KiB/s): min= 0, max= 6002, per=32.10%, avg=4218.00, stdev=2522.96, samples=2
  5601. iops : min= 0, max= 93, avg=65.50, stdev=38.89, samples=2
  5602. write: IOPS=66, BW=4278KiB/s (4381kB/s)(501MiB/120008msec)
  5603. slat (nsec): min=0, max=8021.8k, avg=195191.63, stdev=241106.21
  5604. clat (nsec): min=0, max=930859k, avg=50434385.70, stdev=108648012.81
  5605. lat (nsec): min=0, max=930927k, avg=50630816.52, stdev=108598568.70
  5606. clat percentiles (usec):
  5607. | 1.00th=[ 988], 5.00th=[ 1188], 10.00th=[ 1369], 20.00th=[ 1729],
  5608. | 30.00th=[ 2212], 40.00th=[ 2966], 50.00th=[ 4555], 60.00th=[ 9896],
  5609. | 70.00th=[ 26346], 80.00th=[ 79168], 90.00th=[131597], 95.00th=[261096],
  5610. | 99.00th=[549454], 99.50th=[683672], 99.90th=[884999], 99.95th=[918553],
  5611. | 99.99th=[926942]
  5612. bw ( KiB/s): min= 0, max= 6036, per=76.84%, avg=4276.00, stdev=2489.02, samples=2
  5613. iops : min= 0, max= 94, avg=66.50, stdev=38.89, samples=2
  5614. lat (nsec) : 750=9.89%, 1000=11.90%
  5615. lat (usec) : 2=13.76%, 4=0.02%, 10=0.42%, 20=1.27%, 50=0.87%
  5616. lat (usec) : 100=1.00%, 250=1.09%, 500=0.65%, 750=0.62%, 1000=1.59%
  5617. lat (msec) : 2=14.31%, 4=11.99%, 10=7.35%, 20=3.99%, 50=4.79%
  5618. lat (msec) : 100=5.32%, 250=5.96%, 500=2.30%, 750=0.78%, 1000=0.16%
  5619. cpu : usr=0.09%, sys=0.36%, ctx=16941, majf=0, minf=34
  5620. IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  5621. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  5622. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  5623. issued rwts: total=7909,8019,0,0 short=0,0,0,0 dropped=0,0,0,0
  5624. latency : target=0, window=0, percentile=100.00%, depth=4
  5625.  
  5626. Run status group 0 (all jobs):
  5627. READ: bw=12.8MiB/s (13.5MB/s), 4218KiB/s-8931KiB/s (4319kB/s-9145kB/s), io=1543MiB (1617MB), run=120008-120186msec
  5628. WRITE: bw=5566KiB/s (5699kB/s), 1336KiB/s-4278KiB/s (1369kB/s-4381kB/s), io=660MiB (692MB), run=120008-121394msec
  5629.  
  5630. Disk stats (read/write):
  5631. rbd210: ios=4224/10915, merge=0/2, ticks=600452/894257, in_queue=1487171, util=4.82%
  5632. <example-benchmark-fio-server-352-benchmark-swk2k>
  5633. randwrite: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:35:01 2019
  5634. write: IOPS=19, BW=1250KiB/s (1280kB/s)(149MiB/122256msec)
  5635. slat (nsec): min=0, max=3241.0M, avg=1882431.65, stdev=67552986.62
  5636. clat (nsec): min=0, max=3320.5M, avg=201732832.03, stdev=317272332.29
  5637. lat (nsec): min=0, max=3373.9M, avg=203614730.47, stdev=324396055.25
  5638. clat percentiles (usec):
  5639. | 1.00th=[ 1139], 5.00th=[ 1713], 10.00th=[ 2442],
  5640. | 20.00th=[ 3818], 30.00th=[ 6456], 40.00th=[ 10421],
  5641. | 50.00th=[ 18482], 60.00th=[ 54264], 70.00th=[ 233833],
  5642. | 80.00th=[ 450888], 90.00th=[ 692061], 95.00th=[ 826278],
  5643. | 99.00th=[1132463], 99.50th=[1367344], 99.90th=[3238003],
  5644. | 99.95th=[3305112], 99.99th=[3305112]
  5645. bw ( KiB/s): min= 0, max= 1384, per=23.96%, avg=1270.50, stdev=160.51, samples=2
  5646. iops : min= 0, max= 21, avg=19.50, stdev= 2.12, samples=2
  5647. lat (usec) : 500=0.04%, 1000=0.42%
  5648. lat (msec) : 2=6.71%, 4=13.67%, 10=18.54%, 20=12.33%, 50=7.76%
  5649. lat (msec) : 100=4.49%, 250=7.09%, 500=11.91%, 750=9.77%, 1000=4.82%
  5650. cpu : usr=0.03%, sys=0.08%, ctx=4156, majf=0, minf=29
  5651. IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  5652. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  5653. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  5654. issued rwts: total=0,2384,0,0 short=0,0,0,0 dropped=0,0,0,0
  5655. latency : target=0, window=0, percentile=100.00%, depth=4
  5656. read: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:35:01 2019
  5657. read: IOPS=150, BW=9662KiB/s (9894kB/s)(1137MiB/120533msec)
  5658. slat (nsec): min=0, max=11726k, avg=30343.88, stdev=150033.44
  5659. clat (nsec): min=0, max=1774.4M, avg=26494562.90, stdev=136028998.42
  5660. lat (nsec): min=0, max=1774.5M, avg=26526057.61, stdev=136047028.65
  5661. clat percentiles (nsec):
  5662. | 1.00th=[ 716], 5.00th=[ 724], 10.00th=[ 732],
  5663. | 20.00th=[ 780], 30.00th=[ 796], 40.00th=[ 836],
  5664. | 50.00th=[ 876], 60.00th=[ 916], 70.00th=[ 1112],
  5665. | 80.00th=[ 1512], 90.00th=[ 2965504], 95.00th=[ 36962304],
  5666. | 99.00th=[ 792723456], 99.50th=[ 935329792], 99.90th=[1233125376],
  5667. | 99.95th=[1350565888], 99.99th=[1669332992]
  5668. bw ( KiB/s): min= 0, max= 9968, per=70.80%, avg=9702.50, stdev=375.47, samples=2
  5669. iops : min= 0, max= 155, avg=151.00, stdev= 5.66, samples=2
  5670. lat (nsec) : 750=13.35%, 1000=53.24%
  5671. lat (usec) : 2=14.88%, 4=0.20%, 10=0.15%, 20=0.68%, 50=0.20%
  5672. lat (usec) : 100=0.08%, 250=0.06%, 500=0.10%, 750=0.29%, 1000=1.26%
  5673. lat (msec) : 2=3.73%, 4=2.91%, 10=2.18%, 20=0.86%, 50=1.28%
  5674. lat (msec) : 100=0.60%, 250=0.36%, 500=0.92%, 750=1.40%, 1000=0.96%
  5675. cpu : usr=0.05%, sys=0.19%, ctx=6330, majf=0, minf=30
  5676. IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  5677. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  5678. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  5679. issued rwts: total=18194,0,0,0 short=0,0,0,0 dropped=0,0,0,0
  5680. latency : target=0, window=0, percentile=100.00%, depth=4
  5681. readwrite: (groupid=0, jobs=1): err= 0: pid=14: Wed May 8 20:35:01 2019
  5682. read: IOPS=63, BW=4060KiB/s (4157kB/s)(476MiB/120009msec)
  5683. slat (nsec): min=0, max=2641.9k, avg=25605.77, stdev=72611.64
  5684. clat (nsec): min=0, max=995069k, avg=6187294.75, stdev=39693365.11
  5685. lat (nsec): min=0, max=995130k, avg=6213825.74, stdev=39700139.79
  5686. clat percentiles (nsec):
  5687. | 1.00th=[ 716], 5.00th=[ 724], 10.00th=[ 724],
  5688. | 20.00th=[ 748], 30.00th=[ 844], 40.00th=[ 924],
  5689. | 50.00th=[ 1064], 60.00th=[ 1224], 70.00th=[ 1528],
  5690. | 80.00th=[ 142336], 90.00th=[ 1761280], 95.00th=[ 11993088],
  5691. | 99.00th=[200278016], 99.50th=[320864256], 99.90th=[517996544],
  5692. | 99.95th=[641728512], 99.99th=[994050048]
  5693. bw ( KiB/s): min= 0, max= 6873, per=29.62%, avg=4059.50, stdev=3978.89, samples=2
  5694. iops : min= 0, max= 107, avg=63.00, stdev=62.23, samples=2
  5695. write: IOPS=64, BW=4129KiB/s (4228kB/s)(484MiB/120009msec)
  5696. slat (nsec): min=0, max=6809.1k, avg=193732.28, stdev=258346.27
  5697. clat (nsec): min=0, max=871312k, avg=55941812.81, stdev=118854266.47
  5698. lat (nsec): min=0, max=871345k, avg=56136695.07, stdev=118806735.79
  5699. clat percentiles (usec):
  5700. | 1.00th=[ 979], 5.00th=[ 1188], 10.00th=[ 1385], 20.00th=[ 1696],
  5701. | 30.00th=[ 2147], 40.00th=[ 2868], 50.00th=[ 4178], 60.00th=[ 7504],
  5702. | 70.00th=[ 29230], 80.00th=[ 87557], 90.00th=[160433], 95.00th=[325059],
  5703. | 99.00th=[608175], 99.50th=[666895], 99.90th=[775947], 99.95th=[843056],
  5704. | 99.99th=[868221]
  5705. bw ( KiB/s): min= 0, max= 6961, per=77.84%, avg=4127.00, stdev=4007.88, samples=2
  5706. iops : min= 0, max= 108, avg=64.00, stdev=62.23, samples=2
  5707. lat (nsec) : 750=9.91%, 1000=12.67%
  5708. lat (usec) : 2=12.85%, 4=0.07%, 10=0.40%, 20=1.26%, 50=0.87%
  5709. lat (usec) : 100=1.07%, 250=1.32%, 500=0.80%, 750=0.54%, 1000=1.93%
  5710. lat (msec) : 2=15.05%, 4=12.04%, 10=8.38%, 20=2.77%, 50=3.50%
  5711. lat (msec) : 100=4.94%, 250=6.21%, 500=2.23%, 750=1.11%, 1000=0.09%
  5712. cpu : usr=0.07%, sys=0.36%, ctx=16160, majf=0, minf=34
  5713. IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  5714. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  5715. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  5716. issued rwts: total=7613,7740,0,0 short=0,0,0,0 dropped=0,0,0,0
  5717. latency : target=0, window=0, percentile=100.00%, depth=4
  5718.  
  5719. Run status group 0 (all jobs):
  5720. READ: bw=13.4MiB/s (14.0MB/s), 4060KiB/s-9662KiB/s (4157kB/s-9894kB/s), io=1613MiB (1691MB), run=120009-120533msec
  5721. WRITE: bw=5303KiB/s (5430kB/s), 1250KiB/s-4129KiB/s (1280kB/s-4228kB/s), io=633MiB (664MB), run=120009-122256msec
  5722.  
  5723. Disk stats (read/write):
  5724. rbd118: ios=4729/10780, merge=0/2, ticks=572879/917855, in_queue=1482954, util=4.44%
  5725. <example-benchmark-fio-server-706-benchmark-xxdl7>
  5726. randwrite: (groupid=0, jobs=1): err= 0: pid=11: Wed May 8 20:35:01 2019
  5727. write: IOPS=26, BW=1719KiB/s (1760kB/s)(206MiB/122670msec)
  5728. slat (nsec): min=0, max=673225k, avg=684647.51, stdev=15798809.05
  5729. clat (nsec): min=0, max=1908.0M, avg=146098595.77, stdev=316285717.18
  5730. lat (nsec): min=0, max=1909.0M, avg=146784260.50, stdev=316853377.15
  5731. clat percentiles (usec):
  5732. | 1.00th=[ 1205], 5.00th=[ 1647], 10.00th=[ 1958],
  5733. | 20.00th=[ 2737], 30.00th=[ 3687], 40.00th=[ 4817],
  5734. | 50.00th=[ 6718], 60.00th=[ 9896], 70.00th=[ 16450],
  5735. | 80.00th=[ 240124], 90.00th=[ 566232], 95.00th=[ 901776],
  5736. | 99.00th=[1384121], 99.50th=[1501561], 99.90th=[1887437],
  5737. | 99.95th=[1904215], 99.99th=[1904215]
  5738. bw ( KiB/s): min= 0, max= 2234, per=29.13%, avg=1748.00, stdev=687.31, samples=2
  5739. iops : min= 0, max= 34, avg=26.50, stdev=10.61, samples=2
  5740. lat (usec) : 100=0.03%, 750=0.15%, 1000=0.24%
  5741. lat (msec) : 2=10.27%, 4=22.48%, 10=27.40%, 20=12.00%, 50=5.80%
  5742. lat (msec) : 100=0.94%, 250=0.79%, 500=8.35%, 750=4.86%, 1000=2.89%
  5743. cpu : usr=0.04%, sys=0.12%, ctx=5526, majf=0, minf=30
  5744. IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  5745. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  5746. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  5747. issued rwts: total=0,3292,0,0 short=0,0,0,0 dropped=0,0,0,0
  5748. latency : target=0, window=0, percentile=100.00%, depth=4
  5749. read: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:35:01 2019
  5750. read: IOPS=233, BW=14.6MiB/s (15.3MB/s)(1759MiB/120651msec)
  5751. slat (nsec): min=0, max=525196k, avg=46795.53, stdev=3133292.21
  5752. clat (nsec): min=0, max=1842.0M, avg=17115408.03, stdev=109396301.96
  5753. lat (nsec): min=0, max=1842.1M, avg=17163208.16, stdev=109539992.27
  5754. clat percentiles (nsec):
  5755. | 1.00th=[ 716], 5.00th=[ 732], 10.00th=[ 748],
  5756. | 20.00th=[ 780], 30.00th=[ 796], 40.00th=[ 836],
  5757. | 50.00th=[ 868], 60.00th=[ 900], 70.00th=[ 988],
  5758. | 80.00th=[ 1384], 90.00th=[ 2039808], 95.00th=[ 13565952],
  5759. | 99.00th=[ 708837376], 99.50th=[ 868220928], 99.90th=[1166016512],
  5760. | 99.95th=[1266679808], 99.99th=[1702887424]
  5761. bw ( KiB/s): min= 0, max=17562, per=78.09%, avg=15005.00, stdev=3616.14, samples=2
  5762. iops : min= 0, max= 274, avg=234.00, stdev=56.57, samples=2
  5763. lat (nsec) : 750=10.45%, 1000=59.97%
  5764. lat (usec) : 2=13.45%, 4=0.21%, 10=0.24%, 20=0.76%, 50=0.25%
  5765. lat (usec) : 100=0.13%, 250=0.11%, 500=0.06%, 750=0.26%, 1000=1.06%
  5766. lat (msec) : 2=2.98%, 4=2.55%, 10=1.99%, 20=1.12%, 50=1.38%
  5767. lat (msec) : 100=0.50%, 250=0.35%, 500=0.55%, 750=0.81%, 1000=0.61%
  5768. cpu : usr=0.08%, sys=0.29%, ctx=8279, majf=0, minf=30
  5769. IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  5770. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  5771. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  5772. issued rwts: total=28138,0,0,0 short=0,0,0,0 dropped=0,0,0,0
  5773. latency : target=0, window=0, percentile=100.00%, depth=4
  5774. readwrite: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:35:01 2019
  5775. read: IOPS=67, BW=4309KiB/s (4413kB/s)(505MiB/120043msec)
  5776. slat (nsec): min=0, max=3424.6k, avg=31999.20, stdev=100206.55
  5777. clat (nsec): min=0, max=740074k, avg=9433999.90, stdev=51294559.77
  5778. lat (nsec): min=0, max=740112k, avg=9466955.19, stdev=51304338.52
  5779. clat percentiles (nsec):
  5780. | 1.00th=[ 716], 5.00th=[ 724], 10.00th=[ 740],
  5781. | 20.00th=[ 796], 30.00th=[ 892], 40.00th=[ 1020],
  5782. | 50.00th=[ 1176], 60.00th=[ 1336], 70.00th=[ 18560],
  5783. | 80.00th=[ 839680], 90.00th=[ 4358144], 95.00th=[ 42205184],
  5784. | 99.00th=[287309824], 99.50th=[417333248], 99.90th=[633339904],
  5785. | 99.95th=[700448768], 99.99th=[742391808]
  5786. bw ( KiB/s): min= 0, max= 6309, per=22.43%, avg=4310.00, stdev=2827.01, samples=2
  5787. iops : min= 0, max= 98, avg=67.00, stdev=43.84, samples=2
  5788. write: IOPS=68, BW=4377KiB/s (4482kB/s)(513MiB/120043msec)
  5789. slat (nsec): min=0, max=5049.0k, avg=193015.97, stdev=236476.27
  5790. clat (nsec): min=0, max=1324.3M, avg=49248474.97, stdev=103898012.15
  5791. lat (nsec): min=0, max=1324.3M, avg=49442877.34, stdev=103853368.88
  5792. clat percentiles (usec):
  5793. | 1.00th=[ 1012], 5.00th=[ 1221], 10.00th=[ 1369],
  5794. | 20.00th=[ 1680], 30.00th=[ 2212], 40.00th=[ 3064],
  5795. | 50.00th=[ 4948], 60.00th=[ 10552], 70.00th=[ 26870],
  5796. | 80.00th=[ 74974], 90.00th=[ 137364], 95.00th=[ 256902],
  5797. | 99.00th=[ 513803], 99.50th=[ 599786], 99.90th=[ 859833],
  5798. | 99.95th=[ 868221], 99.99th=[1317012]
  5799. bw ( KiB/s): min= 0, max= 6345, per=72.92%, avg=4376.00, stdev=2784.59, samples=2
  5800. iops : min= 0, max= 99, avg=68.00, stdev=43.84, samples=2
  5801. lat (nsec) : 750=7.50%, 1000=11.70%
  5802. lat (usec) : 2=14.09%, 4=0.07%, 10=0.43%, 20=1.12%, 50=0.98%
  5803. lat (usec) : 100=0.71%, 250=1.18%, 500=0.66%, 750=0.73%, 1000=2.00%
  5804. lat (msec) : 2=15.26%, 4=11.44%, 10=7.63%, 20=4.57%, 50=5.46%
  5805. lat (msec) : 100=5.42%, 250=5.95%, 500=2.44%, 750=0.60%, 1000=0.10%
  5806. cpu : usr=0.11%, sys=0.38%, ctx=18225, majf=0, minf=38
  5807. IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  5808. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  5809. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  5810. issued rwts: total=8083,8206,0,0 short=0,0,0,0 dropped=0,0,0,0
  5811. latency : target=0, window=0, percentile=100.00%, depth=4
  5812.  
  5813. Run status group 0 (all jobs):
  5814. READ: bw=18.8MiB/s (19.7MB/s), 4309KiB/s-14.6MiB/s (4413kB/s-15.3MB/s), io=2264MiB (2374MB), run=120043-120651msec
  5815. WRITE: bw=6002KiB/s (6146kB/s), 1719KiB/s-4377KiB/s (1760kB/s-4482kB/s), io=719MiB (754MB), run=120043-122670msec
  5816.  
  5817. Disk stats (read/write):
  5818. rbd257: ios=6205/11856, merge=0/2, ticks=596108/895995, in_queue=1483029, util=6.30%
  5819. <example-benchmark-fio-server-473-benchmark-s88vn>
  5820. randwrite: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:35:01 2019
  5821. write: IOPS=26, BW=1692KiB/s (1733kB/s)(200MiB/121075msec)
  5822. slat (nsec): min=0, max=225666k, avg=347336.80, stdev=4842478.92
  5823. clat (nsec): min=0, max=3886.0M, avg=150346798.75, stdev=285940322.20
  5824. lat (nsec): min=0, max=3886.2M, avg=150695339.12, stdev=285937535.10
  5825. clat percentiles (usec):
  5826. | 1.00th=[ 1156], 5.00th=[ 1778], 10.00th=[ 2442],
  5827. | 20.00th=[ 3752], 30.00th=[ 5276], 40.00th=[ 7439],
  5828. | 50.00th=[ 10814], 60.00th=[ 17433], 70.00th=[ 54789],
  5829. | 80.00th=[ 304088], 90.00th=[ 566232], 95.00th=[ 750781],
  5830. | 99.00th=[1019216], 99.50th=[1182794], 99.90th=[3506439],
  5831. | 99.95th=[3841983], 99.99th=[3875537]
  5832. bw ( KiB/s): min= 0, max= 2070, per=32.19%, avg=1702.50, stdev=519.72, samples=2
  5833. iops : min= 0, max= 32, avg=26.00, stdev= 8.49, samples=2
  5834. lat (usec) : 500=0.03%, 1000=0.28%
  5835. lat (msec) : 2=6.35%, 4=15.57%, 10=25.95%, 20=13.91%, 50=7.60%
  5836. lat (msec) : 100=2.03%, 250=4.97%, 500=12.16%, 750=6.22%, 1000=3.91%
  5837. cpu : usr=0.04%, sys=0.11%, ctx=5496, majf=0, minf=31
  5838. IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  5839. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  5840. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  5841. issued rwts: total=0,3198,0,0 short=0,0,0,0 dropped=0,0,0,0
  5842. latency : target=0, window=0, percentile=100.00%, depth=4
  5843. read: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:35:01 2019
  5844. read: IOPS=209, BW=13.1MiB/s (13.7MB/s)(1579MiB/120646msec)
  5845. slat (nsec): min=0, max=10947k, avg=35755.18, stdev=179088.85
  5846. clat (nsec): min=0, max=1843.5M, avg=19068443.95, stdev=112056013.01
  5847. lat (nsec): min=0, max=1843.5M, avg=19105175.28, stdev=112070145.59
  5848. clat percentiles (nsec):
  5849. | 1.00th=[ 716], 5.00th=[ 724], 10.00th=[ 740],
  5850. | 20.00th=[ 772], 30.00th=[ 796], 40.00th=[ 844],
  5851. | 50.00th=[ 884], 60.00th=[ 964], 70.00th=[ 1272],
  5852. | 80.00th=[ 839680], 90.00th=[ 3719168], 95.00th=[ 23461888],
  5853. | 99.00th=[ 734003200], 99.50th=[ 868220928], 99.90th=[1132462080],
  5854. | 99.95th=[1266679808], 99.99th=[1619001344]
  5855. bw ( KiB/s): min= 0, max=14679, per=79.45%, avg=13458.00, stdev=1726.75, samples=2
  5856. iops : min= 0, max= 229, avg=210.00, stdev=26.87, samples=2
  5857. lat (nsec) : 750=11.34%, 1000=49.95%
  5858. lat (usec) : 2=15.71%, 4=0.21%, 10=0.19%, 20=1.02%, 50=0.23%
  5859. lat (usec) : 100=0.12%, 250=0.11%, 500=0.14%, 750=0.47%, 1000=1.90%
  5860. lat (msec) : 2=4.98%, 4=3.97%, 10=2.82%, 20=1.45%, 50=1.75%
  5861. lat (msec) : 100=0.66%, 250=0.52%, 500=0.72%, 750=0.84%, 1000=0.71%
  5862. cpu : usr=0.07%, sys=0.28%, ctx=10686, majf=0, minf=30
  5863. IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  5864. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  5865. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  5866. issued rwts: total=25262,0,0,0 short=0,0,0,0 dropped=0,0,0,0
  5867. latency : target=0, window=0, percentile=100.00%, depth=4
  5868. readwrite: (groupid=0, jobs=1): err= 0: pid=14: Wed May 8 20:35:01 2019
  5869. read: IOPS=55, BW=3556KiB/s (3641kB/s)(417MiB/120042msec)
  5870. slat (nsec): min=0, max=1662.4k, avg=27905.04, stdev=68083.88
  5871. clat (nsec): min=0, max=930076k, avg=7187406.33, stdev=38705839.51
  5872. lat (nsec): min=0, max=930119k, avg=7216297.76, stdev=38712737.80
  5873. clat percentiles (nsec):
  5874. | 1.00th=[ 716], 5.00th=[ 724], 10.00th=[ 732],
  5875. | 20.00th=[ 780], 30.00th=[ 868], 40.00th=[ 988],
  5876. | 50.00th=[ 1144], 60.00th=[ 1304], 70.00th=[ 15424],
  5877. | 80.00th=[ 765952], 90.00th=[ 4079616], 95.00th=[ 39059456],
  5878. | 99.00th=[183500800], 99.50th=[283115520], 99.90th=[591396864],
  5879. | 99.95th=[675282944], 99.99th=[926941184]
  5880. bw ( KiB/s): min= 0, max= 5559, per=20.99%, avg=3556.50, stdev=2831.96, samples=2
  5881. iops : min= 0, max= 86, avg=55.00, stdev=43.84, samples=2
  5882. write: IOPS=56, BW=3628KiB/s (3715kB/s)(425MiB/120042msec)
  5883. slat (nsec): min=0, max=6981.7k, avg=189076.12, stdev=241129.58
  5884. clat (nsec): min=0, max=1469.3M, avg=63533638.70, stdev=133759133.98
  5885. lat (nsec): min=0, max=1469.4M, avg=63723896.57, stdev=133715671.18
  5886. clat percentiles (usec):
  5887. | 1.00th=[ 988], 5.00th=[ 1172], 10.00th=[ 1336],
  5888. | 20.00th=[ 1647], 30.00th=[ 2147], 40.00th=[ 3130],
  5889. | 50.00th=[ 5932], 60.00th=[ 16712], 70.00th=[ 44303],
  5890. | 80.00th=[ 96994], 90.00th=[ 162530], 95.00th=[ 329253],
  5891. | 99.00th=[ 700449], 99.50th=[ 801113], 99.90th=[ 952108],
  5892. | 99.95th=[1098908], 99.99th=[1468007]
  5893. bw ( KiB/s): min= 0, max= 5653, per=68.58%, avg=3627.00, stdev=2865.20, samples=2
  5894. iops : min= 0, max= 88, avg=56.50, stdev=44.55, samples=2
  5895. lat (nsec) : 750=8.09%, 1000=12.14%
  5896. lat (usec) : 2=13.44%, 4=0.07%, 10=0.36%, 20=1.26%, 50=0.78%
  5897. lat (usec) : 100=1.02%, 250=1.04%, 500=0.65%, 750=0.75%, 1000=1.96%
  5898. lat (msec) : 2=15.50%, 4=10.12%, 10=5.89%, 20=4.30%, 50=6.07%
  5899. lat (msec) : 100=5.89%, 250=7.10%, 500=2.34%, 750=0.87%, 1000=0.36%
  5900. cpu : usr=0.07%, sys=0.32%, ctx=14974, majf=0, minf=34
  5901. IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  5902. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  5903. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  5904. issued rwts: total=6669,6802,0,0 short=0,0,0,0 dropped=0,0,0,0
  5905. latency : target=0, window=0, percentile=100.00%, depth=4
  5906.  
  5907. Run status group 0 (all jobs):
  5908. READ: bw=16.5MiB/s (17.3MB/s), 3556KiB/s-13.1MiB/s (3641kB/s-13.7MB/s), io=1996MiB (2093MB), run=120042-120646msec
  5909. WRITE: bw=5289KiB/s (5416kB/s), 1692KiB/s-3628KiB/s (1733kB/s-3715kB/s), io=625MiB (656MB), run=120042-121075msec
  5910.  
  5911. Disk stats (read/write):
  5912. rbd146: ios=7112/10366, merge=0/2, ticks=570356/925505, in_queue=1487123, util=5.21%
  5913. <example-benchmark-fio-server-663-benchmark-7t28d>
  5914. randwrite: (groupid=0, jobs=1): err= 0: pid=11: Wed May 8 20:35:01 2019
  5915. write: IOPS=36, BW=2338KiB/s (2394kB/s)(274MiB/120024msec)
  5916. slat (nsec): min=0, max=1108.8M, avg=1563338.65, stdev=34895725.20
  5917. clat (nsec): min=0, max=2157.2M, avg=108691853.62, stdev=172953769.21
  5918. lat (nsec): min=0, max=2157.2M, avg=110458971.82, stdev=181332612.36
  5919. clat percentiles (usec):
  5920. | 1.00th=[ 1012], 5.00th=[ 1254], 10.00th=[ 1483],
  5921. | 20.00th=[ 2040], 30.00th=[ 3425], 40.00th=[ 9634],
  5922. | 50.00th=[ 51119], 60.00th=[ 84411], 70.00th=[ 108528],
  5923. | 80.00th=[ 143655], 90.00th=[ 346031], 95.00th=[ 488637],
  5924. | 99.00th=[ 834667], 99.50th=[ 910164], 99.90th=[1115685],
  5925. | 99.95th=[1182794], 99.99th=[2164261]
  5926. bw ( KiB/s): min= 0, max= 3665, per=35.50%, avg=2337.00, stdev=1878.08, samples=2
  5927. iops : min= 0, max= 57, avg=36.00, stdev=29.70, samples=2
  5928. lat (usec) : 750=0.07%, 1000=0.80%
  5929. lat (msec) : 2=18.44%, 4=12.71%, 10=8.33%, 20=4.41%, 50=5.02%
  5930. lat (msec) : 100=16.80%, 250=20.82%, 500=7.90%, 750=3.33%, 1000=1.21%
  5931. cpu : usr=0.05%, sys=0.15%, ctx=8296, majf=0, minf=26
  5932. IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  5933. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  5934. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  5935. issued rwts: total=0,4381,0,0 short=0,0,0,0 dropped=0,0,0,0
  5936. latency : target=0, window=0, percentile=100.00%, depth=4
  5937. read: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:35:01 2019
  5938. read: IOPS=505, BW=31.4MiB/s (32.9MB/s)(3763MiB/120006msec)
  5939. slat (nsec): min=0, max=1108.8M, avg=95001.81, stdev=8079368.15
  5940. clat (nsec): min=0, max=2157.1M, avg=7861988.18, stdev=51822844.43
  5941. lat (nsec): min=0, max=2157.2M, avg=7957796.46, stdev=52813178.22
  5942. clat percentiles (nsec):
  5943. | 1.00th=[ 708], 5.00th=[ 716], 10.00th=[ 716],
  5944. | 20.00th=[ 724], 30.00th=[ 732], 40.00th=[ 756],
  5945. | 50.00th=[ 836], 60.00th=[ 892], 70.00th=[ 1112],
  5946. | 80.00th=[ 8768], 90.00th=[ 45312], 95.00th=[ 26869760],
  5947. | 99.00th=[ 261095424], 99.50th=[ 421527552], 99.90th=[ 658505728],
  5948. | 99.95th=[ 767557632], 99.99th=[1098907648]
  5949. bw ( KiB/s): min= 0, max=46009, per=88.48%, avg=32107.50, stdev=19659.69, samples=2
  5950. iops : min= 0, max= 727, avg=505.50, stdev=313.25, samples=2
  5951. lat (nsec) : 750=38.28%, 1000=29.30%
  5952. lat (usec) : 2=7.41%, 4=0.05%, 10=6.17%, 20=4.86%, 50=4.02%
  5953. lat (usec) : 100=0.24%, 250=0.07%, 500=0.08%, 750=0.42%, 1000=1.00%
  5954. lat (msec) : 2=1.37%, 4=0.76%, 10=0.40%, 20=0.29%, 50=1.86%
  5955. lat (msec) : 100=1.61%, 250=0.81%, 500=0.77%, 750=0.20%, 1000=0.05%
  5956. cpu : usr=0.14%, sys=0.48%, ctx=11673, majf=0, minf=105
  5957. IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  5958. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  5959. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  5960. issued rwts: total=60714,0,0,0 short=0,0,0,0 dropped=0,0,0,0
  5961. latency : target=0, window=0, percentile=100.00%, depth=4
  5962. readwrite: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:35:01 2019
  5963. read: IOPS=65, BW=4187KiB/s (4288kB/s)(491MiB/120038msec)
  5964. slat (nsec): min=0, max=358717k, avg=71754.97, stdev=4047648.69
  5965. clat (nsec): min=0, max=959407k, avg=3762438.41, stdev=26003167.77
  5966. lat (nsec): min=0, max=959440k, avg=3835087.41, stdev=26316511.26
  5967. clat percentiles (nsec):
  5968. | 1.00th=[ 708], 5.00th=[ 716], 10.00th=[ 724],
  5969. | 20.00th=[ 748], 30.00th=[ 844], 40.00th=[ 932],
  5970. | 50.00th=[ 1080], 60.00th=[ 1256], 70.00th=[ 13504],
  5971. | 80.00th=[ 444416], 90.00th=[ 1957888], 95.00th=[ 8978432],
  5972. | 99.00th=[ 79167488], 99.50th=[152043520], 99.90th=[442499072],
  5973. | 99.95th=[497025024], 99.99th=[960495616]
  5974. bw ( KiB/s): min= 0, max= 7293, per=11.54%, avg=4188.00, stdev=4391.13, samples=2
  5975. iops : min= 0, max= 113, avg=64.50, stdev=68.59, samples=2
  5976. write: IOPS=66, BW=4246KiB/s (4347kB/s)(498MiB/120038msec)
  5977. slat (nsec): min=0, max=616615k, avg=277859.02, stdev=7123826.65
  5978. clat (nsec): min=0, max=1058.7M, avg=56496877.05, stdev=127492223.13
  5979. lat (nsec): min=0, max=1058.8M, avg=56775886.69, stdev=127619809.89
  5980. clat percentiles (usec):
  5981. | 1.00th=[ 996], 5.00th=[ 1188], 10.00th=[ 1336],
  5982. | 20.00th=[ 1614], 30.00th=[ 1958], 40.00th=[ 2540],
  5983. | 50.00th=[ 3884], 60.00th=[ 7111], 70.00th=[ 21627],
  5984. | 80.00th=[ 85459], 90.00th=[ 143655], 95.00th=[ 350225],
  5985. | 99.00th=[ 641729], 99.50th=[ 775947], 99.90th=[ 943719],
  5986. | 99.95th=[ 943719], 99.99th=[1061159]
  5987. bw ( KiB/s): min= 0, max= 7357, per=64.48%, avg=4244.50, stdev=4401.74, samples=2
  5988. iops : min= 0, max= 114, avg=65.50, stdev=68.59, samples=2
  5989. lat (nsec) : 750=10.03%, 1000=12.38%
  5990. lat (usec) : 2=11.41%, 4=0.06%, 10=0.43%, 20=1.47%, 50=0.82%
  5991. lat (usec) : 100=0.96%, 250=1.24%, 500=1.10%, 750=0.87%, 1000=2.21%
  5992. lat (msec) : 2=17.35%, 4=11.41%, 10=7.51%, 20=3.66%, 50=3.38%
  5993. lat (msec) : 100=4.86%, 250=5.46%, 500=2.27%, 750=0.89%, 1000=0.26%
  5994. cpu : usr=0.08%, sys=0.35%, ctx=17449, majf=0, minf=33
  5995. IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  5996. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  5997. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  5998. issued rwts: total=7855,7959,0,0 short=0,0,0,0 dropped=0,0,0,0
  5999. latency : target=0, window=0, percentile=100.00%, depth=4
  6000.  
  6001. Run status group 0 (all jobs):
  6002. READ: bw=35.4MiB/s (37.2MB/s), 4187KiB/s-31.4MiB/s (4288kB/s-32.9MB/s), io=4254MiB (4460MB), run=120006-120038msec
  6003. WRITE: bw=6584KiB/s (6742kB/s), 2338KiB/s-4246KiB/s (2394kB/s-4347kB/s), io=772MiB (809MB), run=120024-120038msec
  6004.  
  6005. Disk stats (read/write):
  6006. rbd284: ios=7426/12487, merge=0/2, ticks=505851/806981, in_queue=1302709, util=12.68%
  6007. <example-benchmark-fio-server-37-benchmark-k5hwt>
  6008. randwrite: (groupid=0, jobs=1): err= 0: pid=11: Wed May 8 20:35:01 2019
  6009. write: IOPS=39, BW=2540KiB/s (2601kB/s)(298MiB/120173msec)
  6010. slat (nsec): min=0, max=696899k, avg=411800.47, stdev=10858781.84
  6011. clat (nsec): min=0, max=1250.4M, avg=100680898.92, stdev=165640343.18
  6012. lat (nsec): min=0, max=1250.6M, avg=101093747.11, stdev=165819419.43
  6013. clat percentiles (usec):
  6014. | 1.00th=[ 971], 5.00th=[ 1205], 10.00th=[ 1418],
  6015. | 20.00th=[ 1909], 30.00th=[ 2868], 40.00th=[ 5735],
  6016. | 50.00th=[ 25035], 60.00th=[ 71828], 70.00th=[ 103285],
  6017. | 80.00th=[ 139461], 90.00th=[ 308282], 95.00th=[ 480248],
  6018. | 99.00th=[ 792724], 99.50th=[ 901776], 99.90th=[1115685],
  6019. | 99.95th=[1115685], 99.99th=[1249903]
  6020. bw ( KiB/s): min= 0, max= 2981, per=39.78%, avg=2541.00, stdev=622.25, samples=2
  6021. iops : min= 0, max= 46, avg=39.00, stdev= 9.90, samples=2
  6022. lat (usec) : 250=0.02%, 500=0.04%, 750=0.04%, 1000=1.26%
  6023. lat (msec) : 2=19.87%, 4=14.23%, 10=8.64%, 20=4.51%, 50=6.53%
  6024. lat (msec) : 100=13.91%, 250=18.74%, 500=7.74%, 750=3.34%, 1000=0.97%
  6025. cpu : usr=0.06%, sys=0.17%, ctx=9069, majf=0, minf=27
  6026. IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  6027. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  6028. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  6029. issued rwts: total=0,4766,0,0 short=0,0,0,0 dropped=0,0,0,0
  6030. latency : target=0, window=0, percentile=100.00%, depth=4
  6031. read: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:35:01 2019
  6032. read: IOPS=381, BW=23.8MiB/s (24.9MB/s)(2856MiB/120067msec)
  6033. slat (nsec): min=0, max=408609k, avg=33102.72, stdev=2270301.13
  6034. clat (nsec): min=0, max=1243.1M, avg=10502054.70, stdev=58942956.54
  6035. lat (nsec): min=0, max=1243.2M, avg=10535980.10, stdev=58995317.19
  6036. clat percentiles (nsec):
  6037. | 1.00th=[ 708], 5.00th=[ 708], 10.00th=[ 716],
  6038. | 20.00th=[ 724], 30.00th=[ 732], 40.00th=[ 756],
  6039. | 50.00th=[ 820], 60.00th=[ 876], 70.00th=[ 988],
  6040. | 80.00th=[ 8640], 90.00th=[ 1155072], 95.00th=[ 49020928],
  6041. | 99.00th=[325058560], 99.50th=[467664896], 99.90th=[734003200],
  6042. | 99.95th=[843055104], 99.99th=[960495616]
  6043. bw ( KiB/s): min= 0, max=37141, per=86.65%, avg=24365.50, stdev=18067.29, samples=2
  6044. iops : min= 0, max= 581, avg=381.00, stdev=282.84, samples=2
  6045. lat (nsec) : 750=38.95%, 1000=31.59%
  6046. lat (usec) : 2=9.13%, 4=0.05%, 10=3.93%, 20=2.24%, 50=1.34%
  6047. lat (usec) : 100=0.13%, 250=0.06%, 500=0.12%, 750=0.56%, 1000=1.34%
  6048. lat (msec) : 2=1.89%, 4=1.03%, 10=0.54%, 20=0.37%, 50=1.82%
  6049. lat (msec) : 100=2.31%, 250=1.27%, 500=0.96%, 750=0.29%, 1000=0.08%
  6050. cpu : usr=0.11%, sys=0.38%, ctx=11528, majf=0, minf=31
  6051. IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  6052. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  6053. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  6054. issued rwts: total=45763,0,0,0 short=0,0,0,0 dropped=0,0,0,0
  6055. latency : target=0, window=0, percentile=100.00%, depth=4
  6056. readwrite: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:35:01 2019
  6057. read: IOPS=58, BW=3769KiB/s (3859kB/s)(442MiB/120004msec)
  6058. slat (nsec): min=0, max=156853k, avg=69439.21, stdev=2318254.95
  6059. clat (nsec): min=0, max=912880k, avg=6950330.88, stdev=42236370.38
  6060. lat (nsec): min=0, max=913024k, avg=7020687.52, stdev=42300601.87
  6061. clat percentiles (nsec):
  6062. | 1.00th=[ 716], 5.00th=[ 724], 10.00th=[ 740],
  6063. | 20.00th=[ 804], 30.00th=[ 876], 40.00th=[ 988],
  6064. | 50.00th=[ 1144], 60.00th=[ 1288], 70.00th=[ 14144],
  6065. | 80.00th=[ 716800], 90.00th=[ 2867200], 95.00th=[ 23986176],
  6066. | 99.00th=[217055232], 99.50th=[325058560], 99.90th=[549453824],
  6067. | 99.95th=[641728512], 99.99th=[910163968]
  6068. bw ( KiB/s): min= 0, max= 5579, per=13.40%, avg=3768.00, stdev=2561.14, samples=2
  6069. iops : min= 0, max= 87, avg=58.50, stdev=40.31, samples=2
  6070. write: IOPS=60, BW=3854KiB/s (3946kB/s)(452MiB/120004msec)
  6071. slat (nsec): min=0, max=17335k, avg=183883.83, stdev=342978.00
  6072. clat (nsec): min=0, max=1171.4M, avg=59432543.96, stdev=120575360.93
  6073. lat (nsec): min=0, max=1171.5M, avg=59617587.11, stdev=120535896.17
  6074. clat percentiles (usec):
  6075. | 1.00th=[ 1020], 5.00th=[ 1221], 10.00th=[ 1418],
  6076. | 20.00th=[ 1762], 30.00th=[ 2343], 40.00th=[ 3490],
  6077. | 50.00th=[ 6718], 60.00th=[ 16057], 70.00th=[ 42730],
  6078. | 80.00th=[ 90702], 90.00th=[ 154141], 95.00th=[ 320865],
  6079. | 99.00th=[ 608175], 99.50th=[ 700449], 99.90th=[ 910164],
  6080. | 99.95th=[ 943719], 99.99th=[1166017]
  6081. bw ( KiB/s): min= 0, max= 5643, per=60.29%, avg=3851.50, stdev=2533.56, samples=2
  6082. iops : min= 0, max= 88, avg=60.00, stdev=39.60, samples=2
  6083. lat (nsec) : 750=6.79%, 1000=13.47%
  6084. lat (usec) : 2=13.65%, 4=0.09%, 10=0.23%, 20=1.02%, 50=0.93%
  6085. lat (usec) : 100=0.99%, 250=1.08%, 500=0.86%, 750=0.66%, 1000=2.05%
  6086. lat (msec) : 2=14.38%, 4=10.35%, 10=7.07%, 20=4.86%, 50=5.50%
  6087. lat (msec) : 100=6.33%, 250=5.87%, 500=2.76%, 750=0.91%, 1000=0.14%
  6088. cpu : usr=0.08%, sys=0.34%, ctx=16157, majf=0, minf=31
  6089. IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  6090. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  6091. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  6092. issued rwts: total=7067,7224,0,0 short=0,0,0,0 dropped=0,0,0,0
  6093. latency : target=0, window=0, percentile=100.00%, depth=4
  6094.  
  6095. Run status group 0 (all jobs):
  6096. READ: bw=27.5MiB/s (28.8MB/s), 3769KiB/s-23.8MiB/s (3859kB/s-24.9MB/s), io=3297MiB (3457MB), run=120004-120067msec
  6097. WRITE: bw=6388KiB/s (6541kB/s), 2540KiB/s-3854KiB/s (2601kB/s-3946kB/s), io=750MiB (786MB), run=120004-120173msec
  6098.  
  6099. Disk stats (read/write):
  6100. rbd12: ios=7283/12276, merge=0/1, ticks=539258/835111, in_queue=1364388, util=12.96%
  6101. Jobs: 7 (f=7): [_(2),m(1)][98.8%][r=455M,w=443M][r=7105,w=6923 IOPS][eta 00m:49s] Jobs: 7 (f=7): [_(2),m(1)][98.7%][r=583M,w=562M][r=9104,w=8780 IOPS][eta 00m:48s] <example-benchmark-fio-server-435-benchmark-5cvqg>
  6102. randwrite: (groupid=0, jobs=1): err= 0: pid=11: Wed May 8 20:35:02 2019
  6103. write: IOPS=22, BW=1412KiB/s (1446kB/s)(166MiB/120304msec)
  6104. slat (nsec): min=0, max=777619k, avg=1045993.83, stdev=24696843.15
  6105. clat (nsec): min=0, max=3272.3M, avg=181650449.15, stdev=336408176.45
  6106. lat (nsec): min=0, max=3273.1M, avg=182696909.40, stdev=336849572.89
  6107. clat percentiles (usec):
  6108. | 1.00th=[ 1139], 5.00th=[ 1467], 10.00th=[ 1811],
  6109. | 20.00th=[ 2540], 30.00th=[ 3720], 40.00th=[ 5997],
  6110. | 50.00th=[ 14746], 60.00th=[ 39584], 70.00th=[ 92799],
  6111. | 80.00th=[ 425722], 90.00th=[ 675283], 95.00th=[ 884999],
  6112. | 99.00th=[1367344], 99.50th=[1669333], 99.90th=[2332034],
  6113. | 99.95th=[2902459], 99.99th=[3271558]
  6114. bw ( KiB/s): min= 0, max= 1468, per=23.39%, avg=1412.00, stdev=79.20, samples=2
  6115. iops : min= 0, max= 22, avg=21.50, stdev= 0.71, samples=2
  6116. lat (usec) : 4=0.04%, 50=0.04%, 100=0.04%, 500=0.04%, 1000=0.23%
  6117. lat (msec) : 2=12.79%, 4=18.75%, 10=15.01%, 20=6.79%, 50=8.45%
  6118. lat (msec) : 100=9.02%, 250=6.56%, 500=5.62%, 750=8.26%, 1000=5.62%
  6119. cpu : usr=0.02%, sys=0.11%, ctx=4893, majf=0, minf=27
  6120. IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  6121. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  6122. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  6123. issued rwts: total=0,2651,0,0 short=0,0,0,0 dropped=0,0,0,0
  6124. latency : target=0, window=0, percentile=100.00%, depth=4
  6125. read: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:35:02 2019
  6126. read: IOPS=540, BW=33.6MiB/s (35.3MB/s)(4039MiB/120089msec)
  6127. slat (nsec): min=0, max=682499k, avg=37730.14, stdev=3821435.40
  6128. clat (nsec): min=0, max=3266.9M, avg=7397397.95, stdev=73619786.81
  6129. lat (nsec): min=0, max=3266.9M, avg=7436102.24, stdev=73726006.00
  6130. clat percentiles (nsec):
  6131. | 1.00th=[ 708], 5.00th=[ 716], 10.00th=[ 724],
  6132. | 20.00th=[ 748], 30.00th=[ 780], 40.00th=[ 804],
  6133. | 50.00th=[ 844], 60.00th=[ 868], 70.00th=[ 900],
  6134. | 80.00th=[ 1144], 90.00th=[ 12608], 95.00th=[ 24448],
  6135. | 99.00th=[ 350224384], 99.50th=[ 591396864], 99.90th=[ 977272832],
  6136. | 99.95th=[1249902592], 99.99th=[1837105152]
  6137. bw ( KiB/s): min= 0, max=54965, per=88.02%, avg=34356.50, stdev=29144.82, samples=2
  6138. iops : min= 0, max= 863, avg=538.50, stdev=458.91, samples=2
  6139. lat (nsec) : 750=20.50%, 1000=57.38%
  6140. lat (usec) : 2=5.04%, 4=0.15%, 10=0.99%, 20=9.96%, 50=1.54%
  6141. lat (usec) : 100=0.15%, 250=0.11%, 500=0.09%, 750=0.15%, 1000=0.35%
  6142. lat (msec) : 2=0.72%, 4=0.49%, 10=0.29%, 20=0.17%, 50=0.41%
  6143. lat (msec) : 100=0.38%, 250=0.09%, 500=0.39%, 750=0.37%, 1000=0.19%
  6144. cpu : usr=0.13%, sys=0.52%, ctx=5705, majf=0, minf=34
  6145. IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  6146. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  6147. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  6148. issued rwts: total=64882,0,0,0 short=0,0,0,0 dropped=0,0,0,0
  6149. latency : target=0, window=0, percentile=100.00%, depth=4
  6150. readwrite: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:35:02 2019
  6151. read: IOPS=71, BW=4595KiB/s (4705kB/s)(538MiB/120003msec)
  6152. slat (nsec): min=0, max=3033.9k, avg=26259.86, stdev=80062.73
  6153. clat (nsec): min=0, max=687222k, avg=4836633.32, stdev=32108832.60
  6154. lat (nsec): min=0, max=687261k, avg=4863797.12, stdev=32118036.94
  6155. clat percentiles (nsec):
  6156. | 1.00th=[ 716], 5.00th=[ 724], 10.00th=[ 740],
  6157. | 20.00th=[ 764], 30.00th=[ 860], 40.00th=[ 972],
  6158. | 50.00th=[ 1160], 60.00th=[ 1352], 70.00th=[ 19072],
  6159. | 80.00th=[ 460800], 90.00th=[ 1597440], 95.00th=[ 7962624],
  6160. | 99.00th=[130547712], 99.50th=[238026752], 99.90th=[434110464],
  6161. | 99.95th=[526385152], 99.99th=[683671552]
  6162. bw ( KiB/s): min= 0, max= 8049, per=11.77%, avg=4593.50, stdev=4886.81, samples=2
  6163. iops : min= 0, max= 125, avg=71.00, stdev=76.37, samples=2
  6164. write: IOPS=72, BW=4638KiB/s (4750kB/s)(544MiB/120003msec)
  6165. slat (nsec): min=0, max=9475.9k, avg=183809.96, stdev=263225.44
  6166. clat (nsec): min=0, max=998063k, avg=50489057.96, stdev=117048370.19
  6167. lat (nsec): min=0, max=998145k, avg=50674028.51, stdev=117014939.57
  6168. clat percentiles (usec):
  6169. | 1.00th=[ 963], 5.00th=[ 1139], 10.00th=[ 1287], 20.00th=[ 1532],
  6170. | 30.00th=[ 1876], 40.00th=[ 2507], 50.00th=[ 3720], 60.00th=[ 7439],
  6171. | 70.00th=[ 21627], 80.00th=[ 71828], 90.00th=[121111], 95.00th=[312476],
  6172. | 99.00th=[574620], 99.50th=[683672], 99.90th=[926942], 99.95th=[935330],
  6173. | 99.99th=[994051]
  6174. bw ( KiB/s): min= 0, max= 8099, per=76.81%, avg=4637.50, stdev=4895.30, samples=2
  6175. iops : min= 0, max= 126, avg=72.00, stdev=76.37, samples=2
  6176. lat (nsec) : 750=7.98%, 1000=12.81%
  6177. lat (usec) : 2=12.09%, 4=0.05%, 10=0.40%, 20=1.62%, 50=1.02%
  6178. lat (usec) : 100=1.18%, 250=1.75%, 500=1.09%, 750=0.99%, 1000=2.65%
  6179. lat (msec) : 2=18.00%, 4=10.84%, 10=6.70%, 20=3.30%, 50=4.31%
  6180. lat (msec) : 100=5.60%, 250=4.41%, 500=2.16%, 750=0.86%, 1000=0.18%
  6181. cpu : usr=0.09%, sys=0.40%, ctx=18391, majf=0, minf=37
  6182. IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  6183. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  6184. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  6185. issued rwts: total=8615,8694,0,0 short=0,0,0,0 dropped=0,0,0,0
  6186. latency : target=0, window=0, percentile=100.00%, depth=4
  6187.  
  6188. Run status group 0 (all jobs):
  6189. READ: bw=38.1MiB/s (39.0MB/s), 4595KiB/s-33.6MiB/s (4705kB/s-35.3MB/s), io=4578MiB (4800MB), run=120003-120089msec
  6190. WRITE: bw=6039KiB/s (6183kB/s), 1412KiB/s-4638KiB/s (1446kB/s-4750kB/s), io=709MiB (744MB), run=120003-120304msec
  6191.  
  6192. Disk stats (read/write):
  6193. rbd110: ios=4843/11517, merge=0/1, ticks=536620/888979, in_queue=1417352, util=7.88%
  6194. Jobs: 6 (f=6): [_(2),m(1)][98.6%][r=487M,w=494M][r=7605,w=7724 IOPS][eta 00m:47s] Jobs: 6 (f=6): [_(2),m(1)][98.6%][r=439M,w=449M][r=6851,w=7007 IOPS][eta 00m:45s] Jobs: 6 (f=6): [_(2),m(1)][98.6%][r=483M,w=472M][r=7539,w=7372 IOPS][eta 00m:45s] Jobs: 6 (f=6): [_(2),m(1)][98.7%][r=423M,w=420M][r=6605,w=6560 IOPS][eta 00m:44s] Jobs: 6 (f=6): [_(2),m(1)][98.7%][r=454M,w=452M][r=7088,w=7060 IOPS][eta 00m:42s] Jobs: 6 (f=6): [_(2),m(1)][98.8%][r=175M,w=169M][r=2727,w=2633 IOPS][eta 00m:41s] Jobs: 6 (f=6): [_(2),m(1)][98.8%][r=452M,w=453M][r=7063,w=7085 IOPS][eta 00m:41s] Jobs: 6 (f=6): [_(2),m(1)][98.8%][r=443M,w=438M][r=6916,w=6842 IOPS][eta 00m:40s] <example-benchmark-fio-server-503-benchmark-6fx6x>
  6195. randwrite: (groupid=0, jobs=1): err= 0: pid=11: Wed May 8 20:35:10 2019
  6196. write: IOPS=57, BW=3665KiB/s (3753kB/s)(430MiB/120026msec)
  6197. slat (nsec): min=0, max=622630k, avg=345543.41, stdev=9113718.13
  6198. clat (nsec): min=0, max=1384.4M, avg=69695686.91, stdev=136504038.48
  6199. lat (nsec): min=0, max=1384.4M, avg=70042389.89, stdev=136680584.74
  6200. clat percentiles (usec):
  6201. | 1.00th=[ 938], 5.00th=[ 1123], 10.00th=[ 1287],
  6202. | 20.00th=[ 1565], 30.00th=[ 1958], 40.00th=[ 2638],
  6203. | 50.00th=[ 4293], 60.00th=[ 16319], 70.00th=[ 63701],
  6204. | 80.00th=[ 101188], 90.00th=[ 208667], 95.00th=[ 371196],
  6205. | 99.00th=[ 675283], 99.50th=[ 759170], 99.90th=[ 918553],
  6206. | 99.95th=[ 935330], 99.99th=[1384121]
  6207. bw ( KiB/s): min= 0, max= 5002, per=29.12%, avg=3663.00, stdev=1893.63, samples=2
  6208. iops : min= 0, max= 78, avg=57.00, stdev=29.70, samples=2
  6209. lat (usec) : 4=0.01%, 500=0.01%, 750=0.09%, 1000=1.89%
  6210. lat (msec) : 2=28.85%, 4=18.14%, 10=8.84%, 20=2.87%, 50=5.82%
  6211. lat (msec) : 100=13.11%, 250=11.95%, 500=5.62%, 750=2.21%, 1000=0.61%
  6212. cpu : usr=0.07%, sys=0.24%, ctx=12882, majf=0, minf=27
  6213. IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  6214. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  6215. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  6216. issued rwts: total=0,6870,0,0 short=0,0,0,0 dropped=0,0,0,0
  6217. latency : target=0, window=0, percentile=100.00%, depth=4
  6218. read: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:35:10 2019
  6219. read: IOPS=363, BW=22.7MiB/s (23.8MB/s)(2725MiB/120010msec)
  6220. slat (nsec): min=0, max=597722k, avg=43845.37, stdev=3451713.75
  6221. clat (nsec): min=0, max=952019k, avg=10981275.55, stdev=56162989.38
  6222. lat (nsec): min=0, max=952050k, avg=11025965.24, stdev=56275129.48
  6223. clat percentiles (nsec):
  6224. | 1.00th=[ 700], 5.00th=[ 708], 10.00th=[ 716],
  6225. | 20.00th=[ 724], 30.00th=[ 732], 40.00th=[ 772],
  6226. | 50.00th=[ 836], 60.00th=[ 884], 70.00th=[ 1048],
  6227. | 80.00th=[ 8768], 90.00th=[ 3489792], 95.00th=[ 52166656],
  6228. | 99.00th=[312475648], 99.50th=[446693376], 99.90th=[683671552],
  6229. | 99.95th=[759169024], 99.99th=[843055104]
  6230. bw ( KiB/s): min= 0, max=33364, per=72.38%, avg=23248.00, stdev=14306.18, samples=2
  6231. iops : min= 0, max= 522, avg=363.50, stdev=224.15, samples=2
  6232. lat (nsec) : 750=35.04%, 1000=33.44%
  6233. lat (usec) : 2=11.20%, 4=0.06%, 10=0.69%, 20=0.35%, 50=0.13%
  6234. lat (usec) : 100=0.09%, 250=0.03%, 500=0.41%, 750=1.28%, 1000=2.38%
  6235. lat (msec) : 2=3.27%, 4=1.88%, 10=0.96%, 20=0.58%, 50=2.99%
  6236. lat (msec) : 100=2.50%, 250=1.42%, 500=0.98%, 750=0.28%, 1000=0.05%
  6237. cpu : usr=0.12%, sys=0.40%, ctx=16531, majf=0, minf=37
  6238. IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  6239. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  6240. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  6241. issued rwts: total=43643,0,0,0 short=0,0,0,0 dropped=0,0,0,0
  6242. latency : target=0, window=0, percentile=100.00%, depth=4
  6243. readwrite: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:35:10 2019
  6244. read: IOPS=138, BW=8872KiB/s (9085kB/s)(1040MiB/120001msec)
  6245. slat (nsec): min=0, max=423034k, avg=64612.42, stdev=3280735.86
  6246. clat (nsec): min=0, max=1022.7M, avg=3889780.33, stdev=26493181.11
  6247. lat (nsec): min=0, max=1022.8M, avg=3955401.07, stdev=26697984.84
  6248. clat percentiles (nsec):
  6249. | 1.00th=[ 716], 5.00th=[ 740], 10.00th=[ 772],
  6250. | 20.00th=[ 900], 30.00th=[ 1128], 40.00th=[ 1544],
  6251. | 50.00th=[ 432128], 60.00th=[ 774144], 70.00th=[ 921600],
  6252. | 80.00th=[ 1269760], 90.00th=[ 2637824], 95.00th=[ 5603328],
  6253. | 99.00th=[ 79167488], 99.50th=[ 183500800], 99.90th=[ 379584512],
  6254. | 99.95th=[ 413138944], 99.99th=[1019215872]
  6255. bw ( KiB/s): min= 0, max= 2046, per=6.37%, avg=2046.00, stdev= 0.00, samples=1
  6256. iops : min= 0, max= 32, avg=32.00, stdev= 0.00, samples=1
  6257. write: IOPS=139, BW=8917KiB/s (9131kB/s)(1045MiB/120001msec)
  6258. slat (nsec): min=0, max=360725k, avg=196120.52, stdev=2797797.23
  6259. clat (nsec): min=0, max=951017k, avg=24667440.35, stdev=78295914.29
  6260. lat (nsec): min=0, max=951106k, avg=24864692.52, stdev=78319107.19
  6261. clat percentiles (usec):
  6262. | 1.00th=[ 914], 5.00th=[ 1057], 10.00th=[ 1139], 20.00th=[ 1270],
  6263. | 30.00th=[ 1401], 40.00th=[ 1565], 50.00th=[ 1860], 60.00th=[ 2507],
  6264. | 70.00th=[ 3916], 80.00th=[ 8455], 90.00th=[ 63701], 95.00th=[128451],
  6265. | 99.00th=[459277], 99.50th=[534774], 99.90th=[717226], 99.95th=[784335],
  6266. | 99.99th=[943719]
  6267. bw ( KiB/s): min= 0, max= 2136, per=16.98%, avg=2136.00, stdev= 0.00, samples=1
  6268. iops : min= 0, max= 33, avg=33.00, stdev= 0.00, samples=1
  6269. lat (nsec) : 750=3.87%, 1000=8.51%
  6270. lat (usec) : 2=7.98%, 4=0.03%, 10=0.07%, 20=0.29%, 50=0.43%
  6271. lat (usec) : 100=0.41%, 250=0.92%, 500=2.95%, 750=3.83%, 1000=8.77%
  6272. lat (msec) : 2=31.61%, 4=12.24%, 10=6.91%, 20=2.28%, 50=2.35%
  6273. lat (msec) : 100=2.80%, 250=2.19%, 500=1.23%, 750=0.30%, 1000=0.04%
  6274. cpu : usr=0.15%, sys=0.66%, ctx=47021, majf=0, minf=32
  6275. IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  6276. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  6277. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  6278. issued rwts: total=16637,16716,0,0 short=0,0,0,0 dropped=0,0,0,0
  6279. latency : target=0, window=0, percentile=100.00%, depth=4
  6280.  
  6281. Run status group 0 (all jobs):
  6282. READ: bw=31.4MiB/s (32.9MB/s), 8872KiB/s-22.7MiB/s (9085kB/s-23.8MB/s), io=3764MiB (3947MB), run=120001-120010msec
  6283. WRITE: bw=12.3MiB/s (12.9MB/s), 3665KiB/s-8917KiB/s (3753kB/s-9131kB/s), io=1475MiB (1546MB), run=120001-120026msec
  6284.  
  6285. Disk stats (read/write):
  6286. rbd208: ios=19607/24303, merge=0/1, ticks=580676/840563, in_queue=1399084, util=21.89%
  6287. Jobs: 5 (f=5): [_(2),m(1)][98.6%][r=401M,w=400M][r=6270,w=6243 IOPS][eta 00m:39s] Jobs: 5 (f=5): [_(2),m(1)][98.7%][r=402M,w=388M][r=6277,w=6068 IOPS][eta 00m:38s] Jobs: 5 (f=5): [_(2),m(1)][98.7%][r=341M,w=330M][r=5323,w=5149 IOPS][eta 00m:37s] Jobs: 5 (f=5): [_(2),m(1)][98.7%][r=372M,w=365M][r=5806,w=5706 IOPS][eta 00m:36s] Jobs: 5 (f=5): [_(2),m(1)][98.8%][r=390M,w=381M][r=6092,w=5949 IOPS][eta 00m:34s] <example-benchmark-fio-server-668-benchmark-bhqmd>
  6288. randwrite: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:35:15 2019
  6289. write: IOPS=15, BW=1003KiB/s (1027kB/s)(119MiB/121774msec)
  6290. slat (nsec): min=0, max=800208k, avg=1315104.52, stdev=25820013.84
  6291. clat (nsec): min=0, max=1877.4M, avg=252516056.14, stdev=421021517.60
  6292. lat (nsec): min=0, max=1877.8M, avg=253830909.29, stdev=421422000.84
  6293. clat percentiles (usec):
  6294. | 1.00th=[ 1020], 5.00th=[ 1467], 10.00th=[ 1860],
  6295. | 20.00th=[ 2573], 30.00th=[ 3654], 40.00th=[ 5276],
  6296. | 50.00th=[ 9110], 60.00th=[ 26084], 70.00th=[ 242222],
  6297. | 80.00th=[ 574620], 90.00th=[ 977273], 95.00th=[1233126],
  6298. | 99.00th=[1484784], 99.50th=[1635779], 99.90th=[1870660],
  6299. | 99.95th=[1870660], 99.99th=[1870660]
  6300. bw ( KiB/s): min= 0, max= 1290, per=7.19%, avg=1011.50, stdev=393.86, samples=2
  6301. iops : min= 0, max= 20, avg=15.50, stdev= 6.36, samples=2
  6302. lat (usec) : 4=0.05%, 500=0.05%, 1000=0.73%
  6303. lat (msec) : 2=10.97%, 4=20.89%, 10=18.43%, 20=6.72%, 50=6.98%
  6304. lat (msec) : 100=3.78%, 250=1.63%, 500=8.61%, 750=6.09%, 1000=5.83%
  6305. cpu : usr=0.01%, sys=0.08%, ctx=3340, majf=0, minf=29
  6306. IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  6307. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  6308. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  6309. issued rwts: total=0,1905,0,0 short=0,0,0,0 dropped=0,0,0,0
  6310. latency : target=0, window=0, percentile=100.00%, depth=4
  6311. read: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:35:15 2019
  6312. read: IOPS=267, BW=16.7MiB/s (17.5MB/s)(2005MiB/120115msec)
  6313. slat (nsec): min=0, max=680756k, avg=56808.85, stdev=4611819.23
  6314. clat (nsec): min=0, max=1845.9M, avg=14932473.14, stdev=106837887.10
  6315. lat (nsec): min=0, max=1845.0M, avg=14990289.36, stdev=106948423.50
  6316. clat percentiles (nsec):
  6317. | 1.00th=[ 708], 5.00th=[ 716], 10.00th=[ 724],
  6318. | 20.00th=[ 764], 30.00th=[ 780], 40.00th=[ 796],
  6319. | 50.00th=[ 836], 60.00th=[ 868], 70.00th=[ 900],
  6320. | 80.00th=[ 1004], 90.00th=[ 12480], 95.00th=[ 2768896],
  6321. | 99.00th=[ 692060160], 99.50th=[ 851443712], 99.90th=[1182793728],
  6322. | 99.95th=[1283457024], 99.99th=[1619001344]
  6323. bw ( KiB/s): min= 0, max=20554, per=67.59%, avg=20554.00, stdev= 0.00, samples=1
  6324. iops : min= 0, max= 321, avg=321.00, stdev= 0.00, samples=1
  6325. lat (nsec) : 750=18.62%, 1000=61.37%
  6326. lat (usec) : 2=9.01%, 4=0.20%, 10=0.18%, 20=1.19%, 50=0.23%
  6327. lat (usec) : 100=0.11%, 250=0.09%, 500=0.09%, 750=0.24%, 1000=0.82%
  6328. lat (msec) : 2=2.05%, 4=1.55%, 10=1.02%, 20=0.52%, 50=0.48%
  6329. lat (msec) : 100=0.19%, 250=0.12%, 500=0.23%, 750=0.97%, 1000=0.50%
  6330. cpu : usr=0.07%, sys=0.29%, ctx=5937, majf=0, minf=30
  6331. IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  6332. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  6333. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  6334. issued rwts: total=32116,0,0,0 short=0,0,0,0 dropped=0,0,0,0
  6335. latency : target=0, window=0, percentile=100.00%, depth=4
  6336. readwrite: (groupid=0, jobs=1): err= 0: pid=14: Wed May 8 20:35:15 2019
  6337. read: IOPS=208, BW=13.0MiB/s (13.7MB/s)(1562MiB/120002msec)
  6338. slat (nsec): min=0, max=3572.6k, avg=32152.11, stdev=62686.37
  6339. clat (nsec): min=0, max=610833k, avg=2659805.67, stdev=24685079.65
  6340. lat (nsec): min=0, max=610861k, avg=2693027.06, stdev=24687759.61
  6341. clat percentiles (nsec):
  6342. | 1.00th=[ 716], 5.00th=[ 724], 10.00th=[ 740],
  6343. | 20.00th=[ 844], 30.00th=[ 1032], 40.00th=[ 1368],
  6344. | 50.00th=[ 268288], 60.00th=[ 700416], 70.00th=[ 831488],
  6345. | 80.00th=[ 1056768], 90.00th=[ 1941504], 95.00th=[ 3325952],
  6346. | 99.00th=[ 41680896], 99.50th=[101187584], 99.90th=[467664896],
  6347. | 99.95th=[509607936], 99.99th=[599785472]
  6348. bw ( KiB/s): min= 0, max=23191, per=43.83%, avg=13330.00, stdev=13945.56, samples=2
  6349. iops : min= 0, max= 362, avg=208.00, stdev=217.79, samples=2
  6350. write: IOPS=207, BW=12.9MiB/s (13.6MB/s)(1554MiB/120002msec)
  6351. slat (nsec): min=0, max=12405k, avg=162543.64, stdev=196155.68
  6352. clat (nsec): min=0, max=677169k, avg=16443104.90, stdev=52031115.06
  6353. lat (nsec): min=0, max=677230k, avg=16606816.60, stdev=52011422.69
  6354. clat percentiles (usec):
  6355. | 1.00th=[ 881], 5.00th=[ 1029], 10.00th=[ 1106], 20.00th=[ 1221],
  6356. | 30.00th=[ 1336], 40.00th=[ 1483], 50.00th=[ 1696], 60.00th=[ 2147],
  6357. | 70.00th=[ 3294], 80.00th=[ 5997], 90.00th=[ 34866], 95.00th=[103285],
  6358. | 99.00th=[263193], 99.50th=[383779], 99.90th=[549454], 99.95th=[591397],
  6359. | 99.99th=[650118]
  6360. bw ( KiB/s): min= 0, max=22936, per=94.24%, avg=13255.50, stdev=13690.29, samples=2
  6361. iops : min= 0, max= 358, avg=206.50, stdev=214.25, samples=2
  6362. lat (nsec) : 750=5.86%, 1000=8.55%
  6363. lat (usec) : 2=6.79%, 4=0.03%, 10=0.24%, 20=0.76%, 50=0.65%
  6364. lat (usec) : 100=0.61%, 250=1.32%, 500=3.16%, 750=3.98%, 1000=9.39%
  6365. lat (msec) : 2=32.63%, 4=11.09%, 10=6.29%, 20=2.00%, 50=1.86%
  6366. lat (msec) : 100=1.92%, 250=2.17%, 500=0.61%, 750=0.11%
  6367. cpu : usr=0.24%, sys=0.94%, ctx=66524, majf=0, minf=34
  6368. IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  6369. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  6370. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  6371. issued rwts: total=24996,24854,0,0 short=0,0,0,0 dropped=0,0,0,0
  6372. latency : target=0, window=0, percentile=100.00%, depth=4
  6373.  
  6374. Run status group 0 (all jobs):
  6375. READ: bw=29.7MiB/s (31.1MB/s), 13.0MiB/s-16.7MiB/s (13.7MB/s-17.5MB/s), io=3567MiB (3740MB), run=120002-120115msec
  6376. WRITE: bw=13.7MiB/s (14.4MB/s), 1003KiB/s-12.9MiB/s (1027kB/s-13.6MB/s), io=1673MiB (1754MB), run=120002-121774msec
  6377.  
  6378. Disk stats (read/write):
  6379. rbd227: ios=15912/26988, merge=0/1, ticks=587346/884907, in_queue=1450898, util=10.17%
  6380. <example-benchmark-fio-server-700-benchmark-lbhjl>
  6381. randwrite: (groupid=0, jobs=1): err= 0: pid=11: Wed May 8 20:35:15 2019
  6382. write: IOPS=46, BW=2954KiB/s (3025kB/s)(346MiB/120020msec)
  6383. slat (nsec): min=0, max=941199k, avg=450397.43, stdev=14470997.85
  6384. clat (nsec): min=0, max=1098.0M, avg=86745193.42, stdev=158139041.80
  6385. lat (nsec): min=0, max=1098.1M, avg=87196573.48, stdev=158617381.78
  6386. clat percentiles (usec):
  6387. | 1.00th=[ 1004], 5.00th=[ 1205], 10.00th=[ 1418],
  6388. | 20.00th=[ 1844], 30.00th=[ 2376], 40.00th=[ 3720],
  6389. | 50.00th=[ 8848], 60.00th=[ 42206], 70.00th=[ 88605],
  6390. | 80.00th=[ 125305], 90.00th=[ 274727], 95.00th=[ 408945],
  6391. | 99.00th=[ 801113], 99.50th=[ 926942], 99.90th=[1044382],
  6392. | 99.95th=[1098908], 99.99th=[1098908]
  6393. bw ( KiB/s): min= 0, max= 2189, per=14.78%, avg=2189.00, stdev= 0.00, samples=1
  6394. iops : min= 0, max= 34, avg=34.00, stdev= 0.00, samples=1
  6395. lat (usec) : 500=0.02%, 1000=0.94%
  6396. lat (msec) : 2=22.85%, 4=17.54%, 10=9.64%, 20=4.55%, 50=5.82%
  6397. lat (msec) : 100=12.14%, 250=15.66%, 500=7.57%, 750=1.99%, 1000=1.07%
  6398. cpu : usr=0.06%, sys=0.19%, ctx=10237, majf=0, minf=30
  6399. IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  6400. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  6401. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  6402. issued rwts: total=0,5537,0,0 short=0,0,0,0 dropped=0,0,0,0
  6403. latency : target=0, window=0, percentile=100.00%, depth=4
  6404. read: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:35:15 2019
  6405. read: IOPS=435, BW=27.2MiB/s (28.5MB/s)(3266MiB/120022msec)
  6406. slat (nsec): min=0, max=519236k, avg=36005.92, stdev=2710731.95
  6407. clat (nsec): min=0, max=1273.1M, avg=9152643.07, stdev=51653585.32
  6408. lat (nsec): min=0, max=1273.1M, avg=9189505.59, stdev=51731663.29
  6409. clat percentiles (nsec):
  6410. | 1.00th=[ 708], 5.00th=[ 708], 10.00th=[ 716],
  6411. | 20.00th=[ 724], 30.00th=[ 732], 40.00th=[ 740],
  6412. | 50.00th=[ 804], 60.00th=[ 876], 70.00th=[ 980],
  6413. | 80.00th=[ 1480], 90.00th=[ 2441216], 95.00th=[ 44826624],
  6414. | 99.00th=[ 254803968], 99.50th=[ 413138944], 99.90th=[ 683671552],
  6415. | 99.95th=[ 750780416], 99.99th=[1019215872]
  6416. bw ( KiB/s): min= 0, max=42058, per=70.17%, avg=27869.00, stdev=20066.28, samples=2
  6417. iops : min= 0, max= 658, avg=435.50, stdev=314.66, samples=2
  6418. lat (nsec) : 750=43.47%, 1000=27.31%
  6419. lat (usec) : 2=9.90%, 4=0.05%, 10=1.01%, 20=0.67%, 50=0.25%
  6420. lat (usec) : 100=0.11%, 250=0.08%, 500=0.90%, 750=1.13%, 1000=1.88%
  6421. lat (msec) : 2=2.63%, 4=1.72%, 10=0.99%, 20=0.62%, 50=2.79%
  6422. lat (msec) : 100=2.38%, 250=1.09%, 500=0.75%, 750=0.22%, 1000=0.04%
  6423. cpu : usr=0.13%, sys=0.48%, ctx=17884, majf=0, minf=32
  6424. IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  6425. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  6426. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  6427. issued rwts: total=52311,0,0,0 short=0,0,0,0 dropped=0,0,0,0
  6428. latency : target=0, window=0, percentile=100.00%, depth=4
  6429. readwrite: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:35:15 2019
  6430. read: IOPS=185, BW=11.6MiB/s (12.1MB/s)(1389MiB/120002msec)
  6431. slat (nsec): min=0, max=2063.9k, avg=37114.12, stdev=56251.65
  6432. clat (nsec): min=0, max=969207k, avg=2652385.66, stdev=19531361.29
  6433. lat (nsec): min=0, max=969226k, avg=2690596.59, stdev=19533369.42
  6434. clat percentiles (nsec):
  6435. | 1.00th=[ 716], 5.00th=[ 740], 10.00th=[ 772],
  6436. | 20.00th=[ 940], 30.00th=[ 1272], 40.00th=[ 288768],
  6437. | 50.00th=[ 626688], 60.00th=[ 782336], 70.00th=[ 913408],
  6438. | 80.00th=[ 1204224], 90.00th=[ 2342912], 95.00th=[ 4227072],
  6439. | 99.00th=[ 53739520], 99.50th=[ 85458944], 99.90th=[333447168],
  6440. | 99.95th=[425721856], 99.99th=[517996544]
  6441. bw ( KiB/s): min= 0, max=22243, per=29.84%, avg=11850.00, stdev=14697.92, samples=2
  6442. iops : min= 0, max= 347, avg=184.50, stdev=229.81, samples=2
  6443. write: IOPS=185, BW=11.6MiB/s (12.1MB/s)(1390MiB/120002msec)
  6444. slat (nsec): min=0, max=11228k, avg=166988.35, stdev=176151.67
  6445. clat (nsec): min=0, max=1018.4M, avg=18784576.99, stdev=68641704.52
  6446. lat (nsec): min=0, max=1018.5M, avg=18952667.28, stdev=68621574.76
  6447. clat percentiles (usec):
  6448. | 1.00th=[ 889], 5.00th=[ 1020], 10.00th=[ 1106], 20.00th=[ 1205],
  6449. | 30.00th=[ 1303], 40.00th=[ 1434], 50.00th=[ 1598], 60.00th=[ 1909],
  6450. | 70.00th=[ 2704], 80.00th=[ 4752], 90.00th=[ 22414], 95.00th=[106431],
  6451. | 99.00th=[383779], 99.50th=[501220], 99.90th=[700449], 99.95th=[725615],
  6452. | 99.99th=[926942]
  6453. bw ( KiB/s): min= 0, max=22164, per=80.07%, avg=11862.00, stdev=14569.23, samples=2
  6454. iops : min= 0, max= 346, avg=185.00, stdev=227.69, samples=2
  6455. lat (nsec) : 750=3.61%, 1000=7.51%
  6456. lat (usec) : 2=5.58%, 4=0.02%, 10=0.09%, 20=0.35%, 50=0.45%
  6457. lat (usec) : 100=0.39%, 250=1.19%, 500=4.59%, 750=4.92%, 1000=10.38%
  6458. lat (msec) : 2=35.97%, 4=11.08%, 10=5.80%, 20=1.87%, 50=1.52%
  6459. lat (msec) : 100=1.77%, 250=1.74%, 500=0.91%, 750=0.23%, 1000=0.02%
  6460. cpu : usr=0.21%, sys=0.82%, ctx=64428, majf=0, minf=103
  6461. IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  6462. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  6463. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  6464. issued rwts: total=22220,22241,0,0 short=0,0,0,0 dropped=0,0,0,0
  6465. latency : target=0, window=0, percentile=100.00%, depth=4
  6466.  
  6467. Run status group 0 (all jobs):
  6468. READ: bw=38.8MiB/s (40.7MB/s), 11.6MiB/s-27.2MiB/s (12.1MB/s-28.5MB/s), io=4655MiB (4881MB), run=120002-120022msec
  6469. WRITE: bw=14.5MiB/s (15.2MB/s), 2954KiB/s-11.6MiB/s (3025kB/s-12.1MB/s), io=1737MiB (1821MB), run=120002-120020msec
  6470.  
  6471. Disk stats (read/write):
  6472. rbd274: ios=26881/29490, merge=0/1, ticks=576168/834987, in_queue=1382881, util=24.06%
  6473. <example-benchmark-fio-server-292-benchmark-bqpc4>
  6474. randwrite: (groupid=0, jobs=1): err= 0: pid=11: Wed May 8 20:35:16 2019
  6475. write: IOPS=31, BW=2033KiB/s (2082kB/s)(239MiB/120229msec)
  6476. slat (nsec): min=0, max=538330k, avg=792992.72, stdev=17014015.86
  6477. clat (nsec): min=0, max=1096.8M, avg=125120214.34, stdev=193494723.33
  6478. lat (nsec): min=0, max=1096.8M, avg=125914117.71, stdev=194040761.47
  6479. clat percentiles (usec):
  6480. | 1.00th=[ 971], 5.00th=[ 1188], 10.00th=[ 1434],
  6481. | 20.00th=[ 2073], 30.00th=[ 3556], 40.00th=[ 7242],
  6482. | 50.00th=[ 22676], 60.00th=[ 85459], 70.00th=[ 125305],
  6483. | 80.00th=[ 210764], 90.00th=[ 392168], 95.00th=[ 549454],
  6484. | 99.00th=[ 868221], 99.50th=[ 918553], 99.90th=[1098908],
  6485. | 99.95th=[1098908], 99.99th=[1098908]
  6486. bw ( KiB/s): min= 0, max= 2264, per=14.20%, avg=2034.50, stdev=324.56, samples=2
  6487. iops : min= 0, max= 35, avg=31.50, stdev= 4.95, samples=2
  6488. lat (usec) : 500=0.05%, 750=0.03%, 1000=1.49%
  6489. lat (msec) : 2=17.42%, 4=13.13%, 10=11.08%, 20=5.87%, 50=5.32%
  6490. lat (msec) : 100=9.90%, 250=18.08%, 500=11.53%, 750=3.90%, 1000=2.04%
  6491. cpu : usr=0.04%, sys=0.14%, ctx=7226, majf=0, minf=29
  6492. IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  6493. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  6494. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  6495. issued rwts: total=0,3817,0,0 short=0,0,0,0 dropped=0,0,0,0
  6496. latency : target=0, window=0, percentile=100.00%, depth=4
  6497. read: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:35:16 2019
  6498. read: IOPS=470, BW=29.4MiB/s (30.8MB/s)(3525MiB/120058msec)
  6499. slat (nsec): min=0, max=528245k, avg=50365.71, stdev=3931550.84
  6500. clat (nsec): min=0, max=976416k, avg=8467552.73, stdev=53420881.21
  6501. lat (nsec): min=0, max=976446k, avg=8518714.93, stdev=53571321.10
  6502. clat percentiles (nsec):
  6503. | 1.00th=[ 708], 5.00th=[ 708], 10.00th=[ 716],
  6504. | 20.00th=[ 724], 30.00th=[ 724], 40.00th=[ 740],
  6505. | 50.00th=[ 812], 60.00th=[ 852], 70.00th=[ 900],
  6506. | 80.00th=[ 1032], 90.00th=[ 23168], 95.00th=[ 27656192],
  6507. | 99.00th=[274726912], 99.50th=[442499072], 99.90th=[700448768],
  6508. | 99.95th=[784334848], 99.99th=[910163968]
  6509. bw ( KiB/s): min= 0, max=45037, per=70.93%, avg=30076.50, stdev=21157.34, samples=2
  6510. iops : min= 0, max= 706, avg=471.00, stdev=332.34, samples=2
  6511. lat (nsec) : 750=41.94%, 1000=37.25%
  6512. lat (usec) : 2=8.02%, 4=0.04%, 10=1.72%, 20=0.97%, 50=0.35%
  6513. lat (usec) : 100=0.10%, 250=0.03%, 500=0.09%, 750=0.46%, 1000=0.92%
  6514. lat (msec) : 2=1.38%, 4=0.84%, 10=0.46%, 20=0.27%, 50=1.30%
  6515. lat (msec) : 100=1.72%, 250=1.03%, 500=0.78%, 750=0.27%, 1000=0.06%
  6516. cpu : usr=0.12%, sys=0.46%, ctx=10834, majf=0, minf=29
  6517. IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  6518. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  6519. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  6520. issued rwts: total=56535,0,0,0 short=0,0,0,0 dropped=0,0,0,0
  6521. latency : target=0, window=0, percentile=100.00%, depth=4
  6522. readwrite: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:35:16 2019
  6523. read: IOPS=192, BW=12.1MiB/s (12.6MB/s)(1446MiB/120008msec)
  6524. slat (nsec): min=0, max=537040k, avg=77887.67, stdev=4761859.59
  6525. clat (nsec): min=0, max=1008.6M, avg=2244051.44, stdev=19678423.59
  6526. lat (nsec): min=0, max=1018.6M, avg=2322966.53, stdev=20791907.39
  6527. clat percentiles (nsec):
  6528. | 1.00th=[ 716], 5.00th=[ 732], 10.00th=[ 748],
  6529. | 20.00th=[ 844], 30.00th=[ 1032], 40.00th=[ 1528],
  6530. | 50.00th=[ 342016], 60.00th=[ 733184], 70.00th=[ 864256],
  6531. | 80.00th=[ 1155072], 90.00th=[ 2179072], 95.00th=[ 3751936],
  6532. | 99.00th=[ 39583744], 99.50th=[ 72876032], 99.90th=[329252864],
  6533. | 99.95th=[463470592], 99.99th=[608174080]
  6534. bw ( KiB/s): min= 0, max=22722, per=29.11%, avg=12342.00, stdev=14679.54, samples=2
  6535. iops : min= 0, max= 355, avg=192.50, stdev=229.81, samples=2
  6536. write: IOPS=192, BW=12.0MiB/s (12.6MB/s)(1444MiB/120008msec)
  6537. slat (nsec): min=0, max=410477k, avg=203439.38, stdev=3377915.14
  6538. clat (nsec): min=0, max=1034.9M, avg=18357830.34, stdev=67889277.51
  6539. lat (nsec): min=0, max=1034.9M, avg=18562367.41, stdev=67958632.05
  6540. clat percentiles (usec):
  6541. | 1.00th=[ 881], 5.00th=[ 1012], 10.00th=[ 1090], 20.00th=[ 1205],
  6542. | 30.00th=[ 1319], 40.00th=[ 1434], 50.00th=[ 1631], 60.00th=[ 1958],
  6543. | 70.00th=[ 2769], 80.00th=[ 4948], 90.00th=[ 20579], 95.00th=[ 98042],
  6544. | 99.00th=[404751], 99.50th=[492831], 99.90th=[608175], 99.95th=[708838],
  6545. | 99.99th=[918553]
  6546. bw ( KiB/s): min= 0, max=22592, per=85.97%, avg=12319.00, stdev=14528.22, samples=2
  6547. iops : min= 0, max= 353, avg=192.00, stdev=227.69, samples=2
  6548. lat (nsec) : 750=5.08%, 1000=9.32%
  6549. lat (usec) : 2=5.93%, 4=0.01%, 10=0.22%, 20=0.58%, 50=0.52%
  6550. lat (usec) : 100=0.53%, 250=1.30%, 500=3.34%, 750=3.99%, 1000=9.36%
  6551. lat (msec) : 2=34.81%, 4=11.10%, 10=6.41%, 20=1.81%, 50=1.59%
  6552. lat (msec) : 100=1.50%, 250=1.44%, 500=0.93%, 750=0.24%, 1000=0.01%
  6553. cpu : usr=0.23%, sys=0.83%, ctx=63019, majf=0, minf=37
  6554. IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  6555. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  6556. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  6557. issued rwts: total=23145,23098,0,0 short=0,0,0,0 dropped=0,0,0,0
  6558. latency : target=0, window=0, percentile=100.00%, depth=4
  6559.  
  6560. Run status group 0 (all jobs):
  6561. READ: bw=41.4MiB/s (43.4MB/s), 12.1MiB/s-29.4MiB/s (12.6MB/s-30.8MB/s), io=4971MiB (5213MB), run=120008-120058msec
  6562. WRITE: bw=13.0MiB/s (14.7MB/s), 2033KiB/s-12.0MiB/s (2082kB/s-12.6MB/s), io=1683MiB (1764MB), run=120008-120229msec
  6563.  
  6564. Disk stats (read/write):
  6565. rbd153: ios=17769/27091, merge=0/1, ticks=542581/849541, in_queue=1369843, util=18.48%
  6566. Jobs: 2 (f=2): [_(2),m(1)][97.1%][r=158M,w=151M][r=2463,w=2356 IOPS][eta 00m:34s] Jobs: 2 (f=2): [_(2),m(1)][97.2%][r=178M,w=165M][r=2784,w=2585 IOPS][eta 00m:33s] Jobs: 2 (f=2): [_(2),m(1)][97.3%][r=172M,w=186M][r=2691,w=2905 IOPS][eta 00m:32s] Jobs: 2 (f=2): [_(2),m(1)][97.4%][r=174M,w=180M][r=2711,w=2820 IOPS][eta 00m:31s] Jobs: 2 (f=2): [_(2),m(1)][97.5%][r=151M,w=157M][r=2364,w=2454 IOPS][eta 00m:30s] Jobs: 2 (f=2): [_(2),m(1)][97.5%][r=34.6M,w=36.0M][r=539,w=561 IOPS][eta 00m:29s] Jobs: 2 (f=2): [_(2),m(1)][97.6%][r=194M,w=176M][r=3028,w=2746 IOPS][eta 00m:28s] Jobs: 2 (f=2): [_(2),m(1)][97.7%][r=182M,w=173M][r=2837,w=2706 IOPS][eta 00m:27s] <example-benchmark-fio-server-561-benchmark-p2zwl>
  6567. randwrite: (groupid=0, jobs=1): err= 0: pid=11: Wed May 8 20:35:23 2019
  6568. write: IOPS=25, BW=1646KiB/s (1686kB/s)(194MiB/120815msec)
  6569. slat (nsec): min=0, max=904337k, avg=1214381.02, stdev=28710695.03
  6570. clat (nsec): min=0, max=2205.6M, avg=154348561.57, stdev=313344406.92
  6571. lat (nsec): min=0, max=2205.7M, avg=155563464.57, stdev=315846325.37
  6572. clat percentiles (usec):
  6573. | 1.00th=[ 1012], 5.00th=[ 1254], 10.00th=[ 1483],
  6574. | 20.00th=[ 1926], 30.00th=[ 2671], 40.00th=[ 3752],
  6575. | 50.00th=[ 5735], 60.00th=[ 11469], 70.00th=[ 44303],
  6576. | 80.00th=[ 183501], 90.00th=[ 641729], 95.00th=[ 868221],
  6577. | 99.00th=[1317012], 99.50th=[1417675], 99.90th=[1988101],
  6578. | 99.95th=[2164261], 99.99th=[2197816]
  6579. bw ( KiB/s): min= 0, max= 2024, per=8.70%, avg=1655.00, stdev=521.84, samples=2
  6580. iops : min= 0, max= 31, avg=25.50, stdev= 7.78, samples=2
  6581. lat (usec) : 500=0.03%, 750=0.06%, 1000=0.81%
  6582. lat (msec) : 2=20.35%, 4=20.32%, 10=16.46%, 20=7.02%, 50=5.96%
  6583. lat (msec) : 100=6.02%, 250=4.35%, 500=3.74%, 750=7.83%, 1000=3.96%
  6584. cpu : usr=0.03%, sys=0.11%, ctx=5679, majf=0, minf=26
  6585. IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  6586. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  6587. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  6588. issued rwts: total=0,3105,0,0 short=0,0,0,0 dropped=0,0,0,0
  6589. latency : target=0, window=0, percentile=100.00%, depth=4
  6590. read: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:35:23 2019
  6591. read: IOPS=488, BW=30.4MiB/s (31.9MB/s)(3658MiB/120210msec)
  6592. slat (nsec): min=0, max=893326k, avg=59818.55, stdev=5752672.97
  6593. clat (nsec): min=0, max=1867.1M, avg=8176169.39, stdev=74702180.11
  6594. lat (nsec): min=0, max=1867.1M, avg=8236986.67, stdev=74929978.25
  6595. clat percentiles (nsec):
  6596. | 1.00th=[ 716], 5.00th=[ 724], 10.00th=[ 724],
  6597. | 20.00th=[ 740], 30.00th=[ 780], 40.00th=[ 796],
  6598. | 50.00th=[ 828], 60.00th=[ 860], 70.00th=[ 892],
  6599. | 80.00th=[ 940], 90.00th=[ 11584], 95.00th=[ 790528],
  6600. | 99.00th=[ 404750336], 99.50th=[ 624951296], 99.90th=[1061158912],
  6601. | 99.95th=[1233125376], 99.99th=[1417674752]
  6602. bw ( KiB/s): min= 0, max=45246, per=64.08%, avg=31208.00, stdev=19852.73, samples=2
  6603. iops : min= 0, max= 710, avg=489.00, stdev=312.54, samples=2
  6604. lat (nsec) : 750=21.79%, 1000=60.61%
  6605. lat (usec) : 2=5.91%, 4=0.14%, 10=0.58%, 20=4.82%, 50=0.60%
  6606. lat (usec) : 100=0.20%, 250=0.09%, 500=0.09%, 750=0.12%, 1000=0.44%
  6607. lat (msec) : 2=1.01%, 4=0.75%, 10=0.49%, 20=0.23%, 50=0.43%
  6608. lat (msec) : 100=0.45%, 250=0.11%, 500=0.45%, 750=0.43%, 1000=0.16%
  6609. cpu : usr=0.11%, sys=0.48%, ctx=6521, majf=0, minf=32
  6610. IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  6611. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  6612. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  6613. issued rwts: total=58746,0,0,0 short=0,0,0,0 dropped=0,0,0,0
  6614. latency : target=0, window=0, percentile=100.00%, depth=4
  6615. readwrite: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:35:23 2019
  6616. read: IOPS=274, BW=17.2MiB/s (17.0MB/s)(2060MiB/120002msec)
  6617. slat (nsec): min=0, max=11991k, avg=33428.44, stdev=87379.39
  6618. clat (nsec): min=0, max=1079.6M, avg=1861929.90, stdev=18135558.66
  6619. lat (nsec): min=0, max=1079.6M, avg=1896414.99, stdev=18137606.47
  6620. clat percentiles (nsec):
  6621. | 1.00th=[ 708], 5.00th=[ 732], 10.00th=[ 756],
  6622. | 20.00th=[ 908], 30.00th=[ 1352], 40.00th=[ 284672],
  6623. | 50.00th=[ 505856], 60.00th=[ 741376], 70.00th=[ 839680],
  6624. | 80.00th=[ 1036288], 90.00th=[ 1843200], 95.00th=[ 3063808],
  6625. | 99.00th=[ 22413312], 99.50th=[ 58458112], 99.90th=[263192576],
  6626. | 99.95th=[375390208], 99.99th=[926941184]
  6627. bw ( KiB/s): min= 0, max=34082, per=36.09%, avg=17577.00, stdev=23341.59, samples=2
  6628. iops : min= 0, max= 532, avg=274.00, stdev=364.87, samples=2
  6629. write: IOPS=273, BW=17.1MiB/s (17.9MB/s)(2051MiB/120002msec)
  6630. slat (nsec): min=0, max=386496k, avg=167469.00, stdev=2137773.06
  6631. clat (nsec): min=0, max=1040.4M, avg=12588891.95, stdev=54899280.21
  6632. lat (nsec): min=0, max=1040.4M, avg=12757420.24, stdev=54996504.54
  6633. clat percentiles (usec):
  6634. | 1.00th=[ 857], 5.00th=[ 988], 10.00th=[ 1057], 20.00th=[ 1156],
  6635. | 30.00th=[ 1237], 40.00th=[ 1336], 50.00th=[ 1450], 60.00th=[ 1647],
  6636. | 70.00th=[ 2114], 80.00th=[ 3392], 90.00th=[ 9110], 95.00th=[ 42730],
  6637. | 99.00th=[337642], 99.50th=[438305], 99.90th=[566232], 99.95th=[608175],
  6638. | 99.99th=[692061]
  6639. bw ( KiB/s): min= 0, max=33863, per=91.97%, avg=17504.00, stdev=23135.12, samples=2
  6640. iops : min= 0, max= 529, avg=273.00, stdev=362.04, samples=2
  6641. lat (nsec) : 750=4.52%, 1000=7.01%
  6642. lat (usec) : 2=4.12%, 4=0.02%, 10=0.13%, 20=0.47%, 50=0.50%
  6643. lat (usec) : 100=0.44%, 250=1.78%, 500=6.01%, 750=5.68%, 1000=11.66%
  6644. lat (msec) : 2=37.37%, 4=10.03%, 10=4.85%, 20=1.67%, 50=1.06%
  6645. lat (msec) : 100=0.95%, 250=0.97%, 500=0.62%, 750=0.15%, 1000=0.01%
  6646. cpu : usr=0.31%, sys=1.09%, ctx=95248, majf=0, minf=34
  6647. IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  6648. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  6649. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  6650. issued rwts: total=32958,32820,0,0 short=0,0,0,0 dropped=0,0,0,0
  6651. latency : target=0, window=0, percentile=100.00%, depth=4
  6652.  
  6653. Run status group 0 (all jobs):
  6654. READ: bw=47.6MiB/s (49.9MB/s), 17.2MiB/s-30.4MiB/s (17.0MB/s-31.9MB/s), io=5717MiB (5995MB), run=120002-120210msec
  6655. WRITE: bw=18.6MiB/s (19.5MB/s), 1646KiB/s-17.1MiB/s (1686kB/s-17.9MB/s), io=2246MiB (2355MB), run=120002-120815msec
  6656.  
  6657. Disk stats (read/write):
  6658. rbd281: ios=24610/36374, merge=0/1, ticks=576701/893934, in_queue=1440916, util=15.37%
  6659. Jobs: 1 (f=1): [_(2),m(1)][94.7%][r=92.5M,w=89.8M][r=1446,w=1403 IOPS][eta 00m:26s] Jobs: 1 (f=1): [_(2),m(1)][94.9%][r=82.6M,w=89.0M][r=1291,w=1391 IOPS][eta 00m:25s] Jobs: 1 (f=1): [_(2),m(1)][95.1%][r=93.7M,w=90.5M][r=1463,w=1413 IOPS][eta 00m:24s] Jobs: 1 (f=1): [_(2),m(1)][95.3%][r=89.0M,w=94.5M][r=1406,w=1476 IOPS][eta 00m:23s] Jobs: 1 (f=1): [_(2),m(1)][95.5%][r=91.4M,w=89.8M][r=1428,w=1403 IOPS][eta 00m:22s] Jobs: 1 (f=1): [_(2),m(1)][95.7%][r=100M,w=97.9M][r=1564,w=1530 IOPS][eta 00m:21s] Jobs: 1 (f=1): [_(2),m(1)][95.9%][r=36.5M,w=39.4M][r=571,w=615 IOPS][eta 00m:20s] Jobs: 1 (f=1): [_(2),m(1)][96.1%][r=36.7M,w=40.8M][r=574,w=638 IOPS][eta 00m:19s] Jobs: 1 (f=1): [_(2),m(1)][96.3%][r=95.3M,w=94.1M][r=1489,w=1470 IOPS][eta 00m:18s] Jobs: 1 (f=1): [_(2),m(1)][96.5%][r=116M,w=114M][r=1811,w=1787 IOPS][eta 00m:17s] Jobs: 1 (f=1): [_(2),m(1)][96.9%][r=113M,w=115M][r=1765,w=1797 IOPS][eta 00m:15s] Jobs: 1 (f=1): [_(2),m(1)][96.9%][r=118M,w=119M][r=1851,w=1857 IOPS][eta 00m:15s] Jobs: 1 (f=1): [_(2),m(1)][97.1%][r=113M,w=117M][r=1769,w=1825 IOPS][eta 00m:14s] Jobs: 1 (f=1): [_(2),m(1)][97.6%][r=117M,w=117M][r=1835,w=1833 IOPS][eta 00m:12s] Jobs: 1 (f=1): [_(2),m(1)][97.6%][r=112M,w=114M][r=1747,w=1778 IOPS][eta 00m:12s] Jobs: 1 (f=1): [_(2),m(1)][97.8%][r=120M,w=118M][r=1871,w=1840 IOPS][eta 00m:11s] Jobs: 1 (f=1): [_(2),m(1)][98.0%][r=114M,w=120M][r=1775,w=1872 IOPS][eta 00m:10s] Jobs: 1 (f=1): [_(2),m(1)][98.4%][r=110M,w=102M][r=1720,w=1599 IOPS][eta 00m:08s] Jobs: 1 (f=1): [_(2),m(1)][98.4%][r=20.8M,w=25.1M][r=324,w=391 IOPS][eta 00m:08s] Jobs: 1 (f=1): [_(2),m(1)][98.6%][r=106M,w=102M][r=1663,w=1601 IOPS][eta 00m:07s] Jobs: 1 (f=1): [_(2),m(1)][98.8%][r=107M,w=107M][r=1677,w=1665 IOPS][eta 00m:06s] Jobs: 1 (f=1): [_(2),m(1)][99.0%][r=107M,w=105M][r=1667,w=1635 IOPS][eta 00m:05s] Jobs: 1 (f=1): [_(2),m(1)][99.2%][r=105M,w=104M][r=1645,w=1622 IOPS][eta 00m:04s] Jobs: 1 (f=1): [_(2),m(1)][99.4%][r=106M,w=111M][r=1663,w=1736 IOPS][eta 00m:03s] Jobs: 1 (f=1): [_(2),m(1)][99.6%][r=107M,w=101M][r=1677,w=1585 IOPS][eta 00m:02s] Jobs: 1 (f=1): [_(2),m(1)][99.8%][r=106M,w=107M][r=1654,w=1665 IOPS][eta 00m:01s] Jobs: 1 (f=1): [_(2),m(1)][100.0%][r=108M,w=107M][r=1686,w=1669 IOPS][eta 00m:00s] <example-benchmark-fio-server-657-benchmark-nfsfh>
  6660. randwrite: (groupid=0, jobs=1): err= 0: pid=11: Wed May 8 20:35:50 2019
  6661. write: IOPS=40, BW=2566KiB/s (2628kB/s)(301MiB/120045msec)
  6662. slat (nsec): min=0, max=482145k, avg=438482.98, stdev=10952692.52
  6663. clat (nsec): min=0, max=1658.2M, avg=99703536.50, stdev=166032787.67
  6664. lat (nsec): min=0, max=1658.3M, avg=100143246.46, stdev=166305537.98
  6665. clat percentiles (usec):
  6666. | 1.00th=[ 947], 5.00th=[ 1188], 10.00th=[ 1418],
  6667. | 20.00th=[ 1909], 30.00th=[ 3163], 40.00th=[ 7635],
  6668. | 50.00th=[ 40109], 60.00th=[ 70779], 70.00th=[ 101188],
  6669. | 80.00th=[ 130548], 90.00th=[ 316670], 95.00th=[ 446694],
  6670. | 99.00th=[ 750781], 99.50th=[ 935330], 99.90th=[1451230],
  6671. | 99.95th=[1602225], 99.99th=[1652556]
  6672. bw ( KiB/s): min= 0, max= 1573, per=3.75%, avg=1573.00, stdev= 0.00, samples=1
  6673. iops : min= 0, max= 24, avg=24.00, stdev= 0.00, samples=1
  6674. lat (usec) : 500=0.02%, 750=0.06%, 1000=1.64%
  6675. lat (msec) : 2=19.38%, 4=12.47%, 10=8.40%, 20=3.24%, 50=7.46%
  6676. lat (msec) : 100=16.80%, 250=18.79%, 500=8.40%, 750=2.35%, 1000=0.64%
  6677. cpu : usr=0.05%, sys=0.16%, ctx=8915, majf=0, minf=26
  6678. IO depths : 1=0.1%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  6679. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  6680. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  6681. issued rwts: total=0,4810,0,0 short=0,0,0,0 dropped=0,0,0,0
  6682. latency : target=0, window=0, percentile=100.00%, depth=4
  6683. read: (groupid=0, jobs=1): err= 0: pid=12: Wed May 8 20:35:50 2019
  6684. read: IOPS=612, BW=38.2MiB/s (40.0MB/s)(4582MiB/120023msec)
  6685. slat (nsec): min=0, max=463662k, avg=27019.50, stdev=2410645.20
  6686. clat (nsec): min=0, max=1544.2M, avg=6507416.01, stdev=46288893.88
  6687. lat (nsec): min=0, max=1544.3M, avg=6535236.00, stdev=46359211.29
  6688. clat percentiles (nsec):
  6689. | 1.00th=[ 708], 5.00th=[ 708], 10.00th=[ 716],
  6690. | 20.00th=[ 716], 30.00th=[ 724], 40.00th=[ 732],
  6691. | 50.00th=[ 740], 60.00th=[ 812], 70.00th=[ 868],
  6692. | 80.00th=[ 948], 90.00th=[ 10432], 95.00th=[ 8454144],
  6693. | 99.00th=[ 191889408], 99.50th=[ 358612992], 99.90th=[ 616562688],
  6694. | 99.95th=[ 750780416], 99.99th=[1149239296]
  6695. bw ( KiB/s): min= 0, max=57456, per=49.78%, avg=39101.00, stdev=25957.89, samples=2
  6696. iops : min= 0, max= 901, avg=612.50, stdev=408.00, samples=2
  6697. lat (nsec) : 750=52.09%, 1000=30.09%
  6698. lat (usec) : 2=6.66%, 4=0.04%, 10=1.10%, 20=0.87%, 50=0.30%
  6699. lat (usec) : 100=0.07%, 250=0.04%, 500=0.30%, 750=0.53%, 1000=0.91%
  6700. lat (msec) : 2=1.09%, 4=0.63%, 10=0.33%, 20=0.24%, 50=1.71%
  6701. lat (msec) : 100=1.41%, 250=0.79%, 500=0.62%, 750=0.14%, 1000=0.04%
  6702. cpu : usr=0.16%, sys=0.56%, ctx=12882, majf=0, minf=108
  6703. IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  6704. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  6705. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  6706. issued rwts: total=73566,0,0,0 short=0,0,0,0 dropped=0,0,0,0
  6707. latency : target=0, window=0, percentile=100.00%, depth=4
  6708. readwrite: (groupid=0, jobs=1): err= 0: pid=13: Wed May 8 20:35:50 2019
  6709. read: IOPS=616, BW=38.5MiB/s (40.4MB/s)(4624MiB/120002msec)
  6710. slat (nsec): min=0, max=4051.2k, avg=34070.14, stdev=38397.97
  6711. clat (nsec): min=0, max=983244k, avg=1157929.31, stdev=11857203.78
  6712. lat (nsec): min=0, max=983271k, avg=1193149.93, stdev=11857821.07
  6713. clat percentiles (nsec):
  6714. | 1.00th=[ 724], 5.00th=[ 780], 10.00th=[ 996],
  6715. | 20.00th=[ 248832], 30.00th=[ 313344], 40.00th=[ 399360],
  6716. | 50.00th=[ 643072], 60.00th=[ 733184], 70.00th=[ 806912],
  6717. | 80.00th=[ 921600], 90.00th=[ 1302528], 95.00th=[ 2211840],
  6718. | 99.00th=[ 5668864], 99.50th=[ 11862016], 99.90th=[ 98041856],
  6719. | 99.95th=[212860928], 99.99th=[692060160]
  6720. bw ( KiB/s): min= 0, max=76727, per=50.24%, avg=39461.00, stdev=52702.08, samples=2
  6721. iops : min= 0, max= 1198, avg=616.00, stdev=823.07, samples=2
  6722. write: IOPS=615, BW=38.5MiB/s (40.4MB/s)(4619MiB/120002msec)
  6723. slat (nsec): min=0, max=32795k, avg=144062.14, stdev=169244.18
  6724. clat (nsec): min=0, max=985893k, avg=5161799.35, stdev=27012639.31
  6725. lat (nsec): min=0, max=985966k, avg=5306860.77, stdev=27005522.14
  6726. clat percentiles (usec):
  6727. | 1.00th=[ 848], 5.00th=[ 979], 10.00th=[ 1057], 20.00th=[ 1139],
  6728. | 30.00th=[ 1221], 40.00th=[ 1287], 50.00th=[ 1352], 60.00th=[ 1434],
  6729. | 70.00th=[ 1565], 80.00th=[ 1860], 90.00th=[ 3392], 95.00th=[ 6783],
  6730. | 99.00th=[111674], 99.50th=[189793], 99.90th=[375391], 99.95th=[413139],
  6731. | 99.99th=[574620]
  6732. bw ( KiB/s): min= 0, max=76539, per=93.92%, avg=39416.50, stdev=52499.14, samples=2
  6733. iops : min= 0, max= 1195, avg=615.00, stdev=820.24, samples=2
  6734. lat (nsec) : 750=1.86%, 1000=3.17%
  6735. lat (usec) : 2=1.65%, 4=0.01%, 10=0.05%, 20=0.16%, 50=0.19%
  6736. lat (usec) : 100=0.18%, 250=2.78%, 500=12.46%, 750=8.78%, 1000=13.88%
  6737. lat (msec) : 2=43.04%, 4=6.77%, 10=2.70%, 20=0.68%, 50=0.48%
  6738. lat (msec) : 100=0.50%, 250=0.48%, 500=0.17%, 750=0.01%, 1000=0.01%
  6739. cpu : usr=0.67%, sys=2.18%, ctx=235909, majf=0, minf=101
  6740. IO depths : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  6741. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  6742. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  6743. issued rwts: total=73990,73908,0,0 short=0,0,0,0 dropped=0,0,0,0
  6744. latency : target=0, window=0, percentile=100.00%, depth=4
  6745.  
  6746. Run status group 0 (all jobs):
  6747. READ: bw=76.7MiB/s (80.4MB/s), 38.2MiB/s-38.5MiB/s (40.0MB/s-40.4MB/s), io=9207MiB (9654MB), run=120002-120023msec
  6748. WRITE: bw=40.0MiB/s (42.0MB/s), 2566KiB/s-38.5MiB/s (2628kB/s-40.4MB/s), io=4920MiB (5159MB), run=120002-120045msec
  6749.  
  6750. Disk stats (read/write):
  6751. rbd266: ios=69727/79136, merge=0/1, ticks=587936/790828, in_queue=1307271, util=33.49%
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement