API tools faq
paste
Advertisement
Guest User

ZFS vs MDADM RAID 10

a guest
Apr 1st, 2020
167
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 200.13 KB | None | 0 0
raw download clone embed print report
  1. Конфигурация сервера:
  2. ====================
  3. proxmox 6.1-8
  4. 2x xeon 14c 2,6 GHz
  5. 512 GB RAM
  6.  
  7. Конфигурация mdadm + thin lvm:
  8. ==============================
  9. те же sas ssd что и для zfs пула но 6 штук в рейд 10
  10.  
  11. Конфигурация zfs пулов.
  12. =======================
  13. 3 пула:
  14. - 8 дисков ssd sas 12 gbps
  15. - 24 диска hdd sas 7k2
  16. - 24 диска hdd sas 7k2
  17.  
  18. ---
  19. удаление метаданных zfs
  20. dd if=/dev/zero of=/dev/sdXX bs=512 count=10
  21. dd if=/dev/zero of=/dev/sdXX bs=512 seek=$(( $(blockdev --getsz /dev/sdXX) - 4096 )) count=1M
  22. ---
  23. корректировка управления подкачкой
  24. sysctl -w vm.swappiness=10
  25. echo "vm.swappiness = 10" >> /etc/sysctl.conf
  26. ---
  27. отправка сообщение о состоянии на почту
  28. apt-get install zfs-zed
  29. ZED_EMAIL_ADDR="root" -> /etc/zfs/zed.d/zed.rc
  30. ---
  31. корректировка размера ARC(128GB)
  32. options zfs zfs_arc_max=137438953472 > /etc/modprobe.d/zfs.conf
  33. update-initramfs -u
  34. ---
  35. создание пула zfs
  36. zpool create -f -o ashift=12 ssd_pool_r10 mirror /dev/mapper/ssd{00,01} mirror /dev/mapper/ssd{02,03} mirror /dev/mapper/ssd{04,05} mirror /dev/mapper/ssd{06,07} mirror /dev/mapper/ssd{08,09}
  37. zfs create ssd_pool_r10/vmdata
  38. zfs set compression=lz4 ssd_pool_r10/vmdata
  39.  
  40. zpool create -f -o ashift=12 hdd_pool_r10 mirror /dev/mapper/hdd{12,13} mirror /dev/mapper/hdd{14,15} mirror /dev/mapper/hdd{16,17} mirror /dev/mapper/hdd{18,19} mirror /dev/mapper/hdd{20,21} mirror /dev/mapper/hdd{22,23} mirror /dev/mapper/hdd{24,25} mirror /dev/mapper/hdd{26,27} mirror /dev/mapper/hdd{28,29} mirror /dev/mapper/hdd{30,31} mirror /dev/mapper/hdd{32,33} mirror /dev/mapper/hdd{34,35}
  41. zfs create hdd_pool_r10/vmdata
  42. zfs set compression=lz4 hdd_pool_r10/vmdata
  43.  
  44. zpool create -f -o ashift=12 hdd_pool_bak raidz2 /dev/mapper/hdd{36,37,38,39,40,41} raidz2 /dev/mapper/hdd{42,43,44,45,46,47} raidz2 /dev/mapper/hdd{48,49,50,51,52,53} raidz2 /dev/mapper/hdd{54,55,56,57,58,59}
  45. zfs create hdd_pool_bak/bkpdata
  46. zfs set compression=lz4 hdd_pool_bak/bkpdata
  47.  
  48. gdisk /dev/mapper/ssd10
  49. n -> 1 -> Enter -> +100G -> Enter
  50. n -> 2 -> Enter -> +100G -> Enter
  51. w -> Y
  52.  
  53. gdisk /dev/mapper/ssd11
  54. n -> 1 -> Enter -> +100G -> Enter
  55. n -> 2 -> Enter -> +100G -> Enter
  56. w -> Y
  57.  
  58. zpool add -f hdd_pool_r10 log mirror /dev/mapper/ssd10-part1 /dev/mapper/ssd11-part1
  59. zpool add -f hdd_pool_bak log mirror /dev/mapper/ssd10-part2 /dev/mapper/ssd11-part2
  60.  
  61. Результаты тестов:
  62. =============================
  63. single_ssd_bench-4k-IOD16.txt
  64. =============================
  65. seqRead: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
  66. seqWrite: (g=1): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
  67. seqWrite/sync: (g=2): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
  68. randRead: (g=3): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
  69. randWrite: (g=4): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
  70. randWrite/sync: (g=5): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
  71. randReadWrite: (g=6): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
  72. randReadWrite/sync: (g=7): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
  73. fio-3.12
  74. Starting 8 processes
  75.  
  76. seqRead: (groupid=0, jobs=1): err= 0: pid=8771: Fri Mar 20 16:09:45 2020
  77. read: IOPS=131k, BW=510MiB/s (535MB/s)(1024MiB/2006msec)
  78. slat (nsec): min=1116, max=2976.9k, avg=6595.80, stdev=46445.86
  79. clat (nsec): min=1308, max=3063.2k, avg=115485.18, stdev=168307.44
  80. lat (usec): min=2, max=3066, avg=122.13, stdev=172.82
  81. clat percentiles (usec):
  82. | 1.00th=[ 29], 5.00th=[ 29], 10.00th=[ 30], 20.00th=[ 30],
  83. | 30.00th=[ 30], 40.00th=[ 30], 50.00th=[ 31], 60.00th=[ 63],
  84. | 70.00th=[ 78], 80.00th=[ 127], 90.00th=[ 400], 95.00th=[ 545],
  85. | 99.00th=[ 660], 99.50th=[ 693], 99.90th=[ 775], 99.95th=[ 824],
  86. | 99.99th=[ 2835]
  87. bw ( KiB/s): min=372785, max=515152, per=80.49%, avg=420724.67, stdev=81779.69, samples=3
  88. iops : min=93196, max=128788, avg=105181.00, stdev=20445.07, samples=3
  89. lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 50=59.32%, 100=16.77%
  90. lat (usec) : 250=9.09%, 500=7.43%, 750=7.25%, 1000=0.13%
  91. lat (msec) : 4=0.02%
  92. cpu : usr=16.56%, sys=35.76%, ctx=2985, majf=0, minf=25
  93. IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
  94. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  95. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
  96. issued rwts: total=262144,0,0,0 short=0,0,0,0 dropped=0,0,0,0
  97. latency : target=0, window=0, percentile=100.00%, depth=16
  98. seqWrite: (groupid=1, jobs=1): err= 0: pid=8777: Fri Mar 20 16:09:45 2020
  99. write: IOPS=279k, BW=1092MiB/s (1145MB/s)(1024MiB/938msec); 0 zone resets
  100. slat (nsec): min=1602, max=26573, avg=2215.28, stdev=495.52
  101. clat (nsec): min=1477, max=100997, avg=54193.82, stdev=2467.06
  102. lat (usec): min=3, max=103, avg=56.47, stdev= 2.55
  103. clat percentiles (nsec):
  104. | 1.00th=[51968], 5.00th=[52480], 10.00th=[52480], 20.00th=[52480],
  105. | 30.00th=[52992], 40.00th=[53504], 50.00th=[54016], 60.00th=[54016],
  106. | 70.00th=[54528], 80.00th=[55040], 90.00th=[56064], 95.00th=[58112],
  107. | 99.00th=[63232], 99.50th=[70144], 99.90th=[80384], 99.95th=[82432],
  108. | 99.99th=[92672]
  109. bw ( MiB/s): min= 1088, max= 1088, per=99.74%, avg=1088.80, stdev= 0.00, samples=1
  110. iops : min=278732, max=278732, avg=278732.00, stdev= 0.00, samples=1
  111. lat (usec) : 2=0.01%, 10=0.01%, 20=0.01%, 50=0.01%, 100=99.99%
  112. lat (usec) : 250=0.01%
  113. cpu : usr=47.60%, sys=52.29%, ctx=1, majf=0, minf=10
  114. IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
  115. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  116. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
  117. issued rwts: total=0,262144,0,0 short=0,0,0,0 dropped=0,0,0,0
  118. latency : target=0, window=0, percentile=100.00%, depth=16
  119. seqWrite/sync: (groupid=2, jobs=1): err= 0: pid=8932: Fri Mar 20 16:09:45 2020
  120. write: IOPS=8936, BW=34.9MiB/s (36.6MB/s)(1024MiB/29335msec); 0 zone resets
  121. slat (nsec): min=2483, max=36677, avg=4765.64, stdev=1449.63
  122. clat (usec): min=2, max=2559, avg=1679.40, stdev=85.53
  123. lat (usec): min=6, max=2564, avg=1684.27, stdev=85.92
  124. clat percentiles (usec):
  125. | 1.00th=[ 1598], 5.00th=[ 1614], 10.00th=[ 1614], 20.00th=[ 1614],
  126. | 30.00th=[ 1631], 40.00th=[ 1631], 50.00th=[ 1631], 60.00th=[ 1647],
  127. | 70.00th=[ 1696], 80.00th=[ 1745], 90.00th=[ 1811], 95.00th=[ 1860],
  128. | 99.00th=[ 1958], 99.50th=[ 1991], 99.90th=[ 2073], 99.95th=[ 2278],
  129. | 99.99th=[ 2409]
  130. bw ( KiB/s): min=26356, max=35981, per=77.39%, avg=27663.40, stdev=1263.03, samples=58
  131. iops : min= 6589, max= 8995, avg=6915.48, stdev=315.74, samples=58
  132. lat (usec) : 4=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
  133. lat (msec) : 2=99.62%, 4=0.38%
  134. fsync/fdatasync/sync_file_range:
  135. sync (nsec): min=70, max=15943, avg=181.68, stdev=252.99
  136. sync percentiles (nsec):
  137. | 1.00th=[ 105], 5.00th=[ 116], 10.00th=[ 124], 20.00th=[ 151],
  138. | 30.00th=[ 163], 40.00th=[ 171], 50.00th=[ 177], 60.00th=[ 181],
  139. | 70.00th=[ 189], 80.00th=[ 197], 90.00th=[ 211], 95.00th=[ 227],
  140. | 99.00th=[ 386], 99.50th=[ 410], 99.90th=[ 482], 99.95th=[ 700],
  141. | 99.99th=[12096]
  142. cpu : usr=4.74%, sys=12.80%, ctx=786390, majf=0, minf=52
  143. IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=200.0%, 32=0.0%, >=64=0.0%
  144. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  145. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
  146. issued rwts: total=0,262144,0,262129 short=0,0,0,0 dropped=0,0,0,0
  147. latency : target=0, window=0, percentile=100.00%, depth=16
  148. randRead: (groupid=3, jobs=1): err= 0: pid=9859: Fri Mar 20 16:09:45 2020
  149. read: IOPS=8551, BW=33.4MiB/s (35.0MB/s)(1024MiB/30655msec)
  150. slat (usec): min=33, max=3099, avg=114.94, stdev=20.68
  151. clat (usec): min=2, max=4846, avg=1754.96, stdev=91.02
  152. lat (usec): min=114, max=4953, avg=1870.14, stdev=94.25
  153. clat percentiles (usec):
  154. | 1.00th=[ 1631], 5.00th=[ 1680], 10.00th=[ 1696], 20.00th=[ 1713],
  155. | 30.00th=[ 1729], 40.00th=[ 1729], 50.00th=[ 1745], 60.00th=[ 1762],
  156. | 70.00th=[ 1778], 80.00th=[ 1795], 90.00th=[ 1827], 95.00th=[ 1860],
  157. | 99.00th=[ 1926], 99.50th=[ 1942], 99.90th=[ 2474], 99.95th=[ 3851],
  158. | 99.99th=[ 4621]
  159. bw ( KiB/s): min=25069, max=33985, per=74.64%, avg=25531.42, stdev=1119.89, samples=60
  160. iops : min= 6267, max= 8496, avg=6382.53, stdev=279.98, samples=60
  161. lat (usec) : 4=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
  162. lat (msec) : 2=99.80%, 4=0.15%, 10=0.05%
  163. cpu : usr=2.00%, sys=6.45%, ctx=262147, majf=0, minf=69
  164. IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
  165. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  166. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
  167. issued rwts: total=262144,0,0,0 short=0,0,0,0 dropped=0,0,0,0
  168. latency : target=0, window=0, percentile=100.00%, depth=16
  169. randWrite: (groupid=4, jobs=1): err= 0: pid=10867: Fri Mar 20 16:09:45 2020
  170. write: IOPS=266k, BW=1041MiB/s (1091MB/s)(1024MiB/984msec); 0 zone resets
  171. slat (nsec): min=1980, max=20252, avg=2317.68, stdev=506.26
  172. clat (nsec): min=1425, max=98552, avg=56826.75, stdev=2094.19
  173. lat (usec): min=4, max=101, avg=59.20, stdev= 2.19
  174. clat percentiles (nsec):
  175. | 1.00th=[54528], 5.00th=[55040], 10.00th=[55040], 20.00th=[55552],
  176. | 30.00th=[55552], 40.00th=[56064], 50.00th=[56576], 60.00th=[56576],
  177. | 70.00th=[57088], 80.00th=[57600], 90.00th=[59136], 95.00th=[60160],
  178. | 99.00th=[64768], 99.50th=[67072], 99.90th=[74240], 99.95th=[78336],
  179. | 99.99th=[83456]
  180. bw ( MiB/s): min= 1038, max= 1038, per=99.75%, avg=1038.02, stdev= 0.00, samples=1
  181. iops : min=265732, max=265732, avg=265732.00, stdev= 0.00, samples=1
  182. lat (usec) : 2=0.01%, 10=0.01%, 20=0.01%, 50=0.01%, 100=99.99%
  183. cpu : usr=42.52%, sys=57.48%, ctx=1, majf=0, minf=8
  184. IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
  185. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  186. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
  187. issued rwts: total=0,262144,0,0 short=0,0,0,0 dropped=0,0,0,0
  188. latency : target=0, window=0, percentile=100.00%, depth=16
  189. randWrite/sync: (groupid=5, jobs=1): err= 0: pid=11032: Fri Mar 20 16:09:45 2020
  190. write: IOPS=8987, BW=35.1MiB/s (36.8MB/s)(1024MiB/29167msec); 0 zone resets
  191. slat (nsec): min=1906, max=248459, avg=4683.98, stdev=1403.52
  192. clat (nsec): min=1974, max=2923.1k, avg=1669749.25, stdev=83375.50
  193. lat (usec): min=6, max=2927, avg=1674.54, stdev=83.75
  194. clat percentiles (usec):
  195. | 1.00th=[ 1614], 5.00th=[ 1614], 10.00th=[ 1614], 20.00th=[ 1614],
  196. | 30.00th=[ 1614], 40.00th=[ 1631], 50.00th=[ 1631], 60.00th=[ 1647],
  197. | 70.00th=[ 1663], 80.00th=[ 1713], 90.00th=[ 1795], 95.00th=[ 1860],
  198. | 99.00th=[ 1942], 99.50th=[ 1991], 99.90th=[ 2147], 99.95th=[ 2311],
  199. | 99.99th=[ 2606]
  200. bw ( KiB/s): min=31241, max=36430, per=92.90%, avg=33396.38, stdev=962.86, samples=58
  201. iops : min= 7810, max= 9107, avg=8348.74, stdev=240.68, samples=58
  202. lat (usec) : 2=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
  203. lat (msec) : 2=99.56%, 4=0.44%
  204. fsync/fdatasync/sync_file_range:
  205. sync (nsec): min=65, max=31335, avg=170.22, stdev=215.63
  206. sync percentiles (nsec):
  207. | 1.00th=[ 97], 5.00th=[ 107], 10.00th=[ 119], 20.00th=[ 147],
  208. | 30.00th=[ 159], 40.00th=[ 163], 50.00th=[ 169], 60.00th=[ 173],
  209. | 70.00th=[ 179], 80.00th=[ 185], 90.00th=[ 193], 95.00th=[ 203],
  210. | 99.00th=[ 354], 99.50th=[ 394], 99.90th=[ 474], 99.95th=[ 502],
  211. | 99.99th=[11968]
  212. cpu : usr=4.48%, sys=12.95%, ctx=786399, majf=0, minf=58
  213. IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=200.0%, 32=0.0%, >=64=0.0%
  214. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  215. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
  216. issued rwts: total=0,262144,0,262129 short=0,0,0,0 dropped=0,0,0,0
  217. latency : target=0, window=0, percentile=100.00%, depth=16
  218. randReadWrite: (groupid=6, jobs=1): err= 0: pid=11877: Fri Mar 20 16:09:45 2020
  219. read: IOPS=8178, BW=31.9MiB/s (33.5MB/s)(511MiB/15999msec)
  220. slat (usec): min=34, max=2903, avg=115.31, stdev=18.90
  221. clat (usec): min=11, max=4079, avg=916.25, stdev=224.57
  222. lat (usec): min=123, max=4310, avg=1031.79, stdev=225.63
  223. clat percentiles (usec):
  224. | 1.00th=[ 400], 5.00th=[ 537], 10.00th=[ 627], 20.00th=[ 734],
  225. | 30.00th=[ 807], 40.00th=[ 857], 50.00th=[ 922], 60.00th=[ 971],
  226. | 70.00th=[ 1037], 80.00th=[ 1090], 90.00th=[ 1205], 95.00th=[ 1287],
  227. | 99.00th=[ 1418], 99.50th=[ 1467], 99.90th=[ 1582], 99.95th=[ 1696],
  228. | 99.99th=[ 3654]
  229. bw ( KiB/s): min=24459, max=32434, per=76.30%, avg=24962.45, stdev=1391.83, samples=31
  230. iops : min= 6114, max= 8108, avg=6240.23, stdev=347.94, samples=31
  231. write: IOPS=8206, BW=32.1MiB/s (33.6MB/s)(513MiB/15999msec); 0 zone resets
  232. slat (nsec): min=1605, max=28032, avg=3210.86, stdev=1220.82
  233. clat (nsec): min=1571, max=4183.4k, avg=916051.28, stdev=226589.57
  234. lat (usec): min=3, max=4185, avg=919.33, stdev=226.63
  235. clat percentiles (usec):
  236. | 1.00th=[ 396], 5.00th=[ 529], 10.00th=[ 627], 20.00th=[ 734],
  237. | 30.00th=[ 807], 40.00th=[ 857], 50.00th=[ 922], 60.00th=[ 971],
  238. | 70.00th=[ 1045], 80.00th=[ 1090], 90.00th=[ 1205], 95.00th=[ 1287],
  239. | 99.00th=[ 1418], 99.50th=[ 1467], 99.90th=[ 1582], 99.95th=[ 1663],
  240. | 99.99th=[ 3654]
  241. bw ( KiB/s): min=23818, max=33411, per=76.29%, avg=25040.87, stdev=1621.63, samples=31
  242. iops : min= 5954, max= 8352, avg=6259.74, stdev=405.35, samples=31
  243. lat (usec) : 2=0.01%, 10=0.01%, 20=0.01%, 100=0.01%, 250=0.05%
  244. lat (usec) : 500=2.68%, 750=21.31%, 1000=42.10%
  245. lat (msec) : 2=33.81%, 4=0.03%, 10=0.01%
  246. cpu : usr=3.56%, sys=8.55%, ctx=130855, majf=0, minf=11
  247. IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
  248. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  249. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
  250. issued rwts: total=130853,131291,0,0 short=0,0,0,0 dropped=0,0,0,0
  251. latency : target=0, window=0, percentile=100.00%, depth=16
  252. randReadWrite/sync: (groupid=7, jobs=1): err= 0: pid=12462: Fri Mar 20 16:09:45 2020
  253. read: IOPS=3187, BW=12.5MiB/s (13.1MB/s)(514MiB/41244msec)
  254. slat (usec): min=49, max=7025, avg=149.00, stdev=186.06
  255. clat (usec): min=23, max=9356, avg=2332.27, stdev=522.99
  256. lat (usec): min=146, max=9509, avg=2481.54, stdev=552.71
  257. clat percentiles (usec):
  258. | 1.00th=[ 1811], 5.00th=[ 1926], 10.00th=[ 1975], 20.00th=[ 2057],
  259. | 30.00th=[ 2114], 40.00th=[ 2180], 50.00th=[ 2212], 60.00th=[ 2278],
  260. | 70.00th=[ 2343], 80.00th=[ 2409], 90.00th=[ 2606], 95.00th=[ 3326],
  261. | 99.00th=[ 4817], 99.50th=[ 5014], 99.90th=[ 5407], 99.95th=[ 6980],
  262. | 99.99th=[ 7963]
  263. bw ( KiB/s): min=11288, max=13400, per=99.96%, avg=12744.37, stdev=280.99, samples=82
  264. iops : min= 2822, max= 3350, avg=3186.05, stdev=70.27, samples=82
  265. write: IOPS=3168, BW=12.4MiB/s (12.0MB/s)(511MiB/41244msec); 0 zone resets
  266. slat (nsec): min=2555, max=32452, avg=4860.56, stdev=1610.31
  267. clat (usec): min=800, max=9560, avg=2389.71, stdev=554.17
  268. lat (usec): min=804, max=9565, avg=2394.69, stdev=554.21
  269. clat percentiles (usec):
  270. | 1.00th=[ 1860], 5.00th=[ 1958], 10.00th=[ 2024], 20.00th=[ 2114],
  271. | 30.00th=[ 2147], 40.00th=[ 2212], 50.00th=[ 2278], 60.00th=[ 2311],
  272. | 70.00th=[ 2376], 80.00th=[ 2474], 90.00th=[ 2671], 95.00th=[ 3589],
  273. | 99.00th=[ 4883], 99.50th=[ 5080], 99.90th=[ 5538], 99.95th=[ 7504],
  274. | 99.99th=[ 8356]
  275. bw ( KiB/s): min=10808, max=13496, per=99.96%, avg=12668.45, stdev=396.06, samples=82
  276. iops : min= 2702, max= 3374, avg=3167.07, stdev=99.02, samples=82
  277. lat (usec) : 50=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
  278. lat (msec) : 2=10.00%, 4=86.45%, 10=3.54%
  279. fsync/fdatasync/sync_file_range:
  280. sync (nsec): min=52, max=12446, avg=154.06, stdev=160.41
  281. sync percentiles (nsec):
  282. | 1.00th=[ 77], 5.00th=[ 94], 10.00th=[ 108], 20.00th=[ 121],
  283. | 30.00th=[ 131], 40.00th=[ 139], 50.00th=[ 143], 60.00th=[ 151],
  284. | 70.00th=[ 157], 80.00th=[ 167], 90.00th=[ 187], 95.00th=[ 262],
  285. | 99.00th=[ 374], 99.50th=[ 398], 99.90th=[ 462], 99.95th=[ 482],
  286. | 99.99th=[11840]
  287. cpu : usr=2.74%, sys=9.37%, ctx=786394, majf=0, minf=14
  288. IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=200.0%, 32=0.0%, >=64=0.0%
  289. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  290. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
  291. issued rwts: total=131456,130688,0,262129 short=0,0,0,0 dropped=0,0,0,0
  292. latency : target=0, window=0, percentile=100.00%, depth=16
  293.  
  294. Run status group 0 (all jobs):
  295. READ: bw=510MiB/s (535MB/s), 510MiB/s-510MiB/s (535MB/s-535MB/s), io=1024MiB (1074MB), run=2006-2006msec
  296.  
  297. Run status group 1 (all jobs):
  298. WRITE: bw=1092MiB/s (1145MB/s), 1092MiB/s-1092MiB/s (1145MB/s-1145MB/s), io=1024MiB (1074MB), run=938-938msec
  299.  
  300. Run status group 2 (all jobs):
  301. WRITE: bw=34.9MiB/s (36.6MB/s), 34.9MiB/s-34.9MiB/s (36.6MB/s-36.6MB/s), io=1024MiB (1074MB), run=29335-29335msec
  302.  
  303. Run status group 3 (all jobs):
  304. READ: bw=33.4MiB/s (35.0MB/s), 33.4MiB/s-33.4MiB/s (35.0MB/s-35.0MB/s), io=1024MiB (1074MB), run=30655-30655msec
  305.  
  306. Run status group 4 (all jobs):
  307. WRITE: bw=1041MiB/s (1091MB/s), 1041MiB/s-1041MiB/s (1091MB/s-1091MB/s), io=1024MiB (1074MB), run=984-984msec
  308.  
  309. Run status group 5 (all jobs):
  310. WRITE: bw=35.1MiB/s (36.8MB/s), 35.1MiB/s-35.1MiB/s (36.8MB/s-36.8MB/s), io=1024MiB (1074MB), run=29167-29167msec
  311.  
  312. Run status group 6 (all jobs):
  313. READ: bw=31.9MiB/s (33.5MB/s), 31.9MiB/s-31.9MiB/s (33.5MB/s-33.5MB/s), io=511MiB (536MB), run=15999-15999msec
  314. WRITE: bw=32.1MiB/s (33.6MB/s), 32.1MiB/s-32.1MiB/s (33.6MB/s-33.6MB/s), io=513MiB (538MB), run=15999-15999msec
  315.  
  316. Run status group 7 (all jobs):
  317. READ: bw=12.5MiB/s (13.1MB/s), 12.5MiB/s-12.5MiB/s (13.1MB/s-13.1MB/s), io=514MiB (538MB), run=41244-41244msec
  318. WRITE: bw=12.4MiB/s (12.0MB/s), 12.4MiB/s-12.4MiB/s (12.0MB/s-12.0MB/s), io=511MiB (535MB), run=41244-41244msec
  319.  
  320. ===============================
  321. ssd_pool_r10_bench-4k-IOD16.txt
  322. ===============================
  323. seqRead: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
  324. seqWrite: (g=1): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
  325. seqWrite/sync: (g=2): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
  326. randRead: (g=3): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
  327. randWrite: (g=4): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
  328. randWrite/sync: (g=5): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
  329. randReadWrite: (g=6): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
  330. randReadWrite/sync: (g=7): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
  331. fio-3.12
  332. Starting 8 processes
  333.  
  334. seqRead: (groupid=0, jobs=1): err= 0: pid=54721: Fri Mar 20 17:57:26 2020
  335. read: IOPS=299k, BW=1166MiB/s (1223MB/s)(1024MiB/878msec)
  336. slat (nsec): min=2154, max=48564, avg=2577.96, stdev=488.99
  337. clat (nsec): min=1436, max=124544, avg=50765.56, stdev=3077.12
  338. lat (usec): min=3, max=130, avg=53.40, stdev= 3.22
  339. clat percentiles (nsec):
  340. | 1.00th=[46336], 5.00th=[47360], 10.00th=[47872], 20.00th=[48384],
  341. | 30.00th=[48896], 40.00th=[49408], 50.00th=[50432], 60.00th=[51456],
  342. | 70.00th=[51968], 80.00th=[52992], 90.00th=[53504], 95.00th=[54528],
  343. | 99.00th=[58624], 99.50th=[63232], 99.90th=[85504], 99.95th=[89600],
  344. | 99.99th=[97792]
  345. bw ( MiB/s): min= 1165, max= 1165, per=99.93%, avg=1165.52, stdev= 0.00, samples=1
  346. iops : min=298372, max=298372, avg=298372.00, stdev= 0.00, samples=1
  347. lat (usec) : 2=0.01%, 10=0.01%, 20=0.01%, 50=47.32%, 100=52.67%
  348. lat (usec) : 250=0.01%
  349. cpu : usr=26.23%, sys=73.55%, ctx=5, majf=0, minf=25
  350. IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
  351. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  352. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
  353. issued rwts: total=262144,0,0,0 short=0,0,0,0 dropped=0,0,0,0
  354. latency : target=0, window=0, percentile=100.00%, depth=16
  355. seqWrite: (groupid=1, jobs=1): err= 0: pid=54722: Fri Mar 20 17:57:26 2020
  356. write: IOPS=149k, BW=582MiB/s (610MB/s)(1024MiB/1760msec); 0 zone resets
  357. slat (usec): min=4, max=156, avg= 5.15, stdev= 2.53
  358. clat (nsec): min=1800, max=392335, avg=101289.43, stdev=16726.42
  359. lat (usec): min=7, max=403, avg=106.52, stdev=17.49
  360. clat percentiles (usec):
  361. | 1.00th=[ 92], 5.00th=[ 93], 10.00th=[ 94], 20.00th=[ 96],
  362. | 30.00th=[ 96], 40.00th=[ 97], 50.00th=[ 97], 60.00th=[ 97],
  363. | 70.00th=[ 98], 80.00th=[ 100], 90.00th=[ 115], 95.00th=[ 127],
  364. | 99.00th=[ 186], 99.50th=[ 210], 99.90th=[ 253], 99.95th=[ 269],
  365. | 99.99th=[ 302]
  366. bw ( KiB/s): min=400336, max=574702, per=77.93%, avg=464318.33, stdev=95998.38, samples=3
  367. iops : min=100084, max=143675, avg=116079.33, stdev=23999.37, samples=3
  368. lat (usec) : 2=0.01%, 10=0.01%, 20=0.01%, 50=0.01%, 100=81.96%
  369. lat (usec) : 250=17.92%, 500=0.11%
  370. cpu : usr=26.09%, sys=73.79%, ctx=102, majf=0, minf=12
  371. IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
  372. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  373. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
  374. issued rwts: total=0,262144,0,0 short=0,0,0,0 dropped=0,0,0,0
  375. latency : target=0, window=0, percentile=100.00%, depth=16
  376. seqWrite/sync: (groupid=2, jobs=1): err= 0: pid=393: Fri Mar 20 17:57:26 2020
  377. write: IOPS=5743, BW=22.4MiB/s (23.5MB/s)(1024MiB/45643msec); 0 zone resets
  378. slat (usec): min=4, max=191, avg= 8.08, stdev= 2.51
  379. clat (usec): min=2, max=54207, avg=2610.85, stdev=1120.32
  380. lat (usec): min=10, max=54258, avg=2619.07, stdev=1120.64
  381. clat percentiles (usec):
  382. | 1.00th=[ 2409], 5.00th=[ 2442], 10.00th=[ 2442], 20.00th=[ 2474],
  383. | 30.00th=[ 2507], 40.00th=[ 2540], 50.00th=[ 2573], 60.00th=[ 2606],
  384. | 70.00th=[ 2638], 80.00th=[ 2671], 90.00th=[ 2737], 95.00th=[ 2802],
  385. | 99.00th=[ 2999], 99.50th=[ 3130], 99.90th=[ 3654], 99.95th=[47449],
  386. | 99.99th=[53216]
  387. bw ( KiB/s): min=15330, max=22371, per=77.86%, avg=17886.32, stdev=811.75, samples=90
  388. iops : min= 3832, max= 5592, avg=4471.23, stdev=202.94, samples=90
  389. lat (usec) : 4=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
  390. lat (msec) : 2=0.01%, 4=99.91%, 10=0.03%, 50=0.01%, 100=0.04%
  391. fsync/fdatasync/sync_file_range:
  392. sync (nsec): min=64, max=13271, avg=105.09, stdev=162.18
  393. sync percentiles (nsec):
  394. | 1.00th=[ 72], 5.00th=[ 78], 10.00th=[ 81], 20.00th=[ 87],
  395. | 30.00th=[ 90], 40.00th=[ 93], 50.00th=[ 95], 60.00th=[ 98],
  396. | 70.00th=[ 101], 80.00th=[ 106], 90.00th=[ 114], 95.00th=[ 199],
  397. | 99.00th=[ 251], 99.50th=[ 262], 99.90th=[ 430], 99.95th=[ 462],
  398. | 99.99th=[11968]
  399. cpu : usr=2.66%, sys=14.23%, ctx=524158, majf=0, minf=16
  400. IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=200.0%, 32=0.0%, >=64=0.0%
  401. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  402. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
  403. issued rwts: total=0,262144,0,262129 short=0,0,0,0 dropped=0,0,0,0
  404. latency : target=0, window=0, percentile=100.00%, depth=16
  405. randRead: (groupid=3, jobs=1): err= 0: pid=48075: Fri Mar 20 17:57:26 2020
  406. read: IOPS=232k, BW=905MiB/s (949MB/s)(1024MiB/1131msec)
  407. slat (nsec): min=2563, max=99479, avg=3447.47, stdev=520.78
  408. clat (nsec): min=1330, max=174843, avg=65242.46, stdev=3139.29
  409. lat (usec): min=4, max=178, avg=68.75, stdev= 3.28
  410. clat percentiles (usec):
  411. | 1.00th=[ 61], 5.00th=[ 63], 10.00th=[ 63], 20.00th=[ 64],
  412. | 30.00th=[ 65], 40.00th=[ 65], 50.00th=[ 65], 60.00th=[ 66],
  413. | 70.00th=[ 67], 80.00th=[ 67], 90.00th=[ 68], 95.00th=[ 70],
  414. | 99.00th=[ 79], 99.50th=[ 83], 99.90th=[ 93], 99.95th=[ 97],
  415. | 99.99th=[ 116]
  416. bw ( KiB/s): min=921407, max=921407, per=99.38%, avg=921407.00, stdev= 0.00, samples=1
  417. iops : min=230351, max=230351, avg=230351.00, stdev= 0.00, samples=1
  418. lat (usec) : 2=0.01%, 10=0.01%, 20=0.01%, 50=0.01%, 100=99.97%
  419. lat (usec) : 250=0.03%
  420. cpu : usr=23.98%, sys=76.02%, ctx=5, majf=0, minf=26
  421. IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
  422. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  423. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
  424. issued rwts: total=262144,0,0,0 short=0,0,0,0 dropped=0,0,0,0
  425. latency : target=0, window=0, percentile=100.00%, depth=16
  426. randWrite: (groupid=4, jobs=1): err= 0: pid=48214: Fri Mar 20 17:57:26 2020
  427. write: IOPS=50.9k, BW=199MiB/s (208MB/s)(1024MiB/5153msec); 0 zone resets
  428. slat (usec): min=4, max=489, avg=17.49, stdev=24.57
  429. clat (nsec): min=1922, max=1795.4k, avg=295572.05, stdev=240817.06
  430. lat (usec): min=6, max=1859, avg=313.19, stdev=255.56
  431. clat percentiles (usec):
  432. | 1.00th=[ 104], 5.00th=[ 112], 10.00th=[ 118], 20.00th=[ 130],
  433. | 30.00th=[ 141], 40.00th=[ 151], 50.00th=[ 172], 60.00th=[ 231],
  434. | 70.00th=[ 314], 80.00th=[ 445], 90.00th=[ 676], 95.00th=[ 848],
  435. | 99.00th=[ 1090], 99.50th=[ 1172], 99.90th=[ 1287], 99.95th=[ 1336],
  436. | 99.99th=[ 1500]
  437. bw ( KiB/s): min=144441, max=232362, per=88.71%, avg=180514.80, stdev=27586.40, samples=10
  438. iops : min=36110, max=58090, avg=45128.30, stdev=6896.55, samples=10
  439. lat (usec) : 2=0.01%, 10=0.01%, 20=0.01%, 50=0.01%, 100=0.05%
  440. lat (usec) : 250=63.10%, 500=19.72%, 750=9.53%, 1000=5.48%
  441. lat (msec) : 2=2.11%
  442. cpu : usr=12.95%, sys=84.05%, ctx=21599, majf=0, minf=53
  443. IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
  444. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  445. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
  446. issued rwts: total=0,262144,0,0 short=0,0,0,0 dropped=0,0,0,0
  447. latency : target=0, window=0, percentile=100.00%, depth=16
  448. randWrite/sync: (groupid=5, jobs=1): err= 0: pid=17134: Fri Mar 20 17:57:26 2020
  449. write: IOPS=5005, BW=19.6MiB/s (20.5MB/s)(1024MiB/52370msec); 0 zone resets
  450. slat (usec): min=6, max=262, avg=12.04, stdev= 7.82
  451. clat (usec): min=2, max=96343, avg=2992.44, stdev=2143.21
  452. lat (usec): min=13, max=96374, avg=3004.66, stdev=2147.99
  453. clat percentiles (usec):
  454. | 1.00th=[ 2474], 5.00th=[ 2507], 10.00th=[ 2540], 20.00th=[ 2573],
  455. | 30.00th=[ 2606], 40.00th=[ 2638], 50.00th=[ 2704], 60.00th=[ 2769],
  456. | 70.00th=[ 2868], 80.00th=[ 2966], 90.00th=[ 3097], 95.00th=[ 3163],
  457. | 99.00th=[16188], 99.50th=[19006], 99.90th=[30278], 99.95th=[40633],
  458. | 99.99th=[47449]
  459. bw ( KiB/s): min= 8321, max=19659, per=77.55%, avg=15527.37, stdev=2785.32, samples=104
  460. iops : min= 2080, max= 4914, avg=3881.48, stdev=696.39, samples=104
  461. lat (usec) : 4=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
  462. lat (msec) : 2=0.01%, 4=98.41%, 10=0.21%, 20=1.00%, 50=0.37%
  463. lat (msec) : 100=0.01%
  464. fsync/fdatasync/sync_file_range:
  465. sync (nsec): min=23, max=25676, avg=129.02, stdev=351.93
  466. sync percentiles (nsec):
  467. | 1.00th=[ 76], 5.00th=[ 81], 10.00th=[ 86], 20.00th=[ 92],
  468. | 30.00th=[ 96], 40.00th=[ 100], 50.00th=[ 103], 60.00th=[ 106],
  469. | 70.00th=[ 110], 80.00th=[ 118], 90.00th=[ 217], 95.00th=[ 251],
  470. | 99.00th=[ 290], 99.50th=[ 334], 99.90th=[ 804], 99.95th=[11968],
  471. | 99.99th=[12608]
  472. cpu : usr=2.60%, sys=16.04%, ctx=524857, majf=0, minf=263
  473. IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=200.0%, 32=0.0%, >=64=0.0%
  474. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  475. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
  476. issued rwts: total=0,262144,0,262129 short=0,0,0,0 dropped=0,0,0,0
  477. latency : target=0, window=0, percentile=100.00%, depth=16
  478. randReadWrite: (groupid=6, jobs=1): err= 0: pid=37080: Fri Mar 20 17:57:26 2020
  479. read: IOPS=36.8k, BW=144MiB/s (151MB/s)(510MiB/3547msec)
  480. slat (usec): min=2, max=170, avg= 9.33, stdev=16.94
  481. clat (usec): min=15, max=1643, avg=202.85, stdev=188.80
  482. lat (usec): min=19, max=1657, avg=212.26, stdev=197.88
  483. clat percentiles (usec):
  484. | 1.00th=[ 79], 5.00th=[ 85], 10.00th=[ 89], 20.00th=[ 94],
  485. | 30.00th=[ 100], 40.00th=[ 109], 50.00th=[ 124], 60.00th=[ 141],
  486. | 70.00th=[ 182], 80.00th=[ 262], 90.00th=[ 457], 95.00th=[ 660],
  487. | 99.00th=[ 938], 99.50th=[ 1037], 99.90th=[ 1221], 99.95th=[ 1270],
  488. | 99.99th=[ 1385]
  489. bw ( KiB/s): min=83903, max=141138, per=71.67%, avg=105598.67, stdev=21152.63, samples=6
  490. iops : min=20975, max=35284, avg=26399.33, stdev=5288.13, samples=6
  491. write: IOPS=37.1k, BW=145MiB/s (152MB/s)(514MiB/3547msec); 0 zone resets
  492. slat (usec): min=4, max=245, avg=14.55, stdev=21.72
  493. clat (nsec): min=1448, max=1670.7k, avg=204385.09, stdev=190639.84
  494. lat (usec): min=6, max=1780, avg=219.04, stdev=204.59
  495. clat percentiles (usec):
  496. | 1.00th=[ 79], 5.00th=[ 85], 10.00th=[ 89], 20.00th=[ 95],
  497. | 30.00th=[ 100], 40.00th=[ 109], 50.00th=[ 125], 60.00th=[ 141],
  498. | 70.00th=[ 182], 80.00th=[ 265], 90.00th=[ 465], 95.00th=[ 668],
  499. | 99.00th=[ 938], 99.50th=[ 1029], 99.90th=[ 1205], 99.95th=[ 1287],
  500. | 99.99th=[ 1418]
  501. bw ( KiB/s): min=83855, max=142260, per=71.58%, avg=106151.67, stdev=21279.07, samples=6
  502. iops : min=20963, max=35565, avg=26537.67, stdev=5319.93, samples=6
  503. lat (usec) : 2=0.01%, 10=0.01%, 20=0.01%, 50=0.01%, 100=30.46%
  504. lat (usec) : 250=48.46%, 500=12.34%, 750=5.38%, 1000=2.73%
  505. lat (msec) : 2=0.63%
  506. cpu : usr=14.33%, sys=84.09%, ctx=7006, majf=0, minf=13
  507. IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
  508. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  509. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
  510. issued rwts: total=130649,131495,0,0 short=0,0,0,0 dropped=0,0,0,0
  511. latency : target=0, window=0, percentile=100.00%, depth=16
  512. randReadWrite/sync: (groupid=7, jobs=1): err= 0: pid=50570: Fri Mar 20 17:57:26 2020
  513. read: IOPS=4891, BW=19.1MiB/s (20.0MB/s)(512MiB/26809msec)
  514. slat (usec): min=2, max=136, avg= 5.64, stdev= 5.54
  515. clat (nsec): min=1639, max=62752k, avg=1442371.64, stdev=1142852.89
  516. lat (usec): min=5, max=62782, avg=1448.11, stdev=1145.60
  517. clat percentiles (usec):
  518. | 1.00th=[ 594], 5.00th=[ 775], 10.00th=[ 914], 20.00th=[ 1057],
  519. | 30.00th=[ 1139], 40.00th=[ 1254], 50.00th=[ 1319], 60.00th=[ 1418],
  520. | 70.00th=[ 1516], 80.00th=[ 1631], 90.00th=[ 1795], 95.00th=[ 1942],
  521. | 99.00th=[ 6849], 99.50th=[ 9503], 99.90th=[13566], 99.95th=[18744],
  522. | 99.99th=[38011]
  523. bw ( KiB/s): min=10648, max=22920, per=99.88%, avg=19541.83, stdev=3426.79, samples=53
  524. iops : min= 2662, max= 5730, avg=4885.43, stdev=856.69, samples=53
  525. write: IOPS=4886, BW=19.1MiB/s (20.0MB/s)(512MiB/26809msec); 0 zone resets
  526. slat (usec): min=5, max=426, avg=10.57, stdev= 7.53
  527. clat (usec): min=170, max=62993, avg=1621.18, stdev=1318.88
  528. lat (usec): min=178, max=63005, avg=1631.90, stdev=1322.95
  529. clat percentiles (usec):
  530. | 1.00th=[ 750], 5.00th=[ 938], 10.00th=[ 1074], 20.00th=[ 1221],
  531. | 30.00th=[ 1303], 40.00th=[ 1418], 50.00th=[ 1500], 60.00th=[ 1582],
  532. | 70.00th=[ 1680], 80.00th=[ 1795], 90.00th=[ 1958], 95.00th=[ 2114],
  533. | 99.00th=[ 7701], 99.50th=[10683], 99.90th=[15533], 99.95th=[28443],
  534. | 99.99th=[38536]
  535. bw ( KiB/s): min=10808, max=22352, per=99.87%, avg=19519.77, stdev=3393.38, samples=53
  536. iops : min= 2702, max= 5588, avg=4879.92, stdev=848.34, samples=53
  537. lat (usec) : 2=0.01%, 10=0.01%, 20=0.01%, 250=0.01%, 500=0.27%
  538. lat (usec) : 750=2.00%, 1000=9.71%
  539. lat (msec) : 2=81.93%, 4=4.73%, 10=0.85%, 20=0.45%, 50=0.06%
  540. lat (msec) : 100=0.01%
  541. fsync/fdatasync/sync_file_range:
  542. sync (nsec): min=24, max=24555, avg=70.13, stdev=167.04
  543. sync percentiles (nsec):
  544. | 1.00th=[ 27], 5.00th=[ 29], 10.00th=[ 29], 20.00th=[ 31],
  545. | 30.00th=[ 32], 40.00th=[ 34], 50.00th=[ 76], 60.00th=[ 83],
  546. | 70.00th=[ 88], 80.00th=[ 95], 90.00th=[ 105], 95.00th=[ 118],
  547. | 99.00th=[ 243], 99.50th=[ 270], 99.90th=[ 358], 99.95th=[ 470],
  548. | 99.99th=[11840]
  549. cpu : usr=3.66%, sys=17.88%, ctx=262352, majf=0, minf=17
  550. IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=200.0%, 32=0.0%, >=64=0.0%
  551. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  552. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
  553. issued rwts: total=131140,131004,0,262129 short=0,0,0,0 dropped=0,0,0,0
  554. latency : target=0, window=0, percentile=100.00%, depth=16
  555.  
  556. Run status group 0 (all jobs):
  557. READ: bw=1166MiB/s (1223MB/s), 1166MiB/s-1166MiB/s (1223MB/s-1223MB/s), io=1024MiB (1074MB), run=878-878msec
  558.  
  559. Run status group 1 (all jobs):
  560. WRITE: bw=582MiB/s (610MB/s), 582MiB/s-582MiB/s (610MB/s-610MB/s), io=1024MiB (1074MB), run=1760-1760msec
  561.  
  562. Run status group 2 (all jobs):
  563. WRITE: bw=22.4MiB/s (23.5MB/s), 22.4MiB/s-22.4MiB/s (23.5MB/s-23.5MB/s), io=1024MiB (1074MB), run=45643-45643msec
  564.  
  565. Run status group 3 (all jobs):
  566. READ: bw=905MiB/s (949MB/s), 905MiB/s-905MiB/s (949MB/s-949MB/s), io=1024MiB (1074MB), run=1131-1131msec
  567.  
  568. Run status group 4 (all jobs):
  569. WRITE: bw=199MiB/s (208MB/s), 199MiB/s-199MiB/s (208MB/s-208MB/s), io=1024MiB (1074MB), run=5153-5153msec
  570.  
  571. Run status group 5 (all jobs):
  572. WRITE: bw=19.6MiB/s (20.5MB/s), 19.6MiB/s-19.6MiB/s (20.5MB/s-20.5MB/s), io=1024MiB (1074MB), run=52370-52370msec
  573.  
  574. Run status group 6 (all jobs):
  575. READ: bw=144MiB/s (151MB/s), 144MiB/s-144MiB/s (151MB/s-151MB/s), io=510MiB (535MB), run=3547-3547msec
  576. WRITE: bw=145MiB/s (152MB/s), 145MiB/s-145MiB/s (152MB/s-152MB/s), io=514MiB (539MB), run=3547-3547msec
  577.  
  578. Run status group 7 (all jobs):
  579. READ: bw=19.1MiB/s (20.0MB/s), 19.1MiB/s-19.1MiB/s (20.0MB/s-20.0MB/s), io=512MiB (537MB), run=26809-26809msec
  580. WRITE: bw=19.1MiB/s (20.0MB/s), 19.1MiB/s-19.1MiB/s (20.0MB/s-20.0MB/s), io=512MiB (537MB), run=26809-26809msec
  581.  
  582. ======================================
  583. ssd_pool_r10_no_lz4_bench-4k-IOD16.txt
  584. ======================================
  585. seqRead: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
  586. seqWrite: (g=1): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
  587. seqWrite/sync: (g=2): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
  588. randRead: (g=3): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
  589. randWrite: (g=4): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
  590. randWrite/sync: (g=5): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
  591. randReadWrite: (g=6): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
  592. randReadWrite/sync: (g=7): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
  593. fio-3.12
  594. Starting 8 processes
  595.  
  596. seqRead: (groupid=0, jobs=1): err= 0: pid=52114: Fri Mar 20 18:48:26 2020
  597. read: IOPS=307k, BW=1200MiB/s (1259MB/s)(1024MiB/853msec)
  598. slat (nsec): min=2127, max=44137, avg=2467.40, stdev=454.31
  599. clat (nsec): min=1544, max=97393, avg=49329.37, stdev=2918.08
  600. lat (usec): min=4, max=108, avg=51.85, stdev= 3.05
  601. clat percentiles (nsec):
  602. | 1.00th=[45824], 5.00th=[45824], 10.00th=[46336], 20.00th=[46848],
  603. | 30.00th=[47360], 40.00th=[48384], 50.00th=[48896], 60.00th=[49920],
  604. | 70.00th=[50432], 80.00th=[51456], 90.00th=[52480], 95.00th=[53504],
  605. | 99.00th=[57088], 99.50th=[59648], 99.90th=[82432], 99.95th=[85504],
  606. | 99.99th=[90624]
  607. bw ( MiB/s): min= 1201, max= 1201, per=100.00%, avg=1201.41, stdev= 0.00, samples=1
  608. iops : min=307561, max=307561, avg=307561.00, stdev= 0.00, samples=1
  609. lat (usec) : 2=0.01%, 10=0.01%, 20=0.01%, 50=61.02%, 100=38.98%
  610. cpu : usr=34.62%, sys=65.14%, ctx=4, majf=0, minf=24
  611. IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
  612. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  613. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
  614. issued rwts: total=262144,0,0,0 short=0,0,0,0 dropped=0,0,0,0
  615. latency : target=0, window=0, percentile=100.00%, depth=16
  616. seqWrite: (groupid=1, jobs=1): err= 0: pid=52132: Fri Mar 20 18:48:26 2020
  617. write: IOPS=149k, BW=583MiB/s (612MB/s)(1024MiB/1755msec); 0 zone resets
  618. slat (usec): min=4, max=112, avg= 5.07, stdev= 2.16
  619. clat (usec): min=2, max=281, avg=100.99, stdev=12.43
  620. lat (usec): min=9, max=289, avg=106.13, stdev=12.94
  621. clat percentiles (usec):
  622. | 1.00th=[ 93], 5.00th=[ 94], 10.00th=[ 94], 20.00th=[ 94],
  623. | 30.00th=[ 95], 40.00th=[ 97], 50.00th=[ 98], 60.00th=[ 100],
  624. | 70.00th=[ 101], 80.00th=[ 102], 90.00th=[ 113], 95.00th=[ 121],
  625. | 99.00th=[ 167], 99.50th=[ 188], 99.90th=[ 215], 99.95th=[ 225],
  626. | 99.99th=[ 247]
  627. bw ( KiB/s): min=512551, max=561497, per=88.67%, avg=529759.00, stdev=27518.51, samples=3
  628. iops : min=128137, max=140374, avg=132439.33, stdev=6879.78, samples=3
  629. lat (usec) : 4=0.01%, 20=0.01%, 50=0.01%, 100=60.52%, 250=39.47%
  630. lat (usec) : 500=0.01%
  631. cpu : usr=29.99%, sys=69.84%, ctx=114, majf=0, minf=12
  632. IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
  633. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  634. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
  635. issued rwts: total=0,262144,0,0 short=0,0,0,0 dropped=0,0,0,0
  636. latency : target=0, window=0, percentile=100.00%, depth=16
  637. seqWrite/sync: (groupid=2, jobs=1): err= 0: pid=54780: Fri Mar 20 18:48:26 2020
  638. write: IOPS=5471, BW=21.4MiB/s (22.4MB/s)(1024MiB/47908msec); 0 zone resets
  639. slat (usec): min=4, max=114, avg= 9.12, stdev= 4.07
  640. clat (usec): min=2, max=52486, avg=2741.86, stdev=1015.60
  641. lat (usec): min=9, max=52567, avg=2751.13, stdev=1016.14
  642. clat percentiles (usec):
  643. | 1.00th=[ 2409], 5.00th=[ 2474], 10.00th=[ 2507], 20.00th=[ 2540],
  644. | 30.00th=[ 2606], 40.00th=[ 2638], 50.00th=[ 2671], 60.00th=[ 2737],
  645. | 70.00th=[ 2802], 80.00th=[ 2900], 90.00th=[ 2999], 95.00th=[ 3064],
  646. | 99.00th=[ 3195], 99.50th=[ 3294], 99.90th=[ 3949], 99.95th=[41157],
  647. | 99.99th=[50594]
  648. bw ( KiB/s): min=12453, max=19941, per=67.20%, avg=14707.46, stdev=1028.30, samples=95
  649. iops : min= 3113, max= 4985, avg=3676.51, stdev=257.11, samples=95
  650. lat (usec) : 4=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
  651. lat (msec) : 2=0.01%, 4=99.90%, 10=0.05%, 50=0.04%, 100=0.01%
  652. fsync/fdatasync/sync_file_range:
  653. sync (nsec): min=35, max=23688, avg=121.64, stdev=414.06
  654. sync percentiles (nsec):
  655. | 1.00th=[ 59], 5.00th=[ 65], 10.00th=[ 70], 20.00th=[ 75],
  656. | 30.00th=[ 81], 40.00th=[ 85], 50.00th=[ 90], 60.00th=[ 96],
  657. | 70.00th=[ 103], 80.00th=[ 121], 90.00th=[ 193], 95.00th=[ 207],
  658. | 99.00th=[ 243], 99.50th=[ 358], 99.90th=[11584], 99.95th=[12096],
  659. | 99.99th=[12480]
  660. cpu : usr=2.92%, sys=15.03%, ctx=523782, majf=0, minf=55
  661. IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=200.0%, 32=0.0%, >=64=0.0%
  662. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  663. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
  664. issued rwts: total=0,262144,0,262129 short=0,0,0,0 dropped=0,0,0,0
  665. latency : target=0, window=0, percentile=100.00%, depth=16
  666. randRead: (groupid=3, jobs=1): err= 0: pid=55449: Fri Mar 20 18:48:26 2020
  667. read: IOPS=239k, BW=932MiB/s (977MB/s)(1024MiB/1099msec)
  668. slat (nsec): min=2419, max=32817, avg=3323.98, stdev=460.96
  669. clat (nsec): min=1482, max=176135, avg=63420.75, stdev=2863.94
  670. lat (usec): min=4, max=179, avg=66.80, stdev= 2.99
  671. clat percentiles (usec):
  672. | 1.00th=[ 60], 5.00th=[ 61], 10.00th=[ 62], 20.00th=[ 62],
  673. | 30.00th=[ 63], 40.00th=[ 63], 50.00th=[ 64], 60.00th=[ 64],
  674. | 70.00th=[ 64], 80.00th=[ 65], 90.00th=[ 66], 95.00th=[ 68],
  675. | 99.00th=[ 76], 99.50th=[ 79], 99.90th=[ 89], 99.95th=[ 92],
  676. | 99.99th=[ 121]
  677. bw ( KiB/s): min=943739, max=945838, per=99.02%, avg=944788.50, stdev=1484.22, samples=2
  678. iops : min=235934, max=236459, avg=236196.50, stdev=371.23, samples=2
  679. lat (usec) : 2=0.01%, 10=0.01%, 20=0.01%, 50=0.01%, 100=99.98%
  680. lat (usec) : 250=0.02%
  681. cpu : usr=19.58%, sys=80.24%, ctx=4, majf=0, minf=26
  682. IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
  683. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  684. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
  685. issued rwts: total=262144,0,0,0 short=0,0,0,0 dropped=0,0,0,0
  686. latency : target=0, window=0, percentile=100.00%, depth=16
  687. randWrite: (groupid=4, jobs=1): err= 0: pid=55455: Fri Mar 20 18:48:26 2020
  688. write: IOPS=51.4k, BW=201MiB/s (210MB/s)(1024MiB/5101msec); 0 zone resets
  689. slat (usec): min=4, max=564, avg=17.36, stdev=24.06
  690. clat (nsec): min=1602, max=1598.1k, avg=292591.99, stdev=227751.12
  691. lat (usec): min=6, max=1679, avg=310.08, stdev=241.74
  692. clat percentiles (usec):
  693. | 1.00th=[ 104], 5.00th=[ 112], 10.00th=[ 121], 20.00th=[ 133],
  694. | 30.00th=[ 141], 40.00th=[ 151], 50.00th=[ 192], 60.00th=[ 231],
  695. | 70.00th=[ 310], 80.00th=[ 441], 90.00th=[ 660], 95.00th=[ 816],
  696. | 99.00th=[ 1020], 99.50th=[ 1074], 99.90th=[ 1172], 99.95th=[ 1221],
  697. | 99.99th=[ 1303]
  698. bw ( KiB/s): min=137582, max=214317, per=79.17%, avg=162740.00, stdev=22069.79, samples=9
  699. iops : min=34395, max=53579, avg=40684.67, stdev=5517.45, samples=9
  700. lat (usec) : 2=0.01%, 10=0.01%, 20=0.01%, 50=0.01%, 100=0.02%
  701. lat (usec) : 250=62.32%, 500=20.82%, 750=9.89%, 1000=5.73%
  702. lat (msec) : 2=1.22%
  703. cpu : usr=10.65%, sys=86.27%, ctx=21754, majf=0, minf=12
  704. IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
  705. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  706. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
  707. issued rwts: total=0,262144,0,0 short=0,0,0,0 dropped=0,0,0,0
  708. latency : target=0, window=0, percentile=100.00%, depth=16
  709. randWrite/sync: (groupid=5, jobs=1): err= 0: pid=25235: Fri Mar 20 18:48:26 2020
  710. write: IOPS=5125, BW=20.0MiB/s (20.0MB/s)(1024MiB/51145msec); 0 zone resets
  711. slat (usec): min=5, max=217, avg=11.13, stdev= 7.03
  712. clat (usec): min=2, max=108760, avg=2921.59, stdev=2206.65
  713. lat (usec): min=9, max=108787, avg=2932.86, stdev=2211.02
  714. clat percentiles (usec):
  715. | 1.00th=[ 2442], 5.00th=[ 2474], 10.00th=[ 2507], 20.00th=[ 2540],
  716. | 30.00th=[ 2573], 40.00th=[ 2606], 50.00th=[ 2638], 60.00th=[ 2704],
  717. | 70.00th=[ 2769], 80.00th=[ 2835], 90.00th=[ 2933], 95.00th=[ 3064],
  718. | 99.00th=[16057], 99.50th=[19530], 99.90th=[33162], 99.95th=[41681],
  719. | 99.99th=[46924]
  720. bw ( KiB/s): min= 7925, max=17882, per=71.88%, avg=14737.25, stdev=2861.31, samples=101
  721. iops : min= 1981, max= 4470, avg=3683.93, stdev=715.36, samples=101
  722. lat (usec) : 4=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
  723. lat (msec) : 2=0.01%, 4=98.48%, 10=0.17%, 20=0.90%, 50=0.43%
  724. lat (msec) : 100=0.01%, 250=0.01%
  725. fsync/fdatasync/sync_file_range:
  726. sync (nsec): min=49, max=14708, avg=96.61, stdev=215.60
  727. sync percentiles (nsec):
  728. | 1.00th=[ 60], 5.00th=[ 65], 10.00th=[ 67], 20.00th=[ 70],
  729. | 30.00th=[ 74], 40.00th=[ 78], 50.00th=[ 82], 60.00th=[ 85],
  730. | 70.00th=[ 89], 80.00th=[ 97], 90.00th=[ 119], 95.00th=[ 201],
  731. | 99.00th=[ 262], 99.50th=[ 286], 99.90th=[ 470], 99.95th=[ 604],
  732. | 99.99th=[12224]
  733. cpu : usr=2.54%, sys=14.97%, ctx=524909, majf=0, minf=147
  734. IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=200.0%, 32=0.0%, >=64=0.0%
  735. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  736. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
  737. issued rwts: total=0,262144,0,262129 short=0,0,0,0 dropped=0,0,0,0
  738. latency : target=0, window=0, percentile=100.00%, depth=16
  739. randReadWrite: (groupid=6, jobs=1): err= 0: pid=36675: Fri Mar 20 18:48:26 2020
  740. read: IOPS=36.5k, BW=142MiB/s (149MB/s)(510MiB/3582msec)
  741. slat (usec): min=2, max=258, avg= 9.48, stdev=17.80
  742. clat (usec): min=8, max=1922, avg=205.58, stdev=202.63
  743. lat (usec): min=10, max=1960, avg=215.14, stdev=212.23
  744. clat percentiles (usec):
  745. | 1.00th=[ 79], 5.00th=[ 85], 10.00th=[ 89], 20.00th=[ 94],
  746. | 30.00th=[ 100], 40.00th=[ 108], 50.00th=[ 122], 60.00th=[ 141],
  747. | 70.00th=[ 176], 80.00th=[ 260], 90.00th=[ 465], 95.00th=[ 668],
  748. | 99.00th=[ 1057], 99.50th=[ 1188], 99.90th=[ 1418], 99.95th=[ 1500],
  749. | 99.99th=[ 1713]
  750. bw ( KiB/s): min=114460, max=179324, per=97.46%, avg=142194.00, stdev=24350.26, samples=7
  751. iops : min=28615, max=44831, avg=35548.14, stdev=6087.59, samples=7
  752. write: IOPS=36.7k, BW=143MiB/s (150MB/s)(514MiB/3582msec); 0 zone resets
  753. slat (usec): min=4, max=408, avg=14.75, stdev=23.09
  754. clat (nsec): min=1636, max=1881.0k, avg=205589.73, stdev=202614.39
  755. lat (usec): min=6, max=2019, avg=220.44, stdev=217.79
  756. clat percentiles (usec):
  757. | 1.00th=[ 79], 5.00th=[ 85], 10.00th=[ 89], 20.00th=[ 95],
  758. | 30.00th=[ 100], 40.00th=[ 108], 50.00th=[ 123], 60.00th=[ 141],
  759. | 70.00th=[ 176], 80.00th=[ 260], 90.00th=[ 465], 95.00th=[ 660],
  760. | 99.00th=[ 1057], 99.50th=[ 1188], 99.90th=[ 1418], 99.95th=[ 1500],
  761. | 99.99th=[ 1680]
  762. bw ( KiB/s): min=114873, max=179848, per=97.48%, avg=143138.00, stdev=24381.25, samples=7
  763. iops : min=28718, max=44962, avg=35784.29, stdev=6095.42, samples=7
  764. lat (usec) : 2=0.01%, 10=0.01%, 20=0.01%, 50=0.01%, 100=30.53%
  765. lat (usec) : 250=48.73%, 500=11.80%, 750=5.41%, 1000=2.26%
  766. lat (msec) : 2=1.26%
  767. cpu : usr=15.08%, sys=83.52%, ctx=6481, majf=0, minf=55
  768. IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
  769. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  770. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
  771. issued rwts: total=130650,131494,0,0 short=0,0,0,0 dropped=0,0,0,0
  772. latency : target=0, window=0, percentile=100.00%, depth=16
  773. randReadWrite/sync: (groupid=7, jobs=1): err= 0: pid=50229: Fri Mar 20 18:48:26 2020
  774. read: IOPS=4911, BW=19.2MiB/s (20.1MB/s)(513MiB/26752msec)
  775. slat (usec): min=2, max=129, avg= 5.51, stdev= 5.27
  776. clat (nsec): min=1820, max=38469k, avg=1439190.66, stdev=1151105.64
  777. lat (usec): min=4, max=38476, avg=1444.80, stdev=1153.56
  778. clat percentiles (usec):
  779. | 1.00th=[ 586], 5.00th=[ 775], 10.00th=[ 906], 20.00th=[ 1057],
  780. | 30.00th=[ 1139], 40.00th=[ 1254], 50.00th=[ 1319], 60.00th=[ 1418],
  781. | 70.00th=[ 1516], 80.00th=[ 1631], 90.00th=[ 1795], 95.00th=[ 1958],
  782. | 99.00th=[ 6718], 99.50th=[ 9503], 99.90th=[13698], 99.95th=[17433],
  783. | 99.99th=[34866]
  784. bw ( KiB/s): min=10752, max=23256, per=99.82%, avg=19607.70, stdev=3450.31, samples=53
  785. iops : min= 2688, max= 5814, avg=4901.91, stdev=862.57, samples=53
  786. write: IOPS=4887, BW=19.1MiB/s (20.0MB/s)(511MiB/26752msec); 0 zone resets
  787. slat (usec): min=5, max=145, avg=10.37, stdev= 7.03
  788. clat (usec): min=203, max=48764, avg=1619.09, stdev=1296.98
  789. lat (usec): min=211, max=48807, avg=1629.59, stdev=1300.65
  790. clat percentiles (usec):
  791. | 1.00th=[ 742], 5.00th=[ 930], 10.00th=[ 1074], 20.00th=[ 1221],
  792. | 30.00th=[ 1303], 40.00th=[ 1418], 50.00th=[ 1483], 60.00th=[ 1582],
  793. | 70.00th=[ 1680], 80.00th=[ 1795], 90.00th=[ 1958], 95.00th=[ 2114],
  794. | 99.00th=[ 7898], 99.50th=[10552], 99.90th=[15270], 99.95th=[30278],
  795. | 99.99th=[36439]
  796. bw ( KiB/s): min=10608, max=22456, per=99.87%, avg=19526.21, stdev=3426.11, samples=53
  797. iops : min= 2652, max= 5614, avg=4881.55, stdev=856.54, samples=53
  798. lat (usec) : 2=0.01%, 10=0.01%, 20=0.01%, 250=0.01%, 500=0.30%
  799. lat (usec) : 750=2.11%, 1000=9.58%
  800. lat (msec) : 2=81.63%, 4=5.02%, 10=0.84%, 20=0.45%, 50=0.06%
  801. fsync/fdatasync/sync_file_range:
  802. sync (nsec): min=17, max=14839, avg=60.33, stdev=153.35
  803. sync percentiles (nsec):
  804. | 1.00th=[ 27], 5.00th=[ 28], 10.00th=[ 29], 20.00th=[ 30],
  805. | 30.00th=[ 31], 40.00th=[ 34], 50.00th=[ 59], 60.00th=[ 65],
  806. | 70.00th=[ 71], 80.00th=[ 77], 90.00th=[ 86], 95.00th=[ 105],
  807. | 99.00th=[ 207], 99.50th=[ 266], 99.90th=[ 338], 99.95th=[ 422],
  808. | 99.99th=[11584]
  809. cpu : usr=3.16%, sys=17.69%, ctx=261772, majf=0, minf=16
  810. IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=200.0%, 32=0.0%, >=64=0.0%
  811. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  812. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
  813. issued rwts: total=131384,130760,0,262129 short=0,0,0,0 dropped=0,0,0,0
  814. latency : target=0, window=0, percentile=100.00%, depth=16
  815.  
  816. Run status group 0 (all jobs):
  817. READ: bw=1200MiB/s (1259MB/s), 1200MiB/s-1200MiB/s (1259MB/s-1259MB/s), io=1024MiB (1074MB), run=853-853msec
  818.  
  819. Run status group 1 (all jobs):
  820. WRITE: bw=583MiB/s (612MB/s), 583MiB/s-583MiB/s (612MB/s-612MB/s), io=1024MiB (1074MB), run=1755-1755msec
  821.  
  822. Run status group 2 (all jobs):
  823. WRITE: bw=21.4MiB/s (22.4MB/s), 21.4MiB/s-21.4MiB/s (22.4MB/s-22.4MB/s), io=1024MiB (1074MB), run=47908-47908msec
  824.  
  825. Run status group 3 (all jobs):
  826. READ: bw=932MiB/s (977MB/s), 932MiB/s-932MiB/s (977MB/s-977MB/s), io=1024MiB (1074MB), run=1099-1099msec
  827.  
  828. Run status group 4 (all jobs):
  829. WRITE: bw=201MiB/s (210MB/s), 201MiB/s-201MiB/s (210MB/s-210MB/s), io=1024MiB (1074MB), run=5101-5101msec
  830.  
  831. Run status group 5 (all jobs):
  832. WRITE: bw=20.0MiB/s (20.0MB/s), 20.0MiB/s-20.0MiB/s (20.0MB/s-20.0MB/s), io=1024MiB (1074MB), run=51145-51145msec
  833.  
  834. Run status group 6 (all jobs):
  835. READ: bw=142MiB/s (149MB/s), 142MiB/s-142MiB/s (149MB/s-149MB/s), io=510MiB (535MB), run=3582-3582msec
  836. WRITE: bw=143MiB/s (150MB/s), 143MiB/s-143MiB/s (150MB/s-150MB/s), io=514MiB (539MB), run=3582-3582msec
  837.  
  838. Run status group 7 (all jobs):
  839. READ: bw=19.2MiB/s (20.1MB/s), 19.2MiB/s-19.2MiB/s (20.1MB/s-20.1MB/s), io=513MiB (538MB), run=26752-26752msec
  840. WRITE: bw=19.1MiB/s (20.0MB/s), 19.1MiB/s-19.1MiB/s (20.0MB/s-20.0MB/s), io=511MiB (536MB), run=26752-26752msec
  841.  
  842. =================================
  843. md_tlvm_6disks_bench-4k-IOD16.txt
  844. =================================
  845. seqRead: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
  846. seqWrite: (g=1): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
  847. seqWrite/sync: (g=2): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
  848. randRead: (g=3): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
  849. randWrite: (g=4): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
  850. randWrite/sync: (g=5): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
  851. randReadWrite: (g=6): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
  852. randReadWrite/sync: (g=7): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
  853. fio-3.12
  854. Starting 8 processes
  855.  
  856. seqRead: (groupid=0, jobs=1): err= 0: pid=9385: Sat Mar 21 09:21:11 2020
  857. read: IOPS=248k, BW=970MiB/s (1017MB/s)(1024MiB/1056msec)
  858. slat (nsec): min=1121, max=3107.1k, avg=3175.13, stdev=23555.47
  859. clat (nsec): min=1384, max=3143.7k, avg=60943.09, stdev=88011.00
  860. lat (usec): min=2, max=3144, avg=64.16, stdev=90.63
  861. clat percentiles (usec):
  862. | 1.00th=[ 30], 5.00th=[ 30], 10.00th=[ 30], 20.00th=[ 30],
  863. | 30.00th=[ 30], 40.00th=[ 31], 50.00th=[ 35], 60.00th=[ 39],
  864. | 70.00th=[ 43], 80.00th=[ 50], 90.00th=[ 225], 95.00th=[ 233],
  865. | 99.00th=[ 247], 99.50th=[ 251], 99.90th=[ 498], 99.95th=[ 1844],
  866. | 99.99th=[ 3032]
  867. bw ( KiB/s): min=998210, max=998210, per=100.00%, avg=998210.00, stdev= 0.00, samples=1
  868. iops : min=249552, max=249552, avg=249552.00, stdev= 0.00, samples=1
  869. lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=80.43%
  870. lat (usec) : 100=7.82%, 250=11.18%, 500=0.46%, 750=0.01%, 1000=0.03%
  871. lat (msec) : 2=0.01%, 4=0.05%
  872. cpu : usr=25.50%, sys=71.56%, ctx=48, majf=0, minf=27
  873. IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
  874. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  875. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
  876. issued rwts: total=262144,0,0,0 short=0,0,0,0 dropped=0,0,0,0
  877. latency : target=0, window=0, percentile=100.00%, depth=16
  878. seqWrite: (groupid=1, jobs=1): err= 0: pid=9418: Sat Mar 21 09:21:11 2020
  879. write: IOPS=284k, BW=1108MiB/s (1162MB/s)(1024MiB/924msec); 0 zone resets
  880. slat (nsec): min=1590, max=30791, avg=2184.64, stdev=478.01
  881. clat (nsec): min=1375, max=109605, avg=53371.78, stdev=2138.40
  882. lat (usec): min=3, max=112, avg=55.61, stdev= 2.21
  883. clat percentiles (nsec):
  884. | 1.00th=[51456], 5.00th=[51456], 10.00th=[51968], 20.00th=[51968],
  885. | 30.00th=[52480], 40.00th=[52480], 50.00th=[52992], 60.00th=[53504],
  886. | 70.00th=[53504], 80.00th=[54016], 90.00th=[55040], 95.00th=[56064],
  887. | 99.00th=[60672], 99.50th=[65280], 99.90th=[78336], 99.95th=[83456],
  888. | 99.99th=[90624]
  889. bw ( MiB/s): min= 1107, max= 1107, per=99.94%, avg=1107.61, stdev= 0.00, samples=1
  890. iops : min=283549, max=283549, avg=283549.00, stdev= 0.00, samples=1
  891. lat (usec) : 2=0.01%, 10=0.01%, 20=0.01%, 50=0.01%, 100=99.99%
  892. lat (usec) : 250=0.01%
  893. cpu : usr=40.20%, sys=59.59%, ctx=2, majf=0, minf=12
  894. IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
  895. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  896. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
  897. issued rwts: total=0,262144,0,0 short=0,0,0,0 dropped=0,0,0,0
  898. latency : target=0, window=0, percentile=100.00%, depth=16
  899. seqWrite/sync: (groupid=2, jobs=1): err= 0: pid=9528: Sat Mar 21 09:21:11 2020
  900. write: IOPS=6618, BW=25.9MiB/s (27.1MB/s)(1024MiB/39607msec); 0 zone resets
  901. slat (nsec): min=2333, max=35529, avg=5667.20, stdev=1132.58
  902. clat (usec): min=2, max=7470, avg=2267.27, stdev=158.14
  903. lat (usec): min=7, max=7476, avg=2273.06, stdev=158.62
  904. clat percentiles (usec):
  905. | 1.00th=[ 2147], 5.00th=[ 2180], 10.00th=[ 2180], 20.00th=[ 2180],
  906. | 30.00th=[ 2212], 40.00th=[ 2212], 50.00th=[ 2212], 60.00th=[ 2245],
  907. | 70.00th=[ 2245], 80.00th=[ 2278], 90.00th=[ 2376], 95.00th=[ 2474],
  908. | 99.00th=[ 2999], 99.50th=[ 3064], 99.90th=[ 3294], 99.95th=[ 3425],
  909. | 99.99th=[ 6194]
  910. bw ( KiB/s): min=17480, max=26117, per=84.03%, avg=22245.27, stdev=921.57, samples=79
  911. iops : min= 4370, max= 6529, avg=5560.96, stdev=230.40, samples=79
  912. lat (usec) : 4=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
  913. lat (msec) : 2=0.01%, 4=99.96%, 10=0.03%
  914. fsync/fdatasync/sync_file_range:
  915. sync (nsec): min=70, max=12275, avg=156.62, stdev=76.13
  916. sync percentiles (nsec):
  917. | 1.00th=[ 94], 5.00th=[ 109], 10.00th=[ 116], 20.00th=[ 126],
  918. | 30.00th=[ 133], 40.00th=[ 141], 50.00th=[ 149], 60.00th=[ 155],
  919. | 70.00th=[ 165], 80.00th=[ 179], 90.00th=[ 205], 95.00th=[ 245],
  920. | 99.00th=[ 302], 99.50th=[ 330], 99.90th=[ 438], 99.95th=[ 510],
  921. | 99.99th=[ 620]
  922. cpu : usr=3.19%, sys=18.69%, ctx=781370, majf=0, minf=55
  923. IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=200.0%, 32=0.0%, >=64=0.0%
  924. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  925. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
  926. issued rwts: total=0,262144,0,262129 short=0,0,0,0 dropped=0,0,0,0
  927. latency : target=0, window=0, percentile=100.00%, depth=16
  928. randRead: (groupid=3, jobs=1): err= 0: pid=10709: Sat Mar 21 09:21:11 2020
  929. read: IOPS=8153, BW=31.9MiB/s (33.4MB/s)(1024MiB/32150msec)
  930. slat (usec): min=36, max=2494, avg=120.34, stdev=14.85
  931. clat (usec): min=2, max=4390, avg=1840.56, stdev=74.01
  932. lat (usec): min=72, max=4516, avg=1961.17, stdev=76.80
  933. clat percentiles (usec):
  934. | 1.00th=[ 1680], 5.00th=[ 1745], 10.00th=[ 1778], 20.00th=[ 1795],
  935. | 30.00th=[ 1811], 40.00th=[ 1827], 50.00th=[ 1827], 60.00th=[ 1844],
  936. | 70.00th=[ 1860], 80.00th=[ 1893], 90.00th=[ 1926], 95.00th=[ 1958],
  937. | 99.00th=[ 2024], 99.50th=[ 2057], 99.90th=[ 2147], 99.95th=[ 2474],
  938. | 99.99th=[ 3458]
  939. bw ( KiB/s): min=28801, max=31921, per=89.61%, avg=29225.56, stdev=445.98, samples=64
  940. iops : min= 7200, max= 7980, avg=7306.03, stdev=111.47, samples=64
  941. lat (usec) : 4=0.01%, 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%
  942. lat (usec) : 1000=0.01%
  943. lat (msec) : 2=98.00%, 4=1.99%, 10=0.01%
  944. cpu : usr=2.05%, sys=9.50%, ctx=262152, majf=0, minf=74
  945. IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
  946. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  947. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
  948. issued rwts: total=262144,0,0,0 short=0,0,0,0 dropped=0,0,0,0
  949. latency : target=0, window=0, percentile=100.00%, depth=16
  950. randWrite: (groupid=4, jobs=1): err= 0: pid=11810: Sat Mar 21 09:21:11 2020
  951. write: IOPS=268k, BW=1047MiB/s (1098MB/s)(1024MiB/978msec); 0 zone resets
  952. slat (nsec): min=1650, max=44672, avg=2334.02, stdev=505.82
  953. clat (nsec): min=1406, max=116633, avg=56454.94, stdev=2219.14
  954. lat (usec): min=3, max=119, avg=58.84, stdev= 2.31
  955. clat percentiles (nsec):
  956. | 1.00th=[54016], 5.00th=[54528], 10.00th=[54528], 20.00th=[55040],
  957. | 30.00th=[55040], 40.00th=[55552], 50.00th=[56064], 60.00th=[56576],
  958. | 70.00th=[56576], 80.00th=[57600], 90.00th=[58624], 95.00th=[59648],
  959. | 99.00th=[64768], 99.50th=[68096], 99.90th=[75264], 99.95th=[80384],
  960. | 99.99th=[98816]
  961. bw ( MiB/s): min= 1041, max= 1041, per=99.42%, avg=1041.01, stdev= 0.00, samples=1
  962. iops : min=266498, max=266498, avg=266498.00, stdev= 0.00, samples=1
  963. lat (usec) : 2=0.01%, 10=0.01%, 20=0.01%, 50=0.01%, 100=99.99%
  964. lat (usec) : 250=0.01%
  965. cpu : usr=42.37%, sys=57.52%, ctx=3, majf=0, minf=10
  966. IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
  967. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  968. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
  969. issued rwts: total=0,262144,0,0 short=0,0,0,0 dropped=0,0,0,0
  970. latency : target=0, window=0, percentile=100.00%, depth=16
  971. randWrite/sync: (groupid=5, jobs=1): err= 0: pid=11840: Sat Mar 21 09:21:11 2020
  972. write: IOPS=6576, BW=25.7MiB/s (26.9MB/s)(1024MiB/39863msec); 0 zone resets
  973. slat (usec): min=2, max=113, avg= 5.63, stdev= 1.28
  974. clat (usec): min=2, max=6988, avg=2281.86, stdev=167.42
  975. lat (usec): min=8, max=6994, avg=2287.61, stdev=168.04
  976. clat percentiles (usec):
  977. | 1.00th=[ 2180], 5.00th=[ 2180], 10.00th=[ 2180], 20.00th=[ 2212],
  978. | 30.00th=[ 2212], 40.00th=[ 2212], 50.00th=[ 2245], 60.00th=[ 2245],
  979. | 70.00th=[ 2278], 80.00th=[ 2278], 90.00th=[ 2409], 95.00th=[ 2573],
  980. | 99.00th=[ 3032], 99.50th=[ 3064], 99.90th=[ 3195], 99.95th=[ 3326],
  981. | 99.99th=[ 5080]
  982. bw ( KiB/s): min=16524, max=25596, per=80.38%, avg=21142.91, stdev=1072.80, samples=79
  983. iops : min= 4131, max= 6399, avg=5285.37, stdev=268.18, samples=79
  984. lat (usec) : 4=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
  985. lat (msec) : 2=0.01%, 4=99.97%, 10=0.02%
  986. fsync/fdatasync/sync_file_range:
  987. sync (nsec): min=28, max=20349, avg=166.10, stdev=78.79
  988. sync percentiles (nsec):
  989. | 1.00th=[ 93], 5.00th=[ 110], 10.00th=[ 120], 20.00th=[ 133],
  990. | 30.00th=[ 141], 40.00th=[ 151], 50.00th=[ 159], 60.00th=[ 169],
  991. | 70.00th=[ 179], 80.00th=[ 193], 90.00th=[ 221], 95.00th=[ 249],
  992. | 99.00th=[ 306], 99.50th=[ 350], 99.90th=[ 482], 99.95th=[ 524],
  993. | 99.99th=[ 588]
  994. cpu : usr=3.77%, sys=18.18%, ctx=779777, majf=0, minf=61
  995. IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=200.0%, 32=0.0%, >=64=0.0%
  996. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  997. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
  998. issued rwts: total=0,262144,0,262129 short=0,0,0,0 dropped=0,0,0,0
  999. latency : target=0, window=0, percentile=100.00%, depth=16
  1000. randReadWrite: (groupid=6, jobs=1): err= 0: pid=13122: Sat Mar 21 09:21:11 2020
  1001. read: IOPS=7749, BW=30.3MiB/s (31.7MB/s)(513MiB/16957msec)
  1002. slat (usec): min=38, max=2878, avg=121.47, stdev=15.44
  1003. clat (usec): min=8, max=3674, avg=971.91, stdev=237.47
  1004. lat (usec): min=138, max=3836, avg=1093.63, stdev=238.32
  1005. clat percentiles (usec):
  1006. | 1.00th=[ 420], 5.00th=[ 562], 10.00th=[ 660], 20.00th=[ 775],
  1007. | 30.00th=[ 857], 40.00th=[ 906], 50.00th=[ 979], 60.00th=[ 1029],
  1008. | 70.00th=[ 1106], 80.00th=[ 1172], 90.00th=[ 1270], 95.00th=[ 1369],
  1009. | 99.00th=[ 1516], 99.50th=[ 1582], 99.90th=[ 1696], 99.95th=[ 1745],
  1010. | 99.99th=[ 1926]
  1011. bw ( KiB/s): min=28537, max=30559, per=93.54%, avg=28996.73, stdev=329.64, samples=33
  1012. iops : min= 7134, max= 7639, avg=7248.82, stdev=82.34, samples=33
  1013. write: IOPS=7709, BW=30.1MiB/s (31.6MB/s)(511MiB/16957msec); 0 zone resets
  1014. slat (nsec): min=1608, max=37389, avg=3602.54, stdev=1500.19
  1015. clat (nsec): min=1520, max=3831.3k, avg=970150.25, stdev=238658.20
  1016. lat (usec): min=5, max=3834, avg=973.81, stdev=238.71
  1017. clat percentiles (usec):
  1018. | 1.00th=[ 420], 5.00th=[ 562], 10.00th=[ 660], 20.00th=[ 775],
  1019. | 30.00th=[ 857], 40.00th=[ 906], 50.00th=[ 979], 60.00th=[ 1029],
  1020. | 70.00th=[ 1106], 80.00th=[ 1172], 90.00th=[ 1270], 95.00th=[ 1369],
  1021. | 99.00th=[ 1516], 99.50th=[ 1582], 99.90th=[ 1696], 99.95th=[ 1762],
  1022. | 99.99th=[ 2900]
  1023. bw ( KiB/s): min=27544, max=31126, per=93.59%, avg=28860.67, stdev=734.09, samples=33
  1024. iops : min= 6886, max= 7781, avg=7214.85, stdev=183.47, samples=33
  1025. lat (usec) : 2=0.01%, 10=0.01%, 100=0.01%, 250=0.05%, 500=1.80%
  1026. lat (usec) : 750=14.43%, 1000=37.41%
  1027. lat (msec) : 2=46.29%, 4=0.01%
  1028. cpu : usr=3.28%, sys=11.88%, ctx=131416, majf=0, minf=14
  1029. IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
  1030. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  1031. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
  1032. issued rwts: total=131411,130733,0,0 short=0,0,0,0 dropped=0,0,0,0
  1033. latency : target=0, window=0, percentile=100.00%, depth=16
  1034. randReadWrite/sync: (groupid=7, jobs=1): err= 0: pid=13769: Sat Mar 21 09:21:11 2020
  1035. read: IOPS=2505, BW=9.79MiB/s (10.3MB/s)(511MiB/52245msec)
  1036. slat (usec): min=56, max=5407, avg=147.28, stdev=94.92
  1037. clat (usec): min=2, max=9446, avg=2965.35, stdev=390.95
  1038. lat (usec): min=151, max=9587, avg=3112.86, stdev=405.85
  1039. clat percentiles (usec):
  1040. | 1.00th=[ 2442], 5.00th=[ 2540], 10.00th=[ 2606], 20.00th=[ 2704],
  1041. | 30.00th=[ 2769], 40.00th=[ 2835], 50.00th=[ 2900], 60.00th=[ 2966],
  1042. | 70.00th=[ 3064], 80.00th=[ 3163], 90.00th=[ 3359], 95.00th=[ 3556],
  1043. | 99.00th=[ 4621], 99.50th=[ 5342], 99.90th=[ 5997], 99.95th=[ 6194],
  1044. | 99.99th=[ 7111]
  1045. bw ( KiB/s): min= 9285, max=10496, per=99.97%, avg=10018.86, stdev=257.21, samples=104
  1046. iops : min= 2321, max= 2624, avg=2504.71, stdev=64.31, samples=104
  1047. write: IOPS=2511, BW=9.81MiB/s (10.3MB/s)(513MiB/52245msec); 0 zone resets
  1048. slat (nsec): min=2989, max=33921, avg=6247.91, stdev=2398.78
  1049. clat (usec): min=295, max=9485, avg=3015.61, stdev=396.03
  1050. lat (usec): min=300, max=9490, avg=3022.00, stdev=396.33
  1051. clat percentiles (usec):
  1052. | 1.00th=[ 2507], 5.00th=[ 2606], 10.00th=[ 2671], 20.00th=[ 2737],
  1053. | 30.00th=[ 2802], 40.00th=[ 2868], 50.00th=[ 2933], 60.00th=[ 3032],
  1054. | 70.00th=[ 3097], 80.00th=[ 3228], 90.00th=[ 3392], 95.00th=[ 3589],
  1055. | 99.00th=[ 4817], 99.50th=[ 5407], 99.90th=[ 6063], 99.95th=[ 6325],
  1056. | 99.99th=[ 7242]
  1057. bw ( KiB/s): min= 9256, max=10896, per=100.00%, avg=10047.22, stdev=343.34, samples=104
  1058. iops : min= 2314, max= 2724, avg=2511.80, stdev=85.83, samples=104
  1059. lat (usec) : 4=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
  1060. lat (msec) : 2=0.01%, 4=98.18%, 10=1.81%
  1061. fsync/fdatasync/sync_file_range:
  1062. sync (nsec): min=32, max=16308, avg=187.17, stdev=150.65
  1063. sync percentiles (nsec):
  1064. | 1.00th=[ 102], 5.00th=[ 118], 10.00th=[ 127], 20.00th=[ 141],
  1065. | 30.00th=[ 147], 40.00th=[ 155], 50.00th=[ 167], 60.00th=[ 175],
  1066. | 70.00th=[ 191], 80.00th=[ 219], 90.00th=[ 255], 95.00th=[ 290],
  1067. | 99.00th=[ 556], 99.50th=[ 572], 99.90th=[ 604], 99.95th=[ 612],
  1068. | 99.99th=[10688]
  1069. cpu : usr=3.14%, sys=14.40%, ctx=779189, majf=0, minf=19
  1070. IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=200.0%, 32=0.0%, >=64=0.0%
  1071. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  1072. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
  1073. issued rwts: total=130910,131234,0,262129 short=0,0,0,0 dropped=0,0,0,0
  1074. latency : target=0, window=0, percentile=100.00%, depth=16
  1075.  
  1076. Run status group 0 (all jobs):
  1077. READ: bw=970MiB/s (1017MB/s), 970MiB/s-970MiB/s (1017MB/s-1017MB/s), io=1024MiB (1074MB), run=1056-1056msec
  1078.  
  1079. Run status group 1 (all jobs):
  1080. WRITE: bw=1108MiB/s (1162MB/s), 1108MiB/s-1108MiB/s (1162MB/s-1162MB/s), io=1024MiB (1074MB), run=924-924msec
  1081.  
  1082. Run status group 2 (all jobs):
  1083. WRITE: bw=25.9MiB/s (27.1MB/s), 25.9MiB/s-25.9MiB/s (27.1MB/s-27.1MB/s), io=1024MiB (1074MB), run=39607-39607msec
  1084.  
  1085. Run status group 3 (all jobs):
  1086. READ: bw=31.9MiB/s (33.4MB/s), 31.9MiB/s-31.9MiB/s (33.4MB/s-33.4MB/s), io=1024MiB (1074MB), run=32150-32150msec
  1087.  
  1088. Run status group 4 (all jobs):
  1089. WRITE: bw=1047MiB/s (1098MB/s), 1047MiB/s-1047MiB/s (1098MB/s-1098MB/s), io=1024MiB (1074MB), run=978-978msec
  1090.  
  1091. Run status group 5 (all jobs):
  1092. WRITE: bw=25.7MiB/s (26.9MB/s), 25.7MiB/s-25.7MiB/s (26.9MB/s-26.9MB/s), io=1024MiB (1074MB), run=39863-39863msec
  1093.  
  1094. Run status group 6 (all jobs):
  1095. READ: bw=30.3MiB/s (31.7MB/s), 30.3MiB/s-30.3MiB/s (31.7MB/s-31.7MB/s), io=513MiB (538MB), run=16957-16957msec
  1096. WRITE: bw=30.1MiB/s (31.6MB/s), 30.1MiB/s-30.1MiB/s (31.6MB/s-31.6MB/s), io=511MiB (535MB), run=16957-16957msec
  1097.  
  1098. Run status group 7 (all jobs):
  1099. READ: bw=9.79MiB/s (10.3MB/s), 9.79MiB/s-9.79MiB/s (10.3MB/s-10.3MB/s), io=511MiB (536MB), run=52245-52245msec
  1100. WRITE: bw=9.81MiB/s (10.3MB/s), 9.81MiB/s-9.81MiB/s (10.3MB/s-10.3MB/s), io=513MiB (538MB), run=52245-52245msec
  1101.  
  1102. ============================
  1103. single_ssd_bench-4k-IOD4.txt
  1104. ============================
  1105. seqRead: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=4
  1106. seqWrite: (g=1): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=4
  1107. seqWrite/sync: (g=2): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=4
  1108. randRead: (g=3): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=4
  1109. randWrite: (g=4): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=4
  1110. randWrite/sync: (g=5): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=4
  1111. randReadWrite: (g=6): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=4
  1112. randReadWrite/sync: (g=7): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=4
  1113. fio-3.12
  1114. Starting 8 processes
  1115.  
  1116. seqRead: (groupid=0, jobs=1): err= 0: pid=3467: Fri Mar 20 16:07:07 2020
  1117. read: IOPS=130k, BW=506MiB/s (531MB/s)(1024MiB/2024msec)
  1118. slat (nsec): min=1078, max=1277.9k, avg=6635.21, stdev=46286.85
  1119. clat (nsec): min=1435, max=1301.5k, avg=23925.88, stdev=80239.21
  1120. lat (usec): min=2, max=1306, avg=30.61, stdev=92.26
  1121. clat percentiles (usec):
  1122. | 1.00th=[ 7], 5.00th=[ 7], 10.00th=[ 7], 20.00th=[ 7],
  1123. | 30.00th=[ 7], 40.00th=[ 7], 50.00th=[ 7], 60.00th=[ 7],
  1124. | 70.00th=[ 7], 80.00th=[ 17], 90.00th=[ 17], 95.00th=[ 33],
  1125. | 99.00th=[ 529], 99.50th=[ 578], 99.90th=[ 668], 99.95th=[ 701],
  1126. | 99.99th=[ 807]
  1127. bw ( KiB/s): min=365869, max=509781, per=80.56%, avg=417345.00, stdev=80224.42, samples=3
  1128. iops : min=91467, max=127445, avg=104336.00, stdev=20056.10, samples=3
  1129. lat (usec) : 2=0.01%, 4=0.01%, 10=73.15%, 20=19.68%, 50=2.47%
  1130. lat (usec) : 100=0.88%, 250=0.98%, 500=1.53%, 750=1.28%, 1000=0.02%
  1131. lat (msec) : 2=0.01%
  1132. cpu : usr=15.62%, sys=36.78%, ctx=2928, majf=0, minf=14
  1133. IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  1134. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  1135. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  1136. issued rwts: total=262144,0,0,0 short=0,0,0,0 dropped=0,0,0,0
  1137. latency : target=0, window=0, percentile=100.00%, depth=4
  1138. seqWrite: (groupid=1, jobs=1): err= 0: pid=3473: Fri Mar 20 16:07:07 2020
  1139. write: IOPS=277k, BW=1081MiB/s (1134MB/s)(1024MiB/947msec); 0 zone resets
  1140. slat (nsec): min=1612, max=36060, avg=2242.44, stdev=517.89
  1141. clat (nsec): min=1490, max=47931, avg=11380.98, stdev=1076.66
  1142. lat (nsec): min=3599, max=50095, avg=13683.78, stdev=1223.15
  1143. clat percentiles (nsec):
  1144. | 1.00th=[10688], 5.00th=[10688], 10.00th=[10688], 20.00th=[10816],
  1145. | 30.00th=[10816], 40.00th=[10944], 50.00th=[10944], 60.00th=[11072],
  1146. | 70.00th=[11712], 80.00th=[11968], 90.00th=[12224], 95.00th=[12864],
  1147. | 99.00th=[14656], 99.50th=[16320], 99.90th=[24448], 99.95th=[27264],
  1148. | 99.99th=[34048]
  1149. bw ( MiB/s): min= 1082, max= 1082, per=100.00%, avg=1082.99, stdev= 0.00, samples=1
  1150. iops : min=277244, max=277244, avg=277244.00, stdev= 0.00, samples=1
  1151. lat (usec) : 2=0.01%, 10=0.03%, 20=99.69%, 50=0.28%
  1152. cpu : usr=40.38%, sys=59.41%, ctx=3, majf=0, minf=11
  1153. IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  1154. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  1155. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  1156. issued rwts: total=0,262144,0,0 short=0,0,0,0 dropped=0,0,0,0
  1157. latency : target=0, window=0, percentile=100.00%, depth=4
  1158. seqWrite/sync: (groupid=2, jobs=1): err= 0: pid=3672: Fri Mar 20 16:07:07 2020
  1159. write: IOPS=9014, BW=35.2MiB/s (36.9MB/s)(1024MiB/29079msec); 0 zone resets
  1160. slat (usec): min=2, max=256, avg= 4.71, stdev= 1.53
  1161. clat (nsec): min=1986, max=1166.2k, avg=333650.62, stdev=24533.35
  1162. lat (usec): min=6, max=1170, avg=338.47, stdev=25.06
  1163. clat percentiles (usec):
  1164. | 1.00th=[ 318], 5.00th=[ 318], 10.00th=[ 322], 20.00th=[ 322],
  1165. | 30.00th=[ 322], 40.00th=[ 326], 50.00th=[ 326], 60.00th=[ 326],
  1166. | 70.00th=[ 330], 80.00th=[ 338], 90.00th=[ 359], 95.00th=[ 383],
  1167. | 99.00th=[ 445], 99.50th=[ 457], 99.90th=[ 474], 99.95th=[ 486],
  1168. | 99.99th=[ 857]
  1169. bw ( KiB/s): min=31211, max=35000, per=91.46%, avg=32979.95, stdev=790.23, samples=58
  1170. iops : min= 7802, max= 8750, avg=8244.60, stdev=197.60, samples=58
  1171. lat (usec) : 2=0.01%, 250=0.01%, 500=99.95%, 750=0.02%, 1000=0.03%
  1172. lat (msec) : 2=0.01%
  1173. fsync/fdatasync/sync_file_range:
  1174. sync (nsec): min=65, max=15527, avg=163.67, stdev=191.52
  1175. sync percentiles (nsec):
  1176. | 1.00th=[ 101], 5.00th=[ 108], 10.00th=[ 113], 20.00th=[ 123],
  1177. | 30.00th=[ 151], 40.00th=[ 161], 50.00th=[ 167], 60.00th=[ 171],
  1178. | 70.00th=[ 177], 80.00th=[ 181], 90.00th=[ 191], 95.00th=[ 199],
  1179. | 99.00th=[ 282], 99.50th=[ 306], 99.90th=[ 410], 99.95th=[ 430],
  1180. | 99.99th=[11840]
  1181. cpu : usr=4.27%, sys=13.32%, ctx=786434, majf=0, minf=54
  1182. IO depths : 1=0.1%, 2=0.1%, 4=200.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  1183. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  1184. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  1185. issued rwts: total=0,262144,0,262141 short=0,0,0,0 dropped=0,0,0,0
  1186. latency : target=0, window=0, percentile=100.00%, depth=4
  1187. randRead: (groupid=3, jobs=1): err= 0: pid=4596: Fri Mar 20 16:07:07 2020
  1188. read: IOPS=8555, BW=33.4MiB/s (35.0MB/s)(1024MiB/30640msec)
  1189. slat (usec): min=33, max=1789, avg=114.86, stdev=12.56
  1190. clat (nsec): min=1982, max=2043.7k, avg=351606.00, stdev=24141.66
  1191. lat (usec): min=129, max=2166, avg=466.68, stdev=28.49
  1192. clat percentiles (usec):
  1193. | 1.00th=[ 302], 5.00th=[ 322], 10.00th=[ 326], 20.00th=[ 334],
  1194. | 30.00th=[ 343], 40.00th=[ 347], 50.00th=[ 351], 60.00th=[ 355],
  1195. | 70.00th=[ 363], 80.00th=[ 367], 90.00th=[ 379], 95.00th=[ 388],
  1196. | 99.00th=[ 416], 99.50th=[ 424], 99.90th=[ 441], 99.95th=[ 449],
  1197. | 99.99th=[ 758]
  1198. bw ( KiB/s): min=32900, max=34373, per=97.18%, avg=33255.43, stdev=211.08, samples=61
  1199. iops : min= 8225, max= 8593, avg=8313.51, stdev=52.76, samples=61
  1200. lat (usec) : 2=0.01%, 250=0.10%, 500=99.88%, 750=0.01%, 1000=0.01%
  1201. lat (msec) : 2=0.01%, 4=0.01%
  1202. cpu : usr=2.04%, sys=6.44%, ctx=262148, majf=0, minf=41
  1203. IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  1204. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  1205. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  1206. issued rwts: total=262144,0,0,0 short=0,0,0,0 dropped=0,0,0,0
  1207. latency : target=0, window=0, percentile=100.00%, depth=4
  1208. randWrite: (groupid=4, jobs=1): err= 0: pid=5675: Fri Mar 20 16:07:07 2020
  1209. write: IOPS=261k, BW=1019MiB/s (1068MB/s)(1024MiB/1005msec); 0 zone resets
  1210. slat (nsec): min=1822, max=25477, avg=2389.06, stdev=526.70
  1211. clat (nsec): min=1476, max=111706, avg=12039.16, stdev=1089.68
  1212. lat (usec): min=3, max=113, avg=14.49, stdev= 1.24
  1213. clat percentiles (nsec):
  1214. | 1.00th=[11200], 5.00th=[11328], 10.00th=[11328], 20.00th=[11456],
  1215. | 30.00th=[11456], 40.00th=[11584], 50.00th=[11584], 60.00th=[11840],
  1216. | 70.00th=[12352], 80.00th=[12608], 90.00th=[12992], 95.00th=[13760],
  1217. | 99.00th=[15552], 99.50th=[16768], 99.90th=[23424], 99.95th=[25728],
  1218. | 99.99th=[31616]
  1219. bw ( KiB/s): min=1039760, max=1039760, per=99.66%, avg=1039760.00, stdev= 0.00, samples=1
  1220. iops : min=259940, max=259940, avg=259940.00, stdev= 0.00, samples=1
  1221. lat (usec) : 2=0.01%, 10=0.01%, 20=99.82%, 50=0.18%, 250=0.01%
  1222. cpu : usr=41.24%, sys=58.67%, ctx=2, majf=0, minf=10
  1223. IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  1224. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  1225. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  1226. issued rwts: total=0,262144,0,0 short=0,0,0,0 dropped=0,0,0,0
  1227. latency : target=0, window=0, percentile=100.00%, depth=4
  1228. randWrite/sync: (groupid=5, jobs=1): err= 0: pid=5803: Fri Mar 20 16:07:07 2020
  1229. write: IOPS=8747, BW=34.2MiB/s (35.8MB/s)(1024MiB/29968msec); 0 zone resets
  1230. slat (nsec): min=2997, max=49736, avg=4941.24, stdev=1663.36
  1231. clat (usec): min=2, max=1204, avg=343.85, stdev=28.55
  1232. lat (usec): min=6, max=1209, avg=348.90, stdev=29.11
  1233. clat percentiles (usec):
  1234. | 1.00th=[ 322], 5.00th=[ 326], 10.00th=[ 326], 20.00th=[ 326],
  1235. | 30.00th=[ 330], 40.00th=[ 330], 50.00th=[ 334], 60.00th=[ 334],
  1236. | 70.00th=[ 343], 80.00th=[ 359], 90.00th=[ 375], 95.00th=[ 396],
  1237. | 99.00th=[ 461], 99.50th=[ 469], 99.90th=[ 482], 99.95th=[ 486],
  1238. | 99.99th=[ 848]
  1239. bw ( KiB/s): min=26367, max=34911, per=79.54%, avg=27830.56, stdev=1184.48, samples=59
  1240. iops : min= 6591, max= 8727, avg=6957.24, stdev=296.08, samples=59
  1241. lat (usec) : 4=0.01%, 250=0.01%, 500=99.95%, 750=0.02%, 1000=0.02%
  1242. lat (msec) : 2=0.01%
  1243. fsync/fdatasync/sync_file_range:
  1244. sync (nsec): min=85, max=17260, avg=163.63, stdev=265.41
  1245. sync percentiles (nsec):
  1246. | 1.00th=[ 103], 5.00th=[ 110], 10.00th=[ 115], 20.00th=[ 121],
  1247. | 30.00th=[ 129], 40.00th=[ 139], 50.00th=[ 161], 60.00th=[ 169],
  1248. | 70.00th=[ 175], 80.00th=[ 181], 90.00th=[ 193], 95.00th=[ 211],
  1249. | 99.00th=[ 302], 99.50th=[ 330], 99.90th=[ 430], 99.95th=[ 6368],
  1250. | 99.99th=[11968]
  1251. cpu : usr=4.75%, sys=13.48%, ctx=786432, majf=0, minf=42
  1252. IO depths : 1=0.1%, 2=0.1%, 4=200.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  1253. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  1254. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  1255. issued rwts: total=0,262144,0,262141 short=0,0,0,0 dropped=0,0,0,0
  1256. latency : target=0, window=0, percentile=100.00%, depth=4
  1257. randReadWrite: (groupid=6, jobs=1): err= 0: pid=6736: Fri Mar 20 16:07:07 2020
  1258. read: IOPS=8148, BW=31.8MiB/s (33.4MB/s)(513MiB/16102msec)
  1259. slat (usec): min=33, max=671, avg=115.56, stdev=10.73
  1260. clat (usec): min=7, max=700, avg=185.72, stdev=98.70
  1261. lat (usec): min=76, max=812, avg=301.52, stdev=99.51
  1262. clat percentiles (usec):
  1263. | 1.00th=[ 14], 5.00th=[ 15], 10.00th=[ 16], 20.00th=[ 120],
  1264. | 30.00th=[ 126], 40.00th=[ 139], 50.00th=[ 198], 60.00th=[ 231],
  1265. | 70.00th=[ 243], 80.00th=[ 258], 90.00th=[ 338], 95.00th=[ 359],
  1266. | 99.00th=[ 383], 99.50th=[ 396], 99.90th=[ 420], 99.95th=[ 429],
  1267. | 99.99th=[ 457]
  1268. bw ( KiB/s): min=23009, max=32608, per=72.06%, avg=23489.16, stdev=1694.36, samples=31
  1269. iops : min= 5752, max= 8152, avg=5871.90, stdev=423.66, samples=31
  1270. write: IOPS=8131, BW=31.8MiB/s (33.3MB/s)(511MiB/16102msec); 0 zone resets
  1271. slat (nsec): min=1630, max=23043, avg=3379.30, stdev=1328.12
  1272. clat (nsec): min=1548, max=682821, avg=184319.20, stdev=98556.77
  1273. lat (usec): min=5, max=686, avg=187.77, stdev=98.61
  1274. clat percentiles (usec):
  1275. | 1.00th=[ 12], 5.00th=[ 13], 10.00th=[ 14], 20.00th=[ 119],
  1276. | 30.00th=[ 124], 40.00th=[ 137], 50.00th=[ 204], 60.00th=[ 231],
  1277. | 70.00th=[ 241], 80.00th=[ 255], 90.00th=[ 334], 95.00th=[ 355],
  1278. | 99.00th=[ 379], 99.50th=[ 392], 99.90th=[ 420], 99.95th=[ 433],
  1279. | 99.99th=[ 449]
  1280. bw ( KiB/s): min=22281, max=32244, per=72.06%, avg=23435.77, stdev=1707.62, samples=31
  1281. iops : min= 5570, max= 8061, avg=5858.61, stdev=426.96, samples=31
  1282. lat (usec) : 2=0.01%, 10=0.01%, 20=11.75%, 50=0.63%, 100=0.02%
  1283. lat (usec) : 250=65.06%, 500=22.54%, 750=0.01%
  1284. cpu : usr=3.43%, sys=8.98%, ctx=131215, majf=0, minf=12
  1285. IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  1286. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  1287. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  1288. issued rwts: total=131215,130929,0,0 short=0,0,0,0 dropped=0,0,0,0
  1289. latency : target=0, window=0, percentile=100.00%, depth=4
  1290. randReadWrite/sync: (groupid=7, jobs=1): err= 0: pid=7377: Fri Mar 20 16:07:07 2020
  1291. read: IOPS=3152, BW=12.3MiB/s (12.9MB/s)(511MiB/41532msec)
  1292. slat (usec): min=76, max=6218, avg=150.63, stdev=191.12
  1293. clat (usec): min=2, max=6441, avg=447.79, stdev=235.01
  1294. lat (usec): min=113, max=6577, avg=598.68, stdev=302.91
  1295. clat percentiles (usec):
  1296. | 1.00th=[ 273], 5.00th=[ 277], 10.00th=[ 285], 20.00th=[ 347],
  1297. | 30.00th=[ 363], 40.00th=[ 400], 50.00th=[ 420], 60.00th=[ 457],
  1298. | 70.00th=[ 474], 80.00th=[ 498], 90.00th=[ 553], 95.00th=[ 586],
  1299. | 99.00th=[ 1532], 99.50th=[ 2442], 99.90th=[ 3130], 99.95th=[ 3294],
  1300. | 99.99th=[ 4948]
  1301. bw ( KiB/s): min=11920, max=13296, per=100.00%, avg=12608.65, stdev=296.19, samples=83
  1302. iops : min= 2980, max= 3324, avg=3152.12, stdev=74.07, samples=83
  1303. write: IOPS=3159, BW=12.3MiB/s (12.9MB/s)(513MiB/41532msec); 0 zone resets
  1304. slat (usec): min=3, max=110, avg= 4.98, stdev= 1.65
  1305. clat (usec): min=319, max=6607, avg=504.68, stdev=271.94
  1306. lat (usec): min=324, max=6612, avg=509.77, stdev=272.01
  1307. clat percentiles (usec):
  1308. | 1.00th=[ 326], 5.00th=[ 326], 10.00th=[ 334], 20.00th=[ 396],
  1309. | 30.00th=[ 412], 40.00th=[ 449], 50.00th=[ 469], 60.00th=[ 506],
  1310. | 70.00th=[ 529], 80.00th=[ 545], 90.00th=[ 603], 95.00th=[ 627],
  1311. | 99.00th=[ 2073], 99.50th=[ 2868], 99.90th=[ 3326], 99.95th=[ 3425],
  1312. | 99.99th=[ 3851]
  1313. bw ( KiB/s): min=11504, max=13392, per=100.00%, avg=12637.95, stdev=373.45, samples=83
  1314. iops : min= 2876, max= 3348, avg=3159.45, stdev=93.37, samples=83
  1315. lat (usec) : 4=0.01%, 250=0.01%, 500=69.56%, 750=28.45%, 1000=0.32%
  1316. lat (msec) : 2=0.78%, 4=0.88%, 10=0.01%
  1317. fsync/fdatasync/sync_file_range:
  1318. sync (nsec): min=54, max=16133, avg=159.08, stdev=163.62
  1319. sync percentiles (nsec):
  1320. | 1.00th=[ 83], 5.00th=[ 94], 10.00th=[ 104], 20.00th=[ 129],
  1321. | 30.00th=[ 135], 40.00th=[ 141], 50.00th=[ 151], 60.00th=[ 161],
  1322. | 70.00th=[ 167], 80.00th=[ 177], 90.00th=[ 187], 95.00th=[ 251],
  1323. | 99.00th=[ 386], 99.50th=[ 402], 99.90th=[ 422], 99.95th=[ 438],
  1324. | 99.99th=[11712]
  1325. cpu : usr=3.06%, sys=9.28%, ctx=786433, majf=0, minf=44
  1326. IO depths : 1=0.1%, 2=0.1%, 4=200.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  1327. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  1328. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  1329. issued rwts: total=130922,131222,0,262141 short=0,0,0,0 dropped=0,0,0,0
  1330. latency : target=0, window=0, percentile=100.00%, depth=4
  1331.  
  1332. Run status group 0 (all jobs):
  1333. READ: bw=506MiB/s (531MB/s), 506MiB/s-506MiB/s (531MB/s-531MB/s), io=1024MiB (1074MB), run=2024-2024msec
  1334.  
  1335. Run status group 1 (all jobs):
  1336. WRITE: bw=1081MiB/s (1134MB/s), 1081MiB/s-1081MiB/s (1134MB/s-1134MB/s), io=1024MiB (1074MB), run=947-947msec
  1337.  
  1338. Run status group 2 (all jobs):
  1339. WRITE: bw=35.2MiB/s (36.9MB/s), 35.2MiB/s-35.2MiB/s (36.9MB/s-36.9MB/s), io=1024MiB (1074MB), run=29079-29079msec
  1340.  
  1341. Run status group 3 (all jobs):
  1342. READ: bw=33.4MiB/s (35.0MB/s), 33.4MiB/s-33.4MiB/s (35.0MB/s-35.0MB/s), io=1024MiB (1074MB), run=30640-30640msec
  1343.  
  1344. Run status group 4 (all jobs):
  1345. WRITE: bw=1019MiB/s (1068MB/s), 1019MiB/s-1019MiB/s (1068MB/s-1068MB/s), io=1024MiB (1074MB), run=1005-1005msec
  1346.  
  1347. Run status group 5 (all jobs):
  1348. WRITE: bw=34.2MiB/s (35.8MB/s), 34.2MiB/s-34.2MiB/s (35.8MB/s-35.8MB/s), io=1024MiB (1074MB), run=29968-29968msec
  1349.  
  1350. Run status group 6 (all jobs):
  1351. READ: bw=31.8MiB/s (33.4MB/s), 31.8MiB/s-31.8MiB/s (33.4MB/s-33.4MB/s), io=513MiB (537MB), run=16102-16102msec
  1352. WRITE: bw=31.8MiB/s (33.3MB/s), 31.8MiB/s-31.8MiB/s (33.3MB/s-33.3MB/s), io=511MiB (536MB), run=16102-16102msec
  1353.  
  1354. Run status group 7 (all jobs):
  1355. READ: bw=12.3MiB/s (12.9MB/s), 12.3MiB/s-12.3MiB/s (12.9MB/s-12.9MB/s), io=511MiB (536MB), run=41532-41532msec
  1356. WRITE: bw=12.3MiB/s (12.9MB/s), 12.3MiB/s-12.3MiB/s (12.9MB/s-12.9MB/s), io=513MiB (537MB), run=41532-41532msec
  1357.  
  1358. ==============================
  1359. ssd_pool_r10_bench-4k-IOD4.txt
  1360. ==============================
  1361. seqRead: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=4
  1362. seqWrite: (g=1): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=4
  1363. seqWrite/sync: (g=2): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=4
  1364. randRead: (g=3): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=4
  1365. randWrite: (g=4): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=4
  1366. randWrite/sync: (g=5): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=4
  1367. randReadWrite: (g=6): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=4
  1368. randReadWrite/sync: (g=7): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=4
  1369. fio-3.12
  1370. Starting 8 processes
  1371. seqRead: Laying out IO file (1 file / 1024MiB)
  1372.  
  1373. seqRead: (groupid=0, jobs=1): err= 0: pid=52488: Fri Mar 20 17:55:06 2020
  1374. read: IOPS=288k, BW=1127MiB/s (1181MB/s)(1024MiB/909msec)
  1375. slat (nsec): min=2421, max=35095, avg=2698.96, stdev=340.48
  1376. clat (nsec): min=1408, max=40443, avg=10921.82, stdev=653.39
  1377. lat (nsec): min=4240, max=61954, avg=13676.03, stdev=751.22
  1378. clat percentiles (nsec):
  1379. | 1.00th=[10304], 5.00th=[10432], 10.00th=[10432], 20.00th=[10560],
  1380. | 30.00th=[10688], 40.00th=[10688], 50.00th=[10816], 60.00th=[10816],
  1381. | 70.00th=[10944], 80.00th=[11072], 90.00th=[11456], 95.00th=[11840],
  1382. | 99.00th=[12480], 99.50th=[12736], 99.90th=[17792], 99.95th=[22656],
  1383. | 99.99th=[29568]
  1384. bw ( MiB/s): min= 1126, max= 1126, per=99.99%, avg=1126.39, stdev= 0.00, samples=1
  1385. iops : min=288355, max=288355, avg=288355.00, stdev= 0.00, samples=1
  1386. lat (usec) : 2=0.01%, 10=0.01%, 20=99.91%, 50=0.09%
  1387. cpu : usr=27.97%, sys=71.48%, ctx=4, majf=7, minf=13
  1388. IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  1389. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  1390. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  1391. issued rwts: total=262144,0,0,0 short=0,0,0,0 dropped=0,0,0,0
  1392. latency : target=0, window=0, percentile=100.00%, depth=4
  1393. seqWrite: (groupid=1, jobs=1): err= 0: pid=52569: Fri Mar 20 17:55:06 2020
  1394. write: IOPS=152k, BW=594MiB/s (623MB/s)(1024MiB/1723msec); 0 zone resets
  1395. slat (usec): min=3, max=103, avg= 5.02, stdev= 1.27
  1396. clat (nsec): min=1582, max=135828, avg=20314.75, stdev=3464.00
  1397. lat (usec): min=6, max=140, avg=25.40, stdev= 4.23
  1398. clat percentiles (nsec):
  1399. | 1.00th=[18560], 5.00th=[18560], 10.00th=[18816], 20.00th=[18816],
  1400. | 30.00th=[18816], 40.00th=[18816], 50.00th=[19072], 60.00th=[19072],
  1401. | 70.00th=[19328], 80.00th=[21888], 90.00th=[23936], 95.00th=[25216],
  1402. | 99.00th=[36096], 99.50th=[42240], 99.90th=[51968], 99.95th=[55552],
  1403. | 99.99th=[66048]
  1404. bw ( KiB/s): min=395740, max=633777, per=81.42%, avg=495518.00, stdev=123596.09, samples=3
  1405. iops : min=98935, max=158444, avg=123879.33, stdev=30898.92, samples=3
  1406. lat (usec) : 2=0.01%, 10=0.01%, 20=74.74%, 50=25.13%, 100=0.13%
  1407. lat (usec) : 250=0.01%
  1408. cpu : usr=25.96%, sys=73.81%, ctx=3, majf=0, minf=12
  1409. IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  1410. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  1411. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  1412. issued rwts: total=0,262144,0,0 short=0,0,0,0 dropped=0,0,0,0
  1413. latency : target=0, window=0, percentile=100.00%, depth=4
  1414. seqWrite/sync: (groupid=2, jobs=1): err= 0: pid=54104: Fri Mar 20 17:55:06 2020
  1415. write: IOPS=5217, BW=20.4MiB/s (21.4MB/s)(1024MiB/50244msec); 0 zone resets
  1416. slat (usec): min=5, max=419, avg= 9.83, stdev= 4.86
  1417. clat (usec): min=4, max=216375, avg=574.31, stdev=591.64
  1418. lat (usec): min=20, max=216403, avg=584.27, stdev=591.97
  1419. clat percentiles (usec):
  1420. | 1.00th=[ 482], 5.00th=[ 490], 10.00th=[ 498], 20.00th=[ 519],
  1421. | 30.00th=[ 537], 40.00th=[ 553], 50.00th=[ 562], 60.00th=[ 578],
  1422. | 70.00th=[ 594], 80.00th=[ 611], 90.00th=[ 644], 95.00th=[ 668],
  1423. | 99.00th=[ 717], 99.50th=[ 758], 99.90th=[ 1057], 99.95th=[ 1745],
  1424. | 99.99th=[36439]
  1425. bw ( KiB/s): min=12713, max=17414, per=72.77%, avg=15187.35, stdev=1010.27, samples=100
  1426. iops : min= 3178, max= 4353, avg=3796.49, stdev=252.61, samples=100
  1427. lat (usec) : 10=0.01%, 250=0.01%, 500=10.82%, 750=88.64%, 1000=0.42%
  1428. lat (msec) : 2=0.08%, 4=0.03%, 10=0.01%, 50=0.01%, 250=0.01%
  1429. fsync/fdatasync/sync_file_range:
  1430. sync (nsec): min=30, max=24631, avg=133.29, stdev=457.35
  1431. sync percentiles (nsec):
  1432. | 1.00th=[ 61], 5.00th=[ 65], 10.00th=[ 68], 20.00th=[ 73],
  1433. | 30.00th=[ 78], 40.00th=[ 83], 50.00th=[ 89], 60.00th=[ 97],
  1434. | 70.00th=[ 116], 80.00th=[ 187], 90.00th=[ 207], 95.00th=[ 221],
  1435. | 99.00th=[ 266], 99.50th=[ 394], 99.90th=[11840], 99.95th=[11968],
  1436. | 99.99th=[12608]
  1437. cpu : usr=2.75%, sys=16.39%, ctx=523763, majf=0, minf=94
  1438. IO depths : 1=0.1%, 2=0.1%, 4=200.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  1439. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  1440. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  1441. issued rwts: total=0,262144,0,262141 short=0,0,0,0 dropped=0,0,0,0
  1442. latency : target=0, window=0, percentile=100.00%, depth=4
  1443. randRead: (groupid=3, jobs=1): err= 0: pid=6011: Fri Mar 20 17:55:06 2020
  1444. read: IOPS=228k, BW=890MiB/s (934MB/s)(1024MiB/1150msec)
  1445. slat (usec): min=2, max=444, avg= 3.53, stdev= 2.90
  1446. clat (nsec): min=1580, max=518141, avg=13686.58, stdev=5597.28
  1447. lat (usec): min=4, max=523, avg=17.27, stdev= 6.62
  1448. clat percentiles (usec):
  1449. | 1.00th=[ 13], 5.00th=[ 13], 10.00th=[ 13], 20.00th=[ 13],
  1450. | 30.00th=[ 13], 40.00th=[ 14], 50.00th=[ 14], 60.00th=[ 14],
  1451. | 70.00th=[ 14], 80.00th=[ 15], 90.00th=[ 15], 95.00th=[ 16],
  1452. | 99.00th=[ 19], 99.50th=[ 26], 99.90th=[ 62], 99.95th=[ 88],
  1453. | 99.99th=[ 359]
  1454. bw ( KiB/s): min=739437, max=884762, per=89.07%, avg=812099.50, stdev=102760.29, samples=2
  1455. iops : min=184859, max=221190, avg=203024.50, stdev=25689.90, samples=2
  1456. lat (usec) : 2=0.01%, 10=0.01%, 20=99.20%, 50=0.65%, 100=0.12%
  1457. lat (usec) : 250=0.01%, 500=0.01%, 750=0.01%
  1458. cpu : usr=27.50%, sys=71.80%, ctx=25, majf=0, minf=15
  1459. IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  1460. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  1461. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  1462. issued rwts: total=262144,0,0,0 short=0,0,0,0 dropped=0,0,0,0
  1463. latency : target=0, window=0, percentile=100.00%, depth=4
  1464. randWrite: (groupid=4, jobs=1): err= 0: pid=6236: Fri Mar 20 17:55:06 2020
  1465. write: IOPS=51.3k, BW=200MiB/s (210MB/s)(1024MiB/5108msec); 0 zone resets
  1466. slat (usec): min=4, max=553, avg=17.53, stdev=26.84
  1467. clat (nsec): min=1498, max=722807, avg=59168.51, stdev=61417.49
  1468. lat (usec): min=6, max=838, avg=76.79, stdev=77.52
  1469. clat percentiles (usec):
  1470. | 1.00th=[ 21], 5.00th=[ 22], 10.00th=[ 23], 20.00th=[ 25],
  1471. | 30.00th=[ 27], 40.00th=[ 29], 50.00th=[ 31], 60.00th=[ 33],
  1472. | 70.00th=[ 38], 80.00th=[ 101], 90.00th=[ 159], 95.00th=[ 186],
  1473. | 99.00th=[ 273], 99.50th=[ 314], 99.90th=[ 482], 99.95th=[ 529],
  1474. | 99.99th=[ 619]
  1475. bw ( KiB/s): min=112682, max=217663, per=68.58%, avg=140775.67, stdev=34314.54, samples=9
  1476. iops : min=28170, max=54415, avg=35193.56, stdev=8578.47, samples=9
  1477. lat (usec) : 2=0.01%, 10=0.01%, 20=0.26%, 50=72.95%, 100=6.50%
  1478. lat (usec) : 250=18.49%, 500=1.71%, 750=0.08%
  1479. cpu : usr=11.47%, sys=84.39%, ctx=22213, majf=0, minf=13
  1480. IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  1481. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  1482. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  1483. issued rwts: total=0,262144,0,0 short=0,0,0,0 dropped=0,0,0,0
  1484. latency : target=0, window=0, percentile=100.00%, depth=4
  1485. randWrite/sync: (groupid=5, jobs=1): err= 0: pid=31445: Fri Mar 20 17:55:06 2020
  1486. write: IOPS=5042, BW=19.7MiB/s (20.7MB/s)(1024MiB/51984msec); 0 zone resets
  1487. slat (usec): min=6, max=311, avg=11.60, stdev= 7.81
  1488. clat (nsec): min=1912, max=110168k, avg=594990.17, stdev=583552.51
  1489. lat (usec): min=9, max=110196, avg=606.76, stdev=587.18
  1490. clat percentiles (usec):
  1491. | 1.00th=[ 478], 5.00th=[ 486], 10.00th=[ 494], 20.00th=[ 506],
  1492. | 30.00th=[ 515], 40.00th=[ 523], 50.00th=[ 537], 60.00th=[ 545],
  1493. | 70.00th=[ 562], 80.00th=[ 586], 90.00th=[ 635], 95.00th=[ 668],
  1494. | 99.00th=[ 2900], 99.50th=[ 3851], 99.90th=[ 5604], 99.95th=[ 6718],
  1495. | 99.99th=[25560]
  1496. bw ( KiB/s): min= 9436, max=20585, per=86.03%, avg=17352.88, stdev=3141.44, samples=103
  1497. iops : min= 2359, max= 5146, avg=4337.87, stdev=785.34, samples=103
  1498. lat (usec) : 2=0.01%, 250=0.01%, 500=16.09%, 750=82.10%, 1000=0.32%
  1499. lat (msec) : 2=0.16%, 4=0.90%, 10=0.42%, 20=0.01%, 50=0.02%
  1500. lat (msec) : 100=0.01%, 250=0.01%
  1501. fsync/fdatasync/sync_file_range:
  1502. sync (nsec): min=50, max=28352, avg=120.38, stdev=331.26
  1503. sync percentiles (nsec):
  1504. | 1.00th=[ 61], 5.00th=[ 67], 10.00th=[ 71], 20.00th=[ 77],
  1505. | 30.00th=[ 82], 40.00th=[ 87], 50.00th=[ 91], 60.00th=[ 96],
  1506. | 70.00th=[ 104], 80.00th=[ 143], 90.00th=[ 197], 95.00th=[ 221],
  1507. | 99.00th=[ 294], 99.50th=[ 362], 99.90th=[ 756], 99.95th=[11840],
  1508. | 99.99th=[12352]
  1509. cpu : usr=2.56%, sys=15.39%, ctx=524790, majf=0, minf=119
  1510. IO depths : 1=0.1%, 2=0.1%, 4=200.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  1511. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  1512. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  1513. issued rwts: total=0,262144,0,262141 short=0,0,0,0 dropped=0,0,0,0
  1514. latency : target=0, window=0, percentile=100.00%, depth=4
  1515. randReadWrite: (groupid=6, jobs=1): err= 0: pid=34288: Fri Mar 20 17:55:06 2020
  1516. read: IOPS=37.6k, BW=147MiB/s (154MB/s)(511MiB/3485msec)
  1517. slat (usec): min=2, max=164, avg= 9.28, stdev=17.03
  1518. clat (usec): min=8, max=598, avg=40.66, stdev=45.70
  1519. lat (usec): min=11, max=609, avg=50.01, stdev=55.17
  1520. clat percentiles (usec):
  1521. | 1.00th=[ 13], 5.00th=[ 15], 10.00th=[ 16], 20.00th=[ 18],
  1522. | 30.00th=[ 20], 40.00th=[ 21], 50.00th=[ 23], 60.00th=[ 25],
  1523. | 70.00th=[ 28], 80.00th=[ 55], 90.00th=[ 98], 95.00th=[ 149],
  1524. | 99.00th=[ 231], 99.50th=[ 255], 99.90th=[ 318], 99.95th=[ 343],
  1525. | 99.99th=[ 400]
  1526. bw ( KiB/s): min=105652, max=160721, per=85.03%, avg=127741.83, stdev=21071.24, samples=6
  1527. iops : min=26413, max=40180, avg=31935.00, stdev=5267.85, samples=6
  1528. write: IOPS=37.7k, BW=147MiB/s (154MB/s)(513MiB/3485msec); 0 zone resets
  1529. slat (usec): min=4, max=232, avg=14.32, stdev=21.71
  1530. clat (nsec): min=1429, max=447931, avg=40400.65, stdev=45317.77
  1531. lat (usec): min=6, max=627, avg=54.81, stdev=59.52
  1532. clat percentiles (usec):
  1533. | 1.00th=[ 13], 5.00th=[ 15], 10.00th=[ 17], 20.00th=[ 18],
  1534. | 30.00th=[ 20], 40.00th=[ 21], 50.00th=[ 22], 60.00th=[ 24],
  1535. | 70.00th=[ 28], 80.00th=[ 54], 90.00th=[ 97], 95.00th=[ 149],
  1536. | 99.00th=[ 229], 99.50th=[ 255], 99.90th=[ 318], 99.95th=[ 347],
  1537. | 99.99th=[ 400]
  1538. bw ( KiB/s): min=105367, max=162797, per=85.03%, avg=128098.17, stdev=21667.47, samples=6
  1539. iops : min=26341, max=40699, avg=32024.17, stdev=5416.89, samples=6
  1540. lat (usec) : 2=0.01%, 10=0.01%, 20=37.21%, 50=41.76%, 100=11.71%
  1541. lat (usec) : 250=8.76%, 500=0.56%, 750=0.01%
  1542. cpu : usr=12.20%, sys=86.28%, ctx=6931, majf=0, minf=40
  1543. IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  1544. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  1545. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  1546. issued rwts: total=130882,131262,0,0 short=0,0,0,0 dropped=0,0,0,0
  1547. latency : target=0, window=0, percentile=100.00%, depth=4
  1548. randReadWrite/sync: (groupid=7, jobs=1): err= 0: pid=47879: Fri Mar 20 17:55:06 2020
  1549. read: IOPS=4804, BW=18.8MiB/s (19.7MB/s)(512MiB/27291msec)
  1550. slat (usec): min=2, max=150, avg= 5.64, stdev= 5.69
  1551. clat (usec): min=12, max=29839, avg=220.10, stdev=302.47
  1552. lat (usec): min=16, max=29933, avg=225.83, stdev=304.39
  1553. clat percentiles (usec):
  1554. | 1.00th=[ 20], 5.00th=[ 21], 10.00th=[ 23], 20.00th=[ 29],
  1555. | 30.00th=[ 180], 40.00th=[ 188], 50.00th=[ 198], 60.00th=[ 212],
  1556. | 70.00th=[ 239], 80.00th=[ 347], 90.00th=[ 383], 95.00th=[ 416],
  1557. | 99.00th=[ 668], 99.50th=[ 1500], 99.90th=[ 3032], 99.95th=[ 3621],
  1558. | 99.99th=[ 7767]
  1559. bw ( KiB/s): min=10896, max=23304, per=99.90%, avg=19200.31, stdev=3483.53, samples=54
  1560. iops : min= 2724, max= 5826, avg=4800.06, stdev=870.88, samples=54
  1561. write: IOPS=4800, BW=18.8MiB/s (19.7MB/s)(512MiB/27291msec); 0 zone resets
  1562. slat (usec): min=5, max=235, avg=10.37, stdev= 7.61
  1563. clat (nsec): min=1778, max=42709k, avg=405510.28, stdev=419590.34
  1564. lat (usec): min=8, max=42783, avg=416.03, stdev=422.74
  1565. clat percentiles (usec):
  1566. | 1.00th=[ 172], 5.00th=[ 178], 10.00th=[ 186], 20.00th=[ 215],
  1567. | 30.00th=[ 334], 40.00th=[ 351], 50.00th=[ 367], 60.00th=[ 392],
  1568. | 70.00th=[ 437], 80.00th=[ 510], 90.00th=[ 562], 95.00th=[ 611],
  1569. | 99.00th=[ 1565], 99.50th=[ 2704], 99.90th=[ 4686], 99.95th=[ 5342],
  1570. | 99.99th=[19792]
  1571. bw ( KiB/s): min=10864, max=22672, per=99.87%, avg=19176.41, stdev=3445.97, samples=54
  1572. iops : min= 2716, max= 5668, avg=4794.09, stdev=861.49, samples=54
  1573. lat (usec) : 2=0.01%, 20=1.34%, 50=10.45%, 100=0.59%, 250=35.16%
  1574. lat (usec) : 500=40.48%, 750=10.77%, 1000=0.21%
  1575. lat (msec) : 2=0.44%, 4=0.45%, 10=0.09%, 20=0.01%, 50=0.01%
  1576. fsync/fdatasync/sync_file_range:
  1577. sync (nsec): min=17, max=15131, avg=72.11, stdev=238.10
  1578. sync percentiles (nsec):
  1579. | 1.00th=[ 27], 5.00th=[ 28], 10.00th=[ 29], 20.00th=[ 30],
  1580. | 30.00th=[ 31], 40.00th=[ 33], 50.00th=[ 70], 60.00th=[ 80],
  1581. | 70.00th=[ 86], 80.00th=[ 94], 90.00th=[ 107], 95.00th=[ 133],
  1582. | 99.00th=[ 237], 99.50th=[ 286], 99.90th=[ 478], 99.95th=[ 828],
  1583. | 99.99th=[12096]
  1584. cpu : usr=3.01%, sys=17.62%, ctx=262313, majf=0, minf=96
  1585. IO depths : 1=0.1%, 2=0.1%, 4=200.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  1586. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  1587. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  1588. issued rwts: total=131132,131012,0,262141 short=0,0,0,0 dropped=0,0,0,0
  1589. latency : target=0, window=0, percentile=100.00%, depth=4
  1590.  
  1591. Run status group 0 (all jobs):
  1592. READ: bw=1127MiB/s (1181MB/s), 1127MiB/s-1127MiB/s (1181MB/s-1181MB/s), io=1024MiB (1074MB), run=909-909msec
  1593.  
  1594. Run status group 1 (all jobs):
  1595. WRITE: bw=594MiB/s (623MB/s), 594MiB/s-594MiB/s (623MB/s-623MB/s), io=1024MiB (1074MB), run=1723-1723msec
  1596.  
  1597. Run status group 2 (all jobs):
  1598. WRITE: bw=20.4MiB/s (21.4MB/s), 20.4MiB/s-20.4MiB/s (21.4MB/s-21.4MB/s), io=1024MiB (1074MB), run=50244-50244msec
  1599.  
  1600. Run status group 3 (all jobs):
  1601. READ: bw=890MiB/s (934MB/s), 890MiB/s-890MiB/s (934MB/s-934MB/s), io=1024MiB (1074MB), run=1150-1150msec
  1602.  
  1603. Run status group 4 (all jobs):
  1604. WRITE: bw=200MiB/s (210MB/s), 200MiB/s-200MiB/s (210MB/s-210MB/s), io=1024MiB (1074MB), run=5108-5108msec
  1605.  
  1606. Run status group 5 (all jobs):
  1607. WRITE: bw=19.7MiB/s (20.7MB/s), 19.7MiB/s-19.7MiB/s (20.7MB/s-20.7MB/s), io=1024MiB (1074MB), run=51984-51984msec
  1608.  
  1609. Run status group 6 (all jobs):
  1610. READ: bw=147MiB/s (154MB/s), 147MiB/s-147MiB/s (154MB/s-154MB/s), io=511MiB (536MB), run=3485-3485msec
  1611. WRITE: bw=147MiB/s (154MB/s), 147MiB/s-147MiB/s (154MB/s-154MB/s), io=513MiB (538MB), run=3485-3485msec
  1612.  
  1613. Run status group 7 (all jobs):
  1614. READ: bw=18.8MiB/s (19.7MB/s), 18.8MiB/s-18.8MiB/s (19.7MB/s-19.7MB/s), io=512MiB (537MB), run=27291-27291msec
  1615. WRITE: bw=18.8MiB/s (19.7MB/s), 18.8MiB/s-18.8MiB/s (19.7MB/s-19.7MB/s), io=512MiB (537MB), run=27291-27291msec
  1616.  
  1617. =====================================
  1618. ssd_pool_r10_no_lz4_bench-4k-IOD4.txt
  1619. =====================================
  1620. seqRead: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=4
  1621. seqWrite: (g=1): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=4
  1622. seqWrite/sync: (g=2): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=4
  1623. randRead: (g=3): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=4
  1624. randWrite: (g=4): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=4
  1625. randWrite/sync: (g=5): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=4
  1626. randReadWrite: (g=6): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=4
  1627. randReadWrite/sync: (g=7): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=4
  1628. fio-3.12
  1629. Starting 8 processes
  1630. seqRead: Laying out IO file (1 file / 1024MiB)
  1631.  
  1632. seqRead: (groupid=0, jobs=1): err= 0: pid=41106: Fri Mar 20 18:46:04 2020
  1633. read: IOPS=311k, BW=1216MiB/s (1275MB/s)(1024MiB/842msec)
  1634. slat (nsec): min=2061, max=37294, avg=2450.84, stdev=386.13
  1635. clat (nsec): min=1406, max=47832, avg=10161.13, stdev=876.84
  1636. lat (nsec): min=3624, max=54762, avg=12670.97, stdev=1066.78
  1637. clat percentiles (nsec):
  1638. | 1.00th=[ 9152], 5.00th=[ 9152], 10.00th=[ 9280], 20.00th=[ 9408],
  1639. | 30.00th=[ 9792], 40.00th=[10048], 50.00th=[10176], 60.00th=[10304],
  1640. | 70.00th=[10432], 80.00th=[10560], 90.00th=[10944], 95.00th=[11328],
  1641. | 99.00th=[12096], 99.50th=[12480], 99.90th=[21376], 99.95th=[23424],
  1642. | 99.99th=[29824]
  1643. bw ( MiB/s): min= 1177, max= 1177, per=96.79%, avg=1177.17, stdev= 0.00, samples=1
  1644. iops : min=301357, max=301357, avg=301357.00, stdev= 0.00, samples=1
  1645. lat (usec) : 2=0.01%, 10=37.66%, 20=62.21%, 50=0.13%
  1646. cpu : usr=28.89%, sys=70.63%, ctx=3, majf=0, minf=12
  1647. IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  1648. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  1649. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  1650. issued rwts: total=262144,0,0,0 short=0,0,0,0 dropped=0,0,0,0
  1651. latency : target=0, window=0, percentile=100.00%, depth=4
  1652. seqWrite: (groupid=1, jobs=1): err= 0: pid=41107: Fri Mar 20 18:46:04 2020
  1653. write: IOPS=154k, BW=602MiB/s (631MB/s)(1024MiB/1701msec); 0 zone resets
  1654. slat (usec): min=3, max=115, avg= 4.94, stdev= 1.16
  1655. clat (nsec): min=1626, max=148185, avg=20054.05, stdev=3005.93
  1656. lat (usec): min=6, max=153, avg=25.06, stdev= 3.66
  1657. clat percentiles (nsec):
  1658. | 1.00th=[18304], 5.00th=[18304], 10.00th=[18560], 20.00th=[18560],
  1659. | 30.00th=[18560], 40.00th=[18816], 50.00th=[18816], 60.00th=[18816],
  1660. | 70.00th=[19328], 80.00th=[21632], 90.00th=[22912], 95.00th=[24960],
  1661. | 99.00th=[31872], 99.50th=[36608], 99.90th=[46848], 99.95th=[50432],
  1662. | 99.99th=[58624]
  1663. bw ( KiB/s): min=388016, max=640550, per=83.43%, avg=514283.00, stdev=178568.50, samples=2
  1664. iops : min=97004, max=160137, avg=128570.50, stdev=44641.77, samples=2
  1665. lat (usec) : 2=0.01%, 10=0.01%, 20=72.45%, 50=27.50%, 100=0.05%
  1666. lat (usec) : 250=0.01%
  1667. cpu : usr=25.76%, sys=74.18%, ctx=6, majf=0, minf=11
  1668. IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  1669. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  1670. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  1671. issued rwts: total=0,262144,0,0 short=0,0,0,0 dropped=0,0,0,0
  1672. latency : target=0, window=0, percentile=100.00%, depth=4
  1673. seqWrite/sync: (groupid=2, jobs=1): err= 0: pid=42361: Fri Mar 20 18:46:04 2020
  1674. write: IOPS=5312, BW=20.8MiB/s (21.8MB/s)(1024MiB/49346msec); 0 zone resets
  1675. slat (usec): min=5, max=145, avg= 9.70, stdev= 4.69
  1676. clat (nsec): min=1814, max=596808k, avg=561205.67, stdev=1240076.64
  1677. lat (usec): min=8, max=596837, avg=571.04, stdev=1240.27
  1678. clat percentiles (usec):
  1679. | 1.00th=[ 478], 5.00th=[ 486], 10.00th=[ 490], 20.00th=[ 502],
  1680. | 30.00th=[ 515], 40.00th=[ 529], 50.00th=[ 545], 60.00th=[ 553],
  1681. | 70.00th=[ 578], 80.00th=[ 603], 90.00th=[ 627], 95.00th=[ 660],
  1682. | 99.00th=[ 709], 99.50th=[ 742], 99.90th=[ 1270], 99.95th=[ 2769],
  1683. | 99.99th=[37487]
  1684. bw ( KiB/s): min= 3767, max=16569, per=69.54%, avg=14777.58, stdev=1589.35, samples=98
  1685. iops : min= 941, max= 4142, avg=3694.02, stdev=397.33, samples=98
  1686. lat (usec) : 2=0.01%, 250=0.01%, 500=17.94%, 750=81.59%, 1000=0.29%
  1687. lat (msec) : 2=0.09%, 4=0.06%, 10=0.01%, 20=0.01%, 50=0.01%
  1688. lat (msec) : 750=0.01%
  1689. fsync/fdatasync/sync_file_range:
  1690. sync (nsec): min=42, max=25506, avg=125.79, stdev=468.39
  1691. sync percentiles (nsec):
  1692. | 1.00th=[ 61], 5.00th=[ 66], 10.00th=[ 69], 20.00th=[ 73],
  1693. | 30.00th=[ 77], 40.00th=[ 81], 50.00th=[ 85], 60.00th=[ 89],
  1694. | 70.00th=[ 99], 80.00th=[ 159], 90.00th=[ 203], 95.00th=[ 215],
  1695. | 99.00th=[ 237], 99.50th=[ 270], 99.90th=[11840], 99.95th=[12096],
  1696. | 99.99th=[12736]
  1697. cpu : usr=2.56%, sys=17.59%, ctx=523690, majf=0, minf=58
  1698. IO depths : 1=0.1%, 2=0.1%, 4=200.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  1699. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  1700. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  1701. issued rwts: total=0,262144,0,262141 short=0,0,0,0 dropped=0,0,0,0
  1702. latency : target=0, window=0, percentile=100.00%, depth=4
  1703. randRead: (groupid=3, jobs=1): err= 0: pid=37660: Fri Mar 20 18:46:04 2020
  1704. read: IOPS=237k, BW=927MiB/s (972MB/s)(1024MiB/1105msec)
  1705. slat (usec): min=2, max=366, avg= 3.38, stdev= 2.03
  1706. clat (nsec): min=1459, max=449683, avg=13175.36, stdev=3991.91
  1707. lat (usec): min=5, max=454, avg=16.62, stdev= 4.78
  1708. clat percentiles (usec):
  1709. | 1.00th=[ 12], 5.00th=[ 12], 10.00th=[ 13], 20.00th=[ 13],
  1710. | 30.00th=[ 13], 40.00th=[ 13], 50.00th=[ 13], 60.00th=[ 13],
  1711. | 70.00th=[ 14], 80.00th=[ 14], 90.00th=[ 14], 95.00th=[ 15],
  1712. | 99.00th=[ 20], 99.50th=[ 25], 99.90th=[ 63], 99.95th=[ 75],
  1713. | 99.99th=[ 116]
  1714. bw ( KiB/s): min=846986, max=923770, per=93.30%, avg=885378.00, stdev=54294.49, samples=2
  1715. iops : min=211746, max=230942, avg=221344.00, stdev=13573.62, samples=2
  1716. lat (usec) : 2=0.01%, 10=0.01%, 20=99.20%, 50=0.65%, 100=0.14%
  1717. lat (usec) : 250=0.01%, 500=0.01%
  1718. cpu : usr=26.63%, sys=72.92%, ctx=20, majf=0, minf=14
  1719. IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  1720. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  1721. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  1722. issued rwts: total=262144,0,0,0 short=0,0,0,0 dropped=0,0,0,0
  1723. latency : target=0, window=0, percentile=100.00%, depth=4
  1724. randWrite: (groupid=4, jobs=1): err= 0: pid=37968: Fri Mar 20 18:46:04 2020
  1725. write: IOPS=51.6k, BW=202MiB/s (212MB/s)(1024MiB/5076msec); 0 zone resets
  1726. slat (usec): min=4, max=543, avg=17.34, stdev=25.35
  1727. clat (nsec): min=1741, max=746155, avg=58812.57, stdev=57354.80
  1728. lat (usec): min=9, max=758, avg=76.25, stdev=72.12
  1729. clat percentiles (usec):
  1730. | 1.00th=[ 21], 5.00th=[ 22], 10.00th=[ 24], 20.00th=[ 26],
  1731. | 30.00th=[ 28], 40.00th=[ 29], 50.00th=[ 31], 60.00th=[ 34],
  1732. | 70.00th=[ 41], 80.00th=[ 99], 90.00th=[ 157], 95.00th=[ 180],
  1733. | 99.00th=[ 247], 99.50th=[ 265], 99.90th=[ 453], 99.95th=[ 498],
  1734. | 99.99th=[ 586]
  1735. bw ( KiB/s): min=120184, max=223898, per=73.60%, avg=152031.11, stdev=31488.76, samples=9
  1736. iops : min=30046, max=55974, avg=38007.44, stdev=7872.13, samples=9
  1737. lat (usec) : 2=0.01%, 20=0.35%, 50=71.69%, 100=8.45%, 250=18.63%
  1738. lat (usec) : 500=0.83%, 750=0.05%
  1739. cpu : usr=11.33%, sys=84.85%, ctx=21599, majf=0, minf=12
  1740. IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  1741. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  1742. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  1743. issued rwts: total=0,262144,0,0 short=0,0,0,0 dropped=0,0,0,0
  1744. latency : target=0, window=0, percentile=100.00%, depth=4
  1745. randWrite/sync: (groupid=5, jobs=1): err= 0: pid=7435: Fri Mar 20 18:46:04 2020
  1746. write: IOPS=4925, BW=19.2MiB/s (20.2MB/s)(1024MiB/53221msec); 0 zone resets
  1747. slat (usec): min=6, max=235, avg=12.77, stdev= 8.06
  1748. clat (usec): min=2, max=123074, avg=609.18, stdev=577.40
  1749. lat (usec): min=10, max=123102, avg=622.08, stdev=581.08
  1750. clat percentiles (usec):
  1751. | 1.00th=[ 486], 5.00th=[ 494], 10.00th=[ 502], 20.00th=[ 515],
  1752. | 30.00th=[ 529], 40.00th=[ 537], 50.00th=[ 553], 60.00th=[ 562],
  1753. | 70.00th=[ 586], 80.00th=[ 603], 90.00th=[ 644], 95.00th=[ 685],
  1754. | 99.00th=[ 2900], 99.50th=[ 3916], 99.90th=[ 5735], 99.95th=[ 6849],
  1755. | 99.99th=[23987]
  1756. bw ( KiB/s): min=10862, max=22885, per=96.79%, avg=19069.78, stdev=3247.39, samples=106
  1757. iops : min= 2715, max= 5721, avg=4767.06, stdev=811.85, samples=106
  1758. lat (usec) : 4=0.01%, 250=0.01%, 500=9.52%, 750=88.26%, 1000=0.73%
  1759. lat (msec) : 2=0.17%, 4=0.85%, 10=0.45%, 20=0.01%, 50=0.02%
  1760. lat (msec) : 250=0.01%
  1761. fsync/fdatasync/sync_file_range:
  1762. sync (nsec): min=41, max=36155, avg=135.66, stdev=412.07
  1763. sync percentiles (nsec):
  1764. | 1.00th=[ 63], 5.00th=[ 70], 10.00th=[ 74], 20.00th=[ 80],
  1765. | 30.00th=[ 84], 40.00th=[ 88], 50.00th=[ 93], 60.00th=[ 100],
  1766. | 70.00th=[ 123], 80.00th=[ 191], 90.00th=[ 213], 95.00th=[ 231],
  1767. | 99.00th=[ 342], 99.50th=[ 426], 99.90th=[11200], 99.95th=[12096],
  1768. | 99.99th=[12864]
  1769. cpu : usr=2.74%, sys=16.62%, ctx=524590, majf=0, minf=218
  1770. IO depths : 1=0.1%, 2=0.1%, 4=200.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  1771. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  1772. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  1773. issued rwts: total=0,262144,0,262141 short=0,0,0,0 dropped=0,0,0,0
  1774. latency : target=0, window=0, percentile=100.00%, depth=4
  1775. randReadWrite: (groupid=6, jobs=1): err= 0: pid=34561: Fri Mar 20 18:46:04 2020
  1776. read: IOPS=38.0k, BW=148MiB/s (156MB/s)(512MiB/3446msec)
  1777. slat (usec): min=2, max=132, avg= 9.06, stdev=16.16
  1778. clat (usec): min=10, max=328, avg=40.11, stdev=42.22
  1779. lat (usec): min=12, max=410, avg=49.25, stdev=51.10
  1780. clat percentiles (usec):
  1781. | 1.00th=[ 13], 5.00th=[ 14], 10.00th=[ 16], 20.00th=[ 18],
  1782. | 30.00th=[ 19], 40.00th=[ 21], 50.00th=[ 23], 60.00th=[ 25],
  1783. | 70.00th=[ 29], 80.00th=[ 57], 90.00th=[ 95], 95.00th=[ 145],
  1784. | 99.00th=[ 210], 99.50th=[ 229], 99.90th=[ 260], 99.95th=[ 269],
  1785. | 99.99th=[ 297]
  1786. bw ( KiB/s): min=84734, max=146995, per=73.39%, avg=111570.17, stdev=22817.78, samples=6
  1787. iops : min=21183, max=36748, avg=27892.17, stdev=5704.27, samples=6
  1788. write: IOPS=38.1k, BW=149MiB/s (156MB/s)(512MiB/3446msec); 0 zone resets
  1789. slat (usec): min=4, max=172, avg=14.21, stdev=20.06
  1790. clat (nsec): min=1636, max=397011, avg=40046.28, stdev=41965.28
  1791. lat (usec): min=6, max=469, avg=54.35, stdev=54.56
  1792. clat percentiles (usec):
  1793. | 1.00th=[ 13], 5.00th=[ 15], 10.00th=[ 16], 20.00th=[ 18],
  1794. | 30.00th=[ 20], 40.00th=[ 21], 50.00th=[ 23], 60.00th=[ 25],
  1795. | 70.00th=[ 29], 80.00th=[ 57], 90.00th=[ 95], 95.00th=[ 143],
  1796. | 99.00th=[ 210], 99.50th=[ 231], 99.90th=[ 262], 99.95th=[ 273],
  1797. | 99.99th=[ 293]
  1798. bw ( KiB/s): min=83685, max=147988, per=73.30%, avg=111600.33, stdev=23238.97, samples=6
  1799. iops : min=20921, max=36997, avg=27899.67, stdev=5809.78, samples=6
  1800. lat (usec) : 2=0.01%, 10=0.01%, 20=36.95%, 50=40.86%, 100=13.46%
  1801. lat (usec) : 250=8.54%, 500=0.18%
  1802. cpu : usr=11.73%, sys=86.88%, ctx=6904, majf=0, minf=12
  1803. IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  1804. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  1805. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  1806. issued rwts: total=130972,131172,0,0 short=0,0,0,0 dropped=0,0,0,0
  1807. latency : target=0, window=0, percentile=100.00%, depth=4
  1808. randReadWrite/sync: (groupid=7, jobs=1): err= 0: pid=48575: Fri Mar 20 18:46:04 2020
  1809. read: IOPS=4826, BW=18.9MiB/s (19.8MB/s)(514MiB/27258msec)
  1810. slat (usec): min=2, max=147, avg= 5.89, stdev= 5.76
  1811. clat (usec): min=13, max=25619, avg=219.72, stdev=285.06
  1812. lat (usec): min=17, max=25708, avg=225.71, stdev=286.98
  1813. clat percentiles (usec):
  1814. | 1.00th=[ 20], 5.00th=[ 22], 10.00th=[ 24], 20.00th=[ 30],
  1815. | 30.00th=[ 180], 40.00th=[ 188], 50.00th=[ 196], 60.00th=[ 210],
  1816. | 70.00th=[ 247], 80.00th=[ 347], 90.00th=[ 379], 95.00th=[ 416],
  1817. | 99.00th=[ 676], 99.50th=[ 1516], 99.90th=[ 3064], 99.95th=[ 3720],
  1818. | 99.99th=[ 7373]
  1819. bw ( KiB/s): min=10152, max=22904, per=99.89%, avg=19284.06, stdev=3611.01, samples=54
  1820. iops : min= 2538, max= 5726, avg=4820.98, stdev=902.72, samples=54
  1821. write: IOPS=4790, BW=18.7MiB/s (19.6MB/s)(510MiB/27258msec); 0 zone resets
  1822. slat (usec): min=5, max=230, avg=10.96, stdev= 7.88
  1823. clat (nsec): min=1713, max=64842k, avg=405558.99, stdev=441039.11
  1824. lat (usec): min=8, max=64871, avg=416.64, stdev=444.32
  1825. clat percentiles (usec):
  1826. | 1.00th=[ 172], 5.00th=[ 178], 10.00th=[ 186], 20.00th=[ 217],
  1827. | 30.00th=[ 334], 40.00th=[ 347], 50.00th=[ 363], 60.00th=[ 383],
  1828. | 70.00th=[ 449], 80.00th=[ 510], 90.00th=[ 553], 95.00th=[ 611],
  1829. | 99.00th=[ 1532], 99.50th=[ 2769], 99.90th=[ 4752], 99.95th=[ 5604],
  1830. | 99.99th=[20317]
  1831. bw ( KiB/s): min=10424, max=22392, per=99.84%, avg=19129.98, stdev=3543.99, samples=54
  1832. iops : min= 2606, max= 5598, avg=4782.48, stdev=885.99, samples=54
  1833. lat (usec) : 2=0.01%, 20=0.96%, 50=10.71%, 100=1.00%, 250=34.17%
  1834. lat (usec) : 500=41.08%, 750=10.87%, 1000=0.23%
  1835. lat (msec) : 2=0.42%, 4=0.44%, 10=0.10%, 20=0.01%, 50=0.01%
  1836. lat (msec) : 100=0.01%
  1837. fsync/fdatasync/sync_file_range:
  1838. sync (nsec): min=17, max=15046, avg=73.66, stdev=220.99
  1839. sync percentiles (nsec):
  1840. | 1.00th=[ 27], 5.00th=[ 28], 10.00th=[ 29], 20.00th=[ 31],
  1841. | 30.00th=[ 32], 40.00th=[ 34], 50.00th=[ 64], 60.00th=[ 72],
  1842. | 70.00th=[ 78], 80.00th=[ 87], 90.00th=[ 133], 95.00th=[ 185],
  1843. | 99.00th=[ 270], 99.50th=[ 310], 99.90th=[ 502], 99.95th=[ 668],
  1844. | 99.99th=[12224]
  1845. cpu : usr=3.63%, sys=18.12%, ctx=261469, majf=0, minf=68
  1846. IO depths : 1=0.1%, 2=0.1%, 4=200.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  1847. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  1848. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  1849. issued rwts: total=131567,130577,0,262141 short=0,0,0,0 dropped=0,0,0,0
  1850. latency : target=0, window=0, percentile=100.00%, depth=4
  1851.  
  1852. Run status group 0 (all jobs):
  1853. READ: bw=1216MiB/s (1275MB/s), 1216MiB/s-1216MiB/s (1275MB/s-1275MB/s), io=1024MiB (1074MB), run=842-842msec
  1854.  
  1855. Run status group 1 (all jobs):
  1856. WRITE: bw=602MiB/s (631MB/s), 602MiB/s-602MiB/s (631MB/s-631MB/s), io=1024MiB (1074MB), run=1701-1701msec
  1857.  
  1858. Run status group 2 (all jobs):
  1859. WRITE: bw=20.8MiB/s (21.8MB/s), 20.8MiB/s-20.8MiB/s (21.8MB/s-21.8MB/s), io=1024MiB (1074MB), run=49346-49346msec
  1860.  
  1861. Run status group 3 (all jobs):
  1862. READ: bw=927MiB/s (972MB/s), 927MiB/s-927MiB/s (972MB/s-972MB/s), io=1024MiB (1074MB), run=1105-1105msec
  1863.  
  1864. Run status group 4 (all jobs):
  1865. WRITE: bw=202MiB/s (212MB/s), 202MiB/s-202MiB/s (212MB/s-212MB/s), io=1024MiB (1074MB), run=5076-5076msec
  1866.  
  1867. Run status group 5 (all jobs):
  1868. WRITE: bw=19.2MiB/s (20.2MB/s), 19.2MiB/s-19.2MiB/s (20.2MB/s-20.2MB/s), io=1024MiB (1074MB), run=53221-53221msec
  1869.  
  1870. Run status group 6 (all jobs):
  1871. READ: bw=148MiB/s (156MB/s), 148MiB/s-148MiB/s (156MB/s-156MB/s), io=512MiB (536MB), run=3446-3446msec
  1872. WRITE: bw=149MiB/s (156MB/s), 149MiB/s-149MiB/s (156MB/s-156MB/s), io=512MiB (537MB), run=3446-3446msec
  1873.  
  1874. Run status group 7 (all jobs):
  1875. READ: bw=18.9MiB/s (19.8MB/s), 18.9MiB/s-18.9MiB/s (19.8MB/s-19.8MB/s), io=514MiB (539MB), run=27258-27258msec
  1876. WRITE: bw=18.7MiB/s (19.6MB/s), 18.7MiB/s-18.7MiB/s (19.6MB/s-19.6MB/s), io=510MiB (535MB), run=27258-27258msec
  1877.  
  1878. ================================
  1879. md_tlvm_6disks_bench-4k-IOD4.txt
  1880. ================================
  1881. seqRead: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=4
  1882. seqWrite: (g=1): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=4
  1883. seqWrite/sync: (g=2): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=4
  1884. randRead: (g=3): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=4
  1885. randWrite: (g=4): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=4
  1886. randWrite/sync: (g=5): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=4
  1887. randReadWrite: (g=6): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=4
  1888. randReadWrite/sync: (g=7): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=4
  1889. fio-3.12
  1890. Starting 8 processes
  1891.  
  1892. seqRead: (groupid=0, jobs=1): err= 0: pid=3156: Sat Mar 21 09:18:00 2020
  1893. read: IOPS=258k, BW=1009MiB/s (1058MB/s)(1024MiB/1015msec)
  1894. slat (nsec): min=1098, max=2896.6k, avg=3038.11, stdev=19732.68
  1895. clat (nsec): min=1355, max=2921.6k, avg=12203.56, stdev=34150.06
  1896. lat (usec): min=2, max=2926, avg=15.28, stdev=39.31
  1897. clat percentiles (usec):
  1898. | 1.00th=[ 7], 5.00th=[ 7], 10.00th=[ 7], 20.00th=[ 7],
  1899. | 30.00th=[ 7], 40.00th=[ 7], 50.00th=[ 7], 60.00th=[ 7],
  1900. | 70.00th=[ 7], 80.00th=[ 9], 90.00th=[ 13], 95.00th=[ 17],
  1901. | 99.00th=[ 202], 99.50th=[ 208], 99.90th=[ 223], 99.95th=[ 229],
  1902. | 99.99th=[ 251]
  1903. bw ( KiB/s): min=1031549, max=1031549, per=99.85%, avg=1031549.00, stdev= 0.00, samples=1
  1904. iops : min=257887, max=257887, avg=257887.00, stdev= 0.00, samples=1
  1905. lat (usec) : 2=0.01%, 4=0.01%, 10=84.75%, 20=11.55%, 50=1.34%
  1906. lat (usec) : 100=0.01%, 250=2.34%, 500=0.01%, 750=0.01%
  1907. lat (msec) : 4=0.01%
  1908. cpu : usr=21.70%, sys=77.22%, ctx=18, majf=0, minf=14
  1909. IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  1910. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  1911. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  1912. issued rwts: total=262144,0,0,0 short=0,0,0,0 dropped=0,0,0,0
  1913. latency : target=0, window=0, percentile=100.00%, depth=4
  1914. seqWrite: (groupid=1, jobs=1): err= 0: pid=3220: Sat Mar 21 09:18:00 2020
  1915. write: IOPS=279k, BW=1092MiB/s (1145MB/s)(1024MiB/938msec); 0 zone resets
  1916. slat (nsec): min=1830, max=118342, avg=2221.16, stdev=533.63
  1917. clat (nsec): min=1464, max=57207, avg=11261.51, stdev=971.66
  1918. lat (usec): min=3, max=138, avg=13.55, stdev= 1.12
  1919. clat percentiles (nsec):
  1920. | 1.00th=[10560], 5.00th=[10688], 10.00th=[10688], 20.00th=[10688],
  1921. | 30.00th=[10816], 40.00th=[10816], 50.00th=[10944], 60.00th=[11072],
  1922. | 70.00th=[11584], 80.00th=[11840], 90.00th=[12096], 95.00th=[12608],
  1923. | 99.00th=[14144], 99.50th=[15936], 99.90th=[22656], 99.95th=[24960],
  1924. | 99.99th=[30336]
  1925. bw ( MiB/s): min= 1089, max= 1089, per=99.80%, avg=1089.50, stdev= 0.00, samples=1
  1926. iops : min=278914, max=278914, avg=278914.00, stdev= 0.00, samples=1
  1927. lat (usec) : 2=0.01%, 10=0.01%, 20=99.82%, 50=0.18%, 100=0.01%
  1928. cpu : usr=40.45%, sys=59.23%, ctx=2, majf=0, minf=11
  1929. IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  1930. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  1931. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  1932. issued rwts: total=0,262144,0,0 short=0,0,0,0 dropped=0,0,0,0
  1933. latency : target=0, window=0, percentile=100.00%, depth=4
  1934. seqWrite/sync: (groupid=2, jobs=1): err= 0: pid=3244: Sat Mar 21 09:18:00 2020
  1935. write: IOPS=6504, BW=25.4MiB/s (26.6MB/s)(1024MiB/40300msec); 0 zone resets
  1936. slat (nsec): min=2492, max=48393, avg=5756.64, stdev=1396.10
  1937. clat (nsec): min=1999, max=2997.8k, avg=462194.24, stdev=52521.09
  1938. lat (usec): min=7, max=3005, avg=468.10, stdev=53.42
  1939. clat percentiles (usec):
  1940. | 1.00th=[ 429], 5.00th=[ 433], 10.00th=[ 433], 20.00th=[ 437],
  1941. | 30.00th=[ 441], 40.00th=[ 441], 50.00th=[ 445], 60.00th=[ 449],
  1942. | 70.00th=[ 453], 80.00th=[ 461], 90.00th=[ 529], 95.00th=[ 603],
  1943. | 99.00th=[ 644], 99.50th=[ 660], 99.90th=[ 807], 99.95th=[ 881],
  1944. | 99.99th=[ 1156]
  1945. bw ( KiB/s): min=17105, max=25882, per=84.05%, avg=21868.79, stdev=1649.34, samples=80
  1946. iops : min= 4276, max= 6470, avg=5466.88, stdev=412.32, samples=80
  1947. lat (usec) : 2=0.01%, 250=0.01%, 500=88.66%, 750=11.19%, 1000=0.13%
  1948. lat (msec) : 2=0.01%, 4=0.01%
  1949. fsync/fdatasync/sync_file_range:
  1950. sync (nsec): min=70, max=12006, avg=167.60, stdev=85.47
  1951. sync percentiles (nsec):
  1952. | 1.00th=[ 100], 5.00th=[ 115], 10.00th=[ 123], 20.00th=[ 135],
  1953. | 30.00th=[ 145], 40.00th=[ 151], 50.00th=[ 159], 60.00th=[ 169],
  1954. | 70.00th=[ 179], 80.00th=[ 191], 90.00th=[ 227], 95.00th=[ 247],
  1955. | 99.00th=[ 298], 99.50th=[ 346], 99.90th=[ 426], 99.95th=[ 486],
  1956. | 99.99th=[ 644]
  1957. cpu : usr=3.58%, sys=19.41%, ctx=783074, majf=0, minf=40
  1958. IO depths : 1=0.1%, 2=0.1%, 4=200.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  1959. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  1960. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  1961. issued rwts: total=0,262144,0,262141 short=0,0,0,0 dropped=0,0,0,0
  1962. latency : target=0, window=0, percentile=100.00%, depth=4
  1963. randRead: (groupid=3, jobs=1): err= 0: pid=4604: Sat Mar 21 09:18:00 2020
  1964. read: IOPS=8122, BW=31.7MiB/s (33.3MB/s)(1024MiB/32272msec)
  1965. slat (usec): min=36, max=2985, avg=120.78, stdev=16.01
  1966. clat (usec): min=2, max=3254, avg=370.35, stdev=30.53
  1967. lat (usec): min=130, max=3386, avg=491.39, stdev=36.00
  1968. clat percentiles (usec):
  1969. | 1.00th=[ 306], 5.00th=[ 334], 10.00th=[ 343], 20.00th=[ 351],
  1970. | 30.00th=[ 359], 40.00th=[ 363], 50.00th=[ 367], 60.00th=[ 375],
  1971. | 70.00th=[ 379], 80.00th=[ 388], 90.00th=[ 400], 95.00th=[ 416],
  1972. | 99.00th=[ 449], 99.50th=[ 461], 99.90th=[ 482], 99.95th=[ 498],
  1973. | 99.99th=[ 963]
  1974. bw ( KiB/s): min=26296, max=32309, per=82.51%, avg=26808.91, stdev=762.91, samples=64
  1975. iops : min= 6574, max= 8077, avg=6701.87, stdev=190.73, samples=64
  1976. lat (usec) : 4=0.01%, 250=0.05%, 500=99.90%, 750=0.03%, 1000=0.01%
  1977. lat (msec) : 2=0.01%, 4=0.01%
  1978. cpu : usr=2.10%, sys=9.56%, ctx=262149, majf=0, minf=16
  1979. IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  1980. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  1981. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  1982. issued rwts: total=262144,0,0,0 short=0,0,0,0 dropped=0,0,0,0
  1983. latency : target=0, window=0, percentile=100.00%, depth=4
  1984. randWrite: (groupid=4, jobs=1): err= 0: pid=5704: Sat Mar 21 09:18:00 2020
  1985. write: IOPS=261k, BW=1021MiB/s (1071MB/s)(1024MiB/1003msec); 0 zone resets
  1986. slat (nsec): min=1789, max=99826, avg=2398.26, stdev=621.92
  1987. clat (nsec): min=1443, max=120866, avg=12012.02, stdev=1355.98
  1988. lat (usec): min=3, max=123, avg=14.48, stdev= 1.55
  1989. clat percentiles (nsec):
  1990. | 1.00th=[11200], 5.00th=[11200], 10.00th=[11328], 20.00th=[11328],
  1991. | 30.00th=[11456], 40.00th=[11456], 50.00th=[11584], 60.00th=[11712],
  1992. | 70.00th=[12224], 80.00th=[12480], 90.00th=[12864], 95.00th=[13632],
  1993. | 99.00th=[15552], 99.50th=[19072], 99.90th=[28800], 99.95th=[33024],
  1994. | 99.99th=[42240]
  1995. bw ( KiB/s): min=1041697, max=1041697, per=99.64%, avg=1041697.00, stdev= 0.00, samples=1
  1996. iops : min=260424, max=260424, avg=260424.00, stdev= 0.00, samples=1
  1997. lat (usec) : 2=0.01%, 10=0.01%, 20=99.55%, 50=0.45%, 100=0.01%
  1998. lat (usec) : 250=0.01%
  1999. cpu : usr=39.02%, sys=60.98%, ctx=3, majf=0, minf=10
  2000. IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  2001. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  2002. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  2003. issued rwts: total=0,262144,0,0 short=0,0,0,0 dropped=0,0,0,0
  2004. latency : target=0, window=0, percentile=100.00%, depth=4
  2005. randWrite/sync: (groupid=5, jobs=1): err= 0: pid=5839: Sat Mar 21 09:18:00 2020
  2006. write: IOPS=6560, BW=25.6MiB/s (26.9MB/s)(1024MiB/39955msec); 0 zone resets
  2007. slat (usec): min=3, max=128, avg= 5.69, stdev= 1.21
  2008. clat (usec): min=2, max=2867, avg=458.20, stdev=43.50
  2009. lat (usec): min=7, max=2875, avg=464.05, stdev=44.03
  2010. clat percentiles (usec):
  2011. | 1.00th=[ 433], 5.00th=[ 437], 10.00th=[ 437], 20.00th=[ 441],
  2012. | 30.00th=[ 445], 40.00th=[ 445], 50.00th=[ 449], 60.00th=[ 449],
  2013. | 70.00th=[ 453], 80.00th=[ 461], 90.00th=[ 474], 95.00th=[ 529],
  2014. | 99.00th=[ 644], 99.50th=[ 668], 99.90th=[ 840], 99.95th=[ 914],
  2015. | 99.99th=[ 1205]
  2016. bw ( KiB/s): min=18919, max=25595, per=87.30%, avg=22910.91, stdev=885.28, samples=79
  2017. iops : min= 4729, max= 6398, avg=5727.34, stdev=221.36, samples=79
  2018. lat (usec) : 4=0.01%, 250=0.01%, 500=94.15%, 750=5.67%, 1000=0.15%
  2019. lat (msec) : 2=0.02%, 4=0.01%
  2020. fsync/fdatasync/sync_file_range:
  2021. sync (nsec): min=74, max=12061, avg=167.76, stdev=87.78
  2022. sync percentiles (nsec):
  2023. | 1.00th=[ 112], 5.00th=[ 123], 10.00th=[ 129], 20.00th=[ 141],
  2024. | 30.00th=[ 149], 40.00th=[ 155], 50.00th=[ 161], 60.00th=[ 169],
  2025. | 70.00th=[ 179], 80.00th=[ 187], 90.00th=[ 207], 95.00th=[ 231],
  2026. | 99.00th=[ 314], 99.50th=[ 370], 99.90th=[ 466], 99.95th=[ 516],
  2027. | 99.99th=[ 740]
  2028. cpu : usr=3.44%, sys=18.60%, ctx=778389, majf=0, minf=16
  2029. IO depths : 1=0.1%, 2=0.1%, 4=200.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  2030. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  2031. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  2032. issued rwts: total=0,262144,0,262141 short=0,0,0,0 dropped=0,0,0,0
  2033. latency : target=0, window=0, percentile=100.00%, depth=4
  2034. randReadWrite: (groupid=6, jobs=1): err= 0: pid=7073: Sat Mar 21 09:18:00 2020
  2035. read: IOPS=7733, BW=30.2MiB/s (31.7MB/s)(512MiB/16938msec)
  2036. slat (usec): min=39, max=2696, avg=121.71, stdev=16.78
  2037. clat (usec): min=9, max=2821, avg=195.84, stdev=105.13
  2038. lat (usec): min=55, max=3107, avg=317.82, stdev=106.85
  2039. clat percentiles (usec):
  2040. | 1.00th=[ 15], 5.00th=[ 16], 10.00th=[ 17], 20.00th=[ 127],
  2041. | 30.00th=[ 133], 40.00th=[ 145], 50.00th=[ 194], 60.00th=[ 245],
  2042. | 70.00th=[ 255], 80.00th=[ 269], 90.00th=[ 355], 95.00th=[ 379],
  2043. | 99.00th=[ 404], 99.50th=[ 420], 99.90th=[ 457], 99.95th=[ 465],
  2044. | 99.99th=[ 498]
  2045. bw ( KiB/s): min=20841, max=30777, per=68.88%, avg=21304.91, stdev=1702.14, samples=33
  2046. iops : min= 5210, max= 7694, avg=5325.85, stdev=425.56, samples=33
  2047. write: IOPS=7743, BW=30.2MiB/s (31.7MB/s)(512MiB/16938msec); 0 zone resets
  2048. slat (nsec): min=1639, max=70960, avg=3514.78, stdev=1408.29
  2049. clat (nsec): min=1491, max=2966.0k, avg=193435.01, stdev=106514.07
  2050. lat (usec): min=5, max=2970, avg=197.02, stdev=106.58
  2051. clat percentiles (usec):
  2052. | 1.00th=[ 12], 5.00th=[ 12], 10.00th=[ 14], 20.00th=[ 124],
  2053. | 30.00th=[ 130], 40.00th=[ 143], 50.00th=[ 186], 60.00th=[ 243],
  2054. | 70.00th=[ 253], 80.00th=[ 269], 90.00th=[ 355], 95.00th=[ 375],
  2055. | 99.00th=[ 404], 99.50th=[ 420], 99.90th=[ 457], 99.95th=[ 465],
  2056. | 99.99th=[ 553]
  2057. bw ( KiB/s): min=20135, max=31206, per=68.87%, avg=21330.36, stdev=1842.98, samples=33
  2058. iops : min= 5033, max= 7801, avg=5332.24, stdev=460.73, samples=33
  2059. lat (usec) : 2=0.01%, 10=0.01%, 20=12.10%, 50=0.47%, 100=0.09%
  2060. lat (usec) : 250=51.85%, 500=35.48%, 750=0.01%
  2061. lat (msec) : 2=0.01%, 4=0.01%
  2062. cpu : usr=3.63%, sys=11.54%, ctx=130984, majf=0, minf=12
  2063. IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  2064. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  2065. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  2066. issued rwts: total=130982,131162,0,0 short=0,0,0,0 dropped=0,0,0,0
  2067. latency : target=0, window=0, percentile=100.00%, depth=4
  2068. randReadWrite/sync: (groupid=7, jobs=1): err= 0: pid=7642: Sat Mar 21 09:18:00 2020
  2069. read: IOPS=2496, BW=9985KiB/s (10.2MB/s)(512MiB/52518msec)
  2070. slat (usec): min=58, max=5149, avg=146.78, stdev=91.75
  2071. clat (usec): min=2, max=5690, avg=573.56, stdev=157.19
  2072. lat (usec): min=126, max=6227, avg=720.59, stdev=186.36
  2073. clat percentiles (usec):
  2074. | 1.00th=[ 375], 5.00th=[ 388], 10.00th=[ 433], 20.00th=[ 457],
  2075. | 30.00th=[ 498], 40.00th=[ 529], 50.00th=[ 562], 60.00th=[ 586],
  2076. | 70.00th=[ 619], 80.00th=[ 660], 90.00th=[ 734], 95.00th=[ 807],
  2077. | 99.00th=[ 922], 99.50th=[ 971], 99.90th=[ 2507], 99.95th=[ 3064],
  2078. | 99.99th=[ 3556]
  2079. bw ( KiB/s): min= 8120, max=10536, per=99.98%, avg=9982.10, stdev=444.64, samples=105
  2080. iops : min= 2030, max= 2634, avg=2495.47, stdev=111.11, samples=105
  2081. write: IOPS=2495, BW=9981KiB/s (10.2MB/s)(512MiB/52518msec); 0 zone resets
  2082. slat (nsec): min=4450, max=35909, avg=6352.54, stdev=2476.65
  2083. clat (usec): min=422, max=5936, avg=630.82, stdev=172.64
  2084. lat (usec): min=427, max=5941, avg=637.33, stdev=173.14
  2085. clat percentiles (usec):
  2086. | 1.00th=[ 441], 5.00th=[ 453], 10.00th=[ 490], 20.00th=[ 523],
  2087. | 30.00th=[ 553], 40.00th=[ 586], 50.00th=[ 611], 60.00th=[ 644],
  2088. | 70.00th=[ 668], 80.00th=[ 709], 90.00th=[ 775], 95.00th=[ 848],
  2089. | 99.00th=[ 979], 99.50th=[ 1106], 99.90th=[ 3163], 99.95th=[ 3392],
  2090. | 99.99th=[ 3785]
  2091. bw ( KiB/s): min= 8248, max=10672, per=99.99%, avg=9979.66, stdev=452.90, samples=105
  2092. iops : min= 2062, max= 2668, avg=2494.86, stdev=113.20, samples=105
  2093. lat (usec) : 4=0.01%, 250=0.01%, 500=20.78%, 750=68.67%, 1000=9.93%
  2094. lat (msec) : 2=0.41%, 4=0.21%, 10=0.01%
  2095. fsync/fdatasync/sync_file_range:
  2096. sync (nsec): min=57, max=12758, avg=213.42, stdev=175.60
  2097. sync percentiles (nsec):
  2098. | 1.00th=[ 102], 5.00th=[ 131], 10.00th=[ 147], 20.00th=[ 165],
  2099. | 30.00th=[ 179], 40.00th=[ 191], 50.00th=[ 203], 60.00th=[ 215],
  2100. | 70.00th=[ 227], 80.00th=[ 241], 90.00th=[ 266], 95.00th=[ 298],
  2101. | 99.00th=[ 524], 99.50th=[ 540], 99.90th=[ 572], 99.95th=[ 596],
  2102. | 99.99th=[11840]
  2103. cpu : usr=3.27%, sys=14.74%, ctx=776627, majf=0, minf=17
  2104. IO depths : 1=0.1%, 2=0.1%, 4=200.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  2105. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  2106. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  2107. issued rwts: total=131098,131046,0,262141 short=0,0,0,0 dropped=0,0,0,0
  2108. latency : target=0, window=0, percentile=100.00%, depth=4
  2109.  
  2110. Run status group 0 (all jobs):
  2111. READ: bw=1009MiB/s (1058MB/s), 1009MiB/s-1009MiB/s (1058MB/s-1058MB/s), io=1024MiB (1074MB), run=1015-1015msec
  2112.  
  2113. Run status group 1 (all jobs):
  2114. WRITE: bw=1092MiB/s (1145MB/s), 1092MiB/s-1092MiB/s (1145MB/s-1145MB/s), io=1024MiB (1074MB), run=938-938msec
  2115.  
  2116. Run status group 2 (all jobs):
  2117. WRITE: bw=25.4MiB/s (26.6MB/s), 25.4MiB/s-25.4MiB/s (26.6MB/s-26.6MB/s), io=1024MiB (1074MB), run=40300-40300msec
  2118.  
  2119. Run status group 3 (all jobs):
  2120. READ: bw=31.7MiB/s (33.3MB/s), 31.7MiB/s-31.7MiB/s (33.3MB/s-33.3MB/s), io=1024MiB (1074MB), run=32272-32272msec
  2121.  
  2122. Run status group 4 (all jobs):
  2123. WRITE: bw=1021MiB/s (1071MB/s), 1021MiB/s-1021MiB/s (1071MB/s-1071MB/s), io=1024MiB (1074MB), run=1003-1003msec
  2124.  
  2125. Run status group 5 (all jobs):
  2126. WRITE: bw=25.6MiB/s (26.9MB/s), 25.6MiB/s-25.6MiB/s (26.9MB/s-26.9MB/s), io=1024MiB (1074MB), run=39955-39955msec
  2127.  
  2128. Run status group 6 (all jobs):
  2129. READ: bw=30.2MiB/s (31.7MB/s), 30.2MiB/s-30.2MiB/s (31.7MB/s-31.7MB/s), io=512MiB (537MB), run=16938-16938msec
  2130. WRITE: bw=30.2MiB/s (31.7MB/s), 30.2MiB/s-30.2MiB/s (31.7MB/s-31.7MB/s), io=512MiB (537MB), run=16938-16938msec
  2131.  
  2132. Run status group 7 (all jobs):
  2133. READ: bw=9985KiB/s (10.2MB/s), 9985KiB/s-9985KiB/s (10.2MB/s-10.2MB/s), io=512MiB (537MB), run=52518-52518msec
  2134. WRITE: bw=9981KiB/s (10.2MB/s), 9981KiB/s-9981KiB/s (10.2MB/s-10.2MB/s), io=512MiB (537MB), run=52518-52518msec
  2135.  
  2136.  
  2137. =============================
  2138. single_hdd_bench-4k-IOD16.txt
  2139. =============================
  2140. seqRead: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
  2141. seqWrite: (g=1): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
  2142. seqWrite/sync: (g=2): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
  2143. randRead: (g=3): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
  2144. randWrite: (g=4): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
  2145. randWrite/sync: (g=5): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
  2146. randReadWrite: (g=6): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
  2147. randReadWrite/sync: (g=7): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
  2148. fio-3.12
  2149. Starting 8 processes
  2150.  
  2151. seqRead: (groupid=0, jobs=1): err= 0: pid=43491: Fri Mar 20 16:01:26 2020
  2152. read: IOPS=39.3k, BW=154MiB/s (161MB/s)(1024MiB/6670msec)
  2153. slat (nsec): min=1179, max=84206k, avg=23919.38, stdev=732455.47
  2154. clat (usec): min=3, max=84251, avg=382.64, stdev=2820.70
  2155. lat (usec): min=7, max=84253, avg=406.64, stdev=2911.97
  2156. clat percentiles (usec):
  2157. | 1.00th=[ 31], 5.00th=[ 37], 10.00th=[ 37], 20.00th=[ 38],
  2158. | 30.00th=[ 38], 40.00th=[ 63], 50.00th=[ 81], 60.00th=[ 82],
  2159. | 70.00th=[ 84], 80.00th=[ 750], 90.00th=[ 914], 95.00th=[ 988],
  2160. | 99.00th=[ 1303], 99.50th=[ 1385], 99.90th=[67634], 99.95th=[67634],
  2161. | 99.99th=[69731]
  2162. bw ( KiB/s): min=12293, max=214672, per=70.95%, avg=111537.17, stdev=65388.42, samples=12
  2163. iops : min= 3073, max=53668, avg=27883.83, stdev=16347.06, samples=12
  2164. lat (usec) : 4=0.01%, 10=0.01%, 20=0.01%, 50=37.24%, 100=39.06%
  2165. lat (usec) : 250=0.50%, 500=0.06%, 750=2.97%, 1000=15.59%
  2166. lat (msec) : 2=4.29%, 4=0.03%, 10=0.03%, 20=0.01%, 50=0.04%
  2167. lat (msec) : 100=0.17%
  2168. cpu : usr=7.38%, sys=16.33%, ctx=4062, majf=0, minf=27
  2169. IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
  2170. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  2171. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
  2172. issued rwts: total=262144,0,0,0 short=0,0,0,0 dropped=0,0,0,0
  2173. latency : target=0, window=0, percentile=100.00%, depth=16
  2174. seqWrite: (groupid=1, jobs=1): err= 0: pid=43679: Fri Mar 20 16:01:26 2020
  2175. write: IOPS=277k, BW=1082MiB/s (1135MB/s)(1024MiB/946msec); 0 zone resets
  2176. slat (nsec): min=1621, max=29202, avg=2242.36, stdev=505.10
  2177. clat (nsec): min=1613, max=108337, avg=54648.89, stdev=3053.67
  2178. lat (usec): min=3, max=111, avg=56.95, stdev= 3.21
  2179. clat percentiles (nsec):
  2180. | 1.00th=[51968], 5.00th=[52480], 10.00th=[52480], 20.00th=[52992],
  2181. | 30.00th=[52992], 40.00th=[53504], 50.00th=[54016], 60.00th=[54016],
  2182. | 70.00th=[54528], 80.00th=[55552], 90.00th=[57600], 95.00th=[61184],
  2183. | 99.00th=[67072], 99.50th=[70144], 99.90th=[76288], 99.95th=[80384],
  2184. | 99.99th=[90624]
  2185. bw ( MiB/s): min= 1078, max= 1078, per=99.62%, avg=1078.31, stdev= 0.00, samples=1
  2186. iops : min=276048, max=276048, avg=276048.00, stdev= 0.00, samples=1
  2187. lat (usec) : 2=0.01%, 10=0.01%, 20=0.01%, 50=0.01%, 100=99.99%
  2188. lat (usec) : 250=0.01%
  2189. cpu : usr=40.95%, sys=58.73%, ctx=1, majf=0, minf=11
  2190. IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
  2191. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  2192. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
  2193. issued rwts: total=0,262144,0,0 short=0,0,0,0 dropped=0,0,0,0
  2194. latency : target=0, window=0, percentile=100.00%, depth=16
  2195. seqWrite/sync: (groupid=2, jobs=1): err= 0: pid=43833: Fri Mar 20 16:01:26 2020
  2196. write: IOPS=115, BW=464KiB/s (475kB/s)(27.2MiB/60005msec); 0 zone resets
  2197. slat (nsec): min=5583, max=48346, avg=18890.61, stdev=5131.06
  2198. clat (msec): min=6, max=385, avg=128.60, stdev=10.23
  2199. lat (msec): min=6, max=385, avg=128.62, stdev=10.23
  2200. clat percentiles (msec):
  2201. | 1.00th=[ 125], 5.00th=[ 126], 10.00th=[ 126], 20.00th=[ 126],
  2202. | 30.00th=[ 126], 40.00th=[ 126], 50.00th=[ 126], 60.00th=[ 126],
  2203. | 70.00th=[ 126], 80.00th=[ 126], 90.00th=[ 150], 95.00th=[ 150],
  2204. | 99.00th=[ 150], 99.50th=[ 153], 99.90th=[ 169], 99.95th=[ 176],
  2205. | 99.99th=[ 384]
  2206. bw ( KiB/s): min= 193, max= 381, per=78.22%, avg=362.15, stdev=18.97, samples=119
  2207. iops : min= 48, max= 95, avg=90.15, stdev= 4.62, samples=119
  2208. lat (msec) : 10=0.03%, 20=0.03%, 50=0.10%, 100=0.17%, 250=99.66%
  2209. lat (msec) : 500=0.01%
  2210. fsync/fdatasync/sync_file_range:
  2211. sync (nsec): min=188, max=13928, avg=563.36, stdev=392.30
  2212. sync percentiles (nsec):
  2213. | 1.00th=[ 330], 5.00th=[ 362], 10.00th=[ 366], 20.00th=[ 374],
  2214. | 30.00th=[ 426], 40.00th=[ 620], 50.00th=[ 628], 60.00th=[ 636],
  2215. | 70.00th=[ 644], 80.00th=[ 644], 90.00th=[ 652], 95.00th=[ 660],
  2216. | 99.00th=[ 668], 99.50th=[ 676], 99.90th=[ 988], 99.95th=[13120],
  2217. | 99.99th=[13888]
  2218. cpu : usr=0.19%, sys=0.84%, ctx=20835, majf=0, minf=13
  2219. IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=199.6%, 32=0.0%, >=64=0.0%
  2220. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  2221. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
  2222. issued rwts: total=0,6957,0,6943 short=0,0,0,0 dropped=0,0,0,0
  2223. latency : target=0, window=0, percentile=100.00%, depth=16
  2224. randRead: (groupid=3, jobs=1): err= 0: pid=45855: Fri Mar 20 16:01:26 2020
  2225. read: IOPS=188, BW=755KiB/s (773kB/s)(44.2MiB/60002msec)
  2226. slat (usec): min=73, max=38019, avg=5290.67, stdev=2845.48
  2227. clat (usec): min=10, max=140765, avg=79437.68, stdev=10759.69
  2228. lat (msec): min=6, max=146, avg=84.73, stdev=11.08
  2229. clat percentiles (msec):
  2230. | 1.00th=[ 55], 5.00th=[ 63], 10.00th=[ 67], 20.00th=[ 71],
  2231. | 30.00th=[ 74], 40.00th=[ 78], 50.00th=[ 80], 60.00th=[ 83],
  2232. | 70.00th=[ 85], 80.00th=[ 88], 90.00th=[ 93], 95.00th=[ 97],
  2233. | 99.00th=[ 107], 99.50th=[ 113], 99.90th=[ 133], 99.95th=[ 134],
  2234. | 99.99th=[ 138]
  2235. bw ( KiB/s): min= 518, max= 684, per=82.18%, avg=619.62, stdev=32.57, samples=119
  2236. iops : min= 129, max= 171, avg=154.54, stdev= 8.10, samples=119
  2237. lat (usec) : 20=0.01%
  2238. lat (msec) : 10=0.01%, 20=0.02%, 50=0.25%, 100=96.79%, 250=2.92%
  2239. cpu : usr=0.19%, sys=0.79%, ctx=11329, majf=0, minf=26
  2240. IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=99.9%, 32=0.0%, >=64=0.0%
  2241. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  2242. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
  2243. issued rwts: total=11320,0,0,0 short=0,0,0,0 dropped=0,0,0,0
  2244. latency : target=0, window=0, percentile=100.00%, depth=16
  2245. randWrite: (groupid=4, jobs=1): err= 0: pid=47832: Fri Mar 20 16:01:26 2020
  2246. write: IOPS=261k, BW=1020MiB/s (1069MB/s)(1024MiB/1004msec); 0 zone resets
  2247. slat (nsec): min=1820, max=108866, avg=2356.81, stdev=557.61
  2248. clat (nsec): min=1411, max=172298, avg=57983.29, stdev=3538.70
  2249. lat (usec): min=3, max=175, avg=60.40, stdev= 3.71
  2250. clat percentiles (usec):
  2251. | 1.00th=[ 56], 5.00th=[ 56], 10.00th=[ 56], 20.00th=[ 57],
  2252. | 30.00th=[ 57], 40.00th=[ 57], 50.00th=[ 58], 60.00th=[ 58],
  2253. | 70.00th=[ 58], 80.00th=[ 59], 90.00th=[ 62], 95.00th=[ 66],
  2254. | 99.00th=[ 74], 99.50th=[ 76], 99.90th=[ 83], 99.95th=[ 88],
  2255. | 99.99th=[ 101]
  2256. bw ( KiB/s): min=1032427, max=1032427, per=98.85%, avg=1032427.00, stdev= 0.00, samples=1
  2257. iops : min=258106, max=258106, avg=258106.00, stdev= 0.00, samples=1
  2258. lat (usec) : 2=0.01%, 10=0.01%, 20=0.01%, 50=0.01%, 100=99.98%
  2259. lat (usec) : 250=0.01%
  2260. cpu : usr=36.99%, sys=62.91%, ctx=1, majf=0, minf=10
  2261. IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
  2262. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  2263. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
  2264. issued rwts: total=0,262144,0,0 short=0,0,0,0 dropped=0,0,0,0
  2265. latency : target=0, window=0, percentile=100.00%, depth=16
  2266. randWrite/sync: (groupid=5, jobs=1): err= 0: pid=48046: Fri Mar 20 16:01:26 2020
  2267. write: IOPS=145, BW=582KiB/s (596kB/s)(34.1MiB/60005msec); 0 zone resets
  2268. slat (nsec): min=3149, max=54433, avg=20231.91, stdev=5406.36
  2269. clat (msec): min=11, max=186, avg=102.76, stdev=10.46
  2270. lat (msec): min=11, max=186, avg=102.78, stdev=10.46
  2271. clat percentiles (msec):
  2272. | 1.00th=[ 82], 5.00th=[ 87], 10.00th=[ 90], 20.00th=[ 94],
  2273. | 30.00th=[ 97], 40.00th=[ 101], 50.00th=[ 103], 60.00th=[ 106],
  2274. | 70.00th=[ 108], 80.00th=[ 111], 90.00th=[ 115], 95.00th=[ 120],
  2275. | 99.00th=[ 128], 99.50th=[ 140], 99.90th=[ 161], 99.95th=[ 167],
  2276. | 99.99th=[ 186]
  2277. bw ( KiB/s): min= 357, max= 488, per=74.57%, avg=434.03, stdev=20.27, samples=119
  2278. iops : min= 89, max= 122, avg=108.09, stdev= 5.11, samples=119
  2279. lat (msec) : 20=0.02%, 50=0.06%, 100=39.18%, 250=60.74%
  2280. fsync/fdatasync/sync_file_range:
  2281. sync (nsec): min=223, max=16470, avg=580.22, stdev=433.66
  2282. sync percentiles (nsec):
  2283. | 1.00th=[ 318], 5.00th=[ 342], 10.00th=[ 374], 20.00th=[ 402],
  2284. | 30.00th=[ 458], 40.00th=[ 604], 50.00th=[ 620], 60.00th=[ 628],
  2285. | 70.00th=[ 660], 80.00th=[ 684], 90.00th=[ 700], 95.00th=[ 700],
  2286. | 99.00th=[ 716], 99.50th=[ 716], 99.90th=[ 884], 99.95th=[14144],
  2287. | 99.99th=[16512]
  2288. cpu : usr=0.21%, sys=1.09%, ctx=26164, majf=0, minf=11
  2289. IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=199.7%, 32=0.0%, >=64=0.0%
  2290. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  2291. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
  2292. issued rwts: total=0,8734,0,8720 short=0,0,0,0 dropped=0,0,0,0
  2293. latency : target=0, window=0, percentile=100.00%, depth=16
  2294. randReadWrite: (groupid=6, jobs=1): err= 0: pid=50032: Fri Mar 20 16:01:26 2020
  2295. read: IOPS=143, BW=575KiB/s (589kB/s)(33.7MiB/60004msec)
  2296. slat (usec): min=180, max=42643, avg=6926.54, stdev=2919.59
  2297. clat (usec): min=5, max=126964, avg=51871.85, stdev=16449.77
  2298. lat (msec): min=3, max=133, avg=58.80, stdev=16.89
  2299. clat percentiles (msec):
  2300. | 1.00th=[ 18], 5.00th=[ 27], 10.00th=[ 32], 20.00th=[ 38],
  2301. | 30.00th=[ 43], 40.00th=[ 47], 50.00th=[ 52], 60.00th=[ 56],
  2302. | 70.00th=[ 61], 80.00th=[ 66], 90.00th=[ 73], 95.00th=[ 81],
  2303. | 99.00th=[ 95], 99.50th=[ 101], 99.90th=[ 113], 99.95th=[ 114],
  2304. | 99.99th=[ 128]
  2305. bw ( KiB/s): min= 386, max= 609, per=77.99%, avg=448.44, stdev=45.82, samples=119
  2306. iops : min= 96, max= 152, avg=111.72, stdev=11.45, samples=119
  2307. write: IOPS=145, BW=584KiB/s (598kB/s)(34.2MiB/60004msec); 0 zone resets
  2308. slat (nsec): min=1788, max=55599, avg=11736.89, stdev=7300.25
  2309. clat (usec): min=166, max=127188, avg=51657.91, stdev=16555.69
  2310. lat (usec): min=173, max=127192, avg=51669.84, stdev=16555.76
  2311. clat percentiles (msec):
  2312. | 1.00th=[ 18], 5.00th=[ 26], 10.00th=[ 31], 20.00th=[ 38],
  2313. | 30.00th=[ 43], 40.00th=[ 47], 50.00th=[ 52], 60.00th=[ 56],
  2314. | 70.00th=[ 61], 80.00th=[ 66], 90.00th=[ 73], 95.00th=[ 81],
  2315. | 99.00th=[ 93], 99.50th=[ 97], 99.90th=[ 113], 99.95th=[ 117],
  2316. | 99.99th=[ 128]
  2317. bw ( KiB/s): min= 274, max= 685, per=77.87%, avg=454.01, stdev=84.63, samples=119
  2318. iops : min= 68, max= 171, avg=113.14, stdev=21.15, samples=119
  2319. lat (usec) : 10=0.01%, 250=0.05%
  2320. lat (msec) : 4=0.01%, 10=0.16%, 20=1.54%, 50=45.28%, 100=52.53%
  2321. lat (msec) : 250=0.44%
  2322. cpu : usr=0.22%, sys=0.73%, ctx=8643, majf=0, minf=13
  2323. IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=99.9%, 32=0.0%, >=64=0.0%
  2324. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  2325. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
  2326. issued rwts: total=8630,8754,0,0 short=0,0,0,0 dropped=0,0,0,0
  2327. latency : target=0, window=0, percentile=100.00%, depth=16
  2328. randReadWrite/sync: (groupid=7, jobs=1): err= 0: pid=52056: Fri Mar 20 16:01:26 2020
  2329. read: IOPS=47, BW=190KiB/s (195kB/s)(11.1MiB/60005msec)
  2330. slat (usec): min=252, max=35561, avg=7296.27, stdev=3038.43
  2331. clat (usec): min=10, max=284378, avg=151108.55, stdev=21763.18
  2332. lat (msec): min=9, max=294, avg=158.41, stdev=22.12
  2333. clat percentiles (msec):
  2334. | 1.00th=[ 109], 5.00th=[ 120], 10.00th=[ 126], 20.00th=[ 133],
  2335. | 30.00th=[ 140], 40.00th=[ 144], 50.00th=[ 150], 60.00th=[ 157],
  2336. | 70.00th=[ 161], 80.00th=[ 169], 90.00th=[ 180], 95.00th=[ 188],
  2337. | 99.00th=[ 207], 99.50th=[ 213], 99.90th=[ 224], 99.95th=[ 224],
  2338. | 99.99th=[ 284]
  2339. bw ( KiB/s): min= 48, max= 272, per=100.00%, avg=189.37, stdev=35.11, samples=120
  2340. iops : min= 12, max= 68, avg=47.28, stdev= 8.78, samples=120
  2341. write: IOPS=48, BW=193KiB/s (198kB/s)(11.3MiB/60005msec); 0 zone resets
  2342. slat (nsec): min=3201, max=41307, avg=19442.63, stdev=5431.39
  2343. clat (msec): min=17, max=311, avg=160.53, stdev=23.54
  2344. lat (msec): min=17, max=311, avg=160.55, stdev=23.54
  2345. clat percentiles (msec):
  2346. | 1.00th=[ 113], 5.00th=[ 126], 10.00th=[ 133], 20.00th=[ 142],
  2347. | 30.00th=[ 148], 40.00th=[ 153], 50.00th=[ 159], 60.00th=[ 165],
  2348. | 70.00th=[ 171], 80.00th=[ 180], 90.00th=[ 190], 95.00th=[ 201],
  2349. | 99.00th=[ 226], 99.50th=[ 228], 99.90th=[ 239], 99.95th=[ 241],
  2350. | 99.99th=[ 313]
  2351. bw ( KiB/s): min= 88, max= 272, per=99.88%, avg=192.77, stdev=33.10, samples=120
  2352. iops : min= 22, max= 68, avg=48.12, stdev= 8.27, samples=120
  2353. lat (usec) : 20=0.02%
  2354. lat (msec) : 20=0.03%, 50=0.03%, 100=0.26%, 250=99.62%, 500=0.03%
  2355. fsync/fdatasync/sync_file_range:
  2356. sync (nsec): min=223, max=13984, avg=556.05, stdev=433.56
  2357. sync percentiles (nsec):
  2358. | 1.00th=[ 338], 5.00th=[ 362], 10.00th=[ 366], 20.00th=[ 378],
  2359. | 30.00th=[ 418], 40.00th=[ 588], 50.00th=[ 604], 60.00th=[ 620],
  2360. | 70.00th=[ 628], 80.00th=[ 636], 90.00th=[ 644], 95.00th=[ 660],
  2361. | 99.00th=[ 684], 99.50th=[ 692], 99.90th=[12352], 99.95th=[13760],
  2362. | 99.99th=[14016]
  2363. cpu : usr=0.17%, sys=0.66%, ctx=17217, majf=0, minf=13
  2364. IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=199.5%, 32=0.0%, >=64=0.0%
  2365. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  2366. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
  2367. issued rwts: total=2850,2901,0,5736 short=0,0,0,0 dropped=0,0,0,0
  2368. latency : target=0, window=0, percentile=100.00%, depth=16
  2369.  
  2370. Run status group 0 (all jobs):
  2371. READ: bw=154MiB/s (161MB/s), 154MiB/s-154MiB/s (161MB/s-161MB/s), io=1024MiB (1074MB), run=6670-6670msec
  2372.  
  2373. Run status group 1 (all jobs):
  2374. WRITE: bw=1082MiB/s (1135MB/s), 1082MiB/s-1082MiB/s (1135MB/s-1135MB/s), io=1024MiB (1074MB), run=946-946msec
  2375.  
  2376. Run status group 2 (all jobs):
  2377. WRITE: bw=464KiB/s (475kB/s), 464KiB/s-464KiB/s (475kB/s-475kB/s), io=27.2MiB (28.5MB), run=60005-60005msec
  2378.  
  2379. Run status group 3 (all jobs):
  2380. READ: bw=755KiB/s (773kB/s), 755KiB/s-755KiB/s (773kB/s-773kB/s), io=44.2MiB (46.4MB), run=60002-60002msec
  2381.  
  2382. Run status group 4 (all jobs):
  2383. WRITE: bw=1020MiB/s (1069MB/s), 1020MiB/s-1020MiB/s (1069MB/s-1069MB/s), io=1024MiB (1074MB), run=1004-1004msec
  2384.  
  2385. Run status group 5 (all jobs):
  2386. WRITE: bw=582KiB/s (596kB/s), 582KiB/s-582KiB/s (596kB/s-596kB/s), io=34.1MiB (35.8MB), run=60005-60005msec
  2387.  
  2388. Run status group 6 (all jobs):
  2389. READ: bw=575KiB/s (589kB/s), 575KiB/s-575KiB/s (589kB/s-589kB/s), io=33.7MiB (35.3MB), run=60004-60004msec
  2390. WRITE: bw=584KiB/s (598kB/s), 584KiB/s-584KiB/s (598kB/s-598kB/s), io=34.2MiB (35.9MB), run=60004-60004msec
  2391.  
  2392. Run status group 7 (all jobs):
  2393. READ: bw=190KiB/s (195kB/s), 190KiB/s-190KiB/s (195kB/s-195kB/s), io=11.1MiB (11.7MB), run=60005-60005msec
  2394. WRITE: bw=193KiB/s (198kB/s), 193KiB/s-193KiB/s (198kB/s-198kB/s), io=11.3MiB (11.9MB), run=60005-60005msec
  2395.  
  2396. ============================
  2397. signle_hdd_bench-4k-IOD4.txt
  2398. ============================
  2399. seqRead: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=4
  2400. seqWrite: (g=1): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=4
  2401. seqWrite/sync: (g=2): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=4
  2402. randRead: (g=3): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=4
  2403. randWrite: (g=4): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=4
  2404. randWrite/sync: (g=5): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=4
  2405. randReadWrite: (g=6): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=4
  2406. randReadWrite/sync: (g=7): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=4
  2407. fio-3.12
  2408. Starting 8 processes
  2409.  
  2410. seqRead: (groupid=0, jobs=1): err= 0: pid=31807: Fri Mar 20 15:56:05 2020
  2411. read: IOPS=58.1k, BW=227MiB/s (238MB/s)(1024MiB/4513msec)
  2412. slat (nsec): min=1140, max=15008k, avg=15599.77, stdev=113206.99
  2413. clat (usec): min=4, max=15033, avg=52.65, stdev=194.87
  2414. lat (usec): min=7, max=15039, avg=68.35, stdev=223.66
  2415. clat percentiles (usec):
  2416. | 1.00th=[ 7], 5.00th=[ 8], 10.00th=[ 8], 20.00th=[ 8],
  2417. | 30.00th=[ 9], 40.00th=[ 9], 50.00th=[ 9], 60.00th=[ 19],
  2418. | 70.00th=[ 20], 80.00th=[ 20], 90.00th=[ 21], 95.00th=[ 34],
  2419. | 99.00th=[ 938], 99.50th=[ 1045], 99.90th=[ 1270], 99.95th=[ 1303],
  2420. | 99.99th=[ 1385]
  2421. bw ( KiB/s): min=150050, max=236769, per=74.63%, avg=173399.50, stdev=26725.62, samples=8
  2422. iops : min=37512, max=59192, avg=43349.38, stdev=6681.46, samples=8
  2423. lat (usec) : 10=59.24%, 20=21.94%, 50=14.12%, 100=0.01%, 250=0.01%
  2424. lat (usec) : 500=0.01%, 750=1.26%, 1000=2.79%
  2425. lat (msec) : 2=0.62%, 10=0.01%, 20=0.01%
  2426. cpu : usr=11.68%, sys=22.25%, ctx=4103, majf=0, minf=14
  2427. IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  2428. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  2429. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  2430. issued rwts: total=262144,0,0,0 short=0,0,0,0 dropped=0,0,0,0
  2431. latency : target=0, window=0, percentile=100.00%, depth=4
  2432. seqWrite: (groupid=1, jobs=1): err= 0: pid=31978: Fri Mar 20 15:56:05 2020
  2433. write: IOPS=251k, BW=979MiB/s (1027MB/s)(1024MiB/1046msec); 0 zone resets
  2434. slat (nsec): min=1883, max=98849, avg=2471.06, stdev=541.55
  2435. clat (nsec): min=1620, max=119311, avg=12521.73, stdev=1098.37
  2436. lat (usec): min=3, max=121, avg=15.07, stdev= 1.29
  2437. clat percentiles (nsec):
  2438. | 1.00th=[11712], 5.00th=[11712], 10.00th=[11840], 20.00th=[11840],
  2439. | 30.00th=[11968], 40.00th=[11968], 50.00th=[12096], 60.00th=[12352],
  2440. | 70.00th=[12864], 80.00th=[13120], 90.00th=[13504], 95.00th=[14272],
  2441. | 99.00th=[16192], 99.50th=[17536], 99.90th=[22400], 99.95th=[25216],
  2442. | 99.99th=[31104]
  2443. bw ( KiB/s): min=995631, max=995631, per=99.32%, avg=995631.00, stdev= 0.00, samples=1
  2444. iops : min=248907, max=248907, avg=248907.00, stdev= 0.00, samples=1
  2445. lat (usec) : 2=0.01%, 10=0.01%, 20=99.83%, 50=0.17%, 250=0.01%
  2446. cpu : usr=40.00%, sys=59.90%, ctx=0, majf=0, minf=12
  2447. IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  2448. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  2449. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  2450. issued rwts: total=0,262144,0,0 short=0,0,0,0 dropped=0,0,0,0
  2451. latency : target=0, window=0, percentile=100.00%, depth=4
  2452. seqWrite/sync: (groupid=2, jobs=1): err= 0: pid=33126: Fri Mar 20 15:56:05 2020
  2453. write: IOPS=115, BW=463KiB/s (474kB/s)(27.1MiB/60001msec); 0 zone resets
  2454. slat (nsec): min=6317, max=95255, avg=20032.68, stdev=5401.74
  2455. clat (msec): min=5, max=407, avg=25.80, stdev= 6.18
  2456. lat (msec): min=5, max=407, avg=25.82, stdev= 6.18
  2457. clat percentiles (msec):
  2458. | 1.00th=[ 26], 5.00th=[ 26], 10.00th=[ 26], 20.00th=[ 26],
  2459. | 30.00th=[ 26], 40.00th=[ 26], 50.00th=[ 26], 60.00th=[ 26],
  2460. | 70.00th=[ 26], 80.00th=[ 26], 90.00th=[ 26], 95.00th=[ 26],
  2461. | 99.00th=[ 51], 99.50th=[ 51], 99.90th=[ 67], 99.95th=[ 75],
  2462. | 99.99th=[ 409]
  2463. bw ( KiB/s): min= 205, max= 342, per=71.48%, avg=330.95, stdev=15.61, samples=119
  2464. iops : min= 51, max= 85, avg=82.36, stdev= 3.82, samples=119
  2465. lat (msec) : 10=0.03%, 20=0.29%, 50=97.98%, 100=1.68%, 500=0.01%
  2466. fsync/fdatasync/sync_file_range:
  2467. sync (nsec): min=230, max=14418, avg=658.66, stdev=673.47
  2468. sync percentiles (nsec):
  2469. | 1.00th=[ 374], 5.00th=[ 434], 10.00th=[ 446], 20.00th=[ 462],
  2470. | 30.00th=[ 510], 40.00th=[ 684], 50.00th=[ 700], 60.00th=[ 708],
  2471. | 70.00th=[ 708], 80.00th=[ 724], 90.00th=[ 732], 95.00th=[ 740],
  2472. | 99.00th=[ 748], 99.50th=[ 756], 99.90th=[13760], 99.95th=[13888],
  2473. | 99.99th=[14400]
  2474. cpu : usr=0.20%, sys=0.88%, ctx=20846, majf=0, minf=13
  2475. IO depths : 1=0.1%, 2=0.1%, 4=199.9%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  2476. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  2477. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  2478. issued rwts: total=0,6947,0,6945 short=0,0,0,0 dropped=0,0,0,0
  2479. latency : target=0, window=0, percentile=100.00%, depth=4
  2480. randRead: (groupid=3, jobs=1): err= 0: pid=35202: Fri Mar 20 15:56:05 2020
  2481. read: IOPS=188, BW=753KiB/s (772kB/s)(44.2MiB/60003msec)
  2482. slat (usec): min=72, max=37926, avg=5298.59, stdev=2827.75
  2483. clat (usec): min=4, max=52663, avg=15926.52, stdev=4946.10
  2484. lat (usec): min=1735, max=57050, avg=21226.20, stdev=5727.55
  2485. clat percentiles (usec):
  2486. | 1.00th=[ 4752], 5.00th=[ 7832], 10.00th=[ 9634], 20.00th=[11731],
  2487. | 30.00th=[13304], 40.00th=[14746], 50.00th=[16057], 60.00th=[17171],
  2488. | 70.00th=[18482], 80.00th=[20055], 90.00th=[22152], 95.00th=[23725],
  2489. | 99.00th=[26608], 99.50th=[27919], 99.90th=[42206], 99.95th=[46924],
  2490. | 99.99th=[50594]
  2491. bw ( KiB/s): min= 465, max= 669, per=74.89%, avg=563.96, stdev=35.21, samples=119
  2492. iops : min= 116, max= 167, avg=140.66, stdev= 8.73, samples=119
  2493. lat (usec) : 10=0.01%, 500=0.03%
  2494. lat (msec) : 2=0.05%, 4=0.56%, 10=10.94%, 20=68.44%, 50=19.96%
  2495. lat (msec) : 100=0.02%
  2496. cpu : usr=0.23%, sys=0.76%, ctx=11310, majf=0, minf=14
  2497. IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  2498. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  2499. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  2500. issued rwts: total=11303,0,0,0 short=0,0,0,0 dropped=0,0,0,0
  2501. latency : target=0, window=0, percentile=100.00%, depth=4
  2502. randWrite: (groupid=4, jobs=1): err= 0: pid=37150: Fri Mar 20 15:56:05 2020
  2503. write: IOPS=237k, BW=925MiB/s (970MB/s)(1024MiB/1107msec); 0 zone resets
  2504. slat (nsec): min=2118, max=30917, avg=2617.58, stdev=528.26
  2505. clat (nsec): min=1833, max=124254, avg=13231.30, stdev=1137.04
  2506. lat (usec): min=4, max=126, avg=15.93, stdev= 1.32
  2507. clat percentiles (nsec):
  2508. | 1.00th=[12224], 5.00th=[12352], 10.00th=[12480], 20.00th=[12608],
  2509. | 30.00th=[12608], 40.00th=[12736], 50.00th=[12864], 60.00th=[12992],
  2510. | 70.00th=[13504], 80.00th=[13760], 90.00th=[14272], 95.00th=[15040],
  2511. | 99.00th=[17024], 99.50th=[18560], 99.90th=[23680], 99.95th=[25728],
  2512. | 99.99th=[31104]
  2513. bw ( KiB/s): min=938407, max=938407, per=99.07%, avg=938407.00, stdev= 0.00, samples=1
  2514. iops : min=234601, max=234601, avg=234601.00, stdev= 0.00, samples=1
  2515. lat (usec) : 2=0.01%, 10=0.01%, 20=99.72%, 50=0.27%, 100=0.01%
  2516. lat (usec) : 250=0.01%
  2517. cpu : usr=46.75%, sys=52.98%, ctx=3, majf=0, minf=9
  2518. IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  2519. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  2520. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  2521. issued rwts: total=0,262144,0,0 short=0,0,0,0 dropped=0,0,0,0
  2522. latency : target=0, window=0, percentile=100.00%, depth=4
  2523. randWrite/sync: (groupid=5, jobs=1): err= 0: pid=37366: Fri Mar 20 15:56:05 2020
  2524. write: IOPS=144, BW=580KiB/s (594kB/s)(33.0MiB/60002msec); 0 zone resets
  2525. slat (usec): min=7, max=300, avg=21.26, stdev= 6.33
  2526. clat (msec): min=5, max=274, avg=20.64, stdev= 5.19
  2527. lat (msec): min=5, max=274, avg=20.66, stdev= 5.19
  2528. clat percentiles (msec):
  2529. | 1.00th=[ 12], 5.00th=[ 14], 10.00th=[ 16], 20.00th=[ 17],
  2530. | 30.00th=[ 19], 40.00th=[ 20], 50.00th=[ 21], 60.00th=[ 22],
  2531. | 70.00th=[ 23], 80.00th=[ 25], 90.00th=[ 27], 95.00th=[ 28],
  2532. | 99.00th=[ 31], 99.50th=[ 32], 99.90th=[ 52], 99.95th=[ 52],
  2533. | 99.99th=[ 275]
  2534. bw ( KiB/s): min= 357, max= 478, per=68.95%, avg=399.19, stdev=20.10, samples=119
  2535. iops : min= 89, max= 119, avg=99.39, stdev= 5.01, samples=119
  2536. lat (msec) : 10=0.32%, 20=44.02%, 50=55.55%, 100=0.09%, 500=0.01%
  2537. fsync/fdatasync/sync_file_range:
  2538. sync (nsec): min=241, max=15402, avg=626.28, stdev=373.20
  2539. sync percentiles (nsec):
  2540. | 1.00th=[ 378], 5.00th=[ 406], 10.00th=[ 426], 20.00th=[ 446],
  2541. | 30.00th=[ 510], 40.00th=[ 676], 50.00th=[ 684], 60.00th=[ 700],
  2542. | 70.00th=[ 708], 80.00th=[ 716], 90.00th=[ 716], 95.00th=[ 724],
  2543. | 99.00th=[ 732], 99.50th=[ 740], 99.90th=[ 892], 99.95th=[13120],
  2544. | 99.99th=[15424]
  2545. cpu : usr=0.30%, sys=1.04%, ctx=26084, majf=0, minf=11
  2546. IO depths : 1=0.1%, 2=0.1%, 4=199.9%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  2547. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  2548. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  2549. issued rwts: total=0,8696,0,8694 short=0,0,0,0 dropped=0,0,0,0
  2550. latency : target=0, window=0, percentile=100.00%, depth=4
  2551. randReadWrite: (groupid=6, jobs=1): err= 0: pid=39400: Fri Mar 20 15:56:05 2020
  2552. read: IOPS=143, BW=576KiB/s (590kB/s)(33.7MiB/60003msec)
  2553. slat (usec): min=141, max=40218, avg=6920.37, stdev=2965.40
  2554. clat (usec): min=10, max=50096, avg=10260.72, stdev=7024.15
  2555. lat (usec): min=225, max=62263, avg=17182.19, stdev=7718.73
  2556. clat percentiles (usec):
  2557. | 1.00th=[ 30], 5.00th=[ 48], 10.00th=[ 62], 20.00th=[ 4146],
  2558. | 30.00th=[ 6194], 40.00th=[ 8029], 50.00th=[ 9634], 60.00th=[11338],
  2559. | 70.00th=[13698], 80.00th=[16188], 90.00th=[19530], 95.00th=[22938],
  2560. | 99.00th=[28443], 99.50th=[30278], 99.90th=[39584], 99.95th=[44303],
  2561. | 99.99th=[50070]
  2562. bw ( KiB/s): min= 344, max= 681, per=72.04%, avg=414.20, stdev=43.52, samples=119
  2563. iops : min= 86, max= 170, avg=103.15, stdev=10.88, samples=119
  2564. write: IOPS=146, BW=586KiB/s (600kB/s)(34.4MiB/60003msec); 0 zone resets
  2565. slat (nsec): min=2213, max=43729, avg=11904.13, stdev=7238.68
  2566. clat (usec): min=12, max=56628, avg=10394.87, stdev=7005.50
  2567. lat (usec): min=15, max=56648, avg=10407.04, stdev=7005.53
  2568. clat percentiles (usec):
  2569. | 1.00th=[ 17], 5.00th=[ 22], 10.00th=[ 40], 20.00th=[ 4293],
  2570. | 30.00th=[ 6456], 40.00th=[ 8225], 50.00th=[ 9765], 60.00th=[11469],
  2571. | 70.00th=[13829], 80.00th=[16319], 90.00th=[19530], 95.00th=[22938],
  2572. | 99.00th=[28705], 99.50th=[31327], 99.90th=[36963], 99.95th=[42206],
  2573. | 99.99th=[56886]
  2574. bw ( KiB/s): min= 235, max= 630, per=71.93%, avg=421.49, stdev=82.50, samples=119
  2575. iops : min= 58, max= 157, avg=104.97, stdev=20.63, samples=119
  2576. lat (usec) : 20=2.36%, 50=6.47%, 100=4.19%, 250=0.10%, 500=0.01%
  2577. lat (usec) : 1000=0.11%
  2578. lat (msec) : 2=0.59%, 4=5.27%, 10=32.65%, 20=39.18%, 50=9.06%
  2579. lat (msec) : 100=0.02%
  2580. cpu : usr=0.24%, sys=0.71%, ctx=8651, majf=0, minf=12
  2581. IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  2582. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  2583. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  2584. issued rwts: total=8637,8795,0,0 short=0,0,0,0 dropped=0,0,0,0
  2585. latency : target=0, window=0, percentile=100.00%, depth=4
  2586. randReadWrite/sync: (groupid=7, jobs=1): err= 0: pid=41402: Fri Mar 20 15:56:05 2020
  2587. read: IOPS=46, BW=184KiB/s (189kB/s)(10.8MiB/60015msec)
  2588. slat (usec): min=262, max=38232, avg=7353.57, stdev=2974.26
  2589. clat (msec): min=9, max=278, avg=27.00, stdev= 9.66
  2590. lat (msec): min=14, max=284, avg=34.35, stdev=10.20
  2591. clat percentiles (msec):
  2592. | 1.00th=[ 14], 5.00th=[ 17], 10.00th=[ 19], 20.00th=[ 21],
  2593. | 30.00th=[ 23], 40.00th=[ 24], 50.00th=[ 27], 60.00th=[ 29],
  2594. | 70.00th=[ 31], 80.00th=[ 33], 90.00th=[ 36], 95.00th=[ 40],
  2595. | 99.00th=[ 49], 99.50th=[ 52], 99.90th=[ 83], 99.95th=[ 257],
  2596. | 99.99th=[ 279]
  2597. bw ( KiB/s): min= 95, max= 271, per=99.96%, avg=183.92, stdev=32.94, samples=120
  2598. iops : min= 23, max= 67, avg=45.83, stdev= 8.24, samples=120
  2599. write: IOPS=48, BW=192KiB/s (197kB/s)(11.3MiB/60015msec); 0 zone resets
  2600. slat (nsec): min=8716, max=49566, avg=20962.73, stdev=5564.84
  2601. clat (msec): min=17, max=286, avg=36.59, stdev=10.14
  2602. lat (msec): min=17, max=286, avg=36.61, stdev=10.14
  2603. clat percentiles (msec):
  2604. | 1.00th=[ 22], 5.00th=[ 25], 10.00th=[ 27], 20.00th=[ 29],
  2605. | 30.00th=[ 31], 40.00th=[ 34], 50.00th=[ 36], 60.00th=[ 38],
  2606. | 70.00th=[ 40], 80.00th=[ 44], 90.00th=[ 50], 95.00th=[ 54],
  2607. | 99.00th=[ 63], 99.50th=[ 68], 99.90th=[ 87], 99.95th=[ 94],
  2608. | 99.99th=[ 288]
  2609. bw ( KiB/s): min= 79, max= 279, per=99.86%, avg=191.72, stdev=33.17, samples=120
  2610. iops : min= 19, max= 69, avg=47.77, stdev= 8.33, samples=120
  2611. lat (msec) : 10=0.02%, 20=7.99%, 50=87.40%, 100=4.54%, 500=0.05%
  2612. fsync/fdatasync/sync_file_range:
  2613. sync (nsec): min=244, max=16460, avg=640.56, stdev=597.47
  2614. sync percentiles (nsec):
  2615. | 1.00th=[ 342], 5.00th=[ 414], 10.00th=[ 422], 20.00th=[ 426],
  2616. | 30.00th=[ 478], 40.00th=[ 692], 50.00th=[ 692], 60.00th=[ 700],
  2617. | 70.00th=[ 700], 80.00th=[ 708], 90.00th=[ 724], 95.00th=[ 764],
  2618. | 99.00th=[ 780], 99.50th=[ 780], 99.90th=[13632], 99.95th=[13760],
  2619. | 99.99th=[16512]
  2620. cpu : usr=0.19%, sys=0.68%, ctx=16932, majf=0, minf=12
  2621. IO depths : 1=0.1%, 2=0.1%, 4=199.9%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  2622. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  2623. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  2624. issued rwts: total=2762,2882,0,5640 short=0,0,0,0 dropped=0,0,0,0
  2625. latency : target=0, window=0, percentile=100.00%, depth=4
  2626.  
  2627. Run status group 0 (all jobs):
  2628. READ: bw=227MiB/s (238MB/s), 227MiB/s-227MiB/s (238MB/s-238MB/s), io=1024MiB (1074MB), run=4513-4513msec
  2629.  
  2630. Run status group 1 (all jobs):
  2631. WRITE: bw=979MiB/s (1027MB/s), 979MiB/s-979MiB/s (1027MB/s-1027MB/s), io=1024MiB (1074MB), run=1046-1046msec
  2632.  
  2633. Run status group 2 (all jobs):
  2634. WRITE: bw=463KiB/s (474kB/s), 463KiB/s-463KiB/s (474kB/s-474kB/s), io=27.1MiB (28.5MB), run=60001-60001msec
  2635.  
  2636. Run status group 3 (all jobs):
  2637. READ: bw=753KiB/s (772kB/s), 753KiB/s-753KiB/s (772kB/s-772kB/s), io=44.2MiB (46.3MB), run=60003-60003msec
  2638.  
  2639. Run status group 4 (all jobs):
  2640. WRITE: bw=925MiB/s (970MB/s), 925MiB/s-925MiB/s (970MB/s-970MB/s), io=1024MiB (1074MB), run=1107-1107msec
  2641.  
  2642. Run status group 5 (all jobs):
  2643. WRITE: bw=580KiB/s (594kB/s), 580KiB/s-580KiB/s (594kB/s-594kB/s), io=33.0MiB (35.6MB), run=60002-60002msec
  2644.  
  2645. Run status group 6 (all jobs):
  2646. READ: bw=576KiB/s (590kB/s), 576KiB/s-576KiB/s (590kB/s-590kB/s), io=33.7MiB (35.4MB), run=60003-60003msec
  2647. WRITE: bw=586KiB/s (600kB/s), 586KiB/s-586KiB/s (600kB/s-600kB/s), io=34.4MiB (36.0MB), run=60003-60003msec
  2648.  
  2649. Run status group 7 (all jobs):
  2650. READ: bw=184KiB/s (189kB/s), 184KiB/s-184KiB/s (189kB/s-189kB/s), io=10.8MiB (11.3MB), run=60015-60015msec
  2651. WRITE: bw=192KiB/s (197kB/s), 192KiB/s-192KiB/s (197kB/s-197kB/s), io=11.3MiB (11.8MB), run=60015-60015msec
  2652.  
  2653. ===========================================
  2654. hdd_pool_r10_24disks_lz4_bench-4k-IOD16.txt
  2655. ===========================================
  2656. seqRead: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
  2657. seqWrite: (g=1): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
  2658. seqWrite/sync: (g=2): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
  2659. randRead: (g=3): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
  2660. randWrite: (g=4): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
  2661. randWrite/sync: (g=5): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
  2662. randReadWrite: (g=6): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
  2663. randReadWrite/sync: (g=7): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
  2664. fio-3.12
  2665. Starting 8 processes
  2666.  
  2667. seqRead: (groupid=0, jobs=1): err= 0: pid=37323: Sun Mar 22 19:43:00 2020
  2668. read: IOPS=274k, BW=1072MiB/s (1124MB/s)(1024MiB/955msec)
  2669. slat (nsec): min=2027, max=66399, avg=2818.62, stdev=750.47
  2670. clat (nsec): min=1435, max=140287, avg=55157.41, stdev=4784.16
  2671. lat (usec): min=4, max=143, avg=58.04, stdev= 4.97
  2672. clat percentiles (usec):
  2673. | 1.00th=[ 50], 5.00th=[ 52], 10.00th=[ 52], 20.00th=[ 53],
  2674. | 30.00th=[ 54], 40.00th=[ 55], 50.00th=[ 55], 60.00th=[ 55],
  2675. | 70.00th=[ 56], 80.00th=[ 56], 90.00th=[ 58], 95.00th=[ 62],
  2676. | 99.00th=[ 80], 99.50th=[ 85], 99.90th=[ 100], 99.95th=[ 104],
  2677. | 99.99th=[ 116]
  2678. bw ( MiB/s): min= 1076, max= 1076, per=100.00%, avg=1076.70, stdev= 0.00, samples=1
  2679. iops : min=275635, max=275635, avg=275635.00, stdev= 0.00, samples=1
  2680. lat (usec) : 2=0.01%, 10=0.01%, 20=0.01%, 50=0.90%, 100=99.01%
  2681. lat (usec) : 250=0.09%
  2682. cpu : usr=31.76%, sys=68.13%, ctx=3, majf=0, minf=25
  2683. IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
  2684. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  2685. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
  2686. issued rwts: total=262144,0,0,0 short=0,0,0,0 dropped=0,0,0,0
  2687. latency : target=0, window=0, percentile=100.00%, depth=16
  2688. seqWrite: (groupid=1, jobs=1): err= 0: pid=37652: Sun Mar 22 19:43:00 2020
  2689. write: IOPS=148k, BW=579MiB/s (607MB/s)(1024MiB/1770msec); 0 zone resets
  2690. slat (nsec): min=4019, max=99001, avg=5216.34, stdev=1595.72
  2691. clat (nsec): min=1525, max=385433, avg=101838.80, stdev=15656.98
  2692. lat (usec): min=6, max=402, avg=107.12, stdev=16.43
  2693. clat percentiles (usec):
  2694. | 1.00th=[ 93], 5.00th=[ 95], 10.00th=[ 96], 20.00th=[ 96],
  2695. | 30.00th=[ 97], 40.00th=[ 97], 50.00th=[ 97], 60.00th=[ 97],
  2696. | 70.00th=[ 98], 80.00th=[ 100], 90.00th=[ 115], 95.00th=[ 130],
  2697. | 99.00th=[ 174], 99.50th=[ 194], 99.90th=[ 247], 99.95th=[ 265],
  2698. | 99.99th=[ 322]
  2699. bw ( KiB/s): min=576673, max=618142, per=100.00%, avg=604129.33, stdev=23779.58, samples=3
  2700. iops : min=144168, max=154535, avg=151032.00, stdev=5944.82, samples=3
  2701. lat (usec) : 2=0.01%, 10=0.01%, 20=0.01%, 50=0.01%, 100=80.27%
  2702. lat (usec) : 250=19.63%, 500=0.09%
  2703. cpu : usr=25.44%, sys=74.39%, ctx=9, majf=0, minf=12
  2704. IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
  2705. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  2706. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
  2707. issued rwts: total=0,262144,0,0 short=0,0,0,0 dropped=0,0,0,0
  2708. latency : target=0, window=0, percentile=100.00%, depth=16
  2709. seqWrite/sync: (groupid=2, jobs=1): err= 0: pid=40377: Sun Mar 22 19:43:00 2020
  2710. write: IOPS=100, BW=403KiB/s (413kB/s)(23.6MiB/60004msec); 0 zone resets
  2711. slat (usec): min=4, max=362, avg=34.70, stdev=11.45
  2712. clat (msec): min=7, max=482, avg=147.96, stdev=35.30
  2713. lat (msec): min=7, max=482, avg=148.00, stdev=35.30
  2714. clat percentiles (msec):
  2715. | 1.00th=[ 121], 5.00th=[ 124], 10.00th=[ 124], 20.00th=[ 126],
  2716. | 30.00th=[ 131], 40.00th=[ 134], 50.00th=[ 136], 60.00th=[ 142],
  2717. | 70.00th=[ 150], 80.00th=[ 161], 90.00th=[ 184], 95.00th=[ 215],
  2718. | 99.00th=[ 284], 99.50th=[ 330], 99.90th=[ 472], 99.95th=[ 472],
  2719. | 99.99th=[ 485]
  2720. bw ( KiB/s): min= 122, max= 368, per=76.55%, avg=308.49, stdev=44.87, samples=119
  2721. iops : min= 30, max= 92, avg=76.76, stdev=11.23, samples=119
  2722. lat (msec) : 10=0.02%, 20=0.05%, 50=0.07%, 100=0.17%, 250=97.17%
  2723. lat (msec) : 500=2.53%
  2724. fsync/fdatasync/sync_file_range:
  2725. sync (nsec): min=209, max=14362, avg=601.76, stdev=472.80
  2726. sync percentiles (nsec):
  2727. | 1.00th=[ 314], 5.00th=[ 378], 10.00th=[ 402], 20.00th=[ 418],
  2728. | 30.00th=[ 442], 40.00th=[ 470], 50.00th=[ 668], 60.00th=[ 692],
  2729. | 70.00th=[ 708], 80.00th=[ 740], 90.00th=[ 764], 95.00th=[ 772],
  2730. | 99.00th=[ 820], 99.50th=[ 820], 99.90th=[12352], 99.95th=[13376],
  2731. | 99.99th=[14400]
  2732. cpu : usr=0.23%, sys=1.74%, ctx=12121, majf=0, minf=47
  2733. IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=199.5%, 32=0.0%, >=64=0.0%
  2734. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  2735. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
  2736. issued rwts: total=0,6047,0,6033 short=0,0,0,0 dropped=0,0,0,0
  2737. latency : target=0, window=0, percentile=100.00%, depth=16
  2738. randRead: (groupid=3, jobs=1): err= 0: pid=45108: Sun Mar 22 19:43:00 2020
  2739. read: IOPS=226k, BW=883MiB/s (926MB/s)(1024MiB/1160msec)
  2740. slat (nsec): min=2691, max=99685, avg=3553.49, stdev=732.97
  2741. clat (nsec): min=1490, max=192567, avg=66859.10, stdev=5788.81
  2742. lat (usec): min=4, max=238, avg=70.47, stdev= 6.09
  2743. clat percentiles (usec):
  2744. | 1.00th=[ 62], 5.00th=[ 63], 10.00th=[ 64], 20.00th=[ 65],
  2745. | 30.00th=[ 65], 40.00th=[ 66], 50.00th=[ 66], 60.00th=[ 67],
  2746. | 70.00th=[ 68], 80.00th=[ 68], 90.00th=[ 72], 95.00th=[ 78],
  2747. | 99.00th=[ 94], 99.50th=[ 98], 99.90th=[ 114], 99.95th=[ 123],
  2748. | 99.99th=[ 169]
  2749. bw ( KiB/s): min=743801, max=882593, per=89.96%, avg=813197.00, stdev=98140.76, samples=2
  2750. iops : min=185950, max=220648, avg=203299.00, stdev=24535.19, samples=2
  2751. lat (usec) : 2=0.01%, 10=0.01%, 20=0.01%, 50=0.01%, 100=99.61%
  2752. lat (usec) : 250=0.38%
  2753. cpu : usr=23.04%, sys=76.79%, ctx=4, majf=0, minf=26
  2754. IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
  2755. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  2756. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
  2757. issued rwts: total=262144,0,0,0 short=0,0,0,0 dropped=0,0,0,0
  2758. latency : target=0, window=0, percentile=100.00%, depth=16
  2759. randWrite: (groupid=4, jobs=1): err= 0: pid=45240: Sun Mar 22 19:43:00 2020
  2760. write: IOPS=88.8k, BW=347MiB/s (364MB/s)(1024MiB/2952msec); 0 zone resets
  2761. slat (usec): min=4, max=8246, avg= 9.57, stdev=33.35
  2762. clat (nsec): min=1646, max=11974k, avg=169460.67, stdev=235709.39
  2763. lat (usec): min=6, max=12051, avg=179.10, stdev=248.86
  2764. clat percentiles (usec):
  2765. | 1.00th=[ 101], 5.00th=[ 104], 10.00th=[ 106], 20.00th=[ 108],
  2766. | 30.00th=[ 110], 40.00th=[ 111], 50.00th=[ 113], 60.00th=[ 115],
  2767. | 70.00th=[ 124], 80.00th=[ 143], 90.00th=[ 255], 95.00th=[ 478],
  2768. | 99.00th=[ 996], 99.50th=[ 1319], 99.90th=[ 2376], 99.95th=[ 3130],
  2769. | 99.99th=[ 8717]
  2770. bw ( KiB/s): min=177947, max=369165, per=72.44%, avg=257297.80, stdev=79983.07, samples=5
  2771. iops : min=44486, max=92291, avg=64323.80, stdev=19996.14, samples=5
  2772. lat (usec) : 2=0.01%, 10=0.01%, 20=0.01%, 50=0.01%, 100=0.40%
  2773. lat (usec) : 250=89.47%, 500=5.44%, 750=2.40%, 1000=1.30%
  2774. lat (msec) : 2=0.77%, 4=0.18%, 10=0.03%, 20=0.01%
  2775. cpu : usr=15.66%, sys=81.40%, ctx=1680, majf=0, minf=11
  2776. IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
  2777. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  2778. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
  2779. issued rwts: total=0,262144,0,0 short=0,0,0,0 dropped=0,0,0,0
  2780. latency : target=0, window=0, percentile=100.00%, depth=16
  2781. randWrite/sync: (groupid=5, jobs=1): err= 0: pid=57297: Sun Mar 22 19:43:00 2020
  2782. write: IOPS=103, BW=416KiB/s (426kB/s)(24.4MiB/60004msec); 0 zone resets
  2783. slat (usec): min=8, max=180, avg=43.58, stdev=15.55
  2784. clat (msec): min=8, max=829, avg=142.49, stdev=39.91
  2785. lat (msec): min=8, max=829, avg=142.54, stdev=39.91
  2786. clat percentiles (msec):
  2787. | 1.00th=[ 112], 5.00th=[ 114], 10.00th=[ 115], 20.00th=[ 121],
  2788. | 30.00th=[ 124], 40.00th=[ 127], 50.00th=[ 132], 60.00th=[ 138],
  2789. | 70.00th=[ 144], 80.00th=[ 157], 90.00th=[ 178], 95.00th=[ 201],
  2790. | 99.00th=[ 338], 99.50th=[ 368], 99.90th=[ 414], 99.95th=[ 422],
  2791. | 99.99th=[ 827]
  2792. bw ( KiB/s): min= 116, max= 511, per=97.26%, avg=403.64, stdev=71.89, samples=118
  2793. iops : min= 29, max= 127, avg=100.59, stdev=17.99, samples=118
  2794. lat (msec) : 10=0.02%, 20=0.02%, 50=0.13%, 100=0.16%, 250=97.02%
  2795. lat (msec) : 500=2.65%, 1000=0.02%
  2796. fsync/fdatasync/sync_file_range:
  2797. sync (nsec): min=174, max=13756, avg=590.56, stdev=480.09
  2798. sync percentiles (nsec):
  2799. | 1.00th=[ 298], 5.00th=[ 366], 10.00th=[ 386], 20.00th=[ 402],
  2800. | 30.00th=[ 430], 40.00th=[ 458], 50.00th=[ 676], 60.00th=[ 684],
  2801. | 70.00th=[ 692], 80.00th=[ 700], 90.00th=[ 732], 95.00th=[ 748],
  2802. | 99.00th=[ 772], 99.50th=[ 796], 99.90th=[12992], 99.95th=[13248],
  2803. | 99.99th=[13760]
  2804. cpu : usr=0.19%, sys=2.14%, ctx=12800, majf=0, minf=54
  2805. IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=199.5%, 32=0.0%, >=64=0.0%
  2806. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  2807. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
  2808. issued rwts: total=0,6235,0,6221 short=0,0,0,0 dropped=0,0,0,0
  2809. latency : target=0, window=0, percentile=100.00%, depth=16
  2810. randReadWrite: (groupid=6, jobs=1): err= 0: pid=9436: Sun Mar 22 19:43:00 2020
  2811. read: IOPS=54.3k, BW=212MiB/s (222MB/s)(512MiB/2416msec)
  2812. slat (usec): min=2, max=1918, avg= 5.93, stdev=13.07
  2813. clat (nsec): min=1512, max=6232.0k, avg=139314.59, stdev=163482.36
  2814. lat (usec): min=4, max=6243, avg=145.31, stdev=170.17
  2815. clat percentiles (usec):
  2816. | 1.00th=[ 73], 5.00th=[ 78], 10.00th=[ 81], 20.00th=[ 85],
  2817. | 30.00th=[ 88], 40.00th=[ 90], 50.00th=[ 93], 60.00th=[ 97],
  2818. | 70.00th=[ 105], 80.00th=[ 130], 90.00th=[ 215], 95.00th=[ 400],
  2819. | 99.00th=[ 857], 99.50th=[ 988], 99.90th=[ 1696], 99.95th=[ 2147],
  2820. | 99.99th=[ 3163]
  2821. bw ( KiB/s): min=101794, max=217663, per=72.09%, avg=156585.25, stdev=58587.99, samples=4
  2822. iops : min=25448, max=54415, avg=39145.75, stdev=14647.03, samples=4
  2823. write: IOPS=54.2k, BW=212MiB/s (222MB/s)(512MiB/2416msec); 0 zone resets
  2824. slat (usec): min=4, max=4822, avg= 9.73, stdev=22.15
  2825. clat (usec): min=5, max=6245, avg=138.24, stdev=152.41
  2826. lat (usec): min=10, max=6324, avg=148.05, stdev=164.71
  2827. clat percentiles (usec):
  2828. | 1.00th=[ 73], 5.00th=[ 78], 10.00th=[ 81], 20.00th=[ 85],
  2829. | 30.00th=[ 88], 40.00th=[ 90], 50.00th=[ 93], 60.00th=[ 97],
  2830. | 70.00th=[ 105], 80.00th=[ 130], 90.00th=[ 215], 95.00th=[ 396],
  2831. | 99.00th=[ 840], 99.50th=[ 971], 99.90th=[ 1663], 99.95th=[ 1958],
  2832. | 99.99th=[ 2868]
  2833. bw ( KiB/s): min=101463, max=216395, per=72.17%, avg=156457.50, stdev=58086.88, samples=4
  2834. iops : min=25367, max=54098, avg=39114.25, stdev=14521.08, samples=4
  2835. lat (usec) : 2=0.01%, 10=0.01%, 20=0.01%, 50=0.01%, 100=64.98%
  2836. lat (usec) : 250=26.53%, 500=4.83%, 750=2.14%, 1000=1.08%
  2837. lat (msec) : 2=0.40%, 4=0.05%, 10=0.01%
  2838. cpu : usr=16.73%, sys=81.66%, ctx=959, majf=0, minf=56
  2839. IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
  2840. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  2841. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
  2842. issued rwts: total=131198,130946,0,0 short=0,0,0,0 dropped=0,0,0,0
  2843. latency : target=0, window=0, percentile=100.00%, depth=16
  2844. randReadWrite/sync: (groupid=7, jobs=1): err= 0: pid=19780: Sun Mar 22 19:43:00 2020
  2845. read: IOPS=109, BW=440KiB/s (451kB/s)(25.8MiB/60016msec)
  2846. slat (usec): min=3, max=124, avg=21.05, stdev=14.96
  2847. clat (msec): min=4, max=343, avg=64.08, stdev=30.89
  2848. lat (msec): min=4, max=343, avg=64.10, stdev=30.89
  2849. clat percentiles (msec):
  2850. | 1.00th=[ 21], 5.00th=[ 31], 10.00th=[ 37], 20.00th=[ 45],
  2851. | 30.00th=[ 51], 40.00th=[ 54], 50.00th=[ 61], 60.00th=[ 65],
  2852. | 70.00th=[ 71], 80.00th=[ 79], 90.00th=[ 92], 95.00th=[ 105],
  2853. | 99.00th=[ 203], 99.50th=[ 275], 99.90th=[ 334], 99.95th=[ 338],
  2854. | 99.99th=[ 342]
  2855. bw ( KiB/s): min= 152, max= 744, per=100.00%, avg=443.11, stdev=110.61, samples=119
  2856. iops : min= 38, max= 186, avg=110.67, stdev=27.70, samples=119
  2857. write: IOPS=106, BW=427KiB/s (438kB/s)(25.0MiB/60016msec); 0 zone resets
  2858. slat (usec): min=9, max=338, avg=32.53, stdev=15.79
  2859. clat (msec): min=13, max=544, avg=73.25, stdev=33.29
  2860. lat (msec): min=13, max=544, avg=73.29, stdev=33.29
  2861. clat percentiles (msec):
  2862. | 1.00th=[ 30], 5.00th=[ 39], 10.00th=[ 45], 20.00th=[ 53],
  2863. | 30.00th=[ 58], 40.00th=[ 63], 50.00th=[ 68], 60.00th=[ 74],
  2864. | 70.00th=[ 79], 80.00th=[ 88], 90.00th=[ 102], 95.00th=[ 118],
  2865. | 99.00th=[ 241], 99.50th=[ 292], 99.90th=[ 334], 99.95th=[ 342],
  2866. | 99.99th=[ 542]
  2867. bw ( KiB/s): min= 208, max= 560, per=100.00%, avg=430.28, stdev=80.95, samples=119
  2868. iops : min= 52, max= 140, avg=107.46, stdev=20.25, samples=119
  2869. lat (msec) : 10=0.05%, 20=0.49%, 50=22.88%, 100=67.85%, 250=7.92%
  2870. lat (msec) : 500=0.81%, 750=0.01%
  2871. fsync/fdatasync/sync_file_range:
  2872. sync (nsec): min=20, max=14343, avg=334.41, stdev=471.13
  2873. sync percentiles (nsec):
  2874. | 1.00th=[ 25], 5.00th=[ 27], 10.00th=[ 29], 20.00th=[ 61],
  2875. | 30.00th=[ 83], 40.00th=[ 110], 50.00th=[ 161], 60.00th=[ 406],
  2876. | 70.00th=[ 474], 80.00th=[ 692], 90.00th=[ 724], 95.00th=[ 756],
  2877. | 99.00th=[ 788], 99.50th=[ 788], 99.90th=[ 1544], 99.95th=[13760],
  2878. | 99.99th=[14144]
  2879. cpu : usr=0.31%, sys=1.92%, ctx=12982, majf=0, minf=19
  2880. IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=199.8%, 32=0.0%, >=64=0.0%
  2881. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  2882. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
  2883. issued rwts: total=6601,6411,0,12998 short=0,0,0,0 dropped=0,0,0,0
  2884. latency : target=0, window=0, percentile=100.00%, depth=16
  2885.  
  2886. Run status group 0 (all jobs):
  2887. READ: bw=1072MiB/s (1124MB/s), 1072MiB/s-1072MiB/s (1124MB/s-1124MB/s), io=1024MiB (1074MB), run=955-955msec
  2888.  
  2889. Run status group 1 (all jobs):
  2890. WRITE: bw=579MiB/s (607MB/s), 579MiB/s-579MiB/s (607MB/s-607MB/s), io=1024MiB (1074MB), run=1770-1770msec
  2891.  
  2892. Run status group 2 (all jobs):
  2893. WRITE: bw=403KiB/s (413kB/s), 403KiB/s-403KiB/s (413kB/s-413kB/s), io=23.6MiB (24.8MB), run=60004-60004msec
  2894.  
  2895. Run status group 3 (all jobs):
  2896. READ: bw=883MiB/s (926MB/s), 883MiB/s-883MiB/s (926MB/s-926MB/s), io=1024MiB (1074MB), run=1160-1160msec
  2897.  
  2898. Run status group 4 (all jobs):
  2899. WRITE: bw=347MiB/s (364MB/s), 347MiB/s-347MiB/s (364MB/s-364MB/s), io=1024MiB (1074MB), run=2952-2952msec
  2900.  
  2901. Run status group 5 (all jobs):
  2902. WRITE: bw=416KiB/s (426kB/s), 416KiB/s-416KiB/s (426kB/s-426kB/s), io=24.4MiB (25.5MB), run=60004-60004msec
  2903.  
  2904. Run status group 6 (all jobs):
  2905. READ: bw=212MiB/s (222MB/s), 212MiB/s-212MiB/s (222MB/s-222MB/s), io=512MiB (537MB), run=2416-2416msec
  2906. WRITE: bw=212MiB/s (222MB/s), 212MiB/s-212MiB/s (222MB/s-222MB/s), io=512MiB (536MB), run=2416-2416msec
  2907.  
  2908. Run status group 7 (all jobs):
  2909. READ: bw=440KiB/s (451kB/s), 440KiB/s-440KiB/s (451kB/s-451kB/s), io=25.8MiB (27.0MB), run=60016-60016msec
  2910. WRITE: bw=427KiB/s (438kB/s), 427KiB/s-427KiB/s (438kB/s-438kB/s), io=25.0MiB (26.3MB), run=60016-60016msec
  2911.  
  2912. ===========================================
  2913. hdd_pool_r10_24disks_lz4_bench-4k-IOD4.txt
  2914. ===========================================
  2915. seqRead: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=4
  2916. seqWrite: (g=1): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=4
  2917. seqWrite/sync: (g=2): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=4
  2918. randRead: (g=3): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=4
  2919. randWrite: (g=4): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=4
  2920. randWrite/sync: (g=5): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=4
  2921. randReadWrite: (g=6): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=4
  2922. randReadWrite/sync: (g=7): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=4
  2923. fio-3.12
  2924. Starting 8 processes
  2925. seqRead: Laying out IO file (1 file / 1024MiB)
  2926.  
  2927. seqRead: (groupid=0, jobs=1): err= 0: pid=46859: Sun Mar 22 19:39:48 2020
  2928. read: IOPS=278k, BW=1086MiB/s (1139MB/s)(1024MiB/943msec)
  2929. slat (nsec): min=2362, max=41041, avg=2780.13, stdev=423.73
  2930. clat (nsec): min=1876, max=52768, avg=11341.44, stdev=863.88
  2931. lat (nsec): min=4806, max=65852, avg=14176.48, stdev=1004.23
  2932. clat percentiles (nsec):
  2933. | 1.00th=[10304], 5.00th=[10560], 10.00th=[10688], 20.00th=[10944],
  2934. | 30.00th=[11072], 40.00th=[11072], 50.00th=[11200], 60.00th=[11328],
  2935. | 70.00th=[11456], 80.00th=[11712], 90.00th=[12096], 95.00th=[12480],
  2936. | 99.00th=[13120], 99.50th=[13888], 99.90th=[22656], 99.95th=[24192],
  2937. | 99.99th=[31872]
  2938. bw ( MiB/s): min= 1083, max= 1083, per=99.76%, avg=1083.29, stdev= 0.00, samples=1
  2939. iops : min=277323, max=277323, avg=277323.00, stdev= 0.00, samples=1
  2940. lat (usec) : 2=0.01%, 10=0.01%, 20=99.79%, 50=0.21%, 100=0.01%
  2941. cpu : usr=34.29%, sys=65.71%, ctx=3, majf=7, minf=13
  2942. IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  2943. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  2944. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  2945. issued rwts: total=262144,0,0,0 short=0,0,0,0 dropped=0,0,0,0
  2946. latency : target=0, window=0, percentile=100.00%, depth=4
  2947. seqWrite: (groupid=1, jobs=1): err= 0: pid=46977: Sun Mar 22 19:39:48 2020
  2948. write: IOPS=152k, BW=593MiB/s (622MB/s)(1024MiB/1726msec); 0 zone resets
  2949. slat (nsec): min=3929, max=96658, avg=5054.64, stdev=1580.56
  2950. clat (nsec): min=1549, max=168978, avg=20357.34, stdev=4391.32
  2951. lat (usec): min=6, max=202, avg=25.48, stdev= 5.46
  2952. clat percentiles (usec):
  2953. | 1.00th=[ 19], 5.00th=[ 19], 10.00th=[ 19], 20.00th=[ 19],
  2954. | 30.00th=[ 19], 40.00th=[ 19], 50.00th=[ 19], 60.00th=[ 20],
  2955. | 70.00th=[ 21], 80.00th=[ 22], 90.00th=[ 23], 95.00th=[ 25],
  2956. | 99.00th=[ 40], 99.50th=[ 48], 99.90th=[ 73], 99.95th=[ 84],
  2957. | 99.99th=[ 113]
  2958. bw ( KiB/s): min=477721, max=632118, per=91.30%, avg=554683.67, stdev=77199.58, samples=3
  2959. iops : min=119430, max=158029, avg=138670.67, stdev=19299.77, samples=3
  2960. lat (usec) : 2=0.01%, 10=0.01%, 20=69.42%, 50=30.17%, 100=0.39%
  2961. lat (usec) : 250=0.02%
  2962. cpu : usr=27.01%, sys=72.87%, ctx=2, majf=0, minf=11
  2963. IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  2964. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  2965. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  2966. issued rwts: total=0,262144,0,0 short=0,0,0,0 dropped=0,0,0,0
  2967. latency : target=0, window=0, percentile=100.00%, depth=4
  2968. seqWrite/sync: (groupid=2, jobs=1): err= 0: pid=49551: Sun Mar 22 19:39:48 2020
  2969. write: IOPS=106, BW=428KiB/s (438kB/s)(25.1MiB/60001msec); 0 zone resets
  2970. slat (usec): min=5, max=178, avg=35.09, stdev=10.54
  2971. clat (msec): min=8, max=237, avg=27.98, stdev=11.53
  2972. lat (msec): min=8, max=237, avg=28.01, stdev=11.53
  2973. clat percentiles (msec):
  2974. | 1.00th=[ 21], 5.00th=[ 21], 10.00th=[ 22], 20.00th=[ 24],
  2975. | 30.00th=[ 24], 40.00th=[ 26], 50.00th=[ 26], 60.00th=[ 28],
  2976. | 70.00th=[ 28], 80.00th=[ 29], 90.00th=[ 32], 95.00th=[ 48],
  2977. | 99.00th=[ 77], 99.50th=[ 95], 99.90th=[ 161], 99.95th=[ 178],
  2978. | 99.99th=[ 239]
  2979. bw ( KiB/s): min= 146, max= 428, per=87.76%, avg=374.71, stdev=60.53, samples=119
  2980. iops : min= 36, max= 107, avg=93.37, stdev=15.17, samples=119
  2981. lat (msec) : 10=0.02%, 20=0.05%, 50=95.76%, 100=3.80%, 250=0.37%
  2982. fsync/fdatasync/sync_file_range:
  2983. sync (nsec): min=230, max=14085, avg=601.64, stdev=403.72
  2984. sync percentiles (nsec):
  2985. | 1.00th=[ 302], 5.00th=[ 370], 10.00th=[ 398], 20.00th=[ 410],
  2986. | 30.00th=[ 462], 40.00th=[ 478], 50.00th=[ 684], 60.00th=[ 700],
  2987. | 70.00th=[ 708], 80.00th=[ 724], 90.00th=[ 740], 95.00th=[ 756],
  2988. | 99.00th=[ 772], 99.50th=[ 780], 99.90th=[ 1992], 99.95th=[13632],
  2989. | 99.99th=[14144]
  2990. cpu : usr=0.22%, sys=1.64%, ctx=12835, majf=0, minf=32
  2991. IO depths : 1=0.1%, 2=0.1%, 4=199.9%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  2992. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  2993. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  2994. issued rwts: total=0,6417,0,6415 short=0,0,0,0 dropped=0,0,0,0
  2995. latency : target=0, window=0, percentile=100.00%, depth=4
  2996. randRead: (groupid=3, jobs=1): err= 0: pid=53778: Sun Mar 22 19:39:48 2020
  2997. read: IOPS=230k, BW=899MiB/s (943MB/s)(1024MiB/1139msec)
  2998. slat (nsec): min=2537, max=387284, avg=3462.61, stdev=996.92
  2999. clat (nsec): min=1783, max=421186, avg=13587.39, stdev=2029.49
  3000. lat (usec): min=4, max=425, avg=17.11, stdev= 2.41
  3001. clat percentiles (nsec):
  3002. | 1.00th=[12224], 5.00th=[12480], 10.00th=[12608], 20.00th=[12864],
  3003. | 30.00th=[12992], 40.00th=[13120], 50.00th=[13376], 60.00th=[13504],
  3004. | 70.00th=[13632], 80.00th=[13888], 90.00th=[14400], 95.00th=[15296],
  3005. | 99.00th=[19072], 99.50th=[20608], 99.90th=[30848], 99.95th=[35584],
  3006. | 99.99th=[45312]
  3007. bw ( KiB/s): min=866000, max=895602, per=95.68%, avg=880801.00, stdev=20931.77, samples=2
  3008. iops : min=216500, max=223900, avg=220200.00, stdev=5232.59, samples=2
  3009. lat (usec) : 2=0.01%, 10=0.01%, 20=99.42%, 50=0.57%, 100=0.01%
  3010. lat (usec) : 250=0.01%, 500=0.01%
  3011. cpu : usr=23.46%, sys=76.54%, ctx=4, majf=0, minf=14
  3012. IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  3013. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  3014. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  3015. issued rwts: total=262144,0,0,0 short=0,0,0,0 dropped=0,0,0,0
  3016. latency : target=0, window=0, percentile=100.00%, depth=4
  3017. randWrite: (groupid=4, jobs=1): err= 0: pid=53879: Sun Mar 22 19:39:48 2020
  3018. write: IOPS=89.4k, BW=349MiB/s (366MB/s)(1024MiB/2931msec); 0 zone resets
  3019. slat (usec): min=4, max=8447, avg= 9.47, stdev=30.73
  3020. clat (nsec): min=1623, max=9247.4k, avg=34180.54, stdev=64247.42
  3021. lat (usec): min=7, max=9414, avg=43.73, stdev=79.05
  3022. clat percentiles (usec):
  3023. | 1.00th=[ 21], 5.00th=[ 21], 10.00th=[ 22], 20.00th=[ 22],
  3024. | 30.00th=[ 22], 40.00th=[ 23], 50.00th=[ 23], 60.00th=[ 24],
  3025. | 70.00th=[ 25], 80.00th=[ 28], 90.00th=[ 37], 95.00th=[ 95],
  3026. | 99.00th=[ 243], 99.50th=[ 338], 99.90th=[ 578], 99.95th=[ 676],
  3027. | 99.99th=[ 1450]
  3028. bw ( KiB/s): min=204680, max=360661, per=77.75%, avg=278158.80, stdev=70393.25, samples=5
  3029. iops : min=51170, max=90165, avg=69539.40, stdev=17598.34, samples=5
  3030. lat (usec) : 2=0.01%, 10=0.01%, 20=0.02%, 50=92.37%, 100=3.04%
  3031. lat (usec) : 250=3.65%, 500=0.75%, 750=0.14%, 1000=0.02%
  3032. lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%
  3033. cpu : usr=17.41%, sys=78.16%, ctx=1872, majf=0, minf=39
  3034. IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  3035. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  3036. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  3037. issued rwts: total=0,262144,0,0 short=0,0,0,0 dropped=0,0,0,0
  3038. latency : target=0, window=0, percentile=100.00%, depth=4
  3039. randWrite/sync: (groupid=5, jobs=1): err= 0: pid=10076: Sun Mar 22 19:39:48 2020
  3040. write: IOPS=103, BW=412KiB/s (422kB/s)(24.1MiB/60004msec); 0 zone resets
  3041. slat (usec): min=11, max=136, avg=44.37, stdev=12.98
  3042. clat (msec): min=11, max=744, avg=28.88, stdev=16.31
  3043. lat (msec): min=11, max=745, avg=28.93, stdev=16.31
  3044. clat percentiles (msec):
  3045. | 1.00th=[ 20], 5.00th=[ 22], 10.00th=[ 23], 20.00th=[ 24],
  3046. | 30.00th=[ 26], 40.00th=[ 26], 50.00th=[ 26], 60.00th=[ 27],
  3047. | 70.00th=[ 28], 80.00th=[ 29], 90.00th=[ 36], 95.00th=[ 50],
  3048. | 99.00th=[ 79], 99.50th=[ 133], 99.90th=[ 194], 99.95th=[ 203],
  3049. | 99.99th=[ 743]
  3050. bw ( KiB/s): min= 122, max= 350, per=73.55%, avg=303.04, stdev=53.44, samples=118
  3051. iops : min= 30, max= 87, avg=75.48, stdev=13.36, samples=118
  3052. lat (msec) : 20=1.41%, 50=93.82%, 100=4.11%, 250=0.63%, 500=0.02%
  3053. lat (msec) : 750=0.02%
  3054. fsync/fdatasync/sync_file_range:
  3055. sync (nsec): min=228, max=13935, avg=631.16, stdev=571.28
  3056. sync percentiles (nsec):
  3057. | 1.00th=[ 286], 5.00th=[ 354], 10.00th=[ 370], 20.00th=[ 398],
  3058. | 30.00th=[ 450], 40.00th=[ 660], 50.00th=[ 692], 60.00th=[ 700],
  3059. | 70.00th=[ 708], 80.00th=[ 724], 90.00th=[ 764], 95.00th=[ 860],
  3060. | 99.00th=[ 1096], 99.50th=[ 1208], 99.90th=[13376], 99.95th=[13504],
  3061. | 99.99th=[13888]
  3062. cpu : usr=0.24%, sys=1.89%, ctx=12624, majf=0, minf=37
  3063. IO depths : 1=0.1%, 2=0.1%, 4=199.9%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  3064. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  3065. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  3066. issued rwts: total=0,6182,0,6180 short=0,0,0,0 dropped=0,0,0,0
  3067. latency : target=0, window=0, percentile=100.00%, depth=4
  3068. randReadWrite: (groupid=6, jobs=1): err= 0: pid=18552: Sun Mar 22 19:39:48 2020
  3069. read: IOPS=52.8k, BW=206MiB/s (216MB/s)(509MiB/2471msec)
  3070. slat (usec): min=2, max=6930, avg= 6.20, stdev=30.58
  3071. clat (usec): min=9, max=6968, avg=29.03, stdev=60.91
  3072. lat (usec): min=12, max=7075, avg=35.30, stdev=73.35
  3073. clat percentiles (usec):
  3074. | 1.00th=[ 12], 5.00th=[ 13], 10.00th=[ 15], 20.00th=[ 16],
  3075. | 30.00th=[ 17], 40.00th=[ 19], 50.00th=[ 20], 60.00th=[ 21],
  3076. | 70.00th=[ 22], 80.00th=[ 25], 90.00th=[ 38], 95.00th=[ 87],
  3077. | 99.00th=[ 208], 99.50th=[ 260], 99.90th=[ 523], 99.95th=[ 676],
  3078. | 99.99th=[ 1991]
  3079. bw ( KiB/s): min=77343, max=213276, per=75.16%, avg=158621.00, stdev=61779.81, samples=4
  3080. iops : min=19335, max=53319, avg=39654.75, stdev=15445.23, samples=4
  3081. write: IOPS=53.3k, BW=208MiB/s (218MB/s)(515MiB/2471msec); 0 zone resets
  3082. slat (usec): min=4, max=4191, avg= 9.84, stdev=26.58
  3083. clat (nsec): min=1741, max=7043.5k, avg=28792.87, stdev=55864.24
  3084. lat (usec): min=8, max=7146, avg=38.72, stdev=69.14
  3085. clat percentiles (usec):
  3086. | 1.00th=[ 13], 5.00th=[ 14], 10.00th=[ 15], 20.00th=[ 16],
  3087. | 30.00th=[ 18], 40.00th=[ 19], 50.00th=[ 20], 60.00th=[ 21],
  3088. | 70.00th=[ 22], 80.00th=[ 25], 90.00th=[ 37], 95.00th=[ 87],
  3089. | 99.00th=[ 206], 99.50th=[ 262], 99.90th=[ 494], 99.95th=[ 676],
  3090. | 99.99th=[ 1532]
  3091. bw ( KiB/s): min=77075, max=217787, per=75.45%, avg=160943.50, stdev=63465.31, samples=4
  3092. iops : min=19268, max=54446, avg=40235.50, stdev=15866.43, samples=4
  3093. lat (usec) : 2=0.01%, 10=0.01%, 20=59.14%, 50=32.18%, 100=4.86%
  3094. lat (usec) : 250=3.27%, 500=0.45%, 750=0.07%, 1000=0.01%
  3095. lat (msec) : 2=0.02%, 4=0.01%, 10=0.01%
  3096. cpu : usr=16.23%, sys=81.34%, ctx=994, majf=0, minf=40
  3097. IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  3098. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  3099. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  3100. issued rwts: total=130367,131777,0,0 short=0,0,0,0 dropped=0,0,0,0
  3101. latency : target=0, window=0, percentile=100.00%, depth=4
  3102. randReadWrite/sync: (groupid=7, jobs=1): err= 0: pid=29054: Sun Mar 22 19:39:48 2020
  3103. read: IOPS=102, BW=410KiB/s (419kB/s)(24.0MiB/60002msec)
  3104. slat (usec): min=3, max=139, avg=20.49, stdev=14.30
  3105. clat (usec): min=20, max=594903, avg=9811.25, stdev=14100.80
  3106. lat (usec): min=24, max=594908, avg=9832.08, stdev=14100.61
  3107. clat percentiles (usec):
  3108. | 1.00th=[ 27], 5.00th=[ 34], 10.00th=[ 41], 20.00th=[ 78],
  3109. | 30.00th=[ 4752], 40.00th=[ 7308], 50.00th=[ 9110], 60.00th=[ 10421],
  3110. | 70.00th=[ 11207], 80.00th=[ 15795], 90.00th=[ 20055], 95.00th=[ 22414],
  3111. | 99.00th=[ 42206], 99.50th=[ 53740], 99.90th=[130548], 99.95th=[132645],
  3112. | 99.99th=[591397]
  3113. bw ( KiB/s): min= 24, max= 680, per=100.00%, avg=409.39, stdev=115.39, samples=120
  3114. iops : min= 6, max= 170, avg=102.29, stdev=28.85, samples=120
  3115. write: IOPS=102, BW=411KiB/s (421kB/s)(24.1MiB/60002msec); 0 zone resets
  3116. slat (usec): min=8, max=193, avg=33.07, stdev=15.04
  3117. clat (msec): min=3, max=602, avg=19.43, stdev=15.12
  3118. lat (msec): min=3, max=602, avg=19.47, stdev=15.12
  3119. clat percentiles (msec):
  3120. | 1.00th=[ 5], 5.00th=[ 6], 10.00th=[ 9], 20.00th=[ 11],
  3121. | 30.00th=[ 13], 40.00th=[ 16], 50.00th=[ 17], 60.00th=[ 21],
  3122. | 70.00th=[ 22], 80.00th=[ 26], 90.00th=[ 32], 95.00th=[ 39],
  3123. | 99.00th=[ 66], 99.50th=[ 94], 99.90th=[ 150], 99.95th=[ 163],
  3124. | 99.99th=[ 600]
  3125. bw ( KiB/s): min= 152, max= 528, per=100.00%, avg=414.03, stdev=77.02, samples=119
  3126. iops : min= 38, max= 132, avg=103.44, stdev=19.26, samples=119
  3127. lat (usec) : 50=6.97%, 100=5.36%, 250=0.22%
  3128. lat (msec) : 4=0.97%, 10=23.84%, 20=37.70%, 50=23.57%, 100=1.06%
  3129. lat (msec) : 250=0.28%, 750=0.02%
  3130. fsync/fdatasync/sync_file_range:
  3131. sync (nsec): min=23, max=15829, avg=348.62, stdev=493.56
  3132. sync percentiles (nsec):
  3133. | 1.00th=[ 27], 5.00th=[ 31], 10.00th=[ 46], 20.00th=[ 86],
  3134. | 30.00th=[ 105], 40.00th=[ 137], 50.00th=[ 253], 60.00th=[ 406],
  3135. | 70.00th=[ 470], 80.00th=[ 700], 90.00th=[ 716], 95.00th=[ 732],
  3136. | 99.00th=[ 772], 99.50th=[ 780], 99.90th=[ 1960], 99.95th=[13760],
  3137. | 99.99th=[15808]
  3138. cpu : usr=0.26%, sys=1.78%, ctx=12418, majf=0, minf=16
  3139. IO depths : 1=0.1%, 2=0.1%, 4=199.9%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  3140. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  3141. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  3142. issued rwts: total=6144,6162,0,12301 short=0,0,0,0 dropped=0,0,0,0
  3143. latency : target=0, window=0, percentile=100.00%, depth=4
  3144.  
  3145. Run status group 0 (all jobs):
  3146. READ: bw=1086MiB/s (1139MB/s), 1086MiB/s-1086MiB/s (1139MB/s-1139MB/s), io=1024MiB (1074MB), run=943-943msec
  3147.  
  3148. Run status group 1 (all jobs):
  3149. WRITE: bw=593MiB/s (622MB/s), 593MiB/s-593MiB/s (622MB/s-622MB/s), io=1024MiB (1074MB), run=1726-1726msec
  3150.  
  3151. Run status group 2 (all jobs):
  3152. WRITE: bw=428KiB/s (438kB/s), 428KiB/s-428KiB/s (438kB/s-438kB/s), io=25.1MiB (26.3MB), run=60001-60001msec
  3153.  
  3154. Run status group 3 (all jobs):
  3155. READ: bw=899MiB/s (943MB/s), 899MiB/s-899MiB/s (943MB/s-943MB/s), io=1024MiB (1074MB), run=1139-1139msec
  3156.  
  3157. Run status group 4 (all jobs):
  3158. WRITE: bw=349MiB/s (366MB/s), 349MiB/s-349MiB/s (366MB/s-366MB/s), io=1024MiB (1074MB), run=2931-2931msec
  3159.  
  3160. Run status group 5 (all jobs):
  3161. WRITE: bw=412KiB/s (422kB/s), 412KiB/s-412KiB/s (422kB/s-422kB/s), io=24.1MiB (25.3MB), run=60004-60004msec
  3162.  
  3163. Run status group 6 (all jobs):
  3164. READ: bw=206MiB/s (216MB/s), 206MiB/s-206MiB/s (216MB/s-216MB/s), io=509MiB (534MB), run=2471-2471msec
  3165. WRITE: bw=208MiB/s (218MB/s), 208MiB/s-208MiB/s (218MB/s-218MB/s), io=515MiB (540MB), run=2471-2471msec
  3166.  
  3167. Run status group 7 (all jobs):
  3168. READ: bw=410KiB/s (419kB/s), 410KiB/s-410KiB/s (419kB/s-419kB/s), io=24.0MiB (25.2MB), run=60002-60002msec
  3169. WRITE: bw=411KiB/s (421kB/s), 411KiB/s-411KiB/s (421kB/s-421kB/s), io=24.1MiB (25.2MB), run=60002-60002msec
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement
Public Pastes
Advertisement
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!