API tools faq
paste
Advertisement
galileo_pkm

Raid O_DIRECT

galileo_pkm
Oct 12th, 2015
111
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 116.28 KB | None | 0 0
raw download clone embed print report
  1. Hard drives: 7x WDC WD30EFRX
  2. Controler: Dell PERC H200
  3.  
  4. [root@storage ~]# mdadm --version
  5. mdadm - v3.3.2 - 21st August 2014
  6.  
  7. [root@storage ~]# mdadm --detail /dev/md2
  8. /dev/md2:
  9. Version : 1.2
  10. Creation Time : Thu Oct 8 04:08:16 2015
  11. Raid Level : raid10
  12. Array Size : 44011520 (41.97 GiB 45.07 GB)
  13. Used Dev Size : 12574720 (11.99 GiB 12.88 GB)
  14. Raid Devices : 7
  15. Total Devices : 7
  16. Persistence : Superblock is persistent
  17.  
  18. Update Time : Mon Oct 12 03:29:20 2015
  19. State : clean
  20. Active Devices : 7
  21. Working Devices : 7
  22. Failed Devices : 0
  23. Spare Devices : 0
  24.  
  25. Layout : far=2
  26. Chunk Size : 128K
  27.  
  28. Name : localhost:2
  29. UUID : 9ccf3697:9b4d1bff:88b49ca9:48a03ed3
  30. Events : 24
  31.  
  32. Number Major Minor RaidDevice State
  33. 0 8 4 0 active sync /dev/sda4
  34. 1 8 36 1 active sync /dev/sdc4
  35. 2 8 20 2 active sync /dev/sdb4
  36. 3 8 52 3 active sync /dev/sdd4
  37. 4 8 84 4 active sync /dev/sdf4
  38. 5 8 68 5 active sync /dev/sde4
  39. 6 8 100 6 active sync /dev/sdg4
  40.  
  41. --------------------------------------------
  42. [root@storage ~]# mdadm --detail /dev/md3
  43. /dev/md3:
  44. Version : 1.2
  45. Creation Time : Thu Oct 8 04:09:20 2015
  46. Raid Level : raid6
  47. Array Size : 14579882880 (13904.46 GiB 14929.80 GB)
  48. Used Dev Size : 2915976576 (2780.89 GiB 2985.96 GB)
  49. Raid Devices : 7
  50. Total Devices : 7
  51. Persistence : Superblock is persistent
  52.  
  53. Update Time : Mon Oct 12 03:26:03 2015
  54. State : clean
  55. Active Devices : 7
  56. Working Devices : 7
  57. Failed Devices : 0
  58. Spare Devices : 0
  59.  
  60. Layout : left-symmetric
  61. Chunk Size : 128K
  62.  
  63. Name : localhost:3
  64. UUID : 567a36e5:8c591b3b:c789c47d:5d4bf67c
  65. Events : 7725
  66.  
  67. Number Major Minor RaidDevice State
  68. 0 8 5 0 active sync /dev/sda5
  69. 1 8 37 1 active sync /dev/sdc5
  70. 2 8 21 2 active sync /dev/sdb5
  71. 3 8 53 3 active sync /dev/sdd5
  72. 4 8 85 4 active sync /dev/sdf5
  73. 5 8 69 5 active sync /dev/sde5
  74. 6 8 101 6 active sync /dev/sdg5
  75.  
  76. [root@storage ~]# blockdev --getra /dev/sda /dev/sdg /dev/md2 /dev/md3
  77. 8192
  78. 8192
  79. 8192
  80. 8192
  81.  
  82.  
  83. =Summary of results
  84.  
  85. == Hard disk, Sequential read, O_DIRECT
  86. ----
  87. bs: 4k, bw: 80833KB/s
  88. bs: 32k, bw: 152000KB/s
  89. bs: 128k, bw: 152809KB/s
  90. ----
  91.  
  92. == Hard disk, Sequential write, O_DIRECT
  93. ----
  94. bs: 16k, bw: 144392KB/s
  95. bs: 32k, bw: 145352KB/s
  96. bs: 128k, bw: 145433KB/s
  97. ----
  98.  
  99. == Raid 10 f2, chunk: 128k, Sequential read, O_DIRECT
  100. ----
  101. bs: 4k, bw: 78136KB/s, per disk bw: 11MB/s, disk avgrq-sz: 8
  102. bs: 32k, bw: 289847KB/s, per disk bw: 41MB/s, disk avgrq-sz: 64
  103. bs: 128k, bw: 409674KB/s, per disk bw: 57MB/s, disk avgrq-sz: 256
  104. bs: 256k, bw: 739344KB/s, per disk bw: 104MB/s, disk avgrq-sz: 256
  105. bs: 896k, bw: 981517KB/s, per disk bw: 133MB/s, disk avgrq-sz: 256
  106. ----
  107.  
  108. == Raid 6, chunk: 128k, Sequential read, O_DIRECT
  109. ----
  110. bs: 4k, bw: 46376KB/s, per disk bw: 6MB/s, disk avgrq-sz: 8
  111. bs: 32k, bw: 155531KB/s, per disk bw: 22MB/s, disk avgrq-sz: 64
  112. bs: 128k, bw: 182024KB/s, per disk bw: 26MB/s, disk avgrq-sz: 256
  113. bs: 256k, bw: 248591KB/s, per disk bw: 34MB/s, disk avgrq-sz: 256
  114. bs: 384k, bw: 299771KB/s, per disk bw: 40MB/s, disk avgrq-sz: 256
  115. bs: 512k, bw: 315374KB/s, per disk bw: 44MB/s, disk avgrq-sz: 256
  116. bs: 640k, bw: 296350KB/s, per disk bw: 44MB/s, disk avgrq-sz: 256
  117. bs: 1280k, bw: 543655KB/s, per disk bw: 75MB/s, disk avgrq-sz: 426
  118. bs: 2560k, bw: 618092KB/s, per disk bw: 83MB/s, disk avgrq-sz: 638
  119. ----
  120.  
  121. == Raid 6, chunk: 128k, Sequential read, Buffered
  122. ----
  123. bs: 2560k, bw: 690546KB/s, per disk bw: 96MB/s, disk avgrq-sz: 512
  124. ----
  125.  
  126. == Raid 6, chunk: 128k, Sequential write, O_DIRECT, stripe_cache_size: 32768
  127. ----
  128. bs: 640K, bw: 382778KB/s, per disk bw: 75MB/s, disk avgrq-sz: 256
  129. bs: 1280K, bw: 405909KB/s, per disk bw: 82MB/s, disk avgrq-sz: 512
  130. ----
  131.  
  132. == Raid 6, chunk: 128k, Sequential write, Buffered, stripe_cache_size: 32768
  133. ----
  134. bs: 1280k, bw: 730527KB/s, per disk bw: 135M/s, disk avgrq-sz: 1024
  135.  
  136.  
  137.  
  138.  
  139.  
  140. /dev/sda4 4k sequential read, O_DIRECT
  141. ===============================================================================================================================
  142. [root@storage ~]# fio --name=write --ioengine=libaio --bs=4k --direct=1 --rw=read --iodepth=1 --size=4G --filename=/dev/sda4
  143. write: (g=0): rw=read, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=1
  144. fio-2.2.8
  145. Starting 1 process
  146. Jobs: 1 (f=1): [R(1)] [100.0% done] [81200KB/0KB/0KB /s] [20.3K/0/0 iops] [eta 00m:00s]
  147. write: (groupid=0, jobs=1): err= 0: pid=2905: Mon Oct 12 03:35:58 2015
  148. read : io=4096.0MB, bw=80834KB/s, iops=20208, runt= 51888msec
  149. slat (usec): min=3, max=44, avg= 3.85, stdev= 0.49
  150. clat (usec): min=29, max=22327, avg=44.09, stdev=36.07
  151. lat (usec): min=43, max=22360, avg=48.30, stdev=36.10
  152. clat percentiles (usec):
  153. | 1.00th=[ 39], 5.00th=[ 40], 10.00th=[ 40], 20.00th=[ 41],
  154. | 30.00th=[ 41], 40.00th=[ 42], 50.00th=[ 43], 60.00th=[ 44],
  155. | 70.00th=[ 44], 80.00th=[ 45], 90.00th=[ 46], 95.00th=[ 47],
  156. | 99.00th=[ 68], 99.50th=[ 70], 99.90th=[ 290], 99.95th=[ 298],
  157. | 99.99th=[ 310]
  158. bw (KB /s): min=64904, max=81456, per=100.00%, avg=80837.28, stdev=2192.44
  159. lat (usec) : 50=97.06%, 100=2.72%, 250=0.01%, 500=0.21%, 750=0.01%
  160. lat (msec) : 10=0.01%, 20=0.01%, 50=0.01%
  161. cpu : usr=4.62%, sys=13.86%, ctx=1048576, majf=0, minf=30
  162. IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  163. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  164. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  165. issued : total=r=1048576/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
  166. latency : target=0, window=0, percentile=100.00%, depth=1
  167.  
  168. Run status group 0 (all jobs):
  169. READ: io=4096.0MB, aggrb=80833KB/s, minb=80833KB/s, maxb=80833KB/s, mint=51888msec, maxt=51888msec
  170.  
  171. Disk stats (read/write):
  172. sda: ios=1044390/0, merge=0/0, ticks=43714/0, in_queue=43534, util=84.13%
  173.  
  174.  
  175. iostat
  176. ------
  177. avg-cpu: %user %nice %system %iowait %steal %idle
  178. 0.68 0.00 1.93 0.00 0.00 97.40
  179.  
  180. Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
  181. sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  182. sda 0.00 0.00 19813.20 0.00 77.40 0.00 8.00 0.84 0.04 0.04 0.00 0.04 84.36
  183. sdd 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  184. sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  185. sde 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  186. sdf 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  187. sdg 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  188. md0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  189. md2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  190. dm-0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  191. dm-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  192. md1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  193. md3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  194. dm-2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  195. dm-3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  196. dm-4 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  197. dm-5 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  198.  
  199.  
  200.  
  201. /dev/sda4 32k sequential read, O_DIRECT
  202. ===============================================================================================================================
  203. [root@storage ~]# fio --name=write --ioengine=libaio --bs=32k --direct=1 --rw=read --iodepth=1 --size=4G --filename=/dev/sda4
  204. write: (g=0): rw=read, bs=32K-32K/32K-32K/32K-32K, ioengine=libaio, iodepth=1
  205. fio-2.2.8
  206. Starting 1 process
  207. Jobs: 1 (f=1): [R(1)] [100.0% done] [143.2MB/0KB/0KB /s] [4582/0/0 iops] [eta 00m:00s]
  208. write: (groupid=0, jobs=1): err= 0: pid=2943: Mon Oct 12 03:42:33 2015
  209. read : io=4096.0MB, bw=152001KB/s, iops=4750, runt= 27594msec
  210. slat (usec): min=4, max=54, avg= 5.59, stdev= 3.24
  211. clat (usec): min=95, max=26201, avg=203.28, stdev=243.94
  212. lat (usec): min=110, max=26210, avg=209.25, stdev=243.91
  213. clat percentiles (usec):
  214. | 1.00th=[ 121], 5.00th=[ 161], 10.00th=[ 167], 20.00th=[ 167],
  215. | 30.00th=[ 169], 40.00th=[ 175], 50.00th=[ 177], 60.00th=[ 177],
  216. | 70.00th=[ 189], 80.00th=[ 201], 90.00th=[ 203], 95.00th=[ 205],
  217. | 99.00th=[ 1448], 99.50th=[ 1480], 99.90th=[ 1496], 99.95th=[ 1512],
  218. | 99.99th=[11328]
  219. bw (KB /s): min=137920, max=161600, per=100.00%, avg=152053.16, stdev=5767.85
  220. lat (usec) : 100=0.01%, 250=98.19%, 500=0.07%, 750=0.03%, 1000=0.04%
  221. lat (msec) : 2=1.63%, 4=0.01%, 10=0.01%, 20=0.02%, 50=0.01%
  222. cpu : usr=1.50%, sys=3.91%, ctx=131074, majf=0, minf=37
  223. IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  224. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  225. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  226. issued : total=r=131072/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
  227. latency : target=0, window=0, percentile=100.00%, depth=1
  228.  
  229. Run status group 0 (all jobs):
  230. READ: io=4096.0MB, aggrb=152000KB/s, minb=152000KB/s, maxb=152000KB/s, mint=27594msec, maxt=27594msec
  231.  
  232. Disk stats (read/write):
  233. sda: ios=130262/3, merge=0/0, ticks=28212/85, in_queue=28263, util=95.36%
  234.  
  235. iostat
  236. ------
  237. avg-cpu: %user %nice %system %iowait %steal %idle
  238. 0.17 0.00 0.57 0.00 0.00 99.26
  239.  
  240. Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
  241. sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  242. sda 0.00 0.00 4773.80 0.00 149.18 0.00 64.00 0.95 0.20 0.20 0.00 0.20 95.08
  243. sdd 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  244. sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  245. sde 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  246. sdf 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  247. sdg 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  248. md0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  249. md2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  250. dm-0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  251. dm-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  252. md1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  253. md3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  254. dm-2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  255. dm-3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  256. dm-4 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  257. dm-5 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  258.  
  259.  
  260. /dev/sda4 128k sequential read, O_DIRECT
  261. ===============================================================================================================================
  262. [root@storage ~]# fio --name=write --ioengine=libaio --bs=128k --direct=1 --rw=read --iodepth=1 --size=4G --filename=/dev/sda4
  263. write: (g=0): rw=read, bs=128K-128K/128K-128K/128K-128K, ioengine=libaio, iodepth=1
  264. fio-2.2.8
  265. Starting 1 process
  266. Jobs: 1 (f=1): [R(1)] [100.0% done] [143.4MB/0KB/0KB /s] [1147/0/0 iops] [eta 00m:00s]
  267. write: (groupid=0, jobs=1): err= 0: pid=2955: Mon Oct 12 03:45:19 2015
  268. read : io=4096.0MB, bw=152809KB/s, iops=1193, runt= 27448msec
  269. slat (usec): min=10, max=82, avg=31.67, stdev=16.57
  270. clat (usec): min=301, max=17821, avg=799.98, stdev=418.66
  271. lat (usec): min=311, max=17902, avg=832.67, stdev=414.96
  272. clat percentiles (usec):
  273. | 1.00th=[ 540], 5.00th=[ 540], 10.00th=[ 612], 20.00th=[ 636],
  274. | 30.00th=[ 644], 40.00th=[ 652], 50.00th=[ 676], 60.00th=[ 756],
  275. | 70.00th=[ 820], 80.00th=[ 836], 90.00th=[ 876], 95.00th=[ 1896],
  276. | 99.00th=[ 2040], 99.50th=[ 2064], 99.90th=[ 3344], 99.95th=[ 5152],
  277. | 99.99th=[12992]
  278. bw (KB /s): min=138219, max=161792, per=100.00%, avg=153028.96, stdev=5135.13
  279. lat (usec) : 500=0.42%, 750=58.97%, 1000=33.79%
  280. lat (msec) : 2=4.37%, 4=2.39%, 10=0.02%, 20=0.04%
  281. cpu : usr=1.04%, sys=5.45%, ctx=32768, majf=0, minf=61
  282. IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  283. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  284. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  285. issued : total=r=32768/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
  286. latency : target=0, window=0, percentile=100.00%, depth=1
  287.  
  288. Run status group 0 (all jobs):
  289. READ: io=4096.0MB, aggrb=152809KB/s, minb=152809KB/s, maxb=152809KB/s, mint=27448msec, maxt=27448msec
  290.  
  291. Disk stats (read/write):
  292. sda: ios=32735/0, merge=0/0, ticks=25794/0, in_queue=25752, util=93.66%
  293.  
  294.  
  295. iostat
  296. ------
  297. avg-cpu: %user %nice %system %iowait %steal %idle
  298. 0.17 0.00 0.87 0.00 0.00 98.96
  299.  
  300. Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
  301. sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  302. sda 0.00 0.00 1113.40 0.00 139.18 0.00 256.00 0.87 0.79 0.79 0.00 0.78 87.28
  303. sdd 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  304. sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  305. sde 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  306. sdf 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  307. sdg 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  308. md0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  309. md2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  310. dm-0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  311. dm-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  312. md1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  313. md3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  314. dm-2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  315. dm-3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  316. dm-4 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  317. dm-5 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  318.  
  319.  
  320. /dev/sdg5 32k sequential write, O_DIRECT
  321. ===============================================================================================================================
  322. [root@storage ~]# fio --name=write --ioengine=libaio --bs=32k --direct=1 --rw=write --iodepth=1 --size=4G --filename=/dev/sdg5
  323. write: (g=0): rw=write, bs=32K-32K/32K-32K/32K-32K, ioengine=libaio, iodepth=1
  324. fio-2.2.8
  325. Starting 1 process
  326. Jobs: 1 (f=1): [W(1)] [96.7% done] [0KB/132.7MB/0KB /s] [0/4245/0 iops] [eta 00m:01s]
  327. write: (groupid=0, jobs=1): err= 0: pid=3685: Mon Oct 12 05:19:48 2015
  328. write: io=4096.0MB, bw=145353KB/s, iops=4542, runt= 28856msec
  329. slat (usec): min=5, max=54, avg= 6.68, stdev= 3.51
  330. clat (usec): min=111, max=12582, avg=211.82, stdev=198.24
  331. lat (usec): min=123, max=12588, avg=218.87, stdev=198.12
  332. clat percentiles (usec):
  333. | 1.00th=[ 122], 5.00th=[ 153], 10.00th=[ 167], 20.00th=[ 177],
  334. | 30.00th=[ 179], 40.00th=[ 179], 50.00th=[ 195], 60.00th=[ 197],
  335. | 70.00th=[ 199], 80.00th=[ 209], 90.00th=[ 211], 95.00th=[ 211],
  336. | 99.00th=[ 1480], 99.50th=[ 1544], 99.90th=[ 1752], 99.95th=[ 1800],
  337. | 99.99th=[ 2928]
  338. bw (KB /s): min=129280, max=153792, per=99.97%, avg=145314.68, stdev=7586.59
  339. lat (usec) : 250=98.15%, 500=0.04%, 750=0.02%, 1000=0.01%
  340. lat (msec) : 2=1.77%, 4=0.01%, 10=0.01%, 20=0.01%
  341. cpu : usr=1.93%, sys=3.75%, ctx=131073, majf=0, minf=26
  342. IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  343. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  344. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  345. issued : total=r=0/w=131072/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
  346. latency : target=0, window=0, percentile=100.00%, depth=1
  347.  
  348. Run status group 0 (all jobs):
  349. WRITE: io=4096.0MB, aggrb=145352KB/s, minb=145352KB/s, maxb=145352KB/s, mint=28856msec, maxt=28856msec
  350.  
  351. Disk stats (read/write):
  352. sdg: ios=0/130118, merge=0/0, ticks=0/27234, in_queue=27207, util=94.68%
  353.  
  354. iostat
  355. ------
  356. avg-cpu: %user %nice %system %iowait %steal %idle
  357. 0.30 0.00 0.44 0.00 0.00 99.26
  358.  
  359. Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
  360. sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  361. sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  362. sdd 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  363. sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  364. sde 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  365. sdf 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  366. sdg 0.00 0.00 0.00 4474.00 0.00 139.81 64.00 0.95 0.21 0.00 0.21 0.21 95.06
  367. md0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  368. md2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  369. dm-0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  370. dm-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  371. md1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  372.  
  373.  
  374. /dev/sdg5 16k sequential write, O_DIRECT
  375. ===============================================================================================================================
  376. [root@storage ~]# fio --name=write --ioengine=libaio --bs=16k --direct=1 --rw=write --iodepth=1 --size=4G --filename=/dev/sdg5
  377. write: (g=0): rw=write, bs=16K-16K/16K-16K/16K-16K, ioengine=libaio, iodepth=1
  378. fio-2.2.8
  379. Starting 1 process
  380. Jobs: 1 (f=1): [W(1)] [100.0% done] [0KB/148.7MB/0KB /s] [0/9513/0 iops] [eta 00m:00s]
  381. write: (groupid=0, jobs=1): err= 0: pid=3703: Mon Oct 12 05:21:36 2015
  382. write: io=4096.0MB, bw=144392KB/s, iops=9024, runt= 29048msec
  383. slat (usec): min=4, max=51, avg= 4.81, stdev= 2.28
  384. clat (usec): min=71, max=57470, avg=104.39, stdev=208.24
  385. lat (usec): min=77, max=57496, avg=109.57, stdev=208.30
  386. clat percentiles (usec):
  387. | 1.00th=[ 81], 5.00th=[ 83], 10.00th=[ 84], 20.00th=[ 85],
  388. | 30.00th=[ 86], 40.00th=[ 86], 50.00th=[ 87], 60.00th=[ 88],
  389. | 70.00th=[ 91], 80.00th=[ 97], 90.00th=[ 111], 95.00th=[ 113],
  390. | 99.00th=[ 145], 99.50th=[ 1352], 99.90th=[ 1624], 99.95th=[ 1688],
  391. | 99.99th=[ 1976]
  392. bw (KB /s): min=106496, max=153760, per=100.00%, avg=144423.53, stdev=9106.88
  393. lat (usec) : 100=80.59%, 250=18.49%, 500=0.01%, 750=0.01%, 1000=0.01%
  394. lat (msec) : 2=0.89%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
  395. lat (msec) : 100=0.01%
  396. cpu : usr=3.13%, sys=6.21%, ctx=262151, majf=0, minf=27
  397. IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  398. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  399. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  400. issued : total=r=0/w=262144/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
  401. latency : target=0, window=0, percentile=100.00%, depth=1
  402.  
  403. Run status group 0 (all jobs):
  404. WRITE: io=4096.0MB, aggrb=144392KB/s, minb=144392KB/s, maxb=144392KB/s, mint=29048msec, maxt=29048msec
  405.  
  406. Disk stats (read/write):
  407. sdg: ios=0/260816, merge=0/2, ticks=0/26745, in_queue=26692, util=91.56%
  408.  
  409.  
  410. iostat
  411. ------
  412. avg-cpu: %user %nice %system %iowait %steal %idle
  413. 0.44 0.00 0.87 0.00 0.00 98.69
  414.  
  415. Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
  416. sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  417. sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  418. sdd 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  419. sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  420. sde 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  421. sdf 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  422. sdg 0.00 0.00 0.00 9193.20 0.00 143.64 32.00 0.92 0.10 0.00 0.10 0.10 91.64
  423. md0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  424. md2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  425. dm-0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  426. dm-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  427. md1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  428.  
  429.  
  430. /dev/sdg5 128k sequential write, O_DIRECT
  431. ===============================================================================================================================
  432. [root@storage ~]# fio --name=write --ioengine=libaio --bs=128k --direct=1 --rw=write --iodepth=1 --size=4G --filename=/dev/sdg5
  433. write: (g=0): rw=write, bs=128K-128K/128K-128K/128K-128K, ioengine=libaio, iodepth=1
  434. fio-2.2.8
  435. Starting 1 process
  436. Jobs: 1 (f=1): [W(1)] [100.0% done] [0KB/130.7MB/0KB /s] [0/1045/0 iops] [eta 00m:00s]
  437. write: (groupid=0, jobs=1): err= 0: pid=3674: Mon Oct 12 05:18:17 2015
  438. write: io=4096.0MB, bw=145434KB/s, iops=1136, runt= 28840msec
  439. slat (usec): min=9, max=86, avg=38.51, stdev=18.41
  440. clat (usec): min=342, max=16258, avg=835.12, stdev=403.56
  441. lat (usec): min=354, max=16314, avg=874.72, stdev=399.64
  442. clat percentiles (usec):
  443. | 1.00th=[ 446], 5.00th=[ 556], 10.00th=[ 612], 20.00th=[ 644],
  444. | 30.00th=[ 652], 40.00th=[ 684], 50.00th=[ 748], 60.00th=[ 804],
  445. | 70.00th=[ 860], 80.00th=[ 892], 90.00th=[ 932], 95.00th=[ 1976],
  446. | 99.00th=[ 2160], 99.50th=[ 2192], 99.90th=[ 2480], 99.95th=[ 3408],
  447. | 99.99th=[11968]
  448. bw (KB /s): min=129536, max=153856, per=100.00%, avg=145539.74, stdev=7445.15
  449. lat (usec) : 500=1.56%, 750=48.71%, 1000=42.54%
  450. lat (msec) : 2=2.67%, 4=4.48%, 10=0.01%, 20=0.02%
  451. cpu : usr=2.31%, sys=4.84%, ctx=32768, majf=0, minf=26
  452. IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  453. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  454. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  455. issued : total=r=0/w=32768/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
  456. latency : target=0, window=0, percentile=100.00%, depth=1
  457.  
  458. Run status group 0 (all jobs):
  459. WRITE: io=4096.0MB, aggrb=145433KB/s, minb=145433KB/s, maxb=145433KB/s, mint=28840msec, maxt=28840msec
  460.  
  461. Disk stats (read/write):
  462. sdg: ios=0/32568, merge=0/0, ticks=0/26726, in_queue=26682, util=92.85%
  463.  
  464.  
  465. iostat
  466. ------
  467. avg-cpu: %user %nice %system %iowait %steal %idle
  468. 0.30 0.00 0.70 0.00 0.00 98.99
  469.  
  470. Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
  471. sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  472. sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  473. sdd 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  474. sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  475. sde 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  476. sdf 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  477. sdg 0.00 0.00 0.00 1147.00 0.00 143.38 256.00 0.94 0.82 0.00 0.82 0.82 93.86
  478. md0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  479. md2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  480. dm-0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  481. dm-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  482. md1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  483.  
  484.  
  485. /dev/md2 4k sequential read, O_DIRECT
  486. ===============================================================================================================================
  487. [root@storage ~]# fio --name=write --ioengine=libaio --bs=4k --direct=1 --rw=read --iodepth=1 --size=4G --filename=/dev/md2
  488. write: (g=0): rw=read, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=1
  489. fio-2.2.8
  490. Starting 1 process
  491. Jobs: 1 (f=1): [R(1)] [100.0% done] [73742KB/0KB/0KB /s] [18.5K/0/0 iops] [eta 00m:00s]
  492. write: (groupid=0, jobs=1): err= 0: pid=2969: Mon Oct 12 03:51:04 2015
  493. read : io=4096.0MB, bw=78137KB/s, iops=19534, runt= 53679msec
  494. slat (usec): min=4, max=50, avg= 4.46, stdev= 0.67
  495. clat (usec): min=30, max=23103, avg=45.19, stdev=91.21
  496. lat (usec): min=44, max=23122, avg=49.99, stdev=91.22
  497. clat percentiles (usec):
  498. | 1.00th=[ 40], 5.00th=[ 40], 10.00th=[ 41], 20.00th=[ 41],
  499. | 30.00th=[ 42], 40.00th=[ 42], 50.00th=[ 43], 60.00th=[ 44],
  500. | 70.00th=[ 44], 80.00th=[ 45], 90.00th=[ 46], 95.00th=[ 63],
  501. | 99.00th=[ 71], 99.50th=[ 73], 99.90th=[ 94], 99.95th=[ 294],
  502. | 99.99th=[ 620]
  503. bw (KB /s): min=38456, max=81472, per=100.00%, avg=78194.69, stdev=7599.04
  504. lat (usec) : 50=93.83%, 100=6.08%, 250=0.01%, 500=0.04%, 750=0.02%
  505. lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
  506. cpu : usr=6.61%, sys=12.97%, ctx=1048581, majf=0, minf=30
  507. IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  508. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  509. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  510. issued : total=r=1048576/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
  511. latency : target=0, window=0, percentile=100.00%, depth=1
  512.  
  513. Run status group 0 (all jobs):
  514. READ: io=4096.0MB, aggrb=78136KB/s, minb=78136KB/s, maxb=78136KB/s, mint=53679msec, maxt=53679msec
  515.  
  516. Disk stats (read/write):
  517. md2: ios=1045376/11, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=149796/10, aggrmerge=0/0, aggrticks=6305/126, aggrin_queue=6404, aggrutil=12.05%
  518. sda: ios=149824/9, merge=0/0, ticks=6316/99, in_queue=6386, util=11.89%
  519. sdb: ios=149792/9, merge=0/1, ticks=6374/126, in_queue=6479, util=11.98%
  520. sdc: ios=149792/9, merge=0/0, ticks=6177/110, in_queue=6267, util=11.66%
  521. sdd: ios=149792/10, merge=0/1, ticks=6342/153, in_queue=6464, util=11.88%
  522. sde: ios=149792/13, merge=0/0, ticks=6380/132, in_queue=6482, util=12.05%
  523. sdf: ios=149792/11, merge=0/0, ticks=6284/121, in_queue=6375, util=11.83%
  524. sdg: ios=149792/11, merge=0/0, ticks=6264/142, in_queue=6377, util=11.86%
  525.  
  526. iostat
  527. ------
  528. avg-cpu: %user %nice %system %iowait %steal %idle
  529. 1.02 0.00 1.90 0.00 0.00 97.09
  530.  
  531. Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
  532. sdb 0.00 0.00 2899.20 0.00 11.33 0.00 8.00 0.11 0.04 0.04 0.00 0.04 11.16
  533. sda 0.00 0.00 2899.20 0.00 11.33 0.00 8.00 0.11 0.04 0.04 0.00 0.04 10.52
  534. sdd 0.00 0.00 2894.00 0.00 11.30 0.00 8.00 0.12 0.04 0.04 0.00 0.04 11.82
  535. sdc 0.00 0.00 2899.20 0.00 11.33 0.00 8.00 0.12 0.04 0.04 0.00 0.04 12.22
  536. sde 0.00 0.00 2899.20 0.00 11.33 0.00 8.00 0.11 0.04 0.04 0.00 0.04 11.28
  537. sdf 0.00 0.00 2894.40 0.00 11.31 0.00 8.00 0.11 0.04 0.04 0.00 0.04 11.48
  538. sdg 0.00 0.00 2899.20 0.00 11.33 0.00 8.00 0.12 0.04 0.04 0.00 0.04 12.22
  539. md0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  540. md2 0.00 0.00 20284.80 0.00 79.24 0.00 8.00 0.00 0.00 0.00 0.00 0.00 0.00
  541. dm-0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  542. dm-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  543. md1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  544. md3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  545. dm-2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  546. dm-3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  547. dm-4 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  548. dm-5 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  549.  
  550.  
  551. /dev/md2 32k sequential read, O_DIRECT
  552. ===============================================================================================================================
  553. [root@storage ~]# fio --name=write --ioengine=libaio --bs=32k --direct=1 --rw=read --iodepth=1 --size=28G --filename=/dev/md2
  554. write: (g=0): rw=read, bs=32K-32K/32K-32K/32K-32K, ioengine=libaio, iodepth=1
  555. fio-2.2.8
  556. Starting 1 process
  557. Jobs: 1 (f=1): [R(1)] [100.0% done] [286.7MB/0KB/0KB /s] [9171/0/0 iops] [eta 00m:00s]
  558. write: (groupid=0, jobs=1): err= 0: pid=3000: Mon Oct 12 03:56:51 2015
  559. read : io=28672MB, bw=289848KB/s, iops=9057, runt=101295msec
  560. slat (usec): min=6, max=57, avg= 6.55, stdev= 0.69
  561. clat (usec): min=81, max=27950, avg=102.33, stdev=122.44
  562. lat (usec): min=100, max=27957, avg=109.22, stdev=122.45
  563. clat percentiles (usec):
  564. | 1.00th=[ 94], 5.00th=[ 95], 10.00th=[ 95], 20.00th=[ 96],
  565. | 30.00th=[ 97], 40.00th=[ 98], 50.00th=[ 98], 60.00th=[ 99],
  566. | 70.00th=[ 99], 80.00th=[ 106], 90.00th=[ 110], 95.00th=[ 113],
  567. | 99.00th=[ 125], 99.50th=[ 133], 99.90th=[ 322], 99.95th=[ 342],
  568. | 99.99th=[ 5344]
  569. bw (KB /s): min=168640, max=295168, per=100.00%, avg=289906.26, stdev=16212.97
  570. lat (usec) : 100=72.99%, 250=26.79%, 500=0.20%, 750=0.01%, 1000=0.01%
  571. lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
  572. cpu : usr=2.99%, sys=8.01%, ctx=917508, majf=0, minf=37
  573. IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  574. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  575. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  576. issued : total=r=917504/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
  577. latency : target=0, window=0, percentile=100.00%, depth=1
  578.  
  579. Run status group 0 (all jobs):
  580. READ: io=28672MB, aggrb=289847KB/s, minb=289847KB/s, maxb=289847KB/s, mint=101295msec, maxt=101295msec
  581.  
  582. Disk stats (read/write):
  583. md2: ios=917353/9, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=131072/9, aggrmerge=0/0, aggrticks=13075/122, aggrin_queue=13169, aggrutil=13.07%
  584. sda: ios=131072/11, merge=0/1, ticks=12999/124, in_queue=13098, util=12.93%
  585. sdb: ios=131072/8, merge=0/0, ticks=13171/123, in_queue=13257, util=13.07%
  586. sdc: ios=131072/9, merge=0/0, ticks=13033/111, in_queue=13118, util=12.95%
  587. sdd: ios=131072/9, merge=0/0, ticks=13102/117, in_queue=13185, util=13.00%
  588. sde: ios=131072/10, merge=0/0, ticks=13055/128, in_queue=13161, util=12.95%
  589. sdf: ios=131072/9, merge=0/0, ticks=13128/109, in_queue=13214, util=13.05%
  590. sdg: ios=131072/12, merge=0/1, ticks=13038/147, in_queue=13151, util=12.93%
  591.  
  592. iostat
  593. ------
  594. avg-cpu: %user %nice %system %iowait %steal %idle
  595. 0.44 0.00 1.28 0.00 0.00 98.29
  596.  
  597. Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
  598. sdb 0.00 0.00 1310.40 0.00 40.95 0.00 64.00 0.14 0.11 0.11 0.00 0.11 13.84
  599. sda 0.00 0.00 1310.40 0.00 40.95 0.00 64.00 0.13 0.10 0.10 0.00 0.10 13.00
  600. sdd 0.00 0.00 1310.80 0.00 40.96 0.00 64.00 0.12 0.09 0.09 0.00 0.09 11.86
  601. sdc 0.00 0.00 1310.40 0.00 40.95 0.00 64.00 0.12 0.09 0.09 0.00 0.09 12.08
  602. sde 0.00 0.00 1310.40 0.00 40.95 0.00 64.00 0.12 0.09 0.09 0.00 0.09 12.38
  603. sdf 0.00 0.00 1311.00 0.00 40.97 0.00 64.00 0.13 0.10 0.10 0.00 0.10 13.40
  604. sdg 0.00 0.00 1310.40 0.00 40.95 0.00 64.00 0.13 0.10 0.10 0.00 0.10 12.74
  605. md0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  606. md2 0.00 0.00 9173.60 0.00 286.68 0.00 64.00 0.00 0.00 0.00 0.00 0.00 0.00
  607. dm-0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  608. dm-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  609. md1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  610. md3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  611. dm-2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  612. dm-3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  613. dm-4 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  614. dm-5 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  615.  
  616.  
  617. /dev/md2 128k sequential read, O_DIRECT
  618. ===============================================================================================================================
  619. [root@storage ~]# fio --name=write --ioengine=libaio --bs=128k --direct=1 --rw=read --iodepth=1 --size=28G --filename=/dev/md2
  620. write: (g=0): rw=read, bs=128K-128K/128K-128K/128K-128K, ioengine=libaio, iodepth=1
  621. fio-2.2.8
  622. Starting 1 process
  623. Jobs: 1 (f=1): [R(1)] [100.0% done] [402.5MB/0KB/0KB /s] [3219/0/0 iops] [eta 00m:00s]
  624. write: (groupid=0, jobs=1): err= 0: pid=3010: Mon Oct 12 03:59:19 2015
  625. read : io=28672MB, bw=409674KB/s, iops=3200, runt= 71667msec
  626. slat (usec): min=12, max=84, avg=13.15, stdev= 2.07
  627. clat (usec): min=273, max=24567, avg=297.75, stdev=162.43
  628. lat (usec): min=292, max=24583, avg=311.25, stdev=162.74
  629. clat percentiles (usec):
  630. | 1.00th=[ 282], 5.00th=[ 282], 10.00th=[ 282], 20.00th=[ 282],
  631. | 30.00th=[ 282], 40.00th=[ 282], 50.00th=[ 286], 60.00th=[ 286],
  632. | 70.00th=[ 310], 80.00th=[ 314], 90.00th=[ 326], 95.00th=[ 330],
  633. | 99.00th=[ 338], 99.50th=[ 370], 99.90th=[ 588], 99.95th=[ 628],
  634. | 99.99th=[ 9664]
  635. bw (KB /s): min=178998, max=413952, per=100.00%, avg=409836.69, stdev=19966.24
  636. lat (usec) : 500=99.66%, 750=0.32%, 1000=0.01%
  637. lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
  638. cpu : usr=0.92%, sys=5.11%, ctx=229376, majf=0, minf=62
  639. IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  640. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  641. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  642. issued : total=r=229376/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
  643. latency : target=0, window=0, percentile=100.00%, depth=1
  644.  
  645. Run status group 0 (all jobs):
  646. READ: io=28672MB, aggrb=409674KB/s, minb=409674KB/s, maxb=409674KB/s, mint=71667msec, maxt=71667msec
  647.  
  648. Disk stats (read/write):
  649. md2: ios=228836/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=32768/0, aggrmerge=0/0, aggrticks=9713/0, aggrin_queue=9701, aggrutil=13.62%
  650. sda: ios=32768/0, merge=0/0, ticks=9775/0, in_queue=9766, util=13.62%
  651. sdb: ios=32768/0, merge=0/0, ticks=9736/0, in_queue=9724, util=13.56%
  652. sdc: ios=32768/0, merge=0/0, ticks=9640/0, in_queue=9630, util=13.43%
  653. sdd: ios=32768/0, merge=0/0, ticks=9693/0, in_queue=9681, util=13.50%
  654. sde: ios=32768/0, merge=0/0, ticks=9763/0, in_queue=9749, util=13.60%
  655. sdf: ios=32768/0, merge=0/0, ticks=9758/0, in_queue=9743, util=13.59%
  656. sdg: ios=32768/0, merge=0/0, ticks=9629/0, in_queue=9616, util=13.41%
  657.  
  658. iostat
  659. ------
  660. avg-cpu: %user %nice %system %iowait %steal %idle
  661. 0.13 0.00 0.80 0.00 0.00 99.06
  662.  
  663. Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
  664. sdb 0.00 0.00 460.40 0.00 57.55 0.00 256.00 0.14 0.29 0.29 0.00 0.29 13.56
  665. sda 0.00 0.00 460.40 0.00 57.55 0.00 256.00 0.14 0.30 0.30 0.00 0.30 13.88
  666. sdd 0.00 0.00 460.40 0.00 57.55 0.00 256.00 0.13 0.29 0.29 0.00 0.29 13.36
  667. sdc 0.00 0.00 460.40 0.00 57.55 0.00 256.00 0.13 0.29 0.29 0.00 0.29 13.40
  668. sde 0.00 0.00 460.60 0.00 57.57 0.00 256.00 0.14 0.31 0.31 0.00 0.30 14.04
  669. sdf 0.00 0.00 460.40 0.00 57.55 0.00 256.00 0.14 0.30 0.30 0.00 0.30 13.62
  670. sdg 0.00 0.00 460.40 0.00 57.55 0.00 256.00 0.13 0.29 0.29 0.00 0.29 13.22
  671. md0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  672. md2 0.00 0.00 3223.00 0.00 402.87 0.00 256.00 0.00 0.00 0.00 0.00 0.00 0.00
  673. dm-0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  674. dm-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  675. md1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  676. md3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  677. dm-2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  678. dm-3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  679. dm-4 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  680. dm-5 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  681.  
  682.  
  683. /dev/md2 256k sequential read, O_DIRECT
  684. ===============================================================================================================================
  685. [root@storage ~]# fio --name=write --ioengine=libaio --bs=256k --direct=1 --rw=read --iodepth=1 --size=28G --filename=/dev/md2
  686. write: (g=0): rw=read, bs=256K-256K/256K-256K/256K-256K, ioengine=libaio, iodepth=1
  687. fio-2.2.8
  688. Starting 1 process
  689. Jobs: 1 (f=1): [R(1)] [100.0% done] [733.3MB/0KB/0KB /s] [2933/0/0 iops] [eta 00m:00s]
  690. write: (groupid=0, jobs=1): err= 0: pid=3053: Mon Oct 12 04:03:13 2015
  691. read : io=28672MB, bw=739345KB/s, iops=2888, runt= 39711msec
  692. slat (usec): min=23, max=150, avg=24.30, stdev= 1.75
  693. clat (usec): min=277, max=28977, avg=320.40, stdev=258.77
  694. lat (usec): min=303, max=29002, avg=345.05, stdev=259.06
  695. clat percentiles (usec):
  696. | 1.00th=[ 282], 5.00th=[ 282], 10.00th=[ 286], 20.00th=[ 306],
  697. | 30.00th=[ 310], 40.00th=[ 310], 50.00th=[ 314], 60.00th=[ 318],
  698. | 70.00th=[ 322], 80.00th=[ 326], 90.00th=[ 330], 95.00th=[ 334],
  699. | 99.00th=[ 450], 99.50th=[ 502], 99.90th=[ 540], 99.95th=[ 660],
  700. | 99.99th=[11840]
  701. bw (KB /s): min=483328, max=753152, per=100.00%, avg=739558.28, stdev=41195.37
  702. lat (usec) : 500=99.49%, 750=0.46%, 1000=0.01%
  703. lat (msec) : 2=0.01%, 4=0.01%, 10=0.02%, 20=0.02%, 50=0.01%
  704. cpu : usr=0.65%, sys=7.98%, ctx=114691, majf=0, minf=93
  705. IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  706. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  707. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  708. issued : total=r=114688/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
  709. latency : target=0, window=0, percentile=100.00%, depth=1
  710.  
  711. Run status group 0 (all jobs):
  712. READ: io=28672MB, aggrb=739344KB/s, minb=739344KB/s, maxb=739344KB/s, mint=39711msec, maxt=39711msec
  713.  
  714. Disk stats (read/write):
  715. md2: ios=229306/3, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=32768/3, aggrmerge=0/0, aggrticks=10228/40, aggrin_queue=10258, aggrutil=26.17%
  716. sda: ios=32768/4, merge=0/0, ticks=10380/45, in_queue=10415, util=26.17%
  717. sdb: ios=32768/3, merge=0/0, ticks=10302/43, in_queue=10331, util=25.97%
  718. sdc: ios=32768/4, merge=0/0, ticks=10258/33, in_queue=10274, util=25.83%
  719. sdd: ios=32768/3, merge=0/0, ticks=10120/39, in_queue=10156, util=25.53%
  720. sde: ios=32768/4, merge=0/0, ticks=10161/42, in_queue=10195, util=25.53%
  721. sdf: ios=32768/3, merge=0/0, ticks=10223/36, in_queue=10243, util=25.66%
  722. sdg: ios=32768/4, merge=0/0, ticks=10152/47, in_queue=10195, util=25.61%
  723.  
  724. iostat
  725. ------
  726. avg-cpu: %user %nice %system %iowait %steal %idle
  727. 0.07 0.00 1.37 0.00 0.00 98.56
  728.  
  729. Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
  730. sdb 0.00 0.00 837.20 0.00 104.65 0.00 256.00 0.26 0.31 0.31 0.00 0.31 25.98
  731. sda 0.00 0.00 837.20 0.00 104.65 0.00 256.00 0.26 0.31 0.31 0.00 0.31 26.22
  732. sdd 0.00 0.00 837.40 0.00 104.68 0.00 256.00 0.25 0.30 0.30 0.00 0.30 25.42
  733. sdc 0.00 0.00 837.20 0.00 104.65 0.00 256.00 0.26 0.32 0.32 0.00 0.32 26.46
  734. sde 0.00 0.00 837.20 0.00 104.65 0.00 256.00 0.26 0.31 0.31 0.00 0.31 25.72
  735. sdf 0.00 0.00 837.40 0.00 104.68 0.00 256.00 0.26 0.31 0.31 0.00 0.31 25.68
  736. sdg 0.00 0.00 837.40 0.00 104.68 0.00 256.00 0.26 0.31 0.31 0.00 0.31 25.84
  737. md0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  738. md2 0.00 0.00 5860.80 0.00 732.60 0.00 256.00 0.00 0.00 0.00 0.00 0.00 0.00
  739. dm-0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  740. dm-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  741. md1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  742. md3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  743. dm-2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  744. dm-3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  745. dm-4 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  746. dm-5 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  747.  
  748. /dev/md2 896k sequential read, O_DIRECT
  749. ===============================================================================================================================
  750. [root@storage ~]# fio --name=write --ioengine=libaio --bs=896k --direct=1 --rw=read --iodepth=1 --size=28G --filename=/dev/md2
  751. write: (g=0): rw=read, bs=896K-896K/896K-896K/896K-896K, ioengine=libaio, iodepth=1
  752. fio-2.2.8
  753. Starting 1 process
  754. Jobs: 1 (f=1): [R(1)] [100.0% done] [922.3MB/0KB/0KB /s] [1053/0/0 iops] [eta 00m:00s]
  755. write: (groupid=0, jobs=1): err= 0: pid=3063: Mon Oct 12 04:05:35 2015
  756. read : io=28672MB, bw=981517KB/s, iops=1095, runt= 29913msec
  757. slat (usec): min=66, max=398, avg=78.10, stdev=36.39
  758. clat (usec): min=425, max=32862, avg=832.60, stdev=536.80
  759. lat (usec): min=541, max=33066, avg=911.09, stdev=532.63
  760. clat percentiles (usec):
  761. | 1.00th=[ 470], 5.00th=[ 516], 10.00th=[ 668], 20.00th=[ 708],
  762. | 30.00th=[ 716], 40.00th=[ 724], 50.00th=[ 748], 60.00th=[ 756],
  763. | 70.00th=[ 756], 80.00th=[ 796], 90.00th=[ 812], 95.00th=[ 2024],
  764. | 99.00th=[ 2096], 99.50th=[ 2128], 99.90th=[ 5024], 99.95th=[10176],
  765. | 99.99th=[23936]
  766. bw (KB /s): min=765184, max=1057280, per=100.00%, avg=981879.80, stdev=47476.81
  767. lat (usec) : 500=2.63%, 750=51.22%, 1000=38.64%
  768. lat (msec) : 2=1.94%, 4=5.45%, 10=0.06%, 20=0.04%, 50=0.01%
  769. cpu : usr=0.29%, sys=9.18%, ctx=32773, majf=0, minf=253
  770. IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  771. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  772. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  773. issued : total=r=32768/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
  774. latency : target=0, window=0, percentile=100.00%, depth=1
  775.  
  776. Run status group 0 (all jobs):
  777. READ: io=28672MB, aggrb=981517KB/s, minb=981517KB/s, maxb=981517KB/s, mint=29913msec, maxt=29913msec
  778.  
  779. Disk stats (read/write):
  780. md2: ios=227492/2, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=32768/3, aggrmerge=0/0, aggrticks=14873/53, aggrin_queue=14910, aggrutil=57.73%
  781. sda: ios=32768/3, merge=0/0, ticks=12495/28, in_queue=12508, util=41.64%
  782. sdb: ios=32768/3, merge=0/0, ticks=14576/38, in_queue=14601, util=48.64%
  783. sdc: ios=32768/3, merge=0/0, ticks=15336/37, in_queue=15360, util=51.12%
  784. sdd: ios=32768/3, merge=0/0, ticks=15473/45, in_queue=15500, util=51.57%
  785. sde: ios=32768/4, merge=0/0, ticks=12421/51, in_queue=12454, util=41.39%
  786. sdf: ios=32768/3, merge=0/0, ticks=16480/64, in_queue=16530, util=54.94%
  787. sdg: ios=32768/4, merge=0/0, ticks=17330/112, in_queue=17421, util=57.73%
  788.  
  789. iostat
  790. ------
  791. avg-cpu: %user %nice %system %iowait %steal %idle
  792. 0.10 0.00 1.48 0.00 0.00 98.42
  793.  
  794. Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
  795. sdb 0.00 0.00 1064.00 0.60 133.00 0.00 255.86 0.53 0.50 0.49 12.67 0.50 52.92
  796. sda 0.00 0.00 1064.00 0.60 133.00 0.00 255.86 0.41 0.39 0.38 9.33 0.38 40.68
  797. sdd 0.00 0.00 1064.00 0.60 133.00 0.00 255.86 0.46 0.43 0.43 15.00 0.43 45.44
  798. sdc 0.00 0.00 1064.00 0.60 133.00 0.00 255.86 0.50 0.47 0.46 12.33 0.47 49.56
  799. sde 0.00 0.00 1064.00 0.80 133.00 0.00 255.81 0.43 0.41 0.40 12.75 0.40 42.54
  800. sdf 0.00 0.00 1064.00 0.60 133.00 0.00 255.86 0.56 0.53 0.52 21.33 0.52 55.46
  801. sdg 0.00 0.00 1063.80 0.80 132.97 0.00 255.81 0.62 0.58 0.56 28.00 0.56 59.64
  802. md0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  803. md2 0.00 0.00 7448.00 0.40 931.00 0.00 255.99 0.00 0.00 0.00 0.00 0.00 0.00
  804. dm-0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  805. dm-1 0.00 0.00 0.00 0.20 0.00 0.00 8.00 0.01 68.00 0.00 68.00 68.00 1.36
  806. md1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  807. md3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  808. dm-2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  809. dm-3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  810. dm-4 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  811. dm-5 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  812.  
  813.  
  814.  
  815. /dev/md3 4k sequential read, O_DIRECT
  816. ===============================================================================================================================
  817. [root@storage ~]# fio --name=write --ioengine=libaio --bs=4k --direct=1 --rw=read --iodepth=1 --size=4G --filename=/dev/md3
  818. write: (g=0): rw=read, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=1
  819. fio-2.2.8
  820. Starting 1 process
  821. Jobs: 1 (f=1): [R(1)] [100.0% done] [46845KB/0KB/0KB /s] [11.8K/0/0 iops] [eta 00m:00s]
  822. write: (groupid=0, jobs=1): err= 0: pid=3108: Mon Oct 12 04:16:35 2015
  823. read : io=4096.0MB, bw=46377KB/s, iops=11594, runt= 90440msec
  824. slat (usec): min=4, max=49, avg= 4.49, stdev= 1.04
  825. clat (usec): min=40, max=48452, avg=80.23, stdev=269.41
  826. lat (usec): min=45, max=48457, avg=85.06, stdev=269.44
  827. clat percentiles (usec):
  828. | 1.00th=[ 67], 5.00th=[ 68], 10.00th=[ 68], 20.00th=[ 69],
  829. | 30.00th=[ 70], 40.00th=[ 70], 50.00th=[ 71], 60.00th=[ 71],
  830. | 70.00th=[ 72], 80.00th=[ 72], 90.00th=[ 76], 95.00th=[ 81],
  831. | 99.00th=[ 94], 99.50th=[ 99], 99.90th=[ 3472], 99.95th=[ 7520],
  832. | 99.99th=[11328]
  833. bw (KB /s): min=32671, max=48720, per=100.00%, avg=46388.28, stdev=1519.29
  834. lat (usec) : 50=0.01%, 100=99.54%, 250=0.25%, 500=0.06%, 750=0.01%
  835. lat (usec) : 1000=0.01%
  836. lat (msec) : 2=0.01%, 4=0.02%, 10=0.07%, 20=0.02%, 50=0.01%
  837. cpu : usr=2.96%, sys=8.48%, ctx=1048581, majf=0, minf=30
  838. IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  839. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  840. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  841. issued : total=r=1048576/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
  842. latency : target=0, window=0, percentile=100.00%, depth=1
  843.  
  844. Run status group 0 (all jobs):
  845. READ: io=4096.0MB, aggrb=46376KB/s, minb=46376KB/s, maxb=46376KB/s, mint=90440msec, maxt=90440msec
  846.  
  847. Disk stats (read/write):
  848. md3: ios=1046721/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=149796/10, aggrmerge=0/0, aggrticks=11636/146, aggrin_queue=11768, aggrutil=13.18%
  849. sda: ios=149792/17, merge=0/2, ticks=11397/160, in_queue=11541, util=12.73%
  850. sdb: ios=149824/8, merge=0/0, ticks=11577/122, in_queue=11684, util=12.92%
  851. sdc: ios=149824/10, merge=0/0, ticks=11595/130, in_queue=11711, util=12.93%
  852. sdd: ios=149792/9, merge=0/0, ticks=11672/111, in_queue=11772, util=13.02%
  853. sde: ios=149792/8, merge=0/0, ticks=11661/150, in_queue=11800, util=12.99%
  854. sdf: ios=149792/9, merge=0/0, ticks=11708/152, in_queue=11843, util=13.00%
  855. sdg: ios=149760/15, merge=0/2, ticks=11845/198, in_queue=12028, util=13.18%
  856.  
  857. iostat
  858. ------
  859. avg-cpu: %user %nice %system %iowait %steal %idle
  860. 0.40 0.00 1.21 0.00 0.00 98.38
  861.  
  862. Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
  863. sdb 0.00 0.00 1657.60 0.00 6.48 0.00 8.00 0.13 0.08 0.08 0.00 0.08 12.82
  864. sda 0.00 0.00 1653.40 0.00 6.46 0.00 8.00 0.12 0.07 0.07 0.00 0.07 12.18
  865. sdd 0.00 0.00 1664.00 0.00 6.50 0.00 8.00 0.13 0.08 0.08 0.00 0.08 13.26
  866. sdc 0.00 0.00 1657.60 0.00 6.48 0.00 8.00 0.13 0.08 0.08 0.00 0.08 12.92
  867. sde 0.00 0.00 1657.60 0.00 6.48 0.00 8.00 0.13 0.08 0.08 0.00 0.08 12.74
  868. sdf 0.00 0.00 1657.60 0.00 6.48 0.00 8.00 0.13 0.08 0.08 0.00 0.08 12.84
  869. sdg 0.00 0.00 1657.60 0.00 6.48 0.00 8.00 0.13 0.08 0.08 0.00 0.08 12.90
  870. md0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  871. md2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  872. dm-0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  873. dm-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  874. md1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  875. md3 0.00 0.00 11605.40 0.00 45.33 0.00 8.00 0.00 0.00 0.00 0.00 0.00 0.00
  876. dm-2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  877. dm-3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  878. dm-4 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  879. dm-5 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  880.  
  881.  
  882. /dev/md3 32k sequential read, O_DIRECT
  883. ===============================================================================================================================
  884. [root@storage ~]# fio --name=write --ioengine=libaio --bs=32k --direct=1 --rw=read --iodepth=1 --size=28G --filename=/dev/md3
  885. write: (g=0): rw=read, bs=32K-32K/32K-32K/32K-32K, ioengine=libaio, iodepth=1
  886. fio-2.2.8
  887. Starting 1 process
  888. Jobs: 1 (f=1): [R(1)] [100.0% done] [151.6MB/0KB/0KB /s] [4848/0/0 iops] [eta 00m:00s]
  889. write: (groupid=0, jobs=1): err= 0: pid=3135: Mon Oct 12 04:24:57 2015
  890. read : io=28672MB, bw=155531KB/s, iops=4860, runt=188773msec
  891. slat (usec): min=6, max=70, avg= 6.64, stdev= 3.09
  892. clat (usec): min=109, max=28416, avg=197.49, stdev=735.66
  893. lat (usec): min=119, max=28425, avg=204.50, stdev=735.70
  894. clat percentiles (usec):
  895. | 1.00th=[ 121], 5.00th=[ 122], 10.00th=[ 123], 20.00th=[ 123],
  896. | 30.00th=[ 124], 40.00th=[ 125], 50.00th=[ 125], 60.00th=[ 126],
  897. | 70.00th=[ 127], 80.00th=[ 143], 90.00th=[ 149], 95.00th=[ 155],
  898. | 99.00th=[ 1384], 99.50th=[ 6688], 99.90th=[11328], 99.95th=[11840],
  899. | 99.99th=[12224]
  900. bw (KB /s): min=128820, max=175238, per=100.00%, avg=155586.96, stdev=9104.90
  901. lat (usec) : 250=98.74%, 500=0.20%, 750=0.01%, 1000=0.01%
  902. lat (msec) : 2=0.13%, 4=0.18%, 10=0.54%, 20=0.21%, 50=0.01%
  903. cpu : usr=1.61%, sys=4.38%, ctx=917508, majf=0, minf=37
  904. IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  905. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  906. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  907. issued : total=r=917504/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
  908. latency : target=0, window=0, percentile=100.00%, depth=1
  909.  
  910. Run status group 0 (all jobs):
  911. READ: io=28672MB, aggrb=155531KB/s, minb=155531KB/s, maxb=155531KB/s, mint=188773msec, maxt=188773msec
  912.  
  913. Disk stats (read/write):
  914. md3: ios=916981/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=131072/7, aggrmerge=0/0, aggrticks=25528/94, aggrin_queue=25606, aggrutil=13.86%
  915. sda: ios=131072/10, merge=0/0, ticks=25471/100, in_queue=25552, util=13.54%
  916. sdb: ios=131076/6, merge=0/0, ticks=25404/73, in_queue=25465, util=13.50%
  917. sdc: ios=131076/8, merge=0/0, ticks=25539/103, in_queue=25619, util=13.55%
  918. sdd: ios=131072/6, merge=0/0, ticks=25355/77, in_queue=25417, util=13.47%
  919. sde: ios=131068/6, merge=0/0, ticks=25469/107, in_queue=25559, util=13.53%
  920. sdf: ios=131072/6, merge=0/0, ticks=25401/91, in_queue=25481, util=13.50%
  921. sdg: ios=131068/8, merge=0/0, ticks=26062/113, in_queue=26155, util=13.86%
  922.  
  923. iostat
  924. ------
  925. avg-cpu: %user %nice %system %iowait %steal %idle
  926. 0.27 0.00 0.60 0.00 0.00 99.13
  927.  
  928. Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
  929. sdb 0.00 0.00 699.20 0.00 21.85 0.00 64.00 0.13 0.19 0.19 0.00 0.19 13.46
  930. sda 0.00 0.00 700.00 0.00 21.88 0.00 64.00 0.14 0.19 0.19 0.00 0.19 13.52
  931. sdd 0.00 0.00 699.20 0.00 21.85 0.00 64.00 0.14 0.20 0.20 0.00 0.20 13.72
  932. sdc 0.00 0.00 699.20 0.00 21.85 0.00 64.00 0.13 0.19 0.19 0.00 0.19 13.34
  933. sde 0.00 0.00 699.40 0.00 21.86 0.00 64.00 0.14 0.20 0.20 0.00 0.20 13.90
  934. sdf 0.00 0.00 699.20 0.00 21.85 0.00 64.00 0.13 0.19 0.19 0.00 0.19 12.98
  935. sdg 0.00 0.00 700.00 0.00 21.88 0.00 64.00 0.14 0.20 0.20 0.00 0.20 13.88
  936. md0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  937. md2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  938. dm-0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  939. dm-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  940. md1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  941. md3 0.00 0.00 4896.00 0.00 153.00 0.00 64.00 0.00 0.00 0.00 0.00 0.00 0.00
  942. dm-2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  943. dm-3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  944. dm-4 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  945. dm-5 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  946.  
  947.  
  948. /dev/md3 128k sequential read, O_DIRECT
  949. ===============================================================================================================================
  950. [root@storage ~]# fio --name=write --ioengine=libaio --bs=128k --direct=1 --rw=read --iodepth=1 --size=28G --filename=/dev/md3
  951. write: (g=0): rw=read, bs=128K-128K/128K-128K/128K-128K, ioengine=libaio, iodepth=1
  952. fio-2.2.8
  953. Starting 1 process
  954. Jobs: 1 (f=1): [R(1)] [100.0% done] [168.5MB/0KB/0KB /s] [1347/0/0 iops] [eta 00m:00s]
  955. write: (groupid=0, jobs=1): err= 0: pid=3145: Mon Oct 12 04:28:51 2015
  956. read : io=28672MB, bw=182024KB/s, iops=1422, runt=161298msec
  957. slat (usec): min=11, max=92, avg=15.22, stdev=10.23
  958. clat (usec): min=275, max=57831, avg=685.86, stdev=1713.16
  959. lat (usec): min=292, max=57845, avg=701.52, stdev=1713.16
  960. clat percentiles (usec):
  961. | 1.00th=[ 302], 5.00th=[ 306], 10.00th=[ 306], 20.00th=[ 310],
  962. | 30.00th=[ 310], 40.00th=[ 310], 50.00th=[ 314], 60.00th=[ 314],
  963. | 70.00th=[ 326], 80.00th=[ 370], 90.00th=[ 414], 95.00th=[ 1032],
  964. | 99.00th=[10688], 99.50th=[11840], 99.90th=[12736], 99.95th=[13120],
  965. | 99.99th=[14144]
  966. bw (KB /s): min=88615, max=214272, per=100.00%, avg=182181.27, stdev=13327.32
  967. lat (usec) : 500=91.36%, 750=3.58%, 1000=0.06%
  968. lat (msec) : 2=0.09%, 4=0.91%, 10=2.72%, 20=1.28%, 50=0.01%
  969. lat (msec) : 100=0.01%
  970. cpu : usr=0.45%, sys=2.85%, ctx=229380, majf=0, minf=61
  971. IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  972. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  973. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  974. issued : total=r=229376/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
  975. latency : target=0, window=0, percentile=100.00%, depth=1
  976.  
  977. Run status group 0 (all jobs):
  978. READ: io=28672MB, aggrb=182024KB/s, minb=182024KB/s, maxb=182024KB/s, mint=161298msec, maxt=161298msec
  979.  
  980. Disk stats (read/write):
  981. md3: ios=229203/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=32768/7, aggrmerge=0/0, aggrticks=22315/107, aggrin_queue=22415, aggrutil=14.14%
  982. sda: ios=32768/10, merge=0/0, ticks=22149/143, in_queue=22281, util=13.78%
  983. sdb: ios=32769/6, merge=0/0, ticks=22649/91, in_queue=22736, util=14.09%
  984. sdc: ios=32769/8, merge=0/0, ticks=22066/100, in_queue=22159, util=13.74%
  985. sdd: ios=32768/6, merge=0/0, ticks=21969/71, in_queue=22037, util=13.67%
  986. sde: ios=32767/6, merge=0/0, ticks=22253/99, in_queue=22348, util=13.87%
  987. sdf: ios=32768/6, merge=0/0, ticks=22703/119, in_queue=22814, util=14.14%
  988. sdg: ios=32767/8, merge=0/0, ticks=22416/126, in_queue=22535, util=13.95%
  989.  
  990. iostat
  991. ------
  992. avg-cpu: %user %nice %system %iowait %steal %idle
  993. 0.07 0.00 0.50 0.00 0.00 99.43
  994.  
  995. Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
  996. sdb 0.00 0.00 206.80 0.00 25.85 0.00 256.00 0.15 0.74 0.74 0.00 0.74 15.30
  997. sda 0.00 0.00 206.60 0.00 25.82 0.00 256.00 0.14 0.66 0.66 0.00 0.66 13.64
  998. sdd 0.00 0.00 206.80 0.00 25.85 0.00 256.00 0.13 0.65 0.65 0.00 0.65 13.48
  999. sdc 0.00 0.00 206.60 0.00 25.82 0.00 256.00 0.13 0.64 0.64 0.00 0.64 13.32
  1000. sde 0.00 0.00 206.80 0.00 25.85 0.00 256.00 0.13 0.63 0.63 0.00 0.63 13.08
  1001. sdf 0.00 0.00 207.00 0.00 25.87 0.00 256.00 0.14 0.69 0.69 0.00 0.69 14.18
  1002. sdg 0.00 0.00 206.60 0.00 25.82 0.00 256.00 0.14 0.66 0.66 0.00 0.67 13.82
  1003. md0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  1004. md2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  1005. dm-0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  1006. dm-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  1007. md1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  1008. md3 0.00 0.00 1447.00 0.00 180.88 0.00 256.00 0.00 0.00 0.00 0.00 0.00 0.00
  1009. dm-2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  1010. dm-3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  1011. dm-4 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  1012. dm-5 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  1013.  
  1014.  
  1015. /dev/md3 256k sequential read, O_DIRECT
  1016. ===============================================================================================================================
  1017. [root@storage ~]# fio --name=write --ioengine=libaio --bs=256k --direct=1 --rw=read --iodepth=1 --size=28G --filename=/dev/md3
  1018. write: (g=0): rw=read, bs=256K-256K/256K-256K/256K-256K, ioengine=libaio, iodepth=1
  1019. fio-2.2.8
  1020. Starting 1 process
  1021. Jobs: 1 (f=1): [R(1)] [100.0% done] [236.8MB/0KB/0KB /s] [947/0/0 iops] [eta 00m:00s]
  1022. write: (groupid=0, jobs=1): err= 0: pid=3307: Mon Oct 12 04:38:07 2015
  1023. read : io=28672MB, bw=248591KB/s, iops=971, runt=118106msec
  1024. slat (usec): min=23, max=146, avg=29.69, stdev=18.80
  1025. clat (usec): min=291, max=44383, avg=997.87, stdev=2351.69
  1026. lat (usec): min=316, max=44410, avg=1028.01, stdev=2351.57
  1027. clat percentiles (usec):
  1028. | 1.00th=[ 306], 5.00th=[ 310], 10.00th=[ 310], 20.00th=[ 310],
  1029. | 30.00th=[ 314], 40.00th=[ 354], 50.00th=[ 366], 60.00th=[ 374],
  1030. | 70.00th=[ 382], 80.00th=[ 410], 90.00th=[ 494], 95.00th=[ 7264],
  1031. | 99.00th=[11840], 99.50th=[12352], 99.90th=[12864], 99.95th=[13120],
  1032. | 99.99th=[15424]
  1033. bw (KB /s): min=191235, max=313344, per=100.00%, avg=248785.65, stdev=21851.51
  1034. lat (usec) : 500=90.04%, 750=1.36%, 1000=0.14%
  1035. lat (msec) : 2=0.56%, 4=1.06%, 10=3.46%, 20=3.37%, 50=0.01%
  1036. cpu : usr=0.29%, sys=3.40%, ctx=114690, majf=0, minf=93
  1037. IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  1038. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  1039. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  1040. issued : total=r=114688/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
  1041. latency : target=0, window=0, percentile=100.00%, depth=1
  1042.  
  1043. Run status group 0 (all jobs):
  1044. READ: io=28672MB, aggrb=248591KB/s, minb=248591KB/s, maxb=248591KB/s, mint=118106msec, maxt=118106msec
  1045.  
  1046. Disk stats (read/write):
  1047. md3: ios=229356/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=32768/10, aggrmerge=0/0, aggrticks=21882/185, aggrin_queue=22062, aggrutil=18.83%
  1048. sda: ios=32768/12, merge=0/1, ticks=22041/195, in_queue=22229, util=18.73%
  1049. sdb: ios=32769/10, merge=0/0, ticks=22062/203, in_queue=22265, util=18.83%
  1050. sdc: ios=32769/14, merge=0/1, ticks=21904/209, in_queue=22107, util=18.66%
  1051. sdd: ios=32768/8, merge=0/0, ticks=21763/186, in_queue=21938, util=18.55%
  1052. sde: ios=32767/9, merge=0/0, ticks=21954/150, in_queue=22098, util=18.71%
  1053. sdf: ios=32768/9, merge=0/0, ticks=21904/142, in_queue=22043, util=18.64%
  1054. sdg: ios=32767/8, merge=0/0, ticks=21551/211, in_queue=21756, util=18.36%
  1055.  
  1056.  
  1057. iostat
  1058. ------
  1059. avg-cpu: %user %nice %system %iowait %steal %idle
  1060. 0.07 0.00 0.54 0.00 0.00 99.40
  1061.  
  1062. Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
  1063. sdb 0.00 0.00 270.00 0.00 33.75 0.00 256.00 0.18 0.66 0.66 0.00 0.66 17.86
  1064. sda 0.00 0.00 269.80 0.00 33.73 0.00 256.00 0.18 0.66 0.66 0.00 0.66 17.90
  1065. sdd 0.00 0.00 270.00 0.00 33.75 0.00 256.00 0.18 0.68 0.68 0.00 0.68 18.38
  1066. sdc 0.00 0.00 269.80 0.00 33.73 0.00 256.00 0.18 0.68 0.68 0.00 0.68 18.34
  1067. sde 0.00 0.00 270.00 0.00 33.75 0.00 256.00 0.18 0.66 0.66 0.00 0.66 17.94
  1068. sdf 0.00 0.00 270.00 0.00 33.75 0.00 256.00 0.18 0.68 0.68 0.00 0.68 18.26
  1069. sdg 0.00 0.00 269.80 0.00 33.73 0.00 256.00 0.20 0.73 0.73 0.00 0.73 19.82
  1070. md0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  1071. md2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  1072. dm-0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  1073. dm-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  1074. md1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  1075. md3 0.00 0.00 1888.80 0.00 236.10 0.00 256.00 0.00 0.00 0.00 0.00 0.00 0.00
  1076. dm-2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  1077. dm-3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  1078. dm-4 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  1079. dm-5 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  1080.  
  1081. /dev/md3 384k sequential read, O_DIRECT
  1082. ===============================================================================================================================
  1083. [root@storage ~]# fio --name=write --ioengine=libaio --bs=384k --direct=1 --rw=read --iodepth=1 --size=28G --filename=/dev/md3
  1084. write: (g=0): rw=read, bs=384K-384K/384K-384K/384K-384K, ioengine=libaio, iodepth=1
  1085. fio-2.2.8
  1086. Starting 1 process
  1087. Jobs: 1 (f=1): [R(1)] [100.0% done] [303.5MB/0KB/0KB /s] [809/0/0 iops] [eta 00m:00s]
  1088. write: (groupid=0, jobs=1): err= 0: pid=3317: Mon Oct 12 04:41:05 2015
  1089. read : io=28672MB, bw=299772KB/s, iops=780, runt= 97942msec
  1090. slat (usec): min=34, max=207, avg=47.95, stdev=31.53
  1091. clat (usec): min=293, max=44016, avg=1230.31, stdev=2662.47
  1092. lat (usec): min=336, max=44145, avg=1278.78, stdev=2662.41
  1093. clat percentiles (usec):
  1094. | 1.00th=[ 310], 5.00th=[ 310], 10.00th=[ 314], 20.00th=[ 358],
  1095. | 30.00th=[ 366], 40.00th=[ 374], 50.00th=[ 382], 60.00th=[ 390],
  1096. | 70.00th=[ 410], 80.00th=[ 442], 90.00th=[ 2128], 95.00th=[ 8768],
  1097. | 99.00th=[12352], 99.50th=[12608], 99.90th=[13504], 99.95th=[13760],
  1098. | 99.99th=[22656]
  1099. bw (KB /s): min=163584, max=762624, per=100.00%, avg=300190.51, stdev=66245.68
  1100. lat (usec) : 500=82.70%, 750=2.31%, 1000=0.78%
  1101. lat (msec) : 2=3.92%, 4=1.99%, 10=4.10%, 20=4.19%, 50=0.02%
  1102. cpu : usr=0.31%, sys=4.24%, ctx=76467, majf=0, minf=125
  1103. IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  1104. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  1105. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  1106. issued : total=r=76459/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
  1107. latency : target=0, window=0, percentile=100.00%, depth=1
  1108.  
  1109. Run status group 0 (all jobs):
  1110. READ: io=28672MB, aggrb=299771KB/s, minb=299771KB/s, maxb=299771KB/s, mint=97942msec, maxt=97942msec
  1111.  
  1112. Disk stats (read/write):
  1113. md3: ios=229164/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=32768/16, aggrmerge=0/1, aggrticks=21368/311, aggrin_queue=21673, aggrutil=22.47%
  1114. sda: ios=32768/13, merge=0/0, ticks=21411/253, in_queue=21658, util=22.07%
  1115. sdb: ios=32769/18, merge=0/0, ticks=21505/375, in_queue=21868, util=22.14%
  1116. sdc: ios=32769/18, merge=0/0, ticks=21171/364, in_queue=21529, util=21.78%
  1117. sdd: ios=32768/14, merge=0/1, ticks=20844/291, in_queue=21125, util=21.46%
  1118. sde: ios=32768/18, merge=0/3, ticks=21801/353, in_queue=22150, util=22.47%
  1119. sdf: ios=32768/15, merge=0/2, ticks=21319/298, in_queue=21612, util=21.95%
  1120. sdg: ios=32767/16, merge=0/2, ticks=21528/244, in_queue=21771, util=22.13%
  1121.  
  1122.  
  1123.  
  1124.  
  1125.  
  1126. iostat
  1127. ------
  1128. avg-cpu: %user %nice %system %iowait %steal %idle
  1129. 0.10 0.00 0.77 0.00 0.00 99.13
  1130.  
  1131. Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
  1132. sdb 0.00 0.00 323.40 0.00 40.42 0.00 256.00 0.21 0.65 0.65 0.00 0.65 21.06
  1133. sda 0.00 0.00 323.20 0.00 40.40 0.00 256.00 0.23 0.70 0.70 0.00 0.70 22.62
  1134. sdd 0.00 0.00 323.20 0.00 40.40 0.00 256.00 0.21 0.66 0.66 0.00 0.66 21.42
  1135. sdc 0.00 0.00 323.00 0.00 40.38 0.00 256.00 0.21 0.66 0.66 0.00 0.66 21.28
  1136. sde 0.00 0.00 323.20 0.00 40.40 0.00 256.00 0.23 0.71 0.71 0.00 0.71 22.82
  1137. sdf 0.00 0.00 323.20 0.00 40.40 0.00 256.00 0.19 0.60 0.60 0.00 0.60 19.38
  1138. sdg 0.00 0.00 323.20 0.00 40.40 0.00 256.00 0.22 0.67 0.67 0.00 0.67 21.68
  1139. md0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  1140. md2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  1141. dm-0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  1142. dm-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  1143. md1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  1144. md3 0.00 0.00 2262.00 0.00 282.75 0.00 256.00 0.00 0.00 0.00 0.00 0.00 0.00
  1145. dm-2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  1146. dm-3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  1147. dm-4 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  1148. dm-5 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  1149.  
  1150.  
  1151.  
  1152. /dev/md3 512k sequential read, O_DIRECT
  1153. ===============================================================================================================================
  1154. [root@storage ~]# fio --name=write --ioengine=libaio --bs=512k --direct=1 --rw=read --iodepth=1 --size=28G --filename=/dev/md3
  1155. write: (g=0): rw=read, bs=512K-512K/512K-512K/512K-512K, ioengine=libaio, iodepth=1
  1156. fio-2.2.8
  1157. Starting 1 process
  1158. Jobs: 1 (f=1): [R(1)] [100.0% done] [296.8MB/0KB/0KB /s] [593/0/0 iops] [eta 00m:00s]
  1159. write: (groupid=0, jobs=1): err= 0: pid=3333: Mon Oct 12 04:44:08 2015
  1160. read : io=28672MB, bw=315375KB/s, iops=615, runt= 93096msec
  1161. slat (usec): min=40, max=229, avg=63.38, stdev=42.49
  1162. clat (usec): min=297, max=45065, avg=1556.81, stdev=2968.96
  1163. lat (usec): min=346, max=45126, avg=1620.79, stdev=2967.91
  1164. clat percentiles (usec):
  1165. | 1.00th=[ 310], 5.00th=[ 326], 10.00th=[ 362], 20.00th=[ 370],
  1166. | 30.00th=[ 382], 40.00th=[ 394], 50.00th=[ 418], 60.00th=[ 454],
  1167. | 70.00th=[ 548], 80.00th=[ 1400], 90.00th=[ 4704], 95.00th=[10304],
  1168. | 99.00th=[12736], 99.50th=[12864], 99.90th=[13888], 99.95th=[14016],
  1169. | 99.99th=[29056]
  1170. bw (KB /s): min=223232, max=691200, per=100.00%, avg=315732.82, stdev=56881.43
  1171. lat (usec) : 500=65.99%, 750=11.81%, 1000=0.56%
  1172. lat (msec) : 2=7.85%, 4=3.25%, 10=5.28%, 20=5.24%, 50=0.02%
  1173. cpu : usr=0.29%, sys=4.39%, ctx=57350, majf=0, minf=157
  1174. IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  1175. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  1176. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  1177. issued : total=r=57344/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
  1178. latency : target=0, window=0, percentile=100.00%, depth=1
  1179.  
  1180. Run status group 0 (all jobs):
  1181. READ: io=28672MB, aggrb=315374KB/s, minb=315374KB/s, maxb=315374KB/s, mint=93096msec, maxt=93096msec
  1182.  
  1183. Disk stats (read/write):
  1184. md3: ios=229268/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=32769/10, aggrmerge=1/0, aggrticks=22200/213, aggrin_queue=22405, aggrutil=24.12%
  1185. sda: ios=32770/11, merge=1/0, ticks=22214/219, in_queue=22426, util=23.92%
  1186. sdb: ios=32770/14, merge=1/0, ticks=22256/154, in_queue=22403, util=23.99%
  1187. sdc: ios=32770/14, merge=1/0, ticks=22348/223, in_queue=22559, util=24.11%
  1188. sdd: ios=32769/10, merge=1/0, ticks=22108/215, in_queue=22316, util=23.84%
  1189. sde: ios=32768/8, merge=1/0, ticks=22076/240, in_queue=22308, util=23.84%
  1190. sdf: ios=32769/8, merge=1/0, ticks=22313/231, in_queue=22532, util=24.12%
  1191. sdg: ios=32768/9, merge=1/0, ticks=22090/210, in_queue=22296, util=23.85%
  1192.  
  1193. iostat
  1194. ------
  1195. avg-cpu: %user %nice %system %iowait %steal %idle
  1196. 0.07 0.00 0.74 0.00 0.00 99.20
  1197.  
  1198. Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
  1199. sdb 0.00 0.00 357.00 0.00 44.62 0.00 256.00 0.23 0.65 0.65 0.00 0.65 23.04
  1200. sda 0.00 0.00 356.80 0.00 44.60 0.00 256.00 0.25 0.70 0.70 0.00 0.70 24.90
  1201. sdd 0.00 0.00 357.00 0.00 44.62 0.00 256.00 0.25 0.72 0.72 0.00 0.71 25.50
  1202. sdc 0.00 0.00 356.80 0.00 44.60 0.00 256.00 0.25 0.71 0.71 0.00 0.71 25.18
  1203. sde 0.00 0.00 356.80 0.00 44.60 0.00 256.00 0.25 0.69 0.69 0.00 0.69 24.62
  1204. sdf 0.00 0.00 356.60 0.00 44.57 0.00 256.00 0.24 0.66 0.66 0.00 0.66 23.66
  1205. sdg 0.00 0.00 356.60 0.00 44.57 0.00 256.00 0.21 0.58 0.58 0.00 0.58 20.86
  1206. md0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  1207. md2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  1208. dm-0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  1209. dm-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  1210. md1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  1211. md3 0.00 0.00 2497.60 0.00 312.20 0.00 256.00 0.00 0.00 0.00 0.00 0.00 0.00
  1212. dm-2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  1213. dm-3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  1214. dm-4 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  1215. dm-5 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  1216.  
  1217.  
  1218. /dev/md3 640k sequential read, O_DIRECT
  1219. ===============================================================================================================================
  1220. [root@storage ~]# fio --name=write --ioengine=libaio --bs=640k --direct=1 --rw=read --iodepth=1 --size=28G --filename=/dev/md3
  1221. write: (g=0): rw=read, bs=640K-640K/640K-640K/640K-640K, ioengine=libaio, iodepth=1
  1222. fio-2.2.8
  1223. Starting 1 process
  1224. Jobs: 1 (f=1): [R(1)] [100.0% done] [266.4MB/0KB/0KB /s] [426/0/0 iops] [eta 00m:00s]
  1225. write: (groupid=0, jobs=1): err= 0: pid=3349: Mon Oct 12 04:47:05 2015
  1226. read : io=28673MB, bw=296351KB/s, iops=463, runt= 99074msec
  1227. slat (usec): min=54, max=304, avg=86.84, stdev=54.68
  1228. clat (usec): min=301, max=47005, avg=2069.29, stdev=3442.02
  1229. lat (usec): min=367, max=47191, avg=2156.57, stdev=3437.50
  1230. clat percentiles (usec):
  1231. | 1.00th=[ 322], 5.00th=[ 362], 10.00th=[ 374], 20.00th=[ 414],
  1232. | 30.00th=[ 442], 40.00th=[ 524], 50.00th=[ 676], 60.00th=[ 692],
  1233. | 70.00th=[ 756], 80.00th=[ 1960], 90.00th=[ 8096], 95.00th=[12224],
  1234. | 99.00th=[12736], 99.50th=[12864], 99.90th=[13888], 99.95th=[15040],
  1235. | 99.99th=[35584]
  1236. bw (KB /s): min=208667, max=701440, per=100.00%, avg=296658.04, stdev=57800.61
  1237. lat (usec) : 500=38.73%, 750=30.72%, 1000=3.68%
  1238. lat (msec) : 2=7.51%, 4=4.70%, 10=6.86%, 20=7.76%, 50=0.04%
  1239. cpu : usr=0.20%, sys=4.44%, ctx=45891, majf=0, minf=189
  1240. IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  1241. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  1242. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  1243. issued : total=r=45876/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
  1244. latency : target=0, window=0, percentile=100.00%, depth=1
  1245.  
  1246. Run status group 0 (all jobs):
  1247. READ: io=28673MB, aggrb=296350KB/s, minb=296350KB/s, maxb=296350KB/s, mint=99074msec, maxt=99074msec
  1248.  
  1249. Disk stats (read/write):
  1250. md3: ios=229365/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=32768/14, aggrmerge=0/0, aggrticks=24351/290, aggrin_queue=24634, aggrutil=25.13%
  1251. sda: ios=32769/19, merge=0/2, ticks=24453/302, in_queue=24746, util=24.90%
  1252. sdb: ios=32769/15, merge=0/0, ticks=24623/263, in_queue=24877, util=24.92%
  1253. sdc: ios=32770/17, merge=0/0, ticks=23848/274, in_queue=24116, util=24.22%
  1254. sdd: ios=32768/13, merge=0/0, ticks=24326/321, in_queue=24639, util=24.59%
  1255. sde: ios=32768/11, merge=0/0, ticks=24188/278, in_queue=24459, util=24.55%
  1256. sdf: ios=32768/12, merge=0/0, ticks=24391/262, in_queue=24650, util=24.72%
  1257. sdg: ios=32768/16, merge=0/2, ticks=24632/336, in_queue=24954, util=25.13%
  1258.  
  1259.  
  1260. iostat
  1261. ------
  1262. avg-cpu: %user %nice %system %iowait %steal %idle
  1263. 0.07 0.00 0.84 0.00 0.00 99.10
  1264.  
  1265. Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
  1266. sdb 0.00 0.00 351.80 0.00 43.97 0.00 256.00 0.27 0.77 0.77 0.00 0.77 27.00
  1267. sda 0.00 0.00 351.80 0.00 43.97 0.00 256.00 0.27 0.76 0.76 0.00 0.76 26.74
  1268. sdd 0.00 0.00 351.80 0.00 43.97 0.00 256.00 0.24 0.69 0.69 0.00 0.69 24.40
  1269. sdc 0.00 0.00 351.40 0.00 43.92 0.00 256.00 0.24 0.67 0.67 0.00 0.67 23.64
  1270. sde 0.00 0.00 352.00 0.00 44.00 0.00 256.00 0.22 0.63 0.63 0.00 0.63 22.26
  1271. sdf 0.00 0.00 351.80 0.00 43.97 0.00 256.00 0.24 0.68 0.68 0.00 0.68 23.88
  1272. sdg 0.00 0.00 351.80 0.00 43.97 0.00 256.00 0.29 0.82 0.82 0.00 0.82 28.80
  1273. md0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  1274. md2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  1275. dm-0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  1276. dm-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  1277. md1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  1278. md3 0.00 0.00 2462.00 0.00 307.75 0.00 256.00 0.00 0.00 0.00 0.00 0.00 0.00
  1279. dm-2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  1280. dm-3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  1281. dm-4 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  1282. dm-5 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  1283.  
  1284.  
  1285. /dev/md3 1280k sequential read, O_DIRECT
  1286. ===============================================================================================================================
  1287. [root@storage ~]# fio --name=write --ioengine=libaio --bs=1280k --direct=1 --rw=read --iodepth=1 --size=28G --filename=/dev/md3
  1288. write: (g=0): rw=read, bs=1280K-1280K/1280K-1280K/1280K-1280K, ioengine=libaio, iodepth=1
  1289. fio-2.2.8
  1290. Starting 1 process
  1291. Jobs: 1 (f=1): [R(1)] [100.0% done] [495.8MB/0KB/0KB /s] [396/0/0 iops] [eta 00m:00s]
  1292. write: (groupid=0, jobs=1): err= 0: pid=3359: Mon Oct 12 04:49:45 2015
  1293. read : io=28673MB, bw=543655KB/s, iops=424, runt= 54006msec
  1294. slat (usec): min=91, max=552, avg=188.97, stdev=73.00
  1295. clat (usec): min=567, max=32518, avg=2159.21, stdev=2457.20
  1296. lat (usec): min=721, max=32619, avg=2348.78, stdev=2444.45
  1297. clat percentiles (usec):
  1298. | 1.00th=[ 636], 5.00th=[ 660], 10.00th=[ 676], 20.00th=[ 708],
  1299. | 30.00th=[ 1192], 40.00th=[ 1256], 50.00th=[ 1400], 60.00th=[ 1736],
  1300. | 70.00th=[ 2040], 80.00th=[ 2416], 90.00th=[ 3376], 95.00th=[ 7968],
  1301. | 99.00th=[13120], 99.50th=[13376], 99.90th=[14400], 99.95th=[15040],
  1302. | 99.99th=[26752]
  1303. bw (KB /s): min=290909, max=770039, per=100.00%, avg=545032.81, stdev=117494.83
  1304. lat (usec) : 750=23.45%, 1000=1.32%
  1305. lat (msec) : 2=43.13%, 4=24.06%, 10=4.24%, 20=3.77%, 50=0.03%
  1306. cpu : usr=0.42%, sys=8.65%, ctx=22946, majf=0, minf=349
  1307. IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  1308. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  1309. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  1310. issued : total=r=22938/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
  1311. latency : target=0, window=0, percentile=100.00%, depth=1
  1312.  
  1313. Run status group 0 (all jobs):
  1314. READ: io=28673MB, aggrb=543655KB/s, minb=543655KB/s, maxb=543655KB/s, mint=54006msec, maxt=54006msec
  1315.  
  1316. Disk stats (read/write):
  1317. md3: ios=229290/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=19661/7, aggrmerge=13107/0, aggrticks=18663/153, aggrin_queue=18809, aggrutil=35.90%
  1318. sda: ios=19661/8, merge=13108/0, ticks=18335/164, in_queue=18493, util=34.03%
  1319. sdb: ios=19661/8, merge=13108/0, ticks=18777/141, in_queue=18913, util=34.85%
  1320. sdc: ios=19662/10, merge=13108/0, ticks=18350/173, in_queue=18514, util=34.05%
  1321. sdd: ios=19661/6, merge=13107/0, ticks=18479/148, in_queue=18622, util=34.31%
  1322. sde: ios=19661/6, merge=13107/0, ticks=18099/154, in_queue=18245, util=33.64%
  1323. sdf: ios=19661/6, merge=13107/0, ticks=19238/108, in_queue=19341, util=35.72%
  1324. sdg: ios=19661/6, merge=13107/0, ticks=19365/186, in_queue=19541, util=35.90%
  1325.  
  1326. iostat
  1327. ------
  1328. avg-cpu: %user %nice %system %iowait %steal %idle
  1329. 0.10 0.00 1.47 0.00 0.00 98.43
  1330.  
  1331. Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
  1332. sdb 241.00 0.00 361.40 0.00 75.30 0.00 426.71 0.39 1.08 1.08 0.00 1.08 39.10
  1333. sda 241.00 0.00 361.60 0.00 75.35 0.00 426.76 0.35 0.97 0.97 0.00 0.97 34.94
  1334. sdd 241.00 0.00 361.40 0.00 75.28 0.00 426.57 0.33 0.92 0.92 0.00 0.92 33.36
  1335. sdc 240.80 0.00 361.80 0.00 75.35 0.00 426.53 0.32 0.88 0.88 0.00 0.88 31.96
  1336. sde 241.20 0.00 361.40 0.00 75.30 0.00 426.71 0.33 0.92 0.92 0.00 0.92 33.36
  1337. sdf 241.20 0.00 361.40 0.00 75.30 0.00 426.71 0.32 0.89 0.89 0.00 0.89 32.02
  1338. sdg 241.00 0.00 361.60 0.00 75.33 0.00 426.62 0.35 0.96 0.96 0.00 0.96 34.82
  1339. md0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  1340. md2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  1341. dm-0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  1342. dm-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  1343. md1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  1344. md3 0.00 0.00 4218.00 0.00 527.25 0.00 256.00 0.00 0.00 0.00 0.00 0.00 0.00
  1345. dm-2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  1346. dm-3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  1347. dm-4 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  1348. dm-5 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  1349.  
  1350.  
  1351. /dev/md3 2560k sequential read, O_DIRECT
  1352. ===============================================================================================================================
  1353. [root@storage ~]# fio --name=write --ioengine=libaio --bs=2560k --direct=1 --rw=read --iodepth=1 --size=28G --filename=/dev/md3
  1354. write: (g=0): rw=read, bs=2560K-2560K/2560K-2560K/2560K-2560K, ioengine=libaio, iodepth=1
  1355. fio-2.2.8
  1356. Starting 1 process
  1357. Jobs: 1 (f=1): [R(1)] [100.0% done] [539.5MB/0KB/0KB /s] [215/0/0 iops] [eta 00m:00s]
  1358. write: (groupid=0, jobs=1): err= 0: pid=3374: Mon Oct 12 04:52:22 2015
  1359. read : io=28673MB, bw=618093KB/s, iops=241, runt= 47502msec
  1360. slat (usec): min=259, max=850, avg=359.39, stdev=55.18
  1361. clat (msec): min=1, max=46, avg= 3.77, stdev= 3.23
  1362. lat (msec): min=1, max=46, avg= 4.13, stdev= 3.22
  1363. clat percentiles (usec):
  1364. | 1.00th=[ 1224], 5.00th=[ 1352], 10.00th=[ 1592], 20.00th=[ 2256],
  1365. | 30.00th=[ 2576], 40.00th=[ 2736], 50.00th=[ 2928], 60.00th=[ 3504],
  1366. | 70.00th=[ 3856], 80.00th=[ 4256], 90.00th=[ 5344], 95.00th=[ 8896],
  1367. | 99.00th=[18560], 99.50th=[22656], 99.90th=[35584], 99.95th=[36096],
  1368. | 99.99th=[44800]
  1369. bw (KB /s): min=430080, max=773120, per=100.00%, avg=619315.79, stdev=78182.70
  1370. lat (msec) : 2=16.92%, 4=57.41%, 10=21.21%, 20=3.66%, 50=0.80%
  1371. cpu : usr=0.26%, sys=9.13%, ctx=11473, majf=0, minf=158
  1372. IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  1373. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  1374. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  1375. issued : total=r=11469/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
  1376. latency : target=0, window=0, percentile=100.00%, depth=1
  1377.  
  1378. Run status group 0 (all jobs):
  1379. READ: io=28673MB, aggrb=618092KB/s, minb=618092KB/s, maxb=618092KB/s, mint=47502msec, maxt=47502msec
  1380.  
  1381. Disk stats (read/write):
  1382. md3: ios=229260/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=13107/3, aggrmerge=19661/0, aggrticks=20111/78, aggrin_queue=20184, aggrutil=47.72%
  1383. sda: ios=13107/4, merge=19662/0, ticks=18837/69, in_queue=18901, util=37.70%
  1384. sdb: ios=13107/4, merge=19662/0, ticks=19449/109, in_queue=19553, util=39.09%
  1385. sdc: ios=13108/5, merge=19662/0, ticks=19896/101, in_queue=19991, util=39.50%
  1386. sdd: ios=13107/3, merge=19661/0, ticks=20025/55, in_queue=20071, util=40.01%
  1387. sde: ios=13108/3, merge=19660/0, ticks=19051/82, in_queue=19127, util=37.87%
  1388. sdf: ios=13108/3, merge=19660/0, ticks=23719/52, in_queue=23769, util=47.72%
  1389. sdg: ios=13107/3, merge=19661/0, ticks=19806/82, in_queue=19882, util=39.77%
  1390.  
  1391. iostat
  1392. ------
  1393. avg-cpu: %user %nice %system %iowait %steal %idle
  1394. 0.07 0.00 1.44 0.00 0.00 98.49
  1395.  
  1396. Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
  1397. sdb 400.80 0.00 267.40 0.80 83.55 0.00 638.00 0.42 1.58 1.50 27.25 1.45 38.82
  1398. sda 401.00 0.00 267.20 0.80 83.50 0.00 638.10 0.41 1.54 1.49 17.25 1.41 37.86
  1399. sdd 400.80 0.00 267.60 0.60 83.55 0.00 638.00 0.40 1.48 1.45 18.33 1.38 36.96
  1400. sdc 400.80 0.00 267.40 1.00 83.58 0.00 637.72 0.46 1.71 1.65 20.20 1.54 41.44
  1401. sde 400.80 0.00 267.40 0.60 83.58 0.00 638.66 0.40 1.49 1.43 27.33 1.36 36.34
  1402. sdf 400.80 0.00 267.40 0.60 83.55 0.00 638.47 0.53 1.98 1.94 17.33 1.86 49.72
  1403. sdg 400.80 0.00 267.20 0.60 83.50 0.00 638.57 0.45 1.66 1.61 27.33 1.54 41.36
  1404. md0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  1405. md2 0.00 0.00 0.00 0.60 0.00 0.00 5.33 0.00 0.00 0.00 0.00 0.00 0.00
  1406. dm-0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  1407. dm-1 0.00 0.00 0.00 0.40 0.00 0.00 8.00 0.02 40.50 0.00 40.50 40.50 1.62
  1408. md1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  1409. md3 0.00 0.00 4676.00 0.00 584.50 0.00 256.00 0.00 0.00 0.00 0.00 0.00 0.00
  1410. dm-2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  1411. dm-3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  1412. dm-4 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  1413. dm-5 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  1414.  
  1415.  
  1416. /dev/md3 2560k sequential read, buffered
  1417. ===============================================================================================================================
  1418. [root@storage ~]# fio --name=write --ioengine=libaio --bs=2560k --direct=0 --rw=read --iodepth=1 --size=28G --filename=/dev/md3
  1419. write: (g=0): rw=read, bs=2560K-2560K/2560K-2560K/2560K-2560K, ioengine=libaio, iodepth=1
  1420. fio-2.2.8
  1421. Starting 1 process
  1422. Jobs: 1 (f=1): [R(1)] [100.0% done] [669.4MB/0KB/0KB /s] [267/0/0 iops] [eta 00m:00s]
  1423. write: (groupid=0, jobs=1): err= 0: pid=3385: Mon Oct 12 04:54:46 2015
  1424. read : io=28673MB, bw=690546KB/s, iops=269, runt= 42518msec
  1425. slat (msec): min=1, max=25, avg= 3.70, stdev= 1.89
  1426. clat (usec): min=1, max=94, avg= 1.72, stdev= 2.48
  1427. lat (msec): min=1, max=25, avg= 3.70, stdev= 1.90
  1428. clat percentiles (usec):
  1429. | 1.00th=[ 1], 5.00th=[ 1], 10.00th=[ 1], 20.00th=[ 1],
  1430. | 30.00th=[ 1], 40.00th=[ 2], 50.00th=[ 2], 60.00th=[ 2],
  1431. | 70.00th=[ 2], 80.00th=[ 2], 90.00th=[ 2], 95.00th=[ 2],
  1432. | 99.00th=[ 3], 99.50th=[ 4], 99.90th=[ 41], 99.95th=[ 75],
  1433. | 99.99th=[ 93]
  1434. bw (KB /s): min=609280, max=757760, per=100.00%, avg=691125.30, stdev=35238.47
  1435. lat (usec) : 2=39.89%, 4=59.45%, 10=0.44%, 20=0.04%, 50=0.10%
  1436. lat (usec) : 100=0.08%
  1437. cpu : usr=0.10%, sys=46.49%, ctx=15370, majf=0, minf=669
  1438. IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  1439. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  1440. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  1441. issued : total=r=11469/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
  1442. latency : target=0, window=0, percentile=100.00%, depth=1
  1443.  
  1444. Run status group 0 (all jobs):
  1445. READ: io=28673MB, aggrb=690546KB/s, minb=690546KB/s, maxb=690546KB/s, mint=42518msec, maxt=42518msec
  1446.  
  1447. Disk stats (read/write):
  1448. md3: ios=7333252/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=16337/0, aggrmerge=1032385/0, aggrticks=33553/0, aggrin_queue=33550, aggrutil=68.46%
  1449. sda: ios=16339/0, merge=1032397/0, ticks=28511/0, in_queue=28509, util=54.87%
  1450. sdb: ios=16334/0, merge=1032402/0, ticks=33197/0, in_queue=33194, util=59.39%
  1451. sdc: ios=16343/0, merge=1032393/0, ticks=29405/0, in_queue=29401, util=56.03%
  1452. sdd: ios=16332/0, merge=1032405/0, ticks=32079/0, in_queue=32076, util=58.48%
  1453. sde: ios=16343/0, merge=1032361/0, ticks=30721/0, in_queue=30719, util=57.63%
  1454. sdf: ios=16346/0, merge=1032361/0, ticks=42313/0, in_queue=42312, util=68.46%
  1455. sdg: ios=16328/0, merge=1032376/0, ticks=38648/0, in_queue=38641, util=64.96%
  1456.  
  1457.  
  1458.  
  1459. iostat
  1460. ------
  1461. avg-cpu: %user %nice %system %iowait %steal %idle
  1462. 0.07 0.00 9.72 7.93 0.00 82.28
  1463.  
  1464. Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
  1465. sdb 24237.00 0.00 383.80 0.00 96.15 0.00 513.07 0.81 2.10 2.10 0.00 1.59 60.90
  1466. sda 24237.00 0.00 384.00 0.00 96.23 0.00 513.20 0.63 1.64 1.64 0.00 1.38 53.00
  1467. sdd 24237.60 0.00 383.40 0.00 96.25 0.00 514.14 0.79 2.05 2.05 0.00 1.58 60.70
  1468. sdc 24236.60 0.00 384.40 0.00 96.20 0.00 512.53 0.64 1.66 1.66 0.00 1.41 54.02
  1469. sde 24236.60 0.00 384.40 0.00 96.23 0.00 512.67 0.71 1.86 1.86 0.00 1.49 57.42
  1470. sdf 24230.60 0.00 384.20 0.00 96.25 0.00 513.07 1.16 3.03 3.03 0.00 1.97 75.50
  1471. sdg 24237.40 0.00 383.80 0.00 96.30 0.00 513.87 0.78 2.02 2.02 0.00 1.55 59.58
  1472. md0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  1473. md2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  1474. dm-0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  1475. dm-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  1476. md1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  1477. md3 0.00 0.00 172343.40 0.00 673.22 0.00 8.00 0.00 0.00 0.00 0.00 0.00 0.00
  1478. dm-2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  1479. dm-3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  1480. dm-4 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  1481. dm-5 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  1482.  
  1483.  
  1484. /dev/md3 640k sequential write, O_DIRECT, stripe_cache_size: 32768
  1485. ===============================================================================================================================
  1486. [root@storage ~]# fio --name=write --ioengine=libaio --bs=640k --direct=1 --rw=write --iodepth=1 --size=28G --filename=/dev/md3
  1487. write: (g=0): rw=write, bs=640K-640K/640K-640K/640K-640K, ioengine=libaio, iodepth=1
  1488. fio-2.2.8
  1489. Starting 1 process
  1490. Jobs: 1 (f=1): [W(1)] [100.0% done] [0KB/328.5MB/0KB /s] [0/525/0 iops] [eta 00m:00s]
  1491. write: (groupid=0, jobs=1): err= 0: pid=3597: Mon Oct 12 05:06:38 2015
  1492. write: io=28673MB, bw=382778KB/s, iops=598, runt= 76704msec
  1493. slat (usec): min=263, max=30318, avg=291.86, stdev=140.62
  1494. clat (usec): min=896, max=62571, avg=1375.69, stdev=823.01
  1495. lat (usec): min=1189, max=62871, avg=1668.13, stdev=834.94
  1496. clat percentiles (usec):
  1497. | 1.00th=[ 932], 5.00th=[ 956], 10.00th=[ 980], 20.00th=[ 996],
  1498. | 30.00th=[ 1004], 40.00th=[ 1004], 50.00th=[ 1020], 60.00th=[ 1032],
  1499. | 70.00th=[ 1320], 80.00th=[ 1992], 90.00th=[ 2320], 95.00th=[ 2576],
  1500. | 99.00th=[ 2928], 99.50th=[ 3056], 99.90th=[ 7584], 99.95th=[11200],
  1501. | 99.99th=[30592]
  1502. bw (KB /s): min=229120, max=405760, per=100.00%, avg=383156.40, stdev=23753.28
  1503. lat (usec) : 1000=29.44%
  1504. lat (msec) : 2=50.74%, 4=19.66%, 10=0.09%, 20=0.04%, 50=0.03%
  1505. lat (msec) : 100=0.01%
  1506. cpu : usr=2.40%, sys=18.73%, ctx=45878, majf=0, minf=27
  1507. IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  1508. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  1509. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  1510. issued : total=r=0/w=45876/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
  1511. latency : target=0, window=0, percentile=100.00%, depth=1
  1512.  
  1513. Run status group 0 (all jobs):
  1514. WRITE: io=28673MB, aggrb=382778KB/s, minb=382778KB/s, maxb=382778KB/s, mint=76704msec, maxt=76704msec
  1515.  
  1516. Disk stats (read/write):
  1517. md3: ios=91/228850, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=22/45888, aggrmerge=0/1422156, aggrticks=3/27322, aggrin_queue=27319, aggrutil=35.72%
  1518. sda: ios=2/45893, merge=0/1422158, ticks=6/27335, in_queue=27336, util=35.35%
  1519. sdb: ios=72/45887, merge=0/1422156, ticks=4/27699, in_queue=27697, util=35.72%
  1520. sdc: ios=17/45888, merge=0/1422156, ticks=1/26824, in_queue=26821, util=34.66%
  1521. sdd: ios=9/45888, merge=0/1422156, ticks=12/27157, in_queue=27160, util=35.06%
  1522. sde: ios=12/45885, merge=0/1422156, ticks=1/27519, in_queue=27514, util=35.52%
  1523. sdf: ios=46/45886, merge=0/1422156, ticks=3/27434, in_queue=27433, util=35.51%
  1524. sdg: ios=1/45890, merge=0/1422158, ticks=0/27289, in_queue=27278, util=35.27%
  1525.  
  1526.  
  1527. iostat
  1528. ------
  1529. avg-cpu: %user %nice %system %iowait %steal %idle
  1530. 0.40 0.00 8.34 0.00 0.00 91.26
  1531.  
  1532. Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
  1533. sdb 0.00 18724.00 0.00 603.80 0.00 75.48 256.00 0.36 0.59 0.00 0.59 0.59 35.56
  1534. sda 0.00 18724.00 0.00 603.80 0.00 75.48 256.00 0.35 0.59 0.00 0.59 0.59 35.38
  1535. sdd 0.00 18724.00 0.00 603.80 0.00 75.48 256.00 0.35 0.57 0.00 0.57 0.57 34.58
  1536. sdc 0.00 18724.00 0.00 604.00 0.00 75.50 256.00 0.35 0.58 0.00 0.58 0.58 35.18
  1537. sde 0.00 18724.00 0.00 603.80 0.00 75.48 256.00 0.35 0.59 0.00 0.59 0.59 35.42
  1538. sdf 0.00 18721.40 0.00 604.00 0.00 75.50 256.00 0.35 0.58 0.00 0.58 0.58 35.22
  1539. sdg 0.00 18720.80 0.00 604.00 0.00 75.50 256.00 0.36 0.59 0.00 0.59 0.59 35.56
  1540. md0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  1541. md2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  1542. dm-0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  1543. dm-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  1544. md1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  1545. md3 0.00 0.00 0.00 3019.00 0.00 377.38 256.00 0.00 0.00 0.00 0.00 0.00 0.00
  1546.  
  1547.  
  1548. /dev/md3 1280k sequential write, O_DIRECT, stripe_cache_size: 32768
  1549. ===============================================================================================================================
  1550. [root@storage ~]# fio --name=write --ioengine=libaio --bs=1280k --direct=1 --rw=write --iodepth=1 --size=28G --filename=/dev/md3
  1551. write: (g=0): rw=write, bs=1280K-1280K/1280K-1280K/1280K-1280K, ioengine=libaio, iodepth=1
  1552. fio-2.2.8
  1553. Starting 1 process
  1554. Jobs: 1 (f=1): [W(1)] [100.0% done] [0KB/394.7MB/0KB /s] [0/315/0 iops] [eta 00m:00s]
  1555. write: (groupid=0, jobs=1): err= 0: pid=3611: Mon Oct 12 05:10:52 2015
  1556. write: io=28673MB, bw=405909KB/s, iops=317, runt= 72333msec
  1557. slat (usec): min=499, max=25857, avg=568.37, stdev=168.78
  1558. clat (msec): min=1, max=37, avg= 2.58, stdev= 1.12
  1559. lat (msec): min=2, max=37, avg= 3.15, stdev= 1.13
  1560. clat percentiles (usec):
  1561. | 1.00th=[ 1880], 5.00th=[ 1896], 10.00th=[ 1912], 20.00th=[ 2096],
  1562. | 30.00th=[ 2128], 40.00th=[ 2128], 50.00th=[ 2288], 60.00th=[ 2576],
  1563. | 70.00th=[ 2800], 80.00th=[ 3024], 90.00th=[ 3344], 95.00th=[ 3600],
  1564. | 99.00th=[ 4320], 99.50th=[ 8384], 99.90th=[15168], 99.95th=[25728],
  1565. | 99.99th=[35584]
  1566. bw (KB /s): min=206946, max=470099, per=100.00%, avg=406344.03, stdev=41260.68
  1567. lat (msec) : 2=15.55%, 4=82.61%, 10=1.48%, 20=0.27%, 50=0.09%
  1568. cpu : usr=1.72%, sys=17.96%, ctx=22941, majf=0, minf=27
  1569. IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  1570. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  1571. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  1572. issued : total=r=0/w=22938/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
  1573. latency : target=0, window=0, percentile=100.00%, depth=1
  1574.  
  1575. Run status group 0 (all jobs):
  1576. WRITE: io=28673MB, aggrb=405909KB/s, minb=405909KB/s, maxb=405909KB/s, mint=72333msec, maxt=72333msec
  1577.  
  1578. Disk stats (read/write):
  1579. md3: ios=91/229180, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=33/22949, aggrmerge=0/1445094, aggrticks=73/28346, aggrin_queue=28414, aggrutil=39.06%
  1580. sda: ios=12/22948, merge=0/1445094, ticks=106/28312, in_queue=28414, util=38.90%
  1581. sdb: ios=78/22950, merge=0/1445094, ticks=24/28515, in_queue=28538, util=39.06%
  1582. sdc: ios=32/22949, merge=0/1445094, ticks=194/27944, in_queue=28134, util=38.32%
  1583. sdd: ios=16/22950, merge=0/1445095, ticks=50/28328, in_queue=28378, util=38.88%
  1584. sde: ios=17/22949, merge=0/1445094, ticks=38/28463, in_queue=28500, util=39.06%
  1585. sdf: ios=74/22949, merge=0/1445095, ticks=57/28471, in_queue=28523, util=39.02%
  1586. sdg: ios=8/22948, merge=0/1445094, ticks=48/28391, in_queue=28414, util=38.83%
  1587.  
  1588.  
  1589. iostat
  1590. ------
  1591. avg-cpu: %user %nice %system %iowait %steal %idle
  1592. 0.30 0.00 8.10 0.00 0.00 91.60
  1593.  
  1594. Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
  1595. sdb 0.00 20638.80 0.00 328.00 0.00 81.90 511.38 0.41 1.24 0.00 1.24 1.20 39.40
  1596. sda 0.00 20638.80 0.00 328.20 0.00 81.90 511.07 0.40 1.23 0.00 1.23 1.19 39.18
  1597. sdd 0.00 20639.00 0.00 328.20 0.00 81.90 511.07 0.41 1.24 0.00 1.24 1.21 39.64
  1598. sdc 0.00 20638.80 0.00 328.20 0.00 81.90 511.07 0.41 1.24 0.00 1.24 1.18 38.86
  1599. sde 0.00 20638.80 0.00 328.40 0.00 81.90 510.76 0.41 1.24 0.00 1.24 1.21 39.74
  1600. sdf 0.00 20639.00 0.00 328.60 0.00 81.95 510.77 0.42 1.27 0.00 1.27 1.22 40.16
  1601. sdg 0.00 20638.80 0.00 328.20 0.00 81.90 511.07 0.41 1.27 0.00 1.27 1.22 40.16
  1602. md0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  1603. md2 0.00 0.00 0.00 1.00 0.00 0.00 8.00 0.00 0.00 0.00 0.00 0.00 0.00
  1604. dm-0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  1605. dm-1 0.00 0.00 0.00 1.00 0.00 0.00 8.00 0.01 14.20 0.00 14.20 13.80 1.38
  1606. md1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  1607. md3 0.00 0.00 0.00 3278.00 0.00 409.75 256.00 0.00 0.00 0.00 0.00 0.00 0.00
  1608.  
  1609. /dev/md3 1280k sequential write, buffered, stripe_cache_size: 32768
  1610. ===============================================================================================================================
  1611. [root@storage ~]# fio --name=write --ioengine=libaio --bs=1280k --direct=0 --rw=write --iodepth=1 --size=28G --filename=/dev/md3
  1612. write: (g=0): rw=write, bs=1280K-1280K/1280K-1280K/1280K-1280K, ioengine=libaio, iodepth=1
  1613. fio-2.2.8
  1614. Starting 1 process
  1615. Jobs: 1 (f=1): [W(1)] [100.0% done] [0KB/0KB/0KB /s] [0/0/0 iops] [eta 00m:00s]
  1616. write: (groupid=0, jobs=1): err= 0: pid=3632: Mon Oct 12 05:12:57 2015
  1617. write: io=28673MB, bw=730528KB/s, iops=570, runt= 40191msec
  1618. slat (usec): min=611, max=29731, avg=1746.97, stdev=2324.38
  1619. clat (usec): min=0, max=131, avg= 1.69, stdev= 1.68
  1620. lat (usec): min=612, max=29745, avg=1749.55, stdev=2325.47
  1621. clat percentiles (usec):
  1622. | 1.00th=[ 1], 5.00th=[ 1], 10.00th=[ 1], 20.00th=[ 1],
  1623. | 30.00th=[ 1], 40.00th=[ 1], 50.00th=[ 1], 60.00th=[ 2],
  1624. | 70.00th=[ 2], 80.00th=[ 2], 90.00th=[ 3], 95.00th=[ 4],
  1625. | 99.00th=[ 5], 99.50th=[ 5], 99.90th=[ 11], 99.95th=[ 16],
  1626. | 99.99th=[ 83]
  1627. bw (KB /s): min=650240, max=1768960, per=100.00%, avg=731268.75, stdev=151261.54
  1628. lat (usec) : 2=58.15%, 4=34.15%, 10=7.58%, 20=0.10%, 50=0.01%
  1629. lat (usec) : 100=0.01%, 250=0.01%
  1630. cpu : usr=2.86%, sys=50.77%, ctx=2688, majf=0, minf=26
  1631. IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  1632. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  1633. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  1634. issued : total=r=0/w=22938/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
  1635. latency : target=0, window=0, percentile=100.00%, depth=1
  1636.  
  1637. Run status group 0 (all jobs):
  1638. WRITE: io=28673MB, aggrb=730527KB/s, minb=730527KB/s, maxb=730527KB/s, mint=40191msec, maxt=40191msec
  1639.  
  1640. Disk stats (read/write):
  1641. md3: ios=29/6975590, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=24/11598, aggrmerge=6/1455760, aggrticks=2737/2088891, aggrin_queue=2096765, aggrutil=95.02%
  1642. sda: ios=7/11589, merge=1/1455772, ticks=431/1967388, in_queue=1973389, util=93.63%
  1643. sdb: ios=52/11599, merge=10/1455762, ticks=9745/2091525, in_queue=2106446, util=94.54%
  1644. sdc: ios=24/11622, merge=1/1455740, ticks=736/1793522, in_queue=1799870, util=93.61%
  1645. sdd: ios=16/11593, merge=10/1455763, ticks=1247/2024760, in_queue=2031207, util=94.13%
  1646. sde: ios=16/11601, merge=10/1455758, ticks=1662/1912224, in_queue=1918867, util=93.75%
  1647. sdf: ios=48/11591, merge=10/1455765, ticks=3621/2325616, in_queue=2334441, util=95.02%
  1648. sdg: ios=5/11596, merge=0/1455760, ticks=1719/2507208, in_queue=2513136, util=94.69%
  1649.  
  1650. iostat
  1651. ------
  1652. avg-cpu: %user %nice %system %iowait %steal %idle
  1653. 0.50 0.00 36.37 7.84 0.00 55.28
  1654.  
  1655. Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
  1656. sdb 0.00 34520.40 0.00 269.80 0.00 134.90 1024.00 44.92 168.41 0.00 168.41 3.67 99.08
  1657. sda 0.00 34519.40 0.00 269.40 0.00 134.70 1024.00 50.75 190.34 0.00 190.34 3.71 99.86
  1658. sdd 0.00 34518.60 0.00 267.80 0.00 133.90 1024.00 48.17 178.99 0.00 178.99 3.71 99.42
  1659. sdc 0.00 34512.40 0.00 271.60 0.00 135.80 1024.00 41.60 156.55 0.00 156.55 3.65 99.20
  1660. sde 0.00 34511.60 0.00 270.60 0.00 135.30 1024.00 46.67 175.15 0.00 175.15 3.66 98.96
  1661. sdf 0.00 34510.40 0.00 268.20 0.00 134.10 1024.00 51.21 190.98 0.00 190.98 3.70 99.26
  1662. sdg 0.00 34510.40 0.00 273.80 0.00 136.90 1024.00 68.30 256.06 0.00 256.06 3.65 100.00
  1663. md0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  1664. md2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  1665. dm-0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  1666. dm-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  1667. md1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
  1668. md3 0.00 0.00 0.00 167424.00 0.00 654.00 8.00 0.00 0.00 0.00 0.00 0.00 0.00
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement
Public Pastes
Advertisement
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!