API tools faq
paste
galileo_pkm

5 disk RAID5, 64K chunk

galileo_pkm
Nov 8th, 2024
45
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 38.78 KB | None | 0 0
raw download clone embed print report
  1.  
  2. 5 disk RAID5, 64K chunk
  3.  
  4. Summary
  5.  
  6. Test BW IOPS
  7. bitmap internal 64M 700KiB/s 174
  8. bitmap internal 128M 702KiB/s 175
  9. bitmap internal 512M 1142KiB/s 285
  10. bitmap internal 1024M 40.4MiB/s 10.3k
  11. bitmap internal 2G 66.5MiB/s 17.0k
  12. bitmap external 64M 67.8MiB/s 17.3k
  13. bitmap external 1024M 76.5MiB/s 19.6k
  14. bitmap none 80.6MiB/s 20.6k
  15. Single disk 1K 54.1MiB/s 55.4k
  16. Single disk 4K 269MiB/s 68.8k
  17.  
  18.  
  19.  
  20.  
  21. AlmaLinux release 9.4 (Seafoam Ocelot)
  22. 5.14.0-427.20.1.el9_4
  23.  
  24.  
  25. nvme list
  26. Node Generic SN Model Namespace Usage Format FW Rev
  27. --------------------- --------------------- -------------------- ---------------------------------------- ---------- -------------------------- ---------------- --------
  28. /dev/nvme0n1 /dev/ng0n1 1460A0F9TSTJ Dell DC NVMe CD8 U.2 960GB 0x1 122.33 GB / 960.20 GB 512 B + 0 B 2.0.0
  29. /dev/nvme1n1 /dev/ng1n1 S6WRNJ0WA04045P Samsung SSD 980 PRO with Heatsink 2TB 0x1 0.00 B / 2.00 TB 512 B + 0 B 5B2QGXA7
  30. /dev/nvme2n1 /dev/ng2n1 S6WRNJ0WA04048B Samsung SSD 980 PRO with Heatsink 2TB 0x1 0.00 B / 2.00 TB 512 B + 0 B 5B2QGXA7
  31. /dev/nvme3n1 /dev/ng3n1 S6WRNJ0W810396H Samsung SSD 980 PRO with Heatsink 2TB 0x1 0.00 B / 2.00 TB 512 B + 0 B 5B2QGXA7
  32. /dev/nvme4n1 /dev/ng4n1 S6WRNJ0W808149N Samsung SSD 980 PRO with Heatsink 2TB 0x1 0.00 B / 2.00 TB 512 B + 0 B 5B2QGXA7
  33. /dev/nvme5n1 /dev/ng5n1 S6WRNJ0WA04043Z Samsung SSD 980 PRO with Heatsink 2TB 0x1 0.00 B / 2.00 TB 512 B + 0 B 5B2QGXA7
  34. /dev/nvme6n1 /dev/ng6n1 PHBT909504AH016N INTEL MEMPEK1J016GAL 0x1 14.40 GB / 14.40 GB 512 B + 0 B K4110420
  35. /dev/nvme7n1 /dev/ng7n1 S6WRNJ0WA04036R Samsung SSD 980 PRO with Heatsink 2TB 0x1 0.00 B / 2.00 TB 512 B + 0 B 5B2QGXA7
  36. /dev/nvme8n1 /dev/ng8n1 S6WRNJ0WA04050H Samsung SSD 980 PRO with Heatsink 2TB 0x1 0.00 B / 2.00 TB 512 B + 0 B 5B2QGXA7
  37.  
  38.  
  39.  
  40.  
  41. bitmap internal 64M
  42. ================================================================
  43. mdadm --verbose --create --assume-clean --bitmap=internal --bitmap-chunk=64M /dev/md/raid5 --name=raid5 --level=5 --chunk=64K --raid-devices=5 /dev/nvme{1..5}n1
  44.  
  45. for dev in /dev/nvme{1..5}n1; do blockdev --setra 256 $dev; done
  46. blockdev --setra 1024 /dev/md/raid5
  47.  
  48. echo 8 > /sys/block/md127/md/group_thread_cnt
  49. echo 8192 > /sys/block/md127/md/stripe_cache_size
  50.  
  51.  
  52. fio --filename=/dev/md/raid5 --direct=1 --rw=randwrite --bs=4k --ioengine=libaio --iodepth=1 --runtime=60 --numjobs=1 --group_reporting --time_based --name=Raid5
  53.  
  54. Raid5: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
  55. fio-3.35
  56. Starting 1 process
  57. Jobs: 1 (f=1): [w(1)][100.0%][w=716KiB/s][w=179 IOPS][eta 00m:00s]
  58. Raid5: (groupid=0, jobs=1): err= 0: pid=4718: Sun Jun 30 02:18:30 2024
  59. write: IOPS=174, BW=700KiB/s (717kB/s)(41.0MiB/60005msec); 0 zone resets
  60. slat (usec): min=4, max=18062, avg=11.28, stdev=176.21
  61. clat (usec): min=46, max=13308, avg=5700.08, stdev=1194.59
  62. lat (usec): min=53, max=22717, avg=5711.36, stdev=1206.03
  63. clat percentiles (usec):
  64. | 1.00th=[ 51], 5.00th=[ 5800], 10.00th=[ 5800], 20.00th=[ 5866],
  65. | 30.00th=[ 5866], 40.00th=[ 5866], 50.00th=[ 5866], 60.00th=[ 5932],
  66. | 70.00th=[ 5932], 80.00th=[ 5932], 90.00th=[ 5932], 95.00th=[ 5997],
  67. | 99.00th=[ 6194], 99.50th=[ 8586], 99.90th=[10290], 99.95th=[13042],
  68. | 99.99th=[13042]
  69. bw ( KiB/s): min= 608, max= 752, per=100.00%, avg=700.03, stdev=20.93, samples=119
  70. iops : min= 152, max= 188, avg=175.01, stdev= 5.23, samples=119
  71. lat (usec) : 50=0.68%, 100=3.23%
  72. lat (msec) : 10=95.99%, 20=0.10%
  73. cpu : usr=0.08%, sys=0.24%, ctx=10503, majf=0, minf=8
  74. IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  75. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  76. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  77. issued rwts: total=0,10499,0,0 short=0,0,0,0 dropped=0,0,0,0
  78. latency : target=0, window=0, percentile=100.00%, depth=1
  79.  
  80. Run status group 0 (all jobs):
  81. WRITE: bw=700KiB/s (717kB/s), 700KiB/s-700KiB/s (717kB/s-717kB/s), io=41.0MiB (43.0MB), run=60005-60005msec
  82.  
  83.  
  84. avg-cpu: %user %nice %system %iowait %steal %idle
  85. 0.00 0.00 0.07 0.00 0.00 99.93
  86.  
  87. Device r/s rMB/s rrqm/s %rrqm r_await rareq-sz w/s wMB/s wrqm/s %wrqm w_await wareq-sz f/s f_await aqu-sz %util
  88. md127 0.00 0.00 0.00 0.00 0.00 0.00 175.47 0.69 0.00 0.00 5.68 4.00 0.00 0.00 1.00 100.00
  89. nvme1n1 69.40 0.27 0.00 0.00 0.01 4.00 237.80 0.93 0.00 0.00 3.55 4.00 168.47 0.81 0.98 95.59
  90. nvme2n1 69.20 0.27 0.00 0.00 0.01 4.00 237.60 0.93 0.00 0.00 3.55 4.00 168.47 0.81 0.98 95.61
  91. nvme3n1 72.20 0.28 0.00 0.00 0.01 4.00 240.60 0.94 0.00 0.00 3.51 4.00 168.47 0.83 0.98 95.29
  92. nvme4n1 68.07 0.27 0.00 0.00 0.02 4.00 236.53 0.92 0.00 0.00 3.57 4.00 168.47 0.81 0.98 95.65
  93. nvme5n1 72.07 0.28 0.00 0.00 0.02 4.00 240.53 0.94 0.00 0.00 3.52 4.00 168.47 0.83 0.99 95.31
  94.  
  95.  
  96. mdadm -X /dev/nvme1n1
  97. Filename : /dev/nvme1n1
  98. Magic : 6d746962
  99. Version : 4
  100. UUID : 77fa1a1b:2f0dd646:adc85c8e:985513a8
  101. Events : 3
  102. Events Cleared : 3
  103. State : OK
  104. Chunksize : 64 MB
  105. Daemon : 5s flush period
  106. Write Mode : Normal
  107. Sync Size : 1953382464 (1862.89 GiB 2000.26 GB)
  108. Bitmap : 29807 bits (chunks), 1517 dirty (5.1%)
  109.  
  110.  
  111. bitmap internal 128M
  112. ================================================================
  113. for dev in /dev/nvme{1..5}n1; do nvme format --force $dev; done
  114.  
  115. mdadm --verbose --create --assume-clean --bitmap=internal --bitmap-chunk=128M /dev/md/raid5 --name=raid5 --level=5 --chunk=64K --raid-devices=5 /dev/nvme{1..5}n1
  116.  
  117. for dev in /dev/nvme{1..5}n1; do blockdev --setra 256 $dev; done
  118. blockdev --setra 1024 /dev/md/raid5
  119.  
  120. echo 8 > /sys/block/md127/md/group_thread_cnt
  121. echo 8192 > /sys/block/md127/md/stripe_cache_size
  122.  
  123.  
  124. fio --filename=/dev/md/raid5 --direct=1 --rw=randwrite --bs=4k --ioengine=libaio --iodepth=1 --runtime=60 --numjobs=1 --group_reporting --time_based --name=Raid5
  125.  
  126. Raid5: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
  127. fio-3.35
  128. Starting 1 process
  129. Jobs: 1 (f=1): [w(1)][100.0%][w=716KiB/s][w=179 IOPS][eta 00m:00s]
  130. Raid5: (groupid=0, jobs=1): err= 0: pid=6283: Sun Jun 30 02:49:06 2024
  131. write: IOPS=175, BW=702KiB/s (719kB/s)(41.1MiB/60002msec); 0 zone resets
  132. slat (usec): min=8, max=18200, avg=16.06, stdev=177.21
  133. clat (usec): min=61, max=20048, avg=5675.78, stdev=1968.88
  134. lat (usec): min=74, max=22975, avg=5691.84, stdev=1976.14
  135. clat percentiles (usec):
  136. | 1.00th=[ 68], 5.00th=[ 73], 10.00th=[ 5866], 20.00th=[ 5932],
  137. | 30.00th=[ 5932], 40.00th=[ 5932], 50.00th=[ 5932], 60.00th=[ 5997],
  138. | 70.00th=[ 5997], 80.00th=[ 5997], 90.00th=[ 5997], 95.00th=[ 6063],
  139. | 99.00th=[14615], 99.50th=[15008], 99.90th=[16188], 99.95th=[16319],
  140. | 99.99th=[16319]
  141. bw ( KiB/s): min= 384, max= 816, per=99.97%, avg=702.12, stdev=72.52, samples=119
  142. iops : min= 96, max= 204, avg=175.53, stdev=18.13, samples=119
  143. lat (usec) : 100=7.62%, 250=0.01%
  144. lat (msec) : 10=90.80%, 20=1.56%, 50=0.01%
  145. cpu : usr=0.11%, sys=0.34%, ctx=10539, majf=0, minf=8
  146. IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  147. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  148. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  149. issued rwts: total=0,10534,0,0 short=0,0,0,0 dropped=0,0,0,0
  150. latency : target=0, window=0, percentile=100.00%, depth=1
  151.  
  152. Run status group 0 (all jobs):
  153. WRITE: bw=702KiB/s (719kB/s), 702KiB/s-702KiB/s (719kB/s-719kB/s), io=41.1MiB (43.1MB), run=60002-60002msec
  154.  
  155.  
  156.  
  157.  
  158.  
  159. avg-cpu: %user %nice %system %iowait %steal %idle
  160. 0.00 0.00 0.08 0.00 0.00 99.92
  161.  
  162. Device r/s rMB/s rrqm/s %rrqm r_await rareq-sz w/s wMB/s wrqm/s %wrqm w_await wareq-sz f/s f_await aqu-sz %util
  163. md127 0.00 0.00 0.00 0.00 0.00 0.00 173.73 0.68 0.00 0.00 5.73 4.00 0.00 0.00 1.00 99.99
  164. nvme1n1 65.87 0.26 0.00 0.00 0.01 4.00 226.07 0.65 0.00 0.00 3.60 2.94 160.20 0.81 0.94 92.46
  165. nvme2n1 71.33 0.28 0.00 0.00 0.02 4.00 231.53 0.67 0.00 0.00 3.50 2.96 160.27 0.84 0.95 91.79
  166. nvme3n1 68.60 0.27 0.00 0.00 0.02 4.00 228.80 0.66 0.00 0.00 3.68 2.95 160.27 0.93 0.99 94.37
  167. nvme4n1 68.87 0.27 0.00 0.00 0.02 4.00 229.07 0.66 0.00 0.00 3.52 2.95 160.20 0.81 0.94 91.59
  168. nvme5n1 72.80 0.28 0.00 0.00 0.02 4.00 233.00 0.68 0.00 0.00 3.53 2.97 160.27 0.87 0.96 92.29
  169.  
  170.  
  171. mdadm -X /dev/nvme1n1
  172. Filename : /dev/nvme1n1
  173. Magic : 6d746962
  174. Version : 4
  175. UUID : 93fdcd4b:ae61a1f8:4d809242:2cd4a4c7
  176. Events : 3
  177. Events Cleared : 3
  178. State : OK
  179. Chunksize : 128 MB
  180. Daemon : 5s flush period
  181. Write Mode : Normal
  182. Sync Size : 1953382464 (1862.89 GiB 2000.26 GB)
  183. Bitmap : 14904 bits (chunks), 1617 dirty (10.8%)
  184.  
  185.  
  186.  
  187.  
  188.  
  189. bitmap internal 512M
  190. ================================================================
  191. for dev in /dev/nvme{1..5}n1; do nvme format --force $dev; done
  192.  
  193. mdadm --verbose --create --assume-clean --bitmap=internal --bitmap-chunk=512M /dev/md/raid5 --name=raid5 --level=5 --chunk=64K --raid-devices=5 /dev/nvme{1..5}n1
  194.  
  195. for dev in /dev/nvme{1..5}n1; do blockdev --setra 256 $dev; done
  196. blockdev --setra 1024 /dev/md/raid5
  197.  
  198. echo 8 > /sys/block/md127/md/group_thread_cnt
  199. echo 8192 > /sys/block/md127/md/stripe_cache_size
  200.  
  201.  
  202. fio --filename=/dev/md/raid5 --direct=1 --rw=randwrite --bs=4k --ioengine=libaio --iodepth=1 --runtime=60 --numjobs=1 --group_reporting --time_based --name=Raid5
  203.  
  204.  
  205. Raid5: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
  206. fio-3.35
  207. Starting 1 process
  208. Jobs: 1 (f=1): [w(1)][100.0%][w=1232KiB/s][w=308 IOPS][eta 00m:00s]
  209. Raid5: (groupid=0, jobs=1): err= 0: pid=6661: Sun Jun 30 02:58:11 2024
  210. write: IOPS=285, BW=1142KiB/s (1169kB/s)(66.9MiB/60006msec); 0 zone resets
  211. slat (usec): min=4, max=18130, avg=10.80, stdev=138.54
  212. clat (usec): min=42, max=13261, avg=3490.08, stdev=2945.95
  213. lat (usec): min=50, max=22827, avg=3500.88, stdev=2949.63
  214. clat percentiles (usec):
  215. | 1.00th=[ 49], 5.00th=[ 51], 10.00th=[ 52], 20.00th=[ 55],
  216. | 30.00th=[ 58], 40.00th=[ 72], 50.00th=[ 5866], 60.00th=[ 5932],
  217. | 70.00th=[ 5932], 80.00th=[ 5932], 90.00th=[ 5997], 95.00th=[ 5997],
  218. | 99.00th=[ 6128], 99.50th=[ 8586], 99.90th=[ 9896], 99.95th=[13042],
  219. | 99.99th=[13042]
  220. bw ( KiB/s): min= 600, max= 1648, per=99.68%, avg=1138.89, stdev=188.44, samples=119
  221. iops : min= 150, max= 412, avg=284.72, stdev=47.11, samples=119
  222. lat (usec) : 50=3.41%, 100=38.62%, 250=0.04%, 500=0.03%
  223. lat (msec) : 10=57.83%, 20=0.07%
  224. cpu : usr=0.09%, sys=0.40%, ctx=17130, majf=0, minf=9
  225. IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  226. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  227. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  228. issued rwts: total=0,17127,0,0 short=0,0,0,0 dropped=0,0,0,0
  229. latency : target=0, window=0, percentile=100.00%, depth=1
  230.  
  231. Run status group 0 (all jobs):
  232. WRITE: bw=1142KiB/s (1169kB/s), 1142KiB/s-1142KiB/s (1169kB/s-1169kB/s), io=66.9MiB (70.2MB), run=60006-60006msec
  233.  
  234.  
  235. avg-cpu: %user %nice %system %iowait %steal %idle
  236. 0.00 0.00 0.10 0.00 0.00 99.90
  237.  
  238. Device r/s rMB/s rrqm/s %rrqm r_await rareq-sz w/s wMB/s wrqm/s %wrqm w_await wareq-sz f/s f_await aqu-sz %util
  239. md127 0.00 0.00 0.00 0.00 0.00 0.00 307.13 1.20 0.00 0.00 3.24 4.00 0.00 0.00 1.00 100.00
  240. nvme1n1 120.47 0.47 0.00 0.00 0.01 4.00 286.07 0.63 0.00 0.00 3.03 2.26 165.60 0.99 1.03 96.58
  241. nvme2n1 123.87 0.48 0.00 0.00 0.01 4.00 289.47 0.65 0.00 0.00 3.00 2.28 165.60 1.00 1.04 96.63
  242. nvme3n1 120.87 0.47 0.00 0.00 0.01 4.00 286.47 0.63 0.00 0.00 3.02 2.27 165.60 1.00 1.03 96.39
  243. nvme4n1 125.00 0.49 0.00 0.00 0.02 4.00 290.60 0.65 0.00 0.00 3.00 2.29 165.60 1.02 1.04 96.54
  244. nvme5n1 124.07 0.48 0.00 0.00 0.02 4.00 289.67 0.65 0.00 0.00 3.01 2.28 165.60 1.03 1.04 96.59
  245.  
  246.  
  247. mdadm -X /dev/nvme1n1
  248. Filename : /dev/nvme1n1
  249. Magic : 6d746962
  250. Version : 4
  251. UUID : 17eadc76:a367542a:feb6e24e:d650576c
  252. Events : 3
  253. Events Cleared : 3
  254. State : OK
  255. Chunksize : 512 MB
  256. Daemon : 5s flush period
  257. Write Mode : Normal
  258. Sync Size : 1953382464 (1862.89 GiB 2000.26 GB)
  259. Bitmap : 3726 bits (chunks), 1977 dirty (53.1%)
  260.  
  261.  
  262. bitmap internal 1024M
  263. ================================================================
  264. for dev in /dev/nvme{1..5}n1; do nvme format --force $dev; done
  265.  
  266. mdadm --verbose --create --assume-clean --bitmap=internal --bitmap-chunk=1024M /dev/md/raid5 --name=raid5 --level=5 --chunk=64K --raid-devices=5 /dev/nvme{1..5}n1
  267.  
  268. for dev in /dev/nvme{1..5}n1; do blockdev --setra 256 $dev; done
  269. blockdev --setra 1024 /dev/md/raid5
  270.  
  271. echo 8 > /sys/block/md127/md/group_thread_cnt
  272. echo 8192 > /sys/block/md127/md/stripe_cache_size
  273.  
  274.  
  275. fio --filename=/dev/md/raid5 --direct=1 --rw=randwrite --bs=4k --ioengine=libaio --iodepth=1 --runtime=60 --numjobs=1 --group_reporting --time_based --name=Raid5
  276.  
  277. fio-3.35
  278. Starting 1 process
  279. Jobs: 1 (f=1): [w(1)][100.0%][w=51.0MiB/s][w=13.1k IOPS][eta 00m:00s]
  280. Raid5: (groupid=0, jobs=1): err= 0: pid=7120: Sun Jun 30 03:08:12 2024
  281. write: IOPS=10.3k, BW=40.4MiB/s (42.4MB/s)(2425MiB/60001msec); 0 zone resets
  282. slat (usec): min=6, max=18135, avg= 8.93, stdev=23.41
  283. clat (usec): min=3, max=10459, avg=86.97, stdev=342.95
  284. lat (usec): min=63, max=22927, avg=95.90, stdev=344.33
  285. clat percentiles (usec):
  286. | 1.00th=[ 62], 5.00th=[ 63], 10.00th=[ 64], 20.00th=[ 65],
  287. | 30.00th=[ 65], 40.00th=[ 66], 50.00th=[ 67], 60.00th=[ 67],
  288. | 70.00th=[ 68], 80.00th=[ 69], 90.00th=[ 70], 95.00th=[ 74],
  289. | 99.00th=[ 133], 99.50th=[ 155], 99.90th=[ 5997], 99.95th=[ 5997],
  290. | 99.99th=[ 6063]
  291. bw ( KiB/s): min= 616, max=52968, per=99.80%, avg=41305.95, stdev=20465.79, samples=119
  292. iops : min= 154, max=13242, avg=10326.47, stdev=5116.44, samples=119
  293. lat (usec) : 4=0.01%, 10=0.01%, 20=0.01%, 100=98.64%, 250=1.00%
  294. lat (usec) : 500=0.01%, 750=0.01%, 1000=0.01%
  295. lat (msec) : 2=0.01%, 4=0.01%, 10=0.33%, 20=0.01%
  296. cpu : usr=1.89%, sys=12.74%, ctx=620837, majf=0, minf=170751
  297. IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  298. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  299. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  300. issued rwts: total=0,620822,0,0 short=0,0,0,0 dropped=0,0,0,0
  301. latency : target=0, window=0, percentile=100.00%, depth=1
  302.  
  303. Run status group 0 (all jobs):
  304. WRITE: bw=40.4MiB/s (42.4MB/s), 40.4MiB/s-40.4MiB/s (42.4MB/s-42.4MB/s), io=2425MiB (2543MB), run=60001-60001msec
  305.  
  306.  
  307. avg-cpu: %user %nice %system %iowait %steal %idle
  308. 0.04 0.00 1.27 0.00 0.00 98.70
  309.  
  310. Device r/s rMB/s rrqm/s %rrqm r_await rareq-sz w/s wMB/s wrqm/s %wrqm w_await wareq-sz f/s f_await aqu-sz %util
  311. md127 0.00 0.00 0.00 0.00 0.00 0.00 18216.93 71.16 0.00 0.00 0.05 4.00 0.00 0.00 0.88 100.00
  312. nvme1n1 7256.20 28.34 0.00 0.00 0.01 4.00 7256.27 28.34 0.00 0.00 0.01 4.00 0.00 0.00 0.19 99.71
  313. nvme2n1 7302.53 28.53 0.00 0.00 0.01 4.00 7302.53 28.53 0.00 0.00 0.01 4.00 0.00 0.00 0.17 99.73
  314. nvme3n1 7278.47 28.43 0.00 0.00 0.01 4.00 7278.53 28.43 0.00 0.00 0.01 4.00 0.00 0.00 0.18 99.57
  315. nvme4n1 7303.93 28.53 0.00 0.00 0.01 4.00 7303.93 28.53 0.00 0.00 0.01 4.00 0.00 0.00 0.21 99.74
  316. nvme5n1 7292.67 28.49 0.00 0.00 0.02 4.00 7292.60 28.49 0.00 0.00 0.02 4.00 0.00 0.00 0.22 99.69
  317.  
  318.  
  319.  
  320. mdadm -X /dev/nvme1n1
  321. Filename : /dev/nvme1n1
  322. Magic : 6d746962
  323. Version : 4
  324. UUID : a0c7ad14:50689e41:e065a166:4935a186
  325. Events : 3
  326. Events Cleared : 3
  327. State : OK
  328. Chunksize : 1 GB
  329. Daemon : 5s flush period
  330. Write Mode : Normal
  331. Sync Size : 1953382464 (1862.89 GiB 2000.26 GB)
  332. Bitmap : 1863 bits (chunks), 1863 dirty (100.0%)
  333.  
  334.  
  335. bitmap internal 2G
  336. ================================================================
  337. for dev in /dev/nvme{1..5}n1; do nvme format --force $dev; done
  338.  
  339. mdadm --verbose --create --assume-clean --bitmap=internal --bitmap-chunk=2G /dev/md/raid5 --name=raid5 --level=5 --chunk=64K --raid-devices=5 /dev/nvme{1..5}n1
  340.  
  341. for dev in /dev/nvme{1..5}n1; do blockdev --setra 256 $dev; done
  342. blockdev --setra 1024 /dev/md/raid5
  343.  
  344. echo 8 > /sys/block/md127/md/group_thread_cnt
  345. echo 8192 > /sys/block/md127/md/stripe_cache_size
  346.  
  347.  
  348. fio --filename=/dev/md/raid5 --direct=1 --rw=randwrite --bs=4k --ioengine=libaio --iodepth=1 --runtime=60 --numjobs=1 --group_reporting --time_based --name=Raid5
  349.  
  350. Raid5: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
  351. fio-3.35
  352. Starting 1 process
  353. Jobs: 1 (f=1): [w(1)][100.0%][w=74.7MiB/s][w=19.1k IOPS][eta 00m:00s]
  354. Raid5: (groupid=0, jobs=1): err= 0: pid=7696: Sun Jun 30 03:30:40 2024
  355. write: IOPS=17.0k, BW=66.5MiB/s (69.8MB/s)(3993MiB/60001msec); 0 zone resets
  356. slat (usec): min=2, max=18094, avg= 4.79, stdev=17.94
  357. clat (usec): min=5, max=10352, avg=53.37, stdev=181.29
  358. lat (usec): min=41, max=22883, avg=58.16, stdev=182.72
  359. clat percentiles (usec):
  360. | 1.00th=[ 43], 5.00th=[ 44], 10.00th=[ 45], 20.00th=[ 46],
  361. | 30.00th=[ 46], 40.00th=[ 47], 50.00th=[ 47], 60.00th=[ 48],
  362. | 70.00th=[ 48], 80.00th=[ 49], 90.00th=[ 50], 95.00th=[ 52],
  363. | 99.00th=[ 90], 99.50th=[ 126], 99.90th=[ 873], 99.95th=[ 5997],
  364. | 99.99th=[ 6063]
  365. bw ( KiB/s): min= 640, max=80168, per=99.91%, avg=68080.94, stdev=21547.29, samples=119
  366. iops : min= 160, max=20042, avg=17020.24, stdev=5386.82, samples=119
  367. lat (usec) : 10=0.01%, 50=92.06%, 100=7.10%, 250=0.73%, 500=0.01%
  368. lat (usec) : 750=0.01%, 1000=0.01%
  369. lat (msec) : 2=0.01%, 4=0.01%, 10=0.09%, 20=0.01%
  370. cpu : usr=2.22%, sys=11.55%, ctx=1022167, majf=0, minf=14
  371. IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  372. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  373. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  374. issued rwts: total=0,1022154,0,0 short=0,0,0,0 dropped=0,0,0,0
  375. latency : target=0, window=0, percentile=100.00%, depth=1
  376.  
  377. Run status group 0 (all jobs):
  378. WRITE: bw=66.5MiB/s (69.8MB/s), 66.5MiB/s-66.5MiB/s (69.8MB/s-69.8MB/s), io=3993MiB (4187MB), run=60001-60001msec
  379.  
  380.  
  381. avg-cpu: %user %nice %system %iowait %steal %idle
  382. 0.04 0.00 1.15 0.00 0.00 98.81
  383.  
  384. Device r/s rMB/s rrqm/s %rrqm r_await rareq-sz w/s wMB/s wrqm/s %wrqm w_await wareq-sz f/s f_await aqu-sz %util
  385. md127 0.00 0.00 0.00 0.00 0.00 0.00 18836.40 73.58 0.00 0.00 0.05 4.00 0.00 0.00 0.87 100.00
  386. nvme1n1 7505.27 29.32 0.00 0.00 0.01 4.00 7505.40 29.32 0.00 0.00 0.01 4.00 0.00 0.00 0.19 99.93
  387. nvme2n1 7510.00 29.34 0.00 0.00 0.01 4.00 7510.07 29.34 0.00 0.00 0.01 4.00 0.00 0.00 0.17 99.90
  388. nvme3n1 7561.40 29.54 0.00 0.00 0.01 4.00 7561.47 29.54 0.00 0.00 0.01 4.00 0.00 0.00 0.19 100.00
  389. nvme4n1 7543.07 29.47 0.00 0.00 0.01 4.00 7543.07 29.47 0.00 0.00 0.01 4.00 0.00 0.00 0.21 99.91
  390. nvme5n1 7552.73 29.50 0.00 0.00 0.01 4.00 7552.80 29.50 0.00 0.00 0.01 4.00 0.00 0.00 0.22 99.91
  391.  
  392.  
  393.  
  394. mdadm -X /dev/nvme1n1
  395. Filename : /dev/nvme1n1
  396. Magic : 6d746962
  397. Version : 4
  398. UUID : 7d8ed7e8:4c2c4b17:723a22e5:4e9b5200
  399. Events : 3
  400. Events Cleared : 3
  401. State : OK
  402. Chunksize : 2 GB
  403. Daemon : 5s flush period
  404. Write Mode : Normal
  405. Sync Size : 1953382464 (1862.89 GiB 2000.26 GB)
  406. Bitmap : 932 bits (chunks), 932 dirty (100.0%)
  407.  
  408.  
  409.  
  410.  
  411.  
  412.  
  413.  
  414.  
  415.  
  416. bitmap external 64M
  417. ================================================================
  418. for dev in /dev/nvme{1..5}n1; do nvme format --force $dev; done
  419.  
  420. mdadm --verbose --create --assume-clean --bitmap=/bitmap/bitmap.bin --bitmap-chunk=64M /dev/md/raid5 --name=raid5 --level=5 --chunk=64K --raid-devices=5 /dev/nvme{1..5}n1
  421.  
  422. for dev in /dev/nvme{1..5}n1; do blockdev --setra 256 $dev; done
  423. blockdev --setra 1024 /dev/md/raid5
  424.  
  425. echo 8 > /sys/block/md127/md/group_thread_cnt
  426. echo 8192 > /sys/block/md127/md/stripe_cache_size
  427.  
  428.  
  429. fio --filename=/dev/md/raid5 --direct=1 --rw=randwrite --bs=4k --ioengine=libaio --iodepth=1 --runtime=60 --numjobs=1 --group_reporting --time_based --name=Raid5
  430.  
  431. Raid5: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
  432. fio-3.35
  433. Starting 1 process
  434. Jobs: 1 (f=1): [w(1)][100.0%][w=67.3MiB/s][w=17.2k IOPS][eta 00m:00s]
  435. Raid5: (groupid=0, jobs=1): err= 0: pid=7912: Sun Jun 30 03:39:11 2024
  436. write: IOPS=17.3k, BW=67.8MiB/s (71.1MB/s)(4066MiB/60001msec); 0 zone resets
  437. slat (usec): min=2, max=21987, avg= 6.11, stdev=22.04
  438. clat (usec): min=3, max=8410, avg=50.79, stdev=27.03
  439. lat (usec): min=42, max=22140, avg=56.90, stdev=35.13
  440. clat percentiles (usec):
  441. | 1.00th=[ 41], 5.00th=[ 42], 10.00th=[ 44], 20.00th=[ 46],
  442. | 30.00th=[ 47], 40.00th=[ 48], 50.00th=[ 49], 60.00th=[ 50],
  443. | 70.00th=[ 51], 80.00th=[ 52], 90.00th=[ 56], 95.00th=[ 68],
  444. | 99.00th=[ 93], 99.50th=[ 124], 99.90th=[ 155], 99.95th=[ 237],
  445. | 99.99th=[ 1037]
  446. bw ( KiB/s): min=38120, max=82576, per=100.00%, avg=69402.96, stdev=7769.33, samples=119
  447. iops : min= 9530, max=20644, avg=17350.76, stdev=1942.33, samples=119
  448. lat (usec) : 4=0.01%, 20=0.01%, 50=67.87%, 100=31.34%, 250=0.76%
  449. lat (usec) : 500=0.01%, 750=0.01%, 1000=0.01%
  450. lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%
  451. cpu : usr=2.23%, sys=14.27%, ctx=1040947, majf=0, minf=233929
  452. IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  453. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  454. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  455. issued rwts: total=0,1040925,0,0 short=0,0,0,0 dropped=0,0,0,0
  456. latency : target=0, window=0, percentile=100.00%, depth=1
  457.  
  458. Run status group 0 (all jobs):
  459. WRITE: bw=67.8MiB/s (71.1MB/s), 67.8MiB/s-67.8MiB/s (71.1MB/s-71.1MB/s), io=4066MiB (4264MB), run=60001-60001msec
  460.  
  461.  
  462. avg-cpu: %user %nice %system %iowait %steal %idle
  463. 0.04 0.00 1.15 0.00 0.00 98.81
  464.  
  465. Device r/s rMB/s rrqm/s %rrqm r_await rareq-sz w/s wMB/s wrqm/s %wrqm w_await wareq-sz f/s f_await aqu-sz %util
  466. md127 0.00 0.00 0.00 0.00 0.00 0.00 18428.60 71.99 0.00 0.00 0.05 4.00 0.00 0.00 0.87 99.99
  467. nvme1n1 7399.40 28.90 0.00 0.00 0.01 4.00 7399.47 28.90 0.00 0.00 0.01 4.00 0.00 0.00 0.17 99.73
  468. nvme2n1 7361.20 28.75 0.00 0.00 0.01 4.00 7361.27 28.75 0.00 0.00 0.01 4.00 0.00 0.00 0.20 99.63
  469. nvme3n1 7376.67 28.82 0.00 0.00 0.01 4.00 7376.73 28.82 0.00 0.00 0.01 4.00 0.00 0.00 0.21 99.63
  470. nvme4n1 7367.27 28.78 0.00 0.00 0.01 4.00 7367.20 28.78 0.00 0.00 0.01 4.00 0.00 0.00 0.18 99.65
  471. nvme5n1 7352.47 28.72 0.00 0.00 0.01 4.00 7352.67 28.72 0.00 0.00 0.01 4.00 0.00 0.00 0.20 99.73
  472. nvme8n1 0.47 0.00 0.00 0.00 0.00 4.00 293.40 1.15 0.00 0.00 0.02 4.00 0.00 0.00 0.01 24.24
  473.  
  474.  
  475.  
  476. mdadm -X /bitmap/bitmap.bin
  477. Filename : /bitmap/bitmap.bin
  478. Magic : 6d746962
  479. Version : 4
  480. UUID : 1e3480e5:1f9d8b8a:53ebc6b7:279afb73
  481. Events : 3
  482. Events Cleared : 3
  483. State : OK
  484. Chunksize : 64 MB
  485. Daemon : 5s flush period
  486. Write Mode : Normal
  487. Sync Size : 1953382464 (1862.89 GiB 2000.26 GB)
  488. Bitmap : 29807 bits (chunks), 29665 dirty (99.5%)
  489.  
  490.  
  491.  
  492. bitmap external 1024M
  493. ================================================================
  494. for dev in /dev/nvme{1..5}n1; do nvme format --force $dev; done
  495.  
  496. mdadm --verbose --create --assume-clean --bitmap=/bitmap/bitmap.bin --bitmap-chunk=1024M /dev/md/raid5 --name=raid5 --level=5 --chunk=64K --raid-devices=5 /dev/nvme{1..5}n1
  497.  
  498. for dev in /dev/nvme{1..5}n1; do blockdev --setra 256 $dev; done
  499. blockdev --setra 1024 /dev/md/raid5
  500.  
  501. echo 8 > /sys/block/md127/md/group_thread_cnt
  502. echo 8192 > /sys/block/md127/md/stripe_cache_size
  503.  
  504.  
  505. fio --filename=/dev/md/raid5 --direct=1 --rw=randwrite --bs=4k --ioengine=libaio --iodepth=1 --runtime=60 --numjobs=1 --group_reporting --time_based --name=Raid5
  506.  
  507. Raid5: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
  508. fio-3.35
  509. Starting 1 process
  510. Jobs: 1 (f=1): [w(1)][100.0%][w=70.6MiB/s][w=18.1k IOPS][eta 00m:00s]
  511. Raid5: (groupid=0, jobs=1): err= 0: pid=8592: Sun Jun 30 03:54:11 2024
  512. write: IOPS=19.6k, BW=76.5MiB/s (80.2MB/s)(4590MiB/60001msec); 0 zone resets
  513. slat (usec): min=2, max=21819, avg= 4.12, stdev=20.16
  514. clat (usec): min=22, max=3706, avg=46.37, stdev=20.38
  515. lat (usec): min=40, max=21951, avg=50.49, stdev=28.81
  516. clat percentiles (usec):
  517. | 1.00th=[ 40], 5.00th=[ 41], 10.00th=[ 42], 20.00th=[ 42],
  518. | 30.00th=[ 43], 40.00th=[ 44], 50.00th=[ 45], 60.00th=[ 47],
  519. | 70.00th=[ 48], 80.00th=[ 49], 90.00th=[ 50], 95.00th=[ 52],
  520. | 99.00th=[ 86], 99.50th=[ 120], 99.90th=[ 157], 99.95th=[ 233],
  521. | 99.99th=[ 906]
  522. bw ( KiB/s): min=61616, max=84728, per=100.00%, avg=78398.66, stdev=5410.81, samples=119
  523. iops : min=15404, max=21182, avg=19599.66, stdev=1352.70, samples=119
  524. lat (usec) : 50=90.10%, 100=9.16%, 250=0.72%, 500=0.01%, 750=0.01%
  525. lat (usec) : 1000=0.01%
  526. lat (msec) : 2=0.01%, 4=0.01%
  527. cpu : usr=2.35%, sys=11.88%, ctx=1175104, majf=0, minf=11
  528. IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  529. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  530. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  531. issued rwts: total=0,1175086,0,0 short=0,0,0,0 dropped=0,0,0,0
  532. latency : target=0, window=0, percentile=100.00%, depth=1
  533.  
  534. Run status group 0 (all jobs):
  535. WRITE: bw=76.5MiB/s (80.2MB/s), 76.5MiB/s-76.5MiB/s (80.2MB/s-80.2MB/s), io=4590MiB (4813MB), run=60001-60001msec
  536.  
  537.  
  538. avg-cpu: %user %nice %system %iowait %steal %idle
  539. 0.04 0.00 1.03 0.00 0.00 98.93
  540.  
  541. Device r/s rMB/s rrqm/s %rrqm r_await rareq-sz w/s wMB/s wrqm/s %wrqm w_await wareq-sz f/s f_await aqu-sz %util
  542. md127 0.00 0.00 0.00 0.00 0.00 0.00 20758.20 81.09 0.00 0.00 0.04 4.00 0.00 0.00 0.89 100.00
  543. nvme1n1 8291.67 32.39 0.00 0.00 0.01 4.00 8291.73 32.39 0.00 0.00 0.01 4.00 0.00 0.00 0.22 99.87
  544. nvme2n1 8270.93 32.31 0.00 0.00 0.01 4.00 8271.07 32.31 0.00 0.00 0.01 4.00 0.00 0.00 0.19 99.79
  545. nvme3n1 8310.67 32.46 0.00 0.00 0.01 4.00 8310.80 32.46 0.00 0.00 0.01 4.00 0.00 0.00 0.20 99.83
  546. nvme4n1 8300.67 32.42 0.00 0.00 0.01 4.00 8300.67 32.42 0.00 0.00 0.01 4.00 0.00 0.00 0.23 99.76
  547. nvme5n1 8342.13 32.59 0.00 0.00 0.02 4.00 8342.13 32.59 0.00 0.00 0.01 4.00 0.00 0.00 0.25 99.85
  548. nvme8n1 0.33 0.00 0.00 0.00 8.40 4.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.33
  549.  
  550.  
  551. mdadm -X /bitmap/bitmap.bin
  552. Filename : /bitmap/bitmap.bin
  553. Magic : 6d746962
  554. Version : 4
  555. UUID : 30e6211d:31ac1204:e6cdadb3:9691d3ee
  556. Events : 3
  557. Events Cleared : 3
  558. State : OK
  559. Chunksize : 1 GB
  560. Daemon : 5s flush period
  561. Write Mode : Normal
  562. Sync Size : 1953382464 (1862.89 GiB 2000.26 GB)
  563. Bitmap : 1863 bits (chunks), 1863 dirty (100.0%)
  564.  
  565.  
  566.  
  567. bitmap none
  568. ================================================================
  569. for dev in /dev/nvme{1..5}n1; do nvme format --force $dev; done
  570.  
  571. mdadm --verbose --create --assume-clean --bitmap=none /dev/md/raid5 --name=raid5 --level=5 --chunk=64K --raid-devices=5 /dev/nvme{1..5}n1
  572.  
  573. for dev in /dev/nvme{1..5}n1; do blockdev --setra 256 $dev; done
  574. blockdev --setra 1024 /dev/md/raid5
  575.  
  576. echo 8 > /sys/block/md127/md/group_thread_cnt
  577. echo 8192 > /sys/block/md127/md/stripe_cache_size
  578.  
  579.  
  580. fio --filename=/dev/md/raid5 --direct=1 --rw=randwrite --bs=4k --ioengine=libaio --iodepth=1 --runtime=60 --numjobs=1 --group_reporting --time_based --name=Raid5
  581.  
  582. Raid5: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
  583. fio-3.35
  584. Starting 1 process
  585. Jobs: 1 (f=1): [w(1)][100.0%][w=82.5MiB/s][w=21.1k IOPS][eta 00m:00s]
  586. Raid5: (groupid=0, jobs=1): err= 0: pid=9158: Sun Jun 30 04:11:01 2024
  587. write: IOPS=20.6k, BW=80.6MiB/s (84.5MB/s)(4833MiB/60001msec); 0 zone resets
  588. slat (usec): min=2, max=13598, avg= 3.50, stdev=12.46
  589. clat (usec): min=4, max=3694, avg=44.31, stdev=21.60
  590. lat (usec): min=39, max=13681, avg=47.81, stdev=24.98
  591. clat percentiles (usec):
  592. | 1.00th=[ 39], 5.00th=[ 40], 10.00th=[ 41], 20.00th=[ 41],
  593. | 30.00th=[ 42], 40.00th=[ 43], 50.00th=[ 43], 60.00th=[ 44],
  594. | 70.00th=[ 45], 80.00th=[ 46], 90.00th=[ 48], 95.00th=[ 50],
  595. | 99.00th=[ 87], 99.50th=[ 117], 99.90th=[ 157], 99.95th=[ 229],
  596. | 99.99th=[ 963]
  597. bw ( KiB/s): min=74112, max=86712, per=100.00%, avg=82486.43, stdev=3696.94, samples=119
  598. iops : min=18528, max=21678, avg=20621.59, stdev=924.23, samples=119
  599. lat (usec) : 10=0.01%, 50=95.91%, 100=3.33%, 250=0.74%, 500=0.01%
  600. lat (usec) : 750=0.01%, 1000=0.01%
  601. lat (msec) : 2=0.01%, 4=0.01%
  602. cpu : usr=2.30%, sys=10.74%, ctx=1237375, majf=0, minf=179597
  603. IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  604. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  605. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  606. issued rwts: total=0,1237359,0,0 short=0,0,0,0 dropped=0,0,0,0
  607. latency : target=0, window=0, percentile=100.00%, depth=1
  608.  
  609. Run status group 0 (all jobs):
  610. WRITE: bw=80.6MiB/s (84.5MB/s), 80.6MiB/s-80.6MiB/s (84.5MB/s-84.5MB/s), io=4833MiB (5068MB), run=60001-60001msec
  611.  
  612.  
  613.  
  614.  
  615. avg-cpu: %user %nice %system %iowait %steal %idle
  616. 0.04 0.00 1.06 0.00 0.00 98.91
  617.  
  618. Device r/s rMB/s rrqm/s %rrqm r_await rareq-sz w/s wMB/s wrqm/s %wrqm w_await wareq-sz f/s f_await aqu-sz %util
  619. md127 0.00 0.00 0.00 0.00 0.00 0.00 20040.87 78.28 0.00 0.00 0.04 4.00 0.00 0.00 0.89 99.99
  620. nvme1n1 8016.80 31.32 0.00 0.00 0.01 4.00 8016.93 31.32 0.00 0.00 0.01 4.00 0.00 0.00 0.21 99.68
  621. nvme2n1 7983.20 31.18 0.00 0.00 0.01 4.00 7983.20 31.18 0.00 0.00 0.01 4.00 0.00 0.00 0.18 99.74
  622. nvme3n1 8030.07 31.37 0.00 0.00 0.01 4.00 8030.20 31.37 0.00 0.00 0.01 4.00 0.00 0.00 0.20 99.62
  623. nvme4n1 8016.40 31.31 0.00 0.00 0.01 4.00 8016.40 31.31 0.00 0.00 0.01 4.00 0.00 0.00 0.23 99.73
  624. nvme5n1 8034.87 31.39 0.00 0.00 0.02 4.00 8035.00 31.39 0.00 0.00 0.01 4.00 0.00 0.00 0.24 99.71
  625.  
  626.  
  627.  
  628.  
  629. single disk 1K RW
  630. ================================================================
  631. fio --filename=/dev/nvme8n1 --direct=1 --rw=randwrite --bs=1k --ioengine=libaio --iodepth=1 --runtime=60 --numjobs=1 --group_reporting --time_based --name=Single
  632. Single: (g=0): rw=randwrite, bs=(R) 1024B-1024B, (W) 1024B-1024B, (T) 1024B-1024B, ioengine=libaio, iodepth=1
  633. fio-3.35
  634. Starting 1 process
  635. Jobs: 1 (f=1): [w(1)][100.0%][w=54.3MiB/s][w=55.6k IOPS][eta 00m:00s]
  636. Single: (groupid=0, jobs=1): err= 0: pid=4471: Sun Jun 30 18:31:56 2024
  637. write: IOPS=55.4k, BW=54.1MiB/s (56.7MB/s)(3244MiB/60001msec); 0 zone resets
  638. slat (usec): min=2, max=2792, avg= 2.71, stdev= 2.12
  639. clat (nsec): min=651, max=8350.9k, avg=14864.41, stdev=5360.57
  640. lat (usec): min=15, max=8403, avg=17.57, stdev= 5.79
  641. clat percentiles (usec):
  642. | 1.00th=[ 15], 5.00th=[ 15], 10.00th=[ 15], 20.00th=[ 15],
  643. | 30.00th=[ 15], 40.00th=[ 15], 50.00th=[ 15], 60.00th=[ 15],
  644. | 70.00th=[ 15], 80.00th=[ 16], 90.00th=[ 16], 95.00th=[ 16],
  645. | 99.00th=[ 18], 99.50th=[ 22], 99.90th=[ 32], 99.95th=[ 33],
  646. | 99.99th=[ 206]
  647. bw ( KiB/s): min=51884, max=56778, per=100.00%, avg=55394.37, stdev=561.60, samples=119
  648. iops : min=51884, max=56778, avg=55394.44, stdev=561.62, samples=119
  649. lat (nsec) : 750=0.01%, 1000=0.01%
  650. lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=99.43%, 50=0.54%
  651. lat (usec) : 100=0.01%, 250=0.02%, 500=0.01%
  652. lat (msec) : 10=0.01%
  653. cpu : usr=3.57%, sys=16.41%, ctx=3321571, majf=0, minf=180742
  654. IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  655. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  656. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  657. issued rwts: total=0,3321653,0,0 short=0,0,0,0 dropped=0,0,0,0
  658. latency : target=0, window=0, percentile=100.00%, depth=1
  659.  
  660. Run status group 0 (all jobs):
  661. WRITE: bw=54.1MiB/s (56.7MB/s), 54.1MiB/s-54.1MiB/s (56.7MB/s-56.7MB/s), io=3244MiB (3401MB), run=60001-60001msec
  662.  
  663. Disk stats (read/write):
  664. nvme8n1: ios=0/3309968, merge=0/0, ticks=0/44637, in_queue=44638, util=99.71%
  665.  
  666.  
  667. avg-cpu: %user %nice %system %iowait %steal %idle
  668. 0.04 0.00 0.42 0.00 0.00 99.54
  669.  
  670. Device r/s rMB/s rrqm/s %rrqm r_await rareq-sz w/s wMB/s wrqm/s %wrqm w_await wareq-sz f/s f_await aqu-sz %util
  671. nvme8n1 0.00 0.00 0.00 0.00 0.00 0.00 55496.93 54.20 0.00 0.00 0.01 1.00 0.00 0.00 0.75 100.00
  672.  
  673.  
  674.  
  675.  
  676.  
  677. single disk 4K RW
  678. ================================================================
  679. blockdev --setra 256 /dev/nvme8n1
  680.  
  681. fio --filename=/dev/nvme8n1 --direct=1 --rw=randwrite --bs=4k --ioengine=libaio --iodepth=1 --runtime=60 --numjobs=1 --group_reporting --time_based --name=Single
  682.  
  683. Single: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
  684. fio-3.35
  685. Starting 1 process
  686. Jobs: 1 (f=1): [w(1)][100.0%][w=270MiB/s][w=69.0k IOPS][eta 00m:00s]
  687. Single: (groupid=0, jobs=1): err= 0: pid=4396: Sun Jun 30 18:21:52 2024
  688. write: IOPS=68.8k, BW=269MiB/s (282MB/s)(15.8GiB/60001msec); 0 zone resets
  689. slat (usec): min=2, max=796, avg= 2.45, stdev= 1.59
  690. clat (nsec): min=652, max=8343.1k, avg=11616.73, stdev=5088.99
  691. lat (usec): min=11, max=8410, avg=14.06, stdev= 5.36
  692. clat percentiles (usec):
  693. | 1.00th=[ 12], 5.00th=[ 12], 10.00th=[ 12], 20.00th=[ 12],
  694. | 30.00th=[ 12], 40.00th=[ 12], 50.00th=[ 12], 60.00th=[ 12],
  695. | 70.00th=[ 12], 80.00th=[ 12], 90.00th=[ 12], 95.00th=[ 12],
  696. | 99.00th=[ 14], 99.50th=[ 17], 99.90th=[ 28], 99.95th=[ 34],
  697. | 99.99th=[ 204]
  698. bw ( KiB/s): min=264072, max=277568, per=100.00%, avg=275629.71, stdev=1902.45, samples=119
  699. iops : min=66018, max=69392, avg=68907.43, stdev=475.55, samples=119
  700. lat (nsec) : 750=0.01%, 1000=0.01%
  701. lat (usec) : 2=0.01%, 4=0.01%, 10=0.04%, 20=99.55%, 50=0.38%
  702. lat (usec) : 100=0.01%, 250=0.02%, 1000=0.01%
  703. lat (msec) : 10=0.01%
  704. cpu : usr=5.20%, sys=21.28%, ctx=4129887, majf=0, minf=45204
  705. IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
  706. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  707. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  708. issued rwts: total=0,4130258,0,0 short=0,0,0,0 dropped=0,0,0,0
  709. latency : target=0, window=0, percentile=100.00%, depth=1
  710.  
  711. Run status group 0 (all jobs):
  712. WRITE: bw=269MiB/s (282MB/s), 269MiB/s-269MiB/s (282MB/s-282MB/s), io=15.8GiB (16.9GB), run=60001-60001msec
  713.  
  714. Disk stats (read/write):
  715. nvme8n1: ios=0/4119593, merge=0/0, ticks=0/40922, in_queue=40923, util=99.89%
  716.  
  717.  
  718. avg-cpu: %user %nice %system %iowait %steal %idle
  719. 0.08 0.00 0.57 0.00 0.00 99.35
  720.  
  721. Device r/s rMB/s rrqm/s %rrqm r_await rareq-sz w/s wMB/s wrqm/s %wrqm w_await wareq-sz f/s f_await aqu-sz %util
  722. nvme8n1 0.00 0.00 0.00 0.00 0.00 0.00 69041.33 269.69 0.00 0.00 0.01 4.00 0.00 0.00 0.68 100.00
  723.  
Advertisement
Add Comment
Please, Sign In to add comment
Public Pastes
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!