API tools faq
paste
galileo_pkm

Is NVMe RAID0 useless (performance-wise) for common workloads?

galileo_pkm
Jun 7th, 2024 (edited)
47
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 16.30 KB | None | 0 0
raw download clone embed print report
  1. Hi all
  2.  
  3. Sorry about the length of this email; I tried to include all that I found relevant.
  4. While building a new PostgreSQL server, I noticed that single disk performance is the
  5. same or better than a RAID0 4-disk array.
  6.  
  7. All benchmarks were done with: pgbench -j 4 -c 512 -P 60 -r -T 300 -b tpcb-like test
  8.  
  9. For example, single disk run:
  10.  
  11. latency average = 17.524 ms
  12. latency stddev = 6.904 ms
  13. tps = 28870
  14.  
  15.  
  16. iostat:
  17.  
  18. avg-cpu: %user %nice %system %iowait %steal %idle
  19. 26.10 0.00 21.02 2.95 0.00 49.93
  20.  
  21. | Device | r/s | rMB/s | rrqm/s | %rrqm | r_await | rareq-sz | w/s | wMB/s | wrqm/s | %wrqm | w_await | wareq-sz | f/s | f_await | aqu-sz | %util |
  22. |---------|:--------:|:------:|:------:|:-----:|:-------:|:--------:|:--------:|:------:|:------:|:-----:|:-------:|:--------:|:------:|:-------:|:------:|:-----:|
  23. | nvme0n1 | 28641.27 | 255.00 | 0.00 | 0.00 | 0.16 | 9.12 | 27665.67 | 458.09 | 0.00 | 0.00 | 0.09 | 16.96 | 251.47 | 1.69 | 7.69 | 98.08 |
  24.  
  25.  
  26.  
  27. RAID0 4 disk, 4K chunk result:
  28.  
  29. latency average = 22.269 ms
  30. latency stddev = 10.825 ms
  31. tps = 22742
  32.  
  33.  
  34. avg-cpu: %user %nice %system %iowait %steal %idle
  35. 23.63 0.00 19.63 1.53 0.00 55.21
  36.  
  37. | Device | r/s | rMB/s | rrqm/s | %rrqm | r_await | rareq-sz | w/s | wMB/s | wrqm/s | %wrqm | w_await | wareq-sz | f/s | f_await | aqu-sz | %util |
  38. |---------|:--------:|:------:|:-------:|:-----:|:-------:|:--------:|:---------:|:------:|:------:|:-----:|:-------:|:--------:|:------:|:-------:|:------:|:-----:|
  39. | md127 | 55359.93 | 216.25 | 0.00 | 0.00 | 0.09 | 4.00 | 105629.07 | 412.61 | 0.00 | 0.00 | 0.04 | 4.00 | 0.00 | 0.00 | 9.02 | 93.76 |
  40. | nvme1n1 | 12763.33 | 54.03 | 1067.47 | 7.72 | 0.08 | 4.33 | 26572.07 | 103.31 | 37.33 | 0.14 | 0.05 | 3.98 | 162.53 | 1.74 | 2.67 | 99.18 |
  41. | nvme3n1 | 12753.07 | 53.97 | 1063.87 | 7.70 | 0.08 | 4.33 | 26560.47 | 103.26 | 37.40 | 0.14 | 0.05 | 3.98 | 162.47 | 1.73 | 2.58 | 99.15 |
  42. | nvme4n1 | 12787.27 | 54.10 | 1062.80 | 7.67 | 0.09 | 4.33 | 26492.73 | 102.99 | 35.67 | 0.13 | 0.05 | 3.98 | 162.53 | 1.69 | 2.67 | 99.07 |
  43. | nvme5n1 | 12796.53 | 54.15 | 1065.60 | 7.69 | 0.09 | 4.33 | 26505.67 | 103.04 | 35.73 | 0.13 | 0.05 | 3.98 | 162.53 | 1.66 | 2.56 | 98.95 |
  44.  
  45.  
  46. BTW, if these tables are mangled in transport or by email clients, I posted this email to https://pastebin.com/raw/hyM3i12d.
  47.  
  48. With this RAID configuration, a single disk wins by a comfortable margin.
  49. PostgreSQL uses 8K I/O, so I would expect a 2x increase in performance?
  50.  
  51. I performed many different tests, tried chunk sizes from 8K to 512K, enabled poll_queues (4 queues),
  52. io_polling, partitioned the drives into 4 partitions and assembled 4 RAID arrays and combined them with
  53. LVM (inspired by this thread: https://yhbt.net/lore/all/5EAED86C53DED2479E3E145969315A2385841062@UMECHPA7B.easf.csd.disa.mil/t/#u), etc.,
  54. and nothing worked. I even tried a ZFS RAID0 zvol with XFS on top of it (performance was abysmal, ~12K tps).
  55. The best that I got was with a plain RAID0 256K chunk, ~31K tps, a 10% uplift compared to a single drive.
  56.  
  57. In the linked thread, it was suggested that this could be an issue with the lack of blk-mq support for MD devices,
  58. but I'm not sure if that applies to this use case.
  59.  
  60. As all my attempts to make this work failed, I ran a series of simple sequential read tests against a single disk and various RAID arrays,
  61. trying to rule out MD as a source of the problem. A sample of the results is below.
  62. Unfortunately, I haven't been able to reach a conclusion, hence this email.
  63.  
  64.  
  65.  
  66. System specs:
  67.  
  68. Dell PowerEdge R7525
  69. Dual AMD EPYC 7313
  70. "Samsung SSD 980 PRO with Heatsink 2TB" drives
  71. All drives under test are connected to the same processor.
  72. AlmaLinux release 9.4 (Seafoam Ocelot)
  73. Kernel 5.14.0-427.18.1.el9_4.x86_64
  74. tuned-adm profile postgresql
  75. poll_queues=4, io_poll = 1
  76.  
  77.  
  78.  
  79. Tests were done using dd_rescue
  80. buffered reads: sync; echo 3 > /proc/sys/vm/drop_caches; dd_rescue -m 64G -b 4k -B 4k /dev/md/raid0 /dev/null
  81. O_DIRECT reads: sync; echo 3 > /proc/sys/vm/drop_caches; dd_rescue -d -m 64G -b 4k -B 4k /dev/md/raid0 /dev/null
  82.  
  83. Summary of the results:
  84.  
  85. | | Read 4K | Read 64K | Read 128K | Read 256K | Read 512K |
  86. |-----------------------------------|---------|----------|-----------|-----------|-----------|
  87. | Single disk buffered | 1.39G/s | 2.27G/s | 2.29G/s | 2.31G/s | 2.32G/s |
  88. | Single disk O_DIRECT | 0.31G/s | 2.18G/s | 2.68G/s | 2.75G/s | 4G/s |
  89. | Raid 0, 4 disk 4K chunk buffered | 1G/s | 1.36G/s | 1.38G/s | 1.38G/s | 1.39G/s |
  90. | Raid 0, 4 disk 4K chunk O_DIRECT | 0.27G/s | 0.94G/s | 0.98G/s | 1.01G/s | 1.04G/s |
  91. | Raid 0, 4 disk 64K chunk buffered | 1.42G/s | 2.2G/s | 2.24G/s | 2.26G/s | 2.26G/s |
  92. | Raid 0, 4 disk 64K chunk O_DIRECT | 0.29G/s | 2.12G/s | 3.86G/s | 5.59G/s | 6.75G/s |
  93.  
  94.  
  95. First thing that stands out is that buffered reads do not scale.
  96.  
  97.  
  98. Buffered reads, single disk vs RAID0 64K chunk
  99. ===============================================================
  100.  
  101.  
  102. Single disk, buffered read 128K
  103.  
  104. avg-cpu: %user %nice %system %iowait %steal %idle
  105. 0.03 0.00 1.99 0.00 0.00 97.98
  106.  
  107. | Device | r/s | rMB/s | rrqm/s | %rrqm | r_await | rareq-sz | aqu-sz | %util |
  108. |---------|----------|---------|--------|-------|---------|----------|--------|-------|
  109. | nvme4n1 | 19874.80 | 2483.20 | 0.60 | 0.00 | 0.68 | 127.94 | 13.60 | 68.74 |
  110.  
  111.  
  112.  
  113. RAID0 64K chunk, buffered read 128K
  114.  
  115. avg-cpu: %user %nice %system %iowait %steal %idle
  116. 0.02 0.00 2.61 0.00 0.00 97.37
  117.  
  118. | Device | r/s | rMB/s | rrqm/s | %rrqm | r_await | rareq-sz | aqu-sz | %util |
  119. |---------|---------:|--------:|--------:|------:|--------:|---------:|-------:|------:|
  120. | md127 | 42779.80 | 2201.60 | 0.00 | 0.00 | 1.04 | 52.70 | 44.37 | 28.84 |
  121. | nvme0n1 | 4410.00 | 551.25 | 6287.60 | 58.78 | 1.24 | 128.00 | 5.48 | 28.76 |
  122. | nvme1n1 | 4405.00 | 550.62 | 6280.00 | 58.77 | 0.93 | 128.00 | 4.10 | 26.54 |
  123. | nvme2n1 | 4430.00 | 551.20 | 6283.00 | 58.65 | 1.20 | 127.41 | 5.33 | 28.72 |
  124. | nvme4n1 | 4407.80 | 550.77 | 6294.40 | 58.81 | 0.89 | 127.95 | 3.90 | 26.42 |
  125.  
  126.  
  127.  
  128. Maybe it's request merging?
  129.  
  130. echo 2 > /sys/block/md127/queue/nomerges
  131. for dev in /sys/block/nvme[0-3]n1/queue/nomerges; do echo 2 > $dev; done
  132.  
  133.  
  134. avg-cpu: %user %nice %system %iowait %steal %idle
  135. 0.01 0.00 2.92 0.00 0.00 97.07
  136.  
  137. | Device | r/s | rMB/s | rrqm/s | %rrqm | r_await | rareq-sz | aqu-sz | %util |
  138. |---------|:--------:|:-------:|:------:|:-----:|:-------:|:--------:|:------:|:-----:|
  139. | md127 | 42619.60 | 2169.60 | 0.00 | 0.00 | 0.71 | 52.13 | 30.14 | 28.82 |
  140. | nvme0n1 | 10641.00 | 542.40 | 0.00 | 0.00 | 0.68 | 52.20 | 7.20 | 27.90 |
  141. | nvme2n1 | 10652.20 | 542.40 | 0.00 | 0.00 | 0.63 | 52.14 | 6.68 | 27.68 |
  142. | nvme5n1 | 10659.60 | 542.40 | 0.00 | 0.00 | 0.77 | 52.10 | 8.22 | 28.70 |
  143. | nvme6n1 | 10666.80 | 542.40 | 0.00 | 0.00 | 0.74 | 52.07 | 7.92 | 28.52 |
  144.  
  145.  
  146. No change, it just prevents the 64K requests sent to the md device to be merged into
  147. 128K requests to the disks.
  148.  
  149. Why is max_sectors_kb for this RAID array limited to 64K when underlying devices are at 128?
  150.  
  151. While I'm at it, it's strange that no matter the device type, max_sectors_kb seems to be limited at 128.
  152. For example: "Dell DC NVMe CD8 U.2 960GB" (re-branded Kioxia KCD8XRUG960G), "Samsung SSD 980 PRO with Heatsink 2TB" and
  153. INTEL MEMPEK1J016GAL (old Optane M10), all vastly different devices, have the same max_sectors_kb.
  154. Also all of these devices show practically the same performance when doing 4K O_DIRECT reads.
  155. As if there is some hard limit, related to latency, somewhere?
  156.  
  157.  
  158.  
  159. RAID0 64K chunk, buffered read 512K
  160.  
  161. avg-cpu: %user %nice %system %iowait %steal %idle
  162. 0.01 0.00 2.12 0.00 0.00 97.87
  163.  
  164. | Device | r/s | rMB/s | rrqm/s | %rrqm | r_await | rareq-sz | aqu-sz | %util |
  165. |---------|:--------:|:-------:|:-------:|:-----:|:-------:|:--------:|:------:|:-----:|
  166. | md127 | 41381.20 | 2284.80 | 0.00 | 0.00 | 1.08 | 56.54 | 44.61 | 30.14 |
  167. | nvme0n1 | 4569.60 | 571.20 | 5773.00 | 55.82 | 1.37 | 128.00 | 6.27 | 30.16 |
  168. | nvme1n1 | 4569.60 | 571.20 | 5767.00 | 55.79 | 0.91 | 128.00 | 4.16 | 27.00 |
  169. | nvme2n1 | 4584.20 | 571.20 | 5768.60 | 55.72 | 1.33 | 127.59 | 6.11 | 30.04 |
  170. | nvme4n1 | 4570.40 | 571.18 | 5775.40 | 55.82 | 0.86 | 127.97 | 3.94 | 26.90 |
  171.  
  172.  
  173. 512K should be 2 full stripes and yet there is no change.
  174.  
  175. This behaviour would explain why I get no appreciable performance uplift with PostgreSQL, but I failed
  176. to find a way to change this. Are there any tunables that can help with this issue?
  177.  
  178.  
  179. I made an attempt to figure out what is going on, so I started by comparing single disk buffered reads versus O_DIRECT.
  180.  
  181.  
  182. Single disk, buffered vs O_DIRECT
  183. ===============================================================
  184.  
  185. Read 4K
  186. ---------------
  187. buffered 1.39G/s
  188.  
  189. avg-cpu: %user %nice %system %iowait %steal %idle
  190. 0.12 0.00 2.57 0.00 0.00 97.31
  191.  
  192. | Device | r/s | rMB/s | rrqm/s | %rrqm | r_await | rareq-sz | aqu-sz | %util |
  193. |---------|----------|---------|--------|-------|---------|----------|--------|-------|
  194. | nvme4n1 | 10534.40 | 1316.80 | 0.00 | 0.00 | 0.90 | 128.00 | 9.51 | 44.38 |
  195.  
  196.  
  197. O_DIRECT 0.31G/s
  198.  
  199. avg-cpu: %user %nice %system %iowait %steal %idle
  200. 0.02 0.00 0.36 0.74 0.00 98.88
  201.  
  202. | Device | r/s | rMB/s | rrqm/s | %rrqm | r_await | rareq-sz | aqu-sz | %util |
  203. |---------|----------|--------|--------|-------|---------|----------|--------|--------|
  204. | nvme4n1 | 82912.40 | 323.88 | 0.00 | 0.00 | 0.01 | 4.00 | 0.69 | 100.00 |
  205.  
  206.  
  207. Makes sense, larger request and queue size.
  208.  
  209.  
  210.  
  211. Read 64K
  212. ---------------
  213. buffered 2.27G/s
  214.  
  215. avg-cpu: %user %nice %system %iowait %steal %idle
  216. 0.02 0.00 3.13 0.00 0.00 96.85
  217.  
  218. | Device | r/s | rMB/s | rrqm/s | %rrqm | r_await | rareq-sz | aqu-sz | %util |
  219. |---------|----------|---------|--------|-------|---------|----------|--------|-------|
  220. | nvme4n1 | 16859.00 | 2107.20 | 0.60 | 0.00 | 0.71 | 127.99 | 11.96 | 59.06 |
  221.  
  222.  
  223. aqu-sz is higher (compared to the 4K result), but await is lower? How?
  224.  
  225.  
  226. O_DIRECT 2.18G/s
  227.  
  228. avg-cpu: %user %nice %system %iowait %steal %idle
  229. 0.03 0.00 0.46 1.09 0.00 98.43
  230.  
  231. | Device | r/s | rMB/s | rrqm/s | %rrqm | r_await | rareq-sz | aqu-sz | %util |
  232. |---------|----------|---------|--------|-------|---------|----------|--------|--------|
  233. | nvme3n1 | 36139.40 | 2258.71 | 0.00 | 0.00 | 0.02 | 64.00 | 0.82 | 100.02 |
  234.  
  235.  
  236. Compared to a buffered 64K read, await is 35 times lower so that's why it's almost the same result?
  237.  
  238.  
  239.  
  240. Read 256K
  241. ---------------
  242.  
  243. buffered 2.31G/s
  244.  
  245. avg-cpu: %user %nice %system %iowait %steal %idle
  246. 0.02 0.00 1.57 0.00 0.00 98.41
  247.  
  248.  
  249. | Device | r/s | rMB/s | rrqm/s | %rrqm | r_await | rareq-sz | aqu-sz | %util |
  250. |---------|----------|---------|--------|-------|---------|----------|--------|-------|
  251. | nvme4n1 | 21662.20 | 2707.62 | 0.60 | 0.00 | 0.68 | 127.99 | 14.79 | 74.42 |
  252.  
  253.  
  254.  
  255. O_DIRECT 2.75G/s
  256.  
  257. avg-cpu: %user %nice %system %iowait %steal %idle
  258. 0.01 0.00 0.42 1.16 0.00 98.41
  259.  
  260. | Device | r/s | rMB/s | rrqm/s | %rrqm | r_await | rareq-sz | aqu-sz | %util |
  261. |---------|----------|---------|--------|-------|---------|----------|--------|--------|
  262. | nvme4n1 | 22542.40 | 2817.80 | 0.00 | 0.00 | 0.07 | 128.00 | 1.50 | 100.00 |
  263.  
  264.  
  265. The same as with 64K reads, but this time O_DIRECT gets higher results due to a deeper queue compared
  266. to the O_DIRECT 64K result. Kind of makes sense.
  267.  
  268.  
  269.  
  270. Read 512K
  271. ---------------
  272.  
  273. buffered 2.32G/s
  274.  
  275. avg-cpu: %user %nice %system %iowait %steal %idle
  276. 0.01 0.00 2.43 0.00 0.00 97.56
  277.  
  278. | Device | r/s | rMB/s | rrqm/s | %rrqm | r_await | rareq-sz | aqu-sz | %util |
  279. |---------|----------|---------|--------|-------|---------|----------|--------|-------|
  280. | nvme4n1 | 18972.60 | 2371.20 | 0.60 | 0.00 | 0.70 | 127.98 | 13.24 | 66.14 |
  281.  
  282.  
  283. O_DIRECT 4G/s
  284.  
  285.  
  286. avg-cpu: %user %nice %system %iowait %steal %idle
  287. 0.01 0.00 0.54 1.00 0.00 98.44
  288.  
  289. | Device | r/s | rMB/s | rrqm/s | %rrqm | r_await | rareq-sz | aqu-sz | %util |
  290. |---------|----------|---------|--------|-------|---------|----------|--------|-------|
  291. | nvme2n1 | 32124.13 | 4015.52 | 0.00 | 0.00 | 0.07 | 128.00 | 2.24 | 97.46 |
  292.  
  293.  
  294. Buffered read has plateaued; maybe this drive can't handle aqu-sz greater than ~11?
  295. O_DIRECT read: Linear increase in aqu-sz, compared to the 256K result, leads to the same performance uplift.
  296. Why is there practically no performance uplift for O_DIRECT reads from 64K to 256K?
  297.  
  298. Not much makes sense so far. Maybe an example where things work as expected (more or less) will bring some clarity.
  299.  
  300.  
  301.  
  302. O_DIRECT 64K single disk vs O_DIRECT RAID0
  303. ===============================================================
  304.  
  305. single disk O_DIRECT 64K 2.18G/s
  306.  
  307. avg-cpu: %user %nice %system %iowait %steal %idle
  308. 0.03 0.00 0.46 1.09 0.00 98.43
  309.  
  310. | Device | r/s | rMB/s | rrqm/s | %rrqm | r_await | rareq-sz | aqu-sz | %util |
  311. |---------|----------|---------|--------|-------|---------|----------|--------|--------|
  312. | nvme3n1 | 36139.40 | 2258.71 | 0.00 | 0.00 | 0.02 | 64.00 | 0.82 | 100.02 |
  313.  
  314.  
  315. single disk O_DIRECT 128K 2.68G/s
  316.  
  317. avg-cpu: %user %nice %system %iowait %steal %idle
  318. 0.02 0.00 0.35 1.20 0.00 98.43
  319.  
  320.  
  321. | Device | r/s | rMB/s | rrqm/s | %rrqm | r_await | rareq-sz | aqu-sz | %util |
  322. |---------|----------|---------|--------|-------|---------|----------|--------|--------|
  323. | nvme3n1 | 22162.20 | 2770.25 | 0.00 | 0.00 | 0.04 | 128.00 | 0.87 | 100.00 |
  324.  
  325.  
  326. Request size doubled yet minor change in BW.
  327.  
  328.  
  329.  
  330. RAID0 O_DIRECT 256K (full stripe) read 5.59G/s
  331.  
  332. avg-cpu: %user %nice %system %iowait %steal %idle
  333. 0.01 0.00 0.91 0.46 0.00 98.61
  334.  
  335. | Device | r/s | rMB/s | rrqm/s | %rrqm | r_await | rareq-sz | aqu-sz | %util |
  336. |---------|:--------:|:-------:|:------:|:-----:|:-------:|:--------:|:------:|:------:|
  337. | md127 | 89868.00 | 5616.75 | 0.00 | 0.00 | 0.02 | 64.00 | 2.23 | 100.00 |
  338. | nvme0n1 | 22467.20 | 1404.20 | 0.00 | 0.00 | 0.03 | 64.00 | 0.57 | 100.00 |
  339. | nvme1n1 | 22467.20 | 1404.20 | 0.00 | 0.00 | 0.02 | 64.00 | 0.55 | 100.00 |
  340. | nvme2n1 | 22467.20 | 1404.19 | 0.00 | 0.00 | 0.02 | 64.00 | 0.53 | 100.00 |
  341. | nvme4n1 | 22467.00 | 1404.19 | 0.00 | 0.00 | 0.02 | 64.00 | 0.53 | 100.00 |
  342.  
  343.  
  344. Compared to a single disk 64K read, this is a 2.5X uplift, a far cry from the theoretical 4X
  345. but at least it's an increase. It seems to align with the increase in aqu-sz: 2.23/0.82 ≈ 5.59G/2.18G.
  346.  
  347. Why is rareq-sz 64? It should be 128 in line with the max_sectors_kb limit?
  348.  
  349.  
  350.  
  351. RAID0 O_DIRECT 512K (2x full stripes) read 6.75G/s
  352.  
  353.  
  354. avg-cpu: %user %nice %system %iowait %steal %idle
  355. 0.01 0.00 1.20 0.30 0.00 98.49
  356.  
  357. | Device | r/s | rMB/s | rrqm/s | %rrqm | r_await | rareq-sz | aqu-sz | %util |
  358. |---------|:---------:|:-------:|:------:|:-----:|:-------:|:--------:|:------:|:------:|
  359. | md127 | 109051.40 | 6815.71 | 0.00 | 0.00 | 0.03 | 64.00 | 3.01 | 100.00 |
  360. | nvme0n1 | 27263.00 | 1703.94 | 0.00 | 0.00 | 0.03 | 64.00 | 0.79 | 100.00 |
  361. | nvme1n1 | 27262.80 | 1703.92 | 0.00 | 0.00 | 0.03 | 64.00 | 0.76 | 100.00 |
  362. | nvme2n1 | 27263.00 | 1703.94 | 0.00 | 0.00 | 0.02 | 64.00 | 0.67 | 100.00 |
  363. | nvme4n1 | 27263.00 | 1703.94 | 0.00 | 0.00 | 0.03 | 64.00 | 0.72 | 100.00 |
  364.  
  365.  
  366. Compared to a single disk 64K read: 3x increase. aqu-sz ratio is 3.01/0.82 = 3.6. Reaching a plateau?
  367.  
  368. Can someone answer some of these questions?
  369.  
  370. Thanks
  371. Dragan
  372.  
  373.  
Advertisement
Add Comment
Please, Sign In to add comment
Public Pastes
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!