API tools faq
paste
Guest User

Untitled

a guest
May 14th, 2018
116
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 59.04 KB | None | 0 0
raw download clone embed print report
  1. root@alice:~# cat /proc/version
  2. Linux version 4.15.0-20-generic (buildd@lgw01-amd64-039) (gcc version 7.3.0 (Ubuntu 7.3.0-16ubuntu3)) #21-Ubuntu SMP Tue Apr 24 06:16:15 UTC 2018
  3.  
  4. root@alice:~# cat /proc/cpuinfo
  5. processor : 0
  6. vendor_id : GenuineIntel
  7. cpu family : 6
  8. model : 85
  9. model name : Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz
  10. stepping : 4
  11. microcode : 0x2000035
  12. cpu MHz : 800.517
  13. cache size : 14080 KB
  14. physical id : 0
  15. siblings : 20
  16. core id : 0
  17. cpu cores : 10
  18. apicid : 0
  19. initial apicid : 0
  20. fpu : yes
  21. fpu_exception : yes
  22. cpuid level : 22
  23. wp : yes
  24. flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cdp_l3 invpcid_single pti intel_ppin mba tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb intel_pt avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts pku ospke
  25. bugs : cpu_meltdown spectre_v1 spectre_v2
  26. bogomips : 4400.00
  27. clflush size : 64
  28. cache_alignment : 64
  29. address sizes : 46 bits physical, 48 bits virtual
  30. power management:
  31. [repeats for processors 1-19]
  32.  
  33. root@alice:~# cat /proc/meminfo
  34. MemTotal: 23252352 kB
  35. MemFree: 673692 kB
  36. MemAvailable: 22073424 kB
  37. Buffers: 145588 kB
  38. Cached: 20598104 kB
  39. SwapCached: 0 kB
  40. Active: 594096 kB
  41. Inactive: 20421860 kB
  42. Active(anon): 129900 kB
  43. Inactive(anon): 144452 kB
  44. Active(file): 464196 kB
  45. Inactive(file): 20277408 kB
  46. Unevictable: 0 kB
  47. Mlocked: 0 kB
  48. SwapTotal: 2097148 kB
  49. SwapFree: 2097148 kB
  50. Dirty: 188 kB
  51. Writeback: 0 kB
  52. AnonPages: 272312 kB
  53. Mapped: 186944 kB
  54. Shmem: 2096 kB
  55. Slab: 1155916 kB
  56. SReclaimable: 1053640 kB
  57. SUnreclaim: 102276 kB
  58. KernelStack: 12016 kB
  59. PageTables: 20388 kB
  60. NFS_Unstable: 0 kB
  61. Bounce: 0 kB
  62. WritebackTmp: 0 kB
  63. CommitLimit: 13723324 kB
  64. Committed_AS: 2024780 kB
  65. VmallocTotal: 34359738367 kB
  66. VmallocUsed: 0 kB
  67. VmallocChunk: 0 kB
  68. HardwareCorrupted: 0 kB
  69. AnonHugePages: 0 kB
  70. ShmemHugePages: 0 kB
  71. ShmemPmdMapped: 0 kB
  72. CmaTotal: 0 kB
  73. CmaFree: 0 kB
  74. HugePages_Total: 0
  75. HugePages_Free: 0
  76. HugePages_Rsvd: 0
  77. HugePages_Surp: 0
  78. Hugepagesize: 2048 kB
  79. DirectMap4k: 331544 kB
  80. DirectMap2M: 15042560 kB
  81. DirectMap1G: 10485760 kB
  82.  
  83. root@alice:~# lspci -vvv
  84. [...]
  85. 65:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller SM961/PM961 (prog-if 02 [NVM Express])
  86. Subsystem: Samsung Electronics Co Ltd NVMe SSD Controller SM961/PM961
  87. Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+
  88. Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
  89. Latency: 0, Cache Line Size: 32 bytes
  90. Interrupt: pin A routed to IRQ 39
  91. NUMA node: 0
  92. Region 0: Memory at e0e00000 (64-bit, non-prefetchable) [size=16K]
  93. Capabilities: [40] Power Management version 3
  94. Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
  95. Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
  96. Capabilities: [50] MSI: Enable- Count=1/32 Maskable- 64bit+
  97. Address: 0000000000000000 Data: 0000
  98. Capabilities: [70] Express (v2) Endpoint, MSI 00
  99. DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited
  100. ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0.000W
  101. DevCtl: Report errors: Correctable+ Non-Fatal+ Fatal+ Unsupported+
  102. RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+ FLReset-
  103. MaxPayload 128 bytes, MaxReadReq 512 bytes
  104. DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr+ TransPend-
  105. LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM L1, Exit Latency L0s unlimited, L1 <64us
  106. ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
  107. LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
  108. ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
  109. LnkSta: Speed 8GT/s, Width x4, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
  110. DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR+, OBFF Not Supported
  111. DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
  112. LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
  113. Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
  114. Compliance De-emphasis: -6dB
  115. LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete+, EqualizationPhase1+
  116. EqualizationPhase2+, EqualizationPhase3+, LinkEqualizationRequest-
  117. Capabilities: [b0] MSI-X: Enable+ Count=8 Masked-
  118. Vector table: BAR=0 offset=00003000
  119. PBA: BAR=0 offset=00002000
  120. Capabilities: [100 v2] Advanced Error Reporting
  121. UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
  122. UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-
  123. UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
  124. CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
  125. CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
  126. AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
  127. Capabilities: [148 v1] Device Serial Number 00-00-00-00-00-00-00-00
  128. Capabilities: [158 v1] Power Budgeting <?>
  129. Capabilities: [168 v1] #19
  130. Capabilities: [188 v1] Latency Tolerance Reporting
  131. Max snoop latency: 0ns
  132. Max no snoop latency: 0ns
  133. Capabilities: [190 v1] L1 PM Substates
  134. L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+ L1_PM_Substates+
  135. PortCommonModeRestoreTime=10us PortTPowerOnTime=10us
  136. L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1-
  137. T_CommonMode=0us LTR1.2_Threshold=0ns
  138. L1SubCtl2: T_PwrOn=10us
  139. Kernel driver in use: nvme
  140. Kernel modules: nvme
  141.  
  142. 66:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller SM961/PM961 (prog-if 02 [NVM Express])
  143. Subsystem: Samsung Electronics Co Ltd NVMe SSD Controller SM961/PM961
  144. Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+
  145. Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
  146. Latency: 0, Cache Line Size: 32 bytes
  147. Interrupt: pin A routed to IRQ 28
  148. NUMA node: 0
  149. Region 0: Memory at e0d00000 (64-bit, non-prefetchable) [size=16K]
  150. Capabilities: [40] Power Management version 3
  151. Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
  152. Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
  153. Capabilities: [50] MSI: Enable- Count=1/32 Maskable- 64bit+
  154. Address: 0000000000000000 Data: 0000
  155. Capabilities: [70] Express (v2) Endpoint, MSI 00
  156. DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited
  157. ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0.000W
  158. DevCtl: Report errors: Correctable+ Non-Fatal+ Fatal+ Unsupported+
  159. RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+ FLReset-
  160. MaxPayload 128 bytes, MaxReadReq 512 bytes
  161. DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr+ TransPend-
  162. LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM L1, Exit Latency L0s unlimited, L1 <64us
  163. ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
  164. LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
  165. ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
  166. LnkSta: Speed 8GT/s, Width x4, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
  167. DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR+, OBFF Not Supported
  168. DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
  169. LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
  170. Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
  171. Compliance De-emphasis: -6dB
  172. LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete+, EqualizationPhase1+
  173. EqualizationPhase2+, EqualizationPhase3+, LinkEqualizationRequest-
  174. Capabilities: [b0] MSI-X: Enable+ Count=8 Masked-
  175. Vector table: BAR=0 offset=00003000
  176. PBA: BAR=0 offset=00002000
  177. Capabilities: [100 v2] Advanced Error Reporting
  178. UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-
  179. UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
  180. UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
  181. CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
  182. CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
  183. AERCap: First Error Pointer: 14, GenCap+ CGenEn- ChkCap+ ChkEn-
  184. Capabilities: [148 v1] Device Serial Number 00-00-00-00-00-00-00-00
  185. Capabilities: [158 v1] Power Budgeting <?>
  186. Capabilities: [168 v1] #19
  187. Capabilities: [188 v1] Latency Tolerance Reporting
  188. Max snoop latency: 0ns
  189. Max no snoop latency: 0ns
  190. Capabilities: [190 v1] L1 PM Substates
  191. L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+ L1_PM_Substates+
  192. PortCommonModeRestoreTime=10us PortTPowerOnTime=10us
  193. L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1-
  194. T_CommonMode=0us LTR1.2_Threshold=0ns
  195. L1SubCtl2: T_PwrOn=10us
  196. Kernel driver in use: nvme
  197. Kernel modules: nvme
  198.  
  199. 67:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller SM961/PM961 (prog-if 02 [NVM Express])
  200. Subsystem: Samsung Electronics Co Ltd NVMe SSD Controller SM961/PM961
  201. Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+
  202. Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
  203. Latency: 0, Cache Line Size: 32 bytes
  204. Interrupt: pin A routed to IRQ 28
  205. NUMA node: 0
  206. Region 0: Memory at e0c00000 (64-bit, non-prefetchable) [size=16K]
  207. Capabilities: [40] Power Management version 3
  208. Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
  209. Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
  210. Capabilities: [50] MSI: Enable- Count=1/32 Maskable- 64bit+
  211. Address: 0000000000000000 Data: 0000
  212. Capabilities: [70] Express (v2) Endpoint, MSI 00
  213. DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited
  214. ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0.000W
  215. DevCtl: Report errors: Correctable+ Non-Fatal+ Fatal+ Unsupported+
  216. RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+ FLReset-
  217. MaxPayload 128 bytes, MaxReadReq 512 bytes
  218. DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr+ TransPend-
  219. LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM L1, Exit Latency L0s unlimited, L1 <64us
  220. ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
  221. LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
  222. ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
  223. LnkSta: Speed 8GT/s, Width x4, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
  224. DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR+, OBFF Not Supported
  225. DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
  226. LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
  227. Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
  228. Compliance De-emphasis: -6dB
  229. LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete+, EqualizationPhase1+
  230. EqualizationPhase2+, EqualizationPhase3+, LinkEqualizationRequest-
  231. Capabilities: [b0] MSI-X: Enable+ Count=8 Masked-
  232. Vector table: BAR=0 offset=00003000
  233. PBA: BAR=0 offset=00002000
  234. Capabilities: [100 v2] Advanced Error Reporting
  235. UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-
  236. UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
  237. UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
  238. CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
  239. CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
  240. AERCap: First Error Pointer: 14, GenCap+ CGenEn- ChkCap+ ChkEn-
  241. Capabilities: [148 v1] Device Serial Number 00-00-00-00-00-00-00-00
  242. Capabilities: [158 v1] Power Budgeting <?>
  243. Capabilities: [168 v1] #19
  244. Capabilities: [188 v1] Latency Tolerance Reporting
  245. Max snoop latency: 0ns
  246. Max no snoop latency: 0ns
  247. Capabilities: [190 v1] L1 PM Substates
  248. L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+ L1_PM_Substates+
  249. PortCommonModeRestoreTime=10us PortTPowerOnTime=10us
  250. L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1-
  251. T_CommonMode=0us LTR1.2_Threshold=0ns
  252. L1SubCtl2: T_PwrOn=10us
  253. Kernel driver in use: nvme
  254. Kernel modules: nvme
  255.  
  256. 68:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller SM961/PM961 (prog-if 02 [NVM Express])
  257. Subsystem: Samsung Electronics Co Ltd NVMe SSD Controller SM961/PM961
  258. Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+
  259. Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
  260. Latency: 0, Cache Line Size: 32 bytes
  261. Interrupt: pin A routed to IRQ 28
  262. NUMA node: 0
  263. Region 0: Memory at e0b00000 (64-bit, non-prefetchable) [size=16K]
  264. Capabilities: [40] Power Management version 3
  265. Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
  266. Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
  267. Capabilities: [50] MSI: Enable- Count=1/32 Maskable- 64bit+
  268. Address: 0000000000000000 Data: 0000
  269. Capabilities: [70] Express (v2) Endpoint, MSI 00
  270. DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited
  271. ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0.000W
  272. DevCtl: Report errors: Correctable+ Non-Fatal+ Fatal+ Unsupported+
  273. RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+ FLReset-
  274. MaxPayload 128 bytes, MaxReadReq 512 bytes
  275. DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr+ TransPend-
  276. LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM L1, Exit Latency L0s unlimited, L1 <64us
  277. ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
  278. LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
  279. ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
  280. LnkSta: Speed 8GT/s, Width x4, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
  281. DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR+, OBFF Not Supported
  282. DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
  283. LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
  284. Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
  285. Compliance De-emphasis: -6dB
  286. LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete+, EqualizationPhase1+
  287. EqualizationPhase2+, EqualizationPhase3+, LinkEqualizationRequest-
  288. Capabilities: [b0] MSI-X: Enable+ Count=8 Masked-
  289. Vector table: BAR=0 offset=00003000
  290. PBA: BAR=0 offset=00002000
  291. Capabilities: [100 v2] Advanced Error Reporting
  292. UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-
  293. UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
  294. UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
  295. CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
  296. CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
  297. AERCap: First Error Pointer: 14, GenCap+ CGenEn- ChkCap+ ChkEn-
  298. Capabilities: [148 v1] Device Serial Number 00-00-00-00-00-00-00-00
  299. Capabilities: [158 v1] Power Budgeting <?>
  300. Capabilities: [168 v1] #19
  301. Capabilities: [188 v1] Latency Tolerance Reporting
  302. Max snoop latency: 0ns
  303. Max no snoop latency: 0ns
  304. Capabilities: [190 v1] L1 PM Substates
  305. L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+ L1_PM_Substates+
  306. PortCommonModeRestoreTime=10us PortTPowerOnTime=10us
  307. L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1-
  308. T_CommonMode=0us LTR1.2_Threshold=0ns
  309. L1SubCtl2: T_PwrOn=10us
  310. Kernel driver in use: nvme
  311. Kernel modules: nvme
  312. [...]
  313.  
  314.  
  315. root@alice:~# mkfs.ext4 -F -v -b 4096 -E lazy_itable_init=0,lazy_journal_init=0 /dev/nvme0n1
  316. mke2fs 1.44.1 (24-Mar-2018)
  317. /dev/nvme0n1 contains a ext4 file system
  318. created on Mon May 14 10:43:28 2018
  319. fs_types for mke2fs.conf resolution: 'ext4'
  320. Discarding device blocks: done
  321. Filesystem label=
  322. OS type: Linux
  323. Block size=4096 (log=2)
  324. Fragment size=4096 (log=2)
  325. Stride=0 blocks, Stripe width=0 blocks
  326. 62513152 inodes, 250051158 blocks
  327. 12502557 blocks (5.00%) reserved for the super user
  328. First data block=0
  329. Maximum filesystem blocks=2399141888
  330. 7631 block groups
  331. 32768 blocks per group, 32768 fragments per group
  332. 8192 inodes per group
  333. Filesystem UUID: 08e33b39-b6b8-439d-9506-20b65f628be3
  334. Superblock backups stored on blocks:
  335. 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
  336. 4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968,
  337. 102400000, 214990848
  338.  
  339. Allocating group tables: done
  340. Writing inode tables: done
  341. Creating journal (262144 blocks): done
  342. Writing superblocks and filesystem accounting information: done
  343.  
  344. root@alice:~# mkfs.ext4 -F -v -b 4096 -E lazy_itable_init=0,lazy_journal_init=0 /dev/nvme1n1
  345. mke2fs 1.44.1 (24-Mar-2018)
  346. /dev/nvme1n1 contains a ext4 file system
  347. created on Mon May 14 10:43:47 2018
  348. fs_types for mke2fs.conf resolution: 'ext4'
  349. Discarding device blocks: done
  350. Filesystem label=
  351. OS type: Linux
  352. Block size=4096 (log=2)
  353. Fragment size=4096 (log=2)
  354. Stride=0 blocks, Stripe width=0 blocks
  355. 62513152 inodes, 250051158 blocks
  356. 12502557 blocks (5.00%) reserved for the super user
  357. First data block=0
  358. Maximum filesystem blocks=2399141888
  359. 7631 block groups
  360. 32768 blocks per group, 32768 fragments per group
  361. 8192 inodes per group
  362. Filesystem UUID: ff72a731-b450-4ee6-b88c-9cdcaae9c275
  363. Superblock backups stored on blocks:
  364. 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
  365. 4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968,
  366. 102400000, 214990848
  367.  
  368. Allocating group tables: done
  369. Writing inode tables: done
  370. Creating journal (262144 blocks): done
  371. Writing superblocks and filesystem accounting information: done
  372.  
  373. root@alice:~# mkfs.ext4 -F -v -b 4096 -E lazy_itable_init=0,lazy_journal_init=0 /dev/nvme2n1
  374. mke2fs 1.44.1 (24-Mar-2018)
  375. /dev/nvme2n1 contains a ext4 file system
  376. last mounted on /data2 on Mon May 14 10:30:53 2018
  377. fs_types for mke2fs.conf resolution: 'ext4'
  378. Discarding device blocks: done
  379. Filesystem label=
  380. OS type: Linux
  381. Block size=4096 (log=2)
  382. Fragment size=4096 (log=2)
  383. Stride=0 blocks, Stripe width=0 blocks
  384. 62513152 inodes, 250051158 blocks
  385. 12502557 blocks (5.00%) reserved for the super user
  386. First data block=0
  387. Maximum filesystem blocks=2399141888
  388. 7631 block groups
  389. 32768 blocks per group, 32768 fragments per group
  390. 8192 inodes per group
  391. Filesystem UUID: ccc9da15-41c1-4260-a845-2e362ab2e86f
  392. Superblock backups stored on blocks:
  393. 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
  394. 4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968,
  395. 102400000, 214990848
  396.  
  397. Allocating group tables: done
  398. Writing inode tables: done
  399. Creating journal (262144 blocks): done
  400. Writing superblocks and filesystem accounting information: done
  401.  
  402. root@alice:~# mkfs.ext4 -F -v -b 4096 -E lazy_itable_init=0,lazy_journal_init=0 /dev/nvme3n1
  403. mke2fs 1.44.1 (24-Mar-2018)
  404. /dev/nvme3n1 contains a ext4 file system
  405. last mounted on /data3 on Mon May 14 10:30:59 2018
  406. fs_types for mke2fs.conf resolution: 'ext4'
  407. Discarding device blocks: done
  408. Filesystem label=
  409. OS type: Linux
  410. Block size=4096 (log=2)
  411. Fragment size=4096 (log=2)
  412. Stride=0 blocks, Stripe width=0 blocks
  413. 62513152 inodes, 250051158 blocks
  414. 12502557 blocks (5.00%) reserved for the super user
  415. First data block=0
  416. Maximum filesystem blocks=2399141888
  417. 7631 block groups
  418. 32768 blocks per group, 32768 fragments per group
  419. 8192 inodes per group
  420. Filesystem UUID: 0469cb89-3c7b-44d4-88c5-83c05cc40f00
  421. Superblock backups stored on blocks:
  422. 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
  423. 4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968,
  424. 102400000, 214990848
  425.  
  426. Allocating group tables: done
  427. Writing inode tables: done
  428. Creating journal (262144 blocks): done
  429. Writing superblocks and filesystem accounting information: done
  430.  
  431. root@alice:~# mount /dev/nvme0n1 /data0 -o lazytime,async
  432. root@alice:~# mount /dev/nvme1n1 /data1 -o lazytime,async
  433. root@alice:~# mount /dev/nvme2n1 /data2 -o lazytime,async
  434. root@alice:~# mount /dev/nvme3n1 /data3 -o lazytime,async
  435.  
  436. root@alice:~# cat ~ryan/fio-seq-write-4x-ext4.job
  437. ; fio-seq-write.job for fiotest
  438.  
  439. [global]
  440. ioengine=libaio
  441. iodepth=32
  442. rw=write
  443. bs=1024K
  444. direct=0
  445. size=256G
  446. numjobs=2
  447. time_based=1
  448. runtime=300
  449. fallocate=native
  450.  
  451. [nvme0]
  452. name=fio-seq-write-0
  453. directory=/data0
  454.  
  455. [nvme0]
  456. name=fio-seq-write-1
  457. directory=/data1
  458.  
  459. [nvme0]
  460. name=fio-seq-write-2
  461. directory=/data2
  462.  
  463. [nvme0]
  464. name=fio-seq-write-3
  465. directory=/data3
  466.  
  467. root@alice:~# fio ~ryan/fio-seq-write-4x-ext4.job
  468. fio-seq-write-0: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=32
  469. ...
  470. fio-seq-write-1: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=32
  471. ...
  472. fio-seq-write-2: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=32
  473. ...
  474. fio-seq-write-3: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=32
  475. ...
  476. fio-3.1
  477. Starting 8 processes
  478. fio-seq-write-0: Laying out IO file (1 file / 262144MiB)
  479. nvme0: Laying out IO file (1 file / 262144MiB)
  480. fio-seq-write-1: Laying out IO file (1 file / 262144MiB)
  481. nvme0: Laying out IO file (1 file / 262144MiB)
  482. fio-seq-write-2: Laying out IO file (1 file / 262144MiB)
  483. nvme0: Laying out IO file (1 file / 262144MiB)
  484. fio-seq-write-3: Laying out IO file (1 file / 262144MiB)
  485. nvme0: Laying out IO file (1 file / 262144MiB)
  486. Jobs: 8 (f=8): [W(8)][100.0%][r=0KiB/s,w=7881MiB/s][r=0,w=7880 IOPS][eta 00m:00s]
  487. fio-seq-write-0: (groupid=0, jobs=1): err= 0: pid=3203: Mon May 14 10:54:16 2018
  488. write: IOPS=938, BW=939MiB/s (984MB/s)(275GiB/300001msec)
  489. slat (usec): min=445, max=232737, avg=1062.50, stdev=1100.92
  490. clat (usec): min=4, max=277060, avg=33023.16, stdev=9446.16
  491. lat (msec): min=2, max=278, avg=34.09, stdev= 9.69
  492. clat percentiles (msec):
  493. | 1.00th=[ 24], 5.00th=[ 26], 10.00th=[ 26], 20.00th=[ 28],
  494. | 30.00th=[ 29], 40.00th=[ 30], 50.00th=[ 31], 60.00th=[ 33],
  495. | 70.00th=[ 35], 80.00th=[ 37], 90.00th=[ 42], 95.00th=[ 48],
  496. | 99.00th=[ 68], 99.50th=[ 79], 99.90th=[ 113], 99.95th=[ 161],
  497. | 99.99th=[ 271]
  498. bw ( KiB/s): min=458322, max=1329548, per=12.39%, avg=969226.41, stdev=146056.23, samples=600
  499. iops : min= 447, max= 1298, avg=946.04, stdev=142.61, samples=600
  500. lat (usec) : 10=0.01%
  501. lat (msec) : 4=0.01%, 10=0.01%, 20=0.07%, 50=95.90%, 100=3.86%
  502. lat (msec) : 250=0.16%, 500=0.01%
  503. cpu : usr=5.03%, sys=93.95%, ctx=55394, majf=0, minf=13
  504. IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
  505. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  506. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
  507. issued rwt: total=0,281620,0, short=0,0,0, dropped=0,0,0
  508. latency : target=0, window=0, percentile=100.00%, depth=32
  509. nvme0: (groupid=0, jobs=1): err= 0: pid=3204: Mon May 14 10:54:16 2018
  510. write: IOPS=961, BW=962MiB/s (1009MB/s)(282GiB/300001msec)
  511. slat (usec): min=454, max=232060, avg=1036.90, stdev=1104.70
  512. clat (usec): min=3, max=276584, avg=32226.72, stdev=9592.49
  513. lat (usec): min=627, max=277782, avg=33263.95, stdev=9837.87
  514. clat percentiles (msec):
  515. | 1.00th=[ 23], 5.00th=[ 25], 10.00th=[ 26], 20.00th=[ 27],
  516. | 30.00th=[ 28], 40.00th=[ 29], 50.00th=[ 31], 60.00th=[ 32],
  517. | 70.00th=[ 34], 80.00th=[ 36], 90.00th=[ 41], 95.00th=[ 46],
  518. | 99.00th=[ 66], 99.50th=[ 78], 99.90th=[ 129], 99.95th=[ 190],
  519. | 99.99th=[ 266]
  520. bw ( KiB/s): min=412903, max=1531870, per=12.70%, avg=993381.82, stdev=151664.04, samples=600
  521. iops : min= 403, max= 1495, avg=969.62, stdev=148.10, samples=600
  522. lat (usec) : 4=0.01%, 750=0.01%
  523. lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.26%, 50=96.19%
  524. lat (msec) : 100=3.37%, 250=0.16%, 500=0.01%
  525. cpu : usr=5.14%, sys=93.85%, ctx=54042, majf=0, minf=12
  526. IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
  527. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  528. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
  529. issued rwt: total=0,288581,0, short=0,0,0, dropped=0,0,0
  530. latency : target=0, window=0, percentile=100.00%, depth=32
  531. fio-seq-write-1: (groupid=0, jobs=1): err= 0: pid=3205: Mon May 14 10:54:16 2018
  532. write: IOPS=960, BW=960MiB/s (1007MB/s)(281GiB/300001msec)
  533. slat (usec): min=444, max=217318, avg=1038.83, stdev=1119.71
  534. clat (usec): min=3, max=260547, avg=32286.95, stdev=9173.17
  535. lat (usec): min=717, max=261384, avg=33326.13, stdev=9407.74
  536. clat percentiles (msec):
  537. | 1.00th=[ 23], 5.00th=[ 26], 10.00th=[ 26], 20.00th=[ 28],
  538. | 30.00th=[ 29], 40.00th=[ 30], 50.00th=[ 31], 60.00th=[ 32],
  539. | 70.00th=[ 34], 80.00th=[ 36], 90.00th=[ 41], 95.00th=[ 45],
  540. | 99.00th=[ 64], 99.50th=[ 71], 99.90th=[ 138], 99.95th=[ 201],
  541. | 99.99th=[ 251]
  542. bw ( KiB/s): min=266322, max=1436903, per=12.67%, avg=991336.74, stdev=141586.31, samples=600
  543. iops : min= 260, max= 1403, avg=967.60, stdev=138.28, samples=600
  544. lat (usec) : 4=0.01%, 750=0.01%
  545. lat (msec) : 2=0.01%, 10=0.01%, 20=0.27%, 50=97.05%, 100=2.50%
  546. lat (msec) : 250=0.17%, 500=0.01%
  547. cpu : usr=5.07%, sys=94.03%, ctx=55774, majf=0, minf=12
  548. IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
  549. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  550. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
  551. issued rwt: total=0,288042,0, short=0,0,0, dropped=0,0,0
  552. latency : target=0, window=0, percentile=100.00%, depth=32
  553. nvme0: (groupid=0, jobs=1): err= 0: pid=3206: Mon May 14 10:54:16 2018
  554. write: IOPS=945, BW=946MiB/s (992MB/s)(277GiB/300001msec)
  555. slat (usec): min=441, max=218295, avg=1054.40, stdev=1145.23
  556. clat (usec): min=3, max=266611, avg=32770.79, stdev=10000.31
  557. lat (usec): min=980, max=267615, avg=33825.57, stdev=10256.53
  558. clat percentiles (msec):
  559. | 1.00th=[ 23], 5.00th=[ 26], 10.00th=[ 26], 20.00th=[ 28],
  560. | 30.00th=[ 29], 40.00th=[ 30], 50.00th=[ 31], 60.00th=[ 33],
  561. | 70.00th=[ 34], 80.00th=[ 37], 90.00th=[ 41], 95.00th=[ 47],
  562. | 99.00th=[ 67], 99.50th=[ 79], 99.90th=[ 146], 99.95th=[ 203],
  563. | 99.99th=[ 251]
  564. bw ( KiB/s): min=331776, max=1521664, per=12.40%, avg=969920.16, stdev=152700.71, samples=599
  565. iops : min= 324, max= 1486, avg=947.00, stdev=149.16, samples=599
  566. lat (usec) : 4=0.01%, 1000=0.01%
  567. lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.22%, 50=95.96%
  568. lat (msec) : 100=3.59%, 250=0.21%, 500=0.01%
  569. cpu : usr=4.97%, sys=94.14%, ctx=54248, majf=0, minf=13
  570. IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
  571. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  572. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
  573. issued rwt: total=0,283790,0, short=0,0,0, dropped=0,0,0
  574. latency : target=0, window=0, percentile=100.00%, depth=32
  575. fio-seq-write-2: (groupid=0, jobs=1): err= 0: pid=3207: Mon May 14 10:54:16 2018
  576. write: IOPS=952, BW=953MiB/s (999MB/s)(279GiB/300001msec)
  577. slat (usec): min=431, max=98207, avg=1046.82, stdev=802.78
  578. clat (usec): min=2, max=217748, avg=32534.98, stdev=8110.36
  579. lat (usec): min=896, max=218611, avg=33582.14, stdev=8334.43
  580. clat percentiles (msec):
  581. | 1.00th=[ 24], 5.00th=[ 26], 10.00th=[ 27], 20.00th=[ 28],
  582. | 30.00th=[ 29], 40.00th=[ 30], 50.00th=[ 31], 60.00th=[ 33],
  583. | 70.00th=[ 34], 80.00th=[ 37], 90.00th=[ 41], 95.00th=[ 46],
  584. | 99.00th=[ 65], 99.50th=[ 73], 99.90th=[ 97], 99.95th=[ 117],
  585. | 99.99th=[ 209]
  586. bw ( KiB/s): min=338580, max=1310967, per=12.58%, avg=984028.66, stdev=130415.10, samples=600
  587. iops : min= 330, max= 1280, avg=960.46, stdev=127.37, samples=600
  588. lat (usec) : 4=0.01%, 1000=0.01%
  589. lat (msec) : 4=0.01%, 10=0.01%, 20=0.06%, 50=96.94%, 100=2.91%
  590. lat (msec) : 250=0.09%
  591. cpu : usr=5.06%, sys=94.24%, ctx=55786, majf=0, minf=13
  592. IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
  593. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  594. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
  595. issued rwt: total=0,285846,0, short=0,0,0, dropped=0,0,0
  596. latency : target=0, window=0, percentile=100.00%, depth=32
  597. nvme0: (groupid=0, jobs=1): err= 0: pid=3208: Mon May 14 10:54:16 2018
  598. write: IOPS=971, BW=972MiB/s (1019MB/s)(285GiB/300001msec)
  599. slat (usec): min=445, max=90863, avg=1026.36, stdev=779.61
  600. clat (usec): min=3, max=192485, avg=31899.51, stdev=8158.90
  601. lat (usec): min=815, max=193342, avg=32926.18, stdev=8386.62
  602. clat percentiles (msec):
  603. | 1.00th=[ 23], 5.00th=[ 25], 10.00th=[ 26], 20.00th=[ 27],
  604. | 30.00th=[ 28], 40.00th=[ 29], 50.00th=[ 31], 60.00th=[ 32],
  605. | 70.00th=[ 34], 80.00th=[ 36], 90.00th=[ 40], 95.00th=[ 44],
  606. | 99.00th=[ 64], 99.50th=[ 74], 99.90th=[ 122], 99.95th=[ 140],
  607. | 99.99th=[ 176]
  608. bw ( KiB/s): min=364088, max=1573161, per=12.83%, avg=1003423.71, stdev=132777.33, samples=600
  609. iops : min= 355, max= 1536, avg=979.43, stdev=129.65, samples=600
  610. lat (usec) : 4=0.01%, 1000=0.01%
  611. lat (msec) : 4=0.01%, 10=0.01%, 20=0.28%, 50=97.32%, 100=2.23%
  612. lat (msec) : 250=0.16%
  613. cpu : usr=5.07%, sys=94.30%, ctx=54593, majf=0, minf=11
  614. IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
  615. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  616. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
  617. issued rwt: total=0,291542,0, short=0,0,0, dropped=0,0,0
  618. latency : target=0, window=0, percentile=100.00%, depth=32
  619. fio-seq-write-3: (groupid=0, jobs=1): err= 0: pid=3209: Mon May 14 10:54:16 2018
  620. write: IOPS=961, BW=961MiB/s (1008MB/s)(282GiB/300001msec)
  621. slat (usec): min=431, max=174252, avg=1037.49, stdev=1030.36
  622. clat (usec): min=2, max=213406, avg=32242.61, stdev=8323.89
  623. lat (usec): min=642, max=214520, avg=33280.41, stdev=8535.02
  624. clat percentiles (msec):
  625. | 1.00th=[ 23], 5.00th=[ 25], 10.00th=[ 26], 20.00th=[ 27],
  626. | 30.00th=[ 29], 40.00th=[ 30], 50.00th=[ 31], 60.00th=[ 32],
  627. | 70.00th=[ 34], 80.00th=[ 36], 90.00th=[ 41], 95.00th=[ 45],
  628. | 99.00th=[ 62], 99.50th=[ 70], 99.90th=[ 110], 99.95th=[ 169],
  629. | 99.99th=[ 207]
  630. bw ( KiB/s): min=556297, max=1430709, per=12.69%, avg=992463.56, stdev=128972.08, samples=600
  631. iops : min= 543, max= 1397, avg=968.72, stdev=125.98, samples=600
  632. lat (usec) : 4=0.01%, 750=0.01%
  633. lat (msec) : 4=0.01%, 10=0.01%, 20=0.23%, 50=97.24%, 100=2.40%
  634. lat (msec) : 250=0.13%
  635. cpu : usr=5.25%, sys=93.46%, ctx=55338, majf=0, minf=11
  636. IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
  637. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  638. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
  639. issued rwt: total=0,288438,0, short=0,0,0, dropped=0,0,0
  640. latency : target=0, window=0, percentile=100.00%, depth=32
  641. nvme0: (groupid=0, jobs=1): err= 0: pid=3210: Mon May 14 10:54:16 2018
  642. write: IOPS=947, BW=947MiB/s (993MB/s)(278GiB/300001msec)
  643. slat (usec): min=445, max=174690, avg=1052.81, stdev=1068.32
  644. clat (usec): min=3, max=248620, avg=32721.43, stdev=10078.38
  645. lat (usec): min=903, max=251248, avg=33774.60, stdev=10345.58
  646. clat percentiles (msec):
  647. | 1.00th=[ 23], 5.00th=[ 25], 10.00th=[ 26], 20.00th=[ 27],
  648. | 30.00th=[ 28], 40.00th=[ 29], 50.00th=[ 31], 60.00th=[ 33],
  649. | 70.00th=[ 34], 80.00th=[ 37], 90.00th=[ 42], 95.00th=[ 48],
  650. | 99.00th=[ 71], 99.50th=[ 86], 99.90th=[ 138], 99.95th=[ 174],
  651. | 99.99th=[ 211]
  652. bw ( KiB/s): min=388129, max=1544258, per=12.50%, avg=977934.93, stdev=156092.60, samples=600
  653. iops : min= 379, max= 1508, avg=954.51, stdev=152.45, samples=600
  654. lat (usec) : 4=0.01%, 1000=0.01%
  655. lat (msec) : 4=0.01%, 10=0.01%, 20=0.30%, 50=95.59%, 100=3.84%
  656. lat (msec) : 250=0.27%
  657. cpu : usr=4.97%, sys=93.63%, ctx=54221, majf=0, minf=12
  658. IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
  659. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  660. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
  661. issued rwt: total=0,284218,0, short=0,0,0, dropped=0,0,0
  662. latency : target=0, window=0, percentile=100.00%, depth=32
  663.  
  664. Run status group 0 (all jobs):
  665. WRITE: bw=7640MiB/s (8011MB/s), 939MiB/s-972MiB/s (984MB/s-1019MB/s), io=2238GiB (2403GB), run=300001-300001msec
  666.  
  667. Disk stats (read/write):
  668. nvme0n1: ios=0/572094, merge=0/73748, ticks=0/45237008, in_queue=45079684, util=92.73%
  669. nvme1n1: ios=0/580504, merge=0/346024, ticks=0/40808792, in_queue=40708512, util=91.27%
  670. nvme2n1: ios=0/579795, merge=0/86749, ticks=0/42895964, in_queue=42801308, util=92.57%
  671. nvme3n1: ios=0/576683, merge=0/22583, ticks=0/52936912, in_queue=52848684, util=95.29%
  672.  
  673. top -d 10 during fio run:
  674. top - 10:49:52 up 38 min, 4 users, load average: 5.49, 2.42, 2.45
  675. Tasks: 363 total, 15 running, 221 sleeping, 0 stopped, 0 zombie
  676. %Cpu(s): 2.0 us, 54.6 sy, 0.0 ni, 42.5 id, 0.8 wa, 0.0 hi, 0.1 si, 0.0 st
  677. KiB Mem : 23252352 total, 428864 free, 1056968 used, 21766520 buff/cache
  678. KiB Swap: 2097148 total, 2097148 free, 0 used. 21538676 avail Mem
  679.  
  680. PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
  681. 3204 root 20 0 745432 37492 1064 R 99.9 0.2 0:35.64 fio
  682. 3208 root 20 0 745456 37508 1076 R 99.8 0.2 0:35.62 fio
  683. 3209 root 20 0 745464 37492 1056 R 99.7 0.2 0:35.65 fio
  684. 3206 root 20 0 745444 37496 1060 R 99.6 0.2 0:35.54 fio
  685. 3210 root 20 0 745468 37504 1076 R 99.6 0.2 0:35.64 fio
  686. 3207 root 20 0 745452 37500 1068 R 99.4 0.2 0:35.54 fio
  687. 3203 root 20 0 745428 37500 1068 R 99.2 0.2 0:35.59 fio
  688. 3205 root 20 0 745440 37500 1068 R 99.0 0.2 0:35.43 fio
  689. 158 root 20 0 0 0 0 R 89.4 0.0 5:09.55 kswapd0
  690. 3027 root 20 0 0 0 0 R 60.9 0.0 0:44.53 kworker/u488:3
  691. 3110 root 20 0 0 0 0 R 56.0 0.0 0:33.54 kworker/u488:0
  692. 3026 root 20 0 0 0 0 R 54.4 0.0 1:22.48 kworker/u488:2
  693. 375 root 20 0 0 0 0 R 50.6 0.0 2:50.14 kworker/u488:18
  694. 3151 root 20 0 0 0 0 S 1.2 0.0 0:00.46 jbd2/nvme0n1-8
  695. 3159 root 20 0 0 0 0 S 1.2 0.0 0:00.26 jbd2/nvme2n1-8
  696. 149 root 20 0 0 0 0 I 1.0 0.0 0:02.55 kworker/0:2
  697. 147 root 20 0 0 0 0 I 0.6 0.0 0:01.55 kworker/4:1
  698. 3155 root 20 0 0 0 0 S 0.6 0.0 0:00.26 jbd2/nvme1n1-8
  699. 3181 root 20 0 712632 274092 270436 S 0.6 1.2 0:01.83 fio
  700. 137 root 20 0 0 0 0 S 0.5 0.0 0:00.13 kcompactd0
  701. 146 root 20 0 0 0 0 I 0.4 0.0 0:01.74 kworker/2:1
  702. 206 root 20 0 0 0 0 R 0.4 0.0 0:01.46 kworker/11:1
  703. 3213 ryan 20 0 51464 4340 3480 R 0.4 0.0 0:00.10 top
  704. 156 root 20 0 0 0 0 I 0.3 0.0 0:01.36 kworker/6:1
  705. 204 root 20 0 0 0 0 I 0.3 0.0 0:01.29 kworker/9:1
  706. 207 root 20 0 0 0 0 I 0.3 0.0 0:00.75 kworker/12:1
  707. 208 root 20 0 0 0 0 I 0.3 0.0 0:00.84 kworker/13:1
  708. 3163 root 20 0 0 0 0 S 0.3 0.0 0:00.40 jbd2/nvme3n1-8
  709. 8 root 20 0 0 0 0 I 0.2 0.0 0:00.68 rcu_sched
  710. 132 root 20 0 0 0 0 I 0.2 0.0 0:00.86 kworker/1:1
  711. 210 root 20 0 0 0 0 I 0.2 0.0 0:01.47 kworker/15:1
  712. 608 root 20 0 0 0 0 I 0.2 0.0 0:00.78 kworker/5:2
  713. 2411 root 20 0 0 0 0 I 0.2 0.0 0:00.82 kworker/7:2
  714. 2934 root 20 0 0 0 0 I 0.2 0.0 0:00.69 kworker/19:2
  715. 28 root 20 0 0 0 0 S 0.1 0.0 0:00.14 ksoftirqd/3
  716. 34 root 20 0 0 0 0 S 0.1 0.0 0:00.12 ksoftirqd/4
  717. 46 root 20 0 0 0 0 S 0.1 0.0 0:00.16 ksoftirqd/6
  718. 133 root 20 0 0 0 0 I 0.1 0.0 0:00.82 kworker/3:1
  719. 143 root 20 0 0 0 0 I 0.1 0.0 0:00.67 kworker/17:1
  720. 144 root 20 0 0 0 0 I 0.1 0.0 0:00.59 kworker/18:1
  721.  
  722.  
  723.  
  724.  
  725. root@alice:~# umount /data{0,1,2,3}
  726.  
  727. root@alice:~# mdadm --create md0 --level=raid0 --raid-devices=4 -f -v /dev/nvme{0,1,2,3}n1
  728. mdadm: chunk size defaults to 512K
  729. mdadm: /dev/nvme0n1 appears to contain an ext2fs file system
  730. size=1000204632K mtime=Mon May 14 10:48:14 2018
  731. mdadm: /dev/nvme1n1 appears to contain an ext2fs file system
  732. size=1000204632K mtime=Mon May 14 10:48:21 2018
  733. mdadm: /dev/nvme2n1 appears to contain an ext2fs file system
  734. size=1000204632K mtime=Mon May 14 10:48:26 2018
  735. mdadm: /dev/nvme3n1 appears to contain an ext2fs file system
  736. size=1000204632K mtime=Mon May 14 10:48:33 2018
  737. Continue creating array? y
  738. mdadm: Defaulting to version 1.2 metadata
  739. mdadm: array /dev/md/md0 started.
  740.  
  741. root@alice:~# cat /proc/mdstat
  742. Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
  743. md127 : active raid0 nvme3n1[3] nvme2n1[2] nvme1n1[1] nvme0n1[0]
  744. 4000292864 blocks super 1.2 512k chunks
  745.  
  746. unused devices: <none>
  747.  
  748. root@alice:~# mkfs.ext4 -F -v -b 4096 -E lazy_itable_init=0,lazy_journal_init=0 /dev/md/md0
  749. mke2fs 1.44.1 (24-Mar-2018)
  750. fs_types for mke2fs.conf resolution: 'ext4'
  751. Discarding device blocks: done
  752. Filesystem label=
  753. OS type: Linux
  754. Block size=4096 (log=2)
  755. Fragment size=4096 (log=2)
  756. Stride=128 blocks, Stripe width=512 blocks
  757. 250019840 inodes, 1000073216 blocks
  758. 50003660 blocks (5.00%) reserved for the super user
  759. First data block=0
  760. Maximum filesystem blocks=3147825152
  761. 30520 block groups
  762. 32768 blocks per group, 32768 fragments per group
  763. 8192 inodes per group
  764. Filesystem UUID: c5ef52e0-c75b-42d5-90f1-30328b769200
  765. Superblock backups stored on blocks:
  766. 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
  767. 4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968,
  768. 102400000, 214990848, 512000000, 550731776, 644972544
  769.  
  770. Allocating group tables: done
  771. Writing inode tables: done
  772. Creating journal (262144 blocks): done
  773. Writing superblocks and filesystem accounting information: done
  774.  
  775. root@alice:~# mount /dev/md/md0 /data0 -o lazytime,async
  776.  
  777. root@alice:~# cat ~ryan/fio-seq-write-raid-ext4.job
  778. ; fio-seq-write.job for fiotest
  779.  
  780. [global]
  781. ioengine=libaio
  782. iodepth=32
  783. rw=write
  784. bs=1024K
  785. direct=0
  786. size=256G
  787. numjobs=8
  788. time_based=1
  789. runtime=300
  790. fallocate=native
  791.  
  792. [raid0]
  793. name=fio-seq-write-0
  794. directory=/data0
  795.  
  796. root@alice:~# fio ~ryan/fio-seq-write-raid-ext4.job
  797. fio-seq-write-0: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=32
  798. ...
  799. fio-3.1
  800. Starting 8 processes
  801. fio-seq-write-0: Laying out IO file (1 file / 262144MiB)
  802. raid0: Laying out IO file (1 file / 262144MiB)
  803. raid0: Laying out IO file (1 file / 262144MiB)
  804. raid0: Laying out IO file (1 file / 262144MiB)
  805. raid0: Laying out IO file (1 file / 262144MiB)
  806. raid0: Laying out IO file (1 file / 262144MiB)
  807. raid0: Laying out IO file (1 file / 262144MiB)
  808. raid0: Laying out IO file (1 file / 262144MiB)
  809. Jobs: 8 (f=8): [W(8)][100.0%][r=0KiB/s,w=4396MiB/s][r=0,w=4396 IOPS][eta 00m:00s]
  810. fio-seq-write-0: (groupid=0, jobs=1): err= 0: pid=3481: Mon May 14 11:20:17 2018
  811. write: IOPS=568, BW=569MiB/s (597MB/s)(167GiB/300001msec)
  812. slat (usec): min=403, max=49298, avg=1754.82, stdev=5147.97
  813. clat (usec): min=3, max=142495, avg=54491.79, stdev=12041.55
  814. lat (usec): min=812, max=145060, avg=56246.94, stdev=12526.21
  815. clat percentiles (msec):
  816. | 1.00th=[ 26], 5.00th=[ 34], 10.00th=[ 47], 20.00th=[ 50],
  817. | 30.00th=[ 51], 40.00th=[ 51], 50.00th=[ 52], 60.00th=[ 54],
  818. | 70.00th=[ 56], 80.00th=[ 60], 90.00th=[ 75], 95.00th=[ 81],
  819. | 99.00th=[ 88], 99.50th=[ 90], 99.90th=[ 103], 99.95th=[ 109],
  820. | 99.99th=[ 122]
  821. bw ( KiB/s): min=432355, max=1083991, per=12.62%, avg=588309.56, stdev=47634.29, samples=600
  822. iops : min= 422, max= 1058, avg=573.97, stdev=46.50, samples=600
  823. lat (usec) : 4=0.01%, 1000=0.01%
  824. lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.12%, 50=34.22%
  825. lat (msec) : 100=65.54%, 250=0.12%
  826. cpu : usr=2.66%, sys=43.60%, ctx=18869, majf=0, minf=12
  827. IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
  828. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  829. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
  830. issued rwt: total=0,170662,0, short=0,0,0, dropped=0,0,0
  831. latency : target=0, window=0, percentile=100.00%, depth=32
  832. raid0: (groupid=0, jobs=1): err= 0: pid=3482: Mon May 14 11:20:17 2018
  833. write: IOPS=569, BW=570MiB/s (597MB/s)(167GiB/300001msec)
  834. slat (usec): min=397, max=49140, avg=1752.69, stdev=5238.94
  835. clat (usec): min=2, max=132907, avg=54423.84, stdev=11971.85
  836. lat (usec): min=594, max=137657, avg=56176.83, stdev=12495.21
  837. clat percentiles (msec):
  838. | 1.00th=[ 26], 5.00th=[ 35], 10.00th=[ 47], 20.00th=[ 50],
  839. | 30.00th=[ 50], 40.00th=[ 51], 50.00th=[ 52], 60.00th=[ 54],
  840. | 70.00th=[ 56], 80.00th=[ 60], 90.00th=[ 77], 95.00th=[ 81],
  841. | 99.00th=[ 88], 99.50th=[ 90], 99.90th=[ 96], 99.95th=[ 104],
  842. | 99.99th=[ 113]
  843. bw ( KiB/s): min=386844, max=1003313, per=12.63%, avg=589055.33, stdev=45642.49, samples=600
  844. iops : min= 377, max= 979, avg=574.68, stdev=44.57, samples=600
  845. lat (usec) : 4=0.01%, 750=0.01%
  846. lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.06%, 50=36.69%
  847. lat (msec) : 100=63.17%, 250=0.07%
  848. cpu : usr=2.59%, sys=42.15%, ctx=17106, majf=0, minf=10
  849. IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
  850. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  851. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
  852. issued rwt: total=0,170875,0, short=0,0,0, dropped=0,0,0
  853. latency : target=0, window=0, percentile=100.00%, depth=32
  854. raid0: (groupid=0, jobs=1): err= 0: pid=3483: Mon May 14 11:20:17 2018
  855. write: IOPS=568, BW=568MiB/s (596MB/s)(167GiB/300001msec)
  856. slat (usec): min=403, max=49422, avg=1756.13, stdev=5124.23
  857. clat (usec): min=2, max=130135, avg=54534.84, stdev=12175.53
  858. lat (usec): min=806, max=134780, avg=56291.31, stdev=12689.55
  859. clat percentiles (msec):
  860. | 1.00th=[ 26], 5.00th=[ 34], 10.00th=[ 47], 20.00th=[ 50],
  861. | 30.00th=[ 50], 40.00th=[ 51], 50.00th=[ 52], 60.00th=[ 54],
  862. | 70.00th=[ 56], 80.00th=[ 61], 90.00th=[ 77], 95.00th=[ 81],
  863. | 99.00th=[ 88], 99.50th=[ 90], 99.90th=[ 102], 99.95th=[ 111],
  864. | 99.99th=[ 125]
  865. bw ( KiB/s): min=417874, max=1017793, per=12.61%, avg=587863.39, stdev=49115.60, samples=600
  866. iops : min= 408, max= 993, avg=573.53, stdev=47.95, samples=600
  867. lat (usec) : 4=0.01%, 1000=0.01%
  868. lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.35%, 50=35.37%
  869. lat (msec) : 100=64.16%, 250=0.11%
  870. cpu : usr=2.77%, sys=44.16%, ctx=17730, majf=0, minf=10
  871. IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
  872. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  873. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
  874. issued rwt: total=0,170527,0, short=0,0,0, dropped=0,0,0
  875. latency : target=0, window=0, percentile=100.00%, depth=32
  876. raid0: (groupid=0, jobs=1): err= 0: pid=3484: Mon May 14 11:20:17 2018
  877. write: IOPS=567, BW=567MiB/s (595MB/s)(166GiB/300014msec)
  878. slat (usec): min=394, max=49830, avg=1759.28, stdev=5121.36
  879. clat (usec): min=5, max=118411, avg=54635.68, stdev=12092.74
  880. lat (msec): min=16, max=120, avg=56.40, stdev=12.61
  881. clat percentiles (msec):
  882. | 1.00th=[ 24], 5.00th=[ 35], 10.00th=[ 47], 20.00th=[ 50],
  883. | 30.00th=[ 50], 40.00th=[ 51], 50.00th=[ 52], 60.00th=[ 54],
  884. | 70.00th=[ 56], 80.00th=[ 61], 90.00th=[ 77], 95.00th=[ 81],
  885. | 99.00th=[ 88], 99.50th=[ 91], 99.90th=[ 105], 99.95th=[ 108],
  886. | 99.99th=[ 113]
  887. bw ( KiB/s): min=386844, max=978488, per=12.59%, avg=586918.18, stdev=49863.80, samples=600
  888. iops : min= 377, max= 955, avg=572.59, stdev=48.70, samples=600
  889. lat (usec) : 10=0.01%
  890. lat (msec) : 20=0.30%, 50=34.97%, 100=64.58%, 250=0.15%
  891. cpu : usr=2.76%, sys=44.30%, ctx=17569, majf=0, minf=12
  892. IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
  893. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  894. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
  895. issued rwt: total=0,170220,0, short=0,0,0, dropped=0,0,0
  896. latency : target=0, window=0, percentile=100.00%, depth=32
  897. raid0: (groupid=0, jobs=1): err= 0: pid=3485: Mon May 14 11:20:17 2018
  898. write: IOPS=568, BW=568MiB/s (596MB/s)(166GiB/300001msec)
  899. slat (usec): min=398, max=49686, avg=1756.51, stdev=5084.53
  900. clat (usec): min=2, max=142685, avg=54549.83, stdev=12402.19
  901. lat (usec): min=970, max=145828, avg=56306.68, stdev=12892.62
  902. clat percentiles (msec):
  903. | 1.00th=[ 24], 5.00th=[ 33], 10.00th=[ 47], 20.00th=[ 50],
  904. | 30.00th=[ 51], 40.00th=[ 51], 50.00th=[ 52], 60.00th=[ 54],
  905. | 70.00th=[ 57], 80.00th=[ 61], 90.00th=[ 77], 95.00th=[ 81],
  906. | 99.00th=[ 89], 99.50th=[ 91], 99.90th=[ 105], 99.95th=[ 113],
  907. | 99.99th=[ 128]
  908. bw ( KiB/s): min=412472, max=1212416, per=12.49%, avg=582434.68, stdev=55233.29, samples=600
  909. iops : min= 402, max= 1184, avg=568.68, stdev=53.96, samples=600
  910. lat (usec) : 4=0.01%, 1000=0.01%
  911. lat (msec) : 10=0.01%, 20=0.48%, 50=33.27%, 100=66.10%, 250=0.15%
  912. cpu : usr=2.78%, sys=44.74%, ctx=18847, majf=0, minf=11
  913. IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
  914. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  915. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
  916. issued rwt: total=0,170483,0, short=0,0,0, dropped=0,0,0
  917. latency : target=0, window=0, percentile=100.00%, depth=32
  918. raid0: (groupid=0, jobs=1): err= 0: pid=3486: Mon May 14 11:20:17 2018
  919. write: IOPS=569, BW=570MiB/s (598MB/s)(167GiB/300002msec)
  920. slat (usec): min=394, max=52439, avg=1751.55, stdev=5174.06
  921. clat (usec): min=2, max=116215, avg=54392.61, stdev=12059.90
  922. lat (usec): min=1351, max=117779, avg=56144.46, stdev=12570.11
  923. clat percentiles (msec):
  924. | 1.00th=[ 26], 5.00th=[ 34], 10.00th=[ 47], 20.00th=[ 50],
  925. | 30.00th=[ 50], 40.00th=[ 51], 50.00th=[ 52], 60.00th=[ 54],
  926. | 70.00th=[ 56], 80.00th=[ 60], 90.00th=[ 75], 95.00th=[ 81],
  927. | 99.00th=[ 88], 99.50th=[ 90], 99.90th=[ 96], 99.95th=[ 101],
  928. | 99.99th=[ 114]
  929. bw ( KiB/s): min=424080, max=1052961, per=12.64%, avg=589383.09, stdev=45070.10, samples=600
  930. iops : min= 414, max= 1028, avg=575.01, stdev=44.02, samples=600
  931. lat (usec) : 4=0.01%
  932. lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.21%, 50=36.48%
  933. lat (msec) : 100=63.26%, 250=0.05%
  934. cpu : usr=2.75%, sys=43.32%, ctx=17479, majf=0, minf=9
  935. IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
  936. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  937. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
  938. issued rwt: total=0,170974,0, short=0,0,0, dropped=0,0,0
  939. latency : target=0, window=0, percentile=100.00%, depth=32
  940. raid0: (groupid=0, jobs=1): err= 0: pid=3487: Mon May 14 11:20:17 2018
  941. write: IOPS=570, BW=570MiB/s (598MB/s)(167GiB/300001msec)
  942. slat (usec): min=396, max=49275, avg=1750.56, stdev=5142.09
  943. clat (usec): min=2, max=128664, avg=54361.51, stdev=12302.24
  944. lat (usec): min=664, max=132226, avg=56112.39, stdev=12821.49
  945. clat percentiles (msec):
  946. | 1.00th=[ 25], 5.00th=[ 33], 10.00th=[ 47], 20.00th=[ 50],
  947. | 30.00th=[ 50], 40.00th=[ 51], 50.00th=[ 52], 60.00th=[ 54],
  948. | 70.00th=[ 56], 80.00th=[ 60], 90.00th=[ 75], 95.00th=[ 81],
  949. | 99.00th=[ 88], 99.50th=[ 91], 99.90th=[ 106], 99.95th=[ 112],
  950. | 99.99th=[ 121]
  951. bw ( KiB/s): min=386844, max=1075717, per=12.65%, avg=589735.88, stdev=52777.49, samples=600
  952. iops : min= 377, max= 1050, avg=575.35, stdev=51.54, samples=600
  953. lat (usec) : 4=0.01%, 750=0.01%
  954. lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.27%, 50=36.40%
  955. lat (msec) : 100=63.16%, 250=0.16%
  956. cpu : usr=2.87%, sys=43.66%, ctx=17657, majf=0, minf=11
  957. IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
  958. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  959. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
  960. issued rwt: total=0,171071,0, short=0,0,0, dropped=0,0,0
  961. latency : target=0, window=0, percentile=100.00%, depth=32
  962. raid0: (groupid=0, jobs=1): err= 0: pid=3488: Mon May 14 11:20:17 2018
  963. write: IOPS=570, BW=570MiB/s (598MB/s)(167GiB/300013msec)
  964. slat (usec): min=403, max=49869, avg=1749.96, stdev=5182.28
  965. clat (usec): min=5, max=142945, avg=54342.17, stdev=11973.60
  966. lat (msec): min=16, max=146, avg=56.09, stdev=12.48
  967. clat percentiles (msec):
  968. | 1.00th=[ 26], 5.00th=[ 34], 10.00th=[ 47], 20.00th=[ 50],
  969. | 30.00th=[ 50], 40.00th=[ 51], 50.00th=[ 52], 60.00th=[ 54],
  970. | 70.00th=[ 56], 80.00th=[ 60], 90.00th=[ 75], 95.00th=[ 81],
  971. | 99.00th=[ 88], 99.50th=[ 90], 99.90th=[ 99], 99.95th=[ 108],
  972. | 99.99th=[ 124]
  973. bw ( KiB/s): min=397187, max=1063305, per=12.66%, avg=590072.26, stdev=44471.04, samples=600
  974. iops : min= 387, max= 1038, avg=575.68, stdev=43.42, samples=600
  975. lat (usec) : 10=0.01%
  976. lat (msec) : 20=0.19%, 50=35.92%, 100=63.81%, 250=0.08%
  977. cpu : usr=2.78%, sys=42.98%, ctx=17195, majf=0, minf=11
  978. IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
  979. submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  980. complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
  981. issued rwt: total=0,171139,0, short=0,0,0, dropped=0,0,0
  982. latency : target=0, window=0, percentile=100.00%, depth=32
  983.  
  984. Run status group 0 (all jobs):
  985. WRITE: bw=4553MiB/s (4774MB/s), 567MiB/s-570MiB/s (595MB/s-598MB/s), io=1334GiB (1432GB), run=300001-300014msec
  986.  
  987. Disk stats (read/write):
  988. md127: ios=0/3077037, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=0/371121, aggrmerge=0/398989, aggrticks=0/479853, aggrin_queue=19987, aggrutil=6.20%
  989. nvme2n1: ios=0/371220, merge=0/394914, ticks=0/456492, in_queue=15252, util=2.95%
  990. nvme1n1: ios=0/369547, merge=0/396226, ticks=0/459500, in_queue=14588, util=2.95%
  991. nvme0n1: ios=0/373542, merge=0/407979, ticks=0/466192, in_queue=24972, util=3.00%
  992. nvme3n1: ios=0/370177, merge=0/396838, ticks=0/537228, in_queue=25136, util=6.20%
  993.  
  994. top -d 10 during fio run:
  995. top - 11:17:37 up 1:06, 4 users, load average: 9.13, 3.95, 2.64
  996. Tasks: 355 total, 6 running, 225 sleeping, 0 stopped, 0 zombie
  997. %Cpu(s): 1.1 us, 25.4 sy, 0.0 ni, 52.5 id, 20.9 wa, 0.0 hi, 0.1 si, 0.0 st
  998. KiB Mem : 23252352 total, 399740 free, 1042156 used, 21810456 buff/cache
  999. KiB Swap: 2097148 total, 2097148 free, 0 used. 21553788 avail Mem
  1000.  
  1001. PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
  1002. 3027 root 20 0 0 0 0 R 99.2 0.0 6:19.05 kworker/u488:3
  1003. 158 root 20 0 0 0 0 S 85.0 0.0 11:33.52 kswapd0
  1004. 3487 root 20 0 745472 37580 1108 R 54.7 0.2 1:10.13 fio
  1005. 3481 root 20 0 745448 37568 1108 R 45.0 0.2 1:07.82 fio
  1006. 3485 root 20 0 745464 37572 1112 D 44.5 0.2 1:08.99 fio
  1007. 3483 root 20 0 745456 37572 1112 D 44.0 0.2 1:10.01 fio
  1008. 3486 root 20 0 745468 37572 1112 D 43.9 0.2 1:09.24 fio
  1009. 3484 root 20 0 745460 37564 1104 D 43.7 0.2 1:10.95 fio
  1010. 3488 root 20 0 745476 37584 1112 R 43.7 0.2 1:07.64 fio
  1011. 3482 root 20 0 745452 37572 1112 D 42.3 0.2 1:06.93 fio
  1012. 3447 root 20 0 0 0 0 S 1.1 0.0 0:01.09 jbd2/md127-8
  1013. 3459 root 20 0 712640 274120 270436 S 0.7 1.2 0:02.62 fio
  1014. 203 root 20 0 0 0 0 I 0.4 0.0 0:02.51 kworker/8:1
  1015. 3450 root 20 0 0 0 0 I 0.3 0.0 0:00.19 kworker/5:0
  1016. 8 root 20 0 0 0 0 I 0.2 0.0 0:01.45 rcu_sched
  1017. 147 root 20 0 0 0 0 I 0.2 0.0 0:03.05 kworker/4:1
  1018. 7 root 20 0 0 0 0 S 0.1 0.0 0:00.35 ksoftirqd/0
  1019. 46 root 20 0 0 0 0 S 0.1 0.0 0:00.33 ksoftirqd/6
  1020. 52 root 20 0 0 0 0 S 0.1 0.0 0:00.27 ksoftirqd/7
  1021. 64 root 20 0 0 0 0 S 0.1 0.0 0:00.26 ksoftirqd/9
  1022. 143 root 20 0 0 0 0 I 0.1 0.0 0:01.18 kworker/17:1
  1023. 146 root 20 0 0 0 0 I 0.1 0.0 0:03.34 kworker/2:1
  1024. 149 root 20 0 0 0 0 I 0.1 0.0 0:04.54 kworker/0:2
  1025. 156 root 20 0 0 0 0 I 0.1 0.0 0:02.92 kworker/6:1
  1026. 204 root 20 0 0 0 0 I 0.1 0.0 0:02.61 kworker/9:1
  1027. 206 root 20 0 0 0 0 I 0.1 0.0 0:02.90 kworker/11:1
  1028. 1433 root -51 0 0 0 0 S 0.1 0.0 0:02.25 irq/236-nvidia
  1029. 2411 root 20 0 0 0 0 I 0.1 0.0 0:01.66 kworker/7:2
  1030. 2934 root 20 0 0 0 0 I 0.1 0.0 0:01.36 kworker/19:2
  1031. 3213 ryan 20 0 51464 4340 3480 R 0.1 0.0 0:03.25 top
  1032. 3451 root 20 0 0 0 0 I 0.1 0.0 0:00.10 kworker/10:0
  1033. 3499 root 20 0 0 0 0 I 0.1 0.0 0:00.04 kworker/12:0
  1034. 1 root 20 0 225464 9340 6792 S 0.0 0.0 0:02.33 systemd
  1035. 2 root 20 0 0 0 0 S 0.0 0.0 0:00.01 kthreadd
  1036. 4 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 kworker/0:0H
  1037. 6 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 mm_percpu_wq
  1038. 9 root 20 0 0 0 0 I 0.0 0.0 0:00.00 rcu_bh
  1039. 10 root rt 0 0 0 0 S 0.0 0.0 0:00.71 migration/0
  1040. 11 root rt 0 0 0 0 S 0.0 0.0 0:00.02 watchdog/0
  1041. 12 root 20 0 0 0 0 S 0.0 0.0 0:00.00 cpuhp/0
  1042.  
  1043. root@alice:~#
Add Comment
Please, Sign In to add comment
Public Pastes
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!