API tools faq
paste
xartin

zfs degraded without an slog

xartin
Aug 11th, 2020 (edited)
140
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 17.20 KB | None | 0 0
raw download clone embed print report
  1. zfs pool degraded without an slog due to sata command timeouts.
  2.  
  3. fenrir ~ # zpool status
  4. pool: home
  5. state: DEGRADED
  6. status: One or more devices has experienced an unrecoverable error. An
  7. attempt was made to correct the error. Applications are unaffected.
  8. action: Determine if the device needs to be replaced, and clear the errors
  9. using 'zpool clear' or replace the device with 'zpool replace'.
  10. see: http://zfsonlinux.org/msg/ZFS-8000-9P
  11. scan: scrub repaired 228K in 3h5m with 0 errors on Sat Nov 24 08:06:24 2018
  12. config:
  13.  
  14. NAME STATE READ WRITE CKSUM
  15. home DEGRADED 0 0 0
  16. raidz1-0 DEGRADED 0 0 0
  17. ata-ST10000NE0004-xxxxxx_xxxxxxxx ONLINE 0 0 0
  18. ata-ST10000NE0004-xxxxxx_xxxxxxxx ONLINE 0 0 0
  19. ata-ST10000NE0004-xxxxxx_xxxxxxxx ONLINE 0 0 0
  20. ata-ST10000NE0004-xxxxxx_xxxxxxxx ONLINE 0 0 0
  21. ata-ST10000NE0004-xxxxxx_xxxxxxxx ONLINE 0 0 0
  22. ata-ST10000NE0004-xxxxxx_xxxxxxxx ONLINE 0 0 0
  23. ata-ST10000NE0004-xxxxxx_xxxxxx11 DEGRADED 0 0 12 too many errors
  24. ata-ST10000NE0004-xxxxxx_xxxxxxxx ONLINE 0 0 0
  25.  
  26. cmd_to: Show SMART command timeout count (ATA).
  27.  
  28.  
  29. fenrir ~ # ZPOOL_SCRIPTS_AS_ROOT=yes zpool status -c smart
  30. pool: home
  31. state: ONLINE
  32. scan: scrub repaired 0B in 1h20m with 0 errors on Sat Nov 24 11:21:54 2018
  33. config:
  34.  
  35. NAME STATE READ WRITE CKSUM health realloc rep_ucor cmd_to temp
  36. home ONLINE 0 0 0
  37. raidz1-0 ONLINE 0 0 0
  38. ata-ST10000NE0004-xxxxxx_xxxxxxxx ONLINE 0 0 0 PASSED 0 0 0 34
  39. ata-ST10000NE0004-xxxxxx_xxxxxxxx ONLINE 0 0 0 PASSED 0 0 0 34
  40. ata-ST10000NE0004-xxxxxx_xxxxxxxx ONLINE 0 0 0 PASSED 0 0 0 33
  41. ata-ST10000NE0004-xxxxxx_xxxxxxxx ONLINE 0 0 0 PASSED 8 0 0 34
  42. ata-ST10000NE0004-xxxxxx_xxxxxxxx ONLINE 0 0 0 PASSED 8 0 8590065844 32
  43. ata-ST10000NE0004-xxxxxx_xxxxxxxx ONLINE 0 0 0 PASSED 0 0 12885098677 34
  44. ata-ST10000NE0004-xxxxxx_xxxxxx11 ONLINE 0 0 0 PASSED 8 0 8590065707 32
  45. ata-ST10000NE0004-xxxxxx_xxxxxxxx ONLINE 0 0 0 PASSED 0 0 12885098699 33
  46.  
  47.  
  48. fenrir ~ # ZPOOL_SCRIPTS_AS_ROOT=yes zpool status -c hours_on
  49. pool: home
  50. state: ONLINE
  51. scan: scrub repaired 0B in 1h20m with 0 errors on Sat Nov 24 11:21:54 2018
  52. config:
  53.  
  54. NAME STATE READ WRITE CKSUM hours_on
  55. home ONLINE 0 0 0
  56. raidz1-0 ONLINE 0 0 0
  57. ata-ST10000NE0004-xxxxxx_xxxxxxxx ONLINE 0 0 0 7510
  58. ata-ST10000NE0004-xxxxxx_xxxxxxxx ONLINE 0 0 0 7510
  59. ata-ST10000NE0004-xxxxxx_xxxxxxxx ONLINE 0 0 0 7510
  60. ata-ST10000NE0004-xxxxxx_xxxxxxxx ONLINE 0 0 0 7510
  61. ata-ST10000NE0004-xxxxxx_xxxxxxxx ONLINE 0 0 0 8550
  62. ata-ST10000NE0004-xxxxxx_xxxxxxxx ONLINE 0 0 0 8550
  63. ata-ST10000NE0004-xxxxxx_xxxxxx11 ONLINE 0 0 0 8663
  64. ata-ST10000NE0004-xxxxxx_xxxxxxxx ONLINE 0 0 0 8663
  65.  
  66.  
  67. [35865.372286] sd 6:0:7:0: attempting task abort! scmd(000000001b20cfe8)
  68. [35865.372321] sd 6:0:7:0: [sdi] tag#9 CDB: Write(16) 8a 00 00 00 00 00 50 21 79 c0 00 00 00 08 00 00
  69. [35865.372353] scsi target6:0:7: handle(0x0020), sas_address(0x300062b20394b747), phy(7)
  70. [35865.372378] scsi target6:0:7: enclosure logical id(0x500062b20394b740), slot(5)
  71. [35865.372402] scsi target6:0:7: enclosure level(0x0000), connector name( )
  72. [35865.439866] sd 6:0:7:0: task abort: SUCCESS scmd(000000001b20cfe8)
  73. [35865.439895] sd 6:0:7:0: [sdi] tag#9 FAILED Result: hostbyte=DID_TIME_OUT driverbyte=DRIVER_OK
  74. [35865.439922] sd 6:0:7:0: [sdi] tag#9 CDB: Write(16) 8a 00 00 00 00 00 50 21 79 c0 00 00 00 08 00 00
  75. [35865.439952] print_req_error: I/O error, dev sdi, sector 1344371136
  76. [35865.439989] sd 6:0:7:0: attempting task abort! scmd(0000000028865f67)
  77. [35865.440011] sd 6:0:7:0: [sdi] tag#5 CDB: Write(16) 8a 00 00 00 00 03 28 f0 a4 10 00 00 00 08 00 00
  78. [35865.440041] scsi target6:0:7: handle(0x0020), sas_address(0x300062b20394b747), phy(7)
  79. [35865.440066] scsi target6:0:7: enclosure logical id(0x500062b20394b740), slot(5)
  80. [35865.440090] scsi target6:0:7: enclosure level(0x0000), connector name( )
  81. [35865.440113] sd 6:0:7:0: task abort: SUCCESS scmd(0000000028865f67)
  82. [35865.440135] sd 6:0:7:0: [sdi] tag#5 FAILED Result: hostbyte=DID_TIME_OUT driverbyte=DRIVER_OK
  83. [35865.440161] sd 6:0:7:0: [sdi] tag#5 CDB: Write(16) 8a 00 00 00 00 03 28 f0 a4 10 00 00 00 08 00 00
  84. [35865.440190] print_req_error: I/O error, dev sdi, sector 13571761168
  85. [35866.184214] sd 6:0:7:0: Power-on or device reset occurred
  86. [81619.335173] sd 6:0:6:0: attempting task abort! scmd(00000000d094d820)
  87. [81619.335208] sd 6:0:6:0: [sdh] tag#2 CDB: ATA command pass through(16) 85 06 20 00 da 00 00 00 00 00 4f 00 c2 00 b0 00
  88. [81619.335244] scsi target6:0:6: handle(0x001f), sas_address(0x300062b20394b746), phy(6)
  89. [81619.336184] scsi target6:0:6: enclosure logical id(0x500062b20394b740), slot(4)
  90. [81619.337133] scsi target6:0:6: enclosure level(0x0000), connector name( )
  91. [81619.433174] sd 6:0:6:0: [sdh] tag#1 FAILED Result: hostbyte=DID_SOFT_ERROR driverbyte=DRIVER_OK
  92. [81619.434132] sd 6:0:6:0: [sdh] tag#1 CDB: Write(16) 8a 00 00 00 00 00 80 06 0b 78 00 00 00 e8 00 00
  93. [81619.434150] sd 6:0:6:0: task abort: SUCCESS scmd(00000000d094d820)
  94. [81619.436033] print_req_error: I/O error, dev sdh, sector 2147879800
  95. [81619.436057] sd 6:0:6:0: [sdh] tag#0 FAILED Result: hostbyte=DID_SOFT_ERROR driverbyte=DRIVER_OK
  96. [81619.439003] sd 6:0:6:0: [sdh] tag#0 CDB: Write(16) 8a 00 00 00 00 00 80 06 0a 98 00 00 00 e0 00 00
  97. [81619.441024] print_req_error: I/O error, dev sdh, sector 2147879576
  98. [81620.047094] sd 6:0:6:0: Power-on or device reset occurred
  99. [85242.164294] sd 6:0:5:0: attempting task abort! scmd(000000007322b4b0)
  100. [85242.165311] sd 6:0:5:0: tag#3 CDB: Test Unit Ready 00 00 00 00 00 00
  101. [85242.166318] scsi target6:0:5: handle(0x001e), sas_address(0x300062b20394b745), phy(5)
  102. [85242.167334] scsi target6:0:5: enclosure logical id(0x500062b20394b740), slot(6)
  103. [85242.168351] scsi target6:0:5: enclosure level(0x0000), connector name( )
  104. [85242.227972] sd 6:0:5:0: task abort: SUCCESS scmd(000000007322b4b0)
  105. [85242.228993] sd 6:0:5:0: attempting task abort! scmd(000000001a81db8a)
  106. [85242.229969] sd 6:0:5:0: [sdg] tag#1 CDB: Write(16) 8a 00 00 00 00 00 83 79 4f e8 00 00 00 e8 00 00
  107. [85242.231972] scsi target6:0:5: handle(0x001e), sas_address(0x300062b20394b745), phy(5)
  108. [85242.232985] scsi target6:0:5: enclosure logical id(0x500062b20394b740), slot(6)
  109. [85242.233965] scsi target6:0:5: enclosure level(0x0000), connector name( )
  110. [85242.234941] sd 6:0:5:0: task abort: SUCCESS scmd(000000001a81db8a)
  111. [85242.235917] sd 6:0:5:0: [sdg] tag#1 FAILED Result: hostbyte=DID_TIME_OUT driverbyte=DRIVER_OK
  112. [85242.236909] sd 6:0:5:0: [sdg] tag#1 CDB: Write(16) 8a 00 00 00 00 00 83 79 4f e8 00 00 00 e8 00 00
  113. [85242.238902] print_req_error: I/O error, dev sdg, sector 2205765608
  114. [85242.239937] sd 6:0:5:0: attempting task abort! scmd(000000007d7e8a77)
  115. [85242.240928] sd 6:0:5:0: [sdg] tag#0 CDB: Write(16) 8a 00 00 00 00 00 83 79 4b e8 00 00 00 48 00 00
  116. [85242.242953] scsi target6:0:5: handle(0x001e), sas_address(0x300062b20394b745), phy(5)
  117. [85242.243988] scsi target6:0:5: enclosure logical id(0x500062b20394b740), slot(6)
  118. [85242.245009] scsi target6:0:5: enclosure level(0x0000), connector name( )
  119. [85242.245996] sd 6:0:5:0: task abort: SUCCESS scmd(000000007d7e8a77)
  120. [85242.246977] sd 6:0:5:0: [sdg] tag#0 FAILED Result: hostbyte=DID_TIME_OUT driverbyte=DRIVER_OK
  121. [85242.247973] sd 6:0:5:0: [sdg] tag#0 CDB: Write(16) 8a 00 00 00 00 00 83 79 4b e8 00 00 00 48 00 00
  122. [85242.249972] print_req_error: I/O error, dev sdg, sector 2205764584
  123. [85242.801292] sd 6:0:5:0: Power-on or device reset occurred
  124. [99142.715879] mpt3sas_cm0: log_info(0x30030109): originator(IOP), code(0x03), sub_code(0x0109)
  125. [99142.723804] mpt3sas_cm0: log_info(0x30030109): originator(IOP), code(0x03), sub_code(0x0109)
  126. [99142.741737] mpt3sas_cm0: log_info(0x30030109): originator(IOP), code(0x03), sub_code(0x0109)
  127. [99142.771173] mpt3sas_cm0: log_info(0x30030109): originator(IOP), code(0x03), sub_code(0x0109)
  128. [99142.776213] mpt3sas_cm0: log_info(0x30030109): originator(IOP), code(0x03), sub_code(0x0109)
  129. [99142.779206] mpt3sas_cm0: log_info(0x30030109): originator(IOP), code(0x03), sub_code(0x0109)
  130. [99142.784336] mpt3sas_cm0: log_info(0x30030109): originator(IOP), code(0x03), sub_code(0x0109)
  131. [99142.787268] mpt3sas_cm0: log_info(0x30030109): originator(IOP), code(0x03), sub_code(0x0109)
  132. [99142.790180] mpt3sas_cm0: log_info(0x30030109): originator(IOP), code(0x03), sub_code(0x0109)
  133. [99142.793216] mpt3sas_cm0: log_info(0x30030109): originator(IOP), code(0x03), sub_code(0x0109)
  134. [99142.796265] mpt3sas_cm0: log_info(0x30030109): originator(IOP), code(0x03), sub_code(0x0109)
  135. [99142.799406] mpt3sas_cm0: log_info(0x30030109): originator(IOP), code(0x03), sub_code(0x0109)
  136. [99142.802272] mpt3sas_cm0: log_info(0x30030109): originator(IOP), code(0x03), sub_code(0x0109)
  137. [99142.805236] mpt3sas_cm0: log_info(0x30030109): originator(IOP), code(0x03), sub_code(0x0109)
  138. [99142.808046] mpt3sas_cm0: log_info(0x30030109): originator(IOP), code(0x03), sub_code(0x0109)
  139. [99142.810910] mpt3sas_cm0: log_info(0x30030109): originator(IOP), code(0x03), sub_code(0x0109)
  140. [99142.811572] mpt3sas_cm0: log_info(0x30030109): originator(IOP), code(0x03), sub_code(0x0109)
  141. [99142.824432] mpt3sas_cm0: log_info(0x30030109): originator(IOP), code(0x03), sub_code(0x0109)
  142. [99142.833663] mpt3sas_cm0: log_info(0x30030109): originator(IOP), code(0x03), sub_code(0x0109)
  143. [99142.845982] mpt3sas_cm0: log_info(0x30030109): originator(IOP), code(0x03), sub_code(0x0109)
  144. [99142.853093] mpt3sas_cm0: log_info(0x30030109): originator(IOP), code(0x03), sub_code(0x0109)
  145. [99142.853675] mpt3sas_cm0: log_info(0x30030109): originator(IOP), code(0x03), sub_code(0x0109)
  146. [99142.866455] mpt3sas_cm0: log_info(0x30030109): originator(IOP), code(0x03), sub_code(0x0109)
  147. [99142.875667] mpt3sas_cm0: log_info(0x30030109): originator(IOP), code(0x03), sub_code(0x0109)
  148. [99142.887911] mpt3sas_cm0: log_info(0x30030109): originator(IOP), code(0x03), sub_code(0x0109)
  149. [99142.894930] mpt3sas_cm0: log_info(0x30030109): originator(IOP), code(0x03), sub_code(0x0109)
  150. [99142.895398] mpt3sas_cm0: log_info(0x30030109): originator(IOP), code(0x03), sub_code(0x0109)
  151. [99142.908028] mpt3sas_cm0: log_info(0x30030109): originator(IOP), code(0x03), sub_code(0x0109)
  152. [99142.917025] mpt3sas_cm0: log_info(0x30030109): originator(IOP), code(0x03), sub_code(0x0109)
  153. [99142.929155] mpt3sas_cm0: log_info(0x30030109): originator(IOP), code(0x03), sub_code(0x0109)
  154. [99142.936157] mpt3sas_cm0: log_info(0x30030109): originator(IOP), code(0x03), sub_code(0x0109)
  155. [99142.936534] mpt3sas_cm0: log_info(0x30030109): originator(IOP), code(0x03), sub_code(0x0109)
  156. [99142.948944] mpt3sas_cm0: log_info(0x30030109): originator(IOP), code(0x03), sub_code(0x0109)
  157. [99142.957880] mpt3sas_cm0: log_info(0x30030109): originator(IOP), code(0x03), sub_code(0x0109)
  158. [99142.969947] mpt3sas_cm0: log_info(0x30030109): originator(IOP), code(0x03), sub_code(0x0109)
  159. [99142.976886] mpt3sas_cm0: log_info(0x30030109): originator(IOP), code(0x03), sub_code(0x0109)
  160. [99142.977196] mpt3sas_cm0: log_info(0x30030109): originator(IOP), code(0x03), sub_code(0x0109)
  161. [99142.990062] mpt3sas_cm0: log_info(0x30030109): originator(IOP), code(0x03), sub_code(0x0109)
  162. [99142.999388] mpt3sas_cm0: log_info(0x30030109): originator(IOP), code(0x03), sub_code(0x0109)
  163. [99143.045115] mpt3sas_cm0: log_info(0x30030109): originator(IOP), code(0x03), sub_code(0x0109)
  164. [99143.056026] mpt3sas_cm0: log_info(0x30030109): originator(IOP), code(0x03), sub_code(0x0109)
  165. [99143.058280] mpt3sas_cm0: log_info(0x30030109): originator(IOP), code(0x03), sub_code(0x0109)
  166. [99143.076612] mpt3sas_cm0: log_info(0x30030109): originator(IOP), code(0x03), sub_code(0x0109)
  167. [99143.085493] mpt3sas_cm0: log_info(0x30030109): originator(IOP), code(0x03), sub_code(0x0109)
  168. [99143.097510] mpt3sas_cm0: log_info(0x30030109): originator(IOP), code(0x03), sub_code(0x0109)
  169. [99143.104314] mpt3sas_cm0: log_info(0x30030109): originator(IOP), code(0x03), sub_code(0x0109)
  170. [99143.104565] mpt3sas_cm0: log_info(0x30030109): originator(IOP), code(0x03), sub_code(0x0109)
  171. [99143.116859] mpt3sas_cm0: log_info(0x30030109): originator(IOP), code(0x03), sub_code(0x0109)
  172. [99143.125696] mpt3sas_cm0: log_info(0x30030109): originator(IOP), code(0x03), sub_code(0x0109)
  173. [99143.137758] mpt3sas_cm0: log_info(0x30030109): originator(IOP), code(0x03), sub_code(0x0109)
  174. [99143.144566] mpt3sas_cm0: log_info(0x30030109): originator(IOP), code(0x03), sub_code(0x0109)
  175. [99143.144819] mpt3sas_cm0: log_info(0x30030109): originator(IOP), code(0x03), sub_code(0x0109)
  176. [99143.157070] mpt3sas_cm0: log_info(0x30030109): originator(IOP), code(0x03), sub_code(0x0109)
  177. [99143.165876] mpt3sas_cm0: log_info(0x30030109): originator(IOP), code(0x03), sub_code(0x0109)
  178. [99143.177846] mpt3sas_cm0: log_info(0x30030109): originator(IOP), code(0x03), sub_code(0x0109)
  179. [103219.663536] sd 6:0:6:0: attempting task abort! scmd(00000000935b6ab1)
  180. [103219.663682] sd 6:0:6:0: [sdh] tag#4 CDB: ATA command pass through(16) 85 06 20 00 da 00 00 00 00 00 4f 00 c2 00 b0 00
  181. [103219.664014] scsi target6:0:6: handle(0x001f), sas_address(0x300062b20394b746), phy(6)
  182. [103219.664223] scsi target6:0:6: enclosure logical id(0x500062b20394b740), slot(4)
  183. [103219.664442] scsi target6:0:6: enclosure level(0x0000), connector name( )
  184. [103219.737975] sd 6:0:6:0: [sdh] tag#2 FAILED Result: hostbyte=DID_SOFT_ERROR driverbyte=DRIVER_OK
  185. [103219.737979] sd 6:0:6:0: [sdh] tag#3 FAILED Result: hostbyte=DID_SOFT_ERROR driverbyte=DRIVER_OK
  186. [103219.737981] sd 6:0:6:0: [sdh] tag#0 FAILED Result: hostbyte=DID_SOFT_ERROR driverbyte=DRIVER_OK
  187. [103219.737986] sd 6:0:6:0: [sdh] tag#0 CDB: Write(16) 8a 00 00 00 00 03 51 b9 49 80 00 00 00 e0 00 00
  188. [103219.737990] print_req_error: I/O error, dev sdh, sector 14255999360
  189. [103219.738231] sd 6:0:6:0: [sdh] tag#2 CDB: Read(16) 88 00 00 00 00 03 51 9f 5a 30 00 00 00 48 00 00
  190. [103219.738235] print_req_error: I/O error, dev sdh, sector 14254299696
  191. [103219.738251] sd 6:0:6:0: task abort: SUCCESS scmd(00000000935b6ab1)
  192. [103219.738550] sd 6:0:6:0: [sdh] tag#3 CDB: Write(16) 8a 00 00 00 00 03 51 b9 4a 60 00 00 00 e8 00 00
  193. [103219.742725] print_req_error: I/O error, dev sdh, sector 14255999584
  194. [103220.440342] sd 6:0:6:0: Power-on or device reset occurred
  195. [103969.730291] sd 6:0:4:0: attempting task abort! scmd(000000006362b3c6)
  196. [103969.730798] sd 6:0:4:0: [sdf] tag#16 CDB: Write(16) 8a 00 00 00 00 03 52 3b ea c8 00 00 00 28 00 00
  197. [103969.731865] scsi target6:0:4: handle(0x001d), sas_address(0x300062b20394b744), phy(4)
  198. [103969.732441] scsi target6:0:4: enclosure logical id(0x500062b20394b740), slot(7)
  199. [103969.733027] scsi target6:0:4: enclosure level(0x0000), connector name( )
  200. [103969.797147] sd 6:0:4:0: task abort: SUCCESS scmd(000000006362b3c6)
  201. [103969.797759] sd 6:0:4:0: [sdf] tag#16 FAILED Result: hostbyte=DID_TIME_OUT driverbyte=DRIVER_OK
  202. [103969.798387] sd 6:0:4:0: [sdf] tag#16 CDB: Write(16) 8a 00 00 00 00 03 52 3b ea c8 00 00 00 28 00 00
  203. [103969.799676] print_req_error: I/O error, dev sdf, sector 14264560328
  204. [103969.800372] sd 6:0:4:0: attempting task abort! scmd(00000000a1dae840)
  205. [103969.801070] sd 6:0:4:0: [sdf] tag#7 CDB: Write(16) 8a 00 00 00 00 03 52 3b ea a0 00 00 00 28 00 00
  206. [103969.802532] scsi target6:0:4: handle(0x001d), sas_address(0x300062b20394b744), phy(4)
  207. [103969.803306] scsi target6:0:4: enclosure logical id(0x500062b20394b740), slot(7)
  208. [103969.804091] scsi target6:0:4: enclosure level(0x0000), connector name( )
  209. [103969.804868] sd 6:0:4:0: task abort: SUCCESS scmd(00000000a1dae840)
  210. [103969.805634] sd 6:0:4:0: [sdf] tag#7 FAILED Result: hostbyte=DID_TIME_OUT driverbyte=DRIVER_OK
  211. [103969.806393] sd 6:0:4:0: [sdf] tag#7 CDB: Write(16) 8a 00 00 00 00 03 52 3b ea a0 00 00 00 28 00 00
  212. [103969.807888] print_req_error: I/O error, dev sdf, sector 14264560288
  213. [103970.433964] sd 6:0:4:0: Power-on or device reset occurred
  214. [154624.760065] sdh: sdh1 sdh9
  215.  
  216. ~/lsi_decode_loginfo $ ./lsi_decode_loginfo.py 0x30030109
  217. Value 30030109h
  218. Type: 30000000h SAS
  219. Origin: 00000000h IOP
  220. Code: 00030000h IOP_LOGINFO_CODE_CONFIG_INVALID_PAGE
  221. Sub Code: 00000100h IOP_LOGINFO_CODE_CONFIG_INVALID_PAGE_RT Route Table Entry not found
  222. unknown 00000009h unknown
Add Comment
Please, Sign In to add comment
Public Pastes
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!