API tools faq
paste
Advertisement
mhammett

ZFS woes post import

mhammett
Mar 11th, 2013
222
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
Bash 15.30 KB | None | 0 0
raw download clone embed print report
  1. I run NexentaStor and had some problems. If I told you everything that has happened, I would be typing for an hour. The short of it...  when pulling a dead drive to RMA, I bumped the power on another, losing the array. I have spent a couple days trying to bring it back, but no luck thus far. Suspecting controller issues, I moved all of the disks to another controller. I have tried different things I have found on Google as well as help from the NexentaStor forum, HardOCP forum and #openIndiana on FreeNode.
  2.  
  3. It processed an import for about 2.25 days that ultimately failed.
  4.  
  5.  
  6. Here is top showing that stuff is actually happening
  7. ==========
  8. load averages:  0.02,  0.02,  0.05;               up 2+12:13:38                                                                                                                                       07:52:54
  9. 108 processes: 106 sleeping, 1 zombie, 1 on cpu
  10. CPU states: 99.0% idle,  0.0% user,  0.9% kernel,  0.0% iowait,  0.0% swap
  11. Kernel: 683 ctxsw, 3 trap, 3986 intr, 504 syscall, 3 flt
  12. Memory: 16G phys mem, 4905M free mem, 1024M total swap, 1024M free swap
  13.  
  14.    PID USERNAME NLWP PRI NICE  SIZE   RES STATE    TIME    CPU COMMAND
  15.  25823 root        1  59    0 2552K 2004K sleep    6:49  0.04% iostat
  16.   7672 root       55  59    0   34M   31M sleep    0:16  0.04% python2.5
  17.  17172 root        1  59    0 3058M 3057M cpu/1    5:47  0.02% top
  18.    589 root       20  59    0   11M 6924K sleep    0:31  0.01% smbd
  19.    930 root        1  59    0   36M 3768K sleep    0:50  0.01% nmdtrace
  20.   2822 root        1  59    0   68M   31M sleep    9:05  0.01% nms
  21.  10062 root        1  59    0 7732K 5216K sleep    0:44  0.00% sshd
  22.    613 root        6  59    0    0K    0K sleep    0:57  0.00% archttp64
  23.    603 root       10  59    0 8056K 6200K sleep    5:51  0.00% syslogd
  24.     11 root       19  59    0   12M   11M sleep    3:29  0.00% svc.configd
  25.  24561 root        1  59    0 3600K 1680K sleep    0:27  0.00% tail
  26.    221 root       31  59    0 7992K 5200K sleep    0:08  0.00% nscd
  27.    284 root        1 100  -20 2728K 1584K sleep    0:12  0.00% xntpd
  28.   6078 root        1  59    0 7732K 5216K sleep    0:32  0.00% sshd
  29.    809 root        1  59    0   16M 6560K sleep    0:07  0.00% apache2
  30.   8281 root        1  59    0 7732K 5216K sleep    0:06  0.00% sshd
  31.    460 root        4  59    0 6324K 3732K sleep    0:03  0.00% inetd
  32.    607 root       22  59    0  442M  432M sleep   23:10  0.00% fmd
  33.    213 root        6  59    0 4228K 2968K sleep    0:03  0.00% devfsadm
  34.    550 root        1  59    0   42M   40M sleep    0:54  0.00% dbus-daemon
  35.    620 root        1  59    0   12M   10M sleep    0:22  0.00% intrd
  36.   8779 root        1  59    0   22M   20M sleep    0:08  0.00% hosts-check
  37.   3564 root        1  59    0   68M   62M sleep    0:07  0.00% nms
  38.   4383 root        1  59    0   68M   62M sleep    0:06  0.00% nms
  39.      9 root       13  59    0   12M   10M sleep    0:03  0.00% svc.startd
  40.    593 root        3  59    0 8304K 6012K sleep    0:03  0.00% hald
  41.   8768 root        1  59    0   25M   23M sleep    0:02  0.00% ses-check
  42.  25621 root        1  59    0 7164K 4620K sleep    0:02  0.00% sshd
  43.  25134 root        1  59    0 7156K 4616K sleep    0:02  0.00% sshd
  44.  25398 root        1  59    0 7156K 4616K sleep    0:02  0.00% sshd
  45.  
  46.  
  47.  
  48. Here is the output of iostat -xn 1
  49. ==========
  50.                     extended device statistics
  51.     r/s    w/s   kr/s   kw/s wait actv wsvc_t asvc_t  %w  %b device
  52.     0.0    0.0    0.0    0.0  0.0  0.0    0.0    0.0   0   0 c0d0
  53.     0.0    0.0    0.0    0.0  0.0  0.0    0.0    0.0   0   0 c0d1
  54.     0.0    0.0    0.0    0.0  0.0  0.0    0.0    0.0   0   0 c1t0d0
  55.     0.0    0.0    0.0    0.0  0.0  0.0    0.0    0.0   0   0 c1t1d0
  56.     0.0    0.0    0.0    0.0  0.0  0.0    0.0    0.0   0   0 c1t5d0
  57.     0.0    0.0    0.0    0.0  0.0  0.0    0.0    0.0   0   0 c1t4d0
  58.     0.0    0.0    0.0    0.0  0.0  0.0    0.0    0.0   0   0 c1t2d0
  59.  
  60.  
  61. Here is a status. it is missing the pool I am trying to recover.
  62. ==========
  63. root@ICSSAN1:/volumes# zpool status
  64.   pool: syspool
  65.  state: ONLINE
  66.  scan: scrub repaired 0 in 1h40m with 0 errors on Mon Mar 11 07:25:01 2013
  67. config:
  68.  
  69.         NAME        STATE     READ WRITE CKSUM
  70.         syspool     ONLINE       0     0     0
  71.           mirror-0  ONLINE       0     0     0
  72.             c0d0s0  ONLINE       0     0     0
  73.             c0d1s0  ONLINE       0     0     0
  74.  
  75. errors: No known data errors
  76.  
  77.  
  78.  
  79. Here is the import query I issued and its results.
  80. ==========
  81. root@ICSSAN1:/volumes# zpool import -fmFX MikesDrives
  82. cannot import 'MikesDrives': one or more devices is currently unavailable
  83.  
  84.  
  85. My /var/log/messages. There were a ton of these happening. I have not seen any in a few hours, probably since when the import finished.
  86. ==========
  87. Mar  9 19:13:55 ICSSAN1 cmlb: [ID 107833 kern.warning] WARNING: /pci@0,0/pci1043,8443@11/disk@5,0 (sd7):
  88. Mar  9 19:13:55 ICSSAN1         primary label corrupt; using backup
  89. Mar  9 19:13:56 ICSSAN1 cmlb: [ID 107833 kern.warning] WARNING: /pci@0,0/pci1043,8443@11/disk@5,0 (sd7):
  90. Mar  9 19:13:56 ICSSAN1         primary label corrupt; using backup
  91. Mar  9 19:13:56 ICSSAN1 cmlb: [ID 107833 kern.warning] WARNING: /pci@0,0/pci1043,8443@11/disk@5,0 (sd7):
  92. Mar  9 19:13:56 ICSSAN1         primary label corrupt; using backup
  93.  
  94.  
  95. This query on /var/log/messages did not return anything.
  96. ==========
  97. root@ICSSAN1:~# tail -n 1000000 /var/log/messages | grep -vE '(primary label corrupt; using backup|ID 107833 kern.warning|smbd|ntp|apache|IRQ17|system performance|DNS|e1000g1|last message)'
  98.  
  99.  
  100. Here is what is running now
  101. ==========
  102. root@ICSSAN1:~# ps aux
  103. USER       PID %CPU %MEM   SZ  RSS TT       S    START  TIME COMMAND
  104. root         3  0.2  0.0    0    0 ?        S   Mar 08 17:44 fsflush
  105. root      8779  0.1  0.22242420892 ?        S 07:04:00  0:09 /usr/bin/perl /var/lib/nza/triggers/hosts-check
  106. root      7672  0.1  0.23483231796 ?        S 05:46:04  0:17 /usr/bin/python /usr/bin/nmv.py
  107. root      2822  0.1  0.26994832160 ?        S   Mar 08  9:05 /usr/bin/perl /lib/svc/method/nms -d
  108. root     25823  0.1  0.0 2552 2004 pts/3    S   Mar 08  6:49 iostat -xn 1
  109. root      8768  0.1  0.22551623868 ?        S 07:03:00  0:02 /usr/bin/perl /var/lib/nza/triggers/ses-check
  110. root     17172  0.1 19.031333883132524 pts/2    S   Mar 09  5:47 top
  111. root         5  0.1  0.0    0    0 ?        S   Mar 08 13:52 zpool-syspool
  112. root      3564  0.1  0.46978463912 ?        S   Mar 08  0:07 /usr/bin/perl /lib/svc/method/nms -s 1 -p 2011 -L
  113. root      8752  0.0  0.1 5804 4136 ?        S 07:02:11  0:00 /usr/bin/perl /usr/bin/netstat.pl e1000g0 e1000g1 60
  114. root      9258  0.0  0.0 2264 1448 pts/7    O 07:56:12  0:00 ps -aux
  115. root        11  0.0  0.11277611528 ?        S   Mar 08  3:29 /lib/svc/bin/svc.configd
  116. root       550  0.0  0.34252441036 ?        S   Mar 08  0:54 dbus-daemon --fork --config-file=/var/lib/nza/nms.conf --print-address
  117. root       607  0.0  2.7452720442668 ?        S   Mar 08 23:10 /usr/lib/fm/fmd/fmd
  118. root       613  0.0  0.1 6988 4540 ?        S   Mar 08  0:57 /usr/local/sbin/archttp64
  119. root       620  0.0  0.11210410356 ?        S   Mar 08  0:22 /usr/perl5/bin/perl /usr/lib/intrd
  120. root       930  0.0  0.137268 3768 ?        S   Mar 08  0:50 /usr/bin/perl /lib/svc/method/nmdtrace -d
  121. root     10062  0.0  0.1 7732 5216 ?        S   Mar 08  0:45 /usr/lib/ssh/sshd
  122. root         0  0.0  0.0    0    0 ?        T   Mar 08  0:01 sched
  123. root         1  0.0  0.0 2612 1632 ?        S   Mar 08  0:00 /sbin/init
  124. root         2  0.0  0.0    0    0 ?        S   Mar 08  0:00 pageout
  125. root         9  0.0  0.11276810340 ?        S   Mar 08  0:03 /lib/svc/bin/svc.startd
  126. dladm       20  0.0  0.0 2972 1888 ?        S   Mar 08  0:00 /sbin/dlmgmtd
  127. daemon      57  0.0  0.1 8452 5052 ?        S   Mar 08  0:00 /lib/crypto/kcfd
  128. root       142  0.0  0.1 6792 2952 ?        S   Mar 08  0:00 /usr/lib/sysevent/syseventd
  129. root       182  0.0  0.0 2180 1552 ?        S   Mar 08  0:00 /usr/lib/power/powerd
  130. root       213  0.0  0.1 4228 2968 ?        S   Mar 08  0:03 devfsadmd
  131. root       215  0.0  0.1 3904 2892 ?        S   Mar 08  0:00 /usr/lib/picl/picld
  132. root       221  0.0  0.1 7992 5200 ?        S   Mar 08  0:08 /usr/sbin/nscd
  133. root       284  0.0  0.0 2728 1584 ?        S   Mar 08  0:12 /usr/lib/inet/xntpd
  134. root       319  0.0  0.0 2424 1068 ?        S   Mar 08  0:00 /usr/lib/inet/in.ndpd
  135. root       395  0.0  0.0 1896 1192 ?        S   Mar 08  0:00 /usr/lib/hotplugd
  136. root       410  0.0  0.0 2016 1336 ?        S   Mar 08  0:00 /usr/sbin/cron
  137. daemon     422  0.0  0.0 3252 1480 ?        S   Mar 08  0:00 /usr/sbin/rpcbind
  138. daemon     435  0.0  0.0 2768 1892 ?        S   Mar 08  0:00 /usr/lib/nfs/nfs4cbd
  139. daemon     436  0.0  0.0 3180 2160 ?        S   Mar 08  0:00 /usr/lib/nfs/statd
  140. root       445  0.0  0.0 2264 1372 ?        S   Mar 08  0:00 /usr/lib/saf/sac -t 300
  141. daemon     455  0.0  0.1 4260 3220 ?        S   Mar 08  0:00 /usr/lib/idmapd
  142. daemon     456  0.0  0.0 2756 1876 ?        S   Mar 08  0:00 /usr/lib/nfs/lockd
  143. root       460  0.0  0.1 6324 3732 ?        S   Mar 08  0:03 /usr/lib/inet/inetd start
  144. root       463  0.0  0.0 1664  908 ?        S   Mar 08  0:00 /usr/lib/utmpd
  145. daemon     473  0.0  0.0 5048 1940 ?        S   Mar 08  0:00 /usr/lib/nfs/nfsmapid
  146. root       484  0.0  0.0 2432 1456 ?        S   Mar 08  0:00 /usr/lib/saf/ttymon
  147. messageb   510  0.0  0.0 3500 1948 ?        S   Mar 08  0:00 /usr/bin/dbus-daemon --system
  148. root       558  0.0  0.0 3484 1984 ?        S   Mar 08  0:00 dbus-daemon --fork --config-file=/var/lib/nza/nmcd.conf --print-address
  149. root       566  0.0  0.0 3484 1984 ?        S   Mar 08  0:00 dbus-daemon --fork --config-file=/var/lib/nza/nmdtrace.conf --print-address
  150. root       572  0.0  0.0 1864 1076 ?        S   Mar 08  0:00 /usr/lib/smbfs/smbiod-svc
  151. root       578  0.0  0.0 2500 1332 ?        S   Mar 08  0:00 /usr/lib/autofs/automountd
  152. root       580  0.0  0.0 2572 1456 ?        S   Mar 08  0:00 /usr/lib/autofs/automountd
  153. root       589  0.0  0.111120 6924 ?        S   Mar 08  0:31 /usr/lib/smbsrv/smbd start
  154. root       593  0.0  0.1 8304 6012 ?        S   Mar 08  0:03 /usr/lib/hal/hald --daemon=yes
  155. root       598  0.0  0.0 5040 2324 ?        S   Mar 08  0:00 hald-runner
  156. root       603  0.0  0.1 8056 6200 ?        S   Mar 08  5:51 /usr/sbin/syslogd
  157. root       622  0.0  0.0 4108 1688 ?        S   Mar 08  0:00 /usr/lib/ssh/sshd
  158. root       623  0.0  0.0 5148 2364 ?        S   Mar 08  0:00 /usr/lib/hal/hald-addon-network-discovery
  159. root       625  0.0  0.0 5152 2364 ?        S   Mar 08  0:00 /usr/lib/hal/hald-addon-cpufreq
  160. root       626  0.0  0.0 4792 2048 ?        S   Mar 08  0:02 /usr/lib/hal/hald-addon-acpi
  161. root       643  0.0  0.0 5396 2284 ?        S   Mar 08  0:00 /usr/lib/rmvolmgr -s
  162. root       809  0.0  0.116276 6560 ?        S   Mar 08  0:07 /usr/sbin/apache2 -k start
  163. www-data   812  0.0  0.014836 1172 ?        S   Mar 08  0:00 /usr/sbin/apache2 -k start
  164. www-data   819  0.0  0.115828 3648 ?        S   Mar 08  0:00 /usr/sbin/apache2 -k start
  165. www-data   820  0.0  0.115828 3648 ?        S   Mar 08  0:00 /usr/sbin/apache2 -k start
  166. root      2826  0.0  0.0 2040 1112 console  S   Mar 08  0:00 /usr/lib/saf/ttymon -g -d /dev/console -l console -m ldterm,ttcompat -h -p ICSSAN1 console login:
  167. root      3451  0.0  0.149444 4472 ?        S   Mar 08  0:00 /usr/bin/perl /usr/bin/nmc -s
  168. root      3547  0.0  0.0 3560 1988 ?        S   Mar 08  0:00 dbus-daemon --fork --config-file=/var/run/nms(1).conf --print-address
  169. root      4367  0.0  0.0 3656 2068 ?        S   Mar 08  0:00 dbus-daemon --fork --config-file=/var/run/nms(2).conf --print-address
  170. root      4383  0.0  0.46978463880 ?        S   Mar 08  0:06 /usr/bin/perl /lib/svc/method/nms -s 2 -p 2012 -L
  171. root      6077  0.0  0.1 7164 4580 ?        S   Mar 08  0:02 /usr/lib/ssh/sshd
  172. root      6078  0.0  0.1 7732 5216 ?        S   Mar 08  0:32 /usr/lib/ssh/sshd
  173. root      6665  0.0  0.0 4452 2048 pts/1    S   Mar 08  0:00 -bash
  174. root      6683  0.0  0.34942447644 pts/1    S   Mar 08  0:01 /usr/bin/perl /usr/bin/nmc
  175. root      7263  0.0  0.0 4588 2388 pts/1    S   Mar 08  0:00 bash
  176. root      7574  0.0  0.0 4444 2252 ?        S 05:45:03  0:00 sh -c nfsstat.pl 60 1>/var/lib/nza/nfs-collector.data.in
  177. root      7594  0.0  0.1 5952 4284 ?        S 05:45:26  0:00 /usr/bin/perl /usr/bin/nfsstat.pl 60
  178. root      7671  0.0  0.0 4996 1900 ?        S 05:46:04  0:00 /usr/bin/python /usr/bin/nmv.py
  179. root      8279  0.0  0.1 7156 4592 ?        S   Mar 08  0:02 /usr/lib/ssh/sshd
  180. root      8281  0.0  0.1 7732 5216 ?        S   Mar 08  0:06 /usr/lib/ssh/sshd
  181. root      8621  0.0  0.0 4452 2092 pts/2    S   Mar 08  0:00 -bash
  182. root      8633  0.0  0.34942447644 pts/2    S   Mar 08  0:01 /usr/bin/perl /usr/bin/nmc
  183. root      8729  0.0  0.22434822652 ?        S 07:02:00  0:01 /usr/bin/perl /var/lib/nza/triggers/volume-check
  184. root      8749  0.0  0.0 4444 2252 ?        S 07:02:09  0:00 sh -c netstat.pl e1000g0 e1000g1 60 1>/var/lib/nza/network-collector.data.in
  185. root      8806  0.0  0.0 4444 2252 ?        S 07:06:05  0:00 sh -c iostat -xnd 60 1>/var/lib/nza/volume-collector.data.in
  186. root      8812  0.0  0.0 2344 1756 ?        S 07:06:07  0:00 iostat -xnd 60
  187. root      8945  0.0  0.0 4584 2380 pts/2    S   Mar 08  0:00 bash
  188. root      9192  0.0  0.0    0    0          Z           0:00  <defunct>
  189. root     10061  0.0  0.1 7164 4596 ?        S   Mar 08  0:02 /usr/lib/ssh/sshd
  190. root     10260  0.0  0.0 4452 2100 pts/3    S   Mar 08  0:00 -bash
  191. root     10271  0.0  0.34942447708 pts/3    S   Mar 08  0:01 /usr/bin/perl /usr/bin/nmc
  192. root     10749  0.0  0.0 4584 2356 pts/3    S   Mar 08  0:00 bash
  193. root     11494  0.0  0.1 7156 4600 ?        S   Mar 08  0:02 /usr/lib/ssh/sshd
  194. root     11495  0.0  0.1 7732 4972 ?        S   Mar 08  0:00 /usr/lib/ssh/sshd
  195. root     11880  0.0  0.0 4452 2100 pts/4    S   Mar 08  0:00 -bash
  196. root     11891  0.0  0.34942447644 pts/4    S   Mar 08  0:01 /usr/bin/perl /usr/bin/nmc
  197. root     12168  0.0  0.0 4584 2356 pts/4    S   Mar 08  0:00 bash
  198. root     13103  0.0  0.1 7164 4604 ?        S   Mar 08  0:02 /usr/lib/ssh/sshd
  199. root     13104  0.0  0.1 7732 4972 ?        S   Mar 08  0:00 /usr/lib/ssh/sshd
  200. root     13312  0.0  0.0 4452 2100 pts/5    S   Mar 08  0:00 -bash
  201. root     13322  0.0  0.34942447644 pts/5    S   Mar 08  0:01 /usr/bin/perl /usr/bin/nmc
  202. root     13677  0.0  0.0 4584 2416 pts/5    S   Mar 08  0:00 bash
  203. root     24561  0.0  0.0 3600 1680 pts/1    S   Mar 09  0:27 tail -f /var/log/messages
  204. root     25134  0.0  0.1 7156 4616 ?        S   Mar 09  0:02 /usr/lib/ssh/sshd
  205. root     25135  0.0  0.1 7964 5428 ?        S   Mar 09  0:00 /usr/lib/ssh/sshd
  206. root     25185  0.0  0.0 4452 2268 pts/6    S   Mar 09  0:00 -bash
  207. root     25193  0.0  0.34934447736 pts/6    S   Mar 09  0:01 /usr/bin/perl /usr/bin/nmc
  208. root     25398  0.0  0.1 7156 4616 ?        S   Mar 09  0:02 /usr/lib/ssh/sshd
  209. root     25399  0.0  0.1 7732 5216 ?        S   Mar 09  0:00 /usr/lib/ssh/sshd
  210. root     25438  0.0  0.0 4452 2268 pts/7    S   Mar 09  0:00 -bash
  211. root     25446  0.0  0.34934447736 pts/7    S   Mar 09  0:01 /usr/bin/perl /usr/bin/nmc
  212. root     25621  0.0  0.1 7164 4620 ?        S   Mar 09  0:02 /usr/lib/ssh/sshd
  213. root     25622  0.0  0.1 7732 5216 ?        S   Mar 09  0:00 /usr/lib/ssh/sshd
  214. root     28877  0.0  0.1 4584 2520 pts/6    S   Mar 09  0:00 /usr/bin/bash
  215. root     29173  0.0  0.1 4584 2520 pts/7    S   Mar 09  0:00 /usr/bin/bash
  216. root     29390  0.0  0.1 4584 2520 ?        S   Mar 09  0:00 /usr/bin/bash
  217.  
  218.  
  219. fmdump -V has not reported any errors in 3 days.
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement
Public Pastes
Advertisement
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!