API tools faq
paste
Advertisement
Guest User

Untitled

a guest
May 9th, 2013
627
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 82.23 KB | None | 0 0
raw download clone embed print report
  1. toast1a# ems_logviewer messages.ems
  2. May 5 07:00:00 [mgmtgwd]: mgmtgwd.jobmgr.private.jobcomplete.success: Private job "log-rotation" [id 1] (logrotation job) completed successfully: .
  3. May 5 07:00:00 [statd]: kern.uptime.filer: 7:00am up 45 mins, 6420586 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 0 iSCSI ops
  4. May 5 07:00:00 [mgmtgwd]: mgmtgwd.jobmgr.jobstart: Started job "Licensing" [id 1]: License Checking.
  5. May 5 07:00:00 [statd]: monitor.shelf.fault: Fault reported on disk storage shelf attached to channel 0b. Please check fans, power, and temperature.
  6. May 5 07:00:00 [statd]: monitor.raiddp.vol.singleDegraded: data disk in RAID group /engdata1/plex0/rg2 is broken.
  7. May 5 07:00:00 [mgmtgwd]: license.check.ok
  8. May 5 07:00:00 [mgmtgwd]: mgmtgwd.jobmgr.jobcomplete.success: Job "Licensing" [id 1] (License Checking) completed successfully: Succeeded.
  9. May 5 07:00:01 [statd]: monitor.nht.count.exceeded
  10. May 5 07:00:01 [statd]: kern.syslog.msg: Cluster is licensed but takeover of partner is disabled
  11. May 5 07:02:26 [coredump_manager]: coredump.findcore.missing: Partial core missing disk 1 of 12
  12. May 5 07:02:26 [coredump_manager]: coredump.findcore.partial: Partial core is missing 1 of 12 disks
  13. May 5 07:02:27 [coredump_manager]: coredump.findcore.missing: Partial core missing disk 1 of 12
  14. May 5 07:02:27 [coredump_manager]: coredump.findcore.partial: Partial core is missing 1 of 12 disks
  15. May 5 07:02:28 [coredump_manager]: coredump.findcore.missing: Partial core missing disk 1 of 12
  16. May 5 07:02:28 [coredump_manager]: coredump.findcore.partial: Partial core is missing 1 of 12 disks
  17. May 5 07:02:29 [coredump_manager]: coredump.findcore.missing: Partial core missing disk 1 of 12
  18. May 5 07:02:29 [coredump_manager]: coredump.findcore.partial: Partial core is missing 1 of 12 disks
  19. May 5 07:02:29 [savecore_admin]: coredump.findcore.missing: Partial core missing disk 1 of 12
  20. May 5 07:02:29 [savecore_admin]: coredump.findcore.partial: Partial core is missing 1 of 12 disks
  21. May 5 07:02:29 [savecore_admin]: coredump.save.started: Saving 4558M to core.101185225.2013-04-22.17_53_15.nz (\"Freeing free block 9009630 in aggregate engdata1.\")
  22. May 5 07:02:29 [savecore_admin]: coredump.save.nospace: Not enough space in root volume to save core
  23. May 5 07:02:29 [savecore_admin]: coredump.save.error: core.101185225.2013-04-22.17_53_15.nz processing encountered error
  24. May 5 07:02:29 [savecore_admin]: coredump.save.started: Saving 4474M to core.101185225.2013-04-22.19_04_41.nz (\"Freeing free block 9009630 in aggregate engdata1.\")
  25. May 5 07:02:29 [savecore_admin]: coredump.save.nospace: Not enough space in root volume to save core
  26. May 5 07:02:29 [savecore_admin]: coredump.save.error: core.101185225.2013-04-22.19_04_41.nz processing encountered error
  27. May 5 07:02:29 [savecore_admin]: coredump.save.started: Saving 4556M to core.101185225.2013-04-22.20_18_41.nz (\"Freeing free block 9009630 in aggregate engdata1.\")
  28. May 5 07:02:29 [savecore_admin]: coredump.save.nospace: Not enough space in root volume to save core
  29. May 5 07:02:29 [savecore_admin]: coredump.save.error: core.101185225.2013-04-22.20_18_41.nz processing encountered error
  30. May 5 07:02:29 [savecore_admin]: coredump.save.started: Saving 4556M to core.101185225.2013-04-22.21_31_46.nz (\"Freeing free block 9009630 in aggregate engdata1.\")
  31. May 5 07:02:29 [savecore_admin]: coredump.save.nospace: Not enough space in root volume to save core
  32. May 5 07:02:29 [savecore_admin]: coredump.save.error: core.101185225.2013-04-22.21_31_46.nz processing encountered error
  33. May 5 07:05:00 [mgmtgwd]: mgmtgwd.jobmgr.private.jobstart: Started private job "snap-hourly" [id 4]: Auto-Snapshot.
  34. May 5 07:07:24 [mgmtgwd]: mgmtgwd.jobmgr.private.jobcomplete.success: Private job "snap-hourly" [id 4] (Auto-Snapshot) completed successfully: .
  35. May 5 07:07:31 [coredump_manager]: coredump.findcore.missing: Partial core missing disk 1 of 12
  36. May 5 07:07:31 [coredump_manager]: coredump.findcore.partial: Partial core is missing 1 of 12 disks
  37. May 5 07:07:34 [coredump_manager]: coredump.findcore.missing: Partial core missing disk 1 of 12
  38. May 5 07:07:34 [coredump_manager]: coredump.findcore.partial: Partial core is missing 1 of 12 disks
  39. May 5 07:07:35 [coredump_manager]: coredump.findcore.missing: Partial core missing disk 1 of 12
  40. May 5 07:07:35 [coredump_manager]: coredump.findcore.partial: Partial core is missing 1 of 12 disks
  41. May 5 07:07:38 [coredump_manager]: coredump.findcore.missing: Partial core missing disk 1 of 12
  42. May 5 07:07:38 [coredump_manager]: coredump.findcore.partial: Partial core is missing 1 of 12 disks
  43. May 5 07:07:39 [savecore_admin]: coredump.findcore.missing: Partial core missing disk 1 of 12
  44. May 5 07:07:39 [savecore_admin]: coredump.findcore.partial: Partial core is missing 1 of 12 disks
  45. May 5 07:07:39 [savecore_admin]: coredump.save.started: Saving 4558M to core.101185225.2013-04-22.17_53_15.nz (\"Freeing free block 9009630 in aggregate engdata1.\")
  46. May 5 07:07:39 [savecore_admin]: coredump.save.nospace: Not enough space in root volume to save core
  47. May 5 07:07:39 [savecore_admin]: coredump.save.error: core.101185225.2013-04-22.17_53_15.nz processing encountered error
  48. May 5 07:07:39 [savecore_admin]: coredump.save.started: Saving 4474M to core.101185225.2013-04-22.19_04_41.nz (\"Freeing free block 9009630 in aggregate engdata1.\")
  49. May 5 07:07:39 [savecore_admin]: coredump.save.nospace: Not enough space in root volume to save core
  50. May 5 07:07:39 [savecore_admin]: coredump.save.error: core.101185225.2013-04-22.19_04_41.nz processing encountered error
  51. May 5 07:07:39 [savecore_admin]: coredump.save.started: Saving 4556M to core.101185225.2013-04-22.20_18_41.nz (\"Freeing free block 9009630 in aggregate engdata1.\")
  52. May 5 07:07:39 [savecore_admin]: coredump.save.nospace: Not enough space in root volume to save core
  53. May 5 07:07:39 [savecore_admin]: coredump.save.error: core.101185225.2013-04-22.20_18_41.nz processing encountered error
  54. May 5 07:07:39 [savecore_admin]: coredump.save.started: Saving 4556M to core.101185225.2013-04-22.21_31_46.nz (\"Freeing free block 9009630 in aggregate engdata1.\")
  55. May 5 07:07:39 [savecore_admin]: coredump.save.nospace: Not enough space in root volume to save core
  56. May 5 07:07:39 [savecore_admin]: coredump.save.error: core.101185225.2013-04-22.21_31_46.nz processing encountered error
  57. May 5 07:07:45 [coredump_manager]: coredump.findcore.missing: Partial core missing disk 1 of 12
  58. May 5 07:07:45 [coredump_manager]: coredump.findcore.partial: Partial core is missing 1 of 12 disks
  59. May 5 07:07:46 [coredump_manager]: coredump.findcore.missing: Partial core missing disk 1 of 12
  60. May 5 07:07:46 [coredump_manager]: coredump.findcore.partial: Partial core is missing 1 of 12 disks
  61. May 5 07:07:48 [coredump_manager]: coredump.findcore.missing: Partial core missing disk 1 of 12
  62. May 5 07:07:48 [coredump_manager]: coredump.findcore.partial: Partial core is missing 1 of 12 disks
  63. May 5 07:07:50 [coredump_manager]: coredump.findcore.missing: Partial core missing disk 1 of 12
  64. May 5 07:07:50 [coredump_manager]: coredump.findcore.partial: Partial core is missing 1 of 12 disks
  65. May 5 07:07:50 [mgmtgwd]: mgmtgwd.jobmgr.private.jobcomplete.success: Private job "coredump-autosave" [id 8] () completed successfully: .
  66. May 5 07:07:54 [savecore_admin]: coredump.findcore.missing: Partial core missing disk 1 of 12
  67. May 5 07:07:54 [savecore_admin]: coredump.findcore.partial: Partial core is missing 1 of 12 disks
  68. May 5 07:07:54 [savecore_admin]: coredump.save.started: Saving 4558M to core.101185225.2013-04-22.17_53_15.nz (\"Freeing free block 9009630 in aggregate engdata1.\")
  69. May 5 07:07:54 [savecore_admin]: coredump.save.nospace: Not enough space in root volume to save core
  70. May 5 07:07:54 [savecore_admin]: coredump.save.error: core.101185225.2013-04-22.17_53_15.nz processing encountered error
  71. May 5 07:07:54 [savecore_admin]: coredump.save.started: Saving 4474M to core.101185225.2013-04-22.19_04_41.nz (\"Freeing free block 9009630 in aggregate engdata1.\")
  72. May 5 07:07:54 [savecore_admin]: coredump.save.nospace: Not enough space in root volume to save core
  73. May 5 07:07:54 [savecore_admin]: coredump.save.error: core.101185225.2013-04-22.19_04_41.nz processing encountered error
  74. May 5 07:07:54 [savecore_admin]: coredump.save.started: Saving 4556M to core.101185225.2013-04-22.20_18_41.nz (\"Freeing free block 9009630 in aggregate engdata1.\")
  75. May 5 07:07:54 [savecore_admin]: coredump.save.nospace: Not enough space in root volume to save core
  76. May 5 07:07:54 [savecore_admin]: coredump.save.error: core.101185225.2013-04-22.20_18_41.nz processing encountered error
  77. May 5 07:07:54 [savecore_admin]: coredump.save.started: Saving 4556M to core.101185225.2013-04-22.21_31_46.nz (\"Freeing free block 9009630 in aggregate engdata1.\")
  78. May 5 07:07:54 [savecore_admin]: coredump.save.nospace: Not enough space in root volume to save core
  79. May 5 07:07:54 [savecore_admin]: coredump.save.error: core.101185225.2013-04-22.21_31_46.nz processing encountered error
  80. May 5 07:10:00 [mgmtgwd]: mgmtgwd.jobmgr.private.jobstart: Started private job "snap-daily" [id 5]: Auto-Snapshot.
  81. May 5 07:10:56 [mgmtgwd]: mgmtgwd.jobmgr.private.jobcomplete.success: Private job "snap-daily" [id 5] (Auto-Snapshot) completed successfully: .
  82. May 5 07:15:00 [mgmtgwd]: mgmtgwd.jobmgr.private.jobstart: Started private job "snap-weekly" [id 6]: Auto-Snapshot.
  83. May 5 07:15:28 [mgmtgwd]: mgmtgwd.jobmgr.private.jobcomplete.success: Private job "snap-weekly" [id 6] (Auto-Snapshot) completed successfully: .
  84. May 5 07:17:01 [registry_control]: kern.syslog.msg: registry: Unexpected error in rollback in regx_commit - Error: No such file or directory (name=options.ssh.access) Error: No such file or directory (name=options.ssh.debug.enable) Error: No such file or directory (name=options.ssh.idle.timeout) Error: No such file or directory (name=options.ssh.passwd_auth.ena
  85. May 5 07:17:01 [registry_control]: reg.transaction.commitFail: registry: Cannot commit transaction in 'Periodic config update'. Error: Invalid argument (name=options.ssh.access) (value=*) Error: Invalid argument (name=options.ssh.debug.enable) (value=off) Error: Invalid argument (name=options.ssh.idle.timeout) (value=600) Error: Invalid argument (name=options.ssh.passwd_auth.enabl
  86. May 5 07:17:52 [console_login_mgr]: kern.syslog.msg: root logged in from console
  87. May 5 07:19:31 [ispfc_main]: disk.ioMediumError: Medium error on disk 0c.51: op 0x28:2f591472:002d sector 794367096 SCSI:medium error - Unrecovered read error - RAID subsystem will use parity information to reconstruct unreadable data (813554c 82d95cc 82d9adc 82d9c6c)
  88. May 5 07:19:35 [raidio_thread]: raid.tetris.media.err: Read error on Disk /engdata1/plex0/rg0/0c.51 Shelf 3 Bay 3 [NETAPP X267_MGRIZ500SSX H3DD] S/N [H817GCGH], block #88263010 during stripe write
  89. May 5 07:19:35 [raidio_thread]: raid.rg.readerr.repair.data: Fixing bad data on Disk /engdata1/plex0/rg0/0c.51 Shelf 3 Bay 3 [NETAPP X267_MGRIZ500SSX H3DD] S/N [H817GCGH], block #88263010
  90. May 5 07:19:36 [ispfc_main]: disk.ioReassignSuccess: disk 0c.51: sector 794367096 was reassigned.
  91. May 5 08:00:00 [statd]: kern.uptime.filer: 8:00am up 1:45 15410774 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 0 iSCSI ops
  92. May 5 08:00:00 [statd]: monitor.shelf.fault: Fault reported on disk storage shelf attached to channel 0b. Please check fans, power, and temperature.
  93. May 5 08:00:00 [statd]: monitor.raiddp.vol.singleDegraded: data disk in RAID group /engdata1/plex0/rg2 is broken.
  94. May 5 08:00:01 [statd]: kern.syslog.msg: Cluster is licensed but takeover of partner is disabled
  95. May 5 08:05:00 [mgmtgwd]: mgmtgwd.jobmgr.private.jobstart: Started private job "snap-hourly" [id 4]: Auto-Snapshot.
  96. May 5 08:05:34 [mgmtgwd]: mgmtgwd.jobmgr.private.jobcomplete.success: Private job "snap-hourly" [id 4] (Auto-Snapshot) completed successfully: .
  97. May 5 08:18:01 [registry_control]: kern.syslog.msg: registry: Unexpected error in rollback in regx_commit - Error: No such file or directory (name=options.ssh.access) Error: No such file or directory (name=options.ssh.debug.enable) Error: No such file or directory (name=options.ssh.idle.timeout) Error: No such file or directory (name=options.ssh.passwd_auth.ena
  98. May 5 08:18:01 [registry_control]: reg.transaction.commitFail: registry: Cannot commit transaction in 'Periodic config update'. Error: Invalid argument (name=options.ssh.access) (value=*) Error: Invalid argument (name=options.ssh.debug.enable) (value=off) Error: Invalid argument (name=options.ssh.idle.timeout) (value=600) Error: Invalid argument (name=options.ssh.passwd_auth.enabl
  99. May 5 08:32:00 [wafl_junkd_aggroffl]: wafl.buf.freeingFreeBlock: Freeing free block 926442036 in aggregate engdata1.\n WAFL inconsistent. Please contact Network Appliance Global Services.
  100. May 5 08:32:01 [wafl_junkd_aggroffl]: sk.panic: Panic String: Freeing free block 926442036 in aggregate engdata1. WAFL inconsistent. Please contact Network Appliance Global Services. in SK process wafl_junkd_aggroffl on release NetApp Release R10.0.1xN_070123_1900
  101. May 5 08:32:01 [wafl_junkd_aggroffl]: coredump.findcore.missing: Partial core missing disk 1 of 12
  102. May 5 08:32:01 [wafl_junkd_aggroffl]: coredump.findcore.partial: Partial core is missing 1 of 12 disks
  103. May 5 08:32:01 [wafl_junkd_aggroffl]: coredump.dump.find.all.used: All available disks contain unsaved cores
  104. May 5 08:32:01 [wafl_junkd_aggroffl]: coredump.dump.failed: Dumpcore failed
  105. May 5 08:33:07 [nic_mgr]: cf.nm.nicTransitionUp: Interconnect link 0 is UP
  106. May 5 08:33:15 [ispfc_timeout_3]: fci.device.quiesce: Adapter unknown encountered a command timeout on 0b device 0b.36 (0x03000024). Quiescing the device.
  107. May 5 08:33:20 [sanown_notify]: diskown.isEnabled: software ownership has been enabled for this system
  108. May 5 08:33:22 [nic_mgr]: cf.nm.nicReset: Initiating soft reset on Cluster Interconnect card 0 due to rendezvous reset
  109. May 5 08:33:22 [rendezvous_proc]: cf.rv.notConnected: Connection for cfo_rv failed
  110. May 5 08:33:22 [nic_mgr]: cf.nm.nicTransitionDown: Cluster Interconnect link 0 is DOWN
  111. May 5 08:33:22 [rendezvous_proc]: cf.rv.notConnected: Connection for cfo_rv failed
  112. May 5 08:33:24 [nic_mgr]: cf.nm.nicTransitionUp: Interconnect link 0 is UP
  113. May 5 08:33:24 [fmmbx_instanceWorke]: fmmb.current.lock.disk: Disk 0c.58 is a local HA mailbox disk.
  114. May 5 08:33:24 [fmmbx_instanceWorke]: fmmb.current.lock.disk: Disk 0a.29 is a local HA mailbox disk.
  115. May 5 08:33:24 [fmmbx_instanceWorke]: kern.syslog.msg: normal mailbox instance on local side
  116. May 5 08:33:24 [rc]: cf.fm.MBstatusOnBoot: status="1"
  117. May 5 08:33:41 [ses_admin]: ses.channel.rescanInitiated: Initiating rescan on channel 0b
  118. May 5 08:33:43 [config_thread]: raid.assim.disk.brokenPreAssim: Broken Disk 0c.60 Shelf ? Bay ? [NETAPP X267_MGRIZ500SSX H3DD] S/N [H814LZ6H] detected prior to assimilation. It should be removed.
  119. May 5 08:33:43 [config_thread]: raid.assim.disk.brokenPreAssim: Broken Disk 0c.53 Shelf ? Bay ? [NETAPP X267_MGRIZ500SSX H3DD] S/N [H8155VEH] detected prior to assimilation. It should be removed.
  120. May 5 08:33:43 [config_thread]: raid.assim.disk.brokenPreAssim: Broken Disk 0c.57 Shelf ? Bay ? [NETAPP X267_MGRIZ500SSX H3DD] S/N [H814R7RH] detected prior to assimilation. It should be removed.
  121. May 5 08:33:43 [config_thread]: raid.assim.disk.brokenPreAssim: Broken Disk 0b.41 Shelf ? Bay ? [NETAPP X267_MGRIZ500SSX H3DD] S/N [H817LZ8H] detected prior to assimilation. It should be removed.
  122. May 5 08:33:43 [config_thread]: raid.assim.disk.brokenPreAssim: Broken Disk 0b.44 Shelf ? Bay ? [NETAPP X267_MGRIZ500SSX H3DD] S/N [H817GR5H] detected prior to assimilation. It should be removed.
  123. May 5 08:33:43 [config_thread]: raid.assim.disk.brokenPreAssim: Broken Disk 0a.23 Shelf ? Bay ? [NETAPP X267_MGRIZ500SSX H3DD] S/N [H814LVPH] detected prior to assimilation. It should be removed.
  124. May 5 08:33:43 [config_thread]: raid.assim.cls.noStableCls: Unable to find a stable consistent label set for all trees. Resorting disks.
  125. May 5 08:33:43 [config_thread]: raid.assim.cls.noStableCls2: Unable to find a stable consistent label set for all trees with Label 2. Resorting disks.
  126. May 5 08:33:43 [fmmbx_instanceWorke]: fmmb.NewDiskSuggestion: count="4", new="0c.58, 0a.29, null, null", old="0c.58, 0a.29"
  127. May 5 08:33:43 [fmmbx_instanceWorke]: fmmb.current.lock.disk: Disk 0d.70 is a partner HA mailbox disk.
  128. May 5 08:33:43 [fmmbx_instanceWorke]: fmmb.current.lock.disk: Disk 0d.67 is a partner HA mailbox disk.
  129. May 5 08:33:43 [fmmbx_instanceWorke]: kern.syslog.msg: normal mailbox instance on partner side
  130. May 5 08:33:43 [rc]: cf.fm.partner: Cluster monitor: partner 'toast1b'
  131. May 5 08:33:43 [rc]: cf.fm.timeMasterStatus: Acting as cluster time master
  132. May 5 08:33:47 [config_thread]: raid.vol.replay.nvram: Performing raid replay on volume(s)
  133. May 5 08:33:47 [rc]: raid.cksum.replay.summary: Replayed 0 checksum blocks.
  134. May 5 08:33:47 [rc]: raid.stripe.replay.summary: Replayed 0 stripes.
  135. May 5 08:33:48 [rc]: wafl.inconsistent.vol: WAFL inconsistent: Aggregate engdata1 is inconsistent. Note: Any new Snapshot copies might contain this inconsistency.
  136. May 5 08:33:48 [rc]: wafl.vol.loading: Loading Aggregate engdata1
  137. May 5 08:33:48 [rc]: wafl.vol.loading: Loading Aggregate aggr0
  138. May 5 08:33:48 [vv_apply_worker16]: wafl.vol.loading: Loading Volume mroot@vserver:06866009-a1fe-11e0-8650-000e0caac15e
  139. May 5 08:33:48 [vv_apply_worker17]: wafl.vol.loading: Loading Volume engdata1_root@vserver:b1a2013d-acca-11e0-98c0-000e0caac15e
  140. May 5 08:33:48 [vv_apply_worker19]: wafl.vol.loading: Loading Volume bluecore@vserver:b1a2013d-acca-11e0-98c0-000e0caac15e
  141. May 5 08:33:48 [vv_apply_worker18]: wafl.vol.loading: Loading Volume engdata1@vserver:b1a2013d-acca-11e0-98c0-000e0caac15e
  142. May 5 08:33:49 [vv_apply_worker19]: wafl.maxdirsize.boot.notice: bluecore@vserver:b1a2013d-acca-11e0-98c0-000e0caac15e: This volume's maxdirsize (102400KB) is higher than the default (35573KB). There may be a performance penalty when doing operations on large directories.
  143. May 5 08:33:49 [rc]: cf.fm.launch: Launching cluster monitor
  144. May 5 08:33:49 [rc]: cf.fm.partner: Cluster monitor: partner 'toast1b'
  145. May 5 08:33:49 [rc]: cf.fm.notkoverClusterDisable: Cluster monitor: cluster takeover disabled (restart)
  146. May 5 08:33:49 [cf_main]: cf.fsm.takeoverOfPartnerDisabled: Cluster monitor: takeover of toast1b disabled (cluster takeover disabled)
  147. May 5 08:34:00 [rc]: license.db.migrate.vol.success: Successfully migrated licenses for volume engdata1
  148. May 5 08:34:00 [rc]: license.db.migrate.vol.success: Successfully migrated licenses for volume aggr0
  149. May 5 08:34:00 [rc]: license.db.migrate.success: Migration complete
  150. May 5 08:34:00 [fmmbx_instanceWorke]: fmmb.NewDiskSuggestion: count="4", new="0c.58, 0a.29, null, null", old="0c.58, 0a.29"
  151. May 5 08:34:00 [rc]: kern.syslog.msg: The system was down for 98 seconds
  152. May 5 08:34:00 [raidio_thread]: raid.disk.unload.done: Unload of Disk 0c.60 Shelf 3 Bay 12 [NETAPP X267_MGRIZ500SSX H3DD] S/N [H814LZ6H] has completed successfully
  153. May 5 08:34:00 [raidio_thread]: raid.disk.unload.done: Unload of Disk 0c.53 Shelf 3 Bay 5 [NETAPP X267_MGRIZ500SSX H3DD] S/N [H8155VEH] has completed successfully
  154. May 5 08:34:00 [raidio_thread]: raid.disk.unload.done: Unload of Disk 0c.57 Shelf 3 Bay 9 [NETAPP X267_MGRIZ500SSX H3DD] S/N [H814R7RH] has completed successfully
  155. May 5 08:34:00 [raidio_thread]: raid.disk.unload.done: Unload of Disk 0b.41 Shelf 2 Bay 9 [NETAPP X267_MGRIZ500SSX H3DD] S/N [H817LZ8H] has completed successfully
  156. May 5 08:34:00 [raidio_thread]: raid.disk.unload.done: Unload of Disk 0b.44 Shelf 2 Bay 12 [NETAPP X267_MGRIZ500SSX H3DD] S/N [H817GR5H] has completed successfully
  157. May 5 08:34:00 [raidio_thread]: raid.disk.unload.done: Unload of Disk 0a.23 Shelf 1 Bay 7 [NETAPP X267_MGRIZ500SSX H3DD] S/N [H814LVPH] has completed successfully
  158. May 5 08:34:00 [rc]: raid.fdr.failed.ok: Disk 0c.60 Shelf 3 Bay 12 [NETAPP X267_MGRIZ500SSX H3DD] S/N [H814LZ6H] successfully deleted from spare pool
  159. May 5 08:34:00 [rc]: raid.fdr.failed.ok: Disk 0c.53 Shelf 3 Bay 5 [NETAPP X267_MGRIZ500SSX H3DD] S/N [H8155VEH] successfully deleted from spare pool
  160. May 5 08:34:00 [rc]: raid.fdr.failed.ok: Disk 0c.57 Shelf 3 Bay 9 [NETAPP X267_MGRIZ500SSX H3DD] S/N [H814R7RH] successfully deleted from spare pool
  161. May 5 08:34:00 [rc]: raid.fdr.failed.ok: Disk 0b.41 Shelf 2 Bay 9 [NETAPP X267_MGRIZ500SSX H3DD] S/N [H817LZ8H] successfully deleted from spare pool
  162. May 5 08:34:00 [rc]: raid.fdr.failed.ok: Disk 0b.44 Shelf 2 Bay 12 [NETAPP X267_MGRIZ500SSX H3DD] S/N [H817GR5H] successfully deleted from spare pool
  163. May 5 08:34:00 [rc]: raid.fdr.failed.ok: Disk 0a.23 Shelf 1 Bay 7 [NETAPP X267_MGRIZ500SSX H3DD] S/N [H814LVPH] successfully deleted from spare pool
  164. May 5 08:34:00 [rc]: kern.syslog.msg: registry: Unexpected error in rollback in regx_commit - Error: No such file or directory (name=options.ssh.access) Error: No such file or directory (name=options.ssh.debug.enable) Error: No such file or directory (name=options.ssh.idle.timeout) Error: No such file or directory (name=options.ssh.passwd_auth.ena
  165. May 5 08:34:00 [rc]: reg.transaction.commitFail: registry: Cannot commit transaction in 'Periodic config update'. Error: Invalid argument (name=options.ssh.access) (value=*) Error: Invalid argument (name=options.ssh.debug.enable) (value=off) Error: Invalid argument (name=options.ssh.idle.timeout) (value=600) Error: Invalid argument (name=options.ssh.passwd_auth.enabl
  166. May 5 08:34:00 [rc]: kern.syslog.msg: sysconfig: validating configuration
  167. May 5 08:34:00 [config_thread]: raid.rg.recons.missing: RAID group /engdata1/plex0/rg2 is missing 1 disk(s).
  168. May 5 08:34:00 [config_thread]: raid.rg.recons.cantStart: The reconstruction cannot start in RAID group /engdata1/plex0/rg2: No 520 bps disk of required size available in spare pool
  169. May 5 08:34:00 [rc]: wafl.inconsistent.vol: WAFL inconsistent: Aggregate engdata1 is inconsistent. Note: Any new Snapshot copies might contain this inconsistency.
  170. May 5 08:34:00 [rc]: mgr.boot.disk_done: NetApp Release ontap boot complete. Last disk update written at Sun May 5 08:31:57 GMT 2013
  171. May 5 08:34:00 [rc]: mgr.boot.reason_abnormal: System rebooted after a panic.
  172. May 5 08:34:00 [rc]: mgr.stack.module: Module kernel loaded from 0x80000000 - 0x100000000.
  173. May 5 08:34:00 [rc]: mgr.stack.module: Module common_kmod.ko loaded from 0x81a00000 - 0x81aa6000.
  174. May 5 08:34:00 [rc]: mgr.stack.module: Module nvram5.ko loaded from 0x81aa6000 - 0x81ab8000.
  175. May 5 08:34:00 [rc]: mgr.stack.module: Module nvram_mgr.ko loaded from 0x81ab8000 - 0x81abd000.
  176. May 5 08:34:00 [rc]: mgr.stack.module: Module nvr.ko loaded from 0x81abd000 - 0x81ad0000.
  177. May 5 08:34:00 [rc]: mgr.stack.module: Module maytag.ko loaded from 0x81ad0000 - 0x858db000.
  178. May 5 08:34:00 [rc]: mgr.stack.module: Module spinvfs.ko loaded from 0x858db000 - 0x85924000.
  179. May 5 08:34:00 [rc]: mgr.stack.module: Module nbladekmod.ko loaded from 0x85924000 - 0x876b8000.
  180. May 5 08:34:00 [rc]: mgr.stack.saved: Reboot with saved panic information in log file
  181. May 5 08:34:00 [rc]: mgr.stack.string: Panic string: Freeing free block 926442036 in aggregate engdata1. WAFL inconsistent. Please contact Network Appliance Global Services. in SK process wafl_junkd_aggroffl on r
  182. May 5 08:34:00 [rc]: mgr.stack.at: Panic occurred at: Sun May 5 08:32:01 2013
  183. May 5 08:34:00 [rc]: mgr.stack.proc: Panic in process: wafl_junkd_aggroffl
  184. May 5 08:34:00 [rc]: mgr.stack.framename: Stack frame 0: maytag.ko::ontap_dumpcore(0x82146b00) + 0xb4
  185. May 5 08:34:00 [rc]: mgr.stack.framename: Stack frame 1: kernel::dumpsys(0x80477260) + 0xf8
  186. May 5 08:34:00 [rc]: mgr.stack.framename: Stack frame 2: kernel::shutdown_get_howto(0x80324460) + 0x5d
  187. May 5 08:34:00 [rc]: mgr.stack.framename: Stack frame 3: kernel::shutdown_get_howto(0x80324460) + 0x45d
  188. May 5 08:34:00 [rc]: mgr.stack.framename: Stack frame 4: kernel::panic(0x80325050) + 0x189
  189. May 5 08:34:00 [rc]: mgr.stack.framename: Stack frame 5: maytag.ko::oscall_panic(0x82148070) + 0x11
  190. May 5 08:34:00 [rc]: mgr.stack.framename: Stack frame 6: maytag.ko::sk_vpanic(0x8226e46e) + 0x86
  191. May 5 08:34:00 [rc]: mgr.stack.framename: Stack frame 7: maytag.ko::sk_vpanic(0x8226e46e) + 0x0
  192. May 5 08:34:00 [rc]: mgr.stack.framename: Stack frame 8: maytag.ko::wafl_junkd_aggroffline_handle_inconsistency_cb(0x829f9ccc) + 0x0
  193. May 5 08:34:00 [rc]: mgr.stack.framename: Stack frame 9: maytag.ko::wafl_junkd_aggroffline_worker(0x829f9ce5) + 0x1c3
  194. May 5 08:34:00 [rc]: mgr.stack.framename: Stack frame 10: maytag.ko::sk_stack_top(0x8226e1b6) + 0x0
  195. May 5 08:34:00 [asup_main]: asup.throttle.drop: Too many autosupport messages in too short a time, throttling autosupport: WAFL INCONSISTENT
  196. May 5 08:34:00 [asup_main]: asup.throttle.drop: Too many autosupport messages in too short a time, throttling autosupport: WAFL INCONSISTENT
  197. May 5 08:34:00 [asup_main]: asup.throttle.drop: Too many autosupport messages in too short a time, throttling autosupport: REBOOT (panic)
  198. May 5 08:34:00 [snmp_trap_main]: snmp.link.up: Interface 5 is up
  199. May 5 08:34:00 [sshd]: kern.syslog.msg: error: Could not load host key: /etc/sshd/ssh_host_key
  200. May 5 08:34:00 [sshd]: kern.syslog.msg: error: Could not load host key: /etc/sshd/ssh_host_rsa_key
  201. May 5 08:34:00 [sshd]: kern.syslog.msg: error: Could not load host key: /etc/sshd/ssh_host_dsa_key
  202. May 5 08:34:01 [coredump_manager]: coredump.findcore.missing: Partial core missing disk 1 of 12
  203. May 5 08:34:01 [coredump_manager]: coredump.findcore.partial: Partial core is missing 1 of 12 disks
  204. May 5 08:34:01 [coredump_manager]: coredump.all.disks.used: No disks available for dumpcore
  205. May 5 08:34:01 [statd]: monitor.raid.brokenDisk: data disk in RAID group /engdata1/plex0/rg2 is broken.
  206. May 5 08:34:01 [statd]: raid.vol.state.online: vol_type="Volume", vol="aggr0"
  207. May 5 08:34:01 [statd]: raid.vol.state.online: vol_type="Volume", vol="engdata1"
  208. May 5 08:34:01 [10/100/1000-VI/e0c]: kern.syslog.msg: Ethernet e0c: Link up
  209. May 5 08:34:01 [10/100/1000-VI/e0d]: kern.syslog.msg: Ethernet e0d: Link up
  210. May 5 08:34:01 [10/100/1000-VI/e0a]: kern.syslog.msg: Ethernet e0a: Link up
  211. May 5 08:34:01 [10/100/1000-VI/e0b]: kern.syslog.msg: Ethernet e0b: Link up
  212. May 5 08:34:01 [config_failed_disk]: raid.fdr.reminder: Failed Disk 0c.60 Shelf 3 Bay 12 [NETAPP X267_MGRIZ500SSX H3DD] S/N [H814LZ6H] is still present in the system and should be removed.
  213. May 5 08:34:01 [config_failed_disk]: raid.fdr.reminder: Failed Disk 0c.53 Shelf 3 Bay 5 [NETAPP X267_MGRIZ500SSX H3DD] S/N [H8155VEH] is still present in the system and should be removed.
  214. May 5 08:34:01 [config_failed_disk]: raid.fdr.reminder: Failed Disk 0c.57 Shelf 3 Bay 9 [NETAPP X267_MGRIZ500SSX H3DD] S/N [H814R7RH] is still present in the system and should be removed.
  215. May 5 08:34:01 [config_failed_disk]: raid.fdr.reminder: Failed Disk 0b.41 Shelf 2 Bay 9 [NETAPP X267_MGRIZ500SSX H3DD] S/N [H817LZ8H] is still present in the system and should be removed.
  216. May 5 08:34:01 [config_failed_disk]: raid.fdr.reminder: Failed Disk 0b.44 Shelf 2 Bay 12 [NETAPP X267_MGRIZ500SSX H3DD] S/N [H817GR5H] is still present in the system and should be removed.
  217. May 5 08:34:01 [config_failed_disk]: raid.fdr.reminder: Failed Disk 0a.23 Shelf 1 Bay 7 [NETAPP X267_MGRIZ500SSX H3DD] S/N [H814LVPH] is still present in the system and should be removed.
  218. May 5 08:34:01 [10/100/1000-VI/e0a]: kern.syslog.msg: Ethernet e0a: Link being reconfigured
  219. May 5 08:34:01 [config_thread]: raid.rg.spares.low: /engdata1/plex0/rg0
  220. May 5 08:34:01 [config_thread]: raid.rg.spares.low: /engdata1/plex0/rg1
  221. May 5 08:34:01 [config_thread]: raid.rg.spares.low: /engdata1/plex0/rg2
  222. May 5 08:34:01 [config_thread]: raid.rg.spares.low: /aggr0/plex0/rg0
  223. May 5 08:34:01 [asup_main]: asup.throttle.drop: Too many autosupport messages in too short a time, throttling autosupport: SPARES_LOW
  224. May 5 08:34:01 [console_login_mgr]: kern.syslog.msg: root logged in from console
  225. May 5 08:34:04 [ses_admin]: ses.drive.shelfAddr.mm: Mismatch between adapter and enclosure services information for device 0b.36 (shelf 2 bay 4).
  226. May 5 08:34:05 [dyn_dev_qual_admin]: dfu.firmwareUpToDate: Firmware is up-to-date on all disk drives
  227. May 5 08:34:05 [statd]: monitor.shelf.accessError: Enclosure services has detected an error in access to shelves on channel 0b.
  228. May 5 08:34:06 [asup_main]: callhome.encl.access: Call home for ENCLOSURE SERVICES ACCESS ERROR
  229. May 5 08:34:06 [10/100/1000-VI/e0a]: kern.syslog.msg: Ethernet e0a: Link up
  230. May 5 08:34:07 [10/100/1000-VI/e0b]: kern.syslog.msg: Ethernet e0b: Link being reconfigured
  231. May 5 08:34:12 [10/100/1000-VI/e0b]: kern.syslog.msg: Ethernet e0b: Link up
  232. May 5 08:34:17 [ses_admin]: ses.status.temperatureWarning: DS14-Mk2-AT shelf 2 on channel 0b temperature warning for Temperature sensor 2: not installed or failed. Current temperature: <N/A> C (<N/A> F). This module is on the rear side of the shelf, at the top center, on shelf module A.
  233. May 5 08:34:17 [ses_admin]: ses.status.electronicsError: DS14-Mk2-AT shelf 2 on channel 0b environmental monitoring error for SES electronics 1: unrecoverable error ; enclosure services hardware failed This module is on the rear side of the shelf, at the top center.
  234. May 5 08:34:18 [statd]: monitor.shelf.fault: Fault reported on disk storage shelf attached to channel 0b. Please check fans, power, and temperature.
  235. May 5 08:34:18 [asup_main]: callhome.shlf.fault: Call home for SHELF_FAULT
  236. May 5 08:34:35 [ses_admin]: ses.status.temperatureWarning: DS14-Mk2-AT shelf 2 on channel 0b temperature warning for Temperature sensor 1: internal communication error. Current temperature: <N/A> C (<N/A> F). This module is on the front side of the shelf, at the right, on the OPS panel.
  237. May 5 08:34:39 [mgmtgwd]: mgmtgwd.jobmgr.private.jobstart: Started private job "coredump-autosave" [id 8]: .
  238. May 5 08:34:40 [coredump_manager]: coredump.findcore.missing: Partial core missing disk 1 of 12
  239. May 5 08:34:40 [coredump_manager]: coredump.findcore.partial: Partial core is missing 1 of 12 disks
  240. May 5 08:34:41 [coredump_manager]: coredump.findcore.missing: Partial core missing disk 1 of 12
  241. May 5 08:34:41 [coredump_manager]: coredump.findcore.partial: Partial core is missing 1 of 12 disks
  242. May 5 08:34:42 [coredump_manager]: coredump.findcore.missing: Partial core missing disk 1 of 12
  243. May 5 08:34:42 [coredump_manager]: coredump.findcore.partial: Partial core is missing 1 of 12 disks
  244. May 5 08:34:42 [mgmtgwd]: mgmtgwd.jobmgr.private.jobstart: Started private job "sync task" [id 7]: sync task.
  245. May 5 08:34:43 [mgmtgwd]: mgmtgwd.jobmgr.private.jobcomplete.success: Private job "sync task" [id 7] (sync task) completed successfully: .
  246. May 5 08:34:43 [coredump_manager]: coredump.findcore.missing: Partial core missing disk 1 of 12
  247. May 5 08:34:43 [coredump_manager]: coredump.findcore.partial: Partial core is missing 1 of 12 disks
  248. May 5 08:34:44 [savecore_admin]: coredump.findcore.missing: Partial core missing disk 1 of 12
  249. May 5 08:34:44 [savecore_admin]: coredump.findcore.partial: Partial core is missing 1 of 12 disks
  250. May 5 08:34:44 [savecore_admin]: coredump.save.started: Saving 4558M to core.101185225.2013-04-22.17_53_15.nz (\"Freeing free block 9009630 in aggregate engdata1.\")
  251. May 5 08:34:44 [savecore_admin]: coredump.save.nospace: Not enough space in root volume to save core
  252. May 5 08:34:44 [savecore_admin]: coredump.save.error: core.101185225.2013-04-22.17_53_15.nz processing encountered error
  253. May 5 08:34:44 [savecore_admin]: coredump.save.started: Saving 4474M to core.101185225.2013-04-22.19_04_41.nz (\"Freeing free block 9009630 in aggregate engdata1.\")
  254. May 5 08:34:44 [savecore_admin]: coredump.save.nospace: Not enough space in root volume to save core
  255. May 5 08:34:44 [savecore_admin]: coredump.save.error: core.101185225.2013-04-22.19_04_41.nz processing encountered error
  256. May 5 08:34:44 [savecore_admin]: coredump.save.started: Saving 4556M to core.101185225.2013-04-22.20_18_41.nz (\"Freeing free block 9009630 in aggregate engdata1.\")
  257. May 5 08:34:44 [savecore_admin]: coredump.save.nospace: Not enough space in root volume to save core
  258. May 5 08:34:44 [savecore_admin]: coredump.save.error: core.101185225.2013-04-22.20_18_41.nz processing encountered error
  259. May 5 08:34:44 [savecore_admin]: coredump.save.started: Saving 4556M to core.101185225.2013-04-22.21_31_46.nz (\"Freeing free block 9009630 in aggregate engdata1.\")
  260. May 5 08:34:44 [savecore_admin]: coredump.save.nospace: Not enough space in root volume to save core
  261. May 5 08:34:44 [savecore_admin]: coredump.save.error: core.101185225.2013-04-22.21_31_46.nz processing encountered error
  262. May 5 08:34:56 [vifMgr]: vifmgr.startedsuccessfully: The Virtual Interface Manager (VIFMgr) started up successfully.
  263. May 5 08:35:02 [config_thread]: raid.rg.media_scrub.start: owner="", rg="/aggr0/plex0/rg0"
  264. May 5 08:35:02 [monitor]: monitor.globalStatus.critical: Cluster failover of toast1b is not possible: cluster takeover disabled. Disk on adapter 0c, shelf 3, bay 12, failed. Disk on adapter 0c, shelf 3, bay 5, failed. Disk on adapter 0c, shelf 3, bay 9, failed. Disk on adapter 0b, shelf 2, bay 9, failed. Disk on adapter 0b, shelf 2, bay 12, failed. Disk on adapter 0a, shelf 1, bay There are not enough spare disks. Disk shelf fault.
  265. May 5 08:35:08 [vifMgr]: vifmgr.vifsuccessfullymoved: VIF clus1 (on virtual server 0), ID 1024, is now hosted on node toast1a, port e0a.
  266. May 5 08:35:09 [vifMgr]: vifmgr.vifsuccessfullymoved: VIF clus2 (on virtual server 0), ID 1023, is now hosted on node toast1a, port e0b.
  267. May 5 08:35:13 [mgmtgwd]: mgmtgwd.jobmgr.private.jobstart: Started private job "sync task" [id 7]: sync task.
  268. May 5 08:35:13 [mgmtgwd]: mgmtgwd.jobmgr.private.jobcomplete.success: Private job "sync task" [id 7] (sync task) completed successfully: .
  269. May 5 08:39:43 [coredump_manager]: coredump.findcore.missing: Partial core missing disk 1 of 12
  270. May 5 08:39:43 [coredump_manager]: coredump.findcore.partial: Partial core is missing 1 of 12 disks
  271. May 5 08:39:44 [coredump_manager]: coredump.findcore.missing: Partial core missing disk 1 of 12
  272. May 5 08:39:44 [coredump_manager]: coredump.findcore.partial: Partial core is missing 1 of 12 disks
  273. May 5 08:39:45 [coredump_manager]: coredump.findcore.missing: Partial core missing disk 1 of 12
  274. May 5 08:39:45 [coredump_manager]: coredump.findcore.partial: Partial core is missing 1 of 12 disks
  275. May 5 08:39:46 [coredump_manager]: coredump.findcore.missing: Partial core missing disk 1 of 12
  276. May 5 08:39:46 [coredump_manager]: coredump.findcore.partial: Partial core is missing 1 of 12 disks
  277. May 5 08:39:47 [savecore_admin]: coredump.findcore.missing: Partial core missing disk 1 of 12
  278. May 5 08:39:47 [savecore_admin]: coredump.findcore.partial: Partial core is missing 1 of 12 disks
  279. May 5 08:39:47 [savecore_admin]: coredump.save.started: Saving 4558M to core.101185225.2013-04-22.17_53_15.nz (\"Freeing free block 9009630 in aggregate engdata1.\")
  280. May 5 08:39:47 [savecore_admin]: coredump.save.nospace: Not enough space in root volume to save core
  281. May 5 08:39:47 [savecore_admin]: coredump.save.error: core.101185225.2013-04-22.17_53_15.nz processing encountered error
  282. May 5 08:39:47 [savecore_admin]: coredump.save.started: Saving 4474M to core.101185225.2013-04-22.19_04_41.nz (\"Freeing free block 9009630 in aggregate engdata1.\")
  283. May 5 08:39:47 [savecore_admin]: coredump.save.nospace: Not enough space in root volume to save core
  284. May 5 08:39:47 [savecore_admin]: coredump.save.error: core.101185225.2013-04-22.19_04_41.nz processing encountered error
  285. May 5 08:39:47 [savecore_admin]: coredump.save.started: Saving 4556M to core.101185225.2013-04-22.20_18_41.nz (\"Freeing free block 9009630 in aggregate engdata1.\")
  286. May 5 08:39:47 [savecore_admin]: coredump.save.nospace: Not enough space in root volume to save core
  287. May 5 08:39:47 [savecore_admin]: coredump.save.error: core.101185225.2013-04-22.20_18_41.nz processing encountered error
  288. May 5 08:39:47 [savecore_admin]: coredump.save.started: Saving 4556M to core.101185225.2013-04-22.21_31_46.nz (\"Freeing free block 9009630 in aggregate engdata1.\")
  289. May 5 08:39:47 [savecore_admin]: coredump.save.nospace: Not enough space in root volume to save core
  290. May 5 08:39:47 [savecore_admin]: coredump.save.error: core.101185225.2013-04-22.21_31_46.nz processing encountered error
  291. May 5 08:44:46 [coredump_manager]: coredump.findcore.missing: Partial core missing disk 1 of 12
  292. May 5 08:44:46 [coredump_manager]: coredump.findcore.partial: Partial core is missing 1 of 12 disks
  293. May 5 08:44:47 [coredump_manager]: coredump.findcore.missing: Partial core missing disk 1 of 12
  294. May 5 08:44:47 [coredump_manager]: coredump.findcore.partial: Partial core is missing 1 of 12 disks
  295. May 5 08:44:48 [coredump_manager]: coredump.findcore.missing: Partial core missing disk 1 of 12
  296. May 5 08:44:48 [coredump_manager]: coredump.findcore.partial: Partial core is missing 1 of 12 disks
  297. May 5 08:44:49 [coredump_manager]: coredump.findcore.missing: Partial core missing disk 1 of 12
  298. May 5 08:44:49 [coredump_manager]: coredump.findcore.partial: Partial core is missing 1 of 12 disks
  299. May 5 08:44:50 [savecore_admin]: coredump.findcore.missing: Partial core missing disk 1 of 12
  300. May 5 08:44:50 [savecore_admin]: coredump.findcore.partial: Partial core is missing 1 of 12 disks
  301. May 5 08:44:50 [savecore_admin]: coredump.save.started: Saving 4558M to core.101185225.2013-04-22.17_53_15.nz (\"Freeing free block 9009630 in aggregate engdata1.\")
  302. May 5 08:44:50 [savecore_admin]: coredump.save.nospace: Not enough space in root volume to save core
  303. May 5 08:44:50 [savecore_admin]: coredump.save.error: core.101185225.2013-04-22.17_53_15.nz processing encountered error
  304. May 5 08:44:50 [savecore_admin]: coredump.save.started: Saving 4474M to core.101185225.2013-04-22.19_04_41.nz (\"Freeing free block 9009630 in aggregate engdata1.\")
  305. May 5 08:44:50 [savecore_admin]: coredump.save.nospace: Not enough space in root volume to save core
  306. May 5 08:44:50 [savecore_admin]: coredump.save.error: core.101185225.2013-04-22.19_04_41.nz processing encountered error
  307. May 5 08:44:50 [savecore_admin]: coredump.save.started: Saving 4556M to core.101185225.2013-04-22.20_18_41.nz (\"Freeing free block 9009630 in aggregate engdata1.\")
  308. May 5 08:44:50 [savecore_admin]: coredump.save.nospace: Not enough space in root volume to save core
  309. May 5 08:44:50 [savecore_admin]: coredump.save.error: core.101185225.2013-04-22.20_18_41.nz processing encountered error
  310. May 5 08:44:50 [savecore_admin]: coredump.save.started: Saving 4556M to core.101185225.2013-04-22.21_31_46.nz (\"Freeing free block 9009630 in aggregate engdata1.\")
  311. May 5 08:44:50 [savecore_admin]: coredump.save.nospace: Not enough space in root volume to save core
  312. May 5 08:44:50 [savecore_admin]: coredump.save.error: core.101185225.2013-04-22.21_31_46.nz processing encountered error
  313. May 5 08:49:49 [coredump_manager]: coredump.findcore.missing: Partial core missing disk 1 of 12
  314. May 5 08:49:49 [coredump_manager]: coredump.findcore.partial: Partial core is missing 1 of 12 disks
  315. May 5 08:49:50 [coredump_manager]: coredump.findcore.missing: Partial core missing disk 1 of 12
  316. May 5 08:49:50 [coredump_manager]: coredump.findcore.partial: Partial core is missing 1 of 12 disks
  317. May 5 08:49:51 [coredump_manager]: coredump.findcore.missing: Partial core missing disk 1 of 12
  318. May 5 08:49:51 [coredump_manager]: coredump.findcore.partial: Partial core is missing 1 of 12 disks
  319. May 5 08:49:52 [coredump_manager]: coredump.findcore.missing: Partial core missing disk 1 of 12
  320. May 5 08:49:52 [coredump_manager]: coredump.findcore.partial: Partial core is missing 1 of 12 disks
  321. May 5 08:49:53 [savecore_admin]: coredump.findcore.missing: Partial core missing disk 1 of 12
  322. May 5 08:49:53 [savecore_admin]: coredump.findcore.partial: Partial core is missing 1 of 12 disks
  323. May 5 08:49:53 [savecore_admin]: coredump.save.started: Saving 4558M to core.101185225.2013-04-22.17_53_15.nz (\"Freeing free block 9009630 in aggregate engdata1.\")
  324. May 5 08:49:53 [savecore_admin]: coredump.save.nospace: Not enough space in root volume to save core
  325. May 5 08:49:53 [savecore_admin]: coredump.save.error: core.101185225.2013-04-22.17_53_15.nz processing encountered error
  326. May 5 08:49:53 [savecore_admin]: coredump.save.started: Saving 4474M to core.101185225.2013-04-22.19_04_41.nz (\"Freeing free block 9009630 in aggregate engdata1.\")
  327. May 5 08:49:53 [savecore_admin]: coredump.save.nospace: Not enough space in root volume to save core
  328. May 5 08:49:53 [savecore_admin]: coredump.save.error: core.101185225.2013-04-22.19_04_41.nz processing encountered error
  329. May 5 08:49:53 [savecore_admin]: coredump.save.started: Saving 4556M to core.101185225.2013-04-22.20_18_41.nz (\"Freeing free block 9009630 in aggregate engdata1.\")
  330. May 5 08:49:53 [savecore_admin]: coredump.save.nospace: Not enough space in root volume to save core
  331. May 5 08:49:53 [savecore_admin]: coredump.save.error: core.101185225.2013-04-22.20_18_41.nz processing encountered error
  332. May 5 08:49:53 [savecore_admin]: coredump.save.started: Saving 4556M to core.101185225.2013-04-22.21_31_46.nz (\"Freeing free block 9009630 in aggregate engdata1.\")
  333. May 5 08:49:53 [savecore_admin]: coredump.save.nospace: Not enough space in root volume to save core
  334. May 5 08:49:53 [savecore_admin]: coredump.save.error: core.101185225.2013-04-22.21_31_46.nz processing encountered error
  335. May 5 08:54:52 [coredump_manager]: coredump.findcore.missing: Partial core missing disk 1 of 12
  336. May 5 08:54:52 [coredump_manager]: coredump.findcore.partial: Partial core is missing 1 of 12 disks
  337. May 5 08:54:53 [coredump_manager]: coredump.findcore.missing: Partial core missing disk 1 of 12
  338. May 5 08:54:53 [coredump_manager]: coredump.findcore.partial: Partial core is missing 1 of 12 disks
  339. May 5 08:54:54 [coredump_manager]: coredump.findcore.missing: Partial core missing disk 1 of 12
  340. May 5 08:54:54 [coredump_manager]: coredump.findcore.partial: Partial core is missing 1 of 12 disks
  341. May 5 08:54:55 [coredump_manager]: coredump.findcore.missing: Partial core missing disk 1 of 12
  342. May 5 08:54:55 [coredump_manager]: coredump.findcore.partial: Partial core is missing 1 of 12 disks
  343. May 5 08:54:56 [savecore_admin]: coredump.findcore.missing: Partial core missing disk 1 of 12
  344. May 5 08:54:56 [savecore_admin]: coredump.findcore.partial: Partial core is missing 1 of 12 disks
  345. May 5 08:54:56 [savecore_admin]: coredump.save.started: Saving 4558M to core.101185225.2013-04-22.17_53_15.nz (\"Freeing free block 9009630 in aggregate engdata1.\")
  346. May 5 08:54:56 [savecore_admin]: coredump.save.nospace: Not enough space in root volume to save core
  347. May 5 08:54:56 [savecore_admin]: coredump.save.error: core.101185225.2013-04-22.17_53_15.nz processing encountered error
  348. May 5 08:54:56 [savecore_admin]: coredump.save.started: Saving 4474M to core.101185225.2013-04-22.19_04_41.nz (\"Freeing free block 9009630 in aggregate engdata1.\")
  349. May 5 08:54:56 [savecore_admin]: coredump.save.nospace: Not enough space in root volume to save core
  350. May 5 08:54:56 [savecore_admin]: coredump.save.error: core.101185225.2013-04-22.19_04_41.nz processing encountered error
  351. May 5 08:54:56 [savecore_admin]: coredump.save.started: Saving 4556M to core.101185225.2013-04-22.20_18_41.nz (\"Freeing free block 9009630 in aggregate engdata1.\")
  352. May 5 08:54:56 [savecore_admin]: coredump.save.nospace: Not enough space in root volume to save core
  353. May 5 08:54:56 [savecore_admin]: coredump.save.error: core.101185225.2013-04-22.20_18_41.nz processing encountered error
  354. May 5 08:54:56 [savecore_admin]: coredump.save.started: Saving 4556M to core.101185225.2013-04-22.21_31_46.nz (\"Freeing free block 9009630 in aggregate engdata1.\")
  355. May 5 08:54:56 [savecore_admin]: coredump.save.nospace: Not enough space in root volume to save core
  356. May 5 08:54:56 [savecore_admin]: coredump.save.error: core.101185225.2013-04-22.21_31_46.nz processing encountered error
  357. May 5 08:59:55 [coredump_manager]: coredump.findcore.missing: Partial core missing disk 1 of 12
  358. May 5 08:59:55 [coredump_manager]: coredump.findcore.partial: Partial core is missing 1 of 12 disks
  359. May 5 08:59:56 [coredump_manager]: coredump.findcore.missing: Partial core missing disk 1 of 12
  360. May 5 08:59:56 [coredump_manager]: coredump.findcore.partial: Partial core is missing 1 of 12 disks
  361. May 5 08:59:57 [coredump_manager]: coredump.findcore.missing: Partial core missing disk 1 of 12
  362. May 5 08:59:57 [coredump_manager]: coredump.findcore.partial: Partial core is missing 1 of 12 disks
  363. May 5 08:59:58 [coredump_manager]: coredump.findcore.missing: Partial core missing disk 1 of 12
  364. May 5 08:59:58 [coredump_manager]: coredump.findcore.partial: Partial core is missing 1 of 12 disks
  365. May 5 08:59:58 [savecore_admin]: coredump.findcore.missing: Partial core missing disk 1 of 12
  366. May 5 08:59:58 [savecore_admin]: coredump.findcore.partial: Partial core is missing 1 of 12 disks
  367. May 5 08:59:58 [savecore_admin]: coredump.save.started: Saving 4558M to core.101185225.2013-04-22.17_53_15.nz (\"Freeing free block 9009630 in aggregate engdata1.\")
  368. May 5 08:59:58 [savecore_admin]: coredump.save.nospace: Not enough space in root volume to save core
  369. May 5 08:59:58 [savecore_admin]: coredump.save.error: core.101185225.2013-04-22.17_53_15.nz processing encountered error
  370. May 5 08:59:58 [savecore_admin]: coredump.save.started: Saving 4474M to core.101185225.2013-04-22.19_04_41.nz (\"Freeing free block 9009630 in aggregate engdata1.\")
  371. May 5 08:59:58 [savecore_admin]: coredump.save.nospace: Not enough space in root volume to save core
  372. May 5 08:59:58 [savecore_admin]: coredump.save.error: core.101185225.2013-04-22.19_04_41.nz processing encountered error
  373. May 5 08:59:58 [savecore_admin]: coredump.save.started: Saving 4556M to core.101185225.2013-04-22.20_18_41.nz (\"Freeing free block 9009630 in aggregate engdata1.\")
  374. May 5 08:59:58 [savecore_admin]: coredump.save.nospace: Not enough space in root volume to save core
  375. May 5 08:59:58 [savecore_admin]: coredump.save.error: core.101185225.2013-04-22.20_18_41.nz processing encountered error
  376. May 5 08:59:58 [savecore_admin]: coredump.save.started: Saving 4556M to core.101185225.2013-04-22.21_31_46.nz (\"Freeing free block 9009630 in aggregate engdata1.\")
  377. May 5 08:59:58 [savecore_admin]: coredump.save.nospace: Not enough space in root volume to save core
  378. May 5 08:59:58 [savecore_admin]: coredump.save.error: core.101185225.2013-04-22.21_31_46.nz processing encountered error
  379. May 5 09:00:00 [statd]: kern.uptime.filer: 9:00am up 26 mins, 2994815 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 0 iSCSI ops
  380. May 5 09:00:00 [statd]: monitor.shelf.fault: Fault reported on disk storage shelf attached to channel 0b. Please check fans, power, and temperature.
  381. May 5 09:00:00 [statd]: monitor.raiddp.vol.singleDegraded: data disk in RAID group /engdata1/plex0/rg2 is broken.
  382. May 5 09:00:01 [asup_main]: asup.throttle.drop: Too many autosupport messages in too short a time, throttling autosupport: DRIVE NHT DATA
  383. May 5 09:00:01 [statd]: monitor.nht.count.exceeded
  384. May 5 09:00:01 [statd]: kern.syslog.msg: Cluster is licensed but takeover of partner is disabled
  385. May 5 09:04:58 [coredump_manager]: coredump.findcore.missing: Partial core missing disk 1 of 12
  386. May 5 09:04:58 [coredump_manager]: coredump.findcore.partial: Partial core is missing 1 of 12 disks
  387. May 5 09:04:59 [coredump_manager]: coredump.findcore.missing: Partial core missing disk 1 of 12
  388. May 5 09:04:59 [coredump_manager]: coredump.findcore.partial: Partial core is missing 1 of 12 disks
  389. May 5 09:05:00 [mgmtgwd]: mgmtgwd.jobmgr.private.jobstart: Started private job "snap-hourly" [id 4]: Auto-Snapshot.
  390. May 5 09:05:00 [coredump_manager]: coredump.findcore.missing: Partial core missing disk 1 of 12
  391. May 5 09:05:00 [coredump_manager]: coredump.findcore.partial: Partial core is missing 1 of 12 disks
  392. May 5 09:05:01 [coredump_manager]: coredump.findcore.missing: Partial core missing disk 1 of 12
  393. May 5 09:05:01 [coredump_manager]: coredump.findcore.partial: Partial core is missing 1 of 12 disks
  394. May 5 09:05:01 [savecore_admin]: coredump.findcore.missing: Partial core missing disk 1 of 12
  395. May 5 09:05:01 [savecore_admin]: coredump.findcore.partial: Partial core is missing 1 of 12 disks
  396. May 5 09:05:01 [savecore_admin]: coredump.save.started: Saving 4558M to core.101185225.2013-04-22.17_53_15.nz (\"Freeing free block 9009630 in aggregate engdata1.\")
  397. May 5 09:05:01 [savecore_admin]: coredump.save.nospace: Not enough space in root volume to save core
  398. May 5 09:05:01 [savecore_admin]: coredump.save.error: core.101185225.2013-04-22.17_53_15.nz processing encountered error
  399. May 5 09:05:01 [savecore_admin]: coredump.save.started: Saving 4474M to core.101185225.2013-04-22.19_04_41.nz (\"Freeing free block 9009630 in aggregate engdata1.\")
  400. May 5 09:05:01 [savecore_admin]: coredump.save.nospace: Not enough space in root volume to save core
  401. May 5 09:05:01 [savecore_admin]: coredump.save.error: core.101185225.2013-04-22.19_04_41.nz processing encountered error
  402. May 5 09:05:01 [savecore_admin]: coredump.save.started: Saving 4556M to core.101185225.2013-04-22.20_18_41.nz (\"Freeing free block 9009630 in aggregate engdata1.\")
  403. May 5 09:05:01 [savecore_admin]: coredump.save.nospace: Not enough space in root volume to save core
  404. May 5 09:05:01 [savecore_admin]: coredump.save.error: core.101185225.2013-04-22.20_18_41.nz processing encountered error
  405. May 5 09:05:01 [savecore_admin]: coredump.save.started: Saving 4556M to core.101185225.2013-04-22.21_31_46.nz (\"Freeing free block 9009630 in aggregate engdata1.\")
  406. May 5 09:05:01 [savecore_admin]: coredump.save.nospace: Not enough space in root volume to save core
  407. May 5 09:05:01 [savecore_admin]: coredump.save.error: core.101185225.2013-04-22.21_31_46.nz processing encountered error
  408. May 5 09:05:39 [mgmtgwd]: mgmtgwd.jobmgr.private.jobcomplete.success: Private job "snap-hourly" [id 4] (Auto-Snapshot) completed successfully: .
  409. May 5 09:10:03 [coredump_manager]: coredump.findcore.missing: Partial core missing disk 1 of 12
  410. May 5 09:10:03 [coredump_manager]: coredump.findcore.partial: Partial core is missing 1 of 12 disks
  411. May 5 09:10:04 [coredump_manager]: coredump.findcore.missing: Partial core missing disk 1 of 12
  412. May 5 09:10:04 [coredump_manager]: coredump.findcore.partial: Partial core is missing 1 of 12 disks
  413. May 5 09:10:08 [coredump_manager]: coredump.findcore.missing: Partial core missing disk 1 of 12
  414. May 5 09:10:08 [coredump_manager]: coredump.findcore.partial: Partial core is missing 1 of 12 disks
  415. May 5 09:10:10 [coredump_manager]: coredump.findcore.missing: Partial core missing disk 1 of 12
  416. May 5 09:10:10 [coredump_manager]: coredump.findcore.partial: Partial core is missing 1 of 12 disks
  417. May 5 09:10:11 [savecore_admin]: coredump.findcore.missing: Partial core missing disk 1 of 12
  418. May 5 09:10:11 [savecore_admin]: coredump.findcore.partial: Partial core is missing 1 of 12 disks
  419. May 5 09:10:11 [savecore_admin]: coredump.save.started: Saving 4558M to core.101185225.2013-04-22.17_53_15.nz (\"Freeing free block 9009630 in aggregate engdata1.\")
  420. May 5 09:10:11 [savecore_admin]: coredump.save.nospace: Not enough space in root volume to save core
  421. May 5 09:10:11 [savecore_admin]: coredump.save.error: core.101185225.2013-04-22.17_53_15.nz processing encountered error
  422. May 5 09:10:11 [savecore_admin]: coredump.save.started: Saving 4474M to core.101185225.2013-04-22.19_04_41.nz (\"Freeing free block 9009630 in aggregate engdata1.\")
  423. May 5 09:10:11 [savecore_admin]: coredump.save.nospace: Not enough space in root volume to save core
  424. May 5 09:10:11 [savecore_admin]: coredump.save.error: core.101185225.2013-04-22.19_04_41.nz processing encountered error
  425. May 5 09:10:11 [savecore_admin]: coredump.save.started: Saving 4556M to core.101185225.2013-04-22.20_18_41.nz (\"Freeing free block 9009630 in aggregate engdata1.\")
  426. May 5 09:10:11 [savecore_admin]: coredump.save.nospace: Not enough space in root volume to save core
  427. May 5 09:10:11 [savecore_admin]: coredump.save.error: core.101185225.2013-04-22.20_18_41.nz processing encountered error
  428. May 5 09:10:11 [savecore_admin]: coredump.save.started: Saving 4556M to core.101185225.2013-04-22.21_31_46.nz (\"Freeing free block 9009630 in aggregate engdata1.\")
  429. May 5 09:10:11 [savecore_admin]: coredump.save.nospace: Not enough space in root volume to save core
  430. May 5 09:10:11 [savecore_admin]: coredump.save.error: core.101185225.2013-04-22.21_31_46.nz processing encountered error
  431. May 5 09:15:10 [coredump_manager]: coredump.findcore.missing: Partial core missing disk 1 of 12
  432. May 5 09:15:10 [coredump_manager]: coredump.findcore.partial: Partial core is missing 1 of 12 disks
  433. May 5 09:15:11 [coredump_manager]: coredump.findcore.missing: Partial core missing disk 1 of 12
  434. May 5 09:15:11 [coredump_manager]: coredump.findcore.partial: Partial core is missing 1 of 12 disks
  435. May 5 09:15:12 [coredump_manager]: coredump.findcore.missing: Partial core missing disk 1 of 12
  436. May 5 09:15:12 [coredump_manager]: coredump.findcore.partial: Partial core is missing 1 of 12 disks
  437. May 5 09:15:13 [coredump_manager]: coredump.findcore.missing: Partial core missing disk 1 of 12
  438. May 5 09:15:13 [coredump_manager]: coredump.findcore.partial: Partial core is missing 1 of 12 disks
  439. May 5 09:15:15 [savecore_admin]: coredump.findcore.missing: Partial core missing disk 1 of 12
  440. May 5 09:15:15 [savecore_admin]: coredump.findcore.partial: Partial core is missing 1 of 12 disks
  441. May 5 09:15:15 [savecore_admin]: coredump.save.started: Saving 4558M to core.101185225.2013-04-22.17_53_15.nz (\"Freeing free block 9009630 in aggregate engdata1.\")
  442. May 5 09:15:15 [savecore_admin]: coredump.save.nospace: Not enough space in root volume to save core
  443. May 5 09:15:15 [savecore_admin]: coredump.save.error: core.101185225.2013-04-22.17_53_15.nz processing encountered error
  444. May 5 09:15:15 [savecore_admin]: coredump.save.started: Saving 4474M to core.101185225.2013-04-22.19_04_41.nz (\"Freeing free block 9009630 in aggregate engdata1.\")
  445. May 5 09:15:15 [savecore_admin]: coredump.save.nospace: Not enough space in root volume to save core
  446. May 5 09:15:15 [savecore_admin]: coredump.save.error: core.101185225.2013-04-22.19_04_41.nz processing encountered error
  447. May 5 09:15:15 [savecore_admin]: coredump.save.started: Saving 4556M to core.101185225.2013-04-22.20_18_41.nz (\"Freeing free block 9009630 in aggregate engdata1.\")
  448. May 5 09:15:15 [savecore_admin]: coredump.save.nospace: Not enough space in root volume to save core
  449. May 5 09:15:15 [savecore_admin]: coredump.save.error: core.101185225.2013-04-22.20_18_41.nz processing encountered error
  450. May 5 09:15:15 [savecore_admin]: coredump.save.started: Saving 4556M to core.101185225.2013-04-22.21_31_46.nz (\"Freeing free block 9009630 in aggregate engdata1.\")
  451. May 5 09:15:15 [savecore_admin]: coredump.save.nospace: Not enough space in root volume to save core
  452. May 5 09:15:15 [savecore_admin]: coredump.save.error: core.101185225.2013-04-22.21_31_46.nz processing encountered error
  453. May 5 09:20:13 [coredump_manager]: coredump.findcore.missing: Partial core missing disk 1 of 12
  454. May 5 09:20:13 [coredump_manager]: coredump.findcore.partial: Partial core is missing 1 of 12 disks
  455. May 5 09:20:14 [coredump_manager]: coredump.findcore.missing: Partial core missing disk 1 of 12
  456. May 5 09:20:14 [coredump_manager]: coredump.findcore.partial: Partial core is missing 1 of 12 disks
  457. May 5 09:20:15 [coredump_manager]: coredump.findcore.missing: Partial core missing disk 1 of 12
  458. May 5 09:20:15 [coredump_manager]: coredump.findcore.partial: Partial core is missing 1 of 12 disks
  459. May 5 09:20:16 [coredump_manager]: coredump.findcore.missing: Partial core missing disk 1 of 12
  460. May 5 09:20:16 [coredump_manager]: coredump.findcore.partial: Partial core is missing 1 of 12 disks
  461. May 5 09:20:17 [savecore_admin]: coredump.findcore.missing: Partial core missing disk 1 of 12
  462. May 5 09:20:17 [savecore_admin]: coredump.findcore.partial: Partial core is missing 1 of 12 disks
  463. May 5 09:20:17 [savecore_admin]: coredump.save.started: Saving 4558M to core.101185225.2013-04-22.17_53_15.nz (\"Freeing free block 9009630 in aggregate engdata1.\")
  464. May 5 09:20:17 [savecore_admin]: coredump.save.nospace: Not enough space in root volume to save core
  465. May 5 09:20:17 [savecore_admin]: coredump.save.error: core.101185225.2013-04-22.17_53_15.nz processing encountered error
  466. May 5 09:20:17 [savecore_admin]: coredump.save.started: Saving 4474M to core.101185225.2013-04-22.19_04_41.nz (\"Freeing free block 9009630 in aggregate engdata1.\")
  467. May 5 09:20:17 [savecore_admin]: coredump.save.nospace: Not enough space in root volume to save core
  468. May 5 09:20:17 [savecore_admin]: coredump.save.error: core.101185225.2013-04-22.19_04_41.nz processing encountered error
  469. May 5 09:20:17 [savecore_admin]: coredump.save.started: Saving 4556M to core.101185225.2013-04-22.20_18_41.nz (\"Freeing free block 9009630 in aggregate engdata1.\")
  470. May 5 09:20:17 [savecore_admin]: coredump.save.nospace: Not enough space in root volume to save core
  471. May 5 09:20:17 [savecore_admin]: coredump.save.error: core.101185225.2013-04-22.20_18_41.nz processing encountered error
  472. May 5 09:20:17 [savecore_admin]: coredump.save.started: Saving 4556M to core.101185225.2013-04-22.21_31_46.nz (\"Freeing free block 9009630 in aggregate engdata1.\")
  473. May 5 09:20:17 [savecore_admin]: coredump.save.nospace: Not enough space in root volume to save core
  474. May 5 09:20:17 [savecore_admin]: coredump.save.error: core.101185225.2013-04-22.21_31_46.nz processing encountered error
  475. May 5 09:25:16 [coredump_manager]: coredump.findcore.missing: Partial core missing disk 1 of 12
  476. May 5 09:25:16 [coredump_manager]: coredump.findcore.partial: Partial core is missing 1 of 12 disks
  477. May 5 09:25:17 [coredump_manager]: coredump.findcore.missing: Partial core missing disk 1 of 12
  478. May 5 09:25:17 [coredump_manager]: coredump.findcore.partial: Partial core is missing 1 of 12 disks
  479. May 5 09:25:18 [coredump_manager]: coredump.findcore.missing: Partial core missing disk 1 of 12
  480. May 5 09:25:18 [coredump_manager]: coredump.findcore.partial: Partial core is missing 1 of 12 disks
  481. May 5 09:25:19 [coredump_manager]: coredump.findcore.missing: Partial core missing disk 1 of 12
  482. May 5 09:25:19 [coredump_manager]: coredump.findcore.partial: Partial core is missing 1 of 12 disks
  483. May 5 09:25:20 [savecore_admin]: coredump.findcore.missing: Partial core missing disk 1 of 12
  484. May 5 09:25:20 [savecore_admin]: coredump.findcore.partial: Partial core is missing 1 of 12 disks
  485. May 5 09:25:20 [savecore_admin]: coredump.save.started: Saving 4558M to core.101185225.2013-04-22.17_53_15.nz (\"Freeing free block 9009630 in aggregate engdata1.\")
  486. May 5 09:25:20 [savecore_admin]: coredump.save.nospace: Not enough space in root volume to save core
  487. May 5 09:25:20 [savecore_admin]: coredump.save.error: core.101185225.2013-04-22.17_53_15.nz processing encountered error
  488. May 5 09:25:20 [savecore_admin]: coredump.save.started: Saving 4474M to core.101185225.2013-04-22.19_04_41.nz (\"Freeing free block 9009630 in aggregate engdata1.\")
  489. May 5 09:25:20 [savecore_admin]: coredump.save.nospace: Not enough space in root volume to save core
  490. May 5 09:25:20 [savecore_admin]: coredump.save.error: core.101185225.2013-04-22.19_04_41.nz processing encountered error
  491. May 5 09:25:20 [savecore_admin]: coredump.save.started: Saving 4556M to core.101185225.2013-04-22.20_18_41.nz (\"Freeing free block 9009630 in aggregate engdata1.\")
  492. May 5 09:25:20 [savecore_admin]: coredump.save.nospace: Not enough space in root volume to save core
  493. May 5 09:25:20 [savecore_admin]: coredump.save.error: core.101185225.2013-04-22.20_18_41.nz processing encountered error
  494. May 5 09:25:20 [savecore_admin]: coredump.save.started: Saving 4556M to core.101185225.2013-04-22.21_31_46.nz (\"Freeing free block 9009630 in aggregate engdata1.\")
  495. May 5 09:25:20 [savecore_admin]: coredump.save.nospace: Not enough space in root volume to save core
  496. May 5 09:25:20 [savecore_admin]: coredump.save.error: core.101185225.2013-04-22.21_31_46.nz processing encountered error
  497. May 5 09:30:19 [coredump_manager]: coredump.findcore.missing: Partial core missing disk 1 of 12
  498. May 5 09:30:19 [coredump_manager]: coredump.findcore.partial: Partial core is missing 1 of 12 disks
  499. May 5 09:30:20 [coredump_manager]: coredump.findcore.missing: Partial core missing disk 1 of 12
  500. May 5 09:30:20 [coredump_manager]: coredump.findcore.partial: Partial core is missing 1 of 12 disks
  501. May 5 09:30:21 [coredump_manager]: coredump.findcore.missing: Partial core missing disk 1 of 12
  502. May 5 09:30:21 [coredump_manager]: coredump.findcore.partial: Partial core is missing 1 of 12 disks
  503. May 5 09:30:22 [coredump_manager]: coredump.findcore.missing: Partial core missing disk 1 of 12
  504. May 5 09:30:22 [coredump_manager]: coredump.findcore.partial: Partial core is missing 1 of 12 disks
  505. May 5 09:30:23 [savecore_admin]: coredump.findcore.missing: Partial core missing disk 1 of 12
  506. May 5 09:30:23 [savecore_admin]: coredump.findcore.partial: Partial core is missing 1 of 12 disks
  507. May 5 09:30:23 [savecore_admin]: coredump.save.started: Saving 4558M to core.101185225.2013-04-22.17_53_15.nz (\"Freeing free block 9009630 in aggregate engdata1.\")
  508. May 5 09:30:23 [savecore_admin]: coredump.save.nospace: Not enough space in root volume to save core
  509. May 5 09:30:23 [savecore_admin]: coredump.save.error: core.101185225.2013-04-22.17_53_15.nz processing encountered error
  510. May 5 09:30:23 [savecore_admin]: coredump.save.started: Saving 4474M to core.101185225.2013-04-22.19_04_41.nz (\"Freeing free block 9009630 in aggregate engdata1.\")
  511. May 5 09:30:23 [savecore_admin]: coredump.save.nospace: Not enough space in root volume to save core
  512. May 5 09:30:23 [savecore_admin]: coredump.save.error: core.101185225.2013-04-22.19_04_41.nz processing encountered error
  513. May 5 09:30:23 [savecore_admin]: coredump.save.started: Saving 4556M to core.101185225.2013-04-22.20_18_41.nz (\"Freeing free block 9009630 in aggregate engdata1.\")
  514. May 5 09:30:23 [savecore_admin]: coredump.save.nospace: Not enough space in root volume to save core
  515. May 5 09:30:23 [savecore_admin]: coredump.save.error: core.101185225.2013-04-22.20_18_41.nz processing encountered error
  516. May 5 09:30:23 [savecore_admin]: coredump.save.started: Saving 4556M to core.101185225.2013-04-22.21_31_46.nz (\"Freeing free block 9009630 in aggregate engdata1.\")
  517. May 5 09:30:23 [savecore_admin]: coredump.save.nospace: Not enough space in root volume to save core
  518. May 5 09:30:23 [savecore_admin]: coredump.save.error: core.101185225.2013-04-22.21_31_46.nz processing encountered error
  519. May 5 09:34:01 [registry_control]: kern.syslog.msg: registry: Unexpected error in rollback in regx_commit - Error: No such file or directory (name=options.ssh.access) Error: No such file or directory (name=options.ssh.debug.enable) Error: No such file or directory (name=options.ssh.idle.timeout) Error: No such file or directory (name=options.ssh.passwd_auth.ena
  520. May 5 09:34:01 [registry_control]: reg.transaction.commitFail: registry: Cannot commit transaction in 'Periodic config update'. Error: Invalid argument (name=options.ssh.access) (value=*) Error: Invalid argument (name=options.ssh.debug.enable) (value=off) Error: Invalid argument (name=options.ssh.idle.timeout) (value=600) Error: Invalid argument (name=options.ssh.passwd_auth.enabl
  521. May 5 09:35:31 [console_login_mgr]: kern.syslog.msg: root logged in from console
  522. May 5 09:40:25 [coredump_manager]: coredump.findcore.missing: Partial core missing disk 1 of 12
  523. May 5 09:40:25 [coredump_manager]: coredump.findcore.partial: Partial core is missing 1 of 12 disks
  524. May 5 09:40:28 [savecore_admin]: coredump.save.started: Saving 4558M to core.101185225.2013-04-22.17_53_15.nz (\"Freeing free block 9009630 in aggregate engdata1.\")
  525. May 5 09:40:28 [savecore_admin]: coredump.save.nospace: Not enough space in root volume to save core
  526. May 5 09:40:28 [savecore_admin]: coredump.save.error: core.101185225.2013-04-22.17_53_15.nz processing encountered error
  527. May 5 09:50:35 [coredump_manager]: coredump.findcore.missing: Partial core missing disk 1 of 12
  528. May 5 09:50:35 [coredump_manager]: coredump.findcore.partial: Partial core is missing 1 of 12 disks
  529. May 5 09:50:39 [savecore_admin]: coredump.save.started: Saving 4558M to core.101185225.2013-04-22.17_53_15.nz (\"Freeing free block 9009630 in aggregate engdata1.\")
  530. May 5 09:50:39 [savecore_admin]: coredump.save.nospace: Not enough space in root volume to save core
  531. May 5 09:50:39 [savecore_admin]: coredump.save.error: core.101185225.2013-04-22.17_53_15.nz processing encountered error
  532. May 5 10:00:00 [statd]: kern.uptime.filer: 10:00am up 1:26 12086600 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 0 iSCSI ops
  533. May 5 10:00:00 [statd]: monitor.shelf.fault: Fault reported on disk storage shelf attached to channel 0b. Please check fans, power, and temperature.
  534. May 5 10:00:00 [statd]: monitor.raiddp.vol.singleDegraded: data disk in RAID group /engdata1/plex0/rg2 is broken.
  535. May 5 10:00:01 [statd]: kern.syslog.msg: Cluster is licensed but takeover of partner is disabled
  536. May 5 10:00:40 [coredump_manager]: coredump.findcore.missing: Partial core missing disk 1 of 12
  537. May 5 10:00:40 [coredump_manager]: coredump.findcore.partial: Partial core is missing 1 of 12 disks
  538. May 5 10:00:43 [savecore_admin]: coredump.save.started: Saving 4558M to core.101185225.2013-04-22.17_53_15.nz (\"Freeing free block 9009630 in aggregate engdata1.\")
  539. May 5 10:00:43 [savecore_admin]: coredump.save.nospace: Not enough space in root volume to save core
  540. May 5 10:00:43 [savecore_admin]: coredump.save.error: core.101185225.2013-04-22.17_53_15.nz processing encountered error
  541. May 5 10:05:00 [mgmtgwd]: mgmtgwd.jobmgr.private.jobstart: Started private job "snap-hourly" [id 4]: Auto-Snapshot.
  542. May 5 10:05:38 [mgmtgwd]: mgmtgwd.jobmgr.private.jobcomplete.success: Private job "snap-hourly" [id 4] (Auto-Snapshot) completed successfully: .
  543. May 5 10:10:46 [coredump_manager]: coredump.findcore.missing: Partial core missing disk 1 of 12
  544. May 5 10:10:46 [coredump_manager]: coredump.findcore.partial: Partial core is missing 1 of 12 disks
  545. May 5 10:10:50 [savecore_admin]: coredump.save.started: Saving 4558M to core.101185225.2013-04-22.17_53_15.nz (\"Freeing free block 9009630 in aggregate engdata1.\")
  546. May 5 10:10:50 [savecore_admin]: coredump.save.nospace: Not enough space in root volume to save core
  547. May 5 10:10:50 [savecore_admin]: coredump.save.error: core.101185225.2013-04-22.17_53_15.nz processing encountered error
  548. May 5 10:20:54 [coredump_manager]: coredump.findcore.missing: Partial core missing disk 1 of 12
  549. May 5 10:20:54 [coredump_manager]: coredump.findcore.partial: Partial core is missing 1 of 12 disks
  550. May 5 10:20:58 [savecore_admin]: coredump.save.started: Saving 4558M to core.101185225.2013-04-22.17_53_15.nz (\"Freeing free block 9009630 in aggregate engdata1.\")
  551. May 5 10:20:58 [savecore_admin]: coredump.save.nospace: Not enough space in root volume to save core
  552. May 5 10:20:58 [savecore_admin]: coredump.save.error: core.101185225.2013-04-22.17_53_15.nz processing encountered error
  553. May 5 10:20:58 [savecore_admin]: coredump.save.started: Saving 4474M to core.101185225.2013-04-22.19_04_41.nz (\"Freeing free block 9009630 in aggregate engdata1.\")
  554. May 5 10:20:58 [savecore_admin]: coredump.save.nospace: Not enough space in root volume to save core
  555. May 5 10:20:58 [savecore_admin]: coredump.save.error: core.101185225.2013-04-22.19_04_41.nz processing encountered error
  556. May 5 10:20:58 [savecore_admin]: coredump.save.started: Saving 4556M to core.101185225.2013-04-22.20_18_41.nz (\"Freeing free block 9009630 in aggregate engdata1.\")
  557. May 5 10:20:58 [savecore_admin]: coredump.save.nospace: Not enough space in root volume to save core
  558. May 5 10:20:58 [savecore_admin]: coredump.save.error: core.101185225.2013-04-22.20_18_41.nz processing encountered error
  559. May 5 10:20:58 [savecore_admin]: coredump.save.started: Saving 4556M to core.101185225.2013-04-22.21_31_46.nz (\"Freeing free block 9009630 in aggregate engdata1.\")
  560. May 5 10:20:58 [savecore_admin]: coredump.save.nospace: Not enough space in root volume to save core
  561. May 5 10:20:58 [savecore_admin]: coredump.save.error: core.101185225.2013-04-22.21_31_46.nz processing encountered error
  562. May 5 10:25:57 [coredump_manager]: coredump.findcore.missing: Partial core missing disk 1 of 12
  563. May 5 10:25:57 [coredump_manager]: coredump.findcore.partial: Partial core is missing 1 of 12 disks
  564. May 5 10:25:58 [coredump_manager]: coredump.findcore.missing: Partial core missing disk 1 of 12
  565. May 5 10:25:58 [coredump_manager]: coredump.findcore.partial: Partial core is missing 1 of 12 disks
  566. May 5 10:25:59 [coredump_manager]: coredump.findcore.missing: Partial core missing disk 1 of 12
  567. May 5 10:25:59 [coredump_manager]: coredump.findcore.partial: Partial core is missing 1 of 12 disks
  568. May 5 10:26:00 [coredump_manager]: coredump.findcore.missing: Partial core missing disk 1 of 12
  569. May 5 10:26:00 [coredump_manager]: coredump.findcore.partial: Partial core is missing 1 of 12 disks
  570. May 5 10:26:01 [savecore_admin]: coredump.findcore.missing: Partial core missing disk 1 of 12
  571. May 5 10:26:01 [savecore_admin]: coredump.findcore.partial: Partial core is missing 1 of 12 disks
  572. May 5 10:26:01 [savecore_admin]: coredump.save.started: Saving 4558M to core.101185225.2013-04-22.17_53_15.nz (\"Freeing free block 9009630 in aggregate engdata1.\")
  573. May 5 10:26:01 [savecore_admin]: coredump.save.nospace: Not enough space in root volume to save core
  574. May 5 10:26:01 [savecore_admin]: coredump.save.error: core.101185225.2013-04-22.17_53_15.nz processing encountered error
  575. May 5 10:26:01 [savecore_admin]: coredump.save.started: Saving 4474M to core.101185225.2013-04-22.19_04_41.nz (\"Freeing free block 9009630 in aggregate engdata1.\")
  576. May 5 10:26:01 [savecore_admin]: coredump.save.nospace: Not enough space in root volume to save core
  577. May 5 10:26:01 [savecore_admin]: coredump.save.error: core.101185225.2013-04-22.19_04_41.nz processing encountered error
  578. May 5 10:26:01 [savecore_admin]: coredump.save.started: Saving 4556M to core.101185225.2013-04-22.20_18_41.nz (\"Freeing free block 9009630 in aggregate engdata1.\")
  579. May 5 10:26:01 [savecore_admin]: coredump.save.nospace: Not enough space in root volume to save core
  580. May 5 10:26:01 [savecore_admin]: coredump.save.error: core.101185225.2013-04-22.20_18_41.nz processing encountered error
  581. May 5 10:26:01 [savecore_admin]: coredump.save.started: Saving 4556M to core.101185225.2013-04-22.21_31_46.nz (\"Freeing free block 9009630 in aggregate engdata1.\")
  582. May 5 10:26:01 [savecore_admin]: coredump.save.nospace: Not enough space in root volume to save core
  583. May 5 10:26:01 [savecore_admin]: coredump.save.error: core.101185225.2013-04-22.21_31_46.nz processing encountered error
  584. May 5 10:27:30 [wafl_junkd_aggroffl]: wafl.buf.freeingFreeBlock: Freeing free block 926442036 in aggregate engdata1.\n WAFL inconsistent. Please contact Network Appliance Global Services.
  585. May 5 10:27:31 [wafl_junkd_aggroffl]: sk.panic: Panic String: Freeing free block 926442036 in aggregate engdata1. WAFL inconsistent. Please contact Network Appliance Global Services. in SK process wafl_junkd_aggroffl on release NetApp Release R10.0.1xN_070123_1900
  586. May 5 10:27:31 [wafl_junkd_aggroffl]: coredump.findcore.missing: Partial core missing disk 1 of 12
  587. May 5 10:27:31 [wafl_junkd_aggroffl]: coredump.findcore.partial: Partial core is missing 1 of 12 disks
  588. May 5 10:27:31 [wafl_junkd_aggroffl]: coredump.dump.find.all.used: All available disks contain unsaved cores
  589. May 5 10:27:31 [wafl_junkd_aggroffl]: coredump.dump.failed: Dumpcore failed
  590. May 5 10:28:37 [nic_mgr]: cf.nm.nicTransitionUp: Interconnect link 0 is UP
  591. May 5 10:28:45 [ispfc_timeout_3]: fci.device.quiesce: Adapter unknown encountered a command timeout on 0b device 0b.36 (0x03000024). Quiescing the device.
  592. May 5 10:28:49 [sanown_notify]: diskown.isEnabled: software ownership has been enabled for this system
  593. May 5 10:28:52 [nic_mgr]: cf.nm.nicReset: Initiating soft reset on Cluster Interconnect card 0 due to rendezvous reset
  594. May 5 10:28:52 [rendezvous_proc]: cf.rv.notConnected: Connection for cfo_rv failed
  595. May 5 10:28:52 [nic_mgr]: cf.nm.nicTransitionDown: Cluster Interconnect link 0 is DOWN
  596. May 5 10:28:52 [rendezvous_proc]: cf.rv.notConnected: Connection for cfo_rv failed
  597. May 5 10:28:54 [nic_mgr]: cf.nm.nicTransitionUp: Interconnect link 0 is UP
  598. May 5 10:28:58 [fmmbx_instanceWorke]: fmmb.current.lock.disk: Disk 0c.58 is a local HA mailbox disk.
  599. May 5 10:28:58 [fmmbx_instanceWorke]: fmmb.current.lock.disk: Disk 0a.29 is a local HA mailbox disk.
  600. May 5 10:28:58 [fmmbx_instanceWorke]: kern.syslog.msg: normal mailbox instance on local side
  601. May 5 10:28:58 [rc]: cf.fm.MBstatusOnBoot: status="1"
  602. May 5 10:29:07 [config_thread]: raid.assim.disk.brokenPreAssim: Broken Disk 0c.60 Shelf ? Bay ? [NETAPP X267_MGRIZ500SSX H3DD] S/N [H814LZ6H] detected prior to assimilation. It should be removed.
  603. May 5 10:29:07 [config_thread]: raid.assim.disk.brokenPreAssim: Broken Disk 0b.44 Shelf ? Bay ? [NETAPP X267_MGRIZ500SSX H3DD] S/N [H817GR5H] detected prior to assimilation. It should be removed.
  604. May 5 10:29:07 [config_thread]: raid.assim.disk.brokenPreAssim: Broken Disk 0b.41 Shelf ? Bay ? [NETAPP X267_MGRIZ500SSX H3DD] S/N [H817LZ8H] detected prior to assimilation. It should be removed.
  605. May 5 10:29:07 [config_thread]: raid.assim.disk.brokenPreAssim: Broken Disk 0a.23 Shelf ? Bay ? [NETAPP X267_MGRIZ500SSX H3DD] S/N [H814LVPH] detected prior to assimilation. It should be removed.
  606. May 5 10:29:07 [config_thread]: raid.assim.disk.brokenPreAssim: Broken Disk 0c.53 Shelf ? Bay ? [NETAPP X267_MGRIZ500SSX H3DD] S/N [H8155VEH] detected prior to assimilation. It should be removed.
  607. May 5 10:29:07 [config_thread]: raid.assim.disk.brokenPreAssim: Broken Disk 0c.57 Shelf ? Bay ? [NETAPP X267_MGRIZ500SSX H3DD] S/N [H814R7RH] detected prior to assimilation. It should be removed.
  608. May 5 10:29:07 [fmmbx_instanceWorke]: fmmb.NewDiskSuggestion: count="4", new="0c.58, 0a.29, null, null", old="0c.58, 0a.29"
  609. May 5 10:29:07 [fmmbx_instanceWorke]: fmmb.current.lock.disk: Disk 0d.70 is a partner HA mailbox disk.
  610. May 5 10:29:07 [fmmbx_instanceWorke]: fmmb.current.lock.disk: Disk 0d.67 is a partner HA mailbox disk.
  611. May 5 10:29:07 [fmmbx_instanceWorke]: kern.syslog.msg: normal mailbox instance on partner side
  612. May 5 10:29:07 [rc]: cf.fm.partner: Cluster monitor: partner 'toast1b'
  613. May 5 10:29:07 [rc]: cf.fm.timeMasterStatus: Acting as cluster time master
  614. May 5 10:29:09 [ses_admin]: ses.channel.rescanInitiated: Initiating rescan on channel 0b
  615. May 5 10:29:14 [config_thread]: raid.vol.replay.nvram: Performing raid replay on volume(s)
  616. May 5 10:29:14 [rc]: raid.cksum.replay.summary: Replayed 0 checksum blocks.
  617. May 5 10:29:14 [rc]: raid.stripe.replay.summary: Replayed 0 stripes.
  618. May 5 10:29:15 [rc]: wafl.inconsistent.vol: WAFL inconsistent: Aggregate engdata1 is inconsistent. Note: Any new Snapshot copies might contain this inconsistency.
  619. May 5 10:29:15 [rc]: wafl.vol.loading: Loading Aggregate engdata1
  620. May 5 10:29:15 [rc]: wafl.vol.loading: Loading Aggregate aggr0
  621. May 5 10:29:15 [vv_apply_worker19]: wafl.vol.loading: Loading Volume bluecore@vserver:b1a2013d-acca-11e0-98c0-000e0caac15e
  622. May 5 10:29:15 [vv_apply_worker16]: wafl.vol.loading: Loading Volume mroot@vserver:06866009-a1fe-11e0-8650-000e0caac15e
  623. May 5 10:29:15 [vv_apply_worker17]: wafl.vol.loading: Loading Volume engdata1_root@vserver:b1a2013d-acca-11e0-98c0-000e0caac15e
  624. May 5 10:29:15 [vv_apply_worker18]: wafl.vol.loading: Loading Volume engdata1@vserver:b1a2013d-acca-11e0-98c0-000e0caac15e
  625. May 5 10:29:16 [vv_apply_worker19]: wafl.maxdirsize.boot.notice: bluecore@vserver:b1a2013d-acca-11e0-98c0-000e0caac15e: This volume's maxdirsize (102400KB) is higher than the default (35573KB). There may be a performance penalty when doing operations on large directories.
  626. May 5 10:29:16 [rc]: cf.fm.launch: Launching cluster monitor
  627. May 5 10:29:16 [rc]: cf.fm.partner: Cluster monitor: partner 'toast1b'
  628. May 5 10:29:16 [rc]: cf.fm.notkoverClusterDisable: Cluster monitor: cluster takeover disabled (restart)
  629. May 5 10:29:16 [cf_main]: cf.fsm.takeoverOfPartnerDisabled: Cluster monitor: takeover of toast1b disabled (cluster takeover disabled)
  630. May 5 10:29:27 [rc]: license.db.migrate.vol.success: Successfully migrated licenses for volume engdata1
  631. May 5 10:29:27 [rc]: license.db.migrate.vol.success: Successfully migrated licenses for volume aggr0
  632. May 5 10:29:27 [rc]: license.db.migrate.success: Migration complete
  633. May 5 10:29:27 [fmmbx_instanceWorke]: fmmb.NewDiskSuggestion: count="4", new="0c.58, 0a.29, null, null", old="0c.58, 0a.29"
  634. May 5 10:29:27 [rc]: kern.syslog.msg: The system was down for 101 seconds
  635. May 5 10:29:27 [raidio_thread]: raid.disk.unload.done: Unload of Disk 0c.60 Shelf 3 Bay 12 [NETAPP X267_MGRIZ500SSX H3DD] S/N [H814LZ6H] has completed successfully
  636. May 5 10:29:27 [raidio_thread]: raid.disk.unload.done: Unload of Disk 0b.44 Shelf 2 Bay 12 [NETAPP X267_MGRIZ500SSX H3DD] S/N [H817GR5H] has completed successfully
  637. May 5 10:29:27 [raidio_thread]: raid.disk.unload.done: Unload of Disk 0b.41 Shelf 2 Bay 9 [NETAPP X267_MGRIZ500SSX H3DD] S/N [H817LZ8H] has completed successfully
  638. May 5 10:29:27 [raidio_thread]: raid.disk.unload.done: Unload of Disk 0a.23 Shelf 1 Bay 7 [NETAPP X267_MGRIZ500SSX H3DD] S/N [H814LVPH] has completed successfully
  639. May 5 10:29:27 [raidio_thread]: raid.disk.unload.done: Unload of Disk 0c.53 Shelf 3 Bay 5 [NETAPP X267_MGRIZ500SSX H3DD] S/N [H8155VEH] has completed successfully
  640. May 5 10:29:27 [raidio_thread]: raid.disk.unload.done: Unload of Disk 0c.57 Shelf 3 Bay 9 [NETAPP X267_MGRIZ500SSX H3DD] S/N [H814R7RH] has completed successfully
  641. May 5 10:29:27 [rc]: raid.fdr.failed.ok: Disk 0c.60 Shelf 3 Bay 12 [NETAPP X267_MGRIZ500SSX H3DD] S/N [H814LZ6H] successfully deleted from spare pool
  642. May 5 10:29:27 [rc]: raid.fdr.failed.ok: Disk 0b.44 Shelf 2 Bay 12 [NETAPP X267_MGRIZ500SSX H3DD] S/N [H817GR5H] successfully deleted from spare pool
  643. May 5 10:29:27 [rc]: raid.fdr.failed.ok: Disk 0b.41 Shelf 2 Bay 9 [NETAPP X267_MGRIZ500SSX H3DD] S/N [H817LZ8H] successfully deleted from spare pool
  644. May 5 10:29:27 [rc]: raid.fdr.failed.ok: Disk 0a.23 Shelf 1 Bay 7 [NETAPP X267_MGRIZ500SSX H3DD] S/N [H814LVPH] successfully deleted from spare pool
  645. May 5 10:29:27 [rc]: raid.fdr.failed.ok: Disk 0c.53 Shelf 3 Bay 5 [NETAPP X267_MGRIZ500SSX H3DD] S/N [H8155VEH] successfully deleted from spare pool
  646. May 5 10:29:27 [rc]: raid.fdr.failed.ok: Disk 0c.57 Shelf 3 Bay 9 [NETAPP X267_MGRIZ500SSX H3DD] S/N [H814R7RH] successfully deleted from spare pool
  647. May 5 10:29:27 [rc]: kern.syslog.msg: registry: Unexpected error in rollback in regx_commit - Error: No such file or directory (name=options.ssh.access) Error: No such file or directory (name=options.ssh.debug.enable) Error: No such file or directory (name=options.ssh.idle.timeout) Error: No such file or directory (name=options.ssh.passwd_auth.ena
  648. May 5 10:29:27 [rc]: reg.transaction.commitFail: registry: Cannot commit transaction in 'Periodic config update'. Error: Invalid argument (name=options.ssh.access) (value=*) Error: Invalid argument (name=options.ssh.debug.enable) (value=off) Error: Invalid argument (name=options.ssh.idle.timeout) (value=600) Error: Invalid argument (name=options.ssh.passwd_auth.enabl
  649. May 5 10:29:27 [rc]: kern.syslog.msg: sysconfig: validating configuration
  650. May 5 10:29:27 [config_thread]: raid.rg.recons.missing: RAID group /engdata1/plex0/rg2 is missing 1 disk(s).
  651. May 5 10:29:27 [config_thread]: raid.rg.recons.cantStart: The reconstruction cannot start in RAID group /engdata1/plex0/rg2: No 520 bps disk of required size available in spare pool
  652. May 5 10:29:27 [rc]: wafl.inconsistent.vol: WAFL inconsistent: Aggregate engdata1 is inconsistent. Note: Any new Snapshot copies might contain this inconsistency.
  653. May 5 10:29:28 [rc]: mgr.boot.disk_done: NetApp Release ontap boot complete. Last disk update written at Sun May 5 10:27:25 GMT 2013
  654. May 5 10:29:28 [rc]: mgr.boot.reason_abnormal: System rebooted after a panic.
  655. May 5 10:29:28 [rc]: mgr.stack.module: Module kernel loaded from 0x80000000 - 0x100000000.
  656. May 5 10:29:28 [rc]: mgr.stack.module: Module common_kmod.ko loaded from 0x81a00000 - 0x81aa6000.
  657. May 5 10:29:28 [rc]: mgr.stack.module: Module nvram5.ko loaded from 0x81aa6000 - 0x81ab8000.
  658. May 5 10:29:28 [rc]: mgr.stack.module: Module nvram_mgr.ko loaded from 0x81ab8000 - 0x81abd000.
  659. May 5 10:29:28 [rc]: mgr.stack.module: Module nvr.ko loaded from 0x81abd000 - 0x81ad0000.
  660. May 5 10:29:28 [rc]: mgr.stack.module: Module maytag.ko loaded from 0x81ad0000 - 0x858db000.
  661. May 5 10:29:28 [rc]: mgr.stack.module: Module spinvfs.ko loaded from 0x858db000 - 0x85924000.
  662. May 5 10:29:28 [rc]: mgr.stack.module: Module nbladekmod.ko loaded from 0x85924000 - 0x876b8000.
  663. May 5 10:29:28 [rc]: mgr.stack.saved: Reboot with saved panic information in log file
  664. May 5 10:29:28 [rc]: mgr.stack.string: Panic string: Freeing free block 926442036 in aggregate engdata1. WAFL inconsistent. Please contact Network Appliance Global Services. in SK process wafl_junkd_aggroffl on r
  665. May 5 10:29:28 [rc]: mgr.stack.at: Panic occurred at: Sun May 5 10:27:31 2013
  666. May 5 10:29:28 [rc]: mgr.stack.proc: Panic in process: wafl_junkd_aggroffl
  667. May 5 10:29:28 [rc]: mgr.stack.framename: Stack frame 0: maytag.ko::ontap_dumpcore(0x82146b00) + 0xb4
  668. May 5 10:29:28 [rc]: mgr.stack.framename: Stack frame 1: kernel::dumpsys(0x80477260) + 0xf8
  669. May 5 10:29:28 [rc]: mgr.stack.framename: Stack frame 2: kernel::shutdown_get_howto(0x80324460) + 0x5d
  670. May 5 10:29:28 [rc]: mgr.stack.framename: Stack frame 3: kernel::shutdown_get_howto(0x80324460) + 0x45d
  671. May 5 10:29:28 [rc]: mgr.stack.framename: Stack frame 4: kernel::panic(0x80325050) + 0x189
  672. May 5 10:29:28 [rc]: mgr.stack.framename: Stack frame 5: maytag.ko::oscall_panic(0x82148070) + 0x11
  673. May 5 10:29:28 [rc]: mgr.stack.framename: Stack frame 6: maytag.ko::sk_vpanic(0x8226e46e) + 0x86
  674. May 5 10:29:28 [rc]: mgr.stack.framename: Stack frame 7: maytag.ko::sk_vpanic(0x8226e46e) + 0x0
  675. May 5 10:29:28 [rc]: mgr.stack.framename: Stack frame 8: maytag.ko::wafl_junkd_aggroffline_handle_inconsistency_cb(0x829f9ccc) + 0x0
  676. May 5 10:29:28 [rc]: mgr.stack.framename: Stack frame 9: maytag.ko::wafl_junkd_aggroffline_worker(0x829f9ce5) + 0x1c3
  677. May 5 10:29:28 [rc]: mgr.stack.framename: Stack frame 10: maytag.ko::sk_stack_top(0x8226e1b6) + 0x0
  678. May 5 10:29:28 [snmp_trap_main]: snmp.link.up: Interface 5 is up
  679. May 5 10:29:28 [sshd]: kern.syslog.msg: error: Could not load host key: /etc/sshd/ssh_host_key
  680. May 5 10:29:28 [sshd]: kern.syslog.msg: error: Could not load host key: /etc/sshd/ssh_host_rsa_key
  681. May 5 10:29:28 [sshd]: kern.syslog.msg: error: Could not load host key: /etc/sshd/ssh_host_dsa_key
  682. May 5 10:29:28 [ispfc_main]: ses.shelf.mmErr: SCSI Enclosure Services (SES) device accessed by 0b.30 (product revision level 3333) is not supported on this platform.
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement
Public Pastes
Advertisement
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!