toast1a# ems_logviewer messages.ems May 5 07:00:00 [mgmtgwd]: mgmtgwd.jobmgr.private.jobcomplete.success: Private job "log-rotation" [id 1] (logrotation job) completed successfully: . May 5 07:00:00 [statd]: kern.uptime.filer: 7:00am up 45 mins, 6420586 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 0 iSCSI ops May 5 07:00:00 [mgmtgwd]: mgmtgwd.jobmgr.jobstart: Started job "Licensing" [id 1]: License Checking. May 5 07:00:00 [statd]: monitor.shelf.fault: Fault reported on disk storage shelf attached to channel 0b. Please check fans, power, and temperature. May 5 07:00:00 [statd]: monitor.raiddp.vol.singleDegraded: data disk in RAID group /engdata1/plex0/rg2 is broken. May 5 07:00:00 [mgmtgwd]: license.check.ok May 5 07:00:00 [mgmtgwd]: mgmtgwd.jobmgr.jobcomplete.success: Job "Licensing" [id 1] (License Checking) completed successfully: Succeeded. May 5 07:00:01 [statd]: monitor.nht.count.exceeded May 5 07:00:01 [statd]: kern.syslog.msg: Cluster is licensed but takeover of partner is disabled May 5 07:02:26 [coredump_manager]: coredump.findcore.missing: Partial core missing disk 1 of 12 May 5 07:02:26 [coredump_manager]: coredump.findcore.partial: Partial core is missing 1 of 12 disks May 5 07:02:27 [coredump_manager]: coredump.findcore.missing: Partial core missing disk 1 of 12 May 5 07:02:27 [coredump_manager]: coredump.findcore.partial: Partial core is missing 1 of 12 disks May 5 07:02:28 [coredump_manager]: coredump.findcore.missing: Partial core missing disk 1 of 12 May 5 07:02:28 [coredump_manager]: coredump.findcore.partial: Partial core is missing 1 of 12 disks May 5 07:02:29 [coredump_manager]: coredump.findcore.missing: Partial core missing disk 1 of 12 May 5 07:02:29 [coredump_manager]: coredump.findcore.partial: Partial core is missing 1 of 12 disks May 5 07:02:29 [savecore_admin]: coredump.findcore.missing: Partial core missing disk 1 of 12 May 5 07:02:29 [savecore_admin]: coredump.findcore.partial: Partial core is missing 1 of 12 disks May 5 07:02:29 [savecore_admin]: coredump.save.started: Saving 4558M to core.101185225.2013-04-22.17_53_15.nz (\"Freeing free block 9009630 in aggregate engdata1.\") May 5 07:02:29 [savecore_admin]: coredump.save.nospace: Not enough space in root volume to save core May 5 07:02:29 [savecore_admin]: coredump.save.error: core.101185225.2013-04-22.17_53_15.nz processing encountered error May 5 07:02:29 [savecore_admin]: coredump.save.started: Saving 4474M to core.101185225.2013-04-22.19_04_41.nz (\"Freeing free block 9009630 in aggregate engdata1.\") May 5 07:02:29 [savecore_admin]: coredump.save.nospace: Not enough space in root volume to save core May 5 07:02:29 [savecore_admin]: coredump.save.error: core.101185225.2013-04-22.19_04_41.nz processing encountered error May 5 07:02:29 [savecore_admin]: coredump.save.started: Saving 4556M to core.101185225.2013-04-22.20_18_41.nz (\"Freeing free block 9009630 in aggregate engdata1.\") May 5 07:02:29 [savecore_admin]: coredump.save.nospace: Not enough space in root volume to save core May 5 07:02:29 [savecore_admin]: coredump.save.error: core.101185225.2013-04-22.20_18_41.nz processing encountered error May 5 07:02:29 [savecore_admin]: coredump.save.started: Saving 4556M to core.101185225.2013-04-22.21_31_46.nz (\"Freeing free block 9009630 in aggregate engdata1.\") May 5 07:02:29 [savecore_admin]: coredump.save.nospace: Not enough space in root volume to save core May 5 07:02:29 [savecore_admin]: coredump.save.error: core.101185225.2013-04-22.21_31_46.nz processing encountered error May 5 07:05:00 [mgmtgwd]: mgmtgwd.jobmgr.private.jobstart: Started private job "snap-hourly" [id 4]: Auto-Snapshot. May 5 07:07:24 [mgmtgwd]: mgmtgwd.jobmgr.private.jobcomplete.success: Private job "snap-hourly" [id 4] (Auto-Snapshot) completed successfully: . May 5 07:07:31 [coredump_manager]: coredump.findcore.missing: Partial core missing disk 1 of 12 May 5 07:07:31 [coredump_manager]: coredump.findcore.partial: Partial core is missing 1 of 12 disks May 5 07:07:34 [coredump_manager]: coredump.findcore.missing: Partial core missing disk 1 of 12 May 5 07:07:34 [coredump_manager]: coredump.findcore.partial: Partial core is missing 1 of 12 disks May 5 07:07:35 [coredump_manager]: coredump.findcore.missing: Partial core missing disk 1 of 12 May 5 07:07:35 [coredump_manager]: coredump.findcore.partial: Partial core is missing 1 of 12 disks May 5 07:07:38 [coredump_manager]: coredump.findcore.missing: Partial core missing disk 1 of 12 May 5 07:07:38 [coredump_manager]: coredump.findcore.partial: Partial core is missing 1 of 12 disks May 5 07:07:39 [savecore_admin]: coredump.findcore.missing: Partial core missing disk 1 of 12 May 5 07:07:39 [savecore_admin]: coredump.findcore.partial: Partial core is missing 1 of 12 disks May 5 07:07:39 [savecore_admin]: coredump.save.started: Saving 4558M to core.101185225.2013-04-22.17_53_15.nz (\"Freeing free block 9009630 in aggregate engdata1.\") May 5 07:07:39 [savecore_admin]: coredump.save.nospace: Not enough space in root volume to save core May 5 07:07:39 [savecore_admin]: coredump.save.error: core.101185225.2013-04-22.17_53_15.nz processing encountered error May 5 07:07:39 [savecore_admin]: coredump.save.started: Saving 4474M to core.101185225.2013-04-22.19_04_41.nz (\"Freeing free block 9009630 in aggregate engdata1.\") May 5 07:07:39 [savecore_admin]: coredump.save.nospace: Not enough space in root volume to save core May 5 07:07:39 [savecore_admin]: coredump.save.error: core.101185225.2013-04-22.19_04_41.nz processing encountered error May 5 07:07:39 [savecore_admin]: coredump.save.started: Saving 4556M to core.101185225.2013-04-22.20_18_41.nz (\"Freeing free block 9009630 in aggregate engdata1.\") May 5 07:07:39 [savecore_admin]: coredump.save.nospace: Not enough space in root volume to save core May 5 07:07:39 [savecore_admin]: coredump.save.error: core.101185225.2013-04-22.20_18_41.nz processing encountered error May 5 07:07:39 [savecore_admin]: coredump.save.started: Saving 4556M to core.101185225.2013-04-22.21_31_46.nz (\"Freeing free block 9009630 in aggregate engdata1.\") May 5 07:07:39 [savecore_admin]: coredump.save.nospace: Not enough space in root volume to save core May 5 07:07:39 [savecore_admin]: coredump.save.error: core.101185225.2013-04-22.21_31_46.nz processing encountered error May 5 07:07:45 [coredump_manager]: coredump.findcore.missing: Partial core missing disk 1 of 12 May 5 07:07:45 [coredump_manager]: coredump.findcore.partial: Partial core is missing 1 of 12 disks May 5 07:07:46 [coredump_manager]: coredump.findcore.missing: Partial core missing disk 1 of 12 May 5 07:07:46 [coredump_manager]: coredump.findcore.partial: Partial core is missing 1 of 12 disks May 5 07:07:48 [coredump_manager]: coredump.findcore.missing: Partial core missing disk 1 of 12 May 5 07:07:48 [coredump_manager]: coredump.findcore.partial: Partial core is missing 1 of 12 disks May 5 07:07:50 [coredump_manager]: coredump.findcore.missing: Partial core missing disk 1 of 12 May 5 07:07:50 [coredump_manager]: coredump.findcore.partial: Partial core is missing 1 of 12 disks May 5 07:07:50 [mgmtgwd]: mgmtgwd.jobmgr.private.jobcomplete.success: Private job "coredump-autosave" [id 8] () completed successfully: . May 5 07:07:54 [savecore_admin]: coredump.findcore.missing: Partial core missing disk 1 of 12 May 5 07:07:54 [savecore_admin]: coredump.findcore.partial: Partial core is missing 1 of 12 disks May 5 07:07:54 [savecore_admin]: coredump.save.started: Saving 4558M to core.101185225.2013-04-22.17_53_15.nz (\"Freeing free block 9009630 in aggregate engdata1.\") May 5 07:07:54 [savecore_admin]: coredump.save.nospace: Not enough space in root volume to save core May 5 07:07:54 [savecore_admin]: coredump.save.error: core.101185225.2013-04-22.17_53_15.nz processing encountered error May 5 07:07:54 [savecore_admin]: coredump.save.started: Saving 4474M to core.101185225.2013-04-22.19_04_41.nz (\"Freeing free block 9009630 in aggregate engdata1.\") May 5 07:07:54 [savecore_admin]: coredump.save.nospace: Not enough space in root volume to save core May 5 07:07:54 [savecore_admin]: coredump.save.error: core.101185225.2013-04-22.19_04_41.nz processing encountered error May 5 07:07:54 [savecore_admin]: coredump.save.started: Saving 4556M to core.101185225.2013-04-22.20_18_41.nz (\"Freeing free block 9009630 in aggregate engdata1.\") May 5 07:07:54 [savecore_admin]: coredump.save.nospace: Not enough space in root volume to save core May 5 07:07:54 [savecore_admin]: coredump.save.error: core.101185225.2013-04-22.20_18_41.nz processing encountered error May 5 07:07:54 [savecore_admin]: coredump.save.started: Saving 4556M to core.101185225.2013-04-22.21_31_46.nz (\"Freeing free block 9009630 in aggregate engdata1.\") May 5 07:07:54 [savecore_admin]: coredump.save.nospace: Not enough space in root volume to save core May 5 07:07:54 [savecore_admin]: coredump.save.error: core.101185225.2013-04-22.21_31_46.nz processing encountered error May 5 07:10:00 [mgmtgwd]: mgmtgwd.jobmgr.private.jobstart: Started private job "snap-daily" [id 5]: Auto-Snapshot. May 5 07:10:56 [mgmtgwd]: mgmtgwd.jobmgr.private.jobcomplete.success: Private job "snap-daily" [id 5] (Auto-Snapshot) completed successfully: . May 5 07:15:00 [mgmtgwd]: mgmtgwd.jobmgr.private.jobstart: Started private job "snap-weekly" [id 6]: Auto-Snapshot. May 5 07:15:28 [mgmtgwd]: mgmtgwd.jobmgr.private.jobcomplete.success: Private job "snap-weekly" [id 6] (Auto-Snapshot) completed successfully: . May 5 07:17:01 [registry_control]: kern.syslog.msg: registry: Unexpected error in rollback in regx_commit - Error: No such file or directory (name=options.ssh.access) Error: No such file or directory (name=options.ssh.debug.enable) Error: No such file or directory (name=options.ssh.idle.timeout) Error: No such file or directory (name=options.ssh.passwd_auth.ena May 5 07:17:01 [registry_control]: reg.transaction.commitFail: registry: Cannot commit transaction in 'Periodic config update'. Error: Invalid argument (name=options.ssh.access) (value=*) Error: Invalid argument (name=options.ssh.debug.enable) (value=off) Error: Invalid argument (name=options.ssh.idle.timeout) (value=600) Error: Invalid argument (name=options.ssh.passwd_auth.enabl May 5 07:17:52 [console_login_mgr]: kern.syslog.msg: root logged in from console May 5 07:19:31 [ispfc_main]: disk.ioMediumError: Medium error on disk 0c.51: op 0x28:2f591472:002d sector 794367096 SCSI:medium error - Unrecovered read error - RAID subsystem will use parity information to reconstruct unreadable data (813554c 82d95cc 82d9adc 82d9c6c) May 5 07:19:35 [raidio_thread]: raid.tetris.media.err: Read error on Disk /engdata1/plex0/rg0/0c.51 Shelf 3 Bay 3 [NETAPP X267_MGRIZ500SSX H3DD] S/N [H817GCGH], block #88263010 during stripe write May 5 07:19:35 [raidio_thread]: raid.rg.readerr.repair.data: Fixing bad data on Disk /engdata1/plex0/rg0/0c.51 Shelf 3 Bay 3 [NETAPP X267_MGRIZ500SSX H3DD] S/N [H817GCGH], block #88263010 May 5 07:19:36 [ispfc_main]: disk.ioReassignSuccess: disk 0c.51: sector 794367096 was reassigned. May 5 08:00:00 [statd]: kern.uptime.filer: 8:00am up 1:45 15410774 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 0 iSCSI ops May 5 08:00:00 [statd]: monitor.shelf.fault: Fault reported on disk storage shelf attached to channel 0b. Please check fans, power, and temperature. May 5 08:00:00 [statd]: monitor.raiddp.vol.singleDegraded: data disk in RAID group /engdata1/plex0/rg2 is broken. May 5 08:00:01 [statd]: kern.syslog.msg: Cluster is licensed but takeover of partner is disabled May 5 08:05:00 [mgmtgwd]: mgmtgwd.jobmgr.private.jobstart: Started private job "snap-hourly" [id 4]: Auto-Snapshot. May 5 08:05:34 [mgmtgwd]: mgmtgwd.jobmgr.private.jobcomplete.success: Private job "snap-hourly" [id 4] (Auto-Snapshot) completed successfully: . May 5 08:18:01 [registry_control]: kern.syslog.msg: registry: Unexpected error in rollback in regx_commit - Error: No such file or directory (name=options.ssh.access) Error: No such file or directory (name=options.ssh.debug.enable) Error: No such file or directory (name=options.ssh.idle.timeout) Error: No such file or directory (name=options.ssh.passwd_auth.ena May 5 08:18:01 [registry_control]: reg.transaction.commitFail: registry: Cannot commit transaction in 'Periodic config update'. Error: Invalid argument (name=options.ssh.access) (value=*) Error: Invalid argument (name=options.ssh.debug.enable) (value=off) Error: Invalid argument (name=options.ssh.idle.timeout) (value=600) Error: Invalid argument (name=options.ssh.passwd_auth.enabl May 5 08:32:00 [wafl_junkd_aggroffl]: wafl.buf.freeingFreeBlock: Freeing free block 926442036 in aggregate engdata1.\n WAFL inconsistent. Please contact Network Appliance Global Services. May 5 08:32:01 [wafl_junkd_aggroffl]: sk.panic: Panic String: Freeing free block 926442036 in aggregate engdata1. WAFL inconsistent. Please contact Network Appliance Global Services. in SK process wafl_junkd_aggroffl on release NetApp Release R10.0.1xN_070123_1900 May 5 08:32:01 [wafl_junkd_aggroffl]: coredump.findcore.missing: Partial core missing disk 1 of 12 May 5 08:32:01 [wafl_junkd_aggroffl]: coredump.findcore.partial: Partial core is missing 1 of 12 disks May 5 08:32:01 [wafl_junkd_aggroffl]: coredump.dump.find.all.used: All available disks contain unsaved cores May 5 08:32:01 [wafl_junkd_aggroffl]: coredump.dump.failed: Dumpcore failed May 5 08:33:07 [nic_mgr]: cf.nm.nicTransitionUp: Interconnect link 0 is UP May 5 08:33:15 [ispfc_timeout_3]: fci.device.quiesce: Adapter unknown encountered a command timeout on 0b device 0b.36 (0x03000024). Quiescing the device. May 5 08:33:20 [sanown_notify]: diskown.isEnabled: software ownership has been enabled for this system May 5 08:33:22 [nic_mgr]: cf.nm.nicReset: Initiating soft reset on Cluster Interconnect card 0 due to rendezvous reset May 5 08:33:22 [rendezvous_proc]: cf.rv.notConnected: Connection for cfo_rv failed May 5 08:33:22 [nic_mgr]: cf.nm.nicTransitionDown: Cluster Interconnect link 0 is DOWN May 5 08:33:22 [rendezvous_proc]: cf.rv.notConnected: Connection for cfo_rv failed May 5 08:33:24 [nic_mgr]: cf.nm.nicTransitionUp: Interconnect link 0 is UP May 5 08:33:24 [fmmbx_instanceWorke]: fmmb.current.lock.disk: Disk 0c.58 is a local HA mailbox disk. May 5 08:33:24 [fmmbx_instanceWorke]: fmmb.current.lock.disk: Disk 0a.29 is a local HA mailbox disk. May 5 08:33:24 [fmmbx_instanceWorke]: kern.syslog.msg: normal mailbox instance on local side May 5 08:33:24 [rc]: cf.fm.MBstatusOnBoot: status="1" May 5 08:33:41 [ses_admin]: ses.channel.rescanInitiated: Initiating rescan on channel 0b May 5 08:33:43 [config_thread]: raid.assim.disk.brokenPreAssim: Broken Disk 0c.60 Shelf ? Bay ? [NETAPP X267_MGRIZ500SSX H3DD] S/N [H814LZ6H] detected prior to assimilation. It should be removed. May 5 08:33:43 [config_thread]: raid.assim.disk.brokenPreAssim: Broken Disk 0c.53 Shelf ? Bay ? [NETAPP X267_MGRIZ500SSX H3DD] S/N [H8155VEH] detected prior to assimilation. It should be removed. May 5 08:33:43 [config_thread]: raid.assim.disk.brokenPreAssim: Broken Disk 0c.57 Shelf ? Bay ? [NETAPP X267_MGRIZ500SSX H3DD] S/N [H814R7RH] detected prior to assimilation. It should be removed. May 5 08:33:43 [config_thread]: raid.assim.disk.brokenPreAssim: Broken Disk 0b.41 Shelf ? Bay ? [NETAPP X267_MGRIZ500SSX H3DD] S/N [H817LZ8H] detected prior to assimilation. It should be removed. May 5 08:33:43 [config_thread]: raid.assim.disk.brokenPreAssim: Broken Disk 0b.44 Shelf ? Bay ? [NETAPP X267_MGRIZ500SSX H3DD] S/N [H817GR5H] detected prior to assimilation. It should be removed. May 5 08:33:43 [config_thread]: raid.assim.disk.brokenPreAssim: Broken Disk 0a.23 Shelf ? Bay ? [NETAPP X267_MGRIZ500SSX H3DD] S/N [H814LVPH] detected prior to assimilation. It should be removed. May 5 08:33:43 [config_thread]: raid.assim.cls.noStableCls: Unable to find a stable consistent label set for all trees. Resorting disks. May 5 08:33:43 [config_thread]: raid.assim.cls.noStableCls2: Unable to find a stable consistent label set for all trees with Label 2. Resorting disks. May 5 08:33:43 [fmmbx_instanceWorke]: fmmb.NewDiskSuggestion: count="4", new="0c.58, 0a.29, null, null", old="0c.58, 0a.29" May 5 08:33:43 [fmmbx_instanceWorke]: fmmb.current.lock.disk: Disk 0d.70 is a partner HA mailbox disk. May 5 08:33:43 [fmmbx_instanceWorke]: fmmb.current.lock.disk: Disk 0d.67 is a partner HA mailbox disk. May 5 08:33:43 [fmmbx_instanceWorke]: kern.syslog.msg: normal mailbox instance on partner side May 5 08:33:43 [rc]: cf.fm.partner: Cluster monitor: partner 'toast1b' May 5 08:33:43 [rc]: cf.fm.timeMasterStatus: Acting as cluster time master May 5 08:33:47 [config_thread]: raid.vol.replay.nvram: Performing raid replay on volume(s) May 5 08:33:47 [rc]: raid.cksum.replay.summary: Replayed 0 checksum blocks. May 5 08:33:47 [rc]: raid.stripe.replay.summary: Replayed 0 stripes. May 5 08:33:48 [rc]: wafl.inconsistent.vol: WAFL inconsistent: Aggregate engdata1 is inconsistent. Note: Any new Snapshot copies might contain this inconsistency. May 5 08:33:48 [rc]: wafl.vol.loading: Loading Aggregate engdata1 May 5 08:33:48 [rc]: wafl.vol.loading: Loading Aggregate aggr0 May 5 08:33:48 [vv_apply_worker16]: wafl.vol.loading: Loading Volume mroot@vserver:06866009-a1fe-11e0-8650-000e0caac15e May 5 08:33:48 [vv_apply_worker17]: wafl.vol.loading: Loading Volume engdata1_root@vserver:b1a2013d-acca-11e0-98c0-000e0caac15e May 5 08:33:48 [vv_apply_worker19]: wafl.vol.loading: Loading Volume bluecore@vserver:b1a2013d-acca-11e0-98c0-000e0caac15e May 5 08:33:48 [vv_apply_worker18]: wafl.vol.loading: Loading Volume engdata1@vserver:b1a2013d-acca-11e0-98c0-000e0caac15e May 5 08:33:49 [vv_apply_worker19]: wafl.maxdirsize.boot.notice: bluecore@vserver:b1a2013d-acca-11e0-98c0-000e0caac15e: This volume's maxdirsize (102400KB) is higher than the default (35573KB). There may be a performance penalty when doing operations on large directories. May 5 08:33:49 [rc]: cf.fm.launch: Launching cluster monitor May 5 08:33:49 [rc]: cf.fm.partner: Cluster monitor: partner 'toast1b' May 5 08:33:49 [rc]: cf.fm.notkoverClusterDisable: Cluster monitor: cluster takeover disabled (restart) May 5 08:33:49 [cf_main]: cf.fsm.takeoverOfPartnerDisabled: Cluster monitor: takeover of toast1b disabled (cluster takeover disabled) May 5 08:34:00 [rc]: license.db.migrate.vol.success: Successfully migrated licenses for volume engdata1 May 5 08:34:00 [rc]: license.db.migrate.vol.success: Successfully migrated licenses for volume aggr0 May 5 08:34:00 [rc]: license.db.migrate.success: Migration complete May 5 08:34:00 [fmmbx_instanceWorke]: fmmb.NewDiskSuggestion: count="4", new="0c.58, 0a.29, null, null", old="0c.58, 0a.29" May 5 08:34:00 [rc]: kern.syslog.msg: The system was down for 98 seconds May 5 08:34:00 [raidio_thread]: raid.disk.unload.done: Unload of Disk 0c.60 Shelf 3 Bay 12 [NETAPP X267_MGRIZ500SSX H3DD] S/N [H814LZ6H] has completed successfully May 5 08:34:00 [raidio_thread]: raid.disk.unload.done: Unload of Disk 0c.53 Shelf 3 Bay 5 [NETAPP X267_MGRIZ500SSX H3DD] S/N [H8155VEH] has completed successfully May 5 08:34:00 [raidio_thread]: raid.disk.unload.done: Unload of Disk 0c.57 Shelf 3 Bay 9 [NETAPP X267_MGRIZ500SSX H3DD] S/N [H814R7RH] has completed successfully May 5 08:34:00 [raidio_thread]: raid.disk.unload.done: Unload of Disk 0b.41 Shelf 2 Bay 9 [NETAPP X267_MGRIZ500SSX H3DD] S/N [H817LZ8H] has completed successfully May 5 08:34:00 [raidio_thread]: raid.disk.unload.done: Unload of Disk 0b.44 Shelf 2 Bay 12 [NETAPP X267_MGRIZ500SSX H3DD] S/N [H817GR5H] has completed successfully May 5 08:34:00 [raidio_thread]: raid.disk.unload.done: Unload of Disk 0a.23 Shelf 1 Bay 7 [NETAPP X267_MGRIZ500SSX H3DD] S/N [H814LVPH] has completed successfully May 5 08:34:00 [rc]: raid.fdr.failed.ok: Disk 0c.60 Shelf 3 Bay 12 [NETAPP X267_MGRIZ500SSX H3DD] S/N [H814LZ6H] successfully deleted from spare pool May 5 08:34:00 [rc]: raid.fdr.failed.ok: Disk 0c.53 Shelf 3 Bay 5 [NETAPP X267_MGRIZ500SSX H3DD] S/N [H8155VEH] successfully deleted from spare pool May 5 08:34:00 [rc]: raid.fdr.failed.ok: Disk 0c.57 Shelf 3 Bay 9 [NETAPP X267_MGRIZ500SSX H3DD] S/N [H814R7RH] successfully deleted from spare pool May 5 08:34:00 [rc]: raid.fdr.failed.ok: Disk 0b.41 Shelf 2 Bay 9 [NETAPP X267_MGRIZ500SSX H3DD] S/N [H817LZ8H] successfully deleted from spare pool May 5 08:34:00 [rc]: raid.fdr.failed.ok: Disk 0b.44 Shelf 2 Bay 12 [NETAPP X267_MGRIZ500SSX H3DD] S/N [H817GR5H] successfully deleted from spare pool May 5 08:34:00 [rc]: raid.fdr.failed.ok: Disk 0a.23 Shelf 1 Bay 7 [NETAPP X267_MGRIZ500SSX H3DD] S/N [H814LVPH] successfully deleted from spare pool May 5 08:34:00 [rc]: kern.syslog.msg: registry: Unexpected error in rollback in regx_commit - Error: No such file or directory (name=options.ssh.access) Error: No such file or directory (name=options.ssh.debug.enable) Error: No such file or directory (name=options.ssh.idle.timeout) Error: No such file or directory (name=options.ssh.passwd_auth.ena May 5 08:34:00 [rc]: reg.transaction.commitFail: registry: Cannot commit transaction in 'Periodic config update'. Error: Invalid argument (name=options.ssh.access) (value=*) Error: Invalid argument (name=options.ssh.debug.enable) (value=off) Error: Invalid argument (name=options.ssh.idle.timeout) (value=600) Error: Invalid argument (name=options.ssh.passwd_auth.enabl May 5 08:34:00 [rc]: kern.syslog.msg: sysconfig: validating configuration May 5 08:34:00 [config_thread]: raid.rg.recons.missing: RAID group /engdata1/plex0/rg2 is missing 1 disk(s). May 5 08:34:00 [config_thread]: raid.rg.recons.cantStart: The reconstruction cannot start in RAID group /engdata1/plex0/rg2: No 520 bps disk of required size available in spare pool May 5 08:34:00 [rc]: wafl.inconsistent.vol: WAFL inconsistent: Aggregate engdata1 is inconsistent. Note: Any new Snapshot copies might contain this inconsistency. May 5 08:34:00 [rc]: mgr.boot.disk_done: NetApp Release ontap boot complete. Last disk update written at Sun May 5 08:31:57 GMT 2013 May 5 08:34:00 [rc]: mgr.boot.reason_abnormal: System rebooted after a panic. May 5 08:34:00 [rc]: mgr.stack.module: Module kernel loaded from 0x80000000 - 0x100000000. May 5 08:34:00 [rc]: mgr.stack.module: Module common_kmod.ko loaded from 0x81a00000 - 0x81aa6000. May 5 08:34:00 [rc]: mgr.stack.module: Module nvram5.ko loaded from 0x81aa6000 - 0x81ab8000. May 5 08:34:00 [rc]: mgr.stack.module: Module nvram_mgr.ko loaded from 0x81ab8000 - 0x81abd000. May 5 08:34:00 [rc]: mgr.stack.module: Module nvr.ko loaded from 0x81abd000 - 0x81ad0000. May 5 08:34:00 [rc]: mgr.stack.module: Module maytag.ko loaded from 0x81ad0000 - 0x858db000. May 5 08:34:00 [rc]: mgr.stack.module: Module spinvfs.ko loaded from 0x858db000 - 0x85924000. May 5 08:34:00 [rc]: mgr.stack.module: Module nbladekmod.ko loaded from 0x85924000 - 0x876b8000. May 5 08:34:00 [rc]: mgr.stack.saved: Reboot with saved panic information in log file May 5 08:34:00 [rc]: mgr.stack.string: Panic string: Freeing free block 926442036 in aggregate engdata1. WAFL inconsistent. Please contact Network Appliance Global Services. in SK process wafl_junkd_aggroffl on r May 5 08:34:00 [rc]: mgr.stack.at: Panic occurred at: Sun May 5 08:32:01 2013 May 5 08:34:00 [rc]: mgr.stack.proc: Panic in process: wafl_junkd_aggroffl May 5 08:34:00 [rc]: mgr.stack.framename: Stack frame 0: maytag.ko::ontap_dumpcore(0x82146b00) + 0xb4 May 5 08:34:00 [rc]: mgr.stack.framename: Stack frame 1: kernel::dumpsys(0x80477260) + 0xf8 May 5 08:34:00 [rc]: mgr.stack.framename: Stack frame 2: kernel::shutdown_get_howto(0x80324460) + 0x5d May 5 08:34:00 [rc]: mgr.stack.framename: Stack frame 3: kernel::shutdown_get_howto(0x80324460) + 0x45d May 5 08:34:00 [rc]: mgr.stack.framename: Stack frame 4: kernel::panic(0x80325050) + 0x189 May 5 08:34:00 [rc]: mgr.stack.framename: Stack frame 5: maytag.ko::oscall_panic(0x82148070) + 0x11 May 5 08:34:00 [rc]: mgr.stack.framename: Stack frame 6: maytag.ko::sk_vpanic(0x8226e46e) + 0x86 May 5 08:34:00 [rc]: mgr.stack.framename: Stack frame 7: maytag.ko::sk_vpanic(0x8226e46e) + 0x0 May 5 08:34:00 [rc]: mgr.stack.framename: Stack frame 8: maytag.ko::wafl_junkd_aggroffline_handle_inconsistency_cb(0x829f9ccc) + 0x0 May 5 08:34:00 [rc]: mgr.stack.framename: Stack frame 9: maytag.ko::wafl_junkd_aggroffline_worker(0x829f9ce5) + 0x1c3 May 5 08:34:00 [rc]: mgr.stack.framename: Stack frame 10: maytag.ko::sk_stack_top(0x8226e1b6) + 0x0 May 5 08:34:00 [asup_main]: asup.throttle.drop: Too many autosupport messages in too short a time, throttling autosupport: WAFL INCONSISTENT May 5 08:34:00 [asup_main]: asup.throttle.drop: Too many autosupport messages in too short a time, throttling autosupport: WAFL INCONSISTENT May 5 08:34:00 [asup_main]: asup.throttle.drop: Too many autosupport messages in too short a time, throttling autosupport: REBOOT (panic) May 5 08:34:00 [snmp_trap_main]: snmp.link.up: Interface 5 is up May 5 08:34:00 [sshd]: kern.syslog.msg: error: Could not load host key: /etc/sshd/ssh_host_key May 5 08:34:00 [sshd]: kern.syslog.msg: error: Could not load host key: /etc/sshd/ssh_host_rsa_key May 5 08:34:00 [sshd]: kern.syslog.msg: error: Could not load host key: /etc/sshd/ssh_host_dsa_key May 5 08:34:01 [coredump_manager]: coredump.findcore.missing: Partial core missing disk 1 of 12 May 5 08:34:01 [coredump_manager]: coredump.findcore.partial: Partial core is missing 1 of 12 disks May 5 08:34:01 [coredump_manager]: coredump.all.disks.used: No disks available for dumpcore May 5 08:34:01 [statd]: monitor.raid.brokenDisk: data disk in RAID group /engdata1/plex0/rg2 is broken. May 5 08:34:01 [statd]: raid.vol.state.online: vol_type="Volume", vol="aggr0" May 5 08:34:01 [statd]: raid.vol.state.online: vol_type="Volume", vol="engdata1" May 5 08:34:01 [10/100/1000-VI/e0c]: kern.syslog.msg: Ethernet e0c: Link up May 5 08:34:01 [10/100/1000-VI/e0d]: kern.syslog.msg: Ethernet e0d: Link up May 5 08:34:01 [10/100/1000-VI/e0a]: kern.syslog.msg: Ethernet e0a: Link up May 5 08:34:01 [10/100/1000-VI/e0b]: kern.syslog.msg: Ethernet e0b: Link up May 5 08:34:01 [config_failed_disk]: raid.fdr.reminder: Failed Disk 0c.60 Shelf 3 Bay 12 [NETAPP X267_MGRIZ500SSX H3DD] S/N [H814LZ6H] is still present in the system and should be removed. May 5 08:34:01 [config_failed_disk]: raid.fdr.reminder: Failed Disk 0c.53 Shelf 3 Bay 5 [NETAPP X267_MGRIZ500SSX H3DD] S/N [H8155VEH] is still present in the system and should be removed. May 5 08:34:01 [config_failed_disk]: raid.fdr.reminder: Failed Disk 0c.57 Shelf 3 Bay 9 [NETAPP X267_MGRIZ500SSX H3DD] S/N [H814R7RH] is still present in the system and should be removed. May 5 08:34:01 [config_failed_disk]: raid.fdr.reminder: Failed Disk 0b.41 Shelf 2 Bay 9 [NETAPP X267_MGRIZ500SSX H3DD] S/N [H817LZ8H] is still present in the system and should be removed. May 5 08:34:01 [config_failed_disk]: raid.fdr.reminder: Failed Disk 0b.44 Shelf 2 Bay 12 [NETAPP X267_MGRIZ500SSX H3DD] S/N [H817GR5H] is still present in the system and should be removed. May 5 08:34:01 [config_failed_disk]: raid.fdr.reminder: Failed Disk 0a.23 Shelf 1 Bay 7 [NETAPP X267_MGRIZ500SSX H3DD] S/N [H814LVPH] is still present in the system and should be removed. May 5 08:34:01 [10/100/1000-VI/e0a]: kern.syslog.msg: Ethernet e0a: Link being reconfigured May 5 08:34:01 [config_thread]: raid.rg.spares.low: /engdata1/plex0/rg0 May 5 08:34:01 [config_thread]: raid.rg.spares.low: /engdata1/plex0/rg1 May 5 08:34:01 [config_thread]: raid.rg.spares.low: /engdata1/plex0/rg2 May 5 08:34:01 [config_thread]: raid.rg.spares.low: /aggr0/plex0/rg0 May 5 08:34:01 [asup_main]: asup.throttle.drop: Too many autosupport messages in too short a time, throttling autosupport: SPARES_LOW May 5 08:34:01 [console_login_mgr]: kern.syslog.msg: root logged in from console May 5 08:34:04 [ses_admin]: ses.drive.shelfAddr.mm: Mismatch between adapter and enclosure services information for device 0b.36 (shelf 2 bay 4). May 5 08:34:05 [dyn_dev_qual_admin]: dfu.firmwareUpToDate: Firmware is up-to-date on all disk drives May 5 08:34:05 [statd]: monitor.shelf.accessError: Enclosure services has detected an error in access to shelves on channel 0b. May 5 08:34:06 [asup_main]: callhome.encl.access: Call home for ENCLOSURE SERVICES ACCESS ERROR May 5 08:34:06 [10/100/1000-VI/e0a]: kern.syslog.msg: Ethernet e0a: Link up May 5 08:34:07 [10/100/1000-VI/e0b]: kern.syslog.msg: Ethernet e0b: Link being reconfigured May 5 08:34:12 [10/100/1000-VI/e0b]: kern.syslog.msg: Ethernet e0b: Link up May 5 08:34:17 [ses_admin]: ses.status.temperatureWarning: DS14-Mk2-AT shelf 2 on channel 0b temperature warning for Temperature sensor 2: not installed or failed. Current temperature: C ( F). This module is on the rear side of the shelf, at the top center, on shelf module A. May 5 08:34:17 [ses_admin]: ses.status.electronicsError: DS14-Mk2-AT shelf 2 on channel 0b environmental monitoring error for SES electronics 1: unrecoverable error ; enclosure services hardware failed This module is on the rear side of the shelf, at the top center. May 5 08:34:18 [statd]: monitor.shelf.fault: Fault reported on disk storage shelf attached to channel 0b. Please check fans, power, and temperature. May 5 08:34:18 [asup_main]: callhome.shlf.fault: Call home for SHELF_FAULT May 5 08:34:35 [ses_admin]: ses.status.temperatureWarning: DS14-Mk2-AT shelf 2 on channel 0b temperature warning for Temperature sensor 1: internal communication error. Current temperature: C ( F). This module is on the front side of the shelf, at the right, on the OPS panel. May 5 08:34:39 [mgmtgwd]: mgmtgwd.jobmgr.private.jobstart: Started private job "coredump-autosave" [id 8]: . May 5 08:34:40 [coredump_manager]: coredump.findcore.missing: Partial core missing disk 1 of 12 May 5 08:34:40 [coredump_manager]: coredump.findcore.partial: Partial core is missing 1 of 12 disks May 5 08:34:41 [coredump_manager]: coredump.findcore.missing: Partial core missing disk 1 of 12 May 5 08:34:41 [coredump_manager]: coredump.findcore.partial: Partial core is missing 1 of 12 disks May 5 08:34:42 [coredump_manager]: coredump.findcore.missing: Partial core missing disk 1 of 12 May 5 08:34:42 [coredump_manager]: coredump.findcore.partial: Partial core is missing 1 of 12 disks May 5 08:34:42 [mgmtgwd]: mgmtgwd.jobmgr.private.jobstart: Started private job "sync task" [id 7]: sync task. May 5 08:34:43 [mgmtgwd]: mgmtgwd.jobmgr.private.jobcomplete.success: Private job "sync task" [id 7] (sync task) completed successfully: . May 5 08:34:43 [coredump_manager]: coredump.findcore.missing: Partial core missing disk 1 of 12 May 5 08:34:43 [coredump_manager]: coredump.findcore.partial: Partial core is missing 1 of 12 disks May 5 08:34:44 [savecore_admin]: coredump.findcore.missing: Partial core missing disk 1 of 12 May 5 08:34:44 [savecore_admin]: coredump.findcore.partial: Partial core is missing 1 of 12 disks May 5 08:34:44 [savecore_admin]: coredump.save.started: Saving 4558M to core.101185225.2013-04-22.17_53_15.nz (\"Freeing free block 9009630 in aggregate engdata1.\") May 5 08:34:44 [savecore_admin]: coredump.save.nospace: Not enough space in root volume to save core May 5 08:34:44 [savecore_admin]: coredump.save.error: core.101185225.2013-04-22.17_53_15.nz processing encountered error May 5 08:34:44 [savecore_admin]: coredump.save.started: Saving 4474M to core.101185225.2013-04-22.19_04_41.nz (\"Freeing free block 9009630 in aggregate engdata1.\") May 5 08:34:44 [savecore_admin]: coredump.save.nospace: Not enough space in root volume to save core May 5 08:34:44 [savecore_admin]: coredump.save.error: core.101185225.2013-04-22.19_04_41.nz processing encountered error May 5 08:34:44 [savecore_admin]: coredump.save.started: Saving 4556M to core.101185225.2013-04-22.20_18_41.nz (\"Freeing free block 9009630 in aggregate engdata1.\") May 5 08:34:44 [savecore_admin]: coredump.save.nospace: Not enough space in root volume to save core May 5 08:34:44 [savecore_admin]: coredump.save.error: core.101185225.2013-04-22.20_18_41.nz processing encountered error May 5 08:34:44 [savecore_admin]: coredump.save.started: Saving 4556M to core.101185225.2013-04-22.21_31_46.nz (\"Freeing free block 9009630 in aggregate engdata1.\") May 5 08:34:44 [savecore_admin]: coredump.save.nospace: Not enough space in root volume to save core May 5 08:34:44 [savecore_admin]: coredump.save.error: core.101185225.2013-04-22.21_31_46.nz processing encountered error May 5 08:34:56 [vifMgr]: vifmgr.startedsuccessfully: The Virtual Interface Manager (VIFMgr) started up successfully. May 5 08:35:02 [config_thread]: raid.rg.media_scrub.start: owner="", rg="/aggr0/plex0/rg0" May 5 08:35:02 [monitor]: monitor.globalStatus.critical: Cluster failover of toast1b is not possible: cluster takeover disabled. Disk on adapter 0c, shelf 3, bay 12, failed. Disk on adapter 0c, shelf 3, bay 5, failed. Disk on adapter 0c, shelf 3, bay 9, failed. Disk on adapter 0b, shelf 2, bay 9, failed. Disk on adapter 0b, shelf 2, bay 12, failed. Disk on adapter 0a, shelf 1, bay There are not enough spare disks. Disk shelf fault. May 5 08:35:08 [vifMgr]: vifmgr.vifsuccessfullymoved: VIF clus1 (on virtual server 0), ID 1024, is now hosted on node toast1a, port e0a. May 5 08:35:09 [vifMgr]: vifmgr.vifsuccessfullymoved: VIF clus2 (on virtual server 0), ID 1023, is now hosted on node toast1a, port e0b. May 5 08:35:13 [mgmtgwd]: mgmtgwd.jobmgr.private.jobstart: Started private job "sync task" [id 7]: sync task. May 5 08:35:13 [mgmtgwd]: mgmtgwd.jobmgr.private.jobcomplete.success: Private job "sync task" [id 7] (sync task) completed successfully: . May 5 08:39:43 [coredump_manager]: coredump.findcore.missing: Partial core missing disk 1 of 12 May 5 08:39:43 [coredump_manager]: coredump.findcore.partial: Partial core is missing 1 of 12 disks May 5 08:39:44 [coredump_manager]: coredump.findcore.missing: Partial core missing disk 1 of 12 May 5 08:39:44 [coredump_manager]: coredump.findcore.partial: Partial core is missing 1 of 12 disks May 5 08:39:45 [coredump_manager]: coredump.findcore.missing: Partial core missing disk 1 of 12 May 5 08:39:45 [coredump_manager]: coredump.findcore.partial: Partial core is missing 1 of 12 disks May 5 08:39:46 [coredump_manager]: coredump.findcore.missing: Partial core missing disk 1 of 12 May 5 08:39:46 [coredump_manager]: coredump.findcore.partial: Partial core is missing 1 of 12 disks May 5 08:39:47 [savecore_admin]: coredump.findcore.missing: Partial core missing disk 1 of 12 May 5 08:39:47 [savecore_admin]: coredump.findcore.partial: Partial core is missing 1 of 12 disks May 5 08:39:47 [savecore_admin]: coredump.save.started: Saving 4558M to core.101185225.2013-04-22.17_53_15.nz (\"Freeing free block 9009630 in aggregate engdata1.\") May 5 08:39:47 [savecore_admin]: coredump.save.nospace: Not enough space in root volume to save core May 5 08:39:47 [savecore_admin]: coredump.save.error: core.101185225.2013-04-22.17_53_15.nz processing encountered error May 5 08:39:47 [savecore_admin]: coredump.save.started: Saving 4474M to core.101185225.2013-04-22.19_04_41.nz (\"Freeing free block 9009630 in aggregate engdata1.\") May 5 08:39:47 [savecore_admin]: coredump.save.nospace: Not enough space in root volume to save core May 5 08:39:47 [savecore_admin]: coredump.save.error: core.101185225.2013-04-22.19_04_41.nz processing encountered error May 5 08:39:47 [savecore_admin]: coredump.save.started: Saving 4556M to core.101185225.2013-04-22.20_18_41.nz (\"Freeing free block 9009630 in aggregate engdata1.\") May 5 08:39:47 [savecore_admin]: coredump.save.nospace: Not enough space in root volume to save core May 5 08:39:47 [savecore_admin]: coredump.save.error: core.101185225.2013-04-22.20_18_41.nz processing encountered error May 5 08:39:47 [savecore_admin]: coredump.save.started: Saving 4556M to core.101185225.2013-04-22.21_31_46.nz (\"Freeing free block 9009630 in aggregate engdata1.\") May 5 08:39:47 [savecore_admin]: coredump.save.nospace: Not enough space in root volume to save core May 5 08:39:47 [savecore_admin]: coredump.save.error: core.101185225.2013-04-22.21_31_46.nz processing encountered error May 5 08:44:46 [coredump_manager]: coredump.findcore.missing: Partial core missing disk 1 of 12 May 5 08:44:46 [coredump_manager]: coredump.findcore.partial: Partial core is missing 1 of 12 disks May 5 08:44:47 [coredump_manager]: coredump.findcore.missing: Partial core missing disk 1 of 12 May 5 08:44:47 [coredump_manager]: coredump.findcore.partial: Partial core is missing 1 of 12 disks May 5 08:44:48 [coredump_manager]: coredump.findcore.missing: Partial core missing disk 1 of 12 May 5 08:44:48 [coredump_manager]: coredump.findcore.partial: Partial core is missing 1 of 12 disks May 5 08:44:49 [coredump_manager]: coredump.findcore.missing: Partial core missing disk 1 of 12 May 5 08:44:49 [coredump_manager]: coredump.findcore.partial: Partial core is missing 1 of 12 disks May 5 08:44:50 [savecore_admin]: coredump.findcore.missing: Partial core missing disk 1 of 12 May 5 08:44:50 [savecore_admin]: coredump.findcore.partial: Partial core is missing 1 of 12 disks May 5 08:44:50 [savecore_admin]: coredump.save.started: Saving 4558M to core.101185225.2013-04-22.17_53_15.nz (\"Freeing free block 9009630 in aggregate engdata1.\") May 5 08:44:50 [savecore_admin]: coredump.save.nospace: Not enough space in root volume to save core May 5 08:44:50 [savecore_admin]: coredump.save.error: core.101185225.2013-04-22.17_53_15.nz processing encountered error May 5 08:44:50 [savecore_admin]: coredump.save.started: Saving 4474M to core.101185225.2013-04-22.19_04_41.nz (\"Freeing free block 9009630 in aggregate engdata1.\") May 5 08:44:50 [savecore_admin]: coredump.save.nospace: Not enough space in root volume to save core May 5 08:44:50 [savecore_admin]: coredump.save.error: core.101185225.2013-04-22.19_04_41.nz processing encountered error May 5 08:44:50 [savecore_admin]: coredump.save.started: Saving 4556M to core.101185225.2013-04-22.20_18_41.nz (\"Freeing free block 9009630 in aggregate engdata1.\") May 5 08:44:50 [savecore_admin]: coredump.save.nospace: Not enough space in root volume to save core May 5 08:44:50 [savecore_admin]: coredump.save.error: core.101185225.2013-04-22.20_18_41.nz processing encountered error May 5 08:44:50 [savecore_admin]: coredump.save.started: Saving 4556M to core.101185225.2013-04-22.21_31_46.nz (\"Freeing free block 9009630 in aggregate engdata1.\") May 5 08:44:50 [savecore_admin]: coredump.save.nospace: Not enough space in root volume to save core May 5 08:44:50 [savecore_admin]: coredump.save.error: core.101185225.2013-04-22.21_31_46.nz processing encountered error May 5 08:49:49 [coredump_manager]: coredump.findcore.missing: Partial core missing disk 1 of 12 May 5 08:49:49 [coredump_manager]: coredump.findcore.partial: Partial core is missing 1 of 12 disks May 5 08:49:50 [coredump_manager]: coredump.findcore.missing: Partial core missing disk 1 of 12 May 5 08:49:50 [coredump_manager]: coredump.findcore.partial: Partial core is missing 1 of 12 disks May 5 08:49:51 [coredump_manager]: coredump.findcore.missing: Partial core missing disk 1 of 12 May 5 08:49:51 [coredump_manager]: coredump.findcore.partial: Partial core is missing 1 of 12 disks May 5 08:49:52 [coredump_manager]: coredump.findcore.missing: Partial core missing disk 1 of 12 May 5 08:49:52 [coredump_manager]: coredump.findcore.partial: Partial core is missing 1 of 12 disks May 5 08:49:53 [savecore_admin]: coredump.findcore.missing: Partial core missing disk 1 of 12 May 5 08:49:53 [savecore_admin]: coredump.findcore.partial: Partial core is missing 1 of 12 disks May 5 08:49:53 [savecore_admin]: coredump.save.started: Saving 4558M to core.101185225.2013-04-22.17_53_15.nz (\"Freeing free block 9009630 in aggregate engdata1.\") May 5 08:49:53 [savecore_admin]: coredump.save.nospace: Not enough space in root volume to save core May 5 08:49:53 [savecore_admin]: coredump.save.error: core.101185225.2013-04-22.17_53_15.nz processing encountered error May 5 08:49:53 [savecore_admin]: coredump.save.started: Saving 4474M to core.101185225.2013-04-22.19_04_41.nz (\"Freeing free block 9009630 in aggregate engdata1.\") May 5 08:49:53 [savecore_admin]: coredump.save.nospace: Not enough space in root volume to save core May 5 08:49:53 [savecore_admin]: coredump.save.error: core.101185225.2013-04-22.19_04_41.nz processing encountered error May 5 08:49:53 [savecore_admin]: coredump.save.started: Saving 4556M to core.101185225.2013-04-22.20_18_41.nz (\"Freeing free block 9009630 in aggregate engdata1.\") May 5 08:49:53 [savecore_admin]: coredump.save.nospace: Not enough space in root volume to save core May 5 08:49:53 [savecore_admin]: coredump.save.error: core.101185225.2013-04-22.20_18_41.nz processing encountered error May 5 08:49:53 [savecore_admin]: coredump.save.started: Saving 4556M to core.101185225.2013-04-22.21_31_46.nz (\"Freeing free block 9009630 in aggregate engdata1.\") May 5 08:49:53 [savecore_admin]: coredump.save.nospace: Not enough space in root volume to save core May 5 08:49:53 [savecore_admin]: coredump.save.error: core.101185225.2013-04-22.21_31_46.nz processing encountered error May 5 08:54:52 [coredump_manager]: coredump.findcore.missing: Partial core missing disk 1 of 12 May 5 08:54:52 [coredump_manager]: coredump.findcore.partial: Partial core is missing 1 of 12 disks May 5 08:54:53 [coredump_manager]: coredump.findcore.missing: Partial core missing disk 1 of 12 May 5 08:54:53 [coredump_manager]: coredump.findcore.partial: Partial core is missing 1 of 12 disks May 5 08:54:54 [coredump_manager]: coredump.findcore.missing: Partial core missing disk 1 of 12 May 5 08:54:54 [coredump_manager]: coredump.findcore.partial: Partial core is missing 1 of 12 disks May 5 08:54:55 [coredump_manager]: coredump.findcore.missing: Partial core missing disk 1 of 12 May 5 08:54:55 [coredump_manager]: coredump.findcore.partial: Partial core is missing 1 of 12 disks May 5 08:54:56 [savecore_admin]: coredump.findcore.missing: Partial core missing disk 1 of 12 May 5 08:54:56 [savecore_admin]: coredump.findcore.partial: Partial core is missing 1 of 12 disks May 5 08:54:56 [savecore_admin]: coredump.save.started: Saving 4558M to core.101185225.2013-04-22.17_53_15.nz (\"Freeing free block 9009630 in aggregate engdata1.\") May 5 08:54:56 [savecore_admin]: coredump.save.nospace: Not enough space in root volume to save core May 5 08:54:56 [savecore_admin]: coredump.save.error: core.101185225.2013-04-22.17_53_15.nz processing encountered error May 5 08:54:56 [savecore_admin]: coredump.save.started: Saving 4474M to core.101185225.2013-04-22.19_04_41.nz (\"Freeing free block 9009630 in aggregate engdata1.\") May 5 08:54:56 [savecore_admin]: coredump.save.nospace: Not enough space in root volume to save core May 5 08:54:56 [savecore_admin]: coredump.save.error: core.101185225.2013-04-22.19_04_41.nz processing encountered error May 5 08:54:56 [savecore_admin]: coredump.save.started: Saving 4556M to core.101185225.2013-04-22.20_18_41.nz (\"Freeing free block 9009630 in aggregate engdata1.\") May 5 08:54:56 [savecore_admin]: coredump.save.nospace: Not enough space in root volume to save core May 5 08:54:56 [savecore_admin]: coredump.save.error: core.101185225.2013-04-22.20_18_41.nz processing encountered error May 5 08:54:56 [savecore_admin]: coredump.save.started: Saving 4556M to core.101185225.2013-04-22.21_31_46.nz (\"Freeing free block 9009630 in aggregate engdata1.\") May 5 08:54:56 [savecore_admin]: coredump.save.nospace: Not enough space in root volume to save core May 5 08:54:56 [savecore_admin]: coredump.save.error: core.101185225.2013-04-22.21_31_46.nz processing encountered error May 5 08:59:55 [coredump_manager]: coredump.findcore.missing: Partial core missing disk 1 of 12 May 5 08:59:55 [coredump_manager]: coredump.findcore.partial: Partial core is missing 1 of 12 disks May 5 08:59:56 [coredump_manager]: coredump.findcore.missing: Partial core missing disk 1 of 12 May 5 08:59:56 [coredump_manager]: coredump.findcore.partial: Partial core is missing 1 of 12 disks May 5 08:59:57 [coredump_manager]: coredump.findcore.missing: Partial core missing disk 1 of 12 May 5 08:59:57 [coredump_manager]: coredump.findcore.partial: Partial core is missing 1 of 12 disks May 5 08:59:58 [coredump_manager]: coredump.findcore.missing: Partial core missing disk 1 of 12 May 5 08:59:58 [coredump_manager]: coredump.findcore.partial: Partial core is missing 1 of 12 disks May 5 08:59:58 [savecore_admin]: coredump.findcore.missing: Partial core missing disk 1 of 12 May 5 08:59:58 [savecore_admin]: coredump.findcore.partial: Partial core is missing 1 of 12 disks May 5 08:59:58 [savecore_admin]: coredump.save.started: Saving 4558M to core.101185225.2013-04-22.17_53_15.nz (\"Freeing free block 9009630 in aggregate engdata1.\") May 5 08:59:58 [savecore_admin]: coredump.save.nospace: Not enough space in root volume to save core May 5 08:59:58 [savecore_admin]: coredump.save.error: core.101185225.2013-04-22.17_53_15.nz processing encountered error May 5 08:59:58 [savecore_admin]: coredump.save.started: Saving 4474M to core.101185225.2013-04-22.19_04_41.nz (\"Freeing free block 9009630 in aggregate engdata1.\") May 5 08:59:58 [savecore_admin]: coredump.save.nospace: Not enough space in root volume to save core May 5 08:59:58 [savecore_admin]: coredump.save.error: core.101185225.2013-04-22.19_04_41.nz processing encountered error May 5 08:59:58 [savecore_admin]: coredump.save.started: Saving 4556M to core.101185225.2013-04-22.20_18_41.nz (\"Freeing free block 9009630 in aggregate engdata1.\") May 5 08:59:58 [savecore_admin]: coredump.save.nospace: Not enough space in root volume to save core May 5 08:59:58 [savecore_admin]: coredump.save.error: core.101185225.2013-04-22.20_18_41.nz processing encountered error May 5 08:59:58 [savecore_admin]: coredump.save.started: Saving 4556M to core.101185225.2013-04-22.21_31_46.nz (\"Freeing free block 9009630 in aggregate engdata1.\") May 5 08:59:58 [savecore_admin]: coredump.save.nospace: Not enough space in root volume to save core May 5 08:59:58 [savecore_admin]: coredump.save.error: core.101185225.2013-04-22.21_31_46.nz processing encountered error May 5 09:00:00 [statd]: kern.uptime.filer: 9:00am up 26 mins, 2994815 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 0 iSCSI ops May 5 09:00:00 [statd]: monitor.shelf.fault: Fault reported on disk storage shelf attached to channel 0b. Please check fans, power, and temperature. May 5 09:00:00 [statd]: monitor.raiddp.vol.singleDegraded: data disk in RAID group /engdata1/plex0/rg2 is broken. May 5 09:00:01 [asup_main]: asup.throttle.drop: Too many autosupport messages in too short a time, throttling autosupport: DRIVE NHT DATA May 5 09:00:01 [statd]: monitor.nht.count.exceeded May 5 09:00:01 [statd]: kern.syslog.msg: Cluster is licensed but takeover of partner is disabled May 5 09:04:58 [coredump_manager]: coredump.findcore.missing: Partial core missing disk 1 of 12 May 5 09:04:58 [coredump_manager]: coredump.findcore.partial: Partial core is missing 1 of 12 disks May 5 09:04:59 [coredump_manager]: coredump.findcore.missing: Partial core missing disk 1 of 12 May 5 09:04:59 [coredump_manager]: coredump.findcore.partial: Partial core is missing 1 of 12 disks May 5 09:05:00 [mgmtgwd]: mgmtgwd.jobmgr.private.jobstart: Started private job "snap-hourly" [id 4]: Auto-Snapshot. May 5 09:05:00 [coredump_manager]: coredump.findcore.missing: Partial core missing disk 1 of 12 May 5 09:05:00 [coredump_manager]: coredump.findcore.partial: Partial core is missing 1 of 12 disks May 5 09:05:01 [coredump_manager]: coredump.findcore.missing: Partial core missing disk 1 of 12 May 5 09:05:01 [coredump_manager]: coredump.findcore.partial: Partial core is missing 1 of 12 disks May 5 09:05:01 [savecore_admin]: coredump.findcore.missing: Partial core missing disk 1 of 12 May 5 09:05:01 [savecore_admin]: coredump.findcore.partial: Partial core is missing 1 of 12 disks May 5 09:05:01 [savecore_admin]: coredump.save.started: Saving 4558M to core.101185225.2013-04-22.17_53_15.nz (\"Freeing free block 9009630 in aggregate engdata1.\") May 5 09:05:01 [savecore_admin]: coredump.save.nospace: Not enough space in root volume to save core May 5 09:05:01 [savecore_admin]: coredump.save.error: core.101185225.2013-04-22.17_53_15.nz processing encountered error May 5 09:05:01 [savecore_admin]: coredump.save.started: Saving 4474M to core.101185225.2013-04-22.19_04_41.nz (\"Freeing free block 9009630 in aggregate engdata1.\") May 5 09:05:01 [savecore_admin]: coredump.save.nospace: Not enough space in root volume to save core May 5 09:05:01 [savecore_admin]: coredump.save.error: core.101185225.2013-04-22.19_04_41.nz processing encountered error May 5 09:05:01 [savecore_admin]: coredump.save.started: Saving 4556M to core.101185225.2013-04-22.20_18_41.nz (\"Freeing free block 9009630 in aggregate engdata1.\") May 5 09:05:01 [savecore_admin]: coredump.save.nospace: Not enough space in root volume to save core May 5 09:05:01 [savecore_admin]: coredump.save.error: core.101185225.2013-04-22.20_18_41.nz processing encountered error May 5 09:05:01 [savecore_admin]: coredump.save.started: Saving 4556M to core.101185225.2013-04-22.21_31_46.nz (\"Freeing free block 9009630 in aggregate engdata1.\") May 5 09:05:01 [savecore_admin]: coredump.save.nospace: Not enough space in root volume to save core May 5 09:05:01 [savecore_admin]: coredump.save.error: core.101185225.2013-04-22.21_31_46.nz processing encountered error May 5 09:05:39 [mgmtgwd]: mgmtgwd.jobmgr.private.jobcomplete.success: Private job "snap-hourly" [id 4] (Auto-Snapshot) completed successfully: . May 5 09:10:03 [coredump_manager]: coredump.findcore.missing: Partial core missing disk 1 of 12 May 5 09:10:03 [coredump_manager]: coredump.findcore.partial: Partial core is missing 1 of 12 disks May 5 09:10:04 [coredump_manager]: coredump.findcore.missing: Partial core missing disk 1 of 12 May 5 09:10:04 [coredump_manager]: coredump.findcore.partial: Partial core is missing 1 of 12 disks May 5 09:10:08 [coredump_manager]: coredump.findcore.missing: Partial core missing disk 1 of 12 May 5 09:10:08 [coredump_manager]: coredump.findcore.partial: Partial core is missing 1 of 12 disks May 5 09:10:10 [coredump_manager]: coredump.findcore.missing: Partial core missing disk 1 of 12 May 5 09:10:10 [coredump_manager]: coredump.findcore.partial: Partial core is missing 1 of 12 disks May 5 09:10:11 [savecore_admin]: coredump.findcore.missing: Partial core missing disk 1 of 12 May 5 09:10:11 [savecore_admin]: coredump.findcore.partial: Partial core is missing 1 of 12 disks May 5 09:10:11 [savecore_admin]: coredump.save.started: Saving 4558M to core.101185225.2013-04-22.17_53_15.nz (\"Freeing free block 9009630 in aggregate engdata1.\") May 5 09:10:11 [savecore_admin]: coredump.save.nospace: Not enough space in root volume to save core May 5 09:10:11 [savecore_admin]: coredump.save.error: core.101185225.2013-04-22.17_53_15.nz processing encountered error May 5 09:10:11 [savecore_admin]: coredump.save.started: Saving 4474M to core.101185225.2013-04-22.19_04_41.nz (\"Freeing free block 9009630 in aggregate engdata1.\") May 5 09:10:11 [savecore_admin]: coredump.save.nospace: Not enough space in root volume to save core May 5 09:10:11 [savecore_admin]: coredump.save.error: core.101185225.2013-04-22.19_04_41.nz processing encountered error May 5 09:10:11 [savecore_admin]: coredump.save.started: Saving 4556M to core.101185225.2013-04-22.20_18_41.nz (\"Freeing free block 9009630 in aggregate engdata1.\") May 5 09:10:11 [savecore_admin]: coredump.save.nospace: Not enough space in root volume to save core May 5 09:10:11 [savecore_admin]: coredump.save.error: core.101185225.2013-04-22.20_18_41.nz processing encountered error May 5 09:10:11 [savecore_admin]: coredump.save.started: Saving 4556M to core.101185225.2013-04-22.21_31_46.nz (\"Freeing free block 9009630 in aggregate engdata1.\") May 5 09:10:11 [savecore_admin]: coredump.save.nospace: Not enough space in root volume to save core May 5 09:10:11 [savecore_admin]: coredump.save.error: core.101185225.2013-04-22.21_31_46.nz processing encountered error May 5 09:15:10 [coredump_manager]: coredump.findcore.missing: Partial core missing disk 1 of 12 May 5 09:15:10 [coredump_manager]: coredump.findcore.partial: Partial core is missing 1 of 12 disks May 5 09:15:11 [coredump_manager]: coredump.findcore.missing: Partial core missing disk 1 of 12 May 5 09:15:11 [coredump_manager]: coredump.findcore.partial: Partial core is missing 1 of 12 disks May 5 09:15:12 [coredump_manager]: coredump.findcore.missing: Partial core missing disk 1 of 12 May 5 09:15:12 [coredump_manager]: coredump.findcore.partial: Partial core is missing 1 of 12 disks May 5 09:15:13 [coredump_manager]: coredump.findcore.missing: Partial core missing disk 1 of 12 May 5 09:15:13 [coredump_manager]: coredump.findcore.partial: Partial core is missing 1 of 12 disks May 5 09:15:15 [savecore_admin]: coredump.findcore.missing: Partial core missing disk 1 of 12 May 5 09:15:15 [savecore_admin]: coredump.findcore.partial: Partial core is missing 1 of 12 disks May 5 09:15:15 [savecore_admin]: coredump.save.started: Saving 4558M to core.101185225.2013-04-22.17_53_15.nz (\"Freeing free block 9009630 in aggregate engdata1.\") May 5 09:15:15 [savecore_admin]: coredump.save.nospace: Not enough space in root volume to save core May 5 09:15:15 [savecore_admin]: coredump.save.error: core.101185225.2013-04-22.17_53_15.nz processing encountered error May 5 09:15:15 [savecore_admin]: coredump.save.started: Saving 4474M to core.101185225.2013-04-22.19_04_41.nz (\"Freeing free block 9009630 in aggregate engdata1.\") May 5 09:15:15 [savecore_admin]: coredump.save.nospace: Not enough space in root volume to save core May 5 09:15:15 [savecore_admin]: coredump.save.error: core.101185225.2013-04-22.19_04_41.nz processing encountered error May 5 09:15:15 [savecore_admin]: coredump.save.started: Saving 4556M to core.101185225.2013-04-22.20_18_41.nz (\"Freeing free block 9009630 in aggregate engdata1.\") May 5 09:15:15 [savecore_admin]: coredump.save.nospace: Not enough space in root volume to save core May 5 09:15:15 [savecore_admin]: coredump.save.error: core.101185225.2013-04-22.20_18_41.nz processing encountered error May 5 09:15:15 [savecore_admin]: coredump.save.started: Saving 4556M to core.101185225.2013-04-22.21_31_46.nz (\"Freeing free block 9009630 in aggregate engdata1.\") May 5 09:15:15 [savecore_admin]: coredump.save.nospace: Not enough space in root volume to save core May 5 09:15:15 [savecore_admin]: coredump.save.error: core.101185225.2013-04-22.21_31_46.nz processing encountered error May 5 09:20:13 [coredump_manager]: coredump.findcore.missing: Partial core missing disk 1 of 12 May 5 09:20:13 [coredump_manager]: coredump.findcore.partial: Partial core is missing 1 of 12 disks May 5 09:20:14 [coredump_manager]: coredump.findcore.missing: Partial core missing disk 1 of 12 May 5 09:20:14 [coredump_manager]: coredump.findcore.partial: Partial core is missing 1 of 12 disks May 5 09:20:15 [coredump_manager]: coredump.findcore.missing: Partial core missing disk 1 of 12 May 5 09:20:15 [coredump_manager]: coredump.findcore.partial: Partial core is missing 1 of 12 disks May 5 09:20:16 [coredump_manager]: coredump.findcore.missing: Partial core missing disk 1 of 12 May 5 09:20:16 [coredump_manager]: coredump.findcore.partial: Partial core is missing 1 of 12 disks May 5 09:20:17 [savecore_admin]: coredump.findcore.missing: Partial core missing disk 1 of 12 May 5 09:20:17 [savecore_admin]: coredump.findcore.partial: Partial core is missing 1 of 12 disks May 5 09:20:17 [savecore_admin]: coredump.save.started: Saving 4558M to core.101185225.2013-04-22.17_53_15.nz (\"Freeing free block 9009630 in aggregate engdata1.\") May 5 09:20:17 [savecore_admin]: coredump.save.nospace: Not enough space in root volume to save core May 5 09:20:17 [savecore_admin]: coredump.save.error: core.101185225.2013-04-22.17_53_15.nz processing encountered error May 5 09:20:17 [savecore_admin]: coredump.save.started: Saving 4474M to core.101185225.2013-04-22.19_04_41.nz (\"Freeing free block 9009630 in aggregate engdata1.\") May 5 09:20:17 [savecore_admin]: coredump.save.nospace: Not enough space in root volume to save core May 5 09:20:17 [savecore_admin]: coredump.save.error: core.101185225.2013-04-22.19_04_41.nz processing encountered error May 5 09:20:17 [savecore_admin]: coredump.save.started: Saving 4556M to core.101185225.2013-04-22.20_18_41.nz (\"Freeing free block 9009630 in aggregate engdata1.\") May 5 09:20:17 [savecore_admin]: coredump.save.nospace: Not enough space in root volume to save core May 5 09:20:17 [savecore_admin]: coredump.save.error: core.101185225.2013-04-22.20_18_41.nz processing encountered error May 5 09:20:17 [savecore_admin]: coredump.save.started: Saving 4556M to core.101185225.2013-04-22.21_31_46.nz (\"Freeing free block 9009630 in aggregate engdata1.\") May 5 09:20:17 [savecore_admin]: coredump.save.nospace: Not enough space in root volume to save core May 5 09:20:17 [savecore_admin]: coredump.save.error: core.101185225.2013-04-22.21_31_46.nz processing encountered error May 5 09:25:16 [coredump_manager]: coredump.findcore.missing: Partial core missing disk 1 of 12 May 5 09:25:16 [coredump_manager]: coredump.findcore.partial: Partial core is missing 1 of 12 disks May 5 09:25:17 [coredump_manager]: coredump.findcore.missing: Partial core missing disk 1 of 12 May 5 09:25:17 [coredump_manager]: coredump.findcore.partial: Partial core is missing 1 of 12 disks May 5 09:25:18 [coredump_manager]: coredump.findcore.missing: Partial core missing disk 1 of 12 May 5 09:25:18 [coredump_manager]: coredump.findcore.partial: Partial core is missing 1 of 12 disks May 5 09:25:19 [coredump_manager]: coredump.findcore.missing: Partial core missing disk 1 of 12 May 5 09:25:19 [coredump_manager]: coredump.findcore.partial: Partial core is missing 1 of 12 disks May 5 09:25:20 [savecore_admin]: coredump.findcore.missing: Partial core missing disk 1 of 12 May 5 09:25:20 [savecore_admin]: coredump.findcore.partial: Partial core is missing 1 of 12 disks May 5 09:25:20 [savecore_admin]: coredump.save.started: Saving 4558M to core.101185225.2013-04-22.17_53_15.nz (\"Freeing free block 9009630 in aggregate engdata1.\") May 5 09:25:20 [savecore_admin]: coredump.save.nospace: Not enough space in root volume to save core May 5 09:25:20 [savecore_admin]: coredump.save.error: core.101185225.2013-04-22.17_53_15.nz processing encountered error May 5 09:25:20 [savecore_admin]: coredump.save.started: Saving 4474M to core.101185225.2013-04-22.19_04_41.nz (\"Freeing free block 9009630 in aggregate engdata1.\") May 5 09:25:20 [savecore_admin]: coredump.save.nospace: Not enough space in root volume to save core May 5 09:25:20 [savecore_admin]: coredump.save.error: core.101185225.2013-04-22.19_04_41.nz processing encountered error May 5 09:25:20 [savecore_admin]: coredump.save.started: Saving 4556M to core.101185225.2013-04-22.20_18_41.nz (\"Freeing free block 9009630 in aggregate engdata1.\") May 5 09:25:20 [savecore_admin]: coredump.save.nospace: Not enough space in root volume to save core May 5 09:25:20 [savecore_admin]: coredump.save.error: core.101185225.2013-04-22.20_18_41.nz processing encountered error May 5 09:25:20 [savecore_admin]: coredump.save.started: Saving 4556M to core.101185225.2013-04-22.21_31_46.nz (\"Freeing free block 9009630 in aggregate engdata1.\") May 5 09:25:20 [savecore_admin]: coredump.save.nospace: Not enough space in root volume to save core May 5 09:25:20 [savecore_admin]: coredump.save.error: core.101185225.2013-04-22.21_31_46.nz processing encountered error May 5 09:30:19 [coredump_manager]: coredump.findcore.missing: Partial core missing disk 1 of 12 May 5 09:30:19 [coredump_manager]: coredump.findcore.partial: Partial core is missing 1 of 12 disks May 5 09:30:20 [coredump_manager]: coredump.findcore.missing: Partial core missing disk 1 of 12 May 5 09:30:20 [coredump_manager]: coredump.findcore.partial: Partial core is missing 1 of 12 disks May 5 09:30:21 [coredump_manager]: coredump.findcore.missing: Partial core missing disk 1 of 12 May 5 09:30:21 [coredump_manager]: coredump.findcore.partial: Partial core is missing 1 of 12 disks May 5 09:30:22 [coredump_manager]: coredump.findcore.missing: Partial core missing disk 1 of 12 May 5 09:30:22 [coredump_manager]: coredump.findcore.partial: Partial core is missing 1 of 12 disks May 5 09:30:23 [savecore_admin]: coredump.findcore.missing: Partial core missing disk 1 of 12 May 5 09:30:23 [savecore_admin]: coredump.findcore.partial: Partial core is missing 1 of 12 disks May 5 09:30:23 [savecore_admin]: coredump.save.started: Saving 4558M to core.101185225.2013-04-22.17_53_15.nz (\"Freeing free block 9009630 in aggregate engdata1.\") May 5 09:30:23 [savecore_admin]: coredump.save.nospace: Not enough space in root volume to save core May 5 09:30:23 [savecore_admin]: coredump.save.error: core.101185225.2013-04-22.17_53_15.nz processing encountered error May 5 09:30:23 [savecore_admin]: coredump.save.started: Saving 4474M to core.101185225.2013-04-22.19_04_41.nz (\"Freeing free block 9009630 in aggregate engdata1.\") May 5 09:30:23 [savecore_admin]: coredump.save.nospace: Not enough space in root volume to save core May 5 09:30:23 [savecore_admin]: coredump.save.error: core.101185225.2013-04-22.19_04_41.nz processing encountered error May 5 09:30:23 [savecore_admin]: coredump.save.started: Saving 4556M to core.101185225.2013-04-22.20_18_41.nz (\"Freeing free block 9009630 in aggregate engdata1.\") May 5 09:30:23 [savecore_admin]: coredump.save.nospace: Not enough space in root volume to save core May 5 09:30:23 [savecore_admin]: coredump.save.error: core.101185225.2013-04-22.20_18_41.nz processing encountered error May 5 09:30:23 [savecore_admin]: coredump.save.started: Saving 4556M to core.101185225.2013-04-22.21_31_46.nz (\"Freeing free block 9009630 in aggregate engdata1.\") May 5 09:30:23 [savecore_admin]: coredump.save.nospace: Not enough space in root volume to save core May 5 09:30:23 [savecore_admin]: coredump.save.error: core.101185225.2013-04-22.21_31_46.nz processing encountered error May 5 09:34:01 [registry_control]: kern.syslog.msg: registry: Unexpected error in rollback in regx_commit - Error: No such file or directory (name=options.ssh.access) Error: No such file or directory (name=options.ssh.debug.enable) Error: No such file or directory (name=options.ssh.idle.timeout) Error: No such file or directory (name=options.ssh.passwd_auth.ena May 5 09:34:01 [registry_control]: reg.transaction.commitFail: registry: Cannot commit transaction in 'Periodic config update'. Error: Invalid argument (name=options.ssh.access) (value=*) Error: Invalid argument (name=options.ssh.debug.enable) (value=off) Error: Invalid argument (name=options.ssh.idle.timeout) (value=600) Error: Invalid argument (name=options.ssh.passwd_auth.enabl May 5 09:35:31 [console_login_mgr]: kern.syslog.msg: root logged in from console May 5 09:40:25 [coredump_manager]: coredump.findcore.missing: Partial core missing disk 1 of 12 May 5 09:40:25 [coredump_manager]: coredump.findcore.partial: Partial core is missing 1 of 12 disks May 5 09:40:28 [savecore_admin]: coredump.save.started: Saving 4558M to core.101185225.2013-04-22.17_53_15.nz (\"Freeing free block 9009630 in aggregate engdata1.\") May 5 09:40:28 [savecore_admin]: coredump.save.nospace: Not enough space in root volume to save core May 5 09:40:28 [savecore_admin]: coredump.save.error: core.101185225.2013-04-22.17_53_15.nz processing encountered error May 5 09:50:35 [coredump_manager]: coredump.findcore.missing: Partial core missing disk 1 of 12 May 5 09:50:35 [coredump_manager]: coredump.findcore.partial: Partial core is missing 1 of 12 disks May 5 09:50:39 [savecore_admin]: coredump.save.started: Saving 4558M to core.101185225.2013-04-22.17_53_15.nz (\"Freeing free block 9009630 in aggregate engdata1.\") May 5 09:50:39 [savecore_admin]: coredump.save.nospace: Not enough space in root volume to save core May 5 09:50:39 [savecore_admin]: coredump.save.error: core.101185225.2013-04-22.17_53_15.nz processing encountered error May 5 10:00:00 [statd]: kern.uptime.filer: 10:00am up 1:26 12086600 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 0 iSCSI ops May 5 10:00:00 [statd]: monitor.shelf.fault: Fault reported on disk storage shelf attached to channel 0b. Please check fans, power, and temperature. May 5 10:00:00 [statd]: monitor.raiddp.vol.singleDegraded: data disk in RAID group /engdata1/plex0/rg2 is broken. May 5 10:00:01 [statd]: kern.syslog.msg: Cluster is licensed but takeover of partner is disabled May 5 10:00:40 [coredump_manager]: coredump.findcore.missing: Partial core missing disk 1 of 12 May 5 10:00:40 [coredump_manager]: coredump.findcore.partial: Partial core is missing 1 of 12 disks May 5 10:00:43 [savecore_admin]: coredump.save.started: Saving 4558M to core.101185225.2013-04-22.17_53_15.nz (\"Freeing free block 9009630 in aggregate engdata1.\") May 5 10:00:43 [savecore_admin]: coredump.save.nospace: Not enough space in root volume to save core May 5 10:00:43 [savecore_admin]: coredump.save.error: core.101185225.2013-04-22.17_53_15.nz processing encountered error May 5 10:05:00 [mgmtgwd]: mgmtgwd.jobmgr.private.jobstart: Started private job "snap-hourly" [id 4]: Auto-Snapshot. May 5 10:05:38 [mgmtgwd]: mgmtgwd.jobmgr.private.jobcomplete.success: Private job "snap-hourly" [id 4] (Auto-Snapshot) completed successfully: . May 5 10:10:46 [coredump_manager]: coredump.findcore.missing: Partial core missing disk 1 of 12 May 5 10:10:46 [coredump_manager]: coredump.findcore.partial: Partial core is missing 1 of 12 disks May 5 10:10:50 [savecore_admin]: coredump.save.started: Saving 4558M to core.101185225.2013-04-22.17_53_15.nz (\"Freeing free block 9009630 in aggregate engdata1.\") May 5 10:10:50 [savecore_admin]: coredump.save.nospace: Not enough space in root volume to save core May 5 10:10:50 [savecore_admin]: coredump.save.error: core.101185225.2013-04-22.17_53_15.nz processing encountered error May 5 10:20:54 [coredump_manager]: coredump.findcore.missing: Partial core missing disk 1 of 12 May 5 10:20:54 [coredump_manager]: coredump.findcore.partial: Partial core is missing 1 of 12 disks May 5 10:20:58 [savecore_admin]: coredump.save.started: Saving 4558M to core.101185225.2013-04-22.17_53_15.nz (\"Freeing free block 9009630 in aggregate engdata1.\") May 5 10:20:58 [savecore_admin]: coredump.save.nospace: Not enough space in root volume to save core May 5 10:20:58 [savecore_admin]: coredump.save.error: core.101185225.2013-04-22.17_53_15.nz processing encountered error May 5 10:20:58 [savecore_admin]: coredump.save.started: Saving 4474M to core.101185225.2013-04-22.19_04_41.nz (\"Freeing free block 9009630 in aggregate engdata1.\") May 5 10:20:58 [savecore_admin]: coredump.save.nospace: Not enough space in root volume to save core May 5 10:20:58 [savecore_admin]: coredump.save.error: core.101185225.2013-04-22.19_04_41.nz processing encountered error May 5 10:20:58 [savecore_admin]: coredump.save.started: Saving 4556M to core.101185225.2013-04-22.20_18_41.nz (\"Freeing free block 9009630 in aggregate engdata1.\") May 5 10:20:58 [savecore_admin]: coredump.save.nospace: Not enough space in root volume to save core May 5 10:20:58 [savecore_admin]: coredump.save.error: core.101185225.2013-04-22.20_18_41.nz processing encountered error May 5 10:20:58 [savecore_admin]: coredump.save.started: Saving 4556M to core.101185225.2013-04-22.21_31_46.nz (\"Freeing free block 9009630 in aggregate engdata1.\") May 5 10:20:58 [savecore_admin]: coredump.save.nospace: Not enough space in root volume to save core May 5 10:20:58 [savecore_admin]: coredump.save.error: core.101185225.2013-04-22.21_31_46.nz processing encountered error May 5 10:25:57 [coredump_manager]: coredump.findcore.missing: Partial core missing disk 1 of 12 May 5 10:25:57 [coredump_manager]: coredump.findcore.partial: Partial core is missing 1 of 12 disks May 5 10:25:58 [coredump_manager]: coredump.findcore.missing: Partial core missing disk 1 of 12 May 5 10:25:58 [coredump_manager]: coredump.findcore.partial: Partial core is missing 1 of 12 disks May 5 10:25:59 [coredump_manager]: coredump.findcore.missing: Partial core missing disk 1 of 12 May 5 10:25:59 [coredump_manager]: coredump.findcore.partial: Partial core is missing 1 of 12 disks May 5 10:26:00 [coredump_manager]: coredump.findcore.missing: Partial core missing disk 1 of 12 May 5 10:26:00 [coredump_manager]: coredump.findcore.partial: Partial core is missing 1 of 12 disks May 5 10:26:01 [savecore_admin]: coredump.findcore.missing: Partial core missing disk 1 of 12 May 5 10:26:01 [savecore_admin]: coredump.findcore.partial: Partial core is missing 1 of 12 disks May 5 10:26:01 [savecore_admin]: coredump.save.started: Saving 4558M to core.101185225.2013-04-22.17_53_15.nz (\"Freeing free block 9009630 in aggregate engdata1.\") May 5 10:26:01 [savecore_admin]: coredump.save.nospace: Not enough space in root volume to save core May 5 10:26:01 [savecore_admin]: coredump.save.error: core.101185225.2013-04-22.17_53_15.nz processing encountered error May 5 10:26:01 [savecore_admin]: coredump.save.started: Saving 4474M to core.101185225.2013-04-22.19_04_41.nz (\"Freeing free block 9009630 in aggregate engdata1.\") May 5 10:26:01 [savecore_admin]: coredump.save.nospace: Not enough space in root volume to save core May 5 10:26:01 [savecore_admin]: coredump.save.error: core.101185225.2013-04-22.19_04_41.nz processing encountered error May 5 10:26:01 [savecore_admin]: coredump.save.started: Saving 4556M to core.101185225.2013-04-22.20_18_41.nz (\"Freeing free block 9009630 in aggregate engdata1.\") May 5 10:26:01 [savecore_admin]: coredump.save.nospace: Not enough space in root volume to save core May 5 10:26:01 [savecore_admin]: coredump.save.error: core.101185225.2013-04-22.20_18_41.nz processing encountered error May 5 10:26:01 [savecore_admin]: coredump.save.started: Saving 4556M to core.101185225.2013-04-22.21_31_46.nz (\"Freeing free block 9009630 in aggregate engdata1.\") May 5 10:26:01 [savecore_admin]: coredump.save.nospace: Not enough space in root volume to save core May 5 10:26:01 [savecore_admin]: coredump.save.error: core.101185225.2013-04-22.21_31_46.nz processing encountered error May 5 10:27:30 [wafl_junkd_aggroffl]: wafl.buf.freeingFreeBlock: Freeing free block 926442036 in aggregate engdata1.\n WAFL inconsistent. Please contact Network Appliance Global Services. May 5 10:27:31 [wafl_junkd_aggroffl]: sk.panic: Panic String: Freeing free block 926442036 in aggregate engdata1. WAFL inconsistent. Please contact Network Appliance Global Services. in SK process wafl_junkd_aggroffl on release NetApp Release R10.0.1xN_070123_1900 May 5 10:27:31 [wafl_junkd_aggroffl]: coredump.findcore.missing: Partial core missing disk 1 of 12 May 5 10:27:31 [wafl_junkd_aggroffl]: coredump.findcore.partial: Partial core is missing 1 of 12 disks May 5 10:27:31 [wafl_junkd_aggroffl]: coredump.dump.find.all.used: All available disks contain unsaved cores May 5 10:27:31 [wafl_junkd_aggroffl]: coredump.dump.failed: Dumpcore failed May 5 10:28:37 [nic_mgr]: cf.nm.nicTransitionUp: Interconnect link 0 is UP May 5 10:28:45 [ispfc_timeout_3]: fci.device.quiesce: Adapter unknown encountered a command timeout on 0b device 0b.36 (0x03000024). Quiescing the device. May 5 10:28:49 [sanown_notify]: diskown.isEnabled: software ownership has been enabled for this system May 5 10:28:52 [nic_mgr]: cf.nm.nicReset: Initiating soft reset on Cluster Interconnect card 0 due to rendezvous reset May 5 10:28:52 [rendezvous_proc]: cf.rv.notConnected: Connection for cfo_rv failed May 5 10:28:52 [nic_mgr]: cf.nm.nicTransitionDown: Cluster Interconnect link 0 is DOWN May 5 10:28:52 [rendezvous_proc]: cf.rv.notConnected: Connection for cfo_rv failed May 5 10:28:54 [nic_mgr]: cf.nm.nicTransitionUp: Interconnect link 0 is UP May 5 10:28:58 [fmmbx_instanceWorke]: fmmb.current.lock.disk: Disk 0c.58 is a local HA mailbox disk. May 5 10:28:58 [fmmbx_instanceWorke]: fmmb.current.lock.disk: Disk 0a.29 is a local HA mailbox disk. May 5 10:28:58 [fmmbx_instanceWorke]: kern.syslog.msg: normal mailbox instance on local side May 5 10:28:58 [rc]: cf.fm.MBstatusOnBoot: status="1" May 5 10:29:07 [config_thread]: raid.assim.disk.brokenPreAssim: Broken Disk 0c.60 Shelf ? Bay ? [NETAPP X267_MGRIZ500SSX H3DD] S/N [H814LZ6H] detected prior to assimilation. It should be removed. May 5 10:29:07 [config_thread]: raid.assim.disk.brokenPreAssim: Broken Disk 0b.44 Shelf ? Bay ? [NETAPP X267_MGRIZ500SSX H3DD] S/N [H817GR5H] detected prior to assimilation. It should be removed. May 5 10:29:07 [config_thread]: raid.assim.disk.brokenPreAssim: Broken Disk 0b.41 Shelf ? Bay ? [NETAPP X267_MGRIZ500SSX H3DD] S/N [H817LZ8H] detected prior to assimilation. It should be removed. May 5 10:29:07 [config_thread]: raid.assim.disk.brokenPreAssim: Broken Disk 0a.23 Shelf ? Bay ? [NETAPP X267_MGRIZ500SSX H3DD] S/N [H814LVPH] detected prior to assimilation. It should be removed. May 5 10:29:07 [config_thread]: raid.assim.disk.brokenPreAssim: Broken Disk 0c.53 Shelf ? Bay ? [NETAPP X267_MGRIZ500SSX H3DD] S/N [H8155VEH] detected prior to assimilation. It should be removed. May 5 10:29:07 [config_thread]: raid.assim.disk.brokenPreAssim: Broken Disk 0c.57 Shelf ? Bay ? [NETAPP X267_MGRIZ500SSX H3DD] S/N [H814R7RH] detected prior to assimilation. It should be removed. May 5 10:29:07 [fmmbx_instanceWorke]: fmmb.NewDiskSuggestion: count="4", new="0c.58, 0a.29, null, null", old="0c.58, 0a.29" May 5 10:29:07 [fmmbx_instanceWorke]: fmmb.current.lock.disk: Disk 0d.70 is a partner HA mailbox disk. May 5 10:29:07 [fmmbx_instanceWorke]: fmmb.current.lock.disk: Disk 0d.67 is a partner HA mailbox disk. May 5 10:29:07 [fmmbx_instanceWorke]: kern.syslog.msg: normal mailbox instance on partner side May 5 10:29:07 [rc]: cf.fm.partner: Cluster monitor: partner 'toast1b' May 5 10:29:07 [rc]: cf.fm.timeMasterStatus: Acting as cluster time master May 5 10:29:09 [ses_admin]: ses.channel.rescanInitiated: Initiating rescan on channel 0b May 5 10:29:14 [config_thread]: raid.vol.replay.nvram: Performing raid replay on volume(s) May 5 10:29:14 [rc]: raid.cksum.replay.summary: Replayed 0 checksum blocks. May 5 10:29:14 [rc]: raid.stripe.replay.summary: Replayed 0 stripes. May 5 10:29:15 [rc]: wafl.inconsistent.vol: WAFL inconsistent: Aggregate engdata1 is inconsistent. Note: Any new Snapshot copies might contain this inconsistency. May 5 10:29:15 [rc]: wafl.vol.loading: Loading Aggregate engdata1 May 5 10:29:15 [rc]: wafl.vol.loading: Loading Aggregate aggr0 May 5 10:29:15 [vv_apply_worker19]: wafl.vol.loading: Loading Volume bluecore@vserver:b1a2013d-acca-11e0-98c0-000e0caac15e May 5 10:29:15 [vv_apply_worker16]: wafl.vol.loading: Loading Volume mroot@vserver:06866009-a1fe-11e0-8650-000e0caac15e May 5 10:29:15 [vv_apply_worker17]: wafl.vol.loading: Loading Volume engdata1_root@vserver:b1a2013d-acca-11e0-98c0-000e0caac15e May 5 10:29:15 [vv_apply_worker18]: wafl.vol.loading: Loading Volume engdata1@vserver:b1a2013d-acca-11e0-98c0-000e0caac15e May 5 10:29:16 [vv_apply_worker19]: wafl.maxdirsize.boot.notice: bluecore@vserver:b1a2013d-acca-11e0-98c0-000e0caac15e: This volume's maxdirsize (102400KB) is higher than the default (35573KB). There may be a performance penalty when doing operations on large directories. May 5 10:29:16 [rc]: cf.fm.launch: Launching cluster monitor May 5 10:29:16 [rc]: cf.fm.partner: Cluster monitor: partner 'toast1b' May 5 10:29:16 [rc]: cf.fm.notkoverClusterDisable: Cluster monitor: cluster takeover disabled (restart) May 5 10:29:16 [cf_main]: cf.fsm.takeoverOfPartnerDisabled: Cluster monitor: takeover of toast1b disabled (cluster takeover disabled) May 5 10:29:27 [rc]: license.db.migrate.vol.success: Successfully migrated licenses for volume engdata1 May 5 10:29:27 [rc]: license.db.migrate.vol.success: Successfully migrated licenses for volume aggr0 May 5 10:29:27 [rc]: license.db.migrate.success: Migration complete May 5 10:29:27 [fmmbx_instanceWorke]: fmmb.NewDiskSuggestion: count="4", new="0c.58, 0a.29, null, null", old="0c.58, 0a.29" May 5 10:29:27 [rc]: kern.syslog.msg: The system was down for 101 seconds May 5 10:29:27 [raidio_thread]: raid.disk.unload.done: Unload of Disk 0c.60 Shelf 3 Bay 12 [NETAPP X267_MGRIZ500SSX H3DD] S/N [H814LZ6H] has completed successfully May 5 10:29:27 [raidio_thread]: raid.disk.unload.done: Unload of Disk 0b.44 Shelf 2 Bay 12 [NETAPP X267_MGRIZ500SSX H3DD] S/N [H817GR5H] has completed successfully May 5 10:29:27 [raidio_thread]: raid.disk.unload.done: Unload of Disk 0b.41 Shelf 2 Bay 9 [NETAPP X267_MGRIZ500SSX H3DD] S/N [H817LZ8H] has completed successfully May 5 10:29:27 [raidio_thread]: raid.disk.unload.done: Unload of Disk 0a.23 Shelf 1 Bay 7 [NETAPP X267_MGRIZ500SSX H3DD] S/N [H814LVPH] has completed successfully May 5 10:29:27 [raidio_thread]: raid.disk.unload.done: Unload of Disk 0c.53 Shelf 3 Bay 5 [NETAPP X267_MGRIZ500SSX H3DD] S/N [H8155VEH] has completed successfully May 5 10:29:27 [raidio_thread]: raid.disk.unload.done: Unload of Disk 0c.57 Shelf 3 Bay 9 [NETAPP X267_MGRIZ500SSX H3DD] S/N [H814R7RH] has completed successfully May 5 10:29:27 [rc]: raid.fdr.failed.ok: Disk 0c.60 Shelf 3 Bay 12 [NETAPP X267_MGRIZ500SSX H3DD] S/N [H814LZ6H] successfully deleted from spare pool May 5 10:29:27 [rc]: raid.fdr.failed.ok: Disk 0b.44 Shelf 2 Bay 12 [NETAPP X267_MGRIZ500SSX H3DD] S/N [H817GR5H] successfully deleted from spare pool May 5 10:29:27 [rc]: raid.fdr.failed.ok: Disk 0b.41 Shelf 2 Bay 9 [NETAPP X267_MGRIZ500SSX H3DD] S/N [H817LZ8H] successfully deleted from spare pool May 5 10:29:27 [rc]: raid.fdr.failed.ok: Disk 0a.23 Shelf 1 Bay 7 [NETAPP X267_MGRIZ500SSX H3DD] S/N [H814LVPH] successfully deleted from spare pool May 5 10:29:27 [rc]: raid.fdr.failed.ok: Disk 0c.53 Shelf 3 Bay 5 [NETAPP X267_MGRIZ500SSX H3DD] S/N [H8155VEH] successfully deleted from spare pool May 5 10:29:27 [rc]: raid.fdr.failed.ok: Disk 0c.57 Shelf 3 Bay 9 [NETAPP X267_MGRIZ500SSX H3DD] S/N [H814R7RH] successfully deleted from spare pool May 5 10:29:27 [rc]: kern.syslog.msg: registry: Unexpected error in rollback in regx_commit - Error: No such file or directory (name=options.ssh.access) Error: No such file or directory (name=options.ssh.debug.enable) Error: No such file or directory (name=options.ssh.idle.timeout) Error: No such file or directory (name=options.ssh.passwd_auth.ena May 5 10:29:27 [rc]: reg.transaction.commitFail: registry: Cannot commit transaction in 'Periodic config update'. Error: Invalid argument (name=options.ssh.access) (value=*) Error: Invalid argument (name=options.ssh.debug.enable) (value=off) Error: Invalid argument (name=options.ssh.idle.timeout) (value=600) Error: Invalid argument (name=options.ssh.passwd_auth.enabl May 5 10:29:27 [rc]: kern.syslog.msg: sysconfig: validating configuration May 5 10:29:27 [config_thread]: raid.rg.recons.missing: RAID group /engdata1/plex0/rg2 is missing 1 disk(s). May 5 10:29:27 [config_thread]: raid.rg.recons.cantStart: The reconstruction cannot start in RAID group /engdata1/plex0/rg2: No 520 bps disk of required size available in spare pool May 5 10:29:27 [rc]: wafl.inconsistent.vol: WAFL inconsistent: Aggregate engdata1 is inconsistent. Note: Any new Snapshot copies might contain this inconsistency. May 5 10:29:28 [rc]: mgr.boot.disk_done: NetApp Release ontap boot complete. Last disk update written at Sun May 5 10:27:25 GMT 2013 May 5 10:29:28 [rc]: mgr.boot.reason_abnormal: System rebooted after a panic. May 5 10:29:28 [rc]: mgr.stack.module: Module kernel loaded from 0x80000000 - 0x100000000. May 5 10:29:28 [rc]: mgr.stack.module: Module common_kmod.ko loaded from 0x81a00000 - 0x81aa6000. May 5 10:29:28 [rc]: mgr.stack.module: Module nvram5.ko loaded from 0x81aa6000 - 0x81ab8000. May 5 10:29:28 [rc]: mgr.stack.module: Module nvram_mgr.ko loaded from 0x81ab8000 - 0x81abd000. May 5 10:29:28 [rc]: mgr.stack.module: Module nvr.ko loaded from 0x81abd000 - 0x81ad0000. May 5 10:29:28 [rc]: mgr.stack.module: Module maytag.ko loaded from 0x81ad0000 - 0x858db000. May 5 10:29:28 [rc]: mgr.stack.module: Module spinvfs.ko loaded from 0x858db000 - 0x85924000. May 5 10:29:28 [rc]: mgr.stack.module: Module nbladekmod.ko loaded from 0x85924000 - 0x876b8000. May 5 10:29:28 [rc]: mgr.stack.saved: Reboot with saved panic information in log file May 5 10:29:28 [rc]: mgr.stack.string: Panic string: Freeing free block 926442036 in aggregate engdata1. WAFL inconsistent. Please contact Network Appliance Global Services. in SK process wafl_junkd_aggroffl on r May 5 10:29:28 [rc]: mgr.stack.at: Panic occurred at: Sun May 5 10:27:31 2013 May 5 10:29:28 [rc]: mgr.stack.proc: Panic in process: wafl_junkd_aggroffl May 5 10:29:28 [rc]: mgr.stack.framename: Stack frame 0: maytag.ko::ontap_dumpcore(0x82146b00) + 0xb4 May 5 10:29:28 [rc]: mgr.stack.framename: Stack frame 1: kernel::dumpsys(0x80477260) + 0xf8 May 5 10:29:28 [rc]: mgr.stack.framename: Stack frame 2: kernel::shutdown_get_howto(0x80324460) + 0x5d May 5 10:29:28 [rc]: mgr.stack.framename: Stack frame 3: kernel::shutdown_get_howto(0x80324460) + 0x45d May 5 10:29:28 [rc]: mgr.stack.framename: Stack frame 4: kernel::panic(0x80325050) + 0x189 May 5 10:29:28 [rc]: mgr.stack.framename: Stack frame 5: maytag.ko::oscall_panic(0x82148070) + 0x11 May 5 10:29:28 [rc]: mgr.stack.framename: Stack frame 6: maytag.ko::sk_vpanic(0x8226e46e) + 0x86 May 5 10:29:28 [rc]: mgr.stack.framename: Stack frame 7: maytag.ko::sk_vpanic(0x8226e46e) + 0x0 May 5 10:29:28 [rc]: mgr.stack.framename: Stack frame 8: maytag.ko::wafl_junkd_aggroffline_handle_inconsistency_cb(0x829f9ccc) + 0x0 May 5 10:29:28 [rc]: mgr.stack.framename: Stack frame 9: maytag.ko::wafl_junkd_aggroffline_worker(0x829f9ce5) + 0x1c3 May 5 10:29:28 [rc]: mgr.stack.framename: Stack frame 10: maytag.ko::sk_stack_top(0x8226e1b6) + 0x0 May 5 10:29:28 [snmp_trap_main]: snmp.link.up: Interface 5 is up May 5 10:29:28 [sshd]: kern.syslog.msg: error: Could not load host key: /etc/sshd/ssh_host_key May 5 10:29:28 [sshd]: kern.syslog.msg: error: Could not load host key: /etc/sshd/ssh_host_rsa_key May 5 10:29:28 [sshd]: kern.syslog.msg: error: Could not load host key: /etc/sshd/ssh_host_dsa_key May 5 10:29:28 [ispfc_main]: ses.shelf.mmErr: SCSI Enclosure Services (SES) device accessed by 0b.30 (product revision level 3333) is not supported on this platform.