Advertisement
Not a member of Pastebin yet?
Sign Up,
it unlocks many cool features!
- Controller-0
- Jul 19 15:49:35 overcloud-controller-cmm7-0 lrmd[59785]: warning: ip-172.17.1.23_monitor_10000 process (PID 799302) timed out
- Jul 19 15:49:35 overcloud-controller-cmm7-0 lrmd[59785]: warning: ip-172.17.1.23_monitor_10000:799302 - timed out after 20000ms
- Jul 19 15:49:35 overcloud-controller-cmm7-0 lrmd[59785]: warning: ip-135.111.247.193_monitor_10000 process (PID 799303) timed out
- Jul 19 15:49:35 overcloud-controller-cmm7-0 lrmd[59785]: warning: ip-172.31.0.24_monitor_10000 process (PID 799304) timed out
- Jul 19 15:49:35 overcloud-controller-cmm7-0 lrmd[59785]: warning: ip-135.111.247.193_monitor_10000:799303 - timed out after 20000ms
- Jul 19 15:49:35 overcloud-controller-cmm7-0 crmd[59788]: error: Result of monitor operation for ip-172.17.1.23 on overcloud-controller-cmm7-0: Timed Out
- Jul 19 15:49:35 overcloud-controller-cmm7-0 crmd[59788]: error: Result of monitor operation for ip-135.111.247.193 on overcloud-controller-cmm7-0: Timed Out
- Jul 19 15:49:36 overcloud-controller-cmm7-0 lrmd[59785]: warning: ip-172.31.0.24_monitor_10000:799304 - timed out after 20000ms
- Jul 19 15:49:36 overcloud-controller-cmm7-0 crmd[59788]: error: Result of monitor operation for ip-172.31.0.24 on overcloud-controller-cmm7-0: Timed Out
- Jul 19 15:49:36 overcloud-controller-cmm7-0 snmptrapd[665777]: 2019-07-19 15:49:36 localhost [UDP: [127.0.0.1]:1161->[127.0.0.1]:162]:#012DISMAN-EVENT-MIB::sysUpTimeInstance = Timeticks: (58
- Jul 19 15:49:36 overcloud-controller-cmm7-0 snmptrapd[665777]: 2019-07-19 15:49:36 localhost [UDP: [127.0.0.1]:1161->[127.0.0.1]:162]:#012DISMAN-EVENT-MIB::sysUpTimeInstance = Timeticks: (58
- Jul 19 15:49:36 overcloud-controller-cmm7-0 IPaddr2(ip-172.17.1.23)[799988]: INFO: IP status = ok, IP_CIP=
- Jul 19 15:49:36 overcloud-controller-cmm7-0 crmd[59788]: notice: Result of stop operation for ip-172.17.1.23 on overcloud-controller-cmm7-0: 0 (ok)
- Jul 19 15:49:37 overcloud-controller-cmm7-0 IPaddr2(ip-172.17.1.23)[800505]: INFO: Adding inet address 172.17.1.23/32 with broadcast address 172.17.1.255 to device vlan141
- Jul 19 15:49:37 overcloud-controller-cmm7-0 IPaddr2(ip-172.17.1.23)[800505]: INFO: Bringing device vlan141 up
- Jul 19 15:49:37 overcloud-controller-cmm7-0 IPaddr2(ip-172.17.1.23)[800505]: INFO: /usr/libexec/heartbeat/send_arp -i 200 -r 5 -p /var/run/resource-agents/send_arp-172.17.1.23 vlan141 172.1
- Jul 19 15:49:37 overcloud-controller-cmm7-0 crmd[59788]: notice: Result of start operation for ip-172.17.1.23 on overcloud-controller-cmm7-0: 0 (ok)
- Jul 19 15:49:41 overcloud-controller-cmm7-0 IPaddr2(ip-172.17.1.23)[800505]: INFO: ARPING 172.17.1.23 from 172.17.1.23 vlan141#012Sent 5 probes (5 broadcast(s))#012Received 0 response(s)
- Jul 19 15:49:41 overcloud-controller-cmm7-0 crmd[59788]: notice: Result of stop operation for zabbix-server on overcloud-controller-cmm7-0: 0 (ok)
- Jul 19 15:49:48 overcloud-controller-cmm7-0 snmptrapd[665777]: NET-SNMP version 5.7.2 AgentX subagent connected
- Jul 19 15:50:58 overcloud-controller-cmm7-0 crmd[59788]: notice: Result of stop operation for alarm-manager on overcloud-controller-cmm7-0: 0 (ok)
- Jul 19 15:50:59 overcloud-controller-cmm7-0 crmd[59788]: notice: Result of stop operation for cbis-vitrage-graph on overcloud-controller-cmm7-0: 0 (ok)
- Jul 19 15:50:59 overcloud-controller-cmm7-0 lrmd[59785]: notice: alarm-manager_start_0:813246:stderr [ Error: No such object: alarm-manager ]
- Jul 19 15:50:59 overcloud-controller-cmm7-0 lrmd[59785]: notice: alarm-manager_start_0:813246:stderr [ Error: No such object: alarm-manager ]
- Jul 19 15:51:00 overcloud-controller-cmm7-0 crmd[59788]: notice: Result of start operation for alarm-manager on overcloud-controller-cmm7-0: 0 (ok)
- Jul 19 15:51:00 overcloud-controller-cmm7-0 lrmd[59785]: notice: cbis-vitrage-graph_start_0:813247:stderr [ Error: No such object: cbis-vitrage-graph ]
- Jul 19 15:51:00 overcloud-controller-cmm7-0 lrmd[59785]: notice: cbis-vitrage-graph_start_0:813247:stderr [ Error: No such object: cbis-vitrage-graph ]
- Jul 19 15:51:00 overcloud-controller-cmm7-0 crmd[59788]: notice: Result of start operation for cbis-vitrage-graph on overcloud-controller-cmm7-0: 0 (ok)
- Jul 19 15:51:00 overcloud-controller-cmm7-0 lrmd[59785]: notice: zabbix-server_start_0:813245:stderr [ Error: No such object: zabbix-server ]
- Jul 19 15:51:00 overcloud-controller-cmm7-0 lrmd[59785]: notice: zabbix-server_start_0:813245:stderr [ Error: No such object: zabbix-server ]
- Jul 19 15:51:00 overcloud-controller-cmm7-0 crmd[59788]: notice: Result of start operation for zabbix-server on overcloud-controller-cmm7-0: 0 (ok)
- Jul 19 20:58:22 overcloud-controller-cmm7-0 pacemaker_remoted[8]: warning: Could not notify client proxy-cib_rw-988636-f4d0e12b/f4d0e12b-c22e-4ed0-814d-b64c6fa47991: Broken pipe
- Jul 19 20:59:54 overcloud-controller-cmm7-0 snmptrapd[665777]: 2019-07-19 20:59:54 localhost [UDP: [127.0.0.1]:1161->[127.0.0.1]:162]:#012DISMAN-EVENT-MIB::sysUpTimeInstance = Timeticks: (18
- Jul 19 20:59:54 overcloud-controller-cmm7-0 snmptrapd[665777]: 2019-07-19 20:59:54 localhost [UDP: [127.0.0.1]:1161->[127.0.0.1]:162]:#012DISMAN-EVENT-MIB::sysUpTimeInstance = Timeticks: (18
- Jul 19 21:00:52 overcloud-controller-cmm7-0 rabbitmq-cluster(rabbitmq)[1000821]: INFO: RabbitMQ is a minority partition, failing monitor
- Jul 19 21:00:54 overcloud-controller-cmm7-0 crmd[59788]: notice: Result of notify operation for rabbitmq on rabbitmq-bundle-0: 0 (ok)
- Jul 19 21:01:04 overcloud-controller-cmm7-0 pacemaker_remoted[8]: notice: rabbitmq_stop_0:1001322:stderr [ Error: unable to connect to node 'rabbit@overcloud-controller-cmm7-0': nodedown ]
- Jul 19 21:01:04 overcloud-controller-cmm7-0 pacemaker_remoted[8]: notice: rabbitmq_stop_0:1001322:stderr [ ]
- Jul 19 21:01:04 overcloud-controller-cmm7-0 pacemaker_remoted[8]: notice: rabbitmq_stop_0:1001322:stderr [ DIAGNOSTICS ]
- Jul 19 21:01:04 overcloud-controller-cmm7-0 pacemaker_remoted[8]: notice: rabbitmq_stop_0:1001322:stderr [ =========== ]
- Jul 19 21:01:04 overcloud-controller-cmm7-0 pacemaker_remoted[8]: notice: rabbitmq_stop_0:1001322:stderr [ ]
- Jul 19 21:01:04 overcloud-controller-cmm7-0 pacemaker_remoted[8]: notice: rabbitmq_stop_0:1001322:stderr [ attempted to contact: ['rabbit@overcloud-controller-cmm7-0'] ]
- Jul 19 21:01:04 overcloud-controller-cmm7-0 pacemaker_remoted[8]: notice: rabbitmq_stop_0:1001322:stderr [ ]
- Jul 19 21:01:04 overcloud-controller-cmm7-0 pacemaker_remoted[8]: notice: rabbitmq_stop_0:1001322:stderr [ rabbit@overcloud-controller-cmm7-0: ]
- Jul 19 21:01:04 overcloud-controller-cmm7-0 pacemaker_remoted[8]: notice: rabbitmq_stop_0:1001322:stderr [ * connected to epmd (port 4369) on overcloud-controller-cmm7-0 ]
- Jul 19 21:01:04 overcloud-controller-cmm7-0 pacemaker_remoted[8]: notice: rabbitmq_stop_0:1001322:stderr [ * epmd reports: node 'rabbit' not running at all ]
- Jul 19 21:01:04 overcloud-controller-cmm7-0 pacemaker_remoted[8]: notice: rabbitmq_stop_0:1001322:stderr [ no other nodes on overcloud-controller-cmm7-0 ]
- Jul 19 21:01:04 overcloud-controller-cmm7-0 pacemaker_remoted[8]: notice: rabbitmq_stop_0:1001322:stderr [ * suggestion: start the node ]
- Jul 19 21:01:04 overcloud-controller-cmm7-0 pacemaker_remoted[8]: notice: rabbitmq_stop_0:1001322:stderr [ ]
- Jul 19 21:01:04 overcloud-controller-cmm7-0 pacemaker_remoted[8]: notice: rabbitmq_stop_0:1001322:stderr [ current node details: ]
- Jul 19 21:01:04 overcloud-controller-cmm7-0 pacemaker_remoted[8]: notice: rabbitmq_stop_0:1001322:stderr [ - node name: 'rabbitmq-cli-52@overcloud-controller-cmm7-0' ]
- Jul 19 21:01:04 overcloud-controller-cmm7-0 pacemaker_remoted[8]: notice: rabbitmq_stop_0:1001322:stderr [ - home dir: /var/lib/rabbitmq ]
- Jul 19 21:01:04 overcloud-controller-cmm7-0 pacemaker_remoted[8]: notice: rabbitmq_stop_0:1001322:stderr [ - cookie hash: 6iGxrPWDr6/m4PB15wvwkA== ]
- Jul 19 21:01:04 overcloud-controller-cmm7-0 pacemaker_remoted[8]: notice: rabbitmq_stop_0:1001322:stderr [ ]
- Jul 19 21:01:04 overcloud-controller-cmm7-0 crmd[59788]: notice: Result of stop operation for rabbitmq on rabbitmq-bundle-0: 0 (ok)
- Jul 19 21:01:14 overcloud-controller-cmm7-0 rabbitmq-cluster(rabbitmq)[1002270]: INFO: RabbitMQ server could not get cluster status from mnesia
- Jul 19 21:01:15 overcloud-controller-cmm7-0 rabbitmq-cluster(rabbitmq)[1002270]: INFO: wiping data directory before joining
- Jul 19 21:01:17 overcloud-controller-cmm7-0 rabbitmq-cluster(rabbitmq)[1002270]: INFO: Forgetting rabbit@overcloud-controller-cmm7-0 via nodes [ rabbit@overcloud-controller-cmm7-1 rabbit@ove
- Jul 19 21:01:17 overcloud-controller-cmm7-0 rabbitmq-cluster(rabbitmq)[1002270]: ERROR: Failed to forget node rabbit@overcloud-controller-cmm7-0 via rabbit@overcloud-controller-cmm7-1.
- Jul 19 21:01:18 overcloud-controller-cmm7-0 rabbitmq-cluster(rabbitmq)[1002270]: ERROR: Failed to forget node rabbit@overcloud-controller-cmm7-0 via rabbit@overcloud-controller-cmm7-2.
- Jul 19 21:01:18 overcloud-controller-cmm7-0 rabbitmq-cluster(rabbitmq)[1002270]: INFO: Joining existing cluster with [ rabbit@overcloud-controller-cmm7-1 rabbit@overcloud-controller-cmm7-2
- Jul 19 21:01:18 overcloud-controller-cmm7-0 rabbitmq-cluster(rabbitmq)[1002270]: INFO: Waiting for server to start
- Controller-2
- Jul 19 15:00:55 overcloud-controller-cmm7-2 snmptrapd[649113]: AgentX master agent failed to respond to ping. Attempting to re-register.
- Jul 19 15:01:01 overcloud-controller-cmm7-2 lrmd[59753]: warning: ip-172.17.4.14_monitor_10000 process (PID 709649) timed out
- Jul 19 15:01:01 overcloud-controller-cmm7-2 lrmd[59753]: warning: ip-172.17.4.14_monitor_10000:709649 - timed out after 20000ms
- Jul 19 15:01:01 overcloud-controller-cmm7-2 lrmd[59753]: warning: ip-172.17.1.10_monitor_10000 process (PID 709650) timed out
- Jul 19 15:01:01 overcloud-controller-cmm7-2 crmd[59756]: error: Result of monitor operation for ip-172.17.4.14 on overcloud-controller-cmm7-2: Timed Out
- Jul 19 15:01:01 overcloud-controller-cmm7-2 crmd[59756]: notice: State transition S_IDLE -> S_POLICY_ENGINE
- Jul 19 15:01:01 overcloud-controller-cmm7-2 lrmd[59753]: warning: ip-172.17.1.10_monitor_10000:709650 - timed out after 20000ms
- Jul 19 15:01:01 overcloud-controller-cmm7-2 crmd[59756]: error: Result of monitor operation for ip-172.17.1.10 on overcloud-controller-cmm7-2: Timed Out
- Jul 19 15:01:05 overcloud-controller-cmm7-2 pengine[59755]: warning: Processing failed monitor of ip-172.17.1.10 on overcloud-controller-cmm7-2: unknown error
- Jul 19 15:01:05 overcloud-controller-cmm7-2 pengine[59755]: warning: Processing failed monitor of ip-172.17.4.14 on overcloud-controller-cmm7-2: unknown error
- Jul 19 15:01:05 overcloud-controller-cmm7-2 pengine[59755]: notice: * Recover ip-172.17.1.10 ( overcloud-controller-cmm7-2 )
- Jul 19 15:01:05 overcloud-controller-cmm7-2 pengine[59755]: notice: * Recover ip-172.17.4.14 ( overcloud-controller-cmm7-2 )
- Jul 19 15:01:05 overcloud-controller-cmm7-2 pengine[59755]: notice: Calculated transition 764, saving inputs in /var/lib/pacemaker/pengine/pe-input-111.bz2
- Jul 19 15:01:05 overcloud-controller-cmm7-2 pengine[59755]: warning: Processing failed monitor of ip-172.17.1.10 on overcloud-controller-cmm7-2: unknown error
- Jul 19 15:01:05 overcloud-controller-cmm7-2 pengine[59755]: warning: Processing failed monitor of ip-172.17.4.14 on overcloud-controller-cmm7-2: unknown error
- Jul 19 15:01:05 overcloud-controller-cmm7-2 pengine[59755]: notice: * Recover ip-172.17.1.10 ( overcloud-controller-cmm7-2 )
- Jul 19 15:01:05 overcloud-controller-cmm7-2 pengine[59755]: notice: * Recover ip-172.17.4.14 ( overcloud-controller-cmm7-2 )
- Jul 19 15:01:05 overcloud-controller-cmm7-2 crmd[59756]: notice: Initiating stop operation ip-172.17.1.10_stop_0 locally on overcloud-controller-cmm7-2
- Jul 19 15:01:05 overcloud-controller-cmm7-2 pengine[59755]: notice: Calculated transition 765, saving inputs in /var/lib/pacemaker/pengine/pe-input-111.bz2
- Jul 19 15:01:05 overcloud-controller-cmm7-2 crmd[59756]: notice: Initiating stop operation ip-172.17.4.14_stop_0 locally on overcloud-controller-cmm7-2
- Jul 19 15:01:05 overcloud-controller-cmm7-2 IPaddr2(ip-172.17.4.14)[710390]: INFO: IP status = ok, IP_CIP=
- Jul 19 15:01:05 overcloud-controller-cmm7-2 IPaddr2(ip-172.17.1.10)[710389]: INFO: IP status = ok, IP_CIP=
- Jul 19 15:01:05 overcloud-controller-cmm7-2 crmd[59756]: notice: Result of stop operation for ip-172.17.1.10 on overcloud-controller-cmm7-2: 0 (ok)
- Jul 19 15:01:05 overcloud-controller-cmm7-2 crmd[59756]: notice: Result of stop operation for ip-172.17.4.14 on overcloud-controller-cmm7-2: 0 (ok)
- Jul 19 15:01:05 overcloud-controller-cmm7-2 crmd[59756]: notice: Initiating start operation ip-172.17.1.10_start_0 locally on overcloud-controller-cmm7-2
- Jul 19 15:01:05 overcloud-controller-cmm7-2 crmd[59756]: notice: Initiating start operation ip-172.17.4.14_start_0 locally on overcloud-controller-cmm7-2
- Jul 19 15:01:05 overcloud-controller-cmm7-2 IPaddr2(ip-172.17.1.10)[710501]: INFO: Adding inet address 172.17.1.10/32 with broadcast address 172.17.1.255 to device vlan141
- Jul 19 15:01:05 overcloud-controller-cmm7-2 IPaddr2(ip-172.17.4.14)[710502]: INFO: Adding inet address 172.17.4.14/32 with broadcast address 172.17.4.255 to device vlan157
- Jul 19 15:01:05 overcloud-controller-cmm7-2 IPaddr2(ip-172.17.1.10)[710501]: INFO: Bringing device vlan141 up
- Jul 19 15:01:05 overcloud-controller-cmm7-2 IPaddr2(ip-172.17.4.14)[710502]: INFO: Bringing device vlan157 up
- .......
- Jul 19 22:31:33 overcloud-controller-cmm7-2 pengine[59755]: warning: Processing failed monitor of ip-172.31.0.24 on overcloud-controller-cmm7-0: unknown error
- Jul 19 22:31:33 overcloud-controller-cmm7-2 pengine[59755]: warning: Processing failed monitor of ip-135.111.247.193 on overcloud-controller-cmm7-0: unknown error
- Jul 19 22:31:33 overcloud-controller-cmm7-2 pengine[59755]: warning: Processing failed monitor of ip-172.17.1.23 on overcloud-controller-cmm7-0: unknown error
- Jul 19 22:31:33 overcloud-controller-cmm7-2 pengine[59755]: warning: Processing failed monitor of ip-172.17.1.10 on overcloud-controller-cmm7-2: unknown error
- Jul 19 22:31:33 overcloud-controller-cmm7-2 pengine[59755]: warning: Processing failed monitor of ip-172.17.4.14 on overcloud-controller-cmm7-2: unknown error
- Jul 19 22:31:33 overcloud-controller-cmm7-2 pengine[59755]: warning: Processing failed monitor of rabbitmq:0 on rabbitmq-bundle-0: unknown error
- Jul 19 22:31:33 overcloud-controller-cmm7-2 pengine[59755]: notice: Calculated transition 801, saving inputs in /var/lib/pacemaker/pengine/pe-input-120.bz2
- Jul 19 22:31:33 overcloud-controller-cmm7-2 crmd[59756]: notice: Transition 801 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-120.bz2)
- Jul 19 22:31:33 overcloud-controller-cmm7-2 crmd[59756]: notice: State transition S_TRANSITION_ENGINE -> S_IDLE
- Jul 19 22:41:12 overcloud-controller-cmm7-2 auditd[18353]: Audit daemon rotating log files
- Jul 19 22:46:33 overcloud-controller-cmm7-2 crmd[59756]: notice: State transition S_IDLE -> S_POLICY_ENGINE
- sriovperformancecompute-cmm7-7:
- [Fri Jul 19 17:03:25 2019] mce: [Hardware Error]: Machine check events logged
- [Fri Jul 19 17:03:25 2019] EDAC sbridge MC0: HANDLING MCE MEMORY ERROR
- [Fri Jul 19 17:03:25 2019] EDAC sbridge MC0: CPU 12: Machine Check Event: 0 Bank 14: 8c000048000800c1
- [Fri Jul 19 17:03:25 2019] EDAC sbridge MC0: TSC 0
- [Fri Jul 19 17:03:25 2019] EDAC sbridge MC0: ADDR 28f9a3b000
- [Fri Jul 19 17:03:25 2019] EDAC sbridge MC0: MISC 122960002000108c
- [Fri Jul 19 17:03:25 2019] EDAC sbridge MC0: PROCESSOR 0:306f2 TIME 1563573817 SOCKET 1 APIC 20
- [Fri Jul 19 17:03:26 2019] EDAC MC2: 1 CE memory scrubbing error on CPU_SrcID#1_Ha#1_Chan#1_DIMM#1 (channel:1 slot:1 page:0x28f9a3b offset:0x0 grain:32 syndrome:0x0 - area:DRAM err_code:0008:00c1 socket:1 ha:1 channel_mask:2 rank:5)
- [Sun Jul 21 14:40:31 2019] mce: [Hardware Error]: Machine check events logged
- [Sun Jul 21 14:40:31 2019] EDAC sbridge MC0: HANDLING MCE MEMORY ERROR
- [Sun Jul 21 14:40:31 2019] EDAC sbridge MC0: CPU 12: Machine Check Event: 0 Bank 14: 8c000048000800c1
- [Sun Jul 21 14:40:31 2019] EDAC sbridge MC0: TSC 0
- [Sun Jul 21 14:40:31 2019] EDAC sbridge MC0: ADDR 28f9a3b000
- [Sun Jul 21 14:40:31 2019] EDAC sbridge MC0: MISC 122960002000108c
- [Sun Jul 21 14:40:31 2019] EDAC sbridge MC0: PROCESSOR 0:306f2 TIME 1563738045 SOCKET 1 APIC 20
- [Sun Jul 21 14:40:32 2019] EDAC MC2: 1 CE memory scrubbing error on CPU_SrcID#1_Ha#1_Chan#1_DIMM#1 (channel:1 slot:1 page:0x28f9a3b offset:0x0 grain:32 syndrome:0x0 - area:DRAM err_code:0008:00c1 socket:1 ha:1 channel_mask:2 rank:5)
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement