Advertisement
Guest User

Untitled

a guest
Jul 22nd, 2019
146
0
Never
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 16.68 KB | None | 0 0
  1. Controller-0
  2.  
  3. Jul 19 15:49:35 overcloud-controller-cmm7-0 lrmd[59785]: warning: ip-172.17.1.23_monitor_10000 process (PID 799302) timed out
  4. Jul 19 15:49:35 overcloud-controller-cmm7-0 lrmd[59785]: warning: ip-172.17.1.23_monitor_10000:799302 - timed out after 20000ms
  5. Jul 19 15:49:35 overcloud-controller-cmm7-0 lrmd[59785]: warning: ip-135.111.247.193_monitor_10000 process (PID 799303) timed out
  6. Jul 19 15:49:35 overcloud-controller-cmm7-0 lrmd[59785]: warning: ip-172.31.0.24_monitor_10000 process (PID 799304) timed out
  7. Jul 19 15:49:35 overcloud-controller-cmm7-0 lrmd[59785]: warning: ip-135.111.247.193_monitor_10000:799303 - timed out after 20000ms
  8. Jul 19 15:49:35 overcloud-controller-cmm7-0 crmd[59788]: error: Result of monitor operation for ip-172.17.1.23 on overcloud-controller-cmm7-0: Timed Out
  9. Jul 19 15:49:35 overcloud-controller-cmm7-0 crmd[59788]: error: Result of monitor operation for ip-135.111.247.193 on overcloud-controller-cmm7-0: Timed Out
  10. Jul 19 15:49:36 overcloud-controller-cmm7-0 lrmd[59785]: warning: ip-172.31.0.24_monitor_10000:799304 - timed out after 20000ms
  11. Jul 19 15:49:36 overcloud-controller-cmm7-0 crmd[59788]: error: Result of monitor operation for ip-172.31.0.24 on overcloud-controller-cmm7-0: Timed Out
  12. Jul 19 15:49:36 overcloud-controller-cmm7-0 snmptrapd[665777]: 2019-07-19 15:49:36 localhost [UDP: [127.0.0.1]:1161->[127.0.0.1]:162]:#012DISMAN-EVENT-MIB::sysUpTimeInstance = Timeticks: (58
  13. Jul 19 15:49:36 overcloud-controller-cmm7-0 snmptrapd[665777]: 2019-07-19 15:49:36 localhost [UDP: [127.0.0.1]:1161->[127.0.0.1]:162]:#012DISMAN-EVENT-MIB::sysUpTimeInstance = Timeticks: (58
  14. Jul 19 15:49:36 overcloud-controller-cmm7-0 IPaddr2(ip-172.17.1.23)[799988]: INFO: IP status = ok, IP_CIP=
  15. Jul 19 15:49:36 overcloud-controller-cmm7-0 crmd[59788]: notice: Result of stop operation for ip-172.17.1.23 on overcloud-controller-cmm7-0: 0 (ok)
  16. Jul 19 15:49:37 overcloud-controller-cmm7-0 IPaddr2(ip-172.17.1.23)[800505]: INFO: Adding inet address 172.17.1.23/32 with broadcast address 172.17.1.255 to device vlan141
  17. Jul 19 15:49:37 overcloud-controller-cmm7-0 IPaddr2(ip-172.17.1.23)[800505]: INFO: Bringing device vlan141 up
  18. Jul 19 15:49:37 overcloud-controller-cmm7-0 IPaddr2(ip-172.17.1.23)[800505]: INFO: /usr/libexec/heartbeat/send_arp -i 200 -r 5 -p /var/run/resource-agents/send_arp-172.17.1.23 vlan141 172.1
  19. Jul 19 15:49:37 overcloud-controller-cmm7-0 crmd[59788]: notice: Result of start operation for ip-172.17.1.23 on overcloud-controller-cmm7-0: 0 (ok)
  20. Jul 19 15:49:41 overcloud-controller-cmm7-0 IPaddr2(ip-172.17.1.23)[800505]: INFO: ARPING 172.17.1.23 from 172.17.1.23 vlan141#012Sent 5 probes (5 broadcast(s))#012Received 0 response(s)
  21. Jul 19 15:49:41 overcloud-controller-cmm7-0 crmd[59788]: notice: Result of stop operation for zabbix-server on overcloud-controller-cmm7-0: 0 (ok)
  22. Jul 19 15:49:48 overcloud-controller-cmm7-0 snmptrapd[665777]: NET-SNMP version 5.7.2 AgentX subagent connected
  23. Jul 19 15:50:58 overcloud-controller-cmm7-0 crmd[59788]: notice: Result of stop operation for alarm-manager on overcloud-controller-cmm7-0: 0 (ok)
  24. Jul 19 15:50:59 overcloud-controller-cmm7-0 crmd[59788]: notice: Result of stop operation for cbis-vitrage-graph on overcloud-controller-cmm7-0: 0 (ok)
  25. Jul 19 15:50:59 overcloud-controller-cmm7-0 lrmd[59785]: notice: alarm-manager_start_0:813246:stderr [ Error: No such object: alarm-manager ]
  26. Jul 19 15:50:59 overcloud-controller-cmm7-0 lrmd[59785]: notice: alarm-manager_start_0:813246:stderr [ Error: No such object: alarm-manager ]
  27. Jul 19 15:51:00 overcloud-controller-cmm7-0 crmd[59788]: notice: Result of start operation for alarm-manager on overcloud-controller-cmm7-0: 0 (ok)
  28. Jul 19 15:51:00 overcloud-controller-cmm7-0 lrmd[59785]: notice: cbis-vitrage-graph_start_0:813247:stderr [ Error: No such object: cbis-vitrage-graph ]
  29. Jul 19 15:51:00 overcloud-controller-cmm7-0 lrmd[59785]: notice: cbis-vitrage-graph_start_0:813247:stderr [ Error: No such object: cbis-vitrage-graph ]
  30. Jul 19 15:51:00 overcloud-controller-cmm7-0 crmd[59788]: notice: Result of start operation for cbis-vitrage-graph on overcloud-controller-cmm7-0: 0 (ok)
  31. Jul 19 15:51:00 overcloud-controller-cmm7-0 lrmd[59785]: notice: zabbix-server_start_0:813245:stderr [ Error: No such object: zabbix-server ]
  32. Jul 19 15:51:00 overcloud-controller-cmm7-0 lrmd[59785]: notice: zabbix-server_start_0:813245:stderr [ Error: No such object: zabbix-server ]
  33. Jul 19 15:51:00 overcloud-controller-cmm7-0 crmd[59788]: notice: Result of start operation for zabbix-server on overcloud-controller-cmm7-0: 0 (ok)
  34.  
  35. Jul 19 20:58:22 overcloud-controller-cmm7-0 pacemaker_remoted[8]: warning: Could not notify client proxy-cib_rw-988636-f4d0e12b/f4d0e12b-c22e-4ed0-814d-b64c6fa47991: Broken pipe
  36. Jul 19 20:59:54 overcloud-controller-cmm7-0 snmptrapd[665777]: 2019-07-19 20:59:54 localhost [UDP: [127.0.0.1]:1161->[127.0.0.1]:162]:#012DISMAN-EVENT-MIB::sysUpTimeInstance = Timeticks: (18
  37. Jul 19 20:59:54 overcloud-controller-cmm7-0 snmptrapd[665777]: 2019-07-19 20:59:54 localhost [UDP: [127.0.0.1]:1161->[127.0.0.1]:162]:#012DISMAN-EVENT-MIB::sysUpTimeInstance = Timeticks: (18
  38. Jul 19 21:00:52 overcloud-controller-cmm7-0 rabbitmq-cluster(rabbitmq)[1000821]: INFO: RabbitMQ is a minority partition, failing monitor
  39. Jul 19 21:00:54 overcloud-controller-cmm7-0 crmd[59788]: notice: Result of notify operation for rabbitmq on rabbitmq-bundle-0: 0 (ok)
  40. Jul 19 21:01:04 overcloud-controller-cmm7-0 pacemaker_remoted[8]: notice: rabbitmq_stop_0:1001322:stderr [ Error: unable to connect to node 'rabbit@overcloud-controller-cmm7-0': nodedown ]
  41. Jul 19 21:01:04 overcloud-controller-cmm7-0 pacemaker_remoted[8]: notice: rabbitmq_stop_0:1001322:stderr [ ]
  42. Jul 19 21:01:04 overcloud-controller-cmm7-0 pacemaker_remoted[8]: notice: rabbitmq_stop_0:1001322:stderr [ DIAGNOSTICS ]
  43. Jul 19 21:01:04 overcloud-controller-cmm7-0 pacemaker_remoted[8]: notice: rabbitmq_stop_0:1001322:stderr [ =========== ]
  44. Jul 19 21:01:04 overcloud-controller-cmm7-0 pacemaker_remoted[8]: notice: rabbitmq_stop_0:1001322:stderr [ ]
  45. Jul 19 21:01:04 overcloud-controller-cmm7-0 pacemaker_remoted[8]: notice: rabbitmq_stop_0:1001322:stderr [ attempted to contact: ['rabbit@overcloud-controller-cmm7-0'] ]
  46. Jul 19 21:01:04 overcloud-controller-cmm7-0 pacemaker_remoted[8]: notice: rabbitmq_stop_0:1001322:stderr [ ]
  47. Jul 19 21:01:04 overcloud-controller-cmm7-0 pacemaker_remoted[8]: notice: rabbitmq_stop_0:1001322:stderr [ rabbit@overcloud-controller-cmm7-0: ]
  48. Jul 19 21:01:04 overcloud-controller-cmm7-0 pacemaker_remoted[8]: notice: rabbitmq_stop_0:1001322:stderr [ * connected to epmd (port 4369) on overcloud-controller-cmm7-0 ]
  49. Jul 19 21:01:04 overcloud-controller-cmm7-0 pacemaker_remoted[8]: notice: rabbitmq_stop_0:1001322:stderr [ * epmd reports: node 'rabbit' not running at all ]
  50. Jul 19 21:01:04 overcloud-controller-cmm7-0 pacemaker_remoted[8]: notice: rabbitmq_stop_0:1001322:stderr [ no other nodes on overcloud-controller-cmm7-0 ]
  51. Jul 19 21:01:04 overcloud-controller-cmm7-0 pacemaker_remoted[8]: notice: rabbitmq_stop_0:1001322:stderr [ * suggestion: start the node ]
  52. Jul 19 21:01:04 overcloud-controller-cmm7-0 pacemaker_remoted[8]: notice: rabbitmq_stop_0:1001322:stderr [ ]
  53. Jul 19 21:01:04 overcloud-controller-cmm7-0 pacemaker_remoted[8]: notice: rabbitmq_stop_0:1001322:stderr [ current node details: ]
  54. Jul 19 21:01:04 overcloud-controller-cmm7-0 pacemaker_remoted[8]: notice: rabbitmq_stop_0:1001322:stderr [ - node name: 'rabbitmq-cli-52@overcloud-controller-cmm7-0' ]
  55. Jul 19 21:01:04 overcloud-controller-cmm7-0 pacemaker_remoted[8]: notice: rabbitmq_stop_0:1001322:stderr [ - home dir: /var/lib/rabbitmq ]
  56. Jul 19 21:01:04 overcloud-controller-cmm7-0 pacemaker_remoted[8]: notice: rabbitmq_stop_0:1001322:stderr [ - cookie hash: 6iGxrPWDr6/m4PB15wvwkA== ]
  57. Jul 19 21:01:04 overcloud-controller-cmm7-0 pacemaker_remoted[8]: notice: rabbitmq_stop_0:1001322:stderr [ ]
  58. Jul 19 21:01:04 overcloud-controller-cmm7-0 crmd[59788]: notice: Result of stop operation for rabbitmq on rabbitmq-bundle-0: 0 (ok)
  59. Jul 19 21:01:14 overcloud-controller-cmm7-0 rabbitmq-cluster(rabbitmq)[1002270]: INFO: RabbitMQ server could not get cluster status from mnesia
  60. Jul 19 21:01:15 overcloud-controller-cmm7-0 rabbitmq-cluster(rabbitmq)[1002270]: INFO: wiping data directory before joining
  61. Jul 19 21:01:17 overcloud-controller-cmm7-0 rabbitmq-cluster(rabbitmq)[1002270]: INFO: Forgetting rabbit@overcloud-controller-cmm7-0 via nodes [ rabbit@overcloud-controller-cmm7-1 rabbit@ove
  62. Jul 19 21:01:17 overcloud-controller-cmm7-0 rabbitmq-cluster(rabbitmq)[1002270]: ERROR: Failed to forget node rabbit@overcloud-controller-cmm7-0 via rabbit@overcloud-controller-cmm7-1.
  63. Jul 19 21:01:18 overcloud-controller-cmm7-0 rabbitmq-cluster(rabbitmq)[1002270]: ERROR: Failed to forget node rabbit@overcloud-controller-cmm7-0 via rabbit@overcloud-controller-cmm7-2.
  64. Jul 19 21:01:18 overcloud-controller-cmm7-0 rabbitmq-cluster(rabbitmq)[1002270]: INFO: Joining existing cluster with [ rabbit@overcloud-controller-cmm7-1 rabbit@overcloud-controller-cmm7-2
  65. Jul 19 21:01:18 overcloud-controller-cmm7-0 rabbitmq-cluster(rabbitmq)[1002270]: INFO: Waiting for server to start
  66.  
  67.  
  68.  
  69.  
  70. Controller-2
  71. Jul 19 15:00:55 overcloud-controller-cmm7-2 snmptrapd[649113]: AgentX master agent failed to respond to ping. Attempting to re-register.
  72. Jul 19 15:01:01 overcloud-controller-cmm7-2 lrmd[59753]: warning: ip-172.17.4.14_monitor_10000 process (PID 709649) timed out
  73. Jul 19 15:01:01 overcloud-controller-cmm7-2 lrmd[59753]: warning: ip-172.17.4.14_monitor_10000:709649 - timed out after 20000ms
  74. Jul 19 15:01:01 overcloud-controller-cmm7-2 lrmd[59753]: warning: ip-172.17.1.10_monitor_10000 process (PID 709650) timed out
  75. Jul 19 15:01:01 overcloud-controller-cmm7-2 crmd[59756]: error: Result of monitor operation for ip-172.17.4.14 on overcloud-controller-cmm7-2: Timed Out
  76. Jul 19 15:01:01 overcloud-controller-cmm7-2 crmd[59756]: notice: State transition S_IDLE -> S_POLICY_ENGINE
  77. Jul 19 15:01:01 overcloud-controller-cmm7-2 lrmd[59753]: warning: ip-172.17.1.10_monitor_10000:709650 - timed out after 20000ms
  78. Jul 19 15:01:01 overcloud-controller-cmm7-2 crmd[59756]: error: Result of monitor operation for ip-172.17.1.10 on overcloud-controller-cmm7-2: Timed Out
  79. Jul 19 15:01:05 overcloud-controller-cmm7-2 pengine[59755]: warning: Processing failed monitor of ip-172.17.1.10 on overcloud-controller-cmm7-2: unknown error
  80. Jul 19 15:01:05 overcloud-controller-cmm7-2 pengine[59755]: warning: Processing failed monitor of ip-172.17.4.14 on overcloud-controller-cmm7-2: unknown error
  81. Jul 19 15:01:05 overcloud-controller-cmm7-2 pengine[59755]: notice: * Recover ip-172.17.1.10 ( overcloud-controller-cmm7-2 )
  82. Jul 19 15:01:05 overcloud-controller-cmm7-2 pengine[59755]: notice: * Recover ip-172.17.4.14 ( overcloud-controller-cmm7-2 )
  83. Jul 19 15:01:05 overcloud-controller-cmm7-2 pengine[59755]: notice: Calculated transition 764, saving inputs in /var/lib/pacemaker/pengine/pe-input-111.bz2
  84. Jul 19 15:01:05 overcloud-controller-cmm7-2 pengine[59755]: warning: Processing failed monitor of ip-172.17.1.10 on overcloud-controller-cmm7-2: unknown error
  85. Jul 19 15:01:05 overcloud-controller-cmm7-2 pengine[59755]: warning: Processing failed monitor of ip-172.17.4.14 on overcloud-controller-cmm7-2: unknown error
  86. Jul 19 15:01:05 overcloud-controller-cmm7-2 pengine[59755]: notice: * Recover ip-172.17.1.10 ( overcloud-controller-cmm7-2 )
  87. Jul 19 15:01:05 overcloud-controller-cmm7-2 pengine[59755]: notice: * Recover ip-172.17.4.14 ( overcloud-controller-cmm7-2 )
  88. Jul 19 15:01:05 overcloud-controller-cmm7-2 crmd[59756]: notice: Initiating stop operation ip-172.17.1.10_stop_0 locally on overcloud-controller-cmm7-2
  89. Jul 19 15:01:05 overcloud-controller-cmm7-2 pengine[59755]: notice: Calculated transition 765, saving inputs in /var/lib/pacemaker/pengine/pe-input-111.bz2
  90. Jul 19 15:01:05 overcloud-controller-cmm7-2 crmd[59756]: notice: Initiating stop operation ip-172.17.4.14_stop_0 locally on overcloud-controller-cmm7-2
  91. Jul 19 15:01:05 overcloud-controller-cmm7-2 IPaddr2(ip-172.17.4.14)[710390]: INFO: IP status = ok, IP_CIP=
  92. Jul 19 15:01:05 overcloud-controller-cmm7-2 IPaddr2(ip-172.17.1.10)[710389]: INFO: IP status = ok, IP_CIP=
  93. Jul 19 15:01:05 overcloud-controller-cmm7-2 crmd[59756]: notice: Result of stop operation for ip-172.17.1.10 on overcloud-controller-cmm7-2: 0 (ok)
  94. Jul 19 15:01:05 overcloud-controller-cmm7-2 crmd[59756]: notice: Result of stop operation for ip-172.17.4.14 on overcloud-controller-cmm7-2: 0 (ok)
  95. Jul 19 15:01:05 overcloud-controller-cmm7-2 crmd[59756]: notice: Initiating start operation ip-172.17.1.10_start_0 locally on overcloud-controller-cmm7-2
  96. Jul 19 15:01:05 overcloud-controller-cmm7-2 crmd[59756]: notice: Initiating start operation ip-172.17.4.14_start_0 locally on overcloud-controller-cmm7-2
  97. Jul 19 15:01:05 overcloud-controller-cmm7-2 IPaddr2(ip-172.17.1.10)[710501]: INFO: Adding inet address 172.17.1.10/32 with broadcast address 172.17.1.255 to device vlan141
  98. Jul 19 15:01:05 overcloud-controller-cmm7-2 IPaddr2(ip-172.17.4.14)[710502]: INFO: Adding inet address 172.17.4.14/32 with broadcast address 172.17.4.255 to device vlan157
  99. Jul 19 15:01:05 overcloud-controller-cmm7-2 IPaddr2(ip-172.17.1.10)[710501]: INFO: Bringing device vlan141 up
  100. Jul 19 15:01:05 overcloud-controller-cmm7-2 IPaddr2(ip-172.17.4.14)[710502]: INFO: Bringing device vlan157 up
  101.  
  102. .......
  103.  
  104. Jul 19 22:31:33 overcloud-controller-cmm7-2 pengine[59755]: warning: Processing failed monitor of ip-172.31.0.24 on overcloud-controller-cmm7-0: unknown error
  105. Jul 19 22:31:33 overcloud-controller-cmm7-2 pengine[59755]: warning: Processing failed monitor of ip-135.111.247.193 on overcloud-controller-cmm7-0: unknown error
  106. Jul 19 22:31:33 overcloud-controller-cmm7-2 pengine[59755]: warning: Processing failed monitor of ip-172.17.1.23 on overcloud-controller-cmm7-0: unknown error
  107. Jul 19 22:31:33 overcloud-controller-cmm7-2 pengine[59755]: warning: Processing failed monitor of ip-172.17.1.10 on overcloud-controller-cmm7-2: unknown error
  108. Jul 19 22:31:33 overcloud-controller-cmm7-2 pengine[59755]: warning: Processing failed monitor of ip-172.17.4.14 on overcloud-controller-cmm7-2: unknown error
  109. Jul 19 22:31:33 overcloud-controller-cmm7-2 pengine[59755]: warning: Processing failed monitor of rabbitmq:0 on rabbitmq-bundle-0: unknown error
  110. Jul 19 22:31:33 overcloud-controller-cmm7-2 pengine[59755]: notice: Calculated transition 801, saving inputs in /var/lib/pacemaker/pengine/pe-input-120.bz2
  111. Jul 19 22:31:33 overcloud-controller-cmm7-2 crmd[59756]: notice: Transition 801 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-120.bz2)
  112. Jul 19 22:31:33 overcloud-controller-cmm7-2 crmd[59756]: notice: State transition S_TRANSITION_ENGINE -> S_IDLE
  113. Jul 19 22:41:12 overcloud-controller-cmm7-2 auditd[18353]: Audit daemon rotating log files
  114. Jul 19 22:46:33 overcloud-controller-cmm7-2 crmd[59756]: notice: State transition S_IDLE -> S_POLICY_ENGINE
  115.  
  116.  
  117.  
  118.  
  119.  
  120. sriovperformancecompute-cmm7-7:
  121. [Fri Jul 19 17:03:25 2019] mce: [Hardware Error]: Machine check events logged
  122. [Fri Jul 19 17:03:25 2019] EDAC sbridge MC0: HANDLING MCE MEMORY ERROR
  123. [Fri Jul 19 17:03:25 2019] EDAC sbridge MC0: CPU 12: Machine Check Event: 0 Bank 14: 8c000048000800c1
  124. [Fri Jul 19 17:03:25 2019] EDAC sbridge MC0: TSC 0
  125. [Fri Jul 19 17:03:25 2019] EDAC sbridge MC0: ADDR 28f9a3b000
  126. [Fri Jul 19 17:03:25 2019] EDAC sbridge MC0: MISC 122960002000108c
  127. [Fri Jul 19 17:03:25 2019] EDAC sbridge MC0: PROCESSOR 0:306f2 TIME 1563573817 SOCKET 1 APIC 20
  128. [Fri Jul 19 17:03:26 2019] EDAC MC2: 1 CE memory scrubbing error on CPU_SrcID#1_Ha#1_Chan#1_DIMM#1 (channel:1 slot:1 page:0x28f9a3b offset:0x0 grain:32 syndrome:0x0 - area:DRAM err_code:0008:00c1 socket:1 ha:1 channel_mask:2 rank:5)
  129. [Sun Jul 21 14:40:31 2019] mce: [Hardware Error]: Machine check events logged
  130. [Sun Jul 21 14:40:31 2019] EDAC sbridge MC0: HANDLING MCE MEMORY ERROR
  131. [Sun Jul 21 14:40:31 2019] EDAC sbridge MC0: CPU 12: Machine Check Event: 0 Bank 14: 8c000048000800c1
  132. [Sun Jul 21 14:40:31 2019] EDAC sbridge MC0: TSC 0
  133. [Sun Jul 21 14:40:31 2019] EDAC sbridge MC0: ADDR 28f9a3b000
  134. [Sun Jul 21 14:40:31 2019] EDAC sbridge MC0: MISC 122960002000108c
  135. [Sun Jul 21 14:40:31 2019] EDAC sbridge MC0: PROCESSOR 0:306f2 TIME 1563738045 SOCKET 1 APIC 20
  136. [Sun Jul 21 14:40:32 2019] EDAC MC2: 1 CE memory scrubbing error on CPU_SrcID#1_Ha#1_Chan#1_DIMM#1 (channel:1 slot:1 page:0x28f9a3b offset:0x0 grain:32 syndrome:0x0 - area:DRAM err_code:0008:00c1 socket:1 ha:1 channel_mask:2 rank:5)
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement