API tools faq
paste
Advertisement
Guest User

MariaDB log for controller 2

a guest
Apr 26th, 2023
237
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 46.59 KB | None | 0 0
raw download clone embed print report
  1. 230414 08:48:21 mysqld_safe Starting mariadbd daemon with databases from /var/lib/mysql/
  2. 230414 08:48:21 mysqld_safe WSREP: Running position recovery with --disable-log-error --pid-file='/var/lib/mysql//controller2-recover.pid'
  3. 230414 08:48:21 mysqld_safe WSREP: Recovered position 00000000-0000-0000-0000-000000000000:-1
  4. 2023-04-14 8:48:21 0 [Note] Starting MariaDB 10.6.12-MariaDB-1:10.6.12+maria~ubu2204-log source revision 4c79e15cc3716f69c044d4287ad2160da8101cdc as process 224
  5. 2023-04-14 8:48:21 0 [Note] WSREP: Loading provider /usr/lib/galera/libgalera_smm.so initial position: 00000000-0000-0000-0000-000000000000:-1
  6. 2023-04-14 8:48:21 0 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib/galera/libgalera_smm.so'
  7. 2023-04-14 8:48:21 0 [Note] WSREP: wsrep_load(): Galera 26.4.14(r06a0c285) by Codership Oy <[email protected]> loaded successfully.
  8. 2023-04-14 8:48:21 0 [Note] WSREP: CRC-32C: using "slicing-by-8" algorithm.
  9. 2023-04-14 8:48:21 0 [Warning] WSREP: Could not open state file for reading: '/var/lib/mysql//grastate.dat'
  10. 2023-04-14 8:48:21 0 [Note] WSREP: Found saved state: 00000000-0000-0000-0000-000000000000:-1, safe_to_bootstrap: 1
  11. 2023-04-14 8:48:21 0 [Note] WSREP: GCache DEBUG: opened preamble:
  12. Version: 0
  13. UUID: 00000000-0000-0000-0000-000000000000
  14. Seqno: -1 - -1
  15. Offset: -1
  16. Synced: 0
  17. 2023-04-14 8:48:21 0 [Note] WSREP: Skipped GCache ring buffer recovery: could not determine history UUID.
  18. 2023-04-14 8:48:21 0 [Note] WSREP: Passing config to GCS: base_dir = /var/lib/mysql/; base_host = 10.0.2.1; base_port = 4567; cert.log_conflicts = no; cert.optimistic_pa = yes; debug = no; evs.auto_evict = 0; evs.delay_margin = PT1S; evs.delayed_keep_period = PT30S; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = 3; evs.send_window = 4; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.user_send_window = 2; evs.view_forget_timeout = PT24H; gcache.dir = /var/lib/mysql/; gcache.keep_pages_size = 0; gcache.keep_plaintext_size = 128M; gcache.mem_size = 0; gcache.name = galera.cache; gcache.page_size = 128M; gcache.recover = yes; gcache.size = 128M; gcomm.thread_prio = ; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 16; gcs.fc_master_slave = no; gcs.fc_single_primary = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.listen_addr = tc
  19. 2023-04-14 8:48:21 0 [Note] WSREP: Start replication
  20. 2023-04-14 8:48:21 0 [Note] WSREP: Connecting with bootstrap option: 0
  21. 2023-04-14 8:48:21 0 [Note] WSREP: Setting GCS initial position to 00000000-0000-0000-0000-000000000000:-1
  22. 2023-04-14 8:48:21 0 [Note] WSREP: Using CRC-32C for message checksums.
  23. 2023-04-14 8:48:21 0 [Note] WSREP: backend: asio
  24. 2023-04-14 8:48:21 0 [Note] WSREP: gcomm thread scheduling priority set to other:0
  25. 2023-04-14 8:48:21 0 [Note] WSREP: access file(/var/lib/mysql//gvwstate.dat) failed(No such file or directory)
  26. 2023-04-14 8:48:21 0 [Note] WSREP: restore pc from disk failed
  27. 2023-04-14 8:48:21 0 [Note] WSREP: GMCast version 0
  28. 2023-04-14 8:48:21 0 [Note] WSREP: (1c4b34e1-ab63, 'tcp://10.0.2.1:4567') listening at tcp://10.0.2.1:4567
  29. 2023-04-14 8:48:21 0 [Note] WSREP: (1c4b34e1-ab63, 'tcp://10.0.2.1:4567') multicast: , ttl: 1
  30. 2023-04-14 8:48:21 0 [Note] WSREP: EVS version 1
  31. 2023-04-14 8:48:21 0 [Note] WSREP: gcomm: connecting to group 'openstack', peer '10.0.1.1:4567,10.0.2.1:4567,10.0.3.1:4567'
  32. 2023-04-14 8:48:21 0 [Note] WSREP: (1c4b34e1-ab63, 'tcp://10.0.2.1:4567') connection established to 1660e5cc-ace5 tcp://10.0.1.1:4567
  33. 2023-04-14 8:48:21 0 [Note] WSREP: (1c4b34e1-ab63, 'tcp://10.0.2.1:4567') turning message relay requesting on, nonlive peers:
  34. 2023-04-14 8:48:22 0 [Note] WSREP: EVS version upgrade 0 -> 1
  35. 2023-04-14 8:48:22 0 [Note] WSREP: declaring 1660e5cc-ace5 at tcp://10.0.1.1:4567 stable
  36. 2023-04-14 8:48:22 0 [Note] WSREP: PC protocol upgrade 0 -> 1
  37. 2023-04-14 8:48:22 0 [Note] WSREP: Node 1660e5cc-ace5 state prim
  38. 2023-04-14 8:48:22 0 [Note] WSREP: view(view_id(PRIM,1660e5cc-ace5,2) memb {
  39. 1660e5cc-ace5,0
  40. 1c4b34e1-ab63,0
  41. } joined {
  42. } left {
  43. } partitioned {
  44. })
  45. 2023-04-14 8:48:22 0 [Note] WSREP: save pc into disk
  46. 2023-04-14 8:48:22 0 [Note] WSREP: discarding pending addr without UUID: tcp://10.0.3.1:4567
  47. 2023-04-14 8:48:22 0 [Note] WSREP: gcomm: connected
  48. 2023-04-14 8:48:22 0 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636
  49. 2023-04-14 8:48:22 0 [Note] WSREP: Shifting CLOSED -> OPEN (TO: 0)
  50. 2023-04-14 8:48:22 0 [Note] WSREP: Opened channel 'openstack'
  51. 2023-04-14 8:48:22 0 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 1, memb_num = 2
  52. 2023-04-14 8:48:22 0 [Note] WSREP: STATE EXCHANGE: Waiting for state UUID.
  53. 2023-04-14 8:48:22 1 [Note] WSREP: Starting rollbacker thread 1
  54. 2023-04-14 8:48:22 0 [Note] WSREP: STATE EXCHANGE: sent state msg: 1c976c68-daa1-11ed-b3ad-1a3d7c9457b5
  55. 2023-04-14 8:48:22 0 [Note] WSREP: STATE EXCHANGE: got state msg: 1c976c68-daa1-11ed-b3ad-1a3d7c9457b5 from 0 (controller1)
  56. 2023-04-14 8:48:22 2 [Note] WSREP: Starting applier thread 2
  57. 2023-04-14 8:48:22 0 [Note] WSREP: STATE EXCHANGE: got state msg: 1c976c68-daa1-11ed-b3ad-1a3d7c9457b5 from 1 (controller2)
  58. 2023-04-14 8:48:22 0 [Note] WSREP: Quorum results:
  59. version = 6,
  60. component = PRIMARY,
  61. conf_id = 1,
  62. members = 1/2 (joined/total),
  63. act_id = 1,
  64. last_appl. = 0,
  65. protocols = 2/10/4 (gcs/repl/appl),
  66. vote policy= 0,
  67. group UUID = 16614045-daa1-11ed-87d5-f6b7c891d885
  68. 2023-04-14 8:48:22 0 [Note] WSREP: Flow-control interval: [23, 23]
  69. 2023-04-14 8:48:22 0 [Note] WSREP: Shifting OPEN -> PRIMARY (TO: 2)
  70. 2023-04-14 8:48:22 2 [Note] WSREP: ####### processing CC 2, local, ordered
  71. 2023-04-14 8:48:22 2 [Note] WSREP: Process first view: 16614045-daa1-11ed-87d5-f6b7c891d885 my uuid: 1c4b34e1-daa1-11ed-ab63-dfd9034b5ed6
  72. 2023-04-14 8:48:22 2 [Note] WSREP: Server controller2 connected to cluster at position 16614045-daa1-11ed-87d5-f6b7c891d885:2 with ID 1c4b34e1-daa1-11ed-ab63-dfd9034b5ed6
  73. 2023-04-14 8:48:22 2 [Note] WSREP: Server status change disconnected -> connected
  74. 2023-04-14 8:48:22 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
  75. 2023-04-14 8:48:22 2 [Note] WSREP: ####### My UUID: 1c4b34e1-daa1-11ed-ab63-dfd9034b5ed6
  76. 2023-04-14 8:48:22 2 [Note] WSREP: Cert index reset to 00000000-0000-0000-0000-000000000000:-1 (proto: 10), state transfer needed: yes
  77. 2023-04-14 8:48:22 0 [Note] WSREP: Service thread queue flushed.
  78. 2023-04-14 8:48:22 2 [Note] WSREP: ####### Assign initial position for certification: 00000000-0000-0000-0000-000000000000:-1, protocol version: -1
  79. 2023-04-14 8:48:22 2 [Note] WSREP: State transfer required:
  80. Group state: 16614045-daa1-11ed-87d5-f6b7c891d885:2
  81. Local state: 00000000-0000-0000-0000-000000000000:-1
  82. 2023-04-14 8:48:22 2 [Note] WSREP: Server status change connected -> joiner
  83. 2023-04-14 8:48:22 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
  84. 2023-04-14 8:48:22 0 [Note] WSREP: Joiner monitor thread started to monitor
  85. 2023-04-14 8:48:22 0 [Note] WSREP: Running: 'wsrep_sst_mariabackup --role 'joiner' --address '10.0.2.1:4444' --datadir '/var/lib/mysql/' --parent 224 --progress 0 --binlog 'mysql-bin' --mysqld-args --basedir=/usr --datadir=/var/lib/mysql/ --plugin-dir=/usr/lib/mysql/plugin --wsrep_provider=/usr/lib/galera/libgalera_smm.so --wsrep_on=ON --log-error=/var/log/kolla/mariadb/mariadb.log --pid-file=/var/lib/mysql/mariadb.pid --port=3306 --wsrep_start_position=00000000-0000-0000-0000-000000000000:-1'
  86. WSREP_SST: [INFO] mariabackup SST started on joiner (20230414 08:48:22.775)
  87. WSREP_SST: [INFO] SSL configuration: CA='', CAPATH='', CERT='', KEY='', MODE='DISABLED', encrypt='0' (20230414 08:48:22.791)
  88. WSREP_SST: [INFO] Progress reporting tool pv not found in path: /usr//bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/sbin:/usr/bin:/sbin:/bin (20230414 08:48:22.828)
  89. WSREP_SST: [INFO] Disabling all progress/rate-limiting (20230414 08:48:22.829)
  90. WSREP_SST: [INFO] Streaming with mbstream (20230414 08:48:22.836)
  91. WSREP_SST: [INFO] Using socat as streamer (20230414 08:48:22.837)
  92. WSREP_SST: [INFO] Evaluating timeout -k 310 300 socat -u TCP-LISTEN:4444,reuseaddr stdio | '/usr//bin/mbstream' -x; RC=( ${PIPESTATUS[@]} ) (20230414 08:48:22.849)
  93. 2023-04-14 8:48:22 2 [Note] WSREP: ####### IST uuid:00000000-0000-0000-0000-000000000000 f: 0, l: 2, STRv: 3
  94. 2023-04-14 8:48:22 2 [Note] WSREP: IST receiver addr using tcp://10.0.2.1:4568
  95. 2023-04-14 8:48:22 2 [Note] WSREP: Prepared IST receiver for 0-2, listening at: tcp://10.0.2.1:4568
  96. 2023-04-14 8:48:22 0 [Note] WSREP: Member 1.0 (controller2) requested state transfer from '*any*'. Selected 0.0 (controller1)(SYNCED) as donor.
  97. 2023-04-14 8:48:22 0 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 2)
  98. 2023-04-14 8:48:22 2 [Note] WSREP: Requesting state transfer: success, donor: 0
  99. 2023-04-14 8:48:22 2 [Note] WSREP: Resetting GCache seqno map due to different histories.
  100. 2023-04-14 8:48:22 2 [Note] WSREP: GCache history reset: 00000000-0000-0000-0000-000000000000:0 -> 16614045-daa1-11ed-87d5-f6b7c891d885:2
  101. WSREP_SST: [INFO] Proceeding with SST (20230414 08:48:23.069)
  102. WSREP_SST: [INFO] Evaluating socat -u TCP-LISTEN:4444,reuseaddr stdio | '/usr//bin/mbstream' -x; RC=( ${PIPESTATUS[@]} ) (20230414 08:48:23.069)
  103. WSREP_SST: [INFO] Cleaning the old binary logs (20230414 08:48:23.076)
  104. WSREP_SST: [INFO] Cleaning the existing datadir and innodb-data/log directories (20230414 08:48:23.078)
  105. WSREP_SST: [INFO] Waiting for SST streaming to complete! (20230414 08:48:23.081)
  106. 2023-04-14 8:48:25 0 [Note] WSREP: (1c4b34e1-ab63, 'tcp://10.0.2.1:4567') turning message relay requesting off
  107. 2023-04-14 9:00:26 0 [Note] WSREP: (1c4b34e1-ab63, 'tcp://10.0.2.1:4567') connection established to cc70ed3a-8652 tcp://10.0.3.1:4567
  108. 2023-04-14 9:00:26 0 [Note] WSREP: (1c4b34e1-ab63, 'tcp://10.0.2.1:4567') turning message relay requesting on, nonlive peers:
  109. 2023-04-14 9:00:27 0 [Note] WSREP: declaring 1660e5cc-ace5 at tcp://10.0.1.1:4567 stable
  110. 2023-04-14 9:00:27 0 [Note] WSREP: declaring cc70ed3a-8652 at tcp://10.0.3.1:4567 stable
  111. 2023-04-14 9:00:27 0 [Note] WSREP: Node 1660e5cc-ace5 state prim
  112. 2023-04-14 9:00:27 0 [Note] WSREP: view(view_id(PRIM,1660e5cc-ace5,3) memb {
  113. 1660e5cc-ace5,0
  114. 1c4b34e1-ab63,0
  115. cc70ed3a-8652,0
  116. } joined {
  117. } left {
  118. } partitioned {
  119. })
  120. 2023-04-14 9:00:27 0 [Note] WSREP: save pc into disk
  121. 2023-04-14 9:00:27 0 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 1, memb_num = 3
  122. 2023-04-14 9:00:27 0 [Note] WSREP: STATE EXCHANGE: Waiting for state UUID.
  123. 2023-04-14 9:00:27 0 [Note] WSREP: STATE EXCHANGE: sent state msg: ccbde6f9-daa2-11ed-a7a1-df980a784101
  124. 2023-04-14 9:00:27 0 [Note] WSREP: STATE EXCHANGE: got state msg: ccbde6f9-daa2-11ed-a7a1-df980a784101 from 0 (controller1)
  125. 2023-04-14 9:00:27 0 [Note] WSREP: STATE EXCHANGE: got state msg: ccbde6f9-daa2-11ed-a7a1-df980a784101 from 1 (controller2)
  126. 2023-04-14 9:00:27 0 [Note] WSREP: STATE EXCHANGE: got state msg: ccbde6f9-daa2-11ed-a7a1-df980a784101 from 2 (controller3)
  127. 2023-04-14 9:00:27 0 [Note] WSREP: Quorum results:
  128. version = 6,
  129. component = PRIMARY,
  130. conf_id = 2,
  131. members = 1/3 (joined/total),
  132. act_id = 2,
  133. last_appl. = 0,
  134. protocols = 2/10/4 (gcs/repl/appl),
  135. vote policy= 0,
  136. group UUID = 16614045-daa1-11ed-87d5-f6b7c891d885
  137. 2023-04-14 9:00:27 0 [Note] WSREP: Flow-control interval: [28, 28]
  138. 2023-04-14 9:00:28 0 [Warning] WSREP: Member 2.0 (controller3) requested state transfer from '*any*', but it is impossible to select State Transfer donor: Resource temporarily unavailable
  139. 2023-04-14 9:00:29 0 [Warning] WSREP: Member 2.0 (controller3) requested state transfer from '*any*', but it is impossible to select State Transfer donor: Resource temporarily unavailable
  140. 2023-04-14 9:00:29 0 [Note] WSREP: (1c4b34e1-ab63, 'tcp://10.0.2.1:4567') turning message relay requesting off
  141. 2023-04-14 9:00:30 0 [Warning] WSREP: Member 2.0 (controller3) requested state transfer from '*any*', but it is impossible to select State Transfer donor: Resource temporarily unavailable
  142. 2023-04-14 9:00:31 0 [Warning] WSREP: Member 2.0 (controller3) requested state transfer from '*any*', but it is impossible to select State Transfer donor: Resource temporarily unavailable
  143. 2023-04-14 9:00:32 0 [Warning] WSREP: Member 2.0 (controller3) requested state transfer from '*any*', but it is impossible to select State Transfer donor: Resource temporarily unavailable
  144. 2023-04-14 9:00:33 0 [Warning] WSREP: Member 2.0 (controller3) requested state transfer from '*any*', but it is impossible to select State Transfer donor: Resource temporarily unavailable
  145. 2023-04-14 9:00:34 0 [Warning] WSREP: Member 2.0 (controller3) requested state transfer from '*any*', but it is impossible to select State Transfer donor: Resource temporarily unavailable
  146. 2023-04-14 9:00:35 0 [Warning] WSREP: Member 2.0 (controller3) requested state transfer from '*any*', but it is impossible to select State Transfer donor: Resource temporarily unavailable
  147. 2023-04-14 9:00:36 0 [Warning] WSREP: Member 2.0 (controller3) requested state transfer from '*any*', but it is impossible to select State Transfer donor: Resource temporarily unavailable
  148. 2023-04-14 9:00:37 0 [Warning] WSREP: Member 2.0 (controller3) requested state transfer from '*any*', but it is impossible to select State Transfer donor: Resource temporarily unavailable
  149. 2023-04-14 9:00:38 0 [Warning] WSREP: Member 2.0 (controller3) requested state transfer from '*any*', but it is impossible to select State Transfer donor: Resource temporarily unavailable
  150. 2023-04-14 9:00:39 0 [Warning] WSREP: Member 2.0 (controller3) requested state transfer from '*any*', but it is impossible to select State Transfer donor: Resource temporarily unavailable
  151. 2023-04-14 9:00:40 0 [Warning] WSREP: Member 2.0 (controller3) requested state transfer from '*any*', but it is impossible to select State Transfer donor: Resource temporarily unavailable
  152. 2023-04-14 9:00:41 0 [Warning] WSREP: Member 2.0 (controller3) requested state transfer from '*any*', but it is impossible to select State Transfer donor: Resource temporarily unavailable
  153. 2023-04-14 9:00:42 0 [Warning] WSREP: Member 2.0 (controller3) requested state transfer from '*any*', but it is impossible to select State Transfer donor: Resource temporarily unavailable
  154. ...
  155.  
  156. ...
  157. 2023-04-14 9:03:59 0 [Warning] WSREP: Member 2.0 (controller3) requested state transfer from '*any*', but it is impossible to select State Transfer donor: Resource temporarily unavailable
  158. 2023-04-14 9:04:00 0 [Warning] WSREP: Member 2.0 (controller3) requested state transfer from '*any*', but it is impossible to select State Transfer donor: Resource temporarily unavailable
  159. 2023-04-14 9:04:01 0 [Warning] WSREP: Member 2.0 (controller3) requested state transfer from '*any*', but it is impossible to select State Transfer donor: Resource temporarily unavailable
  160. 2023-04-14 9:04:02 0 [Warning] WSREP: Member 2.0 (controller3) requested state transfer from '*any*', but it is impossible to select State Transfer donor: Resource temporarily unavailable
  161. 2023-04-14 9:04:03 0 [Warning] WSREP: Member 2.0 (controller3) requested state transfer from '*any*', but it is impossible to select State Transfer donor: Resource temporarily unavailable
  162. 2023-04-14 9:04:04 0 [Warning] WSREP: Member 2.0 (controller3) requested state transfer from '*any*', but it is impossible to select State Transfer donor: Resource temporarily unavailable
  163. 2023-04-14 9:04:05 0 [Warning] WSREP: Member 2.0 (controller3) requested state transfer from '*any*', but it is impossible to select State Transfer donor: Resource temporarily unavailable
  164. 2023-04-14 9:04:06 0 [Warning] WSREP: Member 2.0 (controller3) requested state transfer from '*any*', but it is impossible to select State Transfer donor: Resource temporarily unavailable
  165. 2023-04-14 9:04:07 0 [Warning] WSREP: Member 2.0 (controller3) requested state transfer from '*any*', but it is impossible to select State Transfer donor: Resource temporarily unavailable
  166. 2023-04-14 9:04:08 0 [Warning] WSREP: Member 2.0 (controller3) requested state transfer from '*any*', but it is impossible to select State Transfer donor: Resource temporarily unavailable
  167. 2023-04-14 9:04:09 0 [Warning] WSREP: Member 2.0 (controller3) requested state transfer from '*any*', but it is impossible to select State Transfer donor: Resource temporarily unavailable
  168. 2023-04-14 9:04:10 0 [Warning] WSREP: Member 2.0 (controller3) requested state transfer from '*any*', but it is impossible to select State Transfer donor: Resource temporarily unavailable
  169. 2023-04-14 9:04:11 0 [Warning] WSREP: Member 2.0 (controller3) requested state transfer from '*any*', but it is impossible to select State Transfer donor: Resource temporarily unavailable
  170. 2023-04-14 9:04:11 0 [Warning] WSREP: 0.0 (controller1): State transfer to 1.0 (controller2) failed: -22 (Invalid argument)
  171. 2023-04-14 9:04:11 0 [ERROR] WSREP: ./gcs/src/gcs_group.cpp:gcs_group_handle_join_msg():1207: Will never receive state. Need to abort.
  172. 2023-04-14 9:04:11 0 [Note] WSREP: gcomm: terminating thread
  173. 2023-04-14 9:04:11 0 [Note] WSREP: gcomm: joining thread
  174. 2023-04-14 9:04:11 0 [Note] WSREP: gcomm: closing backend
  175. 2023-04-14 9:04:11 0 [Note] WSREP: view(view_id(NON_PRIM,1660e5cc-ace5,3) memb {
  176. 1c4b34e1-ab63,0
  177. } joined {
  178. } left {
  179. } partitioned {
  180. 1660e5cc-ace5,0
  181. cc70ed3a-8652,0
  182. })
  183. 2023-04-14 9:04:11 0 [Note] WSREP: PC protocol downgrade 1 -> 0
  184. 2023-04-14 9:04:11 0 [Note] WSREP: view((empty))
  185. 2023-04-14 9:04:11 0 [Note] WSREP: gcomm: closed
  186. 2023-04-14 9:04:11 0 [Note] WSREP: /usr/sbin/mariadbd: Terminated.
  187. 230414 09:04:11 mysqld_safe mysqld from pid file /var/lib/mysql/mariadb.pid ended
  188. Terminated
  189. 230414 09:04:22 mysqld_safe Starting mariadbd daemon with databases from /var/lib/mysql/
  190. 230414 09:04:22 mysqld_safe WSREP: Running position recovery with --disable-log-error --pid-file='/var/lib/mysql//controller2-recover.pid'
  191. 230414 09:04:23 mysqld_safe WSREP: Recovered position 00000000-0000-0000-0000-000000000000:-1
  192. 2023-04-14 9:04:23 0 [Note] Starting MariaDB 10.6.12-MariaDB-1:10.6.12+maria~ubu2204-log source revision 4c79e15cc3716f69c044d4287ad2160da8101cdc as process 223
  193. 2023-04-14 9:04:23 0 [Note] WSREP: Loading provider /usr/lib/galera/libgalera_smm.so initial position: 00000000-0000-0000-0000-000000000000:-1
  194. 2023-04-14 9:04:23 0 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib/galera/libgalera_smm.so'
  195. 2023-04-14 9:04:23 0 [Note] WSREP: wsrep_load(): Galera 26.4.14(r06a0c285) by Codership Oy <[email protected]> loaded successfully.
  196. 2023-04-14 9:04:23 0 [Note] WSREP: CRC-32C: using "slicing-by-8" algorithm.
  197. 2023-04-14 9:04:23 0 [Note] WSREP: Found saved state: 00000000-0000-0000-0000-000000000000:-1, safe_to_bootstrap: 1
  198. 2023-04-14 9:04:23 0 [Note] WSREP: GCache DEBUG: opened preamble:
  199. Version: 2
  200. UUID: 16614045-daa1-11ed-87d5-f6b7c891d885
  201. Seqno: -1 - -1
  202. Offset: -1
  203. Synced: 0
  204. 2023-04-14 9:04:23 0 [Note] WSREP: Recovering GCache ring buffer: version: 2, UUID: 16614045-daa1-11ed-87d5-f6b7c891d885, offset: -1
  205. 2023-04-14 9:04:23 0 [Note] WSREP: GCache::RingBuffer initial scan... 0.0% ( 0/134217752 bytes) complete.
  206. 2023-04-14 9:04:23 0 [Note] WSREP: GCache::RingBuffer initial scan...100.0% (134217752/134217752 bytes) complete.
  207. 2023-04-14 9:04:23 0 [Note] WSREP: Recovering GCache ring buffer: Recovery failed, need to do full reset.
  208. 2023-04-14 9:04:23 0 [Note] WSREP: Passing config to GCS: base_dir = /var/lib/mysql/; base_host = 10.0.2.1; base_port = 4567; cert.log_conflicts = no; cert.optimistic_pa = yes; debug = no; evs.auto_evict = 0; evs.delay_margin = PT1S; evs.delayed_keep_period = PT30S; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = 3; evs.send_window = 4; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.user_send_window = 2; evs.view_forget_timeout = PT24H; gcache.dir = /var/lib/mysql/; gcache.keep_pages_size = 0; gcache.keep_plaintext_size = 128M; gcache.mem_size = 0; gcache.name = galera.cache; gcache.page_size = 128M; gcache.recover = yes; gcache.size = 128M; gcomm.thread_prio = ; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 16; gcs.fc_master_slave = no; gcs.fc_single_primary = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.listen_addr = tc
  209. 2023-04-14 9:04:23 0 [Note] WSREP: Start replication
  210. 2023-04-14 9:04:23 0 [Note] WSREP: Connecting with bootstrap option: 0
  211. 2023-04-14 9:04:23 0 [Note] WSREP: Setting GCS initial position to 00000000-0000-0000-0000-000000000000:-1
  212. 2023-04-14 9:04:23 0 [Note] WSREP: Using CRC-32C for message checksums.
  213. 2023-04-14 9:04:23 0 [Note] WSREP: backend: asio
  214. 2023-04-14 9:04:23 0 [Note] WSREP: gcomm thread scheduling priority set to other:0
  215. 2023-04-14 9:04:23 0 [Note] WSREP: access file(/var/lib/mysql//gvwstate.dat) failed(No such file or directory)
  216. 2023-04-14 9:04:23 0 [Note] WSREP: restore pc from disk failed
  217. 2023-04-14 9:04:23 0 [Note] WSREP: GMCast version 0
  218. 2023-04-14 9:04:23 0 [Note] WSREP: (595eed12-a6f0, 'tcp://10.0.2.1:4567') listening at tcp://10.0.2.1:4567
  219. 2023-04-14 9:04:23 0 [Note] WSREP: (595eed12-a6f0, 'tcp://10.0.2.1:4567') multicast: , ttl: 1
  220. 2023-04-14 9:04:23 0 [Note] WSREP: EVS version 1
  221. 2023-04-14 9:04:23 0 [Note] WSREP: gcomm: connecting to group 'openstack', peer '10.0.1.1:4567,10.0.2.1:4567,10.0.3.1:4567'
  222. 2023-04-14 9:04:23 0 [Note] WSREP: (595eed12-a6f0, 'tcp://10.0.2.1:4567') connection established to 1660e5cc-ace5 tcp://10.0.1.1:4567
  223. 2023-04-14 9:04:23 0 [Note] WSREP: (595eed12-a6f0, 'tcp://10.0.2.1:4567') turning message relay requesting on, nonlive peers:
  224. 2023-04-14 9:04:23 0 [Note] WSREP: (595eed12-a6f0, 'tcp://10.0.2.1:4567') connection established to cc70ed3a-8652 tcp://10.0.3.1:4567
  225. 2023-04-14 9:04:23 0 [Note] WSREP: EVS version upgrade 0 -> 1
  226. 2023-04-14 9:04:23 0 [Note] WSREP: declaring 1660e5cc-ace5 at tcp://10.0.1.1:4567 stable
  227. 2023-04-14 9:04:23 0 [Note] WSREP: declaring cc70ed3a-8652 at tcp://10.0.3.1:4567 stable
  228. 2023-04-14 9:04:23 0 [Note] WSREP: PC protocol upgrade 0 -> 1
  229. 2023-04-14 9:04:23 0 [Note] WSREP: Node 1660e5cc-ace5 state prim
  230. 2023-04-14 9:04:23 0 [Note] WSREP: view(view_id(PRIM,1660e5cc-ace5,5) memb {
  231. 1660e5cc-ace5,0
  232. 595eed12-a6f0,0
  233. cc70ed3a-8652,0
  234. } joined {
  235. } left {
  236. } partitioned {
  237. })
  238. 2023-04-14 9:04:23 0 [Note] WSREP: save pc into disk
  239. 2023-04-14 9:04:24 0 [Note] WSREP: gcomm: connected
  240. 2023-04-14 9:04:24 0 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636
  241. 2023-04-14 9:04:24 0 [Note] WSREP: Shifting CLOSED -> OPEN (TO: 0)
  242. 2023-04-14 9:04:24 0 [Note] WSREP: Opened channel 'openstack'
  243. 2023-04-14 9:04:24 0 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 1, memb_num = 3
  244. 2023-04-14 9:04:24 0 [Note] WSREP: STATE EXCHANGE: Waiting for state UUID.
  245. 2023-04-14 9:04:24 0 [Note] WSREP: STATE EXCHANGE: sent state msg: 59ab4e75-daa3-11ed-8243-4330fca8da0a
  246. 2023-04-14 9:04:24 0 [Note] WSREP: STATE EXCHANGE: got state msg: 59ab4e75-daa3-11ed-8243-4330fca8da0a from 0 (controller1)
  247. 2023-04-14 9:04:24 0 [Note] WSREP: STATE EXCHANGE: got state msg: 59ab4e75-daa3-11ed-8243-4330fca8da0a from 2 (controller3)
  248. 2023-04-14 9:04:24 1 [Note] WSREP: Starting applier thread 1
  249. 2023-04-14 9:04:24 2 [Note] WSREP: Starting rollbacker thread 2
  250. 2023-04-14 9:04:24 0 [Note] WSREP: STATE EXCHANGE: got state msg: 59ab4e75-daa3-11ed-8243-4330fca8da0a from 1 (controller2)
  251. 2023-04-14 9:04:24 0 [Note] WSREP: Quorum results:
  252. version = 6,
  253. component = PRIMARY,
  254. conf_id = 4,
  255. members = 1/3 (joined/total),
  256. act_id = 4,
  257. last_appl. = 0,
  258. protocols = 2/10/4 (gcs/repl/appl),
  259. vote policy= 0,
  260. group UUID = 16614045-daa1-11ed-87d5-f6b7c891d885
  261. 2023-04-14 9:04:24 0 [Note] WSREP: Flow-control interval: [28, 28]
  262. 2023-04-14 9:04:24 0 [Note] WSREP: Shifting OPEN -> PRIMARY (TO: 5)
  263. 2023-04-14 9:04:24 1 [Note] WSREP: ####### processing CC 5, local, ordered
  264. 2023-04-14 9:04:24 1 [Note] WSREP: Process first view: 16614045-daa1-11ed-87d5-f6b7c891d885 my uuid: 595eed12-daa3-11ed-a6f0-26aeed3f3fcb
  265. 2023-04-14 9:04:24 1 [Note] WSREP: Server controller2 connected to cluster at position 16614045-daa1-11ed-87d5-f6b7c891d885:5 with ID 595eed12-daa3-11ed-a6f0-26aeed3f3fcb
  266. 2023-04-14 9:04:24 1 [Note] WSREP: Server status change disconnected -> connected
  267. 2023-04-14 9:04:24 1 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
  268. 2023-04-14 9:04:24 1 [Note] WSREP: ####### My UUID: 595eed12-daa3-11ed-a6f0-26aeed3f3fcb
  269. 2023-04-14 9:04:24 1 [Note] WSREP: Cert index reset to 00000000-0000-0000-0000-000000000000:-1 (proto: 10), state transfer needed: yes
  270. 2023-04-14 9:04:24 0 [Note] WSREP: Service thread queue flushed.
  271. 2023-04-14 9:04:24 1 [Note] WSREP: ####### Assign initial position for certification: 00000000-0000-0000-0000-000000000000:-1, protocol version: -1
  272. 2023-04-14 9:04:24 1 [Note] WSREP: State transfer required:
  273. Group state: 16614045-daa1-11ed-87d5-f6b7c891d885:5
  274. Local state: 00000000-0000-0000-0000-000000000000:-1
  275. 2023-04-14 9:04:24 1 [Note] WSREP: Server status change connected -> joiner
  276. 2023-04-14 9:04:24 1 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
  277. 2023-04-14 9:04:24 0 [Note] WSREP: Joiner monitor thread started to monitor
  278. 2023-04-14 9:04:24 0 [Note] WSREP: Running: 'wsrep_sst_mariabackup --role 'joiner' --address '10.0.2.1:4444' --datadir '/var/lib/mysql/' --parent 223 --progress 0 --binlog 'mysql-bin' --mysqld-args --basedir=/usr --datadir=/var/lib/mysql/ --plugin-dir=/usr/lib/mysql/plugin --wsrep_provider=/usr/lib/galera/libgalera_smm.so --wsrep_on=ON --log-error=/var/log/kolla/mariadb/mariadb.log --pid-file=/var/lib/mysql/mariadb.pid --port=3306 --wsrep_start_position=00000000-0000-0000-0000-000000000000:-1'
  279. WSREP_SST: [INFO] mariabackup SST started on joiner (20230414 09:04:24.238)
  280. WSREP_SST: [INFO] SSL configuration: CA='', CAPATH='', CERT='', KEY='', MODE='DISABLED', encrypt='0' (20230414 09:04:24.253)
  281. WSREP_SST: [INFO] Progress reporting tool pv not found in path: /usr//bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/sbin:/usr/bin:/sbin:/bin (20230414 09:04:24.289)
  282. WSREP_SST: [INFO] Disabling all progress/rate-limiting (20230414 09:04:24.289)
  283. WSREP_SST: [INFO] Streaming with mbstream (20230414 09:04:24.295)
  284. WSREP_SST: [INFO] Using socat as streamer (20230414 09:04:24.296)
  285. WSREP_SST: [INFO] Stale sst_in_progress file: /var/lib/mysql/sst_in_progress (20230414 09:04:24.298)
  286. WSREP_SST: [INFO] Evaluating timeout -k 310 300 socat -u TCP-LISTEN:4444,reuseaddr stdio | '/usr//bin/mbstream' -x; RC=( ${PIPESTATUS[@]} ) (20230414 09:04:24.311)
  287. 2023-04-14 9:04:24 1 [Note] WSREP: ####### IST uuid:00000000-0000-0000-0000-000000000000 f: 0, l: 5, STRv: 3
  288. 2023-04-14 9:04:24 1 [Note] WSREP: IST receiver addr using tcp://10.0.2.1:4568
  289. 2023-04-14 9:04:24 1 [Note] WSREP: Prepared IST receiver for 0-5, listening at: tcp://10.0.2.1:4568
  290. 2023-04-14 9:04:24 0 [Warning] WSREP: Member 1.0 (controller2) requested state transfer from '*any*', but it is impossible to select State Transfer donor: Resource temporarily unavailable
  291. 2023-04-14 9:04:24 1 [Note] WSREP: Requesting state transfer failed: -11(Resource temporarily unavailable). Will keep retrying every 1 second(s)
  292. 2023-04-14 9:04:25 0 [Warning] WSREP: Member 1.0 (controller2) requested state transfer from '*any*', but it is impossible to select State Transfer donor: Resource temporarily unavailable
  293. 2023-04-14 9:04:26 0 [Warning] WSREP: Member 1.0 (controller2) requested state transfer from '*any*', but it is impossible to select State Transfer donor: Resource temporarily unavailable
  294. 2023-04-14 9:04:26 0 [Note] WSREP: (595eed12-a6f0, 'tcp://10.0.2.1:4567') turning message relay requesting off
  295. 2023-04-14 9:04:27 0 [Warning] WSREP: Member 1.0 (controller2) requested state transfer from '*any*', but it is impossible to select State Transfer donor: Resource temporarily unavailable
  296. 2023-04-14 9:04:28 0 [Warning] WSREP: Member 1.0 (controller2) requested state transfer from '*any*', but it is impossible to select State Transfer donor: Resource temporarily unavailable
  297. 2023-04-14 9:04:29 0 [Warning] WSREP: Member 1.0 (controller2) requested state transfer from '*any*', but it is impossible to select State Transfer donor: Resource temporarily unavailable
  298. 2023-04-14 9:04:30 0 [Warning] WSREP: Member 1.0 (controller2) requested state transfer from '*any*', but it is impossible to select State Transfer donor: Resource temporarily unavailable
  299. 2023-04-14 9:04:31 0 [Warning] WSREP: Member 1.0 (controller2) requested state transfer from '*any*', but it is impossible to select State Transfer donor: Resource temporarily unavailable
  300. 2023-04-14 9:04:32 0 [Warning] WSREP: Member 1.0 (controller2) requested state transfer from '*any*', but it is impossible to select State Transfer donor: Resource temporarily unavailable
  301. 2023-04-14 9:04:33 0 [Warning] WSREP: Member 1.0 (controller2) requested state transfer from '*any*', but it is impossible to select State Transfer donor: Resource temporarily unavailable
  302. ...
  303. 2023-04-14 9:09:19 0 [Warning] WSREP: Member 1.0 (controller2) requested state transfer from '*any*', but it is impossible to select State Transfer donor: Resource temporarily unavailable
  304. 2023-04-14 9:09:20 0 [Warning] WSREP: Member 1.0 (controller2) requested state transfer from '*any*', but it is impossible to select State Transfer donor: Resource temporarily unavailable
  305. 2023-04-14 9:09:21 0 [Warning] WSREP: Member 1.0 (controller2) requested state transfer from '*any*', but it is impossible to select State Transfer donor: Resource temporarily unavailable
  306. 2023-04-14 9:09:22 0 [Warning] WSREP: Member 1.0 (controller2) requested state transfer from '*any*', but it is impossible to select State Transfer donor: Resource temporarily unavailable
  307. 2023-04-14 9:09:23 0 [Warning] WSREP: Member 1.0 (controller2) requested state transfer from '*any*', but it is impossible to select State Transfer donor: Resource temporarily unavailable
  308. WSREP_SST: [ERROR] Possible timeout in receiving first data from donor in gtid stage: exit codes: 124 0 (20230414 09:09:24.313)
  309. WSREP_SST: [ERROR] Cleanup after exit with status: 32 (20230414 09:09:24.314)
  310. WSREP_SST: [ERROR] Removing /tmp/tmp.gsFL8EkFRm/xtrabackup_galera_info file due to signal (20230414 09:09:24.315)
  311. WSREP_SST: [INFO] Removing the sst_in_progress file (20230414 09:09:24.316)
  312. WSREP_SST: [INFO] Cleaning up temporary directories (20230414 09:09:24.317)
  313. 2023-04-14 9:09:24 0 [ERROR] WSREP: Process completed with error: wsrep_sst_mariabackup --role 'joiner' --address '10.0.2.1:4444' --datadir '/var/lib/mysql/' --parent 223 --progress 0 --binlog 'mysql-bin' --mysqld-args --basedir=/usr --datadir=/var/lib/mysql/ --plugin-dir=/usr/lib/mysql/plugin --wsrep_provider=/usr/lib/galera/libgalera_smm.so --wsrep_on=ON --log-error=/var/log/kolla/mariadb/mariadb.log --pid-file=/var/lib/mysql/mariadb.pid --port=3306 --wsrep_start_position=00000000-0000-0000-0000-000000000000:-1: 32 (Broken pipe)
  314. 2023-04-14 9:09:24 0 [ERROR] WSREP: Failed to read uuid:seqno and wsrep_gtid_domain_id from joiner script.
  315. 2023-04-14 9:09:24 3 [Note] WSREP: SST received
  316. 2023-04-14 9:09:24 3 [Note] WSREP: SST received: 00000000-0000-0000-0000-000000000000:-1
  317. 2023-04-14 9:09:24 0 [Warning] WSREP: Member 1.0 (controller2) requested state transfer from '*any*', but it is impossible to select State Transfer donor: Resource temporarily unavailable
  318. 2023-04-14 9:09:25 0 [Warning] WSREP: Member 1.0 (controller2) requested state transfer from '*any*', but it is impossible to select State Transfer donor: Resource temporarily unavailable
  319. 2023-04-14 9:09:26 0 [Warning] WSREP: Member 1.0 (controller2) requested state transfer from '*any*', but it is impossible to select State Transfer donor: Resource temporarily unavailable
  320. 2023-04-14 9:09:27 0 [Warning] WSREP: Member 1.0 (controller2) requested state transfer from '*any*', but it is impossible to select State Transfer donor: Resource temporarily unavailable
  321. 2023-04-14 9:09:28 0 [Warning] WSREP: Member 1.0 (controller2) requested state transfer from '*any*', but it is impossible to select State Transfer donor: Resource temporarily unavailable
  322. 2023-04-14 9:09:29 0 [Warning] WSREP: Member 1.0 (controller2) requested state transfer from '*any*', but it is impossible to select State Transfer donor: Resource temporarily unavailable
  323. ...
  324.  
  325. ...
  326. 2023-04-14 9:12:26 0 [Warning] WSREP: Member 1.0 (controller2) requested state transfer from '*any*', but it is impossible to select State Transfer donor: Resource temporarily unavailable
  327. 2023-04-14 9:12:27 0 [Warning] WSREP: Member 1.0 (controller2) requested state transfer from '*any*', but it is impossible to select State Transfer donor: Resource temporarily unavailable
  328. 2023-04-14 9:12:28 0 [Warning] WSREP: Member 1.0 (controller2) requested state transfer from '*any*', but it is impossible to select State Transfer donor: Resource temporarily unavailable
  329. 2023-04-14 9:12:29 0 [Warning] WSREP: Member 1.0 (controller2) requested state transfer from '*any*', but it is impossible to select State Transfer donor: Resource temporarily unavailable
  330. 2023-04-14 9:12:30 0 [Warning] WSREP: Member 1.0 (controller2) requested state transfer from '*any*', but it is impossible to select State Transfer donor: Resource temporarily unavailable
  331. 2023-04-14 9:12:31 0 [Note] WSREP: (595eed12-a6f0, 'tcp://10.0.2.1:4567') turning message relay requesting on, nonlive peers: tcp://10.0.1.1:4567
  332. 2023-04-14 9:12:32 0 [Note] WSREP: (595eed12-a6f0, 'tcp://10.0.2.1:4567') connection established to 1660e5cc-ace6 tcp://10.0.1.1:4567
  333. 2023-04-14 9:12:32 0 [Note] WSREP: remote endpoint tcp://10.0.1.1:4567 changed identity 1660e5cc-daa1-11ed-ace5-ea843cdbd9aa -> 1660e5cc-daa1-11ed-ace6-ea843cdbd9aa
  334. 2023-04-14 9:12:36 0 [Note] WSREP: (595eed12-a6f0, 'tcp://10.0.2.1:4567') turning message relay requesting off
  335. 2023-04-14 9:12:36 0 [Note] WSREP: evs::proto(595eed12-a6f0, OPERATIONAL, view_id(REG,1660e5cc-ace5,5)) suspecting node: 1660e5cc-ace5
  336. 2023-04-14 9:12:36 0 [Note] WSREP: evs::proto(595eed12-a6f0, OPERATIONAL, view_id(REG,1660e5cc-ace5,5)) suspected node without join message, declaring inactive
  337. 2023-04-14 9:12:37 0 [Note] WSREP: declaring cc70ed3a-8652 at tcp://10.0.3.1:4567 stable
  338. 2023-04-14 9:12:37 0 [Warning] WSREP: Member 1.0 (controller2) requested state transfer from '*any*', but it is impossible to select State Transfer donor: Resource temporarily unavailable
  339. 2023-04-14 9:12:37 0 [Note] WSREP: Node 595eed12-a6f0 state prim
  340. 2023-04-14 9:12:37 0 [Note] WSREP: view(view_id(PRIM,595eed12-a6f0,6) memb {
  341. 595eed12-a6f0,0
  342. cc70ed3a-8652,0
  343. } joined {
  344. } left {
  345. } partitioned {
  346. 1660e5cc-ace5,0
  347. })
  348. 2023-04-14 9:12:37 0 [Note] WSREP: save pc into disk
  349. 2023-04-14 9:12:37 0 [Note] WSREP: Deferred close timer started for socket with remote endpoint: tcp://10.0.1.1:48007
  350. 2023-04-14 9:12:37 0 [Note] WSREP: forgetting 1660e5cc-ace6 (tcp://10.0.1.1:4567)
  351. 2023-04-14 9:12:37 0 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 2
  352. 2023-04-14 9:12:37 0 [Note] WSREP: Deferred close timer handle_wait Operation aborted. for 0x563e62bd9630
  353. 2023-04-14 9:12:37 0 [Note] WSREP: Deferred close timer destruct
  354. 2023-04-14 9:12:37 0 [Note] WSREP: STATE_EXCHANGE: sent state UUID: 80258a4a-daa4-11ed-b207-ab779892c903
  355. 2023-04-14 9:12:37 0 [Note] WSREP: STATE EXCHANGE: sent state msg: 80258a4a-daa4-11ed-b207-ab779892c903
  356. 2023-04-14 9:12:37 0 [Note] WSREP: STATE EXCHANGE: got state msg: 80258a4a-daa4-11ed-b207-ab779892c903 from 0 (controller2)
  357. 2023-04-14 9:12:37 0 [Note] WSREP: STATE EXCHANGE: got state msg: 80258a4a-daa4-11ed-b207-ab779892c903 from 1 (controller3)
  358. 2023-04-14 9:12:37 0 [Warning] WSREP: Quorum: No node with complete state:
  359.  
  360. Version : 6
  361. Flags : 0x1
  362. Protocols : 2 / 10 / 4
  363. State : PRIMARY
  364. Desync count : 0
  365. Prim state : PRIMARY
  366. Prim UUID : 59ab4e75-daa3-11ed-8243-4330fca8da0a
  367. Prim seqno : 5
  368. First seqno : -1
  369. Last seqno : 5
  370. Commit cut : 0
  371. Last vote : -1.0
  372. Vote policy : 0
  373. Prim JOINED : 1
  374. State UUID : 80258a4a-daa4-11ed-b207-ab779892c903
  375. Group UUID : 16614045-daa1-11ed-87d5-f6b7c891d885
  376. Name : 'controller2'
  377. Incoming addr: '10.0.2.1:3306'
  378.  
  379. Version : 6
  380. Flags : 00
  381. Protocols : 2 / 10 / 4
  382. State : JOINER
  383. Desync count : 0
  384. Prim state : JOINER
  385. Prim UUID : 59ab4e75-daa3-11ed-8243-4330fca8da0a
  386. Prim seqno : 5
  387. First seqno : -1
  388. Last seqno : 5
  389. Commit cut : 0
  390. Last vote : -1.0
  391. Vote policy : 0
  392. Prim JOINED : 1
  393. State UUID : 80258a4a-daa4-11ed-b207-ab779892c903
  394. Group UUID : 16614045-daa1-11ed-87d5-f6b7c891d885
  395. Name : 'controller3'
  396. Incoming addr: '10.0.3.1:3306'
  397.  
  398. 2023-04-14 9:12:37 0 [Warning] WSREP: No re-merged primary component found.
  399. 2023-04-14 9:12:37 0 [Warning] WSREP: No bootstrapped primary component found.
  400. 2023-04-14 9:12:37 0 [ERROR] WSREP: ./gcs/src/gcs_state_msg.cpp:gcs_state_msg_get_quorum():947: Failed to establish quorum.
  401. 2023-04-14 9:12:37 0 [Note] WSREP: Quorum results:
  402. version = 6,
  403. component = NON-PRIMARY,
  404. conf_id = -1,
  405. members = 1/2 (joined/total),
  406. act_id = -1,
  407. last_appl. = 0,
  408. protocols = -1/-1/-1 (gcs/repl/appl),
  409. vote policy= 1,
  410. group UUID = 00000000-0000-0000-0000-000000000000
  411. 2023-04-14 9:12:37 0 [Note] WSREP: Flow-control interval: [23, 23]
  412. 2023-04-14 9:12:37 0 [Note] WSREP: Received NON-PRIMARY.
  413. 2023-04-14 9:12:37 0 [Note] WSREP: Shifting PRIMARY -> OPEN (TO: 5)
  414. 2023-04-14 9:12:38 1 [Warning] WSREP: Send action {0x7f624001f700, 147, STATE_REQUEST} returned -107 (Transport endpoint is not connected)
  415. 2023-04-14 9:12:38 0 [Note] WSREP: (595eed12-a6f0, 'tcp://10.0.2.1:4567') connection established to 1660e5cc-ace6 tcp://10.0.1.1:4567
  416. 2023-04-14 9:12:38 0 [Warning] WSREP: discarding established (time wait) 1660e5cc-ace6 (tcp://10.0.1.1:4567)
  417. 2023-04-14 9:12:38 0 [Note] WSREP: Deferred close timer started for socket with remote endpoint: tcp://10.0.1.1:39565
  418. 2023-04-14 9:12:38 0 [Note] WSREP: Deferred close timer handle_wait Operation aborted. for 0x563e62bd7560
  419. 2023-04-14 9:12:38 0 [Note] WSREP: Deferred close timer destruct
  420. 2023-04-14 9:12:39 1 [Warning] WSREP: Send action {0x7f624001f700, 147, STATE_REQUEST} returned -107 (Transport endpoint is not connected)
  421. 2023-04-14 9:12:40 0 [Note] WSREP: (595eed12-a6f0, 'tcp://10.0.2.1:4567') connection established to 1660e5cc-ace6 tcp://10.0.1.1:4567
  422. 2023-04-14 9:12:40 0 [Warning] WSREP: discarding established (time wait) 1660e5cc-ace6 (tcp://10.0.1.1:4567)
  423. 2023-04-14 9:12:40 0 [Note] WSREP: Deferred close timer started for socket with remote endpoint: tcp://10.0.1.1:58081
  424. 2023-04-14 9:12:40 0 [Note] WSREP: Deferred close timer handle_wait Operation aborted. for 0x7f6244001990
  425. 2023-04-14 9:12:40 0 [Note] WSREP: Deferred close timer destruct
  426. 2023-04-14 9:12:40 0 [Note] WSREP: cleaning up 1660e5cc-ace6 (tcp://10.0.1.1:4567)
  427. 2023-04-14 9:12:40 1 [Warning] WSREP: Send action {0x7f624001f700, 147, STATE_REQUEST} returned -107 (Transport endpoint is not connected)
  428. 2023-04-14 9:12:41 0 [Note] WSREP: (595eed12-a6f0, 'tcp://10.0.2.1:4567') connection established to 1660e5cc-ace6 tcp://10.0.1.1:4567
  429. 2023-04-14 9:12:41 0 [Note] WSREP: (595eed12-a6f0, 'tcp://10.0.2.1:4567') turning message relay requesting on, nonlive peers:
  430. 2023-04-14 9:12:41 1 [Warning] WSREP: Send action {0x7f624001f700, 147, STATE_REQUEST} returned -107 (Transport endpoint is not connected)
  431. 2023-04-14 9:12:42 1 [Warning] WSREP: Send action {0x7f624001f700, 147, STATE_REQUEST} returned -107 (Transport endpoint is not connected)
  432. 2023-04-14 9:12:43 1 [Warning] WSREP: Send action {0x7f624001f700, 147, STATE_REQUEST} returned -107 (Transport endpoint is not connected)
  433. 2023-04-14 9:12:44 0 [Note] WSREP: (595eed12-a6f0, 'tcp://10.0.2.1:4567') turning message relay requesting off
  434. 2023-04-14 9:12:44 1 [Warning] WSREP: Send action {0x7f624001f700, 147, STATE_REQUEST} returned -107 (Transport endpoint is not connected)
  435. 2023-04-14 9:12:45 1 [Warning] WSREP: Send action {0x7f624001f700, 147, STATE_REQUEST} returned -107 (Transport endpoint is not connected)
  436. 2023-04-14 9:12:46 0 [Note] WSREP: declaring cc70ed3a-8652 at tcp://10.0.3.1:4567 stable
  437. 2023-04-14 9:12:46 0 [Note] WSREP: Node 595eed12-a6f0 state prim
  438. 2023-04-14 9:12:46 0 [Warning] WSREP: 595eed12-a6f0 sending install message failed: Resource temporarily unavailable
  439. 2023-04-14 9:12:46 0 [Note] WSREP: declaring 1660e5cc-ace6 at tcp://10.0.1.1:4567 stable
  440. 2023-04-14 9:12:46 0 [Note] WSREP: declaring cc70ed3a-8652 at tcp://10.0.3.1:4567 stable
  441. 2023-04-14 9:12:46 0 [Note] WSREP: Node 595eed12-a6f0 state prim
  442. 2023-04-14 9:12:46 0 [Note] WSREP: view(view_id(PRIM,1660e5cc-ace6,8) memb {
  443. 1660e5cc-ace6,0
  444. 595eed12-a6f0,0
  445. cc70ed3a-8652,0
  446. } joined {
  447. } left {
  448. } partitioned {
  449. })
  450. 2023-04-14 9:12:46 0 [Note] WSREP: save pc into disk
  451. 2023-04-14 9:12:46 0 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 1, memb_num = 3
  452. 2023-04-14 9:12:46 0 [Note] WSREP: STATE EXCHANGE: Waiting for state UUID.
  453. 2023-04-14 9:12:46 0 [Note] WSREP: STATE EXCHANGE: sent state msg: 857f383f-daa4-11ed-b1d5-8b0ce5c246b3
  454. 2023-04-14 9:12:46 0 [Note] WSREP: STATE EXCHANGE: got state msg: 857f383f-daa4-11ed-b1d5-8b0ce5c246b3 from 0 (controller1)
  455. 2023-04-14 9:12:46 0 [Note] WSREP: STATE EXCHANGE: got state msg: 857f383f-daa4-11ed-b1d5-8b0ce5c246b3 from 1 (controller2)
  456. 2023-04-14 9:12:46 0 [Note] WSREP: STATE EXCHANGE: got state msg: 857f383f-daa4-11ed-b1d5-8b0ce5c246b3 from 2 (controller3)
  457. 2023-04-14 9:12:46 0 [Warning] WSREP: Quorum: No node with complete state:
  458.  
  459. Version : 6
  460. Flags : 0x1
  461. Protocols : 2 / 10 / 4
  462. State : NON-PRIMARY
  463. Desync count : 0
  464. Prim state : NON-PRIMARY
  465. Prim UUID : 00000000-0000-0000-0000-000000000000
  466. Prim seqno : -1
  467. First seqno : 1
  468. Last seqno : 5
  469. Commit cut : 5
  470. Last vote : -1.0
  471. Vote policy : 0
  472. Prim JOINED : 0
  473. State UUID : 857f383f-daa4-11ed-b1d5-8b0ce5c246b3
  474. Group UUID : 16614045-daa1-11ed-87d5-f6b7c891d885
  475. Name : 'controller1'
  476. Incoming addr: '10.0.1.1:3306'
  477.  
  478. Version : 6
  479. Flags : 00
  480. Protocols : 2 / 10 / 4
  481. State : NON-PRIMARY
  482. Desync count : 0
  483. Prim state : PRIMARY
  484. Prim UUID : 59ab4e75-daa3-11ed-8243-4330fca8da0a
  485. Prim seqno : 5
  486. First seqno : -1
  487. Last seqno : 5
  488. Commit cut : 0
  489. Last vote : -1.0
  490. Vote policy : 0
  491. Prim JOINED : 1
  492. State UUID : 857f383f-daa4-11ed-b1d5-8b0ce5c246b3
  493. Group UUID : 16614045-daa1-11ed-87d5-f6b7c891d885
  494. Name : 'controller2'
  495. Incoming addr: '10.0.2.1:3306'
  496.  
  497. Version : 6
  498. Flags : 00
  499. Protocols : 2 / 10 / 4
  500. State : NON-PRIMARY
  501. Desync count : 0
  502. Prim state : JOINER
  503. Prim UUID : 59ab4e75-daa3-11ed-8243-4330fca8da0a
  504. Prim seqno : 5
  505. First seqno : -1
  506. Last seqno : 5
  507. Commit cut : 0
  508. Last vote : -1.0
  509. Vote policy : 0
  510. Prim JOINED : 1
  511. State UUID : 857f383f-daa4-11ed-b1d5-8b0ce5c246b3
  512. Group UUID : 16614045-daa1-11ed-87d5-f6b7c891d885
  513. Name : 'controller3'
  514. Incoming addr: '10.0.3.1:3306'
  515.  
  516. 2023-04-14 9:12:46 0 [Warning] WSREP: No re-merged primary component found.
  517. 2023-04-14 9:12:46 0 [Warning] WSREP: No bootstrapped primary component found.
  518. 2023-04-14 9:12:46 0 [ERROR] WSREP: ./gcs/src/gcs_state_msg.cpp:gcs_state_msg_get_quorum():947: Failed to establish quorum.
  519. 2023-04-14 9:12:46 0 [Note] WSREP: Quorum results:
  520. version = 6,
  521. component = NON-PRIMARY,
  522. conf_id = -1,
  523. members = 1/3 (joined/total),
  524. act_id = -1,
  525. last_appl. = 0,
  526. protocols = -1/-1/-1 (gcs/repl/appl),
  527. vote policy= 1,
  528. group UUID = 00000000-0000-0000-0000-000000000000
  529. 2023-04-14 9:12:46 0 [Note] WSREP: Flow-control interval: [28, 28]
  530. 2023-04-14 9:12:46 0 [Note] WSREP: Received NON-PRIMARY.
  531. 2023-04-14 9:12:46 1 [Warning] WSREP: Send action {0x7f624001f700, 147, STATE_REQUEST} returned -107 (Transport endpoint is not connected)
  532. 2023-04-14 9:12:47 1 [Warning] WSREP: Send action {0x7f624001f700, 147, STATE_REQUEST} returned -107 (Transport endpoint is not connected)
  533. 2023-04-14 9:12:48 1 [Warning] WSREP: Send action {0x7f624001f700, 147, STATE_REQUEST} returned -107 (Transport endpoint is not connected)
  534. 2023-04-14 9:12:49 1 [Warning] WSREP: Send action {0x7f624001f700, 147, STATE_REQUEST} returned -107 (Transport endpoint is not connected)
  535. 2023-04-14 9:12:50 1 [Warning] WSREP: Send action {0x7f624001f700, 147, STATE_REQUEST} returned -107 (Transport endpoint is not connected)
  536. 2023-04-14 9:12:51 1 [Warning] WSREP: Send action {0x7f624001f700, 147, STATE_REQUEST} returned -107 (Transport endpoint is not connected)
  537. 2023-04-14 9:12:52 1 [Warning] WSREP: Send action {0x7f624001f700, 147, STATE_REQUEST} returned -107 (Transport endpoint is not connected)
  538. 2023-04-14 9:12:53 1 [Warning] WSREP: Send action {0x7f624001f700, 147, STATE_REQUEST} returned -107 (Transport endpoint is not connected)
  539. 2023-04-14 9:12:54 1 [Warning] WSREP: Send action {0x7f624001f700, 147, STATE_REQUEST} returned -107 (Transport endpoint is not connected)
  540. 2023-04-14 9:12:55 1 [Warning] WSREP: Send action {0x7f624001f700, 147, STATE_REQUEST} returned -107 (Transport endpoint is not connected)
  541. ...
  542.  
  543. ...
  544. 2023-04-21 12:58:57 1 [Warning] WSREP: Send action {0x7f624001f700, 147, STATE_REQUEST} returned -107 (Transport endpoint is not connected)
  545. 2023-04-21 12:58:58 1 [Warning] WSREP: Send action {0x7f624001f700, 147, STATE_REQUEST} returned -107 (Transport endpoint is not connected)
  546. 2023-04-21 12:58:59 1 [Warning] WSREP: Send action {0x7f624001f700, 147, STATE_REQUEST} returned -107 (Transport endpoint is not connected)
  547. 2023-04-21 12:59:00 1 [Warning] WSREP: Send action {0x7f624001f700, 147, STATE_REQUEST} returned -107 (Transport endpoint is not connected)
  548. 2023-04-21 12:59:01 1 [Warning] WSREP: Send action {0x7f624001f700, 147, STATE_REQUEST} returned -107 (Transport endpoint is not connected)
  549. 2023-04-21 12:59:02 1 [Warning] WSREP: Send action {0x7f624001f700, 147, STATE_REQUEST} returned -107 (Transport endpoint is not connected)
  550. 2023-04-21 12:59:03 1 [Warning] WSREP: Send action {0x7f624001f700, 147, STATE_REQUEST} returned -107 (Transport endpoint is not connected)
  551. 2023-04-21 12:59:04 1 [Warning] WSREP: Send action {0x7f624001f700, 147, STATE_REQUEST} returned -107 (Transport endpoint is not connected)
  552. 2023-04-21 12:59:05 1 [Warning] WSREP: Send action {0x7f624001f700, 147, STATE_REQUEST} returned -107 (Transport endpoint is not connected)
  553. 2023-04-21 12:59:06 1 [Warning] WSREP: Send action {0x7f624001f700, 147, STATE_REQUEST} returned -107 (Transport endpoint is not connected)
  554. 2023-04-21 12:59:07 1 [Warning] WSREP: Send action {0x7f624001f700, 147, STATE_REQUEST} returned -107 (Transport endpoint is not connected)
  555.  
Tags: openstack
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement
Public Pastes
Advertisement
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!