jreiners

Untitled

Sep 30th, 2013
651
Never
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
  1. 130930 07:31:29 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
  2. 130930 07:31:29 mysqld_safe Skipping wsrep-recover for empty datadir: /var/lib/mysql
  3. 130930 07:31:29 mysqld_safe Assigning 00000000-0000-0000-0000-000000000000:-1 to wsrep_start_position
  4. 130930 7:31:29 [Note] WSREP: wsrep_start_position var submitted: '00000000-0000-0000-0000-000000000000:-1'
  5. 130930 7:31:29 [Note] WSREP: Read nil XID from storage engines, skipping position init
  6. 130930 7:31:29 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib64/libgalera_smm.so'
  7. 130930 7:31:29 [Note] WSREP: wsrep_load(): Galera 2.7(r157) by Codership Oy <info@codership.com> loaded succesfully.
  8. 130930 7:31:29 [Warning] WSREP: Could not open saved state file for reading: /var/lib/mysql//grastate.dat
  9. 130930 7:31:29 [Note] WSREP: Found saved state: 00000000-0000-0000-0000-000000000000:-1
  10. 130930 7:31:29 [Note] WSREP: Preallocating 134219048/134219048 bytes in '/var/lib/mysql//galera.cache'...
  11. 130930 7:31:29 [Note] WSREP: Passing config to GCS: base_host = 192.168.9.140; base_port = 4567; cert.log_conflicts = no; gcache.dir = /var/lib/mysql/; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = /var/lib/mysql//galera.cache; gcache.page_size = 128M; gcache.size = 128M; gcs.fc_debug = 0; gcs.fc_factor = 1; gcs.fc_limit = 16; gcs.fc_master_slave = NO; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = NO; replicator.causal_read_timeout = PT30S; replicator.commit_order = 3
  12. 130930 7:31:29 [Note] WSREP: Assign initial position for certification: -1, protocol version: -1
  13. 130930 7:31:29 [Note] WSREP: wsrep_sst_grab()
  14. 130930 7:31:29 [Note] WSREP: Start replication
  15. 130930 7:31:29 [Note] WSREP: Setting initial position to 00000000-0000-0000-0000-000000000000:-1
  16.  
  17. 130930 7:31:29 [Note] WSREP: protonet asio version 0
  18. 130930 7:31:29 [Note] WSREP: backend: asio
  19. 130930 7:31:29 [Note] WSREP: GMCast version 0
  20. 130930 7:31:29 [Note] WSREP: (3b11552d-29cc-11e3-b6f6-1e324447576e, 'tcp://0.0.0.0:4567') listening at tcp://0.0.0.0:4567
  21. 130930 7:31:29 [Note] WSREP: (3b11552d-29cc-11e3-b6f6-1e324447576e, 'tcp://0.0.0.0:4567') multicast: , ttl: 1
  22. 130930 7:31:29 [Note] WSREP: EVS version 0
  23. 130930 7:31:29 [Note] WSREP: PC version 0
  24. 130930 7:31:29 [Note] WSREP: gcomm: connecting to group 'joomla', peer '192.168.9.130:,192.168.9.140:,192.168.9.14:,192.168.200.130:,192.168.200.140:'
  25. 130930 7:31:29 [Warning] WSREP: (3b11552d-29cc-11e3-b6f6-1e324447576e, 'tcp://0.0.0.0:4567') address 'tcp://192.168.9.140:4567' points to own listening address, blacklisting
  26. 130930 7:31:29 [Note] WSREP: (3b11552d-29cc-11e3-b6f6-1e324447576e, 'tcp://0.0.0.0:4567') address 'tcp://192.168.9.140:4567' pointing to uuid 3b11552d-29cc-11e3-b6f6-1e324447576e is blacklisted, skipping
  27. 130930 7:31:29 [Note] WSREP: declaring c8b2b1a7-27c9-11e3-a0d3-76954e61f4c4 stable
  28. 130930 7:31:29 [Note] WSREP: Node c8b2b1a7-27c9-11e3-a0d3-76954e61f4c4 state prim
  29. 130930 7:31:29 [Note] WSREP: view(view_id(PRIM,3b11552d-29cc-11e3-b6f6-1e324447576e,427) memb {
  30. 3b11552d-29cc-11e3-b6f6-1e324447576e,
  31. c8b2b1a7-27c9-11e3-a0d3-76954e61f4c4,
  32. } joined {
  33. } left {
  34. } partitioned {
  35. })
  36. 130930 7:31:29 [Note] WSREP: discarding pending addr without UUID: tcp://192.168.200.140:4567
  37. 130930 7:31:29 [Note] WSREP: discarding pending addr without UUID: tcp://192.168.9.130:4567
  38. 130930 7:31:29 [Note] WSREP: discarding pending addr proto entry 0x1ac4620
  39. 130930 7:31:29 [Note] WSREP: discarding pending addr without UUID: tcp://192.168.9.14:4567
  40. 130930 7:31:29 [Note] WSREP: gcomm: connected
  41. 130930 7:31:29 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636
  42. 130930 7:31:29 [Note] WSREP: Shifting CLOSED -> OPEN (TO: 0)
  43. 130930 7:31:29 [Note] WSREP: Opened channel 'joomla'
  44. 130930 7:31:29 [Note] WSREP: Waiting for SST to complete.
  45. 130930 7:31:29 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 2
  46. 130930 7:31:29 [Note] WSREP: STATE_EXCHANGE: sent state UUID: 3b5e266f-29cc-11e3-bb53-a35933296aec
  47. 130930 7:31:29 [Note] WSREP: STATE EXCHANGE: sent state msg: 3b5e266f-29cc-11e3-bb53-a35933296aec
  48. 130930 7:31:29 [Note] WSREP: STATE EXCHANGE: got state msg: 3b5e266f-29cc-11e3-bb53-a35933296aec from 0 (percona5.hotlinesinc.com)
  49. 130930 7:31:29 [Note] WSREP: STATE EXCHANGE: got state msg: 3b5e266f-29cc-11e3-bb53-a35933296aec from 1 (db1.dmz.hotlinesinc.com)
  50. 130930 7:31:29 [Note] WSREP: Quorum results:
  51. version = 2,
  52. component = PRIMARY,
  53. conf_id = 397,
  54. members = 1/2 (joined/total),
  55. act_id = 107149391,
  56. last_appl. = -1,
  57. protocols = 0/4/2 (gcs/repl/appl),
  58. group UUID = 9178165e-feb8-11e2-831d-aaf1b46590ab
  59. 130930 7:31:29 [Note] WSREP: Flow-control interval: [23, 23]
  60. 130930 7:31:29 [Note] WSREP: Shifting OPEN -> PRIMARY (TO: 107149391)
  61. 130930 7:31:29 [Note] WSREP: State transfer required:
  62. Group state: 9178165e-feb8-11e2-831d-aaf1b46590ab:107149391
  63. Local state: 00000000-0000-0000-0000-000000000000:-1
  64. 130930 7:31:29 [Note] WSREP: New cluster view: global state: 9178165e-feb8-11e2-831d-aaf1b46590ab:107149391, view# 398: Primary, number of nodes: 2, my index: 0, protocol version 2
  65. 130930 7:31:29 [Warning] WSREP: Gap in state sequence. Need state transfer.
  66. 130930 7:31:31 [Note] WSREP: Running: 'wsrep_sst_xtrabackup --role 'joiner' --address '192.168.9.140' --auth 'sstuser:s3cret' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --parent '12346''
  67. WSREP_SST: [INFO] Streaming with xbstream (20130930 07:31:31.806)
  68. WSREP_SST: [INFO] Using socat as streamer (20130930 07:31:31.808)
  69. WSREP_SST: [INFO] Evaluating socat -u TCP-LISTEN:4444,reuseaddr stdio | xbstream -x; RC=( ${PIPESTATUS[@]} ) (20130930 07:31:31.818)
  70. 130930 7:31:32 [Note] WSREP: Prepared SST request: xtrabackup|192.168.9.140:4444/xtrabackup_sst
  71. 130930 7:31:32 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
  72. 130930 7:31:32 [Note] WSREP: Assign initial position for certification: 107149391, protocol version: 2
  73. 130930 7:31:32 [Warning] WSREP: Failed to prepare for incremental state transfer: Local state UUID (00000000-0000-0000-0000-000000000000) does not match group state UUID (9178165e-feb8-11e2-831d-aaf1b46590ab): 1 (Operation not permitted)
  74. at galera/src/replicator_str.cpp:prepare_for_IST():436. IST will be unavailable.
  75. 130930 7:31:32 [Note] WSREP: Node 0 (percona5.hotlinesinc.com) requested state transfer from '*any*'. Selected 1 (db1.dmz.hotlinesinc.com)(SYNCED) as donor.
  76. 130930 7:31:32 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 107149394)
  77. 130930 7:31:32 [Note] WSREP: Requesting state transfer: success, donor: 1
  78. xb_stream_read_chunk(): wrong chunk magic at offset 0x0.
  79. 2013/09/30 07:31:32 socat[12519] E write(1, 0xaf43e0, 2692): Broken pipe
  80. WSREP_SST: [ERROR] Error while getting data from donor node: exit codes: 1 1 (20130930 07:31:32.192)
  81. WSREP_SST: [ERROR] Cleanup after exit with status:32 (20130930 07:31:32.194)
  82. WSREP_SST: [INFO] Removing the sst_in_progress file (20130930 07:31:32.197)
  83. 130930 7:31:32 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup --role 'joiner' --address '192.168.9.140' --auth 'sstuser:s3cret' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --parent '12346': 32 (Broken pipe)
  84. 130930 7:31:32 [ERROR] WSREP: Failed to read uuid:seqno from joiner script.
  85. 130930 7:31:32 [ERROR] WSREP: SST failed: 32 (Broken pipe)
  86. 130930 7:31:32 [ERROR] Aborting
  87.  
  88. 130930 7:31:33 [Warning] WSREP: 1 (db1.dmz.hotlinesinc.com): State transfer to 0 (percona5.hotlinesinc.com) failed: -1 (Operation not permitted)
  89. 130930 7:31:33 [ERROR] WSREP: gcs/src/gcs_group.c:gcs_group_handle_join_msg():719: Will never receive state. Need to abort.
  90. 130930 7:31:33 [Note] WSREP: gcomm: terminating thread
  91. 130930 7:31:33 [Note] WSREP: gcomm: joining thread
  92. 130930 7:31:33 [Note] WSREP: gcomm: closing backend
  93. 130930 7:31:34 [Note] WSREP: Closing send monitor...
  94. 130930 7:31:34 [Note] WSREP: Closed send monitor.
  95. 130930 7:31:34 [Note] WSREP: gcomm: terminating thread
  96. 130930 7:31:34 [Note] WSREP: gcomm: joining thread
  97. 130930 7:31:34 [Note] WSREP: gcomm: closing backend
  98. 130930 7:31:34 [ERROR] WSREP: failed to close gcomm backend connection: 131: Forbidden state transition: LEAVING -> LEAVING (FATAL)
  99. at gcomm/src/evs_proto.cpp:shift_to():2185
  100. 130930 7:31:34 [ERROR] WSREP: exception caused by message: evs::msg{version=0,type=5,user_type=255,order=1,seq=14,seq_range=-1,aru_seq=14,flags=4,source=c8b2b1a7-27c9-11e3-a0d3-76954e61f4c4,source_view_id=view_id(REG,3b11552d-29cc-11e3-b6f6-1e324447576e,427),range_uuid=00000000-0000-0000-0000-000000000000,range=[-1,-1],fifo_seq=2454450,node_list=( 3b11552d-29cc-11e3-b6f6-1e324447576e,node: {operational=0,suspected=0,leave_seq=14,view_id=view_id(REG,3b11552d-29cc-11e3-b6f6-1e324447576e,427),safe_seq=14,im_range=[15,14],}
  101. c8b2b1a7-27c9-11e3-a0d3-76954e61f4c4,node: {operational=1,suspected=0,leave_seq=-1,view_id=view_id(REG,3b11552d-29cc-11e3-b6f6-1e324447576e,427),safe_seq=14,im_range=[15,14],}
  102. )
  103. }
  104. state after handling message: evs::proto(evs::proto(3b11552d-29cc-11e3-b6f6-1e324447576e, LEAVING, view_id(REG,3b11552d-29cc-11e3-b6f6-1e324447576e,427)), LEAVING) {
  105. current_view=view(view_id(REG,3b11552d-29cc-11e3-b6f6-1e324447576e,427) memb {
  106. 3b11552d-29cc-11e3-b6f6-1e324447576e,
  107. c8b2b1a7-27c9-11e3-a0d3-76954e61f4c4,
  108. } joined {
  109. } left {
  110. } partitioned {
  111. }),
  112. input_map=evs::input_map: {aru_seq=14,safe_seq=14,node_index=node: {idx=0,range=[15,14],safe_seq=14} node: {idx=1,range=[15,14],safe_seq=14} },
  113. fifo_seq=31,
  114. last_sent=14,
  115. known={
  116. 3b11552d-29cc-11e3-b6f6-1e324447576e,evs::node{operational=1,suspected=0,installed=1,fifo_seq=-1,leave_message=
  117. evs::msg{version=0,type=6,user_type=255,order=1,seq=14,seq_range=-1,aru_seq=14,flags=0,source=3b11552d-29cc-11e3-b6f6-1e324447576e,source_view_id=view_id(REG,3b11552d-29cc-11e3-b6f6-1e324447576e,427),range_uuid=00000000-0000-0000-0000-000000000000,range=[-1,-1],fifo_seq=30,node_list=()
  118. },
  119. }
  120. c8b2b1a7-27c9-11e3-a0d3-76954e61f4c4,evs::node{operational=1,suspected=0,installed=1,fifo_seq=2454450,}
  121. }
  122. }130930 7:31:34 [ERROR] WSREP: failed to close gcomm backend connection: 131: evs::proto(evs::proto(3b11552d-29cc-11e3-b6f6-1e324447576e, LEAVING, view_id(REG,3b11552d-29cc-11e3-b6f6-1e324447576e,427)), LEAVING) {
  123. current_view=view(view_id(REG,3b11552d-29cc-11e3-b6f6-1e324447576e,427) memb {
  124. 3b11552d-29cc-11e3-b6f6-1e324447576e,
  125. c8b2b1a7-27c9-11e3-a0d3-76954e61f4c4,
  126. } joined {
  127. } left {
  128. } partitioned {
  129. }),
  130. input_map=evs::input_map: {aru_seq=14,safe_seq=14,node_index=node: {idx=0,range=[15,14],safe_seq=14} node: {idx=1,range=[15,14],safe_seq=14} },
  131. fifo_seq=31,
  132. last_sent=14,
  133. known={
  134. 3b11552d-29cc-11e3-b6f6-1e324447576e,evs::node{operational=1,suspected=0,installed=1,fifo_seq=-1,leave_message=
  135. evs::msg{version=0,type=6,user_type=255,order=1,seq=14,seq_range=-1,aru_seq=14,flags=0,source=3b11552d-29cc-11e3-b6f6-1e324447576e,source_view_id=view_id(REG,3b11552d-29cc-11e3-b6f6-1e324447576e,427),range_uuid=00000000-0000-0000-0000-000000000000,range=[-1,-1],fifo_seq=30,node_list=()
  136. },
  137. }
  138. c8b2b1a7-27c9-11e3-a0d3-76954e61f4c4,evs::node{oper
  139. 130930 7:31:34 [Note] WSREP: /usr/sbin/mysqld: Terminated.
  140. 130930 07:31:34 mysqld_safe mysqld from pid file /var/lib/mysql/percona5.hotlinesinc.com.pid ended
RAW Paste Data