API tools faq
paste
lifeboy

Untitled

lifeboy
Sep 3rd, 2018
378
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
Bash 12.96 KB | None | 0 0
raw download clone embed print report
  1. On node s1 I see this is the log:
  2. Jul 29 16:20:22 s1 kernel: libceph: mon3 192.168.121.33:6789 socket closed (con state OPEN)
  3. Jul 29 16:20:22 s1 pmxcfs[2719]: [dcdb] notice: members: 1/1218, 3/2719, 4/1304
  4. Jul 29 16:20:22 s1 pmxcfs[2719]: [dcdb] notice: starting data syncronisation
  5. Jul 29 16:20:22 s1 pmxcfs[2719]: [status] notice: members: 1/1218, 3/2719, 4/1304
  6. Jul 29 16:20:22 s1 pmxcfs[2719]: [status] notice: starting data syncronisation
  7. Jul 29 16:20:22 s1 ceph-osd[5828]: 2018-07-29 16:20:22.398059 7f4ab802a700 -1 osd.15 17656 heartbeat_check: no reply from 192.168.121.30:6819 osd.4 since back 2018-07-29 16:19:58.216921 fron
  8. Jul 29 16:20:22 s1 ceph-osd[3546]: 2018-07-29 16:20:22.938311 7f19a1430700 -1 osd.14 17656 heartbeat_check: no reply from 192.168.121.30:6819 osd.4 since back 2018-07-29 16:19:58.859065 fron
  9. Jul 29 16:20:22 s1 ceph-osd[6234]: 2018-07-29 16:20:22.972293 7f7e5bc5f700 -1 osd.10 17656 heartbeat_check: no reply from 192.168.121.30:6819 osd.4 since back 2018-07-29 16:19:56.901979 fron
  10. Jul 29 16:20:22 s1 ceph-osd[5409]: 2018-07-29 16:20:22.995095 7fddb2420700 -1 osd.11 17656 heartbeat_check: no reply from 192.168.121.30:6819 osd.4 since back 2018-07-29 16:19:58.225848 fron
  11. Jul 29 16:20:23 s1 ceph-osd[5828]: 2018-07-29 16:20:23.323712 7f4a9ce7f700 -1 osd.15 17656 heartbeat_check: no reply from 192.168.121.30:6819 osd.4 since back 2018-07-29 16:19:58.216921 fron
  12. Jul 29 16:20:23 s1 ceph-osd[5828]: 2018-07-29 16:20:23.398298 7f4ab802a700 -1 osd.15 17656 heartbeat_check: no reply from 192.168.121.30:6819 osd.4 since back 2018-07-29 16:19:58.216921 fron
  13. Jul 29 16:20:23 s1 ceph-osd[3546]: 2018-07-29 16:20:23.938528 7f19a1430700 -1 osd.14 17656 heartbeat_check: no reply from 192.168.121.30:6819 osd.4 since back 2018-07-29 16:19:58.859065 fron
  14. Jul 29 16:20:23 s1 ceph-osd[6234]: 2018-07-29 16:20:23.972407 7f7e5bc5f700 -1 osd.10 17656 heartbeat_check: no reply from 192.168.121.30:6819 osd.4 since back 2018-07-29 16:19:56.901979 fron
  15. Jul 29 16:20:23 s1 ceph-osd[5409]: 2018-07-29 16:20:23.995218 7fddb2420700 -1 osd.11 17656 heartbeat_check: no reply from 192.168.121.30:6819 osd.4 since back 2018-07-29 16:19:58.225848 fron
  16. Jul 29 16:20:24 s1 ceph-osd[6234]: 2018-07-29 16:20:24.311341 7f7e40d21700 -1 osd.10 17656 heartbeat_check: no reply from 192.168.121.30:6819 osd.4 since back 2018-07-29 16:19:56.901979 fron
  17. Jul 29 16:20:24 s1 ceph-osd[5828]: 2018-07-29 16:20:24.398453 7f4ab802a700 -1 osd.15 17656 heartbeat_check: no reply from 192.168.121.30:6819 osd.4 since back 2018-07-29 16:19:58.216921 fron
  18. Jul 29 16:20:24 s1 corosync[2836]: [TOTEM ] A processor failed, forming new configuration.
  19. Jul 29 16:20:24 s1 ceph-osd[3546]: 2018-07-29 16:20:24.938653 7f19a1430700 -1 osd.14 17656 heartbeat_check: no reply from 192.168.121.30:6819 osd.4 since back 2018-07-29 16:19:58.859065 fron
  20. Jul 29 16:20:24 s1 ceph-osd[6234]: 2018-07-29 16:20:24.972631 7f7e5bc5f700 -1 osd.10 17656 heartbeat_check: no reply from 192.168.121.30:6819 osd.4 since back 2018-07-29 16:19:56.901979 fron
  21. Jul 29 16:20:24 s1 ceph-osd[5409]: 2018-07-29 16:20:24.995316 7fddb2420700 -1 osd.11 17656 heartbeat_check: no reply from 192.168.121.30:6819 osd.4 since back 2018-07-29 16:19:58.225848 fron
  22. Jul 29 16:20:25 s1 ceph-osd[5828]: 2018-07-29 16:20:25.398714 7f4ab802a700 -1 osd.15 17656 heartbeat_check: no reply from 192.168.121.30:6819 osd.4 since back 2018-07-29 16:19:58.216921 fron
  23. Jul 29 16:20:25 s1 ceph-osd[3546]: 2018-07-29 16:20:25.665276 7f19864d0700 -1 osd.14 17656 heartbeat_check: no reply from 192.168.121.30:6819 osd.4 since back 2018-07-29 16:19:58.859065 fron
  24. Jul 29 16:20:25 s1 ceph-osd[3546]: 2018-07-29 16:20:25.938853 7f19a1430700 -1 osd.14 17656 heartbeat_check: no reply from 192.168.121.30:6819 osd.4 since back 2018-07-29 16:19:58.859065 fron
  25. Jul 29 16:20:25 s1 ceph-osd[6234]: 2018-07-29 16:20:25.972853 7f7e5bc5f700 -1 osd.10 17656 heartbeat_check: no reply from 192.168.121.30:6819 osd.4 since back 2018-07-29 16:19:56.901979 fron
  26. Jul 29 16:20:25 s1 ceph-osd[5409]: 2018-07-29 16:20:25.995468 7fddb2420700 -1 osd.11 17656 heartbeat_check: no reply from 192.168.121.30:6819 osd.4 since back 2018-07-29 16:19:58.225848 fron
  27. Jul 29 16:20:26 s1 ceph-osd[5409]: 2018-07-29 16:20:26.033075 7fdd973be700 -1 osd.11 17656 heartbeat_check: no reply from 192.168.121.30:6819 osd.4 since back 2018-07-29 16:19:58.225848 fron
  28. Jul 29 16:20:26 s1 ceph-osd[5828]: 2018-07-29 16:20:26.398978 7f4ab802a700 -1 osd.15 17656 heartbeat_check: no reply from 192.168.121.30:6819 osd.4 since back 2018-07-29 16:19:58.216921 fron
  29. Jul 29 16:20:26 s1 ceph-osd[6234]: 2018-07-29 16:20:26.612339 7f7e40d21700 -1 osd.10 17656 heartbeat_check: no reply from 192.168.121.30:6819 osd.4 since back 2018-07-29 16:19:56.901979 fron
  30. Jul 29 16:20:26 s1 ceph-osd[3546]: 2018-07-29 16:20:26.939014 7f19a1430700 -1 osd.14 17656 heartbeat_check: no reply from 192.168.121.30:6819 osd.4 since back 2018-07-29 16:19:58.859065 fron
  31. Jul 29 16:20:26 s1 ceph-osd[6234]: 2018-07-29 16:20:26.972965 7f7e5bc5f700 -1 osd.10 17656 heartbeat_check: no reply from 192.168.121.30:6819 osd.4 since back 2018-07-29 16:19:56.901979 fron
  32. Jul 29 16:20:26 s1 ceph-osd[5409]: 2018-07-29 16:20:26.995638 7fddb2420700 -1 osd.11 17656 heartbeat_check: no reply from 192.168.121.30:6819 osd.4 since back 2018-07-29 16:19:58.225848 fron
  33. Jul 29 16:20:27 s1 ceph-osd[6234]: 2018-07-29 16:20:27.113406 7f7e40d21700 -1 osd.10 17656 heartbeat_check: no reply from 192.168.121.30:6819 osd.4 since back 2018-07-29 16:19:56.901979 fron
  34. Jul 29 16:20:27 s1 ceph-osd[3546]: 2018-07-29 16:20:27.365976 7f19864d0700 -1 osd.14 17656 heartbeat_check: no reply from 192.168.121.30:6819 osd.4 since back 2018-07-29 16:19:58.859065 fron
  35. Jul 29 16:20:27 s1 ceph-osd[5828]: 2018-07-29 16:20:27.399118 7f4ab802a700 -1 osd.15 17656 heartbeat_check: no reply from 192.168.121.30:6819 osd.4 since back 2018-07-29 16:19:58.216921 fron
  36. Jul 29 16:20:27 s1 ceph-osd[5828]: 2018-07-29 16:20:27.424520 7f4a9ce7f700 -1 osd.15 17656 heartbeat_check: no reply from 192.168.121.30:6819 osd.4 since back 2018-07-29 16:19:58.216921 fron
  37. Jul 29 16:20:27 s1 ceph-osd[5828]: 2018-07-29 16:20:27.925347 7f4a9ce7f700 -1 osd.15 17656 heartbeat_check: no reply from 192.168.121.30:6819 osd.4 since back 2018-07-29 16:19:58.216921 fron
  38. Jul 29 16:20:27 s1 ceph-osd[3546]: 2018-07-29 16:20:27.939131 7f19a1430700 -1 osd.14 17656 heartbeat_check: no reply from 192.168.121.30:6819 osd.4 since back 2018-07-29 16:19:58.859065 fron
  39. Jul 29 16:20:27 s1 ceph-osd[6234]: 2018-07-29 16:20:27.973065 7f7e5bc5f700 -1 osd.10 17656 heartbeat_check: no reply from 192.168.121.30:6819 osd.4 since back 2018-07-29 16:19:56.901979 fron
  40. Jul 29 16:20:27 s1 corosync[2836]: [TOTEM ] A new membership (192.168.121.30:2952) was formed. Members
  41. Jul 29 16:20:27 s1 ceph-osd[5409]: 2018-07-29 16:20:27.995761 7fddb2420700 -1 osd.11 17656 heartbeat_check: no reply from 192.168.121.30:6819 osd.4 since back 2018-07-29 16:19:58.225848 fron
  42. Jul 29 16:20:28 s1 ceph-osd[5409]: 2018-07-29 16:20:28.333658 7fdd973be700 -1 osd.11 17656 heartbeat_check: no reply from 192.168.121.30:6819 osd.4 since back 2018-07-29 16:19:58.225848 fron
  43. Jul 29 16:20:28 s1 ceph-osd[5828]: 2018-07-29 16:20:28.399263 7f4ab802a700 -1 osd.15 17656 heartbeat_check: no reply from 192.168.121.30:6819 osd.4 since back 2018-07-29 16:19:58.216921 fron
  44. Jul 29 16:20:28 s1 ceph-osd[3546]: 2018-07-29 16:20:28.939315 7f19a1430700 -1 osd.14 17656 heartbeat_check: no reply from 192.168.121.30:6819 osd.4 since back 2018-07-29 16:19:58.859065 fron
  45. Jul 29 16:20:28 s1 ceph-osd[6234]: 2018-07-29 16:20:28.973204 7f7e5bc5f700 -1 osd.10 17656 heartbeat_check: no reply from 192.168.121.30:6819 osd.4 since back 2018-07-29 16:19:56.901979 fron
  46. Jul 29 16:20:28 s1 ceph-osd[5409]: 2018-07-29 16:20:28.995861 7fddb2420700 -1 osd.11 17656 heartbeat_check: no reply from 192.168.121.30:6819 osd.4 since back 2018-07-29 16:19:58.225848 fron
  47. lines 352703-352752/352752 (END)
  48.  
  49. Yet I can ping hp1 (192.168.121.30) from s1, I can omping as well, all services seem to be running, but for some reason I get this osd4 error. osd4 is up.
  50.  
  51. # ceph osd tree
  52. ID WEIGHT   TYPE NAME      UP/DOWN REWEIGHT PRIMARY-AFFINITY
  53. -1 18.82312 root default                                    
  54. -3  1.61993     host s1                                      
  55. 10  0.26999         osd.10      up  0.95001          1.00000
  56. 11  0.26999         osd.11      up  1.00000          1.00000
  57. 12  0.26999         osd.12      up  0.95001          1.00000
  58. 13  0.26999         osd.13      up  1.00000          1.00000
  59. 14  0.26999         osd.14      up  0.95001          1.00000
  60. 15  0.26999         osd.15      up  0.95001          1.00000
  61. -4  1.81000     host s2                                      
  62. 16  1.81000         osd.16      up  1.00000          1.00000
  63. -5  1.81839     host sm1                                    
  64.  8  0.90919         osd.8     down        0          1.00000
  65.  5  0.90919         osd.5     down        0          1.00000
  66. -2 10.86163     host hp1                                    
  67.  7  0.67699         osd.7       up  1.00000          1.00000
  68.  9  1.81360         osd.9       up  1.00000          1.00000
  69. 17  0.67699         osd.17      up  1.00000          1.00000
  70. 18  0.67699         osd.18      up  1.00000          1.00000
  71.  0  1.81360         osd.0       up  1.00000          1.00000
  72.  1  0.90430         osd.1       up  1.00000          1.00000
  73.  2  0.90430         osd.2       up  1.00000          1.00000
  74.  3  0.90430         osd.3       up  1.00000          1.00000
  75.  4  1.81360         osd.4       up  1.00000          1.00000
  76.  6  0.67699         osd.6       up  1.00000          1.00000
  77. -6  0.90439     host s4                                      
  78. 19  0.90439         osd.19      up  1.00000          1.00000
  79. -7  0.90439     host s5                                      
  80. 20  0.90439         osd.20    down        0          1.00000
  81. -8  0.90439     host s6                                      
  82. 21  0.90439         osd.21      up  1.00000          1.00000
  83.  
  84. Somehow I think this has to do with using bonded ethernet ports.  But why and what's the problem?
  85.  
  86. # cat /etc/network/interfaces
  87. auto lo
  88. iface lo inet loopback
  89.  
  90. auto eth0
  91. iface eth0 inet manual
  92.  
  93. auto eth1
  94. iface eth1 inet manual
  95.  
  96. auto bond0
  97. iface bond0 inet manual
  98.     slaves eth0 eth1
  99.     bond_miimon 100
  100.     bond_mode 802.3ad
  101.     bond_xmit_hash_policy layer2
  102.  
  103. auto vmbr0
  104. iface vmbr0 inet static
  105.     address 192.168.121.33
  106.     netmask 255.255.255.0
  107.     gateway 192.168.121.1
  108.     bridge_ports bond0
  109.     bridge_stp off
  110.     bridge_fd 0
  111.     post-up /sbin/ethtool -K $IFACE tx off
  112.  
  113. Status:
  114.  
  115. # ip addr show
  116. 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1
  117.    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
  118.    inet 127.0.0.1/8 scope host lo
  119.       valid_lft forever preferred_lft forever
  120.    inet6 ::1/128 scope host
  121.       valid_lft forever preferred_lft forever
  122. 2: eth0: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP group default qlen 1000
  123.    link/ether 00:21:28:8e:b6:52 brd ff:ff:ff:ff:ff:ff
  124. 3: eth1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP group default qlen 1000
  125.    link/ether 00:21:28:8e:b6:52 brd ff:ff:ff:ff:ff:ff
  126. 4: eth2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
  127.    link/ether 00:21:28:8e:b6:54 brd ff:ff:ff:ff:ff:ff
  128. 5: eth3: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
  129.    link/ether 00:21:28:8e:b6:55 brd ff:ff:ff:ff:ff:ff
  130. 6: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr0 state UP group default qlen 1000
  131.    link/ether 00:21:28:8e:b6:52 brd ff:ff:ff:ff:ff:ff
  132. 7: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
  133.    link/ether 00:21:28:8e:b6:52 brd ff:ff:ff:ff:ff:ff
  134.    inet 192.168.121.33/24 brd 192.168.121.255 scope global vmbr0
  135.       valid_lft forever preferred_lft forever
  136.    inet6 fe80::221:28ff:fe8e:b652/64 scope link
  137.       valid_lft forever preferred_lft forever
  138. 8: tap106i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr0 state UNKNOWN group default qlen 1000
  139.    link/ether 16:d3:99:1e:e3:44 brd ff:ff:ff:ff:ff:ff
  140. 9: tap108i0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
  141.    link/ether 7a:07:7f:14:71:a7 brd ff:ff:ff:ff:ff:ff
  142. 10: tap115i0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
  143.    link/ether 92:0b:0e:68:5b:3e brd ff:ff:ff:ff:ff:ff
  144. 11: tap116i0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
  145.    link/ether 12:46:e1:1a:29:5c brd ff:ff:ff:ff:ff:ff
  146. 12: tap121i0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
  147.    link/ether 8e:16:f9:3f:dc:02 brd ff:ff:ff:ff:ff:ff
  148. 13: tap122i0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
  149.    link/ether 42:6b:00:67:16:ab brd ff:ff:ff:ff:ff:ff
  150. 14: tap124i0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
  151.    link/ether ea:f3:ef:eb:68:3a brd ff:ff:ff:ff:ff:ff
Advertisement
Add Comment
Please, Sign In to add comment
Public Pastes
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!