API tools faq
paste
Advertisement
dfarrell07

subm_random_fail_20200826

dfarrell07
Aug 26th, 2020
239
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 448.86 KB | None | 0 0
raw download clone embed print report
  1. 2020-08-25T18:15:32.5540200Z ##[section]Starting: Request a runner to run this job
  2. 2020-08-25T18:15:33.4630358Z Can't find any online and idle self-hosted runner in current repository that matches the required labels: 'ubuntu-latest'
  3. 2020-08-25T18:15:33.4630467Z Can't find any online and idle self-hosted runner in current repository's account/organization that matches the required labels: 'ubuntu-latest'
  4. 2020-08-25T18:15:33.4630676Z Found online and idle hosted runner in current repository's account/organization that matches the required labels: 'ubuntu-latest'
  5. 2020-08-25T18:15:33.6219721Z ##[section]Finishing: Request a runner to run this job
  6. 2020-08-25T18:15:44.3951766Z Current runner version: '2.273.0'
  7. 2020-08-25T18:15:44.3986194Z ##[group]Operating System
  8. 2020-08-25T18:15:44.3987683Z Ubuntu
  9. 2020-08-25T18:15:44.3987860Z 18.04.5
  10. 2020-08-25T18:15:44.3988059Z LTS
  11. 2020-08-25T18:15:44.3988262Z ##[endgroup]
  12. 2020-08-25T18:15:44.3988467Z ##[group]Virtual Environment
  13. 2020-08-25T18:15:44.3988725Z Environment: ubuntu-18.04
  14. 2020-08-25T18:15:44.3988931Z Version: 20200817.1
  15. 2020-08-25T18:15:44.3989268Z Included Software: https://github.com/actions/virtual-environments/blob/ubuntu18/20200817.1/images/linux/Ubuntu1804-README.md
  16. 2020-08-25T18:15:44.3989533Z ##[endgroup]
  17. 2020-08-25T18:15:44.3990481Z Prepare workflow directory
  18. 2020-08-25T18:15:44.4191389Z Prepare all required actions
  19. 2020-08-25T18:15:44.4202823Z Download action repository 'actions/checkout@master'
  20. 2020-08-25T18:15:46.7565811Z ##[group]Run actions/checkout@master
  21. 2020-08-25T18:15:46.7566312Z with:
  22. 2020-08-25T18:15:46.7566688Z repository: submariner-io/lighthouse
  23. 2020-08-25T18:15:46.7567103Z token: ***
  24. 2020-08-25T18:15:46.7567368Z ssh-strict: true
  25. 2020-08-25T18:15:46.7567526Z persist-credentials: true
  26. 2020-08-25T18:15:46.7567790Z clean: true
  27. 2020-08-25T18:15:46.7567998Z fetch-depth: 1
  28. 2020-08-25T18:15:46.7568203Z lfs: false
  29. 2020-08-25T18:15:46.7568405Z submodules: false
  30. 2020-08-25T18:15:46.7568658Z ##[endgroup]
  31. 2020-08-25T18:15:47.6886745Z Syncing repository: submariner-io/lighthouse
  32. 2020-08-25T18:15:47.6892329Z ##[group]Getting Git version info
  33. 2020-08-25T18:15:47.6893231Z Working directory is '/home/runner/work/lighthouse/lighthouse'
  34. 2020-08-25T18:15:47.6893793Z [command]/usr/bin/git version
  35. 2020-08-25T18:15:47.6894031Z git version 2.28.0
  36. 2020-08-25T18:15:47.6894782Z ##[endgroup]
  37. 2020-08-25T18:15:47.6895539Z Deleting the contents of '/home/runner/work/lighthouse/lighthouse'
  38. 2020-08-25T18:15:47.6896917Z ##[group]Initializing the repository
  39. 2020-08-25T18:15:47.6897349Z [command]/usr/bin/git init /home/runner/work/lighthouse/lighthouse
  40. 2020-08-25T18:15:47.6897682Z Initialized empty Git repository in /home/runner/work/lighthouse/lighthouse/.git/
  41. 2020-08-25T18:15:47.6898427Z [command]/usr/bin/git remote add origin https://github.com/submariner-io/lighthouse
  42. 2020-08-25T18:15:47.6898920Z ##[endgroup]
  43. 2020-08-25T18:15:47.6899340Z ##[group]Disabling automatic garbage collection
  44. 2020-08-25T18:15:47.6899959Z [command]/usr/bin/git config --local gc.auto 0
  45. 2020-08-25T18:15:47.6900398Z ##[endgroup]
  46. 2020-08-25T18:15:47.6902992Z ##[group]Setting up auth
  47. 2020-08-25T18:15:47.6970980Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand
  48. 2020-08-25T18:15:47.7006382Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :
  49. 2020-08-25T18:15:47.7007349Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader
  50. 2020-08-25T18:15:47.7008308Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :
  51. 2020-08-25T18:15:47.7009153Z [command]/usr/bin/git config --local http.https://github.com/.extraheader AUTHORIZATION: basic ***
  52. 2020-08-25T18:15:47.7009682Z ##[endgroup]
  53. 2020-08-25T18:15:47.7010096Z ##[group]Fetching the repository
  54. 2020-08-25T18:15:47.7010902Z [command]/usr/bin/git -c protocol.version=2 fetch --no-tags --prune --progress --no-recurse-submodules --depth=1 origin +f91b859307fe960751a14cfccc2555b26a4d3914:refs/remotes/pull/267/merge
  55. 2020-08-25T18:15:49.5476110Z remote: Enumerating objects: 145, done.
  56. 2020-08-25T18:15:49.5476849Z remote: Counting objects: 0% (1/145)
  57. 2020-08-25T18:15:49.5477084Z remote: Counting objects: 1% (2/145)
  58. 2020-08-25T18:15:49.5477519Z remote: Counting objects: 2% (3/145)
  59. 2020-08-25T18:15:49.5477723Z remote: Counting objects: 3% (5/145)
  60. 2020-08-25T18:15:49.5478268Z remote: Counting objects: 4% (6/145)
  61. 2020-08-25T18:15:49.5479522Z remote: Counting objects: 5% (8/145)
  62. 2020-08-25T18:15:49.5479712Z remote: Counting objects: 6% (9/145)
  63. 2020-08-25T18:15:49.5479844Z remote: Counting objects: 7% (11/145)
  64. 2020-08-25T18:15:49.5480051Z remote: Counting objects: 8% (12/145)
  65. 2020-08-25T18:15:49.5480289Z remote: Counting objects: 9% (14/145)
  66. 2020-08-25T18:15:49.5480569Z remote: Counting objects: 10% (15/145)
  67. 2020-08-25T18:15:49.5480750Z remote: Counting objects: 11% (16/145)
  68. 2020-08-25T18:15:49.5480959Z remote: Counting objects: 12% (18/145)
  69. 2020-08-25T18:15:49.5481178Z remote: Counting objects: 13% (19/145)
  70. 2020-08-25T18:15:49.5481365Z remote: Counting objects: 14% (21/145)
  71. 2020-08-25T18:15:49.5481637Z remote: Counting objects: 15% (22/145)
  72. 2020-08-25T18:15:49.5481876Z remote: Counting objects: 16% (24/145)
  73. 2020-08-25T18:15:49.5482203Z remote: Counting objects: 17% (25/145)
  74. 2020-08-25T18:15:49.5482387Z remote: Counting objects: 18% (27/145)
  75. 2020-08-25T18:15:49.5482622Z remote: Counting objects: 19% (28/145)
  76. 2020-08-25T18:15:49.5482801Z remote: Counting objects: 20% (29/145)
  77. 2020-08-25T18:15:49.5483143Z remote: Counting objects: 21% (31/145)
  78. 2020-08-25T18:15:49.5483348Z remote: Counting objects: 22% (32/145)
  79. 2020-08-25T18:15:49.5483472Z remote: Counting objects: 23% (34/145)
  80. 2020-08-25T18:15:49.5484040Z remote: Counting objects: 24% (35/145)
  81. 2020-08-25T18:15:49.5484232Z remote: Counting objects: 25% (37/145)
  82. 2020-08-25T18:15:49.5484407Z remote: Counting objects: 26% (38/145)
  83. 2020-08-25T18:15:49.5484581Z remote: Counting objects: 27% (40/145)
  84. 2020-08-25T18:15:49.5484764Z remote: Counting objects: 28% (41/145)
  85. 2020-08-25T18:15:49.5484939Z remote: Counting objects: 29% (43/145)
  86. 2020-08-25T18:15:49.5485113Z remote: Counting objects: 30% (44/145)
  87. 2020-08-25T18:15:49.5485234Z remote: Counting objects: 31% (45/145)
  88. 2020-08-25T18:15:49.5485462Z remote: Counting objects: 32% (47/145)
  89. 2020-08-25T18:15:49.5485657Z remote: Counting objects: 33% (48/145)
  90. 2020-08-25T18:15:49.5485837Z remote: Counting objects: 34% (50/145)
  91. 2020-08-25T18:15:49.5486013Z remote: Counting objects: 35% (51/145)
  92. 2020-08-25T18:15:49.5486373Z remote: Counting objects: 36% (53/145)
  93. 2020-08-25T18:15:49.5486553Z remote: Counting objects: 37% (54/145)
  94. 2020-08-25T18:15:49.5486734Z remote: Counting objects: 38% (56/145)
  95. 2020-08-25T18:15:49.5487067Z remote: Counting objects: 39% (57/145)
  96. 2020-08-25T18:15:49.5487190Z remote: Counting objects: 40% (58/145)
  97. 2020-08-25T18:15:49.5487389Z remote: Counting objects: 41% (60/145)
  98. 2020-08-25T18:15:49.5487591Z remote: Counting objects: 42% (61/145)
  99. 2020-08-25T18:15:49.5488037Z remote: Counting objects: 43% (63/145)
  100. 2020-08-25T18:15:49.5488217Z remote: Counting objects: 44% (64/145)
  101. 2020-08-25T18:15:49.5488408Z remote: Counting objects: 45% (66/145)
  102. 2020-08-25T18:15:49.5488589Z remote: Counting objects: 46% (67/145)
  103. 2020-08-25T18:15:49.5488768Z remote: Counting objects: 47% (69/145)
  104. 2020-08-25T18:15:49.5488948Z remote: Counting objects: 48% (70/145)
  105. 2020-08-25T18:15:49.5489077Z remote: Counting objects: 49% (72/145)
  106. 2020-08-25T18:15:49.5489298Z remote: Counting objects: 50% (73/145)
  107. 2020-08-25T18:15:49.5489506Z remote: Counting objects: 51% (74/145)
  108. 2020-08-25T18:15:49.5490110Z remote: Counting objects: 52% (76/145)
  109. 2020-08-25T18:15:49.5490324Z remote: Counting objects: 53% (77/145)
  110. 2020-08-25T18:15:49.5490505Z remote: Counting objects: 54% (79/145)
  111. 2020-08-25T18:15:49.5490692Z remote: Counting objects: 55% (80/145)
  112. 2020-08-25T18:15:49.5490871Z remote: Counting objects: 56% (82/145)
  113. 2020-08-25T18:15:49.5490999Z remote: Counting objects: 57% (83/145)
  114. 2020-08-25T18:15:49.5491180Z remote: Counting objects: 58% (85/145)
  115. 2020-08-25T18:15:49.5491533Z remote: Counting objects: 59% (86/145)
  116. 2020-08-25T18:15:49.5491719Z remote: Counting objects: 60% (87/145)
  117. 2020-08-25T18:15:49.5491905Z remote: Counting objects: 61% (89/145)
  118. 2020-08-25T18:15:49.5492086Z remote: Counting objects: 62% (90/145)
  119. 2020-08-25T18:15:49.5492263Z remote: Counting objects: 63% (92/145)
  120. 2020-08-25T18:15:49.5492444Z remote: Counting objects: 64% (93/145)
  121. 2020-08-25T18:15:49.5492571Z remote: Counting objects: 65% (95/145)
  122. 2020-08-25T18:15:49.5492750Z remote: Counting objects: 66% (96/145)
  123. 2020-08-25T18:15:49.5492945Z remote: Counting objects: 67% (98/145)
  124. 2020-08-25T18:15:49.5493398Z remote: Counting objects: 68% (99/145)
  125. 2020-08-25T18:15:49.5493600Z remote: Counting objects: 69% (101/145)
  126. 2020-08-25T18:15:49.5493778Z remote: Counting objects: 70% (102/145)
  127. 2020-08-25T18:15:49.5493955Z remote: Counting objects: 71% (103/145)
  128. 2020-08-25T18:15:49.5494144Z remote: Counting objects: 72% (105/145)
  129. 2020-08-25T18:15:49.5494268Z remote: Counting objects: 73% (106/145)
  130. 2020-08-25T18:15:49.5494443Z remote: Counting objects: 74% (108/145)
  131. 2020-08-25T18:15:49.5494640Z remote: Counting objects: 75% (109/145)
  132. 2020-08-25T18:15:49.5494818Z remote: Counting objects: 76% (111/145)
  133. 2020-08-25T18:15:49.5495017Z remote: Counting objects: 77% (112/145)
  134. 2020-08-25T18:15:49.5495196Z remote: Counting objects: 78% (114/145)
  135. 2020-08-25T18:15:49.5495372Z remote: Counting objects: 79% (115/145)
  136. 2020-08-25T18:15:49.6577327Z remote: Counting objects: 80% (116/145)
  137. 2020-08-25T18:15:49.6577598Z remote: Counting objects: 81% (118/145)
  138. 2020-08-25T18:15:49.6577856Z remote: Counting objects: 82% (119/145)
  139. 2020-08-25T18:15:49.6578040Z remote: Counting objects: 83% (121/145)
  140. 2020-08-25T18:15:49.6578322Z remote: Counting objects: 84% (122/145)
  141. 2020-08-25T18:15:49.6578515Z remote: Counting objects: 85% (124/145)
  142. 2020-08-25T18:15:49.6578707Z remote: Counting objects: 86% (125/145)
  143. 2020-08-25T18:15:49.6578907Z remote: Counting objects: 87% (127/145)
  144. 2020-08-25T18:15:49.6579030Z remote: Counting objects: 88% (128/145)
  145. 2020-08-25T18:15:49.6579948Z remote: Counting objects: 89% (130/145)
  146. 2020-08-25T18:15:49.6580697Z remote: Counting objects: 90% (131/145)
  147. 2020-08-25T18:15:49.6580978Z remote: Counting objects: 91% (132/145)
  148. 2020-08-25T18:15:49.6581154Z remote: Counting objects: 92% (134/145)
  149. 2020-08-25T18:15:49.6581329Z remote: Counting objects: 93% (135/145)
  150. 2020-08-25T18:15:49.6581549Z remote: Counting objects: 94% (137/145)
  151. 2020-08-25T18:15:49.6581730Z remote: Counting objects: 95% (138/145)
  152. 2020-08-25T18:15:49.6581853Z remote: Counting objects: 96% (140/145)
  153. 2020-08-25T18:15:49.6582026Z remote: Counting objects: 97% (141/145)
  154. 2020-08-25T18:15:49.6582209Z remote: Counting objects: 98% (143/145)
  155. 2020-08-25T18:15:49.6582378Z remote: Counting objects: 99% (144/145)
  156. 2020-08-25T18:15:49.6582574Z remote: Counting objects: 100% (145/145)
  157. 2020-08-25T18:15:49.6582751Z remote: Counting objects: 100% (145/145), done.
  158. 2020-08-25T18:15:49.6582940Z remote: Compressing objects: 0% (1/122)
  159. 2020-08-25T18:15:49.6583136Z remote: Compressing objects: 1% (2/122)
  160. 2020-08-25T18:15:49.6583258Z remote: Compressing objects: 2% (3/122)
  161. 2020-08-25T18:15:49.6583429Z remote: Compressing objects: 3% (4/122)
  162. 2020-08-25T18:15:49.6583600Z remote: Compressing objects: 4% (5/122)
  163. 2020-08-25T18:15:49.6583778Z remote: Compressing objects: 5% (7/122)
  164. 2020-08-25T18:15:49.6583953Z remote: Compressing objects: 6% (8/122)
  165. 2020-08-25T18:15:49.6584177Z remote: Compressing objects: 7% (9/122)
  166. 2020-08-25T18:15:49.6584351Z remote: Compressing objects: 8% (10/122)
  167. 2020-08-25T18:15:49.6585028Z remote: Compressing objects: 9% (11/122)
  168. 2020-08-25T18:15:49.6585211Z remote: Compressing objects: 10% (13/122)
  169. 2020-08-25T18:15:49.6585339Z remote: Compressing objects: 11% (14/122)
  170. 2020-08-25T18:15:49.6585510Z remote: Compressing objects: 12% (15/122)
  171. 2020-08-25T18:15:49.6585685Z remote: Compressing objects: 13% (16/122)
  172. 2020-08-25T18:15:49.6897345Z remote: Compressing objects: 14% (18/122)
  173. 2020-08-25T18:15:49.6897667Z remote: Compressing objects: 15% (19/122)
  174. 2020-08-25T18:15:49.6897888Z remote: Compressing objects: 16% (20/122)
  175. 2020-08-25T18:15:49.6898080Z remote: Compressing objects: 17% (21/122)
  176. 2020-08-25T18:15:49.6898456Z remote: Compressing objects: 18% (22/122)
  177. 2020-08-25T18:15:49.6898685Z remote: Compressing objects: 19% (24/122)
  178. 2020-08-25T18:15:49.6898821Z remote: Compressing objects: 20% (25/122)
  179. 2020-08-25T18:15:49.6899010Z remote: Compressing objects: 21% (26/122)
  180. 2020-08-25T18:15:49.6899215Z remote: Compressing objects: 22% (27/122)
  181. 2020-08-25T18:15:49.6899399Z remote: Compressing objects: 23% (29/122)
  182. 2020-08-25T18:15:49.6899641Z remote: Compressing objects: 24% (30/122)
  183. 2020-08-25T18:15:49.6899829Z remote: Compressing objects: 25% (31/122)
  184. 2020-08-25T18:15:49.6900072Z remote: Compressing objects: 26% (32/122)
  185. 2020-08-25T18:15:49.6900258Z remote: Compressing objects: 27% (33/122)
  186. 2020-08-25T18:15:49.6900443Z remote: Compressing objects: 28% (35/122)
  187. 2020-08-25T18:15:49.6900575Z remote: Compressing objects: 29% (36/122)
  188. 2020-08-25T18:15:49.6900761Z remote: Compressing objects: 30% (37/122)
  189. 2020-08-25T18:15:49.6901111Z remote: Compressing objects: 31% (38/122)
  190. 2020-08-25T18:15:49.6901455Z remote: Compressing objects: 32% (40/122)
  191. 2020-08-25T18:15:49.6901667Z remote: Compressing objects: 33% (41/122)
  192. 2020-08-25T18:15:49.6901841Z remote: Compressing objects: 34% (42/122)
  193. 2020-08-25T18:15:49.6902025Z remote: Compressing objects: 35% (43/122)
  194. 2020-08-25T18:15:49.6902359Z remote: Compressing objects: 36% (44/122)
  195. 2020-08-25T18:15:49.6902480Z remote: Compressing objects: 37% (46/122)
  196. 2020-08-25T18:15:49.6902655Z remote: Compressing objects: 38% (47/122)
  197. 2020-08-25T18:15:49.6902825Z remote: Compressing objects: 39% (48/122)
  198. 2020-08-25T18:15:49.6902996Z remote: Compressing objects: 40% (49/122)
  199. 2020-08-25T18:15:49.6903327Z remote: Compressing objects: 41% (51/122)
  200. 2020-08-25T18:15:49.6903518Z remote: Compressing objects: 42% (52/122)
  201. 2020-08-25T18:15:49.6903714Z remote: Compressing objects: 43% (53/122)
  202. 2020-08-25T18:15:49.6903945Z remote: Compressing objects: 44% (54/122)
  203. 2020-08-25T18:15:49.6904117Z remote: Compressing objects: 45% (55/122)
  204. 2020-08-25T18:15:49.6904310Z remote: Compressing objects: 46% (57/122)
  205. 2020-08-25T18:15:49.6904487Z remote: Compressing objects: 47% (58/122)
  206. 2020-08-25T18:15:49.6904608Z remote: Compressing objects: 48% (59/122)
  207. 2020-08-25T18:15:49.6904983Z remote: Compressing objects: 49% (60/122)
  208. 2020-08-25T18:15:49.6905180Z remote: Compressing objects: 50% (61/122)
  209. 2020-08-25T18:15:49.6905568Z remote: Compressing objects: 51% (63/122)
  210. 2020-08-25T18:15:49.6906015Z remote: Compressing objects: 52% (64/122)
  211. 2020-08-25T18:15:49.6906201Z remote: Compressing objects: 53% (65/122)
  212. 2020-08-25T18:15:49.6906384Z remote: Compressing objects: 54% (66/122)
  213. 2020-08-25T18:15:49.6906568Z remote: Compressing objects: 55% (68/122)
  214. 2020-08-25T18:15:49.6906699Z remote: Compressing objects: 56% (69/122)
  215. 2020-08-25T18:15:49.6910173Z remote: Compressing objects: 57% (70/122)
  216. 2020-08-25T18:15:49.6910398Z remote: Compressing objects: 58% (71/122)
  217. 2020-08-25T18:15:49.6910577Z remote: Compressing objects: 59% (72/122)
  218. 2020-08-25T18:15:49.6910986Z remote: Compressing objects: 60% (74/122)
  219. 2020-08-25T18:15:49.6911160Z remote: Compressing objects: 61% (75/122)
  220. 2020-08-25T18:15:49.7281989Z remote: Compressing objects: 62% (76/122)
  221. 2020-08-25T18:15:49.7282221Z remote: Compressing objects: 63% (77/122)
  222. 2020-08-25T18:15:49.7282461Z remote: Compressing objects: 64% (79/122)
  223. 2020-08-25T18:15:49.7282644Z remote: Compressing objects: 65% (80/122)
  224. 2020-08-25T18:15:49.7282822Z remote: Compressing objects: 66% (81/122)
  225. 2020-08-25T18:15:49.7282997Z remote: Compressing objects: 67% (82/122)
  226. 2020-08-25T18:15:49.7283190Z remote: Compressing objects: 68% (83/122)
  227. 2020-08-25T18:15:49.7283464Z remote: Compressing objects: 69% (85/122)
  228. 2020-08-25T18:15:49.7283700Z remote: Compressing objects: 70% (86/122)
  229. 2020-08-25T18:15:49.7283876Z remote: Compressing objects: 71% (87/122)
  230. 2020-08-25T18:15:49.7284052Z remote: Compressing objects: 72% (88/122)
  231. 2020-08-25T18:15:49.7284270Z remote: Compressing objects: 73% (90/122)
  232. 2020-08-25T18:15:49.7284444Z remote: Compressing objects: 74% (91/122)
  233. 2020-08-25T18:15:49.7284618Z remote: Compressing objects: 75% (92/122)
  234. 2020-08-25T18:15:49.7284812Z remote: Compressing objects: 76% (93/122)
  235. 2020-08-25T18:15:49.7284936Z remote: Compressing objects: 77% (94/122)
  236. 2020-08-25T18:15:49.7285110Z remote: Compressing objects: 78% (96/122)
  237. 2020-08-25T18:15:49.7285283Z remote: Compressing objects: 79% (97/122)
  238. 2020-08-25T18:15:49.7285454Z remote: Compressing objects: 80% (98/122)
  239. 2020-08-25T18:15:49.7285625Z remote: Compressing objects: 81% (99/122)
  240. 2020-08-25T18:15:49.7285828Z remote: Compressing objects: 82% (101/122)
  241. 2020-08-25T18:15:49.7286014Z remote: Compressing objects: 83% (102/122)
  242. 2020-08-25T18:15:49.7286191Z remote: Compressing objects: 84% (103/122)
  243. 2020-08-25T18:15:49.7286315Z remote: Compressing objects: 85% (104/122)
  244. 2020-08-25T18:15:49.7286516Z remote: Compressing objects: 86% (105/122)
  245. 2020-08-25T18:15:49.7286690Z remote: Compressing objects: 87% (107/122)
  246. 2020-08-25T18:15:49.7286862Z remote: Compressing objects: 88% (108/122)
  247. 2020-08-25T18:15:49.7287113Z remote: Compressing objects: 89% (109/122)
  248. 2020-08-25T18:15:49.7287283Z remote: Compressing objects: 90% (110/122)
  249. 2020-08-25T18:15:49.7287477Z remote: Compressing objects: 91% (112/122)
  250. 2020-08-25T18:15:49.7287655Z remote: Compressing objects: 92% (113/122)
  251. 2020-08-25T18:15:49.7288010Z remote: Compressing objects: 93% (114/122)
  252. 2020-08-25T18:15:49.7288338Z remote: Compressing objects: 94% (115/122)
  253. 2020-08-25T18:15:49.7288686Z remote: Compressing objects: 95% (116/122)
  254. 2020-08-25T18:15:49.7288988Z remote: Compressing objects: 96% (118/122)
  255. 2020-08-25T18:15:49.7289161Z remote: Compressing objects: 97% (119/122)
  256. 2020-08-25T18:15:49.7289340Z remote: Compressing objects: 98% (120/122)
  257. 2020-08-25T18:15:49.7289509Z remote: Compressing objects: 99% (121/122)
  258. 2020-08-25T18:15:49.7289705Z remote: Compressing objects: 100% (122/122)
  259. 2020-08-25T18:15:49.7289890Z remote: Compressing objects: 100% (122/122), done.
  260. 2020-08-25T18:15:49.7465509Z Receiving objects: 0% (1/145)
  261. 2020-08-25T18:15:49.7465777Z Receiving objects: 1% (2/145)
  262. 2020-08-25T18:15:49.7465987Z Receiving objects: 2% (3/145)
  263. 2020-08-25T18:15:49.7467641Z Receiving objects: 3% (5/145)
  264. 2020-08-25T18:15:49.7467873Z Receiving objects: 4% (6/145)
  265. 2020-08-25T18:15:49.7468036Z Receiving objects: 5% (8/145)
  266. 2020-08-25T18:15:49.7468199Z Receiving objects: 6% (9/145)
  267. 2020-08-25T18:15:49.7468439Z Receiving objects: 7% (11/145)
  268. 2020-08-25T18:15:49.7468562Z Receiving objects: 8% (12/145)
  269. 2020-08-25T18:15:49.7470582Z Receiving objects: 9% (14/145)
  270. 2020-08-25T18:15:49.7471164Z Receiving objects: 10% (15/145)
  271. 2020-08-25T18:15:49.7471335Z Receiving objects: 11% (16/145)
  272. 2020-08-25T18:15:49.7472084Z Receiving objects: 12% (18/145)
  273. 2020-08-25T18:15:49.7472662Z Receiving objects: 13% (19/145)
  274. 2020-08-25T18:15:49.7473285Z Receiving objects: 14% (21/145)
  275. 2020-08-25T18:15:49.7473476Z Receiving objects: 15% (22/145)
  276. 2020-08-25T18:15:49.7473600Z Receiving objects: 16% (24/145)
  277. 2020-08-25T18:15:49.9408035Z Receiving objects: 17% (25/145)
  278. 2020-08-25T18:15:49.9410242Z Receiving objects: 18% (27/145)
  279. 2020-08-25T18:15:49.9416257Z Receiving objects: 19% (28/145)
  280. 2020-08-25T18:15:49.9605339Z Receiving objects: 20% (29/145)
  281. 2020-08-25T18:15:49.9609731Z Receiving objects: 21% (31/145)
  282. 2020-08-25T18:15:49.9610217Z Receiving objects: 22% (32/145)
  283. 2020-08-25T18:15:49.9610332Z Receiving objects: 23% (34/145)
  284. 2020-08-25T18:15:49.9610873Z Receiving objects: 24% (35/145)
  285. 2020-08-25T18:15:49.9612813Z Receiving objects: 25% (37/145)
  286. 2020-08-25T18:15:49.9612949Z Receiving objects: 26% (38/145)
  287. 2020-08-25T18:15:49.9613065Z Receiving objects: 27% (40/145)
  288. 2020-08-25T18:15:49.9613366Z Receiving objects: 28% (41/145)
  289. 2020-08-25T18:15:49.9613713Z Receiving objects: 29% (43/145)
  290. 2020-08-25T18:15:49.9614031Z Receiving objects: 30% (44/145)
  291. 2020-08-25T18:15:49.9614191Z Receiving objects: 31% (45/145)
  292. 2020-08-25T18:15:49.9614314Z Receiving objects: 32% (47/145)
  293. 2020-08-25T18:15:49.9614606Z Receiving objects: 33% (48/145)
  294. 2020-08-25T18:15:49.9614730Z Receiving objects: 34% (50/145)
  295. 2020-08-25T18:15:49.9614856Z Receiving objects: 35% (51/145)
  296. 2020-08-25T18:15:49.9614981Z Receiving objects: 36% (53/145)
  297. 2020-08-25T18:15:49.9615277Z Receiving objects: 37% (54/145)
  298. 2020-08-25T18:15:49.9615397Z Receiving objects: 38% (56/145)
  299. 2020-08-25T18:15:49.9615526Z Receiving objects: 39% (57/145)
  300. 2020-08-25T18:15:49.9615663Z Receiving objects: 40% (58/145)
  301. 2020-08-25T18:15:49.9615792Z Receiving objects: 41% (60/145)
  302. 2020-08-25T18:15:49.9615919Z Receiving objects: 42% (61/145)
  303. 2020-08-25T18:15:49.9616044Z Receiving objects: 43% (63/145)
  304. 2020-08-25T18:15:49.9616171Z Receiving objects: 44% (64/145)
  305. 2020-08-25T18:15:49.9616309Z Receiving objects: 45% (66/145)
  306. 2020-08-25T18:15:49.9616428Z Receiving objects: 46% (67/145)
  307. 2020-08-25T18:15:49.9617872Z Receiving objects: 47% (69/145)
  308. 2020-08-25T18:15:50.0133071Z Receiving objects: 48% (70/145)
  309. 2020-08-25T18:15:50.0133315Z Receiving objects: 49% (72/145)
  310. 2020-08-25T18:15:50.0134376Z Receiving objects: 50% (73/145)
  311. 2020-08-25T18:15:50.0134571Z Receiving objects: 51% (74/145)
  312. 2020-08-25T18:15:50.0143687Z Receiving objects: 52% (76/145)
  313. 2020-08-25T18:15:50.0143883Z Receiving objects: 53% (77/145)
  314. 2020-08-25T18:15:50.0144017Z Receiving objects: 54% (79/145)
  315. 2020-08-25T18:15:50.0144148Z Receiving objects: 55% (80/145)
  316. 2020-08-25T18:15:50.0144274Z Receiving objects: 56% (82/145)
  317. 2020-08-25T18:15:50.0144410Z Receiving objects: 57% (83/145)
  318. 2020-08-25T18:15:50.0144539Z Receiving objects: 58% (85/145)
  319. 2020-08-25T18:15:50.0520411Z Receiving objects: 59% (86/145)
  320. 2020-08-25T18:15:50.0522726Z Receiving objects: 60% (87/145)
  321. 2020-08-25T18:15:50.0523204Z Receiving objects: 61% (89/145)
  322. 2020-08-25T18:15:50.0523557Z Receiving objects: 62% (90/145)
  323. 2020-08-25T18:15:50.0523717Z Receiving objects: 63% (92/145)
  324. 2020-08-25T18:15:50.0524665Z Receiving objects: 64% (93/145)
  325. 2020-08-25T18:15:50.0525151Z Receiving objects: 65% (95/145)
  326. 2020-08-25T18:15:50.0525293Z Receiving objects: 66% (96/145)
  327. 2020-08-25T18:15:50.0525413Z Receiving objects: 67% (98/145)
  328. 2020-08-25T18:15:50.0525553Z Receiving objects: 68% (99/145)
  329. 2020-08-25T18:15:50.0525693Z Receiving objects: 69% (101/145)
  330. 2020-08-25T18:15:50.0525828Z Receiving objects: 70% (102/145)
  331. 2020-08-25T18:15:50.0845873Z Receiving objects: 71% (103/145)
  332. 2020-08-25T18:15:50.0846080Z Receiving objects: 72% (105/145)
  333. 2020-08-25T18:15:50.0846255Z Receiving objects: 73% (106/145)
  334. 2020-08-25T18:15:50.0846466Z Receiving objects: 74% (108/145)
  335. 2020-08-25T18:15:50.0846606Z Receiving objects: 75% (109/145)
  336. 2020-08-25T18:15:50.0846726Z Receiving objects: 76% (111/145)
  337. 2020-08-25T18:15:50.0846866Z Receiving objects: 77% (112/145)
  338. 2020-08-25T18:15:50.0847183Z Receiving objects: 78% (114/145)
  339. 2020-08-25T18:15:50.0847320Z Receiving objects: 79% (115/145)
  340. 2020-08-25T18:15:50.1235558Z Receiving objects: 80% (116/145)
  341. 2020-08-25T18:15:50.1237166Z remote: Total 145 (delta 29), reused 75 (delta 12), pack-reused 0
  342. 2020-08-25T18:15:50.1237329Z Receiving objects: 81% (118/145)
  343. 2020-08-25T18:15:50.1237454Z Receiving objects: 82% (119/145)
  344. 2020-08-25T18:15:50.1237557Z Receiving objects: 83% (121/145)
  345. 2020-08-25T18:15:50.1237954Z Receiving objects: 84% (122/145)
  346. 2020-08-25T18:15:50.1238076Z Receiving objects: 85% (124/145)
  347. 2020-08-25T18:15:50.1244232Z Receiving objects: 86% (125/145)
  348. 2020-08-25T18:15:50.1244544Z Receiving objects: 87% (127/145)
  349. 2020-08-25T18:15:50.1244849Z Receiving objects: 88% (128/145)
  350. 2020-08-25T18:15:50.1245000Z Receiving objects: 89% (130/145)
  351. 2020-08-25T18:15:50.1245121Z Receiving objects: 90% (131/145)
  352. 2020-08-25T18:15:50.1245226Z Receiving objects: 91% (132/145)
  353. 2020-08-25T18:15:50.1245367Z Receiving objects: 92% (134/145)
  354. 2020-08-25T18:15:50.1245487Z Receiving objects: 93% (135/145)
  355. 2020-08-25T18:15:50.1245607Z Receiving objects: 94% (137/145)
  356. 2020-08-25T18:15:50.1249631Z Receiving objects: 95% (138/145)
  357. 2020-08-25T18:15:50.1249944Z Receiving objects: 96% (140/145)
  358. 2020-08-25T18:15:50.1250094Z Receiving objects: 97% (141/145)
  359. 2020-08-25T18:15:50.1250231Z Receiving objects: 98% (143/145)
  360. 2020-08-25T18:15:50.1250350Z Receiving objects: 99% (144/145)
  361. 2020-08-25T18:15:50.1250488Z Receiving objects: 100% (145/145)
  362. 2020-08-25T18:15:50.1250639Z Receiving objects: 100% (145/145), 107.31 KiB | 283.00 KiB/s, done.
  363. 2020-08-25T18:15:50.1250851Z Resolving deltas: 0% (0/29)
  364. 2020-08-25T18:15:50.1251152Z Resolving deltas: 3% (1/29)
  365. 2020-08-25T18:15:50.1251301Z Resolving deltas: 6% (2/29)
  366. 2020-08-25T18:15:50.1251437Z Resolving deltas: 10% (3/29)
  367. 2020-08-25T18:15:50.1255964Z Resolving deltas: 13% (4/29)
  368. 2020-08-25T18:15:50.1256164Z Resolving deltas: 17% (5/29)
  369. 2020-08-25T18:15:50.1256295Z Resolving deltas: 37% (11/29)
  370. 2020-08-25T18:15:50.1258193Z Resolving deltas: 48% (14/29)
  371. 2020-08-25T18:15:50.1260511Z Resolving deltas: 58% (17/29)
  372. 2020-08-25T18:15:50.1262743Z Resolving deltas: 62% (18/29)
  373. 2020-08-25T18:15:50.1266914Z Resolving deltas: 65% (19/29)
  374. 2020-08-25T18:15:50.1268917Z Resolving deltas: 79% (23/29)
  375. 2020-08-25T18:15:50.1270603Z Resolving deltas: 82% (24/29)
  376. 2020-08-25T18:15:50.1272798Z Resolving deltas: 86% (25/29)
  377. 2020-08-25T18:15:50.1274065Z Resolving deltas: 93% (27/29)
  378. 2020-08-25T18:15:50.1278076Z Resolving deltas: 96% (28/29)
  379. 2020-08-25T18:15:50.1278603Z Resolving deltas: 100% (29/29)
  380. 2020-08-25T18:15:50.1279326Z Resolving deltas: 100% (29/29), done.
  381. 2020-08-25T18:15:50.1729201Z From https://github.com/submariner-io/lighthouse
  382. 2020-08-25T18:15:50.1729848Z * [new ref] f91b859307fe960751a14cfccc2555b26a4d3914 -> pull/267/merge
  383. 2020-08-25T18:15:50.1759391Z ##[endgroup]
  384. 2020-08-25T18:15:50.1759769Z ##[group]Determining the checkout info
  385. 2020-08-25T18:15:50.1763002Z ##[endgroup]
  386. 2020-08-25T18:15:50.1763537Z ##[group]Checking out the ref
  387. 2020-08-25T18:15:50.1764636Z [command]/usr/bin/git checkout --progress --force refs/remotes/pull/267/merge
  388. 2020-08-25T18:15:50.1887209Z Note: switching to 'refs/remotes/pull/267/merge'.
  389. 2020-08-25T18:15:50.1903910Z
  390. 2020-08-25T18:15:50.1907451Z You are in 'detached HEAD' state. You can look around, make experimental
  391. 2020-08-25T18:15:50.1907724Z changes and commit them, and you can discard any commits you make in this
  392. 2020-08-25T18:15:50.1907892Z state without impacting any branches by switching back to a branch.
  393. 2020-08-25T18:15:50.1907978Z
  394. 2020-08-25T18:15:50.1908452Z If you want to create a new branch to retain commits you create, you may
  395. 2020-08-25T18:15:50.1909019Z do so (now or later) by using -c with the switch command. Example:
  396. 2020-08-25T18:15:50.1909112Z
  397. 2020-08-25T18:15:50.1909401Z git switch -c <new-branch-name>
  398. 2020-08-25T18:15:50.1909494Z
  399. 2020-08-25T18:15:50.1909624Z Or undo this operation with:
  400. 2020-08-25T18:15:50.1909905Z
  401. 2020-08-25T18:15:50.1910205Z git switch -
  402. 2020-08-25T18:15:50.1910280Z
  403. 2020-08-25T18:15:50.1910422Z Turn off this advice by setting config variable advice.detachedHead to false
  404. 2020-08-25T18:15:50.1910525Z
  405. 2020-08-25T18:15:50.1910680Z HEAD is now at f91b859 Merge 5ee7182eaa7025b210f3bcc9494ac3ccee41c05e into 4e092690c53a155e82bfc69491a39fcadf64a7f3
  406. 2020-08-25T18:15:50.1910904Z ##[endgroup]
  407. 2020-08-25T18:15:50.1911203Z [command]/usr/bin/git log -1
  408. 2020-08-25T18:15:50.1941521Z commit f91b859307fe960751a14cfccc2555b26a4d3914
  409. 2020-08-25T18:15:50.1941990Z Author: Daniel Farrell <dfarrell07@gmail.com>
  410. 2020-08-25T18:15:50.1942675Z Date: Tue Aug 25 14:15:28 2020 -0400
  411. 2020-08-25T18:15:50.1942974Z
  412. 2020-08-25T18:15:50.1943455Z Merge 5ee7182eaa7025b210f3bcc9494ac3ccee41c05e into 4e092690c53a155e82bfc69491a39fcadf64a7f3
  413. 2020-08-25T18:15:50.2075052Z ##[group]Run sudo swapoff -a
  414. 2020-08-25T18:15:50.2075326Z sudo swapoff -a
  415. 2020-08-25T18:15:50.2075462Z sudo rm -f /swapfile
  416. 2020-08-25T18:15:50.2075598Z df -h
  417. 2020-08-25T18:15:50.2075729Z free -h
  418. 2020-08-25T18:15:50.2123229Z shell: /bin/bash -e {0}
  419. 2020-08-25T18:15:50.2123387Z ##[endgroup]
  420. 2020-08-25T18:15:50.2674072Z Filesystem Size Used Avail Use% Mounted on
  421. 2020-08-25T18:15:50.2675341Z udev 3.4G 0 3.4G 0% /dev
  422. 2020-08-25T18:15:50.2675654Z tmpfs 696M 952K 695M 1% /run
  423. 2020-08-25T18:15:50.2675903Z /dev/sda1 84G 58G 26G 70% /
  424. 2020-08-25T18:15:50.2676187Z tmpfs 3.4G 8.0K 3.4G 1% /dev/shm
  425. 2020-08-25T18:15:50.2676457Z tmpfs 5.0M 0 5.0M 0% /run/lock
  426. 2020-08-25T18:15:50.2676719Z tmpfs 3.4G 0 3.4G 0% /sys/fs/cgroup
  427. 2020-08-25T18:15:50.2676993Z /dev/loop0 97M 97M 0 100% /snap/core/9804
  428. 2020-08-25T18:15:50.2677410Z /dev/loop1 40M 40M 0 100% /snap/hub/43
  429. 2020-08-25T18:15:50.2677704Z /dev/sda15 105M 3.6M 101M 4% /boot/efi
  430. 2020-08-25T18:15:50.2678076Z /dev/sdb1 14G 4.1G 9.0G 32% /mnt
  431. 2020-08-25T18:15:50.2715174Z total used free shared buff/cache available
  432. 2020-08-25T18:15:50.2715819Z Mem: 6.8G 491M 4.9G 26M 1.4G 6.0G
  433. 2020-08-25T18:15:50.2716119Z Swap: 0B 0B 0B
  434. 2020-08-25T18:15:50.2772831Z ##[group]Run make e2e using=" operator"
  435. 2020-08-25T18:15:50.2773029Z make e2e using=" operator"
  436. 2020-08-25T18:15:50.2822266Z shell: /bin/bash -e {0}
  437. 2020-08-25T18:15:50.2822510Z ##[endgroup]
  438. 2020-08-25T18:15:50.2940547Z Downloading dapper
  439. 2020-08-25T18:15:50.6296470Z .dapper.tmp version v0.5.2
  440. 2020-08-25T18:15:50.6332540Z ./.dapper -m bind make e2e -- using=operator
  441. 2020-08-25T18:15:51.6009106Z Sending build context to Docker daemon 6.109MB
  442. 2020-08-25T18:15:51.6072073Z
  443. 2020-08-25T18:15:51.6073349Z Step 1/6 : FROM quay.io/submariner/shipyard-dapper-base:0.6.1
  444. 2020-08-25T18:15:52.6196483Z 0.6.1: Pulling from submariner/shipyard-dapper-base
  445. 2020-08-25T18:15:52.6219743Z c7def56d621e: Pulling fs layer
  446. 2020-08-25T18:15:52.6221474Z 3980251caaa6: Pulling fs layer
  447. 2020-08-25T18:15:52.6222374Z 32b7ae3b1936: Pulling fs layer
  448. 2020-08-25T18:15:52.6223035Z 051850cfc290: Pulling fs layer
  449. 2020-08-25T18:15:52.6223644Z d201a62b2776: Pulling fs layer
  450. 2020-08-25T18:15:52.6224483Z 7b6a21e8c93e: Pulling fs layer
  451. 2020-08-25T18:15:52.6225331Z ba1db1a2eda6: Pulling fs layer
  452. 2020-08-25T18:15:52.6225874Z d201a62b2776: Waiting
  453. 2020-08-25T18:15:52.6226368Z 051850cfc290: Waiting
  454. 2020-08-25T18:15:52.6226840Z ba1db1a2eda6: Waiting
  455. 2020-08-25T18:15:52.6227357Z 7b6a21e8c93e: Waiting
  456. 2020-08-25T18:15:53.7019571Z 32b7ae3b1936: Verifying Checksum
  457. 2020-08-25T18:15:53.7021411Z 32b7ae3b1936: Download complete
  458. 2020-08-25T18:15:53.7087086Z c7def56d621e: Verifying Checksum
  459. 2020-08-25T18:15:53.7087297Z c7def56d621e: Download complete
  460. 2020-08-25T18:15:54.0640602Z d201a62b2776: Verifying Checksum
  461. 2020-08-25T18:15:54.0641964Z d201a62b2776: Download complete
  462. 2020-08-25T18:15:54.0971732Z 051850cfc290: Verifying Checksum
  463. 2020-08-25T18:15:54.0972375Z 051850cfc290: Download complete
  464. 2020-08-25T18:15:54.4204932Z 7b6a21e8c93e: Verifying Checksum
  465. 2020-08-25T18:15:54.4205127Z 7b6a21e8c93e: Download complete
  466. 2020-08-25T18:15:54.4715504Z ba1db1a2eda6: Verifying Checksum
  467. 2020-08-25T18:15:54.4715872Z ba1db1a2eda6: Download complete
  468. 2020-08-25T18:15:57.5113340Z c7def56d621e: Pull complete
  469. 2020-08-25T18:16:03.1351941Z 3980251caaa6: Verifying Checksum
  470. 2020-08-25T18:16:03.1352166Z 3980251caaa6: Download complete
  471. 2020-08-25T18:16:15.1682156Z 3980251caaa6: Pull complete
  472. 2020-08-25T18:16:16.0128546Z 32b7ae3b1936: Pull complete
  473. 2020-08-25T18:16:16.0900606Z 051850cfc290: Pull complete
  474. 2020-08-25T18:16:16.1628083Z d201a62b2776: Pull complete
  475. 2020-08-25T18:16:16.2267767Z 7b6a21e8c93e: Pull complete
  476. 2020-08-25T18:16:16.3001883Z ba1db1a2eda6: Pull complete
  477. 2020-08-25T18:16:16.3031224Z Digest: sha256:2c80acb19befccb23203ed1146f73031f6fa599fcbb2f458f48dbe50870696e8
  478. 2020-08-25T18:16:16.3056319Z Status: Downloaded newer image for quay.io/submariner/shipyard-dapper-base:0.6.1
  479. 2020-08-25T18:16:16.3141055Z ---> d8a2f56352b1
  480. 2020-08-25T18:16:16.3141727Z Step 2/6 : ENV DAPPER_ENV="REPO TAG QUAY_USERNAME QUAY_PASSWORD GITHUB_SHA BUILD_ARGS CLUSTERS_ARGS DEPLOY_ARGS RELEASE_ARGS" DAPPER_SOURCE=/go/src/github.com/submariner-io/lighthouse DAPPER_DOCKER_SOCKET=true
  481. 2020-08-25T18:16:24.7192171Z ---> Running in 8efe4ec6e46b
  482. 2020-08-25T18:16:25.8198881Z Removing intermediate container 8efe4ec6e46b
  483. 2020-08-25T18:16:25.8199872Z ---> 3f3acef196fe
  484. 2020-08-25T18:16:25.8200038Z Step 3/6 : ENV DAPPER_OUTPUT=${DAPPER_SOURCE}/output
  485. 2020-08-25T18:16:25.8507283Z ---> Running in 77a85c0f19b7
  486. 2020-08-25T18:16:27.0216919Z Removing intermediate container 77a85c0f19b7
  487. 2020-08-25T18:16:27.0252087Z ---> d47a5a00a868
  488. 2020-08-25T18:16:27.0252257Z Step 4/6 : WORKDIR ${DAPPER_SOURCE}
  489. 2020-08-25T18:16:27.0433888Z ---> Running in cd9e8df8900d
  490. 2020-08-25T18:16:28.1675507Z Removing intermediate container cd9e8df8900d
  491. 2020-08-25T18:16:28.1676850Z ---> c51802b41c03
  492. 2020-08-25T18:16:28.1677329Z Step 5/6 : ENTRYPOINT ["/opt/shipyard/scripts/entry"]
  493. 2020-08-25T18:16:28.1992675Z ---> Running in 1767b463d429
  494. 2020-08-25T18:16:29.3104488Z Removing intermediate container 1767b463d429
  495. 2020-08-25T18:16:29.3129098Z ---> fe29113922a8
  496. 2020-08-25T18:16:29.3129245Z Step 6/6 : CMD ["sh"]
  497. 2020-08-25T18:16:29.3394183Z ---> Running in 4c9bc520cfde
  498. 2020-08-25T18:16:30.4690507Z Removing intermediate container 4c9bc520cfde
  499. 2020-08-25T18:16:30.4691639Z ---> 0b0c093a497a
  500. 2020-08-25T18:16:30.4745856Z Successfully built 0b0c093a497a
  501. 2020-08-25T18:16:30.4803427Z Successfully tagged lighthouse:HEAD
  502. 2020-08-25T18:16:31.0071327Z [lighthouse]$ trap chown -R 1001:116 . exit
  503. 2020-08-25T18:16:31.0076594Z [lighthouse]$ mkdir -p bin dist output
  504. 2020-08-25T18:16:31.0152436Z [lighthouse]$ make e2e -- using=operator
  505. 2020-08-25T18:16:31.0598238Z fatal: No names found, cannot describe anything.
  506. 2020-08-25T18:16:31.0680241Z Makefile:39: warning: overriding recipe for target 'deploy'
  507. 2020-08-25T18:16:31.0680662Z /opt/shipyard/Makefile.inc:36: warning: ignoring old recipe for target 'deploy'
  508. 2020-08-25T18:16:31.0691958Z /opt/shipyard/scripts/e2e.sh cluster1 cluster2
  509. 2020-08-25T18:16:31.0723941Z Downloading shflags 1.0.3
  510. 2020-08-25T18:16:31.0811283Z % Total % Received % Xferd Average Speed Time Time Time Current
  511. 2020-08-25T18:16:31.0811463Z Dload Upload Total Spent Left Speed
  512. 2020-08-25T18:16:31.0811541Z
  513. 2020-08-25T18:16:31.1209905Z 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
  514. 2020-08-25T18:16:31.1216529Z 100 31091 100 31091 0 0 759k 0 --:--:-- --:--:-- --:--:-- 759k
  515. 2020-08-25T18:16:31.1845850Z [lighthouse]$ source /opt/shipyard/scripts/lib/utils
  516. 2020-08-25T18:16:31.1860359Z [lighthouse]$ . /opt/shipyard/scripts/lib/source_only
  517. 2020-08-25T18:16:31.1874290Z [lighthouse]$ script_name=utils
  518. 2020-08-25T18:16:31.1887284Z [lighthouse]$ exec_name=e2e.sh
  519. 2020-08-25T18:16:31.1914041Z [lighthouse]$ source /opt/shipyard/scripts/lib/cluster_settings
  520. 2020-08-25T18:16:31.1924460Z [lighthouse]$ . /opt/shipyard/scripts/lib/source_only
  521. 2020-08-25T18:16:31.1933603Z [lighthouse]$ script_name=cluster_settings
  522. 2020-08-25T18:16:31.1950450Z [lighthouse]$ exec_name=e2e.sh
  523. 2020-08-25T18:16:31.1967232Z [lighthouse]$ broker=cluster1
  524. 2020-08-25T18:16:31.1976352Z [lighthouse]$ declare -gA cluster_nodes
  525. 2020-08-25T18:16:31.1989025Z [lighthouse]$ cluster_nodes[cluster1]=control-plane worker
  526. 2020-08-25T18:16:31.2003538Z [lighthouse]$ cluster_nodes[cluster2]=control-plane worker
  527. 2020-08-25T18:16:31.2018735Z [lighthouse]$ cluster_nodes[cluster3]=control-plane worker worker
  528. 2020-08-25T18:16:31.2026088Z [lighthouse]$ declare -gA cluster_subm
  529. 2020-08-25T18:16:31.2038791Z [lighthouse]$ cluster_subm[cluster1]=true
  530. 2020-08-25T18:16:31.2047156Z [lighthouse]$ cluster_subm[cluster2]=true
  531. 2020-08-25T18:16:31.2064547Z [lighthouse]$ cluster_subm[cluster3]=true
  532. 2020-08-25T18:16:31.2065734Z [lighthouse]$ declare -gA cluster_cni
  533. 2020-08-25T18:16:31.2082805Z [lighthouse]$ cluster_cni[cluster2]=weave
  534. 2020-08-25T18:16:31.2087310Z [lighthouse]$ cluster_cni[cluster3]=weave
  535. 2020-08-25T18:16:31.2100841Z [lighthouse]$ declare_kubeconfig
  536. 2020-08-25T18:16:31.2105735Z [lighthouse]$ declare_kubeconfig
  537. 2020-08-25T18:16:31.2133907Z [lighthouse]$ source /opt/shipyard/scripts/lib/kubecfg
  538. 2020-08-25T18:16:31.2150648Z [lighthouse]$ export KUBECONFIG
  539. 2020-08-25T18:16:31.2176111Z [lighthouse]$ KUBECONFIG=
  540. 2020-08-25T18:16:31.2195339Z [lighthouse]$ find /go/src/github.com/submariner-io/lighthouse/output/kubeconfigs -type f -printf %p:
  541. 2020-08-25T18:16:31.2210968Z find: '/go/src/github.com/submariner-io/lighthouse/output/kubeconfigs': No such file or directory
  542. 2020-08-25T18:16:31.2226728Z [lighthouse]$ :
  543. 2020-08-25T18:16:31.2242578Z [lighthouse]$ deploy_env_once
  544. 2020-08-25T18:16:31.2258973Z [lighthouse]$ deploy_env_once
  545. 2020-08-25T18:16:31.2278090Z 
  546. 2020-08-25T18:16:31.9268416Z [lighthouse]$ make deploy
  547. 2020-08-25T18:16:31.9295873Z make[1]: Entering directory '/go/src/github.com/submariner-io/lighthouse'
  548. 2020-08-25T18:16:31.9721728Z fatal: No names found, cannot describe anything.
  549. 2020-08-25T18:16:31.9826300Z Makefile:39: warning: overriding recipe for target 'deploy'
  550. 2020-08-25T18:16:31.9827419Z /opt/shipyard/Makefile.inc:36: warning: ignoring old recipe for target 'deploy'
  551. 2020-08-25T18:16:31.9898056Z find: 'bin/lighthouse-agent': No such file or directory
  552. 2020-08-25T18:16:32.3287972Z go mod download
  553. 2020-08-25T18:16:58.5861855Z go mod vendor
  554. 2020-08-25T18:17:00.8097861Z /opt/shipyard/scripts/compile.sh bin/lighthouse-agent pkg/agent/main.go
  555. 2020-08-25T18:17:00.8589898Z [lighthouse]$ mkdir -p bin
  556. 2020-08-25T18:17:00.8611163Z Building 'bin/lighthouse-agent' (ldflags: '')
  557. 2020-08-25T18:17:00.8622882Z [lighthouse]$ ldflags=-s -w 
  558. 2020-08-25T18:17:00.8633195Z [lighthouse]$ CGO_ENABLED=0 go build -ldflags -s -w -o bin/lighthouse-agent pkg/agent/main.go
  559. 2020-08-25T18:17:51.8112455Z [lighthouse]$ upx bin/lighthouse-agent
  560. 2020-08-25T18:17:58.3920964Z Ultimate Packer for eXecutables
  561. 2020-08-25T18:17:58.3921777Z Copyright (C) 1996 - 2020
  562. 2020-08-25T18:17:58.3921906Z UPX 3.96 Markus Oberhumer, Laszlo Molnar & John Reiser Jan 23rd 2020
  563. 2020-08-25T18:17:58.3922006Z
  564. 2020-08-25T18:17:58.3922127Z File size Ratio Format Name
  565. 2020-08-25T18:17:58.3922437Z -------------------- ------ ----------- -----------
  566. 2020-08-25T18:17:58.3922754Z 28635136 -> 8552168 29.87% linux/amd64 lighthouse-agent
  567. 2020-08-25T18:17:58.3922827Z
  568. 2020-08-25T18:17:58.3922940Z Packed 1 file.
  569. 2020-08-25T18:17:58.3923269Z /opt/shipyard/scripts/build_image.sh -i lighthouse-agent -f package/Dockerfile.lighthouse-agent
  570. 2020-08-25T18:17:58.4396029Z fatal: No names found, cannot describe anything.
  571. 2020-08-25T18:17:58.5246574Z [lighthouse]$ set -e
  572. 2020-08-25T18:17:58.5252506Z [lighthouse]$ local_image=quay.io/submariner/lighthouse-agent:dev
  573. 2020-08-25T18:17:58.5262519Z [lighthouse]$ cache_image=quay.io/submariner/lighthouse-agent:devel
  574. 2020-08-25T18:17:58.5272750Z [lighthouse]$ cache_flag=
  575. 2020-08-25T18:17:58.5286018Z [lighthouse]$ cache_flag=--cache-from quay.io/submariner/lighthouse-agent:devel
  576. 2020-08-25T18:17:58.5302822Z [lighthouse]$ docker image ls -q quay.io/submariner/lighthouse-agent:devel
  577. 2020-08-25T18:17:58.8604649Z [lighthouse]$ docker pull quay.io/submariner/lighthouse-agent:devel
  578. 2020-08-25T18:18:00.1438681Z devel: Pulling from submariner/lighthouse-agent
  579. 2020-08-25T18:18:00.1438900Z 41ae95b593e0: Pulling fs layer
  580. 2020-08-25T18:18:00.1439193Z f20f68829d13: Pulling fs layer
  581. 2020-08-25T18:18:00.1439312Z 045f279681db: Pulling fs layer
  582. 2020-08-25T18:18:00.1439429Z 4f3197d91727: Pulling fs layer
  583. 2020-08-25T18:18:00.1439576Z 4f3197d91727: Waiting
  584. 2020-08-25T18:18:00.5110156Z 045f279681db: Verifying Checksum
  585. 2020-08-25T18:18:00.5110625Z f20f68829d13: Verifying Checksum
  586. 2020-08-25T18:18:00.5110818Z f20f68829d13: Download complete
  587. 2020-08-25T18:18:00.5110973Z 045f279681db: Download complete
  588. 2020-08-25T18:18:00.9540219Z 4f3197d91727: Verifying Checksum
  589. 2020-08-25T18:18:00.9540514Z 4f3197d91727: Download complete
  590. 2020-08-25T18:18:00.9732701Z 41ae95b593e0: Verifying Checksum
  591. 2020-08-25T18:18:00.9733066Z 41ae95b593e0: Download complete
  592. 2020-08-25T18:18:03.2182421Z 41ae95b593e0: Pull complete
  593. 2020-08-25T18:18:03.2827097Z f20f68829d13: Pull complete
  594. 2020-08-25T18:18:03.3518796Z 045f279681db: Pull complete
  595. 2020-08-25T18:18:03.5129266Z 4f3197d91727: Pull complete
  596. 2020-08-25T18:18:03.5164742Z Digest: sha256:b34135525cd63677b4a81a7be81f04b6cb5fe3a5a886d4ea2b9beaf1380b7a5b
  597. 2020-08-25T18:18:03.5188915Z Status: Downloaded newer image for quay.io/submariner/lighthouse-agent:devel
  598. 2020-08-25T18:18:03.5208290Z quay.io/submariner/lighthouse-agent:devel
  599. 2020-08-25T18:18:03.5253334Z [lighthouse]$ grep FROM package/Dockerfile.lighthouse-agent
  600. 2020-08-25T18:18:03.5326076Z [lighthouse]$ cut -f2 -d 
  601. 2020-08-25T18:18:03.5326405Z [lighthouse]$ grep -v scratch
  602. 2020-08-25T18:18:03.5326975Z [lighthouse]$ cache_flag+= --cache-from registry.access.redhat.com/ubi8/ubi-minimal
  603. 2020-08-25T18:18:03.5327383Z [lighthouse]$ docker pull registry.access.redhat.com/ubi8/ubi-minimal
  604. 2020-08-25T18:18:03.8598244Z Using default tag: latest
  605. 2020-08-25T18:18:04.4950485Z latest: Pulling from ubi8/ubi-minimal
  606. 2020-08-25T18:18:04.5884942Z 41ae95b593e0: Already exists
  607. 2020-08-25T18:18:04.5910506Z f20f68829d13: Already exists
  608. 2020-08-25T18:18:04.6744791Z Digest: sha256:372622021a90893d9e25c298e045c804388c7666f3e756cd48f75d20172d9e55
  609. 2020-08-25T18:18:04.6770844Z Status: Downloaded newer image for registry.access.redhat.com/ubi8/ubi-minimal:latest
  610. 2020-08-25T18:18:04.6808933Z registry.access.redhat.com/ubi8/ubi-minimal:latest
  611. 2020-08-25T18:18:04.6869485Z [lighthouse]$ buildargs_flag=
  612. 2020-08-25T18:18:04.6870035Z [lighthouse]$ docker build -t quay.io/submariner/lighthouse-agent:dev --cache-from quay.io/submariner/lighthouse-agent:devel --cache-from registry.access.redhat.com/ubi8/ubi-minimal -f package/Dockerfile.lighthouse-agent .
  613. 2020-08-25T18:18:06.0145131Z Sending build context to Docker daemon 59.04MB
  614. 2020-08-25T18:18:06.0146237Z
  615. 2020-08-25T18:18:06.0263999Z Step 1/4 : FROM registry.access.redhat.com/ubi8/ubi-minimal
  616. 2020-08-25T18:18:06.0276708Z ---> 86c870596572
  617. 2020-08-25T18:18:06.0276923Z Step 2/4 : WORKDIR /var/submariner
  618. 2020-08-25T18:18:06.0574514Z ---> Using cache
  619. 2020-08-25T18:18:06.0575030Z ---> 27d0c628fac6
  620. 2020-08-25T18:18:06.0575340Z Step 3/4 : COPY bin/lighthouse-agent package/lighthouse-agent.sh /usr/local/bin/
  621. 2020-08-25T18:18:06.0603546Z ---> Using cache
  622. 2020-08-25T18:18:06.0603923Z ---> a2b24ce8b1f5
  623. 2020-08-25T18:18:06.0604213Z Step 4/4 : ENTRYPOINT lighthouse-agent.sh
  624. 2020-08-25T18:18:06.0615879Z ---> Using cache
  625. 2020-08-25T18:18:06.0616153Z ---> 876d66321be3
  626. 2020-08-25T18:18:06.1733998Z Successfully built 876d66321be3
  627. 2020-08-25T18:18:06.1867833Z [lighthouse]$ docker tag quay.io/submariner/lighthouse-agent:dev quay.io/submariner/lighthouse-agent:devel
  628. 2020-08-25T18:18:06.1868222Z Successfully tagged quay.io/submariner/lighthouse-agent:dev
  629. 2020-08-25T18:18:06.5428596Z touch package/.image.lighthouse-agent
  630. 2020-08-25T18:18:06.5491287Z find: 'bin/lighthouse-coredns': No such file or directory
  631. 2020-08-25T18:18:06.9027898Z /opt/shipyard/scripts/compile.sh bin/lighthouse-coredns pkg/coredns/main.go
  632. 2020-08-25T18:18:06.9523647Z [lighthouse]$ mkdir -p bin
  633. 2020-08-25T18:18:06.9542228Z Building 'bin/lighthouse-coredns' (ldflags: '')
  634. 2020-08-25T18:18:06.9553968Z [lighthouse]$ ldflags=-s -w 
  635. 2020-08-25T18:18:06.9566020Z [lighthouse]$ CGO_ENABLED=0 go build -ldflags -s -w -o bin/lighthouse-coredns pkg/coredns/main.go
  636. 2020-08-25T18:18:43.5996389Z [lighthouse]$ upx bin/lighthouse-coredns
  637. 2020-08-25T18:18:53.5100971Z Ultimate Packer for eXecutables
  638. 2020-08-25T18:18:53.5109525Z Copyright (C) 1996 - 2020
  639. 2020-08-25T18:18:53.5109767Z UPX 3.96 Markus Oberhumer, Laszlo Molnar & John Reiser Jan 23rd 2020
  640. 2020-08-25T18:18:53.5109845Z
  641. 2020-08-25T18:18:53.5109961Z File size Ratio Format Name
  642. 2020-08-25T18:18:53.5110287Z -------------------- ------ ----------- -----------
  643. 2020-08-25T18:18:53.5110744Z 41365504 -> 12545880 30.33% linux/amd64 lighthouse-coredns
  644. 2020-08-25T18:18:53.5110809Z
  645. 2020-08-25T18:18:53.5110913Z Packed 1 file.
  646. 2020-08-25T18:18:53.5111220Z /opt/shipyard/scripts/build_image.sh -i lighthouse-coredns -f package/Dockerfile.lighthouse-coredns
  647. 2020-08-25T18:18:53.5583766Z fatal: No names found, cannot describe anything.
  648. 2020-08-25T18:18:53.6397089Z [lighthouse]$ set -e
  649. 2020-08-25T18:18:53.6406249Z [lighthouse]$ local_image=quay.io/submariner/lighthouse-coredns:dev
  650. 2020-08-25T18:18:53.6416175Z [lighthouse]$ cache_image=quay.io/submariner/lighthouse-coredns:devel
  651. 2020-08-25T18:18:53.6427414Z [lighthouse]$ cache_flag=
  652. 2020-08-25T18:18:53.6442061Z [lighthouse]$ cache_flag=--cache-from quay.io/submariner/lighthouse-coredns:devel
  653. 2020-08-25T18:18:53.6455162Z [lighthouse]$ docker image ls -q quay.io/submariner/lighthouse-coredns:devel
  654. 2020-08-25T18:18:53.9692017Z [lighthouse]$ docker pull quay.io/submariner/lighthouse-coredns:devel
  655. 2020-08-25T18:18:55.2562639Z devel: Pulling from submariner/lighthouse-coredns
  656. 2020-08-25T18:18:55.2567150Z 50c5b17671b8: Pulling fs layer
  657. 2020-08-25T18:18:55.2567799Z 074327197609: Pulling fs layer
  658. 2020-08-25T18:18:55.2568958Z cf5b4e39dde7: Pulling fs layer
  659. 2020-08-25T18:18:55.7884159Z cf5b4e39dde7: Verifying Checksum
  660. 2020-08-25T18:18:55.7884407Z cf5b4e39dde7: Download complete
  661. 2020-08-25T18:18:55.8059686Z 074327197609: Verifying Checksum
  662. 2020-08-25T18:18:55.8059854Z 074327197609: Download complete
  663. 2020-08-25T18:18:55.8722352Z 50c5b17671b8: Verifying Checksum
  664. 2020-08-25T18:18:55.8722539Z 50c5b17671b8: Download complete
  665. 2020-08-25T18:18:57.1739922Z 50c5b17671b8: Pull complete
  666. 2020-08-25T18:18:57.5483186Z 074327197609: Pull complete
  667. 2020-08-25T18:18:57.7392574Z cf5b4e39dde7: Pull complete
  668. 2020-08-25T18:18:57.7418269Z Digest: sha256:058418c628d4d9d503eea0e6f78f5a5f5e195bbc2da7230eed715cbb98de7ecc
  669. 2020-08-25T18:18:57.7444108Z Status: Downloaded newer image for quay.io/submariner/lighthouse-coredns:devel
  670. 2020-08-25T18:18:57.7478019Z quay.io/submariner/lighthouse-coredns:devel
  671. 2020-08-25T18:18:57.7501750Z [lighthouse]$ grep FROM package/Dockerfile.lighthouse-coredns
  672. 2020-08-25T18:18:57.7520008Z [lighthouse]$ cut -f2 -d 
  673. 2020-08-25T18:18:57.7529606Z [lighthouse]$ grep -v scratch
  674. 2020-08-25T18:18:57.7536667Z [lighthouse]$ cache_flag+= --cache-from debian:stable-slim
  675. 2020-08-25T18:18:57.7587120Z [lighthouse]$ docker pull debian:stable-slim
  676. 2020-08-25T18:18:59.1444976Z stable-slim: Pulling from library/debian
  677. 2020-08-25T18:18:59.4631559Z 50c5b17671b8: Already exists
  678. 2020-08-25T18:18:59.8377207Z Digest: sha256:317addd6c60b27fc1337d24b9f9b98babf858286962a5ddb689397a487044b93
  679. 2020-08-25T18:18:59.8425406Z Status: Downloaded newer image for debian:stable-slim
  680. 2020-08-25T18:18:59.8430969Z docker.io/library/debian:stable-slim
  681. 2020-08-25T18:18:59.8433880Z [lighthouse]$ buildargs_flag=
  682. 2020-08-25T18:18:59.8508189Z [lighthouse]$ docker build -t quay.io/submariner/lighthouse-coredns:dev --cache-from quay.io/submariner/lighthouse-coredns:devel --cache-from debian:stable-slim -f package/Dockerfile.lighthouse-coredns .
  683. 2020-08-25T18:19:01.2324319Z Sending build context to Docker daemon 71.58MB
  684. 2020-08-25T18:19:01.2324428Z
  685. 2020-08-25T18:19:01.2424903Z Step 1/5 : FROM debian:stable-slim
  686. 2020-08-25T18:19:01.2428745Z ---> 52baa1311484
  687. 2020-08-25T18:19:01.2429123Z Step 2/5 : RUN apt-get update && apt-get -y install ca-certificates && update-ca-certificates
  688. 2020-08-25T18:19:01.2596100Z ---> Using cache
  689. 2020-08-25T18:19:01.2596382Z ---> 073931c855f9
  690. 2020-08-25T18:19:01.2596653Z Step 3/5 : COPY bin/lighthouse-coredns /usr/local/bin/
  691. 2020-08-25T18:19:01.2627788Z ---> Using cache
  692. 2020-08-25T18:19:01.2628251Z ---> f8e79eb1f2fe
  693. 2020-08-25T18:19:01.2628354Z Step 4/5 : EXPOSE 53 53/udp
  694. 2020-08-25T18:19:01.2652773Z ---> Using cache
  695. 2020-08-25T18:19:01.2653052Z ---> ee3457996c9c
  696. 2020-08-25T18:19:01.2653344Z Step 5/5 : ENTRYPOINT ["/usr/local/bin/lighthouse-coredns"]
  697. 2020-08-25T18:19:01.2667579Z ---> Using cache
  698. 2020-08-25T18:19:01.2667846Z ---> 484a3e749abe
  699. 2020-08-25T18:19:01.3761739Z Successfully built 484a3e749abe
  700. 2020-08-25T18:19:01.3831465Z Successfully tagged quay.io/submariner/lighthouse-coredns:dev
  701. 2020-08-25T18:19:01.3856098Z [lighthouse]$ docker tag quay.io/submariner/lighthouse-coredns:dev quay.io/submariner/lighthouse-coredns:devel
  702. 2020-08-25T18:19:01.7168711Z touch package/.image.lighthouse-coredns
  703. 2020-08-25T18:19:01.7190374Z /opt/shipyard/scripts/clusters.sh --cluster_settings /go/src/github.com/submariner-io/lighthouse/scripts/cluster_settings
  704. 2020-08-25T18:19:01.7985768Z Running with: k8s_version=1.17.0, olm_version=0.14.1, olm=false, globalnet=false, registry_inmemory=true, cluster_settings=/go/src/github.com/submariner-io/lighthouse/scripts/cluster_settings, timeout=5m
  705. 2020-08-25T18:19:01.8011217Z [lighthouse]$ source /opt/shipyard/scripts/lib/utils
  706. 2020-08-25T18:19:01.8023309Z [lighthouse]$ . /opt/shipyard/scripts/lib/source_only
  707. 2020-08-25T18:19:01.8035799Z [lighthouse]$ script_name=utils
  708. 2020-08-25T18:19:01.8046065Z [lighthouse]$ exec_name=clusters.sh
  709. 2020-08-25T18:19:01.8064472Z [lighthouse]$ source /opt/shipyard/scripts/lib/cluster_settings
  710. 2020-08-25T18:19:01.8074400Z [lighthouse]$ . /opt/shipyard/scripts/lib/source_only
  711. 2020-08-25T18:19:01.8088693Z [lighthouse]$ script_name=cluster_settings
  712. 2020-08-25T18:19:01.8101466Z [lighthouse]$ exec_name=clusters.sh
  713. 2020-08-25T18:19:01.8122075Z [lighthouse]$ broker=cluster1
  714. 2020-08-25T18:19:01.8133792Z [lighthouse]$ declare -gA cluster_nodes
  715. 2020-08-25T18:19:01.8143951Z [lighthouse]$ cluster_nodes[cluster1]=control-plane worker
  716. 2020-08-25T18:19:01.8156741Z [lighthouse]$ cluster_nodes[cluster2]=control-plane worker
  717. 2020-08-25T18:19:01.8168578Z [lighthouse]$ cluster_nodes[cluster3]=control-plane worker worker
  718. 2020-08-25T18:19:01.8179621Z [lighthouse]$ declare -gA cluster_subm
  719. 2020-08-25T18:19:01.8192058Z [lighthouse]$ cluster_subm[cluster1]=true
  720. 2020-08-25T18:19:01.8205081Z [lighthouse]$ cluster_subm[cluster2]=true
  721. 2020-08-25T18:19:01.8215721Z [lighthouse]$ cluster_subm[cluster3]=true
  722. 2020-08-25T18:19:01.8227068Z [lighthouse]$ declare -gA cluster_cni
  723. 2020-08-25T18:19:01.8240350Z [lighthouse]$ cluster_cni[cluster2]=weave
  724. 2020-08-25T18:19:01.8254848Z [lighthouse]$ cluster_cni[cluster3]=weave
  725. 2020-08-25T18:19:01.8266838Z [lighthouse]$ source /go/src/github.com/submariner-io/lighthouse/scripts/cluster_settings
  726. 2020-08-25T18:19:01.8278298Z [lighthouse]$ . /opt/shipyard/scripts/lib/source_only
  727. 2020-08-25T18:19:01.8289860Z [lighthouse]$ script_name=cluster_settings
  728. 2020-08-25T18:19:01.8302924Z [lighthouse]$ exec_name=clusters.sh
  729. 2020-08-25T18:19:01.8323291Z [lighthouse]$ cluster_nodes[cluster1]=control-plane worker worker
  730. 2020-08-25T18:19:01.8334075Z [lighthouse]$ cluster_nodes[cluster2]=control-plane worker worker
  731. 2020-08-25T18:19:01.8407869Z [lighthouse]$ cat
  732. 2020-08-25T18:19:01.8431413Z [lighthouse]$ typeset -p cluster_nodes
  733. 2020-08-25T18:19:01.8444528Z [lighthouse]$ cut -f 2- -d=
  734. 2020-08-25T18:19:01.8472941Z [lighthouse]$ typeset -p cluster_subm
  735. 2020-08-25T18:19:01.8485607Z [lighthouse]$ cut -f 2- -d=
  736. 2020-08-25T18:19:01.8510559Z Cluster settings::
  737. 2020-08-25T18:19:01.8511026Z broker - 'cluster1'
  738. 2020-08-25T18:19:01.8511324Z clusters - 'cluster1' 'cluster2'
  739. 2020-08-25T18:19:01.8511684Z nodes per cluster - ([cluster3]="control-plane worker worker" [cluster2]="control-plane worker worker" [cluster1]="control-plane worker worker" )
  740. 2020-08-25T18:19:01.8512004Z install submariner - ([cluster2]="true" [cluster1]="true" )
  741. 2020-08-25T18:19:01.8525933Z [lighthouse]$ rm -rf /go/src/github.com/submariner-io/lighthouse/output/kubeconfigs
  742. 2020-08-25T18:19:01.8548384Z [lighthouse]$ mkdir -p /go/src/github.com/submariner-io/lighthouse/output/kubeconfigs
  743. 2020-08-25T18:19:01.8585939Z [lighthouse]$ run_local_registry
  744. 2020-08-25T18:19:01.8597862Z [lighthouse]$ run_local_registry
  745. 2020-08-25T18:19:01.8608129Z [lighthouse]$ registry_running
  746. 2020-08-25T18:19:01.8620168Z [lighthouse]$ registry_running
  747. 2020-08-25T18:19:01.8633249Z [lighthouse]$ docker ps --filter name=^/?kind-registry$
  748. 2020-08-25T18:19:01.8647537Z [lighthouse]$ grep kind-registry
  749. 2020-08-25T18:19:02.2007716Z [lighthouse]$ return 0
  750. 2020-08-25T18:19:02.2010019Z Deploying local registry kind-registry to serve images centrally.
  751. 2020-08-25T18:19:02.2023182Z [lighthouse]$ local volume_flag
  752. 2020-08-25T18:19:02.2038425Z [lighthouse]$ volume_flag=-v /dev/shm/kind-registry:/var/lib/registry
  753. 2020-08-25T18:19:02.2048884Z [lighthouse]$ docker run -d -v /dev/shm/kind-registry:/var/lib/registry -p 5000:5000 --restart=always --name kind-registry registry:2
  754. 2020-08-25T18:19:02.5458863Z Unable to find image 'registry:2' locally
  755. 2020-08-25T18:19:03.4908881Z 2: Pulling from library/registry
  756. 2020-08-25T18:19:03.9136404Z cbdbe7a5bc2a: Already exists
  757. 2020-08-25T18:19:03.9171402Z 47112e65547d: Pulling fs layer
  758. 2020-08-25T18:19:03.9171554Z 46bcb632e506: Pulling fs layer
  759. 2020-08-25T18:19:03.9171680Z c1cc712bcecd: Pulling fs layer
  760. 2020-08-25T18:19:03.9171819Z 3db6272dcbfa: Pulling fs layer
  761. 2020-08-25T18:19:03.9176536Z 3db6272dcbfa: Waiting
  762. 2020-08-25T18:19:04.2671138Z c1cc712bcecd: Verifying Checksum
  763. 2020-08-25T18:19:04.2671292Z c1cc712bcecd: Download complete
  764. 2020-08-25T18:19:04.2963001Z 47112e65547d: Verifying Checksum
  765. 2020-08-25T18:19:04.2963143Z 47112e65547d: Download complete
  766. 2020-08-25T18:19:04.3704043Z 46bcb632e506: Verifying Checksum
  767. 2020-08-25T18:19:04.3704322Z 46bcb632e506: Download complete
  768. 2020-08-25T18:19:04.4334957Z 47112e65547d: Pull complete
  769. 2020-08-25T18:19:04.6206357Z 3db6272dcbfa: Verifying Checksum
  770. 2020-08-25T18:19:04.6206543Z 3db6272dcbfa: Download complete
  771. 2020-08-25T18:19:04.6953716Z 46bcb632e506: Pull complete
  772. 2020-08-25T18:19:04.7662275Z c1cc712bcecd: Pull complete
  773. 2020-08-25T18:19:04.8341098Z 3db6272dcbfa: Pull complete
  774. 2020-08-25T18:19:04.8409604Z Digest: sha256:8be26f81ffea54106bae012c6f349df70f4d5e7e2ec01b143c46e2c03b9e551d
  775. 2020-08-25T18:19:04.8433568Z Status: Downloaded newer image for registry:2
  776. 2020-08-25T18:19:07.2713114Z 34d8f4dcf5539396011d7ee1598edb5e86f114a41282cbbccebe398793ec34d6
  777. 2020-08-25T18:19:07.9720132Z [lighthouse]$ registry_ip=172.17.0.3
  778. 2020-08-25T18:19:07.9739657Z [lighthouse]$ docker inspect -f {{.NetworkSettings.IPAddress}} kind-registry
  779. 2020-08-25T18:19:08.3018868Z [lighthouse]$ declare_cidrs
  780. 2020-08-25T18:19:08.3030880Z [lighthouse]$ declare_cidrs
  781. 2020-08-25T18:19:08.3042248Z [lighthouse]$ declare -gA cluster_CIDRs service_CIDRs global_CIDRs
  782. 2020-08-25T18:19:08.3054165Z [lighthouse]$ i=1
  783. 2020-08-25T18:19:08.3064360Z [lighthouse]$ [cluster1] add_cluster_cidrs 1 cluster1
  784. 2020-08-25T18:19:08.3074975Z [lighthouse]$ [cluster1] add_cluster_cidrs 1 cluster1
  785. 2020-08-25T18:19:08.3087670Z [lighthouse]$ [cluster1] local val=1
  786. 2020-08-25T18:19:08.3098213Z [lighthouse]$ [cluster1] local idx=cluster1
  787. 2020-08-25T18:19:08.3112954Z [lighthouse]$ [cluster1] cluster_CIDRs[cluster1]=10.241.0.0/16
  788. 2020-08-25T18:19:08.3126572Z [lighthouse]$ [cluster1] service_CIDRs[cluster1]=100.91.0.0/16
  789. 2020-08-25T18:19:08.3137726Z [lighthouse]$ [cluster1] i=2
  790. 2020-08-25T18:19:08.3151680Z [lighthouse]$ [cluster2] add_cluster_cidrs 2 cluster2
  791. 2020-08-25T18:19:08.3163920Z [lighthouse]$ [cluster2] add_cluster_cidrs 2 cluster2
  792. 2020-08-25T18:19:08.3175677Z [lighthouse]$ [cluster2] local val=2
  793. 2020-08-25T18:19:08.3185130Z [lighthouse]$ [cluster2] local idx=cluster2
  794. 2020-08-25T18:19:08.3199728Z [lighthouse]$ [cluster2] cluster_CIDRs[cluster2]=10.242.0.0/16
  795. 2020-08-25T18:19:08.3217206Z [lighthouse]$ [cluster2] service_CIDRs[cluster2]=100.92.0.0/16
  796. 2020-08-25T18:19:08.3228092Z [lighthouse]$ [cluster2] i=3
  797. 2020-08-25T18:19:08.3241060Z [lighthouse]$ [cluster2] run_all_clusters with_retries 3 create_kind_cluster
  798. 2020-08-25T18:19:08.3254715Z [lighthouse]$ [cluster2] run_all_clusters with_retries 3 create_kind_cluster
  799. 2020-08-25T18:19:08.3266456Z [lighthouse]$ [cluster2] run_parallel cluster1 cluster2 with_retries 3 create_kind_cluster
  800. 2020-08-25T18:19:08.3281996Z [lighthouse]$ [cluster2] run_parallel cluster1 cluster2 cluster1 cluster2 with_retries 3 create_kind_cluster
  801. 2020-08-25T18:19:08.3294920Z [lighthouse]$ [cluster2] local cmnd=with_retries
  802. 2020-08-25T18:19:08.3305804Z [lighthouse]$ [cluster2] declare -A pids
  803. 2020-08-25T18:19:08.3324438Z [lighthouse]$ [cluster2] eval echo cluster1 cluster2
  804. 2020-08-25T18:19:08.3346919Z [lighthouse]$ [cluster1] pids[cluster1]=3385
  805. 2020-08-25T18:19:08.3350227Z [lighthouse]$ [cluster1] set -o pipefail
  806. 2020-08-25T18:19:08.3369548Z [lighthouse]$ [cluster2] pids[cluster2]=3388
  807. 2020-08-25T18:19:08.3377276Z [lighthouse]$ [cluster1] with_retries 3 create_kind_cluster
  808. 2020-08-25T18:19:08.3389962Z [lighthouse]$ [cluster2] set -o pipefail
  809. 2020-08-25T18:19:08.3403698Z [lighthouse]$ [cluster2] wait 3388
  810. 2020-08-25T18:19:08.3406258Z [lighthouse]$ [cluster1] sed /\[cluster1]/!s/^/[cluster1] /
  811. 2020-08-25T18:19:08.3417846Z [lighthouse]$ [cluster2] with_retries 3 create_kind_cluster
  812. 2020-08-25T18:19:08.3446330Z [lighthouse]$ [cluster2] sed /\[cluster2]/!s/^/[cluster2] /
  813. 2020-08-25T18:22:48.7789171Z [lighthouse]$ [cluster2] with_retries
  814. 2020-08-25T18:22:48.7790538Z [lighthouse]$ [cluster2] local retries
  815. 2020-08-25T18:22:48.7791448Z [lighthouse]$ [cluster2] retries=1 2 3
  816. 2020-08-25T18:22:48.7791787Z [lighthouse]$ [cluster2] eval echo {1..3}
  817. 2020-08-25T18:22:48.7792257Z [lighthouse]$ [cluster2] local cmnd=create_kind_cluster
  818. 2020-08-25T18:22:48.7792782Z [lighthouse]$ [cluster2] wait 3418
  819. 2020-08-25T18:22:48.7793096Z [lighthouse]$ [cluster2] create_kind_cluster
  820. 2020-08-25T18:22:48.7799032Z [lighthouse]$ [cluster2] create_kind_cluster
  821. 2020-08-25T18:22:48.7799839Z [lighthouse]$ [cluster2] export KUBECONFIG=/go/src/github.com/submariner-io/lighthouse/output/kubeconfigs/kind-config-cluster2
  822. 2020-08-25T18:22:48.7800395Z [lighthouse]$ [cluster2] rm -f /go/src/github.com/submariner-io/lighthouse/output/kubeconfigs/kind-config-cluster2
  823. 2020-08-25T18:22:48.7801212Z [lighthouse]$ [cluster2] kind get clusters
  824. 2020-08-25T18:22:48.7801641Z [lighthouse]$ [cluster2] grep -q ^cluster2$
  825. 2020-08-25T18:22:48.7801884Z [cluster2] No kind clusters found.
  826. 2020-08-25T18:22:48.7802099Z [cluster2] Creating KIND cluster...
  827. 2020-08-25T18:22:48.7802499Z [lighthouse]$ [cluster2] generate_cluster_yaml
  828. 2020-08-25T18:22:48.7802928Z [lighthouse]$ [cluster2] generate_cluster_yaml
  829. 2020-08-25T18:22:48.7803358Z [lighthouse]$ [cluster2] local pod_cidr=10.242.0.0/16
  830. 2020-08-25T18:22:48.7803798Z [lighthouse]$ [cluster2] local service_cidr=100.92.0.0/16
  831. 2020-08-25T18:22:48.7804233Z [lighthouse]$ [cluster2] local dns_domain=cluster2.local
  832. 2020-08-25T18:22:48.7804658Z [lighthouse]$ [cluster2] local disable_cni=false
  833. 2020-08-25T18:22:48.7805074Z [lighthouse]$ [cluster2] disable_cni=true
  834. 2020-08-25T18:22:48.7806219Z [lighthouse]$ [cluster2] local nodes
  835. 2020-08-25T18:22:48.7806667Z [lighthouse]$ [cluster2] nodes=
  836. 2020-08-25T18:22:48.7807065Z [cluster2] - role: control-plane
  837. 2020-08-25T18:22:48.7807465Z [lighthouse]$ [cluster2] nodes=
  838. 2020-08-25T18:22:48.7815477Z [cluster2] - role: control-plane
  839. 2020-08-25T18:22:48.7816317Z [cluster2] - role: worker
  840. 2020-08-25T18:22:48.7816916Z [lighthouse]$ [cluster2] nodes=
  841. 2020-08-25T18:22:48.7817517Z [cluster2] - role: control-plane
  842. 2020-08-25T18:22:48.7818060Z [cluster2] - role: worker
  843. 2020-08-25T18:22:48.7818649Z [cluster2] - role: worker
  844. 2020-08-25T18:22:48.7819168Z [cluster2] 
  845. 2020-08-25T18:22:48.7819737Z [cluster2] 
  846. 2020-08-25T18:22:48.7820295Z [lighthouse]$ [cluster2] eval echo "kind: Cluster
  847. 2020-08-25T18:22:48.7820912Z [cluster2] apiVersion: kind.x-k8s.io/v1alpha4
  848. 2020-08-25T18:22:48.7821159Z [cluster2] networking:
  849. 2020-08-25T18:22:48.7821605Z [cluster2] disableDefaultCNI: ${disable_cni}
  850. 2020-08-25T18:22:48.7821839Z [cluster2] podSubnet: ${pod_cidr}
  851. 2020-08-25T18:22:48.7822069Z [cluster2] serviceSubnet: ${service_cidr}
  852. 2020-08-25T18:22:48.7822475Z [cluster2] containerdConfigPatches:
  853. 2020-08-25T18:22:48.7822905Z [cluster2] - |-
  854. 2020-08-25T18:22:48.7823832Z [cluster2] [plugins.\"io.containerd.grpc.v1.cri\".registry.mirrors.\"localhost:5000\"]
  855. 2020-08-25T18:22:48.7824951Z [cluster2] endpoint = [\"http://${registry_ip}:5000\"]
  856. 2020-08-25T18:22:48.7825529Z [cluster2] kubeadmConfigPatches:
  857. 2020-08-25T18:22:48.7825996Z [cluster2] - |
  858. 2020-08-25T18:22:48.7828377Z [cluster2] apiVersion: kubeadm.k8s.io/v1beta2
  859. 2020-08-25T18:22:48.7828676Z [cluster2] kind: ClusterConfiguration
  860. 2020-08-25T18:22:48.7829059Z [cluster2] metadata:
  861. 2020-08-25T18:22:48.7829307Z [cluster2] name: config
  862. 2020-08-25T18:22:48.7829508Z [cluster2] networking:
  863. 2020-08-25T18:22:48.7829883Z [cluster2] podSubnet: ${pod_cidr}
  864. 2020-08-25T18:22:48.7830128Z [cluster2] serviceSubnet: ${service_cidr}
  865. 2020-08-25T18:22:48.7830336Z [cluster2] dnsDomain: ${dns_domain}
  866. 2020-08-25T18:22:48.7831039Z [cluster2] nodes:${nodes}"
  867. 2020-08-25T18:22:48.7831694Z [lighthouse]$ [cluster2] cat /opt/shipyard/scripts/resources/kind-cluster-config.yaml
  868. 2020-08-25T18:22:48.7832325Z [lighthouse]$ [cluster2] local image_flag=
  869. 2020-08-25T18:22:48.7833015Z [lighthouse]$ [cluster2] image_flag=--image=kindest/node:v1.17.0
  870. 2020-08-25T18:22:48.7833905Z [lighthouse]$ [cluster2] kind create cluster --image=kindest/node:v1.17.0 --name=cluster2 --config=/opt/shipyard/scripts/resources/cluster2-config.yaml
  871. 2020-08-25T18:22:48.7834471Z [cluster2] Creating cluster "cluster2" ...
  872. 2020-08-25T18:22:48.7835146Z [cluster2] • Ensuring node image (kindest/node:v1.17.0) 🖼 ...
  873. 2020-08-25T18:22:48.7836054Z [cluster2] ✓ Ensuring node image (kindest/node:v1.17.0) 🖼
  874. 2020-08-25T18:22:48.7836753Z [cluster2] • Preparing nodes 📦 📦 📦 ...
  875. 2020-08-25T18:22:48.7837306Z [cluster2] ✓ Preparing nodes 📦 📦 📦
  876. 2020-08-25T18:22:48.7837898Z [cluster2] • Writing configuration 📜 ...
  877. 2020-08-25T18:22:48.7841729Z [cluster2] ✓ Writing configuration 📜
  878. 2020-08-25T18:22:48.7842018Z [cluster2] • Starting control-plane 🕹️ ...
  879. 2020-08-25T18:22:48.7842270Z [cluster2] ✓ Starting control-plane 🕹️
  880. 2020-08-25T18:22:48.7842574Z [cluster2] • Installing StorageClass 💾 ...
  881. 2020-08-25T18:22:48.7842844Z [cluster2] ✓ Installing StorageClass 💾
  882. 2020-08-25T18:22:48.7843676Z [cluster2] • Joining worker nodes 🚜 ...
  883. 2020-08-25T18:22:48.7843941Z [cluster2] ✓ Joining worker nodes 🚜
  884. 2020-08-25T18:22:48.7845947Z [cluster2] Set kubectl context to "kind-cluster2"
  885. 2020-08-25T18:22:48.7846103Z [cluster2] You can now use your cluster with:
  886. 2020-08-25T18:22:48.7846217Z [cluster2]
  887. 2020-08-25T18:22:48.7846522Z [cluster2] kubectl cluster-info --context kind-cluster2
  888. 2020-08-25T18:22:48.7847130Z [cluster2]
  889. 2020-08-25T18:22:48.7847638Z [cluster2] Have a question, bug, or feature request? Let us know! https://kind.sigs.k8s.io/#community 🙂
  890. 2020-08-25T18:22:48.7847955Z [lighthouse]$ [cluster2] kind_fixup_config
  891. 2020-08-25T18:22:48.7848648Z [lighthouse]$ [cluster2] kind_fixup_config
  892. 2020-08-25T18:22:48.7849107Z [lighthouse]$ [cluster2] local master_ip=172.17.0.5
  893. 2020-08-25T18:22:48.7849980Z [lighthouse]$ [cluster2] docker inspect -f {{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}} cluster2-control-plane
  894. 2020-08-25T18:22:57.1500144Z [lightho[lighthouse]$ [cluster1] with_retries
  895. 2020-08-25T18:22:57.1501071Z [lighthouse]$ [cluster1] local retries
  896. 2020-08-25T18:22:57.1501546Z [lighthouse]$ [cluster1] retries=1 2 3
  897. 2020-08-25T18:22:57.1501972Z [lighthouse]$ [cluster1] eval echo {1..3}
  898. 2020-08-25T18:22:57.1502407Z [lighthouse]$ [cluster1] local cmnd=create_kind_cluster
  899. 2020-08-25T18:22:57.1502826Z [lighthouse]$ [cluster1] wait 3413
  900. 2020-08-25T18:22:57.1503245Z [lighthouse]$ [cluster1] create_kind_cluster
  901. 2020-08-25T18:22:57.1503674Z [lighthouse]$ [cluster1] create_kind_cluster
  902. 2020-08-25T18:22:57.1504238Z [lighthouse]$ [cluster1] export KUBECONFIG=/go/src/github.com/submariner-io/lighthouse/output/kubeconfigs/kind-config-cluster1
  903. 2020-08-25T18:22:57.1505360Z [lighthouse]$ [cluster1] rm -f /go/src/github.com/submariner-io/lighthouse/output/kubeconfigs/kind-config-cluster1
  904. 2020-08-25T18:22:57.1506309Z [lighthouse]$ [cluster1] kind get clusters
  905. 2020-08-25T18:22:57.1506775Z [lighthouse]$ [cluster1] grep -q ^cluster1$
  906. 2020-08-25T18:22:57.1507040Z [cluster1] No kind clusters found.
  907. 2020-08-25T18:22:57.1507269Z [cluster1] Creating KIND cluster...
  908. 2020-08-25T18:22:57.1507850Z [lighthouse]$ [cluster1] generate_cluster_yaml
  909. 2020-08-25T18:22:57.1508292Z [lighthouse]$ [cluster1] generate_cluster_yaml
  910. 2020-08-25T18:22:57.1508744Z [lighthouse]$ [cluster1] local pod_cidr=10.241.0.0/16
  911. 2020-08-25T18:22:57.1509875Z [lighthouse]$ [cluster1] local service_cidr=100.91.0.0/16
  912. 2020-08-25T18:22:57.1510382Z [lighthouse]$ [cluster1] local dns_domain=cluster1.local
  913. 2020-08-25T18:22:57.1510863Z [lighthouse]$ [cluster1] local disable_cni=false
  914. 2020-08-25T18:22:57.1512227Z [lighthouse]$ [cluster1] disable_cni=true
  915. 2020-08-25T18:22:57.1512807Z [lighthouse]$ [cluster1] local nodes
  916. 2020-08-25T18:22:57.1513621Z [lighthouse]$ [cluster1] nodes=
  917. 2020-08-25T18:22:57.1514265Z [cluster1] - role: control-plane
  918. 2020-08-25T18:22:57.1514711Z [lighthouse]$ [cluster1] nodes=
  919. 2020-08-25T18:22:57.1515149Z [cluster1] - role: control-plane
  920. 2020-08-25T18:22:57.1516579Z [cluster1] - role: worker
  921. 2020-08-25T18:22:57.1517048Z [lighthouse]$ [cluster1] nodes=
  922. 2020-08-25T18:22:57.1517479Z [cluster1] - role: control-plane
  923. 2020-08-25T18:22:57.1517890Z [cluster1] - role: worker
  924. 2020-08-25T18:22:57.1518300Z [cluster1] - role: worker
  925. 2020-08-25T18:22:57.1518701Z [cluster1] 
  926. 2020-08-25T18:22:57.1519085Z [cluster1] 
  927. 2020-08-25T18:22:57.1519526Z [lighthouse]$ [cluster1] eval echo "kind: Cluster
  928. 2020-08-25T18:22:57.1520278Z [cluster1] apiVersion: kind.x-k8s.io/v1alpha4
  929. 2020-08-25T18:22:57.1520534Z [cluster1] networking:
  930. 2020-08-25T18:22:57.1520758Z [cluster1] disableDefaultCNI: ${disable_cni}
  931. 2020-08-25T18:22:57.1520984Z [cluster1] podSubnet: ${pod_cidr}
  932. 2020-08-25T18:22:57.1521215Z [cluster1] serviceSubnet: ${service_cidr}
  933. 2020-08-25T18:22:57.1521438Z [cluster1] containerdConfigPatches:
  934. 2020-08-25T18:22:57.1521822Z [cluster1] - |-
  935. 2020-08-25T18:22:57.1522375Z [cluster1] [plugins.\"io.containerd.grpc.v1.cri\".registry.mirrors.\"localhost:5000\"]
  936. 2020-08-25T18:22:57.1523028Z [cluster1] endpoint = [\"http://${registry_ip}:5000\"]
  937. 2020-08-25T18:22:57.1523451Z [cluster1] kubeadmConfigPatches:
  938. 2020-08-25T18:22:57.1524293Z [cluster1] - |
  939. 2020-08-25T18:22:57.1524717Z [cluster1] apiVersion: kubeadm.k8s.io/v1beta2
  940. 2020-08-25T18:22:57.1524881Z [cluster1] kind: ClusterConfiguration
  941. 2020-08-25T18:22:57.1525204Z [cluster1] metadata:
  942. 2020-08-25T18:22:57.1525477Z [cluster1] name: config
  943. 2020-08-25T18:22:57.1525618Z [cluster1] networking:
  944. 2020-08-25T18:22:57.1525758Z [cluster1] podSubnet: ${pod_cidr}
  945. 2020-08-25T18:22:57.1525906Z [cluster1] serviceSubnet: ${service_cidr}
  946. 2020-08-25T18:22:57.1526033Z [cluster1] dnsDomain: ${dns_domain}
  947. 2020-08-25T18:22:57.1526370Z [cluster1] nodes:${nodes}"
  948. 2020-08-25T18:22:57.1526699Z [lighthouse]$ [cluster1] cat /opt/shipyard/scripts/resources/kind-cluster-config.yaml
  949. 2020-08-25T18:22:57.1527006Z [lighthouse]$ [cluster1] local image_flag=
  950. 2020-08-25T18:22:57.1527386Z [lighthouse]$ [cluster1] image_flag=--image=kindest/node:v1.17.0
  951. 2020-08-25T18:22:57.1528247Z [lighthouse]$ [cluster1] kind create cluster --image=kindest/node:v1.17.0 --name=cluster1 --config=/opt/shipyard/scripts/resources/cluster1-config.yaml
  952. 2020-08-25T18:22:57.1528443Z [cluster1] Creating cluster "cluster1" ...
  953. 2020-08-25T18:22:57.1528848Z [cluster1] • Ensuring node image (kindest/node:v1.17.0) 🖼 ...
  954. 2020-08-25T18:22:57.1529360Z [cluster1] ✓ Ensuring node image (kindest/node:v1.17.0) 🖼
  955. 2020-08-25T18:22:57.1529714Z [cluster1] • Preparing nodes 📦 📦 📦 ...
  956. 2020-08-25T18:22:57.1529980Z [cluster1] ✓ Preparing nodes 📦 📦 📦
  957. 2020-08-25T18:22:57.1530248Z [cluster1] • Writing configuration 📜 ...
  958. 2020-08-25T18:22:57.1530644Z [cluster1] ✓ Writing configuration 📜
  959. 2020-08-25T18:22:57.1530974Z [cluster1] • Starting control-plane 🕹️ ...
  960. 2020-08-25T18:22:57.1531243Z [cluster1] ✓ Starting control-plane 🕹️
  961. 2020-08-25T18:22:57.1531514Z [cluster1] • Installing StorageClass 💾 ...
  962. 2020-08-25T18:22:57.1531777Z [cluster1] ✓ Installing StorageClass 💾
  963. 2020-08-25T18:22:57.1532045Z [cluster1] • Joining worker nodes 🚜 ...
  964. 2020-08-25T18:22:57.1532304Z [cluster1] ✓ Joining worker nodes 🚜
  965. 2020-08-25T18:22:57.1532575Z [cluster1] Set kubectl context to "kind-cluster1"
  966. 2020-08-25T18:22:57.1532688Z [cluster1] You can now use your cluster with:
  967. 2020-08-25T18:22:57.1532807Z [cluster1]
  968. 2020-08-25T18:22:57.1533082Z [cluster1] kubectl cluster-info --context kind-cluster1
  969. 2020-08-25T18:22:57.1533236Z [cluster1]
  970. 2020-08-25T18:22:57.1533501Z [cluster1] Thanks for using kind! 😊
  971. 2020-08-25T18:22:57.1533774Z [lighthouse]$ [cluster1] kind_fixup_config
  972. 2020-08-25T18:22:57.1534049Z [lighthouse]$ [cluster1] kind_fixup_config
  973. 2020-08-25T18:22:57.1534342Z [lighthouse]$ [cluster1] local master_ip=172.17.0.4
  974. 2020-08-25T18:22:57.1534729Z [lighthouse]$ [cluster1] docker inspect -f {{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}} cluster1-control-plane
  975. 2020-08-25T18:22:57.1535020Z [lighthouse]$ [cluster1] head -n 1
  976. 2020-08-25T18:23:34.9951061Z [lighthouse]$ [cluster1] sed -i -- s/server: .*/server: https:\/\/172.17.0.4:6443/g /go/src/github.com/submariner-io/lighthouse/output/kubeconfigs/kind-config-cluster1
  977. 2020-08-25T18:23:34.9951626Z [lighthouse]$ [cluster1] sed -i -- s/user: kind-.*/user: cluster1/g /go/src/github.com/submariner-io/lighthouse/output/kubeconfigs/kind-config-cluster1
  978. 2020-08-25T18:23:34.9952678Z [lighthouse]$ [cluster1] sed -i -- s/name: kind-.*/name: cluster1/g /go/src/github.com/submariner-io/lighthouse/output/kubeconfigs/kind-config-cluster1
  979. 2020-08-25T18:23:34.9953333Z [lighthouse]$ [cluster1] sed -i -- s/cluster: kind-.*/cluster: cluster1/g /go/src/github.com/submariner-io/lighthouse/output/kubeconfigs/kind-config-cluster1
  980. 2020-08-25T18:23:34.9953929Z [lighthouse]$ [cluster1] sed -i -- s/current-context: .*/current-context: cluster1/g /go/src/github.com/submariner-io/lighthouse/output/kubeconfigs/kind-config-cluster1
  981. 2020-08-25T18:23:34.9954304Z [lighthouse]$ [cluster1] chmod a+r /go/src/github.com/submariner-io/lighthouse/output/kubeconfigs/kind-config-cluster1
  982. 2020-08-25T18:23:34.9954606Z [lighthouse]$ [cluster1] deploy_cluster_capabilities
  983. 2020-08-25T18:23:34.9954889Z [lighthouse]$ [cluster1] deploy_cluster_capabilities
  984. 2020-08-25T18:23:34.9955159Z [lighthouse]$ [cluster1] deploy_cni
  985. 2020-08-25T18:23:34.9955422Z [lighthouse]$ [cluster1] wait 4624
  986. 2020-08-25T18:23:34.9955690Z [lighthouse]$ [cluster1] deploy_cni
  987. 2020-08-25T18:23:34.9956139Z [lighthouse]$ [cluster1] eval deploy_weave_cni
  988. 2020-08-25T18:23:34.9956417Z [lighthouse]$ [cluster1] deploy_weave_cni
  989. 2020-08-25T18:23:34.9956696Z [lighthouse]$ [cluster1] deploy_weave_cni
  990. 2020-08-25T18:23:34.9956805Z [cluster1] Applying weave network...
  991. 2020-08-25T18:23:34.9957486Z [lighthouse]$ [cluster1] kubectl apply -f https://cloud.weave.works/k8s/net?k8s-version=v1.17.0&env.IPALLOC_RANGE=10.241.0.0/16
  992. 2020-08-25T18:23:34.9957996Z [lighthouse]$ [cluster1] kubectl apply -f https://cloud.weave.works/k8s/net?k8s-version=v1.17.0&env.IPALLOC_RANGE=10.241.0.0/16
  993. 2020-08-25T18:23:34.9958520Z [lighthouse]$ [cluster1] command kubectl --context=cluster1 apply -f https://cloud.weave.works/k8s/net?k8s-version=v1.17.0&env.IPALLOC_RANGE=10.241.0.0/16
  994. 2020-08-25T18:23:34.9958876Z [cluster1] serviceaccount/weave-net created
  995. 2020-08-25T18:23:34.9959201Z [cluster1] clusterrole.rbac.authorization.k8s.io/weave-net created
  996. 2020-08-25T18:23:34.9959527Z [cluster1] clusterrolebinding.rbac.authorization.k8s.io/weave-net created
  997. 2020-08-25T18:23:34.9959834Z [cluster1] role.rbac.authorization.k8s.io/weave-net created
  998. 2020-08-25T18:23:34.9960300Z [cluster1] rolebinding.rbac.authorization.k8s.io/weave-net created
  999. 2020-08-25T18:23:34.9960647Z [cluster1] daemonset.apps/weave-net created
  1000. 2020-08-25T18:23:34.9960939Z [cluster1] Waiting for weave-net pods to be ready...
  1001. 2020-08-25T18:23:34.9961293Z [lighthouse]$ [cluster1] kubectl wait --for=condition=Ready pods -l name=weave-net -n kube-system --timeout=5m
  1002. 2020-08-25T18:23:34.9961860Z [lighthouse]$ [cluster1] kubectl wait --for=condition=Ready pods -l name=weave-net -n kube-system --timeout=5m
  1003. 2020-08-25T18:23:34.9962269Z [lighthouse]$ [cluster1] command kubectl --context=cluster1 wait --for=condition=Ready pods -l name=weave-net -n kube-system --timeout=5m
  1004. 2020-08-25T18:23:34.9963072Z [lighthouse]$ [cluster1] kubectl --context=cluster1 wait --for=condition=Ready pods -l name=weave-net -n kube-system --timeout=5m
  1005. 2020-08-25T18:23:34.9963543Z [cluster1] pod/weave-net-bs5x6 condition met
  1006. 2020-08-25T18:23:34.9964112Z [cluster1] pod/weave-net-tmpqh condition met
  1007. 2020-08-25T18:23:34.9964400Z [cluster1] pod/weave-net-xbjcc condition met
  1008. 2020-08-25T18:23:34.9964693Z [cluster1] Waiting for core-dns deployment to be ready...
  1009. 2020-08-25T18:23:34.9965045Z [lighthouse]$ [cluster1] kubectl -n kube-system rollout status deploy/coredns --timeout=5m
  1010. 2020-08-25T18:23:34.9965770Z [lighthouse]$ [cluster1] kubectl -n kube-system rollout status deploy/coredns --timeout=5m
  1011. 2020-08-25T18:23:34.9966280Z [lighthouse]$ [cluster1] command kubectl --context=cluster1 -n kube-system rollout status deploy/coredns --timeout=5m
  1012. 2020-08-25T18:23:34.9966689Z [lighthouse]$ [cluster1] kubectl --context=cluster1 -n kube-system rollout status deploy/coredns --timeout=5m
  1013. 2020-08-25T18:23:34.9967033Z [cluster1] Waiting for deployment "coredns" rollout to finish: 0 of 2 updated replicas are available...
  1014. 2020-08-25T18:23:34.9967195Z [cluster1] Waiting for deployment "coredns" rollout to finish: 1 of 2 updated replicas are available...
  1015. 2020-08-25T18:23:34.9967347Z [cluster1] deployment "coredns" successfully rolled out
  1016. 2020-08-25T18:23:34.9967691Z [lighthouse]$ [cluster1] return 0
  1017. 2020-08-25T18:23:47.4427061Z use]$ [cluster2] head -n 1
  1018. 2020-08-25T18:23:47.4427778Z [lighthouse]$ [cluster2] sed -i -- s/server: .*/server: https:\/\/172.17.0.5:6443/g /go/src/github.com/submariner-io/lighthouse/output/kubeconfigs/kind-config-cluster2
  1019. 2020-08-25T18:23:47.4428728Z [lighthouse]$ [cluster2] sed -i -- s/user: kind-.*/user: cluster2/g /go/src/github.com/submariner-io/lighthouse/output/kubeconfigs/kind-config-cluster2
  1020. 2020-08-25T18:23:47.4430111Z [lighthouse]$ [cluster2] sed -i -- s/name: kind-.*/name: cluster2/g /go/src/github.com/submariner-io/lighthouse/output/kubeconfigs/kind-config-cluster2
  1021. 2020-08-25T18:23:47.4432275Z [lighthouse]$ [cluster2] sed -i -- s/cluster: kind-.*/cluster: cluster2/g /go/src/github.com/submariner-io/lighthouse/output/kubeconfigs/kind-config-cluster2
  1022. 2020-08-25T18:23:47.4433346Z [lighthouse]$ [cluster2] sed -i -- s/current-context: .*/current-context: cluster2/g /go/src/github.com/submariner-io/lighthouse/output/kubeconfigs/kind-config-cluster2
  1023. 2020-08-25T18:23:47.4434274Z [lighthouse]$ [cluster2] chmod a+r /go/src/github.com/submariner-io/lighthouse/output/kubeconfigs/kind-config-cluster2
  1024. 2020-08-25T18:23:47.4435614Z [lighthouse]$ [cluster2] wait 4487
  1025. 2020-08-25T18:23:47.4436381Z [lighthouse]$ [cluster2] deploy_cluster_capabilities
  1026. 2020-08-25T18:23:47.4437430Z [lighthouse]$ [cluster2] deploy_cluster_capabilities
  1027. 2020-08-25T18:23:47.4438008Z [lighthouse]$ [cluster2] deploy_cni
  1028. 2020-08-25T18:23:47.4439224Z [lighthouse]$ [cluster2] deploy_cni
  1029. 2020-08-25T18:23:47.4440487Z [lighthouse]$ [cluster2] eval deploy_weave_cni
  1030. 2020-08-25T18:23:47.4441176Z [lighthouse]$ [cluster2] deploy_weave_cni
  1031. 2020-08-25T18:23:47.4441712Z [lighthouse]$ [cluster2] deploy_weave_cni
  1032. 2020-08-25T18:23:47.4442058Z [cluster2] Applying weave network...
  1033. 2020-08-25T18:23:47.4443014Z [lighthouse]$ [cluster2] kubectl apply -f https://cloud.weave.works/k8s/net?k8s-version=v1.17.0&env.IPALLOC_RANGE=10.242.0.0/16
  1034. 2020-08-25T18:23:47.4453365Z [lighthouse]$ [cluster2] kubectl apply -f https://cloud.weave.works/k8s/net?k8s-version=v1.17.0&env.IPALLOC_RANGE=10.242.0.0/16
  1035. 2020-08-25T18:23:47.4454245Z [lighthouse]$ [cluster2] command kubectl --context=cluster2 apply -f https://cloud.weave.works/k8s/net?k8s-version=v1.17.0&env.IPALLOC_RANGE=10.242.0.0/16
  1036. 2020-08-25T18:23:47.4454605Z [cluster2] serviceaccount/weave-net created
  1037. 2020-08-25T18:23:47.4454915Z [cluster2] clusterrole.rbac.authorization.k8s.io/weave-net created
  1038. 2020-08-25T18:23:47.4455234Z [cluster2] clusterrolebinding.rbac.authorization.k8s.io/weave-net created
  1039. 2020-08-25T18:23:47.4455559Z [cluster2] role.rbac.authorization.k8s.io/weave-net created
  1040. 2020-08-25T18:23:47.4455866Z [cluster2] rolebinding.rbac.authorization.k8s.io/weave-net created
  1041. 2020-08-25T18:23:47.4456136Z [cluster2] daemonset.apps/weave-net created
  1042. 2020-08-25T18:23:47.4456426Z [cluster2] Waiting for weave-net pods to be ready...
  1043. 2020-08-25T18:23:47.4456794Z [lighthouse]$ [cluster2] kubectl wait --for=condition=Ready pods -l name=weave-net -n kube-system --timeout=5m
  1044. 2020-08-25T18:23:47.4457175Z [lighthouse]$ [cluster2] kubectl wait --for=condition=Ready pods -l name=weave-net -n kube-system --timeout=5m
  1045. 2020-08-25T18:23:47.4457584Z [lighthouse]$ [cluster2] command kubectl --context=cluster2 wait --for=condition=Ready pods -l name=weave-net -n kube-system --timeout=5m
  1046. 2020-08-25T18:23:47.4458136Z [lighthouse]$ [cluster2] kubectl --context=cluster2 wait --for=condition=Ready pods -l name=weave-net -n kube-system --timeout=5m
  1047. 2020-08-25T18:23:47.4458670Z [cluster2] pod/weave-net-dcsvk condition met
  1048. 2020-08-25T18:23:47.4459089Z [cluster2] pod/weave-net-mzrnn condition met
  1049. 2020-08-25T18:23:47.4459346Z [cluster2] pod/weave-net-scmr7 condition met
  1050. 2020-08-25T18:23:47.4459625Z [cluster2] Waiting for core-dns deployment to be ready...
  1051. 2020-08-25T18:23:47.4459943Z [lighthouse]$ [cluster2] kubectl -n kube-system rollout status deploy/coredns --timeout=5m
  1052. 2020-08-25T18:23:47.4460276Z [lighthouse]$ [cluster2] kubectl -n kube-system rollout status deploy/coredns --timeout=5m
  1053. 2020-08-25T18:23:47.4460637Z [lighthouse]$ [cluster2] command kubectl --context=cluster2 -n kube-system rollout status deploy/coredns --timeout=5m
  1054. 2020-08-25T18:23:47.4460994Z [lighthouse]$ [cluster2] kubectl --context=cluster2 -n kube-system rollout status deploy/coredns --timeout=5m
  1055. 2020-08-25T18:23:47.4461327Z [cluster2] Waiting for deployment "coredns" rollout to finish: 0 of 2 updated replicas are available...
  1056. 2020-08-25T18:23:47.4461490Z [cluster2] Waiting for deployment "coredns" rollout to finish: 1 of 2 updated replicas are available...
  1057. 2020-08-25T18:23:47.4461630Z [cluster2] deployment "coredns" successfully rolled out
  1058. 2020-08-25T18:23:47.4461911Z [lighthouse]$ [cluster2] return 0
  1059. 2020-08-25T18:23:47.4505503Z [lighthouse]$ [cluster2] wait 3385
  1060. 2020-08-25T18:23:47.4531454Z [lighthouse]$ [cluster2] print_clusters_message
  1061. 2020-08-25T18:23:47.4559800Z [lighthouse]$ [cluster2] print_clusters_message
  1062. 2020-08-25T18:23:47.4642984Z [lighthouse]$ [cluster2] cat
  1063. 2020-08-25T18:23:47.4686496Z Your virtual cluster(s) are deployed and working properly and can be accessed with:
  1064. 2020-08-25T18:23:47.4686723Z
  1065. 2020-08-25T18:23:47.4687565Z export KUBECONFIG=$(find $(git rev-parse --show-toplevel)/output/kubeconfigs/ -type f -printf %p:)
  1066. 2020-08-25T18:23:47.4687677Z
  1067. 2020-08-25T18:23:47.4687971Z $ kubectl config use-context cluster1 # or cluster2, cluster3..
  1068. 2020-08-25T18:23:47.4688065Z
  1069. 2020-08-25T18:23:47.4688205Z To clean everthing up, just run: make cleanup
  1070. 2020-08-25T18:23:47.4712307Z ./scripts/deploy --cluster_settings /go/src/github.com/submariner-io/lighthouse/scripts/cluster_settings --deploytool_broker_args '--service-discovery'
  1071. 2020-08-25T18:23:47.4853984Z [lighthouse]$ source /opt/shipyard/scripts/lib/deploy_funcs
  1072. 2020-08-25T18:23:47.4873953Z [lighthouse]$ . /opt/shipyard/scripts/lib/source_only
  1073. 2020-08-25T18:23:47.4889123Z [lighthouse]$ script_name=deploy_funcs
  1074. 2020-08-25T18:23:47.4909899Z [lighthouse]$ exec_name=deploy
  1075. 2020-08-25T18:23:47.4942301Z [lighthouse]$ source /opt/shipyard/scripts/lib/utils
  1076. 2020-08-25T18:23:47.4959165Z [lighthouse]$ . /opt/shipyard/scripts/lib/source_only
  1077. 2020-08-25T18:23:47.4969351Z [lighthouse]$ script_name=utils
  1078. 2020-08-25T18:23:47.4998134Z [lighthouse]$ exec_name=deploy
  1079. 2020-08-25T18:23:47.5025490Z [lighthouse]$ source /opt/shipyard/scripts/lib/version
  1080. 2020-08-25T18:23:47.5048246Z [lighthouse]$ . /opt/shipyard/scripts/lib/source_only
  1081. 2020-08-25T18:23:47.5121498Z [lighthouse]$ script_name=version
  1082. 2020-08-25T18:23:47.5134089Z [lighthouse]$ exec_name=deploy
  1083. 2020-08-25T18:23:47.5163088Z [lighthouse]$ git describe --tags --dirty=-dev --exclude=devel --exclude=latest
  1084. 2020-08-25T18:23:47.5764596Z fatal: No names found, cannot describe anything.
  1085. 2020-08-25T18:23:47.5781663Z [lighthouse]$ declare_kubeconfig
  1086. 2020-08-25T18:23:47.5794193Z [lighthouse]$ declare_kubeconfig
  1087. 2020-08-25T18:23:47.5804974Z [lighthouse]$ source /opt/shipyard/scripts/lib/kubecfg
  1088. 2020-08-25T18:23:47.5818095Z [lighthouse]$ export KUBECONFIG
  1089. 2020-08-25T18:23:47.5901056Z [lighthouse]$ KUBECONFIG=/go/src/github.com/submariner-io/lighthouse/output/kubeconfigs/kind-config-cluster2:/go/src/github.com/submariner-io/lighthouse/output/kubeconfigs/kind-config-cluster1:
  1090. 2020-08-25T18:23:47.5916850Z [lighthouse]$ find /go/src/github.com/submariner-io/lighthouse/output/kubeconfigs -type f -printf %p:
  1091. 2020-08-25T18:23:47.5944511Z [lighthouse]$ import_image quay.io/submariner/lighthouse-agent
  1092. 2020-08-25T18:23:47.5957418Z [lighthouse]$ import_image quay.io/submariner/lighthouse-agent
  1093. 2020-08-25T18:23:47.5966927Z [lighthouse]$ local orig_image=quay.io/submariner/lighthouse-agent
  1094. 2020-08-25T18:23:47.5976834Z [lighthouse]$ local versioned_image=quay.io/submariner/lighthouse-agent:dev
  1095. 2020-08-25T18:23:47.5985250Z [lighthouse]$ local local_image=localhost:5000/lighthouse-agent:local
  1096. 2020-08-25T18:23:47.5996043Z [lighthouse]$ docker tag quay.io/submariner/lighthouse-agent:dev localhost:5000/lighthouse-agent:local
  1097. 2020-08-25T18:23:47.9665403Z [lighthouse]$ docker push localhost:5000/lighthouse-agent:local
  1098. 2020-08-25T18:23:48.3392489Z The push refers to repository [localhost:5000/lighthouse-agent]
  1099. 2020-08-25T18:23:48.3610821Z 93db2d8aa80f: Preparing
  1100. 2020-08-25T18:23:48.3611014Z 370fba87584b: Preparing
  1101. 2020-08-25T18:23:48.3611336Z c7e9c12dd629: Preparing
  1102. 2020-08-25T18:23:48.3611702Z 29bad1020e6f: Preparing
  1103. 2020-08-25T18:23:48.9244764Z c7e9c12dd629: Pushed
  1104. 2020-08-25T18:23:48.9245478Z 370fba87584b: Pushed
  1105. 2020-08-25T18:23:49.1329115Z 93db2d8aa80f: Pushed
  1106. 2020-08-25T18:23:59.8966885Z 29bad1020e6f: Pushed
  1107. 2020-08-25T18:23:59.9210630Z local: digest: sha256:b34135525cd63677b4a81a7be81f04b6cb5fe3a5a886d4ea2b9beaf1380b7a5b size: 1155
  1108. 2020-08-25T18:23:59.9285478Z [lighthouse]$ import_image quay.io/submariner/lighthouse-coredns
  1109. 2020-08-25T18:23:59.9306005Z [lighthouse]$ import_image quay.io/submariner/lighthouse-coredns
  1110. 2020-08-25T18:23:59.9317015Z [lighthouse]$ local orig_image=quay.io/submariner/lighthouse-coredns
  1111. 2020-08-25T18:23:59.9364676Z [lighthouse]$ local versioned_image=quay.io/submariner/lighthouse-coredns:dev
  1112. 2020-08-25T18:23:59.9374522Z [lighthouse]$ local local_image=localhost:5000/lighthouse-coredns:local
  1113. 2020-08-25T18:23:59.9385610Z [lighthouse]$ docker tag quay.io/submariner/lighthouse-coredns:dev localhost:5000/lighthouse-coredns:local
  1114. 2020-08-25T18:24:00.3185978Z [lighthouse]$ docker push localhost:5000/lighthouse-coredns:local
  1115. 2020-08-25T18:24:00.6726938Z The push refers to repository [localhost:5000/lighthouse-coredns]
  1116. 2020-08-25T18:24:00.6823350Z d7ed5d9f34a3: Preparing
  1117. 2020-08-25T18:24:00.6823543Z d8e6c6d237be: Preparing
  1118. 2020-08-25T18:24:00.6823672Z 671ce03107cf: Preparing
  1119. 2020-08-25T18:24:03.4491052Z d7ed5d9f34a3: Pushed
  1120. 2020-08-25T18:24:03.7605991Z d8e6c6d237be: Pushed
  1121. 2020-08-25T18:24:07.3947411Z 671ce03107cf: Pushed
  1122. 2020-08-25T18:24:07.4137154Z local: digest: sha256:058418c628d4d9d503eea0e6f78f5a5f5e195bbc2da7230eed715cbb98de7ecc size: 953
  1123. 2020-08-25T18:24:07.4200288Z [lighthouse]$ /opt/shipyard/scripts/deploy.sh --cluster_settings /go/src/github.com/submariner-io/lighthouse/scripts/cluster_settings --deploytool_broker_args --service-discovery
  1124. 2020-08-25T18:24:07.5087034Z Running with: globalnet='false', deploytool='operator', deploytool_broker_args='--service-discovery', deploytool_submariner_args='', cluster_settings='/go/src/github.com/submariner-io/lighthouse/scripts/cluster_settings', timeout=5m
  1125. 2020-08-25T18:24:07.5108446Z [lighthouse]$ source /opt/shipyard/scripts/lib/version
  1126. 2020-08-25T18:24:07.5117725Z [lighthouse]$ . /opt/shipyard/scripts/lib/source_only
  1127. 2020-08-25T18:24:07.5128307Z [lighthouse]$ script_name=version
  1128. 2020-08-25T18:24:07.5138432Z [lighthouse]$ exec_name=deploy.sh
  1129. 2020-08-25T18:24:07.5156828Z [lighthouse]$ git describe --tags --dirty=-dev --exclude=devel --exclude=latest
  1130. 2020-08-25T18:24:07.5587714Z fatal: No names found, cannot describe anything.
  1131. 2020-08-25T18:24:07.5604951Z [lighthouse]$ source /opt/shipyard/scripts/lib/utils
  1132. 2020-08-25T18:24:07.5614146Z [lighthouse]$ . /opt/shipyard/scripts/lib/source_only
  1133. 2020-08-25T18:24:07.5628464Z [lighthouse]$ script_name=utils
  1134. 2020-08-25T18:24:07.5640781Z [lighthouse]$ exec_name=deploy.sh
  1135. 2020-08-25T18:24:07.5664349Z [lighthouse]$ source /opt/shipyard/scripts/lib/deploy_funcs
  1136. 2020-08-25T18:24:07.5680210Z [lighthouse]$ . /opt/shipyard/scripts/lib/source_only
  1137. 2020-08-25T18:24:07.5689122Z [lighthouse]$ script_name=deploy_funcs
  1138. 2020-08-25T18:24:07.5706315Z [lighthouse]$ exec_name=deploy.sh
  1139. 2020-08-25T18:24:07.5727484Z [lighthouse]$ source /opt/shipyard/scripts/lib/cluster_settings
  1140. 2020-08-25T18:24:07.5738218Z [lighthouse]$ . /opt/shipyard/scripts/lib/source_only
  1141. 2020-08-25T18:24:07.5749954Z [lighthouse]$ script_name=cluster_settings
  1142. 2020-08-25T18:24:07.5761972Z [lighthouse]$ exec_name=deploy.sh
  1143. 2020-08-25T18:24:07.5786493Z [lighthouse]$ broker=cluster1
  1144. 2020-08-25T18:24:07.5796037Z [lighthouse]$ declare -gA cluster_nodes
  1145. 2020-08-25T18:24:07.5809856Z [lighthouse]$ cluster_nodes[cluster1]=control-plane worker
  1146. 2020-08-25T18:24:07.5823164Z [lighthouse]$ cluster_nodes[cluster2]=control-plane worker
  1147. 2020-08-25T18:24:07.5833786Z [lighthouse]$ cluster_nodes[cluster3]=control-plane worker worker
  1148. 2020-08-25T18:24:07.5847491Z [lighthouse]$ declare -gA cluster_subm
  1149. 2020-08-25T18:24:07.5860448Z [lighthouse]$ cluster_subm[cluster1]=true
  1150. 2020-08-25T18:24:07.5874737Z [lighthouse]$ cluster_subm[cluster2]=true
  1151. 2020-08-25T18:24:07.5888989Z [lighthouse]$ cluster_subm[cluster3]=true
  1152. 2020-08-25T18:24:07.5900647Z [lighthouse]$ declare -gA cluster_cni
  1153. 2020-08-25T18:24:07.5913192Z [lighthouse]$ cluster_cni[cluster2]=weave
  1154. 2020-08-25T18:24:07.5923892Z [lighthouse]$ cluster_cni[cluster3]=weave
  1155. 2020-08-25T18:24:07.5936782Z [lighthouse]$ source /go/src/github.com/submariner-io/lighthouse/scripts/cluster_settings
  1156. 2020-08-25T18:24:07.5948273Z [lighthouse]$ . /opt/shipyard/scripts/lib/source_only
  1157. 2020-08-25T18:24:07.5960671Z [lighthouse]$ script_name=cluster_settings
  1158. 2020-08-25T18:24:07.5974693Z [lighthouse]$ exec_name=deploy.sh
  1159. 2020-08-25T18:24:07.5995807Z [lighthouse]$ cluster_nodes[cluster1]=control-plane worker worker
  1160. 2020-08-25T18:24:07.6008386Z [lighthouse]$ cluster_nodes[cluster2]=control-plane worker worker
  1161. 2020-08-25T18:24:07.6037330Z [lighthouse]$ declare_cidrs
  1162. 2020-08-25T18:24:07.6047537Z [lighthouse]$ declare_cidrs
  1163. 2020-08-25T18:24:07.6058488Z [lighthouse]$ declare -gA cluster_CIDRs service_CIDRs global_CIDRs
  1164. 2020-08-25T18:24:07.6072421Z [lighthouse]$ i=1
  1165. 2020-08-25T18:24:07.6084086Z [lighthouse]$ [cluster1] add_cluster_cidrs 1 cluster1
  1166. 2020-08-25T18:24:07.6094359Z [lighthouse]$ [cluster1] add_cluster_cidrs 1 cluster1
  1167. 2020-08-25T18:24:07.6105338Z [lighthouse]$ [cluster1] local val=1
  1168. 2020-08-25T18:24:07.6117763Z [lighthouse]$ [cluster1] local idx=cluster1
  1169. 2020-08-25T18:24:07.6132846Z [lighthouse]$ [cluster1] cluster_CIDRs[cluster1]=10.241.0.0/16
  1170. 2020-08-25T18:24:07.6148621Z [lighthouse]$ [cluster1] service_CIDRs[cluster1]=100.91.0.0/16
  1171. 2020-08-25T18:24:07.6162213Z [lighthouse]$ [cluster1] i=2
  1172. 2020-08-25T18:24:07.6180092Z [lighthouse]$ [cluster2] add_cluster_cidrs 2 cluster2
  1173. 2020-08-25T18:24:07.6193072Z [lighthouse]$ [cluster2] add_cluster_cidrs 2 cluster2
  1174. 2020-08-25T18:24:07.6204025Z [lighthouse]$ [cluster2] local val=2
  1175. 2020-08-25T18:24:07.6214537Z [lighthouse]$ [cluster2] local idx=cluster2
  1176. 2020-08-25T18:24:07.6228008Z [lighthouse]$ [cluster2] cluster_CIDRs[cluster2]=10.242.0.0/16
  1177. 2020-08-25T18:24:07.6240083Z [lighthouse]$ [cluster2] service_CIDRs[cluster2]=100.92.0.0/16
  1178. 2020-08-25T18:24:07.6253506Z [lighthouse]$ [cluster2] i=3
  1179. 2020-08-25T18:24:07.6265511Z [lighthouse]$ [cluster2] declare_kubeconfig
  1180. 2020-08-25T18:24:07.6276093Z [lighthouse]$ [cluster2] declare_kubeconfig
  1181. 2020-08-25T18:24:07.6287125Z [lighthouse]$ [cluster2] source /opt/shipyard/scripts/lib/kubecfg
  1182. 2020-08-25T18:24:07.6300487Z [lighthouse]$ [cluster2] export KUBECONFIG
  1183. 2020-08-25T18:24:07.6344115Z [lighthouse]$ [cluster2] KUBECONFIG=/go/src/github.com/submariner-io/lighthouse/output/kubeconfigs/kind-config-cluster2:/go/src/github.com/submariner-io/lighthouse/output/kubeconfigs/kind-config-cluster1:
  1184. 2020-08-25T18:24:07.6362195Z [lighthouse]$ [cluster2] find /go/src/github.com/submariner-io/lighthouse/output/kubeconfigs -type f -printf %p:
  1185. 2020-08-25T18:24:07.6396622Z [lighthouse]$ [cluster2] import_image quay.io/submariner/nettest
  1186. 2020-08-25T18:24:07.6407568Z [lighthouse]$ [cluster2] import_image quay.io/submariner/nettest
  1187. 2020-08-25T18:24:07.6420120Z [lighthouse]$ [cluster2] local orig_image=quay.io/submariner/nettest
  1188. 2020-08-25T18:24:07.6436478Z [lighthouse]$ [cluster2] local versioned_image=quay.io/submariner/nettest:dev
  1189. 2020-08-25T18:24:07.6449491Z [lighthouse]$ [cluster2] local local_image=localhost:5000/nettest:local
  1190. 2020-08-25T18:24:07.6462204Z [lighthouse]$ [cluster2] docker tag quay.io/submariner/nettest:dev localhost:5000/nettest:local
  1191. 2020-08-25T18:24:08.0083913Z Error response from daemon: No such image: quay.io/submariner/nettest:dev
  1192. 2020-08-25T18:24:08.0149277Z [lighthouse]$ [cluster2] docker pull quay.io/submariner/nettest:devel
  1193. 2020-08-25T18:24:09.3245438Z devel: Pulling from submariner/nettest
  1194. 2020-08-25T18:24:09.3245842Z df20fa9351a1: Pulling fs layer
  1195. 2020-08-25T18:24:09.3246319Z 15f35f8557ec: Pulling fs layer
  1196. 2020-08-25T18:24:09.3246428Z aacdef248aba: Pulling fs layer
  1197. 2020-08-25T18:24:09.6979475Z 15f35f8557ec: Verifying Checksum
  1198. 2020-08-25T18:24:09.6979725Z 15f35f8557ec: Download complete
  1199. 2020-08-25T18:24:09.7254432Z df20fa9351a1: Download complete
  1200. 2020-08-25T18:24:10.0049227Z aacdef248aba: Verifying Checksum
  1201. 2020-08-25T18:24:10.0049562Z aacdef248aba: Download complete
  1202. 2020-08-25T18:24:10.1933060Z df20fa9351a1: Pull complete
  1203. 2020-08-25T18:24:10.2646990Z 15f35f8557ec: Pull complete
  1204. 2020-08-25T18:24:10.6352878Z aacdef248aba: Pull complete
  1205. 2020-08-25T18:24:10.6378598Z Digest: sha256:3f8474fd8f3a41eeb84744a75b2e6a1b862dd968001dba5c7c6cc3579a3a716c
  1206. 2020-08-25T18:24:10.6400452Z Status: Downloaded newer image for quay.io/submariner/nettest:devel
  1207. 2020-08-25T18:24:10.6451167Z quay.io/submariner/nettest:devel
  1208. 2020-08-25T18:24:10.6507642Z [lighthouse]$ [cluster2] docker tag quay.io/submariner/nettest:devel quay.io/submariner/nettest:dev
  1209. 2020-08-25T18:24:11.0061621Z [lighthouse]$ [cluster2] docker tag quay.io/submariner/nettest:dev localhost:5000/nettest:local
  1210. 2020-08-25T18:24:11.3737855Z [lighthouse]$ [cluster2] docker push localhost:5000/nettest:local
  1211. 2020-08-25T18:24:11.7336168Z The push refers to repository [localhost:5000/nettest]
  1212. 2020-08-25T18:24:11.7392999Z 23d7b631b983: Preparing
  1213. 2020-08-25T18:24:11.7397285Z e1448740d1c5: Preparing
  1214. 2020-08-25T18:24:11.7397407Z 50644c29ef5a: Preparing
  1215. 2020-08-25T18:24:11.9413603Z e1448740d1c5: Pushed
  1216. 2020-08-25T18:24:12.3234933Z 50644c29ef5a: Pushed
  1217. 2020-08-25T18:24:13.6772995Z 23d7b631b983: Pushed
  1218. 2020-08-25T18:24:13.6996973Z local: digest: sha256:3f8474fd8f3a41eeb84744a75b2e6a1b862dd968001dba5c7c6cc3579a3a716c size: 945
  1219. 2020-08-25T18:24:13.7076669Z [lighthouse]$ [cluster2] import_image quay.io/submariner/submariner
  1220. 2020-08-25T18:24:13.7086248Z [lighthouse]$ [cluster2] import_image quay.io/submariner/submariner
  1221. 2020-08-25T18:24:13.7100602Z [lighthouse]$ [cluster2] local orig_image=quay.io/submariner/submariner
  1222. 2020-08-25T18:24:13.7122598Z [lighthouse]$ [cluster2] local versioned_image=quay.io/submariner/submariner:dev
  1223. 2020-08-25T18:24:13.7133796Z [lighthouse]$ [cluster2] local local_image=localhost:5000/submariner:local
  1224. 2020-08-25T18:24:13.7146633Z [lighthouse]$ [cluster2] docker tag quay.io/submariner/submariner:dev localhost:5000/submariner:local
  1225. 2020-08-25T18:24:14.0762674Z Error response from daemon: No such image: quay.io/submariner/submariner:dev
  1226. 2020-08-25T18:24:14.0817283Z [lighthouse]$ [cluster2] docker pull quay.io/submariner/submariner:devel
  1227. 2020-08-25T18:24:15.3898547Z devel: Pulling from submariner/submariner
  1228. 2020-08-25T18:24:15.3905436Z c7def56d621e: Already exists
  1229. 2020-08-25T18:24:15.3940744Z 8ba88a5a4038: Pulling fs layer
  1230. 2020-08-25T18:24:15.3945481Z 14f5ab309c25: Pulling fs layer
  1231. 2020-08-25T18:24:15.3945773Z 6ca1d86fc328: Pulling fs layer
  1232. 2020-08-25T18:24:15.3946421Z f42626247d58: Pulling fs layer
  1233. 2020-08-25T18:24:15.3947110Z f42626247d58: Waiting
  1234. 2020-08-25T18:24:15.7365339Z 8ba88a5a4038: Verifying Checksum
  1235. 2020-08-25T18:24:15.7365835Z 8ba88a5a4038: Download complete
  1236. 2020-08-25T18:24:15.8150515Z 8ba88a5a4038: Pull complete
  1237. 2020-08-25T18:24:15.9795639Z 6ca1d86fc328: Verifying Checksum
  1238. 2020-08-25T18:24:15.9796328Z 6ca1d86fc328: Download complete
  1239. 2020-08-25T18:24:16.0974279Z f42626247d58: Verifying Checksum
  1240. 2020-08-25T18:24:16.0981841Z f42626247d58: Download complete
  1241. 2020-08-25T18:24:16.1895539Z 14f5ab309c25: Verifying Checksum
  1242. 2020-08-25T18:24:16.1895777Z 14f5ab309c25: Download complete
  1243. 2020-08-25T18:24:17.2671930Z 14f5ab309c25: Pull complete
  1244. 2020-08-25T18:24:17.4682749Z 6ca1d86fc328: Pull complete
  1245. 2020-08-25T18:24:17.5517622Z f42626247d58: Pull complete
  1246. 2020-08-25T18:24:17.5552449Z Digest: sha256:caae9d1efcb18471c66725ea9a3f2720722f6cab5787fee98abbd242c44b7629
  1247. 2020-08-25T18:24:17.5586651Z Status: Downloaded newer image for quay.io/submariner/submariner:devel
  1248. 2020-08-25T18:24:17.5635240Z quay.io/submariner/submariner:devel
  1249. 2020-08-25T18:24:17.5662118Z [lighthouse]$ [cluster2] docker tag quay.io/submariner/submariner:devel quay.io/submariner/submariner:dev
  1250. 2020-08-25T18:24:17.9285908Z [lighthouse]$ [cluster2] docker tag quay.io/submariner/submariner:dev localhost:5000/submariner:local
  1251. 2020-08-25T18:24:18.3028063Z [lighthouse]$ [cluster2] docker push localhost:5000/submariner:local
  1252. 2020-08-25T18:24:18.6414413Z The push refers to repository [localhost:5000/submariner]
  1253. 2020-08-25T18:24:18.6490322Z c9622cc8aa6b: Preparing
  1254. 2020-08-25T18:24:18.6490921Z 58fca44c284b: Preparing
  1255. 2020-08-25T18:24:18.6491103Z 114df4f4b6b7: Preparing
  1256. 2020-08-25T18:24:18.6491222Z c4d24733c9c3: Preparing
  1257. 2020-08-25T18:24:18.6491337Z a344333fd60e: Preparing
  1258. 2020-08-25T18:24:18.7898183Z c4d24733c9c3: Pushed
  1259. 2020-08-25T18:24:19.5524513Z 58fca44c284b: Pushed
  1260. 2020-08-25T18:24:19.6780043Z c9622cc8aa6b: Pushed
  1261. 2020-08-25T18:24:23.4279533Z 114df4f4b6b7: Pushed
  1262. 2020-08-25T18:24:35.4747565Z a344333fd60e: Pushed
  1263. 2020-08-25T18:24:35.5042294Z local: digest: sha256:caae9d1efcb18471c66725ea9a3f2720722f6cab5787fee98abbd242c44b7629 size: 1366
  1264. 2020-08-25T18:24:35.5085636Z [lighthouse]$ [cluster2] import_image quay.io/submariner/submariner-route-agent
  1265. 2020-08-25T18:24:35.5096723Z [lighthouse]$ [cluster2] import_image quay.io/submariner/submariner-route-agent
  1266. 2020-08-25T18:24:35.5110693Z [lighthouse]$ [cluster2] local orig_image=quay.io/submariner/submariner-route-agent
  1267. 2020-08-25T18:24:35.5123898Z [lighthouse]$ [cluster2] local versioned_image=quay.io/submariner/submariner-route-agent:dev
  1268. 2020-08-25T18:24:35.5137730Z [lighthouse]$ [cluster2] local local_image=localhost:5000/submariner-route-agent:local
  1269. 2020-08-25T18:24:35.5153625Z [lighthouse]$ [cluster2] docker tag quay.io/submariner/submariner-route-agent:dev localhost:5000/submariner-route-agent:local
  1270. 2020-08-25T18:24:35.8629962Z Error response from daemon: No such image: quay.io/submariner/submariner-route-agent:dev
  1271. 2020-08-25T18:24:35.8658672Z [lighthouse]$ [cluster2] docker pull quay.io/submariner/submariner-route-agent:devel
  1272. 2020-08-25T18:24:37.1847528Z devel: Pulling from submariner/submariner-route-agent
  1273. 2020-08-25T18:24:37.1847992Z 41ae95b593e0: Already exists
  1274. 2020-08-25T18:24:37.1861613Z f20f68829d13: Already exists
  1275. 2020-08-25T18:24:37.1894599Z 6a29c8bc0278: Pulling fs layer
  1276. 2020-08-25T18:24:37.1894883Z 53a8c0a0de37: Pulling fs layer
  1277. 2020-08-25T18:24:37.1894991Z 46a0fa5b2dc6: Pulling fs layer
  1278. 2020-08-25T18:24:37.1896058Z f5cd48b669c0: Pulling fs layer
  1279. 2020-08-25T18:24:37.1896231Z f5cd48b669c0: Waiting
  1280. 2020-08-25T18:24:37.5447735Z 6a29c8bc0278: Download complete
  1281. 2020-08-25T18:24:37.6606394Z 6a29c8bc0278: Pull complete
  1282. 2020-08-25T18:24:37.6709987Z 53a8c0a0de37: Verifying Checksum
  1283. 2020-08-25T18:24:37.6710551Z 53a8c0a0de37: Download complete
  1284. 2020-08-25T18:24:37.6710711Z 46a0fa5b2dc6: Verifying Checksum
  1285. 2020-08-25T18:24:37.6710899Z 46a0fa5b2dc6: Download complete
  1286. 2020-08-25T18:24:37.9084151Z f5cd48b669c0: Verifying Checksum
  1287. 2020-08-25T18:24:37.9084804Z f5cd48b669c0: Download complete
  1288. 2020-08-25T18:24:37.9676144Z 53a8c0a0de37: Pull complete
  1289. 2020-08-25T18:24:38.1676540Z 46a0fa5b2dc6: Pull complete
  1290. 2020-08-25T18:24:38.2492891Z f5cd48b669c0: Pull complete
  1291. 2020-08-25T18:24:38.2503724Z Digest: sha256:9b3e190e9c6894ac01318c04ebf096a1188bc4f0eec09792d591e5cee2310ed5
  1292. 2020-08-25T18:24:38.2532065Z Status: Downloaded newer image for quay.io/submariner/submariner-route-agent:devel
  1293. 2020-08-25T18:24:38.2532444Z quay.io/submariner/submariner-route-agent:devel
  1294. 2020-08-25T18:24:38.2561241Z [lighthouse]$ [cluster2] docker tag quay.io/submariner/submariner-route-agent:devel quay.io/submariner/submariner-route-agent:dev
  1295. 2020-08-25T18:24:38.6150782Z [lighthouse]$ [cluster2] docker tag quay.io/submariner/submariner-route-agent:dev localhost:5000/submariner-route-agent:local
  1296. 2020-08-25T18:24:38.9612372Z [lighthouse]$ [cluster2] docker push localhost:5000/submariner-route-agent:local
  1297. 2020-08-25T18:24:39.2963580Z The push refers to repository [localhost:5000/submariner-route-agent]
  1298. 2020-08-25T18:24:39.3048265Z 7578f159a1e4: Preparing
  1299. 2020-08-25T18:24:39.3048521Z 068fd7a93cab: Preparing
  1300. 2020-08-25T18:24:39.3048686Z f54eebdb881a: Preparing
  1301. 2020-08-25T18:24:39.3048844Z 3160418cbe20: Preparing
  1302. 2020-08-25T18:24:39.3048943Z c7e9c12dd629: Preparing
  1303. 2020-08-25T18:24:39.3049196Z 29bad1020e6f: Preparing
  1304. 2020-08-25T18:24:39.3116606Z 29bad1020e6f: Waiting
  1305. 2020-08-25T18:24:39.5390510Z c7e9c12dd629: Mounted from lighthouse-agent
  1306. 2020-08-25T18:24:39.9260192Z 3160418cbe20: Pushed
  1307. 2020-08-25T18:24:40.1185917Z 7578f159a1e4: Pushed
  1308. 2020-08-25T18:24:40.1186273Z 068fd7a93cab: Pushed
  1309. 2020-08-25T18:24:40.1187096Z 29bad1020e6f: Mounted from lighthouse-agent
  1310. 2020-08-25T18:24:40.3028691Z f54eebdb881a: Pushed
  1311. 2020-08-25T18:24:40.3240767Z local: digest: sha256:9b3e190e9c6894ac01318c04ebf096a1188bc4f0eec09792d591e5cee2310ed5 size: 1573
  1312. 2020-08-25T18:24:40.3292672Z [lighthouse]$ [cluster2] load_deploytool operator
  1313. 2020-08-25T18:24:40.3306474Z [lighthouse]$ [cluster2] load_deploytool operator
  1314. 2020-08-25T18:24:40.3319910Z [lighthouse]$ [cluster2] local deploytool=operator
  1315. 2020-08-25T18:24:40.3332953Z [lighthouse]$ [cluster2] local deploy_lib=/opt/shipyard/scripts/lib/deploy_operator
  1316. 2020-08-25T18:24:40.3334795Z Will deploy submariner using operator
  1317. 2020-08-25T18:24:40.3348676Z [lighthouse]$ [cluster2] . /opt/shipyard/scripts/lib/deploy_operator
  1318. 2020-08-25T18:24:40.3361739Z [lighthouse]$ [cluster2] . /opt/shipyard/scripts/lib/source_only
  1319. 2020-08-25T18:24:40.3375407Z [lighthouse]$ [cluster2] script_name=deploy_operator
  1320. 2020-08-25T18:24:40.3391695Z [lighthouse]$ [cluster2] exec_name=deploy.sh
  1321. 2020-08-25T18:24:40.3409328Z [lighthouse]$ [cluster2] deploytool_prereqs
  1322. 2020-08-25T18:24:40.3420807Z [lighthouse]$ [cluster2] deploytool_prereqs
  1323. 2020-08-25T18:24:40.3434837Z [lighthouse]$ [cluster2] test -x /go/bin/subctl
  1324. 2020-08-25T18:24:40.3451626Z [lighthouse]$ [cluster2] /go/bin/subctl version
  1325. 2020-08-25T18:24:40.5784089Z subctl version: v0.6.0-rc0
  1326. 2020-08-25T18:24:40.5805021Z [lighthouse]$ [cluster2] import_image quay.io/submariner/submariner-operator
  1327. 2020-08-25T18:24:40.5825974Z [lighthouse]$ [cluster2] import_image quay.io/submariner/submariner-operator
  1328. 2020-08-25T18:24:40.5838933Z [lighthouse]$ [cluster2] local orig_image=quay.io/submariner/submariner-operator
  1329. 2020-08-25T18:24:40.5852939Z [lighthouse]$ [cluster2] local versioned_image=quay.io/submariner/submariner-operator:dev
  1330. 2020-08-25T18:24:40.5866966Z [lighthouse]$ [cluster2] local local_image=localhost:5000/submariner-operator:local
  1331. 2020-08-25T18:24:40.5880251Z [lighthouse]$ [cluster2] docker tag quay.io/submariner/submariner-operator:dev localhost:5000/submariner-operator:local
  1332. 2020-08-25T18:24:40.9262707Z Error response from daemon: No such image: quay.io/submariner/submariner-operator:dev
  1333. 2020-08-25T18:24:40.9289723Z [lighthouse]$ [cluster2] docker pull quay.io/submariner/submariner-operator:devel
  1334. 2020-08-25T18:24:42.2302091Z devel: Pulling from submariner/submariner-operator
  1335. 2020-08-25T18:24:42.2302612Z 48d41cd04709: Pulling fs layer
  1336. 2020-08-25T18:24:43.0637431Z 48d41cd04709: Verifying Checksum
  1337. 2020-08-25T18:24:43.0637863Z 48d41cd04709: Download complete
  1338. 2020-08-25T18:24:43.6111343Z 48d41cd04709: Pull complete
  1339. 2020-08-25T18:24:43.6134270Z Digest: sha256:86f2c7be9e71c3c99385895cfa9affd318ea0d690387e9d9d00b3e6f57fa924c
  1340. 2020-08-25T18:24:43.6158680Z Status: Downloaded newer image for quay.io/submariner/submariner-operator:devel
  1341. 2020-08-25T18:24:43.6314054Z [lighthouse]$ [cluster2] docker tag quay.io/submariner/submariner-operator:devel quay.io/submariner/submariner-operator:dev
  1342. 2020-08-25T18:24:43.6314525Z quay.io/submariner/submariner-operator:devel
  1343. 2020-08-25T18:24:43.9542456Z [lighthouse]$ [cluster2] docker tag quay.io/submariner/submariner-operator:dev localhost:5000/submariner-operator:local
  1344. 2020-08-25T18:24:44.2944562Z [lighthouse]$ [cluster2] docker push localhost:5000/submariner-operator:local
  1345. 2020-08-25T18:24:44.6420109Z The push refers to repository [localhost:5000/submariner-operator]
  1346. 2020-08-25T18:24:44.6490702Z 8c35e4065361: Preparing
  1347. 2020-08-25T18:24:47.4883202Z 8c35e4065361: Pushed
  1348. 2020-08-25T18:24:47.5084088Z local: digest: sha256:86f2c7be9e71c3c99385895cfa9affd318ea0d690387e9d9d00b3e6f57fa924c size: 529
  1349. 2020-08-25T18:24:47.5122643Z [lighthouse]$ [cluster2] run_subm_clusters prepare_cluster submariner-operator
  1350. 2020-08-25T18:24:47.5136903Z [lighthouse]$ [cluster2] run_subm_clusters prepare_cluster submariner-operator
  1351. 2020-08-25T18:24:47.5147576Z [lighthouse]$ [cluster2] declare -a subm_clusters
  1352. 2020-08-25T18:24:47.5185889Z [lighthouse]$ [cluster2] run_parallel cluster1 cluster2 prepare_cluster submariner-operator
  1353. 2020-08-25T18:24:47.5198089Z [lighthouse]$ [cluster2] run_parallel cluster1 cluster2 cluster1 cluster2 prepare_cluster submariner-operator
  1354. 2020-08-25T18:24:47.5211035Z [lighthouse]$ [cluster2] local cmnd=prepare_cluster
  1355. 2020-08-25T18:24:47.5224166Z [lighthouse]$ [cluster2] declare -A pids
  1356. 2020-08-25T18:24:47.5268265Z [lighthouse]$ [cluster2] eval echo cluster1 cluster2
  1357. 2020-08-25T18:24:47.5304224Z [lighthouse]$ [cluster1] set -o pipefail
  1358. 2020-08-25T18:24:47.5309115Z [lighthouse]$ [cluster1] pids[cluster1]=5103
  1359. 2020-08-25T18:24:47.5320971Z [lighthouse]$ [cluster1] prepare_cluster submariner-operator
  1360. 2020-08-25T18:24:47.5325751Z [lighthouse]$ [cluster2] pids[cluster2]=5107
  1361. 2020-08-25T18:24:47.5344371Z [lighthouse]$ [cluster1] sed /\[cluster1]/!s/^/[cluster1] /
  1362. 2020-08-25T18:24:47.5353515Z [lighthouse]$ [cluster2] wait 5107
  1363. 2020-08-25T18:24:47.5359450Z [lighthouse]$ [cluster2] set -o pipefail
  1364. 2020-08-25T18:24:47.5380677Z [lighthouse]$ [cluster2] prepare_cluster submariner-operator
  1365. 2020-08-25T18:24:47.5408483Z [lighthouse]$ [cluster2] sed /\[cluster2]/!s/^/[cluster2] /
  1366. 2020-08-25T18:24:51.3292619Z [lighthouse]$ [cluster2] prepare_cluster
  1367. 2020-08-25T18:24:51.3293490Z [lighthouse]$ [cluster2] local namespace=submariner-operator
  1368. 2020-08-25T18:24:51.3294003Z [cluster2] 
  1369. 2020-08-25T18:24:51.3294596Z [cluster2] 
  1370. 2020-08-25T18:24:51.3295098Z [cluster2] 
  1371. 2020-08-25T18:24:51.3295561Z [lighthouse]$ [cluster2] add_subm_gateway_label
  1372. 2020-08-25T18:24:51.3297722Z [lighthouse]$ [cluster2] add_subm_gateway_label
  1373. 2020-08-25T18:24:51.3298804Z [lighthouse]$ [cluster2] kubectl label node cluster2-worker submariner.io/gateway=true --overwrite
  1374. 2020-08-25T18:24:51.3301756Z [lighthouse]$ [cluster2] kubectl label node cluster2-worker submariner.io/gateway=true --overwrite
  1375. 2020-08-25T18:24:51.3303406Z [lighthouse]$ [cluster2] command kubectl --context=cluster2 label node cluster2-worker submariner.io/gateway=true --overwrite
  1376. 2020-08-25T18:24:51.3304105Z [lighthouse]$ [cluster2] kubectl --context=cluster2 label node cluster2-worker submariner.io/gateway=true --overwrite
  1377. 2020-08-25T18:24:51.3304629Z [cluster2] node/cluster2-worker labeled
  1378. 2020-08-25T18:24:51.3327491Z [lighthouse]$ [cluster2] wait 5103
  1379. 2020-08-25T18:24:51.3923271Z [lighthouse]$ [cluster1] prepare_cluster
  1380. 2020-08-25T18:24:51.3923697Z [lighthouse]$ [cluster1] local namespace=submariner-operator
  1381. 2020-08-25T18:24:51.3924021Z [cluster1] 
  1382. 2020-08-25T18:24:51.3924322Z [cluster1] 
  1383. 2020-08-25T18:24:51.3924559Z [cluster1] 
  1384. 2020-08-25T18:24:51.3925086Z [lighthouse]$ [cluster1] add_subm_gateway_label
  1385. 2020-08-25T18:24:51.3925525Z [lighthouse]$ [cluster1] add_subm_gateway_label
  1386. 2020-08-25T18:24:51.3926088Z [lighthouse]$ [cluster1] kubectl label node cluster1-worker submariner.io/gateway=true --overwrite
  1387. 2020-08-25T18:24:51.3926546Z [lighthouse]$ [cluster1] kubectl label node cluster1-worker submariner.io/gateway=true --overwrite
  1388. 2020-08-25T18:24:51.3927008Z [lighthouse]$ [cluster1] command kubectl --context=cluster1 label node cluster1-worker submariner.io/gateway=true --overwrite
  1389. 2020-08-25T18:24:51.3927452Z [lighthouse]$ [cluster1] kubectl --context=cluster1 label node cluster1-worker submariner.io/gateway=true --overwrite
  1390. 2020-08-25T18:24:51.3927803Z [cluster1] node/cluster1-worker labeled
  1391. 2020-08-25T18:24:51.3950540Z [lighthouse]$ [cluster2] with_context cluster1 setup_broker
  1392. 2020-08-25T18:24:51.3965695Z [lighthouse]$ [cluster2] with_context cluster1 setup_broker
  1393. 2020-08-25T18:24:51.3978172Z [lighthouse]$ [cluster2] local cluster=cluster1
  1394. 2020-08-25T18:24:51.3994045Z [lighthouse]$ [cluster1] local cmnd=setup_broker
  1395. 2020-08-25T18:24:51.4022931Z [lighthouse]$ [cluster1] setup_broker
  1396. 2020-08-25T18:24:51.4037467Z [lighthouse]$ [cluster1] setup_broker
  1397. 2020-08-25T18:24:51.4054947Z [lighthouse]$ [cluster1] local gn
  1398. 2020-08-25T18:24:51.4062258Z Installing broker...
  1399. 2020-08-25T18:24:51.4082116Z [lighthouse]$ [cluster1] cd /go/src/github.com/submariner-io/lighthouse/output
  1400. 2020-08-25T18:24:51.4097887Z [output]$ [cluster1] subctl deploy-broker --kubeconfig /go/src/github.com/submariner-io/lighthouse/output/kubeconfigs/kind-config-cluster1 --service-discovery
  1401. 2020-08-25T18:24:51.6292104Z • Deploying broker ...
  1402. 2020-08-25T18:24:51.8410496Z ✓ Deploying broker
  1403. 2020-08-25T18:24:51.8411003Z • Creating broker-info.subm file ...
  1404. 2020-08-25T18:24:51.8485871Z ✓ Creating broker-info.subm file
  1405. 2020-08-25T18:24:51.8486239Z ✓ A new IPsec PSK will be generated for broker-info.subm
  1406. 2020-08-25T18:24:51.8513768Z [lighthouse]$ [cluster2] install_subm_all_clusters
  1407. 2020-08-25T18:24:51.8531741Z [lighthouse]$ [cluster2] install_subm_all_clusters
  1408. 2020-08-25T18:24:51.8548499Z [lighthouse]$ [cluster2] run_subm_clusters subctl_install_subm
  1409. 2020-08-25T18:24:51.8559630Z [lighthouse]$ [cluster2] run_subm_clusters subctl_install_subm
  1410. 2020-08-25T18:24:51.8569610Z [lighthouse]$ [cluster2] declare -a subm_clusters
  1411. 2020-08-25T18:24:51.8601006Z [lighthouse]$ [cluster2] run_parallel cluster1 cluster2 subctl_install_subm
  1412. 2020-08-25T18:24:51.8613087Z [lighthouse]$ [cluster2] run_parallel cluster1 cluster2 cluster1 cluster2 subctl_install_subm
  1413. 2020-08-25T18:24:51.8624502Z [lighthouse]$ [cluster2] local cmnd=subctl_install_subm
  1414. 2020-08-25T18:24:51.8636169Z [lighthouse]$ [cluster2] declare -A pids
  1415. 2020-08-25T18:24:51.8652595Z [lighthouse]$ [cluster2] eval echo cluster1 cluster2
  1416. 2020-08-25T18:24:51.8670730Z [lighthouse]$ [cluster1] pids[cluster1]=5239
  1417. 2020-08-25T18:24:51.8699034Z [lighthouse]$ [cluster2] pids[cluster2]=5241
  1418. 2020-08-25T18:24:51.8716171Z [lighthouse]$ [cluster1] set -o pipefail
  1419. 2020-08-25T18:24:51.8740027Z [lighthouse]$ [cluster2] wait 5241
  1420. 2020-08-25T18:24:51.8749540Z [lighthouse]$ [cluster1] subctl_install_subm
  1421. 2020-08-25T18:24:51.8758784Z [lighthouse]$ [cluster2] set -o pipefail
  1422. 2020-08-25T18:24:51.8768652Z [lighthouse]$ [cluster1] sed /\[cluster1]/!s/^/[cluster1] /
  1423. 2020-08-25T18:24:51.8785081Z [lighthouse]$ [cluster2] subctl_install_subm
  1424. 2020-08-25T18:24:51.8833098Z [lighthouse]$ [cluster2] sed /\[cluster2]/!s/^/[cluster2] /
  1425. 2020-08-25T18:24:58.9159763Z [lighthouse]$ [cluster2] subctl_install_subm
  1426. 2020-08-25T18:24:58.9160468Z [lighthouse]$ [cluster2] subctl join --kubeconfig /go/src/github.com/submariner-io/lighthouse/output/kubeconfigs/kind-config-cluster2 --clusterid cluster2 --repository localhost:5000 --version local --nattport 4500 --ikeport 500 --colorcodes blue --globalnet-cidr --disable-nat /go/src/github.com/submariner-io/lighthouse/output/broker-info.subm
  1427. 2020-08-25T18:24:58.9161316Z [cluster2] * /go/src/github.com/submariner-io/lighthouse/output/broker-info.subm says broker is at: https://172.17.0.4:6443
  1428. 2020-08-25T18:24:58.9161550Z [cluster2] * There are 1 labeled nodes in the cluster:
  1429. 2020-08-25T18:24:58.9161838Z [cluster2] - cluster2-worker
  1430. 2020-08-25T18:24:58.9162243Z [cluster2] • Discovering network details ...
  1431. 2020-08-25T18:24:58.9162454Z [cluster2] Discovered network details:
  1432. 2020-08-25T18:24:58.9162783Z [cluster2] Network plugin: weave-net
  1433. 2020-08-25T18:24:58.9162965Z [cluster2] Service CIDRs: [100.92.0.0/16]
  1434. 2020-08-25T18:24:58.9163133Z [cluster2] Cluster CIDRs: [10.242.0.0/16]
  1435. 2020-08-25T18:24:58.9163471Z [cluster2] ✓ Discovering network details
  1436. 2020-08-25T18:24:58.9163842Z [cluster2] • Discovering multi cluster details ...
  1437. 2020-08-25T18:24:58.9164137Z [cluster2] • Validating Globalnet configurations ...
  1438. 2020-08-25T18:24:58.9164590Z [cluster2] ✓ Validating Globalnet configurations
  1439. 2020-08-25T18:24:58.9165280Z [cluster2] ✓ Discovering multi cluster details
  1440. 2020-08-25T18:24:58.9165973Z [cluster2] • Deploying the Submariner operator ...
  1441. 2020-08-25T18:24:58.9166699Z [cluster2] ✓ Deploying the Submariner operator
  1442. 2020-08-25T18:24:58.9167080Z [cluster2] ✓ Created operator CRDs
  1443. 2020-08-25T18:24:58.9167433Z [cluster2] ✓ Created operator namespace: submariner-operator
  1444. 2020-08-25T18:24:58.9167862Z [cluster2] ✓ Created operator service account and role
  1445. 2020-08-25T18:24:58.9168126Z [cluster2] ✓ Created lighthouse CRDs
  1446. 2020-08-25T18:24:58.9168525Z [cluster2] ✓ Created lighthouse service account and role
  1447. 2020-08-25T18:24:58.9168878Z [cluster2] ✓ Created Lighthouse service accounts and roles
  1448. 2020-08-25T18:24:58.9169236Z [cluster2] ✓ Deployed the operator successfully
  1449. 2020-08-25T18:24:58.9169687Z [cluster2] • Creating SA for cluster ...
  1450. 2020-08-25T18:24:58.9170109Z [cluster2] ✓ Creating SA for cluster
  1451. 2020-08-25T18:24:58.9170475Z [cluster2] • Deploying Submariner ...
  1452. 2020-08-25T18:24:58.9170790Z [cluster2] ✓ Deploying Submariner
  1453. 2020-08-25T18:24:58.9171055Z [cluster2] ✓ Submariner is up and running
  1454. 2020-08-25T18:24:58.9198565Z [lighthouse]$ [cluster2] wait 5239
  1455. 2020-08-25T18:24:59.3251175Z [lighthouse]$ [cluster1] subctl_install_subm
  1456. 2020-08-25T18:24:59.3252266Z [lighthouse]$ [cluster1] subctl join --kubeconfig /go/src/github.com/submariner-io/lighthouse/output/kubeconfigs/kind-config-cluster1 --clusterid cluster1 --repository localhost:5000 --version local --nattport 4500 --ikeport 500 --colorcodes blue --globalnet-cidr --disable-nat /go/src/github.com/submariner-io/lighthouse/output/broker-info.subm
  1457. 2020-08-25T18:24:59.3252758Z [cluster1] * /go/src/github.com/submariner-io/lighthouse/output/broker-info.subm says broker is at: https://172.17.0.4:6443
  1458. 2020-08-25T18:24:59.3253023Z [cluster1] * There are 1 labeled nodes in the cluster:
  1459. 2020-08-25T18:24:59.3253689Z [cluster1] - cluster1-worker
  1460. 2020-08-25T18:24:59.3253986Z [cluster1] • Discovering network details ...
  1461. 2020-08-25T18:24:59.3254091Z [cluster1] Discovered network details:
  1462. 2020-08-25T18:24:59.3254374Z [cluster1] Network plugin: weave-net
  1463. 2020-08-25T18:24:59.3254561Z [cluster1] Service CIDRs: [100.91.0.0/16]
  1464. 2020-08-25T18:24:59.3254710Z [cluster1] Cluster CIDRs: [10.241.0.0/16]
  1465. 2020-08-25T18:24:59.3255173Z [cluster1] ✓ Discovering network details
  1466. 2020-08-25T18:24:59.3256780Z [cluster1] • Discovering multi cluster details ...
  1467. 2020-08-25T18:24:59.3257551Z [cluster1] • Validating Globalnet configurations ...
  1468. 2020-08-25T18:24:59.3257902Z [cluster1] ✓ Validating Globalnet configurations
  1469. 2020-08-25T18:24:59.3258328Z [cluster1] ✓ Discovering multi cluster details
  1470. 2020-08-25T18:24:59.3258651Z [cluster1] • Deploying the Submariner operator ...
  1471. 2020-08-25T18:24:59.3258973Z [cluster1] ✓ Deploying the Submariner operator
  1472. 2020-08-25T18:24:59.3259274Z [cluster1] ✓ Created operator CRDs
  1473. 2020-08-25T18:24:59.3259665Z [cluster1] ✓ Created operator namespace: submariner-operator
  1474. 2020-08-25T18:24:59.3260173Z [cluster1] ✓ Created operator service account and role
  1475. 2020-08-25T18:24:59.3260875Z [cluster1] ✓ Created lighthouse CRDs
  1476. 2020-08-25T18:24:59.3261759Z [cluster1] ✓ Created lighthouse service account and role
  1477. 2020-08-25T18:24:59.3263390Z [cluster1] ✓ Created Lighthouse service accounts and roles
  1478. 2020-08-25T18:24:59.3264361Z [cluster1] ✓ Deployed the operator successfully
  1479. 2020-08-25T18:24:59.3266238Z [cluster1] • Creating SA for cluster ...
  1480. 2020-08-25T18:24:59.3266648Z [cluster1] ✓ Creating SA for cluster
  1481. 2020-08-25T18:24:59.3266977Z [cluster1] • Deploying Submariner ...
  1482. 2020-08-25T18:24:59.3267282Z [cluster1] ✓ Deploying Submariner
  1483. 2020-08-25T18:24:59.3267916Z [cluster1] ✓ Submariner is up and running
  1484. 2020-08-25T18:24:59.3306925Z [lighthouse]$ [cluster2] with_context cluster2 connectivity_tests cluster1
  1485. 2020-08-25T18:24:59.3459158Z [lighthouse]$ [cluster2] with_context cluster2 connectivity_tests cluster1
  1486. 2020-08-25T18:24:59.3466453Z [lighthouse]$ [cluster2] local cluster=cluster2
  1487. 2020-08-25T18:24:59.3467131Z [lighthouse]$ [cluster2] local cmnd=connectivity_tests
  1488. 2020-08-25T18:24:59.3468070Z [lighthouse]$ [cluster2] connectivity_tests cluster1
  1489. 2020-08-25T18:24:59.3470342Z [lighthouse]$ [cluster2] connectivity_tests
  1490. 2020-08-25T18:24:59.3524502Z [lighthouse]$ [cluster2] target_cluster=cluster1
  1491. 2020-08-25T18:24:59.3524870Z [lighthouse]$ [cluster2] deploy_resource /opt/shipyard/scripts/resources/netshoot.yaml
  1492. 2020-08-25T18:24:59.3525442Z [lighthouse]$ [cluster2] deploy_resource /opt/shipyard/scripts/resources/netshoot.yaml
  1493. 2020-08-25T18:24:59.3525833Z [lighthouse]$ [cluster2] local resource_file=/opt/shipyard/scripts/resources/netshoot.yaml
  1494. 2020-08-25T18:24:59.3561910Z [lighthouse]$ [cluster2] local resource_name
  1495. 2020-08-25T18:24:59.3643288Z [lighthouse]$ [cluster2] resource_name=netshoot
  1496. 2020-08-25T18:24:59.3658896Z [lighthouse]$ [cluster2] basename /opt/shipyard/scripts/resources/netshoot.yaml .yaml
  1497. 2020-08-25T18:24:59.3693450Z [lighthouse]$ [cluster2] kubectl apply -f /opt/shipyard/scripts/resources/netshoot.yaml
  1498. 2020-08-25T18:24:59.3731801Z [lighthouse]$ [cluster2] kubectl apply -f /opt/shipyard/scripts/resources/netshoot.yaml
  1499. 2020-08-25T18:24:59.3834820Z [lighthouse]$ [cluster2] command kubectl --context=cluster2 apply -f /opt/shipyard/scripts/resources/netshoot.yaml
  1500. 2020-08-25T18:24:59.3837739Z [lighthouse]$ [cluster2] kubectl --context=cluster2 apply -f /opt/shipyard/scripts/resources/netshoot.yaml
  1501. 2020-08-25T18:25:00.8798067Z deployment.apps/netshoot created
  1502. 2020-08-25T18:25:00.8900036Z Waiting for netshoot pods to be ready.
  1503. 2020-08-25T18:25:00.8915258Z [lighthouse]$ [cluster2] kubectl rollout status deploy/netshoot --timeout=5m
  1504. 2020-08-25T18:25:00.8935751Z [lighthouse]$ [cluster2] kubectl rollout status deploy/netshoot --timeout=5m
  1505. 2020-08-25T18:25:00.8959796Z [lighthouse]$ [cluster2] command kubectl --context=cluster2 rollout status deploy/netshoot --timeout=5m
  1506. 2020-08-25T18:25:00.8980257Z [lighthouse]$ [cluster2] kubectl --context=cluster2 rollout status deploy/netshoot --timeout=5m
  1507. 2020-08-25T18:25:02.7798543Z Waiting for deployment "netshoot" rollout to finish: 0 of 1 updated replicas are available...
  1508. 2020-08-25T18:25:34.2366857Z deployment "netshoot" successfully rolled out
  1509. 2020-08-25T18:25:34.2638267Z [lighthouse]$ [cluster2] with_context cluster1 deploy_resource /opt/shipyard/scripts/resources/nginx-demo.yaml
  1510. 2020-08-25T18:25:34.2703157Z [lighthouse]$ [cluster2] with_context cluster1 deploy_resource /opt/shipyard/scripts/resources/nginx-demo.yaml
  1511. 2020-08-25T18:25:34.2731651Z [lighthouse]$ [cluster2] local cluster=cluster1
  1512. 2020-08-25T18:25:34.2785659Z [lighthouse]$ [cluster1] local cmnd=deploy_resource
  1513. 2020-08-25T18:25:34.2835212Z [lighthouse]$ [cluster1] deploy_resource /opt/shipyard/scripts/resources/nginx-demo.yaml
  1514. 2020-08-25T18:25:34.2845402Z [lighthouse]$ [cluster1] deploy_resource
  1515. 2020-08-25T18:25:34.2889435Z [lighthouse]$ [cluster1] local resource_file=/opt/shipyard/scripts/resources/nginx-demo.yaml
  1516. 2020-08-25T18:25:34.2900965Z [lighthouse]$ [cluster1] local resource_name
  1517. 2020-08-25T18:25:34.3038020Z [lighthouse]$ [cluster1] resource_name=nginx-demo
  1518. 2020-08-25T18:25:34.3081226Z [lighthouse]$ [cluster1] basename /opt/shipyard/scripts/resources/nginx-demo.yaml .yaml
  1519. 2020-08-25T18:25:34.3144317Z [lighthouse]$ [cluster1] kubectl apply -f /opt/shipyard/scripts/resources/nginx-demo.yaml
  1520. 2020-08-25T18:25:34.3167834Z [lighthouse]$ [cluster1] kubectl apply -f /opt/shipyard/scripts/resources/nginx-demo.yaml
  1521. 2020-08-25T18:25:34.3184622Z [lighthouse]$ [cluster1] command kubectl --context=cluster1 apply -f /opt/shipyard/scripts/resources/nginx-demo.yaml
  1522. 2020-08-25T18:25:34.3200287Z [lighthouse]$ [cluster1] kubectl --context=cluster1 apply -f /opt/shipyard/scripts/resources/nginx-demo.yaml
  1523. 2020-08-25T18:25:38.0726596Z deployment.apps/nginx-demo created
  1524. 2020-08-25T18:25:38.2308253Z service/nginx-demo created
  1525. 2020-08-25T18:25:38.2514257Z Waiting for nginx-demo pods to be ready.
  1526. 2020-08-25T18:25:38.2531087Z [lighthouse]$ [cluster1] kubectl rollout status deploy/nginx-demo --timeout=5m
  1527. 2020-08-25T18:25:38.2560581Z [lighthouse]$ [cluster1] kubectl rollout status deploy/nginx-demo --timeout=5m
  1528. 2020-08-25T18:25:38.2610571Z [lighthouse]$ [cluster1] command kubectl --context=cluster1 rollout status deploy/nginx-demo --timeout=5m
  1529. 2020-08-25T18:25:38.2629258Z [lighthouse]$ [cluster1] kubectl --context=cluster1 rollout status deploy/nginx-demo --timeout=5m
  1530. 2020-08-25T18:25:40.9532371Z Waiting for deployment "nginx-demo" rollout to finish: 0 of 2 updated replicas are available...
  1531. 2020-08-25T18:25:55.1499272Z Waiting for deployment "nginx-demo" rollout to finish: 1 of 2 updated replicas are available...
  1532. 2020-08-25T18:25:59.7933879Z deployment "nginx-demo" successfully rolled out
  1533. 2020-08-25T18:25:59.8093154Z [lighthouse]$ [cluster2] local netshoot_pod nginx_svc_ip
  1534. 2020-08-25T18:26:00.7906590Z [lighthouse]$ [cluster2] netshoot_pod=netshoot-789f6cf54f-6pfdg
  1535. 2020-08-25T18:26:00.7942209Z [lighthouse]$ [cluster2] kubectl get pods -l app=netshoot
  1536. 2020-08-25T18:26:00.7960932Z [lighthouse]$ [cluster2] awk FNR == 2 {print $1}
  1537. 2020-08-25T18:26:00.7971844Z [lighthouse]$ [cluster2] kubectl get pods -l app=netshoot
  1538. 2020-08-25T18:26:00.7986255Z [lighthouse]$ [cluster2] command kubectl --context=cluster2 get pods -l app=netshoot
  1539. 2020-08-25T18:26:00.7996098Z [lighthouse]$ [cluster2] kubectl --context=cluster2 get pods -l app=netshoot
  1540. 2020-08-25T18:26:02.4271932Z [lighthouse]$ [cluster2] nginx_svc_ip=100.91.27.182
  1541. 2020-08-25T18:26:02.4287676Z [lighthouse]$ [cluster2] with_context cluster1 get_svc_ip nginx-demo
  1542. 2020-08-25T18:26:02.4302112Z [lighthouse]$ [cluster2] with_context cluster1 get_svc_ip nginx-demo
  1543. 2020-08-25T18:26:02.4318834Z [lighthouse]$ [cluster2] local cluster=cluster1
  1544. 2020-08-25T18:26:02.4328107Z [lighthouse]$ [cluster1] local cmnd=get_svc_ip
  1545. 2020-08-25T18:26:02.4338873Z [lighthouse]$ [cluster1] get_svc_ip nginx-demo
  1546. 2020-08-25T18:26:02.4349790Z [lighthouse]$ [cluster1] get_svc_ip
  1547. 2020-08-25T18:26:02.4362993Z [lighthouse]$ [cluster1] local svc_name=nginx-demo
  1548. 2020-08-25T18:26:02.4375437Z [lighthouse]$ [cluster1] local svc_ip
  1549. 2020-08-25T18:26:03.3745379Z [lighthouse]$ [cluster1] svc_ip=100.91.27.182
  1550. 2020-08-25T18:26:03.3746111Z [lighthouse]$ [cluster1] kubectl --context=cluster1 get svc -l app=nginx-demo
  1551. 2020-08-25T18:26:03.3746615Z [lighthouse]$ [cluster1] awk FNR == 2 {print $3}
  1552. 2020-08-25T18:26:03.3747110Z [lighthouse]$ [cluster1] kubectl --context=cluster1 get svc -l app=nginx-demo
  1553. 2020-08-25T18:26:03.3747826Z [lighthouse]$ [cluster1] command kubectl --context=cluster1 get svc -l app=nginx-demo
  1554. 2020-08-25T18:26:03.3748154Z [lighthouse]$ [cluster1] kubectl --context=cluster1 get svc -l app=nginx-demo
  1555. 2020-08-25T18:26:04.1374777Z [lighthouse]$ [cluster2] with_retries 5 test_connection netshoot-789f6cf54f-6pfdg 100.91.27.182
  1556. 2020-08-25T18:26:04.1391102Z [lighthouse]$ [cluster2] with_retries 5 test_connection netshoot-789f6cf54f-6pfdg 100.91.27.182
  1557. 2020-08-25T18:26:04.1403825Z [lighthouse]$ [cluster2] local retries
  1558. 2020-08-25T18:26:04.1421901Z [lighthouse]$ [cluster2] retries=1 2 3 4 5
  1559. 2020-08-25T18:26:04.1442761Z [lighthouse]$ [cluster2] eval echo {1..5}
  1560. 2020-08-25T18:26:04.1463180Z [lighthouse]$ [cluster2] local cmnd=test_connection
  1561. 2020-08-25T18:26:04.1480944Z [lighthouse]$ [cluster2] wait 5426
  1562. 2020-08-25T18:26:04.1487043Z [lighthouse]$ [cluster2] test_connection netshoot-789f6cf54f-6pfdg 100.91.27.182
  1563. 2020-08-25T18:26:04.1508944Z [lighthouse]$ [cluster2] test_connection
  1564. 2020-08-25T18:26:04.1515067Z [lighthouse]$ [cluster2] local source_pod=netshoot-789f6cf54f-6pfdg
  1565. 2020-08-25T18:26:04.1534401Z [lighthouse]$ [cluster2] local target_address=100.91.27.182
  1566. 2020-08-25T18:26:04.1534931Z Attempting connectivity between clusters - netshoot-789f6cf54f-6pfdg --> 100.91.27.182
  1567. 2020-08-25T18:26:04.1546582Z [lighthouse]$ [cluster2] kubectl exec netshoot-789f6cf54f-6pfdg -- curl --output /dev/null -m 30 --silent --head --fail 100.91.27.182
  1568. 2020-08-25T18:26:04.1560569Z [lighthouse]$ [cluster2] kubectl exec netshoot-789f6cf54f-6pfdg -- curl --output /dev/null -m 30 --silent --head --fail 100.91.27.182
  1569. 2020-08-25T18:26:04.1576028Z [lighthouse]$ [cluster2] command kubectl --context=cluster2 exec netshoot-789f6cf54f-6pfdg -- curl --output /dev/null -m 30 --silent --head --fail 100.91.27.182
  1570. 2020-08-25T18:26:04.1588070Z [lighthouse]$ [cluster2] kubectl --context=cluster2 exec netshoot-789f6cf54f-6pfdg -- curl --output /dev/null -m 30 --silent --head --fail 100.91.27.182
  1571. 2020-08-25T18:26:06.0974402Z Connection test was successful!
  1572. 2020-08-25T18:26:06.0989491Z [lighthouse]$ [cluster2] return 0
  1573. 2020-08-25T18:26:06.1002370Z [lighthouse]$ [cluster2] remove_resource /opt/shipyard/scripts/resources/netshoot.yaml
  1574. 2020-08-25T18:26:06.1014326Z [lighthouse]$ [cluster2] remove_resource /opt/shipyard/scripts/resources/netshoot.yaml
  1575. 2020-08-25T18:26:06.1027434Z [lighthouse]$ [cluster2] local resource_file=/opt/shipyard/scripts/resources/netshoot.yaml
  1576. 2020-08-25T18:26:06.1041776Z [lighthouse]$ [cluster2] kubectl delete -f /opt/shipyard/scripts/resources/netshoot.yaml
  1577. 2020-08-25T18:26:06.1055971Z [lighthouse]$ [cluster2] kubectl delete -f /opt/shipyard/scripts/resources/netshoot.yaml
  1578. 2020-08-25T18:26:06.1079091Z [lighthouse]$ [cluster2] command kubectl --context=cluster2 delete -f /opt/shipyard/scripts/resources/netshoot.yaml
  1579. 2020-08-25T18:26:06.1096917Z [lighthouse]$ [cluster2] kubectl --context=cluster2 delete -f /opt/shipyard/scripts/resources/netshoot.yaml
  1580. 2020-08-25T18:26:06.9568520Z deployment.apps "netshoot" deleted
  1581. 2020-08-25T18:26:06.9806111Z [lighthouse]$ [cluster2] with_context cluster1 remove_resource /opt/shipyard/scripts/resources/nginx-demo.yaml
  1582. 2020-08-25T18:26:06.9844102Z [lighthouse]$ [cluster2] with_context cluster1 remove_resource /opt/shipyard/scripts/resources/nginx-demo.yaml
  1583. 2020-08-25T18:26:06.9902337Z [lighthouse]$ [cluster2] local cluster=cluster1
  1584. 2020-08-25T18:26:06.9915827Z [lighthouse]$ [cluster1] local cmnd=remove_resource
  1585. 2020-08-25T18:26:06.9930636Z [lighthouse]$ [cluster1] remove_resource /opt/shipyard/scripts/resources/nginx-demo.yaml
  1586. 2020-08-25T18:26:06.9942629Z [lighthouse]$ [cluster1] remove_resource
  1587. 2020-08-25T18:26:06.9956321Z [lighthouse]$ [cluster1] local resource_file=/opt/shipyard/scripts/resources/nginx-demo.yaml
  1588. 2020-08-25T18:26:06.9968861Z [lighthouse]$ [cluster1] kubectl delete -f /opt/shipyard/scripts/resources/nginx-demo.yaml
  1589. 2020-08-25T18:26:06.9980564Z [lighthouse]$ [cluster1] kubectl delete -f /opt/shipyard/scripts/resources/nginx-demo.yaml
  1590. 2020-08-25T18:26:06.9996330Z [lighthouse]$ [cluster1] command kubectl --context=cluster1 delete -f /opt/shipyard/scripts/resources/nginx-demo.yaml
  1591. 2020-08-25T18:26:07.0011076Z [lighthouse]$ [cluster1] kubectl --context=cluster1 delete -f /opt/shipyard/scripts/resources/nginx-demo.yaml
  1592. 2020-08-25T18:26:07.8362007Z deployment.apps "nginx-demo" deleted
  1593. 2020-08-25T18:26:07.9419007Z service "nginx-demo" deleted
  1594. 2020-08-25T18:26:07.9839523Z [lighthouse]$ run_subm_clusters update_coredns_configmap
  1595. 2020-08-25T18:26:07.9914271Z [lighthouse]$ run_subm_clusters update_coredns_configmap
  1596. 2020-08-25T18:26:07.9966612Z [lighthouse]$ declare -a subm_clusters
  1597. 2020-08-25T18:26:08.0077305Z [lighthouse]$ [cluster2] run_parallel cluster1 cluster2 update_coredns_configmap
  1598. 2020-08-25T18:26:08.0132092Z [lighthouse]$ [cluster2] run_parallel cluster1 cluster2 cluster1 cluster2 update_coredns_configmap
  1599. 2020-08-25T18:26:08.0191387Z [lighthouse]$ [cluster2] local cmnd=update_coredns_configmap
  1600. 2020-08-25T18:26:08.0241281Z [lighthouse]$ [cluster2] declare -A pids
  1601. 2020-08-25T18:26:08.0283443Z [lighthouse]$ [cluster2] eval echo cluster1 cluster2
  1602. 2020-08-25T18:26:08.0370486Z [lighthouse]$ [cluster1] pids[cluster1]=5488
  1603. 2020-08-25T18:26:08.0388095Z [lighthouse]$ [cluster1] set -o pipefail
  1604. 2020-08-25T18:26:08.0418644Z [lighthouse]$ [cluster1] update_coredns_configmap
  1605. 2020-08-25T18:26:08.0436233Z [lighthouse]$ [cluster2] set -o pipefail
  1606. 2020-08-25T18:26:08.0442923Z [lighthouse]$ [cluster1] sed /\[cluster1]/!s/^/[cluster1] /
  1607. 2020-08-25T18:26:08.0454777Z [lighthouse]$ [cluster2] pids[cluster2]=5490
  1608. 2020-08-25T18:26:08.0556211Z [lighthouse]$ [cluster2] update_coredns_configmap
  1609. 2020-08-25T18:26:08.0580315Z [lighthouse]$ [cluster2] sed /\[cluster2]/!s/^/[cluster2] /
  1610. 2020-08-25T18:26:08.0608483Z [lighthouse]$ [cluster2] wait 5490
  1611. 2020-08-25T18:26:14.4587440Z [lighthouse]$ [cluster1] update_coredns_configmap
  1612. 2020-08-25T18:26:14.4587796Z [cluster1] 
  1613. 2020-08-25T18:26:14.4588049Z [cluster1] 
  1614. 2020-08-25T18:26:14.4599938Z [lighthouse]$ [cluster1] command kubectl --context=cluster1 get cm -n kube-system coredns -o yaml
  1615. 2020-08-25T18:26:14.4602036Z [lighthouse]$ [cluster1] kubectl --context=cluster1 get cm -n kube-system coredns -o yaml
  1616. 2020-08-25T18:26:14.4602420Z [lighthouse]$ [cluster1] grep -q clusterset /tmp/coremap-cluster1.yaml
  1617. 2020-08-25T18:26:14.4602743Z [lighthouse]$ [cluster1] run_subm_clusters install_service_import
  1618. 2020-08-25T18:26:14.4603057Z [lighthouse]$ [cluster1] run_subm_clusters install_service_import
  1619. 2020-08-25T18:26:14.4603362Z [lighthouse]$ [cluster1] declare -a subm_clusters
  1620. 2020-08-25T18:26:14.4603991Z [cluster1] [lighthouse]$ [cluster2] run_parallel cluster1 cluster2 install_service_import
  1621. 2020-08-25T18:26:14.4604378Z [cluster1] [lighthouse]$ [cluster2] run_parallel cluster1 cluster2 cluster1 cluster2 install_service_import
  1622. 2020-08-25T18:26:14.4604717Z [cluster1] [lighthouse]$ [cluster2] local cmnd=install_service_import
  1623. 2020-08-25T18:26:14.4605374Z [cluster1] [lighthouse]$ [cluster2] declare -A pids
  1624. 2020-08-25T18:26:14.4605766Z [cluster1] [lighthouse]$ [cluster2] eval echo cluster1 cluster2
  1625. 2020-08-25T18:26:14.4606075Z [lighthouse]$ [cluster1] pids[cluster1]=5540
  1626. 2020-08-25T18:26:14.4606371Z [cluster1] [lighthouse]$ [cluster2] set -o pipefail
  1627. 2020-08-25T18:26:14.4606676Z [cluster1] [lighthouse]$ [cluster2] install_service_import
  1628. 2020-08-25T18:26:14.4606998Z [cluster1] [lighthouse]$ [cluster2] sed /\[cluster2]/!s/^/[cluster2] /
  1629. 2020-08-25T18:26:14.4607324Z [cluster1] [lighthouse]$ [cluster2] pids[cluster2]=5543
  1630. 2020-08-25T18:26:14.4607615Z [lighthouse]$ [cluster1] set -o pipefail
  1631. 2020-08-25T18:26:14.4607907Z [lighthouse]$ [cluster1] install_service_import
  1632. 2020-08-25T18:26:14.4608188Z [cluster1] [lighthouse]$ [cluster2] wait 5543
  1633. 2020-08-25T18:26:14.4608497Z [lighthouse]$ [cluster1] sed /\[cluster1]/!s/^/[cluster1] /
  1634. 2020-08-25T18:26:14.4608797Z [lighthouse]$ [cluster1] install_service_import
  1635. 2020-08-25T18:26:14.4609174Z [lighthouse]$ [cluster1] kubectl apply -f /go/src/github.com/submariner-io/lighthouse/package/lighthouse-crd.yaml
  1636. 2020-08-25T18:26:14.4609558Z [lighthouse]$ [cluster1] kubectl apply -f /go/src/github.com/submariner-io/lighthouse/package/lighthouse-crd.yaml
  1637. 2020-08-25T18:26:14.4609974Z [lighthouse]$ [cluster1] command kubectl --context=cluster1 apply -f /go/src/github.com/submariner-io/lighthouse/package/lighthouse-crd.yaml
  1638. 2020-08-25T18:26:14.4610396Z [lighthouse]$ [cluster1] kubectl --context=cluster1 apply -f /go/src/github.com/submariner-io/lighthouse/package/lighthouse-crd.yaml
  1639. 2020-08-25T18:26:14.4610797Z [cluster1] Warning: kubectl apply should be used on resource created by either kubectl create --save-config or kubectl apply
  1640. 2020-08-25T18:26:14.4613296Z [cluster1] customresourcedefinition.apiextensions.k8s.io/serviceimports.lighthouse.submariner.io configured
  1641. 2020-08-25T18:26:14.4613752Z [lighthouse]$ [cluster1] kubectl get crds
  1642. 2020-08-25T18:26:14.4614045Z [lighthouse]$ [cluster1] grep serviceimport
  1643. 2020-08-25T18:26:14.4614330Z [lighthouse]$ [cluster1] kubectl get crds
  1644. 2020-08-25T18:26:14.4614627Z [lighthouse]$ [cluster1] command kubectl --context=cluster1 get crds
  1645. 2020-08-25T18:26:14.4615056Z [lighthouse]$ [cluster1] kubectl --context=cluster1 get crds
  1646. 2020-08-25T18:26:14.4615939Z [cluster1] serviceimports.lighthouse.submariner.io 2020-08-25T18:24:51Z
  1647. 2020-08-25T18:26:14.4616291Z [cluster1] [lighthouse]$ [cluster2] install_service_import
  1648. 2020-08-25T18:26:14.4616698Z [cluster1] [lighthouse]$ [cluster2] kubectl apply -f /go/src/github.com/submariner-io/lighthouse/package/lighthouse-crd.yaml
  1649. 2020-08-25T18:26:14.4617101Z [cluster1] [lighthouse]$ [cluster2] kubectl apply -f /go/src/github.com/submariner-io/lighthouse/package/lighthouse-crd.yaml
  1650. 2020-08-25T18:26:14.4617537Z [cluster1] [lighthouse]$ [cluster2] command kubectl --context=cluster2 apply -f /go/src/github.com/submariner-io/lighthouse/package/lighthouse-crd.yaml
  1651. 2020-08-25T18:26:14.4617960Z [cluster1] [lighthouse]$ [cluster2] kubectl --context=cluster2 apply -f /go/src/github.com/submariner-io/lighthouse/package/lighthouse-crd.yaml
  1652. 2020-08-25T18:26:14.4618133Z [cluster1] [cluster2] customresourcedefinition.apiextensions.k8s.io/serviceimports.lighthouse.submariner.io unchanged
  1653. 2020-08-25T18:26:14.4618447Z [cluster1] [lighthouse]$ [cluster2] kubectl get crds
  1654. 2020-08-25T18:26:14.4618753Z [cluster1] [lighthouse]$ [cluster2] grep serviceimport
  1655. 2020-08-25T18:26:14.4619216Z [cluster1] [lighthouse]$ [cluster2] kubectl get crds
  1656. 2020-08-25T18:26:14.4619549Z [cluster1] [lighthouse]$ [cluster2] command kubectl --context=cluster2 get crds
  1657. 2020-08-25T18:26:14.4619882Z [cluster1] [lighthouse]$ [cluster2] kubectl --context=cluster2 get crds
  1658. 2020-08-25T18:26:14.4620228Z [cluster1] [cluster2] serviceimports.lighthouse.submariner.io 2020-08-25T18:24:52Z
  1659. 2020-08-25T18:26:14.4620531Z [cluster1] [lighthouse]$ [cluster2] wait 5540
  1660. 2020-08-25T18:26:14.4733787Z [lighthouse]$ [cluster2] update_coredns_configmap
  1661. 2020-08-25T18:26:14.4734511Z [cluster2] 
  1662. 2020-08-25T18:26:14.4738029Z [cluster2] 
  1663. 2020-08-25T18:26:14.4738438Z [lighthouse]$ [cluster2] command kubectl --context=cluster2 get cm -n kube-system coredns -o yaml
  1664. 2020-08-25T18:26:14.4738810Z [lighthouse]$ [cluster2] kubectl --context=cluster2 get cm -n kube-system coredns -o yaml
  1665. 2020-08-25T18:26:14.4739167Z [lighthouse]$ [cluster2] grep -q clusterset /tmp/coremap-cluster2.yaml
  1666. 2020-08-25T18:26:14.4739536Z [lighthouse]$ [cluster2] run_subm_clusters install_service_import
  1667. 2020-08-25T18:26:14.4739865Z [lighthouse]$ [cluster2] run_subm_clusters install_service_import
  1668. 2020-08-25T18:26:14.4740171Z [lighthouse]$ [cluster2] declare -a subm_clusters
  1669. 2020-08-25T18:26:14.4740501Z [lighthouse]$ [cluster2] run_parallel cluster1 cluster2 install_service_import
  1670. 2020-08-25T18:26:14.4740850Z [lighthouse]$ [cluster2] run_parallel cluster1 cluster2 cluster1 cluster2 install_service_import
  1671. 2020-08-25T18:26:14.4741173Z [lighthouse]$ [cluster2] local cmnd=install_service_import
  1672. 2020-08-25T18:26:14.4741465Z [lighthouse]$ [cluster2] declare -A pids
  1673. 2020-08-25T18:26:14.4741763Z [lighthouse]$ [cluster2] eval echo cluster1 cluster2
  1674. 2020-08-25T18:26:14.4742073Z [cluster2] [lighthouse]$ [cluster1] pids[cluster1]=5580
  1675. 2020-08-25T18:26:14.4742373Z [lighthouse]$ [cluster2] pids[cluster2]=5582
  1676. 2020-08-25T18:26:14.4742672Z [cluster2] [lighthouse]$ [cluster1] set -o pipefail
  1677. 2020-08-25T18:26:14.4742956Z [lighthouse]$ [cluster2] set -o pipefail
  1678. 2020-08-25T18:26:14.4743405Z [lighthouse]$ [cluster2] install_service_import
  1679. 2020-08-25T18:26:14.4743723Z [lighthouse]$ [cluster2] wait 5582
  1680. 2020-08-25T18:26:14.4744026Z [lighthouse]$ [cluster2] sed /\[cluster2]/!s/^/[cluster2] /
  1681. 2020-08-25T18:26:14.4744336Z [cluster2] [lighthouse]$ [cluster1] install_service_import
  1682. 2020-08-25T18:26:14.4745031Z [cluster2] [lighthouse]$ [cluster1] sed /\[cluster1]/!s/^/[cluster1] /
  1683. 2020-08-25T18:26:14.4745980Z [lighthouse]$ [cluster2] install_service_import
  1684. 2020-08-25T18:26:14.4746464Z [lighthouse]$ [cluster2] kubectl apply -f /go/src/github.com/submariner-io/lighthouse/package/lighthouse-crd.yaml
  1685. 2020-08-25T18:26:14.4746892Z [lighthouse]$ [cluster2] kubectl apply -f /go/src/github.com/submariner-io/lighthouse/package/lighthouse-crd.yaml
  1686. 2020-08-25T18:26:14.4747328Z [lighthouse]$ [cluster2] command kubectl --context=cluster2 apply -f /go/src/github.com/submariner-io/lighthouse/package/lighthouse-crd.yaml
  1687. 2020-08-25T18:26:14.4747749Z [lighthouse]$ [cluster2] kubectl --context=cluster2 apply -f /go/src/github.com/submariner-io/lighthouse/package/lighthouse-crd.yaml
  1688. 2020-08-25T18:26:14.4748141Z [cluster2] Warning: kubectl apply should be used on resource created by either kubectl create --save-config or kubectl apply
  1689. 2020-08-25T18:26:14.4748312Z [cluster2] customresourcedefinition.apiextensions.k8s.io/serviceimports.lighthouse.submariner.io configured
  1690. 2020-08-25T18:26:14.4748600Z [lighthouse]$ [cluster2] kubectl get crds
  1691. 2020-08-25T18:26:14.4748886Z [lighthouse]$ [cluster2] grep serviceimport
  1692. 2020-08-25T18:26:14.4749174Z [lighthouse]$ [cluster2] kubectl get crds
  1693. 2020-08-25T18:26:14.4749490Z [lighthouse]$ [cluster2] command kubectl --context=cluster2 get crds
  1694. 2020-08-25T18:26:14.4750003Z [lighthouse]$ [cluster2] kubectl --context=cluster2 get crds
  1695. 2020-08-25T18:26:14.4750340Z [cluster2] serviceimports.lighthouse.submariner.io 2020-08-25T18:24:52Z
  1696. 2020-08-25T18:26:14.4750635Z [lighthouse]$ [cluster2] wait 5580
  1697. 2020-08-25T18:26:14.4750941Z [cluster2] [lighthouse]$ [cluster1] install_service_import
  1698. 2020-08-25T18:26:14.4751330Z [cluster2] [lighthouse]$ [cluster1] kubectl apply -f /go/src/github.com/submariner-io/lighthouse/package/lighthouse-crd.yaml
  1699. 2020-08-25T18:26:14.4751728Z [cluster2] [lighthouse]$ [cluster1] kubectl apply -f /go/src/github.com/submariner-io/lighthouse/package/lighthouse-crd.yaml
  1700. 2020-08-25T18:26:14.4752142Z [cluster2] [lighthouse]$ [cluster1] command kubectl --context=cluster1 apply -f /go/src/github.com/submariner-io/lighthouse/package/lighthouse-crd.yaml
  1701. 2020-08-25T18:26:14.4752568Z [cluster2] [lighthouse]$ [cluster1] kubectl --context=cluster1 apply -f /go/src/github.com/submariner-io/lighthouse/package/lighthouse-crd.yaml
  1702. 2020-08-25T18:26:14.4752753Z [cluster2] [cluster1] customresourcedefinition.apiextensions.k8s.io/serviceimports.lighthouse.submariner.io unchanged
  1703. 2020-08-25T18:26:14.4753081Z [cluster2] [lighthouse]$ [cluster1] kubectl get crds
  1704. 2020-08-25T18:26:14.4753388Z [cluster2] [lighthouse]$ [cluster1] grep serviceimport
  1705. 2020-08-25T18:26:14.4753690Z [cluster2] [lighthouse]$ [cluster1] kubectl get crds
  1706. 2020-08-25T18:26:14.4754024Z [cluster2] [lighthouse]$ [cluster1] command kubectl --context=cluster1 get crds
  1707. 2020-08-25T18:26:14.4754357Z [cluster2] [lighthouse]$ [cluster1] kubectl --context=cluster1 get crds
  1708. 2020-08-25T18:26:14.4755708Z [cluster2] [cluster1] serviceimports.lighthouse.submariner.io 2020-08-25T18:24:51Z
  1709. 2020-08-25T18:26:14.4756091Z [lighthouse]$ [cluster2] wait 5488
  1710. 2020-08-25T18:26:14.4764142Z make[1]: Leaving directory '/go/src/github.com/submariner-io/lighthouse'
  1711. 2020-08-25T18:26:14.4861882Z [lighthouse]$ declare_kubeconfig
  1712. 2020-08-25T18:26:14.4874906Z [lighthouse]$ declare_kubeconfig
  1713. 2020-08-25T18:26:14.4888310Z [lighthouse]$ source /opt/shipyard/scripts/lib/kubecfg
  1714. 2020-08-25T18:26:14.4913837Z [lighthouse]$ export KUBECONFIG
  1715. 2020-08-25T18:26:14.4963976Z [lighthouse]$ KUBECONFIG=/go/src/github.com/submariner-io/lighthouse/output/kubeconfigs/kind-config-cluster2:/go/src/github.com/submariner-io/lighthouse/output/kubeconfigs/kind-config-cluster1:
  1716. 2020-08-25T18:26:14.4980175Z [lighthouse]$ find /go/src/github.com/submariner-io/lighthouse/output/kubeconfigs -type f -printf %p:
  1717. 2020-08-25T18:26:14.5015547Z [lighthouse]$ test_with_e2e_tests
  1718. 2020-08-25T18:26:14.5024378Z [lighthouse]$ test_with_e2e_tests
  1719. 2020-08-25T18:26:14.5036818Z [lighthouse]$ cd /go/src/github.com/submariner-io/lighthouse/test/e2e
  1720. 2020-08-25T18:26:14.5061381Z [e2e]$ go test -v -timeout 30m -args -ginkgo.v -ginkgo.randomizeAllSpecs -ginkgo.trace -submariner-namespace submariner-operator -dp-context cluster1 -dp-context cluster2 -ginkgo.reportPassed -test.timeout 15m -ginkgo.focus \[.*\] -ginkgo.reportFile /go/src/github.com/submariner-io/lighthouse/output/e2e-junit.xml
  1721. 2020-08-25T18:26:14.5076333Z [e2e]$ tee /go/src/github.com/submariner-io/lighthouse/output/e2e-tests.log
  1722. 2020-08-25T18:26:14.5106169Z [e2e]$ generate_context_flags
  1723. 2020-08-25T18:26:14.5122738Z [e2e]$ generate_context_flags
  1724. 2020-08-25T18:26:14.5135036Z [e2e]$ [cluster1] printf -dp-context cluster1
  1725. 2020-08-25T18:26:14.5147537Z [e2e]$ [cluster2] printf -dp-context cluster2
  1726. 2020-08-25T18:27:15.0164367Z === RUN TestE2E
  1727. 2020-08-25T18:27:15.0183396Z Running Suite: Submariner E2E suite
  1728. 2020-08-25T18:27:15.0185504Z ===================================
  1729. 2020-08-25T18:27:15.0186389Z Random Seed: 1598380035 - Will randomize all specs
  1730. 2020-08-25T18:27:15.0186696Z Will run 7 of 7 specs
  1731. 2020-08-25T18:27:15.0187195Z
  1732. 2020-08-25T18:27:15.0229342Z STEP: Creating kubernetes clients
  1733. 2020-08-25T18:27:15.0283905Z STEP: Creating lighthouse clients
  1734. 2020-08-25T18:27:15.0291218Z [discovery] Test Headless Service Discovery Across Clusters when a pod tries to resolve a headless service which is exported locally and in a remote cluster
  1735. 2020-08-25T18:27:15.0292358Z should resolve the backing pod IPs from both clusters
  1736. 2020-08-25T18:27:15.0293075Z /go/src/github.com/submariner-io/lighthouse/test/e2e/discovery/headless_services.go:25
  1737. 2020-08-25T18:27:15.0293441Z STEP: Creating namespace objects with basename "discovery"
  1738. 2020-08-25T18:27:15.0480126Z STEP: Generated namespace "e2e-tests-discovery-j5vf2" in cluster "cluster1" to execute the tests in
  1739. 2020-08-25T18:27:15.0480600Z STEP: Creating namespace "e2e-tests-discovery-j5vf2" in cluster "cluster2"
  1740. 2020-08-25T18:27:15.0668439Z STEP: Creating an Nginx Deployment on on "cluster2"
  1741. 2020-08-25T18:27:20.1642054Z STEP: Creating a Nginx Headless Service on "cluster2"
  1742. 2020-08-25T18:27:20.1745375Z STEP: Creating serviceExport nginx-headless.e2e-tests-discovery-j5vf2 on "cluster2"
  1743. 2020-08-25T18:27:20.1818269Z STEP: Retrieving ServiceExport nginx-headless.e2e-tests-discovery-j5vf2 on "cluster2"
  1744. 2020-08-25T18:27:25.1949085Z STEP: Creating an Nginx Deployment on on "cluster1"
  1745. 2020-08-25T18:27:30.2206110Z STEP: Creating a Nginx Headless Service on "cluster1"
  1746. 2020-08-25T18:27:30.2254842Z STEP: Creating serviceExport nginx-headless.e2e-tests-discovery-j5vf2 on "cluster1"
  1747. 2020-08-25T18:27:30.2333935Z STEP: Retrieving ServiceExport nginx-headless.e2e-tests-discovery-j5vf2 on "cluster1"
  1748. 2020-08-25T18:27:35.2392804Z STEP: Creating a Netshoot Deployment on "cluster1"
  1749. 2020-08-25T18:27:40.2574356Z STEP: Retrieving Endpoints for nginx-headless on "cluster2"
  1750. 2020-08-25T18:27:40.2620756Z STEP: Retrieving Endpoints for nginx-headless on "cluster1"
  1751. 2020-08-25T18:27:40.2801477Z STEP: Executing "dig +short nginx-headless.e2e-tests-discovery-j5vf2.svc.clusterset.local nginx-headless.e2e-tests-discovery-j5vf2.svc.supercluster.local" to verify IPs [10.242.128.2 10.241.192.4] for service "nginx-headless" "are" discoverable
  1752. 2020-08-25T18:27:40.2806638Z Aug 25 18:27:40.279: INFO: ExecWithOptions {Command:[dig +short nginx-headless.e2e-tests-discovery-j5vf2.svc.clusterset.local nginx-headless.e2e-tests-discovery-j5vf2.svc.supercluster.local] Namespace:e2e-tests-discovery-j5vf2 PodName:netshoot6kzr4-6757486d56-hnxrx ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
  1753. 2020-08-25T18:27:40.3724724Z STEP: Validating that dig result are "10.242.128.2\n10.241.192.4\n10.241.192.4\n10.242.128.2"
  1754. 2020-08-25T18:27:40.3730345Z STEP: Deleting serviceExport nginx-headless.e2e-tests-discovery-j5vf2 on "cluster2"
  1755. 2020-08-25T18:27:45.3856076Z STEP: Executing "dig +short nginx-headless.e2e-tests-discovery-j5vf2.svc.clusterset.local nginx-headless.e2e-tests-discovery-j5vf2.svc.supercluster.local" to verify IPs [10.242.128.2] for service "nginx-headless" "are not" discoverable
  1756. 2020-08-25T18:27:45.3856951Z Aug 25 18:27:45.385: INFO: ExecWithOptions {Command:[dig +short nginx-headless.e2e-tests-discovery-j5vf2.svc.clusterset.local nginx-headless.e2e-tests-discovery-j5vf2.svc.supercluster.local] Namespace:e2e-tests-discovery-j5vf2 PodName:netshoot6kzr4-6757486d56-hnxrx ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
  1757. 2020-08-25T18:27:45.4576324Z STEP: Validating that dig result are not "10.241.192.4\n10.241.192.4"
  1758. 2020-08-25T18:27:45.4578535Z STEP: Executing "dig +short nginx-headless.e2e-tests-discovery-j5vf2.svc.clusterset.local nginx-headless.e2e-tests-discovery-j5vf2.svc.supercluster.local" to verify IPs [10.241.192.4] for service "nginx-headless" "are" discoverable
  1759. 2020-08-25T18:27:45.4579879Z Aug 25 18:27:45.457: INFO: ExecWithOptions {Command:[dig +short nginx-headless.e2e-tests-discovery-j5vf2.svc.clusterset.local nginx-headless.e2e-tests-discovery-j5vf2.svc.supercluster.local] Namespace:e2e-tests-discovery-j5vf2 PodName:netshoot6kzr4-6757486d56-hnxrx ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
  1760. 2020-08-25T18:27:45.5455423Z STEP: Validating that dig result are "10.241.192.4\n10.241.192.4"
  1761. 2020-08-25T18:27:45.5456674Z STEP: Deleting namespace "e2e-tests-discovery-j5vf2" on cluster "cluster1"
  1762. 2020-08-25T18:27:45.5541465Z STEP: Deleting namespace "e2e-tests-discovery-j5vf2" on cluster "cluster2"
  1763. 2020-08-25T18:27:45.5648645Z •
  1764. 2020-08-25T18:27:45.5651276Z ------------------------------
  1765. 2020-08-25T18:27:45.5651611Z STEP: Creating namespace objects with basename "discovery"
  1766. 2020-08-25T18:27:45.5652188Z STEP: Generated namespace "e2e-tests-discovery-j5vf2" in cluster "cluster1" to execute the tests in
  1767. 2020-08-25T18:27:45.5652575Z STEP: Creating namespace "e2e-tests-discovery-j5vf2" in cluster "cluster2"
  1768. 2020-08-25T18:27:45.5652885Z STEP: Creating an Nginx Deployment on on "cluster2"
  1769. 2020-08-25T18:27:45.5653331Z STEP: Creating a Nginx Headless Service on "cluster2"
  1770. 2020-08-25T18:27:45.5653672Z STEP: Creating serviceExport nginx-headless.e2e-tests-discovery-j5vf2 on "cluster2"
  1771. 2020-08-25T18:27:45.5654018Z STEP: Retrieving ServiceExport nginx-headless.e2e-tests-discovery-j5vf2 on "cluster2"
  1772. 2020-08-25T18:27:45.5654317Z STEP: Creating an Nginx Deployment on on "cluster1"
  1773. 2020-08-25T18:27:45.5654605Z STEP: Creating a Nginx Headless Service on "cluster1"
  1774. 2020-08-25T18:27:45.5655084Z STEP: Creating serviceExport nginx-headless.e2e-tests-discovery-j5vf2 on "cluster1"
  1775. 2020-08-25T18:27:45.5655835Z STEP: Retrieving ServiceExport nginx-headless.e2e-tests-discovery-j5vf2 on "cluster1"
  1776. 2020-08-25T18:27:45.5656300Z STEP: Creating a Netshoot Deployment on "cluster1"
  1777. 2020-08-25T18:27:45.5656602Z STEP: Retrieving Endpoints for nginx-headless on "cluster2"
  1778. 2020-08-25T18:27:45.5656907Z STEP: Retrieving Endpoints for nginx-headless on "cluster1"
  1779. 2020-08-25T18:27:45.5657617Z STEP: Executing "dig +short nginx-headless.e2e-tests-discovery-j5vf2.svc.clusterset.local nginx-headless.e2e-tests-discovery-j5vf2.svc.supercluster.local" to verify IPs [10.242.128.2 10.241.192.4] for service "nginx-headless" "are" discoverable
  1780. 2020-08-25T18:27:45.5658577Z Aug 25 18:27:40.279: INFO: ExecWithOptions {Command:[dig +short nginx-headless.e2e-tests-discovery-j5vf2.svc.clusterset.local nginx-headless.e2e-tests-discovery-j5vf2.svc.supercluster.local] Namespace:e2e-tests-discovery-j5vf2 PodName:netshoot6kzr4-6757486d56-hnxrx ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
  1781. 2020-08-25T18:27:45.5659023Z STEP: Validating that dig result are "10.242.128.2\n10.241.192.4\n10.241.192.4\n10.242.128.2"
  1782. 2020-08-25T18:27:45.5659392Z STEP: Deleting serviceExport nginx-headless.e2e-tests-discovery-j5vf2 on "cluster2"
  1783. 2020-08-25T18:27:45.5659920Z STEP: Executing "dig +short nginx-headless.e2e-tests-discovery-j5vf2.svc.clusterset.local nginx-headless.e2e-tests-discovery-j5vf2.svc.supercluster.local" to verify IPs [10.242.128.2] for service "nginx-headless" "are not" discoverable
  1784. 2020-08-25T18:27:45.5660605Z Aug 25 18:27:45.385: INFO: ExecWithOptions {Command:[dig +short nginx-headless.e2e-tests-discovery-j5vf2.svc.clusterset.local nginx-headless.e2e-tests-discovery-j5vf2.svc.supercluster.local] Namespace:e2e-tests-discovery-j5vf2 PodName:netshoot6kzr4-6757486d56-hnxrx ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
  1785. 2020-08-25T18:27:45.5661010Z STEP: Validating that dig result are not "10.241.192.4\n10.241.192.4"
  1786. 2020-08-25T18:27:45.5661531Z STEP: Executing "dig +short nginx-headless.e2e-tests-discovery-j5vf2.svc.clusterset.local nginx-headless.e2e-tests-discovery-j5vf2.svc.supercluster.local" to verify IPs [10.241.192.4] for service "nginx-headless" "are" discoverable
  1787. 2020-08-25T18:27:45.5662382Z Aug 25 18:27:45.457: INFO: ExecWithOptions {Command:[dig +short nginx-headless.e2e-tests-discovery-j5vf2.svc.clusterset.local nginx-headless.e2e-tests-discovery-j5vf2.svc.supercluster.local] Namespace:e2e-tests-discovery-j5vf2 PodName:netshoot6kzr4-6757486d56-hnxrx ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
  1788. 2020-08-25T18:27:45.5662779Z STEP: Validating that dig result are "10.241.192.4\n10.241.192.4"
  1789. 2020-08-25T18:27:45.5663113Z STEP: Deleting namespace "e2e-tests-discovery-j5vf2" on cluster "cluster1"
  1790. 2020-08-25T18:27:45.5663449Z STEP: Deleting namespace "e2e-tests-discovery-j5vf2" on cluster "cluster2"
  1791. 2020-08-25T18:27:45.5663544Z
  1792. 2020-08-25T18:27:45.5663598Z
  1793. 2020-08-25T18:27:45.5663951Z [discovery] Test Service Discovery Across Clusters when there are no active pods for a service
  1794. 2020-08-25T18:27:45.5664261Z should not resolve the service
  1795. 2020-08-25T18:27:45.5664603Z /go/src/github.com/submariner-io/lighthouse/test/e2e/discovery/service_discovery.go:45
  1796. 2020-08-25T18:27:45.5667466Z STEP: Creating namespace objects with basename "discovery"
  1797. 2020-08-25T18:27:45.5808216Z STEP: Generated namespace "e2e-tests-discovery-l8z8t" in cluster "cluster1" to execute the tests in
  1798. 2020-08-25T18:27:45.5808655Z STEP: Creating namespace "e2e-tests-discovery-l8z8t" in cluster "cluster2"
  1799. 2020-08-25T18:27:45.5916513Z STEP: Creating an Nginx Deployment on on "cluster2"
  1800. 2020-08-25T18:27:50.6415172Z STEP: Creating a Nginx Service on "cluster2"
  1801. 2020-08-25T18:27:50.6557131Z STEP: Creating serviceExport nginx-demo.e2e-tests-discovery-l8z8t on "cluster2"
  1802. 2020-08-25T18:27:50.7003214Z STEP: Retrieving ServiceExport nginx-demo.e2e-tests-discovery-l8z8t on "cluster2"
  1803. 2020-08-25T18:27:55.7095534Z STEP: Creating a Netshoot Deployment on "cluster1"
  1804. 2020-08-25T18:28:00.7214101Z STEP: Retrieving service nginx-demo.e2e-tests-discovery-l8z8t on "cluster2"
  1805. 2020-08-25T18:28:00.7256305Z STEP: Retrieving ServiceImport for nginx-demo-e2e-tests-discovery-l8z8t- on "cluster1"
  1806. 2020-08-25T18:28:00.7291942Z STEP: Executing "dig +short nginx-demo.e2e-tests-discovery-l8z8t.svc.clusterset.local nginx-demo.e2e-tests-discovery-l8z8t.svc.supercluster.local" to verify IP "100.92.146.202" for service "nginx-demo" "is" discoverable
  1807. 2020-08-25T18:28:00.7293202Z Aug 25 18:28:00.728: INFO: ExecWithOptions {Command:[dig +short nginx-demo.e2e-tests-discovery-l8z8t.svc.clusterset.local nginx-demo.e2e-tests-discovery-l8z8t.svc.supercluster.local] Namespace:e2e-tests-discovery-l8z8t PodName:netshoot68f7k-6757486d56-7s2kx ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
  1808. 2020-08-25T18:28:00.7953495Z STEP: Validating that dig result is "100.92.146.202\n100.92.146.202"
  1809. 2020-08-25T18:28:00.7953940Z STEP: Setting Nginx deployment replicas to 0
  1810. 2020-08-25T18:28:00.7999211Z STEP: Executing "dig +short nginx-demo.e2e-tests-discovery-l8z8t.svc.clusterset.local nginx-demo.e2e-tests-discovery-l8z8t.svc.supercluster.local" to verify IP "100.92.146.202" for service "nginx-demo" "is not" discoverable
  1811. 2020-08-25T18:28:00.8001775Z Aug 25 18:28:00.799: INFO: ExecWithOptions {Command:[dig +short nginx-demo.e2e-tests-discovery-l8z8t.svc.clusterset.local nginx-demo.e2e-tests-discovery-l8z8t.svc.supercluster.local] Namespace:e2e-tests-discovery-l8z8t PodName:netshoot68f7k-6757486d56-7s2kx ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
  1812. 2020-08-25T18:28:00.9834286Z STEP: Validating that dig result is not ""
  1813. 2020-08-25T18:28:00.9839147Z STEP: Setting Nginx deployment replicas to 2
  1814. 2020-08-25T18:28:00.9930640Z STEP: Executing "dig +short nginx-demo.e2e-tests-discovery-l8z8t.svc.clusterset.local nginx-demo.e2e-tests-discovery-l8z8t.svc.supercluster.local" to verify IP "100.92.146.202" for service "nginx-demo" "is" discoverable
  1815. 2020-08-25T18:28:00.9932048Z Aug 25 18:28:00.992: INFO: ExecWithOptions {Command:[dig +short nginx-demo.e2e-tests-discovery-l8z8t.svc.clusterset.local nginx-demo.e2e-tests-discovery-l8z8t.svc.supercluster.local] Namespace:e2e-tests-discovery-l8z8t PodName:netshoot68f7k-6757486d56-7s2kx ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
  1816. 2020-08-25T18:28:01.1770197Z STEP: Validating that dig result is ""
  1817. 2020-08-25T18:28:06.4102663Z Aug 25 18:28:06.176: INFO: ExecWithOptions {Command:[dig +short nginx-demo.e2e-tests-discovery-l8z8t.svc.clusterset.local nginx-demo.e2e-tests-discovery-l8z8t.svc.supercluster.local] Namespace:e2e-tests-discovery-l8z8t PodName:netshoot68f7k-6757486d56-7s2kx ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
  1818. 2020-08-25T18:28:06.4103234Z STEP: Validating that dig result is "100.92.146.202\n100.92.146.202"
  1819. 2020-08-25T18:28:06.4103606Z STEP: Deleting namespace "e2e-tests-discovery-l8z8t" on cluster "cluster1"
  1820. 2020-08-25T18:28:06.4103930Z STEP: Deleting namespace "e2e-tests-discovery-l8z8t" on cluster "cluster2"
  1821. 2020-08-25T18:28:06.4104189Z •
  1822. 2020-08-25T18:28:06.4104450Z ------------------------------
  1823. 2020-08-25T18:28:06.4105625Z STEP: Creating namespace objects with basename "discovery"
  1824. 2020-08-25T18:28:06.4106156Z STEP: Generated namespace "e2e-tests-discovery-l8z8t" in cluster "cluster1" to execute the tests in
  1825. 2020-08-25T18:28:06.4106493Z STEP: Creating namespace "e2e-tests-discovery-l8z8t" in cluster "cluster2"
  1826. 2020-08-25T18:28:06.4106802Z STEP: Creating an Nginx Deployment on on "cluster2"
  1827. 2020-08-25T18:28:06.4107090Z STEP: Creating a Nginx Service on "cluster2"
  1828. 2020-08-25T18:28:06.4107394Z STEP: Creating serviceExport nginx-demo.e2e-tests-discovery-l8z8t on "cluster2"
  1829. 2020-08-25T18:28:06.4107756Z STEP: Retrieving ServiceExport nginx-demo.e2e-tests-discovery-l8z8t on "cluster2"
  1830. 2020-08-25T18:28:06.4108062Z STEP: Creating a Netshoot Deployment on "cluster1"
  1831. 2020-08-25T18:28:06.4108677Z STEP: Retrieving service nginx-demo.e2e-tests-discovery-l8z8t on "cluster2"
  1832. 2020-08-25T18:28:06.4109095Z STEP: Retrieving ServiceImport for nginx-demo-e2e-tests-discovery-l8z8t- on "cluster1"
  1833. 2020-08-25T18:28:06.4109597Z STEP: Executing "dig +short nginx-demo.e2e-tests-discovery-l8z8t.svc.clusterset.local nginx-demo.e2e-tests-discovery-l8z8t.svc.supercluster.local" to verify IP "100.92.146.202" for service "nginx-demo" "is" discoverable
  1834. 2020-08-25T18:28:06.4110267Z Aug 25 18:28:00.728: INFO: ExecWithOptions {Command:[dig +short nginx-demo.e2e-tests-discovery-l8z8t.svc.clusterset.local nginx-demo.e2e-tests-discovery-l8z8t.svc.supercluster.local] Namespace:e2e-tests-discovery-l8z8t PodName:netshoot68f7k-6757486d56-7s2kx ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
  1835. 2020-08-25T18:28:06.4110669Z STEP: Validating that dig result is "100.92.146.202\n100.92.146.202"
  1836. 2020-08-25T18:28:06.4110969Z STEP: Setting Nginx deployment replicas to 0
  1837. 2020-08-25T18:28:06.4111461Z STEP: Executing "dig +short nginx-demo.e2e-tests-discovery-l8z8t.svc.clusterset.local nginx-demo.e2e-tests-discovery-l8z8t.svc.supercluster.local" to verify IP "100.92.146.202" for service "nginx-demo" "is not" discoverable
  1838. 2020-08-25T18:28:06.4112133Z Aug 25 18:28:00.799: INFO: ExecWithOptions {Command:[dig +short nginx-demo.e2e-tests-discovery-l8z8t.svc.clusterset.local nginx-demo.e2e-tests-discovery-l8z8t.svc.supercluster.local] Namespace:e2e-tests-discovery-l8z8t PodName:netshoot68f7k-6757486d56-7s2kx ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
  1839. 2020-08-25T18:28:06.4112478Z STEP: Validating that dig result is not ""
  1840. 2020-08-25T18:28:06.4112766Z STEP: Setting Nginx deployment replicas to 2
  1841. 2020-08-25T18:28:06.4113640Z STEP: Executing "dig +short nginx-demo.e2e-tests-discovery-l8z8t.svc.clusterset.local nginx-demo.e2e-tests-discovery-l8z8t.svc.supercluster.local" to verify IP "100.92.146.202" for service "nginx-demo" "is" discoverable
  1842. 2020-08-25T18:28:06.4114315Z Aug 25 18:28:00.992: INFO: ExecWithOptions {Command:[dig +short nginx-demo.e2e-tests-discovery-l8z8t.svc.clusterset.local nginx-demo.e2e-tests-discovery-l8z8t.svc.supercluster.local] Namespace:e2e-tests-discovery-l8z8t PodName:netshoot68f7k-6757486d56-7s2kx ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
  1843. 2020-08-25T18:28:06.4114671Z STEP: Validating that dig result is ""
  1844. 2020-08-25T18:28:06.4115274Z Aug 25 18:28:06.176: INFO: ExecWithOptions {Command:[dig +short nginx-demo.e2e-tests-discovery-l8z8t.svc.clusterset.local nginx-demo.e2e-tests-discovery-l8z8t.svc.supercluster.local] Namespace:e2e-tests-discovery-l8z8t PodName:netshoot68f7k-6757486d56-7s2kx ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
  1845. 2020-08-25T18:28:06.4115699Z STEP: Validating that dig result is "100.92.146.202\n100.92.146.202"
  1846. 2020-08-25T18:28:06.4116036Z STEP: Deleting namespace "e2e-tests-discovery-l8z8t" on cluster "cluster1"
  1847. 2020-08-25T18:28:06.4116367Z STEP: Deleting namespace "e2e-tests-discovery-l8z8t" on cluster "cluster2"
  1848. 2020-08-25T18:28:06.4116446Z
  1849. 2020-08-25T18:28:06.4116515Z
  1850. 2020-08-25T18:28:06.4117045Z [discovery] Test Headless Service Discovery Across Clusters when the number of active pods backing a service changes
  1851. 2020-08-25T18:28:06.4117464Z should only resolve the IPs from the active pods
  1852. 2020-08-25T18:28:06.4117972Z /go/src/github.com/submariner-io/lighthouse/test/e2e/discovery/headless_services.go:31
  1853. 2020-08-25T18:28:06.4118283Z STEP: Creating namespace objects with basename "discovery"
  1854. 2020-08-25T18:28:06.4118633Z STEP: Generated namespace "e2e-tests-discovery-qmqpm" in cluster "cluster1" to execute the tests in
  1855. 2020-08-25T18:28:06.4118970Z STEP: Creating namespace "e2e-tests-discovery-qmqpm" in cluster "cluster2"
  1856. 2020-08-25T18:28:06.4119373Z STEP: Creating an Nginx Deployment on on "cluster2"
  1857. 2020-08-25T18:28:11.3609238Z STEP: Setting Nginx deployment replicas to 3
  1858. 2020-08-25T18:28:11.3669093Z STEP: Creating a Nginx Headless Service on "cluster2"
  1859. 2020-08-25T18:28:11.3867379Z STEP: Creating serviceExport nginx-headless.e2e-tests-discovery-qmqpm on "cluster2"
  1860. 2020-08-25T18:28:11.4710979Z STEP: Retrieving ServiceExport nginx-headless.e2e-tests-discovery-qmqpm on "cluster2"
  1861. 2020-08-25T18:28:16.5055545Z STEP: Creating a Netshoot Deployment on "cluster1"
  1862. 2020-08-25T18:28:21.5668408Z STEP: Retrieving Endpoints for nginx-headless on "cluster2"
  1863. 2020-08-25T18:28:21.5704476Z STEP: Executing "dig +short nginx-headless.e2e-tests-discovery-qmqpm.svc.clusterset.local nginx-headless.e2e-tests-discovery-qmqpm.svc.supercluster.local" to verify IPs [10.242.0.6 10.242.128.5 10.242.128.6] for service "nginx-headless" "are" discoverable
  1864. 2020-08-25T18:28:21.5706177Z Aug 25 18:28:21.570: INFO: ExecWithOptions {Command:[dig +short nginx-headless.e2e-tests-discovery-qmqpm.svc.clusterset.local nginx-headless.e2e-tests-discovery-qmqpm.svc.supercluster.local] Namespace:e2e-tests-discovery-qmqpm PodName:netshootl4z5k-6757486d56-x7mxz ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
  1865. 2020-08-25T18:28:21.7801571Z STEP: Validating that dig result are "10.242.0.6\n10.242.128.5\n10.242.128.6\n10.242.0.6\n10.242.128.5\n10.242.128.6"
  1866. 2020-08-25T18:28:21.7802092Z STEP: Setting Nginx deployment replicas to 0
  1867. 2020-08-25T18:28:21.7913789Z STEP: Retrieving Endpoints for nginx-headless on "cluster2"
  1868. 2020-08-25T18:28:26.7976569Z STEP: Executing "dig +short nginx-headless.e2e-tests-discovery-qmqpm.svc.clusterset.local nginx-headless.e2e-tests-discovery-qmqpm.svc.supercluster.local" to verify IPs [] for service "nginx-headless" "are not" discoverable
  1869. 2020-08-25T18:28:26.7977774Z Aug 25 18:28:26.797: INFO: ExecWithOptions {Command:[dig +short nginx-headless.e2e-tests-discovery-qmqpm.svc.clusterset.local nginx-headless.e2e-tests-discovery-qmqpm.svc.supercluster.local] Namespace:e2e-tests-discovery-qmqpm PodName:netshootl4z5k-6757486d56-x7mxz ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
  1870. 2020-08-25T18:28:26.8749090Z STEP: Validating that dig result are not ""
  1871. 2020-08-25T18:28:26.8749636Z STEP: Setting Nginx deployment replicas to 2
  1872. 2020-08-25T18:28:26.8833522Z STEP: Retrieving Endpoints for nginx-headless on "cluster2"
  1873. 2020-08-25T18:28:31.8929384Z STEP: Executing "dig +short nginx-headless.e2e-tests-discovery-qmqpm.svc.clusterset.local nginx-headless.e2e-tests-discovery-qmqpm.svc.supercluster.local" to verify IPs [10.242.0.5 10.242.128.2] for service "nginx-headless" "are" discoverable
  1874. 2020-08-25T18:28:31.8930345Z Aug 25 18:28:31.892: INFO: ExecWithOptions {Command:[dig +short nginx-headless.e2e-tests-discovery-qmqpm.svc.clusterset.local nginx-headless.e2e-tests-discovery-qmqpm.svc.supercluster.local] Namespace:e2e-tests-discovery-qmqpm PodName:netshootl4z5k-6757486d56-x7mxz ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
  1875. 2020-08-25T18:28:31.9792658Z STEP: Validating that dig result are "10.242.0.5\n10.242.128.2\n10.242.0.5\n10.242.128.2"
  1876. 2020-08-25T18:28:31.9805150Z STEP: Deleting namespace "e2e-tests-discovery-qmqpm" on cluster "cluster1"
  1877. 2020-08-25T18:28:31.9918005Z STEP: Deleting namespace "e2e-tests-discovery-qmqpm" on cluster "cluster2"
  1878. 2020-08-25T18:28:32.0113800Z •
  1879. 2020-08-25T18:28:32.0114300Z ------------------------------
  1880. 2020-08-25T18:28:32.0114767Z STEP: Creating namespace objects with basename "discovery"
  1881. 2020-08-25T18:28:32.0115183Z STEP: Generated namespace "e2e-tests-discovery-qmqpm" in cluster "cluster1" to execute the tests in
  1882. 2020-08-25T18:28:32.0128471Z STEP: Creating namespace "e2e-tests-discovery-qmqpm" in cluster "cluster2"
  1883. 2020-08-25T18:28:32.0129057Z STEP: Creating an Nginx Deployment on on "cluster2"
  1884. 2020-08-25T18:28:32.0129390Z STEP: Setting Nginx deployment replicas to 3
  1885. 2020-08-25T18:28:32.0129685Z STEP: Creating a Nginx Headless Service on "cluster2"
  1886. 2020-08-25T18:28:32.0130010Z STEP: Creating serviceExport nginx-headless.e2e-tests-discovery-qmqpm on "cluster2"
  1887. 2020-08-25T18:28:32.0130347Z STEP: Retrieving ServiceExport nginx-headless.e2e-tests-discovery-qmqpm on "cluster2"
  1888. 2020-08-25T18:28:32.0130642Z STEP: Creating a Netshoot Deployment on "cluster1"
  1889. 2020-08-25T18:28:32.0130937Z STEP: Retrieving Endpoints for nginx-headless on "cluster2"
  1890. 2020-08-25T18:28:32.0131463Z STEP: Executing "dig +short nginx-headless.e2e-tests-discovery-qmqpm.svc.clusterset.local nginx-headless.e2e-tests-discovery-qmqpm.svc.supercluster.local" to verify IPs [10.242.0.6 10.242.128.5 10.242.128.6] for service "nginx-headless" "are" discoverable
  1891. 2020-08-25T18:28:32.0132167Z Aug 25 18:28:21.570: INFO: ExecWithOptions {Command:[dig +short nginx-headless.e2e-tests-discovery-qmqpm.svc.clusterset.local nginx-headless.e2e-tests-discovery-qmqpm.svc.supercluster.local] Namespace:e2e-tests-discovery-qmqpm PodName:netshootl4z5k-6757486d56-x7mxz ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
  1892. 2020-08-25T18:28:32.0132634Z STEP: Validating that dig result are "10.242.0.6\n10.242.128.5\n10.242.128.6\n10.242.0.6\n10.242.128.5\n10.242.128.6"
  1893. 2020-08-25T18:28:32.0132938Z STEP: Setting Nginx deployment replicas to 0
  1894. 2020-08-25T18:28:32.0133238Z STEP: Retrieving Endpoints for nginx-headless on "cluster2"
  1895. 2020-08-25T18:28:32.0133984Z STEP: Executing "dig +short nginx-headless.e2e-tests-discovery-qmqpm.svc.clusterset.local nginx-headless.e2e-tests-discovery-qmqpm.svc.supercluster.local" to verify IPs [] for service "nginx-headless" "are not" discoverable
  1896. 2020-08-25T18:28:32.0134864Z Aug 25 18:28:26.797: INFO: ExecWithOptions {Command:[dig +short nginx-headless.e2e-tests-discovery-qmqpm.svc.clusterset.local nginx-headless.e2e-tests-discovery-qmqpm.svc.supercluster.local] Namespace:e2e-tests-discovery-qmqpm PodName:netshootl4z5k-6757486d56-x7mxz ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
  1897. 2020-08-25T18:28:32.0135235Z STEP: Validating that dig result are not ""
  1898. 2020-08-25T18:28:32.0135530Z STEP: Setting Nginx deployment replicas to 2
  1899. 2020-08-25T18:28:32.0135839Z STEP: Retrieving Endpoints for nginx-headless on "cluster2"
  1900. 2020-08-25T18:28:32.0136372Z STEP: Executing "dig +short nginx-headless.e2e-tests-discovery-qmqpm.svc.clusterset.local nginx-headless.e2e-tests-discovery-qmqpm.svc.supercluster.local" to verify IPs [10.242.0.5 10.242.128.2] for service "nginx-headless" "are" discoverable
  1901. 2020-08-25T18:28:32.0137083Z Aug 25 18:28:31.892: INFO: ExecWithOptions {Command:[dig +short nginx-headless.e2e-tests-discovery-qmqpm.svc.clusterset.local nginx-headless.e2e-tests-discovery-qmqpm.svc.supercluster.local] Namespace:e2e-tests-discovery-qmqpm PodName:netshootl4z5k-6757486d56-x7mxz ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
  1902. 2020-08-25T18:28:32.0137518Z STEP: Validating that dig result are "10.242.0.5\n10.242.128.2\n10.242.0.5\n10.242.128.2"
  1903. 2020-08-25T18:28:32.0137857Z STEP: Deleting namespace "e2e-tests-discovery-qmqpm" on cluster "cluster1"
  1904. 2020-08-25T18:28:32.0138191Z STEP: Deleting namespace "e2e-tests-discovery-qmqpm" on cluster "cluster2"
  1905. 2020-08-25T18:28:32.0138289Z
  1906. 2020-08-25T18:28:32.0138404Z
  1907. 2020-08-25T18:28:32.0138967Z [discovery] Test Service Discovery Across Clusters when a pod tries to resolve a service which is present locally and in a remote cluster
  1908. 2020-08-25T18:28:32.0139276Z should resolve the local service
  1909. 2020-08-25T18:28:32.0139609Z /go/src/github.com/submariner-io/lighthouse/test/e2e/discovery/service_discovery.go:33
  1910. 2020-08-25T18:28:32.0140003Z STEP: Creating namespace objects with basename "discovery"
  1911. 2020-08-25T18:28:32.0222654Z STEP: Generated namespace "e2e-tests-discovery-lbtb4" in cluster "cluster1" to execute the tests in
  1912. 2020-08-25T18:28:32.0223036Z STEP: Creating namespace "e2e-tests-discovery-lbtb4" in cluster "cluster2"
  1913. 2020-08-25T18:28:32.0452911Z STEP: Creating an Nginx Deployment on "cluster1"
  1914. 2020-08-25T18:28:37.1111410Z STEP: Creating a Nginx Service on "cluster1"
  1915. 2020-08-25T18:28:37.1423576Z STEP: Creating an Nginx Deployment on "cluster2"
  1916. 2020-08-25T18:28:42.1739493Z STEP: Creating a Nginx Service on "cluster2"
  1917. 2020-08-25T18:28:42.1793986Z STEP: Creating serviceExport nginx-demo.e2e-tests-discovery-lbtb4 on "cluster2"
  1918. 2020-08-25T18:28:42.2130475Z STEP: Retrieving ServiceExport nginx-demo.e2e-tests-discovery-lbtb4 on "cluster2"
  1919. 2020-08-25T18:28:47.2251788Z STEP: Creating a Netshoot Deployment on "cluster1"
  1920. 2020-08-25T18:28:52.2428056Z Aug 25 18:28:52.242: INFO: ExecWithOptions {Command:[cat /etc/resolv.conf] Namespace:e2e-tests-discovery-lbtb4 PodName:netshootqn2cx-6757486d56-l5tpj ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
  1921. 2020-08-25T18:28:52.3182043Z STEP: Executing "dig +short nginx-demo.e2e-tests-discovery-lbtb4.svc.cluster1.local" to verify IP "100.91.161.16" for service "nginx-demo" "is" discoverable
  1922. 2020-08-25T18:28:52.3191898Z Aug 25 18:28:52.317: INFO: ExecWithOptions {Command:[dig +short nginx-demo.e2e-tests-discovery-lbtb4.svc.cluster1.local] Namespace:e2e-tests-discovery-lbtb4 PodName:netshootqn2cx-6757486d56-l5tpj ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
  1923. 2020-08-25T18:28:52.3868357Z STEP: Validating that dig result is "100.91.161.16"
  1924. 2020-08-25T18:28:52.3868766Z STEP: Deleting service "nginx-demo" on "cluster1"
  1925. 2020-08-25T18:28:52.4338204Z STEP: Retrieving service nginx-demo.e2e-tests-discovery-lbtb4 on "cluster2"
  1926. 2020-08-25T18:28:52.4380000Z STEP: Retrieving ServiceImport for nginx-demo-e2e-tests-discovery-lbtb4- on "cluster1"
  1927. 2020-08-25T18:28:52.4466459Z STEP: Executing "dig +short nginx-demo.e2e-tests-discovery-lbtb4.svc.clusterset.local nginx-demo.e2e-tests-discovery-lbtb4.svc.supercluster.local" to verify IP "100.92.26.23" for service "nginx-demo" "is" discoverable
  1928. 2020-08-25T18:28:52.4467202Z Aug 25 18:28:52.446: INFO: ExecWithOptions {Command:[dig +short nginx-demo.e2e-tests-discovery-lbtb4.svc.clusterset.local nginx-demo.e2e-tests-discovery-lbtb4.svc.supercluster.local] Namespace:e2e-tests-discovery-lbtb4 PodName:netshootqn2cx-6757486d56-l5tpj ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
  1929. 2020-08-25T18:28:52.6480993Z STEP: Validating that dig result is "100.92.26.23\n100.92.26.23"
  1930. 2020-08-25T18:28:52.6489328Z STEP: Deleting serviceExport nginx-demo.e2e-tests-discovery-lbtb4 on "cluster2"
  1931. 2020-08-25T18:28:57.6692239Z STEP: Deleting service "nginx-demo" on "cluster2"
  1932. 2020-08-25T18:28:57.7223146Z STEP: Executing "dig +short nginx-demo.e2e-tests-discovery-lbtb4.svc.clusterset.local nginx-demo.e2e-tests-discovery-lbtb4.svc.supercluster.local" to verify IP "100.92.26.23" for service "nginx-demo" "is not" discoverable
  1933. 2020-08-25T18:28:57.7226131Z Aug 25 18:28:57.721: INFO: ExecWithOptions {Command:[dig +short nginx-demo.e2e-tests-discovery-lbtb4.svc.clusterset.local nginx-demo.e2e-tests-discovery-lbtb4.svc.supercluster.local] Namespace:e2e-tests-discovery-lbtb4 PodName:netshootqn2cx-6757486d56-l5tpj ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
  1934. 2020-08-25T18:29:09.8379749Z STEP: Validating that dig result is not ""
  1935. 2020-08-25T18:29:09.8380603Z STEP: Deleting namespace "e2e-tests-discovery-lbtb4" on cluster "cluster1"
  1936. 2020-08-25T18:29:09.8453124Z STEP: Deleting namespace "e2e-tests-discovery-lbtb4" on cluster "cluster2"
  1937. 2020-08-25T18:29:09.8567078Z •
  1938. 2020-08-25T18:29:09.8568820Z ------------------------------
  1939. 2020-08-25T18:29:09.8570494Z STEP: Creating namespace objects with basename "discovery"
  1940. 2020-08-25T18:29:09.8571037Z STEP: Generated namespace "e2e-tests-discovery-lbtb4" in cluster "cluster1" to execute the tests in
  1941. 2020-08-25T18:29:09.8571368Z STEP: Creating namespace "e2e-tests-discovery-lbtb4" in cluster "cluster2"
  1942. 2020-08-25T18:29:09.8571667Z STEP: Creating an Nginx Deployment on "cluster1"
  1943. 2020-08-25T18:29:09.8572153Z STEP: Creating a Nginx Service on "cluster1"
  1944. 2020-08-25T18:29:09.8572605Z STEP: Creating an Nginx Deployment on "cluster2"
  1945. 2020-08-25T18:29:09.8572882Z STEP: Creating a Nginx Service on "cluster2"
  1946. 2020-08-25T18:29:09.8573220Z STEP: Creating serviceExport nginx-demo.e2e-tests-discovery-lbtb4 on "cluster2"
  1947. 2020-08-25T18:29:09.8573537Z STEP: Retrieving ServiceExport nginx-demo.e2e-tests-discovery-lbtb4 on "cluster2"
  1948. 2020-08-25T18:29:09.8573840Z STEP: Creating a Netshoot Deployment on "cluster1"
  1949. 2020-08-25T18:29:09.8574360Z Aug 25 18:28:52.242: INFO: ExecWithOptions {Command:[cat /etc/resolv.conf] Namespace:e2e-tests-discovery-lbtb4 PodName:netshootqn2cx-6757486d56-l5tpj ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
  1950. 2020-08-25T18:29:09.8574819Z STEP: Executing "dig +short nginx-demo.e2e-tests-discovery-lbtb4.svc.cluster1.local" to verify IP "100.91.161.16" for service "nginx-demo" "is" discoverable
  1951. 2020-08-25T18:29:09.8575406Z Aug 25 18:28:52.317: INFO: ExecWithOptions {Command:[dig +short nginx-demo.e2e-tests-discovery-lbtb4.svc.cluster1.local] Namespace:e2e-tests-discovery-lbtb4 PodName:netshootqn2cx-6757486d56-l5tpj ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
  1952. 2020-08-25T18:29:09.8575962Z STEP: Validating that dig result is "100.91.161.16"
  1953. 2020-08-25T18:29:09.8576255Z STEP: Deleting service "nginx-demo" on "cluster1"
  1954. 2020-08-25T18:29:09.8576585Z STEP: Retrieving service nginx-demo.e2e-tests-discovery-lbtb4 on "cluster2"
  1955. 2020-08-25T18:29:09.8576933Z STEP: Retrieving ServiceImport for nginx-demo-e2e-tests-discovery-lbtb4- on "cluster1"
  1956. 2020-08-25T18:29:09.8577418Z STEP: Executing "dig +short nginx-demo.e2e-tests-discovery-lbtb4.svc.clusterset.local nginx-demo.e2e-tests-discovery-lbtb4.svc.supercluster.local" to verify IP "100.92.26.23" for service "nginx-demo" "is" discoverable
  1957. 2020-08-25T18:29:09.8578068Z Aug 25 18:28:52.446: INFO: ExecWithOptions {Command:[dig +short nginx-demo.e2e-tests-discovery-lbtb4.svc.clusterset.local nginx-demo.e2e-tests-discovery-lbtb4.svc.supercluster.local] Namespace:e2e-tests-discovery-lbtb4 PodName:netshootqn2cx-6757486d56-l5tpj ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
  1958. 2020-08-25T18:29:09.8578462Z STEP: Validating that dig result is "100.92.26.23\n100.92.26.23"
  1959. 2020-08-25T18:29:09.8578799Z STEP: Deleting serviceExport nginx-demo.e2e-tests-discovery-lbtb4 on "cluster2"
  1960. 2020-08-25T18:29:09.8579104Z STEP: Deleting service "nginx-demo" on "cluster2"
  1961. 2020-08-25T18:29:09.8579579Z STEP: Executing "dig +short nginx-demo.e2e-tests-discovery-lbtb4.svc.clusterset.local nginx-demo.e2e-tests-discovery-lbtb4.svc.supercluster.local" to verify IP "100.92.26.23" for service "nginx-demo" "is not" discoverable
  1962. 2020-08-25T18:29:09.8580222Z Aug 25 18:28:57.721: INFO: ExecWithOptions {Command:[dig +short nginx-demo.e2e-tests-discovery-lbtb4.svc.clusterset.local nginx-demo.e2e-tests-discovery-lbtb4.svc.supercluster.local] Namespace:e2e-tests-discovery-lbtb4 PodName:netshootqn2cx-6757486d56-l5tpj ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
  1963. 2020-08-25T18:29:09.8580577Z STEP: Validating that dig result is not ""
  1964. 2020-08-25T18:29:09.8580891Z STEP: Deleting namespace "e2e-tests-discovery-lbtb4" on cluster "cluster1"
  1965. 2020-08-25T18:29:09.8581297Z STEP: Deleting namespace "e2e-tests-discovery-lbtb4" on cluster "cluster2"
  1966. 2020-08-25T18:29:09.8581400Z
  1967. 2020-08-25T18:29:09.8581454Z
  1968. 2020-08-25T18:29:09.8581834Z [discovery] Test Service Discovery Across Clusters when a pod tries to resolve a service in a remote cluster
  1969. 2020-08-25T18:29:09.8582154Z should be able to discover the remote service successfully
  1970. 2020-08-25T18:29:09.8582492Z /go/src/github.com/submariner-io/lighthouse/test/e2e/discovery/service_discovery.go:27
  1971. 2020-08-25T18:29:09.8582979Z STEP: Creating namespace objects with basename "discovery"
  1972. 2020-08-25T18:29:09.8682234Z STEP: Generated namespace "e2e-tests-discovery-fmjwl" in cluster "cluster1" to execute the tests in
  1973. 2020-08-25T18:29:09.8682861Z STEP: Creating namespace "e2e-tests-discovery-fmjwl" in cluster "cluster2"
  1974. 2020-08-25T18:29:09.8740961Z STEP: Creating an Nginx Deployment on on "cluster2"
  1975. 2020-08-25T18:29:14.9506732Z STEP: Creating a Nginx Service on "cluster2"
  1976. 2020-08-25T18:29:14.9608555Z STEP: Creating serviceExport nginx-demo.e2e-tests-discovery-fmjwl on "cluster2"
  1977. 2020-08-25T18:29:15.0051794Z STEP: Retrieving ServiceExport nginx-demo.e2e-tests-discovery-fmjwl on "cluster2"
  1978. 2020-08-25T18:29:20.0452051Z STEP: Creating a Netshoot Deployment on "cluster1"
  1979. 2020-08-25T18:29:25.0627780Z STEP: Retrieving service nginx-demo.e2e-tests-discovery-fmjwl on "cluster2"
  1980. 2020-08-25T18:29:25.0647887Z STEP: Retrieving ServiceImport for nginx-demo-e2e-tests-discovery-fmjwl- on "cluster1"
  1981. 2020-08-25T18:29:25.0665234Z STEP: Executing "dig +short nginx-demo.e2e-tests-discovery-fmjwl.svc.clusterset.local nginx-demo.e2e-tests-discovery-fmjwl.svc.supercluster.local" to verify IP "100.92.119.87" for service "nginx-demo" "is" discoverable
  1982. 2020-08-25T18:29:25.0670740Z Aug 25 18:29:25.066: INFO: ExecWithOptions {Command:[dig +short nginx-demo.e2e-tests-discovery-fmjwl.svc.clusterset.local nginx-demo.e2e-tests-discovery-fmjwl.svc.supercluster.local] Namespace:e2e-tests-discovery-fmjwl PodName:netshootwslwk-6757486d56-85p46 ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
  1983. 2020-08-25T18:29:25.1441346Z STEP: Validating that dig result is "100.92.119.87\n100.92.119.87"
  1984. 2020-08-25T18:29:25.1446108Z STEP: Deleting serviceExport nginx-demo.e2e-tests-discovery-fmjwl on "cluster2"
  1985. 2020-08-25T18:29:30.1561557Z STEP: Deleting service "nginx-demo" on "cluster2"
  1986. 2020-08-25T18:29:30.2161929Z STEP: Executing "dig +short nginx-demo.e2e-tests-discovery-fmjwl.svc.clusterset.local nginx-demo.e2e-tests-discovery-fmjwl.svc.supercluster.local" to verify IP "100.92.119.87" for service "nginx-demo" "is not" discoverable
  1987. 2020-08-25T18:29:30.2166786Z Aug 25 18:29:30.215: INFO: ExecWithOptions {Command:[dig +short nginx-demo.e2e-tests-discovery-fmjwl.svc.clusterset.local nginx-demo.e2e-tests-discovery-fmjwl.svc.supercluster.local] Namespace:e2e-tests-discovery-fmjwl PodName:netshootwslwk-6757486d56-85p46 ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
  1988. 2020-08-25T18:29:42.3233765Z STEP: Validating that dig result is not ""
  1989. 2020-08-25T18:29:42.3238171Z STEP: Deleting namespace "e2e-tests-discovery-fmjwl" on cluster "cluster1"
  1990. 2020-08-25T18:29:42.3312666Z STEP: Deleting namespace "e2e-tests-discovery-fmjwl" on cluster "cluster2"
  1991. 2020-08-25T18:29:42.3375243Z •
  1992. 2020-08-25T18:29:42.3376475Z ------------------------------
  1993. 2020-08-25T18:29:42.3376834Z STEP: Creating namespace objects with basename "discovery"
  1994. 2020-08-25T18:29:42.3377537Z STEP: Generated namespace "e2e-tests-discovery-fmjwl" in cluster "cluster1" to execute the tests in
  1995. 2020-08-25T18:29:42.3381044Z STEP: Creating namespace "e2e-tests-discovery-fmjwl" in cluster "cluster2"
  1996. 2020-08-25T18:29:42.3381636Z STEP: Creating an Nginx Deployment on on "cluster2"
  1997. 2020-08-25T18:29:42.3383109Z STEP: Creating a Nginx Service on "cluster2"
  1998. 2020-08-25T18:29:42.3385378Z STEP: Creating serviceExport nginx-demo.e2e-tests-discovery-fmjwl on "cluster2"
  1999. 2020-08-25T18:29:42.3385789Z STEP: Retrieving ServiceExport nginx-demo.e2e-tests-discovery-fmjwl on "cluster2"
  2000. 2020-08-25T18:29:42.3386223Z STEP: Creating a Netshoot Deployment on "cluster1"
  2001. 2020-08-25T18:29:42.3386552Z STEP: Retrieving service nginx-demo.e2e-tests-discovery-fmjwl on "cluster2"
  2002. 2020-08-25T18:29:42.3387169Z STEP: Retrieving ServiceImport for nginx-demo-e2e-tests-discovery-fmjwl- on "cluster1"
  2003. 2020-08-25T18:29:42.3388679Z STEP: Executing "dig +short nginx-demo.e2e-tests-discovery-fmjwl.svc.clusterset.local nginx-demo.e2e-tests-discovery-fmjwl.svc.supercluster.local" to verify IP "100.92.119.87" for service "nginx-demo" "is" discoverable
  2004. 2020-08-25T18:29:42.3390336Z Aug 25 18:29:25.066: INFO: ExecWithOptions {Command:[dig +short nginx-demo.e2e-tests-discovery-fmjwl.svc.clusterset.local nginx-demo.e2e-tests-discovery-fmjwl.svc.supercluster.local] Namespace:e2e-tests-discovery-fmjwl PodName:netshootwslwk-6757486d56-85p46 ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
  2005. 2020-08-25T18:29:42.3391973Z STEP: Validating that dig result is "100.92.119.87\n100.92.119.87"
  2006. 2020-08-25T18:29:42.3396356Z STEP: Deleting serviceExport nginx-demo.e2e-tests-discovery-fmjwl on "cluster2"
  2007. 2020-08-25T18:29:42.3396954Z STEP: Deleting service "nginx-demo" on "cluster2"
  2008. 2020-08-25T18:29:42.3397432Z STEP: Executing "dig +short nginx-demo.e2e-tests-discovery-fmjwl.svc.clusterset.local nginx-demo.e2e-tests-discovery-fmjwl.svc.supercluster.local" to verify IP "100.92.119.87" for service "nginx-demo" "is not" discoverable
  2009. 2020-08-25T18:29:42.3398359Z Aug 25 18:29:30.215: INFO: ExecWithOptions {Command:[dig +short nginx-demo.e2e-tests-discovery-fmjwl.svc.clusterset.local nginx-demo.e2e-tests-discovery-fmjwl.svc.supercluster.local] Namespace:e2e-tests-discovery-fmjwl PodName:netshootwslwk-6757486d56-85p46 ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
  2010. 2020-08-25T18:29:42.3398702Z STEP: Validating that dig result is not ""
  2011. 2020-08-25T18:29:42.3399012Z STEP: Deleting namespace "e2e-tests-discovery-fmjwl" on cluster "cluster1"
  2012. 2020-08-25T18:29:42.3399330Z STEP: Deleting namespace "e2e-tests-discovery-fmjwl" on cluster "cluster2"
  2013. 2020-08-25T18:29:42.3399408Z
  2014. 2020-08-25T18:29:42.3399522Z
  2015. 2020-08-25T18:29:42.3399878Z [discovery] Test Service Discovery Across Clusters when service export is created before the service
  2016. 2020-08-25T18:29:42.3400142Z should resolve the service
  2017. 2020-08-25T18:29:42.3400472Z /go/src/github.com/submariner-io/lighthouse/test/e2e/discovery/service_discovery.go:39
  2018. 2020-08-25T18:29:42.3400774Z STEP: Creating namespace objects with basename "discovery"
  2019. 2020-08-25T18:29:42.3441075Z STEP: Generated namespace "e2e-tests-discovery-kdqft" in cluster "cluster1" to execute the tests in
  2020. 2020-08-25T18:29:42.3443151Z STEP: Creating namespace "e2e-tests-discovery-kdqft" in cluster "cluster2"
  2021. 2020-08-25T18:29:42.3505765Z STEP: Creating an Nginx ServiceExport on "cluster2"
  2022. 2020-08-25T18:29:42.3506111Z STEP: Creating serviceExport nginx-demo.e2e-tests-discovery-kdqft on "cluster2"
  2023. 2020-08-25T18:29:42.3926984Z STEP: Creating an Nginx Deployment on on "cluster2"
  2024. 2020-08-25T18:29:47.4292143Z STEP: Creating a Nginx Service on "cluster2"
  2025. 2020-08-25T18:29:47.4987789Z STEP: Creating a Netshoot Deployment on "cluster1"
  2026. 2020-08-25T18:29:52.5854682Z STEP: Retrieving service nginx-demo.e2e-tests-discovery-kdqft on "cluster2"
  2027. 2020-08-25T18:29:52.5899676Z STEP: Retrieving ServiceImport for nginx-demo-e2e-tests-discovery-kdqft- on "cluster1"
  2028. 2020-08-25T18:29:52.5929680Z STEP: Executing "dig +short nginx-demo.e2e-tests-discovery-kdqft.svc.clusterset.local nginx-demo.e2e-tests-discovery-kdqft.svc.supercluster.local" to verify IP "100.92.210.107" for service "nginx-demo" "is" discoverable
  2029. 2020-08-25T18:29:52.5930762Z Aug 25 18:29:52.592: INFO: ExecWithOptions {Command:[dig +short nginx-demo.e2e-tests-discovery-kdqft.svc.clusterset.local nginx-demo.e2e-tests-discovery-kdqft.svc.supercluster.local] Namespace:e2e-tests-discovery-kdqft PodName:netshoot5g7r5-6757486d56-7p68d ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
  2030. 2020-08-25T18:29:52.6620940Z STEP: Validating that dig result is "100.92.210.107\n100.92.210.107"
  2031. 2020-08-25T18:29:52.6621861Z STEP: Deleting serviceExport nginx-demo.e2e-tests-discovery-kdqft on "cluster2"
  2032. 2020-08-25T18:29:57.6751785Z STEP: Executing "dig +short nginx-demo.e2e-tests-discovery-kdqft.svc.clusterset.local nginx-demo.e2e-tests-discovery-kdqft.svc.supercluster.local" to verify IP "100.92.210.107" for service "nginx-demo" "is not" discoverable
  2033. 2020-08-25T18:29:57.6752602Z Aug 25 18:29:57.674: INFO: ExecWithOptions {Command:[dig +short nginx-demo.e2e-tests-discovery-kdqft.svc.clusterset.local nginx-demo.e2e-tests-discovery-kdqft.svc.supercluster.local] Namespace:e2e-tests-discovery-kdqft PodName:netshoot5g7r5-6757486d56-7p68d ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
  2034. 2020-08-25T18:30:09.7367129Z STEP: Validating that dig result is not ""
  2035. 2020-08-25T18:30:09.7367491Z STEP: Deleting namespace "e2e-tests-discovery-kdqft" on cluster "cluster1"
  2036. 2020-08-25T18:30:09.7410679Z STEP: Deleting namespace "e2e-tests-discovery-kdqft" on cluster "cluster2"
  2037. 2020-08-25T18:30:09.7475695Z •
  2038. 2020-08-25T18:30:09.7476002Z ------------------------------
  2039. 2020-08-25T18:30:09.7485494Z STEP: Creating namespace objects with basename "discovery"
  2040. 2020-08-25T18:30:09.7486351Z STEP: Generated namespace "e2e-tests-discovery-kdqft" in cluster "cluster1" to execute the tests in
  2041. 2020-08-25T18:30:09.7486910Z STEP: Creating namespace "e2e-tests-discovery-kdqft" in cluster "cluster2"
  2042. 2020-08-25T18:30:09.7487216Z STEP: Creating an Nginx ServiceExport on "cluster2"
  2043. 2020-08-25T18:30:09.7487522Z STEP: Creating serviceExport nginx-demo.e2e-tests-discovery-kdqft on "cluster2"
  2044. 2020-08-25T18:30:09.7487812Z STEP: Creating an Nginx Deployment on on "cluster2"
  2045. 2020-08-25T18:30:09.7488082Z STEP: Creating a Nginx Service on "cluster2"
  2046. 2020-08-25T18:30:09.7488358Z STEP: Creating a Netshoot Deployment on "cluster1"
  2047. 2020-08-25T18:30:09.7488648Z STEP: Retrieving service nginx-demo.e2e-tests-discovery-kdqft on "cluster2"
  2048. 2020-08-25T18:30:09.7488979Z STEP: Retrieving ServiceImport for nginx-demo-e2e-tests-discovery-kdqft- on "cluster1"
  2049. 2020-08-25T18:30:09.7489464Z STEP: Executing "dig +short nginx-demo.e2e-tests-discovery-kdqft.svc.clusterset.local nginx-demo.e2e-tests-discovery-kdqft.svc.supercluster.local" to verify IP "100.92.210.107" for service "nginx-demo" "is" discoverable
  2050. 2020-08-25T18:30:09.7490118Z Aug 25 18:29:52.592: INFO: ExecWithOptions {Command:[dig +short nginx-demo.e2e-tests-discovery-kdqft.svc.clusterset.local nginx-demo.e2e-tests-discovery-kdqft.svc.supercluster.local] Namespace:e2e-tests-discovery-kdqft PodName:netshoot5g7r5-6757486d56-7p68d ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
  2051. 2020-08-25T18:30:09.7490520Z STEP: Validating that dig result is "100.92.210.107\n100.92.210.107"
  2052. 2020-08-25T18:30:09.7490844Z STEP: Deleting serviceExport nginx-demo.e2e-tests-discovery-kdqft on "cluster2"
  2053. 2020-08-25T18:30:09.7491319Z STEP: Executing "dig +short nginx-demo.e2e-tests-discovery-kdqft.svc.clusterset.local nginx-demo.e2e-tests-discovery-kdqft.svc.supercluster.local" to verify IP "100.92.210.107" for service "nginx-demo" "is not" discoverable
  2054. 2020-08-25T18:30:09.7492152Z Aug 25 18:29:57.674: INFO: ExecWithOptions {Command:[dig +short nginx-demo.e2e-tests-discovery-kdqft.svc.clusterset.local nginx-demo.e2e-tests-discovery-kdqft.svc.supercluster.local] Namespace:e2e-tests-discovery-kdqft PodName:netshoot5g7r5-6757486d56-7p68d ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
  2055. 2020-08-25T18:30:09.7492549Z STEP: Validating that dig result is not ""
  2056. 2020-08-25T18:30:09.7492853Z STEP: Deleting namespace "e2e-tests-discovery-kdqft" on cluster "cluster1"
  2057. 2020-08-25T18:30:09.7493167Z STEP: Deleting namespace "e2e-tests-discovery-kdqft" on cluster "cluster2"
  2058. 2020-08-25T18:30:09.7493242Z
  2059. 2020-08-25T18:30:09.7493309Z
  2060. 2020-08-25T18:30:09.7493679Z [discovery] Test Headless Service Discovery Across Clusters when a pod tries to resolve a headless service in a remote cluster
  2061. 2020-08-25T18:30:09.7493985Z should resolve the backing pod IPs from the remote cluster
  2062. 2020-08-25T18:30:09.7494312Z /go/src/github.com/submariner-io/lighthouse/test/e2e/discovery/headless_services.go:19
  2063. 2020-08-25T18:30:09.7494611Z STEP: Creating namespace objects with basename "discovery"
  2064. 2020-08-25T18:30:09.7596512Z STEP: Generated namespace "e2e-tests-discovery-pk6g6" in cluster "cluster1" to execute the tests in
  2065. 2020-08-25T18:30:09.7596915Z STEP: Creating namespace "e2e-tests-discovery-pk6g6" in cluster "cluster2"
  2066. 2020-08-25T18:30:09.7881231Z STEP: Creating an Nginx Deployment on on "cluster2"
  2067. 2020-08-25T18:30:14.8476458Z STEP: Creating a Nginx Headless Service on "cluster2"
  2068. 2020-08-25T18:30:14.8614295Z STEP: Creating serviceExport nginx-headless.e2e-tests-discovery-pk6g6 on "cluster2"
  2069. 2020-08-25T18:30:14.8746353Z STEP: Retrieving ServiceExport nginx-headless.e2e-tests-discovery-pk6g6 on "cluster2"
  2070. 2020-08-25T18:30:19.8869425Z STEP: Creating a Netshoot Deployment on "cluster1"
  2071. 2020-08-25T18:33:34.9233552Z STEP: Deleting namespace "e2e-tests-discovery-pk6g6" on cluster "cluster1"
  2072. 2020-08-25T18:33:34.9318373Z STEP: Deleting namespace "e2e-tests-discovery-pk6g6" on cluster "cluster2"
  2073. 2020-08-25T18:33:34.9486217Z
  2074. 2020-08-25T18:33:34.9493589Z • Failure [205.201 seconds]
  2075. 2020-08-25T18:33:34.9493820Z [discovery] Test Headless Service Discovery Across Clusters
  2076. 2020-08-25T18:33:34.9494194Z /go/src/github.com/submariner-io/lighthouse/test/e2e/discovery/headless_services.go:13
  2077. 2020-08-25T18:33:34.9494353Z when a pod tries to resolve a headless service in a remote cluster
  2078. 2020-08-25T18:33:34.9494717Z /go/src/github.com/submariner-io/lighthouse/test/e2e/discovery/headless_services.go:18
  2079. 2020-08-25T18:33:34.9495059Z should resolve the backing pod IPs from the remote cluster [It]
  2080. 2020-08-25T18:33:34.9495417Z /go/src/github.com/submariner-io/lighthouse/test/e2e/discovery/headless_services.go:19
  2081. 2020-08-25T18:33:34.9495549Z
  2082. 2020-08-25T18:33:34.9495907Z Failed to find pods for app netshoot. Actual pod count 2 does not match the expected pod count 1
  2083. 2020-08-25T18:33:34.9496034Z Unexpected error:
  2084. 2020-08-25T18:33:34.9496173Z <*errors.errorString | 0xc0002d83a0>: {
  2085. 2020-08-25T18:33:34.9496304Z s: "timed out waiting for the condition",
  2086. 2020-08-25T18:33:34.9496583Z }
  2087. 2020-08-25T18:33:34.9496700Z timed out waiting for the condition
  2088. 2020-08-25T18:33:34.9496944Z occurred
  2089. 2020-08-25T18:33:34.9497001Z
  2090. 2020-08-25T18:33:34.9497541Z /go/src/github.com/submariner-io/lighthouse/vendor/github.com/submariner-io/shipyard/test/e2e/framework/framework.go:355
  2091. 2020-08-25T18:33:34.9497643Z
  2092. 2020-08-25T18:33:34.9497899Z Full Stack Trace
  2093. 2020-08-25T18:33:34.9498293Z github.com/submariner-io/shipyard/test/e2e/framework.AwaitUntil(0xc00012caa0, 0x1a, 0xc00066f148, 0xc00066f138, 0x8, 0xc00012caa0)
  2094. 2020-08-25T18:33:34.9499041Z /go/src/github.com/submariner-io/lighthouse/vendor/github.com/submariner-io/shipyard/test/e2e/framework/framework.go:355 +0x1c6
  2095. 2020-08-25T18:33:34.9499724Z github.com/submariner-io/shipyard/test/e2e/framework.(*Framework).AwaitPodsByAppLabel(0xc00029d8b0, 0x0, 0x14d9426, 0x8, 0xc000041020, 0x19, 0x1, 0x3684764394e1a201)
  2096. 2020-08-25T18:33:34.9500193Z /go/src/github.com/submariner-io/lighthouse/vendor/github.com/submariner-io/shipyard/test/e2e/framework/pods.go:14 +0xf6
  2097. 2020-08-25T18:33:34.9500569Z github.com/submariner-io/shipyard/test/e2e/framework.create(0xc00029d8b0, 0x0, 0xc0001dbb00, 0x3)
  2098. 2020-08-25T18:33:34.9500971Z /go/src/github.com/submariner-io/lighthouse/vendor/github.com/submariner-io/shipyard/test/e2e/framework/deployments.go:117 +0x1ac
  2099. 2020-08-25T18:33:34.9502268Z github.com/submariner-io/shipyard/test/e2e/framework.(*Framework).NewNetShootDeployment(0xc00029d8b0, 0x0, 0xc0001bd340)
  2100. 2020-08-25T18:33:34.9502718Z /go/src/github.com/submariner-io/lighthouse/vendor/github.com/submariner-io/shipyard/test/e2e/framework/deployments.go:63 +0x2ef
  2101. 2020-08-25T18:33:34.9503096Z github.com/submariner-io/lighthouse/test/e2e/discovery.RunHeadlessDiscoveryTest(0xc000010d20)
  2102. 2020-08-25T18:33:34.9503604Z /go/src/github.com/submariner-io/lighthouse/test/e2e/discovery/headless_services.go:55 +0x352
  2103. 2020-08-25T18:33:34.9504113Z github.com/submariner-io/lighthouse/test/e2e/discovery.glob..func1.1.1()
  2104. 2020-08-25T18:33:34.9504458Z /go/src/github.com/submariner-io/lighthouse/test/e2e/discovery/headless_services.go:20 +0x2a
  2105. 2020-08-25T18:33:34.9505078Z github.com/submariner-io/shipyard/test/e2e.RunE2ETests(0xc0002df680, 0x385e18ff1cf)
  2106. 2020-08-25T18:33:34.9505478Z /go/src/github.com/submariner-io/lighthouse/vendor/github.com/submariner-io/shipyard/test/e2e/e2e.go:58 +0xcf
  2107. 2020-08-25T18:33:34.9505804Z github.com/submariner-io/lighthouse/test/e2e.TestE2E(0xc0002df680)
  2108. 2020-08-25T18:33:34.9506116Z /go/src/github.com/submariner-io/lighthouse/test/e2e/e2e_test.go:11 +0x2b
  2109. 2020-08-25T18:33:34.9506465Z testing.tRunner(0xc0002df680, 0x1578098)
  2110. 2020-08-25T18:33:34.9506598Z /usr/lib/golang/src/testing/testing.go:1039 +0xdc
  2111. 2020-08-25T18:33:34.9506724Z created by testing.(*T).Run
  2112. 2020-08-25T18:33:34.9506858Z /usr/lib/golang/src/testing/testing.go:1090 +0x372
  2113. 2020-08-25T18:33:34.9507184Z ------------------------------
  2114. 2020-08-25T18:33:34.9507250Z
  2115. 2020-08-25T18:33:34.9507576Z JUnit path was configured: /go/src/github.com/submariner-io/lighthouse/output/e2e-junit.xml
  2116. 2020-08-25T18:33:34.9518992Z
  2117. 2020-08-25T18:33:34.9519549Z JUnit report was created: /go/src/github.com/submariner-io/lighthouse/output/e2e-junit.xml
  2118. 2020-08-25T18:33:34.9519636Z
  2119. 2020-08-25T18:33:34.9519705Z
  2120. 2020-08-25T18:33:34.9519960Z Summarizing 1 Failure:
  2121. 2020-08-25T18:33:34.9520041Z
  2122. 2020-08-25T18:33:34.9520546Z [Fail] [discovery] Test Headless Service Discovery Across Clusters when a pod tries to resolve a headless service in a remote cluster [It] should resolve the backing pod IPs from the remote cluster 
  2123. 2020-08-25T18:33:34.9521602Z /go/src/github.com/submariner-io/lighthouse/vendor/github.com/submariner-io/shipyard/test/e2e/framework/framework.go:355
  2124. 2020-08-25T18:33:34.9521715Z
  2125. 2020-08-25T18:33:34.9522163Z Ran 7 of 7 Specs in 379.930 seconds
  2126. 2020-08-25T18:33:34.9522579Z FAIL! -- 6 Passed | 1 Failed | 0 Pending | 0 Skipped
  2127. 2020-08-25T18:33:34.9522864Z --- FAIL: TestE2E (379.93s)
  2128. 2020-08-25T18:33:34.9522985Z FAIL
  2129. 2020-08-25T18:33:34.9549467Z exit status 1
  2130. 2020-08-25T18:33:34.9550122Z FAIL github.com/submariner-io/lighthouse/test/e2e 379.953s
  2131. 2020-08-25T18:33:35.0330348Z make: *** [/opt/shipyard/Makefile.inc:40: e2e] Error 1
  2132. 2020-08-25T18:33:35.0348002Z [lighthouse]$ make e2e -- using=operator
  2133. 2020-08-25T18:33:44.4236318Z time="2020-08-25T18:33:44Z" level=fatal msg="exit status 2"
  2134. 2020-08-25T18:33:44.4250192Z make: *** [e2e] Error 1
  2135. 2020-08-25T18:33:44.4251374Z Makefile.dapper:14: recipe for target 'e2e' failed
  2136. 2020-08-25T18:33:44.4306293Z ##[error]Process completed with exit code 2.
  2137. 2020-08-25T18:33:44.4734727Z ##[group]Run df -h
  2138. 2020-08-25T18:33:44.4735058Z df -h
  2139. 2020-08-25T18:33:44.4735179Z free -h
  2140. 2020-08-25T18:33:44.4735298Z make post-mortem
  2141. 2020-08-25T18:33:44.5865631Z shell: /bin/bash -e {0}
  2142. 2020-08-25T18:33:44.5865970Z ##[endgroup]
  2143. 2020-08-25T18:33:44.6004193Z Filesystem Size Used Avail Use% Mounted on
  2144. 2020-08-25T18:33:44.6004389Z udev 3.4G 0 3.4G 0% /dev
  2145. 2020-08-25T18:33:44.6004523Z tmpfs 696M 1.5M 694M 1% /run
  2146. 2020-08-25T18:33:44.6004650Z /dev/sda1 84G 71G 13G 86% /
  2147. 2020-08-25T18:33:44.6004780Z tmpfs 3.4G 235M 3.2G 7% /dev/shm
  2148. 2020-08-25T18:33:44.6004910Z tmpfs 5.0M 0 5.0M 0% /run/lock
  2149. 2020-08-25T18:33:44.6005039Z tmpfs 3.4G 0 3.4G 0% /sys/fs/cgroup
  2150. 2020-08-25T18:33:44.6005195Z /dev/loop0 97M 97M 0 100% /snap/core/9804
  2151. 2020-08-25T18:33:44.6005329Z /dev/loop1 40M 40M 0 100% /snap/hub/43
  2152. 2020-08-25T18:33:44.6005595Z /dev/sda15 105M 3.6M 101M 4% /boot/efi
  2153. 2020-08-25T18:33:44.6005734Z /dev/sdb1 14G 4.1G 9.0G 32% /mnt
  2154. 2020-08-25T18:33:44.6122180Z total used free shared buff/cache available
  2155. 2020-08-25T18:33:44.6122562Z Mem: 6.8G 3.2G 237M 313M 3.3G 3.2G
  2156. 2020-08-25T18:33:44.6122745Z Swap: 0B 0B 0B
  2157. 2020-08-25T18:33:44.6123338Z ./.dapper -m bind make post-mortem
  2158. 2020-08-25T18:33:47.0707492Z Sending build context to Docker daemon 71.67MB
  2159. 2020-08-25T18:33:47.0707750Z
  2160. 2020-08-25T18:33:47.0813335Z Step 1/6 : FROM quay.io/submariner/shipyard-dapper-base:0.6.1
  2161. 2020-08-25T18:33:47.0830309Z ---> d8a2f56352b1
  2162. 2020-08-25T18:33:47.0834134Z Step 2/6 : ENV DAPPER_ENV="REPO TAG QUAY_USERNAME QUAY_PASSWORD GITHUB_SHA BUILD_ARGS CLUSTERS_ARGS DEPLOY_ARGS RELEASE_ARGS" DAPPER_SOURCE=/go/src/github.com/submariner-io/lighthouse DAPPER_DOCKER_SOCKET=true
  2163. 2020-08-25T18:33:47.0852190Z ---> Using cache
  2164. 2020-08-25T18:33:47.0854372Z ---> 3f3acef196fe
  2165. 2020-08-25T18:33:47.0856436Z Step 3/6 : ENV DAPPER_OUTPUT=${DAPPER_SOURCE}/output
  2166. 2020-08-25T18:33:47.0877327Z ---> Using cache
  2167. 2020-08-25T18:33:47.0877617Z ---> d47a5a00a868
  2168. 2020-08-25T18:33:47.0877785Z Step 4/6 : WORKDIR ${DAPPER_SOURCE}
  2169. 2020-08-25T18:33:47.0892490Z ---> Using cache
  2170. 2020-08-25T18:33:47.0893401Z ---> c51802b41c03
  2171. 2020-08-25T18:33:47.0893584Z Step 5/6 : ENTRYPOINT ["/opt/shipyard/scripts/entry"]
  2172. 2020-08-25T18:33:47.0910657Z ---> Using cache
  2173. 2020-08-25T18:33:47.0915815Z ---> fe29113922a8
  2174. 2020-08-25T18:33:47.0916785Z Step 6/6 : CMD ["sh"]
  2175. 2020-08-25T18:33:47.0931267Z ---> Using cache
  2176. 2020-08-25T18:33:47.0931761Z ---> 0b0c093a497a
  2177. 2020-08-25T18:33:47.2411411Z Successfully built 0b0c093a497a
  2178. 2020-08-25T18:33:47.2462819Z Successfully tagged lighthouse:HEAD
  2179. 2020-08-25T18:33:47.9362411Z [lighthouse]$ trap chown -R 1001:116 . exit
  2180. 2020-08-25T18:33:47.9379912Z [lighthouse]$ mkdir -p bin dist output
  2181. 2020-08-25T18:33:47.9458633Z [lighthouse]$ make post-mortem
  2182. 2020-08-25T18:33:48.0119476Z fatal: No names found, cannot describe anything.
  2183. 2020-08-25T18:33:48.0223597Z Makefile:39: warning: overriding recipe for target 'deploy'
  2184. 2020-08-25T18:33:48.0224455Z /opt/shipyard/Makefile.inc:36: warning: ignoring old recipe for target 'deploy'
  2185. 2020-08-25T18:33:48.0235885Z /opt/shipyard/scripts/post_mortem.sh
  2186. 2020-08-25T18:33:48.0299199Z [lighthouse]$ source /opt/shipyard/scripts/lib/utils
  2187. 2020-08-25T18:33:48.0310217Z [lighthouse]$ . /opt/shipyard/scripts/lib/source_only
  2188. 2020-08-25T18:33:48.0324238Z [lighthouse]$ script_name=utils
  2189. 2020-08-25T18:33:48.0336156Z [lighthouse]$ exec_name=post_mortem.sh
  2190. 2020-08-25T18:33:48.0357046Z [lighthouse]$ declare_kubeconfig
  2191. 2020-08-25T18:33:48.0373185Z [lighthouse]$ declare_kubeconfig
  2192. 2020-08-25T18:33:48.0383836Z [lighthouse]$ source /opt/shipyard/scripts/lib/kubecfg
  2193. 2020-08-25T18:33:48.0399389Z [lighthouse]$ export KUBECONFIG
  2194. 2020-08-25T18:33:48.0433231Z [lighthouse]$ KUBECONFIG=/go/src/github.com/submariner-io/lighthouse/output/kubeconfigs/kind-config-cluster2:/go/src/github.com/submariner-io/lighthouse/output/kubeconfigs/kind-config-cluster1:
  2195. 2020-08-25T18:33:48.0450178Z [lighthouse]$ find /go/src/github.com/submariner-io/lighthouse/output/kubeconfigs -type f -printf %p:
  2196. 2020-08-25T18:33:48.0491596Z [lighthouse]$ kind get clusters
  2197. 2020-08-25T18:33:48.5457623Z [lighthouse]$ run_sequential cluster1 cluster2 post_analyze
  2198. 2020-08-25T18:33:48.5472008Z [lighthouse]$ run_sequential cluster1 cluster2 post_analyze
  2199. 2020-08-25T18:33:48.5486364Z [lighthouse]$ local cmnd=post_analyze
  2200. 2020-08-25T18:33:48.5499534Z [lighthouse]$ eval echo cluster1 cluster2
  2201. 2020-08-25T18:33:48.5516790Z [lighthouse]$ [cluster1] post_analyze
  2202. 2020-08-25T18:33:48.5532203Z [lighthouse]$ [cluster1] sed s/^/[cluster1] /
  2203. 2020-08-25T18:33:48.5557469Z [lighthouse]$ [cluster1] post_analyze
  2204. 2020-08-25T18:33:48.5581089Z [lighthouse]$ [cluster1] kubectl get all --all-namespaces
  2205. 2020-08-25T18:33:48.5596842Z [lighthouse]$ [cluster1] kubectl get all --all-namespaces
  2206. 2020-08-25T18:33:48.5608318Z [lighthouse]$ [cluster1] command kubectl --context=cluster1 get all --all-namespaces
  2207. 2020-08-25T18:33:48.5618694Z [lighthouse]$ [cluster1] kubectl --context=cluster1 get all --all-namespaces
  2208. 2020-08-25T18:33:49.5843791Z
  2209. 2020-08-25T18:33:49.5939485Z
  2210. 2020-08-25T18:33:49.5954250Z
  2211. 2020-08-25T18:33:49.5967439Z
  2212. 2020-08-25T18:33:49.5968837Z [cluster1] ======================= Post mortem cluster1 =======================
  2213. 2020-08-25T18:33:49.5969085Z [cluster1] NAMESPACE NAME READY STATUS RESTARTS AGE
  2214. 2020-08-25T18:33:49.5969969Z [cluster1] e2e-tests-discovery-pk6g6 pod/netshoot4zgww-6757486d56-2jhdt 1/1 Terminating 0 3m30s
  2215. 2020-08-25T18:33:49.5970741Z [cluster1] e2e-tests-discovery-pk6g6 pod/netshootr6n9j-6757486d56-bjwj5 1/1 Terminating 0 3m25s
  2216. 2020-08-25T18:33:49.5971148Z [cluster1] kube-system pod/coredns-6955765f44-qklg2 1/1 Running 0 11m
  2217. 2020-08-25T18:33:49.5971541Z [cluster1] kube-system pod/coredns-6955765f44-rhmqr 1/1 Running 0 11m
  2218. 2020-08-25T18:33:49.5971915Z [cluster1] kube-system pod/etcd-cluster1-control-plane 1/1 Running 0 11m
  2219. 2020-08-25T18:33:49.5972606Z [cluster1] kube-system pod/kube-apiserver-cluster1-control-plane 1/1 Running 0 11m
  2220. 2020-08-25T18:33:49.5972999Z [cluster1] kube-system pod/kube-controller-manager-cluster1-control-plane 1/1 Running 0 11m
  2221. 2020-08-25T18:33:49.5973404Z [cluster1] kube-system pod/kube-proxy-5f684 1/1 Running 0 11m
  2222. 2020-08-25T18:33:49.5973794Z [cluster1] kube-system pod/kube-proxy-hzjfh 1/1 Running 0 11m
  2223. 2020-08-25T18:33:49.5974177Z [cluster1] kube-system pod/kube-proxy-vbv7m 1/1 Running 0 11m
  2224. 2020-08-25T18:33:49.5974562Z [cluster1] kube-system pod/kube-scheduler-cluster1-control-plane 1/1 Running 0 11m
  2225. 2020-08-25T18:33:49.5974969Z [cluster1] kube-system pod/weave-net-bs5x6 2/2 Running 0 10m
  2226. 2020-08-25T18:33:49.5975350Z [cluster1] kube-system pod/weave-net-tmpqh 2/2 Running 0 10m
  2227. 2020-08-25T18:33:49.5975738Z [cluster1] kube-system pod/weave-net-xbjcc 2/2 Running 0 10m
  2228. 2020-08-25T18:33:49.5976276Z [cluster1] local-path-storage pod/local-path-provisioner-7745554f7f-zb2hz 1/1 Running 0 11m
  2229. 2020-08-25T18:33:49.5976720Z [cluster1] submariner-operator pod/submariner-gateway-7xq4m 1/1 Running 0 8m48s
  2230. 2020-08-25T18:33:49.5977118Z [cluster1] submariner-operator pod/submariner-lighthouse-agent-799bd9865b-csrhr 1/1 Running 0 8m48s
  2231. 2020-08-25T18:33:49.5977518Z [cluster1] submariner-operator pod/submariner-lighthouse-coredns-557485fbc-qvfz4 1/1 Running 0 8m47s
  2232. 2020-08-25T18:33:49.5977917Z [cluster1] submariner-operator pod/submariner-lighthouse-coredns-557485fbc-z2jtl 1/1 Running 0 8m47s
  2233. 2020-08-25T18:33:49.5978301Z [cluster1] submariner-operator pod/submariner-operator-6675977db7-ck45d 1/1 Running 0 8m55s
  2234. 2020-08-25T18:33:49.5978705Z [cluster1] submariner-operator pod/submariner-routeagent-7lr7d 1/1 Running 0 8m48s
  2235. 2020-08-25T18:33:49.5979099Z [cluster1] submariner-operator pod/submariner-routeagent-zwzlh 1/1 Running 0 8m48s
  2236. 2020-08-25T18:33:49.5979504Z [cluster1] NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
  2237. 2020-08-25T18:33:49.5979678Z [cluster1] default service/kubernetes ClusterIP 100.91.0.1 <none> 443/TCP 11m
  2238. 2020-08-25T18:33:49.5980095Z [cluster1] kube-system service/kube-dns ClusterIP 100.91.0.10 <none> 53/UDP,53/TCP,9153/TCP 11m
  2239. 2020-08-25T18:33:49.5980510Z [cluster1] submariner-operator service/submariner-lighthouse-coredns ClusterIP 100.91.108.83 <none> 53/UDP 8m47s
  2240. 2020-08-25T18:33:49.5981062Z [cluster1] submariner-operator service/submariner-operator-metrics ClusterIP 100.91.18.224 <none> 8383/TCP,8686/TCP 8m49s
  2241. 2020-08-25T18:33:49.5981484Z [cluster1] NAMESPACE NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
  2242. 2020-08-25T18:33:49.5993819Z
  2243. 2020-08-25T18:33:49.6001484Z
  2244. 2020-08-25T18:33:49.6011862Z
  2245. 2020-08-25T18:33:49.6011971Z
  2246. 2020-08-25T18:33:49.6017159Z
  2247. 2020-08-25T18:33:49.6124924Z [lighthouse]$ [cluster1] kubectl get pods -A
  2248. 2020-08-25T18:33:49.6138691Z [lighthouse]$ [cluster1] tail -n +2
  2249. 2020-08-25T18:33:49.6185598Z [lighthouse]$ [cluster1] grep -v Running
  2250. 2020-08-25T18:33:49.6203999Z [lighthouse]$ [cluster1] kubectl get pods -A
  2251. 2020-08-25T18:33:49.6222972Z [lighthouse]$ [cluster1] sed s/ */;/g
  2252. 2020-08-25T18:33:49.6243305Z [lighthouse]$ [cluster1] command kubectl --context=cluster1 get pods -A
  2253. 2020-08-25T18:33:49.6275455Z [lighthouse]$ [cluster1] kubectl --context=cluster1 get pods -A
  2254. 2020-08-25T18:33:50.4856707Z [lighthouse]$ [cluster1] ns=e2e-tests-discovery-pk6g6
  2255. 2020-08-25T18:33:50.4883574Z [lighthouse]$ [cluster1] cut -f1 -d;
  2256. 2020-08-25T18:33:50.4928047Z [lighthouse]$ [cluster1] name=netshoot4zgww-6757486d56-2jhdt
  2257. 2020-08-25T18:33:50.4949426Z [lighthouse]$ [cluster1] cut -f2 -d;
  2258. 2020-08-25T18:33:50.4978812Z [lighthouse]$ [cluster1] kubectl -n e2e-tests-discovery-pk6g6 describe pod netshoot4zgww-6757486d56-2jhdt
  2259. 2020-08-25T18:33:50.4991673Z [lighthouse]$ [cluster1] kubectl -n e2e-tests-discovery-pk6g6 describe pod netshoot4zgww-6757486d56-2jhdt
  2260. 2020-08-25T18:33:50.5006450Z [lighthouse]$ [cluster1] command kubectl --context=cluster1 -n e2e-tests-discovery-pk6g6 describe pod netshoot4zgww-6757486d56-2jhdt
  2261. 2020-08-25T18:33:50.5048468Z [lighthouse]$ [cluster1] kubectl --context=cluster1 -n e2e-tests-discovery-pk6g6 describe pod netshoot4zgww-6757486d56-2jhdt
  2262. 2020-08-25T18:33:51.3174641Z [cluster1] kube-system daemonset.apps/kube-proxy 3 3 3 3 3 beta.kubernetes.io/os=linux 11m
  2263. 2020-08-25T18:33:51.3175535Z [cluster1] kube-system daemonset.apps/weave-net 3 3 3 3 3 <none> 10m
  2264. 2020-08-25T18:33:51.3176644Z [cluster1] submariner-operator daemonset.apps/submariner-gateway 1 1 1 1 1 submariner.io/gateway=true 8m48s
  2265. 2020-08-25T18:33:51.3177128Z [cluster1] submariner-operator daemonset.apps/submariner-routeagent 2 2 2 2 2 <none> 8m48s
  2266. 2020-08-25T18:33:51.3177522Z [cluster1] NAMESPACE NAME READY UP-TO-DATE AVAILABLE AGE
  2267. 2020-08-25T18:33:51.3177889Z [cluster1] kube-system deployment.apps/coredns 2/2 2 2 11m
  2268. 2020-08-25T18:33:51.3178303Z [cluster1] local-path-storage deployment.apps/local-path-provisioner 1/1 1 1 11m
  2269. 2020-08-25T18:33:51.3178689Z [cluster1] submariner-operator deployment.apps/submariner-lighthouse-agent 1/1 1 1 8m48s
  2270. 2020-08-25T18:33:51.3179077Z [cluster1] submariner-operator deployment.apps/submariner-lighthouse-coredns 2/2 2 2 8m48s
  2271. 2020-08-25T18:33:51.3179456Z [cluster1] submariner-operator deployment.apps/submariner-operator 1/1 1 1 8m55s
  2272. 2020-08-25T18:33:51.3179616Z [cluster1] NAMESPACE NAME DESIRED CURRENT READY AGE
  2273. 2020-08-25T18:33:51.3180176Z [cluster1] kube-system replicaset.apps/coredns-6955765f44 2 2 2 11m
  2274. 2020-08-25T18:33:51.3180795Z [cluster1] local-path-storage replicaset.apps/local-path-provisioner-7745554f7f 1 1 1 11m
  2275. 2020-08-25T18:33:51.3181179Z [cluster1] submariner-operator replicaset.apps/submariner-lighthouse-agent-799bd9865b 1 1 1 8m48s
  2276. 2020-08-25T18:33:51.3181563Z [cluster1] submariner-operator replicaset.apps/submariner-lighthouse-coredns-557485fbc 2 2 2 8m47s
  2277. 2020-08-25T18:33:51.3181941Z [cluster1] submariner-operator replicaset.apps/submariner-operator-6675977db7 1 1 1 8m55s
  2278. 2020-08-25T18:33:51.3182318Z [cluster1] ======================= netshoot4zgww-6757486d56-2jhdt - e2e-tests-discovery-pk6g6 ============================
  2279. 2020-08-25T18:33:51.3182635Z [cluster1] Name: netshoot4zgww-6757486d56-2jhdt
  2280. 2020-08-25T18:33:51.3182920Z [cluster1] Namespace: e2e-tests-discovery-pk6g6
  2281. 2020-08-25T18:33:51.3183053Z [cluster1] Priority: 0
  2282. 2020-08-25T18:33:51.3183356Z [cluster1] Node: cluster1-worker2/172.17.0.6
  2283. 2020-08-25T18:33:51.3183496Z [cluster1] Start Time: Tue, 25 Aug 2020 18:30:19 +0000
  2284. 2020-08-25T18:33:51.3183633Z [cluster1] Labels: app=netshoot
  2285. 2020-08-25T18:33:51.3183940Z [cluster1] pod-template-hash=6757486d56
  2286. 2020-08-25T18:33:51.3184073Z [cluster1] Annotations: <none>
  2287. 2020-08-25T18:33:51.3184202Z [cluster1] Status: Terminating (lasts <invalid>)
  2288. 2020-08-25T18:33:51.3184332Z [cluster1] Termination Grace Period: 30s
  2289. 2020-08-25T18:33:51.3184627Z [cluster1] IP: 10.241.192.4
  2290. 2020-08-25T18:33:51.3185484Z [cluster1] Controlled By: ReplicaSet/netshoot4zgww-6757486d56
  2291. 2020-08-25T18:33:51.3185620Z [cluster1] Containers:
  2292. 2020-08-25T18:33:51.3185735Z [cluster1] netshoot:
  2293. 2020-08-25T18:33:51.3185875Z [cluster1] Container ID: containerd://cc4747c4871aa6cfa831762ed49803e7b415ef99a5ae57bf79c83fcad456cdce
  2294. 2020-08-25T18:33:51.3186031Z [cluster1] Image: quay.io/submariner/nettest
  2295. 2020-08-25T18:33:51.3186303Z [cluster1] Image ID: quay.io/submariner/nettest@sha256:3f8474fd8f3a41eeb84744a75b2e6a1b862dd968001dba5c7c6cc3579a3a716c
  2296. 2020-08-25T18:33:51.3186455Z [cluster1] Port: <none>
  2297. 2020-08-25T18:33:51.3186559Z [cluster1] Host Port: <none>
  2298. 2020-08-25T18:33:51.3186675Z [cluster1] Command:
  2299. 2020-08-25T18:33:51.3186787Z [cluster1] sleep
  2300. 2020-08-25T18:33:51.3186900Z [cluster1] 600
  2301. 2020-08-25T18:33:51.3187133Z [cluster1] State: Running
  2302. 2020-08-25T18:33:51.3187264Z [cluster1] Started: Tue, 25 Aug 2020 18:30:21 +0000
  2303. 2020-08-25T18:33:51.3187391Z [cluster1] Ready: True
  2304. 2020-08-25T18:33:51.3187510Z [cluster1] Restart Count: 0
  2305. 2020-08-25T18:33:51.3187613Z [cluster1] Environment: <none>
  2306. 2020-08-25T18:33:51.3187729Z [cluster1] Mounts:
  2307. 2020-08-25T18:33:51.3188099Z [cluster1] /var/run/secrets/kubernetes.io/serviceaccount from default-token-7ntth (ro)
  2308. 2020-08-25T18:33:51.3188239Z [cluster1] Conditions:
  2309. 2020-08-25T18:33:51.3188356Z [cluster1] Type Status
  2310. 2020-08-25T18:33:51.3188659Z [cluster1] Initialized True
  2311. 2020-08-25T18:33:51.3189163Z [cluster1] Ready True
  2312. 2020-08-25T18:33:51.3189308Z [cluster1] ContainersReady True
  2313. 2020-08-25T18:33:51.3189436Z [cluster1] PodScheduled True
  2314. 2020-08-25T18:33:51.3189554Z [cluster1] Volumes:
  2315. 2020-08-25T18:33:51.3189934Z [cluster1] default-token-7ntth:
  2316. 2020-08-25T18:33:51.3190071Z [cluster1] Type: Secret (a volume populated by a Secret)
  2317. 2020-08-25T18:33:51.3190365Z [cluster1] SecretName: default-token-7ntth
  2318. 2020-08-25T18:33:51.3190493Z [cluster1] Optional: false
  2319. 2020-08-25T18:33:51.3190618Z [cluster1] QoS Class: BestEffort
  2320. 2020-08-25T18:33:51.3190867Z [cluster1] Node-Selectors: <none>
  2321. 2020-08-25T18:33:51.3284894Z [lighthouse]$ [cluster1] kubectl -n e2e-tests-discovery-pk6g6 logs netshoot4zgww-6757486d56-2jhdt
  2322. 2020-08-25T18:33:51.3298870Z [lighthouse]$ [cluster1] kubectl -n e2e-tests-discovery-pk6g6 logs netshoot4zgww-6757486d56-2jhdt
  2323. 2020-08-25T18:33:51.3311835Z [lighthouse]$ [cluster1] command kubectl --context=cluster1 -n e2e-tests-discovery-pk6g6 logs netshoot4zgww-6757486d56-2jhdt
  2324. 2020-08-25T18:33:51.3324845Z [lighthouse]$ [cluster1] kubectl --context=cluster1 -n e2e-tests-discovery-pk6g6 logs netshoot4zgww-6757486d56-2jhdt
  2325. 2020-08-25T18:33:52.1658634Z [lighthouse]$ [cluster1] ns=e2e-tests-discovery-pk6g6
  2326. 2020-08-25T18:33:52.1676595Z [lighthouse]$ [cluster1] cut -f1 -d;
  2327. 2020-08-25T18:33:52.1727728Z [lighthouse]$ [cluster1] name=netshootr6n9j-6757486d56-bjwj5
  2328. 2020-08-25T18:33:52.1751457Z [lighthouse]$ [cluster1] cut -f2 -d;
  2329. 2020-08-25T18:33:52.1788426Z [lighthouse]$ [cluster1] kubectl -n e2e-tests-discovery-pk6g6 describe pod netshootr6n9j-6757486d56-bjwj5
  2330. 2020-08-25T18:33:52.1800963Z [lighthouse]$ [cluster1] kubectl -n e2e-tests-discovery-pk6g6 describe pod netshootr6n9j-6757486d56-bjwj5
  2331. 2020-08-25T18:33:52.1821482Z [lighthouse]$ [cluster1] command kubectl --context=cluster1 -n e2e-tests-discovery-pk6g6 describe pod netshootr6n9j-6757486d56-bjwj5
  2332. 2020-08-25T18:33:52.1834822Z [lighthouse]$ [cluster1] kubectl --context=cluster1 -n e2e-tests-discovery-pk6g6 describe pod netshootr6n9j-6757486d56-bjwj5
  2333. 2020-08-25T18:33:53.0303126Z [lighthouse]$ [cluster1] kubectl -n e2e-tests-discovery-pk6g6 logs netshootr6n9j-6757486d56-bjwj5
  2334. 2020-08-25T18:33:53.0312031Z [lighthouse]$ [cluster1] kubectl -n e2e-tests-discovery-pk6g6 logs netshootr6n9j-6757486d56-bjwj5
  2335. 2020-08-25T18:33:53.0345541Z [lighthouse]$ [cluster1] command kubectl --context=cluster1 -n e2e-tests-discovery-pk6g6 logs netshootr6n9j-6757486d56-bjwj5
  2336. 2020-08-25T18:33:53.0364369Z [lighthouse]$ [cluster1] kubectl --context=cluster1 -n e2e-tests-discovery-pk6g6 logs netshootr6n9j-6757486d56-bjwj5
  2337. 2020-08-25T18:33:53.8383112Z [lighthouse]$ [cluster1] namespace=kube-system
  2338. 2020-08-25T18:33:53.8395774Z [lighthouse]$ [cluster1] kubectl get pods --selector=k8s-app=kube-proxy -n kube-system -o jsonpath={.items[*].metadata.name}
  2339. 2020-08-25T18:33:53.8405845Z [lighthouse]$ [cluster1] kubectl get pods --selector=k8s-app=kube-proxy -n kube-system -o jsonpath={.items[*].metadata.name}
  2340. 2020-08-25T18:33:53.8416871Z [lighthouse]$ [cluster1] command kubectl --context=cluster1 get pods --selector=k8s-app=kube-proxy -n kube-system -o jsonpath={.items[*].metadata.name}
  2341. 2020-08-25T18:33:53.8426945Z [lighthouse]$ [cluster1] kubectl --context=cluster1 get pods --selector=k8s-app=kube-proxy -n kube-system -o jsonpath={.items[*].metadata.name}
  2342. 2020-08-25T18:33:54.6906196Z [lighthouse]$ [cluster1] kubectl -n kube-system logs kube-proxy-5f684
  2343. 2020-08-25T18:33:54.6916584Z [lighthouse]$ [cluster1] kubectl -n kube-system logs kube-proxy-5f684
  2344. 2020-08-25T18:33:54.6930024Z [lighthouse]$ [cluster1] command kubectl --context=cluster1 -n kube-system logs kube-proxy-5f684
  2345. 2020-08-25T18:33:54.6942929Z [lighthouse]$ [cluster1] kubectl --context=cluster1 -n kube-system logs kube-proxy-5f684
  2346. 2020-08-25T18:33:55.5211612Z [cluster1] Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
  2347. 2020-08-25T18:33:55.5216841Z [cluster1] node.kubernetes.io/unreachable:NoExecute for 300s
  2348. 2020-08-25T18:33:55.5216997Z [cluster1] Events: <none>
  2349. 2020-08-25T18:33:55.5217808Z [cluster1] ===================== END netshoot4zgww-6757486d56-2jhdt - e2e-tests-discovery-pk6g6 ==========================
  2350. 2020-08-25T18:33:55.5218233Z [cluster1] ======================= netshootr6n9j-6757486d56-bjwj5 - e2e-tests-discovery-pk6g6 ============================
  2351. 2020-08-25T18:33:55.5218565Z [cluster1] Name: netshootr6n9j-6757486d56-bjwj5
  2352. 2020-08-25T18:33:55.5219215Z [cluster1] Namespace: e2e-tests-discovery-pk6g6
  2353. 2020-08-25T18:33:55.5219348Z [cluster1] Priority: 0
  2354. 2020-08-25T18:33:55.5219969Z [cluster1] Node: cluster1-worker/172.17.0.9
  2355. 2020-08-25T18:33:55.5220122Z [cluster1] Start Time: Tue, 25 Aug 2020 18:30:24 +0000
  2356. 2020-08-25T18:33:55.5220256Z [cluster1] Labels: app=netshoot
  2357. 2020-08-25T18:33:55.5220554Z [cluster1] pod-template-hash=6757486d56
  2358. 2020-08-25T18:33:55.5220688Z [cluster1] Annotations: <none>
  2359. 2020-08-25T18:33:55.5220802Z [cluster1] Status: Terminating (lasts <invalid>)
  2360. 2020-08-25T18:33:55.5220931Z [cluster1] Termination Grace Period: 30s
  2361. 2020-08-25T18:33:55.5221055Z [cluster1] IP: 10.241.128.5
  2362. 2020-08-25T18:33:55.5221367Z [cluster1] Controlled By: ReplicaSet/netshootr6n9j-6757486d56
  2363. 2020-08-25T18:33:55.5221497Z [cluster1] Containers:
  2364. 2020-08-25T18:33:55.5221611Z [cluster1] netshoot:
  2365. 2020-08-25T18:33:55.5221752Z [cluster1] Container ID: containerd://da3ca1ee8183abc53c10cf59d470d7ddb950afb143aef79ceed06095ff82f29e
  2366. 2020-08-25T18:33:55.5221910Z [cluster1] Image: quay.io/submariner/nettest
  2367. 2020-08-25T18:33:55.5222050Z [cluster1] Image ID: quay.io/submariner/nettest@sha256:3f8474fd8f3a41eeb84744a75b2e6a1b862dd968001dba5c7c6cc3579a3a716c
  2368. 2020-08-25T18:33:55.5222199Z [cluster1] Port: <none>
  2369. 2020-08-25T18:33:55.5222318Z [cluster1] Host Port: <none>
  2370. 2020-08-25T18:33:55.5222434Z [cluster1] Command:
  2371. 2020-08-25T18:33:55.5222547Z [cluster1] sleep
  2372. 2020-08-25T18:33:55.5222657Z [cluster1] 600
  2373. 2020-08-25T18:33:55.5222777Z [cluster1] State: Running
  2374. 2020-08-25T18:33:55.5222905Z [cluster1] Started: Tue, 25 Aug 2020 18:30:25 +0000
  2375. 2020-08-25T18:33:55.5223015Z [cluster1] Ready: True
  2376. 2020-08-25T18:33:55.5223133Z [cluster1] Restart Count: 0
  2377. 2020-08-25T18:33:55.5223251Z [cluster1] Environment: <none>
  2378. 2020-08-25T18:33:55.5223367Z [cluster1] Mounts:
  2379. 2020-08-25T18:33:55.5223702Z [cluster1] /var/run/secrets/kubernetes.io/serviceaccount from default-token-7ntth (ro)
  2380. 2020-08-25T18:33:55.5223845Z [cluster1] Conditions:
  2381. 2020-08-25T18:33:55.5223961Z [cluster1] Type Status
  2382. 2020-08-25T18:33:55.5224200Z [cluster1] Initialized True
  2383. 2020-08-25T18:33:55.5224331Z [cluster1] Ready True
  2384. 2020-08-25T18:33:55.5224448Z [cluster1] ContainersReady True
  2385. 2020-08-25T18:33:55.5224565Z [cluster1] PodScheduled True
  2386. 2020-08-25T18:33:55.5225171Z [cluster1] Volumes:
  2387. 2020-08-25T18:33:55.5225500Z [cluster1] default-token-7ntth:
  2388. 2020-08-25T18:33:55.5225632Z [cluster1] Type: Secret (a volume populated by a Secret)
  2389. 2020-08-25T18:33:55.5225915Z [cluster1] SecretName: default-token-7ntth
  2390. 2020-08-25T18:33:55.5226022Z [cluster1] Optional: false
  2391. 2020-08-25T18:33:55.5226143Z [cluster1] QoS Class: BestEffort
  2392. 2020-08-25T18:33:55.5226582Z [cluster1] Node-Selectors: <none>
  2393. 2020-08-25T18:33:55.5226891Z [cluster1] Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
  2394. 2020-08-25T18:33:55.5227049Z [cluster1] node.kubernetes.io/unreachable:NoExecute for 300s
  2395. 2020-08-25T18:33:55.5227185Z [cluster1] Events: <none>
  2396. 2020-08-25T18:33:55.5227718Z [cluster1] ===================== END netshootr6n9j-6757486d56-bjwj5 - e2e-tests-discovery-pk6g6 ==========================
  2397. 2020-08-25T18:33:55.5228240Z [cluster1] +++++++++++++++++++++: Logs for Pod kube-proxy-5f684 in namespace kube-system :++++++++++++++++++++++
  2398. 2020-08-25T18:33:55.5228569Z [cluster1] W0825 18:22:57.761416 1 server_others.go:323] Unknown proxy mode "", assuming iptables proxy
  2399. 2020-08-25T18:33:55.5228709Z [cluster1] I0825 18:22:57.772744 1 node.go:135] Successfully retrieved node IP: 172.17.0.6
  2400. 2020-08-25T18:33:55.5229334Z [cluster1] I0825 18:22:57.772770 1 server_others.go:145] Using iptables Proxier.
  2401. 2020-08-25T18:33:55.5229562Z [cluster1] I0825 18:22:57.773239 1 server.go:571] Version: v1.17.0
  2402. 2020-08-25T18:33:55.5229873Z [cluster1] I0825 18:22:57.773685 1 conntrack.go:52] Setting nf_conntrack_max to 131072
  2403. 2020-08-25T18:33:55.5230538Z [cluster1] I0825 18:22:57.773857 1 conntrack.go:100] Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_established' to 86400
  2404. 2020-08-25T18:33:55.5230914Z [cluster1] I0825 18:22:57.773928 1 conntrack.go:100] Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_close_wait' to 3600
  2405. 2020-08-25T18:33:55.5231058Z [cluster1] I0825 18:22:57.775471 1 config.go:131] Starting endpoints config controller
  2406. 2020-08-25T18:33:55.5231209Z [cluster1] I0825 18:22:57.775487 1 shared_informer.go:197] Waiting for caches to sync for endpoints config
  2407. 2020-08-25T18:33:55.5231357Z [cluster1] I0825 18:22:57.775556 1 config.go:313] Starting service config controller
  2408. 2020-08-25T18:33:55.5231508Z [cluster1] I0825 18:22:57.775561 1 shared_informer.go:197] Waiting for caches to sync for service config
  2409. 2020-08-25T18:33:55.5231654Z [cluster1] I0825 18:22:57.876563 1 shared_informer.go:204] Caches are synced for service config
  2410. 2020-08-25T18:33:55.5318266Z [lighthouse]$ [cluster1] kubectl -n kube-system logs kube-proxy-hzjfh
  2411. 2020-08-25T18:33:55.5337819Z [lighthouse]$ [cluster1] kubectl -n kube-system logs kube-proxy-hzjfh
  2412. 2020-08-25T18:33:55.5353342Z [lighthouse]$ [cluster1] command kubectl --context=cluster1 -n kube-system logs kube-proxy-hzjfh
  2413. 2020-08-25T18:33:55.5371312Z [lighthouse]$ [cluster1] kubectl --context=cluster1 -n kube-system logs kube-proxy-hzjfh
  2414. 2020-08-25T18:33:56.3679878Z [lighthouse]$ [cluster1] kubectl -n kube-system logs kube-proxy-vbv7m
  2415. 2020-08-25T18:33:56.3695125Z [lighthouse]$ [cluster1] kubectl -n kube-system logs kube-proxy-vbv7m
  2416. 2020-08-25T18:33:56.3708272Z [lighthouse]$ [cluster1] command kubectl --context=cluster1 -n kube-system logs kube-proxy-vbv7m
  2417. 2020-08-25T18:33:56.3716955Z [lighthouse]$ [cluster1] kubectl --context=cluster1 -n kube-system logs kube-proxy-vbv7m
  2418. 2020-08-25T18:33:57.2293923Z [lighthouse]$ [cluster1] namespace=submariner-operator
  2419. 2020-08-25T18:33:57.2311852Z [lighthouse]$ [cluster1] kubectl get pods --selector=app=submariner-globalnet -n submariner-operator -o jsonpath={.items[*].metadata.name}
  2420. 2020-08-25T18:33:57.2323743Z [lighthouse]$ [cluster1] kubectl get pods --selector=app=submariner-globalnet -n submariner-operator -o jsonpath={.items[*].metadata.name}
  2421. 2020-08-25T18:33:57.2335761Z [lighthouse]$ [cluster1] command kubectl --context=cluster1 get pods --selector=app=submariner-globalnet -n submariner-operator -o jsonpath={.items[*].metadata.name}
  2422. 2020-08-25T18:33:57.2348719Z [lighthouse]$ [cluster1] kubectl --context=cluster1 get pods --selector=app=submariner-globalnet -n submariner-operator -o jsonpath={.items[*].metadata.name}
  2423. 2020-08-25T18:33:58.0759396Z [lighthouse]$ [cluster1] kubectl get pods --selector=app=submariner-engine -n submariner-operator -o jsonpath={.items[*].metadata.name}
  2424. 2020-08-25T18:33:58.0766934Z [lighthouse]$ [cluster1] kubectl get pods --selector=app=submariner-engine -n submariner-operator -o jsonpath={.items[*].metadata.name}
  2425. 2020-08-25T18:33:58.0782139Z [lighthouse]$ [cluster1] command kubectl --context=cluster1 get pods --selector=app=submariner-engine -n submariner-operator -o jsonpath={.items[*].metadata.name}
  2426. 2020-08-25T18:33:58.0791895Z [lighthouse]$ [cluster1] kubectl --context=cluster1 get pods --selector=app=submariner-engine -n submariner-operator -o jsonpath={.items[*].metadata.name}
  2427. 2020-08-25T18:33:58.8941229Z [lighthouse]$ [cluster1] kubectl -n submariner-operator logs submariner-gateway-7xq4m
  2428. 2020-08-25T18:33:58.8953908Z [lighthouse]$ [cluster1] kubectl -n submariner-operator logs submariner-gateway-7xq4m
  2429. 2020-08-25T18:33:58.8969632Z [lighthouse]$ [cluster1] command kubectl --context=cluster1 -n submariner-operator logs submariner-gateway-7xq4m
  2430. 2020-08-25T18:33:58.8982023Z [lighthouse]$ [cluster1] kubectl --context=cluster1 -n submariner-operator logs submariner-gateway-7xq4m
  2431. 2020-08-25T18:33:59.7237696Z [cluster1] I0825 18:22:57.876646 1 shared_informer.go:204] Caches are synced for endpoints config
  2432. 2020-08-25T18:33:59.7240443Z [cluster1] +++++++++++++++++++++: Logs for Pod kube-proxy-hzjfh in namespace kube-system :++++++++++++++++++++++
  2433. 2020-08-25T18:33:59.7240630Z [cluster1] W0825 18:22:57.872014 1 server_others.go:323] Unknown proxy mode "", assuming iptables proxy
  2434. 2020-08-25T18:33:59.7240793Z [cluster1] I0825 18:22:57.893647 1 node.go:135] Successfully retrieved node IP: 172.17.0.9
  2435. 2020-08-25T18:33:59.7240945Z [cluster1] I0825 18:22:57.896759 1 server_others.go:145] Using iptables Proxier.
  2436. 2020-08-25T18:33:59.7241132Z [cluster1] I0825 18:22:57.897198 1 server.go:571] Version: v1.17.0
  2437. 2020-08-25T18:33:59.7241285Z [cluster1] I0825 18:22:57.897743 1 conntrack.go:52] Setting nf_conntrack_max to 131072
  2438. 2020-08-25T18:33:59.7241706Z [cluster1] I0825 18:22:57.901485 1 conntrack.go:100] Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_established' to 86400
  2439. 2020-08-25T18:33:59.7242107Z [cluster1] I0825 18:22:57.902839 1 conntrack.go:100] Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_close_wait' to 3600
  2440. 2020-08-25T18:33:59.7242525Z [cluster1] I0825 18:22:57.903103 1 config.go:313] Starting service config controller
  2441. 2020-08-25T18:33:59.7242690Z [cluster1] I0825 18:22:57.903112 1 shared_informer.go:197] Waiting for caches to sync for service config
  2442. 2020-08-25T18:33:59.7242851Z [cluster1] I0825 18:22:57.903137 1 config.go:131] Starting endpoints config controller
  2443. 2020-08-25T18:33:59.7243014Z [cluster1] I0825 18:22:57.903146 1 shared_informer.go:197] Waiting for caches to sync for endpoints config
  2444. 2020-08-25T18:33:59.7243161Z [cluster1] I0825 18:22:58.006677 1 shared_informer.go:204] Caches are synced for endpoints config
  2445. 2020-08-25T18:33:59.7243320Z [cluster1] I0825 18:22:58.006817 1 shared_informer.go:204] Caches are synced for service config
  2446. 2020-08-25T18:33:59.7243710Z [cluster1] +++++++++++++++++++++: Logs for Pod kube-proxy-vbv7m in namespace kube-system :++++++++++++++++++++++
  2447. 2020-08-25T18:33:59.7244087Z [cluster1] W0825 18:22:30.284553 1 server_others.go:323] Unknown proxy mode "", assuming iptables proxy
  2448. 2020-08-25T18:33:59.7244297Z [cluster1] I0825 18:22:30.293931 1 node.go:135] Successfully retrieved node IP: 172.17.0.4
  2449. 2020-08-25T18:33:59.7244449Z [cluster1] I0825 18:22:30.294002 1 server_others.go:145] Using iptables Proxier.
  2450. 2020-08-25T18:33:59.7244607Z [cluster1] I0825 18:22:30.294541 1 server.go:571] Version: v1.17.0
  2451. 2020-08-25T18:33:59.7244793Z [cluster1] I0825 18:22:30.295199 1 conntrack.go:52] Setting nf_conntrack_max to 131072
  2452. 2020-08-25T18:33:59.7245232Z [cluster1] I0825 18:22:30.295426 1 conntrack.go:100] Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_established' to 86400
  2453. 2020-08-25T18:33:59.7245625Z [cluster1] I0825 18:22:30.295570 1 conntrack.go:100] Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_close_wait' to 3600
  2454. 2020-08-25T18:33:59.7245787Z [cluster1] I0825 18:22:30.299160 1 config.go:313] Starting service config controller
  2455. 2020-08-25T18:33:59.7245933Z [cluster1] I0825 18:22:30.299317 1 shared_informer.go:197] Waiting for caches to sync for service config
  2456. 2020-08-25T18:33:59.7246090Z [cluster1] I0825 18:22:30.299642 1 config.go:131] Starting endpoints config controller
  2457. 2020-08-25T18:33:59.7246250Z [cluster1] I0825 18:22:30.299961 1 shared_informer.go:197] Waiting for caches to sync for endpoints config
  2458. 2020-08-25T18:33:59.7246411Z [cluster1] I0825 18:22:30.401628 1 shared_informer.go:204] Caches are synced for endpoints config
  2459. 2020-08-25T18:33:59.7246569Z [cluster1] I0825 18:22:30.401677 1 shared_informer.go:204] Caches are synced for service config
  2460. 2020-08-25T18:33:59.7246722Z [cluster1] ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
  2461. 2020-08-25T18:33:59.7246874Z [cluster1] ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
  2462. 2020-08-25T18:33:59.7247428Z [cluster1] +++++++++++++++++++++: Logs for Pod submariner-gateway-7xq4m in namespace submariner-operator :++++++++++++++++++++++
  2463. 2020-08-25T18:33:59.7247724Z [cluster1] + trap 'exit 1' SIGTERM SIGINT
  2464. 2020-08-25T18:33:59.7247861Z [cluster1] + export CHARON_PID_FILE=/var/run/charon.pid
  2465. 2020-08-25T18:33:59.7247994Z [cluster1] + CHARON_PID_FILE=/var/run/charon.pid
  2466. 2020-08-25T18:33:59.7248251Z [cluster1] + rm -f /var/run/charon.pid
  2467. 2020-08-25T18:33:59.7248376Z [cluster1] + SUBMARINER_VERBOSITY=2
  2468. 2020-08-25T18:33:59.7249550Z [cluster1] + '[' false == true ']'
  2469. 2020-08-25T18:33:59.7249944Z [cluster1] + DEBUG=-v=2
  2470. 2020-08-25T18:33:59.7250208Z [cluster1] + mkdir -p /etc/ipsec
  2471. 2020-08-25T18:33:59.7250506Z [cluster1] + sysctl -w net.ipv4.conf.all.send_redirects=0
  2472. 2020-08-25T18:33:59.7250642Z [cluster1] net.ipv4.conf.all.send_redirects = 0
  2473. 2020-08-25T18:33:59.7250934Z [cluster1] iptables-save is present on the host at /usr/sbin/iptables-save
  2474. 2020-08-25T18:33:59.7251111Z [cluster1] + export PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/libexec/strongswan
  2475. 2020-08-25T18:33:59.7251274Z [cluster1] + PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/libexec/strongswan
  2476. 2020-08-25T18:33:59.7251569Z [cluster1] + for f in iptables-save iptables
  2477. 2020-08-25T18:33:59.7272215Z [cluster1] ++ find_iptables_on_host iptables-save
  2478. 2020-08-25T18:33:59.7272653Z [cluster1] ++ chroot /host test -x /usr/sbin/iptables-save
  2479. 2020-08-25T18:33:59.7272787Z [cluster1] ++ echo /usr/sbin
  2480. 2020-08-25T18:33:59.7272903Z [cluster1] ++ return
  2481. 2020-08-25T18:33:59.7273020Z [cluster1] + location=/usr/sbin
  2482. 2020-08-25T18:33:59.7273291Z [cluster1] + '[' /usr/sbin '!=' unknown ']'
  2483. 2020-08-25T18:33:59.7273599Z [cluster1] + echo 'iptables-save is present on the host at /usr/sbin/iptables-save'
  2484. 2020-08-25T18:33:59.7273907Z [cluster1] + sed 's!@@PATH@@!/usr/sbin!' /usr/sbin/iptables-wrapper.in
  2485. 2020-08-25T18:33:59.7274165Z [cluster1] + for f in iptables-save iptables
  2486. 2020-08-25T18:33:59.7274304Z [cluster1] ++ find_iptables_on_host iptables
  2487. 2020-08-25T18:33:59.7274581Z [cluster1] ++ chroot /host test -x /usr/sbin/iptables
  2488. 2020-08-25T18:33:59.7274874Z [cluster1] ++ echo /usr/sbin
  2489. 2020-08-25T18:33:59.7275004Z [cluster1] ++ return
  2490. 2020-08-25T18:33:59.7275120Z [cluster1] + location=/usr/sbin
  2491. 2020-08-25T18:33:59.7275416Z [cluster1] + '[' /usr/sbin '!=' unknown ']'
  2492. 2020-08-25T18:33:59.7275694Z [cluster1] + echo 'iptables is present on the host at /usr/sbin/iptables'
  2493. 2020-08-25T18:33:59.7275999Z [cluster1] + sed 's!@@PATH@@!/usr/sbin!' /usr/sbin/iptables-wrapper.in
  2494. 2020-08-25T18:33:59.7276139Z [cluster1] iptables is present on the host at /usr/sbin/iptables
  2495. 2020-08-25T18:33:59.7276609Z [cluster1] + exec submariner-engine -v=2 -alsologtostderr
  2496. 2020-08-25T18:33:59.7276758Z [cluster1] I0825 18:25:34.402951 1 main.go:67] Starting the submariner gateway engine
  2497. 2020-08-25T18:33:59.7277191Z [cluster1] W0825 18:25:34.404513 1 client_config.go:543] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work.
  2498. 2020-08-25T18:33:59.7277370Z [cluster1] I0825 18:25:34.421627 1 main.go:93] Creating the cable engine
  2499. 2020-08-25T18:33:59.7277528Z [cluster1] I0825 18:25:34.421755 1 syncer.go:48] CableEngine syncer started
  2500. 2020-08-25T18:33:59.7277696Z [cluster1] I0825 18:25:34.433642 1 main.go:232] Gateway leader election config values: main.leaderConfig{LeaseDuration:10, RenewDeadline:5, RetryPeriod:2}
  2501. 2020-08-25T18:33:59.7278100Z [cluster1] I0825 18:25:34.437373 1 main.go:249] Using namespace "submariner-operator" for the leader election lock
  2502. 2020-08-25T18:33:59.7278517Z [cluster1] I0825 18:25:34.437400 1 leaderelection.go:242] attempting to acquire leader lease submariner-operator/submariner-engine-lock...
  2503. 2020-08-25T18:33:59.7279444Z [cluster1] I0825 18:25:34.461417 1 leaderelection.go:252] successfully acquired lease submariner-operator/submariner-engine-lock
  2504. 2020-08-25T18:33:59.7279627Z [cluster1] I0825 18:25:34.466646 1 main.go:134] Creating the tunnel controller
  2505. 2020-08-25T18:33:59.7279937Z [cluster1] I0825 18:25:34.466709 1 main.go:142] Creating the kubernetes central datastore
  2506. 2020-08-25T18:33:59.7280158Z [cluster1] I0825 18:25:34.466741 1 kubernetes.go:60] Rendered API server host: "https://172.17.0.4:6443"
  2507. 2020-08-25T18:33:59.7280953Z [cluster1] I0825 18:25:34.498660 1 event.go:281] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"submariner-operator", Name:"submariner-engine-lock", UID:"92dba329-c5cd-48cf-bc32-ab0d34f96974", APIVersion:"v1", ResourceVersion:"1254", FieldPath:""}): type: 'Normal' reason: 'LeaderElection' cluster1-worker-submariner-engine became leader
  2508. 2020-08-25T18:33:59.7281166Z [cluster1] I0825 18:25:34.701422 1 main.go:152] Creating the datastore syncer
  2509. 2020-08-25T18:33:59.7281328Z [cluster1] I0825 18:25:34.701537 1 datastoresyncer.go:145] Starting the datastore syncer
  2510. 2020-08-25T18:33:59.7281800Z [cluster1] I0825 18:25:34.701549 1 datastoresyncer.go:147] Waiting for informer caches to sync
  2511. 2020-08-25T18:33:59.7283460Z [cluster1] I0825 18:25:34.702198 1 strongswan.go:97] Initializing StrongSwan IPSec driver
  2512. 2020-08-25T18:33:59.7283639Z [cluster1] I0825 18:25:34.702650 1 tunnel.go:47] Starting the tunnel controller
  2513. 2020-08-25T18:33:59.7283792Z [cluster1] I0825 18:25:34.702658 1 tunnel.go:50] Waiting for informer caches to sync
  2514. 2020-08-25T18:33:59.7284313Z [cluster1] I0825 18:25:34.702932 1 strongswan.go:458] Starting charon
  2515. 2020-08-25T18:33:59.7285580Z [cluster1] W0825 18:25:34.730727 1 strongswan.go:541] Failed to connect to charon - retrying: dial unix /var/run/charon.vici: connect: no such file or directory
  2516. 2020-08-25T18:33:59.7286412Z [cluster1] I0825 18:25:34.801710 1 datastoresyncer.go:79] Ensuring we are the only endpoint active for this cluster
  2517. 2020-08-25T18:33:59.7286577Z [cluster1] I0825 18:25:34.803816 1 tunnel.go:58] Tunnel controller started
  2518. 2020-08-25T18:33:59.7286947Z [cluster1] I0825 18:25:34.821896 1 datastoresyncer.go:158] Reconciling local submariner Cluster: types.SubmarinerCluster{ID:"cluster1", Spec:v1.ClusterSpec{ClusterID:"cluster1", ColorCodes:[]string{"blue"}, ServiceCIDR:[]string{"100.91.0.0/16"}, ClusterCIDR:[]string{"10.241.0.0/16"}, GlobalCIDR:[]string{}}}
  2519. 2020-08-25T18:33:59.7287213Z [cluster1] I0825 18:25:34.821923 1 datastoresyncer.go:312] In reconcileClusterCRD: &types.SubmarinerCluster{ID:"cluster1", Spec:v1.ClusterSpec{ClusterID:"cluster1", ColorCodes:[]string{"blue"}, ServiceCIDR:[]string{"100.91.0.0/16"}, ClusterCIDR:[]string{"10.241.0.0/16"}, GlobalCIDR:[]string{}}}
  2520. 2020-08-25T18:33:59.7287416Z [cluster1] I0825 18:25:34.854972 1 datastoresyncer.go:357] Successfully created submariner Cluster "cluster1" in the local datastore
  2521. 2020-08-25T18:33:59.7288370Z [cluster1] I0825 18:25:34.855000 1 datastoresyncer.go:164] Reconciling local submariner Endpoint: types.SubmarinerEndpoint{Spec:v1.EndpointSpec{ClusterID:"cluster1", CableName:"submariner-cable-cluster1-172-17-0-9", Hostname:"cluster1-worker", Subnets:[]string{"100.91.0.0/16", "10.241.0.0/16"}, PrivateIP:"172.17.0.9", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string{}}}
  2522. 2020-08-25T18:33:59.7289358Z [cluster1] I0825 18:25:34.855049 1 datastoresyncer.go:387] In reconcileEndpointCRD: &types.SubmarinerEndpoint{Spec:v1.EndpointSpec{ClusterID:"cluster1", CableName:"submariner-cable-cluster1-172-17-0-9", Hostname:"cluster1-worker", Subnets:[]string{"100.91.0.0/16", "10.241.0.0/16"}, PrivateIP:"172.17.0.9", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string{}}}
  2523. 2020-08-25T18:33:59.7291041Z [cluster1] I0825 18:25:35.185819 1 tunnel.go:95] Tunnel controller processing added or updated submariner Endpoint object: &v1.Endpoint{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"cluster1-submariner-cable-cluster1-172-17-0-9", GenerateName:"", Namespace:"submariner-operator", SelfLink:"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster1-submariner-cable-cluster1-172-17-0-9", UID:"8e28b57c-62e8-4985-9cc5-94c0847f8748", ResourceVersion:"1258", Generation:1, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63733976735, loc:(*time.Location)(0x2155f60)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v1.EndpointSpec{ClusterID:"cluster1", CableName:"submariner-cable-cluster1-172-17-0-9", Hostname:"cluster1-worker", Subnets:[]string{"100.91.0.0/16", "10.241.0.0/16"}, PrivateIP:"172.17.0.9", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  2524. 2020-08-25T18:33:59.7291544Z [cluster1] I0825 18:25:35.185933 1 cableengine.go:94] Not installing cable for local cluster
  2525. 2020-08-25T18:33:59.7292019Z [cluster1] I0825 18:25:35.185940 1 tunnel.go:108] Tunnel controller successfully installed Endpoint cable submariner-cable-cluster1-172-17-0-9 in the engine
  2526. 2020-08-25T18:33:59.7292499Z [cluster1] I0825 18:25:35.209248 1 datastoresyncer.go:432] Successfully created submariner Endpoint "cluster1-submariner-cable-cluster1-172-17-0-9" in the local datastore
  2527. 2020-08-25T18:33:59.7292670Z [cluster1] I0825 18:25:35.209349 1 datastoresyncer.go:177] Datastore syncer started
  2528. 2020-08-25T18:33:59.7292871Z [cluster1] I0825 18:25:35.209889 1 kubernetes.go:317] In SetCluster: &types.SubmarinerCluster{ID:"cluster1", Spec:v1.ClusterSpec{ClusterID:"cluster1", ColorCodes:[]string{"blue"}, ServiceCIDR:[]string{"100.91.0.0/16"}, ClusterCIDR:[]string{"10.241.0.0/16"}, GlobalCIDR:[]string{}}}
  2529. 2020-08-25T18:33:59.7293059Z [cluster1] I0825 18:25:35.735660 1 cableengine.go:73] CableEngine controller started, driver: "strongswan"
  2530. 2020-08-25T18:33:59.7296369Z [cluster1] I0825 18:25:35.779413 1 datastoresyncer.go:287] Processing local submariner Endpoint object: &v1.Endpoint{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"cluster1-submariner-cable-cluster1-172-17-0-9", GenerateName:"", Namespace:"submariner-operator", SelfLink:"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster1-submariner-cable-cluster1-172-17-0-9", UID:"8e28b57c-62e8-4985-9cc5-94c0847f8748", ResourceVersion:"1258", Generation:1, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63733976735, loc:(*time.Location)(0x2155f60)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v1.EndpointSpec{ClusterID:"cluster1", CableName:"submariner-cable-cluster1-172-17-0-9", Hostname:"cluster1-worker", Subnets:[]string{"100.91.0.0/16", "10.241.0.0/16"}, PrivateIP:"172.17.0.9", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  2531. 2020-08-25T18:33:59.7297387Z [cluster1] I0825 18:25:35.779592 1 kubernetes.go:370] In SetEndpoint: &types.SubmarinerEndpoint{Spec:v1.EndpointSpec{ClusterID:"cluster1", CableName:"submariner-cable-cluster1-172-17-0-9", Hostname:"cluster1-worker", Subnets:[]string{"100.91.0.0/16", "10.241.0.0/16"}, PrivateIP:"172.17.0.9", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  2532. 2020-08-25T18:33:59.7297592Z [cluster1] I0825 18:25:35.852769 1 kubernetes.go:240] AddFunc in WatchEndpoints called
  2533. 2020-08-25T18:33:59.7298045Z [cluster1] I0825 18:25:35.854645 1 kubernetes.go:395] Successfully created submariner Endpoint "cluster1-submariner-cable-cluster1-172-17-0-9" in the central datastore
  2534. 2020-08-25T18:33:59.7298219Z [cluster1] I0825 18:25:35.863491 1 kubernetes.go:155] AddFunc in WatchClusters called
  2535. 2020-08-25T18:33:59.7298387Z [cluster1] I0825 18:25:35.869407 1 kubernetes.go:342] Successfully created submariner Cluster "cluster1" in the central datastore
  2536. 2020-08-25T18:33:59.7298645Z [cluster1] I0825 18:25:36.662783 1 kubernetes.go:155] AddFunc in WatchClusters called
  2537. 2020-08-25T18:33:59.7299282Z [cluster1] I0825 18:25:36.662804 1 datastoresyncer.go:312] In reconcileClusterCRD: &types.SubmarinerCluster{ID:"cluster2", Spec:v1.ClusterSpec{ClusterID:"cluster2", ColorCodes:[]string{"blue"}, ServiceCIDR:[]string{"100.92.0.0/16"}, ClusterCIDR:[]string{"10.242.0.0/16"}, GlobalCIDR:[]string{}}}
  2538. 2020-08-25T18:33:59.7299529Z [cluster1] I0825 18:25:36.905089 1 kubernetes.go:240] AddFunc in WatchEndpoints called
  2539. 2020-08-25T18:33:59.7300359Z [cluster1] I0825 18:25:36.905109 1 datastoresyncer.go:387] In reconcileEndpointCRD: &types.SubmarinerEndpoint{Spec:v1.EndpointSpec{ClusterID:"cluster2", CableName:"submariner-cable-cluster2-172-17-0-8", Hostname:"cluster2-worker", Subnets:[]string{"100.92.0.0/16", "10.242.0.0/16"}, PrivateIP:"172.17.0.8", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  2540. 2020-08-25T18:33:59.7300590Z [cluster1] I0825 18:25:36.917959 1 datastoresyncer.go:357] Successfully created submariner Cluster "cluster2" in the local datastore
  2541. 2020-08-25T18:33:59.7301054Z [cluster1] I0825 18:25:37.759659 1 datastoresyncer.go:432] Successfully created submariner Endpoint "cluster2-submariner-cable-cluster2-172-17-0-8" in the local datastore
  2542. 2020-08-25T18:33:59.7302886Z [cluster1] I0825 18:25:37.768211 1 tunnel.go:95] Tunnel controller processing added or updated submariner Endpoint object: &v1.Endpoint{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"cluster2-submariner-cable-cluster2-172-17-0-8", GenerateName:"", Namespace:"submariner-operator", SelfLink:"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster2-submariner-cable-cluster2-172-17-0-8", UID:"4b2f98df-9143-4c8b-b359-f4d3fa6d05bc", ResourceVersion:"1270", Generation:1, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63733976737, loc:(*time.Location)(0x2155f60)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v1.EndpointSpec{ClusterID:"cluster2", CableName:"submariner-cable-cluster2-172-17-0-8", Hostname:"cluster2-worker", Subnets:[]string{"100.92.0.0/16", "10.242.0.0/16"}, PrivateIP:"172.17.0.8", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  2543. 2020-08-25T18:33:59.7303558Z [cluster1] I0825 18:25:37.768727 1 cableengine.go:103] Installing Endpoint cable "submariner-cable-cluster2-172-17-0-8"
  2544. 2020-08-25T18:33:59.7303999Z [cluster1] I0825 18:25:37.816308 1 cableengine.go:131] Successfully installed Endpoint cable "submariner-cable-cluster2-172-17-0-8" with remote IP 172.17.0.8
  2545. 2020-08-25T18:33:59.7304444Z [cluster1] I0825 18:25:37.816350 1 tunnel.go:108] Tunnel controller successfully installed Endpoint cable submariner-cable-cluster2-172-17-0-8 in the engine
  2546. 2020-08-25T18:33:59.7305316Z [cluster1] I0825 18:25:37.816678 1 datastoresyncer.go:271] The updated submariner Endpoint "cluster2" is not for this cluster - skipping updating the datastore
  2547. 2020-08-25T18:33:59.7305664Z [cluster1] I0825 18:26:05.224173 1 kubernetes.go:180] UpdateFunc in WatchClusters called
  2548. 2020-08-25T18:33:59.7305813Z [cluster1] I0825 18:26:05.224217 1 kubernetes.go:180] UpdateFunc in WatchClusters called
  2549. 2020-08-25T18:33:59.7306010Z [cluster1] I0825 18:26:05.224225 1 datastoresyncer.go:312] In reconcileClusterCRD: &types.SubmarinerCluster{ID:"cluster2", Spec:v1.ClusterSpec{ClusterID:"cluster2", ColorCodes:[]string{"blue"}, ServiceCIDR:[]string{"100.92.0.0/16"}, ClusterCIDR:[]string{"10.242.0.0/16"}, GlobalCIDR:[]string{}}}
  2550. 2020-08-25T18:33:59.7306451Z [cluster1] I0825 18:26:05.226526 1 datastoresyncer.go:360] Cluster "cluster2" matched what we received from datastore - not updating
  2551. 2020-08-25T18:33:59.7306743Z [cluster1] I0825 18:26:05.226893 1 kubernetes.go:264] UpdateFunc in WatchEndpoints called
  2552. 2020-08-25T18:33:59.7306901Z [cluster1] I0825 18:26:05.227245 1 kubernetes.go:264] UpdateFunc in WatchEndpoints called
  2553. 2020-08-25T18:33:59.7307805Z [cluster1] I0825 18:26:05.227253 1 datastoresyncer.go:387] In reconcileEndpointCRD: &types.SubmarinerEndpoint{Spec:v1.EndpointSpec{ClusterID:"cluster2", CableName:"submariner-cable-cluster2-172-17-0-8", Hostname:"cluster2-worker", Subnets:[]string{"100.92.0.0/16", "10.242.0.0/16"}, PrivateIP:"172.17.0.8", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  2554. 2020-08-25T18:33:59.7308307Z [cluster1] I0825 18:26:05.229440 1 datastoresyncer.go:435] Endpoint "cluster2-submariner-cable-cluster2-172-17-0-8" matched what we received from datastore - not updating
  2555. 2020-08-25T18:33:59.7308524Z [cluster1] I0825 18:26:34.720754 1 kubernetes.go:317] In SetCluster: &types.SubmarinerCluster{ID:"cluster1", Spec:v1.ClusterSpec{ClusterID:"cluster1", ColorCodes:[]string{"blue"}, ServiceCIDR:[]string{"100.91.0.0/16"}, ClusterCIDR:[]string{"10.241.0.0/16"}, GlobalCIDR:[]string{}}}
  2556. 2020-08-25T18:33:59.7308960Z [cluster1] I0825 18:26:34.724260 1 kubernetes.go:345] Cluster "cluster1" matched what we received from datastore - not updating
  2557. 2020-08-25T18:33:59.7311272Z [cluster1] I0825 18:26:34.728358 1 tunnel.go:95] Tunnel controller processing added or updated submariner Endpoint object: &v1.Endpoint{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"cluster1-submariner-cable-cluster1-172-17-0-9", GenerateName:"", Namespace:"submariner-operator", SelfLink:"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster1-submariner-cable-cluster1-172-17-0-9", UID:"8e28b57c-62e8-4985-9cc5-94c0847f8748", ResourceVersion:"1258", Generation:1, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63733976735, loc:(*time.Location)(0x2155f60)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v1.EndpointSpec{ClusterID:"cluster1", CableName:"submariner-cable-cluster1-172-17-0-9", Hostname:"cluster1-worker", Subnets:[]string{"100.91.0.0/16", "10.241.0.0/16"}, PrivateIP:"172.17.0.9", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  2558. 2020-08-25T18:33:59.7311683Z [cluster1] I0825 18:26:34.728704 1 cableengine.go:94] Not installing cable for local cluster
  2559. 2020-08-25T18:33:59.7312238Z [cluster1] I0825 18:26:34.728727 1 tunnel.go:108] Tunnel controller successfully installed Endpoint cable submariner-cable-cluster1-172-17-0-9 in the engine
  2560. 2020-08-25T18:33:59.7314037Z [cluster1] I0825 18:26:34.728735 1 tunnel.go:95] Tunnel controller processing added or updated submariner Endpoint object: &v1.Endpoint{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"cluster2-submariner-cable-cluster2-172-17-0-8", GenerateName:"", Namespace:"submariner-operator", SelfLink:"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster2-submariner-cable-cluster2-172-17-0-8", UID:"4b2f98df-9143-4c8b-b359-f4d3fa6d05bc", ResourceVersion:"1270", Generation:1, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63733976737, loc:(*time.Location)(0x2155f60)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v1.EndpointSpec{ClusterID:"cluster2", CableName:"submariner-cable-cluster2-172-17-0-8", Hostname:"cluster2-worker", Subnets:[]string{"100.92.0.0/16", "10.242.0.0/16"}, PrivateIP:"172.17.0.8", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  2561. 2020-08-25T18:33:59.7314727Z [cluster1] I0825 18:26:34.728771 1 cableengine.go:103] Installing Endpoint cable "submariner-cable-cluster2-172-17-0-8"
  2562. 2020-08-25T18:33:59.7316484Z [cluster1] I0825 18:26:34.731334 1 datastoresyncer.go:287] Processing local submariner Endpoint object: &v1.Endpoint{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"cluster1-submariner-cable-cluster1-172-17-0-9", GenerateName:"", Namespace:"submariner-operator", SelfLink:"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster1-submariner-cable-cluster1-172-17-0-9", UID:"8e28b57c-62e8-4985-9cc5-94c0847f8748", ResourceVersion:"1258", Generation:1, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63733976735, loc:(*time.Location)(0x2155f60)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v1.EndpointSpec{ClusterID:"cluster1", CableName:"submariner-cable-cluster1-172-17-0-9", Hostname:"cluster1-worker", Subnets:[]string{"100.91.0.0/16", "10.241.0.0/16"}, PrivateIP:"172.17.0.9", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  2563. 2020-08-25T18:33:59.7318136Z [cluster1] I0825 18:26:34.731378 1 kubernetes.go:370] In SetEndpoint: &types.SubmarinerEndpoint{Spec:v1.EndpointSpec{ClusterID:"cluster1", CableName:"submariner-cable-cluster1-172-17-0-9", Hostname:"cluster1-worker", Subnets:[]string{"100.91.0.0/16", "10.241.0.0/16"}, PrivateIP:"172.17.0.9", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  2564. 2020-08-25T18:33:59.7318809Z [cluster1] I0825 18:26:34.733901 1 kubernetes.go:398] Endpoint "cluster1-submariner-cable-cluster1-172-17-0-9" matched what we received from datastore - not updating
  2565. 2020-08-25T18:33:59.7319257Z [cluster1] I0825 18:26:34.735197 1 cableengine.go:117] Cable "submariner-cable-cluster2-172-17-0-8" is already installed - not installing again
  2566. 2020-08-25T18:33:59.7319839Z [cluster1] I0825 18:26:34.735248 1 tunnel.go:108] Tunnel controller successfully installed Endpoint cable submariner-cable-cluster2-172-17-0-8 in the engine
  2567. 2020-08-25T18:33:59.7320333Z [cluster1] I0825 18:26:34.735642 1 datastoresyncer.go:271] The updated submariner Endpoint "cluster2" is not for this cluster - skipping updating the datastore
  2568. 2020-08-25T18:33:59.7320507Z [cluster1] I0825 18:26:35.224264 1 kubernetes.go:180] UpdateFunc in WatchClusters called
  2569. 2020-08-25T18:33:59.7320663Z [cluster1] I0825 18:26:35.224310 1 kubernetes.go:180] UpdateFunc in WatchClusters called
  2570. 2020-08-25T18:33:59.7320856Z [cluster1] I0825 18:26:35.224317 1 datastoresyncer.go:312] In reconcileClusterCRD: &types.SubmarinerCluster{ID:"cluster2", Spec:v1.ClusterSpec{ClusterID:"cluster2", ColorCodes:[]string{"blue"}, ServiceCIDR:[]string{"100.92.0.0/16"}, ClusterCIDR:[]string{"10.242.0.0/16"}, GlobalCIDR:[]string{}}}
  2571. 2020-08-25T18:33:59.7321076Z [cluster1] I0825 18:26:35.227399 1 kubernetes.go:264] UpdateFunc in WatchEndpoints called
  2572. 2020-08-25T18:33:59.7321217Z [cluster1] I0825 18:26:35.227425 1 kubernetes.go:264] UpdateFunc in WatchEndpoints called
  2573. 2020-08-25T18:33:59.7321952Z [cluster1] I0825 18:26:35.227523 1 datastoresyncer.go:387] In reconcileEndpointCRD: &types.SubmarinerEndpoint{Spec:v1.EndpointSpec{ClusterID:"cluster2", CableName:"submariner-cable-cluster2-172-17-0-8", Hostname:"cluster2-worker", Subnets:[]string{"100.92.0.0/16", "10.242.0.0/16"}, PrivateIP:"172.17.0.8", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  2574. 2020-08-25T18:33:59.7322455Z [cluster1] I0825 18:26:35.236547 1 datastoresyncer.go:435] Endpoint "cluster2-submariner-cable-cluster2-172-17-0-8" matched what we received from datastore - not updating
  2575. 2020-08-25T18:33:59.7322993Z [cluster1] I0825 18:26:35.237417 1 datastoresyncer.go:360] Cluster "cluster2" matched what we received from datastore - not updating
  2576. 2020-08-25T18:33:59.7323164Z [cluster1] I0825 18:27:05.224377 1 kubernetes.go:180] UpdateFunc in WatchClusters called
  2577. 2020-08-25T18:33:59.7323316Z [cluster1] I0825 18:27:05.224407 1 kubernetes.go:180] UpdateFunc in WatchClusters called
  2578. 2020-08-25T18:33:59.7323516Z [cluster1] I0825 18:27:05.224413 1 datastoresyncer.go:312] In reconcileClusterCRD: &types.SubmarinerCluster{ID:"cluster2", Spec:v1.ClusterSpec{ClusterID:"cluster2", ColorCodes:[]string{"blue"}, ServiceCIDR:[]string{"100.92.0.0/16"}, ClusterCIDR:[]string{"10.242.0.0/16"}, GlobalCIDR:[]string{}}}
  2579. 2020-08-25T18:33:59.7323702Z [cluster1] I0825 18:27:05.227334 1 kubernetes.go:264] UpdateFunc in WatchEndpoints called
  2580. 2020-08-25T18:33:59.7323920Z [cluster1] I0825 18:27:05.227387 1 kubernetes.go:264] UpdateFunc in WatchEndpoints called
  2581. 2020-08-25T18:33:59.7324638Z [cluster1] I0825 18:27:05.227393 1 datastoresyncer.go:387] In reconcileEndpointCRD: &types.SubmarinerEndpoint{Spec:v1.EndpointSpec{ClusterID:"cluster2", CableName:"submariner-cable-cluster2-172-17-0-8", Hostname:"cluster2-worker", Subnets:[]string{"100.92.0.0/16", "10.242.0.0/16"}, PrivateIP:"172.17.0.8", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  2582. 2020-08-25T18:33:59.7325100Z [cluster1] I0825 18:27:05.228555 1 datastoresyncer.go:360] Cluster "cluster2" matched what we received from datastore - not updating
  2583. 2020-08-25T18:33:59.7326083Z [cluster1] I0825 18:27:05.230194 1 datastoresyncer.go:435] Endpoint "cluster2-submariner-cable-cluster2-172-17-0-8" matched what we received from datastore - not updating
  2584. 2020-08-25T18:33:59.7326326Z [cluster1] I0825 18:27:34.723833 1 kubernetes.go:317] In SetCluster: &types.SubmarinerCluster{ID:"cluster1", Spec:v1.ClusterSpec{ClusterID:"cluster1", ColorCodes:[]string{"blue"}, ServiceCIDR:[]string{"100.91.0.0/16"}, ClusterCIDR:[]string{"10.241.0.0/16"}, GlobalCIDR:[]string{}}}
  2585. 2020-08-25T18:33:59.7326918Z [cluster1] I0825 18:27:34.725958 1 kubernetes.go:345] Cluster "cluster1" matched what we received from datastore - not updating
  2586. 2020-08-25T18:33:59.7328583Z [cluster1] I0825 18:27:34.729096 1 tunnel.go:95] Tunnel controller processing added or updated submariner Endpoint object: &v1.Endpoint{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"cluster1-submariner-cable-cluster1-172-17-0-9", GenerateName:"", Namespace:"submariner-operator", SelfLink:"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster1-submariner-cable-cluster1-172-17-0-9", UID:"8e28b57c-62e8-4985-9cc5-94c0847f8748", ResourceVersion:"1258", Generation:1, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63733976735, loc:(*time.Location)(0x2155f60)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v1.EndpointSpec{ClusterID:"cluster1", CableName:"submariner-cable-cluster1-172-17-0-9", Hostname:"cluster1-worker", Subnets:[]string{"100.91.0.0/16", "10.241.0.0/16"}, PrivateIP:"172.17.0.9", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  2587. 2020-08-25T18:33:59.7329129Z [cluster1] I0825 18:27:34.729181 1 cableengine.go:94] Not installing cable for local cluster
  2588. 2020-08-25T18:33:59.7329557Z [cluster1] I0825 18:27:34.729217 1 tunnel.go:108] Tunnel controller successfully installed Endpoint cable submariner-cable-cluster1-172-17-0-9 in the engine
  2589. 2020-08-25T18:33:59.7331429Z [cluster1] I0825 18:27:34.729224 1 tunnel.go:95] Tunnel controller processing added or updated submariner Endpoint object: &v1.Endpoint{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"cluster2-submariner-cable-cluster2-172-17-0-8", GenerateName:"", Namespace:"submariner-operator", SelfLink:"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster2-submariner-cable-cluster2-172-17-0-8", UID:"4b2f98df-9143-4c8b-b359-f4d3fa6d05bc", ResourceVersion:"1270", Generation:1, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63733976737, loc:(*time.Location)(0x2155f60)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v1.EndpointSpec{ClusterID:"cluster2", CableName:"submariner-cable-cluster2-172-17-0-8", Hostname:"cluster2-worker", Subnets:[]string{"100.92.0.0/16", "10.242.0.0/16"}, PrivateIP:"172.17.0.8", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  2590. 2020-08-25T18:33:59.7332277Z [cluster1] I0825 18:27:34.729251 1 cableengine.go:103] Installing Endpoint cable "submariner-cable-cluster2-172-17-0-8"
  2591. 2020-08-25T18:33:59.7332703Z [cluster1] I0825 18:27:34.731836 1 cableengine.go:117] Cable "submariner-cable-cluster2-172-17-0-8" is already installed - not installing again
  2592. 2020-08-25T18:33:59.7333129Z [cluster1] I0825 18:27:34.731854 1 tunnel.go:108] Tunnel controller successfully installed Endpoint cable submariner-cable-cluster2-172-17-0-8 in the engine
  2593. 2020-08-25T18:33:59.7334748Z [cluster1] I0825 18:27:34.732882 1 datastoresyncer.go:287] Processing local submariner Endpoint object: &v1.Endpoint{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"cluster1-submariner-cable-cluster1-172-17-0-9", GenerateName:"", Namespace:"submariner-operator", SelfLink:"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster1-submariner-cable-cluster1-172-17-0-9", UID:"8e28b57c-62e8-4985-9cc5-94c0847f8748", ResourceVersion:"1258", Generation:1, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63733976735, loc:(*time.Location)(0x2155f60)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v1.EndpointSpec{ClusterID:"cluster1", CableName:"submariner-cable-cluster1-172-17-0-9", Hostname:"cluster1-worker", Subnets:[]string{"100.91.0.0/16", "10.241.0.0/16"}, PrivateIP:"172.17.0.9", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  2594. 2020-08-25T18:33:59.7336418Z [cluster1] I0825 18:27:34.732935 1 kubernetes.go:370] In SetEndpoint: &types.SubmarinerEndpoint{Spec:v1.EndpointSpec{ClusterID:"cluster1", CableName:"submariner-cable-cluster1-172-17-0-9", Hostname:"cluster1-worker", Subnets:[]string{"100.91.0.0/16", "10.241.0.0/16"}, PrivateIP:"172.17.0.9", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  2595. 2020-08-25T18:33:59.7336935Z [cluster1] I0825 18:27:34.735042 1 kubernetes.go:398] Endpoint "cluster1-submariner-cable-cluster1-172-17-0-9" matched what we received from datastore - not updating
  2596. 2020-08-25T18:33:59.7408223Z [cluster1] I0825 18:27:34.736524 1 datastoresyncer.go:271] The updated submariner Endpoint "cluster2" is not for this cluster - skipping updating the datastore
  2597. 2020-08-25T18:33:59.7408616Z [cluster1] I0825 18:27:35.224554 1 kubernetes.go:180] UpdateFunc in WatchClusters called
  2598. 2020-08-25T18:33:59.7408776Z [cluster1] I0825 18:27:35.224602 1 kubernetes.go:180] UpdateFunc in WatchClusters called
  2599. 2020-08-25T18:33:59.7408998Z [cluster1] I0825 18:27:35.224610 1 datastoresyncer.go:312] In reconcileClusterCRD: &types.SubmarinerCluster{ID:"cluster2", Spec:v1.ClusterSpec{ClusterID:"cluster2", ColorCodes:[]string{"blue"}, ServiceCIDR:[]string{"100.92.0.0/16"}, ClusterCIDR:[]string{"10.242.0.0/16"}, GlobalCIDR:[]string{}}}
  2600. 2020-08-25T18:33:59.7410392Z [cluster1] I0825 18:27:35.226861 1 datastoresyncer.go:360] Cluster "cluster2" matched what we received from datastore - not updating
  2601. 2020-08-25T18:33:59.7410815Z [cluster1] I0825 18:27:35.227506 1 kubernetes.go:264] UpdateFunc in WatchEndpoints called
  2602. 2020-08-25T18:33:59.7410976Z [cluster1] I0825 18:27:35.227520 1 kubernetes.go:264] UpdateFunc in WatchEndpoints called
  2603. 2020-08-25T18:33:59.7411823Z [cluster1] I0825 18:27:35.227526 1 datastoresyncer.go:387] In reconcileEndpointCRD: &types.SubmarinerEndpoint{Spec:v1.EndpointSpec{ClusterID:"cluster2", CableName:"submariner-cable-cluster2-172-17-0-8", Hostname:"cluster2-worker", Subnets:[]string{"100.92.0.0/16", "10.242.0.0/16"}, PrivateIP:"172.17.0.8", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  2604. 2020-08-25T18:33:59.7422543Z [cluster1] I0825 18:27:35.229231 1 datastoresyncer.go:435] Endpoint "cluster2-submariner-cable-cluster2-172-17-0-8" matched what we received from datastore - not updating
  2605. 2020-08-25T18:33:59.7423343Z [cluster1] I0825 18:28:05.224608 1 kubernetes.go:180] UpdateFunc in WatchClusters called
  2606. 2020-08-25T18:33:59.7423655Z [cluster1] I0825 18:28:05.224695 1 kubernetes.go:180] UpdateFunc in WatchClusters called
  2607. 2020-08-25T18:33:59.7424152Z [cluster1] I0825 18:28:05.224719 1 datastoresyncer.go:312] In reconcileClusterCRD: &types.SubmarinerCluster{ID:"cluster2", Spec:v1.ClusterSpec{ClusterID:"cluster2", ColorCodes:[]string{"blue"}, ServiceCIDR:[]string{"100.92.0.0/16"}, ClusterCIDR:[]string{"10.242.0.0/16"}, GlobalCIDR:[]string{}}}
  2608. 2020-08-25T18:33:59.7424440Z [cluster1] I0825 18:28:05.227624 1 kubernetes.go:264] UpdateFunc in WatchEndpoints called
  2609. 2020-08-25T18:33:59.7425221Z [cluster1] I0825 18:28:05.227712 1 kubernetes.go:264] UpdateFunc in WatchEndpoints called
  2610. 2020-08-25T18:33:59.7426308Z [cluster1] I0825 18:28:05.227741 1 datastoresyncer.go:387] In reconcileEndpointCRD: &types.SubmarinerEndpoint{Spec:v1.EndpointSpec{ClusterID:"cluster2", CableName:"submariner-cable-cluster2-172-17-0-8", Hostname:"cluster2-worker", Subnets:[]string{"100.92.0.0/16", "10.242.0.0/16"}, PrivateIP:"172.17.0.8", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  2611. 2020-08-25T18:33:59.7427213Z [cluster1] I0825 18:28:05.228920 1 datastoresyncer.go:360] Cluster "cluster2" matched what we received from datastore - not updating
  2612. 2020-08-25T18:33:59.7427854Z [cluster1] I0825 18:28:05.229966 1 datastoresyncer.go:435] Endpoint "cluster2-submariner-cable-cluster2-172-17-0-8" matched what we received from datastore - not updating
  2613. 2020-08-25T18:33:59.7428563Z [cluster1] 00[DMN] Starting IKE charon daemon (strongSwan 5.8.4, Linux 5.3.0-1034-azure, x86_64)
  2614. 2020-08-25T18:33:59.7429080Z [cluster1] 00[CFG] PKCS11 module '<name>' lacks library path
  2615. 2020-08-25T18:33:59.7429542Z [cluster1] 00[LIB] openssl FIPS mode(2) - enabled
  2616. 2020-08-25T18:33:59.7430029Z [cluster1] 00[CFG] loading ca certificates from '/etc/strongswan/ipsec.d/cacerts'
  2617. 2020-08-25T18:33:59.7430536Z [cluster1] 00[CFG] loading aa certificates from '/etc/strongswan/ipsec.d/aacerts'
  2618. 2020-08-25T18:33:59.7431687Z [cluster1] 00[CFG] loading ocsp signer certificates from '/etc/strongswan/ipsec.d/ocspcerts'
  2619. 2020-08-25T18:33:59.7432335Z [cluster1] 00[CFG] loading attribute certificates from '/etc/strongswan/ipsec.d/acerts'
  2620. 2020-08-25T18:33:59.7433935Z [cluster1] 00[CFG] loading crls from '/etc/strongswan/ipsec.d/crls'
  2621. 2020-08-25T18:33:59.7435671Z [cluster1] 00[CFG] loading secrets from '/etc/strongswan/ipsec.secrets'
  2622. 2020-08-25T18:33:59.7437262Z [cluster1] 00[CFG] opening triplet file /etc/strongswan/ipsec.d/triplets.dat failed: No such file or directory
  2623. 2020-08-25T18:33:59.7440032Z [cluster1] 00[CFG] loaded 0 RADIUS server configurations
  2624. 2020-08-25T18:33:59.7441988Z [cluster1] 00[CFG] HA config misses local/remote address
  2625. 2020-08-25T18:33:59.7442609Z [cluster1] 00[CFG] no script for ext-auth script defined, disabled
  2626. 2020-08-25T18:33:59.7449384Z [cluster1] 00[LIB] loaded plugins: charon pkcs11 tpm aesni aes des rc2 sha2 sha1 md4 md5 mgf1 random nonce x509 revocation constraints acert pubkey pkcs1 pkcs7 pkcs8 pkcs12 pgp dnskey sshkey pem openssl gcrypt fips-prf gmp curve25519 chapoly xcbc cmac hmac ctr ccm gcm drbg newhope curl attr kernel-netlink resolve socket-default farp stroke vici updown eap-identity eap-sim eap-aka eap-aka-3gpp eap-aka-3gpp2 eap-md5 eap-gtc eap-mschapv2 eap-dynamic eap-radius eap-tls eap-ttls eap-peap xauth-generic xauth-eap xauth-pam xauth-noauth dhcp led duplicheck unity counters
  2627. 2020-08-25T18:33:59.7450257Z [cluster1] 00[JOB] spawning 16 worker threads
  2628. 2020-08-25T18:33:59.7450591Z [cluster1] 12[NET] received packet: from 172.17.0.8[500] to 172.17.0.9[500] (500 bytes)
  2629. 2020-08-25T18:33:59.7450864Z [cluster1] 12[ENC] parsed IKE_SA_INIT request 0 [ SA KE No N(NATD_S_IP) N(NATD_D_IP) N(FRAG_SUP) N(HASH_ALG) N(REDIR_SUP) ]
  2630. 2020-08-25T18:33:59.7451133Z [cluster1] 12[IKE] no IKE config found for 172.17.0.9...172.17.0.8, sending NO_PROPOSAL_CHOSEN
  2631. 2020-08-25T18:33:59.7451384Z [cluster1] 12[ENC] generating IKE_SA_INIT response 0 [ N(NO_PROP) ]
  2632. 2020-08-25T18:33:59.7451641Z [cluster1] 12[NET] sending packet: from 172.17.0.9[500] to 172.17.0.8[500] (36 bytes)
  2633. 2020-08-25T18:33:59.7452163Z [cluster1] 09[CFG] loaded IKE shared key for: '172.17.0.8'
  2634. 2020-08-25T18:33:59.7452676Z [cluster1] 12[CFG] added vici connection: submariner-cable-cluster2-172-17-0-8
  2635. 2020-08-25T18:33:59.7453627Z [cluster1] 12[CFG] initiating 'submariner-child-submariner-cable-cluster2-172-17-0-8'
  2636. 2020-08-25T18:33:59.7454270Z [cluster1] 12[IKE] initiating IKE_SA submariner-cable-cluster2-172-17-0-8[2] to 172.17.0.8
  2637. 2020-08-25T18:33:59.7454574Z [cluster1] 12[ENC] generating IKE_SA_INIT request 0 [ SA KE No N(NATD_S_IP) N(NATD_D_IP) N(FRAG_SUP) N(HASH_ALG) N(REDIR_SUP) ]
  2638. 2020-08-25T18:33:59.7454830Z [cluster1] 12[NET] sending packet: from 172.17.0.9[500] to 172.17.0.8[500] (500 bytes)
  2639. 2020-08-25T18:33:59.7455081Z [cluster1] 09[NET] received packet: from 172.17.0.8[500] to 172.17.0.9[500] (464 bytes)
  2640. 2020-08-25T18:33:59.7455434Z [cluster1] 09[ENC] parsed IKE_SA_INIT response 0 [ SA KE No N(NATD_S_IP) N(NATD_D_IP) N(FRAG_SUP) N(HASH_ALG) N(CHDLESS_SUP) N(MULT_AUTH) ]
  2641. 2020-08-25T18:33:59.7455851Z [cluster1] 09[CFG] selected proposal: IKE:AES_GCM_16_128/PRF_HMAC_SHA2_256/MODP_2048
  2642. 2020-08-25T18:33:59.7456158Z [cluster1] 09[IKE] remote host is behind NAT
  2643. 2020-08-25T18:33:59.7466168Z [cluster1] 09[IKE] authentication of '172.17.0.9' (myself) with pre-shared key
  2644. 2020-08-25T18:33:59.7466760Z [lighthouse]$ [cluster1] kubectl get Gateway -A -o yaml
  2645. 2020-08-25T18:33:59.7467129Z [cluster1] 09[IKE] establishing CHILD_SA submariner-child-submariner-cable-cluster2-172-17-0-8{1}
  2646. 2020-08-25T18:33:59.7467638Z [cluster1] 09[ENC] generating IKE_AUTH request 1 [ IDi N(INIT_CONTACT) IDr AUTH SA TSi TSr N(MULT_AUTH) N(EAP_ONLY) N(MSG_ID_SYN_SUP) ]
  2647. 2020-08-25T18:33:59.7468057Z [cluster1] 09[NET] sending packet: from 172.17.0.9[4500] to 172.17.0.8[4500] (333 bytes)
  2648. 2020-08-25T18:33:59.7468897Z [cluster1] 11[NET] received packet: from 172.17.0.8[4500] to 172.17.0.9[4500] (257 bytes)
  2649. 2020-08-25T18:33:59.7469230Z [cluster1] 11[ENC] parsed IKE_AUTH response 1 [ IDr AUTH SA TSi TSr ]
  2650. 2020-08-25T18:33:59.7469800Z [cluster1] 11[IKE] authentication of '172.17.0.8' with pre-shared key successful
  2651. 2020-08-25T18:33:59.7470420Z [cluster1] 11[IKE] IKE_SA submariner-cable-cluster2-172-17-0-8[2] established between 172.17.0.9[172.17.0.9]...172.17.0.8[172.17.0.8]
  2652. 2020-08-25T18:33:59.7470700Z [cluster1] 11[IKE] scheduling rekeying in 13090s
  2653. 2020-08-25T18:33:59.7470926Z [cluster1] 11[IKE] maximum IKE_SA lifetime 14530s
  2654. 2020-08-25T18:33:59.7471159Z [cluster1] 11[CFG] selected proposal: ESP:AES_GCM_16_128/NO_EXT_SEQ
  2655. 2020-08-25T18:33:59.7471812Z [cluster1] 11[IKE] CHILD_SA submariner-child-submariner-cable-cluster2-172-17-0-8{1} established with SPIs c1537707_i c191023b_o and TS 10.241.0.0/16 100.91.0.0/16 172.17.0.9/32 === 10.242.0.0/16 100.92.0.0/16 172.17.0.8/32
  2656. 2020-08-25T18:33:59.7472937Z [lighthouse]$ [cluster1] kubectl get Gateway -A -o yaml
  2657. 2020-08-25T18:33:59.7473277Z [cluster1] 10[KNL] interface vethwepg7d5eea1 deleted
  2658. 2020-08-25T18:33:59.7473687Z [cluster1] 11[KNL] interface vethwepl7d5eea1 activated
  2659. 2020-08-25T18:33:59.7474177Z [cluster1] 12[KNL] interface vx-submariner activated
  2660. 2020-08-25T18:33:59.7474629Z [cluster1] 09[KNL] 240.17.0.9 appeared on vx-submariner
  2661. 2020-08-25T18:33:59.7477264Z [cluster1] 13[KNL] interface vethwepl7d5eea1 deactivated
  2662. 2020-08-25T18:33:59.7477402Z [cluster1] 15[KNL] interface vethwepl7d5eea1 deleted
  2663. 2020-08-25T18:33:59.7477528Z [cluster1] 01[KNL] interface vethwepg85eecb8 deleted
  2664. 2020-08-25T18:33:59.7477654Z [cluster1] 07[KNL] interface vethwepl85eecb8 activated
  2665. 2020-08-25T18:33:59.7477766Z [cluster1] 11[KNL] interface vethwepl85eecb8 deactivated
  2666. 2020-08-25T18:33:59.7477893Z [cluster1] 15[KNL] interface vethwepl85eecb8 deleted
  2667. 2020-08-25T18:33:59.7478280Z [cluster1] I0825 18:28:34.724214 1 kubernetes.go:317] In SetCluster: &types.SubmarinerCluster{ID:"cluster1", Spec:v1.ClusterSpec{ClusterID:"cluster1", ColorCodes:[]string{"blue"}, ServiceCIDR:[]string{"100.91.0.0/16"}, ClusterCIDR:[]string{"10.241.0.0/16"}, GlobalCIDR:[]string{}}}
  2668. 2020-08-25T18:33:59.7478822Z [cluster1] I0825 18:28:34.726944 1 kubernetes.go:345] Cluster "cluster1" matched what we received from datastore - not updating
  2669. 2020-08-25T18:33:59.7480614Z [cluster1] I0825 18:28:34.728951 1 tunnel.go:95] Tunnel controller processing added or updated submariner Endpoint object: &v1.Endpoint{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"cluster1-submariner-cable-cluster1-172-17-0-9", GenerateName:"", Namespace:"submariner-operator", SelfLink:"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster1-submariner-cable-cluster1-172-17-0-9", UID:"8e28b57c-62e8-4985-9cc5-94c0847f8748", ResourceVersion:"1258", Generation:1, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63733976735, loc:(*time.Location)(0x2155f60)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v1.EndpointSpec{ClusterID:"cluster1", CableName:"submariner-cable-cluster1-172-17-0-9", Hostname:"cluster1-worker", Subnets:[]string{"100.91.0.0/16", "10.241.0.0/16"}, PrivateIP:"172.17.0.9", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  2670. 2020-08-25T18:33:59.7481011Z [cluster1] I0825 18:28:34.729047 1 cableengine.go:94] Not installing cable for local cluster
  2671. 2020-08-25T18:33:59.7482345Z [cluster1] I0825 18:28:34.729054 1 tunnel.go:108] Tunnel controller successfully installed Endpoint cable submariner-cable-cluster1-172-17-0-9 in the engine
  2672. 2020-08-25T18:33:59.7484749Z [cluster1] I0825 18:28:34.729060 1 tunnel.go:95] Tunnel controller processing added or updated submariner Endpoint object: &v1.Endpoint{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"cluster2-submariner-cable-cluster2-172-17-0-8", GenerateName:"", Namespace:"submariner-operator", SelfLink:"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster2-submariner-cable-cluster2-172-17-0-8", UID:"4b2f98df-9143-4c8b-b359-f4d3fa6d05bc", ResourceVersion:"1270", Generation:1, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63733976737, loc:(*time.Location)(0x2155f60)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v1.EndpointSpec{ClusterID:"cluster2", CableName:"submariner-cable-cluster2-172-17-0-8", Hostname:"cluster2-worker", Subnets:[]string{"100.92.0.0/16", "10.242.0.0/16"}, PrivateIP:"172.17.0.8", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  2673. 2020-08-25T18:33:59.7485373Z [cluster1] I0825 18:28:34.729095 1 cableengine.go:103] Installing Endpoint cable "submariner-cable-cluster2-172-17-0-8"
  2674. 2020-08-25T18:33:59.7487636Z [cluster1] I0825 18:28:34.734948 1 datastoresyncer.go:287] Processing local submariner Endpoint object: &v1.Endpoint{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"cluster1-submariner-cable-cluster1-172-17-0-9", GenerateName:"", Namespace:"submariner-operator", SelfLink:"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster1-submariner-cable-cluster1-172-17-0-9", UID:"8e28b57c-62e8-4985-9cc5-94c0847f8748", ResourceVersion:"1258", Generation:1, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63733976735, loc:(*time.Location)(0x2155f60)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v1.EndpointSpec{ClusterID:"cluster1", CableName:"submariner-cable-cluster1-172-17-0-9", Hostname:"cluster1-worker", Subnets:[]string{"100.91.0.0/16", "10.241.0.0/16"}, PrivateIP:"172.17.0.9", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  2675. 2020-08-25T18:33:59.7488743Z [cluster1] I0825 18:28:34.734988 1 kubernetes.go:370] In SetEndpoint: &types.SubmarinerEndpoint{Spec:v1.EndpointSpec{ClusterID:"cluster1", CableName:"submariner-cable-cluster1-172-17-0-9", Hostname:"cluster1-worker", Subnets:[]string{"100.91.0.0/16", "10.241.0.0/16"}, PrivateIP:"172.17.0.9", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  2676. 2020-08-25T18:33:59.7489238Z [cluster1] I0825 18:28:34.736876 1 cableengine.go:117] Cable "submariner-cable-cluster2-172-17-0-8" is already installed - not installing again
  2677. 2020-08-25T18:33:59.7489679Z [cluster1] I0825 18:28:34.736892 1 tunnel.go:108] Tunnel controller successfully installed Endpoint cable submariner-cable-cluster2-172-17-0-8 in the engine
  2678. 2020-08-25T18:33:59.7490241Z [cluster1] I0825 18:28:34.741190 1 kubernetes.go:398] Endpoint "cluster1-submariner-cable-cluster1-172-17-0-9" matched what we received from datastore - not updating
  2679. 2020-08-25T18:33:59.7490727Z [cluster1] I0825 18:28:34.745002 1 datastoresyncer.go:271] The updated submariner Endpoint "cluster2" is not for this cluster - skipping updating the datastore
  2680. 2020-08-25T18:33:59.7490904Z [cluster1] I0825 18:28:35.224807 1 kubernetes.go:180] UpdateFunc in WatchClusters called
  2681. 2020-08-25T18:33:59.7491062Z [cluster1] I0825 18:28:35.224829 1 kubernetes.go:180] UpdateFunc in WatchClusters called
  2682. 2020-08-25T18:33:59.7491251Z [cluster1] I0825 18:28:35.224836 1 datastoresyncer.go:312] In reconcileClusterCRD: &types.SubmarinerCluster{ID:"cluster2", Spec:v1.ClusterSpec{ClusterID:"cluster2", ColorCodes:[]string{"blue"}, ServiceCIDR:[]string{"100.92.0.0/16"}, ClusterCIDR:[]string{"10.242.0.0/16"}, GlobalCIDR:[]string{}}}
  2683. 2020-08-25T18:33:59.7491724Z [cluster1] I0825 18:28:35.227403 1 datastoresyncer.go:360] Cluster "cluster2" matched what we received from datastore - not updating
  2684. 2020-08-25T18:33:59.7491894Z [cluster1] I0825 18:28:35.227756 1 kubernetes.go:264] UpdateFunc in WatchEndpoints called
  2685. 2020-08-25T18:33:59.7492048Z [cluster1] I0825 18:28:35.227776 1 kubernetes.go:264] UpdateFunc in WatchEndpoints called
  2686. 2020-08-25T18:33:59.7492762Z [cluster1] I0825 18:28:35.227781 1 datastoresyncer.go:387] In reconcileEndpointCRD: &types.SubmarinerEndpoint{Spec:v1.EndpointSpec{ClusterID:"cluster2", CableName:"submariner-cable-cluster2-172-17-0-8", Hostname:"cluster2-worker", Subnets:[]string{"100.92.0.0/16", "10.242.0.0/16"}, PrivateIP:"172.17.0.8", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  2687. 2020-08-25T18:33:59.7493649Z [cluster1] I0825 18:28:35.229829 1 datastoresyncer.go:435] Endpoint "cluster2-submariner-cable-cluster2-172-17-0-8" matched what we received from datastore - not updating
  2688. 2020-08-25T18:33:59.7494081Z [cluster1] I0825 18:29:05.225006 1 kubernetes.go:180] UpdateFunc in WatchClusters called
  2689. 2020-08-25T18:33:59.7494244Z [cluster1] I0825 18:29:05.225111 1 kubernetes.go:180] UpdateFunc in WatchClusters called
  2690. 2020-08-25T18:33:59.7494450Z [cluster1] I0825 18:29:05.225174 1 datastoresyncer.go:312] In reconcileClusterCRD: &types.SubmarinerCluster{ID:"cluster2", Spec:v1.ClusterSpec{ClusterID:"cluster2", ColorCodes:[]string{"blue"}, ServiceCIDR:[]string{"100.92.0.0/16"}, ClusterCIDR:[]string{"10.242.0.0/16"}, GlobalCIDR:[]string{}}}
  2691. 2020-08-25T18:33:59.7494637Z [cluster1] I0825 18:29:05.227863 1 kubernetes.go:264] UpdateFunc in WatchEndpoints called
  2692. 2020-08-25T18:33:59.7494792Z [cluster1] I0825 18:29:05.228038 1 kubernetes.go:264] UpdateFunc in WatchEndpoints called
  2693. 2020-08-25T18:33:59.7495318Z [cluster1] I0825 18:29:05.228327 1 datastoresyncer.go:360] Cluster "cluster2" matched what we received from datastore - not updating
  2694. 2020-08-25T18:33:59.7497564Z [cluster1] I0825 18:29:05.228055 1 datastoresyncer.go:387] In reconcileEndpointCRD: &types.SubmarinerEndpoint{Spec:v1.EndpointSpec{ClusterID:"cluster2", CableName:"submariner-cable-cluster2-172-17-0-8", Hostname:"cluster2-worker", Subnets:[]string{"100.92.0.0/16", "10.242.0.0/16"}, PrivateIP:"172.17.0.8", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  2695. 2020-08-25T18:33:59.7498235Z [cluster1] I0825 18:29:05.230700 1 datastoresyncer.go:435] Endpoint "cluster2-submariner-cable-cluster2-172-17-0-8" matched what we received from datastore - not updating
  2696. 2020-08-25T18:33:59.7498449Z [cluster1] I0825 18:29:34.724591 1 kubernetes.go:317] In SetCluster: &types.SubmarinerCluster{ID:"cluster1", Spec:v1.ClusterSpec{ClusterID:"cluster1", ColorCodes:[]string{"blue"}, ServiceCIDR:[]string{"100.91.0.0/16"}, ClusterCIDR:[]string{"10.241.0.0/16"}, GlobalCIDR:[]string{}}}
  2697. 2020-08-25T18:33:59.7498887Z [cluster1] I0825 18:29:34.727796 1 kubernetes.go:345] Cluster "cluster1" matched what we received from datastore - not updating
  2698. 2020-08-25T18:33:59.7500644Z [cluster1] I0825 18:29:34.729303 1 tunnel.go:95] Tunnel controller processing added or updated submariner Endpoint object: &v1.Endpoint{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"cluster1-submariner-cable-cluster1-172-17-0-9", GenerateName:"", Namespace:"submariner-operator", SelfLink:"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster1-submariner-cable-cluster1-172-17-0-9", UID:"8e28b57c-62e8-4985-9cc5-94c0847f8748", ResourceVersion:"1258", Generation:1, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63733976735, loc:(*time.Location)(0x2155f60)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v1.EndpointSpec{ClusterID:"cluster1", CableName:"submariner-cable-cluster1-172-17-0-9", Hostname:"cluster1-worker", Subnets:[]string{"100.91.0.0/16", "10.241.0.0/16"}, PrivateIP:"172.17.0.9", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  2699. 2020-08-25T18:33:59.7501018Z [cluster1] I0825 18:29:34.729511 1 cableengine.go:94] Not installing cable for local cluster
  2700. 2020-08-25T18:33:59.7501496Z [cluster1] I0825 18:29:34.729646 1 tunnel.go:108] Tunnel controller successfully installed Endpoint cable submariner-cable-cluster1-172-17-0-9 in the engine
  2701. 2020-08-25T18:33:59.7503078Z [cluster1] I0825 18:29:34.729745 1 tunnel.go:95] Tunnel controller processing added or updated submariner Endpoint object: &v1.Endpoint{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"cluster2-submariner-cable-cluster2-172-17-0-8", GenerateName:"", Namespace:"submariner-operator", SelfLink:"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster2-submariner-cable-cluster2-172-17-0-8", UID:"4b2f98df-9143-4c8b-b359-f4d3fa6d05bc", ResourceVersion:"1270", Generation:1, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63733976737, loc:(*time.Location)(0x2155f60)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v1.EndpointSpec{ClusterID:"cluster2", CableName:"submariner-cable-cluster2-172-17-0-8", Hostname:"cluster2-worker", Subnets:[]string{"100.92.0.0/16", "10.242.0.0/16"}, PrivateIP:"172.17.0.8", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  2702. 2020-08-25T18:33:59.7503826Z [cluster1] I0825 18:29:34.730651 1 cableengine.go:103] Installing Endpoint cable "submariner-cable-cluster2-172-17-0-8"
  2703. 2020-08-25T18:33:59.7506402Z [cluster1] I0825 18:29:34.732859 1 datastoresyncer.go:287] Processing local submariner Endpoint object: &v1.Endpoint{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"cluster1-submariner-cable-cluster1-172-17-0-9", GenerateName:"", Namespace:"submariner-operator", SelfLink:"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster1-submariner-cable-cluster1-172-17-0-9", UID:"8e28b57c-62e8-4985-9cc5-94c0847f8748", ResourceVersion:"1258", Generation:1, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63733976735, loc:(*time.Location)(0x2155f60)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v1.EndpointSpec{ClusterID:"cluster1", CableName:"submariner-cable-cluster1-172-17-0-9", Hostname:"cluster1-worker", Subnets:[]string{"100.91.0.0/16", "10.241.0.0/16"}, PrivateIP:"172.17.0.9", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  2704. 2020-08-25T18:33:59.7507809Z [cluster1] I0825 18:29:34.732918 1 kubernetes.go:370] In SetEndpoint: &types.SubmarinerEndpoint{Spec:v1.EndpointSpec{ClusterID:"cluster1", CableName:"submariner-cable-cluster1-172-17-0-9", Hostname:"cluster1-worker", Subnets:[]string{"100.91.0.0/16", "10.241.0.0/16"}, PrivateIP:"172.17.0.9", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  2705. 2020-08-25T18:33:59.7508401Z [cluster1] I0825 18:29:34.734787 1 kubernetes.go:398] Endpoint "cluster1-submariner-cable-cluster1-172-17-0-9" matched what we received from datastore - not updating
  2706. 2020-08-25T18:33:59.7508851Z [cluster1] I0825 18:29:34.737424 1 datastoresyncer.go:271] The updated submariner Endpoint "cluster2" is not for this cluster - skipping updating the datastore
  2707. 2020-08-25T18:33:59.7509815Z [cluster1] I0825 18:29:34.737516 1 cableengine.go:117] Cable "submariner-cable-cluster2-172-17-0-8" is already installed - not installing again
  2708. 2020-08-25T18:33:59.7510319Z [cluster1] I0825 18:29:34.737631 1 tunnel.go:108] Tunnel controller successfully installed Endpoint cable submariner-cable-cluster2-172-17-0-8 in the engine
  2709. 2020-08-25T18:33:59.7510490Z [cluster1] I0825 18:29:35.225132 1 kubernetes.go:180] UpdateFunc in WatchClusters called
  2710. 2020-08-25T18:33:59.7510645Z [cluster1] I0825 18:29:35.225177 1 kubernetes.go:180] UpdateFunc in WatchClusters called
  2711. 2020-08-25T18:33:59.7510969Z [cluster1] I0825 18:29:35.225184 1 datastoresyncer.go:312] In reconcileClusterCRD: &types.SubmarinerCluster{ID:"cluster2", Spec:v1.ClusterSpec{ClusterID:"cluster2", ColorCodes:[]string{"blue"}, ServiceCIDR:[]string{"100.92.0.0/16"}, ClusterCIDR:[]string{"10.242.0.0/16"}, GlobalCIDR:[]string{}}}
  2712. 2020-08-25T18:33:59.7511163Z [cluster1] I0825 18:29:35.227984 1 kubernetes.go:264] UpdateFunc in WatchEndpoints called
  2713. 2020-08-25T18:33:59.7511320Z [cluster1] I0825 18:29:35.227996 1 kubernetes.go:264] UpdateFunc in WatchEndpoints called
  2714. 2020-08-25T18:33:59.7512248Z [cluster1] I0825 18:29:35.228002 1 datastoresyncer.go:387] In reconcileEndpointCRD: &types.SubmarinerEndpoint{Spec:v1.EndpointSpec{ClusterID:"cluster2", CableName:"submariner-cable-cluster2-172-17-0-8", Hostname:"cluster2-worker", Subnets:[]string{"100.92.0.0/16", "10.242.0.0/16"}, PrivateIP:"172.17.0.8", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  2715. 2020-08-25T18:33:59.7512716Z [cluster1] I0825 18:29:35.228640 1 datastoresyncer.go:360] Cluster "cluster2" matched what we received from datastore - not updating
  2716. 2020-08-25T18:33:59.7513177Z [cluster1] I0825 18:29:35.230895 1 datastoresyncer.go:435] Endpoint "cluster2-submariner-cable-cluster2-172-17-0-8" matched what we received from datastore - not updating
  2717. 2020-08-25T18:33:59.7513351Z [cluster1] I0825 18:30:05.225265 1 kubernetes.go:180] UpdateFunc in WatchClusters called
  2718. 2020-08-25T18:33:59.7513510Z [cluster1] I0825 18:30:05.225286 1 kubernetes.go:180] UpdateFunc in WatchClusters called
  2719. 2020-08-25T18:33:59.7513715Z [cluster1] I0825 18:30:05.225293 1 datastoresyncer.go:312] In reconcileClusterCRD: &types.SubmarinerCluster{ID:"cluster2", Spec:v1.ClusterSpec{ClusterID:"cluster2", ColorCodes:[]string{"blue"}, ServiceCIDR:[]string{"100.92.0.0/16"}, ClusterCIDR:[]string{"10.242.0.0/16"}, GlobalCIDR:[]string{}}}
  2720. 2020-08-25T18:33:59.7513906Z [cluster1] I0825 18:30:05.228082 1 kubernetes.go:264] UpdateFunc in WatchEndpoints called
  2721. 2020-08-25T18:33:59.7514060Z [cluster1] I0825 18:30:05.228098 1 kubernetes.go:264] UpdateFunc in WatchEndpoints called
  2722. 2020-08-25T18:33:59.7514782Z [cluster1] I0825 18:30:05.228133 1 datastoresyncer.go:387] In reconcileEndpointCRD: &types.SubmarinerEndpoint{Spec:v1.EndpointSpec{ClusterID:"cluster2", CableName:"submariner-cable-cluster2-172-17-0-8", Hostname:"cluster2-worker", Subnets:[]string{"100.92.0.0/16", "10.242.0.0/16"}, PrivateIP:"172.17.0.8", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  2723. 2020-08-25T18:33:59.7515324Z [cluster1] I0825 18:30:05.233289 1 datastoresyncer.go:360] Cluster "cluster2" matched what we received from datastore - not updating
  2724. 2020-08-25T18:33:59.7515818Z [cluster1] I0825 18:30:05.235577 1 datastoresyncer.go:435] Endpoint "cluster2-submariner-cable-cluster2-172-17-0-8" matched what we received from datastore - not updating
  2725. 2020-08-25T18:33:59.7516048Z [cluster1] I0825 18:30:34.724709 1 kubernetes.go:317] In SetCluster: &types.SubmarinerCluster{ID:"cluster1", Spec:v1.ClusterSpec{ClusterID:"cluster1", ColorCodes:[]string{"blue"}, ServiceCIDR:[]string{"100.91.0.0/16"}, ClusterCIDR:[]string{"10.241.0.0/16"}, GlobalCIDR:[]string{}}}
  2726. 2020-08-25T18:33:59.7516483Z [cluster1] I0825 18:30:34.727839 1 kubernetes.go:345] Cluster "cluster1" matched what we received from datastore - not updating
  2727. 2020-08-25T18:33:59.7518101Z [cluster1] I0825 18:30:34.729604 1 tunnel.go:95] Tunnel controller processing added or updated submariner Endpoint object: &v1.Endpoint{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"cluster1-submariner-cable-cluster1-172-17-0-9", GenerateName:"", Namespace:"submariner-operator", SelfLink:"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster1-submariner-cable-cluster1-172-17-0-9", UID:"8e28b57c-62e8-4985-9cc5-94c0847f8748", ResourceVersion:"1258", Generation:1, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63733976735, loc:(*time.Location)(0x2155f60)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v1.EndpointSpec{ClusterID:"cluster1", CableName:"submariner-cable-cluster1-172-17-0-9", Hostname:"cluster1-worker", Subnets:[]string{"100.91.0.0/16", "10.241.0.0/16"}, PrivateIP:"172.17.0.9", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  2728. 2020-08-25T18:33:59.7518580Z [cluster1] I0825 18:30:34.729680 1 cableengine.go:94] Not installing cable for local cluster
  2729. 2020-08-25T18:33:59.7519048Z [cluster1] I0825 18:30:34.729687 1 tunnel.go:108] Tunnel controller successfully installed Endpoint cable submariner-cable-cluster1-172-17-0-9 in the engine
  2730. 2020-08-25T18:33:59.7520646Z [cluster1] I0825 18:30:34.729692 1 tunnel.go:95] Tunnel controller processing added or updated submariner Endpoint object: &v1.Endpoint{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"cluster2-submariner-cable-cluster2-172-17-0-8", GenerateName:"", Namespace:"submariner-operator", SelfLink:"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster2-submariner-cable-cluster2-172-17-0-8", UID:"4b2f98df-9143-4c8b-b359-f4d3fa6d05bc", ResourceVersion:"1270", Generation:1, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63733976737, loc:(*time.Location)(0x2155f60)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v1.EndpointSpec{ClusterID:"cluster2", CableName:"submariner-cable-cluster2-172-17-0-8", Hostname:"cluster2-worker", Subnets:[]string{"100.92.0.0/16", "10.242.0.0/16"}, PrivateIP:"172.17.0.8", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  2731. 2020-08-25T18:33:59.7521242Z [cluster1] I0825 18:30:34.729718 1 cableengine.go:103] Installing Endpoint cable "submariner-cable-cluster2-172-17-0-8"
  2732. 2020-08-25T18:33:59.7521664Z [cluster1] I0825 18:30:34.732191 1 cableengine.go:117] Cable "submariner-cable-cluster2-172-17-0-8" is already installed - not installing again
  2733. 2020-08-25T18:33:59.7522109Z [cluster1] I0825 18:30:34.732209 1 tunnel.go:108] Tunnel controller successfully installed Endpoint cable submariner-cable-cluster2-172-17-0-8 in the engine
  2734. 2020-08-25T18:33:59.7524689Z [cluster1] I0825 18:30:34.733318 1 datastoresyncer.go:287] Processing local submariner Endpoint object: &v1.Endpoint{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"cluster1-submariner-cable-cluster1-172-17-0-9", GenerateName:"", Namespace:"submariner-operator", SelfLink:"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster1-submariner-cable-cluster1-172-17-0-9", UID:"8e28b57c-62e8-4985-9cc5-94c0847f8748", ResourceVersion:"1258", Generation:1, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63733976735, loc:(*time.Location)(0x2155f60)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v1.EndpointSpec{ClusterID:"cluster1", CableName:"submariner-cable-cluster1-172-17-0-9", Hostname:"cluster1-worker", Subnets:[]string{"100.91.0.0/16", "10.241.0.0/16"}, PrivateIP:"172.17.0.9", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  2735. 2020-08-25T18:33:59.7525657Z [cluster1] I0825 18:30:34.733363 1 kubernetes.go:370] In SetEndpoint: &types.SubmarinerEndpoint{Spec:v1.EndpointSpec{ClusterID:"cluster1", CableName:"submariner-cable-cluster1-172-17-0-9", Hostname:"cluster1-worker", Subnets:[]string{"100.91.0.0/16", "10.241.0.0/16"}, PrivateIP:"172.17.0.9", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  2736. 2020-08-25T18:33:59.7526152Z [cluster1] I0825 18:30:34.735780 1 kubernetes.go:398] Endpoint "cluster1-submariner-cable-cluster1-172-17-0-9" matched what we received from datastore - not updating
  2737. 2020-08-25T18:33:59.7526601Z [cluster1] I0825 18:30:34.738820 1 datastoresyncer.go:271] The updated submariner Endpoint "cluster2" is not for this cluster - skipping updating the datastore
  2738. 2020-08-25T18:33:59.7526873Z [cluster1] I0825 18:30:35.225568 1 kubernetes.go:180] UpdateFunc in WatchClusters called
  2739. 2020-08-25T18:33:59.7527080Z [cluster1] I0825 18:30:35.225590 1 datastoresyncer.go:312] In reconcileClusterCRD: &types.SubmarinerCluster{ID:"cluster2", Spec:v1.ClusterSpec{ClusterID:"cluster2", ColorCodes:[]string{"blue"}, ServiceCIDR:[]string{"100.92.0.0/16"}, ClusterCIDR:[]string{"10.242.0.0/16"}, GlobalCIDR:[]string{}}}
  2740. 2020-08-25T18:33:59.7527542Z [cluster1] I0825 18:30:35.227785 1 datastoresyncer.go:360] Cluster "cluster2" matched what we received from datastore - not updating
  2741. 2020-08-25T18:33:59.7527707Z [cluster1] I0825 18:30:35.227852 1 kubernetes.go:180] UpdateFunc in WatchClusters called
  2742. 2020-08-25T18:33:59.7527865Z [cluster1] I0825 18:30:35.228220 1 kubernetes.go:264] UpdateFunc in WatchEndpoints called
  2743. 2020-08-25T18:33:59.7528023Z [cluster1] I0825 18:30:35.228529 1 kubernetes.go:264] UpdateFunc in WatchEndpoints called
  2744. 2020-08-25T18:33:59.7528741Z [cluster1] I0825 18:30:35.228537 1 datastoresyncer.go:387] In reconcileEndpointCRD: &types.SubmarinerEndpoint{Spec:v1.EndpointSpec{ClusterID:"cluster2", CableName:"submariner-cable-cluster2-172-17-0-8", Hostname:"cluster2-worker", Subnets:[]string{"100.92.0.0/16", "10.242.0.0/16"}, PrivateIP:"172.17.0.8", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  2745. 2020-08-25T18:33:59.7529237Z [cluster1] I0825 18:30:35.230187 1 datastoresyncer.go:435] Endpoint "cluster2-submariner-cable-cluster2-172-17-0-8" matched what we received from datastore - not updating
  2746. 2020-08-25T18:33:59.7529412Z [cluster1] I0825 18:31:05.225716 1 kubernetes.go:180] UpdateFunc in WatchClusters called
  2747. 2020-08-25T18:33:59.7529568Z [cluster1] I0825 18:31:05.225739 1 kubernetes.go:180] UpdateFunc in WatchClusters called
  2748. 2020-08-25T18:33:59.7529836Z [cluster1] I0825 18:31:05.225746 1 datastoresyncer.go:312] In reconcileClusterCRD: &types.SubmarinerCluster{ID:"cluster2", Spec:v1.ClusterSpec{ClusterID:"cluster2", ColorCodes:[]string{"blue"}, ServiceCIDR:[]string{"100.92.0.0/16"}, ClusterCIDR:[]string{"10.242.0.0/16"}, GlobalCIDR:[]string{}}}
  2749. 2020-08-25T18:33:59.7530321Z [cluster1] I0825 18:31:05.228153 1 datastoresyncer.go:360] Cluster "cluster2" matched what we received from datastore - not updating
  2750. 2020-08-25T18:33:59.7530482Z [cluster1] I0825 18:31:05.228368 1 kubernetes.go:264] UpdateFunc in WatchEndpoints called
  2751. 2020-08-25T18:33:59.7530640Z [cluster1] I0825 18:31:05.228385 1 kubernetes.go:264] UpdateFunc in WatchEndpoints called
  2752. 2020-08-25T18:33:59.7531354Z [cluster1] I0825 18:31:05.228391 1 datastoresyncer.go:387] In reconcileEndpointCRD: &types.SubmarinerEndpoint{Spec:v1.EndpointSpec{ClusterID:"cluster2", CableName:"submariner-cable-cluster2-172-17-0-8", Hostname:"cluster2-worker", Subnets:[]string{"100.92.0.0/16", "10.242.0.0/16"}, PrivateIP:"172.17.0.8", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  2753. 2020-08-25T18:33:59.7531871Z [cluster1] I0825 18:31:05.230086 1 datastoresyncer.go:435] Endpoint "cluster2-submariner-cable-cluster2-172-17-0-8" matched what we received from datastore - not updating
  2754. 2020-08-25T18:33:59.7532081Z [cluster1] I0825 18:31:34.725533 1 kubernetes.go:317] In SetCluster: &types.SubmarinerCluster{ID:"cluster1", Spec:v1.ClusterSpec{ClusterID:"cluster1", ColorCodes:[]string{"blue"}, ServiceCIDR:[]string{"100.91.0.0/16"}, ClusterCIDR:[]string{"10.241.0.0/16"}, GlobalCIDR:[]string{}}}
  2755. 2020-08-25T18:33:59.7532518Z [cluster1] I0825 18:31:34.728074 1 kubernetes.go:345] Cluster "cluster1" matched what we received from datastore - not updating
  2756. 2020-08-25T18:33:59.7535295Z [cluster1] I0825 18:31:34.729914 1 tunnel.go:95] Tunnel controller processing added or updated submariner Endpoint object: &v1.Endpoint{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"cluster1-submariner-cable-cluster1-172-17-0-9", GenerateName:"", Namespace:"submariner-operator", SelfLink:"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster1-submariner-cable-cluster1-172-17-0-9", UID:"8e28b57c-62e8-4985-9cc5-94c0847f8748", ResourceVersion:"1258", Generation:1, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63733976735, loc:(*time.Location)(0x2155f60)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v1.EndpointSpec{ClusterID:"cluster1", CableName:"submariner-cable-cluster1-172-17-0-9", Hostname:"cluster1-worker", Subnets:[]string{"100.91.0.0/16", "10.241.0.0/16"}, PrivateIP:"172.17.0.9", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  2757. 2020-08-25T18:33:59.7536276Z [cluster1] I0825 18:31:34.730024 1 cableengine.go:94] Not installing cable for local cluster
  2758. 2020-08-25T18:33:59.7536867Z [cluster1] I0825 18:31:34.730031 1 tunnel.go:108] Tunnel controller successfully installed Endpoint cable submariner-cable-cluster1-172-17-0-9 in the engine
  2759. 2020-08-25T18:33:59.7538806Z [cluster1] I0825 18:31:34.730037 1 tunnel.go:95] Tunnel controller processing added or updated submariner Endpoint object: &v1.Endpoint{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"cluster2-submariner-cable-cluster2-172-17-0-8", GenerateName:"", Namespace:"submariner-operator", SelfLink:"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster2-submariner-cable-cluster2-172-17-0-8", UID:"4b2f98df-9143-4c8b-b359-f4d3fa6d05bc", ResourceVersion:"1270", Generation:1, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63733976737, loc:(*time.Location)(0x2155f60)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v1.EndpointSpec{ClusterID:"cluster2", CableName:"submariner-cable-cluster2-172-17-0-8", Hostname:"cluster2-worker", Subnets:[]string{"100.92.0.0/16", "10.242.0.0/16"}, PrivateIP:"172.17.0.8", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  2760. 2020-08-25T18:33:59.7539504Z [cluster1] I0825 18:31:34.730062 1 cableengine.go:103] Installing Endpoint cable "submariner-cable-cluster2-172-17-0-8"
  2761. 2020-08-25T18:33:59.7539933Z [cluster1] I0825 18:31:34.731944 1 cableengine.go:117] Cable "submariner-cable-cluster2-172-17-0-8" is already installed - not installing again
  2762. 2020-08-25T18:33:59.7540381Z [cluster1] I0825 18:31:34.731979 1 tunnel.go:108] Tunnel controller successfully installed Endpoint cable submariner-cable-cluster2-172-17-0-8 in the engine
  2763. 2020-08-25T18:33:59.7541995Z [cluster1] I0825 18:31:34.732609 1 datastoresyncer.go:287] Processing local submariner Endpoint object: &v1.Endpoint{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"cluster1-submariner-cable-cluster1-172-17-0-9", GenerateName:"", Namespace:"submariner-operator", SelfLink:"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster1-submariner-cable-cluster1-172-17-0-9", UID:"8e28b57c-62e8-4985-9cc5-94c0847f8748", ResourceVersion:"1258", Generation:1, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63733976735, loc:(*time.Location)(0x2155f60)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v1.EndpointSpec{ClusterID:"cluster1", CableName:"submariner-cable-cluster1-172-17-0-9", Hostname:"cluster1-worker", Subnets:[]string{"100.91.0.0/16", "10.241.0.0/16"}, PrivateIP:"172.17.0.9", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  2764. 2020-08-25T18:33:59.7543012Z [cluster1] I0825 18:31:34.732657 1 kubernetes.go:370] In SetEndpoint: &types.SubmarinerEndpoint{Spec:v1.EndpointSpec{ClusterID:"cluster1", CableName:"submariner-cable-cluster1-172-17-0-9", Hostname:"cluster1-worker", Subnets:[]string{"100.91.0.0/16", "10.241.0.0/16"}, PrivateIP:"172.17.0.9", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  2765. 2020-08-25T18:33:59.7543824Z [cluster1] I0825 18:31:34.735008 1 kubernetes.go:398] Endpoint "cluster1-submariner-cable-cluster1-172-17-0-9" matched what we received from datastore - not updating
  2766. 2020-08-25T18:33:59.7544243Z [cluster1] I0825 18:31:34.739215 1 datastoresyncer.go:271] The updated submariner Endpoint "cluster2" is not for this cluster - skipping updating the datastore
  2767. 2020-08-25T18:33:59.7544406Z [cluster1] I0825 18:31:35.225830 1 kubernetes.go:180] UpdateFunc in WatchClusters called
  2768. 2020-08-25T18:33:59.7544555Z [cluster1] I0825 18:31:35.225886 1 kubernetes.go:180] UpdateFunc in WatchClusters called
  2769. 2020-08-25T18:33:59.7545260Z [cluster1] I0825 18:31:35.225893 1 datastoresyncer.go:312] In reconcileClusterCRD: &types.SubmarinerCluster{ID:"cluster2", Spec:v1.ClusterSpec{ClusterID:"cluster2", ColorCodes:[]string{"blue"}, ServiceCIDR:[]string{"100.92.0.0/16"}, ClusterCIDR:[]string{"10.242.0.0/16"}, GlobalCIDR:[]string{}}}
  2770. 2020-08-25T18:33:59.7545443Z [cluster1] I0825 18:31:35.228417 1 kubernetes.go:264] UpdateFunc in WatchEndpoints called
  2771. 2020-08-25T18:33:59.7545594Z [cluster1] I0825 18:31:35.228564 1 kubernetes.go:264] UpdateFunc in WatchEndpoints called
  2772. 2020-08-25T18:33:59.7546689Z [cluster1] I0825 18:31:35.228674 1 datastoresyncer.go:387] In reconcileEndpointCRD: &types.SubmarinerEndpoint{Spec:v1.EndpointSpec{ClusterID:"cluster2", CableName:"submariner-cable-cluster2-172-17-0-8", Hostname:"cluster2-worker", Subnets:[]string{"100.92.0.0/16", "10.242.0.0/16"}, PrivateIP:"172.17.0.8", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  2773. 2020-08-25T18:33:59.7547321Z [cluster1] I0825 18:31:35.228990 1 datastoresyncer.go:360] Cluster "cluster2" matched what we received from datastore - not updating
  2774. 2020-08-25T18:33:59.7547820Z [cluster1] I0825 18:31:35.230940 1 datastoresyncer.go:435] Endpoint "cluster2-submariner-cable-cluster2-172-17-0-8" matched what we received from datastore - not updating
  2775. 2020-08-25T18:33:59.7547996Z [cluster1] I0825 18:32:05.225928 1 kubernetes.go:180] UpdateFunc in WatchClusters called
  2776. 2020-08-25T18:33:59.7548564Z [cluster1] I0825 18:32:05.225960 1 kubernetes.go:180] UpdateFunc in WatchClusters called
  2777. 2020-08-25T18:33:59.7548810Z [cluster1] I0825 18:32:05.225967 1 datastoresyncer.go:312] In reconcileClusterCRD: &types.SubmarinerCluster{ID:"cluster2", Spec:v1.ClusterSpec{ClusterID:"cluster2", ColorCodes:[]string{"blue"}, ServiceCIDR:[]string{"100.92.0.0/16"}, ClusterCIDR:[]string{"10.242.0.0/16"}, GlobalCIDR:[]string{}}}
  2778. 2020-08-25T18:33:59.7550264Z [cluster1] I0825 18:32:05.228343 1 datastoresyncer.go:360] Cluster "cluster2" matched what we received from datastore - not updating
  2779. 2020-08-25T18:33:59.7550448Z [cluster1] I0825 18:32:05.228584 1 kubernetes.go:264] UpdateFunc in WatchEndpoints called
  2780. 2020-08-25T18:33:59.7550607Z [cluster1] I0825 18:32:05.228606 1 kubernetes.go:264] UpdateFunc in WatchEndpoints called
  2781. 2020-08-25T18:33:59.7551344Z [cluster1] I0825 18:32:05.228612 1 datastoresyncer.go:387] In reconcileEndpointCRD: &types.SubmarinerEndpoint{Spec:v1.EndpointSpec{ClusterID:"cluster2", CableName:"submariner-cable-cluster2-172-17-0-8", Hostname:"cluster2-worker", Subnets:[]string{"100.92.0.0/16", "10.242.0.0/16"}, PrivateIP:"172.17.0.8", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  2782. 2020-08-25T18:33:59.7551844Z [cluster1] I0825 18:32:05.230735 1 datastoresyncer.go:435] Endpoint "cluster2-submariner-cable-cluster2-172-17-0-8" matched what we received from datastore - not updating
  2783. 2020-08-25T18:33:59.7552230Z [cluster1] I0825 18:32:34.726014 1 kubernetes.go:317] In SetCluster: &types.SubmarinerCluster{ID:"cluster1", Spec:v1.ClusterSpec{ClusterID:"cluster1", ColorCodes:[]string{"blue"}, ServiceCIDR:[]string{"100.91.0.0/16"}, ClusterCIDR:[]string{"10.241.0.0/16"}, GlobalCIDR:[]string{}}}
  2784. 2020-08-25T18:33:59.7552706Z [cluster1] I0825 18:32:34.729288 1 kubernetes.go:345] Cluster "cluster1" matched what we received from datastore - not updating
  2785. 2020-08-25T18:33:59.7554321Z [cluster1] I0825 18:32:34.730374 1 tunnel.go:95] Tunnel controller processing added or updated submariner Endpoint object: &v1.Endpoint{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"cluster1-submariner-cable-cluster1-172-17-0-9", GenerateName:"", Namespace:"submariner-operator", SelfLink:"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster1-submariner-cable-cluster1-172-17-0-9", UID:"8e28b57c-62e8-4985-9cc5-94c0847f8748", ResourceVersion:"1258", Generation:1, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63733976735, loc:(*time.Location)(0x2155f60)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v1.EndpointSpec{ClusterID:"cluster1", CableName:"submariner-cable-cluster1-172-17-0-9", Hostname:"cluster1-worker", Subnets:[]string{"100.91.0.0/16", "10.241.0.0/16"}, PrivateIP:"172.17.0.9", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  2786. 2020-08-25T18:33:59.7554862Z [cluster1] I0825 18:32:34.730449 1 cableengine.go:94] Not installing cable for local cluster
  2787. 2020-08-25T18:33:59.7555292Z [cluster1] I0825 18:32:34.730456 1 tunnel.go:108] Tunnel controller successfully installed Endpoint cable submariner-cable-cluster1-172-17-0-9 in the engine
  2788. 2020-08-25T18:33:59.7557047Z [cluster1] I0825 18:32:34.730461 1 tunnel.go:95] Tunnel controller processing added or updated submariner Endpoint object: &v1.Endpoint{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"cluster2-submariner-cable-cluster2-172-17-0-8", GenerateName:"", Namespace:"submariner-operator", SelfLink:"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster2-submariner-cable-cluster2-172-17-0-8", UID:"4b2f98df-9143-4c8b-b359-f4d3fa6d05bc", ResourceVersion:"1270", Generation:1, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63733976737, loc:(*time.Location)(0x2155f60)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v1.EndpointSpec{ClusterID:"cluster2", CableName:"submariner-cable-cluster2-172-17-0-8", Hostname:"cluster2-worker", Subnets:[]string{"100.92.0.0/16", "10.242.0.0/16"}, PrivateIP:"172.17.0.8", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  2789. 2020-08-25T18:33:59.7557644Z [cluster1] I0825 18:32:34.730528 1 cableengine.go:103] Installing Endpoint cable "submariner-cable-cluster2-172-17-0-8"
  2790. 2020-08-25T18:33:59.7558212Z [cluster1] I0825 18:32:34.732966 1 cableengine.go:117] Cable "submariner-cable-cluster2-172-17-0-8" is already installed - not installing again
  2791. 2020-08-25T18:33:59.7558862Z [cluster1] I0825 18:32:34.733008 1 tunnel.go:108] Tunnel controller successfully installed Endpoint cable submariner-cable-cluster2-172-17-0-8 in the engine
  2792. 2020-08-25T18:33:59.7561320Z [cluster1] I0825 18:32:34.733033 1 datastoresyncer.go:287] Processing local submariner Endpoint object: &v1.Endpoint{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"cluster1-submariner-cable-cluster1-172-17-0-9", GenerateName:"", Namespace:"submariner-operator", SelfLink:"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster1-submariner-cable-cluster1-172-17-0-9", UID:"8e28b57c-62e8-4985-9cc5-94c0847f8748", ResourceVersion:"1258", Generation:1, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63733976735, loc:(*time.Location)(0x2155f60)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v1.EndpointSpec{ClusterID:"cluster1", CableName:"submariner-cable-cluster1-172-17-0-9", Hostname:"cluster1-worker", Subnets:[]string{"100.91.0.0/16", "10.241.0.0/16"}, PrivateIP:"172.17.0.9", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  2793. 2020-08-25T18:33:59.7562466Z [cluster1] I0825 18:32:34.733085 1 kubernetes.go:370] In SetEndpoint: &types.SubmarinerEndpoint{Spec:v1.EndpointSpec{ClusterID:"cluster1", CableName:"submariner-cable-cluster1-172-17-0-9", Hostname:"cluster1-worker", Subnets:[]string{"100.91.0.0/16", "10.241.0.0/16"}, PrivateIP:"172.17.0.9", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  2794. 2020-08-25T18:33:59.7563533Z [cluster1] I0825 18:32:34.735252 1 kubernetes.go:398] Endpoint "cluster1-submariner-cable-cluster1-172-17-0-9" matched what we received from datastore - not updating
  2795. 2020-08-25T18:33:59.7564081Z [cluster1] I0825 18:32:34.737549 1 datastoresyncer.go:271] The updated submariner Endpoint "cluster2" is not for this cluster - skipping updating the datastore
  2796. 2020-08-25T18:33:59.7564259Z [cluster1] I0825 18:32:35.226138 1 kubernetes.go:180] UpdateFunc in WatchClusters called
  2797. 2020-08-25T18:33:59.7564415Z [cluster1] I0825 18:32:35.226178 1 kubernetes.go:180] UpdateFunc in WatchClusters called
  2798. 2020-08-25T18:33:59.7564771Z [cluster1] I0825 18:32:35.226185 1 datastoresyncer.go:312] In reconcileClusterCRD: &types.SubmarinerCluster{ID:"cluster2", Spec:v1.ClusterSpec{ClusterID:"cluster2", ColorCodes:[]string{"blue"}, ServiceCIDR:[]string{"100.92.0.0/16"}, ClusterCIDR:[]string{"10.242.0.0/16"}, GlobalCIDR:[]string{}}}
  2799. 2020-08-25T18:33:59.7565273Z [cluster1] I0825 18:32:35.228573 1 datastoresyncer.go:360] Cluster "cluster2" matched what we received from datastore - not updating
  2800. 2020-08-25T18:33:59.7565438Z [cluster1] I0825 18:32:35.228645 1 kubernetes.go:264] UpdateFunc in WatchEndpoints called
  2801. 2020-08-25T18:33:59.7565596Z [cluster1] I0825 18:32:35.228660 1 kubernetes.go:264] UpdateFunc in WatchEndpoints called
  2802. 2020-08-25T18:33:59.7566309Z [cluster1] I0825 18:32:35.228665 1 datastoresyncer.go:387] In reconcileEndpointCRD: &types.SubmarinerEndpoint{Spec:v1.EndpointSpec{ClusterID:"cluster2", CableName:"submariner-cable-cluster2-172-17-0-8", Hostname:"cluster2-worker", Subnets:[]string{"100.92.0.0/16", "10.242.0.0/16"}, PrivateIP:"172.17.0.8", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  2803. 2020-08-25T18:33:59.7566820Z [cluster1] I0825 18:32:35.230573 1 datastoresyncer.go:435] Endpoint "cluster2-submariner-cable-cluster2-172-17-0-8" matched what we received from datastore - not updating
  2804. 2020-08-25T18:33:59.7566993Z [cluster1] I0825 18:33:05.226328 1 kubernetes.go:180] UpdateFunc in WatchClusters called
  2805. 2020-08-25T18:33:59.7567145Z [cluster1] I0825 18:33:05.226351 1 kubernetes.go:180] UpdateFunc in WatchClusters called
  2806. 2020-08-25T18:33:59.7567342Z [cluster1] I0825 18:33:05.226358 1 datastoresyncer.go:312] In reconcileClusterCRD: &types.SubmarinerCluster{ID:"cluster2", Spec:v1.ClusterSpec{ClusterID:"cluster2", ColorCodes:[]string{"blue"}, ServiceCIDR:[]string{"100.92.0.0/16"}, ClusterCIDR:[]string{"10.242.0.0/16"}, GlobalCIDR:[]string{}}}
  2807. 2020-08-25T18:33:59.7567531Z [cluster1] I0825 18:33:05.228777 1 kubernetes.go:264] UpdateFunc in WatchEndpoints called
  2808. 2020-08-25T18:33:59.7567955Z [cluster1] I0825 18:33:05.228793 1 kubernetes.go:264] UpdateFunc in WatchEndpoints called
  2809. 2020-08-25T18:33:59.7568680Z [cluster1] I0825 18:33:05.228799 1 datastoresyncer.go:387] In reconcileEndpointCRD: &types.SubmarinerEndpoint{Spec:v1.EndpointSpec{ClusterID:"cluster2", CableName:"submariner-cable-cluster2-172-17-0-8", Hostname:"cluster2-worker", Subnets:[]string{"100.92.0.0/16", "10.242.0.0/16"}, PrivateIP:"172.17.0.8", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  2810. 2020-08-25T18:33:59.7569132Z [cluster1] I0825 18:33:05.230423 1 datastoresyncer.go:360] Cluster "cluster2" matched what we received from datastore - not updating
  2811. 2020-08-25T18:33:59.7569588Z [cluster1] I0825 18:33:05.231279 1 datastoresyncer.go:435] Endpoint "cluster2-submariner-cable-cluster2-172-17-0-8" matched what we received from datastore - not updating
  2812. 2020-08-25T18:33:59.7569799Z [cluster1] I0825 18:33:34.726349 1 kubernetes.go:317] In SetCluster: &types.SubmarinerCluster{ID:"cluster1", Spec:v1.ClusterSpec{ClusterID:"cluster1", ColorCodes:[]string{"blue"}, ServiceCIDR:[]string{"100.91.0.0/16"}, ClusterCIDR:[]string{"10.241.0.0/16"}, GlobalCIDR:[]string{}}}
  2813. 2020-08-25T18:33:59.7570225Z [cluster1] I0825 18:33:34.729194 1 kubernetes.go:345] Cluster "cluster1" matched what we received from datastore - not updating
  2814. 2020-08-25T18:33:59.7572110Z [cluster1] I0825 18:33:34.730630 1 tunnel.go:95] Tunnel controller processing added or updated submariner Endpoint object: &v1.Endpoint{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"cluster1-submariner-cable-cluster1-172-17-0-9", GenerateName:"", Namespace:"submariner-operator", SelfLink:"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster1-submariner-cable-cluster1-172-17-0-9", UID:"8e28b57c-62e8-4985-9cc5-94c0847f8748", ResourceVersion:"1258", Generation:1, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63733976735, loc:(*time.Location)(0x2155f60)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v1.EndpointSpec{ClusterID:"cluster1", CableName:"submariner-cable-cluster1-172-17-0-9", Hostname:"cluster1-worker", Subnets:[]string{"100.91.0.0/16", "10.241.0.0/16"}, PrivateIP:"172.17.0.9", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  2815. 2020-08-25T18:33:59.7572511Z [cluster1] I0825 18:33:34.730720 1 cableengine.go:94] Not installing cable for local cluster
  2816. 2020-08-25T18:33:59.7572977Z [cluster1] I0825 18:33:34.730727 1 tunnel.go:108] Tunnel controller successfully installed Endpoint cable submariner-cable-cluster1-172-17-0-9 in the engine
  2817. 2020-08-25T18:33:59.7574566Z [cluster1] I0825 18:33:34.730733 1 tunnel.go:95] Tunnel controller processing added or updated submariner Endpoint object: &v1.Endpoint{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"cluster2-submariner-cable-cluster2-172-17-0-8", GenerateName:"", Namespace:"submariner-operator", SelfLink:"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster2-submariner-cable-cluster2-172-17-0-8", UID:"4b2f98df-9143-4c8b-b359-f4d3fa6d05bc", ResourceVersion:"1270", Generation:1, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63733976737, loc:(*time.Location)(0x2155f60)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v1.EndpointSpec{ClusterID:"cluster2", CableName:"submariner-cable-cluster2-172-17-0-8", Hostname:"cluster2-worker", Subnets:[]string{"100.92.0.0/16", "10.242.0.0/16"}, PrivateIP:"172.17.0.8", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  2818. 2020-08-25T18:33:59.7575302Z [cluster1] I0825 18:33:34.730764 1 cableengine.go:103] Installing Endpoint cable "submariner-cable-cluster2-172-17-0-8"
  2819. 2020-08-25T18:33:59.7576884Z [cluster1] I0825 18:33:34.733879 1 datastoresyncer.go:287] Processing local submariner Endpoint object: &v1.Endpoint{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"cluster1-submariner-cable-cluster1-172-17-0-9", GenerateName:"", Namespace:"submariner-operator", SelfLink:"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster1-submariner-cable-cluster1-172-17-0-9", UID:"8e28b57c-62e8-4985-9cc5-94c0847f8748", ResourceVersion:"1258", Generation:1, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63733976735, loc:(*time.Location)(0x2155f60)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v1.EndpointSpec{ClusterID:"cluster1", CableName:"submariner-cable-cluster1-172-17-0-9", Hostname:"cluster1-worker", Subnets:[]string{"100.91.0.0/16", "10.241.0.0/16"}, PrivateIP:"172.17.0.9", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  2820. 2020-08-25T18:33:59.7577773Z [cluster1] I0825 18:33:34.733966 1 kubernetes.go:370] In SetEndpoint: &types.SubmarinerEndpoint{Spec:v1.EndpointSpec{ClusterID:"cluster1", CableName:"submariner-cable-cluster1-172-17-0-9", Hostname:"cluster1-worker", Subnets:[]string{"100.91.0.0/16", "10.241.0.0/16"}, PrivateIP:"172.17.0.9", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  2821. 2020-08-25T18:33:59.7578242Z [cluster1] I0825 18:33:34.733975 1 cableengine.go:117] Cable "submariner-cable-cluster2-172-17-0-8" is already installed - not installing again
  2822. 2020-08-25T18:33:59.7578764Z [cluster1] I0825 18:33:34.734097 1 tunnel.go:108] Tunnel controller successfully installed Endpoint cable submariner-cable-cluster2-172-17-0-8 in the engine
  2823. 2020-08-25T18:33:59.7579254Z [cluster1] I0825 18:33:34.736094 1 kubernetes.go:398] Endpoint "cluster1-submariner-cable-cluster1-172-17-0-9" matched what we received from datastore - not updating
  2824. 2020-08-25T18:33:59.7579704Z [cluster1] I0825 18:33:34.737470 1 datastoresyncer.go:271] The updated submariner Endpoint "cluster2" is not for this cluster - skipping updating the datastore
  2825. 2020-08-25T18:33:59.7579877Z [cluster1] I0825 18:33:35.226591 1 kubernetes.go:180] UpdateFunc in WatchClusters called
  2826. 2020-08-25T18:33:59.7580033Z [cluster1] I0825 18:33:35.226615 1 kubernetes.go:180] UpdateFunc in WatchClusters called
  2827. 2020-08-25T18:33:59.7580236Z [cluster1] I0825 18:33:35.226622 1 datastoresyncer.go:312] In reconcileClusterCRD: &types.SubmarinerCluster{ID:"cluster2", Spec:v1.ClusterSpec{ClusterID:"cluster2", ColorCodes:[]string{"blue"}, ServiceCIDR:[]string{"100.92.0.0/16"}, ClusterCIDR:[]string{"10.242.0.0/16"}, GlobalCIDR:[]string{}}}
  2828. 2020-08-25T18:33:59.7580437Z [cluster1] I0825 18:33:35.229354 1 kubernetes.go:264] UpdateFunc in WatchEndpoints called
  2829. 2020-08-25T18:33:59.7580594Z [cluster1] I0825 18:33:35.229501 1 kubernetes.go:264] UpdateFunc in WatchEndpoints called
  2830. 2020-08-25T18:33:59.7596592Z [lighthouse]$ [cluster1] command kubectl --context=cluster1 get Gateway -A -o yaml
  2831. 2020-08-25T18:33:59.7596970Z [lighthouse]$ [cluster1] kubectl --context=cluster1 get Gateway -A -o yaml
  2832. 2020-08-25T18:34:00.5714483Z [cluster1] I0825 18:33:35.229564 1 datastoresyncer.go:387] In reconcileEndpointCRD: &types.SubmarinerEndpoint{Spec:v1.EndpointSpec{ClusterID:"cluster2", CableName:"submariner-cable-cluster2-172-17-0-8", Hostname:"cluster2-worker", Subnets:[]string{"100.92.0.0/16", "10.242.0.0/16"}, PrivateIP:"172.17.0.8", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  2833. 2020-08-25T18:34:00.5715291Z [cluster1] I0825 18:33:35.231864 1 datastoresyncer.go:360] Cluster "cluster2" matched what we received from datastore - not updating
  2834. 2020-08-25T18:34:00.5715749Z [cluster1] I0825 18:33:35.233893 1 datastoresyncer.go:435] Endpoint "cluster2-submariner-cable-cluster2-172-17-0-8" matched what we received from datastore - not updating
  2835. 2020-08-25T18:34:00.5715926Z [cluster1] ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
  2836. 2020-08-25T18:34:00.5716060Z [cluster1] apiVersion: v1
  2837. 2020-08-25T18:34:00.5716176Z [cluster1] items:
  2838. 2020-08-25T18:34:00.5716455Z [cluster1] - apiVersion: submariner.io/v1
  2839. 2020-08-25T18:34:00.5716573Z [cluster1] kind: Gateway
  2840. 2020-08-25T18:34:00.5716675Z [cluster1] metadata:
  2841. 2020-08-25T18:34:00.5716791Z [cluster1] annotations:
  2842. 2020-08-25T18:34:00.5717070Z [cluster1] update-timestamp: "1598380436"
  2843. 2020-08-25T18:34:00.5717364Z [cluster1] creationTimestamp: "2020-08-25T18:25:34Z"
  2844. 2020-08-25T18:34:00.5717499Z [cluster1] generation: 101
  2845. 2020-08-25T18:34:00.5717761Z [cluster1] name: cluster1-worker
  2846. 2020-08-25T18:34:00.5718032Z [cluster1] namespace: submariner-operator
  2847. 2020-08-25T18:34:00.5718140Z [cluster1] resourceVersion: "3781"
  2848. 2020-08-25T18:34:00.5718637Z [cluster1] selfLink: /apis/submariner.io/v1/namespaces/submariner-operator/gateways/cluster1-worker
  2849. 2020-08-25T18:34:00.5718943Z [cluster1] uid: 418a3579-d29c-4549-bcf0-c193ac7deae6
  2850. 2020-08-25T18:34:00.5719063Z [cluster1] status:
  2851. 2020-08-25T18:34:00.5719174Z [cluster1] connections:
  2852. 2020-08-25T18:34:00.5719412Z [cluster1] - endpoint:
  2853. 2020-08-25T18:34:00.5719532Z [cluster1] backend: strongswan
  2854. 2020-08-25T18:34:00.5721375Z [cluster1] cable_name: submariner-cable-cluster2-172-17-0-8
  2855. 2020-08-25T18:34:00.5722492Z [cluster1] cluster_id: cluster2
  2856. 2020-08-25T18:34:00.5723190Z [cluster1] hostname: cluster2-worker
  2857. 2020-08-25T18:34:00.5723340Z [cluster1] nat_enabled: false
  2858. 2020-08-25T18:34:00.5723480Z [cluster1] private_ip: 172.17.0.8
  2859. 2020-08-25T18:34:00.5723604Z [cluster1] public_ip: ""
  2860. 2020-08-25T18:34:00.5723908Z [cluster1] subnets:
  2861. 2020-08-25T18:34:00.5725446Z [cluster1] - 100.92.0.0/16
  2862. 2020-08-25T18:34:00.5725728Z [cluster1] - 10.242.0.0/16
  2863. 2020-08-25T18:34:00.5725838Z [cluster1] status: connected
  2864. 2020-08-25T18:34:00.5726196Z [cluster1] statusMessage: Connected to 172.17.0.8:4500 - encryption alg=AES_GCM_16, keysize=128
  2865. 2020-08-25T18:34:00.5726482Z [cluster1] rekey-time=12591
  2866. 2020-08-25T18:34:00.5727929Z [cluster1] haStatus: active
  2867. 2020-08-25T18:34:00.5728062Z [cluster1] localEndpoint:
  2868. 2020-08-25T18:34:00.5728182Z [cluster1] backend: strongswan
  2869. 2020-08-25T18:34:00.5728558Z [cluster1] cable_name: submariner-cable-cluster1-172-17-0-9
  2870. 2020-08-25T18:34:00.5728677Z [cluster1] cluster_id: cluster1
  2871. 2020-08-25T18:34:00.5728955Z [cluster1] hostname: cluster1-worker
  2872. 2020-08-25T18:34:00.5729092Z [cluster1] nat_enabled: false
  2873. 2020-08-25T18:34:00.5729216Z [cluster1] private_ip: 172.17.0.9
  2874. 2020-08-25T18:34:00.5729338Z [cluster1] public_ip: ""
  2875. 2020-08-25T18:34:00.5729462Z [cluster1] subnets:
  2876. 2020-08-25T18:34:00.5729738Z [cluster1] - 100.91.0.0/16
  2877. 2020-08-25T18:34:00.5730002Z [cluster1] - 10.241.0.0/16
  2878. 2020-08-25T18:34:00.5730109Z [cluster1] statusFailure: ""
  2879. 2020-08-25T18:34:00.5730389Z [cluster1] version: v0.6.0-rc0-6-gd902e2d
  2880. 2020-08-25T18:34:00.5730512Z [cluster1] kind: List
  2881. 2020-08-25T18:34:00.5730630Z [cluster1] metadata:
  2882. 2020-08-25T18:34:00.5730813Z [cluster1] resourceVersion: ""
  2883. 2020-08-25T18:34:00.5730931Z [cluster1] selfLink: ""
  2884. 2020-08-25T18:34:00.5731059Z [cluster1] ===================== END Post mortem cluster1 =====================
  2885. 2020-08-25T18:34:00.5745777Z [lighthouse]$ [cluster2] post_analyze
  2886. 2020-08-25T18:34:00.5761616Z [lighthouse]$ [cluster2] sed s/^/[cluster2] /
  2887. 2020-08-25T18:34:00.5775599Z [lighthouse]$ [cluster2] post_analyze
  2888. 2020-08-25T18:34:00.5806649Z [lighthouse]$ [cluster2] kubectl get all --all-namespaces
  2889. 2020-08-25T18:34:00.5820611Z [lighthouse]$ [cluster2] kubectl get all --all-namespaces
  2890. 2020-08-25T18:34:00.5842719Z [lighthouse]$ [cluster2] command kubectl --context=cluster2 get all --all-namespaces
  2891. 2020-08-25T18:34:00.5857944Z [lighthouse]$ [cluster2] kubectl --context=cluster2 get all --all-namespaces
  2892. 2020-08-25T18:34:01.5648089Z
  2893. 2020-08-25T18:34:01.5648296Z
  2894. 2020-08-25T18:34:01.5667324Z
  2895. 2020-08-25T18:34:01.5678589Z
  2896. 2020-08-25T18:34:01.5678803Z [cluster2] ======================= Post mortem cluster2 =======================
  2897. 2020-08-25T18:34:01.5678971Z [cluster2] NAMESPACE NAME READY STATUS RESTARTS AGE
  2898. 2020-08-25T18:34:01.5679910Z [cluster2] kube-system pod/coredns-6955765f44-mhg65 1/1 Running 0 11m
  2899. 2020-08-25T18:34:01.5680345Z [cluster2] kube-system pod/coredns-6955765f44-z2fdb 1/1 Running 0 11m
  2900. 2020-08-25T18:34:01.5680732Z [cluster2] kube-system pod/etcd-cluster2-control-plane 1/1 Running 0 11m
  2901. 2020-08-25T18:34:01.5681111Z [cluster2] kube-system pod/kube-apiserver-cluster2-control-plane 1/1 Running 0 11m
  2902. 2020-08-25T18:34:01.5681642Z [cluster2] kube-system pod/kube-controller-manager-cluster2-control-plane 1/1 Running 0 11m
  2903. 2020-08-25T18:34:01.5682002Z [cluster2] kube-system pod/kube-proxy-44496 1/1 Running 0 11m
  2904. 2020-08-25T18:34:01.5682395Z [cluster2] kube-system pod/kube-proxy-bbzm9 1/1 Running 0 11m
  2905. 2020-08-25T18:34:01.5682750Z [cluster2] kube-system pod/kube-proxy-jr8sg 1/1 Running 0 11m
  2906. 2020-08-25T18:34:01.5683247Z [cluster2] kube-system pod/kube-scheduler-cluster2-control-plane 1/1 Running 0 11m
  2907. 2020-08-25T18:34:01.5683866Z [cluster2] kube-system pod/weave-net-dcsvk 2/2 Running 0 11m
  2908. 2020-08-25T18:34:01.5684269Z [cluster2] kube-system pod/weave-net-mzrnn 2/2 Running 0 11m
  2909. 2020-08-25T18:34:01.5684633Z [cluster2] kube-system pod/weave-net-scmr7 2/2 Running 0 11m
  2910. 2020-08-25T18:34:01.5684999Z [cluster2] local-path-storage pod/local-path-provisioner-7745554f7f-cg4h8 1/1 Running 0 11m
  2911. 2020-08-25T18:34:01.5685374Z [cluster2] submariner-operator pod/submariner-gateway-v6llm 1/1 Running 0 8m59s
  2912. 2020-08-25T18:34:01.5685796Z [cluster2] submariner-operator pod/submariner-lighthouse-agent-6b486fd8f5-bhzgm 1/1 Running 0 8m59s
  2913. 2020-08-25T18:34:01.5686181Z [cluster2] submariner-operator pod/submariner-lighthouse-coredns-557485fbc-9tmv6 1/1 Running 0 8m57s
  2914. 2020-08-25T18:34:01.5686551Z [cluster2] submariner-operator pod/submariner-lighthouse-coredns-557485fbc-gkrzb 1/1 Running 0 8m58s
  2915. 2020-08-25T18:34:01.5686921Z [cluster2] submariner-operator pod/submariner-operator-6675977db7-lvknj 1/1 Running 0 9m8s
  2916. 2020-08-25T18:34:01.5687285Z [cluster2] submariner-operator pod/submariner-routeagent-mjzcs 1/1 Running 0 8m59s
  2917. 2020-08-25T18:34:01.5687648Z [cluster2] submariner-operator pod/submariner-routeagent-xhj4r 1/1 Running 0 8m59s
  2918. 2020-08-25T18:34:01.5688034Z [cluster2] NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
  2919. 2020-08-25T18:34:01.5688199Z [cluster2] default service/kubernetes ClusterIP 100.92.0.1 <none> 443/TCP 11m
  2920. 2020-08-25T18:34:01.5688791Z [cluster2] kube-system service/kube-dns ClusterIP 100.92.0.10 <none> 53/UDP,53/TCP,9153/TCP 11m
  2921. 2020-08-25T18:34:01.5689197Z [cluster2] submariner-operator service/submariner-lighthouse-coredns ClusterIP 100.92.49.183 <none> 53/UDP 8m56s
  2922. 2020-08-25T18:34:01.5690173Z [cluster2] submariner-operator service/submariner-operator-metrics ClusterIP 100.92.242.245 <none> 8383/TCP,8686/TCP 9m1s
  2923. 2020-08-25T18:34:01.5690587Z [cluster2] NAMESPACE NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
  2924. 2020-08-25T18:34:01.5691012Z [cluster2] kube-system daemonset.apps/kube-proxy 3 3 3 3 3 beta.kubernetes.io/os=linux 11m
  2925. 2020-08-25T18:34:01.5691630Z [cluster2] kube-system daemonset.apps/weave-net 3 3 3 3 3 <none> 11m
  2926. 2020-08-25T18:34:01.5692129Z [cluster2] submariner-operator daemonset.apps/submariner-gateway 1 1 1 1 1 submariner.io/gateway=true 8m59s
  2927. 2020-08-25T18:34:01.5697779Z
  2928. 2020-08-25T18:34:01.5710549Z
  2929. 2020-08-25T18:34:01.5710641Z
  2930. 2020-08-25T18:34:01.5710692Z
  2931. 2020-08-25T18:34:01.5710861Z
  2932. 2020-08-25T18:34:01.5810520Z [lighthouse]$ [cluster2] kubectl get pods -A
  2933. 2020-08-25T18:34:01.5826772Z [lighthouse]$ [cluster2] kubectl get pods -A
  2934. 2020-08-25T18:34:01.5835432Z [lighthouse]$ [cluster2] tail -n +2
  2935. 2020-08-25T18:34:01.5840510Z [lighthouse]$ [cluster2] command kubectl --context=cluster2 get pods -A
  2936. 2020-08-25T18:34:01.5853719Z [lighthouse]$ [cluster2] kubectl --context=cluster2 get pods -A
  2937. 2020-08-25T18:34:01.5854052Z [lighthouse]$ [cluster2] grep -v Running
  2938. 2020-08-25T18:34:01.5878086Z [lighthouse]$ [cluster2] sed s/ */;/g
  2939. 2020-08-25T18:34:02.4114216Z [lighthouse]$ [cluster2] namespace=kube-system
  2940. 2020-08-25T18:34:02.4133201Z [lighthouse]$ [cluster2] kubectl get pods --selector=k8s-app=kube-proxy -n kube-system -o jsonpath={.items[*].metadata.name}
  2941. 2020-08-25T18:34:02.4147493Z [lighthouse]$ [cluster2] kubectl get pods --selector=k8s-app=kube-proxy -n kube-system -o jsonpath={.items[*].metadata.name}
  2942. 2020-08-25T18:34:02.4167550Z [lighthouse]$ [cluster2] command kubectl --context=cluster2 get pods --selector=k8s-app=kube-proxy -n kube-system -o jsonpath={.items[*].metadata.name}
  2943. 2020-08-25T18:34:02.4178937Z [lighthouse]$ [cluster2] kubectl --context=cluster2 get pods --selector=k8s-app=kube-proxy -n kube-system -o jsonpath={.items[*].metadata.name}
  2944. 2020-08-25T18:34:03.2955951Z [lighthouse]$ [cluster2] kubectl -n kube-system logs kube-proxy-44496
  2945. 2020-08-25T18:34:03.2962929Z [lighthouse]$ [cluster2] kubectl -n kube-system logs kube-proxy-44496
  2946. 2020-08-25T18:34:03.2976145Z [lighthouse]$ [cluster2] command kubectl --context=cluster2 -n kube-system logs kube-proxy-44496
  2947. 2020-08-25T18:34:03.2989195Z [lighthouse]$ [cluster2] kubectl --context=cluster2 -n kube-system logs kube-proxy-44496
  2948. 2020-08-25T18:34:04.1001485Z [lighthouse]$ [cluster2] kubectl -n kube-system logs kube-proxy-bbzm9
  2949. 2020-08-25T18:34:04.1014048Z [lighthouse]$ [cluster2] kubectl -n kube-system logs kube-proxy-bbzm9
  2950. 2020-08-25T18:34:04.1027833Z [lighthouse]$ [cluster2] command kubectl --context=cluster2 -n kube-system logs kube-proxy-bbzm9
  2951. 2020-08-25T18:34:04.1041406Z [lighthouse]$ [cluster2] kubectl --context=cluster2 -n kube-system logs kube-proxy-bbzm9
  2952. 2020-08-25T18:34:04.9617063Z [cluster2] submariner-operator daemonset.apps/submariner-routeagent 2 2 2 2 2 <none> 8m59s
  2953. 2020-08-25T18:34:04.9617524Z [cluster2] NAMESPACE NAME READY UP-TO-DATE AVAILABLE AGE
  2954. 2020-08-25T18:34:04.9618246Z [cluster2] kube-system deployment.apps/coredns 2/2 2 2 11m
  2955. 2020-08-25T18:34:04.9618630Z [cluster2] local-path-storage deployment.apps/local-path-provisioner 1/1 1 1 11m
  2956. 2020-08-25T18:34:04.9619014Z [cluster2] submariner-operator deployment.apps/submariner-lighthouse-agent 1/1 1 1 8m59s
  2957. 2020-08-25T18:34:04.9619388Z [cluster2] submariner-operator deployment.apps/submariner-lighthouse-coredns 2/2 2 2 8m58s
  2958. 2020-08-25T18:34:04.9619752Z [cluster2] submariner-operator deployment.apps/submariner-operator 1/1 1 1 9m8s
  2959. 2020-08-25T18:34:04.9619920Z [cluster2] NAMESPACE NAME DESIRED CURRENT READY AGE
  2960. 2020-08-25T18:34:04.9620276Z [cluster2] kube-system replicaset.apps/coredns-6955765f44 2 2 2 11m
  2961. 2020-08-25T18:34:04.9620671Z [cluster2] local-path-storage replicaset.apps/local-path-provisioner-7745554f7f 1 1 1 11m
  2962. 2020-08-25T18:34:04.9621231Z [cluster2] submariner-operator replicaset.apps/submariner-lighthouse-agent-6b486fd8f5 1 1 1 8m59s
  2963. 2020-08-25T18:34:04.9676145Z [cluster2] submariner-operator replicaset.apps/submariner-lighthouse-coredns-557485fbc 2 2 2 8m58s
  2964. 2020-08-25T18:34:04.9676820Z [cluster2] submariner-operator replicaset.apps/submariner-operator-6675977db7 1 1 1 9m8s
  2965. 2020-08-25T18:34:04.9677218Z [cluster2] +++++++++++++++++++++: Logs for Pod kube-proxy-44496 in namespace kube-system :++++++++++++++++++++++
  2966. 2020-08-25T18:34:04.9677387Z [cluster2] W0825 18:22:24.996557 1 server_others.go:323] Unknown proxy mode "", assuming iptables proxy
  2967. 2020-08-25T18:34:04.9677565Z [cluster2] I0825 18:22:25.004718 1 node.go:135] Successfully retrieved node IP: 172.17.0.5
  2968. 2020-08-25T18:34:04.9678234Z [cluster2] I0825 18:22:25.004748 1 server_others.go:145] Using iptables Proxier.
  2969. 2020-08-25T18:34:04.9678504Z [cluster2] I0825 18:22:25.005306 1 server.go:571] Version: v1.17.0
  2970. 2020-08-25T18:34:04.9679013Z [cluster2] I0825 18:22:25.006019 1 conntrack.go:100] Set sysctl 'net/netfilter/nf_conntrack_max' to 131072
  2971. 2020-08-25T18:34:04.9679284Z [cluster2] I0825 18:22:25.006061 1 conntrack.go:52] Setting nf_conntrack_max to 131072
  2972. 2020-08-25T18:34:04.9679859Z [cluster2] I0825 18:22:25.006359 1 conntrack.go:100] Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_established' to 86400
  2973. 2020-08-25T18:34:04.9680291Z [cluster2] I0825 18:22:25.006645 1 conntrack.go:100] Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_close_wait' to 3600
  2974. 2020-08-25T18:34:04.9680449Z [cluster2] I0825 18:22:25.012146 1 config.go:313] Starting service config controller
  2975. 2020-08-25T18:34:04.9680622Z [cluster2] I0825 18:22:25.012170 1 shared_informer.go:197] Waiting for caches to sync for service config
  2976. 2020-08-25T18:34:04.9680783Z [cluster2] I0825 18:22:25.012458 1 config.go:131] Starting endpoints config controller
  2977. 2020-08-25T18:34:04.9680943Z [cluster2] I0825 18:22:25.013445 1 shared_informer.go:197] Waiting for caches to sync for endpoints config
  2978. 2020-08-25T18:34:04.9681102Z [cluster2] I0825 18:22:25.112402 1 shared_informer.go:204] Caches are synced for service config
  2979. 2020-08-25T18:34:04.9681263Z [cluster2] I0825 18:22:25.114532 1 shared_informer.go:204] Caches are synced for endpoints config
  2980. 2020-08-25T18:34:04.9681632Z [cluster2] +++++++++++++++++++++: Logs for Pod kube-proxy-bbzm9 in namespace kube-system :++++++++++++++++++++++
  2981. 2020-08-25T18:34:04.9681796Z [cluster2] W0825 18:22:50.080539 1 server_others.go:323] Unknown proxy mode "", assuming iptables proxy
  2982. 2020-08-25T18:34:04.9681956Z [cluster2] I0825 18:22:50.115220 1 node.go:135] Successfully retrieved node IP: 172.17.0.8
  2983. 2020-08-25T18:34:04.9682255Z [cluster2] I0825 18:22:50.115442 1 server_others.go:145] Using iptables Proxier.
  2984. 2020-08-25T18:34:04.9682422Z [cluster2] I0825 18:22:50.115784 1 server.go:571] Version: v1.17.0
  2985. 2020-08-25T18:34:04.9682576Z [cluster2] I0825 18:22:50.116171 1 conntrack.go:52] Setting nf_conntrack_max to 131072
  2986. 2020-08-25T18:34:04.9683334Z [cluster2] I0825 18:22:50.117047 1 conntrack.go:100] Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_established' to 86400
  2987. 2020-08-25T18:34:04.9683742Z [cluster2] I0825 18:22:50.117113 1 conntrack.go:100] Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_close_wait' to 3600
  2988. 2020-08-25T18:34:04.9683898Z [cluster2] I0825 18:22:50.119543 1 config.go:313] Starting service config controller
  2989. 2020-08-25T18:34:04.9684170Z [cluster2] I0825 18:22:50.119552 1 shared_informer.go:197] Waiting for caches to sync for service config
  2990. 2020-08-25T18:34:04.9723038Z [lighthouse]$ [cluster2] kubectl -n kube-system logs kube-proxy-jr8sg
  2991. 2020-08-25T18:34:04.9732581Z [lighthouse]$ [cluster2] kubectl -n kube-system logs kube-proxy-jr8sg
  2992. 2020-08-25T18:34:04.9745920Z [lighthouse]$ [cluster2] command kubectl --context=cluster2 -n kube-system logs kube-proxy-jr8sg
  2993. 2020-08-25T18:34:04.9758550Z [lighthouse]$ [cluster2] kubectl --context=cluster2 -n kube-system logs kube-proxy-jr8sg
  2994. 2020-08-25T18:34:05.8044197Z [lighthouse]$ [cluster2] namespace=submariner-operator
  2995. 2020-08-25T18:34:05.8060947Z [lighthouse]$ [cluster2] kubectl get pods --selector=app=submariner-globalnet -n submariner-operator -o jsonpath={.items[*].metadata.name}
  2996. 2020-08-25T18:34:05.8079368Z [lighthouse]$ [cluster2] kubectl get pods --selector=app=submariner-globalnet -n submariner-operator -o jsonpath={.items[*].metadata.name}
  2997. 2020-08-25T18:34:05.8098854Z [lighthouse]$ [cluster2] command kubectl --context=cluster2 get pods --selector=app=submariner-globalnet -n submariner-operator -o jsonpath={.items[*].metadata.name}
  2998. 2020-08-25T18:34:05.8115329Z [lighthouse]$ [cluster2] kubectl --context=cluster2 get pods --selector=app=submariner-globalnet -n submariner-operator -o jsonpath={.items[*].metadata.name}
  2999. 2020-08-25T18:34:06.6440552Z [lighthouse]$ [cluster2] kubectl get pods --selector=app=submariner-engine -n submariner-operator -o jsonpath={.items[*].metadata.name}
  3000. 2020-08-25T18:34:06.6452813Z [lighthouse]$ [cluster2] kubectl get pods --selector=app=submariner-engine -n submariner-operator -o jsonpath={.items[*].metadata.name}
  3001. 2020-08-25T18:34:06.6469565Z [lighthouse]$ [cluster2] command kubectl --context=cluster2 get pods --selector=app=submariner-engine -n submariner-operator -o jsonpath={.items[*].metadata.name}
  3002. 2020-08-25T18:34:06.6482588Z [lighthouse]$ [cluster2] kubectl --context=cluster2 get pods --selector=app=submariner-engine -n submariner-operator -o jsonpath={.items[*].metadata.name}
  3003. 2020-08-25T18:34:07.4922124Z [lighthouse]$ [cluster2] kubectl -n submariner-operator logs submariner-gateway-v6llm
  3004. 2020-08-25T18:34:07.4933482Z [lighthouse]$ [cluster2] kubectl -n submariner-operator logs submariner-gateway-v6llm
  3005. 2020-08-25T18:34:07.4951349Z [lighthouse]$ [cluster2] command kubectl --context=cluster2 -n submariner-operator logs submariner-gateway-v6llm
  3006. 2020-08-25T18:34:07.4962915Z [lighthouse]$ [cluster2] kubectl --context=cluster2 -n submariner-operator logs submariner-gateway-v6llm
  3007. 2020-08-25T18:34:08.3173832Z [cluster2] I0825 18:22:50.119603 1 config.go:131] Starting endpoints config controller
  3008. 2020-08-25T18:34:08.3174079Z [cluster2] I0825 18:22:50.119613 1 shared_informer.go:197] Waiting for caches to sync for endpoints config
  3009. 2020-08-25T18:34:08.3174247Z [cluster2] I0825 18:22:50.220186 1 shared_informer.go:204] Caches are synced for endpoints config
  3010. 2020-08-25T18:34:08.3174401Z [cluster2] I0825 18:22:50.220479 1 shared_informer.go:204] Caches are synced for service config
  3011. 2020-08-25T18:34:08.3175586Z [cluster2] +++++++++++++++++++++: Logs for Pod kube-proxy-jr8sg in namespace kube-system :++++++++++++++++++++++
  3012. 2020-08-25T18:34:08.3175841Z [cluster2] W0825 18:22:49.972673 1 server_others.go:323] Unknown proxy mode "", assuming iptables proxy
  3013. 2020-08-25T18:34:08.3175987Z [cluster2] I0825 18:22:50.010774 1 node.go:135] Successfully retrieved node IP: 172.17.0.7
  3014. 2020-08-25T18:34:08.3176114Z [cluster2] I0825 18:22:50.010980 1 server_others.go:145] Using iptables Proxier.
  3015. 2020-08-25T18:34:08.3176304Z [cluster2] I0825 18:22:50.011350 1 server.go:571] Version: v1.17.0
  3016. 2020-08-25T18:34:08.3176450Z [cluster2] I0825 18:22:50.011881 1 conntrack.go:52] Setting nf_conntrack_max to 131072
  3017. 2020-08-25T18:34:08.3176850Z [cluster2] I0825 18:22:50.015144 1 conntrack.go:100] Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_established' to 86400
  3018. 2020-08-25T18:34:08.3177212Z [cluster2] I0825 18:22:50.015230 1 conntrack.go:100] Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_close_wait' to 3600
  3019. 2020-08-25T18:34:08.3177539Z [cluster2] I0825 18:22:50.015936 1 config.go:313] Starting service config controller
  3020. 2020-08-25T18:34:08.3177696Z [cluster2] I0825 18:22:50.015957 1 shared_informer.go:197] Waiting for caches to sync for service config
  3021. 2020-08-25T18:34:08.3177846Z [cluster2] I0825 18:22:50.015971 1 config.go:131] Starting endpoints config controller
  3022. 2020-08-25T18:34:08.3178159Z [cluster2] I0825 18:22:50.015980 1 shared_informer.go:197] Waiting for caches to sync for endpoints config
  3023. 2020-08-25T18:34:08.3178307Z [cluster2] I0825 18:22:50.116137 1 shared_informer.go:204] Caches are synced for endpoints config
  3024. 2020-08-25T18:34:08.3178453Z [cluster2] I0825 18:22:50.116229 1 shared_informer.go:204] Caches are synced for service config
  3025. 2020-08-25T18:34:08.3178575Z [cluster2] ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
  3026. 2020-08-25T18:34:08.3178715Z [cluster2] ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
  3027. 2020-08-25T18:34:08.3179393Z [cluster2] +++++++++++++++++++++: Logs for Pod submariner-gateway-v6llm in namespace submariner-operator :++++++++++++++++++++++
  3028. 2020-08-25T18:34:08.3179720Z [cluster2] + trap 'exit 1' SIGTERM SIGINT
  3029. 2020-08-25T18:34:08.3179839Z [cluster2] + export CHARON_PID_FILE=/var/run/charon.pid
  3030. 2020-08-25T18:34:08.3179961Z [cluster2] + CHARON_PID_FILE=/var/run/charon.pid
  3031. 2020-08-25T18:34:08.3180208Z [cluster2] + rm -f /var/run/charon.pid
  3032. 2020-08-25T18:34:08.3180325Z [cluster2] + SUBMARINER_VERBOSITY=2
  3033. 2020-08-25T18:34:08.3180565Z [cluster2] + '[' false == true ']'
  3034. 2020-08-25T18:34:08.3180775Z [cluster2] + DEBUG=-v=2
  3035. 2020-08-25T18:34:08.3181009Z [cluster2] + mkdir -p /etc/ipsec
  3036. 2020-08-25T18:34:08.3181275Z [cluster2] + sysctl -w net.ipv4.conf.all.send_redirects=0
  3037. 2020-08-25T18:34:08.3181415Z [cluster2] + export PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/libexec/strongswan
  3038. 2020-08-25T18:34:08.3181563Z [cluster2] + PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/libexec/strongswan
  3039. 2020-08-25T18:34:08.3181835Z [cluster2] + for f in iptables-save iptables
  3040. 2020-08-25T18:34:08.3182091Z [cluster2] ++ find_iptables_on_host iptables-save
  3041. 2020-08-25T18:34:08.3182357Z [cluster2] ++ chroot /host test -x /usr/sbin/iptables-save
  3042. 2020-08-25T18:34:08.3182466Z [cluster2] net.ipv4.conf.all.send_redirects = 0
  3043. 2020-08-25T18:34:08.3182582Z [cluster2] ++ echo /usr/sbin
  3044. 2020-08-25T18:34:08.3182691Z [cluster2] ++ return
  3045. 2020-08-25T18:34:08.3182804Z [cluster2] + location=/usr/sbin
  3046. 2020-08-25T18:34:08.3183056Z [cluster2] + '[' /usr/sbin '!=' unknown ']'
  3047. 2020-08-25T18:34:08.3183348Z [cluster2] + echo 'iptables-save is present on the host at /usr/sbin/iptables-save'
  3048. 2020-08-25T18:34:08.3183637Z [cluster2] + sed 's!@@PATH@@!/usr/sbin!' /usr/sbin/iptables-wrapper.in
  3049. 2020-08-25T18:34:08.3183895Z [cluster2] + for f in iptables-save iptables
  3050. 2020-08-25T18:34:08.3183996Z [cluster2] ++ find_iptables_on_host iptables
  3051. 2020-08-25T18:34:08.3185133Z [cluster2] ++ chroot /host test -x /usr/sbin/iptables
  3052. 2020-08-25T18:34:08.3185485Z [cluster2] iptables-save is present on the host at /usr/sbin/iptables-save
  3053. 2020-08-25T18:34:08.3185614Z [cluster2] ++ echo /usr/sbin
  3054. 2020-08-25T18:34:08.3185728Z [cluster2] ++ return
  3055. 2020-08-25T18:34:08.3185841Z [cluster2] + location=/usr/sbin
  3056. 2020-08-25T18:34:08.3186261Z [cluster2] + '[' /usr/sbin '!=' unknown ']'
  3057. 2020-08-25T18:34:08.3186537Z [cluster2] + echo 'iptables is present on the host at /usr/sbin/iptables'
  3058. 2020-08-25T18:34:08.3186805Z [cluster2] + sed 's!@@PATH@@!/usr/sbin!' /usr/sbin/iptables-wrapper.in
  3059. 2020-08-25T18:34:08.3186935Z [cluster2] iptables is present on the host at /usr/sbin/iptables
  3060. 2020-08-25T18:34:08.3187208Z [cluster2] + exec submariner-engine -v=2 -alsologtostderr
  3061. 2020-08-25T18:34:08.3187342Z [cluster2] I0825 18:25:34.821564 1 main.go:67] Starting the submariner gateway engine
  3062. 2020-08-25T18:34:08.3214157Z [cluster2] W0825 18:25:34.828307 1 client_config.go:543] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work.
  3063. 2020-08-25T18:34:08.3214783Z [cluster2] I0825 18:25:34.828988 1 main.go:93] Creating the cable engine
  3064. 2020-08-25T18:34:08.3214946Z [cluster2] I0825 18:25:34.851436 1 syncer.go:48] CableEngine syncer started
  3065. 2020-08-25T18:34:08.3215123Z [cluster2] I0825 18:25:34.852379 1 main.go:232] Gateway leader election config values: main.leaderConfig{LeaseDuration:10, RenewDeadline:5, RetryPeriod:2}
  3066. 2020-08-25T18:34:08.3215571Z [cluster2] I0825 18:25:34.852550 1 main.go:249] Using namespace "submariner-operator" for the leader election lock
  3067. 2020-08-25T18:34:08.3215995Z [cluster2] I0825 18:25:34.852570 1 leaderelection.go:242] attempting to acquire leader lease submariner-operator/submariner-engine-lock...
  3068. 2020-08-25T18:34:08.3216410Z [cluster2] I0825 18:25:35.311800 1 leaderelection.go:252] successfully acquired lease submariner-operator/submariner-engine-lock
  3069. 2020-08-25T18:34:08.3217466Z [cluster2] I0825 18:25:35.312196 1 event.go:281] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"submariner-operator", Name:"submariner-engine-lock", UID:"e325fbdc-e4ec-4597-935e-0158c875863d", APIVersion:"v1", ResourceVersion:"1261", FieldPath:""}): type: 'Normal' reason: 'LeaderElection' cluster2-worker-submariner-engine became leader
  3070. 2020-08-25T18:34:08.3217697Z [cluster2] I0825 18:25:35.312221 1 main.go:134] Creating the tunnel controller
  3071. 2020-08-25T18:34:08.3217852Z [cluster2] I0825 18:25:35.312301 1 main.go:142] Creating the kubernetes central datastore
  3072. 2020-08-25T18:34:08.3218052Z [cluster2] I0825 18:25:35.312327 1 kubernetes.go:60] Rendered API server host: "https://172.17.0.4:6443"
  3073. 2020-08-25T18:34:08.3218221Z [cluster2] I0825 18:25:35.787696 1 main.go:152] Creating the datastore syncer
  3074. 2020-08-25T18:34:08.3218377Z [cluster2] I0825 18:25:35.787828 1 datastoresyncer.go:145] Starting the datastore syncer
  3075. 2020-08-25T18:34:08.3218539Z [cluster2] I0825 18:25:35.787852 1 datastoresyncer.go:147] Waiting for informer caches to sync
  3076. 2020-08-25T18:34:08.3218694Z [cluster2] I0825 18:25:35.788641 1 tunnel.go:47] Starting the tunnel controller
  3077. 2020-08-25T18:34:08.3218828Z [cluster2] I0825 18:25:35.788649 1 tunnel.go:50] Waiting for informer caches to sync
  3078. 2020-08-25T18:34:08.3218979Z [cluster2] I0825 18:25:35.788699 1 strongswan.go:97] Initializing StrongSwan IPSec driver
  3079. 2020-08-25T18:34:08.3219125Z [cluster2] I0825 18:25:35.789227 1 strongswan.go:458] Starting charon
  3080. 2020-08-25T18:34:08.3219601Z [cluster2] W0825 18:25:35.812850 1 strongswan.go:541] Failed to connect to charon - retrying: dial unix /var/run/charon.vici: connect: no such file or directory
  3081. 2020-08-25T18:34:08.3219819Z [cluster2] I0825 18:25:35.890614 1 tunnel.go:58] Tunnel controller started
  3082. 2020-08-25T18:34:08.3220550Z [cluster2] I0825 18:25:35.890719 1 datastoresyncer.go:79] Ensuring we are the only endpoint active for this cluster
  3083. 2020-08-25T18:34:08.3220955Z [cluster2] I0825 18:25:35.894289 1 datastoresyncer.go:158] Reconciling local submariner Cluster: types.SubmarinerCluster{ID:"cluster2", Spec:v1.ClusterSpec{ClusterID:"cluster2", ColorCodes:[]string{"blue"}, ServiceCIDR:[]string{"100.92.0.0/16"}, ClusterCIDR:[]string{"10.242.0.0/16"}, GlobalCIDR:[]string{}}}
  3084. 2020-08-25T18:34:08.3221186Z [cluster2] I0825 18:25:35.894313 1 datastoresyncer.go:312] In reconcileClusterCRD: &types.SubmarinerCluster{ID:"cluster2", Spec:v1.ClusterSpec{ClusterID:"cluster2", ColorCodes:[]string{"blue"}, ServiceCIDR:[]string{"100.92.0.0/16"}, ClusterCIDR:[]string{"10.242.0.0/16"}, GlobalCIDR:[]string{}}}
  3085. 2020-08-25T18:34:08.3221390Z [cluster2] I0825 18:25:35.918289 1 datastoresyncer.go:357] Successfully created submariner Cluster "cluster2" in the local datastore
  3086. 2020-08-25T18:34:08.3222249Z [cluster2] I0825 18:25:35.918309 1 datastoresyncer.go:164] Reconciling local submariner Endpoint: types.SubmarinerEndpoint{Spec:v1.EndpointSpec{ClusterID:"cluster2", CableName:"submariner-cable-cluster2-172-17-0-8", Hostname:"cluster2-worker", Subnets:[]string{"100.92.0.0/16", "10.242.0.0/16"}, PrivateIP:"172.17.0.8", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string{}}}
  3087. 2020-08-25T18:34:08.3223035Z [cluster2] I0825 18:25:35.918347 1 datastoresyncer.go:387] In reconcileEndpointCRD: &types.SubmarinerEndpoint{Spec:v1.EndpointSpec{ClusterID:"cluster2", CableName:"submariner-cable-cluster2-172-17-0-8", Hostname:"cluster2-worker", Subnets:[]string{"100.92.0.0/16", "10.242.0.0/16"}, PrivateIP:"172.17.0.8", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string{}}}
  3088. 2020-08-25T18:34:08.3235228Z [cluster2] I0825 18:25:36.114956 1 tunnel.go:95] Tunnel controller processing added or updated submariner Endpoint object: &v1.Endpoint{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"cluster2-submariner-cable-cluster2-172-17-0-8", GenerateName:"", Namespace:"submariner-operator", SelfLink:"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster2-submariner-cable-cluster2-172-17-0-8", UID:"7522cb4c-547c-4d7e-b1bd-dd836734fb3a", ResourceVersion:"1266", Generation:1, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63733976735, loc:(*time.Location)(0x2155f60)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v1.EndpointSpec{ClusterID:"cluster2", CableName:"submariner-cable-cluster2-172-17-0-8", Hostname:"cluster2-worker", Subnets:[]string{"100.92.0.0/16", "10.242.0.0/16"}, PrivateIP:"172.17.0.8", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  3089. 2020-08-25T18:34:08.3235729Z [cluster2] I0825 18:25:36.115103 1 cableengine.go:94] Not installing cable for local cluster
  3090. 2020-08-25T18:34:08.3236245Z [cluster2] I0825 18:25:36.115123 1 tunnel.go:108] Tunnel controller successfully installed Endpoint cable submariner-cable-cluster2-172-17-0-8 in the engine
  3091. 2020-08-25T18:34:08.3236741Z [cluster2] I0825 18:25:36.126408 1 datastoresyncer.go:432] Successfully created submariner Endpoint "cluster2-submariner-cable-cluster2-172-17-0-8" in the local datastore
  3092. 2020-08-25T18:34:08.3237811Z [cluster2] I0825 18:25:36.138631 1 datastoresyncer.go:177] Datastore syncer started
  3093. 2020-08-25T18:34:08.3238021Z [cluster2] I0825 18:25:36.142120 1 kubernetes.go:317] In SetCluster: &types.SubmarinerCluster{ID:"cluster2", Spec:v1.ClusterSpec{ClusterID:"cluster2", ColorCodes:[]string{"blue"}, ServiceCIDR:[]string{"100.92.0.0/16"}, ClusterCIDR:[]string{"10.242.0.0/16"}, GlobalCIDR:[]string{}}}
  3094. 2020-08-25T18:34:08.3238206Z [cluster2] I0825 18:25:36.143044 1 kubernetes.go:155] AddFunc in WatchClusters called
  3095. 2020-08-25T18:34:08.3238406Z [cluster2] I0825 18:25:36.143070 1 datastoresyncer.go:312] In reconcileClusterCRD: &types.SubmarinerCluster{ID:"cluster1", Spec:v1.ClusterSpec{ClusterID:"cluster1", ColorCodes:[]string{"blue"}, ServiceCIDR:[]string{"100.91.0.0/16"}, ClusterCIDR:[]string{"10.241.0.0/16"}, GlobalCIDR:[]string{}}}
  3096. 2020-08-25T18:34:08.3238754Z [cluster2] I0825 18:25:36.303759 1 kubernetes.go:240] AddFunc in WatchEndpoints called
  3097. 2020-08-25T18:34:08.3239547Z [cluster2] I0825 18:25:36.303780 1 datastoresyncer.go:387] In reconcileEndpointCRD: &types.SubmarinerEndpoint{Spec:v1.EndpointSpec{ClusterID:"cluster1", CableName:"submariner-cable-cluster1-172-17-0-9", Hostname:"cluster1-worker", Subnets:[]string{"100.91.0.0/16", "10.241.0.0/16"}, PrivateIP:"172.17.0.9", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  3098. 2020-08-25T18:34:08.3241163Z [cluster2] I0825 18:25:36.369775 1 datastoresyncer.go:287] Processing local submariner Endpoint object: &v1.Endpoint{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"cluster2-submariner-cable-cluster2-172-17-0-8", GenerateName:"", Namespace:"submariner-operator", SelfLink:"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster2-submariner-cable-cluster2-172-17-0-8", UID:"7522cb4c-547c-4d7e-b1bd-dd836734fb3a", ResourceVersion:"1266", Generation:1, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63733976735, loc:(*time.Location)(0x2155f60)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v1.EndpointSpec{ClusterID:"cluster2", CableName:"submariner-cable-cluster2-172-17-0-8", Hostname:"cluster2-worker", Subnets:[]string{"100.92.0.0/16", "10.242.0.0/16"}, PrivateIP:"172.17.0.8", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  3099. 2020-08-25T18:34:08.3242139Z [cluster2] I0825 18:25:36.369840 1 kubernetes.go:370] In SetEndpoint: &types.SubmarinerEndpoint{Spec:v1.EndpointSpec{ClusterID:"cluster2", CableName:"submariner-cable-cluster2-172-17-0-8", Hostname:"cluster2-worker", Subnets:[]string{"100.92.0.0/16", "10.242.0.0/16"}, PrivateIP:"172.17.0.8", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  3100. 2020-08-25T18:34:08.3242664Z [cluster2] I0825 18:25:36.570979 1 datastoresyncer.go:432] Successfully created submariner Endpoint "cluster1-submariner-cable-cluster1-172-17-0-9" in the local datastore
  3101. 2020-08-25T18:34:08.3243224Z [cluster2] I0825 18:25:36.571095 1 datastoresyncer.go:357] Successfully created submariner Cluster "cluster1" in the local datastore
  3102. 2020-08-25T18:34:08.3244863Z [cluster2] I0825 18:25:36.575946 1 tunnel.go:95] Tunnel controller processing added or updated submariner Endpoint object: &v1.Endpoint{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"cluster1-submariner-cable-cluster1-172-17-0-9", GenerateName:"", Namespace:"submariner-operator", SelfLink:"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster1-submariner-cable-cluster1-172-17-0-9", UID:"e37267ea-4dd1-41b6-9d58-fae12493cc85", ResourceVersion:"1271", Generation:1, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63733976736, loc:(*time.Location)(0x2155f60)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v1.EndpointSpec{ClusterID:"cluster1", CableName:"submariner-cable-cluster1-172-17-0-9", Hostname:"cluster1-worker", Subnets:[]string{"100.91.0.0/16", "10.241.0.0/16"}, PrivateIP:"172.17.0.9", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  3103. 2020-08-25T18:34:08.3245449Z [cluster2] I0825 18:25:36.576016 1 cableengine.go:103] Installing Endpoint cable "submariner-cable-cluster1-172-17-0-9"
  3104. 2020-08-25T18:34:08.3245727Z [cluster2] I0825 18:25:36.655487 1 kubernetes.go:155] AddFunc in WatchClusters called
  3105. 2020-08-25T18:34:08.3245890Z [cluster2] I0825 18:25:36.678694 1 kubernetes.go:342] Successfully created submariner Cluster "cluster2" in the central datastore
  3106. 2020-08-25T18:34:08.3246050Z [cluster2] I0825 18:25:36.814061 1 cableengine.go:73] CableEngine controller started, driver: "strongswan"
  3107. 2020-08-25T18:34:08.3246508Z [cluster2] I0825 18:25:36.847750 1 cableengine.go:131] Successfully installed Endpoint cable "submariner-cable-cluster1-172-17-0-9" with remote IP 172.17.0.9
  3108. 2020-08-25T18:34:08.3246938Z [cluster2] I0825 18:25:36.847777 1 tunnel.go:108] Tunnel controller successfully installed Endpoint cable submariner-cable-cluster1-172-17-0-9 in the engine
  3109. 2020-08-25T18:34:08.3247097Z [cluster2] I0825 18:25:36.885119 1 kubernetes.go:240] AddFunc in WatchEndpoints called
  3110. 2020-08-25T18:34:08.3247542Z [cluster2] I0825 18:25:36.933287 1 kubernetes.go:395] Successfully created submariner Endpoint "cluster2-submariner-cable-cluster2-172-17-0-8" in the central datastore
  3111. 2020-08-25T18:34:08.3247980Z [cluster2] I0825 18:25:36.936664 1 datastoresyncer.go:271] The updated submariner Endpoint "cluster1" is not for this cluster - skipping updating the datastore
  3112. 2020-08-25T18:34:08.3248275Z [cluster2] I0825 18:26:06.143710 1 kubernetes.go:180] UpdateFunc in WatchClusters called
  3113. 2020-08-25T18:34:08.3248472Z [cluster2] I0825 18:26:06.143736 1 datastoresyncer.go:312] In reconcileClusterCRD: &types.SubmarinerCluster{ID:"cluster1", Spec:v1.ClusterSpec{ClusterID:"cluster1", ColorCodes:[]string{"blue"}, ServiceCIDR:[]string{"100.91.0.0/16"}, ClusterCIDR:[]string{"10.241.0.0/16"}, GlobalCIDR:[]string{}}}
  3114. 2020-08-25T18:34:08.3248898Z [cluster2] I0825 18:26:06.147618 1 datastoresyncer.go:360] Cluster "cluster1" matched what we received from datastore - not updating
  3115. 2020-08-25T18:34:08.3249063Z [cluster2] I0825 18:26:06.147635 1 kubernetes.go:180] UpdateFunc in WatchClusters called
  3116. 2020-08-25T18:34:08.3249295Z [cluster2] I0825 18:26:06.303994 1 kubernetes.go:264] UpdateFunc in WatchEndpoints called
  3117. 2020-08-25T18:34:08.3250014Z [cluster2] I0825 18:26:06.304157 1 datastoresyncer.go:387] In reconcileEndpointCRD: &types.SubmarinerEndpoint{Spec:v1.EndpointSpec{ClusterID:"cluster1", CableName:"submariner-cable-cluster1-172-17-0-9", Hostname:"cluster1-worker", Subnets:[]string{"100.91.0.0/16", "10.241.0.0/16"}, PrivateIP:"172.17.0.9", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  3118. 2020-08-25T18:34:08.3250504Z [cluster2] I0825 18:26:06.309259 1 datastoresyncer.go:435] Endpoint "cluster1-submariner-cable-cluster1-172-17-0-9" matched what we received from datastore - not updating
  3119. 2020-08-25T18:34:08.3250701Z [cluster2] I0825 18:26:06.309444 1 kubernetes.go:264] UpdateFunc in WatchEndpoints called
  3120. 2020-08-25T18:34:08.3250906Z [cluster2] I0825 18:26:35.808954 1 kubernetes.go:317] In SetCluster: &types.SubmarinerCluster{ID:"cluster2", Spec:v1.ClusterSpec{ClusterID:"cluster2", ColorCodes:[]string{"blue"}, ServiceCIDR:[]string{"100.92.0.0/16"}, ClusterCIDR:[]string{"10.242.0.0/16"}, GlobalCIDR:[]string{}}}
  3121. 2020-08-25T18:34:08.3252499Z [cluster2] I0825 18:26:35.808899 1 tunnel.go:95] Tunnel controller processing added or updated submariner Endpoint object: &v1.Endpoint{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"cluster2-submariner-cable-cluster2-172-17-0-8", GenerateName:"", Namespace:"submariner-operator", SelfLink:"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster2-submariner-cable-cluster2-172-17-0-8", UID:"7522cb4c-547c-4d7e-b1bd-dd836734fb3a", ResourceVersion:"1266", Generation:1, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63733976735, loc:(*time.Location)(0x2155f60)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v1.EndpointSpec{ClusterID:"cluster2", CableName:"submariner-cable-cluster2-172-17-0-8", Hostname:"cluster2-worker", Subnets:[]string{"100.92.0.0/16", "10.242.0.0/16"}, PrivateIP:"172.17.0.8", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  3122. 2020-08-25T18:34:08.3252966Z [cluster2] I0825 18:26:35.809443 1 cableengine.go:94] Not installing cable for local cluster
  3123. 2020-08-25T18:34:08.3253415Z [cluster2] I0825 18:26:35.809450 1 tunnel.go:108] Tunnel controller successfully installed Endpoint cable submariner-cable-cluster2-172-17-0-8 in the engine
  3124. 2020-08-25T18:34:08.3254976Z [cluster2] I0825 18:26:35.809457 1 tunnel.go:95] Tunnel controller processing added or updated submariner Endpoint object: &v1.Endpoint{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"cluster1-submariner-cable-cluster1-172-17-0-9", GenerateName:"", Namespace:"submariner-operator", SelfLink:"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster1-submariner-cable-cluster1-172-17-0-9", UID:"e37267ea-4dd1-41b6-9d58-fae12493cc85", ResourceVersion:"1271", Generation:1, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63733976736, loc:(*time.Location)(0x2155f60)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v1.EndpointSpec{ClusterID:"cluster1", CableName:"submariner-cable-cluster1-172-17-0-9", Hostname:"cluster1-worker", Subnets:[]string{"100.91.0.0/16", "10.241.0.0/16"}, PrivateIP:"172.17.0.9", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  3125. 2020-08-25T18:34:08.3255580Z [cluster2] I0825 18:26:35.809497 1 cableengine.go:103] Installing Endpoint cable "submariner-cable-cluster1-172-17-0-9"
  3126. 2020-08-25T18:34:08.3256061Z [cluster2] I0825 18:26:35.813634 1 kubernetes.go:345] Cluster "cluster2" matched what we received from datastore - not updating
  3127. 2020-08-25T18:34:08.3257848Z [cluster2] I0825 18:26:35.813654 1 datastoresyncer.go:287] Processing local submariner Endpoint object: &v1.Endpoint{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"cluster2-submariner-cable-cluster2-172-17-0-8", GenerateName:"", Namespace:"submariner-operator", SelfLink:"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster2-submariner-cable-cluster2-172-17-0-8", UID:"7522cb4c-547c-4d7e-b1bd-dd836734fb3a", ResourceVersion:"1266", Generation:1, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63733976735, loc:(*time.Location)(0x2155f60)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v1.EndpointSpec{ClusterID:"cluster2", CableName:"submariner-cable-cluster2-172-17-0-8", Hostname:"cluster2-worker", Subnets:[]string{"100.92.0.0/16", "10.242.0.0/16"}, PrivateIP:"172.17.0.8", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  3128. 2020-08-25T18:34:08.3258716Z [cluster2] I0825 18:26:35.813689 1 kubernetes.go:370] In SetEndpoint: &types.SubmarinerEndpoint{Spec:v1.EndpointSpec{ClusterID:"cluster2", CableName:"submariner-cable-cluster2-172-17-0-8", Hostname:"cluster2-worker", Subnets:[]string{"100.92.0.0/16", "10.242.0.0/16"}, PrivateIP:"172.17.0.8", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  3129. 2020-08-25T18:34:08.3259158Z [cluster2] I0825 18:26:35.815447 1 cableengine.go:117] Cable "submariner-cable-cluster1-172-17-0-9" is already installed - not installing again
  3130. 2020-08-25T18:34:08.3259699Z [cluster2] I0825 18:26:35.815460 1 tunnel.go:108] Tunnel controller successfully installed Endpoint cable submariner-cable-cluster1-172-17-0-9 in the engine
  3131. 2020-08-25T18:34:08.3260144Z [cluster2] I0825 18:26:35.815998 1 kubernetes.go:398] Endpoint "cluster2-submariner-cable-cluster2-172-17-0-8" matched what we received from datastore - not updating
  3132. 2020-08-25T18:34:08.3260559Z [cluster2] I0825 18:26:35.818197 1 datastoresyncer.go:271] The updated submariner Endpoint "cluster1" is not for this cluster - skipping updating the datastore
  3133. 2020-08-25T18:34:08.3260721Z [cluster2] I0825 18:26:36.143839 1 kubernetes.go:180] UpdateFunc in WatchClusters called
  3134. 2020-08-25T18:34:08.3260914Z [cluster2] I0825 18:26:36.143865 1 datastoresyncer.go:312] In reconcileClusterCRD: &types.SubmarinerCluster{ID:"cluster1", Spec:v1.ClusterSpec{ClusterID:"cluster1", ColorCodes:[]string{"blue"}, ServiceCIDR:[]string{"100.91.0.0/16"}, ClusterCIDR:[]string{"10.241.0.0/16"}, GlobalCIDR:[]string{}}}
  3135. 2020-08-25T18:34:08.3261335Z [cluster2] I0825 18:26:36.146927 1 datastoresyncer.go:360] Cluster "cluster1" matched what we received from datastore - not updating
  3136. 2020-08-25T18:34:08.3261491Z [cluster2] I0825 18:26:36.146947 1 kubernetes.go:180] UpdateFunc in WatchClusters called
  3137. 2020-08-25T18:34:08.3261638Z [cluster2] I0825 18:26:36.304120 1 kubernetes.go:264] UpdateFunc in WatchEndpoints called
  3138. 2020-08-25T18:34:08.3262302Z [cluster2] I0825 18:26:36.304143 1 datastoresyncer.go:387] In reconcileEndpointCRD: &types.SubmarinerEndpoint{Spec:v1.EndpointSpec{ClusterID:"cluster1", CableName:"submariner-cable-cluster1-172-17-0-9", Hostname:"cluster1-worker", Subnets:[]string{"100.91.0.0/16", "10.241.0.0/16"}, PrivateIP:"172.17.0.9", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  3139. 2020-08-25T18:34:08.3262770Z [cluster2] I0825 18:26:36.307209 1 datastoresyncer.go:435] Endpoint "cluster1-submariner-cable-cluster1-172-17-0-9" matched what we received from datastore - not updating
  3140. 2020-08-25T18:34:08.3263008Z [cluster2] I0825 18:26:36.307230 1 kubernetes.go:264] UpdateFunc in WatchEndpoints called
  3141. 2020-08-25T18:34:08.3263164Z [cluster2] I0825 18:27:06.143983 1 kubernetes.go:180] UpdateFunc in WatchClusters called
  3142. 2020-08-25T18:34:08.3263542Z [cluster2] I0825 18:27:06.144006 1 datastoresyncer.go:312] In reconcileClusterCRD: &types.SubmarinerCluster{ID:"cluster1", Spec:v1.ClusterSpec{ClusterID:"cluster1", ColorCodes:[]string{"blue"}, ServiceCIDR:[]string{"100.91.0.0/16"}, ClusterCIDR:[]string{"10.241.0.0/16"}, GlobalCIDR:[]string{}}}
  3143. 2020-08-25T18:34:08.3264011Z [cluster2] I0825 18:27:06.146923 1 datastoresyncer.go:360] Cluster "cluster1" matched what we received from datastore - not updating
  3144. 2020-08-25T18:34:08.3264173Z [cluster2] I0825 18:27:06.146956 1 kubernetes.go:180] UpdateFunc in WatchClusters called
  3145. 2020-08-25T18:34:08.3264329Z [cluster2] I0825 18:27:06.304242 1 kubernetes.go:264] UpdateFunc in WatchEndpoints called
  3146. 2020-08-25T18:34:08.3265401Z [cluster2] I0825 18:27:06.304264 1 datastoresyncer.go:387] In reconcileEndpointCRD: &types.SubmarinerEndpoint{Spec:v1.EndpointSpec{ClusterID:"cluster1", CableName:"submariner-cable-cluster1-172-17-0-9", Hostname:"cluster1-worker", Subnets:[]string{"100.91.0.0/16", "10.241.0.0/16"}, PrivateIP:"172.17.0.9", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  3147. 2020-08-25T18:34:08.3265949Z [cluster2] I0825 18:27:06.308391 1 datastoresyncer.go:435] Endpoint "cluster1-submariner-cable-cluster1-172-17-0-9" matched what we received from datastore - not updating
  3148. 2020-08-25T18:34:08.3266123Z [cluster2] I0825 18:27:06.308423 1 kubernetes.go:264] UpdateFunc in WatchEndpoints called
  3149. 2020-08-25T18:34:08.3267890Z [cluster2] I0825 18:27:35.809143 1 tunnel.go:95] Tunnel controller processing added or updated submariner Endpoint object: &v1.Endpoint{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"cluster2-submariner-cable-cluster2-172-17-0-8", GenerateName:"", Namespace:"submariner-operator", SelfLink:"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster2-submariner-cable-cluster2-172-17-0-8", UID:"7522cb4c-547c-4d7e-b1bd-dd836734fb3a", ResourceVersion:"1266", Generation:1, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63733976735, loc:(*time.Location)(0x2155f60)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v1.EndpointSpec{ClusterID:"cluster2", CableName:"submariner-cable-cluster2-172-17-0-8", Hostname:"cluster2-worker", Subnets:[]string{"100.92.0.0/16", "10.242.0.0/16"}, PrivateIP:"172.17.0.8", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  3150. 2020-08-25T18:34:08.3268378Z [cluster2] I0825 18:27:35.809282 1 cableengine.go:94] Not installing cable for local cluster
  3151. 2020-08-25T18:34:08.3268836Z [cluster2] I0825 18:27:35.809290 1 tunnel.go:108] Tunnel controller successfully installed Endpoint cable submariner-cable-cluster2-172-17-0-8 in the engine
  3152. 2020-08-25T18:34:08.3270637Z [cluster2] I0825 18:27:35.809297 1 tunnel.go:95] Tunnel controller processing added or updated submariner Endpoint object: &v1.Endpoint{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"cluster1-submariner-cable-cluster1-172-17-0-9", GenerateName:"", Namespace:"submariner-operator", SelfLink:"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster1-submariner-cable-cluster1-172-17-0-9", UID:"e37267ea-4dd1-41b6-9d58-fae12493cc85", ResourceVersion:"1271", Generation:1, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63733976736, loc:(*time.Location)(0x2155f60)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v1.EndpointSpec{ClusterID:"cluster1", CableName:"submariner-cable-cluster1-172-17-0-9", Hostname:"cluster1-worker", Subnets:[]string{"100.91.0.0/16", "10.241.0.0/16"}, PrivateIP:"172.17.0.9", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  3153. 2020-08-25T18:34:08.3271331Z [cluster2] I0825 18:27:35.809324 1 cableengine.go:103] Installing Endpoint cable "submariner-cable-cluster1-172-17-0-9"
  3154. 2020-08-25T18:34:08.3271527Z [cluster2] I0825 18:27:35.809172 1 kubernetes.go:317] In SetCluster: &types.SubmarinerCluster{ID:"cluster2", Spec:v1.ClusterSpec{ClusterID:"cluster2", ColorCodes:[]string{"blue"}, ServiceCIDR:[]string{"100.92.0.0/16"}, ClusterCIDR:[]string{"10.242.0.0/16"}, GlobalCIDR:[]string{}}}
  3155. 2020-08-25T18:34:09.0182638Z [lighthouse]$ [cluster2] kubectl get Gateway -A -o yaml
  3156. 2020-08-25T18:34:09.0184410Z [cluster2] I0825 18:27:35.812833 1 datastoresyncer.go:287] Processing local submariner Endpoint object: &v1.Endpoint{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"cluster2-submariner-cable-cluster2-172-17-0-8", GenerateName:"", Namespace:"submariner-operator", SelfLink:"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster2-submariner-cable-cluster2-172-17-0-8", UID:"7522cb4c-547c-4d7e-b1bd-dd836734fb3a", ResourceVersion:"1266", Generation:1, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63733976735, loc:(*time.Location)(0x2155f60)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v1.EndpointSpec{ClusterID:"cluster2", CableName:"submariner-cable-cluster2-172-17-0-8", Hostname:"cluster2-worker", Subnets:[]string{"100.92.0.0/16", "10.242.0.0/16"}, PrivateIP:"172.17.0.8", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  3157. 2020-08-25T18:34:09.0193536Z [lighthouse]$ [cluster2] kubectl get Gateway -A -o yaml
  3158. 2020-08-25T18:34:09.0194876Z [cluster2] I0825 18:27:35.812889 1 kubernetes.go:370] In SetEndpoint: &types.SubmarinerEndpoint{Spec:v1.EndpointSpec{ClusterID:"cluster2", CableName:"submariner-cable-cluster2-172-17-0-8", Hostname:"cluster2-worker", Subnets:[]string{"100.92.0.0/16", "10.242.0.0/16"}, PrivateIP:"172.17.0.8", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  3159. 2020-08-25T18:34:09.0196493Z [lighthouse]$ [cluster2] command kubectl --context=cluster2 get Gateway -A -o yaml
  3160. 2020-08-25T18:34:09.0196912Z [cluster2] I0825 18:27:35.816061 1 cableengine.go:117] Cable "submariner-cable-cluster1-172-17-0-9" is already installed - not installing again
  3161. 2020-08-25T18:34:09.0198037Z [lighthouse]$ [cluster2] kubectl --context=cluster2 get Gateway -A -o yaml
  3162. 2020-08-25T18:34:09.0198624Z [cluster2] I0825 18:27:35.816082 1 tunnel.go:108] Tunnel controller successfully installed Endpoint cable submariner-cable-cluster1-172-17-0-9 in the engine
  3163. 2020-08-25T18:34:09.0199414Z [cluster2] I0825 18:27:35.820975 1 kubernetes.go:398] Endpoint "cluster2-submariner-cable-cluster2-172-17-0-8" matched what we received from datastore - not updating
  3164. 2020-08-25T18:34:09.0200151Z [cluster2] I0825 18:27:35.822926 1 kubernetes.go:345] Cluster "cluster2" matched what we received from datastore - not updating
  3165. 2020-08-25T18:34:09.0200738Z [cluster2] I0825 18:27:35.825819 1 datastoresyncer.go:271] The updated submariner Endpoint "cluster1" is not for this cluster - skipping updating the datastore
  3166. 2020-08-25T18:34:09.0201032Z [cluster2] I0825 18:27:36.144083 1 kubernetes.go:180] UpdateFunc in WatchClusters called
  3167. 2020-08-25T18:34:09.0201542Z [cluster2] I0825 18:27:36.144110 1 datastoresyncer.go:312] In reconcileClusterCRD: &types.SubmarinerCluster{ID:"cluster1", Spec:v1.ClusterSpec{ClusterID:"cluster1", ColorCodes:[]string{"blue"}, ServiceCIDR:[]string{"100.91.0.0/16"}, ClusterCIDR:[]string{"10.241.0.0/16"}, GlobalCIDR:[]string{}}}
  3168. 2020-08-25T18:34:09.0202219Z [cluster2] I0825 18:27:36.146755 1 datastoresyncer.go:360] Cluster "cluster1" matched what we received from datastore - not updating
  3169. 2020-08-25T18:34:09.0202513Z [cluster2] I0825 18:27:36.146778 1 kubernetes.go:180] UpdateFunc in WatchClusters called
  3170. 2020-08-25T18:34:09.0202768Z [cluster2] I0825 18:27:36.304316 1 kubernetes.go:264] UpdateFunc in WatchEndpoints called
  3171. 2020-08-25T18:34:09.0203603Z [cluster2] I0825 18:27:36.304359 1 datastoresyncer.go:387] In reconcileEndpointCRD: &types.SubmarinerEndpoint{Spec:v1.EndpointSpec{ClusterID:"cluster1", CableName:"submariner-cable-cluster1-172-17-0-9", Hostname:"cluster1-worker", Subnets:[]string{"100.91.0.0/16", "10.241.0.0/16"}, PrivateIP:"172.17.0.9", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  3172. 2020-08-25T18:34:09.0204295Z [cluster2] I0825 18:27:36.306796 1 datastoresyncer.go:435] Endpoint "cluster1-submariner-cable-cluster1-172-17-0-9" matched what we received from datastore - not updating
  3173. 2020-08-25T18:34:09.0204600Z [cluster2] I0825 18:27:36.306880 1 kubernetes.go:264] UpdateFunc in WatchEndpoints called
  3174. 2020-08-25T18:34:09.0204850Z [cluster2] I0825 18:28:06.144282 1 kubernetes.go:180] UpdateFunc in WatchClusters called
  3175. 2020-08-25T18:34:09.0205146Z [cluster2] I0825 18:28:06.144512 1 datastoresyncer.go:312] In reconcileClusterCRD: &types.SubmarinerCluster{ID:"cluster1", Spec:v1.ClusterSpec{ClusterID:"cluster1", ColorCodes:[]string{"blue"}, ServiceCIDR:[]string{"100.91.0.0/16"}, ClusterCIDR:[]string{"10.241.0.0/16"}, GlobalCIDR:[]string{}}}
  3176. 2020-08-25T18:34:09.0205707Z [cluster2] I0825 18:28:06.146956 1 datastoresyncer.go:360] Cluster "cluster1" matched what we received from datastore - not updating
  3177. 2020-08-25T18:34:09.0206145Z [cluster2] I0825 18:28:06.146985 1 kubernetes.go:180] UpdateFunc in WatchClusters called
  3178. 2020-08-25T18:34:09.0206407Z [cluster2] I0825 18:28:06.304471 1 kubernetes.go:264] UpdateFunc in WatchEndpoints called
  3179. 2020-08-25T18:34:09.0207366Z [cluster2] I0825 18:28:06.304564 1 datastoresyncer.go:387] In reconcileEndpointCRD: &types.SubmarinerEndpoint{Spec:v1.EndpointSpec{ClusterID:"cluster1", CableName:"submariner-cable-cluster1-172-17-0-9", Hostname:"cluster1-worker", Subnets:[]string{"100.91.0.0/16", "10.241.0.0/16"}, PrivateIP:"172.17.0.9", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  3180. 2020-08-25T18:34:09.0208060Z [cluster2] I0825 18:28:06.325151 1 datastoresyncer.go:435] Endpoint "cluster1-submariner-cable-cluster1-172-17-0-9" matched what we received from datastore - not updating
  3181. 2020-08-25T18:34:09.0208382Z [cluster2] I0825 18:28:06.325186 1 kubernetes.go:264] UpdateFunc in WatchEndpoints called
  3182. 2020-08-25T18:34:09.0208889Z [cluster2] 00[DMN] Starting IKE charon daemon (strongSwan 5.8.4, Linux 5.3.0-1034-azure, x86_64)
  3183. 2020-08-25T18:34:09.0209393Z [cluster2] 00[CFG] PKCS11 module '<name>' lacks library path
  3184. 2020-08-25T18:34:09.0209831Z [cluster2] 00[LIB] openssl FIPS mode(2) - enabled
  3185. 2020-08-25T18:34:09.0210725Z [cluster2] 00[CFG] loading ca certificates from '/etc/strongswan/ipsec.d/cacerts'
  3186. 2020-08-25T18:34:09.0211194Z [cluster2] 00[CFG] loading aa certificates from '/etc/strongswan/ipsec.d/aacerts'
  3187. 2020-08-25T18:34:09.0211810Z [cluster2] 00[CFG] loading ocsp signer certificates from '/etc/strongswan/ipsec.d/ocspcerts'
  3188. 2020-08-25T18:34:09.0212349Z [cluster2] 00[CFG] loading attribute certificates from '/etc/strongswan/ipsec.d/acerts'
  3189. 2020-08-25T18:34:09.0212820Z [cluster2] 00[CFG] loading crls from '/etc/strongswan/ipsec.d/crls'
  3190. 2020-08-25T18:34:09.0213289Z [cluster2] 00[CFG] loading secrets from '/etc/strongswan/ipsec.secrets'
  3191. 2020-08-25T18:34:09.0213566Z [cluster2] 00[CFG] opening triplet file /etc/strongswan/ipsec.d/triplets.dat failed: No such file or directory
  3192. 2020-08-25T18:34:09.0213939Z [cluster2] 00[CFG] loaded 0 RADIUS server configurations
  3193. 2020-08-25T18:34:09.0214196Z [cluster2] 00[CFG] HA config misses local/remote address
  3194. 2020-08-25T18:34:09.0214673Z [cluster2] 00[CFG] no script for ext-auth script defined, disabled
  3195. 2020-08-25T18:34:09.0215690Z [cluster2] 00[LIB] loaded plugins: charon pkcs11 tpm aesni aes des rc2 sha2 sha1 md4 md5 mgf1 random nonce x509 revocation constraints acert pubkey pkcs1 pkcs7 pkcs8 pkcs12 pgp dnskey sshkey pem openssl gcrypt fips-prf gmp curve25519 chapoly xcbc cmac hmac ctr ccm gcm drbg newhope curl attr kernel-netlink resolve socket-default farp stroke vici updown eap-identity eap-sim eap-aka eap-aka-3gpp eap-aka-3gpp2 eap-md5 eap-gtc eap-mschapv2 eap-dynamic eap-radius eap-tls eap-ttls eap-peap xauth-generic xauth-eap xauth-pam xauth-noauth dhcp led duplicheck unity counters
  3196. 2020-08-25T18:34:09.0216063Z [cluster2] 00[JOB] spawning 16 worker threads
  3197. 2020-08-25T18:34:09.0216503Z [cluster2] 05[CFG] loaded IKE shared key for: '172.17.0.9'
  3198. 2020-08-25T18:34:09.0216983Z [cluster2] 05[CFG] added vici connection: submariner-cable-cluster1-172-17-0-9
  3199. 2020-08-25T18:34:09.0217483Z [cluster2] 05[CFG] initiating 'submariner-child-submariner-cable-cluster1-172-17-0-9'
  3200. 2020-08-25T18:34:09.0217988Z [cluster2] 05[IKE] initiating IKE_SA submariner-cable-cluster1-172-17-0-9[1] to 172.17.0.9
  3201. 2020-08-25T18:34:09.0218278Z [cluster2] 05[ENC] generating IKE_SA_INIT request 0 [ SA KE No N(NATD_S_IP) N(NATD_D_IP) N(FRAG_SUP) N(HASH_ALG) N(REDIR_SUP) ]
  3202. 2020-08-25T18:34:09.0218800Z [cluster2] 05[NET] sending packet: from 172.17.0.8[500] to 172.17.0.9[500] (500 bytes)
  3203. 2020-08-25T18:34:09.0219285Z [cluster2] 10[NET] received packet: from 172.17.0.9[500] to 172.17.0.8[500] (36 bytes)
  3204. 2020-08-25T18:34:09.0219816Z [cluster2] 10[ENC] parsed IKE_SA_INIT response 0 [ N(NO_PROP) ]
  3205. 2020-08-25T18:34:09.0220049Z [cluster2] 10[IKE] received NO_PROPOSAL_CHOSEN notify error
  3206. 2020-08-25T18:34:09.0220764Z [cluster2] 11[NET] received packet: from 172.17.0.9[500] to 172.17.0.8[500] (500 bytes)
  3207. 2020-08-25T18:34:09.0221270Z [cluster2] 11[ENC] parsed IKE_SA_INIT request 0 [ SA KE No N(NATD_S_IP) N(NATD_D_IP) N(FRAG_SUP) N(HASH_ALG) N(REDIR_SUP) ]
  3208. 2020-08-25T18:34:09.0221687Z [cluster2] 11[IKE] 172.17.0.9 is initiating an IKE_SA
  3209. 2020-08-25T18:34:09.0222124Z [cluster2] 11[CFG] selected proposal: IKE:AES_GCM_16_128/PRF_HMAC_SHA2_256/MODP_2048
  3210. 2020-08-25T18:34:09.0222395Z [cluster2] 11[IKE] remote host is behind NAT
  3211. 2020-08-25T18:34:09.0222641Z [cluster2] 11[ENC] generating IKE_SA_INIT response 0 [ SA KE No N(NATD_S_IP) N(NATD_D_IP) N(FRAG_SUP) N(HASH_ALG) N(CHDLESS_SUP) N(MULT_AUTH) ]
  3212. 2020-08-25T18:34:09.0222911Z [cluster2] 11[NET] sending packet: from 172.17.0.8[500] to 172.17.0.9[500] (464 bytes)
  3213. 2020-08-25T18:34:09.0223140Z [cluster2] 12[NET] received packet: from 172.17.0.9[4500] to 172.17.0.8[4500] (333 bytes)
  3214. 2020-08-25T18:34:09.0223415Z [cluster2] 12[ENC] parsed IKE_AUTH request 1 [ IDi N(INIT_CONTACT) IDr AUTH SA TSi TSr N(MULT_AUTH) N(EAP_ONLY) N(MSG_ID_SYN_SUP) ]
  3215. 2020-08-25T18:34:09.0223666Z [cluster2] 12[CFG] looking for peer configs matching 172.17.0.8[172.17.0.8]...172.17.0.9[172.17.0.9]
  3216. 2020-08-25T18:34:09.0229058Z [cluster2] 12[CFG] selected peer config 'submariner-cable-cluster1-172-17-0-9'
  3217. 2020-08-25T18:34:09.0229583Z [cluster2] 12[IKE] authentication of '172.17.0.9' with pre-shared key successful
  3218. 2020-08-25T18:34:09.0229907Z [cluster2] 12[IKE] authentication of '172.17.0.8' (myself) with pre-shared key
  3219. 2020-08-25T18:34:09.0230298Z [cluster2] 12[IKE] IKE_SA submariner-cable-cluster1-172-17-0-9[2] established between 172.17.0.8[172.17.0.8]...172.17.0.9[172.17.0.9]
  3220. 2020-08-25T18:34:09.0230443Z [cluster2] 12[IKE] scheduling rekeying in 13407s
  3221. 2020-08-25T18:34:09.0230570Z [cluster2] 12[IKE] maximum IKE_SA lifetime 14847s
  3222. 2020-08-25T18:34:09.0230685Z [cluster2] 12[CFG] selected proposal: ESP:AES_GCM_16_128/NO_EXT_SEQ
  3223. 2020-08-25T18:34:09.0231498Z [cluster2] 12[IKE] CHILD_SA submariner-child-submariner-cable-cluster1-172-17-0-9{1} established with SPIs c191023b_i c1537707_o and TS 10.242.0.0/16 100.92.0.0/16 172.17.0.8/32 === 10.241.0.0/16 100.91.0.0/16 172.17.0.9/32
  3224. 2020-08-25T18:34:09.0231700Z [cluster2] 12[ENC] generating IKE_AUTH response 1 [ IDr AUTH SA TSi TSr ]
  3225. 2020-08-25T18:34:09.0231848Z [cluster2] 12[NET] sending packet: from 172.17.0.8[4500] to 172.17.0.9[4500] (257 bytes)
  3226. 2020-08-25T18:34:09.0232186Z [cluster2] 10[KNL] interface vx-submariner activated
  3227. 2020-08-25T18:34:09.0232480Z [cluster2] 14[KNL] 240.17.0.8 appeared on vx-submariner
  3228. 2020-08-25T18:34:09.0232611Z [cluster2] 10[KNL] interface vethwepg000bcdc deleted
  3229. 2020-08-25T18:34:09.0232741Z [cluster2] 13[KNL] interface vethwepl000bcdc activated
  3230. 2020-08-25T18:34:09.0232870Z [cluster2] 15[KNL] interface vethwepg1f38ff7 deleted
  3231. 2020-08-25T18:34:09.0232998Z [cluster2] 06[KNL] interface vethwepl1f38ff7 activated
  3232. 2020-08-25T18:34:09.0233129Z [cluster2] 05[KNL] interface vethwepl000bcdc deactivated
  3233. 2020-08-25T18:34:09.0233246Z [cluster2] 08[KNL] interface vethwepl000bcdc deleted
  3234. 2020-08-25T18:34:09.0233449Z [cluster2] I0825 18:28:35.809229 1 kubernetes.go:317] In SetCluster: &types.SubmarinerCluster{ID:"cluster2", Spec:v1.ClusterSpec{ClusterID:"cluster2", ColorCodes:[]string{"blue"}, ServiceCIDR:[]string{"100.92.0.0/16"}, ClusterCIDR:[]string{"10.242.0.0/16"}, GlobalCIDR:[]string{}}}
  3235. 2020-08-25T18:34:09.0235087Z [cluster2] I0825 18:28:35.809229 1 tunnel.go:95] Tunnel controller processing added or updated submariner Endpoint object: &v1.Endpoint{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"cluster2-submariner-cable-cluster2-172-17-0-8", GenerateName:"", Namespace:"submariner-operator", SelfLink:"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster2-submariner-cable-cluster2-172-17-0-8", UID:"7522cb4c-547c-4d7e-b1bd-dd836734fb3a", ResourceVersion:"1266", Generation:1, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63733976735, loc:(*time.Location)(0x2155f60)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v1.EndpointSpec{ClusterID:"cluster2", CableName:"submariner-cable-cluster2-172-17-0-8", Hostname:"cluster2-worker", Subnets:[]string{"100.92.0.0/16", "10.242.0.0/16"}, PrivateIP:"172.17.0.8", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  3236. 2020-08-25T18:34:09.0235572Z [cluster2] I0825 18:28:35.809321 1 cableengine.go:94] Not installing cable for local cluster
  3237. 2020-08-25T18:34:09.0236036Z [cluster2] I0825 18:28:35.809328 1 tunnel.go:108] Tunnel controller successfully installed Endpoint cable submariner-cable-cluster2-172-17-0-8 in the engine
  3238. 2020-08-25T18:34:09.0238967Z [cluster2] I0825 18:28:35.809335 1 tunnel.go:95] Tunnel controller processing added or updated submariner Endpoint object: &v1.Endpoint{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"cluster1-submariner-cable-cluster1-172-17-0-9", GenerateName:"", Namespace:"submariner-operator", SelfLink:"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster1-submariner-cable-cluster1-172-17-0-9", UID:"e37267ea-4dd1-41b6-9d58-fae12493cc85", ResourceVersion:"1271", Generation:1, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63733976736, loc:(*time.Location)(0x2155f60)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v1.EndpointSpec{ClusterID:"cluster1", CableName:"submariner-cable-cluster1-172-17-0-9", Hostname:"cluster1-worker", Subnets:[]string{"100.91.0.0/16", "10.241.0.0/16"}, PrivateIP:"172.17.0.9", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  3239. 2020-08-25T18:34:09.0239727Z [cluster2] I0825 18:28:35.809361 1 cableengine.go:103] Installing Endpoint cable "submariner-cable-cluster1-172-17-0-9"
  3240. 2020-08-25T18:34:09.0240360Z [cluster2] I0825 18:28:35.812868 1 kubernetes.go:345] Cluster "cluster2" matched what we received from datastore - not updating
  3241. 2020-08-25T18:34:09.0242037Z [cluster2] I0825 18:28:35.813197 1 datastoresyncer.go:287] Processing local submariner Endpoint object: &v1.Endpoint{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"cluster2-submariner-cable-cluster2-172-17-0-8", GenerateName:"", Namespace:"submariner-operator", SelfLink:"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster2-submariner-cable-cluster2-172-17-0-8", UID:"7522cb4c-547c-4d7e-b1bd-dd836734fb3a", ResourceVersion:"1266", Generation:1, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63733976735, loc:(*time.Location)(0x2155f60)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v1.EndpointSpec{ClusterID:"cluster2", CableName:"submariner-cable-cluster2-172-17-0-8", Hostname:"cluster2-worker", Subnets:[]string{"100.92.0.0/16", "10.242.0.0/16"}, PrivateIP:"172.17.0.8", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  3242. 2020-08-25T18:34:09.0242941Z [cluster2] I0825 18:28:35.813237 1 kubernetes.go:370] In SetEndpoint: &types.SubmarinerEndpoint{Spec:v1.EndpointSpec{ClusterID:"cluster2", CableName:"submariner-cable-cluster2-172-17-0-8", Hostname:"cluster2-worker", Subnets:[]string{"100.92.0.0/16", "10.242.0.0/16"}, PrivateIP:"172.17.0.8", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  3243. 2020-08-25T18:34:09.0243426Z [cluster2] I0825 18:28:35.815124 1 kubernetes.go:398] Endpoint "cluster2-submariner-cable-cluster2-172-17-0-8" matched what we received from datastore - not updating
  3244. 2020-08-25T18:34:09.0244022Z [cluster2] I0825 18:28:35.816451 1 datastoresyncer.go:271] The updated submariner Endpoint "cluster1" is not for this cluster - skipping updating the datastore
  3245. 2020-08-25T18:34:09.0244448Z [cluster2] I0825 18:28:35.817471 1 cableengine.go:117] Cable "submariner-cable-cluster1-172-17-0-9" is already installed - not installing again
  3246. 2020-08-25T18:34:09.0244872Z [cluster2] I0825 18:28:35.817533 1 tunnel.go:108] Tunnel controller successfully installed Endpoint cable submariner-cable-cluster1-172-17-0-9 in the engine
  3247. 2020-08-25T18:34:09.0245037Z [cluster2] I0825 18:28:36.144529 1 kubernetes.go:180] UpdateFunc in WatchClusters called
  3248. 2020-08-25T18:34:09.0245237Z [cluster2] I0825 18:28:36.144555 1 datastoresyncer.go:312] In reconcileClusterCRD: &types.SubmarinerCluster{ID:"cluster1", Spec:v1.ClusterSpec{ClusterID:"cluster1", ColorCodes:[]string{"blue"}, ServiceCIDR:[]string{"100.91.0.0/16"}, ClusterCIDR:[]string{"10.241.0.0/16"}, GlobalCIDR:[]string{}}}
  3249. 2020-08-25T18:34:09.0245818Z [cluster2] I0825 18:28:36.147185 1 datastoresyncer.go:360] Cluster "cluster1" matched what we received from datastore - not updating
  3250. 2020-08-25T18:34:09.0245984Z [cluster2] I0825 18:28:36.147243 1 kubernetes.go:180] UpdateFunc in WatchClusters called
  3251. 2020-08-25T18:34:09.0246141Z [cluster2] I0825 18:28:36.305137 1 kubernetes.go:264] UpdateFunc in WatchEndpoints called
  3252. 2020-08-25T18:34:09.0246845Z [cluster2] I0825 18:28:36.305162 1 datastoresyncer.go:387] In reconcileEndpointCRD: &types.SubmarinerEndpoint{Spec:v1.EndpointSpec{ClusterID:"cluster1", CableName:"submariner-cable-cluster1-172-17-0-9", Hostname:"cluster1-worker", Subnets:[]string{"100.91.0.0/16", "10.241.0.0/16"}, PrivateIP:"172.17.0.9", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  3253. 2020-08-25T18:34:09.0247344Z [cluster2] I0825 18:28:36.307768 1 datastoresyncer.go:435] Endpoint "cluster1-submariner-cable-cluster1-172-17-0-9" matched what we received from datastore - not updating
  3254. 2020-08-25T18:34:09.0247630Z [cluster2] I0825 18:28:36.307807 1 kubernetes.go:264] UpdateFunc in WatchEndpoints called
  3255. 2020-08-25T18:34:09.0247796Z [cluster2] I0825 18:29:06.144737 1 kubernetes.go:180] UpdateFunc in WatchClusters called
  3256. 2020-08-25T18:34:09.0247994Z [cluster2] I0825 18:29:06.144785 1 datastoresyncer.go:312] In reconcileClusterCRD: &types.SubmarinerCluster{ID:"cluster1", Spec:v1.ClusterSpec{ClusterID:"cluster1", ColorCodes:[]string{"blue"}, ServiceCIDR:[]string{"100.91.0.0/16"}, ClusterCIDR:[]string{"10.241.0.0/16"}, GlobalCIDR:[]string{}}}
  3257. 2020-08-25T18:34:09.0248447Z [cluster2] I0825 18:29:06.147098 1 datastoresyncer.go:360] Cluster "cluster1" matched what we received from datastore - not updating
  3258. 2020-08-25T18:34:09.0248607Z [cluster2] I0825 18:29:06.147122 1 kubernetes.go:180] UpdateFunc in WatchClusters called
  3259. 2020-08-25T18:34:09.0248769Z [cluster2] I0825 18:29:06.305272 1 kubernetes.go:264] UpdateFunc in WatchEndpoints called
  3260. 2020-08-25T18:34:09.0249459Z [cluster2] I0825 18:29:06.305296 1 datastoresyncer.go:387] In reconcileEndpointCRD: &types.SubmarinerEndpoint{Spec:v1.EndpointSpec{ClusterID:"cluster1", CableName:"submariner-cable-cluster1-172-17-0-9", Hostname:"cluster1-worker", Subnets:[]string{"100.91.0.0/16", "10.241.0.0/16"}, PrivateIP:"172.17.0.9", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  3261. 2020-08-25T18:34:09.0249946Z [cluster2] I0825 18:29:06.307544 1 datastoresyncer.go:435] Endpoint "cluster1-submariner-cable-cluster1-172-17-0-9" matched what we received from datastore - not updating
  3262. 2020-08-25T18:34:09.0250108Z [cluster2] I0825 18:29:06.307579 1 kubernetes.go:264] UpdateFunc in WatchEndpoints called
  3263. 2020-08-25T18:34:09.0252544Z [cluster2] I0825 18:29:35.809447 1 tunnel.go:95] Tunnel controller processing added or updated submariner Endpoint object: &v1.Endpoint{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"cluster2-submariner-cable-cluster2-172-17-0-8", GenerateName:"", Namespace:"submariner-operator", SelfLink:"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster2-submariner-cable-cluster2-172-17-0-8", UID:"7522cb4c-547c-4d7e-b1bd-dd836734fb3a", ResourceVersion:"1266", Generation:1, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63733976735, loc:(*time.Location)(0x2155f60)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v1.EndpointSpec{ClusterID:"cluster2", CableName:"submariner-cable-cluster2-172-17-0-8", Hostname:"cluster2-worker", Subnets:[]string{"100.92.0.0/16", "10.242.0.0/16"}, PrivateIP:"172.17.0.8", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  3264. 2020-08-25T18:34:09.0253119Z [cluster2] I0825 18:29:35.809615 1 cableengine.go:94] Not installing cable for local cluster
  3265. 2020-08-25T18:34:09.0253671Z [cluster2] I0825 18:29:35.809622 1 tunnel.go:108] Tunnel controller successfully installed Endpoint cable submariner-cable-cluster2-172-17-0-8 in the engine
  3266. 2020-08-25T18:34:09.0255713Z [cluster2] I0825 18:29:35.809629 1 tunnel.go:95] Tunnel controller processing added or updated submariner Endpoint object: &v1.Endpoint{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"cluster1-submariner-cable-cluster1-172-17-0-9", GenerateName:"", Namespace:"submariner-operator", SelfLink:"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster1-submariner-cable-cluster1-172-17-0-9", UID:"e37267ea-4dd1-41b6-9d58-fae12493cc85", ResourceVersion:"1271", Generation:1, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63733976736, loc:(*time.Location)(0x2155f60)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v1.EndpointSpec{ClusterID:"cluster1", CableName:"submariner-cable-cluster1-172-17-0-9", Hostname:"cluster1-worker", Subnets:[]string{"100.91.0.0/16", "10.241.0.0/16"}, PrivateIP:"172.17.0.9", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  3267. 2020-08-25T18:34:09.0256663Z [cluster2] I0825 18:29:35.809663 1 cableengine.go:103] Installing Endpoint cable "submariner-cable-cluster1-172-17-0-9"
  3268. 2020-08-25T18:34:09.0256855Z [cluster2] I0825 18:29:35.809906 1 kubernetes.go:317] In SetCluster: &types.SubmarinerCluster{ID:"cluster2", Spec:v1.ClusterSpec{ClusterID:"cluster2", ColorCodes:[]string{"blue"}, ServiceCIDR:[]string{"100.92.0.0/16"}, ClusterCIDR:[]string{"10.242.0.0/16"}, GlobalCIDR:[]string{}}}
  3269. 2020-08-25T18:34:09.0257277Z [cluster2] I0825 18:29:35.812437 1 kubernetes.go:345] Cluster "cluster2" matched what we received from datastore - not updating
  3270. 2020-08-25T18:34:09.0257677Z [cluster2] I0825 18:29:35.814055 1 cableengine.go:117] Cable "submariner-cable-cluster1-172-17-0-9" is already installed - not installing again
  3271. 2020-08-25T18:34:09.0258090Z [cluster2] I0825 18:29:35.814071 1 tunnel.go:108] Tunnel controller successfully installed Endpoint cable submariner-cable-cluster1-172-17-0-9 in the engine
  3272. 2020-08-25T18:34:09.0259584Z [cluster2] I0825 18:29:35.814452 1 datastoresyncer.go:287] Processing local submariner Endpoint object: &v1.Endpoint{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"cluster2-submariner-cable-cluster2-172-17-0-8", GenerateName:"", Namespace:"submariner-operator", SelfLink:"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster2-submariner-cable-cluster2-172-17-0-8", UID:"7522cb4c-547c-4d7e-b1bd-dd836734fb3a", ResourceVersion:"1266", Generation:1, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63733976735, loc:(*time.Location)(0x2155f60)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v1.EndpointSpec{ClusterID:"cluster2", CableName:"submariner-cable-cluster2-172-17-0-8", Hostname:"cluster2-worker", Subnets:[]string{"100.92.0.0/16", "10.242.0.0/16"}, PrivateIP:"172.17.0.8", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  3273. 2020-08-25T18:34:09.0260563Z [cluster2] I0825 18:29:35.814556 1 kubernetes.go:370] In SetEndpoint: &types.SubmarinerEndpoint{Spec:v1.EndpointSpec{ClusterID:"cluster2", CableName:"submariner-cable-cluster2-172-17-0-8", Hostname:"cluster2-worker", Subnets:[]string{"100.92.0.0/16", "10.242.0.0/16"}, PrivateIP:"172.17.0.8", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  3274. 2020-08-25T18:34:09.0261032Z [cluster2] I0825 18:29:35.816021 1 kubernetes.go:398] Endpoint "cluster2-submariner-cable-cluster2-172-17-0-8" matched what we received from datastore - not updating
  3275. 2020-08-25T18:34:09.0261622Z [cluster2] I0825 18:29:35.817575 1 datastoresyncer.go:271] The updated submariner Endpoint "cluster1" is not for this cluster - skipping updating the datastore
  3276. 2020-08-25T18:34:09.0261791Z [cluster2] I0825 18:29:36.144907 1 kubernetes.go:180] UpdateFunc in WatchClusters called
  3277. 2020-08-25T18:34:09.0261989Z [cluster2] I0825 18:29:36.144932 1 datastoresyncer.go:312] In reconcileClusterCRD: &types.SubmarinerCluster{ID:"cluster1", Spec:v1.ClusterSpec{ClusterID:"cluster1", ColorCodes:[]string{"blue"}, ServiceCIDR:[]string{"100.91.0.0/16"}, ClusterCIDR:[]string{"10.241.0.0/16"}, GlobalCIDR:[]string{}}}
  3278. 2020-08-25T18:34:09.0262429Z [cluster2] I0825 18:29:36.147508 1 datastoresyncer.go:360] Cluster "cluster1" matched what we received from datastore - not updating
  3279. 2020-08-25T18:34:09.0262665Z [cluster2] I0825 18:29:36.147539 1 kubernetes.go:180] UpdateFunc in WatchClusters called
  3280. 2020-08-25T18:34:09.0262830Z [cluster2] I0825 18:29:36.305475 1 kubernetes.go:264] UpdateFunc in WatchEndpoints called
  3281. 2020-08-25T18:34:09.0263542Z [cluster2] I0825 18:29:36.305573 1 datastoresyncer.go:387] In reconcileEndpointCRD: &types.SubmarinerEndpoint{Spec:v1.EndpointSpec{ClusterID:"cluster1", CableName:"submariner-cable-cluster1-172-17-0-9", Hostname:"cluster1-worker", Subnets:[]string{"100.91.0.0/16", "10.241.0.0/16"}, PrivateIP:"172.17.0.9", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  3282. 2020-08-25T18:34:09.0264022Z [cluster2] I0825 18:29:36.308428 1 datastoresyncer.go:435] Endpoint "cluster1-submariner-cable-cluster1-172-17-0-9" matched what we received from datastore - not updating
  3283. 2020-08-25T18:34:09.0264197Z [cluster2] I0825 18:29:36.308456 1 kubernetes.go:264] UpdateFunc in WatchEndpoints called
  3284. 2020-08-25T18:34:09.0264402Z [cluster2] I0825 18:30:06.144975 1 kubernetes.go:180] UpdateFunc in WatchClusters called
  3285. 2020-08-25T18:34:09.0264595Z [cluster2] I0825 18:30:06.145005 1 datastoresyncer.go:312] In reconcileClusterCRD: &types.SubmarinerCluster{ID:"cluster1", Spec:v1.ClusterSpec{ClusterID:"cluster1", ColorCodes:[]string{"blue"}, ServiceCIDR:[]string{"100.91.0.0/16"}, ClusterCIDR:[]string{"10.241.0.0/16"}, GlobalCIDR:[]string{}}}
  3286. 2020-08-25T18:34:09.0267438Z [cluster2] I0825 18:30:06.148051 1 datastoresyncer.go:360] Cluster "cluster1" matched what we received from datastore - not updating
  3287. 2020-08-25T18:34:09.0267603Z [cluster2] I0825 18:30:06.148084 1 kubernetes.go:180] UpdateFunc in WatchClusters called
  3288. 2020-08-25T18:34:09.0267757Z [cluster2] I0825 18:30:06.305538 1 kubernetes.go:264] UpdateFunc in WatchEndpoints called
  3289. 2020-08-25T18:34:09.0268199Z [cluster2] I0825 18:30:06.305559 1 kubernetes.go:264] UpdateFunc in WatchEndpoints called
  3290. 2020-08-25T18:34:09.0268962Z [cluster2] I0825 18:30:06.305567 1 datastoresyncer.go:387] In reconcileEndpointCRD: &types.SubmarinerEndpoint{Spec:v1.EndpointSpec{ClusterID:"cluster1", CableName:"submariner-cable-cluster1-172-17-0-9", Hostname:"cluster1-worker", Subnets:[]string{"100.91.0.0/16", "10.241.0.0/16"}, PrivateIP:"172.17.0.9", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  3291. 2020-08-25T18:34:09.0269454Z [cluster2] I0825 18:30:06.307321 1 datastoresyncer.go:435] Endpoint "cluster1-submariner-cable-cluster1-172-17-0-9" matched what we received from datastore - not updating
  3292. 2020-08-25T18:34:09.0271058Z [cluster2] I0825 18:30:35.810909 1 tunnel.go:95] Tunnel controller processing added or updated submariner Endpoint object: &v1.Endpoint{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"cluster2-submariner-cable-cluster2-172-17-0-8", GenerateName:"", Namespace:"submariner-operator", SelfLink:"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster2-submariner-cable-cluster2-172-17-0-8", UID:"7522cb4c-547c-4d7e-b1bd-dd836734fb3a", ResourceVersion:"1266", Generation:1, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63733976735, loc:(*time.Location)(0x2155f60)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v1.EndpointSpec{ClusterID:"cluster2", CableName:"submariner-cable-cluster2-172-17-0-8", Hostname:"cluster2-worker", Subnets:[]string{"100.92.0.0/16", "10.242.0.0/16"}, PrivateIP:"172.17.0.8", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  3293. 2020-08-25T18:34:09.0271381Z [cluster2] I0825 18:30:35.810998 1 cableengine.go:94] Not installing cable for local cluster
  3294. 2020-08-25T18:34:09.0271925Z [cluster2] I0825 18:30:35.811006 1 tunnel.go:108] Tunnel controller successfully installed Endpoint cable submariner-cable-cluster2-172-17-0-8 in the engine
  3295. 2020-08-25T18:34:09.0273673Z [cluster2] I0825 18:30:35.811013 1 tunnel.go:95] Tunnel controller processing added or updated submariner Endpoint object: &v1.Endpoint{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"cluster1-submariner-cable-cluster1-172-17-0-9", GenerateName:"", Namespace:"submariner-operator", SelfLink:"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster1-submariner-cable-cluster1-172-17-0-9", UID:"e37267ea-4dd1-41b6-9d58-fae12493cc85", ResourceVersion:"1271", Generation:1, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63733976736, loc:(*time.Location)(0x2155f60)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v1.EndpointSpec{ClusterID:"cluster1", CableName:"submariner-cable-cluster1-172-17-0-9", Hostname:"cluster1-worker", Subnets:[]string{"100.91.0.0/16", "10.241.0.0/16"}, PrivateIP:"172.17.0.9", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  3296. 2020-08-25T18:34:09.0274297Z [cluster2] I0825 18:30:35.811056 1 cableengine.go:103] Installing Endpoint cable "submariner-cable-cluster1-172-17-0-9"
  3297. 2020-08-25T18:34:09.0274666Z [cluster2] I0825 18:30:35.811304 1 kubernetes.go:317] In SetCluster: &types.SubmarinerCluster{ID:"cluster2", Spec:v1.ClusterSpec{ClusterID:"cluster2", ColorCodes:[]string{"blue"}, ServiceCIDR:[]string{"100.92.0.0/16"}, ClusterCIDR:[]string{"10.242.0.0/16"}, GlobalCIDR:[]string{}}}
  3298. 2020-08-25T18:34:09.0276636Z [cluster2] I0825 18:30:35.812939 1 datastoresyncer.go:287] Processing local submariner Endpoint object: &v1.Endpoint{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"cluster2-submariner-cable-cluster2-172-17-0-8", GenerateName:"", Namespace:"submariner-operator", SelfLink:"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster2-submariner-cable-cluster2-172-17-0-8", UID:"7522cb4c-547c-4d7e-b1bd-dd836734fb3a", ResourceVersion:"1266", Generation:1, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63733976735, loc:(*time.Location)(0x2155f60)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v1.EndpointSpec{ClusterID:"cluster2", CableName:"submariner-cable-cluster2-172-17-0-8", Hostname:"cluster2-worker", Subnets:[]string{"100.92.0.0/16", "10.242.0.0/16"}, PrivateIP:"172.17.0.8", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  3299. 2020-08-25T18:34:09.0277719Z [cluster2] I0825 18:30:35.812983 1 kubernetes.go:370] In SetEndpoint: &types.SubmarinerEndpoint{Spec:v1.EndpointSpec{ClusterID:"cluster2", CableName:"submariner-cable-cluster2-172-17-0-8", Hostname:"cluster2-worker", Subnets:[]string{"100.92.0.0/16", "10.242.0.0/16"}, PrivateIP:"172.17.0.8", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  3300. 2020-08-25T18:34:09.0278177Z [cluster2] I0825 18:30:35.814725 1 kubernetes.go:345] Cluster "cluster2" matched what we received from datastore - not updating
  3301. 2020-08-25T18:34:09.0278632Z [cluster2] I0825 18:30:35.815535 1 kubernetes.go:398] Endpoint "cluster2-submariner-cable-cluster2-172-17-0-8" matched what we received from datastore - not updating
  3302. 2020-08-25T18:34:09.0279084Z [cluster2] I0825 18:30:35.817568 1 datastoresyncer.go:271] The updated submariner Endpoint "cluster1" is not for this cluster - skipping updating the datastore
  3303. 2020-08-25T18:34:09.0279529Z [cluster2] I0825 18:30:35.817986 1 cableengine.go:117] Cable "submariner-cable-cluster1-172-17-0-9" is already installed - not installing again
  3304. 2020-08-25T18:34:09.0280272Z [cluster2] I0825 18:30:35.818016 1 tunnel.go:108] Tunnel controller successfully installed Endpoint cable submariner-cable-cluster1-172-17-0-9 in the engine
  3305. 2020-08-25T18:34:09.0280459Z [cluster2] I0825 18:30:36.145232 1 kubernetes.go:180] UpdateFunc in WatchClusters called
  3306. 2020-08-25T18:34:09.0280666Z [cluster2] I0825 18:30:36.145257 1 datastoresyncer.go:312] In reconcileClusterCRD: &types.SubmarinerCluster{ID:"cluster1", Spec:v1.ClusterSpec{ClusterID:"cluster1", ColorCodes:[]string{"blue"}, ServiceCIDR:[]string{"100.91.0.0/16"}, ClusterCIDR:[]string{"10.241.0.0/16"}, GlobalCIDR:[]string{}}}
  3307. 2020-08-25T18:34:09.0281151Z [cluster2] I0825 18:30:36.147820 1 datastoresyncer.go:360] Cluster "cluster1" matched what we received from datastore - not updating
  3308. 2020-08-25T18:34:09.0281319Z [cluster2] I0825 18:30:36.147843 1 kubernetes.go:180] UpdateFunc in WatchClusters called
  3309. 2020-08-25T18:34:09.0281489Z [cluster2] I0825 18:30:36.305914 1 kubernetes.go:264] UpdateFunc in WatchEndpoints called
  3310. 2020-08-25T18:34:09.0282205Z [cluster2] I0825 18:30:36.306305 1 datastoresyncer.go:387] In reconcileEndpointCRD: &types.SubmarinerEndpoint{Spec:v1.EndpointSpec{ClusterID:"cluster1", CableName:"submariner-cable-cluster1-172-17-0-9", Hostname:"cluster1-worker", Subnets:[]string{"100.91.0.0/16", "10.241.0.0/16"}, PrivateIP:"172.17.0.9", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  3311. 2020-08-25T18:34:09.0283042Z [cluster2] I0825 18:30:36.310549 1 datastoresyncer.go:435] Endpoint "cluster1-submariner-cable-cluster1-172-17-0-9" matched what we received from datastore - not updating
  3312. 2020-08-25T18:34:09.0283225Z [cluster2] I0825 18:30:36.310581 1 kubernetes.go:264] UpdateFunc in WatchEndpoints called
  3313. 2020-08-25T18:34:09.0283382Z [cluster2] I0825 18:31:06.145331 1 kubernetes.go:180] UpdateFunc in WatchClusters called
  3314. 2020-08-25T18:34:09.0283719Z [cluster2] I0825 18:31:06.145354 1 datastoresyncer.go:312] In reconcileClusterCRD: &types.SubmarinerCluster{ID:"cluster1", Spec:v1.ClusterSpec{ClusterID:"cluster1", ColorCodes:[]string{"blue"}, ServiceCIDR:[]string{"100.91.0.0/16"}, ClusterCIDR:[]string{"10.241.0.0/16"}, GlobalCIDR:[]string{}}}
  3315. 2020-08-25T18:34:09.0284190Z [cluster2] I0825 18:31:06.147505 1 datastoresyncer.go:360] Cluster "cluster1" matched what we received from datastore - not updating
  3316. 2020-08-25T18:34:09.0284352Z [cluster2] I0825 18:31:06.147537 1 kubernetes.go:180] UpdateFunc in WatchClusters called
  3317. 2020-08-25T18:34:09.0284508Z [cluster2] I0825 18:31:06.306098 1 kubernetes.go:264] UpdateFunc in WatchEndpoints called
  3318. 2020-08-25T18:34:09.0284664Z [cluster2] I0825 18:31:06.306139 1 kubernetes.go:264] UpdateFunc in WatchEndpoints called
  3319. 2020-08-25T18:34:09.0285377Z [cluster2] I0825 18:31:06.306146 1 datastoresyncer.go:387] In reconcileEndpointCRD: &types.SubmarinerEndpoint{Spec:v1.EndpointSpec{ClusterID:"cluster1", CableName:"submariner-cable-cluster1-172-17-0-9", Hostname:"cluster1-worker", Subnets:[]string{"100.91.0.0/16", "10.241.0.0/16"}, PrivateIP:"172.17.0.9", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  3320. 2020-08-25T18:34:09.0285889Z [cluster2] I0825 18:31:06.308548 1 datastoresyncer.go:435] Endpoint "cluster1-submariner-cable-cluster1-172-17-0-9" matched what we received from datastore - not updating
  3321. 2020-08-25T18:34:09.0286101Z [cluster2] I0825 18:31:35.810277 1 kubernetes.go:317] In SetCluster: &types.SubmarinerCluster{ID:"cluster2", Spec:v1.ClusterSpec{ClusterID:"cluster2", ColorCodes:[]string{"blue"}, ServiceCIDR:[]string{"100.92.0.0/16"}, ClusterCIDR:[]string{"10.242.0.0/16"}, GlobalCIDR:[]string{}}}
  3322. 2020-08-25T18:34:09.0287835Z [cluster2] I0825 18:31:35.811315 1 tunnel.go:95] Tunnel controller processing added or updated submariner Endpoint object: &v1.Endpoint{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"cluster2-submariner-cable-cluster2-172-17-0-8", GenerateName:"", Namespace:"submariner-operator", SelfLink:"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster2-submariner-cable-cluster2-172-17-0-8", UID:"7522cb4c-547c-4d7e-b1bd-dd836734fb3a", ResourceVersion:"1266", Generation:1, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63733976735, loc:(*time.Location)(0x2155f60)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v1.EndpointSpec{ClusterID:"cluster2", CableName:"submariner-cable-cluster2-172-17-0-8", Hostname:"cluster2-worker", Subnets:[]string{"100.92.0.0/16", "10.242.0.0/16"}, PrivateIP:"172.17.0.8", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  3323. 2020-08-25T18:34:09.0288253Z [cluster2] I0825 18:31:35.811547 1 cableengine.go:94] Not installing cable for local cluster
  3324. 2020-08-25T18:34:09.0288729Z [cluster2] I0825 18:31:35.812674 1 tunnel.go:108] Tunnel controller successfully installed Endpoint cable submariner-cable-cluster2-172-17-0-8 in the engine
  3325. 2020-08-25T18:34:09.0290364Z [cluster2] I0825 18:31:35.812682 1 tunnel.go:95] Tunnel controller processing added or updated submariner Endpoint object: &v1.Endpoint{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"cluster1-submariner-cable-cluster1-172-17-0-9", GenerateName:"", Namespace:"submariner-operator", SelfLink:"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster1-submariner-cable-cluster1-172-17-0-9", UID:"e37267ea-4dd1-41b6-9d58-fae12493cc85", ResourceVersion:"1271", Generation:1, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63733976736, loc:(*time.Location)(0x2155f60)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v1.EndpointSpec{ClusterID:"cluster1", CableName:"submariner-cable-cluster1-172-17-0-9", Hostname:"cluster1-worker", Subnets:[]string{"100.91.0.0/16", "10.241.0.0/16"}, PrivateIP:"172.17.0.9", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  3326. 2020-08-25T18:34:09.0291112Z [cluster2] I0825 18:31:35.812755 1 cableengine.go:103] Installing Endpoint cable "submariner-cable-cluster1-172-17-0-9"
  3327. 2020-08-25T18:34:09.0291521Z [cluster2] I0825 18:31:35.813398 1 kubernetes.go:345] Cluster "cluster2" matched what we received from datastore - not updating
  3328. 2020-08-25T18:34:09.0293231Z [cluster2] I0825 18:31:35.815586 1 datastoresyncer.go:287] Processing local submariner Endpoint object: &v1.Endpoint{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"cluster2-submariner-cable-cluster2-172-17-0-8", GenerateName:"", Namespace:"submariner-operator", SelfLink:"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster2-submariner-cable-cluster2-172-17-0-8", UID:"7522cb4c-547c-4d7e-b1bd-dd836734fb3a", ResourceVersion:"1266", Generation:1, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63733976735, loc:(*time.Location)(0x2155f60)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v1.EndpointSpec{ClusterID:"cluster2", CableName:"submariner-cable-cluster2-172-17-0-8", Hostname:"cluster2-worker", Subnets:[]string{"100.92.0.0/16", "10.242.0.0/16"}, PrivateIP:"172.17.0.8", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  3329. 2020-08-25T18:34:09.0294217Z [cluster2] I0825 18:31:35.815801 1 kubernetes.go:370] In SetEndpoint: &types.SubmarinerEndpoint{Spec:v1.EndpointSpec{ClusterID:"cluster2", CableName:"submariner-cable-cluster2-172-17-0-8", Hostname:"cluster2-worker", Subnets:[]string{"100.92.0.0/16", "10.242.0.0/16"}, PrivateIP:"172.17.0.8", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  3330. 2020-08-25T18:34:09.0294704Z [cluster2] I0825 18:31:35.816017 1 cableengine.go:117] Cable "submariner-cable-cluster1-172-17-0-9" is already installed - not installing again
  3331. 2020-08-25T18:34:09.0295134Z [cluster2] I0825 18:31:35.816038 1 tunnel.go:108] Tunnel controller successfully installed Endpoint cable submariner-cable-cluster1-172-17-0-9 in the engine
  3332. 2020-08-25T18:34:09.0295580Z [cluster2] I0825 18:31:35.818505 1 kubernetes.go:398] Endpoint "cluster2-submariner-cable-cluster2-172-17-0-8" matched what we received from datastore - not updating
  3333. 2020-08-25T18:34:09.0296026Z [cluster2] I0825 18:31:35.820416 1 datastoresyncer.go:271] The updated submariner Endpoint "cluster1" is not for this cluster - skipping updating the datastore
  3334. 2020-08-25T18:34:09.0296205Z [cluster2] I0825 18:31:36.145435 1 kubernetes.go:180] UpdateFunc in WatchClusters called
  3335. 2020-08-25T18:34:09.0296405Z [cluster2] I0825 18:31:36.145460 1 datastoresyncer.go:312] In reconcileClusterCRD: &types.SubmarinerCluster{ID:"cluster1", Spec:v1.ClusterSpec{ClusterID:"cluster1", ColorCodes:[]string{"blue"}, ServiceCIDR:[]string{"100.91.0.0/16"}, ClusterCIDR:[]string{"10.241.0.0/16"}, GlobalCIDR:[]string{}}}
  3336. 2020-08-25T18:34:09.0296841Z [cluster2] I0825 18:31:36.147817 1 datastoresyncer.go:360] Cluster "cluster1" matched what we received from datastore - not updating
  3337. 2020-08-25T18:34:09.0296999Z [cluster2] I0825 18:31:36.147978 1 kubernetes.go:180] UpdateFunc in WatchClusters called
  3338. 2020-08-25T18:34:09.0297151Z [cluster2] I0825 18:31:36.306240 1 kubernetes.go:264] UpdateFunc in WatchEndpoints called
  3339. 2020-08-25T18:34:09.0297963Z [cluster2] I0825 18:31:36.306262 1 datastoresyncer.go:387] In reconcileEndpointCRD: &types.SubmarinerEndpoint{Spec:v1.EndpointSpec{ClusterID:"cluster1", CableName:"submariner-cable-cluster1-172-17-0-9", Hostname:"cluster1-worker", Subnets:[]string{"100.91.0.0/16", "10.241.0.0/16"}, PrivateIP:"172.17.0.9", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  3340. 2020-08-25T18:34:09.0298474Z [cluster2] I0825 18:31:36.310580 1 datastoresyncer.go:435] Endpoint "cluster1-submariner-cable-cluster1-172-17-0-9" matched what we received from datastore - not updating
  3341. 2020-08-25T18:34:09.0298643Z [cluster2] I0825 18:31:36.310623 1 kubernetes.go:264] UpdateFunc in WatchEndpoints called
  3342. 2020-08-25T18:34:09.0298778Z [cluster2] I0825 18:32:06.145537 1 kubernetes.go:180] UpdateFunc in WatchClusters called
  3343. 2020-08-25T18:34:09.0298974Z [cluster2] I0825 18:32:06.145563 1 datastoresyncer.go:312] In reconcileClusterCRD: &types.SubmarinerCluster{ID:"cluster1", Spec:v1.ClusterSpec{ClusterID:"cluster1", ColorCodes:[]string{"blue"}, ServiceCIDR:[]string{"100.91.0.0/16"}, ClusterCIDR:[]string{"10.241.0.0/16"}, GlobalCIDR:[]string{}}}
  3344. 2020-08-25T18:34:09.0299429Z [cluster2] I0825 18:32:06.147716 1 datastoresyncer.go:360] Cluster "cluster1" matched what we received from datastore - not updating
  3345. 2020-08-25T18:34:09.0299590Z [cluster2] I0825 18:32:06.147738 1 kubernetes.go:180] UpdateFunc in WatchClusters called
  3346. 2020-08-25T18:34:09.0299728Z [cluster2] I0825 18:32:06.306354 1 kubernetes.go:264] UpdateFunc in WatchEndpoints called
  3347. 2020-08-25T18:34:09.0300414Z [cluster2] I0825 18:32:06.306381 1 datastoresyncer.go:387] In reconcileEndpointCRD: &types.SubmarinerEndpoint{Spec:v1.EndpointSpec{ClusterID:"cluster1", CableName:"submariner-cable-cluster1-172-17-0-9", Hostname:"cluster1-worker", Subnets:[]string{"100.91.0.0/16", "10.241.0.0/16"}, PrivateIP:"172.17.0.9", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  3348. 2020-08-25T18:34:09.0301002Z [cluster2] I0825 18:32:06.308701 1 datastoresyncer.go:435] Endpoint "cluster1-submariner-cable-cluster1-172-17-0-9" matched what we received from datastore - not updating
  3349. 2020-08-25T18:34:09.0301182Z [cluster2] I0825 18:32:06.308725 1 kubernetes.go:264] UpdateFunc in WatchEndpoints called
  3350. 2020-08-25T18:34:09.0305803Z [cluster2] I0825 18:32:35.810521 1 tunnel.go:95] Tunnel controller processing added or updated submariner Endpoint object: &v1.Endpoint{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"cluster2-submariner-cable-cluster2-172-17-0-8", GenerateName:"", Namespace:"submariner-operator", SelfLink:"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster2-submariner-cable-cluster2-172-17-0-8", UID:"7522cb4c-547c-4d7e-b1bd-dd836734fb3a", ResourceVersion:"1266", Generation:1, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63733976735, loc:(*time.Location)(0x2155f60)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v1.EndpointSpec{ClusterID:"cluster2", CableName:"submariner-cable-cluster2-172-17-0-8", Hostname:"cluster2-worker", Subnets:[]string{"100.92.0.0/16", "10.242.0.0/16"}, PrivateIP:"172.17.0.8", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  3351. 2020-08-25T18:34:09.0306383Z [cluster2] I0825 18:32:35.810634 1 cableengine.go:94] Not installing cable for local cluster
  3352. 2020-08-25T18:34:09.0306931Z [cluster2] I0825 18:32:35.810642 1 tunnel.go:108] Tunnel controller successfully installed Endpoint cable submariner-cable-cluster2-172-17-0-8 in the engine
  3353. 2020-08-25T18:34:09.0308714Z [cluster2] I0825 18:32:35.810649 1 tunnel.go:95] Tunnel controller processing added or updated submariner Endpoint object: &v1.Endpoint{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"cluster1-submariner-cable-cluster1-172-17-0-9", GenerateName:"", Namespace:"submariner-operator", SelfLink:"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster1-submariner-cable-cluster1-172-17-0-9", UID:"e37267ea-4dd1-41b6-9d58-fae12493cc85", ResourceVersion:"1271", Generation:1, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63733976736, loc:(*time.Location)(0x2155f60)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v1.EndpointSpec{ClusterID:"cluster1", CableName:"submariner-cable-cluster1-172-17-0-9", Hostname:"cluster1-worker", Subnets:[]string{"100.91.0.0/16", "10.241.0.0/16"}, PrivateIP:"172.17.0.9", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  3354. 2020-08-25T18:34:09.0309549Z [cluster2] I0825 18:32:35.810684 1 cableengine.go:103] Installing Endpoint cable "submariner-cable-cluster1-172-17-0-9"
  3355. 2020-08-25T18:34:09.0309749Z [cluster2] I0825 18:32:35.810632 1 kubernetes.go:317] In SetCluster: &types.SubmarinerCluster{ID:"cluster2", Spec:v1.ClusterSpec{ClusterID:"cluster2", ColorCodes:[]string{"blue"}, ServiceCIDR:[]string{"100.92.0.0/16"}, ClusterCIDR:[]string{"10.242.0.0/16"}, GlobalCIDR:[]string{}}}
  3356. 2020-08-25T18:34:09.0310203Z [cluster2] I0825 18:32:35.812908 1 cableengine.go:117] Cable "submariner-cable-cluster1-172-17-0-9" is already installed - not installing again
  3357. 2020-08-25T18:34:09.0310635Z [cluster2] I0825 18:32:35.812920 1 tunnel.go:108] Tunnel controller successfully installed Endpoint cable submariner-cable-cluster1-172-17-0-9 in the engine
  3358. 2020-08-25T18:34:09.0311031Z [cluster2] I0825 18:32:35.814533 1 kubernetes.go:345] Cluster "cluster2" matched what we received from datastore - not updating
  3359. 2020-08-25T18:34:09.0312650Z [cluster2] I0825 18:32:35.816643 1 datastoresyncer.go:287] Processing local submariner Endpoint object: &v1.Endpoint{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"cluster2-submariner-cable-cluster2-172-17-0-8", GenerateName:"", Namespace:"submariner-operator", SelfLink:"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster2-submariner-cable-cluster2-172-17-0-8", UID:"7522cb4c-547c-4d7e-b1bd-dd836734fb3a", ResourceVersion:"1266", Generation:1, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63733976735, loc:(*time.Location)(0x2155f60)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v1.EndpointSpec{ClusterID:"cluster2", CableName:"submariner-cable-cluster2-172-17-0-8", Hostname:"cluster2-worker", Subnets:[]string{"100.92.0.0/16", "10.242.0.0/16"}, PrivateIP:"172.17.0.8", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  3360. 2020-08-25T18:34:09.0313586Z [cluster2] I0825 18:32:35.816685 1 kubernetes.go:370] In SetEndpoint: &types.SubmarinerEndpoint{Spec:v1.EndpointSpec{ClusterID:"cluster2", CableName:"submariner-cable-cluster2-172-17-0-8", Hostname:"cluster2-worker", Subnets:[]string{"100.92.0.0/16", "10.242.0.0/16"}, PrivateIP:"172.17.0.8", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  3361. 2020-08-25T18:34:09.0314071Z [cluster2] I0825 18:32:35.818881 1 kubernetes.go:398] Endpoint "cluster2-submariner-cable-cluster2-172-17-0-8" matched what we received from datastore - not updating
  3362. 2020-08-25T18:34:09.0314512Z [cluster2] I0825 18:32:35.820846 1 datastoresyncer.go:271] The updated submariner Endpoint "cluster1" is not for this cluster - skipping updating the datastore
  3363. 2020-08-25T18:34:09.0314769Z [cluster2] I0825 18:32:36.145686 1 kubernetes.go:180] UpdateFunc in WatchClusters called
  3364. 2020-08-25T18:34:09.0314927Z [cluster2] I0825 18:32:36.145707 1 kubernetes.go:180] UpdateFunc in WatchClusters called
  3365. 2020-08-25T18:34:09.0315128Z [cluster2] I0825 18:32:36.145714 1 datastoresyncer.go:312] In reconcileClusterCRD: &types.SubmarinerCluster{ID:"cluster1", Spec:v1.ClusterSpec{ClusterID:"cluster1", ColorCodes:[]string{"blue"}, ServiceCIDR:[]string{"100.91.0.0/16"}, ClusterCIDR:[]string{"10.241.0.0/16"}, GlobalCIDR:[]string{}}}
  3366. 2020-08-25T18:34:09.0315577Z [cluster2] I0825 18:32:36.148295 1 datastoresyncer.go:360] Cluster "cluster1" matched what we received from datastore - not updating
  3367. 2020-08-25T18:34:09.0315744Z [cluster2] I0825 18:32:36.306710 1 kubernetes.go:264] UpdateFunc in WatchEndpoints called
  3368. 2020-08-25T18:34:09.0316440Z [cluster2] I0825 18:32:36.306736 1 datastoresyncer.go:387] In reconcileEndpointCRD: &types.SubmarinerEndpoint{Spec:v1.EndpointSpec{ClusterID:"cluster1", CableName:"submariner-cable-cluster1-172-17-0-9", Hostname:"cluster1-worker", Subnets:[]string{"100.91.0.0/16", "10.241.0.0/16"}, PrivateIP:"172.17.0.9", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  3369. 2020-08-25T18:34:09.0317605Z [cluster2] I0825 18:32:36.309815 1 datastoresyncer.go:435] Endpoint "cluster1-submariner-cable-cluster1-172-17-0-9" matched what we received from datastore - not updating
  3370. 2020-08-25T18:34:09.0317783Z [cluster2] I0825 18:32:36.309843 1 kubernetes.go:264] UpdateFunc in WatchEndpoints called
  3371. 2020-08-25T18:34:09.0317937Z [cluster2] I0825 18:33:06.145907 1 kubernetes.go:180] UpdateFunc in WatchClusters called
  3372. 2020-08-25T18:34:09.0318315Z [cluster2] I0825 18:33:06.146115 1 datastoresyncer.go:312] In reconcileClusterCRD: &types.SubmarinerCluster{ID:"cluster1", Spec:v1.ClusterSpec{ClusterID:"cluster1", ColorCodes:[]string{"blue"}, ServiceCIDR:[]string{"100.91.0.0/16"}, ClusterCIDR:[]string{"10.241.0.0/16"}, GlobalCIDR:[]string{}}}
  3373. 2020-08-25T18:34:09.0318935Z [cluster2] I0825 18:33:06.148679 1 datastoresyncer.go:360] Cluster "cluster1" matched what we received from datastore - not updating
  3374. 2020-08-25T18:34:09.0319108Z [cluster2] I0825 18:33:06.148823 1 kubernetes.go:180] UpdateFunc in WatchClusters called
  3375. 2020-08-25T18:34:09.0319260Z [cluster2] I0825 18:33:06.306966 1 kubernetes.go:264] UpdateFunc in WatchEndpoints called
  3376. 2020-08-25T18:34:09.0319408Z [cluster2] I0825 18:33:06.307048 1 kubernetes.go:264] UpdateFunc in WatchEndpoints called
  3377. 2020-08-25T18:34:09.0320121Z [cluster2] I0825 18:33:06.307070 1 datastoresyncer.go:387] In reconcileEndpointCRD: &types.SubmarinerEndpoint{Spec:v1.EndpointSpec{ClusterID:"cluster1", CableName:"submariner-cable-cluster1-172-17-0-9", Hostname:"cluster1-worker", Subnets:[]string{"100.91.0.0/16", "10.241.0.0/16"}, PrivateIP:"172.17.0.9", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  3378. 2020-08-25T18:34:09.0320621Z [cluster2] I0825 18:33:06.309384 1 datastoresyncer.go:435] Endpoint "cluster1-submariner-cable-cluster1-172-17-0-9" matched what we received from datastore - not updating
  3379. 2020-08-25T18:34:09.0322187Z [cluster2] I0825 18:33:35.810837 1 tunnel.go:95] Tunnel controller processing added or updated submariner Endpoint object: &v1.Endpoint{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"cluster2-submariner-cable-cluster2-172-17-0-8", GenerateName:"", Namespace:"submariner-operator", SelfLink:"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster2-submariner-cable-cluster2-172-17-0-8", UID:"7522cb4c-547c-4d7e-b1bd-dd836734fb3a", ResourceVersion:"1266", Generation:1, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63733976735, loc:(*time.Location)(0x2155f60)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v1.EndpointSpec{ClusterID:"cluster2", CableName:"submariner-cable-cluster2-172-17-0-8", Hostname:"cluster2-worker", Subnets:[]string{"100.92.0.0/16", "10.242.0.0/16"}, PrivateIP:"172.17.0.8", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  3380. 2020-08-25T18:34:09.0322660Z [cluster2] I0825 18:33:35.811007 1 cableengine.go:94] Not installing cable for local cluster
  3381. 2020-08-25T18:34:09.0323119Z [cluster2] I0825 18:33:35.811016 1 tunnel.go:108] Tunnel controller successfully installed Endpoint cable submariner-cable-cluster2-172-17-0-8 in the engine
  3382. 2020-08-25T18:34:09.0324661Z [cluster2] I0825 18:33:35.811023 1 tunnel.go:95] Tunnel controller processing added or updated submariner Endpoint object: &v1.Endpoint{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"cluster1-submariner-cable-cluster1-172-17-0-9", GenerateName:"", Namespace:"submariner-operator", SelfLink:"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster1-submariner-cable-cluster1-172-17-0-9", UID:"e37267ea-4dd1-41b6-9d58-fae12493cc85", ResourceVersion:"1271", Generation:1, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63733976736, loc:(*time.Location)(0x2155f60)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v1.EndpointSpec{ClusterID:"cluster1", CableName:"submariner-cable-cluster1-172-17-0-9", Hostname:"cluster1-worker", Subnets:[]string{"100.91.0.0/16", "10.241.0.0/16"}, PrivateIP:"172.17.0.9", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  3383. 2020-08-25T18:34:09.0325271Z [cluster2] I0825 18:33:35.811051 1 cableengine.go:103] Installing Endpoint cable "submariner-cable-cluster1-172-17-0-9"
  3384. 2020-08-25T18:34:09.0325548Z [cluster2] I0825 18:33:35.811319 1 kubernetes.go:317] In SetCluster: &types.SubmarinerCluster{ID:"cluster2", Spec:v1.ClusterSpec{ClusterID:"cluster2", ColorCodes:[]string{"blue"}, ServiceCIDR:[]string{"100.92.0.0/16"}, ClusterCIDR:[]string{"10.242.0.0/16"}, GlobalCIDR:[]string{}}}
  3385. 2020-08-25T18:34:09.0326032Z [cluster2] I0825 18:33:35.815309 1 cableengine.go:117] Cable "submariner-cable-cluster1-172-17-0-9" is already installed - not installing again
  3386. 2020-08-25T18:34:09.0326465Z [cluster2] I0825 18:33:35.816847 1 tunnel.go:108] Tunnel controller successfully installed Endpoint cable submariner-cable-cluster1-172-17-0-9 in the engine
  3387. 2020-08-25T18:34:09.0328009Z [cluster2] I0825 18:33:35.817142 1 datastoresyncer.go:287] Processing local submariner Endpoint object: &v1.Endpoint{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"cluster2-submariner-cable-cluster2-172-17-0-8", GenerateName:"", Namespace:"submariner-operator", SelfLink:"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster2-submariner-cable-cluster2-172-17-0-8", UID:"7522cb4c-547c-4d7e-b1bd-dd836734fb3a", ResourceVersion:"1266", Generation:1, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63733976735, loc:(*time.Location)(0x2155f60)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v1.EndpointSpec{ClusterID:"cluster2", CableName:"submariner-cable-cluster2-172-17-0-8", Hostname:"cluster2-worker", Subnets:[]string{"100.92.0.0/16", "10.242.0.0/16"}, PrivateIP:"172.17.0.8", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  3388. 2020-08-25T18:34:09.0328917Z [cluster2] I0825 18:33:35.817220 1 kubernetes.go:370] In SetEndpoint: &types.SubmarinerEndpoint{Spec:v1.EndpointSpec{ClusterID:"cluster2", CableName:"submariner-cable-cluster2-172-17-0-8", Hostname:"cluster2-worker", Subnets:[]string{"100.92.0.0/16", "10.242.0.0/16"}, PrivateIP:"172.17.0.8", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  3389. 2020-08-25T18:34:09.0329466Z [cluster2] I0825 18:33:35.823799 1 kubernetes.go:345] Cluster "cluster2" matched what we received from datastore - not updating
  3390. 2020-08-25T18:34:09.0329907Z [cluster2] I0825 18:33:35.823872 1 kubernetes.go:398] Endpoint "cluster2-submariner-cable-cluster2-172-17-0-8" matched what we received from datastore - not updating
  3391. 2020-08-25T18:34:09.0330347Z [cluster2] I0825 18:33:35.825546 1 datastoresyncer.go:271] The updated submariner Endpoint "cluster1" is not for this cluster - skipping updating the datastore
  3392. 2020-08-25T18:34:09.0330520Z [cluster2] I0825 18:33:36.146086 1 kubernetes.go:180] UpdateFunc in WatchClusters called
  3393. 2020-08-25T18:34:09.0330724Z [cluster2] I0825 18:33:36.146205 1 datastoresyncer.go:312] In reconcileClusterCRD: &types.SubmarinerCluster{ID:"cluster1", Spec:v1.ClusterSpec{ClusterID:"cluster1", ColorCodes:[]string{"blue"}, ServiceCIDR:[]string{"100.91.0.0/16"}, ClusterCIDR:[]string{"10.241.0.0/16"}, GlobalCIDR:[]string{}}}
  3394. 2020-08-25T18:34:09.0331162Z [cluster2] I0825 18:33:36.148806 1 datastoresyncer.go:360] Cluster "cluster1" matched what we received from datastore - not updating
  3395. 2020-08-25T18:34:09.0331324Z [cluster2] I0825 18:33:36.148832 1 kubernetes.go:180] UpdateFunc in WatchClusters called
  3396. 2020-08-25T18:34:09.0331483Z [cluster2] I0825 18:33:36.307121 1 kubernetes.go:264] UpdateFunc in WatchEndpoints called
  3397. 2020-08-25T18:34:09.0332386Z [cluster2] I0825 18:33:36.307144 1 datastoresyncer.go:387] In reconcileEndpointCRD: &types.SubmarinerEndpoint{Spec:v1.EndpointSpec{ClusterID:"cluster1", CableName:"submariner-cable-cluster1-172-17-0-9", Hostname:"cluster1-worker", Subnets:[]string{"100.91.0.0/16", "10.241.0.0/16"}, PrivateIP:"172.17.0.9", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  3398. 2020-08-25T18:34:09.0333479Z [cluster2] I0825 18:33:36.312842 1 datastoresyncer.go:435] Endpoint "cluster1-submariner-cable-cluster1-172-17-0-9" matched what we received from datastore - not updating
  3399. 2020-08-25T18:34:09.0333776Z [cluster2] I0825 18:33:36.312874 1 kubernetes.go:264] UpdateFunc in WatchEndpoints called
  3400. 2020-08-25T18:34:09.0333935Z [cluster2] I0825 18:34:06.146239 1 kubernetes.go:180] UpdateFunc in WatchClusters called
  3401. 2020-08-25T18:34:09.0334147Z [cluster2] I0825 18:34:06.146392 1 datastoresyncer.go:312] In reconcileClusterCRD: &types.SubmarinerCluster{ID:"cluster1", Spec:v1.ClusterSpec{ClusterID:"cluster1", ColorCodes:[]string{"blue"}, ServiceCIDR:[]string{"100.91.0.0/16"}, ClusterCIDR:[]string{"10.241.0.0/16"}, GlobalCIDR:[]string{}}}
  3402. 2020-08-25T18:34:09.1202729Z [cluster2] I0825 18:34:06.149977 1 datastoresyncer.go:360] Cluster "cluster1" matched what we received from datastore - not updating
  3403. 2020-08-25T18:34:09.1202970Z [cluster2] I0825 18:34:06.149993 1 kubernetes.go:180] UpdateFunc in WatchClusters called
  3404. 2020-08-25T18:34:09.1203146Z [cluster2] I0825 18:34:06.307229 1 kubernetes.go:264] UpdateFunc in WatchEndpoints called
  3405. 2020-08-25T18:34:09.1203936Z [cluster2] I0825 18:34:06.307259 1 datastoresyncer.go:387] In reconcileEndpointCRD: &types.SubmarinerEndpoint{Spec:v1.EndpointSpec{ClusterID:"cluster1", CableName:"submariner-cable-cluster1-172-17-0-9", Hostname:"cluster1-worker", Subnets:[]string{"100.91.0.0/16", "10.241.0.0/16"}, PrivateIP:"172.17.0.9", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  3406. 2020-08-25T18:34:09.1204453Z [cluster2] I0825 18:34:06.311257 1 datastoresyncer.go:435] Endpoint "cluster1-submariner-cable-cluster1-172-17-0-9" matched what we received from datastore - not updating
  3407. 2020-08-25T18:34:09.1204630Z [cluster2] I0825 18:34:06.311325 1 kubernetes.go:264] UpdateFunc in WatchEndpoints called
  3408. 2020-08-25T18:34:09.1205154Z [cluster2] ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
  3409. 2020-08-25T18:34:09.1205291Z [cluster2] apiVersion: v1
  3410. 2020-08-25T18:34:09.1205411Z [cluster2] items:
  3411. 2020-08-25T18:34:09.1205725Z [cluster2] - apiVersion: submariner.io/v1
  3412. 2020-08-25T18:34:09.1205853Z [cluster2] kind: Gateway
  3413. 2020-08-25T18:34:09.1205953Z [cluster2] metadata:
  3414. 2020-08-25T18:34:09.1206073Z [cluster2] annotations:
  3415. 2020-08-25T18:34:09.1206356Z [cluster2] update-timestamp: "1598380447"
  3416. 2020-08-25T18:34:09.1206654Z [cluster2] creationTimestamp: "2020-08-25T18:25:35Z"
  3417. 2020-08-25T18:34:09.1206781Z [cluster2] generation: 103
  3418. 2020-08-25T18:34:09.1207042Z [cluster2] name: cluster2-worker
  3419. 2020-08-25T18:34:09.1207312Z [cluster2] namespace: submariner-operator
  3420. 2020-08-25T18:34:09.1207443Z [cluster2] resourceVersion: "3852"
  3421. 2020-08-25T18:34:09.1207768Z [cluster2] selfLink: /apis/submariner.io/v1/namespaces/submariner-operator/gateways/cluster2-worker
  3422. 2020-08-25T18:34:09.1208089Z [cluster2] uid: 31dba903-0979-47a2-aec8-09ea8ba17eb3
  3423. 2020-08-25T18:34:09.1208212Z [cluster2] status:
  3424. 2020-08-25T18:34:09.1208457Z [cluster2] connections:
  3425. 2020-08-25T18:34:09.1208709Z [cluster2] - endpoint:
  3426. 2020-08-25T18:34:09.1208831Z [cluster2] backend: strongswan
  3427. 2020-08-25T18:34:09.1209130Z [cluster2] cable_name: submariner-cable-cluster1-172-17-0-9
  3428. 2020-08-25T18:34:09.1209266Z [cluster2] cluster_id: cluster1
  3429. 2020-08-25T18:34:09.1210368Z [cluster2] hostname: cluster1-worker
  3430. 2020-08-25T18:34:09.1210515Z [cluster2] nat_enabled: false
  3431. 2020-08-25T18:34:09.1210639Z [cluster2] private_ip: 172.17.0.9
  3432. 2020-08-25T18:34:09.1210763Z [cluster2] public_ip: ""
  3433. 2020-08-25T18:34:09.1210881Z [cluster2] subnets:
  3434. 2020-08-25T18:34:09.1211154Z [cluster2] - 100.91.0.0/16
  3435. 2020-08-25T18:34:09.1211413Z [cluster2] - 10.241.0.0/16
  3436. 2020-08-25T18:34:09.1211532Z [cluster2] status: connected
  3437. 2020-08-25T18:34:09.1211880Z [cluster2] statusMessage: Connected to 172.17.0.9:4500 - encryption alg=AES_GCM_16, keysize=128
  3438. 2020-08-25T18:34:09.1212325Z [cluster2] rekey-time=12897
  3439. 2020-08-25T18:34:09.1212468Z [cluster2] haStatus: active
  3440. 2020-08-25T18:34:09.1212584Z [cluster2] localEndpoint:
  3441. 2020-08-25T18:34:09.1212707Z [cluster2] backend: strongswan
  3442. 2020-08-25T18:34:09.1213042Z [cluster2] cable_name: submariner-cable-cluster2-172-17-0-8
  3443. 2020-08-25T18:34:09.1213174Z [cluster2] cluster_id: cluster2
  3444. 2020-08-25T18:34:09.1213427Z [cluster2] hostname: cluster2-worker
  3445. 2020-08-25T18:34:09.1213551Z [cluster2] nat_enabled: false
  3446. 2020-08-25T18:34:09.1213672Z [cluster2] private_ip: 172.17.0.8
  3447. 2020-08-25T18:34:09.1213790Z [cluster2] public_ip: ""
  3448. 2020-08-25T18:34:09.1213905Z [cluster2] subnets:
  3449. 2020-08-25T18:34:09.1214164Z [cluster2] - 100.92.0.0/16
  3450. 2020-08-25T18:34:09.1214419Z [cluster2] - 10.242.0.0/16
  3451. 2020-08-25T18:34:09.1214529Z [cluster2] statusFailure: ""
  3452. 2020-08-25T18:34:09.1214804Z [cluster2] version: v0.6.0-rc0-6-gd902e2d
  3453. 2020-08-25T18:34:09.1214924Z [cluster2] kind: List
  3454. 2020-08-25T18:34:09.1215044Z [cluster2] metadata:
  3455. 2020-08-25T18:34:09.1215160Z [cluster2] resourceVersion: ""
  3456. 2020-08-25T18:34:09.1215277Z [cluster2] selfLink: ""
  3457. 2020-08-25T18:34:09.1215410Z [cluster2] ===================== END Post mortem cluster2 =====================
  3458. 2020-08-25T18:34:09.1238388Z [lighthouse]$ make post-mortem
  3459. 2020-08-25T18:34:09.4663635Z Post job cleanup.
  3460. 2020-08-25T18:34:09.9282697Z [command]/usr/bin/git version
  3461. 2020-08-25T18:34:09.9361438Z git version 2.28.0
  3462. 2020-08-25T18:34:09.9457642Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand
  3463. 2020-08-25T18:34:09.9514813Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :
  3464. 2020-08-25T18:34:09.9973744Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader
  3465. 2020-08-25T18:34:10.0024203Z http.https://github.com/.extraheader
  3466. 2020-08-25T18:34:10.0035964Z [command]/usr/bin/git config --local --unset-all http.https://github.com/.extraheader
  3467. 2020-08-25T18:34:10.0083193Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :
  3468. 2020-08-25T18:34:10.0518008Z Cleaning up orphan processes
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement
Public Pastes
Advertisement
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!