API tools faq
paste
dfarrell07

subm_random_fail_20201008

dfarrell07
Oct 8th, 2020 (edited)
66
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 867.46 KB | None | 0 0
raw download clone embed print report
  1. 2020-10-08T11:26:10.7962515Z ##[section]Starting: Request a runner to run this job
  2. 2020-10-08T11:26:11.8634934Z Can't find any online and idle self-hosted runner in current repository that matches the required labels: 'ubuntu-latest'
  3. 2020-10-08T11:26:11.8635048Z Can't find any online and idle self-hosted runner in current repository's account/organization that matches the required labels: 'ubuntu-latest'
  4. 2020-10-08T11:26:11.8635299Z Found online and idle hosted runner in current repository's account/organization that matches the required labels: 'ubuntu-latest'
  5. 2020-10-08T11:26:12.0772987Z ##[section]Finishing: Request a runner to run this job
  6. 2020-10-08T11:26:20.4181806Z Current runner version: '2.273.5'
  7. 2020-10-08T11:26:20.4214165Z ##[group]Operating System
  8. 2020-10-08T11:26:20.4214977Z Ubuntu
  9. 2020-10-08T11:26:20.4215314Z 18.04.5
  10. 2020-10-08T11:26:20.4215635Z LTS
  11. 2020-10-08T11:26:20.4215981Z ##[endgroup]
  12. 2020-10-08T11:26:20.4216692Z ##[group]Virtual Environment
  13. 2020-10-08T11:26:20.4217196Z Environment: ubuntu-18.04
  14. 2020-10-08T11:26:20.4217567Z Version: 20201004.1
  15. 2020-10-08T11:26:20.4218474Z Included Software: https://github.com/actions/virtual-environments/blob/ubuntu18/20201004.1/images/linux/Ubuntu1804-README.md
  16. 2020-10-08T11:26:20.4219326Z ##[endgroup]
  17. 2020-10-08T11:26:20.4220464Z Prepare workflow directory
  18. 2020-10-08T11:26:20.4395439Z Prepare all required actions
  19. 2020-10-08T11:26:20.4405136Z Getting action download info
  20. 2020-10-08T11:26:20.8056837Z Download action repository 'actions/checkout@master'
  21. 2020-10-08T11:26:22.8332034Z ##[group]Run actions/checkout@master
  22. 2020-10-08T11:26:22.8332797Z with:
  23. 2020-10-08T11:26:22.8333435Z repository: submariner-io/lighthouse
  24. 2020-10-08T11:26:22.8334192Z token: ***
  25. 2020-10-08T11:26:22.8334510Z ssh-strict: true
  26. 2020-10-08T11:26:22.8334899Z persist-credentials: true
  27. 2020-10-08T11:26:22.8335287Z clean: true
  28. 2020-10-08T11:26:22.8335600Z fetch-depth: 1
  29. 2020-10-08T11:26:22.8335908Z lfs: false
  30. 2020-10-08T11:26:22.8336225Z submodules: false
  31. 2020-10-08T11:26:22.8336552Z ##[endgroup]
  32. 2020-10-08T11:26:23.7616396Z Syncing repository: submariner-io/lighthouse
  33. 2020-10-08T11:26:23.7617062Z ##[group]Getting Git version info
  34. 2020-10-08T11:26:23.7617717Z Working directory is '/home/runner/work/lighthouse/lighthouse'
  35. 2020-10-08T11:26:23.7618188Z [command]/usr/bin/git version
  36. 2020-10-08T11:26:23.7618479Z git version 2.28.0
  37. 2020-10-08T11:26:23.7619110Z ##[endgroup]
  38. 2020-10-08T11:26:23.7619674Z Deleting the contents of '/home/runner/work/lighthouse/lighthouse'
  39. 2020-10-08T11:26:23.7620592Z ##[group]Initializing the repository
  40. 2020-10-08T11:26:23.7621066Z [command]/usr/bin/git init /home/runner/work/lighthouse/lighthouse
  41. 2020-10-08T11:26:23.7621649Z Initialized empty Git repository in /home/runner/work/lighthouse/lighthouse/.git/
  42. 2020-10-08T11:26:23.7622490Z [command]/usr/bin/git remote add origin https://github.com/submariner-io/lighthouse
  43. 2020-10-08T11:26:23.7623000Z ##[endgroup]
  44. 2020-10-08T11:26:23.7623372Z ##[group]Disabling automatic garbage collection
  45. 2020-10-08T11:26:23.7624123Z [command]/usr/bin/git config --local gc.auto 0
  46. 2020-10-08T11:26:23.7624507Z ##[endgroup]
  47. 2020-10-08T11:26:23.7625918Z ##[group]Setting up auth
  48. 2020-10-08T11:26:23.7626620Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand
  49. 2020-10-08T11:26:23.7627767Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :
  50. 2020-10-08T11:26:23.7628916Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader
  51. 2020-10-08T11:26:23.7630550Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :
  52. 2020-10-08T11:26:23.7632017Z [command]/usr/bin/git config --local http.https://github.com/.extraheader AUTHORIZATION: basic ***
  53. 2020-10-08T11:26:23.7632690Z ##[endgroup]
  54. 2020-10-08T11:26:23.7633074Z ##[group]Fetching the repository
  55. 2020-10-08T11:26:23.7634286Z [command]/usr/bin/git -c protocol.version=2 fetch --no-tags --prune --progress --no-recurse-submodules --depth=1 origin +1e22517256c2f4cd884a6d06046225d673b3ae23:refs/remotes/pull/327/merge
  56. 2020-10-08T11:26:23.7635201Z remote: Enumerating objects: 160, done.
  57. 2020-10-08T11:26:23.7635635Z remote: Counting objects: 0% (1/160)
  58. 2020-10-08T11:26:23.7636105Z remote: Counting objects: 1% (2/160)
  59. 2020-10-08T11:26:23.7636459Z remote: Counting objects: 2% (4/160)
  60. 2020-10-08T11:26:23.7636806Z remote: Counting objects: 3% (5/160)
  61. 2020-10-08T11:26:23.7637150Z remote: Counting objects: 4% (7/160)
  62. 2020-10-08T11:26:23.7637664Z remote: Counting objects: 5% (8/160)
  63. 2020-10-08T11:26:23.7638007Z remote: Counting objects: 6% (10/160)
  64. 2020-10-08T11:26:23.7638364Z remote: Counting objects: 7% (12/160)
  65. 2020-10-08T11:26:23.7638731Z remote: Counting objects: 8% (13/160)
  66. 2020-10-08T11:26:23.7639079Z remote: Counting objects: 9% (15/160)
  67. 2020-10-08T11:26:23.7639429Z remote: Counting objects: 10% (16/160)
  68. 2020-10-08T11:26:23.7639900Z remote: Counting objects: 11% (18/160)
  69. 2020-10-08T11:26:23.7640250Z remote: Counting objects: 12% (20/160)
  70. 2020-10-08T11:26:23.7640598Z remote: Counting objects: 13% (21/160)
  71. 2020-10-08T11:26:23.7640950Z remote: Counting objects: 14% (23/160)
  72. 2020-10-08T11:26:23.7641365Z remote: Counting objects: 15% (24/160)
  73. 2020-10-08T11:26:23.7641721Z remote: Counting objects: 16% (26/160)
  74. 2020-10-08T11:26:23.7642071Z remote: Counting objects: 17% (28/160)
  75. 2020-10-08T11:26:23.7642427Z remote: Counting objects: 18% (29/160)
  76. 2020-10-08T11:26:23.7642776Z remote: Counting objects: 19% (31/160)
  77. 2020-10-08T11:26:23.7643125Z remote: Counting objects: 20% (32/160)
  78. 2020-10-08T11:26:23.7643655Z remote: Counting objects: 21% (34/160)
  79. 2020-10-08T11:26:23.7644058Z remote: Counting objects: 22% (36/160)
  80. 2020-10-08T11:26:23.7644456Z remote: Counting objects: 23% (37/160)
  81. 2020-10-08T11:26:23.7644854Z remote: Counting objects: 24% (39/160)
  82. 2020-10-08T11:26:23.7645252Z remote: Counting objects: 25% (40/160)
  83. 2020-10-08T11:26:23.7645648Z remote: Counting objects: 26% (42/160)
  84. 2020-10-08T11:26:23.7646043Z remote: Counting objects: 27% (44/160)
  85. 2020-10-08T11:26:23.7646430Z remote: Counting objects: 28% (45/160)
  86. 2020-10-08T11:26:23.7646826Z remote: Counting objects: 29% (47/160)
  87. 2020-10-08T11:26:23.7647224Z remote: Counting objects: 30% (48/160)
  88. 2020-10-08T11:26:23.7647625Z remote: Counting objects: 31% (50/160)
  89. 2020-10-08T11:26:23.7648020Z remote: Counting objects: 32% (52/160)
  90. 2020-10-08T11:26:23.7648416Z remote: Counting objects: 33% (53/160)
  91. 2020-10-08T11:26:23.7648814Z remote: Counting objects: 34% (55/160)
  92. 2020-10-08T11:26:23.7649210Z remote: Counting objects: 35% (56/160)
  93. 2020-10-08T11:26:23.7649593Z remote: Counting objects: 36% (58/160)
  94. 2020-10-08T11:26:23.7649991Z remote: Counting objects: 37% (60/160)
  95. 2020-10-08T11:26:23.7650387Z remote: Counting objects: 38% (61/160)
  96. 2020-10-08T11:26:23.7650784Z remote: Counting objects: 39% (63/160)
  97. 2020-10-08T11:26:23.7651178Z remote: Counting objects: 40% (64/160)
  98. 2020-10-08T11:26:23.7651574Z remote: Counting objects: 41% (66/160)
  99. 2020-10-08T11:26:23.7651978Z remote: Counting objects: 42% (68/160)
  100. 2020-10-08T11:26:23.7652493Z remote: Counting objects: 43% (69/160)
  101. 2020-10-08T11:26:23.7653467Z remote: Counting objects: 44% (71/160)
  102. 2020-10-08T11:26:23.7653877Z remote: Counting objects: 45% (72/160)
  103. 2020-10-08T11:26:23.7654280Z remote: Counting objects: 46% (74/160)
  104. 2020-10-08T11:26:23.7654679Z remote: Counting objects: 47% (76/160)
  105. 2020-10-08T11:26:23.7655078Z remote: Counting objects: 48% (77/160)
  106. 2020-10-08T11:26:23.7655476Z remote: Counting objects: 49% (79/160)
  107. 2020-10-08T11:26:23.7655874Z remote: Counting objects: 50% (80/160)
  108. 2020-10-08T11:26:23.7656273Z remote: Counting objects: 51% (82/160)
  109. 2020-10-08T11:26:23.7656673Z remote: Counting objects: 52% (84/160)
  110. 2020-10-08T11:26:23.7657055Z remote: Counting objects: 53% (85/160)
  111. 2020-10-08T11:26:23.7657457Z remote: Counting objects: 54% (87/160)
  112. 2020-10-08T11:26:23.7657867Z remote: Counting objects: 55% (88/160)
  113. 2020-10-08T11:26:23.7658269Z remote: Counting objects: 56% (90/160)
  114. 2020-10-08T11:26:23.7658668Z remote: Counting objects: 57% (92/160)
  115. 2020-10-08T11:26:23.7659183Z remote: Counting objects: 58% (93/160)
  116. 2020-10-08T11:26:23.7659581Z remote: Counting objects: 59% (95/160)
  117. 2020-10-08T11:26:23.7659978Z remote: Counting objects: 60% (96/160)
  118. 2020-10-08T11:26:23.7660366Z remote: Counting objects: 61% (98/160)
  119. 2020-10-08T11:26:23.7660765Z remote: Counting objects: 62% (100/160)
  120. 2020-10-08T11:26:23.7661256Z remote: Counting objects: 63% (101/160)
  121. 2020-10-08T11:26:23.7661603Z remote: Counting objects: 64% (103/160)
  122. 2020-10-08T11:26:23.7662144Z remote: Counting objects: 65% (104/160)
  123. 2020-10-08T11:26:23.7662656Z remote: Counting objects: 66% (106/160)
  124. 2020-10-08T11:26:23.7663002Z remote: Counting objects: 67% (108/160)
  125. 2020-10-08T11:26:23.7663428Z remote: Counting objects: 68% (109/160)
  126. 2020-10-08T11:26:23.7663769Z remote: Counting objects: 69% (111/160)
  127. 2020-10-08T11:26:23.7664123Z remote: Counting objects: 70% (112/160)
  128. 2020-10-08T11:26:23.7664480Z remote: Counting objects: 71% (114/160)
  129. 2020-10-08T11:26:23.7664835Z remote: Counting objects: 72% (116/160)
  130. 2020-10-08T11:26:23.7665183Z remote: Counting objects: 73% (117/160)
  131. 2020-10-08T11:26:23.7665702Z remote: Counting objects: 74% (119/160)
  132. 2020-10-08T11:26:23.7666099Z remote: Counting objects: 75% (120/160)
  133. 2020-10-08T11:26:23.7666497Z remote: Counting objects: 76% (122/160)
  134. 2020-10-08T11:26:23.7666898Z remote: Counting objects: 77% (124/160)
  135. 2020-10-08T11:26:23.7667295Z remote: Counting objects: 78% (125/160)
  136. 2020-10-08T11:26:23.7667699Z remote: Counting objects: 79% (127/160)
  137. 2020-10-08T11:26:23.7668082Z remote: Counting objects: 80% (128/160)
  138. 2020-10-08T11:26:23.7668492Z remote: Counting objects: 81% (130/160)
  139. 2020-10-08T11:26:23.7668891Z remote: Counting objects: 82% (132/160)
  140. 2020-10-08T11:26:23.7669288Z remote: Counting objects: 83% (133/160)
  141. 2020-10-08T11:26:23.7669692Z remote: Counting objects: 84% (135/160)
  142. 2020-10-08T11:26:23.7670088Z remote: Counting objects: 85% (136/160)
  143. 2020-10-08T11:26:23.7670485Z remote: Counting objects: 86% (138/160)
  144. 2020-10-08T11:26:23.7670878Z remote: Counting objects: 87% (140/160)
  145. 2020-10-08T11:26:23.7671267Z remote: Counting objects: 88% (141/160)
  146. 2020-10-08T11:26:23.7671775Z remote: Counting objects: 89% (143/160)
  147. 2020-10-08T11:26:23.7672175Z remote: Counting objects: 90% (144/160)
  148. 2020-10-08T11:26:23.7672571Z remote: Counting objects: 91% (146/160)
  149. 2020-10-08T11:26:23.7672966Z remote: Counting objects: 92% (148/160)
  150. 2020-10-08T11:26:23.7673452Z remote: Counting objects: 93% (149/160)
  151. 2020-10-08T11:26:23.7673807Z remote: Counting objects: 94% (151/160)
  152. 2020-10-08T11:26:23.7674157Z remote: Counting objects: 95% (152/160)
  153. 2020-10-08T11:26:23.7674493Z remote: Counting objects: 96% (154/160)
  154. 2020-10-08T11:26:23.7674853Z remote: Counting objects: 97% (156/160)
  155. 2020-10-08T11:26:23.7675201Z remote: Counting objects: 98% (157/160)
  156. 2020-10-08T11:26:23.7675548Z remote: Counting objects: 99% (159/160)
  157. 2020-10-08T11:26:23.7675894Z remote: Counting objects: 100% (160/160)
  158. 2020-10-08T11:26:23.7676258Z remote: Counting objects: 100% (160/160), done.
  159. 2020-10-08T11:26:23.7676639Z remote: Compressing objects: 0% (1/136)
  160. 2020-10-08T11:26:23.7677017Z remote: Compressing objects: 1% (2/136)
  161. 2020-10-08T11:26:23.7677615Z remote: Compressing objects: 2% (3/136)
  162. 2020-10-08T11:26:23.7678084Z remote: Compressing objects: 3% (5/136)
  163. 2020-10-08T11:26:23.7678555Z remote: Compressing objects: 4% (6/136)
  164. 2020-10-08T11:26:23.7679028Z remote: Compressing objects: 5% (7/136)
  165. 2020-10-08T11:26:23.7679495Z remote: Compressing objects: 6% (9/136)
  166. 2020-10-08T11:26:23.7679960Z remote: Compressing objects: 7% (10/136)
  167. 2020-10-08T11:26:23.7680504Z remote: Compressing objects: 8% (11/136)
  168. 2020-10-08T11:26:23.7680969Z remote: Compressing objects: 9% (13/136)
  169. 2020-10-08T11:26:23.7681438Z remote: Compressing objects: 10% (14/136)
  170. 2020-10-08T11:26:23.7681889Z remote: Compressing objects: 11% (15/136)
  171. 2020-10-08T11:26:23.7682353Z remote: Compressing objects: 12% (17/136)
  172. 2020-10-08T11:26:23.7682826Z remote: Compressing objects: 13% (18/136)
  173. 2020-10-08T11:26:23.7683293Z remote: Compressing objects: 14% (20/136)
  174. 2020-10-08T11:26:23.7683756Z remote: Compressing objects: 15% (21/136)
  175. 2020-10-08T11:26:23.7684220Z remote: Compressing objects: 16% (22/136)
  176. 2020-10-08T11:26:23.7684681Z remote: Compressing objects: 17% (24/136)
  177. 2020-10-08T11:26:23.7685192Z remote: Compressing objects: 18% (25/136)
  178. 2020-10-08T11:26:23.7685648Z remote: Compressing objects: 19% (26/136)
  179. 2020-10-08T11:26:23.7686113Z remote: Compressing objects: 20% (28/136)
  180. 2020-10-08T11:26:23.7686585Z remote: Compressing objects: 21% (29/136)
  181. 2020-10-08T11:26:23.7687052Z remote: Compressing objects: 22% (30/136)
  182. 2020-10-08T11:26:23.7687515Z remote: Compressing objects: 23% (32/136)
  183. 2020-10-08T11:26:23.7687980Z remote: Compressing objects: 24% (33/136)
  184. 2020-10-08T11:26:23.7688444Z remote: Compressing objects: 25% (34/136)
  185. 2020-10-08T11:26:23.7688909Z remote: Compressing objects: 26% (36/136)
  186. 2020-10-08T11:26:23.7689373Z remote: Compressing objects: 27% (37/136)
  187. 2020-10-08T11:26:23.7689823Z remote: Compressing objects: 28% (39/136)
  188. 2020-10-08T11:26:23.7690287Z remote: Compressing objects: 29% (40/136)
  189. 2020-10-08T11:26:23.7690750Z remote: Compressing objects: 30% (41/136)
  190. 2020-10-08T11:26:23.7691223Z remote: Compressing objects: 31% (43/136)
  191. 2020-10-08T11:26:23.7691688Z remote: Compressing objects: 32% (44/136)
  192. 2020-10-08T11:26:23.7692151Z remote: Compressing objects: 33% (45/136)
  193. 2020-10-08T11:26:23.7692756Z remote: Compressing objects: 34% (47/136)
  194. 2020-10-08T11:26:23.7693218Z remote: Compressing objects: 35% (48/136)
  195. 2020-10-08T11:26:23.7693668Z remote: Compressing objects: 36% (49/136)
  196. 2020-10-08T11:26:23.7694133Z remote: Compressing objects: 37% (51/136)
  197. 2020-10-08T11:26:23.7694599Z remote: Compressing objects: 38% (52/136)
  198. 2020-10-08T11:26:23.7697816Z remote: Compressing objects: 39% (54/136)
  199. 2020-10-08T11:26:23.7698313Z remote: Compressing objects: 40% (55/136)
  200. 2020-10-08T11:26:23.7698786Z remote: Compressing objects: 41% (56/136)
  201. 2020-10-08T11:26:23.7699253Z remote: Compressing objects: 42% (58/136)
  202. 2020-10-08T11:26:23.7699721Z remote: Compressing objects: 43% (59/136)
  203. 2020-10-08T11:26:23.7700216Z remote: Compressing objects: 44% (60/136)
  204. 2020-10-08T11:26:23.7700668Z remote: Compressing objects: 45% (62/136)
  205. 2020-10-08T11:26:23.7701133Z remote: Compressing objects: 46% (63/136)
  206. 2020-10-08T11:26:23.7701607Z remote: Compressing objects: 47% (64/136)
  207. 2020-10-08T11:26:23.7702071Z remote: Compressing objects: 48% (66/136)
  208. 2020-10-08T11:26:23.7702537Z remote: Compressing objects: 49% (67/136)
  209. 2020-10-08T11:26:23.7703004Z remote: Compressing objects: 50% (68/136)
  210. 2020-10-08T11:26:23.7703471Z remote: Compressing objects: 51% (70/136)
  211. 2020-10-08T11:26:23.7703935Z remote: Compressing objects: 52% (71/136)
  212. 2020-10-08T11:26:23.7704383Z remote: Compressing objects: 53% (73/136)
  213. 2020-10-08T11:26:23.7704845Z remote: Compressing objects: 54% (74/136)
  214. 2020-10-08T11:26:23.7705312Z remote: Compressing objects: 55% (75/136)
  215. 2020-10-08T11:26:23.7705776Z remote: Compressing objects: 56% (77/136)
  216. 2020-10-08T11:26:23.7706249Z remote: Compressing objects: 57% (78/136)
  217. 2020-10-08T11:26:23.7706715Z remote: Compressing objects: 58% (79/136)
  218. 2020-10-08T11:26:23.7707338Z remote: Compressing objects: 59% (81/136)
  219. 2020-10-08T11:26:23.7707806Z remote: Compressing objects: 60% (82/136)
  220. 2020-10-08T11:26:23.7708256Z remote: Compressing objects: 61% (83/136)
  221. 2020-10-08T11:26:23.7708721Z remote: Compressing objects: 62% (85/136)
  222. 2020-10-08T11:26:23.7709186Z remote: Compressing objects: 63% (86/136)
  223. 2020-10-08T11:26:23.7709651Z remote: Compressing objects: 64% (88/136)
  224. 2020-10-08T11:26:23.7710115Z remote: Compressing objects: 65% (89/136)
  225. 2020-10-08T11:26:23.7710582Z remote: Compressing objects: 66% (90/136)
  226. 2020-10-08T11:26:23.7711049Z remote: Compressing objects: 67% (92/136)
  227. 2020-10-08T11:26:23.7711516Z remote: Compressing objects: 68% (93/136)
  228. 2020-10-08T11:26:23.7712156Z remote: Compressing objects: 69% (94/136)
  229. 2020-10-08T11:26:23.7712580Z remote: Compressing objects: 70% (96/136)
  230. 2020-10-08T11:26:23.7713010Z remote: Compressing objects: 71% (97/136)
  231. 2020-10-08T11:26:23.7713888Z remote: Compressing objects: 72% (98/136)
  232. 2020-10-08T11:26:23.7714357Z remote: Compressing objects: 73% (100/136)
  233. 2020-10-08T11:26:23.7714828Z remote: Compressing objects: 74% (101/136)
  234. 2020-10-08T11:26:23.7715298Z remote: Compressing objects: 75% (102/136)
  235. 2020-10-08T11:26:23.7715796Z remote: Compressing objects: 76% (104/136)
  236. 2020-10-08T11:26:23.7716268Z remote: Compressing objects: 77% (105/136)
  237. 2020-10-08T11:26:23.7716719Z remote: Compressing objects: 78% (107/136)
  238. 2020-10-08T11:26:23.7717185Z remote: Compressing objects: 79% (108/136)
  239. 2020-10-08T11:26:23.7717652Z remote: Compressing objects: 80% (109/136)
  240. 2020-10-08T11:26:23.7718118Z remote: Compressing objects: 81% (111/136)
  241. 2020-10-08T11:26:23.7718593Z remote: Compressing objects: 82% (112/136)
  242. 2020-10-08T11:26:23.7719058Z remote: Compressing objects: 83% (113/136)
  243. 2020-10-08T11:26:23.7719523Z remote: Compressing objects: 84% (115/136)
  244. 2020-10-08T11:26:23.7719996Z remote: Compressing objects: 85% (116/136)
  245. 2020-10-08T11:26:23.7720581Z remote: Compressing objects: 86% (117/136)
  246. 2020-10-08T11:26:23.7721002Z remote: Compressing objects: 87% (119/136)
  247. 2020-10-08T11:26:23.7721434Z remote: Compressing objects: 88% (120/136)
  248. 2020-10-08T11:26:23.7721867Z remote: Compressing objects: 89% (122/136)
  249. 2020-10-08T11:26:23.7722302Z remote: Compressing objects: 90% (123/136)
  250. 2020-10-08T11:26:23.7722735Z remote: Compressing objects: 91% (124/136)
  251. 2020-10-08T11:26:23.7723170Z remote: Compressing objects: 92% (126/136)
  252. 2020-10-08T11:26:23.7723605Z remote: Compressing objects: 93% (127/136)
  253. 2020-10-08T11:26:23.7724041Z remote: Compressing objects: 94% (128/136)
  254. 2020-10-08T11:26:23.7724467Z remote: Compressing objects: 95% (130/136)
  255. 2020-10-08T11:26:23.7724901Z remote: Compressing objects: 96% (131/136)
  256. 2020-10-08T11:26:23.7725339Z remote: Compressing objects: 97% (132/136)
  257. 2020-10-08T11:26:23.7725835Z remote: Compressing objects: 98% (134/136)
  258. 2020-10-08T11:26:23.7726283Z remote: Compressing objects: 99% (135/136)
  259. 2020-10-08T11:26:23.7726727Z remote: Compressing objects: 100% (136/136)
  260. 2020-10-08T11:26:23.7727186Z remote: Compressing objects: 100% (136/136), done.
  261. 2020-10-08T11:26:23.7727608Z Receiving objects: 0% (1/160)
  262. 2020-10-08T11:26:23.7727974Z Receiving objects: 1% (2/160)
  263. 2020-10-08T11:26:23.7728337Z Receiving objects: 2% (4/160)
  264. 2020-10-08T11:26:23.7728701Z Receiving objects: 3% (5/160)
  265. 2020-10-08T11:26:23.7729065Z Receiving objects: 4% (7/160)
  266. 2020-10-08T11:26:23.7729429Z Receiving objects: 5% (8/160)
  267. 2020-10-08T11:26:23.7729778Z Receiving objects: 6% (10/160)
  268. 2020-10-08T11:26:23.7730153Z Receiving objects: 7% (12/160)
  269. 2020-10-08T11:26:23.7730517Z Receiving objects: 8% (13/160)
  270. 2020-10-08T11:26:23.7730880Z Receiving objects: 9% (15/160)
  271. 2020-10-08T11:26:23.7731331Z Receiving objects: 10% (16/160)
  272. 2020-10-08T11:26:23.7731695Z Receiving objects: 11% (18/160)
  273. 2020-10-08T11:26:23.7732059Z Receiving objects: 12% (20/160)
  274. 2020-10-08T11:26:23.7732610Z Receiving objects: 13% (21/160)
  275. 2020-10-08T11:26:23.7732963Z Receiving objects: 14% (23/160)
  276. 2020-10-08T11:26:23.7733327Z Receiving objects: 15% (24/160)
  277. 2020-10-08T11:26:23.7733693Z Receiving objects: 16% (26/160)
  278. 2020-10-08T11:26:23.7734057Z Receiving objects: 17% (28/160)
  279. 2020-10-08T11:26:23.7734419Z Receiving objects: 18% (29/160)
  280. 2020-10-08T11:26:23.7734783Z Receiving objects: 19% (31/160)
  281. 2020-10-08T11:26:23.7735146Z Receiving objects: 20% (32/160)
  282. 2020-10-08T11:26:23.7735497Z Receiving objects: 21% (34/160)
  283. 2020-10-08T11:26:23.7735867Z Receiving objects: 22% (36/160)
  284. 2020-10-08T11:26:23.7736233Z Receiving objects: 23% (37/160)
  285. 2020-10-08T11:26:23.7736659Z Receiving objects: 24% (39/160)
  286. 2020-10-08T11:26:23.7737034Z Receiving objects: 25% (40/160)
  287. 2020-10-08T11:26:23.7737393Z Receiving objects: 26% (42/160)
  288. 2020-10-08T11:26:23.7737845Z Receiving objects: 27% (44/160)
  289. 2020-10-08T11:26:23.7738160Z Receiving objects: 28% (45/160)
  290. 2020-10-08T11:26:23.7738468Z Receiving objects: 29% (47/160)
  291. 2020-10-08T11:26:23.7738786Z Receiving objects: 30% (48/160)
  292. 2020-10-08T11:26:23.7739101Z Receiving objects: 31% (50/160)
  293. 2020-10-08T11:26:23.7739418Z Receiving objects: 32% (52/160)
  294. 2020-10-08T11:26:23.7739734Z Receiving objects: 33% (53/160)
  295. 2020-10-08T11:26:23.7740050Z Receiving objects: 34% (55/160)
  296. 2020-10-08T11:26:23.7740365Z Receiving objects: 35% (56/160)
  297. 2020-10-08T11:26:23.7740667Z Receiving objects: 36% (58/160)
  298. 2020-10-08T11:26:23.7740984Z Receiving objects: 37% (60/160)
  299. 2020-10-08T11:26:23.7741303Z Receiving objects: 38% (61/160)
  300. 2020-10-08T11:26:23.7741619Z Receiving objects: 39% (63/160)
  301. 2020-10-08T11:26:23.7741938Z Receiving objects: 40% (64/160)
  302. 2020-10-08T11:26:23.7742258Z Receiving objects: 41% (66/160)
  303. 2020-10-08T11:26:23.7742577Z Receiving objects: 42% (68/160)
  304. 2020-10-08T11:26:23.7742903Z Receiving objects: 43% (69/160)
  305. 2020-10-08T11:26:23.7743205Z Receiving objects: 44% (71/160)
  306. 2020-10-08T11:26:23.7743528Z Receiving objects: 45% (72/160)
  307. 2020-10-08T11:26:23.7743845Z Receiving objects: 46% (74/160)
  308. 2020-10-08T11:26:23.7744160Z Receiving objects: 47% (76/160)
  309. 2020-10-08T11:26:23.7744474Z Receiving objects: 48% (77/160)
  310. 2020-10-08T11:26:23.7744791Z Receiving objects: 49% (79/160)
  311. 2020-10-08T11:26:23.7745111Z Receiving objects: 50% (80/160)
  312. 2020-10-08T11:26:23.7745413Z Receiving objects: 51% (82/160)
  313. 2020-10-08T11:26:23.7745736Z Receiving objects: 52% (84/160)
  314. 2020-10-08T11:26:23.7746057Z Receiving objects: 53% (85/160)
  315. 2020-10-08T11:26:23.7746374Z Receiving objects: 54% (87/160)
  316. 2020-10-08T11:26:23.7746689Z Receiving objects: 55% (88/160)
  317. 2020-10-08T11:26:23.7747010Z Receiving objects: 56% (90/160)
  318. 2020-10-08T11:26:23.7747332Z Receiving objects: 57% (92/160)
  319. 2020-10-08T11:26:23.7747650Z Receiving objects: 58% (93/160)
  320. 2020-10-08T11:26:23.7747953Z Receiving objects: 59% (95/160)
  321. 2020-10-08T11:26:23.7748276Z Receiving objects: 60% (96/160)
  322. 2020-10-08T11:26:23.7748594Z Receiving objects: 61% (98/160)
  323. 2020-10-08T11:26:23.7748912Z Receiving objects: 62% (100/160)
  324. 2020-10-08T11:26:23.7749233Z Receiving objects: 63% (101/160)
  325. 2020-10-08T11:26:23.7749551Z Receiving objects: 64% (103/160)
  326. 2020-10-08T11:26:23.7749867Z Receiving objects: 65% (104/160)
  327. 2020-10-08T11:26:23.7750190Z Receiving objects: 66% (106/160)
  328. 2020-10-08T11:26:23.7750496Z Receiving objects: 67% (108/160)
  329. 2020-10-08T11:26:23.7750812Z Receiving objects: 68% (109/160)
  330. 2020-10-08T11:26:23.7751131Z Receiving objects: 69% (111/160)
  331. 2020-10-08T11:26:23.7751449Z Receiving objects: 70% (112/160)
  332. 2020-10-08T11:26:23.7751766Z Receiving objects: 71% (114/160)
  333. 2020-10-08T11:26:23.7752083Z Receiving objects: 72% (116/160)
  334. 2020-10-08T11:26:23.7752404Z Receiving objects: 73% (117/160)
  335. 2020-10-08T11:26:23.7752708Z Receiving objects: 74% (119/160)
  336. 2020-10-08T11:26:23.7753029Z Receiving objects: 75% (120/160)
  337. 2020-10-08T11:26:23.7753407Z Receiving objects: 76% (122/160)
  338. 2020-10-08T11:26:23.7753728Z Receiving objects: 77% (124/160)
  339. 2020-10-08T11:26:23.7754255Z Receiving objects: 78% (125/160)
  340. 2020-10-08T11:26:23.7754619Z Receiving objects: 79% (127/160)
  341. 2020-10-08T11:26:23.7754987Z Receiving objects: 80% (128/160)
  342. 2020-10-08T11:26:23.7755352Z Receiving objects: 81% (130/160)
  343. 2020-10-08T11:26:23.7755706Z Receiving objects: 82% (132/160)
  344. 2020-10-08T11:26:23.7756073Z Receiving objects: 83% (133/160)
  345. 2020-10-08T11:26:23.7756436Z Receiving objects: 84% (135/160)
  346. 2020-10-08T11:26:23.7756801Z Receiving objects: 85% (136/160)
  347. 2020-10-08T11:26:23.7757165Z Receiving objects: 86% (138/160)
  348. 2020-10-08T11:26:23.7757595Z Receiving objects: 87% (140/160)
  349. 2020-10-08T11:26:23.7757963Z Receiving objects: 88% (141/160)
  350. 2020-10-08T11:26:23.7758571Z Receiving objects: 89% (143/160)
  351. 2020-10-08T11:26:23.7758957Z Receiving objects: 90% (144/160)
  352. 2020-10-08T11:26:23.7759350Z Receiving objects: 91% (146/160)
  353. 2020-10-08T11:26:23.7759750Z Receiving objects: 92% (148/160)
  354. 2020-10-08T11:26:23.7760141Z Receiving objects: 93% (149/160)
  355. 2020-10-08T11:26:23.7760534Z Receiving objects: 94% (151/160)
  356. 2020-10-08T11:26:23.7760928Z Receiving objects: 95% (152/160)
  357. 2020-10-08T11:26:23.7761319Z Receiving objects: 96% (154/160)
  358. 2020-10-08T11:26:23.7761699Z Receiving objects: 97% (156/160)
  359. 2020-10-08T11:26:23.7762093Z Receiving objects: 98% (157/160)
  360. 2020-10-08T11:26:23.7762520Z Receiving objects: 99% (159/160)
  361. 2020-10-08T11:26:23.7763442Z remote: Total 160 (delta 30), reused 82 (delta 13), pack-reused 0
  362. 2020-10-08T11:26:23.7763951Z Receiving objects: 100% (160/160)
  363. 2020-10-08T11:26:23.7764425Z Receiving objects: 100% (160/160), 122.18 KiB | 12.22 MiB/s, done.
  364. 2020-10-08T11:26:23.7764906Z Resolving deltas: 0% (0/30)
  365. 2020-10-08T11:26:23.7765329Z Resolving deltas: 6% (2/30)
  366. 2020-10-08T11:26:23.7765731Z Resolving deltas: 10% (3/30)
  367. 2020-10-08T11:26:23.7766136Z Resolving deltas: 13% (4/30)
  368. 2020-10-08T11:26:23.7766529Z Resolving deltas: 16% (5/30)
  369. 2020-10-08T11:26:23.7766916Z Resolving deltas: 36% (11/30)
  370. 2020-10-08T11:26:23.7767295Z Resolving deltas: 46% (14/30)
  371. 2020-10-08T11:26:23.7767665Z Resolving deltas: 56% (17/30)
  372. 2020-10-08T11:26:23.7768052Z Resolving deltas: 60% (18/30)
  373. 2020-10-08T11:26:23.7768457Z Resolving deltas: 70% (21/30)
  374. 2020-10-08T11:26:23.7768839Z Resolving deltas: 80% (24/30)
  375. 2020-10-08T11:26:23.7769338Z Resolving deltas: 83% (25/30)
  376. 2020-10-08T11:26:23.7769695Z Resolving deltas: 90% (27/30)
  377. 2020-10-08T11:26:23.7770051Z Resolving deltas: 93% (28/30)
  378. 2020-10-08T11:26:23.7770399Z Resolving deltas: 96% (29/30)
  379. 2020-10-08T11:26:23.7770756Z Resolving deltas: 100% (30/30)
  380. 2020-10-08T11:26:23.7771127Z Resolving deltas: 100% (30/30), done.
  381. 2020-10-08T11:26:23.7771817Z From https://github.com/submariner-io/lighthouse
  382. 2020-10-08T11:26:23.7772977Z * [new ref] 1e22517256c2f4cd884a6d06046225d673b3ae23 -> pull/327/merge
  383. 2020-10-08T11:26:23.7773547Z ##[endgroup]
  384. 2020-10-08T11:26:23.7773952Z ##[group]Determining the checkout info
  385. 2020-10-08T11:26:23.7775161Z ##[endgroup]
  386. 2020-10-08T11:26:23.7775515Z ##[group]Checking out the ref
  387. 2020-10-08T11:26:23.7776232Z [command]/usr/bin/git checkout --progress --force refs/remotes/pull/327/merge
  388. 2020-10-08T11:26:23.7786194Z Note: switching to 'refs/remotes/pull/327/merge'.
  389. 2020-10-08T11:26:23.7786524Z
  390. 2020-10-08T11:26:23.7787240Z You are in 'detached HEAD' state. You can look around, make experimental
  391. 2020-10-08T11:26:23.7788014Z changes and commit them, and you can discard any commits you make in this
  392. 2020-10-08T11:26:23.7788678Z state without impacting any branches by switching back to a branch.
  393. 2020-10-08T11:26:23.7789056Z
  394. 2020-10-08T11:26:23.7789530Z If you want to create a new branch to retain commits you create, you may
  395. 2020-10-08T11:26:23.7790354Z do so (now or later) by using -c with the switch command. Example:
  396. 2020-10-08T11:26:23.7790710Z
  397. 2020-10-08T11:26:23.7791226Z git switch -c <new-branch-name>
  398. 2020-10-08T11:26:23.7791531Z
  399. 2020-10-08T11:26:23.7791879Z Or undo this operation with:
  400. 2020-10-08T11:26:23.7792291Z
  401. 2020-10-08T11:26:23.7792744Z git switch -
  402. 2020-10-08T11:26:23.7792960Z
  403. 2020-10-08T11:26:23.7793504Z Turn off this advice by setting config variable advice.detachedHead to false
  404. 2020-10-08T11:26:23.7793965Z
  405. 2020-10-08T11:26:23.7794710Z HEAD is now at 1e22517 Merge c45cd216331ff3b06efa784e08dadefa05a86337 into 07ce0903433101b05a9782b65cceff3644780f5b
  406. 2020-10-08T11:26:23.7795822Z ##[endgroup]
  407. 2020-10-08T11:26:23.7796385Z [command]/usr/bin/git log -1 --format='%H'
  408. 2020-10-08T11:26:23.7797074Z '1e22517256c2f4cd884a6d06046225d673b3ae23'
  409. 2020-10-08T11:26:23.7858928Z ##[group]Run sudo swapoff -a
  410. 2020-10-08T11:26:23.7859419Z sudo swapoff -a
  411. 2020-10-08T11:26:23.7859791Z sudo rm -f /swapfile
  412. 2020-10-08T11:26:23.7860116Z df -h
  413. 2020-10-08T11:26:23.7860416Z free -h
  414. 2020-10-08T11:26:23.7904522Z shell: /bin/bash -e {0}
  415. 2020-10-08T11:26:23.7904872Z ##[endgroup]
  416. 2020-10-08T11:26:23.8616624Z Filesystem Size Used Avail Use% Mounted on
  417. 2020-10-08T11:26:23.8618281Z udev 3.4G 0 3.4G 0% /dev
  418. 2020-10-08T11:26:23.8618976Z tmpfs 696M 680K 695M 1% /run
  419. 2020-10-08T11:26:23.8619606Z /dev/sdb1 84G 62G 22G 75% /
  420. 2020-10-08T11:26:23.8620502Z tmpfs 3.4G 8.0K 3.4G 1% /dev/shm
  421. 2020-10-08T11:26:23.8621152Z tmpfs 5.0M 0 5.0M 0% /run/lock
  422. 2020-10-08T11:26:23.8621811Z tmpfs 3.4G 0 3.4G 0% /sys/fs/cgroup
  423. 2020-10-08T11:26:23.8622491Z /dev/sdb15 105M 3.6M 101M 4% /boot/efi
  424. 2020-10-08T11:26:23.8623182Z /dev/sda1 14G 4.1G 9.0G 32% /mnt
  425. 2020-10-08T11:26:23.8867173Z total used free shared buff/cache available
  426. 2020-10-08T11:26:23.8868349Z Mem: 6.8G 430M 5.0G 25M 1.4G 6.1G
  427. 2020-10-08T11:26:23.8869031Z Swap: 0B 0B 0B
  428. 2020-10-08T11:26:23.8942763Z ##[group]Run make e2e using=" helm"
  429. 2020-10-08T11:26:23.8943443Z make e2e using=" helm"
  430. 2020-10-08T11:26:23.8992107Z shell: /bin/bash -e {0}
  431. 2020-10-08T11:26:23.8992482Z ##[endgroup]
  432. 2020-10-08T11:26:23.9105316Z Downloading dapper
  433. 2020-10-08T11:26:24.8896958Z .dapper.tmp version v0.5.3
  434. 2020-10-08T11:26:24.8953631Z ./.dapper -m bind make e2e -- using=helm
  435. 2020-10-08T11:26:25.8261608Z Sending build context to Docker daemon 6.198MB
  436. 2020-10-08T11:26:25.8307255Z
  437. 2020-10-08T11:26:25.8359546Z Step 1/6 : FROM quay.io/submariner/shipyard-dapper-base:0.7.1
  438. 2020-10-08T11:26:25.9924658Z 0.7.1: Pulling from submariner/shipyard-dapper-base
  439. 2020-10-08T11:26:25.9938940Z bbca93d6c76b: Pulling fs layer
  440. 2020-10-08T11:26:25.9944632Z 1c30167c095e: Pulling fs layer
  441. 2020-10-08T11:26:25.9950245Z a274ee7d3ac9: Pulling fs layer
  442. 2020-10-08T11:26:25.9956688Z 30d9f3b2fbe6: Pulling fs layer
  443. 2020-10-08T11:26:26.0051360Z 96c70f9b1ce0: Pulling fs layer
  444. 2020-10-08T11:26:26.7570206Z a9a281312a19: Pulling fs layer
  445. 2020-10-08T11:26:26.7576292Z 48cc7c7bfa15: Pulling fs layer
  446. 2020-10-08T11:26:26.7582707Z 30d9f3b2fbe6: Waiting
  447. 2020-10-08T11:26:26.7765272Z a9a281312a19: Waiting
  448. 2020-10-08T11:26:26.7765749Z 48cc7c7bfa15: Waiting
  449. 2020-10-08T11:26:26.7766084Z 96c70f9b1ce0: Waiting
  450. 2020-10-08T11:26:26.7766462Z a274ee7d3ac9: Verifying Checksum
  451. 2020-10-08T11:26:26.7766880Z a274ee7d3ac9: Download complete
  452. 2020-10-08T11:26:26.7767427Z bbca93d6c76b: Verifying Checksum
  453. 2020-10-08T11:26:26.7768685Z bbca93d6c76b: Download complete
  454. 2020-10-08T11:26:26.7769101Z 30d9f3b2fbe6: Verifying Checksum
  455. 2020-10-08T11:26:26.7769512Z 30d9f3b2fbe6: Download complete
  456. 2020-10-08T11:26:26.7769915Z a9a281312a19: Verifying Checksum
  457. 2020-10-08T11:26:26.7784046Z a9a281312a19: Download complete
  458. 2020-10-08T11:26:26.7785126Z 96c70f9b1ce0: Verifying Checksum
  459. 2020-10-08T11:26:26.7786170Z 96c70f9b1ce0: Download complete
  460. 2020-10-08T11:26:26.9181655Z 48cc7c7bfa15: Verifying Checksum
  461. 2020-10-08T11:26:26.9182429Z 48cc7c7bfa15: Download complete
  462. 2020-10-08T11:26:28.3015895Z 1c30167c095e: Verifying Checksum
  463. 2020-10-08T11:26:28.3030221Z 1c30167c095e: Download complete
  464. 2020-10-08T11:26:30.6655728Z bbca93d6c76b: Pull complete
  465. 2020-10-08T11:26:36.9804715Z 1c30167c095e: Pull complete
  466. 2020-10-08T11:26:37.6924256Z a274ee7d3ac9: Pull complete
  467. 2020-10-08T11:26:37.7544658Z 30d9f3b2fbe6: Pull complete
  468. 2020-10-08T11:26:37.8235526Z 96c70f9b1ce0: Pull complete
  469. 2020-10-08T11:26:37.8813186Z a9a281312a19: Pull complete
  470. 2020-10-08T11:26:37.9461583Z 48cc7c7bfa15: Pull complete
  471. 2020-10-08T11:26:37.9517957Z Digest: sha256:086697dc5be6c92cac60d506e89a65fbc341d425e59ccea4f12e1e8b71f601b9
  472. 2020-10-08T11:26:37.9535280Z Status: Downloaded newer image for quay.io/submariner/shipyard-dapper-base:0.7.1
  473. 2020-10-08T11:26:37.9545350Z ---> c50512fb3462
  474. 2020-10-08T11:26:37.9546696Z Step 2/6 : ENV DAPPER_ENV="REPO TAG QUAY_USERNAME QUAY_PASSWORD GITHUB_SHA BUILD_ARGS CLUSTERS_ARGS DEPLOY_ARGS RELEASE_ARGS" DAPPER_SOURCE=/go/src/github.com/submariner-io/lighthouse DAPPER_DOCKER_SOCKET=true
  475. 2020-10-08T11:26:46.1888268Z ---> Running in e31fa2c86cd5
  476. 2020-10-08T11:26:47.0388829Z Removing intermediate container e31fa2c86cd5
  477. 2020-10-08T11:26:47.0419900Z ---> c747a35a2763
  478. 2020-10-08T11:26:47.0420664Z Step 3/6 : ENV DAPPER_OUTPUT=${DAPPER_SOURCE}/output
  479. 2020-10-08T11:26:47.0694178Z ---> Running in f04ec293e7d0
  480. 2020-10-08T11:26:48.0406858Z Removing intermediate container f04ec293e7d0
  481. 2020-10-08T11:26:48.0407923Z ---> e1395f25bc2c
  482. 2020-10-08T11:26:48.0408272Z Step 4/6 : WORKDIR ${DAPPER_SOURCE}
  483. 2020-10-08T11:26:48.0709428Z ---> Running in 0e84c30d2323
  484. 2020-10-08T11:26:49.0396314Z Removing intermediate container 0e84c30d2323
  485. 2020-10-08T11:26:49.0397834Z ---> b92d15285148
  486. 2020-10-08T11:26:49.0398554Z Step 5/6 : ENTRYPOINT ["/opt/shipyard/scripts/entry"]
  487. 2020-10-08T11:26:49.0680947Z ---> Running in fdd3ce03584e
  488. 2020-10-08T11:26:50.0473952Z Removing intermediate container fdd3ce03584e
  489. 2020-10-08T11:26:50.0475510Z ---> e388b749e663
  490. 2020-10-08T11:26:50.0476241Z Step 6/6 : CMD ["sh"]
  491. 2020-10-08T11:26:50.0734860Z ---> Running in e0ff9dac6caa
  492. 2020-10-08T11:26:51.0392679Z Removing intermediate container e0ff9dac6caa
  493. 2020-10-08T11:26:51.0393789Z ---> 5cec2e11bcff
  494. 2020-10-08T11:26:51.0474076Z Successfully built 5cec2e11bcff
  495. 2020-10-08T11:26:51.0611952Z Successfully tagged lighthouse:HEAD
  496. 2020-10-08T11:26:51.5210305Z [lighthouse]$ trap chown -R 1001:116 . exit
  497. 2020-10-08T11:26:51.5211003Z [lighthouse]$ mkdir -p bin dist output
  498. 2020-10-08T11:26:51.5298206Z [lighthouse]$ make e2e -- using=helm
  499. 2020-10-08T11:26:51.5705322Z fatal: No names found, cannot describe anything.
  500. 2020-10-08T11:26:51.5804156Z Makefile:41: warning: overriding recipe for target 'deploy'
  501. 2020-10-08T11:26:51.5804697Z /opt/shipyard/scripts/e2e.sh cluster1 cluster2
  502. 2020-10-08T11:26:51.5805646Z /opt/shipyard/Makefile.inc:36: warning: ignoring old recipe for target 'deploy'
  503. 2020-10-08T11:26:51.5836047Z Downloading shflags 1.0.3
  504. 2020-10-08T11:26:51.5913428Z % Total % Received % Xferd Average Speed Time Time Time Current
  505. 2020-10-08T11:26:51.5913930Z Dload Upload Total Spent Left Speed
  506. 2020-10-08T11:26:51.5914241Z
  507. 2020-10-08T11:26:51.6295066Z 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
  508. 2020-10-08T11:26:51.6298701Z 100 31091 100 31091 0 0 759k 0 --:--:-- --:--:-- --:--:-- 759k
  509. 2020-10-08T11:26:51.6726565Z [lighthouse]$ source /opt/shipyard/scripts/lib/utils
  510. 2020-10-08T11:26:51.6735184Z [lighthouse]$ . /opt/shipyard/scripts/lib/source_only
  511. 2020-10-08T11:26:51.6745923Z [lighthouse]$ script_name=utils
  512. 2020-10-08T11:26:51.6754123Z [lighthouse]$ exec_name=e2e.sh
  513. 2020-10-08T11:26:51.6774055Z [lighthouse]$ source /opt/shipyard/scripts/lib/cluster_settings
  514. 2020-10-08T11:26:51.6784199Z [lighthouse]$ . /opt/shipyard/scripts/lib/source_only
  515. 2020-10-08T11:26:51.6797945Z [lighthouse]$ script_name=cluster_settings
  516. 2020-10-08T11:26:51.6806958Z [lighthouse]$ exec_name=e2e.sh
  517. 2020-10-08T11:26:51.6827018Z [lighthouse]$ broker=cluster1
  518. 2020-10-08T11:26:51.6837492Z [lighthouse]$ declare -gA cluster_nodes
  519. 2020-10-08T11:26:51.6847122Z [lighthouse]$ cluster_nodes[cluster1]=control-plane worker
  520. 2020-10-08T11:26:51.6859596Z [lighthouse]$ cluster_nodes[cluster2]=control-plane worker
  521. 2020-10-08T11:26:51.6868412Z [lighthouse]$ cluster_nodes[cluster3]=control-plane worker worker
  522. 2020-10-08T11:26:51.6878775Z [lighthouse]$ declare -gA cluster_subm
  523. 2020-10-08T11:26:51.6887742Z [lighthouse]$ cluster_subm[cluster1]=true
  524. 2020-10-08T11:26:51.6898600Z [lighthouse]$ cluster_subm[cluster2]=true
  525. 2020-10-08T11:26:51.6907620Z [lighthouse]$ cluster_subm[cluster3]=true
  526. 2020-10-08T11:26:51.6922024Z [lighthouse]$ declare -gA cluster_cni
  527. 2020-10-08T11:26:51.6931498Z [lighthouse]$ cluster_cni[cluster2]=weave
  528. 2020-10-08T11:26:51.6942001Z [lighthouse]$ cluster_cni[cluster3]=weave
  529. 2020-10-08T11:26:51.6953049Z [lighthouse]$ declare_kubeconfig
  530. 2020-10-08T11:26:51.6963000Z [lighthouse]$ declare_kubeconfig
  531. 2020-10-08T11:26:51.6972089Z [lighthouse]$ source /opt/shipyard/scripts/lib/kubecfg
  532. 2020-10-08T11:26:51.6988119Z [lighthouse]$ export KUBECONFIG
  533. 2020-10-08T11:26:51.7016172Z [lighthouse]$ KUBECONFIG=
  534. 2020-10-08T11:26:51.7029535Z [lighthouse]$ find /go/src/github.com/submariner-io/lighthouse/output/kubeconfigs -type f -printf %p:
  535. 2020-10-08T11:26:51.7042599Z find: '/go/src/github.com/submariner-io/lighthouse/output/kubeconfigs': No such file or directory
  536. 2020-10-08T11:26:51.7051510Z [lighthouse]$ :
  537. 2020-10-08T11:26:51.7065123Z [lighthouse]$ deploy_env_once
  538. 2020-10-08T11:26:51.7076695Z [lighthouse]$ deploy_env_once
  539. 2020-10-08T11:26:51.7085165Z 
  540. 2020-10-08T11:26:52.3915800Z [lighthouse]$ make deploy
  541. 2020-10-08T11:26:52.3940012Z make[1]: Entering directory '/go/src/github.com/submariner-io/lighthouse'
  542. 2020-10-08T11:26:52.4378394Z fatal: No names found, cannot describe anything.
  543. 2020-10-08T11:26:52.4468371Z Makefile:41: warning: overriding recipe for target 'deploy'
  544. 2020-10-08T11:26:52.4469300Z /opt/shipyard/Makefile.inc:36: warning: ignoring old recipe for target 'deploy'
  545. 2020-10-08T11:26:52.4518079Z find: 'bin/lighthouse-agent': No such file or directory
  546. 2020-10-08T11:26:52.7689420Z go mod download
  547. 2020-10-08T11:27:15.6715793Z go mod vendor
  548. 2020-10-08T11:27:16.3602673Z /opt/shipyard/scripts/compile.sh bin/lighthouse-agent pkg/agent/main.go
  549. 2020-10-08T11:27:16.4055374Z [lighthouse]$ mkdir -p bin
  550. 2020-10-08T11:27:16.4072864Z Building 'bin/lighthouse-agent' (ldflags: '')
  551. 2020-10-08T11:27:16.4083975Z [lighthouse]$ ldflags=-s -w 
  552. 2020-10-08T11:27:16.4095374Z [lighthouse]$ CGO_ENABLED=0 go build -ldflags -s -w -o bin/lighthouse-agent pkg/agent/main.go
  553. 2020-10-08T11:28:01.1720407Z [lighthouse]$ upx bin/lighthouse-agent
  554. 2020-10-08T11:28:05.6298686Z Ultimate Packer for eXecutables
  555. 2020-10-08T11:28:05.6301225Z Copyright (C) 1996 - 2020
  556. 2020-10-08T11:28:05.6302524Z UPX 3.96 Markus Oberhumer, Laszlo Molnar & John Reiser Jan 23rd 2020
  557. 2020-10-08T11:28:05.6303501Z
  558. 2020-10-08T11:28:05.6304436Z File size Ratio Format Name
  559. 2020-10-08T11:28:05.6305997Z -------------------- ------ ----------- -----------
  560. 2020-10-08T11:28:05.6312326Z 28631040 -> 8562640 29.91% linux/amd64 lighthouse-agent
  561. 2020-10-08T11:28:05.6312963Z
  562. 2020-10-08T11:28:05.6313385Z Packed 1 file.
  563. 2020-10-08T11:28:05.6315139Z /opt/shipyard/scripts/build_image.sh -i lighthouse-agent -f package/Dockerfile.lighthouse-agent
  564. 2020-10-08T11:28:05.6748313Z fatal: No names found, cannot describe anything.
  565. 2020-10-08T11:28:05.7551359Z [lighthouse]$ set -e
  566. 2020-10-08T11:28:05.7553453Z [lighthouse]$ local_image=quay.io/submariner/lighthouse-agent:dev
  567. 2020-10-08T11:28:05.7605175Z [lighthouse]$ cache_image=quay.io/submariner/lighthouse-agent:devel
  568. 2020-10-08T11:28:05.7606108Z [lighthouse]$ cache_flag=
  569. 2020-10-08T11:28:05.7619185Z [lighthouse]$ cache_flag=--cache-from quay.io/submariner/lighthouse-agent:devel
  570. 2020-10-08T11:28:05.7620426Z [lighthouse]$ docker image ls -q quay.io/submariner/lighthouse-agent:devel
  571. 2020-10-08T11:28:06.0859420Z [lighthouse]$ docker pull quay.io/submariner/lighthouse-agent:devel
  572. 2020-10-08T11:28:06.5049161Z devel: Pulling from submariner/lighthouse-agent
  573. 2020-10-08T11:28:06.5050286Z 0fd3b5213a9b: Pulling fs layer
  574. 2020-10-08T11:28:06.5050952Z aebb8c556853: Pulling fs layer
  575. 2020-10-08T11:28:06.5051803Z 8abcb662399b: Pulling fs layer
  576. 2020-10-08T11:28:06.5052726Z 4b5265f743fe: Pulling fs layer
  577. 2020-10-08T11:28:06.5053408Z 4b5265f743fe: Waiting
  578. 2020-10-08T11:28:06.5492939Z 8abcb662399b: Verifying Checksum
  579. 2020-10-08T11:28:06.5494153Z 8abcb662399b: Download complete
  580. 2020-10-08T11:28:06.5604117Z aebb8c556853: Verifying Checksum
  581. 2020-10-08T11:28:06.5605217Z aebb8c556853: Download complete
  582. 2020-10-08T11:28:06.6604742Z 4b5265f743fe: Verifying Checksum
  583. 2020-10-08T11:28:06.6605398Z 4b5265f743fe: Download complete
  584. 2020-10-08T11:28:06.8944106Z 0fd3b5213a9b: Verifying Checksum
  585. 2020-10-08T11:28:06.8944581Z 0fd3b5213a9b: Download complete
  586. 2020-10-08T11:28:08.7313448Z 0fd3b5213a9b: Pull complete
  587. 2020-10-08T11:28:08.7896396Z aebb8c556853: Pull complete
  588. 2020-10-08T11:28:08.8475979Z 8abcb662399b: Pull complete
  589. 2020-10-08T11:28:08.9990743Z 4b5265f743fe: Pull complete
  590. 2020-10-08T11:28:09.0011390Z Digest: sha256:1dbecc4760d06cf80fe14eba64d7bd52c284ebd99130ec801f03d4a586245933
  591. 2020-10-08T11:28:09.0046519Z Status: Downloaded newer image for quay.io/submariner/lighthouse-agent:devel
  592. 2020-10-08T11:28:09.0058048Z quay.io/submariner/lighthouse-agent:devel
  593. 2020-10-08T11:28:09.0104069Z [lighthouse]$ grep FROM package/Dockerfile.lighthouse-agent
  594. 2020-10-08T11:28:09.0104952Z [lighthouse]$ cut -f2 -d 
  595. 2020-10-08T11:28:09.0105514Z [lighthouse]$ grep -v scratch
  596. 2020-10-08T11:28:09.0118854Z [lighthouse]$ cache_flag+= --cache-from registry.access.redhat.com/ubi8/ubi-minimal
  597. 2020-10-08T11:28:09.0171555Z [lighthouse]$ docker pull registry.access.redhat.com/ubi8/ubi-minimal
  598. 2020-10-08T11:28:09.3167905Z Using default tag: latest
  599. 2020-10-08T11:28:09.6071879Z latest: Pulling from ubi8/ubi-minimal
  600. 2020-10-08T11:28:09.7304769Z 0fd3b5213a9b: Already exists
  601. 2020-10-08T11:28:09.7325112Z aebb8c556853: Already exists
  602. 2020-10-08T11:28:09.8372964Z Digest: sha256:5cfbaf45ca96806917830c183e9f37df2e913b187aadb32e89fd83fa455ebaa6
  603. 2020-10-08T11:28:09.8392096Z Status: Downloaded newer image for registry.access.redhat.com/ubi8/ubi-minimal:latest
  604. 2020-10-08T11:28:09.8399279Z registry.access.redhat.com/ubi8/ubi-minimal:latest
  605. 2020-10-08T11:28:09.8432726Z [lighthouse]$ buildargs_flag=--build-arg BUILDKIT_INLINE_CACHE=1
  606. 2020-10-08T11:28:09.8495147Z [lighthouse]$ DOCKER_BUILDKIT=1 docker build -t quay.io/submariner/lighthouse-agent:dev --cache-from quay.io/submariner/lighthouse-agent:devel --cache-from registry.access.redhat.com/ubi8/ubi-minimal -f package/Dockerfile.lighthouse-agent --build-arg BUILDKIT_INLINE_CACHE=1 .
  607. 2020-10-08T11:28:10.1962961Z #2 [internal] load .dockerignore
  608. 2020-10-08T11:28:10.2955415Z #2 transferring context: 2B done
  609. 2020-10-08T11:28:10.2955945Z #2 DONE 0.0s
  610. 2020-10-08T11:28:10.2957126Z
  611. 2020-10-08T11:28:10.2958439Z #1 [internal] load build definition from Dockerfile.lighthouse-agent
  612. 2020-10-08T11:28:10.2959128Z #1 transferring dockerfile: 238B done
  613. 2020-10-08T11:28:10.2959495Z #1 DONE 0.0s
  614. 2020-10-08T11:28:10.2959652Z
  615. 2020-10-08T11:28:10.2960604Z #3 [internal] load metadata for registry.access.redhat.com/ubi8/ubi-minimal...
  616. 2020-10-08T11:28:10.2961243Z #3 DONE 0.0s
  617. 2020-10-08T11:28:10.2961451Z
  618. 2020-10-08T11:28:10.2962138Z #4 importing cache manifest from quay.io/submariner/lighthouse-agent:devel
  619. 2020-10-08T11:28:10.2962608Z #4 DONE 0.0s
  620. 2020-10-08T11:28:10.2962815Z
  621. 2020-10-08T11:28:10.2963612Z #5 importing cache manifest from registry.access.redhat.com/ubi8/ubi-minima...
  622. 2020-10-08T11:28:10.2964287Z #5 DONE 0.0s
  623. 2020-10-08T11:28:10.2964423Z
  624. 2020-10-08T11:28:10.2965010Z #6 [1/3] FROM registry.access.redhat.com/ubi8/ubi-minimal
  625. 2020-10-08T11:28:10.2965663Z #6 DONE 0.0s
  626. 2020-10-08T11:28:10.2965801Z
  627. 2020-10-08T11:28:10.2966076Z #8 [internal] load build context
  628. 2020-10-08T11:28:10.2966445Z #8 transferring context: 8.57MB 0.1s done
  629. 2020-10-08T11:28:10.2966750Z #8 DONE 0.1s
  630. 2020-10-08T11:28:10.2966882Z
  631. 2020-10-08T11:28:10.2967177Z #7 [2/3] WORKDIR /var/submariner
  632. 2020-10-08T11:28:10.2967462Z #7 CACHED
  633. 2020-10-08T11:28:10.2967601Z
  634. 2020-10-08T11:28:10.2968234Z #9 [3/3] COPY bin/lighthouse-agent package/lighthouse-agent.sh /usr/local/b...
  635. 2020-10-08T11:28:10.2968743Z #9 CACHED
  636. 2020-10-08T11:28:10.3029342Z
  637. 2020-10-08T11:28:10.3040744Z #10 exporting to image
  638. 2020-10-08T11:28:10.3041077Z #10 exporting layers done
  639. 2020-10-08T11:28:10.3041947Z #10 writing image sha256:8691ff88f1c80f813157a9e3c20edcc3c7d5b888527ac58257fa05a02a4808a0 done
  640. 2020-10-08T11:28:10.3043554Z #10 naming to quay.io/submariner/lighthouse-agent:dev done
  641. 2020-10-08T11:28:10.3044161Z #10 DONE 0.0s
  642. 2020-10-08T11:28:10.3044371Z
  643. 2020-10-08T11:28:10.3044610Z #11 exporting cache
  644. 2020-10-08T11:28:10.3045056Z #11 preparing build cache for export done
  645. 2020-10-08T11:28:10.3045438Z #11 DONE 0.0s
  646. 2020-10-08T11:28:10.3138213Z [lighthouse]$ docker tag quay.io/submariner/lighthouse-agent:dev quay.io/submariner/lighthouse-agent:devel
  647. 2020-10-08T11:28:10.5974993Z touch package/.image.lighthouse-agent
  648. 2020-10-08T11:28:10.6033230Z find: 'bin/lighthouse-coredns': No such file or directory
  649. 2020-10-08T11:28:10.9233144Z /opt/shipyard/scripts/compile.sh bin/lighthouse-coredns pkg/coredns/main.go
  650. 2020-10-08T11:28:10.9649882Z [lighthouse]$ mkdir -p bin
  651. 2020-10-08T11:28:10.9661649Z Building 'bin/lighthouse-coredns' (ldflags: '')
  652. 2020-10-08T11:28:10.9672587Z [lighthouse]$ ldflags=-s -w 
  653. 2020-10-08T11:28:10.9683421Z [lighthouse]$ CGO_ENABLED=0 go build -ldflags -s -w -o bin/lighthouse-coredns pkg/coredns/main.go
  654. 2020-10-08T11:28:44.4257528Z [lighthouse]$ upx bin/lighthouse-coredns
  655. 2020-10-08T11:28:51.2656552Z Ultimate Packer for eXecutables
  656. 2020-10-08T11:28:51.2658459Z Copyright (C) 1996 - 2020
  657. 2020-10-08T11:28:51.2662319Z UPX 3.96 Markus Oberhumer, Laszlo Molnar & John Reiser Jan 23rd 2020
  658. 2020-10-08T11:28:51.2664286Z
  659. 2020-10-08T11:28:51.2665191Z File size Ratio Format Name
  660. 2020-10-08T11:28:51.2666157Z -------------------- ------ ----------- -----------
  661. 2020-10-08T11:28:51.2667127Z 42328064 -> 12750660 30.12% linux/amd64 lighthouse-coredns
  662. 2020-10-08T11:28:51.2667649Z
  663. 2020-10-08T11:28:51.2668127Z Packed 1 file.
  664. 2020-10-08T11:28:51.2669489Z /opt/shipyard/scripts/build_image.sh -i lighthouse-coredns -f package/Dockerfile.lighthouse-coredns
  665. 2020-10-08T11:28:51.3207647Z fatal: No names found, cannot describe anything.
  666. 2020-10-08T11:28:51.4013478Z [lighthouse]$ set -e
  667. 2020-10-08T11:28:51.4024011Z [lighthouse]$ local_image=quay.io/submariner/lighthouse-coredns:dev
  668. 2020-10-08T11:28:51.4033023Z [lighthouse]$ cache_image=quay.io/submariner/lighthouse-coredns:devel
  669. 2020-10-08T11:28:51.4042396Z [lighthouse]$ cache_flag=
  670. 2020-10-08T11:28:51.4056395Z [lighthouse]$ cache_flag=--cache-from quay.io/submariner/lighthouse-coredns:devel
  671. 2020-10-08T11:28:51.4073511Z [lighthouse]$ docker image ls -q quay.io/submariner/lighthouse-coredns:devel
  672. 2020-10-08T11:28:51.7312194Z [lighthouse]$ docker pull quay.io/submariner/lighthouse-coredns:devel
  673. 2020-10-08T11:28:52.1667634Z devel: Pulling from submariner/lighthouse-coredns
  674. 2020-10-08T11:28:52.1669632Z 696098ac4087: Pulling fs layer
  675. 2020-10-08T11:28:52.1671067Z dfb67c3040e6: Pulling fs layer
  676. 2020-10-08T11:28:52.1673365Z e9d8b7f028b4: Pulling fs layer
  677. 2020-10-08T11:28:52.3784473Z e9d8b7f028b4: Verifying Checksum
  678. 2020-10-08T11:28:52.3785069Z e9d8b7f028b4: Download complete
  679. 2020-10-08T11:28:52.3786033Z dfb67c3040e6: Verifying Checksum
  680. 2020-10-08T11:28:52.3786497Z dfb67c3040e6: Download complete
  681. 2020-10-08T11:28:52.4035395Z 696098ac4087: Verifying Checksum
  682. 2020-10-08T11:28:52.4036112Z 696098ac4087: Download complete
  683. 2020-10-08T11:28:53.4802489Z 696098ac4087: Pull complete
  684. 2020-10-08T11:28:54.2862065Z [lighthouse]$ grep FROM package/Dockerfile.lighthouse-coredns
  685. 2020-10-08T11:28:54.2862924Z [lighthouse]$ cut -f2 -d 
  686. 2020-10-08T11:28:54.2863461Z [lighthouse]$ grep -v scratch
  687. 2020-10-08T11:28:54.2864145Z [lighthouse]$ cache_flag+= --cache-from debian:stable-slim
  688. 2020-10-08T11:28:54.2864824Z [lighthouse]$ docker pull debian:stable-slim
  689. 2020-10-08T11:28:54.2865292Z dfb67c3040e6: Pull complete
  690. 2020-10-08T11:28:54.2865675Z e9d8b7f028b4: Pull complete
  691. 2020-10-08T11:28:54.2866635Z Digest: sha256:596c14dfebe9da4398034185ee0d6284ec0243d030a7cc6ffb6443b5e14cc787
  692. 2020-10-08T11:28:54.2867916Z Status: Downloaded newer image for quay.io/submariner/lighthouse-coredns:devel
  693. 2020-10-08T11:28:54.2868803Z quay.io/submariner/lighthouse-coredns:devel
  694. 2020-10-08T11:28:54.4090246Z stable-slim: Pulling from library/debian
  695. 2020-10-08T11:28:54.4568292Z 696098ac4087: Already exists
  696. 2020-10-08T11:28:54.6642335Z Digest: sha256:89ff9e144a438f6bdf89fba6a1fdcb614b6d03bc14433bbb937088ca7c7a7b6d
  697. 2020-10-08T11:28:54.6649141Z [lighthouse]$ buildargs_flag=--build-arg BUILDKIT_INLINE_CACHE=1
  698. 2020-10-08T11:28:54.6650996Z [lighthouse]$ DOCKER_BUILDKIT=1 docker build -t quay.io/submariner/lighthouse-coredns:dev --cache-from quay.io/submariner/lighthouse-coredns:devel --cache-from debian:stable-slim -f package/Dockerfile.lighthouse-coredns --build-arg BUILDKIT_INLINE_CACHE=1 .
  699. 2020-10-08T11:28:54.6696278Z Status: Downloaded newer image for debian:stable-slim
  700. 2020-10-08T11:28:54.6696945Z docker.io/library/debian:stable-slim
  701. 2020-10-08T11:28:54.9848978Z #2 [internal] load .dockerignore
  702. 2020-10-08T11:28:54.9849895Z #2 DONE 0.0s
  703. 2020-10-08T11:28:54.9850776Z
  704. 2020-10-08T11:28:54.9852306Z #1 [internal] load build definition from Dockerfile.lighthouse-coredns
  705. 2020-10-08T11:28:55.1011263Z #1 transferring dockerfile: 284B 0.0s done
  706. 2020-10-08T11:28:55.1011902Z #1 DONE 0.0s
  707. 2020-10-08T11:28:55.1012082Z
  708. 2020-10-08T11:28:55.1012848Z #2 [internal] load .dockerignore
  709. 2020-10-08T11:28:55.1013256Z #2 transferring context: 2B done
  710. 2020-10-08T11:28:55.1013890Z #2 DONE 0.0s
  711. 2020-10-08T11:28:55.1014068Z
  712. 2020-10-08T11:28:55.1015356Z #3 [internal] load metadata for docker.io/library/debian:stable-slim
  713. 2020-10-08T11:28:55.1015785Z #3 DONE 0.0s
  714. 2020-10-08T11:28:55.1015939Z
  715. 2020-10-08T11:28:55.1016838Z #8 importing cache manifest from quay.io/submariner/lighthouse-coredns:deve...
  716. 2020-10-08T11:28:55.1017353Z #8 DONE 0.0s
  717. 2020-10-08T11:28:55.1017614Z
  718. 2020-10-08T11:28:55.1018132Z #9 importing cache manifest from debian:stable-slim
  719. 2020-10-08T11:28:55.1018606Z #9 DONE 0.0s
  720. 2020-10-08T11:28:55.1018774Z
  721. 2020-10-08T11:28:55.1019255Z #4 [1/3] FROM docker.io/library/debian:stable-slim
  722. 2020-10-08T11:28:55.1019716Z #4 DONE 0.0s
  723. 2020-10-08T11:28:55.1019900Z
  724. 2020-10-08T11:28:55.1020174Z #6 [internal] load build context
  725. 2020-10-08T11:28:55.1020544Z #6 transferring context: 12.75MB 0.1s done
  726. 2020-10-08T11:28:55.1376014Z #6 DONE 0.1s
  727. 2020-10-08T11:28:55.1376239Z
  728. 2020-10-08T11:28:55.1377191Z #5 [2/3] RUN apt-get update && apt-get -y install ca-certificates && update...
  729. 2020-10-08T11:28:55.1377634Z #5 CACHED
  730. 2020-10-08T11:28:55.1377792Z
  731. 2020-10-08T11:28:55.1378324Z #7 [3/3] COPY bin/lighthouse-coredns /usr/local/bin/
  732. 2020-10-08T11:28:55.1378711Z #7 CACHED
  733. 2020-10-08T11:28:55.1378867Z
  734. 2020-10-08T11:28:55.1379119Z #10 exporting to image
  735. 2020-10-08T11:28:55.1379437Z #10 exporting layers done
  736. 2020-10-08T11:28:55.1380168Z #10 writing image sha256:e0f6047a56d806a384ed987e1bb68e30f033968e14526609988149d3380a0561 done
  737. 2020-10-08T11:28:55.1381221Z #10 naming to quay.io/submariner/lighthouse-coredns:dev done
  738. 2020-10-08T11:28:55.1381654Z #10 DONE 0.0s
  739. 2020-10-08T11:28:55.1381808Z
  740. 2020-10-08T11:28:55.1382047Z #11 exporting cache
  741. 2020-10-08T11:28:55.1382417Z #11 preparing build cache for export done
  742. 2020-10-08T11:28:55.1382726Z #11 DONE 0.0s
  743. 2020-10-08T11:28:55.1383598Z [lighthouse]$ docker tag quay.io/submariner/lighthouse-coredns:dev quay.io/submariner/lighthouse-coredns:devel
  744. 2020-10-08T11:28:55.4460033Z touch package/.image.lighthouse-coredns
  745. 2020-10-08T11:28:55.4478429Z /opt/shipyard/scripts/clusters.sh --cluster_settings /go/src/github.com/submariner-io/lighthouse/scripts/cluster_settings
  746. 2020-10-08T11:28:55.5130088Z Running with: k8s_version=1.17.0, olm_version=0.14.1, olm=false, globalnet=false, registry_inmemory=true, cluster_settings=/go/src/github.com/submariner-io/lighthouse/scripts/cluster_settings, timeout=5m
  747. 2020-10-08T11:28:55.5150437Z [lighthouse]$ source /opt/shipyard/scripts/lib/utils
  748. 2020-10-08T11:28:55.5159996Z [lighthouse]$ . /opt/shipyard/scripts/lib/source_only
  749. 2020-10-08T11:28:55.5169329Z [lighthouse]$ script_name=utils
  750. 2020-10-08T11:28:55.5178558Z [lighthouse]$ exec_name=clusters.sh
  751. 2020-10-08T11:28:55.5193785Z [lighthouse]$ source /opt/shipyard/scripts/lib/cluster_settings
  752. 2020-10-08T11:28:55.5203196Z [lighthouse]$ . /opt/shipyard/scripts/lib/source_only
  753. 2020-10-08T11:28:55.5211537Z [lighthouse]$ script_name=cluster_settings
  754. 2020-10-08T11:28:55.5224261Z [lighthouse]$ exec_name=clusters.sh
  755. 2020-10-08T11:28:55.5240629Z [lighthouse]$ broker=cluster1
  756. 2020-10-08T11:28:55.5251724Z [lighthouse]$ declare -gA cluster_nodes
  757. 2020-10-08T11:28:55.5263543Z [lighthouse]$ cluster_nodes[cluster1]=control-plane worker
  758. 2020-10-08T11:28:55.5274204Z [lighthouse]$ cluster_nodes[cluster2]=control-plane worker
  759. 2020-10-08T11:28:55.5286067Z [lighthouse]$ cluster_nodes[cluster3]=control-plane worker worker
  760. 2020-10-08T11:28:55.5295031Z [lighthouse]$ declare -gA cluster_subm
  761. 2020-10-08T11:28:55.5307596Z [lighthouse]$ cluster_subm[cluster1]=true
  762. 2020-10-08T11:28:55.5315933Z [lighthouse]$ cluster_subm[cluster2]=true
  763. 2020-10-08T11:28:55.5325879Z [lighthouse]$ cluster_subm[cluster3]=true
  764. 2020-10-08T11:28:55.5336868Z [lighthouse]$ declare -gA cluster_cni
  765. 2020-10-08T11:28:55.5344103Z [lighthouse]$ cluster_cni[cluster2]=weave
  766. 2020-10-08T11:28:55.5353166Z [lighthouse]$ cluster_cni[cluster3]=weave
  767. 2020-10-08T11:28:55.5364108Z [lighthouse]$ source /go/src/github.com/submariner-io/lighthouse/scripts/cluster_settings
  768. 2020-10-08T11:28:55.5372900Z [lighthouse]$ . /opt/shipyard/scripts/lib/source_only
  769. 2020-10-08T11:28:55.5386272Z [lighthouse]$ script_name=cluster_settings
  770. 2020-10-08T11:28:55.5394341Z [lighthouse]$ exec_name=clusters.sh
  771. 2020-10-08T11:28:55.5413951Z [lighthouse]$ cluster_nodes[cluster1]=control-plane worker worker
  772. 2020-10-08T11:28:55.5424011Z [lighthouse]$ cluster_nodes[cluster2]=control-plane worker worker
  773. 2020-10-08T11:28:55.5485407Z [lighthouse]$ cat
  774. 2020-10-08T11:28:55.5504125Z [lighthouse]$ typeset -p cluster_nodes
  775. 2020-10-08T11:28:55.5517094Z [lighthouse]$ cut -f 2- -d=
  776. 2020-10-08T11:28:55.5544437Z [lighthouse]$ typeset -p cluster_subm
  777. 2020-10-08T11:28:55.5556911Z [lighthouse]$ cut -f 2- -d=
  778. 2020-10-08T11:28:55.5575311Z Cluster settings::
  779. 2020-10-08T11:28:55.5576009Z broker - 'cluster1'
  780. 2020-10-08T11:28:55.5576501Z clusters - 'cluster1' 'cluster2'
  781. 2020-10-08T11:28:55.5577502Z nodes per cluster - ([cluster3]="control-plane worker worker" [cluster2]="control-plane worker worker" [cluster1]="control-plane worker worker" )
  782. 2020-10-08T11:28:55.5578475Z install submariner - ([cluster2]="true" [cluster1]="true" )
  783. 2020-10-08T11:28:55.5591423Z [lighthouse]$ rm -rf /go/src/github.com/submariner-io/lighthouse/output/kubeconfigs
  784. 2020-10-08T11:28:55.5610651Z [lighthouse]$ mkdir -p /go/src/github.com/submariner-io/lighthouse/output/kubeconfigs
  785. 2020-10-08T11:28:55.5632548Z [lighthouse]$ run_local_registry
  786. 2020-10-08T11:28:55.5642992Z [lighthouse]$ run_local_registry
  787. 2020-10-08T11:28:55.5653159Z [lighthouse]$ registry_running
  788. 2020-10-08T11:28:55.5663495Z [lighthouse]$ registry_running
  789. 2020-10-08T11:28:55.5674518Z [lighthouse]$ docker ps --filter name=^/?kind-registry$
  790. 2020-10-08T11:28:55.5684611Z [lighthouse]$ grep kind-registry
  791. 2020-10-08T11:28:55.8797145Z [lighthouse]$ return 0
  792. 2020-10-08T11:28:55.8797853Z Deploying local registry kind-registry to serve images centrally.
  793. 2020-10-08T11:28:55.8815508Z [lighthouse]$ local volume_flag
  794. 2020-10-08T11:28:55.8825214Z [lighthouse]$ volume_flag=-v /dev/shm/kind-registry:/var/lib/registry
  795. 2020-10-08T11:28:55.8837300Z [lighthouse]$ docker run -d -v /dev/shm/kind-registry:/var/lib/registry -p 5000:5000 --restart=always --name kind-registry registry:2
  796. 2020-10-08T11:28:56.1877967Z Unable to find image 'registry:2' locally
  797. 2020-10-08T11:28:56.3053563Z 2: Pulling from library/registry
  798. 2020-10-08T11:28:56.3418747Z cbdbe7a5bc2a: Already exists
  799. 2020-10-08T11:28:56.3461006Z 47112e65547d: Pulling fs layer
  800. 2020-10-08T11:28:56.3462087Z 46bcb632e506: Pulling fs layer
  801. 2020-10-08T11:28:56.3462869Z c1cc712bcecd: Pulling fs layer
  802. 2020-10-08T11:28:56.3463553Z 3db6272dcbfa: Pulling fs layer
  803. 2020-10-08T11:28:56.3464171Z 3db6272dcbfa: Waiting
  804. 2020-10-08T11:28:56.5228040Z c1cc712bcecd: Verifying Checksum
  805. 2020-10-08T11:28:56.5228547Z c1cc712bcecd: Download complete
  806. 2020-10-08T11:28:56.6145625Z 47112e65547d: Download complete
  807. 2020-10-08T11:28:56.6983517Z 3db6272dcbfa: Verifying Checksum
  808. 2020-10-08T11:28:56.6983986Z 3db6272dcbfa: Download complete
  809. 2020-10-08T11:28:56.7265418Z 47112e65547d: Pull complete
  810. 2020-10-08T11:28:56.7873147Z 46bcb632e506: Verifying Checksum
  811. 2020-10-08T11:28:56.7873608Z 46bcb632e506: Download complete
  812. 2020-10-08T11:28:57.0015872Z 46bcb632e506: Pull complete
  813. 2020-10-08T11:28:57.0592183Z c1cc712bcecd: Pull complete
  814. 2020-10-08T11:28:57.1165592Z 3db6272dcbfa: Pull complete
  815. 2020-10-08T11:28:57.1191863Z Digest: sha256:8be26f81ffea54106bae012c6f349df70f4d5e7e2ec01b143c46e2c03b9e551d
  816. 2020-10-08T11:28:57.1208114Z Status: Downloaded newer image for registry:2
  817. 2020-10-08T11:29:00.1761778Z 0b9a81c12fb0455fecfb6f1fcf3d0074b811fdcd166ba79f52392aee5edad05c
  818. 2020-10-08T11:29:00.8252852Z [lighthouse]$ registry_ip=172.17.0.3
  819. 2020-10-08T11:29:00.8268601Z [lighthouse]$ docker inspect -f {{.NetworkSettings.IPAddress}} kind-registry
  820. 2020-10-08T11:29:01.1437813Z [lighthouse]$ declare_cidrs
  821. 2020-10-08T11:29:01.1447519Z [lighthouse]$ declare_cidrs
  822. 2020-10-08T11:29:01.1458867Z [lighthouse]$ declare -gA cluster_CIDRs service_CIDRs global_CIDRs
  823. 2020-10-08T11:29:01.1471324Z [lighthouse]$ i=1
  824. 2020-10-08T11:29:01.1484831Z [lighthouse]$ [cluster1] add_cluster_cidrs 1 cluster1
  825. 2020-10-08T11:29:01.1497079Z [lighthouse]$ [cluster1] add_cluster_cidrs 1 cluster1
  826. 2020-10-08T11:29:01.1507051Z [lighthouse]$ [cluster1] local val=1
  827. 2020-10-08T11:29:01.1517717Z [lighthouse]$ [cluster1] local idx=cluster1
  828. 2020-10-08T11:29:01.1527898Z [lighthouse]$ [cluster1] cluster_CIDRs[cluster1]=10.241.0.0/16
  829. 2020-10-08T11:29:01.1540571Z [lighthouse]$ [cluster1] service_CIDRs[cluster1]=100.91.0.0/16
  830. 2020-10-08T11:29:01.1554724Z [lighthouse]$ [cluster1] i=2
  831. 2020-10-08T11:29:01.1565893Z [lighthouse]$ [cluster2] add_cluster_cidrs 2 cluster2
  832. 2020-10-08T11:29:01.1577018Z [lighthouse]$ [cluster2] add_cluster_cidrs 2 cluster2
  833. 2020-10-08T11:29:01.1587880Z [lighthouse]$ [cluster2] local val=2
  834. 2020-10-08T11:29:01.1599231Z [lighthouse]$ [cluster2] local idx=cluster2
  835. 2020-10-08T11:29:01.1612853Z [lighthouse]$ [cluster2] cluster_CIDRs[cluster2]=10.242.0.0/16
  836. 2020-10-08T11:29:01.1624816Z [lighthouse]$ [cluster2] service_CIDRs[cluster2]=100.92.0.0/16
  837. 2020-10-08T11:29:01.1638507Z [lighthouse]$ [cluster2] i=3
  838. 2020-10-08T11:29:01.1651680Z [lighthouse]$ [cluster2] run_all_clusters with_retries 3 create_kind_cluster
  839. 2020-10-08T11:29:01.1665857Z [lighthouse]$ [cluster2] run_all_clusters with_retries 3 create_kind_cluster
  840. 2020-10-08T11:29:01.1679409Z [lighthouse]$ [cluster2] run_parallel cluster1 cluster2 with_retries 3 create_kind_cluster
  841. 2020-10-08T11:29:01.1695056Z [lighthouse]$ [cluster2] run_parallel cluster1 cluster2 cluster1 cluster2 with_retries 3 create_kind_cluster
  842. 2020-10-08T11:29:01.1704909Z [lighthouse]$ [cluster2] local cmnd=with_retries
  843. 2020-10-08T11:29:01.1719357Z [lighthouse]$ [cluster2] declare -A pids
  844. 2020-10-08T11:29:01.1734543Z [lighthouse]$ [cluster2] eval echo cluster1 cluster2
  845. 2020-10-08T11:29:01.1754147Z [lighthouse]$ [cluster1] pids[cluster1]=3406
  846. 2020-10-08T11:29:01.1762262Z [lighthouse]$ [cluster1] set -o pipefail
  847. 2020-10-08T11:29:01.1769164Z [lighthouse]$ [cluster2] pids[cluster2]=3409
  848. 2020-10-08T11:29:01.1777139Z [lighthouse]$ [cluster1] with_retries 3 create_kind_cluster
  849. 2020-10-08T11:29:01.1791206Z [lighthouse]$ [cluster2] wait 3409
  850. 2020-10-08T11:29:01.1794718Z [lighthouse]$ [cluster2] set -o pipefail
  851. 2020-10-08T11:29:01.1801391Z [lighthouse]$ [cluster1] sed /\[cluster1]/!s/^/[cluster1] /
  852. 2020-10-08T11:29:01.1807382Z [lighthouse]$ [cluster2] with_retries 3 create_kind_cluster
  853. 2020-10-08T11:29:01.1823186Z [lighthouse]$ [cluster2] sed /\[cluster2]/!s/^/[cluster2] /
  854. 2020-10-08T11:32:26.2393256Z [lighthouse]$ [cluster1] with_retries
  855. 2020-10-08T11:32:26.2393962Z [lighthouse]$ [cluster1] local retries
  856. 2020-10-08T11:32:26.2394595Z [lighthouse]$ [cluster1] retries=1 2 3
  857. 2020-10-08T11:32:26.2395206Z [lighthouse]$ [cluster1] eval echo {1..3}
  858. 2020-10-08T11:32:26.2395890Z [lighthouse]$ [cluster1] local cmnd=create_kind_cluster
  859. 2020-10-08T11:32:26.2396547Z [lighthouse]$ [cluster1] wait 3435
  860. 2020-10-08T11:32:26.2397172Z [lighthouse]$ [cluster1] create_kind_cluster
  861. 2020-10-08T11:32:26.2397826Z [lighthouse]$ [cluster1] create_kind_cluster
  862. 2020-10-08T11:32:26.2399239Z [lighthouse]$ [cluster1] export KUBECONFIG=/go/src/github.com/submariner-io/lighthouse/output/kubeconfigs/kind-config-cluster1
  863. 2020-10-08T11:32:26.2400765Z [lighthouse]$ [cluster1] rm -f /go/src/github.com/submariner-io/lighthouse/output/kubeconfigs/kind-config-cluster1
  864. 2020-10-08T11:32:26.2401775Z [lighthouse]$ [cluster1] kind get clusters
  865. 2020-10-08T11:32:26.2402431Z [lighthouse]$ [cluster1] grep -q ^cluster1$
  866. 2020-10-08T11:32:26.2402889Z [cluster1] No kind clusters found.
  867. 2020-10-08T11:32:26.2403327Z [cluster1] Creating KIND cluster...
  868. 2020-10-08T11:32:26.2403981Z [lighthouse]$ [cluster1] generate_cluster_yaml
  869. 2020-10-08T11:32:26.2404676Z [lighthouse]$ [cluster1] generate_cluster_yaml
  870. 2020-10-08T11:32:26.2405354Z [lighthouse]$ [cluster1] local pod_cidr=10.241.0.0/16
  871. 2020-10-08T11:32:26.2406013Z [lighthouse]$ [cluster1] local service_cidr=100.91.0.0/16
  872. 2020-10-08T11:32:26.2406770Z [lighthouse]$ [cluster1] local dns_domain=cluster1.local
  873. 2020-10-08T11:32:26.2407517Z [lighthouse]$ [cluster1] local disable_cni=false
  874. 2020-10-08T11:32:26.2408189Z [lighthouse]$ [cluster1] disable_cni=true
  875. 2020-10-08T11:32:26.2408811Z [lighthouse]$ [cluster1] local nodes
  876. 2020-10-08T11:32:26.2409388Z [lighthouse]$ [cluster1] nodes=
  877. 2020-10-08T11:32:26.2409970Z [cluster1] - role: control-plane
  878. 2020-10-08T11:32:26.2410546Z [lighthouse]$ [cluster1] nodes=
  879. 2020-10-08T11:32:26.2411097Z [cluster1] - role: control-plane
  880. 2020-10-08T11:32:26.2411650Z [cluster1] - role: worker
  881. 2020-10-08T11:32:26.2412180Z [lighthouse]$ [cluster1] nodes=
  882. 2020-10-08T11:32:26.2435907Z [cluster1] - role: control-plane
  883. 2020-10-08T11:32:26.2436458Z [cluster1] - role: worker
  884. 2020-10-08T11:32:26.2436946Z [cluster1] - role: worker
  885. 2020-10-08T11:32:26.2437398Z [cluster1] 
  886. 2020-10-08T11:32:26.2437803Z [cluster1] 
  887. 2020-10-08T11:32:26.2438327Z [lighthouse]$ [cluster1] eval echo "---
  888. 2020-10-08T11:32:26.2438743Z [cluster1] kind: Cluster
  889. 2020-10-08T11:32:26.2439493Z [cluster1] apiVersion: kind.x-k8s.io/v1alpha4
  890. 2020-10-08T11:32:26.2440191Z [cluster1] networking:
  891. 2020-10-08T11:32:26.2440679Z [cluster1] disableDefaultCNI: ${disable_cni}
  892. 2020-10-08T11:32:26.2441177Z [cluster1] podSubnet: ${pod_cidr}
  893. 2020-10-08T11:32:26.2441637Z [cluster1] serviceSubnet: ${service_cidr}
  894. 2020-10-08T11:32:26.2442204Z [cluster1] containerdConfigPatches:
  895. 2020-10-08T11:32:26.2442828Z [cluster1] - |-
  896. 2020-10-08T11:32:26.2443661Z [cluster1] [plugins.\"io.containerd.grpc.v1.cri\".registry.mirrors.\"localhost:5000\"]
  897. 2020-10-08T11:32:26.2444477Z [cluster1] endpoint = [\"http://${registry_ip}:5000\"]
  898. 2020-10-08T11:32:26.2445016Z [cluster1] kubeadmConfigPatches:
  899. 2020-10-08T11:32:26.2445578Z [cluster1] - |
  900. 2020-10-08T11:32:26.2446007Z [cluster1] apiVersion: kubeadm.k8s.io/v1beta2
  901. 2020-10-08T11:32:26.2446745Z [cluster1] kind: ClusterConfiguration
  902. 2020-10-08T11:32:26.2447193Z [cluster1] metadata:
  903. 2020-10-08T11:32:26.2447547Z [cluster1] name: config
  904. 2020-10-08T11:32:26.2447906Z [cluster1] networking:
  905. 2020-10-08T11:32:26.2448306Z [cluster1] podSubnet: ${pod_cidr}
  906. 2020-10-08T11:32:26.2448765Z [cluster1] serviceSubnet: ${service_cidr}
  907. 2020-10-08T11:32:26.2449228Z [cluster1] dnsDomain: ${dns_domain}
  908. 2020-10-08T11:32:26.2449769Z [cluster1] nodes: ${nodes}"
  909. 2020-10-08T11:32:26.2454873Z [lighthouse]$ [cluster1] cat /opt/shipyard/scripts/resources/kind-cluster-config.yaml
  910. 2020-10-08T11:32:26.2455883Z [lighthouse]$ [cluster1] local image_flag=
  911. 2020-10-08T11:32:26.2456704Z [lighthouse]$ [cluster1] image_flag=--image=kindest/node:v1.17.0
  912. 2020-10-08T11:32:26.2457999Z [lighthouse]$ [cluster1] kind create cluster --image=kindest/node:v1.17.0 --name=cluster1 --config=/opt/shipyard/scripts/resources/cluster1-config.yaml
  913. 2020-10-08T11:32:26.2458913Z [cluster1] Creating cluster "cluster1" ...
  914. 2020-10-08T11:32:26.2459883Z [cluster1] • Ensuring node image (kindest/node:v1.17.0) 🖼 ...
  915. 2020-10-08T11:32:26.2460677Z [cluster1] ✓ Ensuring node image (kindest/node:v1.17.0) 🖼
  916. 2020-10-08T11:32:26.2461353Z [cluster1] • Preparing nodes 📦 📦 📦 ...
  917. 2020-10-08T11:32:26.2461911Z [cluster1] ✓ Preparing nodes 📦 📦 📦
  918. 2020-10-08T11:32:26.2462488Z [cluster1] • Writing configuration 📜 ...
  919. 2020-10-08T11:32:26.2463113Z [cluster1] ✓ Writing configuration 📜
  920. 2020-10-08T11:32:26.2463752Z [cluster1] • Starting control-plane 🕹️ ...
  921. 2020-10-08T11:32:26.2464386Z [cluster1] ✓ Starting control-plane 🕹️
  922. 2020-10-08T11:32:26.2465033Z [cluster1] • Installing StorageClass 💾 ...
  923. 2020-10-08T11:32:26.2465836Z [cluster1] ✓ Installing StorageClass 💾
  924. 2020-10-08T11:32:26.2466452Z [cluster1] • Joining worker nodes 🚜 ...
  925. 2020-10-08T11:32:26.2467011Z [cluster1] ✓ Joining worker nodes 🚜
  926. 2020-10-08T11:32:26.2467633Z [cluster1] Set kubectl context to "kind-cluster1"
  927. 2020-10-08T11:32:26.2468139Z [cluster1] You can now use your cluster with:
  928. 2020-10-08T11:32:26.2468531Z [cluster1]
  929. 2020-10-08T11:32:26.2469161Z [cluster1] kubectl cluster-info --context kind-cluster1
  930. 2020-10-08T11:32:26.2469613Z [cluster1]
  931. 2020-10-08T11:32:26.2470126Z [cluster1] Thanks for using kind! 😊
  932. 2020-10-08T11:32:26.2470730Z [lighthouse]$ [cluster1] kind_fixup_config
  933. 2020-10-08T11:32:26.2471373Z [lighthouse]$ [cluster1] kind_fixup_config
  934. 2020-10-08T11:32:26.2472015Z [lighthouse]$ [cluster1] local master_ip=172.17.0.8
  935. 2020-10-08T11:32:26.2473187Z [lighthouse]$ [cluster1] docker inspect -f {{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}} cluster1-control-plane
  936. 2020-10-08T11:32:33.3692308Z [lighthouse]$ [cluster1] head -n 1[[lighthouse]$ [cluster2] with_retries
  937. 2020-10-08T11:32:33.3693318Z [lighthouse]$ [cluster2] local retries
  938. 2020-10-08T11:32:33.3693825Z [lighthouse]$ [cluster2] retries=1 2 3
  939. 2020-10-08T11:32:33.3694319Z [lighthouse]$ [cluster2] eval echo {1..3}
  940. 2020-10-08T11:32:33.3694928Z [lighthouse]$ [cluster2] local cmnd=create_kind_cluster
  941. 2020-10-08T11:32:33.3695462Z [lighthouse]$ [cluster2] wait 3438
  942. 2020-10-08T11:32:33.3695969Z [lighthouse]$ [cluster2] create_kind_cluster
  943. 2020-10-08T11:32:33.3696716Z [lighthouse]$ [cluster2] create_kind_cluster
  944. 2020-10-08T11:32:33.3697748Z [lighthouse]$ [cluster2] export KUBECONFIG=/go/src/github.com/submariner-io/lighthouse/output/kubeconfigs/kind-config-cluster2
  945. 2020-10-08T11:32:33.3698959Z [lighthouse]$ [cluster2] rm -f /go/src/github.com/submariner-io/lighthouse/output/kubeconfigs/kind-config-cluster2
  946. 2020-10-08T11:32:33.3699774Z [lighthouse]$ [cluster2] kind get clusters
  947. 2020-10-08T11:32:33.3700314Z [lighthouse]$ [cluster2] grep -q ^cluster2$
  948. 2020-10-08T11:32:33.3700687Z [cluster2] No kind clusters found.
  949. 2020-10-08T11:32:33.3701043Z [cluster2] Creating KIND cluster...
  950. 2020-10-08T11:32:33.3701576Z [lighthouse]$ [cluster2] generate_cluster_yaml
  951. 2020-10-08T11:32:33.3702142Z [lighthouse]$ [cluster2] generate_cluster_yaml
  952. 2020-10-08T11:32:33.3702694Z [lighthouse]$ [cluster2] local pod_cidr=10.242.0.0/16
  953. 2020-10-08T11:32:33.3703231Z [lighthouse]$ [cluster2] local service_cidr=100.92.0.0/16
  954. 2020-10-08T11:32:33.3703837Z [lighthouse]$ [cluster2] local dns_domain=cluster2.local
  955. 2020-10-08T11:32:33.3704441Z [lighthouse]$ [cluster2] local disable_cni=false
  956. 2020-10-08T11:32:33.3704981Z [lighthouse]$ [cluster2] disable_cni=true
  957. 2020-10-08T11:32:33.3705483Z [lighthouse]$ [cluster2] local nodes
  958. 2020-10-08T11:32:33.3705965Z [lighthouse]$ [cluster2] nodes=
  959. 2020-10-08T11:32:33.3706436Z [cluster2] - role: control-plane
  960. 2020-10-08T11:32:33.3706899Z [lighthouse]$ [cluster2] nodes=
  961. 2020-10-08T11:32:33.3707340Z [cluster2] - role: control-plane
  962. 2020-10-08T11:32:33.3707770Z [cluster2] - role: worker
  963. 2020-10-08T11:32:33.3708197Z [lighthouse]$ [cluster2] nodes=
  964. 2020-10-08T11:32:33.3708652Z [cluster2] - role: control-plane
  965. 2020-10-08T11:32:33.3709167Z [cluster2] - role: worker
  966. 2020-10-08T11:32:33.3709575Z [cluster2] - role: worker
  967. 2020-10-08T11:32:33.3709938Z [cluster2] 
  968. 2020-10-08T11:32:33.3710266Z [cluster2] 
  969. 2020-10-08T11:32:33.3710687Z [lighthouse]$ [cluster2] eval echo "---
  970. 2020-10-08T11:32:33.3711007Z [cluster2] kind: Cluster
  971. 2020-10-08T11:32:33.3711499Z [cluster2] apiVersion: kind.x-k8s.io/v1alpha4
  972. 2020-10-08T11:32:33.3711871Z [cluster2] networking:
  973. 2020-10-08T11:32:33.3712264Z [cluster2] disableDefaultCNI: ${disable_cni}
  974. 2020-10-08T11:32:33.3712668Z [cluster2] podSubnet: ${pod_cidr}
  975. 2020-10-08T11:32:33.3713038Z [cluster2] serviceSubnet: ${service_cidr}
  976. 2020-10-08T11:32:33.3713492Z [cluster2] containerdConfigPatches:
  977. 2020-10-08T11:32:33.3713975Z [cluster2] - |-
  978. 2020-10-08T11:32:33.3714589Z [cluster2] [plugins.\"io.containerd.grpc.v1.cri\".registry.mirrors.\"localhost:5000\"]
  979. 2020-10-08T11:32:33.3715352Z [cluster2] endpoint = [\"http://${registry_ip}:5000\"]
  980. 2020-10-08T11:32:33.3715788Z [cluster2] kubeadmConfigPatches:
  981. 2020-10-08T11:32:33.3716253Z [cluster2] - |
  982. 2020-10-08T11:32:33.3716602Z [cluster2] apiVersion: kubeadm.k8s.io/v1beta2
  983. 2020-10-08T11:32:33.3717260Z [cluster2] kind: ClusterConfiguration
  984. 2020-10-08T11:32:33.3717692Z [cluster2] metadata:
  985. 2020-10-08T11:32:33.3718012Z [cluster2] name: config
  986. 2020-10-08T11:32:33.3718344Z [cluster2] networking:
  987. 2020-10-08T11:32:33.3718708Z [cluster2] podSubnet: ${pod_cidr}
  988. 2020-10-08T11:32:33.3719133Z [cluster2] serviceSubnet: ${service_cidr}
  989. 2020-10-08T11:32:33.3719562Z [cluster2] dnsDomain: ${dns_domain}
  990. 2020-10-08T11:32:33.3720055Z [cluster2] nodes: ${nodes}"
  991. 2020-10-08T11:32:33.3721029Z [lighthouse]$ [cluster2] cat /opt/shipyard/scripts/resources/kind-cluster-config.yaml
  992. 2020-10-08T11:32:33.3721939Z [lighthouse]$ [cluster2] local image_flag=
  993. 2020-10-08T11:32:33.3722694Z [lighthouse]$ [cluster2] image_flag=--image=kindest/node:v1.17.0
  994. 2020-10-08T11:32:33.3724639Z [lighthouse]$ [cluster2] kind create cluster --image=kindest/node:v1.17.0 --name=cluster2 --config=/opt/shipyard/scripts/resources/cluster2-config.yaml
  995. 2020-10-08T11:32:33.3725675Z [cluster2] Creating cluster "cluster2" ...
  996. 2020-10-08T11:32:33.3726446Z [cluster2] • Ensuring node image (kindest/node:v1.17.0) 🖼 ...
  997. 2020-10-08T11:32:33.3727176Z [cluster2] ✓ Ensuring node image (kindest/node:v1.17.0) 🖼
  998. 2020-10-08T11:32:33.3727810Z [cluster2] • Preparing nodes 📦 📦 📦 ...
  999. 2020-10-08T11:32:33.3728368Z [cluster2] ✓ Preparing nodes 📦 📦 📦
  1000. 2020-10-08T11:32:33.3729046Z [cluster2] • Writing configuration 📜 ...
  1001. 2020-10-08T11:32:33.3729674Z [cluster2] ✓ Writing configuration 📜
  1002. 2020-10-08T11:32:33.3730312Z [cluster2] • Starting control-plane 🕹️ ...
  1003. 2020-10-08T11:32:33.3730948Z [cluster2] ✓ Starting control-plane 🕹️
  1004. 2020-10-08T11:32:33.3731591Z [cluster2] • Installing StorageClass 💾 ...
  1005. 2020-10-08T11:32:33.3732244Z [cluster2] ✓ Installing StorageClass 💾
  1006. 2020-10-08T11:32:33.3733074Z [cluster2] • Joining worker nodes 🚜 ...
  1007. 2020-10-08T11:32:33.3733646Z [cluster2] ✓ Joining worker nodes 🚜
  1008. 2020-10-08T11:32:33.3734269Z [cluster2] Set kubectl context to "kind-cluster2"
  1009. 2020-10-08T11:32:33.3734800Z [cluster2] You can now use your cluster with:
  1010. 2020-10-08T11:32:33.3735159Z [cluster2]
  1011. 2020-10-08T11:32:33.3735817Z [cluster2] kubectl cluster-info --context kind-cluster2
  1012. 2020-10-08T11:32:33.3736272Z [cluster2]
  1013. 2020-10-08T11:32:33.3736759Z [cluster2] Thanks for using kind! 😊
  1014. 2020-10-08T11:32:33.3737367Z [lighthouse]$ [cluster2] kind_fixup_config
  1015. 2020-10-08T11:32:33.3737995Z [lighthouse]$ [cluster2] kind_fixup_config
  1016. 2020-10-08T11:32:33.3738651Z [lighthouse]$ [cluster2] local master_ip=172.17.0.6
  1017. 2020-10-08T11:32:33.3739820Z [lighthouse]$ [cluster2] docker inspect -f {{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}} cluster2-control-plane
  1018. 2020-10-08T11:33:15.9872733Z [lighthouse]$ [cluster2] head -n 1
  1019. 2020-10-08T11:33:15.9874690Z [lighthouse]$ [cluster1] sed -i -- s/server: .*/server: https:\/\/172.17.0.8:6443/g /go/src/github.com/submariner-io/lighthouse/output/kubeconfigs/kind-config-cluster1
  1020. 2020-10-08T11:33:15.9876555Z [lighthouse]$ [cluster1] sed -i -- s/user: kind-.*/user: cluster1/g /go/src/github.com/submariner-io/lighthouse/output/kubeconfigs/kind-config-cluster1
  1021. 2020-10-08T11:33:15.9878294Z [lighthouse]$ [cluster1] sed -i -- s/name: kind-.*/name: cluster1/g /go/src/github.com/submariner-io/lighthouse/output/kubeconfigs/kind-config-cluster1
  1022. 2020-10-08T11:33:15.9880050Z [lighthouse]$ [cluster1] sed -i -- s/cluster: kind-.*/cluster: cluster1/g /go/src/github.com/submariner-io/lighthouse/output/kubeconfigs/kind-config-cluster1
  1023. 2020-10-08T11:33:15.9881940Z [lighthouse]$ [cluster1] sed -i -- s/current-context: .*/current-context: cluster1/g /go/src/github.com/submariner-io/lighthouse/output/kubeconfigs/kind-config-cluster1
  1024. 2020-10-08T11:33:15.9883612Z [lighthouse]$ [cluster1] chmod a+r /go/src/github.com/submariner-io/lighthouse/output/kubeconfigs/kind-config-cluster1
  1025. 2020-10-08T11:33:15.9884587Z [lighthouse]$ [cluster1] wait 4516
  1026. 2020-10-08T11:33:15.9886413Z [lighthouse]$ [cluster1] deploy_cluster_capabilities
  1027. 2020-10-08T11:33:15.9887444Z [lighthouse]$ [cluster1] deploy_cluster_capabilities
  1028. 2020-10-08T11:33:15.9888240Z [lighthouse]$ [cluster1] deploy_cni
  1029. 2020-10-08T11:33:15.9889035Z [lighthouse]$ [cluster1] deploy_cni
  1030. 2020-10-08T11:33:15.9889631Z [lighthouse]$ [cluster1] eval deploy_weave_cni
  1031. 2020-10-08T11:33:15.9890385Z [lighthouse]$ [cluster1] deploy_weave_cni
  1032. 2020-10-08T11:33:15.9890978Z [lighthouse]$ [cluster1] deploy_weave_cni
  1033. 2020-10-08T11:33:15.9891399Z [cluster1] Applying weave network...
  1034. 2020-10-08T11:33:15.9892618Z [lighthouse]$ [cluster1] kubectl apply -f https://cloud.weave.works/k8s/net?k8s-version=v1.17.0&env.IPALLOC_RANGE=10.241.0.0/16
  1035. 2020-10-08T11:33:15.9894157Z [lighthouse]$ [cluster1] kubectl apply -f https://cloud.weave.works/k8s/net?k8s-version=v1.17.0&env.IPALLOC_RANGE=10.241.0.0/16
  1036. 2020-10-08T11:33:15.9895651Z [lighthouse]$ [cluster1] command kubectl --context=cluster1 apply -f https://cloud.weave.works/k8s/net?k8s-version=v1.17.0&env.IPALLOC_RANGE=10.241.0.0/16
  1037. 2020-10-08T11:33:15.9896940Z [cluster1] serviceaccount/weave-net created
  1038. 2020-10-08T11:33:15.9897937Z [cluster1] clusterrole.rbac.authorization.k8s.io/weave-net created
  1039. 2020-10-08T11:33:15.9899390Z [cluster1] clusterrolebinding.rbac.authorization.k8s.io/weave-net created
  1040. 2020-10-08T11:33:15.9900699Z [cluster1] role.rbac.authorization.k8s.io/weave-net created
  1041. 2020-10-08T11:33:15.9901849Z [cluster1] rolebinding.rbac.authorization.k8s.io/weave-net created
  1042. 2020-10-08T11:33:15.9902812Z [cluster1] daemonset.apps/weave-net created
  1043. 2020-10-08T11:33:15.9903461Z [cluster1] Waiting for weave-net pods to be ready...
  1044. 2020-10-08T11:33:15.9904547Z [lighthouse]$ [cluster1] kubectl wait --for=condition=Ready pods -l name=weave-net -n kube-system --timeout=5m
  1045. 2020-10-08T11:33:15.9905825Z [lighthouse]$ [cluster1] kubectl wait --for=condition=Ready pods -l name=weave-net -n kube-system --timeout=5m
  1046. 2020-10-08T11:33:15.9907093Z [lighthouse]$ [cluster1] command kubectl --context=cluster1 wait --for=condition=Ready pods -l name=weave-net -n kube-system --timeout=5m
  1047. 2020-10-08T11:33:15.9908408Z [lighthouse]$ [cluster1] kubectl --context=cluster1 wait --for=condition=Ready pods -l name=weave-net -n kube-system --timeout=5m
  1048. 2020-10-08T11:33:15.9909364Z [cluster1] pod/weave-net-dxhs2 condition met
  1049. 2020-10-08T11:33:15.9910065Z [cluster1] pod/weave-net-m9c7v condition met
  1050. 2020-10-08T11:33:15.9910761Z [cluster1] pod/weave-net-r7gjb condition met
  1051. 2020-10-08T11:33:15.9911496Z [cluster1] Waiting for core-dns deployment to be ready...
  1052. 2020-10-08T11:33:15.9912397Z [lighthouse]$ [cluster1] kubectl -n kube-system rollout status deploy/coredns --timeout=5m
  1053. 2020-10-08T11:33:15.9913518Z [lighthouse]$ [cluster1] kubectl -n kube-system rollout status deploy/coredns --timeout=5m
  1054. 2020-10-08T11:33:15.9914680Z [lighthouse]$ [cluster1] command kubectl --context=cluster1 -n kube-system rollout status deploy/coredns --timeout=5m
  1055. 2020-10-08T11:33:15.9915861Z [lighthouse]$ [cluster1] kubectl --context=cluster1 -n kube-system rollout status deploy/coredns --timeout=5m
  1056. 2020-10-08T11:33:15.9916771Z [cluster1] Waiting for deployment "coredns" rollout to finish: 0 of 2 updated replicas are available...
  1057. 2020-10-08T11:33:15.9917650Z [cluster1] Waiting for deployment "coredns" rollout to finish: 1 of 2 updated replicas are available...
  1058. 2020-10-08T11:33:15.9918393Z [cluster1] deployment "coredns" successfully rolled out
  1059. 2020-10-08T11:33:15.9919084Z [lighthouse]$ [cluster1] return 0
  1060. 2020-10-08T11:33:16.0144785Z 0m
  1061. 2020-10-08T11:33:16.0146459Z [lighthouse]$ [cluster2] sed -i -- s/server: .*/server: https:\/\/172.17.0.6:6443/g /go/src/github.com/submariner-io/lighthouse/output/kubeconfigs/kind-config-cluster2
  1062. 2020-10-08T11:33:16.0148150Z [lighthouse]$ [cluster2] sed -i -- s/user: kind-.*/user: cluster2/g /go/src/github.com/submariner-io/lighthouse/output/kubeconfigs/kind-config-cluster2
  1063. 2020-10-08T11:33:16.0149806Z [lighthouse]$ [cluster2] sed -i -- s/name: kind-.*/name: cluster2/g /go/src/github.com/submariner-io/lighthouse/output/kubeconfigs/kind-config-cluster2
  1064. 2020-10-08T11:33:16.0151639Z [lighthouse]$ [cluster2] sed -i -- s/cluster: kind-.*/cluster: cluster2/g /go/src/github.com/submariner-io/lighthouse/output/kubeconfigs/kind-config-cluster2
  1065. 2020-10-08T11:33:16.0153456Z [lighthouse]$ [cluster2] sed -i -- s/current-context: .*/current-context: cluster2/g /go/src/github.com/submariner-io/lighthouse/output/kubeconfigs/kind-config-cluster2
  1066. 2020-10-08T11:33:16.0155109Z [lighthouse]$ [cluster2] chmod a+r /go/src/github.com/submariner-io/lighthouse/output/kubeconfigs/kind-config-cluster2
  1067. 2020-10-08T11:33:16.0156200Z [lighthouse]$ [cluster2] wait 4655
  1068. 2020-10-08T11:33:16.0156896Z [lighthouse]$ [cluster2] deploy_cluster_capabilities
  1069. 2020-10-08T11:33:16.0157772Z [lighthouse]$ [cluster2] deploy_cluster_capabilities
  1070. 2020-10-08T11:33:16.0158677Z [lighthouse]$ [cluster2] deploy_cni
  1071. 2020-10-08T11:33:16.0160162Z [lighthouse]$ [cluster2] deploy_cni
  1072. 2020-10-08T11:33:16.0160803Z [lighthouse]$ [cluster2] eval deploy_weave_cni
  1073. 2020-10-08T11:33:16.0161455Z [lighthouse]$ [cluster2] deploy_weave_cni
  1074. 2020-10-08T11:33:16.0162087Z [lighthouse]$ [cluster2] deploy_weave_cni
  1075. 2020-10-08T11:33:16.0162560Z [cluster2] Applying weave network...
  1076. 2020-10-08T11:33:16.0163754Z [lighthouse]$ [cluster2] kubectl apply -f https://cloud.weave.works/k8s/net?k8s-version=v1.17.0&env.IPALLOC_RANGE=10.242.0.0/16
  1077. 2020-10-08T11:33:16.0165379Z [lighthouse]$ [cluster2] kubectl apply -f https://cloud.weave.works/k8s/net?k8s-version=v1.17.0&env.IPALLOC_RANGE=10.242.0.0/16
  1078. 2020-10-08T11:33:16.0166969Z [lighthouse]$ [cluster2] command kubectl --context=cluster2 apply -f https://cloud.weave.works/k8s/net?k8s-version=v1.17.0&env.IPALLOC_RANGE=10.242.0.0/16
  1079. 2020-10-08T11:33:16.8918849Z [lighthouse]$ [cluster2] wait 3406
  1080. 2020-10-08T11:33:16.8919554Z [lighthouse]$ [cluster2] print_clusters_message
  1081. 2020-10-08T11:33:16.8920274Z [lighthouse]$ [cluster2] print_clusters_message
  1082. 2020-10-08T11:33:16.8920896Z [lighthouse]$ [cluster2] cat
  1083. 2020-10-08T11:33:16.8921568Z [lighthouse]$ source /opt/shipyard/scripts/lib/deploy_funcs
  1084. 2020-10-08T11:33:16.8922319Z [lighthouse]$ . /opt/shipyard/scripts/lib/source_only
  1085. 2020-10-08T11:33:16.8922978Z [lighthouse]$ script_name=deploy_funcs
  1086. 2020-10-08T11:33:16.8923557Z [lighthouse]$ exec_name=deploy
  1087. 2020-10-08T11:33:16.8924208Z [lighthouse]$ source /opt/shipyard/scripts/lib/utils
  1088. 2020-10-08T11:33:16.8924928Z [lighthouse]$ . /opt/shipyard/scripts/lib/source_only
  1089. 2020-10-08T11:33:16.8925557Z [lighthouse]$ script_name=utils
  1090. 2020-10-08T11:33:16.8926395Z [lighthouse]$ exec_name=deploy
  1091. 2020-10-08T11:33:16.8927089Z [lighthouse]$ source /opt/shipyard/scripts/lib/version
  1092. 2020-10-08T11:33:16.8927811Z [lighthouse]$ . /opt/shipyard/scripts/lib/source_only
  1093. 2020-10-08T11:33:16.8928453Z [lighthouse]$ script_name=version
  1094. 2020-10-08T11:33:16.8929019Z [lighthouse]$ exec_name=deploy
  1095. 2020-10-08T11:33:16.8929757Z [lighthouse]$ git describe --tags --dirty=-dev --exclude=devel --exclude=latest
  1096. 2020-10-08T11:33:16.8930368Z fatal: No names found, cannot describe anything.
  1097. 2020-10-08T11:33:16.8931004Z [lighthouse]$ declare_kubeconfig
  1098. 2020-10-08T11:33:16.8931613Z [lighthouse]$ declare_kubeconfig
  1099. 2020-10-08T11:33:16.8932303Z [lighthouse]$ source /opt/shipyard/scripts/lib/kubecfg
  1100. 2020-10-08T11:33:16.8933350Z [lighthouse]$ export KUBECONFIG
  1101. 2020-10-08T11:33:16.8934887Z [lighthouse]$ KUBECONFIG=/go/src/github.com/submariner-io/lighthouse/output/kubeconfigs/kind-config-cluster2:/go/src/github.com/submariner-io/lighthouse/output/kubeconfigs/kind-config-cluster1:
  1102. 2020-10-08T11:33:16.8936575Z [lighthouse]$ find /go/src/github.com/submariner-io/lighthouse/output/kubeconfigs -type f -printf %p:
  1103. 2020-10-08T11:33:16.8937607Z [lighthouse]$ import_image quay.io/submariner/lighthouse-agent
  1104. 2020-10-08T11:33:16.8938533Z [lighthouse]$ import_image quay.io/submariner/lighthouse-agent
  1105. 2020-10-08T11:33:16.8939461Z [lighthouse]$ local orig_image=quay.io/submariner/lighthouse-agent
  1106. 2020-10-08T11:33:16.8940487Z [lighthouse]$ local versioned_image=quay.io/submariner/lighthouse-agent:dev
  1107. 2020-10-08T11:33:16.8941572Z [lighthouse]$ local local_image=localhost:5000/lighthouse-agent:local
  1108. 2020-10-08T11:33:16.8942609Z [lighthouse]$ docker tag quay.io/submariner/lighthouse-agent:dev localhost:5000/lighthouse-agent:local
  1109. 2020-10-08T11:33:16.8943595Z [lighthouse]$ docker push localhost:5000/lighthouse-agent:local
  1110. 2020-10-08T11:33:16.8944333Z [cluster2] serviceaccount/weave-net created
  1111. 2020-10-08T11:33:16.8945556Z [cluster2] clusterrole.rbac.authorization.k8s.io/weave-net created
  1112. 2020-10-08T11:33:16.8947344Z [cluster2] clusterrolebinding.rbac.authorization.k8s.io/weave-net created
  1113. 2020-10-08T11:33:16.8948746Z [cluster2] role.rbac.authorization.k8s.io/weave-net created
  1114. 2020-10-08T11:33:16.8949971Z [cluster2] rolebinding.rbac.authorization.k8s.io/weave-net created
  1115. 2020-10-08T11:33:16.8951001Z [cluster2] daemonset.apps/weave-net created
  1116. 2020-10-08T11:33:16.8951706Z [cluster2] Waiting for weave-net pods to be ready...
  1117. 2020-10-08T11:33:16.8952652Z [lighthouse]$ [cluster2] kubectl wait --for=condition=Ready pods -l name=weave-net -n kube-system --timeout=5m
  1118. 2020-10-08T11:33:16.8953795Z [lighthouse]$ [cluster2] kubectl wait --for=condition=Ready pods -l name=weave-net -n kube-system --timeout=5m
  1119. 2020-10-08T11:33:16.8955059Z [lighthouse]$ [cluster2] command kubectl --context=cluster2 wait --for=condition=Ready pods -l name=weave-net -n kube-system --timeout=5m
  1120. 2020-10-08T11:33:16.8956375Z [lighthouse]$ [cluster2] kubectl --context=cluster2 wait --for=condition=Ready pods -l name=weave-net -n kube-system --timeout=5m
  1121. 2020-10-08T11:33:16.8957323Z [cluster2] pod/weave-net-6995p condition met
  1122. 2020-10-08T11:33:16.8958018Z [cluster2] pod/weave-net-ffcfd condition met
  1123. 2020-10-08T11:33:16.8958719Z [cluster2] pod/weave-net-r6qpq condition met
  1124. 2020-10-08T11:33:16.8959443Z [cluster2] Waiting for core-dns deployment to be ready...
  1125. 2020-10-08T11:33:16.8960323Z [lighthouse]$ [cluster2] kubectl -n kube-system rollout status deploy/coredns --timeout=5m
  1126. 2020-10-08T11:33:16.8961343Z [lighthouse]$ [cluster2] kubectl -n kube-system rollout status deploy/coredns --timeout=5m
  1127. 2020-10-08T11:33:16.8962479Z [lighthouse]$ [cluster2] command kubectl --context=cluster2 -n kube-system rollout status deploy/coredns --timeout=5m
  1128. 2020-10-08T11:33:16.8963743Z [lighthouse]$ [cluster2] kubectl --context=cluster2 -n kube-system rollout status deploy/coredns --timeout=5m
  1129. 2020-10-08T11:33:16.8964897Z [cluster2] Waiting for deployment "coredns" rollout to finish: 0 of 2 updated replicas are available...
  1130. 2020-10-08T11:33:16.8965781Z [cluster2] Waiting for deployment "coredns" rollout to finish: 1 of 2 updated replicas are available...
  1131. 2020-10-08T11:33:16.8966519Z [cluster2] deployment "coredns" successfully rolled out
  1132. 2020-10-08T11:33:16.8967263Z [lighthouse]$ [cluster2] return 0
  1133. 2020-10-08T11:33:16.8967832Z Your virtual cluster(s) are deployed and working properly and can be accessed with:
  1134. 2020-10-08T11:33:16.8968240Z
  1135. 2020-10-08T11:33:16.8968979Z export KUBECONFIG=$(find $(git rev-parse --show-toplevel)/output/kubeconfigs/ -type f -printf %p:)
  1136. 2020-10-08T11:33:16.8969472Z
  1137. 2020-10-08T11:33:16.8970167Z $ kubectl config use-context cluster1 # or cluster2, cluster3..
  1138. 2020-10-08T11:33:16.8970525Z
  1139. 2020-10-08T11:33:16.8970839Z To clean everthing up, just run: make cleanup
  1140. 2020-10-08T11:33:16.8979702Z ./scripts/deploy --deploytool helm --cluster_settings /go/src/github.com/submariner-io/lighthouse/scripts/cluster_settings --deploytool_broker_args '--set submariner.serviceDiscovery=true' --deploytool_submariner_args '--set submariner.serviceDiscovery=true,lighthouse.image.repository=localhost:5000/lighthouse-agent,lighthouse.image.tag=local,lighthouseCoredns.image.repository=localhost:5000/lighthouse-coredns,lighthouseCoredns.image.tag=local,serviceAccounts.lighthouse.create=true'
  1141. 2020-10-08T11:33:16.8984263Z The push refers to repository [localhost:5000/lighthouse-agent]
  1142. 2020-10-08T11:33:16.9285923Z b7a0a6f02c22: Preparing
  1143. 2020-10-08T11:33:16.9286449Z 4e5767963082: Preparing
  1144. 2020-10-08T11:33:16.9286776Z 3485805ce47c: Preparing
  1145. 2020-10-08T11:33:16.9287199Z b0e2911c99f3: Preparing
  1146. 2020-10-08T11:33:17.0762802Z 3485805ce47c: Pushed
  1147. 2020-10-08T11:33:17.0763185Z 4e5767963082: Pushed
  1148. 2020-10-08T11:33:17.5560538Z b7a0a6f02c22: Pushed
  1149. 2020-10-08T11:33:27.6438844Z b0e2911c99f3: Pushed
  1150. 2020-10-08T11:33:27.6597855Z local: digest: sha256:1dbecc4760d06cf80fe14eba64d7bd52c284ebd99130ec801f03d4a586245933 size: 1155
  1151. 2020-10-08T11:33:27.6657926Z [lighthouse]$ import_image quay.io/submariner/lighthouse-coredns
  1152. 2020-10-08T11:33:27.6665170Z [lighthouse]$ import_image quay.io/submariner/lighthouse-coredns
  1153. 2020-10-08T11:33:27.6683714Z [lighthouse]$ local orig_image=quay.io/submariner/lighthouse-coredns
  1154. 2020-10-08T11:33:27.6687075Z [lighthouse]$ local versioned_image=quay.io/submariner/lighthouse-coredns:dev
  1155. 2020-10-08T11:33:27.6704188Z [lighthouse]$ local local_image=localhost:5000/lighthouse-coredns:local
  1156. 2020-10-08T11:33:27.6706809Z [lighthouse]$ docker tag quay.io/submariner/lighthouse-coredns:dev localhost:5000/lighthouse-coredns:local
  1157. 2020-10-08T11:33:28.0247050Z [lighthouse]$ docker push localhost:5000/lighthouse-coredns:local
  1158. 2020-10-08T11:33:28.3671724Z The push refers to repository [localhost:5000/lighthouse-coredns]
  1159. 2020-10-08T11:33:28.3729561Z 62dc579b5c53: Preparing
  1160. 2020-10-08T11:33:28.3730030Z 3c2e9ba25480: Preparing
  1161. 2020-10-08T11:33:28.3730347Z 69215f34c115: Preparing
  1162. 2020-10-08T11:33:30.2598201Z 62dc579b5c53: Pushed
  1163. 2020-10-08T11:33:30.2673405Z 3c2e9ba25480: Pushed
  1164. 2020-10-08T11:33:34.8208726Z 69215f34c115: Pushed
  1165. 2020-10-08T11:33:34.8374347Z local: digest: sha256:596c14dfebe9da4398034185ee0d6284ec0243d030a7cc6ffb6443b5e14cc787 size: 953
  1166. 2020-10-08T11:33:34.8434856Z [lighthouse]$ /opt/shipyard/scripts/deploy.sh --deploytool helm --cluster_settings /go/src/github.com/submariner-io/lighthouse/scripts/cluster_settings --deploytool_broker_args --set submariner.serviceDiscovery=true --deploytool_submariner_args --set submariner.serviceDiscovery=true,lighthouse.image.repository=localhost:5000/lighthouse-agent,lighthouse.image.tag=local,lighthouseCoredns.image.repository=localhost:5000/lighthouse-coredns,lighthouseCoredns.image.tag=local,serviceAccounts.lighthouse.create=true
  1167. 2020-10-08T11:33:34.9354983Z Running with: globalnet='false', deploytool='helm', deploytool_broker_args='--set submariner.serviceDiscovery=true', deploytool_submariner_args='--set submariner.serviceDiscovery=true,lighthouse.image.repository=localhost:5000/lighthouse-agent,lighthouse.image.tag=local,lighthouseCoredns.image.repository=localhost:5000/lighthouse-coredns,lighthouseCoredns.image.tag=local,serviceAccounts.lighthouse.create=true', cluster_settings='/go/src/github.com/submariner-io/lighthouse/scripts/cluster_settings', timeout=5m
  1168. 2020-10-08T11:33:34.9375559Z [lighthouse]$ source /opt/shipyard/scripts/lib/version
  1169. 2020-10-08T11:33:34.9385886Z [lighthouse]$ . /opt/shipyard/scripts/lib/source_only
  1170. 2020-10-08T11:33:34.9398075Z [lighthouse]$ script_name=version
  1171. 2020-10-08T11:33:34.9406242Z [lighthouse]$ exec_name=deploy.sh
  1172. 2020-10-08T11:33:34.9427573Z [lighthouse]$ git describe --tags --dirty=-dev --exclude=devel --exclude=latest
  1173. 2020-10-08T11:33:34.9859744Z fatal: No names found, cannot describe anything.
  1174. 2020-10-08T11:33:34.9879919Z [lighthouse]$ source /opt/shipyard/scripts/lib/utils
  1175. 2020-10-08T11:33:34.9891172Z [lighthouse]$ . /opt/shipyard/scripts/lib/source_only
  1176. 2020-10-08T11:33:34.9902829Z [lighthouse]$ script_name=utils
  1177. 2020-10-08T11:33:34.9914734Z [lighthouse]$ exec_name=deploy.sh
  1178. 2020-10-08T11:33:34.9933706Z [lighthouse]$ source /opt/shipyard/scripts/lib/deploy_funcs
  1179. 2020-10-08T11:33:34.9945379Z [lighthouse]$ . /opt/shipyard/scripts/lib/source_only
  1180. 2020-10-08T11:33:34.9957997Z [lighthouse]$ script_name=deploy_funcs
  1181. 2020-10-08T11:33:34.9968997Z [lighthouse]$ exec_name=deploy.sh
  1182. 2020-10-08T11:33:34.9987195Z [lighthouse]$ source /opt/shipyard/scripts/lib/cluster_settings
  1183. 2020-10-08T11:33:35.0003145Z [lighthouse]$ . /opt/shipyard/scripts/lib/source_only
  1184. 2020-10-08T11:33:35.0016271Z [lighthouse]$ script_name=cluster_settings
  1185. 2020-10-08T11:33:35.0027651Z [lighthouse]$ exec_name=deploy.sh
  1186. 2020-10-08T11:33:35.0049086Z [lighthouse]$ broker=cluster1
  1187. 2020-10-08T11:33:35.0060846Z [lighthouse]$ declare -gA cluster_nodes
  1188. 2020-10-08T11:33:35.0071594Z [lighthouse]$ cluster_nodes[cluster1]=control-plane worker
  1189. 2020-10-08T11:33:35.0085067Z [lighthouse]$ cluster_nodes[cluster2]=control-plane worker
  1190. 2020-10-08T11:33:35.0103866Z [lighthouse]$ cluster_nodes[cluster3]=control-plane worker worker
  1191. 2020-10-08T11:33:35.0116554Z [lighthouse]$ declare -gA cluster_subm
  1192. 2020-10-08T11:33:35.0126724Z [lighthouse]$ cluster_subm[cluster1]=true
  1193. 2020-10-08T11:33:35.0138383Z [lighthouse]$ cluster_subm[cluster2]=true
  1194. 2020-10-08T11:33:35.0148515Z [lighthouse]$ cluster_subm[cluster3]=true
  1195. 2020-10-08T11:33:35.0167199Z [lighthouse]$ declare -gA cluster_cni
  1196. 2020-10-08T11:33:35.0178279Z [lighthouse]$ cluster_cni[cluster2]=weave
  1197. 2020-10-08T11:33:35.0191765Z [lighthouse]$ cluster_cni[cluster3]=weave
  1198. 2020-10-08T11:33:35.0203921Z [lighthouse]$ source /go/src/github.com/submariner-io/lighthouse/scripts/cluster_settings
  1199. 2020-10-08T11:33:35.0215655Z [lighthouse]$ . /opt/shipyard/scripts/lib/source_only
  1200. 2020-10-08T11:33:35.0226480Z [lighthouse]$ script_name=cluster_settings
  1201. 2020-10-08T11:33:35.0239228Z [lighthouse]$ exec_name=deploy.sh
  1202. 2020-10-08T11:33:35.0260845Z [lighthouse]$ cluster_nodes[cluster1]=control-plane worker worker
  1203. 2020-10-08T11:33:35.0271577Z [lighthouse]$ cluster_nodes[cluster2]=control-plane worker worker
  1204. 2020-10-08T11:33:35.0300326Z [lighthouse]$ declare_cidrs
  1205. 2020-10-08T11:33:35.0312596Z [lighthouse]$ declare_cidrs
  1206. 2020-10-08T11:33:35.0323211Z [lighthouse]$ declare -gA cluster_CIDRs service_CIDRs global_CIDRs
  1207. 2020-10-08T11:33:35.0335554Z [lighthouse]$ i=1
  1208. 2020-10-08T11:33:35.0347284Z [lighthouse]$ [cluster1] add_cluster_cidrs 1 cluster1
  1209. 2020-10-08T11:33:35.0359885Z [lighthouse]$ [cluster1] add_cluster_cidrs 1 cluster1
  1210. 2020-10-08T11:33:35.0370781Z [lighthouse]$ [cluster1] local val=1
  1211. 2020-10-08T11:33:35.0383217Z [lighthouse]$ [cluster1] local idx=cluster1
  1212. 2020-10-08T11:33:35.0396003Z [lighthouse]$ [cluster1] cluster_CIDRs[cluster1]=10.241.0.0/16
  1213. 2020-10-08T11:33:35.0406680Z [lighthouse]$ [cluster1] service_CIDRs[cluster1]=100.91.0.0/16
  1214. 2020-10-08T11:33:35.0419835Z [lighthouse]$ [cluster1] i=2
  1215. 2020-10-08T11:33:35.0433919Z [lighthouse]$ [cluster2] add_cluster_cidrs 2 cluster2
  1216. 2020-10-08T11:33:35.0445098Z [lighthouse]$ [cluster2] add_cluster_cidrs 2 cluster2
  1217. 2020-10-08T11:33:35.0457100Z [lighthouse]$ [cluster2] local val=2
  1218. 2020-10-08T11:33:35.8954568Z [lighthouse]$ [cluster2] local idx=cluster2
  1219. 2020-10-08T11:33:35.8955333Z [lighthouse]$ [cluster2] cluster_CIDRs[cluster2]=10.242.0.0/16
  1220. 2020-10-08T11:33:35.8956084Z [lighthouse]$ [cluster2] service_CIDRs[cluster2]=100.92.0.0/16
  1221. 2020-10-08T11:33:35.8956692Z [lighthouse]$ [cluster2] i=3
  1222. 2020-10-08T11:33:35.8957337Z [lighthouse]$ [cluster2] declare_kubeconfig
  1223. 2020-10-08T11:33:35.8958031Z [lighthouse]$ [cluster2] declare_kubeconfig
  1224. 2020-10-08T11:33:35.8958813Z [lighthouse]$ [cluster2] source /opt/shipyard/scripts/lib/kubecfg
  1225. 2020-10-08T11:33:35.8959561Z [lighthouse]$ [cluster2] export KUBECONFIG
  1226. 2020-10-08T11:33:35.8961165Z [lighthouse]$ [cluster2] KUBECONFIG=/go/src/github.com/submariner-io/lighthouse/output/kubeconfigs/kind-config-cluster2:/go/src/github.com/submariner-io/lighthouse/output/kubeconfigs/kind-config-cluster1:
  1227. 2020-10-08T11:33:35.8962918Z [lighthouse]$ [cluster2] find /go/src/github.com/submariner-io/lighthouse/output/kubeconfigs -type f -printf %p:
  1228. 2020-10-08T11:33:35.8963931Z [lighthouse]$ [cluster2] import_image quay.io/submariner/nettest
  1229. 2020-10-08T11:33:35.8964776Z [lighthouse]$ [cluster2] import_image quay.io/submariner/nettest
  1230. 2020-10-08T11:33:35.8965626Z [lighthouse]$ [cluster2] local orig_image=quay.io/submariner/nettest
  1231. 2020-10-08T11:33:35.8966814Z [lighthouse]$ [cluster2] local versioned_image=quay.io/submariner/nettest:dev
  1232. 2020-10-08T11:33:35.8967810Z [lighthouse]$ [cluster2] local local_image=localhost:5000/nettest:local
  1233. 2020-10-08T11:33:35.8969118Z [lighthouse]$ [cluster2] docker tag quay.io/submariner/nettest:dev localhost:5000/nettest:local
  1234. 2020-10-08T11:33:35.8969956Z Error response from daemon: No such image: quay.io/submariner/nettest:dev
  1235. 2020-10-08T11:33:35.8970762Z [lighthouse]$ [cluster2] docker pull quay.io/submariner/nettest:devel
  1236. 2020-10-08T11:33:35.9722620Z devel: Pulling from submariner/nettest
  1237. 2020-10-08T11:33:35.9725405Z df20fa9351a1: Pulling fs layer
  1238. 2020-10-08T11:33:35.9725809Z 741138c6c583: Pulling fs layer
  1239. 2020-10-08T11:33:35.9726154Z d34ea735672c: Pulling fs layer
  1240. 2020-10-08T11:33:35.9726504Z c91acd455ec9: Pulling fs layer
  1241. 2020-10-08T11:33:35.9729036Z c91acd455ec9: Waiting
  1242. 2020-10-08T11:33:36.0390988Z 741138c6c583: Verifying Checksum
  1243. 2020-10-08T11:33:36.0394489Z 741138c6c583: Download complete
  1244. 2020-10-08T11:33:36.0555249Z df20fa9351a1: Verifying Checksum
  1245. 2020-10-08T11:33:36.0556501Z df20fa9351a1: Download complete
  1246. 2020-10-08T11:33:36.0819371Z d34ea735672c: Verifying Checksum
  1247. 2020-10-08T11:33:36.0819883Z d34ea735672c: Download complete
  1248. 2020-10-08T11:33:36.1241175Z c91acd455ec9: Verifying Checksum
  1249. 2020-10-08T11:33:36.1241697Z c91acd455ec9: Download complete
  1250. 2020-10-08T11:33:36.5233019Z df20fa9351a1: Pull complete
  1251. 2020-10-08T11:33:36.5933051Z 741138c6c583: Pull complete
  1252. 2020-10-08T11:33:36.8782987Z d34ea735672c: Pull complete
  1253. 2020-10-08T11:33:36.9569491Z c91acd455ec9: Pull complete
  1254. 2020-10-08T11:33:36.9598921Z Digest: sha256:63f21b9d8065d60870c2bca980dbc4dc6358edf669661576b8cbee39e3024a2d
  1255. 2020-10-08T11:33:36.9664107Z Status: Downloaded newer image for quay.io/submariner/nettest:devel
  1256. 2020-10-08T11:33:36.9711728Z quay.io/submariner/nettest:devel
  1257. 2020-10-08T11:33:36.9716985Z [lighthouse]$ [cluster2] docker tag quay.io/submariner/nettest:devel quay.io/submariner/nettest:dev
  1258. 2020-10-08T11:33:37.3193582Z [lighthouse]$ [cluster2] docker tag quay.io/submariner/nettest:dev localhost:5000/nettest:local
  1259. 2020-10-08T11:33:37.6673632Z [lighthouse]$ [cluster2] docker push localhost:5000/nettest:local
  1260. 2020-10-08T11:33:38.0148472Z The push refers to repository [localhost:5000/nettest]
  1261. 2020-10-08T11:33:38.0196354Z 7497bfb0ae54: Preparing
  1262. 2020-10-08T11:33:38.0196905Z 17de069c349e: Preparing
  1263. 2020-10-08T11:33:38.0197214Z b85398c91342: Preparing
  1264. 2020-10-08T11:33:38.0197514Z 50644c29ef5a: Preparing
  1265. 2020-10-08T11:33:38.2573287Z b85398c91342: Pushed
  1266. 2020-10-08T11:33:38.6562275Z 7497bfb0ae54: Pushed
  1267. 2020-10-08T11:33:38.6608200Z 50644c29ef5a: Pushed
  1268. 2020-10-08T11:33:39.8244726Z 17de069c349e: Pushed
  1269. 2020-10-08T11:33:39.8442693Z local: digest: sha256:63f21b9d8065d60870c2bca980dbc4dc6358edf669661576b8cbee39e3024a2d size: 1155
  1270. 2020-10-08T11:33:39.8500332Z [lighthouse]$ [cluster2] import_image quay.io/submariner/submariner
  1271. 2020-10-08T11:33:39.8514859Z [lighthouse]$ [cluster2] import_image quay.io/submariner/submariner
  1272. 2020-10-08T11:33:39.8528251Z [lighthouse]$ [cluster2] local orig_image=quay.io/submariner/submariner
  1273. 2020-10-08T11:33:39.8540229Z [lighthouse]$ [cluster2] local versioned_image=quay.io/submariner/submariner:dev
  1274. 2020-10-08T11:33:39.8550317Z [lighthouse]$ [cluster2] local local_image=localhost:5000/submariner:local
  1275. 2020-10-08T11:33:39.8563359Z [lighthouse]$ [cluster2] docker tag quay.io/submariner/submariner:dev localhost:5000/submariner:local
  1276. 2020-10-08T11:33:40.1994134Z Error response from daemon: No such image: quay.io/submariner/submariner:dev
  1277. 2020-10-08T11:33:40.2038839Z [lighthouse]$ [cluster2] docker pull quay.io/submariner/submariner:devel
  1278. 2020-10-08T11:33:40.6563045Z devel: Pulling from submariner/submariner
  1279. 2020-10-08T11:33:40.6573408Z bbca93d6c76b: Already exists
  1280. 2020-10-08T11:33:40.6594955Z d9c1910875e2: Pulling fs layer
  1281. 2020-10-08T11:33:40.6595397Z 8db3c5475ffa: Pulling fs layer
  1282. 2020-10-08T11:33:40.6595787Z c8120d2c0056: Pulling fs layer
  1283. 2020-10-08T11:33:40.6596135Z 09a9602a21d9: Pulling fs layer
  1284. 2020-10-08T11:33:40.6601009Z 09a9602a21d9: Waiting
  1285. 2020-10-08T11:33:40.7176239Z d9c1910875e2: Verifying Checksum
  1286. 2020-10-08T11:33:40.7176710Z d9c1910875e2: Download complete
  1287. 2020-10-08T11:33:40.8483318Z 09a9602a21d9: Verifying Checksum
  1288. 2020-10-08T11:33:40.8484209Z 09a9602a21d9: Download complete
  1289. 2020-10-08T11:33:40.8537416Z c8120d2c0056: Verifying Checksum
  1290. 2020-10-08T11:33:40.8537916Z c8120d2c0056: Download complete
  1291. 2020-10-08T11:33:40.8584417Z d9c1910875e2: Pull complete
  1292. 2020-10-08T11:33:40.8739675Z 8db3c5475ffa: Verifying Checksum
  1293. 2020-10-08T11:33:40.8740116Z 8db3c5475ffa: Download complete
  1294. 2020-10-08T11:33:41.4202809Z 8db3c5475ffa: Pull complete
  1295. 2020-10-08T11:33:41.5740094Z c8120d2c0056: Pull complete
  1296. 2020-10-08T11:33:41.6372206Z 09a9602a21d9: Pull complete
  1297. 2020-10-08T11:33:41.6404579Z Digest: sha256:cac9e1cce953daead3e41dee69fb7042b607e2a3bee97e8bc62ed20b876e045e
  1298. 2020-10-08T11:33:41.6418747Z Status: Downloaded newer image for quay.io/submariner/submariner:devel
  1299. 2020-10-08T11:33:41.6467921Z [lighthouse]$ [cluster2] docker tag quay.io/submariner/submariner:devel quay.io/submariner/submariner:dev
  1300. 2020-10-08T11:33:41.6469078Z quay.io/submariner/submariner:devel
  1301. 2020-10-08T11:33:42.0145992Z [lighthouse]$ [cluster2] docker tag quay.io/submariner/submariner:dev localhost:5000/submariner:local
  1302. 2020-10-08T11:33:42.3603967Z [lighthouse]$ [cluster2] docker push localhost:5000/submariner:local
  1303. 2020-10-08T11:33:42.7101364Z The push refers to repository [localhost:5000/submariner]
  1304. 2020-10-08T11:33:42.7210189Z e2d9dd84e788: Preparing
  1305. 2020-10-08T11:33:42.7210661Z 017a01bf7efd: Preparing
  1306. 2020-10-08T11:33:42.7211023Z dc8ba3df3f12: Preparing
  1307. 2020-10-08T11:33:42.7211377Z c48026e1f9ff: Preparing
  1308. 2020-10-08T11:33:42.7211717Z fc9509c19a6c: Preparing
  1309. 2020-10-08T11:33:42.8211529Z c48026e1f9ff: Pushed
  1310. 2020-10-08T11:33:44.4574131Z e2d9dd84e788: Pushed
  1311. 2020-10-08T11:33:44.8060716Z 017a01bf7efd: Pushed
  1312. 2020-10-08T11:33:45.8030293Z dc8ba3df3f12: Pushed
  1313. 2020-10-08T11:33:58.6423942Z fc9509c19a6c: Pushed
  1314. 2020-10-08T11:33:58.6593932Z local: digest: sha256:cac9e1cce953daead3e41dee69fb7042b607e2a3bee97e8bc62ed20b876e045e size: 1366
  1315. 2020-10-08T11:33:58.6647768Z [lighthouse]$ [cluster2] import_image quay.io/submariner/submariner-route-agent
  1316. 2020-10-08T11:33:58.6663581Z [lighthouse]$ [cluster2] import_image quay.io/submariner/submariner-route-agent
  1317. 2020-10-08T11:33:58.6664772Z [lighthouse]$ [cluster2] local orig_image=quay.io/submariner/submariner-route-agent
  1318. 2020-10-08T11:33:58.6674663Z [lighthouse]$ [cluster2] local versioned_image=quay.io/submariner/submariner-route-agent:dev
  1319. 2020-10-08T11:33:58.6686103Z [lighthouse]$ [cluster2] local local_image=localhost:5000/submariner-route-agent:local
  1320. 2020-10-08T11:33:58.6698401Z [lighthouse]$ [cluster2] docker tag quay.io/submariner/submariner-route-agent:dev localhost:5000/submariner-route-agent:local
  1321. 2020-10-08T11:33:58.9979131Z Error response from daemon: No such image: quay.io/submariner/submariner-route-agent:dev
  1322. 2020-10-08T11:33:59.0024818Z [lighthouse]$ [cluster2] docker pull quay.io/submariner/submariner-route-agent:devel
  1323. 2020-10-08T11:33:59.4757651Z devel: Pulling from submariner/submariner-route-agent
  1324. 2020-10-08T11:33:59.4758290Z 0fd3b5213a9b: Already exists
  1325. 2020-10-08T11:33:59.4772281Z aebb8c556853: Already exists
  1326. 2020-10-08T11:33:59.4798991Z a10da1370ba4: Pulling fs layer
  1327. 2020-10-08T11:33:59.4799573Z 704148a09378: Pulling fs layer
  1328. 2020-10-08T11:33:59.4799986Z cf840e2779e0: Pulling fs layer
  1329. 2020-10-08T11:33:59.4800409Z e5192a56e366: Pulling fs layer
  1330. 2020-10-08T11:33:59.4809298Z e5192a56e366: Waiting
  1331. 2020-10-08T11:33:59.5809253Z a10da1370ba4: Verifying Checksum
  1332. 2020-10-08T11:33:59.5811031Z a10da1370ba4: Download complete
  1333. 2020-10-08T11:33:59.6055781Z 704148a09378: Verifying Checksum
  1334. 2020-10-08T11:33:59.6056224Z 704148a09378: Download complete
  1335. 2020-10-08T11:33:59.6344993Z cf840e2779e0: Verifying Checksum
  1336. 2020-10-08T11:33:59.6345558Z cf840e2779e0: Download complete
  1337. 2020-10-08T11:33:59.6430209Z e5192a56e366: Verifying Checksum
  1338. 2020-10-08T11:33:59.6430694Z e5192a56e366: Download complete
  1339. 2020-10-08T11:33:59.6661905Z a10da1370ba4: Pull complete
  1340. 2020-10-08T11:33:59.9159270Z 704148a09378: Pull complete
  1341. 2020-10-08T11:34:00.1096104Z cf840e2779e0: Pull complete
  1342. 2020-10-08T11:34:00.1735044Z e5192a56e366: Pull complete
  1343. 2020-10-08T11:34:00.1767258Z Digest: sha256:e7d24a573defc8b532890afadd6218fd8b6b159d913628aa231bbccf4f274f6b
  1344. 2020-10-08T11:34:00.1785254Z Status: Downloaded newer image for quay.io/submariner/submariner-route-agent:devel
  1345. 2020-10-08T11:34:00.1829124Z quay.io/submariner/submariner-route-agent:devel
  1346. 2020-10-08T11:34:00.1874439Z [lighthouse]$ [cluster2] docker tag quay.io/submariner/submariner-route-agent:devel quay.io/submariner/submariner-route-agent:dev
  1347. 2020-10-08T11:34:00.5205842Z [lighthouse]$ [cluster2] docker tag quay.io/submariner/submariner-route-agent:dev localhost:5000/submariner-route-agent:local
  1348. 2020-10-08T11:34:00.8521933Z [lighthouse]$ [cluster2] docker push localhost:5000/submariner-route-agent:local
  1349. 2020-10-08T11:34:01.1849664Z The push refers to repository [localhost:5000/submariner-route-agent]
  1350. 2020-10-08T11:34:01.1908741Z da24f2347902: Preparing
  1351. 2020-10-08T11:34:01.1909197Z 2b98654c6b45: Preparing
  1352. 2020-10-08T11:34:01.1909528Z 74b4a0049f0b: Preparing
  1353. 2020-10-08T11:34:01.1909857Z 22d731db4600: Preparing
  1354. 2020-10-08T11:34:01.1910158Z 3485805ce47c: Preparing
  1355. 2020-10-08T11:34:01.1913316Z b0e2911c99f3: Preparing
  1356. 2020-10-08T11:34:01.1913785Z b0e2911c99f3: Waiting
  1357. 2020-10-08T11:34:01.4201284Z 3485805ce47c: Mounted from lighthouse-agent
  1358. 2020-10-08T11:34:01.7166245Z 22d731db4600: Pushed
  1359. 2020-10-08T11:34:01.9187297Z 2b98654c6b45: Pushed
  1360. 2020-10-08T11:34:01.9191355Z da24f2347902: Pushed
  1361. 2020-10-08T11:34:01.9250173Z b0e2911c99f3: Mounted from lighthouse-agent
  1362. 2020-10-08T11:34:02.1175450Z 74b4a0049f0b: Pushed
  1363. 2020-10-08T11:34:02.1374054Z local: digest: sha256:e7d24a573defc8b532890afadd6218fd8b6b159d913628aa231bbccf4f274f6b size: 1573
  1364. 2020-10-08T11:34:02.1422032Z [lighthouse]$ [cluster2] load_deploytool helm
  1365. 2020-10-08T11:34:02.1434162Z [lighthouse]$ [cluster2] load_deploytool helm
  1366. 2020-10-08T11:34:02.1448173Z [lighthouse]$ [cluster2] local deploytool=helm
  1367. 2020-10-08T11:34:02.1461624Z [lighthouse]$ [cluster2] local deploy_lib=/opt/shipyard/scripts/lib/deploy_helm
  1368. 2020-10-08T11:34:02.1466967Z Will deploy submariner using helm
  1369. 2020-10-08T11:34:02.1480018Z [lighthouse]$ [cluster2] . /opt/shipyard/scripts/lib/deploy_helm
  1370. 2020-10-08T11:34:02.1494869Z [lighthouse]$ [cluster2] . /opt/shipyard/scripts/lib/source_only
  1371. 2020-10-08T11:34:02.1507040Z [lighthouse]$ [cluster2] script_name=deploy_helm
  1372. 2020-10-08T11:34:02.1519520Z [lighthouse]$ [cluster2] exec_name=deploy.sh
  1373. 2020-10-08T11:34:02.1539429Z [lighthouse]$ [cluster2] LC_CTYPE=C tr -dc a-zA-Z0-9
  1374. 2020-10-08T11:34:02.1558161Z [lighthouse]$ [cluster2] fold -w 64
  1375. 2020-10-08T11:34:02.1583086Z [lighthouse]$ [cluster2] head -n 1
  1376. 2020-10-08T11:34:02.1665601Z [lighthouse]$ [cluster2] deploytool_prereqs
  1377. 2020-10-08T11:34:02.1677858Z [lighthouse]$ [cluster2] deploytool_prereqs
  1378. 2020-10-08T11:34:02.1690687Z [lighthouse]$ [cluster2] helm init --client-only
  1379. 2020-10-08T11:34:02.4343234Z Creating /root/.helm
  1380. 2020-10-08T11:34:02.4344827Z Creating /root/.helm/repository
  1381. 2020-10-08T11:34:02.4345278Z Creating /root/.helm/repository/cache
  1382. 2020-10-08T11:34:02.4347175Z Creating /root/.helm/repository/local
  1383. 2020-10-08T11:34:02.4347563Z Creating /root/.helm/plugins
  1384. 2020-10-08T11:34:02.4347926Z Creating /root/.helm/starters
  1385. 2020-10-08T11:34:02.4348304Z Creating /root/.helm/cache/archive
  1386. 2020-10-08T11:34:02.4348838Z Creating /root/.helm/repository/repositories.yaml
  1387. 2020-10-08T11:34:02.4353720Z Adding stable repo with URL: https://kubernetes-charts.storage.googleapis.com
  1388. 2020-10-08T11:34:03.9824390Z Adding local repo with URL: http://127.0.0.1:8879/charts
  1389. 2020-10-08T11:34:03.9831783Z $HELM_HOME has been configured at /root/.helm.
  1390. 2020-10-08T11:34:03.9832906Z Not installing Tiller due to 'client-only' flag having been set
  1391. 2020-10-08T11:34:03.9950594Z [lighthouse]$ [cluster2] helm repo add submariner-latest https://submariner-io.github.io/submariner-charts/charts
  1392. 2020-10-08T11:34:04.4597942Z "submariner-latest" has been added to your repositories
  1393. 2020-10-08T11:34:04.4625119Z [lighthouse]$ [cluster2] run_all_clusters install_helm
  1394. 2020-10-08T11:34:04.4636274Z [lighthouse]$ [cluster2] run_all_clusters install_helm
  1395. 2020-10-08T11:34:04.4651106Z [lighthouse]$ [cluster2] run_parallel cluster1 cluster2 install_helm
  1396. 2020-10-08T11:34:04.4661822Z [lighthouse]$ [cluster2] run_parallel cluster1 cluster2 cluster1 cluster2 install_helm
  1397. 2020-10-08T11:34:04.4675133Z [lighthouse]$ [cluster2] local cmnd=install_helm
  1398. 2020-10-08T11:34:04.4685409Z [lighthouse]$ [cluster2] declare -A pids
  1399. 2020-10-08T11:34:04.4713708Z [lighthouse]$ [cluster2] eval echo cluster1 cluster2
  1400. 2020-10-08T11:34:04.4746317Z [lighthouse]$ [cluster1] pids[cluster1]=5097
  1401. 2020-10-08T11:34:04.4756167Z [lighthouse]$ [cluster1] set -o pipefail
  1402. 2020-10-08T11:34:04.4784964Z [lighthouse]$ [cluster2] pids[cluster2]=5100
  1403. 2020-10-08T11:34:04.4786571Z [lighthouse]$ [cluster2] set -o pipefail
  1404. 2020-10-08T11:34:04.4801103Z [lighthouse]$ [cluster2] install_helm
  1405. 2020-10-08T11:34:04.4806864Z [lighthouse]$ [cluster1] install_helm
  1406. 2020-10-08T11:34:04.4820349Z [lighthouse]$ [cluster2] wait 5100
  1407. 2020-10-08T11:34:04.4824199Z [lighthouse]$ [cluster1] sed /\[cluster1]/!s/^/[cluster1] /
  1408. 2020-10-08T11:34:04.4835709Z [lighthouse]$ [cluster2] sed /\[cluster2]/!s/^/[cluster2] /
  1409. 2020-10-08T11:34:14.4842256Z [lighthouse]$ [cluster1] install_helm
  1410. 2020-10-08T11:34:14.4842795Z [cluster1] 
  1411. 2020-10-08T11:34:14.4843137Z [cluster1] Installing helm...
  1412. 2020-10-08T11:34:14.4844781Z [lighthouse]$ [cluster1] kubectl -n kube-system create serviceaccount tiller
  1413. 2020-10-08T11:34:14.4846913Z [lighthouse]$ [cluster1] kubectl -n kube-system create serviceaccount tiller
  1414. 2020-10-08T11:34:14.4848064Z [lighthouse]$ [cluster1] command kubectl --context=cluster1 -n kube-system create serviceaccount tiller
  1415. 2020-10-08T11:34:14.4849245Z [lighthouse]$ [cluster1] kubectl --context=cluster1 -n kube-system create serviceaccount tiller
  1416. 2020-10-08T11:34:14.4849980Z [cluster1] serviceaccount/tiller created
  1417. 2020-10-08T11:34:14.4851147Z [lighthouse]$ [cluster1] kubectl create clusterrolebinding tiller --clusterrole=cluster-admin --serviceaccount=kube-system:tiller
  1418. 2020-10-08T11:34:14.4853458Z [lighthouse]$ [cluster1] kubectl create clusterrolebinding tiller --clusterrole=cluster-admin --serviceaccount=kube-system:tiller
  1419. 2020-10-08T11:34:14.4855182Z [lighthouse]$ [cluster1] command kubectl --context=cluster1 create clusterrolebinding tiller --clusterrole=cluster-admin --serviceaccount=kube-system:tiller
  1420. 2020-10-08T11:34:14.4857145Z [lighthouse]$ [cluster1] kubectl --context=cluster1 create clusterrolebinding tiller --clusterrole=cluster-admin --serviceaccount=kube-system:tiller
  1421. 2020-10-08T11:34:14.4858662Z [cluster1] clusterrolebinding.rbac.authorization.k8s.io/tiller created
  1422. 2020-10-08T11:34:14.4860105Z [lighthouse]$ [cluster1] helm --kube-context cluster1 init --service-account tiller
  1423. 2020-10-08T11:34:14.4860808Z [cluster1] $HELM_HOME has been configured at /root/.helm.
  1424. 2020-10-08T11:34:14.4861220Z [cluster1]
  1425. 2020-10-08T11:34:14.4862008Z [cluster1] Tiller (the Helm server-side component) has been installed into your Kubernetes Cluster.
  1426. 2020-10-08T11:34:14.4862588Z [cluster1]
  1427. 2020-10-08T11:34:14.4863424Z [cluster1] Please note: by default, Tiller is deployed with an insecure 'allow unauthenticated users' policy.
  1428. 2020-10-08T11:34:14.4864443Z [cluster1] To prevent this, run `helm init` with the --tiller-tls-verify flag.
  1429. 2020-10-08T11:34:14.4865408Z [cluster1] For more information on securing your installation see: https://v2.helm.sh/docs/securing_installation/
  1430. 2020-10-08T11:34:14.4866919Z [lighthouse]$ [cluster1] kubectl -n kube-system rollout status deploy/tiller-deploy --timeout=30s
  1431. 2020-10-08T11:34:14.4868244Z [lighthouse]$ [cluster1] kubectl -n kube-system rollout status deploy/tiller-deploy --timeout=30s
  1432. 2020-10-08T11:34:14.4869697Z [lighthouse]$ [cluster1] command kubectl --context=cluster1 -n kube-system rollout status deploy/tiller-deploy --timeout=30s
  1433. 2020-10-08T11:34:14.4871051Z [lighthouse]$ [cluster1] kubectl --context=cluster1 -n kube-system rollout status deploy/tiller-deploy --timeout=30s
  1434. 2020-10-08T11:34:14.4872259Z [cluster1] Waiting for deployment "tiller-deploy" rollout to finish: 0 of 1 updated replicas are available...
  1435. 2020-10-08T11:34:14.4873255Z [cluster1] deployment "tiller-deploy" successfully rolled out
  1436. 2020-10-08T11:34:19.1899164Z [lighthouse]$ [cluster2] install_helm
  1437. 2020-10-08T11:34:19.1900415Z [cluster2] 
  1438. 2020-10-08T11:34:19.1901239Z [cluster2] Installing helm...
  1439. 2020-10-08T11:34:19.1902437Z [lighthouse]$ [cluster2] kubectl -n kube-system create serviceaccount tiller
  1440. 2020-10-08T11:34:19.1903484Z [lighthouse]$ [cluster2] kubectl -n kube-system create serviceaccount tiller
  1441. 2020-10-08T11:34:19.1904671Z [lighthouse]$ [cluster2] command kubectl --context=cluster2 -n kube-system create serviceaccount tiller
  1442. 2020-10-08T11:34:19.1905850Z [lighthouse]$ [cluster2] kubectl --context=cluster2 -n kube-system create serviceaccount tiller
  1443. 2020-10-08T11:34:19.1906591Z [cluster2] serviceaccount/tiller created
  1444. 2020-10-08T11:34:19.1907756Z [lighthouse]$ [cluster2] kubectl create clusterrolebinding tiller --clusterrole=cluster-admin --serviceaccount=kube-system:tiller
  1445. 2020-10-08T11:34:19.1909327Z [lighthouse]$ [cluster2] kubectl create clusterrolebinding tiller --clusterrole=cluster-admin --serviceaccount=kube-system:tiller
  1446. 2020-10-08T11:34:19.1911231Z [lighthouse]$ [cluster2] command kubectl --context=cluster2 create clusterrolebinding tiller --clusterrole=cluster-admin --serviceaccount=kube-system:tiller
  1447. 2020-10-08T11:34:19.1913034Z [lighthouse]$ [cluster2] kubectl --context=cluster2 create clusterrolebinding tiller --clusterrole=cluster-admin --serviceaccount=kube-system:tiller
  1448. 2020-10-08T11:34:19.1914529Z [cluster2] clusterrolebinding.rbac.authorization.k8s.io/tiller created
  1449. 2020-10-08T11:34:19.1915913Z [lighthouse]$ [cluster2] helm --kube-context cluster2 init --service-account tiller
  1450. 2020-10-08T11:34:19.1916614Z [cluster2] $HELM_HOME has been configured at /root/.helm.
  1451. 2020-10-08T11:34:19.1917022Z [cluster2]
  1452. 2020-10-08T11:34:19.1917808Z [cluster2] Tiller (the Helm server-side component) has been installed into your Kubernetes Cluster.
  1453. 2020-10-08T11:34:19.1918388Z [cluster2]
  1454. 2020-10-08T11:34:19.1919218Z [cluster2] Please note: by default, Tiller is deployed with an insecure 'allow unauthenticated users' policy.
  1455. 2020-10-08T11:34:19.1920247Z [cluster2] To prevent this, run `helm init` with the --tiller-tls-verify flag.
  1456. 2020-10-08T11:34:19.1921170Z [cluster2] For more information on securing your installation see: https://v2.helm.sh/docs/securing_installation/
  1457. 2020-10-08T11:34:19.1922395Z [lighthouse]$ [cluster2] kubectl -n kube-system rollout status deploy/tiller-deploy --timeout=30s
  1458. 2020-10-08T11:34:19.1923504Z [lighthouse]$ [cluster2] kubectl -n kube-system rollout status deploy/tiller-deploy --timeout=30s
  1459. 2020-10-08T11:34:19.1924740Z [lighthouse]$ [cluster2] command kubectl --context=cluster2 -n kube-system rollout status deploy/tiller-deploy --timeout=30s
  1460. 2020-10-08T11:34:19.1926036Z [lighthouse]$ [cluster2] kubectl --context=cluster2 -n kube-system rollout status deploy/tiller-deploy --timeout=30s
  1461. 2020-10-08T11:34:19.1927238Z [cluster2] Waiting for deployment "tiller-deploy" rollout to finish: 0 of 1 updated replicas are available...
  1462. 2020-10-08T11:34:19.1928244Z [cluster2] deployment "tiller-deploy" successfully rolled out
  1463. 2020-10-08T11:34:19.1953299Z [lighthouse]$ [cluster2] wait 5097
  1464. 2020-10-08T11:34:19.1994173Z [lighthouse]$ [cluster2] run_subm_clusters prepare_cluster submariner-operator
  1465. 2020-10-08T11:34:19.2020878Z [lighthouse]$ [cluster2] run_subm_clusters prepare_cluster submariner-operator
  1466. 2020-10-08T11:34:19.2036170Z [lighthouse]$ [cluster2] declare -a subm_clusters
  1467. 2020-10-08T11:34:19.2121185Z [lighthouse]$ [cluster2] run_parallel cluster1 cluster2 prepare_cluster submariner-operator
  1468. 2020-10-08T11:34:19.2143246Z [lighthouse]$ [cluster2] run_parallel cluster1 cluster2 cluster1 cluster2 prepare_cluster submariner-operator
  1469. 2020-10-08T11:34:19.2160289Z [lighthouse]$ [cluster2] local cmnd=prepare_cluster
  1470. 2020-10-08T11:34:19.2179784Z [lighthouse]$ [cluster2] declare -A pids
  1471. 2020-10-08T11:34:19.2203917Z [lighthouse]$ [cluster2] eval echo cluster1 cluster2
  1472. 2020-10-08T11:34:19.2235227Z [lighthouse]$ [cluster1] pids[cluster1]=5229
  1473. 2020-10-08T11:34:19.2256811Z [lighthouse]$ [cluster1] set -o pipefail
  1474. 2020-10-08T11:34:19.2269580Z [lighthouse]$ [cluster1] prepare_cluster submariner-operator
  1475. 2020-10-08T11:34:19.2281449Z [lighthouse]$ [cluster2] set -o pipefail
  1476. 2020-10-08T11:34:19.2301329Z [lighthouse]$ [cluster2] pids[cluster2]=5232
  1477. 2020-10-08T11:34:19.2324438Z [lighthouse]$ [cluster1] sed /\[cluster1]/!s/^/[cluster1] /
  1478. 2020-10-08T11:34:19.2337744Z [lighthouse]$ [cluster2] prepare_cluster submariner-operator
  1479. 2020-10-08T11:34:19.2361921Z [lighthouse]$ [cluster2] wait 5232
  1480. 2020-10-08T11:34:19.2395739Z [lighthouse]$ [cluster2] sed /\[cluster2]/!s/^/[cluster2] /
  1481. 2020-10-08T11:34:22.6620040Z [lighthouse]$ [cluster2] prepare_cluster
  1482. 2020-10-08T11:34:22.6620968Z [lighthouse]$ [cluster2] local namespace=submariner-operator
  1483. 2020-10-08T11:34:22.6621642Z [cluster2] 
  1484. 2020-10-08T11:34:22.6622063Z [cluster2] 
  1485. 2020-10-08T11:34:22.6622722Z [cluster2] 
  1486. 2020-10-08T11:34:22.6623322Z [lighthouse]$ [cluster2] add_subm_gateway_label
  1487. 2020-10-08T11:34:22.6624018Z [lighthouse]$ [cluster2] add_subm_gateway_label
  1488. 2020-10-08T11:34:22.6624991Z [lighthouse]$ [cluster2] kubectl label node cluster2-worker submariner.io/gateway=true --overwrite
  1489. 2020-10-08T11:34:22.6626182Z [lighthouse]$ [cluster2] kubectl label node cluster2-worker submariner.io/gateway=true --overwrite
  1490. 2020-10-08T11:34:22.6627709Z [lighthouse]$ [cluster2] command kubectl --context=cluster2 label node cluster2-worker submariner.io/gateway=true --overwrite
  1491. 2020-10-08T11:34:22.6629088Z [lighthouse]$ [cluster2] kubectl --context=cluster2 label node cluster2-worker submariner.io/gateway=true --overwrite
  1492. 2020-10-08T11:34:22.6630057Z [cluster2] node/cluster2-worker labeled
  1493. 2020-10-08T11:34:22.6665622Z [lighthouse]$ [cluster2] wait 5229
  1494. 2020-10-08T11:34:22.7555105Z [lighthouse]$ [cluster1] prepare_cluster
  1495. 2020-10-08T11:34:22.7556108Z [lighthouse]$ [cluster1] local namespace=submariner-operator
  1496. 2020-10-08T11:34:22.7556779Z [cluster1] 
  1497. 2020-10-08T11:34:22.7557211Z [cluster1] 
  1498. 2020-10-08T11:34:22.7557628Z [cluster1] 
  1499. 2020-10-08T11:34:22.7558202Z [lighthouse]$ [cluster1] add_subm_gateway_label
  1500. 2020-10-08T11:34:22.7558868Z [lighthouse]$ [cluster1] add_subm_gateway_label
  1501. 2020-10-08T11:34:22.7559832Z [lighthouse]$ [cluster1] kubectl label node cluster1-worker submariner.io/gateway=true --overwrite
  1502. 2020-10-08T11:34:22.7561030Z [lighthouse]$ [cluster1] kubectl label node cluster1-worker submariner.io/gateway=true --overwrite
  1503. 2020-10-08T11:34:22.7562352Z [lighthouse]$ [cluster1] command kubectl --context=cluster1 label node cluster1-worker submariner.io/gateway=true --overwrite
  1504. 2020-10-08T11:34:22.7563719Z [lighthouse]$ [cluster1] kubectl --context=cluster1 label node cluster1-worker submariner.io/gateway=true --overwrite
  1505. 2020-10-08T11:34:22.7564690Z [cluster1] node/cluster1-worker labeled
  1506. 2020-10-08T11:34:22.7582290Z [lighthouse]$ [cluster2] with_context cluster1 setup_broker
  1507. 2020-10-08T11:34:22.7597933Z [lighthouse]$ [cluster2] with_context cluster1 setup_broker
  1508. 2020-10-08T11:34:22.7610260Z [lighthouse]$ [cluster2] local cluster=cluster1
  1509. 2020-10-08T11:34:22.7622707Z [lighthouse]$ [cluster1] local cmnd=setup_broker
  1510. 2020-10-08T11:34:22.7634241Z [lighthouse]$ [cluster1] setup_broker
  1511. 2020-10-08T11:34:22.7645837Z [lighthouse]$ [cluster1] setup_broker
  1512. 2020-10-08T11:34:22.7658000Z 
  1513. 2020-10-08T11:34:23.5020328Z Installing submariner broker...
  1514. 2020-10-08T11:34:23.5032341Z [lighthouse]$ [cluster1] helm install submariner-latest/submariner-k8s-broker --kube-context cluster1 --name submariner-k8s-broker --namespace submariner-k8s-broker --set submariner.serviceDiscovery=true
  1515. 2020-10-08T11:34:28.6203551Z NAME: submariner-k8s-broker
  1516. 2020-10-08T11:34:28.6663153Z LAST DEPLOYED: Thu Oct 8 11:34:24 2020
  1517. 2020-10-08T11:34:28.6664228Z NAMESPACE: submariner-k8s-broker
  1518. 2020-10-08T11:34:28.6664686Z STATUS: DEPLOYED
  1519. 2020-10-08T11:34:28.6665037Z
  1520. 2020-10-08T11:34:28.6667615Z RESOURCES:
  1521. 2020-10-08T11:34:28.6670882Z ==> v1/Role
  1522. 2020-10-08T11:34:28.6679124Z NAME AGE
  1523. 2020-10-08T11:34:28.6685683Z submariner-k8s-broker:client 0s
  1524. 2020-10-08T11:34:28.6686064Z
  1525. 2020-10-08T11:34:28.6689386Z ==> v1/RoleBinding
  1526. 2020-10-08T11:34:28.6698960Z NAME AGE
  1527. 2020-10-08T11:34:28.6707031Z submariner-k8s-broker:client 0s
  1528. 2020-10-08T11:34:28.6707366Z
  1529. 2020-10-08T11:34:28.6707659Z ==> v1/ServiceAccount
  1530. 2020-10-08T11:34:28.6708010Z NAME SECRETS AGE
  1531. 2020-10-08T11:34:28.6708688Z submariner-k8s-broker-client 1 0s
  1532. 2020-10-08T11:34:28.6709082Z
  1533. 2020-10-08T11:34:28.6709244Z
  1534. 2020-10-08T11:34:28.6709454Z NOTES:
  1535. 2020-10-08T11:34:28.6709839Z The Submariner Kubernetes Broker is now setup.
  1536. 2020-10-08T11:34:28.6710259Z
  1537. 2020-10-08T11:34:28.6710828Z You can retrieve the server URL by running
  1538. 2020-10-08T11:34:28.6711097Z
  1539. 2020-10-08T11:34:28.6712085Z $ SUBMARINER_BROKER_URL=$(kubectl -n default get endpoints kubernetes -o jsonpath="{.subsets[0].addresses[0].ip}:{.subsets[0].ports[?(@.name=='https')].port}")
  1540. 2020-10-08T11:34:28.6712717Z
  1541. 2020-10-08T11:34:28.6713099Z The broker client token and CA can be retrieved by running
  1542. 2020-10-08T11:34:28.6713412Z
  1543. 2020-10-08T11:34:28.6714867Z $ SUBMARINER_BROKER_CA=$(kubectl -n submariner-k8s-broker get secrets -o jsonpath="{.items[?(@.metadata.annotations['kubernetes\.io/service-account\.name']=='submariner-k8s-broker-client')].data['ca\.crt']}")
  1544. 2020-10-08T11:34:28.6717333Z $ SUBMARINER_BROKER_TOKEN=$(kubectl -n submariner-k8s-broker get secrets -o jsonpath="{.items[?(@.metadata.annotations['kubernetes\.io/service-account\.name']=='submariner-k8s-broker-client')].data.token}"|base64 --decode)
  1545. 2020-10-08T11:34:28.6718631Z
  1546. 2020-10-08T11:34:29.3826427Z [lighthouse]$ [cluster1] submariner_broker_url=172.17.0.8:6443
  1547. 2020-10-08T11:34:29.3844941Z [lighthouse]$ [cluster1] kubectl -n default get endpoints kubernetes -o jsonpath={.subsets[0].addresses[0].ip}:{.subsets[0].ports[?(@.name=='https')].port}
  1548. 2020-10-08T11:34:29.3856736Z [lighthouse]$ [cluster1] kubectl -n default get endpoints kubernetes -o jsonpath={.subsets[0].addresses[0].ip}:{.subsets[0].ports[?(@.name=='https')].port}
  1549. 2020-10-08T11:34:29.3869946Z [lighthouse]$ [cluster1] command kubectl --context=cluster1 -n default get endpoints kubernetes -o jsonpath={.subsets[0].addresses[0].ip}:{.subsets[0].ports[?(@.name=='https')].port}
  1550. 2020-10-08T11:34:30.9218403Z [lighthouse]$ [cluster1] submariner_broker_ca=LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUN5RENDQWJDZ0F3SUJBZ0lCQURBTkJna3Foa2lHOXcwQkFRc0ZBREFWTVJNd0VRWURWUVFERXdwcmRXSmwKY201bGRHVnpNQjRYRFRJd01UQXdPREV4TXpFd05Wb1hEVE13TVRBd05qRXhNekV3TlZvd0ZURVRNQkVHQTFVRQpBeE1LYTNWaVpYSnVaWFJsY3pDQ0FTSXdEUVlKS29aSWh2Y05BUUVCQlFBRGdnRVBBRENDQVFvQ2dnRUJBTjRzClZ4WVdBaHBxcHIzS0lYNWtpelpZNnhaZ2lxc3grdHZDMWhtdzVDN1IzdnB3YUREcVFuazV2M1d0YkRET1h0Z3AKdTd4RU5CaUFUS0FsTFNGYjUxbFNSdHJXUzdjZHhjMzZDK2R1akY4WmVrQlE4aTZSL1JoZnU0VFFoQ0xKZXdIQgpTQ2hRWDVCWmFTWmV1U0swcU9mYm1yMEJ0NmI0c0c1ZERaQnFBeHRWRUtXOVQrNDZOS0w1OEROalBSY1VEeHB3CmJhb3dHSlRpSEFhdW9VTzY5TnBISzZDUGVSeXlNNUtNUk9Ea0tBSTdTRHc4V2ZLK0w3UnpEK2JjbGttTFZkdFAKR2ZPaXFJRWRBblE2ZjhKTWF2U1RjQU11UzRPSThjcTdGNk1Kb1R5bnRpK0lKcUVkVm8xS3ltd05KVjkwUTl1SApPU3BNMU1DOCt1QXBScWZ1cDFVQ0F3RUFBYU1qTUNFd0RnWURWUjBQQVFIL0JBUURBZ0trTUE4R0ExVWRFd0VCCi93UUZNQU1CQWY4d0RRWUpLb1pJaHZjTkFRRUxCUUFEZ2dFQkFOVk1aT1JFeXJZbEhsWFRzYVNIQlBEdGVoWlIKQi9ZYmZOdzRWY3ZKYko2a0NVMm9lTDFNOTBFbDBCdVVnQ2w1WGlnaDh4ZkVmZkxNNE5XbEw2OEtYd1FCRWhJeQpLNFBOSk1lSzExRlloK01rbDFXYVRnazZ6VUg2YVkyTDRJZDNJTk9nSnVWTFI2SSsrbEZwSXJzR2ZUdlkyUUd4CkV4NDhocFNMeXFqdHVwQXd6K2NFeElrSjJvVHhLZ3YxVGlTMitDVWkwa2FUNWZCeEFMcjViNm5kbVZEbllJR3EKOGFBaHpIWkg3QWp6YW16SXdRTVY1TVptUWRWT2VleUtjdDBqTE1pa2I1RHRCWE9KSjlVT1crdmhCRTVPUGY4dQpISEdOdE1oRmJTZ1hnQUt0ekZnM0JoOVJ3VmZ3UCtCNldvYjRxVkZoRVRrdWd3Z1lWT253TlFYZlNzND0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=
  1551. 2020-10-08T11:34:30.9698224Z [lighthouse]$ [cluster1] kubectl -n submariner-k8s-broker get secrets -o jsonpath={.items[?(@.metadata.annotations['kubernetes\.io/service-account\.name']=='submariner-k8s-broker-client')].data['ca\.crt']}
  1552. 2020-10-08T11:34:30.9700687Z [lighthouse]$ [cluster1] kubectl -n submariner-k8s-broker get secrets -o jsonpath={.items[?(@.metadata.annotations['kubernetes\.io/service-account\.name']=='submariner-k8s-broker-client')].data['ca\.crt']}
  1553. 2020-10-08T11:34:30.9703188Z [lighthouse]$ [cluster1] command kubectl --context=cluster1 -n submariner-k8s-broker get secrets -o jsonpath={.items[?(@.metadata.annotations['kubernetes\.io/service-account\.name']=='submariner-k8s-broker-client')].data['ca\.crt']}
  1554. 2020-10-08T11:34:32.4374327Z [lighthouse]$ [cluster1] submariner_broker_token=eyJhbGciOiJSUzI1NiIsImtpZCI6IlRiczIwdFQwY0VEZ3huVXBsLW8xWk9udVpkbjlzb1pESkVFbDRpLVNEazQifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJzdWJtYXJpbmVyLWs4cy1icm9rZXIiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlY3JldC5uYW1lIjoic3VibWFyaW5lci1rOHMtYnJva2VyLWNsaWVudC10b2tlbi1tZ3ZjcSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJzdWJtYXJpbmVyLWs4cy1icm9rZXItY2xpZW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiNWFlODVlMTMtNTg5NS00MzE1LWJiZjYtZGMyOTZjZDU3YzMyIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50OnN1Ym1hcmluZXItazhzLWJyb2tlcjpzdWJtYXJpbmVyLWs4cy1icm9rZXItY2xpZW50In0.UIL8BahihwDz9vKU9qh71BQ2KP0K_11YHRS7qYI-A6LsUfSvqA97TNa115b725MHM-4V3w4iW1uCLCt-iKkUtMZWEOZgMGsi4uQCSPnkNbPC_Xip0caA94_SJ_zyv95zQYjuPKIwBX_pjxJ0KZWqk9dVwsAqcm54dByQ-axKG54Yd85_jqJNopZOuNZYFnIwOdE2Icd4bQypTnMPHLXSx-v1IZL0KW5Sw48KOX8a3XcClTAAwraq-qyCSAS-HEszYlhbwue-zE109lL0Xz4wy4mKEUSHSphMGmzXEsuE93LxGrzc2RqSKE0YFnB2HM6aLCoKXlwsmY67T21gfnycPA
  1555. 2020-10-08T11:34:32.4570443Z [lighthouse]$ [cluster1] kubectl -n submariner-k8s-broker get secrets -o jsonpath={.items[?(@.metadata.annotations['kubernetes\.io/service-account\.name']=='submariner-k8s-broker-client')].data.token}
  1556. 2020-10-08T11:34:32.4571849Z [lighthouse]$ [cluster1] base64 --decode
  1557. 2020-10-08T11:34:32.4573723Z [lighthouse]$ [cluster1] kubectl -n submariner-k8s-broker get secrets -o jsonpath={.items[?(@.metadata.annotations['kubernetes\.io/service-account\.name']=='submariner-k8s-broker-client')].data.token}
  1558. 2020-10-08T11:34:32.4576078Z [lighthouse]$ [cluster1] command kubectl --context=cluster1 -n submariner-k8s-broker get secrets -o jsonpath={.items[?(@.metadata.annotations['kubernetes\.io/service-account\.name']=='submariner-k8s-broker-client')].data.token}
  1559. 2020-10-08T11:34:33.1913552Z [lighthouse]$ [cluster2] install_subm_all_clusters
  1560. 2020-10-08T11:34:33.1924162Z [lighthouse]$ [cluster2] install_subm_all_clusters
  1561. 2020-10-08T11:34:33.1936163Z [lighthouse]$ [cluster2] run_subm_clusters helm_install_subm
  1562. 2020-10-08T11:34:33.1948566Z [lighthouse]$ [cluster2] run_subm_clusters helm_install_subm
  1563. 2020-10-08T11:34:33.1961614Z [lighthouse]$ [cluster2] declare -a subm_clusters
  1564. 2020-10-08T11:34:33.2001074Z [lighthouse]$ [cluster2] run_parallel cluster1 cluster2 helm_install_subm
  1565. 2020-10-08T11:34:33.2014587Z [lighthouse]$ [cluster2] run_parallel cluster1 cluster2 cluster1 cluster2 helm_install_subm
  1566. 2020-10-08T11:34:33.2025708Z [lighthouse]$ [cluster2] local cmnd=helm_install_subm
  1567. 2020-10-08T11:34:33.2036709Z [lighthouse]$ [cluster2] declare -A pids
  1568. 2020-10-08T11:34:33.2055547Z [lighthouse]$ [cluster2] eval echo cluster1 cluster2
  1569. 2020-10-08T11:34:33.2077758Z [lighthouse]$ [cluster1] pids[cluster1]=5443
  1570. 2020-10-08T11:34:33.2081440Z [lighthouse]$ [cluster1] set -o pipefail
  1571. 2020-10-08T11:34:33.2100126Z [lighthouse]$ [cluster1] helm_install_subm
  1572. 2020-10-08T11:34:33.2109542Z [lighthouse]$ [cluster2] pids[cluster2]=5447
  1573. 2020-10-08T11:34:33.2114083Z [lighthouse]$ [cluster2] set -o pipefail
  1574. 2020-10-08T11:34:33.2128480Z [lighthouse]$ [cluster1] sed /\[cluster1]/!s/^/[cluster1] /
  1575. 2020-10-08T11:34:33.2129559Z [lighthouse]$ [cluster2] helm_install_subm
  1576. 2020-10-08T11:34:33.2140423Z [lighthouse]$ [cluster2] wait 5447
  1577. 2020-10-08T11:34:33.2155193Z [lighthouse]$ [cluster2] sed /\[cluster2]/!s/^/[cluster2] /
  1578. 2020-10-08T11:34:35.7494828Z [lighthouse]$ [cluster1] helm_install_subm
  1579. 2020-10-08T11:34:35.7495543Z [lighthouse]$ [cluster1] local crd_create=false
  1580. 2020-10-08T11:34:35.7496045Z [cluster1] 
  1581. 2020-10-08T11:34:35.7496420Z [cluster1] Installing Submariner...
  1582. 2020-10-08T11:34:35.8415287Z [lighthouse]$ [cluster1] helm --kube-context cluster1 install submariner-latest/submariner --name submariner --namespace submariner-operator --set ipsec.psk=euId2xOHjMyEyo3VVxwjuiGT4dguIkzf2nxLuBGqg2nueoBUjY7XQIaqYVORtbLA --set broker.server=172.17.0.8:6443 --set broker.token=eyJhbGciOiJSUzI1NiIsImtpZCI6IlRiczIwdFQwY0VEZ3huVXBsLW8xWk9udVpkbjlzb1pESkVFbDRpLVNEazQifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJzdWJtYXJpbmVyLWs4cy1icm9rZXIiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlY3JldC5uYW1lIjoic3VibWFyaW5lci1rOHMtYnJva2VyLWNsaWVudC10b2tlbi1tZ3ZjcSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJzdWJtYXJpbmVyLWs4cy1icm9rZXItY2xpZW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiNWFlODVlMTMtNTg5NS00MzE1LWJiZjYtZGMyOTZjZDU3YzMyIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50OnN1Ym1hcmluZXItazhzLWJyb2tlcjpzdWJtYXJpbmVyLWs4cy1icm9rZXItY2xpZW50In0.UIL8BahihwDz9vKU9qh71BQ2KP0K_11YHRS7qYI-A6LsUfSvqA97TNa115b725MHM-4V3w4iW1uCLCt-iKkUtMZWEOZgMGsi4uQCSPnkNbPC_Xip0caA94_SJ_zyv95zQYjuPKIwBX_pjxJ0KZWqk9dVwsAqcm54dByQ-axKG54Yd85_jqJNopZOuNZYFnIwOdE2Icd4bQypTnMPHLXSx-v1IZL0KW5Sw48KOX8a3XcClTAAwraq-qyCSAS-HEszYlhbwue-zE109lL0Xz4wy4mKEUSHSphMGmzXEsuE93LxGrzc2RqSKE0YFnB2HM6aLCoKXlwsmY67T21gfnycPA --set broker.namespace=submariner-k8s-broker --set broker.ca=LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUN5RENDQWJDZ0F3SUJBZ0lCQURBTkJna3Foa2lHOXcwQkFRc0ZBREFWTVJNd0VRWURWUVFERXdwcmRXSmwKY201bGRHVnpNQjRYRFRJd01UQXdPREV4TXpFd05Wb1hEVE13TVRBd05qRXhNekV3TlZvd0ZURVRNQkVHQTFVRQpBeE1LYTNWaVpYSnVaWFJsY3pDQ0FTSXdEUVlKS29aSWh2Y05BUUVCQlFBRGdnRVBBRENDQVFvQ2dnRUJBTjRzClZ4WVdBaHBxcHIzS0lYNWtpelpZNnhaZ2lxc3grdHZDMWhtdzVDN1IzdnB3YUREcVFuazV2M1d0YkRET1h0Z3AKdTd4RU5CaUFUS0FsTFNGYjUxbFNSdHJXUzdjZHhjMzZDK2R1akY4WmVrQlE4aTZSL1JoZnU0VFFoQ0xKZXdIQgpTQ2hRWDVCWmFTWmV1U0swcU9mYm1yMEJ0NmI0c0c1ZERaQnFBeHRWRUtXOVQrNDZOS0w1OEROalBSY1VEeHB3CmJhb3dHSlRpSEFhdW9VTzY5TnBISzZDUGVSeXlNNUtNUk9Ea0tBSTdTRHc4V2ZLK0w3UnpEK2JjbGttTFZkdFAKR2ZPaXFJRWRBblE2ZjhKTWF2U1RjQU11UzRPSThjcTdGNk1Kb1R5bnRpK0lKcUVkVm8xS3ltd05KVjkwUTl1SApPU3BNMU1DOCt1QXBScWZ1cDFVQ0F3RUFBYU1qTUNFd0RnWURWUjBQQVFIL0JBUURBZ0trTUE4R0ExVWRFd0VCCi93UUZNQU1CQWY4d0RRWUpLb1pJaHZjTkFRRUxCUUFEZ2dFQkFOVk1aT1JFeXJZbEhsWFRzYVNIQlBEdGVoWlIKQi9ZYmZOdzRWY3ZKYko2a0NVMm9lTDFNOTBFbDBCdVVnQ2w1WGlnaDh4ZkVmZkxNNE5XbEw2OEtYd1FCRWhJeQpLNFBOSk1lSzExRlloK01rbDFXYVRnazZ6VUg2YVkyTDRJZDNJTk9nSnVWTFI2SSsrbEZwSXJzR2ZUdlkyUUd4CkV4NDhocFNMeXFqdHVwQXd6K2NFeElrSjJvVHhLZ3YxVGlTMitDVWkwa2FUNWZCeEFMcjViNm5kbVZEbllJR3EKOGFBaHpIWkg3QWp6YW16SXdRTVY1TVptUWRWT2VleUtjdDBqTE1pa2I1RHRCWE9KSjlVT1crdmhCRTVPUGY4dQpISEdOdE1oRmJTZ1hnQUt0ekZnM0JoOVJ3VmZ3UCtCNldvYjRxVkZoRVRrdWd3Z1lWT253TlFYZlNzND0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo= --set submariner.clusterId=cluster1 --set submariner.clusterCidr=10.241.0.0/16 --set submariner.serviceCidr=100.91.0.0/16 --set submariner.globalCidr= --set serviceAccounts.globalnet.create=false --set submariner.natEnabled=false --set routeAgent.image.repository=localhost:5000/submariner-route-agent --set routeAgent.image.tag=local --set routeAgent.image.pullPolicy=IfNotPresent --set engine.image.repository=localhost:5000/submariner --set engine.image.tag=local --set engine.image.pullPolicy=IfNotPresent --set globalnet.image.repository=localhost:5000/submariner-globalnet --set globalnet.image.tag=local --set globalnet.image.pullPolicy=IfNotPresent --set crd.create=false --set submariner.serviceDiscovery=true,lighthouse.image.repository=localhost:5000/lighthouse-agent,lighthouse.image.tag=local,lighthouseCoredns.image.repository=localhost:5000/lighthouse-coredns,lighthouseCoredns.image.tag=local,serviceAccounts.lighthouse.create=true
  1583. 2020-10-08T11:34:35.9105432Z [cluster1] NAME: submariner
  1584. 2020-10-08T11:34:35.9105893Z [cluster1] LAST DEPLOYED: Thu Oct 8 11:34:34 2020
  1585. 2020-10-08T11:34:35.9113586Z [cluster1] NAMESPACE: submariner-operator
  1586. 2020-10-08T11:34:35.9114186Z [cluster1] STATUS: DEPLOYED
  1587. 2020-10-08T11:34:35.9114500Z [cluster1]
  1588. 2020-10-08T11:34:35.9114780Z [cluster1] RESOURCES:
  1589. 2020-10-08T11:34:35.9115129Z [cluster1] ==> v1/ClusterRole
  1590. 2020-10-08T11:34:35.9115588Z [cluster1] NAME AGE
  1591. 2020-10-08T11:34:35.9115960Z [cluster1] submariner:routeagent 1s
  1592. 2020-10-08T11:34:35.9116396Z [cluster1] submariner:lighthouse 1s
  1593. 2020-10-08T11:34:35.9116742Z [cluster1]
  1594. 2020-10-08T11:34:35.9119887Z [cluster1] ==> v1/ClusterRoleBinding
  1595. 2020-10-08T11:34:35.9120593Z [cluster1] NAME AGE
  1596. 2020-10-08T11:34:35.9120955Z [cluster1] submariner:lighthouse 1s
  1597. 2020-10-08T11:34:35.9121389Z [cluster1] submariner:routeagent 1s
  1598. 2020-10-08T11:34:35.9121743Z [cluster1]
  1599. 2020-10-08T11:34:35.9122148Z [cluster1] ==> v1/ConfigMap
  1600. 2020-10-08T11:34:35.9122444Z [cluster1] NAME DATA AGE
  1601. 2020-10-08T11:34:35.9123357Z [cluster1] submariner-lighthouse-coredns 1 1s
  1602. 2020-10-08T11:34:35.9123807Z [cluster1]
  1603. 2020-10-08T11:34:35.9124065Z [cluster1] ==> v1/DaemonSet
  1604. 2020-10-08T11:34:35.9124698Z [cluster1] NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
  1605. 2020-10-08T11:34:35.9125516Z [cluster1] submariner-gateway 1 1 0 1 0 submariner.io/gateway=true 1s
  1606. 2020-10-08T11:34:35.9126297Z [cluster1] submariner-routeagent 2 2 0 2 0 <none> 1s
  1607. 2020-10-08T11:34:35.9126682Z [cluster1]
  1608. 2020-10-08T11:34:35.9126953Z [cluster1] ==> v1/Deployment
  1609. 2020-10-08T11:34:35.9127473Z [cluster1] NAME READY UP-TO-DATE AVAILABLE AGE
  1610. 2020-10-08T11:34:35.9128339Z [cluster1] submariner-lighthouse-agent 0/1 1 0 1s
  1611. 2020-10-08T11:34:35.9129266Z [cluster1] submariner-lighthouse-coredns 0/2 2 0 1s
  1612. 2020-10-08T11:34:35.9129789Z [cluster1]
  1613. 2020-10-08T11:34:35.9130070Z [cluster1] ==> v1/Pod(related)
  1614. 2020-10-08T11:34:35.9130476Z [cluster1] NAME READY STATUS RESTARTS AGE
  1615. 2020-10-08T11:34:35.9131338Z [cluster1] submariner-gateway-jgnv7 0/1 ContainerCreating 0 1s
  1616. 2020-10-08T11:34:35.9133328Z [cluster1] submariner-lighthouse-agent-5fdd5968d8-gld9p 0/1 ContainerCreating 0 0s
  1617. 2020-10-08T11:34:35.9135540Z [cluster1] submariner-lighthouse-coredns-7466b6679c-w7pcw 0/1 ContainerCreating 0 0s
  1618. 2020-10-08T11:34:35.9137474Z [cluster1] submariner-lighthouse-coredns-7466b6679c-xhdm4 0/1 ContainerCreating 0 0s
  1619. 2020-10-08T11:34:35.9139043Z [cluster1] submariner-routeagent-cljq7 0/1 ContainerCreating 0 0s
  1620. 2020-10-08T11:34:35.9140426Z [cluster1] submariner-routeagent-wmmbs 0/1 ContainerCreating 0 1s
  1621. 2020-10-08T11:34:35.9141073Z [cluster1]
  1622. 2020-10-08T11:34:35.9141364Z [cluster1] ==> v1/Role
  1623. 2020-10-08T11:34:35.9141671Z [cluster1] NAME AGE
  1624. 2020-10-08T11:34:35.9142073Z [cluster1] submariner:routeagent 1s
  1625. 2020-10-08T11:34:35.9142518Z [cluster1] submariner:engine 1s
  1626. 2020-10-08T11:34:35.9142868Z [cluster1]
  1627. 2020-10-08T11:34:35.9143201Z [cluster1] ==> v1/RoleBinding
  1628. 2020-10-08T11:34:35.9143659Z [cluster1] NAME AGE
  1629. 2020-10-08T11:34:35.9144060Z [cluster1] submariner:routeagent 1s
  1630. 2020-10-08T11:34:35.9144485Z [cluster1] submariner:engine 1s
  1631. 2020-10-08T11:34:35.9144832Z [cluster1]
  1632. 2020-10-08T11:34:35.9145137Z [cluster1] ==> v1/Service
  1633. 2020-10-08T11:34:35.9145827Z [cluster1] NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
  1634. 2020-10-08T11:34:35.9146828Z [cluster1] submariner-lighthouse-coredns ClusterIP 100.91.206.45 <none> 53/UDP 1s
  1635. 2020-10-08T11:34:35.9147465Z [cluster1]
  1636. 2020-10-08T11:34:35.9147825Z [cluster1] ==> v1/ServiceAccount
  1637. 2020-10-08T11:34:35.9148237Z [cluster1] NAME SECRETS AGE
  1638. 2020-10-08T11:34:35.9148822Z [cluster1] submariner-engine 1 1s
  1639. 2020-10-08T11:34:35.9149528Z [cluster1] submariner-lighthouse 1 1s
  1640. 2020-10-08T11:34:35.9150365Z [cluster1] submariner-routeagent 1 1s
  1641. 2020-10-08T11:34:35.9150768Z [cluster1]
  1642. 2020-10-08T11:34:35.9151019Z [cluster1]
  1643. 2020-10-08T11:34:35.9151291Z [cluster1] NOTES:
  1644. 2020-10-08T11:34:35.9151661Z [cluster1] Submariner is now installed.
  1645. 2020-10-08T11:34:35.9152541Z [cluster1] If you haven't done so yet, please label a node as `submariner.io/gateway=true` to elect it for running Submariner.
  1646. 2020-10-08T11:34:35.9153127Z [cluster1]
  1647. 2020-10-08T11:34:35.9154009Z [cluster1] By default, Submariner runs with 1 replica. If you have more than one Gateway host, you can scale Submariner to N replicas, and the other Submariner pods will simply join the leader election pool.
  1648. 2020-10-08T11:34:48.0482370Z [lighthouse]$ [cluster2] helm_install_subm
  1649. 2020-10-08T11:34:48.0483134Z [lighthouse]$ [cluster2] local crd_create=false
  1650. 2020-10-08T11:34:48.0483910Z [lighthouse]$ [cluster2] crd_create=true
  1651. 2020-10-08T11:34:48.0484427Z [cluster2] 
  1652. 2020-10-08T11:34:48.0484831Z [cluster2] Installing Submariner...
  1653. 2020-10-08T11:34:48.1219375Z [lighthouse]$ [cluster2] helm --kube-context cluster2 install submariner-latest/submariner --name submariner --namespace submariner-operator --set ipsec.psk=euId2xOHjMyEyo3VVxwjuiGT4dguIkzf2nxLuBGqg2nueoBUjY7XQIaqYVORtbLA --set broker.server=172.17.0.8:6443 --set broker.token=eyJhbGciOiJSUzI1NiIsImtpZCI6IlRiczIwdFQwY0VEZ3huVXBsLW8xWk9udVpkbjlzb1pESkVFbDRpLVNEazQifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJzdWJtYXJpbmVyLWs4cy1icm9rZXIiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlY3JldC5uYW1lIjoic3VibWFyaW5lci1rOHMtYnJva2VyLWNsaWVudC10b2tlbi1tZ3ZjcSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJzdWJtYXJpbmVyLWs4cy1icm9rZXItY2xpZW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiNWFlODVlMTMtNTg5NS00MzE1LWJiZjYtZGMyOTZjZDU3YzMyIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50OnN1Ym1hcmluZXItazhzLWJyb2tlcjpzdWJtYXJpbmVyLWs4cy1icm9rZXItY2xpZW50In0.UIL8BahihwDz9vKU9qh71BQ2KP0K_11YHRS7qYI-A6LsUfSvqA97TNa115b725MHM-4V3w4iW1uCLCt-iKkUtMZWEOZgMGsi4uQCSPnkNbPC_Xip0caA94_SJ_zyv95zQYjuPKIwBX_pjxJ0KZWqk9dVwsAqcm54dByQ-axKG54Yd85_jqJNopZOuNZYFnIwOdE2Icd4bQypTnMPHLXSx-v1IZL0KW5Sw48KOX8a3XcClTAAwraq-qyCSAS-HEszYlhbwue-zE109lL0Xz4wy4mKEUSHSphMGmzXEsuE93LxGrzc2RqSKE0YFnB2HM6aLCoKXlwsmY67T21gfnycPA --set broker.namespace=submariner-k8s-broker --set broker.ca=LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUN5RENDQWJDZ0F3SUJBZ0lCQURBTkJna3Foa2lHOXcwQkFRc0ZBREFWTVJNd0VRWURWUVFERXdwcmRXSmwKY201bGRHVnpNQjRYRFRJd01UQXdPREV4TXpFd05Wb1hEVE13TVRBd05qRXhNekV3TlZvd0ZURVRNQkVHQTFVRQpBeE1LYTNWaVpYSnVaWFJsY3pDQ0FTSXdEUVlKS29aSWh2Y05BUUVCQlFBRGdnRVBBRENDQVFvQ2dnRUJBTjRzClZ4WVdBaHBxcHIzS0lYNWtpelpZNnhaZ2lxc3grdHZDMWhtdzVDN1IzdnB3YUREcVFuazV2M1d0YkRET1h0Z3AKdTd4RU5CaUFUS0FsTFNGYjUxbFNSdHJXUzdjZHhjMzZDK2R1akY4WmVrQlE4aTZSL1JoZnU0VFFoQ0xKZXdIQgpTQ2hRWDVCWmFTWmV1U0swcU9mYm1yMEJ0NmI0c0c1ZERaQnFBeHRWRUtXOVQrNDZOS0w1OEROalBSY1VEeHB3CmJhb3dHSlRpSEFhdW9VTzY5TnBISzZDUGVSeXlNNUtNUk9Ea0tBSTdTRHc4V2ZLK0w3UnpEK2JjbGttTFZkdFAKR2ZPaXFJRWRBblE2ZjhKTWF2U1RjQU11UzRPSThjcTdGNk1Kb1R5bnRpK0lKcUVkVm8xS3ltd05KVjkwUTl1SApPU3BNMU1DOCt1QXBScWZ1cDFVQ0F3RUFBYU1qTUNFd0RnWURWUjBQQVFIL0JBUURBZ0trTUE4R0ExVWRFd0VCCi93UUZNQU1CQWY4d0RRWUpLb1pJaHZjTkFRRUxCUUFEZ2dFQkFOVk1aT1JFeXJZbEhsWFRzYVNIQlBEdGVoWlIKQi9ZYmZOdzRWY3ZKYko2a0NVMm9lTDFNOTBFbDBCdVVnQ2w1WGlnaDh4ZkVmZkxNNE5XbEw2OEtYd1FCRWhJeQpLNFBOSk1lSzExRlloK01rbDFXYVRnazZ6VUg2YVkyTDRJZDNJTk9nSnVWTFI2SSsrbEZwSXJzR2ZUdlkyUUd4CkV4NDhocFNMeXFqdHVwQXd6K2NFeElrSjJvVHhLZ3YxVGlTMitDVWkwa2FUNWZCeEFMcjViNm5kbVZEbllJR3EKOGFBaHpIWkg3QWp6YW16SXdRTVY1TVptUWRWT2VleUtjdDBqTE1pa2I1RHRCWE9KSjlVT1crdmhCRTVPUGY4dQpISEdOdE1oRmJTZ1hnQUt0ekZnM0JoOVJ3VmZ3UCtCNldvYjRxVkZoRVRrdWd3Z1lWT253TlFYZlNzND0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo= --set submariner.clusterId=cluster2 --set submariner.clusterCidr=10.242.0.0/16 --set submariner.serviceCidr=100.92.0.0/16 --set submariner.globalCidr= --set serviceAccounts.globalnet.create=false --set submariner.natEnabled=false --set routeAgent.image.repository=localhost:5000/submariner-route-agent --set routeAgent.image.tag=local --set routeAgent.image.pullPolicy=IfNotPresent --set engine.image.repository=localhost:5000/submariner --set engine.image.tag=local --set engine.image.pullPolicy=IfNotPresent --set globalnet.image.repository=localhost:5000/submariner-globalnet --set globalnet.image.tag=local --set globalnet.image.pullPolicy=IfNotPresent --set crd.create=true --set submariner.serviceDiscovery=true,lighthouse.image.repository=localhost:5000/lighthouse-agent,lighthouse.image.tag=local,lighthouseCoredns.image.repository=localhost:5000/lighthouse-coredns,lighthouseCoredns.image.tag=local,serviceAccounts.lighthouse.create=true
  1654. 2020-10-08T11:34:48.2017310Z [cluster2] NAME: submariner
  1655. 2020-10-08T11:34:48.2017770Z [cluster2] LAST DEPLOYED: Thu Oct 8 11:34:34 2020
  1656. 2020-10-08T11:34:48.2018887Z [cluster2] NAMESPACE: submariner-operator
  1657. 2020-10-08T11:34:48.2019382Z [cluster2] STATUS: DEPLOYED
  1658. 2020-10-08T11:34:48.2019717Z [cluster2]
  1659. 2020-10-08T11:34:48.2020020Z [cluster2] RESOURCES:
  1660. 2020-10-08T11:34:48.2020719Z [lighthouse]$ [cluster2] wait 5443
  1661. 2020-10-08T11:34:48.2021488Z [lighthouse]$ [cluster2] with_context cluster2 with_retries 30 verify_gw_status
  1662. 2020-10-08T11:34:48.2022406Z [lighthouse]$ [cluster2] with_context cluster2 with_retries 30 verify_gw_status
  1663. 2020-10-08T11:34:48.2023208Z [lighthouse]$ [cluster2] local cluster=cluster2
  1664. 2020-10-08T11:34:48.2023937Z [lighthouse]$ [cluster2] local cmnd=with_retries
  1665. 2020-10-08T11:34:48.2024658Z [lighthouse]$ [cluster2] with_retries 30 verify_gw_status
  1666. 2020-10-08T11:34:48.2025324Z [lighthouse]$ [cluster2] with_retries
  1667. 2020-10-08T11:34:48.2025957Z [lighthouse]$ [cluster2] local retries
  1668. 2020-10-08T11:34:48.2026693Z [lighthouse]$ [cluster2] retries=1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30
  1669. 2020-10-08T11:34:48.2027337Z [lighthouse]$ [cluster2] eval echo {1..30}
  1670. 2020-10-08T11:34:48.2028001Z [lighthouse]$ [cluster2] local cmnd=verify_gw_status
  1671. 2020-10-08T11:34:48.2028614Z [lighthouse]$ [cluster2] wait 5512
  1672. 2020-10-08T11:34:48.2029218Z [lighthouse]$ [cluster2] verify_gw_status
  1673. 2020-10-08T11:34:48.2029857Z [lighthouse]$ [cluster2] verify_gw_status
  1674. 2020-10-08T11:34:48.2030494Z [lighthouse]$ [cluster2] sleep_duration=6
  1675. 2020-10-08T11:34:48.2031182Z [lighthouse]$ [cluster2] subctl show connections
  1676. 2020-10-08T11:34:48.2032011Z [lighthouse]$ [cluster2] grep the server could not find the requested resource
  1677. 2020-10-08T11:34:48.2032568Z [cluster2] ==> v1/ClusterRole
  1678. 2020-10-08T11:34:48.3302986Z [cluster2] NAME AGE
  1679. 2020-10-08T11:34:48.3303381Z [cluster2] submariner:routeagent 7s
  1680. 2020-10-08T11:34:48.3303851Z [cluster2] submariner:lighthouse 7s
  1681. 2020-10-08T11:34:48.3304223Z [cluster2]
  1682. 2020-10-08T11:34:48.3304628Z [cluster2] ==> v1/ClusterRoleBinding
  1683. 2020-10-08T11:34:48.3305056Z [cluster2] NAME AGE
  1684. 2020-10-08T11:34:48.3305453Z [cluster2] submariner:lighthouse 7s
  1685. 2020-10-08T11:34:48.3305918Z [cluster2] submariner:routeagent 7s
  1686. 2020-10-08T11:34:48.3306273Z [cluster2]
  1687. 2020-10-08T11:34:48.3306594Z [cluster2] ==> v1/ConfigMap
  1688. 2020-10-08T11:34:48.3306962Z [cluster2] NAME DATA AGE
  1689. 2020-10-08T11:34:48.3308143Z [cluster2] submariner-lighthouse-coredns 1 3s
  1690. 2020-10-08T11:34:48.3308697Z [cluster2]
  1691. 2020-10-08T11:34:48.3309014Z [cluster2] ==> v1/DaemonSet
  1692. 2020-10-08T11:34:48.3309835Z [cluster2] NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
  1693. 2020-10-08T11:34:48.3310847Z [cluster2] submariner-gateway 1 1 0 1 0 submariner.io/gateway=true 3s
  1694. 2020-10-08T11:34:48.3312573Z [cluster2] submariner-routeagent 2 2 0 2 0 <none> 3s
  1695. 2020-10-08T11:34:48.3313053Z [cluster2]
  1696. 2020-10-08T11:34:48.3313380Z [cluster2] ==> v1/Deployment
  1697. 2020-10-08T11:34:48.3314081Z [cluster2] NAME READY UP-TO-DATE AVAILABLE AGE
  1698. 2020-10-08T11:34:48.3314929Z [cluster2] submariner-lighthouse-agent 0/1 1 0 4s
  1699. 2020-10-08T11:34:48.3316394Z [cluster2] submariner-lighthouse-coredns 0/2 2 0 5s
  1700. 2020-10-08T11:34:48.3316963Z [cluster2]
  1701. 2020-10-08T11:34:48.3317282Z [cluster2] ==> v1/Pod(related)
  1702. 2020-10-08T11:34:48.3317942Z [cluster2] NAME READY STATUS RESTARTS AGE
  1703. 2020-10-08T11:34:48.3318882Z [cluster2] submariner-gateway-98rf6 0/1 ContainerCreating 0 5s
  1704. 2020-10-08T11:34:48.3320348Z [cluster2] submariner-lighthouse-agent-9df5c9df-c5kgq 0/1 ContainerCreating 0 5s
  1705. 2020-10-08T11:34:48.3322192Z [cluster2] submariner-lighthouse-coredns-7466b6679c-dppvv 0/1 ContainerCreating 0 5s
  1706. 2020-10-08T11:34:48.3324300Z [cluster2] submariner-lighthouse-coredns-7466b6679c-x5s2r 0/1 ContainerCreating 0 6s
  1707. 2020-10-08T11:34:48.3325863Z [cluster2] submariner-routeagent-clqmx 0/1 ContainerCreating 0 4s
  1708. 2020-10-08T11:34:48.3327079Z [cluster2] submariner-routeagent-gk7px 0/1 ContainerCreating 0 5s
  1709. 2020-10-08T11:34:48.3327718Z [cluster2]
  1710. 2020-10-08T11:34:48.3328007Z [cluster2] ==> v1/Role
  1711. 2020-10-08T11:34:48.3328328Z [cluster2] NAME AGE
  1712. 2020-10-08T11:34:48.3328711Z [cluster2] submariner:routeagent 7s
  1713. 2020-10-08T11:34:48.3329163Z [cluster2] submariner:engine 7s
  1714. 2020-10-08T11:34:48.3329509Z [cluster2]
  1715. 2020-10-08T11:34:48.3329840Z [cluster2] ==> v1/RoleBinding
  1716. 2020-10-08T11:34:48.3330205Z [cluster2] NAME AGE
  1717. 2020-10-08T11:34:48.3330602Z [cluster2] submariner:routeagent 7s
  1718. 2020-10-08T11:34:48.3331043Z [cluster2] submariner:engine 7s
  1719. 2020-10-08T11:34:48.3331392Z [cluster2]
  1720. 2020-10-08T11:34:48.3331679Z [cluster2] ==> v1/Service
  1721. 2020-10-08T11:34:48.3332850Z [cluster2] NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
  1722. 2020-10-08T11:34:48.3333801Z [cluster2] submariner-lighthouse-coredns ClusterIP 100.92.17.108 <none> 53/UDP 3s
  1723. 2020-10-08T11:34:48.3334385Z [cluster2]
  1724. 2020-10-08T11:34:48.3334720Z [cluster2] ==> v1/ServiceAccount
  1725. 2020-10-08T11:34:48.3335107Z [cluster2] NAME SECRETS AGE
  1726. 2020-10-08T11:34:48.3335664Z [cluster2] submariner-engine 1 3s
  1727. 2020-10-08T11:34:48.3336316Z [cluster2] submariner-lighthouse 1 4s
  1728. 2020-10-08T11:34:48.3336999Z [cluster2] submariner-routeagent 1 4s
  1729. 2020-10-08T11:34:48.3337400Z [cluster2]
  1730. 2020-10-08T11:34:48.3337651Z [cluster2]
  1731. 2020-10-08T11:34:48.3437468Z [cluster2] NOTES:
  1732. 2020-10-08T11:34:48.3437847Z [cluster2] Submariner is now installed.
  1733. 2020-10-08T11:34:48.3439139Z [cluster2] If you haven't done so yet, please label a node as `submariner.io/gateway=true` to elect it for running Submariner.
  1734. 2020-10-08T11:34:48.3439734Z [cluster2]
  1735. 2020-10-08T11:34:48.3440550Z [cluster2] By default, Submariner runs with 1 replica. If you have more than one Gateway host, you can scale Submariner to N replicas, and the other Submariner pods will simply join the leader election pool.
  1736. 2020-10-08T11:34:50.9733218Z error reading from submariner client: the server could not find the requested resource (get submariners.submariner.io submariner)
  1737. 2020-10-08T11:34:50.9749440Z [lighthouse]$ [cluster2] return 0
  1738. 2020-10-08T11:34:50.9764232Z [lighthouse]$ [cluster2] return 0
  1739. 2020-10-08T11:34:50.9782425Z [lighthouse]$ [cluster2] with_context cluster2 connectivity_tests cluster1
  1740. 2020-10-08T11:34:50.9798995Z [lighthouse]$ [cluster2] with_context cluster2 connectivity_tests cluster1
  1741. 2020-10-08T11:34:50.9814188Z [lighthouse]$ [cluster2] local cluster=cluster2
  1742. 2020-10-08T11:34:50.9826141Z [lighthouse]$ [cluster2] local cmnd=connectivity_tests
  1743. 2020-10-08T11:34:50.9838325Z [lighthouse]$ [cluster2] connectivity_tests cluster1
  1744. 2020-10-08T11:34:50.9850516Z [lighthouse]$ [cluster2] connectivity_tests
  1745. 2020-10-08T11:34:50.9862418Z [lighthouse]$ [cluster2] target_cluster=cluster1
  1746. 2020-10-08T11:34:50.9879645Z [lighthouse]$ [cluster2] deploy_resource /opt/shipyard/scripts/resources/netshoot.yaml
  1747. 2020-10-08T11:34:50.9889408Z [lighthouse]$ [cluster2] deploy_resource /opt/shipyard/scripts/resources/netshoot.yaml
  1748. 2020-10-08T11:34:50.9900946Z [lighthouse]$ [cluster2] local resource_file=/opt/shipyard/scripts/resources/netshoot.yaml
  1749. 2020-10-08T11:34:50.9910437Z [lighthouse]$ [cluster2] local resource_name
  1750. 2020-10-08T11:34:50.9939894Z [lighthouse]$ [cluster2] resource_name=netshoot
  1751. 2020-10-08T11:34:50.9955459Z [lighthouse]$ [cluster2] basename /opt/shipyard/scripts/resources/netshoot.yaml .yaml
  1752. 2020-10-08T11:34:50.9974532Z [lighthouse]$ [cluster2] kubectl apply -f /opt/shipyard/scripts/resources/netshoot.yaml
  1753. 2020-10-08T11:34:50.9984149Z [lighthouse]$ [cluster2] kubectl apply -f /opt/shipyard/scripts/resources/netshoot.yaml
  1754. 2020-10-08T11:34:50.9997375Z [lighthouse]$ [cluster2] command kubectl --context=cluster2 apply -f /opt/shipyard/scripts/resources/netshoot.yaml
  1755. 2020-10-08T11:34:51.0009476Z [lighthouse]$ [cluster2] kubectl --context=cluster2 apply -f /opt/shipyard/scripts/resources/netshoot.yaml
  1756. 2020-10-08T11:34:53.0749409Z deployment.apps/netshoot created
  1757. 2020-10-08T11:34:53.0940413Z Waiting for netshoot pods to be ready.
  1758. 2020-10-08T11:34:53.1029633Z [lighthouse]$ [cluster2] kubectl rollout status deploy/netshoot --timeout=5m
  1759. 2020-10-08T11:34:53.1095521Z [lighthouse]$ [cluster2] kubectl rollout status deploy/netshoot --timeout=5m
  1760. 2020-10-08T11:34:53.1187907Z [lighthouse]$ [cluster2] command kubectl --context=cluster2 rollout status deploy/netshoot --timeout=5m
  1761. 2020-10-08T11:34:53.1295772Z [lighthouse]$ [cluster2] kubectl --context=cluster2 rollout status deploy/netshoot --timeout=5m
  1762. 2020-10-08T11:34:55.7471139Z Waiting for deployment "netshoot" rollout to finish: 0 of 1 updated replicas are available...
  1763. 2020-10-08T11:35:31.5086536Z deployment "netshoot" successfully rolled out
  1764. 2020-10-08T11:35:31.5169668Z [lighthouse]$ [cluster2] with_context cluster1 deploy_resource /opt/shipyard/scripts/resources/nginx-demo.yaml
  1765. 2020-10-08T11:35:31.5181975Z [lighthouse]$ [cluster2] with_context cluster1 deploy_resource /opt/shipyard/scripts/resources/nginx-demo.yaml
  1766. 2020-10-08T11:35:31.5194526Z [lighthouse]$ [cluster2] local cluster=cluster1
  1767. 2020-10-08T11:35:31.5203556Z [lighthouse]$ [cluster1] local cmnd=deploy_resource
  1768. 2020-10-08T11:35:31.5216498Z [lighthouse]$ [cluster1] deploy_resource /opt/shipyard/scripts/resources/nginx-demo.yaml
  1769. 2020-10-08T11:35:31.5229014Z [lighthouse]$ [cluster1] deploy_resource
  1770. 2020-10-08T11:35:31.5242092Z [lighthouse]$ [cluster1] local resource_file=/opt/shipyard/scripts/resources/nginx-demo.yaml
  1771. 2020-10-08T11:35:31.5255173Z [lighthouse]$ [cluster1] local resource_name
  1772. 2020-10-08T11:35:31.5312029Z [lighthouse]$ [cluster1] resource_name=nginx-demo
  1773. 2020-10-08T11:35:31.5327809Z [lighthouse]$ [cluster1] basename /opt/shipyard/scripts/resources/nginx-demo.yaml .yaml
  1774. 2020-10-08T11:35:31.5346904Z [lighthouse]$ [cluster1] kubectl apply -f /opt/shipyard/scripts/resources/nginx-demo.yaml
  1775. 2020-10-08T11:35:31.5357601Z [lighthouse]$ [cluster1] kubectl apply -f /opt/shipyard/scripts/resources/nginx-demo.yaml
  1776. 2020-10-08T11:35:31.5371829Z [lighthouse]$ [cluster1] command kubectl --context=cluster1 apply -f /opt/shipyard/scripts/resources/nginx-demo.yaml
  1777. 2020-10-08T11:35:31.5383382Z [lighthouse]$ [cluster1] kubectl --context=cluster1 apply -f /opt/shipyard/scripts/resources/nginx-demo.yaml
  1778. 2020-10-08T11:35:32.5536647Z deployment.apps/nginx-demo created
  1779. 2020-10-08T11:35:32.5750466Z service/nginx-demo created
  1780. 2020-10-08T11:35:32.5878617Z Waiting for nginx-demo pods to be ready.
  1781. 2020-10-08T11:35:32.5896120Z [lighthouse]$ [cluster1] kubectl rollout status deploy/nginx-demo --timeout=5m
  1782. 2020-10-08T11:35:32.5911491Z [lighthouse]$ [cluster1] kubectl rollout status deploy/nginx-demo --timeout=5m
  1783. 2020-10-08T11:35:32.5926225Z [lighthouse]$ [cluster1] command kubectl --context=cluster1 rollout status deploy/nginx-demo --timeout=5m
  1784. 2020-10-08T11:35:32.5941176Z [lighthouse]$ [cluster1] kubectl --context=cluster1 rollout status deploy/nginx-demo --timeout=5m
  1785. 2020-10-08T11:35:33.6599980Z Waiting for deployment "nginx-demo" rollout to finish: 0 of 2 updated replicas are available...
  1786. 2020-10-08T11:35:36.2260117Z Waiting for deployment "nginx-demo" rollout to finish: 1 of 2 updated replicas are available...
  1787. 2020-10-08T11:35:36.7465416Z deployment "nginx-demo" successfully rolled out
  1788. 2020-10-08T11:35:36.7574934Z [lighthouse]$ [cluster2] local netshoot_pod nginx_svc_ip
  1789. 2020-10-08T11:35:37.5713918Z [lighthouse]$ [cluster2] netshoot_pod=netshoot-789f6cf54f-jnwxg
  1790. 2020-10-08T11:35:37.5729551Z [lighthouse]$ [cluster2] kubectl get pods -l app=netshoot
  1791. 2020-10-08T11:35:37.5744925Z [lighthouse]$ [cluster2] awk FNR == 2 {print $1}
  1792. 2020-10-08T11:35:37.5752706Z [lighthouse]$ [cluster2] kubectl get pods -l app=netshoot
  1793. 2020-10-08T11:35:37.5767410Z [lighthouse]$ [cluster2] command kubectl --context=cluster2 get pods -l app=netshoot
  1794. 2020-10-08T11:35:37.5782001Z [lighthouse]$ [cluster2] kubectl --context=cluster2 get pods -l app=netshoot
  1795. 2020-10-08T11:35:39.0823294Z [lighthouse]$ [cluster2] nginx_svc_ip=100.91.28.91
  1796. 2020-10-08T11:35:39.0840789Z [lighthouse]$ [cluster2] with_context cluster1 get_svc_ip nginx-demo
  1797. 2020-10-08T11:35:39.0853569Z [lighthouse]$ [cluster2] with_context cluster1 get_svc_ip nginx-demo
  1798. 2020-10-08T11:35:39.0866540Z [lighthouse]$ [cluster2] local cluster=cluster1
  1799. 2020-10-08T11:35:39.0880055Z [lighthouse]$ [cluster1] local cmnd=get_svc_ip
  1800. 2020-10-08T11:35:39.0890308Z [lighthouse]$ [cluster1] get_svc_ip nginx-demo
  1801. 2020-10-08T11:35:39.0904842Z [lighthouse]$ [cluster1] get_svc_ip
  1802. 2020-10-08T11:35:39.0917538Z [lighthouse]$ [cluster1] local svc_name=nginx-demo
  1803. 2020-10-08T11:35:39.0929358Z [lighthouse]$ [cluster1] local svc_ip
  1804. 2020-10-08T11:35:39.8282216Z [lighthouse]$ [cluster1] svc_ip=100.91.28.91
  1805. 2020-10-08T11:35:39.8300930Z [lighthouse]$ [cluster1] kubectl --context=cluster1 get svc -l app=nginx-demo
  1806. 2020-10-08T11:35:39.8315911Z [lighthouse]$ [cluster1] awk FNR == 2 {print $3}
  1807. 2020-10-08T11:35:39.8320682Z [lighthouse]$ [cluster1] kubectl --context=cluster1 get svc -l app=nginx-demo
  1808. 2020-10-08T11:35:39.8337737Z [lighthouse]$ [cluster1] command kubectl --context=cluster1 get svc -l app=nginx-demo
  1809. 2020-10-08T11:35:39.8349892Z [lighthouse]$ [cluster1] kubectl --context=cluster1 get svc -l app=nginx-demo
  1810. 2020-10-08T11:35:40.5719216Z [lighthouse]$ [cluster2] with_retries 5 test_connection netshoot-789f6cf54f-jnwxg 100.91.28.91
  1811. 2020-10-08T11:35:40.5734328Z [lighthouse]$ [cluster2] with_retries 5 test_connection netshoot-789f6cf54f-jnwxg 100.91.28.91
  1812. 2020-10-08T11:35:40.5747012Z [lighthouse]$ [cluster2] local retries
  1813. 2020-10-08T11:35:40.5764674Z [lighthouse]$ [cluster2] retries=1 2 3 4 5
  1814. 2020-10-08T11:35:40.5782440Z [lighthouse]$ [cluster2] eval echo {1..5}
  1815. 2020-10-08T11:35:40.5799340Z [lighthouse]$ [cluster2] local cmnd=test_connection
  1816. 2020-10-08T11:35:40.5816623Z [lighthouse]$ [cluster2] wait 5680
  1817. 2020-10-08T11:35:40.5823950Z [lighthouse]$ [cluster2] test_connection netshoot-789f6cf54f-jnwxg 100.91.28.91
  1818. 2020-10-08T11:35:40.5835692Z [lighthouse]$ [cluster2] test_connection
  1819. 2020-10-08T11:35:40.5846208Z [lighthouse]$ [cluster2] local source_pod=netshoot-789f6cf54f-jnwxg
  1820. 2020-10-08T11:35:40.5858220Z [lighthouse]$ [cluster2] local target_address=100.91.28.91
  1821. 2020-10-08T11:35:40.5859186Z Attempting connectivity between clusters - netshoot-789f6cf54f-jnwxg --> 100.91.28.91
  1822. 2020-10-08T11:35:40.5868805Z [lighthouse]$ [cluster2] wait_time=30
  1823. 2020-10-08T11:35:40.5916574Z [lighthouse]$ [cluster2] start_time=1602156940
  1824. 2020-10-08T11:35:40.5932065Z [lighthouse]$ [cluster2] date +%s
  1825. 2020-10-08T11:35:40.5952762Z [lighthouse]$ [cluster2] kubectl exec netshoot-789f6cf54f-jnwxg -- curl --output /dev/null -m 30 --silent --head --fail 100.91.28.91
  1826. 2020-10-08T11:35:40.5963832Z [lighthouse]$ [cluster2] kubectl exec netshoot-789f6cf54f-jnwxg -- curl --output /dev/null -m 30 --silent --head --fail 100.91.28.91
  1827. 2020-10-08T11:35:40.5977626Z [lighthouse]$ [cluster2] command kubectl --context=cluster2 exec netshoot-789f6cf54f-jnwxg -- curl --output /dev/null -m 30 --silent --head --fail 100.91.28.91
  1828. 2020-10-08T11:35:40.5988186Z [lighthouse]$ [cluster2] kubectl --context=cluster2 exec netshoot-789f6cf54f-jnwxg -- curl --output /dev/null -m 30 --silent --head --fail 100.91.28.91
  1829. 2020-10-08T11:35:41.4253793Z Connection test was successful!
  1830. 2020-10-08T11:35:41.4269204Z [lighthouse]$ [cluster2] return 0
  1831. 2020-10-08T11:35:41.4279561Z [lighthouse]$ [cluster2] remove_resource /opt/shipyard/scripts/resources/netshoot.yaml
  1832. 2020-10-08T11:35:41.4294073Z [lighthouse]$ [cluster2] remove_resource /opt/shipyard/scripts/resources/netshoot.yaml
  1833. 2020-10-08T11:35:41.4307379Z [lighthouse]$ [cluster2] local resource_file=/opt/shipyard/scripts/resources/netshoot.yaml
  1834. 2020-10-08T11:35:41.4322099Z [lighthouse]$ [cluster2] kubectl delete -f /opt/shipyard/scripts/resources/netshoot.yaml
  1835. 2020-10-08T11:35:41.4337829Z [lighthouse]$ [cluster2] kubectl delete -f /opt/shipyard/scripts/resources/netshoot.yaml
  1836. 2020-10-08T11:35:41.4352001Z [lighthouse]$ [cluster2] command kubectl --context=cluster2 delete -f /opt/shipyard/scripts/resources/netshoot.yaml
  1837. 2020-10-08T11:35:41.4367569Z [lighthouse]$ [cluster2] kubectl --context=cluster2 delete -f /opt/shipyard/scripts/resources/netshoot.yaml
  1838. 2020-10-08T11:35:42.1870705Z deployment.apps "netshoot" deleted
  1839. 2020-10-08T11:35:42.2001405Z [lighthouse]$ [cluster2] with_context cluster1 remove_resource /opt/shipyard/scripts/resources/nginx-demo.yaml
  1840. 2020-10-08T11:35:42.2015741Z [lighthouse]$ [cluster2] with_context cluster1 remove_resource /opt/shipyard/scripts/resources/nginx-demo.yaml
  1841. 2020-10-08T11:35:42.2027797Z [lighthouse]$ [cluster2] local cluster=cluster1
  1842. 2020-10-08T11:35:42.2040657Z [lighthouse]$ [cluster1] local cmnd=remove_resource
  1843. 2020-10-08T11:35:42.2053598Z [lighthouse]$ [cluster1] remove_resource /opt/shipyard/scripts/resources/nginx-demo.yaml
  1844. 2020-10-08T11:35:42.2076898Z [lighthouse]$ [cluster1] remove_resource
  1845. 2020-10-08T11:35:42.2089679Z [lighthouse]$ [cluster1] local resource_file=/opt/shipyard/scripts/resources/nginx-demo.yaml
  1846. 2020-10-08T11:35:42.2102306Z [lighthouse]$ [cluster1] kubectl delete -f /opt/shipyard/scripts/resources/nginx-demo.yaml
  1847. 2020-10-08T11:35:42.2123419Z [lighthouse]$ [cluster1] kubectl delete -f /opt/shipyard/scripts/resources/nginx-demo.yaml
  1848. 2020-10-08T11:35:42.2137505Z [lighthouse]$ [cluster1] command kubectl --context=cluster1 delete -f /opt/shipyard/scripts/resources/nginx-demo.yaml
  1849. 2020-10-08T11:35:42.2150454Z [lighthouse]$ [cluster1] kubectl --context=cluster1 delete -f /opt/shipyard/scripts/resources/nginx-demo.yaml
  1850. 2020-10-08T11:35:43.4362849Z [lighthouse]$ run_subm_clusters update_coredns_configmap
  1851. 2020-10-08T11:35:43.4363522Z deployment.apps "nginx-demo" deleted
  1852. 2020-10-08T11:35:43.4370573Z service "nginx-demo" deleted
  1853. 2020-10-08T11:35:43.4373498Z [lighthouse]$ run_subm_clusters update_coredns_configmap
  1854. 2020-10-08T11:35:43.4374602Z [lighthouse]$ declare -a subm_clusters
  1855. 2020-10-08T11:35:43.4375584Z [lighthouse]$ [cluster2] run_parallel cluster1 cluster2 update_coredns_configmap
  1856. 2020-10-08T11:35:43.4376796Z [lighthouse]$ [cluster2] run_parallel cluster1 cluster2 cluster1 cluster2 update_coredns_configmap
  1857. 2020-10-08T11:35:43.4377902Z [lighthouse]$ [cluster2] local cmnd=update_coredns_configmap
  1858. 2020-10-08T11:35:43.4378797Z [lighthouse]$ [cluster2] declare -A pids
  1859. 2020-10-08T11:35:43.4379652Z [lighthouse]$ [cluster2] eval echo cluster1 cluster2
  1860. 2020-10-08T11:35:43.4380629Z [lighthouse]$ [cluster1] pids[cluster1]=5747
  1861. 2020-10-08T11:35:43.4381359Z [lighthouse]$ [cluster1] set -o pipefail
  1862. 2020-10-08T11:35:43.4382388Z [lighthouse]$ [cluster2] pids[cluster2]=5750
  1863. 2020-10-08T11:35:43.4383273Z [lighthouse]$ [cluster1] update_coredns_configmap
  1864. 2020-10-08T11:35:43.4384526Z [lighthouse]$ [cluster2] set -o pipefail
  1865. 2020-10-08T11:35:43.4385367Z [lighthouse]$ [cluster2] wait 5750
  1866. 2020-10-08T11:35:43.4386257Z [lighthouse]$ [cluster1] sed /\[cluster1]/!s/^/[cluster1] /
  1867. 2020-10-08T11:35:43.4387212Z [lighthouse]$ [cluster2] update_coredns_configmap
  1868. 2020-10-08T11:35:43.4388279Z [lighthouse]$ [cluster2] sed /\[cluster2]/!s/^/[cluster2] /
  1869. 2020-10-08T11:35:48.3395735Z [lighthouse]$ [cluster2] update_coredns_configmap
  1870. 2020-10-08T11:35:48.3396363Z [cluster2] 
  1871. 2020-10-08T11:35:48.3396784Z [cluster2] 
  1872. 2020-10-08T11:35:48.3397574Z [lighthouse]$ [cluster2] command kubectl --context=cluster2 get cm -n kube-system coredns -o yaml
  1873. 2020-10-08T11:35:48.3398612Z [lighthouse]$ [cluster2] kubectl --context=cluster2 get cm -n kube-system coredns -o yaml
  1874. 2020-10-08T11:35:48.3399649Z [lighthouse]$ [cluster2] grep -q clusterset /tmp/coremap-cluster2.yaml
  1875. 2020-10-08T11:35:48.3400482Z [lighthouse]$ [cluster2] CLUSTER_IP=100.92.17.108
  1876. 2020-10-08T11:35:48.3401571Z [lighthouse]$ [cluster2] kubectl get svc -n submariner-operator submariner-lighthouse-coredns
  1877. 2020-10-08T11:35:48.3402579Z [lighthouse]$ [cluster2] tail -n 1
  1878. 2020-10-08T11:35:48.3403637Z [lighthouse]$ [cluster2] kubectl get svc -n submariner-operator submariner-lighthouse-coredns
  1879. 2020-10-08T11:35:48.3404863Z [lighthouse]$ [cluster2] awk {print $3;}
  1880. 2020-10-08T11:35:48.3406060Z [lighthouse]$ [cluster2] command kubectl --context=cluster2 get svc -n submariner-operator submariner-lighthouse-coredns
  1881. 2020-10-08T11:35:48.3407733Z [lighthouse]$ [cluster2] kubectl --context=cluster2 get svc -n submariner-operator submariner-lighthouse-coredns
  1882. 2020-10-08T11:35:48.3408714Z [cluster2] 
  1883. 2020-10-08T11:35:48.3409688Z [lighthouse]$ [cluster2] sed -i -e /Corefile:/r /tmp/coredns-cm-100.92.17.108.yaml /tmp/coremap-cluster2.yaml
  1884. 2020-10-08T11:35:48.3410946Z [lighthouse]$ [cluster2] kubectl -n kube-system replace -f /tmp/coremap-cluster2.yaml
  1885. 2020-10-08T11:35:48.3412338Z [lighthouse]$ [cluster2] kubectl -n kube-system replace -f /tmp/coremap-cluster2.yaml
  1886. 2020-10-08T11:35:48.3413757Z [lighthouse]$ [cluster2] command kubectl --context=cluster2 -n kube-system replace -f /tmp/coremap-cluster2.yaml
  1887. 2020-10-08T11:35:48.3415055Z [lighthouse]$ [cluster2] kubectl --context=cluster2 -n kube-system replace -f /tmp/coremap-cluster2.yaml
  1888. 2020-10-08T11:35:48.3415826Z [cluster2] configmap/coredns replaced
  1889. 2020-10-08T11:35:48.3416586Z [lighthouse]$ [cluster2] kubectl -n kube-system describe cm coredns
  1890. 2020-10-08T11:35:48.3417452Z [lighthouse]$ [cluster2] kubectl -n kube-system describe cm coredns
  1891. 2020-10-08T11:35:48.3418438Z [lighthouse]$ [cluster2] command kubectl --context=cluster2 -n kube-system describe cm coredns
  1892. 2020-10-08T11:35:48.3419470Z [lighthouse]$ [cluster2] kubectl --context=cluster2 -n kube-system describe cm coredns
  1893. 2020-10-08T11:35:48.3420058Z [cluster2] Name: coredns
  1894. 2020-10-08T11:35:48.3420608Z [cluster2] Namespace: kube-system
  1895. 2020-10-08T11:35:48.3421012Z [cluster2] Labels: <none>
  1896. 2020-10-08T11:35:48.3421379Z [cluster2] Annotations: <none>
  1897. 2020-10-08T11:35:48.3421721Z [cluster2]
  1898. 2020-10-08T11:35:48.3422004Z [cluster2] Data
  1899. 2020-10-08T11:35:48.3422288Z [cluster2] ====
  1900. 2020-10-08T11:35:48.3422594Z [cluster2] Corefile:
  1901. 2020-10-08T11:35:48.3423031Z [cluster2] ----
  1902. 2020-10-08T11:35:48.3423409Z [cluster2] clusterset.local:53 {
  1903. 2020-10-08T11:35:48.3423805Z [cluster2] forward . 100.92.17.108
  1904. 2020-10-08T11:35:48.3424112Z [cluster2] }
  1905. 2020-10-08T11:35:48.3424387Z [cluster2] .:53 {
  1906. 2020-10-08T11:35:48.3424680Z [cluster2] errors
  1907. 2020-10-08T11:35:48.3424997Z [cluster2] health {
  1908. 2020-10-08T11:35:48.3425332Z [cluster2] lameduck 5s
  1909. 2020-10-08T11:35:48.3425819Z [cluster2] }
  1910. 2020-10-08T11:35:48.3426100Z [cluster2] ready
  1911. 2020-10-08T11:35:48.3426955Z [cluster2] kubernetes cluster2.local in-addr.arpa ip6.arpa {
  1912. 2020-10-08T11:35:48.3427486Z [cluster2] pods insecure
  1913. 2020-10-08T11:35:48.3428096Z [cluster2] fallthrough in-addr.arpa ip6.arpa
  1914. 2020-10-08T11:35:48.3428514Z [cluster2] ttl 30
  1915. 2020-10-08T11:35:48.3428799Z [cluster2] }
  1916. 2020-10-08T11:35:48.3429122Z [cluster2] prometheus :9153
  1917. 2020-10-08T11:35:48.3429518Z [cluster2] forward . /etc/resolv.conf
  1918. 2020-10-08T11:35:48.3429893Z [cluster2] cache 30
  1919. 2020-10-08T11:35:48.3430195Z [cluster2] loop
  1920. 2020-10-08T11:35:48.3430495Z [cluster2] reload
  1921. 2020-10-08T11:35:48.3430833Z [cluster2] loadbalance
  1922. 2020-10-08T11:35:48.3431158Z [cluster2] }
  1923. 2020-10-08T11:35:48.3431429Z [cluster2]
  1924. 2020-10-08T11:35:48.3431727Z [cluster2] Events: <none>
  1925. 2020-10-08T11:35:48.3499643Z [lighthouse]$ [cluster2] wait 5747
  1926. 2020-10-08T11:35:48.3778461Z [lighthouse]$ [cluster1] update_coredns_configmap
  1927. 2020-10-08T11:35:48.3779081Z [cluster1] 
  1928. 2020-10-08T11:35:48.3779682Z [cluster1] 
  1929. 2020-10-08T11:35:48.3780483Z [lighthouse]$ [cluster1] command kubectl --context=cluster1 get cm -n kube-system coredns -o yaml
  1930. 2020-10-08T11:35:48.3781514Z [lighthouse]$ [cluster1] kubectl --context=cluster1 get cm -n kube-system coredns -o yaml
  1931. 2020-10-08T11:35:48.3782521Z [lighthouse]$ [cluster1] grep -q clusterset /tmp/coremap-cluster1.yaml
  1932. 2020-10-08T11:35:48.3783356Z [lighthouse]$ [cluster1] CLUSTER_IP=100.91.206.45
  1933. 2020-10-08T11:35:48.3784443Z [lighthouse]$ [cluster1] kubectl get svc -n submariner-operator submariner-lighthouse-coredns
  1934. 2020-10-08T11:35:48.3785452Z [lighthouse]$ [cluster1] tail -n 1
  1935. 2020-10-08T11:35:48.3786496Z [lighthouse]$ [cluster1] kubectl get svc -n submariner-operator submariner-lighthouse-coredns
  1936. 2020-10-08T11:35:48.3787518Z [lighthouse]$ [cluster1] awk {print $3;}
  1937. 2020-10-08T11:35:48.3788715Z [lighthouse]$ [cluster1] command kubectl --context=cluster1 get svc -n submariner-operator submariner-lighthouse-coredns
  1938. 2020-10-08T11:35:48.3790748Z [lighthouse]$ [cluster1] kubectl --context=cluster1 get svc -n submariner-operator submariner-lighthouse-coredns
  1939. 2020-10-08T11:35:48.3791733Z [cluster1] 
  1940. 2020-10-08T11:35:48.3792710Z [lighthouse]$ [cluster1] sed -i -e /Corefile:/r /tmp/coredns-cm-100.91.206.45.yaml /tmp/coremap-cluster1.yaml
  1941. 2020-10-08T11:35:48.3793968Z [lighthouse]$ [cluster1] kubectl -n kube-system replace -f /tmp/coremap-cluster1.yaml
  1942. 2020-10-08T11:35:48.3795091Z [lighthouse]$ [cluster1] kubectl -n kube-system replace -f /tmp/coremap-cluster1.yaml
  1943. 2020-10-08T11:35:48.3796335Z [lighthouse]$ [cluster1] command kubectl --context=cluster1 -n kube-system replace -f /tmp/coremap-cluster1.yaml
  1944. 2020-10-08T11:35:48.3797653Z [lighthouse]$ [cluster1] kubectl --context=cluster1 -n kube-system replace -f /tmp/coremap-cluster1.yaml
  1945. 2020-10-08T11:35:48.3798421Z [cluster1] configmap/coredns replaced
  1946. 2020-10-08T11:35:48.3799169Z [lighthouse]$ [cluster1] kubectl -n kube-system describe cm coredns
  1947. 2020-10-08T11:35:48.3800040Z [lighthouse]$ [cluster1] kubectl -n kube-system describe cm coredns
  1948. 2020-10-08T11:35:48.3801017Z [lighthouse]$ [cluster1] command kubectl --context=cluster1 -n kube-system describe cm coredns
  1949. 2020-10-08T11:35:48.3802056Z [lighthouse]$ [cluster1] kubectl --context=cluster1 -n kube-system describe cm coredns
  1950. 2020-10-08T11:35:48.3802641Z [cluster1] Name: coredns
  1951. 2020-10-08T11:35:48.3803191Z [cluster1] Namespace: kube-system
  1952. 2020-10-08T11:35:48.3803595Z [cluster1] Labels: <none>
  1953. 2020-10-08T11:35:48.3803976Z [cluster1] Annotations: <none>
  1954. 2020-10-08T11:35:48.3804303Z [cluster1]
  1955. 2020-10-08T11:35:48.3804586Z [cluster1] Data
  1956. 2020-10-08T11:35:48.3804869Z [cluster1] ====
  1957. 2020-10-08T11:35:48.3805179Z [cluster1] Corefile:
  1958. 2020-10-08T11:35:48.3805613Z [cluster1] ----
  1959. 2020-10-08T11:35:48.3806095Z [cluster1] clusterset.local:53 {
  1960. 2020-10-08T11:35:48.3806513Z [cluster1] forward . 100.91.206.45
  1961. 2020-10-08T11:35:48.3806807Z [cluster1] }
  1962. 2020-10-08T11:35:48.3807082Z [cluster1] .:53 {
  1963. 2020-10-08T11:35:48.3807377Z [cluster1] errors
  1964. 2020-10-08T11:35:48.3807691Z [cluster1] health {
  1965. 2020-10-08T11:35:48.3808029Z [cluster1] lameduck 5s
  1966. 2020-10-08T11:35:48.3808339Z [cluster1] }
  1967. 2020-10-08T11:35:48.3808628Z [cluster1] ready
  1968. 2020-10-08T11:35:48.3809296Z [cluster1] kubernetes cluster1.local in-addr.arpa ip6.arpa {
  1969. 2020-10-08T11:35:48.3809817Z [cluster1] pods insecure
  1970. 2020-10-08T11:35:48.3810428Z [cluster1] fallthrough in-addr.arpa ip6.arpa
  1971. 2020-10-08T11:35:48.3810846Z [cluster1] ttl 30
  1972. 2020-10-08T11:35:48.3811130Z [cluster1] }
  1973. 2020-10-08T11:35:48.3811452Z [cluster1] prometheus :9153
  1974. 2020-10-08T11:35:48.3811861Z [cluster1] forward . /etc/resolv.conf
  1975. 2020-10-08T11:35:48.3812222Z [cluster1] cache 30
  1976. 2020-10-08T11:35:48.3813143Z [cluster1] loop
  1977. 2020-10-08T11:35:48.3813449Z [cluster1] reload
  1978. 2020-10-08T11:35:48.3813787Z [cluster1] loadbalance
  1979. 2020-10-08T11:35:48.3814113Z [cluster1] }
  1980. 2020-10-08T11:35:48.3814387Z [cluster1]
  1981. 2020-10-08T11:35:48.3814700Z [cluster1] Events: <none>
  1982. 2020-10-08T11:35:48.3831383Z make[1]: Leaving directory '/go/src/github.com/submariner-io/lighthouse'
  1983. 2020-10-08T11:35:48.3875879Z [lighthouse]$ declare_kubeconfig
  1984. 2020-10-08T11:35:48.3888196Z [lighthouse]$ declare_kubeconfig
  1985. 2020-10-08T11:35:48.3901365Z [lighthouse]$ source /opt/shipyard/scripts/lib/kubecfg
  1986. 2020-10-08T11:35:48.3913697Z [lighthouse]$ export KUBECONFIG
  1987. 2020-10-08T11:35:48.3957333Z [lighthouse]$ KUBECONFIG=/go/src/github.com/submariner-io/lighthouse/output/kubeconfigs/kind-config-cluster2:/go/src/github.com/submariner-io/lighthouse/output/kubeconfigs/kind-config-cluster1:
  1988. 2020-10-08T11:35:48.3973301Z [lighthouse]$ find /go/src/github.com/submariner-io/lighthouse/output/kubeconfigs -type f -printf %p:
  1989. 2020-10-08T11:35:48.4008388Z [lighthouse]$ test_with_e2e_tests
  1990. 2020-10-08T11:35:48.4021112Z [lighthouse]$ test_with_e2e_tests
  1991. 2020-10-08T11:35:48.4033817Z [lighthouse]$ cd /go/src/github.com/submariner-io/lighthouse/test/e2e
  1992. 2020-10-08T11:35:48.4061797Z [e2e]$ go test -v -timeout 30m -args -ginkgo.v -ginkgo.randomizeAllSpecs -ginkgo.trace -submariner-namespace submariner-operator -dp-context cluster1 -dp-context cluster2 -ginkgo.reportPassed -test.timeout 15m -ginkgo.focus \[.*\] -ginkgo.reportFile /go/src/github.com/submariner-io/lighthouse/output/e2e-junit.xml
  1993. 2020-10-08T11:35:48.4074036Z [e2e]$ tee /go/src/github.com/submariner-io/lighthouse/output/e2e-tests.log
  1994. 2020-10-08T11:35:48.4095584Z [e2e]$ generate_context_flags
  1995. 2020-10-08T11:35:48.4106721Z [e2e]$ generate_context_flags
  1996. 2020-10-08T11:35:48.4120901Z [e2e]$ [cluster1] printf -dp-context cluster1
  1997. 2020-10-08T11:35:48.4134824Z [e2e]$ [cluster2] printf -dp-context cluster2
  1998. 2020-10-08T11:36:41.4476306Z === RUN TestE2E
  1999. 2020-10-08T11:36:41.4508078Z Running Suite: Submariner E2E suite
  2000. 2020-10-08T11:36:41.4511785Z ===================================
  2001. 2020-10-08T11:36:41.4516535Z Random Seed: 1602157001 - Will randomize all specs
  2002. 2020-10-08T11:36:41.4518088Z Will run 11 of 11 specs
  2003. 2020-10-08T11:36:41.4518676Z
  2004. 2020-10-08T11:36:41.4520482Z STEP: Creating kubernetes clients
  2005. 2020-10-08T11:36:41.4579302Z STEP: Creating lighthouse clients
  2006. 2020-10-08T11:36:41.4586319Z [discovery] Test Service Discovery Across Clusters when there are no active pods for a service
  2007. 2020-10-08T11:36:41.4587069Z should not resolve the service
  2008. 2020-10-08T11:36:41.4587868Z /go/src/github.com/submariner-io/lighthouse/test/e2e/discovery/service_discovery.go:44
  2009. 2020-10-08T11:36:41.4588736Z STEP: Creating namespace objects with basename "discovery"
  2010. 2020-10-08T11:36:41.4773471Z STEP: Generated namespace "e2e-tests-discovery-2f2zj" in cluster "cluster1" to execute the tests in
  2011. 2020-10-08T11:36:41.4780157Z STEP: Creating namespace "e2e-tests-discovery-2f2zj" in cluster "cluster2"
  2012. 2020-10-08T11:36:41.4989430Z STEP: Creating an Nginx Deployment on on "cluster2"
  2013. 2020-10-08T11:36:46.5637756Z STEP: Creating a Nginx Service on "cluster2"
  2014. 2020-10-08T11:36:46.5733311Z STEP: Creating serviceExport nginx-demo.e2e-tests-discovery-2f2zj on "cluster2"
  2015. 2020-10-08T11:36:46.5990926Z STEP: Retrieving ServiceExport nginx-demo.e2e-tests-discovery-2f2zj on "cluster2"
  2016. 2020-10-08T11:36:51.6076384Z STEP: Creating a Netshoot Deployment on "cluster1"
  2017. 2020-10-08T11:36:56.6225033Z STEP: Retrieving service nginx-demo.e2e-tests-discovery-2f2zj on "cluster2"
  2018. 2020-10-08T11:36:56.6253975Z STEP: Retrieving ServiceImport for nginx-demo-e2e-tests-discovery-2f2zj- on "cluster1"
  2019. 2020-10-08T11:36:56.6281995Z STEP: Executing "dig +short nginx-demo.e2e-tests-discovery-2f2zj.svc.clusterset.local" to verify IP "100.92.226.50" for service "nginx-demo" "is" discoverable
  2020. 2020-10-08T11:36:56.6289137Z Oct 8 11:36:56.627: INFO: ExecWithOptions {Command:[dig +short nginx-demo.e2e-tests-discovery-2f2zj.svc.clusterset.local] Namespace:e2e-tests-discovery-2f2zj PodName:netshootmhgvq-6757486d56-fhc2m ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
  2021. 2020-10-08T11:36:56.6921910Z STEP: Validating that dig result is "100.92.226.50"
  2022. 2020-10-08T11:36:56.6922869Z STEP: Setting Nginx deployment replicas to 0 in cluster "cluster2"
  2023. 2020-10-08T11:36:56.6968242Z STEP: Executing "dig +short nginx-demo.e2e-tests-discovery-2f2zj.svc.clusterset.local" to verify IP "100.92.226.50" for service "nginx-demo" "is not" discoverable
  2024. 2020-10-08T11:36:56.6971871Z Oct 8 11:36:56.696: INFO: ExecWithOptions {Command:[dig +short nginx-demo.e2e-tests-discovery-2f2zj.svc.clusterset.local] Namespace:e2e-tests-discovery-2f2zj PodName:netshootmhgvq-6757486d56-fhc2m ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
  2025. 2020-10-08T11:36:56.8641141Z STEP: Validating that dig result is not ""
  2026. 2020-10-08T11:36:56.8641950Z STEP: Setting Nginx deployment replicas to 2 in cluster "cluster2"
  2027. 2020-10-08T11:36:56.8752737Z STEP: Executing "dig +short nginx-demo.e2e-tests-discovery-2f2zj.svc.clusterset.local" to verify IP "100.92.226.50" for service "nginx-demo" "is" discoverable
  2028. 2020-10-08T11:36:56.8756542Z Oct 8 11:36:56.874: INFO: ExecWithOptions {Command:[dig +short nginx-demo.e2e-tests-discovery-2f2zj.svc.clusterset.local] Namespace:e2e-tests-discovery-2f2zj PodName:netshootmhgvq-6757486d56-fhc2m ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
  2029. 2020-10-08T11:36:56.9921980Z STEP: Validating that dig result is ""
  2030. 2020-10-08T11:37:01.9928266Z Oct 8 11:37:01.991: INFO: ExecWithOptions {Command:[dig +short nginx-demo.e2e-tests-discovery-2f2zj.svc.clusterset.local] Namespace:e2e-tests-discovery-2f2zj PodName:netshootmhgvq-6757486d56-fhc2m ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
  2031. 2020-10-08T11:37:02.0655415Z STEP: Validating that dig result is "100.92.226.50"
  2032. 2020-10-08T11:37:02.0657364Z STEP: Deleting namespace "e2e-tests-discovery-2f2zj" on cluster "cluster1"
  2033. 2020-10-08T11:37:02.0726483Z STEP: Deleting namespace "e2e-tests-discovery-2f2zj" on cluster "cluster2"
  2034. 2020-10-08T11:37:02.0783484Z •
  2035. 2020-10-08T11:37:02.0783908Z ------------------------------
  2036. 2020-10-08T11:37:02.0784496Z STEP: Creating namespace objects with basename "discovery"
  2037. 2020-10-08T11:37:02.0785488Z STEP: Generated namespace "e2e-tests-discovery-2f2zj" in cluster "cluster1" to execute the tests in
  2038. 2020-10-08T11:37:02.0786562Z STEP: Creating namespace "e2e-tests-discovery-2f2zj" in cluster "cluster2"
  2039. 2020-10-08T11:37:02.0787804Z STEP: Creating an Nginx Deployment on on "cluster2"
  2040. 2020-10-08T11:37:02.0788369Z STEP: Creating a Nginx Service on "cluster2"
  2041. 2020-10-08T11:37:02.0789260Z STEP: Creating serviceExport nginx-demo.e2e-tests-discovery-2f2zj on "cluster2"
  2042. 2020-10-08T11:37:02.0790448Z STEP: Retrieving ServiceExport nginx-demo.e2e-tests-discovery-2f2zj on "cluster2"
  2043. 2020-10-08T11:37:02.0791331Z STEP: Creating a Netshoot Deployment on "cluster1"
  2044. 2020-10-08T11:37:02.0792189Z STEP: Retrieving service nginx-demo.e2e-tests-discovery-2f2zj on "cluster2"
  2045. 2020-10-08T11:37:02.0793369Z STEP: Retrieving ServiceImport for nginx-demo-e2e-tests-discovery-2f2zj- on "cluster1"
  2046. 2020-10-08T11:37:02.0795311Z STEP: Executing "dig +short nginx-demo.e2e-tests-discovery-2f2zj.svc.clusterset.local" to verify IP "100.92.226.50" for service "nginx-demo" "is" discoverable
  2047. 2020-10-08T11:37:02.0798912Z Oct 8 11:36:56.627: INFO: ExecWithOptions {Command:[dig +short nginx-demo.e2e-tests-discovery-2f2zj.svc.clusterset.local] Namespace:e2e-tests-discovery-2f2zj PodName:netshootmhgvq-6757486d56-fhc2m ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
  2048. 2020-10-08T11:37:02.0801171Z STEP: Validating that dig result is "100.92.226.50"
  2049. 2020-10-08T11:37:02.0801860Z STEP: Setting Nginx deployment replicas to 0 in cluster "cluster2"
  2050. 2020-10-08T11:37:02.0803708Z STEP: Executing "dig +short nginx-demo.e2e-tests-discovery-2f2zj.svc.clusterset.local" to verify IP "100.92.226.50" for service "nginx-demo" "is not" discoverable
  2051. 2020-10-08T11:37:02.0807331Z Oct 8 11:36:56.696: INFO: ExecWithOptions {Command:[dig +short nginx-demo.e2e-tests-discovery-2f2zj.svc.clusterset.local] Namespace:e2e-tests-discovery-2f2zj PodName:netshootmhgvq-6757486d56-fhc2m ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
  2052. 2020-10-08T11:37:02.0809753Z STEP: Validating that dig result is not ""
  2053. 2020-10-08T11:37:02.0810499Z STEP: Setting Nginx deployment replicas to 2 in cluster "cluster2"
  2054. 2020-10-08T11:37:02.0812275Z STEP: Executing "dig +short nginx-demo.e2e-tests-discovery-2f2zj.svc.clusterset.local" to verify IP "100.92.226.50" for service "nginx-demo" "is" discoverable
  2055. 2020-10-08T11:37:02.0816439Z Oct 8 11:36:56.874: INFO: ExecWithOptions {Command:[dig +short nginx-demo.e2e-tests-discovery-2f2zj.svc.clusterset.local] Namespace:e2e-tests-discovery-2f2zj PodName:netshootmhgvq-6757486d56-fhc2m ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
  2056. 2020-10-08T11:37:02.0818833Z STEP: Validating that dig result is ""
  2057. 2020-10-08T11:37:02.0821391Z Oct 8 11:37:01.991: INFO: ExecWithOptions {Command:[dig +short nginx-demo.e2e-tests-discovery-2f2zj.svc.clusterset.local] Namespace:e2e-tests-discovery-2f2zj PodName:netshootmhgvq-6757486d56-fhc2m ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
  2058. 2020-10-08T11:37:02.0823617Z STEP: Validating that dig result is "100.92.226.50"
  2059. 2020-10-08T11:37:02.0824427Z STEP: Deleting namespace "e2e-tests-discovery-2f2zj" on cluster "cluster1"
  2060. 2020-10-08T11:37:02.0825468Z STEP: Deleting namespace "e2e-tests-discovery-2f2zj" on cluster "cluster2"
  2061. 2020-10-08T11:37:02.0825939Z
  2062. 2020-10-08T11:37:02.0826077Z
  2063. 2020-10-08T11:37:02.0827829Z [discovery] Test Headless Service Discovery Across Clusters when a pod tries to resolve a headless service in a remote cluster
  2064. 2020-10-08T11:37:02.0829764Z should resolve the backing pod IPs from the remote cluster
  2065. 2020-10-08T11:37:02.0830707Z /go/src/github.com/submariner-io/lighthouse/test/e2e/discovery/headless_services.go:17
  2066. 2020-10-08T11:37:02.0832665Z STEP: Creating namespace objects with basename "discovery"
  2067. 2020-10-08T11:37:02.0855586Z STEP: Generated namespace "e2e-tests-discovery-c8br4" in cluster "cluster1" to execute the tests in
  2068. 2020-10-08T11:37:02.0857864Z STEP: Creating namespace "e2e-tests-discovery-c8br4" in cluster "cluster2"
  2069. 2020-10-08T11:37:02.0929016Z STEP: Creating an Nginx Deployment on on "cluster2"
  2070. 2020-10-08T11:37:07.1373924Z STEP: Creating a Nginx Headless Service on "cluster2"
  2071. 2020-10-08T11:37:07.1660551Z STEP: Creating serviceExport nginx-headless.e2e-tests-discovery-c8br4 on "cluster2"
  2072. 2020-10-08T11:37:07.1800572Z STEP: Retrieving ServiceExport nginx-headless.e2e-tests-discovery-c8br4 on "cluster2"
  2073. 2020-10-08T11:37:12.1925887Z STEP: Creating a Netshoot Deployment on "cluster1"
  2074. 2020-10-08T11:37:17.2089051Z STEP: Retrieving Endpoints for nginx-headless on "cluster2"
  2075. 2020-10-08T11:37:17.2121955Z STEP: Executing "dig +short nginx-headless.e2e-tests-discovery-c8br4.svc.clusterset.local" to verify IPs [10.242.192.4] for service "nginx-headless" "are" discoverable
  2076. 2020-10-08T11:37:17.2126190Z Oct 8 11:37:17.211: INFO: ExecWithOptions {Command:[dig +short nginx-headless.e2e-tests-discovery-c8br4.svc.clusterset.local] Namespace:e2e-tests-discovery-c8br4 PodName:netshoothjnr4-6757486d56-hkw7q ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
  2077. 2020-10-08T11:37:17.2946858Z STEP: Validating that dig result are "10.242.192.4"
  2078. 2020-10-08T11:37:17.2948618Z STEP: Deleting serviceExport nginx-headless.e2e-tests-discovery-c8br4 on "cluster2"
  2079. 2020-10-08T11:37:22.9552749Z STEP: Executing "dig +short nginx-headless.e2e-tests-discovery-c8br4.svc.clusterset.local" to verify IPs [10.242.192.4] for service "nginx-headless" "are not" discoverable
  2080. 2020-10-08T11:37:22.9557256Z Oct 8 11:37:22.304: INFO: ExecWithOptions {Command:[dig +short nginx-headless.e2e-tests-discovery-c8br4.svc.clusterset.local] Namespace:e2e-tests-discovery-c8br4 PodName:netshoothjnr4-6757486d56-hkw7q ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
  2081. 2020-10-08T11:37:22.9560063Z STEP: Validating that dig result are not ""
  2082. 2020-10-08T11:37:22.9561128Z STEP: Deleting namespace "e2e-tests-discovery-c8br4" on cluster "cluster1"
  2083. 2020-10-08T11:37:22.9562359Z STEP: Deleting namespace "e2e-tests-discovery-c8br4" on cluster "cluster2"
  2084. 2020-10-08T11:37:22.9563511Z •
  2085. 2020-10-08T11:37:22.9564560Z ------------------------------
  2086. 2020-10-08T11:37:22.9565307Z STEP: Creating namespace objects with basename "discovery"
  2087. 2020-10-08T11:37:22.9566280Z STEP: Generated namespace "e2e-tests-discovery-c8br4" in cluster "cluster1" to execute the tests in
  2088. 2020-10-08T11:37:22.9567350Z STEP: Creating namespace "e2e-tests-discovery-c8br4" in cluster "cluster2"
  2089. 2020-10-08T11:37:22.9568075Z STEP: Creating an Nginx Deployment on on "cluster2"
  2090. 2020-10-08T11:37:22.9568661Z STEP: Creating a Nginx Headless Service on "cluster2"
  2091. 2020-10-08T11:37:22.9569627Z STEP: Creating serviceExport nginx-headless.e2e-tests-discovery-c8br4 on "cluster2"
  2092. 2020-10-08T11:37:22.9571046Z STEP: Retrieving ServiceExport nginx-headless.e2e-tests-discovery-c8br4 on "cluster2"
  2093. 2020-10-08T11:37:22.9572014Z STEP: Creating a Netshoot Deployment on "cluster1"
  2094. 2020-10-08T11:37:22.9573115Z STEP: Retrieving Endpoints for nginx-headless on "cluster2"
  2095. 2020-10-08T11:37:22.9574943Z STEP: Executing "dig +short nginx-headless.e2e-tests-discovery-c8br4.svc.clusterset.local" to verify IPs [10.242.192.4] for service "nginx-headless" "are" discoverable
  2096. 2020-10-08T11:37:22.9578785Z Oct 8 11:37:17.211: INFO: ExecWithOptions {Command:[dig +short nginx-headless.e2e-tests-discovery-c8br4.svc.clusterset.local] Namespace:e2e-tests-discovery-c8br4 PodName:netshoothjnr4-6757486d56-hkw7q ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
  2097. 2020-10-08T11:37:22.9581358Z STEP: Validating that dig result are "10.242.192.4"
  2098. 2020-10-08T11:37:22.9582511Z STEP: Deleting serviceExport nginx-headless.e2e-tests-discovery-c8br4 on "cluster2"
  2099. 2020-10-08T11:37:22.9585287Z STEP: Executing "dig +short nginx-headless.e2e-tests-discovery-c8br4.svc.clusterset.local" to verify IPs [10.242.192.4] for service "nginx-headless" "are not" discoverable
  2100. 2020-10-08T11:37:22.9589276Z Oct 8 11:37:22.304: INFO: ExecWithOptions {Command:[dig +short nginx-headless.e2e-tests-discovery-c8br4.svc.clusterset.local] Namespace:e2e-tests-discovery-c8br4 PodName:netshoothjnr4-6757486d56-hkw7q ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
  2101. 2020-10-08T11:37:22.9591832Z STEP: Validating that dig result are not ""
  2102. 2020-10-08T11:37:22.9592709Z STEP: Deleting namespace "e2e-tests-discovery-c8br4" on cluster "cluster1"
  2103. 2020-10-08T11:37:22.9593914Z STEP: Deleting namespace "e2e-tests-discovery-c8br4" on cluster "cluster2"
  2104. 2020-10-08T11:37:22.9594414Z
  2105. 2020-10-08T11:37:22.9594581Z
  2106. 2020-10-08T11:37:22.9595379Z [discovery] Test Service Discovery Across Clusters when service export is created before the service
  2107. 2020-10-08T11:37:22.9596298Z should resolve the service
  2108. 2020-10-08T11:37:22.9597092Z /go/src/github.com/submariner-io/lighthouse/test/e2e/discovery/service_discovery.go:38
  2109. 2020-10-08T11:37:22.9597945Z STEP: Creating namespace objects with basename "discovery"
  2110. 2020-10-08T11:37:22.9598919Z STEP: Generated namespace "e2e-tests-discovery-z7hjl" in cluster "cluster1" to execute the tests in
  2111. 2020-10-08T11:37:22.9600098Z STEP: Creating namespace "e2e-tests-discovery-z7hjl" in cluster "cluster2"
  2112. 2020-10-08T11:37:22.9600961Z STEP: Creating an Nginx ServiceExport on "cluster2"
  2113. 2020-10-08T11:37:22.9601998Z STEP: Creating serviceExport nginx-demo.e2e-tests-discovery-z7hjl on "cluster2"
  2114. 2020-10-08T11:37:22.9602996Z STEP: Creating an Nginx Deployment on on "cluster2"
  2115. 2020-10-08T11:37:27.4804512Z STEP: Creating a Nginx Service on "cluster2"
  2116. 2020-10-08T11:37:27.5147050Z STEP: Creating a Netshoot Deployment on "cluster1"
  2117. 2020-10-08T11:37:32.6030452Z STEP: Retrieving service nginx-demo.e2e-tests-discovery-z7hjl on "cluster2"
  2118. 2020-10-08T11:37:32.6050348Z STEP: Retrieving ServiceImport for nginx-demo-e2e-tests-discovery-z7hjl- on "cluster1"
  2119. 2020-10-08T11:37:32.6073721Z STEP: Executing "dig +short nginx-demo.e2e-tests-discovery-z7hjl.svc.clusterset.local" to verify IP "100.92.230.255" for service "nginx-demo" "is" discoverable
  2120. 2020-10-08T11:37:32.6076752Z Oct 8 11:37:32.606: INFO: ExecWithOptions {Command:[dig +short nginx-demo.e2e-tests-discovery-z7hjl.svc.clusterset.local] Namespace:e2e-tests-discovery-z7hjl PodName:netshootzqfg9-6757486d56-x6sk8 ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
  2121. 2020-10-08T11:37:32.6802244Z STEP: Validating that dig result is "100.92.230.255"
  2122. 2020-10-08T11:37:32.6809474Z STEP: Deleting serviceExport nginx-demo.e2e-tests-discovery-z7hjl on "cluster2"
  2123. 2020-10-08T11:37:37.6900852Z STEP: Executing "dig +short nginx-demo.e2e-tests-discovery-z7hjl.svc.clusterset.local" to verify IP "100.92.230.255" for service "nginx-demo" "is not" discoverable
  2124. 2020-10-08T11:37:37.6907924Z Oct 8 11:37:37.689: INFO: ExecWithOptions {Command:[dig +short nginx-demo.e2e-tests-discovery-z7hjl.svc.clusterset.local] Namespace:e2e-tests-discovery-z7hjl PodName:netshootzqfg9-6757486d56-x6sk8 ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
  2125. 2020-10-08T11:37:43.7960358Z STEP: Validating that dig result is not ""
  2126. 2020-10-08T11:37:43.7961122Z STEP: Deleting namespace "e2e-tests-discovery-z7hjl" on cluster "cluster1"
  2127. 2020-10-08T11:37:43.8025116Z STEP: Deleting namespace "e2e-tests-discovery-z7hjl" on cluster "cluster2"
  2128. 2020-10-08T11:37:43.8057907Z •
  2129. 2020-10-08T11:37:43.8058324Z ------------------------------
  2130. 2020-10-08T11:37:43.8058901Z STEP: Creating namespace objects with basename "discovery"
  2131. 2020-10-08T11:37:43.8060186Z STEP: Generated namespace "e2e-tests-discovery-z7hjl" in cluster "cluster1" to execute the tests in
  2132. 2020-10-08T11:37:43.8061306Z STEP: Creating namespace "e2e-tests-discovery-z7hjl" in cluster "cluster2"
  2133. 2020-10-08T11:37:43.8062198Z STEP: Creating an Nginx ServiceExport on "cluster2"
  2134. 2020-10-08T11:37:43.8063250Z STEP: Creating serviceExport nginx-demo.e2e-tests-discovery-z7hjl on "cluster2"
  2135. 2020-10-08T11:37:43.8064252Z STEP: Creating an Nginx Deployment on on "cluster2"
  2136. 2020-10-08T11:37:43.8064885Z STEP: Creating a Nginx Service on "cluster2"
  2137. 2020-10-08T11:37:43.8065531Z STEP: Creating a Netshoot Deployment on "cluster1"
  2138. 2020-10-08T11:37:43.8066518Z STEP: Retrieving service nginx-demo.e2e-tests-discovery-z7hjl on "cluster2"
  2139. 2020-10-08T11:37:43.8067880Z STEP: Retrieving ServiceImport for nginx-demo-e2e-tests-discovery-z7hjl- on "cluster1"
  2140. 2020-10-08T11:37:43.8069856Z STEP: Executing "dig +short nginx-demo.e2e-tests-discovery-z7hjl.svc.clusterset.local" to verify IP "100.92.230.255" for service "nginx-demo" "is" discoverable
  2141. 2020-10-08T11:37:43.8073066Z Oct 8 11:37:32.606: INFO: ExecWithOptions {Command:[dig +short nginx-demo.e2e-tests-discovery-z7hjl.svc.clusterset.local] Namespace:e2e-tests-discovery-z7hjl PodName:netshootzqfg9-6757486d56-x6sk8 ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
  2142. 2020-10-08T11:37:43.8075038Z STEP: Validating that dig result is "100.92.230.255"
  2143. 2020-10-08T11:37:43.8075893Z STEP: Deleting serviceExport nginx-demo.e2e-tests-discovery-z7hjl on "cluster2"
  2144. 2020-10-08T11:37:43.8077614Z STEP: Executing "dig +short nginx-demo.e2e-tests-discovery-z7hjl.svc.clusterset.local" to verify IP "100.92.230.255" for service "nginx-demo" "is not" discoverable
  2145. 2020-10-08T11:37:43.8081102Z Oct 8 11:37:37.689: INFO: ExecWithOptions {Command:[dig +short nginx-demo.e2e-tests-discovery-z7hjl.svc.clusterset.local] Namespace:e2e-tests-discovery-z7hjl PodName:netshootzqfg9-6757486d56-x6sk8 ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
  2146. 2020-10-08T11:37:43.8083329Z STEP: Validating that dig result is not ""
  2147. 2020-10-08T11:37:43.8084262Z STEP: Deleting namespace "e2e-tests-discovery-z7hjl" on cluster "cluster1"
  2148. 2020-10-08T11:37:43.8085437Z STEP: Deleting namespace "e2e-tests-discovery-z7hjl" on cluster "cluster2"
  2149. 2020-10-08T11:37:43.8085896Z
  2150. 2020-10-08T11:37:43.8086051Z
  2151. 2020-10-08T11:37:43.8086897Z [discovery] Test Stateful Sets Discovery Across Clusters when a pod tries to resolve a podname from stateful set in a remote cluster
  2152. 2020-10-08T11:37:43.8087893Z should resolve the pod IP from the remote cluster
  2153. 2020-10-08T11:37:43.8088729Z /go/src/github.com/submariner-io/lighthouse/test/e2e/discovery/statefulsets.go:20
  2154. 2020-10-08T11:37:43.8089588Z STEP: Creating namespace objects with basename "discovery"
  2155. 2020-10-08T11:37:43.8166472Z STEP: Generated namespace "e2e-tests-discovery-4rxgg" in cluster "cluster1" to execute the tests in
  2156. 2020-10-08T11:37:43.8167639Z STEP: Creating namespace "e2e-tests-discovery-4rxgg" in cluster "cluster2"
  2157. 2020-10-08T11:37:43.8215540Z STEP: Creating an Nginx Stateful Set on "cluster2"
  2158. 2020-10-08T11:37:48.8782881Z STEP: Creating a Nginx Headless Service on "cluster2"
  2159. 2020-10-08T11:37:48.8981258Z STEP: Creating serviceExport nginx-ss.e2e-tests-discovery-4rxgg on "cluster2"
  2160. 2020-10-08T11:37:48.9162509Z STEP: Retrieving ServiceExport nginx-ss.e2e-tests-discovery-4rxgg on "cluster2"
  2161. 2020-10-08T11:37:53.9293802Z STEP: Creating a Netshoot Deployment on "cluster1"
  2162. 2020-10-08T11:37:58.9585613Z STEP: Retrieving EndpointSlices for nginx-ss on "cluster2"
  2163. 2020-10-08T11:37:58.9623757Z STEP: Executing "dig +short web-0.cluster2.nginx-ss.e2e-tests-discovery-4rxgg.svc.clusterset.local" to verify IPs [10.242.192.4] for pod "web-0.cluster2.nginx-ss" "are" discoverable
  2164. 2020-10-08T11:37:58.9628669Z Oct 8 11:37:58.961: INFO: ExecWithOptions {Command:[dig +short web-0.cluster2.nginx-ss.e2e-tests-discovery-4rxgg.svc.clusterset.local] Namespace:e2e-tests-discovery-4rxgg PodName:netshootnqx7n-6757486d56-fgbpt ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
  2165. 2020-10-08T11:37:59.0459119Z STEP: Validating that dig result are "10.242.192.4"
  2166. 2020-10-08T11:37:59.0460492Z STEP: Deleting serviceExport nginx-ss.e2e-tests-discovery-4rxgg on "cluster2"
  2167. 2020-10-08T11:38:04.0562488Z STEP: Retrieving EndpointSlices for nginx-ss on "cluster2"
  2168. 2020-10-08T11:38:04.0595320Z STEP: Executing "dig +short web-0.cluster2.nginx-ss.e2e-tests-discovery-4rxgg.svc.clusterset.local" to verify IPs [10.242.192.4] for pod "web-0.cluster2.nginx-ss" "are not" discoverable
  2169. 2020-10-08T11:38:04.0601907Z Oct 8 11:38:04.058: INFO: ExecWithOptions {Command:[dig +short web-0.cluster2.nginx-ss.e2e-tests-discovery-4rxgg.svc.clusterset.local] Namespace:e2e-tests-discovery-4rxgg PodName:netshootnqx7n-6757486d56-fgbpt ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
  2170. 2020-10-08T11:38:04.1364614Z STEP: Validating that dig result are not ""
  2171. 2020-10-08T11:38:04.1365988Z STEP: Deleting namespace "e2e-tests-discovery-4rxgg" on cluster "cluster1"
  2172. 2020-10-08T11:38:04.1405333Z STEP: Deleting namespace "e2e-tests-discovery-4rxgg" on cluster "cluster2"
  2173. 2020-10-08T11:38:04.1444384Z •
  2174. 2020-10-08T11:38:04.1444880Z ------------------------------
  2175. 2020-10-08T11:38:04.1447289Z STEP: Creating namespace objects with basename "discovery"
  2176. 2020-10-08T11:38:04.1448667Z STEP: Generated namespace "e2e-tests-discovery-4rxgg" in cluster "cluster1" to execute the tests in
  2177. 2020-10-08T11:38:04.1449926Z STEP: Creating namespace "e2e-tests-discovery-4rxgg" in cluster "cluster2"
  2178. 2020-10-08T11:38:04.1450821Z STEP: Creating an Nginx Stateful Set on "cluster2"
  2179. 2020-10-08T11:38:04.1451632Z STEP: Creating a Nginx Headless Service on "cluster2"
  2180. 2020-10-08T11:38:04.1452952Z STEP: Creating serviceExport nginx-ss.e2e-tests-discovery-4rxgg on "cluster2"
  2181. 2020-10-08T11:38:04.1454739Z STEP: Retrieving ServiceExport nginx-ss.e2e-tests-discovery-4rxgg on "cluster2"
  2182. 2020-10-08T11:38:04.1455799Z STEP: Creating a Netshoot Deployment on "cluster1"
  2183. 2020-10-08T11:38:04.1456589Z STEP: Retrieving EndpointSlices for nginx-ss on "cluster2"
  2184. 2020-10-08T11:38:04.1458837Z STEP: Executing "dig +short web-0.cluster2.nginx-ss.e2e-tests-discovery-4rxgg.svc.clusterset.local" to verify IPs [10.242.192.4] for pod "web-0.cluster2.nginx-ss" "are" discoverable
  2185. 2020-10-08T11:38:04.1463215Z Oct 8 11:37:58.961: INFO: ExecWithOptions {Command:[dig +short web-0.cluster2.nginx-ss.e2e-tests-discovery-4rxgg.svc.clusterset.local] Namespace:e2e-tests-discovery-4rxgg PodName:netshootnqx7n-6757486d56-fgbpt ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
  2186. 2020-10-08T11:38:04.1465963Z STEP: Validating that dig result are "10.242.192.4"
  2187. 2020-10-08T11:38:04.1466983Z STEP: Deleting serviceExport nginx-ss.e2e-tests-discovery-4rxgg on "cluster2"
  2188. 2020-10-08T11:38:04.1468105Z STEP: Retrieving EndpointSlices for nginx-ss on "cluster2"
  2189. 2020-10-08T11:38:04.1470358Z STEP: Executing "dig +short web-0.cluster2.nginx-ss.e2e-tests-discovery-4rxgg.svc.clusterset.local" to verify IPs [10.242.192.4] for pod "web-0.cluster2.nginx-ss" "are not" discoverable
  2190. 2020-10-08T11:38:04.1474730Z Oct 8 11:38:04.058: INFO: ExecWithOptions {Command:[dig +short web-0.cluster2.nginx-ss.e2e-tests-discovery-4rxgg.svc.clusterset.local] Namespace:e2e-tests-discovery-4rxgg PodName:netshootnqx7n-6757486d56-fgbpt ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
  2191. 2020-10-08T11:38:04.1477570Z STEP: Validating that dig result are not ""
  2192. 2020-10-08T11:38:04.1478574Z STEP: Deleting namespace "e2e-tests-discovery-4rxgg" on cluster "cluster1"
  2193. 2020-10-08T11:38:04.1479691Z STEP: Deleting namespace "e2e-tests-discovery-4rxgg" on cluster "cluster2"
  2194. 2020-10-08T11:38:04.1480192Z
  2195. 2020-10-08T11:38:04.1490488Z
  2196. 2020-10-08T11:38:04.1491831Z [discovery] Test Stateful Sets Discovery Across Clusters when the number of active pods backing a stateful set changes
  2197. 2020-10-08T11:38:04.1494153Z should only resolve the IPs from the active pods
  2198. 2020-10-08T11:38:04.1495203Z /go/src/github.com/submariner-io/lighthouse/test/e2e/discovery/statefulsets.go:40
  2199. 2020-10-08T11:38:04.1496296Z STEP: Creating namespace objects with basename "discovery"
  2200. 2020-10-08T11:38:04.1570649Z STEP: Generated namespace "e2e-tests-discovery-srnrh" in cluster "cluster1" to execute the tests in
  2201. 2020-10-08T11:38:04.1573547Z STEP: Creating namespace "e2e-tests-discovery-srnrh" in cluster "cluster2"
  2202. 2020-10-08T11:38:04.1769447Z STEP: Creating an Nginx Stateful Set on on "cluster2"
  2203. 2020-10-08T11:38:09.2273925Z STEP: Setting Nginx statefulset replicas to 3
  2204. 2020-10-08T11:38:09.2317122Z STEP: Creating a Nginx Headless Service on "cluster2"
  2205. 2020-10-08T11:38:09.2470366Z STEP: Creating serviceExport nginx-ss.e2e-tests-discovery-srnrh on "cluster2"
  2206. 2020-10-08T11:38:09.2635174Z STEP: Retrieving ServiceExport nginx-ss.e2e-tests-discovery-srnrh on "cluster2"
  2207. 2020-10-08T11:38:14.2845194Z STEP: Creating a Netshoot Deployment on "cluster1"
  2208. 2020-10-08T11:38:19.2907723Z STEP: Retrieving EndpointSlices for nginx-ss on "cluster2"
  2209. 2020-10-08T11:38:19.2935321Z STEP: Executing "dig +short web-1.cluster2.nginx-ss.e2e-tests-discovery-srnrh.svc.clusterset.local" to verify IPs [10.242.128.5] for pod "web-1.cluster2.nginx-ss" "are" discoverable
  2210. 2020-10-08T11:38:19.2940899Z Oct 8 11:38:19.292: INFO: ExecWithOptions {Command:[dig +short web-1.cluster2.nginx-ss.e2e-tests-discovery-srnrh.svc.clusterset.local] Namespace:e2e-tests-discovery-srnrh PodName:netshootnbvjg-6757486d56-w5phz ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
  2211. 2020-10-08T11:38:19.3700348Z STEP: Validating that dig result are "10.242.128.5"
  2212. 2020-10-08T11:38:19.3702497Z STEP: Executing "dig +short web-0.cluster2.nginx-ss.e2e-tests-discovery-srnrh.svc.clusterset.local" to verify IPs [10.242.192.3] for pod "web-0.cluster2.nginx-ss" "are" discoverable
  2213. 2020-10-08T11:38:19.3706068Z Oct 8 11:38:19.369: INFO: ExecWithOptions {Command:[dig +short web-0.cluster2.nginx-ss.e2e-tests-discovery-srnrh.svc.clusterset.local] Namespace:e2e-tests-discovery-srnrh PodName:netshootnbvjg-6757486d56-w5phz ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
  2214. 2020-10-08T11:38:19.4508270Z STEP: Validating that dig result are "10.242.192.3"
  2215. 2020-10-08T11:38:19.4510444Z STEP: Executing "dig +short web-2.cluster2.nginx-ss.e2e-tests-discovery-srnrh.svc.clusterset.local" to verify IPs [10.242.192.4] for pod "web-2.cluster2.nginx-ss" "are" discoverable
  2216. 2020-10-08T11:38:19.4514512Z Oct 8 11:38:19.450: INFO: ExecWithOptions {Command:[dig +short web-2.cluster2.nginx-ss.e2e-tests-discovery-srnrh.svc.clusterset.local] Namespace:e2e-tests-discovery-srnrh PodName:netshootnbvjg-6757486d56-w5phz ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
  2217. 2020-10-08T11:38:19.5442204Z STEP: Validating that dig result are "10.242.192.4"
  2218. 2020-10-08T11:38:19.5442995Z STEP: Setting Nginx statefulset replicas to 1
  2219. 2020-10-08T11:38:19.5490934Z STEP: Executing "dig +short web-1.cluster2.nginx-ss.e2e-tests-discovery-srnrh.svc.clusterset.local" to verify IPs [10.242.128.5] for pod "web-1.cluster2.nginx-ss" "are not" discoverable
  2220. 2020-10-08T11:38:19.5495785Z Oct 8 11:38:19.548: INFO: ExecWithOptions {Command:[dig +short web-1.cluster2.nginx-ss.e2e-tests-discovery-srnrh.svc.clusterset.local] Namespace:e2e-tests-discovery-srnrh PodName:netshootnbvjg-6757486d56-w5phz ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
  2221. 2020-10-08T11:38:19.6878080Z STEP: Validating that dig result are not "10.242.128.5"
  2222. 2020-10-08T11:38:24.6883844Z Oct 8 11:38:24.687: INFO: ExecWithOptions {Command:[dig +short web-1.cluster2.nginx-ss.e2e-tests-discovery-srnrh.svc.clusterset.local] Namespace:e2e-tests-discovery-srnrh PodName:netshootnbvjg-6757486d56-w5phz ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
  2223. 2020-10-08T11:38:24.7634337Z STEP: Validating that dig result are not "10.242.128.5"
  2224. 2020-10-08T11:38:29.6884722Z Oct 8 11:38:29.687: INFO: ExecWithOptions {Command:[dig +short web-1.cluster2.nginx-ss.e2e-tests-discovery-srnrh.svc.clusterset.local] Namespace:e2e-tests-discovery-srnrh PodName:netshootnbvjg-6757486d56-w5phz ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
  2225. 2020-10-08T11:38:29.7516359Z STEP: Validating that dig result are not "10.242.128.5"
  2226. 2020-10-08T11:38:34.6884386Z Oct 8 11:38:34.687: INFO: ExecWithOptions {Command:[dig +short web-1.cluster2.nginx-ss.e2e-tests-discovery-srnrh.svc.clusterset.local] Namespace:e2e-tests-discovery-srnrh PodName:netshootnbvjg-6757486d56-w5phz ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
  2227. 2020-10-08T11:38:34.7537253Z STEP: Validating that dig result are not ""
  2228. 2020-10-08T11:38:34.7542588Z STEP: Executing "dig +short web-0.cluster2.nginx-ss.e2e-tests-discovery-srnrh.svc.clusterset.local" to verify IPs [10.242.192.3] for pod "web-0.cluster2.nginx-ss" "are" discoverable
  2229. 2020-10-08T11:38:34.7550218Z Oct 8 11:38:34.754: INFO: ExecWithOptions {Command:[dig +short web-0.cluster2.nginx-ss.e2e-tests-discovery-srnrh.svc.clusterset.local] Namespace:e2e-tests-discovery-srnrh PodName:netshootnbvjg-6757486d56-w5phz ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
  2230. 2020-10-08T11:38:34.8144738Z STEP: Validating that dig result are "10.242.192.3"
  2231. 2020-10-08T11:38:34.8146786Z STEP: Executing "dig +short web-2.cluster2.nginx-ss.e2e-tests-discovery-srnrh.svc.clusterset.local" to verify IPs [10.242.192.4] for pod "web-2.cluster2.nginx-ss" "are not" discoverable
  2232. 2020-10-08T11:38:34.8151081Z Oct 8 11:38:34.813: INFO: ExecWithOptions {Command:[dig +short web-2.cluster2.nginx-ss.e2e-tests-discovery-srnrh.svc.clusterset.local] Namespace:e2e-tests-discovery-srnrh PodName:netshootnbvjg-6757486d56-w5phz ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
  2233. 2020-10-08T11:38:34.8805937Z STEP: Validating that dig result are not ""
  2234. 2020-10-08T11:38:34.8806986Z STEP: Deleting serviceExport nginx-ss.e2e-tests-discovery-srnrh on "cluster2"
  2235. 2020-10-08T11:38:39.8898935Z STEP: Deleting namespace "e2e-tests-discovery-srnrh" on cluster "cluster1"
  2236. 2020-10-08T11:38:39.8934667Z STEP: Deleting namespace "e2e-tests-discovery-srnrh" on cluster "cluster2"
  2237. 2020-10-08T11:38:39.8987628Z •
  2238. 2020-10-08T11:38:39.8991187Z ------------------------------
  2239. 2020-10-08T11:38:39.8991918Z STEP: Creating namespace objects with basename "discovery"
  2240. 2020-10-08T11:38:39.8992984Z STEP: Generated namespace "e2e-tests-discovery-srnrh" in cluster "cluster1" to execute the tests in
  2241. 2020-10-08T11:38:39.8995119Z STEP: Creating namespace "e2e-tests-discovery-srnrh" in cluster "cluster2"
  2242. 2020-10-08T11:38:39.8995958Z STEP: Creating an Nginx Stateful Set on on "cluster2"
  2243. 2020-10-08T11:38:39.8996603Z STEP: Setting Nginx statefulset replicas to 3
  2244. 2020-10-08T11:38:39.8997263Z STEP: Creating a Nginx Headless Service on "cluster2"
  2245. 2020-10-08T11:38:39.8998242Z STEP: Creating serviceExport nginx-ss.e2e-tests-discovery-srnrh on "cluster2"
  2246. 2020-10-08T11:38:39.8999529Z STEP: Retrieving ServiceExport nginx-ss.e2e-tests-discovery-srnrh on "cluster2"
  2247. 2020-10-08T11:38:39.9000696Z STEP: Creating a Netshoot Deployment on "cluster1"
  2248. 2020-10-08T11:38:39.9001459Z STEP: Retrieving EndpointSlices for nginx-ss on "cluster2"
  2249. 2020-10-08T11:38:39.9003558Z STEP: Executing "dig +short web-1.cluster2.nginx-ss.e2e-tests-discovery-srnrh.svc.clusterset.local" to verify IPs [10.242.128.5] for pod "web-1.cluster2.nginx-ss" "are" discoverable
  2250. 2020-10-08T11:38:39.9007856Z Oct 8 11:38:19.292: INFO: ExecWithOptions {Command:[dig +short web-1.cluster2.nginx-ss.e2e-tests-discovery-srnrh.svc.clusterset.local] Namespace:e2e-tests-discovery-srnrh PodName:netshootnbvjg-6757486d56-w5phz ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
  2251. 2020-10-08T11:38:39.9010418Z STEP: Validating that dig result are "10.242.128.5"
  2252. 2020-10-08T11:38:39.9012999Z STEP: Executing "dig +short web-0.cluster2.nginx-ss.e2e-tests-discovery-srnrh.svc.clusterset.local" to verify IPs [10.242.192.3] for pod "web-0.cluster2.nginx-ss" "are" discoverable
  2253. 2020-10-08T11:38:39.9017099Z Oct 8 11:38:19.369: INFO: ExecWithOptions {Command:[dig +short web-0.cluster2.nginx-ss.e2e-tests-discovery-srnrh.svc.clusterset.local] Namespace:e2e-tests-discovery-srnrh PodName:netshootnbvjg-6757486d56-w5phz ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
  2254. 2020-10-08T11:38:39.9019659Z STEP: Validating that dig result are "10.242.192.3"
  2255. 2020-10-08T11:38:39.9021751Z STEP: Executing "dig +short web-2.cluster2.nginx-ss.e2e-tests-discovery-srnrh.svc.clusterset.local" to verify IPs [10.242.192.4] for pod "web-2.cluster2.nginx-ss" "are" discoverable
  2256. 2020-10-08T11:38:39.9025778Z Oct 8 11:38:19.450: INFO: ExecWithOptions {Command:[dig +short web-2.cluster2.nginx-ss.e2e-tests-discovery-srnrh.svc.clusterset.local] Namespace:e2e-tests-discovery-srnrh PodName:netshootnbvjg-6757486d56-w5phz ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
  2257. 2020-10-08T11:38:39.9028328Z STEP: Validating that dig result are "10.242.192.4"
  2258. 2020-10-08T11:38:39.9028943Z STEP: Setting Nginx statefulset replicas to 1
  2259. 2020-10-08T11:38:39.9030970Z STEP: Executing "dig +short web-1.cluster2.nginx-ss.e2e-tests-discovery-srnrh.svc.clusterset.local" to verify IPs [10.242.128.5] for pod "web-1.cluster2.nginx-ss" "are not" discoverable
  2260. 2020-10-08T11:38:39.9035212Z Oct 8 11:38:19.548: INFO: ExecWithOptions {Command:[dig +short web-1.cluster2.nginx-ss.e2e-tests-discovery-srnrh.svc.clusterset.local] Namespace:e2e-tests-discovery-srnrh PodName:netshootnbvjg-6757486d56-w5phz ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
  2261. 2020-10-08T11:38:39.9037780Z STEP: Validating that dig result are not "10.242.128.5"
  2262. 2020-10-08T11:38:39.9040708Z Oct 8 11:38:24.687: INFO: ExecWithOptions {Command:[dig +short web-1.cluster2.nginx-ss.e2e-tests-discovery-srnrh.svc.clusterset.local] Namespace:e2e-tests-discovery-srnrh PodName:netshootnbvjg-6757486d56-w5phz ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
  2263. 2020-10-08T11:38:39.9042951Z STEP: Validating that dig result are not "10.242.128.5"
  2264. 2020-10-08T11:38:39.9045376Z Oct 8 11:38:29.687: INFO: ExecWithOptions {Command:[dig +short web-1.cluster2.nginx-ss.e2e-tests-discovery-srnrh.svc.clusterset.local] Namespace:e2e-tests-discovery-srnrh PodName:netshootnbvjg-6757486d56-w5phz ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
  2265. 2020-10-08T11:38:39.9047607Z STEP: Validating that dig result are not "10.242.128.5"
  2266. 2020-10-08T11:38:39.9050027Z Oct 8 11:38:34.687: INFO: ExecWithOptions {Command:[dig +short web-1.cluster2.nginx-ss.e2e-tests-discovery-srnrh.svc.clusterset.local] Namespace:e2e-tests-discovery-srnrh PodName:netshootnbvjg-6757486d56-w5phz ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
  2267. 2020-10-08T11:38:39.9052231Z STEP: Validating that dig result are not ""
  2268. 2020-10-08T11:38:39.9054107Z STEP: Executing "dig +short web-0.cluster2.nginx-ss.e2e-tests-discovery-srnrh.svc.clusterset.local" to verify IPs [10.242.192.3] for pod "web-0.cluster2.nginx-ss" "are" discoverable
  2269. 2020-10-08T11:38:39.9057626Z Oct 8 11:38:34.754: INFO: ExecWithOptions {Command:[dig +short web-0.cluster2.nginx-ss.e2e-tests-discovery-srnrh.svc.clusterset.local] Namespace:e2e-tests-discovery-srnrh PodName:netshootnbvjg-6757486d56-w5phz ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
  2270. 2020-10-08T11:38:39.9059799Z STEP: Validating that dig result are "10.242.192.3"
  2271. 2020-10-08T11:38:39.9061531Z STEP: Executing "dig +short web-2.cluster2.nginx-ss.e2e-tests-discovery-srnrh.svc.clusterset.local" to verify IPs [10.242.192.4] for pod "web-2.cluster2.nginx-ss" "are not" discoverable
  2272. 2020-10-08T11:38:39.9065053Z Oct 8 11:38:34.813: INFO: ExecWithOptions {Command:[dig +short web-2.cluster2.nginx-ss.e2e-tests-discovery-srnrh.svc.clusterset.local] Namespace:e2e-tests-discovery-srnrh PodName:netshootnbvjg-6757486d56-w5phz ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
  2273. 2020-10-08T11:38:39.9067261Z STEP: Validating that dig result are not ""
  2274. 2020-10-08T11:38:39.9068269Z STEP: Deleting serviceExport nginx-ss.e2e-tests-discovery-srnrh on "cluster2"
  2275. 2020-10-08T11:38:39.9069283Z STEP: Deleting namespace "e2e-tests-discovery-srnrh" on cluster "cluster1"
  2276. 2020-10-08T11:38:39.9070832Z STEP: Deleting namespace "e2e-tests-discovery-srnrh" on cluster "cluster2"
  2277. 2020-10-08T11:38:39.9071464Z
  2278. 2020-10-08T11:38:39.9071600Z
  2279. 2020-10-08T11:38:39.9072397Z [discovery] Test Headless Service Discovery Across Clusters when the number of active pods backing a service changes
  2280. 2020-10-08T11:38:39.9073252Z should only resolve the IPs from the active pods
  2281. 2020-10-08T11:38:39.9113413Z /go/src/github.com/submariner-io/lighthouse/test/e2e/discovery/headless_services.go:37
  2282. 2020-10-08T11:38:39.9114675Z STEP: Creating namespace objects with basename "discovery"
  2283. 2020-10-08T11:38:39.9116338Z STEP: Generated namespace "e2e-tests-discovery-qgjk6" in cluster "cluster1" to execute the tests in
  2284. 2020-10-08T11:38:39.9117618Z STEP: Creating namespace "e2e-tests-discovery-qgjk6" in cluster "cluster2"
  2285. 2020-10-08T11:38:39.9294311Z STEP: Creating an Nginx Deployment on on "cluster2"
  2286. 2020-10-08T11:38:44.9807632Z STEP: Setting Nginx deployment replicas to 3 in cluster "cluster2"
  2287. 2020-10-08T11:38:44.9908441Z STEP: Creating a Nginx Headless Service on "cluster2"
  2288. 2020-10-08T11:38:45.0104246Z STEP: Creating serviceExport nginx-headless.e2e-tests-discovery-qgjk6 on "cluster2"
  2289. 2020-10-08T11:38:45.0866506Z STEP: Retrieving ServiceExport nginx-headless.e2e-tests-discovery-qgjk6 on "cluster2"
  2290. 2020-10-08T11:38:50.1211680Z STEP: Creating a Netshoot Deployment on "cluster1"
  2291. 2020-10-08T11:38:55.1315688Z STEP: Retrieving Endpoints for nginx-headless on "cluster2"
  2292. 2020-10-08T11:38:55.1339423Z STEP: Executing "dig +short nginx-headless.e2e-tests-discovery-qgjk6.svc.clusterset.local" to verify IPs [10.242.128.5 10.242.192.4 10.242.192.5] for service "nginx-headless" "are" discoverable
  2293. 2020-10-08T11:38:55.1347434Z Oct 8 11:38:55.133: INFO: ExecWithOptions {Command:[dig +short nginx-headless.e2e-tests-discovery-qgjk6.svc.clusterset.local] Namespace:e2e-tests-discovery-qgjk6 PodName:netshootntk45-6757486d56-g7rl7 ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
  2294. 2020-10-08T11:38:55.2244988Z STEP: Validating that dig result are "10.242.128.5\n10.242.192.4\n10.242.192.5"
  2295. 2020-10-08T11:38:55.2245870Z STEP: Setting Nginx deployment replicas to 0 in cluster "cluster2"
  2296. 2020-10-08T11:38:55.2292319Z STEP: Retrieving Endpoints for nginx-headless on "cluster2"
  2297. 2020-10-08T11:39:00.2369729Z STEP: Executing "dig +short nginx-headless.e2e-tests-discovery-qgjk6.svc.clusterset.local" to verify IPs [] for service "nginx-headless" "are not" discoverable
  2298. 2020-10-08T11:39:00.2374265Z Oct 8 11:39:00.236: INFO: ExecWithOptions {Command:[dig +short nginx-headless.e2e-tests-discovery-qgjk6.svc.clusterset.local] Namespace:e2e-tests-discovery-qgjk6 PodName:netshootntk45-6757486d56-g7rl7 ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
  2299. 2020-10-08T11:39:00.3064996Z STEP: Validating that dig result are not ""
  2300. 2020-10-08T11:39:00.3065808Z STEP: Setting Nginx deployment replicas to 2 in cluster "cluster2"
  2301. 2020-10-08T11:39:00.3109318Z STEP: Retrieving Endpoints for nginx-headless on "cluster2"
  2302. 2020-10-08T11:39:05.3164035Z STEP: Executing "dig +short nginx-headless.e2e-tests-discovery-qgjk6.svc.clusterset.local" to verify IPs [10.242.128.5 10.242.192.3] for service "nginx-headless" "are" discoverable
  2303. 2020-10-08T11:39:05.3169073Z Oct 8 11:39:05.315: INFO: ExecWithOptions {Command:[dig +short nginx-headless.e2e-tests-discovery-qgjk6.svc.clusterset.local] Namespace:e2e-tests-discovery-qgjk6 PodName:netshootntk45-6757486d56-g7rl7 ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
  2304. 2020-10-08T11:39:05.3966799Z STEP: Validating that dig result are "10.242.128.5\n10.242.192.3"
  2305. 2020-10-08T11:39:05.3968118Z STEP: Deleting namespace "e2e-tests-discovery-qgjk6" on cluster "cluster1"
  2306. 2020-10-08T11:39:05.4020091Z STEP: Deleting namespace "e2e-tests-discovery-qgjk6" on cluster "cluster2"
  2307. 2020-10-08T11:39:05.4080378Z •
  2308. 2020-10-08T11:39:05.4087348Z ------------------------------
  2309. 2020-10-08T11:39:05.4088063Z STEP: Creating namespace objects with basename "discovery"
  2310. 2020-10-08T11:39:05.4089130Z STEP: Generated namespace "e2e-tests-discovery-qgjk6" in cluster "cluster1" to execute the tests in
  2311. 2020-10-08T11:39:05.4090291Z STEP: Creating namespace "e2e-tests-discovery-qgjk6" in cluster "cluster2"
  2312. 2020-10-08T11:39:05.4091250Z STEP: Creating an Nginx Deployment on on "cluster2"
  2313. 2020-10-08T11:39:05.4092033Z STEP: Setting Nginx deployment replicas to 3 in cluster "cluster2"
  2314. 2020-10-08T11:39:05.4094330Z STEP: Creating a Nginx Headless Service on "cluster2"
  2315. 2020-10-08T11:39:05.4095538Z STEP: Creating serviceExport nginx-headless.e2e-tests-discovery-qgjk6 on "cluster2"
  2316. 2020-10-08T11:39:05.4097214Z STEP: Retrieving ServiceExport nginx-headless.e2e-tests-discovery-qgjk6 on "cluster2"
  2317. 2020-10-08T11:39:05.4098311Z STEP: Creating a Netshoot Deployment on "cluster1"
  2318. 2020-10-08T11:39:05.4099046Z STEP: Retrieving Endpoints for nginx-headless on "cluster2"
  2319. 2020-10-08T11:39:05.4100952Z STEP: Executing "dig +short nginx-headless.e2e-tests-discovery-qgjk6.svc.clusterset.local" to verify IPs [10.242.128.5 10.242.192.4 10.242.192.5] for service "nginx-headless" "are" discoverable
  2320. 2020-10-08T11:39:05.4104919Z Oct 8 11:38:55.133: INFO: ExecWithOptions {Command:[dig +short nginx-headless.e2e-tests-discovery-qgjk6.svc.clusterset.local] Namespace:e2e-tests-discovery-qgjk6 PodName:netshootntk45-6757486d56-g7rl7 ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
  2321. 2020-10-08T11:39:05.4107576Z STEP: Validating that dig result are "10.242.128.5\n10.242.192.4\n10.242.192.5"
  2322. 2020-10-08T11:39:05.4108360Z STEP: Setting Nginx deployment replicas to 0 in cluster "cluster2"
  2323. 2020-10-08T11:39:05.4109207Z STEP: Retrieving Endpoints for nginx-headless on "cluster2"
  2324. 2020-10-08T11:39:05.4111213Z STEP: Executing "dig +short nginx-headless.e2e-tests-discovery-qgjk6.svc.clusterset.local" to verify IPs [] for service "nginx-headless" "are not" discoverable
  2325. 2020-10-08T11:39:05.4115307Z Oct 8 11:39:00.236: INFO: ExecWithOptions {Command:[dig +short nginx-headless.e2e-tests-discovery-qgjk6.svc.clusterset.local] Namespace:e2e-tests-discovery-qgjk6 PodName:netshootntk45-6757486d56-g7rl7 ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
  2326. 2020-10-08T11:39:05.4117875Z STEP: Validating that dig result are not ""
  2327. 2020-10-08T11:39:05.4118631Z STEP: Setting Nginx deployment replicas to 2 in cluster "cluster2"
  2328. 2020-10-08T11:39:05.4119475Z STEP: Retrieving Endpoints for nginx-headless on "cluster2"
  2329. 2020-10-08T11:39:05.4121501Z STEP: Executing "dig +short nginx-headless.e2e-tests-discovery-qgjk6.svc.clusterset.local" to verify IPs [10.242.128.5 10.242.192.3] for service "nginx-headless" "are" discoverable
  2330. 2020-10-08T11:39:05.4125455Z Oct 8 11:39:05.315: INFO: ExecWithOptions {Command:[dig +short nginx-headless.e2e-tests-discovery-qgjk6.svc.clusterset.local] Namespace:e2e-tests-discovery-qgjk6 PodName:netshootntk45-6757486d56-g7rl7 ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
  2331. 2020-10-08T11:39:05.4128042Z STEP: Validating that dig result are "10.242.128.5\n10.242.192.3"
  2332. 2020-10-08T11:39:05.4128944Z STEP: Deleting namespace "e2e-tests-discovery-qgjk6" on cluster "cluster1"
  2333. 2020-10-08T11:39:05.4130037Z STEP: Deleting namespace "e2e-tests-discovery-qgjk6" on cluster "cluster2"
  2334. 2020-10-08T11:39:05.4130547Z
  2335. 2020-10-08T11:39:05.4130714Z
  2336. 2020-10-08T11:39:05.4131788Z [discovery] Test Headless Service Discovery Across Clusters when a pod tries to resolve a headless service which is exported locally and in a remote cluster
  2337. 2020-10-08T11:39:05.4144760Z should resolve the backing pod IPs from both clusters
  2338. 2020-10-08T11:39:05.4146019Z /go/src/github.com/submariner-io/lighthouse/test/e2e/discovery/headless_services.go:27
  2339. 2020-10-08T11:39:05.4146964Z STEP: Creating namespace objects with basename "discovery"
  2340. 2020-10-08T11:39:05.4200127Z STEP: Generated namespace "e2e-tests-discovery-7vkcd" in cluster "cluster1" to execute the tests in
  2341. 2020-10-08T11:39:05.4201343Z STEP: Creating namespace "e2e-tests-discovery-7vkcd" in cluster "cluster2"
  2342. 2020-10-08T11:39:05.4325345Z STEP: Creating an Nginx Deployment on on "cluster2"
  2343. 2020-10-08T11:39:10.5011946Z STEP: Creating a Nginx Headless Service on "cluster2"
  2344. 2020-10-08T11:39:10.5710342Z STEP: Creating serviceExport nginx-headless.e2e-tests-discovery-7vkcd on "cluster2"
  2345. 2020-10-08T11:39:10.6945201Z STEP: Retrieving ServiceExport nginx-headless.e2e-tests-discovery-7vkcd on "cluster2"
  2346. 2020-10-08T11:39:15.7033399Z STEP: Creating an Nginx Deployment on on "cluster1"
  2347. 2020-10-08T11:39:20.7155180Z STEP: Creating a Nginx Headless Service on "cluster1"
  2348. 2020-10-08T11:39:20.7221707Z STEP: Creating serviceExport nginx-headless.e2e-tests-discovery-7vkcd on "cluster1"
  2349. 2020-10-08T11:39:20.7290196Z STEP: Retrieving ServiceExport nginx-headless.e2e-tests-discovery-7vkcd on "cluster1"
  2350. 2020-10-08T11:39:25.7338274Z STEP: Creating a Netshoot Deployment on "cluster1"
  2351. 2020-10-08T11:39:31.4913943Z STEP: Retrieving Endpoints for nginx-headless on "cluster2"
  2352. 2020-10-08T11:39:31.4915029Z STEP: Retrieving Endpoints for nginx-headless on "cluster1"
  2353. 2020-10-08T11:39:31.4916924Z STEP: Executing "dig +short nginx-headless.e2e-tests-discovery-7vkcd.svc.clusterset.local" to verify IPs [10.242.192.4 10.241.224.4] for service "nginx-headless" "are" discoverable
  2354. 2020-10-08T11:39:31.4920875Z Oct 8 11:39:30.745: INFO: ExecWithOptions {Command:[dig +short nginx-headless.e2e-tests-discovery-7vkcd.svc.clusterset.local] Namespace:e2e-tests-discovery-7vkcd PodName:netshootzjlcv-6757486d56-9f7w6 ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
  2355. 2020-10-08T11:39:31.4923307Z STEP: Validating that dig result are "10.241.224.4"
  2356. 2020-10-08T11:39:35.8211278Z Oct 8 11:39:35.820: INFO: ExecWithOptions {Command:[dig +short nginx-headless.e2e-tests-discovery-7vkcd.svc.clusterset.local] Namespace:e2e-tests-discovery-7vkcd PodName:netshootzjlcv-6757486d56-9f7w6 ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
  2357. 2020-10-08T11:39:35.8914825Z STEP: Validating that dig result are "10.241.224.4"
  2358. 2020-10-08T11:39:40.8214698Z Oct 8 11:39:40.820: INFO: ExecWithOptions {Command:[dig +short nginx-headless.e2e-tests-discovery-7vkcd.svc.clusterset.local] Namespace:e2e-tests-discovery-7vkcd PodName:netshootzjlcv-6757486d56-9f7w6 ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
  2359. 2020-10-08T11:39:40.8851608Z STEP: Validating that dig result are "10.241.224.4"
  2360. 2020-10-08T11:39:45.8211686Z Oct 8 11:39:45.820: INFO: ExecWithOptions {Command:[dig +short nginx-headless.e2e-tests-discovery-7vkcd.svc.clusterset.local] Namespace:e2e-tests-discovery-7vkcd PodName:netshootzjlcv-6757486d56-9f7w6 ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
  2361. 2020-10-08T11:39:45.8873733Z STEP: Validating that dig result are "10.241.224.4"
  2362. 2020-10-08T11:39:50.8211736Z Oct 8 11:39:50.820: INFO: ExecWithOptions {Command:[dig +short nginx-headless.e2e-tests-discovery-7vkcd.svc.clusterset.local] Namespace:e2e-tests-discovery-7vkcd PodName:netshootzjlcv-6757486d56-9f7w6 ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
  2363. 2020-10-08T11:39:50.8900122Z STEP: Validating that dig result are "10.241.224.4"
  2364. 2020-10-08T11:39:55.8211516Z Oct 8 11:39:55.820: INFO: ExecWithOptions {Command:[dig +short nginx-headless.e2e-tests-discovery-7vkcd.svc.clusterset.local] Namespace:e2e-tests-discovery-7vkcd PodName:netshootzjlcv-6757486d56-9f7w6 ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
  2365. 2020-10-08T11:39:55.8901999Z STEP: Validating that dig result are "10.241.224.4"
  2366. 2020-10-08T11:40:00.8211077Z Oct 8 11:40:00.820: INFO: ExecWithOptions {Command:[dig +short nginx-headless.e2e-tests-discovery-7vkcd.svc.clusterset.local] Namespace:e2e-tests-discovery-7vkcd PodName:netshootzjlcv-6757486d56-9f7w6 ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
  2367. 2020-10-08T11:40:00.8856318Z STEP: Validating that dig result are "10.241.224.4"
  2368. 2020-10-08T11:40:05.8211469Z Oct 8 11:40:05.820: INFO: ExecWithOptions {Command:[dig +short nginx-headless.e2e-tests-discovery-7vkcd.svc.clusterset.local] Namespace:e2e-tests-discovery-7vkcd PodName:netshootzjlcv-6757486d56-9f7w6 ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
  2369. 2020-10-08T11:40:05.8777526Z STEP: Validating that dig result are "10.241.224.4"
  2370. 2020-10-08T11:40:10.8320195Z Oct 8 11:40:10.820: INFO: ExecWithOptions {Command:[dig +short nginx-headless.e2e-tests-discovery-7vkcd.svc.clusterset.local] Namespace:e2e-tests-discovery-7vkcd PodName:netshootzjlcv-6757486d56-9f7w6 ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
  2371. 2020-10-08T11:40:10.9067026Z STEP: Validating that dig result are "10.241.224.4"
  2372. 2020-10-08T11:40:15.8210455Z Oct 8 11:40:15.820: INFO: ExecWithOptions {Command:[dig +short nginx-headless.e2e-tests-discovery-7vkcd.svc.clusterset.local] Namespace:e2e-tests-discovery-7vkcd PodName:netshootzjlcv-6757486d56-9f7w6 ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
  2373. 2020-10-08T11:40:15.8808274Z STEP: Validating that dig result are "10.241.224.4"
  2374. 2020-10-08T11:40:20.8215796Z Oct 8 11:40:20.820: INFO: ExecWithOptions {Command:[dig +short nginx-headless.e2e-tests-discovery-7vkcd.svc.clusterset.local] Namespace:e2e-tests-discovery-7vkcd PodName:netshootzjlcv-6757486d56-9f7w6 ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
  2375. 2020-10-08T11:40:20.9014485Z STEP: Validating that dig result are "10.241.224.4"
  2376. 2020-10-08T11:40:25.8213355Z Oct 8 11:40:25.820: INFO: ExecWithOptions {Command:[dig +short nginx-headless.e2e-tests-discovery-7vkcd.svc.clusterset.local] Namespace:e2e-tests-discovery-7vkcd PodName:netshootzjlcv-6757486d56-9f7w6 ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
  2377. 2020-10-08T11:40:25.8852189Z STEP: Validating that dig result are "10.241.224.4"
  2378. 2020-10-08T11:40:30.8214059Z Oct 8 11:40:30.820: INFO: ExecWithOptions {Command:[dig +short nginx-headless.e2e-tests-discovery-7vkcd.svc.clusterset.local] Namespace:e2e-tests-discovery-7vkcd PodName:netshootzjlcv-6757486d56-9f7w6 ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
  2379. 2020-10-08T11:40:30.8981703Z STEP: Validating that dig result are "10.241.224.4"
  2380. 2020-10-08T11:40:35.8211105Z Oct 8 11:40:35.820: INFO: ExecWithOptions {Command:[dig +short nginx-headless.e2e-tests-discovery-7vkcd.svc.clusterset.local] Namespace:e2e-tests-discovery-7vkcd PodName:netshootzjlcv-6757486d56-9f7w6 ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
  2381. 2020-10-08T11:40:35.8962788Z STEP: Validating that dig result are "10.241.224.4"
  2382. 2020-10-08T11:40:40.8211398Z Oct 8 11:40:40.820: INFO: ExecWithOptions {Command:[dig +short nginx-headless.e2e-tests-discovery-7vkcd.svc.clusterset.local] Namespace:e2e-tests-discovery-7vkcd PodName:netshootzjlcv-6757486d56-9f7w6 ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
  2383. 2020-10-08T11:40:40.8978011Z STEP: Validating that dig result are "10.241.224.4"
  2384. 2020-10-08T11:40:45.8211038Z Oct 8 11:40:45.820: INFO: ExecWithOptions {Command:[dig +short nginx-headless.e2e-tests-discovery-7vkcd.svc.clusterset.local] Namespace:e2e-tests-discovery-7vkcd PodName:netshootzjlcv-6757486d56-9f7w6 ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
  2385. 2020-10-08T11:40:45.8826934Z STEP: Validating that dig result are "10.241.224.4"
  2386. 2020-10-08T11:40:50.8209981Z Oct 8 11:40:50.820: INFO: ExecWithOptions {Command:[dig +short nginx-headless.e2e-tests-discovery-7vkcd.svc.clusterset.local] Namespace:e2e-tests-discovery-7vkcd PodName:netshootzjlcv-6757486d56-9f7w6 ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
  2387. 2020-10-08T11:40:50.9054488Z STEP: Validating that dig result are "10.241.224.4"
  2388. 2020-10-08T11:40:55.8212916Z Oct 8 11:40:55.820: INFO: ExecWithOptions {Command:[dig +short nginx-headless.e2e-tests-discovery-7vkcd.svc.clusterset.local] Namespace:e2e-tests-discovery-7vkcd PodName:netshootzjlcv-6757486d56-9f7w6 ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
  2389. 2020-10-08T11:40:55.8896934Z STEP: Validating that dig result are "10.241.224.4"
  2390. 2020-10-08T11:41:00.8213453Z Oct 8 11:41:00.820: INFO: ExecWithOptions {Command:[dig +short nginx-headless.e2e-tests-discovery-7vkcd.svc.clusterset.local] Namespace:e2e-tests-discovery-7vkcd PodName:netshootzjlcv-6757486d56-9f7w6 ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
  2391. 2020-10-08T11:41:00.9007261Z STEP: Validating that dig result are "10.241.224.4"
  2392. 2020-10-08T11:41:05.8212146Z Oct 8 11:41:05.820: INFO: ExecWithOptions {Command:[dig +short nginx-headless.e2e-tests-discovery-7vkcd.svc.clusterset.local] Namespace:e2e-tests-discovery-7vkcd PodName:netshootzjlcv-6757486d56-9f7w6 ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
  2393. 2020-10-08T11:41:05.8800280Z STEP: Validating that dig result are "10.241.224.4"
  2394. 2020-10-08T11:41:10.8227262Z Oct 8 11:41:10.820: INFO: ExecWithOptions {Command:[dig +short nginx-headless.e2e-tests-discovery-7vkcd.svc.clusterset.local] Namespace:e2e-tests-discovery-7vkcd PodName:netshootzjlcv-6757486d56-9f7w6 ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
  2395. 2020-10-08T11:41:10.8914320Z STEP: Validating that dig result are "10.241.224.4"
  2396. 2020-10-08T11:41:15.8211207Z Oct 8 11:41:15.820: INFO: ExecWithOptions {Command:[dig +short nginx-headless.e2e-tests-discovery-7vkcd.svc.clusterset.local] Namespace:e2e-tests-discovery-7vkcd PodName:netshootzjlcv-6757486d56-9f7w6 ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
  2397. 2020-10-08T11:41:15.8853718Z STEP: Validating that dig result are "10.241.224.4"
  2398. 2020-10-08T11:41:20.8213200Z Oct 8 11:41:20.820: INFO: ExecWithOptions {Command:[dig +short nginx-headless.e2e-tests-discovery-7vkcd.svc.clusterset.local] Namespace:e2e-tests-discovery-7vkcd PodName:netshootzjlcv-6757486d56-9f7w6 ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
  2399. 2020-10-08T11:41:20.8855297Z STEP: Validating that dig result are "10.241.224.4"
  2400. 2020-10-08T11:41:25.8211379Z Oct 8 11:41:25.820: INFO: ExecWithOptions {Command:[dig +short nginx-headless.e2e-tests-discovery-7vkcd.svc.clusterset.local] Namespace:e2e-tests-discovery-7vkcd PodName:netshootzjlcv-6757486d56-9f7w6 ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
  2401. 2020-10-08T11:41:25.8874330Z STEP: Validating that dig result are "10.241.224.4"
  2402. 2020-10-08T11:41:30.8211740Z Oct 8 11:41:30.820: INFO: ExecWithOptions {Command:[dig +short nginx-headless.e2e-tests-discovery-7vkcd.svc.clusterset.local] Namespace:e2e-tests-discovery-7vkcd PodName:netshootzjlcv-6757486d56-9f7w6 ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
  2403. 2020-10-08T11:41:30.8934179Z STEP: Validating that dig result are "10.241.224.4"
  2404. 2020-10-08T11:41:35.8210810Z Oct 8 11:41:35.820: INFO: ExecWithOptions {Command:[dig +short nginx-headless.e2e-tests-discovery-7vkcd.svc.clusterset.local] Namespace:e2e-tests-discovery-7vkcd PodName:netshootzjlcv-6757486d56-9f7w6 ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
  2405. 2020-10-08T11:41:35.8857701Z STEP: Validating that dig result are "10.241.224.4"
  2406. 2020-10-08T11:41:40.8210123Z Oct 8 11:41:40.820: INFO: ExecWithOptions {Command:[dig +short nginx-headless.e2e-tests-discovery-7vkcd.svc.clusterset.local] Namespace:e2e-tests-discovery-7vkcd PodName:netshootzjlcv-6757486d56-9f7w6 ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
  2407. 2020-10-08T11:41:40.8952989Z STEP: Validating that dig result are "10.241.224.4"
  2408. 2020-10-08T11:41:45.8213623Z Oct 8 11:41:45.820: INFO: ExecWithOptions {Command:[dig +short nginx-headless.e2e-tests-discovery-7vkcd.svc.clusterset.local] Namespace:e2e-tests-discovery-7vkcd PodName:netshootzjlcv-6757486d56-9f7w6 ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
  2409. 2020-10-08T11:41:45.8949374Z STEP: Validating that dig result are "10.241.224.4"
  2410. 2020-10-08T11:41:50.8211309Z Oct 8 11:41:50.820: INFO: ExecWithOptions {Command:[dig +short nginx-headless.e2e-tests-discovery-7vkcd.svc.clusterset.local] Namespace:e2e-tests-discovery-7vkcd PodName:netshootzjlcv-6757486d56-9f7w6 ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
  2411. 2020-10-08T11:41:50.9000732Z STEP: Validating that dig result are "10.241.224.4"
  2412. 2020-10-08T11:41:55.8210787Z Oct 8 11:41:55.820: INFO: ExecWithOptions {Command:[dig +short nginx-headless.e2e-tests-discovery-7vkcd.svc.clusterset.local] Namespace:e2e-tests-discovery-7vkcd PodName:netshootzjlcv-6757486d56-9f7w6 ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
  2413. 2020-10-08T11:41:55.8848228Z STEP: Validating that dig result are "10.241.224.4"
  2414. 2020-10-08T11:42:00.8215046Z Oct 8 11:42:00.820: INFO: ExecWithOptions {Command:[dig +short nginx-headless.e2e-tests-discovery-7vkcd.svc.clusterset.local] Namespace:e2e-tests-discovery-7vkcd PodName:netshootzjlcv-6757486d56-9f7w6 ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
  2415. 2020-10-08T11:42:00.9024162Z STEP: Validating that dig result are "10.241.224.4"
  2416. 2020-10-08T11:42:05.8218741Z Oct 8 11:42:05.820: INFO: ExecWithOptions {Command:[dig +short nginx-headless.e2e-tests-discovery-7vkcd.svc.clusterset.local] Namespace:e2e-tests-discovery-7vkcd PodName:netshootzjlcv-6757486d56-9f7w6 ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
  2417. 2020-10-08T11:42:05.8869001Z STEP: Validating that dig result are "10.241.224.4"
  2418. 2020-10-08T11:42:10.8210601Z Oct 8 11:42:10.820: INFO: ExecWithOptions {Command:[dig +short nginx-headless.e2e-tests-discovery-7vkcd.svc.clusterset.local] Namespace:e2e-tests-discovery-7vkcd PodName:netshootzjlcv-6757486d56-9f7w6 ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
  2419. 2020-10-08T11:42:10.8861399Z STEP: Validating that dig result are "10.241.224.4"
  2420. 2020-10-08T11:42:15.8214299Z Oct 8 11:42:15.820: INFO: ExecWithOptions {Command:[dig +short nginx-headless.e2e-tests-discovery-7vkcd.svc.clusterset.local] Namespace:e2e-tests-discovery-7vkcd PodName:netshootzjlcv-6757486d56-9f7w6 ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
  2421. 2020-10-08T11:42:15.8895529Z STEP: Validating that dig result are "10.241.224.4"
  2422. 2020-10-08T11:42:20.8211708Z Oct 8 11:42:20.820: INFO: ExecWithOptions {Command:[dig +short nginx-headless.e2e-tests-discovery-7vkcd.svc.clusterset.local] Namespace:e2e-tests-discovery-7vkcd PodName:netshootzjlcv-6757486d56-9f7w6 ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
  2423. 2020-10-08T11:42:20.8921574Z STEP: Validating that dig result are "10.241.224.4"
  2424. 2020-10-08T11:42:26.5291256Z Oct 8 11:42:25.820: INFO: ExecWithOptions {Command:[dig +short nginx-headless.e2e-tests-discovery-7vkcd.svc.clusterset.local] Namespace:e2e-tests-discovery-7vkcd PodName:netshootzjlcv-6757486d56-9f7w6 ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
  2425. 2020-10-08T11:42:26.5296010Z STEP: Validating that dig result are "10.241.224.4"
  2426. 2020-10-08T11:42:30.8213433Z Oct 8 11:42:30.820: INFO: ExecWithOptions {Command:[dig +short nginx-headless.e2e-tests-discovery-7vkcd.svc.clusterset.local] Namespace:e2e-tests-discovery-7vkcd PodName:netshootzjlcv-6757486d56-9f7w6 ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
  2427. 2020-10-08T11:42:30.8973819Z STEP: Validating that dig result are "10.241.224.4"
  2428. 2020-10-08T11:42:35.8213807Z Oct 8 11:42:35.820: INFO: ExecWithOptions {Command:[dig +short nginx-headless.e2e-tests-discovery-7vkcd.svc.clusterset.local] Namespace:e2e-tests-discovery-7vkcd PodName:netshootzjlcv-6757486d56-9f7w6 ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
  2429. 2020-10-08T11:42:35.8852021Z STEP: Validating that dig result are "10.241.224.4"
  2430. 2020-10-08T11:42:40.8215482Z Oct 8 11:42:40.820: INFO: ExecWithOptions {Command:[dig +short nginx-headless.e2e-tests-discovery-7vkcd.svc.clusterset.local] Namespace:e2e-tests-discovery-7vkcd PodName:netshootzjlcv-6757486d56-9f7w6 ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
  2431. 2020-10-08T11:42:40.8891206Z STEP: Validating that dig result are "10.241.224.4"
  2432. 2020-10-08T11:42:40.8902559Z Oct 8 11:42:40.889: INFO: ExecWithOptions {Command:[dig +short nginx-headless.e2e-tests-discovery-7vkcd.svc.clusterset.local] Namespace:e2e-tests-discovery-7vkcd PodName:netshootzjlcv-6757486d56-9f7w6 ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
  2433. 2020-10-08T11:42:40.9584966Z STEP: Validating that dig result are "10.241.224.4"
  2434. 2020-10-08T11:42:40.9593816Z STEP: Deleting namespace "e2e-tests-discovery-7vkcd" on cluster "cluster1"
  2435. 2020-10-08T11:42:40.9710007Z STEP: Deleting namespace "e2e-tests-discovery-7vkcd" on cluster "cluster2"
  2436. 2020-10-08T11:42:40.9829919Z
  2437. 2020-10-08T11:42:40.9837415Z • Failure [215.574 seconds]
  2438. 2020-10-08T11:42:40.9838156Z [discovery] Test Headless Service Discovery Across Clusters
  2439. 2020-10-08T11:42:40.9839242Z /go/src/github.com/submariner-io/lighthouse/test/e2e/discovery/headless_services.go:13
  2440. 2020-10-08T11:42:40.9845216Z when a pod tries to resolve a headless service which is exported locally and in a remote cluster
  2441. 2020-10-08T11:42:40.9846621Z /go/src/github.com/submariner-io/lighthouse/test/e2e/discovery/headless_services.go:26
  2442. 2020-10-08T11:42:40.9847556Z should resolve the backing pod IPs from both clusters [It]
  2443. 2020-10-08T11:42:40.9848498Z /go/src/github.com/submariner-io/lighthouse/test/e2e/discovery/headless_services.go:27
  2444. 2020-10-08T11:42:40.9849004Z
  2445. 2020-10-08T11:42:40.9849730Z Failed to service IP verification. expected execution result "10.241.224.4" to contain "10.242.192.4"
  2446. 2020-10-08T11:42:40.9850274Z Unexpected error:
  2447. 2020-10-08T11:42:40.9850707Z <*errors.errorString | 0xc0002f8390>: {
  2448. 2020-10-08T11:42:40.9851210Z s: "timed out waiting for the condition",
  2449. 2020-10-08T11:42:40.9851541Z }
  2450. 2020-10-08T11:42:40.9851864Z timed out waiting for the condition
  2451. 2020-10-08T11:42:40.9852918Z occurred
  2452. 2020-10-08T11:42:40.9853114Z
  2453. 2020-10-08T11:42:40.9854051Z /go/src/github.com/submariner-io/lighthouse/vendor/github.com/submariner-io/shipyard/test/e2e/framework/framework.go:371
  2454. 2020-10-08T11:42:40.9854689Z
  2455. 2020-10-08T11:42:40.9855094Z Full Stack Trace
  2456. 2020-10-08T11:42:40.9856036Z github.com/submariner-io/shipyard/test/e2e/framework.AwaitUntil(0x14f402f, 0x18, 0xc0000191a8, 0xc000019170, 0x0, 0xc00013fef0)
  2457. 2020-10-08T11:42:40.9857404Z /go/src/github.com/submariner-io/lighthouse/vendor/github.com/submariner-io/shipyard/test/e2e/framework/framework.go:371 +0x1c6
  2458. 2020-10-08T11:42:40.9859134Z github.com/submariner-io/lighthouse/test/e2e/discovery.verifyHeadlessIpsWithDig(0xc0002bd900, 0x0, 0xc0000dc240, 0xc00041b880, 0xc000294ce0, 0x2, 0x2, 0x21649d0, 0x1, 0x1, ...)
  2459. 2020-10-08T11:42:40.9860785Z /go/src/github.com/submariner-io/lighthouse/test/e2e/discovery/headless_services.go:159 +0x544
  2460. 2020-10-08T11:42:40.9862701Z github.com/submariner-io/lighthouse/test/e2e/discovery.RunHeadlessDiscoveryLocalAndRemoteTest(0xc0000b2d20)
  2461. 2020-10-08T11:42:40.9864511Z /go/src/github.com/submariner-io/lighthouse/test/e2e/discovery/headless_services.go:105 +0x7a7
  2462. 2020-10-08T11:42:40.9865624Z github.com/submariner-io/lighthouse/test/e2e/discovery.glob..func1.2.1()
  2463. 2020-10-08T11:42:40.9866746Z /go/src/github.com/submariner-io/lighthouse/test/e2e/discovery/headless_services.go:29 +0x6e
  2464. 2020-10-08T11:42:40.9867810Z github.com/submariner-io/shipyard/test/e2e.RunE2ETests(0xc0003b6000, 0xd0de676cbe)
  2465. 2020-10-08T11:42:40.9868950Z /go/src/github.com/submariner-io/lighthouse/vendor/github.com/submariner-io/shipyard/test/e2e/e2e.go:77 +0x122
  2466. 2020-10-08T11:42:40.9869987Z github.com/submariner-io/lighthouse/test/e2e.TestE2E(0xc0003b6000)
  2467. 2020-10-08T11:42:40.9870957Z /go/src/github.com/submariner-io/lighthouse/test/e2e/e2e_test.go:11 +0x2b
  2468. 2020-10-08T11:42:40.9871498Z testing.tRunner(0xc0003b6000, 0x1581730)
  2469. 2020-10-08T11:42:40.9871942Z /usr/lib/golang/src/testing/testing.go:1054 +0xdc
  2470. 2020-10-08T11:42:40.9872330Z created by testing.(*T).Run
  2471. 2020-10-08T11:42:40.9872718Z /usr/lib/golang/src/testing/testing.go:1105 +0x372
  2472. 2020-10-08T11:42:40.9873209Z ------------------------------
  2473. 2020-10-08T11:42:40.9874090Z [discovery] Test Service Discovery Across Clusters when a pod tries to resolve a service which is present locally and in a remote cluster
  2474. 2020-10-08T11:42:40.9874935Z should resolve the local service
  2475. 2020-10-08T11:42:40.9875731Z /go/src/github.com/submariner-io/lighthouse/test/e2e/discovery/service_discovery.go:32
  2476. 2020-10-08T11:42:40.9876577Z STEP: Creating namespace objects with basename "discovery"
  2477. 2020-10-08T11:42:40.9946221Z STEP: Generated namespace "e2e-tests-discovery-86gct" in cluster "cluster1" to execute the tests in
  2478. 2020-10-08T11:42:40.9947270Z STEP: Creating namespace "e2e-tests-discovery-86gct" in cluster "cluster2"
  2479. 2020-10-08T11:42:40.9985458Z STEP: Creating an Nginx Deployment on "cluster1"
  2480. 2020-10-08T11:42:46.0490645Z STEP: Creating a Nginx Service on "cluster1"
  2481. 2020-10-08T11:42:46.0691325Z STEP: Creating an Nginx Deployment on "cluster2"
  2482. 2020-10-08T11:42:51.0915374Z STEP: Creating a Nginx Service on "cluster2"
  2483. 2020-10-08T11:42:51.0999779Z STEP: Creating serviceExport nginx-demo.e2e-tests-discovery-86gct on "cluster2"
  2484. 2020-10-08T11:42:51.1278399Z STEP: Retrieving ServiceExport nginx-demo.e2e-tests-discovery-86gct on "cluster2"
  2485. 2020-10-08T11:42:56.1441311Z STEP: Creating a Netshoot Deployment on "cluster1"
  2486. 2020-10-08T11:43:01.1628541Z Oct 8 11:43:01.162: INFO: ExecWithOptions {Command:[cat /etc/resolv.conf] Namespace:e2e-tests-discovery-86gct PodName:netshoots7tq4-6757486d56-qxtrg ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
  2487. 2020-10-08T11:43:01.2275658Z STEP: Executing "dig +short nginx-demo.e2e-tests-discovery-86gct.svc.cluster1.local" to verify IP "100.91.81.166" for service "nginx-demo" "is" discoverable
  2488. 2020-10-08T11:43:01.2283109Z Oct 8 11:43:01.227: INFO: ExecWithOptions {Command:[dig +short nginx-demo.e2e-tests-discovery-86gct.svc.cluster1.local] Namespace:e2e-tests-discovery-86gct PodName:netshoots7tq4-6757486d56-qxtrg ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
  2489. 2020-10-08T11:43:01.3503165Z STEP: Validating that dig result is "100.91.81.166"
  2490. 2020-10-08T11:43:01.3504022Z STEP: Deleting service "nginx-demo" on "cluster1"
  2491. 2020-10-08T11:43:01.3854799Z STEP: Retrieving service nginx-demo.e2e-tests-discovery-86gct on "cluster2"
  2492. 2020-10-08T11:43:01.3891202Z STEP: Retrieving ServiceImport for nginx-demo-e2e-tests-discovery-86gct- on "cluster1"
  2493. 2020-10-08T11:43:01.4230476Z STEP: Executing "dig +short nginx-demo.e2e-tests-discovery-86gct.svc.clusterset.local" to verify IP "100.92.200.243" for service "nginx-demo" "is" discoverable
  2494. 2020-10-08T11:43:01.4234150Z Oct 8 11:43:01.422: INFO: ExecWithOptions {Command:[dig +short nginx-demo.e2e-tests-discovery-86gct.svc.clusterset.local] Namespace:e2e-tests-discovery-86gct PodName:netshoots7tq4-6757486d56-qxtrg ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
  2495. 2020-10-08T11:43:01.5381635Z STEP: Validating that dig result is "100.92.200.243"
  2496. 2020-10-08T11:43:01.5382732Z STEP: Deleting serviceExport nginx-demo.e2e-tests-discovery-86gct on "cluster2"
  2497. 2020-10-08T11:43:06.5474120Z STEP: Deleting service "nginx-demo" on "cluster2"
  2498. 2020-10-08T11:43:06.5726929Z STEP: Executing "dig +short nginx-demo.e2e-tests-discovery-86gct.svc.clusterset.local" to verify IP "100.92.200.243" for service "nginx-demo" "is not" discoverable
  2499. 2020-10-08T11:43:06.5730915Z Oct 8 11:43:06.571: INFO: ExecWithOptions {Command:[dig +short nginx-demo.e2e-tests-discovery-86gct.svc.clusterset.local] Namespace:e2e-tests-discovery-86gct PodName:netshoots7tq4-6757486d56-qxtrg ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
  2500. 2020-10-08T11:43:12.7150408Z STEP: Validating that dig result is not ""
  2501. 2020-10-08T11:43:12.7151285Z STEP: Deleting namespace "e2e-tests-discovery-86gct" on cluster "cluster1"
  2502. 2020-10-08T11:43:12.7190954Z STEP: Deleting namespace "e2e-tests-discovery-86gct" on cluster "cluster2"
  2503. 2020-10-08T11:43:12.7246607Z •
  2504. 2020-10-08T11:43:12.7247120Z ------------------------------
  2505. 2020-10-08T11:43:12.7248392Z STEP: Creating namespace objects with basename "discovery"
  2506. 2020-10-08T11:43:12.7249367Z STEP: Generated namespace "e2e-tests-discovery-86gct" in cluster "cluster1" to execute the tests in
  2507. 2020-10-08T11:43:12.7250645Z STEP: Creating namespace "e2e-tests-discovery-86gct" in cluster "cluster2"
  2508. 2020-10-08T11:43:12.7251499Z STEP: Creating an Nginx Deployment on "cluster1"
  2509. 2020-10-08T11:43:12.7252142Z STEP: Creating a Nginx Service on "cluster1"
  2510. 2020-10-08T11:43:12.7253262Z STEP: Creating an Nginx Deployment on "cluster2"
  2511. 2020-10-08T11:43:12.7253889Z STEP: Creating a Nginx Service on "cluster2"
  2512. 2020-10-08T11:43:12.7254879Z STEP: Creating serviceExport nginx-demo.e2e-tests-discovery-86gct on "cluster2"
  2513. 2020-10-08T11:43:12.7256233Z STEP: Retrieving ServiceExport nginx-demo.e2e-tests-discovery-86gct on "cluster2"
  2514. 2020-10-08T11:43:12.7257249Z STEP: Creating a Netshoot Deployment on "cluster1"
  2515. 2020-10-08T11:43:12.7259014Z Oct 8 11:43:01.162: INFO: ExecWithOptions {Command:[cat /etc/resolv.conf] Namespace:e2e-tests-discovery-86gct PodName:netshoots7tq4-6757486d56-qxtrg ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
  2516. 2020-10-08T11:43:12.7261517Z STEP: Executing "dig +short nginx-demo.e2e-tests-discovery-86gct.svc.cluster1.local" to verify IP "100.91.81.166" for service "nginx-demo" "is" discoverable
  2517. 2020-10-08T11:43:12.7264923Z Oct 8 11:43:01.227: INFO: ExecWithOptions {Command:[dig +short nginx-demo.e2e-tests-discovery-86gct.svc.cluster1.local] Namespace:e2e-tests-discovery-86gct PodName:netshoots7tq4-6757486d56-qxtrg ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
  2518. 2020-10-08T11:43:12.7267132Z STEP: Validating that dig result is "100.91.81.166"
  2519. 2020-10-08T11:43:12.7267755Z STEP: Deleting service "nginx-demo" on "cluster1"
  2520. 2020-10-08T11:43:12.7268718Z STEP: Retrieving service nginx-demo.e2e-tests-discovery-86gct on "cluster2"
  2521. 2020-10-08T11:43:12.7270077Z STEP: Retrieving ServiceImport for nginx-demo-e2e-tests-discovery-86gct- on "cluster1"
  2522. 2020-10-08T11:43:12.7272078Z STEP: Executing "dig +short nginx-demo.e2e-tests-discovery-86gct.svc.clusterset.local" to verify IP "100.92.200.243" for service "nginx-demo" "is" discoverable
  2523. 2020-10-08T11:43:12.7275731Z Oct 8 11:43:01.422: INFO: ExecWithOptions {Command:[dig +short nginx-demo.e2e-tests-discovery-86gct.svc.clusterset.local] Namespace:e2e-tests-discovery-86gct PodName:netshoots7tq4-6757486d56-qxtrg ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
  2524. 2020-10-08T11:43:12.7278287Z STEP: Validating that dig result is "100.92.200.243"
  2525. 2020-10-08T11:43:12.7279339Z STEP: Deleting serviceExport nginx-demo.e2e-tests-discovery-86gct on "cluster2"
  2526. 2020-10-08T11:43:12.7280401Z STEP: Deleting service "nginx-demo" on "cluster2"
  2527. 2020-10-08T11:43:12.7282148Z STEP: Executing "dig +short nginx-demo.e2e-tests-discovery-86gct.svc.clusterset.local" to verify IP "100.92.200.243" for service "nginx-demo" "is not" discoverable
  2528. 2020-10-08T11:43:12.7285753Z Oct 8 11:43:06.571: INFO: ExecWithOptions {Command:[dig +short nginx-demo.e2e-tests-discovery-86gct.svc.clusterset.local] Namespace:e2e-tests-discovery-86gct PodName:netshoots7tq4-6757486d56-qxtrg ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
  2529. 2020-10-08T11:43:12.7288166Z STEP: Validating that dig result is not ""
  2530. 2020-10-08T11:43:12.7288970Z STEP: Deleting namespace "e2e-tests-discovery-86gct" on cluster "cluster1"
  2531. 2020-10-08T11:43:12.7289973Z STEP: Deleting namespace "e2e-tests-discovery-86gct" on cluster "cluster2"
  2532. 2020-10-08T11:43:12.7290428Z
  2533. 2020-10-08T11:43:12.7290580Z
  2534. 2020-10-08T11:43:12.7291322Z [discovery] Test Service Discovery Across Clusters when a pod tries to resolve a service in a remote cluster
  2535. 2020-10-08T11:43:12.7293404Z should be able to discover the remote service successfully
  2536. 2020-10-08T11:43:12.7363412Z /go/src/github.com/submariner-io/lighthouse/test/e2e/discovery/service_discovery.go:26
  2537. 2020-10-08T11:43:12.7364325Z STEP: Creating namespace objects with basename "discovery"
  2538. 2020-10-08T11:43:12.7365716Z STEP: Generated namespace "e2e-tests-discovery-9jdkf" in cluster "cluster1" to execute the tests in
  2539. 2020-10-08T11:43:12.7366920Z STEP: Creating namespace "e2e-tests-discovery-9jdkf" in cluster "cluster2"
  2540. 2020-10-08T11:43:12.7521897Z STEP: Creating an Nginx Deployment on on "cluster2"
  2541. 2020-10-08T11:43:17.7795276Z STEP: Creating a Nginx Service on "cluster2"
  2542. 2020-10-08T11:43:17.7943816Z STEP: Creating serviceExport nginx-demo.e2e-tests-discovery-9jdkf on "cluster2"
  2543. 2020-10-08T11:43:17.8299975Z STEP: Retrieving ServiceExport nginx-demo.e2e-tests-discovery-9jdkf on "cluster2"
  2544. 2020-10-08T11:43:22.8395311Z STEP: Creating a Netshoot Deployment on "cluster1"
  2545. 2020-10-08T11:43:27.8490090Z STEP: Retrieving service nginx-demo.e2e-tests-discovery-9jdkf on "cluster2"
  2546. 2020-10-08T11:43:27.8509699Z STEP: Retrieving ServiceImport for nginx-demo-e2e-tests-discovery-9jdkf- on "cluster1"
  2547. 2020-10-08T11:43:27.8541147Z STEP: Executing "dig +short nginx-demo.e2e-tests-discovery-9jdkf.svc.clusterset.local" to verify IP "100.92.199.162" for service "nginx-demo" "is" discoverable
  2548. 2020-10-08T11:43:27.8548638Z Oct 8 11:43:27.853: INFO: ExecWithOptions {Command:[dig +short nginx-demo.e2e-tests-discovery-9jdkf.svc.clusterset.local] Namespace:e2e-tests-discovery-9jdkf PodName:netshootfcfk5-6757486d56-jz86x ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
  2549. 2020-10-08T11:43:27.9294893Z STEP: Validating that dig result is "100.92.199.162"
  2550. 2020-10-08T11:43:27.9299679Z STEP: Deleting serviceExport nginx-demo.e2e-tests-discovery-9jdkf on "cluster2"
  2551. 2020-10-08T11:43:32.9403928Z STEP: Deleting service "nginx-demo" on "cluster2"
  2552. 2020-10-08T11:43:32.9723634Z STEP: Executing "dig +short nginx-demo.e2e-tests-discovery-9jdkf.svc.clusterset.local" to verify IP "100.92.199.162" for service "nginx-demo" "is not" discoverable
  2553. 2020-10-08T11:43:32.9731800Z Oct 8 11:43:32.971: INFO: ExecWithOptions {Command:[dig +short nginx-demo.e2e-tests-discovery-9jdkf.svc.clusterset.local] Namespace:e2e-tests-discovery-9jdkf PodName:netshootfcfk5-6757486d56-jz86x ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
  2554. 2020-10-08T11:43:39.0623969Z STEP: Validating that dig result is not ""
  2555. 2020-10-08T11:43:39.0627557Z STEP: Deleting namespace "e2e-tests-discovery-9jdkf" on cluster "cluster1"
  2556. 2020-10-08T11:43:39.0666410Z STEP: Deleting namespace "e2e-tests-discovery-9jdkf" on cluster "cluster2"
  2557. 2020-10-08T11:43:39.0710792Z •
  2558. 2020-10-08T11:43:39.0711210Z ------------------------------
  2559. 2020-10-08T11:43:39.0711937Z STEP: Creating namespace objects with basename "discovery"
  2560. 2020-10-08T11:43:39.0714599Z STEP: Generated namespace "e2e-tests-discovery-9jdkf" in cluster "cluster1" to execute the tests in
  2561. 2020-10-08T11:43:39.0717321Z STEP: Creating namespace "e2e-tests-discovery-9jdkf" in cluster "cluster2"
  2562. 2020-10-08T11:43:39.0718388Z STEP: Creating an Nginx Deployment on on "cluster2"
  2563. 2020-10-08T11:43:39.0719085Z STEP: Creating a Nginx Service on "cluster2"
  2564. 2020-10-08T11:43:39.0720909Z STEP: Creating serviceExport nginx-demo.e2e-tests-discovery-9jdkf on "cluster2"
  2565. 2020-10-08T11:43:39.0722117Z STEP: Retrieving ServiceExport nginx-demo.e2e-tests-discovery-9jdkf on "cluster2"
  2566. 2020-10-08T11:43:39.0734324Z STEP: Creating a Netshoot Deployment on "cluster1"
  2567. 2020-10-08T11:43:39.0735497Z STEP: Retrieving service nginx-demo.e2e-tests-discovery-9jdkf on "cluster2"
  2568. 2020-10-08T11:43:39.0736939Z STEP: Retrieving ServiceImport for nginx-demo-e2e-tests-discovery-9jdkf- on "cluster1"
  2569. 2020-10-08T11:43:39.0739092Z STEP: Executing "dig +short nginx-demo.e2e-tests-discovery-9jdkf.svc.clusterset.local" to verify IP "100.92.199.162" for service "nginx-demo" "is" discoverable
  2570. 2020-10-08T11:43:39.0742521Z Oct 8 11:43:27.853: INFO: ExecWithOptions {Command:[dig +short nginx-demo.e2e-tests-discovery-9jdkf.svc.clusterset.local] Namespace:e2e-tests-discovery-9jdkf PodName:netshootfcfk5-6757486d56-jz86x ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
  2571. 2020-10-08T11:43:39.0744828Z STEP: Validating that dig result is "100.92.199.162"
  2572. 2020-10-08T11:43:39.0745814Z STEP: Deleting serviceExport nginx-demo.e2e-tests-discovery-9jdkf on "cluster2"
  2573. 2020-10-08T11:43:39.0746811Z STEP: Deleting service "nginx-demo" on "cluster2"
  2574. 2020-10-08T11:43:39.0748455Z STEP: Executing "dig +short nginx-demo.e2e-tests-discovery-9jdkf.svc.clusterset.local" to verify IP "100.92.199.162" for service "nginx-demo" "is not" discoverable
  2575. 2020-10-08T11:43:39.0751884Z Oct 8 11:43:32.971: INFO: ExecWithOptions {Command:[dig +short nginx-demo.e2e-tests-discovery-9jdkf.svc.clusterset.local] Namespace:e2e-tests-discovery-9jdkf PodName:netshootfcfk5-6757486d56-jz86x ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
  2576. 2020-10-08T11:43:39.0754119Z STEP: Validating that dig result is not ""
  2577. 2020-10-08T11:43:39.0754930Z STEP: Deleting namespace "e2e-tests-discovery-9jdkf" on cluster "cluster1"
  2578. 2020-10-08T11:43:39.0755942Z STEP: Deleting namespace "e2e-tests-discovery-9jdkf" on cluster "cluster2"
  2579. 2020-10-08T11:43:39.0756418Z
  2580. 2020-10-08T11:43:39.0756554Z
  2581. 2020-10-08T11:43:39.0757301Z [discovery] Test Service Discovery Across Clusters when there are active pods for a service in only one cluster
  2582. 2020-10-08T11:43:39.0758196Z should not resolve the service on the cluster without active pods
  2583. 2020-10-08T11:43:39.0759088Z /go/src/github.com/submariner-io/lighthouse/test/e2e/discovery/service_discovery.go:50
  2584. 2020-10-08T11:43:39.0768710Z STEP: Creating namespace objects with basename "discovery"
  2585. 2020-10-08T11:43:39.0816378Z STEP: Generated namespace "e2e-tests-discovery-pv46p" in cluster "cluster1" to execute the tests in
  2586. 2020-10-08T11:43:39.0817755Z STEP: Creating namespace "e2e-tests-discovery-pv46p" in cluster "cluster2"
  2587. 2020-10-08T11:43:39.0939019Z STEP: Creating an Nginx Deployment on on "cluster2"
  2588. 2020-10-08T11:43:44.1539973Z STEP: Creating a Nginx Service on "cluster2"
  2589. 2020-10-08T11:43:44.1621248Z STEP: Creating serviceExport nginx-demo.e2e-tests-discovery-pv46p on "cluster2"
  2590. 2020-10-08T11:43:44.1930239Z STEP: Retrieving ServiceExport nginx-demo.e2e-tests-discovery-pv46p on "cluster2"
  2591. 2020-10-08T11:43:49.2071239Z STEP: Creating a Netshoot Deployment on "cluster1"
  2592. 2020-10-08T11:43:54.2157345Z STEP: Retrieving service nginx-demo.e2e-tests-discovery-pv46p on "cluster2"
  2593. 2020-10-08T11:43:54.2170874Z STEP: Retrieving ServiceImport for nginx-demo-e2e-tests-discovery-pv46p- on "cluster1"
  2594. 2020-10-08T11:43:54.2197233Z STEP: Creating an Nginx Deployment on on "cluster1"
  2595. 2020-10-08T11:43:59.2317893Z STEP: Creating a Nginx Service on "cluster1"
  2596. 2020-10-08T11:43:59.2374393Z STEP: Creating serviceExport nginx-demo.e2e-tests-discovery-pv46p on "cluster1"
  2597. 2020-10-08T11:43:59.2563431Z STEP: Retrieving ServiceExport nginx-demo.e2e-tests-discovery-pv46p on "cluster1"
  2598. 2020-10-08T11:44:04.2716311Z STEP: Retrieving service nginx-demo.e2e-tests-discovery-pv46p on "cluster1"
  2599. 2020-10-08T11:44:04.2737100Z STEP: Retrieving ServiceImport for nginx-demo-e2e-tests-discovery-pv46p- on "cluster1"
  2600. 2020-10-08T11:44:04.2760639Z STEP: Executing "dig +short nginx-demo.e2e-tests-discovery-pv46p.svc.clusterset.local" to verify IP "100.91.181.172" for service "nginx-demo" "is" discoverable
  2601. 2020-10-08T11:44:04.2767997Z Oct 8 11:44:04.276: INFO: ExecWithOptions {Command:[dig +short nginx-demo.e2e-tests-discovery-pv46p.svc.clusterset.local] Namespace:e2e-tests-discovery-pv46p PodName:netshootqvtld-6757486d56-pnd7x ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
  2602. 2020-10-08T11:44:04.3581777Z STEP: Validating that dig result is "100.92.225.68"
  2603. 2020-10-08T11:44:09.3587929Z Oct 8 11:44:09.358: INFO: ExecWithOptions {Command:[dig +short nginx-demo.e2e-tests-discovery-pv46p.svc.clusterset.local] Namespace:e2e-tests-discovery-pv46p PodName:netshootqvtld-6757486d56-pnd7x ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
  2604. 2020-10-08T11:44:09.4190425Z STEP: Validating that dig result is "100.91.181.172"
  2605. 2020-10-08T11:44:09.4196798Z STEP: Executing "dig +short nginx-demo.e2e-tests-discovery-pv46p.svc.clusterset.local" to verify IP "100.92.225.68" for service "nginx-demo" "is" discoverable
  2606. 2020-10-08T11:44:09.4204620Z Oct 8 11:44:09.419: INFO: ExecWithOptions {Command:[dig +short nginx-demo.e2e-tests-discovery-pv46p.svc.clusterset.local] Namespace:e2e-tests-discovery-pv46p PodName:netshootqvtld-6757486d56-pnd7x ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
  2607. 2020-10-08T11:44:09.4899308Z STEP: Validating that dig result is "100.92.225.68"
  2608. 2020-10-08T11:44:09.4902648Z STEP: Setting Nginx deployment replicas to 0 in cluster "cluster2"
  2609. 2020-10-08T11:44:09.4953774Z STEP: Executing "dig +short nginx-demo.e2e-tests-discovery-pv46p.svc.clusterset.local" to verify IP "100.91.181.172" for service "nginx-demo" "is" discoverable
  2610. 2020-10-08T11:44:09.4957471Z Oct 8 11:44:09.494: INFO: ExecWithOptions {Command:[dig +short nginx-demo.e2e-tests-discovery-pv46p.svc.clusterset.local] Namespace:e2e-tests-discovery-pv46p PodName:netshootqvtld-6757486d56-pnd7x ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
  2611. 2020-10-08T11:44:09.6297175Z STEP: Validating that dig result is "100.91.181.172"
  2612. 2020-10-08T11:44:09.6319794Z STEP: Executing "dig +short nginx-demo.e2e-tests-discovery-pv46p.svc.clusterset.local" to verify IP "100.92.225.68" for service "nginx-demo" "is not" discoverable
  2613. 2020-10-08T11:44:09.6324786Z Oct 8 11:44:09.630: INFO: ExecWithOptions {Command:[dig +short nginx-demo.e2e-tests-discovery-pv46p.svc.clusterset.local] Namespace:e2e-tests-discovery-pv46p PodName:netshootqvtld-6757486d56-pnd7x ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
  2614. 2020-10-08T11:44:09.8183173Z STEP: Validating that dig result is not "100.91.181.172"
  2615. 2020-10-08T11:44:09.8184009Z STEP: Setting Nginx deployment replicas to 0 in cluster "cluster1"
  2616. 2020-10-08T11:44:09.8244861Z STEP: Executing "dig +short nginx-demo.e2e-tests-discovery-pv46p.svc.clusterset.local" to verify IP "100.91.181.172" for service "nginx-demo" "is not" discoverable
  2617. 2020-10-08T11:44:09.8248727Z Oct 8 11:44:09.823: INFO: ExecWithOptions {Command:[dig +short nginx-demo.e2e-tests-discovery-pv46p.svc.clusterset.local] Namespace:e2e-tests-discovery-pv46p PodName:netshootqvtld-6757486d56-pnd7x ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
  2618. 2020-10-08T11:44:09.9877111Z STEP: Validating that dig result is not ""
  2619. 2020-10-08T11:44:09.9878947Z STEP: Executing "dig +short nginx-demo.e2e-tests-discovery-pv46p.svc.clusterset.local" to verify IP "100.92.225.68" for service "nginx-demo" "is not" discoverable
  2620. 2020-10-08T11:44:09.9887194Z Oct 8 11:44:09.987: INFO: ExecWithOptions {Command:[dig +short nginx-demo.e2e-tests-discovery-pv46p.svc.clusterset.local] Namespace:e2e-tests-discovery-pv46p PodName:netshootqvtld-6757486d56-pnd7x ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
  2621. 2020-10-08T11:44:10.0997206Z STEP: Validating that dig result is not ""
  2622. 2020-10-08T11:44:10.0998142Z STEP: Deleting namespace "e2e-tests-discovery-pv46p" on cluster "cluster1"
  2623. 2020-10-08T11:44:10.1049714Z STEP: Deleting namespace "e2e-tests-discovery-pv46p" on cluster "cluster2"
  2624. 2020-10-08T11:44:10.1106711Z •
  2625. 2020-10-08T11:44:10.1107116Z ------------------------------
  2626. 2020-10-08T11:44:10.1107917Z STEP: Creating namespace objects with basename "discovery"
  2627. 2020-10-08T11:44:10.1108924Z STEP: Generated namespace "e2e-tests-discovery-pv46p" in cluster "cluster1" to execute the tests in
  2628. 2020-10-08T11:44:10.1110022Z STEP: Creating namespace "e2e-tests-discovery-pv46p" in cluster "cluster2"
  2629. 2020-10-08T11:44:10.1110857Z STEP: Creating an Nginx Deployment on on "cluster2"
  2630. 2020-10-08T11:44:10.1111485Z STEP: Creating a Nginx Service on "cluster2"
  2631. 2020-10-08T11:44:10.1112466Z STEP: Creating serviceExport nginx-demo.e2e-tests-discovery-pv46p on "cluster2"
  2632. 2020-10-08T11:44:10.1113821Z STEP: Retrieving ServiceExport nginx-demo.e2e-tests-discovery-pv46p on "cluster2"
  2633. 2020-10-08T11:44:10.1114830Z STEP: Creating a Netshoot Deployment on "cluster1"
  2634. 2020-10-08T11:44:10.1115813Z STEP: Retrieving service nginx-demo.e2e-tests-discovery-pv46p on "cluster2"
  2635. 2020-10-08T11:44:10.1117171Z STEP: Retrieving ServiceImport for nginx-demo-e2e-tests-discovery-pv46p- on "cluster1"
  2636. 2020-10-08T11:44:10.1118181Z STEP: Creating an Nginx Deployment on on "cluster1"
  2637. 2020-10-08T11:44:10.1118807Z STEP: Creating a Nginx Service on "cluster1"
  2638. 2020-10-08T11:44:10.1119799Z STEP: Creating serviceExport nginx-demo.e2e-tests-discovery-pv46p on "cluster1"
  2639. 2020-10-08T11:44:10.1121147Z STEP: Retrieving ServiceExport nginx-demo.e2e-tests-discovery-pv46p on "cluster1"
  2640. 2020-10-08T11:44:10.1122459Z STEP: Retrieving service nginx-demo.e2e-tests-discovery-pv46p on "cluster1"
  2641. 2020-10-08T11:44:10.1123806Z STEP: Retrieving ServiceImport for nginx-demo-e2e-tests-discovery-pv46p- on "cluster1"
  2642. 2020-10-08T11:44:10.1125783Z STEP: Executing "dig +short nginx-demo.e2e-tests-discovery-pv46p.svc.clusterset.local" to verify IP "100.91.181.172" for service "nginx-demo" "is" discoverable
  2643. 2020-10-08T11:44:10.1129175Z Oct 8 11:44:04.276: INFO: ExecWithOptions {Command:[dig +short nginx-demo.e2e-tests-discovery-pv46p.svc.clusterset.local] Namespace:e2e-tests-discovery-pv46p PodName:netshootqvtld-6757486d56-pnd7x ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
  2644. 2020-10-08T11:44:10.1131570Z STEP: Validating that dig result is "100.92.225.68"
  2645. 2020-10-08T11:44:10.1135456Z Oct 8 11:44:09.358: INFO: ExecWithOptions {Command:[dig +short nginx-demo.e2e-tests-discovery-pv46p.svc.clusterset.local] Namespace:e2e-tests-discovery-pv46p PodName:netshootqvtld-6757486d56-pnd7x ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
  2646. 2020-10-08T11:44:10.1137840Z STEP: Validating that dig result is "100.91.181.172"
  2647. 2020-10-08T11:44:10.1139426Z STEP: Executing "dig +short nginx-demo.e2e-tests-discovery-pv46p.svc.clusterset.local" to verify IP "100.92.225.68" for service "nginx-demo" "is" discoverable
  2648. 2020-10-08T11:44:10.1142798Z Oct 8 11:44:09.419: INFO: ExecWithOptions {Command:[dig +short nginx-demo.e2e-tests-discovery-pv46p.svc.clusterset.local] Namespace:e2e-tests-discovery-pv46p PodName:netshootqvtld-6757486d56-pnd7x ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
  2649. 2020-10-08T11:44:10.1145054Z STEP: Validating that dig result is "100.92.225.68"
  2650. 2020-10-08T11:44:10.1145964Z STEP: Setting Nginx deployment replicas to 0 in cluster "cluster2"
  2651. 2020-10-08T11:44:10.1147742Z STEP: Executing "dig +short nginx-demo.e2e-tests-discovery-pv46p.svc.clusterset.local" to verify IP "100.91.181.172" for service "nginx-demo" "is" discoverable
  2652. 2020-10-08T11:44:10.1151214Z Oct 8 11:44:09.494: INFO: ExecWithOptions {Command:[dig +short nginx-demo.e2e-tests-discovery-pv46p.svc.clusterset.local] Namespace:e2e-tests-discovery-pv46p PodName:netshootqvtld-6757486d56-pnd7x ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
  2653. 2020-10-08T11:44:10.1153667Z STEP: Validating that dig result is "100.91.181.172"
  2654. 2020-10-08T11:44:10.1155708Z STEP: Executing "dig +short nginx-demo.e2e-tests-discovery-pv46p.svc.clusterset.local" to verify IP "100.92.225.68" for service "nginx-demo" "is not" discoverable
  2655. 2020-10-08T11:44:10.1159200Z Oct 8 11:44:09.630: INFO: ExecWithOptions {Command:[dig +short nginx-demo.e2e-tests-discovery-pv46p.svc.clusterset.local] Namespace:e2e-tests-discovery-pv46p PodName:netshootqvtld-6757486d56-pnd7x ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
  2656. 2020-10-08T11:44:10.1161969Z STEP: Validating that dig result is not "100.91.181.172"
  2657. 2020-10-08T11:44:10.1162702Z STEP: Setting Nginx deployment replicas to 0 in cluster "cluster1"
  2658. 2020-10-08T11:44:10.1164582Z STEP: Executing "dig +short nginx-demo.e2e-tests-discovery-pv46p.svc.clusterset.local" to verify IP "100.91.181.172" for service "nginx-demo" "is not" discoverable
  2659. 2020-10-08T11:44:10.1168351Z Oct 8 11:44:09.823: INFO: ExecWithOptions {Command:[dig +short nginx-demo.e2e-tests-discovery-pv46p.svc.clusterset.local] Namespace:e2e-tests-discovery-pv46p PodName:netshootqvtld-6757486d56-pnd7x ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
  2660. 2020-10-08T11:44:10.1170956Z STEP: Validating that dig result is not ""
  2661. 2020-10-08T11:44:10.1172947Z STEP: Executing "dig +short nginx-demo.e2e-tests-discovery-pv46p.svc.clusterset.local" to verify IP "100.92.225.68" for service "nginx-demo" "is not" discoverable
  2662. 2020-10-08T11:44:10.1177050Z Oct 8 11:44:09.987: INFO: ExecWithOptions {Command:[dig +short nginx-demo.e2e-tests-discovery-pv46p.svc.clusterset.local] Namespace:e2e-tests-discovery-pv46p PodName:netshootqvtld-6757486d56-pnd7x ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
  2663. 2020-10-08T11:44:10.1179293Z STEP: Validating that dig result is not ""
  2664. 2020-10-08T11:44:10.1180100Z STEP: Deleting namespace "e2e-tests-discovery-pv46p" on cluster "cluster1"
  2665. 2020-10-08T11:44:10.1181112Z STEP: Deleting namespace "e2e-tests-discovery-pv46p" on cluster "cluster2"
  2666. 2020-10-08T11:44:10.1181585Z
  2667. 2020-10-08T11:44:10.1181722Z
  2668. 2020-10-08T11:44:10.1182881Z [discovery] Test Stateful Sets Discovery Across Clusters when a pod tries to resolve a podname from stateful set in a local cluster
  2669. 2020-10-08T11:44:10.1183776Z should resolve the pod IP from the local cluster
  2670. 2020-10-08T11:44:10.1185074Z /go/src/github.com/submariner-io/lighthouse/test/e2e/discovery/statefulsets.go:30
  2671. 2020-10-08T11:44:10.1200660Z STEP: Creating namespace objects with basename "discovery"
  2672. 2020-10-08T11:44:10.1250636Z STEP: Generated namespace "e2e-tests-discovery-wdm99" in cluster "cluster1" to execute the tests in
  2673. 2020-10-08T11:44:10.1251845Z STEP: Creating namespace "e2e-tests-discovery-wdm99" in cluster "cluster2"
  2674. 2020-10-08T11:44:10.1326126Z STEP: Creating an Nginx Stateful Set on on "cluster2"
  2675. 2020-10-08T11:44:15.2002430Z STEP: Creating a Nginx Headless Service on "cluster2"
  2676. 2020-10-08T11:44:15.2098389Z STEP: Creating an Nginx Stateful Set on on "cluster1"
  2677. 2020-10-08T11:44:20.2309352Z STEP: Creating a Nginx Headless Service on "cluster1"
  2678. 2020-10-08T11:44:20.2359467Z STEP: Creating serviceExport nginx-ss.e2e-tests-discovery-wdm99 on "cluster2"
  2679. 2020-10-08T11:44:20.2423828Z STEP: Retrieving ServiceExport nginx-ss.e2e-tests-discovery-wdm99 on "cluster2"
  2680. 2020-10-08T11:44:25.2463382Z STEP: Creating serviceExport nginx-ss.e2e-tests-discovery-wdm99 on "cluster1"
  2681. 2020-10-08T11:44:25.2518886Z STEP: Retrieving ServiceExport nginx-ss.e2e-tests-discovery-wdm99 on "cluster1"
  2682. 2020-10-08T11:44:30.2564789Z STEP: Creating a Netshoot Deployment on "cluster1"
  2683. 2020-10-08T11:44:35.2633214Z STEP: Retrieving EndpointSlices for nginx-ss on "cluster2"
  2684. 2020-10-08T11:44:35.2656036Z STEP: Executing "dig +short web-0.cluster1.nginx-ss.e2e-tests-discovery-wdm99.svc.clusterset.local" to verify IPs [10.241.224.4] for pod "web-0.cluster1.nginx-ss" "are" discoverable
  2685. 2020-10-08T11:44:35.2660687Z Oct 8 11:44:35.264: INFO: ExecWithOptions {Command:[dig +short web-0.cluster1.nginx-ss.e2e-tests-discovery-wdm99.svc.clusterset.local] Namespace:e2e-tests-discovery-wdm99 PodName:netshootjf8wz-6757486d56-82qc9 ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
  2686. 2020-10-08T11:44:35.3411699Z STEP: Validating that dig result are "10.241.224.4"
  2687. 2020-10-08T11:44:35.3414155Z STEP: Executing "dig +short web-0.cluster2.nginx-ss.e2e-tests-discovery-wdm99.svc.clusterset.local" to verify IPs [10.242.192.3] for pod "web-0.cluster2.nginx-ss" "are" discoverable
  2688. 2020-10-08T11:44:35.3417691Z Oct 8 11:44:35.340: INFO: ExecWithOptions {Command:[dig +short web-0.cluster2.nginx-ss.e2e-tests-discovery-wdm99.svc.clusterset.local] Namespace:e2e-tests-discovery-wdm99 PodName:netshootjf8wz-6757486d56-82qc9 ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
  2689. 2020-10-08T11:44:35.4062868Z STEP: Validating that dig result are "10.242.192.3"
  2690. 2020-10-08T11:44:35.4064284Z STEP: Deleting serviceExport nginx-ss.e2e-tests-discovery-wdm99 on "cluster2"
  2691. 2020-10-08T11:44:40.4160447Z STEP: Executing "dig +short web-0.cluster1.nginx-ss.e2e-tests-discovery-wdm99.svc.clusterset.local" to verify IPs [10.241.224.4] for pod "web-0.cluster1.nginx-ss" "are" discoverable
  2692. 2020-10-08T11:44:40.4165078Z Oct 8 11:44:40.415: INFO: ExecWithOptions {Command:[dig +short web-0.cluster1.nginx-ss.e2e-tests-discovery-wdm99.svc.clusterset.local] Namespace:e2e-tests-discovery-wdm99 PodName:netshootjf8wz-6757486d56-82qc9 ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
  2693. 2020-10-08T11:44:40.4822293Z STEP: Validating that dig result are "10.241.224.4"
  2694. 2020-10-08T11:44:40.4824389Z STEP: Executing "dig +short web-0.cluster2.nginx-ss.e2e-tests-discovery-wdm99.svc.clusterset.local" to verify IPs [10.242.192.3] for pod "web-0.cluster2.nginx-ss" "are not" discoverable
  2695. 2020-10-08T11:44:40.4834327Z Oct 8 11:44:40.481: INFO: ExecWithOptions {Command:[dig +short web-0.cluster2.nginx-ss.e2e-tests-discovery-wdm99.svc.clusterset.local] Namespace:e2e-tests-discovery-wdm99 PodName:netshootjf8wz-6757486d56-82qc9 ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
  2696. 2020-10-08T11:44:40.5534909Z STEP: Validating that dig result are not ""
  2697. 2020-10-08T11:44:40.5535927Z STEP: Deleting serviceExport nginx-ss.e2e-tests-discovery-wdm99 on "cluster1"
  2698. 2020-10-08T11:44:45.5667891Z STEP: Retrieving EndpointSlices for nginx-ss on "cluster2"
  2699. 2020-10-08T11:44:45.5684705Z STEP: Deleting namespace "e2e-tests-discovery-wdm99" on cluster "cluster1"
  2700. 2020-10-08T11:44:45.5741289Z STEP: Deleting namespace "e2e-tests-discovery-wdm99" on cluster "cluster2"
  2701. 2020-10-08T11:44:45.5781552Z •
  2702. 2020-10-08T11:44:45.5782341Z ------------------------------
  2703. 2020-10-08T11:44:45.5788634Z STEP: Creating namespace objects with basename "discovery"
  2704. 2020-10-08T11:44:45.5790170Z STEP: Generated namespace "e2e-tests-discovery-wdm99" in cluster "cluster1" to execute the tests in
  2705. 2020-10-08T11:44:45.5791610Z STEP: Creating namespace "e2e-tests-discovery-wdm99" in cluster "cluster2"
  2706. 2020-10-08T11:44:45.5794778Z STEP: Creating an Nginx Stateful Set on on "cluster2"
  2707. 2020-10-08T11:44:45.5795703Z STEP: Creating a Nginx Headless Service on "cluster2"
  2708. 2020-10-08T11:44:45.5796869Z STEP: Creating an Nginx Stateful Set on on "cluster1"
  2709. 2020-10-08T11:44:45.5797751Z STEP: Creating a Nginx Headless Service on "cluster1"
  2710. 2020-10-08T11:44:45.5798850Z STEP: Creating serviceExport nginx-ss.e2e-tests-discovery-wdm99 on "cluster2"
  2711. 2020-10-08T11:44:45.5800219Z STEP: Retrieving ServiceExport nginx-ss.e2e-tests-discovery-wdm99 on "cluster2"
  2712. 2020-10-08T11:44:45.5801815Z STEP: Creating serviceExport nginx-ss.e2e-tests-discovery-wdm99 on "cluster1"
  2713. 2020-10-08T11:44:45.5803258Z STEP: Retrieving ServiceExport nginx-ss.e2e-tests-discovery-wdm99 on "cluster1"
  2714. 2020-10-08T11:44:45.5804364Z STEP: Creating a Netshoot Deployment on "cluster1"
  2715. 2020-10-08T11:44:45.5805221Z STEP: Retrieving EndpointSlices for nginx-ss on "cluster2"
  2716. 2020-10-08T11:44:45.5807540Z STEP: Executing "dig +short web-0.cluster1.nginx-ss.e2e-tests-discovery-wdm99.svc.clusterset.local" to verify IPs [10.241.224.4] for pod "web-0.cluster1.nginx-ss" "are" discoverable
  2717. 2020-10-08T11:44:45.5811953Z Oct 8 11:44:35.264: INFO: ExecWithOptions {Command:[dig +short web-0.cluster1.nginx-ss.e2e-tests-discovery-wdm99.svc.clusterset.local] Namespace:e2e-tests-discovery-wdm99 PodName:netshootjf8wz-6757486d56-82qc9 ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
  2718. 2020-10-08T11:44:45.5814962Z STEP: Validating that dig result are "10.241.224.4"
  2719. 2020-10-08T11:44:45.5817450Z STEP: Executing "dig +short web-0.cluster2.nginx-ss.e2e-tests-discovery-wdm99.svc.clusterset.local" to verify IPs [10.242.192.3] for pod "web-0.cluster2.nginx-ss" "are" discoverable
  2720. 2020-10-08T11:44:45.5821813Z Oct 8 11:44:35.340: INFO: ExecWithOptions {Command:[dig +short web-0.cluster2.nginx-ss.e2e-tests-discovery-wdm99.svc.clusterset.local] Namespace:e2e-tests-discovery-wdm99 PodName:netshootjf8wz-6757486d56-82qc9 ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
  2721. 2020-10-08T11:44:45.5824598Z STEP: Validating that dig result are "10.242.192.3"
  2722. 2020-10-08T11:44:45.5825674Z STEP: Deleting serviceExport nginx-ss.e2e-tests-discovery-wdm99 on "cluster2"
  2723. 2020-10-08T11:44:45.5828217Z STEP: Executing "dig +short web-0.cluster1.nginx-ss.e2e-tests-discovery-wdm99.svc.clusterset.local" to verify IPs [10.241.224.4] for pod "web-0.cluster1.nginx-ss" "are" discoverable
  2724. 2020-10-08T11:44:45.5832622Z Oct 8 11:44:40.415: INFO: ExecWithOptions {Command:[dig +short web-0.cluster1.nginx-ss.e2e-tests-discovery-wdm99.svc.clusterset.local] Namespace:e2e-tests-discovery-wdm99 PodName:netshootjf8wz-6757486d56-82qc9 ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
  2725. 2020-10-08T11:44:45.5835598Z STEP: Validating that dig result are "10.241.224.4"
  2726. 2020-10-08T11:44:45.5837778Z STEP: Executing "dig +short web-0.cluster2.nginx-ss.e2e-tests-discovery-wdm99.svc.clusterset.local" to verify IPs [10.242.192.3] for pod "web-0.cluster2.nginx-ss" "are not" discoverable
  2727. 2020-10-08T11:44:45.5842204Z Oct 8 11:44:40.481: INFO: ExecWithOptions {Command:[dig +short web-0.cluster2.nginx-ss.e2e-tests-discovery-wdm99.svc.clusterset.local] Namespace:e2e-tests-discovery-wdm99 PodName:netshootjf8wz-6757486d56-82qc9 ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
  2728. 2020-10-08T11:44:45.6004408Z STEP: Validating that dig result are not ""
  2729. 2020-10-08T11:44:45.6005759Z STEP: Deleting serviceExport nginx-ss.e2e-tests-discovery-wdm99 on "cluster1"
  2730. 2020-10-08T11:44:45.6006830Z STEP: Retrieving EndpointSlices for nginx-ss on "cluster2"
  2731. 2020-10-08T11:44:45.6007742Z STEP: Deleting namespace "e2e-tests-discovery-wdm99" on cluster "cluster1"
  2732. 2020-10-08T11:44:45.6008741Z STEP: Deleting namespace "e2e-tests-discovery-wdm99" on cluster "cluster2"
  2733. 2020-10-08T11:44:45.6009201Z
  2734. 2020-10-08T11:44:45.6009352Z
  2735. 2020-10-08T11:44:45.6010771Z JUnit path was configured: /go/src/github.com/submariner-io/lighthouse/output/e2e-junit.xml
  2736. 2020-10-08T11:44:45.6065966Z
  2737. 2020-10-08T11:44:45.6067972Z JUnit report was created: /go/src/github.com/submariner-io/lighthouse/output/e2e-junit.xml
  2738. 2020-10-08T11:44:45.6068493Z
  2739. 2020-10-08T11:44:45.6068643Z
  2740. 2020-10-08T11:44:45.6069151Z Summarizing 1 Failure:
  2741. 2020-10-08T11:44:45.6069403Z
  2742. 2020-10-08T11:44:45.6070933Z [Fail] [discovery] Test Headless Service Discovery Across Clusters when a pod tries to resolve a headless service which is exported locally and in a remote cluster [It] should resolve the backing pod IPs from both clusters 
  2743. 2020-10-08T11:44:45.6072611Z /go/src/github.com/submariner-io/lighthouse/vendor/github.com/submariner-io/shipyard/test/e2e/framework/framework.go:371
  2744. 2020-10-08T11:44:45.6073254Z
  2745. 2020-10-08T11:44:45.6073714Z Ran 11 of 11 Specs in 484.128 seconds
  2746. 2020-10-08T11:44:45.6074411Z FAIL! -- 10 Passed | 1 Failed | 0 Pending | 0 Skipped
  2747. 2020-10-08T11:44:45.6074988Z --- FAIL: TestE2E (484.14s)
  2748. 2020-10-08T11:44:45.6075246Z FAIL
  2749. 2020-10-08T11:44:45.6075500Z exit status 1
  2750. 2020-10-08T11:44:45.6076118Z FAIL github.com/submariner-io/lighthouse/test/e2e 484.150s
  2751. 2020-10-08T11:44:46.5629032Z make: *** [/opt/shipyard/Makefile.inc:40: e2e] Error 1
  2752. 2020-10-08T11:44:46.5631626Z [lighthouse]$ make e2e -- using=helm
  2753. 2020-10-08T11:44:53.7359407Z time="2020-10-08T11:44:53Z" level=fatal msg="exit status 2"
  2754. 2020-10-08T11:44:53.7369331Z make: *** [e2e] Error 1
  2755. 2020-10-08T11:44:53.7370206Z Makefile.dapper:14: recipe for target 'e2e' failed
  2756. 2020-10-08T11:44:53.7409633Z ##[error]Process completed with exit code 2.
  2757. 2020-10-08T11:44:53.7540319Z ##[group]Run df -h
  2758. 2020-10-08T11:44:53.7540779Z df -h
  2759. 2020-10-08T11:44:53.7541017Z free -h
  2760. 2020-10-08T11:44:53.7541297Z make post-mortem
  2761. 2020-10-08T11:44:53.8191294Z shell: /bin/bash -e {0}
  2762. 2020-10-08T11:44:53.8191608Z ##[endgroup]
  2763. 2020-10-08T11:44:53.8293817Z Filesystem Size Used Avail Use% Mounted on
  2764. 2020-10-08T11:44:53.8294216Z udev 3.4G 0 3.4G 0% /dev
  2765. 2020-10-08T11:44:53.8294504Z tmpfs 696M 1.3M 695M 1% /run
  2766. 2020-10-08T11:44:53.8294774Z /dev/sdb1 84G 75G 8.9G 90% /
  2767. 2020-10-08T11:44:53.8295072Z tmpfs 3.4G 225M 3.2G 7% /dev/shm
  2768. 2020-10-08T11:44:53.8295482Z tmpfs 5.0M 0 5.0M 0% /run/lock
  2769. 2020-10-08T11:44:53.8295781Z tmpfs 3.4G 0 3.4G 0% /sys/fs/cgroup
  2770. 2020-10-08T11:44:53.8296091Z /dev/sdb15 105M 3.6M 101M 4% /boot/efi
  2771. 2020-10-08T11:44:53.8296381Z /dev/sda1 14G 4.1G 9.0G 32% /mnt
  2772. 2020-10-08T11:44:53.8335243Z total used free shared buff/cache available
  2773. 2020-10-08T11:44:53.8335670Z Mem: 6.8G 3.1G 319M 302M 3.4G 3.4G
  2774. 2020-10-08T11:44:53.8335962Z Swap: 0B 0B 0B
  2775. 2020-10-08T11:44:53.8358933Z ./.dapper -m bind make post-mortem
  2776. 2020-10-08T11:44:55.9269052Z Sending build context to Docker daemon 72.08MB
  2777. 2020-10-08T11:44:55.9269729Z
  2778. 2020-10-08T11:44:55.9353462Z Step 1/6 : FROM quay.io/submariner/shipyard-dapper-base:0.7.1
  2779. 2020-10-08T11:44:55.9359679Z ---> c50512fb3462
  2780. 2020-10-08T11:44:55.9360897Z Step 2/6 : ENV DAPPER_ENV="REPO TAG QUAY_USERNAME QUAY_PASSWORD GITHUB_SHA BUILD_ARGS CLUSTERS_ARGS DEPLOY_ARGS RELEASE_ARGS" DAPPER_SOURCE=/go/src/github.com/submariner-io/lighthouse DAPPER_DOCKER_SOCKET=true
  2781. 2020-10-08T11:44:55.9377698Z ---> Using cache
  2782. 2020-10-08T11:44:55.9378209Z ---> c747a35a2763
  2783. 2020-10-08T11:44:55.9378602Z Step 3/6 : ENV DAPPER_OUTPUT=${DAPPER_SOURCE}/output
  2784. 2020-10-08T11:44:55.9386853Z ---> Using cache
  2785. 2020-10-08T11:44:55.9387367Z ---> e1395f25bc2c
  2786. 2020-10-08T11:44:55.9387706Z Step 4/6 : WORKDIR ${DAPPER_SOURCE}
  2787. 2020-10-08T11:44:55.9397699Z ---> Using cache
  2788. 2020-10-08T11:44:55.9398219Z ---> b92d15285148
  2789. 2020-10-08T11:44:55.9398600Z Step 5/6 : ENTRYPOINT ["/opt/shipyard/scripts/entry"]
  2790. 2020-10-08T11:44:55.9408086Z ---> Using cache
  2791. 2020-10-08T11:44:55.9408631Z ---> e388b749e663
  2792. 2020-10-08T11:44:55.9410350Z Step 6/6 : CMD ["sh"]
  2793. 2020-10-08T11:44:55.9420896Z ---> Using cache
  2794. 2020-10-08T11:44:55.9421363Z ---> 5cec2e11bcff
  2795. 2020-10-08T11:44:56.0339005Z Successfully built 5cec2e11bcff
  2796. 2020-10-08T11:44:56.0377758Z Successfully tagged lighthouse:HEAD
  2797. 2020-10-08T11:44:56.6360084Z [lighthouse]$ trap chown -R 1001:116 . exit
  2798. 2020-10-08T11:44:56.6366753Z [lighthouse]$ mkdir -p bin dist output
  2799. 2020-10-08T11:44:56.6427570Z [lighthouse]$ make post-mortem
  2800. 2020-10-08T11:44:56.6945984Z fatal: No names found, cannot describe anything.
  2801. 2020-10-08T11:44:56.7040517Z Makefile:41: warning: overriding recipe for target 'deploy'
  2802. 2020-10-08T11:44:56.7044688Z /opt/shipyard/Makefile.inc:36: warning: ignoring old recipe for target 'deploy'
  2803. 2020-10-08T11:44:56.7065271Z /opt/shipyard/scripts/post_mortem.sh
  2804. 2020-10-08T11:44:56.7123015Z [lighthouse]$ source /opt/shipyard/scripts/lib/utils
  2805. 2020-10-08T11:44:56.7130930Z [lighthouse]$ . /opt/shipyard/scripts/lib/source_only
  2806. 2020-10-08T11:44:56.7169493Z [lighthouse]$ script_name=utils
  2807. 2020-10-08T11:44:56.7177557Z [lighthouse]$ exec_name=post_mortem.sh
  2808. 2020-10-08T11:44:56.7197233Z [lighthouse]$ declare_kubeconfig
  2809. 2020-10-08T11:44:56.7205589Z [lighthouse]$ declare_kubeconfig
  2810. 2020-10-08T11:44:56.7251285Z [lighthouse]$ source /opt/shipyard/scripts/lib/kubecfg
  2811. 2020-10-08T11:44:56.7259945Z [lighthouse]$ export KUBECONFIG
  2812. 2020-10-08T11:44:56.7336812Z [lighthouse]$ KUBECONFIG=/go/src/github.com/submariner-io/lighthouse/output/kubeconfigs/kind-config-cluster2:/go/src/github.com/submariner-io/lighthouse/output/kubeconfigs/kind-config-cluster1:
  2813. 2020-10-08T11:44:56.7382272Z [lighthouse]$ find /go/src/github.com/submariner-io/lighthouse/output/kubeconfigs -type f -printf %p:
  2814. 2020-10-08T11:44:56.7414553Z [lighthouse]$ kind get clusters
  2815. 2020-10-08T11:44:57.1962077Z [lighthouse]$ run_sequential cluster1 cluster2 post_analyze
  2816. 2020-10-08T11:44:57.1974711Z [lighthouse]$ run_sequential cluster1 cluster2 post_analyze
  2817. 2020-10-08T11:44:57.1983474Z [lighthouse]$ local cmnd=post_analyze
  2818. 2020-10-08T11:44:57.1997665Z [lighthouse]$ eval echo cluster1 cluster2
  2819. 2020-10-08T11:44:57.2016364Z [lighthouse]$ [cluster1] post_analyze
  2820. 2020-10-08T11:44:57.2030592Z [lighthouse]$ [cluster1] sed s/^/[cluster1] /
  2821. 2020-10-08T11:44:57.2042800Z [lighthouse]$ [cluster1] post_analyze
  2822. 2020-10-08T11:44:57.2055032Z [lighthouse]$ [cluster1] kubectl get all --all-namespaces
  2823. 2020-10-08T11:44:57.2067852Z [lighthouse]$ [cluster1] kubectl get all --all-namespaces
  2824. 2020-10-08T11:44:57.2088413Z [lighthouse]$ [cluster1] command kubectl --context=cluster1 get all --all-namespaces
  2825. 2020-10-08T11:44:57.2099524Z [lighthouse]$ [cluster1] kubectl --context=cluster1 get all --all-namespaces
  2826. 2020-10-08T11:44:58.0632514Z
  2827. 2020-10-08T11:44:58.0635242Z
  2828. 2020-10-08T11:44:58.0639268Z
  2829. 2020-10-08T11:44:58.0661780Z
  2830. 2020-10-08T11:44:58.0662465Z [cluster1] ======================= Post mortem cluster1 =======================
  2831. 2020-10-08T11:44:58.0662847Z
  2832. 2020-10-08T11:44:58.0663263Z [cluster1] NAMESPACE NAME READY STATUS RESTARTS AGE
  2833. 2020-10-08T11:44:58.0664917Z [cluster1] e2e-tests-discovery-wdm99 pod/netshootjf8wz-6757486d56-82qc9 1/1 Terminating 0 28s
  2834. 2020-10-08T11:44:58.0666225Z [cluster1] kube-system pod/coredns-6955765f44-mfhxx 1/1 Running 0 12m
  2835. 2020-10-08T11:44:58.0667201Z [cluster1] kube-system pod/coredns-6955765f44-zkh6f 1/1 Running 0 12m
  2836. 2020-10-08T11:44:58.0668223Z [cluster1] kube-system pod/etcd-cluster1-control-plane 1/1 Running 0 13m
  2837. 2020-10-08T11:44:58.0670849Z [cluster1] kube-system pod/kube-apiserver-cluster1-control-plane 1/1 Running 0 13m
  2838. 2020-10-08T11:44:58.0672483Z [cluster1] kube-system pod/kube-controller-manager-cluster1-control-plane 1/1 Running 0 13m
  2839. 2020-10-08T11:44:58.0673799Z [cluster1] kube-system pod/kube-proxy-br57q 1/1 Running 0 12m
  2840. 2020-10-08T11:44:58.0674658Z [cluster1] kube-system pod/kube-proxy-fdlz2 1/1 Running 0 12m
  2841. 2020-10-08T11:44:58.0675753Z [cluster1] kube-system pod/kube-proxy-n9vx9 1/1 Running 0 12m
  2842. 2020-10-08T11:44:58.0677256Z [cluster1] kube-system pod/kube-scheduler-cluster1-control-plane 1/1 Running 0 13m
  2843. 2020-10-08T11:44:58.0680468Z [cluster1] kube-system pod/tiller-deploy-8488d98b4c-kqmmd 1/1 Running 0 10m
  2844. 2020-10-08T11:44:58.0681566Z [cluster1] kube-system pod/weave-net-dxhs2 2/2 Running 0 12m
  2845. 2020-10-08T11:44:58.0682476Z [cluster1] kube-system pod/weave-net-m9c7v 2/2 Running 0 12m
  2846. 2020-10-08T11:44:58.0683378Z [cluster1] kube-system pod/weave-net-r7gjb 2/2 Running 0 12m
  2847. 2020-10-08T11:44:58.0684644Z [cluster1] local-path-storage pod/local-path-provisioner-7745554f7f-vddwf 1/1 Running 0 12m
  2848. 2020-10-08T11:44:58.0686351Z [cluster1] submariner-operator pod/submariner-gateway-jgnv7 1/1 Running 0 10m
  2849. 2020-10-08T11:44:58.0688075Z [cluster1] submariner-operator pod/submariner-lighthouse-agent-5fdd5968d8-gld9p 1/1 Running 0 10m
  2850. 2020-10-08T11:44:58.0689954Z [cluster1] submariner-operator pod/submariner-lighthouse-coredns-7466b6679c-w7pcw 1/1 Running 0 10m
  2851. 2020-10-08T11:44:58.0696299Z [cluster1] submariner-operator pod/submariner-lighthouse-coredns-7466b6679c-xhdm4 1/1 Running 0 10m
  2852. 2020-10-08T11:44:58.0698642Z [cluster1] submariner-operator pod/submariner-routeagent-cljq7 1/1 Running 0 10m
  2853. 2020-10-08T11:44:58.0699967Z [cluster1] submariner-operator pod/submariner-routeagent-wmmbs 1/1 Running 0 10m
  2854. 2020-10-08T11:44:58.0701351Z [cluster1] NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
  2855. 2020-10-08T11:44:58.0702240Z [cluster1] default service/kubernetes ClusterIP 100.91.0.1 <none> 443/TCP 13m
  2856. 2020-10-08T11:44:58.0704653Z [cluster1] kube-system service/kube-dns ClusterIP 100.91.0.10 <none> 53/UDP,53/TCP,9153/TCP 13m
  2857. 2020-10-08T11:44:58.0705866Z [cluster1] kube-system service/tiller-deploy ClusterIP 100.91.195.222 <none> 44134/TCP 10m
  2858. 2020-10-08T11:44:58.0707044Z [cluster1] submariner-operator service/submariner-lighthouse-coredns ClusterIP 100.91.206.45 <none> 53/UDP 10m
  2859. 2020-10-08T11:44:58.0708143Z [cluster1] NAMESPACE NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
  2860. 2020-10-08T11:44:58.0709357Z [cluster1] kube-system daemonset.apps/kube-proxy 3 3 3 3 3 beta.kubernetes.io/os=linux 13m
  2861. 2020-10-08T11:44:58.0792924Z
  2862. 2020-10-08T11:44:58.0793197Z
  2863. 2020-10-08T11:44:58.0793350Z
  2864. 2020-10-08T11:44:58.0793513Z
  2865. 2020-10-08T11:44:58.0816445Z [lighthouse]$ [cluster1] kubectl get pods -A
  2866. 2020-10-08T11:44:58.0832143Z [lighthouse]$ [cluster1] tail -n +2
  2867. 2020-10-08T11:44:58.0846096Z [lighthouse]$ [cluster1] grep -v Running
  2868. 2020-10-08T11:44:58.0857518Z [lighthouse]$ [cluster1] kubectl get pods -A
  2869. 2020-10-08T11:44:58.0864283Z [lighthouse]$ [cluster1] sed s/ */;/g
  2870. 2020-10-08T11:44:58.0886467Z [lighthouse]$ [cluster1] command kubectl --context=cluster1 get pods -A
  2871. 2020-10-08T11:44:58.0896936Z [lighthouse]$ [cluster1] kubectl --context=cluster1 get pods -A
  2872. 2020-10-08T11:44:58.8782789Z [lighthouse]$ [cluster1] ns=e2e-tests-discovery-wdm99
  2873. 2020-10-08T11:44:58.8800309Z [lighthouse]$ [cluster1] cut -f1 -d;
  2874. 2020-10-08T11:44:58.8844283Z [lighthouse]$ [cluster1] name=netshootjf8wz-6757486d56-82qc9
  2875. 2020-10-08T11:44:58.8863210Z [lighthouse]$ [cluster1] cut -f2 -d;
  2876. 2020-10-08T11:44:58.8885670Z [lighthouse]$ [cluster1] kubectl -n e2e-tests-discovery-wdm99 describe pod netshootjf8wz-6757486d56-82qc9
  2877. 2020-10-08T11:44:58.8896265Z [lighthouse]$ [cluster1] kubectl -n e2e-tests-discovery-wdm99 describe pod netshootjf8wz-6757486d56-82qc9
  2878. 2020-10-08T11:44:58.8907326Z [lighthouse]$ [cluster1] command kubectl --context=cluster1 -n e2e-tests-discovery-wdm99 describe pod netshootjf8wz-6757486d56-82qc9
  2879. 2020-10-08T11:44:58.8919225Z [lighthouse]$ [cluster1] kubectl --context=cluster1 -n e2e-tests-discovery-wdm99 describe pod netshootjf8wz-6757486d56-82qc9
  2880. 2020-10-08T11:44:59.6480224Z [cluster1] kube-system daemonset.apps/weave-net 3 3 3 3 3 <none> 12m
  2881. 2020-10-08T11:44:59.6481897Z [cluster1] submariner-operator daemonset.apps/submariner-gateway 1 1 1 1 1 submariner.io/gateway=true 10m
  2882. 2020-10-08T11:44:59.6483248Z [cluster1] submariner-operator daemonset.apps/submariner-routeagent 2 2 2 2 2 <none> 10m
  2883. 2020-10-08T11:44:59.6484273Z [cluster1] NAMESPACE NAME READY UP-TO-DATE AVAILABLE AGE
  2884. 2020-10-08T11:44:59.6485106Z [cluster1] kube-system deployment.apps/coredns 2/2 2 2 13m
  2885. 2020-10-08T11:44:59.6485978Z [cluster1] kube-system deployment.apps/tiller-deploy 1/1 1 1 10m
  2886. 2020-10-08T11:44:59.6487021Z [cluster1] local-path-storage deployment.apps/local-path-provisioner 1/1 1 1 13m
  2887. 2020-10-08T11:44:59.6488288Z [cluster1] submariner-operator deployment.apps/submariner-lighthouse-agent 1/1 1 1 10m
  2888. 2020-10-08T11:44:59.6490261Z [cluster1] submariner-operator deployment.apps/submariner-lighthouse-coredns 2/2 2 2 10m
  2889. 2020-10-08T11:44:59.6491219Z [cluster1] NAMESPACE NAME DESIRED CURRENT READY AGE
  2890. 2020-10-08T11:44:59.6492158Z [cluster1] kube-system replicaset.apps/coredns-6955765f44 2 2 2 12m
  2891. 2020-10-08T11:44:59.6493831Z [cluster1] kube-system replicaset.apps/tiller-deploy-8488d98b4c 1 1 1 10m
  2892. 2020-10-08T11:44:59.6495192Z [cluster1] local-path-storage replicaset.apps/local-path-provisioner-7745554f7f 1 1 1 12m
  2893. 2020-10-08T11:44:59.6496900Z [cluster1] submariner-operator replicaset.apps/submariner-lighthouse-agent-5fdd5968d8 1 1 1 10m
  2894. 2020-10-08T11:44:59.6498813Z [cluster1] submariner-operator replicaset.apps/submariner-lighthouse-coredns-7466b6679c 2 2 2 10m
  2895. 2020-10-08T11:44:59.6500470Z [cluster1] ======================= netshootjf8wz-6757486d56-82qc9 - e2e-tests-discovery-wdm99 ============================
  2896. 2020-10-08T11:44:59.6501621Z [cluster1] Name: netshootjf8wz-6757486d56-82qc9
  2897. 2020-10-08T11:44:59.6502355Z [cluster1] Namespace: e2e-tests-discovery-wdm99
  2898. 2020-10-08T11:44:59.6502798Z [cluster1] Priority: 0
  2899. 2020-10-08T11:44:59.6503294Z [cluster1] Node: cluster1-worker2/172.17.0.7
  2900. 2020-10-08T11:44:59.6503702Z [cluster1] Start Time: Thu, 08 Oct 2020 11:44:30 +0000
  2901. 2020-10-08T11:44:59.6504038Z [cluster1] Labels: app=netshoot
  2902. 2020-10-08T11:44:59.6504544Z [cluster1] pod-template-hash=6757486d56
  2903. 2020-10-08T11:44:59.6504925Z [cluster1] Annotations: <none>
  2904. 2020-10-08T11:44:59.6505316Z [cluster1] Status: Terminating (lasts <invalid>)
  2905. 2020-10-08T11:44:59.6505735Z [cluster1] Termination Grace Period: 30s
  2906. 2020-10-08T11:44:59.6506053Z [cluster1] IP: 10.241.128.4
  2907. 2020-10-08T11:44:59.6506656Z [cluster1] Controlled By: ReplicaSet/netshootjf8wz-6757486d56
  2908. 2020-10-08T11:44:59.6507139Z [cluster1] Containers:
  2909. 2020-10-08T11:44:59.6507427Z [cluster1] netshoot:
  2910. 2020-10-08T11:44:59.6508703Z [cluster1] Container ID: containerd://2a0220cf2b1b929835e23cf4b577d70226ed3875f472ffd18eecd5268fe09481
  2911. 2020-10-08T11:44:59.6510269Z [cluster1] Image: quay.io/submariner/nettest
  2912. 2020-10-08T11:44:59.6511264Z [cluster1] Image ID: quay.io/submariner/nettest@sha256:63f21b9d8065d60870c2bca980dbc4dc6358edf669661576b8cbee39e3024a2d
  2913. 2020-10-08T11:44:59.6512170Z [cluster1] Port: <none>
  2914. 2020-10-08T11:44:59.6512670Z [cluster1] Host Port: <none>
  2915. 2020-10-08T11:44:59.6513015Z [cluster1] Command:
  2916. 2020-10-08T11:44:59.6513334Z [cluster1] sleep
  2917. 2020-10-08T11:44:59.6513628Z [cluster1] 600
  2918. 2020-10-08T11:44:59.6514050Z [cluster1] State: Running
  2919. 2020-10-08T11:44:59.6514427Z [cluster1] Started: Thu, 08 Oct 2020 11:44:30 +0000
  2920. 2020-10-08T11:44:59.6514797Z [cluster1] Ready: True
  2921. 2020-10-08T11:44:59.6515153Z [cluster1] Restart Count: 0
  2922. 2020-10-08T11:44:59.6515545Z [cluster1] Environment: <none>
  2923. 2020-10-08T11:44:59.6515911Z [cluster1] Mounts:
  2924. 2020-10-08T11:44:59.6516777Z [cluster1] /var/run/secrets/kubernetes.io/serviceaccount from default-token-22ghh (ro)
  2925. 2020-10-08T11:44:59.6517444Z [cluster1] Conditions:
  2926. 2020-10-08T11:44:59.6517786Z [cluster1] Type Status
  2927. 2020-10-08T11:44:59.6518169Z [cluster1] Initialized True
  2928. 2020-10-08T11:44:59.6518549Z [cluster1] Ready True
  2929. 2020-10-08T11:44:59.6518964Z [cluster1] ContainersReady True
  2930. 2020-10-08T11:44:59.6519422Z [cluster1] PodScheduled True
  2931. 2020-10-08T11:44:59.6519801Z [cluster1] Volumes:
  2932. 2020-10-08T11:44:59.6520466Z [cluster1] default-token-22ghh:
  2933. 2020-10-08T11:44:59.6521007Z [cluster1] Type: Secret (a volume populated by a Secret)
  2934. 2020-10-08T11:44:59.6521734Z [cluster1] SecretName: default-token-22ghh
  2935. 2020-10-08T11:44:59.6522225Z [cluster1] Optional: false
  2936. 2020-10-08T11:44:59.6522628Z [cluster1] QoS Class: BestEffort
  2937. 2020-10-08T11:44:59.6523208Z [cluster1] Node-Selectors: <none>
  2938. 2020-10-08T11:44:59.6524103Z [cluster1] Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
  2939. 2020-10-08T11:44:59.6525026Z [cluster1] node.kubernetes.io/unreachable:NoExecute for 300s
  2940. 2020-10-08T11:44:59.6565258Z [lighthouse]$ [cluster1] kubectl -n e2e-tests-discovery-wdm99 logs netshootjf8wz-6757486d56-82qc9
  2941. 2020-10-08T11:44:59.6577452Z [lighthouse]$ [cluster1] kubectl -n e2e-tests-discovery-wdm99 logs netshootjf8wz-6757486d56-82qc9
  2942. 2020-10-08T11:44:59.6583538Z [lighthouse]$ [cluster1] command kubectl --context=cluster1 -n e2e-tests-discovery-wdm99 logs netshootjf8wz-6757486d56-82qc9
  2943. 2020-10-08T11:44:59.6594210Z [lighthouse]$ [cluster1] kubectl --context=cluster1 -n e2e-tests-discovery-wdm99 logs netshootjf8wz-6757486d56-82qc9
  2944. 2020-10-08T11:45:00.4288387Z [lighthouse]$ [cluster1] namespace=kube-system
  2945. 2020-10-08T11:45:00.4303417Z [lighthouse]$ [cluster1] kubectl get pods --selector=k8s-app=kube-proxy -n kube-system -o jsonpath={.items[*].metadata.name}
  2946. 2020-10-08T11:45:00.4315795Z [lighthouse]$ [cluster1] kubectl get pods --selector=k8s-app=kube-proxy -n kube-system -o jsonpath={.items[*].metadata.name}
  2947. 2020-10-08T11:45:00.4327412Z [lighthouse]$ [cluster1] command kubectl --context=cluster1 get pods --selector=k8s-app=kube-proxy -n kube-system -o jsonpath={.items[*].metadata.name}
  2948. 2020-10-08T11:45:00.4341672Z [lighthouse]$ [cluster1] kubectl --context=cluster1 get pods --selector=k8s-app=kube-proxy -n kube-system -o jsonpath={.items[*].metadata.name}
  2949. 2020-10-08T11:45:01.2053813Z [lighthouse]$ [cluster1] kubectl -n kube-system logs kube-proxy-br57q
  2950. 2020-10-08T11:45:01.2065415Z [lighthouse]$ [cluster1] kubectl -n kube-system logs kube-proxy-br57q
  2951. 2020-10-08T11:45:01.2077676Z [lighthouse]$ [cluster1] command kubectl --context=cluster1 -n kube-system logs kube-proxy-br57q
  2952. 2020-10-08T11:45:01.2089130Z [lighthouse]$ [cluster1] kubectl --context=cluster1 -n kube-system logs kube-proxy-br57q
  2953. 2020-10-08T11:45:01.9850897Z [lighthouse]$ [cluster1] kubectl -n kube-system logs kube-proxy-fdlz2
  2954. 2020-10-08T11:45:01.9861977Z [lighthouse]$ [cluster1] kubectl -n kube-system logs kube-proxy-fdlz2
  2955. 2020-10-08T11:45:01.9877704Z [lighthouse]$ [cluster1] command kubectl --context=cluster1 -n kube-system logs kube-proxy-fdlz2
  2956. 2020-10-08T11:45:01.9891644Z [lighthouse]$ [cluster1] kubectl --context=cluster1 -n kube-system logs kube-proxy-fdlz2
  2957. 2020-10-08T11:45:02.7498426Z [lighthouse]$ [cluster1] kubectl -n kube-system logs kube-proxy-n9vx9
  2958. 2020-10-08T11:45:02.7508729Z [lighthouse]$ [cluster1] kubectl -n kube-system logs kube-proxy-n9vx9
  2959. 2020-10-08T11:45:02.7523112Z [lighthouse]$ [cluster1] command kubectl --context=cluster1 -n kube-system logs kube-proxy-n9vx9
  2960. 2020-10-08T11:45:02.7535004Z [lighthouse]$ [cluster1] kubectl --context=cluster1 -n kube-system logs kube-proxy-n9vx9
  2961. 2020-10-08T11:45:03.5013931Z [cluster1] Events: <none>
  2962. 2020-10-08T11:45:03.5015547Z [cluster1] ===================== END netshootjf8wz-6757486d56-82qc9 - e2e-tests-discovery-wdm99 ==========================
  2963. 2020-10-08T11:45:03.5018486Z [cluster1] +++++++++++++++++++++: Logs for Pod kube-proxy-br57q in namespace kube-system :++++++++++++++++++++++
  2964. 2020-10-08T11:45:03.5019195Z [cluster1] W1008 11:32:04.065213 1 server_others.go:323] Unknown proxy mode "", assuming iptables proxy
  2965. 2020-10-08T11:45:03.5019822Z [cluster1] I1008 11:32:04.083591 1 node.go:135] Successfully retrieved node IP: 172.17.0.8
  2966. 2020-10-08T11:45:03.5020693Z [cluster1] I1008 11:32:04.083884 1 server_others.go:145] Using iptables Proxier.
  2967. 2020-10-08T11:45:03.5021272Z [cluster1] I1008 11:32:04.085866 1 server.go:571] Version: v1.17.0
  2968. 2020-10-08T11:45:03.5021979Z [cluster1] I1008 11:32:04.086359 1 conntrack.go:100] Set sysctl 'net/netfilter/nf_conntrack_max' to 131072
  2969. 2020-10-08T11:45:03.5022528Z [cluster1] I1008 11:32:04.086415 1 conntrack.go:52] Setting nf_conntrack_max to 131072
  2970. 2020-10-08T11:45:03.5023346Z [cluster1] I1008 11:32:04.089676 1 conntrack.go:100] Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_established' to 86400
  2971. 2020-10-08T11:45:03.5024226Z [cluster1] I1008 11:32:04.089767 1 conntrack.go:100] Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_close_wait' to 3600
  2972. 2020-10-08T11:45:03.5024828Z [cluster1] I1008 11:32:04.116475 1 config.go:313] Starting service config controller
  2973. 2020-10-08T11:45:03.5025384Z [cluster1] I1008 11:32:04.116506 1 shared_informer.go:197] Waiting for caches to sync for service config
  2974. 2020-10-08T11:45:03.5025952Z [cluster1] I1008 11:32:04.119048 1 config.go:131] Starting endpoints config controller
  2975. 2020-10-08T11:45:03.5027177Z [cluster1] I1008 11:32:04.119085 1 shared_informer.go:197] Waiting for caches to sync for endpoints config
  2976. 2020-10-08T11:45:03.5027772Z [cluster1] I1008 11:32:04.232825 1 shared_informer.go:204] Caches are synced for endpoints config
  2977. 2020-10-08T11:45:03.5028336Z [cluster1] I1008 11:32:04.232911 1 shared_informer.go:204] Caches are synced for service config
  2978. 2020-10-08T11:45:03.5029146Z [cluster1] +++++++++++++++++++++: Logs for Pod kube-proxy-fdlz2 in namespace kube-system :++++++++++++++++++++++
  2979. 2020-10-08T11:45:03.5029740Z [cluster1] W1008 11:32:25.304058 1 server_others.go:323] Unknown proxy mode "", assuming iptables proxy
  2980. 2020-10-08T11:45:03.5030448Z [cluster1] I1008 11:32:25.316166 1 node.go:135] Successfully retrieved node IP: 172.17.0.7
  2981. 2020-10-08T11:45:03.5031014Z [cluster1] I1008 11:32:25.316197 1 server_others.go:145] Using iptables Proxier.
  2982. 2020-10-08T11:45:03.5031525Z [cluster1] I1008 11:32:25.316472 1 server.go:571] Version: v1.17.0
  2983. 2020-10-08T11:45:03.5032120Z [cluster1] I1008 11:32:25.316882 1 conntrack.go:52] Setting nf_conntrack_max to 131072
  2984. 2020-10-08T11:45:03.5032934Z [cluster1] I1008 11:32:25.317290 1 conntrack.go:100] Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_established' to 86400
  2985. 2020-10-08T11:45:03.5033822Z [cluster1] I1008 11:32:25.317364 1 conntrack.go:100] Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_close_wait' to 3600
  2986. 2020-10-08T11:45:03.5034668Z [cluster1] I1008 11:32:25.319323 1 config.go:313] Starting service config controller
  2987. 2020-10-08T11:45:03.5035305Z [cluster1] I1008 11:32:25.319339 1 shared_informer.go:197] Waiting for caches to sync for service config
  2988. 2020-10-08T11:45:03.5035942Z [cluster1] I1008 11:32:25.320925 1 config.go:131] Starting endpoints config controller
  2989. 2020-10-08T11:45:03.5038351Z [cluster1] I1008 11:32:25.320971 1 shared_informer.go:197] Waiting for caches to sync for endpoints config
  2990. 2020-10-08T11:45:03.5039081Z [cluster1] I1008 11:32:25.419532 1 shared_informer.go:204] Caches are synced for service config
  2991. 2020-10-08T11:45:03.5039780Z [cluster1] I1008 11:32:25.424850 1 shared_informer.go:204] Caches are synced for endpoints config
  2992. 2020-10-08T11:45:03.5040818Z [cluster1] +++++++++++++++++++++: Logs for Pod kube-proxy-n9vx9 in namespace kube-system :++++++++++++++++++++++
  2993. 2020-10-08T11:45:03.5041549Z [cluster1] W1008 11:32:26.470248 1 server_others.go:323] Unknown proxy mode "", assuming iptables proxy
  2994. 2020-10-08T11:45:03.5042217Z [cluster1] I1008 11:32:26.487546 1 node.go:135] Successfully retrieved node IP: 172.17.0.9
  2995. 2020-10-08T11:45:03.5043016Z [cluster1] I1008 11:32:26.488184 1 server_others.go:145] Using iptables Proxier.
  2996. 2020-10-08T11:45:03.5043545Z [cluster1] I1008 11:32:26.505389 1 server.go:571] Version: v1.17.0
  2997. 2020-10-08T11:45:03.5044096Z [cluster1] I1008 11:32:26.505768 1 conntrack.go:52] Setting nf_conntrack_max to 131072
  2998. 2020-10-08T11:45:03.5045284Z [cluster1] I1008 11:32:26.505883 1 conntrack.go:100] Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_established' to 86400
  2999. 2020-10-08T11:45:03.5047614Z [cluster1] I1008 11:32:26.505945 1 conntrack.go:100] Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_close_wait' to 3600
  3000. 2020-10-08T11:45:03.5048423Z [cluster1] I1008 11:32:26.507854 1 config.go:313] Starting service config controller
  3001. 2020-10-08T11:45:03.5148748Z [lighthouse]$ [cluster1] namespace=submariner-operator
  3002. 2020-10-08T11:45:03.5172125Z [lighthouse]$ [cluster1] kubectl get pods --selector=name=submariner-operator -n submariner-operator -o jsonpath={.items[*].metadata.name}
  3003. 2020-10-08T11:45:03.5187827Z [lighthouse]$ [cluster1] kubectl get pods --selector=name=submariner-operator -n submariner-operator -o jsonpath={.items[*].metadata.name}
  3004. 2020-10-08T11:45:03.5206617Z [lighthouse]$ [cluster1] command kubectl --context=cluster1 get pods --selector=name=submariner-operator -n submariner-operator -o jsonpath={.items[*].metadata.name}
  3005. 2020-10-08T11:45:03.5221486Z [lighthouse]$ [cluster1] kubectl --context=cluster1 get pods --selector=name=submariner-operator -n submariner-operator -o jsonpath={.items[*].metadata.name}
  3006. 2020-10-08T11:45:04.2556826Z [lighthouse]$ [cluster1] namespace=submariner-operator
  3007. 2020-10-08T11:45:04.2569957Z [lighthouse]$ [cluster1] kubectl get pods --selector=app=submariner-globalnet -n submariner-operator -o jsonpath={.items[*].metadata.name}
  3008. 2020-10-08T11:45:04.2580182Z [lighthouse]$ [cluster1] kubectl get pods --selector=app=submariner-globalnet -n submariner-operator -o jsonpath={.items[*].metadata.name}
  3009. 2020-10-08T11:45:04.2593666Z [lighthouse]$ [cluster1] command kubectl --context=cluster1 get pods --selector=app=submariner-globalnet -n submariner-operator -o jsonpath={.items[*].metadata.name}
  3010. 2020-10-08T11:45:04.2605021Z [lighthouse]$ [cluster1] kubectl --context=cluster1 get pods --selector=app=submariner-globalnet -n submariner-operator -o jsonpath={.items[*].metadata.name}
  3011. 2020-10-08T11:45:05.0127378Z [lighthouse]$ [cluster1] kubectl get pods --selector=app=submariner-engine -n submariner-operator -o jsonpath={.items[*].metadata.name}
  3012. 2020-10-08T11:45:05.0136558Z [lighthouse]$ [cluster1] kubectl get pods --selector=app=submariner-engine -n submariner-operator -o jsonpath={.items[*].metadata.name}
  3013. 2020-10-08T11:45:05.0149922Z [lighthouse]$ [cluster1] command kubectl --context=cluster1 get pods --selector=app=submariner-engine -n submariner-operator -o jsonpath={.items[*].metadata.name}
  3014. 2020-10-08T11:45:05.0162719Z [lighthouse]$ [cluster1] kubectl --context=cluster1 get pods --selector=app=submariner-engine -n submariner-operator -o jsonpath={.items[*].metadata.name}
  3015. 2020-10-08T11:45:05.8195556Z [lighthouse]$ [cluster1] kubectl -n submariner-operator logs submariner-gateway-jgnv7
  3016. 2020-10-08T11:45:05.8213417Z [lighthouse]$ [cluster1] kubectl -n submariner-operator logs submariner-gateway-jgnv7
  3017. 2020-10-08T11:45:05.8227001Z [lighthouse]$ [cluster1] command kubectl --context=cluster1 -n submariner-operator logs submariner-gateway-jgnv7
  3018. 2020-10-08T11:45:05.8236153Z [lighthouse]$ [cluster1] kubectl --context=cluster1 -n submariner-operator logs submariner-gateway-jgnv7
  3019. 2020-10-08T11:45:06.5969905Z [cluster1] I1008 11:32:26.507866 1 shared_informer.go:197] Waiting for caches to sync for service config
  3020. 2020-10-08T11:45:06.5970757Z [cluster1] I1008 11:32:26.507906 1 config.go:131] Starting endpoints config controller
  3021. 2020-10-08T11:45:06.5971564Z [cluster1] I1008 11:32:26.507915 1 shared_informer.go:197] Waiting for caches to sync for endpoints config
  3022. 2020-10-08T11:45:06.5972239Z [cluster1] I1008 11:32:26.608030 1 shared_informer.go:204] Caches are synced for service config
  3023. 2020-10-08T11:45:06.5973005Z [cluster1] I1008 11:32:26.608118 1 shared_informer.go:204] Caches are synced for endpoints config
  3024. 2020-10-08T11:45:06.5973714Z [cluster1] ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
  3025. 2020-10-08T11:45:06.5974031Z [cluster1] ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
  3026. 2020-10-08T11:45:06.5974335Z [cluster1] ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
  3027. 2020-10-08T11:45:06.5975668Z [cluster1] +++++++++++++++++++++: Logs for Pod submariner-gateway-jgnv7 in namespace submariner-operator :++++++++++++++++++++++
  3028. 2020-10-08T11:45:06.5976534Z [cluster1] + trap 'exit 1' SIGTERM SIGINT
  3029. 2020-10-08T11:45:06.5976990Z [cluster1] + export CHARON_PID_FILE=/var/run/charon.pid
  3030. 2020-10-08T11:45:06.5977647Z [cluster1] + CHARON_PID_FILE=/var/run/charon.pid
  3031. 2020-10-08T11:45:06.5978243Z [cluster1] + rm -f /var/run/charon.pid
  3032. 2020-10-08T11:45:06.5978677Z [cluster1] + SUBMARINER_VERBOSITY=2
  3033. 2020-10-08T11:45:06.5979218Z [cluster1] + '[' false == true ']'
  3034. 2020-10-08T11:45:06.5979679Z [cluster1] + DEBUG=-v=2
  3035. 2020-10-08T11:45:06.5980164Z [cluster1] + mkdir -p /etc/ipsec
  3036. 2020-10-08T11:45:06.5980861Z [cluster1] + sysctl -w net.ipv4.conf.all.send_redirects=0
  3037. 2020-10-08T11:45:06.5981532Z [cluster1] net.ipv4.conf.all.send_redirects = 0
  3038. 2020-10-08T11:45:06.5982248Z [cluster1] + export PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/libexec/strongswan
  3039. 2020-10-08T11:45:06.5983013Z [cluster1] + PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/libexec/strongswan
  3040. 2020-10-08T11:45:06.5983793Z [cluster1] + for f in iptables-save iptables
  3041. 2020-10-08T11:45:06.5984465Z [cluster1] ++ find_iptables_on_host iptables-save
  3042. 2020-10-08T11:45:06.5985166Z [cluster1] ++ chroot /host test -x /usr/sbin/iptables-save
  3043. 2020-10-08T11:45:06.5985964Z [cluster1] chroot: cannot change root directory to '/host': No such file or directory
  3044. 2020-10-08T11:45:06.5986744Z [cluster1] ++ chroot /host test -x /sbin/iptables-save
  3045. 2020-10-08T11:45:06.5987565Z [cluster1] chroot: cannot change root directory to '/host': No such file or directory
  3046. 2020-10-08T11:45:06.5988091Z [cluster1] ++ echo unknown
  3047. 2020-10-08T11:45:06.5988456Z [cluster1] + location=unknown
  3048. 2020-10-08T11:45:06.5989002Z [cluster1] + '[' unknown '!=' unknown ']'
  3049. 2020-10-08T11:45:06.5989955Z [cluster1] + echo 'WARNING: not using iptables wrapper because iptables was not detected on the'
  3050. 2020-10-08T11:45:06.5990853Z [cluster1] WARNING: not using iptables wrapper because iptables was not detected on the
  3051. 2020-10-08T11:45:06.5991693Z [cluster1] + echo 'host at the following paths [/usr/sbin, /sbin].'
  3052. 2020-10-08T11:45:06.5992221Z [cluster1] host at the following paths [/usr/sbin, /sbin].
  3053. 2020-10-08T11:45:06.5993051Z [cluster1] + echo 'Either the host file system isn'\''t mounted or the host does not have iptables'
  3054. 2020-10-08T11:45:06.5993959Z [cluster1] Either the host file system isn't mounted or the host does not have iptables
  3055. 2020-10-08T11:45:06.5994989Z [cluster1] + echo 'installed. The pod will use the image installed iptables version.'
  3056. 2020-10-08T11:45:06.5995823Z [cluster1] installed. The pod will use the image installed iptables version.
  3057. 2020-10-08T11:45:06.5996615Z [cluster1] + for f in iptables-save iptables
  3058. 2020-10-08T11:45:06.5997098Z [cluster1] ++ find_iptables_on_host iptables
  3059. 2020-10-08T11:45:06.5997731Z [cluster1] ++ chroot /host test -x /usr/sbin/iptables
  3060. 2020-10-08T11:45:06.5998511Z [cluster1] chroot: cannot change root directory to '/host': No such file or directory
  3061. 2020-10-08T11:45:06.5999239Z [cluster1] ++ chroot /host test -x /sbin/iptables
  3062. 2020-10-08T11:45:06.6000151Z [cluster1] chroot: cannot change root directory to '/host': No such file or directory
  3063. 2020-10-08T11:45:06.6000678Z [cluster1] ++ echo unknown
  3064. 2020-10-08T11:45:06.6001047Z [cluster1] + location=unknown
  3065. 2020-10-08T11:45:06.6001671Z [cluster1] + '[' unknown '!=' unknown ']'
  3066. 2020-10-08T11:45:06.6002410Z [cluster1] + echo 'WARNING: not using iptables wrapper because iptables was not detected on the'
  3067. 2020-10-08T11:45:06.6003218Z [cluster1] WARNING: not using iptables wrapper because iptables was not detected on the
  3068. 2020-10-08T11:45:06.6004000Z [cluster1] + echo 'host at the following paths [/usr/sbin, /sbin].'
  3069. 2020-10-08T11:45:06.6004501Z [cluster1] host at the following paths [/usr/sbin, /sbin].
  3070. 2020-10-08T11:45:06.6005255Z [cluster1] + echo 'Either the host file system isn'\''t mounted or the host does not have iptables'
  3071. 2020-10-08T11:45:06.6006089Z [cluster1] Either the host file system isn't mounted or the host does not have iptables
  3072. 2020-10-08T11:45:06.6006928Z [cluster1] + echo 'installed. The pod will use the image installed iptables version.'
  3073. 2020-10-08T11:45:06.6007862Z [cluster1] installed. The pod will use the image installed iptables version.
  3074. 2020-10-08T11:45:06.6008688Z [cluster1] + exec submariner-engine -v=2 -alsologtostderr
  3075. 2020-10-08T11:45:06.6009310Z [cluster1] I1008 11:34:55.312338 1 main.go:71] Starting the submariner gateway engine
  3076. 2020-10-08T11:45:06.6135572Z [cluster1] W1008 11:34:55.323096 1 client_config.go:543] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work.
  3077. 2020-10-08T11:45:06.6136524Z [cluster1] I1008 11:34:55.345840 1 main.go:99] Creating the cable engine
  3078. 2020-10-08T11:45:06.6137055Z [cluster1] I1008 11:34:55.347383 1 syncer.go:51] CableEngine syncer started
  3079. 2020-10-08T11:45:06.6137922Z [cluster1] I1008 11:34:55.387798 1 main.go:263] Gateway leader election config values: main.leaderConfig{LeaseDuration:10, RenewDeadline:5, RetryPeriod:2}
  3080. 2020-10-08T11:45:06.6139376Z [cluster1] I1008 11:34:55.388175 1 main.go:280] Using namespace "submariner-operator" for the leader election lock
  3081. 2020-10-08T11:45:06.6140883Z [cluster1] I1008 11:34:55.388359 1 leaderelection.go:242] attempting to acquire leader lease submariner-operator/submariner-engine-lock...
  3082. 2020-10-08T11:45:06.6142501Z [cluster1] I1008 11:34:55.458610 1 leaderelection.go:252] successfully acquired lease submariner-operator/submariner-engine-lock
  3083. 2020-10-08T11:45:06.6145535Z [cluster1] I1008 11:34:55.458936 1 event.go:281] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"submariner-operator", Name:"submariner-engine-lock", UID:"b1df2726-10ab-42fd-88fa-f6fa91db4ebf", APIVersion:"v1", ResourceVersion:"1204", FieldPath:""}): type: 'Normal' reason: 'LeaderElection' cluster1-worker-submariner-engine became leader
  3084. 2020-10-08T11:45:06.6147489Z [cluster1] I1008 11:34:55.458978 1 main.go:133] Creating the tunnel controller
  3085. 2020-10-08T11:45:06.6148122Z [cluster1] I1008 11:34:55.459029 1 main.go:141] Creating the kubernetes central datastore
  3086. 2020-10-08T11:45:06.6148967Z [cluster1] I1008 11:34:55.459056 1 kubernetes.go:60] Rendered API server host: "https://172.17.0.8:6443"
  3087. 2020-10-08T11:45:06.6149620Z [cluster1] I1008 11:34:55.757448 1 main.go:151] Creating the datastore syncer
  3088. 2020-10-08T11:45:06.6150236Z [cluster1] I1008 11:34:55.757558 1 strongswan.go:99] Initializing StrongSwan IPSec driver
  3089. 2020-10-08T11:45:06.6151062Z [cluster1] I1008 11:34:55.759006 1 strongswan.go:460] Starting charon
  3090. 2020-10-08T11:45:06.6152123Z [cluster1] W1008 11:34:55.761632 1 strongswan.go:543] Failed to connect to charon - retrying: dial unix /var/run/charon.vici: connect: no such file or directory
  3091. 2020-10-08T11:45:06.6153145Z [cluster1] 00[DMN] Starting IKE charon daemon (strongSwan 5.9.0, Linux 5.4.0-1026-azure, x86_64)
  3092. 2020-10-08T11:45:06.6154115Z [cluster1] 00[CFG] PKCS11 module '<name>' lacks library path
  3093. 2020-10-08T11:45:06.6154746Z [cluster1] 00[LIB] openssl FIPS mode(0) - disabled
  3094. 2020-10-08T11:45:06.6155679Z [cluster1] 00[CFG] loading ca certificates from '/etc/strongswan/ipsec.d/cacerts'
  3095. 2020-10-08T11:45:06.6156663Z [cluster1] 00[CFG] loading aa certificates from '/etc/strongswan/ipsec.d/aacerts'
  3096. 2020-10-08T11:45:06.6157652Z [cluster1] 00[CFG] loading ocsp signer certificates from '/etc/strongswan/ipsec.d/ocspcerts'
  3097. 2020-10-08T11:45:06.6158664Z [cluster1] 00[CFG] loading attribute certificates from '/etc/strongswan/ipsec.d/acerts'
  3098. 2020-10-08T11:45:06.6159689Z [cluster1] 00[CFG] loading crls from '/etc/strongswan/ipsec.d/crls'
  3099. 2020-10-08T11:45:06.6160516Z [cluster1] 00[CFG] loading secrets from '/etc/strongswan/ipsec.secrets'
  3100. 2020-10-08T11:45:06.6161323Z [cluster1] 00[CFG] opening triplet file /etc/strongswan/ipsec.d/triplets.dat failed: No such file or directory
  3101. 2020-10-08T11:45:06.6162080Z [cluster1] 00[CFG] loaded 0 RADIUS server configurations
  3102. 2020-10-08T11:45:06.6162632Z [cluster1] 00[CFG] HA config misses local/remote address
  3103. 2020-10-08T11:45:06.6163366Z [cluster1] 00[CFG] no script for ext-auth script defined, disabled
  3104. 2020-10-08T11:45:06.6166523Z [cluster1] 00[LIB] loaded plugins: charon pkcs11 tpm aesni aes des rc2 sha2 sha1 md4 md5 mgf1 random nonce x509 revocation constraints acert pubkey pkcs1 pkcs7 pkcs8 pkcs12 pgp dnskey sshkey pem openssl gcrypt fips-prf gmp curve25519 chapoly xcbc cmac hmac ctr ccm gcm drbg newhope curl attr kernel-netlink resolve socket-default farp stroke vici updown eap-identity eap-sim eap-aka eap-aka-3gpp eap-aka-3gpp2 eap-md5 eap-gtc eap-mschapv2 eap-dynamic eap-radius eap-tls eap-ttls eap-peap xauth-generic xauth-eap xauth-pam xauth-noauth dhcp led duplicheck unity counters
  3105. 2020-10-08T11:45:06.6168937Z [cluster1] 00[JOB] spawning 16 worker threads
  3106. 2020-10-08T11:45:06.6169575Z [cluster1] I1008 11:34:56.763023 1 cableengine.go:71] CableEngine controller started, driver: "strongswan"
  3107. 2020-10-08T11:45:06.6170344Z [cluster1] I1008 11:34:56.763089 1 datastoresyncer.go:145] Starting the datastore syncer
  3108. 2020-10-08T11:45:06.6171056Z [cluster1] I1008 11:34:56.763122 1 datastoresyncer.go:147] Waiting for informer caches to sync
  3109. 2020-10-08T11:45:06.6171826Z [cluster1] I1008 11:34:56.763147 1 datastoresyncer.go:79] Ensuring we are the only endpoint active for this cluster
  3110. 2020-10-08T11:45:06.6172589Z [cluster1] I1008 11:34:56.763107 1 tunnel.go:47] Starting the tunnel controller
  3111. 2020-10-08T11:45:06.6173140Z [cluster1] I1008 11:34:56.764041 1 tunnel.go:50] Waiting for informer caches to sync
  3112. 2020-10-08T11:45:06.6173694Z [cluster1] I1008 11:34:56.764088 1 tunnel.go:58] Tunnel controller started
  3113. 2020-10-08T11:45:06.6175025Z [cluster1] I1008 11:34:56.765829 1 datastoresyncer.go:158] Reconciling local submariner Cluster: types.SubmarinerCluster{ID:"cluster1", Spec:v1.ClusterSpec{ClusterID:"cluster1", ColorCodes:[]string{"blue"}, ServiceCIDR:[]string{"100.91.0.0/16"}, ClusterCIDR:[]string{"10.241.0.0/16"}, GlobalCIDR:[]string{}}}
  3114. 2020-10-08T11:45:06.6177391Z [cluster1] I1008 11:34:56.767037 1 datastoresyncer.go:312] In reconcileClusterCRD: &types.SubmarinerCluster{ID:"cluster1", Spec:v1.ClusterSpec{ClusterID:"cluster1", ColorCodes:[]string{"blue"}, ServiceCIDR:[]string{"100.91.0.0/16"}, ClusterCIDR:[]string{"10.241.0.0/16"}, GlobalCIDR:[]string{}}}
  3115. 2020-10-08T11:45:06.6179135Z [cluster1] I1008 11:34:56.779166 1 datastoresyncer.go:357] Successfully created submariner Cluster "cluster1" in the local datastore
  3116. 2020-10-08T11:45:06.6294445Z [cluster1] I1008 11:34:56.779888 1 datastoresyncer.go:164] Reconciling local submariner Endpoint: types.SubmarinerEndpoint{Spec:v1.EndpointSpec{ClusterID:"cluster1", CableName:"submariner-cable-cluster1-172-17-0-9", Hostname:"cluster1-worker", Subnets:[]string{"100.91.0.0/16", "10.241.0.0/16"}, PrivateIP:"172.17.0.9", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string{}}}
  3117. 2020-10-08T11:45:06.6298168Z [cluster1] I1008 11:34:56.780056 1 datastoresyncer.go:387] In reconcileEndpointCRD: &types.SubmarinerEndpoint{Spec:v1.EndpointSpec{ClusterID:"cluster1", CableName:"submariner-cable-cluster1-172-17-0-9", Hostname:"cluster1-worker", Subnets:[]string{"100.91.0.0/16", "10.241.0.0/16"}, PrivateIP:"172.17.0.9", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string{}}}
  3118. 2020-10-08T11:45:06.6305144Z [cluster1] I1008 11:34:56.809954 1 tunnel.go:95] Tunnel controller processing added or updated submariner Endpoint object: &v1.Endpoint{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"cluster1-submariner-cable-cluster1-172-17-0-9", GenerateName:"", Namespace:"submariner-operator", SelfLink:"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster1-submariner-cable-cluster1-172-17-0-9", UID:"23aee4db-4473-4b17-8f51-aedd32e3a5ff", ResourceVersion:"1212", Generation:1, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63737753696, loc:(*time.Location)(0x225ef80)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v1.EndpointSpec{ClusterID:"cluster1", CableName:"submariner-cable-cluster1-172-17-0-9", Hostname:"cluster1-worker", Subnets:[]string{"100.91.0.0/16", "10.241.0.0/16"}, PrivateIP:"172.17.0.9", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  3119. 2020-10-08T11:45:06.6309958Z [cluster1] I1008 11:34:56.819483 1 cableengine.go:92] Not installing cable for local cluster
  3120. 2020-10-08T11:45:06.6313271Z [cluster1] I1008 11:34:56.819672 1 tunnel.go:108] Tunnel controller successfully installed Endpoint cable submariner-cable-cluster1-172-17-0-9 in the engine
  3121. 2020-10-08T11:45:06.6316197Z [cluster1] I1008 11:34:56.819971 1 datastoresyncer.go:432] Successfully created submariner Endpoint "cluster1-submariner-cable-cluster1-172-17-0-9" in the local datastore
  3122. 2020-10-08T11:45:06.6318483Z [cluster1] I1008 11:34:56.821564 1 datastoresyncer.go:177] Datastore syncer started
  3123. 2020-10-08T11:45:06.6320166Z [cluster1] I1008 11:34:56.827159 1 kubernetes.go:317] In SetCluster: &types.SubmarinerCluster{ID:"cluster1", Spec:v1.ClusterSpec{ClusterID:"cluster1", ColorCodes:[]string{"blue"}, ServiceCIDR:[]string{"100.91.0.0/16"}, ClusterCIDR:[]string{"10.241.0.0/16"}, GlobalCIDR:[]string{}}}
  3124. 2020-10-08T11:45:06.6328021Z [cluster1] I1008 11:34:56.837794 1 datastoresyncer.go:287] Processing local submariner Endpoint object: &v1.Endpoint{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"cluster1-submariner-cable-cluster1-172-17-0-9", GenerateName:"", Namespace:"submariner-operator", SelfLink:"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster1-submariner-cable-cluster1-172-17-0-9", UID:"23aee4db-4473-4b17-8f51-aedd32e3a5ff", ResourceVersion:"1212", Generation:1, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63737753696, loc:(*time.Location)(0x225ef80)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v1.EndpointSpec{ClusterID:"cluster1", CableName:"submariner-cable-cluster1-172-17-0-9", Hostname:"cluster1-worker", Subnets:[]string{"100.91.0.0/16", "10.241.0.0/16"}, PrivateIP:"172.17.0.9", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  3125. 2020-10-08T11:45:06.6335687Z [cluster1] I1008 11:34:56.841675 1 kubernetes.go:370] In SetEndpoint: &types.SubmarinerEndpoint{Spec:v1.EndpointSpec{ClusterID:"cluster1", CableName:"submariner-cable-cluster1-172-17-0-9", Hostname:"cluster1-worker", Subnets:[]string{"100.91.0.0/16", "10.241.0.0/16"}, PrivateIP:"172.17.0.9", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  3126. 2020-10-08T11:45:06.6337414Z [cluster1] I1008 11:34:56.851803 1 kubernetes.go:342] Successfully created submariner Cluster "cluster1" in the central datastore
  3127. 2020-10-08T11:45:06.6338113Z [cluster1] I1008 11:34:56.852732 1 kubernetes.go:155] AddFunc in WatchClusters called
  3128. 2020-10-08T11:45:06.6338704Z [cluster1] I1008 11:34:56.860123 1 kubernetes.go:240] AddFunc in WatchEndpoints called
  3129. 2020-10-08T11:45:06.6340226Z [cluster1] I1008 11:34:56.861095 1 kubernetes.go:395] Successfully created submariner Endpoint "cluster1-submariner-cable-cluster1-172-17-0-9" in the central datastore
  3130. 2020-10-08T11:45:06.6341839Z [cluster1] 16[KNL] interface vx-submariner activated
  3131. 2020-10-08T11:45:06.6342503Z [cluster1] 12[KNL] 240.17.0.9 appeared on vx-submariner
  3132. 2020-10-08T11:45:06.6343073Z [cluster1] I1008 11:35:26.093615 1 kubernetes.go:155] AddFunc in WatchClusters called
  3133. 2020-10-08T11:45:06.6344462Z [cluster1] I1008 11:35:26.093709 1 datastoresyncer.go:312] In reconcileClusterCRD: &types.SubmarinerCluster{ID:"cluster2", Spec:v1.ClusterSpec{ClusterID:"cluster2", ColorCodes:[]string{"blue"}, ServiceCIDR:[]string{"100.92.0.0/16"}, ClusterCIDR:[]string{"10.242.0.0/16"}, GlobalCIDR:[]string{}}}
  3134. 2020-10-08T11:45:06.6347965Z [cluster1] I1008 11:35:26.110720 1 datastoresyncer.go:357] Successfully created submariner Cluster "cluster2" in the local datastore
  3135. 2020-10-08T11:45:06.6348993Z [cluster1] I1008 11:35:26.119120 1 kubernetes.go:240] AddFunc in WatchEndpoints called
  3136. 2020-10-08T11:45:06.6351805Z [cluster1] I1008 11:35:26.119139 1 datastoresyncer.go:387] In reconcileEndpointCRD: &types.SubmarinerEndpoint{Spec:v1.EndpointSpec{ClusterID:"cluster2", CableName:"submariner-cable-cluster2-172-17-0-5", Hostname:"cluster2-worker", Subnets:[]string{"100.92.0.0/16", "10.242.0.0/16"}, PrivateIP:"172.17.0.5", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  3137. 2020-10-08T11:45:06.6355583Z [cluster1] 05[NET] received packet: from 172.17.0.5[500] to 172.17.0.9[500] (500 bytes)
  3138. 2020-10-08T11:45:06.6356203Z [cluster1] 05[ENC] parsed IKE_SA_INIT request 0 [ SA KE No N(NATD_S_IP) N(NATD_D_IP) N(FRAG_SUP) N(HASH_ALG) N(REDIR_SUP) ]
  3139. 2020-10-08T11:45:06.6356846Z [cluster1] 05[IKE] no IKE config found for 172.17.0.9...172.17.0.5, sending NO_PROPOSAL_CHOSEN
  3140. 2020-10-08T11:45:06.6357401Z [cluster1] 05[ENC] generating IKE_SA_INIT response 0 [ N(NO_PROP) ]
  3141. 2020-10-08T11:45:06.6357903Z [cluster1] 05[NET] sending packet: from 172.17.0.9[500] to 172.17.0.5[500] (36 bytes)
  3142. 2020-10-08T11:45:06.6366662Z [cluster1] I1008 11:35:26.133703 1 tunnel.go:95] Tunnel controller processing added or updated submariner Endpoint object: &v1.Endpoint{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"cluster2-submariner-cable-cluster2-172-17-0-5", GenerateName:"", Namespace:"submariner-operator", SelfLink:"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster2-submariner-cable-cluster2-172-17-0-5", UID:"474bcd76-5748-4372-9832-4a15a4b1804d", ResourceVersion:"1332", Generation:1, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63737753726, loc:(*time.Location)(0x225ef80)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v1.EndpointSpec{ClusterID:"cluster2", CableName:"submariner-cable-cluster2-172-17-0-5", Hostname:"cluster2-worker", Subnets:[]string{"100.92.0.0/16", "10.242.0.0/16"}, PrivateIP:"172.17.0.5", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  3143. 2020-10-08T11:45:06.6373087Z [cluster1] I1008 11:35:26.133993 1 cableengine.go:101] Installing Endpoint cable "submariner-cable-cluster2-172-17-0-5"
  3144. 2020-10-08T11:45:06.6374232Z [cluster1] 12[CFG] loaded IKE shared key for: '172.17.0.5'
  3145. 2020-10-08T11:45:06.6375239Z [cluster1] 04[CFG] added vici connection: submariner-cable-cluster2-172-17-0-5
  3146. 2020-10-08T11:45:06.6376997Z [cluster1] 04[CFG] initiating 'submariner-child-submariner-cable-cluster2-172-17-0-5'
  3147. 2020-10-08T11:45:06.6378771Z [cluster1] 04[IKE] initiating IKE_SA submariner-cable-cluster2-172-17-0-5[2] to 172.17.0.5
  3148. 2020-10-08T11:45:06.6379779Z [cluster1] 04[ENC] generating IKE_SA_INIT request 0 [ SA KE No N(NATD_S_IP) N(NATD_D_IP) N(FRAG_SUP) N(HASH_ALG) N(REDIR_SUP) ]
  3149. 2020-10-08T11:45:06.6380440Z [cluster1] 04[NET] sending packet: from 172.17.0.9[500] to 172.17.0.5[500] (500 bytes)
  3150. 2020-10-08T11:45:06.6382403Z [cluster1] I1008 11:35:26.140558 1 cableengine.go:150] Successfully installed Endpoint cable "submariner-cable-cluster2-172-17-0-5" with remote IP 172.17.0.5
  3151. 2020-10-08T11:45:06.6384777Z [cluster1] I1008 11:35:26.140584 1 tunnel.go:108] Tunnel controller successfully installed Endpoint cable submariner-cable-cluster2-172-17-0-5 in the engine
  3152. 2020-10-08T11:45:06.6386975Z [cluster1] I1008 11:35:26.142189 1 datastoresyncer.go:432] Successfully created submariner Endpoint "cluster2-submariner-cable-cluster2-172-17-0-5" in the local datastore
  3153. 2020-10-08T11:45:06.6389522Z [cluster1] I1008 11:35:26.159946 1 datastoresyncer.go:271] The updated submariner Endpoint "cluster2" is not for this cluster - skipping updating the datastore
  3154. 2020-10-08T11:45:06.6390399Z [cluster1] 16[NET] received packet: from 172.17.0.5[500] to 172.17.0.9[500] (464 bytes)
  3155. 2020-10-08T11:45:06.6391105Z [cluster1] 16[ENC] parsed IKE_SA_INIT response 0 [ SA KE No N(NATD_S_IP) N(NATD_D_IP) N(FRAG_SUP) N(HASH_ALG) N(CHDLESS_SUP) N(MULT_AUTH) ]
  3156. 2020-10-08T11:45:06.6391840Z [cluster1] 16[CFG] selected proposal: IKE:AES_GCM_16_128/PRF_HMAC_SHA2_256/MODP_2048
  3157. 2020-10-08T11:45:06.6392355Z [cluster1] 16[IKE] remote host is behind NAT
  3158. 2020-10-08T11:45:06.6393097Z [cluster1] 16[IKE] authentication of '172.17.0.9' (myself) with pre-shared key
  3159. 2020-10-08T11:45:06.6394659Z [cluster1] 16[IKE] establishing CHILD_SA submariner-child-submariner-cable-cluster2-172-17-0-5{1}
  3160. 2020-10-08T11:45:06.6396018Z [cluster1] 16[ENC] generating IKE_AUTH request 1 [ IDi N(INIT_CONTACT) IDr AUTH SA TSi TSr N(MULT_AUTH) N(EAP_ONLY) N(MSG_ID_SYN_SUP) ]
  3161. 2020-10-08T11:45:06.6396664Z [cluster1] 16[NET] sending packet: from 172.17.0.9[4500] to 172.17.0.5[4500] (333 bytes)
  3162. 2020-10-08T11:45:06.6397147Z [cluster1] 14[NET] received packet: from 172.17.0.5[4500] to 172.17.0.9[4500] (257 bytes)
  3163. 2020-10-08T11:45:06.6397646Z [cluster1] 14[ENC] parsed IKE_AUTH response 1 [ IDr AUTH SA TSi TSr ]
  3164. 2020-10-08T11:45:06.6398416Z [cluster1] 14[IKE] authentication of '172.17.0.5' with pre-shared key successful
  3165. 2020-10-08T11:45:06.6399561Z [cluster1] 14[IKE] IKE_SA submariner-cable-cluster2-172-17-0-5[2] established between 172.17.0.9[172.17.0.9]...172.17.0.5[172.17.0.5]
  3166. 2020-10-08T11:45:06.6400370Z [cluster1] 14[IKE] scheduling rekeying in 14012s
  3167. 2020-10-08T11:45:06.6400784Z [cluster1] 14[IKE] maximum IKE_SA lifetime 15452s
  3168. 2020-10-08T11:45:06.6401636Z [cluster1] 14[CFG] selected proposal: ESP:AES_GCM_16_128/NO_EXT_SEQ
  3169. 2020-10-08T11:45:06.6403327Z [cluster1] 14[IKE] CHILD_SA submariner-child-submariner-cable-cluster2-172-17-0-5{1} established with SPIs c3e8a822_i cc5f34fc_o and TS 10.241.0.0/16 100.91.0.0/16 172.17.0.9/32 === 10.242.0.0/16 100.92.0.0/16 172.17.0.5/32
  3170. 2020-10-08T11:45:06.6404727Z [cluster1] I1008 11:35:26.835099 1 kubernetes.go:180] UpdateFunc in WatchClusters called
  3171. 2020-10-08T11:45:06.6405427Z [cluster1] I1008 11:35:26.835289 1 kubernetes.go:180] UpdateFunc in WatchClusters called
  3172. 2020-10-08T11:45:06.6407041Z [cluster1] I1008 11:35:26.835305 1 datastoresyncer.go:312] In reconcileClusterCRD: &types.SubmarinerCluster{ID:"cluster2", Spec:v1.ClusterSpec{ClusterID:"cluster2", ColorCodes:[]string{"blue"}, ServiceCIDR:[]string{"100.92.0.0/16"}, ClusterCIDR:[]string{"10.242.0.0/16"}, GlobalCIDR:[]string{}}}
  3173. 2020-10-08T11:45:06.6408593Z [cluster1] I1008 11:35:26.838070 1 datastoresyncer.go:360] Cluster "cluster2" matched what we received from datastore - not updating
  3174. 2020-10-08T11:45:06.6409319Z [cluster1] I1008 11:35:26.838263 1 kubernetes.go:264] UpdateFunc in WatchEndpoints called
  3175. 2020-10-08T11:45:06.6409942Z [cluster1] I1008 11:35:26.838275 1 kubernetes.go:264] UpdateFunc in WatchEndpoints called
  3176. 2020-10-08T11:45:06.6413856Z [cluster1] I1008 11:35:26.838296 1 datastoresyncer.go:387] In reconcileEndpointCRD: &types.SubmarinerEndpoint{Spec:v1.EndpointSpec{ClusterID:"cluster2", CableName:"submariner-cable-cluster2-172-17-0-5", Hostname:"cluster2-worker", Subnets:[]string{"100.92.0.0/16", "10.242.0.0/16"}, PrivateIP:"172.17.0.5", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  3177. 2020-10-08T11:45:06.6417251Z [cluster1] I1008 11:35:26.840373 1 datastoresyncer.go:435] Endpoint "cluster2-submariner-cable-cluster2-172-17-0-5" matched what we received from datastore - not updating
  3178. 2020-10-08T11:45:06.6418594Z [cluster1] 15[KNL] interface vethwepgcc56be4 deleted
  3179. 2020-10-08T11:45:06.6419208Z [cluster1] 06[KNL] interface vethweplcc56be4 activated
  3180. 2020-10-08T11:45:06.6419842Z [cluster1] 16[KNL] interface vethweplcc56be4 deactivated
  3181. 2020-10-08T11:45:06.6420463Z [cluster1] 05[KNL] interface vethweplcc56be4 deleted
  3182. 2020-10-08T11:45:06.6428878Z [cluster1] I1008 11:35:55.765865 1 tunnel.go:95] Tunnel controller processing added or updated submariner Endpoint object: &v1.Endpoint{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"cluster1-submariner-cable-cluster1-172-17-0-9", GenerateName:"", Namespace:"submariner-operator", SelfLink:"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster1-submariner-cable-cluster1-172-17-0-9", UID:"23aee4db-4473-4b17-8f51-aedd32e3a5ff", ResourceVersion:"1212", Generation:1, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63737753696, loc:(*time.Location)(0x225ef80)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v1.EndpointSpec{ClusterID:"cluster1", CableName:"submariner-cable-cluster1-172-17-0-9", Hostname:"cluster1-worker", Subnets:[]string{"100.91.0.0/16", "10.241.0.0/16"}, PrivateIP:"172.17.0.9", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  3183. 2020-10-08T11:45:06.6435532Z [cluster1] I1008 11:35:55.765988 1 cableengine.go:92] Not installing cable for local cluster
  3184. 2020-10-08T11:45:06.6437088Z [cluster1] I1008 11:35:55.766004 1 tunnel.go:108] Tunnel controller successfully installed Endpoint cable submariner-cable-cluster1-172-17-0-9 in the engine
  3185. 2020-10-08T11:45:06.6445184Z [cluster1] I1008 11:35:55.766021 1 tunnel.go:95] Tunnel controller processing added or updated submariner Endpoint object: &v1.Endpoint{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"cluster2-submariner-cable-cluster2-172-17-0-5", GenerateName:"", Namespace:"submariner-operator", SelfLink:"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster2-submariner-cable-cluster2-172-17-0-5", UID:"474bcd76-5748-4372-9832-4a15a4b1804d", ResourceVersion:"1332", Generation:1, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63737753726, loc:(*time.Location)(0x225ef80)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v1.EndpointSpec{ClusterID:"cluster2", CableName:"submariner-cable-cluster2-172-17-0-5", Hostname:"cluster2-worker", Subnets:[]string{"100.92.0.0/16", "10.242.0.0/16"}, PrivateIP:"172.17.0.5", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  3186. 2020-10-08T11:45:06.6451483Z [cluster1] I1008 11:35:55.766086 1 cableengine.go:101] Installing Endpoint cable "submariner-cable-cluster2-172-17-0-5"
  3187. 2020-10-08T11:45:06.6453511Z [cluster1] I1008 11:35:55.767123 1 kubernetes.go:317] In SetCluster: &types.SubmarinerCluster{ID:"cluster1", Spec:v1.ClusterSpec{ClusterID:"cluster1", ColorCodes:[]string{"blue"}, ServiceCIDR:[]string{"100.91.0.0/16"}, ClusterCIDR:[]string{"10.241.0.0/16"}, GlobalCIDR:[]string{}}}
  3188. 2020-10-08T11:45:06.6462343Z [cluster1] I1008 11:35:55.769216 1 datastoresyncer.go:287] Processing local submariner Endpoint object: &v1.Endpoint{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"cluster1-submariner-cable-cluster1-172-17-0-9", GenerateName:"", Namespace:"submariner-operator", SelfLink:"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster1-submariner-cable-cluster1-172-17-0-9", UID:"23aee4db-4473-4b17-8f51-aedd32e3a5ff", ResourceVersion:"1212", Generation:1, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63737753696, loc:(*time.Location)(0x225ef80)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v1.EndpointSpec{ClusterID:"cluster1", CableName:"submariner-cable-cluster1-172-17-0-9", Hostname:"cluster1-worker", Subnets:[]string{"100.91.0.0/16", "10.241.0.0/16"}, PrivateIP:"172.17.0.9", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  3189. 2020-10-08T11:45:06.6470109Z [cluster1] I1008 11:35:55.769317 1 kubernetes.go:370] In SetEndpoint: &types.SubmarinerEndpoint{Spec:v1.EndpointSpec{ClusterID:"cluster1", CableName:"submariner-cable-cluster1-172-17-0-9", Hostname:"cluster1-worker", Subnets:[]string{"100.91.0.0/16", "10.241.0.0/16"}, PrivateIP:"172.17.0.9", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  3190. 2020-10-08T11:45:06.6472871Z [cluster1] I1008 11:35:55.770902 1 cableengine.go:128] Cable "submariner-cable-cluster2-172-17-0-5" is already installed - not installing again
  3191. 2020-10-08T11:45:06.6474843Z [cluster1] I1008 11:35:55.770956 1 tunnel.go:108] Tunnel controller successfully installed Endpoint cable submariner-cable-cluster2-172-17-0-5 in the engine
  3192. 2020-10-08T11:45:06.6476782Z [cluster1] I1008 11:35:55.771607 1 kubernetes.go:398] Endpoint "cluster1-submariner-cable-cluster1-172-17-0-9" matched what we received from datastore - not updating
  3193. 2020-10-08T11:45:06.6478373Z [cluster1] I1008 11:35:55.773560 1 kubernetes.go:345] Cluster "cluster1" matched what we received from datastore - not updating
  3194. 2020-10-08T11:45:06.6479605Z [cluster1] I1008 11:35:55.778055 1 datastoresyncer.go:271] The updated submariner Endpoint "cluster2" is not for this cluster - skipping updating the datastore
  3195. 2020-10-08T11:45:06.6480509Z [cluster1] I1008 11:35:56.835249 1 kubernetes.go:180] UpdateFunc in WatchClusters called
  3196. 2020-10-08T11:45:06.6482573Z [cluster1] I1008 11:35:56.835308 1 kubernetes.go:180] UpdateFunc in WatchClusters called
  3197. 2020-10-08T11:45:06.6484127Z [cluster1] I1008 11:35:56.835322 1 datastoresyncer.go:312] In reconcileClusterCRD: &types.SubmarinerCluster{ID:"cluster2", Spec:v1.ClusterSpec{ClusterID:"cluster2", ColorCodes:[]string{"blue"}, ServiceCIDR:[]string{"100.92.0.0/16"}, ClusterCIDR:[]string{"10.242.0.0/16"}, GlobalCIDR:[]string{}}}
  3198. 2020-10-08T11:45:06.6485528Z [cluster1] I1008 11:35:56.839884 1 kubernetes.go:264] UpdateFunc in WatchEndpoints called
  3199. 2020-10-08T11:45:06.6486361Z [cluster1] I1008 11:35:56.839936 1 kubernetes.go:264] UpdateFunc in WatchEndpoints called
  3200. 2020-10-08T11:45:06.6488912Z [cluster1] I1008 11:35:56.839956 1 datastoresyncer.go:387] In reconcileEndpointCRD: &types.SubmarinerEndpoint{Spec:v1.EndpointSpec{ClusterID:"cluster2", CableName:"submariner-cable-cluster2-172-17-0-5", Hostname:"cluster2-worker", Subnets:[]string{"100.92.0.0/16", "10.242.0.0/16"}, PrivateIP:"172.17.0.5", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  3201. 2020-10-08T11:45:06.6491270Z [cluster1] I1008 11:35:56.840337 1 datastoresyncer.go:360] Cluster "cluster2" matched what we received from datastore - not updating
  3202. 2020-10-08T11:45:06.6493284Z [cluster1] I1008 11:35:56.842833 1 datastoresyncer.go:435] Endpoint "cluster2-submariner-cable-cluster2-172-17-0-5" matched what we received from datastore - not updating
  3203. 2020-10-08T11:45:06.6494602Z [cluster1] I1008 11:36:26.835347 1 kubernetes.go:180] UpdateFunc in WatchClusters called
  3204. 2020-10-08T11:45:06.6495387Z [cluster1] I1008 11:36:26.835370 1 kubernetes.go:180] UpdateFunc in WatchClusters called
  3205. 2020-10-08T11:45:06.6496787Z [cluster1] I1008 11:36:26.835378 1 datastoresyncer.go:312] In reconcileClusterCRD: &types.SubmarinerCluster{ID:"cluster2", Spec:v1.ClusterSpec{ClusterID:"cluster2", ColorCodes:[]string{"blue"}, ServiceCIDR:[]string{"100.92.0.0/16"}, ClusterCIDR:[]string{"10.242.0.0/16"}, GlobalCIDR:[]string{}}}
  3206. 2020-10-08T11:45:06.6498685Z [cluster1] I1008 11:36:26.840023 1 datastoresyncer.go:360] Cluster "cluster2" matched what we received from datastore - not updating
  3207. 2020-10-08T11:45:06.6499522Z [cluster1] I1008 11:36:26.840112 1 kubernetes.go:264] UpdateFunc in WatchEndpoints called
  3208. 2020-10-08T11:45:06.6501744Z [cluster1] I1008 11:36:26.840123 1 kubernetes.go:264] UpdateFunc in WatchEndpoints called
  3209. 2020-10-08T11:45:06.6504785Z [cluster1] I1008 11:36:26.840128 1 datastoresyncer.go:387] In reconcileEndpointCRD: &types.SubmarinerEndpoint{Spec:v1.EndpointSpec{ClusterID:"cluster2", CableName:"submariner-cable-cluster2-172-17-0-5", Hostname:"cluster2-worker", Subnets:[]string{"100.92.0.0/16", "10.242.0.0/16"}, PrivateIP:"172.17.0.5", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  3210. 2020-10-08T11:45:06.6507696Z [cluster1] I1008 11:36:26.842383 1 datastoresyncer.go:435] Endpoint "cluster2-submariner-cable-cluster2-172-17-0-5" matched what we received from datastore - not updating
  3211. 2020-10-08T11:45:06.6514909Z [cluster1] I1008 11:36:55.765364 1 tunnel.go:95] Tunnel controller processing added or updated submariner Endpoint object: &v1.Endpoint{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"cluster1-submariner-cable-cluster1-172-17-0-9", GenerateName:"", Namespace:"submariner-operator", SelfLink:"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster1-submariner-cable-cluster1-172-17-0-9", UID:"23aee4db-4473-4b17-8f51-aedd32e3a5ff", ResourceVersion:"1212", Generation:1, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63737753696, loc:(*time.Location)(0x225ef80)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v1.EndpointSpec{ClusterID:"cluster1", CableName:"submariner-cable-cluster1-172-17-0-9", Hostname:"cluster1-worker", Subnets:[]string{"100.91.0.0/16", "10.241.0.0/16"}, PrivateIP:"172.17.0.9", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  3212. 2020-10-08T11:45:06.6520298Z [cluster1] I1008 11:36:55.765481 1 cableengine.go:92] Not installing cable for local cluster
  3213. 2020-10-08T11:45:06.6522280Z [cluster1] I1008 11:36:55.765504 1 tunnel.go:108] Tunnel controller successfully installed Endpoint cable submariner-cable-cluster1-172-17-0-9 in the engine
  3214. 2020-10-08T11:45:06.6529377Z [cluster1] I1008 11:36:55.765518 1 tunnel.go:95] Tunnel controller processing added or updated submariner Endpoint object: &v1.Endpoint{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"cluster2-submariner-cable-cluster2-172-17-0-5", GenerateName:"", Namespace:"submariner-operator", SelfLink:"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster2-submariner-cable-cluster2-172-17-0-5", UID:"474bcd76-5748-4372-9832-4a15a4b1804d", ResourceVersion:"1332", Generation:1, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63737753726, loc:(*time.Location)(0x225ef80)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v1.EndpointSpec{ClusterID:"cluster2", CableName:"submariner-cable-cluster2-172-17-0-5", Hostname:"cluster2-worker", Subnets:[]string{"100.92.0.0/16", "10.242.0.0/16"}, PrivateIP:"172.17.0.5", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  3215. 2020-10-08T11:45:06.6535951Z [cluster1] I1008 11:36:55.765593 1 cableengine.go:101] Installing Endpoint cable "submariner-cable-cluster2-172-17-0-5"
  3216. 2020-10-08T11:45:06.6537537Z [cluster1] I1008 11:36:55.766883 1 kubernetes.go:317] In SetCluster: &types.SubmarinerCluster{ID:"cluster1", Spec:v1.ClusterSpec{ClusterID:"cluster1", ColorCodes:[]string{"blue"}, ServiceCIDR:[]string{"100.91.0.0/16"}, ClusterCIDR:[]string{"10.241.0.0/16"}, GlobalCIDR:[]string{}}}
  3217. 2020-10-08T11:45:06.6545752Z [cluster1] I1008 11:36:55.768180 1 datastoresyncer.go:287] Processing local submariner Endpoint object: &v1.Endpoint{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"cluster1-submariner-cable-cluster1-172-17-0-9", GenerateName:"", Namespace:"submariner-operator", SelfLink:"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster1-submariner-cable-cluster1-172-17-0-9", UID:"23aee4db-4473-4b17-8f51-aedd32e3a5ff", ResourceVersion:"1212", Generation:1, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63737753696, loc:(*time.Location)(0x225ef80)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v1.EndpointSpec{ClusterID:"cluster1", CableName:"submariner-cable-cluster1-172-17-0-9", Hostname:"cluster1-worker", Subnets:[]string{"100.91.0.0/16", "10.241.0.0/16"}, PrivateIP:"172.17.0.9", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  3218. 2020-10-08T11:45:06.6553299Z [cluster1] I1008 11:36:55.768231 1 kubernetes.go:370] In SetEndpoint: &types.SubmarinerEndpoint{Spec:v1.EndpointSpec{ClusterID:"cluster1", CableName:"submariner-cable-cluster1-172-17-0-9", Hostname:"cluster1-worker", Subnets:[]string{"100.91.0.0/16", "10.241.0.0/16"}, PrivateIP:"172.17.0.9", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  3219. 2020-10-08T11:45:06.6555464Z [cluster1] I1008 11:36:55.769585 1 kubernetes.go:345] Cluster "cluster1" matched what we received from datastore - not updating
  3220. 2020-10-08T11:45:06.6557080Z [cluster1] I1008 11:36:55.769904 1 kubernetes.go:398] Endpoint "cluster1-submariner-cable-cluster1-172-17-0-9" matched what we received from datastore - not updating
  3221. 2020-10-08T11:45:06.6558955Z [cluster1] I1008 11:36:55.772389 1 cableengine.go:128] Cable "submariner-cable-cluster2-172-17-0-5" is already installed - not installing again
  3222. 2020-10-08T11:45:06.6561779Z [cluster1] I1008 11:36:55.772423 1 tunnel.go:108] Tunnel controller successfully installed Endpoint cable submariner-cable-cluster2-172-17-0-5 in the engine
  3223. 2020-10-08T11:45:06.6563586Z [cluster1] I1008 11:36:55.772842 1 datastoresyncer.go:271] The updated submariner Endpoint "cluster2" is not for this cluster - skipping updating the datastore
  3224. 2020-10-08T11:45:06.6564393Z [cluster1] I1008 11:36:56.835449 1 kubernetes.go:180] UpdateFunc in WatchClusters called
  3225. 2020-10-08T11:45:06.6565001Z [cluster1] I1008 11:36:56.835476 1 kubernetes.go:180] UpdateFunc in WatchClusters called
  3226. 2020-10-08T11:45:06.6566233Z [cluster1] I1008 11:36:56.835485 1 datastoresyncer.go:312] In reconcileClusterCRD: &types.SubmarinerCluster{ID:"cluster2", Spec:v1.ClusterSpec{ClusterID:"cluster2", ColorCodes:[]string{"blue"}, ServiceCIDR:[]string{"100.92.0.0/16"}, ClusterCIDR:[]string{"10.242.0.0/16"}, GlobalCIDR:[]string{}}}
  3227. 2020-10-08T11:45:06.6567791Z [cluster1] I1008 11:36:56.839369 1 datastoresyncer.go:360] Cluster "cluster2" matched what we received from datastore - not updating
  3228. 2020-10-08T11:45:06.6568525Z [cluster1] I1008 11:36:56.840254 1 kubernetes.go:264] UpdateFunc in WatchEndpoints called
  3229. 2020-10-08T11:45:06.6569216Z [cluster1] I1008 11:36:56.840270 1 kubernetes.go:264] UpdateFunc in WatchEndpoints called
  3230. 2020-10-08T11:45:06.6571676Z [cluster1] I1008 11:36:56.840277 1 datastoresyncer.go:387] In reconcileEndpointCRD: &types.SubmarinerEndpoint{Spec:v1.EndpointSpec{ClusterID:"cluster2", CableName:"submariner-cable-cluster2-172-17-0-5", Hostname:"cluster2-worker", Subnets:[]string{"100.92.0.0/16", "10.242.0.0/16"}, PrivateIP:"172.17.0.5", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  3231. 2020-10-08T11:45:06.6575728Z [cluster1] I1008 11:36:56.842585 1 datastoresyncer.go:435] Endpoint "cluster2-submariner-cable-cluster2-172-17-0-5" matched what we received from datastore - not updating
  3232. 2020-10-08T11:45:06.6577151Z [cluster1] I1008 11:37:26.835590 1 kubernetes.go:180] UpdateFunc in WatchClusters called
  3233. 2020-10-08T11:45:06.6577906Z [cluster1] I1008 11:37:26.835619 1 kubernetes.go:180] UpdateFunc in WatchClusters called
  3234. 2020-10-08T11:45:06.6579440Z [cluster1] I1008 11:37:26.835642 1 datastoresyncer.go:312] In reconcileClusterCRD: &types.SubmarinerCluster{ID:"cluster2", Spec:v1.ClusterSpec{ClusterID:"cluster2", ColorCodes:[]string{"blue"}, ServiceCIDR:[]string{"100.92.0.0/16"}, ClusterCIDR:[]string{"10.242.0.0/16"}, GlobalCIDR:[]string{}}}
  3235. 2020-10-08T11:45:06.6580950Z [cluster1] I1008 11:37:26.840507 1 kubernetes.go:264] UpdateFunc in WatchEndpoints called
  3236. 2020-10-08T11:45:06.6582456Z [cluster1] I1008 11:37:26.840535 1 kubernetes.go:264] UpdateFunc in WatchEndpoints called
  3237. 2020-10-08T11:45:06.6585178Z [cluster1] I1008 11:37:26.840545 1 datastoresyncer.go:387] In reconcileEndpointCRD: &types.SubmarinerEndpoint{Spec:v1.EndpointSpec{ClusterID:"cluster2", CableName:"submariner-cable-cluster2-172-17-0-5", Hostname:"cluster2-worker", Subnets:[]string{"100.92.0.0/16", "10.242.0.0/16"}, PrivateIP:"172.17.0.5", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  3238. 2020-10-08T11:45:06.6587231Z [cluster1] I1008 11:37:26.841056 1 datastoresyncer.go:360] Cluster "cluster2" matched what we received from datastore - not updating
  3239. 2020-10-08T11:45:06.6588961Z [cluster1] I1008 11:37:26.842361 1 datastoresyncer.go:435] Endpoint "cluster2-submariner-cable-cluster2-172-17-0-5" matched what we received from datastore - not updating
  3240. 2020-10-08T11:45:06.6590203Z [cluster1] 04[KNL] interface vethwepg4c7a9ee deleted
  3241. 2020-10-08T11:45:06.6590768Z [cluster1] 11[KNL] interface vethwepl4c7a9ee activated
  3242. 2020-10-08T11:45:06.6597774Z [cluster1] I1008 11:37:55.765569 1 tunnel.go:95] Tunnel controller processing added or updated submariner Endpoint object: &v1.Endpoint{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"cluster1-submariner-cable-cluster1-172-17-0-9", GenerateName:"", Namespace:"submariner-operator", SelfLink:"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster1-submariner-cable-cluster1-172-17-0-9", UID:"23aee4db-4473-4b17-8f51-aedd32e3a5ff", ResourceVersion:"1212", Generation:1, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63737753696, loc:(*time.Location)(0x225ef80)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v1.EndpointSpec{ClusterID:"cluster1", CableName:"submariner-cable-cluster1-172-17-0-9", Hostname:"cluster1-worker", Subnets:[]string{"100.91.0.0/16", "10.241.0.0/16"}, PrivateIP:"172.17.0.9", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  3243. 2020-10-08T11:45:06.6604818Z [cluster1] I1008 11:37:55.765677 1 cableengine.go:92] Not installing cable for local cluster
  3244. 2020-10-08T11:45:06.6606270Z [cluster1] I1008 11:37:55.765690 1 tunnel.go:108] Tunnel controller successfully installed Endpoint cable submariner-cable-cluster1-172-17-0-9 in the engine
  3245. 2020-10-08T11:45:06.6613502Z [cluster1] I1008 11:37:55.765700 1 tunnel.go:95] Tunnel controller processing added or updated submariner Endpoint object: &v1.Endpoint{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"cluster2-submariner-cable-cluster2-172-17-0-5", GenerateName:"", Namespace:"submariner-operator", SelfLink:"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster2-submariner-cable-cluster2-172-17-0-5", UID:"474bcd76-5748-4372-9832-4a15a4b1804d", ResourceVersion:"1332", Generation:1, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63737753726, loc:(*time.Location)(0x225ef80)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v1.EndpointSpec{ClusterID:"cluster2", CableName:"submariner-cable-cluster2-172-17-0-5", Hostname:"cluster2-worker", Subnets:[]string{"100.92.0.0/16", "10.242.0.0/16"}, PrivateIP:"172.17.0.5", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  3246. 2020-10-08T11:45:06.6619703Z [cluster1] I1008 11:37:55.765734 1 cableengine.go:101] Installing Endpoint cable "submariner-cable-cluster2-172-17-0-5"
  3247. 2020-10-08T11:45:06.6621825Z [cluster1] I1008 11:37:55.767270 1 kubernetes.go:317] In SetCluster: &types.SubmarinerCluster{ID:"cluster1", Spec:v1.ClusterSpec{ClusterID:"cluster1", ColorCodes:[]string{"blue"}, ServiceCIDR:[]string{"100.91.0.0/16"}, ClusterCIDR:[]string{"10.241.0.0/16"}, GlobalCIDR:[]string{}}}
  3248. 2020-10-08T11:45:06.6624085Z [lighthouse]$ [cluster1] kubectl get pods --selector=app=submariner-routeagent -n submariner-operator -o jsonpath={.items[*].metadata.name}
  3249. 2020-10-08T11:45:06.6625631Z [lighthouse]$ [cluster1] kubectl get pods --selector=app=submariner-routeagent -n submariner-operator -o jsonpath={.items[*].metadata.name}
  3250. 2020-10-08T11:45:06.6628294Z [lighthouse]$ [cluster1] command kubectl --context=cluster1 get pods --selector=app=submariner-routeagent -n submariner-operator -o jsonpath={.items[*].metadata.name}
  3251. 2020-10-08T11:45:06.6629990Z [lighthouse]$ [cluster1] kubectl --context=cluster1 get pods --selector=app=submariner-routeagent -n submariner-operator -o jsonpath={.items[*].metadata.name}
  3252. 2020-10-08T11:45:06.6636540Z [cluster1] I1008 11:37:55.768853 1 datastoresyncer.go:287] Processing local submariner Endpoint object: &v1.Endpoint{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"cluster1-submariner-cable-cluster1-172-17-0-9", GenerateName:"", Namespace:"submariner-operator", SelfLink:"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster1-submariner-cable-cluster1-172-17-0-9", UID:"23aee4db-4473-4b17-8f51-aedd32e3a5ff", ResourceVersion:"1212", Generation:1, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63737753696, loc:(*time.Location)(0x225ef80)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v1.EndpointSpec{ClusterID:"cluster1", CableName:"submariner-cable-cluster1-172-17-0-9", Hostname:"cluster1-worker", Subnets:[]string{"100.91.0.0/16", "10.241.0.0/16"}, PrivateIP:"172.17.0.9", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  3253. 2020-10-08T11:45:06.6643404Z [cluster1] I1008 11:37:55.769897 1 kubernetes.go:370] In SetEndpoint: &types.SubmarinerEndpoint{Spec:v1.EndpointSpec{ClusterID:"cluster1", CableName:"submariner-cable-cluster1-172-17-0-9", Hostname:"cluster1-worker", Subnets:[]string{"100.91.0.0/16", "10.241.0.0/16"}, PrivateIP:"172.17.0.9", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  3254. 2020-10-08T11:45:06.6645803Z [cluster1] I1008 11:37:55.769880 1 cableengine.go:128] Cable "submariner-cable-cluster2-172-17-0-5" is already installed - not installing again
  3255. 2020-10-08T11:45:06.6647648Z [cluster1] I1008 11:37:55.770371 1 tunnel.go:108] Tunnel controller successfully installed Endpoint cable submariner-cable-cluster2-172-17-0-5 in the engine
  3256. 2020-10-08T11:45:06.6649074Z [cluster1] I1008 11:37:55.771496 1 kubernetes.go:345] Cluster "cluster1" matched what we received from datastore - not updating
  3257. 2020-10-08T11:45:06.6650682Z [cluster1] I1008 11:37:55.772574 1 kubernetes.go:398] Endpoint "cluster1-submariner-cable-cluster1-172-17-0-9" matched what we received from datastore - not updating
  3258. 2020-10-08T11:45:06.6652711Z [cluster1] I1008 11:37:55.774376 1 datastoresyncer.go:271] The updated submariner Endpoint "cluster2" is not for this cluster - skipping updating the datastore
  3259. 2020-10-08T11:45:06.6653632Z [cluster1] I1008 11:37:56.835722 1 kubernetes.go:180] UpdateFunc in WatchClusters called
  3260. 2020-10-08T11:45:06.6654368Z [cluster1] I1008 11:37:56.835751 1 kubernetes.go:180] UpdateFunc in WatchClusters called
  3261. 2020-10-08T11:45:06.6656524Z [cluster1] I1008 11:37:56.835764 1 datastoresyncer.go:312] In reconcileClusterCRD: &types.SubmarinerCluster{ID:"cluster2", Spec:v1.ClusterSpec{ClusterID:"cluster2", ColorCodes:[]string{"blue"}, ServiceCIDR:[]string{"100.92.0.0/16"}, ClusterCIDR:[]string{"10.242.0.0/16"}, GlobalCIDR:[]string{}}}
  3262. 2020-10-08T11:45:06.6658756Z [cluster1] I1008 11:37:56.839039 1 datastoresyncer.go:360] Cluster "cluster2" matched what we received from datastore - not updating
  3263. 2020-10-08T11:45:06.6659613Z [cluster1] I1008 11:37:56.840638 1 kubernetes.go:264] UpdateFunc in WatchEndpoints called
  3264. 2020-10-08T11:45:06.6660326Z [cluster1] I1008 11:37:56.840653 1 kubernetes.go:264] UpdateFunc in WatchEndpoints called
  3265. 2020-10-08T11:45:06.6663515Z [cluster1] I1008 11:37:56.840662 1 datastoresyncer.go:387] In reconcileEndpointCRD: &types.SubmarinerEndpoint{Spec:v1.EndpointSpec{ClusterID:"cluster2", CableName:"submariner-cable-cluster2-172-17-0-5", Hostname:"cluster2-worker", Subnets:[]string{"100.92.0.0/16", "10.242.0.0/16"}, PrivateIP:"172.17.0.5", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  3266. 2020-10-08T11:45:06.6666441Z [cluster1] I1008 11:37:56.842397 1 datastoresyncer.go:435] Endpoint "cluster2-submariner-cable-cluster2-172-17-0-5" matched what we received from datastore - not updating
  3267. 2020-10-08T11:45:06.6667715Z [cluster1] 04[KNL] interface vethwepl4c7a9ee deactivated
  3268. 2020-10-08T11:45:06.6668295Z [cluster1] 05[KNL] interface vethwepl4c7a9ee deleted
  3269. 2020-10-08T11:45:06.6668919Z [cluster1] I1008 11:38:26.835871 1 kubernetes.go:180] UpdateFunc in WatchClusters called
  3270. 2020-10-08T11:45:06.6669610Z [cluster1] I1008 11:38:26.835891 1 kubernetes.go:180] UpdateFunc in WatchClusters called
  3271. 2020-10-08T11:45:06.6671134Z [cluster1] I1008 11:38:26.835911 1 datastoresyncer.go:312] In reconcileClusterCRD: &types.SubmarinerCluster{ID:"cluster2", Spec:v1.ClusterSpec{ClusterID:"cluster2", ColorCodes:[]string{"blue"}, ServiceCIDR:[]string{"100.92.0.0/16"}, ClusterCIDR:[]string{"10.242.0.0/16"}, GlobalCIDR:[]string{}}}
  3272. 2020-10-08T11:45:06.6672877Z [cluster1] I1008 11:38:26.840732 1 kubernetes.go:264] UpdateFunc in WatchEndpoints called
  3273. 2020-10-08T11:45:06.6673589Z [cluster1] I1008 11:38:26.840788 1 kubernetes.go:264] UpdateFunc in WatchEndpoints called
  3274. 2020-10-08T11:45:06.6676122Z [cluster1] I1008 11:38:26.840807 1 datastoresyncer.go:387] In reconcileEndpointCRD: &types.SubmarinerEndpoint{Spec:v1.EndpointSpec{ClusterID:"cluster2", CableName:"submariner-cable-cluster2-172-17-0-5", Hostname:"cluster2-worker", Subnets:[]string{"100.92.0.0/16", "10.242.0.0/16"}, PrivateIP:"172.17.0.5", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  3275. 2020-10-08T11:45:06.6678471Z [cluster1] I1008 11:38:26.842272 1 datastoresyncer.go:360] Cluster "cluster2" matched what we received from datastore - not updating
  3276. 2020-10-08T11:45:06.6680432Z [cluster1] I1008 11:38:26.844494 1 datastoresyncer.go:435] Endpoint "cluster2-submariner-cable-cluster2-172-17-0-5" matched what we received from datastore - not updating
  3277. 2020-10-08T11:45:06.6688789Z [cluster1] I1008 11:38:55.766042 1 tunnel.go:95] Tunnel controller processing added or updated submariner Endpoint object: &v1.Endpoint{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"cluster1-submariner-cable-cluster1-172-17-0-9", GenerateName:"", Namespace:"submariner-operator", SelfLink:"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster1-submariner-cable-cluster1-172-17-0-9", UID:"23aee4db-4473-4b17-8f51-aedd32e3a5ff", ResourceVersion:"1212", Generation:1, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63737753696, loc:(*time.Location)(0x225ef80)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v1.EndpointSpec{ClusterID:"cluster1", CableName:"submariner-cable-cluster1-172-17-0-9", Hostname:"cluster1-worker", Subnets:[]string{"100.91.0.0/16", "10.241.0.0/16"}, PrivateIP:"172.17.0.9", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  3278. 2020-10-08T11:45:06.6694743Z [cluster1] I1008 11:38:55.766141 1 cableengine.go:92] Not installing cable for local cluster
  3279. 2020-10-08T11:45:06.6696178Z [cluster1] I1008 11:38:55.766288 1 tunnel.go:108] Tunnel controller successfully installed Endpoint cable submariner-cable-cluster1-172-17-0-9 in the engine
  3280. 2020-10-08T11:45:06.6703608Z [cluster1] I1008 11:38:55.766380 1 tunnel.go:95] Tunnel controller processing added or updated submariner Endpoint object: &v1.Endpoint{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"cluster2-submariner-cable-cluster2-172-17-0-5", GenerateName:"", Namespace:"submariner-operator", SelfLink:"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster2-submariner-cable-cluster2-172-17-0-5", UID:"474bcd76-5748-4372-9832-4a15a4b1804d", ResourceVersion:"1332", Generation:1, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63737753726, loc:(*time.Location)(0x225ef80)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v1.EndpointSpec{ClusterID:"cluster2", CableName:"submariner-cable-cluster2-172-17-0-5", Hostname:"cluster2-worker", Subnets:[]string{"100.92.0.0/16", "10.242.0.0/16"}, PrivateIP:"172.17.0.5", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  3281. 2020-10-08T11:45:06.6709441Z [cluster1] I1008 11:38:55.766508 1 cableengine.go:101] Installing Endpoint cable "submariner-cable-cluster2-172-17-0-5"
  3282. 2020-10-08T11:45:06.6711042Z [cluster1] I1008 11:38:55.767583 1 kubernetes.go:317] In SetCluster: &types.SubmarinerCluster{ID:"cluster1", Spec:v1.ClusterSpec{ClusterID:"cluster1", ColorCodes:[]string{"blue"}, ServiceCIDR:[]string{"100.91.0.0/16"}, ClusterCIDR:[]string{"10.241.0.0/16"}, GlobalCIDR:[]string{}}}
  3283. 2020-10-08T11:45:06.6718907Z [cluster1] I1008 11:38:55.769013 1 datastoresyncer.go:287] Processing local submariner Endpoint object: &v1.Endpoint{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"cluster1-submariner-cable-cluster1-172-17-0-9", GenerateName:"", Namespace:"submariner-operator", SelfLink:"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster1-submariner-cable-cluster1-172-17-0-9", UID:"23aee4db-4473-4b17-8f51-aedd32e3a5ff", ResourceVersion:"1212", Generation:1, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63737753696, loc:(*time.Location)(0x225ef80)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v1.EndpointSpec{ClusterID:"cluster1", CableName:"submariner-cable-cluster1-172-17-0-9", Hostname:"cluster1-worker", Subnets:[]string{"100.91.0.0/16", "10.241.0.0/16"}, PrivateIP:"172.17.0.9", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  3284. 2020-10-08T11:45:06.6727825Z [cluster1] I1008 11:38:55.769062 1 kubernetes.go:370] In SetEndpoint: &types.SubmarinerEndpoint{Spec:v1.EndpointSpec{ClusterID:"cluster1", CableName:"submariner-cable-cluster1-172-17-0-9", Hostname:"cluster1-worker", Subnets:[]string{"100.91.0.0/16", "10.241.0.0/16"}, PrivateIP:"172.17.0.9", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  3285. 2020-10-08T11:45:06.6730189Z [cluster1] I1008 11:38:55.771150 1 kubernetes.go:345] Cluster "cluster1" matched what we received from datastore - not updating
  3286. 2020-10-08T11:45:06.6731697Z [cluster1] I1008 11:38:55.772107 1 cableengine.go:128] Cable "submariner-cable-cluster2-172-17-0-5" is already installed - not installing again
  3287. 2020-10-08T11:45:06.6733690Z [cluster1] I1008 11:38:55.772163 1 tunnel.go:108] Tunnel controller successfully installed Endpoint cable submariner-cable-cluster2-172-17-0-5 in the engine
  3288. 2020-10-08T11:45:06.6736381Z [cluster1] I1008 11:38:55.773519 1 kubernetes.go:398] Endpoint "cluster1-submariner-cable-cluster1-172-17-0-9" matched what we received from datastore - not updating
  3289. 2020-10-08T11:45:06.6738253Z [cluster1] I1008 11:38:55.775290 1 datastoresyncer.go:271] The updated submariner Endpoint "cluster2" is not for this cluster - skipping updating the datastore
  3290. 2020-10-08T11:45:06.6739237Z [cluster1] I1008 11:38:56.836008 1 kubernetes.go:180] UpdateFunc in WatchClusters called
  3291. 2020-10-08T11:45:06.6740022Z [cluster1] I1008 11:38:56.836042 1 kubernetes.go:180] UpdateFunc in WatchClusters called
  3292. 2020-10-08T11:45:06.6742188Z [cluster1] I1008 11:38:56.836056 1 datastoresyncer.go:312] In reconcileClusterCRD: &types.SubmarinerCluster{ID:"cluster2", Spec:v1.ClusterSpec{ClusterID:"cluster2", ColorCodes:[]string{"blue"}, ServiceCIDR:[]string{"100.92.0.0/16"}, ClusterCIDR:[]string{"10.242.0.0/16"}, GlobalCIDR:[]string{}}}
  3293. 2020-10-08T11:45:06.6744168Z [cluster1] I1008 11:38:56.838253 1 datastoresyncer.go:360] Cluster "cluster2" matched what we received from datastore - not updating
  3294. 2020-10-08T11:45:06.6745071Z [cluster1] I1008 11:38:56.840856 1 kubernetes.go:264] UpdateFunc in WatchEndpoints called
  3295. 2020-10-08T11:45:06.6745828Z [cluster1] I1008 11:38:56.840879 1 kubernetes.go:264] UpdateFunc in WatchEndpoints called
  3296. 2020-10-08T11:45:06.6748645Z [cluster1] I1008 11:38:56.840900 1 datastoresyncer.go:387] In reconcileEndpointCRD: &types.SubmarinerEndpoint{Spec:v1.EndpointSpec{ClusterID:"cluster2", CableName:"submariner-cable-cluster2-172-17-0-5", Hostname:"cluster2-worker", Subnets:[]string{"100.92.0.0/16", "10.242.0.0/16"}, PrivateIP:"172.17.0.5", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  3297. 2020-10-08T11:45:06.6751783Z [cluster1] I1008 11:38:56.843107 1 datastoresyncer.go:435] Endpoint "cluster2-submariner-cable-cluster2-172-17-0-5" matched what we received from datastore - not updating
  3298. 2020-10-08T11:45:06.6753640Z [cluster1] 10[KNL] interface vethwepg2c525b8 deleted
  3299. 2020-10-08T11:45:06.6754252Z [cluster1] 16[KNL] interface vethwepl2c525b8 activated
  3300. 2020-10-08T11:45:06.6754991Z [cluster1] I1008 11:39:26.836137 1 kubernetes.go:180] UpdateFunc in WatchClusters called
  3301. 2020-10-08T11:45:06.6755883Z [cluster1] I1008 11:39:26.836163 1 kubernetes.go:180] UpdateFunc in WatchClusters called
  3302. 2020-10-08T11:45:06.6757410Z [cluster1] I1008 11:39:26.836178 1 datastoresyncer.go:312] In reconcileClusterCRD: &types.SubmarinerCluster{ID:"cluster2", Spec:v1.ClusterSpec{ClusterID:"cluster2", ColorCodes:[]string{"blue"}, ServiceCIDR:[]string{"100.92.0.0/16"}, ClusterCIDR:[]string{"10.242.0.0/16"}, GlobalCIDR:[]string{}}}
  3303. 2020-10-08T11:45:06.6759011Z [cluster1] I1008 11:39:26.841040 1 kubernetes.go:264] UpdateFunc in WatchEndpoints called
  3304. 2020-10-08T11:45:06.6759876Z [cluster1] I1008 11:39:26.841072 1 kubernetes.go:264] UpdateFunc in WatchEndpoints called
  3305. 2020-10-08T11:45:06.6763348Z [cluster1] I1008 11:39:26.841085 1 datastoresyncer.go:387] In reconcileEndpointCRD: &types.SubmarinerEndpoint{Spec:v1.EndpointSpec{ClusterID:"cluster2", CableName:"submariner-cable-cluster2-172-17-0-5", Hostname:"cluster2-worker", Subnets:[]string{"100.92.0.0/16", "10.242.0.0/16"}, PrivateIP:"172.17.0.5", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  3306. 2020-10-08T11:45:06.6766309Z [cluster1] I1008 11:39:26.842842 1 datastoresyncer.go:360] Cluster "cluster2" matched what we received from datastore - not updating
  3307. 2020-10-08T11:45:06.6768199Z [cluster1] I1008 11:39:26.844903 1 datastoresyncer.go:435] Endpoint "cluster2-submariner-cable-cluster2-172-17-0-5" matched what we received from datastore - not updating
  3308. 2020-10-08T11:45:06.6776539Z [cluster1] I1008 11:39:55.766162 1 tunnel.go:95] Tunnel controller processing added or updated submariner Endpoint object: &v1.Endpoint{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"cluster1-submariner-cable-cluster1-172-17-0-9", GenerateName:"", Namespace:"submariner-operator", SelfLink:"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster1-submariner-cable-cluster1-172-17-0-9", UID:"23aee4db-4473-4b17-8f51-aedd32e3a5ff", ResourceVersion:"1212", Generation:1, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63737753696, loc:(*time.Location)(0x225ef80)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v1.EndpointSpec{ClusterID:"cluster1", CableName:"submariner-cable-cluster1-172-17-0-9", Hostname:"cluster1-worker", Subnets:[]string{"100.91.0.0/16", "10.241.0.0/16"}, PrivateIP:"172.17.0.9", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  3309. 2020-10-08T11:45:06.6783167Z [cluster1] I1008 11:39:55.766259 1 cableengine.go:92] Not installing cable for local cluster
  3310. 2020-10-08T11:45:06.6784733Z [cluster1] I1008 11:39:55.766271 1 tunnel.go:108] Tunnel controller successfully installed Endpoint cable submariner-cable-cluster1-172-17-0-9 in the engine
  3311. 2020-10-08T11:45:06.6792049Z [cluster1] I1008 11:39:55.766281 1 tunnel.go:95] Tunnel controller processing added or updated submariner Endpoint object: &v1.Endpoint{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"cluster2-submariner-cable-cluster2-172-17-0-5", GenerateName:"", Namespace:"submariner-operator", SelfLink:"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster2-submariner-cable-cluster2-172-17-0-5", UID:"474bcd76-5748-4372-9832-4a15a4b1804d", ResourceVersion:"1332", Generation:1, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63737753726, loc:(*time.Location)(0x225ef80)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v1.EndpointSpec{ClusterID:"cluster2", CableName:"submariner-cable-cluster2-172-17-0-5", Hostname:"cluster2-worker", Subnets:[]string{"100.92.0.0/16", "10.242.0.0/16"}, PrivateIP:"172.17.0.5", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  3312. 2020-10-08T11:45:06.6799228Z [cluster1] I1008 11:39:55.766318 1 cableengine.go:101] Installing Endpoint cable "submariner-cable-cluster2-172-17-0-5"
  3313. 2020-10-08T11:45:06.6802249Z [cluster1] I1008 11:39:55.768439 1 kubernetes.go:317] In SetCluster: &types.SubmarinerCluster{ID:"cluster1", Spec:v1.ClusterSpec{ClusterID:"cluster1", ColorCodes:[]string{"blue"}, ServiceCIDR:[]string{"100.91.0.0/16"}, ClusterCIDR:[]string{"10.241.0.0/16"}, GlobalCIDR:[]string{}}}
  3314. 2020-10-08T11:45:06.6809422Z [cluster1] I1008 11:39:55.769214 1 datastoresyncer.go:287] Processing local submariner Endpoint object: &v1.Endpoint{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"cluster1-submariner-cable-cluster1-172-17-0-9", GenerateName:"", Namespace:"submariner-operator", SelfLink:"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster1-submariner-cable-cluster1-172-17-0-9", UID:"23aee4db-4473-4b17-8f51-aedd32e3a5ff", ResourceVersion:"1212", Generation:1, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63737753696, loc:(*time.Location)(0x225ef80)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v1.EndpointSpec{ClusterID:"cluster1", CableName:"submariner-cable-cluster1-172-17-0-9", Hostname:"cluster1-worker", Subnets:[]string{"100.91.0.0/16", "10.241.0.0/16"}, PrivateIP:"172.17.0.9", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  3315. 2020-10-08T11:45:06.6817718Z [cluster1] I1008 11:39:55.769345 1 kubernetes.go:370] In SetEndpoint: &types.SubmarinerEndpoint{Spec:v1.EndpointSpec{ClusterID:"cluster1", CableName:"submariner-cable-cluster1-172-17-0-9", Hostname:"cluster1-worker", Subnets:[]string{"100.91.0.0/16", "10.241.0.0/16"}, PrivateIP:"172.17.0.9", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  3316. 2020-10-08T11:45:06.6820113Z [cluster1] I1008 11:39:55.773608 1 kubernetes.go:345] Cluster "cluster1" matched what we received from datastore - not updating
  3317. 2020-10-08T11:45:06.6822772Z [cluster1] I1008 11:39:55.775577 1 kubernetes.go:398] Endpoint "cluster1-submariner-cable-cluster1-172-17-0-9" matched what we received from datastore - not updating
  3318. 2020-10-08T11:45:06.6824859Z [cluster1] I1008 11:39:55.776813 1 cableengine.go:128] Cable "submariner-cable-cluster2-172-17-0-5" is already installed - not installing again
  3319. 2020-10-08T11:45:06.6826731Z [cluster1] I1008 11:39:55.776932 1 tunnel.go:108] Tunnel controller successfully installed Endpoint cable submariner-cable-cluster2-172-17-0-5 in the engine
  3320. 2020-10-08T11:45:06.6828994Z [cluster1] I1008 11:39:55.779702 1 datastoresyncer.go:271] The updated submariner Endpoint "cluster2" is not for this cluster - skipping updating the datastore
  3321. 2020-10-08T11:45:06.6830205Z [cluster1] I1008 11:39:56.836456 1 kubernetes.go:180] UpdateFunc in WatchClusters called
  3322. 2020-10-08T11:45:06.6831101Z [cluster1] I1008 11:39:56.836542 1 kubernetes.go:180] UpdateFunc in WatchClusters called
  3323. 2020-10-08T11:45:06.6834533Z [cluster1] I1008 11:39:56.836554 1 datastoresyncer.go:312] In reconcileClusterCRD: &types.SubmarinerCluster{ID:"cluster2", Spec:v1.ClusterSpec{ClusterID:"cluster2", ColorCodes:[]string{"blue"}, ServiceCIDR:[]string{"100.92.0.0/16"}, ClusterCIDR:[]string{"10.242.0.0/16"}, GlobalCIDR:[]string{}}}
  3324. 2020-10-08T11:45:06.6836965Z [cluster1] I1008 11:39:56.839831 1 datastoresyncer.go:360] Cluster "cluster2" matched what we received from datastore - not updating
  3325. 2020-10-08T11:45:06.6838138Z [cluster1] I1008 11:39:56.841225 1 kubernetes.go:264] UpdateFunc in WatchEndpoints called
  3326. 2020-10-08T11:45:06.6838916Z [cluster1] I1008 11:39:56.841312 1 kubernetes.go:264] UpdateFunc in WatchEndpoints called
  3327. 2020-10-08T11:45:06.6842405Z [cluster1] I1008 11:39:56.841324 1 datastoresyncer.go:387] In reconcileEndpointCRD: &types.SubmarinerEndpoint{Spec:v1.EndpointSpec{ClusterID:"cluster2", CableName:"submariner-cable-cluster2-172-17-0-5", Hostname:"cluster2-worker", Subnets:[]string{"100.92.0.0/16", "10.242.0.0/16"}, PrivateIP:"172.17.0.5", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  3328. 2020-10-08T11:45:06.6845708Z [cluster1] I1008 11:39:56.843328 1 datastoresyncer.go:435] Endpoint "cluster2-submariner-cable-cluster2-172-17-0-5" matched what we received from datastore - not updating
  3329. 2020-10-08T11:45:06.6847124Z [cluster1] I1008 11:40:26.836589 1 kubernetes.go:180] UpdateFunc in WatchClusters called
  3330. 2020-10-08T11:45:06.6847884Z [cluster1] I1008 11:40:26.836640 1 kubernetes.go:180] UpdateFunc in WatchClusters called
  3331. 2020-10-08T11:45:06.6850121Z [cluster1] I1008 11:40:26.836659 1 datastoresyncer.go:312] In reconcileClusterCRD: &types.SubmarinerCluster{ID:"cluster2", Spec:v1.ClusterSpec{ClusterID:"cluster2", ColorCodes:[]string{"blue"}, ServiceCIDR:[]string{"100.92.0.0/16"}, ClusterCIDR:[]string{"10.242.0.0/16"}, GlobalCIDR:[]string{}}}
  3332. 2020-10-08T11:45:06.6851948Z [cluster1] I1008 11:40:26.840064 1 datastoresyncer.go:360] Cluster "cluster2" matched what we received from datastore - not updating
  3333. 2020-10-08T11:45:06.6854143Z [cluster1] I1008 11:40:26.841313 1 kubernetes.go:264] UpdateFunc in WatchEndpoints called
  3334. 2020-10-08T11:45:06.6854874Z [cluster1] I1008 11:40:26.841325 1 kubernetes.go:264] UpdateFunc in WatchEndpoints called
  3335. 2020-10-08T11:45:06.6858256Z [cluster1] I1008 11:40:26.841331 1 datastoresyncer.go:387] In reconcileEndpointCRD: &types.SubmarinerEndpoint{Spec:v1.EndpointSpec{ClusterID:"cluster2", CableName:"submariner-cable-cluster2-172-17-0-5", Hostname:"cluster2-worker", Subnets:[]string{"100.92.0.0/16", "10.242.0.0/16"}, PrivateIP:"172.17.0.5", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  3336. 2020-10-08T11:45:06.6862926Z [cluster1] I1008 11:40:26.843343 1 datastoresyncer.go:435] Endpoint "cluster2-submariner-cable-cluster2-172-17-0-5" matched what we received from datastore - not updating
  3337. 2020-10-08T11:45:06.6870923Z [cluster1] I1008 11:40:55.766442 1 tunnel.go:95] Tunnel controller processing added or updated submariner Endpoint object: &v1.Endpoint{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"cluster1-submariner-cable-cluster1-172-17-0-9", GenerateName:"", Namespace:"submariner-operator", SelfLink:"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster1-submariner-cable-cluster1-172-17-0-9", UID:"23aee4db-4473-4b17-8f51-aedd32e3a5ff", ResourceVersion:"1212", Generation:1, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63737753696, loc:(*time.Location)(0x225ef80)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v1.EndpointSpec{ClusterID:"cluster1", CableName:"submariner-cable-cluster1-172-17-0-9", Hostname:"cluster1-worker", Subnets:[]string{"100.91.0.0/16", "10.241.0.0/16"}, PrivateIP:"172.17.0.9", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  3338. 2020-10-08T11:45:06.6877035Z [cluster1] I1008 11:40:55.766663 1 cableengine.go:92] Not installing cable for local cluster
  3339. 2020-10-08T11:45:06.6878546Z [cluster1] I1008 11:40:55.766704 1 tunnel.go:108] Tunnel controller successfully installed Endpoint cable submariner-cable-cluster1-172-17-0-9 in the engine
  3340. 2020-10-08T11:45:06.6900408Z [cluster1] I1008 11:40:55.766727 1 tunnel.go:95] Tunnel controller processing added or updated submariner Endpoint object: &v1.Endpoint{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"cluster2-submariner-cable-cluster2-172-17-0-5", GenerateName:"", Namespace:"submariner-operator", SelfLink:"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster2-submariner-cable-cluster2-172-17-0-5", UID:"474bcd76-5748-4372-9832-4a15a4b1804d", ResourceVersion:"1332", Generation:1, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63737753726, loc:(*time.Location)(0x225ef80)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v1.EndpointSpec{ClusterID:"cluster2", CableName:"submariner-cable-cluster2-172-17-0-5", Hostname:"cluster2-worker", Subnets:[]string{"100.92.0.0/16", "10.242.0.0/16"}, PrivateIP:"172.17.0.5", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  3341. 2020-10-08T11:45:06.6913624Z [cluster1] I1008 11:40:55.766809 1 cableengine.go:101] Installing Endpoint cable "submariner-cable-cluster2-172-17-0-5"
  3342. 2020-10-08T11:45:06.6915258Z [cluster1] I1008 11:40:55.768138 1 kubernetes.go:317] In SetCluster: &types.SubmarinerCluster{ID:"cluster1", Spec:v1.ClusterSpec{ClusterID:"cluster1", ColorCodes:[]string{"blue"}, ServiceCIDR:[]string{"100.91.0.0/16"}, ClusterCIDR:[]string{"10.241.0.0/16"}, GlobalCIDR:[]string{}}}
  3343. 2020-10-08T11:45:06.6924480Z [cluster1] I1008 11:40:55.768648 1 datastoresyncer.go:287] Processing local submariner Endpoint object: &v1.Endpoint{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"cluster1-submariner-cable-cluster1-172-17-0-9", GenerateName:"", Namespace:"submariner-operator", SelfLink:"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster1-submariner-cable-cluster1-172-17-0-9", UID:"23aee4db-4473-4b17-8f51-aedd32e3a5ff", ResourceVersion:"1212", Generation:1, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63737753696, loc:(*time.Location)(0x225ef80)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v1.EndpointSpec{ClusterID:"cluster1", CableName:"submariner-cable-cluster1-172-17-0-9", Hostname:"cluster1-worker", Subnets:[]string{"100.91.0.0/16", "10.241.0.0/16"}, PrivateIP:"172.17.0.9", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  3344. 2020-10-08T11:45:06.6931249Z [cluster1] I1008 11:40:55.768696 1 kubernetes.go:370] In SetEndpoint: &types.SubmarinerEndpoint{Spec:v1.EndpointSpec{ClusterID:"cluster1", CableName:"submariner-cable-cluster1-172-17-0-9", Hostname:"cluster1-worker", Subnets:[]string{"100.91.0.0/16", "10.241.0.0/16"}, PrivateIP:"172.17.0.9", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  3345. 2020-10-08T11:45:06.6935575Z [cluster1] I1008 11:40:55.771761 1 kubernetes.go:345] Cluster "cluster1" matched what we received from datastore - not updating
  3346. 2020-10-08T11:45:06.6937132Z [cluster1] I1008 11:40:55.772977 1 kubernetes.go:398] Endpoint "cluster1-submariner-cable-cluster1-172-17-0-9" matched what we received from datastore - not updating
  3347. 2020-10-08T11:45:06.6938659Z [cluster1] I1008 11:40:55.774538 1 datastoresyncer.go:271] The updated submariner Endpoint "cluster2" is not for this cluster - skipping updating the datastore
  3348. 2020-10-08T11:45:06.6940018Z [cluster1] I1008 11:40:55.775107 1 cableengine.go:128] Cable "submariner-cable-cluster2-172-17-0-5" is already installed - not installing again
  3349. 2020-10-08T11:45:06.6942135Z [cluster1] I1008 11:40:55.775125 1 tunnel.go:108] Tunnel controller successfully installed Endpoint cable submariner-cable-cluster2-172-17-0-5 in the engine
  3350. 2020-10-08T11:45:06.6943111Z [cluster1] I1008 11:40:56.836913 1 kubernetes.go:180] UpdateFunc in WatchClusters called
  3351. 2020-10-08T11:45:06.6943705Z [cluster1] I1008 11:40:56.836954 1 kubernetes.go:180] UpdateFunc in WatchClusters called
  3352. 2020-10-08T11:45:06.6977551Z [cluster1] I1008 11:40:56.836988 1 datastoresyncer.go:312] In reconcileClusterCRD: &types.SubmarinerCluster{ID:"cluster2", Spec:v1.ClusterSpec{ClusterID:"cluster2", ColorCodes:[]string{"blue"}, ServiceCIDR:[]string{"100.92.0.0/16"}, ClusterCIDR:[]string{"10.242.0.0/16"}, GlobalCIDR:[]string{}}}
  3353. 2020-10-08T11:45:06.6979970Z [cluster1] I1008 11:40:56.840740 1 datastoresyncer.go:360] Cluster "cluster2" matched what we received from datastore - not updating
  3354. 2020-10-08T11:45:06.6980818Z [cluster1] I1008 11:40:56.841495 1 kubernetes.go:264] UpdateFunc in WatchEndpoints called
  3355. 2020-10-08T11:45:06.6991804Z [cluster1] I1008 11:40:56.842167 1 kubernetes.go:264] UpdateFunc in WatchEndpoints called
  3356. 2020-10-08T11:45:06.6994171Z [cluster1] I1008 11:40:56.842178 1 datastoresyncer.go:387] In reconcileEndpointCRD: &types.SubmarinerEndpoint{Spec:v1.EndpointSpec{ClusterID:"cluster2", CableName:"submariner-cable-cluster2-172-17-0-5", Hostname:"cluster2-worker", Subnets:[]string{"100.92.0.0/16", "10.242.0.0/16"}, PrivateIP:"172.17.0.5", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  3357. 2020-10-08T11:45:06.6997085Z [cluster1] I1008 11:40:56.844143 1 datastoresyncer.go:435] Endpoint "cluster2-submariner-cable-cluster2-172-17-0-5" matched what we received from datastore - not updating
  3358. 2020-10-08T11:45:06.6998394Z [cluster1] I1008 11:41:26.837052 1 kubernetes.go:180] UpdateFunc in WatchClusters called
  3359. 2020-10-08T11:45:06.6999921Z [cluster1] I1008 11:41:26.837078 1 datastoresyncer.go:312] In reconcileClusterCRD: &types.SubmarinerCluster{ID:"cluster2", Spec:v1.ClusterSpec{ClusterID:"cluster2", ColorCodes:[]string{"blue"}, ServiceCIDR:[]string{"100.92.0.0/16"}, ClusterCIDR:[]string{"10.242.0.0/16"}, GlobalCIDR:[]string{}}}
  3360. 2020-10-08T11:45:06.7008045Z [cluster1] I1008 11:41:26.839328 1 datastoresyncer.go:360] Cluster "cluster2" matched what we received from datastore - not updating
  3361. 2020-10-08T11:45:06.7008794Z [cluster1] I1008 11:41:26.839350 1 kubernetes.go:180] UpdateFunc in WatchClusters called
  3362. 2020-10-08T11:45:06.7009437Z [cluster1] I1008 11:41:26.841485 1 kubernetes.go:264] UpdateFunc in WatchEndpoints called
  3363. 2020-10-08T11:45:06.7010059Z [cluster1] I1008 11:41:26.841499 1 kubernetes.go:264] UpdateFunc in WatchEndpoints called
  3364. 2020-10-08T11:45:06.7012238Z [cluster1] I1008 11:41:26.841505 1 datastoresyncer.go:387] In reconcileEndpointCRD: &types.SubmarinerEndpoint{Spec:v1.EndpointSpec{ClusterID:"cluster2", CableName:"submariner-cable-cluster2-172-17-0-5", Hostname:"cluster2-worker", Subnets:[]string{"100.92.0.0/16", "10.242.0.0/16"}, PrivateIP:"172.17.0.5", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  3365. 2020-10-08T11:45:06.7014833Z [cluster1] I1008 11:41:26.843564 1 datastoresyncer.go:435] Endpoint "cluster2-submariner-cable-cluster2-172-17-0-5" matched what we received from datastore - not updating
  3366. 2020-10-08T11:45:06.7022617Z [cluster1] I1008 11:41:55.766790 1 tunnel.go:95] Tunnel controller processing added or updated submariner Endpoint object: &v1.Endpoint{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"cluster1-submariner-cable-cluster1-172-17-0-9", GenerateName:"", Namespace:"submariner-operator", SelfLink:"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster1-submariner-cable-cluster1-172-17-0-9", UID:"23aee4db-4473-4b17-8f51-aedd32e3a5ff", ResourceVersion:"1212", Generation:1, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63737753696, loc:(*time.Location)(0x225ef80)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v1.EndpointSpec{ClusterID:"cluster1", CableName:"submariner-cable-cluster1-172-17-0-9", Hostname:"cluster1-worker", Subnets:[]string{"100.91.0.0/16", "10.241.0.0/16"}, PrivateIP:"172.17.0.9", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  3367. 2020-10-08T11:45:06.7028104Z [cluster1] I1008 11:41:55.767480 1 cableengine.go:92] Not installing cable for local cluster
  3368. 2020-10-08T11:45:06.7029533Z [cluster1] I1008 11:41:55.767988 1 tunnel.go:108] Tunnel controller successfully installed Endpoint cable submariner-cable-cluster1-172-17-0-9 in the engine
  3369. 2020-10-08T11:45:06.7036456Z [cluster1] I1008 11:41:55.768337 1 tunnel.go:95] Tunnel controller processing added or updated submariner Endpoint object: &v1.Endpoint{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"cluster2-submariner-cable-cluster2-172-17-0-5", GenerateName:"", Namespace:"submariner-operator", SelfLink:"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster2-submariner-cable-cluster2-172-17-0-5", UID:"474bcd76-5748-4372-9832-4a15a4b1804d", ResourceVersion:"1332", Generation:1, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63737753726, loc:(*time.Location)(0x225ef80)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v1.EndpointSpec{ClusterID:"cluster2", CableName:"submariner-cable-cluster2-172-17-0-5", Hostname:"cluster2-worker", Subnets:[]string{"100.92.0.0/16", "10.242.0.0/16"}, PrivateIP:"172.17.0.5", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  3370. 2020-10-08T11:45:06.7043656Z [cluster1] I1008 11:41:55.768913 1 cableengine.go:101] Installing Endpoint cable "submariner-cable-cluster2-172-17-0-5"
  3371. 2020-10-08T11:45:06.7045368Z [cluster1] I1008 11:41:55.769103 1 kubernetes.go:317] In SetCluster: &types.SubmarinerCluster{ID:"cluster1", Spec:v1.ClusterSpec{ClusterID:"cluster1", ColorCodes:[]string{"blue"}, ServiceCIDR:[]string{"100.91.0.0/16"}, ClusterCIDR:[]string{"10.241.0.0/16"}, GlobalCIDR:[]string{}}}
  3372. 2020-10-08T11:45:06.7053320Z [cluster1] I1008 11:41:55.769368 1 datastoresyncer.go:287] Processing local submariner Endpoint object: &v1.Endpoint{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"cluster1-submariner-cable-cluster1-172-17-0-9", GenerateName:"", Namespace:"submariner-operator", SelfLink:"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster1-submariner-cable-cluster1-172-17-0-9", UID:"23aee4db-4473-4b17-8f51-aedd32e3a5ff", ResourceVersion:"1212", Generation:1, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63737753696, loc:(*time.Location)(0x225ef80)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v1.EndpointSpec{ClusterID:"cluster1", CableName:"submariner-cable-cluster1-172-17-0-9", Hostname:"cluster1-worker", Subnets:[]string{"100.91.0.0/16", "10.241.0.0/16"}, PrivateIP:"172.17.0.9", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  3373. 2020-10-08T11:45:06.7060319Z [cluster1] I1008 11:41:55.770028 1 kubernetes.go:370] In SetEndpoint: &types.SubmarinerEndpoint{Spec:v1.EndpointSpec{ClusterID:"cluster1", CableName:"submariner-cable-cluster1-172-17-0-9", Hostname:"cluster1-worker", Subnets:[]string{"100.91.0.0/16", "10.241.0.0/16"}, PrivateIP:"172.17.0.9", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  3374. 2020-10-08T11:45:06.7063611Z [cluster1] I1008 11:41:55.773577 1 kubernetes.go:398] Endpoint "cluster1-submariner-cable-cluster1-172-17-0-9" matched what we received from datastore - not updating
  3375. 2020-10-08T11:45:06.7064988Z [cluster1] I1008 11:41:55.774369 1 kubernetes.go:345] Cluster "cluster1" matched what we received from datastore - not updating
  3376. 2020-10-08T11:45:06.7066077Z [cluster1] I1008 11:41:55.776052 1 datastoresyncer.go:271] The updated submariner Endpoint "cluster2" is not for this cluster - skipping updating the datastore
  3377. 2020-10-08T11:45:06.7067492Z [cluster1] I1008 11:41:55.777397 1 cableengine.go:128] Cable "submariner-cable-cluster2-172-17-0-5" is already installed - not installing again
  3378. 2020-10-08T11:45:06.7068999Z [cluster1] I1008 11:41:55.777411 1 tunnel.go:108] Tunnel controller successfully installed Endpoint cable submariner-cable-cluster2-172-17-0-5 in the engine
  3379. 2020-10-08T11:45:06.7069969Z [cluster1] I1008 11:41:56.837281 1 kubernetes.go:180] UpdateFunc in WatchClusters called
  3380. 2020-10-08T11:45:06.7070578Z [cluster1] I1008 11:41:56.837310 1 kubernetes.go:180] UpdateFunc in WatchClusters called
  3381. 2020-10-08T11:45:06.7071808Z [cluster1] I1008 11:41:56.837334 1 datastoresyncer.go:312] In reconcileClusterCRD: &types.SubmarinerCluster{ID:"cluster2", Spec:v1.ClusterSpec{ClusterID:"cluster2", ColorCodes:[]string{"blue"}, ServiceCIDR:[]string{"100.92.0.0/16"}, ClusterCIDR:[]string{"10.242.0.0/16"}, GlobalCIDR:[]string{}}}
  3382. 2020-10-08T11:45:06.7073340Z [cluster1] I1008 11:41:56.841729 1 datastoresyncer.go:360] Cluster "cluster2" matched what we received from datastore - not updating
  3383. 2020-10-08T11:45:06.7074071Z [cluster1] I1008 11:41:56.841573 1 kubernetes.go:264] UpdateFunc in WatchEndpoints called
  3384. 2020-10-08T11:45:06.7074695Z [cluster1] I1008 11:41:56.841778 1 kubernetes.go:264] UpdateFunc in WatchEndpoints called
  3385. 2020-10-08T11:45:06.7077234Z [cluster1] I1008 11:41:56.841789 1 datastoresyncer.go:387] In reconcileEndpointCRD: &types.SubmarinerEndpoint{Spec:v1.EndpointSpec{ClusterID:"cluster2", CableName:"submariner-cable-cluster2-172-17-0-5", Hostname:"cluster2-worker", Subnets:[]string{"100.92.0.0/16", "10.242.0.0/16"}, PrivateIP:"172.17.0.5", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  3386. 2020-10-08T11:45:06.7080106Z [cluster1] I1008 11:41:56.843761 1 datastoresyncer.go:435] Endpoint "cluster2-submariner-cable-cluster2-172-17-0-5" matched what we received from datastore - not updating
  3387. 2020-10-08T11:45:06.7082429Z [cluster1] I1008 11:42:26.837363 1 kubernetes.go:180] UpdateFunc in WatchClusters called
  3388. 2020-10-08T11:45:06.7083152Z [cluster1] I1008 11:42:26.837422 1 kubernetes.go:180] UpdateFunc in WatchClusters called
  3389. 2020-10-08T11:45:06.7084551Z [cluster1] I1008 11:42:26.837438 1 datastoresyncer.go:312] In reconcileClusterCRD: &types.SubmarinerCluster{ID:"cluster2", Spec:v1.ClusterSpec{ClusterID:"cluster2", ColorCodes:[]string{"blue"}, ServiceCIDR:[]string{"100.92.0.0/16"}, ClusterCIDR:[]string{"10.242.0.0/16"}, GlobalCIDR:[]string{}}}
  3390. 2020-10-08T11:45:06.7086361Z [cluster1] I1008 11:42:26.839780 1 datastoresyncer.go:360] Cluster "cluster2" matched what we received from datastore - not updating
  3391. 2020-10-08T11:45:06.7087245Z [cluster1] I1008 11:42:26.842173 1 kubernetes.go:264] UpdateFunc in WatchEndpoints called
  3392. 2020-10-08T11:45:06.7087953Z [cluster1] I1008 11:42:26.842193 1 kubernetes.go:264] UpdateFunc in WatchEndpoints called
  3393. 2020-10-08T11:45:06.7090692Z [cluster1] I1008 11:42:26.842199 1 datastoresyncer.go:387] In reconcileEndpointCRD: &types.SubmarinerEndpoint{Spec:v1.EndpointSpec{ClusterID:"cluster2", CableName:"submariner-cable-cluster2-172-17-0-5", Hostname:"cluster2-worker", Subnets:[]string{"100.92.0.0/16", "10.242.0.0/16"}, PrivateIP:"172.17.0.5", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  3394. 2020-10-08T11:45:06.7093892Z [cluster1] I1008 11:42:26.843808 1 datastoresyncer.go:435] Endpoint "cluster2-submariner-cable-cluster2-172-17-0-5" matched what we received from datastore - not updating
  3395. 2020-10-08T11:45:06.7095254Z [cluster1] 14[KNL] interface vethwepl2c525b8 deactivated
  3396. 2020-10-08T11:45:06.7095873Z [cluster1] 15[KNL] interface vethwepl2c525b8 deleted
  3397. 2020-10-08T11:45:06.7104412Z [cluster1] I1008 11:42:55.767060 1 tunnel.go:95] Tunnel controller processing added or updated submariner Endpoint object: &v1.Endpoint{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"cluster1-submariner-cable-cluster1-172-17-0-9", GenerateName:"", Namespace:"submariner-operator", SelfLink:"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster1-submariner-cable-cluster1-172-17-0-9", UID:"23aee4db-4473-4b17-8f51-aedd32e3a5ff", ResourceVersion:"1212", Generation:1, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63737753696, loc:(*time.Location)(0x225ef80)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v1.EndpointSpec{ClusterID:"cluster1", CableName:"submariner-cable-cluster1-172-17-0-9", Hostname:"cluster1-worker", Subnets:[]string{"100.91.0.0/16", "10.241.0.0/16"}, PrivateIP:"172.17.0.9", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  3398. 2020-10-08T11:45:06.7110515Z [cluster1] I1008 11:42:55.767328 1 cableengine.go:92] Not installing cable for local cluster
  3399. 2020-10-08T11:45:06.7112153Z [cluster1] I1008 11:42:55.767511 1 tunnel.go:108] Tunnel controller successfully installed Endpoint cable submariner-cable-cluster1-172-17-0-9 in the engine
  3400. 2020-10-08T11:45:06.7119560Z [cluster1] I1008 11:42:55.767675 1 tunnel.go:95] Tunnel controller processing added or updated submariner Endpoint object: &v1.Endpoint{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"cluster2-submariner-cable-cluster2-172-17-0-5", GenerateName:"", Namespace:"submariner-operator", SelfLink:"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster2-submariner-cable-cluster2-172-17-0-5", UID:"474bcd76-5748-4372-9832-4a15a4b1804d", ResourceVersion:"1332", Generation:1, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63737753726, loc:(*time.Location)(0x225ef80)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v1.EndpointSpec{ClusterID:"cluster2", CableName:"submariner-cable-cluster2-172-17-0-5", Hostname:"cluster2-worker", Subnets:[]string{"100.92.0.0/16", "10.242.0.0/16"}, PrivateIP:"172.17.0.5", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  3401. 2020-10-08T11:45:06.7126575Z [cluster1] I1008 11:42:55.767856 1 cableengine.go:101] Installing Endpoint cable "submariner-cable-cluster2-172-17-0-5"
  3402. 2020-10-08T11:45:06.7128406Z [cluster1] I1008 11:42:55.768570 1 kubernetes.go:317] In SetCluster: &types.SubmarinerCluster{ID:"cluster1", Spec:v1.ClusterSpec{ClusterID:"cluster1", ColorCodes:[]string{"blue"}, ServiceCIDR:[]string{"100.91.0.0/16"}, ClusterCIDR:[]string{"10.241.0.0/16"}, GlobalCIDR:[]string{}}}
  3403. 2020-10-08T11:45:06.7136368Z [cluster1] I1008 11:42:55.770453 1 datastoresyncer.go:287] Processing local submariner Endpoint object: &v1.Endpoint{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"cluster1-submariner-cable-cluster1-172-17-0-9", GenerateName:"", Namespace:"submariner-operator", SelfLink:"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster1-submariner-cable-cluster1-172-17-0-9", UID:"23aee4db-4473-4b17-8f51-aedd32e3a5ff", ResourceVersion:"1212", Generation:1, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63737753696, loc:(*time.Location)(0x225ef80)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v1.EndpointSpec{ClusterID:"cluster1", CableName:"submariner-cable-cluster1-172-17-0-9", Hostname:"cluster1-worker", Subnets:[]string{"100.91.0.0/16", "10.241.0.0/16"}, PrivateIP:"172.17.0.9", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  3404. 2020-10-08T11:45:06.7144787Z [cluster1] I1008 11:42:55.770502 1 kubernetes.go:370] In SetEndpoint: &types.SubmarinerEndpoint{Spec:v1.EndpointSpec{ClusterID:"cluster1", CableName:"submariner-cable-cluster1-172-17-0-9", Hostname:"cluster1-worker", Subnets:[]string{"100.91.0.0/16", "10.241.0.0/16"}, PrivateIP:"172.17.0.9", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  3405. 2020-10-08T11:45:06.7147111Z [cluster1] I1008 11:42:55.771371 1 kubernetes.go:345] Cluster "cluster1" matched what we received from datastore - not updating
  3406. 2020-10-08T11:45:06.7148611Z [cluster1] I1008 11:42:55.772784 1 cableengine.go:128] Cable "submariner-cable-cluster2-172-17-0-5" is already installed - not installing again
  3407. 2020-10-08T11:45:06.7150478Z [cluster1] I1008 11:42:55.772802 1 tunnel.go:108] Tunnel controller successfully installed Endpoint cable submariner-cable-cluster2-172-17-0-5 in the engine
  3408. 2020-10-08T11:45:06.7152573Z [cluster1] I1008 11:42:55.772790 1 kubernetes.go:398] Endpoint "cluster1-submariner-cable-cluster1-172-17-0-9" matched what we received from datastore - not updating
  3409. 2020-10-08T11:45:06.7154426Z [cluster1] I1008 11:42:55.774781 1 datastoresyncer.go:271] The updated submariner Endpoint "cluster2" is not for this cluster - skipping updating the datastore
  3410. 2020-10-08T11:45:06.7155408Z [cluster1] I1008 11:42:56.837551 1 kubernetes.go:180] UpdateFunc in WatchClusters called
  3411. 2020-10-08T11:45:06.7156156Z [cluster1] I1008 11:42:56.837590 1 kubernetes.go:180] UpdateFunc in WatchClusters called
  3412. 2020-10-08T11:45:06.7157676Z [cluster1] I1008 11:42:56.837602 1 datastoresyncer.go:312] In reconcileClusterCRD: &types.SubmarinerCluster{ID:"cluster2", Spec:v1.ClusterSpec{ClusterID:"cluster2", ColorCodes:[]string{"blue"}, ServiceCIDR:[]string{"100.92.0.0/16"}, ClusterCIDR:[]string{"10.242.0.0/16"}, GlobalCIDR:[]string{}}}
  3413. 2020-10-08T11:45:06.7160914Z [cluster1] I1008 11:42:56.840558 1 datastoresyncer.go:360] Cluster "cluster2" matched what we received from datastore - not updating
  3414. 2020-10-08T11:45:06.7162503Z [cluster1] I1008 11:42:56.842288 1 kubernetes.go:264] UpdateFunc in WatchEndpoints called
  3415. 2020-10-08T11:45:06.7163277Z [cluster1] I1008 11:42:56.842304 1 kubernetes.go:264] UpdateFunc in WatchEndpoints called
  3416. 2020-10-08T11:45:06.7166239Z [cluster1] I1008 11:42:56.842310 1 datastoresyncer.go:387] In reconcileEndpointCRD: &types.SubmarinerEndpoint{Spec:v1.EndpointSpec{ClusterID:"cluster2", CableName:"submariner-cable-cluster2-172-17-0-5", Hostname:"cluster2-worker", Subnets:[]string{"100.92.0.0/16", "10.242.0.0/16"}, PrivateIP:"172.17.0.5", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  3417. 2020-10-08T11:45:06.7169531Z [cluster1] I1008 11:42:56.844209 1 datastoresyncer.go:435] Endpoint "cluster2-submariner-cable-cluster2-172-17-0-5" matched what we received from datastore - not updating
  3418. 2020-10-08T11:45:06.7171204Z [cluster1] 12[KNL] interface vethwepgf2f5998 deleted
  3419. 2020-10-08T11:45:06.7171901Z [cluster1] 07[KNL] interface vethweplf2f5998 activated
  3420. 2020-10-08T11:45:06.7172742Z [cluster1] I1008 11:43:26.837724 1 kubernetes.go:180] UpdateFunc in WatchClusters called
  3421. 2020-10-08T11:45:06.7173569Z [cluster1] I1008 11:43:26.840052 1 kubernetes.go:180] UpdateFunc in WatchClusters called
  3422. 2020-10-08T11:45:06.7174972Z [cluster1] I1008 11:43:26.840137 1 datastoresyncer.go:312] In reconcileClusterCRD: &types.SubmarinerCluster{ID:"cluster2", Spec:v1.ClusterSpec{ClusterID:"cluster2", ColorCodes:[]string{"blue"}, ServiceCIDR:[]string{"100.92.0.0/16"}, ClusterCIDR:[]string{"10.242.0.0/16"}, GlobalCIDR:[]string{}}}
  3423. 2020-10-08T11:45:06.7177062Z [cluster1] I1008 11:43:26.842132 1 datastoresyncer.go:360] Cluster "cluster2" matched what we received from datastore - not updating
  3424. 2020-10-08T11:45:06.7178078Z [cluster1] I1008 11:43:26.842396 1 kubernetes.go:264] UpdateFunc in WatchEndpoints called
  3425. 2020-10-08T11:45:06.7178848Z [cluster1] I1008 11:43:26.842444 1 kubernetes.go:264] UpdateFunc in WatchEndpoints called
  3426. 2020-10-08T11:45:06.7182070Z [cluster1] I1008 11:43:26.842456 1 datastoresyncer.go:387] In reconcileEndpointCRD: &types.SubmarinerEndpoint{Spec:v1.EndpointSpec{ClusterID:"cluster2", CableName:"submariner-cable-cluster2-172-17-0-5", Hostname:"cluster2-worker", Subnets:[]string{"100.92.0.0/16", "10.242.0.0/16"}, PrivateIP:"172.17.0.5", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  3427. 2020-10-08T11:45:06.7185248Z [cluster1] I1008 11:43:26.844588 1 datastoresyncer.go:435] Endpoint "cluster2-submariner-cable-cluster2-172-17-0-5" matched what we received from datastore - not updating
  3428. 2020-10-08T11:45:06.7186519Z [cluster1] 09[KNL] interface vethweplf2f5998 deactivated
  3429. 2020-10-08T11:45:06.7187095Z [cluster1] 11[KNL] interface vethweplf2f5998 deleted
  3430. 2020-10-08T11:45:06.7193671Z [cluster1] I1008 11:43:55.767337 1 tunnel.go:95] Tunnel controller processing added or updated submariner Endpoint object: &v1.Endpoint{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"cluster1-submariner-cable-cluster1-172-17-0-9", GenerateName:"", Namespace:"submariner-operator", SelfLink:"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster1-submariner-cable-cluster1-172-17-0-9", UID:"23aee4db-4473-4b17-8f51-aedd32e3a5ff", ResourceVersion:"1212", Generation:1, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63737753696, loc:(*time.Location)(0x225ef80)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v1.EndpointSpec{ClusterID:"cluster1", CableName:"submariner-cable-cluster1-172-17-0-9", Hostname:"cluster1-worker", Subnets:[]string{"100.91.0.0/16", "10.241.0.0/16"}, PrivateIP:"172.17.0.9", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  3431. 2020-10-08T11:45:06.7199493Z [cluster1] I1008 11:43:55.767442 1 cableengine.go:92] Not installing cable for local cluster
  3432. 2020-10-08T11:45:06.7200973Z [cluster1] I1008 11:43:55.767706 1 tunnel.go:108] Tunnel controller successfully installed Endpoint cable submariner-cable-cluster1-172-17-0-9 in the engine
  3433. 2020-10-08T11:45:06.7209267Z [cluster1] I1008 11:43:55.768064 1 tunnel.go:95] Tunnel controller processing added or updated submariner Endpoint object: &v1.Endpoint{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"cluster2-submariner-cable-cluster2-172-17-0-5", GenerateName:"", Namespace:"submariner-operator", SelfLink:"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster2-submariner-cable-cluster2-172-17-0-5", UID:"474bcd76-5748-4372-9832-4a15a4b1804d", ResourceVersion:"1332", Generation:1, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63737753726, loc:(*time.Location)(0x225ef80)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v1.EndpointSpec{ClusterID:"cluster2", CableName:"submariner-cable-cluster2-172-17-0-5", Hostname:"cluster2-worker", Subnets:[]string{"100.92.0.0/16", "10.242.0.0/16"}, PrivateIP:"172.17.0.5", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  3434. 2020-10-08T11:45:06.7215391Z [cluster1] I1008 11:43:55.768411 1 cableengine.go:101] Installing Endpoint cable "submariner-cable-cluster2-172-17-0-5"
  3435. 2020-10-08T11:45:06.7216987Z [cluster1] I1008 11:43:55.768918 1 kubernetes.go:317] In SetCluster: &types.SubmarinerCluster{ID:"cluster1", Spec:v1.ClusterSpec{ClusterID:"cluster1", ColorCodes:[]string{"blue"}, ServiceCIDR:[]string{"100.91.0.0/16"}, ClusterCIDR:[]string{"10.241.0.0/16"}, GlobalCIDR:[]string{}}}
  3436. 2020-10-08T11:45:06.7224829Z [cluster1] I1008 11:43:55.771794 1 datastoresyncer.go:287] Processing local submariner Endpoint object: &v1.Endpoint{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"cluster1-submariner-cable-cluster1-172-17-0-9", GenerateName:"", Namespace:"submariner-operator", SelfLink:"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster1-submariner-cable-cluster1-172-17-0-9", UID:"23aee4db-4473-4b17-8f51-aedd32e3a5ff", ResourceVersion:"1212", Generation:1, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63737753696, loc:(*time.Location)(0x225ef80)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v1.EndpointSpec{ClusterID:"cluster1", CableName:"submariner-cable-cluster1-172-17-0-9", Hostname:"cluster1-worker", Subnets:[]string{"100.91.0.0/16", "10.241.0.0/16"}, PrivateIP:"172.17.0.9", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  3437. 2020-10-08T11:45:06.7232111Z [cluster1] I1008 11:43:55.771863 1 kubernetes.go:370] In SetEndpoint: &types.SubmarinerEndpoint{Spec:v1.EndpointSpec{ClusterID:"cluster1", CableName:"submariner-cable-cluster1-172-17-0-9", Hostname:"cluster1-worker", Subnets:[]string{"100.91.0.0/16", "10.241.0.0/16"}, PrivateIP:"172.17.0.9", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  3438. 2020-10-08T11:45:06.7234587Z [cluster1] I1008 11:43:55.773180 1 cableengine.go:128] Cable "submariner-cable-cluster2-172-17-0-5" is already installed - not installing again
  3439. 2020-10-08T11:45:06.7236331Z [cluster1] I1008 11:43:55.773198 1 tunnel.go:108] Tunnel controller successfully installed Endpoint cable submariner-cable-cluster2-172-17-0-5 in the engine
  3440. 2020-10-08T11:45:06.7238122Z [cluster1] I1008 11:43:55.773308 1 kubernetes.go:345] Cluster "cluster1" matched what we received from datastore - not updating
  3441. 2020-10-08T11:45:06.7239743Z [cluster1] I1008 11:43:55.774058 1 kubernetes.go:398] Endpoint "cluster1-submariner-cable-cluster1-172-17-0-9" matched what we received from datastore - not updating
  3442. 2020-10-08T11:45:06.7242328Z [cluster1] I1008 11:43:55.775562 1 datastoresyncer.go:271] The updated submariner Endpoint "cluster2" is not for this cluster - skipping updating the datastore
  3443. 2020-10-08T11:45:06.7243249Z [cluster1] I1008 11:43:56.837841 1 kubernetes.go:180] UpdateFunc in WatchClusters called
  3444. 2020-10-08T11:45:06.7243946Z [cluster1] I1008 11:43:56.837869 1 kubernetes.go:180] UpdateFunc in WatchClusters called
  3445. 2020-10-08T11:45:06.7245444Z [cluster1] I1008 11:43:56.837876 1 datastoresyncer.go:312] In reconcileClusterCRD: &types.SubmarinerCluster{ID:"cluster2", Spec:v1.ClusterSpec{ClusterID:"cluster2", ColorCodes:[]string{"blue"}, ServiceCIDR:[]string{"100.92.0.0/16"}, ClusterCIDR:[]string{"10.242.0.0/16"}, GlobalCIDR:[]string{}}}
  3446. 2020-10-08T11:45:06.7247277Z [cluster1] I1008 11:43:56.841199 1 datastoresyncer.go:360] Cluster "cluster2" matched what we received from datastore - not updating
  3447. 2020-10-08T11:45:06.7248008Z [cluster1] I1008 11:43:56.842520 1 kubernetes.go:264] UpdateFunc in WatchEndpoints called
  3448. 2020-10-08T11:45:06.7248819Z [cluster1] I1008 11:43:56.842539 1 kubernetes.go:264] UpdateFunc in WatchEndpoints called
  3449. 2020-10-08T11:45:06.7251295Z [cluster1] I1008 11:43:56.842545 1 datastoresyncer.go:387] In reconcileEndpointCRD: &types.SubmarinerEndpoint{Spec:v1.EndpointSpec{ClusterID:"cluster2", CableName:"submariner-cable-cluster2-172-17-0-5", Hostname:"cluster2-worker", Subnets:[]string{"100.92.0.0/16", "10.242.0.0/16"}, PrivateIP:"172.17.0.5", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  3450. 2020-10-08T11:45:06.7254410Z [cluster1] I1008 11:43:56.844426 1 datastoresyncer.go:435] Endpoint "cluster2-submariner-cable-cluster2-172-17-0-5" matched what we received from datastore - not updating
  3451. 2020-10-08T11:45:06.7255657Z [cluster1] 14[KNL] interface vethwepgf4684dd deleted
  3452. 2020-10-08T11:45:06.7256218Z [cluster1] 15[KNL] interface vethweplf4684dd activated
  3453. 2020-10-08T11:45:06.7256853Z [cluster1] I1008 11:44:26.837974 1 kubernetes.go:180] UpdateFunc in WatchClusters called
  3454. 2020-10-08T11:45:06.7257530Z [cluster1] I1008 11:44:26.837995 1 kubernetes.go:180] UpdateFunc in WatchClusters called
  3455. 2020-10-08T11:45:06.7258929Z [cluster1] I1008 11:44:26.838012 1 datastoresyncer.go:312] In reconcileClusterCRD: &types.SubmarinerCluster{ID:"cluster2", Spec:v1.ClusterSpec{ClusterID:"cluster2", ColorCodes:[]string{"blue"}, ServiceCIDR:[]string{"100.92.0.0/16"}, ClusterCIDR:[]string{"10.242.0.0/16"}, GlobalCIDR:[]string{}}}
  3456. 2020-10-08T11:45:06.7260699Z [cluster1] I1008 11:44:26.840659 1 datastoresyncer.go:360] Cluster "cluster2" matched what we received from datastore - not updating
  3457. 2020-10-08T11:45:06.7262141Z [cluster1] I1008 11:44:26.842651 1 kubernetes.go:264] UpdateFunc in WatchEndpoints called
  3458. 2020-10-08T11:45:06.7262865Z [cluster1] I1008 11:44:26.842685 1 kubernetes.go:264] UpdateFunc in WatchEndpoints called
  3459. 2020-10-08T11:45:06.7266064Z [cluster1] I1008 11:44:26.842724 1 datastoresyncer.go:387] In reconcileEndpointCRD: &types.SubmarinerEndpoint{Spec:v1.EndpointSpec{ClusterID:"cluster2", CableName:"submariner-cable-cluster2-172-17-0-5", Hostname:"cluster2-worker", Subnets:[]string{"100.92.0.0/16", "10.242.0.0/16"}, PrivateIP:"172.17.0.5", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  3460. 2020-10-08T11:45:06.7269119Z [cluster1] I1008 11:44:26.844469 1 datastoresyncer.go:435] Endpoint "cluster2-submariner-cable-cluster2-172-17-0-5" matched what we received from datastore - not updating
  3461. 2020-10-08T11:45:06.7270393Z [cluster1] 08[KNL] interface vethweplf4684dd deactivated
  3462. 2020-10-08T11:45:06.7270968Z [cluster1] 09[KNL] interface vethweplf4684dd deleted
  3463. 2020-10-08T11:45:06.7278043Z [cluster1] I1008 11:44:55.767569 1 tunnel.go:95] Tunnel controller processing added or updated submariner Endpoint object: &v1.Endpoint{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"cluster1-submariner-cable-cluster1-172-17-0-9", GenerateName:"", Namespace:"submariner-operator", SelfLink:"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster1-submariner-cable-cluster1-172-17-0-9", UID:"23aee4db-4473-4b17-8f51-aedd32e3a5ff", ResourceVersion:"1212", Generation:1, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63737753696, loc:(*time.Location)(0x225ef80)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v1.EndpointSpec{ClusterID:"cluster1", CableName:"submariner-cable-cluster1-172-17-0-9", Hostname:"cluster1-worker", Subnets:[]string{"100.91.0.0/16", "10.241.0.0/16"}, PrivateIP:"172.17.0.9", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  3464. 2020-10-08T11:45:06.7284415Z [cluster1] I1008 11:44:55.767692 1 cableengine.go:92] Not installing cable for local cluster
  3465. 2020-10-08T11:45:06.7286132Z [cluster1] I1008 11:44:55.767701 1 tunnel.go:108] Tunnel controller successfully installed Endpoint cable submariner-cable-cluster1-172-17-0-9 in the engine
  3466. 2020-10-08T11:45:06.7293525Z [cluster1] I1008 11:44:55.767707 1 tunnel.go:95] Tunnel controller processing added or updated submariner Endpoint object: &v1.Endpoint{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"cluster2-submariner-cable-cluster2-172-17-0-5", GenerateName:"", Namespace:"submariner-operator", SelfLink:"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster2-submariner-cable-cluster2-172-17-0-5", UID:"474bcd76-5748-4372-9832-4a15a4b1804d", ResourceVersion:"1332", Generation:1, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63737753726, loc:(*time.Location)(0x225ef80)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v1.EndpointSpec{ClusterID:"cluster2", CableName:"submariner-cable-cluster2-172-17-0-5", Hostname:"cluster2-worker", Subnets:[]string{"100.92.0.0/16", "10.242.0.0/16"}, PrivateIP:"172.17.0.5", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  3467. 2020-10-08T11:45:06.7299368Z [cluster1] I1008 11:44:55.767744 1 cableengine.go:101] Installing Endpoint cable "submariner-cable-cluster2-172-17-0-5"
  3468. 2020-10-08T11:45:06.7301022Z [cluster1] I1008 11:44:55.769205 1 kubernetes.go:317] In SetCluster: &types.SubmarinerCluster{ID:"cluster1", Spec:v1.ClusterSpec{ClusterID:"cluster1", ColorCodes:[]string{"blue"}, ServiceCIDR:[]string{"100.91.0.0/16"}, ClusterCIDR:[]string{"10.241.0.0/16"}, GlobalCIDR:[]string{}}}
  3469. 2020-10-08T11:45:06.7303591Z [cluster1] I1008 11:44:55.771773 1 cableengine.go:128] Cable "submariner-cable-cluster2-172-17-0-5" is already installed - not installing again
  3470. 2020-10-08T11:45:06.7305323Z [cluster1] I1008 11:44:55.775802 1 tunnel.go:108] Tunnel controller successfully installed Endpoint cable submariner-cable-cluster2-172-17-0-5 in the engine
  3471. 2020-10-08T11:45:06.7312290Z [cluster1] I1008 11:44:55.773438 1 datastoresyncer.go:287] Processing local submariner Endpoint object: &v1.Endpoint{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"cluster1-submariner-cable-cluster1-172-17-0-9", GenerateName:"", Namespace:"submariner-operator", SelfLink:"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster1-submariner-cable-cluster1-172-17-0-9", UID:"23aee4db-4473-4b17-8f51-aedd32e3a5ff", ResourceVersion:"1212", Generation:1, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63737753696, loc:(*time.Location)(0x225ef80)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v1.EndpointSpec{ClusterID:"cluster1", CableName:"submariner-cable-cluster1-172-17-0-9", Hostname:"cluster1-worker", Subnets:[]string{"100.91.0.0/16", "10.241.0.0/16"}, PrivateIP:"172.17.0.9", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  3472. 2020-10-08T11:45:06.7319614Z [cluster1] I1008 11:44:55.775880 1 kubernetes.go:370] In SetEndpoint: &types.SubmarinerEndpoint{Spec:v1.EndpointSpec{ClusterID:"cluster1", CableName:"submariner-cable-cluster1-172-17-0-9", Hostname:"cluster1-worker", Subnets:[]string{"100.91.0.0/16", "10.241.0.0/16"}, PrivateIP:"172.17.0.9", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  3473. 2020-10-08T11:45:06.7323418Z [cluster1] I1008 11:44:55.776881 1 kubernetes.go:345] Cluster "cluster1" matched what we received from datastore - not updating
  3474. 2020-10-08T11:45:06.7325948Z [cluster1] I1008 11:44:55.777903 1 kubernetes.go:398] Endpoint "cluster1-submariner-cable-cluster1-172-17-0-9" matched what we received from datastore - not updating
  3475. 2020-10-08T11:45:06.7327936Z [cluster1] I1008 11:44:55.780895 1 datastoresyncer.go:271] The updated submariner Endpoint "cluster2" is not for this cluster - skipping updating the datastore
  3476. 2020-10-08T11:45:06.7328847Z [cluster1] I1008 11:44:56.839763 1 kubernetes.go:180] UpdateFunc in WatchClusters called
  3477. 2020-10-08T11:45:06.7329547Z [cluster1] I1008 11:44:56.839793 1 kubernetes.go:180] UpdateFunc in WatchClusters called
  3478. 2020-10-08T11:45:06.7331574Z [cluster1] I1008 11:44:56.839801 1 datastoresyncer.go:312] In reconcileClusterCRD: &types.SubmarinerCluster{ID:"cluster2", Spec:v1.ClusterSpec{ClusterID:"cluster2", ColorCodes:[]string{"blue"}, ServiceCIDR:[]string{"100.92.0.0/16"}, ClusterCIDR:[]string{"10.242.0.0/16"}, GlobalCIDR:[]string{}}}
  3479. 2020-10-08T11:45:06.7333204Z [cluster1] I1008 11:44:56.842729 1 kubernetes.go:264] UpdateFunc in WatchEndpoints called
  3480. 2020-10-08T11:45:06.7333979Z [cluster1] I1008 11:44:56.842754 1 kubernetes.go:264] UpdateFunc in WatchEndpoints called
  3481. 2020-10-08T11:45:06.7336711Z [cluster1] I1008 11:44:56.842765 1 datastoresyncer.go:387] In reconcileEndpointCRD: &types.SubmarinerEndpoint{Spec:v1.EndpointSpec{ClusterID:"cluster2", CableName:"submariner-cable-cluster2-172-17-0-5", Hostname:"cluster2-worker", Subnets:[]string{"100.92.0.0/16", "10.242.0.0/16"}, PrivateIP:"172.17.0.5", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  3482. 2020-10-08T11:45:07.3991916Z [lighthouse]$ [cluster1] kubectl -n submariner-operator logs submariner-routeagent-cljq7
  3483. 2020-10-08T11:45:07.4003493Z [lighthouse]$ [cluster1] kubectl -n submariner-operator logs submariner-routeagent-cljq7
  3484. 2020-10-08T11:45:07.4014839Z [lighthouse]$ [cluster1] command kubectl --context=cluster1 -n submariner-operator logs submariner-routeagent-cljq7
  3485. 2020-10-08T11:45:07.4026499Z [lighthouse]$ [cluster1] kubectl --context=cluster1 -n submariner-operator logs submariner-routeagent-cljq7
  3486. 2020-10-08T11:45:08.1588759Z [cluster1] I1008 11:44:56.848489 1 datastoresyncer.go:435] Endpoint "cluster2-submariner-cable-cluster2-172-17-0-5" matched what we received from datastore - not updating
  3487. 2020-10-08T11:45:08.1590790Z [cluster1] I1008 11:44:56.850104 1 datastoresyncer.go:360] Cluster "cluster2" matched what we received from datastore - not updating
  3488. 2020-10-08T11:45:08.1591524Z [cluster1] ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
  3489. 2020-10-08T11:45:08.1592557Z [cluster1] +++++++++++++++++++++: Logs for Pod submariner-routeagent-cljq7 in namespace submariner-operator :++++++++++++++++++++++
  3490. 2020-10-08T11:45:08.1593531Z [cluster1] + trap 'exit 1' SIGTERM SIGINT
  3491. 2020-10-08T11:45:08.1593977Z [cluster1] + SUBMARINER_VERBOSITY=2
  3492. 2020-10-08T11:45:08.1594521Z [cluster1] + '[' false == true ']'
  3493. 2020-10-08T11:45:08.1594986Z [cluster1] + DEBUG=-v=2
  3494. 2020-10-08T11:45:08.1595537Z [cluster1] + for f in iptables-save iptables
  3495. 2020-10-08T11:45:08.1596208Z [cluster1] ++ find_iptables_on_host iptables-save
  3496. 2020-10-08T11:45:08.1596914Z [cluster1] ++ chroot /host test -x /usr/sbin/iptables-save
  3497. 2020-10-08T11:45:08.1597363Z [cluster1] ++ echo /usr/sbin
  3498. 2020-10-08T11:45:08.1597692Z [cluster1] ++ return
  3499. 2020-10-08T11:45:08.1598041Z [cluster1] + location=/usr/sbin
  3500. 2020-10-08T11:45:08.1598578Z [cluster1] + '[' /usr/sbin '!=' unknown ']'
  3501. 2020-10-08T11:45:08.1599539Z [cluster1] + echo 'iptables-save is present on the host at /usr/sbin/iptables-save'
  3502. 2020-10-08T11:45:08.1600465Z [cluster1] + sed 's!@@PATH@@!/usr/sbin!' /usr/sbin/iptables-wrapper.in
  3503. 2020-10-08T11:45:08.1601335Z [cluster1] iptables-save is present on the host at /usr/sbin/iptables-save
  3504. 2020-10-08T11:45:08.1601983Z [cluster1] iptables is present on the host at /usr/sbin/iptables
  3505. 2020-10-08T11:45:08.1602865Z [cluster1] + for f in iptables-save iptables
  3506. 2020-10-08T11:45:08.1603351Z [cluster1] ++ find_iptables_on_host iptables
  3507. 2020-10-08T11:45:08.1603986Z [cluster1] ++ chroot /host test -x /usr/sbin/iptables
  3508. 2020-10-08T11:45:08.1604400Z [cluster1] ++ echo /usr/sbin
  3509. 2020-10-08T11:45:08.1604730Z [cluster1] ++ return
  3510. 2020-10-08T11:45:08.1605064Z [cluster1] + location=/usr/sbin
  3511. 2020-10-08T11:45:08.1605602Z [cluster1] + '[' /usr/sbin '!=' unknown ']'
  3512. 2020-10-08T11:45:08.1606280Z [cluster1] + echo 'iptables is present on the host at /usr/sbin/iptables'
  3513. 2020-10-08T11:45:08.1607102Z [cluster1] + sed 's!@@PATH@@!/usr/sbin!' /usr/sbin/iptables-wrapper.in
  3514. 2020-10-08T11:45:08.1608159Z [cluster1] + exec submariner-route-agent -v=2 -alsologtostderr
  3515. 2020-10-08T11:45:08.1609186Z [cluster1] I1008 11:35:11.400572 1 main.go:47] Starting submariner-route-agent
  3516. 2020-10-08T11:45:08.1610319Z [cluster1] W1008 11:35:11.405697 1 client_config.go:543] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work.
  3517. 2020-10-08T11:45:08.1611400Z [cluster1] I1008 11:35:11.416231 1 route.go:195] Starting Route Controller. ClusterID: cluster1, localClusterCIDR: [10.241.0.0/16], localServiceCIDR: [100.91.0.0/16]
  3518. 2020-10-08T11:45:08.1612265Z [cluster1] I1008 11:35:11.416292 1 route.go:199] Waiting for Endpoint informer caches to sync.
  3519. 2020-10-08T11:45:08.1617508Z [cluster1] I1008 11:35:11.457073 1 route.go:736] Enqueueing endpoint for route controller {"kind":"Endpoint","apiVersion":"submariner.io/v1","metadata":{"name":"cluster1-submariner-cable-cluster1-172-17-0-9","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster1-submariner-cable-cluster1-172-17-0-9","uid":"23aee4db-4473-4b17-8f51-aedd32e3a5ff","resourceVersion":"1212","generation":1,"creationTimestamp":"2020-10-08T11:34:56Z"},"spec":{"cluster_id":"cluster1","cable_name":"submariner-cable-cluster1-172-17-0-9","hostname":"cluster1-worker","subnets":["100.91.0.0/16","10.241.0.0/16"],"private_ip":"172.17.0.9","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
  3520. 2020-10-08T11:45:08.1621320Z [cluster1] I1008 11:35:11.465347 1 route.go:751] Enqueueing sm-route-agent-pod event, ip: 172.17.0.7
  3521. 2020-10-08T11:45:08.1622230Z [cluster1] I1008 11:35:11.465423 1 route.go:751] Enqueueing sm-route-agent-pod event, ip: 172.17.0.9
  3522. 2020-10-08T11:45:08.1622847Z [cluster1] I1008 11:35:11.516646 1 driver.go:37] Interface "lo" has "127.0.0.1" address
  3523. 2020-10-08T11:45:08.1623382Z [cluster1] I1008 11:35:11.516845 1 driver.go:37] Interface "weave" has "10.241.224.0" address
  3524. 2020-10-08T11:45:08.1624027Z [cluster1] I1008 11:35:11.516860 1 driver.go:42] Found CNI Interface "weave" that has IP "10.241.224.0" from ClusterCIDR "10.241.0.0/16"
  3525. 2020-10-08T11:45:08.1624854Z [cluster1] I1008 11:35:11.517875 1 driver.go:60] Successfully configured rp_filter to loose mode(2) on cniInterface "weave"
  3526. 2020-10-08T11:45:08.1625935Z [cluster1] I1008 11:35:11.522902 1 iptables.go:22] Install/ensure SUBMARINER-POSTROUTING chain exists
  3527. 2020-10-08T11:45:08.1627392Z [cluster1] I1008 11:35:11.547406 1 iptables.go:28] Insert SUBMARINER-POSTROUTING rule that has rules for inter-cluster traffic
  3528. 2020-10-08T11:45:08.1628466Z [cluster1] I1008 11:35:11.555708 1 iptables.go:34] Install/ensure SUBMARINER-INPUT chain exists
  3529. 2020-10-08T11:45:08.1629436Z [cluster1] I1008 11:35:11.574988 1 iptables.go:45] Allow VxLAN incoming traffic in SUBMARINER-INPUT Chain
  3530. 2020-10-08T11:45:08.1630500Z [cluster1] I1008 11:35:11.588604 1 iptables.go:53] Insert rule to allow traffic over vx-submariner interface in FORWARDing Chain
  3531. 2020-10-08T11:45:08.1631818Z [cluster1] I1008 11:35:11.609176 1 iptables.go:65] Installing rule for host network to remote cluster communication: -s 240.0.0.0/8 -o vx-submariner -j SNAT --to 10.241.224.0
  3532. 2020-10-08T11:45:08.1632961Z [cluster1] I1008 11:35:11.623704 1 route.go:257] In cluster1, podIP of submariner-route-agent[0] is 172.17.0.9
  3533. 2020-10-08T11:45:08.1633978Z [cluster1] I1008 11:35:11.623776 1 route.go:257] In cluster1, podIP of submariner-route-agent[1] is 172.17.0.7
  3534. 2020-10-08T11:45:08.1635088Z [cluster1] I1008 11:35:11.623805 1 route.go:271] Hostname is "cluster1-worker" and routeAgentNodeName is "cluster1-worker"
  3535. 2020-10-08T11:45:08.1636462Z [cluster1] I1008 11:35:11.641737 1 driver.go:87] Successfully annotated node "cluster1-worker" with cniIfaceIP "10.241.224.0"
  3536. 2020-10-08T11:45:08.1637282Z [cluster1] I1008 11:35:11.641834 1 route.go:288] Route agent workers started
  3537. 2020-10-08T11:45:08.1639058Z [cluster1] I1008 11:35:11.743330 1 route.go:634] Processing local Endpoint cluster1-submariner-cable-cluster1-172-17-0-9 with IP 172.17.0.9, Host cluster1-worker for this cluster
  3538. 2020-10-08T11:45:08.1640407Z [cluster1] I1008 11:35:11.756335 1 route.go:851] Processing route {Ifindex: 18 Dst: <nil> Src: <nil> Gw: 172.17.0.1 Flags: [] Table: 254}
  3539. 2020-10-08T11:45:08.1640972Z [cluster1] I1008 11:35:11.756588 1 route.go:854] Found nil gw or dst
  3540. 2020-10-08T11:45:08.1641536Z [cluster1] I1008 11:35:11.756666 1 route.go:851] Processing route {Ifindex: 4 Dst: 10.241.0.0/16 Src: 10.241.224.0 Gw: <nil> Flags: [] Table: 254}
  3541. 2020-10-08T11:45:08.1642234Z [cluster1] I1008 11:35:11.757153 1 route.go:854] Found nil gw or dst
  3542. 2020-10-08T11:45:08.1642795Z [cluster1] I1008 11:35:11.757289 1 route.go:851] Processing route {Ifindex: 18 Dst: 172.17.0.0/16 Src: 172.17.0.9 Gw: <nil> Flags: [] Table: 254}
  3543. 2020-10-08T11:45:08.1643347Z [cluster1] I1008 11:35:11.757376 1 route.go:854] Found nil gw or dst
  3544. 2020-10-08T11:45:08.1643890Z [cluster1] I1008 11:35:11.757464 1 route.go:645] This route agent is running on the active gateway node
  3545. 2020-10-08T11:45:08.1644833Z [cluster1] I1008 11:35:11.757548 1 route.go:653] Creating the vxlan interface: vx-submariner on the gateway node
  3546. 2020-10-08T11:45:08.1646126Z [cluster1] I1008 11:35:11.758587 1 route.go:540] Processing submariner-route-agent pod "submariner-routeagent-wmmbs" with host IP "172.17.0.7", pod IP "172.17.0.7"
  3547. 2020-10-08T11:45:08.1647333Z [cluster1] I1008 11:35:11.765167 1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.9 00:00:00:00:00:00
  3548. 2020-10-08T11:45:08.1649015Z [cluster1] I1008 11:35:11.768190 1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.7 00:00:00:00:00:00
  3549. 2020-10-08T11:45:08.1658564Z [cluster1] I1008 11:35:11.768727 1 route.go:478] Successfully configured rp_filter to loose mode(2) on vx-submariner
  3550. 2020-10-08T11:45:08.1659499Z [cluster1] I1008 11:35:11.769608 1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.7 00:00:00:00:00:00
  3551. 2020-10-08T11:45:08.1660867Z [cluster1] I1008 11:35:11.770471 1 route.go:558] FDB entry added on the Gateway node's vxlan iface for route-agent pod "submariner-routeagent-wmmbs", IP "172.17.0.7"
  3552. 2020-10-08T11:45:08.1662461Z [cluster1] I1008 11:35:11.770690 1 route.go:568] Successfully processed submariner-route-agent pod "submariner-routeagent-wmmbs"
  3553. 2020-10-08T11:45:08.1664051Z [cluster1] I1008 11:35:12.084643 1 route.go:540] Processing submariner-route-agent pod "submariner-routeagent-cljq7" with host IP "172.17.0.9", pod IP "172.17.0.9"
  3554. 2020-10-08T11:45:08.1665112Z [cluster1] I1008 11:35:12.084855 1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.9 00:00:00:00:00:00
  3555. 2020-10-08T11:45:08.1666118Z [cluster1] I1008 11:35:12.084950 1 route.go:558] FDB entry added on the Gateway node's vxlan iface for route-agent pod "submariner-routeagent-cljq7", IP "172.17.0.9"
  3556. 2020-10-08T11:45:08.1667519Z [cluster1] I1008 11:35:12.085008 1 route.go:568] Successfully processed submariner-route-agent pod "submariner-routeagent-cljq7"
  3557. 2020-10-08T11:45:08.1671420Z [cluster1] I1008 11:35:26.128887 1 route.go:736] Enqueueing endpoint for route controller {"metadata":{"name":"cluster2-submariner-cable-cluster2-172-17-0-5","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster2-submariner-cable-cluster2-172-17-0-5","uid":"474bcd76-5748-4372-9832-4a15a4b1804d","resourceVersion":"1332","generation":1,"creationTimestamp":"2020-10-08T11:35:26Z"},"spec":{"cluster_id":"cluster2","cable_name":"submariner-cable-cluster2-172-17-0-5","hostname":"cluster2-worker","subnets":["100.92.0.0/16","10.242.0.0/16"],"private_ip":"172.17.0.5","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
  3558. 2020-10-08T11:45:08.1675538Z [cluster1] I1008 11:35:26.146044 1 route.go:620] Updating iptable rules for remote cluster's Endpoint "cluster2-submariner-cable-cluster2-172-17-0-5" with subnets [100.92.0.0/16 10.242.0.0/16]
  3559. 2020-10-08T11:45:08.1677423Z [cluster1] I1008 11:35:26.150581 1 iptables.go:83] Installing iptables rule for outgoing traffic: -s 10.241.0.0/16 -d 100.92.0.0/16 -j ACCEPT
  3560. 2020-10-08T11:45:08.1678530Z [cluster1] I1008 11:35:26.168341 1 iptables.go:91] Installing iptables rule for incoming traffic: -s 100.92.0.0/16 -d 10.241.0.0/16 -j ACCEPT
  3561. 2020-10-08T11:45:08.1679623Z [cluster1] I1008 11:35:26.218814 1 iptables.go:83] Installing iptables rule for outgoing traffic: -s 10.241.0.0/16 -d 10.242.0.0/16 -j ACCEPT
  3562. 2020-10-08T11:45:08.1680710Z [cluster1] I1008 11:35:26.238609 1 iptables.go:91] Installing iptables rule for incoming traffic: -s 10.242.0.0/16 -d 10.241.0.0/16 -j ACCEPT
  3563. 2020-10-08T11:45:08.1685383Z [cluster1] I1008 11:35:41.457227 1 route.go:736] Enqueueing endpoint for route controller {"kind":"Endpoint","apiVersion":"submariner.io/v1","metadata":{"name":"cluster1-submariner-cable-cluster1-172-17-0-9","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster1-submariner-cable-cluster1-172-17-0-9","uid":"23aee4db-4473-4b17-8f51-aedd32e3a5ff","resourceVersion":"1212","generation":1,"creationTimestamp":"2020-10-08T11:34:56Z"},"spec":{"cluster_id":"cluster1","cable_name":"submariner-cable-cluster1-172-17-0-9","hostname":"cluster1-worker","subnets":["100.91.0.0/16","10.241.0.0/16"],"private_ip":"172.17.0.9","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
  3564. 2020-10-08T11:45:08.1694648Z [cluster1] I1008 11:35:41.457308 1 route.go:736] Enqueueing endpoint for route controller {"metadata":{"name":"cluster2-submariner-cable-cluster2-172-17-0-5","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster2-submariner-cable-cluster2-172-17-0-5","uid":"474bcd76-5748-4372-9832-4a15a4b1804d","resourceVersion":"1332","generation":1,"creationTimestamp":"2020-10-08T11:35:26Z"},"spec":{"cluster_id":"cluster2","cable_name":"submariner-cable-cluster2-172-17-0-5","hostname":"cluster2-worker","subnets":["100.92.0.0/16","10.242.0.0/16"],"private_ip":"172.17.0.5","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
  3565. 2020-10-08T11:45:08.1698897Z [cluster1] I1008 11:35:41.465078 1 route.go:634] Processing local Endpoint cluster1-submariner-cable-cluster1-172-17-0-9 with IP 172.17.0.9, Host cluster1-worker for this cluster
  3566. 2020-10-08T11:45:08.1700518Z [cluster1] I1008 11:35:41.465295 1 route.go:851] Processing route {Ifindex: 15 Dst: 240.0.0.0/8 Src: 240.17.0.9 Gw: <nil> Flags: [] Table: 254}
  3567. 2020-10-08T11:45:08.1701236Z [cluster1] I1008 11:35:41.465316 1 route.go:854] Found nil gw or dst
  3568. 2020-10-08T11:45:08.1701774Z [cluster1] I1008 11:35:41.465323 1 route.go:645] This route agent is running on the active gateway node
  3569. 2020-10-08T11:45:08.1702724Z [cluster1] I1008 11:35:41.465330 1 route.go:653] Creating the vxlan interface: vx-submariner on the gateway node
  3570. 2020-10-08T11:45:08.1703606Z [cluster1] I1008 11:35:41.465459 1 vxlan.go:61] VxLAN interface already exists with same configuration.
  3571. 2020-10-08T11:45:08.1704301Z [cluster1] I1008 11:35:41.465487 1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.7 00:00:00:00:00:00
  3572. 2020-10-08T11:45:08.1704910Z [cluster1] I1008 11:35:41.465511 1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.9 00:00:00:00:00:00
  3573. 2020-10-08T11:45:08.1705948Z [cluster1] I1008 11:35:41.465563 1 route.go:478] Successfully configured rp_filter to loose mode(2) on vx-submariner
  3574. 2020-10-08T11:45:08.1707695Z [cluster1] I1008 11:35:41.467666 1 route.go:620] Updating iptable rules for remote cluster's Endpoint "cluster2-submariner-cable-cluster2-172-17-0-5" with subnets [100.92.0.0/16 10.242.0.0/16]
  3575. 2020-10-08T11:45:08.1712659Z [cluster1] I1008 11:36:11.457368 1 route.go:736] Enqueueing endpoint for route controller {"kind":"Endpoint","apiVersion":"submariner.io/v1","metadata":{"name":"cluster1-submariner-cable-cluster1-172-17-0-9","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster1-submariner-cable-cluster1-172-17-0-9","uid":"23aee4db-4473-4b17-8f51-aedd32e3a5ff","resourceVersion":"1212","generation":1,"creationTimestamp":"2020-10-08T11:34:56Z"},"spec":{"cluster_id":"cluster1","cable_name":"submariner-cable-cluster1-172-17-0-9","hostname":"cluster1-worker","subnets":["100.91.0.0/16","10.241.0.0/16"],"private_ip":"172.17.0.9","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
  3576. 2020-10-08T11:45:08.1720415Z [cluster1] I1008 11:36:11.457422 1 route.go:736] Enqueueing endpoint for route controller {"metadata":{"name":"cluster2-submariner-cable-cluster2-172-17-0-5","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster2-submariner-cable-cluster2-172-17-0-5","uid":"474bcd76-5748-4372-9832-4a15a4b1804d","resourceVersion":"1332","generation":1,"creationTimestamp":"2020-10-08T11:35:26Z"},"spec":{"cluster_id":"cluster2","cable_name":"submariner-cable-cluster2-172-17-0-5","hostname":"cluster2-worker","subnets":["100.92.0.0/16","10.242.0.0/16"],"private_ip":"172.17.0.5","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
  3577. 2020-10-08T11:45:08.1726926Z [cluster1] I1008 11:36:11.464838 1 route.go:634] Processing local Endpoint cluster1-submariner-cable-cluster1-172-17-0-9 with IP 172.17.0.9, Host cluster1-worker for this cluster
  3578. 2020-10-08T11:45:08.1728217Z [cluster1] I1008 11:36:11.465011 1 route.go:851] Processing route {Ifindex: 15 Dst: 240.0.0.0/8 Src: 240.17.0.9 Gw: <nil> Flags: [] Table: 254}
  3579. 2020-10-08T11:45:08.1728828Z [cluster1] I1008 11:36:11.465031 1 route.go:854] Found nil gw or dst
  3580. 2020-10-08T11:45:08.1729388Z [cluster1] I1008 11:36:11.465039 1 route.go:645] This route agent is running on the active gateway node
  3581. 2020-10-08T11:45:08.1730386Z [cluster1] I1008 11:36:11.465047 1 route.go:653] Creating the vxlan interface: vx-submariner on the gateway node
  3582. 2020-10-08T11:45:08.1731209Z [cluster1] I1008 11:36:11.465166 1 vxlan.go:61] VxLAN interface already exists with same configuration.
  3583. 2020-10-08T11:45:08.1731952Z [cluster1] I1008 11:36:11.465196 1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.9 00:00:00:00:00:00
  3584. 2020-10-08T11:45:08.1732741Z [cluster1] I1008 11:36:11.465221 1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.7 00:00:00:00:00:00
  3585. 2020-10-08T11:45:08.1733758Z [cluster1] I1008 11:36:11.465356 1 route.go:478] Successfully configured rp_filter to loose mode(2) on vx-submariner
  3586. 2020-10-08T11:45:08.1737257Z [cluster1] I1008 11:36:11.467202 1 route.go:620] Updating iptable rules for remote cluster's Endpoint "cluster2-submariner-cable-cluster2-172-17-0-5" with subnets [100.92.0.0/16 10.242.0.0/16]
  3587. 2020-10-08T11:45:08.1742699Z [cluster1] I1008 11:36:41.457500 1 route.go:736] Enqueueing endpoint for route controller {"kind":"Endpoint","apiVersion":"submariner.io/v1","metadata":{"name":"cluster1-submariner-cable-cluster1-172-17-0-9","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster1-submariner-cable-cluster1-172-17-0-9","uid":"23aee4db-4473-4b17-8f51-aedd32e3a5ff","resourceVersion":"1212","generation":1,"creationTimestamp":"2020-10-08T11:34:56Z"},"spec":{"cluster_id":"cluster1","cable_name":"submariner-cable-cluster1-172-17-0-9","hostname":"cluster1-worker","subnets":["100.91.0.0/16","10.241.0.0/16"],"private_ip":"172.17.0.9","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
  3588. 2020-10-08T11:45:08.1750425Z [cluster1] I1008 11:36:41.457568 1 route.go:736] Enqueueing endpoint for route controller {"metadata":{"name":"cluster2-submariner-cable-cluster2-172-17-0-5","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster2-submariner-cable-cluster2-172-17-0-5","uid":"474bcd76-5748-4372-9832-4a15a4b1804d","resourceVersion":"1332","generation":1,"creationTimestamp":"2020-10-08T11:35:26Z"},"spec":{"cluster_id":"cluster2","cable_name":"submariner-cable-cluster2-172-17-0-5","hostname":"cluster2-worker","subnets":["100.92.0.0/16","10.242.0.0/16"],"private_ip":"172.17.0.5","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
  3589. 2020-10-08T11:45:08.1754148Z [cluster1] I1008 11:36:41.464923 1 route.go:634] Processing local Endpoint cluster1-submariner-cable-cluster1-172-17-0-9 with IP 172.17.0.9, Host cluster1-worker for this cluster
  3590. 2020-10-08T11:45:08.1756539Z [cluster1] I1008 11:36:41.465078 1 route.go:851] Processing route {Ifindex: 15 Dst: 240.0.0.0/8 Src: 240.17.0.9 Gw: <nil> Flags: [] Table: 254}
  3591. 2020-10-08T11:45:08.1757112Z [cluster1] I1008 11:36:41.465098 1 route.go:854] Found nil gw or dst
  3592. 2020-10-08T11:45:08.1757650Z [cluster1] I1008 11:36:41.465116 1 route.go:645] This route agent is running on the active gateway node
  3593. 2020-10-08T11:45:08.1758623Z [cluster1] I1008 11:36:41.465123 1 route.go:653] Creating the vxlan interface: vx-submariner on the gateway node
  3594. 2020-10-08T11:45:08.1759390Z [cluster1] I1008 11:36:41.465214 1 vxlan.go:61] VxLAN interface already exists with same configuration.
  3595. 2020-10-08T11:45:08.1760081Z [cluster1] I1008 11:36:41.465284 1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.9 00:00:00:00:00:00
  3596. 2020-10-08T11:45:08.1760706Z [cluster1] I1008 11:36:41.465310 1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.7 00:00:00:00:00:00
  3597. 2020-10-08T11:45:08.1761638Z [cluster1] I1008 11:36:41.465361 1 route.go:478] Successfully configured rp_filter to loose mode(2) on vx-submariner
  3598. 2020-10-08T11:45:08.1763210Z [cluster1] I1008 11:36:41.467187 1 route.go:620] Updating iptable rules for remote cluster's Endpoint "cluster2-submariner-cable-cluster2-172-17-0-5" with subnets [100.92.0.0/16 10.242.0.0/16]
  3599. 2020-10-08T11:45:08.1769552Z [cluster1] I1008 11:37:11.457687 1 route.go:736] Enqueueing endpoint for route controller {"metadata":{"name":"cluster2-submariner-cable-cluster2-172-17-0-5","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster2-submariner-cable-cluster2-172-17-0-5","uid":"474bcd76-5748-4372-9832-4a15a4b1804d","resourceVersion":"1332","generation":1,"creationTimestamp":"2020-10-08T11:35:26Z"},"spec":{"cluster_id":"cluster2","cable_name":"submariner-cable-cluster2-172-17-0-5","hostname":"cluster2-worker","subnets":["100.92.0.0/16","10.242.0.0/16"],"private_ip":"172.17.0.5","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
  3600. 2020-10-08T11:45:08.1778214Z [cluster1] I1008 11:37:11.457757 1 route.go:736] Enqueueing endpoint for route controller {"kind":"Endpoint","apiVersion":"submariner.io/v1","metadata":{"name":"cluster1-submariner-cable-cluster1-172-17-0-9","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster1-submariner-cable-cluster1-172-17-0-9","uid":"23aee4db-4473-4b17-8f51-aedd32e3a5ff","resourceVersion":"1212","generation":1,"creationTimestamp":"2020-10-08T11:34:56Z"},"spec":{"cluster_id":"cluster1","cable_name":"submariner-cable-cluster1-172-17-0-9","hostname":"cluster1-worker","subnets":["100.91.0.0/16","10.241.0.0/16"],"private_ip":"172.17.0.9","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
  3601. 2020-10-08T11:45:08.1783451Z [cluster1] I1008 11:37:11.465141 1 route.go:620] Updating iptable rules for remote cluster's Endpoint "cluster2-submariner-cable-cluster2-172-17-0-5" with subnets [100.92.0.0/16 10.242.0.0/16]
  3602. 2020-10-08T11:45:08.1785559Z [cluster1] I1008 11:37:11.466789 1 route.go:634] Processing local Endpoint cluster1-submariner-cable-cluster1-172-17-0-9 with IP 172.17.0.9, Host cluster1-worker for this cluster
  3603. 2020-10-08T11:45:08.1786843Z [cluster1] I1008 11:37:11.467136 1 route.go:851] Processing route {Ifindex: 15 Dst: 240.0.0.0/8 Src: 240.17.0.9 Gw: <nil> Flags: [] Table: 254}
  3604. 2020-10-08T11:45:08.1787407Z [cluster1] I1008 11:37:11.467172 1 route.go:854] Found nil gw or dst
  3605. 2020-10-08T11:45:08.1788252Z [cluster1] I1008 11:37:11.467180 1 route.go:645] This route agent is running on the active gateway node
  3606. 2020-10-08T11:45:08.1789360Z [cluster1] I1008 11:37:11.467188 1 route.go:653] Creating the vxlan interface: vx-submariner on the gateway node
  3607. 2020-10-08T11:45:08.1790113Z [cluster1] I1008 11:37:11.467362 1 vxlan.go:61] VxLAN interface already exists with same configuration.
  3608. 2020-10-08T11:45:08.1790797Z [cluster1] I1008 11:37:11.467392 1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.9 00:00:00:00:00:00
  3609. 2020-10-08T11:45:08.1791572Z [cluster1] I1008 11:37:11.467441 1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.7 00:00:00:00:00:00
  3610. 2020-10-08T11:45:08.1792671Z [cluster1] I1008 11:37:11.467491 1 route.go:478] Successfully configured rp_filter to loose mode(2) on vx-submariner
  3611. 2020-10-08T11:45:08.1797561Z [cluster1] I1008 11:37:41.457821 1 route.go:736] Enqueueing endpoint for route controller {"kind":"Endpoint","apiVersion":"submariner.io/v1","metadata":{"name":"cluster1-submariner-cable-cluster1-172-17-0-9","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster1-submariner-cable-cluster1-172-17-0-9","uid":"23aee4db-4473-4b17-8f51-aedd32e3a5ff","resourceVersion":"1212","generation":1,"creationTimestamp":"2020-10-08T11:34:56Z"},"spec":{"cluster_id":"cluster1","cable_name":"submariner-cable-cluster1-172-17-0-9","hostname":"cluster1-worker","subnets":["100.91.0.0/16","10.241.0.0/16"],"private_ip":"172.17.0.9","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
  3612. 2020-10-08T11:45:08.1805638Z [cluster1] I1008 11:37:41.457931 1 route.go:736] Enqueueing endpoint for route controller {"metadata":{"name":"cluster2-submariner-cable-cluster2-172-17-0-5","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster2-submariner-cable-cluster2-172-17-0-5","uid":"474bcd76-5748-4372-9832-4a15a4b1804d","resourceVersion":"1332","generation":1,"creationTimestamp":"2020-10-08T11:35:26Z"},"spec":{"cluster_id":"cluster2","cable_name":"submariner-cable-cluster2-172-17-0-5","hostname":"cluster2-worker","subnets":["100.92.0.0/16","10.242.0.0/16"],"private_ip":"172.17.0.5","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
  3613. 2020-10-08T11:45:08.1809980Z [cluster1] I1008 11:37:41.465606 1 route.go:634] Processing local Endpoint cluster1-submariner-cable-cluster1-172-17-0-9 with IP 172.17.0.9, Host cluster1-worker for this cluster
  3614. 2020-10-08T11:45:08.1811260Z [cluster1] I1008 11:37:41.465906 1 route.go:851] Processing route {Ifindex: 15 Dst: 240.0.0.0/8 Src: 240.17.0.9 Gw: <nil> Flags: [] Table: 254}
  3615. 2020-10-08T11:45:08.1811819Z [cluster1] I1008 11:37:41.465938 1 route.go:854] Found nil gw or dst
  3616. 2020-10-08T11:45:08.1812338Z [cluster1] I1008 11:37:41.465947 1 route.go:645] This route agent is running on the active gateway node
  3617. 2020-10-08T11:45:08.1813495Z [cluster1] I1008 11:37:41.465955 1 route.go:653] Creating the vxlan interface: vx-submariner on the gateway node
  3618. 2020-10-08T11:45:08.1814248Z [cluster1] I1008 11:37:41.466074 1 vxlan.go:61] VxLAN interface already exists with same configuration.
  3619. 2020-10-08T11:45:08.1814938Z [cluster1] I1008 11:37:41.466110 1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.9 00:00:00:00:00:00
  3620. 2020-10-08T11:45:08.1815563Z [cluster1] I1008 11:37:41.466136 1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.7 00:00:00:00:00:00
  3621. 2020-10-08T11:45:08.1816504Z [cluster1] I1008 11:37:41.466190 1 route.go:478] Successfully configured rp_filter to loose mode(2) on vx-submariner
  3622. 2020-10-08T11:45:08.1818634Z [cluster1] I1008 11:37:41.467744 1 route.go:620] Updating iptable rules for remote cluster's Endpoint "cluster2-submariner-cable-cluster2-172-17-0-5" with subnets [100.92.0.0/16 10.242.0.0/16]
  3623. 2020-10-08T11:45:08.1823422Z [cluster1] I1008 11:38:11.457943 1 route.go:736] Enqueueing endpoint for route controller {"kind":"Endpoint","apiVersion":"submariner.io/v1","metadata":{"name":"cluster1-submariner-cable-cluster1-172-17-0-9","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster1-submariner-cable-cluster1-172-17-0-9","uid":"23aee4db-4473-4b17-8f51-aedd32e3a5ff","resourceVersion":"1212","generation":1,"creationTimestamp":"2020-10-08T11:34:56Z"},"spec":{"cluster_id":"cluster1","cable_name":"submariner-cable-cluster1-172-17-0-9","hostname":"cluster1-worker","subnets":["100.91.0.0/16","10.241.0.0/16"],"private_ip":"172.17.0.9","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
  3624. 2020-10-08T11:45:08.1829644Z [cluster1] I1008 11:38:11.458009 1 route.go:736] Enqueueing endpoint for route controller {"metadata":{"name":"cluster2-submariner-cable-cluster2-172-17-0-5","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster2-submariner-cable-cluster2-172-17-0-5","uid":"474bcd76-5748-4372-9832-4a15a4b1804d","resourceVersion":"1332","generation":1,"creationTimestamp":"2020-10-08T11:35:26Z"},"spec":{"cluster_id":"cluster2","cable_name":"submariner-cable-cluster2-172-17-0-5","hostname":"cluster2-worker","subnets":["100.92.0.0/16","10.242.0.0/16"],"private_ip":"172.17.0.5","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
  3625. 2020-10-08T11:45:08.1833753Z [cluster1] I1008 11:38:11.465224 1 route.go:634] Processing local Endpoint cluster1-submariner-cable-cluster1-172-17-0-9 with IP 172.17.0.9, Host cluster1-worker for this cluster
  3626. 2020-10-08T11:45:08.1835037Z [cluster1] I1008 11:38:11.465499 1 route.go:851] Processing route {Ifindex: 15 Dst: 240.0.0.0/8 Src: 240.17.0.9 Gw: <nil> Flags: [] Table: 254}
  3627. 2020-10-08T11:45:08.1835601Z [cluster1] I1008 11:38:11.465527 1 route.go:854] Found nil gw or dst
  3628. 2020-10-08T11:45:08.1836140Z [cluster1] I1008 11:38:11.465536 1 route.go:645] This route agent is running on the active gateway node
  3629. 2020-10-08T11:45:08.1837064Z [cluster1] I1008 11:38:11.465544 1 route.go:653] Creating the vxlan interface: vx-submariner on the gateway node
  3630. 2020-10-08T11:45:08.1837817Z [cluster1] I1008 11:38:11.465665 1 vxlan.go:61] VxLAN interface already exists with same configuration.
  3631. 2020-10-08T11:45:08.1838506Z [cluster1] I1008 11:38:11.465697 1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.9 00:00:00:00:00:00
  3632. 2020-10-08T11:45:08.1839134Z [cluster1] I1008 11:38:11.465724 1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.7 00:00:00:00:00:00
  3633. 2020-10-08T11:45:08.1840399Z [cluster1] I1008 11:38:11.465770 1 route.go:478] Successfully configured rp_filter to loose mode(2) on vx-submariner
  3634. 2020-10-08T11:45:08.1842190Z [cluster1] I1008 11:38:11.467371 1 route.go:620] Updating iptable rules for remote cluster's Endpoint "cluster2-submariner-cable-cluster2-172-17-0-5" with subnets [100.92.0.0/16 10.242.0.0/16]
  3635. 2020-10-08T11:45:08.1847320Z [cluster1] I1008 11:38:41.458093 1 route.go:736] Enqueueing endpoint for route controller {"kind":"Endpoint","apiVersion":"submariner.io/v1","metadata":{"name":"cluster1-submariner-cable-cluster1-172-17-0-9","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster1-submariner-cable-cluster1-172-17-0-9","uid":"23aee4db-4473-4b17-8f51-aedd32e3a5ff","resourceVersion":"1212","generation":1,"creationTimestamp":"2020-10-08T11:34:56Z"},"spec":{"cluster_id":"cluster1","cable_name":"submariner-cable-cluster1-172-17-0-9","hostname":"cluster1-worker","subnets":["100.91.0.0/16","10.241.0.0/16"],"private_ip":"172.17.0.9","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
  3636. 2020-10-08T11:45:08.1855240Z [cluster1] I1008 11:38:41.458163 1 route.go:736] Enqueueing endpoint for route controller {"metadata":{"name":"cluster2-submariner-cable-cluster2-172-17-0-5","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster2-submariner-cable-cluster2-172-17-0-5","uid":"474bcd76-5748-4372-9832-4a15a4b1804d","resourceVersion":"1332","generation":1,"creationTimestamp":"2020-10-08T11:35:26Z"},"spec":{"cluster_id":"cluster2","cable_name":"submariner-cable-cluster2-172-17-0-5","hostname":"cluster2-worker","subnets":["100.92.0.0/16","10.242.0.0/16"],"private_ip":"172.17.0.5","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
  3637. 2020-10-08T11:45:08.1859311Z [cluster1] I1008 11:38:41.467926 1 route.go:634] Processing local Endpoint cluster1-submariner-cable-cluster1-172-17-0-9 with IP 172.17.0.9, Host cluster1-worker for this cluster
  3638. 2020-10-08T11:45:08.1860558Z [cluster1] I1008 11:38:41.468082 1 route.go:851] Processing route {Ifindex: 15 Dst: 240.0.0.0/8 Src: 240.17.0.9 Gw: <nil> Flags: [] Table: 254}
  3639. 2020-10-08T11:45:08.1861170Z [cluster1] I1008 11:38:41.468101 1 route.go:854] Found nil gw or dst
  3640. 2020-10-08T11:45:08.1861643Z [cluster1] I1008 11:38:41.468109 1 route.go:645] This route agent is running on the active gateway node
  3641. 2020-10-08T11:45:08.1862588Z [cluster1] I1008 11:38:41.468117 1 route.go:653] Creating the vxlan interface: vx-submariner on the gateway node
  3642. 2020-10-08T11:45:08.1863442Z [cluster1] I1008 11:38:41.468254 1 vxlan.go:61] VxLAN interface already exists with same configuration.
  3643. 2020-10-08T11:45:08.1864978Z [cluster1] I1008 11:38:41.468282 1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.9 00:00:00:00:00:00
  3644. 2020-10-08T11:45:08.1865618Z [cluster1] I1008 11:38:41.468306 1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.7 00:00:00:00:00:00
  3645. 2020-10-08T11:45:08.1866763Z [cluster1] I1008 11:38:41.468346 1 route.go:478] Successfully configured rp_filter to loose mode(2) on vx-submariner
  3646. 2020-10-08T11:45:08.1868483Z [cluster1] I1008 11:38:41.471927 1 route.go:620] Updating iptable rules for remote cluster's Endpoint "cluster2-submariner-cable-cluster2-172-17-0-5" with subnets [100.92.0.0/16 10.242.0.0/16]
  3647. 2020-10-08T11:45:08.1870401Z [lighthouse]$ [cluster1] kubectl -n submariner-operator logs submariner-routeagent-wmmbs
  3648. 2020-10-08T11:45:08.1871823Z [lighthouse]$ [cluster1] kubectl -n submariner-operator logs submariner-routeagent-wmmbs
  3649. 2020-10-08T11:45:08.1873361Z [lighthouse]$ [cluster1] command kubectl --context=cluster1 -n submariner-operator logs submariner-routeagent-wmmbs
  3650. 2020-10-08T11:45:08.1874951Z [lighthouse]$ [cluster1] kubectl --context=cluster1 -n submariner-operator logs submariner-routeagent-wmmbs
  3651. 2020-10-08T11:45:08.1880116Z [cluster1] I1008 11:39:11.458176 1 route.go:736] Enqueueing endpoint for route controller {"kind":"Endpoint","apiVersion":"submariner.io/v1","metadata":{"name":"cluster1-submariner-cable-cluster1-172-17-0-9","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster1-submariner-cable-cluster1-172-17-0-9","uid":"23aee4db-4473-4b17-8f51-aedd32e3a5ff","resourceVersion":"1212","generation":1,"creationTimestamp":"2020-10-08T11:34:56Z"},"spec":{"cluster_id":"cluster1","cable_name":"submariner-cable-cluster1-172-17-0-9","hostname":"cluster1-worker","subnets":["100.91.0.0/16","10.241.0.0/16"],"private_ip":"172.17.0.9","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
  3652. 2020-10-08T11:45:08.1889594Z [cluster1] I1008 11:39:11.458245 1 route.go:736] Enqueueing endpoint for route controller {"metadata":{"name":"cluster2-submariner-cable-cluster2-172-17-0-5","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster2-submariner-cable-cluster2-172-17-0-5","uid":"474bcd76-5748-4372-9832-4a15a4b1804d","resourceVersion":"1332","generation":1,"creationTimestamp":"2020-10-08T11:35:26Z"},"spec":{"cluster_id":"cluster2","cable_name":"submariner-cable-cluster2-172-17-0-5","hostname":"cluster2-worker","subnets":["100.92.0.0/16","10.242.0.0/16"],"private_ip":"172.17.0.5","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
  3653. 2020-10-08T11:45:08.1894321Z [cluster1] I1008 11:39:11.465904 1 route.go:634] Processing local Endpoint cluster1-submariner-cable-cluster1-172-17-0-9 with IP 172.17.0.9, Host cluster1-worker for this cluster
  3654. 2020-10-08T11:45:08.1896142Z [cluster1] I1008 11:39:11.466058 1 route.go:851] Processing route {Ifindex: 15 Dst: 240.0.0.0/8 Src: 240.17.0.9 Gw: <nil> Flags: [] Table: 254}
  3655. 2020-10-08T11:45:08.1896820Z [cluster1] I1008 11:39:11.466082 1 route.go:854] Found nil gw or dst
  3656. 2020-10-08T11:45:08.1897361Z [cluster1] I1008 11:39:11.466090 1 route.go:645] This route agent is running on the active gateway node
  3657. 2020-10-08T11:45:08.1898322Z [cluster1] I1008 11:39:11.466098 1 route.go:653] Creating the vxlan interface: vx-submariner on the gateway node
  3658. 2020-10-08T11:45:08.1899069Z [cluster1] I1008 11:39:11.466207 1 vxlan.go:61] VxLAN interface already exists with same configuration.
  3659. 2020-10-08T11:45:08.1899758Z [cluster1] I1008 11:39:11.466243 1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.9 00:00:00:00:00:00
  3660. 2020-10-08T11:45:08.1900397Z [cluster1] I1008 11:39:11.466268 1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.7 00:00:00:00:00:00
  3661. 2020-10-08T11:45:08.1901358Z [cluster1] I1008 11:39:11.466318 1 route.go:478] Successfully configured rp_filter to loose mode(2) on vx-submariner
  3662. 2020-10-08T11:45:08.1902952Z [cluster1] I1008 11:39:11.468110 1 route.go:620] Updating iptable rules for remote cluster's Endpoint "cluster2-submariner-cable-cluster2-172-17-0-5" with subnets [100.92.0.0/16 10.242.0.0/16]
  3663. 2020-10-08T11:45:08.1907906Z [cluster1] I1008 11:39:41.458446 1 route.go:736] Enqueueing endpoint for route controller {"kind":"Endpoint","apiVersion":"submariner.io/v1","metadata":{"name":"cluster1-submariner-cable-cluster1-172-17-0-9","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster1-submariner-cable-cluster1-172-17-0-9","uid":"23aee4db-4473-4b17-8f51-aedd32e3a5ff","resourceVersion":"1212","generation":1,"creationTimestamp":"2020-10-08T11:34:56Z"},"spec":{"cluster_id":"cluster1","cable_name":"submariner-cable-cluster1-172-17-0-9","hostname":"cluster1-worker","subnets":["100.91.0.0/16","10.241.0.0/16"],"private_ip":"172.17.0.9","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
  3664. 2020-10-08T11:45:08.1914866Z [cluster1] I1008 11:39:41.458629 1 route.go:736] Enqueueing endpoint for route controller {"metadata":{"name":"cluster2-submariner-cable-cluster2-172-17-0-5","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster2-submariner-cable-cluster2-172-17-0-5","uid":"474bcd76-5748-4372-9832-4a15a4b1804d","resourceVersion":"1332","generation":1,"creationTimestamp":"2020-10-08T11:35:26Z"},"spec":{"cluster_id":"cluster2","cable_name":"submariner-cable-cluster2-172-17-0-5","hostname":"cluster2-worker","subnets":["100.92.0.0/16","10.242.0.0/16"],"private_ip":"172.17.0.5","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
  3665. 2020-10-08T11:45:08.1919746Z [cluster1] I1008 11:39:41.465768 1 route.go:634] Processing local Endpoint cluster1-submariner-cable-cluster1-172-17-0-9 with IP 172.17.0.9, Host cluster1-worker for this cluster
  3666. 2020-10-08T11:45:08.1921033Z [cluster1] I1008 11:39:41.465965 1 route.go:851] Processing route {Ifindex: 15 Dst: 240.0.0.0/8 Src: 240.17.0.9 Gw: <nil> Flags: [] Table: 254}
  3667. 2020-10-08T11:45:08.1921596Z [cluster1] I1008 11:39:41.465992 1 route.go:854] Found nil gw or dst
  3668. 2020-10-08T11:45:08.1922133Z [cluster1] I1008 11:39:41.466000 1 route.go:645] This route agent is running on the active gateway node
  3669. 2020-10-08T11:45:08.1923045Z [cluster1] I1008 11:39:41.466008 1 route.go:653] Creating the vxlan interface: vx-submariner on the gateway node
  3670. 2020-10-08T11:45:08.1923796Z [cluster1] I1008 11:39:41.466124 1 vxlan.go:61] VxLAN interface already exists with same configuration.
  3671. 2020-10-08T11:45:08.1924512Z [cluster1] I1008 11:39:41.466162 1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.9 00:00:00:00:00:00
  3672. 2020-10-08T11:45:08.1925209Z [cluster1] I1008 11:39:41.466194 1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.7 00:00:00:00:00:00
  3673. 2020-10-08T11:45:08.1926153Z [cluster1] I1008 11:39:41.466247 1 route.go:478] Successfully configured rp_filter to loose mode(2) on vx-submariner
  3674. 2020-10-08T11:45:08.1927732Z [cluster1] I1008 11:39:41.467764 1 route.go:620] Updating iptable rules for remote cluster's Endpoint "cluster2-submariner-cable-cluster2-172-17-0-5" with subnets [100.92.0.0/16 10.242.0.0/16]
  3675. 2020-10-08T11:45:08.1932073Z [cluster1] I1008 11:40:11.458508 1 route.go:736] Enqueueing endpoint for route controller {"metadata":{"name":"cluster2-submariner-cable-cluster2-172-17-0-5","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster2-submariner-cable-cluster2-172-17-0-5","uid":"474bcd76-5748-4372-9832-4a15a4b1804d","resourceVersion":"1332","generation":1,"creationTimestamp":"2020-10-08T11:35:26Z"},"spec":{"cluster_id":"cluster2","cable_name":"submariner-cable-cluster2-172-17-0-5","hostname":"cluster2-worker","subnets":["100.92.0.0/16","10.242.0.0/16"],"private_ip":"172.17.0.5","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
  3676. 2020-10-08T11:45:08.1938925Z [cluster1] I1008 11:40:11.458581 1 route.go:736] Enqueueing endpoint for route controller {"kind":"Endpoint","apiVersion":"submariner.io/v1","metadata":{"name":"cluster1-submariner-cable-cluster1-172-17-0-9","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster1-submariner-cable-cluster1-172-17-0-9","uid":"23aee4db-4473-4b17-8f51-aedd32e3a5ff","resourceVersion":"1212","generation":1,"creationTimestamp":"2020-10-08T11:34:56Z"},"spec":{"cluster_id":"cluster1","cable_name":"submariner-cable-cluster1-172-17-0-9","hostname":"cluster1-worker","subnets":["100.91.0.0/16","10.241.0.0/16"],"private_ip":"172.17.0.9","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
  3677. 2020-10-08T11:45:08.1942786Z [cluster1] I1008 11:40:11.465785 1 route.go:620] Updating iptable rules for remote cluster's Endpoint "cluster2-submariner-cable-cluster2-172-17-0-5" with subnets [100.92.0.0/16 10.242.0.0/16]
  3678. 2020-10-08T11:45:08.1944616Z [cluster1] I1008 11:40:11.467570 1 route.go:634] Processing local Endpoint cluster1-submariner-cable-cluster1-172-17-0-9 with IP 172.17.0.9, Host cluster1-worker for this cluster
  3679. 2020-10-08T11:45:08.1945736Z [cluster1] I1008 11:40:11.467730 1 route.go:851] Processing route {Ifindex: 15 Dst: 240.0.0.0/8 Src: 240.17.0.9 Gw: <nil> Flags: [] Table: 254}
  3680. 2020-10-08T11:45:08.1946224Z [cluster1] I1008 11:40:11.467753 1 route.go:854] Found nil gw or dst
  3681. 2020-10-08T11:45:08.1946896Z [cluster1] I1008 11:40:11.467761 1 route.go:645] This route agent is running on the active gateway node
  3682. 2020-10-08T11:45:08.1947811Z [cluster1] I1008 11:40:11.467769 1 route.go:653] Creating the vxlan interface: vx-submariner on the gateway node
  3683. 2020-10-08T11:45:08.1948653Z [cluster1] I1008 11:40:11.467878 1 vxlan.go:61] VxLAN interface already exists with same configuration.
  3684. 2020-10-08T11:45:08.1949342Z [cluster1] I1008 11:40:11.467912 1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.9 00:00:00:00:00:00
  3685. 2020-10-08T11:45:08.1949976Z [cluster1] I1008 11:40:11.467941 1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.7 00:00:00:00:00:00
  3686. 2020-10-08T11:45:08.1950918Z [cluster1] I1008 11:40:11.467987 1 route.go:478] Successfully configured rp_filter to loose mode(2) on vx-submariner
  3687. 2020-10-08T11:45:08.1955706Z [cluster1] I1008 11:40:41.458713 1 route.go:736] Enqueueing endpoint for route controller {"kind":"Endpoint","apiVersion":"submariner.io/v1","metadata":{"name":"cluster1-submariner-cable-cluster1-172-17-0-9","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster1-submariner-cable-cluster1-172-17-0-9","uid":"23aee4db-4473-4b17-8f51-aedd32e3a5ff","resourceVersion":"1212","generation":1,"creationTimestamp":"2020-10-08T11:34:56Z"},"spec":{"cluster_id":"cluster1","cable_name":"submariner-cable-cluster1-172-17-0-9","hostname":"cluster1-worker","subnets":["100.91.0.0/16","10.241.0.0/16"],"private_ip":"172.17.0.9","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
  3688. 2020-10-08T11:45:08.1962829Z [cluster1] I1008 11:40:41.458802 1 route.go:736] Enqueueing endpoint for route controller {"metadata":{"name":"cluster2-submariner-cable-cluster2-172-17-0-5","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster2-submariner-cable-cluster2-172-17-0-5","uid":"474bcd76-5748-4372-9832-4a15a4b1804d","resourceVersion":"1332","generation":1,"creationTimestamp":"2020-10-08T11:35:26Z"},"spec":{"cluster_id":"cluster2","cable_name":"submariner-cable-cluster2-172-17-0-5","hostname":"cluster2-worker","subnets":["100.92.0.0/16","10.242.0.0/16"],"private_ip":"172.17.0.5","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
  3689. 2020-10-08T11:45:08.1967926Z [cluster1] I1008 11:40:41.466604 1 route.go:634] Processing local Endpoint cluster1-submariner-cable-cluster1-172-17-0-9 with IP 172.17.0.9, Host cluster1-worker for this cluster
  3690. 2020-10-08T11:45:08.1969040Z [cluster1] I1008 11:40:41.466782 1 route.go:851] Processing route {Ifindex: 15 Dst: 240.0.0.0/8 Src: 240.17.0.9 Gw: <nil> Flags: [] Table: 254}
  3691. 2020-10-08T11:45:08.1969531Z [cluster1] I1008 11:40:41.466807 1 route.go:854] Found nil gw or dst
  3692. 2020-10-08T11:45:08.1970002Z [cluster1] I1008 11:40:41.466814 1 route.go:645] This route agent is running on the active gateway node
  3693. 2020-10-08T11:45:08.1970804Z [cluster1] I1008 11:40:41.466822 1 route.go:653] Creating the vxlan interface: vx-submariner on the gateway node
  3694. 2020-10-08T11:45:08.1971458Z [cluster1] I1008 11:40:41.466934 1 vxlan.go:61] VxLAN interface already exists with same configuration.
  3695. 2020-10-08T11:45:08.1972062Z [cluster1] I1008 11:40:41.466960 1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.7 00:00:00:00:00:00
  3696. 2020-10-08T11:45:08.1972699Z [cluster1] I1008 11:40:41.466983 1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.9 00:00:00:00:00:00
  3697. 2020-10-08T11:45:08.1973505Z [cluster1] I1008 11:40:41.467051 1 route.go:478] Successfully configured rp_filter to loose mode(2) on vx-submariner
  3698. 2020-10-08T11:45:08.1975660Z [cluster1] I1008 11:40:41.468858 1 route.go:620] Updating iptable rules for remote cluster's Endpoint "cluster2-submariner-cable-cluster2-172-17-0-5" with subnets [100.92.0.0/16 10.242.0.0/16]
  3699. 2020-10-08T11:45:08.1980120Z [cluster1] I1008 11:41:11.458813 1 route.go:736] Enqueueing endpoint for route controller {"kind":"Endpoint","apiVersion":"submariner.io/v1","metadata":{"name":"cluster1-submariner-cable-cluster1-172-17-0-9","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster1-submariner-cable-cluster1-172-17-0-9","uid":"23aee4db-4473-4b17-8f51-aedd32e3a5ff","resourceVersion":"1212","generation":1,"creationTimestamp":"2020-10-08T11:34:56Z"},"spec":{"cluster_id":"cluster1","cable_name":"submariner-cable-cluster1-172-17-0-9","hostname":"cluster1-worker","subnets":["100.91.0.0/16","10.241.0.0/16"],"private_ip":"172.17.0.9","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
  3700. 2020-10-08T11:45:08.1986445Z [cluster1] I1008 11:41:11.458880 1 route.go:736] Enqueueing endpoint for route controller {"metadata":{"name":"cluster2-submariner-cable-cluster2-172-17-0-5","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster2-submariner-cable-cluster2-172-17-0-5","uid":"474bcd76-5748-4372-9832-4a15a4b1804d","resourceVersion":"1332","generation":1,"creationTimestamp":"2020-10-08T11:35:26Z"},"spec":{"cluster_id":"cluster2","cable_name":"submariner-cable-cluster2-172-17-0-5","hostname":"cluster2-worker","subnets":["100.92.0.0/16","10.242.0.0/16"],"private_ip":"172.17.0.5","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
  3701. 2020-10-08T11:45:08.1990743Z [cluster1] I1008 11:41:11.466467 1 route.go:634] Processing local Endpoint cluster1-submariner-cable-cluster1-172-17-0-9 with IP 172.17.0.9, Host cluster1-worker for this cluster
  3702. 2020-10-08T11:45:08.1992025Z [cluster1] I1008 11:41:11.466631 1 route.go:851] Processing route {Ifindex: 15 Dst: 240.0.0.0/8 Src: 240.17.0.9 Gw: <nil> Flags: [] Table: 254}
  3703. 2020-10-08T11:45:08.1992585Z [cluster1] I1008 11:41:11.466655 1 route.go:854] Found nil gw or dst
  3704. 2020-10-08T11:45:08.1993120Z [cluster1] I1008 11:41:11.466663 1 route.go:645] This route agent is running on the active gateway node
  3705. 2020-10-08T11:45:08.1994019Z [cluster1] I1008 11:41:11.466670 1 route.go:653] Creating the vxlan interface: vx-submariner on the gateway node
  3706. 2020-10-08T11:45:08.1995519Z [cluster1] I1008 11:41:11.466778 1 vxlan.go:61] VxLAN interface already exists with same configuration.
  3707. 2020-10-08T11:45:08.1996371Z [cluster1] I1008 11:41:11.466813 1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.9 00:00:00:00:00:00
  3708. 2020-10-08T11:45:08.1997134Z [cluster1] I1008 11:41:11.466837 1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.7 00:00:00:00:00:00
  3709. 2020-10-08T11:45:08.1998105Z [cluster1] I1008 11:41:11.466905 1 route.go:478] Successfully configured rp_filter to loose mode(2) on vx-submariner
  3710. 2020-10-08T11:45:08.1999722Z [cluster1] I1008 11:41:11.468758 1 route.go:620] Updating iptable rules for remote cluster's Endpoint "cluster2-submariner-cable-cluster2-172-17-0-5" with subnets [100.92.0.0/16 10.242.0.0/16]
  3711. 2020-10-08T11:45:08.2004717Z [cluster1] I1008 11:41:41.458910 1 route.go:736] Enqueueing endpoint for route controller {"kind":"Endpoint","apiVersion":"submariner.io/v1","metadata":{"name":"cluster1-submariner-cable-cluster1-172-17-0-9","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster1-submariner-cable-cluster1-172-17-0-9","uid":"23aee4db-4473-4b17-8f51-aedd32e3a5ff","resourceVersion":"1212","generation":1,"creationTimestamp":"2020-10-08T11:34:56Z"},"spec":{"cluster_id":"cluster1","cable_name":"submariner-cable-cluster1-172-17-0-9","hostname":"cluster1-worker","subnets":["100.91.0.0/16","10.241.0.0/16"],"private_ip":"172.17.0.9","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
  3712. 2020-10-08T11:45:08.2010899Z [cluster1] I1008 11:41:41.459010 1 route.go:736] Enqueueing endpoint for route controller {"metadata":{"name":"cluster2-submariner-cable-cluster2-172-17-0-5","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster2-submariner-cable-cluster2-172-17-0-5","uid":"474bcd76-5748-4372-9832-4a15a4b1804d","resourceVersion":"1332","generation":1,"creationTimestamp":"2020-10-08T11:35:26Z"},"spec":{"cluster_id":"cluster2","cable_name":"submariner-cable-cluster2-172-17-0-5","hostname":"cluster2-worker","subnets":["100.92.0.0/16","10.242.0.0/16"],"private_ip":"172.17.0.5","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
  3713. 2020-10-08T11:45:08.2014875Z [cluster1] I1008 11:41:41.466344 1 route.go:634] Processing local Endpoint cluster1-submariner-cable-cluster1-172-17-0-9 with IP 172.17.0.9, Host cluster1-worker for this cluster
  3714. 2020-10-08T11:45:08.2015997Z [cluster1] I1008 11:41:41.466541 1 route.go:851] Processing route {Ifindex: 15 Dst: 240.0.0.0/8 Src: 240.17.0.9 Gw: <nil> Flags: [] Table: 254}
  3715. 2020-10-08T11:45:08.2016485Z [cluster1] I1008 11:41:41.466567 1 route.go:854] Found nil gw or dst
  3716. 2020-10-08T11:45:08.2016958Z [cluster1] I1008 11:41:41.466575 1 route.go:645] This route agent is running on the active gateway node
  3717. 2020-10-08T11:45:08.2017763Z [cluster1] I1008 11:41:41.466583 1 route.go:653] Creating the vxlan interface: vx-submariner on the gateway node
  3718. 2020-10-08T11:45:08.2018639Z [cluster1] I1008 11:41:41.466694 1 vxlan.go:61] VxLAN interface already exists with same configuration.
  3719. 2020-10-08T11:45:08.2019435Z [cluster1] I1008 11:41:41.466721 1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.9 00:00:00:00:00:00
  3720. 2020-10-08T11:45:08.2020063Z [cluster1] I1008 11:41:41.466756 1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.7 00:00:00:00:00:00
  3721. 2020-10-08T11:45:08.2021003Z [cluster1] I1008 11:41:41.466796 1 route.go:478] Successfully configured rp_filter to loose mode(2) on vx-submariner
  3722. 2020-10-08T11:45:08.2022588Z [cluster1] I1008 11:41:41.468141 1 route.go:620] Updating iptable rules for remote cluster's Endpoint "cluster2-submariner-cable-cluster2-172-17-0-5" with subnets [100.92.0.0/16 10.242.0.0/16]
  3723. 2020-10-08T11:45:08.2027583Z [cluster1] I1008 11:42:11.459049 1 route.go:736] Enqueueing endpoint for route controller {"kind":"Endpoint","apiVersion":"submariner.io/v1","metadata":{"name":"cluster1-submariner-cable-cluster1-172-17-0-9","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster1-submariner-cable-cluster1-172-17-0-9","uid":"23aee4db-4473-4b17-8f51-aedd32e3a5ff","resourceVersion":"1212","generation":1,"creationTimestamp":"2020-10-08T11:34:56Z"},"spec":{"cluster_id":"cluster1","cable_name":"submariner-cable-cluster1-172-17-0-9","hostname":"cluster1-worker","subnets":["100.91.0.0/16","10.241.0.0/16"],"private_ip":"172.17.0.9","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
  3724. 2020-10-08T11:45:08.2034567Z [cluster1] I1008 11:42:11.459115 1 route.go:736] Enqueueing endpoint for route controller {"metadata":{"name":"cluster2-submariner-cable-cluster2-172-17-0-5","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster2-submariner-cable-cluster2-172-17-0-5","uid":"474bcd76-5748-4372-9832-4a15a4b1804d","resourceVersion":"1332","generation":1,"creationTimestamp":"2020-10-08T11:35:26Z"},"spec":{"cluster_id":"cluster2","cable_name":"submariner-cable-cluster2-172-17-0-5","hostname":"cluster2-worker","subnets":["100.92.0.0/16","10.242.0.0/16"],"private_ip":"172.17.0.5","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
  3725. 2020-10-08T11:45:08.2038941Z [cluster1] I1008 11:42:11.466593 1 route.go:634] Processing local Endpoint cluster1-submariner-cable-cluster1-172-17-0-9 with IP 172.17.0.9, Host cluster1-worker for this cluster
  3726. 2020-10-08T11:45:08.2041978Z [cluster1] I1008 11:42:11.466883 1 route.go:851] Processing route {Ifindex: 15 Dst: 240.0.0.0/8 Src: 240.17.0.9 Gw: <nil> Flags: [] Table: 254}
  3727. 2020-10-08T11:45:08.2042590Z [cluster1] I1008 11:42:11.466912 1 route.go:854] Found nil gw or dst
  3728. 2020-10-08T11:45:08.2044436Z [cluster1] I1008 11:42:11.466921 1 route.go:645] This route agent is running on the active gateway node
  3729. 2020-10-08T11:45:08.2045796Z [cluster1] I1008 11:42:11.466930 1 route.go:653] Creating the vxlan interface: vx-submariner on the gateway node
  3730. 2020-10-08T11:45:08.2046615Z [cluster1] I1008 11:42:11.467160 1 vxlan.go:61] VxLAN interface already exists with same configuration.
  3731. 2020-10-08T11:45:08.2047500Z [cluster1] I1008 11:42:11.467247 1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.9 00:00:00:00:00:00
  3732. 2020-10-08T11:45:08.2048181Z [cluster1] I1008 11:42:11.467350 1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.7 00:00:00:00:00:00
  3733. 2020-10-08T11:45:08.2049203Z [cluster1] I1008 11:42:11.467415 1 route.go:478] Successfully configured rp_filter to loose mode(2) on vx-submariner
  3734. 2020-10-08T11:45:08.2050903Z [cluster1] I1008 11:42:11.469158 1 route.go:620] Updating iptable rules for remote cluster's Endpoint "cluster2-submariner-cable-cluster2-172-17-0-5" with subnets [100.92.0.0/16 10.242.0.0/16]
  3735. 2020-10-08T11:45:08.2056367Z [cluster1] I1008 11:42:41.459131 1 route.go:736] Enqueueing endpoint for route controller {"kind":"Endpoint","apiVersion":"submariner.io/v1","metadata":{"name":"cluster1-submariner-cable-cluster1-172-17-0-9","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster1-submariner-cable-cluster1-172-17-0-9","uid":"23aee4db-4473-4b17-8f51-aedd32e3a5ff","resourceVersion":"1212","generation":1,"creationTimestamp":"2020-10-08T11:34:56Z"},"spec":{"cluster_id":"cluster1","cable_name":"submariner-cable-cluster1-172-17-0-9","hostname":"cluster1-worker","subnets":["100.91.0.0/16","10.241.0.0/16"],"private_ip":"172.17.0.9","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
  3736. 2020-10-08T11:45:08.2064014Z [cluster1] I1008 11:42:41.459198 1 route.go:736] Enqueueing endpoint for route controller {"metadata":{"name":"cluster2-submariner-cable-cluster2-172-17-0-5","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster2-submariner-cable-cluster2-172-17-0-5","uid":"474bcd76-5748-4372-9832-4a15a4b1804d","resourceVersion":"1332","generation":1,"creationTimestamp":"2020-10-08T11:35:26Z"},"spec":{"cluster_id":"cluster2","cable_name":"submariner-cable-cluster2-172-17-0-5","hostname":"cluster2-worker","subnets":["100.92.0.0/16","10.242.0.0/16"],"private_ip":"172.17.0.5","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
  3737. 2020-10-08T11:45:08.2068378Z [cluster1] I1008 11:42:41.466405 1 route.go:634] Processing local Endpoint cluster1-submariner-cable-cluster1-172-17-0-9 with IP 172.17.0.9, Host cluster1-worker for this cluster
  3738. 2020-10-08T11:45:08.2069658Z [cluster1] I1008 11:42:41.466578 1 route.go:851] Processing route {Ifindex: 15 Dst: 240.0.0.0/8 Src: 240.17.0.9 Gw: <nil> Flags: [] Table: 254}
  3739. 2020-10-08T11:45:08.2070220Z [cluster1] I1008 11:42:41.466604 1 route.go:854] Found nil gw or dst
  3740. 2020-10-08T11:45:08.2070825Z [cluster1] I1008 11:42:41.466613 1 route.go:645] This route agent is running on the active gateway node
  3741. 2020-10-08T11:45:08.2071625Z [cluster1] I1008 11:42:41.466620 1 route.go:653] Creating the vxlan interface: vx-submariner on the gateway node
  3742. 2020-10-08T11:45:08.2072262Z [cluster1] I1008 11:42:41.466730 1 vxlan.go:61] VxLAN interface already exists with same configuration.
  3743. 2020-10-08T11:45:08.2072869Z [cluster1] I1008 11:42:41.466763 1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.7 00:00:00:00:00:00
  3744. 2020-10-08T11:45:08.2073416Z [cluster1] I1008 11:42:41.466794 1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.9 00:00:00:00:00:00
  3745. 2020-10-08T11:45:08.2074229Z [cluster1] I1008 11:42:41.466840 1 route.go:478] Successfully configured rp_filter to loose mode(2) on vx-submariner
  3746. 2020-10-08T11:45:08.2075608Z [cluster1] I1008 11:42:41.468289 1 route.go:620] Updating iptable rules for remote cluster's Endpoint "cluster2-submariner-cable-cluster2-172-17-0-5" with subnets [100.92.0.0/16 10.242.0.0/16]
  3747. 2020-10-08T11:45:08.2079958Z [cluster1] I1008 11:43:11.459260 1 route.go:736] Enqueueing endpoint for route controller {"kind":"Endpoint","apiVersion":"submariner.io/v1","metadata":{"name":"cluster1-submariner-cable-cluster1-172-17-0-9","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster1-submariner-cable-cluster1-172-17-0-9","uid":"23aee4db-4473-4b17-8f51-aedd32e3a5ff","resourceVersion":"1212","generation":1,"creationTimestamp":"2020-10-08T11:34:56Z"},"spec":{"cluster_id":"cluster1","cable_name":"submariner-cable-cluster1-172-17-0-9","hostname":"cluster1-worker","subnets":["100.91.0.0/16","10.241.0.0/16"],"private_ip":"172.17.0.9","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
  3748. 2020-10-08T11:45:08.2088542Z [cluster1] I1008 11:43:11.459324 1 route.go:736] Enqueueing endpoint for route controller {"metadata":{"name":"cluster2-submariner-cable-cluster2-172-17-0-5","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster2-submariner-cable-cluster2-172-17-0-5","uid":"474bcd76-5748-4372-9832-4a15a4b1804d","resourceVersion":"1332","generation":1,"creationTimestamp":"2020-10-08T11:35:26Z"},"spec":{"cluster_id":"cluster2","cable_name":"submariner-cable-cluster2-172-17-0-5","hostname":"cluster2-worker","subnets":["100.92.0.0/16","10.242.0.0/16"],"private_ip":"172.17.0.5","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
  3749. 2020-10-08T11:45:08.2092993Z [cluster1] I1008 11:43:11.466360 1 route.go:634] Processing local Endpoint cluster1-submariner-cable-cluster1-172-17-0-9 with IP 172.17.0.9, Host cluster1-worker for this cluster
  3750. 2020-10-08T11:45:08.2094277Z [cluster1] I1008 11:43:11.466564 1 route.go:851] Processing route {Ifindex: 15 Dst: 240.0.0.0/8 Src: 240.17.0.9 Gw: <nil> Flags: [] Table: 254}
  3751. 2020-10-08T11:45:08.2094839Z [cluster1] I1008 11:43:11.466589 1 route.go:854] Found nil gw or dst
  3752. 2020-10-08T11:45:08.2096019Z [cluster1] I1008 11:43:11.466597 1 route.go:645] This route agent is running on the active gateway node
  3753. 2020-10-08T11:45:08.2096981Z [cluster1] I1008 11:43:11.466605 1 route.go:653] Creating the vxlan interface: vx-submariner on the gateway node
  3754. 2020-10-08T11:45:08.2097738Z [cluster1] I1008 11:43:11.466736 1 vxlan.go:61] VxLAN interface already exists with same configuration.
  3755. 2020-10-08T11:45:08.2098496Z [cluster1] I1008 11:43:11.466773 1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.9 00:00:00:00:00:00
  3756. 2020-10-08T11:45:08.2099044Z [cluster1] I1008 11:43:11.466801 1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.7 00:00:00:00:00:00
  3757. 2020-10-08T11:45:08.2099860Z [cluster1] I1008 11:43:11.466855 1 route.go:478] Successfully configured rp_filter to loose mode(2) on vx-submariner
  3758. 2020-10-08T11:45:08.2101249Z [cluster1] I1008 11:43:11.468647 1 route.go:620] Updating iptable rules for remote cluster's Endpoint "cluster2-submariner-cable-cluster2-172-17-0-5" with subnets [100.92.0.0/16 10.242.0.0/16]
  3759. 2020-10-08T11:45:08.2106183Z [cluster1] I1008 11:43:41.459398 1 route.go:736] Enqueueing endpoint for route controller {"kind":"Endpoint","apiVersion":"submariner.io/v1","metadata":{"name":"cluster1-submariner-cable-cluster1-172-17-0-9","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster1-submariner-cable-cluster1-172-17-0-9","uid":"23aee4db-4473-4b17-8f51-aedd32e3a5ff","resourceVersion":"1212","generation":1,"creationTimestamp":"2020-10-08T11:34:56Z"},"spec":{"cluster_id":"cluster1","cable_name":"submariner-cable-cluster1-172-17-0-9","hostname":"cluster1-worker","subnets":["100.91.0.0/16","10.241.0.0/16"],"private_ip":"172.17.0.9","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
  3760. 2020-10-08T11:45:08.2112768Z [cluster1] I1008 11:43:41.459617 1 route.go:736] Enqueueing endpoint for route controller {"metadata":{"name":"cluster2-submariner-cable-cluster2-172-17-0-5","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster2-submariner-cable-cluster2-172-17-0-5","uid":"474bcd76-5748-4372-9832-4a15a4b1804d","resourceVersion":"1332","generation":1,"creationTimestamp":"2020-10-08T11:35:26Z"},"spec":{"cluster_id":"cluster2","cable_name":"submariner-cable-cluster2-172-17-0-5","hostname":"cluster2-worker","subnets":["100.92.0.0/16","10.242.0.0/16"],"private_ip":"172.17.0.5","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
  3761. 2020-10-08T11:45:08.2117455Z [cluster1] I1008 11:43:41.467295 1 route.go:634] Processing local Endpoint cluster1-submariner-cable-cluster1-172-17-0-9 with IP 172.17.0.9, Host cluster1-worker for this cluster
  3762. 2020-10-08T11:45:08.2118735Z [cluster1] I1008 11:43:41.467705 1 route.go:851] Processing route {Ifindex: 15 Dst: 240.0.0.0/8 Src: 240.17.0.9 Gw: <nil> Flags: [] Table: 254}
  3763. 2020-10-08T11:45:08.2119296Z [cluster1] I1008 11:43:41.467737 1 route.go:854] Found nil gw or dst
  3764. 2020-10-08T11:45:08.2119835Z [cluster1] I1008 11:43:41.467746 1 route.go:645] This route agent is running on the active gateway node
  3765. 2020-10-08T11:45:08.2120751Z [cluster1] I1008 11:43:41.467755 1 route.go:653] Creating the vxlan interface: vx-submariner on the gateway node
  3766. 2020-10-08T11:45:08.2121584Z [cluster1] I1008 11:43:41.467992 1 vxlan.go:61] VxLAN interface already exists with same configuration.
  3767. 2020-10-08T11:45:08.2122272Z [cluster1] I1008 11:43:41.468078 1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.9 00:00:00:00:00:00
  3768. 2020-10-08T11:45:08.2122902Z [cluster1] I1008 11:43:41.468115 1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.7 00:00:00:00:00:00
  3769. 2020-10-08T11:45:08.2123846Z [cluster1] I1008 11:43:41.468208 1 route.go:478] Successfully configured rp_filter to loose mode(2) on vx-submariner
  3770. 2020-10-08T11:45:08.2125458Z [cluster1] I1008 11:43:41.470104 1 route.go:620] Updating iptable rules for remote cluster's Endpoint "cluster2-submariner-cable-cluster2-172-17-0-5" with subnets [100.92.0.0/16 10.242.0.0/16]
  3771. 2020-10-08T11:45:08.2130784Z [cluster1] I1008 11:44:11.459582 1 route.go:736] Enqueueing endpoint for route controller {"kind":"Endpoint","apiVersion":"submariner.io/v1","metadata":{"name":"cluster1-submariner-cable-cluster1-172-17-0-9","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster1-submariner-cable-cluster1-172-17-0-9","uid":"23aee4db-4473-4b17-8f51-aedd32e3a5ff","resourceVersion":"1212","generation":1,"creationTimestamp":"2020-10-08T11:34:56Z"},"spec":{"cluster_id":"cluster1","cable_name":"submariner-cable-cluster1-172-17-0-9","hostname":"cluster1-worker","subnets":["100.91.0.0/16","10.241.0.0/16"],"private_ip":"172.17.0.9","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
  3772. 2020-10-08T11:45:08.2136927Z [cluster1] I1008 11:44:11.459823 1 route.go:736] Enqueueing endpoint for route controller {"metadata":{"name":"cluster2-submariner-cable-cluster2-172-17-0-5","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster2-submariner-cable-cluster2-172-17-0-5","uid":"474bcd76-5748-4372-9832-4a15a4b1804d","resourceVersion":"1332","generation":1,"creationTimestamp":"2020-10-08T11:35:26Z"},"spec":{"cluster_id":"cluster2","cable_name":"submariner-cable-cluster2-172-17-0-5","hostname":"cluster2-worker","subnets":["100.92.0.0/16","10.242.0.0/16"],"private_ip":"172.17.0.5","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
  3773. 2020-10-08T11:45:08.2140615Z [cluster1] I1008 11:44:11.467366 1 route.go:634] Processing local Endpoint cluster1-submariner-cable-cluster1-172-17-0-9 with IP 172.17.0.9, Host cluster1-worker for this cluster
  3774. 2020-10-08T11:45:08.2141741Z [cluster1] I1008 11:44:11.467580 1 route.go:851] Processing route {Ifindex: 15 Dst: 240.0.0.0/8 Src: 240.17.0.9 Gw: <nil> Flags: [] Table: 254}
  3775. 2020-10-08T11:45:08.2142230Z [cluster1] I1008 11:44:11.467606 1 route.go:854] Found nil gw or dst
  3776. 2020-10-08T11:45:08.2142706Z [cluster1] I1008 11:44:11.467613 1 route.go:645] This route agent is running on the active gateway node
  3777. 2020-10-08T11:45:08.2143503Z [cluster1] I1008 11:44:11.467620 1 route.go:653] Creating the vxlan interface: vx-submariner on the gateway node
  3778. 2020-10-08T11:45:08.2144231Z [cluster1] I1008 11:44:11.467814 1 vxlan.go:61] VxLAN interface already exists with same configuration.
  3779. 2020-10-08T11:45:08.2145005Z [cluster1] I1008 11:44:11.467893 1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.9 00:00:00:00:00:00
  3780. 2020-10-08T11:45:08.2145628Z [cluster1] I1008 11:44:11.467962 1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.7 00:00:00:00:00:00
  3781. 2020-10-08T11:45:08.2146570Z [cluster1] I1008 11:44:11.468058 1 route.go:478] Successfully configured rp_filter to loose mode(2) on vx-submariner
  3782. 2020-10-08T11:45:08.2148450Z [cluster1] I1008 11:44:11.470002 1 route.go:620] Updating iptable rules for remote cluster's Endpoint "cluster2-submariner-cable-cluster2-172-17-0-5" with subnets [100.92.0.0/16 10.242.0.0/16]
  3783. 2020-10-08T11:45:08.2153779Z [cluster1] I1008 11:44:41.459773 1 route.go:736] Enqueueing endpoint for route controller {"kind":"Endpoint","apiVersion":"submariner.io/v1","metadata":{"name":"cluster1-submariner-cable-cluster1-172-17-0-9","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster1-submariner-cable-cluster1-172-17-0-9","uid":"23aee4db-4473-4b17-8f51-aedd32e3a5ff","resourceVersion":"1212","generation":1,"creationTimestamp":"2020-10-08T11:34:56Z"},"spec":{"cluster_id":"cluster1","cable_name":"submariner-cable-cluster1-172-17-0-9","hostname":"cluster1-worker","subnets":["100.91.0.0/16","10.241.0.0/16"],"private_ip":"172.17.0.9","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
  3784. 2020-10-08T11:45:08.2161107Z [cluster1] I1008 11:44:41.459916 1 route.go:736] Enqueueing endpoint for route controller {"metadata":{"name":"cluster2-submariner-cable-cluster2-172-17-0-5","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster2-submariner-cable-cluster2-172-17-0-5","uid":"474bcd76-5748-4372-9832-4a15a4b1804d","resourceVersion":"1332","generation":1,"creationTimestamp":"2020-10-08T11:35:26Z"},"spec":{"cluster_id":"cluster2","cable_name":"submariner-cable-cluster2-172-17-0-5","hostname":"cluster2-worker","subnets":["100.92.0.0/16","10.242.0.0/16"],"private_ip":"172.17.0.5","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
  3785. 2020-10-08T11:45:08.2166091Z [cluster1] I1008 11:44:41.466961 1 route.go:634] Processing local Endpoint cluster1-submariner-cable-cluster1-172-17-0-9 with IP 172.17.0.9, Host cluster1-worker for this cluster
  3786. 2020-10-08T11:45:08.2167219Z [cluster1] I1008 11:44:41.467147 1 route.go:851] Processing route {Ifindex: 15 Dst: 240.0.0.0/8 Src: 240.17.0.9 Gw: <nil> Flags: [] Table: 254}
  3787. 2020-10-08T11:45:08.2167710Z [cluster1] I1008 11:44:41.467170 1 route.go:854] Found nil gw or dst
  3788. 2020-10-08T11:45:08.2168179Z [cluster1] I1008 11:44:41.467177 1 route.go:645] This route agent is running on the active gateway node
  3789. 2020-10-08T11:45:08.2168981Z [cluster1] I1008 11:44:41.467183 1 route.go:653] Creating the vxlan interface: vx-submariner on the gateway node
  3790. 2020-10-08T11:45:08.2169646Z [cluster1] I1008 11:44:41.467283 1 vxlan.go:61] VxLAN interface already exists with same configuration.
  3791. 2020-10-08T11:45:08.2170243Z [cluster1] I1008 11:44:41.467315 1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.9 00:00:00:00:00:00
  3792. 2020-10-08T11:45:08.2170794Z [cluster1] I1008 11:44:41.467343 1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.7 00:00:00:00:00:00
  3793. 2020-10-08T11:45:08.2171606Z [cluster1] I1008 11:44:41.467390 1 route.go:478] Successfully configured rp_filter to loose mode(2) on vx-submariner
  3794. 2020-10-08T11:45:08.2173435Z [cluster1] I1008 11:44:41.468675 1 route.go:620] Updating iptable rules for remote cluster's Endpoint "cluster2-submariner-cable-cluster2-172-17-0-5" with subnets [100.92.0.0/16 10.242.0.0/16]
  3795. 2020-10-08T11:45:08.9237020Z [cluster1] +++++++++++++++++++++: Logs for Pod submariner-routeagent-wmmbs in namespace submariner-operator :++++++++++++++++++++++
  3796. 2020-10-08T11:45:08.9238617Z [cluster1] + trap 'exit 1' SIGTERM SIGINT
  3797. 2020-10-08T11:45:08.9239090Z [cluster1] + SUBMARINER_VERBOSITY=2
  3798. 2020-10-08T11:45:08.9239664Z [cluster1] + '[' false == true ']'
  3799. 2020-10-08T11:45:08.9240129Z [cluster1] + DEBUG=-v=2
  3800. 2020-10-08T11:45:08.9240683Z [cluster1] + for f in iptables-save iptables
  3801. 2020-10-08T11:45:08.9247167Z [cluster1] ++ find_iptables_on_host iptables-save
  3802. 2020-10-08T11:45:08.9248038Z [cluster1] ++ chroot /host test -x /usr/sbin/iptables-save
  3803. 2020-10-08T11:45:08.9248460Z [cluster1] ++ echo /usr/sbin
  3804. 2020-10-08T11:45:08.9248765Z [cluster1] ++ return
  3805. 2020-10-08T11:45:08.9249090Z [cluster1] + location=/usr/sbin
  3806. 2020-10-08T11:45:08.9249593Z [cluster1] + '[' /usr/sbin '!=' unknown ']'
  3807. 2020-10-08T11:45:08.9250279Z [cluster1] + echo 'iptables-save is present on the host at /usr/sbin/iptables-save'
  3808. 2020-10-08T11:45:08.9251110Z [cluster1] + sed 's!@@PATH@@!/usr/sbin!' /usr/sbin/iptables-wrapper.in
  3809. 2020-10-08T11:45:08.9251795Z [cluster1] + for f in iptables-save iptables
  3810. 2020-10-08T11:45:08.9258120Z [cluster1] ++ find_iptables_on_host iptables
  3811. 2020-10-08T11:45:08.9258755Z [cluster1] ++ chroot /host test -x /usr/sbin/iptables
  3812. 2020-10-08T11:45:08.9259095Z [cluster1] ++ echo /usr/sbin
  3813. 2020-10-08T11:45:08.9259365Z [cluster1] ++ return
  3814. 2020-10-08T11:45:08.9259649Z [cluster1] + location=/usr/sbin
  3815. 2020-10-08T11:45:08.9260073Z [cluster1] + '[' /usr/sbin '!=' unknown ']'
  3816. 2020-10-08T11:45:08.9260624Z [cluster1] + echo 'iptables is present on the host at /usr/sbin/iptables'
  3817. 2020-10-08T11:45:08.9261290Z [cluster1] + sed 's!@@PATH@@!/usr/sbin!' /usr/sbin/iptables-wrapper.in
  3818. 2020-10-08T11:45:08.9262024Z [cluster1] + exec submariner-route-agent -v=2 -alsologtostderr
  3819. 2020-10-08T11:45:08.9262929Z [cluster1] iptables-save is present on the host at /usr/sbin/iptables-save
  3820. 2020-10-08T11:45:08.9263455Z [cluster1] iptables is present on the host at /usr/sbin/iptables
  3821. 2020-10-08T11:45:08.9264157Z [cluster1] I1008 11:34:55.507289 1 main.go:47] Starting submariner-route-agent
  3822. 2020-10-08T11:45:08.9265155Z [cluster1] W1008 11:34:55.507604 1 client_config.go:543] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work.
  3823. 2020-10-08T11:45:08.9266106Z [cluster1] I1008 11:34:55.513305 1 route.go:195] Starting Route Controller. ClusterID: cluster1, localClusterCIDR: [10.241.0.0/16], localServiceCIDR: [100.91.0.0/16]
  3824. 2020-10-08T11:45:08.9266866Z [cluster1] I1008 11:34:55.513323 1 route.go:199] Waiting for Endpoint informer caches to sync.
  3825. 2020-10-08T11:45:08.9267630Z [cluster1] I1008 11:34:55.540388 1 route.go:751] Enqueueing sm-route-agent-pod event, ip: 172.17.0.7
  3826. 2020-10-08T11:45:08.9268424Z [cluster1] I1008 11:34:55.540404 1 route.go:751] Enqueueing sm-route-agent-pod event, ip: 172.17.0.9
  3827. 2020-10-08T11:45:08.9268964Z [cluster1] I1008 11:34:55.616314 1 driver.go:37] Interface "lo" has "127.0.0.1" address
  3828. 2020-10-08T11:45:08.9269438Z [cluster1] I1008 11:34:55.616407 1 driver.go:37] Interface "weave" has "10.241.128.0" address
  3829. 2020-10-08T11:45:08.9269987Z [cluster1] I1008 11:34:55.616414 1 driver.go:42] Found CNI Interface "weave" that has IP "10.241.128.0" from ClusterCIDR "10.241.0.0/16"
  3830. 2020-10-08T11:45:08.9270677Z [cluster1] I1008 11:34:55.616492 1 driver.go:60] Successfully configured rp_filter to loose mode(2) on cniInterface "weave"
  3831. 2020-10-08T11:45:08.9271613Z [cluster1] I1008 11:34:55.714863 1 iptables.go:22] Install/ensure SUBMARINER-POSTROUTING chain exists
  3832. 2020-10-08T11:45:08.9272639Z [cluster1] I1008 11:34:56.605371 1 iptables.go:28] Insert SUBMARINER-POSTROUTING rule that has rules for inter-cluster traffic
  3833. 2020-10-08T11:45:08.9273583Z [cluster1] I1008 11:34:56.634556 1 iptables.go:34] Install/ensure SUBMARINER-INPUT chain exists
  3834. 2020-10-08T11:45:08.9274435Z [cluster1] I1008 11:34:56.717507 1 iptables.go:45] Allow VxLAN incoming traffic in SUBMARINER-INPUT Chain
  3835. 2020-10-08T11:45:08.9275634Z [cluster1] I1008 11:34:56.738783 1 iptables.go:53] Insert rule to allow traffic over vx-submariner interface in FORWARDing Chain
  3836. 2020-10-08T11:45:08.9277184Z [cluster1] I1008 11:34:56.762287 1 iptables.go:65] Installing rule for host network to remote cluster communication: -s 240.0.0.0/8 -o vx-submariner -j SNAT --to 10.241.128.0
  3837. 2020-10-08T11:45:08.9281573Z [cluster1] I1008 11:34:56.802632 1 route.go:736] Enqueueing endpoint for route controller {"metadata":{"name":"cluster1-submariner-cable-cluster1-172-17-0-9","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster1-submariner-cable-cluster1-172-17-0-9","uid":"23aee4db-4473-4b17-8f51-aedd32e3a5ff","resourceVersion":"1212","generation":1,"creationTimestamp":"2020-10-08T11:34:56Z"},"spec":{"cluster_id":"cluster1","cable_name":"submariner-cable-cluster1-172-17-0-9","hostname":"cluster1-worker","subnets":["100.91.0.0/16","10.241.0.0/16"],"private_ip":"172.17.0.9","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
  3838. 2020-10-08T11:45:08.9285297Z [cluster1] I1008 11:34:56.818585 1 route.go:257] In cluster1, podIP of submariner-route-agent[0] is 172.17.0.9
  3839. 2020-10-08T11:45:08.9286527Z [cluster1] I1008 11:34:56.818601 1 route.go:257] In cluster1, podIP of submariner-route-agent[1] is 172.17.0.7
  3840. 2020-10-08T11:45:08.9287683Z [cluster1] I1008 11:34:56.818607 1 route.go:271] Hostname is "cluster1-worker2" and routeAgentNodeName is "cluster1-worker2"
  3841. 2020-10-08T11:45:08.9288881Z [cluster1] I1008 11:34:56.877716 1 driver.go:87] Successfully annotated node "cluster1-worker2" with cniIfaceIP "10.241.128.0"
  3842. 2020-10-08T11:45:08.9289585Z [cluster1] I1008 11:34:56.877738 1 route.go:288] Route agent workers started
  3843. 2020-10-08T11:45:08.9291061Z [cluster1] I1008 11:34:56.882050 1 route.go:634] Processing local Endpoint cluster1-submariner-cable-cluster1-172-17-0-9 with IP 172.17.0.9, Host cluster1-worker for this cluster
  3844. 2020-10-08T11:45:08.9292467Z [cluster1] I1008 11:34:56.882070 1 route.go:677] This route agent is running on a non-gateway/non-active node
  3845. 2020-10-08T11:45:08.9293341Z [cluster1] I1008 11:34:56.882079 1 route.go:689] Creating the vxlan interface vx-submariner with gateway node IP 172.17.0.9
  3846. 2020-10-08T11:45:08.9294486Z [cluster1] I1008 11:34:56.897157 1 route.go:540] Processing submariner-route-agent pod "submariner-routeagent-wmmbs" with host IP "172.17.0.7", pod IP "172.17.0.7"
  3847. 2020-10-08T11:45:08.9295287Z [cluster1] I1008 11:34:56.916992 1 route.go:866] Reconciling routes to gw: 240.17.0.9
  3848. 2020-10-08T11:45:08.9295824Z [cluster1] I1008 11:34:56.917132 1 route.go:882] Processing route {Ifindex: 16 Dst: 240.0.0.0/8 Src: 240.17.0.7 Gw: <nil> Flags: [] Table: 254}
  3849. 2020-10-08T11:45:08.9296314Z [cluster1] I1008 11:34:56.917164 1 route.go:885] Found nil gw or dst
  3850. 2020-10-08T11:45:08.9297440Z [cluster1] I1008 11:34:56.917245 1 route.go:715] Successfully processed local Endpoint "cluster1-submariner-cable-cluster1-172-17-0-9"
  3851. 2020-10-08T11:45:08.9299317Z [cluster1] I1008 11:34:56.917264 1 route.go:568] Successfully processed submariner-route-agent pod "submariner-routeagent-wmmbs"
  3852. 2020-10-08T11:45:08.9300897Z [cluster1] I1008 11:34:56.919700 1 route.go:540] Processing submariner-route-agent pod "submariner-routeagent-cljq7" with host IP "172.17.0.9", pod IP "172.17.0.9"
  3853. 2020-10-08T11:45:08.9302625Z [cluster1] I1008 11:34:56.919722 1 route.go:568] Successfully processed submariner-route-agent pod "submariner-routeagent-cljq7"
  3854. 2020-10-08T11:45:08.9307560Z [cluster1] I1008 11:35:25.523908 1 route.go:736] Enqueueing endpoint for route controller {"metadata":{"name":"cluster1-submariner-cable-cluster1-172-17-0-9","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster1-submariner-cable-cluster1-172-17-0-9","uid":"23aee4db-4473-4b17-8f51-aedd32e3a5ff","resourceVersion":"1212","generation":1,"creationTimestamp":"2020-10-08T11:34:56Z"},"spec":{"cluster_id":"cluster1","cable_name":"submariner-cable-cluster1-172-17-0-9","hostname":"cluster1-worker","subnets":["100.91.0.0/16","10.241.0.0/16"],"private_ip":"172.17.0.9","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
  3855. 2020-10-08T11:45:08.9312133Z [cluster1] I1008 11:35:25.532928 1 route.go:634] Processing local Endpoint cluster1-submariner-cable-cluster1-172-17-0-9 with IP 172.17.0.9, Host cluster1-worker for this cluster
  3856. 2020-10-08T11:45:08.9313439Z [cluster1] I1008 11:35:25.532949 1 route.go:677] This route agent is running on a non-gateway/non-active node
  3857. 2020-10-08T11:45:08.9314282Z [cluster1] I1008 11:35:25.532958 1 route.go:689] Creating the vxlan interface vx-submariner with gateway node IP 172.17.0.9
  3858. 2020-10-08T11:45:08.9315111Z [cluster1] I1008 11:35:25.533152 1 vxlan.go:61] VxLAN interface already exists with same configuration.
  3859. 2020-10-08T11:45:08.9315740Z [cluster1] I1008 11:35:25.540590 1 route.go:866] Reconciling routes to gw: 240.17.0.9
  3860. 2020-10-08T11:45:08.9316351Z [cluster1] I1008 11:35:25.540708 1 route.go:882] Processing route {Ifindex: 16 Dst: 240.0.0.0/8 Src: 240.17.0.7 Gw: <nil> Flags: [] Table: 254}
  3861. 2020-10-08T11:45:08.9316979Z [cluster1] I1008 11:35:25.540727 1 route.go:885] Found nil gw or dst
  3862. 2020-10-08T11:45:08.9318287Z [cluster1] I1008 11:35:25.540802 1 route.go:715] Successfully processed local Endpoint "cluster1-submariner-cable-cluster1-172-17-0-9"
  3863. 2020-10-08T11:45:08.9352809Z [cluster1] I1008 11:35:26.128067 1 route.go:736] Enqueueing endpoint for route controller {"metadata":{"name":"cluster2-submariner-cable-cluster2-172-17-0-5","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster2-submariner-cable-cluster2-172-17-0-5","uid":"474bcd76-5748-4372-9832-4a15a4b1804d","resourceVersion":"1332","generation":1,"creationTimestamp":"2020-10-08T11:35:26Z"},"spec":{"cluster_id":"cluster2","cable_name":"submariner-cable-cluster2-172-17-0-5","hostname":"cluster2-worker","subnets":["100.92.0.0/16","10.242.0.0/16"],"private_ip":"172.17.0.5","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
  3864. 2020-10-08T11:45:08.9357260Z [cluster1] I1008 11:35:26.155381 1 route.go:620] Updating iptable rules for remote cluster's Endpoint "cluster2-submariner-cable-cluster2-172-17-0-5" with subnets [100.92.0.0/16 10.242.0.0/16]
  3865. 2020-10-08T11:45:08.9359083Z [cluster1] I1008 11:35:26.163096 1 iptables.go:83] Installing iptables rule for outgoing traffic: -s 10.241.0.0/16 -d 100.92.0.0/16 -j ACCEPT
  3866. 2020-10-08T11:45:08.9360183Z [cluster1] I1008 11:35:26.173175 1 iptables.go:91] Installing iptables rule for incoming traffic: -s 100.92.0.0/16 -d 10.241.0.0/16 -j ACCEPT
  3867. 2020-10-08T11:45:08.9361274Z [cluster1] I1008 11:35:26.200727 1 iptables.go:83] Installing iptables rule for outgoing traffic: -s 10.241.0.0/16 -d 10.242.0.0/16 -j ACCEPT
  3868. 2020-10-08T11:45:08.9362362Z [cluster1] I1008 11:35:26.214318 1 iptables.go:91] Installing iptables rule for incoming traffic: -s 10.242.0.0/16 -d 10.241.0.0/16 -j ACCEPT
  3869. 2020-10-08T11:45:08.9367128Z [cluster1] I1008 11:35:55.524031 1 route.go:736] Enqueueing endpoint for route controller {"metadata":{"name":"cluster1-submariner-cable-cluster1-172-17-0-9","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster1-submariner-cable-cluster1-172-17-0-9","uid":"23aee4db-4473-4b17-8f51-aedd32e3a5ff","resourceVersion":"1212","generation":1,"creationTimestamp":"2020-10-08T11:34:56Z"},"spec":{"cluster_id":"cluster1","cable_name":"submariner-cable-cluster1-172-17-0-9","hostname":"cluster1-worker","subnets":["100.91.0.0/16","10.241.0.0/16"],"private_ip":"172.17.0.9","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
  3870. 2020-10-08T11:45:08.9374941Z [cluster1] I1008 11:35:55.524095 1 route.go:736] Enqueueing endpoint for route controller {"metadata":{"name":"cluster2-submariner-cable-cluster2-172-17-0-5","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster2-submariner-cable-cluster2-172-17-0-5","uid":"474bcd76-5748-4372-9832-4a15a4b1804d","resourceVersion":"1332","generation":1,"creationTimestamp":"2020-10-08T11:35:26Z"},"spec":{"cluster_id":"cluster2","cable_name":"submariner-cable-cluster2-172-17-0-5","hostname":"cluster2-worker","subnets":["100.92.0.0/16","10.242.0.0/16"],"private_ip":"172.17.0.5","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
  3871. 2020-10-08T11:45:08.9379198Z [cluster1] I1008 11:35:55.531967 1 route.go:634] Processing local Endpoint cluster1-submariner-cable-cluster1-172-17-0-9 with IP 172.17.0.9, Host cluster1-worker for this cluster
  3872. 2020-10-08T11:45:08.9380701Z [cluster1] I1008 11:35:55.531986 1 route.go:677] This route agent is running on a non-gateway/non-active node
  3873. 2020-10-08T11:45:08.9381677Z [cluster1] I1008 11:35:55.531995 1 route.go:689] Creating the vxlan interface vx-submariner with gateway node IP 172.17.0.9
  3874. 2020-10-08T11:45:08.9382432Z [cluster1] I1008 11:35:55.532179 1 vxlan.go:61] VxLAN interface already exists with same configuration.
  3875. 2020-10-08T11:45:08.9383140Z [cluster1] I1008 11:35:55.535889 1 route.go:866] Reconciling routes to gw: 240.17.0.9
  3876. 2020-10-08T11:45:08.9383752Z [cluster1] I1008 11:35:55.536363 1 route.go:882] Processing route {Ifindex: 16 Dst: 240.0.0.0/8 Src: 240.17.0.7 Gw: <nil> Flags: [] Table: 254}
  3877. 2020-10-08T11:45:08.9384307Z [cluster1] I1008 11:35:55.536387 1 route.go:885] Found nil gw or dst
  3878. 2020-10-08T11:45:08.9385846Z [cluster1] I1008 11:35:55.539306 1 route.go:715] Successfully processed local Endpoint "cluster1-submariner-cable-cluster1-172-17-0-9"
  3879. 2020-10-08T11:45:08.9388022Z [cluster1] I1008 11:35:55.541172 1 route.go:620] Updating iptable rules for remote cluster's Endpoint "cluster2-submariner-cable-cluster2-172-17-0-5" with subnets [100.92.0.0/16 10.242.0.0/16]
  3880. 2020-10-08T11:45:08.9393247Z [cluster1] I1008 11:36:25.524112 1 route.go:736] Enqueueing endpoint for route controller {"metadata":{"name":"cluster1-submariner-cable-cluster1-172-17-0-9","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster1-submariner-cable-cluster1-172-17-0-9","uid":"23aee4db-4473-4b17-8f51-aedd32e3a5ff","resourceVersion":"1212","generation":1,"creationTimestamp":"2020-10-08T11:34:56Z"},"spec":{"cluster_id":"cluster1","cable_name":"submariner-cable-cluster1-172-17-0-9","hostname":"cluster1-worker","subnets":["100.91.0.0/16","10.241.0.0/16"],"private_ip":"172.17.0.9","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
  3881. 2020-10-08T11:45:08.9400771Z [cluster1] I1008 11:36:25.524201 1 route.go:736] Enqueueing endpoint for route controller {"metadata":{"name":"cluster2-submariner-cable-cluster2-172-17-0-5","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster2-submariner-cable-cluster2-172-17-0-5","uid":"474bcd76-5748-4372-9832-4a15a4b1804d","resourceVersion":"1332","generation":1,"creationTimestamp":"2020-10-08T11:35:26Z"},"spec":{"cluster_id":"cluster2","cable_name":"submariner-cable-cluster2-172-17-0-5","hostname":"cluster2-worker","subnets":["100.92.0.0/16","10.242.0.0/16"],"private_ip":"172.17.0.5","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
  3882. 2020-10-08T11:45:08.9405298Z [cluster1] I1008 11:36:25.531580 1 route.go:634] Processing local Endpoint cluster1-submariner-cable-cluster1-172-17-0-9 with IP 172.17.0.9, Host cluster1-worker for this cluster
  3883. 2020-10-08T11:45:08.9406920Z [cluster1] I1008 11:36:25.531621 1 route.go:677] This route agent is running on a non-gateway/non-active node
  3884. 2020-10-08T11:45:08.9407967Z [cluster1] I1008 11:36:25.531631 1 route.go:689] Creating the vxlan interface vx-submariner with gateway node IP 172.17.0.9
  3885. 2020-10-08T11:45:08.9408785Z [cluster1] I1008 11:36:25.531800 1 vxlan.go:61] VxLAN interface already exists with same configuration.
  3886. 2020-10-08T11:45:08.9409462Z [cluster1] I1008 11:36:25.537129 1 route.go:866] Reconciling routes to gw: 240.17.0.9
  3887. 2020-10-08T11:45:08.9411865Z [cluster1] I1008 11:36:25.537303 1 route.go:882] Processing route {Ifindex: 16 Dst: 10.242.0.0/16 Src: <nil> Gw: 240.17.0.9 Flags: [] Table: 254}
  3888. 2020-10-08T11:45:08.9412780Z [cluster1] I1008 11:36:25.537335 1 route.go:888] Found route {Ifindex: 16 Dst: 10.242.0.0/16 Src: <nil> Gw: 240.17.0.9 Flags: [] Table: 254} with gw 240.17.0.9 already installed
  3889. 2020-10-08T11:45:08.9413572Z [cluster1] I1008 11:36:25.537345 1 route.go:882] Processing route {Ifindex: 16 Dst: 100.92.0.0/16 Src: <nil> Gw: 240.17.0.9 Flags: [] Table: 254}
  3890. 2020-10-08T11:45:08.9414352Z [cluster1] I1008 11:36:25.537359 1 route.go:888] Found route {Ifindex: 16 Dst: 100.92.0.0/16 Src: <nil> Gw: 240.17.0.9 Flags: [] Table: 254} with gw 240.17.0.9 already installed
  3891. 2020-10-08T11:45:08.9415135Z [cluster1] I1008 11:36:25.537367 1 route.go:882] Processing route {Ifindex: 16 Dst: 240.0.0.0/8 Src: 240.17.0.7 Gw: <nil> Flags: [] Table: 254}
  3892. 2020-10-08T11:45:08.9415731Z [cluster1] I1008 11:36:25.537375 1 route.go:885] Found nil gw or dst
  3893. 2020-10-08T11:45:08.9416347Z [cluster1] I1008 11:36:25.537425 1 route.go:924] Found equivalent route, not adding
  3894. 2020-10-08T11:45:08.9416944Z [cluster1] I1008 11:36:25.537433 1 route.go:924] Found equivalent route, not adding
  3895. 2020-10-08T11:45:08.9418509Z [cluster1] I1008 11:36:25.537461 1 route.go:715] Successfully processed local Endpoint "cluster1-submariner-cable-cluster1-172-17-0-9"
  3896. 2020-10-08T11:45:08.9421011Z [cluster1] I1008 11:36:25.539129 1 route.go:620] Updating iptable rules for remote cluster's Endpoint "cluster2-submariner-cable-cluster2-172-17-0-5" with subnets [100.92.0.0/16 10.242.0.0/16]
  3897. 2020-10-08T11:45:08.9426510Z [cluster1] I1008 11:36:55.524219 1 route.go:736] Enqueueing endpoint for route controller {"metadata":{"name":"cluster1-submariner-cable-cluster1-172-17-0-9","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster1-submariner-cable-cluster1-172-17-0-9","uid":"23aee4db-4473-4b17-8f51-aedd32e3a5ff","resourceVersion":"1212","generation":1,"creationTimestamp":"2020-10-08T11:34:56Z"},"spec":{"cluster_id":"cluster1","cable_name":"submariner-cable-cluster1-172-17-0-9","hostname":"cluster1-worker","subnets":["100.91.0.0/16","10.241.0.0/16"],"private_ip":"172.17.0.9","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
  3898. 2020-10-08T11:45:08.9434787Z [cluster1] I1008 11:36:55.524288 1 route.go:736] Enqueueing endpoint for route controller {"metadata":{"name":"cluster2-submariner-cable-cluster2-172-17-0-5","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster2-submariner-cable-cluster2-172-17-0-5","uid":"474bcd76-5748-4372-9832-4a15a4b1804d","resourceVersion":"1332","generation":1,"creationTimestamp":"2020-10-08T11:35:26Z"},"spec":{"cluster_id":"cluster2","cable_name":"submariner-cable-cluster2-172-17-0-5","hostname":"cluster2-worker","subnets":["100.92.0.0/16","10.242.0.0/16"],"private_ip":"172.17.0.5","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
  3899. 2020-10-08T11:45:08.9439873Z [cluster1] I1008 11:36:55.531795 1 route.go:634] Processing local Endpoint cluster1-submariner-cable-cluster1-172-17-0-9 with IP 172.17.0.9, Host cluster1-worker for this cluster
  3900. 2020-10-08T11:45:08.9448869Z [cluster1] I1008 11:36:55.531817 1 route.go:677] This route agent is running on a non-gateway/non-active node
  3901. 2020-10-08T11:45:08.9449881Z [cluster1] I1008 11:36:55.531825 1 route.go:689] Creating the vxlan interface vx-submariner with gateway node IP 172.17.0.9
  3902. 2020-10-08T11:45:08.9451832Z [cluster1] I1008 11:36:55.532077 1 vxlan.go:61] VxLAN interface already exists with same configuration.
  3903. 2020-10-08T11:45:08.9452576Z [cluster1] I1008 11:36:55.535168 1 route.go:866] Reconciling routes to gw: 240.17.0.9
  3904. 2020-10-08T11:45:08.9453195Z [cluster1] I1008 11:36:55.535293 1 route.go:882] Processing route {Ifindex: 16 Dst: 10.242.0.0/16 Src: <nil> Gw: 240.17.0.9 Flags: [] Table: 254}
  3905. 2020-10-08T11:45:08.9454167Z [cluster1] I1008 11:36:55.535314 1 route.go:888] Found route {Ifindex: 16 Dst: 10.242.0.0/16 Src: <nil> Gw: 240.17.0.9 Flags: [] Table: 254} with gw 240.17.0.9 already installed
  3906. 2020-10-08T11:45:08.9456071Z [cluster1] I1008 11:36:55.535322 1 route.go:882] Processing route {Ifindex: 16 Dst: 100.92.0.0/16 Src: <nil> Gw: 240.17.0.9 Flags: [] Table: 254}
  3907. 2020-10-08T11:45:08.9456871Z [cluster1] I1008 11:36:55.535331 1 route.go:888] Found route {Ifindex: 16 Dst: 100.92.0.0/16 Src: <nil> Gw: 240.17.0.9 Flags: [] Table: 254} with gw 240.17.0.9 already installed
  3908. 2020-10-08T11:45:08.9457512Z [cluster1] I1008 11:36:55.535338 1 route.go:882] Processing route {Ifindex: 16 Dst: 240.0.0.0/8 Src: 240.17.0.7 Gw: <nil> Flags: [] Table: 254}
  3909. 2020-10-08T11:45:08.9457995Z [cluster1] I1008 11:36:55.535346 1 route.go:885] Found nil gw or dst
  3910. 2020-10-08T11:45:08.9458608Z [cluster1] I1008 11:36:55.535394 1 route.go:924] Found equivalent route, not adding
  3911. 2020-10-08T11:45:08.9459498Z [cluster1] I1008 11:36:55.535401 1 route.go:924] Found equivalent route, not adding
  3912. 2020-10-08T11:45:08.9461044Z [cluster1] I1008 11:36:55.535430 1 route.go:715] Successfully processed local Endpoint "cluster1-submariner-cable-cluster1-172-17-0-9"
  3913. 2020-10-08T11:45:08.9463849Z [cluster1] I1008 11:36:55.536954 1 route.go:620] Updating iptable rules for remote cluster's Endpoint "cluster2-submariner-cable-cluster2-172-17-0-5" with subnets [100.92.0.0/16 10.242.0.0/16]
  3914. 2020-10-08T11:45:08.9469083Z [cluster1] I1008 11:37:25.524335 1 route.go:736] Enqueueing endpoint for route controller {"metadata":{"name":"cluster1-submariner-cable-cluster1-172-17-0-9","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster1-submariner-cable-cluster1-172-17-0-9","uid":"23aee4db-4473-4b17-8f51-aedd32e3a5ff","resourceVersion":"1212","generation":1,"creationTimestamp":"2020-10-08T11:34:56Z"},"spec":{"cluster_id":"cluster1","cable_name":"submariner-cable-cluster1-172-17-0-9","hostname":"cluster1-worker","subnets":["100.91.0.0/16","10.241.0.0/16"],"private_ip":"172.17.0.9","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
  3915. 2020-10-08T11:45:08.9476638Z [cluster1] I1008 11:37:25.524402 1 route.go:736] Enqueueing endpoint for route controller {"metadata":{"name":"cluster2-submariner-cable-cluster2-172-17-0-5","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster2-submariner-cable-cluster2-172-17-0-5","uid":"474bcd76-5748-4372-9832-4a15a4b1804d","resourceVersion":"1332","generation":1,"creationTimestamp":"2020-10-08T11:35:26Z"},"spec":{"cluster_id":"cluster2","cable_name":"submariner-cable-cluster2-172-17-0-5","hostname":"cluster2-worker","subnets":["100.92.0.0/16","10.242.0.0/16"],"private_ip":"172.17.0.5","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
  3916. 2020-10-08T11:45:08.9481182Z [cluster1] I1008 11:37:25.532312 1 route.go:634] Processing local Endpoint cluster1-submariner-cable-cluster1-172-17-0-9 with IP 172.17.0.9, Host cluster1-worker for this cluster
  3917. 2020-10-08T11:45:08.9482799Z [cluster1] I1008 11:37:25.532333 1 route.go:677] This route agent is running on a non-gateway/non-active node
  3918. 2020-10-08T11:45:08.9483855Z [cluster1] I1008 11:37:25.532342 1 route.go:689] Creating the vxlan interface vx-submariner with gateway node IP 172.17.0.9
  3919. 2020-10-08T11:45:08.9485177Z [cluster1] I1008 11:37:25.532496 1 vxlan.go:61] VxLAN interface already exists with same configuration.
  3920. 2020-10-08T11:45:08.9486012Z [cluster1] I1008 11:37:25.535971 1 route.go:866] Reconciling routes to gw: 240.17.0.9
  3921. 2020-10-08T11:45:08.9486674Z [cluster1] I1008 11:37:25.536091 1 route.go:882] Processing route {Ifindex: 16 Dst: 10.242.0.0/16 Src: <nil> Gw: 240.17.0.9 Flags: [] Table: 254}
  3922. 2020-10-08T11:45:08.9487462Z [cluster1] I1008 11:37:25.536117 1 route.go:888] Found route {Ifindex: 16 Dst: 10.242.0.0/16 Src: <nil> Gw: 240.17.0.9 Flags: [] Table: 254} with gw 240.17.0.9 already installed
  3923. 2020-10-08T11:45:08.9488358Z [cluster1] I1008 11:37:25.536126 1 route.go:882] Processing route {Ifindex: 16 Dst: 100.92.0.0/16 Src: <nil> Gw: 240.17.0.9 Flags: [] Table: 254}
  3924. 2020-10-08T11:45:08.9489135Z [cluster1] I1008 11:37:25.536180 1 route.go:888] Found route {Ifindex: 16 Dst: 100.92.0.0/16 Src: <nil> Gw: 240.17.0.9 Flags: [] Table: 254} with gw 240.17.0.9 already installed
  3925. 2020-10-08T11:45:08.9489896Z [cluster1] I1008 11:37:25.536192 1 route.go:882] Processing route {Ifindex: 16 Dst: 240.0.0.0/8 Src: 240.17.0.7 Gw: <nil> Flags: [] Table: 254}
  3926. 2020-10-08T11:45:08.9490489Z [cluster1] I1008 11:37:25.536200 1 route.go:885] Found nil gw or dst
  3927. 2020-10-08T11:45:08.9491027Z [cluster1] I1008 11:37:25.536264 1 route.go:924] Found equivalent route, not adding
  3928. 2020-10-08T11:45:08.9491635Z [cluster1] I1008 11:37:25.536279 1 route.go:924] Found equivalent route, not adding
  3929. 2020-10-08T11:45:08.9493269Z [cluster1] I1008 11:37:25.536307 1 route.go:715] Successfully processed local Endpoint "cluster1-submariner-cable-cluster1-172-17-0-9"
  3930. 2020-10-08T11:45:08.9496379Z [cluster1] I1008 11:37:25.538217 1 route.go:620] Updating iptable rules for remote cluster's Endpoint "cluster2-submariner-cable-cluster2-172-17-0-5" with subnets [100.92.0.0/16 10.242.0.0/16]
  3931. 2020-10-08T11:45:08.9501624Z [cluster1] I1008 11:37:55.524465 1 route.go:736] Enqueueing endpoint for route controller {"metadata":{"name":"cluster1-submariner-cable-cluster1-172-17-0-9","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster1-submariner-cable-cluster1-172-17-0-9","uid":"23aee4db-4473-4b17-8f51-aedd32e3a5ff","resourceVersion":"1212","generation":1,"creationTimestamp":"2020-10-08T11:34:56Z"},"spec":{"cluster_id":"cluster1","cable_name":"submariner-cable-cluster1-172-17-0-9","hostname":"cluster1-worker","subnets":["100.91.0.0/16","10.241.0.0/16"],"private_ip":"172.17.0.9","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
  3932. 2020-10-08T11:45:08.9509034Z [cluster1] I1008 11:37:55.524520 1 route.go:736] Enqueueing endpoint for route controller {"metadata":{"name":"cluster2-submariner-cable-cluster2-172-17-0-5","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster2-submariner-cable-cluster2-172-17-0-5","uid":"474bcd76-5748-4372-9832-4a15a4b1804d","resourceVersion":"1332","generation":1,"creationTimestamp":"2020-10-08T11:35:26Z"},"spec":{"cluster_id":"cluster2","cable_name":"submariner-cable-cluster2-172-17-0-5","hostname":"cluster2-worker","subnets":["100.92.0.0/16","10.242.0.0/16"],"private_ip":"172.17.0.5","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
  3933. 2020-10-08T11:45:08.9514611Z [cluster1] I1008 11:37:55.533111 1 route.go:634] Processing local Endpoint cluster1-submariner-cable-cluster1-172-17-0-9 with IP 172.17.0.9, Host cluster1-worker for this cluster
  3934. 2020-10-08T11:45:08.9516121Z [cluster1] I1008 11:37:55.533156 1 route.go:677] This route agent is running on a non-gateway/non-active node
  3935. 2020-10-08T11:45:08.9517101Z [cluster1] I1008 11:37:55.533164 1 route.go:689] Creating the vxlan interface vx-submariner with gateway node IP 172.17.0.9
  3936. 2020-10-08T11:45:08.9517862Z [cluster1] I1008 11:37:55.533380 1 vxlan.go:61] VxLAN interface already exists with same configuration.
  3937. 2020-10-08T11:45:08.9518538Z [cluster1] I1008 11:37:55.536560 1 route.go:866] Reconciling routes to gw: 240.17.0.9
  3938. 2020-10-08T11:45:08.9519071Z [cluster1] I1008 11:37:55.536682 1 route.go:882] Processing route {Ifindex: 16 Dst: 10.242.0.0/16 Src: <nil> Gw: 240.17.0.9 Flags: [] Table: 254}
  3939. 2020-10-08T11:45:08.9519933Z [cluster1] I1008 11:37:55.536704 1 route.go:888] Found route {Ifindex: 16 Dst: 10.242.0.0/16 Src: <nil> Gw: 240.17.0.9 Flags: [] Table: 254} with gw 240.17.0.9 already installed
  3940. 2020-10-08T11:45:08.9520656Z [cluster1] I1008 11:37:55.536713 1 route.go:882] Processing route {Ifindex: 16 Dst: 100.92.0.0/16 Src: <nil> Gw: 240.17.0.9 Flags: [] Table: 254}
  3941. 2020-10-08T11:45:08.9521489Z [cluster1] I1008 11:37:55.536722 1 route.go:888] Found route {Ifindex: 16 Dst: 100.92.0.0/16 Src: <nil> Gw: 240.17.0.9 Flags: [] Table: 254} with gw 240.17.0.9 already installed
  3942. 2020-10-08T11:45:08.9522214Z [cluster1] I1008 11:37:55.536730 1 route.go:882] Processing route {Ifindex: 16 Dst: 240.0.0.0/8 Src: 240.17.0.7 Gw: <nil> Flags: [] Table: 254}
  3943. 2020-10-08T11:45:08.9522951Z [cluster1] I1008 11:37:55.536739 1 route.go:885] Found nil gw or dst
  3944. 2020-10-08T11:45:08.9523487Z [cluster1] I1008 11:37:55.536792 1 route.go:924] Found equivalent route, not adding
  3945. 2020-10-08T11:45:08.9524095Z [cluster1] I1008 11:37:55.536801 1 route.go:924] Found equivalent route, not adding
  3946. 2020-10-08T11:45:08.9525667Z [cluster1] I1008 11:37:55.536848 1 route.go:715] Successfully processed local Endpoint "cluster1-submariner-cable-cluster1-172-17-0-9"
  3947. 2020-10-08T11:45:08.9527859Z [cluster1] I1008 11:37:55.539250 1 route.go:620] Updating iptable rules for remote cluster's Endpoint "cluster2-submariner-cable-cluster2-172-17-0-5" with subnets [100.92.0.0/16 10.242.0.0/16]
  3948. 2020-10-08T11:45:08.9533399Z [cluster1] I1008 11:38:25.524582 1 route.go:736] Enqueueing endpoint for route controller {"metadata":{"name":"cluster1-submariner-cable-cluster1-172-17-0-9","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster1-submariner-cable-cluster1-172-17-0-9","uid":"23aee4db-4473-4b17-8f51-aedd32e3a5ff","resourceVersion":"1212","generation":1,"creationTimestamp":"2020-10-08T11:34:56Z"},"spec":{"cluster_id":"cluster1","cable_name":"submariner-cable-cluster1-172-17-0-9","hostname":"cluster1-worker","subnets":["100.91.0.0/16","10.241.0.0/16"],"private_ip":"172.17.0.9","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
  3949. 2020-10-08T11:45:08.9541319Z [cluster1] I1008 11:38:25.524681 1 route.go:736] Enqueueing endpoint for route controller {"metadata":{"name":"cluster2-submariner-cable-cluster2-172-17-0-5","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster2-submariner-cable-cluster2-172-17-0-5","uid":"474bcd76-5748-4372-9832-4a15a4b1804d","resourceVersion":"1332","generation":1,"creationTimestamp":"2020-10-08T11:35:26Z"},"spec":{"cluster_id":"cluster2","cable_name":"submariner-cable-cluster2-172-17-0-5","hostname":"cluster2-worker","subnets":["100.92.0.0/16","10.242.0.0/16"],"private_ip":"172.17.0.5","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
  3950. 2020-10-08T11:45:08.9546011Z [cluster1] I1008 11:38:25.532678 1 route.go:634] Processing local Endpoint cluster1-submariner-cable-cluster1-172-17-0-9 with IP 172.17.0.9, Host cluster1-worker for this cluster
  3951. 2020-10-08T11:45:08.9547646Z [cluster1] I1008 11:38:25.532699 1 route.go:677] This route agent is running on a non-gateway/non-active node
  3952. 2020-10-08T11:45:08.9548700Z [cluster1] I1008 11:38:25.532708 1 route.go:689] Creating the vxlan interface vx-submariner with gateway node IP 172.17.0.9
  3953. 2020-10-08T11:45:08.9549521Z [cluster1] I1008 11:38:25.532899 1 vxlan.go:61] VxLAN interface already exists with same configuration.
  3954. 2020-10-08T11:45:08.9550187Z [cluster1] I1008 11:38:25.535972 1 route.go:866] Reconciling routes to gw: 240.17.0.9
  3955. 2020-10-08T11:45:08.9550840Z [cluster1] I1008 11:38:25.536101 1 route.go:882] Processing route {Ifindex: 16 Dst: 10.242.0.0/16 Src: <nil> Gw: 240.17.0.9 Flags: [] Table: 254}
  3956. 2020-10-08T11:45:08.9551628Z [cluster1] I1008 11:38:25.536129 1 route.go:888] Found route {Ifindex: 16 Dst: 10.242.0.0/16 Src: <nil> Gw: 240.17.0.9 Flags: [] Table: 254} with gw 240.17.0.9 already installed
  3957. 2020-10-08T11:45:08.9552416Z [cluster1] I1008 11:38:25.536139 1 route.go:882] Processing route {Ifindex: 16 Dst: 100.92.0.0/16 Src: <nil> Gw: 240.17.0.9 Flags: [] Table: 254}
  3958. 2020-10-08T11:45:08.9553386Z [cluster1] I1008 11:38:25.536148 1 route.go:888] Found route {Ifindex: 16 Dst: 100.92.0.0/16 Src: <nil> Gw: 240.17.0.9 Flags: [] Table: 254} with gw 240.17.0.9 already installed
  3959. 2020-10-08T11:45:08.9554118Z [cluster1] I1008 11:38:25.536155 1 route.go:882] Processing route {Ifindex: 16 Dst: 240.0.0.0/8 Src: 240.17.0.7 Gw: <nil> Flags: [] Table: 254}
  3960. 2020-10-08T11:45:08.9554866Z [cluster1] I1008 11:38:25.536163 1 route.go:885] Found nil gw or dst
  3961. 2020-10-08T11:45:08.9555491Z [cluster1] I1008 11:38:25.536211 1 route.go:924] Found equivalent route, not adding
  3962. 2020-10-08T11:45:08.9556052Z [cluster1] I1008 11:38:25.536219 1 route.go:924] Found equivalent route, not adding
  3963. 2020-10-08T11:45:08.9557429Z [cluster1] I1008 11:38:25.536247 1 route.go:715] Successfully processed local Endpoint "cluster1-submariner-cable-cluster1-172-17-0-9"
  3964. 2020-10-08T11:45:08.9560259Z [cluster1] I1008 11:38:25.537935 1 route.go:620] Updating iptable rules for remote cluster's Endpoint "cluster2-submariner-cable-cluster2-172-17-0-5" with subnets [100.92.0.0/16 10.242.0.0/16]
  3965. 2020-10-08T11:45:08.9565184Z [cluster1] I1008 11:38:55.524640 1 route.go:736] Enqueueing endpoint for route controller {"metadata":{"name":"cluster2-submariner-cable-cluster2-172-17-0-5","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster2-submariner-cable-cluster2-172-17-0-5","uid":"474bcd76-5748-4372-9832-4a15a4b1804d","resourceVersion":"1332","generation":1,"creationTimestamp":"2020-10-08T11:35:26Z"},"spec":{"cluster_id":"cluster2","cable_name":"submariner-cable-cluster2-172-17-0-5","hostname":"cluster2-worker","subnets":["100.92.0.0/16","10.242.0.0/16"],"private_ip":"172.17.0.5","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
  3966. 2020-10-08T11:45:08.9573515Z [cluster1] I1008 11:38:55.524696 1 route.go:736] Enqueueing endpoint for route controller {"metadata":{"name":"cluster1-submariner-cable-cluster1-172-17-0-9","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster1-submariner-cable-cluster1-172-17-0-9","uid":"23aee4db-4473-4b17-8f51-aedd32e3a5ff","resourceVersion":"1212","generation":1,"creationTimestamp":"2020-10-08T11:34:56Z"},"spec":{"cluster_id":"cluster1","cable_name":"submariner-cable-cluster1-172-17-0-9","hostname":"cluster1-worker","subnets":["100.91.0.0/16","10.241.0.0/16"],"private_ip":"172.17.0.9","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
  3967. 2020-10-08T11:45:08.9578697Z [cluster1] I1008 11:38:55.535458 1 route.go:620] Updating iptable rules for remote cluster's Endpoint "cluster2-submariner-cable-cluster2-172-17-0-5" with subnets [100.92.0.0/16 10.242.0.0/16]
  3968. 2020-10-08T11:45:08.9581060Z [cluster1] I1008 11:38:55.538683 1 route.go:634] Processing local Endpoint cluster1-submariner-cable-cluster1-172-17-0-9 with IP 172.17.0.9, Host cluster1-worker for this cluster
  3969. 2020-10-08T11:45:08.9582555Z [cluster1] I1008 11:38:55.538701 1 route.go:677] This route agent is running on a non-gateway/non-active node
  3970. 2020-10-08T11:45:08.9583538Z [cluster1] I1008 11:38:55.538709 1 route.go:689] Creating the vxlan interface vx-submariner with gateway node IP 172.17.0.9
  3971. 2020-10-08T11:45:08.9584298Z [cluster1] I1008 11:38:55.538865 1 vxlan.go:61] VxLAN interface already exists with same configuration.
  3972. 2020-10-08T11:45:08.9585225Z [cluster1] I1008 11:38:55.542696 1 route.go:866] Reconciling routes to gw: 240.17.0.9
  3973. 2020-10-08T11:45:08.9585986Z [cluster1] I1008 11:38:55.542824 1 route.go:882] Processing route {Ifindex: 16 Dst: 10.242.0.0/16 Src: <nil> Gw: 240.17.0.9 Flags: [] Table: 254}
  3974. 2020-10-08T11:45:08.9586728Z [cluster1] I1008 11:38:55.542845 1 route.go:888] Found route {Ifindex: 16 Dst: 10.242.0.0/16 Src: <nil> Gw: 240.17.0.9 Flags: [] Table: 254} with gw 240.17.0.9 already installed
  3975. 2020-10-08T11:45:08.9587482Z [cluster1] I1008 11:38:55.542854 1 route.go:882] Processing route {Ifindex: 16 Dst: 100.92.0.0/16 Src: <nil> Gw: 240.17.0.9 Flags: [] Table: 254}
  3976. 2020-10-08T11:45:08.9588203Z [cluster1] I1008 11:38:55.542862 1 route.go:888] Found route {Ifindex: 16 Dst: 100.92.0.0/16 Src: <nil> Gw: 240.17.0.9 Flags: [] Table: 254} with gw 240.17.0.9 already installed
  3977. 2020-10-08T11:45:08.9589348Z [cluster1] I1008 11:38:55.542870 1 route.go:882] Processing route {Ifindex: 16 Dst: 240.0.0.0/8 Src: 240.17.0.7 Gw: <nil> Flags: [] Table: 254}
  3978. 2020-10-08T11:45:08.9589899Z [cluster1] I1008 11:38:55.542877 1 route.go:885] Found nil gw or dst
  3979. 2020-10-08T11:45:08.9590379Z [cluster1] I1008 11:38:55.542926 1 route.go:924] Found equivalent route, not adding
  3980. 2020-10-08T11:45:08.9590938Z [cluster1] I1008 11:38:55.542933 1 route.go:924] Found equivalent route, not adding
  3981. 2020-10-08T11:45:08.9592343Z [cluster1] I1008 11:38:55.542962 1 route.go:715] Successfully processed local Endpoint "cluster1-submariner-cable-cluster1-172-17-0-9"
  3982. 2020-10-08T11:45:08.9597761Z [cluster1] I1008 11:39:25.524779 1 route.go:736] Enqueueing endpoint for route controller {"metadata":{"name":"cluster1-submariner-cable-cluster1-172-17-0-9","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster1-submariner-cable-cluster1-172-17-0-9","uid":"23aee4db-4473-4b17-8f51-aedd32e3a5ff","resourceVersion":"1212","generation":1,"creationTimestamp":"2020-10-08T11:34:56Z"},"spec":{"cluster_id":"cluster1","cable_name":"submariner-cable-cluster1-172-17-0-9","hostname":"cluster1-worker","subnets":["100.91.0.0/16","10.241.0.0/16"],"private_ip":"172.17.0.9","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
  3983. 2020-10-08T11:45:08.9604851Z [cluster1] I1008 11:39:25.524851 1 route.go:736] Enqueueing endpoint for route controller {"metadata":{"name":"cluster2-submariner-cable-cluster2-172-17-0-5","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster2-submariner-cable-cluster2-172-17-0-5","uid":"474bcd76-5748-4372-9832-4a15a4b1804d","resourceVersion":"1332","generation":1,"creationTimestamp":"2020-10-08T11:35:26Z"},"spec":{"cluster_id":"cluster2","cable_name":"submariner-cable-cluster2-172-17-0-5","hostname":"cluster2-worker","subnets":["100.92.0.0/16","10.242.0.0/16"],"private_ip":"172.17.0.5","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
  3984. 2020-10-08T11:45:08.9608521Z [cluster1] I1008 11:39:25.532226 1 route.go:634] Processing local Endpoint cluster1-submariner-cable-cluster1-172-17-0-9 with IP 172.17.0.9, Host cluster1-worker for this cluster
  3985. 2020-10-08T11:45:08.9610013Z [cluster1] I1008 11:39:25.532246 1 route.go:677] This route agent is running on a non-gateway/non-active node
  3986. 2020-10-08T11:45:08.9610989Z [cluster1] I1008 11:39:25.532255 1 route.go:689] Creating the vxlan interface vx-submariner with gateway node IP 172.17.0.9
  3987. 2020-10-08T11:45:08.9611744Z [cluster1] I1008 11:39:25.532414 1 vxlan.go:61] VxLAN interface already exists with same configuration.
  3988. 2020-10-08T11:45:08.9612542Z [cluster1] I1008 11:39:25.535390 1 route.go:866] Reconciling routes to gw: 240.17.0.9
  3989. 2020-10-08T11:45:08.9613170Z [cluster1] I1008 11:39:25.535511 1 route.go:882] Processing route {Ifindex: 16 Dst: 10.242.0.0/16 Src: <nil> Gw: 240.17.0.9 Flags: [] Table: 254}
  3990. 2020-10-08T11:45:08.9613986Z [cluster1] I1008 11:39:25.535532 1 route.go:888] Found route {Ifindex: 16 Dst: 10.242.0.0/16 Src: <nil> Gw: 240.17.0.9 Flags: [] Table: 254} with gw 240.17.0.9 already installed
  3991. 2020-10-08T11:45:08.9614715Z [cluster1] I1008 11:39:25.535540 1 route.go:882] Processing route {Ifindex: 16 Dst: 100.92.0.0/16 Src: <nil> Gw: 240.17.0.9 Flags: [] Table: 254}
  3992. 2020-10-08T11:45:08.9615438Z [cluster1] I1008 11:39:25.535548 1 route.go:888] Found route {Ifindex: 16 Dst: 100.92.0.0/16 Src: <nil> Gw: 240.17.0.9 Flags: [] Table: 254} with gw 240.17.0.9 already installed
  3993. 2020-10-08T11:45:08.9616162Z [cluster1] I1008 11:39:25.535555 1 route.go:882] Processing route {Ifindex: 16 Dst: 240.0.0.0/8 Src: 240.17.0.7 Gw: <nil> Flags: [] Table: 254}
  3994. 2020-10-08T11:45:08.9616713Z [cluster1] I1008 11:39:25.535563 1 route.go:885] Found nil gw or dst
  3995. 2020-10-08T11:45:08.9617300Z [cluster1] I1008 11:39:25.535609 1 route.go:924] Found equivalent route, not adding
  3996. 2020-10-08T11:45:08.9617866Z [cluster1] I1008 11:39:25.535616 1 route.go:924] Found equivalent route, not adding
  3997. 2020-10-08T11:45:08.9619251Z [cluster1] I1008 11:39:25.535643 1 route.go:715] Successfully processed local Endpoint "cluster1-submariner-cable-cluster1-172-17-0-9"
  3998. 2020-10-08T11:45:08.9621324Z [cluster1] I1008 11:39:25.537374 1 route.go:620] Updating iptable rules for remote cluster's Endpoint "cluster2-submariner-cable-cluster2-172-17-0-5" with subnets [100.92.0.0/16 10.242.0.0/16]
  3999. 2020-10-08T11:45:08.9625459Z [cluster1] I1008 11:39:55.524895 1 route.go:736] Enqueueing endpoint for route controller {"metadata":{"name":"cluster1-submariner-cable-cluster1-172-17-0-9","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster1-submariner-cable-cluster1-172-17-0-9","uid":"23aee4db-4473-4b17-8f51-aedd32e3a5ff","resourceVersion":"1212","generation":1,"creationTimestamp":"2020-10-08T11:34:56Z"},"spec":{"cluster_id":"cluster1","cable_name":"submariner-cable-cluster1-172-17-0-9","hostname":"cluster1-worker","subnets":["100.91.0.0/16","10.241.0.0/16"],"private_ip":"172.17.0.9","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
  4000. 2020-10-08T11:45:08.9632292Z [cluster1] I1008 11:39:55.524955 1 route.go:736] Enqueueing endpoint for route controller {"metadata":{"name":"cluster2-submariner-cable-cluster2-172-17-0-5","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster2-submariner-cable-cluster2-172-17-0-5","uid":"474bcd76-5748-4372-9832-4a15a4b1804d","resourceVersion":"1332","generation":1,"creationTimestamp":"2020-10-08T11:35:26Z"},"spec":{"cluster_id":"cluster2","cable_name":"submariner-cable-cluster2-172-17-0-5","hostname":"cluster2-worker","subnets":["100.92.0.0/16","10.242.0.0/16"],"private_ip":"172.17.0.5","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
  4001. 2020-10-08T11:45:08.9636195Z [cluster1] I1008 11:39:55.532413 1 route.go:634] Processing local Endpoint cluster1-submariner-cable-cluster1-172-17-0-9 with IP 172.17.0.9, Host cluster1-worker for this cluster
  4002. 2020-10-08T11:45:08.9637796Z [cluster1] I1008 11:39:55.532434 1 route.go:677] This route agent is running on a non-gateway/non-active node
  4003. 2020-10-08T11:45:08.9639289Z [cluster1] I1008 11:39:55.532442 1 route.go:689] Creating the vxlan interface vx-submariner with gateway node IP 172.17.0.9
  4004. 2020-10-08T11:45:08.9640184Z [cluster1] I1008 11:39:55.532594 1 vxlan.go:61] VxLAN interface already exists with same configuration.
  4005. 2020-10-08T11:45:08.9640801Z [cluster1] I1008 11:39:55.535765 1 route.go:866] Reconciling routes to gw: 240.17.0.9
  4006. 2020-10-08T11:45:08.9641409Z [cluster1] I1008 11:39:55.535931 1 route.go:882] Processing route {Ifindex: 16 Dst: 10.242.0.0/16 Src: <nil> Gw: 240.17.0.9 Flags: [] Table: 254}
  4007. 2020-10-08T11:45:08.9642149Z [cluster1] I1008 11:39:55.535960 1 route.go:888] Found route {Ifindex: 16 Dst: 10.242.0.0/16 Src: <nil> Gw: 240.17.0.9 Flags: [] Table: 254} with gw 240.17.0.9 already installed
  4008. 2020-10-08T11:45:08.9642877Z [cluster1] I1008 11:39:55.536001 1 route.go:882] Processing route {Ifindex: 16 Dst: 100.92.0.0/16 Src: <nil> Gw: 240.17.0.9 Flags: [] Table: 254}
  4009. 2020-10-08T11:45:08.9643606Z [cluster1] I1008 11:39:55.536010 1 route.go:888] Found route {Ifindex: 16 Dst: 100.92.0.0/16 Src: <nil> Gw: 240.17.0.9 Flags: [] Table: 254} with gw 240.17.0.9 already installed
  4010. 2020-10-08T11:45:08.9644331Z [cluster1] I1008 11:39:55.536023 1 route.go:882] Processing route {Ifindex: 16 Dst: 240.0.0.0/8 Src: 240.17.0.7 Gw: <nil> Flags: [] Table: 254}
  4011. 2020-10-08T11:45:08.9644881Z [cluster1] I1008 11:39:55.536030 1 route.go:885] Found nil gw or dst
  4012. 2020-10-08T11:45:08.9645382Z [cluster1] I1008 11:39:55.536105 1 route.go:924] Found equivalent route, not adding
  4013. 2020-10-08T11:45:08.9646681Z [cluster1] I1008 11:39:55.536118 1 route.go:924] Found equivalent route, not adding
  4014. 2020-10-08T11:45:08.9648328Z [cluster1] I1008 11:39:55.536172 1 route.go:715] Successfully processed local Endpoint "cluster1-submariner-cable-cluster1-172-17-0-9"
  4015. 2020-10-08T11:45:08.9650352Z [cluster1] I1008 11:39:55.537854 1 route.go:620] Updating iptable rules for remote cluster's Endpoint "cluster2-submariner-cable-cluster2-172-17-0-5" with subnets [100.92.0.0/16 10.242.0.0/16]
  4016. 2020-10-08T11:45:08.9655993Z [cluster1] I1008 11:40:25.525088 1 route.go:736] Enqueueing endpoint for route controller {"metadata":{"name":"cluster1-submariner-cable-cluster1-172-17-0-9","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster1-submariner-cable-cluster1-172-17-0-9","uid":"23aee4db-4473-4b17-8f51-aedd32e3a5ff","resourceVersion":"1212","generation":1,"creationTimestamp":"2020-10-08T11:34:56Z"},"spec":{"cluster_id":"cluster1","cable_name":"submariner-cable-cluster1-172-17-0-9","hostname":"cluster1-worker","subnets":["100.91.0.0/16","10.241.0.0/16"],"private_ip":"172.17.0.9","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
  4017. 2020-10-08T11:45:08.9664223Z [cluster1] I1008 11:40:25.525208 1 route.go:736] Enqueueing endpoint for route controller {"metadata":{"name":"cluster2-submariner-cable-cluster2-172-17-0-5","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster2-submariner-cable-cluster2-172-17-0-5","uid":"474bcd76-5748-4372-9832-4a15a4b1804d","resourceVersion":"1332","generation":1,"creationTimestamp":"2020-10-08T11:35:26Z"},"spec":{"cluster_id":"cluster2","cable_name":"submariner-cable-cluster2-172-17-0-5","hostname":"cluster2-worker","subnets":["100.92.0.0/16","10.242.0.0/16"],"private_ip":"172.17.0.5","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
  4018. 2020-10-08T11:45:08.9668817Z [cluster1] I1008 11:40:25.532696 1 route.go:634] Processing local Endpoint cluster1-submariner-cable-cluster1-172-17-0-9 with IP 172.17.0.9, Host cluster1-worker for this cluster
  4019. 2020-10-08T11:45:08.9670308Z [lighthouse]$ [cluster1] subctl show all
  4020. 2020-10-08T11:45:08.9671089Z [cluster1] I1008 11:40:25.532719 1 route.go:677] This route agent is running on a non-gateway/non-active node
  4021. 2020-10-08T11:45:08.9672073Z [cluster1] I1008 11:40:25.532727 1 route.go:689] Creating the vxlan interface vx-submariner with gateway node IP 172.17.0.9
  4022. 2020-10-08T11:45:08.9672736Z [cluster1] I1008 11:40:25.532960 1 vxlan.go:61] VxLAN interface already exists with same configuration.
  4023. 2020-10-08T11:45:08.9673288Z [cluster1] I1008 11:40:25.535797 1 route.go:866] Reconciling routes to gw: 240.17.0.9
  4024. 2020-10-08T11:45:08.9673821Z [cluster1] I1008 11:40:25.535907 1 route.go:882] Processing route {Ifindex: 16 Dst: 10.242.0.0/16 Src: <nil> Gw: 240.17.0.9 Flags: [] Table: 254}
  4025. 2020-10-08T11:45:08.9674467Z [cluster1] I1008 11:40:25.535932 1 route.go:888] Found route {Ifindex: 16 Dst: 10.242.0.0/16 Src: <nil> Gw: 240.17.0.9 Flags: [] Table: 254} with gw 240.17.0.9 already installed
  4026. 2020-10-08T11:45:08.9675107Z [cluster1] I1008 11:40:25.535940 1 route.go:882] Processing route {Ifindex: 16 Dst: 100.92.0.0/16 Src: <nil> Gw: 240.17.0.9 Flags: [] Table: 254}
  4027. 2020-10-08T11:45:08.9675739Z [cluster1] I1008 11:40:25.535954 1 route.go:888] Found route {Ifindex: 16 Dst: 100.92.0.0/16 Src: <nil> Gw: 240.17.0.9 Flags: [] Table: 254} with gw 240.17.0.9 already installed
  4028. 2020-10-08T11:45:08.9676589Z [cluster1] I1008 11:40:25.535960 1 route.go:882] Processing route {Ifindex: 16 Dst: 240.0.0.0/8 Src: 240.17.0.7 Gw: <nil> Flags: [] Table: 254}
  4029. 2020-10-08T11:45:08.9677139Z [cluster1] I1008 11:40:25.535973 1 route.go:885] Found nil gw or dst
  4030. 2020-10-08T11:45:08.9677635Z [cluster1] I1008 11:40:25.536014 1 route.go:924] Found equivalent route, not adding
  4031. 2020-10-08T11:45:08.9678194Z [cluster1] I1008 11:40:25.536025 1 route.go:924] Found equivalent route, not adding
  4032. 2020-10-08T11:45:08.9679707Z [cluster1] I1008 11:40:25.536050 1 route.go:715] Successfully processed local Endpoint "cluster1-submariner-cable-cluster1-172-17-0-9"
  4033. 2020-10-08T11:45:08.9681874Z [cluster1] I1008 11:40:25.537718 1 route.go:620] Updating iptable rules for remote cluster's Endpoint "cluster2-submariner-cable-cluster2-172-17-0-5" with subnets [100.92.0.0/16 10.242.0.0/16]
  4034. 2020-10-08T11:45:08.9686640Z [cluster1] I1008 11:40:55.525189 1 route.go:736] Enqueueing endpoint for route controller {"metadata":{"name":"cluster1-submariner-cable-cluster1-172-17-0-9","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster1-submariner-cable-cluster1-172-17-0-9","uid":"23aee4db-4473-4b17-8f51-aedd32e3a5ff","resourceVersion":"1212","generation":1,"creationTimestamp":"2020-10-08T11:34:56Z"},"spec":{"cluster_id":"cluster1","cable_name":"submariner-cable-cluster1-172-17-0-9","hostname":"cluster1-worker","subnets":["100.91.0.0/16","10.241.0.0/16"],"private_ip":"172.17.0.9","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
  4035. 2020-10-08T11:45:08.9694278Z [cluster1] I1008 11:40:55.525438 1 route.go:736] Enqueueing endpoint for route controller {"metadata":{"name":"cluster2-submariner-cable-cluster2-172-17-0-5","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster2-submariner-cable-cluster2-172-17-0-5","uid":"474bcd76-5748-4372-9832-4a15a4b1804d","resourceVersion":"1332","generation":1,"creationTimestamp":"2020-10-08T11:35:26Z"},"spec":{"cluster_id":"cluster2","cable_name":"submariner-cable-cluster2-172-17-0-5","hostname":"cluster2-worker","subnets":["100.92.0.0/16","10.242.0.0/16"],"private_ip":"172.17.0.5","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
  4036. 2020-10-08T11:45:08.9698726Z [cluster1] I1008 11:40:55.532749 1 route.go:634] Processing local Endpoint cluster1-submariner-cable-cluster1-172-17-0-9 with IP 172.17.0.9, Host cluster1-worker for this cluster
  4037. 2020-10-08T11:45:08.9700360Z [cluster1] I1008 11:40:55.532776 1 route.go:677] This route agent is running on a non-gateway/non-active node
  4038. 2020-10-08T11:45:08.9701914Z [cluster1] I1008 11:40:55.532786 1 route.go:689] Creating the vxlan interface vx-submariner with gateway node IP 172.17.0.9
  4039. 2020-10-08T11:45:08.9702735Z [cluster1] I1008 11:40:55.532944 1 vxlan.go:61] VxLAN interface already exists with same configuration.
  4040. 2020-10-08T11:45:08.9703416Z [cluster1] I1008 11:40:55.535615 1 route.go:866] Reconciling routes to gw: 240.17.0.9
  4041. 2020-10-08T11:45:08.9704181Z [cluster1] I1008 11:40:55.535808 1 route.go:882] Processing route {Ifindex: 16 Dst: 10.242.0.0/16 Src: <nil> Gw: 240.17.0.9 Flags: [] Table: 254}
  4042. 2020-10-08T11:45:08.9704964Z [cluster1] I1008 11:40:55.535832 1 route.go:888] Found route {Ifindex: 16 Dst: 10.242.0.0/16 Src: <nil> Gw: 240.17.0.9 Flags: [] Table: 254} with gw 240.17.0.9 already installed
  4043. 2020-10-08T11:45:08.9705744Z [cluster1] I1008 11:40:55.535839 1 route.go:882] Processing route {Ifindex: 16 Dst: 100.92.0.0/16 Src: <nil> Gw: 240.17.0.9 Flags: [] Table: 254}
  4044. 2020-10-08T11:45:08.9706513Z [cluster1] I1008 11:40:55.535846 1 route.go:888] Found route {Ifindex: 16 Dst: 100.92.0.0/16 Src: <nil> Gw: 240.17.0.9 Flags: [] Table: 254} with gw 240.17.0.9 already installed
  4045. 2020-10-08T11:45:08.9707291Z [cluster1] I1008 11:40:55.535853 1 route.go:882] Processing route {Ifindex: 16 Dst: 240.0.0.0/8 Src: 240.17.0.7 Gw: <nil> Flags: [] Table: 254}
  4046. 2020-10-08T11:45:08.9707886Z [cluster1] I1008 11:40:55.536002 1 route.go:885] Found nil gw or dst
  4047. 2020-10-08T11:45:08.9708420Z [cluster1] I1008 11:40:55.536096 1 route.go:924] Found equivalent route, not adding
  4048. 2020-10-08T11:45:08.9709030Z [cluster1] I1008 11:40:55.536108 1 route.go:924] Found equivalent route, not adding
  4049. 2020-10-08T11:45:08.9710540Z [cluster1] I1008 11:40:55.536168 1 route.go:715] Successfully processed local Endpoint "cluster1-submariner-cable-cluster1-172-17-0-9"
  4050. 2020-10-08T11:45:08.9712821Z [cluster1] I1008 11:40:55.537615 1 route.go:620] Updating iptable rules for remote cluster's Endpoint "cluster2-submariner-cable-cluster2-172-17-0-5" with subnets [100.92.0.0/16 10.242.0.0/16]
  4051. 2020-10-08T11:45:08.9717968Z [cluster1] I1008 11:41:25.525366 1 route.go:736] Enqueueing endpoint for route controller {"metadata":{"name":"cluster1-submariner-cable-cluster1-172-17-0-9","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster1-submariner-cable-cluster1-172-17-0-9","uid":"23aee4db-4473-4b17-8f51-aedd32e3a5ff","resourceVersion":"1212","generation":1,"creationTimestamp":"2020-10-08T11:34:56Z"},"spec":{"cluster_id":"cluster1","cable_name":"submariner-cable-cluster1-172-17-0-9","hostname":"cluster1-worker","subnets":["100.91.0.0/16","10.241.0.0/16"],"private_ip":"172.17.0.9","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
  4052. 2020-10-08T11:45:08.9725808Z [cluster1] I1008 11:41:25.525460 1 route.go:736] Enqueueing endpoint for route controller {"metadata":{"name":"cluster2-submariner-cable-cluster2-172-17-0-5","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster2-submariner-cable-cluster2-172-17-0-5","uid":"474bcd76-5748-4372-9832-4a15a4b1804d","resourceVersion":"1332","generation":1,"creationTimestamp":"2020-10-08T11:35:26Z"},"spec":{"cluster_id":"cluster2","cable_name":"submariner-cable-cluster2-172-17-0-5","hostname":"cluster2-worker","subnets":["100.92.0.0/16","10.242.0.0/16"],"private_ip":"172.17.0.5","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
  4053. 2020-10-08T11:45:08.9730585Z [cluster1] I1008 11:41:25.533133 1 route.go:634] Processing local Endpoint cluster1-submariner-cable-cluster1-172-17-0-9 with IP 172.17.0.9, Host cluster1-worker for this cluster
  4054. 2020-10-08T11:45:08.9732208Z [cluster1] I1008 11:41:25.533153 1 route.go:677] This route agent is running on a non-gateway/non-active node
  4055. 2020-10-08T11:45:08.9733373Z [cluster1] I1008 11:41:25.533162 1 route.go:689] Creating the vxlan interface vx-submariner with gateway node IP 172.17.0.9
  4056. 2020-10-08T11:45:08.9734192Z [cluster1] I1008 11:41:25.533415 1 vxlan.go:61] VxLAN interface already exists with same configuration.
  4057. 2020-10-08T11:45:08.9734870Z [cluster1] I1008 11:41:25.536424 1 route.go:866] Reconciling routes to gw: 240.17.0.9
  4058. 2020-10-08T11:45:08.9735531Z [cluster1] I1008 11:41:25.536677 1 route.go:882] Processing route {Ifindex: 16 Dst: 10.242.0.0/16 Src: <nil> Gw: 240.17.0.9 Flags: [] Table: 254}
  4059. 2020-10-08T11:45:08.9736318Z [cluster1] I1008 11:41:25.536704 1 route.go:888] Found route {Ifindex: 16 Dst: 10.242.0.0/16 Src: <nil> Gw: 240.17.0.9 Flags: [] Table: 254} with gw 240.17.0.9 already installed
  4060. 2020-10-08T11:45:08.9737103Z [cluster1] I1008 11:41:25.536805 1 route.go:882] Processing route {Ifindex: 16 Dst: 100.92.0.0/16 Src: <nil> Gw: 240.17.0.9 Flags: [] Table: 254}
  4061. 2020-10-08T11:45:08.9737887Z [cluster1] I1008 11:41:25.536824 1 route.go:888] Found route {Ifindex: 16 Dst: 100.92.0.0/16 Src: <nil> Gw: 240.17.0.9 Flags: [] Table: 254} with gw 240.17.0.9 already installed
  4062. 2020-10-08T11:45:08.9738663Z [cluster1] I1008 11:41:25.536832 1 route.go:882] Processing route {Ifindex: 16 Dst: 240.0.0.0/8 Src: 240.17.0.7 Gw: <nil> Flags: [] Table: 254}
  4063. 2020-10-08T11:45:08.9739254Z [cluster1] I1008 11:41:25.536839 1 route.go:885] Found nil gw or dst
  4064. 2020-10-08T11:45:08.9739790Z [cluster1] I1008 11:41:25.536951 1 route.go:924] Found equivalent route, not adding
  4065. 2020-10-08T11:45:08.9740397Z [cluster1] I1008 11:41:25.536966 1 route.go:924] Found equivalent route, not adding
  4066. 2020-10-08T11:45:08.9741850Z [cluster1] I1008 11:41:25.537044 1 route.go:715] Successfully processed local Endpoint "cluster1-submariner-cable-cluster1-172-17-0-9"
  4067. 2020-10-08T11:45:08.9745383Z [cluster1] I1008 11:41:25.538732 1 route.go:620] Updating iptable rules for remote cluster's Endpoint "cluster2-submariner-cable-cluster2-172-17-0-5" with subnets [100.92.0.0/16 10.242.0.0/16]
  4068. 2020-10-08T11:45:08.9753649Z [cluster1] I1008 11:41:55.525563 1 route.go:736] Enqueueing endpoint for route controller {"metadata":{"name":"cluster2-submariner-cable-cluster2-172-17-0-5","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster2-submariner-cable-cluster2-172-17-0-5","uid":"474bcd76-5748-4372-9832-4a15a4b1804d","resourceVersion":"1332","generation":1,"creationTimestamp":"2020-10-08T11:35:26Z"},"spec":{"cluster_id":"cluster2","cable_name":"submariner-cable-cluster2-172-17-0-5","hostname":"cluster2-worker","subnets":["100.92.0.0/16","10.242.0.0/16"],"private_ip":"172.17.0.5","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
  4069. 2020-10-08T11:45:08.9760538Z [cluster1] I1008 11:41:55.525620 1 route.go:736] Enqueueing endpoint for route controller {"metadata":{"name":"cluster1-submariner-cable-cluster1-172-17-0-9","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster1-submariner-cable-cluster1-172-17-0-9","uid":"23aee4db-4473-4b17-8f51-aedd32e3a5ff","resourceVersion":"1212","generation":1,"creationTimestamp":"2020-10-08T11:34:56Z"},"spec":{"cluster_id":"cluster1","cable_name":"submariner-cable-cluster1-172-17-0-9","hostname":"cluster1-worker","subnets":["100.91.0.0/16","10.241.0.0/16"],"private_ip":"172.17.0.9","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
  4070. 2020-10-08T11:45:08.9765974Z [cluster1] I1008 11:41:55.533218 1 route.go:620] Updating iptable rules for remote cluster's Endpoint "cluster2-submariner-cable-cluster2-172-17-0-5" with subnets [100.92.0.0/16 10.242.0.0/16]
  4071. 2020-10-08T11:45:08.9768737Z [cluster1] I1008 11:41:55.534707 1 route.go:634] Processing local Endpoint cluster1-submariner-cable-cluster1-172-17-0-9 with IP 172.17.0.9, Host cluster1-worker for this cluster
  4072. 2020-10-08T11:45:08.9770262Z [cluster1] I1008 11:41:55.534726 1 route.go:677] This route agent is running on a non-gateway/non-active node
  4073. 2020-10-08T11:45:08.9772034Z [cluster1] I1008 11:41:55.534734 1 route.go:689] Creating the vxlan interface vx-submariner with gateway node IP 172.17.0.9
  4074. 2020-10-08T11:45:08.9773154Z [cluster1] I1008 11:41:55.534884 1 vxlan.go:61] VxLAN interface already exists with same configuration.
  4075. 2020-10-08T11:45:08.9773837Z [cluster1] I1008 11:41:55.537629 1 route.go:866] Reconciling routes to gw: 240.17.0.9
  4076. 2020-10-08T11:45:08.9774499Z [cluster1] I1008 11:41:55.537741 1 route.go:882] Processing route {Ifindex: 16 Dst: 10.242.0.0/16 Src: <nil> Gw: 240.17.0.9 Flags: [] Table: 254}
  4077. 2020-10-08T11:45:08.9775290Z [cluster1] I1008 11:41:55.537766 1 route.go:888] Found route {Ifindex: 16 Dst: 10.242.0.0/16 Src: <nil> Gw: 240.17.0.9 Flags: [] Table: 254} with gw 240.17.0.9 already installed
  4078. 2020-10-08T11:45:08.9776076Z [cluster1] I1008 11:41:55.537775 1 route.go:882] Processing route {Ifindex: 16 Dst: 100.92.0.0/16 Src: <nil> Gw: 240.17.0.9 Flags: [] Table: 254}
  4079. 2020-10-08T11:45:08.9776869Z [cluster1] I1008 11:41:55.537782 1 route.go:888] Found route {Ifindex: 16 Dst: 100.92.0.0/16 Src: <nil> Gw: 240.17.0.9 Flags: [] Table: 254} with gw 240.17.0.9 already installed
  4080. 2020-10-08T11:45:08.9777648Z [cluster1] I1008 11:41:55.537789 1 route.go:882] Processing route {Ifindex: 16 Dst: 240.0.0.0/8 Src: 240.17.0.7 Gw: <nil> Flags: [] Table: 254}
  4081. 2020-10-08T11:45:08.9778244Z [cluster1] I1008 11:41:55.537800 1 route.go:885] Found nil gw or dst
  4082. 2020-10-08T11:45:08.9778891Z [cluster1] I1008 11:41:55.537840 1 route.go:924] Found equivalent route, not adding
  4083. 2020-10-08T11:45:08.9779456Z [cluster1] I1008 11:41:55.537848 1 route.go:924] Found equivalent route, not adding
  4084. 2020-10-08T11:45:08.9780854Z [cluster1] I1008 11:41:55.537871 1 route.go:715] Successfully processed local Endpoint "cluster1-submariner-cable-cluster1-172-17-0-9"
  4085. 2020-10-08T11:45:08.9785414Z [cluster1] I1008 11:42:25.525686 1 route.go:736] Enqueueing endpoint for route controller {"metadata":{"name":"cluster1-submariner-cable-cluster1-172-17-0-9","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster1-submariner-cable-cluster1-172-17-0-9","uid":"23aee4db-4473-4b17-8f51-aedd32e3a5ff","resourceVersion":"1212","generation":1,"creationTimestamp":"2020-10-08T11:34:56Z"},"spec":{"cluster_id":"cluster1","cable_name":"submariner-cable-cluster1-172-17-0-9","hostname":"cluster1-worker","subnets":["100.91.0.0/16","10.241.0.0/16"],"private_ip":"172.17.0.9","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
  4086. 2020-10-08T11:45:08.9791990Z [cluster1] I1008 11:42:25.525775 1 route.go:736] Enqueueing endpoint for route controller {"metadata":{"name":"cluster2-submariner-cable-cluster2-172-17-0-5","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster2-submariner-cable-cluster2-172-17-0-5","uid":"474bcd76-5748-4372-9832-4a15a4b1804d","resourceVersion":"1332","generation":1,"creationTimestamp":"2020-10-08T11:35:26Z"},"spec":{"cluster_id":"cluster2","cable_name":"submariner-cable-cluster2-172-17-0-5","hostname":"cluster2-worker","subnets":["100.92.0.0/16","10.242.0.0/16"],"private_ip":"172.17.0.5","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
  4087. 2020-10-08T11:45:08.9796857Z [cluster1] I1008 11:42:25.532926 1 route.go:634] Processing local Endpoint cluster1-submariner-cable-cluster1-172-17-0-9 with IP 172.17.0.9, Host cluster1-worker for this cluster
  4088. 2020-10-08T11:45:08.9798480Z [cluster1] I1008 11:42:25.532951 1 route.go:677] This route agent is running on a non-gateway/non-active node
  4089. 2020-10-08T11:45:08.9799532Z [cluster1] I1008 11:42:25.532961 1 route.go:689] Creating the vxlan interface vx-submariner with gateway node IP 172.17.0.9
  4090. 2020-10-08T11:45:08.9800742Z [cluster1] I1008 11:42:25.533146 1 vxlan.go:61] VxLAN interface already exists with same configuration.
  4091. 2020-10-08T11:45:08.9802064Z [cluster1] I1008 11:42:25.536097 1 route.go:866] Reconciling routes to gw: 240.17.0.9
  4092. 2020-10-08T11:45:08.9802692Z [cluster1] I1008 11:42:25.536215 1 route.go:882] Processing route {Ifindex: 16 Dst: 10.242.0.0/16 Src: <nil> Gw: 240.17.0.9 Flags: [] Table: 254}
  4093. 2020-10-08T11:45:08.9803426Z [cluster1] I1008 11:42:25.536237 1 route.go:888] Found route {Ifindex: 16 Dst: 10.242.0.0/16 Src: <nil> Gw: 240.17.0.9 Flags: [] Table: 254} with gw 240.17.0.9 already installed
  4094. 2020-10-08T11:45:08.9804135Z [cluster1] I1008 11:42:25.536246 1 route.go:882] Processing route {Ifindex: 16 Dst: 100.92.0.0/16 Src: <nil> Gw: 240.17.0.9 Flags: [] Table: 254}
  4095. 2020-10-08T11:45:08.9804856Z [cluster1] I1008 11:42:25.536253 1 route.go:888] Found route {Ifindex: 16 Dst: 100.92.0.0/16 Src: <nil> Gw: 240.17.0.9 Flags: [] Table: 254} with gw 240.17.0.9 already installed
  4096. 2020-10-08T11:45:08.9805572Z [cluster1] I1008 11:42:25.536260 1 route.go:882] Processing route {Ifindex: 16 Dst: 240.0.0.0/8 Src: 240.17.0.7 Gw: <nil> Flags: [] Table: 254}
  4097. 2020-10-08T11:45:08.9806130Z [cluster1] I1008 11:42:25.536267 1 route.go:885] Found nil gw or dst
  4098. 2020-10-08T11:45:08.9806624Z [cluster1] I1008 11:42:25.536455 1 route.go:924] Found equivalent route, not adding
  4099. 2020-10-08T11:45:08.9807254Z [cluster1] I1008 11:42:25.536550 1 route.go:924] Found equivalent route, not adding
  4100. 2020-10-08T11:45:08.9808477Z [cluster1] I1008 11:42:25.536600 1 route.go:715] Successfully processed local Endpoint "cluster1-submariner-cable-cluster1-172-17-0-9"
  4101. 2020-10-08T11:45:08.9810241Z [cluster1] I1008 11:42:25.538155 1 route.go:620] Updating iptable rules for remote cluster's Endpoint "cluster2-submariner-cable-cluster2-172-17-0-5" with subnets [100.92.0.0/16 10.242.0.0/16]
  4102. 2020-10-08T11:45:08.9814726Z [cluster1] I1008 11:42:55.525952 1 route.go:736] Enqueueing endpoint for route controller {"metadata":{"name":"cluster1-submariner-cable-cluster1-172-17-0-9","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster1-submariner-cable-cluster1-172-17-0-9","uid":"23aee4db-4473-4b17-8f51-aedd32e3a5ff","resourceVersion":"1212","generation":1,"creationTimestamp":"2020-10-08T11:34:56Z"},"spec":{"cluster_id":"cluster1","cable_name":"submariner-cable-cluster1-172-17-0-9","hostname":"cluster1-worker","subnets":["100.91.0.0/16","10.241.0.0/16"],"private_ip":"172.17.0.9","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
  4103. 2020-10-08T11:45:08.9821962Z [cluster1] I1008 11:42:55.526348 1 route.go:736] Enqueueing endpoint for route controller {"metadata":{"name":"cluster2-submariner-cable-cluster2-172-17-0-5","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster2-submariner-cable-cluster2-172-17-0-5","uid":"474bcd76-5748-4372-9832-4a15a4b1804d","resourceVersion":"1332","generation":1,"creationTimestamp":"2020-10-08T11:35:26Z"},"spec":{"cluster_id":"cluster2","cable_name":"submariner-cable-cluster2-172-17-0-5","hostname":"cluster2-worker","subnets":["100.92.0.0/16","10.242.0.0/16"],"private_ip":"172.17.0.5","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
  4104. 2020-10-08T11:45:08.9826773Z [cluster1] I1008 11:42:55.533906 1 route.go:634] Processing local Endpoint cluster1-submariner-cable-cluster1-172-17-0-9 with IP 172.17.0.9, Host cluster1-worker for this cluster
  4105. 2020-10-08T11:45:08.9828480Z [cluster1] I1008 11:42:55.533925 1 route.go:677] This route agent is running on a non-gateway/non-active node
  4106. 2020-10-08T11:45:08.9829483Z [cluster1] I1008 11:42:55.533933 1 route.go:689] Creating the vxlan interface vx-submariner with gateway node IP 172.17.0.9
  4107. 2020-10-08T11:45:08.9830240Z [cluster1] I1008 11:42:55.534217 1 vxlan.go:61] VxLAN interface already exists with same configuration.
  4108. 2020-10-08T11:45:08.9830866Z [cluster1] I1008 11:42:55.537130 1 route.go:866] Reconciling routes to gw: 240.17.0.9
  4109. 2020-10-08T11:45:08.9831474Z [cluster1] I1008 11:42:55.537495 1 route.go:882] Processing route {Ifindex: 16 Dst: 10.242.0.0/16 Src: <nil> Gw: 240.17.0.9 Flags: [] Table: 254}
  4110. 2020-10-08T11:45:08.9832207Z [cluster1] I1008 11:42:55.537541 1 route.go:888] Found route {Ifindex: 16 Dst: 10.242.0.0/16 Src: <nil> Gw: 240.17.0.9 Flags: [] Table: 254} with gw 240.17.0.9 already installed
  4111. 2020-10-08T11:45:08.9832925Z [cluster1] I1008 11:42:55.537591 1 route.go:882] Processing route {Ifindex: 16 Dst: 100.92.0.0/16 Src: <nil> Gw: 240.17.0.9 Flags: [] Table: 254}
  4112. 2020-10-08T11:45:08.9833644Z [cluster1] I1008 11:42:55.537615 1 route.go:888] Found route {Ifindex: 16 Dst: 100.92.0.0/16 Src: <nil> Gw: 240.17.0.9 Flags: [] Table: 254} with gw 240.17.0.9 already installed
  4113. 2020-10-08T11:45:08.9834363Z [cluster1] I1008 11:42:55.537664 1 route.go:882] Processing route {Ifindex: 16 Dst: 240.0.0.0/8 Src: 240.17.0.7 Gw: <nil> Flags: [] Table: 254}
  4114. 2020-10-08T11:45:08.9834915Z [cluster1] I1008 11:42:55.537686 1 route.go:885] Found nil gw or dst
  4115. 2020-10-08T11:45:08.9835423Z [cluster1] I1008 11:42:55.537811 1 route.go:924] Found equivalent route, not adding
  4116. 2020-10-08T11:45:08.9835982Z [cluster1] I1008 11:42:55.537835 1 route.go:924] Found equivalent route, not adding
  4117. 2020-10-08T11:45:08.9837495Z [cluster1] I1008 11:42:55.537883 1 route.go:715] Successfully processed local Endpoint "cluster1-submariner-cable-cluster1-172-17-0-9"
  4118. 2020-10-08T11:45:08.9839773Z [cluster1] I1008 11:42:55.539602 1 route.go:620] Updating iptable rules for remote cluster's Endpoint "cluster2-submariner-cable-cluster2-172-17-0-5" with subnets [100.92.0.0/16 10.242.0.0/16]
  4119. 2020-10-08T11:45:08.9846399Z [cluster1] I1008 11:43:25.526040 1 route.go:736] Enqueueing endpoint for route controller {"metadata":{"name":"cluster1-submariner-cable-cluster1-172-17-0-9","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster1-submariner-cable-cluster1-172-17-0-9","uid":"23aee4db-4473-4b17-8f51-aedd32e3a5ff","resourceVersion":"1212","generation":1,"creationTimestamp":"2020-10-08T11:34:56Z"},"spec":{"cluster_id":"cluster1","cable_name":"submariner-cable-cluster1-172-17-0-9","hostname":"cluster1-worker","subnets":["100.91.0.0/16","10.241.0.0/16"],"private_ip":"172.17.0.9","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
  4120. 2020-10-08T11:45:08.9854031Z [cluster1] I1008 11:43:25.526095 1 route.go:736] Enqueueing endpoint for route controller {"metadata":{"name":"cluster2-submariner-cable-cluster2-172-17-0-5","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster2-submariner-cable-cluster2-172-17-0-5","uid":"474bcd76-5748-4372-9832-4a15a4b1804d","resourceVersion":"1332","generation":1,"creationTimestamp":"2020-10-08T11:35:26Z"},"spec":{"cluster_id":"cluster2","cable_name":"submariner-cable-cluster2-172-17-0-5","hostname":"cluster2-worker","subnets":["100.92.0.0/16","10.242.0.0/16"],"private_ip":"172.17.0.5","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
  4121. 2020-10-08T11:45:08.9858667Z [cluster1] I1008 11:43:25.533334 1 route.go:634] Processing local Endpoint cluster1-submariner-cable-cluster1-172-17-0-9 with IP 172.17.0.9, Host cluster1-worker for this cluster
  4122. 2020-10-08T11:45:08.9860288Z [cluster1] I1008 11:43:25.533355 1 route.go:677] This route agent is running on a non-gateway/non-active node
  4123. 2020-10-08T11:45:08.9861343Z [cluster1] I1008 11:43:25.533364 1 route.go:689] Creating the vxlan interface vx-submariner with gateway node IP 172.17.0.9
  4124. 2020-10-08T11:45:08.9862139Z [cluster1] I1008 11:43:25.533518 1 vxlan.go:61] VxLAN interface already exists with same configuration.
  4125. 2020-10-08T11:45:08.9862816Z [cluster1] I1008 11:43:25.536459 1 route.go:866] Reconciling routes to gw: 240.17.0.9
  4126. 2020-10-08T11:45:08.9863473Z [cluster1] I1008 11:43:25.536577 1 route.go:882] Processing route {Ifindex: 16 Dst: 10.242.0.0/16 Src: <nil> Gw: 240.17.0.9 Flags: [] Table: 254}
  4127. 2020-10-08T11:45:08.9864265Z [cluster1] I1008 11:43:25.536598 1 route.go:888] Found route {Ifindex: 16 Dst: 10.242.0.0/16 Src: <nil> Gw: 240.17.0.9 Flags: [] Table: 254} with gw 240.17.0.9 already installed
  4128. 2020-10-08T11:45:08.9865050Z [cluster1] I1008 11:43:25.536606 1 route.go:882] Processing route {Ifindex: 16 Dst: 100.92.0.0/16 Src: <nil> Gw: 240.17.0.9 Flags: [] Table: 254}
  4129. 2020-10-08T11:45:08.9865932Z [cluster1] I1008 11:43:25.536614 1 route.go:888] Found route {Ifindex: 16 Dst: 100.92.0.0/16 Src: <nil> Gw: 240.17.0.9 Flags: [] Table: 254} with gw 240.17.0.9 already installed
  4130. 2020-10-08T11:45:08.9866718Z [cluster1] I1008 11:43:25.536621 1 route.go:882] Processing route {Ifindex: 16 Dst: 240.0.0.0/8 Src: 240.17.0.7 Gw: <nil> Flags: [] Table: 254}
  4131. 2020-10-08T11:45:08.9867313Z [cluster1] I1008 11:43:25.536628 1 route.go:885] Found nil gw or dst
  4132. 2020-10-08T11:45:08.9867850Z [cluster1] I1008 11:43:25.536677 1 route.go:924] Found equivalent route, not adding
  4133. 2020-10-08T11:45:08.9868459Z [cluster1] I1008 11:43:25.536684 1 route.go:924] Found equivalent route, not adding
  4134. 2020-10-08T11:45:08.9869945Z [cluster1] I1008 11:43:25.536711 1 route.go:715] Successfully processed local Endpoint "cluster1-submariner-cable-cluster1-172-17-0-9"
  4135. 2020-10-08T11:45:08.9872131Z [cluster1] I1008 11:43:25.538376 1 route.go:620] Updating iptable rules for remote cluster's Endpoint "cluster2-submariner-cable-cluster2-172-17-0-5" with subnets [100.92.0.0/16 10.242.0.0/16]
  4136. 2020-10-08T11:45:08.9877096Z [cluster1] I1008 11:43:55.526176 1 route.go:736] Enqueueing endpoint for route controller {"metadata":{"name":"cluster1-submariner-cable-cluster1-172-17-0-9","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster1-submariner-cable-cluster1-172-17-0-9","uid":"23aee4db-4473-4b17-8f51-aedd32e3a5ff","resourceVersion":"1212","generation":1,"creationTimestamp":"2020-10-08T11:34:56Z"},"spec":{"cluster_id":"cluster1","cable_name":"submariner-cable-cluster1-172-17-0-9","hostname":"cluster1-worker","subnets":["100.91.0.0/16","10.241.0.0/16"],"private_ip":"172.17.0.9","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
  4137. 2020-10-08T11:45:08.9883026Z [cluster1] I1008 11:43:55.526245 1 route.go:736] Enqueueing endpoint for route controller {"metadata":{"name":"cluster2-submariner-cable-cluster2-172-17-0-5","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster2-submariner-cable-cluster2-172-17-0-5","uid":"474bcd76-5748-4372-9832-4a15a4b1804d","resourceVersion":"1332","generation":1,"creationTimestamp":"2020-10-08T11:35:26Z"},"spec":{"cluster_id":"cluster2","cable_name":"submariner-cable-cluster2-172-17-0-5","hostname":"cluster2-worker","subnets":["100.92.0.0/16","10.242.0.0/16"],"private_ip":"172.17.0.5","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
  4138. 2020-10-08T11:45:08.9887701Z [cluster1] I1008 11:43:55.533745 1 route.go:634] Processing local Endpoint cluster1-submariner-cable-cluster1-172-17-0-9 with IP 172.17.0.9, Host cluster1-worker for this cluster
  4139. 2020-10-08T11:45:08.9889389Z [cluster1] I1008 11:43:55.533767 1 route.go:677] This route agent is running on a non-gateway/non-active node
  4140. 2020-10-08T11:45:08.9890375Z [cluster1] I1008 11:43:55.533776 1 route.go:689] Creating the vxlan interface vx-submariner with gateway node IP 172.17.0.9
  4141. 2020-10-08T11:45:08.9891127Z [cluster1] I1008 11:43:55.533957 1 vxlan.go:61] VxLAN interface already exists with same configuration.
  4142. 2020-10-08T11:45:08.9891754Z [cluster1] I1008 11:43:55.537061 1 route.go:866] Reconciling routes to gw: 240.17.0.9
  4143. 2020-10-08T11:45:08.9892456Z [cluster1] I1008 11:43:55.537194 1 route.go:882] Processing route {Ifindex: 16 Dst: 10.242.0.0/16 Src: <nil> Gw: 240.17.0.9 Flags: [] Table: 254}
  4144. 2020-10-08T11:45:08.9893188Z [cluster1] I1008 11:43:55.537225 1 route.go:888] Found route {Ifindex: 16 Dst: 10.242.0.0/16 Src: <nil> Gw: 240.17.0.9 Flags: [] Table: 254} with gw 240.17.0.9 already installed
  4145. 2020-10-08T11:45:08.9893927Z [cluster1] I1008 11:43:55.537275 1 route.go:882] Processing route {Ifindex: 16 Dst: 100.92.0.0/16 Src: <nil> Gw: 240.17.0.9 Flags: [] Table: 254}
  4146. 2020-10-08T11:45:08.9894643Z [cluster1] I1008 11:43:55.537285 1 route.go:888] Found route {Ifindex: 16 Dst: 100.92.0.0/16 Src: <nil> Gw: 240.17.0.9 Flags: [] Table: 254} with gw 240.17.0.9 already installed
  4147. 2020-10-08T11:45:08.9895363Z [cluster1] I1008 11:43:55.537292 1 route.go:882] Processing route {Ifindex: 16 Dst: 240.0.0.0/8 Src: 240.17.0.7 Gw: <nil> Flags: [] Table: 254}
  4148. 2020-10-08T11:45:08.9895897Z [cluster1] I1008 11:43:55.537307 1 route.go:885] Found nil gw or dst
  4149. 2020-10-08T11:45:08.9896393Z [cluster1] I1008 11:43:55.537358 1 route.go:924] Found equivalent route, not adding
  4150. 2020-10-08T11:45:08.9896957Z [cluster1] I1008 11:43:55.537393 1 route.go:924] Found equivalent route, not adding
  4151. 2020-10-08T11:45:08.9898325Z [cluster1] I1008 11:43:55.537424 1 route.go:715] Successfully processed local Endpoint "cluster1-submariner-cable-cluster1-172-17-0-9"
  4152. 2020-10-08T11:45:08.9900879Z [cluster1] I1008 11:43:55.538917 1 route.go:620] Updating iptable rules for remote cluster's Endpoint "cluster2-submariner-cable-cluster2-172-17-0-5" with subnets [100.92.0.0/16 10.242.0.0/16]
  4153. 2020-10-08T11:45:08.9905305Z [cluster1] I1008 11:44:25.526281 1 route.go:736] Enqueueing endpoint for route controller {"metadata":{"name":"cluster1-submariner-cable-cluster1-172-17-0-9","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster1-submariner-cable-cluster1-172-17-0-9","uid":"23aee4db-4473-4b17-8f51-aedd32e3a5ff","resourceVersion":"1212","generation":1,"creationTimestamp":"2020-10-08T11:34:56Z"},"spec":{"cluster_id":"cluster1","cable_name":"submariner-cable-cluster1-172-17-0-9","hostname":"cluster1-worker","subnets":["100.91.0.0/16","10.241.0.0/16"],"private_ip":"172.17.0.9","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
  4154. 2020-10-08T11:45:08.9911974Z [cluster1] I1008 11:44:25.526366 1 route.go:736] Enqueueing endpoint for route controller {"metadata":{"name":"cluster2-submariner-cable-cluster2-172-17-0-5","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster2-submariner-cable-cluster2-172-17-0-5","uid":"474bcd76-5748-4372-9832-4a15a4b1804d","resourceVersion":"1332","generation":1,"creationTimestamp":"2020-10-08T11:35:26Z"},"spec":{"cluster_id":"cluster2","cable_name":"submariner-cable-cluster2-172-17-0-5","hostname":"cluster2-worker","subnets":["100.92.0.0/16","10.242.0.0/16"],"private_ip":"172.17.0.5","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
  4155. 2020-10-08T11:45:08.9916546Z [cluster1] I1008 11:44:25.533790 1 route.go:634] Processing local Endpoint cluster1-submariner-cable-cluster1-172-17-0-9 with IP 172.17.0.9, Host cluster1-worker for this cluster
  4156. 2020-10-08T11:45:08.9918052Z [cluster1] I1008 11:44:25.533807 1 route.go:677] This route agent is running on a non-gateway/non-active node
  4157. 2020-10-08T11:45:08.9919148Z [cluster1] I1008 11:44:25.533816 1 route.go:689] Creating the vxlan interface vx-submariner with gateway node IP 172.17.0.9
  4158. 2020-10-08T11:45:08.9919905Z [cluster1] I1008 11:44:25.533994 1 vxlan.go:61] VxLAN interface already exists with same configuration.
  4159. 2020-10-08T11:45:08.9920533Z [cluster1] I1008 11:44:25.536930 1 route.go:866] Reconciling routes to gw: 240.17.0.9
  4160. 2020-10-08T11:45:08.9921144Z [cluster1] I1008 11:44:25.537198 1 route.go:882] Processing route {Ifindex: 16 Dst: 10.242.0.0/16 Src: <nil> Gw: 240.17.0.9 Flags: [] Table: 254}
  4161. 2020-10-08T11:45:08.9921872Z [cluster1] I1008 11:44:25.537273 1 route.go:888] Found route {Ifindex: 16 Dst: 10.242.0.0/16 Src: <nil> Gw: 240.17.0.9 Flags: [] Table: 254} with gw 240.17.0.9 already installed
  4162. 2020-10-08T11:45:08.9922595Z [cluster1] I1008 11:44:25.537283 1 route.go:882] Processing route {Ifindex: 16 Dst: 100.92.0.0/16 Src: <nil> Gw: 240.17.0.9 Flags: [] Table: 254}
  4163. 2020-10-08T11:45:08.9923409Z [cluster1] I1008 11:44:25.537292 1 route.go:888] Found route {Ifindex: 16 Dst: 100.92.0.0/16 Src: <nil> Gw: 240.17.0.9 Flags: [] Table: 254} with gw 240.17.0.9 already installed
  4164. 2020-10-08T11:45:08.9924039Z [cluster1] I1008 11:44:25.537299 1 route.go:882] Processing route {Ifindex: 16 Dst: 240.0.0.0/8 Src: 240.17.0.7 Gw: <nil> Flags: [] Table: 254}
  4165. 2020-10-08T11:45:08.9924522Z [cluster1] I1008 11:44:25.537306 1 route.go:885] Found nil gw or dst
  4166. 2020-10-08T11:45:08.9924959Z [cluster1] I1008 11:44:25.537440 1 route.go:924] Found equivalent route, not adding
  4167. 2020-10-08T11:45:08.9925446Z [cluster1] I1008 11:44:25.537455 1 route.go:924] Found equivalent route, not adding
  4168. 2020-10-08T11:45:08.9926619Z [cluster1] I1008 11:44:25.537528 1 route.go:715] Successfully processed local Endpoint "cluster1-submariner-cable-cluster1-172-17-0-9"
  4169. 2020-10-08T11:45:08.9929067Z [cluster1] I1008 11:44:25.539089 1 route.go:620] Updating iptable rules for remote cluster's Endpoint "cluster2-submariner-cable-cluster2-172-17-0-5" with subnets [100.92.0.0/16 10.242.0.0/16]
  4170. 2020-10-08T11:45:08.9936963Z [cluster1] I1008 11:44:55.526405 1 route.go:736] Enqueueing endpoint for route controller {"metadata":{"name":"cluster1-submariner-cable-cluster1-172-17-0-9","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster1-submariner-cable-cluster1-172-17-0-9","uid":"23aee4db-4473-4b17-8f51-aedd32e3a5ff","resourceVersion":"1212","generation":1,"creationTimestamp":"2020-10-08T11:34:56Z"},"spec":{"cluster_id":"cluster1","cable_name":"submariner-cable-cluster1-172-17-0-9","hostname":"cluster1-worker","subnets":["100.91.0.0/16","10.241.0.0/16"],"private_ip":"172.17.0.9","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
  4171. 2020-10-08T11:45:08.9944985Z [cluster1] I1008 11:44:55.526479 1 route.go:736] Enqueueing endpoint for route controller {"metadata":{"name":"cluster2-submariner-cable-cluster2-172-17-0-5","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster2-submariner-cable-cluster2-172-17-0-5","uid":"474bcd76-5748-4372-9832-4a15a4b1804d","resourceVersion":"1332","generation":1,"creationTimestamp":"2020-10-08T11:35:26Z"},"spec":{"cluster_id":"cluster2","cable_name":"submariner-cable-cluster2-172-17-0-5","hostname":"cluster2-worker","subnets":["100.92.0.0/16","10.242.0.0/16"],"private_ip":"172.17.0.5","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
  4172. 2020-10-08T11:45:08.9959484Z [cluster1] I1008 11:44:55.533710 1 route.go:634] Processing local Endpoint cluster1-submariner-cable-cluster1-172-17-0-9 with IP 172.17.0.9, Host cluster1-worker for this cluster
  4173. 2020-10-08T11:45:08.9960827Z [cluster1] I1008 11:44:55.533742 1 route.go:677] This route agent is running on a non-gateway/non-active node
  4174. 2020-10-08T11:45:08.9961682Z [cluster1] I1008 11:44:55.533750 1 route.go:689] Creating the vxlan interface vx-submariner with gateway node IP 172.17.0.9
  4175. 2020-10-08T11:45:08.9962505Z [cluster1] I1008 11:44:55.533925 1 vxlan.go:61] VxLAN interface already exists with same configuration.
  4176. 2020-10-08T11:45:08.9963064Z [cluster1] I1008 11:44:55.536878 1 route.go:866] Reconciling routes to gw: 240.17.0.9
  4177. 2020-10-08T11:45:08.9963581Z [cluster1] I1008 11:44:55.537019 1 route.go:882] Processing route {Ifindex: 16 Dst: 10.242.0.0/16 Src: <nil> Gw: 240.17.0.9 Flags: [] Table: 254}
  4178. 2020-10-08T11:45:08.9964220Z [cluster1] I1008 11:44:55.537040 1 route.go:888] Found route {Ifindex: 16 Dst: 10.242.0.0/16 Src: <nil> Gw: 240.17.0.9 Flags: [] Table: 254} with gw 240.17.0.9 already installed
  4179. 2020-10-08T11:45:08.9964854Z [cluster1] I1008 11:44:55.537048 1 route.go:882] Processing route {Ifindex: 16 Dst: 100.92.0.0/16 Src: <nil> Gw: 240.17.0.9 Flags: [] Table: 254}
  4180. 2020-10-08T11:45:08.9965487Z [cluster1] I1008 11:44:55.537057 1 route.go:888] Found route {Ifindex: 16 Dst: 100.92.0.0/16 Src: <nil> Gw: 240.17.0.9 Flags: [] Table: 254} with gw 240.17.0.9 already installed
  4181. 2020-10-08T11:45:08.9966127Z [cluster1] I1008 11:44:55.537064 1 route.go:882] Processing route {Ifindex: 16 Dst: 240.0.0.0/8 Src: 240.17.0.7 Gw: <nil> Flags: [] Table: 254}
  4182. 2020-10-08T11:45:08.9966608Z [cluster1] I1008 11:44:55.537071 1 route.go:885] Found nil gw or dst
  4183. 2020-10-08T11:45:08.9967043Z [cluster1] I1008 11:44:55.537128 1 route.go:924] Found equivalent route, not adding
  4184. 2020-10-08T11:45:08.9967532Z [cluster1] I1008 11:44:55.537135 1 route.go:924] Found equivalent route, not adding
  4185. 2020-10-08T11:45:08.9968730Z [cluster1] I1008 11:44:55.537173 1 route.go:715] Successfully processed local Endpoint "cluster1-submariner-cable-cluster1-172-17-0-9"
  4186. 2020-10-08T11:45:08.9970815Z [cluster1] I1008 11:44:55.538692 1 route.go:620] Updating iptable rules for remote cluster's Endpoint "cluster2-submariner-cable-cluster2-172-17-0-5" with subnets [100.92.0.0/16 10.242.0.0/16]
  4187. 2020-10-08T11:45:09.2057409Z Error obtaining the Submariner resource: the server could not find the requested resource (get submariners.submariner.io submariner)
  4188. 2020-10-08T11:45:09.2082278Z
  4189. 2020-10-08T11:45:09.2084416Z subctl version: v0.7.0-pre1-30-g7a9e04d
  4190. 2020-10-08T11:45:09.2084698Z
  4191. 2020-10-08T11:45:09.2085034Z [cluster1] ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
  4192. 2020-10-08T11:45:09.2085351Z [cluster1]
  4193. 2020-10-08T11:45:09.2085784Z [cluster1] Showing information for cluster "cluster2":
  4194. 2020-10-08T11:45:09.2086322Z [cluster1] Discovered network details:
  4195. 2020-10-08T11:45:09.2086974Z [cluster1] Network plugin: weave-net
  4196. 2020-10-08T11:45:09.2087402Z [cluster1] Service CIDRs: [100.92.0.0/16]
  4197. 2020-10-08T11:45:09.2087779Z [cluster1] Cluster CIDRs: [10.242.0.0/16]
  4198. 2020-10-08T11:45:09.2088106Z [cluster1]
  4199. 2020-10-08T11:45:09.2088479Z [cluster1] ===================== END Post mortem cluster1 =====================
  4200. 2020-10-08T11:45:09.2095882Z [lighthouse]$ [cluster2] post_analyze
  4201. 2020-10-08T11:45:09.2109217Z [lighthouse]$ [cluster2] sed s/^/[cluster2] /
  4202. 2020-10-08T11:45:09.2112433Z [lighthouse]$ [cluster2] post_analyze
  4203. 2020-10-08T11:45:09.2126661Z [lighthouse]$ [cluster2] kubectl get all --all-namespaces
  4204. 2020-10-08T11:45:09.2141320Z [lighthouse]$ [cluster2] kubectl get all --all-namespaces
  4205. 2020-10-08T11:45:09.2151668Z [lighthouse]$ [cluster2] command kubectl --context=cluster2 get all --all-namespaces
  4206. 2020-10-08T11:45:09.2160815Z [lighthouse]$ [cluster2] kubectl --context=cluster2 get all --all-namespaces
  4207. 2020-10-08T11:45:10.0689096Z
  4208. 2020-10-08T11:45:10.0694415Z
  4209. 2020-10-08T11:45:10.0701624Z
  4210. 2020-10-08T11:45:10.0709293Z
  4211. 2020-10-08T11:45:10.0754762Z
  4212. 2020-10-08T11:45:10.0755157Z [cluster2] ======================= Post mortem cluster2 =======================
  4213. 2020-10-08T11:45:10.0756053Z [cluster2] NAMESPACE NAME READY STATUS RESTARTS AGE
  4214. 2020-10-08T11:45:10.0757733Z [cluster2] kube-system pod/coredns-6955765f44-h4lf6 1/1 Running 0 13m
  4215. 2020-10-08T11:45:10.0758831Z [cluster2] kube-system pod/coredns-6955765f44-lpklx 1/1 Running 0 13m
  4216. 2020-10-08T11:45:10.0759975Z [cluster2] kube-system pod/etcd-cluster2-control-plane 1/1 Running 0 13m
  4217. 2020-10-08T11:45:10.0761361Z [cluster2] kube-system pod/kube-apiserver-cluster2-control-plane 1/1 Running 0 13m
  4218. 2020-10-08T11:45:10.0763079Z [cluster2] kube-system pod/kube-controller-manager-cluster2-control-plane 1/1 Running 0 13m
  4219. 2020-10-08T11:45:10.0764512Z [cluster2] kube-system pod/kube-proxy-9xvqn 1/1 Running 0 12m
  4220. 2020-10-08T11:45:10.0765486Z [cluster2] kube-system pod/kube-proxy-gxzxt 1/1 Running 0 12m
  4221. 2020-10-08T11:45:10.0766455Z [cluster2] kube-system pod/kube-proxy-lz5qm 1/1 Running 0 13m
  4222. 2020-10-08T11:45:10.0767751Z [cluster2] kube-system pod/kube-scheduler-cluster2-control-plane 1/1 Running 0 13m
  4223. 2020-10-08T11:45:10.0768962Z [cluster2] kube-system pod/tiller-deploy-8488d98b4c-frkql 1/1 Running 0 11m
  4224. 2020-10-08T11:45:10.0769917Z [cluster2] kube-system pod/weave-net-6995p 2/2 Running 0 12m
  4225. 2020-10-08T11:45:10.0770748Z [cluster2] kube-system pod/weave-net-ffcfd 2/2 Running 0 12m
  4226. 2020-10-08T11:45:10.0771591Z [cluster2] kube-system pod/weave-net-r6qpq 2/2 Running 0 12m
  4227. 2020-10-08T11:45:10.0772841Z [cluster2] local-path-storage pod/local-path-provisioner-7745554f7f-p7dgn 1/1 Running 0 13m
  4228. 2020-10-08T11:45:10.0774626Z [cluster2] submariner-operator pod/submariner-gateway-98rf6 1/1 Running 0 10m
  4229. 2020-10-08T11:45:10.0776227Z [cluster2] submariner-operator pod/submariner-lighthouse-agent-9df5c9df-c5kgq 1/1 Running 0 10m
  4230. 2020-10-08T11:45:10.0779924Z [cluster2] submariner-operator pod/submariner-lighthouse-coredns-7466b6679c-dppvv 1/1 Running 0 10m
  4231. 2020-10-08T11:45:10.0781844Z [cluster2] submariner-operator pod/submariner-lighthouse-coredns-7466b6679c-x5s2r 1/1 Running 0 10m
  4232. 2020-10-08T11:45:10.0783669Z [cluster2] submariner-operator pod/submariner-routeagent-clqmx 1/1 Running 0 10m
  4233. 2020-10-08T11:45:10.0784971Z [cluster2] submariner-operator pod/submariner-routeagent-gk7px 1/1 Running 0 10m
  4234. 2020-10-08T11:45:10.0785919Z [cluster2] NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
  4235. 2020-10-08T11:45:10.0786641Z [cluster2] default service/kubernetes ClusterIP 100.92.0.1 <none> 443/TCP 13m
  4236. 2020-10-08T11:45:10.0787451Z [cluster2] kube-system service/kube-dns ClusterIP 100.92.0.10 <none> 53/UDP,53/TCP,9153/TCP 13m
  4237. 2020-10-08T11:45:10.0788450Z [cluster2] kube-system service/tiller-deploy ClusterIP 100.92.218.47 <none> 44134/TCP 11m
  4238. 2020-10-08T11:45:10.0789650Z [cluster2] submariner-operator service/submariner-lighthouse-coredns ClusterIP 100.92.17.108 <none> 53/UDP 10m
  4239. 2020-10-08T11:45:10.0790898Z [cluster2] NAMESPACE NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
  4240. 2020-10-08T11:45:10.0791997Z [cluster2] kube-system daemonset.apps/kube-proxy 3 3 3 3 3 beta.kubernetes.io/os=linux 13m
  4241. 2020-10-08T11:45:10.0793341Z [cluster2] kube-system daemonset.apps/weave-net 3 3 3 3 3 <none> 12m
  4242. 2020-10-08T11:45:10.0794567Z [cluster2] submariner-operator daemonset.apps/submariner-gateway 1 1 1 1 1 submariner.io/gateway=true 10m
  4243. 2020-10-08T11:45:10.0795986Z [cluster2] submariner-operator daemonset.apps/submariner-routeagent 2 2 2 2 2 <none> 10m
  4244. 2020-10-08T11:45:10.0808299Z
  4245. 2020-10-08T11:45:10.0812298Z
  4246. 2020-10-08T11:45:10.0816384Z
  4247. 2020-10-08T11:45:10.0820020Z
  4248. 2020-10-08T11:45:10.0913449Z [lighthouse]$ [cluster2] kubectl get pods -A
  4249. 2020-10-08T11:45:10.0938027Z [lighthouse]$ [cluster2] kubectl get pods -A
  4250. 2020-10-08T11:45:10.0942493Z [lighthouse]$ [cluster2] tail -n +2
  4251. 2020-10-08T11:45:10.0956451Z [lighthouse]$ [cluster2] grep -v Running
  4252. 2020-10-08T11:45:10.0971918Z [lighthouse]$ [cluster2] sed s/ */;/g
  4253. 2020-10-08T11:45:10.0996499Z [lighthouse]$ [cluster2] command kubectl --context=cluster2 get pods -A
  4254. 2020-10-08T11:45:10.1021288Z [lighthouse]$ [cluster2] kubectl --context=cluster2 get pods -A
  4255. 2020-10-08T11:45:10.8574959Z [lighthouse]$ [cluster2] namespace=kube-system
  4256. 2020-10-08T11:45:10.8587802Z [lighthouse]$ [cluster2] kubectl get pods --selector=k8s-app=kube-proxy -n kube-system -o jsonpath={.items[*].metadata.name}
  4257. 2020-10-08T11:45:10.8598925Z [lighthouse]$ [cluster2] kubectl get pods --selector=k8s-app=kube-proxy -n kube-system -o jsonpath={.items[*].metadata.name}
  4258. 2020-10-08T11:45:10.8610564Z [lighthouse]$ [cluster2] command kubectl --context=cluster2 get pods --selector=k8s-app=kube-proxy -n kube-system -o jsonpath={.items[*].metadata.name}
  4259. 2020-10-08T11:45:10.8624189Z [lighthouse]$ [cluster2] kubectl --context=cluster2 get pods --selector=k8s-app=kube-proxy -n kube-system -o jsonpath={.items[*].metadata.name}
  4260. 2020-10-08T11:45:11.6422042Z [lighthouse]$ [cluster2] kubectl -n kube-system logs kube-proxy-9xvqn
  4261. 2020-10-08T11:45:11.6430400Z [lighthouse]$ [cluster2] kubectl -n kube-system logs kube-proxy-9xvqn
  4262. 2020-10-08T11:45:11.6441309Z [lighthouse]$ [cluster2] command kubectl --context=cluster2 -n kube-system logs kube-proxy-9xvqn
  4263. 2020-10-08T11:45:11.6453592Z [lighthouse]$ [cluster2] kubectl --context=cluster2 -n kube-system logs kube-proxy-9xvqn
  4264. 2020-10-08T11:45:12.3986661Z [lighthouse]$ [cluster2] kubectl -n kube-system logs kube-proxy-gxzxt
  4265. 2020-10-08T11:45:12.3998686Z [lighthouse]$ [cluster2] kubectl -n kube-system logs kube-proxy-gxzxt
  4266. 2020-10-08T11:45:12.4008954Z [lighthouse]$ [cluster2] command kubectl --context=cluster2 -n kube-system logs kube-proxy-gxzxt
  4267. 2020-10-08T11:45:12.4018863Z [lighthouse]$ [cluster2] kubectl --context=cluster2 -n kube-system logs kube-proxy-gxzxt
  4268. 2020-10-08T11:45:13.1380383Z [cluster2] NAMESPACE NAME READY UP-TO-DATE AVAILABLE AGE
  4269. 2020-10-08T11:45:13.1381211Z [cluster2] kube-system deployment.apps/coredns 2/2 2 2 13m
  4270. 2020-10-08T11:45:13.1381986Z [cluster2] kube-system deployment.apps/tiller-deploy 1/1 1 1 11m
  4271. 2020-10-08T11:45:13.1382972Z [cluster2] local-path-storage deployment.apps/local-path-provisioner 1/1 1 1 13m
  4272. 2020-10-08T11:45:13.1384334Z [cluster2] submariner-operator deployment.apps/submariner-lighthouse-agent 1/1 1 1 10m
  4273. 2020-10-08T11:45:13.1385721Z [cluster2] submariner-operator deployment.apps/submariner-lighthouse-coredns 2/2 2 2 10m
  4274. 2020-10-08T11:45:13.1386607Z [cluster2] NAMESPACE NAME DESIRED CURRENT READY AGE
  4275. 2020-10-08T11:45:13.1387612Z [cluster2] kube-system replicaset.apps/coredns-6955765f44 2 2 2 13m
  4276. 2020-10-08T11:45:13.1388638Z [cluster2] kube-system replicaset.apps/tiller-deploy-8488d98b4c 1 1 1 11m
  4277. 2020-10-08T11:45:13.1389894Z [cluster2] local-path-storage replicaset.apps/local-path-provisioner-7745554f7f 1 1 1 13m
  4278. 2020-10-08T11:45:13.1391425Z [cluster2] submariner-operator replicaset.apps/submariner-lighthouse-agent-9df5c9df 1 1 1 10m
  4279. 2020-10-08T11:45:13.1393244Z [cluster2] submariner-operator replicaset.apps/submariner-lighthouse-coredns-7466b6679c 2 2 2 10m
  4280. 2020-10-08T11:45:13.1394613Z [cluster2] +++++++++++++++++++++: Logs for Pod kube-proxy-9xvqn in namespace kube-system :++++++++++++++++++++++
  4281. 2020-10-08T11:45:13.1395311Z [cluster2] W1008 11:32:34.680931 1 server_others.go:323] Unknown proxy mode "", assuming iptables proxy
  4282. 2020-10-08T11:45:13.1395941Z [cluster2] I1008 11:32:34.693744 1 node.go:135] Successfully retrieved node IP: 172.17.0.4
  4283. 2020-10-08T11:45:13.1396508Z [cluster2] I1008 11:32:34.693847 1 server_others.go:145] Using iptables Proxier.
  4284. 2020-10-08T11:45:13.1397108Z [cluster2] I1008 11:32:34.694110 1 server.go:571] Version: v1.17.0
  4285. 2020-10-08T11:45:13.1397743Z [cluster2] I1008 11:32:34.706279 1 conntrack.go:52] Setting nf_conntrack_max to 131072
  4286. 2020-10-08T11:45:13.1398683Z [cluster2] I1008 11:32:34.706994 1 conntrack.go:100] Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_established' to 86400
  4287. 2020-10-08T11:45:13.1399710Z [cluster2] I1008 11:32:34.707124 1 conntrack.go:100] Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_close_wait' to 3600
  4288. 2020-10-08T11:45:13.1400407Z [cluster2] I1008 11:32:34.708711 1 config.go:131] Starting endpoints config controller
  4289. 2020-10-08T11:45:13.1401055Z [cluster2] I1008 11:32:34.708781 1 shared_informer.go:197] Waiting for caches to sync for endpoints config
  4290. 2020-10-08T11:45:13.1401700Z [cluster2] I1008 11:32:34.708821 1 config.go:313] Starting service config controller
  4291. 2020-10-08T11:45:13.1402338Z [cluster2] I1008 11:32:34.708868 1 shared_informer.go:197] Waiting for caches to sync for service config
  4292. 2020-10-08T11:45:13.1402988Z [cluster2] I1008 11:32:34.811963 1 shared_informer.go:204] Caches are synced for service config
  4293. 2020-10-08T11:45:13.1403636Z [cluster2] I1008 11:32:34.811964 1 shared_informer.go:204] Caches are synced for endpoints config
  4294. 2020-10-08T11:45:13.1404523Z [cluster2] +++++++++++++++++++++: Logs for Pod kube-proxy-gxzxt in namespace kube-system :++++++++++++++++++++++
  4295. 2020-10-08T11:45:13.1405198Z [cluster2] W1008 11:32:33.270690 1 server_others.go:323] Unknown proxy mode "", assuming iptables proxy
  4296. 2020-10-08T11:45:13.1405816Z [cluster2] I1008 11:32:33.286236 1 node.go:135] Successfully retrieved node IP: 172.17.0.5
  4297. 2020-10-08T11:45:13.1406460Z [cluster2] I1008 11:32:33.286261 1 server_others.go:145] Using iptables Proxier.
  4298. 2020-10-08T11:45:13.1406966Z [cluster2] I1008 11:32:33.286641 1 server.go:571] Version: v1.17.0
  4299. 2020-10-08T11:45:13.1407472Z [cluster2] I1008 11:32:33.287087 1 conntrack.go:52] Setting nf_conntrack_max to 131072
  4300. 2020-10-08T11:45:13.1408418Z [cluster2] I1008 11:32:33.291515 1 conntrack.go:100] Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_established' to 86400
  4301. 2020-10-08T11:45:13.1409427Z [cluster2] I1008 11:32:33.291668 1 conntrack.go:100] Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_close_wait' to 3600
  4302. 2020-10-08T11:45:13.1410115Z [cluster2] I1008 11:32:33.294546 1 config.go:313] Starting service config controller
  4303. 2020-10-08T11:45:13.1410751Z [cluster2] I1008 11:32:33.294603 1 shared_informer.go:197] Waiting for caches to sync for service config
  4304. 2020-10-08T11:45:13.1411395Z [cluster2] I1008 11:32:33.294713 1 config.go:131] Starting endpoints config controller
  4305. 2020-10-08T11:45:13.1455887Z [lighthouse]$ [cluster2] kubectl -n kube-system logs kube-proxy-lz5qm
  4306. 2020-10-08T11:45:13.1466007Z [lighthouse]$ [cluster2] kubectl -n kube-system logs kube-proxy-lz5qm
  4307. 2020-10-08T11:45:13.1477576Z [lighthouse]$ [cluster2] command kubectl --context=cluster2 -n kube-system logs kube-proxy-lz5qm
  4308. 2020-10-08T11:45:13.1488460Z [lighthouse]$ [cluster2] kubectl --context=cluster2 -n kube-system logs kube-proxy-lz5qm
  4309. 2020-10-08T11:45:13.9043598Z [lighthouse]$ [cluster2] namespace=submariner-operator
  4310. 2020-10-08T11:45:13.9058977Z [lighthouse]$ [cluster2] kubectl get pods --selector=name=submariner-operator -n submariner-operator -o jsonpath={.items[*].metadata.name}
  4311. 2020-10-08T11:45:13.9071991Z [lighthouse]$ [cluster2] kubectl get pods --selector=name=submariner-operator -n submariner-operator -o jsonpath={.items[*].metadata.name}
  4312. 2020-10-08T11:45:13.9087369Z [lighthouse]$ [cluster2] command kubectl --context=cluster2 get pods --selector=name=submariner-operator -n submariner-operator -o jsonpath={.items[*].metadata.name}
  4313. 2020-10-08T11:45:13.9100249Z [lighthouse]$ [cluster2] kubectl --context=cluster2 get pods --selector=name=submariner-operator -n submariner-operator -o jsonpath={.items[*].metadata.name}
  4314. 2020-10-08T11:45:14.6517246Z [lighthouse]$ [cluster2] namespace=submariner-operator
  4315. 2020-10-08T11:45:14.6529902Z [lighthouse]$ [cluster2] kubectl get pods --selector=app=submariner-globalnet -n submariner-operator -o jsonpath={.items[*].metadata.name}
  4316. 2020-10-08T11:45:14.6540379Z [lighthouse]$ [cluster2] kubectl get pods --selector=app=submariner-globalnet -n submariner-operator -o jsonpath={.items[*].metadata.name}
  4317. 2020-10-08T11:45:14.6554152Z [lighthouse]$ [cluster2] command kubectl --context=cluster2 get pods --selector=app=submariner-globalnet -n submariner-operator -o jsonpath={.items[*].metadata.name}
  4318. 2020-10-08T11:45:14.6564231Z [lighthouse]$ [cluster2] kubectl --context=cluster2 get pods --selector=app=submariner-globalnet -n submariner-operator -o jsonpath={.items[*].metadata.name}
  4319. 2020-10-08T11:45:15.4223081Z [lighthouse]$ [cluster2] kubectl get pods --selector=app=submariner-engine -n submariner-operator -o jsonpath={.items[*].metadata.name}
  4320. 2020-10-08T11:45:15.4235304Z [lighthouse]$ [cluster2] kubectl get pods --selector=app=submariner-engine -n submariner-operator -o jsonpath={.items[*].metadata.name}
  4321. 2020-10-08T11:45:15.4247456Z [lighthouse]$ [cluster2] command kubectl --context=cluster2 get pods --selector=app=submariner-engine -n submariner-operator -o jsonpath={.items[*].metadata.name}
  4322. 2020-10-08T11:45:15.4259886Z [lighthouse]$ [cluster2] kubectl --context=cluster2 get pods --selector=app=submariner-engine -n submariner-operator -o jsonpath={.items[*].metadata.name}
  4323. 2020-10-08T11:45:16.1752337Z [lighthouse]$ [cluster2] kubectl -n submariner-operator logs submariner-gateway-98rf6
  4324. 2020-10-08T11:45:16.1764044Z [lighthouse]$ [cluster2] kubectl -n submariner-operator logs submariner-gateway-98rf6
  4325. 2020-10-08T11:45:16.1774724Z [lighthouse]$ [cluster2] command kubectl --context=cluster2 -n submariner-operator logs submariner-gateway-98rf6
  4326. 2020-10-08T11:45:16.1789085Z [lighthouse]$ [cluster2] kubectl --context=cluster2 -n submariner-operator logs submariner-gateway-98rf6
  4327. 2020-10-08T11:45:16.9341177Z [cluster2] I1008 11:32:33.294750 1 shared_informer.go:197] Waiting for caches to sync for endpoints config
  4328. 2020-10-08T11:45:16.9342346Z [cluster2] I1008 11:32:33.394837 1 shared_informer.go:204] Caches are synced for service config
  4329. 2020-10-08T11:45:16.9342996Z [cluster2] I1008 11:32:33.395074 1 shared_informer.go:204] Caches are synced for endpoints config
  4330. 2020-10-08T11:45:16.9344320Z [cluster2] +++++++++++++++++++++: Logs for Pod kube-proxy-lz5qm in namespace kube-system :++++++++++++++++++++++
  4331. 2020-10-08T11:45:16.9345171Z [cluster2] W1008 11:32:12.253197 1 server_others.go:323] Unknown proxy mode "", assuming iptables proxy
  4332. 2020-10-08T11:45:16.9346117Z [cluster2] I1008 11:32:12.261403 1 node.go:135] Successfully retrieved node IP: 172.17.0.6
  4333. 2020-10-08T11:45:16.9346691Z [cluster2] I1008 11:32:12.261431 1 server_others.go:145] Using iptables Proxier.
  4334. 2020-10-08T11:45:16.9347209Z [cluster2] I1008 11:32:12.261785 1 server.go:571] Version: v1.17.0
  4335. 2020-10-08T11:45:16.9347722Z [cluster2] I1008 11:32:12.262221 1 conntrack.go:52] Setting nf_conntrack_max to 131072
  4336. 2020-10-08T11:45:16.9348726Z [cluster2] I1008 11:32:12.262472 1 conntrack.go:100] Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_established' to 86400
  4337. 2020-10-08T11:45:16.9349757Z [cluster2] I1008 11:32:12.262518 1 conntrack.go:100] Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_close_wait' to 3600
  4338. 2020-10-08T11:45:16.9350443Z [cluster2] I1008 11:32:12.265304 1 config.go:313] Starting service config controller
  4339. 2020-10-08T11:45:16.9351087Z [cluster2] I1008 11:32:12.265385 1 shared_informer.go:197] Waiting for caches to sync for service config
  4340. 2020-10-08T11:45:16.9351734Z [cluster2] I1008 11:32:12.265420 1 config.go:131] Starting endpoints config controller
  4341. 2020-10-08T11:45:16.9352379Z [cluster2] I1008 11:32:12.265446 1 shared_informer.go:197] Waiting for caches to sync for endpoints config
  4342. 2020-10-08T11:45:16.9353046Z [cluster2] I1008 11:32:12.365534 1 shared_informer.go:204] Caches are synced for service config
  4343. 2020-10-08T11:45:16.9353756Z [cluster2] I1008 11:32:12.366322 1 shared_informer.go:204] Caches are synced for endpoints config
  4344. 2020-10-08T11:45:16.9354182Z [cluster2] ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
  4345. 2020-10-08T11:45:16.9354459Z [cluster2] ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
  4346. 2020-10-08T11:45:16.9354738Z [cluster2] ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
  4347. 2020-10-08T11:45:16.9355533Z [cluster2] +++++++++++++++++++++: Logs for Pod submariner-gateway-98rf6 in namespace submariner-operator :++++++++++++++++++++++
  4348. 2020-10-08T11:45:16.9356284Z [cluster2] + trap 'exit 1' SIGTERM SIGINT
  4349. 2020-10-08T11:45:16.9356898Z [cluster2] + export CHARON_PID_FILE=/var/run/charon.pid
  4350. 2020-10-08T11:45:16.9357371Z [cluster2] + CHARON_PID_FILE=/var/run/charon.pid
  4351. 2020-10-08T11:45:16.9357920Z [cluster2] + rm -f /var/run/charon.pid
  4352. 2020-10-08T11:45:16.9358320Z [cluster2] + SUBMARINER_VERBOSITY=2
  4353. 2020-10-08T11:45:16.9358826Z [cluster2] + '[' false == true ']'
  4354. 2020-10-08T11:45:16.9359258Z [cluster2] + DEBUG=-v=2
  4355. 2020-10-08T11:45:16.9359705Z [cluster2] + mkdir -p /etc/ipsec
  4356. 2020-10-08T11:45:16.9360354Z [cluster2] + sysctl -w net.ipv4.conf.all.send_redirects=0
  4357. 2020-10-08T11:45:16.9360978Z [cluster2] net.ipv4.conf.all.send_redirects = 0
  4358. 2020-10-08T11:45:16.9361643Z [cluster2] + export PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/libexec/strongswan
  4359. 2020-10-08T11:45:16.9362529Z [cluster2] + PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/libexec/strongswan
  4360. 2020-10-08T11:45:16.9363263Z [cluster2] + for f in iptables-save iptables
  4361. 2020-10-08T11:45:16.9363893Z [cluster2] ++ find_iptables_on_host iptables-save
  4362. 2020-10-08T11:45:16.9364549Z [cluster2] ++ chroot /host test -x /usr/sbin/iptables-save
  4363. 2020-10-08T11:45:16.9365312Z [cluster2] chroot: cannot change root directory to '/host': No such file or directory
  4364. 2020-10-08T11:45:16.9366245Z [cluster2] ++ chroot /host test -x /sbin/iptables-save
  4365. 2020-10-08T11:45:16.9367054Z [cluster2] chroot: cannot change root directory to '/host': No such file or directory
  4366. 2020-10-08T11:45:16.9367582Z [cluster2] ++ echo unknown
  4367. 2020-10-08T11:45:16.9367954Z [cluster2] + location=unknown
  4368. 2020-10-08T11:45:16.9368479Z [cluster2] + '[' unknown '!=' unknown ']'
  4369. 2020-10-08T11:45:16.9369276Z [cluster2] + echo 'WARNING: not using iptables wrapper because iptables was not detected on the'
  4370. 2020-10-08T11:45:16.9370060Z [cluster2] WARNING: not using iptables wrapper because iptables was not detected on the
  4371. 2020-10-08T11:45:16.9379150Z [cluster2] + echo 'host at the following paths [/usr/sbin, /sbin].'
  4372. 2020-10-08T11:45:16.9379966Z [cluster2] host at the following paths [/usr/sbin, /sbin].
  4373. 2020-10-08T11:45:16.9381041Z [cluster2] + echo 'Either the host file system isn'\''t mounted or the host does not have iptables'
  4374. 2020-10-08T11:45:16.9381962Z [cluster2] Either the host file system isn't mounted or the host does not have iptables
  4375. 2020-10-08T11:45:16.9382876Z [cluster2] + echo 'installed. The pod will use the image installed iptables version.'
  4376. 2020-10-08T11:45:16.9383604Z [cluster2] installed. The pod will use the image installed iptables version.
  4377. 2020-10-08T11:45:16.9384369Z [cluster2] + for f in iptables-save iptables
  4378. 2020-10-08T11:45:16.9384837Z [cluster2] ++ find_iptables_on_host iptables
  4379. 2020-10-08T11:45:16.9385474Z [cluster2] ++ chroot /host test -x /usr/sbin/iptables
  4380. 2020-10-08T11:45:16.9386260Z [cluster2] chroot: cannot change root directory to '/host': No such file or directory
  4381. 2020-10-08T11:45:16.9387031Z [cluster2] ++ chroot /host test -x /sbin/iptables
  4382. 2020-10-08T11:45:16.9387806Z [cluster2] chroot: cannot change root directory to '/host': No such file or directory
  4383. 2020-10-08T11:45:16.9388334Z [cluster2] ++ echo unknown
  4384. 2020-10-08T11:45:16.9388703Z [cluster2] + location=unknown
  4385. 2020-10-08T11:45:16.9389245Z [cluster2] + '[' unknown '!=' unknown ']'
  4386. 2020-10-08T11:45:16.9390024Z [cluster2] + echo 'WARNING: not using iptables wrapper because iptables was not detected on the'
  4387. 2020-10-08T11:45:16.9390802Z [cluster2] WARNING: not using iptables wrapper because iptables was not detected on the
  4388. 2020-10-08T11:45:16.9392451Z [cluster2] + echo 'host at the following paths [/usr/sbin, /sbin].'
  4389. 2020-10-08T11:45:16.9393009Z [cluster2] host at the following paths [/usr/sbin, /sbin].
  4390. 2020-10-08T11:45:16.9393871Z [cluster2] + echo 'Either the host file system isn'\''t mounted or the host does not have iptables'
  4391. 2020-10-08T11:45:16.9394784Z [cluster2] Either the host file system isn't mounted or the host does not have iptables
  4392. 2020-10-08T11:45:16.9395700Z [cluster2] + echo 'installed. The pod will use the image installed iptables version.'
  4393. 2020-10-08T11:45:16.9396428Z [cluster2] installed. The pod will use the image installed iptables version.
  4394. 2020-10-08T11:45:16.9397301Z [cluster2] + exec submariner-engine -v=2 -alsologtostderr
  4395. 2020-10-08T11:45:16.9397970Z [cluster2] I1008 11:35:24.436434 1 main.go:71] Starting the submariner gateway engine
  4396. 2020-10-08T11:45:16.9399116Z [cluster2] W1008 11:35:24.436931 1 client_config.go:543] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work.
  4397. 2020-10-08T11:45:16.9399955Z [cluster2] I1008 11:35:24.437365 1 main.go:99] Creating the cable engine
  4398. 2020-10-08T11:45:16.9400520Z [cluster2] I1008 11:35:24.437563 1 syncer.go:51] CableEngine syncer started
  4399. 2020-10-08T11:45:16.9401639Z [cluster2] I1008 11:35:24.439235 1 main.go:263] Gateway leader election config values: main.leaderConfig{LeaseDuration:10, RenewDeadline:5, RetryPeriod:2}
  4400. 2020-10-08T11:45:16.9402990Z [cluster2] I1008 11:35:24.439557 1 main.go:280] Using namespace "submariner-operator" for the leader election lock
  4401. 2020-10-08T11:45:16.9404409Z [cluster2] I1008 11:35:24.439577 1 leaderelection.go:242] attempting to acquire leader lease submariner-operator/submariner-engine-lock...
  4402. 2020-10-08T11:45:16.9406031Z [cluster2] I1008 11:35:24.939929 1 leaderelection.go:252] successfully acquired lease submariner-operator/submariner-engine-lock
  4403. 2020-10-08T11:45:16.9408774Z [cluster2] I1008 11:35:24.940191 1 event.go:281] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"submariner-operator", Name:"submariner-engine-lock", UID:"86a1526e-48c7-4b31-96e3-7b3da2d3c22e", APIVersion:"v1", ResourceVersion:"1271", FieldPath:""}): type: 'Normal' reason: 'LeaderElection' cluster2-worker-submariner-engine became leader
  4404. 2020-10-08T11:45:16.9410644Z [cluster2] I1008 11:35:24.940269 1 main.go:133] Creating the tunnel controller
  4405. 2020-10-08T11:45:16.9411375Z [cluster2] I1008 11:35:24.940361 1 main.go:141] Creating the kubernetes central datastore
  4406. 2020-10-08T11:45:16.9412160Z [cluster2] I1008 11:35:24.940420 1 kubernetes.go:60] Rendered API server host: "https://172.17.0.8:6443"
  4407. 2020-10-08T11:45:16.9423127Z [cluster2] I1008 11:35:25.045037 1 main.go:151] Creating the datastore syncer
  4408. 2020-10-08T11:45:16.9423806Z [cluster2] I1008 11:35:25.046008 1 strongswan.go:99] Initializing StrongSwan IPSec driver
  4409. 2020-10-08T11:45:16.9424465Z [cluster2] I1008 11:35:25.049037 1 strongswan.go:460] Starting charon
  4410. 2020-10-08T11:45:16.9425689Z [cluster2] W1008 11:35:25.050610 1 strongswan.go:543] Failed to connect to charon - retrying: dial unix /var/run/charon.vici: connect: no such file or directory
  4411. 2020-10-08T11:45:16.9426794Z [cluster2] 00[DMN] Starting IKE charon daemon (strongSwan 5.9.0, Linux 5.4.0-1026-azure, x86_64)
  4412. 2020-10-08T11:45:16.9427584Z [cluster2] 00[CFG] PKCS11 module '<name>' lacks library path
  4413. 2020-10-08T11:45:16.9428266Z [cluster2] 00[LIB] openssl FIPS mode(0) - disabled
  4414. 2020-10-08T11:45:16.9429074Z [cluster2] 00[CFG] loading ca certificates from '/etc/strongswan/ipsec.d/cacerts'
  4415. 2020-10-08T11:45:16.9429992Z [cluster2] 00[CFG] loading aa certificates from '/etc/strongswan/ipsec.d/aacerts'
  4416. 2020-10-08T11:45:16.9430978Z [cluster2] 00[CFG] loading ocsp signer certificates from '/etc/strongswan/ipsec.d/ocspcerts'
  4417. 2020-10-08T11:45:16.9432095Z [cluster2] 00[CFG] loading attribute certificates from '/etc/strongswan/ipsec.d/acerts'
  4418. 2020-10-08T11:45:16.9432915Z [cluster2] 00[CFG] loading crls from '/etc/strongswan/ipsec.d/crls'
  4419. 2020-10-08T11:45:16.9433678Z [cluster2] 00[CFG] loading secrets from '/etc/strongswan/ipsec.secrets'
  4420. 2020-10-08T11:45:16.9434422Z [cluster2] 00[CFG] opening triplet file /etc/strongswan/ipsec.d/triplets.dat failed: No such file or directory
  4421. 2020-10-08T11:45:16.9435122Z [cluster2] 00[CFG] loaded 0 RADIUS server configurations
  4422. 2020-10-08T11:45:16.9435643Z [cluster2] 00[CFG] HA config misses local/remote address
  4423. 2020-10-08T11:45:16.9436336Z [cluster2] 00[CFG] no script for ext-auth script defined, disabled
  4424. 2020-10-08T11:45:16.9439374Z [cluster2] 00[LIB] loaded plugins: charon pkcs11 tpm aesni aes des rc2 sha2 sha1 md4 md5 mgf1 random nonce x509 revocation constraints acert pubkey pkcs1 pkcs7 pkcs8 pkcs12 pgp dnskey sshkey pem openssl gcrypt fips-prf gmp curve25519 chapoly xcbc cmac hmac ctr ccm gcm drbg newhope curl attr kernel-netlink resolve socket-default farp stroke vici updown eap-identity eap-sim eap-aka eap-aka-3gpp eap-aka-3gpp2 eap-md5 eap-gtc eap-mschapv2 eap-dynamic eap-radius eap-tls eap-ttls eap-peap xauth-generic xauth-eap xauth-pam xauth-noauth dhcp led duplicheck unity counters
  4425. 2020-10-08T11:45:16.9441589Z [cluster2] 00[JOB] spawning 16 worker threads
  4426. 2020-10-08T11:45:16.9442272Z [cluster2] I1008 11:35:26.051808 1 cableengine.go:71] CableEngine controller started, driver: "strongswan"
  4427. 2020-10-08T11:45:16.9443226Z [cluster2] I1008 11:35:26.051932 1 datastoresyncer.go:145] Starting the datastore syncer
  4428. 2020-10-08T11:45:16.9443999Z [cluster2] I1008 11:35:26.051974 1 datastoresyncer.go:147] Waiting for informer caches to sync
  4429. 2020-10-08T11:45:16.9444833Z [cluster2] I1008 11:35:26.052000 1 datastoresyncer.go:79] Ensuring we are the only endpoint active for this cluster
  4430. 2020-10-08T11:45:16.9445580Z [cluster2] I1008 11:35:26.051957 1 tunnel.go:47] Starting the tunnel controller
  4431. 2020-10-08T11:45:16.9446166Z [cluster2] I1008 11:35:26.052316 1 tunnel.go:50] Waiting for informer caches to sync
  4432. 2020-10-08T11:45:16.9446754Z [cluster2] I1008 11:35:26.052335 1 tunnel.go:58] Tunnel controller started
  4433. 2020-10-08T11:45:16.9448272Z [cluster2] I1008 11:35:26.055860 1 datastoresyncer.go:158] Reconciling local submariner Cluster: types.SubmarinerCluster{ID:"cluster2", Spec:v1.ClusterSpec{ClusterID:"cluster2", ColorCodes:[]string{"blue"}, ServiceCIDR:[]string{"100.92.0.0/16"}, ClusterCIDR:[]string{"10.242.0.0/16"}, GlobalCIDR:[]string{}}}
  4434. 2020-10-08T11:45:16.9462548Z [cluster2] I1008 11:35:26.055909 1 datastoresyncer.go:312] In reconcileClusterCRD: &types.SubmarinerCluster{ID:"cluster2", Spec:v1.ClusterSpec{ClusterID:"cluster2", ColorCodes:[]string{"blue"}, ServiceCIDR:[]string{"100.92.0.0/16"}, ClusterCIDR:[]string{"10.242.0.0/16"}, GlobalCIDR:[]string{}}}
  4435. 2020-10-08T11:45:16.9464191Z [cluster2] I1008 11:35:26.066212 1 datastoresyncer.go:357] Successfully created submariner Cluster "cluster2" in the local datastore
  4436. 2020-10-08T11:45:16.9466832Z [cluster2] I1008 11:35:26.066336 1 datastoresyncer.go:164] Reconciling local submariner Endpoint: types.SubmarinerEndpoint{Spec:v1.EndpointSpec{ClusterID:"cluster2", CableName:"submariner-cable-cluster2-172-17-0-5", Hostname:"cluster2-worker", Subnets:[]string{"100.92.0.0/16", "10.242.0.0/16"}, PrivateIP:"172.17.0.5", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string{}}}
  4437. 2020-10-08T11:45:16.9471320Z [cluster2] I1008 11:35:26.066404 1 datastoresyncer.go:387] In reconcileEndpointCRD: &types.SubmarinerEndpoint{Spec:v1.EndpointSpec{ClusterID:"cluster2", CableName:"submariner-cable-cluster2-172-17-0-5", Hostname:"cluster2-worker", Subnets:[]string{"100.92.0.0/16", "10.242.0.0/16"}, PrivateIP:"172.17.0.5", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string{}}}
  4438. 2020-10-08T11:45:16.9481559Z [cluster2] I1008 11:35:26.074893 1 datastoresyncer.go:432] Successfully created submariner Endpoint "cluster2-submariner-cable-cluster2-172-17-0-5" in the local datastore
  4439. 2020-10-08T11:45:16.9482916Z [cluster2] I1008 11:35:26.075723 1 datastoresyncer.go:177] Datastore syncer started
  4440. 2020-10-08T11:45:16.9490129Z [cluster2] I1008 11:35:26.076867 1 kubernetes.go:317] In SetCluster: &types.SubmarinerCluster{ID:"cluster2", Spec:v1.ClusterSpec{ClusterID:"cluster2", ColorCodes:[]string{"blue"}, ServiceCIDR:[]string{"100.92.0.0/16"}, ClusterCIDR:[]string{"10.242.0.0/16"}, GlobalCIDR:[]string{}}}
  4441. 2020-10-08T11:45:16.9534199Z [cluster2] I1008 11:35:26.082263 1 tunnel.go:95] Tunnel controller processing added or updated submariner Endpoint object: &v1.Endpoint{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"cluster2-submariner-cable-cluster2-172-17-0-5", GenerateName:"", Namespace:"submariner-operator", SelfLink:"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster2-submariner-cable-cluster2-172-17-0-5", UID:"6a117963-5044-4b34-9df6-57ebb8a4981d", ResourceVersion:"1284", Generation:1, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63737753726, loc:(*time.Location)(0x225ef80)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v1.EndpointSpec{ClusterID:"cluster2", CableName:"submariner-cable-cluster2-172-17-0-5", Hostname:"cluster2-worker", Subnets:[]string{"100.92.0.0/16", "10.242.0.0/16"}, PrivateIP:"172.17.0.5", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  4442. 2020-10-08T11:45:16.9539625Z [cluster2] I1008 11:35:26.082344 1 cableengine.go:92] Not installing cable for local cluster
  4443. 2020-10-08T11:45:16.9541202Z [cluster2] I1008 11:35:26.082357 1 tunnel.go:108] Tunnel controller successfully installed Endpoint cable submariner-cable-cluster2-172-17-0-5 in the engine
  4444. 2020-10-08T11:45:16.9542405Z [cluster2] 10[KNL] interface vx-submariner activated
  4445. 2020-10-08T11:45:16.9550971Z [cluster2] I1008 11:35:26.095860 1 datastoresyncer.go:287] Processing local submariner Endpoint object: &v1.Endpoint{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"cluster2-submariner-cable-cluster2-172-17-0-5", GenerateName:"", Namespace:"submariner-operator", SelfLink:"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster2-submariner-cable-cluster2-172-17-0-5", UID:"6a117963-5044-4b34-9df6-57ebb8a4981d", ResourceVersion:"1284", Generation:1, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63737753726, loc:(*time.Location)(0x225ef80)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v1.EndpointSpec{ClusterID:"cluster2", CableName:"submariner-cable-cluster2-172-17-0-5", Hostname:"cluster2-worker", Subnets:[]string{"100.92.0.0/16", "10.242.0.0/16"}, PrivateIP:"172.17.0.5", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  4446. 2020-10-08T11:45:16.9557088Z [cluster2] 01[KNL] 240.17.0.5 appeared on vx-submariner
  4447. 2020-10-08T11:45:16.9559526Z [cluster2] I1008 11:35:26.097560 1 kubernetes.go:370] In SetEndpoint: &types.SubmarinerEndpoint{Spec:v1.EndpointSpec{ClusterID:"cluster2", CableName:"submariner-cable-cluster2-172-17-0-5", Hostname:"cluster2-worker", Subnets:[]string{"100.92.0.0/16", "10.242.0.0/16"}, PrivateIP:"172.17.0.5", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  4448. 2020-10-08T11:45:16.9561524Z [cluster2] I1008 11:35:26.097485 1 kubernetes.go:240] AddFunc in WatchEndpoints called
  4449. 2020-10-08T11:45:16.9564171Z [cluster2] I1008 11:35:26.097705 1 datastoresyncer.go:387] In reconcileEndpointCRD: &types.SubmarinerEndpoint{Spec:v1.EndpointSpec{ClusterID:"cluster1", CableName:"submariner-cable-cluster1-172-17-0-9", Hostname:"cluster1-worker", Subnets:[]string{"100.91.0.0/16", "10.241.0.0/16"}, PrivateIP:"172.17.0.9", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  4450. 2020-10-08T11:45:16.9566443Z [cluster2] I1008 11:35:26.105137 1 kubernetes.go:155] AddFunc in WatchClusters called
  4451. 2020-10-08T11:45:16.9567840Z [cluster2] I1008 11:35:26.105167 1 datastoresyncer.go:312] In reconcileClusterCRD: &types.SubmarinerCluster{ID:"cluster1", Spec:v1.ClusterSpec{ClusterID:"cluster1", ColorCodes:[]string{"blue"}, ServiceCIDR:[]string{"100.91.0.0/16"}, ClusterCIDR:[]string{"10.241.0.0/16"}, GlobalCIDR:[]string{}}}
  4452. 2020-10-08T11:45:16.9570181Z [cluster2] I1008 11:35:26.114059 1 datastoresyncer.go:432] Successfully created submariner Endpoint "cluster1-submariner-cable-cluster1-172-17-0-9" in the local datastore
  4453. 2020-10-08T11:45:16.9571720Z [cluster2] I1008 11:35:26.116254 1 kubernetes.go:342] Successfully created submariner Cluster "cluster2" in the central datastore
  4454. 2020-10-08T11:45:16.9578527Z [cluster2] I1008 11:35:26.119845 1 tunnel.go:95] Tunnel controller processing added or updated submariner Endpoint object: &v1.Endpoint{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"cluster1-submariner-cable-cluster1-172-17-0-9", GenerateName:"", Namespace:"submariner-operator", SelfLink:"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster1-submariner-cable-cluster1-172-17-0-9", UID:"60d04732-7b08-4685-b72b-cb2c12702817", ResourceVersion:"1285", Generation:1, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63737753726, loc:(*time.Location)(0x225ef80)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v1.EndpointSpec{ClusterID:"cluster1", CableName:"submariner-cable-cluster1-172-17-0-9", Hostname:"cluster1-worker", Subnets:[]string{"100.91.0.0/16", "10.241.0.0/16"}, PrivateIP:"172.17.0.9", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  4455. 2020-10-08T11:45:16.9584452Z [cluster2] I1008 11:35:26.123416 1 cableengine.go:101] Installing Endpoint cable "submariner-cable-cluster1-172-17-0-9"
  4456. 2020-10-08T11:45:16.9585461Z [cluster2] I1008 11:35:26.120314 1 kubernetes.go:240] AddFunc in WatchEndpoints called
  4457. 2020-10-08T11:45:16.9587129Z [cluster2] I1008 11:35:26.123353 1 kubernetes.go:395] Successfully created submariner Endpoint "cluster2-submariner-cable-cluster2-172-17-0-5" in the central datastore
  4458. 2020-10-08T11:45:16.9588541Z [cluster2] 07[CFG] loaded IKE shared key for: '172.17.0.9'
  4459. 2020-10-08T11:45:16.9589473Z [cluster2] 05[CFG] added vici connection: submariner-cable-cluster1-172-17-0-9
  4460. 2020-10-08T11:45:16.9591015Z [cluster2] 05[CFG] initiating 'submariner-child-submariner-cable-cluster1-172-17-0-9'
  4461. 2020-10-08T11:45:16.9592528Z [cluster2] 05[IKE] initiating IKE_SA submariner-cable-cluster1-172-17-0-9[1] to 172.17.0.9
  4462. 2020-10-08T11:45:16.9593344Z [cluster2] 05[ENC] generating IKE_SA_INIT request 0 [ SA KE No N(NATD_S_IP) N(NATD_D_IP) N(FRAG_SUP) N(HASH_ALG) N(REDIR_SUP) ]
  4463. 2020-10-08T11:45:16.9593939Z [cluster2] 05[NET] sending packet: from 172.17.0.5[500] to 172.17.0.9[500] (500 bytes)
  4464. 2020-10-08T11:45:16.9594363Z [cluster2] 10[NET] received packet: from 172.17.0.9[500] to 172.17.0.5[500] (36 bytes)
  4465. 2020-10-08T11:45:16.9594796Z [cluster2] 10[ENC] parsed IKE_SA_INIT response 0 [ N(NO_PROP) ]
  4466. 2020-10-08T11:45:16.9595235Z [cluster2] 10[IKE] received NO_PROPOSAL_CHOSEN notify error
  4467. 2020-10-08T11:45:16.9596191Z [cluster2] I1008 11:35:26.137354 1 datastoresyncer.go:271] The updated submariner Endpoint "cluster1" is not for this cluster - skipping updating the datastore
  4468. 2020-10-08T11:45:16.9600410Z [cluster2] I1008 11:35:26.137529 1 datastoresyncer.go:357] Successfully created submariner Cluster "cluster1" in the local datastore
  4469. 2020-10-08T11:45:16.9601148Z [cluster2] I1008 11:35:26.137557 1 kubernetes.go:155] AddFunc in WatchClusters called
  4470. 2020-10-08T11:45:16.9602425Z [cluster2] I1008 11:35:26.137721 1 cableengine.go:150] Successfully installed Endpoint cable "submariner-cable-cluster1-172-17-0-9" with remote IP 172.17.0.9
  4471. 2020-10-08T11:45:16.9603296Z [cluster2] 08[NET] received packet: from 172.17.0.9[500] to 172.17.0.5[500] (500 bytes)
  4472. 2020-10-08T11:45:16.9603841Z [cluster2] 08[ENC] parsed IKE_SA_INIT request 0 [ SA KE No N(NATD_S_IP) N(NATD_D_IP) N(FRAG_SUP) N(HASH_ALG) N(REDIR_SUP) ]
  4473. 2020-10-08T11:45:16.9604666Z [cluster2] 08[IKE] 172.17.0.9 is initiating an IKE_SA
  4474. 2020-10-08T11:45:16.9605164Z [cluster2] 08[CFG] selected proposal: IKE:AES_GCM_16_128/PRF_HMAC_SHA2_256/MODP_2048
  4475. 2020-10-08T11:45:16.9606350Z [cluster2] I1008 11:35:26.153947 1 tunnel.go:108] Tunnel controller successfully installed Endpoint cable submariner-cable-cluster1-172-17-0-9 in the engine
  4476. 2020-10-08T11:45:16.9607198Z [cluster2] 08[IKE] remote host is behind NAT
  4477. 2020-10-08T11:45:16.9607747Z [cluster2] 08[ENC] generating IKE_SA_INIT response 0 [ SA KE No N(NATD_S_IP) N(NATD_D_IP) N(FRAG_SUP) N(HASH_ALG) N(CHDLESS_SUP) N(MULT_AUTH) ]
  4478. 2020-10-08T11:45:16.9608318Z [cluster2] 08[NET] sending packet: from 172.17.0.5[500] to 172.17.0.9[500] (464 bytes)
  4479. 2020-10-08T11:45:16.9608737Z [cluster2] 11[NET] received packet: from 172.17.0.9[4500] to 172.17.0.5[4500] (333 bytes)
  4480. 2020-10-08T11:45:16.9609412Z [cluster2] 11[ENC] parsed IKE_AUTH request 1 [ IDi N(INIT_CONTACT) IDr AUTH SA TSi TSr N(MULT_AUTH) N(EAP_ONLY) N(MSG_ID_SYN_SUP) ]
  4481. 2020-10-08T11:45:16.9609977Z [cluster2] 11[CFG] looking for peer configs matching 172.17.0.5[172.17.0.5]...172.17.0.9[172.17.0.9]
  4482. 2020-10-08T11:45:16.9610831Z [cluster2] 11[CFG] selected peer config 'submariner-cable-cluster1-172-17-0-9'
  4483. 2020-10-08T11:45:16.9611743Z [cluster2] 11[IKE] authentication of '172.17.0.9' with pre-shared key successful
  4484. 2020-10-08T11:45:16.9612612Z [cluster2] 11[IKE] authentication of '172.17.0.5' (myself) with pre-shared key
  4485. 2020-10-08T11:45:16.9613830Z [cluster2] 11[IKE] IKE_SA submariner-cable-cluster1-172-17-0-9[2] established between 172.17.0.5[172.17.0.5]...172.17.0.9[172.17.0.9]
  4486. 2020-10-08T11:45:16.9614687Z [cluster2] 11[IKE] scheduling rekeying in 12964s
  4487. 2020-10-08T11:45:16.9615066Z [cluster2] 11[IKE] maximum IKE_SA lifetime 14404s
  4488. 2020-10-08T11:45:16.9615477Z [cluster2] 11[CFG] selected proposal: ESP:AES_GCM_16_128/NO_EXT_SEQ
  4489. 2020-10-08T11:45:16.9617038Z [cluster2] 11[IKE] CHILD_SA submariner-child-submariner-cable-cluster1-172-17-0-9{1} established with SPIs cc5f34fc_i c3e8a822_o and TS 10.242.0.0/16 100.92.0.0/16 172.17.0.5/32 === 10.241.0.0/16 100.91.0.0/16 172.17.0.9/32
  4490. 2020-10-08T11:45:16.9618206Z [cluster2] 11[ENC] generating IKE_AUTH response 1 [ IDr AUTH SA TSi TSr ]
  4491. 2020-10-08T11:45:16.9618665Z [cluster2] 11[NET] sending packet: from 172.17.0.5[4500] to 172.17.0.9[4500] (257 bytes)
  4492. 2020-10-08T11:45:16.9619196Z [cluster2] I1008 11:35:56.097628 1 kubernetes.go:264] UpdateFunc in WatchEndpoints called
  4493. 2020-10-08T11:45:16.9621374Z [cluster2] I1008 11:35:56.097654 1 datastoresyncer.go:387] In reconcileEndpointCRD: &types.SubmarinerEndpoint{Spec:v1.EndpointSpec{ClusterID:"cluster1", CableName:"submariner-cable-cluster1-172-17-0-9", Hostname:"cluster1-worker", Subnets:[]string{"100.91.0.0/16", "10.241.0.0/16"}, PrivateIP:"172.17.0.9", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  4494. 2020-10-08T11:45:16.9623889Z [cluster2] I1008 11:35:56.100145 1 datastoresyncer.go:435] Endpoint "cluster1-submariner-cable-cluster1-172-17-0-9" matched what we received from datastore - not updating
  4495. 2020-10-08T11:45:16.9625038Z [cluster2] I1008 11:35:56.100184 1 kubernetes.go:264] UpdateFunc in WatchEndpoints called
  4496. 2020-10-08T11:45:16.9625658Z [cluster2] I1008 11:35:56.105152 1 kubernetes.go:180] UpdateFunc in WatchClusters called
  4497. 2020-10-08T11:45:16.9627213Z [cluster2] I1008 11:35:56.105194 1 datastoresyncer.go:312] In reconcileClusterCRD: &types.SubmarinerCluster{ID:"cluster1", Spec:v1.ClusterSpec{ClusterID:"cluster1", ColorCodes:[]string{"blue"}, ServiceCIDR:[]string{"100.91.0.0/16"}, ClusterCIDR:[]string{"10.241.0.0/16"}, GlobalCIDR:[]string{}}}
  4498. 2020-10-08T11:45:16.9629388Z [cluster2] I1008 11:35:56.107309 1 datastoresyncer.go:360] Cluster "cluster1" matched what we received from datastore - not updating
  4499. 2020-10-08T11:45:16.9630354Z [cluster2] I1008 11:35:56.107340 1 kubernetes.go:180] UpdateFunc in WatchClusters called
  4500. 2020-10-08T11:45:16.9631018Z [cluster2] 14[KNL] interface vethwepl6787a7e deactivated
  4501. 2020-10-08T11:45:16.9631586Z [cluster2] 09[KNL] interface vethwepl6787a7e deleted
  4502. 2020-10-08T11:45:16.9638139Z [cluster2] I1008 11:36:25.057583 1 tunnel.go:95] Tunnel controller processing added or updated submariner Endpoint object: &v1.Endpoint{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"cluster2-submariner-cable-cluster2-172-17-0-5", GenerateName:"", Namespace:"submariner-operator", SelfLink:"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster2-submariner-cable-cluster2-172-17-0-5", UID:"6a117963-5044-4b34-9df6-57ebb8a4981d", ResourceVersion:"1284", Generation:1, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63737753726, loc:(*time.Location)(0x225ef80)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v1.EndpointSpec{ClusterID:"cluster2", CableName:"submariner-cable-cluster2-172-17-0-5", Hostname:"cluster2-worker", Subnets:[]string{"100.92.0.0/16", "10.242.0.0/16"}, PrivateIP:"172.17.0.5", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  4503. 2020-10-08T11:45:16.9644508Z [cluster2] I1008 11:36:25.057676 1 cableengine.go:92] Not installing cable for local cluster
  4504. 2020-10-08T11:45:16.9645937Z [cluster2] I1008 11:36:25.057931 1 tunnel.go:108] Tunnel controller successfully installed Endpoint cable submariner-cable-cluster2-172-17-0-5 in the engine
  4505. 2020-10-08T11:45:16.9653452Z [cluster2] I1008 11:36:25.057948 1 tunnel.go:95] Tunnel controller processing added or updated submariner Endpoint object: &v1.Endpoint{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"cluster1-submariner-cable-cluster1-172-17-0-9", GenerateName:"", Namespace:"submariner-operator", SelfLink:"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster1-submariner-cable-cluster1-172-17-0-9", UID:"60d04732-7b08-4685-b72b-cb2c12702817", ResourceVersion:"1285", Generation:1, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63737753726, loc:(*time.Location)(0x225ef80)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v1.EndpointSpec{ClusterID:"cluster1", CableName:"submariner-cable-cluster1-172-17-0-9", Hostname:"cluster1-worker", Subnets:[]string{"100.91.0.0/16", "10.241.0.0/16"}, PrivateIP:"172.17.0.9", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  4506. 2020-10-08T11:45:16.9659908Z [cluster2] I1008 11:36:25.058021 1 cableengine.go:101] Installing Endpoint cable "submariner-cable-cluster1-172-17-0-9"
  4507. 2020-10-08T11:45:16.9666965Z [cluster2] I1008 11:36:25.059899 1 datastoresyncer.go:287] Processing local submariner Endpoint object: &v1.Endpoint{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"cluster2-submariner-cable-cluster2-172-17-0-5", GenerateName:"", Namespace:"submariner-operator", SelfLink:"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster2-submariner-cable-cluster2-172-17-0-5", UID:"6a117963-5044-4b34-9df6-57ebb8a4981d", ResourceVersion:"1284", Generation:1, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63737753726, loc:(*time.Location)(0x225ef80)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v1.EndpointSpec{ClusterID:"cluster2", CableName:"submariner-cable-cluster2-172-17-0-5", Hostname:"cluster2-worker", Subnets:[]string{"100.92.0.0/16", "10.242.0.0/16"}, PrivateIP:"172.17.0.5", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  4508. 2020-10-08T11:45:16.9673850Z [cluster2] I1008 11:36:25.059988 1 kubernetes.go:370] In SetEndpoint: &types.SubmarinerEndpoint{Spec:v1.EndpointSpec{ClusterID:"cluster2", CableName:"submariner-cable-cluster2-172-17-0-5", Hostname:"cluster2-worker", Subnets:[]string{"100.92.0.0/16", "10.242.0.0/16"}, PrivateIP:"172.17.0.5", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  4509. 2020-10-08T11:45:16.9676885Z [cluster2] I1008 11:36:25.061327 1 kubernetes.go:317] In SetCluster: &types.SubmarinerCluster{ID:"cluster2", Spec:v1.ClusterSpec{ClusterID:"cluster2", ColorCodes:[]string{"blue"}, ServiceCIDR:[]string{"100.92.0.0/16"}, ClusterCIDR:[]string{"10.242.0.0/16"}, GlobalCIDR:[]string{}}}
  4510. 2020-10-08T11:45:16.9679136Z [cluster2] I1008 11:36:25.063460 1 kubernetes.go:398] Endpoint "cluster2-submariner-cable-cluster2-172-17-0-5" matched what we received from datastore - not updating
  4511. 2020-10-08T11:45:16.9680893Z [cluster2] I1008 11:36:25.066348 1 kubernetes.go:345] Cluster "cluster2" matched what we received from datastore - not updating
  4512. 2020-10-08T11:45:16.9682313Z [cluster2] I1008 11:36:25.066459 1 datastoresyncer.go:271] The updated submariner Endpoint "cluster1" is not for this cluster - skipping updating the datastore
  4513. 2020-10-08T11:45:16.9683835Z [cluster2] I1008 11:36:25.066836 1 cableengine.go:128] Cable "submariner-cable-cluster1-172-17-0-9" is already installed - not installing again
  4514. 2020-10-08T11:45:16.9685569Z [cluster2] I1008 11:36:25.066912 1 tunnel.go:108] Tunnel controller successfully installed Endpoint cable submariner-cable-cluster1-172-17-0-9 in the engine
  4515. 2020-10-08T11:45:16.9686997Z [cluster2] I1008 11:36:26.097861 1 kubernetes.go:264] UpdateFunc in WatchEndpoints called
  4516. 2020-10-08T11:45:16.9689595Z [cluster2] I1008 11:36:26.097951 1 datastoresyncer.go:387] In reconcileEndpointCRD: &types.SubmarinerEndpoint{Spec:v1.EndpointSpec{ClusterID:"cluster1", CableName:"submariner-cable-cluster1-172-17-0-9", Hostname:"cluster1-worker", Subnets:[]string{"100.91.0.0/16", "10.241.0.0/16"}, PrivateIP:"172.17.0.9", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  4517. 2020-10-08T11:45:16.9692577Z [cluster2] I1008 11:36:26.100794 1 datastoresyncer.go:435] Endpoint "cluster1-submariner-cable-cluster1-172-17-0-9" matched what we received from datastore - not updating
  4518. 2020-10-08T11:45:16.9693903Z [cluster2] I1008 11:36:26.100841 1 kubernetes.go:264] UpdateFunc in WatchEndpoints called
  4519. 2020-10-08T11:45:16.9694613Z [cluster2] I1008 11:36:26.105268 1 kubernetes.go:180] UpdateFunc in WatchClusters called
  4520. 2020-10-08T11:45:16.9696108Z [cluster2] I1008 11:36:26.105301 1 datastoresyncer.go:312] In reconcileClusterCRD: &types.SubmarinerCluster{ID:"cluster1", Spec:v1.ClusterSpec{ClusterID:"cluster1", ColorCodes:[]string{"blue"}, ServiceCIDR:[]string{"100.91.0.0/16"}, ClusterCIDR:[]string{"10.241.0.0/16"}, GlobalCIDR:[]string{}}}
  4521. 2020-10-08T11:45:16.9697654Z [cluster2] I1008 11:36:26.107391 1 datastoresyncer.go:360] Cluster "cluster1" matched what we received from datastore - not updating
  4522. 2020-10-08T11:45:16.9698805Z [cluster2] I1008 11:36:26.107434 1 kubernetes.go:180] UpdateFunc in WatchClusters called
  4523. 2020-10-08T11:45:16.9699355Z [cluster2] 14[KNL] interface vethwepg4679fb2 deleted
  4524. 2020-10-08T11:45:16.9699841Z [cluster2] 12[KNL] interface vethwepl4679fb2 activated
  4525. 2020-10-08T11:45:16.9700399Z [cluster2] I1008 11:36:56.097981 1 kubernetes.go:264] UpdateFunc in WatchEndpoints called
  4526. 2020-10-08T11:45:16.9703517Z [cluster2] I1008 11:36:56.098018 1 datastoresyncer.go:387] In reconcileEndpointCRD: &types.SubmarinerEndpoint{Spec:v1.EndpointSpec{ClusterID:"cluster1", CableName:"submariner-cable-cluster1-172-17-0-9", Hostname:"cluster1-worker", Subnets:[]string{"100.91.0.0/16", "10.241.0.0/16"}, PrivateIP:"172.17.0.9", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  4527. 2020-10-08T11:45:16.9706697Z [cluster2] I1008 11:36:56.101820 1 datastoresyncer.go:435] Endpoint "cluster1-submariner-cable-cluster1-172-17-0-9" matched what we received from datastore - not updating
  4528. 2020-10-08T11:45:16.9708123Z [cluster2] I1008 11:36:56.101858 1 kubernetes.go:264] UpdateFunc in WatchEndpoints called
  4529. 2020-10-08T11:45:16.9708887Z [cluster2] I1008 11:36:56.105313 1 kubernetes.go:180] UpdateFunc in WatchClusters called
  4530. 2020-10-08T11:45:16.9710402Z [cluster2] I1008 11:36:56.105345 1 datastoresyncer.go:312] In reconcileClusterCRD: &types.SubmarinerCluster{ID:"cluster1", Spec:v1.ClusterSpec{ClusterID:"cluster1", ColorCodes:[]string{"blue"}, ServiceCIDR:[]string{"100.91.0.0/16"}, ClusterCIDR:[]string{"10.241.0.0/16"}, GlobalCIDR:[]string{}}}
  4531. 2020-10-08T11:45:16.9712529Z [cluster2] I1008 11:36:56.106988 1 datastoresyncer.go:360] Cluster "cluster1" matched what we received from datastore - not updating
  4532. 2020-10-08T11:45:16.9713427Z [cluster2] I1008 11:36:56.107011 1 kubernetes.go:180] UpdateFunc in WatchClusters called
  4533. 2020-10-08T11:45:16.9714129Z [cluster2] 06[KNL] interface vethwepl4679fb2 deactivated
  4534. 2020-10-08T11:45:16.9714746Z [cluster2] 10[KNL] interface vethwepl4679fb2 deleted
  4535. 2020-10-08T11:45:16.9715430Z [cluster2] 09[KNL] interface vethwepgc51ca28 deleted
  4536. 2020-10-08T11:45:16.9716199Z [cluster2] 11[KNL] interface vethweplc51ca28 activated
  4537. 2020-10-08T11:45:16.9716786Z [cluster2] 10[KNL] interface vethwepg3ea2c8e deleted
  4538. 2020-10-08T11:45:16.9717389Z [cluster2] 15[KNL] interface vethwepl3ea2c8e activated
  4539. 2020-10-08T11:45:16.9718023Z [cluster2] 14[KNL] interface vethweplc51ca28 deactivated
  4540. 2020-10-08T11:45:16.9719168Z [cluster2] 08[KNL] interface vethweplc51ca28 deleted
  4541. 2020-10-08T11:45:16.9719865Z [cluster2] 15[KNL] interface vethwepgac89716 deleted
  4542. 2020-10-08T11:45:16.9720422Z [cluster2] 09[KNL] interface vethweplac89716 activated
  4543. 2020-10-08T11:45:16.9727642Z [cluster2] I1008 11:37:25.061538 1 tunnel.go:95] Tunnel controller processing added or updated submariner Endpoint object: &v1.Endpoint{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"cluster2-submariner-cable-cluster2-172-17-0-5", GenerateName:"", Namespace:"submariner-operator", SelfLink:"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster2-submariner-cable-cluster2-172-17-0-5", UID:"6a117963-5044-4b34-9df6-57ebb8a4981d", ResourceVersion:"1284", Generation:1, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63737753726, loc:(*time.Location)(0x225ef80)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v1.EndpointSpec{ClusterID:"cluster2", CableName:"submariner-cable-cluster2-172-17-0-5", Hostname:"cluster2-worker", Subnets:[]string{"100.92.0.0/16", "10.242.0.0/16"}, PrivateIP:"172.17.0.5", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  4544. 2020-10-08T11:45:16.9732293Z [cluster2] I1008 11:37:25.061925 1 cableengine.go:92] Not installing cable for local cluster
  4545. 2020-10-08T11:45:16.9733686Z [cluster2] I1008 11:37:25.061937 1 tunnel.go:108] Tunnel controller successfully installed Endpoint cable submariner-cable-cluster2-172-17-0-5 in the engine
  4546. 2020-10-08T11:45:16.9739720Z [cluster2] I1008 11:37:25.061945 1 tunnel.go:95] Tunnel controller processing added or updated submariner Endpoint object: &v1.Endpoint{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"cluster1-submariner-cable-cluster1-172-17-0-9", GenerateName:"", Namespace:"submariner-operator", SelfLink:"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster1-submariner-cable-cluster1-172-17-0-9", UID:"60d04732-7b08-4685-b72b-cb2c12702817", ResourceVersion:"1285", Generation:1, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63737753726, loc:(*time.Location)(0x225ef80)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v1.EndpointSpec{ClusterID:"cluster1", CableName:"submariner-cable-cluster1-172-17-0-9", Hostname:"cluster1-worker", Subnets:[]string{"100.91.0.0/16", "10.241.0.0/16"}, PrivateIP:"172.17.0.9", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  4547. 2020-10-08T11:45:16.9745123Z [cluster2] I1008 11:37:25.062985 1 cableengine.go:101] Installing Endpoint cable "submariner-cable-cluster1-172-17-0-9"
  4548. 2020-10-08T11:45:16.9746599Z [cluster2] I1008 11:37:25.062656 1 kubernetes.go:317] In SetCluster: &types.SubmarinerCluster{ID:"cluster2", Spec:v1.ClusterSpec{ClusterID:"cluster2", ColorCodes:[]string{"blue"}, ServiceCIDR:[]string{"100.92.0.0/16"}, ClusterCIDR:[]string{"10.242.0.0/16"}, GlobalCIDR:[]string{}}}
  4549. 2020-10-08T11:45:16.9752765Z [cluster2] I1008 11:37:25.067189 1 datastoresyncer.go:287] Processing local submariner Endpoint object: &v1.Endpoint{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"cluster2-submariner-cable-cluster2-172-17-0-5", GenerateName:"", Namespace:"submariner-operator", SelfLink:"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster2-submariner-cable-cluster2-172-17-0-5", UID:"6a117963-5044-4b34-9df6-57ebb8a4981d", ResourceVersion:"1284", Generation:1, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63737753726, loc:(*time.Location)(0x225ef80)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v1.EndpointSpec{ClusterID:"cluster2", CableName:"submariner-cable-cluster2-172-17-0-5", Hostname:"cluster2-worker", Subnets:[]string{"100.92.0.0/16", "10.242.0.0/16"}, PrivateIP:"172.17.0.5", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  4550. 2020-10-08T11:45:16.9761069Z [cluster2] I1008 11:37:25.068334 1 kubernetes.go:370] In SetEndpoint: &types.SubmarinerEndpoint{Spec:v1.EndpointSpec{ClusterID:"cluster2", CableName:"submariner-cable-cluster2-172-17-0-5", Hostname:"cluster2-worker", Subnets:[]string{"100.92.0.0/16", "10.242.0.0/16"}, PrivateIP:"172.17.0.5", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  4551. 2020-10-08T11:45:16.9763654Z [cluster2] I1008 11:37:25.067711 1 kubernetes.go:345] Cluster "cluster2" matched what we received from datastore - not updating
  4552. 2020-10-08T11:45:16.9768313Z [cluster2] I1008 11:37:25.075715 1 kubernetes.go:398] Endpoint "cluster2-submariner-cable-cluster2-172-17-0-5" matched what we received from datastore - not updating
  4553. 2020-10-08T11:45:16.9770486Z [cluster2] I1008 11:37:25.077444 1 datastoresyncer.go:271] The updated submariner Endpoint "cluster1" is not for this cluster - skipping updating the datastore
  4554. 2020-10-08T11:45:16.9772272Z [cluster2] I1008 11:37:25.086209 1 cableengine.go:128] Cable "submariner-cable-cluster1-172-17-0-9" is already installed - not installing again
  4555. 2020-10-08T11:45:16.9774481Z [cluster2] I1008 11:37:25.086227 1 tunnel.go:108] Tunnel controller successfully installed Endpoint cable submariner-cable-cluster1-172-17-0-9 in the engine
  4556. 2020-10-08T11:45:16.9778053Z [cluster2] I1008 11:37:26.098095 1 kubernetes.go:264] UpdateFunc in WatchEndpoints called
  4557. 2020-10-08T11:45:16.9781092Z [cluster2] I1008 11:37:26.098119 1 datastoresyncer.go:387] In reconcileEndpointCRD: &types.SubmarinerEndpoint{Spec:v1.EndpointSpec{ClusterID:"cluster1", CableName:"submariner-cable-cluster1-172-17-0-9", Hostname:"cluster1-worker", Subnets:[]string{"100.91.0.0/16", "10.241.0.0/16"}, PrivateIP:"172.17.0.9", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  4558. 2020-10-08T11:45:16.9787133Z [cluster2] I1008 11:37:26.099800 1 datastoresyncer.go:435] Endpoint "cluster1-submariner-cable-cluster1-172-17-0-9" matched what we received from datastore - not updating
  4559. 2020-10-08T11:45:16.9788689Z [cluster2] I1008 11:37:26.099826 1 kubernetes.go:264] UpdateFunc in WatchEndpoints called
  4560. 2020-10-08T11:45:16.9789525Z [cluster2] I1008 11:37:26.105605 1 kubernetes.go:180] UpdateFunc in WatchClusters called
  4561. 2020-10-08T11:45:16.9791222Z [cluster2] I1008 11:37:26.105639 1 datastoresyncer.go:312] In reconcileClusterCRD: &types.SubmarinerCluster{ID:"cluster1", Spec:v1.ClusterSpec{ClusterID:"cluster1", ColorCodes:[]string{"blue"}, ServiceCIDR:[]string{"100.91.0.0/16"}, ClusterCIDR:[]string{"10.241.0.0/16"}, GlobalCIDR:[]string{}}}
  4562. 2020-10-08T11:45:16.9794119Z [cluster2] I1008 11:37:26.107486 1 datastoresyncer.go:360] Cluster "cluster1" matched what we received from datastore - not updating
  4563. 2020-10-08T11:45:16.9798132Z [cluster2] I1008 11:37:26.107505 1 kubernetes.go:180] UpdateFunc in WatchClusters called
  4564. 2020-10-08T11:45:16.9798916Z [cluster2] 08[KNL] interface vethwepl3ea2c8e deactivated
  4565. 2020-10-08T11:45:16.9799498Z [cluster2] 14[KNL] interface vethwepl3ea2c8e deleted
  4566. 2020-10-08T11:45:16.9800053Z [cluster2] 09[KNL] interface vethwepg60c98c8 deleted
  4567. 2020-10-08T11:45:16.9800607Z [cluster2] 01[KNL] interface vethwepl60c98c8 activated
  4568. 2020-10-08T11:45:16.9801769Z [cluster2] 01[KNL] interface vethweplac89716 deactivated
  4569. 2020-10-08T11:45:16.9802487Z [cluster2] 12[KNL] interface vethweplac89716 deleted
  4570. 2020-10-08T11:45:16.9803112Z [cluster2] I1008 11:37:56.098223 1 kubernetes.go:264] UpdateFunc in WatchEndpoints called
  4571. 2020-10-08T11:45:16.9806013Z [cluster2] I1008 11:37:56.098264 1 datastoresyncer.go:387] In reconcileEndpointCRD: &types.SubmarinerEndpoint{Spec:v1.EndpointSpec{ClusterID:"cluster1", CableName:"submariner-cable-cluster1-172-17-0-9", Hostname:"cluster1-worker", Subnets:[]string{"100.91.0.0/16", "10.241.0.0/16"}, PrivateIP:"172.17.0.9", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  4572. 2020-10-08T11:45:16.9809115Z [cluster2] I1008 11:37:56.101372 1 datastoresyncer.go:435] Endpoint "cluster1-submariner-cable-cluster1-172-17-0-9" matched what we received from datastore - not updating
  4573. 2020-10-08T11:45:16.9810432Z [cluster2] I1008 11:37:56.101404 1 kubernetes.go:264] UpdateFunc in WatchEndpoints called
  4574. 2020-10-08T11:45:16.9811143Z [cluster2] I1008 11:37:56.105695 1 kubernetes.go:180] UpdateFunc in WatchClusters called
  4575. 2020-10-08T11:45:16.9812687Z [cluster2] I1008 11:37:56.105710 1 datastoresyncer.go:312] In reconcileClusterCRD: &types.SubmarinerCluster{ID:"cluster1", Spec:v1.ClusterSpec{ClusterID:"cluster1", ColorCodes:[]string{"blue"}, ServiceCIDR:[]string{"100.91.0.0/16"}, ClusterCIDR:[]string{"10.241.0.0/16"}, GlobalCIDR:[]string{}}}
  4576. 2020-10-08T11:45:16.9814735Z [cluster2] I1008 11:37:56.107195 1 datastoresyncer.go:360] Cluster "cluster1" matched what we received from datastore - not updating
  4577. 2020-10-08T11:45:16.9815584Z [cluster2] I1008 11:37:56.107216 1 kubernetes.go:180] UpdateFunc in WatchClusters called
  4578. 2020-10-08T11:45:16.9816209Z [cluster2] 01[KNL] interface vethwepg250815d deleted
  4579. 2020-10-08T11:45:16.9816741Z [cluster2] 05[KNL] interface vethwepl250815d activated
  4580. 2020-10-08T11:45:16.9817855Z [cluster2] 05[KNL] interface vethwepl60c98c8 deactivated
  4581. 2020-10-08T11:45:16.9818437Z [cluster2] 10[KNL] interface vethwepl60c98c8 deleted
  4582. 2020-10-08T11:45:16.9818980Z [cluster2] 12[KNL] interface vethwepg5a07cf7 deleted
  4583. 2020-10-08T11:45:16.9819535Z [cluster2] 09[KNL] interface vethwepl5a07cf7 activated
  4584. 2020-10-08T11:45:16.9820116Z [cluster2] 14[KNL] interface vethwepl5a07cf7 deactivated
  4585. 2020-10-08T11:45:16.9820683Z [cluster2] 11[KNL] interface vethwepl5a07cf7 deleted
  4586. 2020-10-08T11:45:16.9821891Z [cluster2] I1008 11:38:25.061563 1 kubernetes.go:317] In SetCluster: &types.SubmarinerCluster{ID:"cluster2", Spec:v1.ClusterSpec{ClusterID:"cluster2", ColorCodes:[]string{"blue"}, ServiceCIDR:[]string{"100.92.0.0/16"}, ClusterCIDR:[]string{"10.242.0.0/16"}, GlobalCIDR:[]string{}}}
  4587. 2020-10-08T11:45:16.9829858Z [cluster2] I1008 11:38:25.061548 1 tunnel.go:95] Tunnel controller processing added or updated submariner Endpoint object: &v1.Endpoint{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"cluster2-submariner-cable-cluster2-172-17-0-5", GenerateName:"", Namespace:"submariner-operator", SelfLink:"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster2-submariner-cable-cluster2-172-17-0-5", UID:"6a117963-5044-4b34-9df6-57ebb8a4981d", ResourceVersion:"1284", Generation:1, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63737753726, loc:(*time.Location)(0x225ef80)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v1.EndpointSpec{ClusterID:"cluster2", CableName:"submariner-cable-cluster2-172-17-0-5", Hostname:"cluster2-worker", Subnets:[]string{"100.92.0.0/16", "10.242.0.0/16"}, PrivateIP:"172.17.0.5", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  4588. 2020-10-08T11:45:16.9843724Z [cluster2] I1008 11:38:25.061821 1 cableengine.go:92] Not installing cable for local cluster
  4589. 2020-10-08T11:45:16.9845731Z [cluster2] I1008 11:38:25.061838 1 tunnel.go:108] Tunnel controller successfully installed Endpoint cable submariner-cable-cluster2-172-17-0-5 in the engine
  4590. 2020-10-08T11:45:16.9852900Z [cluster2] I1008 11:38:25.061852 1 tunnel.go:95] Tunnel controller processing added or updated submariner Endpoint object: &v1.Endpoint{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"cluster1-submariner-cable-cluster1-172-17-0-9", GenerateName:"", Namespace:"submariner-operator", SelfLink:"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster1-submariner-cable-cluster1-172-17-0-9", UID:"60d04732-7b08-4685-b72b-cb2c12702817", ResourceVersion:"1285", Generation:1, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63737753726, loc:(*time.Location)(0x225ef80)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v1.EndpointSpec{ClusterID:"cluster1", CableName:"submariner-cable-cluster1-172-17-0-9", Hostname:"cluster1-worker", Subnets:[]string{"100.91.0.0/16", "10.241.0.0/16"}, PrivateIP:"172.17.0.9", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  4591. 2020-10-08T11:45:16.9861419Z [cluster2] I1008 11:38:25.061927 1 cableengine.go:101] Installing Endpoint cable "submariner-cable-cluster1-172-17-0-9"
  4592. 2020-10-08T11:45:16.9863196Z [cluster2] I1008 11:38:25.064803 1 cableengine.go:128] Cable "submariner-cable-cluster1-172-17-0-9" is already installed - not installing again
  4593. 2020-10-08T11:45:16.9883569Z [cluster2] I1008 11:38:25.064821 1 tunnel.go:108] Tunnel controller successfully installed Endpoint cable submariner-cable-cluster1-172-17-0-9 in the engine
  4594. 2020-10-08T11:45:16.9886344Z [cluster2] I1008 11:38:25.066702 1 kubernetes.go:345] Cluster "cluster2" matched what we received from datastore - not updating
  4595. 2020-10-08T11:45:16.9892698Z [cluster2] I1008 11:38:25.067593 1 datastoresyncer.go:287] Processing local submariner Endpoint object: &v1.Endpoint{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"cluster2-submariner-cable-cluster2-172-17-0-5", GenerateName:"", Namespace:"submariner-operator", SelfLink:"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster2-submariner-cable-cluster2-172-17-0-5", UID:"6a117963-5044-4b34-9df6-57ebb8a4981d", ResourceVersion:"1284", Generation:1, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63737753726, loc:(*time.Location)(0x225ef80)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v1.EndpointSpec{ClusterID:"cluster2", CableName:"submariner-cable-cluster2-172-17-0-5", Hostname:"cluster2-worker", Subnets:[]string{"100.92.0.0/16", "10.242.0.0/16"}, PrivateIP:"172.17.0.5", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  4596. 2020-10-08T11:45:16.9902147Z [cluster2] I1008 11:38:25.067642 1 kubernetes.go:370] In SetEndpoint: &types.SubmarinerEndpoint{Spec:v1.EndpointSpec{ClusterID:"cluster2", CableName:"submariner-cable-cluster2-172-17-0-5", Hostname:"cluster2-worker", Subnets:[]string{"100.92.0.0/16", "10.242.0.0/16"}, PrivateIP:"172.17.0.5", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  4597. 2020-10-08T11:45:16.9907231Z [cluster2] I1008 11:38:25.069831 1 kubernetes.go:398] Endpoint "cluster2-submariner-cable-cluster2-172-17-0-5" matched what we received from datastore - not updating
  4598. 2020-10-08T11:45:16.9909212Z [cluster2] I1008 11:38:25.071530 1 datastoresyncer.go:271] The updated submariner Endpoint "cluster1" is not for this cluster - skipping updating the datastore
  4599. 2020-10-08T11:45:16.9910742Z [cluster2] I1008 11:38:26.098459 1 kubernetes.go:264] UpdateFunc in WatchEndpoints called
  4600. 2020-10-08T11:45:16.9912955Z [cluster2] I1008 11:38:26.098537 1 datastoresyncer.go:387] In reconcileEndpointCRD: &types.SubmarinerEndpoint{Spec:v1.EndpointSpec{ClusterID:"cluster1", CableName:"submariner-cable-cluster1-172-17-0-9", Hostname:"cluster1-worker", Subnets:[]string{"100.91.0.0/16", "10.241.0.0/16"}, PrivateIP:"172.17.0.9", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  4601. 2020-10-08T11:45:16.9915770Z [cluster2] I1008 11:38:26.101181 1 datastoresyncer.go:435] Endpoint "cluster1-submariner-cable-cluster1-172-17-0-9" matched what we received from datastore - not updating
  4602. 2020-10-08T11:45:16.9917399Z [cluster2] I1008 11:38:26.101208 1 kubernetes.go:264] UpdateFunc in WatchEndpoints called
  4603. 2020-10-08T11:45:16.9918117Z [cluster2] I1008 11:38:26.105774 1 kubernetes.go:180] UpdateFunc in WatchClusters called
  4604. 2020-10-08T11:45:16.9919523Z [cluster2] I1008 11:38:26.105790 1 datastoresyncer.go:312] In reconcileClusterCRD: &types.SubmarinerCluster{ID:"cluster1", Spec:v1.ClusterSpec{ClusterID:"cluster1", ColorCodes:[]string{"blue"}, ServiceCIDR:[]string{"100.91.0.0/16"}, ClusterCIDR:[]string{"10.241.0.0/16"}, GlobalCIDR:[]string{}}}
  4605. 2020-10-08T11:45:16.9921323Z [cluster2] I1008 11:38:26.107447 1 datastoresyncer.go:360] Cluster "cluster1" matched what we received from datastore - not updating
  4606. 2020-10-08T11:45:16.9922153Z [cluster2] I1008 11:38:26.107474 1 kubernetes.go:180] UpdateFunc in WatchClusters called
  4607. 2020-10-08T11:45:16.9922786Z [cluster2] 12[KNL] interface vethwepga738e5b deleted
  4608. 2020-10-08T11:45:16.9923346Z [cluster2] 10[KNL] interface vethwepla738e5b activated
  4609. 2020-10-08T11:45:16.9923903Z [cluster2] 16[KNL] interface vethwepg5be3553 deleted
  4610. 2020-10-08T11:45:16.9924462Z [cluster2] 06[KNL] interface vethwepl5be3553 activated
  4611. 2020-10-08T11:45:16.9925664Z [cluster2] 14[KNL] interface vethwepl250815d deactivated
  4612. 2020-10-08T11:45:16.9926209Z [cluster2] 07[KNL] interface vethwepl250815d deleted
  4613. 2020-10-08T11:45:16.9926772Z [cluster2] 05[KNL] interface vethwepla738e5b deactivated
  4614. 2020-10-08T11:45:16.9927539Z [cluster2] 14[KNL] interface vethwepla738e5b deleted
  4615. 2020-10-08T11:45:16.9928118Z [cluster2] 10[KNL] interface vethwepl5be3553 deactivated
  4616. 2020-10-08T11:45:16.9928694Z [cluster2] 08[KNL] interface vethwepl5be3553 deleted
  4617. 2020-10-08T11:45:16.9929369Z [cluster2] I1008 11:38:56.098591 1 kubernetes.go:264] UpdateFunc in WatchEndpoints called
  4618. 2020-10-08T11:45:16.9932089Z [cluster2] I1008 11:38:56.098630 1 datastoresyncer.go:387] In reconcileEndpointCRD: &types.SubmarinerEndpoint{Spec:v1.EndpointSpec{ClusterID:"cluster1", CableName:"submariner-cable-cluster1-172-17-0-9", Hostname:"cluster1-worker", Subnets:[]string{"100.91.0.0/16", "10.241.0.0/16"}, PrivateIP:"172.17.0.9", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  4619. 2020-10-08T11:45:16.9935649Z [cluster2] I1008 11:38:56.101092 1 datastoresyncer.go:435] Endpoint "cluster1-submariner-cable-cluster1-172-17-0-9" matched what we received from datastore - not updating
  4620. 2020-10-08T11:45:16.9936818Z [cluster2] I1008 11:38:56.101133 1 kubernetes.go:264] UpdateFunc in WatchEndpoints called
  4621. 2020-10-08T11:45:16.9937438Z [cluster2] I1008 11:38:56.105904 1 kubernetes.go:180] UpdateFunc in WatchClusters called
  4622. 2020-10-08T11:45:16.9938974Z [cluster2] I1008 11:38:56.105946 1 datastoresyncer.go:312] In reconcileClusterCRD: &types.SubmarinerCluster{ID:"cluster1", Spec:v1.ClusterSpec{ClusterID:"cluster1", ColorCodes:[]string{"blue"}, ServiceCIDR:[]string{"100.91.0.0/16"}, ClusterCIDR:[]string{"10.241.0.0/16"}, GlobalCIDR:[]string{}}}
  4623. 2020-10-08T11:45:16.9940580Z [cluster2] I1008 11:38:56.107989 1 datastoresyncer.go:360] Cluster "cluster1" matched what we received from datastore - not updating
  4624. 2020-10-08T11:45:16.9941308Z [cluster2] I1008 11:38:56.108015 1 kubernetes.go:180] UpdateFunc in WatchClusters called
  4625. 2020-10-08T11:45:16.9942248Z [cluster2] 01[KNL] interface vethwepgdaff059 deleted
  4626. 2020-10-08T11:45:16.9944276Z [cluster2] 06[KNL] interface vethwepldaff059 activated
  4627. 2020-10-08T11:45:16.9948883Z [cluster2] 01[KNL] interface vethwepgf65a7f2 deleted
  4628. 2020-10-08T11:45:16.9949393Z [cluster2] 06[KNL] interface vethweplf65a7f2 activated
  4629. 2020-10-08T11:45:16.9950758Z [cluster2] 14[KNL] interface vethwepldaff059 deactivated
  4630. 2020-10-08T11:45:16.9951502Z [cluster2] 10[KNL] interface vethwepldaff059 deleted
  4631. 2020-10-08T11:45:16.9952697Z [cluster2] I1008 11:39:25.061758 1 kubernetes.go:317] In SetCluster: &types.SubmarinerCluster{ID:"cluster2", Spec:v1.ClusterSpec{ClusterID:"cluster2", ColorCodes:[]string{"blue"}, ServiceCIDR:[]string{"100.92.0.0/16"}, ClusterCIDR:[]string{"10.242.0.0/16"}, GlobalCIDR:[]string{}}}
  4632. 2020-10-08T11:45:16.9960130Z [cluster2] I1008 11:39:25.061759 1 tunnel.go:95] Tunnel controller processing added or updated submariner Endpoint object: &v1.Endpoint{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"cluster2-submariner-cable-cluster2-172-17-0-5", GenerateName:"", Namespace:"submariner-operator", SelfLink:"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster2-submariner-cable-cluster2-172-17-0-5", UID:"6a117963-5044-4b34-9df6-57ebb8a4981d", ResourceVersion:"1284", Generation:1, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63737753726, loc:(*time.Location)(0x225ef80)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v1.EndpointSpec{ClusterID:"cluster2", CableName:"submariner-cable-cluster2-172-17-0-5", Hostname:"cluster2-worker", Subnets:[]string{"100.92.0.0/16", "10.242.0.0/16"}, PrivateIP:"172.17.0.5", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  4633. 2020-10-08T11:45:16.9965614Z [cluster2] I1008 11:39:25.062016 1 cableengine.go:92] Not installing cable for local cluster
  4634. 2020-10-08T11:45:16.9967039Z [cluster2] I1008 11:39:25.062032 1 tunnel.go:108] Tunnel controller successfully installed Endpoint cable submariner-cable-cluster2-172-17-0-5 in the engine
  4635. 2020-10-08T11:45:16.9973900Z [cluster2] I1008 11:39:25.062048 1 tunnel.go:95] Tunnel controller processing added or updated submariner Endpoint object: &v1.Endpoint{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"cluster1-submariner-cable-cluster1-172-17-0-9", GenerateName:"", Namespace:"submariner-operator", SelfLink:"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster1-submariner-cable-cluster1-172-17-0-9", UID:"60d04732-7b08-4685-b72b-cb2c12702817", ResourceVersion:"1285", Generation:1, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63737753726, loc:(*time.Location)(0x225ef80)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v1.EndpointSpec{ClusterID:"cluster1", CableName:"submariner-cable-cluster1-172-17-0-9", Hostname:"cluster1-worker", Subnets:[]string{"100.91.0.0/16", "10.241.0.0/16"}, PrivateIP:"172.17.0.9", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  4636. 2020-10-08T11:45:16.9979935Z [cluster2] I1008 11:39:25.062104 1 cableengine.go:101] Installing Endpoint cable "submariner-cable-cluster1-172-17-0-9"
  4637. 2020-10-08T11:45:16.9981359Z [cluster2] I1008 11:39:25.064560 1 cableengine.go:128] Cable "submariner-cable-cluster1-172-17-0-9" is already installed - not installing again
  4638. 2020-10-08T11:45:16.9982871Z [cluster2] I1008 11:39:25.064577 1 tunnel.go:108] Tunnel controller successfully installed Endpoint cable submariner-cable-cluster1-172-17-0-9 in the engine
  4639. 2020-10-08T11:45:16.9984105Z [cluster2] I1008 11:39:25.066480 1 kubernetes.go:345] Cluster "cluster2" matched what we received from datastore - not updating
  4640. 2020-10-08T11:45:16.9990948Z [cluster2] I1008 11:39:25.066608 1 datastoresyncer.go:287] Processing local submariner Endpoint object: &v1.Endpoint{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"cluster2-submariner-cable-cluster2-172-17-0-5", GenerateName:"", Namespace:"submariner-operator", SelfLink:"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster2-submariner-cable-cluster2-172-17-0-5", UID:"6a117963-5044-4b34-9df6-57ebb8a4981d", ResourceVersion:"1284", Generation:1, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63737753726, loc:(*time.Location)(0x225ef80)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v1.EndpointSpec{ClusterID:"cluster2", CableName:"submariner-cable-cluster2-172-17-0-5", Hostname:"cluster2-worker", Subnets:[]string{"100.92.0.0/16", "10.242.0.0/16"}, PrivateIP:"172.17.0.5", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  4641. 2020-10-08T11:45:16.9998750Z [cluster2] I1008 11:39:25.066656 1 kubernetes.go:370] In SetEndpoint: &types.SubmarinerEndpoint{Spec:v1.EndpointSpec{ClusterID:"cluster2", CableName:"submariner-cable-cluster2-172-17-0-5", Hostname:"cluster2-worker", Subnets:[]string{"100.92.0.0/16", "10.242.0.0/16"}, PrivateIP:"172.17.0.5", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  4642. 2020-10-08T11:45:17.0002009Z [cluster2] I1008 11:39:25.069149 1 kubernetes.go:398] Endpoint "cluster2-submariner-cable-cluster2-172-17-0-5" matched what we received from datastore - not updating
  4643. 2020-10-08T11:45:17.0004288Z [cluster2] I1008 11:39:25.071072 1 datastoresyncer.go:271] The updated submariner Endpoint "cluster1" is not for this cluster - skipping updating the datastore
  4644. 2020-10-08T11:45:17.0006913Z [cluster2] I1008 11:39:26.098712 1 kubernetes.go:264] UpdateFunc in WatchEndpoints called
  4645. 2020-10-08T11:45:17.0009491Z [cluster2] I1008 11:39:26.098734 1 datastoresyncer.go:387] In reconcileEndpointCRD: &types.SubmarinerEndpoint{Spec:v1.EndpointSpec{ClusterID:"cluster1", CableName:"submariner-cable-cluster1-172-17-0-9", Hostname:"cluster1-worker", Subnets:[]string{"100.91.0.0/16", "10.241.0.0/16"}, PrivateIP:"172.17.0.9", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  4646. 2020-10-08T11:45:17.0012022Z [cluster2] I1008 11:39:26.101152 1 datastoresyncer.go:435] Endpoint "cluster1-submariner-cable-cluster1-172-17-0-9" matched what we received from datastore - not updating
  4647. 2020-10-08T11:45:17.0013357Z [cluster2] I1008 11:39:26.101175 1 kubernetes.go:264] UpdateFunc in WatchEndpoints called
  4648. 2020-10-08T11:45:17.0013981Z [cluster2] I1008 11:39:26.106026 1 kubernetes.go:180] UpdateFunc in WatchClusters called
  4649. 2020-10-08T11:45:17.0016390Z [cluster2] I1008 11:39:26.106062 1 kubernetes.go:180] UpdateFunc in WatchClusters called
  4650. 2020-10-08T11:45:17.0019945Z [cluster2] I1008 11:39:26.106076 1 datastoresyncer.go:312] In reconcileClusterCRD: &types.SubmarinerCluster{ID:"cluster1", Spec:v1.ClusterSpec{ClusterID:"cluster1", ColorCodes:[]string{"blue"}, ServiceCIDR:[]string{"100.91.0.0/16"}, ClusterCIDR:[]string{"10.241.0.0/16"}, GlobalCIDR:[]string{}}}
  4651. 2020-10-08T11:45:17.0022037Z [cluster2] I1008 11:39:26.107723 1 datastoresyncer.go:360] Cluster "cluster1" matched what we received from datastore - not updating
  4652. 2020-10-08T11:45:17.0022945Z [cluster2] I1008 11:39:56.098928 1 kubernetes.go:264] UpdateFunc in WatchEndpoints called
  4653. 2020-10-08T11:45:17.0027054Z [cluster2] I1008 11:39:56.098953 1 datastoresyncer.go:387] In reconcileEndpointCRD: &types.SubmarinerEndpoint{Spec:v1.EndpointSpec{ClusterID:"cluster1", CableName:"submariner-cable-cluster1-172-17-0-9", Hostname:"cluster1-worker", Subnets:[]string{"100.91.0.0/16", "10.241.0.0/16"}, PrivateIP:"172.17.0.9", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  4654. 2020-10-08T11:45:17.0030382Z [cluster2] I1008 11:39:56.102255 1 datastoresyncer.go:435] Endpoint "cluster1-submariner-cable-cluster1-172-17-0-9" matched what we received from datastore - not updating
  4655. 2020-10-08T11:45:17.0031833Z [cluster2] I1008 11:39:56.102285 1 kubernetes.go:264] UpdateFunc in WatchEndpoints called
  4656. 2020-10-08T11:45:17.0032546Z [cluster2] I1008 11:39:56.106155 1 kubernetes.go:180] UpdateFunc in WatchClusters called
  4657. 2020-10-08T11:45:17.0033951Z [cluster2] I1008 11:39:56.106183 1 datastoresyncer.go:312] In reconcileClusterCRD: &types.SubmarinerCluster{ID:"cluster1", Spec:v1.ClusterSpec{ClusterID:"cluster1", ColorCodes:[]string{"blue"}, ServiceCIDR:[]string{"100.91.0.0/16"}, ClusterCIDR:[]string{"10.241.0.0/16"}, GlobalCIDR:[]string{}}}
  4658. 2020-10-08T11:45:17.0036390Z [cluster2] I1008 11:39:56.107814 1 datastoresyncer.go:360] Cluster "cluster1" matched what we received from datastore - not updating
  4659. 2020-10-08T11:45:17.0037220Z [cluster2] I1008 11:39:56.107834 1 kubernetes.go:180] UpdateFunc in WatchClusters called
  4660. 2020-10-08T11:45:17.0038899Z [lighthouse]$ [cluster2] kubectl get pods --selector=app=submariner-routeagent -n submariner-operator -o jsonpath={.items[*].metadata.name}
  4661. 2020-10-08T11:45:17.0041128Z [lighthouse]$ [cluster2] kubectl get pods --selector=app=submariner-routeagent -n submariner-operator -o jsonpath={.items[*].metadata.name}
  4662. 2020-10-08T11:45:17.0043392Z [lighthouse]$ [cluster2] command kubectl --context=cluster2 get pods --selector=app=submariner-routeagent -n submariner-operator -o jsonpath={.items[*].metadata.name}
  4663. 2020-10-08T11:45:17.0046129Z [lighthouse]$ [cluster2] kubectl --context=cluster2 get pods --selector=app=submariner-routeagent -n submariner-operator -o jsonpath={.items[*].metadata.name}
  4664. 2020-10-08T11:45:17.0053137Z [cluster2] I1008 11:40:25.061933 1 tunnel.go:95] Tunnel controller processing added or updated submariner Endpoint object: &v1.Endpoint{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"cluster2-submariner-cable-cluster2-172-17-0-5", GenerateName:"", Namespace:"submariner-operator", SelfLink:"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster2-submariner-cable-cluster2-172-17-0-5", UID:"6a117963-5044-4b34-9df6-57ebb8a4981d", ResourceVersion:"1284", Generation:1, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63737753726, loc:(*time.Location)(0x225ef80)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v1.EndpointSpec{ClusterID:"cluster2", CableName:"submariner-cable-cluster2-172-17-0-5", Hostname:"cluster2-worker", Subnets:[]string{"100.92.0.0/16", "10.242.0.0/16"}, PrivateIP:"172.17.0.5", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  4665. 2020-10-08T11:45:17.0057869Z [cluster2] I1008 11:40:25.062037 1 cableengine.go:92] Not installing cable for local cluster
  4666. 2020-10-08T11:45:17.0059238Z [cluster2] I1008 11:40:25.062105 1 tunnel.go:108] Tunnel controller successfully installed Endpoint cable submariner-cable-cluster2-172-17-0-5 in the engine
  4667. 2020-10-08T11:45:17.0065910Z [cluster2] I1008 11:40:25.062120 1 tunnel.go:95] Tunnel controller processing added or updated submariner Endpoint object: &v1.Endpoint{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"cluster1-submariner-cable-cluster1-172-17-0-9", GenerateName:"", Namespace:"submariner-operator", SelfLink:"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster1-submariner-cable-cluster1-172-17-0-9", UID:"60d04732-7b08-4685-b72b-cb2c12702817", ResourceVersion:"1285", Generation:1, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63737753726, loc:(*time.Location)(0x225ef80)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v1.EndpointSpec{ClusterID:"cluster1", CableName:"submariner-cable-cluster1-172-17-0-9", Hostname:"cluster1-worker", Subnets:[]string{"100.91.0.0/16", "10.241.0.0/16"}, PrivateIP:"172.17.0.9", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  4668. 2020-10-08T11:45:17.0071177Z [cluster2] I1008 11:40:25.062216 1 cableengine.go:101] Installing Endpoint cable "submariner-cable-cluster1-172-17-0-9"
  4669. 2020-10-08T11:45:17.0073311Z [cluster2] I1008 11:40:25.062242 1 kubernetes.go:317] In SetCluster: &types.SubmarinerCluster{ID:"cluster2", Spec:v1.ClusterSpec{ClusterID:"cluster2", ColorCodes:[]string{"blue"}, ServiceCIDR:[]string{"100.92.0.0/16"}, ClusterCIDR:[]string{"10.242.0.0/16"}, GlobalCIDR:[]string{}}}
  4670. 2020-10-08T11:45:17.0081041Z [cluster2] I1008 11:40:25.064806 1 datastoresyncer.go:287] Processing local submariner Endpoint object: &v1.Endpoint{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"cluster2-submariner-cable-cluster2-172-17-0-5", GenerateName:"", Namespace:"submariner-operator", SelfLink:"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster2-submariner-cable-cluster2-172-17-0-5", UID:"6a117963-5044-4b34-9df6-57ebb8a4981d", ResourceVersion:"1284", Generation:1, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63737753726, loc:(*time.Location)(0x225ef80)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v1.EndpointSpec{ClusterID:"cluster2", CableName:"submariner-cable-cluster2-172-17-0-5", Hostname:"cluster2-worker", Subnets:[]string{"100.92.0.0/16", "10.242.0.0/16"}, PrivateIP:"172.17.0.5", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  4671. 2020-10-08T11:45:17.0087544Z [cluster2] I1008 11:40:25.065078 1 kubernetes.go:370] In SetEndpoint: &types.SubmarinerEndpoint{Spec:v1.EndpointSpec{ClusterID:"cluster2", CableName:"submariner-cable-cluster2-172-17-0-5", Hostname:"cluster2-worker", Subnets:[]string{"100.92.0.0/16", "10.242.0.0/16"}, PrivateIP:"172.17.0.5", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  4672. 2020-10-08T11:45:17.0090231Z [cluster2] I1008 11:40:25.066748 1 kubernetes.go:398] Endpoint "cluster2-submariner-cable-cluster2-172-17-0-5" matched what we received from datastore - not updating
  4673. 2020-10-08T11:45:17.0091604Z [cluster2] I1008 11:40:25.067134 1 kubernetes.go:345] Cluster "cluster2" matched what we received from datastore - not updating
  4674. 2020-10-08T11:45:17.0092904Z [cluster2] I1008 11:40:25.068944 1 cableengine.go:128] Cable "submariner-cable-cluster1-172-17-0-9" is already installed - not installing again
  4675. 2020-10-08T11:45:17.0095037Z [cluster2] I1008 11:40:25.068963 1 tunnel.go:108] Tunnel controller successfully installed Endpoint cable submariner-cable-cluster1-172-17-0-9 in the engine
  4676. 2020-10-08T11:45:17.0096498Z [cluster2] I1008 11:40:25.070269 1 datastoresyncer.go:271] The updated submariner Endpoint "cluster1" is not for this cluster - skipping updating the datastore
  4677. 2020-10-08T11:45:17.0097301Z [cluster2] I1008 11:40:26.099052 1 kubernetes.go:264] UpdateFunc in WatchEndpoints called
  4678. 2020-10-08T11:45:17.0097921Z [cluster2] I1008 11:40:26.099097 1 kubernetes.go:264] UpdateFunc in WatchEndpoints called
  4679. 2020-10-08T11:45:17.0100080Z [cluster2] I1008 11:40:26.099111 1 datastoresyncer.go:387] In reconcileEndpointCRD: &types.SubmarinerEndpoint{Spec:v1.EndpointSpec{ClusterID:"cluster1", CableName:"submariner-cable-cluster1-172-17-0-9", Hostname:"cluster1-worker", Subnets:[]string{"100.91.0.0/16", "10.241.0.0/16"}, PrivateIP:"172.17.0.9", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  4680. 2020-10-08T11:45:17.0102679Z [cluster2] I1008 11:40:26.101169 1 datastoresyncer.go:435] Endpoint "cluster1-submariner-cable-cluster1-172-17-0-9" matched what we received from datastore - not updating
  4681. 2020-10-08T11:45:17.0103823Z [cluster2] I1008 11:40:26.106373 1 kubernetes.go:180] UpdateFunc in WatchClusters called
  4682. 2020-10-08T11:45:17.0108701Z [cluster2] I1008 11:40:26.106404 1 datastoresyncer.go:312] In reconcileClusterCRD: &types.SubmarinerCluster{ID:"cluster1", Spec:v1.ClusterSpec{ClusterID:"cluster1", ColorCodes:[]string{"blue"}, ServiceCIDR:[]string{"100.91.0.0/16"}, ClusterCIDR:[]string{"10.241.0.0/16"}, GlobalCIDR:[]string{}}}
  4683. 2020-10-08T11:45:17.0110529Z [cluster2] I1008 11:40:26.108105 1 datastoresyncer.go:360] Cluster "cluster1" matched what we received from datastore - not updating
  4684. 2020-10-08T11:45:17.0111259Z [cluster2] I1008 11:40:26.108149 1 kubernetes.go:180] UpdateFunc in WatchClusters called
  4685. 2020-10-08T11:45:17.0111878Z [cluster2] I1008 11:40:56.099424 1 kubernetes.go:264] UpdateFunc in WatchEndpoints called
  4686. 2020-10-08T11:45:17.0114063Z [cluster2] I1008 11:40:56.099508 1 datastoresyncer.go:387] In reconcileEndpointCRD: &types.SubmarinerEndpoint{Spec:v1.EndpointSpec{ClusterID:"cluster1", CableName:"submariner-cable-cluster1-172-17-0-9", Hostname:"cluster1-worker", Subnets:[]string{"100.91.0.0/16", "10.241.0.0/16"}, PrivateIP:"172.17.0.9", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  4687. 2020-10-08T11:45:17.0117163Z [cluster2] I1008 11:40:56.103277 1 datastoresyncer.go:435] Endpoint "cluster1-submariner-cable-cluster1-172-17-0-9" matched what we received from datastore - not updating
  4688. 2020-10-08T11:45:17.0118545Z [cluster2] I1008 11:40:56.103340 1 kubernetes.go:264] UpdateFunc in WatchEndpoints called
  4689. 2020-10-08T11:45:17.0119303Z [cluster2] I1008 11:40:56.106584 1 kubernetes.go:180] UpdateFunc in WatchClusters called
  4690. 2020-10-08T11:45:17.0120543Z [cluster2] I1008 11:40:56.106617 1 datastoresyncer.go:312] In reconcileClusterCRD: &types.SubmarinerCluster{ID:"cluster1", Spec:v1.ClusterSpec{ClusterID:"cluster1", ColorCodes:[]string{"blue"}, ServiceCIDR:[]string{"100.91.0.0/16"}, ClusterCIDR:[]string{"10.241.0.0/16"}, GlobalCIDR:[]string{}}}
  4691. 2020-10-08T11:45:17.0122116Z [cluster2] I1008 11:40:56.108487 1 datastoresyncer.go:360] Cluster "cluster1" matched what we received from datastore - not updating
  4692. 2020-10-08T11:45:17.0122840Z [cluster2] I1008 11:40:56.108524 1 kubernetes.go:180] UpdateFunc in WatchClusters called
  4693. 2020-10-08T11:45:17.0129735Z [cluster2] I1008 11:41:25.062121 1 tunnel.go:95] Tunnel controller processing added or updated submariner Endpoint object: &v1.Endpoint{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"cluster2-submariner-cable-cluster2-172-17-0-5", GenerateName:"", Namespace:"submariner-operator", SelfLink:"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster2-submariner-cable-cluster2-172-17-0-5", UID:"6a117963-5044-4b34-9df6-57ebb8a4981d", ResourceVersion:"1284", Generation:1, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63737753726, loc:(*time.Location)(0x225ef80)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v1.EndpointSpec{ClusterID:"cluster2", CableName:"submariner-cable-cluster2-172-17-0-5", Hostname:"cluster2-worker", Subnets:[]string{"100.92.0.0/16", "10.242.0.0/16"}, PrivateIP:"172.17.0.5", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  4694. 2020-10-08T11:45:17.0137163Z [cluster2] I1008 11:41:25.062237 1 cableengine.go:92] Not installing cable for local cluster
  4695. 2020-10-08T11:45:17.0138515Z [cluster2] I1008 11:41:25.062250 1 tunnel.go:108] Tunnel controller successfully installed Endpoint cable submariner-cable-cluster2-172-17-0-5 in the engine
  4696. 2020-10-08T11:45:17.0145319Z [cluster2] I1008 11:41:25.062263 1 tunnel.go:95] Tunnel controller processing added or updated submariner Endpoint object: &v1.Endpoint{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"cluster1-submariner-cable-cluster1-172-17-0-9", GenerateName:"", Namespace:"submariner-operator", SelfLink:"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster1-submariner-cable-cluster1-172-17-0-9", UID:"60d04732-7b08-4685-b72b-cb2c12702817", ResourceVersion:"1285", Generation:1, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63737753726, loc:(*time.Location)(0x225ef80)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v1.EndpointSpec{ClusterID:"cluster1", CableName:"submariner-cable-cluster1-172-17-0-9", Hostname:"cluster1-worker", Subnets:[]string{"100.91.0.0/16", "10.241.0.0/16"}, PrivateIP:"172.17.0.9", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  4697. 2020-10-08T11:45:17.0151678Z [cluster2] I1008 11:41:25.062300 1 cableengine.go:101] Installing Endpoint cable "submariner-cable-cluster1-172-17-0-9"
  4698. 2020-10-08T11:45:17.0153739Z [cluster2] I1008 11:41:25.062302 1 kubernetes.go:317] In SetCluster: &types.SubmarinerCluster{ID:"cluster2", Spec:v1.ClusterSpec{ClusterID:"cluster2", ColorCodes:[]string{"blue"}, ServiceCIDR:[]string{"100.92.0.0/16"}, ClusterCIDR:[]string{"10.242.0.0/16"}, GlobalCIDR:[]string{}}}
  4699. 2020-10-08T11:45:17.0157828Z [cluster2] I1008 11:41:25.065423 1 kubernetes.go:345] Cluster "cluster2" matched what we received from datastore - not updating
  4700. 2020-10-08T11:45:17.0159363Z [cluster2] I1008 11:41:25.067316 1 cableengine.go:128] Cable "submariner-cable-cluster1-172-17-0-9" is already installed - not installing again
  4701. 2020-10-08T11:45:17.0161250Z [cluster2] I1008 11:41:25.067396 1 tunnel.go:108] Tunnel controller successfully installed Endpoint cable submariner-cable-cluster1-172-17-0-9 in the engine
  4702. 2020-10-08T11:45:17.0169829Z [cluster2] I1008 11:41:25.069396 1 datastoresyncer.go:287] Processing local submariner Endpoint object: &v1.Endpoint{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"cluster2-submariner-cable-cluster2-172-17-0-5", GenerateName:"", Namespace:"submariner-operator", SelfLink:"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster2-submariner-cable-cluster2-172-17-0-5", UID:"6a117963-5044-4b34-9df6-57ebb8a4981d", ResourceVersion:"1284", Generation:1, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63737753726, loc:(*time.Location)(0x225ef80)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v1.EndpointSpec{ClusterID:"cluster2", CableName:"submariner-cable-cluster2-172-17-0-5", Hostname:"cluster2-worker", Subnets:[]string{"100.92.0.0/16", "10.242.0.0/16"}, PrivateIP:"172.17.0.5", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  4703. 2020-10-08T11:45:17.0179291Z [cluster2] I1008 11:41:25.069548 1 kubernetes.go:370] In SetEndpoint: &types.SubmarinerEndpoint{Spec:v1.EndpointSpec{ClusterID:"cluster2", CableName:"submariner-cable-cluster2-172-17-0-5", Hostname:"cluster2-worker", Subnets:[]string{"100.92.0.0/16", "10.242.0.0/16"}, PrivateIP:"172.17.0.5", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  4704. 2020-10-08T11:45:17.0182218Z [cluster2] I1008 11:41:25.071561 1 kubernetes.go:398] Endpoint "cluster2-submariner-cable-cluster2-172-17-0-5" matched what we received from datastore - not updating
  4705. 2020-10-08T11:45:17.0188338Z [cluster2] I1008 11:41:25.073171 1 datastoresyncer.go:271] The updated submariner Endpoint "cluster1" is not for this cluster - skipping updating the datastore
  4706. 2020-10-08T11:45:17.0189160Z [cluster2] I1008 11:41:26.099694 1 kubernetes.go:264] UpdateFunc in WatchEndpoints called
  4707. 2020-10-08T11:45:17.0191330Z [cluster2] I1008 11:41:26.099731 1 datastoresyncer.go:387] In reconcileEndpointCRD: &types.SubmarinerEndpoint{Spec:v1.EndpointSpec{ClusterID:"cluster1", CableName:"submariner-cable-cluster1-172-17-0-9", Hostname:"cluster1-worker", Subnets:[]string{"100.91.0.0/16", "10.241.0.0/16"}, PrivateIP:"172.17.0.9", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  4708. 2020-10-08T11:45:17.0194313Z [cluster2] I1008 11:41:26.102932 1 datastoresyncer.go:435] Endpoint "cluster1-submariner-cable-cluster1-172-17-0-9" matched what we received from datastore - not updating
  4709. 2020-10-08T11:45:17.0196996Z [cluster2] I1008 11:41:26.102973 1 kubernetes.go:264] UpdateFunc in WatchEndpoints called
  4710. 2020-10-08T11:45:17.0197726Z [cluster2] I1008 11:41:26.106787 1 kubernetes.go:180] UpdateFunc in WatchClusters called
  4711. 2020-10-08T11:45:17.0202241Z [cluster2] I1008 11:41:26.106812 1 datastoresyncer.go:312] In reconcileClusterCRD: &types.SubmarinerCluster{ID:"cluster1", Spec:v1.ClusterSpec{ClusterID:"cluster1", ColorCodes:[]string{"blue"}, ServiceCIDR:[]string{"100.91.0.0/16"}, ClusterCIDR:[]string{"10.241.0.0/16"}, GlobalCIDR:[]string{}}}
  4712. 2020-10-08T11:45:17.0209786Z [cluster2] I1008 11:41:26.108493 1 datastoresyncer.go:360] Cluster "cluster1" matched what we received from datastore - not updating
  4713. 2020-10-08T11:45:17.0210602Z [cluster2] I1008 11:41:26.108517 1 kubernetes.go:180] UpdateFunc in WatchClusters called
  4714. 2020-10-08T11:45:17.0212066Z [cluster2] I1008 11:41:56.099815 1 kubernetes.go:264] UpdateFunc in WatchEndpoints called
  4715. 2020-10-08T11:45:17.0217331Z [cluster2] I1008 11:41:56.099840 1 datastoresyncer.go:387] In reconcileEndpointCRD: &types.SubmarinerEndpoint{Spec:v1.EndpointSpec{ClusterID:"cluster1", CableName:"submariner-cable-cluster1-172-17-0-9", Hostname:"cluster1-worker", Subnets:[]string{"100.91.0.0/16", "10.241.0.0/16"}, PrivateIP:"172.17.0.9", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  4716. 2020-10-08T11:45:17.0224419Z [cluster2] I1008 11:41:56.102023 1 datastoresyncer.go:435] Endpoint "cluster1-submariner-cable-cluster1-172-17-0-9" matched what we received from datastore - not updating
  4717. 2020-10-08T11:45:17.0238245Z [cluster2] I1008 11:41:56.102051 1 kubernetes.go:264] UpdateFunc in WatchEndpoints called
  4718. 2020-10-08T11:45:17.0239136Z [cluster2] I1008 11:41:56.106880 1 kubernetes.go:180] UpdateFunc in WatchClusters called
  4719. 2020-10-08T11:45:17.0240885Z [cluster2] I1008 11:41:56.106908 1 datastoresyncer.go:312] In reconcileClusterCRD: &types.SubmarinerCluster{ID:"cluster1", Spec:v1.ClusterSpec{ClusterID:"cluster1", ColorCodes:[]string{"blue"}, ServiceCIDR:[]string{"100.91.0.0/16"}, ClusterCIDR:[]string{"10.241.0.0/16"}, GlobalCIDR:[]string{}}}
  4720. 2020-10-08T11:45:17.0243397Z [cluster2] I1008 11:41:56.109316 1 datastoresyncer.go:360] Cluster "cluster1" matched what we received from datastore - not updating
  4721. 2020-10-08T11:45:17.0244239Z [cluster2] I1008 11:41:56.109401 1 kubernetes.go:180] UpdateFunc in WatchClusters called
  4722. 2020-10-08T11:45:17.0250730Z [cluster2] I1008 11:42:25.062340 1 kubernetes.go:317] In SetCluster: &types.SubmarinerCluster{ID:"cluster2", Spec:v1.ClusterSpec{ClusterID:"cluster2", ColorCodes:[]string{"blue"}, ServiceCIDR:[]string{"100.92.0.0/16"}, ClusterCIDR:[]string{"10.242.0.0/16"}, GlobalCIDR:[]string{}}}
  4723. 2020-10-08T11:45:17.0268056Z [cluster2] I1008 11:42:25.062533 1 tunnel.go:95] Tunnel controller processing added or updated submariner Endpoint object: &v1.Endpoint{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"cluster2-submariner-cable-cluster2-172-17-0-5", GenerateName:"", Namespace:"submariner-operator", SelfLink:"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster2-submariner-cable-cluster2-172-17-0-5", UID:"6a117963-5044-4b34-9df6-57ebb8a4981d", ResourceVersion:"1284", Generation:1, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63737753726, loc:(*time.Location)(0x225ef80)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v1.EndpointSpec{ClusterID:"cluster2", CableName:"submariner-cable-cluster2-172-17-0-5", Hostname:"cluster2-worker", Subnets:[]string{"100.92.0.0/16", "10.242.0.0/16"}, PrivateIP:"172.17.0.5", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  4724. 2020-10-08T11:45:17.0272907Z [cluster2] I1008 11:42:25.062648 1 cableengine.go:92] Not installing cable for local cluster
  4725. 2020-10-08T11:45:17.0274155Z [cluster2] I1008 11:42:25.062663 1 tunnel.go:108] Tunnel controller successfully installed Endpoint cable submariner-cable-cluster2-172-17-0-5 in the engine
  4726. 2020-10-08T11:45:17.0280825Z [cluster2] I1008 11:42:25.062674 1 tunnel.go:95] Tunnel controller processing added or updated submariner Endpoint object: &v1.Endpoint{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"cluster1-submariner-cable-cluster1-172-17-0-9", GenerateName:"", Namespace:"submariner-operator", SelfLink:"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster1-submariner-cable-cluster1-172-17-0-9", UID:"60d04732-7b08-4685-b72b-cb2c12702817", ResourceVersion:"1285", Generation:1, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63737753726, loc:(*time.Location)(0x225ef80)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v1.EndpointSpec{ClusterID:"cluster1", CableName:"submariner-cable-cluster1-172-17-0-9", Hostname:"cluster1-worker", Subnets:[]string{"100.91.0.0/16", "10.241.0.0/16"}, PrivateIP:"172.17.0.9", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  4727. 2020-10-08T11:45:17.0286094Z [cluster2] I1008 11:42:25.062708 1 cableengine.go:101] Installing Endpoint cable "submariner-cable-cluster1-172-17-0-9"
  4728. 2020-10-08T11:45:17.0327437Z [cluster2] I1008 11:42:25.065215 1 datastoresyncer.go:287] Processing local submariner Endpoint object: &v1.Endpoint{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"cluster2-submariner-cable-cluster2-172-17-0-5", GenerateName:"", Namespace:"submariner-operator", SelfLink:"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster2-submariner-cable-cluster2-172-17-0-5", UID:"6a117963-5044-4b34-9df6-57ebb8a4981d", ResourceVersion:"1284", Generation:1, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63737753726, loc:(*time.Location)(0x225ef80)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v1.EndpointSpec{ClusterID:"cluster2", CableName:"submariner-cable-cluster2-172-17-0-5", Hostname:"cluster2-worker", Subnets:[]string{"100.92.0.0/16", "10.242.0.0/16"}, PrivateIP:"172.17.0.5", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  4729. 2020-10-08T11:45:17.0336557Z [cluster2] I1008 11:42:25.065866 1 kubernetes.go:370] In SetEndpoint: &types.SubmarinerEndpoint{Spec:v1.EndpointSpec{ClusterID:"cluster2", CableName:"submariner-cable-cluster2-172-17-0-5", Hostname:"cluster2-worker", Subnets:[]string{"100.92.0.0/16", "10.242.0.0/16"}, PrivateIP:"172.17.0.5", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  4730. 2020-10-08T11:45:17.0339144Z [cluster2] I1008 11:42:25.067735 1 kubernetes.go:345] Cluster "cluster2" matched what we received from datastore - not updating
  4731. 2020-10-08T11:45:17.0340893Z [cluster2] I1008 11:42:25.068855 1 kubernetes.go:398] Endpoint "cluster2-submariner-cable-cluster2-172-17-0-5" matched what we received from datastore - not updating
  4732. 2020-10-08T11:45:17.0342927Z [cluster2] I1008 11:42:25.069778 1 cableengine.go:128] Cable "submariner-cable-cluster1-172-17-0-9" is already installed - not installing again
  4733. 2020-10-08T11:45:17.0345211Z [cluster2] I1008 11:42:25.069795 1 tunnel.go:108] Tunnel controller successfully installed Endpoint cable submariner-cable-cluster1-172-17-0-9 in the engine
  4734. 2020-10-08T11:45:17.0346936Z [cluster2] I1008 11:42:25.070878 1 datastoresyncer.go:271] The updated submariner Endpoint "cluster1" is not for this cluster - skipping updating the datastore
  4735. 2020-10-08T11:45:17.0347939Z [cluster2] I1008 11:42:26.099931 1 kubernetes.go:264] UpdateFunc in WatchEndpoints called
  4736. 2020-10-08T11:45:17.0373239Z [cluster2] I1008 11:42:26.100047 1 datastoresyncer.go:387] In reconcileEndpointCRD: &types.SubmarinerEndpoint{Spec:v1.EndpointSpec{ClusterID:"cluster1", CableName:"submariner-cable-cluster1-172-17-0-9", Hostname:"cluster1-worker", Subnets:[]string{"100.91.0.0/16", "10.241.0.0/16"}, PrivateIP:"172.17.0.9", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  4737. 2020-10-08T11:45:17.0376898Z [cluster2] I1008 11:42:26.102169 1 datastoresyncer.go:435] Endpoint "cluster1-submariner-cable-cluster1-172-17-0-9" matched what we received from datastore - not updating
  4738. 2020-10-08T11:45:17.0378237Z [cluster2] I1008 11:42:26.102337 1 kubernetes.go:264] UpdateFunc in WatchEndpoints called
  4739. 2020-10-08T11:45:17.0378950Z [cluster2] I1008 11:42:26.106961 1 kubernetes.go:180] UpdateFunc in WatchClusters called
  4740. 2020-10-08T11:45:17.0380391Z [cluster2] I1008 11:42:26.106980 1 datastoresyncer.go:312] In reconcileClusterCRD: &types.SubmarinerCluster{ID:"cluster1", Spec:v1.ClusterSpec{ClusterID:"cluster1", ColorCodes:[]string{"blue"}, ServiceCIDR:[]string{"100.91.0.0/16"}, ClusterCIDR:[]string{"10.241.0.0/16"}, GlobalCIDR:[]string{}}}
  4741. 2020-10-08T11:45:17.0381943Z [cluster2] I1008 11:42:26.108672 1 datastoresyncer.go:360] Cluster "cluster1" matched what we received from datastore - not updating
  4742. 2020-10-08T11:45:17.0382664Z [cluster2] I1008 11:42:26.108689 1 kubernetes.go:180] UpdateFunc in WatchClusters called
  4743. 2020-10-08T11:45:17.0383211Z [cluster2] 01[KNL] interface vethwepg4e16d28 deleted
  4744. 2020-10-08T11:45:17.0383791Z [cluster2] 05[KNL] interface vethwepl4e16d28 activated
  4745. 2020-10-08T11:45:17.0384233Z [cluster2] 10[KNL] interface vethweplf65a7f2 deactivated
  4746. 2020-10-08T11:45:17.0385006Z [cluster2] 15[KNL] interface vethweplf65a7f2 deleted
  4747. 2020-10-08T11:45:17.0385558Z [cluster2] I1008 11:42:56.100191 1 kubernetes.go:264] UpdateFunc in WatchEndpoints called
  4748. 2020-10-08T11:45:17.0387905Z [cluster2] I1008 11:42:56.100216 1 datastoresyncer.go:387] In reconcileEndpointCRD: &types.SubmarinerEndpoint{Spec:v1.EndpointSpec{ClusterID:"cluster1", CableName:"submariner-cable-cluster1-172-17-0-9", Hostname:"cluster1-worker", Subnets:[]string{"100.91.0.0/16", "10.241.0.0/16"}, PrivateIP:"172.17.0.9", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  4749. 2020-10-08T11:45:17.0390536Z [cluster2] I1008 11:42:56.102541 1 datastoresyncer.go:435] Endpoint "cluster1-submariner-cable-cluster1-172-17-0-9" matched what we received from datastore - not updating
  4750. 2020-10-08T11:45:17.0391694Z [cluster2] I1008 11:42:56.102574 1 kubernetes.go:264] UpdateFunc in WatchEndpoints called
  4751. 2020-10-08T11:45:17.0392309Z [cluster2] I1008 11:42:56.107162 1 kubernetes.go:180] UpdateFunc in WatchClusters called
  4752. 2020-10-08T11:45:17.0392912Z [cluster2] I1008 11:42:56.107207 1 kubernetes.go:180] UpdateFunc in WatchClusters called
  4753. 2020-10-08T11:45:17.0394206Z [cluster2] I1008 11:42:56.107251 1 datastoresyncer.go:312] In reconcileClusterCRD: &types.SubmarinerCluster{ID:"cluster1", Spec:v1.ClusterSpec{ClusterID:"cluster1", ColorCodes:[]string{"blue"}, ServiceCIDR:[]string{"100.91.0.0/16"}, ClusterCIDR:[]string{"10.241.0.0/16"}, GlobalCIDR:[]string{}}}
  4754. 2020-10-08T11:45:17.0396258Z [cluster2] I1008 11:42:56.109101 1 datastoresyncer.go:360] Cluster "cluster1" matched what we received from datastore - not updating
  4755. 2020-10-08T11:45:17.0396913Z [cluster2] 09[KNL] interface vethwepg9464469 deleted
  4756. 2020-10-08T11:45:17.0397390Z [cluster2] 08[KNL] interface vethwepl9464469 activated
  4757. 2020-10-08T11:45:17.0397891Z [cluster2] 10[KNL] interface vethwepl4e16d28 deactivated
  4758. 2020-10-08T11:45:17.0398386Z [cluster2] 05[KNL] interface vethwepl4e16d28 deleted
  4759. 2020-10-08T11:45:17.0408748Z [cluster2] I1008 11:43:25.062580 1 tunnel.go:95] Tunnel controller processing added or updated submariner Endpoint object: &v1.Endpoint{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"cluster2-submariner-cable-cluster2-172-17-0-5", GenerateName:"", Namespace:"submariner-operator", SelfLink:"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster2-submariner-cable-cluster2-172-17-0-5", UID:"6a117963-5044-4b34-9df6-57ebb8a4981d", ResourceVersion:"1284", Generation:1, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63737753726, loc:(*time.Location)(0x225ef80)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v1.EndpointSpec{ClusterID:"cluster2", CableName:"submariner-cable-cluster2-172-17-0-5", Hostname:"cluster2-worker", Subnets:[]string{"100.92.0.0/16", "10.242.0.0/16"}, PrivateIP:"172.17.0.5", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  4760. 2020-10-08T11:45:17.0416930Z [cluster2] I1008 11:43:25.063025 1 cableengine.go:92] Not installing cable for local cluster
  4761. 2020-10-08T11:45:17.0418132Z [cluster2] I1008 11:43:25.063669 1 tunnel.go:108] Tunnel controller successfully installed Endpoint cable submariner-cable-cluster2-172-17-0-5 in the engine
  4762. 2020-10-08T11:45:17.0419422Z [cluster2] I1008 11:43:25.062728 1 kubernetes.go:317] In SetCluster: &types.SubmarinerCluster{ID:"cluster2", Spec:v1.ClusterSpec{ClusterID:"cluster2", ColorCodes:[]string{"blue"}, ServiceCIDR:[]string{"100.92.0.0/16"}, ClusterCIDR:[]string{"10.242.0.0/16"}, GlobalCIDR:[]string{}}}
  4763. 2020-10-08T11:45:17.0426092Z [cluster2] I1008 11:43:25.064273 1 tunnel.go:95] Tunnel controller processing added or updated submariner Endpoint object: &v1.Endpoint{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"cluster1-submariner-cable-cluster1-172-17-0-9", GenerateName:"", Namespace:"submariner-operator", SelfLink:"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster1-submariner-cable-cluster1-172-17-0-9", UID:"60d04732-7b08-4685-b72b-cb2c12702817", ResourceVersion:"1285", Generation:1, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63737753726, loc:(*time.Location)(0x225ef80)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v1.EndpointSpec{ClusterID:"cluster1", CableName:"submariner-cable-cluster1-172-17-0-9", Hostname:"cluster1-worker", Subnets:[]string{"100.91.0.0/16", "10.241.0.0/16"}, PrivateIP:"172.17.0.9", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  4764. 2020-10-08T11:45:17.0431718Z [cluster2] I1008 11:43:25.064490 1 cableengine.go:101] Installing Endpoint cable "submariner-cable-cluster1-172-17-0-9"
  4765. 2020-10-08T11:45:17.0439770Z [cluster2] I1008 11:43:25.065627 1 datastoresyncer.go:287] Processing local submariner Endpoint object: &v1.Endpoint{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"cluster2-submariner-cable-cluster2-172-17-0-5", GenerateName:"", Namespace:"submariner-operator", SelfLink:"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster2-submariner-cable-cluster2-172-17-0-5", UID:"6a117963-5044-4b34-9df6-57ebb8a4981d", ResourceVersion:"1284", Generation:1, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63737753726, loc:(*time.Location)(0x225ef80)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v1.EndpointSpec{ClusterID:"cluster2", CableName:"submariner-cable-cluster2-172-17-0-5", Hostname:"cluster2-worker", Subnets:[]string{"100.92.0.0/16", "10.242.0.0/16"}, PrivateIP:"172.17.0.5", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  4766. 2020-10-08T11:45:17.0446336Z [cluster2] I1008 11:43:25.065734 1 kubernetes.go:370] In SetEndpoint: &types.SubmarinerEndpoint{Spec:v1.EndpointSpec{ClusterID:"cluster2", CableName:"submariner-cable-cluster2-172-17-0-5", Hostname:"cluster2-worker", Subnets:[]string{"100.92.0.0/16", "10.242.0.0/16"}, PrivateIP:"172.17.0.5", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  4767. 2020-10-08T11:45:17.0447971Z [cluster2] I1008 11:43:25.067736 1 kubernetes.go:345] Cluster "cluster2" matched what we received from datastore - not updating
  4768. 2020-10-08T11:45:17.0449185Z [cluster2] I1008 11:43:25.069110 1 kubernetes.go:398] Endpoint "cluster2-submariner-cable-cluster2-172-17-0-5" matched what we received from datastore - not updating
  4769. 2020-10-08T11:45:17.0450491Z [cluster2] I1008 11:43:25.071040 1 datastoresyncer.go:271] The updated submariner Endpoint "cluster1" is not for this cluster - skipping updating the datastore
  4770. 2020-10-08T11:45:17.0451633Z [cluster2] I1008 11:43:25.071740 1 cableengine.go:128] Cable "submariner-cable-cluster1-172-17-0-9" is already installed - not installing again
  4771. 2020-10-08T11:45:17.0453414Z [cluster2] I1008 11:43:25.071768 1 tunnel.go:108] Tunnel controller successfully installed Endpoint cable submariner-cable-cluster1-172-17-0-9 in the engine
  4772. 2020-10-08T11:45:17.0454265Z [cluster2] I1008 11:43:26.100492 1 kubernetes.go:264] UpdateFunc in WatchEndpoints called
  4773. 2020-10-08T11:45:17.0457282Z [cluster2] I1008 11:43:26.100556 1 datastoresyncer.go:387] In reconcileEndpointCRD: &types.SubmarinerEndpoint{Spec:v1.EndpointSpec{ClusterID:"cluster1", CableName:"submariner-cable-cluster1-172-17-0-9", Hostname:"cluster1-worker", Subnets:[]string{"100.91.0.0/16", "10.241.0.0/16"}, PrivateIP:"172.17.0.9", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  4774. 2020-10-08T11:45:17.0459596Z [cluster2] I1008 11:43:26.102645 1 datastoresyncer.go:435] Endpoint "cluster1-submariner-cable-cluster1-172-17-0-9" matched what we received from datastore - not updating
  4775. 2020-10-08T11:45:17.0461416Z [cluster2] I1008 11:43:26.102670 1 kubernetes.go:264] UpdateFunc in WatchEndpoints called
  4776. 2020-10-08T11:45:17.0462077Z [cluster2] I1008 11:43:26.107294 1 kubernetes.go:180] UpdateFunc in WatchClusters called
  4777. 2020-10-08T11:45:17.0463763Z [cluster2] I1008 11:43:26.107314 1 datastoresyncer.go:312] In reconcileClusterCRD: &types.SubmarinerCluster{ID:"cluster1", Spec:v1.ClusterSpec{ClusterID:"cluster1", ColorCodes:[]string{"blue"}, ServiceCIDR:[]string{"100.91.0.0/16"}, ClusterCIDR:[]string{"10.241.0.0/16"}, GlobalCIDR:[]string{}}}
  4778. 2020-10-08T11:45:17.0466484Z [cluster2] I1008 11:43:26.108918 1 datastoresyncer.go:360] Cluster "cluster1" matched what we received from datastore - not updating
  4779. 2020-10-08T11:45:17.0467347Z [cluster2] I1008 11:43:26.108962 1 kubernetes.go:180] UpdateFunc in WatchClusters called
  4780. 2020-10-08T11:45:17.0468039Z [cluster2] 06[KNL] interface vethwepgcd52737 deleted
  4781. 2020-10-08T11:45:17.0468534Z [cluster2] 16[KNL] interface vethweplcd52737 activated
  4782. 2020-10-08T11:45:17.0469035Z [cluster2] 14[KNL] interface vethwepl9464469 deactivated
  4783. 2020-10-08T11:45:17.0469521Z [cluster2] 15[KNL] interface vethwepl9464469 deleted
  4784. 2020-10-08T11:45:17.0470049Z [cluster2] I1008 11:43:56.100598 1 kubernetes.go:264] UpdateFunc in WatchEndpoints called
  4785. 2020-10-08T11:45:17.0472350Z [cluster2] I1008 11:43:56.100622 1 datastoresyncer.go:387] In reconcileEndpointCRD: &types.SubmarinerEndpoint{Spec:v1.EndpointSpec{ClusterID:"cluster1", CableName:"submariner-cable-cluster1-172-17-0-9", Hostname:"cluster1-worker", Subnets:[]string{"100.91.0.0/16", "10.241.0.0/16"}, PrivateIP:"172.17.0.9", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  4786. 2020-10-08T11:45:17.0475833Z [cluster2] I1008 11:43:56.102467 1 datastoresyncer.go:435] Endpoint "cluster1-submariner-cable-cluster1-172-17-0-9" matched what we received from datastore - not updating
  4787. 2020-10-08T11:45:17.0477000Z [cluster2] I1008 11:43:56.102494 1 kubernetes.go:264] UpdateFunc in WatchEndpoints called
  4788. 2020-10-08T11:45:17.0477620Z [cluster2] I1008 11:43:56.107399 1 kubernetes.go:180] UpdateFunc in WatchClusters called
  4789. 2020-10-08T11:45:17.0479061Z [cluster2] I1008 11:43:56.107430 1 datastoresyncer.go:312] In reconcileClusterCRD: &types.SubmarinerCluster{ID:"cluster1", Spec:v1.ClusterSpec{ClusterID:"cluster1", ColorCodes:[]string{"blue"}, ServiceCIDR:[]string{"100.91.0.0/16"}, ClusterCIDR:[]string{"10.241.0.0/16"}, GlobalCIDR:[]string{}}}
  4790. 2020-10-08T11:45:17.0480860Z [cluster2] I1008 11:43:56.108873 1 datastoresyncer.go:360] Cluster "cluster1" matched what we received from datastore - not updating
  4791. 2020-10-08T11:45:17.0481685Z [cluster2] I1008 11:43:56.108893 1 kubernetes.go:180] UpdateFunc in WatchClusters called
  4792. 2020-10-08T11:45:17.0482343Z [cluster2] 06[KNL] interface vethweplcd52737 deactivated
  4793. 2020-10-08T11:45:17.0482902Z [cluster2] 01[KNL] interface vethweplcd52737 deleted
  4794. 2020-10-08T11:45:17.0483450Z [cluster2] 05[KNL] interface vethwepg72df167 deleted
  4795. 2020-10-08T11:45:17.0484010Z [cluster2] 11[KNL] interface vethwepl72df167 activated
  4796. 2020-10-08T11:45:17.0485910Z [cluster2] I1008 11:44:25.062821 1 kubernetes.go:317] In SetCluster: &types.SubmarinerCluster{ID:"cluster2", Spec:v1.ClusterSpec{ClusterID:"cluster2", ColorCodes:[]string{"blue"}, ServiceCIDR:[]string{"100.92.0.0/16"}, ClusterCIDR:[]string{"10.242.0.0/16"}, GlobalCIDR:[]string{}}}
  4797. 2020-10-08T11:45:17.0494189Z [cluster2] I1008 11:44:25.062816 1 tunnel.go:95] Tunnel controller processing added or updated submariner Endpoint object: &v1.Endpoint{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"cluster1-submariner-cable-cluster1-172-17-0-9", GenerateName:"", Namespace:"submariner-operator", SelfLink:"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster1-submariner-cable-cluster1-172-17-0-9", UID:"60d04732-7b08-4685-b72b-cb2c12702817", ResourceVersion:"1285", Generation:1, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63737753726, loc:(*time.Location)(0x225ef80)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v1.EndpointSpec{ClusterID:"cluster1", CableName:"submariner-cable-cluster1-172-17-0-9", Hostname:"cluster1-worker", Subnets:[]string{"100.91.0.0/16", "10.241.0.0/16"}, PrivateIP:"172.17.0.9", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  4798. 2020-10-08T11:45:17.0501162Z [cluster2] I1008 11:44:25.062974 1 cableengine.go:101] Installing Endpoint cable "submariner-cable-cluster1-172-17-0-9"
  4799. 2020-10-08T11:45:17.0502618Z [cluster2] I1008 11:44:25.065145 1 kubernetes.go:345] Cluster "cluster2" matched what we received from datastore - not updating
  4800. 2020-10-08T11:45:17.0504015Z [cluster2] I1008 11:44:25.067665 1 cableengine.go:128] Cable "submariner-cable-cluster1-172-17-0-9" is already installed - not installing again
  4801. 2020-10-08T11:45:17.0506260Z [cluster2] I1008 11:44:25.067702 1 tunnel.go:108] Tunnel controller successfully installed Endpoint cable submariner-cable-cluster1-172-17-0-9 in the engine
  4802. 2020-10-08T11:45:17.0512755Z [cluster2] I1008 11:44:25.067726 1 tunnel.go:95] Tunnel controller processing added or updated submariner Endpoint object: &v1.Endpoint{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"cluster2-submariner-cable-cluster2-172-17-0-5", GenerateName:"", Namespace:"submariner-operator", SelfLink:"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster2-submariner-cable-cluster2-172-17-0-5", UID:"6a117963-5044-4b34-9df6-57ebb8a4981d", ResourceVersion:"1284", Generation:1, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63737753726, loc:(*time.Location)(0x225ef80)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v1.EndpointSpec{ClusterID:"cluster2", CableName:"submariner-cable-cluster2-172-17-0-5", Hostname:"cluster2-worker", Subnets:[]string{"100.92.0.0/16", "10.242.0.0/16"}, PrivateIP:"172.17.0.5", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  4803. 2020-10-08T11:45:17.0518432Z [cluster2] I1008 11:44:25.067799 1 cableengine.go:92] Not installing cable for local cluster
  4804. 2020-10-08T11:45:17.0519691Z [cluster2] I1008 11:44:25.067817 1 tunnel.go:108] Tunnel controller successfully installed Endpoint cable submariner-cable-cluster2-172-17-0-5 in the engine
  4805. 2020-10-08T11:45:17.0521086Z [cluster2] I1008 11:44:25.068818 1 datastoresyncer.go:271] The updated submariner Endpoint "cluster1" is not for this cluster - skipping updating the datastore
  4806. 2020-10-08T11:45:17.0527476Z [cluster2] I1008 11:44:25.070356 1 datastoresyncer.go:287] Processing local submariner Endpoint object: &v1.Endpoint{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"cluster2-submariner-cable-cluster2-172-17-0-5", GenerateName:"", Namespace:"submariner-operator", SelfLink:"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster2-submariner-cable-cluster2-172-17-0-5", UID:"6a117963-5044-4b34-9df6-57ebb8a4981d", ResourceVersion:"1284", Generation:1, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63737753726, loc:(*time.Location)(0x225ef80)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v1.EndpointSpec{ClusterID:"cluster2", CableName:"submariner-cable-cluster2-172-17-0-5", Hostname:"cluster2-worker", Subnets:[]string{"100.92.0.0/16", "10.242.0.0/16"}, PrivateIP:"172.17.0.5", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  4807. 2020-10-08T11:45:17.0533666Z [cluster2] I1008 11:44:25.070404 1 kubernetes.go:370] In SetEndpoint: &types.SubmarinerEndpoint{Spec:v1.EndpointSpec{ClusterID:"cluster2", CableName:"submariner-cable-cluster2-172-17-0-5", Hostname:"cluster2-worker", Subnets:[]string{"100.92.0.0/16", "10.242.0.0/16"}, PrivateIP:"172.17.0.5", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  4808. 2020-10-08T11:45:17.0538180Z [cluster2] I1008 11:44:25.072391 1 kubernetes.go:398] Endpoint "cluster2-submariner-cable-cluster2-172-17-0-5" matched what we received from datastore - not updating
  4809. 2020-10-08T11:45:17.0539351Z [cluster2] I1008 11:44:26.100647 1 kubernetes.go:264] UpdateFunc in WatchEndpoints called
  4810. 2020-10-08T11:45:17.0540007Z [cluster2] I1008 11:44:26.100704 1 kubernetes.go:264] UpdateFunc in WatchEndpoints called
  4811. 2020-10-08T11:45:17.0541884Z [cluster2] I1008 11:44:26.100728 1 datastoresyncer.go:387] In reconcileEndpointCRD: &types.SubmarinerEndpoint{Spec:v1.EndpointSpec{ClusterID:"cluster1", CableName:"submariner-cable-cluster1-172-17-0-9", Hostname:"cluster1-worker", Subnets:[]string{"100.91.0.0/16", "10.241.0.0/16"}, PrivateIP:"172.17.0.9", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  4812. 2020-10-08T11:45:17.0545816Z [cluster2] I1008 11:44:26.103330 1 datastoresyncer.go:435] Endpoint "cluster1-submariner-cable-cluster1-172-17-0-9" matched what we received from datastore - not updating
  4813. 2020-10-08T11:45:17.0546973Z [cluster2] I1008 11:44:26.107523 1 kubernetes.go:180] UpdateFunc in WatchClusters called
  4814. 2020-10-08T11:45:17.0548157Z [cluster2] I1008 11:44:26.107541 1 datastoresyncer.go:312] In reconcileClusterCRD: &types.SubmarinerCluster{ID:"cluster1", Spec:v1.ClusterSpec{ClusterID:"cluster1", ColorCodes:[]string{"blue"}, ServiceCIDR:[]string{"100.91.0.0/16"}, ClusterCIDR:[]string{"10.241.0.0/16"}, GlobalCIDR:[]string{}}}
  4815. 2020-10-08T11:45:17.0549550Z [cluster2] I1008 11:44:26.110456 1 datastoresyncer.go:360] Cluster "cluster1" matched what we received from datastore - not updating
  4816. 2020-10-08T11:45:17.0550175Z [cluster2] I1008 11:44:26.110477 1 kubernetes.go:180] UpdateFunc in WatchClusters called
  4817. 2020-10-08T11:45:17.0550667Z [cluster2] 01[KNL] interface vethwepl72df167 deactivated
  4818. 2020-10-08T11:45:17.0551087Z [cluster2] 05[KNL] interface vethwepl72df167 deleted
  4819. 2020-10-08T11:45:17.0551563Z [cluster2] I1008 11:44:56.100794 1 kubernetes.go:264] UpdateFunc in WatchEndpoints called
  4820. 2020-10-08T11:45:17.6874801Z [lighthouse]$ [cluster2] kubectl -n submariner-operator logs submariner-routeagent-clqmx
  4821. 2020-10-08T11:45:17.6884228Z [lighthouse]$ [cluster2] kubectl -n submariner-operator logs submariner-routeagent-clqmx
  4822. 2020-10-08T11:45:17.6900319Z [lighthouse]$ [cluster2] command kubectl --context=cluster2 -n submariner-operator logs submariner-routeagent-clqmx
  4823. 2020-10-08T11:45:17.6910228Z [lighthouse]$ [cluster2] kubectl --context=cluster2 -n submariner-operator logs submariner-routeagent-clqmx
  4824. 2020-10-08T11:45:18.4638966Z [cluster2] I1008 11:44:56.100843 1 datastoresyncer.go:387] In reconcileEndpointCRD: &types.SubmarinerEndpoint{Spec:v1.EndpointSpec{ClusterID:"cluster1", CableName:"submariner-cable-cluster1-172-17-0-9", Hostname:"cluster1-worker", Subnets:[]string{"100.91.0.0/16", "10.241.0.0/16"}, PrivateIP:"172.17.0.9", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
  4825. 2020-10-08T11:45:18.4643654Z [cluster2] I1008 11:44:56.103516 1 datastoresyncer.go:435] Endpoint "cluster1-submariner-cable-cluster1-172-17-0-9" matched what we received from datastore - not updating
  4826. 2020-10-08T11:45:18.4645423Z [cluster2] I1008 11:44:56.103558 1 kubernetes.go:264] UpdateFunc in WatchEndpoints called
  4827. 2020-10-08T11:45:18.4646385Z [cluster2] I1008 11:44:56.107613 1 kubernetes.go:180] UpdateFunc in WatchClusters called
  4828. 2020-10-08T11:45:18.4648662Z [cluster2] I1008 11:44:56.107650 1 datastoresyncer.go:312] In reconcileClusterCRD: &types.SubmarinerCluster{ID:"cluster1", Spec:v1.ClusterSpec{ClusterID:"cluster1", ColorCodes:[]string{"blue"}, ServiceCIDR:[]string{"100.91.0.0/16"}, ClusterCIDR:[]string{"10.241.0.0/16"}, GlobalCIDR:[]string{}}}
  4829. 2020-10-08T11:45:18.4652067Z [cluster2] I1008 11:44:56.109672 1 datastoresyncer.go:360] Cluster "cluster1" matched what we received from datastore - not updating
  4830. 2020-10-08T11:45:18.4653501Z [cluster2] I1008 11:44:56.109706 1 kubernetes.go:180] UpdateFunc in WatchClusters called
  4831. 2020-10-08T11:45:18.4654193Z [cluster2] ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
  4832. 2020-10-08T11:45:18.4657613Z [cluster2] +++++++++++++++++++++: Logs for Pod submariner-routeagent-clqmx in namespace submariner-operator :++++++++++++++++++++++
  4833. 2020-10-08T11:45:18.4658719Z [cluster2] + trap 'exit 1' SIGTERM SIGINT
  4834. 2020-10-08T11:45:18.4661131Z [cluster2] + SUBMARINER_VERBOSITY=2
  4835. 2020-10-08T11:45:18.4661730Z [cluster2] + '[' false == true ']'
  4836. 2020-10-08T11:45:18.4662195Z [cluster2] + DEBUG=-v=2
  4837. 2020-10-08T11:45:18.4662748Z [cluster2] + for f in iptables-save iptables
  4838. 2020-10-08T11:45:18.4663422Z [cluster2] ++ find_iptables_on_host iptables-save
  4839. 2020-10-08T11:45:18.4664132Z [cluster2] ++ chroot /host test -x /usr/sbin/iptables-save
  4840. 2020-10-08T11:45:18.4664584Z [cluster2] ++ echo /usr/sbin
  4841. 2020-10-08T11:45:18.4664916Z [cluster2] ++ return
  4842. 2020-10-08T11:45:18.4665377Z [cluster2] + location=/usr/sbin
  4843. 2020-10-08T11:45:18.4665926Z [cluster2] + '[' /usr/sbin '!=' unknown ']'
  4844. 2020-10-08T11:45:18.4666797Z [cluster2] + echo 'iptables-save is present on the host at /usr/sbin/iptables-save'
  4845. 2020-10-08T11:45:18.4667627Z [cluster2] + sed 's!@@PATH@@!/usr/sbin!' /usr/sbin/iptables-wrapper.in
  4846. 2020-10-08T11:45:18.4668311Z [cluster2] + for f in iptables-save iptables
  4847. 2020-10-08T11:45:18.4668760Z [cluster2] ++ find_iptables_on_host iptables
  4848. 2020-10-08T11:45:18.4669344Z [cluster2] ++ chroot /host test -x /usr/sbin/iptables
  4849. 2020-10-08T11:45:18.4669730Z [cluster2] ++ echo /usr/sbin
  4850. 2020-10-08T11:45:18.4670036Z [cluster2] ++ return
  4851. 2020-10-08T11:45:18.4670444Z [cluster2] + location=/usr/sbin
  4852. 2020-10-08T11:45:18.4670950Z [cluster2] + '[' /usr/sbin '!=' unknown ']'
  4853. 2020-10-08T11:45:18.4671559Z [cluster2] + echo 'iptables is present on the host at /usr/sbin/iptables'
  4854. 2020-10-08T11:45:18.4672316Z [cluster2] + sed 's!@@PATH@@!/usr/sbin!' /usr/sbin/iptables-wrapper.in
  4855. 2020-10-08T11:45:18.4673117Z [cluster2] iptables-save is present on the host at /usr/sbin/iptables-save
  4856. 2020-10-08T11:45:18.4673715Z [cluster2] iptables is present on the host at /usr/sbin/iptables
  4857. 2020-10-08T11:45:18.4674518Z [cluster2] + exec submariner-route-agent -v=2 -alsologtostderr
  4858. 2020-10-08T11:45:18.4675434Z [cluster2] I1008 11:35:11.991000 1 main.go:47] Starting submariner-route-agent
  4859. 2020-10-08T11:45:18.4676557Z [cluster2] W1008 11:35:11.991280 1 client_config.go:543] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work.
  4860. 2020-10-08T11:45:18.4677648Z [cluster2] I1008 11:35:11.992861 1 route.go:195] Starting Route Controller. ClusterID: cluster2, localClusterCIDR: [10.242.0.0/16], localServiceCIDR: [100.92.0.0/16]
  4861. 2020-10-08T11:45:18.4678515Z [cluster2] I1008 11:35:11.992895 1 route.go:199] Waiting for Endpoint informer caches to sync.
  4862. 2020-10-08T11:45:18.4679383Z [cluster2] I1008 11:35:12.013417 1 route.go:751] Enqueueing sm-route-agent-pod event, ip: 172.17.0.4
  4863. 2020-10-08T11:45:18.4680321Z [cluster2] I1008 11:35:12.013434 1 route.go:751] Enqueueing sm-route-agent-pod event, ip: 172.17.0.5
  4864. 2020-10-08T11:45:18.4680931Z [cluster2] I1008 11:35:12.093790 1 driver.go:37] Interface "lo" has "127.0.0.1" address
  4865. 2020-10-08T11:45:18.4681605Z [cluster2] I1008 11:35:12.093989 1 driver.go:37] Interface "weave" has "10.242.128.0" address
  4866. 2020-10-08T11:45:18.4682245Z [cluster2] I1008 11:35:12.094022 1 driver.go:42] Found CNI Interface "weave" that has IP "10.242.128.0" from ClusterCIDR "10.242.0.0/16"
  4867. 2020-10-08T11:45:18.4683034Z [cluster2] I1008 11:35:12.094112 1 driver.go:60] Successfully configured rp_filter to loose mode(2) on cniInterface "weave"
  4868. 2020-10-08T11:45:18.4684108Z [cluster2] I1008 11:35:12.242298 1 iptables.go:22] Install/ensure SUBMARINER-POSTROUTING chain exists
  4869. 2020-10-08T11:45:18.4685282Z [cluster2] I1008 11:35:12.253751 1 iptables.go:28] Insert SUBMARINER-POSTROUTING rule that has rules for inter-cluster traffic
  4870. 2020-10-08T11:45:18.4686355Z [cluster2] I1008 11:35:12.261443 1 iptables.go:34] Install/ensure SUBMARINER-INPUT chain exists
  4871. 2020-10-08T11:45:18.4687329Z [cluster2] I1008 11:35:12.279546 1 iptables.go:45] Allow VxLAN incoming traffic in SUBMARINER-INPUT Chain
  4872. 2020-10-08T11:45:18.4688491Z [cluster2] I1008 11:35:12.290016 1 iptables.go:53] Insert rule to allow traffic over vx-submariner interface in FORWARDing Chain
  4873. 2020-10-08T11:45:18.4689717Z [cluster2] I1008 11:35:12.298118 1 iptables.go:65] Installing rule for host network to remote cluster communication: -s 240.0.0.0/8 -o vx-submariner -j SNAT --to 10.242.128.0
  4874. 2020-10-08T11:45:18.4690833Z [cluster2] I1008 11:35:12.793562 1 route.go:257] In cluster2, podIP of submariner-route-agent[0] is 172.17.0.4
  4875. 2020-10-08T11:45:18.4691851Z [cluster2] I1008 11:35:12.793579 1 route.go:257] In cluster2, podIP of submariner-route-agent[1] is 172.17.0.5
  4876. 2020-10-08T11:45:18.4693111Z [cluster2] I1008 11:35:12.793584 1 route.go:271] Hostname is "cluster2-worker2" and routeAgentNodeName is "cluster2-worker2"
  4877. 2020-10-08T11:45:18.4694300Z [cluster2] I1008 11:35:12.872994 1 driver.go:87] Successfully annotated node "cluster2-worker2" with cniIfaceIP "10.242.128.0"
  4878. 2020-10-08T11:45:18.4695006Z [cluster2] I1008 11:35:12.873015 1 route.go:288] Route agent workers started
  4879. 2020-10-08T11:45:18.4696173Z [cluster2] I1008 11:35:12.932833 1 route.go:540] Processing submariner-route-agent pod "submariner-routeagent-clqmx" with host IP "172.17.0.4", pod IP "172.17.0.4"
  4880. 2020-10-08T11:45:18.4697494Z [cluster2] I1008 11:35:12.932853 1 route.go:568] Successfully processed submariner-route-agent pod "submariner-routeagent-clqmx"
  4881. 2020-10-08T11:45:18.4698851Z [cluster2] I1008 11:35:12.937924 1 route.go:540] Processing submariner-route-agent pod "submariner-routeagent-gk7px" with host IP "172.17.0.5", pod IP "172.17.0.5"
  4882. 2020-10-08T11:45:18.4700180Z [cluster2] I1008 11:35:12.937941 1 route.go:568] Successfully processed submariner-route-agent pod "submariner-routeagent-gk7px"
  4883. 2020-10-08T11:45:18.4704429Z [cluster2] I1008 11:35:26.072120 1 route.go:736] Enqueueing endpoint for route controller {"metadata":{"name":"cluster2-submariner-cable-cluster2-172-17-0-5","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster2-submariner-cable-cluster2-172-17-0-5","uid":"6a117963-5044-4b34-9df6-57ebb8a4981d","resourceVersion":"1284","generation":1,"creationTimestamp":"2020-10-08T11:35:26Z"},"spec":{"cluster_id":"cluster2","cable_name":"submariner-cable-cluster2-172-17-0-5","hostname":"cluster2-worker","subnets":["100.92.0.0/16","10.242.0.0/16"],"private_ip":"172.17.0.5","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
  4884. 2020-10-08T11:45:18.4708993Z [cluster2] I1008 11:35:26.079324 1 route.go:634] Processing local Endpoint cluster2-submariner-cable-cluster2-172-17-0-5 with IP 172.17.0.5, Host cluster2-worker for this cluster
  4885. 2020-10-08T11:45:18.4710609Z [cluster2] I1008 11:35:26.079346 1 route.go:677] This route agent is running on a non-gateway/non-active node
  4886. 2020-10-08T11:45:18.4711656Z [cluster2] I1008 11:35:26.079354 1 route.go:689] Creating the vxlan interface vx-submariner with gateway node IP 172.17.0.5
  4887. 2020-10-08T11:45:18.4712433Z [cluster2] I1008 11:35:26.101473 1 route.go:866] Reconciling routes to gw: 240.17.0.5
  4888. 2020-10-08T11:45:18.4713097Z [cluster2] I1008 11:35:26.101603 1 route.go:882] Processing route {Ifindex: 18 Dst: 240.0.0.0/8 Src: 240.17.0.4 Gw: <nil> Flags: [] Table: 254}
  4889. 2020-10-08T11:45:18.4713698Z [cluster2] I1008 11:35:26.101628 1 route.go:885] Found nil gw or dst
  4890. 2020-10-08T11:45:18.4715102Z [cluster2] I1008 11:35:26.101687 1 route.go:715] Successfully processed local Endpoint "cluster2-submariner-cable-cluster2-172-17-0-5"
  4891. 2020-10-08T11:45:18.4747380Z [cluster2] I1008 11:35:26.108971 1 route.go:736] Enqueueing endpoint for route controller {"metadata":{"name":"cluster1-submariner-cable-cluster1-172-17-0-9","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster1-submariner-cable-cluster1-172-17-0-9","uid":"60d04732-7b08-4685-b72b-cb2c12702817","resourceVersion":"1285","generation":1,"creationTimestamp":"2020-10-08T11:35:26Z"},"spec":{"cluster_id":"cluster1","cable_name":"submariner-cable-cluster1-172-17-0-9","hostname":"cluster1-worker","subnets":["100.91.0.0/16","10.241.0.0/16"],"private_ip":"172.17.0.9","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
  4892. 2020-10-08T11:45:18.4759601Z [cluster2] I1008 11:35:26.129973 1 route.go:620] Updating iptable rules for remote cluster's Endpoint "cluster1-submariner-cable-cluster1-172-17-0-9" with subnets [100.91.0.0/16 10.241.0.0/16]
  4893. 2020-10-08T11:45:18.4761972Z [cluster2] I1008 11:35:26.146965 1 iptables.go:83] Installing iptables rule for outgoing traffic: -s 10.242.0.0/16 -d 100.91.0.0/16 -j ACCEPT
  4894. 2020-10-08T11:45:18.4764827Z [cluster2] I1008 11:35:26.195693 1 iptables.go:91] Installing iptables rule for incoming traffic: -s 100.91.0.0/16 -d 10.242.0.0/16 -j ACCEPT
  4895. 2020-10-08T11:45:18.4766322Z [cluster2] I1008 11:35:26.226105 1 iptables.go:83] Installing iptables rule for outgoing traffic: -s 10.242.0.0/16 -d 10.241.0.0/16 -j ACCEPT
  4896. 2020-10-08T11:45:18.4770840Z [cluster2] I1008 11:35:26.239823 1 iptables.go:91] Installing iptables rule for incoming traffic: -s 10.241.0.0/16 -d 10.242.0.0/16 -j ACCEPT
  4897. 2020-10-08T11:45:18.4777523Z [cluster2] I1008 11:35:42.000637 1 route.go:736] Enqueueing endpoint for route controller {"metadata":{"name":"cluster1-submariner-cable-cluster1-172-17-0-9","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster1-submariner-cable-cluster1-172-17-0-9","uid":"60d04732-7b08-4685-b72b-cb2c12702817","resourceVersion":"1285","generation":1,"creationTimestamp":"2020-10-08T11:35:26Z"},"spec":{"cluster_id":"cluster1","cable_name":"submariner-cable-cluster1-172-17-0-9","hostname":"cluster1-worker","subnets":["100.91.0.0/16","10.241.0.0/16"],"private_ip":"172.17.0.9","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
  4898. 2020-10-08T11:45:18.4784356Z [cluster2] I1008 11:35:42.000736 1 route.go:736] Enqueueing endpoint for route controller {"metadata":{"name":"cluster2-submariner-cable-cluster2-172-17-0-5","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster2-submariner-cable-cluster2-172-17-0-5","uid":"6a117963-5044-4b34-9df6-57ebb8a4981d","resourceVersion":"1284","generation":1,"creationTimestamp":"2020-10-08T11:35:26Z"},"spec":{"cluster_id":"cluster2","cable_name":"submariner-cable-cluster2-172-17-0-5","hostname":"cluster2-worker","subnets":["100.92.0.0/16","10.242.0.0/16"],"private_ip":"172.17.0.5","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
  4899. 2020-10-08T11:45:18.4788634Z [cluster2] I1008 11:35:42.007640 1 route.go:620] Updating iptable rules for remote cluster's Endpoint "cluster1-submariner-cable-cluster1-172-17-0-9" with subnets [100.91.0.0/16 10.241.0.0/16]
  4900. 2020-10-08T11:45:18.4790750Z [cluster2] I1008 11:35:42.010152 1 route.go:634] Processing local Endpoint cluster2-submariner-cable-cluster2-172-17-0-5 with IP 172.17.0.5, Host cluster2-worker for this cluster
  4901. 2020-10-08T11:45:18.4792492Z [cluster2] I1008 11:35:42.010167 1 route.go:677] This route agent is running on a non-gateway/non-active node
  4902. 2020-10-08T11:45:18.4793699Z [cluster2] I1008 11:35:42.010175 1 route.go:689] Creating the vxlan interface vx-submariner with gateway node IP 172.17.0.5
  4903. 2020-10-08T11:45:18.4794522Z [cluster2] I1008 11:35:42.010327 1 vxlan.go:61] VxLAN interface already exists with same configuration.
  4904. 2020-10-08T11:45:18.4795203Z [cluster2] I1008 11:35:42.015980 1 route.go:866] Reconciling routes to gw: 240.17.0.5
  4905. 2020-10-08T11:45:18.4795860Z [cluster2] I1008 11:35:42.016080 1 route.go:882] Processing route {Ifindex: 18 Dst: 240.0.0.0/8 Src: 240.17.0.4 Gw: <nil> Flags: [] Table: 254}
  4906. 2020-10-08T11:45:18.4796555Z [cluster2] I1008 11:35:42.016099 1 route.go:885] Found nil gw or dst
  4907. 2020-10-08T11:45:18.4798132Z [cluster2] I1008 11:35:42.016209 1 route.go:715] Successfully processed local Endpoint "cluster2-submariner-cable-cluster2-172-17-0-5"
  4908. 2020-10-08T11:45:18.4803181Z [cluster2] I1008 11:36:12.000794 1 route.go:736] Enqueueing endpoint for route controller {"metadata":{"name":"cluster2-submariner-cable-cluster2-172-17-0-5","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster2-submariner-cable-cluster2-172-17-0-5","uid":"6a117963-5044-4b34-9df6-57ebb8a4981d","resourceVersion":"1284","generation":1,"creationTimestamp":"2020-10-08T11:35:26Z"},"spec":{"cluster_id":"cluster2","cable_name":"submariner-cable-cluster2-172-17-0-5","hostname":"cluster2-worker","subnets":["100.92.0.0/16","10.242.0.0/16"],"private_ip":"172.17.0.5","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
  4909. 2020-10-08T11:45:18.4810425Z [cluster2] I1008 11:36:12.000856 1 route.go:736] Enqueueing endpoint for route controller {"metadata":{"name":"cluster1-submariner-cable-cluster1-172-17-0-9","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster1-submariner-cable-cluster1-172-17-0-9","uid":"60d04732-7b08-4685-b72b-cb2c12702817","resourceVersion":"1285","generation":1,"creationTimestamp":"2020-10-08T11:35:26Z"},"spec":{"cluster_id":"cluster1","cable_name":"submariner-cable-cluster1-172-17-0-9","hostname":"cluster1-worker","subnets":["100.91.0.0/16","10.241.0.0/16"],"private_ip":"172.17.0.9","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
  4910. 2020-10-08T11:45:18.4818687Z [cluster2] I1008 11:36:12.008560 1 route.go:634] Processing local Endpoint cluster2-submariner-cable-cluster2-172-17-0-5 with IP 172.17.0.5, Host cluster2-worker for this cluster
  4911. 2020-10-08T11:45:18.4820917Z [cluster2] I1008 11:36:12.008648 1 route.go:677] This route agent is running on a non-gateway/non-active node
  4912. 2020-10-08T11:45:18.4822358Z [cluster2] I1008 11:36:12.008672 1 route.go:689] Creating the vxlan interface vx-submariner with gateway node IP 172.17.0.5
  4913. 2020-10-08T11:45:18.4823286Z [cluster2] I1008 11:36:12.008879 1 vxlan.go:61] VxLAN interface already exists with same configuration.
  4914. 2020-10-08T11:45:18.4824024Z [cluster2] I1008 11:36:12.014879 1 route.go:866] Reconciling routes to gw: 240.17.0.5
  4915. 2020-10-08T11:45:18.4824769Z [cluster2] I1008 11:36:12.015009 1 route.go:882] Processing route {Ifindex: 18 Dst: 10.241.0.0/16 Src: <nil> Gw: 240.17.0.5 Flags: [] Table: 254}
  4916. 2020-10-08T11:45:18.4825604Z [cluster2] I1008 11:36:12.015032 1 route.go:888] Found route {Ifindex: 18 Dst: 10.241.0.0/16 Src: <nil> Gw: 240.17.0.5 Flags: [] Table: 254} with gw 240.17.0.5 already installed
  4917. 2020-10-08T11:45:18.4826435Z [cluster2] I1008 11:36:12.015041 1 route.go:882] Processing route {Ifindex: 18 Dst: 100.91.0.0/16 Src: <nil> Gw: 240.17.0.5 Flags: [] Table: 254}
  4918. 2020-10-08T11:45:18.4827263Z [cluster2] I1008 11:36:12.015050 1 route.go:888] Found route {Ifindex: 18 Dst: 100.91.0.0/16 Src: <nil> Gw: 240.17.0.5 Flags: [] Table: 254} with gw 240.17.0.5 already installed
  4919. 2020-10-08T11:45:18.4828226Z [cluster2] I1008 11:36:12.015059 1 route.go:882] Processing route {Ifindex: 18 Dst: 240.0.0.0/8 Src: 240.17.0.4 Gw: <nil> Flags: [] Table: 254}
  4920. 2020-10-08T11:45:18.4828913Z [cluster2] I1008 11:36:12.015083 1 route.go:885] Found nil gw or dst
  4921. 2020-10-08T11:45:18.4829540Z [cluster2] I1008 11:36:12.015134 1 route.go:924] Found equivalent route, not adding
  4922. 2020-10-08T11:45:18.4830227Z [cluster2] I1008 11:36:12.015143 1 route.go:924] Found equivalent route, not adding
  4923. 2020-10-08T11:45:18.4832450Z [cluster2] I1008 11:36:12.015171 1 route.go:715] Successfully processed local Endpoint "cluster2-submariner-cable-cluster2-172-17-0-5"
  4924. 2020-10-08T11:45:18.4834679Z [cluster2] I1008 11:36:12.022711 1 route.go:620] Updating iptable rules for remote cluster's Endpoint "cluster1-submariner-cable-cluster1-172-17-0-9" with subnets [100.91.0.0/16 10.241.0.0/16]
  4925. 2020-10-08T11:45:18.4840182Z [cluster2] I1008 11:36:42.000911 1 route.go:736] Enqueueing endpoint for route controller {"metadata":{"name":"cluster2-submariner-cable-cluster2-172-17-0-5","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster2-submariner-cable-cluster2-172-17-0-5","uid":"6a117963-5044-4b34-9df6-57ebb8a4981d","resourceVersion":"1284","generation":1,"creationTimestamp":"2020-10-08T11:35:26Z"},"spec":{"cluster_id":"cluster2","cable_name":"submariner-cable-cluster2-172-17-0-5","hostname":"cluster2-worker","subnets":["100.92.0.0/16","10.242.0.0/16"],"private_ip":"172.17.0.5","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
  4926. 2020-10-08T11:45:18.4856663Z [cluster2] I1008 11:36:42.000998 1 route.go:736] Enqueueing endpoint for route controller {"metadata":{"name":"cluster1-submariner-cable-cluster1-172-17-0-9","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster1-submariner-cable-cluster1-172-17-0-9","uid":"60d04732-7b08-4685-b72b-cb2c12702817","resourceVersion":"1285","generation":1,"creationTimestamp":"2020-10-08T11:35:26Z"},"spec":{"cluster_id":"cluster1","cable_name":"submariner-cable-cluster1-172-17-0-9","hostname":"cluster1-worker","subnets":["100.91.0.0/16","10.241.0.0/16"],"private_ip":"172.17.0.9","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
  4927. 2020-10-08T11:45:18.4864125Z [cluster2] I1008 11:36:42.008254 1 route.go:634] Processing local Endpoint cluster2-submariner-cable-cluster2-172-17-0-5 with IP 172.17.0.5, Host cluster2-worker for this cluster
  4928. 2020-10-08T11:45:18.4866640Z [cluster2] I1008 11:36:42.008272 1 route.go:677] This route agent is running on a non-gateway/non-active node
  4929. 2020-10-08T11:45:18.4868048Z [cluster2] I1008 11:36:42.008287 1 route.go:689] Creating the vxlan interface vx-submariner with gateway node IP 172.17.0.5
  4930. 2020-10-08T11:45:18.4869011Z [cluster2] I1008 11:36:42.008457 1 vxlan.go:61] VxLAN interface already exists with same configuration.
  4931. 2020-10-08T11:45:18.4870153Z [cluster2] I1008 11:36:42.011987 1 route.go:866] Reconciling routes to gw: 240.17.0.5
  4932. 2020-10-08T11:45:18.4871078Z [cluster2] I1008 11:36:42.012104 1 route.go:882] Processing route {Ifindex: 18 Dst: 10.241.0.0/16 Src: <nil> Gw: 240.17.0.5 Flags: [] Table: 254}
  4933. 2020-10-08T11:45:18.4872298Z [cluster2] I1008 11:36:42.012137 1 route.go:888] Found route {Ifindex: 18 Dst: 10.241.0.0/16 Src: <nil> Gw: 240.17.0.5 Flags: [] Table: 254} with gw 240.17.0.5 already installed
  4934. 2020-10-08T11:45:18.4873523Z [cluster2] I1008 11:36:42.012148 1 route.go:882] Processing route {Ifindex: 18 Dst: 100.91.0.0/16 Src: <nil> Gw: 240.17.0.5 Flags: [] Table: 254}
  4935. 2020-10-08T11:45:18.4874598Z [cluster2] I1008 11:36:42.012162 1 route.go:888] Found route {Ifindex: 18 Dst: 100.91.0.0/16 Src: <nil> Gw: 240.17.0.5 Flags: [] Table: 254} with gw 240.17.0.5 already installed
  4936. 2020-10-08T11:45:18.4875617Z [cluster2] I1008 11:36:42.012170 1 route.go:882] Processing route {Ifindex: 18 Dst: 240.0.0.0/8 Src: 240.17.0.4 Gw: <nil> Flags: [] Table: 254}
  4937. 2020-10-08T11:45:18.4876791Z [cluster2] I1008 11:36:42.012185 1 route.go:885] Found nil gw or dst
  4938. 2020-10-08T11:45:18.4877298Z [cluster2] I1008 11:36:42.012244 1 route.go:924] Found equivalent route, not adding
  4939. 2020-10-08T11:45:18.4878089Z [cluster2] I1008 11:36:42.012259 1 route.go:924] Found equivalent route, not adding
  4940. 2020-10-08T11:45:18.4880249Z [cluster2] I1008 11:36:42.012287 1 route.go:715] Successfully processed local Endpoint "cluster2-submariner-cable-cluster2-172-17-0-5"
  4941. 2020-10-08T11:45:18.4894906Z [cluster2] I1008 11:36:42.014136 1 route.go:620] Updating iptable rules for remote cluster's Endpoint "cluster1-submariner-cable-cluster1-172-17-0-9" with subnets [100.91.0.0/16 10.241.0.0/16]
  4942. 2020-10-08T11:45:18.4941770Z [cluster2] I1008 11:37:12.001149 1 route.go:736] Enqueueing endpoint for route controller {"metadata":{"name":"cluster2-submariner-cable-cluster2-172-17-0-5","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster2-submariner-cable-cluster2-172-17-0-5","uid":"6a117963-5044-4b34-9df6-57ebb8a4981d","resourceVersion":"1284","generation":1,"creationTimestamp":"2020-10-08T11:35:26Z"},"spec":{"cluster_id":"cluster2","cable_name":"submariner-cable-cluster2-172-17-0-5","hostname":"cluster2-worker","subnets":["100.92.0.0/16","10.242.0.0/16"],"private_ip":"172.17.0.5","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
  4943. 2020-10-08T11:45:18.4947879Z [cluster2] I1008 11:37:12.001291 1 route.go:736] Enqueueing endpoint for route controller {"metadata":{"name":"cluster1-submariner-cable-cluster1-172-17-0-9","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster1-submariner-cable-cluster1-172-17-0-9","uid":"60d04732-7b08-4685-b72b-cb2c12702817","resourceVersion":"1285","generation":1,"creationTimestamp":"2020-10-08T11:35:26Z"},"spec":{"cluster_id":"cluster1","cable_name":"submariner-cable-cluster1-172-17-0-9","hostname":"cluster1-worker","subnets":["100.91.0.0/16","10.241.0.0/16"],"private_ip":"172.17.0.9","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
  4944. 2020-10-08T11:45:18.4951555Z [cluster2] I1008 11:37:12.008648 1 route.go:634] Processing local Endpoint cluster2-submariner-cable-cluster2-172-17-0-5 with IP 172.17.0.5, Host cluster2-worker for this cluster
  4945. 2020-10-08T11:45:18.4952868Z [cluster2] I1008 11:37:12.008668 1 route.go:677] This route agent is running on a non-gateway/non-active node
  4946. 2020-10-08T11:45:18.4953887Z [cluster2] I1008 11:37:12.008677 1 route.go:689] Creating the vxlan interface vx-submariner with gateway node IP 172.17.0.5
  4947. 2020-10-08T11:45:18.4954649Z [cluster2] I1008 11:37:12.008840 1 vxlan.go:61] VxLAN interface already exists with same configuration.
  4948. 2020-10-08T11:45:18.4955281Z [cluster2] I1008 11:37:12.011458 1 route.go:866] Reconciling routes to gw: 240.17.0.5
  4949. 2020-10-08T11:45:18.4955893Z [cluster2] I1008 11:37:12.011558 1 route.go:882] Processing route {Ifindex: 18 Dst: 10.241.0.0/16 Src: <nil> Gw: 240.17.0.5 Flags: [] Table: 254}
  4950. 2020-10-08T11:45:18.4956628Z [cluster2] I1008 11:37:12.011576 1 route.go:888] Found route {Ifindex: 18 Dst: 10.241.0.0/16 Src: <nil> Gw: 240.17.0.5 Flags: [] Table: 254} with gw 240.17.0.5 already installed
  4951. 2020-10-08T11:45:18.4957355Z [cluster2] I1008 11:37:12.011582 1 route.go:882] Processing route {Ifindex: 18 Dst: 100.91.0.0/16 Src: <nil> Gw: 240.17.0.5 Flags: [] Table: 254}
  4952. 2020-10-08T11:45:18.4958075Z [cluster2] I1008 11:37:12.011589 1 route.go:888] Found route {Ifindex: 18 Dst: 100.91.0.0/16 Src: <nil> Gw: 240.17.0.5 Flags: [] Table: 254} with gw 240.17.0.5 already installed
  4953. 2020-10-08T11:45:18.4958792Z [cluster2] I1008 11:37:12.011594 1 route.go:882] Processing route {Ifindex: 18 Dst: 240.0.0.0/8 Src: 240.17.0.4 Gw: <nil> Flags: [] Table: 254}
  4954. 2020-10-08T11:45:18.4959338Z [cluster2] I1008 11:37:12.011599 1 route.go:885] Found nil gw or dst
  4955. 2020-10-08T11:45:18.4959832Z [cluster2] I1008 11:37:12.011637 1 route.go:924] Found equivalent route, not adding
  4956. 2020-10-08T11:45:18.4960475Z [cluster2] I1008 11:37:12.011642 1 route.go:924] Found equivalent route, not adding
  4957. 2020-10-08T11:45:18.4961855Z [cluster2] I1008 11:37:12.011663 1 route.go:715] Successfully processed local Endpoint "cluster2-submariner-cable-cluster2-172-17-0-5"
  4958. 2020-10-08T11:45:18.4963873Z [cluster2] I1008 11:37:12.013092 1 route.go:620] Updating iptable rules for remote cluster's Endpoint "cluster1-submariner-cable-cluster1-172-17-0-9" with subnets [100.91.0.0/16 10.241.0.0/16]
  4959. 2020-10-08T11:45:18.4968886Z [cluster2] I1008 11:37:42.001232 1 route.go:736] Enqueueing endpoint for route controller {"metadata":{"name":"cluster2-submariner-cable-cluster2-172-17-0-5","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster2-submariner-cable-cluster2-172-17-0-5","uid":"6a117963-5044-4b34-9df6-57ebb8a4981d","resourceVersion":"1284","generation":1,"creationTimestamp":"2020-10-08T11:35:26Z"},"spec":{"cluster_id":"cluster2","cable_name":"submariner-cable-cluster2-172-17-0-5","hostname":"cluster2-worker","subnets":["100.92.0.0/16","10.242.0.0/16"],"private_ip":"172.17.0.5","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
  4960. 2020-10-08T11:45:18.4976896Z [cluster2] I1008 11:37:42.001333 1 route.go:736] Enqueueing endpoint for route controller {"metadata":{"name":"cluster1-submariner-cable-cluster1-172-17-0-9","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster1-submariner-cable-cluster1-172-17-0-9","uid":"60d04732-7b08-4685-b72b-cb2c12702817","resourceVersion":"1285","generation":1,"creationTimestamp":"2020-10-08T11:35:26Z"},"spec":{"cluster_id":"cluster1","cable_name":"submariner-cable-cluster1-172-17-0-9","hostname":"cluster1-worker","subnets":["100.91.0.0/16","10.241.0.0/16"],"private_ip":"172.17.0.9","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
  4961. 2020-10-08T11:45:18.4980925Z [cluster2] I1008 11:37:42.008428 1 route.go:634] Processing local Endpoint cluster2-submariner-cable-cluster2-172-17-0-5 with IP 172.17.0.5, Host cluster2-worker for this cluster
  4962. 2020-10-08T11:45:18.4982434Z [cluster2] I1008 11:37:42.008451 1 route.go:677] This route agent is running on a non-gateway/non-active node
  4963. 2020-10-08T11:45:18.4983410Z [cluster2] I1008 11:37:42.008461 1 route.go:689] Creating the vxlan interface vx-submariner with gateway node IP 172.17.0.5
  4964. 2020-10-08T11:45:18.4984169Z [cluster2] I1008 11:37:42.008665 1 vxlan.go:61] VxLAN interface already exists with same configuration.
  4965. 2020-10-08T11:45:18.4984800Z [cluster2] I1008 11:37:42.011791 1 route.go:866] Reconciling routes to gw: 240.17.0.5
  4966. 2020-10-08T11:45:18.4985411Z [cluster2] I1008 11:37:42.011919 1 route.go:882] Processing route {Ifindex: 18 Dst: 10.241.0.0/16 Src: <nil> Gw: 240.17.0.5 Flags: [] Table: 254}
  4967. 2020-10-08T11:45:18.4986145Z [cluster2] I1008 11:37:42.011946 1 route.go:888] Found route {Ifindex: 18 Dst: 10.241.0.0/16 Src: <nil> Gw: 240.17.0.5 Flags: [] Table: 254} with gw 240.17.0.5 already installed
  4968. 2020-10-08T11:45:18.4986953Z [cluster2] I1008 11:37:42.011955 1 route.go:882] Processing route {Ifindex: 18 Dst: 100.91.0.0/16 Src: <nil> Gw: 240.17.0.5 Flags: [] Table: 254}
  4969. 2020-10-08T11:45:18.4987675Z [cluster2] I1008 11:37:42.011964 1 route.go:888] Found route {Ifindex: 18 Dst: 100.91.0.0/16 Src: <nil> Gw: 240.17.0.5 Flags: [] Table: 254} with gw 240.17.0.5 already installed
  4970. 2020-10-08T11:45:18.4988396Z [cluster2] I1008 11:37:42.011971 1 route.go:882] Processing route {Ifindex: 18 Dst: 240.0.0.0/8 Src: 240.17.0.4 Gw: <nil> Flags: [] Table: 254}
  4971. 2020-10-08T11:45:18.4988944Z [cluster2] I1008 11:37:42.011979 1 route.go:885] Found nil gw or dst
  4972. 2020-10-08T11:45:18.4989442Z [cluster2] I1008 11:37:42.012027 1 route.go:924] Found equivalent route, not adding
  4973. 2020-10-08T11:45:18.4990002Z [cluster2] I1008 11:37:42.012034 1 route.go:924] Found equivalent route, not adding
  4974. 2020-10-08T11:45:18.4991818Z [cluster2] I1008 11:37:42.012062 1 route.go:715] Successfully processed local Endpoint "cluster2-submariner-cable-cluster2-172-17-0-5"
  4975. 2020-10-08T11:45:18.5003621Z [cluster2] I1008 11:37:42.013872 1 route.go:620] Updating iptable rules for remote cluster's Endpoint "cluster1-submariner-cable-cluster1-172-17-0-9" with subnets [100.91.0.0/16 10.241.0.0/16]
  4976. 2020-10-08T11:45:18.5008329Z [cluster2] I1008 11:38:12.001503 1 route.go:736] Enqueueing endpoint for route controller {"metadata":{"name":"cluster2-submariner-cable-cluster2-172-17-0-5","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster2-submariner-cable-cluster2-172-17-0-5","uid":"6a117963-5044-4b34-9df6-57ebb8a4981d","resourceVersion":"1284","generation":1,"creationTimestamp":"2020-10-08T11:35:26Z"},"spec":{"cluster_id":"cluster2","cable_name":"submariner-cable-cluster2-172-17-0-5","hostname":"cluster2-worker","subnets":["100.92.0.0/16","10.242.0.0/16"],"private_ip":"172.17.0.5","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
  4977. 2020-10-08T11:45:18.5015468Z [cluster2] I1008 11:38:12.001581 1 route.go:736] Enqueueing endpoint for route controller {"metadata":{"name":"cluster1-submariner-cable-cluster1-172-17-0-9","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster1-submariner-cable-cluster1-172-17-0-9","uid":"60d04732-7b08-4685-b72b-cb2c12702817","resourceVersion":"1285","generation":1,"creationTimestamp":"2020-10-08T11:35:26Z"},"spec":{"cluster_id":"cluster1","cable_name":"submariner-cable-cluster1-172-17-0-9","hostname":"cluster1-worker","subnets":["100.91.0.0/16","10.241.0.0/16"],"private_ip":"172.17.0.9","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
  4978. 2020-10-08T11:45:18.5020277Z [cluster2] I1008 11:38:12.040871 1 route.go:634] Processing local Endpoint cluster2-submariner-cable-cluster2-172-17-0-5 with IP 172.17.0.5, Host cluster2-worker for this cluster
  4979. 2020-10-08T11:45:18.5024955Z [cluster2] I1008 11:38:12.040898 1 route.go:677] This route agent is running on a non-gateway/non-active node
  4980. 2020-10-08T11:45:18.5026034Z [cluster2] I1008 11:38:12.040908 1 route.go:689] Creating the vxlan interface vx-submariner with gateway node IP 172.17.0.5
  4981. 2020-10-08T11:45:18.5026854Z [cluster2] I1008 11:38:12.041092 1 vxlan.go:61] VxLAN interface already exists with same configuration.
  4982. 2020-10-08T11:45:18.5027536Z [cluster2] I1008 11:38:12.045989 1 route.go:866] Reconciling routes to gw: 240.17.0.5
  4983. 2020-10-08T11:45:18.5028195Z [cluster2] I1008 11:38:12.046218 1 route.go:882] Processing route {Ifindex: 18 Dst: 10.241.0.0/16 Src: <nil> Gw: 240.17.0.5 Flags: [] Table: 254}
  4984. 2020-10-08T11:45:18.5029075Z [cluster2] I1008 11:38:12.046289 1 route.go:888] Found route {Ifindex: 18 Dst: 10.241.0.0/16 Src: <nil> Gw: 240.17.0.5 Flags: [] Table: 254} with gw 240.17.0.5 already installed
  4985. 2020-10-08T11:45:18.5029806Z [cluster2] I1008 11:38:12.046315 1 route.go:882] Processing route {Ifindex: 18 Dst: 100.91.0.0/16 Src: <nil> Gw: 240.17.0.5 Flags: [] Table: 254}
  4986. 2020-10-08T11:45:18.5030537Z [cluster2] I1008 11:38:12.046399 1 route.go:888] Found route {Ifindex: 18 Dst: 100.91.0.0/16 Src: <nil> Gw: 240.17.0.5 Flags: [] Table: 254} with gw 240.17.0.5 already installed
  4987. 2020-10-08T11:45:18.5031254Z [cluster2] I1008 11:38:12.046472 1 route.go:882] Processing route {Ifindex: 18 Dst: 240.0.0.0/8 Src: 240.17.0.4 Gw: <nil> Flags: [] Table: 254}
  4988. 2020-10-08T11:45:18.5031808Z [cluster2] I1008 11:38:12.046546 1 route.go:885] Found nil gw or dst
  4989. 2020-10-08T11:45:18.5032307Z [cluster2] I1008 11:38:12.046667 1 route.go:924] Found equivalent route, not adding
  4990. 2020-10-08T11:45:18.5032971Z [cluster2] I1008 11:38:12.046714 1 route.go:924] Found equivalent route, not adding
  4991. 2020-10-08T11:45:18.5034141Z [cluster2] I1008 11:38:12.046785 1 route.go:715] Successfully processed local Endpoint "cluster2-submariner-cable-cluster2-172-17-0-5"
  4992. 2020-10-08T11:45:18.5037123Z [cluster2] I1008 11:38:12.095425 1 route.go:620] Updating iptable rules for remote cluster's Endpoint "cluster1-submariner-cable-cluster1-172-17-0-9" with subnets [100.91.0.0/16 10.241.0.0/16]
  4993. 2020-10-08T11:45:18.5041236Z [cluster2] I1008 11:38:42.001565 1 route.go:736] Enqueueing endpoint for route controller {"metadata":{"name":"cluster1-submariner-cable-cluster1-172-17-0-9","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster1-submariner-cable-cluster1-172-17-0-9","uid":"60d04732-7b08-4685-b72b-cb2c12702817","resourceVersion":"1285","generation":1,"creationTimestamp":"2020-10-08T11:35:26Z"},"spec":{"cluster_id":"cluster1","cable_name":"submariner-cable-cluster1-172-17-0-9","hostname":"cluster1-worker","subnets":["100.91.0.0/16","10.241.0.0/16"],"private_ip":"172.17.0.9","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
  4994. 2020-10-08T11:45:18.5047087Z [cluster2] I1008 11:38:42.001635 1 route.go:736] Enqueueing endpoint for route controller {"metadata":{"name":"cluster2-submariner-cable-cluster2-172-17-0-5","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster2-submariner-cable-cluster2-172-17-0-5","uid":"6a117963-5044-4b34-9df6-57ebb8a4981d","resourceVersion":"1284","generation":1,"creationTimestamp":"2020-10-08T11:35:26Z"},"spec":{"cluster_id":"cluster2","cable_name":"submariner-cable-cluster2-172-17-0-5","hostname":"cluster2-worker","subnets":["100.92.0.0/16","10.242.0.0/16"],"private_ip":"172.17.0.5","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
  4995. 2020-10-08T11:45:18.5050813Z [cluster2] I1008 11:38:42.009849 1 route.go:620] Updating iptable rules for remote cluster's Endpoint "cluster1-submariner-cable-cluster1-172-17-0-9" with subnets [100.91.0.0/16 10.241.0.0/16]
  4996. 2020-10-08T11:45:18.5053604Z [cluster2] I1008 11:38:42.016647 1 route.go:634] Processing local Endpoint cluster2-submariner-cable-cluster2-172-17-0-5 with IP 172.17.0.5, Host cluster2-worker for this cluster
  4997. 2020-10-08T11:45:18.5055040Z [cluster2] I1008 11:38:42.016662 1 route.go:677] This route agent is running on a non-gateway/non-active node
  4998. 2020-10-08T11:45:18.5055893Z [cluster2] I1008 11:38:42.016671 1 route.go:689] Creating the vxlan interface vx-submariner with gateway node IP 172.17.0.5
  4999. 2020-10-08T11:45:18.5056761Z [cluster2] I1008 11:38:42.016825 1 vxlan.go:61] VxLAN interface already exists with same configuration.
  5000. 2020-10-08T11:45:18.5057401Z [cluster2] I1008 11:38:42.024279 1 route.go:866] Reconciling routes to gw: 240.17.0.5
  5001. 2020-10-08T11:45:18.5058010Z [cluster2] I1008 11:38:42.024421 1 route.go:882] Processing route {Ifindex: 18 Dst: 10.241.0.0/16 Src: <nil> Gw: 240.17.0.5 Flags: [] Table: 254}
  5002. 2020-10-08T11:45:18.5058742Z [cluster2] I1008 11:38:42.024445 1 route.go:888] Found route {Ifindex: 18 Dst: 10.241.0.0/16 Src: <nil> Gw: 240.17.0.5 Flags: [] Table: 254} with gw 240.17.0.5 already installed
  5003. 2020-10-08T11:45:18.5059473Z [cluster2] I1008 11:38:42.024455 1 route.go:882] Processing route {Ifindex: 18 Dst: 100.91.0.0/16 Src: <nil> Gw: 240.17.0.5 Flags: [] Table: 254}
  5004. 2020-10-08T11:45:18.5060189Z [cluster2] I1008 11:38:42.024464 1 route.go:888] Found route {Ifindex: 18 Dst: 100.91.0.0/16 Src: <nil> Gw: 240.17.0.5 Flags: [] Table: 254} with gw 240.17.0.5 already installed
  5005. 2020-10-08T11:45:18.5060907Z [cluster2] I1008 11:38:42.024471 1 route.go:882] Processing route {Ifindex: 18 Dst: 240.0.0.0/8 Src: 240.17.0.4 Gw: <nil> Flags: [] Table: 254}
  5006. 2020-10-08T11:45:18.5061458Z [cluster2] I1008 11:38:42.024479 1 route.go:885] Found nil gw or dst
  5007. 2020-10-08T11:45:18.5061953Z [cluster2] I1008 11:38:42.024528 1 route.go:924] Found equivalent route, not adding
  5008. 2020-10-08T11:45:18.5062516Z [cluster2] I1008 11:38:42.024536 1 route.go:924] Found equivalent route, not adding
  5009. 2020-10-08T11:45:18.5063874Z [cluster2] I1008 11:38:42.024564 1 route.go:715] Successfully processed local Endpoint "cluster2-submariner-cable-cluster2-172-17-0-5"
  5010. 2020-10-08T11:45:18.5068844Z [cluster2] I1008 11:39:12.001907 1 route.go:736] Enqueueing endpoint for route controller {"metadata":{"name":"cluster2-submariner-cable-cluster2-172-17-0-5","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster2-submariner-cable-cluster2-172-17-0-5","uid":"6a117963-5044-4b34-9df6-57ebb8a4981d","resourceVersion":"1284","generation":1,"creationTimestamp":"2020-10-08T11:35:26Z"},"spec":{"cluster_id":"cluster2","cable_name":"submariner-cable-cluster2-172-17-0-5","hostname":"cluster2-worker","subnets":["100.92.0.0/16","10.242.0.0/16"],"private_ip":"172.17.0.5","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
  5011. 2020-10-08T11:45:18.5075602Z [cluster2] I1008 11:39:12.001968 1 route.go:736] Enqueueing endpoint for route controller {"metadata":{"name":"cluster1-submariner-cable-cluster1-172-17-0-9","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster1-submariner-cable-cluster1-172-17-0-9","uid":"60d04732-7b08-4685-b72b-cb2c12702817","resourceVersion":"1285","generation":1,"creationTimestamp":"2020-10-08T11:35:26Z"},"spec":{"cluster_id":"cluster1","cable_name":"submariner-cable-cluster1-172-17-0-9","hostname":"cluster1-worker","subnets":["100.91.0.0/16","10.241.0.0/16"],"private_ip":"172.17.0.9","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
  5012. 2020-10-08T11:45:18.5079933Z [cluster2] I1008 11:39:12.009892 1 route.go:634] Processing local Endpoint cluster2-submariner-cable-cluster2-172-17-0-5 with IP 172.17.0.5, Host cluster2-worker for this cluster
  5013. 2020-10-08T11:45:18.5081436Z [cluster2] I1008 11:39:12.009913 1 route.go:677] This route agent is running on a non-gateway/non-active node
  5014. 2020-10-08T11:45:18.5082413Z [cluster2] I1008 11:39:12.009923 1 route.go:689] Creating the vxlan interface vx-submariner with gateway node IP 172.17.0.5
  5015. 2020-10-08T11:45:18.5083181Z [cluster2] I1008 11:39:12.010090 1 vxlan.go:61] VxLAN interface already exists with same configuration.
  5016. 2020-10-08T11:45:18.5083815Z [cluster2] I1008 11:39:12.012654 1 route.go:866] Reconciling routes to gw: 240.17.0.5
  5017. 2020-10-08T11:45:18.5084425Z [cluster2] I1008 11:39:12.012920 1 route.go:882] Processing route {Ifindex: 18 Dst: 10.241.0.0/16 Src: <nil> Gw: 240.17.0.5 Flags: [] Table: 254}
  5018. 2020-10-08T11:45:18.5085156Z [cluster2] I1008 11:39:12.012948 1 route.go:888] Found route {Ifindex: 18 Dst: 10.241.0.0/16 Src: <nil> Gw: 240.17.0.5 Flags: [] Table: 254} with gw 240.17.0.5 already installed
  5019. 2020-10-08T11:45:18.5086035Z [cluster2] I1008 11:39:12.012958 1 route.go:882] Processing route {Ifindex: 18 Dst: 100.91.0.0/16 Src: <nil> Gw: 240.17.0.5 Flags: [] Table: 254}
  5020. 2020-10-08T11:45:18.5086815Z [cluster2] I1008 11:39:12.012967 1 route.go:888] Found route {Ifindex: 18 Dst: 100.91.0.0/16 Src: <nil> Gw: 240.17.0.5 Flags: [] Table: 254} with gw 240.17.0.5 already installed
  5021. 2020-10-08T11:45:18.5087683Z [cluster2] I1008 11:39:12.012979 1 route.go:882] Processing route {Ifindex: 18 Dst: 240.0.0.0/8 Src: 240.17.0.4 Gw: <nil> Flags: [] Table: 254}
  5022. 2020-10-08T11:45:18.5088275Z [cluster2] I1008 11:39:12.012986 1 route.go:885] Found nil gw or dst
  5023. 2020-10-08T11:45:18.5088810Z [cluster2] I1008 11:39:12.013032 1 route.go:924] Found equivalent route, not adding
  5024. 2020-10-08T11:45:18.5089417Z [cluster2] I1008 11:39:12.013040 1 route.go:924] Found equivalent route, not adding
  5025. 2020-10-08T11:45:18.5090891Z [cluster2] I1008 11:39:12.013067 1 route.go:715] Successfully processed local Endpoint "cluster2-submariner-cable-cluster2-172-17-0-5"
  5026. 2020-10-08T11:45:18.5095233Z [cluster2] I1008 11:39:12.014692 1 route.go:620] Updating iptable rules for remote cluster's Endpoint "cluster1-submariner-cable-cluster1-172-17-0-9" with subnets [100.91.0.0/16 10.241.0.0/16]
  5027. 2020-10-08T11:45:18.5099869Z [cluster2] I1008 11:39:42.002009 1 route.go:736] Enqueueing endpoint for route controller {"metadata":{"name":"cluster1-submariner-cable-cluster1-172-17-0-9","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster1-submariner-cable-cluster1-172-17-0-9","uid":"60d04732-7b08-4685-b72b-cb2c12702817","resourceVersion":"1285","generation":1,"creationTimestamp":"2020-10-08T11:35:26Z"},"spec":{"cluster_id":"cluster1","cable_name":"submariner-cable-cluster1-172-17-0-9","hostname":"cluster1-worker","subnets":["100.91.0.0/16","10.241.0.0/16"],"private_ip":"172.17.0.9","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
  5028. 2020-10-08T11:45:18.5106197Z [cluster2] I1008 11:39:42.002157 1 route.go:736] Enqueueing endpoint for route controller {"metadata":{"name":"cluster2-submariner-cable-cluster2-172-17-0-5","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster2-submariner-cable-cluster2-172-17-0-5","uid":"6a117963-5044-4b34-9df6-57ebb8a4981d","resourceVersion":"1284","generation":1,"creationTimestamp":"2020-10-08T11:35:26Z"},"spec":{"cluster_id":"cluster2","cable_name":"submariner-cable-cluster2-172-17-0-5","hostname":"cluster2-worker","subnets":["100.92.0.0/16","10.242.0.0/16"],"private_ip":"172.17.0.5","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
  5029. 2020-10-08T11:45:18.5110465Z [cluster2] I1008 11:39:42.009373 1 route.go:620] Updating iptable rules for remote cluster's Endpoint "cluster1-submariner-cable-cluster1-172-17-0-9" with subnets [100.91.0.0/16 10.241.0.0/16]
  5030. 2020-10-08T11:45:18.5112578Z [cluster2] I1008 11:39:42.010856 1 route.go:634] Processing local Endpoint cluster2-submariner-cable-cluster2-172-17-0-5 with IP 172.17.0.5, Host cluster2-worker for this cluster
  5031. 2020-10-08T11:45:18.5114079Z [cluster2] I1008 11:39:42.010877 1 route.go:677] This route agent is running on a non-gateway/non-active node
  5032. 2020-10-08T11:45:18.5115130Z [cluster2] I1008 11:39:42.010885 1 route.go:689] Creating the vxlan interface vx-submariner with gateway node IP 172.17.0.5
  5033. 2020-10-08T11:45:18.5115896Z [cluster2] I1008 11:39:42.011109 1 vxlan.go:61] VxLAN interface already exists with same configuration.
  5034. 2020-10-08T11:45:18.5116527Z [cluster2] I1008 11:39:42.014180 1 route.go:866] Reconciling routes to gw: 240.17.0.5
  5035. 2020-10-08T11:45:18.5117936Z [cluster2] I1008 11:39:42.014328 1 route.go:882] Processing route {Ifindex: 18 Dst: 10.241.0.0/16 Src: <nil> Gw: 240.17.0.5 Flags: [] Table: 254}
  5036. 2020-10-08T11:45:18.5119347Z [cluster2] I1008 11:39:42.014356 1 route.go:888] Found route {Ifindex: 18 Dst: 10.241.0.0/16 Src: <nil> Gw: 240.17.0.5 Flags: [] Table: 254} with gw 240.17.0.5 already installed
  5037. 2020-10-08T11:45:18.5120079Z [cluster2] I1008 11:39:42.014366 1 route.go:882] Processing route {Ifindex: 18 Dst: 100.91.0.0/16 Src: <nil> Gw: 240.17.0.5 Flags: [] Table: 254}
  5038. 2020-10-08T11:45:18.5120799Z [cluster2] I1008 11:39:42.014380 1 route.go:888] Found route {Ifindex: 18 Dst: 100.91.0.0/16 Src: <nil> Gw: 240.17.0.5 Flags: [] Table: 254} with gw 240.17.0.5 already installed
  5039. 2020-10-08T11:45:18.5121527Z [cluster2] I1008 11:39:42.014387 1 route.go:882] Processing route {Ifindex: 18 Dst: 240.0.0.0/8 Src: 240.17.0.4 Gw: <nil> Flags: [] Table: 254}
  5040. 2020-10-08T11:45:18.5122080Z [cluster2] I1008 11:39:42.014401 1 route.go:885] Found nil gw or dst
  5041. 2020-10-08T11:45:18.5124405Z [cluster2] I1008 11:39:42.014448 1 route.go:924] Found equivalent route, not adding
  5042. 2020-10-08T11:45:18.5125046Z [cluster2] I1008 11:39:42.014461 1 route.go:924] Found equivalent route, not adding
  5043. 2020-10-08T11:45:18.5127624Z [cluster2] I1008 11:39:42.014488 1 route.go:715] Successfully processed local Endpoint "cluster2-submariner-cable-cluster2-172-17-0-5"
  5044. 2020-10-08T11:45:18.5132876Z [cluster2] I1008 11:40:12.002119 1 route.go:736] Enqueueing endpoint for route controller {"metadata":{"name":"cluster2-submariner-cable-cluster2-172-17-0-5","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster2-submariner-cable-cluster2-172-17-0-5","uid":"6a117963-5044-4b34-9df6-57ebb8a4981d","resourceVersion":"1284","generation":1,"creationTimestamp":"2020-10-08T11:35:26Z"},"spec":{"cluster_id":"cluster2","cable_name":"submariner-cable-cluster2-172-17-0-5","hostname":"cluster2-worker","subnets":["100.92.0.0/16","10.242.0.0/16"],"private_ip":"172.17.0.5","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
  5045. 2020-10-08T11:45:18.5140938Z [cluster2] I1008 11:40:12.002202 1 route.go:736] Enqueueing endpoint for route controller {"metadata":{"name":"cluster1-submariner-cable-cluster1-172-17-0-9","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster1-submariner-cable-cluster1-172-17-0-9","uid":"60d04732-7b08-4685-b72b-cb2c12702817","resourceVersion":"1285","generation":1,"creationTimestamp":"2020-10-08T11:35:26Z"},"spec":{"cluster_id":"cluster1","cable_name":"submariner-cable-cluster1-172-17-0-9","hostname":"cluster1-worker","subnets":["100.91.0.0/16","10.241.0.0/16"],"private_ip":"172.17.0.9","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
  5046. 2020-10-08T11:45:18.5145208Z [cluster2] I1008 11:40:12.011773 1 route.go:634] Processing local Endpoint cluster2-submariner-cable-cluster2-172-17-0-5 with IP 172.17.0.5, Host cluster2-worker for this cluster
  5047. 2020-10-08T11:45:18.5146710Z [cluster2] I1008 11:40:12.011794 1 route.go:677] This route agent is running on a non-gateway/non-active node
  5048. 2020-10-08T11:45:18.5147687Z [cluster2] I1008 11:40:12.011804 1 route.go:689] Creating the vxlan interface vx-submariner with gateway node IP 172.17.0.5
  5049. 2020-10-08T11:45:18.5148444Z [cluster2] I1008 11:40:12.011984 1 vxlan.go:61] VxLAN interface already exists with same configuration.
  5050. 2020-10-08T11:45:18.5149073Z [cluster2] I1008 11:40:12.015132 1 route.go:866] Reconciling routes to gw: 240.17.0.5
  5051. 2020-10-08T11:45:18.5149686Z [cluster2] I1008 11:40:12.015303 1 route.go:882] Processing route {Ifindex: 18 Dst: 10.241.0.0/16 Src: <nil> Gw: 240.17.0.5 Flags: [] Table: 254}
  5052. 2020-10-08T11:45:18.5150421Z [cluster2] I1008 11:40:12.015346 1 route.go:888] Found route {Ifindex: 18 Dst: 10.241.0.0/16 Src: <nil> Gw: 240.17.0.5 Flags: [] Table: 254} with gw 240.17.0.5 already installed
  5053. 2020-10-08T11:45:18.5151143Z [cluster2] I1008 11:40:12.015363 1 route.go:882] Processing route {Ifindex: 18 Dst: 100.91.0.0/16 Src: <nil> Gw: 240.17.0.5 Flags: [] Table: 254}
  5054. 2020-10-08T11:45:18.5151859Z [cluster2] I1008 11:40:12.015372 1 route.go:888] Found route {Ifindex: 18 Dst: 100.91.0.0/16 Src: <nil> Gw: 240.17.0.5 Flags: [] Table: 254} with gw 240.17.0.5 already installed
  5055. 2020-10-08T11:45:18.5152577Z [cluster2] I1008 11:40:12.015385 1 route.go:882] Processing route {Ifindex: 18 Dst: 240.0.0.0/8 Src: 240.17.0.4 Gw: <nil> Flags: [] Table: 254}
  5056. 2020-10-08T11:45:18.5153127Z [cluster2] I1008 11:40:12.015393 1 route.go:885] Found nil gw or dst
  5057. 2020-10-08T11:45:18.5153628Z [cluster2] I1008 11:40:12.015441 1 route.go:924] Found equivalent route, not adding
  5058. 2020-10-08T11:45:18.5154192Z [cluster2] I1008 11:40:12.015454 1 route.go:924] Found equivalent route, not adding
  5059. 2020-10-08T11:45:18.5155546Z [cluster2] I1008 11:40:12.015483 1 route.go:715] Successfully processed local Endpoint "cluster2-submariner-cable-cluster2-172-17-0-5"
  5060. 2020-10-08T11:45:18.5157561Z [cluster2] I1008 11:40:12.017120 1 route.go:620] Updating iptable rules for remote cluster's Endpoint "cluster1-submariner-cable-cluster1-172-17-0-9" with subnets [100.91.0.0/16 10.241.0.0/16]
  5061. 2020-10-08T11:45:18.5162704Z [cluster2] I1008 11:40:42.002238 1 route.go:736] Enqueueing endpoint for route controller {"metadata":{"name":"cluster2-submariner-cable-cluster2-172-17-0-5","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster2-submariner-cable-cluster2-172-17-0-5","uid":"6a117963-5044-4b34-9df6-57ebb8a4981d","resourceVersion":"1284","generation":1,"creationTimestamp":"2020-10-08T11:35:26Z"},"spec":{"cluster_id":"cluster2","cable_name":"submariner-cable-cluster2-172-17-0-5","hostname":"cluster2-worker","subnets":["100.92.0.0/16","10.242.0.0/16"],"private_ip":"172.17.0.5","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
  5062. 2020-10-08T11:45:18.5170726Z [cluster2] I1008 11:40:42.002311 1 route.go:736] Enqueueing endpoint for route controller {"metadata":{"name":"cluster1-submariner-cable-cluster1-172-17-0-9","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster1-submariner-cable-cluster1-172-17-0-9","uid":"60d04732-7b08-4685-b72b-cb2c12702817","resourceVersion":"1285","generation":1,"creationTimestamp":"2020-10-08T11:35:26Z"},"spec":{"cluster_id":"cluster1","cable_name":"submariner-cable-cluster1-172-17-0-9","hostname":"cluster1-worker","subnets":["100.91.0.0/16","10.241.0.0/16"],"private_ip":"172.17.0.9","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
  5063. 2020-10-08T11:45:18.5175584Z [cluster2] I1008 11:40:42.009403 1 route.go:634] Processing local Endpoint cluster2-submariner-cable-cluster2-172-17-0-5 with IP 172.17.0.5, Host cluster2-worker for this cluster
  5064. 2020-10-08T11:45:18.5177299Z [cluster2] I1008 11:40:42.009434 1 route.go:677] This route agent is running on a non-gateway/non-active node
  5065. 2020-10-08T11:45:18.5178355Z [cluster2] I1008 11:40:42.009443 1 route.go:689] Creating the vxlan interface vx-submariner with gateway node IP 172.17.0.5
  5066. 2020-10-08T11:45:18.5179165Z [cluster2] I1008 11:40:42.009593 1 vxlan.go:61] VxLAN interface already exists with same configuration.
  5067. 2020-10-08T11:45:18.5179875Z [cluster2] I1008 11:40:42.014208 1 route.go:866] Reconciling routes to gw: 240.17.0.5
  5068. 2020-10-08T11:45:18.5180532Z [cluster2] I1008 11:40:42.014325 1 route.go:882] Processing route {Ifindex: 18 Dst: 10.241.0.0/16 Src: <nil> Gw: 240.17.0.5 Flags: [] Table: 254}
  5069. 2020-10-08T11:45:18.5181322Z [cluster2] I1008 11:40:42.014349 1 route.go:888] Found route {Ifindex: 18 Dst: 10.241.0.0/16 Src: <nil> Gw: 240.17.0.5 Flags: [] Table: 254} with gw 240.17.0.5 already installed
  5070. 2020-10-08T11:45:18.5182094Z [cluster2] I1008 11:40:42.014358 1 route.go:882] Processing route {Ifindex: 18 Dst: 100.91.0.0/16 Src: <nil> Gw: 240.17.0.5 Flags: [] Table: 254}
  5071. 2020-10-08T11:45:18.5183035Z [cluster2] I1008 11:40:42.014370 1 route.go:888] Found route {Ifindex: 18 Dst: 100.91.0.0/16 Src: <nil> Gw: 240.17.0.5 Flags: [] Table: 254} with gw 240.17.0.5 already installed
  5072. 2020-10-08T11:45:18.5183822Z [cluster2] I1008 11:40:42.014384 1 route.go:882] Processing route {Ifindex: 18 Dst: 240.0.0.0/8 Src: 240.17.0.4 Gw: <nil> Flags: [] Table: 254}
  5073. 2020-10-08T11:45:18.5184412Z [cluster2] I1008 11:40:42.014391 1 route.go:885] Found nil gw or dst
  5074. 2020-10-08T11:45:18.5184949Z [cluster2] I1008 11:40:42.014436 1 route.go:924] Found equivalent route, not adding
  5075. 2020-10-08T11:45:18.5185557Z [cluster2] I1008 11:40:42.014448 1 route.go:924] Found equivalent route, not adding
  5076. 2020-10-08T11:45:18.5187057Z [cluster2] I1008 11:40:42.014474 1 route.go:715] Successfully processed local Endpoint "cluster2-submariner-cable-cluster2-172-17-0-5"
  5077. 2020-10-08T11:45:18.5189268Z [cluster2] I1008 11:40:42.016083 1 route.go:620] Updating iptable rules for remote cluster's Endpoint "cluster1-submariner-cable-cluster1-172-17-0-9" with subnets [100.91.0.0/16 10.241.0.0/16]
  5078. 2020-10-08T11:45:18.5197088Z [cluster2] I1008 11:41:12.002332 1 route.go:736] Enqueueing endpoint for route controller {"metadata":{"name":"cluster2-submariner-cable-cluster2-172-17-0-5","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster2-submariner-cable-cluster2-172-17-0-5","uid":"6a117963-5044-4b34-9df6-57ebb8a4981d","resourceVersion":"1284","generation":1,"creationTimestamp":"2020-10-08T11:35:26Z"},"spec":{"cluster_id":"cluster2","cable_name":"submariner-cable-cluster2-172-17-0-5","hostname":"cluster2-worker","subnets":["100.92.0.0/16","10.242.0.0/16"],"private_ip":"172.17.0.5","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
  5079. 2020-10-08T11:45:18.5203000Z [cluster2] I1008 11:41:12.002393 1 route.go:736] Enqueueing endpoint for route controller {"metadata":{"name":"cluster1-submariner-cable-cluster1-172-17-0-9","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster1-submariner-cable-cluster1-172-17-0-9","uid":"60d04732-7b08-4685-b72b-cb2c12702817","resourceVersion":"1285","generation":1,"creationTimestamp":"2020-10-08T11:35:26Z"},"spec":{"cluster_id":"cluster1","cable_name":"submariner-cable-cluster1-172-17-0-9","hostname":"cluster1-worker","subnets":["100.91.0.0/16","10.241.0.0/16"],"private_ip":"172.17.0.9","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
  5080. 2020-10-08T11:45:18.5207332Z [cluster2] I1008 11:41:12.009401 1 route.go:634] Processing local Endpoint cluster2-submariner-cable-cluster2-172-17-0-5 with IP 172.17.0.5, Host cluster2-worker for this cluster
  5081. 2020-10-08T11:45:18.5208931Z [cluster2] I1008 11:41:12.009420 1 route.go:677] This route agent is running on a non-gateway/non-active node
  5082. 2020-10-08T11:45:18.5209919Z [cluster2] I1008 11:41:12.009430 1 route.go:689] Creating the vxlan interface vx-submariner with gateway node IP 172.17.0.5
  5083. 2020-10-08T11:45:18.5210670Z [cluster2] I1008 11:41:12.009590 1 vxlan.go:61] VxLAN interface already exists with same configuration.
  5084. 2020-10-08T11:45:18.5211301Z [cluster2] I1008 11:41:12.012853 1 route.go:866] Reconciling routes to gw: 240.17.0.5
  5085. 2020-10-08T11:45:18.5211984Z [cluster2] I1008 11:41:12.012966 1 route.go:882] Processing route {Ifindex: 18 Dst: 10.241.0.0/16 Src: <nil> Gw: 240.17.0.5 Flags: [] Table: 254}
  5086. 2020-10-08T11:45:18.5213733Z [cluster2] I1008 11:41:12.012991 1 route.go:888] Found route {Ifindex: 18 Dst: 10.241.0.0/16 Src: <nil> Gw: 240.17.0.5 Flags: [] Table: 254} with gw 240.17.0.5 already installed
  5087. 2020-10-08T11:45:18.5214471Z [cluster2] I1008 11:41:12.013000 1 route.go:882] Processing route {Ifindex: 18 Dst: 100.91.0.0/16 Src: <nil> Gw: 240.17.0.5 Flags: [] Table: 254}
  5088. 2020-10-08T11:45:18.5215194Z [cluster2] I1008 11:41:12.013008 1 route.go:888] Found route {Ifindex: 18 Dst: 100.91.0.0/16 Src: <nil> Gw: 240.17.0.5 Flags: [] Table: 254} with gw 240.17.0.5 already installed
  5089. 2020-10-08T11:45:18.5215912Z [cluster2] I1008 11:41:12.013015 1 route.go:882] Processing route {Ifindex: 18 Dst: 240.0.0.0/8 Src: 240.17.0.4 Gw: <nil> Flags: [] Table: 254}
  5090. 2020-10-08T11:45:18.5216460Z [cluster2] I1008 11:41:12.013023 1 route.go:885] Found nil gw or dst
  5091. 2020-10-08T11:45:18.5216956Z [cluster2] I1008 11:41:12.013072 1 route.go:924] Found equivalent route, not adding
  5092. 2020-10-08T11:45:18.5217520Z [cluster2] I1008 11:41:12.013079 1 route.go:924] Found equivalent route, not adding
  5093. 2020-10-08T11:45:18.5218906Z [cluster2] I1008 11:41:12.013105 1 route.go:715] Successfully processed local Endpoint "cluster2-submariner-cable-cluster2-172-17-0-5"
  5094. 2020-10-08T11:45:18.5220934Z [cluster2] I1008 11:41:12.015107 1 route.go:620] Updating iptable rules for remote cluster's Endpoint "cluster1-submariner-cable-cluster1-172-17-0-9" with subnets [100.91.0.0/16 10.241.0.0/16]
  5095. 2020-10-08T11:45:18.5225603Z [cluster2] I1008 11:41:42.002448 1 route.go:736] Enqueueing endpoint for route controller {"metadata":{"name":"cluster2-submariner-cable-cluster2-172-17-0-5","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster2-submariner-cable-cluster2-172-17-0-5","uid":"6a117963-5044-4b34-9df6-57ebb8a4981d","resourceVersion":"1284","generation":1,"creationTimestamp":"2020-10-08T11:35:26Z"},"spec":{"cluster_id":"cluster2","cable_name":"submariner-cable-cluster2-172-17-0-5","hostname":"cluster2-worker","subnets":["100.92.0.0/16","10.242.0.0/16"],"private_ip":"172.17.0.5","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
  5096. 2020-10-08T11:45:18.5232780Z [cluster2] I1008 11:41:42.002537 1 route.go:736] Enqueueing endpoint for route controller {"metadata":{"name":"cluster1-submariner-cable-cluster1-172-17-0-9","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster1-submariner-cable-cluster1-172-17-0-9","uid":"60d04732-7b08-4685-b72b-cb2c12702817","resourceVersion":"1285","generation":1,"creationTimestamp":"2020-10-08T11:35:26Z"},"spec":{"cluster_id":"cluster1","cable_name":"submariner-cable-cluster1-172-17-0-9","hostname":"cluster1-worker","subnets":["100.91.0.0/16","10.241.0.0/16"],"private_ip":"172.17.0.9","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
  5097. 2020-10-08T11:45:18.5236492Z [cluster2] I1008 11:41:42.009346 1 route.go:634] Processing local Endpoint cluster2-submariner-cable-cluster2-172-17-0-5 with IP 172.17.0.5, Host cluster2-worker for this cluster
  5098. 2020-10-08T11:45:18.5237805Z [cluster2] I1008 11:41:42.009365 1 route.go:677] This route agent is running on a non-gateway/non-active node
  5099. 2020-10-08T11:45:18.5239368Z [cluster2] I1008 11:41:42.009373 1 route.go:689] Creating the vxlan interface vx-submariner with gateway node IP 172.17.0.5
  5100. 2020-10-08T11:45:18.5240021Z [cluster2] I1008 11:41:42.009550 1 vxlan.go:61] VxLAN interface already exists with same configuration.
  5101. 2020-10-08T11:45:18.5240570Z [cluster2] I1008 11:41:42.012876 1 route.go:866] Reconciling routes to gw: 240.17.0.5
  5102. 2020-10-08T11:45:18.5241451Z [cluster2] I1008 11:41:42.013006 1 route.go:882] Processing route {Ifindex: 18 Dst: 10.241.0.0/16 Src: <nil> Gw: 240.17.0.5 Flags: [] Table: 254}
  5103. 2020-10-08T11:45:18.5242209Z [cluster2] I1008 11:41:42.013033 1 route.go:888] Found route {Ifindex: 18 Dst: 10.241.0.0/16 Src: <nil> Gw: 240.17.0.5 Flags: [] Table: 254} with gw 240.17.0.5 already installed
  5104. 2020-10-08T11:45:18.5242849Z [cluster2] I1008 11:41:42.013042 1 route.go:882] Processing route {Ifindex: 18 Dst: 100.91.0.0/16 Src: <nil> Gw: 240.17.0.5 Flags: [] Table: 254}
  5105. 2020-10-08T11:45:18.5243487Z [cluster2] I1008 11:41:42.013051 1 route.go:888] Found route {Ifindex: 18 Dst: 100.91.0.0/16 Src: <nil> Gw: 240.17.0.5 Flags: [] Table: 254} with gw 240.17.0.5 already installed
  5106. 2020-10-08T11:45:18.5244116Z [cluster2] I1008 11:41:42.013058 1 route.go:882] Processing route {Ifindex: 18 Dst: 240.0.0.0/8 Src: 240.17.0.4 Gw: <nil> Flags: [] Table: 254}
  5107. 2020-10-08T11:45:18.5244597Z [cluster2] I1008 11:41:42.013066 1 route.go:885] Found nil gw or dst
  5108. 2020-10-08T11:45:18.5245032Z [cluster2] I1008 11:41:42.013115 1 route.go:924] Found equivalent route, not adding
  5109. 2020-10-08T11:45:18.5245525Z [cluster2] I1008 11:41:42.013124 1 route.go:924] Found equivalent route, not adding
  5110. 2020-10-08T11:45:18.5246744Z [cluster2] I1008 11:41:42.013153 1 route.go:715] Successfully processed local Endpoint "cluster2-submariner-cable-cluster2-172-17-0-5"
  5111. 2020-10-08T11:45:18.5248515Z [cluster2] I1008 11:41:42.015336 1 route.go:620] Updating iptable rules for remote cluster's Endpoint "cluster1-submariner-cable-cluster1-172-17-0-9" with subnets [100.91.0.0/16 10.241.0.0/16]
  5112. 2020-10-08T11:45:18.5252637Z [cluster2] I1008 11:42:12.002577 1 route.go:736] Enqueueing endpoint for route controller {"metadata":{"name":"cluster1-submariner-cable-cluster1-172-17-0-9","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster1-submariner-cable-cluster1-172-17-0-9","uid":"60d04732-7b08-4685-b72b-cb2c12702817","resourceVersion":"1285","generation":1,"creationTimestamp":"2020-10-08T11:35:26Z"},"spec":{"cluster_id":"cluster1","cable_name":"submariner-cable-cluster1-172-17-0-9","hostname":"cluster1-worker","subnets":["100.91.0.0/16","10.241.0.0/16"],"private_ip":"172.17.0.9","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
  5113. 2020-10-08T11:45:18.5258590Z [cluster2] I1008 11:42:12.002655 1 route.go:736] Enqueueing endpoint for route controller {"metadata":{"name":"cluster2-submariner-cable-cluster2-172-17-0-5","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster2-submariner-cable-cluster2-172-17-0-5","uid":"6a117963-5044-4b34-9df6-57ebb8a4981d","resourceVersion":"1284","generation":1,"creationTimestamp":"2020-10-08T11:35:26Z"},"spec":{"cluster_id":"cluster2","cable_name":"submariner-cable-cluster2-172-17-0-5","hostname":"cluster2-worker","subnets":["100.92.0.0/16","10.242.0.0/16"],"private_ip":"172.17.0.5","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
  5114. 2020-10-08T11:45:18.5272317Z [cluster2] I1008 11:42:12.010083 1 route.go:620] Updating iptable rules for remote cluster's Endpoint "cluster1-submariner-cable-cluster1-172-17-0-9" with subnets [100.91.0.0/16 10.241.0.0/16]
  5115. 2020-10-08T11:45:18.5274446Z [cluster2] I1008 11:42:12.012086 1 route.go:634] Processing local Endpoint cluster2-submariner-cable-cluster2-172-17-0-5 with IP 172.17.0.5, Host cluster2-worker for this cluster
  5116. 2020-10-08T11:45:18.5275951Z [cluster2] I1008 11:42:12.012102 1 route.go:677] This route agent is running on a non-gateway/non-active node
  5117. 2020-10-08T11:45:18.5278686Z [cluster2] I1008 11:42:12.012110 1 route.go:689] Creating the vxlan interface vx-submariner with gateway node IP 172.17.0.5
  5118. 2020-10-08T11:45:18.5279456Z [cluster2] I1008 11:42:12.012323 1 vxlan.go:61] VxLAN interface already exists with same configuration.
  5119. 2020-10-08T11:45:18.5280123Z [cluster2] I1008 11:42:12.015703 1 route.go:866] Reconciling routes to gw: 240.17.0.5
  5120. 2020-10-08T11:45:18.5280728Z [cluster2] I1008 11:42:12.015834 1 route.go:882] Processing route {Ifindex: 18 Dst: 10.241.0.0/16 Src: <nil> Gw: 240.17.0.5 Flags: [] Table: 254}
  5121. 2020-10-08T11:45:18.5281453Z [cluster2] I1008 11:42:12.015911 1 route.go:888] Found route {Ifindex: 18 Dst: 10.241.0.0/16 Src: <nil> Gw: 240.17.0.5 Flags: [] Table: 254} with gw 240.17.0.5 already installed
  5122. 2020-10-08T11:45:18.5282174Z [cluster2] I1008 11:42:12.015944 1 route.go:882] Processing route {Ifindex: 18 Dst: 100.91.0.0/16 Src: <nil> Gw: 240.17.0.5 Flags: [] Table: 254}
  5123. 2020-10-08T11:45:18.5282909Z [cluster2] I1008 11:42:12.015978 1 route.go:888] Found route {Ifindex: 18 Dst: 100.91.0.0/16 Src: <nil> Gw: 240.17.0.5 Flags: [] Table: 254} with gw 240.17.0.5 already installed
  5124. 2020-10-08T11:45:18.5283626Z [cluster2] I1008 11:42:12.016018 1 route.go:882] Processing route {Ifindex: 18 Dst: 240.0.0.0/8 Src: 240.17.0.4 Gw: <nil> Flags: [] Table: 254}
  5125. 2020-10-08T11:45:18.5284513Z [cluster2] I1008 11:42:12.016055 1 route.go:885] Found nil gw or dst
  5126. 2020-10-08T11:45:18.5285061Z [cluster2] I1008 11:42:12.016139 1 route.go:924] Found equivalent route, not adding
  5127. 2020-10-08T11:45:18.5285668Z [cluster2] I1008 11:42:12.016154 1 route.go:924] Found equivalent route, not adding
  5128. 2020-10-08T11:45:18.5287210Z [cluster2] I1008 11:42:12.016183 1 route.go:715] Successfully processed local Endpoint "cluster2-submariner-cable-cluster2-172-17-0-5"
  5129. 2020-10-08T11:45:18.5294296Z [cluster2] I1008 11:42:42.002701 1 route.go:736] Enqueueing endpoint for route controller {"metadata":{"name":"cluster2-submariner-cable-cluster2-172-17-0-5","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster2-submariner-cable-cluster2-172-17-0-5","uid":"6a117963-5044-4b34-9df6-57ebb8a4981d","resourceVersion":"1284","generation":1,"creationTimestamp":"2020-10-08T11:35:26Z"},"spec":{"cluster_id":"cluster2","cable_name":"submariner-cable-cluster2-172-17-0-5","hostname":"cluster2-worker","subnets":["100.92.0.0/16","10.242.0.0/16"],"private_ip":"172.17.0.5","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
  5130. 2020-10-08T11:45:18.5301959Z [cluster2] I1008 11:42:42.002775 1 route.go:736] Enqueueing endpoint for route controller {"metadata":{"name":"cluster1-submariner-cable-cluster1-172-17-0-9","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster1-submariner-cable-cluster1-172-17-0-9","uid":"60d04732-7b08-4685-b72b-cb2c12702817","resourceVersion":"1285","generation":1,"creationTimestamp":"2020-10-08T11:35:26Z"},"spec":{"cluster_id":"cluster1","cable_name":"submariner-cable-cluster1-172-17-0-9","hostname":"cluster1-worker","subnets":["100.91.0.0/16","10.241.0.0/16"],"private_ip":"172.17.0.9","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
  5131. 2020-10-08T11:45:18.5309980Z [cluster2] I1008 11:42:42.009857 1 route.go:634] Processing local Endpoint cluster2-submariner-cable-cluster2-172-17-0-5 with IP 172.17.0.5, Host cluster2-worker for this cluster
  5132. 2020-10-08T11:45:18.5312815Z [cluster2] I1008 11:42:42.009877 1 route.go:677] This route agent is running on a non-gateway/non-active node
  5133. 2020-10-08T11:45:18.5313808Z [cluster2] I1008 11:42:42.009885 1 route.go:689] Creating the vxlan interface vx-submariner with gateway node IP 172.17.0.5
  5134. 2020-10-08T11:45:18.5314566Z [cluster2] I1008 11:42:42.010050 1 vxlan.go:61] VxLAN interface already exists with same configuration.
  5135. 2020-10-08T11:45:18.5315385Z [cluster2] I1008 11:42:42.012837 1 route.go:866] Reconciling routes to gw: 240.17.0.5
  5136. 2020-10-08T11:45:18.5315996Z [cluster2] I1008 11:42:42.012963 1 route.go:882] Processing route {Ifindex: 18 Dst: 10.241.0.0/16 Src: <nil> Gw: 240.17.0.5 Flags: [] Table: 254}
  5137. 2020-10-08T11:45:18.5316725Z [cluster2] I1008 11:42:42.012985 1 route.go:888] Found route {Ifindex: 18 Dst: 10.241.0.0/16 Src: <nil> Gw: 240.17.0.5 Flags: [] Table: 254} with gw 240.17.0.5 already installed
  5138. 2020-10-08T11:45:18.5317449Z [cluster2] I1008 11:42:42.012994 1 route.go:882] Processing route {Ifindex: 18 Dst: 100.91.0.0/16 Src: <nil> Gw: 240.17.0.5 Flags: [] Table: 254}
  5139. 2020-10-08T11:45:18.5318170Z [cluster2] I1008 11:42:42.013003 1 route.go:888] Found route {Ifindex: 18 Dst: 100.91.0.0/16 Src: <nil> Gw: 240.17.0.5 Flags: [] Table: 254} with gw 240.17.0.5 already installed
  5140. 2020-10-08T11:45:18.5318889Z [cluster2] I1008 11:42:42.013011 1 route.go:882] Processing route {Ifindex: 18 Dst: 240.0.0.0/8 Src: 240.17.0.4 Gw: <nil> Flags: [] Table: 254}
  5141. 2020-10-08T11:45:18.5319444Z [cluster2] I1008 11:42:42.013019 1 route.go:885] Found nil gw or dst
  5142. 2020-10-08T11:45:18.5319944Z [cluster2] I1008 11:42:42.013068 1 route.go:924] Found equivalent route, not adding
  5143. 2020-10-08T11:45:18.5320506Z [cluster2] I1008 11:42:42.013076 1 route.go:924] Found equivalent route, not adding
  5144. 2020-10-08T11:45:18.5321867Z [cluster2] I1008 11:42:42.013104 1 route.go:715] Successfully processed local Endpoint "cluster2-submariner-cable-cluster2-172-17-0-5"
  5145. 2020-10-08T11:45:18.5324776Z [cluster2] I1008 11:42:42.014806 1 route.go:620] Updating iptable rules for remote cluster's Endpoint "cluster1-submariner-cable-cluster1-172-17-0-9" with subnets [100.91.0.0/16 10.241.0.0/16]
  5146. 2020-10-08T11:45:18.5329782Z [cluster2] I1008 11:43:12.002778 1 route.go:736] Enqueueing endpoint for route controller {"metadata":{"name":"cluster2-submariner-cable-cluster2-172-17-0-5","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster2-submariner-cable-cluster2-172-17-0-5","uid":"6a117963-5044-4b34-9df6-57ebb8a4981d","resourceVersion":"1284","generation":1,"creationTimestamp":"2020-10-08T11:35:26Z"},"spec":{"cluster_id":"cluster2","cable_name":"submariner-cable-cluster2-172-17-0-5","hostname":"cluster2-worker","subnets":["100.92.0.0/16","10.242.0.0/16"],"private_ip":"172.17.0.5","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
  5147. 2020-10-08T11:45:18.5337261Z [cluster2] I1008 11:43:12.002837 1 route.go:736] Enqueueing endpoint for route controller {"metadata":{"name":"cluster1-submariner-cable-cluster1-172-17-0-9","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster1-submariner-cable-cluster1-172-17-0-9","uid":"60d04732-7b08-4685-b72b-cb2c12702817","resourceVersion":"1285","generation":1,"creationTimestamp":"2020-10-08T11:35:26Z"},"spec":{"cluster_id":"cluster1","cable_name":"submariner-cable-cluster1-172-17-0-9","hostname":"cluster1-worker","subnets":["100.91.0.0/16","10.241.0.0/16"],"private_ip":"172.17.0.9","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
  5148. 2020-10-08T11:45:18.5341846Z [cluster2] I1008 11:43:12.010179 1 route.go:634] Processing local Endpoint cluster2-submariner-cable-cluster2-172-17-0-5 with IP 172.17.0.5, Host cluster2-worker for this cluster
  5149. 2020-10-08T11:45:18.5354392Z [cluster2] I1008 11:43:12.010197 1 route.go:677] This route agent is running on a non-gateway/non-active node
  5150. 2020-10-08T11:45:18.5355395Z [cluster2] I1008 11:43:12.010206 1 route.go:689] Creating the vxlan interface vx-submariner with gateway node IP 172.17.0.5
  5151. 2020-10-08T11:45:18.5356237Z [cluster2] I1008 11:43:12.010374 1 vxlan.go:61] VxLAN interface already exists with same configuration.
  5152. 2020-10-08T11:45:18.5357334Z [lighthouse]$ [cluster2] kubectl -n submariner-operator logs submariner-routeagent-gk7px
  5153. 2020-10-08T11:45:18.5358542Z [lighthouse]$ [cluster2] kubectl -n submariner-operator logs submariner-routeagent-gk7px
  5154. 2020-10-08T11:45:18.5359983Z [lighthouse]$ [cluster2] command kubectl --context=cluster2 -n submariner-operator logs submariner-routeagent-gk7px
  5155. 2020-10-08T11:45:18.5361267Z [lighthouse]$ [cluster2] kubectl --context=cluster2 -n submariner-operator logs submariner-routeagent-gk7px
  5156. 2020-10-08T11:45:18.5362232Z [cluster2] I1008 11:43:12.013330 1 route.go:866] Reconciling routes to gw: 240.17.0.5
  5157. 2020-10-08T11:45:18.5362849Z [cluster2] I1008 11:43:12.014034 1 route.go:882] Processing route {Ifindex: 18 Dst: 10.241.0.0/16 Src: <nil> Gw: 240.17.0.5 Flags: [] Table: 254}
  5158. 2020-10-08T11:45:18.5363581Z [cluster2] I1008 11:43:12.014099 1 route.go:888] Found route {Ifindex: 18 Dst: 10.241.0.0/16 Src: <nil> Gw: 240.17.0.5 Flags: [] Table: 254} with gw 240.17.0.5 already installed
  5159. 2020-10-08T11:45:18.5365613Z [cluster2] I1008 11:43:12.014124 1 route.go:882] Processing route {Ifindex: 18 Dst: 100.91.0.0/16 Src: <nil> Gw: 240.17.0.5 Flags: [] Table: 254}
  5160. 2020-10-08T11:45:18.5366352Z [cluster2] I1008 11:43:12.014144 1 route.go:888] Found route {Ifindex: 18 Dst: 100.91.0.0/16 Src: <nil> Gw: 240.17.0.5 Flags: [] Table: 254} with gw 240.17.0.5 already installed
  5161. 2020-10-08T11:45:18.5367076Z [cluster2] I1008 11:43:12.014175 1 route.go:882] Processing route {Ifindex: 18 Dst: 240.0.0.0/8 Src: 240.17.0.4 Gw: <nil> Flags: [] Table: 254}
  5162. 2020-10-08T11:45:18.5367623Z [cluster2] I1008 11:43:12.014210 1 route.go:885] Found nil gw or dst
  5163. 2020-10-08T11:45:18.5368121Z [cluster2] I1008 11:43:12.014303 1 route.go:924] Found equivalent route, not adding
  5164. 2020-10-08T11:45:18.5368685Z [cluster2] I1008 11:43:12.014331 1 route.go:924] Found equivalent route, not adding
  5165. 2020-10-08T11:45:18.5370106Z [cluster2] I1008 11:43:12.014387 1 route.go:715] Successfully processed local Endpoint "cluster2-submariner-cable-cluster2-172-17-0-5"
  5166. 2020-10-08T11:45:18.5372133Z [cluster2] I1008 11:43:12.016583 1 route.go:620] Updating iptable rules for remote cluster's Endpoint "cluster1-submariner-cable-cluster1-172-17-0-9" with subnets [100.91.0.0/16 10.241.0.0/16]
  5167. 2020-10-08T11:45:18.5377915Z [cluster2] I1008 11:43:42.003010 1 route.go:736] Enqueueing endpoint for route controller {"metadata":{"name":"cluster2-submariner-cable-cluster2-172-17-0-5","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster2-submariner-cable-cluster2-172-17-0-5","uid":"6a117963-5044-4b34-9df6-57ebb8a4981d","resourceVersion":"1284","generation":1,"creationTimestamp":"2020-10-08T11:35:26Z"},"spec":{"cluster_id":"cluster2","cable_name":"submariner-cable-cluster2-172-17-0-5","hostname":"cluster2-worker","subnets":["100.92.0.0/16","10.242.0.0/16"],"private_ip":"172.17.0.5","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
  5168. 2020-10-08T11:45:18.5384002Z [cluster2] I1008 11:43:42.003112 1 route.go:736] Enqueueing endpoint for route controller {"metadata":{"name":"cluster1-submariner-cable-cluster1-172-17-0-9","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster1-submariner-cable-cluster1-172-17-0-9","uid":"60d04732-7b08-4685-b72b-cb2c12702817","resourceVersion":"1285","generation":1,"creationTimestamp":"2020-10-08T11:35:26Z"},"spec":{"cluster_id":"cluster1","cable_name":"submariner-cable-cluster1-172-17-0-9","hostname":"cluster1-worker","subnets":["100.91.0.0/16","10.241.0.0/16"],"private_ip":"172.17.0.9","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
  5169. 2020-10-08T11:45:18.5388318Z [cluster2] I1008 11:43:42.010111 1 route.go:634] Processing local Endpoint cluster2-submariner-cable-cluster2-172-17-0-5 with IP 172.17.0.5, Host cluster2-worker for this cluster
  5170. 2020-10-08T11:45:18.5389820Z [cluster2] I1008 11:43:42.010148 1 route.go:677] This route agent is running on a non-gateway/non-active node
  5171. 2020-10-08T11:45:18.5390797Z [cluster2] I1008 11:43:42.010158 1 route.go:689] Creating the vxlan interface vx-submariner with gateway node IP 172.17.0.5
  5172. 2020-10-08T11:45:18.5391630Z [cluster2] I1008 11:43:42.010354 1 vxlan.go:61] VxLAN interface already exists with same configuration.
  5173. 2020-10-08T11:45:18.5392259Z [cluster2] I1008 11:43:42.013339 1 route.go:866] Reconciling routes to gw: 240.17.0.5
  5174. 2020-10-08T11:45:18.5392863Z [cluster2] I1008 11:43:42.013558 1 route.go:882] Processing route {Ifindex: 18 Dst: 10.241.0.0/16 Src: <nil> Gw: 240.17.0.5 Flags: [] Table: 254}
  5175. 2020-10-08T11:45:18.5393592Z [cluster2] I1008 11:43:42.013587 1 route.go:888] Found route {Ifindex: 18 Dst: 10.241.0.0/16 Src: <nil> Gw: 240.17.0.5 Flags: [] Table: 254} with gw 240.17.0.5 already installed
  5176. 2020-10-08T11:45:18.5394303Z [cluster2] I1008 11:43:42.013596 1 route.go:882] Processing route {Ifindex: 18 Dst: 100.91.0.0/16 Src: <nil> Gw: 240.17.0.5 Flags: [] Table: 254}
  5177. 2020-10-08T11:45:18.5395030Z [cluster2] I1008 11:43:42.013604 1 route.go:888] Found route {Ifindex: 18 Dst: 100.91.0.0/16 Src: <nil> Gw: 240.17.0.5 Flags: [] Table: 254} with gw 240.17.0.5 already installed
  5178. 2020-10-08T11:45:18.5395754Z [cluster2] I1008 11:43:42.013617 1 route.go:882] Processing route {Ifindex: 18 Dst: 240.0.0.0/8 Src: 240.17.0.4 Gw: <nil> Flags: [] Table: 254}
  5179. 2020-10-08T11:45:18.5396306Z [cluster2] I1008 11:43:42.013624 1 route.go:885] Found nil gw or dst
  5180. 2020-10-08T11:45:18.5396807Z [cluster2] I1008 11:43:42.013671 1 route.go:924] Found equivalent route, not adding
  5181. 2020-10-08T11:45:18.5397369Z [cluster2] I1008 11:43:42.013683 1 route.go:924] Found equivalent route, not adding
  5182. 2020-10-08T11:45:18.5398752Z [cluster2] I1008 11:43:42.013711 1 route.go:715] Successfully processed local Endpoint "cluster2-submariner-cable-cluster2-172-17-0-5"
  5183. 2020-10-08T11:45:18.5400513Z [cluster2] I1008 11:43:42.015424 1 route.go:620] Updating iptable rules for remote cluster's Endpoint "cluster1-submariner-cable-cluster1-172-17-0-9" with subnets [100.91.0.0/16 10.241.0.0/16]
  5184. 2020-10-08T11:45:18.5406268Z [cluster2] I1008 11:44:12.003106 1 route.go:736] Enqueueing endpoint for route controller {"metadata":{"name":"cluster2-submariner-cable-cluster2-172-17-0-5","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster2-submariner-cable-cluster2-172-17-0-5","uid":"6a117963-5044-4b34-9df6-57ebb8a4981d","resourceVersion":"1284","generation":1,"creationTimestamp":"2020-10-08T11:35:26Z"},"spec":{"cluster_id":"cluster2","cable_name":"submariner-cable-cluster2-172-17-0-5","hostname":"cluster2-worker","subnets":["100.92.0.0/16","10.242.0.0/16"],"private_ip":"172.17.0.5","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
  5185. 2020-10-08T11:45:18.5425568Z [cluster2] I1008 11:44:12.003192 1 route.go:736] Enqueueing endpoint for route controller {"metadata":{"name":"cluster1-submariner-cable-cluster1-172-17-0-9","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster1-submariner-cable-cluster1-172-17-0-9","uid":"60d04732-7b08-4685-b72b-cb2c12702817","resourceVersion":"1285","generation":1,"creationTimestamp":"2020-10-08T11:35:26Z"},"spec":{"cluster_id":"cluster1","cable_name":"submariner-cable-cluster1-172-17-0-9","hostname":"cluster1-worker","subnets":["100.91.0.0/16","10.241.0.0/16"],"private_ip":"172.17.0.9","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
  5186. 2020-10-08T11:45:18.5430331Z [cluster2] I1008 11:44:12.010761 1 route.go:634] Processing local Endpoint cluster2-submariner-cable-cluster2-172-17-0-5 with IP 172.17.0.5, Host cluster2-worker for this cluster
  5187. 2020-10-08T11:45:18.5431964Z [cluster2] I1008 11:44:12.010782 1 route.go:677] This route agent is running on a non-gateway/non-active node
  5188. 2020-10-08T11:45:18.5433099Z [cluster2] I1008 11:44:12.010791 1 route.go:689] Creating the vxlan interface vx-submariner with gateway node IP 172.17.0.5
  5189. 2020-10-08T11:45:18.5433858Z [cluster2] I1008 11:44:12.010953 1 vxlan.go:61] VxLAN interface already exists with same configuration.
  5190. 2020-10-08T11:45:18.5434569Z [cluster2] I1008 11:44:12.013867 1 route.go:866] Reconciling routes to gw: 240.17.0.5
  5191. 2020-10-08T11:45:18.5435182Z [cluster2] I1008 11:44:12.013984 1 route.go:882] Processing route {Ifindex: 18 Dst: 10.241.0.0/16 Src: <nil> Gw: 240.17.0.5 Flags: [] Table: 254}
  5192. 2020-10-08T11:45:18.5435913Z [cluster2] I1008 11:44:12.014012 1 route.go:888] Found route {Ifindex: 18 Dst: 10.241.0.0/16 Src: <nil> Gw: 240.17.0.5 Flags: [] Table: 254} with gw 240.17.0.5 already installed
  5193. 2020-10-08T11:45:18.5436642Z [cluster2] I1008 11:44:12.014021 1 route.go:882] Processing route {Ifindex: 18 Dst: 100.91.0.0/16 Src: <nil> Gw: 240.17.0.5 Flags: [] Table: 254}
  5194. 2020-10-08T11:45:18.5437360Z [cluster2] I1008 11:44:12.014029 1 route.go:888] Found route {Ifindex: 18 Dst: 100.91.0.0/16 Src: <nil> Gw: 240.17.0.5 Flags: [] Table: 254} with gw 240.17.0.5 already installed
  5195. 2020-10-08T11:45:18.5438082Z [cluster2] I1008 11:44:12.014042 1 route.go:882] Processing route {Ifindex: 18 Dst: 240.0.0.0/8 Src: 240.17.0.4 Gw: <nil> Flags: [] Table: 254}
  5196. 2020-10-08T11:45:18.5438637Z [cluster2] I1008 11:44:12.014050 1 route.go:885] Found nil gw or dst
  5197. 2020-10-08T11:45:18.5439136Z [cluster2] I1008 11:44:12.014097 1 route.go:924] Found equivalent route, not adding
  5198. 2020-10-08T11:45:18.5439696Z [cluster2] I1008 11:44:12.014104 1 route.go:924] Found equivalent route, not adding
  5199. 2020-10-08T11:45:18.5441052Z [cluster2] I1008 11:44:12.014131 1 route.go:715] Successfully processed local Endpoint "cluster2-submariner-cable-cluster2-172-17-0-5"
  5200. 2020-10-08T11:45:18.5443077Z [cluster2] I1008 11:44:12.015499 1 route.go:620] Updating iptable rules for remote cluster's Endpoint "cluster1-submariner-cable-cluster1-172-17-0-9" with subnets [100.91.0.0/16 10.241.0.0/16]
  5201. 2020-10-08T11:45:18.5447610Z [cluster2] I1008 11:44:42.003394 1 route.go:736] Enqueueing endpoint for route controller {"metadata":{"name":"cluster2-submariner-cable-cluster2-172-17-0-5","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster2-submariner-cable-cluster2-172-17-0-5","uid":"6a117963-5044-4b34-9df6-57ebb8a4981d","resourceVersion":"1284","generation":1,"creationTimestamp":"2020-10-08T11:35:26Z"},"spec":{"cluster_id":"cluster2","cable_name":"submariner-cable-cluster2-172-17-0-5","hostname":"cluster2-worker","subnets":["100.92.0.0/16","10.242.0.0/16"],"private_ip":"172.17.0.5","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
  5202. 2020-10-08T11:45:18.5456259Z [cluster2] I1008 11:44:42.003484 1 route.go:736] Enqueueing endpoint for route controller {"metadata":{"name":"cluster1-submariner-cable-cluster1-172-17-0-9","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster1-submariner-cable-cluster1-172-17-0-9","uid":"60d04732-7b08-4685-b72b-cb2c12702817","resourceVersion":"1285","generation":1,"creationTimestamp":"2020-10-08T11:35:26Z"},"spec":{"cluster_id":"cluster1","cable_name":"submariner-cable-cluster1-172-17-0-9","hostname":"cluster1-worker","subnets":["100.91.0.0/16","10.241.0.0/16"],"private_ip":"172.17.0.9","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
  5203. 2020-10-08T11:45:18.5460070Z [cluster2] I1008 11:44:42.010410 1 route.go:634] Processing local Endpoint cluster2-submariner-cable-cluster2-172-17-0-5 with IP 172.17.0.5, Host cluster2-worker for this cluster
  5204. 2020-10-08T11:45:18.5462135Z [cluster2] I1008 11:44:42.010426 1 route.go:677] This route agent is running on a non-gateway/non-active node
  5205. 2020-10-08T11:45:18.5462995Z [cluster2] I1008 11:44:42.010434 1 route.go:689] Creating the vxlan interface vx-submariner with gateway node IP 172.17.0.5
  5206. 2020-10-08T11:45:18.5463876Z [cluster2] I1008 11:44:42.010586 1 vxlan.go:61] VxLAN interface already exists with same configuration.
  5207. 2020-10-08T11:45:18.5464713Z [cluster2] I1008 11:44:42.013621 1 route.go:866] Reconciling routes to gw: 240.17.0.5
  5208. 2020-10-08T11:45:18.5465452Z [cluster2] I1008 11:44:42.013785 1 route.go:882] Processing route {Ifindex: 18 Dst: 10.241.0.0/16 Src: <nil> Gw: 240.17.0.5 Flags: [] Table: 254}
  5209. 2020-10-08T11:45:18.5466184Z [cluster2] I1008 11:44:42.013816 1 route.go:888] Found route {Ifindex: 18 Dst: 10.241.0.0/16 Src: <nil> Gw: 240.17.0.5 Flags: [] Table: 254} with gw 240.17.0.5 already installed
  5210. 2020-10-08T11:45:18.5466914Z [cluster2] I1008 11:44:42.013839 1 route.go:882] Processing route {Ifindex: 18 Dst: 100.91.0.0/16 Src: <nil> Gw: 240.17.0.5 Flags: [] Table: 254}
  5211. 2020-10-08T11:45:18.5467636Z [cluster2] I1008 11:44:42.013851 1 route.go:888] Found route {Ifindex: 18 Dst: 100.91.0.0/16 Src: <nil> Gw: 240.17.0.5 Flags: [] Table: 254} with gw 240.17.0.5 already installed
  5212. 2020-10-08T11:45:18.5468357Z [cluster2] I1008 11:44:42.013865 1 route.go:882] Processing route {Ifindex: 18 Dst: 240.0.0.0/8 Src: 240.17.0.4 Gw: <nil> Flags: [] Table: 254}
  5213. 2020-10-08T11:45:18.5468909Z [cluster2] I1008 11:44:42.013873 1 route.go:885] Found nil gw or dst
  5214. 2020-10-08T11:45:18.5469416Z [cluster2] I1008 11:44:42.013941 1 route.go:924] Found equivalent route, not adding
  5215. 2020-10-08T11:45:18.5469977Z [cluster2] I1008 11:44:42.013956 1 route.go:924] Found equivalent route, not adding
  5216. 2020-10-08T11:45:18.5471377Z [cluster2] I1008 11:44:42.014025 1 route.go:715] Successfully processed local Endpoint "cluster2-submariner-cable-cluster2-172-17-0-5"
  5217. 2020-10-08T11:45:18.5473409Z [cluster2] I1008 11:44:42.015507 1 route.go:620] Updating iptable rules for remote cluster's Endpoint "cluster1-submariner-cable-cluster1-172-17-0-9" with subnets [100.91.0.0/16 10.241.0.0/16]
  5218. 2020-10-08T11:45:19.3026144Z [cluster2] I1008 11:45:12.003548 1 route.go:736] Enqueueing endpoint for route controller {"metadata":{"name":"cluster2-submariner-cable-cluster2-172-17-0-5","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster2-submariner-cable-cluster2-172-17-0-5","uid":"6a117963-5044-4b34-9df6-57ebb8a4981d","resourceVersion":"1284","generation":1,"creationTimestamp":"2020-10-08T11:35:26Z"},"spec":{"cluster_id":"cluster2","cable_name":"submariner-cable-cluster2-172-17-0-5","hostname":"cluster2-worker","subnets":["100.92.0.0/16","10.242.0.0/16"],"private_ip":"172.17.0.5","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
  5219. 2020-10-08T11:45:19.3036128Z [cluster2] I1008 11:45:12.003618 1 route.go:736] Enqueueing endpoint for route controller {"metadata":{"name":"cluster1-submariner-cable-cluster1-172-17-0-9","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster1-submariner-cable-cluster1-172-17-0-9","uid":"60d04732-7b08-4685-b72b-cb2c12702817","resourceVersion":"1285","generation":1,"creationTimestamp":"2020-10-08T11:35:26Z"},"spec":{"cluster_id":"cluster1","cable_name":"submariner-cable-cluster1-172-17-0-9","hostname":"cluster1-worker","subnets":["100.91.0.0/16","10.241.0.0/16"],"private_ip":"172.17.0.9","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
  5220. 2020-10-08T11:45:19.3040229Z [cluster2] I1008 11:45:12.011069 1 route.go:634] Processing local Endpoint cluster2-submariner-cable-cluster2-172-17-0-5 with IP 172.17.0.5, Host cluster2-worker for this cluster
  5221. 2020-10-08T11:45:19.3041566Z [cluster2] I1008 11:45:12.011088 1 route.go:677] This route agent is running on a non-gateway/non-active node
  5222. 2020-10-08T11:45:19.3042423Z [cluster2] I1008 11:45:12.011097 1 route.go:689] Creating the vxlan interface vx-submariner with gateway node IP 172.17.0.5
  5223. 2020-10-08T11:45:19.3043086Z [cluster2] I1008 11:45:12.011254 1 vxlan.go:61] VxLAN interface already exists with same configuration.
  5224. 2020-10-08T11:45:19.3043640Z [cluster2] I1008 11:45:12.014164 1 route.go:866] Reconciling routes to gw: 240.17.0.5
  5225. 2020-10-08T11:45:19.3044391Z [cluster2] I1008 11:45:12.014269 1 route.go:882] Processing route {Ifindex: 18 Dst: 10.241.0.0/16 Src: <nil> Gw: 240.17.0.5 Flags: [] Table: 254}
  5226. 2020-10-08T11:45:19.3045213Z [cluster2] I1008 11:45:12.014287 1 route.go:888] Found route {Ifindex: 18 Dst: 10.241.0.0/16 Src: <nil> Gw: 240.17.0.5 Flags: [] Table: 254} with gw 240.17.0.5 already installed
  5227. 2020-10-08T11:45:19.3045943Z [cluster2] I1008 11:45:12.014295 1 route.go:882] Processing route {Ifindex: 18 Dst: 100.91.0.0/16 Src: <nil> Gw: 240.17.0.5 Flags: [] Table: 254}
  5228. 2020-10-08T11:45:19.3046668Z [cluster2] I1008 11:45:12.014302 1 route.go:888] Found route {Ifindex: 18 Dst: 100.91.0.0/16 Src: <nil> Gw: 240.17.0.5 Flags: [] Table: 254} with gw 240.17.0.5 already installed
  5229. 2020-10-08T11:45:19.3047391Z [cluster2] I1008 11:45:12.014309 1 route.go:882] Processing route {Ifindex: 18 Dst: 240.0.0.0/8 Src: 240.17.0.4 Gw: <nil> Flags: [] Table: 254}
  5230. 2020-10-08T11:45:19.3047942Z [cluster2] I1008 11:45:12.014315 1 route.go:885] Found nil gw or dst
  5231. 2020-10-08T11:45:19.3048439Z [cluster2] I1008 11:45:12.014358 1 route.go:924] Found equivalent route, not adding
  5232. 2020-10-08T11:45:19.3049010Z [cluster2] I1008 11:45:12.014364 1 route.go:924] Found equivalent route, not adding
  5233. 2020-10-08T11:45:19.3050388Z [cluster2] I1008 11:45:12.014388 1 route.go:715] Successfully processed local Endpoint "cluster2-submariner-cable-cluster2-172-17-0-5"
  5234. 2020-10-08T11:45:19.3052846Z [cluster2] I1008 11:45:12.016360 1 route.go:620] Updating iptable rules for remote cluster's Endpoint "cluster1-submariner-cable-cluster1-172-17-0-9" with subnets [100.91.0.0/16 10.241.0.0/16]
  5235. 2020-10-08T11:45:19.3054600Z [cluster2] +++++++++++++++++++++: Logs for Pod submariner-routeagent-gk7px in namespace submariner-operator :++++++++++++++++++++++
  5236. 2020-10-08T11:45:19.3055391Z [cluster2] + trap 'exit 1' SIGTERM SIGINT
  5237. 2020-10-08T11:45:19.3055740Z [cluster2] + SUBMARINER_VERBOSITY=2
  5238. 2020-10-08T11:45:19.3056181Z [cluster2] + '[' false == true ']'
  5239. 2020-10-08T11:45:19.3056556Z [cluster2] + DEBUG=-v=2
  5240. 2020-10-08T11:45:19.3057016Z [cluster2] + for f in iptables-save iptables
  5241. 2020-10-08T11:45:19.3057561Z [cluster2] ++ find_iptables_on_host iptables-save
  5242. 2020-10-08T11:45:19.3058143Z [cluster2] ++ chroot /host test -x /usr/sbin/iptables-save
  5243. 2020-10-08T11:45:19.3058510Z [cluster2] ++ echo /usr/sbin
  5244. 2020-10-08T11:45:19.3058779Z [cluster2] ++ return
  5245. 2020-10-08T11:45:19.3059128Z [cluster2] + location=/usr/sbin
  5246. 2020-10-08T11:45:19.3059573Z [cluster2] + '[' /usr/sbin '!=' unknown ']'
  5247. 2020-10-08T11:45:19.3060188Z [cluster2] + echo 'iptables-save is present on the host at /usr/sbin/iptables-save'
  5248. 2020-10-08T11:45:19.3060908Z [cluster2] + sed 's!@@PATH@@!/usr/sbin!' /usr/sbin/iptables-wrapper.in
  5249. 2020-10-08T11:45:19.3061502Z [cluster2] + for f in iptables-save iptables
  5250. 2020-10-08T11:45:19.3061892Z [cluster2] ++ find_iptables_on_host iptables
  5251. 2020-10-08T11:45:19.3062404Z [cluster2] ++ chroot /host test -x /usr/sbin/iptables
  5252. 2020-10-08T11:45:19.3062740Z [cluster2] ++ echo /usr/sbin
  5253. 2020-10-08T11:45:19.3062991Z [cluster2] ++ return
  5254. 2020-10-08T11:45:19.3063276Z [cluster2] + location=/usr/sbin
  5255. 2020-10-08T11:45:19.3063714Z [cluster2] + '[' /usr/sbin '!=' unknown ']'
  5256. 2020-10-08T11:45:19.3064345Z [cluster2] + echo 'iptables is present on the host at /usr/sbin/iptables'
  5257. 2020-10-08T11:45:19.3065022Z [cluster2] + sed 's!@@PATH@@!/usr/sbin!' /usr/sbin/iptables-wrapper.in
  5258. 2020-10-08T11:45:19.3065726Z [cluster2] iptables-save is present on the host at /usr/sbin/iptables-save
  5259. 2020-10-08T11:45:19.3066250Z [cluster2] iptables is present on the host at /usr/sbin/iptables
  5260. 2020-10-08T11:45:19.3066945Z [cluster2] + exec submariner-route-agent -v=2 -alsologtostderr
  5261. 2020-10-08T11:45:19.3067714Z [cluster2] I1008 11:35:11.183882 1 main.go:47] Starting submariner-route-agent
  5262. 2020-10-08T11:45:19.3068696Z [cluster2] W1008 11:35:11.184979 1 client_config.go:543] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work.
  5263. 2020-10-08T11:45:19.3069646Z [cluster2] I1008 11:35:11.186698 1 route.go:195] Starting Route Controller. ClusterID: cluster2, localClusterCIDR: [10.242.0.0/16], localServiceCIDR: [100.92.0.0/16]
  5264. 2020-10-08T11:45:19.3070462Z [cluster2] I1008 11:35:11.186720 1 route.go:199] Waiting for Endpoint informer caches to sync.
  5265. 2020-10-08T11:45:19.3071231Z [cluster2] I1008 11:35:11.287280 1 route.go:751] Enqueueing sm-route-agent-pod event, ip: 172.17.0.4
  5266. 2020-10-08T11:45:19.3072208Z [cluster2] I1008 11:35:11.287302 1 route.go:751] Enqueueing sm-route-agent-pod event, ip: 172.17.0.5
  5267. 2020-10-08T11:45:19.3072855Z [cluster2] I1008 11:35:11.293687 1 driver.go:37] Interface "lo" has "127.0.0.1" address
  5268. 2020-10-08T11:45:19.3073320Z [cluster2] I1008 11:35:11.293752 1 driver.go:37] Interface "weave" has "10.242.192.0" address
  5269. 2020-10-08T11:45:19.3073876Z [cluster2] I1008 11:35:11.293759 1 driver.go:42] Found CNI Interface "weave" that has IP "10.242.192.0" from ClusterCIDR "10.242.0.0/16"
  5270. 2020-10-08T11:45:19.3074564Z [cluster2] I1008 11:35:11.294118 1 driver.go:60] Successfully configured rp_filter to loose mode(2) on cniInterface "weave"
  5271. 2020-10-08T11:45:19.3075517Z [cluster2] I1008 11:35:11.313409 1 iptables.go:22] Install/ensure SUBMARINER-POSTROUTING chain exists
  5272. 2020-10-08T11:45:19.3076639Z [cluster2] I1008 11:35:11.378668 1 iptables.go:28] Insert SUBMARINER-POSTROUTING rule that has rules for inter-cluster traffic
  5273. 2020-10-08T11:45:19.3077689Z [cluster2] I1008 11:35:11.431321 1 iptables.go:34] Install/ensure SUBMARINER-INPUT chain exists
  5274. 2020-10-08T11:45:19.3078545Z [cluster2] I1008 11:35:11.482873 1 iptables.go:45] Allow VxLAN incoming traffic in SUBMARINER-INPUT Chain
  5275. 2020-10-08T11:45:19.3079479Z [cluster2] I1008 11:35:11.492645 1 iptables.go:53] Insert rule to allow traffic over vx-submariner interface in FORWARDing Chain
  5276. 2020-10-08T11:45:19.3080533Z [cluster2] I1008 11:35:11.503912 1 iptables.go:65] Installing rule for host network to remote cluster communication: -s 240.0.0.0/8 -o vx-submariner -j SNAT --to 10.242.192.0
  5277. 2020-10-08T11:45:19.3081613Z [cluster2] I1008 11:35:11.517551 1 route.go:257] In cluster2, podIP of submariner-route-agent[0] is 172.17.0.4
  5278. 2020-10-08T11:45:19.3087577Z [cluster2] I1008 11:35:11.518009 1 route.go:257] In cluster2, podIP of submariner-route-agent[1] is 172.17.0.5
  5279. 2020-10-08T11:45:19.3089850Z [cluster2] I1008 11:35:11.518025 1 route.go:271] Hostname is "cluster2-worker" and routeAgentNodeName is "cluster2-worker"
  5280. 2020-10-08T11:45:19.3091034Z [cluster2] I1008 11:35:11.546386 1 driver.go:87] Successfully annotated node "cluster2-worker" with cniIfaceIP "10.242.192.0"
  5281. 2020-10-08T11:45:19.3091730Z [cluster2] I1008 11:35:11.546410 1 route.go:288] Route agent workers started
  5282. 2020-10-08T11:45:19.3093083Z [cluster2] I1008 11:35:11.736355 1 route.go:540] Processing submariner-route-agent pod "submariner-routeagent-clqmx" with host IP "172.17.0.4", pod IP "172.17.0.4"
  5283. 2020-10-08T11:45:19.3094632Z [cluster2] I1008 11:35:11.736383 1 route.go:568] Successfully processed submariner-route-agent pod "submariner-routeagent-clqmx"
  5284. 2020-10-08T11:45:19.3096349Z [cluster2] I1008 11:35:11.777183 1 route.go:540] Processing submariner-route-agent pod "submariner-routeagent-gk7px" with host IP "172.17.0.5", pod IP "172.17.0.5"
  5285. 2020-10-08T11:45:19.3097898Z [cluster2] I1008 11:35:11.779731 1 route.go:568] Successfully processed submariner-route-agent pod "submariner-routeagent-gk7px"
  5286. 2020-10-08T11:45:19.3101980Z [cluster2] I1008 11:35:26.075376 1 route.go:736] Enqueueing endpoint for route controller {"metadata":{"name":"cluster2-submariner-cable-cluster2-172-17-0-5","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster2-submariner-cable-cluster2-172-17-0-5","uid":"6a117963-5044-4b34-9df6-57ebb8a4981d","resourceVersion":"1284","generation":1,"creationTimestamp":"2020-10-08T11:35:26Z"},"spec":{"cluster_id":"cluster2","cable_name":"submariner-cable-cluster2-172-17-0-5","hostname":"cluster2-worker","subnets":["100.92.0.0/16","10.242.0.0/16"],"private_ip":"172.17.0.5","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
  5287. 2020-10-08T11:45:19.3110615Z [cluster2] I1008 11:35:26.086662 1 route.go:634] Processing local Endpoint cluster2-submariner-cable-cluster2-172-17-0-5 with IP 172.17.0.5, Host cluster2-worker for this cluster
  5288. 2020-10-08T11:45:19.3111768Z [cluster2] I1008 11:35:26.086853 1 route.go:851] Processing route {Ifindex: 10 Dst: <nil> Src: <nil> Gw: 172.17.0.1 Flags: [] Table: 254}
  5289. 2020-10-08T11:45:19.3112266Z [cluster2] I1008 11:35:26.086911 1 route.go:854] Found nil gw or dst
  5290. 2020-10-08T11:45:19.3112928Z [cluster2] I1008 11:35:26.086915 1 route.go:851] Processing route {Ifindex: 4 Dst: 10.242.0.0/16 Src: 10.242.192.0 Gw: <nil> Flags: [] Table: 254}
  5291. 2020-10-08T11:45:19.3113479Z [cluster2] I1008 11:35:26.086925 1 route.go:854] Found nil gw or dst
  5292. 2020-10-08T11:45:19.3114040Z [cluster2] I1008 11:35:26.086928 1 route.go:851] Processing route {Ifindex: 10 Dst: 172.17.0.0/16 Src: 172.17.0.5 Gw: <nil> Flags: [] Table: 254}
  5293. 2020-10-08T11:45:19.3114589Z [cluster2] I1008 11:35:26.086948 1 route.go:854] Found nil gw or dst
  5294. 2020-10-08T11:45:19.3115134Z [cluster2] I1008 11:35:26.086956 1 route.go:645] This route agent is running on the active gateway node
  5295. 2020-10-08T11:45:19.3116063Z [cluster2] I1008 11:35:26.086964 1 route.go:653] Creating the vxlan interface: vx-submariner on the gateway node
  5296. 2020-10-08T11:45:19.3116758Z [cluster2] I1008 11:35:26.095891 1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.4 00:00:00:00:00:00
  5297. 2020-10-08T11:45:19.3117385Z [cluster2] I1008 11:35:26.096092 1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.5 00:00:00:00:00:00
  5298. 2020-10-08T11:45:19.3119319Z [cluster2] I1008 11:35:26.096562 1 route.go:478] Successfully configured rp_filter to loose mode(2) on vx-submariner
  5299. 2020-10-08T11:45:19.3124427Z [cluster2] I1008 11:35:26.123834 1 route.go:736] Enqueueing endpoint for route controller {"metadata":{"name":"cluster1-submariner-cable-cluster1-172-17-0-9","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster1-submariner-cable-cluster1-172-17-0-9","uid":"60d04732-7b08-4685-b72b-cb2c12702817","resourceVersion":"1285","generation":1,"creationTimestamp":"2020-10-08T11:35:26Z"},"spec":{"cluster_id":"cluster1","cable_name":"submariner-cable-cluster1-172-17-0-9","hostname":"cluster1-worker","subnets":["100.91.0.0/16","10.241.0.0/16"],"private_ip":"172.17.0.9","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
  5300. 2020-10-08T11:45:19.3129154Z [cluster2] I1008 11:35:26.145340 1 route.go:620] Updating iptable rules for remote cluster's Endpoint "cluster1-submariner-cable-cluster1-172-17-0-9" with subnets [100.91.0.0/16 10.241.0.0/16]
  5301. 2020-10-08T11:45:19.3130688Z [cluster2] I1008 11:35:26.188366 1 iptables.go:83] Installing iptables rule for outgoing traffic: -s 10.242.0.0/16 -d 100.91.0.0/16 -j ACCEPT
  5302. 2020-10-08T11:45:19.3131712Z [cluster2] I1008 11:35:26.218784 1 iptables.go:91] Installing iptables rule for incoming traffic: -s 100.91.0.0/16 -d 10.242.0.0/16 -j ACCEPT
  5303. 2020-10-08T11:45:19.3133001Z [cluster2] I1008 11:35:26.262277 1 iptables.go:83] Installing iptables rule for outgoing traffic: -s 10.242.0.0/16 -d 10.241.0.0/16 -j ACCEPT
  5304. 2020-10-08T11:45:19.3134033Z [cluster2] I1008 11:35:26.270281 1 iptables.go:91] Installing iptables rule for incoming traffic: -s 10.241.0.0/16 -d 10.242.0.0/16 -j ACCEPT
  5305. 2020-10-08T11:45:19.3139318Z [cluster2] I1008 11:35:41.228070 1 route.go:736] Enqueueing endpoint for route controller {"metadata":{"name":"cluster2-submariner-cable-cluster2-172-17-0-5","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster2-submariner-cable-cluster2-172-17-0-5","uid":"6a117963-5044-4b34-9df6-57ebb8a4981d","resourceVersion":"1284","generation":1,"creationTimestamp":"2020-10-08T11:35:26Z"},"spec":{"cluster_id":"cluster2","cable_name":"submariner-cable-cluster2-172-17-0-5","hostname":"cluster2-worker","subnets":["100.92.0.0/16","10.242.0.0/16"],"private_ip":"172.17.0.5","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
  5306. 2020-10-08T11:45:19.3190928Z [cluster2] I1008 11:35:41.228283 1 route.go:736] Enqueueing endpoint for route controller {"metadata":{"name":"cluster1-submariner-cable-cluster1-172-17-0-9","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster1-submariner-cable-cluster1-172-17-0-9","uid":"60d04732-7b08-4685-b72b-cb2c12702817","resourceVersion":"1285","generation":1,"creationTimestamp":"2020-10-08T11:35:26Z"},"spec":{"cluster_id":"cluster1","cable_name":"submariner-cable-cluster1-172-17-0-9","hostname":"cluster1-worker","subnets":["100.91.0.0/16","10.241.0.0/16"],"private_ip":"172.17.0.9","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
  5307. 2020-10-08T11:45:19.3195389Z [cluster2] I1008 11:35:41.236172 1 route.go:634] Processing local Endpoint cluster2-submariner-cable-cluster2-172-17-0-5 with IP 172.17.0.5, Host cluster2-worker for this cluster
  5308. 2020-10-08T11:45:19.3196963Z [cluster2] I1008 11:35:41.236367 1 route.go:851] Processing route {Ifindex: 16 Dst: 240.0.0.0/8 Src: 240.17.0.5 Gw: <nil> Flags: [] Table: 254}
  5309. 2020-10-08T11:45:19.3197573Z [cluster2] I1008 11:35:41.236463 1 route.go:854] Found nil gw or dst
  5310. 2020-10-08T11:45:19.3198152Z [cluster2] I1008 11:35:41.236481 1 route.go:645] This route agent is running on the active gateway node
  5311. 2020-10-08T11:45:19.3199155Z [cluster2] I1008 11:35:41.236489 1 route.go:653] Creating the vxlan interface: vx-submariner on the gateway node
  5312. 2020-10-08T11:45:19.3199968Z [cluster2] I1008 11:35:41.236644 1 vxlan.go:61] VxLAN interface already exists with same configuration.
  5313. 2020-10-08T11:45:19.3200800Z [cluster2] I1008 11:35:41.236682 1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.4 00:00:00:00:00:00
  5314. 2020-10-08T11:45:19.3201481Z [cluster2] I1008 11:35:41.236708 1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.5 00:00:00:00:00:00
  5315. 2020-10-08T11:45:19.3202675Z [cluster2] I1008 11:35:41.236774 1 route.go:478] Successfully configured rp_filter to loose mode(2) on vx-submariner
  5316. 2020-10-08T11:45:19.3204271Z [cluster2] I1008 11:35:41.239698 1 route.go:620] Updating iptable rules for remote cluster's Endpoint "cluster1-submariner-cable-cluster1-172-17-0-9" with subnets [100.91.0.0/16 10.241.0.0/16]
  5317. 2020-10-08T11:45:19.3209108Z [cluster2] I1008 11:36:11.228909 1 route.go:736] Enqueueing endpoint for route controller {"metadata":{"name":"cluster2-submariner-cable-cluster2-172-17-0-5","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster2-submariner-cable-cluster2-172-17-0-5","uid":"6a117963-5044-4b34-9df6-57ebb8a4981d","resourceVersion":"1284","generation":1,"creationTimestamp":"2020-10-08T11:35:26Z"},"spec":{"cluster_id":"cluster2","cable_name":"submariner-cable-cluster2-172-17-0-5","hostname":"cluster2-worker","subnets":["100.92.0.0/16","10.242.0.0/16"],"private_ip":"172.17.0.5","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
  5318. 2020-10-08T11:45:19.3215839Z [cluster2] I1008 11:36:11.228969 1 route.go:736] Enqueueing endpoint for route controller {"metadata":{"name":"cluster1-submariner-cable-cluster1-172-17-0-9","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster1-submariner-cable-cluster1-172-17-0-9","uid":"60d04732-7b08-4685-b72b-cb2c12702817","resourceVersion":"1285","generation":1,"creationTimestamp":"2020-10-08T11:35:26Z"},"spec":{"cluster_id":"cluster1","cable_name":"submariner-cable-cluster1-172-17-0-9","hostname":"cluster1-worker","subnets":["100.91.0.0/16","10.241.0.0/16"],"private_ip":"172.17.0.9","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
  5319. 2020-10-08T11:45:19.3220460Z [cluster2] I1008 11:36:11.292777 1 route.go:634] Processing local Endpoint cluster2-submariner-cable-cluster2-172-17-0-5 with IP 172.17.0.5, Host cluster2-worker for this cluster
  5320. 2020-10-08T11:45:19.3221928Z [cluster2] I1008 11:36:11.292955 1 route.go:851] Processing route {Ifindex: 16 Dst: 240.0.0.0/8 Src: 240.17.0.5 Gw: <nil> Flags: [] Table: 254}
  5321. 2020-10-08T11:45:19.3222537Z [cluster2] I1008 11:36:11.292984 1 route.go:854] Found nil gw or dst
  5322. 2020-10-08T11:45:19.3223115Z [cluster2] I1008 11:36:11.292993 1 route.go:645] This route agent is running on the active gateway node
  5323. 2020-10-08T11:45:19.3224118Z [cluster2] I1008 11:36:11.293001 1 route.go:653] Creating the vxlan interface: vx-submariner on the gateway node
  5324. 2020-10-08T11:45:19.3228942Z [cluster2] I1008 11:36:11.293118 1 vxlan.go:61] VxLAN interface already exists with same configuration.
  5325. 2020-10-08T11:45:19.3229777Z [cluster2] I1008 11:36:11.293157 1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.4 00:00:00:00:00:00
  5326. 2020-10-08T11:45:19.3230528Z [cluster2] I1008 11:36:11.293182 1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.5 00:00:00:00:00:00
  5327. 2020-10-08T11:45:19.3231825Z [cluster2] I1008 11:36:11.293266 1 route.go:478] Successfully configured rp_filter to loose mode(2) on vx-submariner
  5328. 2020-10-08T11:45:19.3233723Z [cluster2] I1008 11:36:11.295549 1 route.go:620] Updating iptable rules for remote cluster's Endpoint "cluster1-submariner-cable-cluster1-172-17-0-9" with subnets [100.91.0.0/16 10.241.0.0/16]
  5329. 2020-10-08T11:45:19.3239207Z [cluster2] I1008 11:36:41.230026 1 route.go:736] Enqueueing endpoint for route controller {"metadata":{"name":"cluster1-submariner-cable-cluster1-172-17-0-9","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster1-submariner-cable-cluster1-172-17-0-9","uid":"60d04732-7b08-4685-b72b-cb2c12702817","resourceVersion":"1285","generation":1,"creationTimestamp":"2020-10-08T11:35:26Z"},"spec":{"cluster_id":"cluster1","cable_name":"submariner-cable-cluster1-172-17-0-9","hostname":"cluster1-worker","subnets":["100.91.0.0/16","10.241.0.0/16"],"private_ip":"172.17.0.9","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
  5330. 2020-10-08T11:45:19.3247191Z [cluster2] I1008 11:36:41.230144 1 route.go:736] Enqueueing endpoint for route controller {"metadata":{"name":"cluster2-submariner-cable-cluster2-172-17-0-5","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster2-submariner-cable-cluster2-172-17-0-5","uid":"6a117963-5044-4b34-9df6-57ebb8a4981d","resourceVersion":"1284","generation":1,"creationTimestamp":"2020-10-08T11:35:26Z"},"spec":{"cluster_id":"cluster2","cable_name":"submariner-cable-cluster2-172-17-0-5","hostname":"cluster2-worker","subnets":["100.92.0.0/16","10.242.0.0/16"],"private_ip":"172.17.0.5","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
  5331. 2020-10-08T11:45:19.3252289Z [cluster2] I1008 11:36:41.238144 1 route.go:620] Updating iptable rules for remote cluster's Endpoint "cluster1-submariner-cable-cluster1-172-17-0-9" with subnets [100.91.0.0/16 10.241.0.0/16]
  5332. 2020-10-08T11:45:19.3254894Z [cluster2] I1008 11:36:41.240084 1 route.go:634] Processing local Endpoint cluster2-submariner-cable-cluster2-172-17-0-5 with IP 172.17.0.5, Host cluster2-worker for this cluster
  5333. 2020-10-08T11:45:19.3256335Z [cluster2] I1008 11:36:41.240301 1 route.go:851] Processing route {Ifindex: 16 Dst: 240.0.0.0/8 Src: 240.17.0.5 Gw: <nil> Flags: [] Table: 254}
  5334. 2020-10-08T11:45:19.3256895Z [cluster2] I1008 11:36:41.240329 1 route.go:854] Found nil gw or dst
  5335. 2020-10-08T11:45:19.3257431Z [cluster2] I1008 11:36:41.240338 1 route.go:645] This route agent is running on the active gateway node
  5336. 2020-10-08T11:45:19.3258392Z [cluster2] I1008 11:36:41.240346 1 route.go:653] Creating the vxlan interface: vx-submariner on the gateway node
  5337. 2020-10-08T11:45:19.3259140Z [cluster2] I1008 11:36:41.240474 1 vxlan.go:61] VxLAN interface already exists with same configuration.
  5338. 2020-10-08T11:45:19.3259823Z [cluster2] I1008 11:36:41.240502 1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.4 00:00:00:00:00:00
  5339. 2020-10-08T11:45:19.3260539Z [cluster2] I1008 11:36:41.240526 1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.5 00:00:00:00:00:00
  5340. 2020-10-08T11:45:19.3261481Z [cluster2] I1008 11:36:41.240569 1 route.go:478] Successfully configured rp_filter to loose mode(2) on vx-submariner
  5341. 2020-10-08T11:45:19.3266560Z [cluster2] I1008 11:37:11.229590 1 route.go:736] Enqueueing endpoint for route controller {"metadata":{"name":"cluster2-submariner-cable-cluster2-172-17-0-5","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster2-submariner-cable-cluster2-172-17-0-5","uid":"6a117963-5044-4b34-9df6-57ebb8a4981d","resourceVersion":"1284","generation":1,"creationTimestamp":"2020-10-08T11:35:26Z"},"spec":{"cluster_id":"cluster2","cable_name":"submariner-cable-cluster2-172-17-0-5","hostname":"cluster2-worker","subnets":["100.92.0.0/16","10.242.0.0/16"],"private_ip":"172.17.0.5","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
  5342. 2020-10-08T11:45:19.3274595Z [cluster2] I1008 11:37:11.229657 1 route.go:736] Enqueueing endpoint for route controller {"metadata":{"name":"cluster1-submariner-cable-cluster1-172-17-0-9","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster1-submariner-cable-cluster1-172-17-0-9","uid":"60d04732-7b08-4685-b72b-cb2c12702817","resourceVersion":"1285","generation":1,"creationTimestamp":"2020-10-08T11:35:26Z"},"spec":{"cluster_id":"cluster1","cable_name":"submariner-cable-cluster1-172-17-0-9","hostname":"cluster1-worker","subnets":["100.91.0.0/16","10.241.0.0/16"],"private_ip":"172.17.0.9","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
  5343. 2020-10-08T11:45:19.3279091Z [cluster2] I1008 11:37:11.237741 1 route.go:634] Processing local Endpoint cluster2-submariner-cable-cluster2-172-17-0-5 with IP 172.17.0.5, Host cluster2-worker for this cluster
  5344. 2020-10-08T11:45:19.3280378Z [cluster2] I1008 11:37:11.237966 1 route.go:851] Processing route {Ifindex: 16 Dst: 240.0.0.0/8 Src: 240.17.0.5 Gw: <nil> Flags: [] Table: 254}
  5345. 2020-10-08T11:45:19.3280944Z [cluster2] I1008 11:37:11.237992 1 route.go:854] Found nil gw or dst
  5346. 2020-10-08T11:45:19.3281479Z [cluster2] I1008 11:37:11.238000 1 route.go:645] This route agent is running on the active gateway node
  5347. 2020-10-08T11:45:19.3282968Z [cluster2] I1008 11:37:11.238007 1 route.go:653] Creating the vxlan interface: vx-submariner on the gateway node
  5348. 2020-10-08T11:45:19.3283719Z [cluster2] I1008 11:37:11.238138 1 vxlan.go:61] VxLAN interface already exists with same configuration.
  5349. 2020-10-08T11:45:19.3284903Z [cluster2] I1008 11:37:11.238172 1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.4 00:00:00:00:00:00
  5350. 2020-10-08T11:45:19.3285977Z [cluster2] I1008 11:37:11.238195 1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.5 00:00:00:00:00:00
  5351. 2020-10-08T11:45:19.3287064Z [cluster2] I1008 11:37:11.238237 1 route.go:478] Successfully configured rp_filter to loose mode(2) on vx-submariner
  5352. 2020-10-08T11:45:19.3288739Z [cluster2] I1008 11:37:11.239922 1 route.go:620] Updating iptable rules for remote cluster's Endpoint "cluster1-submariner-cable-cluster1-172-17-0-9" with subnets [100.91.0.0/16 10.241.0.0/16]
  5353. 2020-10-08T11:45:19.3293932Z [cluster2] I1008 11:37:41.229720 1 route.go:736] Enqueueing endpoint for route controller {"metadata":{"name":"cluster2-submariner-cable-cluster2-172-17-0-5","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster2-submariner-cable-cluster2-172-17-0-5","uid":"6a117963-5044-4b34-9df6-57ebb8a4981d","resourceVersion":"1284","generation":1,"creationTimestamp":"2020-10-08T11:35:26Z"},"spec":{"cluster_id":"cluster2","cable_name":"submariner-cable-cluster2-172-17-0-5","hostname":"cluster2-worker","subnets":["100.92.0.0/16","10.242.0.0/16"],"private_ip":"172.17.0.5","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
  5354. 2020-10-08T11:45:19.3310150Z [cluster2] I1008 11:37:41.229807 1 route.go:736] Enqueueing endpoint for route controller {"metadata":{"name":"cluster1-submariner-cable-cluster1-172-17-0-9","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster1-submariner-cable-cluster1-172-17-0-9","uid":"60d04732-7b08-4685-b72b-cb2c12702817","resourceVersion":"1285","generation":1,"creationTimestamp":"2020-10-08T11:35:26Z"},"spec":{"cluster_id":"cluster1","cable_name":"submariner-cable-cluster1-172-17-0-9","hostname":"cluster1-worker","subnets":["100.91.0.0/16","10.241.0.0/16"],"private_ip":"172.17.0.9","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
  5355. 2020-10-08T11:45:19.3313862Z [cluster2] I1008 11:37:41.237285 1 route.go:634] Processing local Endpoint cluster2-submariner-cable-cluster2-172-17-0-5 with IP 172.17.0.5, Host cluster2-worker for this cluster
  5356. 2020-10-08T11:45:19.3314987Z [cluster2] I1008 11:37:41.237460 1 route.go:851] Processing route {Ifindex: 16 Dst: 240.0.0.0/8 Src: 240.17.0.5 Gw: <nil> Flags: [] Table: 254}
  5357. 2020-10-08T11:45:19.3315487Z [cluster2] I1008 11:37:41.237480 1 route.go:854] Found nil gw or dst
  5358. 2020-10-08T11:45:19.3315940Z [cluster2] I1008 11:37:41.237489 1 route.go:645] This route agent is running on the active gateway node
  5359. 2020-10-08T11:45:19.3316741Z [cluster2] I1008 11:37:41.237497 1 route.go:653] Creating the vxlan interface: vx-submariner on the gateway node
  5360. 2020-10-08T11:45:19.3317394Z [cluster2] I1008 11:37:41.237608 1 vxlan.go:61] VxLAN interface already exists with same configuration.
  5361. 2020-10-08T11:45:19.3318229Z [cluster2] I1008 11:37:41.237637 1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.4 00:00:00:00:00:00
  5362. 2020-10-08T11:45:19.3318854Z [cluster2] I1008 11:37:41.237661 1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.5 00:00:00:00:00:00
  5363. 2020-10-08T11:45:19.3319783Z [cluster2] I1008 11:37:41.237717 1 route.go:478] Successfully configured rp_filter to loose mode(2) on vx-submariner
  5364. 2020-10-08T11:45:19.3321368Z [cluster2] I1008 11:37:41.239585 1 route.go:620] Updating iptable rules for remote cluster's Endpoint "cluster1-submariner-cable-cluster1-172-17-0-9" with subnets [100.91.0.0/16 10.241.0.0/16]
  5365. 2020-10-08T11:45:19.3326022Z [cluster2] I1008 11:38:11.229885 1 route.go:736] Enqueueing endpoint for route controller {"metadata":{"name":"cluster2-submariner-cable-cluster2-172-17-0-5","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster2-submariner-cable-cluster2-172-17-0-5","uid":"6a117963-5044-4b34-9df6-57ebb8a4981d","resourceVersion":"1284","generation":1,"creationTimestamp":"2020-10-08T11:35:26Z"},"spec":{"cluster_id":"cluster2","cable_name":"submariner-cable-cluster2-172-17-0-5","hostname":"cluster2-worker","subnets":["100.92.0.0/16","10.242.0.0/16"],"private_ip":"172.17.0.5","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
  5366. 2020-10-08T11:45:19.3333303Z [cluster2] I1008 11:38:11.229949 1 route.go:736] Enqueueing endpoint for route controller {"metadata":{"name":"cluster1-submariner-cable-cluster1-172-17-0-9","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster1-submariner-cable-cluster1-172-17-0-9","uid":"60d04732-7b08-4685-b72b-cb2c12702817","resourceVersion":"1285","generation":1,"creationTimestamp":"2020-10-08T11:35:26Z"},"spec":{"cluster_id":"cluster1","cable_name":"submariner-cable-cluster1-172-17-0-9","hostname":"cluster1-worker","subnets":["100.91.0.0/16","10.241.0.0/16"],"private_ip":"172.17.0.9","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
  5367. 2020-10-08T11:45:19.3336965Z [cluster2] I1008 11:38:11.237774 1 route.go:634] Processing local Endpoint cluster2-submariner-cable-cluster2-172-17-0-5 with IP 172.17.0.5, Host cluster2-worker for this cluster
  5368. 2020-10-08T11:45:19.3338085Z [cluster2] I1008 11:38:11.237929 1 route.go:851] Processing route {Ifindex: 16 Dst: 240.0.0.0/8 Src: 240.17.0.5 Gw: <nil> Flags: [] Table: 254}
  5369. 2020-10-08T11:45:19.3338636Z [cluster2] I1008 11:38:11.237948 1 route.go:854] Found nil gw or dst
  5370. 2020-10-08T11:45:19.3339103Z [cluster2] I1008 11:38:11.237956 1 route.go:645] This route agent is running on the active gateway node
  5371. 2020-10-08T11:45:19.3339910Z [cluster2] I1008 11:38:11.237963 1 route.go:653] Creating the vxlan interface: vx-submariner on the gateway node
  5372. 2020-10-08T11:45:19.3340563Z [cluster2] I1008 11:38:11.238072 1 vxlan.go:61] VxLAN interface already exists with same configuration.
  5373. 2020-10-08T11:45:19.3341159Z [cluster2] I1008 11:38:11.238098 1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.4 00:00:00:00:00:00
  5374. 2020-10-08T11:45:19.3341709Z [cluster2] I1008 11:38:11.238121 1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.5 00:00:00:00:00:00
  5375. 2020-10-08T11:45:19.3342517Z [cluster2] I1008 11:38:11.238158 1 route.go:478] Successfully configured rp_filter to loose mode(2) on vx-submariner
  5376. 2020-10-08T11:45:19.3343912Z [cluster2] I1008 11:38:11.240230 1 route.go:620] Updating iptable rules for remote cluster's Endpoint "cluster1-submariner-cable-cluster1-172-17-0-9" with subnets [100.91.0.0/16 10.241.0.0/16]
  5377. 2020-10-08T11:45:19.3351665Z [cluster2] I1008 11:38:41.230013 1 route.go:736] Enqueueing endpoint for route controller {"metadata":{"name":"cluster2-submariner-cable-cluster2-172-17-0-5","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster2-submariner-cable-cluster2-172-17-0-5","uid":"6a117963-5044-4b34-9df6-57ebb8a4981d","resourceVersion":"1284","generation":1,"creationTimestamp":"2020-10-08T11:35:26Z"},"spec":{"cluster_id":"cluster2","cable_name":"submariner-cable-cluster2-172-17-0-5","hostname":"cluster2-worker","subnets":["100.92.0.0/16","10.242.0.0/16"],"private_ip":"172.17.0.5","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
  5378. 2020-10-08T11:45:19.3358233Z [cluster2] I1008 11:38:41.230084 1 route.go:736] Enqueueing endpoint for route controller {"metadata":{"name":"cluster1-submariner-cable-cluster1-172-17-0-9","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster1-submariner-cable-cluster1-172-17-0-9","uid":"60d04732-7b08-4685-b72b-cb2c12702817","resourceVersion":"1285","generation":1,"creationTimestamp":"2020-10-08T11:35:26Z"},"spec":{"cluster_id":"cluster1","cable_name":"submariner-cable-cluster1-172-17-0-9","hostname":"cluster1-worker","subnets":["100.91.0.0/16","10.241.0.0/16"],"private_ip":"172.17.0.9","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
  5379. 2020-10-08T11:45:19.3362482Z [cluster2] I1008 11:38:41.238978 1 route.go:634] Processing local Endpoint cluster2-submariner-cable-cluster2-172-17-0-5 with IP 172.17.0.5, Host cluster2-worker for this cluster
  5380. 2020-10-08T11:45:19.3363881Z [cluster2] I1008 11:38:41.239153 1 route.go:851] Processing route {Ifindex: 16 Dst: 240.0.0.0/8 Src: 240.17.0.5 Gw: <nil> Flags: [] Table: 254}
  5381. 2020-10-08T11:45:19.3364853Z [cluster2] I1008 11:38:41.239182 1 route.go:854] Found nil gw or dst
  5382. 2020-10-08T11:45:19.3365538Z [cluster2] I1008 11:38:41.239191 1 route.go:645] This route agent is running on the active gateway node
  5383. 2020-10-08T11:45:19.3366500Z [cluster2] I1008 11:38:41.239199 1 route.go:653] Creating the vxlan interface: vx-submariner on the gateway node
  5384. 2020-10-08T11:45:19.3367246Z [cluster2] I1008 11:38:41.239317 1 vxlan.go:61] VxLAN interface already exists with same configuration.
  5385. 2020-10-08T11:45:19.3367930Z [cluster2] I1008 11:38:41.239353 1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.4 00:00:00:00:00:00
  5386. 2020-10-08T11:45:19.3368539Z [cluster2] I1008 11:38:41.239384 1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.5 00:00:00:00:00:00
  5387. 2020-10-08T11:45:19.3369465Z [cluster2] I1008 11:38:41.239441 1 route.go:478] Successfully configured rp_filter to loose mode(2) on vx-submariner
  5388. 2020-10-08T11:45:19.3371158Z [cluster2] I1008 11:38:41.240835 1 route.go:620] Updating iptable rules for remote cluster's Endpoint "cluster1-submariner-cable-cluster1-172-17-0-9" with subnets [100.91.0.0/16 10.241.0.0/16]
  5389. 2020-10-08T11:45:19.3377387Z [cluster2] I1008 11:39:11.230146 1 route.go:736] Enqueueing endpoint for route controller {"metadata":{"name":"cluster2-submariner-cable-cluster2-172-17-0-5","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster2-submariner-cable-cluster2-172-17-0-5","uid":"6a117963-5044-4b34-9df6-57ebb8a4981d","resourceVersion":"1284","generation":1,"creationTimestamp":"2020-10-08T11:35:26Z"},"spec":{"cluster_id":"cluster2","cable_name":"submariner-cable-cluster2-172-17-0-5","hostname":"cluster2-worker","subnets":["100.92.0.0/16","10.242.0.0/16"],"private_ip":"172.17.0.5","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
  5390. 2020-10-08T11:45:19.3384470Z [cluster2] I1008 11:39:11.230244 1 route.go:736] Enqueueing endpoint for route controller {"metadata":{"name":"cluster1-submariner-cable-cluster1-172-17-0-9","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster1-submariner-cable-cluster1-172-17-0-9","uid":"60d04732-7b08-4685-b72b-cb2c12702817","resourceVersion":"1285","generation":1,"creationTimestamp":"2020-10-08T11:35:26Z"},"spec":{"cluster_id":"cluster1","cable_name":"submariner-cable-cluster1-172-17-0-9","hostname":"cluster1-worker","subnets":["100.91.0.0/16","10.241.0.0/16"],"private_ip":"172.17.0.9","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
  5391. 2020-10-08T11:45:19.3389916Z [cluster2] I1008 11:39:11.237599 1 route.go:634] Processing local Endpoint cluster2-submariner-cable-cluster2-172-17-0-5 with IP 172.17.0.5, Host cluster2-worker for this cluster
  5392. 2020-10-08T11:45:19.3391210Z [cluster2] I1008 11:39:11.237777 1 route.go:851] Processing route {Ifindex: 16 Dst: 240.0.0.0/8 Src: 240.17.0.5 Gw: <nil> Flags: [] Table: 254}
  5393. 2020-10-08T11:45:19.3391790Z [cluster2] I1008 11:39:11.237805 1 route.go:854] Found nil gw or dst
  5394. 2020-10-08T11:45:19.3392499Z [cluster2] I1008 11:39:11.237813 1 route.go:645] This route agent is running on the active gateway node
  5395. 2020-10-08T11:45:19.3393469Z [cluster2] I1008 11:39:11.237822 1 route.go:653] Creating the vxlan interface: vx-submariner on the gateway node
  5396. 2020-10-08T11:45:19.3394686Z [cluster2] I1008 11:39:11.237938 1 vxlan.go:61] VxLAN interface already exists with same configuration.
  5397. 2020-10-08T11:45:19.3395511Z [cluster2] I1008 11:39:11.237976 1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.4 00:00:00:00:00:00
  5398. 2020-10-08T11:45:19.3396518Z [cluster2] I1008 11:39:11.238000 1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.5 00:00:00:00:00:00
  5399. 2020-10-08T11:45:19.3398497Z [cluster2] I1008 11:39:11.238052 1 route.go:478] Successfully configured rp_filter to loose mode(2) on vx-submariner
  5400. 2020-10-08T11:45:19.3400286Z [cluster2] I1008 11:39:11.239826 1 route.go:620] Updating iptable rules for remote cluster's Endpoint "cluster1-submariner-cable-cluster1-172-17-0-9" with subnets [100.91.0.0/16 10.241.0.0/16]
  5401. 2020-10-08T11:45:19.3405228Z [cluster2] I1008 11:39:41.230274 1 route.go:736] Enqueueing endpoint for route controller {"metadata":{"name":"cluster2-submariner-cable-cluster2-172-17-0-5","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster2-submariner-cable-cluster2-172-17-0-5","uid":"6a117963-5044-4b34-9df6-57ebb8a4981d","resourceVersion":"1284","generation":1,"creationTimestamp":"2020-10-08T11:35:26Z"},"spec":{"cluster_id":"cluster2","cable_name":"submariner-cable-cluster2-172-17-0-5","hostname":"cluster2-worker","subnets":["100.92.0.0/16","10.242.0.0/16"],"private_ip":"172.17.0.5","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
  5402. 2020-10-08T11:45:19.3413209Z [cluster2] I1008 11:39:41.230350 1 route.go:736] Enqueueing endpoint for route controller {"metadata":{"name":"cluster1-submariner-cable-cluster1-172-17-0-9","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster1-submariner-cable-cluster1-172-17-0-9","uid":"60d04732-7b08-4685-b72b-cb2c12702817","resourceVersion":"1285","generation":1,"creationTimestamp":"2020-10-08T11:35:26Z"},"spec":{"cluster_id":"cluster1","cable_name":"submariner-cable-cluster1-172-17-0-9","hostname":"cluster1-worker","subnets":["100.91.0.0/16","10.241.0.0/16"],"private_ip":"172.17.0.9","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
  5403. 2020-10-08T11:45:19.3419586Z [cluster2] I1008 11:39:41.237810 1 route.go:634] Processing local Endpoint cluster2-submariner-cable-cluster2-172-17-0-5 with IP 172.17.0.5, Host cluster2-worker for this cluster
  5404. 2020-10-08T11:45:19.3420949Z [cluster2] I1008 11:39:41.237993 1 route.go:851] Processing route {Ifindex: 16 Dst: 240.0.0.0/8 Src: 240.17.0.5 Gw: <nil> Flags: [] Table: 254}
  5405. 2020-10-08T11:45:19.3421455Z [cluster2] I1008 11:39:41.238019 1 route.go:854] Found nil gw or dst
  5406. 2020-10-08T11:45:19.3421923Z [cluster2] I1008 11:39:41.238027 1 route.go:645] This route agent is running on the active gateway node
  5407. 2020-10-08T11:45:19.3422724Z [cluster2] I1008 11:39:41.238034 1 route.go:653] Creating the vxlan interface: vx-submariner on the gateway node
  5408. 2020-10-08T11:45:19.3423375Z [cluster2] I1008 11:39:41.238137 1 vxlan.go:61] VxLAN interface already exists with same configuration.
  5409. 2020-10-08T11:45:19.3423980Z [cluster2] I1008 11:39:41.238172 1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.4 00:00:00:00:00:00
  5410. 2020-10-08T11:45:19.3424531Z [cluster2] I1008 11:39:41.238202 1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.5 00:00:00:00:00:00
  5411. 2020-10-08T11:45:19.3425341Z [cluster2] I1008 11:39:41.238243 1 route.go:478] Successfully configured rp_filter to loose mode(2) on vx-submariner
  5412. 2020-10-08T11:45:19.3427272Z [cluster2] I1008 11:39:41.239940 1 route.go:620] Updating iptable rules for remote cluster's Endpoint "cluster1-submariner-cable-cluster1-172-17-0-9" with subnets [100.91.0.0/16 10.241.0.0/16]
  5413. 2020-10-08T11:45:19.3431361Z [cluster2] I1008 11:40:11.230479 1 route.go:736] Enqueueing endpoint for route controller {"metadata":{"name":"cluster2-submariner-cable-cluster2-172-17-0-5","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster2-submariner-cable-cluster2-172-17-0-5","uid":"6a117963-5044-4b34-9df6-57ebb8a4981d","resourceVersion":"1284","generation":1,"creationTimestamp":"2020-10-08T11:35:26Z"},"spec":{"cluster_id":"cluster2","cable_name":"submariner-cable-cluster2-172-17-0-5","hostname":"cluster2-worker","subnets":["100.92.0.0/16","10.242.0.0/16"],"private_ip":"172.17.0.5","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
  5414. 2020-10-08T11:45:19.3439373Z [cluster2] I1008 11:40:11.230546 1 route.go:736] Enqueueing endpoint for route controller {"metadata":{"name":"cluster1-submariner-cable-cluster1-172-17-0-9","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster1-submariner-cable-cluster1-172-17-0-9","uid":"60d04732-7b08-4685-b72b-cb2c12702817","resourceVersion":"1285","generation":1,"creationTimestamp":"2020-10-08T11:35:26Z"},"spec":{"cluster_id":"cluster1","cable_name":"submariner-cable-cluster1-172-17-0-9","hostname":"cluster1-worker","subnets":["100.91.0.0/16","10.241.0.0/16"],"private_ip":"172.17.0.9","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
  5415. 2020-10-08T11:45:19.3443949Z [cluster2] I1008 11:40:11.238437 1 route.go:634] Processing local Endpoint cluster2-submariner-cable-cluster2-172-17-0-5 with IP 172.17.0.5, Host cluster2-worker for this cluster
  5416. 2020-10-08T11:45:19.3445402Z [cluster2] I1008 11:40:11.238618 1 route.go:851] Processing route {Ifindex: 16 Dst: 240.0.0.0/8 Src: 240.17.0.5 Gw: <nil> Flags: [] Table: 254}
  5417. 2020-10-08T11:45:19.3446032Z [cluster2] I1008 11:40:11.238646 1 route.go:854] Found nil gw or dst
  5418. 2020-10-08T11:45:19.3446571Z [cluster2] I1008 11:40:11.238654 1 route.go:645] This route agent is running on the active gateway node
  5419. 2020-10-08T11:45:19.3447504Z [cluster2] I1008 11:40:11.238662 1 route.go:653] Creating the vxlan interface: vx-submariner on the gateway node
  5420. 2020-10-08T11:45:19.3448249Z [cluster2] I1008 11:40:11.238770 1 vxlan.go:61] VxLAN interface already exists with same configuration.
  5421. 2020-10-08T11:45:19.3449001Z [cluster2] I1008 11:40:11.238807 1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.5 00:00:00:00:00:00
  5422. 2020-10-08T11:45:19.3449546Z [cluster2] I1008 11:40:11.238833 1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.4 00:00:00:00:00:00
  5423. 2020-10-08T11:45:19.3450358Z [cluster2] I1008 11:40:11.238886 1 route.go:478] Successfully configured rp_filter to loose mode(2) on vx-submariner
  5424. 2020-10-08T11:45:19.3451751Z [cluster2] I1008 11:40:11.243461 1 route.go:620] Updating iptable rules for remote cluster's Endpoint "cluster1-submariner-cable-cluster1-172-17-0-9" with subnets [100.91.0.0/16 10.241.0.0/16]
  5425. 2020-10-08T11:45:19.3456878Z [cluster2] I1008 11:40:41.230632 1 route.go:736] Enqueueing endpoint for route controller {"metadata":{"name":"cluster1-submariner-cable-cluster1-172-17-0-9","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster1-submariner-cable-cluster1-172-17-0-9","uid":"60d04732-7b08-4685-b72b-cb2c12702817","resourceVersion":"1285","generation":1,"creationTimestamp":"2020-10-08T11:35:26Z"},"spec":{"cluster_id":"cluster1","cable_name":"submariner-cable-cluster1-172-17-0-9","hostname":"cluster1-worker","subnets":["100.91.0.0/16","10.241.0.0/16"],"private_ip":"172.17.0.9","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
  5426. 2020-10-08T11:45:19.3464737Z [cluster2] I1008 11:40:41.230697 1 route.go:736] Enqueueing endpoint for route controller {"metadata":{"name":"cluster2-submariner-cable-cluster2-172-17-0-5","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster2-submariner-cable-cluster2-172-17-0-5","uid":"6a117963-5044-4b34-9df6-57ebb8a4981d","resourceVersion":"1284","generation":1,"creationTimestamp":"2020-10-08T11:35:26Z"},"spec":{"cluster_id":"cluster2","cable_name":"submariner-cable-cluster2-172-17-0-5","hostname":"cluster2-worker","subnets":["100.92.0.0/16","10.242.0.0/16"],"private_ip":"172.17.0.5","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
  5427. 2020-10-08T11:45:19.3469200Z [cluster2] I1008 11:40:41.238873 1 route.go:620] Updating iptable rules for remote cluster's Endpoint "cluster1-submariner-cable-cluster1-172-17-0-9" with subnets [100.91.0.0/16 10.241.0.0/16]
  5428. 2020-10-08T11:45:19.3471312Z [cluster2] I1008 11:40:41.242478 1 route.go:634] Processing local Endpoint cluster2-submariner-cable-cluster2-172-17-0-5 with IP 172.17.0.5, Host cluster2-worker for this cluster
  5429. 2020-10-08T11:45:19.3472658Z [cluster2] I1008 11:40:41.243020 1 route.go:851] Processing route {Ifindex: 16 Dst: 240.0.0.0/8 Src: 240.17.0.5 Gw: <nil> Flags: [] Table: 254}
  5430. 2020-10-08T11:45:19.3473220Z [cluster2] I1008 11:40:41.243068 1 route.go:854] Found nil gw or dst
  5431. 2020-10-08T11:45:19.3473756Z [cluster2] I1008 11:40:41.243077 1 route.go:645] This route agent is running on the active gateway node
  5432. 2020-10-08T11:45:19.3475161Z [cluster2] I1008 11:40:41.243085 1 route.go:653] Creating the vxlan interface: vx-submariner on the gateway node
  5433. 2020-10-08T11:45:19.3476059Z [cluster2] I1008 11:40:41.243229 1 vxlan.go:61] VxLAN interface already exists with same configuration.
  5434. 2020-10-08T11:45:19.3476748Z [cluster2] I1008 11:40:41.243267 1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.4 00:00:00:00:00:00
  5435. 2020-10-08T11:45:19.3477374Z [cluster2] I1008 11:40:41.243562 1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.5 00:00:00:00:00:00
  5436. 2020-10-08T11:45:19.3478967Z [cluster2] I1008 11:40:41.243696 1 route.go:478] Successfully configured rp_filter to loose mode(2) on vx-submariner
  5437. 2020-10-08T11:45:19.3483137Z [cluster2] I1008 11:41:11.230751 1 route.go:736] Enqueueing endpoint for route controller {"metadata":{"name":"cluster2-submariner-cable-cluster2-172-17-0-5","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster2-submariner-cable-cluster2-172-17-0-5","uid":"6a117963-5044-4b34-9df6-57ebb8a4981d","resourceVersion":"1284","generation":1,"creationTimestamp":"2020-10-08T11:35:26Z"},"spec":{"cluster_id":"cluster2","cable_name":"submariner-cable-cluster2-172-17-0-5","hostname":"cluster2-worker","subnets":["100.92.0.0/16","10.242.0.0/16"],"private_ip":"172.17.0.5","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
  5438. 2020-10-08T11:45:19.3489021Z [cluster2] I1008 11:41:11.230870 1 route.go:736] Enqueueing endpoint for route controller {"metadata":{"name":"cluster1-submariner-cable-cluster1-172-17-0-9","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster1-submariner-cable-cluster1-172-17-0-9","uid":"60d04732-7b08-4685-b72b-cb2c12702817","resourceVersion":"1285","generation":1,"creationTimestamp":"2020-10-08T11:35:26Z"},"spec":{"cluster_id":"cluster1","cable_name":"submariner-cable-cluster1-172-17-0-9","hostname":"cluster1-worker","subnets":["100.91.0.0/16","10.241.0.0/16"],"private_ip":"172.17.0.9","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
  5439. 2020-10-08T11:45:19.3492820Z [cluster2] I1008 11:41:11.238203 1 route.go:634] Processing local Endpoint cluster2-submariner-cable-cluster2-172-17-0-5 with IP 172.17.0.5, Host cluster2-worker for this cluster
  5440. 2020-10-08T11:45:19.3493935Z [cluster2] I1008 11:41:11.238381 1 route.go:851] Processing route {Ifindex: 16 Dst: 240.0.0.0/8 Src: 240.17.0.5 Gw: <nil> Flags: [] Table: 254}
  5441. 2020-10-08T11:45:19.3494432Z [cluster2] I1008 11:41:11.238406 1 route.go:854] Found nil gw or dst
  5442. 2020-10-08T11:45:19.3495073Z [cluster2] I1008 11:41:11.238414 1 route.go:645] This route agent is running on the active gateway node
  5443. 2020-10-08T11:45:19.3495999Z [cluster2] I1008 11:41:11.238422 1 route.go:653] Creating the vxlan interface: vx-submariner on the gateway node
  5444. 2020-10-08T11:45:19.3496744Z [cluster2] I1008 11:41:11.238524 1 vxlan.go:61] VxLAN interface already exists with same configuration.
  5445. 2020-10-08T11:45:19.3497429Z [cluster2] I1008 11:41:11.238558 1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.5 00:00:00:00:00:00
  5446. 2020-10-08T11:45:19.3498388Z [cluster2] I1008 11:41:11.238588 1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.4 00:00:00:00:00:00
  5447. 2020-10-08T11:45:19.3499476Z [cluster2] I1008 11:41:11.238636 1 route.go:478] Successfully configured rp_filter to loose mode(2) on vx-submariner
  5448. 2020-10-08T11:45:19.3501147Z [cluster2] I1008 11:41:11.240299 1 route.go:620] Updating iptable rules for remote cluster's Endpoint "cluster1-submariner-cable-cluster1-172-17-0-9" with subnets [100.91.0.0/16 10.241.0.0/16]
  5449. 2020-10-08T11:45:19.3505632Z [cluster2] I1008 11:41:41.230838 1 route.go:736] Enqueueing endpoint for route controller {"metadata":{"name":"cluster2-submariner-cable-cluster2-172-17-0-5","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster2-submariner-cable-cluster2-172-17-0-5","uid":"6a117963-5044-4b34-9df6-57ebb8a4981d","resourceVersion":"1284","generation":1,"creationTimestamp":"2020-10-08T11:35:26Z"},"spec":{"cluster_id":"cluster2","cable_name":"submariner-cable-cluster2-172-17-0-5","hostname":"cluster2-worker","subnets":["100.92.0.0/16","10.242.0.0/16"],"private_ip":"172.17.0.5","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
  5450. 2020-10-08T11:45:19.3511819Z [cluster2] I1008 11:41:41.231100 1 route.go:736] Enqueueing endpoint for route controller {"metadata":{"name":"cluster1-submariner-cable-cluster1-172-17-0-9","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster1-submariner-cable-cluster1-172-17-0-9","uid":"60d04732-7b08-4685-b72b-cb2c12702817","resourceVersion":"1285","generation":1,"creationTimestamp":"2020-10-08T11:35:26Z"},"spec":{"cluster_id":"cluster1","cable_name":"submariner-cable-cluster1-172-17-0-9","hostname":"cluster1-worker","subnets":["100.91.0.0/16","10.241.0.0/16"],"private_ip":"172.17.0.9","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
  5451. 2020-10-08T11:45:19.3517568Z [cluster2] I1008 11:41:41.238698 1 route.go:634] Processing local Endpoint cluster2-submariner-cable-cluster2-172-17-0-5 with IP 172.17.0.5, Host cluster2-worker for this cluster
  5452. 2020-10-08T11:45:19.3519054Z [cluster2] I1008 11:41:41.239035 1 route.go:851] Processing route {Ifindex: 16 Dst: 240.0.0.0/8 Src: 240.17.0.5 Gw: <nil> Flags: [] Table: 254}
  5453. 2020-10-08T11:45:19.3519564Z [cluster2] I1008 11:41:41.239067 1 route.go:854] Found nil gw or dst
  5454. 2020-10-08T11:45:19.3520021Z [cluster2] I1008 11:41:41.239076 1 route.go:645] This route agent is running on the active gateway node
  5455. 2020-10-08T11:45:19.3520864Z [cluster2] I1008 11:41:41.239084 1 route.go:653] Creating the vxlan interface: vx-submariner on the gateway node
  5456. 2020-10-08T11:45:19.3521517Z [cluster2] I1008 11:41:41.239287 1 vxlan.go:61] VxLAN interface already exists with same configuration.
  5457. 2020-10-08T11:45:19.3522117Z [cluster2] I1008 11:41:41.239402 1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.4 00:00:00:00:00:00
  5458. 2020-10-08T11:45:19.3522662Z [cluster2] I1008 11:41:41.239440 1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.5 00:00:00:00:00:00
  5459. 2020-10-08T11:45:19.3523476Z [cluster2] I1008 11:41:41.239592 1 route.go:478] Successfully configured rp_filter to loose mode(2) on vx-submariner
  5460. 2020-10-08T11:45:19.3524854Z [cluster2] I1008 11:41:41.241548 1 route.go:620] Updating iptable rules for remote cluster's Endpoint "cluster1-submariner-cable-cluster1-172-17-0-9" with subnets [100.91.0.0/16 10.241.0.0/16]
  5461. 2020-10-08T11:45:19.3528906Z [cluster2] I1008 11:42:11.231051 1 route.go:736] Enqueueing endpoint for route controller {"metadata":{"name":"cluster2-submariner-cable-cluster2-172-17-0-5","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster2-submariner-cable-cluster2-172-17-0-5","uid":"6a117963-5044-4b34-9df6-57ebb8a4981d","resourceVersion":"1284","generation":1,"creationTimestamp":"2020-10-08T11:35:26Z"},"spec":{"cluster_id":"cluster2","cable_name":"submariner-cable-cluster2-172-17-0-5","hostname":"cluster2-worker","subnets":["100.92.0.0/16","10.242.0.0/16"],"private_ip":"172.17.0.5","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
  5462. 2020-10-08T11:45:19.3534976Z [cluster2] I1008 11:42:11.231136 1 route.go:736] Enqueueing endpoint for route controller {"metadata":{"name":"cluster1-submariner-cable-cluster1-172-17-0-9","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster1-submariner-cable-cluster1-172-17-0-9","uid":"60d04732-7b08-4685-b72b-cb2c12702817","resourceVersion":"1285","generation":1,"creationTimestamp":"2020-10-08T11:35:26Z"},"spec":{"cluster_id":"cluster1","cable_name":"submariner-cable-cluster1-172-17-0-9","hostname":"cluster1-worker","subnets":["100.91.0.0/16","10.241.0.0/16"],"private_ip":"172.17.0.9","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
  5463. 2020-10-08T11:45:19.3539206Z [cluster2] I1008 11:42:11.238731 1 route.go:634] Processing local Endpoint cluster2-submariner-cable-cluster2-172-17-0-5 with IP 172.17.0.5, Host cluster2-worker for this cluster
  5464. 2020-10-08T11:45:19.3540327Z [cluster2] I1008 11:42:11.238903 1 route.go:851] Processing route {Ifindex: 16 Dst: 240.0.0.0/8 Src: 240.17.0.5 Gw: <nil> Flags: [] Table: 254}
  5465. 2020-10-08T11:45:19.3540817Z [cluster2] I1008 11:42:11.238929 1 route.go:854] Found nil gw or dst
  5466. 2020-10-08T11:45:19.3541367Z [cluster2] I1008 11:42:11.238937 1 route.go:645] This route agent is running on the active gateway node
  5467. 2020-10-08T11:45:19.3542169Z [cluster2] I1008 11:42:11.238953 1 route.go:653] Creating the vxlan interface: vx-submariner on the gateway node
  5468. 2020-10-08T11:45:19.3542821Z [cluster2] I1008 11:42:11.239063 1 vxlan.go:61] VxLAN interface already exists with same configuration.
  5469. 2020-10-08T11:45:19.3543416Z [cluster2] I1008 11:42:11.239097 1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.4 00:00:00:00:00:00
  5470. 2020-10-08T11:45:19.3543965Z [cluster2] I1008 11:42:11.239129 1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.5 00:00:00:00:00:00
  5471. 2020-10-08T11:45:19.3544775Z [cluster2] I1008 11:42:11.239175 1 route.go:478] Successfully configured rp_filter to loose mode(2) on vx-submariner
  5472. 2020-10-08T11:45:19.3546623Z [cluster2] I1008 11:42:11.240777 1 route.go:620] Updating iptable rules for remote cluster's Endpoint "cluster1-submariner-cable-cluster1-172-17-0-9" with subnets [100.91.0.0/16 10.241.0.0/16]
  5473. 2020-10-08T11:45:19.3550709Z [cluster2] I1008 11:42:41.231189 1 route.go:736] Enqueueing endpoint for route controller {"metadata":{"name":"cluster1-submariner-cable-cluster1-172-17-0-9","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster1-submariner-cable-cluster1-172-17-0-9","uid":"60d04732-7b08-4685-b72b-cb2c12702817","resourceVersion":"1285","generation":1,"creationTimestamp":"2020-10-08T11:35:26Z"},"spec":{"cluster_id":"cluster1","cable_name":"submariner-cable-cluster1-172-17-0-9","hostname":"cluster1-worker","subnets":["100.91.0.0/16","10.241.0.0/16"],"private_ip":"172.17.0.9","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
  5474. 2020-10-08T11:45:19.3556572Z [cluster2] I1008 11:42:41.231257 1 route.go:736] Enqueueing endpoint for route controller {"metadata":{"name":"cluster2-submariner-cable-cluster2-172-17-0-5","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster2-submariner-cable-cluster2-172-17-0-5","uid":"6a117963-5044-4b34-9df6-57ebb8a4981d","resourceVersion":"1284","generation":1,"creationTimestamp":"2020-10-08T11:35:26Z"},"spec":{"cluster_id":"cluster2","cable_name":"submariner-cable-cluster2-172-17-0-5","hostname":"cluster2-worker","subnets":["100.92.0.0/16","10.242.0.0/16"],"private_ip":"172.17.0.5","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
  5475. 2020-10-08T11:45:19.3560237Z [cluster2] I1008 11:42:41.238611 1 route.go:620] Updating iptable rules for remote cluster's Endpoint "cluster1-submariner-cable-cluster1-172-17-0-9" with subnets [100.91.0.0/16 10.241.0.0/16]
  5476. 2020-10-08T11:45:19.3562073Z [cluster2] I1008 11:42:41.240210 1 route.go:634] Processing local Endpoint cluster2-submariner-cable-cluster2-172-17-0-5 with IP 172.17.0.5, Host cluster2-worker for this cluster
  5477. 2020-10-08T11:45:19.3563476Z [cluster2] I1008 11:42:41.240401 1 route.go:851] Processing route {Ifindex: 16 Dst: 240.0.0.0/8 Src: 240.17.0.5 Gw: <nil> Flags: [] Table: 254}
  5478. 2020-10-08T11:45:19.3563965Z [cluster2] I1008 11:42:41.240427 1 route.go:854] Found nil gw or dst
  5479. 2020-10-08T11:45:19.3564431Z [cluster2] I1008 11:42:41.240436 1 route.go:645] This route agent is running on the active gateway node
  5480. 2020-10-08T11:45:19.3565671Z [cluster2] I1008 11:42:41.240444 1 route.go:653] Creating the vxlan interface: vx-submariner on the gateway node
  5481. 2020-10-08T11:45:19.3566336Z [cluster2] I1008 11:42:41.240552 1 vxlan.go:61] VxLAN interface already exists with same configuration.
  5482. 2020-10-08T11:45:19.3566933Z [cluster2] I1008 11:42:41.240588 1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.4 00:00:00:00:00:00
  5483. 2020-10-08T11:45:19.3567468Z [cluster2] I1008 11:42:41.240613 1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.5 00:00:00:00:00:00
  5484. 2020-10-08T11:45:19.3568655Z [cluster2] I1008 11:42:41.240688 1 route.go:478] Successfully configured rp_filter to loose mode(2) on vx-submariner
  5485. 2020-10-08T11:45:19.3572571Z [cluster2] I1008 11:43:11.231413 1 route.go:736] Enqueueing endpoint for route controller {"metadata":{"name":"cluster2-submariner-cable-cluster2-172-17-0-5","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster2-submariner-cable-cluster2-172-17-0-5","uid":"6a117963-5044-4b34-9df6-57ebb8a4981d","resourceVersion":"1284","generation":1,"creationTimestamp":"2020-10-08T11:35:26Z"},"spec":{"cluster_id":"cluster2","cable_name":"submariner-cable-cluster2-172-17-0-5","hostname":"cluster2-worker","subnets":["100.92.0.0/16","10.242.0.0/16"],"private_ip":"172.17.0.5","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
  5486. 2020-10-08T11:45:19.3578653Z [cluster2] I1008 11:43:11.231522 1 route.go:736] Enqueueing endpoint for route controller {"metadata":{"name":"cluster1-submariner-cable-cluster1-172-17-0-9","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster1-submariner-cable-cluster1-172-17-0-9","uid":"60d04732-7b08-4685-b72b-cb2c12702817","resourceVersion":"1285","generation":1,"creationTimestamp":"2020-10-08T11:35:26Z"},"spec":{"cluster_id":"cluster1","cable_name":"submariner-cable-cluster1-172-17-0-9","hostname":"cluster1-worker","subnets":["100.91.0.0/16","10.241.0.0/16"],"private_ip":"172.17.0.9","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
  5487. 2020-10-08T11:45:19.3582323Z [cluster2] I1008 11:43:11.239357 1 route.go:634] Processing local Endpoint cluster2-submariner-cable-cluster2-172-17-0-5 with IP 172.17.0.5, Host cluster2-worker for this cluster
  5488. 2020-10-08T11:45:19.3583444Z [cluster2] I1008 11:43:11.239659 1 route.go:851] Processing route {Ifindex: 16 Dst: 240.0.0.0/8 Src: 240.17.0.5 Gw: <nil> Flags: [] Table: 254}
  5489. 2020-10-08T11:45:19.3584238Z [cluster2] I1008 11:43:11.239688 1 route.go:854] Found nil gw or dst
  5490. 2020-10-08T11:45:19.3584832Z [cluster2] I1008 11:43:11.239696 1 route.go:645] This route agent is running on the active gateway node
  5491. 2020-10-08T11:45:19.3585678Z [cluster2] I1008 11:43:11.239724 1 route.go:653] Creating the vxlan interface: vx-submariner on the gateway node
  5492. 2020-10-08T11:45:19.3586329Z [cluster2] I1008 11:43:11.239883 1 vxlan.go:61] VxLAN interface already exists with same configuration.
  5493. 2020-10-08T11:45:19.3586921Z [cluster2] I1008 11:43:11.239973 1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.4 00:00:00:00:00:00
  5494. 2020-10-08T11:45:19.3587469Z [cluster2] I1008 11:43:11.240046 1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.5 00:00:00:00:00:00
  5495. 2020-10-08T11:45:19.3588275Z [cluster2] I1008 11:43:11.240155 1 route.go:478] Successfully configured rp_filter to loose mode(2) on vx-submariner
  5496. 2020-10-08T11:45:19.3591337Z [cluster2] I1008 11:43:11.242855 1 route.go:620] Updating iptable rules for remote cluster's Endpoint "cluster1-submariner-cable-cluster1-172-17-0-9" with subnets [100.91.0.0/16 10.241.0.0/16]
  5497. 2020-10-08T11:45:19.3596282Z [cluster2] I1008 11:43:41.231602 1 route.go:736] Enqueueing endpoint for route controller {"metadata":{"name":"cluster2-submariner-cable-cluster2-172-17-0-5","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster2-submariner-cable-cluster2-172-17-0-5","uid":"6a117963-5044-4b34-9df6-57ebb8a4981d","resourceVersion":"1284","generation":1,"creationTimestamp":"2020-10-08T11:35:26Z"},"spec":{"cluster_id":"cluster2","cable_name":"submariner-cable-cluster2-172-17-0-5","hostname":"cluster2-worker","subnets":["100.92.0.0/16","10.242.0.0/16"],"private_ip":"172.17.0.5","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
  5498. 2020-10-08T11:45:19.3603867Z [cluster2] I1008 11:43:41.231691 1 route.go:736] Enqueueing endpoint for route controller {"metadata":{"name":"cluster1-submariner-cable-cluster1-172-17-0-9","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster1-submariner-cable-cluster1-172-17-0-9","uid":"60d04732-7b08-4685-b72b-cb2c12702817","resourceVersion":"1285","generation":1,"creationTimestamp":"2020-10-08T11:35:26Z"},"spec":{"cluster_id":"cluster1","cable_name":"submariner-cable-cluster1-172-17-0-9","hostname":"cluster1-worker","subnets":["100.91.0.0/16","10.241.0.0/16"],"private_ip":"172.17.0.9","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
  5499. 2020-10-08T11:45:19.3608411Z [cluster2] I1008 11:43:41.239111 1 route.go:634] Processing local Endpoint cluster2-submariner-cable-cluster2-172-17-0-5 with IP 172.17.0.5, Host cluster2-worker for this cluster
  5500. 2020-10-08T11:45:19.3609533Z [cluster2] I1008 11:43:41.239298 1 route.go:851] Processing route {Ifindex: 16 Dst: 240.0.0.0/8 Src: 240.17.0.5 Gw: <nil> Flags: [] Table: 254}
  5501. 2020-10-08T11:45:19.3610198Z [cluster2] I1008 11:43:41.239322 1 route.go:854] Found nil gw or dst
  5502. 2020-10-08T11:45:19.3610944Z [cluster2] I1008 11:43:41.239329 1 route.go:645] This route agent is running on the active gateway node
  5503. 2020-10-08T11:45:19.3611937Z [cluster2] I1008 11:43:41.239336 1 route.go:653] Creating the vxlan interface: vx-submariner on the gateway node
  5504. 2020-10-08T11:45:19.3613365Z [cluster2] I1008 11:43:41.239467 1 vxlan.go:61] VxLAN interface already exists with same configuration.
  5505. 2020-10-08T11:45:19.3614105Z [cluster2] I1008 11:43:41.239521 1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.5 00:00:00:00:00:00
  5506. 2020-10-08T11:45:19.3614784Z [cluster2] I1008 11:43:41.239544 1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.4 00:00:00:00:00:00
  5507. 2020-10-08T11:45:19.3615808Z [cluster2] I1008 11:43:41.239593 1 route.go:478] Successfully configured rp_filter to loose mode(2) on vx-submariner
  5508. 2020-10-08T11:45:19.3617498Z [cluster2] I1008 11:43:41.242832 1 route.go:620] Updating iptable rules for remote cluster's Endpoint "cluster1-submariner-cable-cluster1-172-17-0-9" with subnets [100.91.0.0/16 10.241.0.0/16]
  5509. 2020-10-08T11:45:19.3622650Z [cluster2] I1008 11:44:11.231751 1 route.go:736] Enqueueing endpoint for route controller {"metadata":{"name":"cluster2-submariner-cable-cluster2-172-17-0-5","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster2-submariner-cable-cluster2-172-17-0-5","uid":"6a117963-5044-4b34-9df6-57ebb8a4981d","resourceVersion":"1284","generation":1,"creationTimestamp":"2020-10-08T11:35:26Z"},"spec":{"cluster_id":"cluster2","cable_name":"submariner-cable-cluster2-172-17-0-5","hostname":"cluster2-worker","subnets":["100.92.0.0/16","10.242.0.0/16"],"private_ip":"172.17.0.5","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
  5510. 2020-10-08T11:45:19.3631024Z [cluster2] I1008 11:44:11.231822 1 route.go:736] Enqueueing endpoint for route controller {"metadata":{"name":"cluster1-submariner-cable-cluster1-172-17-0-9","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster1-submariner-cable-cluster1-172-17-0-9","uid":"60d04732-7b08-4685-b72b-cb2c12702817","resourceVersion":"1285","generation":1,"creationTimestamp":"2020-10-08T11:35:26Z"},"spec":{"cluster_id":"cluster1","cable_name":"submariner-cable-cluster1-172-17-0-9","hostname":"cluster1-worker","subnets":["100.91.0.0/16","10.241.0.0/16"],"private_ip":"172.17.0.9","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
  5511. 2020-10-08T11:45:19.3636814Z [cluster2] I1008 11:44:11.239652 1 route.go:634] Processing local Endpoint cluster2-submariner-cable-cluster2-172-17-0-5 with IP 172.17.0.5, Host cluster2-worker for this cluster
  5512. 2020-10-08T11:45:19.3638296Z [cluster2] I1008 11:44:11.239816 1 route.go:851] Processing route {Ifindex: 16 Dst: 240.0.0.0/8 Src: 240.17.0.5 Gw: <nil> Flags: [] Table: 254}
  5513. 2020-10-08T11:45:19.3638861Z [cluster2] I1008 11:44:11.239843 1 route.go:854] Found nil gw or dst
  5514. 2020-10-08T11:45:19.3639491Z [cluster2] I1008 11:44:11.239850 1 route.go:645] This route agent is running on the active gateway node
  5515. 2020-10-08T11:45:19.3641602Z [cluster2] I1008 11:44:11.239858 1 route.go:653] Creating the vxlan interface: vx-submariner on the gateway node
  5516. 2020-10-08T11:45:19.3642268Z [cluster2] I1008 11:44:11.239971 1 vxlan.go:61] VxLAN interface already exists with same configuration.
  5517. 2020-10-08T11:45:19.3642871Z [cluster2] I1008 11:44:11.240005 1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.4 00:00:00:00:00:00
  5518. 2020-10-08T11:45:19.3643416Z [cluster2] I1008 11:44:11.240034 1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.5 00:00:00:00:00:00
  5519. 2020-10-08T11:45:19.3644264Z [cluster2] I1008 11:44:11.240081 1 route.go:478] Successfully configured rp_filter to loose mode(2) on vx-submariner
  5520. 2020-10-08T11:45:19.3645652Z [cluster2] I1008 11:44:11.241762 1 route.go:620] Updating iptable rules for remote cluster's Endpoint "cluster1-submariner-cable-cluster1-172-17-0-9" with subnets [100.91.0.0/16 10.241.0.0/16]
  5521. 2020-10-08T11:45:19.3650157Z [cluster2] I1008 11:44:41.231835 1 route.go:736] Enqueueing endpoint for route controller {"metadata":{"name":"cluster2-submariner-cable-cluster2-172-17-0-5","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster2-submariner-cable-cluster2-172-17-0-5","uid":"6a117963-5044-4b34-9df6-57ebb8a4981d","resourceVersion":"1284","generation":1,"creationTimestamp":"2020-10-08T11:35:26Z"},"spec":{"cluster_id":"cluster2","cable_name":"submariner-cable-cluster2-172-17-0-5","hostname":"cluster2-worker","subnets":["100.92.0.0/16","10.242.0.0/16"],"private_ip":"172.17.0.5","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
  5522. 2020-10-08T11:45:19.3657218Z [cluster2] I1008 11:44:41.231959 1 route.go:736] Enqueueing endpoint for route controller {"metadata":{"name":"cluster1-submariner-cable-cluster1-172-17-0-9","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster1-submariner-cable-cluster1-172-17-0-9","uid":"60d04732-7b08-4685-b72b-cb2c12702817","resourceVersion":"1285","generation":1,"creationTimestamp":"2020-10-08T11:35:26Z"},"spec":{"cluster_id":"cluster1","cable_name":"submariner-cable-cluster1-172-17-0-9","hostname":"cluster1-worker","subnets":["100.91.0.0/16","10.241.0.0/16"],"private_ip":"172.17.0.9","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
  5523. 2020-10-08T11:45:19.3661042Z [cluster2] I1008 11:44:41.239305 1 route.go:634] Processing local Endpoint cluster2-submariner-cable-cluster2-172-17-0-5 with IP 172.17.0.5, Host cluster2-worker for this cluster
  5524. 2020-10-08T11:45:19.3662161Z [cluster2] I1008 11:44:41.239481 1 route.go:851] Processing route {Ifindex: 16 Dst: 240.0.0.0/8 Src: 240.17.0.5 Gw: <nil> Flags: [] Table: 254}
  5525. 2020-10-08T11:45:19.3662861Z [lighthouse]$ [cluster2] subctl show all
  5526. 2020-10-08T11:45:19.5497551Z Error obtaining the Submariner resource: the server could not find the requested resource (get submariners.submariner.io submariner)
  5527. 2020-10-08T11:45:19.5498320Z
  5528. 2020-10-08T11:45:19.5499518Z subctl version: v0.7.0-pre1-30-g7a9e04d
  5529. 2020-10-08T11:45:19.5501096Z
  5530. 2020-10-08T11:45:19.5516228Z [cluster2] I1008 11:44:41.239507 1 route.go:854] Found nil gw or dst
  5531. 2020-10-08T11:45:19.5516860Z [cluster2] I1008 11:44:41.239515 1 route.go:645] This route agent is running on the active gateway node
  5532. 2020-10-08T11:45:19.5517992Z [cluster2] I1008 11:44:41.239523 1 route.go:653] Creating the vxlan interface: vx-submariner on the gateway node
  5533. 2020-10-08T11:45:19.5518872Z [cluster2] I1008 11:44:41.239643 1 vxlan.go:61] VxLAN interface already exists with same configuration.
  5534. 2020-10-08T11:45:19.5519852Z [cluster2] I1008 11:44:41.239681 1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.4 00:00:00:00:00:00
  5535. 2020-10-08T11:45:19.5520494Z [cluster2] I1008 11:44:41.239712 1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.5 00:00:00:00:00:00
  5536. 2020-10-08T11:45:19.5521694Z [cluster2] I1008 11:44:41.239767 1 route.go:478] Successfully configured rp_filter to loose mode(2) on vx-submariner
  5537. 2020-10-08T11:45:19.5523439Z [cluster2] I1008 11:44:41.241292 1 route.go:620] Updating iptable rules for remote cluster's Endpoint "cluster1-submariner-cable-cluster1-172-17-0-9" with subnets [100.91.0.0/16 10.241.0.0/16]
  5538. 2020-10-08T11:45:19.5528297Z [cluster2] I1008 11:45:11.231966 1 route.go:736] Enqueueing endpoint for route controller {"metadata":{"name":"cluster1-submariner-cable-cluster1-172-17-0-9","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster1-submariner-cable-cluster1-172-17-0-9","uid":"60d04732-7b08-4685-b72b-cb2c12702817","resourceVersion":"1285","generation":1,"creationTimestamp":"2020-10-08T11:35:26Z"},"spec":{"cluster_id":"cluster1","cable_name":"submariner-cable-cluster1-172-17-0-9","hostname":"cluster1-worker","subnets":["100.91.0.0/16","10.241.0.0/16"],"private_ip":"172.17.0.9","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
  5539. 2020-10-08T11:45:19.5535268Z [cluster2] I1008 11:45:11.232063 1 route.go:736] Enqueueing endpoint for route controller {"metadata":{"name":"cluster2-submariner-cable-cluster2-172-17-0-5","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster2-submariner-cable-cluster2-172-17-0-5","uid":"6a117963-5044-4b34-9df6-57ebb8a4981d","resourceVersion":"1284","generation":1,"creationTimestamp":"2020-10-08T11:35:26Z"},"spec":{"cluster_id":"cluster2","cable_name":"submariner-cable-cluster2-172-17-0-5","hostname":"cluster2-worker","subnets":["100.92.0.0/16","10.242.0.0/16"],"private_ip":"172.17.0.5","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
  5540. 2020-10-08T11:45:19.5539485Z [cluster2] I1008 11:45:11.240140 1 route.go:620] Updating iptable rules for remote cluster's Endpoint "cluster1-submariner-cable-cluster1-172-17-0-9" with subnets [100.91.0.0/16 10.241.0.0/16]
  5541. 2020-10-08T11:45:19.5541608Z [cluster2] I1008 11:45:11.242729 1 route.go:634] Processing local Endpoint cluster2-submariner-cable-cluster2-172-17-0-5 with IP 172.17.0.5, Host cluster2-worker for this cluster
  5542. 2020-10-08T11:45:19.5542893Z [cluster2] I1008 11:45:11.242926 1 route.go:851] Processing route {Ifindex: 16 Dst: 240.0.0.0/8 Src: 240.17.0.5 Gw: <nil> Flags: [] Table: 254}
  5543. 2020-10-08T11:45:19.5543450Z [cluster2] I1008 11:45:11.242945 1 route.go:854] Found nil gw or dst
  5544. 2020-10-08T11:45:19.5543985Z [cluster2] I1008 11:45:11.242954 1 route.go:645] This route agent is running on the active gateway node
  5545. 2020-10-08T11:45:19.5544991Z [cluster2] I1008 11:45:11.242962 1 route.go:653] Creating the vxlan interface: vx-submariner on the gateway node
  5546. 2020-10-08T11:45:19.5545741Z [cluster2] I1008 11:45:11.243070 1 vxlan.go:61] VxLAN interface already exists with same configuration.
  5547. 2020-10-08T11:45:19.5546531Z [cluster2] I1008 11:45:11.243173 1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.5 00:00:00:00:00:00
  5548. 2020-10-08T11:45:19.5547150Z [cluster2] I1008 11:45:11.243206 1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.4 00:00:00:00:00:00
  5549. 2020-10-08T11:45:19.5548096Z [cluster2] I1008 11:45:11.243249 1 route.go:478] Successfully configured rp_filter to loose mode(2) on vx-submariner
  5550. 2020-10-08T11:45:19.5548651Z [cluster2] ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
  5551. 2020-10-08T11:45:19.5548940Z [cluster2]
  5552. 2020-10-08T11:45:19.5549350Z [cluster2] Showing information for cluster "cluster2":
  5553. 2020-10-08T11:45:19.5549847Z [cluster2] Discovered network details:
  5554. 2020-10-08T11:45:19.5550430Z [cluster2] Network plugin: weave-net
  5555. 2020-10-08T11:45:19.5550819Z [cluster2] Service CIDRs: [100.92.0.0/16]
  5556. 2020-10-08T11:45:19.5551252Z [cluster2] Cluster CIDRs: [10.242.0.0/16]
  5557. 2020-10-08T11:45:19.5551506Z [cluster2]
  5558. 2020-10-08T11:45:19.5551901Z [cluster2] ===================== END Post mortem cluster2 =====================
  5559. 2020-10-08T11:45:19.5565367Z [lighthouse]$ make post-mortem
  5560. 2020-10-08T11:45:19.9435807Z Post job cleanup.
  5561. 2020-10-08T11:45:20.3862670Z [command]/usr/bin/git version
  5562. 2020-10-08T11:45:20.3926519Z git version 2.28.0
  5563. 2020-10-08T11:45:20.3997599Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand
  5564. 2020-10-08T11:45:20.4048161Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :
  5565. 2020-10-08T11:45:20.4397848Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader
  5566. 2020-10-08T11:45:20.4432258Z http.https://github.com/.extraheader
  5567. 2020-10-08T11:45:20.4447949Z [command]/usr/bin/git config --local --unset-all http.https://github.com/.extraheader
  5568. 2020-10-08T11:45:20.4485459Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :
  5569. 2020-10-08T11:45:20.4807496Z Cleaning up orphan processes
Add Comment
Please, Sign In to add comment
Public Pastes
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!