subm_random_fail_20201102-3

2020-11-02T14:08:17.8576801Z ##[section]Starting: Request a runner to run this job
2020-11-02T14:08:18.1379948Z Can't find any online and idle self-hosted runner in current repository that matches the required labels: 'ubuntu-latest'
2020-11-02T14:08:18.1380093Z Can't find any online and idle self-hosted runner in current repository's account/organization that matches the required labels: 'ubuntu-latest'
2020-11-02T14:08:18.1381102Z Found online and idle hosted runner in current repository's account/organization that matches the required labels: 'ubuntu-latest'
2020-11-02T14:08:18.2989754Z ##[section]Finishing: Request a runner to run this job
2020-11-02T14:08:27.2919301Z Current runner version: '2.273.5'
2020-11-02T14:08:27.2952516Z ##[group]Operating System
2020-11-02T14:08:27.2953381Z Ubuntu
2020-11-02T14:08:27.2953709Z 18.04.5
2020-11-02T14:08:27.2954015Z LTS
2020-11-02T14:08:27.2954293Z ##[endgroup]
2020-11-02T14:08:27.2954708Z ##[group]Virtual Environment
2020-11-02T14:08:27.2955220Z Environment: ubuntu-18.04
2020-11-02T14:08:27.2955644Z Version: 20201026.1
2020-11-02T14:08:27.2956519Z Included Software: https://github.com/actions/virtual-environments/blob/ubuntu18/20201026.1/images/linux/Ubuntu1804-README.md
2020-11-02T14:08:27.2957351Z ##[endgroup]
2020-11-02T14:08:27.2958739Z Prepare workflow directory
2020-11-02T14:08:27.3165165Z Prepare all required actions
2020-11-02T14:08:27.3176057Z Getting action download info
2020-11-02T14:08:27.5559295Z Download action repository 'actions/checkout@v2'
2020-11-02T14:08:29.6522606Z ##[group]Run actions/checkout@v2
2020-11-02T14:08:29.6523269Z with:
2020-11-02T14:08:29.6523889Z   repository: submariner-io/lighthouse
2020-11-02T14:08:29.6524783Z   token: ***
2020-11-02T14:08:29.6525184Z   ssh-strict: true
2020-11-02T14:08:29.6525624Z   persist-credentials: true
2020-11-02T14:08:29.6525994Z   clean: true
2020-11-02T14:08:29.6526394Z   fetch-depth: 1
2020-11-02T14:08:29.6526742Z   lfs: false
2020-11-02T14:08:29.6527093Z   submodules: false
2020-11-02T14:08:29.6527462Z ##[endgroup]
2020-11-02T14:08:30.5743153Z Syncing repository: submariner-io/lighthouse
2020-11-02T14:08:30.5744571Z ##[group]Getting Git version info
2020-11-02T14:08:30.5745953Z Working directory is '/home/runner/work/lighthouse/lighthouse'
2020-11-02T14:08:30.5746852Z [command]/usr/bin/git version
2020-11-02T14:08:30.5751084Z git version 2.29.0
2020-11-02T14:08:30.5752461Z ##[endgroup]
2020-11-02T14:08:30.5753290Z Deleting the contents of '/home/runner/work/lighthouse/lighthouse'
2020-11-02T14:08:30.5754659Z ##[group]Initializing the repository
2020-11-02T14:08:30.5755273Z [command]/usr/bin/git init /home/runner/work/lighthouse/lighthouse
2020-11-02T14:08:30.5756015Z Initialized empty Git repository in /home/runner/work/lighthouse/lighthouse/.git/
2020-11-02T14:08:30.5757152Z [command]/usr/bin/git remote add origin https://github.com/submariner-io/lighthouse
2020-11-02T14:08:30.5758235Z ##[endgroup]
2020-11-02T14:08:30.5758765Z ##[group]Disabling automatic garbage collection
2020-11-02T14:08:30.5759622Z [command]/usr/bin/git config --local gc.auto 0
2020-11-02T14:08:30.5760080Z ##[endgroup]
2020-11-02T14:08:30.5761799Z ##[group]Setting up auth
2020-11-02T14:08:30.5762699Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand
2020-11-02T14:08:30.5764154Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :
2020-11-02T14:08:30.5765753Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader
2020-11-02T14:08:30.5767673Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :
2020-11-02T14:08:30.5769217Z [command]/usr/bin/git config --local http.https://github.com/.extraheader AUTHORIZATION: basic ***
2020-11-02T14:08:30.5769993Z ##[endgroup]
2020-11-02T14:08:30.5770428Z ##[group]Fetching the repository
2020-11-02T14:08:30.5771775Z [command]/usr/bin/git -c protocol.version=2 fetch --no-tags --prune --progress --no-recurse-submodules --depth=1 origin +25fe91a012b5b20d25c52f76752ac29f22abd74d:refs/remotes/pull/360/merge
2020-11-02T14:08:30.5772832Z remote: Enumerating objects: 161, done.
2020-11-02T14:08:30.5774051Z remote: Counting objects:   0% (1/161)
2020-11-02T14:08:30.5774455Z remote: Counting objects:   1% (2/161)
2020-11-02T14:08:30.5774984Z remote: Counting objects:   2% (4/161)
2020-11-02T14:08:30.5775447Z remote: Counting objects:   3% (5/161)
2020-11-02T14:08:30.5775893Z remote: Counting objects:   4% (7/161)
2020-11-02T14:08:30.5776595Z remote: Counting objects:   5% (9/161)
2020-11-02T14:08:30.5777050Z remote: Counting objects:   6% (10/161)
2020-11-02T14:08:30.5777517Z remote: Counting objects:   7% (12/161)
2020-11-02T14:08:30.5777982Z remote: Counting objects:   8% (13/161)
2020-11-02T14:08:30.5778427Z remote: Counting objects:   9% (15/161)
2020-11-02T14:08:30.5778874Z remote: Counting objects:  10% (17/161)
2020-11-02T14:08:30.5779270Z remote: Counting objects:  11% (18/161)
2020-11-02T14:08:30.5779719Z remote: Counting objects:  12% (20/161)
2020-11-02T14:08:30.5780167Z remote: Counting objects:  13% (21/161)
2020-11-02T14:08:30.5780636Z remote: Counting objects:  14% (23/161)
2020-11-02T14:08:30.5781436Z remote: Counting objects:  15% (25/161)
2020-11-02T14:08:30.5781925Z remote: Counting objects:  16% (26/161)
2020-11-02T14:08:30.5782548Z remote: Counting objects:  17% (28/161)
2020-11-02T14:08:30.5783002Z remote: Counting objects:  18% (29/161)
2020-11-02T14:08:30.5783395Z remote: Counting objects:  19% (31/161)
2020-11-02T14:08:30.5783842Z remote: Counting objects:  20% (33/161)
2020-11-02T14:08:30.5784482Z remote: Counting objects:  21% (34/161)
2020-11-02T14:08:30.5784945Z remote: Counting objects:  22% (36/161)
2020-11-02T14:08:30.5785460Z remote: Counting objects:  23% (38/161)
2020-11-02T14:08:30.5785927Z remote: Counting objects:  24% (39/161)
2020-11-02T14:08:30.5786390Z remote: Counting objects:  25% (41/161)
2020-11-02T14:08:30.5786851Z remote: Counting objects:  26% (42/161)
2020-11-02T14:08:30.5787257Z remote: Counting objects:  27% (44/161)
2020-11-02T14:08:30.5787734Z remote: Counting objects:  28% (46/161)
2020-11-02T14:08:30.5788200Z remote: Counting objects:  29% (47/161)
2020-11-02T14:08:30.5788664Z remote: Counting objects:  30% (49/161)
2020-11-02T14:08:30.5789133Z remote: Counting objects:  31% (50/161)
2020-11-02T14:08:30.5789631Z remote: Counting objects:  32% (52/161)
2020-11-02T14:08:30.5790103Z remote: Counting objects:  33% (54/161)
2020-11-02T14:08:30.5790567Z remote: Counting objects:  34% (55/161)
2020-11-02T14:08:30.5790973Z remote: Counting objects:  35% (57/161)
2020-11-02T14:08:30.5791624Z remote: Counting objects:  36% (58/161)
2020-11-02T14:08:30.5792099Z remote: Counting objects:  37% (60/161)
2020-11-02T14:08:30.5792562Z remote: Counting objects:  38% (62/161)
2020-11-02T14:08:30.5793026Z remote: Counting objects:  39% (63/161)
2020-11-02T14:08:30.5793507Z remote: Counting objects:  40% (65/161)
2020-11-02T14:08:30.5794002Z remote: Counting objects:  41% (67/161)
2020-11-02T14:08:30.5794483Z remote: Counting objects:  42% (68/161)
2020-11-02T14:08:30.5795234Z remote: Counting objects:  43% (70/161)
2020-11-02T14:08:30.5795685Z remote: Counting objects:  44% (71/161)
2020-11-02T14:08:30.5796170Z remote: Counting objects:  45% (73/161)
2020-11-02T14:08:30.5796690Z remote: Counting objects:  46% (75/161)
2020-11-02T14:08:30.5797161Z remote: Counting objects:  47% (76/161)
2020-11-02T14:08:30.5797722Z remote: Counting objects:  48% (78/161)
2020-11-02T14:08:30.5798198Z remote: Counting objects:  49% (79/161)
2020-11-02T14:08:30.5798698Z remote: Counting objects:  50% (81/161)
2020-11-02T14:08:30.5799164Z remote: Counting objects:  51% (83/161)
2020-11-02T14:08:30.5799570Z remote: Counting objects:  52% (84/161)
2020-11-02T14:08:30.5800034Z remote: Counting objects:  53% (86/161)
2020-11-02T14:08:30.5800501Z remote: Counting objects:  54% (87/161)
2020-11-02T14:08:30.5800974Z remote: Counting objects:  55% (89/161)
2020-11-02T14:08:30.5804408Z remote: Counting objects:  56% (91/161)
2020-11-02T14:08:30.5804932Z remote: Counting objects:  57% (92/161)
2020-11-02T14:08:30.5805581Z remote: Counting objects:  58% (94/161)
2020-11-02T14:08:30.5806089Z remote: Counting objects:  59% (95/161)
2020-11-02T14:08:30.5806501Z remote: Counting objects:  60% (97/161)
2020-11-02T14:08:30.5806967Z remote: Counting objects:  61% (99/161)
2020-11-02T14:08:30.5807435Z remote: Counting objects:  62% (100/161)
2020-11-02T14:08:30.5807899Z remote: Counting objects:  63% (102/161)
2020-11-02T14:08:30.5808364Z remote: Counting objects:  64% (104/161)
2020-11-02T14:08:30.5808839Z remote: Counting objects:  65% (105/161)
2020-11-02T14:08:30.5809307Z remote: Counting objects:  66% (107/161)
2020-11-02T14:08:30.5809846Z remote: Counting objects:  67% (108/161)
2020-11-02T14:08:30.5810361Z remote: Counting objects:  68% (110/161)
2020-11-02T14:08:30.5810890Z remote: Counting objects:  69% (112/161)
2020-11-02T14:08:30.5811360Z remote: Counting objects:  70% (113/161)
2020-11-02T14:08:30.5811838Z remote: Counting objects:  71% (115/161)
2020-11-02T14:08:30.5812318Z remote: Counting objects:  72% (116/161)
2020-11-02T14:08:30.5812809Z remote: Counting objects:  73% (118/161)
2020-11-02T14:08:30.5813275Z remote: Counting objects:  74% (120/161)
2020-11-02T14:08:30.5813738Z remote: Counting objects:  75% (121/161)
2020-11-02T14:08:30.5814401Z remote: Counting objects:  76% (123/161)
2020-11-02T14:08:30.5814875Z remote: Counting objects:  77% (124/161)
2020-11-02T14:08:30.5815325Z remote: Counting objects:  78% (126/161)
2020-11-02T14:08:30.5815720Z remote: Counting objects:  79% (128/161)
2020-11-02T14:08:30.5816170Z remote: Counting objects:  80% (129/161)
2020-11-02T14:08:30.5816644Z remote: Counting objects:  81% (131/161)
2020-11-02T14:08:30.5817095Z remote: Counting objects:  82% (133/161)
2020-11-02T14:08:30.5817540Z remote: Counting objects:  83% (134/161)
2020-11-02T14:08:30.5817993Z remote: Counting objects:  84% (136/161)
2020-11-02T14:08:30.5818456Z remote: Counting objects:  85% (137/161)
2020-11-02T14:08:30.5818928Z remote: Counting objects:  86% (139/161)
2020-11-02T14:08:30.5819327Z remote: Counting objects:  87% (141/161)
2020-11-02T14:08:30.5819959Z remote: Counting objects:  88% (142/161)
2020-11-02T14:08:30.5820426Z remote: Counting objects:  89% (144/161)
2020-11-02T14:08:30.5821065Z remote: Counting objects:  90% (145/161)
2020-11-02T14:08:30.5821720Z remote: Counting objects:  91% (147/161)
2020-11-02T14:08:30.5822183Z remote: Counting objects:  92% (149/161)
2020-11-02T14:08:30.5822648Z remote: Counting objects:  93% (150/161)
2020-11-02T14:08:30.5823122Z remote: Counting objects:  94% (152/161)
2020-11-02T14:08:30.5823534Z remote: Counting objects:  95% (153/161)
2020-11-02T14:08:30.5824026Z remote: Counting objects:  96% (155/161)
2020-11-02T14:08:30.5824506Z remote: Counting objects:  97% (157/161)
2020-11-02T14:08:30.5824992Z remote: Counting objects:  98% (158/161)
2020-11-02T14:08:30.5826192Z remote: Counting objects:  99% (160/161)
2020-11-02T14:08:30.5826780Z remote: Counting objects: 100% (161/161)
2020-11-02T14:08:30.5827268Z remote: Counting objects: 100% (161/161), done.
2020-11-02T14:08:30.5827774Z remote: Compressing objects:   0% (1/138)
2020-11-02T14:08:30.5828282Z remote: Compressing objects:   1% (2/138)
2020-11-02T14:08:30.5828733Z remote: Compressing objects:   2% (3/138)
2020-11-02T14:08:30.5829280Z remote: Compressing objects:   3% (5/138)
2020-11-02T14:08:30.5829786Z remote: Compressing objects:   4% (6/138)
2020-11-02T14:08:30.5830319Z remote: Compressing objects:   5% (7/138)
2020-11-02T14:08:30.5830819Z remote: Compressing objects:   6% (9/138)
2020-11-02T14:08:30.5831576Z remote: Compressing objects:   7% (10/138)
2020-11-02T14:08:30.5832269Z remote: Compressing objects:   8% (12/138)
2020-11-02T14:08:30.5832776Z remote: Compressing objects:   9% (13/138)
2020-11-02T14:08:30.5833227Z remote: Compressing objects:  10% (14/138)
2020-11-02T14:08:30.5833732Z remote: Compressing objects:  11% (16/138)
2020-11-02T14:08:30.5834274Z remote: Compressing objects:  12% (17/138)
2020-11-02T14:08:30.5834803Z remote: Compressing objects:  13% (18/138)
2020-11-02T14:08:30.5835308Z remote: Compressing objects:  14% (20/138)
2020-11-02T14:08:30.5835979Z remote: Compressing objects:  15% (21/138)
2020-11-02T14:08:30.5837250Z remote: Compressing objects:  16% (23/138)
2020-11-02T14:08:30.5837775Z remote: Compressing objects:  17% (24/138)
2020-11-02T14:08:30.5838368Z remote: Compressing objects:  18% (25/138)
2020-11-02T14:08:30.5838901Z remote: Compressing objects:  19% (27/138)
2020-11-02T14:08:30.5839404Z remote: Compressing objects:  20% (28/138)
2020-11-02T14:08:30.5839990Z remote: Compressing objects:  21% (29/138)
2020-11-02T14:08:30.5840492Z remote: Compressing objects:  22% (31/138)
2020-11-02T14:08:30.5841015Z remote: Compressing objects:  23% (32/138)
2020-11-02T14:08:30.5841515Z remote: Compressing objects:  24% (34/138)
2020-11-02T14:08:30.5842015Z remote: Compressing objects:  25% (35/138)
2020-11-02T14:08:30.5842511Z remote: Compressing objects:  26% (36/138)
2020-11-02T14:08:30.5842956Z remote: Compressing objects:  27% (38/138)
2020-11-02T14:08:30.5843456Z remote: Compressing objects:  28% (39/138)
2020-11-02T14:08:30.5844124Z remote: Compressing objects:  29% (41/138)
2020-11-02T14:08:30.5844634Z remote: Compressing objects:  30% (42/138)
2020-11-02T14:08:30.5845128Z remote: Compressing objects:  31% (43/138)
2020-11-02T14:08:30.5845605Z remote: Compressing objects:  32% (45/138)
2020-11-02T14:08:30.5846089Z remote: Compressing objects:  33% (46/138)
2020-11-02T14:08:30.5847001Z remote: Compressing objects:  34% (47/138)
2020-11-02T14:08:30.5847631Z remote: Compressing objects:  35% (49/138)
2020-11-02T14:08:30.5848302Z remote: Compressing objects:  36% (50/138)
2020-11-02T14:08:30.5848814Z remote: Compressing objects:  37% (52/138)
2020-11-02T14:08:30.5849330Z remote: Compressing objects:  38% (53/138)
2020-11-02T14:08:30.5849868Z remote: Compressing objects:  39% (54/138)
2020-11-02T14:08:30.5850367Z remote: Compressing objects:  40% (56/138)
2020-11-02T14:08:30.5850865Z remote: Compressing objects:  41% (57/138)
2020-11-02T14:08:30.5851513Z remote: Compressing objects:  42% (58/138)
2020-11-02T14:08:30.5851994Z remote: Compressing objects:  43% (60/138)
2020-11-02T14:08:30.5852436Z remote: Compressing objects:  44% (61/138)
2020-11-02T14:08:30.5852921Z remote: Compressing objects:  45% (63/138)
2020-11-02T14:08:30.5853420Z remote: Compressing objects:  46% (64/138)
2020-11-02T14:08:30.5856879Z remote: Compressing objects:  47% (65/138)
2020-11-02T14:08:30.5857412Z remote: Compressing objects:  48% (67/138)
2020-11-02T14:08:30.5857879Z remote: Compressing objects:  49% (68/138)
2020-11-02T14:08:30.5858344Z remote: Compressing objects:  50% (69/138)
2020-11-02T14:08:30.5858806Z remote: Compressing objects:  51% (71/138)
2020-11-02T14:08:30.5859218Z remote: Compressing objects:  52% (72/138)
2020-11-02T14:08:30.5859968Z remote: Compressing objects:  53% (74/138)
2020-11-02T14:08:30.5860469Z remote: Compressing objects:  54% (75/138)
2020-11-02T14:08:30.5860988Z remote: Compressing objects:  55% (76/138)
2020-11-02T14:08:30.5861485Z remote: Compressing objects:  56% (78/138)
2020-11-02T14:08:30.5862021Z remote: Compressing objects:  57% (79/138)
2020-11-02T14:08:30.5862524Z remote: Compressing objects:  58% (81/138)
2020-11-02T14:08:30.5863347Z remote: Compressing objects:  59% (82/138)
2020-11-02T14:08:30.5863828Z remote: Compressing objects:  60% (83/138)
2020-11-02T14:08:30.5864260Z remote: Compressing objects:  61% (85/138)
2020-11-02T14:08:30.5864780Z remote: Compressing objects:  62% (86/138)
2020-11-02T14:08:30.5865277Z remote: Compressing objects:  63% (87/138)
2020-11-02T14:08:30.5865937Z remote: Compressing objects:  64% (89/138)
2020-11-02T14:08:30.5866438Z remote: Compressing objects:  65% (90/138)
2020-11-02T14:08:30.5866965Z remote: Compressing objects:  66% (92/138)
2020-11-02T14:08:30.5867470Z remote: Compressing objects:  67% (93/138)
2020-11-02T14:08:30.5867969Z remote: Compressing objects:  68% (94/138)
2020-11-02T14:08:30.5868650Z remote: Compressing objects:  69% (96/138)
2020-11-02T14:08:30.5869155Z remote: Compressing objects:  70% (97/138)
2020-11-02T14:08:30.5869660Z remote: Compressing objects:  71% (98/138)
2020-11-02T14:08:30.5870159Z remote: Compressing objects:  72% (100/138)
2020-11-02T14:08:30.5870997Z remote: Compressing objects:  73% (101/138)
2020-11-02T14:08:30.5871850Z remote: Compressing objects:  74% (103/138)
2020-11-02T14:08:30.5872392Z remote: Compressing objects:  75% (104/138)
2020-11-02T14:08:30.5872899Z remote: Compressing objects:  76% (105/138)
2020-11-02T14:08:30.5873403Z remote: Compressing objects:  77% (107/138)
2020-11-02T14:08:30.5873856Z remote: Compressing objects:  78% (108/138)
2020-11-02T14:08:30.5874373Z remote: Compressing objects:  79% (110/138)
2020-11-02T14:08:30.5875032Z remote: Compressing objects:  80% (111/138)
2020-11-02T14:08:30.5875517Z remote: Compressing objects:  81% (112/138)
2020-11-02T14:08:30.5876011Z remote: Compressing objects:  82% (114/138)
2020-11-02T14:08:30.5876498Z remote: Compressing objects:  83% (115/138)
2020-11-02T14:08:30.5877002Z remote: Compressing objects:  84% (116/138)
2020-11-02T14:08:30.5877499Z remote: Compressing objects:  85% (118/138)
2020-11-02T14:08:30.5877931Z remote: Compressing objects:  86% (119/138)
2020-11-02T14:08:30.5878418Z remote: Compressing objects:  87% (121/138)
2020-11-02T14:08:30.5878918Z remote: Compressing objects:  88% (122/138)
2020-11-02T14:08:30.5879401Z remote: Compressing objects:  89% (123/138)
2020-11-02T14:08:30.5879886Z remote: Compressing objects:  90% (125/138)
2020-11-02T14:08:30.5880370Z remote: Compressing objects:  91% (126/138)
2020-11-02T14:08:30.5880852Z remote: Compressing objects:  92% (127/138)
2020-11-02T14:08:30.5881379Z remote: Compressing objects:  93% (129/138)
2020-11-02T14:08:30.5881814Z remote: Compressing objects:  94% (130/138)
2020-11-02T14:08:30.5882311Z remote: Compressing objects:  95% (132/138)
2020-11-02T14:08:30.5882810Z remote: Compressing objects:  96% (133/138)
2020-11-02T14:08:30.5883296Z remote: Compressing objects:  97% (134/138)
2020-11-02T14:08:30.5883861Z remote: Compressing objects:  98% (136/138)
2020-11-02T14:08:30.5884296Z remote: Compressing objects:  99% (137/138)
2020-11-02T14:08:30.5884783Z remote: Compressing objects: 100% (138/138)
2020-11-02T14:08:30.5885286Z remote: Compressing objects: 100% (138/138), done.
2020-11-02T14:08:30.5885778Z Receiving objects:   0% (1/161)
2020-11-02T14:08:30.5886193Z Receiving objects:   1% (2/161)
2020-11-02T14:08:30.5886617Z Receiving objects:   2% (4/161)
2020-11-02T14:08:30.5887027Z Receiving objects:   3% (5/161)
2020-11-02T14:08:30.5887435Z Receiving objects:   4% (7/161)
2020-11-02T14:08:30.5887790Z Receiving objects:   5% (9/161)
2020-11-02T14:08:30.5888206Z Receiving objects:   6% (10/161)
2020-11-02T14:08:30.5888632Z Receiving objects:   7% (12/161)
2020-11-02T14:08:30.5889047Z Receiving objects:   8% (13/161)
2020-11-02T14:08:30.5889480Z Receiving objects:   9% (15/161)
2020-11-02T14:08:30.5889908Z Receiving objects:  10% (17/161)
2020-11-02T14:08:30.5890427Z Receiving objects:  11% (18/161)
2020-11-02T14:08:30.5890786Z Receiving objects:  12% (20/161)
2020-11-02T14:08:30.5891200Z Receiving objects:  13% (21/161)
2020-11-02T14:08:30.5891612Z Receiving objects:  14% (23/161)
2020-11-02T14:08:30.5892025Z Receiving objects:  15% (25/161)
2020-11-02T14:08:30.5892438Z Receiving objects:  16% (26/161)
2020-11-02T14:08:30.6094610Z Receiving objects:  17% (28/161)
2020-11-02T14:08:30.6095184Z Receiving objects:  18% (29/161)
2020-11-02T14:08:30.6096243Z Receiving objects:  19% (31/161)
2020-11-02T14:08:30.6096835Z Receiving objects:  20% (33/161)
2020-11-02T14:08:30.6097184Z Receiving objects:  21% (34/161)
2020-11-02T14:08:30.6097719Z Receiving objects:  22% (36/161)
2020-11-02T14:08:30.6098079Z Receiving objects:  23% (38/161)
2020-11-02T14:08:30.6098630Z Receiving objects:  24% (39/161)
2020-11-02T14:08:30.6099024Z Receiving objects:  25% (41/161)
2020-11-02T14:08:30.6099379Z Receiving objects:  26% (42/161)
2020-11-02T14:08:30.6100085Z Receiving objects:  27% (44/161)
2020-11-02T14:08:30.6100446Z Receiving objects:  28% (46/161)
2020-11-02T14:08:30.6100984Z Receiving objects:  29% (47/161)
2020-11-02T14:08:30.6101332Z Receiving objects:  30% (49/161)
2020-11-02T14:08:30.6101676Z Receiving objects:  31% (50/161)
2020-11-02T14:08:30.6102025Z Receiving objects:  32% (52/161)
2020-11-02T14:08:30.6102363Z Receiving objects:  33% (54/161)
2020-11-02T14:08:30.6102715Z Receiving objects:  34% (55/161)
2020-11-02T14:08:30.6103063Z Receiving objects:  35% (57/161)
2020-11-02T14:08:30.6103411Z Receiving objects:  36% (58/161)
2020-11-02T14:08:30.6103762Z Receiving objects:  37% (60/161)
2020-11-02T14:08:30.6104110Z Receiving objects:  38% (62/161)
2020-11-02T14:08:30.6104456Z Receiving objects:  39% (63/161)
2020-11-02T14:08:30.6105008Z Receiving objects:  40% (65/161)
2020-11-02T14:08:30.6105364Z Receiving objects:  41% (67/161)
2020-11-02T14:08:30.6105981Z Receiving objects:  42% (68/161)
2020-11-02T14:08:30.6106364Z Receiving objects:  43% (70/161)
2020-11-02T14:08:30.6106750Z Receiving objects:  44% (71/161)
2020-11-02T14:08:30.6107121Z Receiving objects:  45% (73/161)
2020-11-02T14:08:30.6107497Z Receiving objects:  46% (75/161)
2020-11-02T14:08:30.6107872Z Receiving objects:  47% (76/161)
2020-11-02T14:08:30.6108237Z Receiving objects:  48% (78/161)
2020-11-02T14:08:30.6108610Z Receiving objects:  49% (79/161)
2020-11-02T14:08:30.6108982Z Receiving objects:  50% (81/161)
2020-11-02T14:08:30.6109522Z Receiving objects:  51% (83/161)
2020-11-02T14:08:30.6109883Z Receiving objects:  52% (84/161)
2020-11-02T14:08:30.6110243Z Receiving objects:  53% (86/161)
2020-11-02T14:08:30.6110602Z Receiving objects:  54% (87/161)
2020-11-02T14:08:30.6110959Z Receiving objects:  55% (89/161)
2020-11-02T14:08:30.6111459Z Receiving objects:  56% (91/161)
2020-11-02T14:08:30.6111929Z Receiving objects:  57% (92/161)
2020-11-02T14:08:30.6112290Z Receiving objects:  58% (94/161)
2020-11-02T14:08:30.6112652Z Receiving objects:  59% (95/161)
2020-11-02T14:08:30.6113379Z Receiving objects:  60% (97/161)
2020-11-02T14:08:30.6113750Z Receiving objects:  61% (99/161)
2020-11-02T14:08:30.6114392Z Receiving objects:  62% (100/161)
2020-11-02T14:08:30.6114793Z Receiving objects:  63% (102/161)
2020-11-02T14:08:30.6115174Z Receiving objects:  64% (104/161)
2020-11-02T14:08:30.6115550Z Receiving objects:  65% (105/161)
2020-11-02T14:08:30.6115926Z Receiving objects:  66% (107/161)
2020-11-02T14:08:30.6116300Z Receiving objects:  67% (108/161)
2020-11-02T14:08:30.6116674Z Receiving objects:  68% (110/161)
2020-11-02T14:08:30.6117118Z Receiving objects:  69% (112/161)
2020-11-02T14:08:30.6117492Z Receiving objects:  70% (113/161)
2020-11-02T14:08:30.6117859Z Receiving objects:  71% (115/161)
2020-11-02T14:08:30.6118235Z Receiving objects:  72% (116/161)
2020-11-02T14:08:30.6118619Z Receiving objects:  73% (118/161)
2020-11-02T14:08:30.6119157Z Receiving objects:  74% (120/161)
2020-11-02T14:08:30.6119712Z Receiving objects:  75% (121/161)
2020-11-02T14:08:30.6120419Z Receiving objects:  76% (123/161)
2020-11-02T14:08:30.6120781Z Receiving objects:  77% (124/161)
2020-11-02T14:08:30.6121133Z Receiving objects:  78% (126/161)
2020-11-02T14:08:30.6121495Z Receiving objects:  79% (128/161)
2020-11-02T14:08:30.6121856Z Receiving objects:  80% (129/161)
2020-11-02T14:08:30.6122221Z Receiving objects:  81% (131/161)
2020-11-02T14:08:30.6122774Z Receiving objects:  82% (133/161)
2020-11-02T14:08:30.6123150Z Receiving objects:  83% (134/161)
2020-11-02T14:08:30.6123524Z Receiving objects:  84% (136/161)
2020-11-02T14:08:30.6123896Z Receiving objects:  85% (137/161)
2020-11-02T14:08:30.6124261Z Receiving objects:  86% (139/161)
2020-11-02T14:08:30.6124634Z Receiving objects:  87% (141/161)
2020-11-02T14:08:30.6125008Z Receiving objects:  88% (142/161)
2020-11-02T14:08:30.6125464Z Receiving objects:  89% (144/161)
2020-11-02T14:08:30.6125855Z Receiving objects:  90% (145/161)
2020-11-02T14:08:30.6126231Z Receiving objects:  91% (147/161)
2020-11-02T14:08:30.6126602Z Receiving objects:  92% (149/161)
2020-11-02T14:08:30.6126983Z Receiving objects:  93% (150/161)
2020-11-02T14:08:30.6127350Z Receiving objects:  94% (152/161)
2020-11-02T14:08:30.6127724Z Receiving objects:  95% (153/161)
2020-11-02T14:08:30.6128100Z Receiving objects:  96% (155/161)
2020-11-02T14:08:30.6197622Z Receiving objects:  97% (157/161)
2020-11-02T14:08:30.6198901Z remote: Total 161 (delta 29), reused 79 (delta 12), pack-reused 0
2020-11-02T14:08:30.6199599Z Receiving objects:  98% (158/161)
2020-11-02T14:08:30.6200134Z Receiving objects:  99% (160/161)
2020-11-02T14:08:30.6200650Z Receiving objects: 100% (161/161)
2020-11-02T14:08:30.6201217Z Receiving objects: 100% (161/161), 124.30 KiB | 6.54 MiB/s, done.
2020-11-02T14:08:30.6201767Z Resolving deltas:   0% (0/29)
2020-11-02T14:08:30.6202293Z Resolving deltas:   3% (1/29)
2020-11-02T14:08:30.6202801Z Resolving deltas:   6% (2/29)
2020-11-02T14:08:30.6203300Z Resolving deltas:  10% (3/29)
2020-11-02T14:08:30.6203798Z Resolving deltas:  13% (4/29)
2020-11-02T14:08:30.6204310Z Resolving deltas:  17% (5/29)
2020-11-02T14:08:30.6204809Z Resolving deltas:  20% (6/29)
2020-11-02T14:08:30.6205305Z Resolving deltas:  24% (7/29)
2020-11-02T14:08:30.6205802Z Resolving deltas:  27% (8/29)
2020-11-02T14:08:30.6206279Z Resolving deltas:  31% (9/29)
2020-11-02T14:08:30.6206778Z Resolving deltas:  34% (10/29)
2020-11-02T14:08:30.6207282Z Resolving deltas:  37% (11/29)
2020-11-02T14:08:30.6207782Z Resolving deltas:  41% (12/29)
2020-11-02T14:08:30.6208280Z Resolving deltas:  44% (13/29)
2020-11-02T14:08:30.6208785Z Resolving deltas:  48% (14/29)
2020-11-02T14:08:30.6209282Z Resolving deltas:  51% (15/29)
2020-11-02T14:08:30.6209779Z Resolving deltas:  55% (16/29)
2020-11-02T14:08:30.6210277Z Resolving deltas:  58% (17/29)
2020-11-02T14:08:30.6210774Z Resolving deltas:  62% (18/29)
2020-11-02T14:08:30.6211281Z Resolving deltas:  65% (19/29)
2020-11-02T14:08:30.6211785Z Resolving deltas:  68% (20/29)
2020-11-02T14:08:30.6212281Z Resolving deltas:  72% (21/29)
2020-11-02T14:08:30.6212774Z Resolving deltas:  75% (22/29)
2020-11-02T14:08:30.6213276Z Resolving deltas:  79% (23/29)
2020-11-02T14:08:30.6213776Z Resolving deltas:  82% (24/29)
2020-11-02T14:08:30.6214274Z Resolving deltas:  86% (25/29)
2020-11-02T14:08:30.6214792Z Resolving deltas:  89% (26/29)
2020-11-02T14:08:30.6215296Z Resolving deltas:  93% (27/29)
2020-11-02T14:08:30.6215797Z Resolving deltas:  96% (28/29)
2020-11-02T14:08:30.6216537Z Resolving deltas: 100% (29/29)
2020-11-02T14:08:30.6217293Z Resolving deltas: 100% (29/29), done.
2020-11-02T14:08:30.6457642Z From https://github.com/submariner-io/lighthouse
2020-11-02T14:08:30.6458632Z  * [new ref]         25fe91a012b5b20d25c52f76752ac29f22abd74d -> pull/360/merge
2020-11-02T14:08:30.6484460Z ##[endgroup]
2020-11-02T14:08:30.6484921Z ##[group]Determining the checkout info
2020-11-02T14:08:30.6487646Z ##[endgroup]
2020-11-02T14:08:30.6488212Z ##[group]Checking out the ref
2020-11-02T14:08:30.6493229Z [command]/usr/bin/git checkout --progress --force refs/remotes/pull/360/merge
2020-11-02T14:08:30.6630957Z Note: switching to 'refs/remotes/pull/360/merge'.
2020-11-02T14:08:30.6631800Z
2020-11-02T14:08:30.6632609Z You are in 'detached HEAD' state. You can look around, make experimental
2020-11-02T14:08:30.6633739Z changes and commit them, and you can discard any commits you make in this
2020-11-02T14:08:30.6634395Z state without impacting any branches by switching back to a branch.
2020-11-02T14:08:30.6646258Z
2020-11-02T14:08:30.6646933Z If you want to create a new branch to retain commits you create, you may
2020-11-02T14:08:30.6648067Z do so (now or later) by using -c with the switch command. Example:
2020-11-02T14:08:30.6648425Z
2020-11-02T14:08:30.6648943Z   git switch -c <new-branch-name>
2020-11-02T14:08:30.6649218Z
2020-11-02T14:08:30.6649541Z Or undo this operation with:
2020-11-02T14:08:30.6650200Z
2020-11-02T14:08:30.6650715Z   git switch -
2020-11-02T14:08:30.6650928Z
2020-11-02T14:08:30.6651446Z Turn off this advice by setting config variable advice.detachedHead to false
2020-11-02T14:08:30.6651900Z
2020-11-02T14:08:30.6652825Z HEAD is now at 25fe91a Merge 47d41a93de16d1a9d0659d20eeb3748b2b36c1b4 into 8c67c8fcbbf25fd20b0df8594e9929c7ee137630
2020-11-02T14:08:30.6653696Z ##[endgroup]
2020-11-02T14:08:30.6691554Z [command]/usr/bin/git log -1 --format='%H'
2020-11-02T14:08:30.6729418Z '25fe91a012b5b20d25c52f76752ac29f22abd74d'
2020-11-02T14:08:30.6936294Z ##[group]Run sudo swapoff -a
2020-11-02T14:08:30.6936749Z sudo swapoff -a
2020-11-02T14:08:30.6937075Z sudo rm -f /swapfile
2020-11-02T14:08:30.6937364Z df -h
2020-11-02T14:08:30.6937617Z free -h
2020-11-02T14:08:30.6985877Z shell: /bin/bash -e {0}
2020-11-02T14:08:30.6986382Z ##[endgroup]
2020-11-02T14:08:30.8414016Z Filesystem      Size  Used Avail Use% Mounted on
2020-11-02T14:08:30.8414926Z udev            3.4G     0  3.4G   0% /dev
2020-11-02T14:08:30.8415445Z tmpfs           696M  680K  695M   1% /run
2020-11-02T14:08:30.8415940Z /dev/sda1        84G   63G   21G  76% /
2020-11-02T14:08:30.8416711Z tmpfs           3.4G  8.0K  3.4G   1% /dev/shm
2020-11-02T14:08:30.8417479Z tmpfs           5.0M     0  5.0M   0% /run/lock
2020-11-02T14:08:30.8417868Z tmpfs           3.4G     0  3.4G   0% /sys/fs/cgroup
2020-11-02T14:08:30.8418252Z /dev/sda15      105M  3.6M  101M   4% /boot/efi
2020-11-02T14:08:30.8592966Z /dev/sdb1        14G  4.1G  9.0G  32% /mnt
2020-11-02T14:08:30.8593648Z               total        used        free      shared  buff/cache   available
2020-11-02T14:08:30.8594447Z Mem:           6.8G        450M        4.9G         25M        1.4G        6.0G
2020-11-02T14:08:30.8594967Z Swap:            0B          0B          0B
2020-11-02T14:08:30.8644829Z ##[group]Run make e2e using=" helm"
2020-11-02T14:08:30.8645268Z make e2e using=" helm"
2020-11-02T14:08:30.8688230Z shell: /bin/bash -e {0}
2020-11-02T14:08:30.8688548Z ##[endgroup]
2020-11-02T14:08:30.8797347Z Downloading dapper
2020-11-02T14:08:31.6544976Z .dapper.tmp version v0.5.3
2020-11-02T14:08:31.6955468Z ./.dapper -m bind make e2e  -- using=helm
2020-11-02T14:08:33.0623987Z Sending build context to Docker daemon  6.209MB
2020-11-02T14:08:33.0660823Z
2020-11-02T14:08:33.0662808Z Step 1/6 : FROM quay.io/submariner/shipyard-dapper-base:devel
2020-11-02T14:08:33.0663781Z devel: Pulling from submariner/shipyard-dapper-base
2020-11-02T14:08:33.0665125Z ee7e89337106: Pulling fs layer
2020-11-02T14:08:33.0665492Z 15b785f9b2f0: Pulling fs layer
2020-11-02T14:08:33.0666640Z f22377f2f0d9: Pulling fs layer
2020-11-02T14:08:33.0666994Z 14821382b808: Pulling fs layer
2020-11-02T14:08:33.0667312Z 72c0738702f0: Pulling fs layer
2020-11-02T14:08:33.0667851Z 213eace38d21: Pulling fs layer
2020-11-02T14:08:33.0668240Z 555aaae525c9: Pulling fs layer
2020-11-02T14:08:33.0668555Z 72c0738702f0: Waiting
2020-11-02T14:08:33.0668847Z 14821382b808: Waiting
2020-11-02T14:08:33.0669154Z 213eace38d21: Waiting
2020-11-02T14:08:33.0669814Z 555aaae525c9: Waiting
2020-11-02T14:08:33.6230488Z ee7e89337106: Verifying Checksum
2020-11-02T14:08:33.6237414Z ee7e89337106: Download complete
2020-11-02T14:08:33.6347305Z f22377f2f0d9: Verifying Checksum
2020-11-02T14:08:33.6355681Z f22377f2f0d9: Download complete
2020-11-02T14:08:33.7369184Z 14821382b808: Verifying Checksum
2020-11-02T14:08:33.7372298Z 14821382b808: Download complete
2020-11-02T14:08:33.7389561Z 72c0738702f0: Verifying Checksum
2020-11-02T14:08:33.7392700Z 72c0738702f0: Download complete
2020-11-02T14:08:33.8394173Z 213eace38d21: Verifying Checksum
2020-11-02T14:08:33.8398107Z 213eace38d21: Download complete
2020-11-02T14:08:33.8785674Z 555aaae525c9: Verifying Checksum
2020-11-02T14:08:33.8786188Z 555aaae525c9: Download complete
2020-11-02T14:08:35.2259821Z 15b785f9b2f0: Verifying Checksum
2020-11-02T14:08:35.2260450Z 15b785f9b2f0: Download complete
2020-11-02T14:08:38.5439969Z ee7e89337106: Pull complete
2020-11-02T14:08:46.3456042Z 15b785f9b2f0: Pull complete
2020-11-02T14:08:51.9844773Z f22377f2f0d9: Pull complete
2020-11-02T14:08:52.0491312Z 14821382b808: Pull complete
2020-11-02T14:08:53.6317617Z 72c0738702f0: Pull complete
2020-11-02T14:08:54.5684701Z 213eace38d21: Pull complete
2020-11-02T14:08:54.5686307Z 555aaae525c9: Pull complete
2020-11-02T14:08:54.5687382Z Digest: sha256:2d5f6b3b673026b5f6db375a578397f919124c2d9cd35679fc323e68091fc40e
2020-11-02T14:08:54.5690910Z Status: Downloaded newer image for quay.io/submariner/shipyard-dapper-base:devel
2020-11-02T14:08:54.5691694Z  ---> 45327d47bb4a
2020-11-02T14:08:54.5692954Z Step 2/6 : ENV DAPPER_ENV="REPO TAG QUAY_USERNAME QUAY_PASSWORD GITHUB_SHA BUILD_ARGS CLUSTERS_ARGS DEPLOY_ARGS RELEASE_ARGS"     DAPPER_SOURCE=/go/src/github.com/submariner-io/lighthouse DAPPER_DOCKER_SOCKET=true
2020-11-02T14:08:55.9979008Z  ---> Running in 29fda15ef07a
2020-11-02T14:08:56.8616184Z Removing intermediate container 29fda15ef07a
2020-11-02T14:08:56.8618781Z  ---> d738e368da0f
2020-11-02T14:08:56.8620412Z Step 3/6 : ENV DAPPER_OUTPUT=${DAPPER_SOURCE}/output
2020-11-02T14:08:56.8916403Z  ---> Running in c06ea0aa80bb
2020-11-02T14:08:57.7855071Z Removing intermediate container c06ea0aa80bb
2020-11-02T14:08:57.7856240Z  ---> a08ac595035e
2020-11-02T14:08:57.7908830Z Step 4/6 : WORKDIR ${DAPPER_SOURCE}
2020-11-02T14:08:57.8132200Z  ---> Running in 18d96c1f6d9a
2020-11-02T14:08:58.7148094Z Removing intermediate container 18d96c1f6d9a
2020-11-02T14:08:58.7149322Z  ---> b8ceb069cd5c
2020-11-02T14:08:58.7149920Z Step 5/6 : ENTRYPOINT ["/opt/shipyard/scripts/entry"]
2020-11-02T14:08:58.7454480Z  ---> Running in bf8082165506
2020-11-02T14:08:59.6116631Z Removing intermediate container bf8082165506
2020-11-02T14:08:59.6117691Z  ---> 43f258a32a70
2020-11-02T14:08:59.6117967Z Step 6/6 : CMD ["sh"]
2020-11-02T14:08:59.6368972Z  ---> Running in 58c827412bdb
2020-11-02T14:09:00.5342516Z Removing intermediate container 58c827412bdb
2020-11-02T14:09:00.5343960Z  ---> c35f75223205
2020-11-02T14:09:00.5475037Z Successfully built c35f75223205
2020-11-02T14:09:00.5512045Z Successfully tagged lighthouse:HEAD
2020-11-02T14:09:01.1167613Z [lighthouse]$ trap chown -R 1001:116 . exit
2020-11-02T14:09:01.1187270Z [lighthouse]$ mkdir -p bin dist output
2020-11-02T14:09:01.1188400Z [lighthouse]$ make e2e -- using=helm
2020-11-02T14:09:01.1738387Z fatal: No names found, cannot describe anything.
2020-11-02T14:09:01.1824027Z Makefile:40: warning: overriding recipe for target 'deploy'
2020-11-02T14:09:01.1824923Z /opt/shipyard/Makefile.inc:36: warning: ignoring old recipe for target 'deploy'
2020-11-02T14:09:01.1865078Z /opt/shipyard/scripts/e2e.sh cluster1 cluster2
2020-11-02T14:09:01.1865527Z Downloading shflags 1.0.3
2020-11-02T14:09:01.2007866Z   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
2020-11-02T14:09:01.2015947Z                                  Dload  Upload   Total   Spent    Left  Speed
2020-11-02T14:09:01.2016899Z
2020-11-02T14:09:01.2442486Z   0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
2020-11-02T14:09:01.2443190Z 100 31091  100 31091    0     0   690k      0 --:--:-- --:--:-- --:--:--  690k
2020-11-02T14:09:01.2929388Z [lighthouse]$ source /opt/shipyard/scripts/lib/utils
2020-11-02T14:09:01.2940343Z [lighthouse]$ . /opt/shipyard/scripts/lib/source_only
2020-11-02T14:09:01.2947203Z [lighthouse]$ script_name=utils
2020-11-02T14:09:01.2962380Z [lighthouse]$ exec_name=e2e.sh
2020-11-02T14:09:01.2981032Z [lighthouse]$ source /opt/shipyard/scripts/lib/cluster_settings
2020-11-02T14:09:01.2994159Z [lighthouse]$ . /opt/shipyard/scripts/lib/source_only
2020-11-02T14:09:01.3005383Z [lighthouse]$ script_name=cluster_settings
2020-11-02T14:09:01.3015271Z [lighthouse]$ exec_name=e2e.sh
2020-11-02T14:09:01.3039032Z [lighthouse]$ broker=cluster1
2020-11-02T14:09:01.3051573Z [lighthouse]$ declare -gA cluster_nodes
2020-11-02T14:09:01.3063089Z [lighthouse]$ cluster_nodes[cluster1]=control-plane worker
2020-11-02T14:09:01.3075981Z [lighthouse]$ cluster_nodes[cluster2]=control-plane worker
2020-11-02T14:09:01.3087330Z [lighthouse]$ cluster_nodes[cluster3]=control-plane worker worker
2020-11-02T14:09:01.3099990Z [lighthouse]$ declare -gA cluster_subm
2020-11-02T14:09:01.3111966Z [lighthouse]$ cluster_subm[cluster1]=true
2020-11-02T14:09:01.3119940Z [lighthouse]$ cluster_subm[cluster2]=true
2020-11-02T14:09:01.3131052Z [lighthouse]$ cluster_subm[cluster3]=true
2020-11-02T14:09:01.3142112Z [lighthouse]$ declare -gA cluster_cni
2020-11-02T14:09:01.3154655Z [lighthouse]$ cluster_cni[cluster2]=weave
2020-11-02T14:09:01.3171258Z [lighthouse]$ cluster_cni[cluster3]=weave
2020-11-02T14:09:01.3190421Z [lighthouse]$ declare_kubeconfig
2020-11-02T14:09:01.3203162Z [lighthouse]$ declare_kubeconfig
2020-11-02T14:09:01.3215111Z [lighthouse]$ source /opt/shipyard/scripts/lib/kubecfg
2020-11-02T14:09:01.3225984Z [lighthouse]$ export KUBECONFIG
2020-11-02T14:09:01.3258276Z [lighthouse]$ KUBECONFIG=
2020-11-02T14:09:01.3272645Z [lighthouse]$ find /go/src/github.com/submariner-io/lighthouse/output/kubeconfigs -type f -printf %p:
2020-11-02T14:09:01.3287570Z find: '/go/src/github.com/submariner-io/lighthouse/output/kubeconfigs': No such file or directory
2020-11-02T14:09:01.3300338Z [lighthouse]$ :
2020-11-02T14:09:01.3317490Z [lighthouse]$ deploy_env_once
2020-11-02T14:09:01.3328422Z [lighthouse]$ deploy_env_once
2020-11-02T14:09:01.3349884Z 
2020-11-02T14:09:01.5985640Z [lighthouse]$ make deploy
2020-11-02T14:09:01.6016634Z make[1]: Entering directory '/go/src/github.com/submariner-io/lighthouse'
2020-11-02T14:09:01.6464043Z fatal: No names found, cannot describe anything.
2020-11-02T14:09:01.6546166Z Makefile:40: warning: overriding recipe for target 'deploy'
2020-11-02T14:09:01.6547027Z /opt/shipyard/Makefile.inc:36: warning: ignoring old recipe for target 'deploy'
2020-11-02T14:09:01.6615162Z find: 'bin/lighthouse-agent': No such file or directory
2020-11-02T14:09:01.9866118Z go mod download
2020-11-02T14:09:28.0957801Z go mod vendor
2020-11-02T14:09:28.9814397Z /opt/shipyard/scripts/compile.sh bin/lighthouse-agent pkg/agent/main.go
2020-11-02T14:09:29.0314702Z [lighthouse]$ mkdir -p bin
2020-11-02T14:09:29.0331785Z Building 'bin/lighthouse-agent' (ldflags: '')
2020-11-02T14:09:29.0343514Z [lighthouse]$ ldflags=-s -w 
2020-11-02T14:09:29.0355808Z [lighthouse]$ CGO_ENABLED=0 go build -ldflags -s -w  -o bin/lighthouse-agent pkg/agent/main.go
2020-11-02T14:10:13.4022041Z [lighthouse]$ upx bin/lighthouse-agent
2020-11-02T14:10:19.2482614Z                        Ultimate Packer for eXecutables
2020-11-02T14:10:19.2483761Z                           Copyright (C) 1996 - 2020
2020-11-02T14:10:19.2484221Z UPX 3.96        Markus Oberhumer, Laszlo Molnar & John Reiser   Jan 23rd 2020
2020-11-02T14:10:19.2484576Z
2020-11-02T14:10:19.2484873Z         File size         Ratio      Format      Name
2020-11-02T14:10:19.2485636Z    --------------------   ------   -----------   -----------
2020-11-02T14:10:19.2486180Z   26943488 ->   7833544   29.07%   linux/amd64   lighthouse-agent
2020-11-02T14:10:19.2486485Z
2020-11-02T14:10:19.2486729Z Packed 1 file.
2020-11-02T14:10:19.2487661Z /opt/shipyard/scripts/build_image.sh -i lighthouse-agent -f package/Dockerfile.lighthouse-agent
2020-11-02T14:10:19.2986495Z fatal: No names found, cannot describe anything.
2020-11-02T14:10:19.3811906Z [lighthouse]$ set -e
2020-11-02T14:10:19.3818716Z [lighthouse]$ local_image=quay.io/submariner/lighthouse-agent:dev
2020-11-02T14:10:19.3829809Z [lighthouse]$ cache_image=quay.io/submariner/lighthouse-agent:devel
2020-11-02T14:10:19.3838734Z [lighthouse]$ cache_flag=
2020-11-02T14:10:19.3850325Z [lighthouse]$ cache_flag=--cache-from quay.io/submariner/lighthouse-agent:devel
2020-11-02T14:10:19.3867236Z [lighthouse]$ docker image ls -q quay.io/submariner/lighthouse-agent:devel
2020-11-02T14:10:19.7110872Z [lighthouse]$ docker pull quay.io/submariner/lighthouse-agent:devel
2020-11-02T14:10:20.1747671Z devel: Pulling from submariner/lighthouse-agent
2020-11-02T14:10:20.1748551Z 0fd3b5213a9b: Pulling fs layer
2020-11-02T14:10:20.1749074Z aebb8c556853: Pulling fs layer
2020-11-02T14:10:20.1749454Z 8abcb662399b: Pulling fs layer
2020-11-02T14:10:20.1749801Z d69f5364b4cc: Pulling fs layer
2020-11-02T14:10:20.1750141Z d69f5364b4cc: Waiting
2020-11-02T14:10:20.2685752Z aebb8c556853: Verifying Checksum
2020-11-02T14:10:20.2686225Z aebb8c556853: Download complete
2020-11-02T14:10:20.2714697Z 8abcb662399b: Verifying Checksum
2020-11-02T14:10:20.2715185Z 8abcb662399b: Download complete
2020-11-02T14:10:20.4169517Z d69f5364b4cc: Verifying Checksum
2020-11-02T14:10:20.4177385Z d69f5364b4cc: Download complete
2020-11-02T14:10:20.6611859Z 0fd3b5213a9b: Verifying Checksum
2020-11-02T14:10:20.6612841Z 0fd3b5213a9b: Download complete
2020-11-02T14:10:22.6917415Z 0fd3b5213a9b: Pull complete
2020-11-02T14:10:22.7559558Z aebb8c556853: Pull complete
2020-11-02T14:10:22.8288073Z 8abcb662399b: Pull complete
2020-11-02T14:10:22.9837955Z d69f5364b4cc: Pull complete
2020-11-02T14:10:22.9868616Z Digest: sha256:acdd2e7c54ba2f84a38696786441ade2a5a0950d987b9f0b948ef6556b88827c
2020-11-02T14:10:22.9914895Z Status: Downloaded newer image for quay.io/submariner/lighthouse-agent:devel
2020-11-02T14:10:22.9954381Z quay.io/submariner/lighthouse-agent:devel
2020-11-02T14:10:22.9959281Z [lighthouse]$ grep FROM package/Dockerfile.lighthouse-agent
2020-11-02T14:10:22.9960127Z [lighthouse]$ cut -f2 -d 
2020-11-02T14:10:22.9960666Z [lighthouse]$ grep -v scratch
2020-11-02T14:10:22.9974283Z [lighthouse]$ cache_flag+= --cache-from registry.access.redhat.com/ubi8/ubi-minimal
2020-11-02T14:10:22.9984614Z [lighthouse]$ docker pull registry.access.redhat.com/ubi8/ubi-minimal
2020-11-02T14:10:23.3124300Z Using default tag: latest
2020-11-02T14:10:23.5517342Z latest: Pulling from ubi8/ubi-minimal
2020-11-02T14:10:23.6596901Z 0fd3b5213a9b: Already exists
2020-11-02T14:10:23.6616443Z aebb8c556853: Already exists
2020-11-02T14:10:23.7711612Z Digest: sha256:5cfbaf45ca96806917830c183e9f37df2e913b187aadb32e89fd83fa455ebaa6
2020-11-02T14:10:23.7762429Z Status: Downloaded newer image for registry.access.redhat.com/ubi8/ubi-minimal:latest
2020-11-02T14:10:23.7772443Z registry.access.redhat.com/ubi8/ubi-minimal:latest
2020-11-02T14:10:23.7823563Z [lighthouse]$ buildargs_flag=--build-arg BUILDKIT_INLINE_CACHE=1
2020-11-02T14:10:23.7826169Z [lighthouse]$ DOCKER_BUILDKIT=1 docker build -t quay.io/submariner/lighthouse-agent:dev --cache-from quay.io/submariner/lighthouse-agent:devel --cache-from registry.access.redhat.com/ubi8/ubi-minimal -f package/Dockerfile.lighthouse-agent --build-arg BUILDKIT_INLINE_CACHE=1 .
2020-11-02T14:10:24.1983982Z #1 [internal] load .dockerignore
2020-11-02T14:10:24.1984633Z #1 transferring context: 2B done
2020-11-02T14:10:24.1984926Z #1 DONE 0.0s
2020-11-02T14:10:24.1985320Z
2020-11-02T14:10:24.1986453Z #2 [internal] load build definition from Dockerfile.lighthouse-agent
2020-11-02T14:10:24.1987085Z #2 transferring dockerfile: 238B done
2020-11-02T14:10:24.1987390Z #2 DONE 0.0s
2020-11-02T14:10:24.1987566Z
2020-11-02T14:10:24.1988296Z #3 [internal] load metadata for registry.access.redhat.com/ubi8/ubi-minimal...
2020-11-02T14:10:24.1988853Z #3 DONE 0.0s
2020-11-02T14:10:24.1989008Z
2020-11-02T14:10:24.1989625Z #4 [1/3] FROM registry.access.redhat.com/ubi8/ubi-minimal
2020-11-02T14:10:24.1990091Z #4 DONE 0.0s
2020-11-02T14:10:24.1990244Z
2020-11-02T14:10:24.1990972Z #8 importing cache manifest from registry.access.redhat.com/ubi8/ubi-minima...
2020-11-02T14:10:24.1991811Z #8 DONE 0.0s
2020-11-02T14:10:24.1991970Z
2020-11-02T14:10:24.1992655Z #9 importing cache manifest from quay.io/submariner/lighthouse-agent:devel
2020-11-02T14:10:24.1993131Z #9 DONE 0.0s
2020-11-02T14:10:24.1993301Z
2020-11-02T14:10:24.1993575Z #6 [internal] load build context
2020-11-02T14:10:24.1993948Z #6 transferring context: 7.84MB 0.1s done
2020-11-02T14:10:24.2379479Z #6 DONE 0.1s
2020-11-02T14:10:24.2393691Z
2020-11-02T14:10:24.2394518Z #5 [2/3] WORKDIR /var/submariner
2020-11-02T14:10:24.2394997Z #5 CACHED
2020-11-02T14:10:24.2395332Z
2020-11-02T14:10:24.2396777Z #7 [3/3] COPY bin/lighthouse-agent package/lighthouse-agent.sh /usr/local/b...
2020-11-02T14:10:24.2397454Z #7 CACHED
2020-11-02T14:10:24.2397748Z
2020-11-02T14:10:24.2398100Z #10 exporting to image
2020-11-02T14:10:24.2398531Z #10 exporting layers done
2020-11-02T14:10:24.2399450Z #10 writing image sha256:719dff01b005de55b21a9122162942ac7e43a27aeeeec31a17f33bebc6c2f8e4 done
2020-11-02T14:10:24.2401121Z #10 naming to quay.io/submariner/lighthouse-agent:dev done
2020-11-02T14:10:24.2402882Z #10 DONE 0.0s
2020-11-02T14:10:24.2403326Z
2020-11-02T14:10:24.2404344Z #11 exporting cache
2020-11-02T14:10:24.2404709Z #11 preparing build cache for export done
2020-11-02T14:10:24.2405048Z #11 DONE 0.0s
2020-11-02T14:10:24.2411562Z [lighthouse]$ docker tag quay.io/submariner/lighthouse-agent:dev quay.io/submariner/lighthouse-agent:devel
2020-11-02T14:10:24.5460132Z touch package/.image.lighthouse-agent
2020-11-02T14:10:24.5517310Z find: 'bin/lighthouse-coredns': No such file or directory
2020-11-02T14:10:24.8750582Z /opt/shipyard/scripts/compile.sh bin/lighthouse-coredns pkg/coredns/main.go
2020-11-02T14:10:24.9215768Z [lighthouse]$ mkdir -p bin
2020-11-02T14:10:24.9233260Z Building 'bin/lighthouse-coredns' (ldflags: '')
2020-11-02T14:10:24.9243944Z [lighthouse]$ ldflags=-s -w 
2020-11-02T14:10:24.9256646Z [lighthouse]$ CGO_ENABLED=0 go build -ldflags -s -w  -o bin/lighthouse-coredns pkg/coredns/main.go
2020-11-02T14:10:56.2600805Z [lighthouse]$ upx bin/lighthouse-coredns
2020-11-02T14:11:04.8517826Z                        Ultimate Packer for eXecutables
2020-11-02T14:11:04.8520279Z                           Copyright (C) 1996 - 2020
2020-11-02T14:11:04.8520729Z UPX 3.96        Markus Oberhumer, Laszlo Molnar & John Reiser   Jan 23rd 2020
2020-11-02T14:11:04.8521088Z
2020-11-02T14:11:04.8521354Z         File size         Ratio      Format      Name
2020-11-02T14:11:04.8521853Z    --------------------   ------   -----------   -----------
2020-11-02T14:11:04.8522409Z   39477248 ->  11747520   29.76%   linux/amd64   lighthouse-coredns
2020-11-02T14:11:04.8522872Z
2020-11-02T14:11:04.8523083Z Packed 1 file.
2020-11-02T14:11:04.8524119Z /opt/shipyard/scripts/build_image.sh -i lighthouse-coredns -f package/Dockerfile.lighthouse-coredns
2020-11-02T14:11:04.8964654Z fatal: No names found, cannot describe anything.
2020-11-02T14:11:04.9765496Z [lighthouse]$ set -e
2020-11-02T14:11:04.9776706Z [lighthouse]$ local_image=quay.io/submariner/lighthouse-coredns:dev
2020-11-02T14:11:04.9786468Z [lighthouse]$ cache_image=quay.io/submariner/lighthouse-coredns:devel
2020-11-02T14:11:04.9799845Z [lighthouse]$ cache_flag=
2020-11-02T14:11:04.9812842Z [lighthouse]$ cache_flag=--cache-from quay.io/submariner/lighthouse-coredns:devel
2020-11-02T14:11:04.9832542Z [lighthouse]$ docker image ls -q quay.io/submariner/lighthouse-coredns:devel
2020-11-02T14:11:05.3021177Z [lighthouse]$ docker pull quay.io/submariner/lighthouse-coredns:devel
2020-11-02T14:11:05.7757766Z devel: Pulling from submariner/lighthouse-coredns
2020-11-02T14:11:05.7759060Z 58db8e8debc5: Pulling fs layer
2020-11-02T14:11:05.7759540Z c15e840cda60: Pulling fs layer
2020-11-02T14:11:05.7759995Z a633d0345583: Pulling fs layer
2020-11-02T14:11:05.9744091Z a633d0345583: Verifying Checksum
2020-11-02T14:11:05.9745301Z a633d0345583: Download complete
2020-11-02T14:11:06.0018681Z c15e840cda60: Verifying Checksum
2020-11-02T14:11:06.0019188Z c15e840cda60: Download complete
2020-11-02T14:11:06.1815046Z 58db8e8debc5: Verifying Checksum
2020-11-02T14:11:06.1815569Z 58db8e8debc5: Download complete
2020-11-02T14:11:07.4340131Z 58db8e8debc5: Pull complete
2020-11-02T14:11:07.8289048Z c15e840cda60: Pull complete
2020-11-02T14:11:08.0158280Z a633d0345583: Pull complete
2020-11-02T14:11:08.0184043Z Digest: sha256:e5c63ef292997ce6af66be1730ad3541a613e6653068635326e59302d215ba20
2020-11-02T14:11:08.0206463Z Status: Downloaded newer image for quay.io/submariner/lighthouse-coredns:devel
2020-11-02T14:11:08.0242610Z quay.io/submariner/lighthouse-coredns:devel
2020-11-02T14:11:08.0280951Z [lighthouse]$ grep FROM package/Dockerfile.lighthouse-coredns
2020-11-02T14:11:08.0281798Z [lighthouse]$ cut -f2 -d 
2020-11-02T14:11:08.0282297Z [lighthouse]$ grep -v scratch
2020-11-02T14:11:08.0294031Z [lighthouse]$ cache_flag+= --cache-from debian:stable-slim
2020-11-02T14:11:08.0305318Z [lighthouse]$ docker pull debian:stable-slim
2020-11-02T14:11:08.6740276Z stable-slim: Pulling from library/debian
2020-11-02T14:11:08.7620123Z 58db8e8debc5: Already exists
2020-11-02T14:11:08.8453052Z Digest: sha256:b488c06160cb6b7b0fc33081de1001c194ebcd9a7fff9dfff27381cd350dcbe8
2020-11-02T14:11:08.8477529Z Status: Downloaded newer image for debian:stable-slim
2020-11-02T14:11:08.8507848Z docker.io/library/debian:stable-slim
2020-11-02T14:11:08.8592619Z [lighthouse]$ buildargs_flag=--build-arg BUILDKIT_INLINE_CACHE=1
2020-11-02T14:11:08.8595073Z [lighthouse]$ DOCKER_BUILDKIT=1 docker build -t quay.io/submariner/lighthouse-coredns:dev --cache-from quay.io/submariner/lighthouse-coredns:devel --cache-from debian:stable-slim -f package/Dockerfile.lighthouse-coredns --build-arg BUILDKIT_INLINE_CACHE=1 .
2020-11-02T14:11:09.2813772Z #1 [internal] load .dockerignore
2020-11-02T14:11:09.2820686Z #1 transferring context: 2B done
2020-11-02T14:11:09.2821092Z #1 DONE 0.0s
2020-11-02T14:11:09.2821497Z
2020-11-02T14:11:09.2822719Z #2 [internal] load build definition from Dockerfile.lighthouse-coredns
2020-11-02T14:11:09.2823397Z #2 transferring dockerfile: 284B done
2020-11-02T14:11:09.2823732Z #2 DONE 0.0s
2020-11-02T14:11:09.2823906Z
2020-11-02T14:11:09.2824505Z #3 [internal] load metadata for docker.io/library/debian:stable-slim
2020-11-02T14:11:09.2825132Z #3 DONE 0.0s
2020-11-02T14:11:09.2825315Z
2020-11-02T14:11:09.2826028Z #8 importing cache manifest from quay.io/submariner/lighthouse-coredns:deve...
2020-11-02T14:11:09.2826551Z #8 DONE 0.0s
2020-11-02T14:11:09.2826734Z
2020-11-02T14:11:09.2827285Z #9 importing cache manifest from debian:stable-slim
2020-11-02T14:11:09.2827689Z #9 DONE 0.0s
2020-11-02T14:11:09.2827852Z
2020-11-02T14:11:09.2828734Z #4 [1/3] FROM docker.io/library/debian:stable-slim
2020-11-02T14:11:09.2829080Z #4 DONE 0.0s
2020-11-02T14:11:09.2829232Z
2020-11-02T14:11:09.2829504Z #6 [internal] load build context
2020-11-02T14:11:09.3186020Z #6 transferring context: 11.75MB 0.1s done
2020-11-02T14:11:09.3186386Z #6 DONE 0.1s
2020-11-02T14:11:09.3186544Z
2020-11-02T14:11:09.3187379Z #5 [2/3] RUN apt-get update && apt-get -y install ca-certificates && update...
2020-11-02T14:11:09.3187819Z #5 CACHED
2020-11-02T14:11:09.3187999Z
2020-11-02T14:11:09.3188514Z #7 [3/3] COPY bin/lighthouse-coredns /usr/local/bin/
2020-11-02T14:11:09.3189114Z #7 CACHED
2020-11-02T14:11:09.3189288Z
2020-11-02T14:11:09.3189535Z #10 exporting to image
2020-11-02T14:11:09.3189850Z #10 exporting layers done
2020-11-02T14:11:09.3190693Z #10 writing image sha256:0c50ecdf78c1659c2b6b4d91cf8c3c629a3a825b419ef681b7bd95996b4ebd82 done
2020-11-02T14:11:09.3192367Z #10 naming to quay.io/submariner/lighthouse-coredns:dev done
2020-11-02T14:11:09.3192811Z #10 DONE 0.0s
2020-11-02T14:11:09.3192983Z
2020-11-02T14:11:09.3193208Z #11 exporting cache
2020-11-02T14:11:09.3193559Z #11 preparing build cache for export done
2020-11-02T14:11:09.3193872Z #11 DONE 0.0s
2020-11-02T14:11:09.3277298Z [lighthouse]$ docker tag quay.io/submariner/lighthouse-coredns:dev quay.io/submariner/lighthouse-coredns:devel
2020-11-02T14:11:09.6420520Z touch package/.image.lighthouse-coredns
2020-11-02T14:11:09.6445029Z /opt/shipyard/scripts/clusters.sh --cluster_settings /go/src/github.com/submariner-io/lighthouse/scripts/cluster_settings
2020-11-02T14:11:09.7201631Z Running with: k8s_version=1.17.0, olm_version=0.14.1, olm=false, globalnet=false, registry_inmemory=true, cluster_settings=/go/src/github.com/submariner-io/lighthouse/scripts/cluster_settings, timeout=5m
2020-11-02T14:11:09.7230048Z [lighthouse]$ source /opt/shipyard/scripts/lib/utils
2020-11-02T14:11:09.7240141Z [lighthouse]$ . /opt/shipyard/scripts/lib/source_only
2020-11-02T14:11:09.7252349Z [lighthouse]$ script_name=utils
2020-11-02T14:11:09.7266777Z [lighthouse]$ exec_name=clusters.sh
2020-11-02T14:11:09.7288305Z [lighthouse]$ source /opt/shipyard/scripts/lib/cluster_settings
2020-11-02T14:11:09.7299733Z [lighthouse]$ . /opt/shipyard/scripts/lib/source_only
2020-11-02T14:11:09.7312667Z [lighthouse]$ script_name=cluster_settings
2020-11-02T14:11:09.7324335Z [lighthouse]$ exec_name=clusters.sh
2020-11-02T14:11:09.7344223Z [lighthouse]$ broker=cluster1
2020-11-02T14:11:09.7355045Z [lighthouse]$ declare -gA cluster_nodes
2020-11-02T14:11:09.7367250Z [lighthouse]$ cluster_nodes[cluster1]=control-plane worker
2020-11-02T14:11:09.7379268Z [lighthouse]$ cluster_nodes[cluster2]=control-plane worker
2020-11-02T14:11:09.7392025Z [lighthouse]$ cluster_nodes[cluster3]=control-plane worker worker
2020-11-02T14:11:09.7405383Z [lighthouse]$ declare -gA cluster_subm
2020-11-02T14:11:09.7417503Z [lighthouse]$ cluster_subm[cluster1]=true
2020-11-02T14:11:09.7429966Z [lighthouse]$ cluster_subm[cluster2]=true
2020-11-02T14:11:09.7442031Z [lighthouse]$ cluster_subm[cluster3]=true
2020-11-02T14:11:09.7454243Z [lighthouse]$ declare -gA cluster_cni
2020-11-02T14:11:09.7466691Z [lighthouse]$ cluster_cni[cluster2]=weave
2020-11-02T14:11:09.7478499Z [lighthouse]$ cluster_cni[cluster3]=weave
2020-11-02T14:11:09.7495932Z [lighthouse]$ source /go/src/github.com/submariner-io/lighthouse/scripts/cluster_settings
2020-11-02T14:11:09.7510192Z [lighthouse]$ . /opt/shipyard/scripts/lib/source_only
2020-11-02T14:11:09.7523208Z [lighthouse]$ script_name=cluster_settings
2020-11-02T14:11:09.7535788Z [lighthouse]$ exec_name=clusters.sh
2020-11-02T14:11:09.7559125Z [lighthouse]$ cluster_nodes[cluster1]=control-plane worker worker
2020-11-02T14:11:09.7571502Z [lighthouse]$ cluster_nodes[cluster2]=control-plane worker worker
2020-11-02T14:11:09.7640377Z [lighthouse]$ cat
2020-11-02T14:11:09.7662286Z [lighthouse]$ typeset -p cluster_nodes
2020-11-02T14:11:09.7675966Z [lighthouse]$ cut -f 2- -d=
2020-11-02T14:11:09.7706440Z [lighthouse]$ typeset -p cluster_subm
2020-11-02T14:11:09.7724581Z [lighthouse]$ cut -f 2- -d=
2020-11-02T14:11:09.7750699Z Cluster settings::
2020-11-02T14:11:09.7752237Z   broker - 'cluster1'
2020-11-02T14:11:09.7753271Z   clusters - 'cluster1' 'cluster2'
2020-11-02T14:11:09.7755406Z   nodes per cluster - ([cluster3]="control-plane worker worker" [cluster2]="control-plane worker worker" [cluster1]="control-plane worker worker" )
2020-11-02T14:11:09.7757781Z   install submariner - ([cluster2]="true" [cluster1]="true" )
2020-11-02T14:11:09.7779117Z [lighthouse]$ rm -rf /go/src/github.com/submariner-io/lighthouse/output/kubeconfigs
2020-11-02T14:11:09.7805902Z [lighthouse]$ mkdir -p /go/src/github.com/submariner-io/lighthouse/output/kubeconfigs
2020-11-02T14:11:09.7832252Z [lighthouse]$ run_local_registry
2020-11-02T14:11:09.7842975Z [lighthouse]$ run_local_registry
2020-11-02T14:11:09.7853958Z [lighthouse]$ registry_running
2020-11-02T14:11:09.7864807Z [lighthouse]$ registry_running
2020-11-02T14:11:09.7877972Z [lighthouse]$ docker ps --filter name=^/?kind-registry$
2020-11-02T14:11:09.7890971Z [lighthouse]$ grep kind-registry
2020-11-02T14:11:10.1276263Z [lighthouse]$ return 0
2020-11-02T14:11:10.1280141Z Deploying local registry kind-registry to serve images centrally.
2020-11-02T14:11:10.1291914Z [lighthouse]$ local volume_flag
2020-11-02T14:11:10.1305042Z [lighthouse]$ volume_flag=-v /dev/shm/kind-registry:/var/lib/registry
2020-11-02T14:11:10.1318145Z [lighthouse]$ docker run -d -v /dev/shm/kind-registry:/var/lib/registry -p 5000:5000 --restart=always --name kind-registry registry:2
2020-11-02T14:11:10.4504880Z Unable to find image 'registry:2' locally
2020-11-02T14:11:10.6207002Z 2: Pulling from library/registry
2020-11-02T14:11:10.7255930Z cbdbe7a5bc2a: Already exists
2020-11-02T14:11:10.7377957Z 47112e65547d: Pulling fs layer
2020-11-02T14:11:10.7378327Z 46bcb632e506: Pulling fs layer
2020-11-02T14:11:10.7378707Z c1cc712bcecd: Pulling fs layer
2020-11-02T14:11:10.7379071Z 3db6272dcbfa: Pulling fs layer
2020-11-02T14:11:10.7379408Z 3db6272dcbfa: Waiting
2020-11-02T14:11:10.8182613Z c1cc712bcecd: Verifying Checksum
2020-11-02T14:11:10.8183056Z c1cc712bcecd: Download complete
2020-11-02T14:11:10.8367095Z 47112e65547d: Download complete
2020-11-02T14:11:10.9004944Z 3db6272dcbfa: Verifying Checksum
2020-11-02T14:11:10.9005432Z 3db6272dcbfa: Download complete
2020-11-02T14:11:10.9362467Z 46bcb632e506: Download complete
2020-11-02T14:11:10.9820996Z 47112e65547d: Pull complete
2020-11-02T14:11:11.2310394Z 46bcb632e506: Pull complete
2020-11-02T14:11:11.2978529Z c1cc712bcecd: Pull complete
2020-11-02T14:11:11.3633458Z 3db6272dcbfa: Pull complete
2020-11-02T14:11:11.3668075Z Digest: sha256:8be26f81ffea54106bae012c6f349df70f4d5e7e2ec01b143c46e2c03b9e551d
2020-11-02T14:11:11.3686720Z Status: Downloaded newer image for registry:2
2020-11-02T14:11:14.5519881Z c3300880df941641135eb7a53278bd9cc89eaa314b54b22f1864c965277f06ef
2020-11-02T14:11:15.2878666Z [lighthouse]$ registry_ip=172.17.0.3
2020-11-02T14:11:15.2892227Z [lighthouse]$ docker inspect -f {{.NetworkSettings.IPAddress}} kind-registry
2020-11-02T14:11:15.6110878Z [lighthouse]$ declare_cidrs
2020-11-02T14:11:15.6124852Z [lighthouse]$ declare_cidrs
2020-11-02T14:11:15.6135010Z [lighthouse]$ declare -gA cluster_CIDRs service_CIDRs global_CIDRs
2020-11-02T14:11:15.6148669Z [lighthouse]$ i=1
2020-11-02T14:11:15.6163949Z [lighthouse]$ [cluster1] add_cluster_cidrs 1 cluster1
2020-11-02T14:11:15.6182847Z [lighthouse]$ [cluster1] add_cluster_cidrs 1 cluster1
2020-11-02T14:11:15.6199358Z [lighthouse]$ [cluster1] local val=1
2020-11-02T14:11:15.6213030Z [lighthouse]$ [cluster1] local idx=cluster1
2020-11-02T14:11:15.6226562Z [lighthouse]$ [cluster1] cluster_CIDRs[cluster1]=10.1.0.0/16
2020-11-02T14:11:15.6241332Z [lighthouse]$ [cluster1] service_CIDRs[cluster1]=100.1.0.0/16
2020-11-02T14:11:15.6253313Z [lighthouse]$ [cluster1] i=2
2020-11-02T14:11:15.6266733Z [lighthouse]$ [cluster2] add_cluster_cidrs 2 cluster2
2020-11-02T14:11:15.6281323Z [lighthouse]$ [cluster2] add_cluster_cidrs 2 cluster2
2020-11-02T14:11:15.6293594Z [lighthouse]$ [cluster2] local val=2
2020-11-02T14:11:15.6307999Z [lighthouse]$ [cluster2] local idx=cluster2
2020-11-02T14:11:15.6321990Z [lighthouse]$ [cluster2] cluster_CIDRs[cluster2]=10.2.0.0/16
2020-11-02T14:11:15.6427074Z [lighthouse]$ [cluster2] service_CIDRs[cluster2]=100.2.0.0/16
2020-11-02T14:11:15.6432481Z [lighthouse]$ [cluster2] i=3
2020-11-02T14:11:15.6433234Z [lighthouse]$ [cluster2] run_all_clusters with_retries 3 create_kind_cluster
2020-11-02T14:11:15.6434339Z [lighthouse]$ [cluster2] run_all_clusters with_retries 3 create_kind_cluster
2020-11-02T14:11:15.6435315Z [lighthouse]$ [cluster2] run_parallel cluster1 cluster2 with_retries 3 create_kind_cluster
2020-11-02T14:11:15.6436490Z [lighthouse]$ [cluster2] run_parallel cluster1 cluster2 cluster1 cluster2 with_retries 3 create_kind_cluster
2020-11-02T14:11:15.6437406Z [lighthouse]$ [cluster2] local cmnd=with_retries
2020-11-02T14:11:15.6438074Z [lighthouse]$ [cluster2] declare -A pids
2020-11-02T14:11:15.6447959Z [lighthouse]$ [cluster2] eval echo cluster1 cluster2
2020-11-02T14:11:15.6470101Z [lighthouse]$ [cluster1] pids[cluster1]=3423
2020-11-02T14:11:15.6476960Z [lighthouse]$ [cluster1] set -o pipefail
2020-11-02T14:11:15.6489976Z [lighthouse]$ [cluster1] with_retries 3 create_kind_cluster
2020-11-02T14:11:15.6493097Z [lighthouse]$ [cluster2] pids[cluster2]=3426
2020-11-02T14:11:15.6509290Z [lighthouse]$ [cluster1] sed /\[cluster1]/!s/^/[cluster1] /
2020-11-02T14:11:15.6519453Z [lighthouse]$ [cluster2] wait 3426
2020-11-02T14:11:15.6529836Z [lighthouse]$ [cluster2] set -o pipefail
2020-11-02T14:11:15.6553290Z [lighthouse]$ [cluster2] with_retries 3 create_kind_cluster
2020-11-02T14:11:16.6132335Z [lighthouse]$ [cluster2] sed /\[cluster2]/!s/^/[cluster2] /
2020-11-02T14:14:48.1579483Z [lighthouse]$ [cluster1] with_retries
2020-11-02T14:14:48.1581127Z [lighthouse]$ [cluster1] local retries=3
2020-11-02T14:14:48.1582594Z [lighthouse]$ [cluster1] local cmnd=create_kind_cluster
2020-11-02T14:14:48.1583893Z [lighthouse]$ [cluster1] iteration=1
2020-11-02T14:14:48.1585031Z [lighthouse]$ [cluster1] wait 3444
2020-11-02T14:14:48.1586364Z [lighthouse]$ [cluster1] create_kind_cluster
2020-11-02T14:14:48.1587667Z [lighthouse]$ [cluster1] create_kind_cluster
2020-11-02T14:14:48.1589442Z [lighthouse]$ [cluster1] export KUBECONFIG=/go/src/github.com/submariner-io/lighthouse/output/kubeconfigs/kind-config-cluster1
2020-11-02T14:14:48.1592260Z [lighthouse]$ [cluster1] rm -f /go/src/github.com/submariner-io/lighthouse/output/kubeconfigs/kind-config-cluster1
2020-11-02T14:14:48.1594028Z [lighthouse]$ [cluster1] kind get clusters
2020-11-02T14:14:48.1595322Z [lighthouse]$ [cluster1] grep -q ^cluster1$
2020-11-02T14:14:48.1596215Z [cluster1] No kind clusters found.
2020-11-02T14:14:48.1597722Z [cluster1] Creating KIND cluster...
2020-11-02T14:14:48.1599283Z [lighthouse]$ [cluster1] generate_cluster_yaml
2020-11-02T14:14:48.1602753Z [lighthouse]$ [cluster1] generate_cluster_yaml
2020-11-02T14:14:48.1603513Z [lighthouse]$ [cluster1] local pod_cidr=10.1.0.0/16
2020-11-02T14:14:48.1604191Z [lighthouse]$ [cluster1] local service_cidr=100.1.0.0/16
2020-11-02T14:14:48.1604938Z [lighthouse]$ [cluster1] local dns_domain=cluster1.local
2020-11-02T14:14:48.1605685Z [lighthouse]$ [cluster1] local disable_cni=false
2020-11-02T14:14:48.1606353Z [lighthouse]$ [cluster1] disable_cni=true
2020-11-02T14:14:48.1607142Z [lighthouse]$ [cluster1] local nodes
2020-11-02T14:14:48.1607748Z [lighthouse]$ [cluster1] nodes=
2020-11-02T14:14:48.1608334Z [cluster1] - role: control-plane
2020-11-02T14:14:48.1608908Z [lighthouse]$ [cluster1] nodes=
2020-11-02T14:14:48.1609474Z [cluster1] - role: control-plane
2020-11-02T14:14:48.1610011Z [cluster1] - role: worker
2020-11-02T14:14:48.1610855Z [lighthouse]$ [cluster1] nodes=
2020-11-02T14:14:48.1611402Z [cluster1] - role: control-plane
2020-11-02T14:14:48.1611925Z [cluster1] - role: worker
2020-11-02T14:14:48.1612413Z [cluster1] - role: worker
2020-11-02T14:14:48.1612868Z [cluster1] 
2020-11-02T14:14:48.1613312Z [cluster1] 
2020-11-02T14:14:48.1613826Z [lighthouse]$ [cluster1] eval echo "---
2020-11-02T14:14:48.1614220Z [cluster1] kind: Cluster
2020-11-02T14:14:48.1614827Z [cluster1] apiVersion: kind.x-k8s.io/v1alpha4
2020-11-02T14:14:48.1615264Z [cluster1] networking:
2020-11-02T14:14:48.1615738Z [cluster1]   disableDefaultCNI: ${disable_cni}
2020-11-02T14:14:48.1616237Z [cluster1]   podSubnet: ${pod_cidr}
2020-11-02T14:14:48.1616693Z [cluster1]   serviceSubnet: ${service_cidr}
2020-11-02T14:14:48.1617259Z [cluster1] containerdConfigPatches:
2020-11-02T14:14:48.1617860Z [cluster1]   - |-
2020-11-02T14:14:48.1618665Z [cluster1]     [plugins.\"io.containerd.grpc.v1.cri\".registry.mirrors.\"localhost:5000\"]
2020-11-02T14:14:48.1620856Z [cluster1]       endpoint = [\"http://${registry_ip}:5000\"]
2020-11-02T14:14:48.1621377Z [cluster1] kubeadmConfigPatches:
2020-11-02T14:14:48.1622056Z [cluster1]   - |
2020-11-02T14:14:48.1622482Z [cluster1]     apiVersion: kubeadm.k8s.io/v1beta2
2020-11-02T14:14:48.1623042Z [cluster1]     kind: ClusterConfiguration
2020-11-02T14:14:48.1623516Z [cluster1]     metadata:
2020-11-02T14:14:48.1623859Z [cluster1]       name: config
2020-11-02T14:14:48.1624215Z [cluster1]     networking:
2020-11-02T14:14:48.1624588Z [cluster1]       podSubnet: ${pod_cidr}
2020-11-02T14:14:48.1625044Z [cluster1]       serviceSubnet: ${service_cidr}
2020-11-02T14:14:48.1625518Z [cluster1]       dnsDomain: ${dns_domain}
2020-11-02T14:14:48.1626073Z [cluster1] nodes: ${nodes}"
2020-11-02T14:14:48.1626940Z [lighthouse]$ [cluster1] cat /opt/shipyard/scripts/resources/kind-cluster-config.yaml
2020-11-02T14:14:48.1627833Z [lighthouse]$ [cluster1] local image_flag=
2020-11-02T14:14:48.1628622Z [lighthouse]$ [cluster1] image_flag=--image=kindest/node:v1.17.0
2020-11-02T14:14:48.1629903Z [lighthouse]$ [cluster1] kind create cluster --image=kindest/node:v1.17.0 --name=cluster1 --config=/opt/shipyard/scripts/resources/cluster1-config.yaml
2020-11-02T14:14:48.1630817Z [cluster1] Creating cluster "cluster1" ...
2020-11-02T14:14:48.1640148Z [cluster1]  • Ensuring node image (kindest/node:v1.17.0) 🖼  ...
2020-11-02T14:14:48.1641255Z [cluster1]  ✓ Ensuring node image (kindest/node:v1.17.0) 🖼
2020-11-02T14:14:48.1642305Z [cluster1]  • Preparing nodes 📦 📦 📦   ...
2020-11-02T14:14:48.1642946Z [cluster1]  ✓ Preparing nodes 📦 📦 📦
2020-11-02T14:14:48.1644132Z [cluster1]  • Writing configuration 📜  ...
2020-11-02T14:14:48.1645122Z [cluster1]  ✓ Writing configuration 📜
2020-11-02T14:14:48.1645989Z [cluster1]  • Starting control-plane 🕹️  ...
2020-11-02T14:14:48.1646854Z [cluster1]  ✓ Starting control-plane 🕹️
2020-11-02T14:14:48.1647703Z [cluster1]  • Installing StorageClass 💾  ...
2020-11-02T14:14:48.1648869Z [cluster1]  ✓ Installing StorageClass 💾
2020-11-02T14:14:48.1649732Z [cluster1]  • Joining worker nodes 🚜  ...
2020-11-02T14:14:48.1650528Z [cluster1]  ✓ Joining worker nodes 🚜
2020-11-02T14:14:48.1651435Z [cluster1] Set kubectl context to "kind-cluster1"
2020-11-02T14:14:48.1652145Z [cluster1] You can now use your cluster with:
2020-11-02T14:14:48.1652729Z [cluster1]
2020-11-02T14:14:48.1653581Z [cluster1] kubectl cluster-info --context kind-cluster1
2020-11-02T14:14:48.1654117Z [cluster1]
2020-11-02T14:14:48.1657505Z [cluster1] Not sure what to do next? 😅 Check out https://kind.sigs.k8s.io/docs/user/quick-start/
2020-11-02T14:14:48.1658831Z [lighthouse]$ [cluster1] kind_fixup_config
2020-11-02T14:14:48.1659743Z [lighthouse]$ [cluster1] kind_fixup_config
2020-11-02T14:14:48.1660636Z [lighthouse]$ [cluster1] local master_ip=172.17.0.4
2020-11-02T14:14:48.1662153Z [lighthouse]$ [cluster1] docker inspect -f {{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}} cluster1-control-plane
2020-11-02T14:14:57.3216215Z [lighthouse]$ [cluster1] head[lighthouse]$ [cluster2] with_retries
2020-11-02T14:14:57.3217301Z [lighthouse]$ [cluster2] local retries=3
2020-11-02T14:14:57.3218152Z [lighthouse]$ [cluster2] local cmnd=create_kind_cluster
2020-11-02T14:14:57.3218816Z [lighthouse]$ [cluster2] iteration=1
2020-11-02T14:14:57.3219428Z [lighthouse]$ [cluster2] create_kind_cluster
2020-11-02T14:14:57.3220198Z [lighthouse]$ [cluster2] wait 3454
2020-11-02T14:14:57.3221053Z [lighthouse]$ [cluster2] create_kind_cluster
2020-11-02T14:14:57.3222314Z [lighthouse]$ [cluster2] export KUBECONFIG=/go/src/github.com/submariner-io/lighthouse/output/kubeconfigs/kind-config-cluster2
2020-11-02T14:14:57.3223872Z [lighthouse]$ [cluster2] rm -f /go/src/github.com/submariner-io/lighthouse/output/kubeconfigs/kind-config-cluster2
2020-11-02T14:14:57.3225412Z [lighthouse]$ [cluster2] kind get clusters
2020-11-02T14:14:57.3226056Z [lighthouse]$ [cluster2] grep -q ^cluster2$
2020-11-02T14:14:57.3226654Z [cluster2] No kind clusters found.
2020-11-02T14:14:57.3227089Z [cluster2] Creating KIND cluster...
2020-11-02T14:14:57.3228082Z [lighthouse]$ [cluster2] generate_cluster_yaml
2020-11-02T14:14:57.3229108Z [lighthouse]$ [cluster2] generate_cluster_yaml
2020-11-02T14:14:57.3229942Z [lighthouse]$ [cluster2] local pod_cidr=10.2.0.0/16
2020-11-02T14:14:57.3230951Z [lighthouse]$ [cluster2] local service_cidr=100.2.0.0/16
2020-11-02T14:14:57.3232115Z [lighthouse]$ [cluster2] local dns_domain=cluster2.local
2020-11-02T14:14:57.3232808Z [lighthouse]$ [cluster2] local disable_cni=false
2020-11-02T14:14:57.3233399Z [lighthouse]$ [cluster2] disable_cni=true
2020-11-02T14:14:57.3233965Z [lighthouse]$ [cluster2] local nodes
2020-11-02T14:14:57.3234495Z [lighthouse]$ [cluster2] nodes=
2020-11-02T14:14:57.3235024Z [cluster2] - role: control-plane
2020-11-02T14:14:57.3235560Z [lighthouse]$ [cluster2] nodes=
2020-11-02T14:14:57.3236068Z [cluster2] - role: control-plane
2020-11-02T14:14:57.3236734Z [cluster2] - role: worker
2020-11-02T14:14:57.3237242Z [lighthouse]$ [cluster2] nodes=
2020-11-02T14:14:57.3237752Z [cluster2] - role: control-plane
2020-11-02T14:14:57.3238244Z [cluster2] - role: worker
2020-11-02T14:14:57.3238708Z [cluster2] - role: worker
2020-11-02T14:14:57.3239139Z [cluster2] 
2020-11-02T14:14:57.3239539Z [cluster2] 
2020-11-02T14:14:57.3242498Z [lighthouse]$ [cluster2] eval echo "---
2020-11-02T14:14:57.3242907Z [cluster2] kind: Cluster
2020-11-02T14:14:57.3243956Z [cluster2] apiVersion: kind.x-k8s.io/v1alpha4
2020-11-02T14:14:57.3244399Z [cluster2] networking:
2020-11-02T14:14:57.3244832Z [cluster2]   disableDefaultCNI: ${disable_cni}
2020-11-02T14:14:57.3245274Z [cluster2]   podSubnet: ${pod_cidr}
2020-11-02T14:14:57.3245685Z [cluster2]   serviceSubnet: ${service_cidr}
2020-11-02T14:14:57.3246204Z [cluster2] containerdConfigPatches:
2020-11-02T14:14:57.3246979Z [cluster2]   - |-
2020-11-02T14:14:57.3247667Z [cluster2]     [plugins.\"io.containerd.grpc.v1.cri\".registry.mirrors.\"localhost:5000\"]
2020-11-02T14:14:57.3248607Z [cluster2]       endpoint = [\"http://${registry_ip}:5000\"]
2020-11-02T14:14:57.3249304Z [cluster2] kubeadmConfigPatches:
2020-11-02T14:14:57.3250032Z [cluster2]   - |
2020-11-02T14:14:57.3250482Z [cluster2]     apiVersion: kubeadm.k8s.io/v1beta2
2020-11-02T14:14:57.3251025Z [cluster2]     kind: ClusterConfiguration
2020-11-02T14:14:57.3251654Z [cluster2]     metadata:
2020-11-02T14:14:57.3252010Z [cluster2]       name: config
2020-11-02T14:14:57.3252351Z [cluster2]     networking:
2020-11-02T14:14:57.3252742Z [cluster2]       podSubnet: ${pod_cidr}
2020-11-02T14:14:57.3258717Z [cluster2]       serviceSubnet: ${service_cidr}
2020-11-02T14:14:57.3259237Z [cluster2]       dnsDomain: ${dns_domain}
2020-11-02T14:14:57.3259911Z [cluster2] nodes: ${nodes}"
2020-11-02T14:14:57.3261215Z [lighthouse]$ [cluster2] cat /opt/shipyard/scripts/resources/kind-cluster-config.yaml
2020-11-02T14:14:57.3262174Z [lighthouse]$ [cluster2] local image_flag=
2020-11-02T14:14:57.3262977Z [lighthouse]$ [cluster2] image_flag=--image=kindest/node:v1.17.0
2020-11-02T14:14:57.3264994Z [lighthouse]$ [cluster2] kind create cluster --image=kindest/node:v1.17.0 --name=cluster2 --config=/opt/shipyard/scripts/resources/cluster2-config.yaml
2020-11-02T14:14:57.3266064Z [cluster2] Creating cluster "cluster2" ...
2020-11-02T14:14:57.3266963Z [cluster2]  • Ensuring node image (kindest/node:v1.17.0) 🖼  ...
2020-11-02T14:14:57.3267751Z [cluster2]  ✓ Ensuring node image (kindest/node:v1.17.0) 🖼
2020-11-02T14:14:57.3268508Z [cluster2]  • Preparing nodes 📦 📦 📦   ...
2020-11-02T14:14:57.3269072Z [cluster2]  ✓ Preparing nodes 📦 📦 📦
2020-11-02T14:14:57.3270059Z [cluster2]  • Writing configuration 📜  ...
2020-11-02T14:14:57.3270649Z [cluster2]  ✓ Writing configuration 📜
2020-11-02T14:14:57.3271628Z [cluster2]  • Starting control-plane 🕹️  ...
2020-11-02T14:14:57.3272813Z [cluster2]  ✓ Starting control-plane 🕹️
2020-11-02T14:14:57.3273468Z [cluster2]  • Installing StorageClass 💾  ...
2020-11-02T14:14:57.3274262Z [cluster2]  ✓ Installing StorageClass 💾
2020-11-02T14:14:57.3274878Z [cluster2]  • Joining worker nodes 🚜  ...
2020-11-02T14:14:57.3275449Z [cluster2]  ✓ Joining worker nodes 🚜
2020-11-02T14:14:57.3276095Z [cluster2] Set kubectl context to "kind-cluster2"
2020-11-02T14:14:57.3276601Z [cluster2] You can now use your cluster with:
2020-11-02T14:14:57.3276971Z [cluster2]
2020-11-02T14:14:57.3277621Z [cluster2] kubectl cluster-info --context kind-cluster2
2020-11-02T14:14:57.3278077Z [cluster2]
2020-11-02T14:14:57.3278582Z [cluster2] Thanks for using kind! 😊
2020-11-02T14:14:57.3279197Z [lighthouse]$ [cluster2] kind_fixup_config
2020-11-02T14:14:57.3280025Z [lighthouse]$ [cluster2] kind_fixup_config
2020-11-02T14:14:57.3280697Z [lighthouse]$ [cluster2] local master_ip=172.17.0.7
2020-11-02T14:14:57.3281957Z [lighthouse]$ [cluster2] docker inspect -f {{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}} cluster2-control-plane
2020-11-02T14:14:57.3282914Z [lighthouse]$ [cluster2] head -n 1
2020-11-02T14:15:34.0437344Z [lighthouse]$ [cluster2] sed -i -- s/server: . -n 1
2020-11-02T14:15:34.0439367Z [lighthouse]$ [cluster1] sed -i -- s/server: .*/server: https:\/\/172.17.0.4:6443/g /go/src/github.com/submariner-io/lighthouse/output/kubeconfigs/kind-config-cluster1
2020-11-02T14:15:34.0441821Z [lighthouse]$ [cluster1] sed -i -- s/user: kind-.*/user: cluster1/g /go/src/github.com/submariner-io/lighthouse/output/kubeconfigs/kind-config-cluster1
2020-11-02T14:15:34.0443900Z [lighthouse]$ [cluster1] sed -i -- s/name: kind-.*/name: cluster1/g /go/src/github.com/submariner-io/lighthouse/output/kubeconfigs/kind-config-cluster1
2020-11-02T14:15:34.0447762Z [lighthouse]$ [cluster1] sed -i -- s/cluster: kind-.*/cluster: cluster1/g /go/src/github.com/submariner-io/lighthouse/output/kubeconfigs/kind-config-cluster1
2020-11-02T14:15:34.0449650Z [lighthouse]$ [cluster1] sed -i -- s/current-context: .*/current-context: cluster1/g /go/src/github.com/submariner-io/lighthouse/output/kubeconfigs/kind-config-cluster1
2020-11-02T14:15:34.0451287Z [lighthouse]$ [cluster1] chmod a+r /go/src/github.com/submariner-io/lighthouse/output/kubeconfigs/kind-config-cluster1
2020-11-02T14:15:34.0452263Z [lighthouse]$ [cluster1] wait 4524
2020-11-02T14:15:34.0452954Z [lighthouse]$ [cluster1] deploy_cluster_capabilities
2020-11-02T14:15:34.0453723Z [lighthouse]$ [cluster1] deploy_cluster_capabilities
2020-11-02T14:15:34.0454562Z [lighthouse]$ [cluster1] deploy_cni
2020-11-02T14:15:34.0455125Z [lighthouse]$ [cluster1] deploy_cni
2020-11-02T14:15:34.0455721Z [lighthouse]$ [cluster1] eval deploy_weave_cni
2020-11-02T14:15:34.0456568Z [lighthouse]$ [cluster1] deploy_weave_cni
2020-11-02T14:15:34.0457166Z [lighthouse]$ [cluster1] deploy_weave_cni
2020-11-02T14:15:34.0457789Z [cluster1] Applying weave network...
2020-11-02T14:15:34.0459049Z [lighthouse]$ [cluster1] kubectl apply -f https://cloud.weave.works/k8s/net?k8s-version=v1.17.0&env.IPALLOC_RANGE=10.1.0.0/16
2020-11-02T14:15:34.0460551Z [lighthouse]$ [cluster1] kubectl apply -f https://cloud.weave.works/k8s/net?k8s-version=v1.17.0&env.IPALLOC_RANGE=10.1.0.0/16
2020-11-02T14:15:34.0462479Z [lighthouse]$ [cluster1] command kubectl --context=cluster1 apply -f https://cloud.weave.works/k8s/net?k8s-version=v1.17.0&env.IPALLOC_RANGE=10.1.0.0/16
2020-11-02T14:15:34.0463616Z [cluster1] serviceaccount/weave-net created
2020-11-02T14:15:34.0465138Z [cluster1] clusterrole.rbac.authorization.k8s.io/weave-net created
2020-11-02T14:15:34.0466487Z [cluster1] clusterrolebinding.rbac.authorization.k8s.io/weave-net created
2020-11-02T14:15:34.0467879Z [cluster1] role.rbac.authorization.k8s.io/weave-net created
2020-11-02T14:15:34.0469000Z [cluster1] rolebinding.rbac.authorization.k8s.io/weave-net created
2020-11-02T14:15:34.0469930Z [cluster1] daemonset.apps/weave-net created
2020-11-02T14:15:34.0470948Z [cluster1] Waiting for weave-net pods to be ready...
2020-11-02T14:15:34.0472203Z [lighthouse]$ [cluster1] kubectl wait --for=condition=Ready pods -l name=weave-net -n kube-system --timeout=5m
2020-11-02T14:15:34.0473346Z [lighthouse]$ [cluster1] kubectl wait --for=condition=Ready pods -l name=weave-net -n kube-system --timeout=5m
2020-11-02T14:15:34.0474914Z [lighthouse]$ [cluster1] command kubectl --context=cluster1 wait --for=condition=Ready pods -l name=weave-net -n kube-system --timeout=5m
2020-11-02T14:15:34.0476130Z [lighthouse]$ [cluster1] kubectl --context=cluster1 wait --for=condition=Ready pods -l name=weave-net -n kube-system --timeout=5m
2020-11-02T14:15:34.0477212Z [cluster1] pod/weave-net-5jpsf condition met
2020-11-02T14:15:34.0477896Z [cluster1] pod/weave-net-r4dz7 condition met
2020-11-02T14:15:34.0478902Z [cluster1] pod/weave-net-w85hf condition met
2020-11-02T14:15:34.0479621Z [cluster1] Waiting for core-dns deployment to be ready...
2020-11-02T14:15:34.0480499Z [lighthouse]$ [cluster1] kubectl -n kube-system rollout status deploy/coredns --timeout=5m
2020-11-02T14:15:34.0481618Z [lighthouse]$ [cluster1] kubectl -n kube-system rollout status deploy/coredns --timeout=5m
2020-11-02T14:15:34.0482669Z [lighthouse]$ [cluster1] command kubectl --context=cluster1 -n kube-system rollout status deploy/coredns --timeout=5m
2020-11-02T14:15:34.0483889Z [lighthouse]$ [cluster1] kubectl --context=cluster1 -n kube-system rollout status deploy/coredns --timeout=5m
2020-11-02T14:15:34.0484752Z [cluster1] Waiting for deployment "coredns" rollout to finish: 0 of 2 updated replicas are available...
2020-11-02T14:15:34.0485565Z [cluster1] Waiting for deployment "coredns" rollout to finish: 1 of 2 updated replicas are available...
2020-11-02T14:15:34.0486240Z [cluster1] deployment "coredns" successfully rolled out
2020-11-02T14:15:34.0486910Z [lighthouse]$ [cluster1] return 0
2020-11-02T14:15:42.2064752Z */server: https:\/\/172.17.0.7:6443/g /go/src/github.com/submariner-io/lighthouse/output/kubeconfigs/kind-config-cluster2
2020-11-02T14:15:42.2066577Z [lighthouse]$ [cluster2] sed -i -- s/user: kind-.*/user: cluster2/g /go/src/github.com/submariner-io/lighthouse/output/kubeconfigs/kind-config-cluster2
2020-11-02T14:15:42.2068228Z [lighthouse]$ [cluster2] sed -i -- s/name: kind-.*/name: cluster2/g /go/src/github.com/submariner-io/lighthouse/output/kubeconfigs/kind-config-cluster2
2020-11-02T14:15:42.2069875Z [lighthouse]$ [cluster2] sed -i -- s/cluster: kind-.*/cluster: cluster2/g /go/src/github.com/submariner-io/lighthouse/output/kubeconfigs/kind-config-cluster2
2020-11-02T14:15:42.2072261Z [lighthouse]$ [cluster2] sed -i -- s/current-context: .*/current-context: cluster2/g /go/src/github.com/submariner-io/lighthouse/output/kubeconfigs/kind-config-cluster2
2020-11-02T14:15:42.2074451Z [lighthouse]$ [cluster2] chmod a+r /go/src/github.com/submariner-io/lighthouse/output/kubeconfigs/kind-config-cluster2
2020-11-02T14:15:42.2075356Z [lighthouse]$ [cluster2] wait 4668
2020-11-02T14:15:42.2076175Z [lighthouse]$ [cluster2] deploy_cluster_capabilities
2020-11-02T14:15:42.2076924Z [lighthouse]$ [cluster2] deploy_cluster_capabilities
2020-11-02T14:15:42.2077575Z [lighthouse]$ [cluster2] deploy_cni
2020-11-02T14:15:42.2078136Z [lighthouse]$ [cluster2] deploy_cni
2020-11-02T14:15:42.2078946Z [lighthouse]$ [cluster2] eval deploy_weave_cni
2020-11-02T14:15:42.2079603Z [lighthouse]$ [cluster2] deploy_weave_cni
2020-11-02T14:15:42.2080238Z [lighthouse]$ [cluster2] deploy_weave_cni
2020-11-02T14:15:42.2080856Z [cluster2] Applying weave network...
2020-11-02T14:15:42.2082367Z [lighthouse]$ [cluster2] kubectl apply -f https://cloud.weave.works/k8s/net?k8s-version=v1.17.0&env.IPALLOC_RANGE=10.2.0.0/16
2020-11-02T14:15:42.2084522Z [lighthouse]$ [cluster2] kubectl apply -f https://cloud.weave.works/k8s/net?k8s-version=v1.17.0&env.IPALLOC_RANGE=10.2.0.0/16
2020-11-02T14:15:42.2086010Z [lighthouse]$ [cluster2] command kubectl --context=cluster2 apply -f https://cloud.weave.works/k8s/net?k8s-version=v1.17.0&env.IPALLOC_RANGE=10.2.0.0/16
2020-11-02T14:15:42.2087087Z [cluster2] serviceaccount/weave-net created
2020-11-02T14:15:42.2088740Z [cluster2] clusterrole.rbac.authorization.k8s.io/weave-net created
2020-11-02T14:15:42.2090361Z [cluster2] clusterrolebinding.rbac.authorization.k8s.io/weave-net created
2020-11-02T14:15:42.2091714Z [cluster2] role.rbac.authorization.k8s.io/weave-net created
2020-11-02T14:15:42.2093083Z [cluster2] rolebinding.rbac.authorization.k8s.io/weave-net created
2020-11-02T14:15:42.2094123Z [cluster2] daemonset.apps/weave-net created
2020-11-02T14:15:42.2094839Z [cluster2] Waiting for weave-net pods to be ready...
2020-11-02T14:15:42.2095937Z [lighthouse]$ [cluster2] kubectl wait --for=condition=Ready pods -l name=weave-net -n kube-system --timeout=5m
2020-11-02T14:15:42.2097385Z [lighthouse]$ [cluster2] kubectl wait --for=condition=Ready pods -l name=weave-net -n kube-system --timeout=5m
2020-11-02T14:15:42.2098635Z [lighthouse]$ [cluster2] command kubectl --context=cluster2 wait --for=condition=Ready pods -l name=weave-net -n kube-system --timeout=5m
2020-11-02T14:15:42.2100117Z [lighthouse]$ [cluster2] kubectl --context=cluster2 wait --for=condition=Ready pods -l name=weave-net -n kube-system --timeout=5m
2020-11-02T14:15:42.2101129Z [cluster2] pod/weave-net-8k8k4 condition met
2020-11-02T14:15:42.2101816Z [cluster2] pod/weave-net-wjrv2 condition met
2020-11-02T14:15:42.2103845Z [cluster2] pod/weave-net-xz7dq condition met
2020-11-02T14:15:42.2104719Z [cluster2] Waiting for core-dns deployment to be ready...
2020-11-02T14:15:42.2105574Z [lighthouse]$ [cluster2] kubectl -n kube-system rollout status deploy/coredns --timeout=5m
2020-11-02T14:15:42.2106701Z [lighthouse]$ [cluster2] kubectl -n kube-system rollout status deploy/coredns --timeout=5m
2020-11-02T14:15:42.2108199Z [lighthouse]$ [cluster2] command kubectl --context=cluster2 -n kube-system rollout status deploy/coredns --timeout=5m
2020-11-02T14:15:42.2109393Z [lighthouse]$ [cluster2] kubectl --context=cluster2 -n kube-system rollout status deploy/coredns --timeout=5m
2020-11-02T14:15:42.2110708Z [cluster2] Waiting for deployment "coredns" rollout to finish: 0 of 2 updated replicas are available...
2020-11-02T14:15:42.2111607Z [cluster2] Waiting for deployment "coredns" rollout to finish: 1 of 2 updated replicas are available...
2020-11-02T14:15:42.2112295Z [cluster2] deployment "coredns" successfully rolled out
2020-11-02T14:15:42.2112939Z [lighthouse]$ [cluster2] return 0
2020-11-02T14:15:42.2133337Z [lighthouse]$ [cluster2] wait 3423
2020-11-02T14:15:42.2170428Z [lighthouse]$ [cluster2] print_clusters_message
2020-11-02T14:15:42.2187022Z [lighthouse]$ [cluster2] print_clusters_message
2020-11-02T14:15:42.2233310Z [lighthouse]$ [cluster2] cat
2020-11-02T14:15:42.2255754Z Your virtual cluster(s) are deployed and working properly and can be accessed with:
2020-11-02T14:15:42.2256261Z
2020-11-02T14:15:42.2257185Z export KUBECONFIG=$(find $(git rev-parse --show-toplevel)/output/kubeconfigs/ -type f -printf %p:)
2020-11-02T14:15:42.2257664Z
2020-11-02T14:15:42.2258283Z $ kubectl config use-context cluster1 # or cluster2, cluster3..
2020-11-02T14:15:42.2258664Z
2020-11-02T14:15:42.2259021Z To clean everthing up, just run: make cleanup
2020-11-02T14:15:42.2270819Z ./scripts/deploy --deploytool helm --cluster_settings /go/src/github.com/submariner-io/lighthouse/scripts/cluster_settings --deploytool_broker_args '--set submariner.serviceDiscovery=true' --deploytool_submariner_args '--set submariner.serviceDiscovery=true,lighthouse.image.repository=localhost:5000/lighthouse-agent,lighthouse.image.tag=local,lighthouseCoredns.image.repository=localhost:5000/lighthouse-coredns,lighthouseCoredns.image.tag=local,serviceAccounts.lighthouse.create=true'
2020-11-02T14:15:42.2436491Z [lighthouse]$ source /opt/shipyard/scripts/lib/deploy_funcs
2020-11-02T14:15:42.2452516Z [lighthouse]$ . /opt/shipyard/scripts/lib/source_only
2020-11-02T14:15:42.2484903Z [lighthouse]$ script_name=deploy_funcs
2020-11-02T14:15:42.2500450Z [lighthouse]$ exec_name=deploy
2020-11-02T14:15:42.2553599Z [lighthouse]$ source /opt/shipyard/scripts/lib/utils
2020-11-02T14:15:42.2572752Z [lighthouse]$ . /opt/shipyard/scripts/lib/source_only
2020-11-02T14:15:42.2591222Z [lighthouse]$ script_name=utils
2020-11-02T14:15:42.2604523Z [lighthouse]$ exec_name=deploy
2020-11-02T14:15:42.2648634Z [lighthouse]$ source /opt/shipyard/scripts/lib/version
2020-11-02T14:15:42.2674068Z [lighthouse]$ . /opt/shipyard/scripts/lib/source_only
2020-11-02T14:15:42.2690865Z [lighthouse]$ script_name=version
2020-11-02T14:15:42.2716084Z [lighthouse]$ exec_name=deploy
2020-11-02T14:15:42.2739744Z [lighthouse]$ git describe --tags --dirty=-dev --exclude=devel --exclude=latest
2020-11-02T14:15:42.3307588Z fatal: No names found, cannot describe anything.
2020-11-02T14:15:42.3327435Z [lighthouse]$ declare_kubeconfig
2020-11-02T14:15:42.3335370Z [lighthouse]$ declare_kubeconfig
2020-11-02T14:15:42.3347389Z [lighthouse]$ source /opt/shipyard/scripts/lib/kubecfg
2020-11-02T14:15:42.3360197Z [lighthouse]$ export KUBECONFIG
2020-11-02T14:15:42.3407440Z [lighthouse]$ KUBECONFIG=/go/src/github.com/submariner-io/lighthouse/output/kubeconfigs/kind-config-cluster2:/go/src/github.com/submariner-io/lighthouse/output/kubeconfigs/kind-config-cluster1:
2020-11-02T14:15:42.3420468Z [lighthouse]$ find /go/src/github.com/submariner-io/lighthouse/output/kubeconfigs -type f -printf %p:
2020-11-02T14:15:42.3447313Z [lighthouse]$ import_image quay.io/submariner/lighthouse-agent
2020-11-02T14:15:42.3458319Z [lighthouse]$ import_image quay.io/submariner/lighthouse-agent
2020-11-02T14:15:42.3468263Z [lighthouse]$ local orig_image=quay.io/submariner/lighthouse-agent
2020-11-02T14:15:42.3479068Z [lighthouse]$ local versioned_image=quay.io/submariner/lighthouse-agent:dev
2020-11-02T14:15:42.3488443Z [lighthouse]$ local local_image=localhost:5000/lighthouse-agent:local
2020-11-02T14:15:42.3502082Z [lighthouse]$ docker tag quay.io/submariner/lighthouse-agent:dev localhost:5000/lighthouse-agent:local
2020-11-02T14:15:42.6858456Z [lighthouse]$ docker push localhost:5000/lighthouse-agent:local
2020-11-02T14:15:43.0345824Z The push refers to repository [localhost:5000/lighthouse-agent]
2020-11-02T14:15:43.0556168Z 3fe1858422d1: Preparing
2020-11-02T14:15:43.0556711Z 4e5767963082: Preparing
2020-11-02T14:15:43.0557248Z 3485805ce47c: Preparing
2020-11-02T14:15:43.0557568Z b0e2911c99f3: Preparing
2020-11-02T14:15:43.1407398Z 4e5767963082: Pushed
2020-11-02T14:15:43.1407724Z 3485805ce47c: Pushed
2020-11-02T14:15:43.6503547Z 3fe1858422d1: Pushed
2020-11-02T14:15:54.7060035Z [lighthouse]$ import_image quay.io/submariner/lighthouse-coredns
2020-11-02T14:15:54.7060759Z b0e2911c99f3: Pushed
2020-11-02T14:15:54.7061691Z [lighthouse]$ import_image quay.io/submariner/lighthouse-coredns
2020-11-02T14:15:54.7062559Z [lighthouse]$ local orig_image=quay.io/submariner/lighthouse-coredns
2020-11-02T14:15:54.7063492Z [lighthouse]$ local versioned_image=quay.io/submariner/lighthouse-coredns:dev
2020-11-02T14:15:54.7064408Z [lighthouse]$ local local_image=localhost:5000/lighthouse-coredns:local
2020-11-02T14:15:54.7065458Z [lighthouse]$ docker tag quay.io/submariner/lighthouse-coredns:dev localhost:5000/lighthouse-coredns:local
2020-11-02T14:15:54.7066486Z [lighthouse]$ docker push localhost:5000/lighthouse-coredns:local
2020-11-02T14:15:54.7068017Z local: digest: sha256:acdd2e7c54ba2f84a38696786441ade2a5a0950d987b9f0b948ef6556b88827c size: 1155
2020-11-02T14:15:54.8301208Z The push refers to repository [localhost:5000/lighthouse-coredns]
2020-11-02T14:15:54.8356746Z 24dd4ae37bb0: Preparing
2020-11-02T14:15:54.8357395Z a5db64596524: Preparing
2020-11-02T14:15:54.8357917Z 6b1fecdd2d1b: Preparing
2020-11-02T14:15:55.8714029Z 24dd4ae37bb0: Pushed
2020-11-02T14:15:57.4500265Z a5db64596524: Pushed
2020-11-02T14:16:01.5666986Z 6b1fecdd2d1b: Pushed
2020-11-02T14:16:01.5855579Z local: digest: sha256:e5c63ef292997ce6af66be1730ad3541a613e6653068635326e59302d215ba20 size: 953
2020-11-02T14:16:01.5915208Z [lighthouse]$ /opt/shipyard/scripts/deploy.sh --deploytool helm --cluster_settings /go/src/github.com/submariner-io/lighthouse/scripts/cluster_settings --deploytool_broker_args --set submariner.serviceDiscovery=true --deploytool_submariner_args --set submariner.serviceDiscovery=true,lighthouse.image.repository=localhost:5000/lighthouse-agent,lighthouse.image.tag=local,lighthouseCoredns.image.repository=localhost:5000/lighthouse-coredns,lighthouseCoredns.image.tag=local,serviceAccounts.lighthouse.create=true
2020-11-02T14:16:01.6924541Z Running with: globalnet='false', deploytool='helm', deploytool_broker_args='--set submariner.serviceDiscovery=true', deploytool_submariner_args='--set submariner.serviceDiscovery=true,lighthouse.image.repository=localhost:5000/lighthouse-agent,lighthouse.image.tag=local,lighthouseCoredns.image.repository=localhost:5000/lighthouse-coredns,lighthouseCoredns.image.tag=local,serviceAccounts.lighthouse.create=true', cluster_settings='/go/src/github.com/submariner-io/lighthouse/scripts/cluster_settings', timeout=5m
2020-11-02T14:16:01.6958649Z [lighthouse]$ source /opt/shipyard/scripts/lib/version
2020-11-02T14:16:01.6975022Z [lighthouse]$ . /opt/shipyard/scripts/lib/source_only
2020-11-02T14:16:01.6988184Z [lighthouse]$ script_name=version
2020-11-02T14:16:01.7002836Z [lighthouse]$ exec_name=deploy.sh
2020-11-02T14:16:01.7025414Z [lighthouse]$ git describe --tags --dirty=-dev --exclude=devel --exclude=latest
2020-11-02T14:16:01.7490012Z fatal: No names found, cannot describe anything.
2020-11-02T14:16:01.7528342Z [lighthouse]$ source /opt/shipyard/scripts/lib/utils
2020-11-02T14:16:01.7529122Z [lighthouse]$ . /opt/shipyard/scripts/lib/source_only
2020-11-02T14:16:01.7537543Z [lighthouse]$ script_name=utils
2020-11-02T14:16:01.7552672Z [lighthouse]$ exec_name=deploy.sh
2020-11-02T14:16:01.7575224Z [lighthouse]$ source /opt/shipyard/scripts/lib/deploy_funcs
2020-11-02T14:16:01.7585487Z [lighthouse]$ . /opt/shipyard/scripts/lib/source_only
2020-11-02T14:16:01.7601518Z [lighthouse]$ script_name=deploy_funcs
2020-11-02T14:16:01.7612521Z [lighthouse]$ exec_name=deploy.sh
2020-11-02T14:16:01.7637962Z [lighthouse]$ source /opt/shipyard/scripts/lib/cluster_settings
2020-11-02T14:16:01.7649138Z [lighthouse]$ . /opt/shipyard/scripts/lib/source_only
2020-11-02T14:16:01.7662878Z [lighthouse]$ script_name=cluster_settings
2020-11-02T14:16:01.7673462Z [lighthouse]$ exec_name=deploy.sh
2020-11-02T14:16:01.7697668Z [lighthouse]$ broker=cluster1
2020-11-02T14:16:01.7710673Z [lighthouse]$ declare -gA cluster_nodes
2020-11-02T14:16:01.7728183Z [lighthouse]$ cluster_nodes[cluster1]=control-plane worker
2020-11-02T14:16:01.7744879Z [lighthouse]$ cluster_nodes[cluster2]=control-plane worker
2020-11-02T14:16:01.7759839Z [lighthouse]$ cluster_nodes[cluster3]=control-plane worker worker
2020-11-02T14:16:01.7779613Z [lighthouse]$ declare -gA cluster_subm
2020-11-02T14:16:01.7794689Z [lighthouse]$ cluster_subm[cluster1]=true
2020-11-02T14:16:01.7810033Z [lighthouse]$ cluster_subm[cluster2]=true
2020-11-02T14:16:01.7823459Z [lighthouse]$ cluster_subm[cluster3]=true
2020-11-02T14:16:01.7839338Z [lighthouse]$ declare -gA cluster_cni
2020-11-02T14:16:01.7852086Z [lighthouse]$ cluster_cni[cluster2]=weave
2020-11-02T14:16:01.7862619Z [lighthouse]$ cluster_cni[cluster3]=weave
2020-11-02T14:16:01.7881211Z [lighthouse]$ source /go/src/github.com/submariner-io/lighthouse/scripts/cluster_settings
2020-11-02T14:16:01.7893886Z [lighthouse]$ . /opt/shipyard/scripts/lib/source_only
2020-11-02T14:16:01.7908160Z [lighthouse]$ script_name=cluster_settings
2020-11-02T14:16:01.7922816Z [lighthouse]$ exec_name=deploy.sh
2020-11-02T14:16:01.7960307Z [lighthouse]$ cluster_nodes[cluster1]=control-plane worker worker
2020-11-02T14:16:01.7961453Z [lighthouse]$ cluster_nodes[cluster2]=control-plane worker worker
2020-11-02T14:16:01.7992203Z [lighthouse]$ declare_cidrs
2020-11-02T14:16:01.8005349Z [lighthouse]$ declare_cidrs
2020-11-02T14:16:01.8018248Z [lighthouse]$ declare -gA cluster_CIDRs service_CIDRs global_CIDRs
2020-11-02T14:16:01.8034172Z [lighthouse]$ i=1
2020-11-02T14:16:01.8048413Z [lighthouse]$ [cluster1] add_cluster_cidrs 1 cluster1
2020-11-02T14:16:01.8062386Z [lighthouse]$ [cluster1] add_cluster_cidrs 1 cluster1
2020-11-02T14:16:01.8073431Z [lighthouse]$ [cluster1] local val=1
2020-11-02T14:16:01.8085281Z [lighthouse]$ [cluster1] local idx=cluster1
2020-11-02T14:16:01.8100209Z [lighthouse]$ [cluster1] cluster_CIDRs[cluster1]=10.1.0.0/16
2020-11-02T14:16:01.8113580Z [lighthouse]$ [cluster1] service_CIDRs[cluster1]=100.1.0.0/16
2020-11-02T14:16:01.8127923Z [lighthouse]$ [cluster1] i=2
2020-11-02T14:16:01.8142848Z [lighthouse]$ [cluster2] add_cluster_cidrs 2 cluster2
2020-11-02T14:16:01.8154715Z [lighthouse]$ [cluster2] add_cluster_cidrs 2 cluster2
2020-11-02T14:16:01.8166336Z [lighthouse]$ [cluster2] local val=2
2020-11-02T14:16:01.8189629Z [lighthouse]$ [cluster2] local idx=cluster2
2020-11-02T14:16:01.8201893Z [lighthouse]$ [cluster2] cluster_CIDRs[cluster2]=10.2.0.0/16
2020-11-02T14:16:01.8213358Z [lighthouse]$ [cluster2] service_CIDRs[cluster2]=100.2.0.0/16
2020-11-02T14:16:01.8227836Z [lighthouse]$ [cluster2] i=3
2020-11-02T14:16:01.8245492Z [lighthouse]$ [cluster2] declare_kubeconfig
2020-11-02T14:16:01.8258156Z [lighthouse]$ [cluster2] declare_kubeconfig
2020-11-02T14:16:01.8273174Z [lighthouse]$ [cluster2] source /opt/shipyard/scripts/lib/kubecfg
2020-11-02T14:16:01.8284492Z [lighthouse]$ [cluster2] export KUBECONFIG
2020-11-02T14:16:01.8329684Z [lighthouse]$ [cluster2] KUBECONFIG=/go/src/github.com/submariner-io/lighthouse/output/kubeconfigs/kind-config-cluster2:/go/src/github.com/submariner-io/lighthouse/output/kubeconfigs/kind-config-cluster1:
2020-11-02T14:16:01.8351100Z [lighthouse]$ [cluster2] find /go/src/github.com/submariner-io/lighthouse/output/kubeconfigs -type f -printf %p:
2020-11-02T14:16:01.8384038Z [lighthouse]$ [cluster2] import_image quay.io/submariner/nettest
2020-11-02T14:16:01.8401160Z [lighthouse]$ [cluster2] import_image quay.io/submariner/nettest
2020-11-02T14:16:01.8416060Z [lighthouse]$ [cluster2] local orig_image=quay.io/submariner/nettest
2020-11-02T14:16:01.8428841Z [lighthouse]$ [cluster2] local versioned_image=quay.io/submariner/nettest:dev
2020-11-02T14:16:01.8443472Z [lighthouse]$ [cluster2] local local_image=localhost:5000/nettest:local
2020-11-02T14:16:01.8456693Z [lighthouse]$ [cluster2] docker tag quay.io/submariner/nettest:dev localhost:5000/nettest:local
2020-11-02T14:16:02.1868801Z Error response from daemon: No such image: quay.io/submariner/nettest:dev
2020-11-02T14:16:02.1902650Z [lighthouse]$ [cluster2] docker pull quay.io/submariner/nettest:devel
2020-11-02T14:16:02.6894482Z devel: Pulling from submariner/nettest
2020-11-02T14:16:02.6894923Z 188c0c94c7c5: Pulling fs layer
2020-11-02T14:16:02.6895267Z 56cefa24c928: Pulling fs layer
2020-11-02T14:16:02.6895578Z 629d5148bcbc: Pulling fs layer
2020-11-02T14:16:02.6896228Z 630288c35ea1: Pulling fs layer
2020-11-02T14:16:02.6899097Z 630288c35ea1: Waiting
2020-11-02T14:16:02.7674874Z 56cefa24c928: Verifying Checksum
2020-11-02T14:16:02.7675298Z 56cefa24c928: Download complete
2020-11-02T14:16:02.8523611Z 188c0c94c7c5: Verifying Checksum
2020-11-02T14:16:02.8524083Z 188c0c94c7c5: Download complete
2020-11-02T14:16:02.8937582Z 629d5148bcbc: Verifying Checksum
2020-11-02T14:16:02.8938030Z 629d5148bcbc: Download complete
2020-11-02T14:16:02.8943162Z 630288c35ea1: Download complete
2020-11-02T14:16:03.3310099Z 188c0c94c7c5: Pull complete
2020-11-02T14:16:03.4005711Z 56cefa24c928: Pull complete
2020-11-02T14:16:03.7469873Z 629d5148bcbc: Pull complete
2020-11-02T14:16:03.8312689Z 630288c35ea1: Pull complete
2020-11-02T14:16:03.8349998Z Digest: sha256:897b8f7645e3942b17fdd1e4560c4e64e19b97020677912e39f075b68e5e943a
2020-11-02T14:16:03.8376362Z Status: Downloaded newer image for quay.io/submariner/nettest:devel
2020-11-02T14:16:03.8406518Z quay.io/submariner/nettest:devel
2020-11-02T14:16:03.8422608Z [lighthouse]$ [cluster2] docker tag quay.io/submariner/nettest:devel quay.io/submariner/nettest:dev
2020-11-02T14:16:04.1769249Z [lighthouse]$ [cluster2] docker tag quay.io/submariner/nettest:dev localhost:5000/nettest:local
2020-11-02T14:16:04.4983323Z [lighthouse]$ [cluster2] docker push localhost:5000/nettest:local
2020-11-02T14:16:04.8106543Z The push refers to repository [localhost:5000/nettest]
2020-11-02T14:16:04.8214068Z 38c1b9018ad3: Preparing
2020-11-02T14:16:04.8218158Z f0cf3a781ab4: Preparing
2020-11-02T14:16:04.8218617Z d26ed8773de6: Preparing
2020-11-02T14:16:04.8218983Z ace0eda3e3be: Preparing
2020-11-02T14:16:05.7877190Z 38c1b9018ad3: Pushed
2020-11-02T14:16:05.7878435Z d26ed8773de6: Pushed
2020-11-02T14:16:05.7987705Z ace0eda3e3be: Pushed
2020-11-02T14:16:06.5138302Z f0cf3a781ab4: Pushed
2020-11-02T14:16:06.5330277Z local: digest: sha256:897b8f7645e3942b17fdd1e4560c4e64e19b97020677912e39f075b68e5e943a size: 1155
2020-11-02T14:16:06.5390772Z [lighthouse]$ [cluster2] import_image quay.io/submariner/submariner
2020-11-02T14:16:06.5395398Z [lighthouse]$ [cluster2] import_image quay.io/submariner/submariner
2020-11-02T14:16:06.5401190Z [lighthouse]$ [cluster2] local orig_image=quay.io/submariner/submariner
2020-11-02T14:16:06.5415216Z [lighthouse]$ [cluster2] local versioned_image=quay.io/submariner/submariner:dev
2020-11-02T14:16:06.5432390Z [lighthouse]$ [cluster2] local local_image=localhost:5000/submariner:local
2020-11-02T14:16:06.5443003Z [lighthouse]$ [cluster2] docker tag quay.io/submariner/submariner:dev localhost:5000/submariner:local
2020-11-02T14:16:06.8637626Z Error response from daemon: No such image: quay.io/submariner/submariner:dev
2020-11-02T14:16:06.8676756Z [lighthouse]$ [cluster2] docker pull quay.io/submariner/submariner:devel
2020-11-02T14:16:07.3838431Z devel: Pulling from submariner/submariner
2020-11-02T14:16:07.3844111Z ee7e89337106: Already exists
2020-11-02T14:16:07.3878989Z 5a0589ba61c0: Pulling fs layer
2020-11-02T14:16:07.3883932Z 0cdb65106189: Pulling fs layer
2020-11-02T14:16:07.3888125Z cf3ac1366717: Pulling fs layer
2020-11-02T14:16:07.3893546Z cc9875344a58: Pulling fs layer
2020-11-02T14:16:07.3893910Z cc9875344a58: Waiting
2020-11-02T14:16:07.5152339Z 5a0589ba61c0: Download complete
2020-11-02T14:16:07.6385104Z 5a0589ba61c0: Pull complete
2020-11-02T14:16:07.6449426Z cc9875344a58: Verifying Checksum
2020-11-02T14:16:07.6449856Z cc9875344a58: Download complete
2020-11-02T14:16:07.6508318Z cf3ac1366717: Verifying Checksum
2020-11-02T14:16:07.6508796Z cf3ac1366717: Download complete
2020-11-02T14:16:07.9601085Z 0cdb65106189: Verifying Checksum
2020-11-02T14:16:07.9601611Z 0cdb65106189: Download complete
2020-11-02T14:16:08.9684203Z 0cdb65106189: Pull complete
2020-11-02T14:16:09.1482697Z cf3ac1366717: Pull complete
2020-11-02T14:16:09.2174927Z cc9875344a58: Pull complete
2020-11-02T14:16:09.2207526Z Digest: sha256:a09f015cb6ff6096ebbdfcbb65004dc02ead21ec444d6eed517b93a2599b549c
2020-11-02T14:16:09.2230527Z Status: Downloaded newer image for quay.io/submariner/submariner:devel
2020-11-02T14:16:09.2267205Z quay.io/submariner/submariner:devel
2020-11-02T14:16:09.2276931Z [lighthouse]$ [cluster2] docker tag quay.io/submariner/submariner:devel quay.io/submariner/submariner:dev
2020-11-02T14:16:09.5443155Z [lighthouse]$ [cluster2] docker tag quay.io/submariner/submariner:dev localhost:5000/submariner:local
2020-11-02T14:16:09.8668343Z [lighthouse]$ [cluster2] docker push localhost:5000/submariner:local
2020-11-02T14:16:10.2053137Z The push refers to repository [localhost:5000/submariner]
2020-11-02T14:16:10.2053787Z 214822d64306: Preparing
2020-11-02T14:16:10.2054143Z 265c78c63c02: Preparing
2020-11-02T14:16:10.2054709Z 33567521673b: Preparing
2020-11-02T14:16:10.2054999Z e66734129b2b: Preparing
2020-11-02T14:16:10.2055414Z f3c6dcc5e8bc: Preparing
2020-11-02T14:16:10.8534265Z e66734129b2b: Pushed
2020-11-02T14:16:10.8537863Z 265c78c63c02: Pushed
2020-11-02T14:16:12.0308203Z 214822d64306: Pushed
2020-11-02T14:16:15.0127390Z 33567521673b: Pushed
2020-11-02T14:16:25.0882638Z f3c6dcc5e8bc: Pushed
2020-11-02T14:16:25.1128230Z local: digest: sha256:a09f015cb6ff6096ebbdfcbb65004dc02ead21ec444d6eed517b93a2599b549c size: 1366
2020-11-02T14:16:25.1175952Z [lighthouse]$ [cluster2] import_image quay.io/submariner/submariner-route-agent
2020-11-02T14:16:25.1190417Z [lighthouse]$ [cluster2] import_image quay.io/submariner/submariner-route-agent
2020-11-02T14:16:25.1204308Z [lighthouse]$ [cluster2] local orig_image=quay.io/submariner/submariner-route-agent
2020-11-02T14:16:25.1227521Z [lighthouse]$ [cluster2] local versioned_image=quay.io/submariner/submariner-route-agent:dev
2020-11-02T14:16:25.1232003Z [lighthouse]$ [cluster2] local local_image=localhost:5000/submariner-route-agent:local
2020-11-02T14:16:25.1244259Z [lighthouse]$ [cluster2] docker tag quay.io/submariner/submariner-route-agent:dev localhost:5000/submariner-route-agent:local
2020-11-02T14:16:25.4490837Z Error response from daemon: No such image: quay.io/submariner/submariner-route-agent:dev
2020-11-02T14:16:25.4519489Z [lighthouse]$ [cluster2] docker pull quay.io/submariner/submariner-route-agent:devel
2020-11-02T14:16:25.9327281Z devel: Pulling from submariner/submariner-route-agent
2020-11-02T14:16:25.9328106Z 0fd3b5213a9b: Already exists
2020-11-02T14:16:25.9347218Z aebb8c556853: Already exists
2020-11-02T14:16:25.9386113Z d91bb8e8f11d: Pulling fs layer
2020-11-02T14:16:25.9386773Z fa68030320d0: Pulling fs layer
2020-11-02T14:16:25.9387205Z 4efd1d91b692: Pulling fs layer
2020-11-02T14:16:25.9387761Z 19f2d599f144: Pulling fs layer
2020-11-02T14:16:25.9388204Z 19f2d599f144: Waiting
2020-11-02T14:16:26.0638713Z d91bb8e8f11d: Verifying Checksum
2020-11-02T14:16:26.0639559Z d91bb8e8f11d: Download complete
2020-11-02T14:16:26.1426878Z d91bb8e8f11d: Pull complete
2020-11-02T14:16:26.2090794Z 19f2d599f144: Verifying Checksum
2020-11-02T14:16:26.2092368Z 19f2d599f144: Download complete
2020-11-02T14:16:26.2104130Z 4efd1d91b692: Verifying Checksum
2020-11-02T14:16:26.2104598Z 4efd1d91b692: Download complete
2020-11-02T14:16:26.3227649Z fa68030320d0: Verifying Checksum
2020-11-02T14:16:26.3228184Z fa68030320d0: Download complete
2020-11-02T14:16:26.5872648Z fa68030320d0: Pull complete
2020-11-02T14:16:26.7587975Z 4efd1d91b692: Pull complete
2020-11-02T14:16:26.8315945Z 19f2d599f144: Pull complete
2020-11-02T14:16:26.8347887Z Digest: sha256:5fa93aefa786fd7e7482781429016adfaabf7e088c956033935373e0e71e5b4a
2020-11-02T14:16:26.8385980Z Status: Downloaded newer image for quay.io/submariner/submariner-route-agent:devel
2020-11-02T14:16:26.8446056Z quay.io/submariner/submariner-route-agent:devel
2020-11-02T14:16:26.8506776Z [lighthouse]$ [cluster2] docker tag quay.io/submariner/submariner-route-agent:devel quay.io/submariner/submariner-route-agent:dev
2020-11-02T14:16:27.1857740Z [lighthouse]$ [cluster2] docker tag quay.io/submariner/submariner-route-agent:dev localhost:5000/submariner-route-agent:local
2020-11-02T14:16:27.5261171Z [lighthouse]$ [cluster2] docker push localhost:5000/submariner-route-agent:local
2020-11-02T14:16:27.8456162Z The push refers to repository [localhost:5000/submariner-route-agent]
2020-11-02T14:16:27.8522141Z d39a494dd3f1: Preparing
2020-11-02T14:16:27.8522508Z c12a8b02ebf4: Preparing
2020-11-02T14:16:27.8522936Z 47dc8ae8c4ab: Preparing
2020-11-02T14:16:27.8523321Z e9192cd260b1: Preparing
2020-11-02T14:16:27.8523712Z 3485805ce47c: Preparing
2020-11-02T14:16:27.8556820Z b0e2911c99f3: Preparing
2020-11-02T14:16:27.8557186Z b0e2911c99f3: Waiting
2020-11-02T14:16:28.4333659Z 3485805ce47c: Mounted from lighthouse-agent
2020-11-02T14:16:28.5380188Z c12a8b02ebf4: Pushed
2020-11-02T14:16:28.5380513Z e9192cd260b1: Pushed
2020-11-02T14:16:28.5380915Z d39a494dd3f1: Pushed
2020-11-02T14:16:28.5430470Z b0e2911c99f3: Mounted from lighthouse-agent
2020-11-02T14:16:28.7502234Z 47dc8ae8c4ab: Pushed
2020-11-02T14:16:28.7694858Z local: digest: sha256:5fa93aefa786fd7e7482781429016adfaabf7e088c956033935373e0e71e5b4a size: 1573
2020-11-02T14:16:28.7745938Z [lighthouse]$ [cluster2] load_deploytool helm
2020-11-02T14:16:28.7758415Z [lighthouse]$ [cluster2] load_deploytool helm
2020-11-02T14:16:28.7776167Z [lighthouse]$ [cluster2] local deploytool=helm
2020-11-02T14:16:28.7779141Z [lighthouse]$ [cluster2] local deploy_lib=/opt/shipyard/scripts/lib/deploy_helm
2020-11-02T14:16:28.7784865Z Will deploy submariner using helm
2020-11-02T14:16:28.7797568Z [lighthouse]$ [cluster2] . /opt/shipyard/scripts/lib/deploy_helm
2020-11-02T14:16:28.7807956Z [lighthouse]$ [cluster2] . /opt/shipyard/scripts/lib/source_only
2020-11-02T14:16:28.7819694Z [lighthouse]$ [cluster2] script_name=deploy_helm
2020-11-02T14:16:28.7835933Z [lighthouse]$ [cluster2] exec_name=deploy.sh
2020-11-02T14:16:28.7853713Z [lighthouse]$ [cluster2] LC_CTYPE=C tr -dc a-zA-Z0-9
2020-11-02T14:16:28.7869927Z [lighthouse]$ [cluster2] fold -w 64
2020-11-02T14:16:28.7888151Z [lighthouse]$ [cluster2] head -n 1
2020-11-02T14:16:28.7934584Z [lighthouse]$ [cluster2] deploytool_prereqs
2020-11-02T14:16:28.7948096Z [lighthouse]$ [cluster2] deploytool_prereqs
2020-11-02T14:16:28.7963400Z [lighthouse]$ [cluster2] helm init --client-only
2020-11-02T14:16:29.1061257Z Creating /root/.helm
2020-11-02T14:16:29.1061810Z Creating /root/.helm/repository
2020-11-02T14:16:29.1062351Z Creating /root/.helm/repository/cache
2020-11-02T14:16:29.1062853Z Creating /root/.helm/repository/local
2020-11-02T14:16:29.1063315Z Creating /root/.helm/plugins
2020-11-02T14:16:29.1063743Z Creating /root/.helm/starters
2020-11-02T14:16:29.1064369Z Creating /root/.helm/cache/archive
2020-11-02T14:16:29.1065120Z Creating /root/.helm/repository/repositories.yaml
2020-11-02T14:16:29.1066984Z Adding stable repo with URL: https://kubernetes-charts.storage.googleapis.com
2020-11-02T14:16:30.7517108Z Adding local repo with URL: http://127.0.0.1:8879/charts
2020-11-02T14:16:30.7525195Z $HELM_HOME has been configured at /root/.helm.
2020-11-02T14:16:30.7526457Z Not installing Tiller due to 'client-only' flag having been set
2020-11-02T14:16:30.7585160Z [lighthouse]$ [cluster2] helm repo add submariner-latest https://submariner-io.github.io/submariner-charts/charts
2020-11-02T14:16:31.3390538Z "submariner-latest" has been added to your repositories
2020-11-02T14:16:31.3417966Z [lighthouse]$ [cluster2] run_all_clusters install_helm
2020-11-02T14:16:31.3431901Z [lighthouse]$ [cluster2] run_all_clusters install_helm
2020-11-02T14:16:31.3452668Z [lighthouse]$ [cluster2] run_parallel cluster1 cluster2 install_helm
2020-11-02T14:16:31.3466087Z [lighthouse]$ [cluster2] run_parallel cluster1 cluster2 cluster1 cluster2 install_helm
2020-11-02T14:16:31.3480123Z [lighthouse]$ [cluster2] local cmnd=install_helm
2020-11-02T14:16:31.3491683Z [lighthouse]$ [cluster2] declare -A pids
2020-11-02T14:16:31.3510762Z [lighthouse]$ [cluster2] eval echo cluster1 cluster2
2020-11-02T14:16:31.3532642Z [lighthouse]$ [cluster1] pids[cluster1]=5104
2020-11-02T14:16:31.3537828Z [lighthouse]$ [cluster1] set -o pipefail
2020-11-02T14:16:31.3558107Z [lighthouse]$ [cluster2] pids[cluster2]=5107
2020-11-02T14:16:31.3564114Z [lighthouse]$ [cluster1] install_helm
2020-11-02T14:16:31.3576809Z [lighthouse]$ [cluster2] set -o pipefail
2020-11-02T14:16:31.3587286Z [lighthouse]$ [cluster2] wait 5107
2020-11-02T14:16:31.3601297Z [lighthouse]$ [cluster1] sed /\[cluster1]/!s/^/[cluster1] /
2020-11-02T14:16:31.3606692Z [lighthouse]$ [cluster2] install_helm
2020-11-02T14:16:31.3627530Z [lighthouse]$ [cluster2] sed /\[cluster2]/!s/^/[cluster2] /
2020-11-02T14:16:46.3220116Z [lighthouse]$ [cluster2] install_helm
2020-11-02T14:16:46.3222413Z [cluster2] 
2020-11-02T14:16:46.3222818Z [cluster2] Installing helm...
2020-11-02T14:16:46.3223740Z [lighthouse]$ [cluster2] kubectl -n kube-system create serviceaccount tiller
2020-11-02T14:16:46.3225104Z [lighthouse]$ [cluster2] kubectl -n kube-system create serviceaccount tiller
2020-11-02T14:16:46.3226266Z [lighthouse]$ [cluster2] command kubectl --context=cluster2 -n kube-system create serviceaccount tiller
2020-11-02T14:16:46.3227583Z [lighthouse]$ [cluster2] kubectl --context=cluster2 -n kube-system create serviceaccount tiller
2020-11-02T14:16:46.3228909Z [cluster2] serviceaccount/tiller created
2020-11-02T14:16:46.3230263Z [lighthouse]$ [cluster2] kubectl create clusterrolebinding tiller --clusterrole=cluster-admin --serviceaccount=kube-system:tiller
2020-11-02T14:16:46.3232269Z [lighthouse]$ [cluster2] kubectl create clusterrolebinding tiller --clusterrole=cluster-admin --serviceaccount=kube-system:tiller
2020-11-02T14:16:46.3234058Z [lighthouse]$ [cluster2] command kubectl --context=cluster2 create clusterrolebinding tiller --clusterrole=cluster-admin --serviceaccount=kube-system:tiller
2020-11-02T14:16:46.3236067Z [lighthouse]$ [cluster2] kubectl --context=cluster2 create clusterrolebinding tiller --clusterrole=cluster-admin --serviceaccount=kube-system:tiller
2020-11-02T14:16:46.3237808Z [cluster2] clusterrolebinding.rbac.authorization.k8s.io/tiller created
2020-11-02T14:16:46.3239100Z [lighthouse]$ [cluster2] helm --kube-context cluster2 init --service-account tiller
2020-11-02T14:16:46.3239796Z [cluster2] $HELM_HOME has been configured at /root/.helm.
2020-11-02T14:16:46.3240283Z [cluster2]
2020-11-02T14:16:46.3241083Z [cluster2] Tiller (the Helm server-side component) has been installed into your Kubernetes Cluster.
2020-11-02T14:16:46.3241654Z [cluster2]
2020-11-02T14:16:46.3242762Z [cluster2] Please note: by default, Tiller is deployed with an insecure 'allow unauthenticated users' policy.
2020-11-02T14:16:46.3244185Z [cluster2] To prevent this, run `helm init` with the --tiller-tls-verify flag.
2020-11-02T14:16:46.3245604Z [cluster2] For more information on securing your installation see: https://v2.helm.sh/docs/securing_installation/
2020-11-02T14:16:46.3246910Z [lighthouse]$ [cluster2] kubectl -n kube-system rollout status deploy/tiller-deploy --timeout=30s
2020-11-02T14:16:46.3248022Z [lighthouse]$ [cluster2] kubectl -n kube-system rollout status deploy/tiller-deploy --timeout=30s
2020-11-02T14:16:46.3249579Z [lighthouse]$ [cluster2] command kubectl --context=cluster2 -n kube-system rollout status deploy/tiller-deploy --timeout=30s
2020-11-02T14:16:46.3250812Z [lighthouse]$ [cluster2] kubectl --context=cluster2 -n kube-system rollout status deploy/tiller-deploy --timeout=30s
2020-11-02T14:16:46.3252290Z [cluster2] Waiting for deployment "tiller-deploy" rollout to finish: 0 of 1 updated replicas are available...
2020-11-02T14:16:46.3253195Z [cluster2] deployment "tiller-deploy" successfully rolled out
2020-11-02T14:16:46.3296586Z [lighthouse]$ [cluster2] wait 5104
2020-11-02T14:16:47.9233192Z [lighthouse]$ [cluster1] install_helm
2020-11-02T14:16:47.9233814Z [cluster1] 
2020-11-02T14:16:47.9234149Z [cluster1] Installing helm...
2020-11-02T14:16:47.9235041Z [lighthouse]$ [cluster1] kubectl -n kube-system create serviceaccount tiller
2020-11-02T14:16:47.9236039Z [lighthouse]$ [cluster1] kubectl -n kube-system create serviceaccount tiller
2020-11-02T14:16:47.9237322Z [lighthouse]$ [cluster1] command kubectl --context=cluster1 -n kube-system create serviceaccount tiller
2020-11-02T14:16:47.9238461Z [lighthouse]$ [cluster1] kubectl --context=cluster1 -n kube-system create serviceaccount tiller
2020-11-02T14:16:47.9239199Z [cluster1] serviceaccount/tiller created
2020-11-02T14:16:47.9240329Z [lighthouse]$ [cluster1] kubectl create clusterrolebinding tiller --clusterrole=cluster-admin --serviceaccount=kube-system:tiller
2020-11-02T14:16:47.9241823Z [lighthouse]$ [cluster1] kubectl create clusterrolebinding tiller --clusterrole=cluster-admin --serviceaccount=kube-system:tiller
2020-11-02T14:16:47.9243468Z [lighthouse]$ [cluster1] command kubectl --context=cluster1 create clusterrolebinding tiller --clusterrole=cluster-admin --serviceaccount=kube-system:tiller
2020-11-02T14:16:47.9245374Z [lighthouse]$ [cluster1] kubectl --context=cluster1 create clusterrolebinding tiller --clusterrole=cluster-admin --serviceaccount=kube-system:tiller
2020-11-02T14:16:47.9246988Z [cluster1] clusterrolebinding.rbac.authorization.k8s.io/tiller created
2020-11-02T14:16:47.9248324Z [lighthouse]$ [cluster1] helm --kube-context cluster1 init --service-account tiller
2020-11-02T14:16:47.9249050Z [cluster1] $HELM_HOME has been configured at /root/.helm.
2020-11-02T14:16:47.9249502Z [cluster1]
2020-11-02T14:16:47.9250738Z [cluster1] Tiller (the Helm server-side component) has been installed into your Kubernetes Cluster.
2020-11-02T14:16:47.9251337Z [cluster1]
2020-11-02T14:16:47.9252182Z [cluster1] Please note: by default, Tiller is deployed with an insecure 'allow unauthenticated users' policy.
2020-11-02T14:16:47.9253211Z [cluster1] To prevent this, run `helm init` with the --tiller-tls-verify flag.
2020-11-02T14:16:47.9254320Z [cluster1] For more information on securing your installation see: https://v2.helm.sh/docs/securing_installation/
2020-11-02T14:16:47.9255723Z [lighthouse]$ [cluster1] kubectl -n kube-system rollout status deploy/tiller-deploy --timeout=30s
2020-11-02T14:16:47.9256856Z [lighthouse]$ [cluster1] kubectl -n kube-system rollout status deploy/tiller-deploy --timeout=30s
2020-11-02T14:16:47.9258211Z [lighthouse]$ [cluster1] command kubectl --context=cluster1 -n kube-system rollout status deploy/tiller-deploy --timeout=30s
2020-11-02T14:16:47.9259688Z [lighthouse]$ [cluster1] kubectl --context=cluster1 -n kube-system rollout status deploy/tiller-deploy --timeout=30s
2020-11-02T14:16:47.9261070Z [cluster1] Waiting for deployment "tiller-deploy" rollout to finish: 0 of 1 updated replicas are available...
2020-11-02T14:16:47.9262546Z [cluster1] deployment "tiller-deploy" successfully rolled out
2020-11-02T14:16:47.9283746Z [lighthouse]$ [cluster2] run_subm_clusters prepare_cluster submariner-operator
2020-11-02T14:16:47.9298498Z [lighthouse]$ [cluster2] run_subm_clusters prepare_cluster submariner-operator
2020-11-02T14:16:47.9319210Z [lighthouse]$ [cluster2] declare -a subm_clusters
2020-11-02T14:16:47.9352584Z [lighthouse]$ [cluster2] run_parallel cluster1 cluster2 prepare_cluster submariner-operator
2020-11-02T14:16:47.9386650Z [lighthouse]$ [cluster2] run_parallel cluster1 cluster2 cluster1 cluster2 prepare_cluster submariner-operator
2020-11-02T14:16:47.9420697Z [lighthouse]$ [cluster2] local cmnd=prepare_cluster
2020-11-02T14:16:47.9457880Z [lighthouse]$ [cluster2] declare -A pids
2020-11-02T14:16:47.9481911Z [lighthouse]$ [cluster2] eval echo cluster1 cluster2
2020-11-02T14:16:47.9562271Z [lighthouse]$ [cluster1] pids[cluster1]=5221
2020-11-02T14:16:47.9628076Z [lighthouse]$ [cluster2] pids[cluster2]=5223
2020-11-02T14:16:47.9651210Z [lighthouse]$ [cluster1] set -o pipefail
2020-11-02T14:16:47.9667568Z [lighthouse]$ [cluster2] set -o pipefail
2020-11-02T14:16:47.9694138Z [lighthouse]$ [cluster2] wait 5223
2020-11-02T14:16:47.9699802Z [lighthouse]$ [cluster1] prepare_cluster submariner-operator
2020-11-02T14:16:47.9783056Z [lighthouse]$ [cluster2] prepare_cluster submariner-operator
2020-11-02T14:16:47.9878280Z [lighthouse]$ [cluster1] sed /\[cluster1]/!s/^/[cluster1] /
2020-11-02T14:16:48.0001714Z [lighthouse]$ [cluster2] sed /\[cluster2]/!s/^/[cluster2] /
2020-11-02T14:16:49.3775853Z [lighthouse]$ [cluster2] prepare_cluster
2020-11-02T14:16:49.3776786Z [lighthouse]$ [cluster2] local namespace=submariner-operator
2020-11-02T14:16:49.3777421Z [cluster2] 
2020-11-02T14:16:49.3777844Z [cluster2] 
2020-11-02T14:16:49.3778212Z [cluster2] 
2020-11-02T14:16:49.3778873Z [lighthouse]$ [cluster2] add_subm_gateway_label
2020-11-02T14:16:49.3779537Z [lighthouse]$ [cluster2] add_subm_gateway_label
2020-11-02T14:16:49.3780435Z [lighthouse]$ [cluster2] kubectl label node cluster2-worker submariner.io/gateway=true --overwrite
2020-11-02T14:16:49.3782172Z [lighthouse]$ [cluster2] kubectl label node cluster2-worker submariner.io/gateway=true --overwrite
2020-11-02T14:16:49.3783597Z [lighthouse]$ [cluster2] command kubectl --context=cluster2 label node cluster2-worker submariner.io/gateway=true --overwrite
2020-11-02T14:16:49.3785158Z [lighthouse]$ [cluster2] kubectl --context=cluster2 label node cluster2-worker submariner.io/gateway=true --overwrite
2020-11-02T14:16:49.3786132Z [cluster2] node/cluster2-worker labeled
2020-11-02T14:16:49.3799852Z [lighthouse]$ [cluster2] wait 5221
2020-11-02T14:16:49.4979256Z [lighthouse]$ [cluster1] prepare_cluster
2020-11-02T14:16:49.4980336Z [lighthouse]$ [cluster1] local namespace=submariner-operator
2020-11-02T14:16:49.4980976Z [cluster1] 
2020-11-02T14:16:49.4981537Z [cluster1] 
2020-11-02T14:16:49.4981976Z [cluster1] 
2020-11-02T14:16:49.4982554Z [lighthouse]$ [cluster1] add_subm_gateway_label
2020-11-02T14:16:49.4983228Z [lighthouse]$ [cluster1] add_subm_gateway_label
2020-11-02T14:16:49.4984180Z [lighthouse]$ [cluster1] kubectl label node cluster1-worker submariner.io/gateway=true --overwrite
2020-11-02T14:16:49.4985410Z [lighthouse]$ [cluster1] kubectl label node cluster1-worker submariner.io/gateway=true --overwrite
2020-11-02T14:16:49.4986683Z [lighthouse]$ [cluster1] command kubectl --context=cluster1 label node cluster1-worker submariner.io/gateway=true --overwrite
2020-11-02T14:16:49.4988234Z [lighthouse]$ [cluster1] kubectl --context=cluster1 label node cluster1-worker submariner.io/gateway=true --overwrite
2020-11-02T14:16:49.4989174Z [cluster1] node/cluster1-worker labeled
2020-11-02T14:16:49.5009278Z [lighthouse]$ [cluster2] with_context cluster1 setup_broker
2020-11-02T14:16:49.5021991Z [lighthouse]$ [cluster2] with_context cluster1 setup_broker
2020-11-02T14:16:49.5036388Z [lighthouse]$ [cluster2] local cluster=cluster1
2020-11-02T14:16:49.5048185Z [lighthouse]$ [cluster1] local cmnd=setup_broker
2020-11-02T14:16:49.5064461Z [lighthouse]$ [cluster1] setup_broker
2020-11-02T14:16:49.5076798Z [lighthouse]$ [cluster1] setup_broker
2020-11-02T14:16:49.5091522Z 
2020-11-02T14:16:49.7957822Z Installing submariner broker...
2020-11-02T14:16:49.7975216Z [lighthouse]$ [cluster1] helm install submariner-latest/submariner-k8s-broker --kube-context cluster1 --name submariner-k8s-broker --namespace submariner-k8s-broker --set submariner.serviceDiscovery=true
2020-11-02T14:16:56.8744222Z NAME:   submariner-k8s-broker
2020-11-02T14:16:56.9236077Z LAST DEPLOYED: Mon Nov  2 14:16:50 2020
2020-11-02T14:16:56.9237186Z NAMESPACE: submariner-k8s-broker
2020-11-02T14:16:56.9237672Z STATUS: DEPLOYED
2020-11-02T14:16:56.9237876Z
2020-11-02T14:16:56.9238552Z RESOURCES:
2020-11-02T14:16:56.9238834Z ==> v1/Role
2020-11-02T14:16:56.9239104Z NAME                          AGE
2020-11-02T14:16:56.9239715Z submariner-k8s-broker:client  0s
2020-11-02T14:16:56.9240007Z
2020-11-02T14:16:56.9240467Z ==> v1/RoleBinding
2020-11-02T14:16:56.9240819Z NAME                          AGE
2020-11-02T14:16:56.9241401Z submariner-k8s-broker:client  0s
2020-11-02T14:16:56.9242394Z
2020-11-02T14:16:56.9242770Z ==> v1/ServiceAccount
2020-11-02T14:16:56.9248955Z NAME                          SECRETS  AGE
2020-11-02T14:16:56.9250185Z submariner-k8s-broker-client  1        0s
2020-11-02T14:16:56.9488091Z
2020-11-02T14:16:56.9488248Z
2020-11-02T14:16:56.9488467Z NOTES:
2020-11-02T14:16:56.9488809Z The Submariner Kubernetes Broker is now setup.
2020-11-02T14:16:56.9489095Z
2020-11-02T14:16:56.9489374Z You can retrieve the server URL by running
2020-11-02T14:16:56.9489631Z
2020-11-02T14:16:56.9491179Z   $ SUBMARINER_BROKER_URL=$(kubectl -n default get endpoints kubernetes -o jsonpath="{.subsets[0].addresses[0].ip}:{.subsets[0].ports[?(@.name=='https')].port}")
2020-11-02T14:16:56.9491769Z
2020-11-02T14:16:56.9492138Z The broker client token and CA can be retrieved by running
2020-11-02T14:16:56.9492456Z
2020-11-02T14:16:56.9494166Z   $ SUBMARINER_BROKER_CA=$(kubectl -n submariner-k8s-broker get secrets -o jsonpath="{.items[?(@.metadata.annotations['kubernetes\.io/service-account\.name']=='submariner-k8s-broker-client')].data['ca\.crt']}")
2020-11-02T14:16:56.9496750Z   $ SUBMARINER_BROKER_TOKEN=$(kubectl -n submariner-k8s-broker get secrets -o jsonpath="{.items[?(@.metadata.annotations['kubernetes\.io/service-account\.name']=='submariner-k8s-broker-client')].data.token}"|base64 --decode)
2020-11-02T14:16:56.9497909Z
2020-11-02T14:16:57.2263202Z [lighthouse]$ [cluster1] submariner_broker_url=172.17.0.4:6443
2020-11-02T14:16:57.2283283Z [lighthouse]$ [cluster1] kubectl -n default get endpoints kubernetes -o jsonpath={.subsets[0].addresses[0].ip}:{.subsets[0].ports[?(@.name=='https')].port}
2020-11-02T14:16:57.2301446Z [lighthouse]$ [cluster1] kubectl -n default get endpoints kubernetes -o jsonpath={.subsets[0].addresses[0].ip}:{.subsets[0].ports[?(@.name=='https')].port}
2020-11-02T14:16:57.2329783Z [lighthouse]$ [cluster1] command kubectl --context=cluster1 -n default get endpoints kubernetes -o jsonpath={.subsets[0].addresses[0].ip}:{.subsets[0].ports[?(@.name=='https')].port}
2020-11-02T14:16:57.8727131Z [lighthouse]$ [cluster1] submariner_broker_ca=LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUN5RENDQWJDZ0F3SUJBZ0lCQURBTkJna3Foa2lHOXcwQkFRc0ZBREFWTVJNd0VRWURWUVFERXdwcmRXSmwKY201bGRHVnpNQjRYRFRJd01URXdNakUwTVRNeU5Wb1hEVE13TVRBek1URTBNVE15TlZvd0ZURVRNQkVHQTFVRQpBeE1LYTNWaVpYSnVaWFJsY3pDQ0FTSXdEUVlKS29aSWh2Y05BUUVCQlFBRGdnRVBBRENDQVFvQ2dnRUJBSmRFClVQQjFMRGNCRWIxcmlEZDlBM0F6Y1B6ejZMTklNSUUwbmFxTEZNUHhpamtXRWlkUUNXQVhtSy9tYzVtdE9FMEwKd2xlWkdrT3h5NHIwTHJoVnF5ZXF6Ty9kUlZ1UjB2Sm1yZHl4bU1lbHROOE5kMHdxNFNnRlZZVnNnZDZzL21USwpWU0R1Zll2UTIrR1NsT0UrdXVmeHAzeVgwSkJPRHFLbVJNNHFUSm00WGVoNEs2MTM4alVvclF0azBVYTY4ZDZOCmpwNEJDSFhQUXF3OVh1RlR5dFZxNHJEbkJTWEhCVkRhRitqVk9nOWljRXo0Q2RjdnF0NHNOWlEzSEZIU3lOVUYKc0xmMUlmbXVIdVBtdThVSW90dmRUY2RHQXNlV3F6ZStqSENCK2lFcjRNUE0zWCtVRlpiK0JIbkRYLzNiT1BQSAp6emlVRHlLek9YRENQUExuKzVNQ0F3RUFBYU1qTUNFd0RnWURWUjBQQVFIL0JBUURBZ0trTUE4R0ExVWRFd0VCCi93UUZNQU1CQWY4d0RRWUpLb1pJaHZjTkFRRUxCUUFEZ2dFQkFFMlVDSllRaGNuUTlhL1VKOHBhVkpjQjJFTngKWFYxNlBxUjVNNmtrNTNiYmVkSFpvOS9UYUlCTWNscCtHWEMrWVJHbjc0SC9qdTJ3MjJvdTkwT2kyYWNBNC8zOApPaGpmVWE3bEVidjdoRWEvVXJUV1BLaHNVdDV2KzVwNXNhcERYQUkvYzl3cENreEx0bUIza0tLY0daMGl1dDJOCmtVcnIzQ0huaXJnV2x4N0wvNjJUbTVaU0ROSlhuNHN5WW81VkVwVityd3ozYzhFdUdvazVCb2N4aFdocC9RREEKbU44S1MwYzR2OENJelhZL205MThWZjlKWkVhYytTOW4xbzVjb1MrTFQxRitSaXlVRm9iRXV3TkNiU0JaSUtoQgpXL2JWZzN5R29LZDVST1NJV2VnNW5WaVNLa0FaMG9FTFh0V1BybWtQbzh3VjNzN2tOVUVGNnR1dFU4UT0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=
2020-11-02T14:16:57.9246384Z [lighthouse]$ [cluster1] kubectl -n submariner-k8s-broker get secrets -o jsonpath={.items[?(@.metadata.annotations['kubernetes\.io/service-account\.name']=='submariner-k8s-broker-client')].data['ca\.crt']}
2020-11-02T14:16:57.9248739Z [lighthouse]$ [cluster1] kubectl -n submariner-k8s-broker get secrets -o jsonpath={.items[?(@.metadata.annotations['kubernetes\.io/service-account\.name']=='submariner-k8s-broker-client')].data['ca\.crt']}
2020-11-02T14:16:57.9250940Z [lighthouse]$ [cluster1] command kubectl --context=cluster1 -n submariner-k8s-broker get secrets -o jsonpath={.items[?(@.metadata.annotations['kubernetes\.io/service-account\.name']=='submariner-k8s-broker-client')].data['ca\.crt']}
2020-11-02T14:16:58.4532425Z [lighthouse]$ [cluster1] submariner_broker_token=eyJhbGciOiJSUzI1NiIsImtpZCI6ImlyU2tGeU5tbXVrVU9uWXV5TmlSVFJUcWN3R294d3hhQW9yVmhuSXRES0UifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJzdWJtYXJpbmVyLWs4cy1icm9rZXIiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlY3JldC5uYW1lIjoic3VibWFyaW5lci1rOHMtYnJva2VyLWNsaWVudC10b2tlbi13d205diIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJzdWJtYXJpbmVyLWs4cy1icm9rZXItY2xpZW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiMTg0NDg5OTYtNjNjMi00MTNmLTg3YzAtMDUzMDc3M2JjZjM4Iiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50OnN1Ym1hcmluZXItazhzLWJyb2tlcjpzdWJtYXJpbmVyLWs4cy1icm9rZXItY2xpZW50In0.nwAj_lgs9UdkEk8R9x80tM4kWOlR6trkevh0uqNqG-c0atQOsEQdSZRoZgiE6UnsJJrCq2rY5Intjp4xoZCf5Qe1X4Gv3h0HtNVOdICdk5n6X3mzzaS_-RVocNIkaZncjnx93c7K7vvuHmSFmUmmlHQguxWVOjkZOPgwkUVLNeACSRY9slPTqx9jgf7wLh3wfUNGxQm6T-sjtaJ698jOQ6aI3AZY2KltLXUIZt-VAh-SMlZcceUo-BAH3jPBEGjog3mSk7Lfyi22jueNehYkx1EF8xjNIl49qtF6wasB7NMRQwnXCfKsN-yp_qCSsTTNvXz9CAMb3glZkPpXPdh-hA
2020-11-02T14:16:58.4666212Z [lighthouse]$ [cluster1] kubectl -n submariner-k8s-broker get secrets -o jsonpath={.items[?(@.metadata.annotations['kubernetes\.io/service-account\.name']=='submariner-k8s-broker-client')].data.token}
2020-11-02T14:16:58.4668713Z [lighthouse]$ [cluster1] kubectl -n submariner-k8s-broker get secrets -o jsonpath={.items[?(@.metadata.annotations['kubernetes\.io/service-account\.name']=='submariner-k8s-broker-client')].data.token}
2020-11-02T14:16:58.4670356Z [lighthouse]$ [cluster1] base64 --decode
2020-11-02T14:16:58.4672171Z [lighthouse]$ [cluster1] command kubectl --context=cluster1 -n submariner-k8s-broker get secrets -o jsonpath={.items[?(@.metadata.annotations['kubernetes\.io/service-account\.name']=='submariner-k8s-broker-client')].data.token}
2020-11-02T14:16:58.7570759Z [lighthouse]$ [cluster2] install_subm_all_clusters
2020-11-02T14:16:58.7583660Z [lighthouse]$ [cluster2] install_subm_all_clusters
2020-11-02T14:16:58.7599522Z [lighthouse]$ [cluster2] run_subm_clusters helm_install_subm
2020-11-02T14:16:58.7614560Z [lighthouse]$ [cluster2] run_subm_clusters helm_install_subm
2020-11-02T14:16:58.7627428Z [lighthouse]$ [cluster2] declare -a subm_clusters
2020-11-02T14:16:58.7661584Z [lighthouse]$ [cluster2] run_parallel cluster1 cluster2 helm_install_subm
2020-11-02T14:16:58.7675623Z [lighthouse]$ [cluster2] run_parallel cluster1 cluster2 cluster1 cluster2 helm_install_subm
2020-11-02T14:16:58.7688344Z [lighthouse]$ [cluster2] local cmnd=helm_install_subm
2020-11-02T14:16:58.7702192Z [lighthouse]$ [cluster2] declare -A pids
2020-11-02T14:16:58.7722940Z [lighthouse]$ [cluster2] eval echo cluster1 cluster2
2020-11-02T14:16:58.7745573Z [lighthouse]$ [cluster1] pids[cluster1]=5412
2020-11-02T14:16:58.7758411Z [lighthouse]$ [cluster1] set -o pipefail
2020-11-02T14:16:58.7779380Z [lighthouse]$ [cluster2] pids[cluster2]=5415
2020-11-02T14:16:58.7786157Z [lighthouse]$ [cluster2] set -o pipefail
2020-11-02T14:16:58.7790789Z [lighthouse]$ [cluster1] helm_install_subm
2020-11-02T14:16:58.7828463Z [lighthouse]$ [cluster2] wait 5415
2020-11-02T14:16:58.7840507Z [lighthouse]$ [cluster1] sed /\[cluster1]/!s/^/[cluster1] /
2020-11-02T14:16:58.7861434Z [lighthouse]$ [cluster2] helm_install_subm
2020-11-02T14:16:58.7894961Z [lighthouse]$ [cluster2] sed /\[cluster2]/!s/^/[cluster2] /
2020-11-02T14:17:01.3411030Z [lighthouse]$ [cluster1] helm_install_subm
2020-11-02T14:17:01.3411818Z [lighthouse]$ [cluster1] local crd_create=false
2020-11-02T14:17:01.3412377Z [cluster1] 
2020-11-02T14:17:01.3412775Z [cluster1] Installing Submariner...
2020-11-02T14:17:01.4631099Z [lighthouse]$ [cluster1] helm --kube-context cluster1 install submariner-latest/submariner --name submariner --namespace submariner-operator --set ipsec.psk=jmDoylkTDloLDfmRpRiK12XeIFK8EgDIFHW1sTZjJYlzVIjBwO2ajdnVKEiT7U4E --set broker.server=172.17.0.4:6443 --set broker.token=eyJhbGciOiJSUzI1NiIsImtpZCI6ImlyU2tGeU5tbXVrVU9uWXV5TmlSVFJUcWN3R294d3hhQW9yVmhuSXRES0UifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJzdWJtYXJpbmVyLWs4cy1icm9rZXIiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlY3JldC5uYW1lIjoic3VibWFyaW5lci1rOHMtYnJva2VyLWNsaWVudC10b2tlbi13d205diIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJzdWJtYXJpbmVyLWs4cy1icm9rZXItY2xpZW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiMTg0NDg5OTYtNjNjMi00MTNmLTg3YzAtMDUzMDc3M2JjZjM4Iiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50OnN1Ym1hcmluZXItazhzLWJyb2tlcjpzdWJtYXJpbmVyLWs4cy1icm9rZXItY2xpZW50In0.nwAj_lgs9UdkEk8R9x80tM4kWOlR6trkevh0uqNqG-c0atQOsEQdSZRoZgiE6UnsJJrCq2rY5Intjp4xoZCf5Qe1X4Gv3h0HtNVOdICdk5n6X3mzzaS_-RVocNIkaZncjnx93c7K7vvuHmSFmUmmlHQguxWVOjkZOPgwkUVLNeACSRY9slPTqx9jgf7wLh3wfUNGxQm6T-sjtaJ698jOQ6aI3AZY2KltLXUIZt-VAh-SMlZcceUo-BAH3jPBEGjog3mSk7Lfyi22jueNehYkx1EF8xjNIl49qtF6wasB7NMRQwnXCfKsN-yp_qCSsTTNvXz9CAMb3glZkPpXPdh-hA --set broker.namespace=submariner-k8s-broker --set broker.ca=LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUN5RENDQWJDZ0F3SUJBZ0lCQURBTkJna3Foa2lHOXcwQkFRc0ZBREFWTVJNd0VRWURWUVFERXdwcmRXSmwKY201bGRHVnpNQjRYRFRJd01URXdNakUwTVRNeU5Wb1hEVE13TVRBek1URTBNVE15TlZvd0ZURVRNQkVHQTFVRQpBeE1LYTNWaVpYSnVaWFJsY3pDQ0FTSXdEUVlKS29aSWh2Y05BUUVCQlFBRGdnRVBBRENDQVFvQ2dnRUJBSmRFClVQQjFMRGNCRWIxcmlEZDlBM0F6Y1B6ejZMTklNSUUwbmFxTEZNUHhpamtXRWlkUUNXQVhtSy9tYzVtdE9FMEwKd2xlWkdrT3h5NHIwTHJoVnF5ZXF6Ty9kUlZ1UjB2Sm1yZHl4bU1lbHROOE5kMHdxNFNnRlZZVnNnZDZzL21USwpWU0R1Zll2UTIrR1NsT0UrdXVmeHAzeVgwSkJPRHFLbVJNNHFUSm00WGVoNEs2MTM4alVvclF0azBVYTY4ZDZOCmpwNEJDSFhQUXF3OVh1RlR5dFZxNHJEbkJTWEhCVkRhRitqVk9nOWljRXo0Q2RjdnF0NHNOWlEzSEZIU3lOVUYKc0xmMUlmbXVIdVBtdThVSW90dmRUY2RHQXNlV3F6ZStqSENCK2lFcjRNUE0zWCtVRlpiK0JIbkRYLzNiT1BQSAp6emlVRHlLek9YRENQUExuKzVNQ0F3RUFBYU1qTUNFd0RnWURWUjBQQVFIL0JBUURBZ0trTUE4R0ExVWRFd0VCCi93UUZNQU1CQWY4d0RRWUpLb1pJaHZjTkFRRUxCUUFEZ2dFQkFFMlVDSllRaGNuUTlhL1VKOHBhVkpjQjJFTngKWFYxNlBxUjVNNmtrNTNiYmVkSFpvOS9UYUlCTWNscCtHWEMrWVJHbjc0SC9qdTJ3MjJvdTkwT2kyYWNBNC8zOApPaGpmVWE3bEVidjdoRWEvVXJUV1BLaHNVdDV2KzVwNXNhcERYQUkvYzl3cENreEx0bUIza0tLY0daMGl1dDJOCmtVcnIzQ0huaXJnV2x4N0wvNjJUbTVaU0ROSlhuNHN5WW81VkVwVityd3ozYzhFdUdvazVCb2N4aFdocC9RREEKbU44S1MwYzR2OENJelhZL205MThWZjlKWkVhYytTOW4xbzVjb1MrTFQxRitSaXlVRm9iRXV3TkNiU0JaSUtoQgpXL2JWZzN5R29LZDVST1NJV2VnNW5WaVNLa0FaMG9FTFh0V1BybWtQbzh3VjNzN2tOVUVGNnR1dFU4UT0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo= --set submariner.clusterId=cluster1 --set submariner.clusterCidr=10.1.0.0/16 --set submariner.serviceCidr=100.1.0.0/16 --set submariner.globalCidr= --set serviceAccounts.globalnet.create=false --set submariner.natEnabled=false --set routeAgent.image.repository=localhost:5000/submariner-route-agent --set routeAgent.image.tag=local --set routeAgent.image.pullPolicy=IfNotPresent --set engine.image.repository=localhost:5000/submariner --set engine.image.tag=local --set engine.image.pullPolicy=IfNotPresent --set globalnet.image.repository=localhost:5000/submariner-globalnet --set globalnet.image.tag=local --set globalnet.image.pullPolicy=IfNotPresent --set crd.create=false --set submariner.serviceDiscovery=true,lighthouse.image.repository=localhost:5000/lighthouse-agent,lighthouse.image.tag=local,lighthouseCoredns.image.repository=localhost:5000/lighthouse-coredns,lighthouseCoredns.image.tag=local,serviceAccounts.lighthouse.create=true
2020-11-02T14:17:01.5885125Z [cluster1] NAME:   submariner
2020-11-02T14:17:01.5885645Z [cluster1] LAST DEPLOYED: Mon Nov  2 14:16:59 2020
2020-11-02T14:17:01.5886992Z [cluster1] NAMESPACE: submariner-operator
2020-11-02T14:17:01.5887496Z [cluster1] STATUS: DEPLOYED
2020-11-02T14:17:01.5887826Z [cluster1]
2020-11-02T14:17:01.5888146Z [cluster1] RESOURCES:
2020-11-02T14:17:01.5888516Z [cluster1] ==> v1/ClusterRole
2020-11-02T14:17:01.5889355Z [cluster1] NAME                   AGE
2020-11-02T14:17:01.5889745Z [cluster1] submariner:lighthouse  1s
2020-11-02T14:17:01.5890228Z [cluster1] submariner:routeagent  1s
2020-11-02T14:17:01.5890598Z [cluster1]
2020-11-02T14:17:01.5890991Z [cluster1] ==> v1/ClusterRoleBinding
2020-11-02T14:17:01.5891409Z [cluster1] NAME                   AGE
2020-11-02T14:17:01.5891800Z [cluster1] submariner:lighthouse  1s
2020-11-02T14:17:01.5892262Z [cluster1] submariner:routeagent  1s
2020-11-02T14:17:01.5892633Z [cluster1]
2020-11-02T14:17:01.5892933Z [cluster1] ==> v1/ConfigMap
2020-11-02T14:17:01.5893850Z [cluster1] NAME                           DATA  AGE
2020-11-02T14:17:01.5894732Z [cluster1] submariner-lighthouse-coredns  1     0s
2020-11-02T14:17:01.5895269Z [cluster1]
2020-11-02T14:17:01.5895869Z [cluster1] ==> v1/DaemonSet
2020-11-02T14:17:01.5896872Z [cluster1] NAME                   DESIRED  CURRENT  READY  UP-TO-DATE  AVAILABLE  NODE SELECTOR               AGE
2020-11-02T14:17:01.5898015Z [cluster1] submariner-gateway     1        1        0      1           0          submariner.io/gateway=true  0s
2020-11-02T14:17:01.5898963Z [cluster1] submariner-routeagent  2        2        0      2           0          <none>                      0s
2020-11-02T14:17:01.5899427Z [cluster1]
2020-11-02T14:17:01.5899732Z [cluster1] ==> v1/Deployment
2020-11-02T14:17:01.5900365Z [cluster1] NAME                           READY  UP-TO-DATE  AVAILABLE  AGE
2020-11-02T14:17:01.5901192Z [cluster1] submariner-lighthouse-agent    0/1    1           0          0s
2020-11-02T14:17:01.5902151Z [cluster1] submariner-lighthouse-coredns  0/2    2           0          0s
2020-11-02T14:17:01.5902688Z [cluster1]
2020-11-02T14:17:01.5903008Z [cluster1] ==> v1/Pod(related)
2020-11-02T14:17:01.5903449Z [cluster1] NAME                                            READY  STATUS             RESTARTS  AGE
2020-11-02T14:17:01.5904368Z [cluster1] submariner-gateway-xs2nq                        0/1    ContainerCreating  0         1s
2020-11-02T14:17:01.5905754Z [cluster1] submariner-lighthouse-agent-68c5565ff8-zhrpm    0/1    ContainerCreating  0         1s
2020-11-02T14:17:01.5936771Z [cluster1] submariner-lighthouse-coredns-7466b6679c-h8rnx  0/1    ContainerCreating  0         1s
2020-11-02T14:17:01.5939555Z [cluster1] submariner-lighthouse-coredns-7466b6679c-n8242  0/1    ContainerCreating  0         1s
2020-11-02T14:17:01.5963484Z [cluster1] submariner-routeagent-ctc55                     0/1    ContainerCreating  0         1s
2020-11-02T14:17:01.5964849Z [cluster1] submariner-routeagent-lfc7d                     0/1    ContainerCreating  0         1s
2020-11-02T14:17:01.5965733Z [cluster1]
2020-11-02T14:17:01.5966010Z [cluster1] ==> v1/Role
2020-11-02T14:17:01.5966311Z [cluster1] NAME                   AGE
2020-11-02T14:17:01.5966697Z [cluster1] submariner:routeagent  1s
2020-11-02T14:17:01.5967271Z [cluster1] submariner:engine      1s
2020-11-02T14:17:01.5967784Z [cluster1]
2020-11-02T14:17:01.5968111Z [cluster1] ==> v1/RoleBinding
2020-11-02T14:17:01.5968454Z [cluster1] NAME                   AGE
2020-11-02T14:17:01.5968837Z [cluster1] submariner:routeagent  1s
2020-11-02T14:17:01.5969269Z [cluster1] submariner:engine      1s
2020-11-02T14:17:01.5969604Z [cluster1]
2020-11-02T14:17:01.5969879Z [cluster1] ==> v1/Service
2020-11-02T14:17:01.5970574Z [cluster1] NAME                           TYPE       CLUSTER-IP    EXTERNAL-IP  PORT(S)  AGE
2020-11-02T14:17:01.5971710Z [cluster1] submariner-lighthouse-coredns  ClusterIP  100.1.80.194  <none>       53/UDP   0s
2020-11-02T14:17:01.5972482Z [cluster1]
2020-11-02T14:17:01.5972826Z [cluster1] ==> v1/ServiceAccount
2020-11-02T14:17:01.5973220Z [cluster1] NAME                   SECRETS  AGE
2020-11-02T14:17:01.5973805Z [cluster1] submariner-engine      1        0s
2020-11-02T14:17:01.5974479Z [cluster1] submariner-lighthouse  1        0s
2020-11-02T14:17:01.5975509Z [cluster1] submariner-routeagent  1        0s
2020-11-02T14:17:01.6028071Z [cluster1]
2020-11-02T14:17:01.6028587Z [cluster1]
2020-11-02T14:17:01.6028895Z [cluster1] NOTES:
2020-11-02T14:17:01.6029439Z [cluster1] Submariner is now installed.
2020-11-02T14:17:01.6030547Z [cluster1] If you haven't done so yet, please label a node as `submariner.io/gateway=true` to elect it for running Submariner.
2020-11-02T14:17:01.6031140Z [cluster1]
2020-11-02T14:17:01.6033013Z [cluster1] By default, Submariner runs with 1 replica. If you have more than one Gateway host, you can scale Submariner to N replicas, and the other Submariner pods will simply join the leader election pool.
2020-11-02T14:17:10.0466829Z [lighthouse]$ [cluster2] helm_install_subm
2020-11-02T14:17:10.0467531Z [lighthouse]$ [cluster2] local crd_create=false
2020-11-02T14:17:10.0468157Z [lighthouse]$ [cluster2] crd_create=true
2020-11-02T14:17:10.0468883Z [cluster2] 
2020-11-02T14:17:10.0469273Z [cluster2] Installing Submariner...
2020-11-02T14:17:10.1369681Z [lighthouse]$ [cluster2] helm --kube-context cluster2 install submariner-latest/submariner --name submariner --namespace submariner-operator --set ipsec.psk=jmDoylkTDloLDfmRpRiK12XeIFK8EgDIFHW1sTZjJYlzVIjBwO2ajdnVKEiT7U4E --set broker.server=172.17.0.4:6443 --set broker.token=eyJhbGciOiJSUzI1NiIsImtpZCI6ImlyU2tGeU5tbXVrVU9uWXV5TmlSVFJUcWN3R294d3hhQW9yVmhuSXRES0UifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJzdWJtYXJpbmVyLWs4cy1icm9rZXIiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlY3JldC5uYW1lIjoic3VibWFyaW5lci1rOHMtYnJva2VyLWNsaWVudC10b2tlbi13d205diIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJzdWJtYXJpbmVyLWs4cy1icm9rZXItY2xpZW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiMTg0NDg5OTYtNjNjMi00MTNmLTg3YzAtMDUzMDc3M2JjZjM4Iiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50OnN1Ym1hcmluZXItazhzLWJyb2tlcjpzdWJtYXJpbmVyLWs4cy1icm9rZXItY2xpZW50In0.nwAj_lgs9UdkEk8R9x80tM4kWOlR6trkevh0uqNqG-c0atQOsEQdSZRoZgiE6UnsJJrCq2rY5Intjp4xoZCf5Qe1X4Gv3h0HtNVOdICdk5n6X3mzzaS_-RVocNIkaZncjnx93c7K7vvuHmSFmUmmlHQguxWVOjkZOPgwkUVLNeACSRY9slPTqx9jgf7wLh3wfUNGxQm6T-sjtaJ698jOQ6aI3AZY2KltLXUIZt-VAh-SMlZcceUo-BAH3jPBEGjog3mSk7Lfyi22jueNehYkx1EF8xjNIl49qtF6wasB7NMRQwnXCfKsN-yp_qCSsTTNvXz9CAMb3glZkPpXPdh-hA --set broker.namespace=submariner-k8s-broker --set broker.ca=LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUN5RENDQWJDZ0F3SUJBZ0lCQURBTkJna3Foa2lHOXcwQkFRc0ZBREFWTVJNd0VRWURWUVFERXdwcmRXSmwKY201bGRHVnpNQjRYRFRJd01URXdNakUwTVRNeU5Wb1hEVE13TVRBek1URTBNVE15TlZvd0ZURVRNQkVHQTFVRQpBeE1LYTNWaVpYSnVaWFJsY3pDQ0FTSXdEUVlKS29aSWh2Y05BUUVCQlFBRGdnRVBBRENDQVFvQ2dnRUJBSmRFClVQQjFMRGNCRWIxcmlEZDlBM0F6Y1B6ejZMTklNSUUwbmFxTEZNUHhpamtXRWlkUUNXQVhtSy9tYzVtdE9FMEwKd2xlWkdrT3h5NHIwTHJoVnF5ZXF6Ty9kUlZ1UjB2Sm1yZHl4bU1lbHROOE5kMHdxNFNnRlZZVnNnZDZzL21USwpWU0R1Zll2UTIrR1NsT0UrdXVmeHAzeVgwSkJPRHFLbVJNNHFUSm00WGVoNEs2MTM4alVvclF0azBVYTY4ZDZOCmpwNEJDSFhQUXF3OVh1RlR5dFZxNHJEbkJTWEhCVkRhRitqVk9nOWljRXo0Q2RjdnF0NHNOWlEzSEZIU3lOVUYKc0xmMUlmbXVIdVBtdThVSW90dmRUY2RHQXNlV3F6ZStqSENCK2lFcjRNUE0zWCtVRlpiK0JIbkRYLzNiT1BQSAp6emlVRHlLek9YRENQUExuKzVNQ0F3RUFBYU1qTUNFd0RnWURWUjBQQVFIL0JBUURBZ0trTUE4R0ExVWRFd0VCCi93UUZNQU1CQWY4d0RRWUpLb1pJaHZjTkFRRUxCUUFEZ2dFQkFFMlVDSllRaGNuUTlhL1VKOHBhVkpjQjJFTngKWFYxNlBxUjVNNmtrNTNiYmVkSFpvOS9UYUlCTWNscCtHWEMrWVJHbjc0SC9qdTJ3MjJvdTkwT2kyYWNBNC8zOApPaGpmVWE3bEVidjdoRWEvVXJUV1BLaHNVdDV2KzVwNXNhcERYQUkvYzl3cENreEx0bUIza0tLY0daMGl1dDJOCmtVcnIzQ0huaXJnV2x4N0wvNjJUbTVaU0ROSlhuNHN5WW81VkVwVityd3ozYzhFdUdvazVCb2N4aFdocC9RREEKbU44S1MwYzR2OENJelhZL205MThWZjlKWkVhYytTOW4xbzVjb1MrTFQxRitSaXlVRm9iRXV3TkNiU0JaSUtoQgpXL2JWZzN5R29LZDVST1NJV2VnNW5WaVNLa0FaMG9FTFh0V1BybWtQbzh3VjNzN2tOVUVGNnR1dFU4UT0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo= --set submariner.clusterId=cluster2 --set submariner.clusterCidr=10.2.0.0/16 --set submariner.serviceCidr=100.2.0.0/16 --set submariner.globalCidr= --set serviceAccounts.globalnet.create=false --set submariner.natEnabled=false --set routeAgent.image.repository=localhost:5000/submariner-route-agent --set routeAgent.image.tag=local --set routeAgent.image.pullPolicy=IfNotPresent --set engine.image.repository=localhost:5000/submariner --set engine.image.tag=local --set engine.image.pullPolicy=IfNotPresent --set globalnet.image.repository=localhost:5000/submariner-globalnet --set globalnet.image.tag=local --set globalnet.image.pullPolicy=IfNotPresent --set crd.create=true --set submariner.serviceDiscovery=true,lighthouse.image.repository=localhost:5000/lighthouse-agent,lighthouse.image.tag=local,lighthouseCoredns.image.repository=localhost:5000/lighthouse-coredns,lighthouseCoredns.image.tag=local,serviceAccounts.lighthouse.create=true
2020-11-02T14:17:10.2015564Z [cluster2] NAME:   submariner
2020-11-02T14:17:10.2016123Z [cluster2] LAST DEPLOYED: Mon Nov  2 14:16:59 2020
2020-11-02T14:17:10.2017411Z [cluster2] NAMESPACE: submariner-operator
2020-11-02T14:17:10.2017904Z [cluster2] STATUS: DEPLOYED
2020-11-02T14:17:10.2018842Z [cluster2]
2020-11-02T14:17:10.2019303Z [cluster2] RESOURCES:
2020-11-02T14:17:10.2020149Z [lighthouse]$ [cluster2] wait 5412
2020-11-02T14:17:10.2021045Z [lighthouse]$ [cluster2] with_context cluster2 with_retries 30 verify_gw_status
2020-11-02T14:17:10.2022148Z [lighthouse]$ [cluster2] with_context cluster2 with_retries 30 verify_gw_status
2020-11-02T14:17:10.2022975Z [lighthouse]$ [cluster2] local cluster=cluster2
2020-11-02T14:17:10.2023679Z [lighthouse]$ [cluster2] local cmnd=with_retries
2020-11-02T14:17:10.2024406Z [lighthouse]$ [cluster2] with_retries 30 verify_gw_status
2020-11-02T14:17:10.2025156Z [lighthouse]$ [cluster2] with_retries
2020-11-02T14:17:10.2025785Z [lighthouse]$ [cluster2] local retries=30
2020-11-02T14:17:10.2026465Z [lighthouse]$ [cluster2] local cmnd=verify_gw_status
2020-11-02T14:17:10.2027127Z [lighthouse]$ [cluster2] iteration=1
2020-11-02T14:17:10.2027711Z [lighthouse]$ [cluster2] wait 5476
2020-11-02T14:17:10.2028323Z [lighthouse]$ [cluster2] verify_gw_status
2020-11-02T14:17:10.2028965Z [lighthouse]$ [cluster2] verify_gw_status
2020-11-02T14:17:10.2029608Z [lighthouse]$ [cluster2] sleep_duration=6
2020-11-02T14:17:10.2030299Z [lighthouse]$ [cluster2] subctl show connections
2020-11-02T14:17:10.2031937Z [lighthouse]$ [cluster2] grep the server could not find the requested resource
2020-11-02T14:17:10.2032561Z [cluster2] ==> v1/ClusterRole
2020-11-02T14:17:10.2032921Z [cluster2] NAME                   AGE
2020-11-02T14:17:10.2033304Z [cluster2] submariner:routeagent  2s
2020-11-02T14:17:10.2033768Z [cluster2] submariner:lighthouse  2s
2020-11-02T14:17:10.2034138Z [cluster2]
2020-11-02T14:17:10.2034531Z [cluster2] ==> v1/ClusterRoleBinding
2020-11-02T14:17:10.2034949Z [cluster2] NAME                   AGE
2020-11-02T14:17:10.2035344Z [cluster2] submariner:lighthouse  2s
2020-11-02T14:17:10.2035816Z [cluster2] submariner:routeagent  2s
2020-11-02T14:17:10.2036181Z [cluster2]
2020-11-02T14:17:10.2036478Z [cluster2] ==> v1/ConfigMap
2020-11-02T14:17:10.2036837Z [cluster2] NAME                           DATA  AGE
2020-11-02T14:17:10.2037651Z [cluster2] submariner-lighthouse-coredns  1     1s
2020-11-02T14:17:10.2038179Z [cluster2]
2020-11-02T14:17:10.2038499Z [cluster2] ==> v1/DaemonSet
2020-11-02T14:17:10.2039284Z [cluster2] NAME                   DESIRED  CURRENT  READY  UP-TO-DATE  AVAILABLE  NODE SELECTOR               AGE
2020-11-02T14:17:10.2040273Z [cluster2] submariner-gateway     1        1        0      1           0          submariner.io/gateway=true  1s
2020-11-02T14:17:10.2041214Z [cluster2] submariner-routeagent  2        2        0      2           0          <none>                      1s
2020-11-02T14:17:10.2041677Z [cluster2]
2020-11-02T14:17:10.2041982Z [cluster2] ==> v1/Deployment
2020-11-02T14:17:10.2042620Z [cluster2] NAME                           READY  UP-TO-DATE  AVAILABLE  AGE
2020-11-02T14:17:10.2043452Z [cluster2] submariner-lighthouse-agent    0/1    1           0          1s
2020-11-02T14:17:10.2044410Z [cluster2] submariner-lighthouse-coredns  0/2    2           0          1s
2020-11-02T14:17:10.2044946Z [cluster2]
2020-11-02T14:17:10.2045266Z [cluster2] ==> v1/Pod(related)
2020-11-02T14:17:10.2045866Z [cluster2] NAME                                            READY  STATUS             RESTARTS  AGE
2020-11-02T14:17:10.2046905Z [cluster2] submariner-gateway-98th8                        0/1    ContainerCreating  0         1s
2020-11-02T14:17:10.2048190Z [cluster2] submariner-lighthouse-agent-594fff74b7-kq8qj    0/1    ContainerCreating  0         1s
2020-11-02T14:17:10.2049830Z [cluster2] submariner-lighthouse-coredns-7466b6679c-x49tl  0/1    ContainerCreating  0         2s
2020-11-02T14:17:10.2051958Z [cluster2] submariner-lighthouse-coredns-7466b6679c-zzcjh  0/1    ContainerCreating  0         2s
2020-11-02T14:17:10.2053577Z [cluster2] submariner-routeagent-4w68n                     0/1    ContainerCreating  0         1s
2020-11-02T14:17:10.2056440Z [cluster2] submariner-routeagent-ct9kx                     0/1    ContainerCreating  0         1s
2020-11-02T14:17:10.2057222Z [cluster2]
2020-11-02T14:17:10.2057689Z [cluster2] ==> v1/Role
2020-11-02T14:17:10.2058009Z [cluster2] NAME                   AGE
2020-11-02T14:17:10.2058422Z [cluster2] submariner:routeagent  2s
2020-11-02T14:17:10.2058848Z [cluster2] submariner:engine      2s
2020-11-02T14:17:10.2059194Z [cluster2]
2020-11-02T14:17:10.2059525Z [cluster2] ==> v1/RoleBinding
2020-11-02T14:17:10.2059880Z [cluster2] NAME                   AGE
2020-11-02T14:17:10.2060276Z [cluster2] submariner:routeagent  2s
2020-11-02T14:17:10.2060716Z [cluster2] submariner:engine      2s
2020-11-02T14:17:10.2061061Z [cluster2]
2020-11-02T14:17:10.2061345Z [cluster2] ==> v1/Service
2020-11-02T14:17:10.2062036Z [cluster2] NAME                           TYPE       CLUSTER-IP    EXTERNAL-IP  PORT(S)  AGE
2020-11-02T14:17:10.2063068Z [cluster2] submariner-lighthouse-coredns  ClusterIP  100.2.249.85  <none>       53/UDP   1s
2020-11-02T14:17:10.2063676Z [cluster2]
2020-11-02T14:17:10.2064031Z [cluster2] ==> v1/ServiceAccount
2020-11-02T14:17:10.2064438Z [cluster2] NAME                   SECRETS  AGE
2020-11-02T14:17:10.2065041Z [cluster2] submariner-engine      1        1s
2020-11-02T14:17:10.2065943Z [cluster2] submariner-lighthouse  1        1s
2020-11-02T14:17:10.2066659Z [cluster2] submariner-routeagent  1        1s
2020-11-02T14:17:10.2067085Z [cluster2]
2020-11-02T14:17:10.2067357Z [cluster2]
2020-11-02T14:17:10.2067649Z [cluster2] NOTES:
2020-11-02T14:17:10.2068045Z [cluster2] Submariner is now installed.
2020-11-02T14:17:10.2069013Z [cluster2] If you haven't done so yet, please label a node as `submariner.io/gateway=true` to elect it for running Submariner.
2020-11-02T14:17:10.2069861Z [cluster2]
2020-11-02T14:17:10.2070910Z [cluster2] By default, Submariner runs with 1 replica. If you have more than one Gateway host, you can scale Submariner to N replicas, and the other Submariner pods will simply join the leader election pool.
2020-11-02T14:17:13.3155412Z error reading from submariner client: the server could not find the requested resource (get submariners.submariner.io submariner)
2020-11-02T14:17:13.3168390Z [lighthouse]$ [cluster2] return 0
2020-11-02T14:17:13.3192433Z [lighthouse]$ [cluster2] return 0
2020-11-02T14:17:13.3217136Z [lighthouse]$ [cluster2] with_context cluster2 connectivity_tests cluster1
2020-11-02T14:17:13.3258591Z [lighthouse]$ [cluster2] with_context cluster2 connectivity_tests cluster1
2020-11-02T14:17:13.3273945Z [lighthouse]$ [cluster2] local cluster=cluster2
2020-11-02T14:17:13.3513871Z [lighthouse]$ [cluster2] local cmnd=connectivity_tests
2020-11-02T14:17:13.3599358Z [lighthouse]$ [cluster2] connectivity_tests cluster1
2020-11-02T14:17:13.3616644Z [lighthouse]$ [cluster2] connectivity_tests
2020-11-02T14:17:13.3631563Z [lighthouse]$ [cluster2] target_cluster=cluster1
2020-11-02T14:17:13.3643675Z [lighthouse]$ [cluster2] deploy_resource /opt/shipyard/scripts/resources/netshoot.yaml
2020-11-02T14:17:13.3657254Z [lighthouse]$ [cluster2] deploy_resource /opt/shipyard/scripts/resources/netshoot.yaml
2020-11-02T14:17:13.3699320Z [lighthouse]$ [cluster2] local resource_file=/opt/shipyard/scripts/resources/netshoot.yaml
2020-11-02T14:17:13.3728059Z [lighthouse]$ [cluster2] local resource_name
2020-11-02T14:17:13.3843082Z [lighthouse]$ [cluster2] resource_name=netshoot
2020-11-02T14:17:13.3875312Z [lighthouse]$ [cluster2] basename /opt/shipyard/scripts/resources/netshoot.yaml .yaml
2020-11-02T14:17:13.4038817Z [lighthouse]$ [cluster2] kubectl apply -f /opt/shipyard/scripts/resources/netshoot.yaml
2020-11-02T14:17:13.4124063Z [lighthouse]$ [cluster2] kubectl apply -f /opt/shipyard/scripts/resources/netshoot.yaml
2020-11-02T14:17:13.4150843Z [lighthouse]$ [cluster2] command kubectl --context=cluster2 apply -f /opt/shipyard/scripts/resources/netshoot.yaml
2020-11-02T14:17:13.4230746Z [lighthouse]$ [cluster2] kubectl --context=cluster2 apply -f /opt/shipyard/scripts/resources/netshoot.yaml
2020-11-02T14:17:15.8907991Z deployment.apps/netshoot created
2020-11-02T14:17:15.8955175Z Waiting for netshoot pods to be ready.
2020-11-02T14:17:15.8973392Z [lighthouse]$ [cluster2] kubectl rollout status deploy/netshoot --timeout=5m
2020-11-02T14:17:15.8986023Z [lighthouse]$ [cluster2] kubectl rollout status deploy/netshoot --timeout=5m
2020-11-02T14:17:15.8999986Z [lighthouse]$ [cluster2] command kubectl --context=cluster2 rollout status deploy/netshoot --timeout=5m
2020-11-02T14:17:15.9011058Z [lighthouse]$ [cluster2] kubectl --context=cluster2 rollout status deploy/netshoot --timeout=5m
2020-11-02T14:17:16.5197134Z Waiting for deployment "netshoot" rollout to finish: 0 out of 1 new replicas have been updated...
2020-11-02T14:17:17.8382830Z Waiting for deployment "netshoot" rollout to finish: 0 of 1 updated replicas are available...
2020-11-02T14:17:53.4544804Z deployment "netshoot" successfully rolled out
2020-11-02T14:17:53.4587560Z [lighthouse]$ [cluster2] with_context cluster1 deploy_resource /opt/shipyard/scripts/resources/nginx-demo.yaml
2020-11-02T14:17:53.4601759Z [lighthouse]$ [cluster2] with_context cluster1 deploy_resource /opt/shipyard/scripts/resources/nginx-demo.yaml
2020-11-02T14:17:53.4614529Z [lighthouse]$ [cluster2] local cluster=cluster1
2020-11-02T14:17:53.4629378Z [lighthouse]$ [cluster1] local cmnd=deploy_resource
2020-11-02T14:17:53.4643899Z [lighthouse]$ [cluster1] deploy_resource /opt/shipyard/scripts/resources/nginx-demo.yaml
2020-11-02T14:17:53.4656925Z [lighthouse]$ [cluster1] deploy_resource
2020-11-02T14:17:53.4670092Z [lighthouse]$ [cluster1] local resource_file=/opt/shipyard/scripts/resources/nginx-demo.yaml
2020-11-02T14:17:53.4686650Z [lighthouse]$ [cluster1] local resource_name
2020-11-02T14:17:53.4722026Z [lighthouse]$ [cluster1] resource_name=nginx-demo
2020-11-02T14:17:53.4741601Z [lighthouse]$ [cluster1] basename /opt/shipyard/scripts/resources/nginx-demo.yaml .yaml
2020-11-02T14:17:53.4768265Z [lighthouse]$ [cluster1] kubectl apply -f /opt/shipyard/scripts/resources/nginx-demo.yaml
2020-11-02T14:17:53.4787073Z [lighthouse]$ [cluster1] kubectl apply -f /opt/shipyard/scripts/resources/nginx-demo.yaml
2020-11-02T14:17:53.4800033Z [lighthouse]$ [cluster1] command kubectl --context=cluster1 apply -f /opt/shipyard/scripts/resources/nginx-demo.yaml
2020-11-02T14:17:53.4820642Z [lighthouse]$ [cluster1] kubectl --context=cluster1 apply -f /opt/shipyard/scripts/resources/nginx-demo.yaml
2020-11-02T14:17:53.9644136Z deployment.apps/nginx-demo created
2020-11-02T14:17:53.9816895Z service/nginx-demo created
2020-11-02T14:17:53.9864490Z Waiting for nginx-demo pods to be ready.
2020-11-02T14:17:53.9884859Z [lighthouse]$ [cluster1] kubectl rollout status deploy/nginx-demo --timeout=5m
2020-11-02T14:17:53.9941434Z [lighthouse]$ [cluster1] kubectl rollout status deploy/nginx-demo --timeout=5m
2020-11-02T14:17:53.9998928Z [lighthouse]$ [cluster1] command kubectl --context=cluster1 rollout status deploy/nginx-demo --timeout=5m
2020-11-02T14:17:54.0024047Z [lighthouse]$ [cluster1] kubectl --context=cluster1 rollout status deploy/nginx-demo --timeout=5m
2020-11-02T14:17:54.5843524Z Waiting for deployment "nginx-demo" rollout to finish: 0 of 2 updated replicas are available...
2020-11-02T14:17:57.2673674Z Waiting for deployment "nginx-demo" rollout to finish: 1 of 2 updated replicas are available...
2020-11-02T14:17:57.7458563Z deployment "nginx-demo" successfully rolled out
2020-11-02T14:17:57.7505573Z [lighthouse]$ [cluster2] local netshoot_pod nginx_svc_ip
2020-11-02T14:17:58.1635163Z [lighthouse]$ [cluster2] netshoot_pod=netshoot-789f6cf54f-p49hk
2020-11-02T14:17:58.1655739Z [lighthouse]$ [cluster2] kubectl get pods -l app=netshoot
2020-11-02T14:17:58.1678192Z [lighthouse]$ [cluster2] awk FNR == 2 {print $1}
2020-11-02T14:17:58.1678954Z [lighthouse]$ [cluster2] kubectl get pods -l app=netshoot
2020-11-02T14:17:58.1691485Z [lighthouse]$ [cluster2] command kubectl --context=cluster2 get pods -l app=netshoot
2020-11-02T14:17:58.1705971Z [lighthouse]$ [cluster2] kubectl --context=cluster2 get pods -l app=netshoot
2020-11-02T14:17:58.7714750Z [lighthouse]$ [cluster2] nginx_svc_ip=100.1.129.145
2020-11-02T14:17:58.7736988Z [lighthouse]$ [cluster2] with_context cluster1 get_svc_ip nginx-demo
2020-11-02T14:17:58.7750071Z [lighthouse]$ [cluster2] with_context cluster1 get_svc_ip nginx-demo
2020-11-02T14:17:58.7763924Z [lighthouse]$ [cluster2] local cluster=cluster1
2020-11-02T14:17:58.7775602Z [lighthouse]$ [cluster1] local cmnd=get_svc_ip
2020-11-02T14:17:58.7787811Z [lighthouse]$ [cluster1] get_svc_ip nginx-demo
2020-11-02T14:17:58.7800076Z [lighthouse]$ [cluster1] get_svc_ip
2020-11-02T14:17:58.7811437Z [lighthouse]$ [cluster1] local svc_name=nginx-demo
2020-11-02T14:17:58.7825688Z [lighthouse]$ [cluster1] local svc_ip
2020-11-02T14:17:59.0790708Z [lighthouse]$ [cluster1] svc_ip=100.1.129.145
2020-11-02T14:17:59.0814797Z [lighthouse]$ [cluster1] kubectl --context=cluster1 get svc -l app=nginx-demo
2020-11-02T14:17:59.0833513Z [lighthouse]$ [cluster1] kubectl --context=cluster1 get svc -l app=nginx-demo
2020-11-02T14:17:59.0835040Z [lighthouse]$ [cluster1] awk FNR == 2 {print $3}
2020-11-02T14:17:59.0850726Z [lighthouse]$ [cluster1] command kubectl --context=cluster1 get svc -l app=nginx-demo
2020-11-02T14:17:59.0889842Z [lighthouse]$ [cluster1] kubectl --context=cluster1 get svc -l app=nginx-demo
2020-11-02T14:17:59.3813233Z [lighthouse]$ [cluster2] with_retries 5 test_connection netshoot-789f6cf54f-p49hk 100.1.129.145
2020-11-02T14:17:59.3830127Z [lighthouse]$ [cluster2] with_retries 5 test_connection netshoot-789f6cf54f-p49hk 100.1.129.145
2020-11-02T14:17:59.3841287Z [lighthouse]$ [cluster2] local retries=5
2020-11-02T14:17:59.3854533Z [lighthouse]$ [cluster2] local cmnd=test_connection
2020-11-02T14:17:59.3865579Z [lighthouse]$ [cluster2] iteration=1
2020-11-02T14:17:59.3892046Z [lighthouse]$ [cluster2] wait 5636
2020-11-02T14:17:59.3898310Z [lighthouse]$ [cluster2] test_connection netshoot-789f6cf54f-p49hk 100.1.129.145
2020-11-02T14:17:59.3913173Z [lighthouse]$ [cluster2] test_connection
2020-11-02T14:17:59.3923799Z [lighthouse]$ [cluster2] local source_pod=netshoot-789f6cf54f-p49hk
2020-11-02T14:17:59.3937299Z [lighthouse]$ [cluster2] local target_address=100.1.129.145
2020-11-02T14:17:59.3943597Z Attempting connectivity between clusters - netshoot-789f6cf54f-p49hk --> 100.1.129.145
2020-11-02T14:17:59.3954146Z [lighthouse]$ [cluster2] wait_time=30
2020-11-02T14:17:59.3998143Z [lighthouse]$ [cluster2] start_time=1604326679
2020-11-02T14:17:59.4051291Z [lighthouse]$ [cluster2] date +%s
2020-11-02T14:17:59.4075507Z [lighthouse]$ [cluster2] kubectl exec netshoot-789f6cf54f-p49hk -- curl --output /dev/null -m 30 --silent --head --fail 100.1.129.145
2020-11-02T14:17:59.4088167Z [lighthouse]$ [cluster2] kubectl exec netshoot-789f6cf54f-p49hk -- curl --output /dev/null -m 30 --silent --head --fail 100.1.129.145
2020-11-02T14:17:59.4100529Z [lighthouse]$ [cluster2] command kubectl --context=cluster2 exec netshoot-789f6cf54f-p49hk -- curl --output /dev/null -m 30 --silent --head --fail 100.1.129.145
2020-11-02T14:17:59.4115547Z [lighthouse]$ [cluster2] kubectl --context=cluster2 exec netshoot-789f6cf54f-p49hk -- curl --output /dev/null -m 30 --silent --head --fail 100.1.129.145
2020-11-02T14:17:59.7865964Z Connection test was successful!
2020-11-02T14:17:59.7885292Z [lighthouse]$ [cluster2] return 0
2020-11-02T14:17:59.7900112Z [lighthouse]$ [cluster2] remove_resource /opt/shipyard/scripts/resources/netshoot.yaml
2020-11-02T14:17:59.7913779Z [lighthouse]$ [cluster2] remove_resource /opt/shipyard/scripts/resources/netshoot.yaml
2020-11-02T14:17:59.7926288Z [lighthouse]$ [cluster2] local resource_file=/opt/shipyard/scripts/resources/netshoot.yaml
2020-11-02T14:17:59.7942197Z [lighthouse]$ [cluster2] kubectl delete -f /opt/shipyard/scripts/resources/netshoot.yaml
2020-11-02T14:17:59.7956347Z [lighthouse]$ [cluster2] kubectl delete -f /opt/shipyard/scripts/resources/netshoot.yaml
2020-11-02T14:17:59.7971633Z [lighthouse]$ [cluster2] command kubectl --context=cluster2 delete -f /opt/shipyard/scripts/resources/netshoot.yaml
2020-11-02T14:17:59.7985210Z [lighthouse]$ [cluster2] kubectl --context=cluster2 delete -f /opt/shipyard/scripts/resources/netshoot.yaml
2020-11-02T14:18:00.0943484Z deployment.apps "netshoot" deleted
2020-11-02T14:18:00.1012374Z [lighthouse]$ [cluster2] with_context cluster1 remove_resource /opt/shipyard/scripts/resources/nginx-demo.yaml
2020-11-02T14:18:00.1040015Z [lighthouse]$ [cluster2] with_context cluster1 remove_resource /opt/shipyard/scripts/resources/nginx-demo.yaml
2020-11-02T14:18:00.1056147Z [lighthouse]$ [cluster2] local cluster=cluster1
2020-11-02T14:18:00.1075357Z [lighthouse]$ [cluster1] local cmnd=remove_resource
2020-11-02T14:18:00.1091950Z [lighthouse]$ [cluster1] remove_resource /opt/shipyard/scripts/resources/nginx-demo.yaml
2020-11-02T14:18:00.1103343Z [lighthouse]$ [cluster1] remove_resource
2020-11-02T14:18:00.1120771Z [lighthouse]$ [cluster1] local resource_file=/opt/shipyard/scripts/resources/nginx-demo.yaml
2020-11-02T14:18:00.1142735Z [lighthouse]$ [cluster1] kubectl delete -f /opt/shipyard/scripts/resources/nginx-demo.yaml
2020-11-02T14:18:00.1166957Z [lighthouse]$ [cluster1] kubectl delete -f /opt/shipyard/scripts/resources/nginx-demo.yaml
2020-11-02T14:18:00.1187745Z [lighthouse]$ [cluster1] command kubectl --context=cluster1 delete -f /opt/shipyard/scripts/resources/nginx-demo.yaml
2020-11-02T14:18:00.1209025Z [lighthouse]$ [cluster1] kubectl --context=cluster1 delete -f /opt/shipyard/scripts/resources/nginx-demo.yaml
2020-11-02T14:18:00.4264660Z deployment.apps "nginx-demo" deleted
2020-11-02T14:18:00.5090501Z service "nginx-demo" deleted
2020-11-02T14:18:00.5547549Z [lighthouse]$ run_subm_clusters update_coredns_configmap
2020-11-02T14:18:00.5564450Z [lighthouse]$ run_subm_clusters update_coredns_configmap
2020-11-02T14:18:00.5577641Z [lighthouse]$ declare -a subm_clusters
2020-11-02T14:18:00.5644620Z [lighthouse]$ [cluster2] run_parallel cluster1 cluster2 update_coredns_configmap
2020-11-02T14:18:00.5689147Z [lighthouse]$ [cluster2] run_parallel cluster1 cluster2 cluster1 cluster2 update_coredns_configmap
2020-11-02T14:18:00.5708506Z [lighthouse]$ [cluster2] local cmnd=update_coredns_configmap
2020-11-02T14:18:00.5738614Z [lighthouse]$ [cluster2] declare -A pids
2020-11-02T14:18:00.5819801Z [lighthouse]$ [cluster2] eval echo cluster1 cluster2
2020-11-02T14:18:00.5851243Z [lighthouse]$ [cluster1] pids[cluster1]=5697
2020-11-02T14:18:00.5947828Z [lighthouse]$ [cluster1] set -o pipefail
2020-11-02T14:18:00.5962774Z [lighthouse]$ [cluster1] update_coredns_configmap
2020-11-02T14:18:00.5983658Z [lighthouse]$ [cluster2] pids[cluster2]=5700
2020-11-02T14:18:00.6113291Z [lighthouse]$ [cluster2] set -o pipefail
2020-11-02T14:18:00.6132496Z [lighthouse]$ [cluster2] update_coredns_configmap
2020-11-02T14:18:00.6167467Z [lighthouse]$ [cluster1] sed /\[cluster1]/!s/^/[cluster1] /
2020-11-02T14:18:00.6198307Z [lighthouse]$ [cluster2] wait 5700
2020-11-02T14:18:00.6218280Z [lighthouse]$ [cluster2] sed /\[cluster2]/!s/^/[cluster2] /
2020-11-02T14:18:03.1865545Z [lighthouse]$ [cluster2] update_coredns_configmap
2020-11-02T14:18:03.1866394Z [cluster2] 
2020-11-02T14:18:03.1866833Z [cluster2] 
2020-11-02T14:18:03.1867623Z [lighthouse]$ [cluster2] command kubectl --context=cluster2 get cm -n kube-system coredns -o yaml
2020-11-02T14:18:03.1868682Z [lighthouse]$ [cluster2] kubectl --context=cluster2 get cm -n kube-system coredns -o yaml
2020-11-02T14:18:03.1869836Z [lighthouse]$ [cluster2] grep -q clusterset /tmp/coremap-cluster2.yaml
2020-11-02T14:18:03.1870655Z [lighthouse]$ [cluster2] CLUSTER_IP=100.2.249.85
2020-11-02T14:18:03.1872092Z [lighthouse]$ [cluster2] kubectl get svc -n submariner-operator submariner-lighthouse-coredns
2020-11-02T14:18:03.1873096Z [lighthouse]$ [cluster2] tail -n 1
2020-11-02T14:18:03.1873651Z [lighthouse]$ [cluster2] awk {print $3;}
2020-11-02T14:18:03.1874667Z [lighthouse]$ [cluster2] kubectl get svc -n submariner-operator submariner-lighthouse-coredns
2020-11-02T14:18:03.1876167Z [lighthouse]$ [cluster2] command kubectl --context=cluster2 get svc -n submariner-operator submariner-lighthouse-coredns
2020-11-02T14:18:03.1878054Z [lighthouse]$ [cluster2] kubectl --context=cluster2 get svc -n submariner-operator submariner-lighthouse-coredns
2020-11-02T14:18:03.1879191Z [cluster2] 
2020-11-02T14:18:03.1880258Z [lighthouse]$ [cluster2] sed -i -e /Corefile:/r /tmp/coredns-cm-100.2.249.85.yaml /tmp/coremap-cluster2.yaml
2020-11-02T14:18:03.1881388Z [lighthouse]$ [cluster2] kubectl -n kube-system replace -f /tmp/coremap-cluster2.yaml
2020-11-02T14:18:03.1882414Z [lighthouse]$ [cluster2] kubectl -n kube-system replace -f /tmp/coremap-cluster2.yaml
2020-11-02T14:18:03.1883545Z [lighthouse]$ [cluster2] command kubectl --context=cluster2 -n kube-system replace -f /tmp/coremap-cluster2.yaml
2020-11-02T14:18:03.1884900Z [lighthouse]$ [cluster2] kubectl --context=cluster2 -n kube-system replace -f /tmp/coremap-cluster2.yaml
2020-11-02T14:18:03.1885612Z [cluster2] configmap/coredns replaced
2020-11-02T14:18:03.1886310Z [lighthouse]$ [cluster2] kubectl -n kube-system describe cm coredns
2020-11-02T14:18:03.1887115Z [lighthouse]$ [cluster2] kubectl -n kube-system describe cm coredns
2020-11-02T14:18:03.1888033Z [lighthouse]$ [cluster2] command kubectl --context=cluster2 -n kube-system describe cm coredns
2020-11-02T14:18:03.1888997Z [lighthouse]$ [cluster2] kubectl --context=cluster2 -n kube-system describe cm coredns
2020-11-02T14:18:03.1889544Z [cluster2] Name:         coredns
2020-11-02T14:18:03.1890062Z [cluster2] Namespace:    kube-system
2020-11-02T14:18:03.1890437Z [cluster2] Labels:       <none>
2020-11-02T14:18:03.1890795Z [cluster2] Annotations:  <none>
2020-11-02T14:18:03.1891096Z [cluster2]
2020-11-02T14:18:03.1891363Z [cluster2] Data
2020-11-02T14:18:03.1891630Z [cluster2] ====
2020-11-02T14:18:03.1891923Z [cluster2] Corefile:
2020-11-02T14:18:03.1892348Z [cluster2] ----
2020-11-02T14:18:03.1892703Z [cluster2] clusterset.local:53 {
2020-11-02T14:18:03.1893082Z [cluster2]     forward . 100.2.249.85
2020-11-02T14:18:03.1893353Z [cluster2] }
2020-11-02T14:18:03.1893612Z [cluster2] .:53 {
2020-11-02T14:18:03.1893887Z [cluster2]     errors
2020-11-02T14:18:03.1894407Z [cluster2]     health {
2020-11-02T14:18:03.1894730Z [cluster2]        lameduck 5s
2020-11-02T14:18:03.1895032Z [cluster2]     }
2020-11-02T14:18:03.1895312Z [cluster2]     ready
2020-11-02T14:18:03.1895947Z [cluster2]     kubernetes cluster2.local in-addr.arpa ip6.arpa {
2020-11-02T14:18:03.1896452Z [cluster2]        pods insecure
2020-11-02T14:18:03.1897046Z [cluster2]        fallthrough in-addr.arpa ip6.arpa
2020-11-02T14:18:03.1897448Z [cluster2]        ttl 30
2020-11-02T14:18:03.1898047Z [cluster2]     }
2020-11-02T14:18:03.1898338Z [cluster2]     prometheus :9153
2020-11-02T14:18:03.1898706Z [cluster2]     forward . /etc/resolv.conf
2020-11-02T14:18:03.1899149Z [cluster2]     cache 30
2020-11-02T14:18:03.1899417Z [cluster2]     loop
2020-11-02T14:18:03.1899693Z [cluster2]     reload
2020-11-02T14:18:03.1900002Z [cluster2]     loadbalance
2020-11-02T14:18:03.1900292Z [cluster2] }
2020-11-02T14:18:03.1900540Z [cluster2]
2020-11-02T14:18:03.1900820Z [cluster2] Events:  <none>
2020-11-02T14:18:03.1924654Z [lighthouse]$ [cluster2] wait 5697
2020-11-02T14:18:03.2458961Z [lighthouse]$ [cluster1] update_coredns_configmap
2020-11-02T14:18:03.2459629Z [cluster1] 
2020-11-02T14:18:03.2460003Z [cluster1] 
2020-11-02T14:18:03.2460714Z [lighthouse]$ [cluster1] command kubectl --context=cluster1 get cm -n kube-system coredns -o yaml
2020-11-02T14:18:03.2461643Z [lighthouse]$ [cluster1] kubectl --context=cluster1 get cm -n kube-system coredns -o yaml
2020-11-02T14:18:03.2462537Z [lighthouse]$ [cluster1] grep -q clusterset /tmp/coremap-cluster1.yaml
2020-11-02T14:18:03.2463297Z [lighthouse]$ [cluster1] CLUSTER_IP=100.1.80.194
2020-11-02T14:18:03.2464928Z [lighthouse]$ [cluster1] kubectl get svc -n submariner-operator submariner-lighthouse-coredns
2020-11-02T14:18:03.2465911Z [lighthouse]$ [cluster1] tail -n 1
2020-11-02T14:18:03.2466503Z [lighthouse]$ [cluster1] awk {print $3;}
2020-11-02T14:18:03.2467804Z [lighthouse]$ [cluster1] kubectl get svc -n submariner-operator submariner-lighthouse-coredns
2020-11-02T14:18:03.2469628Z [lighthouse]$ [cluster1] command kubectl --context=cluster1 get svc -n submariner-operator submariner-lighthouse-coredns
2020-11-02T14:18:03.2471110Z [lighthouse]$ [cluster1] kubectl --context=cluster1 get svc -n submariner-operator submariner-lighthouse-coredns
2020-11-02T14:18:03.2472425Z [cluster1] 
2020-11-02T14:18:03.2475401Z [lighthouse]$ [cluster1] sed -i -e /Corefile:/r /tmp/coredns-cm-100.1.80.194.yaml /tmp/coremap-cluster1.yaml
2020-11-02T14:18:03.2476642Z [lighthouse]$ [cluster1] kubectl -n kube-system replace -f /tmp/coremap-cluster1.yaml
2020-11-02T14:18:03.2477678Z [lighthouse]$ [cluster1] kubectl -n kube-system replace -f /tmp/coremap-cluster1.yaml
2020-11-02T14:18:03.2478816Z [lighthouse]$ [cluster1] command kubectl --context=cluster1 -n kube-system replace -f /tmp/coremap-cluster1.yaml
2020-11-02T14:18:03.2480165Z [lighthouse]$ [cluster1] kubectl --context=cluster1 -n kube-system replace -f /tmp/coremap-cluster1.yaml
2020-11-02T14:18:03.2480844Z [cluster1] configmap/coredns replaced
2020-11-02T14:18:03.2481531Z [lighthouse]$ [cluster1] kubectl -n kube-system describe cm coredns
2020-11-02T14:18:03.2482309Z [lighthouse]$ [cluster1] kubectl -n kube-system describe cm coredns
2020-11-02T14:18:03.2483169Z [lighthouse]$ [cluster1] command kubectl --context=cluster1 -n kube-system describe cm coredns
2020-11-02T14:18:03.2484193Z [lighthouse]$ [cluster1] kubectl --context=cluster1 -n kube-system describe cm coredns
2020-11-02T14:18:03.2484728Z [cluster1] Name:         coredns
2020-11-02T14:18:03.2485576Z [cluster1] Namespace:    kube-system
2020-11-02T14:18:03.2485940Z [cluster1] Labels:       <none>
2020-11-02T14:18:03.2486463Z [cluster1] Annotations:  <none>
2020-11-02T14:18:03.2486794Z [cluster1]
2020-11-02T14:18:03.2487044Z [cluster1] Data
2020-11-02T14:18:03.2487312Z [cluster1] ====
2020-11-02T14:18:03.2487596Z [cluster1] Corefile:
2020-11-02T14:18:03.2488176Z [cluster1] ----
2020-11-02T14:18:03.2488512Z [cluster1] clusterset.local:53 {
2020-11-02T14:18:03.2488879Z [cluster1]     forward . 100.1.80.194
2020-11-02T14:18:03.2489331Z [cluster1] }
2020-11-02T14:18:03.2489574Z [cluster1] .:53 {
2020-11-02T14:18:03.2489846Z [cluster1]     errors
2020-11-02T14:18:03.2490140Z [cluster1]     health {
2020-11-02T14:18:03.2490801Z [cluster1]        lameduck 5s
2020-11-02T14:18:03.2491090Z [cluster1]     }
2020-11-02T14:18:03.2491358Z [cluster1]     ready
2020-11-02T14:18:03.2491990Z [cluster1]     kubernetes cluster1.local in-addr.arpa ip6.arpa {
2020-11-02T14:18:03.2492620Z [cluster1]        pods insecure
2020-11-02T14:18:03.2493242Z [cluster1]        fallthrough in-addr.arpa ip6.arpa
2020-11-02T14:18:03.2493632Z [cluster1]        ttl 30
2020-11-02T14:18:03.2494073Z [cluster1]     }
2020-11-02T14:18:03.2494382Z [cluster1]     prometheus :9153
2020-11-02T14:18:03.2494772Z [cluster1]     forward . /etc/resolv.conf
2020-11-02T14:18:03.2495493Z [cluster1]     cache 30
2020-11-02T14:18:03.2495748Z [cluster1]     loop
2020-11-02T14:18:03.2496018Z [cluster1]     reload
2020-11-02T14:18:03.2496324Z [cluster1]     loadbalance
2020-11-02T14:18:03.2496614Z [cluster1] }
2020-11-02T14:18:03.2496857Z [cluster1]
2020-11-02T14:18:03.2497135Z [cluster1] Events:  <none>
2020-11-02T14:18:03.2497846Z make[1]: Leaving directory '/go/src/github.com/submariner-io/lighthouse'
2020-11-02T14:18:03.2527113Z [lighthouse]$ declare_kubeconfig
2020-11-02T14:18:03.2538192Z [lighthouse]$ declare_kubeconfig
2020-11-02T14:18:03.2550537Z [lighthouse]$ source /opt/shipyard/scripts/lib/kubecfg
2020-11-02T14:18:03.2559902Z [lighthouse]$ export KUBECONFIG
2020-11-02T14:18:03.2602674Z [lighthouse]$ KUBECONFIG=/go/src/github.com/submariner-io/lighthouse/output/kubeconfigs/kind-config-cluster2:/go/src/github.com/submariner-io/lighthouse/output/kubeconfigs/kind-config-cluster1:
2020-11-02T14:18:03.2616868Z [lighthouse]$ find /go/src/github.com/submariner-io/lighthouse/output/kubeconfigs -type f -printf %p:
2020-11-02T14:18:03.2651427Z [lighthouse]$ test_with_e2e_tests
2020-11-02T14:18:03.2660988Z [lighthouse]$ test_with_e2e_tests
2020-11-02T14:18:03.2674231Z [lighthouse]$ cd /go/src/github.com/submariner-io/lighthouse/test/e2e
2020-11-02T14:18:03.2697168Z [e2e]$ go test -v -timeout 30m -args -ginkgo.v -ginkgo.randomizeAllSpecs -ginkgo.trace -submariner-namespace submariner-operator -dp-context cluster1 -dp-context cluster2 -ginkgo.reportPassed -test.timeout 15m -ginkgo.focus \[.*\] -ginkgo.reportFile /go/src/github.com/submariner-io/lighthouse/output/e2e-junit.xml
2020-11-02T14:18:03.2717221Z [e2e]$ tee /go/src/github.com/submariner-io/lighthouse/output/e2e-tests.log
2020-11-02T14:18:03.2740257Z [e2e]$ generate_context_flags
2020-11-02T14:18:03.2752362Z [e2e]$ generate_context_flags
2020-11-02T14:18:03.2764414Z [e2e]$ [cluster1] printf  -dp-context cluster1
2020-11-02T14:18:03.2775755Z [e2e]$ [cluster2] printf  -dp-context cluster2
2020-11-02T14:19:06.3727452Z === RUN   TestE2E
2020-11-02T14:19:06.3766339Z Running Suite: Submariner E2E suite
2020-11-02T14:19:06.3770968Z ===================================
2020-11-02T14:19:06.3779671Z Random Seed: 1604326746 - Will randomize all specs
2020-11-02T14:19:06.3837139Z Will run 11 of 11 specs
2020-11-02T14:19:06.3840806Z
2020-11-02T14:19:06.3858270Z STEP: Creating kubernetes clients
2020-11-02T14:19:06.3902010Z STEP: Creating lighthouse clients
2020-11-02T14:19:06.3906421Z [discovery] Test Service Discovery Across Clusters when a pod tries to resolve a service which is present locally and in a remote cluster
2020-11-02T14:19:06.3907838Z   should resolve the local service
2020-11-02T14:19:06.3908868Z   /go/src/github.com/submariner-io/lighthouse/test/e2e/discovery/service_discovery.go:32
2020-11-02T14:19:06.3909978Z STEP: Creating namespace objects with basename "discovery"
2020-11-02T14:19:06.4067861Z STEP: Generated namespace "e2e-tests-discovery-8k966" in cluster "cluster1" to execute the tests in
2020-11-02T14:19:06.4073290Z STEP: Creating namespace "e2e-tests-discovery-8k966" in cluster "cluster2"
2020-11-02T14:19:06.4213102Z STEP: Creating an Nginx Deployment on "cluster1"
2020-11-02T14:19:11.7583283Z STEP: Creating a Nginx Service on "cluster1"
2020-11-02T14:19:11.7587262Z STEP: Creating an Nginx Deployment on "cluster2"
2020-11-02T14:19:16.5372084Z STEP: Creating a Nginx Service on "cluster2"
2020-11-02T14:19:16.5427721Z STEP: Creating serviceExport nginx-demo.e2e-tests-discovery-8k966 on "cluster2"
2020-11-02T14:19:16.5788021Z STEP: Retrieving ServiceExport nginx-demo.e2e-tests-discovery-8k966 on "cluster2"
2020-11-02T14:19:21.5931836Z STEP: Creating a Netshoot Deployment on "cluster1"
2020-11-02T14:19:26.6241072Z Nov  2 14:19:26.623: INFO: ExecWithOptions {Command:[cat /etc/resolv.conf] Namespace:e2e-tests-discovery-8k966 PodName:netshootqgnrd-6757486d56-t7wn5 ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
2020-11-02T14:19:26.6997707Z STEP: Executing "dig +short nginx-demo.e2e-tests-discovery-8k966.svc.cluster1.local" to verify IP "100.1.170.129" for service "nginx-demo" "is" discoverable
2020-11-02T14:19:26.7001074Z Nov  2 14:19:26.699: INFO: ExecWithOptions {Command:[dig +short nginx-demo.e2e-tests-discovery-8k966.svc.cluster1.local] Namespace:e2e-tests-discovery-8k966 PodName:netshootqgnrd-6757486d56-t7wn5 ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
2020-11-02T14:19:26.7905274Z STEP: Validating that dig result is "100.1.170.129"
2020-11-02T14:19:26.7906092Z STEP: Deleting service "nginx-demo" on "cluster1"
2020-11-02T14:19:26.8230046Z STEP: Retrieving service nginx-demo.e2e-tests-discovery-8k966 on "cluster2"
2020-11-02T14:19:26.8271763Z STEP: Retrieving ServiceImport for nginx-demo-e2e-tests-discovery-8k966- on "cluster1"
2020-11-02T14:19:26.8393071Z STEP: Executing "dig +short nginx-demo.e2e-tests-discovery-8k966.svc.clusterset.local" to verify IP "100.2.183.98" for service "nginx-demo" "is" discoverable
2020-11-02T14:19:26.8396180Z Nov  2 14:19:26.838: INFO: ExecWithOptions {Command:[dig +short nginx-demo.e2e-tests-discovery-8k966.svc.clusterset.local] Namespace:e2e-tests-discovery-8k966 PodName:netshootqgnrd-6757486d56-t7wn5 ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
2020-11-02T14:19:27.0357153Z STEP: Validating that dig result is ""
2020-11-02T14:19:32.0361461Z Nov  2 14:19:32.035: INFO: ExecWithOptions {Command:[dig +short nginx-demo.e2e-tests-discovery-8k966.svc.clusterset.local] Namespace:e2e-tests-discovery-8k966 PodName:netshootqgnrd-6757486d56-t7wn5 ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
2020-11-02T14:19:32.1339357Z STEP: Validating that dig result is ""
2020-11-02T14:19:37.0366081Z Nov  2 14:19:37.035: INFO: ExecWithOptions {Command:[dig +short nginx-demo.e2e-tests-discovery-8k966.svc.clusterset.local] Namespace:e2e-tests-discovery-8k966 PodName:netshootqgnrd-6757486d56-t7wn5 ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
2020-11-02T14:19:37.1141617Z STEP: Validating that dig result is "100.2.183.98"
2020-11-02T14:19:37.1147737Z STEP: Deleting serviceExport nginx-demo.e2e-tests-discovery-8k966 on "cluster2"
2020-11-02T14:19:42.1266161Z STEP: Deleting service "nginx-demo" on "cluster2"
2020-11-02T14:19:42.1898097Z STEP: Executing "dig +short nginx-demo.e2e-tests-discovery-8k966.svc.clusterset.local" to verify IP "100.2.183.98" for service "nginx-demo" "is not" discoverable
2020-11-02T14:19:42.1906515Z Nov  2 14:19:42.189: INFO: ExecWithOptions {Command:[dig +short nginx-demo.e2e-tests-discovery-8k966.svc.clusterset.local] Namespace:e2e-tests-discovery-8k966 PodName:netshootqgnrd-6757486d56-t7wn5 ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
2020-11-02T14:19:42.3574786Z STEP: Validating that dig result is not ""
2020-11-02T14:19:42.3584044Z STEP: Deleting namespace "e2e-tests-discovery-8k966" on cluster "cluster1"
2020-11-02T14:19:42.3638399Z STEP: Deleting namespace "e2e-tests-discovery-8k966" on cluster "cluster2"
2020-11-02T14:19:42.3700662Z •
2020-11-02T14:19:42.3701314Z ------------------------------
2020-11-02T14:19:42.3702426Z STEP: Creating namespace objects with basename "discovery"
2020-11-02T14:19:42.3703563Z STEP: Generated namespace "e2e-tests-discovery-8k966" in cluster "cluster1" to execute the tests in
2020-11-02T14:19:42.3705195Z STEP: Creating namespace "e2e-tests-discovery-8k966" in cluster "cluster2"
2020-11-02T14:19:42.3706094Z STEP: Creating an Nginx Deployment on "cluster1"
2020-11-02T14:19:42.3706736Z STEP: Creating a Nginx Service on "cluster1"
2020-11-02T14:19:42.3707776Z STEP: Creating an Nginx Deployment on "cluster2"
2020-11-02T14:19:42.3708424Z STEP: Creating a Nginx Service on "cluster2"
2020-11-02T14:19:42.3709423Z STEP: Creating serviceExport nginx-demo.e2e-tests-discovery-8k966 on "cluster2"
2020-11-02T14:19:42.3711119Z STEP: Retrieving ServiceExport nginx-demo.e2e-tests-discovery-8k966 on "cluster2"
2020-11-02T14:19:42.3712652Z STEP: Creating a Netshoot Deployment on "cluster1"
2020-11-02T14:19:42.3714343Z Nov  2 14:19:26.623: INFO: ExecWithOptions {Command:[cat /etc/resolv.conf] Namespace:e2e-tests-discovery-8k966 PodName:netshootqgnrd-6757486d56-t7wn5 ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
2020-11-02T14:19:42.3717454Z STEP: Executing "dig +short nginx-demo.e2e-tests-discovery-8k966.svc.cluster1.local" to verify IP "100.1.170.129" for service "nginx-demo" "is" discoverable
2020-11-02T14:19:42.3721396Z Nov  2 14:19:26.699: INFO: ExecWithOptions {Command:[dig +short nginx-demo.e2e-tests-discovery-8k966.svc.cluster1.local] Namespace:e2e-tests-discovery-8k966 PodName:netshootqgnrd-6757486d56-t7wn5 ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
2020-11-02T14:19:42.3724257Z STEP: Validating that dig result is "100.1.170.129"
2020-11-02T14:19:42.3724940Z STEP: Deleting service "nginx-demo" on "cluster1"
2020-11-02T14:19:42.3726464Z STEP: Retrieving service nginx-demo.e2e-tests-discovery-8k966 on "cluster2"
2020-11-02T14:19:42.3727773Z STEP: Retrieving ServiceImport for nginx-demo-e2e-tests-discovery-8k966- on "cluster1"
2020-11-02T14:19:42.3729629Z STEP: Executing "dig +short nginx-demo.e2e-tests-discovery-8k966.svc.clusterset.local" to verify IP "100.2.183.98" for service "nginx-demo" "is" discoverable
2020-11-02T14:19:42.3732769Z Nov  2 14:19:26.838: INFO: ExecWithOptions {Command:[dig +short nginx-demo.e2e-tests-discovery-8k966.svc.clusterset.local] Namespace:e2e-tests-discovery-8k966 PodName:netshootqgnrd-6757486d56-t7wn5 ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
2020-11-02T14:19:42.3735035Z STEP: Validating that dig result is ""
2020-11-02T14:19:42.3737346Z Nov  2 14:19:32.035: INFO: ExecWithOptions {Command:[dig +short nginx-demo.e2e-tests-discovery-8k966.svc.clusterset.local] Namespace:e2e-tests-discovery-8k966 PodName:netshootqgnrd-6757486d56-t7wn5 ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
2020-11-02T14:19:42.3739811Z STEP: Validating that dig result is ""
2020-11-02T14:19:42.3758024Z Nov  2 14:19:37.035: INFO: ExecWithOptions {Command:[dig +short nginx-demo.e2e-tests-discovery-8k966.svc.clusterset.local] Namespace:e2e-tests-discovery-8k966 PodName:netshootqgnrd-6757486d56-t7wn5 ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
2020-11-02T14:19:42.3760594Z STEP: Validating that dig result is "100.2.183.98"
2020-11-02T14:19:42.3761607Z STEP: Deleting serviceExport nginx-demo.e2e-tests-discovery-8k966 on "cluster2"
2020-11-02T14:19:42.3762616Z STEP: Deleting service "nginx-demo" on "cluster2"
2020-11-02T14:19:42.3764343Z STEP: Executing "dig +short nginx-demo.e2e-tests-discovery-8k966.svc.clusterset.local" to verify IP "100.2.183.98" for service "nginx-demo" "is not" discoverable
2020-11-02T14:19:42.3767782Z Nov  2 14:19:42.189: INFO: ExecWithOptions {Command:[dig +short nginx-demo.e2e-tests-discovery-8k966.svc.clusterset.local] Namespace:e2e-tests-discovery-8k966 PodName:netshootqgnrd-6757486d56-t7wn5 ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
2020-11-02T14:19:42.3769999Z STEP: Validating that dig result is not ""
2020-11-02T14:19:42.3770815Z STEP: Deleting namespace "e2e-tests-discovery-8k966" on cluster "cluster1"
2020-11-02T14:19:42.3771957Z STEP: Deleting namespace "e2e-tests-discovery-8k966" on cluster "cluster2"
2020-11-02T14:19:42.3772390Z
2020-11-02T14:19:42.3772540Z
2020-11-02T14:19:42.3773282Z [discovery] Test Service Discovery Across Clusters when there are active pods for a service in only one cluster
2020-11-02T14:19:42.3774159Z   should not resolve the service on the cluster without active pods
2020-11-02T14:19:42.3775018Z   /go/src/github.com/submariner-io/lighthouse/test/e2e/discovery/service_discovery.go:50
2020-11-02T14:19:42.3775843Z STEP: Creating namespace objects with basename "discovery"
2020-11-02T14:19:42.3859307Z STEP: Generated namespace "e2e-tests-discovery-smfnt" in cluster "cluster1" to execute the tests in
2020-11-02T14:19:42.3860462Z STEP: Creating namespace "e2e-tests-discovery-smfnt" in cluster "cluster2"
2020-11-02T14:19:42.4022821Z STEP: Creating an Nginx Deployment on on "cluster2"
2020-11-02T14:19:47.4484759Z STEP: Creating a Nginx Service on "cluster2"
2020-11-02T14:19:47.4874759Z STEP: Creating serviceExport nginx-demo.e2e-tests-discovery-smfnt on "cluster2"
2020-11-02T14:19:47.5028416Z STEP: Retrieving ServiceExport nginx-demo.e2e-tests-discovery-smfnt on "cluster2"
2020-11-02T14:19:52.5253616Z STEP: Creating a Netshoot Deployment on "cluster1"
2020-11-02T14:19:57.5331342Z STEP: Retrieving service nginx-demo.e2e-tests-discovery-smfnt on "cluster2"
2020-11-02T14:19:57.5352079Z STEP: Retrieving ServiceImport for nginx-demo-e2e-tests-discovery-smfnt- on "cluster1"
2020-11-02T14:19:57.5378975Z STEP: Creating an Nginx Deployment on on "cluster1"
2020-11-02T14:20:02.5450840Z STEP: Creating a Nginx Service on "cluster1"
2020-11-02T14:20:02.5500190Z STEP: Creating serviceExport nginx-demo.e2e-tests-discovery-smfnt on "cluster1"
2020-11-02T14:20:02.5742168Z STEP: Retrieving ServiceExport nginx-demo.e2e-tests-discovery-smfnt on "cluster1"
2020-11-02T14:20:07.5895355Z STEP: Retrieving service nginx-demo.e2e-tests-discovery-smfnt on "cluster1"
2020-11-02T14:20:07.5909593Z STEP: Retrieving ServiceImport for nginx-demo-e2e-tests-discovery-smfnt- on "cluster1"
2020-11-02T14:20:07.5937493Z STEP: Executing "dig +short nginx-demo.e2e-tests-discovery-smfnt.svc.clusterset.local" to verify IP "100.1.236.203" for service "nginx-demo" "is" discoverable
2020-11-02T14:20:07.5941352Z Nov  2 14:20:07.592: INFO: ExecWithOptions {Command:[dig +short nginx-demo.e2e-tests-discovery-smfnt.svc.clusterset.local] Namespace:e2e-tests-discovery-smfnt PodName:netshoothkzmx-6757486d56-cpv2r ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
2020-11-02T14:20:07.6696062Z STEP: Validating that dig result is "100.2.196.144"
2020-11-02T14:20:12.6705789Z Nov  2 14:20:12.669: INFO: ExecWithOptions {Command:[dig +short nginx-demo.e2e-tests-discovery-smfnt.svc.clusterset.local] Namespace:e2e-tests-discovery-smfnt PodName:netshoothkzmx-6757486d56-cpv2r ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
2020-11-02T14:20:12.7763631Z STEP: Validating that dig result is "100.1.236.203"
2020-11-02T14:20:12.7769488Z STEP: Executing "dig +short nginx-demo.e2e-tests-discovery-smfnt.svc.clusterset.local" to verify IP "100.2.196.144" for service "nginx-demo" "is" discoverable
2020-11-02T14:20:12.7776820Z Nov  2 14:20:12.777: INFO: ExecWithOptions {Command:[dig +short nginx-demo.e2e-tests-discovery-smfnt.svc.clusterset.local] Namespace:e2e-tests-discovery-smfnt PodName:netshoothkzmx-6757486d56-cpv2r ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
2020-11-02T14:20:12.8566330Z STEP: Validating that dig result is "100.2.196.144"
2020-11-02T14:20:12.8567140Z STEP: Setting Nginx deployment replicas to 0 in cluster "cluster2"
2020-11-02T14:20:12.8612070Z STEP: Executing "dig +short nginx-demo.e2e-tests-discovery-smfnt.svc.clusterset.local" to verify IP "100.1.236.203" for service "nginx-demo" "is" discoverable
2020-11-02T14:20:12.8615529Z Nov  2 14:20:12.860: INFO: ExecWithOptions {Command:[dig +short nginx-demo.e2e-tests-discovery-smfnt.svc.clusterset.local] Namespace:e2e-tests-discovery-smfnt PodName:netshoothkzmx-6757486d56-cpv2r ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
2020-11-02T14:20:13.0526031Z STEP: Validating that dig result is "100.1.236.203"
2020-11-02T14:20:13.0534453Z STEP: Executing "dig +short nginx-demo.e2e-tests-discovery-smfnt.svc.clusterset.local" to verify IP "100.2.196.144" for service "nginx-demo" "is not" discoverable
2020-11-02T14:20:13.0542189Z Nov  2 14:20:13.046: INFO: ExecWithOptions {Command:[dig +short nginx-demo.e2e-tests-discovery-smfnt.svc.clusterset.local] Namespace:e2e-tests-discovery-smfnt PodName:netshoothkzmx-6757486d56-cpv2r ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
2020-11-02T14:20:13.1652447Z STEP: Validating that dig result is not "100.1.236.203"
2020-11-02T14:20:13.1656106Z STEP: Setting Nginx deployment replicas to 0 in cluster "cluster1"
2020-11-02T14:20:13.1737744Z STEP: Executing "dig +short nginx-demo.e2e-tests-discovery-smfnt.svc.clusterset.local" to verify IP "100.1.236.203" for service "nginx-demo" "is not" discoverable
2020-11-02T14:20:13.1744467Z Nov  2 14:20:13.173: INFO: ExecWithOptions {Command:[dig +short nginx-demo.e2e-tests-discovery-smfnt.svc.clusterset.local] Namespace:e2e-tests-discovery-smfnt PodName:netshoothkzmx-6757486d56-cpv2r ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
2020-11-02T14:20:13.4124170Z STEP: Validating that dig result is not ""
2020-11-02T14:20:13.4126221Z STEP: Executing "dig +short nginx-demo.e2e-tests-discovery-smfnt.svc.clusterset.local" to verify IP "100.2.196.144" for service "nginx-demo" "is not" discoverable
2020-11-02T14:20:13.4129748Z Nov  2 14:20:13.411: INFO: ExecWithOptions {Command:[dig +short nginx-demo.e2e-tests-discovery-smfnt.svc.clusterset.local] Namespace:e2e-tests-discovery-smfnt PodName:netshoothkzmx-6757486d56-cpv2r ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
2020-11-02T14:20:13.5067259Z STEP: Validating that dig result is not ""
2020-11-02T14:20:13.5068175Z STEP: Deleting namespace "e2e-tests-discovery-smfnt" on cluster "cluster1"
2020-11-02T14:20:13.5135175Z STEP: Deleting namespace "e2e-tests-discovery-smfnt" on cluster "cluster2"
2020-11-02T14:20:13.5264172Z •
2020-11-02T14:20:13.5278170Z ------------------------------
2020-11-02T14:20:13.5281212Z STEP: Creating namespace objects with basename "discovery"
2020-11-02T14:20:13.5286092Z STEP: Generated namespace "e2e-tests-discovery-smfnt" in cluster "cluster1" to execute the tests in
2020-11-02T14:20:13.5293921Z STEP: Creating namespace "e2e-tests-discovery-smfnt" in cluster "cluster2"
2020-11-02T14:20:13.5300230Z STEP: Creating an Nginx Deployment on on "cluster2"
2020-11-02T14:20:13.5310953Z STEP: Creating a Nginx Service on "cluster2"
2020-11-02T14:20:13.5312687Z STEP: Creating serviceExport nginx-demo.e2e-tests-discovery-smfnt on "cluster2"
2020-11-02T14:20:13.5314417Z STEP: Retrieving ServiceExport nginx-demo.e2e-tests-discovery-smfnt on "cluster2"
2020-11-02T14:20:13.5319400Z STEP: Creating a Netshoot Deployment on "cluster1"
2020-11-02T14:20:13.5321495Z STEP: Retrieving service nginx-demo.e2e-tests-discovery-smfnt on "cluster2"
2020-11-02T14:20:13.5327668Z STEP: Retrieving ServiceImport for nginx-demo-e2e-tests-discovery-smfnt- on "cluster1"
2020-11-02T14:20:13.5333524Z STEP: Creating an Nginx Deployment on on "cluster1"
2020-11-02T14:20:13.5336429Z STEP: Creating a Nginx Service on "cluster1"
2020-11-02T14:20:13.5340888Z STEP: Creating serviceExport nginx-demo.e2e-tests-discovery-smfnt on "cluster1"
2020-11-02T14:20:13.5350398Z STEP: Retrieving ServiceExport nginx-demo.e2e-tests-discovery-smfnt on "cluster1"
2020-11-02T14:20:13.5357967Z STEP: Retrieving service nginx-demo.e2e-tests-discovery-smfnt on "cluster1"
2020-11-02T14:20:13.5371793Z STEP: Retrieving ServiceImport for nginx-demo-e2e-tests-discovery-smfnt- on "cluster1"
2020-11-02T14:20:13.5380957Z STEP: Executing "dig +short nginx-demo.e2e-tests-discovery-smfnt.svc.clusterset.local" to verify IP "100.1.236.203" for service "nginx-demo" "is" discoverable
2020-11-02T14:20:13.5386969Z Nov  2 14:20:07.592: INFO: ExecWithOptions {Command:[dig +short nginx-demo.e2e-tests-discovery-smfnt.svc.clusterset.local] Namespace:e2e-tests-discovery-smfnt PodName:netshoothkzmx-6757486d56-cpv2r ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
2020-11-02T14:20:13.5392851Z STEP: Validating that dig result is "100.2.196.144"
2020-11-02T14:20:13.5402696Z Nov  2 14:20:12.669: INFO: ExecWithOptions {Command:[dig +short nginx-demo.e2e-tests-discovery-smfnt.svc.clusterset.local] Namespace:e2e-tests-discovery-smfnt PodName:netshoothkzmx-6757486d56-cpv2r ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
2020-11-02T14:20:13.5417418Z STEP: Validating that dig result is "100.1.236.203"
2020-11-02T14:20:13.5419395Z STEP: Executing "dig +short nginx-demo.e2e-tests-discovery-smfnt.svc.clusterset.local" to verify IP "100.2.196.144" for service "nginx-demo" "is" discoverable
2020-11-02T14:20:13.5422890Z Nov  2 14:20:12.777: INFO: ExecWithOptions {Command:[dig +short nginx-demo.e2e-tests-discovery-smfnt.svc.clusterset.local] Namespace:e2e-tests-discovery-smfnt PodName:netshoothkzmx-6757486d56-cpv2r ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
2020-11-02T14:20:13.5425491Z STEP: Validating that dig result is "100.2.196.144"
2020-11-02T14:20:13.5426195Z STEP: Setting Nginx deployment replicas to 0 in cluster "cluster2"
2020-11-02T14:20:13.5427900Z STEP: Executing "dig +short nginx-demo.e2e-tests-discovery-smfnt.svc.clusterset.local" to verify IP "100.1.236.203" for service "nginx-demo" "is" discoverable
2020-11-02T14:20:13.5431149Z Nov  2 14:20:12.860: INFO: ExecWithOptions {Command:[dig +short nginx-demo.e2e-tests-discovery-smfnt.svc.clusterset.local] Namespace:e2e-tests-discovery-smfnt PodName:netshoothkzmx-6757486d56-cpv2r ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
2020-11-02T14:20:13.5433821Z STEP: Validating that dig result is "100.1.236.203"
2020-11-02T14:20:13.5435581Z STEP: Executing "dig +short nginx-demo.e2e-tests-discovery-smfnt.svc.clusterset.local" to verify IP "100.2.196.144" for service "nginx-demo" "is not" discoverable
2020-11-02T14:20:13.5438934Z Nov  2 14:20:13.046: INFO: ExecWithOptions {Command:[dig +short nginx-demo.e2e-tests-discovery-smfnt.svc.clusterset.local] Namespace:e2e-tests-discovery-smfnt PodName:netshoothkzmx-6757486d56-cpv2r ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
2020-11-02T14:20:13.5529497Z STEP: Validating that dig result is not "100.1.236.203"
2020-11-02T14:20:13.5530436Z STEP: Setting Nginx deployment replicas to 0 in cluster "cluster1"
2020-11-02T14:20:13.5534058Z STEP: Executing "dig +short nginx-demo.e2e-tests-discovery-smfnt.svc.clusterset.local" to verify IP "100.1.236.203" for service "nginx-demo" "is not" discoverable
2020-11-02T14:20:13.5537859Z Nov  2 14:20:13.173: INFO: ExecWithOptions {Command:[dig +short nginx-demo.e2e-tests-discovery-smfnt.svc.clusterset.local] Namespace:e2e-tests-discovery-smfnt PodName:netshoothkzmx-6757486d56-cpv2r ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
2020-11-02T14:20:13.5540332Z STEP: Validating that dig result is not ""
2020-11-02T14:20:13.5542280Z STEP: Executing "dig +short nginx-demo.e2e-tests-discovery-smfnt.svc.clusterset.local" to verify IP "100.2.196.144" for service "nginx-demo" "is not" discoverable
2020-11-02T14:20:13.5545807Z Nov  2 14:20:13.411: INFO: ExecWithOptions {Command:[dig +short nginx-demo.e2e-tests-discovery-smfnt.svc.clusterset.local] Namespace:e2e-tests-discovery-smfnt PodName:netshoothkzmx-6757486d56-cpv2r ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
2020-11-02T14:20:13.5547987Z STEP: Validating that dig result is not ""
2020-11-02T14:20:13.5549100Z STEP: Deleting namespace "e2e-tests-discovery-smfnt" on cluster "cluster1"
2020-11-02T14:20:13.5550390Z STEP: Deleting namespace "e2e-tests-discovery-smfnt" on cluster "cluster2"
2020-11-02T14:20:13.5550878Z
2020-11-02T14:20:13.5551198Z
2020-11-02T14:20:13.5578979Z [discovery] Test Headless Service Discovery Across Clusters when a pod tries to resolve a headless service which is exported locally and in a remote cluster
2020-11-02T14:20:13.5580049Z   should resolve the backing pod IPs from both clusters
2020-11-02T14:20:13.5580942Z   /go/src/github.com/submariner-io/lighthouse/test/e2e/discovery/headless_services.go:27
2020-11-02T14:20:13.5583798Z STEP: Creating namespace objects with basename "discovery"
2020-11-02T14:20:13.5684216Z STEP: Generated namespace "e2e-tests-discovery-r9fcv" in cluster "cluster1" to execute the tests in
2020-11-02T14:20:13.5691949Z STEP: Creating namespace "e2e-tests-discovery-r9fcv" in cluster "cluster2"
2020-11-02T14:20:13.5825701Z STEP: Creating an Nginx Deployment on on "cluster2"
2020-11-02T14:20:18.6834757Z STEP: Creating a Nginx Headless Service on "cluster2"
2020-11-02T14:20:18.7052452Z STEP: Creating serviceExport nginx-headless.e2e-tests-discovery-r9fcv on "cluster2"
2020-11-02T14:20:18.7599607Z STEP: Retrieving ServiceExport nginx-headless.e2e-tests-discovery-r9fcv on "cluster2"
2020-11-02T14:20:23.7867451Z STEP: Creating an Nginx Deployment on on "cluster1"
2020-11-02T14:20:28.8034603Z STEP: Creating a Nginx Headless Service on "cluster1"
2020-11-02T14:20:28.8181848Z STEP: Creating serviceExport nginx-headless.e2e-tests-discovery-r9fcv on "cluster1"
2020-11-02T14:20:28.8238167Z STEP: Retrieving ServiceExport nginx-headless.e2e-tests-discovery-r9fcv on "cluster1"
2020-11-02T14:20:33.8278942Z STEP: Creating a Netshoot Deployment on "cluster1"
2020-11-02T14:20:38.8378427Z STEP: Retrieving Endpoints for nginx-headless on "cluster2"
2020-11-02T14:20:38.8411838Z STEP: Retrieving Endpoints for nginx-headless on "cluster1"
2020-11-02T14:20:38.8476613Z STEP: Executing "dig +short nginx-headless.e2e-tests-discovery-r9fcv.svc.clusterset.local" to verify IPs [10.2.128.4 10.1.224.4] for service "nginx-headless" "are" discoverable
2020-11-02T14:20:38.8481046Z Nov  2 14:20:38.847: INFO: ExecWithOptions {Command:[dig +short nginx-headless.e2e-tests-discovery-r9fcv.svc.clusterset.local] Namespace:e2e-tests-discovery-r9fcv PodName:netshoothq4jr-6757486d56-z444w ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
2020-11-02T14:20:38.9277878Z STEP: Validating that dig result are "10.2.128.4\n10.1.224.4"
2020-11-02T14:20:38.9279159Z STEP: Deleting serviceExport nginx-headless.e2e-tests-discovery-r9fcv on "cluster2"
2020-11-02T14:20:43.9412136Z STEP: Executing "dig +short nginx-headless.e2e-tests-discovery-r9fcv.svc.clusterset.local" to verify IPs [10.2.128.4] for service "nginx-headless" "are not" discoverable
2020-11-02T14:20:43.9418745Z Nov  2 14:20:43.940: INFO: ExecWithOptions {Command:[dig +short nginx-headless.e2e-tests-discovery-r9fcv.svc.clusterset.local] Namespace:e2e-tests-discovery-r9fcv PodName:netshoothq4jr-6757486d56-z444w ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
2020-11-02T14:20:44.0136315Z STEP: Validating that dig result are not "10.1.224.4"
2020-11-02T14:20:44.0144935Z STEP: Executing "dig +short nginx-headless.e2e-tests-discovery-r9fcv.svc.clusterset.local" to verify IPs [10.1.224.4] for service "nginx-headless" "are" discoverable
2020-11-02T14:20:44.0153439Z Nov  2 14:20:44.014: INFO: ExecWithOptions {Command:[dig +short nginx-headless.e2e-tests-discovery-r9fcv.svc.clusterset.local] Namespace:e2e-tests-discovery-r9fcv PodName:netshoothq4jr-6757486d56-z444w ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
2020-11-02T14:20:44.0811632Z STEP: Validating that dig result are "10.1.224.4"
2020-11-02T14:20:44.0812576Z STEP: Deleting namespace "e2e-tests-discovery-r9fcv" on cluster "cluster1"
2020-11-02T14:20:44.0858832Z STEP: Deleting namespace "e2e-tests-discovery-r9fcv" on cluster "cluster2"
2020-11-02T14:20:44.0939406Z •
2020-11-02T14:20:44.0939857Z ------------------------------
2020-11-02T14:20:44.0940479Z STEP: Creating namespace objects with basename "discovery"
2020-11-02T14:20:44.0942023Z STEP: Generated namespace "e2e-tests-discovery-r9fcv" in cluster "cluster1" to execute the tests in
2020-11-02T14:20:44.0943314Z STEP: Creating namespace "e2e-tests-discovery-r9fcv" in cluster "cluster2"
2020-11-02T14:20:44.0944213Z STEP: Creating an Nginx Deployment on on "cluster2"
2020-11-02T14:20:44.0944940Z STEP: Creating a Nginx Headless Service on "cluster2"
2020-11-02T14:20:44.0946180Z STEP: Creating serviceExport nginx-headless.e2e-tests-discovery-r9fcv on "cluster2"
2020-11-02T14:20:44.0947879Z STEP: Retrieving ServiceExport nginx-headless.e2e-tests-discovery-r9fcv on "cluster2"
2020-11-02T14:20:44.0949147Z STEP: Creating an Nginx Deployment on on "cluster1"
2020-11-02T14:20:44.0949906Z STEP: Creating a Nginx Headless Service on "cluster1"
2020-11-02T14:20:44.0951055Z STEP: Creating serviceExport nginx-headless.e2e-tests-discovery-r9fcv on "cluster1"
2020-11-02T14:20:44.0953163Z STEP: Retrieving ServiceExport nginx-headless.e2e-tests-discovery-r9fcv on "cluster1"
2020-11-02T14:20:44.0954316Z STEP: Creating a Netshoot Deployment on "cluster1"
2020-11-02T14:20:44.0955108Z STEP: Retrieving Endpoints for nginx-headless on "cluster2"
2020-11-02T14:20:44.0955954Z STEP: Retrieving Endpoints for nginx-headless on "cluster1"
2020-11-02T14:20:44.0957889Z STEP: Executing "dig +short nginx-headless.e2e-tests-discovery-r9fcv.svc.clusterset.local" to verify IPs [10.2.128.4 10.1.224.4] for service "nginx-headless" "are" discoverable
2020-11-02T14:20:44.0961748Z Nov  2 14:20:38.847: INFO: ExecWithOptions {Command:[dig +short nginx-headless.e2e-tests-discovery-r9fcv.svc.clusterset.local] Namespace:e2e-tests-discovery-r9fcv PodName:netshoothq4jr-6757486d56-z444w ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
2020-11-02T14:20:44.0964272Z STEP: Validating that dig result are "10.2.128.4\n10.1.224.4"
2020-11-02T14:20:44.0965402Z STEP: Deleting serviceExport nginx-headless.e2e-tests-discovery-r9fcv on "cluster2"
2020-11-02T14:20:44.0967826Z STEP: Executing "dig +short nginx-headless.e2e-tests-discovery-r9fcv.svc.clusterset.local" to verify IPs [10.2.128.4] for service "nginx-headless" "are not" discoverable
2020-11-02T14:20:44.0972010Z Nov  2 14:20:43.940: INFO: ExecWithOptions {Command:[dig +short nginx-headless.e2e-tests-discovery-r9fcv.svc.clusterset.local] Namespace:e2e-tests-discovery-r9fcv PodName:netshoothq4jr-6757486d56-z444w ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
2020-11-02T14:20:44.0974738Z STEP: Validating that dig result are not "10.1.224.4"
2020-11-02T14:20:44.0976662Z STEP: Executing "dig +short nginx-headless.e2e-tests-discovery-r9fcv.svc.clusterset.local" to verify IPs [10.1.224.4] for service "nginx-headless" "are" discoverable
2020-11-02T14:20:44.0980298Z Nov  2 14:20:44.014: INFO: ExecWithOptions {Command:[dig +short nginx-headless.e2e-tests-discovery-r9fcv.svc.clusterset.local] Namespace:e2e-tests-discovery-r9fcv PodName:netshoothq4jr-6757486d56-z444w ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
2020-11-02T14:20:44.0982818Z STEP: Validating that dig result are "10.1.224.4"
2020-11-02T14:20:44.0983681Z STEP: Deleting namespace "e2e-tests-discovery-r9fcv" on cluster "cluster1"
2020-11-02T14:20:44.0984752Z STEP: Deleting namespace "e2e-tests-discovery-r9fcv" on cluster "cluster2"
2020-11-02T14:20:44.0985393Z
2020-11-02T14:20:44.0985531Z
2020-11-02T14:20:44.1003082Z [discovery] Test Stateful Sets Discovery Across Clusters when a pod tries to resolve a podname from stateful set in a local cluster
2020-11-02T14:20:44.1004458Z   should resolve the pod IP from the local cluster
2020-11-02T14:20:44.1005514Z   /go/src/github.com/submariner-io/lighthouse/test/e2e/discovery/statefulsets.go:30
2020-11-02T14:20:44.1006619Z STEP: Creating namespace objects with basename "discovery"
2020-11-02T14:20:44.1023981Z STEP: Generated namespace "e2e-tests-discovery-f24mc" in cluster "cluster1" to execute the tests in
2020-11-02T14:20:44.1025458Z STEP: Creating namespace "e2e-tests-discovery-f24mc" in cluster "cluster2"
2020-11-02T14:20:44.1154045Z STEP: Creating an Nginx Stateful Set on on "cluster2"
2020-11-02T14:20:49.1917151Z STEP: Creating a Nginx Headless Service on "cluster2"
2020-11-02T14:20:49.2030921Z STEP: Creating an Nginx Stateful Set on on "cluster1"
2020-11-02T14:20:54.2748138Z STEP: Creating a Nginx Headless Service on "cluster1"
2020-11-02T14:20:54.2824015Z STEP: Creating serviceExport nginx-ss.e2e-tests-discovery-f24mc on "cluster2"
2020-11-02T14:20:54.2876012Z STEP: Retrieving ServiceExport nginx-ss.e2e-tests-discovery-f24mc on "cluster2"
2020-11-02T14:20:59.2920410Z STEP: Creating serviceExport nginx-ss.e2e-tests-discovery-f24mc on "cluster1"
2020-11-02T14:20:59.2989759Z STEP: Retrieving ServiceExport nginx-ss.e2e-tests-discovery-f24mc on "cluster1"
2020-11-02T14:21:04.3098696Z STEP: Creating a Netshoot Deployment on "cluster1"
2020-11-02T14:21:09.3221118Z STEP: Retrieving EndpointSlices for nginx-ss on "cluster2"
2020-11-02T14:21:09.3254304Z STEP: Executing "dig +short web-0.cluster1.nginx-ss.e2e-tests-discovery-f24mc.svc.clusterset.local" to verify IPs [10.1.224.4] for pod "web-0.cluster1.nginx-ss" "are" discoverable
2020-11-02T14:21:09.3290053Z Nov  2 14:21:09.325: INFO: ExecWithOptions {Command:[dig +short web-0.cluster1.nginx-ss.e2e-tests-discovery-f24mc.svc.clusterset.local] Namespace:e2e-tests-discovery-f24mc PodName:netshootn9md2-6757486d56-8xt9l ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
2020-11-02T14:21:09.4027059Z STEP: Validating that dig result are "10.1.224.4"
2020-11-02T14:21:09.4029466Z STEP: Executing "dig +short web-0.cluster2.nginx-ss.e2e-tests-discovery-f24mc.svc.clusterset.local" to verify IPs [10.2.192.2] for pod "web-0.cluster2.nginx-ss" "are" discoverable
2020-11-02T14:21:09.4034320Z Nov  2 14:21:09.402: INFO: ExecWithOptions {Command:[dig +short web-0.cluster2.nginx-ss.e2e-tests-discovery-f24mc.svc.clusterset.local] Namespace:e2e-tests-discovery-f24mc PodName:netshootn9md2-6757486d56-8xt9l ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
2020-11-02T14:21:09.4951856Z STEP: Validating that dig result are "10.2.192.2"
2020-11-02T14:21:09.4957751Z STEP: Deleting serviceExport nginx-ss.e2e-tests-discovery-f24mc on "cluster2"
2020-11-02T14:21:14.5073634Z STEP: Executing "dig +short web-0.cluster1.nginx-ss.e2e-tests-discovery-f24mc.svc.clusterset.local" to verify IPs [10.1.224.4] for pod "web-0.cluster1.nginx-ss" "are" discoverable
2020-11-02T14:21:14.5077780Z Nov  2 14:21:14.506: INFO: ExecWithOptions {Command:[dig +short web-0.cluster1.nginx-ss.e2e-tests-discovery-f24mc.svc.clusterset.local] Namespace:e2e-tests-discovery-f24mc PodName:netshootn9md2-6757486d56-8xt9l ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
2020-11-02T14:21:14.5815546Z STEP: Validating that dig result are "10.1.224.4"
2020-11-02T14:21:14.5817468Z STEP: Executing "dig +short web-0.cluster2.nginx-ss.e2e-tests-discovery-f24mc.svc.clusterset.local" to verify IPs [10.2.192.2] for pod "web-0.cluster2.nginx-ss" "are not" discoverable
2020-11-02T14:21:14.5821652Z Nov  2 14:21:14.580: INFO: ExecWithOptions {Command:[dig +short web-0.cluster2.nginx-ss.e2e-tests-discovery-f24mc.svc.clusterset.local] Namespace:e2e-tests-discovery-f24mc PodName:netshootn9md2-6757486d56-8xt9l ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
2020-11-02T14:21:14.6487973Z STEP: Validating that dig result are not ""
2020-11-02T14:21:14.6488993Z STEP: Deleting serviceExport nginx-ss.e2e-tests-discovery-f24mc on "cluster1"
2020-11-02T14:21:19.6818563Z STEP: Retrieving EndpointSlices for nginx-ss on "cluster2"
2020-11-02T14:21:19.6845306Z STEP: Deleting namespace "e2e-tests-discovery-f24mc" on cluster "cluster1"
2020-11-02T14:21:19.7048719Z STEP: Deleting namespace "e2e-tests-discovery-f24mc" on cluster "cluster2"
2020-11-02T14:21:19.7131878Z •
2020-11-02T14:21:19.7132678Z ------------------------------
2020-11-02T14:21:19.7133560Z STEP: Creating namespace objects with basename "discovery"
2020-11-02T14:21:19.7139425Z STEP: Generated namespace "e2e-tests-discovery-f24mc" in cluster "cluster1" to execute the tests in
2020-11-02T14:21:19.7140524Z STEP: Creating namespace "e2e-tests-discovery-f24mc" in cluster "cluster2"
2020-11-02T14:21:19.7141414Z STEP: Creating an Nginx Stateful Set on on "cluster2"
2020-11-02T14:21:19.7142585Z STEP: Creating a Nginx Headless Service on "cluster2"
2020-11-02T14:21:19.7143304Z STEP: Creating an Nginx Stateful Set on on "cluster1"
2020-11-02T14:21:19.7144023Z STEP: Creating a Nginx Headless Service on "cluster1"
2020-11-02T14:21:19.7145051Z STEP: Creating serviceExport nginx-ss.e2e-tests-discovery-f24mc on "cluster2"
2020-11-02T14:21:19.7146454Z STEP: Retrieving ServiceExport nginx-ss.e2e-tests-discovery-f24mc on "cluster2"
2020-11-02T14:21:19.7147691Z STEP: Creating serviceExport nginx-ss.e2e-tests-discovery-f24mc on "cluster1"
2020-11-02T14:21:19.7148994Z STEP: Retrieving ServiceExport nginx-ss.e2e-tests-discovery-f24mc on "cluster1"
2020-11-02T14:21:19.7149950Z STEP: Creating a Netshoot Deployment on "cluster1"
2020-11-02T14:21:19.7150687Z STEP: Retrieving EndpointSlices for nginx-ss on "cluster2"
2020-11-02T14:21:19.7152967Z STEP: Executing "dig +short web-0.cluster1.nginx-ss.e2e-tests-discovery-f24mc.svc.clusterset.local" to verify IPs [10.1.224.4] for pod "web-0.cluster1.nginx-ss" "are" discoverable
2020-11-02T14:21:19.7157092Z Nov  2 14:21:09.325: INFO: ExecWithOptions {Command:[dig +short web-0.cluster1.nginx-ss.e2e-tests-discovery-f24mc.svc.clusterset.local] Namespace:e2e-tests-discovery-f24mc PodName:netshootn9md2-6757486d56-8xt9l ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
2020-11-02T14:21:19.7159581Z STEP: Validating that dig result are "10.1.224.4"
2020-11-02T14:21:19.7161872Z STEP: Executing "dig +short web-0.cluster2.nginx-ss.e2e-tests-discovery-f24mc.svc.clusterset.local" to verify IPs [10.2.192.2] for pod "web-0.cluster2.nginx-ss" "are" discoverable
2020-11-02T14:21:19.7166406Z Nov  2 14:21:09.402: INFO: ExecWithOptions {Command:[dig +short web-0.cluster2.nginx-ss.e2e-tests-discovery-f24mc.svc.clusterset.local] Namespace:e2e-tests-discovery-f24mc PodName:netshootn9md2-6757486d56-8xt9l ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
2020-11-02T14:21:19.7168968Z STEP: Validating that dig result are "10.2.192.2"
2020-11-02T14:21:19.7170140Z STEP: Deleting serviceExport nginx-ss.e2e-tests-discovery-f24mc on "cluster2"
2020-11-02T14:21:19.7172707Z STEP: Executing "dig +short web-0.cluster1.nginx-ss.e2e-tests-discovery-f24mc.svc.clusterset.local" to verify IPs [10.1.224.4] for pod "web-0.cluster1.nginx-ss" "are" discoverable
2020-11-02T14:21:19.7176628Z Nov  2 14:21:14.506: INFO: ExecWithOptions {Command:[dig +short web-0.cluster1.nginx-ss.e2e-tests-discovery-f24mc.svc.clusterset.local] Namespace:e2e-tests-discovery-f24mc PodName:netshootn9md2-6757486d56-8xt9l ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
2020-11-02T14:21:19.7179104Z STEP: Validating that dig result are "10.1.224.4"
2020-11-02T14:21:19.7181248Z STEP: Executing "dig +short web-0.cluster2.nginx-ss.e2e-tests-discovery-f24mc.svc.clusterset.local" to verify IPs [10.2.192.2] for pod "web-0.cluster2.nginx-ss" "are not" discoverable
2020-11-02T14:21:19.7185578Z Nov  2 14:21:14.580: INFO: ExecWithOptions {Command:[dig +short web-0.cluster2.nginx-ss.e2e-tests-discovery-f24mc.svc.clusterset.local] Namespace:e2e-tests-discovery-f24mc PodName:netshootn9md2-6757486d56-8xt9l ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
2020-11-02T14:21:19.7187964Z STEP: Validating that dig result are not ""
2020-11-02T14:21:19.7188895Z STEP: Deleting serviceExport nginx-ss.e2e-tests-discovery-f24mc on "cluster1"
2020-11-02T14:21:19.7190074Z STEP: Retrieving EndpointSlices for nginx-ss on "cluster2"
2020-11-02T14:21:19.7190983Z STEP: Deleting namespace "e2e-tests-discovery-f24mc" on cluster "cluster1"
2020-11-02T14:21:19.7205398Z STEP: Deleting namespace "e2e-tests-discovery-f24mc" on cluster "cluster2"
2020-11-02T14:21:19.7205890Z
2020-11-02T14:21:19.7206024Z
2020-11-02T14:21:19.7206885Z [discovery] Test Stateful Sets Discovery Across Clusters when a pod tries to resolve a podname from stateful set in a remote cluster
2020-11-02T14:21:19.7207766Z   should resolve the pod IP from the remote cluster
2020-11-02T14:21:19.7209073Z   /go/src/github.com/submariner-io/lighthouse/test/e2e/discovery/statefulsets.go:20
2020-11-02T14:21:19.7210392Z STEP: Creating namespace objects with basename "discovery"
2020-11-02T14:21:19.7369338Z STEP: Generated namespace "e2e-tests-discovery-c7xss" in cluster "cluster1" to execute the tests in
2020-11-02T14:21:19.7370456Z STEP: Creating namespace "e2e-tests-discovery-c7xss" in cluster "cluster2"
2020-11-02T14:21:19.7636384Z STEP: Creating an Nginx Stateful Set on "cluster2"
2020-11-02T14:21:24.8019316Z STEP: Creating a Nginx Headless Service on "cluster2"
2020-11-02T14:21:24.8357949Z STEP: Creating serviceExport nginx-ss.e2e-tests-discovery-c7xss on "cluster2"
2020-11-02T14:21:24.8639672Z STEP: Retrieving ServiceExport nginx-ss.e2e-tests-discovery-c7xss on "cluster2"
2020-11-02T14:21:29.8723066Z STEP: Creating a Netshoot Deployment on "cluster1"
2020-11-02T14:21:34.8818861Z STEP: Retrieving EndpointSlices for nginx-ss on "cluster2"
2020-11-02T14:21:34.8840351Z STEP: Executing "dig +short web-0.cluster2.nginx-ss.e2e-tests-discovery-c7xss.svc.clusterset.local" to verify IPs [10.2.128.4] for pod "web-0.cluster2.nginx-ss" "are" discoverable
2020-11-02T14:21:34.8845074Z Nov  2 14:21:34.883: INFO: ExecWithOptions {Command:[dig +short web-0.cluster2.nginx-ss.e2e-tests-discovery-c7xss.svc.clusterset.local] Namespace:e2e-tests-discovery-c7xss PodName:netshootlc57r-6757486d56-khfr8 ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
2020-11-02T14:21:34.9691005Z STEP: Validating that dig result are "10.2.128.4"
2020-11-02T14:21:34.9696022Z STEP: Deleting serviceExport nginx-ss.e2e-tests-discovery-c7xss on "cluster2"
2020-11-02T14:21:40.2959336Z STEP: Retrieving EndpointSlices for nginx-ss on "cluster2"
2020-11-02T14:21:40.2961721Z STEP: Executing "dig +short web-0.cluster2.nginx-ss.e2e-tests-discovery-c7xss.svc.clusterset.local" to verify IPs [10.2.128.4] for pod "web-0.cluster2.nginx-ss" "are not" discoverable
2020-11-02T14:21:40.2966191Z Nov  2 14:21:39.981: INFO: ExecWithOptions {Command:[dig +short web-0.cluster2.nginx-ss.e2e-tests-discovery-c7xss.svc.clusterset.local] Namespace:e2e-tests-discovery-c7xss PodName:netshootlc57r-6757486d56-khfr8 ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
2020-11-02T14:21:40.2969165Z STEP: Validating that dig result are not ""
2020-11-02T14:21:40.2970023Z STEP: Deleting namespace "e2e-tests-discovery-c7xss" on cluster "cluster1"
2020-11-02T14:21:40.2971393Z STEP: Deleting namespace "e2e-tests-discovery-c7xss" on cluster "cluster2"
2020-11-02T14:21:40.2972057Z •
2020-11-02T14:21:40.2972453Z ------------------------------
2020-11-02T14:21:40.2973057Z STEP: Creating namespace objects with basename "discovery"
2020-11-02T14:21:40.2974077Z STEP: Generated namespace "e2e-tests-discovery-c7xss" in cluster "cluster1" to execute the tests in
2020-11-02T14:21:40.2975221Z STEP: Creating namespace "e2e-tests-discovery-c7xss" in cluster "cluster2"
2020-11-02T14:21:40.2976268Z STEP: Creating an Nginx Stateful Set on "cluster2"
2020-11-02T14:21:40.2977018Z STEP: Creating a Nginx Headless Service on "cluster2"
2020-11-02T14:21:40.2978347Z STEP: Creating serviceExport nginx-ss.e2e-tests-discovery-c7xss on "cluster2"
2020-11-02T14:21:40.2979582Z STEP: Retrieving ServiceExport nginx-ss.e2e-tests-discovery-c7xss on "cluster2"
2020-11-02T14:21:40.2980537Z STEP: Creating a Netshoot Deployment on "cluster1"
2020-11-02T14:21:40.2981303Z STEP: Retrieving EndpointSlices for nginx-ss on "cluster2"
2020-11-02T14:21:40.2983672Z STEP: Executing "dig +short web-0.cluster2.nginx-ss.e2e-tests-discovery-c7xss.svc.clusterset.local" to verify IPs [10.2.128.4] for pod "web-0.cluster2.nginx-ss" "are" discoverable
2020-11-02T14:21:40.2987697Z Nov  2 14:21:34.883: INFO: ExecWithOptions {Command:[dig +short web-0.cluster2.nginx-ss.e2e-tests-discovery-c7xss.svc.clusterset.local] Namespace:e2e-tests-discovery-c7xss PodName:netshootlc57r-6757486d56-khfr8 ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
2020-11-02T14:21:40.2990181Z STEP: Validating that dig result are "10.2.128.4"
2020-11-02T14:21:40.2991132Z STEP: Deleting serviceExport nginx-ss.e2e-tests-discovery-c7xss on "cluster2"
2020-11-02T14:21:40.2992707Z STEP: Retrieving EndpointSlices for nginx-ss on "cluster2"
2020-11-02T14:21:40.2994776Z STEP: Executing "dig +short web-0.cluster2.nginx-ss.e2e-tests-discovery-c7xss.svc.clusterset.local" to verify IPs [10.2.128.4] for pod "web-0.cluster2.nginx-ss" "are not" discoverable
2020-11-02T14:21:40.2999740Z Nov  2 14:21:39.981: INFO: ExecWithOptions {Command:[dig +short web-0.cluster2.nginx-ss.e2e-tests-discovery-c7xss.svc.clusterset.local] Namespace:e2e-tests-discovery-c7xss PodName:netshootlc57r-6757486d56-khfr8 ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
2020-11-02T14:21:40.3002320Z STEP: Validating that dig result are not ""
2020-11-02T14:21:40.3003279Z STEP: Deleting namespace "e2e-tests-discovery-c7xss" on cluster "cluster1"
2020-11-02T14:21:40.3004328Z STEP: Deleting namespace "e2e-tests-discovery-c7xss" on cluster "cluster2"
2020-11-02T14:21:40.3004793Z
2020-11-02T14:21:40.3004931Z
2020-11-02T14:21:40.3005665Z [discovery] Test Service Discovery Across Clusters when service export is created before the service
2020-11-02T14:21:40.3006689Z   should resolve the service
2020-11-02T14:21:40.3007509Z   /go/src/github.com/submariner-io/lighthouse/test/e2e/discovery/service_discovery.go:38
2020-11-02T14:21:40.3008408Z STEP: Creating namespace objects with basename "discovery"
2020-11-02T14:21:40.3009406Z STEP: Generated namespace "e2e-tests-discovery-nh4mt" in cluster "cluster1" to execute the tests in
2020-11-02T14:21:40.3010502Z STEP: Creating namespace "e2e-tests-discovery-nh4mt" in cluster "cluster2"
2020-11-02T14:21:40.3011371Z STEP: Creating an Nginx ServiceExport on "cluster2"
2020-11-02T14:21:40.3012405Z STEP: Creating serviceExport nginx-demo.e2e-tests-discovery-nh4mt on "cluster2"
2020-11-02T14:21:40.3013552Z STEP: Creating an Nginx Deployment on on "cluster2"
2020-11-02T14:21:45.1583324Z STEP: Creating a Nginx Service on "cluster2"
2020-11-02T14:21:45.1695164Z STEP: Creating a Netshoot Deployment on "cluster1"
2020-11-02T14:21:50.2070911Z STEP: Retrieving service nginx-demo.e2e-tests-discovery-nh4mt on "cluster2"
2020-11-02T14:21:50.2092576Z STEP: Retrieving ServiceImport for nginx-demo-e2e-tests-discovery-nh4mt- on "cluster1"
2020-11-02T14:21:50.2121957Z STEP: Executing "dig +short nginx-demo.e2e-tests-discovery-nh4mt.svc.clusterset.local" to verify IP "100.2.19.88" for service "nginx-demo" "is" discoverable
2020-11-02T14:21:50.2125579Z Nov  2 14:21:50.211: INFO: ExecWithOptions {Command:[dig +short nginx-demo.e2e-tests-discovery-nh4mt.svc.clusterset.local] Namespace:e2e-tests-discovery-nh4mt PodName:netshootdmsx7-6757486d56-vzgqm ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
2020-11-02T14:21:50.3181224Z STEP: Validating that dig result is "100.2.19.88"
2020-11-02T14:21:50.3185006Z STEP: Deleting serviceExport nginx-demo.e2e-tests-discovery-nh4mt on "cluster2"
2020-11-02T14:21:55.3373557Z STEP: Executing "dig +short nginx-demo.e2e-tests-discovery-nh4mt.svc.clusterset.local" to verify IP "100.2.19.88" for service "nginx-demo" "is not" discoverable
2020-11-02T14:21:55.3376898Z Nov  2 14:21:55.336: INFO: ExecWithOptions {Command:[dig +short nginx-demo.e2e-tests-discovery-nh4mt.svc.clusterset.local] Namespace:e2e-tests-discovery-nh4mt PodName:netshootdmsx7-6757486d56-vzgqm ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
2020-11-02T14:21:55.4099407Z STEP: Validating that dig result is not ""
2020-11-02T14:21:55.4107988Z STEP: Deleting namespace "e2e-tests-discovery-nh4mt" on cluster "cluster1"
2020-11-02T14:21:55.4174059Z STEP: Deleting namespace "e2e-tests-discovery-nh4mt" on cluster "cluster2"
2020-11-02T14:21:55.4224402Z •
2020-11-02T14:21:55.4230304Z ------------------------------
2020-11-02T14:21:55.4231137Z STEP: Creating namespace objects with basename "discovery"
2020-11-02T14:21:55.4232592Z STEP: Generated namespace "e2e-tests-discovery-nh4mt" in cluster "cluster1" to execute the tests in
2020-11-02T14:21:55.4233694Z STEP: Creating namespace "e2e-tests-discovery-nh4mt" in cluster "cluster2"
2020-11-02T14:21:55.4234848Z STEP: Creating an Nginx ServiceExport on "cluster2"
2020-11-02T14:21:55.4236133Z STEP: Creating serviceExport nginx-demo.e2e-tests-discovery-nh4mt on "cluster2"
2020-11-02T14:21:55.4237507Z STEP: Creating an Nginx Deployment on on "cluster2"
2020-11-02T14:21:55.4238190Z STEP: Creating a Nginx Service on "cluster2"
2020-11-02T14:21:55.4238885Z STEP: Creating a Netshoot Deployment on "cluster1"
2020-11-02T14:21:55.4240147Z STEP: Retrieving service nginx-demo.e2e-tests-discovery-nh4mt on "cluster2"
2020-11-02T14:21:55.4242412Z STEP: Retrieving ServiceImport for nginx-demo-e2e-tests-discovery-nh4mt- on "cluster1"
2020-11-02T14:21:55.4244370Z STEP: Executing "dig +short nginx-demo.e2e-tests-discovery-nh4mt.svc.clusterset.local" to verify IP "100.2.19.88" for service "nginx-demo" "is" discoverable
2020-11-02T14:21:55.4247804Z Nov  2 14:21:50.211: INFO: ExecWithOptions {Command:[dig +short nginx-demo.e2e-tests-discovery-nh4mt.svc.clusterset.local] Namespace:e2e-tests-discovery-nh4mt PodName:netshootdmsx7-6757486d56-vzgqm ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
2020-11-02T14:21:55.4250037Z STEP: Validating that dig result is "100.2.19.88"
2020-11-02T14:21:55.4250974Z STEP: Deleting serviceExport nginx-demo.e2e-tests-discovery-nh4mt on "cluster2"
2020-11-02T14:21:55.4253564Z STEP: Executing "dig +short nginx-demo.e2e-tests-discovery-nh4mt.svc.clusterset.local" to verify IP "100.2.19.88" for service "nginx-demo" "is not" discoverable
2020-11-02T14:21:55.4257095Z Nov  2 14:21:55.336: INFO: ExecWithOptions {Command:[dig +short nginx-demo.e2e-tests-discovery-nh4mt.svc.clusterset.local] Namespace:e2e-tests-discovery-nh4mt PodName:netshootdmsx7-6757486d56-vzgqm ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
2020-11-02T14:21:55.4259955Z STEP: Validating that dig result is not ""
2020-11-02T14:21:55.4260821Z STEP: Deleting namespace "e2e-tests-discovery-nh4mt" on cluster "cluster1"
2020-11-02T14:21:55.4261872Z STEP: Deleting namespace "e2e-tests-discovery-nh4mt" on cluster "cluster2"
2020-11-02T14:21:55.4262371Z
2020-11-02T14:21:55.4262519Z
2020-11-02T14:21:55.4263468Z [discovery] Test Service Discovery Across Clusters when a pod tries to resolve a service in a remote cluster
2020-11-02T14:21:55.4264577Z   should be able to discover the remote service successfully
2020-11-02T14:21:55.4265701Z   /go/src/github.com/submariner-io/lighthouse/test/e2e/discovery/service_discovery.go:26
2020-11-02T14:21:55.4266599Z STEP: Creating namespace objects with basename "discovery"
2020-11-02T14:21:55.4397812Z STEP: Generated namespace "e2e-tests-discovery-tnj5k" in cluster "cluster1" to execute the tests in
2020-11-02T14:21:55.4398946Z STEP: Creating namespace "e2e-tests-discovery-tnj5k" in cluster "cluster2"
2020-11-02T14:21:55.4614619Z STEP: Creating an Nginx Deployment on on "cluster2"
2020-11-02T14:22:00.5370382Z STEP: Creating a Nginx Service on "cluster2"
2020-11-02T14:22:00.5981189Z STEP: Creating serviceExport nginx-demo.e2e-tests-discovery-tnj5k on "cluster2"
2020-11-02T14:22:00.6876566Z STEP: Retrieving ServiceExport nginx-demo.e2e-tests-discovery-tnj5k on "cluster2"
2020-11-02T14:22:05.7213873Z STEP: Creating a Netshoot Deployment on "cluster1"
2020-11-02T14:22:10.7321254Z STEP: Retrieving service nginx-demo.e2e-tests-discovery-tnj5k on "cluster2"
2020-11-02T14:22:10.7341762Z STEP: Retrieving ServiceImport for nginx-demo-e2e-tests-discovery-tnj5k- on "cluster1"
2020-11-02T14:22:10.7361905Z STEP: Executing "dig +short nginx-demo.e2e-tests-discovery-tnj5k.svc.clusterset.local" to verify IP "100.2.73.183" for service "nginx-demo" "is" discoverable
2020-11-02T14:22:10.7370960Z Nov  2 14:22:10.735: INFO: ExecWithOptions {Command:[dig +short nginx-demo.e2e-tests-discovery-tnj5k.svc.clusterset.local] Namespace:e2e-tests-discovery-tnj5k PodName:netshootbk7vw-6757486d56-qcbgc ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
2020-11-02T14:22:10.8126895Z STEP: Validating that dig result is "100.2.73.183"
2020-11-02T14:22:10.8127999Z STEP: Deleting serviceExport nginx-demo.e2e-tests-discovery-tnj5k on "cluster2"
2020-11-02T14:22:16.3062027Z STEP: Deleting service "nginx-demo" on "cluster2"
2020-11-02T14:22:16.3064965Z STEP: Executing "dig +short nginx-demo.e2e-tests-discovery-tnj5k.svc.clusterset.local" to verify IP "100.2.73.183" for service "nginx-demo" "is not" discoverable
2020-11-02T14:22:16.3068920Z Nov  2 14:22:15.872: INFO: ExecWithOptions {Command:[dig +short nginx-demo.e2e-tests-discovery-tnj5k.svc.clusterset.local] Namespace:e2e-tests-discovery-tnj5k PodName:netshootbk7vw-6757486d56-qcbgc ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
2020-11-02T14:22:16.3073034Z STEP: Validating that dig result is not ""
2020-11-02T14:22:16.3074729Z STEP: Deleting namespace "e2e-tests-discovery-tnj5k" on cluster "cluster1"
2020-11-02T14:22:16.3076153Z STEP: Deleting namespace "e2e-tests-discovery-tnj5k" on cluster "cluster2"
2020-11-02T14:22:16.3077251Z •
2020-11-02T14:22:16.3079356Z ------------------------------
2020-11-02T14:22:16.3080161Z STEP: Creating namespace objects with basename "discovery"
2020-11-02T14:22:16.3081410Z STEP: Generated namespace "e2e-tests-discovery-tnj5k" in cluster "cluster1" to execute the tests in
2020-11-02T14:22:16.3082646Z STEP: Creating namespace "e2e-tests-discovery-tnj5k" in cluster "cluster2"
2020-11-02T14:22:16.3083601Z STEP: Creating an Nginx Deployment on on "cluster2"
2020-11-02T14:22:16.3084682Z STEP: Creating a Nginx Service on "cluster2"
2020-11-02T14:22:16.3085948Z STEP: Creating serviceExport nginx-demo.e2e-tests-discovery-tnj5k on "cluster2"
2020-11-02T14:22:16.3087652Z STEP: Retrieving ServiceExport nginx-demo.e2e-tests-discovery-tnj5k on "cluster2"
2020-11-02T14:22:16.3088716Z STEP: Creating a Netshoot Deployment on "cluster1"
2020-11-02T14:22:16.3090139Z STEP: Retrieving service nginx-demo.e2e-tests-discovery-tnj5k on "cluster2"
2020-11-02T14:22:16.3091554Z STEP: Retrieving ServiceImport for nginx-demo-e2e-tests-discovery-tnj5k- on "cluster1"
2020-11-02T14:22:16.3093873Z STEP: Executing "dig +short nginx-demo.e2e-tests-discovery-tnj5k.svc.clusterset.local" to verify IP "100.2.73.183" for service "nginx-demo" "is" discoverable
2020-11-02T14:22:16.3098003Z Nov  2 14:22:10.735: INFO: ExecWithOptions {Command:[dig +short nginx-demo.e2e-tests-discovery-tnj5k.svc.clusterset.local] Namespace:e2e-tests-discovery-tnj5k PodName:netshootbk7vw-6757486d56-qcbgc ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
2020-11-02T14:22:16.3100533Z STEP: Validating that dig result is "100.2.73.183"
2020-11-02T14:22:16.3101564Z STEP: Deleting serviceExport nginx-demo.e2e-tests-discovery-tnj5k on "cluster2"
2020-11-02T14:22:16.3102615Z STEP: Deleting service "nginx-demo" on "cluster2"
2020-11-02T14:22:16.3104295Z STEP: Executing "dig +short nginx-demo.e2e-tests-discovery-tnj5k.svc.clusterset.local" to verify IP "100.2.73.183" for service "nginx-demo" "is not" discoverable
2020-11-02T14:22:16.3107736Z Nov  2 14:22:15.872: INFO: ExecWithOptions {Command:[dig +short nginx-demo.e2e-tests-discovery-tnj5k.svc.clusterset.local] Namespace:e2e-tests-discovery-tnj5k PodName:netshootbk7vw-6757486d56-qcbgc ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
2020-11-02T14:22:16.3110066Z STEP: Validating that dig result is not ""
2020-11-02T14:22:16.3110936Z STEP: Deleting namespace "e2e-tests-discovery-tnj5k" on cluster "cluster1"
2020-11-02T14:22:16.3112493Z STEP: Deleting namespace "e2e-tests-discovery-tnj5k" on cluster "cluster2"
2020-11-02T14:22:16.3112980Z
2020-11-02T14:22:16.3113148Z
2020-11-02T14:22:16.3113891Z [discovery] Test Service Discovery Across Clusters when there are no active pods for a service
2020-11-02T14:22:16.3114660Z   should not resolve the service
2020-11-02T14:22:16.3115512Z   /go/src/github.com/submariner-io/lighthouse/test/e2e/discovery/service_discovery.go:44
2020-11-02T14:22:16.3116580Z STEP: Creating namespace objects with basename "discovery"
2020-11-02T14:22:16.3117671Z STEP: Generated namespace "e2e-tests-discovery-2b27h" in cluster "cluster1" to execute the tests in
2020-11-02T14:22:16.3118801Z STEP: Creating namespace "e2e-tests-discovery-2b27h" in cluster "cluster2"
2020-11-02T14:22:16.3120165Z STEP: Creating an Nginx Deployment on on "cluster2"
2020-11-02T14:22:21.1617167Z STEP: Creating a Nginx Service on "cluster2"
2020-11-02T14:22:21.1836539Z STEP: Creating serviceExport nginx-demo.e2e-tests-discovery-2b27h on "cluster2"
2020-11-02T14:22:21.2612212Z STEP: Retrieving ServiceExport nginx-demo.e2e-tests-discovery-2b27h on "cluster2"
2020-11-02T14:22:26.3850701Z STEP: Creating a Netshoot Deployment on "cluster1"
2020-11-02T14:22:31.3954055Z STEP: Retrieving service nginx-demo.e2e-tests-discovery-2b27h on "cluster2"
2020-11-02T14:22:31.4113639Z STEP: Retrieving ServiceImport for nginx-demo-e2e-tests-discovery-2b27h- on "cluster1"
2020-11-02T14:22:31.4116605Z STEP: Executing "dig +short nginx-demo.e2e-tests-discovery-2b27h.svc.clusterset.local" to verify IP "100.2.2.27" for service "nginx-demo" "is" discoverable
2020-11-02T14:22:31.4121558Z Nov  2 14:22:31.405: INFO: ExecWithOptions {Command:[dig +short nginx-demo.e2e-tests-discovery-2b27h.svc.clusterset.local] Namespace:e2e-tests-discovery-2b27h PodName:netshootqlfxp-6757486d56-qqk8p ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
2020-11-02T14:22:31.5128134Z STEP: Validating that dig result is "100.2.2.27"
2020-11-02T14:22:31.5132195Z STEP: Setting Nginx deployment replicas to 0 in cluster "cluster2"
2020-11-02T14:22:31.5191495Z STEP: Executing "dig +short nginx-demo.e2e-tests-discovery-2b27h.svc.clusterset.local" to verify IP "100.2.2.27" for service "nginx-demo" "is not" discoverable
2020-11-02T14:22:31.5199808Z Nov  2 14:22:31.518: INFO: ExecWithOptions {Command:[dig +short nginx-demo.e2e-tests-discovery-2b27h.svc.clusterset.local] Namespace:e2e-tests-discovery-2b27h PodName:netshootqlfxp-6757486d56-qqk8p ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
2020-11-02T14:22:31.6742712Z STEP: Validating that dig result is not ""
2020-11-02T14:22:31.6743579Z STEP: Setting Nginx deployment replicas to 2 in cluster "cluster2"
2020-11-02T14:22:31.6993126Z STEP: Executing "dig +short nginx-demo.e2e-tests-discovery-2b27h.svc.clusterset.local" to verify IP "100.2.2.27" for service "nginx-demo" "is" discoverable
2020-11-02T14:22:31.6996810Z Nov  2 14:22:31.698: INFO: ExecWithOptions {Command:[dig +short nginx-demo.e2e-tests-discovery-2b27h.svc.clusterset.local] Namespace:e2e-tests-discovery-2b27h PodName:netshootqlfxp-6757486d56-qqk8p ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
2020-11-02T14:22:31.8711044Z STEP: Validating that dig result is ""
2020-11-02T14:22:36.8715154Z Nov  2 14:22:36.870: INFO: ExecWithOptions {Command:[dig +short nginx-demo.e2e-tests-discovery-2b27h.svc.clusterset.local] Namespace:e2e-tests-discovery-2b27h PodName:netshootqlfxp-6757486d56-qqk8p ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
2020-11-02T14:22:36.9568455Z STEP: Validating that dig result is "100.2.2.27"
2020-11-02T14:22:36.9572218Z STEP: Deleting namespace "e2e-tests-discovery-2b27h" on cluster "cluster1"
2020-11-02T14:22:36.9621678Z STEP: Deleting namespace "e2e-tests-discovery-2b27h" on cluster "cluster2"
2020-11-02T14:22:36.9671402Z •
2020-11-02T14:22:36.9671893Z ------------------------------
2020-11-02T14:22:36.9672527Z STEP: Creating namespace objects with basename "discovery"
2020-11-02T14:22:36.9673578Z STEP: Generated namespace "e2e-tests-discovery-2b27h" in cluster "cluster1" to execute the tests in
2020-11-02T14:22:36.9675041Z STEP: Creating namespace "e2e-tests-discovery-2b27h" in cluster "cluster2"
2020-11-02T14:22:36.9676001Z STEP: Creating an Nginx Deployment on on "cluster2"
2020-11-02T14:22:36.9676672Z STEP: Creating a Nginx Service on "cluster2"
2020-11-02T14:22:36.9677703Z STEP: Creating serviceExport nginx-demo.e2e-tests-discovery-2b27h on "cluster2"
2020-11-02T14:22:36.9679239Z STEP: Retrieving ServiceExport nginx-demo.e2e-tests-discovery-2b27h on "cluster2"
2020-11-02T14:22:36.9680348Z STEP: Creating a Netshoot Deployment on "cluster1"
2020-11-02T14:22:36.9681381Z STEP: Retrieving service nginx-demo.e2e-tests-discovery-2b27h on "cluster2"
2020-11-02T14:22:36.9682780Z STEP: Retrieving ServiceImport for nginx-demo-e2e-tests-discovery-2b27h- on "cluster1"
2020-11-02T14:22:36.9685178Z STEP: Executing "dig +short nginx-demo.e2e-tests-discovery-2b27h.svc.clusterset.local" to verify IP "100.2.2.27" for service "nginx-demo" "is" discoverable
2020-11-02T14:22:36.9688584Z Nov  2 14:22:31.405: INFO: ExecWithOptions {Command:[dig +short nginx-demo.e2e-tests-discovery-2b27h.svc.clusterset.local] Namespace:e2e-tests-discovery-2b27h PodName:netshootqlfxp-6757486d56-qqk8p ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
2020-11-02T14:22:36.9690894Z STEP: Validating that dig result is "100.2.2.27"
2020-11-02T14:22:36.9691824Z STEP: Setting Nginx deployment replicas to 0 in cluster "cluster2"
2020-11-02T14:22:36.9693545Z STEP: Executing "dig +short nginx-demo.e2e-tests-discovery-2b27h.svc.clusterset.local" to verify IP "100.2.2.27" for service "nginx-demo" "is not" discoverable
2020-11-02T14:22:36.9696938Z Nov  2 14:22:31.518: INFO: ExecWithOptions {Command:[dig +short nginx-demo.e2e-tests-discovery-2b27h.svc.clusterset.local] Namespace:e2e-tests-discovery-2b27h PodName:netshootqlfxp-6757486d56-qqk8p ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
2020-11-02T14:22:36.9699232Z STEP: Validating that dig result is not ""
2020-11-02T14:22:36.9699975Z STEP: Setting Nginx deployment replicas to 2 in cluster "cluster2"
2020-11-02T14:22:36.9701671Z STEP: Executing "dig +short nginx-demo.e2e-tests-discovery-2b27h.svc.clusterset.local" to verify IP "100.2.2.27" for service "nginx-demo" "is" discoverable
2020-11-02T14:22:36.9705046Z Nov  2 14:22:31.698: INFO: ExecWithOptions {Command:[dig +short nginx-demo.e2e-tests-discovery-2b27h.svc.clusterset.local] Namespace:e2e-tests-discovery-2b27h PodName:netshootqlfxp-6757486d56-qqk8p ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
2020-11-02T14:22:36.9707329Z STEP: Validating that dig result is ""
2020-11-02T14:22:36.9710036Z Nov  2 14:22:36.870: INFO: ExecWithOptions {Command:[dig +short nginx-demo.e2e-tests-discovery-2b27h.svc.clusterset.local] Namespace:e2e-tests-discovery-2b27h PodName:netshootqlfxp-6757486d56-qqk8p ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
2020-11-02T14:22:36.9718254Z STEP: Validating that dig result is "100.2.2.27"
2020-11-02T14:22:36.9719760Z STEP: Deleting namespace "e2e-tests-discovery-2b27h" on cluster "cluster1"
2020-11-02T14:22:36.9723751Z STEP: Deleting namespace "e2e-tests-discovery-2b27h" on cluster "cluster2"
2020-11-02T14:22:36.9724446Z
2020-11-02T14:22:36.9724598Z
2020-11-02T14:22:36.9725576Z [discovery] Test Headless Service Discovery Across Clusters when the number of active pods backing a service changes
2020-11-02T14:22:36.9727181Z   should only resolve the IPs from the active pods
2020-11-02T14:22:36.9728065Z   /go/src/github.com/submariner-io/lighthouse/test/e2e/discovery/headless_services.go:37
2020-11-02T14:22:36.9728989Z STEP: Creating namespace objects with basename "discovery"
2020-11-02T14:22:36.9801182Z STEP: Generated namespace "e2e-tests-discovery-zzgr8" in cluster "cluster1" to execute the tests in
2020-11-02T14:22:36.9802612Z STEP: Creating namespace "e2e-tests-discovery-zzgr8" in cluster "cluster2"
2020-11-02T14:22:37.0040766Z STEP: Creating an Nginx Deployment on on "cluster2"
2020-11-02T14:22:42.0508094Z STEP: Setting Nginx deployment replicas to 3 in cluster "cluster2"
2020-11-02T14:22:42.0814704Z STEP: Creating a Nginx Headless Service on "cluster2"
2020-11-02T14:22:42.1156100Z STEP: Creating serviceExport nginx-headless.e2e-tests-discovery-zzgr8 on "cluster2"
2020-11-02T14:22:42.1412969Z STEP: Retrieving ServiceExport nginx-headless.e2e-tests-discovery-zzgr8 on "cluster2"
2020-11-02T14:22:47.1488016Z STEP: Creating a Netshoot Deployment on "cluster1"
2020-11-02T14:22:52.1611374Z STEP: Retrieving Endpoints for nginx-headless on "cluster2"
2020-11-02T14:22:52.1645160Z STEP: Executing "dig +short nginx-headless.e2e-tests-discovery-zzgr8.svc.clusterset.local" to verify IPs [10.2.128.5 10.2.192.3 10.2.192.4] for service "nginx-headless" "are" discoverable
2020-11-02T14:22:52.1653783Z Nov  2 14:22:52.163: INFO: ExecWithOptions {Command:[dig +short nginx-headless.e2e-tests-discovery-zzgr8.svc.clusterset.local] Namespace:e2e-tests-discovery-zzgr8 PodName:netshootjprxk-6757486d56-jg8xc ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
2020-11-02T14:22:52.2464009Z STEP: Validating that dig result are "10.2.128.5\n10.2.192.3\n10.2.192.4"
2020-11-02T14:22:52.2469347Z STEP: Setting Nginx deployment replicas to 0 in cluster "cluster2"
2020-11-02T14:22:52.2514694Z STEP: Retrieving Endpoints for nginx-headless on "cluster2"
2020-11-02T14:22:57.2610473Z STEP: Executing "dig +short nginx-headless.e2e-tests-discovery-zzgr8.svc.clusterset.local" to verify IPs [] for service "nginx-headless" "are not" discoverable
2020-11-02T14:22:57.2618135Z Nov  2 14:22:57.260: INFO: ExecWithOptions {Command:[dig +short nginx-headless.e2e-tests-discovery-zzgr8.svc.clusterset.local] Namespace:e2e-tests-discovery-zzgr8 PodName:netshootjprxk-6757486d56-jg8xc ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
2020-11-02T14:22:57.3580774Z STEP: Validating that dig result are not ""
2020-11-02T14:22:57.3587188Z STEP: Setting Nginx deployment replicas to 2 in cluster "cluster2"
2020-11-02T14:22:57.3635914Z STEP: Retrieving Endpoints for nginx-headless on "cluster2"
2020-11-02T14:23:02.3728025Z STEP: Executing "dig +short nginx-headless.e2e-tests-discovery-zzgr8.svc.clusterset.local" to verify IPs [10.2.128.4 10.2.192.2] for service "nginx-headless" "are" discoverable
2020-11-02T14:23:02.3732056Z Nov  2 14:23:02.372: INFO: ExecWithOptions {Command:[dig +short nginx-headless.e2e-tests-discovery-zzgr8.svc.clusterset.local] Namespace:e2e-tests-discovery-zzgr8 PodName:netshootjprxk-6757486d56-jg8xc ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
2020-11-02T14:23:02.4473767Z STEP: Validating that dig result are "10.2.128.4\n10.2.192.2"
2020-11-02T14:23:02.4479618Z STEP: Deleting namespace "e2e-tests-discovery-zzgr8" on cluster "cluster1"
2020-11-02T14:23:02.4533782Z STEP: Deleting namespace "e2e-tests-discovery-zzgr8" on cluster "cluster2"
2020-11-02T14:23:02.4593033Z •
2020-11-02T14:23:02.4593652Z ------------------------------
2020-11-02T14:23:02.4594335Z STEP: Creating namespace objects with basename "discovery"
2020-11-02T14:23:02.4595380Z STEP: Generated namespace "e2e-tests-discovery-zzgr8" in cluster "cluster1" to execute the tests in
2020-11-02T14:23:02.4596530Z STEP: Creating namespace "e2e-tests-discovery-zzgr8" in cluster "cluster2"
2020-11-02T14:23:02.4597748Z STEP: Creating an Nginx Deployment on on "cluster2"
2020-11-02T14:23:02.4598530Z STEP: Setting Nginx deployment replicas to 3 in cluster "cluster2"
2020-11-02T14:23:02.4599303Z STEP: Creating a Nginx Headless Service on "cluster2"
2020-11-02T14:23:02.4600691Z STEP: Creating serviceExport nginx-headless.e2e-tests-discovery-zzgr8 on "cluster2"
2020-11-02T14:23:02.4602602Z STEP: Retrieving ServiceExport nginx-headless.e2e-tests-discovery-zzgr8 on "cluster2"
2020-11-02T14:23:02.4603701Z STEP: Creating a Netshoot Deployment on "cluster1"
2020-11-02T14:23:02.4604474Z STEP: Retrieving Endpoints for nginx-headless on "cluster2"
2020-11-02T14:23:02.4606324Z STEP: Executing "dig +short nginx-headless.e2e-tests-discovery-zzgr8.svc.clusterset.local" to verify IPs [10.2.128.5 10.2.192.3 10.2.192.4] for service "nginx-headless" "are" discoverable
2020-11-02T14:23:02.4610134Z Nov  2 14:22:52.163: INFO: ExecWithOptions {Command:[dig +short nginx-headless.e2e-tests-discovery-zzgr8.svc.clusterset.local] Namespace:e2e-tests-discovery-zzgr8 PodName:netshootjprxk-6757486d56-jg8xc ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
2020-11-02T14:23:02.4612660Z STEP: Validating that dig result are "10.2.128.5\n10.2.192.3\n10.2.192.4"
2020-11-02T14:23:02.4613437Z STEP: Setting Nginx deployment replicas to 0 in cluster "cluster2"
2020-11-02T14:23:02.4614274Z STEP: Retrieving Endpoints for nginx-headless on "cluster2"
2020-11-02T14:23:02.4616276Z STEP: Executing "dig +short nginx-headless.e2e-tests-discovery-zzgr8.svc.clusterset.local" to verify IPs [] for service "nginx-headless" "are not" discoverable
2020-11-02T14:23:02.4620045Z Nov  2 14:22:57.260: INFO: ExecWithOptions {Command:[dig +short nginx-headless.e2e-tests-discovery-zzgr8.svc.clusterset.local] Namespace:e2e-tests-discovery-zzgr8 PodName:netshootjprxk-6757486d56-jg8xc ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
2020-11-02T14:23:02.4622390Z STEP: Validating that dig result are not ""
2020-11-02T14:23:02.4623110Z STEP: Setting Nginx deployment replicas to 2 in cluster "cluster2"
2020-11-02T14:23:02.4624110Z STEP: Retrieving Endpoints for nginx-headless on "cluster2"
2020-11-02T14:23:02.4626016Z STEP: Executing "dig +short nginx-headless.e2e-tests-discovery-zzgr8.svc.clusterset.local" to verify IPs [10.2.128.4 10.2.192.2] for service "nginx-headless" "are" discoverable
2020-11-02T14:23:02.4630061Z Nov  2 14:23:02.372: INFO: ExecWithOptions {Command:[dig +short nginx-headless.e2e-tests-discovery-zzgr8.svc.clusterset.local] Namespace:e2e-tests-discovery-zzgr8 PodName:netshootjprxk-6757486d56-jg8xc ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
2020-11-02T14:23:02.4632778Z STEP: Validating that dig result are "10.2.128.4\n10.2.192.2"
2020-11-02T14:23:02.4633666Z STEP: Deleting namespace "e2e-tests-discovery-zzgr8" on cluster "cluster1"
2020-11-02T14:23:02.4634741Z STEP: Deleting namespace "e2e-tests-discovery-zzgr8" on cluster "cluster2"
2020-11-02T14:23:02.4635395Z
2020-11-02T14:23:02.4647279Z
2020-11-02T14:23:02.4648433Z [discovery] Test Headless Service Discovery Across Clusters when a pod tries to resolve a headless service in a remote cluster
2020-11-02T14:23:02.4649555Z   should resolve the backing pod IPs from the remote cluster
2020-11-02T14:23:02.4650771Z   /go/src/github.com/submariner-io/lighthouse/test/e2e/discovery/headless_services.go:17
2020-11-02T14:23:02.4652112Z STEP: Creating namespace objects with basename "discovery"
2020-11-02T14:23:02.4699046Z STEP: Generated namespace "e2e-tests-discovery-kk4mr" in cluster "cluster1" to execute the tests in
2020-11-02T14:23:02.4703317Z STEP: Creating namespace "e2e-tests-discovery-kk4mr" in cluster "cluster2"
2020-11-02T14:23:02.4802470Z STEP: Creating an Nginx Deployment on on "cluster2"
2020-11-02T14:23:07.5611318Z STEP: Creating a Nginx Headless Service on "cluster2"
2020-11-02T14:23:07.5726969Z STEP: Creating serviceExport nginx-headless.e2e-tests-discovery-kk4mr on "cluster2"
2020-11-02T14:23:07.6017042Z STEP: Retrieving ServiceExport nginx-headless.e2e-tests-discovery-kk4mr on "cluster2"
2020-11-02T14:23:12.6166230Z STEP: Creating a Netshoot Deployment on "cluster1"
2020-11-02T14:23:17.6290681Z STEP: Retrieving Endpoints for nginx-headless on "cluster2"
2020-11-02T14:23:17.6321870Z STEP: Executing "dig +short nginx-headless.e2e-tests-discovery-kk4mr.svc.clusterset.local" to verify IPs [10.2.192.3] for service "nginx-headless" "are" discoverable
2020-11-02T14:23:17.6326297Z Nov  2 14:23:17.631: INFO: ExecWithOptions {Command:[dig +short nginx-headless.e2e-tests-discovery-kk4mr.svc.clusterset.local] Namespace:e2e-tests-discovery-kk4mr PodName:netshoot69v6n-6757486d56-9fnhn ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
2020-11-02T14:23:17.7588864Z STEP: Validating that dig result are ""
2020-11-02T14:23:23.3222577Z Nov  2 14:23:22.758: INFO: ExecWithOptions {Command:[dig +short nginx-headless.e2e-tests-discovery-kk4mr.svc.clusterset.local] Namespace:e2e-tests-discovery-kk4mr PodName:netshoot69v6n-6757486d56-9fnhn ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
2020-11-02T14:23:23.3225163Z STEP: Validating that dig result are ""
2020-11-02T14:23:27.7589915Z Nov  2 14:23:27.758: INFO: ExecWithOptions {Command:[dig +short nginx-headless.e2e-tests-discovery-kk4mr.svc.clusterset.local] Namespace:e2e-tests-discovery-kk4mr PodName:netshoot69v6n-6757486d56-9fnhn ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
2020-11-02T14:23:27.8617227Z STEP: Validating that dig result are ""
2020-11-02T14:23:32.7591776Z Nov  2 14:23:32.758: INFO: ExecWithOptions {Command:[dig +short nginx-headless.e2e-tests-discovery-kk4mr.svc.clusterset.local] Namespace:e2e-tests-discovery-kk4mr PodName:netshoot69v6n-6757486d56-9fnhn ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
2020-11-02T14:23:32.8224263Z STEP: Validating that dig result are ""
2020-11-02T14:23:37.7593147Z Nov  2 14:23:37.758: INFO: ExecWithOptions {Command:[dig +short nginx-headless.e2e-tests-discovery-kk4mr.svc.clusterset.local] Namespace:e2e-tests-discovery-kk4mr PodName:netshoot69v6n-6757486d56-9fnhn ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
2020-11-02T14:23:37.8477610Z STEP: Validating that dig result are ""
2020-11-02T14:23:42.7592036Z Nov  2 14:23:42.758: INFO: ExecWithOptions {Command:[dig +short nginx-headless.e2e-tests-discovery-kk4mr.svc.clusterset.local] Namespace:e2e-tests-discovery-kk4mr PodName:netshoot69v6n-6757486d56-9fnhn ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
2020-11-02T14:23:42.8334154Z STEP: Validating that dig result are ""
2020-11-02T14:23:47.7593551Z Nov  2 14:23:47.758: INFO: ExecWithOptions {Command:[dig +short nginx-headless.e2e-tests-discovery-kk4mr.svc.clusterset.local] Namespace:e2e-tests-discovery-kk4mr PodName:netshoot69v6n-6757486d56-9fnhn ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
2020-11-02T14:23:47.8451702Z STEP: Validating that dig result are ""
2020-11-02T14:23:52.7591973Z Nov  2 14:23:52.758: INFO: ExecWithOptions {Command:[dig +short nginx-headless.e2e-tests-discovery-kk4mr.svc.clusterset.local] Namespace:e2e-tests-discovery-kk4mr PodName:netshoot69v6n-6757486d56-9fnhn ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
2020-11-02T14:23:52.8266634Z STEP: Validating that dig result are ""
2020-11-02T14:23:57.7593085Z Nov  2 14:23:57.758: INFO: ExecWithOptions {Command:[dig +short nginx-headless.e2e-tests-discovery-kk4mr.svc.clusterset.local] Namespace:e2e-tests-discovery-kk4mr PodName:netshoot69v6n-6757486d56-9fnhn ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
2020-11-02T14:23:57.8374386Z STEP: Validating that dig result are ""
2020-11-02T14:24:02.7590558Z Nov  2 14:24:02.758: INFO: ExecWithOptions {Command:[dig +short nginx-headless.e2e-tests-discovery-kk4mr.svc.clusterset.local] Namespace:e2e-tests-discovery-kk4mr PodName:netshoot69v6n-6757486d56-9fnhn ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
2020-11-02T14:24:02.8300421Z STEP: Validating that dig result are ""
2020-11-02T14:24:07.7589020Z Nov  2 14:24:07.758: INFO: ExecWithOptions {Command:[dig +short nginx-headless.e2e-tests-discovery-kk4mr.svc.clusterset.local] Namespace:e2e-tests-discovery-kk4mr PodName:netshoot69v6n-6757486d56-9fnhn ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
2020-11-02T14:24:07.8297591Z STEP: Validating that dig result are ""
2020-11-02T14:24:12.7591985Z Nov  2 14:24:12.758: INFO: ExecWithOptions {Command:[dig +short nginx-headless.e2e-tests-discovery-kk4mr.svc.clusterset.local] Namespace:e2e-tests-discovery-kk4mr PodName:netshoot69v6n-6757486d56-9fnhn ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
2020-11-02T14:24:12.8260193Z STEP: Validating that dig result are ""
2020-11-02T14:24:17.7600451Z Nov  2 14:24:17.758: INFO: ExecWithOptions {Command:[dig +short nginx-headless.e2e-tests-discovery-kk4mr.svc.clusterset.local] Namespace:e2e-tests-discovery-kk4mr PodName:netshoot69v6n-6757486d56-9fnhn ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
2020-11-02T14:24:17.8402717Z STEP: Validating that dig result are ""
2020-11-02T14:24:22.7592201Z Nov  2 14:24:22.758: INFO: ExecWithOptions {Command:[dig +short nginx-headless.e2e-tests-discovery-kk4mr.svc.clusterset.local] Namespace:e2e-tests-discovery-kk4mr PodName:netshoot69v6n-6757486d56-9fnhn ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
2020-11-02T14:24:22.8400224Z STEP: Validating that dig result are ""
2020-11-02T14:24:27.7592288Z Nov  2 14:24:27.758: INFO: ExecWithOptions {Command:[dig +short nginx-headless.e2e-tests-discovery-kk4mr.svc.clusterset.local] Namespace:e2e-tests-discovery-kk4mr PodName:netshoot69v6n-6757486d56-9fnhn ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
2020-11-02T14:24:27.8296028Z STEP: Validating that dig result are ""
2020-11-02T14:24:32.7636291Z Nov  2 14:24:32.758: INFO: ExecWithOptions {Command:[dig +short nginx-headless.e2e-tests-discovery-kk4mr.svc.clusterset.local] Namespace:e2e-tests-discovery-kk4mr PodName:netshoot69v6n-6757486d56-9fnhn ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
2020-11-02T14:24:32.8335396Z STEP: Validating that dig result are ""
2020-11-02T14:24:37.7591779Z Nov  2 14:24:37.758: INFO: ExecWithOptions {Command:[dig +short nginx-headless.e2e-tests-discovery-kk4mr.svc.clusterset.local] Namespace:e2e-tests-discovery-kk4mr PodName:netshoot69v6n-6757486d56-9fnhn ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
2020-11-02T14:24:37.8260579Z STEP: Validating that dig result are ""
2020-11-02T14:24:42.7590835Z Nov  2 14:24:42.758: INFO: ExecWithOptions {Command:[dig +short nginx-headless.e2e-tests-discovery-kk4mr.svc.clusterset.local] Namespace:e2e-tests-discovery-kk4mr PodName:netshoot69v6n-6757486d56-9fnhn ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
2020-11-02T14:24:42.8244085Z STEP: Validating that dig result are ""
2020-11-02T14:24:47.7592796Z Nov  2 14:24:47.758: INFO: ExecWithOptions {Command:[dig +short nginx-headless.e2e-tests-discovery-kk4mr.svc.clusterset.local] Namespace:e2e-tests-discovery-kk4mr PodName:netshoot69v6n-6757486d56-9fnhn ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
2020-11-02T14:24:47.8381701Z STEP: Validating that dig result are ""
2020-11-02T14:24:52.7622222Z Nov  2 14:24:52.758: INFO: ExecWithOptions {Command:[dig +short nginx-headless.e2e-tests-discovery-kk4mr.svc.clusterset.local] Namespace:e2e-tests-discovery-kk4mr PodName:netshoot69v6n-6757486d56-9fnhn ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
2020-11-02T14:24:52.8231162Z STEP: Validating that dig result are ""
2020-11-02T14:24:57.7591806Z Nov  2 14:24:57.758: INFO: ExecWithOptions {Command:[dig +short nginx-headless.e2e-tests-discovery-kk4mr.svc.clusterset.local] Namespace:e2e-tests-discovery-kk4mr PodName:netshoot69v6n-6757486d56-9fnhn ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
2020-11-02T14:24:57.8286942Z STEP: Validating that dig result are ""
2020-11-02T14:25:02.7591034Z Nov  2 14:25:02.758: INFO: ExecWithOptions {Command:[dig +short nginx-headless.e2e-tests-discovery-kk4mr.svc.clusterset.local] Namespace:e2e-tests-discovery-kk4mr PodName:netshoot69v6n-6757486d56-9fnhn ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
2020-11-02T14:25:02.8342643Z STEP: Validating that dig result are ""
2020-11-02T14:25:07.7590584Z Nov  2 14:25:07.758: INFO: ExecWithOptions {Command:[dig +short nginx-headless.e2e-tests-discovery-kk4mr.svc.clusterset.local] Namespace:e2e-tests-discovery-kk4mr PodName:netshoot69v6n-6757486d56-9fnhn ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
2020-11-02T14:25:07.8319304Z STEP: Validating that dig result are ""
2020-11-02T14:25:12.7591050Z Nov  2 14:25:12.758: INFO: ExecWithOptions {Command:[dig +short nginx-headless.e2e-tests-discovery-kk4mr.svc.clusterset.local] Namespace:e2e-tests-discovery-kk4mr PodName:netshoot69v6n-6757486d56-9fnhn ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
2020-11-02T14:25:12.8329069Z STEP: Validating that dig result are ""
2020-11-02T14:25:17.7594907Z Nov  2 14:25:17.758: INFO: ExecWithOptions {Command:[dig +short nginx-headless.e2e-tests-discovery-kk4mr.svc.clusterset.local] Namespace:e2e-tests-discovery-kk4mr PodName:netshoot69v6n-6757486d56-9fnhn ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
2020-11-02T14:25:17.8402978Z STEP: Validating that dig result are ""
2020-11-02T14:25:22.7634655Z Nov  2 14:25:22.758: INFO: ExecWithOptions {Command:[dig +short nginx-headless.e2e-tests-discovery-kk4mr.svc.clusterset.local] Namespace:e2e-tests-discovery-kk4mr PodName:netshoot69v6n-6757486d56-9fnhn ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
2020-11-02T14:25:22.8290318Z STEP: Validating that dig result are ""
2020-11-02T14:25:27.7592169Z Nov  2 14:25:27.758: INFO: ExecWithOptions {Command:[dig +short nginx-headless.e2e-tests-discovery-kk4mr.svc.clusterset.local] Namespace:e2e-tests-discovery-kk4mr PodName:netshoot69v6n-6757486d56-9fnhn ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
2020-11-02T14:25:27.8488191Z STEP: Validating that dig result are ""
2020-11-02T14:25:32.7592149Z Nov  2 14:25:32.758: INFO: ExecWithOptions {Command:[dig +short nginx-headless.e2e-tests-discovery-kk4mr.svc.clusterset.local] Namespace:e2e-tests-discovery-kk4mr PodName:netshoot69v6n-6757486d56-9fnhn ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
2020-11-02T14:25:32.8370157Z STEP: Validating that dig result are ""
2020-11-02T14:25:37.7591636Z Nov  2 14:25:37.758: INFO: ExecWithOptions {Command:[dig +short nginx-headless.e2e-tests-discovery-kk4mr.svc.clusterset.local] Namespace:e2e-tests-discovery-kk4mr PodName:netshoot69v6n-6757486d56-9fnhn ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
2020-11-02T14:25:37.8427069Z STEP: Validating that dig result are ""
2020-11-02T14:25:42.7592049Z Nov  2 14:25:42.758: INFO: ExecWithOptions {Command:[dig +short nginx-headless.e2e-tests-discovery-kk4mr.svc.clusterset.local] Namespace:e2e-tests-discovery-kk4mr PodName:netshoot69v6n-6757486d56-9fnhn ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
2020-11-02T14:25:42.8263489Z STEP: Validating that dig result are ""
2020-11-02T14:25:47.7592622Z Nov  2 14:25:47.758: INFO: ExecWithOptions {Command:[dig +short nginx-headless.e2e-tests-discovery-kk4mr.svc.clusterset.local] Namespace:e2e-tests-discovery-kk4mr PodName:netshoot69v6n-6757486d56-9fnhn ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
2020-11-02T14:25:47.8402269Z STEP: Validating that dig result are ""
2020-11-02T14:25:52.7591985Z Nov  2 14:25:52.758: INFO: ExecWithOptions {Command:[dig +short nginx-headless.e2e-tests-discovery-kk4mr.svc.clusterset.local] Namespace:e2e-tests-discovery-kk4mr PodName:netshoot69v6n-6757486d56-9fnhn ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
2020-11-02T14:25:52.8253543Z STEP: Validating that dig result are ""
2020-11-02T14:25:57.7591747Z Nov  2 14:25:57.758: INFO: ExecWithOptions {Command:[dig +short nginx-headless.e2e-tests-discovery-kk4mr.svc.clusterset.local] Namespace:e2e-tests-discovery-kk4mr PodName:netshoot69v6n-6757486d56-9fnhn ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
2020-11-02T14:25:57.8305916Z STEP: Validating that dig result are ""
2020-11-02T14:26:02.7593194Z Nov  2 14:26:02.758: INFO: ExecWithOptions {Command:[dig +short nginx-headless.e2e-tests-discovery-kk4mr.svc.clusterset.local] Namespace:e2e-tests-discovery-kk4mr PodName:netshoot69v6n-6757486d56-9fnhn ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
2020-11-02T14:26:02.8257799Z STEP: Validating that dig result are ""
2020-11-02T14:26:07.7599602Z Nov  2 14:26:07.758: INFO: ExecWithOptions {Command:[dig +short nginx-headless.e2e-tests-discovery-kk4mr.svc.clusterset.local] Namespace:e2e-tests-discovery-kk4mr PodName:netshoot69v6n-6757486d56-9fnhn ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
2020-11-02T14:26:07.8232164Z STEP: Validating that dig result are ""
2020-11-02T14:26:12.7592604Z Nov  2 14:26:12.758: INFO: ExecWithOptions {Command:[dig +short nginx-headless.e2e-tests-discovery-kk4mr.svc.clusterset.local] Namespace:e2e-tests-discovery-kk4mr PodName:netshoot69v6n-6757486d56-9fnhn ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
2020-11-02T14:26:12.8319487Z STEP: Validating that dig result are ""
2020-11-02T14:26:17.7591737Z Nov  2 14:26:17.758: INFO: ExecWithOptions {Command:[dig +short nginx-headless.e2e-tests-discovery-kk4mr.svc.clusterset.local] Namespace:e2e-tests-discovery-kk4mr PodName:netshoot69v6n-6757486d56-9fnhn ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
2020-11-02T14:26:17.8434215Z STEP: Validating that dig result are ""
2020-11-02T14:26:22.7589983Z Nov  2 14:26:22.758: INFO: ExecWithOptions {Command:[dig +short nginx-headless.e2e-tests-discovery-kk4mr.svc.clusterset.local] Namespace:e2e-tests-discovery-kk4mr PodName:netshoot69v6n-6757486d56-9fnhn ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
2020-11-02T14:26:22.8293038Z STEP: Validating that dig result are ""
2020-11-02T14:26:27.7593939Z Nov  2 14:26:27.758: INFO: ExecWithOptions {Command:[dig +short nginx-headless.e2e-tests-discovery-kk4mr.svc.clusterset.local] Namespace:e2e-tests-discovery-kk4mr PodName:netshoot69v6n-6757486d56-9fnhn ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
2020-11-02T14:26:27.8288557Z STEP: Validating that dig result are ""
2020-11-02T14:26:27.8292105Z Nov  2 14:26:27.827: INFO: ExecWithOptions {Command:[dig +short nginx-headless.e2e-tests-discovery-kk4mr.svc.clusterset.local] Namespace:e2e-tests-discovery-kk4mr PodName:netshoot69v6n-6757486d56-9fnhn ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
2020-11-02T14:26:27.9053917Z STEP: Validating that dig result are ""
2020-11-02T14:26:27.9062683Z STEP: Deleting namespace "e2e-tests-discovery-kk4mr" on cluster "cluster1"
2020-11-02T14:26:27.9189000Z STEP: Deleting namespace "e2e-tests-discovery-kk4mr" on cluster "cluster2"
2020-11-02T14:26:27.9334442Z
2020-11-02T14:26:27.9336177Z • Failure [205.469 seconds]
2020-11-02T14:26:27.9355742Z [discovery] Test Headless Service Discovery Across Clusters
2020-11-02T14:26:27.9357035Z /go/src/github.com/submariner-io/lighthouse/test/e2e/discovery/headless_services.go:13
2020-11-02T14:26:27.9358262Z   when a pod tries to resolve a headless service in a remote cluster
2020-11-02T14:26:27.9362482Z   /go/src/github.com/submariner-io/lighthouse/test/e2e/discovery/headless_services.go:16
2020-11-02T14:26:27.9364252Z     should resolve the backing pod IPs from the remote cluster [It]
2020-11-02T14:26:27.9367116Z     /go/src/github.com/submariner-io/lighthouse/test/e2e/discovery/headless_services.go:17
2020-11-02T14:26:27.9368733Z
2020-11-02T14:26:27.9369531Z     Failed to  service IP verification. expected execution result "" to contain "10.2.192.3"
2020-11-02T14:26:27.9370083Z     Unexpected error:
2020-11-02T14:26:27.9371144Z         <*errors.errorString | 0xc0002e2370>: {
2020-11-02T14:26:27.9372370Z             s: "timed out waiting for the condition",
2020-11-02T14:26:27.9373729Z         }
2020-11-02T14:26:27.9374403Z         timed out waiting for the condition
2020-11-02T14:26:27.9375076Z     occurred
2020-11-02T14:26:27.9375261Z
2020-11-02T14:26:27.9376145Z     /go/src/github.com/submariner-io/lighthouse/vendor/github.com/submariner-io/shipyard/test/e2e/framework/framework.go:416
2020-11-02T14:26:27.9378211Z
2020-11-02T14:26:27.9379191Z     Full Stack Trace
2020-11-02T14:26:27.9380178Z     github.com/submariner-io/shipyard/test/e2e/framework.AwaitUntil(0x1534853, 0x18, 0xc0007ff1e0, 0xc0007ff1a8, 0x0, 0xc0004b63c0)
2020-11-02T14:26:27.9381537Z     	/go/src/github.com/submariner-io/lighthouse/vendor/github.com/submariner-io/shipyard/test/e2e/framework/framework.go:416 +0x1c6
2020-11-02T14:26:27.9383354Z     github.com/submariner-io/lighthouse/test/e2e/discovery.verifyHeadlessIpsWithDig(0xc0002a9db0, 0x0, 0xc000412900, 0xc0001cc1c0, 0xc00057b1d0, 0x1, 0x1, 0x1f99e00, 0x1, 0x1, ...)
2020-11-02T14:26:27.9384916Z     	/go/src/github.com/submariner-io/lighthouse/test/e2e/discovery/headless_services.go:159 +0x52f
2020-11-02T14:26:27.9386774Z     github.com/submariner-io/lighthouse/test/e2e/discovery.RunHeadlessDiscoveryTest(0xc00011ed60)
2020-11-02T14:26:27.9388178Z     	/go/src/github.com/submariner-io/lighthouse/test/e2e/discovery/headless_services.go:67 +0x44b
2020-11-02T14:26:27.9389253Z     github.com/submariner-io/lighthouse/test/e2e/discovery.glob..func1.1.1()
2020-11-02T14:26:27.9390977Z     	/go/src/github.com/submariner-io/lighthouse/test/e2e/discovery/headless_services.go:19 +0x6e
2020-11-02T14:26:27.9416838Z     github.com/submariner-io/shipyard/test/e2e.RunE2ETests(0xc000281b00, 0xa0bcee9748)
2020-11-02T14:26:27.9418159Z     	/go/src/github.com/submariner-io/lighthouse/vendor/github.com/submariner-io/shipyard/test/e2e/e2e.go:77 +0x125
2020-11-02T14:26:27.9419314Z     github.com/submariner-io/lighthouse/test/e2e.TestE2E(0xc000281b00)
2020-11-02T14:26:27.9420534Z     	/go/src/github.com/submariner-io/lighthouse/test/e2e/e2e_test.go:11 +0x2b
2020-11-02T14:26:27.9421102Z     testing.tRunner(0xc000281b00, 0x15c1d68)
2020-11-02T14:26:27.9421540Z     	/usr/lib/golang/src/testing/testing.go:1127 +0xef
2020-11-02T14:26:27.9421915Z     created by testing.(*T).Run
2020-11-02T14:26:27.9422291Z     	/usr/lib/golang/src/testing/testing.go:1178 +0x386
2020-11-02T14:26:27.9423149Z ------------------------------
2020-11-02T14:26:27.9424034Z [discovery] Test Stateful Sets Discovery Across Clusters when the number of active pods backing a stateful set changes
2020-11-02T14:26:27.9425089Z   should only resolve the IPs from the active pods
2020-11-02T14:26:27.9425947Z   /go/src/github.com/submariner-io/lighthouse/test/e2e/discovery/statefulsets.go:40
2020-11-02T14:26:27.9426993Z STEP: Creating namespace objects with basename "discovery"
2020-11-02T14:26:27.9485968Z STEP: Generated namespace "e2e-tests-discovery-lkkrq" in cluster "cluster1" to execute the tests in
2020-11-02T14:26:27.9487295Z STEP: Creating namespace "e2e-tests-discovery-lkkrq" in cluster "cluster2"
2020-11-02T14:26:27.9713071Z STEP: Creating an Nginx Stateful Set on on "cluster2"
2020-11-02T14:26:33.0404560Z STEP: Setting Nginx statefulset replicas to 3
2020-11-02T14:26:33.0546925Z STEP: Creating a Nginx Headless Service on "cluster2"
2020-11-02T14:26:33.0760551Z STEP: Creating serviceExport nginx-ss.e2e-tests-discovery-lkkrq on "cluster2"
2020-11-02T14:26:33.1213914Z STEP: Retrieving ServiceExport nginx-ss.e2e-tests-discovery-lkkrq on "cluster2"
2020-11-02T14:26:38.1609346Z STEP: Creating a Netshoot Deployment on "cluster1"
2020-11-02T14:26:43.1742556Z STEP: Retrieving EndpointSlices for nginx-ss on "cluster2"
2020-11-02T14:26:43.1768106Z STEP: Executing "dig +short web-0.cluster2.nginx-ss.e2e-tests-discovery-lkkrq.svc.clusterset.local" to verify IPs [10.2.128.4] for pod "web-0.cluster2.nginx-ss" "are" discoverable
2020-11-02T14:26:43.1772112Z Nov  2 14:26:43.176: INFO: ExecWithOptions {Command:[dig +short web-0.cluster2.nginx-ss.e2e-tests-discovery-lkkrq.svc.clusterset.local] Namespace:e2e-tests-discovery-lkkrq PodName:netshootxxgqg-6757486d56-fm96g ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
2020-11-02T14:26:43.2521819Z STEP: Validating that dig result are "10.2.128.4"
2020-11-02T14:26:43.2524188Z STEP: Executing "dig +short web-1.cluster2.nginx-ss.e2e-tests-discovery-lkkrq.svc.clusterset.local" to verify IPs [10.2.192.2] for pod "web-1.cluster2.nginx-ss" "are" discoverable
2020-11-02T14:26:43.2532191Z Nov  2 14:26:43.251: INFO: ExecWithOptions {Command:[dig +short web-1.cluster2.nginx-ss.e2e-tests-discovery-lkkrq.svc.clusterset.local] Namespace:e2e-tests-discovery-lkkrq PodName:netshootxxgqg-6757486d56-fm96g ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
2020-11-02T14:26:43.3544514Z STEP: Validating that dig result are "10.2.192.2"
2020-11-02T14:26:43.3553444Z STEP: Executing "dig +short web-2.cluster2.nginx-ss.e2e-tests-discovery-lkkrq.svc.clusterset.local" to verify IPs [10.2.192.3] for pod "web-2.cluster2.nginx-ss" "are" discoverable
2020-11-02T14:26:43.3557079Z Nov  2 14:26:43.354: INFO: ExecWithOptions {Command:[dig +short web-2.cluster2.nginx-ss.e2e-tests-discovery-lkkrq.svc.clusterset.local] Namespace:e2e-tests-discovery-lkkrq PodName:netshootxxgqg-6757486d56-fm96g ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
2020-11-02T14:26:43.4237419Z STEP: Validating that dig result are "10.2.192.3"
2020-11-02T14:26:43.4243123Z STEP: Setting Nginx statefulset replicas to 1
2020-11-02T14:26:43.4319296Z STEP: Executing "dig +short web-0.cluster2.nginx-ss.e2e-tests-discovery-lkkrq.svc.clusterset.local" to verify IPs [10.2.128.4] for pod "web-0.cluster2.nginx-ss" "are" discoverable
2020-11-02T14:26:43.4324006Z Nov  2 14:26:43.431: INFO: ExecWithOptions {Command:[dig +short web-0.cluster2.nginx-ss.e2e-tests-discovery-lkkrq.svc.clusterset.local] Namespace:e2e-tests-discovery-lkkrq PodName:netshootxxgqg-6757486d56-fm96g ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
2020-11-02T14:26:43.5307004Z STEP: Validating that dig result are "10.2.128.4"
2020-11-02T14:26:43.5311080Z STEP: Executing "dig +short web-1.cluster2.nginx-ss.e2e-tests-discovery-lkkrq.svc.clusterset.local" to verify IPs [10.2.192.2] for pod "web-1.cluster2.nginx-ss" "are not" discoverable
2020-11-02T14:26:43.5316715Z Nov  2 14:26:43.530: INFO: ExecWithOptions {Command:[dig +short web-1.cluster2.nginx-ss.e2e-tests-discovery-lkkrq.svc.clusterset.local] Namespace:e2e-tests-discovery-lkkrq PodName:netshootxxgqg-6757486d56-fm96g ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
2020-11-02T14:26:43.6633131Z STEP: Validating that dig result are not "10.2.192.2"
2020-11-02T14:26:48.6638401Z Nov  2 14:26:48.662: INFO: ExecWithOptions {Command:[dig +short web-1.cluster2.nginx-ss.e2e-tests-discovery-lkkrq.svc.clusterset.local] Namespace:e2e-tests-discovery-lkkrq PodName:netshootxxgqg-6757486d56-fm96g ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
2020-11-02T14:26:48.7457014Z STEP: Validating that dig result are not ""
2020-11-02T14:26:48.7465230Z STEP: Executing "dig +short web-2.cluster2.nginx-ss.e2e-tests-discovery-lkkrq.svc.clusterset.local" to verify IPs [10.2.192.3] for pod "web-2.cluster2.nginx-ss" "are not" discoverable
2020-11-02T14:26:48.7469205Z Nov  2 14:26:48.745: INFO: ExecWithOptions {Command:[dig +short web-2.cluster2.nginx-ss.e2e-tests-discovery-lkkrq.svc.clusterset.local] Namespace:e2e-tests-discovery-lkkrq PodName:netshootxxgqg-6757486d56-fm96g ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
2020-11-02T14:26:48.8309947Z STEP: Validating that dig result are not ""
2020-11-02T14:26:48.8311006Z STEP: Deleting serviceExport nginx-ss.e2e-tests-discovery-lkkrq on "cluster2"
2020-11-02T14:26:53.8953402Z STEP: Deleting namespace "e2e-tests-discovery-lkkrq" on cluster "cluster1"
2020-11-02T14:26:53.9070069Z STEP: Deleting namespace "e2e-tests-discovery-lkkrq" on cluster "cluster2"
2020-11-02T14:26:53.9141986Z •
2020-11-02T14:26:53.9146892Z ------------------------------
2020-11-02T14:26:53.9157525Z STEP: Creating namespace objects with basename "discovery"
2020-11-02T14:26:53.9158751Z STEP: Generated namespace "e2e-tests-discovery-lkkrq" in cluster "cluster1" to execute the tests in
2020-11-02T14:26:53.9159954Z STEP: Creating namespace "e2e-tests-discovery-lkkrq" in cluster "cluster2"
2020-11-02T14:26:53.9161035Z STEP: Creating an Nginx Stateful Set on on "cluster2"
2020-11-02T14:26:53.9161971Z STEP: Setting Nginx statefulset replicas to 3
2020-11-02T14:26:53.9162722Z STEP: Creating a Nginx Headless Service on "cluster2"
2020-11-02T14:26:53.9163783Z STEP: Creating serviceExport nginx-ss.e2e-tests-discovery-lkkrq on "cluster2"
2020-11-02T14:26:53.9165145Z STEP: Retrieving ServiceExport nginx-ss.e2e-tests-discovery-lkkrq on "cluster2"
2020-11-02T14:26:53.9166210Z STEP: Creating a Netshoot Deployment on "cluster1"
2020-11-02T14:26:53.9167025Z STEP: Retrieving EndpointSlices for nginx-ss on "cluster2"
2020-11-02T14:26:53.9169265Z STEP: Executing "dig +short web-0.cluster2.nginx-ss.e2e-tests-discovery-lkkrq.svc.clusterset.local" to verify IPs [10.2.128.4] for pod "web-0.cluster2.nginx-ss" "are" discoverable
2020-11-02T14:26:53.9173538Z Nov  2 14:26:43.176: INFO: ExecWithOptions {Command:[dig +short web-0.cluster2.nginx-ss.e2e-tests-discovery-lkkrq.svc.clusterset.local] Namespace:e2e-tests-discovery-lkkrq PodName:netshootxxgqg-6757486d56-fm96g ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
2020-11-02T14:26:53.9176182Z STEP: Validating that dig result are "10.2.128.4"
2020-11-02T14:26:53.9178223Z STEP: Executing "dig +short web-1.cluster2.nginx-ss.e2e-tests-discovery-lkkrq.svc.clusterset.local" to verify IPs [10.2.192.2] for pod "web-1.cluster2.nginx-ss" "are" discoverable
2020-11-02T14:26:53.9182697Z Nov  2 14:26:43.251: INFO: ExecWithOptions {Command:[dig +short web-1.cluster2.nginx-ss.e2e-tests-discovery-lkkrq.svc.clusterset.local] Namespace:e2e-tests-discovery-lkkrq PodName:netshootxxgqg-6757486d56-fm96g ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
2020-11-02T14:26:53.9186225Z STEP: Validating that dig result are "10.2.192.2"
2020-11-02T14:26:53.9188278Z STEP: Executing "dig +short web-2.cluster2.nginx-ss.e2e-tests-discovery-lkkrq.svc.clusterset.local" to verify IPs [10.2.192.3] for pod "web-2.cluster2.nginx-ss" "are" discoverable
2020-11-02T14:26:53.9193335Z Nov  2 14:26:43.354: INFO: ExecWithOptions {Command:[dig +short web-2.cluster2.nginx-ss.e2e-tests-discovery-lkkrq.svc.clusterset.local] Namespace:e2e-tests-discovery-lkkrq PodName:netshootxxgqg-6757486d56-fm96g ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
2020-11-02T14:26:53.9196086Z STEP: Validating that dig result are "10.2.192.3"
2020-11-02T14:26:53.9196795Z STEP: Setting Nginx statefulset replicas to 1
2020-11-02T14:26:53.9198863Z STEP: Executing "dig +short web-0.cluster2.nginx-ss.e2e-tests-discovery-lkkrq.svc.clusterset.local" to verify IPs [10.2.128.4] for pod "web-0.cluster2.nginx-ss" "are" discoverable
2020-11-02T14:26:53.9203639Z Nov  2 14:26:43.431: INFO: ExecWithOptions {Command:[dig +short web-0.cluster2.nginx-ss.e2e-tests-discovery-lkkrq.svc.clusterset.local] Namespace:e2e-tests-discovery-lkkrq PodName:netshootxxgqg-6757486d56-fm96g ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
2020-11-02T14:26:53.9206570Z STEP: Validating that dig result are "10.2.128.4"
2020-11-02T14:26:53.9208897Z STEP: Executing "dig +short web-1.cluster2.nginx-ss.e2e-tests-discovery-lkkrq.svc.clusterset.local" to verify IPs [10.2.192.2] for pod "web-1.cluster2.nginx-ss" "are not" discoverable
2020-11-02T14:26:53.9213599Z Nov  2 14:26:43.530: INFO: ExecWithOptions {Command:[dig +short web-1.cluster2.nginx-ss.e2e-tests-discovery-lkkrq.svc.clusterset.local] Namespace:e2e-tests-discovery-lkkrq PodName:netshootxxgqg-6757486d56-fm96g ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
2020-11-02T14:26:53.9216316Z STEP: Validating that dig result are not "10.2.192.2"
2020-11-02T14:26:53.9219190Z Nov  2 14:26:48.662: INFO: ExecWithOptions {Command:[dig +short web-1.cluster2.nginx-ss.e2e-tests-discovery-lkkrq.svc.clusterset.local] Namespace:e2e-tests-discovery-lkkrq PodName:netshootxxgqg-6757486d56-fm96g ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
2020-11-02T14:26:53.9222018Z STEP: Validating that dig result are not ""
2020-11-02T14:26:53.9224080Z STEP: Executing "dig +short web-2.cluster2.nginx-ss.e2e-tests-discovery-lkkrq.svc.clusterset.local" to verify IPs [10.2.192.3] for pod "web-2.cluster2.nginx-ss" "are not" discoverable
2020-11-02T14:26:53.9228500Z Nov  2 14:26:48.745: INFO: ExecWithOptions {Command:[dig +short web-2.cluster2.nginx-ss.e2e-tests-discovery-lkkrq.svc.clusterset.local] Namespace:e2e-tests-discovery-lkkrq PodName:netshootxxgqg-6757486d56-fm96g ContainerName:netshoot Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false}
2020-11-02T14:26:53.9256628Z STEP: Validating that dig result are not ""
2020-11-02T14:26:53.9258164Z STEP: Deleting serviceExport nginx-ss.e2e-tests-discovery-lkkrq on "cluster2"
2020-11-02T14:26:53.9259394Z STEP: Deleting namespace "e2e-tests-discovery-lkkrq" on cluster "cluster1"
2020-11-02T14:26:53.9260493Z STEP: Deleting namespace "e2e-tests-discovery-lkkrq" on cluster "cluster2"
2020-11-02T14:26:53.9261000Z
2020-11-02T14:26:53.9261148Z
2020-11-02T14:26:53.9261916Z JUnit path was configured: /go/src/github.com/submariner-io/lighthouse/output/e2e-junit.xml
2020-11-02T14:26:53.9262484Z
2020-11-02T14:26:53.9263435Z JUnit report was created: /go/src/github.com/submariner-io/lighthouse/output/e2e-junit.xml
2020-11-02T14:26:53.9263938Z
2020-11-02T14:26:53.9264106Z
2020-11-02T14:26:53.9264784Z Summarizing 1 Failure:
2020-11-02T14:26:53.9265048Z
2020-11-02T14:26:53.9266236Z [Fail] [discovery] Test Headless Service Discovery Across Clusters when a pod tries to resolve a headless service in a remote cluster [It] should resolve the backing pod IPs from the remote cluster 
2020-11-02T14:26:53.9267998Z /go/src/github.com/submariner-io/lighthouse/vendor/github.com/submariner-io/shipyard/test/e2e/framework/framework.go:416
2020-11-02T14:26:53.9268616Z
2020-11-02T14:26:53.9269103Z Ran 11 of 11 Specs in 467.538 seconds
2020-11-02T14:26:53.9269836Z FAIL! -- 10 Passed | 1 Failed | 0 Pending | 0 Skipped
2020-11-02T14:26:53.9270617Z --- FAIL: TestE2E (467.54s)
2020-11-02T14:26:53.9270891Z FAIL
2020-11-02T14:26:53.9271143Z exit status 1
2020-11-02T14:26:53.9271977Z FAIL	github.com/submariner-io/lighthouse/test/e2e	467.565s
2020-11-02T14:26:54.0050385Z make: *** [/opt/shipyard/Makefile.inc:40: e2e] Error 1
2020-11-02T14:26:54.0106623Z [lighthouse]$ make e2e -- using=helm
2020-11-02T14:27:03.0708088Z time="2020-11-02T14:27:03Z" level=fatal msg="exit status 2"
2020-11-02T14:27:03.0720240Z make: *** [e2e] Error 1
2020-11-02T14:27:03.0721278Z Makefile.dapper:14: recipe for target 'e2e' failed
2020-11-02T14:27:03.0780387Z ##[error]Process completed with exit code 2.
2020-11-02T14:27:03.0926889Z ##[group]Run df -h
2020-11-02T14:27:03.0927295Z df -h
2020-11-02T14:27:03.0927526Z free -h
2020-11-02T14:27:03.0927802Z make post-mortem
2020-11-02T14:27:03.4859851Z shell: /bin/bash -e {0}
2020-11-02T14:27:03.4860149Z ##[endgroup]
2020-11-02T14:27:03.5006868Z Filesystem      Size  Used Avail Use% Mounted on
2020-11-02T14:27:03.5007247Z udev            3.4G     0  3.4G   0% /dev
2020-11-02T14:27:03.5007525Z tmpfs           696M  1.2M  695M   1% /run
2020-11-02T14:27:03.5007786Z /dev/sda1        84G   76G  7.8G  91% /
2020-11-02T14:27:03.5008240Z tmpfs           3.4G  223M  3.2G   7% /dev/shm
2020-11-02T14:27:03.5008680Z tmpfs           5.0M     0  5.0M   0% /run/lock
2020-11-02T14:27:03.5008973Z tmpfs           3.4G     0  3.4G   0% /sys/fs/cgroup
2020-11-02T14:27:03.5009290Z /dev/sda15      105M  3.6M  101M   4% /boot/efi
2020-11-02T14:27:03.5009574Z /dev/sdb1        14G  4.1G  9.0G  32% /mnt
2020-11-02T14:27:03.5057663Z               total        used        free      shared  buff/cache   available
2020-11-02T14:27:03.5059734Z Mem:           6.8G        3.1G        336M        300M        3.4G        3.4G
2020-11-02T14:27:03.5060277Z Swap:            0B          0B          0B
2020-11-02T14:27:03.5094855Z ./.dapper -m bind make post-mortem
2020-11-02T14:27:05.9036413Z Sending build context to Docker daemon  70.83MB
2020-11-02T14:27:05.9036857Z
2020-11-02T14:27:05.9156735Z Step 1/6 : FROM quay.io/submariner/shipyard-dapper-base:devel
2020-11-02T14:27:05.9157827Z  ---> 45327d47bb4a
2020-11-02T14:27:05.9159211Z Step 2/6 : ENV DAPPER_ENV="REPO TAG QUAY_USERNAME QUAY_PASSWORD GITHUB_SHA BUILD_ARGS CLUSTERS_ARGS DEPLOY_ARGS RELEASE_ARGS"     DAPPER_SOURCE=/go/src/github.com/submariner-io/lighthouse DAPPER_DOCKER_SOCKET=true
2020-11-02T14:27:05.9161111Z  ---> Using cache
2020-11-02T14:27:05.9161762Z  ---> d738e368da0f
2020-11-02T14:27:05.9162289Z Step 3/6 : ENV DAPPER_OUTPUT=${DAPPER_SOURCE}/output
2020-11-02T14:27:05.9166207Z  ---> Using cache
2020-11-02T14:27:05.9167195Z  ---> a08ac595035e
2020-11-02T14:27:05.9168084Z Step 4/6 : WORKDIR ${DAPPER_SOURCE}
2020-11-02T14:27:05.9216010Z  ---> Using cache
2020-11-02T14:27:05.9217255Z  ---> b8ceb069cd5c
2020-11-02T14:27:05.9218093Z Step 5/6 : ENTRYPOINT ["/opt/shipyard/scripts/entry"]
2020-11-02T14:27:05.9251281Z  ---> Using cache
2020-11-02T14:27:05.9252083Z  ---> 43f258a32a70
2020-11-02T14:27:05.9252536Z Step 6/6 : CMD ["sh"]
2020-11-02T14:27:05.9280175Z  ---> Using cache
2020-11-02T14:27:05.9281508Z  ---> c35f75223205
2020-11-02T14:27:06.0381992Z Successfully built c35f75223205
2020-11-02T14:27:06.0444491Z Successfully tagged lighthouse:HEAD
2020-11-02T14:27:06.5353411Z [lighthouse]$ trap chown -R 1001:116 . exit
2020-11-02T14:27:06.5376503Z [lighthouse]$ mkdir -p bin dist output
2020-11-02T14:27:06.5404708Z [lighthouse]$ make post-mortem
2020-11-02T14:27:06.5993622Z fatal: No names found, cannot describe anything.
2020-11-02T14:27:06.6143912Z Makefile:40: warning: overriding recipe for target 'deploy'
2020-11-02T14:27:06.6145008Z /opt/shipyard/Makefile.inc:36: warning: ignoring old recipe for target 'deploy'
2020-11-02T14:27:06.6159826Z /opt/shipyard/scripts/post_mortem.sh
2020-11-02T14:27:06.6221126Z [lighthouse]$ source /opt/shipyard/scripts/lib/utils
2020-11-02T14:27:06.6245790Z [lighthouse]$ . /opt/shipyard/scripts/lib/source_only
2020-11-02T14:27:06.6258143Z [lighthouse]$ script_name=utils
2020-11-02T14:27:06.6269244Z [lighthouse]$ exec_name=post_mortem.sh
2020-11-02T14:27:06.6293362Z [lighthouse]$ declare_kubeconfig
2020-11-02T14:27:06.6305711Z [lighthouse]$ declare_kubeconfig
2020-11-02T14:27:06.6320819Z [lighthouse]$ source /opt/shipyard/scripts/lib/kubecfg
2020-11-02T14:27:06.6334757Z [lighthouse]$ export KUBECONFIG
2020-11-02T14:27:06.6370026Z [lighthouse]$ KUBECONFIG=/go/src/github.com/submariner-io/lighthouse/output/kubeconfigs/kind-config-cluster2:/go/src/github.com/submariner-io/lighthouse/output/kubeconfigs/kind-config-cluster1:
2020-11-02T14:27:06.6386303Z [lighthouse]$ find /go/src/github.com/submariner-io/lighthouse/output/kubeconfigs -type f -printf %p:
2020-11-02T14:27:06.6440237Z [lighthouse]$ kind get clusters
2020-11-02T14:27:07.0607362Z [lighthouse]$ run_sequential cluster1 cluster2 post_analyze
2020-11-02T14:27:07.0619174Z [lighthouse]$ run_sequential cluster1 cluster2 post_analyze
2020-11-02T14:27:07.0634649Z [lighthouse]$ local cmnd=post_analyze
2020-11-02T14:27:07.0651730Z [lighthouse]$ eval echo cluster1 cluster2
2020-11-02T14:27:07.0666245Z [lighthouse]$ [cluster1] post_analyze
2020-11-02T14:27:07.0680914Z [lighthouse]$ [cluster1] sed s/^/[cluster1] /
2020-11-02T14:27:07.0702138Z [lighthouse]$ [cluster1] post_analyze
2020-11-02T14:27:07.0726387Z [lighthouse]$ [cluster1] kubectl get all --all-namespaces
2020-11-02T14:27:07.0739566Z [lighthouse]$ [cluster1] kubectl get all --all-namespaces
2020-11-02T14:27:07.0752790Z [lighthouse]$ [cluster1] command kubectl --context=cluster1 get all --all-namespaces
2020-11-02T14:27:07.0766659Z [lighthouse]$ [cluster1] kubectl --context=cluster1 get all --all-namespaces
2020-11-02T14:27:07.4938133Z [cluster1] ======================= Post mortem cluster1 =======================
2020-11-02T14:27:07.4938710Z [cluster1] NAMESPACE                   NAME                                                 READY   STATUS        RESTARTS   AGE
2020-11-02T14:27:07.4940291Z [cluster1] e2e-tests-discovery-lkkrq   pod/netshootxxgqg-6757486d56-fm96g                   1/1     Terminating   0          29s
2020-11-02T14:27:07.4941498Z [cluster1] kube-system                 pod/coredns-6955765f44-hqt7w                         1/1     Running       0          12m
2020-11-02T14:27:07.4942481Z [cluster1] kube-system                 pod/coredns-6955765f44-lnvrd                         1/1     Running       0          12m
2020-11-02T14:27:07.4943519Z [cluster1] kube-system                 pod/etcd-cluster1-control-plane                      1/1     Running       0          13m
2020-11-02T14:27:07.4944772Z [cluster1] kube-system                 pod/kube-apiserver-cluster1-control-plane            1/1     Running       0          13m
2020-11-02T14:27:07.4946343Z [cluster1] kube-system                 pod/kube-controller-manager-cluster1-control-plane   1/1     Running       0          13m
2020-11-02T14:27:07.4947645Z [cluster1] kube-system                 pod/kube-proxy-lvp44                                 1/1     Running       0          12m
2020-11-02T14:27:07.4948525Z [cluster1] kube-system                 pod/kube-proxy-pbkjw                                 1/1     Running       0          12m
2020-11-02T14:27:07.4949401Z [cluster1] kube-system                 pod/kube-proxy-sx5r4                                 1/1     Running       0          12m
2020-11-02T14:27:07.4950554Z [cluster1] kube-system                 pod/kube-scheduler-cluster1-control-plane            1/1     Running       0          13m
2020-11-02T14:27:07.4951996Z [cluster1] kube-system                 pod/tiller-deploy-8488d98b4c-7wwcm                   1/1     Running       0          10m
2020-11-02T14:27:07.4952990Z [cluster1] kube-system                 pod/weave-net-5jpsf                                  2/2     Running       0          12m
2020-11-02T14:27:07.4953848Z [cluster1] kube-system                 pod/weave-net-r4dz7                                  2/2     Running       0          12m
2020-11-02T14:27:07.4954685Z [cluster1] kube-system                 pod/weave-net-w85hf                                  2/2     Running       0          12m
2020-11-02T14:27:07.4955866Z [cluster1] local-path-storage          pod/local-path-provisioner-7745554f7f-nhlhf          1/1     Running       0          12m
2020-11-02T14:27:07.4958247Z [cluster1] submariner-operator         pod/submariner-gateway-xs2nq                         1/1     Running       0          10m
2020-11-02T14:27:07.4959999Z [cluster1] submariner-operator         pod/submariner-lighthouse-agent-68c5565ff8-zhrpm     1/1     Running       0          10m
2020-11-02T14:27:07.4962232Z [cluster1] submariner-operator         pod/submariner-lighthouse-coredns-7466b6679c-h8rnx   1/1     Running       0          10m
2020-11-02T14:27:07.4964169Z [cluster1] submariner-operator         pod/submariner-lighthouse-coredns-7466b6679c-n8242   1/1     Running       0          10m
2020-11-02T14:27:07.4966052Z [cluster1] submariner-operator         pod/submariner-routeagent-ctc55                      1/1     Running       0          10m
2020-11-02T14:27:07.4967588Z [cluster1] submariner-operator         pod/submariner-routeagent-lfc7d                      1/1     Running       0          10m
2020-11-02T14:27:07.4968266Z [cluster1]
2020-11-02T14:27:07.4969060Z [cluster1] NAMESPACE             NAME                                    TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)                  AGE
2020-11-02T14:27:07.4969762Z [cluster1] default               service/kubernetes                      ClusterIP   100.1.0.1      <none>        443/TCP                  13m
2020-11-02T14:27:07.4970733Z [cluster1] kube-system           service/kube-dns                        ClusterIP   100.1.0.10     <none>        53/UDP,53/TCP,9153/TCP   13m
2020-11-02T14:27:07.4971733Z [cluster1] kube-system           service/tiller-deploy                   ClusterIP   100.1.21.22    <none>        44134/TCP                10m
2020-11-02T14:27:07.4973093Z [cluster1] submariner-operator   service/submariner-lighthouse-coredns   ClusterIP   100.1.80.194   <none>        53/UDP                   10m
2020-11-02T14:27:07.4973807Z [cluster1]
2020-11-02T14:27:07.4974608Z [cluster1] NAMESPACE             NAME                                   DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR                 AGE
2020-11-02T14:27:07.4975737Z [cluster1] kube-system           daemonset.apps/kube-proxy              3         3         3       3            3           beta.kubernetes.io/os=linux   13m
2020-11-02T14:27:07.5017733Z [lighthouse]$ [cluster1] kubectl get pods -A
2020-11-02T14:27:07.5033026Z [lighthouse]$ [cluster1] tail -n +2
2020-11-02T14:27:07.5042234Z [lighthouse]$ [cluster1] kubectl get pods -A
2020-11-02T14:27:07.5053121Z [lighthouse]$ [cluster1] grep -v Running
2020-11-02T14:27:07.5073484Z [lighthouse]$ [cluster1] command kubectl --context=cluster1 get pods -A
2020-11-02T14:27:07.5084472Z [lighthouse]$ [cluster1] sed s/  */;/g
2020-11-02T14:27:07.5090416Z [lighthouse]$ [cluster1] kubectl --context=cluster1 get pods -A
2020-11-02T14:27:07.8045019Z [lighthouse]$ [cluster1] ns=e2e-tests-discovery-lkkrq
2020-11-02T14:27:07.8059901Z [lighthouse]$ [cluster1] cut -f1 -d;
2020-11-02T14:27:07.8098535Z [lighthouse]$ [cluster1] name=netshootxxgqg-6757486d56-fm96g
2020-11-02T14:27:07.8121079Z [lighthouse]$ [cluster1] cut -f2 -d;
2020-11-02T14:27:07.8146765Z [lighthouse]$ [cluster1] kubectl -n e2e-tests-discovery-lkkrq describe pod netshootxxgqg-6757486d56-fm96g
2020-11-02T14:27:07.8162818Z [lighthouse]$ [cluster1] kubectl -n e2e-tests-discovery-lkkrq describe pod netshootxxgqg-6757486d56-fm96g
2020-11-02T14:27:07.8173328Z [lighthouse]$ [cluster1] command kubectl --context=cluster1 -n e2e-tests-discovery-lkkrq describe pod netshootxxgqg-6757486d56-fm96g
2020-11-02T14:27:07.8200846Z [lighthouse]$ [cluster1] kubectl --context=cluster1 -n e2e-tests-discovery-lkkrq describe pod netshootxxgqg-6757486d56-fm96g
2020-11-02T14:27:08.1188855Z [cluster1] kube-system           daemonset.apps/weave-net               3         3         3       3            3           <none>                        12m
2020-11-02T14:27:08.1190254Z [cluster1] submariner-operator   daemonset.apps/submariner-gateway      1         1         1       1            1           submariner.io/gateway=true    10m
2020-11-02T14:27:08.1192932Z [cluster1] submariner-operator   daemonset.apps/submariner-routeagent   2         2         2       2            2           <none>                        10m
2020-11-02T14:27:08.1193625Z [cluster1]
2020-11-02T14:27:08.1194354Z [cluster1] NAMESPACE             NAME                                            READY   UP-TO-DATE   AVAILABLE   AGE
2020-11-02T14:27:08.1195235Z [cluster1] kube-system           deployment.apps/coredns                         2/2     2            2           13m
2020-11-02T14:27:08.1196156Z [cluster1] kube-system           deployment.apps/tiller-deploy                   1/1     1            1           10m
2020-11-02T14:27:08.1197243Z [cluster1] local-path-storage    deployment.apps/local-path-provisioner          1/1     1            1           12m
2020-11-02T14:27:08.1198686Z [cluster1] submariner-operator   deployment.apps/submariner-lighthouse-agent     1/1     1            1           10m
2020-11-02T14:27:08.1200067Z [cluster1] submariner-operator   deployment.apps/submariner-lighthouse-coredns   2/2     2            2           10m
2020-11-02T14:27:08.1200797Z [cluster1]
2020-11-02T14:27:08.1201224Z [cluster1] NAMESPACE             NAME                                                       DESIRED   CURRENT   READY   AGE
2020-11-02T14:27:08.1202317Z [cluster1] kube-system           replicaset.apps/coredns-6955765f44                         2         2         2       12m
2020-11-02T14:27:08.1203371Z [cluster1] kube-system           replicaset.apps/tiller-deploy-8488d98b4c                   1         1         1       10m
2020-11-02T14:27:08.1204679Z [cluster1] local-path-storage    replicaset.apps/local-path-provisioner-7745554f7f          1         1         1       12m
2020-11-02T14:27:08.1206296Z [cluster1] submariner-operator   replicaset.apps/submariner-lighthouse-agent-68c5565ff8     1         1         1       10m
2020-11-02T14:27:08.1208724Z [cluster1] submariner-operator   replicaset.apps/submariner-lighthouse-coredns-7466b6679c   2         2         2       10m
2020-11-02T14:27:08.1210246Z [cluster1] ======================= netshootxxgqg-6757486d56-fm96g - e2e-tests-discovery-lkkrq ============================
2020-11-02T14:27:08.1211307Z [cluster1] Name:                      netshootxxgqg-6757486d56-fm96g
2020-11-02T14:27:08.1212168Z [cluster1] Namespace:                 e2e-tests-discovery-lkkrq
2020-11-02T14:27:08.1212676Z [cluster1] Priority:                  0
2020-11-02T14:27:08.1213268Z [cluster1] Node:                      cluster1-worker2/172.17.0.6
2020-11-02T14:27:08.1213714Z [cluster1] Start Time:                Mon, 02 Nov 2020 14:26:38 +0000
2020-11-02T14:27:08.1214093Z [cluster1] Labels:                    app=netshoot
2020-11-02T14:27:08.1214760Z [cluster1]                            pod-template-hash=6757486d56
2020-11-02T14:27:08.1215218Z [cluster1] Annotations:               <none>
2020-11-02T14:27:08.1215689Z [cluster1] Status:                    Terminating (lasts <invalid>)
2020-11-02T14:27:08.1216175Z [cluster1] Termination Grace Period:  30s
2020-11-02T14:27:08.1216557Z [cluster1] IP:                        10.1.0.6
2020-11-02T14:27:08.1217026Z [cluster1] IPs:
2020-11-02T14:27:08.1217322Z [cluster1]   IP:           10.1.0.6
2020-11-02T14:27:08.1218048Z [cluster1] Controlled By:  ReplicaSet/netshootxxgqg-6757486d56
2020-11-02T14:27:08.1218629Z [cluster1] Containers:
2020-11-02T14:27:08.1218984Z [cluster1]   netshoot:
2020-11-02T14:27:08.1220368Z [cluster1]     Container ID:  containerd://609493df2435b22640df56e195ebb74207a2121e9e36b770295336458e3c1130
2020-11-02T14:27:08.1221825Z [cluster1]     Image:         quay.io/submariner/nettest
2020-11-02T14:27:08.1222847Z [cluster1]     Image ID:      quay.io/submariner/nettest@sha256:63f21b9d8065d60870c2bca980dbc4dc6358edf669661576b8cbee39e3024a2d
2020-11-02T14:27:08.1223784Z [cluster1]     Port:          <none>
2020-11-02T14:27:08.1224288Z [cluster1]     Host Port:     <none>
2020-11-02T14:27:08.1224899Z [cluster1]     Command:
2020-11-02T14:27:08.1225182Z [cluster1]       sleep
2020-11-02T14:27:08.1225454Z [cluster1]       600
2020-11-02T14:27:08.1225911Z [cluster1]     State:          Running
2020-11-02T14:27:08.1226283Z [cluster1]       Started:      Mon, 02 Nov 2020 14:26:39 +0000
2020-11-02T14:27:08.1226626Z [cluster1]     Ready:          True
2020-11-02T14:27:08.1226958Z [cluster1]     Restart Count:  0
2020-11-02T14:27:08.1227324Z [cluster1]     Environment:    <none>
2020-11-02T14:27:08.1227647Z [cluster1]     Mounts:
2020-11-02T14:27:08.1228480Z [cluster1]       /var/run/secrets/kubernetes.io/serviceaccount from default-token-lc8db (ro)
2020-11-02T14:27:08.1229099Z [cluster1] Conditions:
2020-11-02T14:27:08.1229432Z [cluster1]   Type              Status
2020-11-02T14:27:08.1229785Z [cluster1]   Initialized       True
2020-11-02T14:27:08.1230136Z [cluster1]   Ready             True
2020-11-02T14:27:08.1230522Z [cluster1]   ContainersReady   True
2020-11-02T14:27:08.1230948Z [cluster1]   PodScheduled      True
2020-11-02T14:27:08.1231619Z [cluster1] Volumes:
2020-11-02T14:27:08.1232566Z [cluster1]   default-token-lc8db:
2020-11-02T14:27:08.1233117Z [cluster1]     Type:        Secret (a volume populated by a Secret)
2020-11-02T14:27:08.1234038Z [cluster1]     SecretName:  default-token-lc8db
2020-11-02T14:27:08.1234508Z [cluster1]     Optional:    false
2020-11-02T14:27:08.1234893Z [cluster1] QoS Class:       BestEffort
2020-11-02T14:27:08.1235648Z [cluster1] Node-Selectors:  <none>
2020-11-02T14:27:08.1236454Z [cluster1] Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
2020-11-02T14:27:08.1275927Z [lighthouse]$ [cluster1] kubectl -n e2e-tests-discovery-lkkrq logs netshootxxgqg-6757486d56-fm96g
2020-11-02T14:27:08.1290533Z [lighthouse]$ [cluster1] kubectl -n e2e-tests-discovery-lkkrq logs netshootxxgqg-6757486d56-fm96g
2020-11-02T14:27:08.1303998Z [lighthouse]$ [cluster1] command kubectl --context=cluster1 -n e2e-tests-discovery-lkkrq logs netshootxxgqg-6757486d56-fm96g
2020-11-02T14:27:08.1325186Z [lighthouse]$ [cluster1] kubectl --context=cluster1 -n e2e-tests-discovery-lkkrq logs netshootxxgqg-6757486d56-fm96g
2020-11-02T14:27:08.4415259Z [lighthouse]$ [cluster1] namespace=kube-system
2020-11-02T14:27:08.4433395Z [lighthouse]$ [cluster1] kubectl get pods --selector=k8s-app=kube-proxy -n kube-system -o jsonpath={.items[*].metadata.name}
2020-11-02T14:27:08.4446848Z [lighthouse]$ [cluster1] kubectl get pods --selector=k8s-app=kube-proxy -n kube-system -o jsonpath={.items[*].metadata.name}
2020-11-02T14:27:08.4461419Z [lighthouse]$ [cluster1] command kubectl --context=cluster1 get pods --selector=k8s-app=kube-proxy -n kube-system -o jsonpath={.items[*].metadata.name}
2020-11-02T14:27:08.4475458Z [lighthouse]$ [cluster1] kubectl --context=cluster1 get pods --selector=k8s-app=kube-proxy -n kube-system -o jsonpath={.items[*].metadata.name}
2020-11-02T14:27:08.7492105Z [lighthouse]$ [cluster1] kubectl -n kube-system logs kube-proxy-lvp44
2020-11-02T14:27:08.7503855Z [lighthouse]$ [cluster1] kubectl -n kube-system logs kube-proxy-lvp44
2020-11-02T14:27:08.7517450Z [lighthouse]$ [cluster1] command kubectl --context=cluster1 -n kube-system logs kube-proxy-lvp44
2020-11-02T14:27:08.7531491Z [lighthouse]$ [cluster1] kubectl --context=cluster1 -n kube-system logs kube-proxy-lvp44
2020-11-02T14:27:09.0518554Z [lighthouse]$ [cluster1] kubectl -n kube-system logs kube-proxy-pbkjw
2020-11-02T14:27:09.0531204Z [lighthouse]$ [cluster1] kubectl -n kube-system logs kube-proxy-pbkjw
2020-11-02T14:27:09.0563115Z [lighthouse]$ [cluster1] command kubectl --context=cluster1 -n kube-system logs kube-proxy-pbkjw
2020-11-02T14:27:09.0568216Z [lighthouse]$ [cluster1] kubectl --context=cluster1 -n kube-system logs kube-proxy-pbkjw
2020-11-02T14:27:09.3763022Z [lighthouse]$ [cluster1] kubectl -n kube-system logs kube-proxy-sx5r4
2020-11-02T14:27:09.3771414Z [lighthouse]$ [cluster1] kubectl -n kube-system logs kube-proxy-sx5r4
2020-11-02T14:27:09.3783707Z [lighthouse]$ [cluster1] command kubectl --context=cluster1 -n kube-system logs kube-proxy-sx5r4
2020-11-02T14:27:09.3795018Z [lighthouse]$ [cluster1] kubectl --context=cluster1 -n kube-system logs kube-proxy-sx5r4
2020-11-02T14:27:09.6809189Z [cluster1]                  node.kubernetes.io/unreachable:NoExecute for 300s
2020-11-02T14:27:09.6810497Z [cluster1] Events:          <none>
2020-11-02T14:27:09.6812019Z [cluster1] ===================== END netshootxxgqg-6757486d56-fm96g - e2e-tests-discovery-lkkrq ==========================
2020-11-02T14:27:09.6813194Z [cluster1] +++++++++++++++++++++: Logs for Pod kube-proxy-lvp44 in namespace kube-system :++++++++++++++++++++++
2020-11-02T14:27:09.6813969Z [cluster1] W1102 14:14:49.696276       1 server_others.go:323] Unknown proxy mode "", assuming iptables proxy
2020-11-02T14:27:09.6815159Z [cluster1] I1102 14:14:49.709656       1 node.go:135] Successfully retrieved node IP: 172.17.0.6
2020-11-02T14:27:09.6815813Z [cluster1] I1102 14:14:49.709694       1 server_others.go:145] Using iptables Proxier.
2020-11-02T14:27:09.6816638Z [cluster1] I1102 14:14:49.710223       1 server.go:571] Version: v1.17.0
2020-11-02T14:27:09.6817196Z [cluster1] I1102 14:14:49.711259       1 conntrack.go:52] Setting nf_conntrack_max to 131072
2020-11-02T14:27:09.6818302Z [cluster1] I1102 14:14:49.712847       1 conntrack.go:100] Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_established' to 86400
2020-11-02T14:27:09.6820477Z [cluster1] I1102 14:14:49.712927       1 conntrack.go:100] Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_close_wait' to 3600
2020-11-02T14:27:09.6821233Z [cluster1] I1102 14:14:49.717606       1 config.go:313] Starting service config controller
2020-11-02T14:27:09.6822238Z [cluster1] I1102 14:14:49.717627       1 shared_informer.go:197] Waiting for caches to sync for service config
2020-11-02T14:27:09.6823422Z [cluster1] I1102 14:14:49.717649       1 config.go:131] Starting endpoints config controller
2020-11-02T14:27:09.6825068Z [cluster1] I1102 14:14:49.717659       1 shared_informer.go:197] Waiting for caches to sync for endpoints config
2020-11-02T14:27:09.6825802Z [cluster1] I1102 14:14:49.817779       1 shared_informer.go:204] Caches are synced for service config
2020-11-02T14:27:09.6826504Z [cluster1] I1102 14:14:49.817878       1 shared_informer.go:204] Caches are synced for endpoints config
2020-11-02T14:27:09.6827563Z [cluster1] +++++++++++++++++++++: Logs for Pod kube-proxy-pbkjw in namespace kube-system :++++++++++++++++++++++
2020-11-02T14:27:09.6828291Z [cluster1] W1102 14:14:49.728073       1 server_others.go:323] Unknown proxy mode "", assuming iptables proxy
2020-11-02T14:27:09.6828954Z [cluster1] I1102 14:14:49.744156       1 node.go:135] Successfully retrieved node IP: 172.17.0.5
2020-11-02T14:27:09.6829561Z [cluster1] I1102 14:14:49.744305       1 server_others.go:145] Using iptables Proxier.
2020-11-02T14:27:09.6830090Z [cluster1] I1102 14:14:49.744734       1 server.go:571] Version: v1.17.0
2020-11-02T14:27:09.6830638Z [cluster1] I1102 14:14:49.745352       1 conntrack.go:52] Setting nf_conntrack_max to 131072
2020-11-02T14:27:09.6832146Z [cluster1] I1102 14:14:49.745677       1 conntrack.go:100] Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_established' to 86400
2020-11-02T14:27:09.6833291Z [cluster1] I1102 14:14:49.745935       1 conntrack.go:100] Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_close_wait' to 3600
2020-11-02T14:27:09.6834029Z [cluster1] I1102 14:14:49.748024       1 config.go:313] Starting service config controller
2020-11-02T14:27:09.6834712Z [cluster1] I1102 14:14:49.755336       1 shared_informer.go:197] Waiting for caches to sync for service config
2020-11-02T14:27:09.6835577Z [cluster1] I1102 14:14:49.755489       1 config.go:131] Starting endpoints config controller
2020-11-02T14:27:09.6836289Z [cluster1] I1102 14:14:49.755635       1 shared_informer.go:197] Waiting for caches to sync for endpoints config
2020-11-02T14:27:09.6837577Z [cluster1] I1102 14:14:49.859090       1 shared_informer.go:204] Caches are synced for endpoints config
2020-11-02T14:27:09.6838370Z [cluster1] I1102 14:14:49.859286       1 shared_informer.go:204] Caches are synced for service config
2020-11-02T14:27:09.6839387Z [cluster1] +++++++++++++++++++++: Logs for Pod kube-proxy-sx5r4 in namespace kube-system :++++++++++++++++++++++
2020-11-02T14:27:09.6840114Z [cluster1] W1102 14:14:26.087927       1 server_others.go:323] Unknown proxy mode "", assuming iptables proxy
2020-11-02T14:27:09.6840759Z [cluster1] I1102 14:14:26.111771       1 node.go:135] Successfully retrieved node IP: 172.17.0.4
2020-11-02T14:27:09.6841361Z [cluster1] I1102 14:14:26.111800       1 server_others.go:145] Using iptables Proxier.
2020-11-02T14:27:09.6841992Z [cluster1] I1102 14:14:26.112054       1 server.go:571] Version: v1.17.0
2020-11-02T14:27:09.6842833Z [cluster1] I1102 14:14:26.112438       1 conntrack.go:100] Set sysctl 'net/netfilter/nf_conntrack_max' to 131072
2020-11-02T14:27:09.6843508Z [cluster1] I1102 14:14:26.112463       1 conntrack.go:52] Setting nf_conntrack_max to 131072
2020-11-02T14:27:09.6844514Z [cluster1] I1102 14:14:26.117469       1 conntrack.go:100] Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_established' to 86400
2020-11-02T14:27:09.6845618Z [cluster1] I1102 14:14:26.117585       1 conntrack.go:100] Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_close_wait' to 3600
2020-11-02T14:27:09.6906304Z [lighthouse]$ [cluster1] namespace=submariner-operator
2020-11-02T14:27:09.6920547Z [lighthouse]$ [cluster1] kubectl get pods --selector=name=submariner-operator -n submariner-operator -o jsonpath={.items[*].metadata.name}
2020-11-02T14:27:09.6945417Z [lighthouse]$ [cluster1] kubectl get pods --selector=name=submariner-operator -n submariner-operator -o jsonpath={.items[*].metadata.name}
2020-11-02T14:27:09.6951140Z [lighthouse]$ [cluster1] command kubectl --context=cluster1 get pods --selector=name=submariner-operator -n submariner-operator -o jsonpath={.items[*].metadata.name}
2020-11-02T14:27:09.6994432Z [lighthouse]$ [cluster1] kubectl --context=cluster1 get pods --selector=name=submariner-operator -n submariner-operator -o jsonpath={.items[*].metadata.name}
2020-11-02T14:27:09.9958238Z [lighthouse]$ [cluster1] namespace=submariner-operator
2020-11-02T14:27:09.9970654Z [lighthouse]$ [cluster1] kubectl get pods --selector=app=submariner-globalnet -n submariner-operator -o jsonpath={.items[*].metadata.name}
2020-11-02T14:27:09.9982274Z [lighthouse]$ [cluster1] kubectl get pods --selector=app=submariner-globalnet -n submariner-operator -o jsonpath={.items[*].metadata.name}
2020-11-02T14:27:09.9994875Z [lighthouse]$ [cluster1] command kubectl --context=cluster1 get pods --selector=app=submariner-globalnet -n submariner-operator -o jsonpath={.items[*].metadata.name}
2020-11-02T14:27:10.0012017Z [lighthouse]$ [cluster1] kubectl --context=cluster1 get pods --selector=app=submariner-globalnet -n submariner-operator -o jsonpath={.items[*].metadata.name}
2020-11-02T14:27:10.2879482Z [lighthouse]$ [cluster1] kubectl get pods --selector=app=submariner-engine -n submariner-operator -o jsonpath={.items[*].metadata.name}
2020-11-02T14:27:10.2893027Z [lighthouse]$ [cluster1] kubectl get pods --selector=app=submariner-engine -n submariner-operator -o jsonpath={.items[*].metadata.name}
2020-11-02T14:27:10.2906403Z [lighthouse]$ [cluster1] command kubectl --context=cluster1 get pods --selector=app=submariner-engine -n submariner-operator -o jsonpath={.items[*].metadata.name}
2020-11-02T14:27:10.2919612Z [lighthouse]$ [cluster1] kubectl --context=cluster1 get pods --selector=app=submariner-engine -n submariner-operator -o jsonpath={.items[*].metadata.name}
2020-11-02T14:27:10.5833265Z [lighthouse]$ [cluster1] kubectl -n submariner-operator logs submariner-gateway-xs2nq
2020-11-02T14:27:10.5847439Z [lighthouse]$ [cluster1] kubectl -n submariner-operator logs submariner-gateway-xs2nq
2020-11-02T14:27:10.5863706Z [lighthouse]$ [cluster1] command kubectl --context=cluster1 -n submariner-operator logs submariner-gateway-xs2nq
2020-11-02T14:27:10.5877011Z [lighthouse]$ [cluster1] kubectl --context=cluster1 -n submariner-operator logs submariner-gateway-xs2nq
2020-11-02T14:27:10.9244152Z [cluster1] I1102 14:14:26.133224       1 config.go:131] Starting endpoints config controller
2020-11-02T14:27:10.9244917Z [cluster1] I1102 14:14:26.133239       1 shared_informer.go:197] Waiting for caches to sync for endpoints config
2020-11-02T14:27:10.9245754Z [cluster1] I1102 14:14:26.133275       1 config.go:313] Starting service config controller
2020-11-02T14:27:10.9246430Z [cluster1] I1102 14:14:26.133312       1 shared_informer.go:197] Waiting for caches to sync for service config
2020-11-02T14:27:10.9247124Z [cluster1] I1102 14:14:26.238711       1 shared_informer.go:204] Caches are synced for endpoints config
2020-11-02T14:27:10.9247829Z [cluster1] I1102 14:14:26.238800       1 shared_informer.go:204] Caches are synced for service config
2020-11-02T14:27:10.9248330Z [cluster1] ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
2020-11-02T14:27:10.9248656Z [cluster1] ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
2020-11-02T14:27:10.9248979Z [cluster1] ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
2020-11-02T14:27:10.9250367Z [cluster1] +++++++++++++++++++++: Logs for Pod submariner-gateway-xs2nq in namespace submariner-operator :++++++++++++++++++++++
2020-11-02T14:27:10.9251257Z [cluster1] + trap 'exit 1' SIGTERM SIGINT
2020-11-02T14:27:10.9251713Z [cluster1] + export CHARON_PID_FILE=/var/run/charon.pid
2020-11-02T14:27:10.9252203Z [cluster1] + CHARON_PID_FILE=/var/run/charon.pid
2020-11-02T14:27:10.9252790Z [cluster1] + rm -f /var/run/charon.pid
2020-11-02T14:27:10.9253217Z [cluster1] + SUBMARINER_VERBOSITY=2
2020-11-02T14:27:10.9253907Z [cluster1] + '[' false == true ']'
2020-11-02T14:27:10.9254352Z [cluster1] + DEBUG=-v=2
2020-11-02T14:27:10.9254812Z [cluster1] + mkdir -p /etc/ipsec
2020-11-02T14:27:10.9255463Z [cluster1] + sysctl -w net.ipv4.conf.all.send_redirects=0
2020-11-02T14:27:10.9256054Z [cluster1] net.ipv4.conf.all.send_redirects = 0
2020-11-02T14:27:10.9256721Z [cluster1] + export PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/libexec/strongswan
2020-11-02T14:27:10.9257441Z [cluster1] + PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/libexec/strongswan
2020-11-02T14:27:10.9258184Z [cluster1] + for f in iptables-save iptables
2020-11-02T14:27:10.9258821Z [cluster1] ++ find_iptables_on_host iptables-save
2020-11-02T14:27:10.9259482Z [cluster1] ++ chroot /host test -x /usr/sbin/iptables-save
2020-11-02T14:27:10.9260247Z [cluster1] chroot: cannot change root directory to '/host': No such file or directory
2020-11-02T14:27:10.9260989Z [cluster1] ++ chroot /host test -x /sbin/iptables-save
2020-11-02T14:27:10.9262153Z [cluster1] chroot: cannot change root directory to '/host': No such file or directory
2020-11-02T14:27:10.9262664Z [cluster1] ++ echo unknown
2020-11-02T14:27:10.9263033Z [cluster1] + location=unknown
2020-11-02T14:27:10.9263588Z [cluster1] + '[' unknown '!=' unknown ']'
2020-11-02T14:27:10.9264392Z [cluster1] + echo 'WARNING: not using iptables wrapper because iptables was not detected on the'
2020-11-02T14:27:10.9265474Z [cluster1] WARNING: not using iptables wrapper because iptables was not detected on the
2020-11-02T14:27:10.9266261Z [cluster1] + echo 'host at the following paths [/usr/sbin, /sbin].'
2020-11-02T14:27:10.9266774Z [cluster1] host at the following paths [/usr/sbin, /sbin].
2020-11-02T14:27:10.9267558Z [cluster1] + echo 'Either the host file system isn'\''t mounted or the host does not have iptables'
2020-11-02T14:27:10.9268415Z [cluster1] Either the host file system isn't mounted or the host does not have iptables
2020-11-02T14:27:10.9269556Z [cluster1] + echo 'installed. The pod will use the image installed iptables version.'
2020-11-02T14:27:10.9270353Z [cluster1] installed. The pod will use the image installed iptables version.
2020-11-02T14:27:10.9271114Z [cluster1] + for f in iptables-save iptables
2020-11-02T14:27:10.9271778Z [cluster1] ++ find_iptables_on_host iptables
2020-11-02T14:27:10.9272650Z [cluster1] ++ chroot /host test -x /usr/sbin/iptables
2020-11-02T14:27:10.9273412Z [cluster1] chroot: cannot change root directory to '/host': No such file or directory
2020-11-02T14:27:10.9274135Z [cluster1] ++ chroot /host test -x /sbin/iptables
2020-11-02T14:27:10.9274886Z [cluster1] chroot: cannot change root directory to '/host': No such file or directory
2020-11-02T14:27:10.9275544Z [cluster1] ++ echo unknown
2020-11-02T14:27:10.9275873Z [cluster1] + location=unknown
2020-11-02T14:27:10.9277711Z [cluster1] + '[' unknown '!=' unknown ']'
2020-11-02T14:27:10.9278669Z [cluster1] + echo 'WARNING: not using iptables wrapper because iptables was not detected on the'
2020-11-02T14:27:10.9279458Z [cluster1] WARNING: not using iptables wrapper because iptables was not detected on the
2020-11-02T14:27:10.9280459Z [cluster1] + echo 'host at the following paths [/usr/sbin, /sbin].'
2020-11-02T14:27:10.9280984Z [cluster1] host at the following paths [/usr/sbin, /sbin].
2020-11-02T14:27:10.9281792Z [cluster1] + echo 'Either the host file system isn'\''t mounted or the host does not have iptables'
2020-11-02T14:27:10.9282677Z [cluster1] Either the host file system isn't mounted or the host does not have iptables
2020-11-02T14:27:10.9283561Z [cluster1] + echo 'installed. The pod will use the image installed iptables version.'
2020-11-02T14:27:10.9284240Z [cluster1] installed. The pod will use the image installed iptables version.
2020-11-02T14:27:10.9285273Z [cluster1] + exec submariner-engine -v=2 -alsologtostderr
2020-11-02T14:27:10.9285908Z [cluster1] I1102 14:17:36.217255       1 main.go:70] Starting the submariner gateway engine
2020-11-02T14:27:10.9287039Z [cluster1] W1102 14:17:36.227259       1 client_config.go:543] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
2020-11-02T14:27:10.9288024Z [cluster1] I1102 14:17:36.230974       1 main.go:95] Creating the cable engine
2020-11-02T14:27:10.9288728Z [cluster1] I1102 14:17:36.239445       1 syncer.go:61] CableEngine syncer started
2020-11-02T14:27:10.9289608Z [cluster1] I1102 14:17:41.857841       1 main.go:252] Gateway leader election config values: main.leaderConfig{LeaseDuration:10, RenewDeadline:5, RetryPeriod:2}
2020-11-02T14:27:10.9290870Z [cluster1] I1102 14:17:41.858488       1 main.go:269] Using namespace "submariner-operator" for the leader election lock
2020-11-02T14:27:10.9292339Z [cluster1] I1102 14:17:41.859285       1 leaderelection.go:242] attempting to acquire leader lease  submariner-operator/submariner-engine-lock...
2020-11-02T14:27:10.9293811Z [cluster1] I1102 14:17:41.870333       1 leaderelection.go:252] successfully acquired lease submariner-operator/submariner-engine-lock
2020-11-02T14:27:10.9296598Z [cluster1] I1102 14:17:41.870369       1 event.go:281] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"submariner-operator", Name:"submariner-engine-lock", UID:"7ce5907d-cdf8-48eb-82d9-a65d6186ae16", APIVersion:"v1", ResourceVersion:"1301", FieldPath:""}): type: 'Normal' reason: 'LeaderElection' cluster1-worker-submariner-engine became leader
2020-11-02T14:27:10.9299388Z [cluster1] I1102 14:17:41.870612       1 main.go:144] Creating the datastore syncer
2020-11-02T14:27:10.9300052Z [cluster1] I1102 14:17:41.872099       1 strongswan.go:99] Initializing StrongSwan IPSec driver
2020-11-02T14:27:10.9300698Z [cluster1] I1102 14:17:42.056357       1 strongswan.go:460] Starting charon
2020-11-02T14:27:10.9301843Z [cluster1] W1102 14:17:42.057028       1 strongswan.go:543] Failed to connect to charon - retrying: dial unix /var/run/charon.vici: connect: no such file or directory
2020-11-02T14:27:10.9303175Z [cluster1] 00[DMN] Starting IKE charon daemon (strongSwan 5.9.0, Linux 5.4.0-1031-azure, x86_64)
2020-11-02T14:27:10.9304060Z [cluster1] 00[CFG] PKCS11 module '<name>' lacks library path
2020-11-02T14:27:10.9304773Z [cluster1] 00[LIB] openssl FIPS mode(0) - disabled
2020-11-02T14:27:10.9305727Z [cluster1] 00[CFG] loading ca certificates from '/etc/strongswan/ipsec.d/cacerts'
2020-11-02T14:27:10.9306605Z [cluster1] 00[CFG] loading aa certificates from '/etc/strongswan/ipsec.d/aacerts'
2020-11-02T14:27:10.9307546Z [cluster1] 00[CFG] loading ocsp signer certificates from '/etc/strongswan/ipsec.d/ocspcerts'
2020-11-02T14:27:10.9308568Z [cluster1] 00[CFG] loading attribute certificates from '/etc/strongswan/ipsec.d/acerts'
2020-11-02T14:27:10.9309420Z [cluster1] 00[CFG] loading crls from '/etc/strongswan/ipsec.d/crls'
2020-11-02T14:27:10.9310217Z [cluster1] 00[CFG] loading secrets from '/etc/strongswan/ipsec.secrets'
2020-11-02T14:27:10.9310992Z [cluster1] 00[CFG] opening triplet file /etc/strongswan/ipsec.d/triplets.dat failed: No such file or directory
2020-11-02T14:27:10.9324545Z [cluster1] 00[CFG] loaded 0 RADIUS server configurations
2020-11-02T14:27:10.9325467Z [cluster1] 00[CFG] HA config misses local/remote address
2020-11-02T14:27:10.9326332Z [cluster1] 00[CFG] no script for ext-auth script defined, disabled
2020-11-02T14:27:10.9329060Z [cluster1] 00[LIB] loaded plugins: charon pkcs11 tpm aesni aes des rc2 sha2 sha1 md4 md5 mgf1 random nonce x509 revocation constraints acert pubkey pkcs1 pkcs7 pkcs8 pkcs12 pgp dnskey sshkey pem openssl gcrypt fips-prf gmp curve25519 chapoly xcbc cmac hmac ctr ccm gcm drbg newhope curl attr kernel-netlink resolve socket-default farp stroke vici updown eap-identity eap-sim eap-aka eap-aka-3gpp eap-aka-3gpp2 eap-md5 eap-gtc eap-mschapv2 eap-dynamic eap-radius eap-tls eap-ttls eap-peap xauth-generic xauth-eap xauth-pam xauth-noauth dhcp led duplicheck unity counters
2020-11-02T14:27:10.9331112Z [cluster1] 00[JOB] spawning 16 worker threads
2020-11-02T14:27:10.9331982Z [cluster1] I1102 14:17:43.058136       1 cableengine.go:71] CableEngine controller started, driver: "strongswan"
2020-11-02T14:27:10.9333204Z [cluster1] I1102 14:17:43.058189       1 datastoresyncer.go:60] Starting the datastore syncer
2020-11-02T14:27:10.9333847Z [cluster1] I1102 14:17:43.058558       1 tunnel.go:22] Starting the tunnel controller
2020-11-02T14:27:10.9334728Z [cluster1] I1102 14:17:43.059071       1 helpers.go:75] Found schema.GroupVersionResource{Group:"submariner.io", Version:"v1", Resource:"endpoints"}
2020-11-02T14:27:10.9335660Z [cluster1] I1102 14:17:43.059859       1 resource_syncer.go:180] Starting syncer "Tunnel controller"
2020-11-02T14:27:10.9336386Z [cluster1] I1102 14:17:43.062423       1 resource_syncer.go:193] Syncer "Tunnel controller" waiting for informer cache to sync
2020-11-02T14:27:10.9337218Z [cluster1] I1102 14:17:43.162591       1 resource_syncer.go:202] Syncer "Tunnel controller" started
2020-11-02T14:27:10.9338110Z [cluster1] I1102 14:17:48.674623       1 helpers.go:75] Found schema.GroupVersionResource{Group:"submariner.io", Version:"v1", Resource:"clusters"}
2020-11-02T14:27:10.9339291Z [cluster1] I1102 14:17:49.125229       1 helpers.go:75] Found schema.GroupVersionResource{Group:"submariner.io", Version:"v1", Resource:"clusters"}
2020-11-02T14:27:10.9340469Z [cluster1] I1102 14:17:49.158747       1 helpers.go:75] Found schema.GroupVersionResource{Group:"submariner.io", Version:"v1", Resource:"clusters"}
2020-11-02T14:27:10.9342199Z [cluster1] I1102 14:17:49.162318       1 helpers.go:75] Found schema.GroupVersionResource{Group:"submariner.io", Version:"v1", Resource:"clusters"}
2020-11-02T14:27:10.9344100Z [cluster1] I1102 14:17:49.162722       1 helpers.go:75] Found schema.GroupVersionResource{Group:"submariner.io", Version:"v1", Resource:"endpoints"}
2020-11-02T14:27:10.9346120Z [cluster1] I1102 14:17:49.162990       1 helpers.go:75] Found schema.GroupVersionResource{Group:"submariner.io", Version:"v1", Resource:"endpoints"}
2020-11-02T14:27:10.9347422Z [cluster1] I1102 14:17:49.163396       1 datastoresyncer.go:80] Starting the broker syncer
2020-11-02T14:27:10.9348565Z [cluster1] I1102 14:17:49.166021       1 resource_syncer.go:180] Starting syncer "local -> broker for *v1.Cluster"
2020-11-02T14:27:10.9349620Z [cluster1] I1102 14:17:49.166311       1 resource_syncer.go:193] Syncer "local -> broker for *v1.Cluster" waiting for informer cache to sync
2020-11-02T14:27:10.9350587Z [cluster1] I1102 14:17:49.270766       1 resource_syncer.go:202] Syncer "local -> broker for *v1.Cluster" started
2020-11-02T14:27:10.9351685Z [cluster1] I1102 14:17:49.271243       1 resource_syncer.go:180] Starting syncer "broker -> local for *v1.Cluster"
2020-11-02T14:27:10.9352635Z [cluster1] I1102 14:17:49.271895       1 resource_syncer.go:202] Syncer "broker -> local for *v1.Cluster" started
2020-11-02T14:27:10.9353538Z [cluster1] I1102 14:17:49.274746       1 resource_syncer.go:180] Starting syncer "local -> broker for *v1.Endpoint"
2020-11-02T14:27:10.9354559Z [cluster1] I1102 14:17:49.275177       1 resource_syncer.go:193] Syncer "local -> broker for *v1.Endpoint" waiting for informer cache to sync
2020-11-02T14:27:10.9358404Z [cluster1] I1102 14:17:49.279837       1 resource_syncer.go:264] Syncer "broker -> local for *v1.Cluster" retrieved created resource: &unstructured.Unstructured{Object:map[string]interface {}{"apiVersion":"submariner.io/v1", "kind":"Cluster", "metadata":map[string]interface {}{"creationTimestamp":"2020-11-02T14:17:48Z", "generation":1, "labels":map[string]interface {}{"submariner-io/clusterID":"cluster2"}, "name":"cluster2", "namespace":"submariner-k8s-broker", "resourceVersion":"1322", "selfLink":"/apis/submariner.io/v1/namespaces/submariner-k8s-broker/clusters/cluster2", "uid":"e0a3f487-3b0b-4398-947d-a1c32af739e8"}, "spec":map[string]interface {}{"cluster_cidr":[]interface {}{"10.2.0.0/16"}, "cluster_id":"cluster2", "color_codes":[]interface {}{"blue"}, "global_cidr":[]interface {}{}, "service_cidr":[]interface {}{"100.2.0.0/16"}}}}
2020-11-02T14:27:10.9364421Z [cluster1] I1102 14:17:49.282524       1 resource_syncer.go:276] Syncing resource: &unstructured.Unstructured{Object:map[string]interface {}{"apiVersion":"submariner.io/v1", "kind":"Cluster", "metadata":map[string]interface {}{"creationTimestamp":"2020-11-02T14:17:48Z", "generation":1, "labels":map[string]interface {}{"submariner-io/clusterID":"cluster2"}, "name":"cluster2", "namespace":"submariner-k8s-broker", "resourceVersion":"1322", "selfLink":"/apis/submariner.io/v1/namespaces/submariner-k8s-broker/clusters/cluster2", "uid":"e0a3f487-3b0b-4398-947d-a1c32af739e8"}, "spec":map[string]interface {}{"cluster_cidr":[]interface {}{"10.2.0.0/16"}, "cluster_id":"cluster2", "color_codes":[]interface {}{"blue"}, "global_cidr":[]interface {}{}, "service_cidr":[]interface {}{"100.2.0.0/16"}}}}
2020-11-02T14:27:10.9370519Z [cluster1] I1102 14:17:49.282834       1 federator.go:42] In Distribute for &unstructured.Unstructured{Object:map[string]interface {}{"apiVersion":"submariner.io/v1", "kind":"Cluster", "metadata":map[string]interface {}{"creationTimestamp":"2020-11-02T14:17:48Z", "generation":1, "labels":map[string]interface {}{"submariner-io/clusterID":"cluster2"}, "name":"cluster2", "namespace":"submariner-k8s-broker", "resourceVersion":"1322", "selfLink":"/apis/submariner.io/v1/namespaces/submariner-k8s-broker/clusters/cluster2", "uid":"e0a3f487-3b0b-4398-947d-a1c32af739e8"}, "spec":map[string]interface {}{"cluster_cidr":[]interface {}{"10.2.0.0/16"}, "cluster_id":"cluster2", "color_codes":[]interface {}{"blue"}, "global_cidr":[]interface {}{}, "service_cidr":[]interface {}{"100.2.0.0/16"}}}}
2020-11-02T14:27:10.9373767Z [cluster1] I1102 14:17:49.283205       1 helpers.go:75] Found schema.GroupVersionResource{Group:"submariner.io", Version:"v1", Resource:"clusters"}
2020-11-02T14:27:10.9376765Z [cluster1] I1102 14:17:49.285300       1 create_or_update.go:32] Creating resource: &unstructured.Unstructured{Object:map[string]interface {}{"apiVersion":"submariner.io/v1", "kind":"Cluster", "metadata":map[string]interface {}{"labels":map[string]interface {}{"submariner-io/clusterID":"cluster2"}, "name":"cluster2", "namespace":"submariner-operator"}, "spec":map[string]interface {}{"cluster_cidr":[]interface {}{"10.2.0.0/16"}, "cluster_id":"cluster2", "color_codes":[]interface {}{"blue"}, "global_cidr":[]interface {}{}, "service_cidr":[]interface {}{"100.2.0.0/16"}}}}
2020-11-02T14:27:10.9379359Z [cluster1] I1102 14:17:49.299163       1 resource_syncer.go:285] Syncer "broker -> local for *v1.Cluster" successfully synced "submariner-k8s-broker/cluster2"
2020-11-02T14:27:10.9384174Z [cluster1] I1102 14:17:49.305074       1 resource_syncer.go:264] Syncer "local -> broker for *v1.Cluster" retrieved created resource: &unstructured.Unstructured{Object:map[string]interface {}{"apiVersion":"submariner.io/v1", "kind":"Cluster", "metadata":map[string]interface {}{"creationTimestamp":"2020-11-02T14:17:49Z", "generation":1, "labels":map[string]interface {}{"submariner-io/clusterID":"cluster2"}, "name":"cluster2", "namespace":"submariner-operator", "resourceVersion":"1331", "selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/clusters/cluster2", "uid":"d4ecee52-ae5d-4b68-8aff-afe47258fb62"}, "spec":map[string]interface {}{"cluster_cidr":[]interface {}{"10.2.0.0/16"}, "cluster_id":"cluster2", "color_codes":[]interface {}{"blue"}, "global_cidr":[]interface {}{}, "service_cidr":[]interface {}{"100.2.0.0/16"}}}}
2020-11-02T14:27:10.9387961Z [cluster1] I1102 14:17:49.305911       1 resource_syncer.go:437] Found cluster ID label "cluster2" - not syncing resource "cluster2"
2020-11-02T14:27:10.9388934Z [cluster1] I1102 14:17:49.375716       1 resource_syncer.go:202] Syncer "local -> broker for *v1.Endpoint" started
2020-11-02T14:27:10.9389849Z [cluster1] I1102 14:17:49.376248       1 resource_syncer.go:180] Starting syncer "broker -> local for *v1.Endpoint"
2020-11-02T14:27:10.9390759Z [cluster1] I1102 14:17:49.376527       1 resource_syncer.go:202] Syncer "broker -> local for *v1.Endpoint" started
2020-11-02T14:27:10.9391847Z [cluster1] I1102 14:17:49.377280       1 datastoresyncer.go:108] Ensuring we are the only endpoint active for this cluster
2020-11-02T14:27:10.9393290Z [cluster1] I1102 14:17:49.377510       1 datastoresyncer.go:143] Creating local submariner Cluster: types.SubmarinerCluster{ID:"cluster1", Spec:v1.ClusterSpec{ClusterID:"cluster1", ColorCodes:[]string{"blue"}, ServiceCIDR:[]string{"100.1.0.0/16"}, ClusterCIDR:[]string{"10.1.0.0/16"}, GlobalCIDR:[]string{}}}
2020-11-02T14:27:10.9405259Z [cluster1] I1102 14:17:49.377826       1 federator.go:42] In Distribute for &v1.Cluster{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"cluster1", GenerateName:"", Namespace:"", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v1.ClusterSpec{ClusterID:"cluster1", ColorCodes:[]string{"blue"}, ServiceCIDR:[]string{"100.1.0.0/16"}, ClusterCIDR:[]string{"10.1.0.0/16"}, GlobalCIDR:[]string{}}}
2020-11-02T14:27:10.9409639Z [cluster1] I1102 14:17:49.379302       1 helpers.go:75] Found schema.GroupVersionResource{Group:"submariner.io", Version:"v1", Resource:"clusters"}
2020-11-02T14:27:10.9416103Z [cluster1] I1102 14:17:49.383746       1 resource_syncer.go:264] Syncer "broker -> local for *v1.Endpoint" retrieved created resource: &unstructured.Unstructured{Object:map[string]interface {}{"apiVersion":"submariner.io/v1", "kind":"Endpoint", "metadata":map[string]interface {}{"creationTimestamp":"2020-11-02T14:17:48Z", "generation":1, "labels":map[string]interface {}{"submariner-io/clusterID":"cluster2"}, "name":"cluster2-submariner-cable-cluster2-172-17-0-9", "namespace":"submariner-k8s-broker", "resourceVersion":"1323", "selfLink":"/apis/submariner.io/v1/namespaces/submariner-k8s-broker/endpoints/cluster2-submariner-cable-cluster2-172-17-0-9", "uid":"c683ad51-5f24-4408-857a-1f081edbeeb5"}, "spec":map[string]interface {}{"backend":"strongswan", "cable_name":"submariner-cable-cluster2-172-17-0-9", "cluster_id":"cluster2", "hostname":"cluster2-worker", "nat_enabled":false, "private_ip":"172.17.0.9", "public_ip":"", "subnets":[]interface {}{"100.2.0.0/16", "10.2.0.0/16"}}}}
2020-11-02T14:27:10.9460910Z [cluster1] I1102 14:17:49.384582       1 resource_syncer.go:276] Syncing resource: &unstructured.Unstructured{Object:map[string]interface {}{"apiVersion":"submariner.io/v1", "kind":"Endpoint", "metadata":map[string]interface {}{"creationTimestamp":"2020-11-02T14:17:48Z", "generation":1, "labels":map[string]interface {}{"submariner-io/clusterID":"cluster2"}, "name":"cluster2-submariner-cable-cluster2-172-17-0-9", "namespace":"submariner-k8s-broker", "resourceVersion":"1323", "selfLink":"/apis/submariner.io/v1/namespaces/submariner-k8s-broker/endpoints/cluster2-submariner-cable-cluster2-172-17-0-9", "uid":"c683ad51-5f24-4408-857a-1f081edbeeb5"}, "spec":map[string]interface {}{"backend":"strongswan", "cable_name":"submariner-cable-cluster2-172-17-0-9", "cluster_id":"cluster2", "hostname":"cluster2-worker", "nat_enabled":false, "private_ip":"172.17.0.9", "public_ip":"", "subnets":[]interface {}{"100.2.0.0/16", "10.2.0.0/16"}}}}
2020-11-02T14:27:10.9469776Z [cluster1] I1102 14:17:49.384957       1 federator.go:42] In Distribute for &unstructured.Unstructured{Object:map[string]interface {}{"apiVersion":"submariner.io/v1", "kind":"Endpoint", "metadata":map[string]interface {}{"creationTimestamp":"2020-11-02T14:17:48Z", "generation":1, "labels":map[string]interface {}{"submariner-io/clusterID":"cluster2"}, "name":"cluster2-submariner-cable-cluster2-172-17-0-9", "namespace":"submariner-k8s-broker", "resourceVersion":"1323", "selfLink":"/apis/submariner.io/v1/namespaces/submariner-k8s-broker/endpoints/cluster2-submariner-cable-cluster2-172-17-0-9", "uid":"c683ad51-5f24-4408-857a-1f081edbeeb5"}, "spec":map[string]interface {}{"backend":"strongswan", "cable_name":"submariner-cable-cluster2-172-17-0-9", "cluster_id":"cluster2", "hostname":"cluster2-worker", "nat_enabled":false, "private_ip":"172.17.0.9", "public_ip":"", "subnets":[]interface {}{"100.2.0.0/16", "10.2.0.0/16"}}}}
2020-11-02T14:27:10.9474522Z [cluster1] I1102 14:17:49.385392       1 helpers.go:75] Found schema.GroupVersionResource{Group:"submariner.io", Version:"v1", Resource:"endpoints"}
2020-11-02T14:27:10.9478049Z [cluster1] I1102 14:17:49.384217       1 create_or_update.go:32] Creating resource: &unstructured.Unstructured{Object:map[string]interface {}{"apiVersion":"submariner.io/v1", "kind":"Cluster", "metadata":map[string]interface {}{"name":"cluster1", "namespace":"submariner-operator"}, "spec":map[string]interface {}{"cluster_cidr":[]interface {}{"10.1.0.0/16"}, "cluster_id":"cluster1", "color_codes":[]interface {}{"blue"}, "global_cidr":[]interface {}{}, "service_cidr":[]interface {}{"100.1.0.0/16"}}}}
2020-11-02T14:27:10.9483125Z [cluster1] I1102 14:17:49.391054       1 create_or_update.go:32] Creating resource: &unstructured.Unstructured{Object:map[string]interface {}{"apiVersion":"submariner.io/v1", "kind":"Endpoint", "metadata":map[string]interface {}{"labels":map[string]interface {}{"submariner-io/clusterID":"cluster2"}, "name":"cluster2-submariner-cable-cluster2-172-17-0-9", "namespace":"submariner-operator"}, "spec":map[string]interface {}{"backend":"strongswan", "cable_name":"submariner-cable-cluster2-172-17-0-9", "cluster_id":"cluster2", "hostname":"cluster2-worker", "nat_enabled":false, "private_ip":"172.17.0.9", "public_ip":"", "subnets":[]interface {}{"100.2.0.0/16", "10.2.0.0/16"}}}}
2020-11-02T14:27:10.9489399Z [cluster1] I1102 14:17:49.399183       1 resource_syncer.go:264] Syncer "local -> broker for *v1.Cluster" retrieved created resource: &unstructured.Unstructured{Object:map[string]interface {}{"apiVersion":"submariner.io/v1", "kind":"Cluster", "metadata":map[string]interface {}{"creationTimestamp":"2020-11-02T14:17:49Z", "generation":1, "name":"cluster1", "namespace":"submariner-operator", "resourceVersion":"1332", "selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/clusters/cluster1", "uid":"8c6e5c0d-c84d-4763-8d5e-0e270e7de7b9"}, "spec":map[string]interface {}{"cluster_cidr":[]interface {}{"10.1.0.0/16"}, "cluster_id":"cluster1", "color_codes":[]interface {}{"blue"}, "global_cidr":[]interface {}{}, "service_cidr":[]interface {}{"100.1.0.0/16"}}}}
2020-11-02T14:27:10.9495439Z [cluster1] I1102 14:17:49.399255       1 resource_syncer.go:276] Syncing resource: &unstructured.Unstructured{Object:map[string]interface {}{"apiVersion":"submariner.io/v1", "kind":"Cluster", "metadata":map[string]interface {}{"creationTimestamp":"2020-11-02T14:17:49Z", "generation":1, "name":"cluster1", "namespace":"submariner-operator", "resourceVersion":"1332", "selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/clusters/cluster1", "uid":"8c6e5c0d-c84d-4763-8d5e-0e270e7de7b9"}, "spec":map[string]interface {}{"cluster_cidr":[]interface {}{"10.1.0.0/16"}, "cluster_id":"cluster1", "color_codes":[]interface {}{"blue"}, "global_cidr":[]interface {}{}, "service_cidr":[]interface {}{"100.1.0.0/16"}}}}
2020-11-02T14:27:10.9500819Z [cluster1] I1102 14:17:49.399297       1 federator.go:42] In Distribute for &unstructured.Unstructured{Object:map[string]interface {}{"apiVersion":"submariner.io/v1", "kind":"Cluster", "metadata":map[string]interface {}{"creationTimestamp":"2020-11-02T14:17:49Z", "generation":1, "name":"cluster1", "namespace":"submariner-operator", "resourceVersion":"1332", "selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/clusters/cluster1", "uid":"8c6e5c0d-c84d-4763-8d5e-0e270e7de7b9"}, "spec":map[string]interface {}{"cluster_cidr":[]interface {}{"10.1.0.0/16"}, "cluster_id":"cluster1", "color_codes":[]interface {}{"blue"}, "global_cidr":[]interface {}{}, "service_cidr":[]interface {}{"100.1.0.0/16"}}}}
2020-11-02T14:27:10.9503853Z [cluster1] I1102 14:17:49.399372       1 helpers.go:75] Found schema.GroupVersionResource{Group:"submariner.io", Version:"v1", Resource:"clusters"}
2020-11-02T14:27:10.9505950Z [cluster1] I1102 14:17:49.407316       1 resource_syncer.go:285] Syncer "broker -> local for *v1.Endpoint" successfully synced "submariner-k8s-broker/cluster2-submariner-cable-cluster2-172-17-0-9"
2020-11-02T14:27:10.9550488Z [cluster1] I1102 14:17:49.407443       1 datastoresyncer.go:156] Creating local submariner Endpoint: types.SubmarinerEndpoint{Spec:v1.EndpointSpec{ClusterID:"cluster1", CableName:"submariner-cable-cluster1-172-17-0-5", Hostname:"cluster1-worker", Subnets:[]string{"100.1.0.0/16", "10.1.0.0/16"}, PrivateIP:"172.17.0.5", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string{}}}
2020-11-02T14:27:10.9557265Z [cluster1] I1102 14:17:49.407479       1 federator.go:42] In Distribute for &v1.Endpoint{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"cluster1-submariner-cable-cluster1-172-17-0-5", GenerateName:"", Namespace:"", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v1.EndpointSpec{ClusterID:"cluster1", CableName:"submariner-cable-cluster1-172-17-0-5", Hostname:"cluster1-worker", Subnets:[]string{"100.1.0.0/16", "10.1.0.0/16"}, PrivateIP:"172.17.0.5", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string{}}}
2020-11-02T14:27:10.9562035Z [cluster1] I1102 14:17:49.407690       1 helpers.go:75] Found schema.GroupVersionResource{Group:"submariner.io", Version:"v1", Resource:"endpoints"}
2020-11-02T14:27:10.9568223Z [cluster1] I1102 14:17:49.413523       1 resource_syncer.go:264] Syncer "local -> broker for *v1.Endpoint" retrieved created resource: &unstructured.Unstructured{Object:map[string]interface {}{"apiVersion":"submariner.io/v1", "kind":"Endpoint", "metadata":map[string]interface {}{"creationTimestamp":"2020-11-02T14:17:49Z", "generation":1, "labels":map[string]interface {}{"submariner-io/clusterID":"cluster2"}, "name":"cluster2-submariner-cable-cluster2-172-17-0-9", "namespace":"submariner-operator", "resourceVersion":"1333", "selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster2-submariner-cable-cluster2-172-17-0-9", "uid":"6c3f2c01-0843-4317-be97-7f6ad9cedad0"}, "spec":map[string]interface {}{"backend":"strongswan", "cable_name":"submariner-cable-cluster2-172-17-0-9", "cluster_id":"cluster2", "hostname":"cluster2-worker", "nat_enabled":false, "private_ip":"172.17.0.9", "public_ip":"", "subnets":[]interface {}{"100.2.0.0/16", "10.2.0.0/16"}}}}
2020-11-02T14:27:10.9573892Z [cluster1] I1102 14:17:49.413630       1 resource_syncer.go:437] Found cluster ID label "cluster2" - not syncing resource "cluster2-submariner-cable-cluster2-172-17-0-9"
2020-11-02T14:27:10.9580369Z [cluster1] I1102 14:17:49.413668       1 resource_syncer.go:264] Syncer "Tunnel controller" retrieved created resource: &unstructured.Unstructured{Object:map[string]interface {}{"apiVersion":"submariner.io/v1", "kind":"Endpoint", "metadata":map[string]interface {}{"creationTimestamp":"2020-11-02T14:17:49Z", "generation":1, "labels":map[string]interface {}{"submariner-io/clusterID":"cluster2"}, "name":"cluster2-submariner-cable-cluster2-172-17-0-9", "namespace":"submariner-operator", "resourceVersion":"1333", "selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster2-submariner-cable-cluster2-172-17-0-9", "uid":"6c3f2c01-0843-4317-be97-7f6ad9cedad0"}, "spec":map[string]interface {}{"backend":"strongswan", "cable_name":"submariner-cable-cluster2-172-17-0-9", "cluster_id":"cluster2", "hostname":"cluster2-worker", "nat_enabled":false, "private_ip":"172.17.0.9", "public_ip":"", "subnets":[]interface {}{"100.2.0.0/16", "10.2.0.0/16"}}}}
2020-11-02T14:27:10.9592178Z [cluster1] I1102 14:17:49.413840       1 tunnel.go:56] Tunnel controller processing added or updated submariner Endpoint object: &v1.Endpoint{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"cluster2-submariner-cable-cluster2-172-17-0-9", GenerateName:"", Namespace:"submariner-operator", SelfLink:"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster2-submariner-cable-cluster2-172-17-0-9", UID:"6c3f2c01-0843-4317-be97-7f6ad9cedad0", ResourceVersion:"1333", Generation:1, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63739923469, loc:(*time.Location)(0x2020aa0)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{"submariner-io/clusterID":"cluster2"}, Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v1.EndpointSpec{ClusterID:"cluster2", CableName:"submariner-cable-cluster2-172-17-0-9", Hostname:"cluster2-worker", Subnets:[]string{"100.2.0.0/16", "10.2.0.0/16"}, PrivateIP:"172.17.0.9", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
2020-11-02T14:27:10.9599976Z [cluster1] I1102 14:17:49.413886       1 cableengine.go:101] Installing Endpoint cable "submariner-cable-cluster2-172-17-0-9"
2020-11-02T14:27:10.9603655Z [cluster1] I1102 14:17:49.420528       1 create_or_update.go:32] Creating resource: &unstructured.Unstructured{Object:map[string]interface {}{"apiVersion":"submariner.io/v1", "kind":"Endpoint", "metadata":map[string]interface {}{"name":"cluster1-submariner-cable-cluster1-172-17-0-5", "namespace":"submariner-operator"}, "spec":map[string]interface {}{"backend":"strongswan", "cable_name":"submariner-cable-cluster1-172-17-0-5", "cluster_id":"cluster1", "hostname":"cluster1-worker", "nat_enabled":false, "private_ip":"172.17.0.5", "public_ip":"", "subnets":[]interface {}{"100.1.0.0/16", "10.1.0.0/16"}}}}
2020-11-02T14:27:10.9609165Z [cluster1] I1102 14:17:49.421357       1 create_or_update.go:32] Creating resource: &unstructured.Unstructured{Object:map[string]interface {}{"apiVersion":"submariner.io/v1", "kind":"Cluster", "metadata":map[string]interface {}{"labels":map[string]interface {}{"submariner-io/clusterID":"cluster1"}, "name":"cluster1", "namespace":"submariner-k8s-broker"}, "spec":map[string]interface {}{"cluster_cidr":[]interface {}{"10.1.0.0/16"}, "cluster_id":"cluster1", "color_codes":[]interface {}{"blue"}, "global_cidr":[]interface {}{}, "service_cidr":[]interface {}{"100.1.0.0/16"}}}}
2020-11-02T14:27:10.9611355Z [cluster1] 14[CFG] loaded IKE shared key for: '172.17.0.9'
2020-11-02T14:27:10.9611932Z [cluster1] I1102 14:17:49.448821       1 datastoresyncer.go:99] Datastore syncer started
2020-11-02T14:27:10.9613013Z [cluster1] 03[CFG] added vici connection: submariner-cable-cluster2-172-17-0-9
2020-11-02T14:27:10.9614745Z [cluster1] 03[CFG] initiating 'submariner-child-submariner-cable-cluster2-172-17-0-9'
2020-11-02T14:27:10.9616367Z [cluster1] 03[IKE] initiating IKE_SA submariner-cable-cluster2-172-17-0-9[1] to 172.17.0.9
2020-11-02T14:27:10.9617969Z [cluster1] 03[ENC] generating IKE_SA_INIT request 0 [ SA KE No N(NATD_S_IP) N(NATD_D_IP) N(FRAG_SUP) N(HASH_ALG) N(REDIR_SUP) ]
2020-11-02T14:27:10.9618588Z [cluster1] 03[NET] sending packet: from 172.17.0.5[500] to 172.17.0.9[500] (500 bytes)
2020-11-02T14:27:10.9619812Z [cluster1] I1102 14:17:49.469503       1 cableengine.go:150] Successfully installed Endpoint cable "submariner-cable-cluster2-172-17-0-9" with remote IP 172.17.0.9
2020-11-02T14:27:10.9621704Z [cluster1] I1102 14:17:49.476099       1 tunnel.go:68] Tunnel controller successfully installed Endpoint cable submariner-cable-cluster2-172-17-0-9 in the engine
2020-11-02T14:27:10.9627155Z [cluster1] I1102 14:17:49.475960       1 resource_syncer.go:264] Syncer "local -> broker for *v1.Endpoint" retrieved created resource: &unstructured.Unstructured{Object:map[string]interface {}{"apiVersion":"submariner.io/v1", "kind":"Endpoint", "metadata":map[string]interface {}{"creationTimestamp":"2020-11-02T14:17:49Z", "generation":1, "name":"cluster1-submariner-cable-cluster1-172-17-0-5", "namespace":"submariner-operator", "resourceVersion":"1336", "selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster1-submariner-cable-cluster1-172-17-0-5", "uid":"8ad9dd2a-90d6-46e2-8306-555f0b4b595b"}, "spec":map[string]interface {}{"backend":"strongswan", "cable_name":"submariner-cable-cluster1-172-17-0-5", "cluster_id":"cluster1", "hostname":"cluster1-worker", "nat_enabled":false, "private_ip":"172.17.0.5", "public_ip":"", "subnets":[]interface {}{"100.1.0.0/16", "10.1.0.0/16"}}}}
2020-11-02T14:27:10.9636295Z [cluster1] I1102 14:17:49.481754       1 resource_syncer.go:276] Syncing resource: &unstructured.Unstructured{Object:map[string]interface {}{"apiVersion":"submariner.io/v1", "kind":"Endpoint", "metadata":map[string]interface {}{"creationTimestamp":"2020-11-02T14:17:49Z", "generation":1, "name":"cluster1-submariner-cable-cluster1-172-17-0-5", "namespace":"submariner-operator", "resourceVersion":"1336", "selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster1-submariner-cable-cluster1-172-17-0-5", "uid":"8ad9dd2a-90d6-46e2-8306-555f0b4b595b"}, "spec":map[string]interface {}{"backend":"strongswan", "cable_name":"submariner-cable-cluster1-172-17-0-5", "cluster_id":"cluster1", "hostname":"cluster1-worker", "nat_enabled":false, "private_ip":"172.17.0.5", "public_ip":"", "subnets":[]interface {}{"100.1.0.0/16", "10.1.0.0/16"}}}}
2020-11-02T14:27:10.9640921Z [cluster1] I1102 14:17:49.481737       1 resource_syncer.go:352] Syncer "Tunnel controller": transform function returned nil - not syncing - requeue: false
2020-11-02T14:27:10.9646428Z [cluster1] I1102 14:17:49.486095       1 resource_syncer.go:264] Syncer "Tunnel controller" retrieved created resource: &unstructured.Unstructured{Object:map[string]interface {}{"apiVersion":"submariner.io/v1", "kind":"Endpoint", "metadata":map[string]interface {}{"creationTimestamp":"2020-11-02T14:17:49Z", "generation":1, "name":"cluster1-submariner-cable-cluster1-172-17-0-5", "namespace":"submariner-operator", "resourceVersion":"1336", "selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster1-submariner-cable-cluster1-172-17-0-5", "uid":"8ad9dd2a-90d6-46e2-8306-555f0b4b595b"}, "spec":map[string]interface {}{"backend":"strongswan", "cable_name":"submariner-cable-cluster1-172-17-0-5", "cluster_id":"cluster1", "hostname":"cluster1-worker", "nat_enabled":false, "private_ip":"172.17.0.5", "public_ip":"", "subnets":[]interface {}{"100.1.0.0/16", "10.1.0.0/16"}}}}
2020-11-02T14:27:10.9651517Z [cluster1] I1102 14:17:49.487427       1 resource_syncer.go:285] Syncer "local -> broker for *v1.Cluster" successfully synced "submariner-operator/cluster1"
2020-11-02T14:27:10.9656661Z [cluster1] I1102 14:17:49.483489       1 federator.go:42] In Distribute for &unstructured.Unstructured{Object:map[string]interface {}{"apiVersion":"submariner.io/v1", "kind":"Endpoint", "metadata":map[string]interface {}{"creationTimestamp":"2020-11-02T14:17:49Z", "generation":1, "name":"cluster1-submariner-cable-cluster1-172-17-0-5", "namespace":"submariner-operator", "resourceVersion":"1336", "selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster1-submariner-cable-cluster1-172-17-0-5", "uid":"8ad9dd2a-90d6-46e2-8306-555f0b4b595b"}, "spec":map[string]interface {}{"backend":"strongswan", "cable_name":"submariner-cable-cluster1-172-17-0-5", "cluster_id":"cluster1", "hostname":"cluster1-worker", "nat_enabled":false, "private_ip":"172.17.0.5", "public_ip":"", "subnets":[]interface {}{"100.1.0.0/16", "10.1.0.0/16"}}}}
2020-11-02T14:27:10.9661336Z [cluster1] I1102 14:17:49.501779       1 helpers.go:75] Found schema.GroupVersionResource{Group:"submariner.io", Version:"v1", Resource:"endpoints"}
2020-11-02T14:27:10.9668123Z [cluster1] I1102 14:17:49.492190       1 tunnel.go:56] Tunnel controller processing added or updated submariner Endpoint object: &v1.Endpoint{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"cluster1-submariner-cable-cluster1-172-17-0-5", GenerateName:"", Namespace:"submariner-operator", SelfLink:"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster1-submariner-cable-cluster1-172-17-0-5", UID:"8ad9dd2a-90d6-46e2-8306-555f0b4b595b", ResourceVersion:"1336", Generation:1, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63739923469, loc:(*time.Location)(0x2020aa0)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v1.EndpointSpec{ClusterID:"cluster1", CableName:"submariner-cable-cluster1-172-17-0-5", Hostname:"cluster1-worker", Subnets:[]string{"100.1.0.0/16", "10.1.0.0/16"}, PrivateIP:"172.17.0.5", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
2020-11-02T14:27:10.9721236Z [cluster1] 02[NET] received packet: from 172.17.0.9[500] to 172.17.0.5[500] (36 bytes)
2020-11-02T14:27:10.9722006Z [cluster1] 02[ENC] parsed IKE_SA_INIT response 0 [ N(NO_PROP) ]
2020-11-02T14:27:10.9722537Z [cluster1] 02[IKE] received NO_PROPOSAL_CHOSEN notify error
2020-11-02T14:27:10.9726787Z [cluster1] I1102 14:17:49.493495       1 resource_syncer.go:264] Syncer "broker -> local for *v1.Cluster" retrieved created resource: &unstructured.Unstructured{Object:map[string]interface {}{"apiVersion":"submariner.io/v1", "kind":"Cluster", "metadata":map[string]interface {}{"creationTimestamp":"2020-11-02T14:17:49Z", "generation":1, "labels":map[string]interface {}{"submariner-io/clusterID":"cluster1"}, "name":"cluster1", "namespace":"submariner-k8s-broker", "resourceVersion":"1335", "selfLink":"/apis/submariner.io/v1/namespaces/submariner-k8s-broker/clusters/cluster1", "uid":"3ec30dab-da2c-4e45-85a4-f8b7e55f322d"}, "spec":map[string]interface {}{"cluster_cidr":[]interface {}{"10.1.0.0/16"}, "cluster_id":"cluster1", "color_codes":[]interface {}{"blue"}, "global_cidr":[]interface {}{}, "service_cidr":[]interface {}{"100.1.0.0/16"}}}}
2020-11-02T14:27:10.9730690Z [cluster1] I1102 14:17:49.528071       1 resource_syncer.go:443] Cluster ID label "cluster1" not present or matches local cluster ID "cluster1" - not syncing resource "cluster1"
2020-11-02T14:27:10.9731627Z [cluster1] I1102 14:17:49.528291       1 cableengine.go:92] Not installing cable for local cluster
2020-11-02T14:27:10.9733061Z [cluster1] I1102 14:17:49.528306       1 tunnel.go:68] Tunnel controller successfully installed Endpoint cable submariner-cable-cluster1-172-17-0-5 in the engine
2020-11-02T14:27:10.9734631Z [cluster1] I1102 14:17:49.528574       1 resource_syncer.go:352] Syncer "Tunnel controller": transform function returned nil - not syncing - requeue: false
2020-11-02T14:27:10.9739217Z [cluster1] I1102 14:17:49.537907       1 create_or_update.go:32] Creating resource: &unstructured.Unstructured{Object:map[string]interface {}{"apiVersion":"submariner.io/v1", "kind":"Endpoint", "metadata":map[string]interface {}{"labels":map[string]interface {}{"submariner-io/clusterID":"cluster1"}, "name":"cluster1-submariner-cable-cluster1-172-17-0-5", "namespace":"submariner-k8s-broker"}, "spec":map[string]interface {}{"backend":"strongswan", "cable_name":"submariner-cable-cluster1-172-17-0-5", "cluster_id":"cluster1", "hostname":"cluster1-worker", "nat_enabled":false, "private_ip":"172.17.0.5", "public_ip":"", "subnets":[]interface {}{"100.1.0.0/16", "10.1.0.0/16"}}}}
2020-11-02T14:27:10.9743298Z [cluster1] I1102 14:17:49.553624       1 resource_syncer.go:285] Syncer "local -> broker for *v1.Endpoint" successfully synced "submariner-operator/cluster1-submariner-cable-cluster1-172-17-0-5"
2020-11-02T14:27:10.9750770Z [cluster1] I1102 14:17:49.560916       1 resource_syncer.go:264] Syncer "broker -> local for *v1.Endpoint" retrieved created resource: &unstructured.Unstructured{Object:map[string]interface {}{"apiVersion":"submariner.io/v1", "kind":"Endpoint", "metadata":map[string]interface {}{"creationTimestamp":"2020-11-02T14:17:49Z", "generation":1, "labels":map[string]interface {}{"submariner-io/clusterID":"cluster1"}, "name":"cluster1-submariner-cable-cluster1-172-17-0-5", "namespace":"submariner-k8s-broker", "resourceVersion":"1337", "selfLink":"/apis/submariner.io/v1/namespaces/submariner-k8s-broker/endpoints/cluster1-submariner-cable-cluster1-172-17-0-5", "uid":"29dd0c54-3258-411c-a553-9ac43bfda669"}, "spec":map[string]interface {}{"backend":"strongswan", "cable_name":"submariner-cable-cluster1-172-17-0-5", "cluster_id":"cluster1", "hostname":"cluster1-worker", "nat_enabled":false, "private_ip":"172.17.0.5", "public_ip":"", "subnets":[]interface {}{"100.1.0.0/16", "10.1.0.0/16"}}}}
2020-11-02T14:27:10.9777496Z [cluster1] I1102 14:17:49.561248       1 resource_syncer.go:443] Cluster ID label "cluster1" not present or matches local cluster ID "cluster1" - not syncing resource "cluster1-submariner-cable-cluster1-172-17-0-5"
2020-11-02T14:27:10.9778969Z [cluster1] 12[NET] received packet: from 172.17.0.9[500] to 172.17.0.5[500] (500 bytes)
2020-11-02T14:27:10.9779611Z [cluster1] 12[ENC] parsed IKE_SA_INIT request 0 [ SA KE No N(NATD_S_IP) N(NATD_D_IP) N(FRAG_SUP) N(HASH_ALG) N(REDIR_SUP) ]
2020-11-02T14:27:10.9780206Z [cluster1] 12[IKE] 172.17.0.9 is initiating an IKE_SA
2020-11-02T14:27:10.9780712Z [cluster1] 12[CFG] selected proposal: IKE:AES_GCM_16_128/PRF_HMAC_SHA2_256/MODP_2048
2020-11-02T14:27:10.9781216Z [cluster1] 12[IKE] remote host is behind NAT
2020-11-02T14:27:10.9781865Z [cluster1] 12[ENC] generating IKE_SA_INIT response 0 [ SA KE No N(NATD_S_IP) N(NATD_D_IP) N(FRAG_SUP) N(HASH_ALG) N(CHDLESS_SUP) N(MULT_AUTH) ]
2020-11-02T14:27:10.9782542Z [cluster1] 12[NET] sending packet: from 172.17.0.5[500] to 172.17.0.9[500] (464 bytes)
2020-11-02T14:27:10.9783038Z [cluster1] 14[NET] received packet: from 172.17.0.9[4500] to 172.17.0.5[4500] (333 bytes)
2020-11-02T14:27:10.9783901Z [cluster1] 14[ENC] parsed IKE_AUTH request 1 [ IDi N(INIT_CONTACT) IDr AUTH SA TSi TSr N(MULT_AUTH) N(EAP_ONLY) N(MSG_ID_SYN_SUP) ]
2020-11-02T14:27:10.9784808Z [cluster1] 14[CFG] looking for peer configs matching 172.17.0.5[172.17.0.5]...172.17.0.9[172.17.0.9]
2020-11-02T14:27:10.9786035Z [cluster1] 14[CFG] selected peer config 'submariner-cable-cluster2-172-17-0-9'
2020-11-02T14:27:10.9787357Z [cluster1] 14[IKE] authentication of '172.17.0.9' with pre-shared key successful
2020-11-02T14:27:10.9788212Z [cluster1] 14[IKE] authentication of '172.17.0.5' (myself) with pre-shared key
2020-11-02T14:27:10.9789922Z [lighthouse]$ [cluster1] kubectl get pods --selector=app=submariner-routeagent -n submariner-operator -o jsonpath={.items[*].metadata.name}
2020-11-02T14:27:10.9796487Z [lighthouse]$ [cluster1] kubectl get pods --selector=app=submariner-routeagent -n submariner-operator -o jsonpath={.items[*].metadata.name}
2020-11-02T14:27:10.9798553Z [lighthouse]$ [cluster1] command kubectl --context=cluster1 get pods --selector=app=submariner-routeagent -n submariner-operator -o jsonpath={.items[*].metadata.name}
2020-11-02T14:27:10.9800448Z [lighthouse]$ [cluster1] kubectl --context=cluster1 get pods --selector=app=submariner-routeagent -n submariner-operator -o jsonpath={.items[*].metadata.name}
2020-11-02T14:27:10.9802192Z [cluster1] 14[IKE] IKE_SA submariner-cable-cluster2-172-17-0-9[2] established between 172.17.0.5[172.17.0.5]...172.17.0.9[172.17.0.9]
2020-11-02T14:27:10.9803007Z [cluster1] 14[IKE] scheduling rekeying in 13946s
2020-11-02T14:27:10.9803460Z [cluster1] 14[IKE] maximum IKE_SA lifetime 15386s
2020-11-02T14:27:10.9803947Z [cluster1] 14[CFG] selected proposal: ESP:AES_GCM_16_128/NO_EXT_SEQ
2020-11-02T14:27:10.9804700Z [cluster1] 15[KNL] interface vx-submariner activated
2020-11-02T14:27:10.9805383Z [cluster1] 16[KNL] 240.17.0.5 appeared on vx-submariner
2020-11-02T14:27:10.9807246Z [cluster1] 14[IKE] CHILD_SA submariner-child-submariner-cable-cluster2-172-17-0-9{1} established with SPIs c62a309f_i c65121a0_o and TS 10.1.0.0/16 100.1.0.0/16 172.17.0.5/32 === 10.2.0.0/16 100.2.0.0/16 172.17.0.9/32
2020-11-02T14:27:10.9808616Z [cluster1] 14[ENC] generating IKE_AUTH response 1 [ IDr AUTH SA TSi TSr ]
2020-11-02T14:27:10.9809182Z [cluster1] 14[NET] sending packet: from 172.17.0.5[4500] to 172.17.0.9[4500] (257 bytes)
2020-11-02T14:27:10.9809746Z [cluster1] 04[KNL] interface vethwepgd3c4845 deleted
2020-11-02T14:27:10.9810345Z [cluster1] 13[KNL] interface vethwepld3c4845 activated
2020-11-02T14:27:10.9810963Z [cluster1] 11[KNL] interface vethwepld3c4845 deactivated
2020-11-02T14:27:10.9811730Z [cluster1] 14[KNL] interface vethwepld3c4845 deleted
2020-11-02T14:27:10.9812318Z [cluster1] 04[KNL] interface vethwepga91fbe4 deleted
2020-11-02T14:27:11.2884721Z [lighthouse]$ [cluster1] kubectl -n submariner-operator logs submariner-routeagent-ctc55
2020-11-02T14:27:11.2892564Z [lighthouse]$ [cluster1] kubectl -n submariner-operator logs submariner-routeagent-ctc55
2020-11-02T14:27:11.2904346Z [lighthouse]$ [cluster1] command kubectl --context=cluster1 -n submariner-operator logs submariner-routeagent-ctc55
2020-11-02T14:27:11.2920435Z [lighthouse]$ [cluster1] kubectl --context=cluster1 -n submariner-operator logs submariner-routeagent-ctc55
2020-11-02T14:27:11.6002431Z [cluster1] 11[KNL] interface vethwepla91fbe4 activated
2020-11-02T14:27:11.6003134Z [cluster1] 12[KNL] interface vethwepla91fbe4 deactivated
2020-11-02T14:27:11.6003691Z [cluster1] 08[KNL] interface vethwepla91fbe4 deleted
2020-11-02T14:27:11.6004221Z [cluster1] 14[KNL] interface vethwepga5ed316 deleted
2020-11-02T14:27:11.6004905Z [cluster1] 16[KNL] interface vethwepla5ed316 activated
2020-11-02T14:27:11.6005449Z [cluster1] 11[KNL] interface vethwepla5ed316 deactivated
2020-11-02T14:27:11.6005975Z [cluster1] 13[KNL] interface vethwepla5ed316 deleted
2020-11-02T14:27:11.6006480Z [cluster1] 11[KNL] interface vethwepg43d6b0e deleted
2020-11-02T14:27:11.6007250Z [cluster1] 03[KNL] interface vethwepl43d6b0e activated
2020-11-02T14:27:11.6007780Z [cluster1] 02[KNL] interface vethwepl43d6b0e deactivated
2020-11-02T14:27:11.6008400Z [cluster1] 04[KNL] interface vethwepl43d6b0e deleted
2020-11-02T14:27:11.6008913Z [cluster1] 10[KNL] interface vethwepg0db27e4 deleted
2020-11-02T14:27:11.6009410Z [cluster1] 16[KNL] interface vethwepl0db27e4 activated
2020-11-02T14:27:11.6009943Z [cluster1] 10[KNL] interface vethwepl0db27e4 deactivated
2020-11-02T14:27:11.6010467Z [cluster1] 12[KNL] interface vethwepl0db27e4 deleted
2020-11-02T14:27:11.6010967Z [cluster1] 11[KNL] interface vethwepg050a898 deleted
2020-11-02T14:27:11.6011475Z [cluster1] 14[KNL] interface vethwepl050a898 activated
2020-11-02T14:27:11.6011999Z [cluster1] 16[KNL] interface vethwepl050a898 deactivated
2020-11-02T14:27:11.6012513Z [cluster1] 04[KNL] interface vethwepl050a898 deleted
2020-11-02T14:27:11.6012907Z [cluster1] ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
2020-11-02T14:27:11.6014250Z [cluster1] +++++++++++++++++++++: Logs for Pod submariner-routeagent-ctc55 in namespace submariner-operator :++++++++++++++++++++++
2020-11-02T14:27:11.6015118Z [cluster1] + trap 'exit 1' SIGTERM SIGINT
2020-11-02T14:27:11.6015523Z [cluster1] + SUBMARINER_VERBOSITY=2
2020-11-02T14:27:11.6016035Z [cluster1] + '[' false == true ']'
2020-11-02T14:27:11.6016474Z [cluster1] + DEBUG=-v=2
2020-11-02T14:27:11.6016998Z [cluster1] + for f in iptables-save iptables
2020-11-02T14:27:11.6017612Z [cluster1] ++ find_iptables_on_host iptables-save
2020-11-02T14:27:11.6018253Z [cluster1] ++ chroot /host test -x /usr/sbin/iptables-save
2020-11-02T14:27:11.6018649Z [cluster1] ++ echo /usr/sbin
2020-11-02T14:27:11.6019108Z [cluster1] ++ return
2020-11-02T14:27:11.6019426Z [cluster1] + location=/usr/sbin
2020-11-02T14:27:11.6019950Z [cluster1] + '[' /usr/sbin '!=' unknown ']'
2020-11-02T14:27:11.6020667Z [cluster1] + echo 'iptables-save is present on the host at /usr/sbin/iptables-save'
2020-11-02T14:27:11.6021965Z [cluster1] + sed 's!@@PATH@@!/usr/sbin!' /usr/sbin/iptables-wrapper.in
2020-11-02T14:27:11.6022944Z [cluster1] iptables-save is present on the host at /usr/sbin/iptables-save
2020-11-02T14:27:11.6023745Z [cluster1] + for f in iptables-save iptables
2020-11-02T14:27:11.6024237Z [cluster1] ++ find_iptables_on_host iptables
2020-11-02T14:27:11.6025045Z [cluster1] ++ chroot /host test -x /usr/sbin/iptables
2020-11-02T14:27:11.6025439Z [cluster1] ++ echo /usr/sbin
2020-11-02T14:27:11.6025746Z [cluster1] ++ return
2020-11-02T14:27:11.6026393Z [cluster1] + location=/usr/sbin
2020-11-02T14:27:11.6026900Z [cluster1] + '[' /usr/sbin '!=' unknown ']'
2020-11-02T14:27:11.6027739Z [cluster1] + echo 'iptables is present on the host at /usr/sbin/iptables'
2020-11-02T14:27:11.6028531Z [cluster1] + sed 's!@@PATH@@!/usr/sbin!' /usr/sbin/iptables-wrapper.in
2020-11-02T14:27:11.6029331Z [cluster1] + exec submariner-route-agent -v=2 -alsologtostderr
2020-11-02T14:27:11.6029907Z [cluster1] iptables is present on the host at /usr/sbin/iptables
2020-11-02T14:27:11.6030681Z [cluster1] I1102 14:17:35.483795       1 main.go:47] Starting submariner-route-agent
2020-11-02T14:27:11.6032269Z [cluster1] W1102 14:17:35.499539       1 client_config.go:543] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
2020-11-02T14:27:11.6033899Z [cluster1] I1102 14:17:35.515087       1 route.go:192] Starting Route Controller. ClusterID: cluster1, localClusterCIDR: [10.1.0.0/16], localServiceCIDR: [100.1.0.0/16]
2020-11-02T14:27:11.6042965Z [cluster1] I1102 14:17:35.515119       1 route.go:196] Waiting for Endpoint informer caches to sync.
2020-11-02T14:27:11.6044667Z [cluster1] I1102 14:17:35.724140       1 route.go:656] Enqueueing sm-route-agent-pod event, ip: 172.17.0.5
2020-11-02T14:27:11.6047099Z [cluster1] I1102 14:17:35.724172       1 route.go:656] Enqueueing sm-route-agent-pod event, ip: 172.17.0.6
2020-11-02T14:27:11.6047960Z [cluster1] I1102 14:17:35.724689       1 driver.go:37] Interface "lo" has "127.0.0.1" address
2020-11-02T14:27:11.6049627Z [cluster1] I1102 14:17:35.724832       1 driver.go:37] Interface "weave" has "10.1.224.0" address
2020-11-02T14:27:11.6050585Z [cluster1] I1102 14:17:35.724841       1 driver.go:42] Found CNI Interface "weave" that has IP "10.1.224.0" from ClusterCIDR "10.1.0.0/16"
2020-11-02T14:27:11.6051557Z [cluster1] I1102 14:17:35.724906       1 driver.go:60] Successfully configured rp_filter to loose mode(2) on cniInterface "weave"
2020-11-02T14:27:11.6053184Z [cluster1] I1102 14:17:35.748818       1 iptables.go:22] Install/ensure SUBMARINER-POSTROUTING chain exists
2020-11-02T14:27:11.6054595Z [cluster1] I1102 14:17:36.215337       1 iptables.go:28] Insert SUBMARINER-POSTROUTING rule that has rules for inter-cluster traffic
2020-11-02T14:27:11.6056256Z [cluster1] I1102 14:17:36.272795       1 iptables.go:34] Install/ensure SUBMARINER-INPUT chain exists
2020-11-02T14:27:11.6058139Z [cluster1] I1102 14:17:36.501646       1 iptables.go:45] Allow VxLAN incoming traffic in SUBMARINER-INPUT Chain
2020-11-02T14:27:11.6059713Z [cluster1] I1102 14:17:36.544004       1 iptables.go:53] Insert rule to allow traffic over vx-submariner interface in FORWARDing Chain
2020-11-02T14:27:11.6061204Z [cluster1] I1102 14:17:36.580807       1 iptables.go:65] Installing rule for host network to remote cluster communication: -s 240.0.0.0/8 -o vx-submariner -j SNAT --to 10.1.224.0
2020-11-02T14:27:11.6062630Z [cluster1] I1102 14:17:36.754932       1 route.go:254] In cluster1, podIP of submariner-route-agent[0] is 172.17.0.5
2020-11-02T14:27:11.6063746Z [cluster1] I1102 14:17:36.755167       1 route.go:254] In cluster1, podIP of submariner-route-agent[1] is 172.17.0.6
2020-11-02T14:27:11.6065579Z [cluster1] I1102 14:17:36.755340       1 route.go:268] Hostname is "cluster1-worker" and routeAgentNodeName is "cluster1-worker"
2020-11-02T14:27:11.6066683Z [cluster1] I1102 14:17:36.797753       1 driver.go:87] Successfully annotated node "cluster1-worker" with cniIfaceIP "10.1.224.0"
2020-11-02T14:27:11.6067527Z [cluster1] I1102 14:17:36.801477       1 route.go:286] Route agent workers started
2020-11-02T14:27:11.6068677Z [cluster1] I1102 14:17:36.801565       1 route.go:507] Processing submariner-route-agent pod "submariner-routeagent-ctc55" with host IP "172.17.0.5", pod IP "172.17.0.5"
2020-11-02T14:27:11.6070375Z [cluster1] I1102 14:17:36.801619       1 route.go:531] Successfully processed submariner-route-agent pod "submariner-routeagent-ctc55"
2020-11-02T14:27:11.6095674Z [cluster1] I1102 14:17:36.801654       1 route.go:507] Processing submariner-route-agent pod "submariner-routeagent-lfc7d" with host IP "172.17.0.6", pod IP "172.17.0.6"
2020-11-02T14:27:11.6097289Z [cluster1] I1102 14:17:36.801674       1 route.go:531] Successfully processed submariner-route-agent pod "submariner-routeagent-lfc7d"
2020-11-02T14:27:11.6099669Z [cluster1] I1102 14:17:49.414636       1 route.go:558] Updating iptable rules for remote cluster's Endpoint "cluster2-submariner-cable-cluster2-172-17-0-9" with subnets [100.2.0.0/16 10.2.0.0/16]
2020-11-02T14:27:11.6101217Z [cluster1] I1102 14:17:49.454584       1 iptables.go:83] Installing iptables rule for outgoing traffic: -s 10.1.0.0/16 -d 100.2.0.0/16 -j ACCEPT
2020-11-02T14:27:11.6102689Z [cluster1] I1102 14:17:49.512016       1 iptables.go:91] Installing iptables rule for incoming traffic: -s 100.2.0.0/16 -d 10.1.0.0/16 -j ACCEPT
2020-11-02T14:27:11.6104094Z [cluster1] I1102 14:17:49.584452       1 iptables.go:83] Installing iptables rule for outgoing traffic: -s 10.1.0.0/16 -d 10.2.0.0/16 -j ACCEPT
2020-11-02T14:27:11.6105282Z [cluster1] I1102 14:17:49.617648       1 iptables.go:91] Installing iptables rule for incoming traffic: -s 10.2.0.0/16 -d 10.1.0.0/16 -j ACCEPT
2020-11-02T14:27:11.6107455Z [cluster1] I1102 14:17:49.668277       1 route.go:570] Processing local Endpoint cluster1-submariner-cable-cluster1-172-17-0-5 with IP 172.17.0.5, Host cluster1-worker for this cluster
2020-11-02T14:27:11.6108796Z [cluster1] I1102 14:17:49.669309       1 route.go:756] Processing route {Ifindex: 10 Dst: <nil> Src: <nil> Gw: 172.17.0.1 Flags: [] Table: 254}
2020-11-02T14:27:11.6109594Z [cluster1] I1102 14:17:49.669549       1 route.go:759] Found nil gw or dst
2020-11-02T14:27:11.6110275Z [cluster1] I1102 14:17:49.669644       1 route.go:756] Processing route {Ifindex: 4 Dst: 10.1.0.0/16 Src: 10.1.224.0 Gw: <nil> Flags: [] Table: 254}
2020-11-02T14:27:11.6110878Z [cluster1] I1102 14:17:49.669758       1 route.go:759] Found nil gw or dst
2020-11-02T14:27:11.6111883Z [cluster1] I1102 14:17:49.669852       1 route.go:756] Processing route {Ifindex: 10 Dst: 172.17.0.0/16 Src: 172.17.0.5 Gw: <nil> Flags: [] Table: 254}
2020-11-02T14:27:11.6112476Z [cluster1] I1102 14:17:49.671413       1 route.go:759] Found nil gw or dst
2020-11-02T14:27:11.6113058Z [cluster1] I1102 14:17:49.671599       1 route.go:582] This route agent is running on the active gateway node
2020-11-02T14:27:11.6114162Z [cluster1] I1102 14:17:49.671733       1 route.go:590] Creating the vxlan interface: vx-submariner on the gateway node
2020-11-02T14:27:11.6114921Z [cluster1] I1102 14:17:49.677052       1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.5 00:00:00:00:00:00
2020-11-02T14:27:11.6115594Z [cluster1] I1102 14:17:49.677964       1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.6 00:00:00:00:00:00
2020-11-02T14:27:11.6116628Z [cluster1] I1102 14:17:49.678217       1 route.go:464] Successfully configured rp_filter to loose mode(2) on vx-submariner
2020-11-02T14:27:11.6118665Z [cluster1] I1102 14:18:05.579179       1 route.go:558] Updating iptable rules for remote cluster's Endpoint "cluster2-submariner-cable-cluster2-172-17-0-9" with subnets [100.2.0.0/16 10.2.0.0/16]
2020-11-02T14:27:11.6120862Z [cluster1] I1102 14:18:05.579217       1 route.go:570] Processing local Endpoint cluster1-submariner-cable-cluster1-172-17-0-5 with IP 172.17.0.5, Host cluster1-worker for this cluster
2020-11-02T14:27:11.6122439Z [cluster1] I1102 14:18:05.579442       1 route.go:756] Processing route {Ifindex: 16 Dst: 240.0.0.0/8 Src: 240.17.0.5 Gw: <nil> Flags: [] Table: 254}
2020-11-02T14:27:11.6123029Z [cluster1] I1102 14:18:05.579464       1 route.go:759] Found nil gw or dst
2020-11-02T14:27:11.6123655Z [cluster1] I1102 14:18:05.579471       1 route.go:582] This route agent is running on the active gateway node
2020-11-02T14:27:11.6124667Z [cluster1] I1102 14:18:05.579478       1 route.go:590] Creating the vxlan interface: vx-submariner on the gateway node
2020-11-02T14:27:11.6125444Z [cluster1] I1102 14:18:05.579647       1 vxlan.go:61] VxLAN interface already exists with same configuration.
2020-11-02T14:27:11.6126413Z [cluster1] I1102 14:18:05.579685       1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.5 00:00:00:00:00:00
2020-11-02T14:27:11.6127036Z [cluster1] I1102 14:18:05.579713       1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.6 00:00:00:00:00:00
2020-11-02T14:27:11.6128186Z [cluster1] I1102 14:18:05.579774       1 route.go:464] Successfully configured rp_filter to loose mode(2) on vx-submariner
2020-11-02T14:27:11.6129805Z [cluster1] I1102 14:18:35.578962       1 route.go:558] Updating iptable rules for remote cluster's Endpoint "cluster2-submariner-cable-cluster2-172-17-0-9" with subnets [100.2.0.0/16 10.2.0.0/16]
2020-11-02T14:27:11.6132360Z [cluster1] I1102 14:18:35.579008       1 route.go:570] Processing local Endpoint cluster1-submariner-cable-cluster1-172-17-0-5 with IP 172.17.0.5, Host cluster1-worker for this cluster
2020-11-02T14:27:11.6133686Z [cluster1] I1102 14:18:35.579296       1 route.go:756] Processing route {Ifindex: 16 Dst: 240.0.0.0/8 Src: 240.17.0.5 Gw: <nil> Flags: [] Table: 254}
2020-11-02T14:27:11.6134286Z [cluster1] I1102 14:18:35.579327       1 route.go:759] Found nil gw or dst
2020-11-02T14:27:11.6138373Z [cluster1] I1102 14:18:35.579335       1 route.go:582] This route agent is running on the active gateway node
2020-11-02T14:27:11.6140170Z [cluster1] I1102 14:18:35.579341       1 route.go:590] Creating the vxlan interface: vx-submariner on the gateway node
2020-11-02T14:27:11.6141117Z [cluster1] I1102 14:18:35.579524       1 vxlan.go:61] VxLAN interface already exists with same configuration.
2020-11-02T14:27:11.6152097Z [cluster1] I1102 14:18:35.579560       1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.6 00:00:00:00:00:00
2020-11-02T14:27:11.6152853Z [cluster1] I1102 14:18:35.579590       1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.5 00:00:00:00:00:00
2020-11-02T14:27:11.6154694Z [cluster1] I1102 14:18:35.579670       1 route.go:464] Successfully configured rp_filter to loose mode(2) on vx-submariner
2020-11-02T14:27:11.6156792Z [cluster1] I1102 14:19:05.579264       1 route.go:570] Processing local Endpoint cluster1-submariner-cable-cluster1-172-17-0-5 with IP 172.17.0.5, Host cluster1-worker for this cluster
2020-11-02T14:27:11.6158090Z [cluster1] I1102 14:19:05.579471       1 route.go:756] Processing route {Ifindex: 16 Dst: 240.0.0.0/8 Src: 240.17.0.5 Gw: <nil> Flags: [] Table: 254}
2020-11-02T14:27:11.6160935Z [cluster1] I1102 14:19:05.579496       1 route.go:759] Found nil gw or dst
2020-11-02T14:27:11.6161491Z [cluster1] I1102 14:19:05.579503       1 route.go:582] This route agent is running on the active gateway node
2020-11-02T14:27:11.6162555Z [cluster1] I1102 14:19:05.579511       1 route.go:590] Creating the vxlan interface: vx-submariner on the gateway node
2020-11-02T14:27:11.6163708Z [cluster1] I1102 14:19:05.579647       1 vxlan.go:61] VxLAN interface already exists with same configuration.
2020-11-02T14:27:11.6164655Z [cluster1] I1102 14:19:05.579686       1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.5 00:00:00:00:00:00
2020-11-02T14:27:11.6165478Z [cluster1] I1102 14:19:05.579715       1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.6 00:00:00:00:00:00
2020-11-02T14:27:11.6167479Z [cluster1] I1102 14:19:05.579761       1 route.go:464] Successfully configured rp_filter to loose mode(2) on vx-submariner
2020-11-02T14:27:11.6169433Z [cluster1] I1102 14:19:05.579817       1 route.go:558] Updating iptable rules for remote cluster's Endpoint "cluster2-submariner-cable-cluster2-172-17-0-9" with subnets [100.2.0.0/16 10.2.0.0/16]
2020-11-02T14:27:11.6174136Z [cluster1] I1102 14:19:35.579560       1 route.go:558] Updating iptable rules for remote cluster's Endpoint "cluster2-submariner-cable-cluster2-172-17-0-9" with subnets [100.2.0.0/16 10.2.0.0/16]
2020-11-02T14:27:11.6177495Z [cluster1] I1102 14:19:35.579785       1 route.go:570] Processing local Endpoint cluster1-submariner-cable-cluster1-172-17-0-5 with IP 172.17.0.5, Host cluster1-worker for this cluster
2020-11-02T14:27:11.6183563Z [cluster1] I1102 14:19:35.580437       1 route.go:756] Processing route {Ifindex: 16 Dst: 240.0.0.0/8 Src: 240.17.0.5 Gw: <nil> Flags: [] Table: 254}
2020-11-02T14:27:11.6184181Z [cluster1] I1102 14:19:35.580523       1 route.go:759] Found nil gw or dst
2020-11-02T14:27:11.6184767Z [cluster1] I1102 14:19:35.580532       1 route.go:582] This route agent is running on the active gateway node
2020-11-02T14:27:11.6187513Z [cluster1] I1102 14:19:35.580539       1 route.go:590] Creating the vxlan interface: vx-submariner on the gateway node
2020-11-02T14:27:11.6188523Z [cluster1] I1102 14:19:35.580834       1 vxlan.go:61] VxLAN interface already exists with same configuration.
2020-11-02T14:27:11.6189384Z [cluster1] I1102 14:19:35.580885       1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.5 00:00:00:00:00:00
2020-11-02T14:27:11.6190187Z [cluster1] I1102 14:19:35.580922       1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.6 00:00:00:00:00:00
2020-11-02T14:27:11.6191508Z [cluster1] I1102 14:19:35.581020       1 route.go:464] Successfully configured rp_filter to loose mode(2) on vx-submariner
2020-11-02T14:27:11.6193895Z [cluster1] I1102 14:20:05.579446       1 route.go:570] Processing local Endpoint cluster1-submariner-cable-cluster1-172-17-0-5 with IP 172.17.0.5, Host cluster1-worker for this cluster
2020-11-02T14:27:11.6195340Z [cluster1] I1102 14:20:05.579837       1 route.go:756] Processing route {Ifindex: 16 Dst: 240.0.0.0/8 Src: 240.17.0.5 Gw: <nil> Flags: [] Table: 254}
2020-11-02T14:27:11.6196149Z [cluster1] I1102 14:20:05.579868       1 route.go:759] Found nil gw or dst
2020-11-02T14:27:11.6196698Z [cluster1] I1102 14:20:05.579875       1 route.go:582] This route agent is running on the active gateway node
2020-11-02T14:27:11.6197953Z [cluster1] I1102 14:20:05.579882       1 route.go:590] Creating the vxlan interface: vx-submariner on the gateway node
2020-11-02T14:27:11.6198732Z [cluster1] I1102 14:20:05.580039       1 vxlan.go:61] VxLAN interface already exists with same configuration.
2020-11-02T14:27:11.6201480Z [cluster1] I1102 14:20:05.580094       1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.5 00:00:00:00:00:00
2020-11-02T14:27:11.6202814Z [cluster1] I1102 14:20:05.580168       1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.6 00:00:00:00:00:00
2020-11-02T14:27:11.6204222Z [cluster1] I1102 14:20:05.580224       1 route.go:464] Successfully configured rp_filter to loose mode(2) on vx-submariner
2020-11-02T14:27:11.6206222Z [cluster1] I1102 14:20:05.580321       1 route.go:558] Updating iptable rules for remote cluster's Endpoint "cluster2-submariner-cable-cluster2-172-17-0-9" with subnets [100.2.0.0/16 10.2.0.0/16]
2020-11-02T14:27:11.6208404Z [cluster1] I1102 14:20:35.579738       1 route.go:570] Processing local Endpoint cluster1-submariner-cable-cluster1-172-17-0-5 with IP 172.17.0.5, Host cluster1-worker for this cluster
2020-11-02T14:27:11.6209637Z [cluster1] I1102 14:20:35.580039       1 route.go:756] Processing route {Ifindex: 16 Dst: 240.0.0.0/8 Src: 240.17.0.5 Gw: <nil> Flags: [] Table: 254}
2020-11-02T14:27:11.6210196Z [cluster1] I1102 14:20:35.580069       1 route.go:759] Found nil gw or dst
2020-11-02T14:27:11.6210913Z [cluster1] I1102 14:20:35.580077       1 route.go:582] This route agent is running on the active gateway node
2020-11-02T14:27:11.6213230Z [cluster1] I1102 14:20:35.580087       1 route.go:590] Creating the vxlan interface: vx-submariner on the gateway node
2020-11-02T14:27:11.6214010Z [cluster1] I1102 14:20:35.580238       1 vxlan.go:61] VxLAN interface already exists with same configuration.
2020-11-02T14:27:11.6214675Z [cluster1] I1102 14:20:35.580274       1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.5 00:00:00:00:00:00
2020-11-02T14:27:11.6215297Z [cluster1] I1102 14:20:35.580304       1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.6 00:00:00:00:00:00
2020-11-02T14:27:11.6216270Z [cluster1] I1102 14:20:35.580361       1 route.go:464] Successfully configured rp_filter to loose mode(2) on vx-submariner
2020-11-02T14:27:11.6218641Z [cluster1] I1102 14:20:35.580425       1 route.go:558] Updating iptable rules for remote cluster's Endpoint "cluster2-submariner-cable-cluster2-172-17-0-9" with subnets [100.2.0.0/16 10.2.0.0/16]
2020-11-02T14:27:11.6220984Z [cluster1] I1102 14:21:05.579971       1 route.go:558] Updating iptable rules for remote cluster's Endpoint "cluster2-submariner-cable-cluster2-172-17-0-9" with subnets [100.2.0.0/16 10.2.0.0/16]
2020-11-02T14:27:11.6223626Z [cluster1] I1102 14:21:05.580047       1 route.go:570] Processing local Endpoint cluster1-submariner-cable-cluster1-172-17-0-5 with IP 172.17.0.5, Host cluster1-worker for this cluster
2020-11-02T14:27:11.6225126Z [cluster1] I1102 14:21:05.580211       1 route.go:756] Processing route {Ifindex: 16 Dst: 240.0.0.0/8 Src: 240.17.0.5 Gw: <nil> Flags: [] Table: 254}
2020-11-02T14:27:11.6225891Z [cluster1] I1102 14:21:05.580239       1 route.go:759] Found nil gw or dst
2020-11-02T14:27:11.6226451Z [cluster1] I1102 14:21:05.580300       1 route.go:582] This route agent is running on the active gateway node
2020-11-02T14:27:11.6227436Z [cluster1] I1102 14:21:05.580314       1 route.go:590] Creating the vxlan interface: vx-submariner on the gateway node
2020-11-02T14:27:11.6228213Z [cluster1] I1102 14:21:05.580503       1 vxlan.go:61] VxLAN interface already exists with same configuration.
2020-11-02T14:27:11.6229240Z [cluster1] I1102 14:21:05.580540       1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.5 00:00:00:00:00:00
2020-11-02T14:27:11.6229993Z [cluster1] I1102 14:21:05.580568       1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.6 00:00:00:00:00:00
2020-11-02T14:27:11.6231019Z [cluster1] I1102 14:21:05.580626       1 route.go:464] Successfully configured rp_filter to loose mode(2) on vx-submariner
2020-11-02T14:27:11.6233003Z [cluster1] I1102 14:21:35.580085       1 route.go:570] Processing local Endpoint cluster1-submariner-cable-cluster1-172-17-0-5 with IP 172.17.0.5, Host cluster1-worker for this cluster
2020-11-02T14:27:11.6234329Z [cluster1] I1102 14:21:35.580294       1 route.go:756] Processing route {Ifindex: 16 Dst: 240.0.0.0/8 Src: 240.17.0.5 Gw: <nil> Flags: [] Table: 254}
2020-11-02T14:27:11.6234872Z [cluster1] I1102 14:21:35.580328       1 route.go:759] Found nil gw or dst
2020-11-02T14:27:11.6235410Z [cluster1] I1102 14:21:35.580336       1 route.go:582] This route agent is running on the active gateway node
2020-11-02T14:27:11.6236366Z [cluster1] I1102 14:21:35.580343       1 route.go:590] Creating the vxlan interface: vx-submariner on the gateway node
2020-11-02T14:27:11.6237115Z [cluster1] I1102 14:21:35.580491       1 vxlan.go:61] VxLAN interface already exists with same configuration.
2020-11-02T14:27:11.6237795Z [cluster1] I1102 14:21:35.580537       1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.5 00:00:00:00:00:00
2020-11-02T14:27:11.6238417Z [cluster1] I1102 14:21:35.580591       1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.6 00:00:00:00:00:00
2020-11-02T14:27:11.6239372Z [cluster1] I1102 14:21:35.580653       1 route.go:464] Successfully configured rp_filter to loose mode(2) on vx-submariner
2020-11-02T14:27:11.6240931Z [cluster1] I1102 14:21:35.580716       1 route.go:558] Updating iptable rules for remote cluster's Endpoint "cluster2-submariner-cable-cluster2-172-17-0-9" with subnets [100.2.0.0/16 10.2.0.0/16]
2020-11-02T14:27:11.6242959Z [cluster1] I1102 14:22:05.580221       1 route.go:558] Updating iptable rules for remote cluster's Endpoint "cluster2-submariner-cable-cluster2-172-17-0-9" with subnets [100.2.0.0/16 10.2.0.0/16]
2020-11-02T14:27:11.6245067Z [cluster1] I1102 14:22:05.580337       1 route.go:570] Processing local Endpoint cluster1-submariner-cable-cluster1-172-17-0-5 with IP 172.17.0.5, Host cluster1-worker for this cluster
2020-11-02T14:27:11.6246254Z [cluster1] I1102 14:22:05.580626       1 route.go:756] Processing route {Ifindex: 16 Dst: 240.0.0.0/8 Src: 240.17.0.5 Gw: <nil> Flags: [] Table: 254}
2020-11-02T14:27:11.6246791Z [cluster1] I1102 14:22:05.580692       1 route.go:759] Found nil gw or dst
2020-11-02T14:27:11.6247307Z [cluster1] I1102 14:22:05.580701       1 route.go:582] This route agent is running on the active gateway node
2020-11-02T14:27:11.6248222Z [cluster1] I1102 14:22:05.580708       1 route.go:590] Creating the vxlan interface: vx-submariner on the gateway node
2020-11-02T14:27:11.6248952Z [cluster1] I1102 14:22:05.580933       1 vxlan.go:61] VxLAN interface already exists with same configuration.
2020-11-02T14:27:11.6249610Z [cluster1] I1102 14:22:05.581009       1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.5 00:00:00:00:00:00
2020-11-02T14:27:11.6250192Z [cluster1] I1102 14:22:05.581059       1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.6 00:00:00:00:00:00
2020-11-02T14:27:11.6251116Z [cluster1] I1102 14:22:05.581165       1 route.go:464] Successfully configured rp_filter to loose mode(2) on vx-submariner
2020-11-02T14:27:11.6252613Z [cluster1] I1102 14:22:35.580350       1 route.go:558] Updating iptable rules for remote cluster's Endpoint "cluster2-submariner-cable-cluster2-172-17-0-9" with subnets [100.2.0.0/16 10.2.0.0/16]
2020-11-02T14:27:11.6256337Z [cluster1] I1102 14:22:35.580414       1 route.go:570] Processing local Endpoint cluster1-submariner-cable-cluster1-172-17-0-5 with IP 172.17.0.5, Host cluster1-worker for this cluster
2020-11-02T14:27:11.6258701Z [cluster1] I1102 14:22:35.580570       1 route.go:756] Processing route {Ifindex: 16 Dst: 240.0.0.0/8 Src: 240.17.0.5 Gw: <nil> Flags: [] Table: 254}
2020-11-02T14:27:11.6259377Z [cluster1] I1102 14:22:35.580596       1 route.go:759] Found nil gw or dst
2020-11-02T14:27:11.6259909Z [cluster1] I1102 14:22:35.580603       1 route.go:582] This route agent is running on the active gateway node
2020-11-02T14:27:11.6261084Z [cluster1] I1102 14:22:35.580609       1 route.go:590] Creating the vxlan interface: vx-submariner on the gateway node
2020-11-02T14:27:11.6262499Z [cluster1] I1102 14:22:35.580723       1 vxlan.go:61] VxLAN interface already exists with same configuration.
2020-11-02T14:27:11.6263410Z [cluster1] I1102 14:22:35.580755       1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.5 00:00:00:00:00:00
2020-11-02T14:27:11.6264062Z [cluster1] I1102 14:22:35.580782       1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.6 00:00:00:00:00:00
2020-11-02T14:27:11.6265541Z [cluster1] I1102 14:22:35.580840       1 route.go:464] Successfully configured rp_filter to loose mode(2) on vx-submariner
2020-11-02T14:27:11.6267486Z [cluster1] I1102 14:23:05.580467       1 route.go:558] Updating iptable rules for remote cluster's Endpoint "cluster2-submariner-cable-cluster2-172-17-0-9" with subnets [100.2.0.0/16 10.2.0.0/16]
2020-11-02T14:27:11.6269606Z [cluster1] I1102 14:23:05.580510       1 route.go:570] Processing local Endpoint cluster1-submariner-cable-cluster1-172-17-0-5 with IP 172.17.0.5, Host cluster1-worker for this cluster
2020-11-02T14:27:11.6271143Z [cluster1] I1102 14:23:05.580726       1 route.go:756] Processing route {Ifindex: 16 Dst: 240.0.0.0/8 Src: 240.17.0.5 Gw: <nil> Flags: [] Table: 254}
2020-11-02T14:27:11.6271844Z [cluster1] I1102 14:23:05.580756       1 route.go:759] Found nil gw or dst
2020-11-02T14:27:11.6272348Z [cluster1] I1102 14:23:05.580765       1 route.go:582] This route agent is running on the active gateway node
2020-11-02T14:27:11.6273314Z [cluster1] I1102 14:23:05.580772       1 route.go:590] Creating the vxlan interface: vx-submariner on the gateway node
2020-11-02T14:27:11.6274040Z [cluster1] I1102 14:23:05.580898       1 vxlan.go:61] VxLAN interface already exists with same configuration.
2020-11-02T14:27:11.6274696Z [cluster1] I1102 14:23:05.580941       1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.5 00:00:00:00:00:00
2020-11-02T14:27:11.6275300Z [cluster1] I1102 14:23:05.580982       1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.6 00:00:00:00:00:00
2020-11-02T14:27:11.6276224Z [cluster1] I1102 14:23:05.581045       1 route.go:464] Successfully configured rp_filter to loose mode(2) on vx-submariner
2020-11-02T14:27:11.6277729Z [cluster1] I1102 14:23:35.580626       1 route.go:558] Updating iptable rules for remote cluster's Endpoint "cluster2-submariner-cable-cluster2-172-17-0-9" with subnets [100.2.0.0/16 10.2.0.0/16]
2020-11-02T14:27:11.6280981Z [cluster1] I1102 14:23:35.580665       1 route.go:570] Processing local Endpoint cluster1-submariner-cable-cluster1-172-17-0-5 with IP 172.17.0.5, Host cluster1-worker for this cluster
2020-11-02T14:27:11.6282216Z [cluster1] I1102 14:23:35.580955       1 route.go:756] Processing route {Ifindex: 16 Dst: 240.0.0.0/8 Src: 240.17.0.5 Gw: <nil> Flags: [] Table: 254}
2020-11-02T14:27:11.6282757Z [cluster1] I1102 14:23:35.580989       1 route.go:759] Found nil gw or dst
2020-11-02T14:27:11.6283617Z [cluster1] I1102 14:23:35.580998       1 route.go:582] This route agent is running on the active gateway node
2020-11-02T14:27:11.6285105Z [cluster1] I1102 14:23:35.581005       1 route.go:590] Creating the vxlan interface: vx-submariner on the gateway node
2020-11-02T14:27:11.6285885Z [cluster1] I1102 14:23:35.581149       1 vxlan.go:61] VxLAN interface already exists with same configuration.
2020-11-02T14:27:11.6286588Z [cluster1] I1102 14:23:35.581215       1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.6 00:00:00:00:00:00
2020-11-02T14:27:11.6287446Z [cluster1] I1102 14:23:35.581259       1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.5 00:00:00:00:00:00
2020-11-02T14:27:11.6288567Z [cluster1] I1102 14:23:35.581314       1 route.go:464] Successfully configured rp_filter to loose mode(2) on vx-submariner
2020-11-02T14:27:11.6291047Z [cluster1] I1102 14:24:05.580838       1 route.go:558] Updating iptable rules for remote cluster's Endpoint "cluster2-submariner-cable-cluster2-172-17-0-9" with subnets [100.2.0.0/16 10.2.0.0/16]
2020-11-02T14:27:11.6295051Z [cluster1] I1102 14:24:05.580883       1 route.go:570] Processing local Endpoint cluster1-submariner-cable-cluster1-172-17-0-5 with IP 172.17.0.5, Host cluster1-worker for this cluster
2020-11-02T14:27:11.6296287Z [cluster1] I1102 14:24:05.581303       1 route.go:756] Processing route {Ifindex: 16 Dst: 240.0.0.0/8 Src: 240.17.0.5 Gw: <nil> Flags: [] Table: 254}
2020-11-02T14:27:11.6297289Z [cluster1] I1102 14:24:05.581339       1 route.go:759] Found nil gw or dst
2020-11-02T14:27:11.6298038Z [cluster1] I1102 14:24:05.581348       1 route.go:582] This route agent is running on the active gateway node
2020-11-02T14:27:11.6299113Z [cluster1] I1102 14:24:05.581356       1 route.go:590] Creating the vxlan interface: vx-submariner on the gateway node
2020-11-02T14:27:11.6299867Z [cluster1] I1102 14:24:05.581553       1 vxlan.go:61] VxLAN interface already exists with same configuration.
2020-11-02T14:27:11.6301238Z [lighthouse]$ [cluster1] kubectl -n submariner-operator logs submariner-routeagent-lfc7d
2020-11-02T14:27:11.6302961Z [lighthouse]$ [cluster1] kubectl -n submariner-operator logs submariner-routeagent-lfc7d
2020-11-02T14:27:11.6304421Z [lighthouse]$ [cluster1] command kubectl --context=cluster1 -n submariner-operator logs submariner-routeagent-lfc7d
2020-11-02T14:27:11.6307474Z [lighthouse]$ [cluster1] kubectl --context=cluster1 -n submariner-operator logs submariner-routeagent-lfc7d
2020-11-02T14:27:11.6308534Z [cluster1] I1102 14:24:05.581651       1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.5 00:00:00:00:00:00
2020-11-02T14:27:11.6309227Z [cluster1] I1102 14:24:05.581704       1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.6 00:00:00:00:00:00
2020-11-02T14:27:11.6310467Z [cluster1] I1102 14:24:05.581811       1 route.go:464] Successfully configured rp_filter to loose mode(2) on vx-submariner
2020-11-02T14:27:11.6313526Z [cluster1] I1102 14:24:35.580988       1 route.go:558] Updating iptable rules for remote cluster's Endpoint "cluster2-submariner-cable-cluster2-172-17-0-9" with subnets [100.2.0.0/16 10.2.0.0/16]
2020-11-02T14:27:11.6315732Z [cluster1] I1102 14:24:35.581070       1 route.go:570] Processing local Endpoint cluster1-submariner-cable-cluster1-172-17-0-5 with IP 172.17.0.5, Host cluster1-worker for this cluster
2020-11-02T14:27:11.6317056Z [cluster1] I1102 14:24:35.581270       1 route.go:756] Processing route {Ifindex: 16 Dst: 240.0.0.0/8 Src: 240.17.0.5 Gw: <nil> Flags: [] Table: 254}
2020-11-02T14:27:11.6317664Z [cluster1] I1102 14:24:35.581310       1 route.go:759] Found nil gw or dst
2020-11-02T14:27:11.6318226Z [cluster1] I1102 14:24:35.581318       1 route.go:582] This route agent is running on the active gateway node
2020-11-02T14:27:11.6319588Z [cluster1] I1102 14:24:35.581325       1 route.go:590] Creating the vxlan interface: vx-submariner on the gateway node
2020-11-02T14:27:11.6320530Z [cluster1] I1102 14:24:35.581489       1 vxlan.go:61] VxLAN interface already exists with same configuration.
2020-11-02T14:27:11.6321189Z [cluster1] I1102 14:24:35.581535       1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.5 00:00:00:00:00:00
2020-11-02T14:27:11.6321791Z [cluster1] I1102 14:24:35.581575       1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.6 00:00:00:00:00:00
2020-11-02T14:27:11.6322731Z [cluster1] I1102 14:24:35.581634       1 route.go:464] Successfully configured rp_filter to loose mode(2) on vx-submariner
2020-11-02T14:27:11.6324484Z [cluster1] I1102 14:25:05.581152       1 route.go:570] Processing local Endpoint cluster1-submariner-cable-cluster1-172-17-0-5 with IP 172.17.0.5, Host cluster1-worker for this cluster
2020-11-02T14:27:11.6325763Z [cluster1] I1102 14:25:05.581339       1 route.go:756] Processing route {Ifindex: 16 Dst: 240.0.0.0/8 Src: 240.17.0.5 Gw: <nil> Flags: [] Table: 254}
2020-11-02T14:27:11.6326315Z [cluster1] I1102 14:25:05.581410       1 route.go:759] Found nil gw or dst
2020-11-02T14:27:11.6326833Z [cluster1] I1102 14:25:05.581418       1 route.go:582] This route agent is running on the active gateway node
2020-11-02T14:27:11.6327778Z [cluster1] I1102 14:25:05.581425       1 route.go:590] Creating the vxlan interface: vx-submariner on the gateway node
2020-11-02T14:27:11.6328573Z [cluster1] I1102 14:25:05.581566       1 vxlan.go:61] VxLAN interface already exists with same configuration.
2020-11-02T14:27:11.6329229Z [cluster1] I1102 14:25:05.581620       1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.5 00:00:00:00:00:00
2020-11-02T14:27:11.6329839Z [cluster1] I1102 14:25:05.581670       1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.6 00:00:00:00:00:00
2020-11-02T14:27:11.6330750Z [cluster1] I1102 14:25:05.581755       1 route.go:464] Successfully configured rp_filter to loose mode(2) on vx-submariner
2020-11-02T14:27:11.6332480Z [cluster1] I1102 14:25:05.581838       1 route.go:558] Updating iptable rules for remote cluster's Endpoint "cluster2-submariner-cable-cluster2-172-17-0-9" with subnets [100.2.0.0/16 10.2.0.0/16]
2020-11-02T14:27:11.6336489Z [cluster1] I1102 14:25:35.581261       1 route.go:570] Processing local Endpoint cluster1-submariner-cable-cluster1-172-17-0-5 with IP 172.17.0.5, Host cluster1-worker for this cluster
2020-11-02T14:27:11.6338321Z [cluster1] I1102 14:25:35.581644       1 route.go:756] Processing route {Ifindex: 16 Dst: 240.0.0.0/8 Src: 240.17.0.5 Gw: <nil> Flags: [] Table: 254}
2020-11-02T14:27:11.6338991Z [cluster1] I1102 14:25:35.581678       1 route.go:759] Found nil gw or dst
2020-11-02T14:27:11.6339557Z [cluster1] I1102 14:25:35.581686       1 route.go:582] This route agent is running on the active gateway node
2020-11-02T14:27:11.6342537Z [cluster1] I1102 14:25:35.581693       1 route.go:590] Creating the vxlan interface: vx-submariner on the gateway node
2020-11-02T14:27:11.6343659Z [cluster1] I1102 14:25:35.581988       1 vxlan.go:61] VxLAN interface already exists with same configuration.
2020-11-02T14:27:11.6344518Z [cluster1] I1102 14:25:35.582102       1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.5 00:00:00:00:00:00
2020-11-02T14:27:11.6345306Z [cluster1] I1102 14:25:35.582148       1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.6 00:00:00:00:00:00
2020-11-02T14:27:11.6347079Z [cluster1] I1102 14:25:35.582258       1 route.go:464] Successfully configured rp_filter to loose mode(2) on vx-submariner
2020-11-02T14:27:11.6349695Z [cluster1] I1102 14:25:35.582331       1 route.go:558] Updating iptable rules for remote cluster's Endpoint "cluster2-submariner-cable-cluster2-172-17-0-9" with subnets [100.2.0.0/16 10.2.0.0/16]
2020-11-02T14:27:11.6352919Z [cluster1] I1102 14:26:05.581510       1 route.go:570] Processing local Endpoint cluster1-submariner-cable-cluster1-172-17-0-5 with IP 172.17.0.5, Host cluster1-worker for this cluster
2020-11-02T14:27:11.6354339Z [cluster1] I1102 14:26:05.581776       1 route.go:756] Processing route {Ifindex: 16 Dst: 240.0.0.0/8 Src: 240.17.0.5 Gw: <nil> Flags: [] Table: 254}
2020-11-02T14:27:11.6354921Z [cluster1] I1102 14:26:05.581811       1 route.go:759] Found nil gw or dst
2020-11-02T14:27:11.6355641Z [cluster1] I1102 14:26:05.581819       1 route.go:582] This route agent is running on the active gateway node
2020-11-02T14:27:11.6356755Z [cluster1] I1102 14:26:05.581825       1 route.go:590] Creating the vxlan interface: vx-submariner on the gateway node
2020-11-02T14:27:11.6357479Z [cluster1] I1102 14:26:05.581990       1 vxlan.go:61] VxLAN interface already exists with same configuration.
2020-11-02T14:27:11.6358327Z [cluster1] I1102 14:26:05.582035       1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.5 00:00:00:00:00:00
2020-11-02T14:27:11.6359459Z [cluster1] I1102 14:26:05.582070       1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.6 00:00:00:00:00:00
2020-11-02T14:27:11.6360547Z [cluster1] I1102 14:26:05.582120       1 route.go:464] Successfully configured rp_filter to loose mode(2) on vx-submariner
2020-11-02T14:27:11.6362750Z [cluster1] I1102 14:26:05.582177       1 route.go:558] Updating iptable rules for remote cluster's Endpoint "cluster2-submariner-cable-cluster2-172-17-0-9" with subnets [100.2.0.0/16 10.2.0.0/16]
2020-11-02T14:27:11.6365199Z [cluster1] I1102 14:26:35.581705       1 route.go:558] Updating iptable rules for remote cluster's Endpoint "cluster2-submariner-cable-cluster2-172-17-0-9" with subnets [100.2.0.0/16 10.2.0.0/16]
2020-11-02T14:27:11.6367422Z [cluster1] I1102 14:26:35.581743       1 route.go:570] Processing local Endpoint cluster1-submariner-cable-cluster1-172-17-0-5 with IP 172.17.0.5, Host cluster1-worker for this cluster
2020-11-02T14:27:11.6368664Z [cluster1] I1102 14:26:35.581945       1 route.go:756] Processing route {Ifindex: 16 Dst: 240.0.0.0/8 Src: 240.17.0.5 Gw: <nil> Flags: [] Table: 254}
2020-11-02T14:27:11.6369221Z [cluster1] I1102 14:26:35.581995       1 route.go:759] Found nil gw or dst
2020-11-02T14:27:11.6369760Z [cluster1] I1102 14:26:35.582018       1 route.go:582] This route agent is running on the active gateway node
2020-11-02T14:27:11.6371070Z [cluster1] I1102 14:26:35.582044       1 route.go:590] Creating the vxlan interface: vx-submariner on the gateway node
2020-11-02T14:27:11.6371818Z [cluster1] I1102 14:26:35.582200       1 vxlan.go:61] VxLAN interface already exists with same configuration.
2020-11-02T14:27:11.6373994Z [cluster1] I1102 14:26:35.582245       1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.5 00:00:00:00:00:00
2020-11-02T14:27:11.6374671Z [cluster1] I1102 14:26:35.582276       1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.6 00:00:00:00:00:00
2020-11-02T14:27:11.6375753Z [cluster1] I1102 14:26:35.582324       1 route.go:464] Successfully configured rp_filter to loose mode(2) on vx-submariner
2020-11-02T14:27:11.6377473Z [cluster1] I1102 14:27:05.581771       1 route.go:558] Updating iptable rules for remote cluster's Endpoint "cluster2-submariner-cable-cluster2-172-17-0-9" with subnets [100.2.0.0/16 10.2.0.0/16]
2020-11-02T14:27:11.6379503Z [cluster1] I1102 14:27:05.581818       1 route.go:570] Processing local Endpoint cluster1-submariner-cable-cluster1-172-17-0-5 with IP 172.17.0.5, Host cluster1-worker for this cluster
2020-11-02T14:27:11.6380740Z [cluster1] I1102 14:27:05.582010       1 route.go:756] Processing route {Ifindex: 16 Dst: 240.0.0.0/8 Src: 240.17.0.5 Gw: <nil> Flags: [] Table: 254}
2020-11-02T14:27:11.6381735Z [cluster1] I1102 14:27:05.582030       1 route.go:759] Found nil gw or dst
2020-11-02T14:27:11.6382575Z [cluster1] I1102 14:27:05.582038       1 route.go:582] This route agent is running on the active gateway node
2020-11-02T14:27:11.6384670Z [cluster1] I1102 14:27:05.582045       1 route.go:590] Creating the vxlan interface: vx-submariner on the gateway node
2020-11-02T14:27:11.6385620Z [cluster1] I1102 14:27:05.582176       1 vxlan.go:61] VxLAN interface already exists with same configuration.
2020-11-02T14:27:11.6386300Z [cluster1] I1102 14:27:05.582211       1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.5 00:00:00:00:00:00
2020-11-02T14:27:11.6386919Z [cluster1] I1102 14:27:05.582241       1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.6 00:00:00:00:00:00
2020-11-02T14:27:11.9280613Z [cluster1] I1102 14:27:05.582297       1 route.go:464] Successfully configured rp_filter to loose mode(2) on vx-submariner
2020-11-02T14:27:11.9282247Z [cluster1] +++++++++++++++++++++: Logs for Pod submariner-routeagent-lfc7d in namespace submariner-operator :++++++++++++++++++++++
2020-11-02T14:27:11.9283509Z [cluster1] + trap 'exit 1' SIGTERM SIGINT
2020-11-02T14:27:11.9284119Z [cluster1] + SUBMARINER_VERBOSITY=2
2020-11-02T14:27:11.9284660Z [cluster1] + '[' false == true ']'
2020-11-02T14:27:11.9285250Z [cluster1] + DEBUG=-v=2
2020-11-02T14:27:11.9285830Z [cluster1] + for f in iptables-save iptables
2020-11-02T14:27:11.9286691Z [cluster1] ++ find_iptables_on_host iptables-save
2020-11-02T14:27:11.9287405Z [cluster1] ++ chroot /host test -x /usr/sbin/iptables-save
2020-11-02T14:27:11.9288216Z [cluster1] ++ echo /usr/sbin
2020-11-02T14:27:11.9288560Z [cluster1] ++ return
2020-11-02T14:27:11.9289136Z [cluster1] + location=/usr/sbin
2020-11-02T14:27:11.9293571Z [cluster1] + '[' /usr/sbin '!=' unknown ']'
2020-11-02T14:27:11.9294452Z [cluster1] + echo 'iptables-save is present on the host at /usr/sbin/iptables-save'
2020-11-02T14:27:11.9295296Z [cluster1] + sed 's!@@PATH@@!/usr/sbin!' /usr/sbin/iptables-wrapper.in
2020-11-02T14:27:11.9296288Z [cluster1] iptables-save is present on the host at /usr/sbin/iptables-save
2020-11-02T14:27:11.9297016Z [cluster1] + for f in iptables-save iptables
2020-11-02T14:27:11.9297467Z [cluster1] ++ find_iptables_on_host iptables
2020-11-02T14:27:11.9298331Z [cluster1] ++ chroot /host test -x /usr/sbin/iptables
2020-11-02T14:27:11.9298735Z [cluster1] ++ echo /usr/sbin
2020-11-02T14:27:11.9299054Z [cluster1] ++ return
2020-11-02T14:27:11.9299393Z [cluster1] + location=/usr/sbin
2020-11-02T14:27:11.9299904Z [cluster1] + '[' /usr/sbin '!=' unknown ']'
2020-11-02T14:27:11.9300800Z [cluster1] + echo 'iptables is present on the host at /usr/sbin/iptables'
2020-11-02T14:27:11.9302131Z [cluster1] + sed 's!@@PATH@@!/usr/sbin!' /usr/sbin/iptables-wrapper.in
2020-11-02T14:27:11.9304951Z [cluster1] + exec submariner-route-agent -v=2 -alsologtostderr
2020-11-02T14:27:11.9305618Z [cluster1] iptables is present on the host at /usr/sbin/iptables
2020-11-02T14:27:11.9306593Z [cluster1] I1102 14:17:35.663973       1 main.go:47] Starting submariner-route-agent
2020-11-02T14:27:11.9307709Z [cluster1] W1102 14:17:35.665430       1 client_config.go:543] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
2020-11-02T14:27:11.9308790Z [cluster1] I1102 14:17:35.672863       1 route.go:192] Starting Route Controller. ClusterID: cluster1, localClusterCIDR: [10.1.0.0/16], localServiceCIDR: [100.1.0.0/16]
2020-11-02T14:27:11.9309941Z [cluster1] I1102 14:17:35.694618       1 route.go:196] Waiting for Endpoint informer caches to sync.
2020-11-02T14:27:11.9314445Z [cluster1] I1102 14:17:35.749765       1 route.go:656] Enqueueing sm-route-agent-pod event, ip: 172.17.0.5
2020-11-02T14:27:11.9315496Z [cluster1] I1102 14:17:35.751122       1 route.go:656] Enqueueing sm-route-agent-pod event, ip: 172.17.0.6
2020-11-02T14:27:11.9316132Z [cluster1] I1102 14:17:35.795941       1 driver.go:37] Interface "lo" has "127.0.0.1" address
2020-11-02T14:27:11.9316683Z [cluster1] I1102 14:17:35.796034       1 driver.go:37] Interface "weave" has "10.1.0.1" address
2020-11-02T14:27:11.9317478Z [cluster1] I1102 14:17:35.796082       1 driver.go:42] Found CNI Interface "weave" that has IP "10.1.0.1" from ClusterCIDR "10.1.0.0/16"
2020-11-02T14:27:11.9318284Z [cluster1] I1102 14:17:35.796258       1 driver.go:60] Successfully configured rp_filter to loose mode(2) on cniInterface "weave"
2020-11-02T14:27:11.9319354Z [cluster1] I1102 14:17:35.819397       1 iptables.go:22] Install/ensure SUBMARINER-POSTROUTING chain exists
2020-11-02T14:27:11.9320511Z [cluster1] I1102 14:17:36.137808       1 iptables.go:28] Insert SUBMARINER-POSTROUTING rule that has rules for inter-cluster traffic
2020-11-02T14:27:11.9321577Z [cluster1] I1102 14:17:36.163402       1 iptables.go:34] Install/ensure SUBMARINER-INPUT chain exists
2020-11-02T14:27:11.9322544Z [cluster1] I1102 14:17:36.219946       1 iptables.go:45] Allow VxLAN incoming traffic in SUBMARINER-INPUT Chain
2020-11-02T14:27:11.9323609Z [cluster1] I1102 14:17:36.255017       1 iptables.go:53] Insert rule to allow traffic over vx-submariner interface in FORWARDing Chain
2020-11-02T14:27:11.9325026Z [cluster1] I1102 14:17:36.294237       1 iptables.go:65] Installing rule for host network to remote cluster communication: -s 240.0.0.0/8 -o vx-submariner -j SNAT --to 10.1.0.1
2020-11-02T14:27:11.9326234Z [cluster1] I1102 14:17:36.355387       1 route.go:254] In cluster1, podIP of submariner-route-agent[0] is 172.17.0.5
2020-11-02T14:27:11.9327276Z [cluster1] I1102 14:17:36.355413       1 route.go:254] In cluster1, podIP of submariner-route-agent[1] is 172.17.0.6
2020-11-02T14:27:11.9328413Z [cluster1] I1102 14:17:36.355419       1 route.go:268] Hostname is "cluster1-worker2" and routeAgentNodeName is "cluster1-worker2"
2020-11-02T14:27:11.9329583Z [cluster1] I1102 14:17:36.369924       1 driver.go:87] Successfully annotated node "cluster1-worker2" with cniIfaceIP "10.1.0.1"
2020-11-02T14:27:11.9330279Z [cluster1] I1102 14:17:36.369949       1 route.go:286] Route agent workers started
2020-11-02T14:27:11.9331435Z [cluster1] I1102 14:17:36.369974       1 route.go:507] Processing submariner-route-agent pod "submariner-routeagent-ctc55" with host IP "172.17.0.5", pod IP "172.17.0.5"
2020-11-02T14:27:11.9333009Z [cluster1] I1102 14:17:36.369981       1 route.go:531] Successfully processed submariner-route-agent pod "submariner-routeagent-ctc55"
2020-11-02T14:27:11.9335068Z [cluster1] I1102 14:17:36.370004       1 route.go:507] Processing submariner-route-agent pod "submariner-routeagent-lfc7d" with host IP "172.17.0.6", pod IP "172.17.0.6"
2020-11-02T14:27:11.9336665Z [cluster1] I1102 14:17:36.370009       1 route.go:531] Successfully processed submariner-route-agent pod "submariner-routeagent-lfc7d"
2020-11-02T14:27:11.9338834Z [cluster1] I1102 14:17:49.412203       1 route.go:558] Updating iptable rules for remote cluster's Endpoint "cluster2-submariner-cable-cluster2-172-17-0-9" with subnets [100.2.0.0/16 10.2.0.0/16]
2020-11-02T14:27:11.9340325Z [cluster1] I1102 14:17:49.426962       1 iptables.go:83] Installing iptables rule for outgoing traffic: -s 10.1.0.0/16 -d 100.2.0.0/16 -j ACCEPT
2020-11-02T14:27:11.9341337Z [cluster1] I1102 14:17:49.445119       1 iptables.go:91] Installing iptables rule for incoming traffic: -s 100.2.0.0/16 -d 10.1.0.0/16 -j ACCEPT
2020-11-02T14:27:11.9342780Z [cluster1] I1102 14:17:49.485944       1 iptables.go:83] Installing iptables rule for outgoing traffic: -s 10.1.0.0/16 -d 10.2.0.0/16 -j ACCEPT
2020-11-02T14:27:11.9343853Z [cluster1] I1102 14:17:49.501221       1 iptables.go:91] Installing iptables rule for incoming traffic: -s 10.2.0.0/16 -d 10.1.0.0/16 -j ACCEPT
2020-11-02T14:27:11.9345649Z [cluster1] I1102 14:17:49.539983       1 route.go:570] Processing local Endpoint cluster1-submariner-cable-cluster1-172-17-0-5 with IP 172.17.0.5, Host cluster1-worker for this cluster
2020-11-02T14:27:11.9347656Z [cluster1] I1102 14:17:49.540121       1 route.go:612] This route agent is running on a non-gateway/non-active node
2020-11-02T14:27:11.9348663Z [cluster1] I1102 14:17:49.540231       1 route.go:623] Creating the vxlan interface vx-submariner with gateway node IP 172.17.0.5
2020-11-02T14:27:11.9349506Z [cluster1] I1102 14:17:49.564254       1 route.go:771] Reconciling routes to gw: 240.17.0.5
2020-11-02T14:27:11.9350145Z [cluster1] I1102 14:17:49.564520       1 route.go:787] Processing route {Ifindex: 16 Dst: 240.0.0.0/8 Src: 240.17.0.6 Gw: <nil> Flags: [] Table: 254}
2020-11-02T14:27:11.9350721Z [cluster1] I1102 14:17:49.564656       1 route.go:790] Found nil gw or dst
2020-11-02T14:27:11.9377606Z [cluster1] I1102 14:17:49.564863       1 route.go:647] Successfully processed local Endpoint "cluster1-submariner-cable-cluster1-172-17-0-5"
2020-11-02T14:27:11.9381770Z [cluster1] I1102 14:18:05.712758       1 route.go:570] Processing local Endpoint cluster1-submariner-cable-cluster1-172-17-0-5 with IP 172.17.0.5, Host cluster1-worker for this cluster
2020-11-02T14:27:11.9383458Z [cluster1] I1102 14:18:05.712862       1 route.go:612] This route agent is running on a non-gateway/non-active node
2020-11-02T14:27:11.9384437Z [cluster1] I1102 14:18:05.712880       1 route.go:623] Creating the vxlan interface vx-submariner with gateway node IP 172.17.0.5
2020-11-02T14:27:11.9385431Z [cluster1] I1102 14:18:05.713105       1 vxlan.go:61] VxLAN interface already exists with same configuration.
2020-11-02T14:27:11.9386064Z [cluster1] I1102 14:18:05.718156       1 route.go:771] Reconciling routes to gw: 240.17.0.5
2020-11-02T14:27:11.9386672Z [cluster1] I1102 14:18:05.718310       1 route.go:787] Processing route {Ifindex: 16 Dst: 10.2.0.0/16 Src: <nil> Gw: 240.17.0.5 Flags: [] Table: 254}
2020-11-02T14:27:11.9387756Z [cluster1] I1102 14:18:05.718336       1 route.go:793] Found route {Ifindex: 16 Dst: 10.2.0.0/16 Src: <nil> Gw: 240.17.0.5 Flags: [] Table: 254} with gw 240.17.0.5 already installed
2020-11-02T14:27:11.9388622Z [cluster1] I1102 14:18:05.718346       1 route.go:787] Processing route {Ifindex: 16 Dst: 100.2.0.0/16 Src: <nil> Gw: 240.17.0.5 Flags: [] Table: 254}
2020-11-02T14:27:11.9405077Z [cluster1] I1102 14:18:05.718354       1 route.go:793] Found route {Ifindex: 16 Dst: 100.2.0.0/16 Src: <nil> Gw: 240.17.0.5 Flags: [] Table: 254} with gw 240.17.0.5 already installed
2020-11-02T14:27:11.9405943Z [cluster1] I1102 14:18:05.718363       1 route.go:787] Processing route {Ifindex: 16 Dst: 240.0.0.0/8 Src: 240.17.0.6 Gw: <nil> Flags: [] Table: 254}
2020-11-02T14:27:11.9406548Z [cluster1] I1102 14:18:05.718380       1 route.go:790] Found nil gw or dst
2020-11-02T14:27:11.9407085Z [cluster1] I1102 14:18:05.718443       1 route.go:829] Found equivalent route, not adding
2020-11-02T14:27:11.9407690Z [cluster1] I1102 14:18:05.718452       1 route.go:829] Found equivalent route, not adding
2020-11-02T14:27:11.9409467Z [cluster1] I1102 14:18:05.718499       1 route.go:647] Successfully processed local Endpoint "cluster1-submariner-cable-cluster1-172-17-0-5"
2020-11-02T14:27:11.9412236Z [cluster1] I1102 14:18:05.718532       1 route.go:558] Updating iptable rules for remote cluster's Endpoint "cluster2-submariner-cable-cluster2-172-17-0-9" with subnets [100.2.0.0/16 10.2.0.0/16]
2020-11-02T14:27:11.9414395Z [cluster1] I1102 14:18:35.712810       1 route.go:558] Updating iptable rules for remote cluster's Endpoint "cluster2-submariner-cable-cluster2-172-17-0-9" with subnets [100.2.0.0/16 10.2.0.0/16]
2020-11-02T14:27:11.9416803Z [cluster1] I1102 14:18:35.712880       1 route.go:570] Processing local Endpoint cluster1-submariner-cable-cluster1-172-17-0-5 with IP 172.17.0.5, Host cluster1-worker for this cluster
2020-11-02T14:27:11.9418550Z [cluster1] I1102 14:18:35.712891       1 route.go:612] This route agent is running on a non-gateway/non-active node
2020-11-02T14:27:11.9420126Z [cluster1] I1102 14:18:35.712900       1 route.go:623] Creating the vxlan interface vx-submariner with gateway node IP 172.17.0.5
2020-11-02T14:27:11.9420835Z [cluster1] I1102 14:18:35.713151       1 vxlan.go:61] VxLAN interface already exists with same configuration.
2020-11-02T14:27:11.9421589Z [cluster1] I1102 14:18:35.717490       1 route.go:771] Reconciling routes to gw: 240.17.0.5
2020-11-02T14:27:11.9422181Z [cluster1] I1102 14:18:35.717639       1 route.go:787] Processing route {Ifindex: 16 Dst: 10.2.0.0/16 Src: <nil> Gw: 240.17.0.5 Flags: [] Table: 254}
2020-11-02T14:27:11.9422887Z [cluster1] I1102 14:18:35.717663       1 route.go:793] Found route {Ifindex: 16 Dst: 10.2.0.0/16 Src: <nil> Gw: 240.17.0.5 Flags: [] Table: 254} with gw 240.17.0.5 already installed
2020-11-02T14:27:11.9423591Z [cluster1] I1102 14:18:35.717673       1 route.go:787] Processing route {Ifindex: 16 Dst: 100.2.0.0/16 Src: <nil> Gw: 240.17.0.5 Flags: [] Table: 254}
2020-11-02T14:27:11.9424296Z [cluster1] I1102 14:18:35.717683       1 route.go:793] Found route {Ifindex: 16 Dst: 100.2.0.0/16 Src: <nil> Gw: 240.17.0.5 Flags: [] Table: 254} with gw 240.17.0.5 already installed
2020-11-02T14:27:11.9425152Z [cluster1] I1102 14:18:35.717691       1 route.go:787] Processing route {Ifindex: 16 Dst: 240.0.0.0/8 Src: 240.17.0.6 Gw: <nil> Flags: [] Table: 254}
2020-11-02T14:27:11.9425669Z [cluster1] I1102 14:18:35.717700       1 route.go:790] Found nil gw or dst
2020-11-02T14:27:11.9426480Z [cluster1] I1102 14:18:35.717778       1 route.go:829] Found equivalent route, not adding
2020-11-02T14:27:11.9427085Z [cluster1] I1102 14:18:35.717787       1 route.go:829] Found equivalent route, not adding
2020-11-02T14:27:11.9428398Z [cluster1] I1102 14:18:35.717817       1 route.go:647] Successfully processed local Endpoint "cluster1-submariner-cable-cluster1-172-17-0-5"
2020-11-02T14:27:11.9430396Z [cluster1] I1102 14:19:05.712936       1 route.go:558] Updating iptable rules for remote cluster's Endpoint "cluster2-submariner-cable-cluster2-172-17-0-9" with subnets [100.2.0.0/16 10.2.0.0/16]
2020-11-02T14:27:11.9441822Z [cluster1] I1102 14:19:05.712975       1 route.go:570] Processing local Endpoint cluster1-submariner-cable-cluster1-172-17-0-5 with IP 172.17.0.5, Host cluster1-worker for this cluster
2020-11-02T14:27:11.9443694Z [cluster1] I1102 14:19:05.712984       1 route.go:612] This route agent is running on a non-gateway/non-active node
2020-11-02T14:27:11.9444775Z [cluster1] I1102 14:19:05.712994       1 route.go:623] Creating the vxlan interface vx-submariner with gateway node IP 172.17.0.5
2020-11-02T14:27:11.9445597Z [cluster1] I1102 14:19:05.713201       1 vxlan.go:61] VxLAN interface already exists with same configuration.
2020-11-02T14:27:11.9446273Z [cluster1] I1102 14:19:05.716715       1 route.go:771] Reconciling routes to gw: 240.17.0.5
2020-11-02T14:27:11.9446906Z [cluster1] I1102 14:19:05.716849       1 route.go:787] Processing route {Ifindex: 16 Dst: 10.2.0.0/16 Src: <nil> Gw: 240.17.0.5 Flags: [] Table: 254}
2020-11-02T14:27:11.9447694Z [cluster1] I1102 14:19:05.716889       1 route.go:793] Found route {Ifindex: 16 Dst: 10.2.0.0/16 Src: <nil> Gw: 240.17.0.5 Flags: [] Table: 254} with gw 240.17.0.5 already installed
2020-11-02T14:27:11.9448478Z [cluster1] I1102 14:19:05.716898       1 route.go:787] Processing route {Ifindex: 16 Dst: 100.2.0.0/16 Src: <nil> Gw: 240.17.0.5 Flags: [] Table: 254}
2020-11-02T14:27:11.9449734Z [cluster1] I1102 14:19:05.716906       1 route.go:793] Found route {Ifindex: 16 Dst: 100.2.0.0/16 Src: <nil> Gw: 240.17.0.5 Flags: [] Table: 254} with gw 240.17.0.5 already installed
2020-11-02T14:27:11.9450445Z [cluster1] I1102 14:19:05.716914       1 route.go:787] Processing route {Ifindex: 16 Dst: 240.0.0.0/8 Src: 240.17.0.6 Gw: <nil> Flags: [] Table: 254}
2020-11-02T14:27:11.9451171Z [cluster1] I1102 14:19:05.716922       1 route.go:790] Found nil gw or dst
2020-11-02T14:27:11.9452053Z [cluster1] I1102 14:19:05.716987       1 route.go:829] Found equivalent route, not adding
2020-11-02T14:27:11.9452655Z [cluster1] I1102 14:19:05.716997       1 route.go:829] Found equivalent route, not adding
2020-11-02T14:27:11.9454081Z [cluster1] I1102 14:19:05.717031       1 route.go:647] Successfully processed local Endpoint "cluster1-submariner-cable-cluster1-172-17-0-5"
2020-11-02T14:27:11.9456157Z [cluster1] I1102 14:19:35.713210       1 route.go:558] Updating iptable rules for remote cluster's Endpoint "cluster2-submariner-cable-cluster2-172-17-0-9" with subnets [100.2.0.0/16 10.2.0.0/16]
2020-11-02T14:27:11.9459413Z [cluster1] I1102 14:19:35.713250       1 route.go:570] Processing local Endpoint cluster1-submariner-cable-cluster1-172-17-0-5 with IP 172.17.0.5, Host cluster1-worker for this cluster
2020-11-02T14:27:11.9461078Z [cluster1] I1102 14:19:35.713259       1 route.go:612] This route agent is running on a non-gateway/non-active node
2020-11-02T14:27:11.9462229Z [cluster1] I1102 14:19:35.713269       1 route.go:623] Creating the vxlan interface vx-submariner with gateway node IP 172.17.0.5
2020-11-02T14:27:11.9462986Z [cluster1] I1102 14:19:35.713443       1 vxlan.go:61] VxLAN interface already exists with same configuration.
2020-11-02T14:27:11.9463611Z [cluster1] I1102 14:19:35.716514       1 route.go:771] Reconciling routes to gw: 240.17.0.5
2020-11-02T14:27:11.9464402Z [cluster1] I1102 14:19:35.716675       1 route.go:787] Processing route {Ifindex: 16 Dst: 10.2.0.0/16 Src: <nil> Gw: 240.17.0.5 Flags: [] Table: 254}
2020-11-02T14:27:11.9465948Z [cluster1] I1102 14:19:35.716707       1 route.go:793] Found route {Ifindex: 16 Dst: 10.2.0.0/16 Src: <nil> Gw: 240.17.0.5 Flags: [] Table: 254} with gw 240.17.0.5 already installed
2020-11-02T14:27:11.9466774Z [cluster1] I1102 14:19:35.716718       1 route.go:787] Processing route {Ifindex: 16 Dst: 100.2.0.0/16 Src: <nil> Gw: 240.17.0.5 Flags: [] Table: 254}
2020-11-02T14:27:11.9467866Z [cluster1] I1102 14:19:35.716727       1 route.go:793] Found route {Ifindex: 16 Dst: 100.2.0.0/16 Src: <nil> Gw: 240.17.0.5 Flags: [] Table: 254} with gw 240.17.0.5 already installed
2020-11-02T14:27:11.9488430Z [cluster1] I1102 14:19:35.716761       1 route.go:787] Processing route {Ifindex: 16 Dst: 240.0.0.0/8 Src: 240.17.0.6 Gw: <nil> Flags: [] Table: 254}
2020-11-02T14:27:11.9489717Z [cluster1] I1102 14:19:35.716769       1 route.go:790] Found nil gw or dst
2020-11-02T14:27:11.9490440Z [cluster1] I1102 14:19:35.716844       1 route.go:829] Found equivalent route, not adding
2020-11-02T14:27:11.9491057Z [cluster1] I1102 14:19:35.716860       1 route.go:829] Found equivalent route, not adding
2020-11-02T14:27:11.9496332Z [cluster1] I1102 14:19:35.716893       1 route.go:647] Successfully processed local Endpoint "cluster1-submariner-cable-cluster1-172-17-0-5"
2020-11-02T14:27:11.9509982Z [cluster1] I1102 14:20:05.713457       1 route.go:558] Updating iptable rules for remote cluster's Endpoint "cluster2-submariner-cable-cluster2-172-17-0-9" with subnets [100.2.0.0/16 10.2.0.0/16]
2020-11-02T14:27:11.9512301Z [cluster1] I1102 14:20:05.713527       1 route.go:570] Processing local Endpoint cluster1-submariner-cable-cluster1-172-17-0-5 with IP 172.17.0.5, Host cluster1-worker for this cluster
2020-11-02T14:27:11.9513750Z [cluster1] I1102 14:20:05.713538       1 route.go:612] This route agent is running on a non-gateway/non-active node
2020-11-02T14:27:11.9514691Z [cluster1] I1102 14:20:05.713562       1 route.go:623] Creating the vxlan interface vx-submariner with gateway node IP 172.17.0.5
2020-11-02T14:27:11.9516151Z [cluster1] I1102 14:20:05.713858       1 vxlan.go:61] VxLAN interface already exists with same configuration.
2020-11-02T14:27:11.9517480Z [cluster1] I1102 14:20:05.717095       1 route.go:771] Reconciling routes to gw: 240.17.0.5
2020-11-02T14:27:11.9548731Z [cluster1] I1102 14:20:05.717245       1 route.go:787] Processing route {Ifindex: 16 Dst: 10.2.0.0/16 Src: <nil> Gw: 240.17.0.5 Flags: [] Table: 254}
2020-11-02T14:27:11.9549459Z [cluster1] I1102 14:20:05.717276       1 route.go:793] Found route {Ifindex: 16 Dst: 10.2.0.0/16 Src: <nil> Gw: 240.17.0.5 Flags: [] Table: 254} with gw 240.17.0.5 already installed
2020-11-02T14:27:11.9550167Z [cluster1] I1102 14:20:05.717286       1 route.go:787] Processing route {Ifindex: 16 Dst: 100.2.0.0/16 Src: <nil> Gw: 240.17.0.5 Flags: [] Table: 254}
2020-11-02T14:27:11.9550875Z [cluster1] I1102 14:20:05.717300       1 route.go:793] Found route {Ifindex: 16 Dst: 100.2.0.0/16 Src: <nil> Gw: 240.17.0.5 Flags: [] Table: 254} with gw 240.17.0.5 already installed
2020-11-02T14:27:11.9551801Z [cluster1] I1102 14:20:05.717308       1 route.go:787] Processing route {Ifindex: 16 Dst: 240.0.0.0/8 Src: 240.17.0.6 Gw: <nil> Flags: [] Table: 254}
2020-11-02T14:27:11.9552519Z [cluster1] I1102 14:20:05.717315       1 route.go:790] Found nil gw or dst
2020-11-02T14:27:11.9553308Z [cluster1] I1102 14:20:05.717424       1 route.go:829] Found equivalent route, not adding
2020-11-02T14:27:11.9553893Z [cluster1] I1102 14:20:05.717434       1 route.go:829] Found equivalent route, not adding
2020-11-02T14:27:11.9554868Z [lighthouse]$ [cluster1] subctl show all
2020-11-02T14:27:11.9556401Z [cluster1] I1102 14:20:05.717465       1 route.go:647] Successfully processed local Endpoint "cluster1-submariner-cable-cluster1-172-17-0-5"
2020-11-02T14:27:11.9558812Z [cluster1] I1102 14:20:35.713483       1 route.go:570] Processing local Endpoint cluster1-submariner-cable-cluster1-172-17-0-5 with IP 172.17.0.5, Host cluster1-worker for this cluster
2020-11-02T14:27:11.9560382Z [cluster1] I1102 14:20:35.713518       1 route.go:612] This route agent is running on a non-gateway/non-active node
2020-11-02T14:27:11.9561621Z [cluster1] I1102 14:20:35.713530       1 route.go:623] Creating the vxlan interface vx-submariner with gateway node IP 172.17.0.5
2020-11-02T14:27:11.9562546Z [cluster1] I1102 14:20:35.713735       1 vxlan.go:61] VxLAN interface already exists with same configuration.
2020-11-02T14:27:11.9563173Z [cluster1] I1102 14:20:35.716771       1 route.go:771] Reconciling routes to gw: 240.17.0.5
2020-11-02T14:27:11.9563776Z [cluster1] I1102 14:20:35.717074       1 route.go:787] Processing route {Ifindex: 16 Dst: 10.2.0.0/16 Src: <nil> Gw: 240.17.0.5 Flags: [] Table: 254}
2020-11-02T14:27:11.9564712Z [cluster1] I1102 14:20:35.717108       1 route.go:793] Found route {Ifindex: 16 Dst: 10.2.0.0/16 Src: <nil> Gw: 240.17.0.5 Flags: [] Table: 254} with gw 240.17.0.5 already installed
2020-11-02T14:27:11.9565465Z [cluster1] I1102 14:20:35.717119       1 route.go:787] Processing route {Ifindex: 16 Dst: 100.2.0.0/16 Src: <nil> Gw: 240.17.0.5 Flags: [] Table: 254}
2020-11-02T14:27:11.9566474Z [cluster1] I1102 14:20:35.717128       1 route.go:793] Found route {Ifindex: 16 Dst: 100.2.0.0/16 Src: <nil> Gw: 240.17.0.5 Flags: [] Table: 254} with gw 240.17.0.5 already installed
2020-11-02T14:27:11.9567253Z [cluster1] I1102 14:20:35.717235       1 route.go:787] Processing route {Ifindex: 16 Dst: 240.0.0.0/8 Src: 240.17.0.6 Gw: <nil> Flags: [] Table: 254}
2020-11-02T14:27:11.9567988Z [cluster1] I1102 14:20:35.717261       1 route.go:790] Found nil gw or dst
2020-11-02T14:27:11.9568499Z [cluster1] I1102 14:20:35.717426       1 route.go:829] Found equivalent route, not adding
2020-11-02T14:27:11.9569230Z [cluster1] I1102 14:20:35.717547       1 route.go:829] Found equivalent route, not adding
2020-11-02T14:27:11.9570744Z [cluster1] I1102 14:20:35.717619       1 route.go:647] Successfully processed local Endpoint "cluster1-submariner-cable-cluster1-172-17-0-5"
2020-11-02T14:27:11.9572981Z [cluster1] I1102 14:20:35.717712       1 route.go:558] Updating iptable rules for remote cluster's Endpoint "cluster2-submariner-cable-cluster2-172-17-0-9" with subnets [100.2.0.0/16 10.2.0.0/16]
2020-11-02T14:27:11.9575006Z [cluster1] I1102 14:21:05.713692       1 route.go:558] Updating iptable rules for remote cluster's Endpoint "cluster2-submariner-cable-cluster2-172-17-0-9" with subnets [100.2.0.0/16 10.2.0.0/16]
2020-11-02T14:27:11.9577145Z [cluster1] I1102 14:21:05.713738       1 route.go:570] Processing local Endpoint cluster1-submariner-cable-cluster1-172-17-0-5 with IP 172.17.0.5, Host cluster1-worker for this cluster
2020-11-02T14:27:11.9578540Z [cluster1] I1102 14:21:05.713748       1 route.go:612] This route agent is running on a non-gateway/non-active node
2020-11-02T14:27:11.9608153Z [cluster1] I1102 14:21:05.713756       1 route.go:623] Creating the vxlan interface vx-submariner with gateway node IP 172.17.0.5
2020-11-02T14:27:11.9609474Z [cluster1] I1102 14:21:05.714061       1 vxlan.go:61] VxLAN interface already exists with same configuration.
2020-11-02T14:27:11.9610145Z [cluster1] I1102 14:21:05.717360       1 route.go:771] Reconciling routes to gw: 240.17.0.5
2020-11-02T14:27:11.9610784Z [cluster1] I1102 14:21:05.717482       1 route.go:787] Processing route {Ifindex: 16 Dst: 10.2.0.0/16 Src: <nil> Gw: 240.17.0.5 Flags: [] Table: 254}
2020-11-02T14:27:11.9611546Z [cluster1] I1102 14:21:05.717504       1 route.go:793] Found route {Ifindex: 16 Dst: 10.2.0.0/16 Src: <nil> Gw: 240.17.0.5 Flags: [] Table: 254} with gw 240.17.0.5 already installed
2020-11-02T14:27:11.9612463Z [cluster1] I1102 14:21:05.717513       1 route.go:787] Processing route {Ifindex: 16 Dst: 100.2.0.0/16 Src: <nil> Gw: 240.17.0.5 Flags: [] Table: 254}
2020-11-02T14:27:11.9613348Z [cluster1] I1102 14:21:05.717522       1 route.go:793] Found route {Ifindex: 16 Dst: 100.2.0.0/16 Src: <nil> Gw: 240.17.0.5 Flags: [] Table: 254} with gw 240.17.0.5 already installed
2020-11-02T14:27:11.9614050Z [cluster1] I1102 14:21:05.717529       1 route.go:787] Processing route {Ifindex: 16 Dst: 240.0.0.0/8 Src: 240.17.0.6 Gw: <nil> Flags: [] Table: 254}
2020-11-02T14:27:11.9614764Z [cluster1] I1102 14:21:05.717537       1 route.go:790] Found nil gw or dst
2020-11-02T14:27:11.9615319Z [cluster1] I1102 14:21:05.717597       1 route.go:829] Found equivalent route, not adding
2020-11-02T14:27:11.9615869Z [cluster1] I1102 14:21:05.717606       1 route.go:829] Found equivalent route, not adding
2020-11-02T14:27:11.9617213Z [cluster1] I1102 14:21:05.717636       1 route.go:647] Successfully processed local Endpoint "cluster1-submariner-cable-cluster1-172-17-0-5"
2020-11-02T14:27:11.9619077Z [cluster1] I1102 14:21:35.713784       1 route.go:558] Updating iptable rules for remote cluster's Endpoint "cluster2-submariner-cable-cluster2-172-17-0-9" with subnets [100.2.0.0/16 10.2.0.0/16]
2020-11-02T14:27:11.9621669Z [cluster1] I1102 14:21:35.713823       1 route.go:570] Processing local Endpoint cluster1-submariner-cable-cluster1-172-17-0-5 with IP 172.17.0.5, Host cluster1-worker for this cluster
2020-11-02T14:27:11.9623239Z [cluster1] I1102 14:21:35.713832       1 route.go:612] This route agent is running on a non-gateway/non-active node
2020-11-02T14:27:11.9624753Z [cluster1] I1102 14:21:35.713841       1 route.go:623] Creating the vxlan interface vx-submariner with gateway node IP 172.17.0.5
2020-11-02T14:27:11.9625455Z [cluster1] I1102 14:21:35.714013       1 vxlan.go:61] VxLAN interface already exists with same configuration.
2020-11-02T14:27:11.9626023Z [cluster1] I1102 14:21:35.717157       1 route.go:771] Reconciling routes to gw: 240.17.0.5
2020-11-02T14:27:11.9626772Z [cluster1] I1102 14:21:35.717387       1 route.go:787] Processing route {Ifindex: 16 Dst: 10.2.0.0/16 Src: <nil> Gw: 240.17.0.5 Flags: [] Table: 254}
2020-11-02T14:27:11.9630955Z [cluster1] I1102 14:21:35.717450       1 route.go:793] Found route {Ifindex: 16 Dst: 10.2.0.0/16 Src: <nil> Gw: 240.17.0.5 Flags: [] Table: 254} with gw 240.17.0.5 already installed
2020-11-02T14:27:11.9631919Z [cluster1] I1102 14:21:35.717469       1 route.go:787] Processing route {Ifindex: 16 Dst: 100.2.0.0/16 Src: <nil> Gw: 240.17.0.5 Flags: [] Table: 254}
2020-11-02T14:27:11.9632991Z [cluster1] I1102 14:21:35.717479       1 route.go:793] Found route {Ifindex: 16 Dst: 100.2.0.0/16 Src: <nil> Gw: 240.17.0.5 Flags: [] Table: 254} with gw 240.17.0.5 already installed
2020-11-02T14:27:11.9633672Z [cluster1] I1102 14:21:35.717497       1 route.go:787] Processing route {Ifindex: 16 Dst: 240.0.0.0/8 Src: 240.17.0.6 Gw: <nil> Flags: [] Table: 254}
2020-11-02T14:27:11.9634193Z [cluster1] I1102 14:21:35.717506       1 route.go:790] Found nil gw or dst
2020-11-02T14:27:11.9634659Z [cluster1] I1102 14:21:35.717585       1 route.go:829] Found equivalent route, not adding
2020-11-02T14:27:11.9635184Z [cluster1] I1102 14:21:35.717603       1 route.go:829] Found equivalent route, not adding
2020-11-02T14:27:11.9636539Z [cluster1] I1102 14:21:35.717652       1 route.go:647] Successfully processed local Endpoint "cluster1-submariner-cable-cluster1-172-17-0-5"
2020-11-02T14:27:11.9638361Z [cluster1] I1102 14:22:05.714181       1 route.go:558] Updating iptable rules for remote cluster's Endpoint "cluster2-submariner-cable-cluster2-172-17-0-9" with subnets [100.2.0.0/16 10.2.0.0/16]
2020-11-02T14:27:11.9640247Z [cluster1] I1102 14:22:05.714224       1 route.go:570] Processing local Endpoint cluster1-submariner-cable-cluster1-172-17-0-5 with IP 172.17.0.5, Host cluster1-worker for this cluster
2020-11-02T14:27:11.9641768Z [cluster1] I1102 14:22:05.714259       1 route.go:612] This route agent is running on a non-gateway/non-active node
2020-11-02T14:27:11.9642681Z [cluster1] I1102 14:22:05.714269       1 route.go:623] Creating the vxlan interface vx-submariner with gateway node IP 172.17.0.5
2020-11-02T14:27:11.9644034Z [cluster1] I1102 14:22:05.714468       1 vxlan.go:61] VxLAN interface already exists with same configuration.
2020-11-02T14:27:11.9644684Z [cluster1] I1102 14:22:05.717819       1 route.go:771] Reconciling routes to gw: 240.17.0.5
2020-11-02T14:27:11.9645724Z [cluster1] I1102 14:22:05.717953       1 route.go:787] Processing route {Ifindex: 16 Dst: 10.2.0.0/16 Src: <nil> Gw: 240.17.0.5 Flags: [] Table: 254}
2020-11-02T14:27:11.9646564Z [cluster1] I1102 14:22:05.717985       1 route.go:793] Found route {Ifindex: 16 Dst: 10.2.0.0/16 Src: <nil> Gw: 240.17.0.5 Flags: [] Table: 254} with gw 240.17.0.5 already installed
2020-11-02T14:27:11.9647356Z [cluster1] I1102 14:22:05.717996       1 route.go:787] Processing route {Ifindex: 16 Dst: 100.2.0.0/16 Src: <nil> Gw: 240.17.0.5 Flags: [] Table: 254}
2020-11-02T14:27:11.9648141Z [cluster1] I1102 14:22:05.718005       1 route.go:793] Found route {Ifindex: 16 Dst: 100.2.0.0/16 Src: <nil> Gw: 240.17.0.5 Flags: [] Table: 254} with gw 240.17.0.5 already installed
2020-11-02T14:27:11.9648922Z [cluster1] I1102 14:22:05.718019       1 route.go:787] Processing route {Ifindex: 16 Dst: 240.0.0.0/8 Src: 240.17.0.6 Gw: <nil> Flags: [] Table: 254}
2020-11-02T14:27:11.9649518Z [cluster1] I1102 14:22:05.718027       1 route.go:790] Found nil gw or dst
2020-11-02T14:27:11.9650055Z [cluster1] I1102 14:22:05.718084       1 route.go:829] Found equivalent route, not adding
2020-11-02T14:27:11.9650659Z [cluster1] I1102 14:22:05.718093       1 route.go:829] Found equivalent route, not adding
2020-11-02T14:27:11.9652274Z [cluster1] I1102 14:22:05.718129       1 route.go:647] Successfully processed local Endpoint "cluster1-submariner-cable-cluster1-172-17-0-5"
2020-11-02T14:27:11.9654777Z [cluster1] I1102 14:22:35.714189       1 route.go:558] Updating iptable rules for remote cluster's Endpoint "cluster2-submariner-cable-cluster2-172-17-0-9" with subnets [100.2.0.0/16 10.2.0.0/16]
2020-11-02T14:27:11.9657438Z [cluster1] I1102 14:22:35.714247       1 route.go:570] Processing local Endpoint cluster1-submariner-cable-cluster1-172-17-0-5 with IP 172.17.0.5, Host cluster1-worker for this cluster
2020-11-02T14:27:11.9659064Z [cluster1] I1102 14:22:35.714256       1 route.go:612] This route agent is running on a non-gateway/non-active node
2020-11-02T14:27:11.9660041Z [cluster1] I1102 14:22:35.714271       1 route.go:623] Creating the vxlan interface vx-submariner with gateway node IP 172.17.0.5
2020-11-02T14:27:11.9660793Z [cluster1] I1102 14:22:35.714633       1 vxlan.go:61] VxLAN interface already exists with same configuration.
2020-11-02T14:27:11.9661789Z [cluster1] I1102 14:22:35.718720       1 route.go:771] Reconciling routes to gw: 240.17.0.5
2020-11-02T14:27:11.9662441Z [cluster1] I1102 14:22:35.718936       1 route.go:787] Processing route {Ifindex: 16 Dst: 10.2.0.0/16 Src: <nil> Gw: 240.17.0.5 Flags: [] Table: 254}
2020-11-02T14:27:11.9663224Z [cluster1] I1102 14:22:35.718976       1 route.go:793] Found route {Ifindex: 16 Dst: 10.2.0.0/16 Src: <nil> Gw: 240.17.0.5 Flags: [] Table: 254} with gw 240.17.0.5 already installed
2020-11-02T14:27:11.9664007Z [cluster1] I1102 14:22:35.719009       1 route.go:787] Processing route {Ifindex: 16 Dst: 100.2.0.0/16 Src: <nil> Gw: 240.17.0.5 Flags: [] Table: 254}
2020-11-02T14:27:11.9664936Z [cluster1] I1102 14:22:35.719019       1 route.go:793] Found route {Ifindex: 16 Dst: 100.2.0.0/16 Src: <nil> Gw: 240.17.0.5 Flags: [] Table: 254} with gw 240.17.0.5 already installed
2020-11-02T14:27:11.9665985Z [cluster1] I1102 14:22:35.719027       1 route.go:787] Processing route {Ifindex: 16 Dst: 240.0.0.0/8 Src: 240.17.0.6 Gw: <nil> Flags: [] Table: 254}
2020-11-02T14:27:11.9666520Z [cluster1] I1102 14:22:35.719036       1 route.go:790] Found nil gw or dst
2020-11-02T14:27:11.9666997Z [cluster1] I1102 14:22:35.719174       1 route.go:829] Found equivalent route, not adding
2020-11-02T14:27:11.9667620Z [cluster1] I1102 14:22:35.719188       1 route.go:829] Found equivalent route, not adding
2020-11-02T14:27:11.9669307Z [cluster1] I1102 14:22:35.719223       1 route.go:647] Successfully processed local Endpoint "cluster1-submariner-cable-cluster1-172-17-0-5"
2020-11-02T14:27:11.9683240Z [cluster1] I1102 14:23:05.714272       1 route.go:558] Updating iptable rules for remote cluster's Endpoint "cluster2-submariner-cable-cluster2-172-17-0-9" with subnets [100.2.0.0/16 10.2.0.0/16]
2020-11-02T14:27:11.9685782Z [cluster1] I1102 14:23:05.714321       1 route.go:570] Processing local Endpoint cluster1-submariner-cable-cluster1-172-17-0-5 with IP 172.17.0.5, Host cluster1-worker for this cluster
2020-11-02T14:27:11.9687393Z [cluster1] I1102 14:23:05.714335       1 route.go:612] This route agent is running on a non-gateway/non-active node
2020-11-02T14:27:11.9688441Z [cluster1] I1102 14:23:05.714346       1 route.go:623] Creating the vxlan interface vx-submariner with gateway node IP 172.17.0.5
2020-11-02T14:27:11.9689255Z [cluster1] I1102 14:23:05.714543       1 vxlan.go:61] VxLAN interface already exists with same configuration.
2020-11-02T14:27:11.9689931Z [cluster1] I1102 14:23:05.717789       1 route.go:771] Reconciling routes to gw: 240.17.0.5
2020-11-02T14:27:11.9690579Z [cluster1] I1102 14:23:05.717940       1 route.go:787] Processing route {Ifindex: 16 Dst: 10.2.0.0/16 Src: <nil> Gw: 240.17.0.5 Flags: [] Table: 254}
2020-11-02T14:27:11.9691423Z [cluster1] I1102 14:23:05.717974       1 route.go:793] Found route {Ifindex: 16 Dst: 10.2.0.0/16 Src: <nil> Gw: 240.17.0.5 Flags: [] Table: 254} with gw 240.17.0.5 already installed
2020-11-02T14:27:11.9692214Z [cluster1] I1102 14:23:05.717985       1 route.go:787] Processing route {Ifindex: 16 Dst: 100.2.0.0/16 Src: <nil> Gw: 240.17.0.5 Flags: [] Table: 254}
2020-11-02T14:27:11.9693335Z [cluster1] I1102 14:23:05.718002       1 route.go:793] Found route {Ifindex: 16 Dst: 100.2.0.0/16 Src: <nil> Gw: 240.17.0.5 Flags: [] Table: 254} with gw 240.17.0.5 already installed
2020-11-02T14:27:11.9694063Z [cluster1] I1102 14:23:05.718011       1 route.go:787] Processing route {Ifindex: 16 Dst: 240.0.0.0/8 Src: 240.17.0.6 Gw: <nil> Flags: [] Table: 254}
2020-11-02T14:27:11.9694600Z [cluster1] I1102 14:23:05.718025       1 route.go:790] Found nil gw or dst
2020-11-02T14:27:11.9695096Z [cluster1] I1102 14:23:05.718090       1 route.go:829] Found equivalent route, not adding
2020-11-02T14:27:11.9695657Z [cluster1] I1102 14:23:05.718107       1 route.go:829] Found equivalent route, not adding
2020-11-02T14:27:11.9697004Z [cluster1] I1102 14:23:05.718160       1 route.go:647] Successfully processed local Endpoint "cluster1-submariner-cable-cluster1-172-17-0-5"
2020-11-02T14:27:11.9699776Z [cluster1] I1102 14:23:35.714441       1 route.go:558] Updating iptable rules for remote cluster's Endpoint "cluster2-submariner-cable-cluster2-172-17-0-9" with subnets [100.2.0.0/16 10.2.0.0/16]
2020-11-02T14:27:11.9701944Z [cluster1] I1102 14:23:35.714478       1 route.go:570] Processing local Endpoint cluster1-submariner-cable-cluster1-172-17-0-5 with IP 172.17.0.5, Host cluster1-worker for this cluster
2020-11-02T14:27:11.9703503Z [cluster1] I1102 14:23:35.714487       1 route.go:612] This route agent is running on a non-gateway/non-active node
2020-11-02T14:27:11.9704546Z [cluster1] I1102 14:23:35.714497       1 route.go:623] Creating the vxlan interface vx-submariner with gateway node IP 172.17.0.5
2020-11-02T14:27:11.9705360Z [cluster1] I1102 14:23:35.714709       1 vxlan.go:61] VxLAN interface already exists with same configuration.
2020-11-02T14:27:11.9706031Z [cluster1] I1102 14:23:35.717695       1 route.go:771] Reconciling routes to gw: 240.17.0.5
2020-11-02T14:27:11.9706674Z [cluster1] I1102 14:23:35.717828       1 route.go:787] Processing route {Ifindex: 16 Dst: 10.2.0.0/16 Src: <nil> Gw: 240.17.0.5 Flags: [] Table: 254}
2020-11-02T14:27:11.9707458Z [cluster1] I1102 14:23:35.717880       1 route.go:793] Found route {Ifindex: 16 Dst: 10.2.0.0/16 Src: <nil> Gw: 240.17.0.5 Flags: [] Table: 254} with gw 240.17.0.5 already installed
2020-11-02T14:27:11.9708239Z [cluster1] I1102 14:23:35.717889       1 route.go:787] Processing route {Ifindex: 16 Dst: 100.2.0.0/16 Src: <nil> Gw: 240.17.0.5 Flags: [] Table: 254}
2020-11-02T14:27:11.9709021Z [cluster1] I1102 14:23:35.717900       1 route.go:793] Found route {Ifindex: 16 Dst: 100.2.0.0/16 Src: <nil> Gw: 240.17.0.5 Flags: [] Table: 254} with gw 240.17.0.5 already installed
2020-11-02T14:27:11.9709907Z [cluster1] I1102 14:23:35.717907       1 route.go:787] Processing route {Ifindex: 16 Dst: 240.0.0.0/8 Src: 240.17.0.6 Gw: <nil> Flags: [] Table: 254}
2020-11-02T14:27:11.9710558Z [cluster1] I1102 14:23:35.717915       1 route.go:790] Found nil gw or dst
2020-11-02T14:27:11.9711078Z [cluster1] I1102 14:23:35.717970       1 route.go:829] Found equivalent route, not adding
2020-11-02T14:27:11.9712233Z [cluster1] I1102 14:23:35.717979       1 route.go:829] Found equivalent route, not adding
2020-11-02T14:27:11.9713713Z [cluster1] I1102 14:23:35.718022       1 route.go:647] Successfully processed local Endpoint "cluster1-submariner-cable-cluster1-172-17-0-5"
2020-11-02T14:27:11.9716058Z [cluster1] I1102 14:24:05.714629       1 route.go:558] Updating iptable rules for remote cluster's Endpoint "cluster2-submariner-cable-cluster2-172-17-0-9" with subnets [100.2.0.0/16 10.2.0.0/16]
2020-11-02T14:27:11.9718073Z [cluster1] I1102 14:24:05.714671       1 route.go:570] Processing local Endpoint cluster1-submariner-cable-cluster1-172-17-0-5 with IP 172.17.0.5, Host cluster1-worker for this cluster
2020-11-02T14:27:11.9724027Z [cluster1] I1102 14:24:05.714681       1 route.go:612] This route agent is running on a non-gateway/non-active node
2020-11-02T14:27:11.9725028Z [cluster1] I1102 14:24:05.714690       1 route.go:623] Creating the vxlan interface vx-submariner with gateway node IP 172.17.0.5
2020-11-02T14:27:11.9725788Z [cluster1] I1102 14:24:05.714922       1 vxlan.go:61] VxLAN interface already exists with same configuration.
2020-11-02T14:27:11.9726417Z [cluster1] I1102 14:24:05.718225       1 route.go:771] Reconciling routes to gw: 240.17.0.5
2020-11-02T14:27:11.9727023Z [cluster1] I1102 14:24:05.718367       1 route.go:787] Processing route {Ifindex: 16 Dst: 10.2.0.0/16 Src: <nil> Gw: 240.17.0.5 Flags: [] Table: 254}
2020-11-02T14:27:11.9727756Z [cluster1] I1102 14:24:05.718407       1 route.go:793] Found route {Ifindex: 16 Dst: 10.2.0.0/16 Src: <nil> Gw: 240.17.0.5 Flags: [] Table: 254} with gw 240.17.0.5 already installed
2020-11-02T14:27:11.9728496Z [cluster1] I1102 14:24:05.718417       1 route.go:787] Processing route {Ifindex: 16 Dst: 100.2.0.0/16 Src: <nil> Gw: 240.17.0.5 Flags: [] Table: 254}
2020-11-02T14:27:11.9729229Z [cluster1] I1102 14:24:05.718426       1 route.go:793] Found route {Ifindex: 16 Dst: 100.2.0.0/16 Src: <nil> Gw: 240.17.0.5 Flags: [] Table: 254} with gw 240.17.0.5 already installed
2020-11-02T14:27:11.9729956Z [cluster1] I1102 14:24:05.718440       1 route.go:787] Processing route {Ifindex: 16 Dst: 240.0.0.0/8 Src: 240.17.0.6 Gw: <nil> Flags: [] Table: 254}
2020-11-02T14:27:11.9730511Z [cluster1] I1102 14:24:05.718448       1 route.go:790] Found nil gw or dst
2020-11-02T14:27:11.9731007Z [cluster1] I1102 14:24:05.718505       1 route.go:829] Found equivalent route, not adding
2020-11-02T14:27:11.9731925Z [cluster1] I1102 14:24:05.718514       1 route.go:829] Found equivalent route, not adding
2020-11-02T14:27:11.9733367Z [cluster1] I1102 14:24:05.718566       1 route.go:647] Successfully processed local Endpoint "cluster1-submariner-cable-cluster1-172-17-0-5"
2020-11-02T14:27:11.9735477Z [cluster1] I1102 14:24:35.714812       1 route.go:570] Processing local Endpoint cluster1-submariner-cable-cluster1-172-17-0-5 with IP 172.17.0.5, Host cluster1-worker for this cluster
2020-11-02T14:27:11.9737042Z [cluster1] I1102 14:24:35.714850       1 route.go:612] This route agent is running on a non-gateway/non-active node
2020-11-02T14:27:11.9738416Z [cluster1] I1102 14:24:35.714863       1 route.go:623] Creating the vxlan interface vx-submariner with gateway node IP 172.17.0.5
2020-11-02T14:27:11.9739230Z [cluster1] I1102 14:24:35.715105       1 vxlan.go:61] VxLAN interface already exists with same configuration.
2020-11-02T14:27:11.9739901Z [cluster1] I1102 14:24:35.718644       1 route.go:771] Reconciling routes to gw: 240.17.0.5
2020-11-02T14:27:11.9740552Z [cluster1] I1102 14:24:35.718885       1 route.go:787] Processing route {Ifindex: 16 Dst: 10.2.0.0/16 Src: <nil> Gw: 240.17.0.5 Flags: [] Table: 254}
2020-11-02T14:27:11.9741755Z [cluster1] I1102 14:24:35.718929       1 route.go:793] Found route {Ifindex: 16 Dst: 10.2.0.0/16 Src: <nil> Gw: 240.17.0.5 Flags: [] Table: 254} with gw 240.17.0.5 already installed
2020-11-02T14:27:11.9742719Z [cluster1] I1102 14:24:35.718940       1 route.go:787] Processing route {Ifindex: 16 Dst: 100.2.0.0/16 Src: <nil> Gw: 240.17.0.5 Flags: [] Table: 254}
2020-11-02T14:27:11.9743506Z [cluster1] I1102 14:24:35.718949       1 route.go:793] Found route {Ifindex: 16 Dst: 100.2.0.0/16 Src: <nil> Gw: 240.17.0.5 Flags: [] Table: 254} with gw 240.17.0.5 already installed
2020-11-02T14:27:11.9744291Z [cluster1] I1102 14:24:35.718957       1 route.go:787] Processing route {Ifindex: 16 Dst: 240.0.0.0/8 Src: 240.17.0.6 Gw: <nil> Flags: [] Table: 254}
2020-11-02T14:27:11.9745305Z [cluster1] I1102 14:24:35.718965       1 route.go:790] Found nil gw or dst
2020-11-02T14:27:11.9745852Z [cluster1] I1102 14:24:35.719075       1 route.go:829] Found equivalent route, not adding
2020-11-02T14:27:11.9746467Z [cluster1] I1102 14:24:35.719092       1 route.go:829] Found equivalent route, not adding
2020-11-02T14:27:11.9748142Z [cluster1] I1102 14:24:35.719197       1 route.go:647] Successfully processed local Endpoint "cluster1-submariner-cable-cluster1-172-17-0-5"
2020-11-02T14:27:11.9750074Z [cluster1] I1102 14:24:35.719221       1 route.go:558] Updating iptable rules for remote cluster's Endpoint "cluster2-submariner-cable-cluster2-172-17-0-9" with subnets [100.2.0.0/16 10.2.0.0/16]
2020-11-02T14:27:11.9752443Z [cluster1] I1102 14:25:05.714923       1 route.go:558] Updating iptable rules for remote cluster's Endpoint "cluster2-submariner-cable-cluster2-172-17-0-9" with subnets [100.2.0.0/16 10.2.0.0/16]
2020-11-02T14:27:11.9754558Z [cluster1] I1102 14:25:05.714963       1 route.go:570] Processing local Endpoint cluster1-submariner-cable-cluster1-172-17-0-5 with IP 172.17.0.5, Host cluster1-worker for this cluster
2020-11-02T14:27:11.9757097Z [cluster1] I1102 14:25:05.714972       1 route.go:612] This route agent is running on a non-gateway/non-active node
2020-11-02T14:27:11.9758314Z [cluster1] I1102 14:25:05.714985       1 route.go:623] Creating the vxlan interface vx-submariner with gateway node IP 172.17.0.5
2020-11-02T14:27:11.9759094Z [cluster1] I1102 14:25:05.715375       1 vxlan.go:61] VxLAN interface already exists with same configuration.
2020-11-02T14:27:11.9759933Z [cluster1] I1102 14:25:05.718452       1 route.go:771] Reconciling routes to gw: 240.17.0.5
2020-11-02T14:27:11.9761626Z [cluster1] I1102 14:25:05.718592       1 route.go:787] Processing route {Ifindex: 16 Dst: 10.2.0.0/16 Src: <nil> Gw: 240.17.0.5 Flags: [] Table: 254}
2020-11-02T14:27:11.9762441Z [cluster1] I1102 14:25:05.718626       1 route.go:793] Found route {Ifindex: 16 Dst: 10.2.0.0/16 Src: <nil> Gw: 240.17.0.5 Flags: [] Table: 254} with gw 240.17.0.5 already installed
2020-11-02T14:27:11.9763227Z [cluster1] I1102 14:25:05.718637       1 route.go:787] Processing route {Ifindex: 16 Dst: 100.2.0.0/16 Src: <nil> Gw: 240.17.0.5 Flags: [] Table: 254}
2020-11-02T14:27:11.9764337Z [cluster1] I1102 14:25:05.718664       1 route.go:793] Found route {Ifindex: 16 Dst: 100.2.0.0/16 Src: <nil> Gw: 240.17.0.5 Flags: [] Table: 254} with gw 240.17.0.5 already installed
2020-11-02T14:27:11.9765041Z [cluster1] I1102 14:25:05.718683       1 route.go:787] Processing route {Ifindex: 16 Dst: 240.0.0.0/8 Src: 240.17.0.6 Gw: <nil> Flags: [] Table: 254}
2020-11-02T14:27:11.9765770Z [cluster1] I1102 14:25:05.718695       1 route.go:790] Found nil gw or dst
2020-11-02T14:27:11.9766266Z [cluster1] I1102 14:25:05.718771       1 route.go:829] Found equivalent route, not adding
2020-11-02T14:27:11.9766825Z [cluster1] I1102 14:25:05.718789       1 route.go:829] Found equivalent route, not adding
2020-11-02T14:27:11.9768374Z [cluster1] I1102 14:25:05.718821       1 route.go:647] Successfully processed local Endpoint "cluster1-submariner-cable-cluster1-172-17-0-5"
2020-11-02T14:27:11.9770309Z [cluster1] I1102 14:25:35.715060       1 route.go:558] Updating iptable rules for remote cluster's Endpoint "cluster2-submariner-cable-cluster2-172-17-0-9" with subnets [100.2.0.0/16 10.2.0.0/16]
2020-11-02T14:27:11.9772868Z [cluster1] I1102 14:25:35.715101       1 route.go:570] Processing local Endpoint cluster1-submariner-cable-cluster1-172-17-0-5 with IP 172.17.0.5, Host cluster1-worker for this cluster
2020-11-02T14:27:11.9774396Z [cluster1] I1102 14:25:35.715110       1 route.go:612] This route agent is running on a non-gateway/non-active node
2020-11-02T14:27:11.9775404Z [cluster1] I1102 14:25:35.715119       1 route.go:623] Creating the vxlan interface vx-submariner with gateway node IP 172.17.0.5
2020-11-02T14:27:11.9776341Z [cluster1] I1102 14:25:35.715380       1 vxlan.go:61] VxLAN interface already exists with same configuration.
2020-11-02T14:27:11.9776965Z [cluster1] I1102 14:25:35.718507       1 route.go:771] Reconciling routes to gw: 240.17.0.5
2020-11-02T14:27:11.9777564Z [cluster1] I1102 14:25:35.718633       1 route.go:787] Processing route {Ifindex: 16 Dst: 10.2.0.0/16 Src: <nil> Gw: 240.17.0.5 Flags: [] Table: 254}
2020-11-02T14:27:11.9778302Z [cluster1] I1102 14:25:35.718664       1 route.go:793] Found route {Ifindex: 16 Dst: 10.2.0.0/16 Src: <nil> Gw: 240.17.0.5 Flags: [] Table: 254} with gw 240.17.0.5 already installed
2020-11-02T14:27:11.9779030Z [cluster1] I1102 14:25:35.718674       1 route.go:787] Processing route {Ifindex: 16 Dst: 100.2.0.0/16 Src: <nil> Gw: 240.17.0.5 Flags: [] Table: 254}
2020-11-02T14:27:11.9779760Z [cluster1] I1102 14:25:35.718689       1 route.go:793] Found route {Ifindex: 16 Dst: 100.2.0.0/16 Src: <nil> Gw: 240.17.0.5 Flags: [] Table: 254} with gw 240.17.0.5 already installed
2020-11-02T14:27:11.9780484Z [cluster1] I1102 14:25:35.718697       1 route.go:787] Processing route {Ifindex: 16 Dst: 240.0.0.0/8 Src: 240.17.0.6 Gw: <nil> Flags: [] Table: 254}
2020-11-02T14:27:11.9781037Z [cluster1] I1102 14:25:35.718705       1 route.go:790] Found nil gw or dst
2020-11-02T14:27:11.9781727Z [cluster1] I1102 14:25:35.718764       1 route.go:829] Found equivalent route, not adding
2020-11-02T14:27:11.9782311Z [cluster1] I1102 14:25:35.718782       1 route.go:829] Found equivalent route, not adding
2020-11-02T14:27:11.9783911Z [cluster1] I1102 14:25:35.718818       1 route.go:647] Successfully processed local Endpoint "cluster1-submariner-cable-cluster1-172-17-0-5"
2020-11-02T14:27:11.9785980Z [cluster1] I1102 14:26:05.715258       1 route.go:558] Updating iptable rules for remote cluster's Endpoint "cluster2-submariner-cable-cluster2-172-17-0-9" with subnets [100.2.0.0/16 10.2.0.0/16]
2020-11-02T14:27:11.9788285Z [cluster1] I1102 14:26:05.715305       1 route.go:570] Processing local Endpoint cluster1-submariner-cable-cluster1-172-17-0-5 with IP 172.17.0.5, Host cluster1-worker for this cluster
2020-11-02T14:27:11.9789785Z [cluster1] I1102 14:26:05.715314       1 route.go:612] This route agent is running on a non-gateway/non-active node
2020-11-02T14:27:11.9790825Z [cluster1] I1102 14:26:05.715338       1 route.go:623] Creating the vxlan interface vx-submariner with gateway node IP 172.17.0.5
2020-11-02T14:27:11.9791744Z [cluster1] I1102 14:26:05.715580       1 vxlan.go:61] VxLAN interface already exists with same configuration.
2020-11-02T14:27:11.9792395Z [cluster1] I1102 14:26:05.719369       1 route.go:771] Reconciling routes to gw: 240.17.0.5
2020-11-02T14:27:11.9793023Z [cluster1] I1102 14:26:05.719567       1 route.go:787] Processing route {Ifindex: 16 Dst: 10.2.0.0/16 Src: <nil> Gw: 240.17.0.5 Flags: [] Table: 254}
2020-11-02T14:27:11.9793776Z [cluster1] I1102 14:26:05.719611       1 route.go:793] Found route {Ifindex: 16 Dst: 10.2.0.0/16 Src: <nil> Gw: 240.17.0.5 Flags: [] Table: 254} with gw 240.17.0.5 already installed
2020-11-02T14:27:11.9794528Z [cluster1] I1102 14:26:05.719622       1 route.go:787] Processing route {Ifindex: 16 Dst: 100.2.0.0/16 Src: <nil> Gw: 240.17.0.5 Flags: [] Table: 254}
2020-11-02T14:27:11.9795280Z [cluster1] I1102 14:26:05.719631       1 route.go:793] Found route {Ifindex: 16 Dst: 100.2.0.0/16 Src: <nil> Gw: 240.17.0.5 Flags: [] Table: 254} with gw 240.17.0.5 already installed
2020-11-02T14:27:11.9796236Z [cluster1] I1102 14:26:05.719639       1 route.go:787] Processing route {Ifindex: 16 Dst: 240.0.0.0/8 Src: 240.17.0.6 Gw: <nil> Flags: [] Table: 254}
2020-11-02T14:27:11.9796814Z [cluster1] I1102 14:26:05.719647       1 route.go:790] Found nil gw or dst
2020-11-02T14:27:11.9797327Z [cluster1] I1102 14:26:05.719710       1 route.go:829] Found equivalent route, not adding
2020-11-02T14:27:11.9797907Z [cluster1] I1102 14:26:05.719719       1 route.go:829] Found equivalent route, not adding
2020-11-02T14:27:11.9799456Z [cluster1] I1102 14:26:05.719750       1 route.go:647] Successfully processed local Endpoint "cluster1-submariner-cable-cluster1-172-17-0-5"
2020-11-02T14:27:11.9801472Z [cluster1] I1102 14:26:35.715326       1 route.go:570] Processing local Endpoint cluster1-submariner-cable-cluster1-172-17-0-5 with IP 172.17.0.5, Host cluster1-worker for this cluster
2020-11-02T14:27:11.9803073Z [cluster1] I1102 14:26:35.715358       1 route.go:612] This route agent is running on a non-gateway/non-active node
2020-11-02T14:27:11.9804046Z [cluster1] I1102 14:26:35.715369       1 route.go:623] Creating the vxlan interface vx-submariner with gateway node IP 172.17.0.5
2020-11-02T14:27:11.9804798Z [cluster1] I1102 14:26:35.715591       1 vxlan.go:61] VxLAN interface already exists with same configuration.
2020-11-02T14:27:11.9805593Z [cluster1] I1102 14:26:35.718830       1 route.go:771] Reconciling routes to gw: 240.17.0.5
2020-11-02T14:27:11.9806240Z [cluster1] I1102 14:26:35.718976       1 route.go:787] Processing route {Ifindex: 16 Dst: 10.2.0.0/16 Src: <nil> Gw: 240.17.0.5 Flags: [] Table: 254}
2020-11-02T14:27:11.9807023Z [cluster1] I1102 14:26:35.719011       1 route.go:793] Found route {Ifindex: 16 Dst: 10.2.0.0/16 Src: <nil> Gw: 240.17.0.5 Flags: [] Table: 254} with gw 240.17.0.5 already installed
2020-11-02T14:27:11.9807803Z [cluster1] I1102 14:26:35.719034       1 route.go:787] Processing route {Ifindex: 16 Dst: 100.2.0.0/16 Src: <nil> Gw: 240.17.0.5 Flags: [] Table: 254}
2020-11-02T14:27:11.9808592Z [cluster1] I1102 14:26:35.719054       1 route.go:793] Found route {Ifindex: 16 Dst: 100.2.0.0/16 Src: <nil> Gw: 240.17.0.5 Flags: [] Table: 254} with gw 240.17.0.5 already installed
2020-11-02T14:27:11.9809524Z [cluster1] I1102 14:26:35.719062       1 route.go:787] Processing route {Ifindex: 16 Dst: 240.0.0.0/8 Src: 240.17.0.6 Gw: <nil> Flags: [] Table: 254}
2020-11-02T14:27:11.9810097Z [cluster1] I1102 14:26:35.719070       1 route.go:790] Found nil gw or dst
2020-11-02T14:27:11.9810611Z [cluster1] I1102 14:26:35.719194       1 route.go:829] Found equivalent route, not adding
2020-11-02T14:27:11.9811337Z [cluster1] I1102 14:26:35.719221       1 route.go:829] Found equivalent route, not adding
2020-11-02T14:27:11.9815169Z [cluster1] I1102 14:26:35.719255       1 route.go:647] Successfully processed local Endpoint "cluster1-submariner-cable-cluster1-172-17-0-5"
2020-11-02T14:27:11.9817311Z [cluster1] I1102 14:26:35.719272       1 route.go:558] Updating iptable rules for remote cluster's Endpoint "cluster2-submariner-cable-cluster2-172-17-0-9" with subnets [100.2.0.0/16 10.2.0.0/16]
2020-11-02T14:27:11.9819471Z [cluster1] I1102 14:27:05.715379       1 route.go:558] Updating iptable rules for remote cluster's Endpoint "cluster2-submariner-cable-cluster2-172-17-0-9" with subnets [100.2.0.0/16 10.2.0.0/16]
2020-11-02T14:27:11.9821593Z [cluster1] I1102 14:27:05.715420       1 route.go:570] Processing local Endpoint cluster1-submariner-cable-cluster1-172-17-0-5 with IP 172.17.0.5, Host cluster1-worker for this cluster
2020-11-02T14:27:11.9823262Z [cluster1] I1102 14:27:05.715430       1 route.go:612] This route agent is running on a non-gateway/non-active node
2020-11-02T14:27:11.9824265Z [cluster1] I1102 14:27:05.715440       1 route.go:623] Creating the vxlan interface vx-submariner with gateway node IP 172.17.0.5
2020-11-02T14:27:11.9825397Z [cluster1] I1102 14:27:05.715652       1 vxlan.go:61] VxLAN interface already exists with same configuration.
2020-11-02T14:27:11.9826198Z [cluster1] I1102 14:27:05.718829       1 route.go:771] Reconciling routes to gw: 240.17.0.5
2020-11-02T14:27:11.9826922Z [cluster1] I1102 14:27:05.718952       1 route.go:787] Processing route {Ifindex: 16 Dst: 10.2.0.0/16 Src: <nil> Gw: 240.17.0.5 Flags: [] Table: 254}
2020-11-02T14:27:11.9827690Z [cluster1] I1102 14:27:05.718974       1 route.go:793] Found route {Ifindex: 16 Dst: 10.2.0.0/16 Src: <nil> Gw: 240.17.0.5 Flags: [] Table: 254} with gw 240.17.0.5 already installed
2020-11-02T14:27:11.9828445Z [cluster1] I1102 14:27:05.718988       1 route.go:787] Processing route {Ifindex: 16 Dst: 100.2.0.0/16 Src: <nil> Gw: 240.17.0.5 Flags: [] Table: 254}
2020-11-02T14:27:11.9829199Z [cluster1] I1102 14:27:05.718998       1 route.go:793] Found route {Ifindex: 16 Dst: 100.2.0.0/16 Src: <nil> Gw: 240.17.0.5 Flags: [] Table: 254} with gw 240.17.0.5 already installed
2020-11-02T14:27:11.9830287Z [cluster1] I1102 14:27:05.719005       1 route.go:787] Processing route {Ifindex: 16 Dst: 240.0.0.0/8 Src: 240.17.0.6 Gw: <nil> Flags: [] Table: 254}
2020-11-02T14:27:11.9831023Z [cluster1] I1102 14:27:05.719013       1 route.go:790] Found nil gw or dst
2020-11-02T14:27:11.9831658Z [cluster1] I1102 14:27:05.719074       1 route.go:829] Found equivalent route, not adding
2020-11-02T14:27:11.9832217Z [cluster1] I1102 14:27:05.719083       1 route.go:829] Found equivalent route, not adding
2020-11-02T14:27:11.9833589Z [cluster1] I1102 14:27:05.719112       1 route.go:647] Successfully processed local Endpoint "cluster1-submariner-cable-cluster1-172-17-0-5"
2020-11-02T14:27:12.2565397Z Error obtaining the Submariner resource: the server could not find the requested resource (get submariners.submariner.io submariner)
2020-11-02T14:27:12.2566261Z
2020-11-02T14:27:12.2567468Z subctl version: v0.7.0-8-g282818e
2020-11-02T14:27:12.2567720Z
2020-11-02T14:27:12.2588914Z [cluster1] ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
2020-11-02T14:27:12.2589267Z [cluster1]
2020-11-02T14:27:12.2589700Z [cluster1] Showing information for cluster "cluster2":
2020-11-02T14:27:12.2590219Z [cluster1]     Discovered network details:
2020-11-02T14:27:12.2591176Z [cluster1]         Network plugin:  weave-net
2020-11-02T14:27:12.2592047Z [cluster1]         Service CIDRs:   [100.2.0.0/16]
2020-11-02T14:27:12.2592421Z [cluster1]         Cluster CIDRs:   [10.2.0.0/16]
2020-11-02T14:27:12.2592739Z [cluster1]
2020-11-02T14:27:12.2593096Z [cluster1] ===================== END Post mortem cluster1 =====================
2020-11-02T14:27:12.2610035Z [lighthouse]$ [cluster2] post_analyze
2020-11-02T14:27:12.2626769Z [lighthouse]$ [cluster2] sed s/^/[cluster2] /
2020-11-02T14:27:12.2636041Z [lighthouse]$ [cluster2] post_analyze
2020-11-02T14:27:12.2650792Z [lighthouse]$ [cluster2] kubectl get all --all-namespaces
2020-11-02T14:27:12.2664979Z [lighthouse]$ [cluster2] kubectl get all --all-namespaces
2020-11-02T14:27:12.2684668Z [lighthouse]$ [cluster2] command kubectl --context=cluster2 get all --all-namespaces
2020-11-02T14:27:12.2700397Z [lighthouse]$ [cluster2] kubectl --context=cluster2 get all --all-namespaces
2020-11-02T14:27:12.7333076Z [cluster2] ======================= Post mortem cluster2 =======================
2020-11-02T14:27:12.7333710Z [cluster2] NAMESPACE             NAME                                                 READY   STATUS    RESTARTS   AGE
2020-11-02T14:27:12.7335052Z [cluster2] kube-system           pod/coredns-6955765f44-9d7gr                         1/1     Running   0          12m
2020-11-02T14:27:12.7336002Z [cluster2] kube-system           pod/coredns-6955765f44-bqrv6                         1/1     Running   0          12m
2020-11-02T14:27:12.7337001Z [cluster2] kube-system           pod/etcd-cluster2-control-plane                      1/1     Running   0          12m
2020-11-02T14:27:12.7338214Z [cluster2] kube-system           pod/kube-apiserver-cluster2-control-plane            1/1     Running   0          12m
2020-11-02T14:27:12.7340130Z [cluster2] kube-system           pod/kube-controller-manager-cluster2-control-plane   1/1     Running   0          12m
2020-11-02T14:27:12.7341701Z [cluster2] kube-system           pod/kube-proxy-6g8nn                                 1/1     Running   0          12m
2020-11-02T14:27:12.7342653Z [cluster2] kube-system           pod/kube-proxy-pk77v                                 1/1     Running   0          12m
2020-11-02T14:27:12.7343540Z [cluster2] kube-system           pod/kube-proxy-vbnbj                                 1/1     Running   0          12m
2020-11-02T14:27:12.7345167Z [cluster2] kube-system           pod/kube-scheduler-cluster2-control-plane            1/1     Running   0          12m
2020-11-02T14:27:12.7346591Z [cluster2] kube-system           pod/tiller-deploy-8488d98b4c-ngwv9                   1/1     Running   0          10m
2020-11-02T14:27:12.7347566Z [cluster2] kube-system           pod/weave-net-8k8k4                                  2/2     Running   0          12m
2020-11-02T14:27:12.7348420Z [cluster2] kube-system           pod/weave-net-wjrv2                                  2/2     Running   0          12m
2020-11-02T14:27:12.7349264Z [cluster2] kube-system           pod/weave-net-xz7dq                                  2/2     Running   0          12m
2020-11-02T14:27:12.7350416Z [cluster2] local-path-storage    pod/local-path-provisioner-7745554f7f-dt6rv          1/1     Running   0          12m
2020-11-02T14:27:12.7352321Z [cluster2] submariner-operator   pod/submariner-gateway-98th8                         1/1     Running   0          10m
2020-11-02T14:27:12.7353841Z [cluster2] submariner-operator   pod/submariner-lighthouse-agent-594fff74b7-kq8qj     1/1     Running   0          10m
2020-11-02T14:27:12.7355797Z [cluster2] submariner-operator   pod/submariner-lighthouse-coredns-7466b6679c-x49tl   1/1     Running   0          10m
2020-11-02T14:27:12.7357627Z [cluster2] submariner-operator   pod/submariner-lighthouse-coredns-7466b6679c-zzcjh   1/1     Running   0          10m
2020-11-02T14:27:12.7359159Z [cluster2] submariner-operator   pod/submariner-routeagent-4w68n                      1/1     Running   0          10m
2020-11-02T14:27:12.7360602Z [cluster2] submariner-operator   pod/submariner-routeagent-ct9kx                      1/1     Running   0          10m
2020-11-02T14:27:12.7361211Z [cluster2]
2020-11-02T14:27:12.7361907Z [cluster2] NAMESPACE             NAME                                    TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                  AGE
2020-11-02T14:27:12.7362537Z [cluster2] default               service/kubernetes                      ClusterIP   100.2.0.1       <none>        443/TCP                  13m
2020-11-02T14:27:12.7363426Z [cluster2] kube-system           service/kube-dns                        ClusterIP   100.2.0.10      <none>        53/UDP,53/TCP,9153/TCP   12m
2020-11-02T14:27:12.7364350Z [cluster2] kube-system           service/tiller-deploy                   ClusterIP   100.2.132.122   <none>        44134/TCP                10m
2020-11-02T14:27:12.7365506Z [cluster2] submariner-operator   service/submariner-lighthouse-coredns   ClusterIP   100.2.249.85    <none>        53/UDP                   10m
2020-11-02T14:27:12.7366344Z [cluster2]
2020-11-02T14:27:12.7367141Z [cluster2] NAMESPACE             NAME                                   DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR                 AGE
2020-11-02T14:27:12.7368204Z [cluster2] kube-system           daemonset.apps/kube-proxy              3         3         3       3            3           beta.kubernetes.io/os=linux   12m
2020-11-02T14:27:12.7369451Z [cluster2] kube-system           daemonset.apps/weave-net               3         3         3       3            3           <none>                        12m
2020-11-02T14:27:12.7370783Z [cluster2] submariner-operator   daemonset.apps/submariner-gateway      1         1         1       1            1           submariner.io/gateway=true    10m
2020-11-02T14:27:12.7404027Z [lighthouse]$ [cluster2] kubectl get pods -A
2020-11-02T14:27:12.7420745Z [lighthouse]$ [cluster2] kubectl get pods -A
2020-11-02T14:27:12.7428245Z [lighthouse]$ [cluster2] tail -n +2
2020-11-02T14:27:12.7436816Z [lighthouse]$ [cluster2] command kubectl --context=cluster2 get pods -A
2020-11-02T14:27:12.7451800Z [lighthouse]$ [cluster2] grep -v Running
2020-11-02T14:27:12.7460378Z [lighthouse]$ [cluster2] kubectl --context=cluster2 get pods -A
2020-11-02T14:27:12.7471663Z [lighthouse]$ [cluster2] sed s/  */;/g
2020-11-02T14:27:13.0669918Z [lighthouse]$ [cluster2] namespace=kube-system
2020-11-02T14:27:13.0693531Z [lighthouse]$ [cluster2] kubectl get pods --selector=k8s-app=kube-proxy -n kube-system -o jsonpath={.items[*].metadata.name}
2020-11-02T14:27:13.0708677Z [lighthouse]$ [cluster2] kubectl get pods --selector=k8s-app=kube-proxy -n kube-system -o jsonpath={.items[*].metadata.name}
2020-11-02T14:27:13.0728015Z [lighthouse]$ [cluster2] command kubectl --context=cluster2 get pods --selector=k8s-app=kube-proxy -n kube-system -o jsonpath={.items[*].metadata.name}
2020-11-02T14:27:13.0747304Z [lighthouse]$ [cluster2] kubectl --context=cluster2 get pods --selector=k8s-app=kube-proxy -n kube-system -o jsonpath={.items[*].metadata.name}
2020-11-02T14:27:13.3738858Z [lighthouse]$ [cluster2] kubectl -n kube-system logs kube-proxy-6g8nn
2020-11-02T14:27:13.3749632Z [lighthouse]$ [cluster2] kubectl -n kube-system logs kube-proxy-6g8nn
2020-11-02T14:27:13.3762870Z [lighthouse]$ [cluster2] command kubectl --context=cluster2 -n kube-system logs kube-proxy-6g8nn
2020-11-02T14:27:13.3773071Z [lighthouse]$ [cluster2] kubectl --context=cluster2 -n kube-system logs kube-proxy-6g8nn
2020-11-02T14:27:13.6878214Z [lighthouse]$ [cluster2] kubectl -n kube-system logs kube-proxy-pk77v
2020-11-02T14:27:13.6888172Z [lighthouse]$ [cluster2] kubectl -n kube-system logs kube-proxy-pk77v
2020-11-02T14:27:13.6898341Z [lighthouse]$ [cluster2] command kubectl --context=cluster2 -n kube-system logs kube-proxy-pk77v
2020-11-02T14:27:13.6910291Z [lighthouse]$ [cluster2] kubectl --context=cluster2 -n kube-system logs kube-proxy-pk77v
2020-11-02T14:27:13.9906502Z [cluster2] submariner-operator   daemonset.apps/submariner-routeagent   2         2         2       2            2           <none>                        10m
2020-11-02T14:27:13.9907267Z [cluster2]
2020-11-02T14:27:13.9907935Z [cluster2] NAMESPACE             NAME                                            READY   UP-TO-DATE   AVAILABLE   AGE
2020-11-02T14:27:13.9908719Z [cluster2] kube-system           deployment.apps/coredns                         2/2     2            2           12m
2020-11-02T14:27:13.9909549Z [cluster2] kube-system           deployment.apps/tiller-deploy                   1/1     1            1           10m
2020-11-02T14:27:13.9910533Z [cluster2] local-path-storage    deployment.apps/local-path-provisioner          1/1     1            1           12m
2020-11-02T14:27:13.9911890Z [cluster2] submariner-operator   deployment.apps/submariner-lighthouse-agent     1/1     1            1           10m
2020-11-02T14:27:13.9913201Z [cluster2] submariner-operator   deployment.apps/submariner-lighthouse-coredns   2/2     2            2           10m
2020-11-02T14:27:13.9913877Z [cluster2]
2020-11-02T14:27:13.9914274Z [cluster2] NAMESPACE             NAME                                                       DESIRED   CURRENT   READY   AGE
2020-11-02T14:27:13.9915466Z [cluster2] kube-system           replicaset.apps/coredns-6955765f44                         2         2         2       12m
2020-11-02T14:27:13.9916506Z [cluster2] kube-system           replicaset.apps/tiller-deploy-8488d98b4c                   1         1         1       10m
2020-11-02T14:27:13.9917772Z [cluster2] local-path-storage    replicaset.apps/local-path-provisioner-7745554f7f          1         1         1       12m
2020-11-02T14:27:13.9920592Z [cluster2] submariner-operator   replicaset.apps/submariner-lighthouse-agent-594fff74b7     1         1         1       10m
2020-11-02T14:27:13.9922621Z [cluster2] submariner-operator   replicaset.apps/submariner-lighthouse-coredns-7466b6679c   2         2         2       10m
2020-11-02T14:27:13.9924257Z [cluster2] +++++++++++++++++++++: Logs for Pod kube-proxy-6g8nn in namespace kube-system :++++++++++++++++++++++
2020-11-02T14:27:13.9925257Z [cluster2] W1102 14:14:31.562451       1 server_others.go:323] Unknown proxy mode "", assuming iptables proxy
2020-11-02T14:27:13.9925857Z [cluster2] I1102 14:14:31.570952       1 node.go:135] Successfully retrieved node IP: 172.17.0.7
2020-11-02T14:27:13.9926403Z [cluster2] I1102 14:14:31.570982       1 server_others.go:145] Using iptables Proxier.
2020-11-02T14:27:13.9926899Z [cluster2] I1102 14:14:31.571494       1 server.go:571] Version: v1.17.0
2020-11-02T14:27:13.9927390Z [cluster2] I1102 14:14:31.572030       1 conntrack.go:52] Setting nf_conntrack_max to 131072
2020-11-02T14:27:13.9928521Z [cluster2] I1102 14:14:31.572249       1 conntrack.go:100] Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_established' to 86400
2020-11-02T14:27:13.9929546Z [cluster2] I1102 14:14:31.572403       1 conntrack.go:100] Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_close_wait' to 3600
2020-11-02T14:27:13.9930234Z [cluster2] I1102 14:14:31.575340       1 config.go:313] Starting service config controller
2020-11-02T14:27:13.9931026Z [cluster2] I1102 14:14:31.575452       1 shared_informer.go:197] Waiting for caches to sync for service config
2020-11-02T14:27:13.9931651Z [cluster2] I1102 14:14:31.575542       1 config.go:131] Starting endpoints config controller
2020-11-02T14:27:13.9932287Z [cluster2] I1102 14:14:31.575619       1 shared_informer.go:197] Waiting for caches to sync for endpoints config
2020-11-02T14:27:13.9932914Z [cluster2] I1102 14:14:31.675739       1 shared_informer.go:204] Caches are synced for service config
2020-11-02T14:27:13.9933548Z [cluster2] I1102 14:14:31.675965       1 shared_informer.go:204] Caches are synced for endpoints config
2020-11-02T14:27:13.9934414Z [cluster2] +++++++++++++++++++++: Logs for Pod kube-proxy-pk77v in namespace kube-system :++++++++++++++++++++++
2020-11-02T14:27:13.9935242Z [cluster2] W1102 14:14:59.040452       1 server_others.go:323] Unknown proxy mode "", assuming iptables proxy
2020-11-02T14:27:13.9936322Z [cluster2] I1102 14:14:59.095078       1 node.go:135] Successfully retrieved node IP: 172.17.0.8
2020-11-02T14:27:13.9936920Z [cluster2] I1102 14:14:59.095112       1 server_others.go:145] Using iptables Proxier.
2020-11-02T14:27:13.9937453Z [cluster2] I1102 14:14:59.095533       1 server.go:571] Version: v1.17.0
2020-11-02T14:27:13.9937992Z [cluster2] I1102 14:14:59.095939       1 conntrack.go:52] Setting nf_conntrack_max to 131072
2020-11-02T14:27:13.9938995Z [cluster2] I1102 14:14:59.096108       1 conntrack.go:100] Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_established' to 86400
2020-11-02T14:27:13.9940104Z [cluster2] I1102 14:14:59.096175       1 conntrack.go:100] Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_close_wait' to 3600
2020-11-02T14:27:13.9940860Z [cluster2] I1102 14:14:59.097244       1 config.go:131] Starting endpoints config controller
2020-11-02T14:27:13.9941564Z [cluster2] I1102 14:14:59.097259       1 shared_informer.go:197] Waiting for caches to sync for endpoints config
2020-11-02T14:27:13.9942425Z [cluster2] I1102 14:14:59.097289       1 config.go:313] Starting service config controller
2020-11-02T14:27:13.9985665Z [lighthouse]$ [cluster2] kubectl -n kube-system logs kube-proxy-vbnbj
2020-11-02T14:27:13.9997995Z [lighthouse]$ [cluster2] kubectl -n kube-system logs kube-proxy-vbnbj
2020-11-02T14:27:14.0013969Z [lighthouse]$ [cluster2] command kubectl --context=cluster2 -n kube-system logs kube-proxy-vbnbj
2020-11-02T14:27:14.0027154Z [lighthouse]$ [cluster2] kubectl --context=cluster2 -n kube-system logs kube-proxy-vbnbj
2020-11-02T14:27:14.3241702Z [lighthouse]$ [cluster2] namespace=submariner-operator
2020-11-02T14:27:14.3263455Z [lighthouse]$ [cluster2] kubectl get pods --selector=name=submariner-operator -n submariner-operator -o jsonpath={.items[*].metadata.name}
2020-11-02T14:27:14.3279033Z [lighthouse]$ [cluster2] kubectl get pods --selector=name=submariner-operator -n submariner-operator -o jsonpath={.items[*].metadata.name}
2020-11-02T14:27:14.3292384Z [lighthouse]$ [cluster2] command kubectl --context=cluster2 get pods --selector=name=submariner-operator -n submariner-operator -o jsonpath={.items[*].metadata.name}
2020-11-02T14:27:14.3305615Z [lighthouse]$ [cluster2] kubectl --context=cluster2 get pods --selector=name=submariner-operator -n submariner-operator -o jsonpath={.items[*].metadata.name}
2020-11-02T14:27:14.6179623Z [lighthouse]$ [cluster2] namespace=submariner-operator
2020-11-02T14:27:14.6196211Z [lighthouse]$ [cluster2] kubectl get pods --selector=app=submariner-globalnet -n submariner-operator -o jsonpath={.items[*].metadata.name}
2020-11-02T14:27:14.6209497Z [lighthouse]$ [cluster2] kubectl get pods --selector=app=submariner-globalnet -n submariner-operator -o jsonpath={.items[*].metadata.name}
2020-11-02T14:27:14.6225294Z [lighthouse]$ [cluster2] command kubectl --context=cluster2 get pods --selector=app=submariner-globalnet -n submariner-operator -o jsonpath={.items[*].metadata.name}
2020-11-02T14:27:14.6246133Z [lighthouse]$ [cluster2] kubectl --context=cluster2 get pods --selector=app=submariner-globalnet -n submariner-operator -o jsonpath={.items[*].metadata.name}
2020-11-02T14:27:14.9169710Z [lighthouse]$ [cluster2] kubectl get pods --selector=app=submariner-engine -n submariner-operator -o jsonpath={.items[*].metadata.name}
2020-11-02T14:27:14.9200099Z [lighthouse]$ [cluster2] kubectl get pods --selector=app=submariner-engine -n submariner-operator -o jsonpath={.items[*].metadata.name}
2020-11-02T14:27:14.9212585Z [lighthouse]$ [cluster2] command kubectl --context=cluster2 get pods --selector=app=submariner-engine -n submariner-operator -o jsonpath={.items[*].metadata.name}
2020-11-02T14:27:14.9224027Z [lighthouse]$ [cluster2] kubectl --context=cluster2 get pods --selector=app=submariner-engine -n submariner-operator -o jsonpath={.items[*].metadata.name}
2020-11-02T14:27:15.2301537Z [lighthouse]$ [cluster2] kubectl -n submariner-operator logs submariner-gateway-98th8
2020-11-02T14:27:15.2315467Z [lighthouse]$ [cluster2] kubectl -n submariner-operator logs submariner-gateway-98th8
2020-11-02T14:27:15.2333213Z [lighthouse]$ [cluster2] command kubectl --context=cluster2 -n submariner-operator logs submariner-gateway-98th8
2020-11-02T14:27:15.2354892Z [lighthouse]$ [cluster2] kubectl --context=cluster2 -n submariner-operator logs submariner-gateway-98th8
2020-11-02T14:27:15.5314763Z [cluster2] I1102 14:14:59.097292       1 shared_informer.go:197] Waiting for caches to sync for service config
2020-11-02T14:27:15.5315762Z [cluster2] I1102 14:14:59.199023       1 shared_informer.go:204] Caches are synced for service config
2020-11-02T14:27:15.5316431Z [cluster2] I1102 14:14:59.199237       1 shared_informer.go:204] Caches are synced for endpoints config
2020-11-02T14:27:15.5317757Z [cluster2] +++++++++++++++++++++: Logs for Pod kube-proxy-vbnbj in namespace kube-system :++++++++++++++++++++++
2020-11-02T14:27:15.5318446Z [cluster2] W1102 14:14:59.022141       1 server_others.go:323] Unknown proxy mode "", assuming iptables proxy
2020-11-02T14:27:15.5319383Z [cluster2] I1102 14:14:59.452117       1 node.go:135] Successfully retrieved node IP: 172.17.0.9
2020-11-02T14:27:15.5319908Z [cluster2] I1102 14:14:59.452153       1 server_others.go:145] Using iptables Proxier.
2020-11-02T14:27:15.5320387Z [cluster2] I1102 14:14:59.455958       1 server.go:571] Version: v1.17.0
2020-11-02T14:27:15.5320861Z [cluster2] I1102 14:14:59.456391       1 conntrack.go:52] Setting nf_conntrack_max to 131072
2020-11-02T14:27:15.5322496Z [cluster2] I1102 14:14:59.460296       1 conntrack.go:100] Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_established' to 86400
2020-11-02T14:27:15.5323984Z [cluster2] I1102 14:14:59.460362       1 conntrack.go:100] Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_close_wait' to 3600
2020-11-02T14:27:15.5324723Z [cluster2] I1102 14:14:59.469390       1 config.go:313] Starting service config controller
2020-11-02T14:27:15.5325405Z [cluster2] I1102 14:14:59.469406       1 shared_informer.go:197] Waiting for caches to sync for service config
2020-11-02T14:27:15.5326077Z [cluster2] I1102 14:14:59.469427       1 config.go:131] Starting endpoints config controller
2020-11-02T14:27:15.5326786Z [cluster2] I1102 14:14:59.469435       1 shared_informer.go:197] Waiting for caches to sync for endpoints config
2020-11-02T14:27:15.5327950Z [cluster2] I1102 14:14:59.569577       1 shared_informer.go:204] Caches are synced for service config
2020-11-02T14:27:15.5328582Z [cluster2] I1102 14:14:59.569681       1 shared_informer.go:204] Caches are synced for endpoints config
2020-11-02T14:27:15.5329053Z [cluster2] ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
2020-11-02T14:27:15.5329355Z [cluster2] ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
2020-11-02T14:27:15.5329811Z [cluster2] ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
2020-11-02T14:27:15.5330645Z [cluster2] +++++++++++++++++++++: Logs for Pod submariner-gateway-98th8 in namespace submariner-operator :++++++++++++++++++++++
2020-11-02T14:27:15.5331431Z [cluster2] + trap 'exit 1' SIGTERM SIGINT
2020-11-02T14:27:15.5331855Z [cluster2] + export CHARON_PID_FILE=/var/run/charon.pid
2020-11-02T14:27:15.5332296Z [cluster2] + CHARON_PID_FILE=/var/run/charon.pid
2020-11-02T14:27:15.5332801Z [cluster2] + rm -f /var/run/charon.pid
2020-11-02T14:27:15.5333182Z [cluster2] + SUBMARINER_VERBOSITY=2
2020-11-02T14:27:15.5333662Z [cluster2] + '[' false == true ']'
2020-11-02T14:27:15.5334071Z [cluster2] + DEBUG=-v=2
2020-11-02T14:27:15.5334502Z [cluster2] + mkdir -p /etc/ipsec
2020-11-02T14:27:15.5335489Z [cluster2] + sysctl -w net.ipv4.conf.all.send_redirects=0
2020-11-02T14:27:15.5336095Z [cluster2] net.ipv4.conf.all.send_redirects = 0
2020-11-02T14:27:15.5336739Z [cluster2] + export PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/libexec/strongswan
2020-11-02T14:27:15.5337457Z [cluster2] + PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/libexec/strongswan
2020-11-02T14:27:15.5338188Z [cluster2] + for f in iptables-save iptables
2020-11-02T14:27:15.5338816Z [cluster2] ++ find_iptables_on_host iptables-save
2020-11-02T14:27:15.5339473Z [cluster2] ++ chroot /host test -x /usr/sbin/iptables-save
2020-11-02T14:27:15.5340234Z [cluster2] chroot: cannot change root directory to '/host': No such file or directory
2020-11-02T14:27:15.5340970Z [cluster2] ++ chroot /host test -x /sbin/iptables-save
2020-11-02T14:27:15.5342092Z [cluster2] chroot: cannot change root directory to '/host': No such file or directory
2020-11-02T14:27:15.5342625Z [cluster2] ++ echo unknown
2020-11-02T14:27:15.5342976Z [cluster2] + location=unknown
2020-11-02T14:27:15.5343808Z [cluster2] + '[' unknown '!=' unknown ']'
2020-11-02T14:27:15.5344617Z [cluster2] + echo 'WARNING: not using iptables wrapper because iptables was not detected on the'
2020-11-02T14:27:15.5345393Z [cluster2] WARNING: not using iptables wrapper because iptables was not detected on the
2020-11-02T14:27:15.5346389Z [cluster2] + echo 'host at the following paths [/usr/sbin, /sbin].'
2020-11-02T14:27:15.5346895Z [cluster2] host at the following paths [/usr/sbin, /sbin].
2020-11-02T14:27:15.5347676Z [cluster2] + echo 'Either the host file system isn'\''t mounted or the host does not have iptables'
2020-11-02T14:27:15.5348988Z [cluster2] Either the host file system isn't mounted or the host does not have iptables
2020-11-02T14:27:15.5350158Z [cluster2] + echo 'installed. The pod will use the image installed iptables version.'
2020-11-02T14:27:15.5350895Z [cluster2] installed. The pod will use the image installed iptables version.
2020-11-02T14:27:15.5351783Z [cluster2] + for f in iptables-save iptables
2020-11-02T14:27:15.5352219Z [cluster2] ++ find_iptables_on_host iptables
2020-11-02T14:27:15.5390207Z [cluster2] ++ chroot /host test -x /usr/sbin/iptables
2020-11-02T14:27:15.5391016Z [cluster2] chroot: cannot change root directory to '/host': No such file or directory
2020-11-02T14:27:15.5392003Z [cluster2] ++ chroot /host test -x /sbin/iptables
2020-11-02T14:27:15.5392700Z [cluster2] chroot: cannot change root directory to '/host': No such file or directory
2020-11-02T14:27:15.5393164Z [cluster2] ++ echo unknown
2020-11-02T14:27:15.5393490Z [cluster2] + location=unknown
2020-11-02T14:27:15.5394123Z [cluster2] + '[' unknown '!=' unknown ']'
2020-11-02T14:27:15.5394824Z [cluster2] + echo 'WARNING: not using iptables wrapper because iptables was not detected on the'
2020-11-02T14:27:15.5395521Z [cluster2] WARNING: not using iptables wrapper because iptables was not detected on the
2020-11-02T14:27:15.5396238Z [cluster2] + echo 'host at the following paths [/usr/sbin, /sbin].'
2020-11-02T14:27:15.5396716Z [cluster2] host at the following paths [/usr/sbin, /sbin].
2020-11-02T14:27:15.5397447Z [cluster2] + echo 'Either the host file system isn'\''t mounted or the host does not have iptables'
2020-11-02T14:27:15.5398246Z [cluster2] Either the host file system isn't mounted or the host does not have iptables
2020-11-02T14:27:15.5399041Z [cluster2] + echo 'installed. The pod will use the image installed iptables version.'
2020-11-02T14:27:15.5399869Z [cluster2] installed. The pod will use the image installed iptables version.
2020-11-02T14:27:15.5400645Z [cluster2] + exec submariner-engine -v=2 -alsologtostderr
2020-11-02T14:27:15.5401388Z [cluster2] I1102 14:17:35.254358       1 main.go:70] Starting the submariner gateway engine
2020-11-02T14:27:15.5403177Z [cluster2] W1102 14:17:35.259404       1 client_config.go:543] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
2020-11-02T14:27:15.5404184Z [cluster2] I1102 14:17:35.259821       1 main.go:95] Creating the cable engine
2020-11-02T14:27:15.5404722Z [cluster2] I1102 14:17:35.262322       1 syncer.go:61] CableEngine syncer started
2020-11-02T14:27:15.5405606Z [cluster2] I1102 14:17:40.866685       1 main.go:252] Gateway leader election config values: main.leaderConfig{LeaseDuration:10, RenewDeadline:5, RetryPeriod:2}
2020-11-02T14:27:15.5407217Z [cluster2] I1102 14:17:40.866991       1 main.go:269] Using namespace "submariner-operator" for the leader election lock
2020-11-02T14:27:15.5409262Z [cluster2] I1102 14:17:40.867018       1 leaderelection.go:242] attempting to acquire leader lease  submariner-operator/submariner-engine-lock...
2020-11-02T14:27:15.5410732Z [cluster2] I1102 14:17:40.881590       1 leaderelection.go:252] successfully acquired lease submariner-operator/submariner-engine-lock
2020-11-02T14:27:15.5413319Z [cluster2] I1102 14:17:40.881857       1 event.go:281] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"submariner-operator", Name:"submariner-engine-lock", UID:"178d6442-df04-4405-8429-cea22fefc758", APIVersion:"v1", ResourceVersion:"1250", FieldPath:""}): type: 'Normal' reason: 'LeaderElection' cluster2-worker-submariner-engine became leader
2020-11-02T14:27:15.5414918Z [cluster2] I1102 14:17:40.881930       1 main.go:144] Creating the datastore syncer
2020-11-02T14:27:15.5415507Z [cluster2] I1102 14:17:40.881990       1 strongswan.go:99] Initializing StrongSwan IPSec driver
2020-11-02T14:27:15.5416088Z [cluster2] I1102 14:17:40.882457       1 strongswan.go:460] Starting charon
2020-11-02T14:27:15.5417529Z [cluster2] W1102 14:17:40.890498       1 strongswan.go:543] Failed to connect to charon - retrying: dial unix /var/run/charon.vici: connect: no such file or directory
2020-11-02T14:27:15.5419166Z [cluster2] 00[DMN] Starting IKE charon daemon (strongSwan 5.9.0, Linux 5.4.0-1031-azure, x86_64)
2020-11-02T14:27:15.5420110Z [cluster2] 00[CFG] PKCS11 module '<name>' lacks library path
2020-11-02T14:27:15.5420870Z [cluster2] 00[LIB] openssl FIPS mode(0) - disabled
2020-11-02T14:27:15.5421712Z [cluster2] 00[CFG] loading ca certificates from '/etc/strongswan/ipsec.d/cacerts'
2020-11-02T14:27:15.5422636Z [cluster2] 00[CFG] loading aa certificates from '/etc/strongswan/ipsec.d/aacerts'
2020-11-02T14:27:15.5423612Z [cluster2] 00[CFG] loading ocsp signer certificates from '/etc/strongswan/ipsec.d/ocspcerts'
2020-11-02T14:27:15.5424615Z [cluster2] 00[CFG] loading attribute certificates from '/etc/strongswan/ipsec.d/acerts'
2020-11-02T14:27:15.5425641Z [cluster2] 00[CFG] loading crls from '/etc/strongswan/ipsec.d/crls'
2020-11-02T14:27:15.5426644Z [cluster2] 00[CFG] loading secrets from '/etc/strongswan/ipsec.secrets'
2020-11-02T14:27:15.5427448Z [cluster2] 00[CFG] opening triplet file /etc/strongswan/ipsec.d/triplets.dat failed: No such file or directory
2020-11-02T14:27:15.5428203Z [cluster2] 00[CFG] loaded 0 RADIUS server configurations
2020-11-02T14:27:15.5428742Z [cluster2] 00[CFG] HA config misses local/remote address
2020-11-02T14:27:15.5429790Z [cluster2] 00[CFG] no script for ext-auth script defined, disabled
2020-11-02T14:27:15.5436697Z [cluster2] 00[LIB] loaded plugins: charon pkcs11 tpm aesni aes des rc2 sha2 sha1 md4 md5 mgf1 random nonce x509 revocation constraints acert pubkey pkcs1 pkcs7 pkcs8 pkcs12 pgp dnskey sshkey pem openssl gcrypt fips-prf gmp curve25519 chapoly xcbc cmac hmac ctr ccm gcm drbg newhope curl attr kernel-netlink resolve socket-default farp stroke vici updown eap-identity eap-sim eap-aka eap-aka-3gpp eap-aka-3gpp2 eap-md5 eap-gtc eap-mschapv2 eap-dynamic eap-radius eap-tls eap-ttls eap-peap xauth-generic xauth-eap xauth-pam xauth-noauth dhcp led duplicheck unity counters
2020-11-02T14:27:15.5438665Z [cluster2] 00[JOB] spawning 16 worker threads
2020-11-02T14:27:15.5439464Z [cluster2] I1102 14:17:42.043212       1 cableengine.go:71] CableEngine controller started, driver: "strongswan"
2020-11-02T14:27:15.5440659Z [cluster2] I1102 14:17:42.043607       1 datastoresyncer.go:60] Starting the datastore syncer
2020-11-02T14:27:15.5441792Z [cluster2] I1102 14:17:42.043632       1 tunnel.go:22] Starting the tunnel controller
2020-11-02T14:27:15.5443282Z [cluster2] I1102 14:17:42.066012       1 helpers.go:75] Found schema.GroupVersionResource{Group:"submariner.io", Version:"v1", Resource:"endpoints"}
2020-11-02T14:27:15.5444280Z [cluster2] I1102 14:17:42.067095       1 resource_syncer.go:180] Starting syncer "Tunnel controller"
2020-11-02T14:27:15.5445057Z [cluster2] I1102 14:17:42.067170       1 resource_syncer.go:193] Syncer "Tunnel controller" waiting for informer cache to sync
2020-11-02T14:27:15.5445819Z [cluster2] I1102 14:17:42.167324       1 resource_syncer.go:202] Syncer "Tunnel controller" started
2020-11-02T14:27:15.5447125Z [cluster2] I1102 14:17:47.652400       1 helpers.go:75] Found schema.GroupVersionResource{Group:"submariner.io", Version:"v1", Resource:"clusters"}
2020-11-02T14:27:15.5448582Z [cluster2] I1102 14:17:47.809453       1 helpers.go:75] Found schema.GroupVersionResource{Group:"submariner.io", Version:"v1", Resource:"clusters"}
2020-11-02T14:27:15.5449872Z [cluster2] I1102 14:17:47.849338       1 helpers.go:75] Found schema.GroupVersionResource{Group:"submariner.io", Version:"v1", Resource:"clusters"}
2020-11-02T14:27:15.5459365Z [cluster2] I1102 14:17:47.849525       1 helpers.go:75] Found schema.GroupVersionResource{Group:"submariner.io", Version:"v1", Resource:"clusters"}
2020-11-02T14:27:15.5460928Z [cluster2] I1102 14:17:47.849734       1 helpers.go:75] Found schema.GroupVersionResource{Group:"submariner.io", Version:"v1", Resource:"endpoints"}
2020-11-02T14:27:15.5462510Z [cluster2] I1102 14:17:47.849908       1 helpers.go:75] Found schema.GroupVersionResource{Group:"submariner.io", Version:"v1", Resource:"endpoints"}
2020-11-02T14:27:15.5463503Z [cluster2] I1102 14:17:47.850012       1 datastoresyncer.go:80] Starting the broker syncer
2020-11-02T14:27:15.5464936Z [cluster2] I1102 14:17:47.850936       1 resource_syncer.go:180] Starting syncer "local -> broker for *v1.Cluster"
2020-11-02T14:27:15.5466066Z [cluster2] I1102 14:17:47.850971       1 resource_syncer.go:193] Syncer "local -> broker for *v1.Cluster" waiting for informer cache to sync
2020-11-02T14:27:15.5468416Z [cluster2] I1102 14:17:47.951228       1 resource_syncer.go:202] Syncer "local -> broker for *v1.Cluster" started
2020-11-02T14:27:15.5469805Z [cluster2] I1102 14:17:47.951282       1 resource_syncer.go:180] Starting syncer "broker -> local for *v1.Cluster"
2020-11-02T14:27:15.5470764Z [cluster2] I1102 14:17:47.951297       1 resource_syncer.go:202] Syncer "broker -> local for *v1.Cluster" started
2020-11-02T14:27:15.5472900Z [cluster2] I1102 14:17:47.951307       1 resource_syncer.go:180] Starting syncer "local -> broker for *v1.Endpoint"
2020-11-02T14:27:15.5473896Z [cluster2] I1102 14:17:47.951319       1 resource_syncer.go:193] Syncer "local -> broker for *v1.Endpoint" waiting for informer cache to sync
2020-11-02T14:27:15.5474851Z [cluster2] I1102 14:17:48.051458       1 resource_syncer.go:202] Syncer "local -> broker for *v1.Endpoint" started
2020-11-02T14:27:15.5475729Z [cluster2] I1102 14:17:48.051500       1 resource_syncer.go:180] Starting syncer "broker -> local for *v1.Endpoint"
2020-11-02T14:27:15.5476604Z [cluster2] I1102 14:17:48.051518       1 resource_syncer.go:202] Syncer "broker -> local for *v1.Endpoint" started
2020-11-02T14:27:15.5477332Z [cluster2] I1102 14:17:48.051528       1 datastoresyncer.go:108] Ensuring we are the only endpoint active for this cluster
2020-11-02T14:27:15.5478709Z [cluster2] I1102 14:17:48.051540       1 datastoresyncer.go:143] Creating local submariner Cluster: types.SubmarinerCluster{ID:"cluster2", Spec:v1.ClusterSpec{ClusterID:"cluster2", ColorCodes:[]string{"blue"}, ServiceCIDR:[]string{"100.2.0.0/16"}, ClusterCIDR:[]string{"10.2.0.0/16"}, GlobalCIDR:[]string{}}}
2020-11-02T14:27:15.5482243Z [cluster2] I1102 14:17:48.051603       1 federator.go:42] In Distribute for &v1.Cluster{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"cluster2", GenerateName:"", Namespace:"", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v1.ClusterSpec{ClusterID:"cluster2", ColorCodes:[]string{"blue"}, ServiceCIDR:[]string{"100.2.0.0/16"}, ClusterCIDR:[]string{"10.2.0.0/16"}, GlobalCIDR:[]string{}}}
2020-11-02T14:27:15.5486186Z [cluster2] I1102 14:17:48.051731       1 helpers.go:75] Found schema.GroupVersionResource{Group:"submariner.io", Version:"v1", Resource:"clusters"}
2020-11-02T14:27:15.5489413Z [cluster2] I1102 14:17:48.053995       1 create_or_update.go:32] Creating resource: &unstructured.Unstructured{Object:map[string]interface {}{"apiVersion":"submariner.io/v1", "kind":"Cluster", "metadata":map[string]interface {}{"name":"cluster2", "namespace":"submariner-operator"}, "spec":map[string]interface {}{"cluster_cidr":[]interface {}{"10.2.0.0/16"}, "cluster_id":"cluster2", "color_codes":[]interface {}{"blue"}, "global_cidr":[]interface {}{}, "service_cidr":[]interface {}{"100.2.0.0/16"}}}}
2020-11-02T14:27:15.5493578Z [cluster2] I1102 14:17:48.057551       1 datastoresyncer.go:156] Creating local submariner Endpoint: types.SubmarinerEndpoint{Spec:v1.EndpointSpec{ClusterID:"cluster2", CableName:"submariner-cable-cluster2-172-17-0-9", Hostname:"cluster2-worker", Subnets:[]string{"100.2.0.0/16", "10.2.0.0/16"}, PrivateIP:"172.17.0.9", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string{}}}
2020-11-02T14:27:15.5508744Z [cluster2] I1102 14:17:48.057863       1 federator.go:42] In Distribute for &v1.Endpoint{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"cluster2-submariner-cable-cluster2-172-17-0-9", GenerateName:"", Namespace:"", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v1.EndpointSpec{ClusterID:"cluster2", CableName:"submariner-cable-cluster2-172-17-0-9", Hostname:"cluster2-worker", Subnets:[]string{"100.2.0.0/16", "10.2.0.0/16"}, PrivateIP:"172.17.0.9", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string{}}}
2020-11-02T14:27:15.5513808Z [cluster2] I1102 14:17:48.058050       1 helpers.go:75] Found schema.GroupVersionResource{Group:"submariner.io", Version:"v1", Resource:"endpoints"}
2020-11-02T14:27:15.5518562Z [cluster2] I1102 14:17:48.064704       1 resource_syncer.go:264] Syncer "local -> broker for *v1.Cluster" retrieved created resource: &unstructured.Unstructured{Object:map[string]interface {}{"apiVersion":"submariner.io/v1", "kind":"Cluster", "metadata":map[string]interface {}{"creationTimestamp":"2020-11-02T14:17:48Z", "generation":1, "name":"cluster2", "namespace":"submariner-operator", "resourceVersion":"1278", "selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/clusters/cluster2", "uid":"6f5da36a-99eb-4a8d-8eb7-ed75657d8587"}, "spec":map[string]interface {}{"cluster_cidr":[]interface {}{"10.2.0.0/16"}, "cluster_id":"cluster2", "color_codes":[]interface {}{"blue"}, "global_cidr":[]interface {}{}, "service_cidr":[]interface {}{"100.2.0.0/16"}}}}
2020-11-02T14:27:15.5525029Z [cluster2] I1102 14:17:48.064803       1 resource_syncer.go:276] Syncing resource: &unstructured.Unstructured{Object:map[string]interface {}{"apiVersion":"submariner.io/v1", "kind":"Cluster", "metadata":map[string]interface {}{"creationTimestamp":"2020-11-02T14:17:48Z", "generation":1, "name":"cluster2", "namespace":"submariner-operator", "resourceVersion":"1278", "selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/clusters/cluster2", "uid":"6f5da36a-99eb-4a8d-8eb7-ed75657d8587"}, "spec":map[string]interface {}{"cluster_cidr":[]interface {}{"10.2.0.0/16"}, "cluster_id":"cluster2", "color_codes":[]interface {}{"blue"}, "global_cidr":[]interface {}{}, "service_cidr":[]interface {}{"100.2.0.0/16"}}}}
2020-11-02T14:27:15.5531704Z [cluster2] I1102 14:17:48.064842       1 federator.go:42] In Distribute for &unstructured.Unstructured{Object:map[string]interface {}{"apiVersion":"submariner.io/v1", "kind":"Cluster", "metadata":map[string]interface {}{"creationTimestamp":"2020-11-02T14:17:48Z", "generation":1, "name":"cluster2", "namespace":"submariner-operator", "resourceVersion":"1278", "selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/clusters/cluster2", "uid":"6f5da36a-99eb-4a8d-8eb7-ed75657d8587"}, "spec":map[string]interface {}{"cluster_cidr":[]interface {}{"10.2.0.0/16"}, "cluster_id":"cluster2", "color_codes":[]interface {}{"blue"}, "global_cidr":[]interface {}{}, "service_cidr":[]interface {}{"100.2.0.0/16"}}}}
2020-11-02T14:27:15.5535100Z [cluster2] I1102 14:17:48.064895       1 helpers.go:75] Found schema.GroupVersionResource{Group:"submariner.io", Version:"v1", Resource:"clusters"}
2020-11-02T14:27:15.5592880Z [cluster2] I1102 14:17:48.066595       1 create_or_update.go:32] Creating resource: &unstructured.Unstructured{Object:map[string]interface {}{"apiVersion":"submariner.io/v1", "kind":"Cluster", "metadata":map[string]interface {}{"labels":map[string]interface {}{"submariner-io/clusterID":"cluster2"}, "name":"cluster2", "namespace":"submariner-k8s-broker"}, "spec":map[string]interface {}{"cluster_cidr":[]interface {}{"10.2.0.0/16"}, "cluster_id":"cluster2", "color_codes":[]interface {}{"blue"}, "global_cidr":[]interface {}{}, "service_cidr":[]interface {}{"100.2.0.0/16"}}}}
2020-11-02T14:27:15.5598177Z [cluster2] I1102 14:17:48.086687       1 create_or_update.go:32] Creating resource: &unstructured.Unstructured{Object:map[string]interface {}{"apiVersion":"submariner.io/v1", "kind":"Endpoint", "metadata":map[string]interface {}{"name":"cluster2-submariner-cable-cluster2-172-17-0-9", "namespace":"submariner-operator"}, "spec":map[string]interface {}{"backend":"strongswan", "cable_name":"submariner-cable-cluster2-172-17-0-9", "cluster_id":"cluster2", "hostname":"cluster2-worker", "nat_enabled":false, "private_ip":"172.17.0.9", "public_ip":"", "subnets":[]interface {}{"100.2.0.0/16", "10.2.0.0/16"}}}}
2020-11-02T14:27:15.5601542Z [cluster2] I1102 14:17:48.089557       1 resource_syncer.go:285] Syncer "local -> broker for *v1.Cluster" successfully synced "submariner-operator/cluster2"
2020-11-02T14:27:15.5605557Z [cluster2] I1102 14:17:48.094636       1 resource_syncer.go:264] Syncer "broker -> local for *v1.Cluster" retrieved created resource: &unstructured.Unstructured{Object:map[string]interface {}{"apiVersion":"submariner.io/v1", "kind":"Cluster", "metadata":map[string]interface {}{"creationTimestamp":"2020-11-02T14:17:48Z", "generation":1, "labels":map[string]interface {}{"submariner-io/clusterID":"cluster2"}, "name":"cluster2", "namespace":"submariner-k8s-broker", "resourceVersion":"1322", "selfLink":"/apis/submariner.io/v1/namespaces/submariner-k8s-broker/clusters/cluster2", "uid":"e0a3f487-3b0b-4398-947d-a1c32af739e8"}, "spec":map[string]interface {}{"cluster_cidr":[]interface {}{"10.2.0.0/16"}, "cluster_id":"cluster2", "color_codes":[]interface {}{"blue"}, "global_cidr":[]interface {}{}, "service_cidr":[]interface {}{"100.2.0.0/16"}}}}
2020-11-02T14:27:15.5608943Z [cluster2] I1102 14:17:48.094812       1 resource_syncer.go:443] Cluster ID label "cluster2" not present or matches local cluster ID "cluster2" - not syncing resource "cluster2"
2020-11-02T14:27:15.5609813Z [cluster2] I1102 14:17:48.109066       1 datastoresyncer.go:99] Datastore syncer started
2020-11-02T14:27:15.5611280Z [lighthouse]$ [cluster2] kubectl get pods --selector=app=submariner-routeagent -n submariner-operator -o jsonpath={.items[*].metadata.name}
2020-11-02T14:27:15.5612723Z [lighthouse]$ [cluster2] kubectl get pods --selector=app=submariner-routeagent -n submariner-operator -o jsonpath={.items[*].metadata.name}
2020-11-02T14:27:15.5614424Z [lighthouse]$ [cluster2] command kubectl --context=cluster2 get pods --selector=app=submariner-routeagent -n submariner-operator -o jsonpath={.items[*].metadata.name}
2020-11-02T14:27:15.5616218Z [lighthouse]$ [cluster2] kubectl --context=cluster2 get pods --selector=app=submariner-routeagent -n submariner-operator -o jsonpath={.items[*].metadata.name}
2020-11-02T14:27:15.5622779Z [cluster2] I1102 14:17:48.114324       1 resource_syncer.go:264] Syncer "local -> broker for *v1.Endpoint" retrieved created resource: &unstructured.Unstructured{Object:map[string]interface {}{"apiVersion":"submariner.io/v1", "kind":"Endpoint", "metadata":map[string]interface {}{"creationTimestamp":"2020-11-02T14:17:48Z", "generation":1, "name":"cluster2-submariner-cable-cluster2-172-17-0-9", "namespace":"submariner-operator", "resourceVersion":"1279", "selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster2-submariner-cable-cluster2-172-17-0-9", "uid":"1112af78-f959-40d1-b570-c351ce0c602a"}, "spec":map[string]interface {}{"backend":"strongswan", "cable_name":"submariner-cable-cluster2-172-17-0-9", "cluster_id":"cluster2", "hostname":"cluster2-worker", "nat_enabled":false, "private_ip":"172.17.0.9", "public_ip":"", "subnets":[]interface {}{"100.2.0.0/16", "10.2.0.0/16"}}}}
2020-11-02T14:27:15.5631745Z [cluster2] I1102 14:17:48.114430       1 resource_syncer.go:276] Syncing resource: &unstructured.Unstructured{Object:map[string]interface {}{"apiVersion":"submariner.io/v1", "kind":"Endpoint", "metadata":map[string]interface {}{"creationTimestamp":"2020-11-02T14:17:48Z", "generation":1, "name":"cluster2-submariner-cable-cluster2-172-17-0-9", "namespace":"submariner-operator", "resourceVersion":"1279", "selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster2-submariner-cable-cluster2-172-17-0-9", "uid":"1112af78-f959-40d1-b570-c351ce0c602a"}, "spec":map[string]interface {}{"backend":"strongswan", "cable_name":"submariner-cable-cluster2-172-17-0-9", "cluster_id":"cluster2", "hostname":"cluster2-worker", "nat_enabled":false, "private_ip":"172.17.0.9", "public_ip":"", "subnets":[]interface {}{"100.2.0.0/16", "10.2.0.0/16"}}}}
2020-11-02T14:27:15.5639284Z [cluster2] I1102 14:17:48.114500       1 federator.go:42] In Distribute for &unstructured.Unstructured{Object:map[string]interface {}{"apiVersion":"submariner.io/v1", "kind":"Endpoint", "metadata":map[string]interface {}{"creationTimestamp":"2020-11-02T14:17:48Z", "generation":1, "name":"cluster2-submariner-cable-cluster2-172-17-0-9", "namespace":"submariner-operator", "resourceVersion":"1279", "selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster2-submariner-cable-cluster2-172-17-0-9", "uid":"1112af78-f959-40d1-b570-c351ce0c602a"}, "spec":map[string]interface {}{"backend":"strongswan", "cable_name":"submariner-cable-cluster2-172-17-0-9", "cluster_id":"cluster2", "hostname":"cluster2-worker", "nat_enabled":false, "private_ip":"172.17.0.9", "public_ip":"", "subnets":[]interface {}{"100.2.0.0/16", "10.2.0.0/16"}}}}
2020-11-02T14:27:15.5643134Z [cluster2] I1102 14:17:48.114578       1 helpers.go:75] Found schema.GroupVersionResource{Group:"submariner.io", Version:"v1", Resource:"endpoints"}
2020-11-02T14:27:15.5648004Z [cluster2] I1102 14:17:48.114862       1 resource_syncer.go:264] Syncer "Tunnel controller" retrieved created resource: &unstructured.Unstructured{Object:map[string]interface {}{"apiVersion":"submariner.io/v1", "kind":"Endpoint", "metadata":map[string]interface {}{"creationTimestamp":"2020-11-02T14:17:48Z", "generation":1, "name":"cluster2-submariner-cable-cluster2-172-17-0-9", "namespace":"submariner-operator", "resourceVersion":"1279", "selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster2-submariner-cable-cluster2-172-17-0-9", "uid":"1112af78-f959-40d1-b570-c351ce0c602a"}, "spec":map[string]interface {}{"backend":"strongswan", "cable_name":"submariner-cable-cluster2-172-17-0-9", "cluster_id":"cluster2", "hostname":"cluster2-worker", "nat_enabled":false, "private_ip":"172.17.0.9", "public_ip":"", "subnets":[]interface {}{"100.2.0.0/16", "10.2.0.0/16"}}}}
2020-11-02T14:27:15.5657518Z [cluster2] I1102 14:17:48.115245       1 tunnel.go:56] Tunnel controller processing added or updated submariner Endpoint object: &v1.Endpoint{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"cluster2-submariner-cable-cluster2-172-17-0-9", GenerateName:"", Namespace:"submariner-operator", SelfLink:"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster2-submariner-cable-cluster2-172-17-0-9", UID:"1112af78-f959-40d1-b570-c351ce0c602a", ResourceVersion:"1279", Generation:1, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63739923468, loc:(*time.Location)(0x2020aa0)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v1.EndpointSpec{ClusterID:"cluster2", CableName:"submariner-cable-cluster2-172-17-0-9", Hostname:"cluster2-worker", Subnets:[]string{"100.2.0.0/16", "10.2.0.0/16"}, PrivateIP:"172.17.0.9", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
2020-11-02T14:27:15.5662774Z [cluster2] I1102 14:17:48.115341       1 cableengine.go:92] Not installing cable for local cluster
2020-11-02T14:27:15.5664111Z [cluster2] I1102 14:17:48.115381       1 tunnel.go:68] Tunnel controller successfully installed Endpoint cable submariner-cable-cluster2-172-17-0-9 in the engine
2020-11-02T14:27:15.5665985Z [cluster2] I1102 14:17:48.115396       1 resource_syncer.go:352] Syncer "Tunnel controller": transform function returned nil - not syncing - requeue: false
2020-11-02T14:27:15.5670595Z [cluster2] I1102 14:17:48.116864       1 create_or_update.go:32] Creating resource: &unstructured.Unstructured{Object:map[string]interface {}{"apiVersion":"submariner.io/v1", "kind":"Endpoint", "metadata":map[string]interface {}{"labels":map[string]interface {}{"submariner-io/clusterID":"cluster2"}, "name":"cluster2-submariner-cable-cluster2-172-17-0-9", "namespace":"submariner-k8s-broker"}, "spec":map[string]interface {}{"backend":"strongswan", "cable_name":"submariner-cable-cluster2-172-17-0-9", "cluster_id":"cluster2", "hostname":"cluster2-worker", "nat_enabled":false, "private_ip":"172.17.0.9", "public_ip":"", "subnets":[]interface {}{"100.2.0.0/16", "10.2.0.0/16"}}}}
2020-11-02T14:27:15.5715003Z [cluster2] I1102 14:17:48.120616       1 resource_syncer.go:285] Syncer "local -> broker for *v1.Endpoint" successfully synced "submariner-operator/cluster2-submariner-cable-cluster2-172-17-0-9"
2020-11-02T14:27:15.5720845Z [cluster2] I1102 14:17:48.125906       1 resource_syncer.go:264] Syncer "broker -> local for *v1.Endpoint" retrieved created resource: &unstructured.Unstructured{Object:map[string]interface {}{"apiVersion":"submariner.io/v1", "kind":"Endpoint", "metadata":map[string]interface {}{"creationTimestamp":"2020-11-02T14:17:48Z", "generation":1, "labels":map[string]interface {}{"submariner-io/clusterID":"cluster2"}, "name":"cluster2-submariner-cable-cluster2-172-17-0-9", "namespace":"submariner-k8s-broker", "resourceVersion":"1323", "selfLink":"/apis/submariner.io/v1/namespaces/submariner-k8s-broker/endpoints/cluster2-submariner-cable-cluster2-172-17-0-9", "uid":"c683ad51-5f24-4408-857a-1f081edbeeb5"}, "spec":map[string]interface {}{"backend":"strongswan", "cable_name":"submariner-cable-cluster2-172-17-0-9", "cluster_id":"cluster2", "hostname":"cluster2-worker", "nat_enabled":false, "private_ip":"172.17.0.9", "public_ip":"", "subnets":[]interface {}{"100.2.0.0/16", "10.2.0.0/16"}}}}
2020-11-02T14:27:15.5726307Z [cluster2] I1102 14:17:48.132559       1 resource_syncer.go:443] Cluster ID label "cluster2" not present or matches local cluster ID "cluster2" - not syncing resource "cluster2-submariner-cable-cluster2-172-17-0-9"
2020-11-02T14:27:15.5731846Z [cluster2] I1102 14:17:49.440191       1 resource_syncer.go:264] Syncer "broker -> local for *v1.Cluster" retrieved created resource: &unstructured.Unstructured{Object:map[string]interface {}{"apiVersion":"submariner.io/v1", "kind":"Cluster", "metadata":map[string]interface {}{"creationTimestamp":"2020-11-02T14:17:49Z", "generation":1, "labels":map[string]interface {}{"submariner-io/clusterID":"cluster1"}, "name":"cluster1", "namespace":"submariner-k8s-broker", "resourceVersion":"1335", "selfLink":"/apis/submariner.io/v1/namespaces/submariner-k8s-broker/clusters/cluster1", "uid":"3ec30dab-da2c-4e45-85a4-f8b7e55f322d"}, "spec":map[string]interface {}{"cluster_cidr":[]interface {}{"10.1.0.0/16"}, "cluster_id":"cluster1", "color_codes":[]interface {}{"blue"}, "global_cidr":[]interface {}{}, "service_cidr":[]interface {}{"100.1.0.0/16"}}}}
2020-11-02T14:27:15.5738464Z [cluster2] I1102 14:17:49.440267       1 resource_syncer.go:276] Syncing resource: &unstructured.Unstructured{Object:map[string]interface {}{"apiVersion":"submariner.io/v1", "kind":"Cluster", "metadata":map[string]interface {}{"creationTimestamp":"2020-11-02T14:17:49Z", "generation":1, "labels":map[string]interface {}{"submariner-io/clusterID":"cluster1"}, "name":"cluster1", "namespace":"submariner-k8s-broker", "resourceVersion":"1335", "selfLink":"/apis/submariner.io/v1/namespaces/submariner-k8s-broker/clusters/cluster1", "uid":"3ec30dab-da2c-4e45-85a4-f8b7e55f322d"}, "spec":map[string]interface {}{"cluster_cidr":[]interface {}{"10.1.0.0/16"}, "cluster_id":"cluster1", "color_codes":[]interface {}{"blue"}, "global_cidr":[]interface {}{}, "service_cidr":[]interface {}{"100.1.0.0/16"}}}}
2020-11-02T14:27:15.5745387Z [cluster2] I1102 14:17:49.440325       1 federator.go:42] In Distribute for &unstructured.Unstructured{Object:map[string]interface {}{"apiVersion":"submariner.io/v1", "kind":"Cluster", "metadata":map[string]interface {}{"creationTimestamp":"2020-11-02T14:17:49Z", "generation":1, "labels":map[string]interface {}{"submariner-io/clusterID":"cluster1"}, "name":"cluster1", "namespace":"submariner-k8s-broker", "resourceVersion":"1335", "selfLink":"/apis/submariner.io/v1/namespaces/submariner-k8s-broker/clusters/cluster1", "uid":"3ec30dab-da2c-4e45-85a4-f8b7e55f322d"}, "spec":map[string]interface {}{"cluster_cidr":[]interface {}{"10.1.0.0/16"}, "cluster_id":"cluster1", "color_codes":[]interface {}{"blue"}, "global_cidr":[]interface {}{}, "service_cidr":[]interface {}{"100.1.0.0/16"}}}}
2020-11-02T14:27:15.5749168Z [cluster2] I1102 14:17:49.440399       1 helpers.go:75] Found schema.GroupVersionResource{Group:"submariner.io", Version:"v1", Resource:"clusters"}
2020-11-02T14:27:15.5753105Z [cluster2] I1102 14:17:49.450060       1 create_or_update.go:32] Creating resource: &unstructured.Unstructured{Object:map[string]interface {}{"apiVersion":"submariner.io/v1", "kind":"Cluster", "metadata":map[string]interface {}{"labels":map[string]interface {}{"submariner-io/clusterID":"cluster1"}, "name":"cluster1", "namespace":"submariner-operator"}, "spec":map[string]interface {}{"cluster_cidr":[]interface {}{"10.1.0.0/16"}, "cluster_id":"cluster1", "color_codes":[]interface {}{"blue"}, "global_cidr":[]interface {}{}, "service_cidr":[]interface {}{"100.1.0.0/16"}}}}
2020-11-02T14:27:15.5755940Z [cluster2] I1102 14:17:49.454865       1 resource_syncer.go:285] Syncer "broker -> local for *v1.Cluster" successfully synced "submariner-k8s-broker/cluster1"
2020-11-02T14:27:15.5760235Z [cluster2] I1102 14:17:49.461220       1 resource_syncer.go:264] Syncer "local -> broker for *v1.Cluster" retrieved created resource: &unstructured.Unstructured{Object:map[string]interface {}{"apiVersion":"submariner.io/v1", "kind":"Cluster", "metadata":map[string]interface {}{"creationTimestamp":"2020-11-02T14:17:49Z", "generation":1, "labels":map[string]interface {}{"submariner-io/clusterID":"cluster1"}, "name":"cluster1", "namespace":"submariner-operator", "resourceVersion":"1289", "selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/clusters/cluster1", "uid":"0640e236-721b-4486-b2ed-9505817e37c5"}, "spec":map[string]interface {}{"cluster_cidr":[]interface {}{"10.1.0.0/16"}, "cluster_id":"cluster1", "color_codes":[]interface {}{"blue"}, "global_cidr":[]interface {}{}, "service_cidr":[]interface {}{"100.1.0.0/16"}}}}
2020-11-02T14:27:15.5763834Z [cluster2] I1102 14:17:49.463904       1 resource_syncer.go:437] Found cluster ID label "cluster1" - not syncing resource "cluster1"
2020-11-02T14:27:15.5764674Z [cluster2] 16[NET] received packet: from 172.17.0.5[500] to 172.17.0.9[500] (500 bytes)
2020-11-02T14:27:15.5765302Z [cluster2] 16[ENC] parsed IKE_SA_INIT request 0 [ SA KE No N(NATD_S_IP) N(NATD_D_IP) N(FRAG_SUP) N(HASH_ALG) N(REDIR_SUP) ]
2020-11-02T14:27:15.5766353Z [cluster2] 16[IKE] no IKE config found for 172.17.0.9...172.17.0.5, sending NO_PROPOSAL_CHOSEN
2020-11-02T14:27:15.5766945Z [cluster2] 16[ENC] generating IKE_SA_INIT response 0 [ N(NO_PROP) ]
2020-11-02T14:27:15.5767620Z [cluster2] 16[NET] sending packet: from 172.17.0.9[500] to 172.17.0.5[500] (36 bytes)
2020-11-02T14:27:15.5773410Z [cluster2] I1102 14:17:49.568301       1 resource_syncer.go:264] Syncer "broker -> local for *v1.Endpoint" retrieved created resource: &unstructured.Unstructured{Object:map[string]interface {}{"apiVersion":"submariner.io/v1", "kind":"Endpoint", "metadata":map[string]interface {}{"creationTimestamp":"2020-11-02T14:17:49Z", "generation":1, "labels":map[string]interface {}{"submariner-io/clusterID":"cluster1"}, "name":"cluster1-submariner-cable-cluster1-172-17-0-5", "namespace":"submariner-k8s-broker", "resourceVersion":"1337", "selfLink":"/apis/submariner.io/v1/namespaces/submariner-k8s-broker/endpoints/cluster1-submariner-cable-cluster1-172-17-0-5", "uid":"29dd0c54-3258-411c-a553-9ac43bfda669"}, "spec":map[string]interface {}{"backend":"strongswan", "cable_name":"submariner-cable-cluster1-172-17-0-5", "cluster_id":"cluster1", "hostname":"cluster1-worker", "nat_enabled":false, "private_ip":"172.17.0.5", "public_ip":"", "subnets":[]interface {}{"100.1.0.0/16", "10.1.0.0/16"}}}}
2020-11-02T14:27:15.5782927Z [cluster2] I1102 14:17:49.568522       1 resource_syncer.go:276] Syncing resource: &unstructured.Unstructured{Object:map[string]interface {}{"apiVersion":"submariner.io/v1", "kind":"Endpoint", "metadata":map[string]interface {}{"creationTimestamp":"2020-11-02T14:17:49Z", "generation":1, "labels":map[string]interface {}{"submariner-io/clusterID":"cluster1"}, "name":"cluster1-submariner-cable-cluster1-172-17-0-5", "namespace":"submariner-k8s-broker", "resourceVersion":"1337", "selfLink":"/apis/submariner.io/v1/namespaces/submariner-k8s-broker/endpoints/cluster1-submariner-cable-cluster1-172-17-0-5", "uid":"29dd0c54-3258-411c-a553-9ac43bfda669"}, "spec":map[string]interface {}{"backend":"strongswan", "cable_name":"submariner-cable-cluster1-172-17-0-5", "cluster_id":"cluster1", "hostname":"cluster1-worker", "nat_enabled":false, "private_ip":"172.17.0.5", "public_ip":"", "subnets":[]interface {}{"100.1.0.0/16", "10.1.0.0/16"}}}}
2020-11-02T14:27:15.5793026Z [cluster2] I1102 14:17:49.568617       1 federator.go:42] In Distribute for &unstructured.Unstructured{Object:map[string]interface {}{"apiVersion":"submariner.io/v1", "kind":"Endpoint", "metadata":map[string]interface {}{"creationTimestamp":"2020-11-02T14:17:49Z", "generation":1, "labels":map[string]interface {}{"submariner-io/clusterID":"cluster1"}, "name":"cluster1-submariner-cable-cluster1-172-17-0-5", "namespace":"submariner-k8s-broker", "resourceVersion":"1337", "selfLink":"/apis/submariner.io/v1/namespaces/submariner-k8s-broker/endpoints/cluster1-submariner-cable-cluster1-172-17-0-5", "uid":"29dd0c54-3258-411c-a553-9ac43bfda669"}, "spec":map[string]interface {}{"backend":"strongswan", "cable_name":"submariner-cable-cluster1-172-17-0-5", "cluster_id":"cluster1", "hostname":"cluster1-worker", "nat_enabled":false, "private_ip":"172.17.0.5", "public_ip":"", "subnets":[]interface {}{"100.1.0.0/16", "10.1.0.0/16"}}}}
2020-11-02T14:27:15.5797784Z [cluster2] I1102 14:17:49.568790       1 helpers.go:75] Found schema.GroupVersionResource{Group:"submariner.io", Version:"v1", Resource:"endpoints"}
2020-11-02T14:27:15.5802223Z [cluster2] I1102 14:17:49.593239       1 create_or_update.go:32] Creating resource: &unstructured.Unstructured{Object:map[string]interface {}{"apiVersion":"submariner.io/v1", "kind":"Endpoint", "metadata":map[string]interface {}{"labels":map[string]interface {}{"submariner-io/clusterID":"cluster1"}, "name":"cluster1-submariner-cable-cluster1-172-17-0-5", "namespace":"submariner-operator"}, "spec":map[string]interface {}{"backend":"strongswan", "cable_name":"submariner-cable-cluster1-172-17-0-5", "cluster_id":"cluster1", "hostname":"cluster1-worker", "nat_enabled":false, "private_ip":"172.17.0.5", "public_ip":"", "subnets":[]interface {}{"100.1.0.0/16", "10.1.0.0/16"}}}}
2020-11-02T14:27:15.5807578Z [cluster2] I1102 14:17:49.616575       1 resource_syncer.go:285] Syncer "broker -> local for *v1.Endpoint" successfully synced "submariner-k8s-broker/cluster1-submariner-cable-cluster1-172-17-0-5"
2020-11-02T14:27:15.5813744Z [cluster2] I1102 14:17:49.621564       1 resource_syncer.go:264] Syncer "Tunnel controller" retrieved created resource: &unstructured.Unstructured{Object:map[string]interface {}{"apiVersion":"submariner.io/v1", "kind":"Endpoint", "metadata":map[string]interface {}{"creationTimestamp":"2020-11-02T14:17:49Z", "generation":1, "labels":map[string]interface {}{"submariner-io/clusterID":"cluster1"}, "name":"cluster1-submariner-cable-cluster1-172-17-0-5", "namespace":"submariner-operator", "resourceVersion":"1295", "selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster1-submariner-cable-cluster1-172-17-0-5", "uid":"ece540ec-fe35-4d78-a530-da0e585138dc"}, "spec":map[string]interface {}{"backend":"strongswan", "cable_name":"submariner-cable-cluster1-172-17-0-5", "cluster_id":"cluster1", "hostname":"cluster1-worker", "nat_enabled":false, "private_ip":"172.17.0.5", "public_ip":"", "subnets":[]interface {}{"100.1.0.0/16", "10.1.0.0/16"}}}}
2020-11-02T14:27:15.5822750Z [cluster2] I1102 14:17:49.621689       1 resource_syncer.go:264] Syncer "local -> broker for *v1.Endpoint" retrieved created resource: &unstructured.Unstructured{Object:map[string]interface {}{"apiVersion":"submariner.io/v1", "kind":"Endpoint", "metadata":map[string]interface {}{"creationTimestamp":"2020-11-02T14:17:49Z", "generation":1, "labels":map[string]interface {}{"submariner-io/clusterID":"cluster1"}, "name":"cluster1-submariner-cable-cluster1-172-17-0-5", "namespace":"submariner-operator", "resourceVersion":"1295", "selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster1-submariner-cable-cluster1-172-17-0-5", "uid":"ece540ec-fe35-4d78-a530-da0e585138dc"}, "spec":map[string]interface {}{"backend":"strongswan", "cable_name":"submariner-cable-cluster1-172-17-0-5", "cluster_id":"cluster1", "hostname":"cluster1-worker", "nat_enabled":false, "private_ip":"172.17.0.5", "public_ip":"", "subnets":[]interface {}{"100.1.0.0/16", "10.1.0.0/16"}}}}
2020-11-02T14:27:15.5828576Z [cluster2] I1102 14:17:49.621877       1 resource_syncer.go:437] Found cluster ID label "cluster1" - not syncing resource "cluster1-submariner-cable-cluster1-172-17-0-5"
2020-11-02T14:27:15.5840826Z [cluster2] I1102 14:17:49.622072       1 tunnel.go:56] Tunnel controller processing added or updated submariner Endpoint object: &v1.Endpoint{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"cluster1-submariner-cable-cluster1-172-17-0-5", GenerateName:"", Namespace:"submariner-operator", SelfLink:"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster1-submariner-cable-cluster1-172-17-0-5", UID:"ece540ec-fe35-4d78-a530-da0e585138dc", ResourceVersion:"1295", Generation:1, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63739923469, loc:(*time.Location)(0x2020aa0)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{"submariner-io/clusterID":"cluster1"}, Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v1.EndpointSpec{ClusterID:"cluster1", CableName:"submariner-cable-cluster1-172-17-0-5", Hostname:"cluster1-worker", Subnets:[]string{"100.1.0.0/16", "10.1.0.0/16"}, PrivateIP:"172.17.0.5", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
2020-11-02T14:27:15.5848037Z [cluster2] I1102 14:17:49.622176       1 cableengine.go:101] Installing Endpoint cable "submariner-cable-cluster1-172-17-0-5"
2020-11-02T14:27:15.5849290Z [cluster2] 07[CFG] loaded IKE shared key for: '172.17.0.5'
2020-11-02T14:27:15.5851796Z [cluster2] 11[CFG] added vici connection: submariner-cable-cluster1-172-17-0-5
2020-11-02T14:27:15.5853786Z [cluster2] 11[CFG] initiating 'submariner-child-submariner-cable-cluster1-172-17-0-5'
2020-11-02T14:27:15.5855474Z [cluster2] 11[IKE] initiating IKE_SA submariner-cable-cluster1-172-17-0-5[2] to 172.17.0.5
2020-11-02T14:27:15.5856444Z [cluster2] 11[ENC] generating IKE_SA_INIT request 0 [ SA KE No N(NATD_S_IP) N(NATD_D_IP) N(FRAG_SUP) N(HASH_ALG) N(REDIR_SUP) ]
2020-11-02T14:27:15.5857106Z [cluster2] 11[NET] sending packet: from 172.17.0.9[500] to 172.17.0.5[500] (500 bytes)
2020-11-02T14:27:15.5858554Z [cluster2] I1102 14:17:49.664462       1 cableengine.go:150] Successfully installed Endpoint cable "submariner-cable-cluster1-172-17-0-5" with remote IP 172.17.0.5
2020-11-02T14:27:15.5860324Z [cluster2] I1102 14:17:49.664602       1 tunnel.go:68] Tunnel controller successfully installed Endpoint cable submariner-cable-cluster1-172-17-0-5 in the engine
2020-11-02T14:27:15.5861337Z [cluster2] 15[NET] received packet: from 172.17.0.5[500] to 172.17.0.9[500] (464 bytes)
2020-11-02T14:27:15.5862655Z [cluster2] I1102 14:17:49.664618       1 resource_syncer.go:352] Syncer "Tunnel controller": transform function returned nil - not syncing - requeue: false
2020-11-02T14:27:15.5863579Z [cluster2] 15[ENC] parsed IKE_SA_INIT response 0 [ SA KE No N(NATD_S_IP) N(NATD_D_IP) N(FRAG_SUP) N(HASH_ALG) N(CHDLESS_SUP) N(MULT_AUTH) ]
2020-11-02T14:27:15.5864485Z [cluster2] 15[CFG] selected proposal: IKE:AES_GCM_16_128/PRF_HMAC_SHA2_256/MODP_2048
2020-11-02T14:27:15.5865162Z [cluster2] 15[IKE] remote host is behind NAT
2020-11-02T14:27:15.5865889Z [cluster2] 15[IKE] authentication of '172.17.0.9' (myself) with pre-shared key
2020-11-02T14:27:15.5867312Z [cluster2] 15[IKE] establishing CHILD_SA submariner-child-submariner-cable-cluster1-172-17-0-5{1}
2020-11-02T14:27:15.5868657Z [cluster2] 15[ENC] generating IKE_AUTH request 1 [ IDi N(INIT_CONTACT) IDr AUTH SA TSi TSr N(MULT_AUTH) N(EAP_ONLY) N(MSG_ID_SYN_SUP) ]
2020-11-02T14:27:15.5869481Z [cluster2] 15[NET] sending packet: from 172.17.0.9[4500] to 172.17.0.5[4500] (333 bytes)
2020-11-02T14:27:15.8562181Z [lighthouse]$ [cluster2] kubectl -n submariner-operator logs submariner-routeagent-4w68n
2020-11-02T14:27:15.8574445Z [lighthouse]$ [cluster2] kubectl -n submariner-operator logs submariner-routeagent-4w68n
2020-11-02T14:27:15.8587171Z [lighthouse]$ [cluster2] command kubectl --context=cluster2 -n submariner-operator logs submariner-routeagent-4w68n
2020-11-02T14:27:15.8600943Z [lighthouse]$ [cluster2] kubectl --context=cluster2 -n submariner-operator logs submariner-routeagent-4w68n
2020-11-02T14:27:16.1620246Z [cluster2] 14[NET] received packet: from 172.17.0.5[4500] to 172.17.0.9[4500] (257 bytes)
2020-11-02T14:27:16.1620864Z [cluster2] 14[ENC] parsed IKE_AUTH response 1 [ IDr AUTH SA TSi TSr ]
2020-11-02T14:27:16.1622134Z [cluster2] 14[IKE] authentication of '172.17.0.5' with pre-shared key successful
2020-11-02T14:27:16.1623354Z [cluster2] 14[IKE] IKE_SA submariner-cable-cluster1-172-17-0-5[2] established between 172.17.0.9[172.17.0.9]...172.17.0.5[172.17.0.5]
2020-11-02T14:27:16.1624398Z [cluster2] 14[IKE] scheduling rekeying in 13606s
2020-11-02T14:27:16.1624899Z [cluster2] 14[IKE] maximum IKE_SA lifetime 15046s
2020-11-02T14:27:16.1625404Z [cluster2] 14[CFG] selected proposal: ESP:AES_GCM_16_128/NO_EXT_SEQ
2020-11-02T14:27:16.1627091Z [cluster2] 14[IKE] CHILD_SA submariner-child-submariner-cable-cluster1-172-17-0-5{1} established with SPIs c65121a0_i c62a309f_o and TS 10.2.0.0/16 100.2.0.0/16 172.17.0.9/32 === 10.1.0.0/16 100.1.0.0/16 172.17.0.5/32
2020-11-02T14:27:16.1628599Z [cluster2] 13[KNL] interface vx-submariner activated
2020-11-02T14:27:16.1629313Z [cluster2] 05[KNL] 240.17.0.9 appeared on vx-submariner
2020-11-02T14:27:16.1629872Z [cluster2] 12[KNL] interface vethwepl0d2e11f deactivated
2020-11-02T14:27:16.1630477Z [cluster2] 09[KNL] interface vethwepl0d2e11f deleted
2020-11-02T14:27:16.1631061Z [cluster2] 09[KNL] interface vethwepg34eb985 deleted
2020-11-02T14:27:16.1632242Z [cluster2] 11[KNL] interface vethwepl34eb985 activated
2020-11-02T14:27:16.1632843Z [cluster2] 12[KNL] interface vethwepgebbac63 deleted
2020-11-02T14:27:16.1633453Z [cluster2] 06[KNL] interface vethweplebbac63 activated
2020-11-02T14:27:16.1634078Z [cluster2] 11[KNL] interface vethwepl34eb985 deactivated
2020-11-02T14:27:16.1634676Z [cluster2] 16[KNL] interface vethwepl34eb985 deleted
2020-11-02T14:27:16.1635286Z [cluster2] 15[KNL] interface vethweplebbac63 deactivated
2020-11-02T14:27:16.1637645Z [cluster2] 14[KNL] interface vethweplebbac63 deleted
2020-11-02T14:27:16.1638262Z [cluster2] 08[KNL] interface vethwepg52eb18e deleted
2020-11-02T14:27:16.1638855Z [cluster2] 11[KNL] interface vethwepl52eb18e activated
2020-11-02T14:27:16.1639470Z [cluster2] 09[KNL] interface vethwepl52eb18e deactivated
2020-11-02T14:27:16.1640074Z [cluster2] 07[KNL] interface vethwepl52eb18e deleted
2020-11-02T14:27:16.1640648Z [cluster2] 07[KNL] interface vethwepgf890302 deleted
2020-11-02T14:27:16.1641233Z [cluster2] 08[KNL] interface vethweplf890302 activated
2020-11-02T14:27:16.1644478Z [cluster2] 12[KNL] interface vethweplf890302 deactivated
2020-11-02T14:27:16.1645059Z [cluster2] 11[KNL] interface vethweplf890302 deleted
2020-11-02T14:27:16.1645784Z [cluster2] 16[KNL] interface vethwepgad638fc deleted
2020-11-02T14:27:16.1646386Z [cluster2] 08[KNL] interface vethweplad638fc activated
2020-11-02T14:27:16.1649117Z [cluster2] 12[KNL] interface vethweplad638fc deactivated
2020-11-02T14:27:16.1649728Z [cluster2] 08[KNL] interface vethweplad638fc deleted
2020-11-02T14:27:16.1650309Z [cluster2] 12[KNL] interface vethwepg4fd6e6a deleted
2020-11-02T14:27:16.1650903Z [cluster2] 09[KNL] interface vethwepl4fd6e6a activated
2020-11-02T14:27:16.1651491Z [cluster2] 09[KNL] interface vethwepg56a4284 deleted
2020-11-02T14:27:16.1652055Z [cluster2] 12[KNL] interface vethwepl56a4284 activated
2020-11-02T14:27:16.1654585Z [cluster2] 07[KNL] interface vethwepg8aaabf0 deleted
2020-11-02T14:27:16.1655205Z [cluster2] 13[KNL] interface vethwepl8aaabf0 activated
2020-11-02T14:27:16.1656000Z [cluster2] 09[KNL] interface vethwepl4fd6e6a deactivated
2020-11-02T14:27:16.1656748Z [cluster2] 15[KNL] interface vethwepl4fd6e6a deleted
2020-11-02T14:27:16.1657375Z [cluster2] 11[KNL] interface vethwepl8aaabf0 deactivated
2020-11-02T14:27:16.1659969Z [cluster2] 14[KNL] interface vethwepl8aaabf0 deleted
2020-11-02T14:27:16.1660588Z [cluster2] 07[KNL] interface vethwepl56a4284 deactivated
2020-11-02T14:27:16.1661187Z [cluster2] 15[KNL] interface vethwepl56a4284 deleted
2020-11-02T14:27:16.1661740Z [cluster2] 09[KNL] interface vethwepgb001169 deleted
2020-11-02T14:27:16.1662319Z [cluster2] 16[KNL] interface vethweplb001169 activated
2020-11-02T14:27:16.1662896Z [cluster2] 12[KNL] interface vethwepg8812880 deleted
2020-11-02T14:27:16.1663466Z [cluster2] 16[KNL] interface vethwepl8812880 activated
2020-11-02T14:27:16.1664066Z [cluster2] 11[KNL] interface vethweplb001169 deactivated
2020-11-02T14:27:16.1664658Z [cluster2] 13[KNL] interface vethweplb001169 deleted
2020-11-02T14:27:16.1665290Z [cluster2] 09[KNL] interface vethwepl8812880 deactivated
2020-11-02T14:27:16.1665872Z [cluster2] 16[KNL] interface vethwepl8812880 deleted
2020-11-02T14:27:16.1666426Z [cluster2] 09[KNL] interface vethwepg971ce4b deleted
2020-11-02T14:27:16.1667007Z [cluster2] 13[KNL] interface vethwepl971ce4b activated
2020-11-02T14:27:16.1667598Z [cluster2] 07[KNL] interface vethwepgdbd24ca deleted
2020-11-02T14:27:16.1668196Z [cluster2] 06[KNL] interface vethwepldbd24ca activated
2020-11-02T14:27:16.1668820Z [cluster2] 15[KNL] interface vethwepldbd24ca deactivated
2020-11-02T14:27:16.1669429Z [cluster2] 08[KNL] interface vethwepldbd24ca deleted
2020-11-02T14:27:16.1670033Z [cluster2] 10[KNL] interface vethwepl971ce4b deactivated
2020-11-02T14:27:16.1670626Z [cluster2] 13[KNL] interface vethwepl971ce4b deleted
2020-11-02T14:27:16.1671060Z [cluster2] ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
2020-11-02T14:27:16.1672373Z [cluster2] +++++++++++++++++++++: Logs for Pod submariner-routeagent-4w68n in namespace submariner-operator :++++++++++++++++++++++
2020-11-02T14:27:16.1673359Z [cluster2] + trap 'exit 1' SIGTERM SIGINT
2020-11-02T14:27:16.1673806Z [cluster2] + SUBMARINER_VERBOSITY=2
2020-11-02T14:27:16.1674362Z [cluster2] + '[' false == true ']'
2020-11-02T14:27:16.1674836Z [cluster2] + DEBUG=-v=2
2020-11-02T14:27:16.1676411Z [cluster2] + for f in iptables-save iptables
2020-11-02T14:27:16.1677090Z [cluster2] ++ find_iptables_on_host iptables-save
2020-11-02T14:27:16.1677798Z [cluster2] ++ chroot /host test -x /usr/sbin/iptables-save
2020-11-02T14:27:16.1678247Z [cluster2] ++ echo /usr/sbin
2020-11-02T14:27:16.1678560Z [cluster2] ++ return
2020-11-02T14:27:16.1678907Z [cluster2] + location=/usr/sbin
2020-11-02T14:27:16.1679454Z [cluster2] + '[' /usr/sbin '!=' unknown ']'
2020-11-02T14:27:16.1680208Z [cluster2] + echo 'iptables-save is present on the host at /usr/sbin/iptables-save'
2020-11-02T14:27:16.1681100Z [cluster2] + sed 's!@@PATH@@!/usr/sbin!' /usr/sbin/iptables-wrapper.in
2020-11-02T14:27:16.1682197Z [cluster2] iptables-save is present on the host at /usr/sbin/iptables-save
2020-11-02T14:27:16.1682960Z [cluster2] + for f in iptables-save iptables
2020-11-02T14:27:16.1683542Z [cluster2] ++ find_iptables_on_host iptables
2020-11-02T14:27:16.1684204Z [cluster2] ++ chroot /host test -x /usr/sbin/iptables
2020-11-02T14:27:16.1684620Z [cluster2] ++ echo /usr/sbin
2020-11-02T14:27:16.1684947Z [cluster2] ++ return
2020-11-02T14:27:16.1685294Z [cluster2] + location=/usr/sbin
2020-11-02T14:27:16.1685844Z [cluster2] + '[' /usr/sbin '!=' unknown ']'
2020-11-02T14:27:16.1686524Z [cluster2] + echo 'iptables is present on the host at /usr/sbin/iptables'
2020-11-02T14:27:16.1687336Z [cluster2] + sed 's!@@PATH@@!/usr/sbin!' /usr/sbin/iptables-wrapper.in
2020-11-02T14:27:16.1687953Z [cluster2] iptables is present on the host at /usr/sbin/iptables
2020-11-02T14:27:16.1688787Z [cluster2] + exec submariner-route-agent -v=2 -alsologtostderr
2020-11-02T14:27:16.1689740Z [cluster2] I1102 14:17:35.538306       1 main.go:47] Starting submariner-route-agent
2020-11-02T14:27:16.1690945Z [cluster2] W1102 14:17:35.545626       1 client_config.go:543] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
2020-11-02T14:27:16.1692093Z [cluster2] I1102 14:17:35.654523       1 route.go:192] Starting Route Controller. ClusterID: cluster2, localClusterCIDR: [10.2.0.0/16], localServiceCIDR: [100.2.0.0/16]
2020-11-02T14:27:16.1693017Z [cluster2] I1102 14:17:35.654554       1 route.go:196] Waiting for Endpoint informer caches to sync.
2020-11-02T14:27:16.1693955Z [cluster2] I1102 14:17:35.706096       1 route.go:656] Enqueueing sm-route-agent-pod event, ip: 172.17.0.9
2020-11-02T14:27:16.1694919Z [cluster2] I1102 14:17:35.706131       1 route.go:656] Enqueueing sm-route-agent-pod event, ip: 172.17.0.8
2020-11-02T14:27:16.1695570Z [cluster2] I1102 14:17:35.755072       1 driver.go:37] Interface "lo" has "127.0.0.1" address
2020-11-02T14:27:16.1696147Z [cluster2] I1102 14:17:35.755273       1 driver.go:37] Interface "weave" has "10.2.128.0" address
2020-11-02T14:27:16.1696828Z [cluster2] I1102 14:17:35.755283       1 driver.go:42] Found CNI Interface "weave" that has IP "10.2.128.0" from ClusterCIDR "10.2.0.0/16"
2020-11-02T14:27:16.1697670Z [cluster2] I1102 14:17:35.755413       1 driver.go:60] Successfully configured rp_filter to loose mode(2) on cniInterface "weave"
2020-11-02T14:27:16.1729093Z [cluster2] I1102 14:17:35.764260       1 iptables.go:22] Install/ensure SUBMARINER-POSTROUTING chain exists
2020-11-02T14:27:16.1730562Z [cluster2] I1102 14:17:36.112888       1 iptables.go:28] Insert SUBMARINER-POSTROUTING rule that has rules for inter-cluster traffic
2020-11-02T14:27:16.1731721Z [cluster2] I1102 14:17:36.136936       1 iptables.go:34] Install/ensure SUBMARINER-INPUT chain exists
2020-11-02T14:27:16.1732762Z [cluster2] I1102 14:17:36.246161       1 iptables.go:45] Allow VxLAN incoming traffic in SUBMARINER-INPUT Chain
2020-11-02T14:27:16.1733928Z [cluster2] I1102 14:17:36.300095       1 iptables.go:53] Insert rule to allow traffic over vx-submariner interface in FORWARDing Chain
2020-11-02T14:27:16.1735278Z [cluster2] I1102 14:17:36.335737       1 iptables.go:65] Installing rule for host network to remote cluster communication: -s 240.0.0.0/8 -o vx-submariner -j SNAT --to 10.2.128.0
2020-11-02T14:27:16.1736483Z [cluster2] I1102 14:17:36.404908       1 route.go:254] In cluster2, podIP of submariner-route-agent[0] is 172.17.0.8
2020-11-02T14:27:16.1737567Z [cluster2] I1102 14:17:36.404934       1 route.go:254] In cluster2, podIP of submariner-route-agent[1] is 172.17.0.9
2020-11-02T14:27:16.1738788Z [cluster2] I1102 14:17:36.404940       1 route.go:268] Hostname is "cluster2-worker2" and routeAgentNodeName is "cluster2-worker2"
2020-11-02T14:27:16.1740045Z [cluster2] I1102 14:17:36.424892       1 driver.go:87] Successfully annotated node "cluster2-worker2" with cniIfaceIP "10.2.128.0"
2020-11-02T14:27:16.1740797Z [cluster2] I1102 14:17:36.424943       1 route.go:286] Route agent workers started
2020-11-02T14:27:16.1742380Z [cluster2] I1102 14:17:36.424972       1 route.go:507] Processing submariner-route-agent pod "submariner-routeagent-ct9kx" with host IP "172.17.0.9", pod IP "172.17.0.9"
2020-11-02T14:27:16.1744027Z [cluster2] I1102 14:17:36.424980       1 route.go:531] Successfully processed submariner-route-agent pod "submariner-routeagent-ct9kx"
2020-11-02T14:27:16.1745911Z [cluster2] I1102 14:17:36.425014       1 route.go:507] Processing submariner-route-agent pod "submariner-routeagent-4w68n" with host IP "172.17.0.8", pod IP "172.17.0.8"
2020-11-02T14:27:16.1750295Z [cluster2] I1102 14:17:36.425021       1 route.go:531] Successfully processed submariner-route-agent pod "submariner-routeagent-4w68n"
2020-11-02T14:27:16.1752501Z [cluster2] I1102 14:17:48.109760       1 route.go:570] Processing local Endpoint cluster2-submariner-cable-cluster2-172-17-0-9 with IP 172.17.0.9, Host cluster2-worker for this cluster
2020-11-02T14:27:16.1754094Z [cluster2] I1102 14:17:48.109786       1 route.go:612] This route agent is running on a non-gateway/non-active node
2020-11-02T14:27:16.1755143Z [cluster2] I1102 14:17:48.109798       1 route.go:623] Creating the vxlan interface vx-submariner with gateway node IP 172.17.0.9
2020-11-02T14:27:16.1755839Z [cluster2] I1102 14:17:48.127359       1 route.go:771] Reconciling routes to gw: 240.17.0.9
2020-11-02T14:27:16.1756493Z [cluster2] I1102 14:17:48.127516       1 route.go:787] Processing route {Ifindex: 15 Dst: 240.0.0.0/8 Src: 240.17.0.8 Gw: <nil> Flags: [] Table: 254}
2020-11-02T14:27:16.1757093Z [cluster2] I1102 14:17:48.127548       1 route.go:790] Found nil gw or dst
2020-11-02T14:27:16.1758438Z [cluster2] I1102 14:17:48.127621       1 route.go:647] Successfully processed local Endpoint "cluster2-submariner-cable-cluster2-172-17-0-9"
2020-11-02T14:27:16.1760522Z [cluster2] I1102 14:17:49.618055       1 route.go:558] Updating iptable rules for remote cluster's Endpoint "cluster1-submariner-cable-cluster1-172-17-0-5" with subnets [100.1.0.0/16 10.1.0.0/16]
2020-11-02T14:27:16.1762120Z [cluster2] I1102 14:17:49.657760       1 iptables.go:83] Installing iptables rule for outgoing traffic: -s 10.2.0.0/16 -d 100.1.0.0/16 -j ACCEPT
2020-11-02T14:27:16.1763205Z [cluster2] I1102 14:17:49.704061       1 iptables.go:91] Installing iptables rule for incoming traffic: -s 100.1.0.0/16 -d 10.2.0.0/16 -j ACCEPT
2020-11-02T14:27:16.1764280Z [cluster2] I1102 14:17:49.728398       1 iptables.go:83] Installing iptables rule for outgoing traffic: -s 10.2.0.0/16 -d 10.1.0.0/16 -j ACCEPT
2020-11-02T14:27:16.1765346Z [cluster2] I1102 14:17:49.747590       1 iptables.go:91] Installing iptables rule for incoming traffic: -s 10.1.0.0/16 -d 10.2.0.0/16 -j ACCEPT
2020-11-02T14:27:16.1766993Z [cluster2] I1102 14:18:05.669549       1 route.go:570] Processing local Endpoint cluster2-submariner-cable-cluster2-172-17-0-9 with IP 172.17.0.9, Host cluster2-worker for this cluster
2020-11-02T14:27:16.1768549Z [cluster2] I1102 14:18:05.669584       1 route.go:612] This route agent is running on a non-gateway/non-active node
2020-11-02T14:27:16.1769603Z [cluster2] I1102 14:18:05.669606       1 route.go:623] Creating the vxlan interface vx-submariner with gateway node IP 172.17.0.9
2020-11-02T14:27:16.1770410Z [cluster2] I1102 14:18:05.669897       1 vxlan.go:61] VxLAN interface already exists with same configuration.
2020-11-02T14:27:16.1771087Z [cluster2] I1102 14:18:05.676666       1 route.go:771] Reconciling routes to gw: 240.17.0.9
2020-11-02T14:27:16.1771717Z [cluster2] I1102 14:18:05.676823       1 route.go:787] Processing route {Ifindex: 15 Dst: 240.0.0.0/8 Src: 240.17.0.8 Gw: <nil> Flags: [] Table: 254}
2020-11-02T14:27:16.1772315Z [cluster2] I1102 14:18:05.676857       1 route.go:790] Found nil gw or dst
2020-11-02T14:27:16.1773662Z [cluster2] I1102 14:18:05.677025       1 route.go:647] Successfully processed local Endpoint "cluster2-submariner-cable-cluster2-172-17-0-9"
2020-11-02T14:27:16.1775726Z [cluster2] I1102 14:18:05.677053       1 route.go:558] Updating iptable rules for remote cluster's Endpoint "cluster1-submariner-cable-cluster1-172-17-0-5" with subnets [100.1.0.0/16 10.1.0.0/16]
2020-11-02T14:27:16.1779536Z [cluster2] I1102 14:18:35.669656       1 route.go:570] Processing local Endpoint cluster2-submariner-cable-cluster2-172-17-0-9 with IP 172.17.0.9, Host cluster2-worker for this cluster
2020-11-02T14:27:16.1781166Z [cluster2] I1102 14:18:35.669684       1 route.go:612] This route agent is running on a non-gateway/non-active node
2020-11-02T14:27:16.1782210Z [cluster2] I1102 14:18:35.669705       1 route.go:623] Creating the vxlan interface vx-submariner with gateway node IP 172.17.0.9
2020-11-02T14:27:16.1783020Z [cluster2] I1102 14:18:35.669906       1 vxlan.go:61] VxLAN interface already exists with same configuration.
2020-11-02T14:27:16.1783692Z [cluster2] I1102 14:18:35.674385       1 route.go:771] Reconciling routes to gw: 240.17.0.9
2020-11-02T14:27:16.1784346Z [cluster2] I1102 14:18:35.674544       1 route.go:787] Processing route {Ifindex: 15 Dst: 10.1.0.0/16 Src: <nil> Gw: 240.17.0.9 Flags: [] Table: 254}
2020-11-02T14:27:16.1785143Z [cluster2] I1102 14:18:35.674567       1 route.go:793] Found route {Ifindex: 15 Dst: 10.1.0.0/16 Src: <nil> Gw: 240.17.0.9 Flags: [] Table: 254} with gw 240.17.0.9 already installed
2020-11-02T14:27:16.1785931Z [cluster2] I1102 14:18:35.674577       1 route.go:787] Processing route {Ifindex: 15 Dst: 100.1.0.0/16 Src: <nil> Gw: 240.17.0.9 Flags: [] Table: 254}
2020-11-02T14:27:16.1786718Z [cluster2] I1102 14:18:35.674585       1 route.go:793] Found route {Ifindex: 15 Dst: 100.1.0.0/16 Src: <nil> Gw: 240.17.0.9 Flags: [] Table: 254} with gw 240.17.0.9 already installed
2020-11-02T14:27:16.1788153Z [lighthouse]$ [cluster2] kubectl -n submariner-operator logs submariner-routeagent-ct9kx
2020-11-02T14:27:16.1789108Z [cluster2] I1102 14:18:35.674593       1 route.go:787] Processing route {Ifindex: 15 Dst: 240.0.0.0/8 Src: 240.17.0.8 Gw: <nil> Flags: [] Table: 254}
2020-11-02T14:27:16.1789705Z [cluster2] I1102 14:18:35.674601       1 route.go:790] Found nil gw or dst
2020-11-02T14:27:16.1790246Z [cluster2] I1102 14:18:35.674656       1 route.go:829] Found equivalent route, not adding
2020-11-02T14:27:16.1790855Z [cluster2] I1102 14:18:35.674665       1 route.go:829] Found equivalent route, not adding
2020-11-02T14:27:16.1798328Z [cluster2] I1102 14:18:35.674694       1 route.go:647] Successfully processed local Endpoint "cluster2-submariner-cable-cluster2-172-17-0-9"
2020-11-02T14:27:16.1800488Z [cluster2] I1102 14:18:35.674711       1 route.go:558] Updating iptable rules for remote cluster's Endpoint "cluster1-submariner-cable-cluster1-172-17-0-5" with subnets [100.1.0.0/16 10.1.0.0/16]
2020-11-02T14:27:16.1802872Z [cluster2] I1102 14:19:05.669767       1 route.go:570] Processing local Endpoint cluster2-submariner-cable-cluster2-172-17-0-9 with IP 172.17.0.9, Host cluster2-worker for this cluster
2020-11-02T14:27:16.1804485Z [cluster2] I1102 14:19:05.669798       1 route.go:612] This route agent is running on a non-gateway/non-active node
2020-11-02T14:27:16.1805554Z [cluster2] I1102 14:19:05.669810       1 route.go:623] Creating the vxlan interface vx-submariner with gateway node IP 172.17.0.9
2020-11-02T14:27:16.1806373Z [cluster2] I1102 14:19:05.670051       1 vxlan.go:61] VxLAN interface already exists with same configuration.
2020-11-02T14:27:16.1807048Z [cluster2] I1102 14:19:05.676059       1 route.go:771] Reconciling routes to gw: 240.17.0.9
2020-11-02T14:27:16.1807686Z [cluster2] I1102 14:19:05.676211       1 route.go:787] Processing route {Ifindex: 15 Dst: 10.1.0.0/16 Src: <nil> Gw: 240.17.0.9 Flags: [] Table: 254}
2020-11-02T14:27:16.1808472Z [cluster2] I1102 14:19:05.676237       1 route.go:793] Found route {Ifindex: 15 Dst: 10.1.0.0/16 Src: <nil> Gw: 240.17.0.9 Flags: [] Table: 254} with gw 240.17.0.9 already installed
2020-11-02T14:27:16.1809263Z [cluster2] I1102 14:19:05.676246       1 route.go:787] Processing route {Ifindex: 15 Dst: 100.1.0.0/16 Src: <nil> Gw: 240.17.0.9 Flags: [] Table: 254}
2020-11-02T14:27:16.1810271Z [cluster2] I1102 14:19:05.676254       1 route.go:793] Found route {Ifindex: 15 Dst: 100.1.0.0/16 Src: <nil> Gw: 240.17.0.9 Flags: [] Table: 254} with gw 240.17.0.9 already installed
2020-11-02T14:27:16.1811130Z [cluster2] I1102 14:19:05.676261       1 route.go:787] Processing route {Ifindex: 15 Dst: 240.0.0.0/8 Src: 240.17.0.8 Gw: <nil> Flags: [] Table: 254}
2020-11-02T14:27:16.1811800Z [cluster2] I1102 14:19:05.676269       1 route.go:790] Found nil gw or dst
2020-11-02T14:27:16.1812334Z [cluster2] I1102 14:19:05.676328       1 route.go:829] Found equivalent route, not adding
2020-11-02T14:27:16.1812936Z [cluster2] I1102 14:19:05.676336       1 route.go:829] Found equivalent route, not adding
2020-11-02T14:27:16.1814457Z [cluster2] I1102 14:19:05.676372       1 route.go:647] Successfully processed local Endpoint "cluster2-submariner-cable-cluster2-172-17-0-9"
2020-11-02T14:27:16.1816535Z [cluster2] I1102 14:19:05.676387       1 route.go:558] Updating iptable rules for remote cluster's Endpoint "cluster1-submariner-cable-cluster1-172-17-0-5" with subnets [100.1.0.0/16 10.1.0.0/16]
2020-11-02T14:27:16.1819005Z [cluster2] I1102 14:19:35.670121       1 route.go:570] Processing local Endpoint cluster2-submariner-cable-cluster2-172-17-0-9 with IP 172.17.0.9, Host cluster2-worker for this cluster
2020-11-02T14:27:16.1820572Z [cluster2] I1102 14:19:35.670165       1 route.go:612] This route agent is running on a non-gateway/non-active node
2020-11-02T14:27:16.1821615Z [cluster2] I1102 14:19:35.670176       1 route.go:623] Creating the vxlan interface vx-submariner with gateway node IP 172.17.0.9
2020-11-02T14:27:16.1822425Z [cluster2] I1102 14:19:35.670378       1 vxlan.go:61] VxLAN interface already exists with same configuration.
2020-11-02T14:27:16.1823269Z [cluster2] I1102 14:19:35.674545       1 route.go:771] Reconciling routes to gw: 240.17.0.9
2020-11-02T14:27:16.1823923Z [cluster2] I1102 14:19:35.675222       1 route.go:787] Processing route {Ifindex: 15 Dst: 10.1.0.0/16 Src: <nil> Gw: 240.17.0.9 Flags: [] Table: 254}
2020-11-02T14:27:16.1824719Z [cluster2] I1102 14:19:35.675301       1 route.go:793] Found route {Ifindex: 15 Dst: 10.1.0.0/16 Src: <nil> Gw: 240.17.0.9 Flags: [] Table: 254} with gw 240.17.0.9 already installed
2020-11-02T14:27:16.1825509Z [cluster2] I1102 14:19:35.675320       1 route.go:787] Processing route {Ifindex: 15 Dst: 100.1.0.0/16 Src: <nil> Gw: 240.17.0.9 Flags: [] Table: 254}
2020-11-02T14:27:16.1826278Z [cluster2] I1102 14:19:35.675330       1 route.go:793] Found route {Ifindex: 15 Dst: 100.1.0.0/16 Src: <nil> Gw: 240.17.0.9 Flags: [] Table: 254} with gw 240.17.0.9 already installed
2020-11-02T14:27:16.1827059Z [cluster2] I1102 14:19:35.675354       1 route.go:787] Processing route {Ifindex: 15 Dst: 240.0.0.0/8 Src: 240.17.0.8 Gw: <nil> Flags: [] Table: 254}
2020-11-02T14:27:16.1827653Z [cluster2] I1102 14:19:35.675382       1 route.go:790] Found nil gw or dst
2020-11-02T14:27:16.1828190Z [cluster2] I1102 14:19:35.675493       1 route.go:829] Found equivalent route, not adding
2020-11-02T14:27:16.1828796Z [cluster2] I1102 14:19:35.675513       1 route.go:829] Found equivalent route, not adding
2020-11-02T14:27:16.1830257Z [cluster2] I1102 14:19:35.678088       1 route.go:647] Successfully processed local Endpoint "cluster2-submariner-cable-cluster2-172-17-0-9"
2020-11-02T14:27:16.1833218Z [cluster2] I1102 14:19:35.678139       1 route.go:558] Updating iptable rules for remote cluster's Endpoint "cluster1-submariner-cable-cluster1-172-17-0-5" with subnets [100.1.0.0/16 10.1.0.0/16]
2020-11-02T14:27:16.1835607Z [cluster2] I1102 14:20:05.670168       1 route.go:570] Processing local Endpoint cluster2-submariner-cable-cluster2-172-17-0-9 with IP 172.17.0.9, Host cluster2-worker for this cluster
2020-11-02T14:27:16.1837209Z [cluster2] I1102 14:20:05.670205       1 route.go:612] This route agent is running on a non-gateway/non-active node
2020-11-02T14:27:16.1838254Z [cluster2] I1102 14:20:05.670217       1 route.go:623] Creating the vxlan interface vx-submariner with gateway node IP 172.17.0.9
2020-11-02T14:27:16.1839348Z [cluster2] I1102 14:20:05.670428       1 vxlan.go:61] VxLAN interface already exists with same configuration.
2020-11-02T14:27:16.1840059Z [cluster2] I1102 14:20:05.673989       1 route.go:771] Reconciling routes to gw: 240.17.0.9
2020-11-02T14:27:16.1840690Z [cluster2] I1102 14:20:05.674178       1 route.go:787] Processing route {Ifindex: 15 Dst: 10.1.0.0/16 Src: <nil> Gw: 240.17.0.9 Flags: [] Table: 254}
2020-11-02T14:27:16.1841643Z [cluster2] I1102 14:20:05.674218       1 route.go:793] Found route {Ifindex: 15 Dst: 10.1.0.0/16 Src: <nil> Gw: 240.17.0.9 Flags: [] Table: 254} with gw 240.17.0.9 already installed
2020-11-02T14:27:16.1842425Z [cluster2] I1102 14:20:05.674229       1 route.go:787] Processing route {Ifindex: 15 Dst: 100.1.0.0/16 Src: <nil> Gw: 240.17.0.9 Flags: [] Table: 254}
2020-11-02T14:27:16.1843214Z [cluster2] I1102 14:20:05.674258       1 route.go:793] Found route {Ifindex: 15 Dst: 100.1.0.0/16 Src: <nil> Gw: 240.17.0.9 Flags: [] Table: 254} with gw 240.17.0.9 already installed
2020-11-02T14:27:16.1844004Z [cluster2] I1102 14:20:05.674267       1 route.go:787] Processing route {Ifindex: 15 Dst: 240.0.0.0/8 Src: 240.17.0.8 Gw: <nil> Flags: [] Table: 254}
2020-11-02T14:27:16.1844582Z [cluster2] I1102 14:20:05.674637       1 route.go:790] Found nil gw or dst
2020-11-02T14:27:16.1845113Z [cluster2] I1102 14:20:05.674715       1 route.go:829] Found equivalent route, not adding
2020-11-02T14:27:16.1845713Z [cluster2] I1102 14:20:05.674725       1 route.go:829] Found equivalent route, not adding
2020-11-02T14:27:16.1847346Z [cluster2] I1102 14:20:05.674756       1 route.go:647] Successfully processed local Endpoint "cluster2-submariner-cable-cluster2-172-17-0-9"
2020-11-02T14:27:16.1849536Z [cluster2] I1102 14:20:05.674773       1 route.go:558] Updating iptable rules for remote cluster's Endpoint "cluster1-submariner-cable-cluster1-172-17-0-5" with subnets [100.1.0.0/16 10.1.0.0/16]
2020-11-02T14:27:16.1851694Z [cluster2] I1102 14:20:35.670334       1 route.go:570] Processing local Endpoint cluster2-submariner-cable-cluster2-172-17-0-9 with IP 172.17.0.9, Host cluster2-worker for this cluster
2020-11-02T14:27:16.1853261Z [cluster2] I1102 14:20:35.670397       1 route.go:612] This route agent is running on a non-gateway/non-active node
2020-11-02T14:27:16.1854308Z [cluster2] I1102 14:20:35.670409       1 route.go:623] Creating the vxlan interface vx-submariner with gateway node IP 172.17.0.9
2020-11-02T14:27:16.1855109Z [cluster2] I1102 14:20:35.670707       1 vxlan.go:61] VxLAN interface already exists with same configuration.
2020-11-02T14:27:16.1855781Z [cluster2] I1102 14:20:35.676109       1 route.go:771] Reconciling routes to gw: 240.17.0.9
2020-11-02T14:27:16.1856427Z [cluster2] I1102 14:20:35.676427       1 route.go:787] Processing route {Ifindex: 15 Dst: 10.1.0.0/16 Src: <nil> Gw: 240.17.0.9 Flags: [] Table: 254}
2020-11-02T14:27:16.1857271Z [cluster2] I1102 14:20:35.676510       1 route.go:793] Found route {Ifindex: 15 Dst: 10.1.0.0/16 Src: <nil> Gw: 240.17.0.9 Flags: [] Table: 254} with gw 240.17.0.9 already installed
2020-11-02T14:27:16.1858067Z [cluster2] I1102 14:20:35.676548       1 route.go:787] Processing route {Ifindex: 15 Dst: 100.1.0.0/16 Src: <nil> Gw: 240.17.0.9 Flags: [] Table: 254}
2020-11-02T14:27:16.1858852Z [cluster2] I1102 14:20:35.676574       1 route.go:793] Found route {Ifindex: 15 Dst: 100.1.0.0/16 Src: <nil> Gw: 240.17.0.9 Flags: [] Table: 254} with gw 240.17.0.9 already installed
2020-11-02T14:27:16.1859632Z [cluster2] I1102 14:20:35.676609       1 route.go:787] Processing route {Ifindex: 15 Dst: 240.0.0.0/8 Src: 240.17.0.8 Gw: <nil> Flags: [] Table: 254}
2020-11-02T14:27:16.1860229Z [cluster2] I1102 14:20:35.676630       1 route.go:790] Found nil gw or dst
2020-11-02T14:27:16.1860746Z [cluster2] I1102 14:20:35.676754       1 route.go:829] Found equivalent route, not adding
2020-11-02T14:27:16.1861350Z [cluster2] I1102 14:20:35.676774       1 route.go:829] Found equivalent route, not adding
2020-11-02T14:27:16.1862774Z [cluster2] I1102 14:20:35.676809       1 route.go:647] Successfully processed local Endpoint "cluster2-submariner-cable-cluster2-172-17-0-9"
2020-11-02T14:27:16.1865146Z [cluster2] I1102 14:20:35.676849       1 route.go:558] Updating iptable rules for remote cluster's Endpoint "cluster1-submariner-cable-cluster1-172-17-0-5" with subnets [100.1.0.0/16 10.1.0.0/16]
2020-11-02T14:27:16.1867342Z [cluster2] I1102 14:21:05.670641       1 route.go:558] Updating iptable rules for remote cluster's Endpoint "cluster1-submariner-cable-cluster1-172-17-0-5" with subnets [100.1.0.0/16 10.1.0.0/16]
2020-11-02T14:27:16.1869676Z [cluster2] I1102 14:21:05.670682       1 route.go:570] Processing local Endpoint cluster2-submariner-cable-cluster2-172-17-0-9 with IP 172.17.0.9, Host cluster2-worker for this cluster
2020-11-02T14:27:16.1875845Z [cluster2] I1102 14:21:05.670692       1 route.go:612] This route agent is running on a non-gateway/non-active node
2020-11-02T14:27:16.1877291Z [cluster2] I1102 14:21:05.670701       1 route.go:623] Creating the vxlan interface vx-submariner with gateway node IP 172.17.0.9
2020-11-02T14:27:16.1878071Z [cluster2] I1102 14:21:05.670962       1 vxlan.go:61] VxLAN interface already exists with same configuration.
2020-11-02T14:27:16.1878699Z [cluster2] I1102 14:21:05.673993       1 route.go:771] Reconciling routes to gw: 240.17.0.9
2020-11-02T14:27:16.1879323Z [cluster2] I1102 14:21:05.674131       1 route.go:787] Processing route {Ifindex: 15 Dst: 10.1.0.0/16 Src: <nil> Gw: 240.17.0.9 Flags: [] Table: 254}
2020-11-02T14:27:16.1880231Z [cluster2] I1102 14:21:05.674177       1 route.go:793] Found route {Ifindex: 15 Dst: 10.1.0.0/16 Src: <nil> Gw: 240.17.0.9 Flags: [] Table: 254} with gw 240.17.0.9 already installed
2020-11-02T14:27:16.1880989Z [cluster2] I1102 14:21:05.674203       1 route.go:787] Processing route {Ifindex: 15 Dst: 100.1.0.0/16 Src: <nil> Gw: 240.17.0.9 Flags: [] Table: 254}
2020-11-02T14:27:16.1881741Z [cluster2] I1102 14:21:05.674212       1 route.go:793] Found route {Ifindex: 15 Dst: 100.1.0.0/16 Src: <nil> Gw: 240.17.0.9 Flags: [] Table: 254} with gw 240.17.0.9 already installed
2020-11-02T14:27:16.1882503Z [cluster2] I1102 14:21:05.674220       1 route.go:787] Processing route {Ifindex: 15 Dst: 240.0.0.0/8 Src: 240.17.0.8 Gw: <nil> Flags: [] Table: 254}
2020-11-02T14:27:16.1883262Z [cluster2] I1102 14:21:05.674227       1 route.go:790] Found nil gw or dst
2020-11-02T14:27:16.1884009Z [cluster2] I1102 14:21:05.674284       1 route.go:829] Found equivalent route, not adding
2020-11-02T14:27:16.1884771Z [cluster2] I1102 14:21:05.674293       1 route.go:829] Found equivalent route, not adding
2020-11-02T14:27:16.1886223Z [cluster2] I1102 14:21:05.674340       1 route.go:647] Successfully processed local Endpoint "cluster2-submariner-cable-cluster2-172-17-0-9"
2020-11-02T14:27:16.1888621Z [cluster2] I1102 14:21:35.670916       1 route.go:570] Processing local Endpoint cluster2-submariner-cable-cluster2-172-17-0-9 with IP 172.17.0.9, Host cluster2-worker for this cluster
2020-11-02T14:27:16.1890522Z [cluster2] I1102 14:21:35.670943       1 route.go:612] This route agent is running on a non-gateway/non-active node
2020-11-02T14:27:16.1891569Z [cluster2] I1102 14:21:35.670954       1 route.go:623] Creating the vxlan interface vx-submariner with gateway node IP 172.17.0.9
2020-11-02T14:27:16.1892375Z [cluster2] I1102 14:21:35.671229       1 vxlan.go:61] VxLAN interface already exists with same configuration.
2020-11-02T14:27:16.1893045Z [cluster2] I1102 14:21:35.674670       1 route.go:771] Reconciling routes to gw: 240.17.0.9
2020-11-02T14:27:16.1893884Z [cluster2] I1102 14:21:35.674815       1 route.go:787] Processing route {Ifindex: 15 Dst: 10.1.0.0/16 Src: <nil> Gw: 240.17.0.9 Flags: [] Table: 254}
2020-11-02T14:27:16.1894781Z [cluster2] I1102 14:21:35.674850       1 route.go:793] Found route {Ifindex: 15 Dst: 10.1.0.0/16 Src: <nil> Gw: 240.17.0.9 Flags: [] Table: 254} with gw 240.17.0.9 already installed
2020-11-02T14:27:16.1895691Z [cluster2] I1102 14:21:35.674860       1 route.go:787] Processing route {Ifindex: 15 Dst: 100.1.0.0/16 Src: <nil> Gw: 240.17.0.9 Flags: [] Table: 254}
2020-11-02T14:27:16.1896888Z [cluster2] I1102 14:21:35.674869       1 route.go:793] Found route {Ifindex: 15 Dst: 100.1.0.0/16 Src: <nil> Gw: 240.17.0.9 Flags: [] Table: 254} with gw 240.17.0.9 already installed
2020-11-02T14:27:16.1897760Z [cluster2] I1102 14:21:35.674877       1 route.go:787] Processing route {Ifindex: 15 Dst: 240.0.0.0/8 Src: 240.17.0.8 Gw: <nil> Flags: [] Table: 254}
2020-11-02T14:27:16.1898358Z [cluster2] I1102 14:21:35.674885       1 route.go:790] Found nil gw or dst
2020-11-02T14:27:16.1898936Z [cluster2] I1102 14:21:35.674954       1 route.go:829] Found equivalent route, not adding
2020-11-02T14:27:16.1899539Z [cluster2] I1102 14:21:35.674988       1 route.go:829] Found equivalent route, not adding
2020-11-02T14:27:16.1900990Z [cluster2] I1102 14:21:35.675023       1 route.go:647] Successfully processed local Endpoint "cluster2-submariner-cable-cluster2-172-17-0-9"
2020-11-02T14:27:16.1903066Z [cluster2] I1102 14:21:35.675071       1 route.go:558] Updating iptable rules for remote cluster's Endpoint "cluster1-submariner-cable-cluster1-172-17-0-5" with subnets [100.1.0.0/16 10.1.0.0/16]
2020-11-02T14:27:16.1905571Z [cluster2] I1102 14:22:05.670925       1 route.go:570] Processing local Endpoint cluster2-submariner-cable-cluster2-172-17-0-9 with IP 172.17.0.9, Host cluster2-worker for this cluster
2020-11-02T14:27:16.1907396Z [cluster2] I1102 14:22:05.670975       1 route.go:612] This route agent is running on a non-gateway/non-active node
2020-11-02T14:27:16.1908436Z [cluster2] I1102 14:22:05.670987       1 route.go:623] Creating the vxlan interface vx-submariner with gateway node IP 172.17.0.9
2020-11-02T14:27:16.1909239Z [cluster2] I1102 14:22:05.671367       1 vxlan.go:61] VxLAN interface already exists with same configuration.
2020-11-02T14:27:16.1909910Z [cluster2] I1102 14:22:05.674647       1 route.go:771] Reconciling routes to gw: 240.17.0.9
2020-11-02T14:27:16.1910560Z [cluster2] I1102 14:22:05.674786       1 route.go:787] Processing route {Ifindex: 15 Dst: 10.1.0.0/16 Src: <nil> Gw: 240.17.0.9 Flags: [] Table: 254}
2020-11-02T14:27:16.1911510Z [cluster2] I1102 14:22:05.674819       1 route.go:793] Found route {Ifindex: 15 Dst: 10.1.0.0/16 Src: <nil> Gw: 240.17.0.9 Flags: [] Table: 254} with gw 240.17.0.9 already installed
2020-11-02T14:27:16.1912302Z [cluster2] I1102 14:22:05.674829       1 route.go:787] Processing route {Ifindex: 15 Dst: 100.1.0.0/16 Src: <nil> Gw: 240.17.0.9 Flags: [] Table: 254}
2020-11-02T14:27:16.1913087Z [cluster2] I1102 14:22:05.674837       1 route.go:793] Found route {Ifindex: 15 Dst: 100.1.0.0/16 Src: <nil> Gw: 240.17.0.9 Flags: [] Table: 254} with gw 240.17.0.9 already installed
2020-11-02T14:27:16.1914228Z [cluster2] I1102 14:22:05.674859       1 route.go:787] Processing route {Ifindex: 15 Dst: 240.0.0.0/8 Src: 240.17.0.8 Gw: <nil> Flags: [] Table: 254}
2020-11-02T14:27:16.1914822Z [cluster2] I1102 14:22:05.674867       1 route.go:790] Found nil gw or dst
2020-11-02T14:27:16.1915359Z [cluster2] I1102 14:22:05.674935       1 route.go:829] Found equivalent route, not adding
2020-11-02T14:27:16.1915964Z [cluster2] I1102 14:22:05.674944       1 route.go:829] Found equivalent route, not adding
2020-11-02T14:27:16.1918053Z [cluster2] I1102 14:22:05.674972       1 route.go:647] Successfully processed local Endpoint "cluster2-submariner-cable-cluster2-172-17-0-9"
2020-11-02T14:27:16.1920292Z [cluster2] I1102 14:22:05.674988       1 route.go:558] Updating iptable rules for remote cluster's Endpoint "cluster1-submariner-cable-cluster1-172-17-0-5" with subnets [100.1.0.0/16 10.1.0.0/16]
2020-11-02T14:27:16.1922460Z [cluster2] I1102 14:22:35.671102       1 route.go:570] Processing local Endpoint cluster2-submariner-cable-cluster2-172-17-0-9 with IP 172.17.0.9, Host cluster2-worker for this cluster
2020-11-02T14:27:16.1923998Z [cluster2] I1102 14:22:35.671334       1 route.go:612] This route agent is running on a non-gateway/non-active node
2020-11-02T14:27:16.1925037Z [cluster2] I1102 14:22:35.671392       1 route.go:623] Creating the vxlan interface vx-submariner with gateway node IP 172.17.0.9
2020-11-02T14:27:16.1926035Z [cluster2] I1102 14:22:35.671659       1 vxlan.go:61] VxLAN interface already exists with same configuration.
2020-11-02T14:27:16.1926707Z [cluster2] I1102 14:22:35.675065       1 route.go:771] Reconciling routes to gw: 240.17.0.9
2020-11-02T14:27:16.1927354Z [cluster2] I1102 14:22:35.675377       1 route.go:787] Processing route {Ifindex: 15 Dst: 10.1.0.0/16 Src: <nil> Gw: 240.17.0.9 Flags: [] Table: 254}
2020-11-02T14:27:16.1928286Z [cluster2] I1102 14:22:35.675414       1 route.go:793] Found route {Ifindex: 15 Dst: 10.1.0.0/16 Src: <nil> Gw: 240.17.0.9 Flags: [] Table: 254} with gw 240.17.0.9 already installed
2020-11-02T14:27:16.1929303Z [cluster2] I1102 14:22:35.675424       1 route.go:787] Processing route {Ifindex: 15 Dst: 100.1.0.0/16 Src: <nil> Gw: 240.17.0.9 Flags: [] Table: 254}
2020-11-02T14:27:16.1930087Z [cluster2] I1102 14:22:35.675432       1 route.go:793] Found route {Ifindex: 15 Dst: 100.1.0.0/16 Src: <nil> Gw: 240.17.0.9 Flags: [] Table: 254} with gw 240.17.0.9 already installed
2020-11-02T14:27:16.1930874Z [cluster2] I1102 14:22:35.675673       1 route.go:787] Processing route {Ifindex: 15 Dst: 240.0.0.0/8 Src: 240.17.0.8 Gw: <nil> Flags: [] Table: 254}
2020-11-02T14:27:16.1931468Z [cluster2] I1102 14:22:35.675729       1 route.go:790] Found nil gw or dst
2020-11-02T14:27:16.1933084Z [cluster2] I1102 14:22:35.675841       1 route.go:829] Found equivalent route, not adding
2020-11-02T14:27:16.1933719Z [cluster2] I1102 14:22:35.675886       1 route.go:829] Found equivalent route, not adding
2020-11-02T14:27:16.1935254Z [cluster2] I1102 14:22:35.675973       1 route.go:647] Successfully processed local Endpoint "cluster2-submariner-cable-cluster2-172-17-0-9"
2020-11-02T14:27:16.1937651Z [cluster2] I1102 14:22:35.676049       1 route.go:558] Updating iptable rules for remote cluster's Endpoint "cluster1-submariner-cable-cluster1-172-17-0-5" with subnets [100.1.0.0/16 10.1.0.0/16]
2020-11-02T14:27:16.1939817Z [cluster2] I1102 14:23:05.671246       1 route.go:558] Updating iptable rules for remote cluster's Endpoint "cluster1-submariner-cable-cluster1-172-17-0-5" with subnets [100.1.0.0/16 10.1.0.0/16]
2020-11-02T14:27:16.1955419Z [cluster2] I1102 14:23:05.671314       1 route.go:570] Processing local Endpoint cluster2-submariner-cable-cluster2-172-17-0-9 with IP 172.17.0.9, Host cluster2-worker for this cluster
2020-11-02T14:27:16.1957485Z [cluster2] I1102 14:23:05.671323       1 route.go:612] This route agent is running on a non-gateway/non-active node
2020-11-02T14:27:16.1958540Z [cluster2] I1102 14:23:05.671332       1 route.go:623] Creating the vxlan interface vx-submariner with gateway node IP 172.17.0.9
2020-11-02T14:27:16.1960364Z [cluster2] I1102 14:23:05.671505       1 vxlan.go:61] VxLAN interface already exists with same configuration.
2020-11-02T14:27:16.1961021Z [cluster2] I1102 14:23:05.675421       1 route.go:771] Reconciling routes to gw: 240.17.0.9
2020-11-02T14:27:16.1961833Z [cluster2] I1102 14:23:05.675611       1 route.go:787] Processing route {Ifindex: 15 Dst: 10.1.0.0/16 Src: <nil> Gw: 240.17.0.9 Flags: [] Table: 254}
2020-11-02T14:27:16.1962594Z [cluster2] I1102 14:23:05.675647       1 route.go:793] Found route {Ifindex: 15 Dst: 10.1.0.0/16 Src: <nil> Gw: 240.17.0.9 Flags: [] Table: 254} with gw 240.17.0.9 already installed
2020-11-02T14:27:16.1963349Z [cluster2] I1102 14:23:05.675658       1 route.go:787] Processing route {Ifindex: 15 Dst: 100.1.0.0/16 Src: <nil> Gw: 240.17.0.9 Flags: [] Table: 254}
2020-11-02T14:27:16.1965545Z [cluster2] I1102 14:23:05.675667       1 route.go:793] Found route {Ifindex: 15 Dst: 100.1.0.0/16 Src: <nil> Gw: 240.17.0.9 Flags: [] Table: 254} with gw 240.17.0.9 already installed
2020-11-02T14:27:16.1966280Z [cluster2] I1102 14:23:05.675680       1 route.go:787] Processing route {Ifindex: 15 Dst: 240.0.0.0/8 Src: 240.17.0.8 Gw: <nil> Flags: [] Table: 254}
2020-11-02T14:27:16.1966834Z [cluster2] I1102 14:23:05.675689       1 route.go:790] Found nil gw or dst
2020-11-02T14:27:16.1967700Z [cluster2] I1102 14:23:05.675769       1 route.go:829] Found equivalent route, not adding
2020-11-02T14:27:16.1968540Z [cluster2] I1102 14:23:05.675786       1 route.go:829] Found equivalent route, not adding
2020-11-02T14:27:16.1970175Z [cluster2] I1102 14:23:05.675818       1 route.go:647] Successfully processed local Endpoint "cluster2-submariner-cable-cluster2-172-17-0-9"
2020-11-02T14:27:16.1972532Z [cluster2] I1102 14:23:35.671259       1 route.go:570] Processing local Endpoint cluster2-submariner-cable-cluster2-172-17-0-9 with IP 172.17.0.9, Host cluster2-worker for this cluster
2020-11-02T14:27:16.1974036Z [cluster2] I1102 14:23:35.671306       1 route.go:612] This route agent is running on a non-gateway/non-active node
2020-11-02T14:27:16.1975401Z [cluster2] I1102 14:23:35.671318       1 route.go:623] Creating the vxlan interface vx-submariner with gateway node IP 172.17.0.9
2020-11-02T14:27:16.1976181Z [cluster2] I1102 14:23:35.671535       1 vxlan.go:61] VxLAN interface already exists with same configuration.
2020-11-02T14:27:16.1976820Z [cluster2] I1102 14:23:35.674935       1 route.go:771] Reconciling routes to gw: 240.17.0.9
2020-11-02T14:27:16.1977448Z [cluster2] I1102 14:23:35.675082       1 route.go:787] Processing route {Ifindex: 15 Dst: 10.1.0.0/16 Src: <nil> Gw: 240.17.0.9 Flags: [] Table: 254}
2020-11-02T14:27:16.1978413Z [cluster2] I1102 14:23:35.675107       1 route.go:793] Found route {Ifindex: 15 Dst: 10.1.0.0/16 Src: <nil> Gw: 240.17.0.9 Flags: [] Table: 254} with gw 240.17.0.9 already installed
2020-11-02T14:27:16.1979196Z [cluster2] I1102 14:23:35.675116       1 route.go:787] Processing route {Ifindex: 15 Dst: 100.1.0.0/16 Src: <nil> Gw: 240.17.0.9 Flags: [] Table: 254}
2020-11-02T14:27:16.1979978Z [cluster2] I1102 14:23:35.675185       1 route.go:793] Found route {Ifindex: 15 Dst: 100.1.0.0/16 Src: <nil> Gw: 240.17.0.9 Flags: [] Table: 254} with gw 240.17.0.9 already installed
2020-11-02T14:27:16.1980756Z [cluster2] I1102 14:23:35.675193       1 route.go:787] Processing route {Ifindex: 15 Dst: 240.0.0.0/8 Src: 240.17.0.8 Gw: <nil> Flags: [] Table: 254}
2020-11-02T14:27:16.1981620Z [cluster2] I1102 14:23:35.675201       1 route.go:790] Found nil gw or dst
2020-11-02T14:27:16.1982155Z [cluster2] I1102 14:23:35.675256       1 route.go:829] Found equivalent route, not adding
2020-11-02T14:27:16.1982757Z [cluster2] I1102 14:23:35.675264       1 route.go:829] Found equivalent route, not adding
2020-11-02T14:27:16.1985720Z [cluster2] I1102 14:23:35.675294       1 route.go:647] Successfully processed local Endpoint "cluster2-submariner-cable-cluster2-172-17-0-9"
2020-11-02T14:27:16.1988268Z [cluster2] I1102 14:23:35.675309       1 route.go:558] Updating iptable rules for remote cluster's Endpoint "cluster1-submariner-cable-cluster1-172-17-0-5" with subnets [100.1.0.0/16 10.1.0.0/16]
2020-11-02T14:27:16.1990732Z [cluster2] I1102 14:24:05.671487       1 route.go:558] Updating iptable rules for remote cluster's Endpoint "cluster1-submariner-cable-cluster1-172-17-0-5" with subnets [100.1.0.0/16 10.1.0.0/16]
2020-11-02T14:27:16.1998095Z [cluster2] I1102 14:24:05.671530       1 route.go:570] Processing local Endpoint cluster2-submariner-cable-cluster2-172-17-0-9 with IP 172.17.0.9, Host cluster2-worker for this cluster
2020-11-02T14:27:16.2003482Z [cluster2] I1102 14:24:05.671560       1 route.go:612] This route agent is running on a non-gateway/non-active node
2020-11-02T14:27:16.2008388Z [cluster2] I1102 14:24:05.671570       1 route.go:623] Creating the vxlan interface vx-submariner with gateway node IP 172.17.0.9
2020-11-02T14:27:16.2009223Z [cluster2] I1102 14:24:05.671924       1 vxlan.go:61] VxLAN interface already exists with same configuration.
2020-11-02T14:27:16.2009902Z [cluster2] I1102 14:24:05.675395       1 route.go:771] Reconciling routes to gw: 240.17.0.9
2020-11-02T14:27:16.2010535Z [cluster2] I1102 14:24:05.675551       1 route.go:787] Processing route {Ifindex: 15 Dst: 10.1.0.0/16 Src: <nil> Gw: 240.17.0.9 Flags: [] Table: 254}
2020-11-02T14:27:16.2011631Z [cluster2] I1102 14:24:05.675585       1 route.go:793] Found route {Ifindex: 15 Dst: 10.1.0.0/16 Src: <nil> Gw: 240.17.0.9 Flags: [] Table: 254} with gw 240.17.0.9 already installed
2020-11-02T14:27:16.2012439Z [cluster2] I1102 14:24:05.675595       1 route.go:787] Processing route {Ifindex: 15 Dst: 100.1.0.0/16 Src: <nil> Gw: 240.17.0.9 Flags: [] Table: 254}
2020-11-02T14:27:16.2013226Z [cluster2] I1102 14:24:05.675611       1 route.go:793] Found route {Ifindex: 15 Dst: 100.1.0.0/16 Src: <nil> Gw: 240.17.0.9 Flags: [] Table: 254} with gw 240.17.0.9 already installed
2020-11-02T14:27:16.2014010Z [cluster2] I1102 14:24:05.675619       1 route.go:787] Processing route {Ifindex: 15 Dst: 240.0.0.0/8 Src: 240.17.0.8 Gw: <nil> Flags: [] Table: 254}
2020-11-02T14:27:16.2014605Z [cluster2] I1102 14:24:05.675627       1 route.go:790] Found nil gw or dst
2020-11-02T14:27:16.2015139Z [cluster2] I1102 14:24:05.675684       1 route.go:829] Found equivalent route, not adding
2020-11-02T14:27:16.2015746Z [cluster2] I1102 14:24:05.675701       1 route.go:829] Found equivalent route, not adding
2020-11-02T14:27:16.2017262Z [cluster2] I1102 14:24:05.675733       1 route.go:647] Successfully processed local Endpoint "cluster2-submariner-cable-cluster2-172-17-0-9"
2020-11-02T14:27:16.2020304Z [cluster2] I1102 14:24:35.671680       1 route.go:558] Updating iptable rules for remote cluster's Endpoint "cluster1-submariner-cable-cluster1-172-17-0-5" with subnets [100.1.0.0/16 10.1.0.0/16]
2020-11-02T14:27:16.2022489Z [cluster2] I1102 14:24:35.671722       1 route.go:570] Processing local Endpoint cluster2-submariner-cable-cluster2-172-17-0-9 with IP 172.17.0.9, Host cluster2-worker for this cluster
2020-11-02T14:27:16.2024052Z [cluster2] I1102 14:24:35.671737       1 route.go:612] This route agent is running on a non-gateway/non-active node
2020-11-02T14:27:16.2027144Z [cluster2] I1102 14:24:35.671748       1 route.go:623] Creating the vxlan interface vx-submariner with gateway node IP 172.17.0.9
2020-11-02T14:27:16.2027971Z [cluster2] I1102 14:24:35.672232       1 vxlan.go:61] VxLAN interface already exists with same configuration.
2020-11-02T14:27:16.2028650Z [cluster2] I1102 14:24:35.675990       1 route.go:771] Reconciling routes to gw: 240.17.0.9
2020-11-02T14:27:16.2029300Z [cluster2] I1102 14:24:35.676150       1 route.go:787] Processing route {Ifindex: 15 Dst: 10.1.0.0/16 Src: <nil> Gw: 240.17.0.9 Flags: [] Table: 254}
2020-11-02T14:27:16.2030292Z [cluster2] I1102 14:24:35.676185       1 route.go:793] Found route {Ifindex: 15 Dst: 10.1.0.0/16 Src: <nil> Gw: 240.17.0.9 Flags: [] Table: 254} with gw 240.17.0.9 already installed
2020-11-02T14:27:16.2031087Z [cluster2] I1102 14:24:35.676196       1 route.go:787] Processing route {Ifindex: 15 Dst: 100.1.0.0/16 Src: <nil> Gw: 240.17.0.9 Flags: [] Table: 254}
2020-11-02T14:27:16.2032018Z [cluster2] I1102 14:24:35.676205       1 route.go:793] Found route {Ifindex: 15 Dst: 100.1.0.0/16 Src: <nil> Gw: 240.17.0.9 Flags: [] Table: 254} with gw 240.17.0.9 already installed
2020-11-02T14:27:16.2033311Z [cluster2] I1102 14:24:35.676213       1 route.go:787] Processing route {Ifindex: 15 Dst: 240.0.0.0/8 Src: 240.17.0.8 Gw: <nil> Flags: [] Table: 254}
2020-11-02T14:27:16.2033925Z [cluster2] I1102 14:24:35.676229       1 route.go:790] Found nil gw or dst
2020-11-02T14:27:16.2034458Z [cluster2] I1102 14:24:35.676288       1 route.go:829] Found equivalent route, not adding
2020-11-02T14:27:16.2035064Z [cluster2] I1102 14:24:35.676297       1 route.go:829] Found equivalent route, not adding
2020-11-02T14:27:16.2036780Z [cluster2] I1102 14:24:35.676328       1 route.go:647] Successfully processed local Endpoint "cluster2-submariner-cable-cluster2-172-17-0-9"
2020-11-02T14:27:16.2039182Z [cluster2] I1102 14:25:05.671862       1 route.go:558] Updating iptable rules for remote cluster's Endpoint "cluster1-submariner-cable-cluster1-172-17-0-5" with subnets [100.1.0.0/16 10.1.0.0/16]
2020-11-02T14:27:16.2041422Z [cluster2] I1102 14:25:05.671913       1 route.go:570] Processing local Endpoint cluster2-submariner-cable-cluster2-172-17-0-9 with IP 172.17.0.9, Host cluster2-worker for this cluster
2020-11-02T14:27:16.2043454Z [cluster2] I1102 14:25:05.671923       1 route.go:612] This route agent is running on a non-gateway/non-active node
2020-11-02T14:27:16.2044544Z [cluster2] I1102 14:25:05.671932       1 route.go:623] Creating the vxlan interface vx-submariner with gateway node IP 172.17.0.9
2020-11-02T14:27:16.2045358Z [cluster2] I1102 14:25:05.672124       1 vxlan.go:61] VxLAN interface already exists with same configuration.
2020-11-02T14:27:16.2046046Z [cluster2] I1102 14:25:05.675573       1 route.go:771] Reconciling routes to gw: 240.17.0.9
2020-11-02T14:27:16.2046689Z [cluster2] I1102 14:25:05.675778       1 route.go:787] Processing route {Ifindex: 15 Dst: 10.1.0.0/16 Src: <nil> Gw: 240.17.0.9 Flags: [] Table: 254}
2020-11-02T14:27:16.2047476Z [cluster2] I1102 14:25:05.675807       1 route.go:793] Found route {Ifindex: 15 Dst: 10.1.0.0/16 Src: <nil> Gw: 240.17.0.9 Flags: [] Table: 254} with gw 240.17.0.9 already installed
2020-11-02T14:27:16.2048268Z [cluster2] I1102 14:25:05.675826       1 route.go:787] Processing route {Ifindex: 15 Dst: 100.1.0.0/16 Src: <nil> Gw: 240.17.0.9 Flags: [] Table: 254}
2020-11-02T14:27:16.2049037Z [cluster2] I1102 14:25:05.675835       1 route.go:793] Found route {Ifindex: 15 Dst: 100.1.0.0/16 Src: <nil> Gw: 240.17.0.9 Flags: [] Table: 254} with gw 240.17.0.9 already installed
2020-11-02T14:27:16.2049815Z [cluster2] I1102 14:25:05.675842       1 route.go:787] Processing route {Ifindex: 15 Dst: 240.0.0.0/8 Src: 240.17.0.8 Gw: <nil> Flags: [] Table: 254}
2020-11-02T14:27:16.2050410Z [cluster2] I1102 14:25:05.675860       1 route.go:790] Found nil gw or dst
2020-11-02T14:27:16.2051092Z [cluster2] I1102 14:25:05.675917       1 route.go:829] Found equivalent route, not adding
2020-11-02T14:27:16.2051674Z [cluster2] I1102 14:25:05.675926       1 route.go:829] Found equivalent route, not adding
2020-11-02T14:27:16.2053052Z [cluster2] I1102 14:25:05.675964       1 route.go:647] Successfully processed local Endpoint "cluster2-submariner-cable-cluster2-172-17-0-9"
2020-11-02T14:27:16.2056468Z [cluster2] I1102 14:25:35.672026       1 route.go:570] Processing local Endpoint cluster2-submariner-cable-cluster2-172-17-0-9 with IP 172.17.0.9, Host cluster2-worker for this cluster
2020-11-02T14:27:16.2058046Z [cluster2] I1102 14:25:35.672066       1 route.go:612] This route agent is running on a non-gateway/non-active node
2020-11-02T14:27:16.2059092Z [cluster2] I1102 14:25:35.672300       1 route.go:623] Creating the vxlan interface vx-submariner with gateway node IP 172.17.0.9
2020-11-02T14:27:16.2060028Z [cluster2] I1102 14:25:35.672600       1 vxlan.go:61] VxLAN interface already exists with same configuration.
2020-11-02T14:27:16.2060676Z [cluster2] I1102 14:25:35.676114       1 route.go:771] Reconciling routes to gw: 240.17.0.9
2020-11-02T14:27:16.2061298Z [cluster2] I1102 14:25:35.676253       1 route.go:787] Processing route {Ifindex: 15 Dst: 10.1.0.0/16 Src: <nil> Gw: 240.17.0.9 Flags: [] Table: 254}
2020-11-02T14:27:16.2062273Z [cluster2] I1102 14:25:35.676287       1 route.go:793] Found route {Ifindex: 15 Dst: 10.1.0.0/16 Src: <nil> Gw: 240.17.0.9 Flags: [] Table: 254} with gw 240.17.0.9 already installed
2020-11-02T14:27:16.2063055Z [cluster2] I1102 14:25:35.676297       1 route.go:787] Processing route {Ifindex: 15 Dst: 100.1.0.0/16 Src: <nil> Gw: 240.17.0.9 Flags: [] Table: 254}
2020-11-02T14:27:16.2063839Z [cluster2] I1102 14:25:35.676306       1 route.go:793] Found route {Ifindex: 15 Dst: 100.1.0.0/16 Src: <nil> Gw: 240.17.0.9 Flags: [] Table: 254} with gw 240.17.0.9 already installed
2020-11-02T14:27:16.2064617Z [cluster2] I1102 14:25:35.676319       1 route.go:787] Processing route {Ifindex: 15 Dst: 240.0.0.0/8 Src: 240.17.0.8 Gw: <nil> Flags: [] Table: 254}
2020-11-02T14:27:16.2065366Z [cluster2] I1102 14:25:35.676327       1 route.go:790] Found nil gw or dst
2020-11-02T14:27:16.2066249Z [cluster2] I1102 14:25:35.676387       1 route.go:829] Found equivalent route, not adding
2020-11-02T14:27:16.2067018Z [cluster2] I1102 14:25:35.676396       1 route.go:829] Found equivalent route, not adding
2020-11-02T14:27:16.2068583Z [cluster2] I1102 14:25:35.676425       1 route.go:647] Successfully processed local Endpoint "cluster2-submariner-cable-cluster2-172-17-0-9"
2020-11-02T14:27:16.2070703Z [cluster2] I1102 14:25:35.676441       1 route.go:558] Updating iptable rules for remote cluster's Endpoint "cluster1-submariner-cable-cluster1-172-17-0-5" with subnets [100.1.0.0/16 10.1.0.0/16]
2020-11-02T14:27:16.2074510Z [cluster2] I1102 14:26:05.672165       1 route.go:570] Processing local Endpoint cluster2-submariner-cable-cluster2-172-17-0-9 with IP 172.17.0.9, Host cluster2-worker for this cluster
2020-11-02T14:27:16.2076338Z [cluster2] I1102 14:26:05.672196       1 route.go:612] This route agent is running on a non-gateway/non-active node
2020-11-02T14:27:16.2077393Z [cluster2] I1102 14:26:05.672207       1 route.go:623] Creating the vxlan interface vx-submariner with gateway node IP 172.17.0.9
2020-11-02T14:27:16.2078216Z [cluster2] I1102 14:26:05.672423       1 vxlan.go:61] VxLAN interface already exists with same configuration.
2020-11-02T14:27:16.2078892Z [cluster2] I1102 14:26:05.675771       1 route.go:771] Reconciling routes to gw: 240.17.0.9
2020-11-02T14:27:16.2079542Z [cluster2] I1102 14:26:05.675903       1 route.go:787] Processing route {Ifindex: 15 Dst: 10.1.0.0/16 Src: <nil> Gw: 240.17.0.9 Flags: [] Table: 254}
2020-11-02T14:27:16.2080327Z [cluster2] I1102 14:26:05.675946       1 route.go:793] Found route {Ifindex: 15 Dst: 10.1.0.0/16 Src: <nil> Gw: 240.17.0.9 Flags: [] Table: 254} with gw 240.17.0.9 already installed
2020-11-02T14:27:16.2081114Z [cluster2] I1102 14:26:05.675956       1 route.go:787] Processing route {Ifindex: 15 Dst: 100.1.0.0/16 Src: <nil> Gw: 240.17.0.9 Flags: [] Table: 254}
2020-11-02T14:27:16.2081902Z [cluster2] I1102 14:26:05.675972       1 route.go:793] Found route {Ifindex: 15 Dst: 100.1.0.0/16 Src: <nil> Gw: 240.17.0.9 Flags: [] Table: 254} with gw 240.17.0.9 already installed
2020-11-02T14:27:16.2082688Z [cluster2] I1102 14:26:05.675987       1 route.go:787] Processing route {Ifindex: 15 Dst: 240.0.0.0/8 Src: 240.17.0.8 Gw: <nil> Flags: [] Table: 254}
2020-11-02T14:27:16.2083332Z [cluster2] I1102 14:26:05.675996       1 route.go:790] Found nil gw or dst
2020-11-02T14:27:16.2083847Z [cluster2] I1102 14:26:05.676088       1 route.go:829] Found equivalent route, not adding
2020-11-02T14:27:16.2084454Z [cluster2] I1102 14:26:05.676105       1 route.go:829] Found equivalent route, not adding
2020-11-02T14:27:16.2085982Z [cluster2] I1102 14:26:05.676136       1 route.go:647] Successfully processed local Endpoint "cluster2-submariner-cable-cluster2-172-17-0-9"
2020-11-02T14:27:16.2088057Z [cluster2] I1102 14:26:05.676182       1 route.go:558] Updating iptable rules for remote cluster's Endpoint "cluster1-submariner-cable-cluster1-172-17-0-5" with subnets [100.1.0.0/16 10.1.0.0/16]
2020-11-02T14:27:16.2091511Z [cluster2] I1102 14:26:35.672451       1 route.go:570] Processing local Endpoint cluster2-submariner-cable-cluster2-172-17-0-9 with IP 172.17.0.9, Host cluster2-worker for this cluster
2020-11-02T14:27:16.2093117Z [cluster2] I1102 14:26:35.672499       1 route.go:612] This route agent is running on a non-gateway/non-active node
2020-11-02T14:27:16.2094167Z [cluster2] I1102 14:26:35.672511       1 route.go:623] Creating the vxlan interface vx-submariner with gateway node IP 172.17.0.9
2020-11-02T14:27:16.2094978Z [cluster2] I1102 14:26:35.676147       1 vxlan.go:61] VxLAN interface already exists with same configuration.
2020-11-02T14:27:16.2095647Z [cluster2] I1102 14:26:35.679610       1 route.go:771] Reconciling routes to gw: 240.17.0.9
2020-11-02T14:27:16.2096296Z [cluster2] I1102 14:26:35.679782       1 route.go:787] Processing route {Ifindex: 15 Dst: 10.1.0.0/16 Src: <nil> Gw: 240.17.0.9 Flags: [] Table: 254}
2020-11-02T14:27:16.2097078Z [cluster2] I1102 14:26:35.679818       1 route.go:793] Found route {Ifindex: 15 Dst: 10.1.0.0/16 Src: <nil> Gw: 240.17.0.9 Flags: [] Table: 254} with gw 240.17.0.9 already installed
2020-11-02T14:27:16.2098710Z [cluster2] I1102 14:26:35.679838       1 route.go:787] Processing route {Ifindex: 15 Dst: 100.1.0.0/16 Src: <nil> Gw: 240.17.0.9 Flags: [] Table: 254}
2020-11-02T14:27:16.2099726Z [cluster2] I1102 14:26:35.679848       1 route.go:793] Found route {Ifindex: 15 Dst: 100.1.0.0/16 Src: <nil> Gw: 240.17.0.9 Flags: [] Table: 254} with gw 240.17.0.9 already installed
2020-11-02T14:27:16.2101117Z [lighthouse]$ [cluster2] kubectl -n submariner-operator logs submariner-routeagent-ct9kx
2020-11-02T14:27:16.2102607Z [lighthouse]$ [cluster2] command kubectl --context=cluster2 -n submariner-operator logs submariner-routeagent-ct9kx
2020-11-02T14:27:16.2104140Z [lighthouse]$ [cluster2] kubectl --context=cluster2 -n submariner-operator logs submariner-routeagent-ct9kx
2020-11-02T14:27:16.4916201Z [cluster2] I1102 14:26:35.679863       1 route.go:787] Processing route {Ifindex: 15 Dst: 240.0.0.0/8 Src: 240.17.0.8 Gw: <nil> Flags: [] Table: 254}
2020-11-02T14:27:16.4917912Z [cluster2] I1102 14:26:35.679871       1 route.go:790] Found nil gw or dst
2020-11-02T14:27:16.4918448Z [cluster2] I1102 14:26:35.679951       1 route.go:829] Found equivalent route, not adding
2020-11-02T14:27:16.4919279Z [cluster2] I1102 14:26:35.679967       1 route.go:829] Found equivalent route, not adding
2020-11-02T14:27:16.4921242Z [cluster2] I1102 14:26:35.680000       1 route.go:647] Successfully processed local Endpoint "cluster2-submariner-cable-cluster2-172-17-0-9"
2020-11-02T14:27:16.4923466Z [cluster2] I1102 14:26:35.680026       1 route.go:558] Updating iptable rules for remote cluster's Endpoint "cluster1-submariner-cable-cluster1-172-17-0-5" with subnets [100.1.0.0/16 10.1.0.0/16]
2020-11-02T14:27:16.4925637Z [cluster2] I1102 14:27:05.672516       1 route.go:570] Processing local Endpoint cluster2-submariner-cable-cluster2-172-17-0-9 with IP 172.17.0.9, Host cluster2-worker for this cluster
2020-11-02T14:27:16.4927208Z [cluster2] I1102 14:27:05.672574       1 route.go:612] This route agent is running on a non-gateway/non-active node
2020-11-02T14:27:16.4928260Z [cluster2] I1102 14:27:05.672586       1 route.go:623] Creating the vxlan interface vx-submariner with gateway node IP 172.17.0.9
2020-11-02T14:27:16.4929068Z [cluster2] I1102 14:27:05.672847       1 vxlan.go:61] VxLAN interface already exists with same configuration.
2020-11-02T14:27:16.4929742Z [cluster2] I1102 14:27:05.675951       1 route.go:771] Reconciling routes to gw: 240.17.0.9
2020-11-02T14:27:16.4930393Z [cluster2] I1102 14:27:05.676098       1 route.go:787] Processing route {Ifindex: 15 Dst: 10.1.0.0/16 Src: <nil> Gw: 240.17.0.9 Flags: [] Table: 254}
2020-11-02T14:27:16.4931180Z [cluster2] I1102 14:27:05.676124       1 route.go:793] Found route {Ifindex: 15 Dst: 10.1.0.0/16 Src: <nil> Gw: 240.17.0.9 Flags: [] Table: 254} with gw 240.17.0.9 already installed
2020-11-02T14:27:16.4931968Z [cluster2] I1102 14:27:05.676134       1 route.go:787] Processing route {Ifindex: 15 Dst: 100.1.0.0/16 Src: <nil> Gw: 240.17.0.9 Flags: [] Table: 254}
2020-11-02T14:27:16.4932763Z [cluster2] I1102 14:27:05.676143       1 route.go:793] Found route {Ifindex: 15 Dst: 100.1.0.0/16 Src: <nil> Gw: 240.17.0.9 Flags: [] Table: 254} with gw 240.17.0.9 already installed
2020-11-02T14:27:16.4933546Z [cluster2] I1102 14:27:05.676151       1 route.go:787] Processing route {Ifindex: 15 Dst: 240.0.0.0/8 Src: 240.17.0.8 Gw: <nil> Flags: [] Table: 254}
2020-11-02T14:27:16.4934141Z [cluster2] I1102 14:27:05.676159       1 route.go:790] Found nil gw or dst
2020-11-02T14:27:16.4934675Z [cluster2] I1102 14:27:05.676214       1 route.go:829] Found equivalent route, not adding
2020-11-02T14:27:16.4935394Z [cluster2] I1102 14:27:05.676223       1 route.go:829] Found equivalent route, not adding
2020-11-02T14:27:16.4936873Z [cluster2] I1102 14:27:05.676252       1 route.go:647] Successfully processed local Endpoint "cluster2-submariner-cable-cluster2-172-17-0-9"
2020-11-02T14:27:16.4939208Z [cluster2] I1102 14:27:05.676268       1 route.go:558] Updating iptable rules for remote cluster's Endpoint "cluster1-submariner-cable-cluster1-172-17-0-5" with subnets [100.1.0.0/16 10.1.0.0/16]
2020-11-02T14:27:16.4941094Z [cluster2] +++++++++++++++++++++: Logs for Pod submariner-routeagent-ct9kx in namespace submariner-operator :++++++++++++++++++++++
2020-11-02T14:27:16.4942085Z [cluster2] + trap 'exit 1' SIGTERM SIGINT
2020-11-02T14:27:16.4942531Z [cluster2] + SUBMARINER_VERBOSITY=2
2020-11-02T14:27:16.4943082Z [cluster2] + '[' false == true ']'
2020-11-02T14:27:16.4943555Z [cluster2] + DEBUG=-v=2
2020-11-02T14:27:16.4944148Z [cluster2] + for f in iptables-save iptables
2020-11-02T14:27:16.4944819Z [cluster2] ++ find_iptables_on_host iptables-save
2020-11-02T14:27:16.4945532Z [cluster2] ++ chroot /host test -x /usr/sbin/iptables-save
2020-11-02T14:27:16.4946326Z [cluster2] iptables-save is present on the host at /usr/sbin/iptables-save
2020-11-02T14:27:16.4947046Z [cluster2] ++ echo /usr/sbin
2020-11-02T14:27:16.4947396Z [cluster2] ++ return
2020-11-02T14:27:16.4947742Z [cluster2] + location=/usr/sbin
2020-11-02T14:27:16.4948330Z [cluster2] + '[' /usr/sbin '!=' unknown ']'
2020-11-02T14:27:16.4949096Z [cluster2] + echo 'iptables-save is present on the host at /usr/sbin/iptables-save'
2020-11-02T14:27:16.4949985Z [cluster2] + sed 's!@@PATH@@!/usr/sbin!' /usr/sbin/iptables-wrapper.in
2020-11-02T14:27:16.4950875Z [cluster2] + for f in iptables-save iptables
2020-11-02T14:27:16.4951560Z [cluster2] ++ find_iptables_on_host iptables
2020-11-02T14:27:16.4952236Z [cluster2] ++ chroot /host test -x /usr/sbin/iptables
2020-11-02T14:27:16.4952643Z [cluster2] ++ echo /usr/sbin
2020-11-02T14:27:16.4952960Z [cluster2] ++ return
2020-11-02T14:27:16.4953294Z [cluster2] + location=/usr/sbin
2020-11-02T14:27:16.4953821Z [cluster2] + '[' /usr/sbin '!=' unknown ']'
2020-11-02T14:27:16.4954482Z [cluster2] + echo 'iptables is present on the host at /usr/sbin/iptables'
2020-11-02T14:27:16.4955269Z [cluster2] + sed 's!@@PATH@@!/usr/sbin!' /usr/sbin/iptables-wrapper.in
2020-11-02T14:27:16.4956124Z [cluster2] + exec submariner-route-agent -v=2 -alsologtostderr
2020-11-02T14:27:16.4956784Z [cluster2] iptables is present on the host at /usr/sbin/iptables
2020-11-02T14:27:16.4957605Z [cluster2] I1102 14:17:49.211874       1 main.go:47] Starting submariner-route-agent
2020-11-02T14:27:16.4967130Z [cluster2] W1102 14:17:49.216923       1 client_config.go:543] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
2020-11-02T14:27:16.4968628Z [cluster2] I1102 14:17:49.223870       1 route.go:192] Starting Route Controller. ClusterID: cluster2, localClusterCIDR: [10.2.0.0/16], localServiceCIDR: [100.2.0.0/16]
2020-11-02T14:27:16.4974821Z [cluster2] I1102 14:17:49.241832       1 route.go:196] Waiting for Endpoint informer caches to sync.
2020-11-02T14:27:16.4976107Z [cluster2] I1102 14:17:49.312949       1 route.go:656] Enqueueing sm-route-agent-pod event, ip: 172.17.0.9
2020-11-02T14:27:16.4978733Z [cluster2] I1102 14:17:49.312990       1 route.go:656] Enqueueing sm-route-agent-pod event, ip: 172.17.0.8
2020-11-02T14:27:16.4979779Z [cluster2] I1102 14:17:49.348776       1 driver.go:37] Interface "lo" has "127.0.0.1" address
2020-11-02T14:27:16.4980488Z [cluster2] I1102 14:17:49.348878       1 driver.go:37] Interface "weave" has "10.2.192.0" address
2020-11-02T14:27:16.4981278Z [cluster2] I1102 14:17:49.348886       1 driver.go:42] Found CNI Interface "weave" that has IP "10.2.192.0" from ClusterCIDR "10.2.0.0/16"
2020-11-02T14:27:16.4983420Z [cluster2] I1102 14:17:49.348966       1 driver.go:60] Successfully configured rp_filter to loose mode(2) on cniInterface "weave"
2020-11-02T14:27:16.4984832Z [cluster2] I1102 14:17:49.355200       1 iptables.go:22] Install/ensure SUBMARINER-POSTROUTING chain exists
2020-11-02T14:27:16.4988627Z [cluster2] I1102 14:17:49.369220       1 iptables.go:28] Insert SUBMARINER-POSTROUTING rule that has rules for inter-cluster traffic
2020-11-02T14:27:16.4990152Z [cluster2] I1102 14:17:49.396025       1 iptables.go:34] Install/ensure SUBMARINER-INPUT chain exists
2020-11-02T14:27:16.4993727Z [cluster2] I1102 14:17:49.625219       1 route.go:656] Enqueueing sm-route-agent-pod event, ip: 172.17.0.9
2020-11-02T14:27:16.4994847Z [cluster2] I1102 14:17:49.689869       1 iptables.go:45] Allow VxLAN incoming traffic in SUBMARINER-INPUT Chain
2020-11-02T14:27:16.4996477Z [cluster2] I1102 14:17:49.738890       1 iptables.go:53] Insert rule to allow traffic over vx-submariner interface in FORWARDing Chain
2020-11-02T14:27:16.4997831Z [cluster2] I1102 14:17:49.763471       1 iptables.go:65] Installing rule for host network to remote cluster communication: -s 240.0.0.0/8 -o vx-submariner -j SNAT --to 10.2.192.0
2020-11-02T14:27:16.4999045Z [cluster2] I1102 14:17:49.783677       1 iptables.go:83] Installing iptables rule for outgoing traffic: -s 10.2.0.0/16 -d 100.1.0.0/16 -j ACCEPT
2020-11-02T14:27:16.5000120Z [cluster2] I1102 14:17:49.791529       1 iptables.go:91] Installing iptables rule for incoming traffic: -s 100.1.0.0/16 -d 10.2.0.0/16 -j ACCEPT
2020-11-02T14:27:16.5001210Z [cluster2] I1102 14:17:49.816153       1 iptables.go:83] Installing iptables rule for outgoing traffic: -s 10.2.0.0/16 -d 10.1.0.0/16 -j ACCEPT
2020-11-02T14:27:16.5002426Z [cluster2] I1102 14:17:49.829852       1 iptables.go:91] Installing iptables rule for incoming traffic: -s 10.1.0.0/16 -d 10.2.0.0/16 -j ACCEPT
2020-11-02T14:27:16.5003445Z [cluster2] I1102 14:17:49.875885       1 route.go:254] In cluster2, podIP of submariner-route-agent[0] is 172.17.0.8
2020-11-02T14:27:16.5004481Z [cluster2] I1102 14:17:49.875914       1 route.go:254] In cluster2, podIP of submariner-route-agent[1] is 172.17.0.9
2020-11-02T14:27:16.5005626Z [cluster2] I1102 14:17:49.875920       1 route.go:268] Hostname is "cluster2-worker" and routeAgentNodeName is "cluster2-worker"
2020-11-02T14:27:16.5006815Z [cluster2] I1102 14:17:49.889457       1 driver.go:87] Successfully annotated node "cluster2-worker" with cniIfaceIP "10.2.192.0"
2020-11-02T14:27:16.5007530Z [cluster2] I1102 14:17:49.889490       1 route.go:286] Route agent workers started
2020-11-02T14:27:16.5008728Z [cluster2] I1102 14:17:49.889516       1 route.go:507] Processing submariner-route-agent pod "submariner-routeagent-ct9kx" with host IP "172.17.0.9", pod IP "172.17.0.9"
2020-11-02T14:27:16.5010591Z [cluster2] I1102 14:17:49.889543       1 route.go:531] Successfully processed submariner-route-agent pod "submariner-routeagent-ct9kx"
2020-11-02T14:27:16.5012160Z [cluster2] I1102 14:17:49.889551       1 route.go:507] Processing submariner-route-agent pod "submariner-routeagent-4w68n" with host IP "172.17.0.8", pod IP "172.17.0.8"
2020-11-02T14:27:16.5013801Z [cluster2] I1102 14:17:49.889556       1 route.go:531] Successfully processed submariner-route-agent pod "submariner-routeagent-4w68n"
2020-11-02T14:27:16.5015567Z [cluster2] I1102 14:17:49.889571       1 route.go:570] Processing local Endpoint cluster2-submariner-cable-cluster2-172-17-0-9 with IP 172.17.0.9, Host cluster2-worker for this cluster
2020-11-02T14:27:16.5016807Z [cluster2] I1102 14:17:49.889783       1 route.go:756] Processing route {Ifindex: 18 Dst: <nil> Src: <nil> Gw: 172.17.0.1 Flags: [] Table: 254}
2020-11-02T14:27:16.5017371Z [cluster2] I1102 14:17:49.889853       1 route.go:759] Found nil gw or dst
2020-11-02T14:27:16.5018044Z [cluster2] I1102 14:17:49.889858       1 route.go:756] Processing route {Ifindex: 4 Dst: 10.2.0.0/16 Src: 10.2.192.0 Gw: <nil> Flags: [] Table: 254}
2020-11-02T14:27:16.5018603Z [cluster2] I1102 14:17:49.889869       1 route.go:759] Found nil gw or dst
2020-11-02T14:27:16.5019533Z [cluster2] I1102 14:17:49.889873       1 route.go:756] Processing route {Ifindex: 18 Dst: 172.17.0.0/16 Src: 172.17.0.9 Gw: <nil> Flags: [] Table: 254}
2020-11-02T14:27:16.5020130Z [cluster2] I1102 14:17:49.889886       1 route.go:759] Found nil gw or dst
2020-11-02T14:27:16.5020851Z [cluster2] I1102 14:17:49.889894       1 route.go:582] This route agent is running on the active gateway node
2020-11-02T14:27:16.5032005Z [cluster2] I1102 14:17:49.889901       1 route.go:590] Creating the vxlan interface: vx-submariner on the gateway node
2020-11-02T14:27:16.5032988Z [cluster2] I1102 14:17:49.890539       1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.8 00:00:00:00:00:00
2020-11-02T14:27:16.5033636Z [cluster2] I1102 14:17:49.890601       1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.9 00:00:00:00:00:00
2020-11-02T14:27:16.5034631Z [cluster2] I1102 14:17:49.891578       1 route.go:464] Successfully configured rp_filter to loose mode(2) on vx-submariner
2020-11-02T14:27:16.5036340Z [cluster2] I1102 14:17:49.892170       1 route.go:558] Updating iptable rules for remote cluster's Endpoint "cluster1-submariner-cable-cluster1-172-17-0-5" with subnets [100.1.0.0/16 10.1.0.0/16]
2020-11-02T14:27:16.5039761Z [cluster2] I1102 14:18:19.250969       1 route.go:570] Processing local Endpoint cluster2-submariner-cable-cluster2-172-17-0-9 with IP 172.17.0.9, Host cluster2-worker for this cluster
2020-11-02T14:27:16.5041121Z [cluster2] I1102 14:18:19.251363       1 route.go:756] Processing route {Ifindex: 13 Dst: 240.0.0.0/8 Src: 240.17.0.9 Gw: <nil> Flags: [] Table: 254}
2020-11-02T14:27:16.5041723Z [cluster2] I1102 14:18:19.251386       1 route.go:759] Found nil gw or dst
2020-11-02T14:27:16.5042596Z [cluster2] I1102 14:18:19.251412       1 route.go:582] This route agent is running on the active gateway node
2020-11-02T14:27:16.5043649Z [cluster2] I1102 14:18:19.251420       1 route.go:590] Creating the vxlan interface: vx-submariner on the gateway node
2020-11-02T14:27:16.5044426Z [cluster2] I1102 14:18:19.251613       1 vxlan.go:61] VxLAN interface already exists with same configuration.
2020-11-02T14:27:16.5045889Z [cluster2] I1102 14:18:19.251673       1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.8 00:00:00:00:00:00
2020-11-02T14:27:16.5046551Z [cluster2] I1102 14:18:19.251705       1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.9 00:00:00:00:00:00
2020-11-02T14:27:16.5047605Z [cluster2] I1102 14:18:19.251824       1 route.go:464] Successfully configured rp_filter to loose mode(2) on vx-submariner
2020-11-02T14:27:16.5049205Z [cluster2] I1102 14:18:19.251934       1 route.go:558] Updating iptable rules for remote cluster's Endpoint "cluster1-submariner-cable-cluster1-172-17-0-5" with subnets [100.1.0.0/16 10.1.0.0/16]
2020-11-02T14:27:16.5051517Z [cluster2] I1102 14:18:49.251324       1 route.go:570] Processing local Endpoint cluster2-submariner-cable-cluster2-172-17-0-9 with IP 172.17.0.9, Host cluster2-worker for this cluster
2020-11-02T14:27:16.5054928Z [cluster2] I1102 14:18:49.251580       1 route.go:756] Processing route {Ifindex: 13 Dst: 240.0.0.0/8 Src: 240.17.0.9 Gw: <nil> Flags: [] Table: 254}
2020-11-02T14:27:16.5055535Z [cluster2] I1102 14:18:49.251616       1 route.go:759] Found nil gw or dst
2020-11-02T14:27:16.5056070Z [cluster2] I1102 14:18:49.251625       1 route.go:582] This route agent is running on the active gateway node
2020-11-02T14:27:16.5057111Z [cluster2] I1102 14:18:49.251632       1 route.go:590] Creating the vxlan interface: vx-submariner on the gateway node
2020-11-02T14:27:16.5059426Z [cluster2] I1102 14:18:49.251785       1 vxlan.go:61] VxLAN interface already exists with same configuration.
2020-11-02T14:27:16.5060171Z [cluster2] I1102 14:18:49.251835       1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.8 00:00:00:00:00:00
2020-11-02T14:27:16.5060842Z [cluster2] I1102 14:18:49.251880       1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.9 00:00:00:00:00:00
2020-11-02T14:27:16.5061957Z [cluster2] I1102 14:18:49.251963       1 route.go:464] Successfully configured rp_filter to loose mode(2) on vx-submariner
2020-11-02T14:27:16.5063866Z [cluster2] I1102 14:18:49.252052       1 route.go:558] Updating iptable rules for remote cluster's Endpoint "cluster1-submariner-cable-cluster1-172-17-0-5" with subnets [100.1.0.0/16 10.1.0.0/16]
2020-11-02T14:27:16.5065875Z [cluster2] I1102 14:19:19.251384       1 route.go:570] Processing local Endpoint cluster2-submariner-cable-cluster2-172-17-0-9 with IP 172.17.0.9, Host cluster2-worker for this cluster
2020-11-02T14:27:16.5067335Z [cluster2] I1102 14:19:19.251700       1 route.go:756] Processing route {Ifindex: 13 Dst: 240.0.0.0/8 Src: 240.17.0.9 Gw: <nil> Flags: [] Table: 254}
2020-11-02T14:27:16.5067895Z [cluster2] I1102 14:19:19.251771       1 route.go:759] Found nil gw or dst
2020-11-02T14:27:16.5068430Z [cluster2] I1102 14:19:19.251779       1 route.go:582] This route agent is running on the active gateway node
2020-11-02T14:27:16.5069386Z [cluster2] I1102 14:19:19.251786       1 route.go:590] Creating the vxlan interface: vx-submariner on the gateway node
2020-11-02T14:27:16.5070132Z [cluster2] I1102 14:19:19.251936       1 vxlan.go:61] VxLAN interface already exists with same configuration.
2020-11-02T14:27:16.5070836Z [cluster2] I1102 14:19:19.251984       1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.8 00:00:00:00:00:00
2020-11-02T14:27:16.5071823Z [cluster2] I1102 14:19:19.252044       1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.9 00:00:00:00:00:00
2020-11-02T14:27:16.5073773Z [cluster2] I1102 14:19:19.252106       1 route.go:464] Successfully configured rp_filter to loose mode(2) on vx-submariner
2020-11-02T14:27:16.5076608Z [cluster2] I1102 14:19:19.252201       1 route.go:558] Updating iptable rules for remote cluster's Endpoint "cluster1-submariner-cable-cluster1-172-17-0-5" with subnets [100.1.0.0/16 10.1.0.0/16]
2020-11-02T14:27:16.5078903Z [cluster2] I1102 14:19:49.251387       1 route.go:570] Processing local Endpoint cluster2-submariner-cable-cluster2-172-17-0-9 with IP 172.17.0.9, Host cluster2-worker for this cluster
2020-11-02T14:27:16.5080366Z [cluster2] I1102 14:19:49.251608       1 route.go:756] Processing route {Ifindex: 13 Dst: 240.0.0.0/8 Src: 240.17.0.9 Gw: <nil> Flags: [] Table: 254}
2020-11-02T14:27:16.5081102Z [cluster2] I1102 14:19:49.251662       1 route.go:759] Found nil gw or dst
2020-11-02T14:27:16.5081643Z [cluster2] I1102 14:19:49.251669       1 route.go:582] This route agent is running on the active gateway node
2020-11-02T14:27:16.5082572Z [cluster2] I1102 14:19:49.251676       1 route.go:590] Creating the vxlan interface: vx-submariner on the gateway node
2020-11-02T14:27:16.5083315Z [cluster2] I1102 14:19:49.251831       1 vxlan.go:61] VxLAN interface already exists with same configuration.
2020-11-02T14:27:16.5083992Z [cluster2] I1102 14:19:49.251867       1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.8 00:00:00:00:00:00
2020-11-02T14:27:16.5085073Z [cluster2] I1102 14:19:49.251896       1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.9 00:00:00:00:00:00
2020-11-02T14:27:16.5086256Z [cluster2] I1102 14:19:49.251980       1 route.go:464] Successfully configured rp_filter to loose mode(2) on vx-submariner
2020-11-02T14:27:16.5088289Z [cluster2] I1102 14:19:49.252055       1 route.go:558] Updating iptable rules for remote cluster's Endpoint "cluster1-submariner-cable-cluster1-172-17-0-5" with subnets [100.1.0.0/16 10.1.0.0/16]
2020-11-02T14:27:16.5090728Z [cluster2] I1102 14:20:19.251561       1 route.go:570] Processing local Endpoint cluster2-submariner-cable-cluster2-172-17-0-9 with IP 172.17.0.9, Host cluster2-worker for this cluster
2020-11-02T14:27:16.5092052Z [cluster2] I1102 14:20:19.251850       1 route.go:756] Processing route {Ifindex: 13 Dst: 240.0.0.0/8 Src: 240.17.0.9 Gw: <nil> Flags: [] Table: 254}
2020-11-02T14:27:16.5092653Z [cluster2] I1102 14:20:19.251876       1 route.go:759] Found nil gw or dst
2020-11-02T14:27:16.5093376Z [cluster2] I1102 14:20:19.251884       1 route.go:582] This route agent is running on the active gateway node
2020-11-02T14:27:16.5094390Z [cluster2] I1102 14:20:19.251891       1 route.go:590] Creating the vxlan interface: vx-submariner on the gateway node
2020-11-02T14:27:16.5095499Z [cluster2] I1102 14:20:19.252024       1 vxlan.go:61] VxLAN interface already exists with same configuration.
2020-11-02T14:27:16.5096336Z [cluster2] I1102 14:20:19.252060       1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.8 00:00:00:00:00:00
2020-11-02T14:27:16.5097021Z [cluster2] I1102 14:20:19.252090       1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.9 00:00:00:00:00:00
2020-11-02T14:27:16.5098431Z [cluster2] I1102 14:20:19.252134       1 route.go:464] Successfully configured rp_filter to loose mode(2) on vx-submariner
2020-11-02T14:27:16.5100241Z [cluster2] I1102 14:20:19.252206       1 route.go:558] Updating iptable rules for remote cluster's Endpoint "cluster1-submariner-cable-cluster1-172-17-0-5" with subnets [100.1.0.0/16 10.1.0.0/16]
2020-11-02T14:27:16.5102570Z [cluster2] I1102 14:20:49.251652       1 route.go:570] Processing local Endpoint cluster2-submariner-cable-cluster2-172-17-0-9 with IP 172.17.0.9, Host cluster2-worker for this cluster
2020-11-02T14:27:16.5103890Z [cluster2] I1102 14:20:49.251848       1 route.go:756] Processing route {Ifindex: 13 Dst: 240.0.0.0/8 Src: 240.17.0.9 Gw: <nil> Flags: [] Table: 254}
2020-11-02T14:27:16.5104660Z [cluster2] I1102 14:20:49.251870       1 route.go:759] Found nil gw or dst
2020-11-02T14:27:16.5105360Z [cluster2] I1102 14:20:49.251877       1 route.go:582] This route agent is running on the active gateway node
2020-11-02T14:27:16.5106276Z [cluster2] I1102 14:20:49.251884       1 route.go:590] Creating the vxlan interface: vx-submariner on the gateway node
2020-11-02T14:27:16.5107020Z [cluster2] I1102 14:20:49.252050       1 vxlan.go:61] VxLAN interface already exists with same configuration.
2020-11-02T14:27:16.5107695Z [cluster2] I1102 14:20:49.252086       1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.9 00:00:00:00:00:00
2020-11-02T14:27:16.5108315Z [cluster2] I1102 14:20:49.252116       1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.8 00:00:00:00:00:00
2020-11-02T14:27:16.5109230Z [cluster2] I1102 14:20:49.252159       1 route.go:464] Successfully configured rp_filter to loose mode(2) on vx-submariner
2020-11-02T14:27:16.5110773Z [cluster2] I1102 14:20:49.252218       1 route.go:558] Updating iptable rules for remote cluster's Endpoint "cluster1-submariner-cable-cluster1-172-17-0-5" with subnets [100.1.0.0/16 10.1.0.0/16]
2020-11-02T14:27:16.5112992Z [cluster2] I1102 14:21:19.252515       1 route.go:570] Processing local Endpoint cluster2-submariner-cable-cluster2-172-17-0-9 with IP 172.17.0.9, Host cluster2-worker for this cluster
2020-11-02T14:27:16.5114227Z [cluster2] I1102 14:21:19.252709       1 route.go:756] Processing route {Ifindex: 13 Dst: 240.0.0.0/8 Src: 240.17.0.9 Gw: <nil> Flags: [] Table: 254}
2020-11-02T14:27:16.5114784Z [cluster2] I1102 14:21:19.252743       1 route.go:759] Found nil gw or dst
2020-11-02T14:27:16.5115316Z [cluster2] I1102 14:21:19.252751       1 route.go:582] This route agent is running on the active gateway node
2020-11-02T14:27:16.5116235Z [cluster2] I1102 14:21:19.252759       1 route.go:590] Creating the vxlan interface: vx-submariner on the gateway node
2020-11-02T14:27:16.5116985Z [cluster2] I1102 14:21:19.252904       1 vxlan.go:61] VxLAN interface already exists with same configuration.
2020-11-02T14:27:16.5117664Z [cluster2] I1102 14:21:19.252955       1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.8 00:00:00:00:00:00
2020-11-02T14:27:16.5118283Z [cluster2] I1102 14:21:19.253021       1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.9 00:00:00:00:00:00
2020-11-02T14:27:16.5119588Z [cluster2] I1102 14:21:19.253084       1 route.go:464] Successfully configured rp_filter to loose mode(2) on vx-submariner
2020-11-02T14:27:16.5121230Z [cluster2] I1102 14:21:19.253152       1 route.go:558] Updating iptable rules for remote cluster's Endpoint "cluster1-submariner-cable-cluster1-172-17-0-5" with subnets [100.1.0.0/16 10.1.0.0/16]
2020-11-02T14:27:16.5123486Z [cluster2] I1102 14:21:49.252737       1 route.go:570] Processing local Endpoint cluster2-submariner-cable-cluster2-172-17-0-9 with IP 172.17.0.9, Host cluster2-worker for this cluster
2020-11-02T14:27:16.5124891Z [cluster2] I1102 14:21:49.252971       1 route.go:756] Processing route {Ifindex: 13 Dst: 240.0.0.0/8 Src: 240.17.0.9 Gw: <nil> Flags: [] Table: 254}
2020-11-02T14:27:16.5125535Z [cluster2] I1102 14:21:49.253003       1 route.go:759] Found nil gw or dst
2020-11-02T14:27:16.5126094Z [cluster2] I1102 14:21:49.253011       1 route.go:582] This route agent is running on the active gateway node
2020-11-02T14:27:16.5127196Z [cluster2] I1102 14:21:49.253018       1 route.go:590] Creating the vxlan interface: vx-submariner on the gateway node
2020-11-02T14:27:16.5127967Z [cluster2] I1102 14:21:49.253176       1 vxlan.go:61] VxLAN interface already exists with same configuration.
2020-11-02T14:27:16.5128668Z [cluster2] I1102 14:21:49.253241       1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.8 00:00:00:00:00:00
2020-11-02T14:27:16.5129308Z [cluster2] I1102 14:21:49.253282       1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.9 00:00:00:00:00:00
2020-11-02T14:27:16.5130279Z [cluster2] I1102 14:21:49.253340       1 route.go:464] Successfully configured rp_filter to loose mode(2) on vx-submariner
2020-11-02T14:27:16.5131867Z [cluster2] I1102 14:21:49.253428       1 route.go:558] Updating iptable rules for remote cluster's Endpoint "cluster1-submariner-cable-cluster1-172-17-0-5" with subnets [100.1.0.0/16 10.1.0.0/16]
2020-11-02T14:27:16.5134662Z [cluster2] I1102 14:22:19.252948       1 route.go:570] Processing local Endpoint cluster2-submariner-cable-cluster2-172-17-0-9 with IP 172.17.0.9, Host cluster2-worker for this cluster
2020-11-02T14:27:16.5135893Z [cluster2] I1102 14:22:19.253241       1 route.go:756] Processing route {Ifindex: 13 Dst: 240.0.0.0/8 Src: 240.17.0.9 Gw: <nil> Flags: [] Table: 254}
2020-11-02T14:27:16.5136446Z [cluster2] I1102 14:22:19.253266       1 route.go:759] Found nil gw or dst
2020-11-02T14:27:16.5136979Z [cluster2] I1102 14:22:19.253274       1 route.go:582] This route agent is running on the active gateway node
2020-11-02T14:27:16.5137897Z [cluster2] I1102 14:22:19.253290       1 route.go:590] Creating the vxlan interface: vx-submariner on the gateway node
2020-11-02T14:27:16.5138653Z [cluster2] I1102 14:22:19.253423       1 vxlan.go:61] VxLAN interface already exists with same configuration.
2020-11-02T14:27:16.5139734Z [cluster2] I1102 14:22:19.253461       1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.8 00:00:00:00:00:00
2020-11-02T14:27:16.5140353Z [cluster2] I1102 14:22:19.253493       1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.9 00:00:00:00:00:00
2020-11-02T14:27:16.5141661Z [cluster2] I1102 14:22:19.253559       1 route.go:464] Successfully configured rp_filter to loose mode(2) on vx-submariner
2020-11-02T14:27:16.5143286Z [cluster2] I1102 14:22:19.253624       1 route.go:558] Updating iptable rules for remote cluster's Endpoint "cluster1-submariner-cable-cluster1-172-17-0-5" with subnets [100.1.0.0/16 10.1.0.0/16]
2020-11-02T14:27:16.5145732Z [cluster2] I1102 14:22:49.253235       1 route.go:570] Processing local Endpoint cluster2-submariner-cable-cluster2-172-17-0-9 with IP 172.17.0.9, Host cluster2-worker for this cluster
2020-11-02T14:27:16.5146966Z [cluster2] I1102 14:22:49.253474       1 route.go:756] Processing route {Ifindex: 13 Dst: 240.0.0.0/8 Src: 240.17.0.9 Gw: <nil> Flags: [] Table: 254}
2020-11-02T14:27:16.5147524Z [cluster2] I1102 14:22:49.253504       1 route.go:759] Found nil gw or dst
2020-11-02T14:27:16.5148155Z [cluster2] I1102 14:22:49.253512       1 route.go:582] This route agent is running on the active gateway node
2020-11-02T14:27:16.5149075Z [cluster2] I1102 14:22:49.253535       1 route.go:590] Creating the vxlan interface: vx-submariner on the gateway node
2020-11-02T14:27:16.5149813Z [cluster2] I1102 14:22:49.253675       1 vxlan.go:61] VxLAN interface already exists with same configuration.
2020-11-02T14:27:16.5150486Z [cluster2] I1102 14:22:49.253710       1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.8 00:00:00:00:00:00
2020-11-02T14:27:16.5151103Z [cluster2] I1102 14:22:49.253741       1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.9 00:00:00:00:00:00
2020-11-02T14:27:16.5152700Z [cluster2] I1102 14:22:49.253788       1 route.go:464] Successfully configured rp_filter to loose mode(2) on vx-submariner
2020-11-02T14:27:16.5154274Z [cluster2] I1102 14:22:49.253849       1 route.go:558] Updating iptable rules for remote cluster's Endpoint "cluster1-submariner-cable-cluster1-172-17-0-5" with subnets [100.1.0.0/16 10.1.0.0/16]
2020-11-02T14:27:16.5156279Z [cluster2] I1102 14:23:19.253487       1 route.go:570] Processing local Endpoint cluster2-submariner-cable-cluster2-172-17-0-9 with IP 172.17.0.9, Host cluster2-worker for this cluster
2020-11-02T14:27:16.5157508Z [cluster2] I1102 14:23:19.253698       1 route.go:756] Processing route {Ifindex: 13 Dst: 240.0.0.0/8 Src: 240.17.0.9 Gw: <nil> Flags: [] Table: 254}
2020-11-02T14:27:16.5158063Z [cluster2] I1102 14:23:19.253731       1 route.go:759] Found nil gw or dst
2020-11-02T14:27:16.5158602Z [cluster2] I1102 14:23:19.253739       1 route.go:582] This route agent is running on the active gateway node
2020-11-02T14:27:16.5159517Z [cluster2] I1102 14:23:19.253746       1 route.go:590] Creating the vxlan interface: vx-submariner on the gateway node
2020-11-02T14:27:16.5160238Z [cluster2] I1102 14:23:19.253888       1 vxlan.go:61] VxLAN interface already exists with same configuration.
2020-11-02T14:27:16.5160911Z [cluster2] I1102 14:23:19.253937       1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.9 00:00:00:00:00:00
2020-11-02T14:27:16.5161532Z [cluster2] I1102 14:23:19.253978       1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.8 00:00:00:00:00:00
2020-11-02T14:27:16.5162639Z [cluster2] I1102 14:23:19.254045       1 route.go:464] Successfully configured rp_filter to loose mode(2) on vx-submariner
2020-11-02T14:27:16.5171048Z [cluster2] I1102 14:23:19.254115       1 route.go:558] Updating iptable rules for remote cluster's Endpoint "cluster1-submariner-cable-cluster1-172-17-0-5" with subnets [100.1.0.0/16 10.1.0.0/16]
2020-11-02T14:27:16.5173620Z [cluster2] I1102 14:23:49.253608       1 route.go:558] Updating iptable rules for remote cluster's Endpoint "cluster1-submariner-cable-cluster1-172-17-0-5" with subnets [100.1.0.0/16 10.1.0.0/16]
2020-11-02T14:27:16.5175859Z [cluster2] I1102 14:23:49.253655       1 route.go:570] Processing local Endpoint cluster2-submariner-cable-cluster2-172-17-0-9 with IP 172.17.0.9, Host cluster2-worker for this cluster
2020-11-02T14:27:16.5177534Z [cluster2] I1102 14:23:49.253901       1 route.go:756] Processing route {Ifindex: 13 Dst: 240.0.0.0/8 Src: 240.17.0.9 Gw: <nil> Flags: [] Table: 254}
2020-11-02T14:27:16.5178098Z [cluster2] I1102 14:23:49.253933       1 route.go:759] Found nil gw or dst
2020-11-02T14:27:16.5178636Z [cluster2] I1102 14:23:49.253941       1 route.go:582] This route agent is running on the active gateway node
2020-11-02T14:27:16.5181099Z [cluster2] I1102 14:23:49.253948       1 route.go:590] Creating the vxlan interface: vx-submariner on the gateway node
2020-11-02T14:27:16.5182091Z [cluster2] I1102 14:23:49.254101       1 vxlan.go:61] VxLAN interface already exists with same configuration.
2020-11-02T14:27:16.5182828Z [cluster2] I1102 14:23:49.254149       1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.8 00:00:00:00:00:00
2020-11-02T14:27:16.5183504Z [cluster2] I1102 14:23:49.254187       1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.9 00:00:00:00:00:00
2020-11-02T14:27:16.5184858Z [cluster2] I1102 14:23:49.254245       1 route.go:464] Successfully configured rp_filter to loose mode(2) on vx-submariner
2020-11-02T14:27:16.5186418Z [cluster2] I1102 14:24:19.253684       1 route.go:570] Processing local Endpoint cluster2-submariner-cable-cluster2-172-17-0-9 with IP 172.17.0.9, Host cluster2-worker for this cluster
2020-11-02T14:27:16.5187649Z [cluster2] I1102 14:24:19.253991       1 route.go:756] Processing route {Ifindex: 13 Dst: 240.0.0.0/8 Src: 240.17.0.9 Gw: <nil> Flags: [] Table: 254}
2020-11-02T14:27:16.5188509Z [cluster2] I1102 14:24:19.254030       1 route.go:759] Found nil gw or dst
2020-11-02T14:27:16.5189122Z [cluster2] I1102 14:24:19.254039       1 route.go:582] This route agent is running on the active gateway node
2020-11-02T14:27:16.5190107Z [cluster2] I1102 14:24:19.254046       1 route.go:590] Creating the vxlan interface: vx-submariner on the gateway node
2020-11-02T14:27:16.5191074Z [cluster2] I1102 14:24:19.254199       1 vxlan.go:61] VxLAN interface already exists with same configuration.
2020-11-02T14:27:16.5192132Z [cluster2] I1102 14:24:19.254261       1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.8 00:00:00:00:00:00
2020-11-02T14:27:16.5192757Z [cluster2] I1102 14:24:19.254325       1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.9 00:00:00:00:00:00
2020-11-02T14:27:16.5193717Z [cluster2] I1102 14:24:19.254385       1 route.go:464] Successfully configured rp_filter to loose mode(2) on vx-submariner
2020-11-02T14:27:16.5195276Z [cluster2] I1102 14:24:19.254459       1 route.go:558] Updating iptable rules for remote cluster's Endpoint "cluster1-submariner-cable-cluster1-172-17-0-5" with subnets [100.1.0.0/16 10.1.0.0/16]
2020-11-02T14:27:16.5197291Z [cluster2] I1102 14:24:49.253874       1 route.go:570] Processing local Endpoint cluster2-submariner-cable-cluster2-172-17-0-9 with IP 172.17.0.9, Host cluster2-worker for this cluster
2020-11-02T14:27:16.5198515Z [cluster2] I1102 14:24:49.254116       1 route.go:756] Processing route {Ifindex: 13 Dst: 240.0.0.0/8 Src: 240.17.0.9 Gw: <nil> Flags: [] Table: 254}
2020-11-02T14:27:16.5199071Z [cluster2] I1102 14:24:49.254146       1 route.go:759] Found nil gw or dst
2020-11-02T14:27:16.5199605Z [cluster2] I1102 14:24:49.254154       1 route.go:582] This route agent is running on the active gateway node
2020-11-02T14:27:16.5200523Z [cluster2] I1102 14:24:49.254162       1 route.go:590] Creating the vxlan interface: vx-submariner on the gateway node
2020-11-02T14:27:16.5201264Z [cluster2] I1102 14:24:49.254303       1 vxlan.go:61] VxLAN interface already exists with same configuration.
2020-11-02T14:27:16.5201933Z [cluster2] I1102 14:24:49.254338       1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.8 00:00:00:00:00:00
2020-11-02T14:27:16.5202553Z [cluster2] I1102 14:24:49.254368       1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.9 00:00:00:00:00:00
2020-11-02T14:27:16.5203488Z [cluster2] I1102 14:24:49.254421       1 route.go:464] Successfully configured rp_filter to loose mode(2) on vx-submariner
2020-11-02T14:27:16.5205021Z [cluster2] I1102 14:24:49.254485       1 route.go:558] Updating iptable rules for remote cluster's Endpoint "cluster1-submariner-cable-cluster1-172-17-0-5" with subnets [100.1.0.0/16 10.1.0.0/16]
2020-11-02T14:27:16.5207025Z [cluster2] I1102 14:25:19.254098       1 route.go:570] Processing local Endpoint cluster2-submariner-cable-cluster2-172-17-0-9 with IP 172.17.0.9, Host cluster2-worker for this cluster
2020-11-02T14:27:16.5208250Z [cluster2] I1102 14:25:19.254346       1 route.go:756] Processing route {Ifindex: 13 Dst: 240.0.0.0/8 Src: 240.17.0.9 Gw: <nil> Flags: [] Table: 254}
2020-11-02T14:27:16.5208813Z [cluster2] I1102 14:25:19.254381       1 route.go:759] Found nil gw or dst
2020-11-02T14:27:16.5209351Z [cluster2] I1102 14:25:19.254390       1 route.go:582] This route agent is running on the active gateway node
2020-11-02T14:27:16.5210427Z [cluster2] I1102 14:25:19.254398       1 route.go:590] Creating the vxlan interface: vx-submariner on the gateway node
2020-11-02T14:27:16.5211261Z [cluster2] I1102 14:25:19.254574       1 vxlan.go:61] VxLAN interface already exists with same configuration.
2020-11-02T14:27:16.5211920Z [cluster2] I1102 14:25:19.254620       1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.8 00:00:00:00:00:00
2020-11-02T14:27:16.5212517Z [cluster2] I1102 14:25:19.254663       1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.9 00:00:00:00:00:00
2020-11-02T14:27:16.5216060Z [cluster2] I1102 14:25:19.254733       1 route.go:464] Successfully configured rp_filter to loose mode(2) on vx-submariner
2020-11-02T14:27:16.5220972Z [cluster2] I1102 14:25:19.254823       1 route.go:558] Updating iptable rules for remote cluster's Endpoint "cluster1-submariner-cable-cluster1-172-17-0-5" with subnets [100.1.0.0/16 10.1.0.0/16]
2020-11-02T14:27:16.5225191Z [cluster2] I1102 14:25:49.254316       1 route.go:570] Processing local Endpoint cluster2-submariner-cable-cluster2-172-17-0-9 with IP 172.17.0.9, Host cluster2-worker for this cluster
2020-11-02T14:27:16.5226671Z [cluster2] I1102 14:25:49.254550       1 route.go:756] Processing route {Ifindex: 13 Dst: 240.0.0.0/8 Src: 240.17.0.9 Gw: <nil> Flags: [] Table: 254}
2020-11-02T14:27:16.5227408Z [cluster2] I1102 14:25:49.254585       1 route.go:759] Found nil gw or dst
2020-11-02T14:27:16.5227930Z [cluster2] I1102 14:25:49.254593       1 route.go:582] This route agent is running on the active gateway node
2020-11-02T14:27:16.5228860Z [cluster2] I1102 14:25:49.254601       1 route.go:590] Creating the vxlan interface: vx-submariner on the gateway node
2020-11-02T14:27:16.5229620Z [cluster2] I1102 14:25:49.254728       1 vxlan.go:61] VxLAN interface already exists with same configuration.
2020-11-02T14:27:16.5230300Z [cluster2] I1102 14:25:49.254792       1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.9 00:00:00:00:00:00
2020-11-02T14:27:16.5230924Z [cluster2] I1102 14:25:49.254828       1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.8 00:00:00:00:00:00
2020-11-02T14:27:16.5232104Z [cluster2] I1102 14:25:49.254890       1 route.go:464] Successfully configured rp_filter to loose mode(2) on vx-submariner
2020-11-02T14:27:16.5233645Z [cluster2] I1102 14:25:49.254968       1 route.go:558] Updating iptable rules for remote cluster's Endpoint "cluster1-submariner-cable-cluster1-172-17-0-5" with subnets [100.1.0.0/16 10.1.0.0/16]
2020-11-02T14:27:16.5235878Z [cluster2] I1102 14:26:19.254488       1 route.go:570] Processing local Endpoint cluster2-submariner-cable-cluster2-172-17-0-9 with IP 172.17.0.9, Host cluster2-worker for this cluster
2020-11-02T14:27:16.5237709Z [cluster2] I1102 14:26:19.254728       1 route.go:756] Processing route {Ifindex: 13 Dst: 240.0.0.0/8 Src: 240.17.0.9 Gw: <nil> Flags: [] Table: 254}
2020-11-02T14:27:16.5238545Z [cluster2] I1102 14:26:19.254759       1 route.go:759] Found nil gw or dst
2020-11-02T14:27:16.5239098Z [cluster2] I1102 14:26:19.254767       1 route.go:582] This route agent is running on the active gateway node
2020-11-02T14:27:16.5240253Z [cluster2] I1102 14:26:19.254775       1 route.go:590] Creating the vxlan interface: vx-submariner on the gateway node
2020-11-02T14:27:16.5241052Z [cluster2] I1102 14:26:19.254922       1 vxlan.go:61] VxLAN interface already exists with same configuration.
2020-11-02T14:27:16.5241928Z [cluster2] I1102 14:26:19.254971       1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.8 00:00:00:00:00:00
2020-11-02T14:27:16.5242553Z [cluster2] I1102 14:26:19.255011       1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.9 00:00:00:00:00:00
2020-11-02T14:27:16.5243525Z [cluster2] I1102 14:26:19.255074       1 route.go:464] Successfully configured rp_filter to loose mode(2) on vx-submariner
2020-11-02T14:27:16.5245112Z [cluster2] I1102 14:26:19.255193       1 route.go:558] Updating iptable rules for remote cluster's Endpoint "cluster1-submariner-cable-cluster1-172-17-0-5" with subnets [100.1.0.0/16 10.1.0.0/16]
2020-11-02T14:27:16.5247188Z [cluster2] I1102 14:26:49.254581       1 route.go:570] Processing local Endpoint cluster2-submariner-cable-cluster2-172-17-0-9 with IP 172.17.0.9, Host cluster2-worker for this cluster
2020-11-02T14:27:16.5248458Z [cluster2] I1102 14:26:49.254807       1 route.go:756] Processing route {Ifindex: 13 Dst: 240.0.0.0/8 Src: 240.17.0.9 Gw: <nil> Flags: [] Table: 254}
2020-11-02T14:27:16.5249033Z [cluster2] I1102 14:26:49.254841       1 route.go:759] Found nil gw or dst
2020-11-02T14:27:16.5249585Z [cluster2] I1102 14:26:49.254849       1 route.go:582] This route agent is running on the active gateway node
2020-11-02T14:27:16.5251048Z [cluster2] I1102 14:26:49.254856       1 route.go:590] Creating the vxlan interface: vx-submariner on the gateway node
2020-11-02T14:27:16.5251862Z [cluster2] I1102 14:26:49.254994       1 vxlan.go:61] VxLAN interface already exists with same configuration.
2020-11-02T14:27:16.5252590Z [cluster2] I1102 14:26:49.255057       1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.8 00:00:00:00:00:00
2020-11-02T14:27:16.5253574Z [lighthouse]$ [cluster2] subctl show all
2020-11-02T14:27:16.7610956Z Error obtaining the Submariner resource: the server could not find the requested resource (get submariners.submariner.io submariner)
2020-11-02T14:27:16.7611742Z
2020-11-02T14:27:16.7612598Z subctl version: v0.7.0-8-g282818e
2020-11-02T14:27:16.7612814Z
2020-11-02T14:27:16.7640714Z [cluster2] I1102 14:26:49.255093       1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.9 00:00:00:00:00:00
2020-11-02T14:27:16.7641941Z [cluster2] I1102 14:26:49.255224       1 route.go:464] Successfully configured rp_filter to loose mode(2) on vx-submariner
2020-11-02T14:27:16.7643569Z [cluster2] I1102 14:26:49.255291       1 route.go:558] Updating iptable rules for remote cluster's Endpoint "cluster1-submariner-cable-cluster1-172-17-0-5" with subnets [100.1.0.0/16 10.1.0.0/16]
2020-11-02T14:27:16.7644630Z [cluster2] ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
2020-11-02T14:27:16.7644931Z [cluster2]
2020-11-02T14:27:16.7645585Z [cluster2] Showing information for cluster "cluster2":
2020-11-02T14:27:16.7646116Z [cluster2]     Discovered network details:
2020-11-02T14:27:16.7646976Z [cluster2]         Network plugin:  weave-net
2020-11-02T14:27:16.7647382Z [cluster2]         Service CIDRs:   [100.2.0.0/16]
2020-11-02T14:27:16.7647885Z [cluster2]         Cluster CIDRs:   [10.2.0.0/16]
2020-11-02T14:27:16.7648193Z [cluster2]
2020-11-02T14:27:16.7648549Z [cluster2] ===================== END Post mortem cluster2 =====================
2020-11-02T14:27:16.7670896Z [lighthouse]$ make post-mortem
2020-11-02T14:27:17.1423635Z Post job cleanup.
2020-11-02T14:27:17.5857164Z [command]/usr/bin/git version
2020-11-02T14:27:17.5942615Z git version 2.29.0
2020-11-02T14:27:17.6009474Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand
2020-11-02T14:27:17.6061181Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :
2020-11-02T14:27:17.6538637Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader
2020-11-02T14:27:17.6578278Z http.https://github.com/.extraheader
2020-11-02T14:27:17.6593911Z [command]/usr/bin/git config --local --unset-all http.https://github.com/.extraheader
2020-11-02T14:27:17.6694026Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :
2020-11-02T14:27:17.7445464Z Cleaning up orphan processes