subm_random_fail_20201008-1

2020-10-08T11:04:25.2009269Z ##[section]Starting: Request a runner to run this job
2020-10-08T11:04:25.7544605Z Can't find any online and idle self-hosted runner in current repository that matches the required labels: 'ubuntu-latest'
2020-10-08T11:04:25.7544738Z Can't find any online and idle self-hosted runner in current repository's account/organization that matches the required labels: 'ubuntu-latest'
2020-10-08T11:04:25.7545309Z Found online and idle hosted runner in current repository's account/organization that matches the required labels: 'ubuntu-latest'
2020-10-08T11:04:25.9023425Z ##[section]Finishing: Request a runner to run this job
2020-10-08T11:04:33.5339491Z Current runner version: '2.273.5'
2020-10-08T11:04:33.5368618Z ##[group]Operating System
2020-10-08T11:04:33.5369407Z Ubuntu
2020-10-08T11:04:33.5369685Z 18.04.5
2020-10-08T11:04:33.5369941Z LTS
2020-10-08T11:04:33.5370205Z ##[endgroup]
2020-10-08T11:04:33.5370565Z ##[group]Virtual Environment
2020-10-08T11:04:33.5371001Z Environment: ubuntu-18.04
2020-10-08T11:04:33.5371363Z Version: 20201004.1
2020-10-08T11:04:33.5372139Z Included Software: https://github.com/actions/virtual-environments/blob/ubuntu18/20201004.1/images/linux/Ubuntu1804-README.md
2020-10-08T11:04:33.5372909Z ##[endgroup]
2020-10-08T11:04:33.5374260Z Prepare workflow directory
2020-10-08T11:04:33.5642628Z Prepare all required actions
2020-10-08T11:04:33.5658294Z Download action repository 'actions/checkout@v2'
2020-10-08T11:04:36.3132579Z ##[group]Run actions/checkout@v2
2020-10-08T11:04:36.3133185Z with:
2020-10-08T11:04:36.3133777Z   repository: submariner-io/submariner
2020-10-08T11:04:36.3134606Z   token: ***
2020-10-08T11:04:36.3134937Z   ssh-strict: true
2020-10-08T11:04:36.3135398Z   persist-credentials: true
2020-10-08T11:04:36.3135795Z   clean: true
2020-10-08T11:04:36.3136122Z   fetch-depth: 1
2020-10-08T11:04:36.3136440Z   lfs: false
2020-10-08T11:04:36.3136750Z   submodules: false
2020-10-08T11:04:36.3137088Z ##[endgroup]
2020-10-08T11:04:38.5864057Z Syncing repository: submariner-io/submariner
2020-10-08T11:04:38.6146182Z ##[group]Getting Git version info
2020-10-08T11:04:38.6147655Z Working directory is '/home/runner/work/submariner/submariner'
2020-10-08T11:04:38.6192833Z [command]/usr/bin/git version
2020-10-08T11:04:38.8308531Z git version 2.28.0
2020-10-08T11:04:38.8336355Z ##[endgroup]
2020-10-08T11:04:38.8343862Z Deleting the contents of '/home/runner/work/submariner/submariner'
2020-10-08T11:04:38.8349608Z ##[group]Initializing the repository
2020-10-08T11:04:38.8353725Z [command]/usr/bin/git init /home/runner/work/submariner/submariner
2020-10-08T11:04:38.8718053Z Initialized empty Git repository in /home/runner/work/submariner/submariner/.git/
2020-10-08T11:04:38.8731228Z [command]/usr/bin/git remote add origin https://github.com/submariner-io/submariner
2020-10-08T11:04:38.8907151Z ##[endgroup]
2020-10-08T11:04:38.8908319Z ##[group]Disabling automatic garbage collection
2020-10-08T11:04:38.8912979Z [command]/usr/bin/git config --local gc.auto 0
2020-10-08T11:04:38.8955250Z ##[endgroup]
2020-10-08T11:04:38.8962673Z ##[group]Setting up auth
2020-10-08T11:04:38.8969526Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand
2020-10-08T11:04:38.9007559Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :
2020-10-08T11:04:39.2145871Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader
2020-10-08T11:04:39.2196065Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :
2020-10-08T11:04:39.2513757Z [command]/usr/bin/git config --local http.https://github.com/.extraheader AUTHORIZATION: basic ***
2020-10-08T11:04:39.2602144Z ##[endgroup]
2020-10-08T11:04:39.2614046Z ##[group]Fetching the repository
2020-10-08T11:04:39.2640480Z [command]/usr/bin/git -c protocol.version=2 fetch --no-tags --prune --progress --no-recurse-submodules --depth=1 origin +a94878ad4fe368d3bb1efd0be1df1b354e805ae3:refs/remotes/pull/865/merge
2020-10-08T11:04:39.6807746Z remote: Enumerating objects: 221, done.
2020-10-08T11:04:39.6809182Z remote: Counting objects:   0% (1/221)
2020-10-08T11:04:39.6826857Z remote: Counting objects:   1% (3/221)
2020-10-08T11:04:39.6828200Z remote: Counting objects:   2% (5/221)
2020-10-08T11:04:39.6833705Z remote: Counting objects:   3% (7/221)
2020-10-08T11:04:39.6838180Z remote: Counting objects:   4% (9/221)
2020-10-08T11:04:39.6842456Z remote: Counting objects:   5% (12/221)
2020-10-08T11:04:39.6847136Z remote: Counting objects:   6% (14/221)
2020-10-08T11:04:39.6848354Z remote: Counting objects:   7% (16/221)
2020-10-08T11:04:39.6849766Z remote: Counting objects:   8% (18/221)
2020-10-08T11:04:39.6850828Z remote: Counting objects:   9% (20/221)
2020-10-08T11:04:39.6852059Z remote: Counting objects:  10% (23/221)
2020-10-08T11:04:39.6853187Z remote: Counting objects:  11% (25/221)
2020-10-08T11:04:39.6854186Z remote: Counting objects:  12% (27/221)
2020-10-08T11:04:39.6855175Z remote: Counting objects:  13% (29/221)
2020-10-08T11:04:39.6856405Z remote: Counting objects:  14% (31/221)
2020-10-08T11:04:39.6857801Z remote: Counting objects:  15% (34/221)
2020-10-08T11:04:39.6859095Z remote: Counting objects:  16% (36/221)
2020-10-08T11:04:39.6861808Z remote: Counting objects:  17% (38/221)
2020-10-08T11:04:39.6862322Z remote: Counting objects:  18% (40/221)
2020-10-08T11:04:39.6862917Z remote: Counting objects:  19% (42/221)
2020-10-08T11:04:39.6863452Z remote: Counting objects:  20% (45/221)
2020-10-08T11:04:39.6863930Z remote: Counting objects:  21% (47/221)
2020-10-08T11:04:39.6864353Z remote: Counting objects:  22% (49/221)
2020-10-08T11:04:39.6864755Z remote: Counting objects:  23% (51/221)
2020-10-08T11:04:39.6865119Z remote: Counting objects:  24% (54/221)
2020-10-08T11:04:39.6865494Z remote: Counting objects:  25% (56/221)
2020-10-08T11:04:39.6865868Z remote: Counting objects:  26% (58/221)
2020-10-08T11:04:39.6866241Z remote: Counting objects:  27% (60/221)
2020-10-08T11:04:39.6866705Z remote: Counting objects:  28% (62/221)
2020-10-08T11:04:39.6867088Z remote: Counting objects:  29% (65/221)
2020-10-08T11:04:39.6869684Z remote: Counting objects:  30% (67/221)
2020-10-08T11:04:39.6870086Z remote: Counting objects:  31% (69/221)
2020-10-08T11:04:39.6871279Z remote: Counting objects:  32% (71/221)
2020-10-08T11:04:39.6871670Z remote: Counting objects:  33% (73/221)
2020-10-08T11:04:39.6872032Z remote: Counting objects:  34% (76/221)
2020-10-08T11:04:39.6872407Z remote: Counting objects:  35% (78/221)
2020-10-08T11:04:39.6872782Z remote: Counting objects:  36% (80/221)
2020-10-08T11:04:39.6873155Z remote: Counting objects:  37% (82/221)
2020-10-08T11:04:39.6873530Z remote: Counting objects:  38% (84/221)
2020-10-08T11:04:39.6873902Z remote: Counting objects:  39% (87/221)
2020-10-08T11:04:39.6874273Z remote: Counting objects:  40% (89/221)
2020-10-08T11:04:39.6874644Z remote: Counting objects:  41% (91/221)
2020-10-08T11:04:39.6875195Z remote: Counting objects:  42% (93/221)
2020-10-08T11:04:39.6875584Z remote: Counting objects:  43% (96/221)
2020-10-08T11:04:39.6877865Z remote: Counting objects:  44% (98/221)
2020-10-08T11:04:39.6879266Z remote: Counting objects:  45% (100/221)
2020-10-08T11:04:39.6879789Z remote: Counting objects:  46% (102/221)
2020-10-08T11:04:39.6880183Z remote: Counting objects:  47% (104/221)
2020-10-08T11:04:39.6880666Z remote: Counting objects:  48% (107/221)
2020-10-08T11:04:39.6881042Z remote: Counting objects:  49% (109/221)
2020-10-08T11:04:39.6881520Z remote: Counting objects:  50% (111/221)
2020-10-08T11:04:39.6881889Z remote: Counting objects:  51% (113/221)
2020-10-08T11:04:39.6882360Z remote: Counting objects:  52% (115/221)
2020-10-08T11:04:39.6882829Z remote: Counting objects:  53% (118/221)
2020-10-08T11:04:39.6883216Z remote: Counting objects:  54% (120/221)
2020-10-08T11:04:39.6883706Z remote: Counting objects:  55% (122/221)
2020-10-08T11:04:39.6884085Z remote: Counting objects:  56% (124/221)
2020-10-08T11:04:39.6884562Z remote: Counting objects:  57% (126/221)
2020-10-08T11:04:39.6885302Z remote: Counting objects:  58% (129/221)
2020-10-08T11:04:39.6885775Z remote: Counting objects:  59% (131/221)
2020-10-08T11:04:39.6886201Z remote: Counting objects:  60% (133/221)
2020-10-08T11:04:39.6886577Z remote: Counting objects:  61% (135/221)
2020-10-08T11:04:39.6886952Z remote: Counting objects:  62% (138/221)
2020-10-08T11:04:39.6887409Z remote: Counting objects:  63% (140/221)
2020-10-08T11:04:39.6887782Z remote: Counting objects:  64% (142/221)
2020-10-08T11:04:39.6888154Z remote: Counting objects:  65% (144/221)
2020-10-08T11:04:39.6888524Z remote: Counting objects:  66% (146/221)
2020-10-08T11:04:39.6888896Z remote: Counting objects:  67% (149/221)
2020-10-08T11:04:39.6889773Z remote: Counting objects:  68% (151/221)
2020-10-08T11:04:39.6890176Z remote: Counting objects:  69% (153/221)
2020-10-08T11:04:39.6890551Z remote: Counting objects:  70% (155/221)
2020-10-08T11:04:39.6891482Z remote: Counting objects:  71% (157/221)
2020-10-08T11:04:39.6892241Z remote: Counting objects:  72% (160/221)
2020-10-08T11:04:39.6892618Z remote: Counting objects:  73% (162/221)
2020-10-08T11:04:39.6892987Z remote: Counting objects:  74% (164/221)
2020-10-08T11:04:39.6894074Z remote: Counting objects:  75% (166/221)
2020-10-08T11:04:39.6894469Z remote: Counting objects:  76% (168/221)
2020-10-08T11:04:39.6894841Z remote: Counting objects:  77% (171/221)
2020-10-08T11:04:39.6895548Z remote: Counting objects:  78% (173/221)
2020-10-08T11:04:39.6898357Z remote: Counting objects:  79% (175/221)
2020-10-08T11:04:39.6898743Z remote: Counting objects:  80% (177/221)
2020-10-08T11:04:39.6899135Z remote: Counting objects:  81% (180/221)
2020-10-08T11:04:39.6899512Z remote: Counting objects:  82% (182/221)
2020-10-08T11:04:39.6900314Z remote: Counting objects:  83% (184/221)
2020-10-08T11:04:39.6900872Z remote: Counting objects:  84% (186/221)
2020-10-08T11:04:39.6901261Z remote: Counting objects:  85% (188/221)
2020-10-08T11:04:39.6901632Z remote: Counting objects:  86% (191/221)
2020-10-08T11:04:39.6902003Z remote: Counting objects:  87% (193/221)
2020-10-08T11:04:39.6902359Z remote: Counting objects:  88% (195/221)
2020-10-08T11:04:39.6902730Z remote: Counting objects:  89% (197/221)
2020-10-08T11:04:39.6903104Z remote: Counting objects:  90% (199/221)
2020-10-08T11:04:39.6903474Z remote: Counting objects:  91% (202/221)
2020-10-08T11:04:39.6903844Z remote: Counting objects:  92% (204/221)
2020-10-08T11:04:39.6904215Z remote: Counting objects:  93% (206/221)
2020-10-08T11:04:39.6904595Z remote: Counting objects:  94% (208/221)
2020-10-08T11:04:39.6904970Z remote: Counting objects:  95% (210/221)
2020-10-08T11:04:39.6905355Z remote: Counting objects:  96% (213/221)
2020-10-08T11:04:39.6905725Z remote: Counting objects:  97% (215/221)
2020-10-08T11:04:39.6906091Z remote: Counting objects:  98% (217/221)
2020-10-08T11:04:39.6906462Z remote: Counting objects:  99% (219/221)
2020-10-08T11:04:39.6906833Z remote: Counting objects: 100% (221/221)
2020-10-08T11:04:39.6907222Z remote: Counting objects: 100% (221/221), done.
2020-10-08T11:04:39.6907634Z remote: Compressing objects:   0% (1/181)
2020-10-08T11:04:39.6908040Z remote: Compressing objects:   1% (2/181)
2020-10-08T11:04:39.6908447Z remote: Compressing objects:   2% (4/181)
2020-10-08T11:04:39.6908854Z remote: Compressing objects:   3% (6/181)
2020-10-08T11:04:39.6909261Z remote: Compressing objects:   4% (8/181)
2020-10-08T11:04:39.6909662Z remote: Compressing objects:   5% (10/181)
2020-10-08T11:04:39.6910075Z remote: Compressing objects:   6% (11/181)
2020-10-08T11:04:39.6910485Z remote: Compressing objects:   7% (13/181)
2020-10-08T11:04:39.6910896Z remote: Compressing objects:   8% (15/181)
2020-10-08T11:04:39.6911465Z remote: Compressing objects:   9% (17/181)
2020-10-08T11:04:39.6911873Z remote: Compressing objects:  10% (19/181)
2020-10-08T11:04:39.6912282Z remote: Compressing objects:  11% (20/181)
2020-10-08T11:04:39.6912690Z remote: Compressing objects:  12% (22/181)
2020-10-08T11:04:39.6913084Z remote: Compressing objects:  13% (24/181)
2020-10-08T11:04:39.6913492Z remote: Compressing objects:  14% (26/181)
2020-10-08T11:04:39.6913900Z remote: Compressing objects:  15% (28/181)
2020-10-08T11:04:39.6914307Z remote: Compressing objects:  16% (29/181)
2020-10-08T11:04:39.6914714Z remote: Compressing objects:  17% (31/181)
2020-10-08T11:04:39.6915194Z remote: Compressing objects:  18% (33/181)
2020-10-08T11:04:39.6915619Z remote: Compressing objects:  19% (35/181)
2020-10-08T11:04:39.6916027Z remote: Compressing objects:  20% (37/181)
2020-10-08T11:04:39.6916627Z remote: Compressing objects:  21% (39/181)
2020-10-08T11:04:39.6917049Z remote: Compressing objects:  22% (40/181)
2020-10-08T11:04:39.6917455Z remote: Compressing objects:  23% (42/181)
2020-10-08T11:04:39.6917862Z remote: Compressing objects:  24% (44/181)
2020-10-08T11:04:39.6918270Z remote: Compressing objects:  25% (46/181)
2020-10-08T11:04:39.6918676Z remote: Compressing objects:  26% (48/181)
2020-10-08T11:04:39.6919083Z remote: Compressing objects:  27% (49/181)
2020-10-08T11:04:39.6938979Z remote: Compressing objects:  28% (51/181)
2020-10-08T11:04:39.6939607Z remote: Compressing objects:  29% (53/181)
2020-10-08T11:04:39.6940152Z remote: Compressing objects:  30% (55/181)
2020-10-08T11:04:39.6942557Z remote: Compressing objects:  31% (57/181)
2020-10-08T11:04:39.6943277Z remote: Compressing objects:  32% (58/181)
2020-10-08T11:04:39.6943852Z remote: Compressing objects:  33% (60/181)
2020-10-08T11:04:39.6944409Z remote: Compressing objects:  34% (62/181)
2020-10-08T11:04:39.6944947Z remote: Compressing objects:  35% (64/181)
2020-10-08T11:04:39.6945485Z remote: Compressing objects:  36% (66/181)
2020-10-08T11:04:39.6946067Z remote: Compressing objects:  37% (67/181)
2020-10-08T11:04:39.6946616Z remote: Compressing objects:  38% (69/181)
2020-10-08T11:04:39.6947217Z remote: Compressing objects:  39% (71/181)
2020-10-08T11:04:39.6948935Z remote: Compressing objects:  40% (73/181)
2020-10-08T11:04:39.6949543Z remote: Compressing objects:  41% (75/181)
2020-10-08T11:04:39.6953081Z remote: Compressing objects:  42% (77/181)
2020-10-08T11:04:39.6958405Z remote: Compressing objects:  43% (78/181)
2020-10-08T11:04:39.6961915Z remote: Compressing objects:  44% (80/181)
2020-10-08T11:04:39.6966384Z remote: Compressing objects:  45% (82/181)
2020-10-08T11:04:39.6972359Z remote: Compressing objects:  46% (84/181)
2020-10-08T11:04:39.6973044Z remote: Compressing objects:  47% (86/181)
2020-10-08T11:04:39.6977890Z remote: Compressing objects:  48% (87/181)
2020-10-08T11:04:39.6997669Z remote: Compressing objects:  49% (89/181)
2020-10-08T11:04:39.6999537Z remote: Compressing objects:  50% (91/181)
2020-10-08T11:04:39.7007692Z remote: Compressing objects:  51% (93/181)
2020-10-08T11:04:39.7015208Z remote: Compressing objects:  52% (95/181)
2020-10-08T11:04:39.7033523Z remote: Compressing objects:  53% (96/181)
2020-10-08T11:04:39.7034119Z remote: Compressing objects:  54% (98/181)
2020-10-08T11:04:39.7034676Z remote: Compressing objects:  55% (100/181)
2020-10-08T11:04:39.7039768Z remote: Compressing objects:  56% (102/181)
2020-10-08T11:04:39.7041552Z remote: Compressing objects:  57% (104/181)
2020-10-08T11:04:39.7063513Z remote: Compressing objects:  58% (105/181)
2020-10-08T11:04:39.7064440Z remote: Compressing objects:  59% (107/181)
2020-10-08T11:04:39.7065180Z remote: Compressing objects:  60% (109/181)
2020-10-08T11:04:39.7065777Z remote: Compressing objects:  61% (111/181)
2020-10-08T11:04:39.7066209Z remote: Compressing objects:  62% (113/181)
2020-10-08T11:04:39.7066621Z remote: Compressing objects:  63% (115/181)
2020-10-08T11:04:39.7067162Z remote: Compressing objects:  64% (116/181)
2020-10-08T11:04:39.7067582Z remote: Compressing objects:  65% (118/181)
2020-10-08T11:04:39.7067992Z remote: Compressing objects:  66% (120/181)
2020-10-08T11:04:39.7068403Z remote: Compressing objects:  67% (122/181)
2020-10-08T11:04:39.7068814Z remote: Compressing objects:  68% (124/181)
2020-10-08T11:04:39.7069356Z remote: Compressing objects:  69% (125/181)
2020-10-08T11:04:39.7069790Z remote: Compressing objects:  70% (127/181)
2020-10-08T11:04:39.7070198Z remote: Compressing objects:  71% (129/181)
2020-10-08T11:04:39.7070904Z remote: Compressing objects:  72% (131/181)
2020-10-08T11:04:39.7072069Z remote: Compressing objects:  73% (133/181)
2020-10-08T11:04:39.7072479Z remote: Compressing objects:  74% (134/181)
2020-10-08T11:04:39.7073259Z remote: Compressing objects:  75% (136/181)
2020-10-08T11:04:39.7073678Z remote: Compressing objects:  76% (138/181)
2020-10-08T11:04:39.7074087Z remote: Compressing objects:  77% (140/181)
2020-10-08T11:04:39.7074498Z remote: Compressing objects:  78% (142/181)
2020-10-08T11:04:39.7074906Z remote: Compressing objects:  79% (143/181)
2020-10-08T11:04:39.7075316Z remote: Compressing objects:  80% (145/181)
2020-10-08T11:04:39.7075725Z remote: Compressing objects:  81% (147/181)
2020-10-08T11:04:39.7076150Z remote: Compressing objects:  82% (149/181)
2020-10-08T11:04:39.7076951Z remote: Compressing objects:  83% (151/181)
2020-10-08T11:04:39.7077511Z remote: Compressing objects:  84% (153/181)
2020-10-08T11:04:39.7078082Z remote: Compressing objects:  85% (154/181)
2020-10-08T11:04:39.7078626Z remote: Compressing objects:  86% (156/181)
2020-10-08T11:04:39.7079161Z remote: Compressing objects:  87% (158/181)
2020-10-08T11:04:39.7079705Z remote: Compressing objects:  88% (160/181)
2020-10-08T11:04:39.7080240Z remote: Compressing objects:  89% (162/181)
2020-10-08T11:04:39.7080776Z remote: Compressing objects:  90% (163/181)
2020-10-08T11:04:39.7081316Z remote: Compressing objects:  91% (165/181)
2020-10-08T11:04:39.7081852Z remote: Compressing objects:  92% (167/181)
2020-10-08T11:04:39.7082389Z remote: Compressing objects:  93% (169/181)
2020-10-08T11:04:39.7082941Z remote: Compressing objects:  94% (171/181)
2020-10-08T11:04:39.7083476Z remote: Compressing objects:  95% (172/181)
2020-10-08T11:04:39.7084010Z remote: Compressing objects:  96% (174/181)
2020-10-08T11:04:39.7084547Z remote: Compressing objects:  97% (176/181)
2020-10-08T11:04:39.7085096Z remote: Compressing objects:  98% (178/181)
2020-10-08T11:04:39.7085613Z remote: Compressing objects:  99% (180/181)
2020-10-08T11:04:39.7086152Z remote: Compressing objects: 100% (181/181)
2020-10-08T11:04:39.7086706Z remote: Compressing objects: 100% (181/181), done.
2020-10-08T11:04:39.7290695Z Receiving objects:   0% (1/221)
2020-10-08T11:04:39.7291816Z Receiving objects:   1% (3/221)
2020-10-08T11:04:39.7294568Z Receiving objects:   2% (5/221)
2020-10-08T11:04:39.7297074Z Receiving objects:   3% (7/221)
2020-10-08T11:04:39.7297667Z Receiving objects:   4% (9/221)
2020-10-08T11:04:39.7298175Z Receiving objects:   5% (12/221)
2020-10-08T11:04:39.7300429Z Receiving objects:   6% (14/221)
2020-10-08T11:04:39.7301002Z Receiving objects:   7% (16/221)
2020-10-08T11:04:39.7301496Z Receiving objects:   8% (18/221)
2020-10-08T11:04:39.7301986Z Receiving objects:   9% (20/221)
2020-10-08T11:04:39.7302460Z Receiving objects:  10% (23/221)
2020-10-08T11:04:39.7303235Z Receiving objects:  11% (25/221)
2020-10-08T11:04:39.7305512Z Receiving objects:  12% (27/221)
2020-10-08T11:04:39.7306084Z Receiving objects:  13% (29/221)
2020-10-08T11:04:39.7306580Z Receiving objects:  14% (31/221)
2020-10-08T11:04:39.7358373Z Receiving objects:  15% (34/221)
2020-10-08T11:04:39.7358762Z Receiving objects:  16% (36/221)
2020-10-08T11:04:39.7366252Z Receiving objects:  17% (38/221)
2020-10-08T11:04:39.7369205Z Receiving objects:  18% (40/221)
2020-10-08T11:04:39.7369569Z Receiving objects:  19% (42/221)
2020-10-08T11:04:39.7369930Z Receiving objects:  20% (45/221)
2020-10-08T11:04:39.7370268Z Receiving objects:  21% (47/221)
2020-10-08T11:04:39.7370603Z Receiving objects:  22% (49/221)
2020-10-08T11:04:39.7371111Z Receiving objects:  23% (51/221)
2020-10-08T11:04:39.7371460Z Receiving objects:  24% (54/221)
2020-10-08T11:04:39.7371803Z Receiving objects:  25% (56/221)
2020-10-08T11:04:39.7373918Z Receiving objects:  26% (58/221)
2020-10-08T11:04:39.7374309Z Receiving objects:  27% (60/221)
2020-10-08T11:04:39.7374647Z Receiving objects:  28% (62/221)
2020-10-08T11:04:39.7374982Z Receiving objects:  29% (65/221)
2020-10-08T11:04:39.7375316Z Receiving objects:  30% (67/221)
2020-10-08T11:04:39.7375771Z Receiving objects:  31% (69/221)
2020-10-08T11:04:39.7376096Z Receiving objects:  32% (71/221)
2020-10-08T11:04:39.7378086Z Receiving objects:  33% (73/221)
2020-10-08T11:04:39.7378454Z Receiving objects:  34% (76/221)
2020-10-08T11:04:39.7380402Z Receiving objects:  35% (78/221)
2020-10-08T11:04:39.7380767Z Receiving objects:  36% (80/221)
2020-10-08T11:04:39.7382674Z Receiving objects:  37% (82/221)
2020-10-08T11:04:39.7383034Z Receiving objects:  38% (84/221)
2020-10-08T11:04:39.7385118Z Receiving objects:  39% (87/221)
2020-10-08T11:04:39.7385494Z Receiving objects:  40% (89/221)
2020-10-08T11:04:39.7385816Z Receiving objects:  41% (91/221)
2020-10-08T11:04:39.7386153Z Receiving objects:  42% (93/221)
2020-10-08T11:04:39.7386489Z Receiving objects:  43% (96/221)
2020-10-08T11:04:39.7386836Z Receiving objects:  44% (98/221)
2020-10-08T11:04:39.7387172Z Receiving objects:  45% (100/221)
2020-10-08T11:04:39.7389141Z Receiving objects:  46% (102/221)
2020-10-08T11:04:39.7389672Z Receiving objects:  47% (104/221)
2020-10-08T11:04:39.7390152Z Receiving objects:  48% (107/221)
2020-10-08T11:04:39.7390620Z Receiving objects:  49% (109/221)
2020-10-08T11:04:39.7391081Z Receiving objects:  50% (111/221)
2020-10-08T11:04:39.7391531Z Receiving objects:  51% (113/221)
2020-10-08T11:04:39.7427934Z Receiving objects:  52% (115/221)
2020-10-08T11:04:39.7428575Z Receiving objects:  53% (118/221)
2020-10-08T11:04:39.7429097Z Receiving objects:  54% (120/221)
2020-10-08T11:04:39.7429590Z Receiving objects:  55% (122/221)
2020-10-08T11:04:39.7430105Z Receiving objects:  56% (124/221)
2020-10-08T11:04:39.7430603Z Receiving objects:  57% (126/221)
2020-10-08T11:04:39.7431095Z Receiving objects:  58% (129/221)
2020-10-08T11:04:39.7431576Z Receiving objects:  59% (131/221)
2020-10-08T11:04:39.7432083Z Receiving objects:  60% (133/221)
2020-10-08T11:04:39.7432563Z Receiving objects:  61% (135/221)
2020-10-08T11:04:39.7433041Z Receiving objects:  62% (138/221)
2020-10-08T11:04:39.7433509Z Receiving objects:  63% (140/221)
2020-10-08T11:04:39.7434016Z Receiving objects:  64% (142/221)
2020-10-08T11:04:39.7434505Z Receiving objects:  65% (144/221)
2020-10-08T11:04:39.7434992Z Receiving objects:  66% (146/221)
2020-10-08T11:04:39.7435475Z Receiving objects:  67% (149/221)
2020-10-08T11:04:39.7436045Z Receiving objects:  68% (151/221)
2020-10-08T11:04:39.7436813Z Receiving objects:  69% (153/221)
2020-10-08T11:04:39.7437367Z Receiving objects:  70% (155/221)
2020-10-08T11:04:39.7437861Z Receiving objects:  71% (157/221)
2020-10-08T11:04:39.7438376Z Receiving objects:  72% (160/221)
2020-10-08T11:04:39.7438874Z Receiving objects:  73% (162/221)
2020-10-08T11:04:39.7439360Z Receiving objects:  74% (164/221)
2020-10-08T11:04:39.7439833Z Receiving objects:  75% (166/221)
2020-10-08T11:04:39.7440581Z Receiving objects:  76% (168/221)
2020-10-08T11:04:39.7441084Z Receiving objects:  77% (171/221)
2020-10-08T11:04:39.7441584Z Receiving objects:  78% (173/221)
2020-10-08T11:04:39.7442072Z Receiving objects:  79% (175/221)
2020-10-08T11:04:39.7443042Z Receiving objects:  80% (177/221)
2020-10-08T11:04:39.7444850Z remote: Total 221 (delta 26), reused 98 (delta 16), pack-reused 0
2020-10-08T11:04:39.7448433Z Receiving objects:  81% (180/221)
2020-10-08T11:04:39.7448801Z Receiving objects:  82% (182/221)
2020-10-08T11:04:39.7449143Z Receiving objects:  83% (184/221)
2020-10-08T11:04:39.7449479Z Receiving objects:  84% (186/221)
2020-10-08T11:04:39.7449804Z Receiving objects:  85% (188/221)
2020-10-08T11:04:39.7450143Z Receiving objects:  86% (191/221)
2020-10-08T11:04:39.7450634Z Receiving objects:  87% (193/221)
2020-10-08T11:04:39.7451000Z Receiving objects:  88% (195/221)
2020-10-08T11:04:39.7451338Z Receiving objects:  89% (197/221)
2020-10-08T11:04:39.7451678Z Receiving objects:  90% (199/221)
2020-10-08T11:04:39.7452027Z Receiving objects:  91% (202/221)
2020-10-08T11:04:39.7452364Z Receiving objects:  92% (204/221)
2020-10-08T11:04:39.7452686Z Receiving objects:  93% (206/221)
2020-10-08T11:04:39.7453024Z Receiving objects:  94% (208/221)
2020-10-08T11:04:39.7453361Z Receiving objects:  95% (210/221)
2020-10-08T11:04:39.7453698Z Receiving objects:  96% (213/221)
2020-10-08T11:04:39.7454033Z Receiving objects:  97% (215/221)
2020-10-08T11:04:39.7454372Z Receiving objects:  98% (217/221)
2020-10-08T11:04:39.7454708Z Receiving objects:  99% (219/221)
2020-10-08T11:04:39.7455045Z Receiving objects: 100% (221/221)
2020-10-08T11:04:39.7455423Z Receiving objects: 100% (221/221), 426.66 KiB | 19.39 MiB/s, done.
2020-10-08T11:04:39.7455807Z Resolving deltas:   0% (0/26)
2020-10-08T11:04:39.7456145Z Resolving deltas:   3% (1/26)
2020-10-08T11:04:39.7456474Z Resolving deltas:   7% (2/26)
2020-10-08T11:04:39.7456802Z Resolving deltas:  11% (3/26)
2020-10-08T11:04:39.7457129Z Resolving deltas:  19% (5/26)
2020-10-08T11:04:39.7457462Z Resolving deltas:  26% (7/26)
2020-10-08T11:04:39.7457774Z Resolving deltas:  30% (8/26)
2020-10-08T11:04:39.7458102Z Resolving deltas:  50% (13/26)
2020-10-08T11:04:39.7458430Z Resolving deltas:  57% (15/26)
2020-10-08T11:04:39.7458754Z Resolving deltas:  61% (16/26)
2020-10-08T11:04:39.7459081Z Resolving deltas:  65% (17/26)
2020-10-08T11:04:39.7459407Z Resolving deltas:  69% (18/26)
2020-10-08T11:04:39.7459735Z Resolving deltas:  73% (19/26)
2020-10-08T11:04:39.7460059Z Resolving deltas:  76% (20/26)
2020-10-08T11:04:39.7460372Z Resolving deltas:  80% (21/26)
2020-10-08T11:04:39.7460698Z Resolving deltas:  84% (22/26)
2020-10-08T11:04:39.7461024Z Resolving deltas:  88% (23/26)
2020-10-08T11:04:39.7463277Z Resolving deltas:  92% (24/26)
2020-10-08T11:04:39.7463837Z Resolving deltas:  96% (25/26)
2020-10-08T11:04:39.7467943Z Resolving deltas: 100% (26/26)
2020-10-08T11:04:39.7468316Z Resolving deltas: 100% (26/26), done.
2020-10-08T11:04:39.8171135Z From https://github.com/submariner-io/submariner
2020-10-08T11:04:39.8172440Z  * [new ref]         a94878ad4fe368d3bb1efd0be1df1b354e805ae3 -> pull/865/merge
2020-10-08T11:04:39.8211056Z ##[endgroup]
2020-10-08T11:04:39.8212081Z ##[group]Determining the checkout info
2020-10-08T11:04:39.8213372Z ##[endgroup]
2020-10-08T11:04:39.8214068Z ##[group]Checking out the ref
2020-10-08T11:04:39.8219112Z [command]/usr/bin/git checkout --progress --force refs/remotes/pull/865/merge
2020-10-08T11:04:39.8404013Z Note: switching to 'refs/remotes/pull/865/merge'.
2020-10-08T11:04:39.8404929Z
2020-10-08T11:04:39.8406001Z You are in 'detached HEAD' state. You can look around, make experimental
2020-10-08T11:04:39.8407026Z changes and commit them, and you can discard any commits you make in this
2020-10-08T11:04:39.8407965Z state without impacting any branches by switching back to a branch.
2020-10-08T11:04:39.8408580Z
2020-10-08T11:04:39.8409258Z If you want to create a new branch to retain commits you create, you may
2020-10-08T11:04:39.8410389Z do so (now or later) by using -c with the switch command. Example:
2020-10-08T11:04:39.8411317Z
2020-10-08T11:04:39.8412194Z   git switch -c <new-branch-name>
2020-10-08T11:04:39.8412825Z
2020-10-08T11:04:39.8414968Z Or undo this operation with:
2020-10-08T11:04:39.8415514Z
2020-10-08T11:04:39.8416318Z   git switch -
2020-10-08T11:04:39.8416882Z
2020-10-08T11:04:39.8417657Z Turn off this advice by setting config variable advice.detachedHead to false
2020-10-08T11:04:39.8421336Z
2020-10-08T11:04:39.8422110Z HEAD is now at a94878a Merge 49b43b007e6c08a3a277e1a6e6a891d1ff127880 into 4b8bc77fab50f53c61dd58d4f96f8425634c07b1
2020-10-08T11:04:39.8422994Z ##[endgroup]
2020-10-08T11:04:39.8627001Z [command]/usr/bin/git log -1 --format='%H'
2020-10-08T11:04:39.8662551Z 'a94878ad4fe368d3bb1efd0be1df1b354e805ae3'
2020-10-08T11:04:39.8801244Z ##[group]Run sudo swapoff -a
2020-10-08T11:04:39.8801755Z sudo swapoff -a
2020-10-08T11:04:39.8802165Z sudo rm -f /swapfile
2020-10-08T11:04:39.8802482Z df -h
2020-10-08T11:04:39.8802734Z free -h
2020-10-08T11:04:39.8846326Z shell: /bin/bash -e {0}
2020-10-08T11:04:39.8846680Z ##[endgroup]
2020-10-08T11:04:40.1000130Z Filesystem      Size  Used Avail Use% Mounted on
2020-10-08T11:04:40.1001005Z udev            3.4G     0  3.4G   0% /dev
2020-10-08T11:04:40.1001520Z tmpfs           696M  680K  695M   1% /run
2020-10-08T11:04:40.1002151Z /dev/sda1        84G   62G   22G  75% /
2020-10-08T11:04:40.1002667Z tmpfs           3.4G  8.0K  3.4G   1% /dev/shm
2020-10-08T11:04:40.1003167Z tmpfs           5.0M     0  5.0M   0% /run/lock
2020-10-08T11:04:40.1003898Z tmpfs           3.4G     0  3.4G   0% /sys/fs/cgroup
2020-10-08T11:04:40.1004263Z /dev/sda15      105M  3.6M  101M   4% /boot/efi
2020-10-08T11:04:40.1004784Z /dev/sdb1        14G  4.1G  9.0G  32% /mnt
2020-10-08T11:04:40.1100521Z               total        used        free      shared  buff/cache   available
2020-10-08T11:04:40.1101434Z Mem:           6.8G        429M        5.0G         25M        1.4G        6.1G
2020-10-08T11:04:40.1102072Z Swap:            0B          0B          0B
2020-10-08T11:04:40.1154866Z ##[group]Run make e2e using=" helm "
2020-10-08T11:04:40.1155263Z make e2e using=" helm "
2020-10-08T11:04:40.1195319Z shell: /bin/bash -e {0}
2020-10-08T11:04:40.1195662Z ##[endgroup]
2020-10-08T11:04:40.1609450Z Downloading dapper
2020-10-08T11:04:40.4006754Z .dapper.tmp version v0.5.3
2020-10-08T11:04:40.4192276Z ./.dapper -m bind make e2e  -- using=helm\
2020-10-08T11:04:43.7893733Z Sending build context to Docker daemon  6.961MB
2020-10-08T11:04:43.7917352Z
2020-10-08T11:04:43.7981798Z Step 1/5 : FROM quay.io/submariner/shipyard-dapper-base:0.7.1
2020-10-08T11:04:44.0389135Z 0.7.1: Pulling from submariner/shipyard-dapper-base
2020-10-08T11:04:44.0598224Z bbca93d6c76b: Pulling fs layer
2020-10-08T11:04:44.0598617Z 1c30167c095e: Pulling fs layer
2020-10-08T11:04:44.0598962Z a274ee7d3ac9: Pulling fs layer
2020-10-08T11:04:44.0599309Z 30d9f3b2fbe6: Pulling fs layer
2020-10-08T11:04:44.0599709Z 96c70f9b1ce0: Pulling fs layer
2020-10-08T11:04:44.0600151Z a9a281312a19: Pulling fs layer
2020-10-08T11:04:44.0600496Z 48cc7c7bfa15: Pulling fs layer
2020-10-08T11:04:44.0611283Z 96c70f9b1ce0: Waiting
2020-10-08T11:04:44.0611618Z a9a281312a19: Waiting
2020-10-08T11:04:44.0614458Z 48cc7c7bfa15: Waiting
2020-10-08T11:04:44.0614778Z 30d9f3b2fbe6: Waiting
2020-10-08T11:04:44.6062278Z a274ee7d3ac9: Verifying Checksum
2020-10-08T11:04:44.6072629Z a274ee7d3ac9: Download complete
2020-10-08T11:04:44.7369650Z 30d9f3b2fbe6: Verifying Checksum
2020-10-08T11:04:44.7439142Z 30d9f3b2fbe6: Download complete
2020-10-08T11:04:44.8449140Z 96c70f9b1ce0: Verifying Checksum
2020-10-08T11:04:44.8450446Z 96c70f9b1ce0: Download complete
2020-10-08T11:04:44.8960660Z bbca93d6c76b: Verifying Checksum
2020-10-08T11:04:44.8964357Z bbca93d6c76b: Download complete
2020-10-08T11:04:44.9397140Z a9a281312a19: Verifying Checksum
2020-10-08T11:04:44.9400940Z a9a281312a19: Download complete
2020-10-08T11:04:44.9981987Z 48cc7c7bfa15: Verifying Checksum
2020-10-08T11:04:44.9982725Z 48cc7c7bfa15: Download complete
2020-10-08T11:04:46.2063226Z 1c30167c095e: Verifying Checksum
2020-10-08T11:04:46.2063645Z 1c30167c095e: Download complete
2020-10-08T11:04:49.0068382Z bbca93d6c76b: Pull complete
2020-10-08T11:04:56.8069304Z 1c30167c095e: Pull complete
2020-10-08T11:05:00.2568141Z a274ee7d3ac9: Pull complete
2020-10-08T11:05:00.3154442Z 30d9f3b2fbe6: Pull complete
2020-10-08T11:05:00.3745321Z 96c70f9b1ce0: Pull complete
2020-10-08T11:05:00.4320606Z a9a281312a19: Pull complete
2020-10-08T11:05:00.5074473Z 48cc7c7bfa15: Pull complete
2020-10-08T11:05:00.5103177Z Digest: sha256:086697dc5be6c92cac60d506e89a65fbc341d425e59ccea4f12e1e8b71f601b9
2020-10-08T11:05:00.5123336Z Status: Downloaded newer image for quay.io/submariner/shipyard-dapper-base:0.7.1
2020-10-08T11:05:00.5131821Z  ---> c50512fb3462
2020-10-08T11:05:00.5133108Z Step 2/5 : ENV DAPPER_ENV="REPO TAG QUAY_USERNAME QUAY_PASSWORD GITHUB_SHA CLUSTERS_ARGS DEPLOY_ARGS IMAGES_ARGS RELEASE_ARGS"     DAPPER_SOURCE=/go/src/github.com/submariner-io/submariner DAPPER_DOCKER_SOCKET=true
2020-10-08T11:05:03.9666185Z  ---> Running in 435c0d34b9c6
2020-10-08T11:05:04.7760625Z Removing intermediate container 435c0d34b9c6
2020-10-08T11:05:04.7761799Z  ---> 107ec502674d
2020-10-08T11:05:04.7762169Z Step 3/5 : ENV DAPPER_OUTPUT=${DAPPER_SOURCE}/output
2020-10-08T11:05:04.8054177Z  ---> Running in d4f8a6ec9bea
2020-10-08T11:05:05.6274519Z Removing intermediate container d4f8a6ec9bea
2020-10-08T11:05:05.6275711Z  ---> 959c7a109766
2020-10-08T11:05:05.6276045Z Step 4/5 : WORKDIR ${DAPPER_SOURCE}
2020-10-08T11:05:05.6543717Z  ---> Running in 47f9323e9cfd
2020-10-08T11:05:06.4523551Z Removing intermediate container 47f9323e9cfd
2020-10-08T11:05:06.4524744Z  ---> 380b60cb98ca
2020-10-08T11:05:06.4525127Z Step 5/5 : ENTRYPOINT ["/opt/shipyard/scripts/entry"]
2020-10-08T11:05:06.4800570Z  ---> Running in d60020666bb4
2020-10-08T11:05:07.2706588Z Removing intermediate container d60020666bb4
2020-10-08T11:05:07.2707948Z  ---> 9b8bd61411c6
2020-10-08T11:05:07.2777458Z Successfully built 9b8bd61411c6
2020-10-08T11:05:07.2955772Z Successfully tagged submariner:HEAD
2020-10-08T11:05:08.2728187Z [submariner]$ trap chown -R 1001:116 . exit
2020-10-08T11:05:08.2729319Z [submariner]$ mkdir -p bin dist output
2020-10-08T11:05:08.2749925Z [submariner]$ make e2e -- using=helm 
2020-10-08T11:05:08.3161289Z fatal: No names found, cannot describe anything.
2020-10-08T11:05:08.3283996Z /opt/shipyard/scripts/e2e.sh --focus .\* cluster2 cluster3 cluster1
2020-10-08T11:05:08.3313805Z Downloading shflags 1.0.3
2020-10-08T11:05:08.3389166Z   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
2020-10-08T11:05:08.3389709Z                                  Dload  Upload   Total   Spent    Left  Speed
2020-10-08T11:05:08.3389988Z
2020-10-08T11:05:08.4081884Z   0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
2020-10-08T11:05:08.4082495Z 100 31091  100 31091    0     0   433k      0 --:--:-- --:--:-- --:--:--  433k
2020-10-08T11:05:08.4641869Z [submariner]$ source /opt/shipyard/scripts/lib/utils
2020-10-08T11:05:08.4651976Z [submariner]$ . /opt/shipyard/scripts/lib/source_only
2020-10-08T11:05:08.4660042Z [submariner]$ script_name=utils
2020-10-08T11:05:08.4669970Z [submariner]$ exec_name=e2e.sh
2020-10-08T11:05:08.4689006Z [submariner]$ source /opt/shipyard/scripts/lib/cluster_settings
2020-10-08T11:05:08.4699066Z [submariner]$ . /opt/shipyard/scripts/lib/source_only
2020-10-08T11:05:08.4708464Z [submariner]$ script_name=cluster_settings
2020-10-08T11:05:08.4719132Z [submariner]$ exec_name=e2e.sh
2020-10-08T11:05:08.4739377Z [submariner]$ broker=cluster1
2020-10-08T11:05:08.4748652Z [submariner]$ declare -gA cluster_nodes
2020-10-08T11:05:08.4760890Z [submariner]$ cluster_nodes[cluster1]=control-plane worker
2020-10-08T11:05:08.4769763Z [submariner]$ cluster_nodes[cluster2]=control-plane worker
2020-10-08T11:05:08.4782542Z [submariner]$ cluster_nodes[cluster3]=control-plane worker worker
2020-10-08T11:05:08.4791995Z [submariner]$ declare -gA cluster_subm
2020-10-08T11:05:08.4802353Z [submariner]$ cluster_subm[cluster1]=true
2020-10-08T11:05:08.4814643Z [submariner]$ cluster_subm[cluster2]=true
2020-10-08T11:05:08.4823597Z [submariner]$ cluster_subm[cluster3]=true
2020-10-08T11:05:08.4835725Z [submariner]$ declare -gA cluster_cni
2020-10-08T11:05:08.4845098Z [submariner]$ cluster_cni[cluster2]=weave
2020-10-08T11:05:08.4857350Z [submariner]$ cluster_cni[cluster3]=weave
2020-10-08T11:05:08.4869155Z [submariner]$ declare_kubeconfig
2020-10-08T11:05:08.4883626Z [submariner]$ declare_kubeconfig
2020-10-08T11:05:08.4891489Z [submariner]$ source /opt/shipyard/scripts/lib/kubecfg
2020-10-08T11:05:08.4903620Z [submariner]$ export KUBECONFIG
2020-10-08T11:05:08.4943357Z [submariner]$ KUBECONFIG=
2020-10-08T11:05:08.4958066Z [submariner]$ find /go/src/github.com/submariner-io/submariner/output/kubeconfigs -type f -printf %p:
2020-10-08T11:05:08.4973796Z find: '/go/src/github.com/submariner-io/submariner/output/kubeconfigs': No such file or directory
2020-10-08T11:05:08.4987025Z [submariner]$ :
2020-10-08T11:05:08.5000888Z [submariner]$ deploy_env_once
2020-10-08T11:05:08.5010674Z [submariner]$ deploy_env_once
2020-10-08T11:05:08.5022599Z 
2020-10-08T11:05:09.1438742Z [submariner]$ make deploy
2020-10-08T11:05:09.1465909Z make[1]: Entering directory '/go/src/github.com/submariner-io/submariner'
2020-10-08T11:05:09.1852422Z fatal: No names found, cannot describe anything.
2020-10-08T11:05:09.2033751Z /opt/shipyard/scripts/clusters.sh --cluster_settings /go/src/github.com/submariner-io/submariner/scripts/cluster_settings
2020-10-08T11:05:09.2921319Z Running with: k8s_version=1.17.0, olm_version=0.14.1, olm=false, globalnet=false, registry_inmemory=true, cluster_settings=/go/src/github.com/submariner-io/submariner/scripts/cluster_settings, timeout=5m
2020-10-08T11:05:09.2943131Z [submariner]$ source /opt/shipyard/scripts/lib/utils
2020-10-08T11:05:09.2956343Z [submariner]$ . /opt/shipyard/scripts/lib/source_only
2020-10-08T11:05:09.2964501Z [submariner]$ script_name=utils
2020-10-08T11:05:09.2976370Z [submariner]$ exec_name=clusters.sh
2020-10-08T11:05:09.2993242Z [submariner]$ source /opt/shipyard/scripts/lib/cluster_settings
2020-10-08T11:05:09.3000944Z [submariner]$ . /opt/shipyard/scripts/lib/source_only
2020-10-08T11:05:09.3012688Z [submariner]$ script_name=cluster_settings
2020-10-08T11:05:09.3019991Z [submariner]$ exec_name=clusters.sh
2020-10-08T11:05:09.3041799Z [submariner]$ broker=cluster1
2020-10-08T11:05:09.3049136Z [submariner]$ declare -gA cluster_nodes
2020-10-08T11:05:09.3058726Z [submariner]$ cluster_nodes[cluster1]=control-plane worker
2020-10-08T11:05:09.3071183Z [submariner]$ cluster_nodes[cluster2]=control-plane worker
2020-10-08T11:05:09.3079976Z [submariner]$ cluster_nodes[cluster3]=control-plane worker worker
2020-10-08T11:05:09.3089990Z [submariner]$ declare -gA cluster_subm
2020-10-08T11:05:09.3100989Z [submariner]$ cluster_subm[cluster1]=true
2020-10-08T11:05:09.3111400Z [submariner]$ cluster_subm[cluster2]=true
2020-10-08T11:05:09.3124104Z [submariner]$ cluster_subm[cluster3]=true
2020-10-08T11:05:09.3133080Z [submariner]$ declare -gA cluster_cni
2020-10-08T11:05:09.3141501Z [submariner]$ cluster_cni[cluster2]=weave
2020-10-08T11:05:09.3153280Z [submariner]$ cluster_cni[cluster3]=weave
2020-10-08T11:05:09.3164690Z [submariner]$ source /go/src/github.com/submariner-io/submariner/scripts/cluster_settings
2020-10-08T11:05:09.3176952Z [submariner]$ cluster_nodes[cluster1]=control-plane worker
2020-10-08T11:05:09.3189695Z [submariner]$ cluster_nodes[cluster2]=control-plane worker worker
2020-10-08T11:05:09.3199481Z [submariner]$ cluster_nodes[cluster3]=control-plane worker worker
2020-10-08T11:05:09.3254792Z [submariner]$ cat
2020-10-08T11:05:09.3275705Z [submariner]$ typeset -p cluster_nodes
2020-10-08T11:05:09.3289400Z [submariner]$ cut -f 2- -d=
2020-10-08T11:05:09.3317348Z [submariner]$ typeset -p cluster_subm
2020-10-08T11:05:09.3331875Z [submariner]$ cut -f 2- -d=
2020-10-08T11:05:09.3354152Z Cluster settings::
2020-10-08T11:05:09.3354850Z   broker - 'cluster1'
2020-10-08T11:05:09.3355406Z   clusters - 'cluster1' 'cluster2' 'cluster3'
2020-10-08T11:05:09.3356646Z   nodes per cluster - ([cluster3]="control-plane worker worker" [cluster2]="control-plane worker worker" [cluster1]="control-plane worker" )
2020-10-08T11:05:09.3357756Z   install submariner - ([cluster3]="true" [cluster2]="true" [cluster1]="true" )
2020-10-08T11:05:09.3369291Z [submariner]$ rm -rf /go/src/github.com/submariner-io/submariner/output/kubeconfigs
2020-10-08T11:05:09.3388589Z [submariner]$ mkdir -p /go/src/github.com/submariner-io/submariner/output/kubeconfigs
2020-10-08T11:05:09.3411522Z [submariner]$ run_local_registry
2020-10-08T11:05:09.3421746Z [submariner]$ run_local_registry
2020-10-08T11:05:09.3432844Z [submariner]$ registry_running
2020-10-08T11:05:09.3443796Z [submariner]$ registry_running
2020-10-08T11:05:09.3452881Z [submariner]$ docker ps --filter name=^/?kind-registry$
2020-10-08T11:05:09.3466926Z [submariner]$ grep kind-registry
2020-10-08T11:05:09.6568478Z [submariner]$ return 0
2020-10-08T11:05:09.6572135Z Deploying local registry kind-registry to serve images centrally.
2020-10-08T11:05:09.6581905Z [submariner]$ local volume_flag
2020-10-08T11:05:09.6596883Z [submariner]$ volume_flag=-v /dev/shm/kind-registry:/var/lib/registry
2020-10-08T11:05:09.6607360Z [submariner]$ docker run -d -v /dev/shm/kind-registry:/var/lib/registry -p 5000:5000 --restart=always --name kind-registry registry:2
2020-10-08T11:05:09.9722643Z Unable to find image 'registry:2' locally
2020-10-08T11:05:10.1270854Z 2: Pulling from library/registry
2020-10-08T11:05:10.1935159Z cbdbe7a5bc2a: Already exists
2020-10-08T11:05:10.2035139Z 47112e65547d: Pulling fs layer
2020-10-08T11:05:10.2035589Z 46bcb632e506: Pulling fs layer
2020-10-08T11:05:10.2035930Z c1cc712bcecd: Pulling fs layer
2020-10-08T11:05:10.2036290Z 3db6272dcbfa: Pulling fs layer
2020-10-08T11:05:10.2036817Z 3db6272dcbfa: Waiting
2020-10-08T11:05:10.4004776Z c1cc712bcecd: Verifying Checksum
2020-10-08T11:05:10.4006230Z c1cc712bcecd: Download complete
2020-10-08T11:05:10.5041339Z 47112e65547d: Verifying Checksum
2020-10-08T11:05:10.5041788Z 47112e65547d: Download complete
2020-10-08T11:05:10.5774393Z 3db6272dcbfa: Verifying Checksum
2020-10-08T11:05:10.5778958Z 3db6272dcbfa: Download complete
2020-10-08T11:05:10.6537402Z 47112e65547d: Pull complete
2020-10-08T11:05:10.6550469Z 46bcb632e506: Verifying Checksum
2020-10-08T11:05:10.6550912Z 46bcb632e506: Download complete
2020-10-08T11:05:10.8953910Z 46bcb632e506: Pull complete
2020-10-08T11:05:10.9515205Z c1cc712bcecd: Pull complete
2020-10-08T11:05:11.0087216Z 3db6272dcbfa: Pull complete
2020-10-08T11:05:11.0112120Z Digest: sha256:8be26f81ffea54106bae012c6f349df70f4d5e7e2ec01b143c46e2c03b9e551d
2020-10-08T11:05:11.0130997Z Status: Downloaded newer image for registry:2
2020-10-08T11:05:11.3618264Z 3516823d48c6b985c791c423a03513981b91f239623c453021f622eb449dcb13
2020-10-08T11:05:12.2296337Z [submariner]$ registry_ip=172.17.0.3
2020-10-08T11:05:12.2311375Z [submariner]$ docker inspect -f {{.NetworkSettings.IPAddress}} kind-registry
2020-10-08T11:05:12.5391140Z [submariner]$ declare_cidrs
2020-10-08T11:05:12.5403564Z [submariner]$ declare_cidrs
2020-10-08T11:05:12.5413454Z [submariner]$ declare -gA cluster_CIDRs service_CIDRs global_CIDRs
2020-10-08T11:05:12.5425982Z [submariner]$ i=1
2020-10-08T11:05:12.5437779Z [submariner]$ [cluster1] add_cluster_cidrs 1 cluster1
2020-10-08T11:05:12.5450465Z [submariner]$ [cluster1] add_cluster_cidrs 1 cluster1
2020-10-08T11:05:12.5460999Z [submariner]$ [cluster1] local val=1
2020-10-08T11:05:12.5472592Z [submariner]$ [cluster1] local idx=cluster1
2020-10-08T11:05:12.5484704Z [submariner]$ [cluster1] cluster_CIDRs[cluster1]=10.241.0.0/16
2020-10-08T11:05:12.5500624Z [submariner]$ [cluster1] service_CIDRs[cluster1]=100.91.0.0/16
2020-10-08T11:05:12.5512490Z [submariner]$ [cluster1] i=2
2020-10-08T11:05:12.5526624Z [submariner]$ [cluster2] add_cluster_cidrs 2 cluster2
2020-10-08T11:05:12.5537026Z [submariner]$ [cluster2] add_cluster_cidrs 2 cluster2
2020-10-08T11:05:12.5549304Z [submariner]$ [cluster2] local val=2
2020-10-08T11:05:12.5559796Z [submariner]$ [cluster2] local idx=cluster2
2020-10-08T11:05:12.5573378Z [submariner]$ [cluster2] cluster_CIDRs[cluster2]=10.242.0.0/16
2020-10-08T11:05:12.5584128Z [submariner]$ [cluster2] service_CIDRs[cluster2]=100.92.0.0/16
2020-10-08T11:05:12.5598063Z [submariner]$ [cluster2] i=3
2020-10-08T11:05:12.5609792Z [submariner]$ [cluster3] add_cluster_cidrs 3 cluster3
2020-10-08T11:05:12.5622385Z [submariner]$ [cluster3] add_cluster_cidrs 3 cluster3
2020-10-08T11:05:12.5632496Z [submariner]$ [cluster3] local val=3
2020-10-08T11:05:12.5645249Z [submariner]$ [cluster3] local idx=cluster3
2020-10-08T11:05:12.5659656Z [submariner]$ [cluster3] cluster_CIDRs[cluster3]=10.243.0.0/16
2020-10-08T11:05:12.5674884Z [submariner]$ [cluster3] service_CIDRs[cluster3]=100.93.0.0/16
2020-10-08T11:05:12.5691298Z [submariner]$ [cluster3] i=4
2020-10-08T11:05:12.5708329Z [submariner]$ [cluster3] run_all_clusters with_retries 3 create_kind_cluster
2020-10-08T11:05:12.5723888Z [submariner]$ [cluster3] run_all_clusters with_retries 3 create_kind_cluster
2020-10-08T11:05:12.5740514Z [submariner]$ [cluster3] run_parallel cluster1 cluster2 cluster3 with_retries 3 create_kind_cluster
2020-10-08T11:05:12.5754189Z [submariner]$ [cluster3] run_parallel cluster1 cluster2 cluster3 cluster1 cluster2 cluster3 with_retries 3 create_kind_cluster
2020-10-08T11:05:12.5763655Z [submariner]$ [cluster3] local cmnd=with_retries
2020-10-08T11:05:12.5776048Z [submariner]$ [cluster3] declare -A pids
2020-10-08T11:05:12.5790691Z [submariner]$ [cluster3] eval echo cluster1 cluster2 cluster3
2020-10-08T11:05:12.5812699Z [submariner]$ [cluster1] pids[cluster1]=342
2020-10-08T11:05:12.5820154Z [submariner]$ [cluster1] set -o pipefail
2020-10-08T11:05:12.5839066Z [submariner]$ [cluster2] pids[cluster2]=346
2020-10-08T11:05:12.5846270Z [submariner]$ [cluster2] set -o pipefail
2020-10-08T11:05:12.5859898Z [submariner]$ [cluster1] with_retries 3 create_kind_cluster
2020-10-08T11:05:12.5871469Z [submariner]$ [cluster2] with_retries 3 create_kind_cluster
2020-10-08T11:05:12.5876693Z [submariner]$ [cluster3] set -o pipefail
2020-10-08T11:05:12.5891316Z [submariner]$ [cluster1] sed /\[cluster1]/!s/^/[cluster1] /
2020-10-08T11:05:12.5900800Z [submariner]$ [cluster2] sed /\[cluster2]/!s/^/[cluster2] /
2020-10-08T11:05:12.5905578Z [submariner]$ [cluster3] pids[cluster3]=349
2020-10-08T11:05:12.5917074Z [submariner]$ [cluster3] with_retries 3 create_kind_cluster
2020-10-08T11:05:12.5935595Z [submariner]$ [cluster3] wait 349
2020-10-08T11:05:12.5952392Z [submariner]$ [cluster3] sed /\[cluster3]/!s/^/[cluster3] /
2020-10-08T11:09:45.1935230Z [submariner]$ [cluster1] with_retries
2020-10-08T11:09:45.1936078Z [submariner]$ [cluster1] local retries
2020-10-08T11:09:45.1936682Z [submariner]$ [cluster1] retries=1 2 3
2020-10-08T11:09:45.1937263Z [submariner]$ [cluster1] eval echo {1..3}
2020-10-08T11:09:45.4573548Z [submariner]$ [cluster1] local cmnd=create_kind_cluster
2020-10-08T11:09:45.4574215Z [submariner]$ [cluster1] wait 388
2020-10-08T11:09:45.4574818Z [submariner]$ [cluster1] create_kind_cluster
2020-10-08T11:09:45.4575468Z [submariner]$ [cluster1] create_kind_cluster
2020-10-08T11:09:45.4576619Z [submariner]$ [cluster1] export KUBECONFIG=/go/src/github.com/submariner-io/submariner/output/kubeconfigs/kind-config-cluster1
2020-10-08T11:09:45.4578017Z [submariner]$ [cluster1] rm -f /go/src/github.com/submariner-io/submariner/output/kubeconfigs/kind-config-cluster1
2020-10-08T11:09:45.4578960Z [submariner]$ [cluster1] kind get clusters
2020-10-08T11:09:45.4579580Z [submariner]$ [cluster1] grep -q ^cluster1$
2020-10-08T11:09:45.4580007Z [cluster1] No kind clusters found.
2020-10-08T11:09:45.4580398Z [cluster1] Creating KIND cluster...
2020-10-08T11:09:45.4581026Z [submariner]$ [cluster1] generate_cluster_yaml
2020-10-08T11:09:45.4581704Z [submariner]$ [cluster1] generate_cluster_yaml
2020-10-08T11:09:45.4582350Z [submariner]$ [cluster1] local pod_cidr=10.241.0.0/16
2020-10-08T11:09:45.4582992Z [submariner]$ [cluster1] local service_cidr=100.91.0.0/16
2020-10-08T11:09:45.4583700Z [submariner]$ [cluster1] local dns_domain=cluster1.local
2020-10-08T11:09:45.4584411Z [submariner]$ [cluster1] local disable_cni=false
2020-10-08T11:09:45.4585222Z [submariner]$ [cluster1] local nodes
2020-10-08T11:09:45.4586074Z [submariner]$ [cluster1] nodes=
2020-10-08T11:09:45.4587930Z [cluster1] - role: control-plane
2020-10-08T11:09:45.4588568Z [submariner]$ [cluster1] nodes=
2020-10-08T11:09:45.4589112Z [cluster1] - role: control-plane
2020-10-08T11:09:45.4589625Z [cluster1] - role: worker
2020-10-08T11:09:45.4590060Z [cluster1] 
2020-10-08T11:09:45.4590465Z [cluster1] 
2020-10-08T11:09:45.4591116Z [submariner]$ [cluster1] eval echo "---
2020-10-08T11:09:45.4591665Z [cluster1] kind: Cluster
2020-10-08T11:09:45.4592341Z [cluster1] apiVersion: kind.x-k8s.io/v1alpha4
2020-10-08T11:09:45.4592771Z [cluster1] networking:
2020-10-08T11:09:45.4593225Z [cluster1]   disableDefaultCNI: ${disable_cni}
2020-10-08T11:09:45.4593686Z [cluster1]   podSubnet: ${pod_cidr}
2020-10-08T11:09:45.4594112Z [cluster1]   serviceSubnet: ${service_cidr}
2020-10-08T11:09:45.4594644Z [cluster1] containerdConfigPatches:
2020-10-08T11:09:45.4595204Z [cluster1]   - |-
2020-10-08T11:09:45.4595982Z [cluster1]     [plugins.\"io.containerd.grpc.v1.cri\".registry.mirrors.\"localhost:5000\"]
2020-10-08T11:09:45.4597169Z [cluster1]       endpoint = [\"http://${registry_ip}:5000\"]
2020-10-08T11:09:45.4597665Z [cluster1] kubeadmConfigPatches:
2020-10-08T11:09:45.4598310Z [cluster1]   - |
2020-10-08T11:09:45.4598716Z [cluster1]     apiVersion: kubeadm.k8s.io/v1beta2
2020-10-08T11:09:45.4599248Z [cluster1]     kind: ClusterConfiguration
2020-10-08T11:09:45.4599810Z [cluster1]     metadata:
2020-10-08T11:09:45.4600128Z [cluster1]       name: config
2020-10-08T11:09:45.4600463Z [cluster1]     networking:
2020-10-08T11:09:45.4600832Z [cluster1]       podSubnet: ${pod_cidr}
2020-10-08T11:09:45.4601261Z [cluster1]       serviceSubnet: ${service_cidr}
2020-10-08T11:09:45.4601699Z [cluster1]       dnsDomain: ${dns_domain}
2020-10-08T11:09:45.4602253Z [cluster1] nodes: ${nodes}"
2020-10-08T11:09:45.4603084Z [submariner]$ [cluster1] cat /opt/shipyard/scripts/resources/kind-cluster-config.yaml
2020-10-08T11:09:45.4603934Z [submariner]$ [cluster1] local image_flag=
2020-10-08T11:09:45.4604661Z [submariner]$ [cluster1] image_flag=--image=kindest/node:v1.17.0
2020-10-08T11:09:45.4605967Z [submariner]$ [cluster1] kind create cluster --image=kindest/node:v1.17.0 --name=cluster1 --config=/opt/shipyard/scripts/resources/cluster1-config.yaml
2020-10-08T11:09:45.4607120Z [cluster1] Creating cluster "cluster1" ...
2020-10-08T11:09:45.4608076Z [cluster1]  • Ensuring node image (kindest/node:v1.17.0) 🖼  ...
2020-10-08T11:09:45.4608920Z [cluster1]  ✓ Ensuring node image (kindest/node:v1.17.0) 🖼
2020-10-08T11:09:45.4609560Z [cluster1]  • Preparing nodes 📦 📦   ...
2020-10-08T11:09:45.4610090Z [cluster1]  ✓ Preparing nodes 📦 📦
2020-10-08T11:09:45.4610657Z [cluster1]  • Writing configuration 📜  ...
2020-10-08T11:09:45.4611255Z [cluster1]  ✓ Writing configuration 📜
2020-10-08T11:09:45.4611846Z [cluster1]  • Starting control-plane 🕹️  ...
2020-10-08T11:09:45.4612449Z [cluster1]  ✓ Starting control-plane 🕹️
2020-10-08T11:09:45.4613010Z [cluster1]  • Installing CNI 🔌  ...
2020-10-08T11:09:45.4613526Z [cluster1]  ✓ Installing CNI 🔌
2020-10-08T11:09:45.4614102Z [cluster1]  • Installing StorageClass 💾  ...
2020-10-08T11:09:45.4614720Z [cluster1]  ✓ Installing StorageClass 💾
2020-10-08T11:09:45.4615303Z [cluster1]  • Joining worker nodes 🚜  ...
2020-10-08T11:09:45.4615841Z [cluster1]  ✓ Joining worker nodes 🚜
2020-10-08T11:09:45.4616481Z [cluster1] Set kubectl context to "kind-cluster1"
2020-10-08T11:09:45.4616977Z [cluster1] You can now use your cluster with:
2020-10-08T11:09:45.4617320Z [cluster1]
2020-10-08T11:09:45.4618072Z [cluster1] kubectl cluster-info --context kind-cluster1
2020-10-08T11:09:45.4618508Z [cluster1]
2020-10-08T11:09:45.4619014Z [cluster1] Thanks for using kind! 😊
2020-10-08T11:09:45.4619595Z [submariner]$ [cluster1] kind_fixup_config
2020-10-08T11:09:45.4620192Z [submariner]$ [cluster1] kind_fixup_config
2020-10-08T11:09:45.4620821Z [submariner]$ [cluster1] local master_ip=172.17.0.10
2020-10-08T11:09:45.4621909Z [submariner]$ [cluster1] docker inspect -f {{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}} cluster1-control-plane
2020-10-08T11:09:45.4622880Z [submariner]$ [cluster1] head -n 1
2020-10-08T11:09:45.6603018Z [submariner]$ [cluster1] sed -i -- s/server: .*/server: https:\/\/172.17.0.10:6443/g /go/src/github.com/submariner-io/submariner/output/kubeconfigs/kind-config-cluster1
2020-10-08T11:09:45.6604879Z [submariner]$ [cluster1] sed -i -- s/user: kind-.*/user: cluster1/g /go/src/github.com/submariner-io/submariner/output/kubeconfigs/kind-config-cluster1
2020-10-08T11:09:45.6606505Z [submariner]$ [cluster1] sed -i -- s/name: kind-.*/name: cluster1/g /go/src/github.com/submariner-io/submariner/output/kubeconfigs/kind-config-cluster1
2020-10-08T11:09:45.6608065Z [submariner]$ [cluster1] sed -i -- s/cluster: kind-.*/cluster: cluster1/g /go/src/github.com/submariner-io/submariner/output/kubeconfigs/kind-config-cluster1
2020-10-08T11:09:45.6609750Z [submariner]$ [cluster1] sed -i -- s/current-context: .*/current-context: cluster1/g /go/src/github.com/submariner-io/submariner/output/kubeconfigs/kind-config-cluster1
2020-10-08T11:09:45.6611294Z [submariner]$ [cluster1] chmod a+r /go/src/github.com/submariner-io/submariner/output/kubeconfigs/kind-config-cluster1
2020-10-08T11:09:45.6612237Z [submariner]$ [cluster1] wait 1807
2020-10-08T11:09:45.6613950Z [submariner]$ [cluster1] deploy_cluster_capabilities
2020-10-08T11:09:45.6614839Z [submariner]$ [cluster1] deploy_cluster_capabilities
2020-10-08T11:09:45.6615498Z [submariner]$ [cluster1] deploy_cni
2020-10-08T11:09:45.6616071Z [submariner]$ [cluster1] deploy_cni
2020-10-08T11:09:45.6616634Z [submariner]$ [cluster1] return 0
2020-10-08T11:09:45.6617189Z [submariner]$ [cluster1] return 0
2020-10-08T11:10:03.4853610Z [submariner]$ [cluster3] with_retries
2020-10-08T11:10:03.4854309Z [submariner]$ [cluster3] local retries
2020-10-08T11:10:03.4854901Z [submariner]$ [cluster3] retries=1 2 3
2020-10-08T11:10:03.4855491Z [submariner]$ [cluster3] eval echo {1..3}
2020-10-08T11:10:03.4856153Z [submariner]$ [cluster3] local cmnd=create_kind_cluster
2020-10-08T11:10:03.4856789Z [submariner]$ [cluster3] wait 384
2020-10-08T11:10:03.4857392Z [submariner]$ [cluster3] create_kind_cluster
2020-10-08T11:10:03.4858387Z [submariner]$ [cluster3] create_kind_cluster
2020-10-08T11:10:03.4859497Z [submariner]$ [cluster3] export KUBECONFIG=/go/src/github.com/submariner-io/submariner/output/kubeconfigs/kind-config-cluster3
2020-10-08T11:10:03.4860897Z [submariner]$ [cluster3] rm -f /go/src/github.com/submariner-io/submariner/output/kubeconfigs/kind-config-cluster3
2020-10-08T11:10:03.4861858Z [submariner]$ [cluster3] kind get clusters
2020-10-08T11:10:03.4862489Z [submariner]$ [cluster3] grep -q ^cluster3$
2020-10-08T11:10:03.4862899Z [cluster3] No kind clusters found.
2020-10-08T11:10:03.4863301Z [cluster3] Creating KIND cluster...
2020-10-08T11:10:03.4863938Z [submariner]$ [cluster3] generate_cluster_yaml
2020-10-08T11:10:03.4864607Z [submariner]$ [cluster3] generate_cluster_yaml
2020-10-08T11:10:03.4865261Z [submariner]$ [cluster3] local pod_cidr=10.243.0.0/16
2020-10-08T11:10:03.4865908Z [submariner]$ [cluster3] local service_cidr=100.93.0.0/16
2020-10-08T11:10:03.4867756Z [submariner]$ [cluster3] local dns_domain=cluster3.local
2020-10-08T11:10:03.4868986Z [submariner]$ [cluster3] local disable_cni=false
2020-10-08T11:10:03.4869641Z [submariner]$ [cluster3] disable_cni=true
2020-10-08T11:10:03.4871632Z [submariner]$ [cluster3] local nodes
2020-10-08T11:10:03.4872371Z [submariner]$ [cluster3] nodes=
2020-10-08T11:10:03.4872946Z [cluster3] - role: control-plane
2020-10-08T11:10:03.4873506Z [submariner]$ [cluster3] nodes=
2020-10-08T11:10:03.4874059Z [cluster3] - role: control-plane
2020-10-08T11:10:03.4874579Z [cluster3] - role: worker
2020-10-08T11:10:03.4875098Z [submariner]$ [cluster3] nodes=
2020-10-08T11:10:03.4875632Z [cluster3] - role: control-plane
2020-10-08T11:10:03.4876143Z [cluster3] - role: worker
2020-10-08T11:10:03.4897503Z [cluster3] - role: worker
2020-10-08T11:10:03.4898020Z [cluster3] 
2020-10-08T11:10:03.4898463Z [cluster3] 
2020-10-08T11:10:03.4899175Z [submariner]$ [cluster3] eval echo "---
2020-10-08T11:10:03.4899573Z [cluster3] kind: Cluster
2020-10-08T11:10:03.4900204Z [cluster3] apiVersion: kind.x-k8s.io/v1alpha4
2020-10-08T11:10:03.4900631Z [cluster3] networking:
2020-10-08T11:10:03.4901083Z [cluster3]   disableDefaultCNI: ${disable_cni}
2020-10-08T11:10:03.4901543Z [cluster3]   podSubnet: ${pod_cidr}
2020-10-08T11:10:03.4901969Z [cluster3]   serviceSubnet: ${service_cidr}
2020-10-08T11:10:03.4902501Z [cluster3] containerdConfigPatches:
2020-10-08T11:10:03.4903083Z [cluster3]   - |-
2020-10-08T11:10:03.4903824Z [cluster3]     [plugins.\"io.containerd.grpc.v1.cri\".registry.mirrors.\"localhost:5000\"]
2020-10-08T11:10:03.4904559Z [cluster3]       endpoint = [\"http://${registry_ip}:5000\"]
2020-10-08T11:10:03.4905051Z [cluster3] kubeadmConfigPatches:
2020-10-08T11:10:03.4905617Z [cluster3]   - |
2020-10-08T11:10:03.4906014Z [cluster3]     apiVersion: kubeadm.k8s.io/v1beta2
2020-10-08T11:10:03.4906552Z [cluster3]     kind: ClusterConfiguration
2020-10-08T11:10:03.4906984Z [cluster3]     metadata:
2020-10-08T11:10:03.4907307Z [cluster3]       name: config
2020-10-08T11:10:03.4907626Z [cluster3]     networking:
2020-10-08T11:10:03.4907996Z [cluster3]       podSubnet: ${pod_cidr}
2020-10-08T11:10:03.4908423Z [cluster3]       serviceSubnet: ${service_cidr}
2020-10-08T11:10:03.4908856Z [cluster3]       dnsDomain: ${dns_domain}
2020-10-08T11:10:03.4909394Z [cluster3] nodes: ${nodes}"
2020-10-08T11:10:03.4910313Z [submariner]$ [cluster3] cat /opt/shipyard/scripts/resources/kind-cluster-config.yaml
2020-10-08T11:10:03.4911171Z [submariner]$ [cluster3] local image_flag=
2020-10-08T11:10:03.4911915Z [submariner]$ [cluster3] image_flag=--image=kindest/node:v1.17.0
2020-10-08T11:10:03.4913368Z [submariner]$ [cluster3] kind create cluster --image=kindest/node:v1.17.0 --name=cluster3 --config=/opt/shipyard/scripts/resources/cluster3-config.yaml
2020-10-08T11:10:03.4914360Z [cluster3] Creating cluster "cluster3" ...
2020-10-08T11:10:03.4915142Z [cluster3]  • Ensuring node image (kindest/node:v1.17.0) 🖼  ...
2020-10-08T11:10:03.4915871Z [cluster3]  ✓ Ensuring node image (kindest/node:v1.17.0) 🖼
2020-10-08T11:10:03.4916935Z [cluster3]  • Preparing nodes 📦 📦 📦   ...
2020-10-08T11:10:03.4917552Z [cluster3]  ✓ Preparing nodes 📦 📦 📦
2020-10-08T11:10:03.4918131Z [cluster3]  • Writing configuration 📜  ...
2020-10-08T11:10:03.4918792Z [cluster3]  ✓ Writing configuration 📜
2020-10-08T11:10:03.4919409Z [cluster3]  • Starting control-plane 🕹️  ...
2020-10-08T11:10:03.4920009Z [cluster3]  ✓ Starting control-plane 🕹️
2020-10-08T11:10:03.4920633Z [cluster3]  • Installing StorageClass 💾  ...
2020-10-08T11:10:03.4921260Z [cluster3]  ✓ Installing StorageClass 💾
2020-10-08T11:10:03.4921852Z [cluster3]  • Joining worker nodes 🚜  ...
2020-10-08T11:10:03.4922398Z [cluster3]  ✓ Joining worker nodes 🚜
2020-10-08T11:10:03.4923019Z [cluster3] Set kubectl context to "kind-cluster3"
2020-10-08T11:10:03.4923544Z [cluster3] You can now use your cluster with:
2020-10-08T11:10:03.4923903Z [cluster3]
2020-10-08T11:10:03.4924505Z [cluster3] kubectl cluster-info --context kind-cluster3
2020-10-08T11:10:03.4924928Z [cluster3]
2020-10-08T11:10:03.4925410Z [cluster3] Thanks for using kind! 😊
2020-10-08T11:10:03.4925996Z [submariner]$ [cluster3] kind_fixup_config
2020-10-08T11:10:03.4926619Z [submariner]$ [cluster3] kind_fixup_config
2020-10-08T11:10:03.4927253Z [submariner]$ [cluster3] local master_ip=172.17.0.9
2020-10-08T11:10:03.4928346Z [submariner]$ [cluster3] docker inspect -f {{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}} cluster3-control-plane
2020-10-08T11:10:03.5308468Z [submariner]$ [cluster3] head -n 1[0[submariner]$ [cluster2] with_retries
2020-10-08T11:10:03.5309278Z [submariner]$ [cluster2] local retries
2020-10-08T11:10:03.5309885Z [submariner]$ [cluster2] retries=1 2 3
2020-10-08T11:10:03.5310556Z [submariner]$ [cluster2] eval echo {1..3}
2020-10-08T11:10:03.5311415Z [submariner]$ [cluster2] local cmnd=create_kind_cluster
2020-10-08T11:10:03.5312176Z [submariner]$ [cluster2] create_kind_cluster
2020-10-08T11:10:03.5313912Z [submariner]$ [cluster2] wait 390
2020-10-08T11:10:03.5314522Z [submariner]$ [cluster2] create_kind_cluster
2020-10-08T11:10:03.5315858Z [submariner]$ [cluster2] export KUBECONFIG=/go/src/github.com/submariner-io/submariner/output/kubeconfigs/kind-config-cluster2
2020-10-08T11:10:03.5317586Z [submariner]$ [cluster2] rm -f /go/src/github.com/submariner-io/submariner/output/kubeconfigs/kind-config-cluster2
2020-10-08T11:10:03.5318555Z [submariner]$ [cluster2] kind get clusters
2020-10-08T11:10:03.5319179Z [submariner]$ [cluster2] grep -q ^cluster2$
2020-10-08T11:10:03.5319605Z [cluster2] No kind clusters found.
2020-10-08T11:10:03.5320012Z [cluster2] Creating KIND cluster...
2020-10-08T11:10:03.5320859Z [submariner]$ [cluster2] generate_cluster_yaml
2020-10-08T11:10:03.5321558Z [submariner]$ [cluster2] generate_cluster_yaml
2020-10-08T11:10:03.5322340Z [submariner]$ [cluster2] local pod_cidr=10.242.0.0/16
2020-10-08T11:10:03.5323025Z [submariner]$ [cluster2] local service_cidr=100.92.0.0/16
2020-10-08T11:10:03.5323723Z [submariner]$ [cluster2] local dns_domain=cluster2.local
2020-10-08T11:10:03.5324433Z [submariner]$ [cluster2] local disable_cni=false
2020-10-08T11:10:03.5325068Z [submariner]$ [cluster2] disable_cni=true
2020-10-08T11:10:03.5325667Z [submariner]$ [cluster2] local nodes
2020-10-08T11:10:03.5326230Z [submariner]$ [cluster2] nodes=
2020-10-08T11:10:03.5326784Z [cluster2] - role: control-plane
2020-10-08T11:10:03.5327334Z [submariner]$ [cluster2] nodes=
2020-10-08T11:10:03.5327877Z [cluster2] - role: control-plane
2020-10-08T11:10:03.5328377Z [cluster2] - role: worker
2020-10-08T11:10:03.5328888Z [submariner]$ [cluster2] nodes=
2020-10-08T11:10:03.5329661Z [cluster2] - role: control-plane
2020-10-08T11:10:03.5330175Z [cluster2] - role: worker
2020-10-08T11:10:03.5330651Z [cluster2] - role: worker
2020-10-08T11:10:03.5331088Z [cluster2] 
2020-10-08T11:10:03.5331495Z [cluster2] 
2020-10-08T11:10:03.5331979Z [submariner]$ [cluster2] eval echo "---
2020-10-08T11:10:03.5332345Z [cluster2] kind: Cluster
2020-10-08T11:10:03.5332930Z [cluster2] apiVersion: kind.x-k8s.io/v1alpha4
2020-10-08T11:10:03.5333357Z [cluster2] networking:
2020-10-08T11:10:03.5333805Z [cluster2]   disableDefaultCNI: ${disable_cni}
2020-10-08T11:10:03.5334268Z [cluster2]   podSubnet: ${pod_cidr}
2020-10-08T11:10:03.5334693Z [cluster2]   serviceSubnet: ${service_cidr}
2020-10-08T11:10:03.5336860Z [cluster2] containerdConfigPatches:
2020-10-08T11:10:03.5337621Z [cluster2]   - |-
2020-10-08T11:10:03.5338315Z [cluster2]     [plugins.\"io.containerd.grpc.v1.cri\".registry.mirrors.\"localhost:5000\"]
2020-10-08T11:10:03.5339061Z [cluster2]       endpoint = [\"http://${registry_ip}:5000\"]
2020-10-08T11:10:03.5339557Z [cluster2] kubeadmConfigPatches:
2020-10-08T11:10:03.5340123Z [cluster2]   - |
2020-10-08T11:10:03.5340522Z [cluster2]     apiVersion: kubeadm.k8s.io/v1beta2
2020-10-08T11:10:03.5341050Z [cluster2]     kind: ClusterConfiguration
2020-10-08T11:10:03.5341463Z [cluster2]     metadata:
2020-10-08T11:10:03.5341790Z [cluster2]       name: config
2020-10-08T11:10:03.5342126Z [cluster2]     networking:
2020-10-08T11:10:03.5342491Z [cluster2]       podSubnet: ${pod_cidr}
2020-10-08T11:10:03.5342918Z [cluster2]       serviceSubnet: ${service_cidr}
2020-10-08T11:10:03.5343556Z [cluster2]       dnsDomain: ${dns_domain}
2020-10-08T11:10:03.5344129Z [cluster2] nodes: ${nodes}"
2020-10-08T11:10:03.5344962Z [submariner]$ [cluster2] cat /opt/shipyard/scripts/resources/kind-cluster-config.yaml
2020-10-08T11:10:03.5345802Z [submariner]$ [cluster2] local image_flag=
2020-10-08T11:10:03.5346538Z [submariner]$ [cluster2] image_flag=--image=kindest/node:v1.17.0
2020-10-08T11:10:03.5347903Z [submariner]$ [cluster2] kind create cluster --image=kindest/node:v1.17.0 --name=cluster2 --config=/opt/shipyard/scripts/resources/cluster2-config.yaml
2020-10-08T11:10:03.5348776Z [cluster2] Creating cluster "cluster2" ...
2020-10-08T11:10:03.5349522Z [cluster2]  • Ensuring node image (kindest/node:v1.17.0) 🖼  ...
2020-10-08T11:10:03.5350307Z [cluster2]  ✓ Ensuring node image (kindest/node:v1.17.0) 🖼
2020-10-08T11:10:03.5350943Z [cluster2]  • Preparing nodes 📦 📦 📦   ...
2020-10-08T11:10:03.5351480Z [cluster2]  ✓ Preparing nodes 📦 📦 📦
2020-10-08T11:10:03.5352050Z [cluster2]  • Writing configuration 📜  ...
2020-10-08T11:10:03.5352632Z [cluster2]  ✓ Writing configuration 📜
2020-10-08T11:10:03.5353238Z [cluster2]  • Starting control-plane 🕹️  ...
2020-10-08T11:10:03.5353845Z [cluster2]  ✓ Starting control-plane 🕹️
2020-10-08T11:10:03.5354463Z [cluster2]  • Installing StorageClass 💾  ...
2020-10-08T11:10:03.5355140Z [cluster2]  ✓ Installing StorageClass 💾
2020-10-08T11:10:03.5355750Z [cluster2]  • Joining worker nodes 🚜  ...
2020-10-08T11:10:03.5356303Z [cluster2]  ✓ Joining worker nodes 🚜
2020-10-08T11:10:03.5391346Z [cluster2] Set kubectl context to "kind-cluster2"
2020-10-08T11:10:03.5391825Z [cluster2] You can now use your cluster with:
2020-10-08T11:10:03.5392168Z [cluster2]
2020-10-08T11:10:03.5392788Z [cluster2] kubectl cluster-info --context kind-cluster2
2020-10-08T11:10:03.5393210Z [cluster2]
2020-10-08T11:10:03.5394089Z [cluster2] Have a question, bug, or feature request? Let us know! https://kind.sigs.k8s.io/#community 🙂
2020-10-08T11:10:03.5394987Z [submariner]$ [cluster2] kind_fixup_config
2020-10-08T11:10:03.5395607Z [submariner]$ [cluster2] kind_fixup_config
2020-10-08T11:10:03.5396241Z [submariner]$ [cluster2] local master_ip=172.17.0.11
2020-10-08T11:11:00.5254342Z [submariner]$ [cluster2] docker inspect -f {{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}} cluster2-control-plane
2020-10-08T11:11:00.5255916Z [submariner]$ [cluster2] head -n 1
2020-10-08T11:11:00.5257074Z [submariner]$ [cluster2] sed -i -- s/server: .*/server: https:\/\/172.17.0.11:6443/g /go/src/github.com/submariner-io/submariner/output/kubeconfigs/kind-config-cluster2
2020-10-08T11:11:00.5258635Z [submariner]$ [cluster2] sed -i -- s/user: kind-.*/user: cluster2/g /go/src/github.com/submariner-io/submariner/output/kubeconfigs/kind-config-cluster2
2020-10-08T11:11:00.5260178Z [submariner]$ [cluster2] sed -i -- s/name: kind-.*/name: cluster2/g /go/src/github.com/submariner-io/submariner/output/kubeconfigs/kind-config-cluster2
2020-10-08T11:11:00.5261750Z [submariner]$ [cluster2] sed -i -- s/cluster: kind-.*/cluster: cluster2/g /go/src/github.com/submariner-io/submariner/output/kubeconfigs/kind-config-cluster2
2020-10-08T11:11:00.5263436Z [submariner]$ [cluster2] sed -i -- s/current-context: .*/current-context: cluster2/g /go/src/github.com/submariner-io/submariner/output/kubeconfigs/kind-config-cluster2
2020-10-08T11:11:00.5265007Z [submariner]$ [cluster2] chmod a+r /go/src/github.com/submariner-io/submariner/output/kubeconfigs/kind-config-cluster2
2020-10-08T11:11:00.5265952Z [submariner]$ [cluster2] wait 2041
2020-10-08T11:11:00.5266610Z [submariner]$ [cluster2] deploy_cluster_capabilities
2020-10-08T11:11:00.5267365Z [submariner]$ [cluster2] deploy_cluster_capabilities
2020-10-08T11:11:00.5268032Z [submariner]$ [cluster2] deploy_cni
2020-10-08T11:11:00.5268625Z [submariner]$ [cluster2] deploy_cni
2020-10-08T11:11:00.5269254Z [submariner]$ [cluster2] eval deploy_weave_cni
2020-10-08T11:11:00.5269894Z [submariner]$ [cluster2] deploy_weave_cni
2020-10-08T11:11:00.5270522Z [submariner]$ [cluster2] deploy_weave_cni
2020-10-08T11:11:00.5270962Z [cluster2] Applying weave network...
2020-10-08T11:11:00.5272021Z [submariner]$ [cluster2] kubectl apply -f https://cloud.weave.works/k8s/net?k8s-version=v1.17.0&env.IPALLOC_RANGE=10.242.0.0/16
2020-10-08T11:11:00.5273610Z [submariner]$ [cluster2] kubectl apply -f https://cloud.weave.works/k8s/net?k8s-version=v1.17.0&env.IPALLOC_RANGE=10.242.0.0/16
2020-10-08T11:11:00.5275254Z [submariner]$ [cluster2] command kubectl --context=cluster2 apply -f https://cloud.weave.works/k8s/net?k8s-version=v1.17.0&env.IPALLOC_RANGE=10.242.0.0/16
2020-10-08T11:11:00.5276348Z [cluster2] serviceaccount/weave-net created
2020-10-08T11:11:00.5277810Z [cluster2] clusterrole.rbac.authorization.k8s.io/weave-net created
2020-10-08T11:11:00.5279264Z [cluster2] clusterrolebinding.rbac.authorization.k8s.io/weave-net created
2020-10-08T11:11:00.5280574Z [cluster2] role.rbac.authorization.k8s.io/weave-net created
2020-10-08T11:11:00.5281745Z [cluster2] rolebinding.rbac.authorization.k8s.io/weave-net created
2020-10-08T11:11:00.5282971Z [cluster2] daemonset.apps/weave-net created
2020-10-08T11:11:00.5283669Z [cluster2] Waiting for weave-net pods to be ready...
2020-10-08T11:11:00.5284594Z [submariner]$ [cluster2] kubectl wait --for=condition=Ready pods -l name=weave-net -n kube-system --timeout=5m
2020-10-08T11:11:00.5285689Z [submariner]$ [cluster2] kubectl wait --for=condition=Ready pods -l name=weave-net -n kube-system --timeout=5m
2020-10-08T11:11:00.5286896Z [submariner]$ [cluster2] command kubectl --context=cluster2 wait --for=condition=Ready pods -l name=weave-net -n kube-system --timeout=5m
2020-10-08T11:11:00.5288143Z [submariner]$ [cluster2] kubectl --context=cluster2 wait --for=condition=Ready pods -l name=weave-net -n kube-system --timeout=5m
2020-10-08T11:11:00.5289064Z [cluster2] pod/weave-net-66csr condition met
2020-10-08T11:11:00.5289749Z [cluster2] pod/weave-net-ggqbp condition met
2020-10-08T11:11:00.5290436Z [cluster2] pod/weave-net-w2lrf condition met
2020-10-08T11:11:00.5291142Z [cluster2] Waiting for core-dns deployment to be ready...
2020-10-08T11:11:00.5292030Z [submariner]$ [cluster2] kubectl -n kube-system rollout status deploy/coredns --timeout=5m
2020-10-08T11:11:00.5293217Z [submariner]$ [cluster2] kubectl -n kube-system rollout status deploy/coredns --timeout=5m
2020-10-08T11:11:00.5294305Z [submariner]$ [cluster2] command kubectl --context=cluster2 -n kube-system rollout status deploy/coredns --timeout=5m
2020-10-08T11:11:00.5295426Z [submariner]$ [cluster2] kubectl --context=cluster2 -n kube-system rollout status deploy/coredns --timeout=5m
2020-10-08T11:11:00.5296273Z [cluster2] Waiting for deployment "coredns" rollout to finish: 0 of 2 updated replicas are available...
2020-10-08T11:11:00.5297084Z [cluster2] Waiting for deployment "coredns" rollout to finish: 1 of 2 updated replicas are available...
2020-10-08T11:11:00.5297770Z [cluster2] deployment "coredns" successfully rolled out
2020-10-08T11:11:00.5298435Z [submariner]$ [cluster2] return 0
2020-10-08T11:11:04.4345523Z m
2020-10-08T11:11:04.4347671Z [submariner]$ [cluster3] sed -i -- s/server: .*/server: https:\/\/172.17.0.9:6443/g /go/src/github.com/submariner-io/submariner/output/kubeconfigs/kind-config-cluster3
2020-10-08T11:11:04.4349532Z [submariner]$ [cluster3] sed -i -- s/user: kind-.*/user: cluster3/g /go/src/github.com/submariner-io/submariner/output/kubeconfigs/kind-config-cluster3
2020-10-08T11:11:04.4384241Z [submariner]$ [cluster3] wait 346
2020-10-08T11:11:04.4385438Z [submariner]$ [cluster3] sed -i -- s/name: kind-.*/name: cluster3/g /go/src/github.com/submariner-io/submariner/output/kubeconfigs/kind-config-cluster3
2020-10-08T11:11:04.4387027Z [submariner]$ [cluster3] sed -i -- s/cluster: kind-.*/cluster: cluster3/g /go/src/github.com/submariner-io/submariner/output/kubeconfigs/kind-config-cluster3
2020-10-08T11:11:04.4388719Z [submariner]$ [cluster3] sed -i -- s/current-context: .*/current-context: cluster3/g /go/src/github.com/submariner-io/submariner/output/kubeconfigs/kind-config-cluster3
2020-10-08T11:11:04.4390826Z [submariner]$ [cluster3] chmod a+r /go/src/github.com/submariner-io/submariner/output/kubeconfigs/kind-config-cluster3
2020-10-08T11:11:04.4391841Z [submariner]$ [cluster3] wait 2045
2020-10-08T11:11:04.4392502Z [submariner]$ [cluster3] deploy_cluster_capabilities
2020-10-08T11:11:04.4393240Z [submariner]$ [cluster3] deploy_cluster_capabilities
2020-10-08T11:11:04.4393884Z [submariner]$ [cluster3] deploy_cni
2020-10-08T11:11:04.4394923Z [submariner]$ [cluster3] deploy_cni
2020-10-08T11:11:04.4395551Z [submariner]$ [cluster3] eval deploy_weave_cni
2020-10-08T11:11:04.4396332Z [submariner]$ [cluster3] deploy_weave_cni
2020-10-08T11:11:04.4397132Z [submariner]$ [cluster3] deploy_weave_cni
2020-10-08T11:11:04.4397575Z [cluster3] Applying weave network...
2020-10-08T11:11:04.4399599Z [submariner]$ [cluster3] kubectl apply -f https://cloud.weave.works/k8s/net?k8s-version=v1.17.0&env.IPALLOC_RANGE=10.243.0.0/16
2020-10-08T11:11:04.4401025Z [submariner]$ [cluster3] kubectl apply -f https://cloud.weave.works/k8s/net?k8s-version=v1.17.0&env.IPALLOC_RANGE=10.243.0.0/16
2020-10-08T11:11:04.4403735Z [submariner]$ [cluster3] command kubectl --context=cluster3 apply -f https://cloud.weave.works/k8s/net?k8s-version=v1.17.0&env.IPALLOC_RANGE=10.243.0.0/16
2020-10-08T11:11:04.4404815Z [cluster3] serviceaccount/weave-net created
2020-10-08T11:11:04.4405818Z [cluster3] clusterrole.rbac.authorization.k8s.io/weave-net created
2020-10-08T11:11:04.4407246Z [cluster3] clusterrolebinding.rbac.authorization.k8s.io/weave-net created
2020-10-08T11:11:04.4408750Z [cluster3] role.rbac.authorization.k8s.io/weave-net created
2020-10-08T11:11:04.4409886Z [cluster3] rolebinding.rbac.authorization.k8s.io/weave-net created
2020-10-08T11:11:04.4410848Z [cluster3] daemonset.apps/weave-net created
2020-10-08T11:11:04.4411515Z [cluster3] Waiting for weave-net pods to be ready...
2020-10-08T11:11:04.4459294Z [submariner]$ [cluster3] kubectl wait --for=condition=Ready pods -l name=weave-net -n kube-system --timeout=5m
2020-10-08T11:11:04.4460742Z [submariner]$ [cluster3] kubectl wait --for=condition=Ready pods -l name=weave-net -n kube-system --timeout=5m
2020-10-08T11:11:04.4462109Z [submariner]$ [cluster3] command kubectl --context=cluster3 wait --for=condition=Ready pods -l name=weave-net -n kube-system --timeout=5m
2020-10-08T11:11:04.4463695Z [submariner]$ [cluster3] kubectl --context=cluster3 wait --for=condition=Ready pods -l name=weave-net -n kube-system --timeout=5m
2020-10-08T11:11:04.4465297Z [cluster3] pod/weave-net-mj2rh condition met
2020-10-08T11:11:04.4465982Z [cluster3] pod/weave-net-qgtj8 condition met
2020-10-08T11:11:04.4466650Z [cluster3] pod/weave-net-zsslf condition met
2020-10-08T11:11:04.4467332Z [cluster3] Waiting for core-dns deployment to be ready...
2020-10-08T11:11:04.4468209Z [submariner]$ [cluster3] kubectl -n kube-system rollout status deploy/coredns --timeout=5m
2020-10-08T11:11:04.4469199Z [submariner]$ [cluster3] kubectl -n kube-system rollout status deploy/coredns --timeout=5m
2020-10-08T11:11:04.4470276Z [submariner]$ [cluster3] command kubectl --context=cluster3 -n kube-system rollout status deploy/coredns --timeout=5m
2020-10-08T11:11:04.4471396Z [submariner]$ [cluster3] kubectl --context=cluster3 -n kube-system rollout status deploy/coredns --timeout=5m
2020-10-08T11:11:04.4472242Z [cluster3] Waiting for deployment "coredns" rollout to finish: 0 of 2 updated replicas are available...
2020-10-08T11:11:04.4473054Z [cluster3] Waiting for deployment "coredns" rollout to finish: 1 of 2 updated replicas are available...
2020-10-08T11:11:04.4473740Z [cluster3] deployment "coredns" successfully rolled out
2020-10-08T11:11:04.4474387Z [submariner]$ [cluster3] return 0
2020-10-08T11:11:04.4475033Z [submariner]$ [cluster3] wait 342
2020-10-08T11:11:04.4475647Z [submariner]$ [cluster3] print_clusters_message
2020-10-08T11:11:04.4497373Z [submariner]$ [cluster3] print_clusters_message
2020-10-08T11:11:04.4520559Z [submariner]$ [cluster3] cat
2020-10-08T11:11:04.4534859Z Your virtual cluster(s) are deployed and working properly and can be accessed with:
2020-10-08T11:11:04.4535286Z
2020-10-08T11:11:04.4536181Z export KUBECONFIG=$(find $(git rev-parse --show-toplevel)/output/kubeconfigs/ -type f -printf %p:)
2020-10-08T11:11:04.4536630Z
2020-10-08T11:11:04.4537219Z $ kubectl config use-context cluster1 # or cluster2, cluster3..
2020-10-08T11:11:04.4537575Z
2020-10-08T11:11:04.4537875Z To clean everthing up, just run: make cleanup
2020-10-08T11:11:04.4567895Z go mod download
2020-10-08T11:11:21.3585931Z go mod vendor
2020-10-08T11:11:27.2550917Z fatal: No names found, cannot describe anything.
2020-10-08T11:11:27.2559521Z /opt/shipyard/scripts/compile.sh bin/submariner-engine main.go --ldflags '-X main.VERSION='
2020-10-08T11:11:27.3269647Z [submariner]$ mkdir -p bin
2020-10-08T11:11:27.3292800Z Building 'bin/submariner-engine' (ldflags: '-X main.VERSION=')
2020-10-08T11:11:27.3304378Z [submariner]$ ldflags=-s -w -X main.VERSION=
2020-10-08T11:11:27.3319226Z [submariner]$ CGO_ENABLED=0 go build -ldflags -s -w -X main.VERSION= -o bin/submariner-engine main.go
2020-10-08T11:12:37.7026891Z [submariner]$ upx bin/submariner-engine
2020-10-08T11:12:44.4728429Z                        Ultimate Packer for eXecutables
2020-10-08T11:12:44.4729547Z                           Copyright (C) 1996 - 2020
2020-10-08T11:12:44.4729979Z UPX 3.96        Markus Oberhumer, Laszlo Molnar & John Reiser   Jan 23rd 2020
2020-10-08T11:12:44.4730309Z
2020-10-08T11:12:44.4730586Z         File size         Ratio      Format      Name
2020-10-08T11:12:44.4731300Z    --------------------   ------   -----------   -----------
2020-10-08T11:12:44.4731867Z   31801344 ->   9689168   30.47%   linux/amd64   submariner-engine
2020-10-08T11:12:44.4732161Z
2020-10-08T11:12:44.4732388Z Packed 1 file.
2020-10-08T11:12:44.5209179Z fatal: No names found, cannot describe anything.
2020-10-08T11:12:44.5217443Z /opt/shipyard/scripts/compile.sh bin/submariner-route-agent ./pkg/routeagent/main.go --ldflags '-X main.VERSION='
2020-10-08T11:12:44.5707505Z [submariner]$ mkdir -p bin
2020-10-08T11:12:44.5724212Z Building 'bin/submariner-route-agent' (ldflags: '-X main.VERSION=')
2020-10-08T11:12:44.5733500Z [submariner]$ ldflags=-s -w -X main.VERSION=
2020-10-08T11:12:44.5744921Z [submariner]$ CGO_ENABLED=0 go build -ldflags -s -w -X main.VERSION= -o bin/submariner-route-agent ./pkg/routeagent/main.go
2020-10-08T11:13:09.0401024Z [submariner]$ upx bin/submariner-route-agent
2020-10-08T11:13:15.4283331Z                        Ultimate Packer for eXecutables
2020-10-08T11:13:15.4284886Z                           Copyright (C) 1996 - 2020
2020-10-08T11:13:15.4285791Z UPX 3.96        Markus Oberhumer, Laszlo Molnar & John Reiser   Jan 23rd 2020
2020-10-08T11:13:15.4286314Z
2020-10-08T11:13:15.4286779Z         File size         Ratio      Format      Name
2020-10-08T11:13:15.4287533Z    --------------------   ------   -----------   -----------
2020-10-08T11:13:15.4288365Z   31866880 ->   9445496   29.64%   linux/amd64   submariner-route-agent
2020-10-08T11:13:15.4288861Z
2020-10-08T11:13:15.4289216Z Packed 1 file.
2020-10-08T11:13:15.4742899Z fatal: No names found, cannot describe anything.
2020-10-08T11:13:15.4753249Z /opt/shipyard/scripts/compile.sh bin/submariner-globalnet ./pkg/globalnet/main.go --ldflags '-X main.VERSION='
2020-10-08T11:13:15.5227210Z [submariner]$ mkdir -p bin
2020-10-08T11:13:15.5244613Z Building 'bin/submariner-globalnet' (ldflags: '-X main.VERSION=')
2020-10-08T11:13:15.5255844Z [submariner]$ ldflags=-s -w -X main.VERSION=
2020-10-08T11:13:15.5268065Z [submariner]$ CGO_ENABLED=0 go build -ldflags -s -w -X main.VERSION= -o bin/submariner-globalnet ./pkg/globalnet/main.go
2020-10-08T11:13:20.0214249Z [submariner]$ upx bin/submariner-globalnet
2020-10-08T11:13:26.4846682Z                        Ultimate Packer for eXecutables
2020-10-08T11:13:26.4847840Z                           Copyright (C) 1996 - 2020
2020-10-08T11:13:26.4848273Z UPX 3.96        Markus Oberhumer, Laszlo Molnar & John Reiser   Jan 23rd 2020
2020-10-08T11:13:26.4848587Z
2020-10-08T11:13:26.4848863Z         File size         Ratio      Format      Name
2020-10-08T11:13:26.4849351Z    --------------------   ------   -----------   -----------
2020-10-08T11:13:26.4850121Z   31866880 ->   9444660   29.64%   linux/amd64   submariner-globalnet
2020-10-08T11:13:26.4850428Z
2020-10-08T11:13:26.4850659Z Packed 1 file.
2020-10-08T11:13:26.8629327Z /opt/shipyard/scripts/build_image.sh -i submariner -f package/Dockerfile.submariner
2020-10-08T11:13:26.9044187Z fatal: No names found, cannot describe anything.
2020-10-08T11:13:26.9883658Z [submariner]$ set -e
2020-10-08T11:13:26.9892308Z [submariner]$ local_image=quay.io/submariner/submariner:dev
2020-10-08T11:13:26.9904604Z [submariner]$ cache_image=quay.io/submariner/submariner:devel
2020-10-08T11:13:26.9914287Z [submariner]$ cache_flag=
2020-10-08T11:13:26.9926553Z [submariner]$ cache_flag=--cache-from quay.io/submariner/submariner:devel
2020-10-08T11:13:26.9947182Z [submariner]$ docker image ls -q quay.io/submariner/submariner:devel
2020-10-08T11:13:27.3059029Z [submariner]$ docker pull quay.io/submariner/submariner:devel
2020-10-08T11:13:27.8161833Z devel: Pulling from submariner/submariner
2020-10-08T11:13:27.8178328Z bbca93d6c76b: Already exists
2020-10-08T11:13:27.8228184Z d9c1910875e2: Pulling fs layer
2020-10-08T11:13:27.8228598Z 8db3c5475ffa: Pulling fs layer
2020-10-08T11:13:27.8228946Z c8120d2c0056: Pulling fs layer
2020-10-08T11:13:27.8229272Z 09a9602a21d9: Pulling fs layer
2020-10-08T11:13:27.8233516Z 09a9602a21d9: Waiting
2020-10-08T11:13:27.8987431Z d9c1910875e2: Download complete
2020-10-08T11:13:28.0091382Z c8120d2c0056: Verifying Checksum
2020-10-08T11:13:28.0092057Z c8120d2c0056: Download complete
2020-10-08T11:13:28.0092432Z 09a9602a21d9: Verifying Checksum
2020-10-08T11:13:28.0092783Z 09a9602a21d9: Download complete
2020-10-08T11:13:28.0126725Z 8db3c5475ffa: Verifying Checksum
2020-10-08T11:13:28.0130303Z 8db3c5475ffa: Download complete
2020-10-08T11:13:28.7281126Z d9c1910875e2: Pull complete
2020-10-08T11:13:29.3255497Z 8db3c5475ffa: Pull complete
2020-10-08T11:13:29.5028451Z c8120d2c0056: Pull complete
2020-10-08T11:13:29.5702450Z 09a9602a21d9: Pull complete
2020-10-08T11:13:29.5725481Z Digest: sha256:cac9e1cce953daead3e41dee69fb7042b607e2a3bee97e8bc62ed20b876e045e
2020-10-08T11:13:29.5750450Z Status: Downloaded newer image for quay.io/submariner/submariner:devel
2020-10-08T11:13:29.5776522Z quay.io/submariner/submariner:devel
2020-10-08T11:13:29.5825979Z [submariner]$ grep FROM package/Dockerfile.submariner
2020-10-08T11:13:29.5827196Z [submariner]$ cut -f2 -d 
2020-10-08T11:13:29.5827941Z [submariner]$ grep -v scratch
2020-10-08T11:13:29.5840517Z [submariner]$ cache_flag+= --cache-from fedora:32
2020-10-08T11:13:29.5851660Z [submariner]$ docker pull fedora:32
2020-10-08T11:13:30.2397693Z 32: Pulling from library/fedora
2020-10-08T11:13:30.2933981Z bbca93d6c76b: Already exists
2020-10-08T11:13:30.4906114Z Digest: sha256:93b99d8dbb27450b6848d17669cce2bfe871bcd7534fc3a2bb8bd83c0f026ba1
2020-10-08T11:13:30.4925193Z Status: Downloaded newer image for fedora:32
2020-10-08T11:13:30.4925666Z docker.io/library/fedora:32
2020-10-08T11:13:30.4953229Z [submariner]$ buildargs_flag=--build-arg BUILDKIT_INLINE_CACHE=1
2020-10-08T11:13:30.4967070Z [submariner]$ DOCKER_BUILDKIT=1 docker build -t quay.io/submariner/submariner:dev --cache-from quay.io/submariner/submariner:devel --cache-from fedora:32 -f package/Dockerfile.submariner --build-arg BUILDKIT_INLINE_CACHE=1 .
2020-10-08T11:13:30.9610971Z #2 [internal] load build definition from Dockerfile.submariner
2020-10-08T11:13:30.9611716Z #2 transferring dockerfile: 581B done
2020-10-08T11:13:30.9612078Z #2 DONE 0.0s
2020-10-08T11:13:30.9612238Z
2020-10-08T11:13:30.9612763Z #1 [internal] load .dockerignore
2020-10-08T11:13:30.9613176Z #1 transferring context: 2B done
2020-10-08T11:13:30.9613480Z #1 DONE 0.0s
2020-10-08T11:13:30.9613635Z
2020-10-08T11:13:30.9613999Z #3 [internal] load metadata for docker.io/library/fedora:32
2020-10-08T11:13:30.9614372Z #3 DONE 0.0s
2020-10-08T11:13:30.9614540Z
2020-10-08T11:13:30.9614835Z #9 importing cache manifest from fedora:32
2020-10-08T11:13:30.9615159Z #9 DONE 0.0s
2020-10-08T11:13:30.9615326Z
2020-10-08T11:13:30.9615740Z #11 importing cache manifest from quay.io/submariner/submariner:devel
2020-10-08T11:13:30.9616155Z #11 DONE 0.0s
2020-10-08T11:13:30.9616324Z
2020-10-08T11:13:30.9616613Z #10 [1/5] FROM docker.io/library/fedora:32
2020-10-08T11:13:30.9616915Z #10 DONE 0.0s
2020-10-08T11:13:30.9617068Z
2020-10-08T11:13:30.9617337Z #6 [internal] load build context
2020-10-08T11:13:30.9617716Z #6 transferring context: 9.69MB 0.1s done
2020-10-08T11:13:31.1109975Z #6 DONE 0.1s
2020-10-08T11:13:31.1110576Z
2020-10-08T11:13:31.1111071Z #4 [2/5] WORKDIR /var/submariner
2020-10-08T11:13:31.1111522Z #4 CACHED
2020-10-08T11:13:31.1111798Z
2020-10-08T11:13:31.1112962Z #5 [3/5] RUN dnf -y install --nodocs --setopt=install_weak_deps=0          ...
2020-10-08T11:13:31.1113529Z #5 CACHED
2020-10-08T11:13:31.1113818Z
2020-10-08T11:13:31.1114644Z #7 [4/5] COPY package/submariner.sh package/pluto bin/submariner-engine /us...
2020-10-08T11:13:32.2700529Z #7 DONE 1.3s
2020-10-08T11:13:32.2701155Z
2020-10-08T11:13:32.2702290Z #8 [5/5] COPY package/iptables-wrapper.in /usr/sbin/
2020-10-08T11:13:32.2702757Z #8 DONE 0.0s
2020-10-08T11:13:33.3560184Z
2020-10-08T11:13:33.3560654Z #12 exporting to image
2020-10-08T11:13:33.3561017Z #12 exporting layers done
2020-10-08T11:13:33.3561860Z #12 writing image sha256:ca693abf15425ab175763b06748a36dc766a4c72d3e82d1ef40342bcc2683d17 done
2020-10-08T11:13:33.3563897Z #12 naming to quay.io/submariner/submariner:dev done
2020-10-08T11:13:33.3564888Z #12 DONE 0.0s
2020-10-08T11:13:33.3565050Z
2020-10-08T11:13:33.3565344Z #13 exporting cache
2020-10-08T11:13:33.3565756Z #13 preparing build cache for export done
2020-10-08T11:13:33.3566923Z #13 DONE 0.0s
2020-10-08T11:13:33.3601656Z [submariner]$ docker tag quay.io/submariner/submariner:dev quay.io/submariner/submariner:devel
2020-10-08T11:13:33.6989731Z touch package/.image.submariner
2020-10-08T11:13:34.0252600Z /opt/shipyard/scripts/build_image.sh -i submariner-route-agent -f package/Dockerfile.submariner-route-agent
2020-10-08T11:13:34.0984087Z fatal: No names found, cannot describe anything.
2020-10-08T11:13:34.1818329Z [submariner]$ set -e
2020-10-08T11:13:34.1839750Z [submariner]$ local_image=quay.io/submariner/submariner-route-agent:dev
2020-10-08T11:13:34.1850392Z [submariner]$ cache_image=quay.io/submariner/submariner-route-agent:devel
2020-10-08T11:13:34.1861598Z [submariner]$ cache_flag=
2020-10-08T11:13:34.1875201Z [submariner]$ cache_flag=--cache-from quay.io/submariner/submariner-route-agent:devel
2020-10-08T11:13:34.1892828Z [submariner]$ docker image ls -q quay.io/submariner/submariner-route-agent:devel
2020-10-08T11:13:34.5406572Z [submariner]$ docker pull quay.io/submariner/submariner-route-agent:devel
2020-10-08T11:13:35.0098567Z devel: Pulling from submariner/submariner-route-agent
2020-10-08T11:13:35.0099310Z 0fd3b5213a9b: Pulling fs layer
2020-10-08T11:13:35.0099708Z aebb8c556853: Pulling fs layer
2020-10-08T11:13:35.0100048Z a10da1370ba4: Pulling fs layer
2020-10-08T11:13:35.0100408Z 704148a09378: Pulling fs layer
2020-10-08T11:13:35.0100804Z cf840e2779e0: Pulling fs layer
2020-10-08T11:13:35.0101288Z e5192a56e366: Pulling fs layer
2020-10-08T11:13:35.0101615Z 704148a09378: Waiting
2020-10-08T11:13:35.0101930Z cf840e2779e0: Waiting
2020-10-08T11:13:35.0102254Z e5192a56e366: Waiting
2020-10-08T11:13:35.0822202Z a10da1370ba4: Verifying Checksum
2020-10-08T11:13:35.0822668Z a10da1370ba4: Download complete
2020-10-08T11:13:35.0823472Z aebb8c556853: Verifying Checksum
2020-10-08T11:13:35.0823947Z aebb8c556853: Download complete
2020-10-08T11:13:35.2553017Z 704148a09378: Verifying Checksum
2020-10-08T11:13:35.2553454Z 704148a09378: Download complete
2020-10-08T11:13:35.3247610Z cf840e2779e0: Verifying Checksum
2020-10-08T11:13:35.3248089Z cf840e2779e0: Download complete
2020-10-08T11:13:35.3477440Z e5192a56e366: Verifying Checksum
2020-10-08T11:13:35.3487434Z e5192a56e366: Download complete
2020-10-08T11:13:35.4946592Z 0fd3b5213a9b: Verifying Checksum
2020-10-08T11:13:35.4947265Z 0fd3b5213a9b: Download complete
2020-10-08T11:13:37.9075533Z 0fd3b5213a9b: Pull complete
2020-10-08T11:13:37.9749027Z aebb8c556853: Pull complete
2020-10-08T11:13:38.0534951Z a10da1370ba4: Pull complete
2020-10-08T11:13:38.3001279Z 704148a09378: Pull complete
2020-10-08T11:13:38.4959568Z cf840e2779e0: Pull complete
2020-10-08T11:13:38.5754206Z e5192a56e366: Pull complete
2020-10-08T11:13:38.5782382Z Digest: sha256:e7d24a573defc8b532890afadd6218fd8b6b159d913628aa231bbccf4f274f6b
2020-10-08T11:13:38.5803476Z Status: Downloaded newer image for quay.io/submariner/submariner-route-agent:devel
2020-10-08T11:13:38.5834543Z quay.io/submariner/submariner-route-agent:devel
2020-10-08T11:13:38.5867218Z [submariner]$ grep FROM package/Dockerfile.submariner-route-agent
2020-10-08T11:13:38.5882365Z [submariner]$ cut -f2 -d 
2020-10-08T11:13:38.5883022Z [submariner]$ grep -v scratch
2020-10-08T11:13:38.5899605Z [submariner]$ cache_flag+= --cache-from registry.access.redhat.com/ubi8/ubi-minimal
2020-10-08T11:13:38.5908053Z [submariner]$ docker pull registry.access.redhat.com/ubi8/ubi-minimal
2020-10-08T11:13:38.9162445Z Using default tag: latest
2020-10-08T11:13:39.1765668Z latest: Pulling from ubi8/ubi-minimal
2020-10-08T11:13:39.3020975Z 0fd3b5213a9b: Already exists
2020-10-08T11:13:39.3035305Z aebb8c556853: Already exists
2020-10-08T11:13:39.4256463Z Digest: sha256:5cfbaf45ca96806917830c183e9f37df2e913b187aadb32e89fd83fa455ebaa6
2020-10-08T11:13:39.4274032Z Status: Downloaded newer image for registry.access.redhat.com/ubi8/ubi-minimal:latest
2020-10-08T11:13:39.4282445Z registry.access.redhat.com/ubi8/ubi-minimal:latest
2020-10-08T11:13:39.4308236Z [submariner]$ buildargs_flag=--build-arg BUILDKIT_INLINE_CACHE=1
2020-10-08T11:13:39.4322236Z [submariner]$ DOCKER_BUILDKIT=1 docker build -t quay.io/submariner/submariner-route-agent:dev --cache-from quay.io/submariner/submariner-route-agent:devel --cache-from registry.access.redhat.com/ubi8/ubi-minimal -f package/Dockerfile.submariner-route-agent --build-arg BUILDKIT_INLINE_CACHE=1 .
2020-10-08T11:13:39.7449369Z #2 [internal] load .dockerignore
2020-10-08T11:13:39.8483179Z #2 transferring context: 2B done
2020-10-08T11:13:39.8483655Z #2 DONE 0.0s
2020-10-08T11:13:39.8483833Z
2020-10-08T11:13:39.8485154Z #1 [internal] load build definition from Dockerfile.submariner-route-agent
2020-10-08T11:13:39.8485880Z #1 transferring dockerfile: 513B done
2020-10-08T11:13:39.8486224Z #1 DONE 0.0s
2020-10-08T11:13:39.8486407Z
2020-10-08T11:13:39.8487165Z #3 [internal] load metadata for registry.access.redhat.com/ubi8/ubi-minimal...
2020-10-08T11:13:39.8487729Z #3 DONE 0.0s
2020-10-08T11:13:39.8487887Z
2020-10-08T11:13:39.8488523Z #4 [1/5] FROM registry.access.redhat.com/ubi8/ubi-minimal
2020-10-08T11:13:39.8489000Z #4 DONE 0.0s
2020-10-08T11:13:39.8489155Z
2020-10-08T11:13:39.8489893Z #10 importing cache manifest from registry.access.redhat.com/ubi8/ubi-minima...
2020-10-08T11:13:39.8490459Z #10 DONE 0.0s
2020-10-08T11:13:39.8490613Z
2020-10-08T11:13:39.8491305Z #11 importing cache manifest from quay.io/submariner/submariner-route-agent:...
2020-10-08T11:13:39.8491831Z #11 DONE 0.0s
2020-10-08T11:13:39.8492000Z
2020-10-08T11:13:39.8492259Z #7 [internal] load build context
2020-10-08T11:13:39.8492640Z #7 transferring context: 9.45MB 0.1s done
2020-10-08T11:13:39.9974869Z #7 DONE 0.1s
2020-10-08T11:13:39.9978702Z
2020-10-08T11:13:39.9979512Z #5 [2/5] WORKDIR /var/submariner
2020-10-08T11:13:39.9980058Z #5 CACHED
2020-10-08T11:13:39.9980697Z
2020-10-08T11:13:39.9982126Z #6 [3/5] RUN microdnf -y install --nodocs iproute iptables &&     microdnf ...
2020-10-08T11:13:39.9982860Z #6 CACHED
2020-10-08T11:13:39.9983226Z
2020-10-08T11:13:39.9984313Z #8 [4/5] COPY package/submariner-route-agent.sh bin/submariner-route-agent ...
2020-10-08T11:13:42.8545704Z #8 DONE 2.8s
2020-10-08T11:13:42.8546040Z
2020-10-08T11:13:42.8547112Z #9 [5/5] COPY package/iptables-wrapper.in /usr/sbin/
2020-10-08T11:13:42.8547519Z #9 DONE 0.0s
2020-10-08T11:13:43.1104948Z
2020-10-08T11:13:43.1105522Z #12 exporting to image
2020-10-08T11:13:43.1106003Z #12 exporting layers done
2020-10-08T11:13:43.1279201Z #12 writing image sha256:47e8f65d6d797aa39df1736fc41b82693532fcc86289fd0d06ad967f211958f1 done
2020-10-08T11:13:43.1280938Z #12 naming to quay.io/submariner/submariner-route-agent:dev done
2020-10-08T11:13:43.1281425Z #12 DONE 0.0s
2020-10-08T11:13:43.1281617Z
2020-10-08T11:13:43.1281864Z #13 exporting cache
2020-10-08T11:13:43.1282233Z #13 preparing build cache for export done
2020-10-08T11:13:43.1282552Z #13 DONE 0.0s
2020-10-08T11:13:43.1283574Z [submariner]$ docker tag quay.io/submariner/submariner-route-agent:dev quay.io/submariner/submariner-route-agent:devel
2020-10-08T11:13:43.4377967Z touch package/.image.submariner-route-agent
2020-10-08T11:13:43.7866065Z /opt/shipyard/scripts/build_image.sh -i submariner-globalnet -f package/Dockerfile.submariner-globalnet
2020-10-08T11:13:43.8262398Z fatal: No names found, cannot describe anything.
2020-10-08T11:13:43.9366782Z [submariner]$ set -e
2020-10-08T11:13:43.9377851Z [submariner]$ local_image=quay.io/submariner/submariner-globalnet:dev
2020-10-08T11:13:43.9391890Z [submariner]$ cache_image=quay.io/submariner/submariner-globalnet:devel
2020-10-08T11:13:43.9404443Z [submariner]$ cache_flag=
2020-10-08T11:13:43.9416884Z [submariner]$ cache_flag=--cache-from quay.io/submariner/submariner-globalnet:devel
2020-10-08T11:13:43.9435379Z [submariner]$ docker image ls -q quay.io/submariner/submariner-globalnet:devel
2020-10-08T11:13:44.2826954Z [submariner]$ docker pull quay.io/submariner/submariner-globalnet:devel
2020-10-08T11:13:44.7590245Z devel: Pulling from submariner/submariner-globalnet
2020-10-08T11:13:44.7590843Z 0fd3b5213a9b: Already exists
2020-10-08T11:13:44.7608370Z aebb8c556853: Already exists
2020-10-08T11:13:44.7629298Z a10da1370ba4: Already exists
2020-10-08T11:13:44.7649976Z 704148a09378: Already exists
2020-10-08T11:13:44.7671165Z af5a11458838: Pulling fs layer
2020-10-08T11:13:44.7672653Z 9b526c5f9b6a: Pulling fs layer
2020-10-08T11:13:44.8375665Z 9b526c5f9b6a: Verifying Checksum
2020-10-08T11:13:44.8376183Z 9b526c5f9b6a: Download complete
2020-10-08T11:13:44.9222785Z af5a11458838: Verifying Checksum
2020-10-08T11:13:44.9223213Z af5a11458838: Download complete
2020-10-08T11:13:45.0777495Z af5a11458838: Pull complete
2020-10-08T11:13:45.1422141Z 9b526c5f9b6a: Pull complete
2020-10-08T11:13:45.1447161Z Digest: sha256:4d0998abadd198b967a0db55d85507cc1e9a665db91e5068c2b2d32810f96f11
2020-10-08T11:13:45.1484115Z Status: Downloaded newer image for quay.io/submariner/submariner-globalnet:devel
2020-10-08T11:13:45.1516727Z quay.io/submariner/submariner-globalnet:devel
2020-10-08T11:13:45.1527145Z [submariner]$ grep FROM package/Dockerfile.submariner-globalnet
2020-10-08T11:13:45.1530932Z [submariner]$ cut -f2 -d 
2020-10-08T11:13:45.1531787Z [submariner]$ grep -v scratch
2020-10-08T11:13:45.1544313Z [submariner]$ cache_flag+= --cache-from registry.access.redhat.com/ubi8/ubi-minimal
2020-10-08T11:13:45.1549997Z [submariner]$ docker pull registry.access.redhat.com/ubi8/ubi-minimal
2020-10-08T11:13:45.4715401Z Using default tag: latest
2020-10-08T11:13:45.6972543Z latest: Pulling from ubi8/ubi-minimal
2020-10-08T11:13:45.8350576Z Digest: sha256:5cfbaf45ca96806917830c183e9f37df2e913b187aadb32e89fd83fa455ebaa6
2020-10-08T11:13:45.8352782Z Status: Image is up to date for registry.access.redhat.com/ubi8/ubi-minimal:latest
2020-10-08T11:13:45.8355740Z registry.access.redhat.com/ubi8/ubi-minimal:latest
2020-10-08T11:13:45.8382725Z [submariner]$ buildargs_flag=--build-arg BUILDKIT_INLINE_CACHE=1
2020-10-08T11:13:45.8400950Z [submariner]$ DOCKER_BUILDKIT=1 docker build -t quay.io/submariner/submariner-globalnet:dev --cache-from quay.io/submariner/submariner-globalnet:devel --cache-from registry.access.redhat.com/ubi8/ubi-minimal -f package/Dockerfile.submariner-globalnet --build-arg BUILDKIT_INLINE_CACHE=1 .
2020-10-08T11:13:46.1573961Z #2 [internal] load .dockerignore
2020-10-08T11:13:46.2643472Z #2 transferring context: 2B done
2020-10-08T11:13:46.2644143Z #2 DONE 0.0s
2020-10-08T11:13:46.2644310Z
2020-10-08T11:13:46.2645548Z #1 [internal] load build definition from Dockerfile.submariner-globalnet
2020-10-08T11:13:46.2646240Z #1 transferring dockerfile: 505B done
2020-10-08T11:13:46.2646564Z #1 DONE 0.0s
2020-10-08T11:13:46.2646750Z
2020-10-08T11:13:46.2647526Z #3 [internal] load metadata for registry.access.redhat.com/ubi8/ubi-minimal...
2020-10-08T11:13:46.2648095Z #3 DONE 0.0s
2020-10-08T11:13:46.2648267Z
2020-10-08T11:13:46.2648953Z #10 importing cache manifest from quay.io/submariner/submariner-globalnet:de...
2020-10-08T11:13:46.2649468Z #10 DONE 0.0s
2020-10-08T11:13:46.2649638Z
2020-10-08T11:13:46.2650279Z #4 [1/5] FROM registry.access.redhat.com/ubi8/ubi-minimal
2020-10-08T11:13:46.2650740Z #4 DONE 0.0s
2020-10-08T11:13:46.2650912Z
2020-10-08T11:13:46.2651655Z #11 importing cache manifest from registry.access.redhat.com/ubi8/ubi-minima...
2020-10-08T11:13:46.2652220Z #11 DONE 0.0s
2020-10-08T11:13:46.2652377Z
2020-10-08T11:13:46.2652651Z #7 [internal] load build context
2020-10-08T11:13:46.2653029Z #7 transferring context: 9.45MB 0.1s done
2020-10-08T11:13:46.4139744Z #7 DONE 0.1s
2020-10-08T11:13:46.4141127Z
2020-10-08T11:13:46.4141513Z #5 [2/5] WORKDIR /var/submariner
2020-10-08T11:13:46.5038323Z #5 CACHED
2020-10-08T11:13:46.5038494Z
2020-10-08T11:13:46.5039613Z #6 [3/5] RUN microdnf -y install --nodocs iproute iptables &&     microdnf ...
2020-10-08T11:13:46.5040029Z #6 CACHED
2020-10-08T11:13:46.5040204Z
2020-10-08T11:13:46.5040974Z #8 [4/5] COPY package/submariner-globalnet.sh bin/submariner-globalnet /usr...
2020-10-08T11:13:46.7164826Z #8 DONE 0.4s
2020-10-08T11:13:46.7165345Z
2020-10-08T11:13:46.7166520Z #9 [5/5] COPY package/iptables-wrapper.in /usr/sbin/
2020-10-08T11:13:46.7166933Z #9 DONE 0.0s
2020-10-08T11:13:47.0971585Z
2020-10-08T11:13:47.0972445Z #12 exporting to image
2020-10-08T11:13:47.0972923Z #12 exporting layers done
2020-10-08T11:13:47.0973895Z #12 writing image sha256:e50a25f4f89223b3a81e7de8b8a6d1b6880cab5606555eea404b7ae8900846ec done
2020-10-08T11:13:47.0976341Z #12 naming to quay.io/submariner/submariner-globalnet:dev done
2020-10-08T11:13:47.0977000Z #12 DONE 0.0s
2020-10-08T11:13:47.0977300Z
2020-10-08T11:13:47.0977670Z #13 exporting cache
2020-10-08T11:13:47.0978165Z #13 preparing build cache for export done
2020-10-08T11:13:47.0978614Z #13 DONE 0.0s
2020-10-08T11:13:47.1039845Z [submariner]$ docker tag quay.io/submariner/submariner-globalnet:dev quay.io/submariner/submariner-globalnet:devel
2020-10-08T11:13:47.4141152Z touch package/.image.submariner-globalnet
2020-10-08T11:13:47.4163428Z /opt/shipyard/scripts/deploy.sh --deploytool helm
2020-10-08T11:13:47.4820929Z Running with: globalnet='false', deploytool='helm', deploytool_broker_args='', deploytool_submariner_args='', cluster_settings='', timeout=5m
2020-10-08T11:13:47.4842556Z [submariner]$ source /opt/shipyard/scripts/lib/version
2020-10-08T11:13:47.4854914Z [submariner]$ . /opt/shipyard/scripts/lib/source_only
2020-10-08T11:13:47.4864940Z [submariner]$ script_name=version
2020-10-08T11:13:47.4878846Z [submariner]$ exec_name=deploy.sh
2020-10-08T11:13:47.4896470Z [submariner]$ git describe --tags --dirty=-dev --exclude=devel --exclude=latest
2020-10-08T11:13:47.5286558Z fatal: No names found, cannot describe anything.
2020-10-08T11:13:47.5305965Z [submariner]$ source /opt/shipyard/scripts/lib/utils
2020-10-08T11:13:47.5322385Z [submariner]$ . /opt/shipyard/scripts/lib/source_only
2020-10-08T11:13:47.5333007Z [submariner]$ script_name=utils
2020-10-08T11:13:47.5347791Z [submariner]$ exec_name=deploy.sh
2020-10-08T11:13:47.5367356Z [submariner]$ source /opt/shipyard/scripts/lib/deploy_funcs
2020-10-08T11:13:47.5380110Z [submariner]$ . /opt/shipyard/scripts/lib/source_only
2020-10-08T11:13:47.5393124Z [submariner]$ script_name=deploy_funcs
2020-10-08T11:13:47.5403967Z [submariner]$ exec_name=deploy.sh
2020-10-08T11:13:47.5424429Z [submariner]$ source /opt/shipyard/scripts/lib/cluster_settings
2020-10-08T11:13:47.5439094Z [submariner]$ . /opt/shipyard/scripts/lib/source_only
2020-10-08T11:13:47.5451943Z [submariner]$ script_name=cluster_settings
2020-10-08T11:13:47.5466725Z [submariner]$ exec_name=deploy.sh
2020-10-08T11:13:47.5489158Z [submariner]$ broker=cluster1
2020-10-08T11:13:47.5500705Z [submariner]$ declare -gA cluster_nodes
2020-10-08T11:13:47.5514119Z [submariner]$ cluster_nodes[cluster1]=control-plane worker
2020-10-08T11:13:47.5525938Z [submariner]$ cluster_nodes[cluster2]=control-plane worker
2020-10-08T11:13:47.5539532Z [submariner]$ cluster_nodes[cluster3]=control-plane worker worker
2020-10-08T11:13:47.5550493Z [submariner]$ declare -gA cluster_subm
2020-10-08T11:13:47.5564328Z [submariner]$ cluster_subm[cluster1]=true
2020-10-08T11:13:47.5574449Z [submariner]$ cluster_subm[cluster2]=true
2020-10-08T11:13:47.5588184Z [submariner]$ cluster_subm[cluster3]=true
2020-10-08T11:13:47.5599382Z [submariner]$ declare -gA cluster_cni
2020-10-08T11:13:47.5612598Z [submariner]$ cluster_cni[cluster2]=weave
2020-10-08T11:13:47.5625002Z [submariner]$ cluster_cni[cluster3]=weave
2020-10-08T11:13:47.5636076Z [submariner]$ declare_cidrs
2020-10-08T11:13:47.5654981Z [submariner]$ declare_cidrs
2020-10-08T11:13:47.5669237Z [submariner]$ declare -gA cluster_CIDRs service_CIDRs global_CIDRs
2020-10-08T11:13:47.5685010Z [submariner]$ i=1
2020-10-08T11:13:47.5731681Z [submariner]$ [cluster1] add_cluster_cidrs 1 cluster1
2020-10-08T11:13:47.5752417Z [submariner]$ [cluster1] add_cluster_cidrs 1 cluster1
2020-10-08T11:13:47.5765579Z [submariner]$ [cluster1] local val=1
2020-10-08T11:13:47.5776967Z [submariner]$ [cluster1] local idx=cluster1
2020-10-08T11:13:47.5788903Z [submariner]$ [cluster1] cluster_CIDRs[cluster1]=10.241.0.0/16
2020-10-08T11:13:47.5804001Z [submariner]$ [cluster1] service_CIDRs[cluster1]=100.91.0.0/16
2020-10-08T11:13:47.5816869Z [submariner]$ [cluster1] i=2
2020-10-08T11:13:47.5831198Z [submariner]$ [cluster2] add_cluster_cidrs 2 cluster2
2020-10-08T11:13:47.5842887Z [submariner]$ [cluster2] add_cluster_cidrs 2 cluster2
2020-10-08T11:13:47.5852385Z [submariner]$ [cluster2] local val=2
2020-10-08T11:13:47.5864791Z [submariner]$ [cluster2] local idx=cluster2
2020-10-08T11:13:47.5876165Z [submariner]$ [cluster2] cluster_CIDRs[cluster2]=10.242.0.0/16
2020-10-08T11:13:47.5888798Z [submariner]$ [cluster2] service_CIDRs[cluster2]=100.92.0.0/16
2020-10-08T11:13:47.5900091Z [submariner]$ [cluster2] i=3
2020-10-08T11:13:47.5911199Z [submariner]$ [cluster3] add_cluster_cidrs 3 cluster3
2020-10-08T11:13:47.5924672Z [submariner]$ [cluster3] add_cluster_cidrs 3 cluster3
2020-10-08T11:13:47.5934435Z [submariner]$ [cluster3] local val=3
2020-10-08T11:13:47.5946981Z [submariner]$ [cluster3] local idx=cluster3
2020-10-08T11:13:47.6011717Z [submariner]$ [cluster3] cluster_CIDRs[cluster3]=10.243.0.0/16
2020-10-08T11:13:47.6028090Z [submariner]$ [cluster3] service_CIDRs[cluster3]=100.93.0.0/16
2020-10-08T11:13:47.6040670Z [submariner]$ [cluster3] i=4
2020-10-08T11:13:47.6050924Z [submariner]$ [cluster3] declare_kubeconfig
2020-10-08T11:13:47.6063955Z [submariner]$ [cluster3] declare_kubeconfig
2020-10-08T11:13:47.6073907Z [submariner]$ [cluster3] source /opt/shipyard/scripts/lib/kubecfg
2020-10-08T11:13:47.6089412Z [submariner]$ [cluster3] export KUBECONFIG
2020-10-08T11:13:47.6126518Z [submariner]$ [cluster3] KUBECONFIG=/go/src/github.com/submariner-io/submariner/output/kubeconfigs/kind-config-cluster3:/go/src/github.com/submariner-io/submariner/output/kubeconfigs/kind-config-cluster2:/go/src/github.com/submariner-io/submariner/output/kubeconfigs/kind-config-cluster1:
2020-10-08T11:13:47.6149596Z [submariner]$ [cluster3] find /go/src/github.com/submariner-io/submariner/output/kubeconfigs -type f -printf %p:
2020-10-08T11:13:47.6179578Z [submariner]$ [cluster3] import_image quay.io/submariner/nettest
2020-10-08T11:13:47.6191421Z [submariner]$ [cluster3] import_image quay.io/submariner/nettest
2020-10-08T11:13:47.6204045Z [submariner]$ [cluster3] local orig_image=quay.io/submariner/nettest
2020-10-08T11:13:47.6217596Z [submariner]$ [cluster3] local versioned_image=quay.io/submariner/nettest:dev
2020-10-08T11:13:47.6230261Z [submariner]$ [cluster3] local local_image=localhost:5000/nettest:local
2020-10-08T11:13:47.6243576Z [submariner]$ [cluster3] docker tag quay.io/submariner/nettest:dev localhost:5000/nettest:local
2020-10-08T11:13:47.9471703Z Error response from daemon: No such image: quay.io/submariner/nettest:dev
2020-10-08T11:13:47.9525253Z [submariner]$ [cluster3] docker pull quay.io/submariner/nettest:devel
2020-10-08T11:13:48.4101410Z devel: Pulling from submariner/nettest
2020-10-08T11:13:48.4101953Z df20fa9351a1: Pulling fs layer
2020-10-08T11:13:48.4104053Z 741138c6c583: Pulling fs layer
2020-10-08T11:13:48.4104775Z d34ea735672c: Pulling fs layer
2020-10-08T11:13:48.4108653Z c91acd455ec9: Pulling fs layer
2020-10-08T11:13:48.4112728Z c91acd455ec9: Waiting
2020-10-08T11:13:48.4885105Z 741138c6c583: Verifying Checksum
2020-10-08T11:13:48.4885562Z 741138c6c583: Download complete
2020-10-08T11:13:48.5465816Z df20fa9351a1: Verifying Checksum
2020-10-08T11:13:48.5466258Z df20fa9351a1: Download complete
2020-10-08T11:13:48.5519802Z d34ea735672c: Verifying Checksum
2020-10-08T11:13:48.5521005Z d34ea735672c: Download complete
2020-10-08T11:13:48.5818074Z c91acd455ec9: Verifying Checksum
2020-10-08T11:13:48.5818603Z c91acd455ec9: Download complete
2020-10-08T11:13:48.7200088Z df20fa9351a1: Pull complete
2020-10-08T11:13:48.7979212Z 741138c6c583: Pull complete
2020-10-08T11:13:49.1798950Z d34ea735672c: Pull complete
2020-10-08T11:13:49.2489214Z c91acd455ec9: Pull complete
2020-10-08T11:13:49.2516328Z Digest: sha256:63f21b9d8065d60870c2bca980dbc4dc6358edf669661576b8cbee39e3024a2d
2020-10-08T11:13:49.2541263Z Status: Downloaded newer image for quay.io/submariner/nettest:devel
2020-10-08T11:13:49.2560878Z quay.io/submariner/nettest:devel
2020-10-08T11:13:49.2570018Z [submariner]$ [cluster3] docker tag quay.io/submariner/nettest:devel quay.io/submariner/nettest:dev
2020-10-08T11:13:49.5876122Z [submariner]$ [cluster3] docker tag quay.io/submariner/nettest:dev localhost:5000/nettest:local
2020-10-08T11:13:49.9145537Z [submariner]$ [cluster3] docker push localhost:5000/nettest:local
2020-10-08T11:13:50.2621714Z The push refers to repository [localhost:5000/nettest]
2020-10-08T11:13:50.3227387Z 7497bfb0ae54: Preparing
2020-10-08T11:13:50.3227741Z 17de069c349e: Preparing
2020-10-08T11:13:50.3228054Z b85398c91342: Preparing
2020-10-08T11:13:50.3228356Z 50644c29ef5a: Preparing
2020-10-08T11:13:51.1470070Z b85398c91342: Pushed
2020-10-08T11:13:51.2007624Z 7497bfb0ae54: Pushed
2020-10-08T11:13:51.2029329Z 50644c29ef5a: Pushed
2020-10-08T11:13:52.2206611Z 17de069c349e: Pushed
2020-10-08T11:13:52.2395745Z local: digest: sha256:63f21b9d8065d60870c2bca980dbc4dc6358edf669661576b8cbee39e3024a2d size: 1155
2020-10-08T11:13:52.2430433Z [submariner]$ [cluster3] import_image quay.io/submariner/submariner
2020-10-08T11:13:52.2447477Z [submariner]$ [cluster3] import_image quay.io/submariner/submariner
2020-10-08T11:13:52.2458809Z [submariner]$ [cluster3] local orig_image=quay.io/submariner/submariner
2020-10-08T11:13:52.2472497Z [submariner]$ [cluster3] local versioned_image=quay.io/submariner/submariner:dev
2020-10-08T11:13:52.2485677Z [submariner]$ [cluster3] local local_image=localhost:5000/submariner:local
2020-10-08T11:13:52.2498734Z [submariner]$ [cluster3] docker tag quay.io/submariner/submariner:dev localhost:5000/submariner:local
2020-10-08T11:13:52.5681192Z [submariner]$ [cluster3] docker push localhost:5000/submariner:local
2020-10-08T11:13:52.8785886Z The push refers to repository [localhost:5000/submariner]
2020-10-08T11:13:52.8853401Z af8ae9c4c94b: Preparing
2020-10-08T11:13:52.8853834Z befa3b30a784: Preparing
2020-10-08T11:13:52.8857000Z dc8ba3df3f12: Preparing
2020-10-08T11:13:52.8860994Z c48026e1f9ff: Preparing
2020-10-08T11:13:52.8861625Z fc9509c19a6c: Preparing
2020-10-08T11:13:53.8668510Z af8ae9c4c94b: Pushed
2020-10-08T11:13:54.1261353Z c48026e1f9ff: Pushed
2020-10-08T11:13:54.3670452Z befa3b30a784: Pushed
2020-10-08T11:13:55.5609158Z dc8ba3df3f12: Pushed
2020-10-08T11:14:09.0502021Z fc9509c19a6c: Pushed
2020-10-08T11:14:09.0721868Z local: digest: sha256:f3296b9c0b8c78b8b988bee4262df84e46e31719d41d1c978501114581b7fa00 size: 1366
2020-10-08T11:14:09.0756179Z [submariner]$ [cluster3] import_image quay.io/submariner/submariner-route-agent
2020-10-08T11:14:09.0768168Z [submariner]$ [cluster3] import_image quay.io/submariner/submariner-route-agent
2020-10-08T11:14:09.0780225Z [submariner]$ [cluster3] local orig_image=quay.io/submariner/submariner-route-agent
2020-10-08T11:14:09.0792461Z [submariner]$ [cluster3] local versioned_image=quay.io/submariner/submariner-route-agent:dev
2020-10-08T11:14:09.0805303Z [submariner]$ [cluster3] local local_image=localhost:5000/submariner-route-agent:local
2020-10-08T11:14:09.0822140Z [submariner]$ [cluster3] docker tag quay.io/submariner/submariner-route-agent:dev localhost:5000/submariner-route-agent:local
2020-10-08T11:14:09.3976817Z [submariner]$ [cluster3] docker push localhost:5000/submariner-route-agent:local
2020-10-08T11:14:09.7189691Z The push refers to repository [localhost:5000/submariner-route-agent]
2020-10-08T11:14:09.7266920Z c9f706aff5cd: Preparing
2020-10-08T11:14:09.7269517Z 0cecab1fe207: Preparing
2020-10-08T11:14:09.7270172Z 74b4a0049f0b: Preparing
2020-10-08T11:14:09.7270522Z 22d731db4600: Preparing
2020-10-08T11:14:09.7270806Z 3485805ce47c: Preparing
2020-10-08T11:14:09.7271104Z b0e2911c99f3: Preparing
2020-10-08T11:14:09.7276249Z b0e2911c99f3: Waiting
2020-10-08T11:14:09.8397907Z c9f706aff5cd: Pushed
2020-10-08T11:14:09.9434972Z 3485805ce47c: Pushed
2020-10-08T11:14:10.3962904Z 22d731db4600: Pushed
2020-10-08T11:14:10.9589595Z 0cecab1fe207: Pushed
2020-10-08T11:14:10.9641426Z 74b4a0049f0b: Pushed
2020-10-08T11:14:20.3402066Z b0e2911c99f3: Pushed
2020-10-08T11:14:20.3729956Z local: digest: sha256:68b822318b15686cadd208bea1d2d59f8e65b50c70e31f0e5c52995e9084ce7f size: 1573
2020-10-08T11:14:20.3833565Z [submariner]$ [cluster3] load_deploytool helm
2020-10-08T11:14:20.3899138Z [submariner]$ [cluster3] load_deploytool helm
2020-10-08T11:14:20.3949336Z [submariner]$ [cluster3] local deploytool=helm
2020-10-08T11:14:20.4010984Z [submariner]$ [cluster3] local deploy_lib=/opt/shipyard/scripts/lib/deploy_helm
2020-10-08T11:14:20.4024013Z Will deploy submariner using helm
2020-10-08T11:14:20.4048975Z [submariner]$ [cluster3] . /opt/shipyard/scripts/lib/deploy_helm
2020-10-08T11:14:20.4073094Z [submariner]$ [cluster3] . /opt/shipyard/scripts/lib/source_only
2020-10-08T11:14:20.4098242Z [submariner]$ [cluster3] script_name=deploy_helm
2020-10-08T11:14:20.4117853Z [submariner]$ [cluster3] exec_name=deploy.sh
2020-10-08T11:14:20.4141715Z [submariner]$ [cluster3] LC_CTYPE=C tr -dc a-zA-Z0-9
2020-10-08T11:14:20.4182359Z [submariner]$ [cluster3] fold -w 64
2020-10-08T11:14:20.4261174Z [submariner]$ [cluster3] head -n 1
2020-10-08T11:14:20.4334804Z [submariner]$ [cluster3] deploytool_prereqs
2020-10-08T11:14:20.4362005Z [submariner]$ [cluster3] deploytool_prereqs
2020-10-08T11:14:20.4399499Z [submariner]$ [cluster3] helm init --client-only
2020-10-08T11:14:20.7032940Z Creating /root/.helm
2020-10-08T11:14:20.7034309Z Creating /root/.helm/repository
2020-10-08T11:14:20.7034911Z Creating /root/.helm/repository/cache
2020-10-08T11:14:20.7035327Z Creating /root/.helm/repository/local
2020-10-08T11:14:20.7035721Z Creating /root/.helm/plugins
2020-10-08T11:14:20.7036081Z Creating /root/.helm/starters
2020-10-08T11:14:20.7037359Z Creating /root/.helm/cache/archive
2020-10-08T11:14:20.7037899Z Creating /root/.helm/repository/repositories.yaml
2020-10-08T11:14:20.7039683Z Adding stable repo with URL: https://kubernetes-charts.storage.googleapis.com
2020-10-08T11:14:22.4114228Z Adding local repo with URL: http://127.0.0.1:8879/charts
2020-10-08T11:14:22.4130984Z $HELM_HOME has been configured at /root/.helm.
2020-10-08T11:14:22.4137264Z Not installing Tiller due to 'client-only' flag having been set
2020-10-08T11:14:22.4288327Z [submariner]$ [cluster3] helm repo add submariner-latest https://submariner-io.github.io/submariner-charts/charts
2020-10-08T11:14:22.8699349Z "submariner-latest" has been added to your repositories
2020-10-08T11:14:22.8725462Z [submariner]$ [cluster3] run_all_clusters install_helm
2020-10-08T11:14:22.8737102Z [submariner]$ [cluster3] run_all_clusters install_helm
2020-10-08T11:14:22.8748728Z [submariner]$ [cluster3] run_parallel cluster1 cluster2 cluster3 install_helm
2020-10-08T11:14:22.8763041Z [submariner]$ [cluster3] run_parallel cluster1 cluster2 cluster3 cluster1 cluster2 cluster3 install_helm
2020-10-08T11:14:22.8772485Z [submariner]$ [cluster3] local cmnd=install_helm
2020-10-08T11:14:22.8786682Z [submariner]$ [cluster3] declare -A pids
2020-10-08T11:14:22.8802325Z [submariner]$ [cluster3] eval echo cluster1 cluster2 cluster3
2020-10-08T11:14:22.8820701Z [submariner]$ [cluster1] pids[cluster1]=5145
2020-10-08T11:14:22.8826846Z [submariner]$ [cluster1] set -o pipefail
2020-10-08T11:14:22.8841496Z [submariner]$ [cluster2] set -o pipefail
2020-10-08T11:14:22.8846664Z [submariner]$ [cluster2] pids[cluster2]=5148
2020-10-08T11:14:22.8853406Z [submariner]$ [cluster1] install_helm
2020-10-08T11:14:22.8868126Z [submariner]$ [cluster1] sed /\[cluster1]/!s/^/[cluster1] /
2020-10-08T11:14:22.8877862Z [submariner]$ [cluster2] install_helm
2020-10-08T11:14:22.8887377Z [submariner]$ [cluster3] pids[cluster3]=5154
2020-10-08T11:14:22.8905431Z [submariner]$ [cluster3] set -o pipefail
2020-10-08T11:14:22.8923050Z [submariner]$ [cluster2] sed /\[cluster2]/!s/^/[cluster2] /
2020-10-08T11:14:22.8932758Z [submariner]$ [cluster3] wait 5154
2020-10-08T11:14:22.8949288Z [submariner]$ [cluster3] install_helm
2020-10-08T11:14:22.8974118Z [submariner]$ [cluster3] sed /\[cluster3]/!s/^/[cluster3] /
2020-10-08T11:14:40.1915137Z [submariner]$ [cluster1] install_helm
2020-10-08T11:14:40.1915722Z [cluster1] 
2020-10-08T11:14:40.1916059Z [cluster1] Installing helm...
2020-10-08T11:14:40.1917390Z [submariner]$ [cluster1] kubectl -n kube-system create serviceaccount tiller
2020-10-08T11:14:40.1918345Z [submariner]$ [cluster1] kubectl -n kube-system create serviceaccount tiller
2020-10-08T11:14:40.1919393Z [submariner]$ [cluster1] command kubectl --context=cluster1 -n kube-system create serviceaccount tiller
2020-10-08T11:14:40.1920756Z [submariner]$ [cluster1] kubectl --context=cluster1 -n kube-system create serviceaccount tiller
2020-10-08T11:14:40.1921466Z [cluster1] serviceaccount/tiller created
2020-10-08T11:14:40.1922639Z [submariner]$ [cluster1] kubectl create clusterrolebinding tiller --clusterrole=cluster-admin --serviceaccount=kube-system:tiller
2020-10-08T11:14:40.1924111Z [submariner]$ [cluster1] kubectl create clusterrolebinding tiller --clusterrole=cluster-admin --serviceaccount=kube-system:tiller
2020-10-08T11:14:40.1925895Z [submariner]$ [cluster1] command kubectl --context=cluster1 create clusterrolebinding tiller --clusterrole=cluster-admin --serviceaccount=kube-system:tiller
2020-10-08T11:14:40.1927540Z [submariner]$ [cluster1] kubectl --context=cluster1 create clusterrolebinding tiller --clusterrole=cluster-admin --serviceaccount=kube-system:tiller
2020-10-08T11:14:40.1928890Z [cluster1] clusterrolebinding.rbac.authorization.k8s.io/tiller created
2020-10-08T11:14:40.1930167Z [submariner]$ [cluster1] helm --kube-context cluster1 init --service-account tiller
2020-10-08T11:14:40.1930829Z [cluster1] $HELM_HOME has been configured at /root/.helm.
2020-10-08T11:14:40.1931210Z [cluster1]
2020-10-08T11:14:40.1931970Z [cluster1] Tiller (the Helm server-side component) has been installed into your Kubernetes Cluster.
2020-10-08T11:14:40.1932509Z [cluster1]
2020-10-08T11:14:40.1933302Z [cluster1] Please note: by default, Tiller is deployed with an insecure 'allow unauthenticated users' policy.
2020-10-08T11:14:40.1934261Z [cluster1] To prevent this, run `helm init` with the --tiller-tls-verify flag.
2020-10-08T11:14:40.1935127Z [cluster1] For more information on securing your installation see: https://v2.helm.sh/docs/securing_installation/
2020-10-08T11:14:40.1936281Z [submariner]$ [cluster1] kubectl -n kube-system rollout status deploy/tiller-deploy --timeout=30s
2020-10-08T11:14:40.1937327Z [submariner]$ [cluster1] kubectl -n kube-system rollout status deploy/tiller-deploy --timeout=30s
2020-10-08T11:14:40.1938706Z [submariner]$ [cluster1] command kubectl --context=cluster1 -n kube-system rollout status deploy/tiller-deploy --timeout=30s
2020-10-08T11:14:40.1939923Z [submariner]$ [cluster1] kubectl --context=cluster1 -n kube-system rollout status deploy/tiller-deploy --timeout=30s
2020-10-08T11:14:40.1941064Z [cluster1] Waiting for deployment "tiller-deploy" rollout to finish: 0 of 1 updated replicas are available...
2020-10-08T11:14:40.1942008Z [cluster1] deployment "tiller-deploy" successfully rolled out
2020-10-08T11:14:40.7903198Z [submariner]$ [cluster2] install_helm
2020-10-08T11:14:40.7904225Z [cluster2] 
2020-10-08T11:14:40.7905170Z [cluster2] Installing helm...
2020-10-08T11:14:40.7906202Z [submariner]$ [cluster2] kubectl -n kube-system create serviceaccount tiller
2020-10-08T11:14:40.7907354Z [submariner]$ [cluster2] kubectl -n kube-system create serviceaccount tiller
2020-10-08T11:14:40.7908631Z [submariner]$ [cluster2] command kubectl --context=cluster2 -n kube-system create serviceaccount tiller
2020-10-08T11:14:40.7910464Z [submariner]$ [cluster2] kubectl --context=cluster2 -n kube-system create serviceaccount tiller
2020-10-08T11:14:40.7912023Z [cluster2] serviceaccount/tiller created
2020-10-08T11:14:40.7913231Z [submariner]$ [cluster2] kubectl create clusterrolebinding tiller --clusterrole=cluster-admin --serviceaccount=kube-system:tiller
2020-10-08T11:14:40.7914702Z [submariner]$ [cluster2] kubectl create clusterrolebinding tiller --clusterrole=cluster-admin --serviceaccount=kube-system:tiller
2020-10-08T11:14:40.7916280Z [submariner]$ [cluster2] command kubectl --context=cluster2 create clusterrolebinding tiller --clusterrole=cluster-admin --serviceaccount=kube-system:tiller
2020-10-08T11:14:40.7918796Z [submariner]$ [cluster2] kubectl --context=cluster2 create clusterrolebinding tiller --clusterrole=cluster-admin --serviceaccount=kube-system:tiller
2020-10-08T11:14:40.7921000Z [cluster2] clusterrolebinding.rbac.authorization.k8s.io/tiller created
2020-10-08T11:14:40.7922581Z [submariner]$ [cluster2] helm --kube-context cluster2 init --service-account tiller
2020-10-08T11:14:40.7923239Z [cluster2] $HELM_HOME has been configured at /root/.helm.
2020-10-08T11:14:40.7923604Z [cluster2]
2020-10-08T11:14:40.7924407Z [cluster2] Tiller (the Helm server-side component) has been installed into your Kubernetes Cluster.
2020-10-08T11:14:40.7924951Z [cluster2]
2020-10-08T11:14:40.7925746Z [cluster2] Please note: by default, Tiller is deployed with an insecure 'allow unauthenticated users' policy.
2020-10-08T11:14:40.7926714Z [cluster2] To prevent this, run `helm init` with the --tiller-tls-verify flag.
2020-10-08T11:14:40.7927583Z [cluster2] For more information on securing your installation see: https://v2.helm.sh/docs/securing_installation/
2020-10-08T11:14:40.7928731Z [submariner]$ [cluster2] kubectl -n kube-system rollout status deploy/tiller-deploy --timeout=30s
2020-10-08T11:14:40.7929795Z [submariner]$ [cluster2] kubectl -n kube-system rollout status deploy/tiller-deploy --timeout=30s
2020-10-08T11:14:40.7930951Z [submariner]$ [cluster2] command kubectl --context=cluster2 -n kube-system rollout status deploy/tiller-deploy --timeout=30s
2020-10-08T11:14:40.7932156Z [submariner]$ [cluster2] kubectl --context=cluster2 -n kube-system rollout status deploy/tiller-deploy --timeout=30s
2020-10-08T11:14:40.7933289Z [cluster2] Waiting for deployment "tiller-deploy" rollout to finish: 0 of 1 updated replicas are available...
2020-10-08T11:14:40.7934216Z [cluster2] deployment "tiller-deploy" successfully rolled out
2020-10-08T11:14:41.4411625Z [submariner]$ [cluster3] install_helm
2020-10-08T11:14:41.4412193Z [cluster3] 
2020-10-08T11:14:41.4412516Z [cluster3] Installing helm...
2020-10-08T11:14:41.4413281Z [submariner]$ [cluster3] kubectl -n kube-system create serviceaccount tiller
2020-10-08T11:14:41.4414582Z [submariner]$ [cluster3] kubectl -n kube-system create serviceaccount tiller
2020-10-08T11:14:41.4415637Z [submariner]$ [cluster3] command kubectl --context=cluster3 -n kube-system create serviceaccount tiller
2020-10-08T11:14:41.4416934Z [submariner]$ [cluster3] kubectl --context=cluster3 -n kube-system create serviceaccount tiller
2020-10-08T11:14:41.4417630Z [cluster3] serviceaccount/tiller created
2020-10-08T11:14:41.4418783Z [submariner]$ [cluster3] kubectl create clusterrolebinding tiller --clusterrole=cluster-admin --serviceaccount=kube-system:tiller
2020-10-08T11:14:41.4420320Z [submariner]$ [cluster3] kubectl create clusterrolebinding tiller --clusterrole=cluster-admin --serviceaccount=kube-system:tiller
2020-10-08T11:14:41.4422072Z [submariner]$ [cluster3] command kubectl --context=cluster3 create clusterrolebinding tiller --clusterrole=cluster-admin --serviceaccount=kube-system:tiller
2020-10-08T11:14:41.4423751Z [submariner]$ [cluster3] kubectl --context=cluster3 create clusterrolebinding tiller --clusterrole=cluster-admin --serviceaccount=kube-system:tiller
2020-10-08T11:14:41.4425108Z [cluster3] clusterrolebinding.rbac.authorization.k8s.io/tiller created
2020-10-08T11:14:41.4426385Z [submariner]$ [cluster3] helm --kube-context cluster3 init --service-account tiller
2020-10-08T11:14:41.4427034Z [cluster3] $HELM_HOME has been configured at /root/.helm.
2020-10-08T11:14:41.4427638Z [cluster3]
2020-10-08T11:14:41.4428568Z [cluster3] Tiller (the Helm server-side component) has been installed into your Kubernetes Cluster.
2020-10-08T11:14:41.4429275Z [cluster3]
2020-10-08T11:14:41.4430121Z [cluster3] Please note: by default, Tiller is deployed with an insecure 'allow unauthenticated users' policy.
2020-10-08T11:14:41.4431635Z [cluster3] To prevent this, run `helm init` with the --tiller-tls-verify flag.
2020-10-08T11:14:41.4432675Z [cluster3] For more information on securing your installation see: https://v2.helm.sh/docs/securing_installation/
2020-10-08T11:14:41.4434054Z [submariner]$ [cluster3] kubectl -n kube-system rollout status deploy/tiller-deploy --timeout=30s
2020-10-08T11:14:41.4435177Z [submariner]$ [cluster3] kubectl -n kube-system rollout status deploy/tiller-deploy --timeout=30s
2020-10-08T11:14:41.4436332Z [submariner]$ [cluster3] command kubectl --context=cluster3 -n kube-system rollout status deploy/tiller-deploy --timeout=30s
2020-10-08T11:14:41.4472131Z [submariner]$ [cluster3] kubectl --context=cluster3 -n kube-system rollout status deploy/tiller-deploy --timeout=30s
2020-10-08T11:14:41.4473310Z [cluster3] Waiting for deployment "tiller-deploy" rollout to finish: 0 of 1 updated replicas are available...
2020-10-08T11:14:41.4474241Z [cluster3] deployment "tiller-deploy" successfully rolled out
2020-10-08T11:14:41.4475804Z [submariner]$ [cluster3] wait 5148
2020-10-08T11:14:41.4476369Z [submariner]$ [cluster3] wait 5145
2020-10-08T11:14:41.4503981Z [submariner]$ [cluster3] run_subm_clusters prepare_cluster submariner-operator
2020-10-08T11:14:41.4522178Z [submariner]$ [cluster3] run_subm_clusters prepare_cluster submariner-operator
2020-10-08T11:14:41.4556047Z [submariner]$ [cluster3] declare -a subm_clusters
2020-10-08T11:14:41.4646401Z [submariner]$ [cluster3] run_parallel cluster1 cluster2 cluster3 prepare_cluster submariner-operator
2020-10-08T11:14:41.4700587Z [submariner]$ [cluster3] run_parallel cluster1 cluster2 cluster3 cluster1 cluster2 cluster3 prepare_cluster submariner-operator
2020-10-08T11:14:41.4721252Z [submariner]$ [cluster3] local cmnd=prepare_cluster
2020-10-08T11:14:41.4763001Z [submariner]$ [cluster3] declare -A pids
2020-10-08T11:14:41.4811419Z [submariner]$ [cluster3] eval echo cluster1 cluster2 cluster3
2020-10-08T11:14:41.4829179Z [submariner]$ [cluster1] pids[cluster1]=5340
2020-10-08T11:14:41.4864173Z [submariner]$ [cluster1] set -o pipefail
2020-10-08T11:14:41.4872745Z [submariner]$ [cluster2] pids[cluster2]=5343
2020-10-08T11:14:41.4889309Z [submariner]$ [cluster1] prepare_cluster submariner-operator
2020-10-08T11:14:41.4901493Z [submariner]$ [cluster3] pids[cluster3]=5346
2020-10-08T11:14:41.4928704Z [submariner]$ [cluster2] set -o pipefail
2020-10-08T11:14:41.4958661Z [submariner]$ [cluster3] set -o pipefail
2020-10-08T11:14:41.5017342Z [submariner]$ [cluster3] wait 5346
2020-10-08T11:14:41.5034066Z [submariner]$ [cluster1] sed /\[cluster1]/!s/^/[cluster1] /
2020-10-08T11:14:41.5041782Z [submariner]$ [cluster3] prepare_cluster submariner-operator
2020-10-08T11:14:41.5050988Z [submariner]$ [cluster2] prepare_cluster submariner-operator
2020-10-08T11:14:41.5132060Z [submariner]$ [cluster3] sed /\[cluster3]/!s/^/[cluster3] /
2020-10-08T11:14:41.5152991Z [submariner]$ [cluster2] sed /\[cluster2]/!s/^/[cluster2] /
2020-10-08T11:14:46.7476378Z [submariner]$ [cluster2] prepare_cluster
2020-10-08T11:14:46.7478365Z [submariner]$ [cluster2] local namespace=submariner-operator
2020-10-08T11:14:46.7478995Z [cluster2] 
2020-10-08T11:14:46.7479410Z [cluster2] 
2020-10-08T11:14:46.7479819Z [cluster2] 
2020-10-08T11:14:46.7480369Z [submariner]$ [cluster2] add_subm_gateway_label
2020-10-08T11:14:46.7481014Z [submariner]$ [cluster2] add_subm_gateway_label
2020-10-08T11:14:46.7481922Z [submariner]$ [cluster2] kubectl label node cluster2-worker submariner.io/gateway=true --overwrite
2020-10-08T11:14:46.7483047Z [submariner]$ [cluster2] kubectl label node cluster2-worker submariner.io/gateway=true --overwrite
2020-10-08T11:14:46.7484283Z [submariner]$ [cluster2] command kubectl --context=cluster2 label node cluster2-worker submariner.io/gateway=true --overwrite
2020-10-08T11:14:46.7485715Z [submariner]$ [cluster2] kubectl --context=cluster2 label node cluster2-worker submariner.io/gateway=true --overwrite
2020-10-08T11:14:46.7486964Z [cluster2] node/cluster2-worker labeled
2020-10-08T11:14:46.8866668Z [submariner]$ [cluster3] prepare_cluster
2020-10-08T11:14:46.8867493Z [submariner]$ [cluster3] local namespace=submariner-operator
2020-10-08T11:14:46.8868137Z [cluster3] 
2020-10-08T11:14:46.8868553Z [cluster3] 
2020-10-08T11:14:46.8868964Z [cluster3] 
2020-10-08T11:14:46.8869513Z [submariner]$ [cluster3] add_subm_gateway_label
2020-10-08T11:14:46.8870164Z [submariner]$ [cluster3] add_subm_gateway_label
2020-10-08T11:14:46.8871071Z [submariner]$ [cluster3] kubectl label node cluster3-worker submariner.io/gateway=true --overwrite
2020-10-08T11:14:46.8872193Z [submariner]$ [cluster3] kubectl label node cluster3-worker submariner.io/gateway=true --overwrite
2020-10-08T11:14:46.8873420Z [submariner]$ [cluster3] command kubectl --context=cluster3 label node cluster3-worker submariner.io/gateway=true --overwrite
2020-10-08T11:14:46.8874741Z [submariner]$ [cluster3] kubectl --context=cluster3 label node cluster3-worker submariner.io/gateway=true --overwrite
2020-10-08T11:14:46.8875656Z [cluster3] node/cluster3-worker labeled
2020-10-08T11:14:46.8900691Z [submariner]$ [cluster3] wait 5343
2020-10-08T11:14:46.8918654Z [submariner]$ [cluster3] wait 5340
2020-10-08T11:14:47.0009288Z [submariner]$ [cluster1] prepare_cluster
2020-10-08T11:14:47.0010185Z [submariner]$ [cluster1] local namespace=submariner-operator
2020-10-08T11:14:47.0010824Z [cluster1] 
2020-10-08T11:14:47.0011244Z [cluster1] 
2020-10-08T11:14:47.0011653Z [cluster1] 
2020-10-08T11:14:47.0012187Z [submariner]$ [cluster1] add_subm_gateway_label
2020-10-08T11:14:47.0012845Z [submariner]$ [cluster1] add_subm_gateway_label
2020-10-08T11:14:47.0013758Z [submariner]$ [cluster1] kubectl label node cluster1-worker submariner.io/gateway=true --overwrite
2020-10-08T11:14:47.0015267Z [submariner]$ [cluster1] kubectl label node cluster1-worker submariner.io/gateway=true --overwrite
2020-10-08T11:14:47.0016504Z [submariner]$ [cluster1] command kubectl --context=cluster1 label node cluster1-worker submariner.io/gateway=true --overwrite
2020-10-08T11:14:47.0017785Z [submariner]$ [cluster1] kubectl --context=cluster1 label node cluster1-worker submariner.io/gateway=true --overwrite
2020-10-08T11:14:47.0018697Z [cluster1] node/cluster1-worker labeled
2020-10-08T11:14:47.0040331Z [submariner]$ [cluster3] with_context cluster1 setup_broker
2020-10-08T11:14:47.0053493Z [submariner]$ [cluster3] with_context cluster1 setup_broker
2020-10-08T11:14:47.0063451Z [submariner]$ [cluster3] local cluster=cluster1
2020-10-08T11:14:47.0078168Z [submariner]$ [cluster1] local cmnd=setup_broker
2020-10-08T11:14:47.0090875Z [submariner]$ [cluster1] setup_broker
2020-10-08T11:14:47.0103729Z [submariner]$ [cluster1] setup_broker
2020-10-08T11:14:47.0118624Z 
2020-10-08T11:14:47.7044807Z Installing submariner broker...
2020-10-08T11:14:47.7063504Z [submariner]$ [cluster1] helm install submariner-latest/submariner-k8s-broker --kube-context cluster1 --name submariner-k8s-broker --namespace submariner-k8s-broker
2020-10-08T11:14:51.8280093Z NAME:   submariner-k8s-broker
2020-10-08T11:14:51.8651249Z LAST DEPLOYED: Thu Oct  8 11:14:48 2020
2020-10-08T11:14:51.8652940Z NAMESPACE: submariner-k8s-broker
2020-10-08T11:14:51.8654348Z STATUS: DEPLOYED
2020-10-08T11:14:51.8654549Z
2020-10-08T11:14:51.8659047Z RESOURCES:
2020-10-08T11:14:51.8659315Z ==> v1/Role
2020-10-08T11:14:51.8659554Z NAME                          AGE
2020-10-08T11:14:51.8660242Z submariner-k8s-broker:client  0s
2020-10-08T11:14:51.8660541Z
2020-10-08T11:14:51.8660789Z ==> v1/RoleBinding
2020-10-08T11:14:51.8661060Z NAME                          AGE
2020-10-08T11:14:51.8661594Z submariner-k8s-broker:client  0s
2020-10-08T11:14:51.8661918Z
2020-10-08T11:14:51.8662184Z ==> v1/ServiceAccount
2020-10-08T11:14:51.8662766Z NAME                          SECRETS  AGE
2020-10-08T11:14:51.8663477Z submariner-k8s-broker-client  1        0s
2020-10-08T11:14:51.8663834Z
2020-10-08T11:14:51.8663987Z
2020-10-08T11:14:51.8664195Z NOTES:
2020-10-08T11:14:51.8664550Z The Submariner Kubernetes Broker is now setup.
2020-10-08T11:14:51.8664832Z
2020-10-08T11:14:51.8665135Z You can retrieve the server URL by running
2020-10-08T11:14:51.8665382Z
2020-10-08T11:14:51.8666285Z   $ SUBMARINER_BROKER_URL=$(kubectl -n default get endpoints kubernetes -o jsonpath="{.subsets[0].addresses[0].ip}:{.subsets[0].ports[?(@.name=='https')].port}")
2020-10-08T11:14:51.8666873Z
2020-10-08T11:14:51.8667227Z The broker client token and CA can be retrieved by running
2020-10-08T11:14:51.8667519Z
2020-10-08T11:14:51.8668869Z   $ SUBMARINER_BROKER_CA=$(kubectl -n submariner-k8s-broker get secrets -o jsonpath="{.items[?(@.metadata.annotations['kubernetes\.io/service-account\.name']=='submariner-k8s-broker-client')].data['ca\.crt']}")
2020-10-08T11:14:51.8671144Z   $ SUBMARINER_BROKER_TOKEN=$(kubectl -n submariner-k8s-broker get secrets -o jsonpath="{.items[?(@.metadata.annotations['kubernetes\.io/service-account\.name']=='submariner-k8s-broker-client')].data.token}"|base64 --decode)
2020-10-08T11:14:51.8672423Z
2020-10-08T11:14:52.5767769Z [submariner]$ [cluster1] submariner_broker_url=172.17.0.10:6443
2020-10-08T11:14:52.5784350Z [submariner]$ [cluster1] kubectl -n default get endpoints kubernetes -o jsonpath={.subsets[0].addresses[0].ip}:{.subsets[0].ports[?(@.name=='https')].port}
2020-10-08T11:14:52.5800334Z [submariner]$ [cluster1] kubectl -n default get endpoints kubernetes -o jsonpath={.subsets[0].addresses[0].ip}:{.subsets[0].ports[?(@.name=='https')].port}
2020-10-08T11:14:52.5833573Z [submariner]$ [cluster1] command kubectl --context=cluster1 -n default get endpoints kubernetes -o jsonpath={.subsets[0].addresses[0].ip}:{.subsets[0].ports[?(@.name=='https')].port}
2020-10-08T11:14:54.0700137Z [submariner]$ [cluster1] submariner_broker_ca=LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUN5RENDQWJDZ0F3SUJBZ0lCQURBTkJna3Foa2lHOXcwQkFRc0ZBREFWTVJNd0VRWURWUVFERXdwcmRXSmwKY201bGRHVnpNQjRYRFRJd01UQXdPREV4TURjMU0xb1hEVE13TVRBd05qRXhNRGMxTTFvd0ZURVRNQkVHQTFVRQpBeE1LYTNWaVpYSnVaWFJsY3pDQ0FTSXdEUVlKS29aSWh2Y05BUUVCQlFBRGdnRVBBRENDQVFvQ2dnRUJBTE9VCnFqcXE0OEsxOS9ST2tQRkU3bXRuL2VjV011QzRGQlZTTE5tVjBvYW5iZzE2cDZSd2xPankvaUJQcGdUaFcyVWgKcU5Denl6TDY1K1dPeEdGWVR4c2JKdHkyb2hhWHJ1WEYzNlE1QjJSbDA1bTlvTXE5S0FOTlhjcUcxR09rY1pXQgpsQkJhWGtEaTlMODlnd2c1ajMyQ0E5dSswS0UxdUFCRjRBcUpackkxcTlXMEc1bjIrMXJjNXBQMDFaaG1KVHBxCk1XTWNXNTF0TmkvOGhmcjFCTENzQXl0K1J1MVY2QkJ4VmdMaU9kM3Jaak1VZlhaWUk0QWxvL0Nacnl3K2NZb2IKelRXdTU2WlFSb05PSGpLdHN2TG1jTG5JNTVtY0Z6ZHZ2V3MrZE9aYVhtTTRwL3lGN2pZcDA4Z0UrdGdrWU1DUQpuSjdhVnBQTlNhNnZlOVVNWDJNQ0F3RUFBYU1qTUNFd0RnWURWUjBQQVFIL0JBUURBZ0trTUE4R0ExVWRFd0VCCi93UUZNQU1CQWY4d0RRWUpLb1pJaHZjTkFRRUxCUUFEZ2dFQkFKeUZZQnVkaFd3VmNzNGdOek9zb1Q0VVJHbUsKRjhuTWpQU0RsMzFGdFZybVVHdlI5NzR0ZlJKRmRNeEVxbENVa0xERmYyOEs1QmFFVzJLWmtoYzRtSEdwL1V0Rgp2dDM2UmFQYU9aYUFPZEhVZ2ZoSUxrcFZKdHo2RTVRejZxdUF1RnJFZFRGYnR6R0o3L3B3dUpES29sRU04SUtsCmhjaHZyaUtZc0JpVVhadW1sbzJhNWNaS3RTNkZjbzRGdFE3VkZQWVJIbldHT3BDVWo0bXZmc1kvMFZqWmRPbFAKQ0Ewa0h2WkUxUmVmemFkcm9OZ2d5VVB1WkUvbHhmRDU0Z2VpTWVhaWd3dll1aDJzWG85Zllkd3ZrOEUwcXNxVApYT1pvcmpQN2tTV1VqUktQOUpjdGJMT3oreDhWbk43WlBRRnJGc3h5UXB5V1VYelYwWm5mM2NDYWJ6bz0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=
2020-10-08T11:14:54.1334360Z [submariner]$ [cluster1] kubectl -n submariner-k8s-broker get secrets -o jsonpath={.items[?(@.metadata.annotations['kubernetes\.io/service-account\.name']=='submariner-k8s-broker-client')].data['ca\.crt']}
2020-10-08T11:14:54.1336603Z [submariner]$ [cluster1] kubectl -n submariner-k8s-broker get secrets -o jsonpath={.items[?(@.metadata.annotations['kubernetes\.io/service-account\.name']=='submariner-k8s-broker-client')].data['ca\.crt']}
2020-10-08T11:14:54.1345413Z [submariner]$ [cluster1] command kubectl --context=cluster1 -n submariner-k8s-broker get secrets -o jsonpath={.items[?(@.metadata.annotations['kubernetes\.io/service-account\.name']=='submariner-k8s-broker-client')].data['ca\.crt']}
2020-10-08T11:14:55.5352902Z [submariner]$ [cluster1] submariner_broker_token=eyJhbGciOiJSUzI1NiIsImtpZCI6IlMtS2FJYkJ2Rmw5MFAxNzB0T1BxRjNnQVFuMGJtVjNIZm01dUIxNnFDbjAifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJzdWJtYXJpbmVyLWs4cy1icm9rZXIiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlY3JldC5uYW1lIjoic3VibWFyaW5lci1rOHMtYnJva2VyLWNsaWVudC10b2tlbi1oeHNtZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJzdWJtYXJpbmVyLWs4cy1icm9rZXItY2xpZW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiNDU3M2QzOTYtNTkxYy00YmM1LThjM2UtMTRiZTM3NTAwNDVlIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50OnN1Ym1hcmluZXItazhzLWJyb2tlcjpzdWJtYXJpbmVyLWs4cy1icm9rZXItY2xpZW50In0.WcQ5F2WEbTZrkaKszOqOmOuycVZVTP50VucMQtPSIDtX1XYuW9mT2ypYMx44TVlbCr7oR7Lnx3J2NyJly_UejF0R_30JiqV4icQhbY26i77oOpXjcuG8DJvn4aNbQZeVbQ9m-h452Hjv2r_SVXwTA-WwUKjxsuS-P9YXzJ2_qx9QkDNv8QD7xFd9qSGFORhtX9NEPQy_yRykKcXoZ7zYGPalYiKoN1-BrZpN7bxL-_kmBoFbAKVQPAdIMjtK9GrpM0N9ON5xsEgupgUsxeuySDlxYtBeiJrPrXU2v4iL_eZf-tvF-S8f7G69tHpTYUwmRnPxXRukzcSnoCWIFUMUXA
2020-10-08T11:14:55.5638539Z [submariner]$ [cluster1] kubectl -n submariner-k8s-broker get secrets -o jsonpath={.items[?(@.metadata.annotations['kubernetes\.io/service-account\.name']=='submariner-k8s-broker-client')].data.token}
2020-10-08T11:14:55.5639946Z [submariner]$ [cluster1] base64 --decode
2020-10-08T11:14:55.5641429Z [submariner]$ [cluster1] kubectl -n submariner-k8s-broker get secrets -o jsonpath={.items[?(@.metadata.annotations['kubernetes\.io/service-account\.name']=='submariner-k8s-broker-client')].data.token}
2020-10-08T11:14:55.5643733Z [submariner]$ [cluster1] command kubectl --context=cluster1 -n submariner-k8s-broker get secrets -o jsonpath={.items[?(@.metadata.annotations['kubernetes\.io/service-account\.name']=='submariner-k8s-broker-client')].data.token}
2020-10-08T11:14:56.2503840Z [submariner]$ [cluster3] install_subm_all_clusters
2020-10-08T11:14:56.2519500Z [submariner]$ [cluster3] install_subm_all_clusters
2020-10-08T11:14:56.2533106Z [submariner]$ [cluster3] run_subm_clusters helm_install_subm
2020-10-08T11:14:56.2544405Z [submariner]$ [cluster3] run_subm_clusters helm_install_subm
2020-10-08T11:14:56.2558763Z [submariner]$ [cluster3] declare -a subm_clusters
2020-10-08T11:14:56.2604548Z [submariner]$ [cluster3] run_parallel cluster1 cluster2 cluster3 helm_install_subm
2020-10-08T11:14:56.2628869Z [submariner]$ [cluster3] run_parallel cluster1 cluster2 cluster3 cluster1 cluster2 cluster3 helm_install_subm
2020-10-08T11:14:56.2642030Z [submariner]$ [cluster3] local cmnd=helm_install_subm
2020-10-08T11:14:56.2655098Z [submariner]$ [cluster3] declare -A pids
2020-10-08T11:14:56.2672627Z [submariner]$ [cluster3] eval echo cluster1 cluster2 cluster3
2020-10-08T11:14:56.2697884Z [submariner]$ [cluster1] pids[cluster1]=5609
2020-10-08T11:14:56.2705359Z [submariner]$ [cluster1] set -o pipefail
2020-10-08T11:14:56.2718418Z [submariner]$ [cluster1] helm_install_subm
2020-10-08T11:14:56.2730586Z [submariner]$ [cluster2] pids[cluster2]=5612
2020-10-08T11:14:56.2736259Z [submariner]$ [cluster1] sed /\[cluster1]/!s/^/[cluster1] /
2020-10-08T11:14:56.2759582Z [submariner]$ [cluster2] set -o pipefail
2020-10-08T11:14:56.2772790Z [submariner]$ [cluster3] pids[cluster3]=5618
2020-10-08T11:14:56.2791118Z [submariner]$ [cluster2] helm_install_subm
2020-10-08T11:14:56.2816995Z [submariner]$ [cluster3] set -o pipefail
2020-10-08T11:14:56.2828378Z [submariner]$ [cluster3] wait 5618
2020-10-08T11:14:56.2851073Z [submariner]$ [cluster2] sed /\[cluster2]/!s/^/[cluster2] /
2020-10-08T11:14:56.2863288Z [submariner]$ [cluster3] helm_install_subm
2020-10-08T11:14:56.2903602Z [submariner]$ [cluster3] sed /\[cluster3]/!s/^/[cluster3] /
2020-10-08T11:15:00.5994773Z [submariner]$ [cluster2] helm_install_subm
2020-10-08T11:15:00.5995833Z [submariner]$ [cluster2] local crd_create=false
2020-10-08T11:15:00.5997216Z [submariner]$ [cluster2] crd_create=true
2020-10-08T11:15:00.5997953Z [cluster2] 
2020-10-08T11:15:00.5998469Z [cluster2] Installing Submariner...
2020-10-08T11:15:00.6892129Z [submariner]$ [cluster2] helm --kube-context cluster2 install submariner-latest/submariner --name submariner --namespace submariner-operator --set ipsec.psk=TmriFXoiNRKlFNnq8hiuYI9Do4lS5A2UFE16Ph3D1RCTXaY9gcfuNglm6fX0bBxM --set broker.server=172.17.0.10:6443 --set broker.token=eyJhbGciOiJSUzI1NiIsImtpZCI6IlMtS2FJYkJ2Rmw5MFAxNzB0T1BxRjNnQVFuMGJtVjNIZm01dUIxNnFDbjAifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJzdWJtYXJpbmVyLWs4cy1icm9rZXIiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlY3JldC5uYW1lIjoic3VibWFyaW5lci1rOHMtYnJva2VyLWNsaWVudC10b2tlbi1oeHNtZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJzdWJtYXJpbmVyLWs4cy1icm9rZXItY2xpZW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiNDU3M2QzOTYtNTkxYy00YmM1LThjM2UtMTRiZTM3NTAwNDVlIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50OnN1Ym1hcmluZXItazhzLWJyb2tlcjpzdWJtYXJpbmVyLWs4cy1icm9rZXItY2xpZW50In0.WcQ5F2WEbTZrkaKszOqOmOuycVZVTP50VucMQtPSIDtX1XYuW9mT2ypYMx44TVlbCr7oR7Lnx3J2NyJly_UejF0R_30JiqV4icQhbY26i77oOpXjcuG8DJvn4aNbQZeVbQ9m-h452Hjv2r_SVXwTA-WwUKjxsuS-P9YXzJ2_qx9QkDNv8QD7xFd9qSGFORhtX9NEPQy_yRykKcXoZ7zYGPalYiKoN1-BrZpN7bxL-_kmBoFbAKVQPAdIMjtK9GrpM0N9ON5xsEgupgUsxeuySDlxYtBeiJrPrXU2v4iL_eZf-tvF-S8f7G69tHpTYUwmRnPxXRukzcSnoCWIFUMUXA --set broker.namespace=submariner-k8s-broker --set broker.ca=LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUN5RENDQWJDZ0F3SUJBZ0lCQURBTkJna3Foa2lHOXcwQkFRc0ZBREFWTVJNd0VRWURWUVFERXdwcmRXSmwKY201bGRHVnpNQjRYRFRJd01UQXdPREV4TURjMU0xb1hEVE13TVRBd05qRXhNRGMxTTFvd0ZURVRNQkVHQTFVRQpBeE1LYTNWaVpYSnVaWFJsY3pDQ0FTSXdEUVlKS29aSWh2Y05BUUVCQlFBRGdnRVBBRENDQVFvQ2dnRUJBTE9VCnFqcXE0OEsxOS9ST2tQRkU3bXRuL2VjV011QzRGQlZTTE5tVjBvYW5iZzE2cDZSd2xPankvaUJQcGdUaFcyVWgKcU5Denl6TDY1K1dPeEdGWVR4c2JKdHkyb2hhWHJ1WEYzNlE1QjJSbDA1bTlvTXE5S0FOTlhjcUcxR09rY1pXQgpsQkJhWGtEaTlMODlnd2c1ajMyQ0E5dSswS0UxdUFCRjRBcUpackkxcTlXMEc1bjIrMXJjNXBQMDFaaG1KVHBxCk1XTWNXNTF0TmkvOGhmcjFCTENzQXl0K1J1MVY2QkJ4VmdMaU9kM3Jaak1VZlhaWUk0QWxvL0Nacnl3K2NZb2IKelRXdTU2WlFSb05PSGpLdHN2TG1jTG5JNTVtY0Z6ZHZ2V3MrZE9aYVhtTTRwL3lGN2pZcDA4Z0UrdGdrWU1DUQpuSjdhVnBQTlNhNnZlOVVNWDJNQ0F3RUFBYU1qTUNFd0RnWURWUjBQQVFIL0JBUURBZ0trTUE4R0ExVWRFd0VCCi93UUZNQU1CQWY4d0RRWUpLb1pJaHZjTkFRRUxCUUFEZ2dFQkFKeUZZQnVkaFd3VmNzNGdOek9zb1Q0VVJHbUsKRjhuTWpQU0RsMzFGdFZybVVHdlI5NzR0ZlJKRmRNeEVxbENVa0xERmYyOEs1QmFFVzJLWmtoYzRtSEdwL1V0Rgp2dDM2UmFQYU9aYUFPZEhVZ2ZoSUxrcFZKdHo2RTVRejZxdUF1RnJFZFRGYnR6R0o3L3B3dUpES29sRU04SUtsCmhjaHZyaUtZc0JpVVhadW1sbzJhNWNaS3RTNkZjbzRGdFE3VkZQWVJIbldHT3BDVWo0bXZmc1kvMFZqWmRPbFAKQ0Ewa0h2WkUxUmVmemFkcm9OZ2d5VVB1WkUvbHhmRDU0Z2VpTWVhaWd3dll1aDJzWG85Zllkd3ZrOEUwcXNxVApYT1pvcmpQN2tTV1VqUktQOUpjdGJMT3oreDhWbk43WlBRRnJGc3h5UXB5V1VYelYwWm5mM2NDYWJ6bz0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo= --set submariner.clusterId=cluster2 --set submariner.clusterCidr=10.242.0.0/16 --set submariner.serviceCidr=100.92.0.0/16 --set submariner.globalCidr= --set serviceAccounts.globalnet.create=false --set submariner.natEnabled=false --set routeAgent.image.repository=localhost:5000/submariner-route-agent --set routeAgent.image.tag=local --set routeAgent.image.pullPolicy=IfNotPresent --set engine.image.repository=localhost:5000/submariner --set engine.image.tag=local --set engine.image.pullPolicy=IfNotPresent --set globalnet.image.repository=localhost:5000/submariner-globalnet --set globalnet.image.tag=local --set globalnet.image.pullPolicy=IfNotPresent --set crd.create=true
2020-10-08T11:15:00.8271257Z [cluster2] NAME:   submariner
2020-10-08T11:15:00.8271692Z [cluster2] LAST DEPLOYED: Thu Oct  8 11:14:58 2020
2020-10-08T11:15:00.8272826Z [cluster2] NAMESPACE: submariner-operator
2020-10-08T11:15:00.8273540Z [cluster2] STATUS: DEPLOYED
2020-10-08T11:15:00.8273856Z [cluster2]
2020-10-08T11:15:00.8274156Z [cluster2] RESOURCES:
2020-10-08T11:15:00.8274503Z [cluster2] ==> v1/ClusterRole
2020-10-08T11:15:00.8274839Z [cluster2] NAME                   AGE
2020-10-08T11:15:00.8275213Z [cluster2] submariner:routeagent  0s
2020-10-08T11:15:00.8275560Z [cluster2]
2020-10-08T11:15:00.8275930Z [cluster2] ==> v1/ClusterRoleBinding
2020-10-08T11:15:00.8276306Z [cluster2] NAME                   AGE
2020-10-08T11:15:00.8277062Z [cluster2] submariner:routeagent  0s
2020-10-08T11:15:00.8277415Z [cluster2]
2020-10-08T11:15:00.8277714Z [cluster2] ==> v1/DaemonSet
2020-10-08T11:15:00.8278560Z [cluster2] NAME                   DESIRED  CURRENT  READY  UP-TO-DATE  AVAILABLE  NODE SELECTOR               AGE
2020-10-08T11:15:00.8279513Z [cluster2] submariner-gateway     1        1        0      1           0          submariner.io/gateway=true  0s
2020-10-08T11:15:00.8280436Z [cluster2] submariner-routeagent  2        2        0      2           0          <none>                      0s
2020-10-08T11:15:00.8280875Z [cluster2]
2020-10-08T11:15:00.8281176Z [cluster2] ==> v1/Pod(related)
2020-10-08T11:15:00.8281562Z [cluster2] NAME                         READY  STATUS             RESTARTS  AGE
2020-10-08T11:15:00.8282413Z [cluster2] submariner-gateway-6q4bk     0/1    ContainerCreating  0         0s
2020-10-08T11:15:00.8283470Z [cluster2] submariner-routeagent-mwn97  0/1    ContainerCreating  0         0s
2020-10-08T11:15:00.8284560Z [cluster2] submariner-routeagent-xhztm  0/1    ContainerCreating  0         0s
2020-10-08T11:15:00.8285141Z [cluster2]
2020-10-08T11:15:00.8315046Z [cluster2] ==> v1/Role
2020-10-08T11:15:00.8315369Z [cluster2] NAME                   AGE
2020-10-08T11:15:00.8315724Z [cluster2] submariner:engine      0s
2020-10-08T11:15:00.8316123Z [cluster2] submariner:routeagent  0s
2020-10-08T11:15:00.8316648Z [cluster2]
2020-10-08T11:15:00.8317195Z [cluster2] ==> v1/RoleBinding
2020-10-08T11:15:00.8317540Z [cluster2] NAME                   AGE
2020-10-08T11:15:00.8317912Z [cluster2] submariner:routeagent  0s
2020-10-08T11:15:00.8318324Z [cluster2] submariner:engine      0s
2020-10-08T11:15:00.8318646Z [cluster2]
2020-10-08T11:15:00.8318978Z [cluster2] ==> v1/ServiceAccount
2020-10-08T11:15:00.8319346Z [cluster2] NAME                   SECRETS  AGE
2020-10-08T11:15:00.8320123Z [cluster2] submariner-engine      1        0s
2020-10-08T11:15:00.8320802Z [cluster2] submariner-routeagent  1        0s
2020-10-08T11:15:00.8321201Z [cluster2]
2020-10-08T11:15:00.8321456Z [cluster2]
2020-10-08T11:15:00.8321727Z [cluster2] NOTES:
2020-10-08T11:15:00.8322096Z [cluster2] Submariner is now installed.
2020-10-08T11:15:00.8323008Z [cluster2] If you haven't done so yet, please label a node as `submariner.io/gateway=true` to elect it for running Submariner.
2020-10-08T11:15:00.8323599Z [cluster2]
2020-10-08T11:15:00.8324421Z [cluster2] By default, Submariner runs with 1 replica. If you have more than one Gateway host, you can scale Submariner to N replicas, and the other Submariner pods will simply join the leader election pool.
2020-10-08T11:15:13.7954687Z [submariner]$ [cluster1] helm_install_subm
2020-10-08T11:15:13.7955520Z [submariner]$ [cluster1] local crd_create=false
2020-10-08T11:15:13.7956062Z [cluster1] 
2020-10-08T11:15:13.7957174Z [cluster1] Installing Submariner...
2020-10-08T11:15:13.8597180Z [submariner]$ [cluster1] helm --kube-context cluster1 install submariner-latest/submariner --name submariner --namespace submariner-operator --set ipsec.psk=TmriFXoiNRKlFNnq8hiuYI9Do4lS5A2UFE16Ph3D1RCTXaY9gcfuNglm6fX0bBxM --set broker.server=172.17.0.10:6443 --set broker.token=eyJhbGciOiJSUzI1NiIsImtpZCI6IlMtS2FJYkJ2Rmw5MFAxNzB0T1BxRjNnQVFuMGJtVjNIZm01dUIxNnFDbjAifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJzdWJtYXJpbmVyLWs4cy1icm9rZXIiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlY3JldC5uYW1lIjoic3VibWFyaW5lci1rOHMtYnJva2VyLWNsaWVudC10b2tlbi1oeHNtZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJzdWJtYXJpbmVyLWs4cy1icm9rZXItY2xpZW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiNDU3M2QzOTYtNTkxYy00YmM1LThjM2UtMTRiZTM3NTAwNDVlIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50OnN1Ym1hcmluZXItazhzLWJyb2tlcjpzdWJtYXJpbmVyLWs4cy1icm9rZXItY2xpZW50In0.WcQ5F2WEbTZrkaKszOqOmOuycVZVTP50VucMQtPSIDtX1XYuW9mT2ypYMx44TVlbCr7oR7Lnx3J2NyJly_UejF0R_30JiqV4icQhbY26i77oOpXjcuG8DJvn4aNbQZeVbQ9m-h452Hjv2r_SVXwTA-WwUKjxsuS-P9YXzJ2_qx9QkDNv8QD7xFd9qSGFORhtX9NEPQy_yRykKcXoZ7zYGPalYiKoN1-BrZpN7bxL-_kmBoFbAKVQPAdIMjtK9GrpM0N9ON5xsEgupgUsxeuySDlxYtBeiJrPrXU2v4iL_eZf-tvF-S8f7G69tHpTYUwmRnPxXRukzcSnoCWIFUMUXA --set broker.namespace=submariner-k8s-broker --set broker.ca=LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUN5RENDQWJDZ0F3SUJBZ0lCQURBTkJna3Foa2lHOXcwQkFRc0ZBREFWTVJNd0VRWURWUVFERXdwcmRXSmwKY201bGRHVnpNQjRYRFRJd01UQXdPREV4TURjMU0xb1hEVE13TVRBd05qRXhNRGMxTTFvd0ZURVRNQkVHQTFVRQpBeE1LYTNWaVpYSnVaWFJsY3pDQ0FTSXdEUVlKS29aSWh2Y05BUUVCQlFBRGdnRVBBRENDQVFvQ2dnRUJBTE9VCnFqcXE0OEsxOS9ST2tQRkU3bXRuL2VjV011QzRGQlZTTE5tVjBvYW5iZzE2cDZSd2xPankvaUJQcGdUaFcyVWgKcU5Denl6TDY1K1dPeEdGWVR4c2JKdHkyb2hhWHJ1WEYzNlE1QjJSbDA1bTlvTXE5S0FOTlhjcUcxR09rY1pXQgpsQkJhWGtEaTlMODlnd2c1ajMyQ0E5dSswS0UxdUFCRjRBcUpackkxcTlXMEc1bjIrMXJjNXBQMDFaaG1KVHBxCk1XTWNXNTF0TmkvOGhmcjFCTENzQXl0K1J1MVY2QkJ4VmdMaU9kM3Jaak1VZlhaWUk0QWxvL0Nacnl3K2NZb2IKelRXdTU2WlFSb05PSGpLdHN2TG1jTG5JNTVtY0Z6ZHZ2V3MrZE9aYVhtTTRwL3lGN2pZcDA4Z0UrdGdrWU1DUQpuSjdhVnBQTlNhNnZlOVVNWDJNQ0F3RUFBYU1qTUNFd0RnWURWUjBQQVFIL0JBUURBZ0trTUE4R0ExVWRFd0VCCi93UUZNQU1CQWY4d0RRWUpLb1pJaHZjTkFRRUxCUUFEZ2dFQkFKeUZZQnVkaFd3VmNzNGdOek9zb1Q0VVJHbUsKRjhuTWpQU0RsMzFGdFZybVVHdlI5NzR0ZlJKRmRNeEVxbENVa0xERmYyOEs1QmFFVzJLWmtoYzRtSEdwL1V0Rgp2dDM2UmFQYU9aYUFPZEhVZ2ZoSUxrcFZKdHo2RTVRejZxdUF1RnJFZFRGYnR6R0o3L3B3dUpES29sRU04SUtsCmhjaHZyaUtZc0JpVVhadW1sbzJhNWNaS3RTNkZjbzRGdFE3VkZQWVJIbldHT3BDVWo0bXZmc1kvMFZqWmRPbFAKQ0Ewa0h2WkUxUmVmemFkcm9OZ2d5VVB1WkUvbHhmRDU0Z2VpTWVhaWd3dll1aDJzWG85Zllkd3ZrOEUwcXNxVApYT1pvcmpQN2tTV1VqUktQOUpjdGJMT3oreDhWbk43WlBRRnJGc3h5UXB5V1VYelYwWm5mM2NDYWJ6bz0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo= --set submariner.clusterId=cluster1 --set submariner.clusterCidr=10.241.0.0/16 --set submariner.serviceCidr=100.91.0.0/16 --set submariner.globalCidr= --set serviceAccounts.globalnet.create=false --set submariner.natEnabled=false --set routeAgent.image.repository=localhost:5000/submariner-route-agent --set routeAgent.image.tag=local --set routeAgent.image.pullPolicy=IfNotPresent --set engine.image.repository=localhost:5000/submariner --set engine.image.tag=local --set engine.image.pullPolicy=IfNotPresent --set globalnet.image.repository=localhost:5000/submariner-globalnet --set globalnet.image.tag=local --set globalnet.image.pullPolicy=IfNotPresent --set crd.create=false
2020-10-08T11:15:13.9208666Z [cluster1] Error: failed to download "submariner-latest/submariner" (hint: running `helm repo update` may help)
2020-10-08T11:15:29.7701890Z [submariner]$ [cluster3] helm_install_subm
2020-10-08T11:15:29.7709682Z [submariner]$ [cluster3] local crd_create=false
2020-10-08T11:15:29.7710351Z [submariner]$ [cluster3] crd_create=true
2020-10-08T11:15:29.7710912Z [cluster3] 
2020-10-08T11:15:29.7711287Z [cluster3] Installing Submariner...
2020-10-08T11:15:29.8460114Z [submariner]$ [cluster3] helm --kube-context cluster3 install submariner-latest/submariner --name submariner --namespace submariner-operator --set ipsec.psk=TmriFXoiNRKlFNnq8hiuYI9Do4lS5A2UFE16Ph3D1RCTXaY9gcfuNglm6fX0bBxM --set broker.server=172.17.0.10:6443 --set broker.token=eyJhbGciOiJSUzI1NiIsImtpZCI6IlMtS2FJYkJ2Rmw5MFAxNzB0T1BxRjNnQVFuMGJtVjNIZm01dUIxNnFDbjAifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJzdWJtYXJpbmVyLWs4cy1icm9rZXIiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlY3JldC5uYW1lIjoic3VibWFyaW5lci1rOHMtYnJva2VyLWNsaWVudC10b2tlbi1oeHNtZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJzdWJtYXJpbmVyLWs4cy1icm9rZXItY2xpZW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiNDU3M2QzOTYtNTkxYy00YmM1LThjM2UtMTRiZTM3NTAwNDVlIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50OnN1Ym1hcmluZXItazhzLWJyb2tlcjpzdWJtYXJpbmVyLWs4cy1icm9rZXItY2xpZW50In0.WcQ5F2WEbTZrkaKszOqOmOuycVZVTP50VucMQtPSIDtX1XYuW9mT2ypYMx44TVlbCr7oR7Lnx3J2NyJly_UejF0R_30JiqV4icQhbY26i77oOpXjcuG8DJvn4aNbQZeVbQ9m-h452Hjv2r_SVXwTA-WwUKjxsuS-P9YXzJ2_qx9QkDNv8QD7xFd9qSGFORhtX9NEPQy_yRykKcXoZ7zYGPalYiKoN1-BrZpN7bxL-_kmBoFbAKVQPAdIMjtK9GrpM0N9ON5xsEgupgUsxeuySDlxYtBeiJrPrXU2v4iL_eZf-tvF-S8f7G69tHpTYUwmRnPxXRukzcSnoCWIFUMUXA --set broker.namespace=submariner-k8s-broker --set broker.ca=LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUN5RENDQWJDZ0F3SUJBZ0lCQURBTkJna3Foa2lHOXcwQkFRc0ZBREFWTVJNd0VRWURWUVFERXdwcmRXSmwKY201bGRHVnpNQjRYRFRJd01UQXdPREV4TURjMU0xb1hEVE13TVRBd05qRXhNRGMxTTFvd0ZURVRNQkVHQTFVRQpBeE1LYTNWaVpYSnVaWFJsY3pDQ0FTSXdEUVlKS29aSWh2Y05BUUVCQlFBRGdnRVBBRENDQVFvQ2dnRUJBTE9VCnFqcXE0OEsxOS9ST2tQRkU3bXRuL2VjV011QzRGQlZTTE5tVjBvYW5iZzE2cDZSd2xPankvaUJQcGdUaFcyVWgKcU5Denl6TDY1K1dPeEdGWVR4c2JKdHkyb2hhWHJ1WEYzNlE1QjJSbDA1bTlvTXE5S0FOTlhjcUcxR09rY1pXQgpsQkJhWGtEaTlMODlnd2c1ajMyQ0E5dSswS0UxdUFCRjRBcUpackkxcTlXMEc1bjIrMXJjNXBQMDFaaG1KVHBxCk1XTWNXNTF0TmkvOGhmcjFCTENzQXl0K1J1MVY2QkJ4VmdMaU9kM3Jaak1VZlhaWUk0QWxvL0Nacnl3K2NZb2IKelRXdTU2WlFSb05PSGpLdHN2TG1jTG5JNTVtY0Z6ZHZ2V3MrZE9aYVhtTTRwL3lGN2pZcDA4Z0UrdGdrWU1DUQpuSjdhVnBQTlNhNnZlOVVNWDJNQ0F3RUFBYU1qTUNFd0RnWURWUjBQQVFIL0JBUURBZ0trTUE4R0ExVWRFd0VCCi93UUZNQU1CQWY4d0RRWUpLb1pJaHZjTkFRRUxCUUFEZ2dFQkFKeUZZQnVkaFd3VmNzNGdOek9zb1Q0VVJHbUsKRjhuTWpQU0RsMzFGdFZybVVHdlI5NzR0ZlJKRmRNeEVxbENVa0xERmYyOEs1QmFFVzJLWmtoYzRtSEdwL1V0Rgp2dDM2UmFQYU9aYUFPZEhVZ2ZoSUxrcFZKdHo2RTVRejZxdUF1RnJFZFRGYnR6R0o3L3B3dUpES29sRU04SUtsCmhjaHZyaUtZc0JpVVhadW1sbzJhNWNaS3RTNkZjbzRGdFE3VkZQWVJIbldHT3BDVWo0bXZmc1kvMFZqWmRPbFAKQ0Ewa0h2WkUxUmVmemFkcm9OZ2d5VVB1WkUvbHhmRDU0Z2VpTWVhaWd3dll1aDJzWG85Zllkd3ZrOEUwcXNxVApYT1pvcmpQN2tTV1VqUktQOUpjdGJMT3oreDhWbk43WlBRRnJGc3h5UXB5V1VYelYwWm5mM2NDYWJ6bz0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo= --set submariner.clusterId=cluster3 --set submariner.clusterCidr=10.243.0.0/16 --set submariner.serviceCidr=100.93.0.0/16 --set submariner.globalCidr= --set serviceAccounts.globalnet.create=false --set submariner.natEnabled=false --set routeAgent.image.repository=localhost:5000/submariner-route-agent --set routeAgent.image.tag=local --set routeAgent.image.pullPolicy=IfNotPresent --set engine.image.repository=localhost:5000/submariner --set engine.image.tag=local --set engine.image.pullPolicy=IfNotPresent --set globalnet.image.repository=localhost:5000/submariner-globalnet --set globalnet.image.tag=local --set globalnet.image.pullPolicy=IfNotPresent --set crd.create=true
2020-10-08T11:15:29.9503412Z [cluster3] NAME:   submariner
2020-10-08T11:15:29.9503847Z [cluster3] LAST DEPLOYED: Thu Oct  8 11:15:25 2020
2020-10-08T11:15:29.9505534Z [cluster3] NAMESPACE: submariner-operator
2020-10-08T11:15:29.9505992Z [cluster3] STATUS: DEPLOYED
2020-10-08T11:15:29.9506305Z [cluster3]
2020-10-08T11:15:29.9507411Z [cluster3] RESOURCES:
2020-10-08T11:15:29.9507990Z [cluster3] ==> v1/ClusterRole
2020-10-08T11:15:29.9508347Z [cluster3] NAME                   AGE
2020-10-08T11:15:29.9508891Z [cluster3] submariner:routeagent  0s
2020-10-08T11:15:29.9509241Z [cluster3]
2020-10-08T11:15:29.9509592Z [cluster3] ==> v1/ClusterRoleBinding
2020-10-08T11:15:29.9509985Z [cluster3] NAME                   AGE
2020-10-08T11:15:29.9510355Z [cluster3] submariner:routeagent  0s
2020-10-08T11:15:29.9510701Z [cluster3]
2020-10-08T11:15:29.9511001Z [cluster3] ==> v1/DaemonSet
2020-10-08T11:15:29.9511866Z [submariner]$ [cluster3] wait 5612
2020-10-08T11:15:29.9512462Z [submariner]$ [cluster3] wait 5609
2020-10-08T11:15:29.9512918Z make[1]: *** [/opt/shipyard/Makefile.inc:36: deploy] Error 1
2020-10-08T11:15:29.9513407Z make: *** [/opt/shipyard/Makefile.inc:40: e2e] Error 2
2020-10-08T11:15:29.9514019Z [submariner]$ make e2e -- using=helm 
2020-10-08T11:15:29.9514782Z [cluster3] NAME                   DESIRED  CURRENT  READY  UP-TO-DATE  AVAILABLE  NODE SELECTOR               AGE
2020-10-08T11:15:29.9516025Z [cluster3] submariner-gateway     1        1        0      1           0          submariner.io/gateway=true  0s
2020-10-08T11:15:29.9518492Z [cluster3] submariner-routeagent  2        2        0      2           0          <none>                      0s
2020-10-08T11:15:29.9518944Z [cluster3]
2020-10-08T11:15:29.9519246Z [cluster3] ==> v1/Pod(related)
2020-10-08T11:15:29.9519652Z [cluster3] NAME                         READY  STATUS             RESTARTS  AGE
2020-10-08T11:15:29.9520518Z [cluster3] submariner-gateway-pkb8n     0/1    ContainerCreating  0         0s
2020-10-08T11:15:29.9521582Z [cluster3] submariner-routeagent-596j7  0/1    ContainerCreating  0         0s
2020-10-08T11:15:29.9522673Z [cluster3] submariner-routeagent-jll6h  0/1    ContainerCreating  0         0s
2020-10-08T11:15:29.9523251Z [cluster3]
2020-10-08T11:15:29.9523510Z [cluster3] ==> v1/Role
2020-10-08T11:15:29.9523810Z [cluster3] NAME                   AGE
2020-10-08T11:15:29.9524193Z [cluster3] submariner:routeagent  0s
2020-10-08T11:15:29.9524608Z [cluster3] submariner:engine      0s
2020-10-08T11:15:29.9524930Z [cluster3]
2020-10-08T11:15:29.9525242Z [cluster3] ==> v1/RoleBinding
2020-10-08T11:15:29.9525575Z [cluster3] NAME                   AGE
2020-10-08T11:15:29.9525931Z [cluster3] submariner:routeagent  0s
2020-10-08T11:15:29.9526344Z [cluster3] submariner:engine      0s
2020-10-08T11:15:29.9526670Z [cluster3]
2020-10-08T11:15:29.9527003Z [cluster3] ==> v1/ServiceAccount
2020-10-08T11:15:29.9527385Z [cluster3] NAME                   SECRETS  AGE
2020-10-08T11:15:29.9527990Z [cluster3] submariner-engine      1        0s
2020-10-08T11:15:29.9528665Z [cluster3] submariner-routeagent  1        0s
2020-10-08T11:15:29.9529047Z [cluster3]
2020-10-08T11:15:29.9529306Z [cluster3]
2020-10-08T11:15:29.9529582Z [cluster3] NOTES:
2020-10-08T11:15:29.9529950Z [cluster3] Submariner is now installed.
2020-10-08T11:15:29.9532412Z [cluster3] If you haven't done so yet, please label a node as `submariner.io/gateway=true` to elect it for running Submariner.
2020-10-08T11:15:29.9533207Z [cluster3]
2020-10-08T11:15:29.9534029Z [cluster3] By default, Submariner runs with 1 replica. If you have more than one Gateway host, you can scale Submariner to N replicas, and the other Submariner pods will simply join the leader election pool.
2020-10-08T11:15:29.9535311Z make[1]: Leaving directory '/go/src/github.com/submariner-io/submariner'
2020-10-08T11:15:36.5235066Z time="2020-10-08T11:15:36Z" level=fatal msg="exit status 2"
2020-10-08T11:15:36.5246371Z make: *** [e2e] Error 1
2020-10-08T11:15:36.5247175Z Makefile.dapper:14: recipe for target 'e2e' failed
2020-10-08T11:15:36.5314762Z ##[error]Process completed with exit code 2.
2020-10-08T11:15:36.5420418Z ##[group]Run df -h
2020-10-08T11:15:36.5420839Z df -h
2020-10-08T11:15:36.5421079Z free -h
2020-10-08T11:15:36.5421362Z make post-mortem
2020-10-08T11:15:36.6451699Z shell: /bin/bash -e {0}
2020-10-08T11:15:36.6452011Z ##[endgroup]
2020-10-08T11:15:36.6641826Z Filesystem      Size  Used Avail Use% Mounted on
2020-10-08T11:15:36.6642648Z udev            3.4G     0  3.4G   0% /dev
2020-10-08T11:15:36.6643773Z tmpfs           696M  1.4M  695M   1% /run
2020-10-08T11:15:36.6644363Z /dev/sda1        84G   76G  7.8G  91% /
2020-10-08T11:15:36.6644727Z tmpfs           3.4G  163M  3.3G   5% /dev/shm
2020-10-08T11:15:36.6645023Z tmpfs           5.0M     0  5.0M   0% /run/lock
2020-10-08T11:15:36.6645333Z tmpfs           3.4G     0  3.4G   0% /sys/fs/cgroup
2020-10-08T11:15:36.6645637Z /dev/sda15      105M  3.6M  101M   4% /boot/efi
2020-10-08T11:15:36.6645935Z /dev/sdb1        14G  4.1G  9.0G  32% /mnt
2020-10-08T11:15:36.6749938Z               total        used        free      shared  buff/cache   available
2020-10-08T11:15:36.6750847Z Mem:           6.8G        3.3G        238M        257M        3.3G        3.1G
2020-10-08T11:15:36.6751368Z Swap:            0B          0B          0B
2020-10-08T11:15:36.6798073Z ./.dapper -m bind make post-mortem
2020-10-08T11:15:47.3484792Z Sending build context to Docker daemon  69.85MB
2020-10-08T11:15:47.3486673Z
2020-10-08T11:15:47.3628832Z Step 1/5 : FROM quay.io/submariner/shipyard-dapper-base:0.7.1
2020-10-08T11:15:47.3638769Z  ---> c50512fb3462
2020-10-08T11:15:47.3642004Z Step 2/5 : ENV DAPPER_ENV="REPO TAG QUAY_USERNAME QUAY_PASSWORD GITHUB_SHA CLUSTERS_ARGS DEPLOY_ARGS IMAGES_ARGS RELEASE_ARGS"     DAPPER_SOURCE=/go/src/github.com/submariner-io/submariner DAPPER_DOCKER_SOCKET=true
2020-10-08T11:15:47.3658142Z  ---> Using cache
2020-10-08T11:15:47.3659048Z  ---> 107ec502674d
2020-10-08T11:15:47.3662686Z Step 3/5 : ENV DAPPER_OUTPUT=${DAPPER_SOURCE}/output
2020-10-08T11:15:47.3677750Z  ---> Using cache
2020-10-08T11:15:47.3678718Z  ---> 959c7a109766
2020-10-08T11:15:47.3679244Z Step 4/5 : WORKDIR ${DAPPER_SOURCE}
2020-10-08T11:15:47.3693052Z  ---> Using cache
2020-10-08T11:15:47.3693746Z  ---> 380b60cb98ca
2020-10-08T11:15:47.3694351Z Step 5/5 : ENTRYPOINT ["/opt/shipyard/scripts/entry"]
2020-10-08T11:15:47.3811678Z  ---> Using cache
2020-10-08T11:15:47.3812839Z  ---> 9b8bd61411c6
2020-10-08T11:15:47.5494758Z Successfully built 9b8bd61411c6
2020-10-08T11:15:47.5544743Z Successfully tagged submariner:HEAD
2020-10-08T11:15:49.4096381Z [submariner]$ trap chown -R 1001:116 . exit
2020-10-08T11:15:49.4134271Z [submariner]$ mkdir -p bin dist output
2020-10-08T11:15:49.4244354Z [submariner]$ make post-mortem
2020-10-08T11:15:49.5345913Z fatal: No names found, cannot describe anything.
2020-10-08T11:15:49.5619771Z /opt/shipyard/scripts/post_mortem.sh
2020-10-08T11:15:49.5787576Z [submariner]$ source /opt/shipyard/scripts/lib/utils
2020-10-08T11:15:49.5802687Z [submariner]$ . /opt/shipyard/scripts/lib/source_only
2020-10-08T11:15:49.5835243Z [submariner]$ script_name=utils
2020-10-08T11:15:49.5835886Z [submariner]$ exec_name=post_mortem.sh
2020-10-08T11:15:49.5847007Z [submariner]$ declare_kubeconfig
2020-10-08T11:15:49.5858154Z [submariner]$ declare_kubeconfig
2020-10-08T11:15:49.5869126Z [submariner]$ source /opt/shipyard/scripts/lib/kubecfg
2020-10-08T11:15:49.5886308Z [submariner]$ export KUBECONFIG
2020-10-08T11:15:49.5926482Z [submariner]$ KUBECONFIG=/go/src/github.com/submariner-io/submariner/output/kubeconfigs/kind-config-cluster3:/go/src/github.com/submariner-io/submariner/output/kubeconfigs/kind-config-cluster2:/go/src/github.com/submariner-io/submariner/output/kubeconfigs/kind-config-cluster1:
2020-10-08T11:15:49.5957544Z [submariner]$ find /go/src/github.com/submariner-io/submariner/output/kubeconfigs -type f -printf %p:
2020-10-08T11:15:49.6075069Z [submariner]$ kind get clusters
2020-10-08T11:15:50.1639052Z [submariner]$ run_sequential cluster1 cluster2 cluster3 post_analyze
2020-10-08T11:15:50.1652969Z [submariner]$ run_sequential cluster1 cluster2 cluster3 post_analyze
2020-10-08T11:15:50.1666778Z [submariner]$ local cmnd=post_analyze
2020-10-08T11:15:50.1695322Z [submariner]$ eval echo cluster1 cluster2 cluster3
2020-10-08T11:15:50.1718322Z [submariner]$ [cluster1] post_analyze
2020-10-08T11:15:50.1734831Z [submariner]$ [cluster1] sed s/^/[cluster1] /
2020-10-08T11:15:50.1756995Z [submariner]$ [cluster1] post_analyze
2020-10-08T11:15:50.1773294Z [submariner]$ [cluster1] kubectl get all --all-namespaces
2020-10-08T11:15:50.1789861Z [submariner]$ [cluster1] kubectl get all --all-namespaces
2020-10-08T11:15:50.1818208Z [submariner]$ [cluster1] command kubectl --context=cluster1 get all --all-namespaces
2020-10-08T11:15:50.1841400Z [submariner]$ [cluster1] kubectl --context=cluster1 get all --all-namespaces
2020-10-08T11:15:51.5391418Z
2020-10-08T11:15:51.5391792Z
2020-10-08T11:15:51.5392071Z
2020-10-08T11:15:51.5392209Z
2020-10-08T11:15:51.5392396Z
2020-10-08T11:15:51.5401825Z
2020-10-08T11:15:51.5402096Z
2020-10-08T11:15:51.5402266Z
2020-10-08T11:15:51.5402492Z
2020-10-08T11:15:51.5534028Z [submariner]$ [cluster1] kubectl get pods -A
2020-10-08T11:15:51.5546469Z [submariner]$ [cluster1] tail -n +2
2020-10-08T11:15:51.5571500Z [submariner]$ [cluster1] grep -v Running
2020-10-08T11:15:51.5578227Z [submariner]$ [cluster1] kubectl get pods -A
2020-10-08T11:15:51.5597625Z [submariner]$ [cluster1] sed s/  */;/g
2020-10-08T11:15:51.5609708Z [submariner]$ [cluster1] command kubectl --context=cluster1 get pods -A
2020-10-08T11:15:51.5637233Z [submariner]$ [cluster1] kubectl --context=cluster1 get pods -A
2020-10-08T11:15:52.5668877Z [submariner]$ [cluster1] namespace=kube-system
2020-10-08T11:15:52.5680627Z [submariner]$ [cluster1] kubectl get pods --selector=k8s-app=kube-proxy -n kube-system -o jsonpath={.items[*].metadata.name}
2020-10-08T11:15:52.5690005Z [submariner]$ [cluster1] kubectl get pods --selector=k8s-app=kube-proxy -n kube-system -o jsonpath={.items[*].metadata.name}
2020-10-08T11:15:52.5700887Z [submariner]$ [cluster1] command kubectl --context=cluster1 get pods --selector=k8s-app=kube-proxy -n kube-system -o jsonpath={.items[*].metadata.name}
2020-10-08T11:15:52.5710558Z [submariner]$ [cluster1] kubectl --context=cluster1 get pods --selector=k8s-app=kube-proxy -n kube-system -o jsonpath={.items[*].metadata.name}
2020-10-08T11:15:54.0294502Z [submariner]$ [cluster1] kubectl -n kube-system logs kube-proxy-jrjgn
2020-10-08T11:15:54.0307056Z [submariner]$ [cluster1] kubectl -n kube-system logs kube-proxy-jrjgn
2020-10-08T11:15:54.0326120Z [submariner]$ [cluster1] command kubectl --context=cluster1 -n kube-system logs kube-proxy-jrjgn
2020-10-08T11:15:54.0340370Z [submariner]$ [cluster1] kubectl --context=cluster1 -n kube-system logs kube-proxy-jrjgn
2020-10-08T11:15:55.3049442Z [cluster1] ======================= Post mortem cluster1 =======================
2020-10-08T11:15:55.3050093Z [cluster1] NAMESPACE            NAME                                                 READY   STATUS    RESTARTS   AGE
2020-10-08T11:15:55.3051780Z [cluster1] kube-system          pod/coredns-6955765f44-dw47b                         1/1     Running   0          6m33s
2020-10-08T11:15:55.3052750Z [cluster1] kube-system          pod/coredns-6955765f44-jjr7s                         1/1     Running   0          6m33s
2020-10-08T11:15:55.3053766Z [cluster1] kube-system          pod/etcd-cluster1-control-plane                      1/1     Running   0          6m54s
2020-10-08T11:15:55.3054725Z [cluster1] kube-system          pod/kindnet-5ccvs                                    1/1     Running   0          6m33s
2020-10-08T11:15:55.3055743Z [cluster1] kube-system          pod/kindnet-wtrjk                                    1/1     Running   0          6m11s
2020-10-08T11:15:55.3056940Z [cluster1] kube-system          pod/kube-apiserver-cluster1-control-plane            1/1     Running   0          6m54s
2020-10-08T11:15:55.3058510Z [cluster1] kube-system          pod/kube-controller-manager-cluster1-control-plane   1/1     Running   0          6m54s
2020-10-08T11:15:55.3059790Z [cluster1] kube-system          pod/kube-proxy-jrjgn                                 1/1     Running   0          6m11s
2020-10-08T11:15:55.3060652Z [cluster1] kube-system          pod/kube-proxy-v5q8z                                 1/1     Running   0          6m33s
2020-10-08T11:15:55.3061785Z [cluster1] kube-system          pod/kube-scheduler-cluster1-control-plane            1/1     Running   0          6m54s
2020-10-08T11:15:55.3063041Z [cluster1] kube-system          pod/tiller-deploy-8488d98b4c-tk9pf                   1/1     Running   0          84s
2020-10-08T11:15:55.3064357Z [cluster1] local-path-storage   pod/local-path-provisioner-7745554f7f-lr754          1/1     Running   0          6m33s
2020-10-08T11:15:55.3065550Z [cluster1] NAMESPACE     NAME                    TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                  AGE
2020-10-08T11:15:55.3066203Z [cluster1] default       service/kubernetes      ClusterIP   100.91.0.1      <none>        443/TCP                  6m50s
2020-10-08T11:15:55.3067118Z [cluster1] kube-system   service/kube-dns        ClusterIP   100.91.0.10     <none>        53/UDP,53/TCP,9153/TCP   6m48s
2020-10-08T11:15:55.3068052Z [cluster1] kube-system   service/tiller-deploy   ClusterIP   100.91.86.216   <none>        44134/TCP                84s
2020-10-08T11:15:55.3069037Z [cluster1] NAMESPACE     NAME                        DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR                 AGE
2020-10-08T11:15:55.3070009Z [cluster1] kube-system   daemonset.apps/kindnet      2         2         2       2            2           <none>                        6m43s
2020-10-08T11:15:55.3071253Z [cluster1] kube-system   daemonset.apps/kube-proxy   2         2         2       2            2           beta.kubernetes.io/os=linux   6m48s
2020-10-08T11:15:55.3072218Z [cluster1] NAMESPACE            NAME                                     READY   UP-TO-DATE   AVAILABLE   AGE
2020-10-08T11:15:55.3073066Z [cluster1] kube-system          deployment.apps/coredns                  2/2     2            2           6m48s
2020-10-08T11:15:55.3073923Z [cluster1] kube-system          deployment.apps/tiller-deploy            1/1     1            1           84s
2020-10-08T11:15:55.3074962Z [cluster1] local-path-storage   deployment.apps/local-path-provisioner   1/1     1            1           6m39s
2020-10-08T11:15:55.3075716Z [cluster1] NAMESPACE            NAME                                                DESIRED   CURRENT   READY   AGE
2020-10-08T11:15:55.3077041Z [cluster1] kube-system          replicaset.apps/coredns-6955765f44                  2         2         2       6m33s
2020-10-08T11:15:55.3078133Z [cluster1] kube-system          replicaset.apps/tiller-deploy-8488d98b4c            1         1         1       84s
2020-10-08T11:15:55.3079389Z [cluster1] local-path-storage   replicaset.apps/local-path-provisioner-7745554f7f   1         1         1       6m33s
2020-10-08T11:15:55.3080908Z [cluster1] +++++++++++++++++++++: Logs for Pod kube-proxy-jrjgn in namespace kube-system :++++++++++++++++++++++
2020-10-08T11:15:55.3081611Z [cluster1] W1008 11:09:46.538102       1 server_others.go:323] Unknown proxy mode "", assuming iptables proxy
2020-10-08T11:15:55.3082236Z [cluster1] I1008 11:09:46.546532       1 node.go:135] Successfully retrieved node IP: 172.17.0.4
2020-10-08T11:15:55.3082800Z [cluster1] I1008 11:09:46.546565       1 server_others.go:145] Using iptables Proxier.
2020-10-08T11:15:55.3083318Z [cluster1] I1008 11:09:46.546939       1 server.go:571] Version: v1.17.0
2020-10-08T11:15:55.3083941Z [cluster1] I1008 11:09:46.547494       1 conntrack.go:52] Setting nf_conntrack_max to 131072
2020-10-08T11:15:55.3155940Z [submariner]$ [cluster1] kubectl -n kube-system logs kube-proxy-v5q8z
2020-10-08T11:15:55.3168481Z [submariner]$ [cluster1] kubectl -n kube-system logs kube-proxy-v5q8z
2020-10-08T11:15:55.3183419Z [submariner]$ [cluster1] command kubectl --context=cluster1 -n kube-system logs kube-proxy-v5q8z
2020-10-08T11:15:55.3199724Z [submariner]$ [cluster1] kubectl --context=cluster1 -n kube-system logs kube-proxy-v5q8z
2020-10-08T11:15:56.1879026Z [submariner]$ [cluster1] namespace=submariner-operator
2020-10-08T11:15:56.1895660Z [submariner]$ [cluster1] kubectl get pods --selector=name=submariner-operator -n submariner-operator -o jsonpath={.items[*].metadata.name}
2020-10-08T11:15:56.1907223Z [submariner]$ [cluster1] kubectl get pods --selector=name=submariner-operator -n submariner-operator -o jsonpath={.items[*].metadata.name}
2020-10-08T11:15:56.1918745Z [submariner]$ [cluster1] command kubectl --context=cluster1 get pods --selector=name=submariner-operator -n submariner-operator -o jsonpath={.items[*].metadata.name}
2020-10-08T11:15:56.1931815Z [submariner]$ [cluster1] kubectl --context=cluster1 get pods --selector=name=submariner-operator -n submariner-operator -o jsonpath={.items[*].metadata.name}
2020-10-08T11:15:56.9309473Z [submariner]$ [cluster1] namespace=submariner-operator
2020-10-08T11:15:56.9323958Z [submariner]$ [cluster1] kubectl get pods --selector=app=submariner-globalnet -n submariner-operator -o jsonpath={.items[*].metadata.name}
2020-10-08T11:15:56.9335023Z [submariner]$ [cluster1] kubectl get pods --selector=app=submariner-globalnet -n submariner-operator -o jsonpath={.items[*].metadata.name}
2020-10-08T11:15:56.9348166Z [submariner]$ [cluster1] command kubectl --context=cluster1 get pods --selector=app=submariner-globalnet -n submariner-operator -o jsonpath={.items[*].metadata.name}
2020-10-08T11:15:56.9359766Z [submariner]$ [cluster1] kubectl --context=cluster1 get pods --selector=app=submariner-globalnet -n submariner-operator -o jsonpath={.items[*].metadata.name}
2020-10-08T11:15:57.6885195Z [submariner]$ [cluster1] kubectl get pods --selector=app=submariner-engine -n submariner-operator -o jsonpath={.items[*].metadata.name}
2020-10-08T11:15:57.6910512Z [submariner]$ [cluster1] kubectl get pods --selector=app=submariner-engine -n submariner-operator -o jsonpath={.items[*].metadata.name}
2020-10-08T11:15:57.6924476Z [submariner]$ [cluster1] command kubectl --context=cluster1 get pods --selector=app=submariner-engine -n submariner-operator -o jsonpath={.items[*].metadata.name}
2020-10-08T11:15:57.6936284Z [submariner]$ [cluster1] kubectl --context=cluster1 get pods --selector=app=submariner-engine -n submariner-operator -o jsonpath={.items[*].metadata.name}
2020-10-08T11:15:58.4277455Z [submariner]$ [cluster1] kubectl get pods --selector=app=submariner-routeagent -n submariner-operator -o jsonpath={.items[*].metadata.name}
2020-10-08T11:15:58.4288930Z [submariner]$ [cluster1] kubectl get pods --selector=app=submariner-routeagent -n submariner-operator -o jsonpath={.items[*].metadata.name}
2020-10-08T11:15:58.4301671Z [submariner]$ [cluster1] command kubectl --context=cluster1 get pods --selector=app=submariner-routeagent -n submariner-operator -o jsonpath={.items[*].metadata.name}
2020-10-08T11:15:58.4313783Z [submariner]$ [cluster1] kubectl --context=cluster1 get pods --selector=app=submariner-routeagent -n submariner-operator -o jsonpath={.items[*].metadata.name}
2020-10-08T11:15:59.1587016Z [submariner]$ [cluster1] subctl show all
2020-10-08T11:15:59.4196897Z Error obtaining the Submariner resource: the server could not find the requested resource (get submariners.submariner.io submariner)
2020-10-08T11:15:59.4197700Z
2020-10-08T11:15:59.4198696Z subctl version: v0.7.0-pre1-30-g7a9e04d
2020-10-08T11:15:59.4198972Z
2020-10-08T11:15:59.4237222Z [cluster1] I1008 11:09:46.547797       1 conntrack.go:100] Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_established' to 86400
2020-10-08T11:15:59.4246089Z [cluster1] I1008 11:09:46.547863       1 conntrack.go:100] Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_close_wait' to 3600
2020-10-08T11:15:59.4246902Z [cluster1] I1008 11:09:46.549760       1 config.go:313] Starting service config controller
2020-10-08T11:15:59.4247544Z [cluster1] I1008 11:09:46.549908       1 shared_informer.go:197] Waiting for caches to sync for service config
2020-10-08T11:15:59.4248192Z [cluster1] I1008 11:09:46.556364       1 config.go:131] Starting endpoints config controller
2020-10-08T11:15:59.4248855Z [cluster1] I1008 11:09:46.556396       1 shared_informer.go:197] Waiting for caches to sync for endpoints config
2020-10-08T11:15:59.4249525Z [cluster1] I1008 11:09:46.650154       1 shared_informer.go:204] Caches are synced for service config
2020-10-08T11:15:59.4250507Z [cluster1] I1008 11:09:46.656633       1 shared_informer.go:204] Caches are synced for endpoints config
2020-10-08T11:15:59.4251756Z [cluster1] +++++++++++++++++++++: Logs for Pod kube-proxy-v5q8z in namespace kube-system :++++++++++++++++++++++
2020-10-08T11:15:59.4252756Z [cluster1] W1008 11:09:26.543907       1 server_others.go:323] Unknown proxy mode "", assuming iptables proxy
2020-10-08T11:15:59.4253739Z [cluster1] I1008 11:09:26.565777       1 node.go:135] Successfully retrieved node IP: 172.17.0.10
2020-10-08T11:15:59.4254293Z [cluster1] I1008 11:09:26.565989       1 server_others.go:145] Using iptables Proxier.
2020-10-08T11:15:59.4254809Z [cluster1] I1008 11:09:26.566392       1 server.go:571] Version: v1.17.0
2020-10-08T11:15:59.4255852Z [cluster1] I1008 11:09:26.567009       1 conntrack.go:100] Set sysctl 'net/netfilter/nf_conntrack_max' to 131072
2020-10-08T11:15:59.4256499Z [cluster1] I1008 11:09:26.567173       1 conntrack.go:52] Setting nf_conntrack_max to 131072
2020-10-08T11:15:59.4257495Z [cluster1] I1008 11:09:26.569040       1 conntrack.go:100] Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_established' to 86400
2020-10-08T11:15:59.4258547Z [cluster1] I1008 11:09:26.569340       1 conntrack.go:100] Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_close_wait' to 3600
2020-10-08T11:15:59.4259378Z [cluster1] I1008 11:09:26.571822       1 config.go:313] Starting service config controller
2020-10-08T11:15:59.4260014Z [cluster1] I1008 11:09:26.571837       1 shared_informer.go:197] Waiting for caches to sync for service config
2020-10-08T11:15:59.4260658Z [cluster1] I1008 11:09:26.571865       1 config.go:131] Starting endpoints config controller
2020-10-08T11:15:59.4261316Z [cluster1] I1008 11:09:26.571874       1 shared_informer.go:197] Waiting for caches to sync for endpoints config
2020-10-08T11:15:59.4261987Z [cluster1] I1008 11:09:26.671987       1 shared_informer.go:204] Caches are synced for service config
2020-10-08T11:15:59.4262645Z [cluster1] I1008 11:09:26.672081       1 shared_informer.go:204] Caches are synced for endpoints config
2020-10-08T11:15:59.4263117Z [cluster1] ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
2020-10-08T11:15:59.4263436Z [cluster1] ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
2020-10-08T11:15:59.4263964Z [cluster1] ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
2020-10-08T11:15:59.4264282Z [cluster1] ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
2020-10-08T11:15:59.4264737Z [cluster1] ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
2020-10-08T11:15:59.4265023Z [cluster1]
2020-10-08T11:15:59.4265424Z [cluster1] Showing information for cluster "cluster3":
2020-10-08T11:15:59.4266101Z [cluster1]     Discovered network details:
2020-10-08T11:15:59.4267446Z [cluster1]         Network plugin:  weave-net
2020-10-08T11:15:59.4267859Z [cluster1]         Service CIDRs:   [100.93.0.0/16]
2020-10-08T11:15:59.4268328Z [cluster1]         Cluster CIDRs:   [10.243.0.0/16]
2020-10-08T11:15:59.4268655Z [cluster1]
2020-10-08T11:15:59.4269001Z [cluster1] ===================== END Post mortem cluster1 =====================
2020-10-08T11:15:59.4269916Z [submariner]$ [cluster2] post_analyze
2020-10-08T11:15:59.4270546Z [submariner]$ [cluster2] sed s/^/[cluster2] /
2020-10-08T11:15:59.4284842Z [submariner]$ [cluster2] post_analyze
2020-10-08T11:15:59.4300899Z [submariner]$ [cluster2] kubectl get all --all-namespaces
2020-10-08T11:15:59.4311838Z [submariner]$ [cluster2] kubectl get all --all-namespaces
2020-10-08T11:15:59.4324329Z [submariner]$ [cluster2] command kubectl --context=cluster2 get all --all-namespaces
2020-10-08T11:15:59.4335375Z [submariner]$ [cluster2] kubectl --context=cluster2 get all --all-namespaces
2020-10-08T11:16:00.2989999Z
2020-10-08T11:16:00.2993703Z
2020-10-08T11:16:00.2999148Z
2020-10-08T11:16:00.3006356Z
2020-10-08T11:16:00.3012052Z
2020-10-08T11:16:00.3019363Z [cluster2] ======================= Post mortem cluster2 =======================
2020-10-08T11:16:00.3019951Z [cluster2] NAMESPACE             NAME                                                 READY   STATUS    RESTARTS   AGE
2020-10-08T11:16:00.3021453Z [cluster2] kube-system           pod/coredns-6955765f44-7rlfb                         1/1     Running   0          6m29s
2020-10-08T11:16:00.3023581Z [cluster2] kube-system           pod/coredns-6955765f44-g9jlz                         1/1     Running   0          6m29s
2020-10-08T11:16:00.3025012Z [cluster2] kube-system           pod/etcd-cluster2-control-plane                      1/1     Running   0          6m41s
2020-10-08T11:16:00.3029478Z [cluster2] kube-system           pod/kube-apiserver-cluster2-control-plane            1/1     Running   1          6m41s
2020-10-08T11:16:00.3031799Z [cluster2] kube-system           pod/kube-controller-manager-cluster2-control-plane   1/1     Running   0          6m41s
2020-10-08T11:16:00.3033158Z [cluster2] kube-system           pod/kube-proxy-4922b                                 1/1     Running   0          6m6s
2020-10-08T11:16:00.3034038Z [cluster2] kube-system           pod/kube-proxy-fmkdh                                 1/1     Running   0          6m29s
2020-10-08T11:16:00.3034932Z [cluster2] kube-system           pod/kube-proxy-grrnf                                 1/1     Running   0          6m7s
2020-10-08T11:16:00.3036242Z [cluster2] kube-system           pod/kube-scheduler-cluster2-control-plane            1/1     Running   0          6m41s
2020-10-08T11:16:00.3038147Z [cluster2] kube-system           pod/tiller-deploy-8488d98b4c-gjnmv                   1/1     Running   0          93s
2020-10-08T11:16:00.3039519Z [cluster2] kube-system           pod/weave-net-66csr                                  2/2     Running   0          5m46s
2020-10-08T11:16:00.3040395Z [cluster2] kube-system           pod/weave-net-ggqbp                                  2/2     Running   0          5m46s
2020-10-08T11:16:00.3041263Z [cluster2] kube-system           pod/weave-net-w2lrf                                  2/2     Running   0          5m46s
2020-10-08T11:16:00.3042454Z [cluster2] local-path-storage    pod/local-path-provisioner-7745554f7f-lkdq9          1/1     Running   0          6m29s
2020-10-08T11:16:00.3044075Z [cluster2] submariner-operator   pod/submariner-gateway-6q4bk                         1/1     Running   0          60s
2020-10-08T11:16:00.3045290Z [cluster2] submariner-operator   pod/submariner-routeagent-mwn97                      1/1     Running   0          60s
2020-10-08T11:16:00.3046544Z [cluster2] submariner-operator   pod/submariner-routeagent-xhztm                      1/1     Running   0          60s
2020-10-08T11:16:00.3047617Z [cluster2] NAMESPACE     NAME                    TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)                  AGE
2020-10-08T11:16:00.3048400Z [cluster2] default       service/kubernetes      ClusterIP   100.92.0.1     <none>        443/TCP                  6m46s
2020-10-08T11:16:00.3049360Z [cluster2] kube-system   service/kube-dns        ClusterIP   100.92.0.10    <none>        53/UDP,53/TCP,9153/TCP   6m43s
2020-10-08T11:16:00.3050306Z [cluster2] kube-system   service/tiller-deploy   ClusterIP   100.92.175.4   <none>        44134/TCP                93s
2020-10-08T11:16:00.3051346Z [cluster2] NAMESPACE             NAME                                   DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR                 AGE
2020-10-08T11:16:00.3052472Z [cluster2] kube-system           daemonset.apps/kube-proxy              3         3         3       3            3           beta.kubernetes.io/os=linux   6m43s
2020-10-08T11:16:00.3053536Z [cluster2] kube-system           daemonset.apps/weave-net               3         3         3       3            3           <none>                        5m46s
2020-10-08T11:16:00.3054700Z [cluster2] submariner-operator   daemonset.apps/submariner-gateway      1         1         1       1            1           submariner.io/gateway=true    60s
2020-10-08T11:16:00.3056024Z [cluster2] submariner-operator   daemonset.apps/submariner-routeagent   2         2         2       2            2           <none>                        60s
2020-10-08T11:16:00.3057059Z [cluster2] NAMESPACE            NAME                                     READY   UP-TO-DATE   AVAILABLE   AGE
2020-10-08T11:16:00.3058155Z [cluster2] kube-system          deployment.apps/coredns                  2/2     2            2           6m43s
2020-10-08T11:16:00.3059040Z [cluster2] kube-system          deployment.apps/tiller-deploy            1/1     1            1           93s
2020-10-08T11:16:00.3060086Z [cluster2] local-path-storage   deployment.apps/local-path-provisioner   1/1     1            1           6m35s
2020-10-08T11:16:00.3060846Z [cluster2] NAMESPACE            NAME                                                DESIRED   CURRENT   READY   AGE
2020-10-08T11:16:00.3061206Z
2020-10-08T11:16:00.3061363Z
2020-10-08T11:16:00.3061509Z
2020-10-08T11:16:00.3061661Z
2020-10-08T11:16:00.3176967Z [submariner]$ [cluster2] kubectl get pods -A
2020-10-08T11:16:00.3187917Z [submariner]$ [cluster2] tail -n +2
2020-10-08T11:16:00.3199246Z [submariner]$ [cluster2] kubectl get pods -A
2020-10-08T11:16:00.3210659Z [submariner]$ [cluster2] grep -v Running
2020-10-08T11:16:00.3221490Z [submariner]$ [cluster2] command kubectl --context=cluster2 get pods -A
2020-10-08T11:16:00.3238651Z [submariner]$ [cluster2] kubectl --context=cluster2 get pods -A
2020-10-08T11:16:00.3254957Z [submariner]$ [cluster2] sed s/  */;/g
2020-10-08T11:16:01.0690360Z [submariner]$ [cluster2] namespace=kube-system
2020-10-08T11:16:01.0706339Z [submariner]$ [cluster2] kubectl get pods --selector=k8s-app=kube-proxy -n kube-system -o jsonpath={.items[*].metadata.name}
2020-10-08T11:16:01.0719406Z [submariner]$ [cluster2] kubectl get pods --selector=k8s-app=kube-proxy -n kube-system -o jsonpath={.items[*].metadata.name}
2020-10-08T11:16:01.0734715Z [submariner]$ [cluster2] command kubectl --context=cluster2 get pods --selector=k8s-app=kube-proxy -n kube-system -o jsonpath={.items[*].metadata.name}
2020-10-08T11:16:01.0745642Z [submariner]$ [cluster2] kubectl --context=cluster2 get pods --selector=k8s-app=kube-proxy -n kube-system -o jsonpath={.items[*].metadata.name}
2020-10-08T11:16:01.8507946Z [submariner]$ [cluster2] kubectl -n kube-system logs kube-proxy-4922b
2020-10-08T11:16:01.8519421Z [submariner]$ [cluster2] kubectl -n kube-system logs kube-proxy-4922b
2020-10-08T11:16:01.8532324Z [submariner]$ [cluster2] command kubectl --context=cluster2 -n kube-system logs kube-proxy-4922b
2020-10-08T11:16:01.8545967Z [submariner]$ [cluster2] kubectl --context=cluster2 -n kube-system logs kube-proxy-4922b
2020-10-08T11:16:02.6468866Z [submariner]$ [cluster2] kubectl -n kube-system logs kube-proxy-fmkdh
2020-10-08T11:16:02.6474116Z [submariner]$ [cluster2] kubectl -n kube-system logs kube-proxy-fmkdh
2020-10-08T11:16:02.6484698Z [submariner]$ [cluster2] command kubectl --context=cluster2 -n kube-system logs kube-proxy-fmkdh
2020-10-08T11:16:02.6496797Z [submariner]$ [cluster2] kubectl --context=cluster2 -n kube-system logs kube-proxy-fmkdh
2020-10-08T11:16:03.4037304Z [submariner]$ [cluster2] kubectl -n kube-system logs kube-proxy-grrnf
2020-10-08T11:16:03.4048035Z [submariner]$ [cluster2] kubectl -n kube-system logs kube-proxy-grrnf
2020-10-08T11:16:03.4058799Z [submariner]$ [cluster2] command kubectl --context=cluster2 -n kube-system logs kube-proxy-grrnf
2020-10-08T11:16:03.4070076Z [submariner]$ [cluster2] kubectl --context=cluster2 -n kube-system logs kube-proxy-grrnf
2020-10-08T11:16:04.1718100Z [cluster2] kube-system          replicaset.apps/coredns-6955765f44                  2         2         2       6m29s
2020-10-08T11:16:04.1719779Z [cluster2] kube-system          replicaset.apps/tiller-deploy-8488d98b4c            1         1         1       93s
2020-10-08T11:16:04.1721310Z [cluster2] local-path-storage   replicaset.apps/local-path-provisioner-7745554f7f   1         1         1       6m29s
2020-10-08T11:16:04.1722709Z [cluster2] +++++++++++++++++++++: Logs for Pod kube-proxy-4922b in namespace kube-system :++++++++++++++++++++++
2020-10-08T11:16:04.1723565Z [cluster2] W1008 11:10:14.004608       1 server_others.go:323] Unknown proxy mode "", assuming iptables proxy
2020-10-08T11:16:04.1724329Z [cluster2] I1008 11:10:14.018520       1 node.go:135] Successfully retrieved node IP: 172.17.0.8
2020-10-08T11:16:04.1725004Z [cluster2] I1008 11:10:14.018627       1 server_others.go:145] Using iptables Proxier.
2020-10-08T11:16:04.1725630Z [cluster2] I1008 11:10:14.018987       1 server.go:571] Version: v1.17.0
2020-10-08T11:16:04.1726445Z [cluster2] I1008 11:10:14.019550       1 conntrack.go:52] Setting nf_conntrack_max to 131072
2020-10-08T11:16:04.1727608Z [cluster2] I1008 11:10:14.019814       1 conntrack.go:100] Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_established' to 86400
2020-10-08T11:16:04.1728886Z [cluster2] I1008 11:10:14.019894       1 conntrack.go:100] Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_close_wait' to 3600
2020-10-08T11:16:04.1729735Z [cluster2] I1008 11:10:14.023012       1 config.go:313] Starting service config controller
2020-10-08T11:16:04.1730661Z [cluster2] I1008 11:10:14.023025       1 shared_informer.go:197] Waiting for caches to sync for service config
2020-10-08T11:16:04.1731576Z [cluster2] I1008 11:10:14.026571       1 config.go:131] Starting endpoints config controller
2020-10-08T11:16:04.1732347Z [cluster2] I1008 11:10:14.031328       1 shared_informer.go:197] Waiting for caches to sync for endpoints config
2020-10-08T11:16:04.1733108Z [cluster2] I1008 11:10:14.123263       1 shared_informer.go:204] Caches are synced for service config
2020-10-08T11:16:04.1733877Z [cluster2] I1008 11:10:14.132367       1 shared_informer.go:204] Caches are synced for endpoints config
2020-10-08T11:16:04.1734982Z [cluster2] +++++++++++++++++++++: Logs for Pod kube-proxy-fmkdh in namespace kube-system :++++++++++++++++++++++
2020-10-08T11:16:04.1735819Z [cluster2] W1008 11:09:38.182370       1 server_others.go:323] Unknown proxy mode "", assuming iptables proxy
2020-10-08T11:16:04.1736860Z [cluster2] I1008 11:09:38.261173       1 node.go:135] Successfully retrieved node IP: 172.17.0.11
2020-10-08T11:16:04.1737660Z [cluster2] I1008 11:09:38.261445       1 server_others.go:145] Using iptables Proxier.
2020-10-08T11:16:04.1738285Z [cluster2] I1008 11:09:38.262079       1 server.go:571] Version: v1.17.0
2020-10-08T11:16:04.1738907Z [cluster2] I1008 11:09:38.262963       1 conntrack.go:52] Setting nf_conntrack_max to 131072
2020-10-08T11:16:04.1740066Z [cluster2] I1008 11:09:38.266584       1 conntrack.go:100] Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_established' to 86400
2020-10-08T11:16:04.1741471Z [cluster2] I1008 11:09:38.266819       1 conntrack.go:100] Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_close_wait' to 3600
2020-10-08T11:16:04.1743073Z [cluster2] I1008 11:09:38.267104       1 config.go:313] Starting service config controller
2020-10-08T11:16:04.1743945Z [cluster2] I1008 11:09:38.267219       1 shared_informer.go:197] Waiting for caches to sync for service config
2020-10-08T11:16:04.1744707Z [cluster2] I1008 11:09:38.267507       1 config.go:131] Starting endpoints config controller
2020-10-08T11:16:04.1745467Z [cluster2] I1008 11:09:38.273788       1 shared_informer.go:197] Waiting for caches to sync for endpoints config
2020-10-08T11:16:04.1746234Z [cluster2] I1008 11:09:38.367792       1 shared_informer.go:204] Caches are synced for service config
2020-10-08T11:16:04.1746980Z [cluster2] I1008 11:09:38.374105       1 shared_informer.go:204] Caches are synced for endpoints config
2020-10-08T11:16:04.1748353Z [cluster2] +++++++++++++++++++++: Logs for Pod kube-proxy-grrnf in namespace kube-system :++++++++++++++++++++++
2020-10-08T11:16:04.1749205Z [cluster2] W1008 11:10:05.686651       1 server_others.go:323] Unknown proxy mode "", assuming iptables proxy
2020-10-08T11:16:04.1749938Z [cluster2] I1008 11:10:05.697219       1 node.go:135] Successfully retrieved node IP: 172.17.0.7
2020-10-08T11:16:04.1750621Z [cluster2] I1008 11:10:05.697273       1 server_others.go:145] Using iptables Proxier.
2020-10-08T11:16:04.1751228Z [cluster2] I1008 11:10:05.697915       1 server.go:571] Version: v1.17.0
2020-10-08T11:16:04.1751836Z [cluster2] I1008 11:10:05.698657       1 conntrack.go:52] Setting nf_conntrack_max to 131072
2020-10-08T11:16:04.1752957Z [cluster2] I1008 11:10:05.701186       1 conntrack.go:100] Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_established' to 86400
2020-10-08T11:16:04.1754199Z [cluster2] I1008 11:10:05.701575       1 conntrack.go:100] Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_close_wait' to 3600
2020-10-08T11:16:04.1874963Z [submariner]$ [cluster2] namespace=submariner-operator
2020-10-08T11:16:04.1893209Z [submariner]$ [cluster2] kubectl get pods --selector=name=submariner-operator -n submariner-operator -o jsonpath={.items[*].metadata.name}
2020-10-08T11:16:04.1907414Z [submariner]$ [cluster2] kubectl get pods --selector=name=submariner-operator -n submariner-operator -o jsonpath={.items[*].metadata.name}
2020-10-08T11:16:04.1918961Z [submariner]$ [cluster2] command kubectl --context=cluster2 get pods --selector=name=submariner-operator -n submariner-operator -o jsonpath={.items[*].metadata.name}
2020-10-08T11:16:04.1930548Z [submariner]$ [cluster2] kubectl --context=cluster2 get pods --selector=name=submariner-operator -n submariner-operator -o jsonpath={.items[*].metadata.name}
2020-10-08T11:16:04.9863057Z [submariner]$ [cluster2] namespace=submariner-operator
2020-10-08T11:16:04.9885323Z [submariner]$ [cluster2] kubectl get pods --selector=app=submariner-globalnet -n submariner-operator -o jsonpath={.items[*].metadata.name}
2020-10-08T11:16:04.9913292Z [submariner]$ [cluster2] kubectl get pods --selector=app=submariner-globalnet -n submariner-operator -o jsonpath={.items[*].metadata.name}
2020-10-08T11:16:04.9926548Z [submariner]$ [cluster2] command kubectl --context=cluster2 get pods --selector=app=submariner-globalnet -n submariner-operator -o jsonpath={.items[*].metadata.name}
2020-10-08T11:16:04.9949016Z [submariner]$ [cluster2] kubectl --context=cluster2 get pods --selector=app=submariner-globalnet -n submariner-operator -o jsonpath={.items[*].metadata.name}
2020-10-08T11:16:05.7668397Z [submariner]$ [cluster2] kubectl get pods --selector=app=submariner-engine -n submariner-operator -o jsonpath={.items[*].metadata.name}
2020-10-08T11:16:05.7688837Z [submariner]$ [cluster2] kubectl get pods --selector=app=submariner-engine -n submariner-operator -o jsonpath={.items[*].metadata.name}
2020-10-08T11:16:05.7702938Z [submariner]$ [cluster2] command kubectl --context=cluster2 get pods --selector=app=submariner-engine -n submariner-operator -o jsonpath={.items[*].metadata.name}
2020-10-08T11:16:05.7714480Z [submariner]$ [cluster2] kubectl --context=cluster2 get pods --selector=app=submariner-engine -n submariner-operator -o jsonpath={.items[*].metadata.name}
2020-10-08T11:16:06.5152240Z [submariner]$ [cluster2] kubectl -n submariner-operator logs submariner-gateway-6q4bk
2020-10-08T11:16:06.5164531Z [submariner]$ [cluster2] kubectl -n submariner-operator logs submariner-gateway-6q4bk
2020-10-08T11:16:06.5178052Z [submariner]$ [cluster2] command kubectl --context=cluster2 -n submariner-operator logs submariner-gateway-6q4bk
2020-10-08T11:16:06.5190859Z [submariner]$ [cluster2] kubectl --context=cluster2 -n submariner-operator logs submariner-gateway-6q4bk
2020-10-08T11:16:07.2926723Z [cluster2] I1008 11:10:05.702638       1 config.go:313] Starting service config controller
2020-10-08T11:16:07.2927522Z [cluster2] I1008 11:10:05.702788       1 shared_informer.go:197] Waiting for caches to sync for service config
2020-10-08T11:16:07.2928966Z [cluster2] I1008 11:10:05.702922       1 config.go:131] Starting endpoints config controller
2020-10-08T11:16:07.2929701Z [cluster2] I1008 11:10:05.703033       1 shared_informer.go:197] Waiting for caches to sync for endpoints config
2020-10-08T11:16:07.2930395Z [cluster2] I1008 11:10:05.803022       1 shared_informer.go:204] Caches are synced for service config
2020-10-08T11:16:07.2931049Z [cluster2] I1008 11:10:05.803319       1 shared_informer.go:204] Caches are synced for endpoints config
2020-10-08T11:16:07.2931541Z [cluster2] ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
2020-10-08T11:16:07.2931858Z [cluster2] ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
2020-10-08T11:16:07.2932178Z [cluster2] ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
2020-10-08T11:16:07.2933614Z [cluster2] +++++++++++++++++++++: Logs for Pod submariner-gateway-6q4bk in namespace submariner-operator :++++++++++++++++++++++
2020-10-08T11:16:07.2934492Z [cluster2] + trap 'exit 1' SIGTERM SIGINT
2020-10-08T11:16:07.2934948Z [cluster2] + export CHARON_PID_FILE=/var/run/charon.pid
2020-10-08T11:16:07.2935436Z [cluster2] + CHARON_PID_FILE=/var/run/charon.pid
2020-10-08T11:16:07.2936028Z [cluster2] + rm -f /var/run/charon.pid
2020-10-08T11:16:07.2936663Z [cluster2] + SUBMARINER_VERBOSITY=2
2020-10-08T11:16:07.2937248Z [cluster2] + '[' false == true ']'
2020-10-08T11:16:07.2937710Z [cluster2] + DEBUG=-v=2
2020-10-08T11:16:07.2938190Z [cluster2] + mkdir -p /etc/ipsec
2020-10-08T11:16:07.2938850Z [cluster2] + sysctl -w net.ipv4.conf.all.send_redirects=0
2020-10-08T11:16:07.2939468Z [cluster2] net.ipv4.conf.all.send_redirects = 0
2020-10-08T11:16:07.2940140Z [cluster2] + export PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/libexec/strongswan
2020-10-08T11:16:07.2940878Z [cluster2] + PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/libexec/strongswan
2020-10-08T11:16:07.2941643Z [cluster2] + for f in iptables-save iptables
2020-10-08T11:16:07.2942302Z [cluster2] ++ find_iptables_on_host iptables-save
2020-10-08T11:16:07.2943290Z [cluster2] ++ chroot /host test -x /usr/sbin/iptables-save
2020-10-08T11:16:07.2944082Z [cluster2] chroot: cannot change root directory to '/host': No such file or directory
2020-10-08T11:16:07.2944840Z [cluster2] ++ chroot /host test -x /sbin/iptables-save
2020-10-08T11:16:07.2945600Z [cluster2] chroot: cannot change root directory to '/host': No such file or directory
2020-10-08T11:16:07.2946095Z [cluster2] ++ echo unknown
2020-10-08T11:16:07.2946445Z [cluster2] + location=unknown
2020-10-08T11:16:07.2946981Z [cluster2] + '[' unknown '!=' unknown ']'
2020-10-08T11:16:07.2947748Z [cluster2] + echo 'WARNING: not using iptables wrapper because iptables was not detected on the'
2020-10-08T11:16:07.2948612Z [cluster2] WARNING: not using iptables wrapper because iptables was not detected on the
2020-10-08T11:16:07.2949475Z [cluster2] + echo 'host at the following paths [/usr/sbin, /sbin].'
2020-10-08T11:16:07.2949987Z [cluster2] host at the following paths [/usr/sbin, /sbin].
2020-10-08T11:16:07.2950801Z [cluster2] + echo 'Either the host file system isn'\''t mounted or the host does not have iptables'
2020-10-08T11:16:07.2951656Z [cluster2] Either the host file system isn't mounted or the host does not have iptables
2020-10-08T11:16:07.2952522Z [cluster2] + echo 'installed. The pod will use the image installed iptables version.'
2020-10-08T11:16:07.2953197Z [cluster2] installed. The pod will use the image installed iptables version.
2020-10-08T11:16:07.2953929Z [cluster2] + for f in iptables-save iptables
2020-10-08T11:16:07.2954382Z [cluster2] ++ find_iptables_on_host iptables
2020-10-08T11:16:07.2955004Z [cluster2] ++ chroot /host test -x /usr/sbin/iptables
2020-10-08T11:16:07.2955764Z [cluster2] chroot: cannot change root directory to '/host': No such file or directory
2020-10-08T11:16:07.2956781Z [cluster2] ++ chroot /host test -x /sbin/iptables
2020-10-08T11:16:07.2957769Z [cluster2] chroot: cannot change root directory to '/host': No such file or directory
2020-10-08T11:16:07.2958254Z [cluster2] ++ echo unknown
2020-10-08T11:16:07.2958614Z [cluster2] + location=unknown
2020-10-08T11:16:07.2959184Z [cluster2] + '[' unknown '!=' unknown ']'
2020-10-08T11:16:07.2961379Z [cluster2] + echo 'WARNING: not using iptables wrapper because iptables was not detected on the'
2020-10-08T11:16:07.2962138Z [cluster2] WARNING: not using iptables wrapper because iptables was not detected on the
2020-10-08T11:16:07.2962976Z [cluster2] + echo 'host at the following paths [/usr/sbin, /sbin].'
2020-10-08T11:16:07.2963648Z [cluster2] host at the following paths [/usr/sbin, /sbin].
2020-10-08T11:16:07.2967976Z [cluster2] + echo 'Either the host file system isn'\''t mounted or the host does not have iptables'
2020-10-08T11:16:07.2968882Z [cluster2] Either the host file system isn't mounted or the host does not have iptables
2020-10-08T11:16:07.2969751Z [cluster2] + echo 'installed. The pod will use the image installed iptables version.'
2020-10-08T11:16:07.2970482Z [cluster2] installed. The pod will use the image installed iptables version.
2020-10-08T11:16:07.2971343Z [cluster2] + exec submariner-engine -v=2 -alsologtostderr
2020-10-08T11:16:07.2971964Z [cluster2] I1008 11:15:23.771620       1 main.go:71] Starting the submariner gateway engine
2020-10-08T11:16:07.2973066Z [cluster2] W1008 11:15:23.772274       1 client_config.go:543] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
2020-10-08T11:16:07.2973838Z [cluster2] I1008 11:15:23.772664       1 main.go:99] Creating the cable engine
2020-10-08T11:16:07.2974364Z [cluster2] I1008 11:15:23.772828       1 syncer.go:51] CableEngine syncer started
2020-10-08T11:16:07.2975231Z [cluster2] I1008 11:15:23.773620       1 main.go:263] Gateway leader election config values: main.leaderConfig{LeaseDuration:10, RenewDeadline:5, RetryPeriod:2}
2020-10-08T11:16:07.2976485Z [cluster2] I1008 11:15:23.773846       1 main.go:280] Using namespace "submariner-operator" for the leader election lock
2020-10-08T11:16:07.2977815Z [cluster2] I1008 11:15:23.773860       1 leaderelection.go:242] attempting to acquire leader lease  submariner-operator/submariner-engine-lock...
2020-10-08T11:16:07.2979557Z [cluster2] I1008 11:15:23.793337       1 leaderelection.go:252] successfully acquired lease submariner-operator/submariner-engine-lock
2020-10-08T11:16:07.2982084Z [cluster2] I1008 11:15:23.793385       1 event.go:281] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"submariner-operator", Name:"submariner-engine-lock", UID:"6dc49e52-6264-4743-bf95-01e2d507bd86", APIVersion:"v1", ResourceVersion:"1623", FieldPath:""}): type: 'Normal' reason: 'LeaderElection' cluster2-worker-submariner-engine became leader
2020-10-08T11:16:07.2983897Z [cluster2] I1008 11:15:23.793419       1 main.go:133] Creating the tunnel controller
2020-10-08T11:16:07.2984500Z [cluster2] I1008 11:15:23.793500       1 main.go:141] Creating the kubernetes central datastore
2020-10-08T11:16:07.2985224Z [cluster2] I1008 11:15:23.793532       1 kubernetes.go:60] Rendered API server host: "https://172.17.0.10:6443"
2020-10-08T11:16:07.2985892Z [cluster2] I1008 11:15:23.851879       1 main.go:151] Creating the datastore syncer
2020-10-08T11:16:07.2986507Z [cluster2] I1008 11:15:23.852351       1 strongswan.go:99] Initializing StrongSwan IPSec driver
2020-10-08T11:16:07.2987112Z [cluster2] I1008 11:15:23.855619       1 strongswan.go:460] Starting charon
2020-10-08T11:16:07.2988202Z [cluster2] W1008 11:15:23.860082       1 strongswan.go:543] Failed to connect to charon - retrying: dial unix /var/run/charon.vici: connect: no such file or directory
2020-10-08T11:16:07.2989253Z [cluster2] 00[DMN] Starting IKE charon daemon (strongSwan 5.9.0, Linux 5.4.0-1026-azure, x86_64)
2020-10-08T11:16:07.2990005Z [cluster2] 00[CFG] PKCS11 module '<name>' lacks library path
2020-10-08T11:16:07.2990650Z [cluster2] 00[LIB] openssl FIPS mode(0) - disabled
2020-10-08T11:16:07.2991869Z [cluster2] 00[CFG] loading ca certificates from '/etc/strongswan/ipsec.d/cacerts'
2020-10-08T11:16:07.2993505Z [cluster2] 00[CFG] loading aa certificates from '/etc/strongswan/ipsec.d/aacerts'
2020-10-08T11:16:07.2994470Z [cluster2] 00[CFG] loading ocsp signer certificates from '/etc/strongswan/ipsec.d/ocspcerts'
2020-10-08T11:16:07.2995432Z [cluster2] 00[CFG] loading attribute certificates from '/etc/strongswan/ipsec.d/acerts'
2020-10-08T11:16:07.2996273Z [cluster2] 00[CFG] loading crls from '/etc/strongswan/ipsec.d/crls'
2020-10-08T11:16:07.2997882Z [cluster2] 00[CFG] loading secrets from '/etc/strongswan/ipsec.secrets'
2020-10-08T11:16:07.2998640Z [cluster2] 00[CFG] opening triplet file /etc/strongswan/ipsec.d/triplets.dat failed: No such file or directory
2020-10-08T11:16:07.2999325Z [cluster2] 00[CFG] loaded 0 RADIUS server configurations
2020-10-08T11:16:07.2999980Z [cluster2] 00[CFG] HA config misses local/remote address
2020-10-08T11:16:07.3000857Z [cluster2] 00[CFG] no script for ext-auth script defined, disabled
2020-10-08T11:16:07.3003629Z [cluster2] 00[LIB] loaded plugins: charon pkcs11 tpm aesni aes des rc2 sha2 sha1 md4 md5 mgf1 random nonce x509 revocation constraints acert pubkey pkcs1 pkcs7 pkcs8 pkcs12 pgp dnskey sshkey pem openssl gcrypt fips-prf gmp curve25519 chapoly xcbc cmac hmac ctr ccm gcm drbg newhope curl attr kernel-netlink resolve socket-default farp stroke vici updown eap-identity eap-sim eap-aka eap-aka-3gpp eap-aka-3gpp2 eap-md5 eap-gtc eap-mschapv2 eap-dynamic eap-radius eap-tls eap-ttls eap-peap xauth-generic xauth-eap xauth-pam xauth-noauth dhcp led duplicheck unity counters
2020-10-08T11:16:07.3005689Z [cluster2] 00[JOB] spawning 16 worker threads
2020-10-08T11:16:07.3006322Z [cluster2] I1008 11:15:24.861345       1 cableengine.go:71] CableEngine controller started, driver: "strongswan"
2020-10-08T11:16:07.3007088Z [cluster2] I1008 11:15:24.861396       1 datastoresyncer.go:145] Starting the datastore syncer
2020-10-08T11:16:07.3007797Z [cluster2] I1008 11:15:24.861409       1 datastoresyncer.go:147] Waiting for informer caches to sync
2020-10-08T11:16:07.3008569Z [cluster2] I1008 11:15:24.861426       1 datastoresyncer.go:79] Ensuring we are the only endpoint active for this cluster
2020-10-08T11:16:07.3009427Z [cluster2] I1008 11:15:24.861441       1 tunnel.go:47] Starting the tunnel controller
2020-10-08T11:16:07.3009990Z [cluster2] I1008 11:15:24.861459       1 tunnel.go:50] Waiting for informer caches to sync
2020-10-08T11:16:07.3010540Z [cluster2] I1008 11:15:24.861473       1 tunnel.go:58] Tunnel controller started
2020-10-08T11:16:07.3011866Z [cluster2] I1008 11:15:24.863972       1 datastoresyncer.go:158] Reconciling local submariner Cluster: types.SubmarinerCluster{ID:"cluster2", Spec:v1.ClusterSpec{ClusterID:"cluster2", ColorCodes:[]string{"blue"}, ServiceCIDR:[]string{"100.92.0.0/16"}, ClusterCIDR:[]string{"10.242.0.0/16"}, GlobalCIDR:[]string{}}}
2020-10-08T11:16:07.3014028Z [cluster2] I1008 11:15:24.864056       1 datastoresyncer.go:312] In reconcileClusterCRD: &types.SubmarinerCluster{ID:"cluster2", Spec:v1.ClusterSpec{ClusterID:"cluster2", ColorCodes:[]string{"blue"}, ServiceCIDR:[]string{"100.92.0.0/16"}, ClusterCIDR:[]string{"10.242.0.0/16"}, GlobalCIDR:[]string{}}}
2020-10-08T11:16:07.3015585Z [cluster2] I1008 11:15:24.868823       1 datastoresyncer.go:357] Successfully created submariner Cluster "cluster2" in the local datastore
2020-10-08T11:16:07.3018273Z [cluster2] I1008 11:15:24.868878       1 datastoresyncer.go:164] Reconciling local submariner Endpoint: types.SubmarinerEndpoint{Spec:v1.EndpointSpec{ClusterID:"cluster2", CableName:"submariner-cable-cluster2-172-17-0-8", Hostname:"cluster2-worker", Subnets:[]string{"100.92.0.0/16", "10.242.0.0/16"}, PrivateIP:"172.17.0.8", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string{}}}
2020-10-08T11:16:07.3021979Z [cluster2] I1008 11:15:24.868907       1 datastoresyncer.go:387] In reconcileEndpointCRD: &types.SubmarinerEndpoint{Spec:v1.EndpointSpec{ClusterID:"cluster2", CableName:"submariner-cable-cluster2-172-17-0-8", Hostname:"cluster2-worker", Subnets:[]string{"100.92.0.0/16", "10.242.0.0/16"}, PrivateIP:"172.17.0.8", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string{}}}
2020-10-08T11:16:07.3024830Z [cluster2] I1008 11:15:24.877001       1 datastoresyncer.go:432] Successfully created submariner Endpoint "cluster2-submariner-cable-cluster2-172-17-0-8" in the local datastore
2020-10-08T11:16:07.3026131Z [cluster2] I1008 11:15:24.877065       1 datastoresyncer.go:177] Datastore syncer started
2020-10-08T11:16:07.3027388Z [cluster2] I1008 11:15:24.877663       1 kubernetes.go:317] In SetCluster: &types.SubmarinerCluster{ID:"cluster2", Spec:v1.ClusterSpec{ClusterID:"cluster2", ColorCodes:[]string{"blue"}, ServiceCIDR:[]string{"100.92.0.0/16"}, ClusterCIDR:[]string{"10.242.0.0/16"}, GlobalCIDR:[]string{}}}
2020-10-08T11:16:07.3034453Z [cluster2] I1008 11:15:24.880757       1 tunnel.go:95] Tunnel controller processing added or updated submariner Endpoint object: &v1.Endpoint{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"cluster2-submariner-cable-cluster2-172-17-0-8", GenerateName:"", Namespace:"submariner-operator", SelfLink:"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster2-submariner-cable-cluster2-172-17-0-8", UID:"0ca8b389-f4b4-453d-a2cb-7b856a53e2e6", ResourceVersion:"1629", Generation:1, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63737752524, loc:(*time.Location)(0x225ef80)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v1.EndpointSpec{ClusterID:"cluster2", CableName:"submariner-cable-cluster2-172-17-0-8", Hostname:"cluster2-worker", Subnets:[]string{"100.92.0.0/16", "10.242.0.0/16"}, PrivateIP:"172.17.0.8", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
2020-10-08T11:16:07.3039856Z [cluster2] I1008 11:15:24.880919       1 cableengine.go:92] Not installing cable for local cluster
2020-10-08T11:16:07.3041486Z [cluster2] I1008 11:15:24.880989       1 tunnel.go:108] Tunnel controller successfully installed Endpoint cable submariner-cable-cluster2-172-17-0-8 in the engine
2020-10-08T11:16:07.3042690Z [cluster2] 08[KNL] interface vx-submariner activated
2020-10-08T11:16:07.3043888Z [cluster2] 11[KNL] 240.17.0.8 appeared on vx-submariner
2020-10-08T11:16:07.3050400Z [cluster2] I1008 11:15:24.898603       1 datastoresyncer.go:287] Processing local submariner Endpoint object: &v1.Endpoint{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"cluster2-submariner-cable-cluster2-172-17-0-8", GenerateName:"", Namespace:"submariner-operator", SelfLink:"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster2-submariner-cable-cluster2-172-17-0-8", UID:"0ca8b389-f4b4-453d-a2cb-7b856a53e2e6", ResourceVersion:"1629", Generation:1, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63737752524, loc:(*time.Location)(0x225ef80)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v1.EndpointSpec{ClusterID:"cluster2", CableName:"submariner-cable-cluster2-172-17-0-8", Hostname:"cluster2-worker", Subnets:[]string{"100.92.0.0/16", "10.242.0.0/16"}, PrivateIP:"172.17.0.8", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
2020-10-08T11:16:07.3057359Z [cluster2] I1008 11:15:24.898666       1 kubernetes.go:370] In SetEndpoint: &types.SubmarinerEndpoint{Spec:v1.EndpointSpec{ClusterID:"cluster2", CableName:"submariner-cable-cluster2-172-17-0-8", Hostname:"cluster2-worker", Subnets:[]string{"100.92.0.0/16", "10.242.0.0/16"}, PrivateIP:"172.17.0.8", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
2020-10-08T11:16:07.3059164Z [cluster2] I1008 11:15:24.902427       1 kubernetes.go:155] AddFunc in WatchClusters called
2020-10-08T11:16:07.3059975Z [cluster2] I1008 11:15:24.903057       1 kubernetes.go:342] Successfully created submariner Cluster "cluster2" in the central datastore
2020-10-08T11:16:07.3060778Z [cluster2] I1008 11:15:24.905923       1 kubernetes.go:240] AddFunc in WatchEndpoints called
2020-10-08T11:16:07.3067918Z [cluster2] I1008 11:15:24.906371       1 kubernetes.go:395] Successfully created submariner Endpoint "cluster2-submariner-cable-cluster2-172-17-0-8" in the central datastore
2020-10-08T11:16:07.3069375Z [cluster2] I1008 11:15:48.648651       1 kubernetes.go:155] AddFunc in WatchClusters called
2020-10-08T11:16:07.3071184Z [cluster2] I1008 11:15:48.648690       1 datastoresyncer.go:312] In reconcileClusterCRD: &types.SubmarinerCluster{ID:"cluster3", Spec:v1.ClusterSpec{ClusterID:"cluster3", ColorCodes:[]string{"blue"}, ServiceCIDR:[]string{"100.93.0.0/16"}, ClusterCIDR:[]string{"10.243.0.0/16"}, GlobalCIDR:[]string{}}}
2020-10-08T11:16:07.3072748Z [cluster2] I1008 11:15:48.656321       1 datastoresyncer.go:357] Successfully created submariner Cluster "cluster3" in the local datastore
2020-10-08T11:16:07.3073587Z [cluster2] I1008 11:15:48.661045       1 kubernetes.go:240] AddFunc in WatchEndpoints called
2020-10-08T11:16:07.3076375Z [cluster2] I1008 11:15:48.661067       1 datastoresyncer.go:387] In reconcileEndpointCRD: &types.SubmarinerEndpoint{Spec:v1.EndpointSpec{ClusterID:"cluster3", CableName:"submariner-cable-cluster3-172-17-0-5", Hostname:"cluster3-worker", Subnets:[]string{"100.93.0.0/16", "10.243.0.0/16"}, PrivateIP:"172.17.0.5", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
2020-10-08T11:16:07.3078443Z [cluster2] 16[NET] received packet: from 172.17.0.5[500] to 172.17.0.8[500] (500 bytes)
2020-10-08T11:16:07.3079059Z [cluster2] 16[ENC] parsed IKE_SA_INIT request 0 [ SA KE No N(NATD_S_IP) N(NATD_D_IP) N(FRAG_SUP) N(HASH_ALG) N(REDIR_SUP) ]
2020-10-08T11:16:07.3079895Z [cluster2] 16[IKE] no IKE config found for 172.17.0.8...172.17.0.5, sending NO_PROPOSAL_CHOSEN
2020-10-08T11:16:07.3080457Z [cluster2] 16[ENC] generating IKE_SA_INIT response 0 [ N(NO_PROP) ]
2020-10-08T11:16:07.3080965Z [cluster2] 16[NET] sending packet: from 172.17.0.8[500] to 172.17.0.5[500] (36 bytes)
2020-10-08T11:16:07.3090959Z [cluster2] I1008 11:15:48.725822       1 tunnel.go:95] Tunnel controller processing added or updated submariner Endpoint object: &v1.Endpoint{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"cluster3-submariner-cable-cluster3-172-17-0-5", GenerateName:"", Namespace:"submariner-operator", SelfLink:"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster3-submariner-cable-cluster3-172-17-0-5", UID:"9871d64e-98a5-4a6b-9983-faf9b0a1d129", ResourceVersion:"1713", Generation:1, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63737752548, loc:(*time.Location)(0x225ef80)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v1.EndpointSpec{ClusterID:"cluster3", CableName:"submariner-cable-cluster3-172-17-0-5", Hostname:"cluster3-worker", Subnets:[]string{"100.93.0.0/16", "10.243.0.0/16"}, PrivateIP:"172.17.0.5", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
2020-10-08T11:16:07.3105216Z [cluster2] I1008 11:15:48.725944       1 cableengine.go:101] Installing Endpoint cable "submariner-cable-cluster3-172-17-0-5"
2020-10-08T11:16:07.3108849Z [cluster2] 07[CFG] loaded IKE shared key for: '172.17.0.5'
2020-10-08T11:16:07.3110431Z [cluster2] I1008 11:15:48.727770       1 datastoresyncer.go:432] Successfully created submariner Endpoint "cluster3-submariner-cable-cluster3-172-17-0-5" in the local datastore
2020-10-08T11:16:07.3112141Z [cluster2] 12[CFG] added vici connection: submariner-cable-cluster3-172-17-0-5
2020-10-08T11:16:07.3113762Z [cluster2] 12[CFG] initiating 'submariner-child-submariner-cable-cluster3-172-17-0-5'
2020-10-08T11:16:07.3119770Z [cluster2] 12[IKE] initiating IKE_SA submariner-cable-cluster3-172-17-0-5[2] to 172.17.0.5
2020-10-08T11:16:07.3121132Z [cluster2] 12[ENC] generating IKE_SA_INIT request 0 [ SA KE No N(NATD_S_IP) N(NATD_D_IP) N(FRAG_SUP) N(HASH_ALG) N(REDIR_SUP) ]
2020-10-08T11:16:07.3122627Z [cluster2] 12[NET] sending packet: from 172.17.0.8[500] to 172.17.0.5[500] (500 bytes)
2020-10-08T11:16:07.3123864Z [cluster2] I1008 11:15:48.746683       1 datastoresyncer.go:271] The updated submariner Endpoint "cluster3" is not for this cluster - skipping updating the datastore
2020-10-08T11:16:07.3125448Z [cluster2] I1008 11:15:48.751217       1 cableengine.go:150] Successfully installed Endpoint cable "submariner-cable-cluster3-172-17-0-5" with remote IP 172.17.0.5
2020-10-08T11:16:07.3127205Z [cluster2] I1008 11:15:48.751245       1 tunnel.go:108] Tunnel controller successfully installed Endpoint cable submariner-cable-cluster3-172-17-0-5 in the engine
2020-10-08T11:16:07.3128201Z [cluster2] 09[NET] received packet: from 172.17.0.5[500] to 172.17.0.8[500] (464 bytes)
2020-10-08T11:16:07.3128849Z [cluster2] 09[ENC] parsed IKE_SA_INIT response 0 [ SA KE No N(NATD_S_IP) N(NATD_D_IP) N(FRAG_SUP) N(HASH_ALG) N(CHDLESS_SUP) N(MULT_AUTH) ]
2020-10-08T11:16:07.3129530Z [cluster2] 09[CFG] selected proposal: IKE:AES_GCM_16_128/PRF_HMAC_SHA2_256/MODP_2048
2020-10-08T11:16:07.3130023Z [cluster2] 09[IKE] remote host is behind NAT
2020-10-08T11:16:07.3130742Z [cluster2] 09[IKE] authentication of '172.17.0.8' (myself) with pre-shared key
2020-10-08T11:16:07.3132173Z [cluster2] 09[IKE] establishing CHILD_SA submariner-child-submariner-cable-cluster3-172-17-0-5{1}
2020-10-08T11:16:07.3134323Z [cluster2] 09[ENC] generating IKE_AUTH request 1 [ IDi N(INIT_CONTACT) IDr AUTH SA TSi TSr N(MULT_AUTH) N(EAP_ONLY) N(MSG_ID_SYN_SUP) ]
2020-10-08T11:16:07.3135007Z [cluster2] 09[NET] sending packet: from 172.17.0.8[4500] to 172.17.0.5[4500] (333 bytes)
2020-10-08T11:16:07.3135713Z [cluster2] 07[NET] received packet: from 172.17.0.5[4500] to 172.17.0.8[4500] (257 bytes)
2020-10-08T11:16:07.3136221Z [cluster2] 07[ENC] parsed IKE_AUTH response 1 [ IDr AUTH SA TSi TSr ]
2020-10-08T11:16:07.3137109Z [cluster2] 07[IKE] authentication of '172.17.0.5' with pre-shared key successful
2020-10-08T11:16:07.3138275Z [cluster2] 07[IKE] IKE_SA submariner-cable-cluster3-172-17-0-5[2] established between 172.17.0.8[172.17.0.8]...172.17.0.5[172.17.0.5]
2020-10-08T11:16:07.3139069Z [cluster2] 07[IKE] scheduling rekeying in 13534s
2020-10-08T11:16:07.3139502Z [cluster2] 07[IKE] maximum IKE_SA lifetime 14974s
2020-10-08T11:16:07.3140091Z [cluster2] 07[CFG] selected proposal: ESP:AES_GCM_16_128/NO_EXT_SEQ
2020-10-08T11:16:07.3142620Z [cluster2] 07[IKE] CHILD_SA submariner-child-submariner-cable-cluster3-172-17-0-5{1} established with SPIs cec24d50_i cdb0c4b9_o and TS 10.242.0.0/16 100.92.0.0/16 172.17.0.8/32 === 10.243.0.0/16 100.93.0.0/16 172.17.0.5/32
2020-10-08T11:16:07.3144015Z [cluster2] I1008 11:15:54.881546       1 kubernetes.go:180] UpdateFunc in WatchClusters called
2020-10-08T11:16:07.3145425Z [cluster2] I1008 11:15:54.881575       1 datastoresyncer.go:312] In reconcileClusterCRD: &types.SubmarinerCluster{ID:"cluster3", Spec:v1.ClusterSpec{ClusterID:"cluster3", ColorCodes:[]string{"blue"}, ServiceCIDR:[]string{"100.93.0.0/16"}, ClusterCIDR:[]string{"10.243.0.0/16"}, GlobalCIDR:[]string{}}}
2020-10-08T11:16:07.3146813Z [cluster2] I1008 11:15:54.881565       1 kubernetes.go:264] UpdateFunc in WatchEndpoints called
2020-10-08T11:16:07.3147524Z [cluster2] I1008 11:15:54.881939       1 kubernetes.go:264] UpdateFunc in WatchEndpoints called
2020-10-08T11:16:07.3153050Z [submariner]$ [cluster2] kubectl get pods --selector=app=submariner-routeagent -n submariner-operator -o jsonpath={.items[*].metadata.name}
2020-10-08T11:16:07.3154750Z [submariner]$ [cluster2] kubectl get pods --selector=app=submariner-routeagent -n submariner-operator -o jsonpath={.items[*].metadata.name}
2020-10-08T11:16:07.3156637Z [submariner]$ [cluster2] command kubectl --context=cluster2 get pods --selector=app=submariner-routeagent -n submariner-operator -o jsonpath={.items[*].metadata.name}
2020-10-08T11:16:07.3158398Z [submariner]$ [cluster2] kubectl --context=cluster2 get pods --selector=app=submariner-routeagent -n submariner-operator -o jsonpath={.items[*].metadata.name}
2020-10-08T11:16:08.0784397Z [submariner]$ [cluster2] kubectl -n submariner-operator logs submariner-routeagent-mwn97
2020-10-08T11:16:08.0796203Z [submariner]$ [cluster2] kubectl -n submariner-operator logs submariner-routeagent-mwn97
2020-10-08T11:16:08.0814335Z [submariner]$ [cluster2] command kubectl --context=cluster2 -n submariner-operator logs submariner-routeagent-mwn97
2020-10-08T11:16:08.0831641Z [submariner]$ [cluster2] kubectl --context=cluster2 -n submariner-operator logs submariner-routeagent-mwn97
2020-10-08T11:16:08.8298995Z [cluster2] I1008 11:15:54.882074       1 datastoresyncer.go:387] In reconcileEndpointCRD: &types.SubmarinerEndpoint{Spec:v1.EndpointSpec{ClusterID:"cluster3", CableName:"submariner-cable-cluster3-172-17-0-5", Hostname:"cluster3-worker", Subnets:[]string{"100.93.0.0/16", "10.243.0.0/16"}, PrivateIP:"172.17.0.5", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
2020-10-08T11:16:08.8301420Z [cluster2] I1008 11:15:55.147842       1 datastoresyncer.go:360] Cluster "cluster3" matched what we received from datastore - not updating
2020-10-08T11:16:08.8302383Z [cluster2] I1008 11:15:55.147873       1 kubernetes.go:180] UpdateFunc in WatchClusters called
2020-10-08T11:16:08.8304041Z [cluster2] I1008 11:15:55.150384       1 datastoresyncer.go:435] Endpoint "cluster3-submariner-cable-cluster3-172-17-0-5" matched what we received from datastore - not updating
2020-10-08T11:16:08.8306359Z [cluster2] ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
2020-10-08T11:16:08.8307741Z [cluster2] +++++++++++++++++++++: Logs for Pod submariner-routeagent-mwn97 in namespace submariner-operator :++++++++++++++++++++++
2020-10-08T11:16:08.8308670Z [cluster2] + trap 'exit 1' SIGTERM SIGINT
2020-10-08T11:16:08.8309071Z [cluster2] + SUBMARINER_VERBOSITY=2
2020-10-08T11:16:08.8309608Z [cluster2] + '[' false == true ']'
2020-10-08T11:16:08.8310070Z [cluster2] + DEBUG=-v=2
2020-10-08T11:16:08.8310625Z [cluster2] + for f in iptables-save iptables
2020-10-08T11:16:08.8311281Z [cluster2] ++ find_iptables_on_host iptables-save
2020-10-08T11:16:08.8312263Z [cluster2] ++ chroot /host test -x /usr/sbin/iptables-save
2020-10-08T11:16:08.8312690Z [cluster2] ++ echo /usr/sbin
2020-10-08T11:16:08.8313164Z [cluster2] ++ return
2020-10-08T11:16:08.8313495Z [cluster2] + location=/usr/sbin
2020-10-08T11:16:08.8314075Z [cluster2] + '[' /usr/sbin '!=' unknown ']'
2020-10-08T11:16:08.8314816Z [cluster2] + echo 'iptables-save is present on the host at /usr/sbin/iptables-save'
2020-10-08T11:16:08.8316089Z [cluster2] + sed 's!@@PATH@@!/usr/sbin!' /usr/sbin/iptables-wrapper.in
2020-10-08T11:16:08.8317568Z [cluster2] iptables-save is present on the host at /usr/sbin/iptables-save
2020-10-08T11:16:08.8318315Z [cluster2] + for f in iptables-save iptables
2020-10-08T11:16:08.8318773Z [cluster2] ++ find_iptables_on_host iptables
2020-10-08T11:16:08.8319394Z [cluster2] ++ chroot /host test -x /usr/sbin/iptables
2020-10-08T11:16:08.8319878Z [cluster2] iptables is present on the host at /usr/sbin/iptables
2020-10-08T11:16:08.8320309Z [cluster2] ++ echo /usr/sbin
2020-10-08T11:16:08.8320618Z [cluster2] ++ return
2020-10-08T11:16:08.8321098Z [cluster2] + location=/usr/sbin
2020-10-08T11:16:08.8321678Z [cluster2] + '[' /usr/sbin '!=' unknown ']'
2020-10-08T11:16:08.8322347Z [cluster2] + echo 'iptables is present on the host at /usr/sbin/iptables'
2020-10-08T11:16:08.8323136Z [cluster2] + sed 's!@@PATH@@!/usr/sbin!' /usr/sbin/iptables-wrapper.in
2020-10-08T11:16:08.8324005Z [cluster2] + exec submariner-route-agent -v=2 -alsologtostderr
2020-10-08T11:16:08.8324907Z [cluster2] I1008 11:15:14.856192       1 main.go:47] Starting submariner-route-agent
2020-10-08T11:16:08.8326054Z [cluster2] W1008 11:15:14.856456       1 client_config.go:543] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
2020-10-08T11:16:08.8327129Z [cluster2] I1008 11:15:14.862162       1 route.go:195] Starting Route Controller. ClusterID: cluster2, localClusterCIDR: [10.242.0.0/16], localServiceCIDR: [100.92.0.0/16]
2020-10-08T11:16:08.8327996Z [cluster2] I1008 11:15:14.862201       1 route.go:199] Waiting for Endpoint informer caches to sync.
2020-10-08T11:16:08.8328903Z [cluster2] I1008 11:15:14.912253       1 route.go:751] Enqueueing sm-route-agent-pod event, ip: 172.17.0.7
2020-10-08T11:16:08.8329831Z [cluster2] I1008 11:15:14.912276       1 route.go:751] Enqueueing sm-route-agent-pod event, ip: 172.17.0.8
2020-10-08T11:16:08.8330444Z [cluster2] I1008 11:15:14.964435       1 driver.go:37] Interface "lo" has "127.0.0.1" address
2020-10-08T11:16:08.8330986Z [cluster2] I1008 11:15:14.964547       1 driver.go:37] Interface "weave" has "10.242.128.0" address
2020-10-08T11:16:08.8331620Z [cluster2] I1008 11:15:14.964555       1 driver.go:42] Found CNI Interface "weave" that has IP "10.242.128.0" from ClusterCIDR "10.242.0.0/16"
2020-10-08T11:16:08.8332408Z [cluster2] I1008 11:15:14.964662       1 driver.go:60] Successfully configured rp_filter to loose mode(2) on cniInterface "weave"
2020-10-08T11:16:08.8333502Z [cluster2] I1008 11:15:14.970049       1 iptables.go:22] Install/ensure SUBMARINER-POSTROUTING chain exists
2020-10-08T11:16:08.8334705Z [cluster2] I1008 11:15:15.013688       1 route.go:751] Enqueueing sm-route-agent-pod event, ip: 172.17.0.8
2020-10-08T11:16:08.8336014Z [cluster2] I1008 11:15:15.022876       1 iptables.go:28] Insert SUBMARINER-POSTROUTING rule that has rules for inter-cluster traffic
2020-10-08T11:16:08.8337121Z [cluster2] I1008 11:15:15.032169       1 iptables.go:34] Install/ensure SUBMARINER-INPUT chain exists
2020-10-08T11:16:08.8338330Z [cluster2] I1008 11:15:15.057142       1 iptables.go:45] Allow VxLAN incoming traffic in SUBMARINER-INPUT Chain
2020-10-08T11:16:08.8339433Z [cluster2] I1008 11:15:15.067974       1 iptables.go:53] Insert rule to allow traffic over vx-submariner interface in FORWARDing Chain
2020-10-08T11:16:08.8340683Z [cluster2] I1008 11:15:15.083003       1 iptables.go:65] Installing rule for host network to remote cluster communication: -s 240.0.0.0/8 -o vx-submariner -j SNAT --to 10.242.128.0
2020-10-08T11:16:08.8362699Z [cluster2] I1008 11:15:15.102757       1 route.go:257] In cluster2, podIP of submariner-route-agent[0] is 172.17.0.8
2020-10-08T11:16:08.8363973Z [cluster2] I1008 11:15:15.102779       1 route.go:257] In cluster2, podIP of submariner-route-agent[1] is 172.17.0.7
2020-10-08T11:16:08.8365199Z [cluster2] I1008 11:15:15.102784       1 route.go:271] Hostname is "cluster2-worker" and routeAgentNodeName is "cluster2-worker"
2020-10-08T11:16:08.8366408Z [cluster2] I1008 11:15:15.112562       1 driver.go:87] Successfully annotated node "cluster2-worker" with cniIfaceIP "10.242.128.0"
2020-10-08T11:16:08.8367105Z [cluster2] I1008 11:15:15.112586       1 route.go:288] Route agent workers started
2020-10-08T11:16:08.8368300Z [cluster2] I1008 11:15:15.115191       1 route.go:540] Processing submariner-route-agent pod "submariner-routeagent-xhztm" with host IP "172.17.0.7", pod IP "172.17.0.7"
2020-10-08T11:16:08.8370037Z [cluster2] I1008 11:15:15.115224       1 route.go:568] Successfully processed submariner-route-agent pod "submariner-routeagent-xhztm"
2020-10-08T11:16:08.8371653Z [cluster2] I1008 11:15:15.122162       1 route.go:540] Processing submariner-route-agent pod "submariner-routeagent-mwn97" with host IP "172.17.0.8", pod IP "172.17.0.8"
2020-10-08T11:16:08.8373187Z [cluster2] I1008 11:15:15.122250       1 route.go:568] Successfully processed submariner-route-agent pod "submariner-routeagent-mwn97"
2020-10-08T11:16:08.8377619Z [cluster2] I1008 11:15:24.876155       1 route.go:736] Enqueueing endpoint for route controller {"metadata":{"name":"cluster2-submariner-cable-cluster2-172-17-0-8","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster2-submariner-cable-cluster2-172-17-0-8","uid":"0ca8b389-f4b4-453d-a2cb-7b856a53e2e6","resourceVersion":"1629","generation":1,"creationTimestamp":"2020-10-08T11:15:24Z"},"spec":{"cluster_id":"cluster2","cable_name":"submariner-cable-cluster2-172-17-0-8","hostname":"cluster2-worker","subnets":["100.92.0.0/16","10.242.0.0/16"],"private_ip":"172.17.0.8","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
2020-10-08T11:16:08.8381830Z [cluster2] I1008 11:15:24.894701       1 route.go:634] Processing local Endpoint cluster2-submariner-cable-cluster2-172-17-0-8 with IP 172.17.0.8, Host cluster2-worker for this cluster
2020-10-08T11:16:08.8383101Z [cluster2] I1008 11:15:24.894925       1 route.go:851] Processing route {Ifindex: 16 Dst: <nil> Src: <nil> Gw: 172.17.0.1 Flags: [] Table: 254}
2020-10-08T11:16:08.8383681Z [cluster2] I1008 11:15:24.894969       1 route.go:854] Found nil gw or dst
2020-10-08T11:16:08.8384249Z [cluster2] I1008 11:15:24.894973       1 route.go:851] Processing route {Ifindex: 4 Dst: 10.242.0.0/16 Src: 10.242.128.0 Gw: <nil> Flags: [] Table: 254}
2020-10-08T11:16:08.8384810Z [cluster2] I1008 11:15:24.894983       1 route.go:854] Found nil gw or dst
2020-10-08T11:16:08.8385372Z [cluster2] I1008 11:15:24.894986       1 route.go:851] Processing route {Ifindex: 16 Dst: 172.17.0.0/16 Src: 172.17.0.8 Gw: <nil> Flags: [] Table: 254}
2020-10-08T11:16:08.8385932Z [cluster2] I1008 11:15:24.894993       1 route.go:854] Found nil gw or dst
2020-10-08T11:16:08.8386476Z [cluster2] I1008 11:15:24.895001       1 route.go:645] This route agent is running on the active gateway node
2020-10-08T11:16:08.8387433Z [cluster2] I1008 11:15:24.895009       1 route.go:653] Creating the vxlan interface: vx-submariner on the gateway node
2020-10-08T11:16:08.8388326Z [cluster2] I1008 11:15:24.895605       1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.7 00:00:00:00:00:00
2020-10-08T11:16:08.8388955Z [cluster2] I1008 11:15:24.895799       1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.8 00:00:00:00:00:00
2020-10-08T11:16:08.8389955Z [cluster2] I1008 11:15:24.895875       1 route.go:478] Successfully configured rp_filter to loose mode(2) on vx-submariner
2020-10-08T11:16:08.8394390Z [cluster2] I1008 11:15:44.897402       1 route.go:736] Enqueueing endpoint for route controller {"metadata":{"name":"cluster2-submariner-cable-cluster2-172-17-0-8","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster2-submariner-cable-cluster2-172-17-0-8","uid":"0ca8b389-f4b4-453d-a2cb-7b856a53e2e6","resourceVersion":"1629","generation":1,"creationTimestamp":"2020-10-08T11:15:24Z"},"spec":{"cluster_id":"cluster2","cable_name":"submariner-cable-cluster2-172-17-0-8","hostname":"cluster2-worker","subnets":["100.92.0.0/16","10.242.0.0/16"],"private_ip":"172.17.0.8","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
2020-10-08T11:16:08.8530306Z [submariner]$ [cluster2] kubectl -n submariner-operator logs submariner-routeagent-xhztm
2020-10-08T11:16:08.8543716Z [submariner]$ [cluster2] kubectl -n submariner-operator logs submariner-routeagent-xhztm
2020-10-08T11:16:08.8560327Z [submariner]$ [cluster2] command kubectl --context=cluster2 -n submariner-operator logs submariner-routeagent-xhztm
2020-10-08T11:16:08.8579284Z [submariner]$ [cluster2] kubectl --context=cluster2 -n submariner-operator logs submariner-routeagent-xhztm
2020-10-08T11:16:09.6119880Z [cluster2] I1008 11:15:44.969457       1 route.go:634] Processing local Endpoint cluster2-submariner-cable-cluster2-172-17-0-8 with IP 172.17.0.8, Host cluster2-worker for this cluster
2020-10-08T11:16:09.6122526Z [cluster2] I1008 11:15:44.969687       1 route.go:851] Processing route {Ifindex: 11 Dst: 240.0.0.0/8 Src: 240.17.0.8 Gw: <nil> Flags: [] Table: 254}
2020-10-08T11:16:09.6123118Z [cluster2] I1008 11:15:44.969711       1 route.go:854] Found nil gw or dst
2020-10-08T11:16:09.6123658Z [cluster2] I1008 11:15:44.969719       1 route.go:645] This route agent is running on the active gateway node
2020-10-08T11:16:09.6124695Z [cluster2] I1008 11:15:44.969727       1 route.go:653] Creating the vxlan interface: vx-submariner on the gateway node
2020-10-08T11:16:09.6125446Z [cluster2] I1008 11:15:44.969886       1 vxlan.go:61] VxLAN interface already exists with same configuration.
2020-10-08T11:16:09.6126132Z [cluster2] I1008 11:15:44.969940       1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.8 00:00:00:00:00:00
2020-10-08T11:16:09.6126960Z [cluster2] I1008 11:15:44.969970       1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.7 00:00:00:00:00:00
2020-10-08T11:16:09.6127958Z [cluster2] I1008 11:15:44.970053       1 route.go:478] Successfully configured rp_filter to loose mode(2) on vx-submariner
2020-10-08T11:16:09.6132119Z [cluster2] I1008 11:15:48.720542       1 route.go:736] Enqueueing endpoint for route controller {"metadata":{"name":"cluster3-submariner-cable-cluster3-172-17-0-5","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster3-submariner-cable-cluster3-172-17-0-5","uid":"9871d64e-98a5-4a6b-9983-faf9b0a1d129","resourceVersion":"1713","generation":1,"creationTimestamp":"2020-10-08T11:15:48Z"},"spec":{"cluster_id":"cluster3","cable_name":"submariner-cable-cluster3-172-17-0-5","hostname":"cluster3-worker","subnets":["100.93.0.0/16","10.243.0.0/16"],"private_ip":"172.17.0.5","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
2020-10-08T11:16:09.6136522Z [cluster2] I1008 11:15:48.747047       1 route.go:620] Updating iptable rules for remote cluster's Endpoint "cluster3-submariner-cable-cluster3-172-17-0-5" with subnets [100.93.0.0/16 10.243.0.0/16]
2020-10-08T11:16:09.6138337Z [cluster2] I1008 11:15:48.793537       1 iptables.go:83] Installing iptables rule for outgoing traffic: -s 10.242.0.0/16 -d 100.93.0.0/16 -j ACCEPT
2020-10-08T11:16:09.6139385Z [cluster2] I1008 11:15:48.856542       1 iptables.go:91] Installing iptables rule for incoming traffic: -s 100.93.0.0/16 -d 10.242.0.0/16 -j ACCEPT
2020-10-08T11:16:09.6140417Z [cluster2] I1008 11:15:48.913432       1 iptables.go:83] Installing iptables rule for outgoing traffic: -s 10.242.0.0/16 -d 10.243.0.0/16 -j ACCEPT
2020-10-08T11:16:09.6141452Z [cluster2] I1008 11:15:49.015375       1 iptables.go:91] Installing iptables rule for incoming traffic: -s 10.243.0.0/16 -d 10.242.0.0/16 -j ACCEPT
2020-10-08T11:16:09.6142766Z [cluster2] +++++++++++++++++++++: Logs for Pod submariner-routeagent-xhztm in namespace submariner-operator :++++++++++++++++++++++
2020-10-08T11:16:09.6143737Z [cluster2] + trap 'exit 1' SIGTERM SIGINT
2020-10-08T11:16:09.6144156Z [cluster2] + SUBMARINER_VERBOSITY=2
2020-10-08T11:16:09.6144672Z [cluster2] + '[' false == true ']'
2020-10-08T11:16:09.6145145Z [cluster2] + DEBUG=-v=2
2020-10-08T11:16:09.6145700Z [cluster2] + for f in iptables-save iptables
2020-10-08T11:16:09.6146354Z [cluster2] ++ find_iptables_on_host iptables-save
2020-10-08T11:16:09.6147236Z [cluster2] ++ chroot /host test -x /usr/sbin/iptables-save
2020-10-08T11:16:09.6147669Z [cluster2] ++ echo /usr/sbin
2020-10-08T11:16:09.6148299Z [cluster2] ++ return
2020-10-08T11:16:09.6148647Z [cluster2] + location=/usr/sbin
2020-10-08T11:16:09.6149236Z [cluster2] + '[' /usr/sbin '!=' unknown ']'
2020-10-08T11:16:09.6149974Z [cluster2] + echo 'iptables-save is present on the host at /usr/sbin/iptables-save'
2020-10-08T11:16:09.6150825Z [cluster2] + sed 's!@@PATH@@!/usr/sbin!' /usr/sbin/iptables-wrapper.in
2020-10-08T11:16:09.6151665Z [cluster2] iptables-save is present on the host at /usr/sbin/iptables-save
2020-10-08T11:16:09.6152402Z [cluster2] + for f in iptables-save iptables
2020-10-08T11:16:09.6152856Z [cluster2] ++ find_iptables_on_host iptables
2020-10-08T11:16:09.6153482Z [cluster2] ++ chroot /host test -x /usr/sbin/iptables
2020-10-08T11:16:09.6153873Z [cluster2] ++ echo /usr/sbin
2020-10-08T11:16:09.6154166Z [cluster2] ++ return
2020-10-08T11:16:09.6154496Z [cluster2] + location=/usr/sbin
2020-10-08T11:16:09.6154949Z [cluster2] iptables is present on the host at /usr/sbin/iptables
2020-10-08T11:16:09.6155569Z [cluster2] + '[' /usr/sbin '!=' unknown ']'
2020-10-08T11:16:09.6156226Z [cluster2] + echo 'iptables is present on the host at /usr/sbin/iptables'
2020-10-08T11:16:09.6158108Z [cluster2] + sed 's!@@PATH@@!/usr/sbin!' /usr/sbin/iptables-wrapper.in
2020-10-08T11:16:09.6158974Z [cluster2] + exec submariner-route-agent -v=2 -alsologtostderr
2020-10-08T11:16:09.6165083Z [cluster2] I1008 11:15:14.672841       1 main.go:47] Starting submariner-route-agent
2020-10-08T11:16:09.6166435Z [cluster2] W1008 11:15:14.673138       1 client_config.go:543] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
2020-10-08T11:16:09.6167709Z [cluster2] I1008 11:15:14.675965       1 route.go:195] Starting Route Controller. ClusterID: cluster2, localClusterCIDR: [10.242.0.0/16], localServiceCIDR: [100.92.0.0/16]
2020-10-08T11:16:09.6168578Z [cluster2] I1008 11:15:14.675994       1 route.go:199] Waiting for Endpoint informer caches to sync.
2020-10-08T11:16:09.6169508Z [cluster2] I1008 11:15:14.723061       1 route.go:751] Enqueueing sm-route-agent-pod event, ip: 172.17.0.7
2020-10-08T11:16:09.6170435Z [cluster2] I1008 11:15:14.723436       1 route.go:751] Enqueueing sm-route-agent-pod event, ip: 172.17.0.8
2020-10-08T11:16:09.6171049Z [cluster2] I1008 11:15:14.776872       1 driver.go:37] Interface "lo" has "127.0.0.1" address
2020-10-08T11:16:09.6171592Z [cluster2] I1008 11:15:14.777177       1 driver.go:37] Interface "weave" has "10.242.0.1" address
2020-10-08T11:16:09.6172230Z [cluster2] I1008 11:15:14.777300       1 driver.go:42] Found CNI Interface "weave" that has IP "10.242.0.1" from ClusterCIDR "10.242.0.0/16"
2020-10-08T11:16:09.6173200Z [cluster2] I1008 11:15:14.777503       1 driver.go:60] Successfully configured rp_filter to loose mode(2) on cniInterface "weave"
2020-10-08T11:16:09.6174929Z [cluster2] I1008 11:15:14.783567       1 iptables.go:22] Install/ensure SUBMARINER-POSTROUTING chain exists
2020-10-08T11:16:09.6176518Z [cluster2] I1008 11:15:14.812023       1 iptables.go:28] Insert SUBMARINER-POSTROUTING rule that has rules for inter-cluster traffic
2020-10-08T11:16:09.6177682Z [cluster2] I1008 11:15:14.821550       1 iptables.go:34] Install/ensure SUBMARINER-INPUT chain exists
2020-10-08T11:16:09.6178678Z [cluster2] I1008 11:15:14.848171       1 iptables.go:45] Allow VxLAN incoming traffic in SUBMARINER-INPUT Chain
2020-10-08T11:16:09.6179967Z [cluster2] I1008 11:15:14.866594       1 iptables.go:53] Insert rule to allow traffic over vx-submariner interface in FORWARDing Chain
2020-10-08T11:16:09.6181281Z [cluster2] I1008 11:15:14.883567       1 iptables.go:65] Installing rule for host network to remote cluster communication: -s 240.0.0.0/8 -o vx-submariner -j SNAT --to 10.242.0.1
2020-10-08T11:16:09.6182437Z [cluster2] I1008 11:15:14.904029       1 route.go:257] In cluster2, podIP of submariner-route-agent[0] is 172.17.0.8
2020-10-08T11:16:09.6183475Z [cluster2] I1008 11:15:14.904172       1 route.go:257] In cluster2, podIP of submariner-route-agent[1] is 172.17.0.7
2020-10-08T11:16:09.6184619Z [cluster2] I1008 11:15:14.904252       1 route.go:271] Hostname is "cluster2-worker2" and routeAgentNodeName is "cluster2-worker2"
2020-10-08T11:16:09.6185823Z [cluster2] I1008 11:15:14.922143       1 driver.go:87] Successfully annotated node "cluster2-worker2" with cniIfaceIP "10.242.0.1"
2020-10-08T11:16:09.6186525Z [cluster2] I1008 11:15:14.922264       1 route.go:288] Route agent workers started
2020-10-08T11:16:09.6187718Z [cluster2] I1008 11:15:14.932154       1 route.go:540] Processing submariner-route-agent pod "submariner-routeagent-xhztm" with host IP "172.17.0.7", pod IP "172.17.0.7"
2020-10-08T11:16:09.6189248Z [cluster2] I1008 11:15:14.932183       1 route.go:568] Successfully processed submariner-route-agent pod "submariner-routeagent-xhztm"
2020-10-08T11:16:09.6190810Z [cluster2] I1008 11:15:14.935400       1 route.go:540] Processing submariner-route-agent pod "submariner-routeagent-mwn97" with host IP "172.17.0.8", pod IP "172.17.0.8"
2020-10-08T11:16:09.6192332Z [cluster2] I1008 11:15:14.935423       1 route.go:568] Successfully processed submariner-route-agent pod "submariner-routeagent-mwn97"
2020-10-08T11:16:09.6197065Z [cluster2] I1008 11:15:24.874895       1 route.go:736] Enqueueing endpoint for route controller {"metadata":{"name":"cluster2-submariner-cable-cluster2-172-17-0-8","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster2-submariner-cable-cluster2-172-17-0-8","uid":"0ca8b389-f4b4-453d-a2cb-7b856a53e2e6","resourceVersion":"1629","generation":1,"creationTimestamp":"2020-10-08T11:15:24Z"},"spec":{"cluster_id":"cluster2","cable_name":"submariner-cable-cluster2-172-17-0-8","hostname":"cluster2-worker","subnets":["100.92.0.0/16","10.242.0.0/16"],"private_ip":"172.17.0.8","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
2020-10-08T11:16:09.6201308Z [cluster2] I1008 11:15:24.886543       1 route.go:634] Processing local Endpoint cluster2-submariner-cable-cluster2-172-17-0-8 with IP 172.17.0.8, Host cluster2-worker for this cluster
2020-10-08T11:16:09.6202807Z [cluster2] I1008 11:15:24.886568       1 route.go:677] This route agent is running on a non-gateway/non-active node
2020-10-08T11:16:09.6203807Z [cluster2] I1008 11:15:24.886578       1 route.go:689] Creating the vxlan interface vx-submariner with gateway node IP 172.17.0.8
2020-10-08T11:16:09.6204446Z [cluster2] I1008 11:15:24.891224       1 route.go:866] Reconciling routes to gw: 240.17.0.8
2020-10-08T11:16:09.6398133Z [submariner]$ [cluster2] subctl show all
2020-10-08T11:16:09.9001952Z Error obtaining the Submariner resource: the server could not find the requested resource (get submariners.submariner.io submariner)
2020-10-08T11:16:09.9003501Z
2020-10-08T11:16:09.9004682Z subctl version: v0.7.0-pre1-30-g7a9e04d
2020-10-08T11:16:09.9005115Z
2020-10-08T11:16:09.9042073Z [cluster2] I1008 11:15:24.891363       1 route.go:882] Processing route {Ifindex: 13 Dst: 240.0.0.0/8 Src: 240.17.0.7 Gw: <nil> Flags: [] Table: 254}
2020-10-08T11:16:09.9043013Z [cluster2] I1008 11:15:24.891392       1 route.go:885] Found nil gw or dst
2020-10-08T11:16:09.9045239Z [cluster2] I1008 11:15:24.891461       1 route.go:715] Successfully processed local Endpoint "cluster2-submariner-cable-cluster2-172-17-0-8"
2020-10-08T11:16:09.9050507Z [cluster2] I1008 11:15:44.723216       1 route.go:736] Enqueueing endpoint for route controller {"metadata":{"name":"cluster2-submariner-cable-cluster2-172-17-0-8","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster2-submariner-cable-cluster2-172-17-0-8","uid":"0ca8b389-f4b4-453d-a2cb-7b856a53e2e6","resourceVersion":"1629","generation":1,"creationTimestamp":"2020-10-08T11:15:24Z"},"spec":{"cluster_id":"cluster2","cable_name":"submariner-cable-cluster2-172-17-0-8","hostname":"cluster2-worker","subnets":["100.92.0.0/16","10.242.0.0/16"],"private_ip":"172.17.0.8","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
2020-10-08T11:16:09.9057419Z [cluster2] I1008 11:15:44.962550       1 route.go:634] Processing local Endpoint cluster2-submariner-cable-cluster2-172-17-0-8 with IP 172.17.0.8, Host cluster2-worker for this cluster
2020-10-08T11:16:09.9059060Z [cluster2] I1008 11:15:44.962574       1 route.go:677] This route agent is running on a non-gateway/non-active node
2020-10-08T11:16:09.9060232Z [cluster2] I1008 11:15:44.962583       1 route.go:689] Creating the vxlan interface vx-submariner with gateway node IP 172.17.0.8
2020-10-08T11:16:09.9061007Z [cluster2] I1008 11:15:44.962768       1 vxlan.go:61] VxLAN interface already exists with same configuration.
2020-10-08T11:16:09.9061630Z [cluster2] I1008 11:15:44.966983       1 route.go:866] Reconciling routes to gw: 240.17.0.8
2020-10-08T11:16:09.9062247Z [cluster2] I1008 11:15:44.967198       1 route.go:882] Processing route {Ifindex: 13 Dst: 240.0.0.0/8 Src: 240.17.0.7 Gw: <nil> Flags: [] Table: 254}
2020-10-08T11:16:09.9062811Z [cluster2] I1008 11:15:44.967219       1 route.go:885] Found nil gw or dst
2020-10-08T11:16:09.9064277Z [cluster2] I1008 11:15:44.967298       1 route.go:715] Successfully processed local Endpoint "cluster2-submariner-cable-cluster2-172-17-0-8"
2020-10-08T11:16:09.9068856Z [cluster2] I1008 11:15:48.725007       1 route.go:736] Enqueueing endpoint for route controller {"metadata":{"name":"cluster3-submariner-cable-cluster3-172-17-0-5","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster3-submariner-cable-cluster3-172-17-0-5","uid":"9871d64e-98a5-4a6b-9983-faf9b0a1d129","resourceVersion":"1713","generation":1,"creationTimestamp":"2020-10-08T11:15:48Z"},"spec":{"cluster_id":"cluster3","cable_name":"submariner-cable-cluster3-172-17-0-5","hostname":"cluster3-worker","subnets":["100.93.0.0/16","10.243.0.0/16"],"private_ip":"172.17.0.5","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
2020-10-08T11:16:09.9073419Z [cluster2] I1008 11:15:48.754784       1 route.go:620] Updating iptable rules for remote cluster's Endpoint "cluster3-submariner-cable-cluster3-172-17-0-5" with subnets [100.93.0.0/16 10.243.0.0/16]
2020-10-08T11:16:09.9075669Z [cluster2] I1008 11:15:48.761878       1 iptables.go:83] Installing iptables rule for outgoing traffic: -s 10.242.0.0/16 -d 100.93.0.0/16 -j ACCEPT
2020-10-08T11:16:09.9077495Z [cluster2] I1008 11:15:48.805865       1 iptables.go:91] Installing iptables rule for incoming traffic: -s 100.93.0.0/16 -d 10.242.0.0/16 -j ACCEPT
2020-10-08T11:16:09.9078630Z [cluster2] I1008 11:15:48.871092       1 iptables.go:83] Installing iptables rule for outgoing traffic: -s 10.242.0.0/16 -d 10.243.0.0/16 -j ACCEPT
2020-10-08T11:16:09.9079916Z [cluster2] I1008 11:15:48.956887       1 iptables.go:91] Installing iptables rule for incoming traffic: -s 10.243.0.0/16 -d 10.242.0.0/16 -j ACCEPT
2020-10-08T11:16:09.9080458Z [cluster2] ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
2020-10-08T11:16:09.9080732Z [cluster2]
2020-10-08T11:16:09.9081137Z [cluster2] Showing information for cluster "cluster3":
2020-10-08T11:16:09.9081635Z [cluster2]     Discovered network details:
2020-10-08T11:16:09.9082247Z [cluster2]         Network plugin:  weave-net
2020-10-08T11:16:09.9082642Z [cluster2]         Service CIDRs:   [100.93.0.0/16]
2020-10-08T11:16:09.9083004Z [cluster2]         Cluster CIDRs:   [10.243.0.0/16]
2020-10-08T11:16:09.9083312Z [cluster2]
2020-10-08T11:16:09.9083770Z [cluster2] ===================== END Post mortem cluster2 =====================
2020-10-08T11:16:09.9085304Z [submariner]$ [cluster3] post_analyze
2020-10-08T11:16:09.9087652Z [submariner]$ [cluster3] post_analyze
2020-10-08T11:16:09.9182724Z [submariner]$ [cluster3] kubectl get all --all-namespaces
2020-10-08T11:16:09.9194472Z [submariner]$ [cluster3] sed s/^/[cluster3] /
2020-10-08T11:16:09.9195223Z [submariner]$ [cluster3] kubectl get all --all-namespaces
2020-10-08T11:16:09.9200033Z [submariner]$ [cluster3] command kubectl --context=cluster3 get all --all-namespaces
2020-10-08T11:16:09.9211791Z [submariner]$ [cluster3] kubectl --context=cluster3 get all --all-namespaces
2020-10-08T11:16:10.8223966Z
2020-10-08T11:16:10.8224322Z
2020-10-08T11:16:10.8228089Z
2020-10-08T11:16:10.8237323Z
2020-10-08T11:16:10.8242143Z
2020-10-08T11:16:10.8249462Z
2020-10-08T11:16:10.8249675Z
2020-10-08T11:16:10.8249815Z
2020-10-08T11:16:10.8250178Z [cluster3] ======================= Post mortem cluster3 =======================
2020-10-08T11:16:10.8250685Z [cluster3] NAMESPACE             NAME                                                 READY   STATUS    RESTARTS   AGE
2020-10-08T11:16:10.8254109Z [cluster3] kube-system           pod/coredns-6955765f44-mjnkz                         1/1     Running   0          6m40s
2020-10-08T11:16:10.8255104Z [cluster3] kube-system           pod/coredns-6955765f44-q29sm                         1/1     Running   0          6m40s
2020-10-08T11:16:10.8256180Z [cluster3] kube-system           pod/etcd-cluster3-control-plane                      1/1     Running   0          6m53s
2020-10-08T11:16:10.8257454Z [cluster3] kube-system           pod/kube-apiserver-cluster3-control-plane            1/1     Running   1          6m53s
2020-10-08T11:16:10.8259007Z [cluster3] kube-system           pod/kube-controller-manager-cluster3-control-plane   1/1     Running   0          6m53s
2020-10-08T11:16:10.8260339Z [cluster3] kube-system           pod/kube-proxy-2xjmm                                 1/1     Running   0          6m17s
2020-10-08T11:16:10.8261218Z [cluster3] kube-system           pod/kube-proxy-slh5v                                 1/1     Running   0          6m40s
2020-10-08T11:16:10.8262119Z [cluster3] kube-system           pod/kube-proxy-tgpm6                                 1/1     Running   0          6m16s
2020-10-08T11:16:10.8263259Z [cluster3] kube-system           pod/kube-scheduler-cluster3-control-plane            1/1     Running   0          6m53s
2020-10-08T11:16:10.8264528Z [cluster3] kube-system           pod/tiller-deploy-8488d98b4c-dzqvt                   1/1     Running   0          102s
2020-10-08T11:16:10.8266275Z [cluster3] kube-system           pod/weave-net-mj2rh                                  2/2     Running   0          5m56s
2020-10-08T11:16:10.8267208Z [cluster3] kube-system           pod/weave-net-qgtj8                                  2/2     Running   0          5m56s
2020-10-08T11:16:10.8268079Z [cluster3] kube-system           pod/weave-net-zsslf                                  2/2     Running   0          5m56s
2020-10-08T11:16:10.8269257Z [cluster3] local-path-storage    pod/local-path-provisioner-7745554f7f-bf5bt          1/1     Running   0          6m40s
2020-10-08T11:16:10.8270971Z [cluster3] submariner-operator   pod/submariner-gateway-pkb8n                         1/1     Running   0          41s
2020-10-08T11:16:10.8272177Z [cluster3] submariner-operator   pod/submariner-routeagent-596j7                      1/1     Running   0          41s
2020-10-08T11:16:10.8273413Z [cluster3] submariner-operator   pod/submariner-routeagent-jll6h                      1/1     Running   0          41s
2020-10-08T11:16:10.8274477Z [cluster3] NAMESPACE     NAME                    TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                  AGE
2020-10-08T11:16:10.8275108Z [cluster3] default       service/kubernetes      ClusterIP   100.93.0.1       <none>        443/TCP                  6m57s
2020-10-08T11:16:10.8276152Z [cluster3] kube-system   service/kube-dns        ClusterIP   100.93.0.10      <none>        53/UDP,53/TCP,9153/TCP   6m55s
2020-10-08T11:16:10.8278068Z [cluster3] kube-system   service/tiller-deploy   ClusterIP   100.93.244.133   <none>        44134/TCP                102s
2020-10-08T11:16:10.8279099Z [cluster3] NAMESPACE             NAME                                   DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR                 AGE
2020-10-08T11:16:10.8280216Z [cluster3] kube-system           daemonset.apps/kube-proxy              3         3         3       3            3           beta.kubernetes.io/os=linux   6m55s
2020-10-08T11:16:10.8281262Z [cluster3] kube-system           daemonset.apps/weave-net               3         3         3       3            3           <none>                        5m56s
2020-10-08T11:16:10.8282410Z [cluster3] submariner-operator   daemonset.apps/submariner-gateway      1         1         1       1            1           submariner.io/gateway=true    41s
2020-10-08T11:16:10.8283717Z [cluster3] submariner-operator   daemonset.apps/submariner-routeagent   2         2         2       2            2           <none>                        41s
2020-10-08T11:16:10.8284734Z [cluster3] NAMESPACE            NAME                                     READY   UP-TO-DATE   AVAILABLE   AGE
2020-10-08T11:16:10.8285738Z [cluster3] kube-system          deployment.apps/coredns                  2/2     2            2           6m55s
2020-10-08T11:16:10.8286647Z [cluster3] kube-system          deployment.apps/tiller-deploy            1/1     1            1           102s
2020-10-08T11:16:10.8287675Z [cluster3] local-path-storage   deployment.apps/local-path-provisioner   1/1     1            1           6m45s
2020-10-08T11:16:10.8288918Z
2020-10-08T11:16:10.8398373Z [submariner]$ [cluster3] kubectl get pods -A
2020-10-08T11:16:10.8410008Z [submariner]$ [cluster3] tail -n +2
2020-10-08T11:16:10.8419393Z [submariner]$ [cluster3] kubectl get pods -A
2020-10-08T11:16:10.8438202Z [submariner]$ [cluster3] command kubectl --context=cluster3 get pods -A
2020-10-08T11:16:10.8448891Z [submariner]$ [cluster3] grep -v Running
2020-10-08T11:16:10.8464600Z [submariner]$ [cluster3] sed s/  */;/g
2020-10-08T11:16:10.8465312Z [submariner]$ [cluster3] kubectl --context=cluster3 get pods -A
2020-10-08T11:16:11.6713200Z [submariner]$ [cluster3] namespace=kube-system
2020-10-08T11:16:11.6728307Z [submariner]$ [cluster3] kubectl get pods --selector=k8s-app=kube-proxy -n kube-system -o jsonpath={.items[*].metadata.name}
2020-10-08T11:16:11.6740227Z [submariner]$ [cluster3] kubectl get pods --selector=k8s-app=kube-proxy -n kube-system -o jsonpath={.items[*].metadata.name}
2020-10-08T11:16:11.6775977Z [submariner]$ [cluster3] command kubectl --context=cluster3 get pods --selector=k8s-app=kube-proxy -n kube-system -o jsonpath={.items[*].metadata.name}
2020-10-08T11:16:11.6789443Z [submariner]$ [cluster3] kubectl --context=cluster3 get pods --selector=k8s-app=kube-proxy -n kube-system -o jsonpath={.items[*].metadata.name}
2020-10-08T11:16:12.4196293Z [submariner]$ [cluster3] kubectl -n kube-system logs kube-proxy-2xjmm
2020-10-08T11:16:12.4209991Z [submariner]$ [cluster3] kubectl -n kube-system logs kube-proxy-2xjmm
2020-10-08T11:16:12.4221976Z [submariner]$ [cluster3] command kubectl --context=cluster3 -n kube-system logs kube-proxy-2xjmm
2020-10-08T11:16:12.4231345Z [submariner]$ [cluster3] kubectl --context=cluster3 -n kube-system logs kube-proxy-2xjmm
2020-10-08T11:16:13.2257583Z [submariner]$ [cluster3] kubectl -n kube-system logs kube-proxy-slh5v
2020-10-08T11:16:13.2279631Z [submariner]$ [cluster3] kubectl -n kube-system logs kube-proxy-slh5v
2020-10-08T11:16:13.2337601Z [submariner]$ [cluster3] command kubectl --context=cluster3 -n kube-system logs kube-proxy-slh5v
2020-10-08T11:16:13.2339330Z [submariner]$ [cluster3] kubectl --context=cluster3 -n kube-system logs kube-proxy-slh5v
2020-10-08T11:16:13.9784622Z [submariner]$ [cluster3] kubectl -n kube-system logs kube-proxy-tgpm6
2020-10-08T11:16:13.9795198Z [submariner]$ [cluster3] kubectl -n kube-system logs kube-proxy-tgpm6
2020-10-08T11:16:13.9806712Z [submariner]$ [cluster3] command kubectl --context=cluster3 -n kube-system logs kube-proxy-tgpm6
2020-10-08T11:16:13.9817961Z [submariner]$ [cluster3] kubectl --context=cluster3 -n kube-system logs kube-proxy-tgpm6
2020-10-08T11:16:14.7249203Z [cluster3] NAMESPACE            NAME                                                DESIRED   CURRENT   READY   AGE
2020-10-08T11:16:14.7251702Z [cluster3] kube-system          replicaset.apps/coredns-6955765f44                  2         2         2       6m40s
2020-10-08T11:16:14.7252760Z [cluster3] kube-system          replicaset.apps/tiller-deploy-8488d98b4c            1         1         1       102s
2020-10-08T11:16:14.7254050Z [cluster3] local-path-storage   replicaset.apps/local-path-provisioner-7745554f7f   1         1         1       6m40s
2020-10-08T11:16:14.7255223Z [cluster3] +++++++++++++++++++++: Logs for Pod kube-proxy-2xjmm in namespace kube-system :++++++++++++++++++++++
2020-10-08T11:16:14.7255924Z [cluster3] W1008 11:10:12.131220       1 server_others.go:323] Unknown proxy mode "", assuming iptables proxy
2020-10-08T11:16:14.7256551Z [cluster3] I1008 11:10:12.163129       1 node.go:135] Successfully retrieved node IP: 172.17.0.5
2020-10-08T11:16:14.7257121Z [cluster3] I1008 11:10:12.163169       1 server_others.go:145] Using iptables Proxier.
2020-10-08T11:16:14.7257637Z [cluster3] I1008 11:10:12.163664       1 server.go:571] Version: v1.17.0
2020-10-08T11:16:14.7258129Z [cluster3] I1008 11:10:12.164239       1 conntrack.go:52] Setting nf_conntrack_max to 131072
2020-10-08T11:16:14.7259102Z [cluster3] I1008 11:10:12.164430       1 conntrack.go:100] Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_established' to 86400
2020-10-08T11:16:14.7260149Z [cluster3] I1008 11:10:12.164596       1 conntrack.go:100] Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_close_wait' to 3600
2020-10-08T11:16:14.7260845Z [cluster3] I1008 11:10:12.166793       1 config.go:131] Starting endpoints config controller
2020-10-08T11:16:14.7261509Z [cluster3] I1008 11:10:12.166916       1 shared_informer.go:197] Waiting for caches to sync for endpoints config
2020-10-08T11:16:14.7262151Z [cluster3] I1008 11:10:12.181170       1 config.go:313] Starting service config controller
2020-10-08T11:16:14.7262787Z [cluster3] I1008 11:10:12.181204       1 shared_informer.go:197] Waiting for caches to sync for service config
2020-10-08T11:16:14.7263454Z [cluster3] I1008 11:10:12.267275       1 shared_informer.go:204] Caches are synced for endpoints config
2020-10-08T11:16:14.7264105Z [cluster3] I1008 11:10:12.281373       1 shared_informer.go:204] Caches are synced for service config
2020-10-08T11:16:14.7265011Z [cluster3] +++++++++++++++++++++: Logs for Pod kube-proxy-slh5v in namespace kube-system :++++++++++++++++++++++
2020-10-08T11:16:14.7265694Z [cluster3] W1008 11:09:38.393048       1 server_others.go:323] Unknown proxy mode "", assuming iptables proxy
2020-10-08T11:16:14.7266571Z [cluster3] I1008 11:09:38.428519       1 node.go:135] Successfully retrieved node IP: 172.17.0.9
2020-10-08T11:16:14.7267119Z [cluster3] I1008 11:09:38.428560       1 server_others.go:145] Using iptables Proxier.
2020-10-08T11:16:14.7267622Z [cluster3] I1008 11:09:38.429060       1 server.go:571] Version: v1.17.0
2020-10-08T11:16:14.7268128Z [cluster3] I1008 11:09:38.429425       1 conntrack.go:52] Setting nf_conntrack_max to 131072
2020-10-08T11:16:14.7269143Z [cluster3] I1008 11:09:38.429604       1 conntrack.go:100] Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_established' to 86400
2020-10-08T11:16:14.7270189Z [cluster3] I1008 11:09:38.429658       1 conntrack.go:100] Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_close_wait' to 3600
2020-10-08T11:16:14.7271009Z [cluster3] I1008 11:09:38.500042       1 config.go:313] Starting service config controller
2020-10-08T11:16:14.7271668Z [cluster3] I1008 11:09:38.500065       1 shared_informer.go:197] Waiting for caches to sync for service config
2020-10-08T11:16:14.7272314Z [cluster3] I1008 11:09:38.500941       1 config.go:131] Starting endpoints config controller
2020-10-08T11:16:14.7272971Z [cluster3] I1008 11:09:38.500956       1 shared_informer.go:197] Waiting for caches to sync for endpoints config
2020-10-08T11:16:14.7273636Z [cluster3] I1008 11:09:38.600324       1 shared_informer.go:204] Caches are synced for service config
2020-10-08T11:16:14.7274287Z [cluster3] I1008 11:09:38.601397       1 shared_informer.go:204] Caches are synced for endpoints config
2020-10-08T11:16:14.7275240Z [cluster3] +++++++++++++++++++++: Logs for Pod kube-proxy-tgpm6 in namespace kube-system :++++++++++++++++++++++
2020-10-08T11:16:14.7276123Z [cluster3] W1008 11:10:10.973580       1 server_others.go:323] Unknown proxy mode "", assuming iptables proxy
2020-10-08T11:16:14.7277168Z [cluster3] I1008 11:10:11.123249       1 node.go:135] Successfully retrieved node IP: 172.17.0.6
2020-10-08T11:16:14.7277738Z [cluster3] I1008 11:10:11.123287       1 server_others.go:145] Using iptables Proxier.
2020-10-08T11:16:14.7278249Z [cluster3] I1008 11:10:11.123547       1 server.go:571] Version: v1.17.0
2020-10-08T11:16:14.7278754Z [cluster3] I1008 11:10:11.123957       1 conntrack.go:52] Setting nf_conntrack_max to 131072
2020-10-08T11:16:14.7279776Z [cluster3] I1008 11:10:11.124030       1 conntrack.go:100] Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_established' to 86400
2020-10-08T11:16:14.7281050Z [cluster3] I1008 11:10:11.124073       1 conntrack.go:100] Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_close_wait' to 3600
2020-10-08T11:16:14.7430503Z [submariner]$ [cluster3] namespace=submariner-operator
2020-10-08T11:16:14.7552809Z [submariner]$ [cluster3] kubectl get pods --selector=name=submariner-operator -n submariner-operator -o jsonpath={.items[*].metadata.name}
2020-10-08T11:16:14.7588180Z [submariner]$ [cluster3] kubectl get pods --selector=name=submariner-operator -n submariner-operator -o jsonpath={.items[*].metadata.name}
2020-10-08T11:16:14.7601188Z [submariner]$ [cluster3] command kubectl --context=cluster3 get pods --selector=name=submariner-operator -n submariner-operator -o jsonpath={.items[*].metadata.name}
2020-10-08T11:16:14.7614326Z [submariner]$ [cluster3] kubectl --context=cluster3 get pods --selector=name=submariner-operator -n submariner-operator -o jsonpath={.items[*].metadata.name}
2020-10-08T11:16:15.5531012Z [submariner]$ [cluster3] namespace=submariner-operator
2020-10-08T11:16:15.5542694Z [submariner]$ [cluster3] kubectl get pods --selector=app=submariner-globalnet -n submariner-operator -o jsonpath={.items[*].metadata.name}
2020-10-08T11:16:15.5554493Z [submariner]$ [cluster3] kubectl get pods --selector=app=submariner-globalnet -n submariner-operator -o jsonpath={.items[*].metadata.name}
2020-10-08T11:16:15.5568368Z [submariner]$ [cluster3] command kubectl --context=cluster3 get pods --selector=app=submariner-globalnet -n submariner-operator -o jsonpath={.items[*].metadata.name}
2020-10-08T11:16:15.5581347Z [submariner]$ [cluster3] kubectl --context=cluster3 get pods --selector=app=submariner-globalnet -n submariner-operator -o jsonpath={.items[*].metadata.name}
2020-10-08T11:16:16.3017676Z [submariner]$ [cluster3] kubectl get pods --selector=app=submariner-engine -n submariner-operator -o jsonpath={.items[*].metadata.name}
2020-10-08T11:16:16.3032132Z [submariner]$ [cluster3] kubectl get pods --selector=app=submariner-engine -n submariner-operator -o jsonpath={.items[*].metadata.name}
2020-10-08T11:16:16.3046989Z [submariner]$ [cluster3] command kubectl --context=cluster3 get pods --selector=app=submariner-engine -n submariner-operator -o jsonpath={.items[*].metadata.name}
2020-10-08T11:16:16.3060719Z [submariner]$ [cluster3] kubectl --context=cluster3 get pods --selector=app=submariner-engine -n submariner-operator -o jsonpath={.items[*].metadata.name}
2020-10-08T11:16:17.0496609Z [submariner]$ [cluster3] kubectl -n submariner-operator logs submariner-gateway-pkb8n
2020-10-08T11:16:17.0504987Z [submariner]$ [cluster3] kubectl -n submariner-operator logs submariner-gateway-pkb8n
2020-10-08T11:16:17.0514430Z [submariner]$ [cluster3] command kubectl --context=cluster3 -n submariner-operator logs submariner-gateway-pkb8n
2020-10-08T11:16:17.0526715Z [submariner]$ [cluster3] kubectl --context=cluster3 -n submariner-operator logs submariner-gateway-pkb8n
2020-10-08T11:16:17.8131097Z [cluster3] I1008 11:10:11.125617       1 config.go:131] Starting endpoints config controller
2020-10-08T11:16:17.8131816Z [cluster3] I1008 11:10:11.125639       1 shared_informer.go:197] Waiting for caches to sync for endpoints config
2020-10-08T11:16:17.8132499Z [cluster3] I1008 11:10:11.125695       1 config.go:313] Starting service config controller
2020-10-08T11:16:17.8133127Z [cluster3] I1008 11:10:11.125717       1 shared_informer.go:197] Waiting for caches to sync for service config
2020-10-08T11:16:17.8133948Z [cluster3] I1008 11:10:11.226484       1 shared_informer.go:204] Caches are synced for endpoints config
2020-10-08T11:16:17.8134600Z [cluster3] I1008 11:10:11.226484       1 shared_informer.go:204] Caches are synced for service config
2020-10-08T11:16:17.8135075Z [cluster3] ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
2020-10-08T11:16:17.8135389Z [cluster3] ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
2020-10-08T11:16:17.8135705Z [cluster3] ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
2020-10-08T11:16:17.8137084Z [cluster3] +++++++++++++++++++++: Logs for Pod submariner-gateway-pkb8n in namespace submariner-operator :++++++++++++++++++++++
2020-10-08T11:16:17.8137965Z [cluster3] + trap 'exit 1' SIGTERM SIGINT
2020-10-08T11:16:17.8138426Z [cluster3] + export CHARON_PID_FILE=/var/run/charon.pid
2020-10-08T11:16:17.8138901Z [cluster3] + CHARON_PID_FILE=/var/run/charon.pid
2020-10-08T11:16:17.8141950Z [cluster3] + rm -f /var/run/charon.pid
2020-10-08T11:16:17.8142359Z [cluster3] + SUBMARINER_VERBOSITY=2
2020-10-08T11:16:17.8142962Z [cluster3] + '[' false == true ']'
2020-10-08T11:16:17.8143416Z [cluster3] + DEBUG=-v=2
2020-10-08T11:16:17.8143888Z [cluster3] + mkdir -p /etc/ipsec
2020-10-08T11:16:17.8144556Z [cluster3] + sysctl -w net.ipv4.conf.all.send_redirects=0
2020-10-08T11:16:17.8145334Z [cluster3] net.ipv4.conf.all.send_redirects = 0
2020-10-08T11:16:17.8146004Z [cluster3] + export PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/libexec/strongswan
2020-10-08T11:16:17.8146727Z [cluster3] + PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/libexec/strongswan
2020-10-08T11:16:17.8147513Z [cluster3] + for f in iptables-save iptables
2020-10-08T11:16:17.8149151Z [cluster3] ++ find_iptables_on_host iptables-save
2020-10-08T11:16:17.8150089Z [cluster3] ++ chroot /host test -x /usr/sbin/iptables-save
2020-10-08T11:16:17.8150885Z [cluster3] chroot: cannot change root directory to '/host': No such file or directory
2020-10-08T11:16:17.8151924Z [cluster3] ++ chroot /host test -x /sbin/iptables-save
2020-10-08T11:16:17.8152692Z [cluster3] chroot: cannot change root directory to '/host': No such file or directory
2020-10-08T11:16:17.8153181Z [cluster3] ++ echo unknown
2020-10-08T11:16:17.8153531Z [cluster3] + location=unknown
2020-10-08T11:16:17.8154227Z [cluster3] + '[' unknown '!=' unknown ']'
2020-10-08T11:16:17.8155038Z [cluster3] + echo 'WARNING: not using iptables wrapper because iptables was not detected on the'
2020-10-08T11:16:17.8155765Z [cluster3] WARNING: not using iptables wrapper because iptables was not detected on the
2020-10-08T11:16:17.8157324Z [cluster3] + echo 'host at the following paths [/usr/sbin, /sbin].'
2020-10-08T11:16:17.8157873Z [cluster3] host at the following paths [/usr/sbin, /sbin].
2020-10-08T11:16:17.8158758Z [cluster3] + echo 'Either the host file system isn'\''t mounted or the host does not have iptables'
2020-10-08T11:16:17.8159637Z [cluster3] Either the host file system isn't mounted or the host does not have iptables
2020-10-08T11:16:17.8160500Z [cluster3] + echo 'installed. The pod will use the image installed iptables version.'
2020-10-08T11:16:17.8161173Z [cluster3] installed. The pod will use the image installed iptables version.
2020-10-08T11:16:17.8161898Z [cluster3] + for f in iptables-save iptables
2020-10-08T11:16:17.8162332Z [cluster3] ++ find_iptables_on_host iptables
2020-10-08T11:16:17.8162942Z [cluster3] ++ chroot /host test -x /usr/sbin/iptables
2020-10-08T11:16:17.8163687Z [cluster3] chroot: cannot change root directory to '/host': No such file or directory
2020-10-08T11:16:17.8164391Z [cluster3] ++ chroot /host test -x /sbin/iptables
2020-10-08T11:16:17.8165131Z [cluster3] chroot: cannot change root directory to '/host': No such file or directory
2020-10-08T11:16:17.8165618Z [cluster3] ++ echo unknown
2020-10-08T11:16:17.8165965Z [cluster3] + location=unknown
2020-10-08T11:16:17.8166491Z [cluster3] + '[' unknown '!=' unknown ']'
2020-10-08T11:16:17.8167240Z [cluster3] + echo 'WARNING: not using iptables wrapper because iptables was not detected on the'
2020-10-08T11:16:17.8167967Z [cluster3] WARNING: not using iptables wrapper because iptables was not detected on the
2020-10-08T11:16:17.8168755Z [cluster3] + echo 'host at the following paths [/usr/sbin, /sbin].'
2020-10-08T11:16:17.8169262Z [cluster3] host at the following paths [/usr/sbin, /sbin].
2020-10-08T11:16:17.8170053Z [cluster3] + echo 'Either the host file system isn'\''t mounted or the host does not have iptables'
2020-10-08T11:16:17.8170911Z [cluster3] Either the host file system isn't mounted or the host does not have iptables
2020-10-08T11:16:17.8171775Z [cluster3] + echo 'installed. The pod will use the image installed iptables version.'
2020-10-08T11:16:17.8172821Z [cluster3] installed. The pod will use the image installed iptables version.
2020-10-08T11:16:17.8173753Z [cluster3] + exec submariner-engine -v=2 -alsologtostderr
2020-10-08T11:16:17.8174384Z [cluster3] I1008 11:15:47.295789       1 main.go:71] Starting the submariner gateway engine
2020-10-08T11:16:17.8175471Z [cluster3] W1008 11:15:47.299166       1 client_config.go:543] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
2020-10-08T11:16:17.8176242Z [cluster3] I1008 11:15:47.302779       1 main.go:99] Creating the cable engine
2020-10-08T11:16:17.8176865Z [cluster3] I1008 11:15:47.309637       1 syncer.go:51] CableEngine syncer started
2020-10-08T11:16:17.8178273Z [cluster3] I1008 11:15:47.323373       1 main.go:263] Gateway leader election config values: main.leaderConfig{LeaseDuration:10, RenewDeadline:5, RetryPeriod:2}
2020-10-08T11:16:17.8179810Z [cluster3] I1008 11:15:47.324821       1 main.go:280] Using namespace "submariner-operator" for the leader election lock
2020-10-08T11:16:17.8181155Z [cluster3] I1008 11:15:47.325228       1 leaderelection.go:242] attempting to acquire leader lease  submariner-operator/submariner-engine-lock...
2020-10-08T11:16:17.8182876Z [cluster3] I1008 11:15:47.405863       1 leaderelection.go:252] successfully acquired lease submariner-operator/submariner-engine-lock
2020-10-08T11:16:17.8185398Z [cluster3] I1008 11:15:47.405934       1 event.go:281] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"submariner-operator", Name:"submariner-engine-lock", UID:"40ae1462-b2b2-4184-bf6e-d00e1538653a", APIVersion:"v1", ResourceVersion:"1689", FieldPath:""}): type: 'Normal' reason: 'LeaderElection' cluster3-worker-submariner-engine became leader
2020-10-08T11:16:17.8187111Z [cluster3] I1008 11:15:47.406073       1 main.go:133] Creating the tunnel controller
2020-10-08T11:16:17.8187791Z [cluster3] I1008 11:15:47.406131       1 main.go:141] Creating the kubernetes central datastore
2020-10-08T11:16:17.8188533Z [cluster3] I1008 11:15:47.406177       1 kubernetes.go:60] Rendered API server host: "https://172.17.0.10:6443"
2020-10-08T11:16:17.8189190Z [cluster3] I1008 11:15:47.513457       1 main.go:151] Creating the datastore syncer
2020-10-08T11:16:17.8189805Z [cluster3] I1008 11:15:47.513649       1 strongswan.go:99] Initializing StrongSwan IPSec driver
2020-10-08T11:16:17.8190406Z [cluster3] I1008 11:15:47.514015       1 strongswan.go:460] Starting charon
2020-10-08T11:16:17.8191498Z [cluster3] W1008 11:15:47.525824       1 strongswan.go:543] Failed to connect to charon - retrying: dial unix /var/run/charon.vici: connect: no such file or directory
2020-10-08T11:16:17.8192536Z [cluster3] 00[DMN] Starting IKE charon daemon (strongSwan 5.9.0, Linux 5.4.0-1026-azure, x86_64)
2020-10-08T11:16:17.8193277Z [cluster3] 00[CFG] PKCS11 module '<name>' lacks library path
2020-10-08T11:16:17.8193904Z [cluster3] 00[LIB] openssl FIPS mode(0) - disabled
2020-10-08T11:16:17.8194666Z [cluster3] 00[CFG] loading ca certificates from '/etc/strongswan/ipsec.d/cacerts'
2020-10-08T11:16:17.8195544Z [cluster3] 00[CFG] loading aa certificates from '/etc/strongswan/ipsec.d/aacerts'
2020-10-08T11:16:17.8205341Z [cluster3] 00[CFG] loading ocsp signer certificates from '/etc/strongswan/ipsec.d/ocspcerts'
2020-10-08T11:16:17.8206725Z [cluster3] 00[CFG] loading attribute certificates from '/etc/strongswan/ipsec.d/acerts'
2020-10-08T11:16:17.8207592Z [cluster3] 00[CFG] loading crls from '/etc/strongswan/ipsec.d/crls'
2020-10-08T11:16:17.8208382Z [cluster3] 00[CFG] loading secrets from '/etc/strongswan/ipsec.secrets'
2020-10-08T11:16:17.8209135Z [cluster3] 00[CFG] opening triplet file /etc/strongswan/ipsec.d/triplets.dat failed: No such file or directory
2020-10-08T11:16:17.8209836Z [cluster3] 00[CFG] loaded 0 RADIUS server configurations
2020-10-08T11:16:17.8210345Z [cluster3] 00[CFG] HA config misses local/remote address
2020-10-08T11:16:17.8211060Z [cluster3] 00[CFG] no script for ext-auth script defined, disabled
2020-10-08T11:16:17.8213838Z [cluster3] 00[LIB] loaded plugins: charon pkcs11 tpm aesni aes des rc2 sha2 sha1 md4 md5 mgf1 random nonce x509 revocation constraints acert pubkey pkcs1 pkcs7 pkcs8 pkcs12 pgp dnskey sshkey pem openssl gcrypt fips-prf gmp curve25519 chapoly xcbc cmac hmac ctr ccm gcm drbg newhope curl attr kernel-netlink resolve socket-default farp stroke vici updown eap-identity eap-sim eap-aka eap-aka-3gpp eap-aka-3gpp2 eap-md5 eap-gtc eap-mschapv2 eap-dynamic eap-radius eap-tls eap-ttls eap-peap xauth-generic xauth-eap xauth-pam xauth-noauth dhcp led duplicheck unity counters
2020-10-08T11:16:17.8215899Z [cluster3] 00[JOB] spawning 16 worker threads
2020-10-08T11:16:17.8216530Z [cluster3] I1008 11:15:48.528028       1 cableengine.go:71] CableEngine controller started, driver: "strongswan"
2020-10-08T11:16:17.8217294Z [cluster3] I1008 11:15:48.528107       1 datastoresyncer.go:145] Starting the datastore syncer
2020-10-08T11:16:17.8217999Z [cluster3] I1008 11:15:48.528176       1 datastoresyncer.go:147] Waiting for informer caches to sync
2020-10-08T11:16:17.8218766Z [cluster3] I1008 11:15:48.528205       1 datastoresyncer.go:79] Ensuring we are the only endpoint active for this cluster
2020-10-08T11:16:17.8220266Z [cluster3] I1008 11:15:48.528149       1 tunnel.go:47] Starting the tunnel controller
2020-10-08T11:16:17.8221086Z [cluster3] I1008 11:15:48.530315       1 tunnel.go:50] Waiting for informer caches to sync
2020-10-08T11:16:17.8221638Z [cluster3] I1008 11:15:48.530334       1 tunnel.go:58] Tunnel controller started
2020-10-08T11:16:17.8222955Z [cluster3] I1008 11:15:48.530957       1 datastoresyncer.go:158] Reconciling local submariner Cluster: types.SubmarinerCluster{ID:"cluster3", Spec:v1.ClusterSpec{ClusterID:"cluster3", ColorCodes:[]string{"blue"}, ServiceCIDR:[]string{"100.93.0.0/16"}, ClusterCIDR:[]string{"10.243.0.0/16"}, GlobalCIDR:[]string{}}}
2020-10-08T11:16:17.8225167Z [cluster3] I1008 11:15:48.530991       1 datastoresyncer.go:312] In reconcileClusterCRD: &types.SubmarinerCluster{ID:"cluster3", Spec:v1.ClusterSpec{ClusterID:"cluster3", ColorCodes:[]string{"blue"}, ServiceCIDR:[]string{"100.93.0.0/16"}, ClusterCIDR:[]string{"10.243.0.0/16"}, GlobalCIDR:[]string{}}}
2020-10-08T11:16:17.8226721Z [cluster3] I1008 11:15:48.542703       1 datastoresyncer.go:357] Successfully created submariner Cluster "cluster3" in the local datastore
2020-10-08T11:16:17.8229471Z [cluster3] I1008 11:15:48.543058       1 datastoresyncer.go:164] Reconciling local submariner Endpoint: types.SubmarinerEndpoint{Spec:v1.EndpointSpec{ClusterID:"cluster3", CableName:"submariner-cable-cluster3-172-17-0-5", Hostname:"cluster3-worker", Subnets:[]string{"100.93.0.0/16", "10.243.0.0/16"}, PrivateIP:"172.17.0.5", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string{}}}
2020-10-08T11:16:17.8233151Z [cluster3] I1008 11:15:48.543097       1 datastoresyncer.go:387] In reconcileEndpointCRD: &types.SubmarinerEndpoint{Spec:v1.EndpointSpec{ClusterID:"cluster3", CableName:"submariner-cable-cluster3-172-17-0-5", Hostname:"cluster3-worker", Subnets:[]string{"100.93.0.0/16", "10.243.0.0/16"}, PrivateIP:"172.17.0.5", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string{}}}
2020-10-08T11:16:17.8288376Z [cluster3] I1008 11:15:48.563405       1 datastoresyncer.go:432] Successfully created submariner Endpoint "cluster3-submariner-cable-cluster3-172-17-0-5" in the local datastore
2020-10-08T11:16:17.8289718Z [cluster3] I1008 11:15:48.565515       1 datastoresyncer.go:177] Datastore syncer started
2020-10-08T11:16:17.8290990Z [cluster3] I1008 11:15:48.569259       1 kubernetes.go:317] In SetCluster: &types.SubmarinerCluster{ID:"cluster3", Spec:v1.ClusterSpec{ClusterID:"cluster3", ColorCodes:[]string{"blue"}, ServiceCIDR:[]string{"100.93.0.0/16"}, ClusterCIDR:[]string{"10.243.0.0/16"}, GlobalCIDR:[]string{}}}
2020-10-08T11:16:17.8298235Z [cluster3] I1008 11:15:48.570033       1 tunnel.go:95] Tunnel controller processing added or updated submariner Endpoint object: &v1.Endpoint{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"cluster3-submariner-cable-cluster3-172-17-0-5", GenerateName:"", Namespace:"submariner-operator", SelfLink:"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster3-submariner-cable-cluster3-172-17-0-5", UID:"bcbc8629-05e5-4a7c-b55c-840e9e8463a6", ResourceVersion:"1696", Generation:1, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63737752548, loc:(*time.Location)(0x225ef80)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v1.EndpointSpec{ClusterID:"cluster3", CableName:"submariner-cable-cluster3-172-17-0-5", Hostname:"cluster3-worker", Subnets:[]string{"100.93.0.0/16", "10.243.0.0/16"}, PrivateIP:"172.17.0.5", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
2020-10-08T11:16:17.8303451Z [cluster3] I1008 11:15:48.573628       1 cableengine.go:92] Not installing cable for local cluster
2020-10-08T11:16:17.8306590Z [cluster3] I1008 11:15:48.573644       1 tunnel.go:108] Tunnel controller successfully installed Endpoint cable submariner-cable-cluster3-172-17-0-5 in the engine
2020-10-08T11:16:17.8314217Z [cluster3] I1008 11:15:48.606026       1 datastoresyncer.go:287] Processing local submariner Endpoint object: &v1.Endpoint{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"cluster3-submariner-cable-cluster3-172-17-0-5", GenerateName:"", Namespace:"submariner-operator", SelfLink:"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster3-submariner-cable-cluster3-172-17-0-5", UID:"bcbc8629-05e5-4a7c-b55c-840e9e8463a6", ResourceVersion:"1696", Generation:1, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63737752548, loc:(*time.Location)(0x225ef80)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v1.EndpointSpec{ClusterID:"cluster3", CableName:"submariner-cable-cluster3-172-17-0-5", Hostname:"cluster3-worker", Subnets:[]string{"100.93.0.0/16", "10.243.0.0/16"}, PrivateIP:"172.17.0.5", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
2020-10-08T11:16:17.8322997Z [cluster3] I1008 11:15:48.606102       1 kubernetes.go:370] In SetEndpoint: &types.SubmarinerEndpoint{Spec:v1.EndpointSpec{ClusterID:"cluster3", CableName:"submariner-cable-cluster3-172-17-0-5", Hostname:"cluster3-worker", Subnets:[]string{"100.93.0.0/16", "10.243.0.0/16"}, PrivateIP:"172.17.0.5", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
2020-10-08T11:16:17.8325841Z [cluster3] I1008 11:15:48.664685       1 kubernetes.go:155] AddFunc in WatchClusters called
2020-10-08T11:16:17.8327631Z [cluster3] I1008 11:15:48.664707       1 datastoresyncer.go:312] In reconcileClusterCRD: &types.SubmarinerCluster{ID:"cluster2", Spec:v1.ClusterSpec{ClusterID:"cluster2", ColorCodes:[]string{"blue"}, ServiceCIDR:[]string{"100.92.0.0/16"}, ClusterCIDR:[]string{"10.242.0.0/16"}, GlobalCIDR:[]string{}}}
2020-10-08T11:16:17.8329052Z [cluster3] I1008 11:15:48.667487       1 kubernetes.go:240] AddFunc in WatchEndpoints called
2020-10-08T11:16:17.8331605Z [cluster3] I1008 11:15:48.667560       1 datastoresyncer.go:387] In reconcileEndpointCRD: &types.SubmarinerEndpoint{Spec:v1.EndpointSpec{ClusterID:"cluster2", CableName:"submariner-cable-cluster2-172-17-0-8", Hostname:"cluster2-worker", Subnets:[]string{"100.92.0.0/16", "10.242.0.0/16"}, PrivateIP:"172.17.0.8", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
2020-10-08T11:16:17.8335075Z [cluster3] I1008 11:15:48.675438       1 kubernetes.go:395] Successfully created submariner Endpoint "cluster3-submariner-cable-cluster3-172-17-0-5" in the central datastore
2020-10-08T11:16:17.8342509Z [cluster3] I1008 11:15:48.680496       1 tunnel.go:95] Tunnel controller processing added or updated submariner Endpoint object: &v1.Endpoint{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"cluster2-submariner-cable-cluster2-172-17-0-8", GenerateName:"", Namespace:"submariner-operator", SelfLink:"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster2-submariner-cable-cluster2-172-17-0-8", UID:"ea0afa4a-05a3-4175-b625-82869baf0e4e", ResourceVersion:"1697", Generation:1, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63737752548, loc:(*time.Location)(0x225ef80)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v1.EndpointSpec{ClusterID:"cluster2", CableName:"submariner-cable-cluster2-172-17-0-8", Hostname:"cluster2-worker", Subnets:[]string{"100.92.0.0/16", "10.242.0.0/16"}, PrivateIP:"172.17.0.8", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
2020-10-08T11:16:17.8391856Z [cluster3] I1008 11:15:48.680589       1 cableengine.go:101] Installing Endpoint cable "submariner-cable-cluster2-172-17-0-8"
2020-10-08T11:16:17.8392923Z [cluster3] 04[CFG] loaded IKE shared key for: '172.17.0.8'
2020-10-08T11:16:17.8393858Z [cluster3] 08[CFG] added vici connection: submariner-cable-cluster2-172-17-0-8
2020-10-08T11:16:17.8395444Z [cluster3] 08[CFG] initiating 'submariner-child-submariner-cable-cluster2-172-17-0-8'
2020-10-08T11:16:17.8397618Z [cluster3] 08[IKE] initiating IKE_SA submariner-cable-cluster2-172-17-0-8[1] to 172.17.0.8
2020-10-08T11:16:17.8398706Z [cluster3] 08[ENC] generating IKE_SA_INIT request 0 [ SA KE No N(NATD_S_IP) N(NATD_D_IP) N(FRAG_SUP) N(HASH_ALG) N(REDIR_SUP) ]
2020-10-08T11:16:17.8399350Z [cluster3] 08[NET] sending packet: from 172.17.0.5[500] to 172.17.0.8[500] (500 bytes)
2020-10-08T11:16:17.8426286Z [submariner]$ [cluster3] kubectl get pods --selector=app=submariner-routeagent -n submariner-operator -o jsonpath={.items[*].metadata.name}
2020-10-08T11:16:17.8440231Z [submariner]$ [cluster3] kubectl get pods --selector=app=submariner-routeagent -n submariner-operator -o jsonpath={.items[*].metadata.name}
2020-10-08T11:16:17.8455063Z [submariner]$ [cluster3] command kubectl --context=cluster3 get pods --selector=app=submariner-routeagent -n submariner-operator -o jsonpath={.items[*].metadata.name}
2020-10-08T11:16:17.8471018Z [submariner]$ [cluster3] kubectl --context=cluster3 get pods --selector=app=submariner-routeagent -n submariner-operator -o jsonpath={.items[*].metadata.name}
2020-10-08T11:16:18.5991527Z [submariner]$ [cluster3] kubectl -n submariner-operator logs submariner-routeagent-596j7
2020-10-08T11:16:18.6005490Z [submariner]$ [cluster3] kubectl -n submariner-operator logs submariner-routeagent-596j7
2020-10-08T11:16:18.6014949Z [submariner]$ [cluster3] command kubectl --context=cluster3 -n submariner-operator logs submariner-routeagent-596j7
2020-10-08T11:16:18.6027325Z [submariner]$ [cluster3] kubectl --context=cluster3 -n submariner-operator logs submariner-routeagent-596j7
2020-10-08T11:16:19.3417155Z [cluster3] 11[NET] received packet: from 172.17.0.8[500] to 172.17.0.5[500] (36 bytes)
2020-10-08T11:16:19.3417818Z [cluster3] 11[ENC] parsed IKE_SA_INIT response 0 [ N(NO_PROP) ]
2020-10-08T11:16:19.3418333Z [cluster3] 11[IKE] received NO_PROPOSAL_CHOSEN notify error
2020-10-08T11:16:19.3420076Z [cluster3] I1008 11:15:48.690215       1 cableengine.go:150] Successfully installed Endpoint cable "submariner-cable-cluster2-172-17-0-8" with remote IP 172.17.0.8
2020-10-08T11:16:19.3421872Z [cluster3] I1008 11:15:48.690353       1 tunnel.go:108] Tunnel controller successfully installed Endpoint cable submariner-cable-cluster2-172-17-0-8 in the engine
2020-10-08T11:16:19.3422867Z [cluster3] 14[NET] received packet: from 172.17.0.8[500] to 172.17.0.5[500] (500 bytes)
2020-10-08T11:16:19.3423474Z [cluster3] 14[ENC] parsed IKE_SA_INIT request 0 [ SA KE No N(NATD_S_IP) N(NATD_D_IP) N(FRAG_SUP) N(HASH_ALG) N(REDIR_SUP) ]
2020-10-08T11:16:19.3424055Z [cluster3] 14[IKE] 172.17.0.8 is initiating an IKE_SA
2020-10-08T11:16:19.3424544Z [cluster3] 14[CFG] selected proposal: IKE:AES_GCM_16_128/PRF_HMAC_SHA2_256/MODP_2048
2020-10-08T11:16:19.3426108Z [cluster3] I1008 11:15:48.745186       1 datastoresyncer.go:432] Successfully created submariner Endpoint "cluster2-submariner-cable-cluster2-172-17-0-8" in the local datastore
2020-10-08T11:16:19.3427407Z [cluster3] I1008 11:15:48.745211       1 kubernetes.go:240] AddFunc in WatchEndpoints called
2020-10-08T11:16:19.3428251Z [cluster3] I1008 11:15:48.745293       1 datastoresyncer.go:357] Successfully created submariner Cluster "cluster2" in the local datastore
2020-10-08T11:16:19.3429072Z [cluster3] I1008 11:15:48.745307       1 kubernetes.go:155] AddFunc in WatchClusters called
2020-10-08T11:16:19.3429863Z [cluster3] I1008 11:15:48.745585       1 kubernetes.go:342] Successfully created submariner Cluster "cluster3" in the central datastore
2020-10-08T11:16:19.3431437Z [cluster3] I1008 11:15:48.745780       1 datastoresyncer.go:271] The updated submariner Endpoint "cluster2" is not for this cluster - skipping updating the datastore
2020-10-08T11:16:19.3432211Z [cluster3] 14[IKE] remote host is behind NAT
2020-10-08T11:16:19.3432840Z [cluster3] 14[ENC] generating IKE_SA_INIT response 0 [ SA KE No N(NATD_S_IP) N(NATD_D_IP) N(FRAG_SUP) N(HASH_ALG) N(CHDLESS_SUP) N(MULT_AUTH) ]
2020-10-08T11:16:19.3433499Z [cluster3] 14[NET] sending packet: from 172.17.0.5[500] to 172.17.0.8[500] (464 bytes)
2020-10-08T11:16:19.3433981Z [cluster3] 06[NET] received packet: from 172.17.0.8[4500] to 172.17.0.5[4500] (333 bytes)
2020-10-08T11:16:19.3434956Z [cluster3] 06[ENC] parsed IKE_AUTH request 1 [ IDi N(INIT_CONTACT) IDr AUTH SA TSi TSr N(MULT_AUTH) N(EAP_ONLY) N(MSG_ID_SYN_SUP) ]
2020-10-08T11:16:19.3435627Z [cluster3] 06[CFG] looking for peer configs matching 172.17.0.5[172.17.0.5]...172.17.0.8[172.17.0.8]
2020-10-08T11:16:19.3437219Z [cluster3] 06[CFG] selected peer config 'submariner-cable-cluster2-172-17-0-8'
2020-10-08T11:16:19.3438458Z [cluster3] 06[IKE] authentication of '172.17.0.8' with pre-shared key successful
2020-10-08T11:16:19.3439277Z [cluster3] 06[IKE] authentication of '172.17.0.5' (myself) with pre-shared key
2020-10-08T11:16:19.3440398Z [cluster3] 06[IKE] IKE_SA submariner-cable-cluster2-172-17-0-8[2] established between 172.17.0.5[172.17.0.5]...172.17.0.8[172.17.0.8]
2020-10-08T11:16:19.3441196Z [cluster3] 06[IKE] scheduling rekeying in 13755s
2020-10-08T11:16:19.3441614Z [cluster3] 06[IKE] maximum IKE_SA lifetime 15195s
2020-10-08T11:16:19.3442082Z [cluster3] 06[CFG] selected proposal: ESP:AES_GCM_16_128/NO_EXT_SEQ
2020-10-08T11:16:19.3443736Z [cluster3] 06[IKE] CHILD_SA submariner-child-submariner-cable-cluster2-172-17-0-8{1} established with SPIs cdb0c4b9_i cec24d50_o and TS 10.243.0.0/16 100.93.0.0/16 172.17.0.5/32 === 10.242.0.0/16 100.92.0.0/16 172.17.0.8/32
2020-10-08T11:16:19.3445217Z [cluster3] 06[ENC] generating IKE_AUTH response 1 [ IDr AUTH SA TSi TSr ]
2020-10-08T11:16:19.3445752Z [cluster3] 06[NET] sending packet: from 172.17.0.5[4500] to 172.17.0.8[4500] (257 bytes)
2020-10-08T11:16:19.3446487Z [cluster3] 10[KNL] interface vx-submariner activated
2020-10-08T11:16:19.3447173Z [cluster3] 11[KNL] 240.17.0.5 appeared on vx-submariner
2020-10-08T11:16:19.3447722Z [cluster3] ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
2020-10-08T11:16:19.3448695Z [cluster3] +++++++++++++++++++++: Logs for Pod submariner-routeagent-596j7 in namespace submariner-operator :++++++++++++++++++++++
2020-10-08T11:16:19.3449595Z [cluster3] + trap 'exit 1' SIGTERM SIGINT
2020-10-08T11:16:19.3450017Z [cluster3] + SUBMARINER_VERBOSITY=2
2020-10-08T11:16:19.3450549Z [cluster3] + '[' false == true ']'
2020-10-08T11:16:19.3450986Z [cluster3] + DEBUG=-v=2
2020-10-08T11:16:19.3451533Z [cluster3] + for f in iptables-save iptables
2020-10-08T11:16:19.3452178Z [cluster3] ++ find_iptables_on_host iptables-save
2020-10-08T11:16:19.3452864Z [cluster3] ++ chroot /host test -x /usr/sbin/iptables-save
2020-10-08T11:16:19.3453638Z [cluster3] iptables-save is present on the host at /usr/sbin/iptables-save
2020-10-08T11:16:19.3454131Z [cluster3] ++ echo /usr/sbin
2020-10-08T11:16:19.3454439Z [cluster3] ++ return
2020-10-08T11:16:19.3454767Z [cluster3] + location=/usr/sbin
2020-10-08T11:16:19.3455273Z [cluster3] + '[' /usr/sbin '!=' unknown ']'
2020-10-08T11:16:19.3465854Z [cluster3] + echo 'iptables-save is present on the host at /usr/sbin/iptables-save'
2020-10-08T11:16:19.3466824Z [cluster3] + sed 's!@@PATH@@!/usr/sbin!' /usr/sbin/iptables-wrapper.in
2020-10-08T11:16:19.3467703Z [cluster3] + for f in iptables-save iptables
2020-10-08T11:16:19.3468360Z [cluster3] ++ find_iptables_on_host iptables
2020-10-08T11:16:19.3469021Z [cluster3] ++ chroot /host test -x /usr/sbin/iptables
2020-10-08T11:16:19.3469415Z [cluster3] ++ echo /usr/sbin
2020-10-08T11:16:19.3469721Z [cluster3] ++ return
2020-10-08T11:16:19.3470244Z [cluster3] + location=/usr/sbin
2020-10-08T11:16:19.3470816Z [cluster3] + '[' /usr/sbin '!=' unknown ']'
2020-10-08T11:16:19.3471675Z [cluster3] + echo 'iptables is present on the host at /usr/sbin/iptables'
2020-10-08T11:16:19.3472524Z [cluster3] + sed 's!@@PATH@@!/usr/sbin!' /usr/sbin/iptables-wrapper.in
2020-10-08T11:16:19.3473380Z [cluster3] + exec submariner-route-agent -v=2 -alsologtostderr
2020-10-08T11:16:19.3474016Z [cluster3] iptables is present on the host at /usr/sbin/iptables
2020-10-08T11:16:19.3474832Z [cluster3] I1008 11:15:47.046887       1 main.go:47] Starting submariner-route-agent
2020-10-08T11:16:19.3476093Z [cluster3] W1008 11:15:47.047758       1 client_config.go:543] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
2020-10-08T11:16:19.3477736Z [cluster3] I1008 11:15:47.072344       1 route.go:195] Starting Route Controller. ClusterID: cluster3, localClusterCIDR: [10.243.0.0/16], localServiceCIDR: [100.93.0.0/16]
2020-10-08T11:16:19.3478616Z [cluster3] I1008 11:15:47.072394       1 route.go:199] Waiting for Endpoint informer caches to sync.
2020-10-08T11:16:19.3479597Z [cluster3] I1008 11:15:47.186028       1 route.go:751] Enqueueing sm-route-agent-pod event, ip: 172.17.0.6
2020-10-08T11:16:19.3480523Z [cluster3] I1008 11:15:47.186055       1 route.go:751] Enqueueing sm-route-agent-pod event, ip: 172.17.0.5
2020-10-08T11:16:19.3481134Z [cluster3] I1008 11:15:47.273050       1 driver.go:37] Interface "lo" has "127.0.0.1" address
2020-10-08T11:16:19.3481670Z [cluster3] I1008 11:15:47.273135       1 driver.go:37] Interface "weave" has "10.243.128.0" address
2020-10-08T11:16:19.3482313Z [cluster3] I1008 11:15:47.273143       1 driver.go:42] Found CNI Interface "weave" that has IP "10.243.128.0" from ClusterCIDR "10.243.0.0/16"
2020-10-08T11:16:19.3483102Z [cluster3] I1008 11:15:47.273230       1 driver.go:60] Successfully configured rp_filter to loose mode(2) on cniInterface "weave"
2020-10-08T11:16:19.3484190Z [cluster3] I1008 11:15:47.291869       1 iptables.go:22] Install/ensure SUBMARINER-POSTROUTING chain exists
2020-10-08T11:16:19.3485553Z [cluster3] I1008 11:15:47.310255       1 iptables.go:28] Insert SUBMARINER-POSTROUTING rule that has rules for inter-cluster traffic
2020-10-08T11:16:19.3486899Z [cluster3] I1008 11:15:47.329451       1 iptables.go:34] Install/ensure SUBMARINER-INPUT chain exists
2020-10-08T11:16:19.3488035Z [cluster3] I1008 11:15:47.386465       1 iptables.go:45] Allow VxLAN incoming traffic in SUBMARINER-INPUT Chain
2020-10-08T11:16:19.3489139Z [cluster3] I1008 11:15:47.416670       1 iptables.go:53] Insert rule to allow traffic over vx-submariner interface in FORWARDing Chain
2020-10-08T11:16:19.3490387Z [cluster3] I1008 11:15:47.437255       1 iptables.go:65] Installing rule for host network to remote cluster communication: -s 240.0.0.0/8 -o vx-submariner -j SNAT --to 10.243.128.0
2020-10-08T11:16:19.3491532Z [cluster3] I1008 11:15:47.479993       1 route.go:257] In cluster3, podIP of submariner-route-agent[0] is 172.17.0.6
2020-10-08T11:16:19.3492567Z [cluster3] I1008 11:15:47.480022       1 route.go:257] In cluster3, podIP of submariner-route-agent[1] is 172.17.0.5
2020-10-08T11:16:19.3493733Z [cluster3] I1008 11:15:47.480053       1 route.go:271] Hostname is "cluster3-worker2" and routeAgentNodeName is "cluster3-worker2"
2020-10-08T11:16:19.3494933Z [cluster3] I1008 11:15:47.507349       1 driver.go:87] Successfully annotated node "cluster3-worker2" with cniIfaceIP "10.243.128.0"
2020-10-08T11:16:19.3495629Z [cluster3] I1008 11:15:47.507722       1 route.go:288] Route agent workers started
2020-10-08T11:16:19.3496813Z [cluster3] I1008 11:15:47.527028       1 route.go:540] Processing submariner-route-agent pod "submariner-routeagent-596j7" with host IP "172.17.0.6", pod IP "172.17.0.6"
2020-10-08T11:16:19.3498330Z [cluster3] I1008 11:15:47.527116       1 route.go:568] Successfully processed submariner-route-agent pod "submariner-routeagent-596j7"
2020-10-08T11:16:19.3499884Z [cluster3] I1008 11:15:47.537296       1 route.go:540] Processing submariner-route-agent pod "submariner-routeagent-jll6h" with host IP "172.17.0.5", pod IP "172.17.0.5"
2020-10-08T11:16:19.3502447Z [cluster3] I1008 11:15:47.537434       1 route.go:568] Successfully processed submariner-route-agent pod "submariner-routeagent-jll6h"
2020-10-08T11:16:19.3507160Z [cluster3] I1008 11:15:48.557390       1 route.go:736] Enqueueing endpoint for route controller {"metadata":{"name":"cluster3-submariner-cable-cluster3-172-17-0-5","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster3-submariner-cable-cluster3-172-17-0-5","uid":"bcbc8629-05e5-4a7c-b55c-840e9e8463a6","resourceVersion":"1696","generation":1,"creationTimestamp":"2020-10-08T11:15:48Z"},"spec":{"cluster_id":"cluster3","cable_name":"submariner-cable-cluster3-172-17-0-5","hostname":"cluster3-worker","subnets":["100.93.0.0/16","10.243.0.0/16"],"private_ip":"172.17.0.5","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
2020-10-08T11:16:19.3511484Z [cluster3] I1008 11:15:48.605588       1 route.go:634] Processing local Endpoint cluster3-submariner-cable-cluster3-172-17-0-5 with IP 172.17.0.5, Host cluster3-worker for this cluster
2020-10-08T11:16:19.3513664Z [cluster3] I1008 11:15:48.605651       1 route.go:677] This route agent is running on a non-gateway/non-active node
2020-10-08T11:16:19.3514677Z [cluster3] I1008 11:15:48.605661       1 route.go:689] Creating the vxlan interface vx-submariner with gateway node IP 172.17.0.5
2020-10-08T11:16:19.3515329Z [cluster3] I1008 11:15:48.611253       1 route.go:866] Reconciling routes to gw: 240.17.0.5
2020-10-08T11:16:19.3515932Z [cluster3] I1008 11:15:48.611602       1 route.go:882] Processing route {Ifindex: 14 Dst: 240.0.0.0/8 Src: 240.17.0.6 Gw: <nil> Flags: [] Table: 254}
2020-10-08T11:16:19.3516700Z [cluster3] I1008 11:15:48.611975       1 route.go:885] Found nil gw or dst
2020-10-08T11:16:19.3518047Z [cluster3] I1008 11:15:48.612198       1 route.go:715] Successfully processed local Endpoint "cluster3-submariner-cable-cluster3-172-17-0-5"
2020-10-08T11:16:19.3522634Z [cluster3] I1008 11:15:48.673967       1 route.go:736] Enqueueing endpoint for route controller {"metadata":{"name":"cluster2-submariner-cable-cluster2-172-17-0-8","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster2-submariner-cable-cluster2-172-17-0-8","uid":"ea0afa4a-05a3-4175-b625-82869baf0e4e","resourceVersion":"1697","generation":1,"creationTimestamp":"2020-10-08T11:15:48Z"},"spec":{"cluster_id":"cluster2","cable_name":"submariner-cable-cluster2-172-17-0-8","hostname":"cluster2-worker","subnets":["100.92.0.0/16","10.242.0.0/16"],"private_ip":"172.17.0.8","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
2020-10-08T11:16:19.3526808Z [cluster3] I1008 11:15:48.756128       1 route.go:620] Updating iptable rules for remote cluster's Endpoint "cluster2-submariner-cable-cluster2-172-17-0-8" with subnets [100.92.0.0/16 10.242.0.0/16]
2020-10-08T11:16:19.3528361Z [cluster3] I1008 11:15:48.769669       1 iptables.go:83] Installing iptables rule for outgoing traffic: -s 10.243.0.0/16 -d 100.92.0.0/16 -j ACCEPT
2020-10-08T11:16:19.3529443Z [cluster3] I1008 11:15:48.858938       1 iptables.go:91] Installing iptables rule for incoming traffic: -s 100.92.0.0/16 -d 10.243.0.0/16 -j ACCEPT
2020-10-08T11:16:19.3530467Z [cluster3] I1008 11:15:48.917254       1 iptables.go:83] Installing iptables rule for outgoing traffic: -s 10.243.0.0/16 -d 10.242.0.0/16 -j ACCEPT
2020-10-08T11:16:19.3531489Z [cluster3] I1008 11:15:48.969159       1 iptables.go:91] Installing iptables rule for incoming traffic: -s 10.242.0.0/16 -d 10.243.0.0/16 -j ACCEPT
2020-10-08T11:16:19.3539284Z [cluster3] I1008 11:16:17.160722       1 route.go:736] Enqueueing endpoint for route controller {"metadata":{"name":"cluster3-submariner-cable-cluster3-172-17-0-5","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster3-submariner-cable-cluster3-172-17-0-5","uid":"bcbc8629-05e5-4a7c-b55c-840e9e8463a6","resourceVersion":"1696","generation":1,"creationTimestamp":"2020-10-08T11:15:48Z"},"spec":{"cluster_id":"cluster3","cable_name":"submariner-cable-cluster3-172-17-0-5","hostname":"cluster3-worker","subnets":["100.93.0.0/16","10.243.0.0/16"],"private_ip":"172.17.0.5","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
2020-10-08T11:16:19.3611254Z [submariner]$ [cluster3] kubectl -n submariner-operator logs submariner-routeagent-jll6h
2020-10-08T11:16:19.3627558Z [submariner]$ [cluster3] kubectl -n submariner-operator logs submariner-routeagent-jll6h
2020-10-08T11:16:19.3645640Z [submariner]$ [cluster3] command kubectl --context=cluster3 -n submariner-operator logs submariner-routeagent-jll6h
2020-10-08T11:16:19.3675167Z [submariner]$ [cluster3] kubectl --context=cluster3 -n submariner-operator logs submariner-routeagent-jll6h
2020-10-08T11:16:20.4216261Z [submariner]$ [cluster3] subctl show all
2020-10-08T11:16:20.4220572Z [cluster3] I1008 11:16:17.160794       1 route.go:736] Enqueueing endpoint for route controller {"metadata":{"name":"cluster2-submariner-cable-cluster2-172-17-0-8","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster2-submariner-cable-cluster2-172-17-0-8","uid":"ea0afa4a-05a3-4175-b625-82869baf0e4e","resourceVersion":"1697","generation":1,"creationTimestamp":"2020-10-08T11:15:48Z"},"spec":{"cluster_id":"cluster2","cable_name":"submariner-cable-cluster2-172-17-0-8","hostname":"cluster2-worker","subnets":["100.92.0.0/16","10.242.0.0/16"],"private_ip":"172.17.0.8","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
2020-10-08T11:16:20.4224891Z Error obtaining the Submariner resource: the server could not find the requested resource (get submariners.submariner.io submariner)
2020-10-08T11:16:20.4225571Z
2020-10-08T11:16:20.4226269Z subctl version: v0.7.0-pre1-30-g7a9e04d
2020-10-08T11:16:20.4226549Z
2020-10-08T11:16:20.4227032Z [submariner]$ make post-mortem
2020-10-08T11:16:20.4228542Z [cluster3] I1008 11:16:17.173280       1 route.go:634] Processing local Endpoint cluster3-submariner-cable-cluster3-172-17-0-5 with IP 172.17.0.5, Host cluster3-worker for this cluster
2020-10-08T11:16:20.4230045Z [cluster3] I1008 11:16:17.173313       1 route.go:677] This route agent is running on a non-gateway/non-active node
2020-10-08T11:16:20.4231171Z [cluster3] I1008 11:16:17.173323       1 route.go:689] Creating the vxlan interface vx-submariner with gateway node IP 172.17.0.5
2020-10-08T11:16:20.4231936Z [cluster3] I1008 11:16:17.173518       1 vxlan.go:61] VxLAN interface already exists with same configuration.
2020-10-08T11:16:20.4232569Z [cluster3] I1008 11:16:17.177467       1 route.go:866] Reconciling routes to gw: 240.17.0.5
2020-10-08T11:16:20.4233179Z [cluster3] I1008 11:16:17.177591       1 route.go:882] Processing route {Ifindex: 14 Dst: 240.0.0.0/8 Src: 240.17.0.6 Gw: <nil> Flags: [] Table: 254}
2020-10-08T11:16:20.4233725Z [cluster3] I1008 11:16:17.177614       1 route.go:885] Found nil gw or dst
2020-10-08T11:16:20.4235039Z [cluster3] I1008 11:16:17.177753       1 route.go:715] Successfully processed local Endpoint "cluster3-submariner-cable-cluster3-172-17-0-5"
2020-10-08T11:16:20.4237480Z [cluster3] I1008 11:16:17.181531       1 route.go:620] Updating iptable rules for remote cluster's Endpoint "cluster2-submariner-cable-cluster2-172-17-0-8" with subnets [100.92.0.0/16 10.242.0.0/16]
2020-10-08T11:16:20.4239257Z [cluster3] +++++++++++++++++++++: Logs for Pod submariner-routeagent-jll6h in namespace submariner-operator :++++++++++++++++++++++
2020-10-08T11:16:20.4240354Z [cluster3] + trap 'exit 1' SIGTERM SIGINT
2020-10-08T11:16:20.4240790Z [cluster3] + SUBMARINER_VERBOSITY=2
2020-10-08T11:16:20.4241368Z [cluster3] + '[' false == true ']'
2020-10-08T11:16:20.4241824Z [cluster3] + DEBUG=-v=2
2020-10-08T11:16:20.4243251Z [cluster3] + for f in iptables-save iptables
2020-10-08T11:16:20.4244323Z [cluster3] ++ find_iptables_on_host iptables-save
2020-10-08T11:16:20.4244994Z [cluster3] ++ chroot /host test -x /usr/sbin/iptables-save
2020-10-08T11:16:20.4245413Z [cluster3] ++ echo /usr/sbin
2020-10-08T11:16:20.4245719Z [cluster3] ++ return
2020-10-08T11:16:20.4246047Z [cluster3] + location=/usr/sbin
2020-10-08T11:16:20.4246577Z [cluster3] + '[' /usr/sbin '!=' unknown ']'
2020-10-08T11:16:20.4247304Z [cluster3] + echo 'iptables-save is present on the host at /usr/sbin/iptables-save'
2020-10-08T11:16:20.4248152Z [cluster3] + sed 's!@@PATH@@!/usr/sbin!' /usr/sbin/iptables-wrapper.in
2020-10-08T11:16:20.4248854Z [cluster3] + for f in iptables-save iptables
2020-10-08T11:16:20.4249290Z [cluster3] ++ find_iptables_on_host iptables
2020-10-08T11:16:20.4250045Z [cluster3] ++ chroot /host test -x /usr/sbin/iptables
2020-10-08T11:16:20.4250859Z [cluster3] iptables-save is present on the host at /usr/sbin/iptables-save
2020-10-08T11:16:20.4251461Z [cluster3] iptables is present on the host at /usr/sbin/iptables
2020-10-08T11:16:20.4251896Z [cluster3] ++ echo /usr/sbin
2020-10-08T11:16:20.4252203Z [cluster3] ++ return
2020-10-08T11:16:20.4252532Z [cluster3] + location=/usr/sbin
2020-10-08T11:16:20.4253065Z [cluster3] + '[' /usr/sbin '!=' unknown ']'
2020-10-08T11:16:20.4253706Z [cluster3] + echo 'iptables is present on the host at /usr/sbin/iptables'
2020-10-08T11:16:20.4254488Z [cluster3] + sed 's!@@PATH@@!/usr/sbin!' /usr/sbin/iptables-wrapper.in
2020-10-08T11:16:20.4255338Z [cluster3] + exec submariner-route-agent -v=2 -alsologtostderr
2020-10-08T11:16:20.4256247Z [cluster3] I1008 11:15:55.743426       1 main.go:47] Starting submariner-route-agent
2020-10-08T11:16:20.4257391Z [cluster3] W1008 11:15:55.743906       1 client_config.go:543] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
2020-10-08T11:16:20.4258463Z [cluster3] I1008 11:15:55.745658       1 route.go:195] Starting Route Controller. ClusterID: cluster3, localClusterCIDR: [10.243.0.0/16], localServiceCIDR: [100.93.0.0/16]
2020-10-08T11:16:20.4259327Z [cluster3] I1008 11:15:55.745691       1 route.go:199] Waiting for Endpoint informer caches to sync.
2020-10-08T11:16:20.4263618Z [cluster3] I1008 11:15:55.768597       1 route.go:736] Enqueueing endpoint for route controller {"kind":"Endpoint","apiVersion":"submariner.io/v1","metadata":{"name":"cluster3-submariner-cable-cluster3-172-17-0-5","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster3-submariner-cable-cluster3-172-17-0-5","uid":"bcbc8629-05e5-4a7c-b55c-840e9e8463a6","resourceVersion":"1696","generation":1,"creationTimestamp":"2020-10-08T11:15:48Z"},"spec":{"cluster_id":"cluster3","cable_name":"submariner-cable-cluster3-172-17-0-5","hostname":"cluster3-worker","subnets":["100.93.0.0/16","10.243.0.0/16"],"private_ip":"172.17.0.5","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
2020-10-08T11:16:20.4270697Z [cluster3] I1008 11:15:55.769440       1 route.go:736] Enqueueing endpoint for route controller {"kind":"Endpoint","apiVersion":"submariner.io/v1","metadata":{"name":"cluster2-submariner-cable-cluster2-172-17-0-8","namespace":"submariner-operator","selfLink":"/apis/submariner.io/v1/namespaces/submariner-operator/endpoints/cluster2-submariner-cable-cluster2-172-17-0-8","uid":"ea0afa4a-05a3-4175-b625-82869baf0e4e","resourceVersion":"1697","generation":1,"creationTimestamp":"2020-10-08T11:15:48Z"},"spec":{"cluster_id":"cluster2","cable_name":"submariner-cable-cluster2-172-17-0-8","hostname":"cluster2-worker","subnets":["100.92.0.0/16","10.242.0.0/16"],"private_ip":"172.17.0.8","public_ip":"","nat_enabled":false,"backend":"strongswan"}}
2020-10-08T11:16:20.4275191Z [cluster3] I1008 11:15:55.777484       1 route.go:751] Enqueueing sm-route-agent-pod event, ip: 172.17.0.6
2020-10-08T11:16:20.4276134Z [cluster3] I1008 11:15:55.777515       1 route.go:751] Enqueueing sm-route-agent-pod event, ip: 172.17.0.5
2020-10-08T11:16:20.4276894Z [cluster3] I1008 11:15:55.846414       1 driver.go:37] Interface "lo" has "127.0.0.1" address
2020-10-08T11:16:20.4277595Z [cluster3] I1008 11:15:55.846570       1 driver.go:37] Interface "weave" has "10.243.0.1" address
2020-10-08T11:16:20.4278234Z [cluster3] I1008 11:15:55.846580       1 driver.go:42] Found CNI Interface "weave" that has IP "10.243.0.1" from ClusterCIDR "10.243.0.0/16"
2020-10-08T11:16:20.4279021Z [cluster3] I1008 11:15:55.846698       1 driver.go:60] Successfully configured rp_filter to loose mode(2) on cniInterface "weave"
2020-10-08T11:16:20.4280181Z [cluster3] I1008 11:15:55.852063       1 iptables.go:22] Install/ensure SUBMARINER-POSTROUTING chain exists
2020-10-08T11:16:20.4281481Z [cluster3] I1008 11:15:55.866797       1 iptables.go:28] Insert SUBMARINER-POSTROUTING rule that has rules for inter-cluster traffic
2020-10-08T11:16:20.4282623Z [cluster3] I1008 11:15:55.875530       1 iptables.go:34] Install/ensure SUBMARINER-INPUT chain exists
2020-10-08T11:16:20.4283619Z [cluster3] I1008 11:15:55.911910       1 iptables.go:45] Allow VxLAN incoming traffic in SUBMARINER-INPUT Chain
2020-10-08T11:16:20.4284717Z [cluster3] I1008 11:15:55.921849       1 iptables.go:53] Insert rule to allow traffic over vx-submariner interface in FORWARDing Chain
2020-10-08T11:16:20.4286648Z [cluster3] I1008 11:15:55.934957       1 iptables.go:65] Installing rule for host network to remote cluster communication: -s 240.0.0.0/8 -o vx-submariner -j SNAT --to 10.243.0.1
2020-10-08T11:16:20.4287843Z [cluster3] I1008 11:15:55.951873       1 iptables.go:83] Installing iptables rule for outgoing traffic: -s 10.243.0.0/16 -d 100.92.0.0/16 -j ACCEPT
2020-10-08T11:16:20.4288871Z [cluster3] I1008 11:15:55.960405       1 iptables.go:91] Installing iptables rule for incoming traffic: -s 100.92.0.0/16 -d 10.243.0.0/16 -j ACCEPT
2020-10-08T11:16:20.4289909Z [cluster3] I1008 11:15:55.976417       1 iptables.go:83] Installing iptables rule for outgoing traffic: -s 10.243.0.0/16 -d 10.242.0.0/16 -j ACCEPT
2020-10-08T11:16:20.4290934Z [cluster3] I1008 11:15:55.985180       1 iptables.go:91] Installing iptables rule for incoming traffic: -s 10.242.0.0/16 -d 10.243.0.0/16 -j ACCEPT
2020-10-08T11:16:20.4291955Z [cluster3] I1008 11:15:55.998434       1 route.go:257] In cluster3, podIP of submariner-route-agent[0] is 172.17.0.6
2020-10-08T11:16:20.4292983Z [cluster3] I1008 11:15:55.998457       1 route.go:257] In cluster3, podIP of submariner-route-agent[1] is 172.17.0.5
2020-10-08T11:16:20.4294114Z [cluster3] I1008 11:15:55.998463       1 route.go:271] Hostname is "cluster3-worker" and routeAgentNodeName is "cluster3-worker"
2020-10-08T11:16:20.4295279Z [cluster3] I1008 11:15:56.016105       1 driver.go:87] Successfully annotated node "cluster3-worker" with cniIfaceIP "10.243.0.1"
2020-10-08T11:16:20.4295969Z [cluster3] I1008 11:15:56.016137       1 route.go:288] Route agent workers started
2020-10-08T11:16:20.4297433Z [cluster3] I1008 11:15:56.027412       1 route.go:634] Processing local Endpoint cluster3-submariner-cable-cluster3-172-17-0-5 with IP 172.17.0.5, Host cluster3-worker for this cluster
2020-10-08T11:16:20.4298900Z [cluster3] I1008 11:15:56.028116       1 route.go:851] Processing route {Ifindex: 10 Dst: <nil> Src: <nil> Gw: 172.17.0.1 Flags: [] Table: 254}
2020-10-08T11:16:20.4299467Z [cluster3] I1008 11:15:56.028319       1 route.go:854] Found nil gw or dst
2020-10-08T11:16:20.4300029Z [cluster3] I1008 11:15:56.028325       1 route.go:851] Processing route {Ifindex: 4 Dst: 10.243.0.0/16 Src: 10.243.0.1 Gw: <nil> Flags: [] Table: 254}
2020-10-08T11:16:20.4300587Z [cluster3] I1008 11:15:56.028339       1 route.go:854] Found nil gw or dst
2020-10-08T11:16:20.4301149Z [cluster3] I1008 11:15:56.028343       1 route.go:851] Processing route {Ifindex: 10 Dst: 172.17.0.0/16 Src: 172.17.0.5 Gw: <nil> Flags: [] Table: 254}
2020-10-08T11:16:20.4301712Z [cluster3] I1008 11:15:56.028351       1 route.go:854] Found nil gw or dst
2020-10-08T11:16:20.4302233Z [cluster3] I1008 11:15:56.028385       1 route.go:645] This route agent is running on the active gateway node
2020-10-08T11:16:20.4303554Z [cluster3] I1008 11:15:56.028396       1 route.go:653] Creating the vxlan interface: vx-submariner on the gateway node
2020-10-08T11:16:20.4304441Z [cluster3] I1008 11:15:56.032073       1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.6 00:00:00:00:00:00
2020-10-08T11:16:20.4305066Z [cluster3] I1008 11:15:56.032336       1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.5 00:00:00:00:00:00
2020-10-08T11:16:20.4306067Z [cluster3] I1008 11:15:56.035407       1 route.go:478] Successfully configured rp_filter to loose mode(2) on vx-submariner
2020-10-08T11:16:20.4307502Z [cluster3] I1008 11:15:56.053396       1 route.go:540] Processing submariner-route-agent pod "submariner-routeagent-596j7" with host IP "172.17.0.6", pod IP "172.17.0.6"
2020-10-08T11:16:20.4308473Z [cluster3] I1008 11:15:56.053760       1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.6 00:00:00:00:00:00
2020-10-08T11:16:20.4309695Z [cluster3] I1008 11:15:56.053913       1 route.go:558] FDB entry added on the Gateway node's vxlan iface for route-agent pod "submariner-routeagent-596j7", IP "172.17.0.6"
2020-10-08T11:16:20.4311169Z [cluster3] I1008 11:15:56.054029       1 route.go:568] Successfully processed submariner-route-agent pod "submariner-routeagent-596j7"
2020-10-08T11:16:20.4312719Z [cluster3] I1008 11:15:56.060898       1 route.go:540] Processing submariner-route-agent pod "submariner-routeagent-jll6h" with host IP "172.17.0.5", pod IP "172.17.0.5"
2020-10-08T11:16:20.4313665Z [cluster3] I1008 11:15:56.061177       1 vxlan.go:159] Successfully added the bridge fdb entry 172.17.0.5 00:00:00:00:00:00
2020-10-08T11:16:20.4314993Z [cluster3] I1008 11:15:56.061192       1 route.go:558] FDB entry added on the Gateway node's vxlan iface for route-agent pod "submariner-routeagent-jll6h", IP "172.17.0.5"
2020-10-08T11:16:20.4316613Z [cluster3] I1008 11:15:56.061199       1 route.go:568] Successfully processed submariner-route-agent pod "submariner-routeagent-jll6h"
2020-10-08T11:16:20.4318497Z [cluster3] I1008 11:15:56.061912       1 route.go:620] Updating iptable rules for remote cluster's Endpoint "cluster2-submariner-cable-cluster2-172-17-0-8" with subnets [100.92.0.0/16 10.242.0.0/16]
2020-10-08T11:16:20.4319530Z [cluster3] ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
2020-10-08T11:16:20.4319818Z [cluster3]
2020-10-08T11:16:20.4320206Z [cluster3] Showing information for cluster "cluster3":
2020-10-08T11:16:20.4320699Z [cluster3]     Discovered network details:
2020-10-08T11:16:20.4321309Z [cluster3]         Network plugin:  weave-net
2020-10-08T11:16:20.4321703Z [cluster3]         Service CIDRs:   [100.93.0.0/16]
2020-10-08T11:16:20.4322066Z [cluster3]         Cluster CIDRs:   [10.243.0.0/16]
2020-10-08T11:16:20.4322379Z [cluster3]
2020-10-08T11:16:20.4322725Z [cluster3] ===================== END Post mortem cluster3 =====================
2020-10-08T11:16:21.0460946Z Post job cleanup.
2020-10-08T11:16:21.5524560Z [command]/usr/bin/git version
2020-10-08T11:16:21.5594725Z git version 2.28.0
2020-10-08T11:16:21.5681673Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand
2020-10-08T11:16:21.5735779Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :
2020-10-08T11:16:21.6146679Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader
2020-10-08T11:16:21.6185007Z http.https://github.com/.extraheader
2020-10-08T11:16:21.6194471Z [command]/usr/bin/git config --local --unset-all http.https://github.com/.extraheader
2020-10-08T11:16:21.6239791Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :
2020-10-08T11:16:21.6686396Z Cleaning up orphan processes