API tools faq
paste
Advertisement
wojcieh

master 2 logs when joining the cluster

wojcieh
Jan 13th, 2021
167
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 101.35 KB | None | 0 0
raw download clone embed print report
  1. Jan 13 15:59:40 kubernetes-master2 kubelet[5528]: k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil(0x4a721f0, 0x12a05f200, 0x0, 0xc000aec701, 0xc0000a80c0)
  2. Jan 13 15:59:40 kubernetes-master2 kubelet[5528]: #011/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0x98
  3. Jan 13 15:59:40 kubernetes-master2 kubelet[5528]: k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.Until(...)
  4. Jan 13 15:59:40 kubernetes-master2 kubelet[5528]: #011/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:90
  5. Jan 13 15:59:40 kubernetes-master2 kubelet[5528]: k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.Forever(0x4a721f0, 0x12a05f200)
  6. Jan 13 15:59:40 kubernetes-master2 kubelet[5528]: #011/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:81 +0x4f
  7. Jan 13 15:59:40 kubernetes-master2 kubelet[5528]: created by k8s.io/kubernetes/vendor/k8s.io/component-base/logs.InitLogs
  8. Jan 13 15:59:40 kubernetes-master2 kubelet[5528]: #011/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/component-base/logs/logs.go:58 +0x8a
  9. Jan 13 15:59:40 kubernetes-master2 systemd[1]: kubelet.service: Main process exited, code=exited, status=255/EXCEPTION
  10. Jan 13 15:59:40 kubernetes-master2 systemd[1]: kubelet.service: Failed with result 'exit-code'.
  11. Jan 13 15:59:50 kubernetes-master2 systemd[1]: kubelet.service: Scheduled restart job, restart counter is at 120.
  12. Jan 13 15:59:50 kubernetes-master2 systemd[1]: Stopped kubelet: The Kubernetes Node Agent.
  13. Jan 13 15:59:50 kubernetes-master2 systemd[1]: Started kubelet: The Kubernetes Node Agent.
  14. Jan 13 15:59:50 kubernetes-master2 kubelet[5562]: F0113 15:59:50.570006 5562 server.go:198] failed to load Kubelet config file /var/lib/kubelet/config.yaml, error failed to read kubelet config file "/var/lib/kubelet/config.yaml", error: open /var/lib/kubelet/config.yaml: no such file or directory
  15. Jan 13 15:59:50 kubernetes-master2 kubelet[5562]: goroutine 1 [running]:
  16. Jan 13 15:59:50 kubernetes-master2 kubelet[5562]: k8s.io/kubernetes/vendor/k8s.io/klog/v2.stacks(0xc000122001, 0xc000136840, 0xfb, 0x14d)
  17. Jan 13 15:59:50 kubernetes-master2 kubelet[5562]: #011/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/klog/v2/klog.go:1026 +0xb9
  18. Jan 13 15:59:50 kubernetes-master2 kubelet[5562]: k8s.io/kubernetes/vendor/k8s.io/klog/v2.(*loggingT).output(0x70c9460, 0xc000000003, 0x0, 0x0, 0xc000170620, 0x6f34162, 0x9, 0xc6, 0x411b00)
  19. Jan 13 15:59:50 kubernetes-master2 kubelet[5562]: #011/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/klog/v2/klog.go:975 +0x19b
  20. Jan 13 15:59:50 kubernetes-master2 kubelet[5562]: k8s.io/kubernetes/vendor/k8s.io/klog/v2.(*loggingT).printDepth(0x70c9460, 0xc000000003, 0x0, 0x0, 0x0, 0x0, 0x1, 0xc00005ee60, 0x1, 0x1)
  21. Jan 13 15:59:50 kubernetes-master2 kubelet[5562]: #011/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/klog/v2/klog.go:732 +0x16f
  22. Jan 13 15:59:50 kubernetes-master2 kubelet[5562]: k8s.io/kubernetes/vendor/k8s.io/klog/v2.(*loggingT).print(...)
  23. Jan 13 15:59:50 kubernetes-master2 kubelet[5562]: #011/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/klog/v2/klog.go:714
  24. Jan 13 15:59:50 kubernetes-master2 kubelet[5562]: k8s.io/kubernetes/vendor/k8s.io/klog/v2.Fatal(...)
  25. Jan 13 15:59:50 kubernetes-master2 kubelet[5562]: #011/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/klog/v2/klog.go:1482
  26. Jan 13 15:59:50 kubernetes-master2 kubelet[5562]: k8s.io/kubernetes/cmd/kubelet/app.NewKubeletCommand.func1(0xc000159b80, 0xc000124010, 0x3, 0x3)
  27. Jan 13 15:59:50 kubernetes-master2 kubelet[5562]: #011/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubelet/app/server.go:198 +0xe5a
  28. Jan 13 15:59:50 kubernetes-master2 kubelet[5562]: k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).execute(0xc000159b80, 0xc000124010, 0x3, 0x3, 0xc000159b80, 0xc000124010)
  29. Jan 13 15:59:50 kubernetes-master2 kubelet[5562]: #011/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:854 +0x2c2
  30. Jan 13 15:59:50 kubernetes-master2 kubelet[5562]: k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0xc000159b80, 0x1659d2aef0e08987, 0x70c9020, 0x409b25)
  31. Jan 13 15:59:50 kubernetes-master2 kubelet[5562]: #011/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:958 +0x375
  32. Jan 13 15:59:50 kubernetes-master2 kubelet[5562]: k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).Execute(...)
  33. Jan 13 15:59:50 kubernetes-master2 kubelet[5562]: #011/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:895
  34. Jan 13 15:59:50 kubernetes-master2 kubelet[5562]: main.main()
  35. Jan 13 15:59:50 kubernetes-master2 kubelet[5562]: #011_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubelet/kubelet.go:41 +0xe5
  36. Jan 13 15:59:50 kubernetes-master2 kubelet[5562]: goroutine 19 [chan receive]:
  37. Jan 13 15:59:50 kubernetes-master2 kubelet[5562]: k8s.io/kubernetes/vendor/k8s.io/klog/v2.(*loggingT).flushDaemon(0x70c9460)
  38. Jan 13 15:59:50 kubernetes-master2 kubelet[5562]: #011/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/klog/v2/klog.go:1169 +0x8b
  39. Jan 13 15:59:50 kubernetes-master2 kubelet[5562]: created by k8s.io/kubernetes/vendor/k8s.io/klog/v2.init.0
  40. Jan 13 15:59:50 kubernetes-master2 kubelet[5562]: #011/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/klog/v2/klog.go:417 +0xdf
  41. Jan 13 15:59:50 kubernetes-master2 kubelet[5562]: goroutine 90 [select]:
  42. Jan 13 15:59:50 kubernetes-master2 kubelet[5562]: k8s.io/kubernetes/vendor/go.opencensus.io/stats/view.(*worker).start(0xc000135400)
  43. Jan 13 15:59:50 kubernetes-master2 kubelet[5562]: #011/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/go.opencensus.io/stats/view/worker.go:154 +0x105
  44. Jan 13 15:59:50 kubernetes-master2 kubelet[5562]: created by k8s.io/kubernetes/vendor/go.opencensus.io/stats/view.init.0
  45. Jan 13 15:59:50 kubernetes-master2 kubelet[5562]: #011/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/go.opencensus.io/stats/view/worker.go:32 +0x57
  46. Jan 13 15:59:50 kubernetes-master2 kubelet[5562]: goroutine 102 [select]:
  47. Jan 13 15:59:50 kubernetes-master2 kubelet[5562]: k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0x4a721f0, 0x4f0bde0, 0xc0002a8000, 0x1, 0xc0001000c0)
  48. Jan 13 15:59:50 kubernetes-master2 kubelet[5562]: #011/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:167 +0x149
  49. Jan 13 15:59:50 kubernetes-master2 kubelet[5562]: k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil(0x4a721f0, 0x12a05f200, 0x0, 0xc0001ee601, 0xc0001000c0)
  50. Jan 13 15:59:50 kubernetes-master2 kubelet[5562]: #011/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0x98
  51. Jan 13 15:59:50 kubernetes-master2 kubelet[5562]: k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.Until(...)
  52. Jan 13 15:59:50 kubernetes-master2 kubelet[5562]: #011/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:90
  53. Jan 13 15:59:50 kubernetes-master2 kubelet[5562]: k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.Forever(0x4a721f0, 0x12a05f200)
  54. Jan 13 15:59:50 kubernetes-master2 kubelet[5562]: #011/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:81 +0x4f
  55. Jan 13 15:59:50 kubernetes-master2 kubelet[5562]: created by k8s.io/kubernetes/vendor/k8s.io/component-base/logs.InitLogs
  56. Jan 13 15:59:50 kubernetes-master2 kubelet[5562]: #011/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/component-base/logs/logs.go:58 +0x8a
  57. Jan 13 15:59:50 kubernetes-master2 systemd[1]: kubelet.service: Main process exited, code=exited, status=255/EXCEPTION
  58. Jan 13 15:59:50 kubernetes-master2 systemd[1]: kubelet.service: Failed with result 'exit-code'.
  59. Jan 13 15:59:55 kubernetes-master2 systemd[1]: Configuration file /lib/systemd/system/kubelet.service is marked world-inaccessible. This has no effect as configuration data is accessible via APIs without restrictions. Proceeding anyway.
  60. Jan 13 15:59:55 kubernetes-master2 systemd[1]: Configuration file /etc/systemd/system/kubelet.service.d/10-kubeadm.conf is marked world-inaccessible. This has no effect as configuration data is accessible via APIs without restrictions. Proceeding anyway.
  61. Jan 13 15:59:55 kubernetes-master2 systemd[1]: Configuration file /lib/systemd/system/kubelet.service is marked world-inaccessible. This has no effect as configuration data is accessible via APIs without restrictions. Proceeding anyway.
  62. Jan 13 15:59:55 kubernetes-master2 systemd[1]: Configuration file /etc/systemd/system/kubelet.service.d/10-kubeadm.conf is marked world-inaccessible. This has no effect as configuration data is accessible via APIs without restrictions. Proceeding anyway.
  63. Jan 13 15:59:59 kubernetes-master2 systemd[1]: Stopped kubelet: The Kubernetes Node Agent.
  64. Jan 13 15:59:59 kubernetes-master2 systemd[1]: Reloading.
  65. Jan 13 15:59:59 kubernetes-master2 systemd[1]: /lib/systemd/system/dbus.socket:5: ListenStream= references a path below legacy directory /var/run/, updating /var/run/dbus/system_bus_socket → /run/dbus/system_bus_socket; please update the unit file accordingly.
  66. Jan 13 15:59:59 kubernetes-master2 systemd[1]: Configuration file /lib/systemd/system/kubelet.service is marked world-inaccessible. This has no effect as configuration data is accessible via APIs without restrictions. Proceeding anyway.
  67. Jan 13 15:59:59 kubernetes-master2 systemd[1]: Configuration file /etc/systemd/system/kubelet.service.d/10-kubeadm.conf is marked world-inaccessible. This has no effect as configuration data is accessible via APIs without restrictions. Proceeding anyway.
  68. Jan 13 15:59:59 kubernetes-master2 systemd[1]: /lib/systemd/system/docker.socket:6: ListenStream= references a path below legacy directory /var/run/, updating /var/run/docker.sock → /run/docker.sock; please update the unit file accordingly.
  69. Jan 13 15:59:59 kubernetes-master2 systemd[1]: Started kubelet: The Kubernetes Node Agent.
  70. Jan 13 16:00:00 kubernetes-master2 systemd[1]: Started Kubernetes systemd probe.
  71. Jan 13 16:00:00 kubernetes-master2 kubelet[5807]: I0113 16:00:00.366078 5807 server.go:416] Version: v1.20.1
  72. Jan 13 16:00:00 kubernetes-master2 kubelet[5807]: I0113 16:00:00.367513 5807 server.go:837] Client rotation is on, will bootstrap in background
  73. Jan 13 16:00:00 kubernetes-master2 systemd[1]: run-rc37265db04a74219b6ea8f59d58af67a.scope: Succeeded.
  74. Jan 13 16:00:00 kubernetes-master2 kubelet[5807]: I0113 16:00:00.382682 5807 dynamic_cafile_content.go:167] Starting client-ca-bundle::/etc/kubernetes/pki/ca.crt
  75. Jan 13 16:00:00 kubernetes-master2 kubelet[5807]: I0113 16:00:00.480789 5807 server.go:645] --cgroups-per-qos enabled, but --cgroup-root was not specified. defaulting to /
  76. Jan 13 16:00:00 kubernetes-master2 kubelet[5807]: I0113 16:00:00.481609 5807 container_manager_linux.go:274] container manager verified user specified cgroup-root exists: []
  77. Jan 13 16:00:00 kubernetes-master2 kubelet[5807]: I0113 16:00:00.481654 5807 container_manager_linux.go:279] Creating Container Manager object based on Node Config: {RuntimeCgroupsName: SystemCgroupsName: KubeletCgroupsName: ContainerRuntime:docker CgroupsPerQOS:true CgroupRoot:/ CgroupDriver:systemd KubeletRootDir:/var/lib/kubelet ProtectKernelDefaults:false NodeAllocatableConfig:{KubeReservedCgroupName: SystemReservedCgroupName: ReservedSystemCPUs: EnforceNodeAllocatable:map[pods:{}] KubeReserved:map[] SystemReserved:map[] HardEvictionThresholds:[{Signal:memory.available Operator:LessThan Value:{Quantity:100Mi Percentage:0} GracePeriod:0s MinReclaim:<nil>} {Signal:nodefs.available Operator:LessThan Value:{Quantity:<nil> Percentage:0.1} GracePeriod:0s MinReclaim:<nil>} {Signal:nodefs.inodesFree Operator:LessThan Value:{Quantity:<nil> Percentage:0.05} GracePeriod:0s MinReclaim:<nil>} {Signal:imagefs.available Operator:LessThan Value:{Quantity:<nil> Percentage:0.15} GracePeriod:0s MinReclaim:<nil>}]} QOSReserved:map[] ExperimentalCPUManagerPolicy:none ExperimentalTopologyManagerScope:container ExperimentalCPUManagerReconcilePeriod:10s ExperimentalPodPidsLimit:-1 EnforceCPULimits:true CPUCFSQuotaPeriod:100ms ExperimentalTopologyManagerPolicy:none}
  78. Jan 13 16:00:00 kubernetes-master2 kubelet[5807]: I0113 16:00:00.482024 5807 topology_manager.go:120] [topologymanager] Creating topology manager with none policy per container scope
  79. Jan 13 16:00:00 kubernetes-master2 kubelet[5807]: I0113 16:00:00.482060 5807 container_manager_linux.go:310] [topologymanager] Initializing Topology Manager with none policy and container-level scope
  80. Jan 13 16:00:00 kubernetes-master2 kubelet[5807]: I0113 16:00:00.482072 5807 container_manager_linux.go:315] Creating device plugin manager: true
  81. Jan 13 16:00:00 kubernetes-master2 kubelet[5807]: W0113 16:00:00.482295 5807 kubelet.go:297] Using dockershim is deprecated, please consider using a full-fledged CRI implementation
  82. Jan 13 16:00:00 kubernetes-master2 kubelet[5807]: I0113 16:00:00.483187 5807 client.go:77] Connecting to docker on unix:///var/run/docker.sock
  83. Jan 13 16:00:00 kubernetes-master2 kubelet[5807]: I0113 16:00:00.483214 5807 client.go:94] Start docker client with request timeout=2m0s
  84. Jan 13 16:00:00 kubernetes-master2 kubelet[5807]: W0113 16:00:00.496745 5807 docker_service.go:559] Hairpin mode set to "promiscuous-bridge" but kubenet is not enabled, falling back to "hairpin-veth"
  85. Jan 13 16:00:00 kubernetes-master2 kubelet[5807]: I0113 16:00:00.496785 5807 docker_service.go:240] Hairpin mode set to "hairpin-veth"
  86. Jan 13 16:00:00 kubernetes-master2 kubelet[5807]: W0113 16:00:00.496906 5807 cni.go:239] Unable to update cni config: no networks found in /etc/cni/net.d
  87. Jan 13 16:00:00 kubernetes-master2 kubelet[5807]: W0113 16:00:00.500393 5807 cni.go:239] Unable to update cni config: no networks found in /etc/cni/net.d
  88. Jan 13 16:00:00 kubernetes-master2 kubelet[5807]: I0113 16:00:00.500776 5807 docker_service.go:255] Docker cri networking managed by cni
  89. Jan 13 16:00:00 kubernetes-master2 kubelet[5807]: W0113 16:00:00.500863 5807 cni.go:239] Unable to update cni config: no networks found in /etc/cni/net.d
  90. Jan 13 16:00:00 kubernetes-master2 kubelet[5807]: I0113 16:00:00.514517 5807 docker_service.go:260] Docker Info: &{ID:ZJPZ:5PX5:H3KS:7XMT:AT4K:IYXO:V2DE:K6Q5:RRDJ:46NY:7ZZV:U2TQ Containers:0 ContainersRunning:0 ContainersPaused:0 ContainersStopped:0 Images:10 Driver:overlay2 DriverStatus:[[Backing Filesystem extfs] [Supports d_type true] [Native Overlay Diff true]] SystemStatus:[] Plugins:{Volume:[local] Network:[bridge host ipvlan macvlan null overlay] Authorization:[] Log:[awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog]} MemoryLimit:true SwapLimit:false KernelMemory:true KernelMemoryTCP:true CPUCfsPeriod:true CPUCfsQuota:true CPUShares:true CPUSet:true PidsLimit:true IPv4Forwarding:true BridgeNfIptables:true BridgeNfIP6tables:true Debug:false NFd:22 OomKillDisable:true NGoroutines:35 SystemTime:2021-01-13T16:00:00.502655257+01:00 LoggingDriver:json-file CgroupDriver:systemd NEventsListener:0 KernelVersion:5.4.0-60-generic OperatingSystem:Ubuntu 20.04.1 LTS OSType:linux Architecture:x86_64 IndexServerAddress:https://index.docker.io/v1/ RegistryConfig:0xc0009988c0 NCPU:2 MemTotal:4127334400 GenericResources:[] DockerRootDir:/var/lib/docker HTTPProxy: HTTPSProxy: NoProxy: Name:kubernetes-master2 Labels:[] ExperimentalBuild:false ServerVersion:19.03.11 ClusterStore: ClusterAdvertise: Runtimes:map[runc:{Path:runc Args:[]}] DefaultRuntime:runc Swarm:{NodeID: NodeAddr: LocalNodeState:inactive ControlAvailable:false Error: RemoteManagers:[] Nodes:0 Managers:0 Cluster:<nil> Warnings:[]} LiveRestoreEnabled:false Isolation: InitBinary:docker-init ContainerdCommit:{ID:7ad184331fa3e55e52b890ea95e65ba581ae3429 Expected:7ad184331fa3e55e52b890ea95e65ba581ae3429} RuncCommit:{ID:dc9208a3303feef5b3839f4323d9beb36df0a9dd Expected:dc9208a3303feef5b3839f4323d9beb36df0a9dd} InitCommit:{ID:fec3683 Expected:fec3683} SecurityOptions:[name=apparmor name=seccomp,profile=default] ProductLicense: Warnings:[WARNING: No swap limit support]}
  91. Jan 13 16:00:00 kubernetes-master2 kubelet[5807]: I0113 16:00:00.514624 5807 docker_service.go:273] Setting cgroupDriver to systemd
  92. Jan 13 16:00:00 kubernetes-master2 kubelet[5807]: I0113 16:00:00.538200 5807 remote_runtime.go:62] parsed scheme: ""
  93. Jan 13 16:00:00 kubernetes-master2 kubelet[5807]: I0113 16:00:00.538237 5807 remote_runtime.go:62] scheme "" not registered, fallback to default scheme
  94. Jan 13 16:00:00 kubernetes-master2 kubelet[5807]: I0113 16:00:00.538398 5807 passthrough.go:48] ccResolverWrapper: sending update to cc: {[{/var/run/dockershim.sock <nil> 0 <nil>}] <nil> <nil>}
  95. Jan 13 16:00:00 kubernetes-master2 kubelet[5807]: I0113 16:00:00.538419 5807 clientconn.go:948] ClientConn switching balancer to "pick_first"
  96. Jan 13 16:00:00 kubernetes-master2 kubelet[5807]: I0113 16:00:00.538486 5807 remote_image.go:50] parsed scheme: ""
  97. Jan 13 16:00:00 kubernetes-master2 kubelet[5807]: I0113 16:00:00.538500 5807 remote_image.go:50] scheme "" not registered, fallback to default scheme
  98. Jan 13 16:00:00 kubernetes-master2 kubelet[5807]: I0113 16:00:00.538517 5807 passthrough.go:48] ccResolverWrapper: sending update to cc: {[{/var/run/dockershim.sock <nil> 0 <nil>}] <nil> <nil>}
  99. Jan 13 16:00:00 kubernetes-master2 kubelet[5807]: I0113 16:00:00.538528 5807 clientconn.go:948] ClientConn switching balancer to "pick_first"
  100. Jan 13 16:00:00 kubernetes-master2 kubelet[5807]: I0113 16:00:00.538575 5807 kubelet.go:262] Adding pod path: /etc/kubernetes/manifests
  101. Jan 13 16:00:00 kubernetes-master2 kubelet[5807]: I0113 16:00:00.538628 5807 kubelet.go:273] Watching apiserver
  102. Jan 13 16:00:00 kubernetes-master2 kubelet[5807]: I0113 16:00:00.562210 5807 kuberuntime_manager.go:216] Container runtime docker initialized, version: 19.03.11, apiVersion: 1.40.0
  103. Jan 13 16:00:05 kubernetes-master2 kubelet[5807]: W0113 16:00:05.501629 5807 cni.go:239] Unable to update cni config: no networks found in /etc/cni/net.d
  104. Jan 13 16:00:06 kubernetes-master2 kubelet[5807]: E0113 16:00:06.844564 5807 aws_credentials.go:77] while getting AWS credentials NoCredentialProviders: no valid providers in chain. Deprecated.
  105. Jan 13 16:00:06 kubernetes-master2 kubelet[5807]: #011For verbose messaging see aws.Config.CredentialsChainVerboseErrors
  106. Jan 13 16:00:06 kubernetes-master2 kubelet[5807]: I0113 16:00:06.849755 5807 server.go:1176] Started kubelet
  107. Jan 13 16:00:06 kubernetes-master2 kubelet[5807]: E0113 16:00:06.851695 5807 kubelet.go:1271] Image garbage collection failed once. Stats initialization may not have completed yet: failed to get imageFs info: unable to find data in memory cache
  108. Jan 13 16:00:06 kubernetes-master2 kubelet[5807]: I0113 16:00:06.853761 5807 fs_resource_analyzer.go:64] Starting FS ResourceAnalyzer
  109. Jan 13 16:00:06 kubernetes-master2 kubelet[5807]: I0113 16:00:06.855311 5807 server.go:148] Starting to listen on 0.0.0.0:10250
  110. Jan 13 16:00:06 kubernetes-master2 kubelet[5807]: I0113 16:00:06.857684 5807 server.go:409] Adding debug handlers to kubelet server.
  111. Jan 13 16:00:06 kubernetes-master2 kubelet[5807]: I0113 16:00:06.891237 5807 volume_manager.go:271] Starting Kubelet Volume Manager
  112. Jan 13 16:00:06 kubernetes-master2 kubelet[5807]: I0113 16:00:06.893980 5807 desired_state_of_world_populator.go:142] Desired state populator starts to run
  113. Jan 13 16:00:06 kubernetes-master2 kubelet[5807]: E0113 16:00:06.908308 5807 nodelease.go:49] failed to get node "kubernetes-master2" when trying to set owner ref to the node lease: nodes "kubernetes-master2" not found
  114. Jan 13 16:00:06 kubernetes-master2 kubelet[5807]: E0113 16:00:06.912378 5807 kubelet.go:2160] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
  115. Jan 13 16:00:06 kubernetes-master2 kubelet[5807]: E0113 16:00:06.929861 5807 remote_runtime.go:332] ContainerStatus "76151f13dc95e6b282fbd0ec5326900a2ab16f13ae5b2243481ff5c480becabe" from runtime service failed: rpc error: code = Unknown desc = Error: No such container: 76151f13dc95e6b282fbd0ec5326900a2ab16f13ae5b2243481ff5c480becabe
  116. Jan 13 16:00:06 kubernetes-master2 kubelet[5807]: I0113 16:00:06.929909 5807 kuberuntime_gc.go:347] Error getting ContainerStatus for containerID "76151f13dc95e6b282fbd0ec5326900a2ab16f13ae5b2243481ff5c480becabe": rpc error: code = Unknown desc = Error: No such container: 76151f13dc95e6b282fbd0ec5326900a2ab16f13ae5b2243481ff5c480becabe
  117. Jan 13 16:00:06 kubernetes-master2 kubelet[5807]: E0113 16:00:06.931198 5807 remote_runtime.go:332] ContainerStatus "a982dbf20a3ca21c4331784e1791205b65b4e96e2f0e0f49193a7dd886ad70f5" from runtime service failed: rpc error: code = Unknown desc = Error: No such container: a982dbf20a3ca21c4331784e1791205b65b4e96e2f0e0f49193a7dd886ad70f5
  118. Jan 13 16:00:06 kubernetes-master2 kubelet[5807]: I0113 16:00:06.931242 5807 kuberuntime_gc.go:347] Error getting ContainerStatus for containerID "a982dbf20a3ca21c4331784e1791205b65b4e96e2f0e0f49193a7dd886ad70f5": rpc error: code = Unknown desc = Error: No such container: a982dbf20a3ca21c4331784e1791205b65b4e96e2f0e0f49193a7dd886ad70f5
  119. Jan 13 16:00:06 kubernetes-master2 kubelet[5807]: E0113 16:00:06.933065 5807 remote_runtime.go:332] ContainerStatus "99451bf13179f2f0606901daf2115b159e68bbd545613381c9ec4071a9bef508" from runtime service failed: rpc error: code = Unknown desc = Error: No such container: 99451bf13179f2f0606901daf2115b159e68bbd545613381c9ec4071a9bef508
  120. Jan 13 16:00:06 kubernetes-master2 kubelet[5807]: I0113 16:00:06.933095 5807 kuberuntime_gc.go:347] Error getting ContainerStatus for containerID "99451bf13179f2f0606901daf2115b159e68bbd545613381c9ec4071a9bef508": rpc error: code = Unknown desc = Error: No such container: 99451bf13179f2f0606901daf2115b159e68bbd545613381c9ec4071a9bef508
  121. Jan 13 16:00:06 kubernetes-master2 kubelet[5807]: E0113 16:00:06.934280 5807 remote_runtime.go:332] ContainerStatus "c9279741a943fbc1bcd6047764adb846c87a7c90f87c210883e75576e80b9ff2" from runtime service failed: rpc error: code = Unknown desc = Error: No such container: c9279741a943fbc1bcd6047764adb846c87a7c90f87c210883e75576e80b9ff2
  122. Jan 13 16:00:06 kubernetes-master2 kubelet[5807]: I0113 16:00:06.934514 5807 kuberuntime_gc.go:347] Error getting ContainerStatus for containerID "c9279741a943fbc1bcd6047764adb846c87a7c90f87c210883e75576e80b9ff2": rpc error: code = Unknown desc = Error: No such container: c9279741a943fbc1bcd6047764adb846c87a7c90f87c210883e75576e80b9ff2
  123. Jan 13 16:00:06 kubernetes-master2 kubelet[5807]: E0113 16:00:06.936077 5807 remote_runtime.go:332] ContainerStatus "c22e37144fa563ea5fd8f71620341bcd1d91054eb9fa460f70cd73fcefac285c" from runtime service failed: rpc error: code = Unknown desc = Error: No such container: c22e37144fa563ea5fd8f71620341bcd1d91054eb9fa460f70cd73fcefac285c
  124. Jan 13 16:00:06 kubernetes-master2 kubelet[5807]: I0113 16:00:06.936292 5807 kuberuntime_gc.go:347] Error getting ContainerStatus for containerID "c22e37144fa563ea5fd8f71620341bcd1d91054eb9fa460f70cd73fcefac285c": rpc error: code = Unknown desc = Error: No such container: c22e37144fa563ea5fd8f71620341bcd1d91054eb9fa460f70cd73fcefac285c
  125. Jan 13 16:00:06 kubernetes-master2 kubelet[5807]: E0113 16:00:06.938126 5807 remote_runtime.go:332] ContainerStatus "268c81b52d97ee68c8d8b54160321e2dce8d24d11576f7e59040c0a1b443aeb5" from runtime service failed: rpc error: code = Unknown desc = Error: No such container: 268c81b52d97ee68c8d8b54160321e2dce8d24d11576f7e59040c0a1b443aeb5
  126. Jan 13 16:00:06 kubernetes-master2 kubelet[5807]: I0113 16:00:06.938158 5807 kuberuntime_gc.go:347] Error getting ContainerStatus for containerID "268c81b52d97ee68c8d8b54160321e2dce8d24d11576f7e59040c0a1b443aeb5": rpc error: code = Unknown desc = Error: No such container: 268c81b52d97ee68c8d8b54160321e2dce8d24d11576f7e59040c0a1b443aeb5
  127. Jan 13 16:00:06 kubernetes-master2 kubelet[5807]: E0113 16:00:06.939790 5807 remote_runtime.go:332] ContainerStatus "fe25320a9211b5dedf24a6a94365f2fdfe4db215d2259ff38db4011366184575" from runtime service failed: rpc error: code = Unknown desc = Error: No such container: fe25320a9211b5dedf24a6a94365f2fdfe4db215d2259ff38db4011366184575
  128. Jan 13 16:00:06 kubernetes-master2 kubelet[5807]: I0113 16:00:06.939820 5807 kuberuntime_gc.go:347] Error getting ContainerStatus for containerID "fe25320a9211b5dedf24a6a94365f2fdfe4db215d2259ff38db4011366184575": rpc error: code = Unknown desc = Error: No such container: fe25320a9211b5dedf24a6a94365f2fdfe4db215d2259ff38db4011366184575
  129. Jan 13 16:00:06 kubernetes-master2 kubelet[5807]: E0113 16:00:06.941466 5807 remote_runtime.go:332] ContainerStatus "18a602ac0b152a9aa6f5b009f41b2487534fe696a7701ebb17402bad43148c16" from runtime service failed: rpc error: code = Unknown desc = Error: No such container: 18a602ac0b152a9aa6f5b009f41b2487534fe696a7701ebb17402bad43148c16
  130. Jan 13 16:00:06 kubernetes-master2 kubelet[5807]: I0113 16:00:06.941503 5807 kuberuntime_gc.go:347] Error getting ContainerStatus for containerID "18a602ac0b152a9aa6f5b009f41b2487534fe696a7701ebb17402bad43148c16": rpc error: code = Unknown desc = Error: No such container: 18a602ac0b152a9aa6f5b009f41b2487534fe696a7701ebb17402bad43148c16
  131. Jan 13 16:00:06 kubernetes-master2 kubelet[5807]: E0113 16:00:06.943594 5807 remote_runtime.go:332] ContainerStatus "c0531a94bfa4c33a9fcdcd6ee75139634a538b99187268efdd96d5eb17ec0642" from runtime service failed: rpc error: code = Unknown desc = Error: No such container: c0531a94bfa4c33a9fcdcd6ee75139634a538b99187268efdd96d5eb17ec0642
  132. Jan 13 16:00:06 kubernetes-master2 kubelet[5807]: I0113 16:00:06.943886 5807 kuberuntime_gc.go:347] Error getting ContainerStatus for containerID "c0531a94bfa4c33a9fcdcd6ee75139634a538b99187268efdd96d5eb17ec0642": rpc error: code = Unknown desc = Error: No such container: c0531a94bfa4c33a9fcdcd6ee75139634a538b99187268efdd96d5eb17ec0642
  133. Jan 13 16:00:06 kubernetes-master2 kubelet[5807]: E0113 16:00:06.946259 5807 remote_runtime.go:332] ContainerStatus "6c0de10ba61ea18c9fdfc8d3ee484acae1045d01f7e2ff2af21ece697f75f3b7" from runtime service failed: rpc error: code = Unknown desc = Error: No such container: 6c0de10ba61ea18c9fdfc8d3ee484acae1045d01f7e2ff2af21ece697f75f3b7
  134. Jan 13 16:00:06 kubernetes-master2 kubelet[5807]: I0113 16:00:06.946295 5807 kuberuntime_gc.go:347] Error getting ContainerStatus for containerID "6c0de10ba61ea18c9fdfc8d3ee484acae1045d01f7e2ff2af21ece697f75f3b7": rpc error: code = Unknown desc = Error: No such container: 6c0de10ba61ea18c9fdfc8d3ee484acae1045d01f7e2ff2af21ece697f75f3b7
  135. Jan 13 16:00:06 kubernetes-master2 kubelet[5807]: I0113 16:00:06.948934 5807 client.go:86] parsed scheme: "unix"
  136. Jan 13 16:00:06 kubernetes-master2 kubelet[5807]: I0113 16:00:06.948954 5807 client.go:86] scheme "unix" not registered, fallback to default scheme
  137. Jan 13 16:00:06 kubernetes-master2 kubelet[5807]: I0113 16:00:06.948997 5807 passthrough.go:48] ccResolverWrapper: sending update to cc: {[{unix:///run/containerd/containerd.sock <nil> 0 <nil>}] <nil> <nil>}
  138. Jan 13 16:00:06 kubernetes-master2 kubelet[5807]: I0113 16:00:06.949013 5807 clientconn.go:948] ClientConn switching balancer to "pick_first"
  139. Jan 13 16:00:06 kubernetes-master2 kubelet[5807]: E0113 16:00:06.949420 5807 remote_runtime.go:332] ContainerStatus "f94b5a462538ef0a11101452ff80e8985b1df390260e0d6dff0ce76432dadd6f" from runtime service failed: rpc error: code = Unknown desc = Error: No such container: f94b5a462538ef0a11101452ff80e8985b1df390260e0d6dff0ce76432dadd6f
  140. Jan 13 16:00:06 kubernetes-master2 kubelet[5807]: I0113 16:00:06.949466 5807 kuberuntime_gc.go:347] Error getting ContainerStatus for containerID "f94b5a462538ef0a11101452ff80e8985b1df390260e0d6dff0ce76432dadd6f": rpc error: code = Unknown desc = Error: No such container: f94b5a462538ef0a11101452ff80e8985b1df390260e0d6dff0ce76432dadd6f
  141. Jan 13 16:00:06 kubernetes-master2 kubelet[5807]: E0113 16:00:06.957405 5807 remote_runtime.go:332] ContainerStatus "5e6c4ed48213988b9015ed6f040068fc77d17af5de77098571d973f9f2a1b987" from runtime service failed: rpc error: code = Unknown desc = Error: No such container: 5e6c4ed48213988b9015ed6f040068fc77d17af5de77098571d973f9f2a1b987
  142. Jan 13 16:00:06 kubernetes-master2 kubelet[5807]: I0113 16:00:06.957654 5807 kuberuntime_gc.go:347] Error getting ContainerStatus for containerID "5e6c4ed48213988b9015ed6f040068fc77d17af5de77098571d973f9f2a1b987": rpc error: code = Unknown desc = Error: No such container: 5e6c4ed48213988b9015ed6f040068fc77d17af5de77098571d973f9f2a1b987
  143. Jan 13 16:00:07 kubernetes-master2 kubelet[5807]: E0113 16:00:07.004332 5807 kubelet.go:2240] node "kubernetes-master2" not found
  144. Jan 13 16:00:07 kubernetes-master2 kubelet[5807]: E0113 16:00:07.006434 5807 remote_runtime.go:332] ContainerStatus "93d55c166937abd39e8219db7b4b2c902c817c338d334b51c73adac08dfecb66" from runtime service failed: rpc error: code = Unknown desc = Error: No such container: 93d55c166937abd39e8219db7b4b2c902c817c338d334b51c73adac08dfecb66
  145. Jan 13 16:00:07 kubernetes-master2 kubelet[5807]: I0113 16:00:07.006685 5807 kuberuntime_gc.go:347] Error getting ContainerStatus for containerID "93d55c166937abd39e8219db7b4b2c902c817c338d334b51c73adac08dfecb66": rpc error: code = Unknown desc = Error: No such container: 93d55c166937abd39e8219db7b4b2c902c817c338d334b51c73adac08dfecb66
  146. Jan 13 16:00:07 kubernetes-master2 kubelet[5807]: E0113 16:00:07.011298 5807 remote_runtime.go:332] ContainerStatus "6814593ad10242c567998466d3321c7b13b6a4353f788230593a1a26b3867d58" from runtime service failed: rpc error: code = Unknown desc = Error: No such container: 6814593ad10242c567998466d3321c7b13b6a4353f788230593a1a26b3867d58
  147. Jan 13 16:00:07 kubernetes-master2 kubelet[5807]: I0113 16:00:07.011610 5807 kuberuntime_gc.go:347] Error getting ContainerStatus for containerID "6814593ad10242c567998466d3321c7b13b6a4353f788230593a1a26b3867d58": rpc error: code = Unknown desc = Error: No such container: 6814593ad10242c567998466d3321c7b13b6a4353f788230593a1a26b3867d58
  148. Jan 13 16:00:07 kubernetes-master2 kubelet[5807]: I0113 16:00:07.022255 5807 kubelet_network_linux.go:56] Initialized IPv4 iptables rules.
  149. Jan 13 16:00:07 kubernetes-master2 kubelet[5807]: I0113 16:00:07.022326 5807 status_manager.go:158] Starting to sync pod status with apiserver
  150. Jan 13 16:00:07 kubernetes-master2 kubelet[5807]: I0113 16:00:07.022349 5807 kubelet.go:1799] Starting kubelet main sync loop.
  151. Jan 13 16:00:07 kubernetes-master2 kubelet[5807]: E0113 16:00:07.023536 5807 kubelet.go:1823] skipping pod synchronization - [container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]
  152. Jan 13 16:00:07 kubernetes-master2 kubelet[5807]: E0113 16:00:07.036325 5807 remote_runtime.go:332] ContainerStatus "ca871fd9b8ef36091c321797cef30823e6a60d5a3f35883113b2996acf73c995" from runtime service failed: rpc error: code = Unknown desc = Error: No such container: ca871fd9b8ef36091c321797cef30823e6a60d5a3f35883113b2996acf73c995
  153. Jan 13 16:00:07 kubernetes-master2 kubelet[5807]: I0113 16:00:07.036597 5807 kuberuntime_gc.go:347] Error getting ContainerStatus for containerID "ca871fd9b8ef36091c321797cef30823e6a60d5a3f35883113b2996acf73c995": rpc error: code = Unknown desc = Error: No such container: ca871fd9b8ef36091c321797cef30823e6a60d5a3f35883113b2996acf73c995
  154. Jan 13 16:00:07 kubernetes-master2 kubelet[5807]: I0113 16:00:07.054425 5807 kubelet_node_status.go:71] Attempting to register node kubernetes-master2
  155. Jan 13 16:00:07 kubernetes-master2 kubelet[5807]: I0113 16:00:07.075447 5807 kubelet_node_status.go:74] Successfully registered node kubernetes-master2
  156. Jan 13 16:00:07 kubernetes-master2 kubelet[5807]: I0113 16:00:07.101659 5807 cpu_manager.go:193] [cpumanager] starting with none policy
  157. Jan 13 16:00:07 kubernetes-master2 kubelet[5807]: I0113 16:00:07.101989 5807 cpu_manager.go:194] [cpumanager] reconciling every 10s
  158. Jan 13 16:00:07 kubernetes-master2 kubelet[5807]: I0113 16:00:07.102165 5807 state_mem.go:36] [cpumanager] initializing new in-memory state store
  159. Jan 13 16:00:07 kubernetes-master2 kubelet[5807]: I0113 16:00:07.103608 5807 policy_none.go:43] [cpumanager] none policy: Start
  160. Jan 13 16:00:07 kubernetes-master2 systemd[1]: Created slice libcontainer container kubepods.slice.
  161. Jan 13 16:00:07 kubernetes-master2 kubelet[5807]: E0113 16:00:07.131474 5807 kubelet.go:1823] skipping pod synchronization - container runtime status check may not have completed yet
  162. Jan 13 16:00:07 kubernetes-master2 systemd[1]: Created slice libcontainer container kubepods-burstable.slice.
  163. Jan 13 16:00:07 kubernetes-master2 systemd[1]: Created slice libcontainer container kubepods-besteffort.slice.
  164. Jan 13 16:00:07 kubernetes-master2 kubelet[5807]: W0113 16:00:07.166136 5807 manager.go:594] Failed to retrieve checkpoint for "kubelet_internal_checkpoint": checkpoint is not found
  165. Jan 13 16:00:07 kubernetes-master2 kubelet[5807]: I0113 16:00:07.166955 5807 plugin_manager.go:114] Starting Kubelet Plugin Manager
  166. Jan 13 16:00:07 kubernetes-master2 kubelet[5807]: I0113 16:00:07.331957 5807 topology_manager.go:187] [topologymanager] Topology Admit Handler
  167. Jan 13 16:00:07 kubernetes-master2 kubelet[5807]: I0113 16:00:07.332467 5807 topology_manager.go:187] [topologymanager] Topology Admit Handler
  168. Jan 13 16:00:07 kubernetes-master2 kubelet[5807]: I0113 16:00:07.332558 5807 topology_manager.go:187] [topologymanager] Topology Admit Handler
  169. Jan 13 16:00:07 kubernetes-master2 kubelet[5807]: I0113 16:00:07.332945 5807 topology_manager.go:187] [topologymanager] Topology Admit Handler
  170. Jan 13 16:00:07 kubernetes-master2 kubelet[5807]: I0113 16:00:07.333437 5807 topology_manager.go:187] [topologymanager] Topology Admit Handler
  171. Jan 13 16:00:07 kubernetes-master2 systemd[1]: Created slice libcontainer container kubepods-burstable-pod9be8cb4627e7e5ad4c3f8acabd4b49b3.slice.
  172. Jan 13 16:00:07 kubernetes-master2 systemd[1]: Created slice libcontainer container kubepods-besteffort-pod283b30fc_6fef_442d_96e2_d59031ee6f76.slice.
  173. Jan 13 16:00:07 kubernetes-master2 systemd[1]: Created slice libcontainer container kubepods-burstable-pod7a2e5860052277dd81c04d93d6d3fde5.slice.
  174. Jan 13 16:00:07 kubernetes-master2 systemd[1]: Created slice libcontainer container kubepods-burstable-podc61f75a63a6b7c302751a6cc76c53045.slice.
  175. Jan 13 16:00:07 kubernetes-master2 systemd[1]: Created slice libcontainer container kubepods-burstable-pod1880486d_a3bc_4ebc_bd32_219d7f80429d.slice.
  176. Jan 13 16:00:07 kubernetes-master2 kubelet[5807]: I0113 16:00:07.436615 5807 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "ca-certs" (UniqueName: "kubernetes.io/host-path/c61f75a63a6b7c302751a6cc76c53045-ca-certs") pod "kube-controller-manager-kubernetes-master2" (UID: "c61f75a63a6b7c302751a6cc76c53045")
  177. Jan 13 16:00:07 kubernetes-master2 kubelet[5807]: I0113 16:00:07.461805 5807 topology_manager.go:187] [topologymanager] Topology Admit Handler
  178. Jan 13 16:00:07 kubernetes-master2 kubelet[5807]: W0113 16:00:07.461964 5807 predicate.go:113] Failed to admit pod etcd-kubernetes-master2_kube-system(2850093eb023d5a9a20d573fb918b641) - Unexpected error while attempting to recover from admission failure: preemption: error finding a set of pods to preempt: no set of running pods found to reclaim resources: [(res: ephemeral-storage, q: 104857600), ]
  179. Jan 13 16:00:07 kubernetes-master2 kubelet[5807]: I0113 16:00:07.536855 5807 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "var-lib-calico" (UniqueName: "kubernetes.io/host-path/1880486d-a3bc-4ebc-bd32-219d7f80429d-var-lib-calico") pod "calico-node-kxvtq" (UID: "1880486d-a3bc-4ebc-bd32-219d7f80429d")
  180. Jan 13 16:00:07 kubernetes-master2 kubelet[5807]: I0113 16:00:07.537142 5807 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "cni-bin-dir" (UniqueName: "kubernetes.io/host-path/1880486d-a3bc-4ebc-bd32-219d7f80429d-cni-bin-dir") pod "calico-node-kxvtq" (UID: "1880486d-a3bc-4ebc-bd32-219d7f80429d")
  181. Jan 13 16:00:07 kubernetes-master2 kubelet[5807]: I0113 16:00:07.537312 5807 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "host-local-net-dir" (UniqueName: "kubernetes.io/host-path/1880486d-a3bc-4ebc-bd32-219d7f80429d-host-local-net-dir") pod "calico-node-kxvtq" (UID: "1880486d-a3bc-4ebc-bd32-219d7f80429d")
  182. Jan 13 16:00:07 kubernetes-master2 kubelet[5807]: I0113 16:00:07.537614 5807 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "calico-node-token-4725q" (UniqueName: "kubernetes.io/secret/1880486d-a3bc-4ebc-bd32-219d7f80429d-calico-node-token-4725q") pod "calico-node-kxvtq" (UID: "1880486d-a3bc-4ebc-bd32-219d7f80429d")
  183. Jan 13 16:00:07 kubernetes-master2 kubelet[5807]: I0113 16:00:07.537782 5807 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "k8s-certs" (UniqueName: "kubernetes.io/host-path/7a2e5860052277dd81c04d93d6d3fde5-k8s-certs") pod "kube-apiserver-kubernetes-master2" (UID: "7a2e5860052277dd81c04d93d6d3fde5")
  184. Jan 13 16:00:07 kubernetes-master2 kubelet[5807]: I0113 16:00:07.537951 5807 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "etc-pki" (UniqueName: "kubernetes.io/host-path/c61f75a63a6b7c302751a6cc76c53045-etc-pki") pod "kube-controller-manager-kubernetes-master2" (UID: "c61f75a63a6b7c302751a6cc76c53045")
  185. Jan 13 16:00:07 kubernetes-master2 kubelet[5807]: I0113 16:00:07.537977 5807 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "var-run-calico" (UniqueName: "kubernetes.io/host-path/1880486d-a3bc-4ebc-bd32-219d7f80429d-var-run-calico") pod "calico-node-kxvtq" (UID: "1880486d-a3bc-4ebc-bd32-219d7f80429d")
  186. Jan 13 16:00:07 kubernetes-master2 kubelet[5807]: I0113 16:00:07.538033 5807 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "cni-net-dir" (UniqueName: "kubernetes.io/host-path/1880486d-a3bc-4ebc-bd32-219d7f80429d-cni-net-dir") pod "calico-node-kxvtq" (UID: "1880486d-a3bc-4ebc-bd32-219d7f80429d")
  187. Jan 13 16:00:07 kubernetes-master2 kubelet[5807]: I0113 16:00:07.538054 5807 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "cni-log-dir" (UniqueName: "kubernetes.io/host-path/1880486d-a3bc-4ebc-bd32-219d7f80429d-cni-log-dir") pod "calico-node-kxvtq" (UID: "1880486d-a3bc-4ebc-bd32-219d7f80429d")
  188. Jan 13 16:00:07 kubernetes-master2 kubelet[5807]: I0113 16:00:07.538078 5807 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "k8s-certs" (UniqueName: "kubernetes.io/host-path/c61f75a63a6b7c302751a6cc76c53045-k8s-certs") pod "kube-controller-manager-kubernetes-master2" (UID: "c61f75a63a6b7c302751a6cc76c53045")
  189. Jan 13 16:00:07 kubernetes-master2 kubelet[5807]: I0113 16:00:07.538125 5807 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "kube-proxy" (UniqueName: "kubernetes.io/configmap/283b30fc-6fef-442d-96e2-d59031ee6f76-kube-proxy") pod "kube-proxy-c4rb5" (UID: "283b30fc-6fef-442d-96e2-d59031ee6f76")
  190. Jan 13 16:00:07 kubernetes-master2 kubelet[5807]: I0113 16:00:07.538164 5807 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "usr-local-share-ca-certificates" (UniqueName: "kubernetes.io/host-path/7a2e5860052277dd81c04d93d6d3fde5-usr-local-share-ca-certificates") pod "kube-apiserver-kubernetes-master2" (UID: "7a2e5860052277dd81c04d93d6d3fde5")
  191. Jan 13 16:00:07 kubernetes-master2 kubelet[5807]: I0113 16:00:07.538236 5807 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "xtables-lock" (UniqueName: "kubernetes.io/host-path/1880486d-a3bc-4ebc-bd32-219d7f80429d-xtables-lock") pod "calico-node-kxvtq" (UID: "1880486d-a3bc-4ebc-bd32-219d7f80429d")
  192. Jan 13 16:00:07 kubernetes-master2 kubelet[5807]: I0113 16:00:07.538350 5807 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "flexvol-driver-host" (UniqueName: "kubernetes.io/host-path/1880486d-a3bc-4ebc-bd32-219d7f80429d-flexvol-driver-host") pod "calico-node-kxvtq" (UID: "1880486d-a3bc-4ebc-bd32-219d7f80429d")
  193. Jan 13 16:00:07 kubernetes-master2 kubelet[5807]: I0113 16:00:07.538436 5807 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "ca-certs" (UniqueName: "kubernetes.io/host-path/7a2e5860052277dd81c04d93d6d3fde5-ca-certs") pod "kube-apiserver-kubernetes-master2" (UID: "7a2e5860052277dd81c04d93d6d3fde5")
  194. Jan 13 16:00:07 kubernetes-master2 kubelet[5807]: I0113 16:00:07.538576 5807 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "etc-ca-certificates" (UniqueName: "kubernetes.io/host-path/c61f75a63a6b7c302751a6cc76c53045-etc-ca-certificates") pod "kube-controller-manager-kubernetes-master2" (UID: "c61f75a63a6b7c302751a6cc76c53045")
  195. Jan 13 16:00:07 kubernetes-master2 kubelet[5807]: I0113 16:00:07.538630 5807 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "flexvolume-dir" (UniqueName: "kubernetes.io/host-path/c61f75a63a6b7c302751a6cc76c53045-flexvolume-dir") pod "kube-controller-manager-kubernetes-master2" (UID: "c61f75a63a6b7c302751a6cc76c53045")
  196. Jan 13 16:00:07 kubernetes-master2 kubelet[5807]: I0113 16:00:07.538685 5807 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "usr-share-ca-certificates" (UniqueName: "kubernetes.io/host-path/c61f75a63a6b7c302751a6cc76c53045-usr-share-ca-certificates") pod "kube-controller-manager-kubernetes-master2" (UID: "c61f75a63a6b7c302751a6cc76c53045")
  197. Jan 13 16:00:07 kubernetes-master2 kubelet[5807]: I0113 16:00:07.538739 5807 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "lib-modules" (UniqueName: "kubernetes.io/host-path/283b30fc-6fef-442d-96e2-d59031ee6f76-lib-modules") pod "kube-proxy-c4rb5" (UID: "283b30fc-6fef-442d-96e2-d59031ee6f76")
  198. Jan 13 16:00:07 kubernetes-master2 kubelet[5807]: I0113 16:00:07.538792 5807 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "lib-modules" (UniqueName: "kubernetes.io/host-path/1880486d-a3bc-4ebc-bd32-219d7f80429d-lib-modules") pod "calico-node-kxvtq" (UID: "1880486d-a3bc-4ebc-bd32-219d7f80429d")
  199. Jan 13 16:00:07 kubernetes-master2 kubelet[5807]: I0113 16:00:07.538843 5807 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "usr-share-ca-certificates" (UniqueName: "kubernetes.io/host-path/7a2e5860052277dd81c04d93d6d3fde5-usr-share-ca-certificates") pod "kube-apiserver-kubernetes-master2" (UID: "7a2e5860052277dd81c04d93d6d3fde5")
  200. Jan 13 16:00:07 kubernetes-master2 kubelet[5807]: I0113 16:00:07.538865 5807 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "xtables-lock" (UniqueName: "kubernetes.io/host-path/283b30fc-6fef-442d-96e2-d59031ee6f76-xtables-lock") pod "kube-proxy-c4rb5" (UID: "283b30fc-6fef-442d-96e2-d59031ee6f76")
  201. Jan 13 16:00:07 kubernetes-master2 kubelet[5807]: I0113 16:00:07.538921 5807 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "usr-local-share-ca-certificates" (UniqueName: "kubernetes.io/host-path/c61f75a63a6b7c302751a6cc76c53045-usr-local-share-ca-certificates") pod "kube-controller-manager-kubernetes-master2" (UID: "c61f75a63a6b7c302751a6cc76c53045")
  202. Jan 13 16:00:07 kubernetes-master2 kubelet[5807]: I0113 16:00:07.538976 5807 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "kubeconfig" (UniqueName: "kubernetes.io/host-path/9be8cb4627e7e5ad4c3f8acabd4b49b3-kubeconfig") pod "kube-scheduler-kubernetes-master2" (UID: "9be8cb4627e7e5ad4c3f8acabd4b49b3")
  203. Jan 13 16:00:07 kubernetes-master2 kubelet[5807]: I0113 16:00:07.539131 5807 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "etc-ca-certificates" (UniqueName: "kubernetes.io/host-path/7a2e5860052277dd81c04d93d6d3fde5-etc-ca-certificates") pod "kube-apiserver-kubernetes-master2" (UID: "7a2e5860052277dd81c04d93d6d3fde5")
  204. Jan 13 16:00:07 kubernetes-master2 kubelet[5807]: I0113 16:00:07.539662 5807 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "etc-pki" (UniqueName: "kubernetes.io/host-path/7a2e5860052277dd81c04d93d6d3fde5-etc-pki") pod "kube-apiserver-kubernetes-master2" (UID: "7a2e5860052277dd81c04d93d6d3fde5")
  205. Jan 13 16:00:07 kubernetes-master2 kubelet[5807]: I0113 16:00:07.539757 5807 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "kube-proxy-token-ccqrj" (UniqueName: "kubernetes.io/secret/283b30fc-6fef-442d-96e2-d59031ee6f76-kube-proxy-token-ccqrj") pod "kube-proxy-c4rb5" (UID: "283b30fc-6fef-442d-96e2-d59031ee6f76")
  206. Jan 13 16:00:07 kubernetes-master2 kubelet[5807]: I0113 16:00:07.539781 5807 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "sysfs" (UniqueName: "kubernetes.io/host-path/1880486d-a3bc-4ebc-bd32-219d7f80429d-sysfs") pod "calico-node-kxvtq" (UID: "1880486d-a3bc-4ebc-bd32-219d7f80429d")
  207. Jan 13 16:00:07 kubernetes-master2 kubelet[5807]: I0113 16:00:07.539799 5807 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "policysync" (UniqueName: "kubernetes.io/host-path/1880486d-a3bc-4ebc-bd32-219d7f80429d-policysync") pod "calico-node-kxvtq" (UID: "1880486d-a3bc-4ebc-bd32-219d7f80429d")
  208. Jan 13 16:00:07 kubernetes-master2 kubelet[5807]: I0113 16:00:07.540017 5807 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "kubeconfig" (UniqueName: "kubernetes.io/host-path/c61f75a63a6b7c302751a6cc76c53045-kubeconfig") pod "kube-controller-manager-kubernetes-master2" (UID: "c61f75a63a6b7c302751a6cc76c53045")
  209. Jan 13 16:00:07 kubernetes-master2 kubelet[5807]: I0113 16:00:07.540085 5807 reconciler.go:157] Reconciler: start to sync state
  210. Jan 13 16:00:07 kubernetes-master2 systemd[1]: var-lib-docker-overlay2-1e90e7b19d87a2e1b37dea5c9b41feca649a083dc16afa348a21a58fbfb577c2\x2dinit-merged.mount: Succeeded.
  211. Jan 13 16:00:07 kubernetes-master2 systemd[1333]: var-lib-docker-overlay2-1e90e7b19d87a2e1b37dea5c9b41feca649a083dc16afa348a21a58fbfb577c2\x2dinit-merged.mount: Succeeded.
  212. Jan 13 16:00:08 kubernetes-master2 systemd[1]: var-lib-docker-overlay2-1e90e7b19d87a2e1b37dea5c9b41feca649a083dc16afa348a21a58fbfb577c2-merged.mount: Succeeded.
  213. Jan 13 16:00:08 kubernetes-master2 systemd[1333]: var-lib-docker-overlay2-1e90e7b19d87a2e1b37dea5c9b41feca649a083dc16afa348a21a58fbfb577c2-merged.mount: Succeeded.
  214. Jan 13 16:00:08 kubernetes-master2 containerd[822]: time="2021-01-13T16:00:08.154613958+01:00" level=info msg="shim containerd-shim started" address="/containerd-shim/moby/93ed11e6504900909acc5b8ce6f9e9e7de3cd305005f33e7a535b3609373a82c/shim.sock" debug=false pid=6051
  215. Jan 13 16:00:08 kubernetes-master2 systemd[1]: Started libcontainer container 93ed11e6504900909acc5b8ce6f9e9e7de3cd305005f33e7a535b3609373a82c.
  216. Jan 13 16:00:08 kubernetes-master2 kernel: [ 1258.969469] cgroup: cgroup: disabling cgroup2 socket matching due to net_prio or net_cls activation
  217. Jan 13 16:00:08 kubernetes-master2 systemd[1]: var-lib-docker-overlay2-629839e26c89ab40c5f4c410cf9454d9aed22c31d3526fed43c96a3560413766\x2dinit-merged.mount: Succeeded.
  218. Jan 13 16:00:08 kubernetes-master2 systemd[1333]: var-lib-docker-overlay2-629839e26c89ab40c5f4c410cf9454d9aed22c31d3526fed43c96a3560413766\x2dinit-merged.mount: Succeeded.
  219. Jan 13 16:00:08 kubernetes-master2 systemd[1333]: var-lib-docker-overlay2-629839e26c89ab40c5f4c410cf9454d9aed22c31d3526fed43c96a3560413766-merged.mount: Succeeded.
  220. Jan 13 16:00:08 kubernetes-master2 systemd[1]: var-lib-docker-overlay2-629839e26c89ab40c5f4c410cf9454d9aed22c31d3526fed43c96a3560413766-merged.mount: Succeeded.
  221. Jan 13 16:00:08 kubernetes-master2 containerd[822]: time="2021-01-13T16:00:08.355445045+01:00" level=info msg="shim containerd-shim started" address="/containerd-shim/moby/f0715a7302b0d7923f26b69b88ff48baaf683eb49801b8a0faa5109f9634f581/shim.sock" debug=false pid=6099
  222. Jan 13 16:00:08 kubernetes-master2 kubelet[5807]: W0113 16:00:08.379083 5807 pod_container_deletor.go:79] Container "93ed11e6504900909acc5b8ce6f9e9e7de3cd305005f33e7a535b3609373a82c" not found in pod's containers
  223. Jan 13 16:00:08 kubernetes-master2 systemd[1]: Started libcontainer container f0715a7302b0d7923f26b69b88ff48baaf683eb49801b8a0faa5109f9634f581.
  224. Jan 13 16:00:08 kubernetes-master2 systemd[1]: var-lib-docker-overlay2-ea5806fb64f1575bd62f7f8f66370732d1b187075720821d35a303b2677344f6\x2dinit-merged.mount: Succeeded.
  225. Jan 13 16:00:08 kubernetes-master2 systemd[1333]: var-lib-docker-overlay2-ea5806fb64f1575bd62f7f8f66370732d1b187075720821d35a303b2677344f6\x2dinit-merged.mount: Succeeded.
  226. Jan 13 16:00:08 kubernetes-master2 systemd[1333]: var-lib-docker-overlay2-ea5806fb64f1575bd62f7f8f66370732d1b187075720821d35a303b2677344f6-merged.mount: Succeeded.
  227. Jan 13 16:00:08 kubernetes-master2 systemd[1]: var-lib-docker-overlay2-ea5806fb64f1575bd62f7f8f66370732d1b187075720821d35a303b2677344f6-merged.mount: Succeeded.
  228. Jan 13 16:00:08 kubernetes-master2 kubelet[5807]: I0113 16:00:08.448483 5807 request.go:655] Throttling request took 1.113788554s, request: GET:https://kubernetes-cluster.homelab01.local:8443/api/v1/namespaces/kube-system/configmaps?fieldSelector=metadata.name%3Dkubernetes-services-endpoint&limit=500&resourceVersion=0
  229. Jan 13 16:00:08 kubernetes-master2 containerd[822]: time="2021-01-13T16:00:08.501895109+01:00" level=info msg="shim containerd-shim started" address="/containerd-shim/moby/a98a6f89297f3e0b38362161c1bd30b37897df3ff8f132f8c2ecc9d95d12ce2b/shim.sock" debug=false pid=6135
  230. Jan 13 16:00:08 kubernetes-master2 systemd[1]: Started libcontainer container a98a6f89297f3e0b38362161c1bd30b37897df3ff8f132f8c2ecc9d95d12ce2b.
  231. Jan 13 16:00:08 kubernetes-master2 systemd[1]: run-docker-runtime\x2drunc-moby-a98a6f89297f3e0b38362161c1bd30b37897df3ff8f132f8c2ecc9d95d12ce2b-runc.5mpw1W.mount: Succeeded.
  232. Jan 13 16:00:08 kubernetes-master2 systemd[1333]: run-docker-runtime\x2drunc-moby-a98a6f89297f3e0b38362161c1bd30b37897df3ff8f132f8c2ecc9d95d12ce2b-runc.5mpw1W.mount: Succeeded.
  233. Jan 13 16:00:08 kubernetes-master2 systemd[1]: var-lib-docker-overlay2-672b3c7e49610437a3b7e677d79f9bd0498e45938074147e36616c1a871c6d79\x2dinit-merged.mount: Succeeded.
  234. Jan 13 16:00:08 kubernetes-master2 systemd[1333]: var-lib-docker-overlay2-672b3c7e49610437a3b7e677d79f9bd0498e45938074147e36616c1a871c6d79\x2dinit-merged.mount: Succeeded.
  235. Jan 13 16:00:08 kubernetes-master2 systemd[1]: var-lib-docker-overlay2-672b3c7e49610437a3b7e677d79f9bd0498e45938074147e36616c1a871c6d79-merged.mount: Succeeded.
  236. Jan 13 16:00:08 kubernetes-master2 systemd[1333]: var-lib-docker-overlay2-672b3c7e49610437a3b7e677d79f9bd0498e45938074147e36616c1a871c6d79-merged.mount: Succeeded.
  237. Jan 13 16:00:08 kubernetes-master2 containerd[822]: time="2021-01-13T16:00:08.723869074+01:00" level=info msg="shim containerd-shim started" address="/containerd-shim/moby/0da0b070081a87c3004d885e32a77177fd4a51bfe6e5e58e93908ec52cd271f9/shim.sock" debug=false pid=6180
  238. Jan 13 16:00:08 kubernetes-master2 systemd[1]: Started libcontainer container 0da0b070081a87c3004d885e32a77177fd4a51bfe6e5e58e93908ec52cd271f9.
  239. Jan 13 16:00:08 kubernetes-master2 kernel: [ 1259.706799] IPVS: Registered protocols (TCP, UDP, SCTP, AH, ESP)
  240. Jan 13 16:00:08 kubernetes-master2 kernel: [ 1259.706821] IPVS: Connection hash table configured (size=4096, memory=64Kbytes)
  241. Jan 13 16:00:08 kubernetes-master2 kernel: [ 1259.706962] IPVS: ipvs loaded.
  242. Jan 13 16:00:08 kubernetes-master2 kernel: [ 1259.713376] IPVS: [rr] scheduler registered.
  243. Jan 13 16:00:08 kubernetes-master2 kernel: [ 1259.718334] IPVS: [wrr] scheduler registered.
  244. Jan 13 16:00:09 kubernetes-master2 kernel: [ 1259.723507] IPVS: [sh] scheduler registered.
  245. Jan 13 16:00:09 kubernetes-master2 systemd[1]: docker-0da0b070081a87c3004d885e32a77177fd4a51bfe6e5e58e93908ec52cd271f9.scope: Succeeded.
  246. Jan 13 16:00:09 kubernetes-master2 containerd[822]: time="2021-01-13T16:00:09.161991978+01:00" level=info msg="shim reaped" id=0da0b070081a87c3004d885e32a77177fd4a51bfe6e5e58e93908ec52cd271f9
  247. Jan 13 16:00:09 kubernetes-master2 dockerd[827]: time="2021-01-13T16:00:09.172102541+01:00" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
  248. Jan 13 16:00:09 kubernetes-master2 systemd[1333]: var-lib-docker-overlay2-672b3c7e49610437a3b7e677d79f9bd0498e45938074147e36616c1a871c6d79-merged.mount: Succeeded.
  249. Jan 13 16:00:09 kubernetes-master2 systemd[1]: var-lib-docker-overlay2-672b3c7e49610437a3b7e677d79f9bd0498e45938074147e36616c1a871c6d79-merged.mount: Succeeded.
  250. Jan 13 16:00:09 kubernetes-master2 systemd[1333]: var-lib-docker-overlay2-f63bce07c6261a07bfa9dc67f9b20d5bce666b462382ed5e7600e2bdf1ddca5d\x2dinit-merged.mount: Succeeded.
  251. Jan 13 16:00:09 kubernetes-master2 systemd[1]: var-lib-docker-overlay2-f63bce07c6261a07bfa9dc67f9b20d5bce666b462382ed5e7600e2bdf1ddca5d\x2dinit-merged.mount: Succeeded.
  252. Jan 13 16:00:09 kubernetes-master2 systemd[1333]: var-lib-docker-overlay2-f63bce07c6261a07bfa9dc67f9b20d5bce666b462382ed5e7600e2bdf1ddca5d-merged.mount: Succeeded.
  253. Jan 13 16:00:09 kubernetes-master2 systemd[1]: var-lib-docker-overlay2-f63bce07c6261a07bfa9dc67f9b20d5bce666b462382ed5e7600e2bdf1ddca5d-merged.mount: Succeeded.
  254. Jan 13 16:00:09 kubernetes-master2 containerd[822]: time="2021-01-13T16:00:09.480011800+01:00" level=info msg="shim containerd-shim started" address="/containerd-shim/moby/944c8e9d2f7090506cd7032acbe6cc32bd82989b342ca003b010b939ae5b6e5b/shim.sock" debug=false pid=6245
  255. Jan 13 16:00:09 kubernetes-master2 systemd[1]: Started libcontainer container 944c8e9d2f7090506cd7032acbe6cc32bd82989b342ca003b010b939ae5b6e5b.
  256. Jan 13 16:00:10 kubernetes-master2 kubelet[5807]: I0113 16:00:10.370609 5807 transport.go:132] certificate rotation detected, shutting down client connections to start using new credentials
  257. Jan 13 16:00:10 kubernetes-master2 kubelet[5807]: E0113 16:00:10.372062 5807 kubelet.go:1635] Failed creating a mirror pod for "kube-controller-manager-kubernetes-master2_kube-system(c61f75a63a6b7c302751a6cc76c53045)": Post "https://kubernetes-cluster.homelab01.local:8443/api/v1/namespaces/kube-system/pods": read tcp 192.168.255.202:54400->192.168.255.200:8443: use of closed network connection
  258. Jan 13 16:00:10 kubernetes-master2 kubelet[5807]: E0113 16:00:10.372883 5807 kubelet.go:1635] Failed creating a mirror pod for "kube-scheduler-kubernetes-master2_kube-system(9be8cb4627e7e5ad4c3f8acabd4b49b3)": Post "https://kubernetes-cluster.homelab01.local:8443/api/v1/namespaces/kube-system/pods": read tcp 192.168.255.202:54400->192.168.255.200:8443: use of closed network connection
  259. Jan 13 16:00:10 kubernetes-master2 kubelet[5807]: E0113 16:00:10.373187 5807 kubelet.go:1635] Failed creating a mirror pod for "kube-apiserver-kubernetes-master2_kube-system(7a2e5860052277dd81c04d93d6d3fde5)": Post "https://kubernetes-cluster.homelab01.local:8443/api/v1/namespaces/kube-system/pods": read tcp 192.168.255.202:54400->192.168.255.200:8443: use of closed network connection
  260. Jan 13 16:00:10 kubernetes-master2 kubelet[5807]: E0113 16:00:10.374382 5807 event.go:273] Unable to write event: '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"etcd-kubernetes-master2.1659d2b2dfc68206", GenerateName:"", Namespace:"kube-system", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, InvolvedObject:v1.ObjectReference{Kind:"Pod", Namespace:"kube-system", Name:"etcd-kubernetes-master2", UID:"2850093eb023d5a9a20d573fb918b641", APIVersion:"v1", ResourceVersion:"", FieldPath:""}, Reason:"UnexpectedAdmissionError", Message:"Unexpected error while attempting to recover from admission failure: preemption: error finding a set of pods to preempt: no set of running pods found to reclaim resources: [(res: ephemeral-storage, q: 104857600), ]", Source:v1.EventSource{Component:"kubelet", Host:"kubernetes-master2"}, FirstTimestamp:v1.Time{Time:time.Time{wall:0xbff7e09ddb899c06, ext:7590531320, loc:(*time.Location)(0x70c9020)}}, LastTimestamp:v1.Time{Time:time.Time{wall:0xbff7e09ddb899c06, ext:7590531320, loc:(*time.Location)(0x70c9020)}}, Count:1, Type:"Warning", EventTime:v1.MicroTime{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, Series:(*v1.EventSeries)(nil), Action:"", Related:(*v1.ObjectReference)(nil), ReportingController:"", ReportingInstance:""}': 'Post "https://kubernetes-cluster.homelab01.local:8443/api/v1/namespaces/kube-system/events": read tcp 192.168.255.202:54400->192.168.255.200:8443: use of closed network connection'(may retry after sleeping)
  261. Jan 13 16:00:10 kubernetes-master2 kubelet[5807]: E0113 16:00:10.374740 5807 csi_plugin.go:293] Failed to initialize CSINode: error updating CSINode annotation: timed out waiting for the condition; caused by: Post "https://kubernetes-cluster.homelab01.local:8443/apis/storage.k8s.io/v1/csinodes": read tcp 192.168.255.202:54400->192.168.255.200:8443: use of closed network connection
  262. Jan 13 16:00:10 kubernetes-master2 systemd[1333]: var-lib-docker-overlay2-d86b97e7d2f2e336e1bd65f08bb7dfe9c3fa74dc6163e36460fd8e26678ac4d2\x2dinit-merged.mount: Succeeded.
  263. Jan 13 16:00:10 kubernetes-master2 systemd[1]: var-lib-docker-overlay2-d86b97e7d2f2e336e1bd65f08bb7dfe9c3fa74dc6163e36460fd8e26678ac4d2\x2dinit-merged.mount: Succeeded.
  264. Jan 13 16:00:10 kubernetes-master2 systemd[1333]: var-lib-docker-overlay2-1027965cda4a243cd992c553ca78e6fbc5bb682b0d378b24d292ca3a66386e51\x2dinit-merged.mount: Succeeded.
  265. Jan 13 16:00:10 kubernetes-master2 systemd[1333]: var-lib-docker-overlay2-d86b97e7d2f2e336e1bd65f08bb7dfe9c3fa74dc6163e36460fd8e26678ac4d2-merged.mount: Succeeded.
  266. Jan 13 16:00:10 kubernetes-master2 systemd[1]: var-lib-docker-overlay2-d86b97e7d2f2e336e1bd65f08bb7dfe9c3fa74dc6163e36460fd8e26678ac4d2-merged.mount: Succeeded.
  267. Jan 13 16:00:10 kubernetes-master2 systemd[1333]: var-lib-docker-overlay2-1027965cda4a243cd992c553ca78e6fbc5bb682b0d378b24d292ca3a66386e51-merged.mount: Succeeded.
  268. Jan 13 16:00:10 kubernetes-master2 kubelet[5807]: W0113 16:00:10.503657 5807 cni.go:239] Unable to update cni config: no networks found in /etc/cni/net.d
  269. Jan 13 16:00:10 kubernetes-master2 systemd[1]: var-lib-docker-overlay2-1027965cda4a243cd992c553ca78e6fbc5bb682b0d378b24d292ca3a66386e51-merged.mount: Succeeded.
  270. Jan 13 16:00:10 kubernetes-master2 systemd[1333]: var-lib-docker-overlay2-e1037800ec2597e559a5cbaa0f9bb7681b8d2ca8e2d8788de584cbd3bbfcb3ac-merged.mount: Succeeded.
  271. Jan 13 16:00:10 kubernetes-master2 systemd[1]: var-lib-docker-overlay2-e1037800ec2597e559a5cbaa0f9bb7681b8d2ca8e2d8788de584cbd3bbfcb3ac-merged.mount: Succeeded.
  272. Jan 13 16:00:10 kubernetes-master2 containerd[822]: time="2021-01-13T16:00:10.695548662+01:00" level=info msg="shim containerd-shim started" address="/containerd-shim/moby/07e398986851fe20a3fc31c48bc4b44d23b84814be2728ccc3fa57a3f1c783cc/shim.sock" debug=false pid=6289
  273. Jan 13 16:00:10 kubernetes-master2 systemd[1]: Started libcontainer container 07e398986851fe20a3fc31c48bc4b44d23b84814be2728ccc3fa57a3f1c783cc.
  274. Jan 13 16:00:10 kubernetes-master2 systemd[1]: docker-944c8e9d2f7090506cd7032acbe6cc32bd82989b342ca003b010b939ae5b6e5b.scope: Succeeded.
  275. Jan 13 16:00:10 kubernetes-master2 containerd[822]: time="2021-01-13T16:00:10.770685641+01:00" level=info msg="shim containerd-shim started" address="/containerd-shim/moby/e2b2c84b58822d1c4887d977f7aea4cb7fdf63a8970fa40d375d0d1fba1586fc/shim.sock" debug=false pid=6313
  276. Jan 13 16:00:10 kubernetes-master2 containerd[822]: time="2021-01-13T16:00:10.774677293+01:00" level=info msg="shim containerd-shim started" address="/containerd-shim/moby/863c58b21e380703c0718a6aa750b1a11391fa894dd24087cd81acf7b39ad9da/shim.sock" debug=false pid=6316
  277. Jan 13 16:00:10 kubernetes-master2 systemd[1]: Started libcontainer container e2b2c84b58822d1c4887d977f7aea4cb7fdf63a8970fa40d375d0d1fba1586fc.
  278. Jan 13 16:00:10 kubernetes-master2 systemd[1]: Started libcontainer container 863c58b21e380703c0718a6aa750b1a11391fa894dd24087cd81acf7b39ad9da.
  279. Jan 13 16:00:11 kubernetes-master2 systemd[1333]: var-lib-docker-overlay2-e196dce50d4211f5f4298037aa92e94efd2ad85f8f5b7a4540367d9bb2485ec0\x2dinit-merged.mount: Succeeded.
  280. Jan 13 16:00:11 kubernetes-master2 systemd[1]: var-lib-docker-overlay2-e196dce50d4211f5f4298037aa92e94efd2ad85f8f5b7a4540367d9bb2485ec0\x2dinit-merged.mount: Succeeded.
  281. Jan 13 16:00:11 kubernetes-master2 systemd[1333]: var-lib-docker-overlay2-e196dce50d4211f5f4298037aa92e94efd2ad85f8f5b7a4540367d9bb2485ec0-merged.mount: Succeeded.
  282. Jan 13 16:00:11 kubernetes-master2 systemd[1]: var-lib-docker-overlay2-e196dce50d4211f5f4298037aa92e94efd2ad85f8f5b7a4540367d9bb2485ec0-merged.mount: Succeeded.
  283. Jan 13 16:00:11 kubernetes-master2 containerd[822]: time="2021-01-13T16:00:11.057553728+01:00" level=info msg="shim reaped" id=944c8e9d2f7090506cd7032acbe6cc32bd82989b342ca003b010b939ae5b6e5b
  284. Jan 13 16:00:11 kubernetes-master2 dockerd[827]: time="2021-01-13T16:00:11.060859855+01:00" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
  285. Jan 13 16:00:11 kubernetes-master2 systemd[1]: var-lib-docker-overlay2-f63bce07c6261a07bfa9dc67f9b20d5bce666b462382ed5e7600e2bdf1ddca5d-merged.mount: Succeeded.
  286. Jan 13 16:00:11 kubernetes-master2 systemd[1333]: var-lib-docker-overlay2-f63bce07c6261a07bfa9dc67f9b20d5bce666b462382ed5e7600e2bdf1ddca5d-merged.mount: Succeeded.
  287. Jan 13 16:00:11 kubernetes-master2 containerd[822]: time="2021-01-13T16:00:11.111263878+01:00" level=info msg="shim containerd-shim started" address="/containerd-shim/moby/053a627745fd71aa3b3860db018ed6c35ad3570ba812a8bb23b834662cc11805/shim.sock" debug=false pid=6408
  288. Jan 13 16:00:11 kubernetes-master2 systemd[1333]: var-lib-docker-overlay2-12422c5325057fcae118f288a331ca30c68f54709bb168bb5277f260674597bc\x2dinit-merged.mount: Succeeded.
  289. Jan 13 16:00:11 kubernetes-master2 systemd[1]: var-lib-docker-overlay2-12422c5325057fcae118f288a331ca30c68f54709bb168bb5277f260674597bc\x2dinit-merged.mount: Succeeded.
  290. Jan 13 16:00:11 kubernetes-master2 systemd[1]: Started libcontainer container 053a627745fd71aa3b3860db018ed6c35ad3570ba812a8bb23b834662cc11805.
  291. Jan 13 16:00:11 kubernetes-master2 systemd[1333]: var-lib-docker-overlay2-12422c5325057fcae118f288a331ca30c68f54709bb168bb5277f260674597bc-merged.mount: Succeeded.
  292. Jan 13 16:00:11 kubernetes-master2 systemd[1]: var-lib-docker-overlay2-12422c5325057fcae118f288a331ca30c68f54709bb168bb5277f260674597bc-merged.mount: Succeeded.
  293. Jan 13 16:00:11 kubernetes-master2 systemd[1]: var-lib-docker-overlay2-8455342db84b6a4aba0515d1643fa6e933367bf33008265151662b4b5b199b08\x2dinit-merged.mount: Succeeded.
  294. Jan 13 16:00:11 kubernetes-master2 systemd[1333]: var-lib-docker-overlay2-8455342db84b6a4aba0515d1643fa6e933367bf33008265151662b4b5b199b08\x2dinit-merged.mount: Succeeded.
  295. Jan 13 16:00:11 kubernetes-master2 containerd[822]: time="2021-01-13T16:00:11.258143766+01:00" level=info msg="shim containerd-shim started" address="/containerd-shim/moby/11f271809bc27bddbbea0a2c7b485cd71989990903d4709c03920ecf3fa1747b/shim.sock" debug=false pid=6442
  296. Jan 13 16:00:11 kubernetes-master2 systemd[1333]: var-lib-docker-overlay2-8455342db84b6a4aba0515d1643fa6e933367bf33008265151662b4b5b199b08-merged.mount: Succeeded.
  297. Jan 13 16:00:11 kubernetes-master2 systemd[1]: var-lib-docker-overlay2-8455342db84b6a4aba0515d1643fa6e933367bf33008265151662b4b5b199b08-merged.mount: Succeeded.
  298. Jan 13 16:00:11 kubernetes-master2 systemd[1]: Started libcontainer container 11f271809bc27bddbbea0a2c7b485cd71989990903d4709c03920ecf3fa1747b.
  299. Jan 13 16:00:11 kubernetes-master2 containerd[822]: time="2021-01-13T16:00:11.317526769+01:00" level=info msg="shim containerd-shim started" address="/containerd-shim/moby/8d98914e6ba98f84651a2cef63cd1bc846f828b5554808adc1d32e7fada3d0db/shim.sock" debug=false pid=6458
  300. Jan 13 16:00:11 kubernetes-master2 systemd[1]: Started libcontainer container 8d98914e6ba98f84651a2cef63cd1bc846f828b5554808adc1d32e7fada3d0db.
  301. Jan 13 16:00:11 kubernetes-master2 systemd[1333]: var-lib-docker-overlay2-f7c7128e565187801897b46ee1879560d745a757fc5787c5f5e7f03dd14d43d8\x2dinit-merged.mount: Succeeded.
  302. Jan 13 16:00:11 kubernetes-master2 systemd[1]: var-lib-docker-overlay2-f7c7128e565187801897b46ee1879560d745a757fc5787c5f5e7f03dd14d43d8\x2dinit-merged.mount: Succeeded.
  303. Jan 13 16:00:11 kubernetes-master2 containerd[822]: time="2021-01-13T16:00:11.638507700+01:00" level=info msg="shim containerd-shim started" address="/containerd-shim/moby/daa0a49e82e38b1bea49327081d08c149ad59bdc8a918a4b9b4729d9a5da8108/shim.sock" debug=false pid=6524
  304. Jan 13 16:00:11 kubernetes-master2 systemd[1]: Started libcontainer container daa0a49e82e38b1bea49327081d08c149ad59bdc8a918a4b9b4729d9a5da8108.
  305. Jan 13 16:00:12 kubernetes-master2 systemd[1]: docker-daa0a49e82e38b1bea49327081d08c149ad59bdc8a918a4b9b4729d9a5da8108.scope: Succeeded.
  306. Jan 13 16:00:12 kubernetes-master2 kubelet[5807]: E0113 16:00:12.236507 5807 kubelet.go:2160] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
  307. Jan 13 16:00:12 kubernetes-master2 containerd[822]: time="2021-01-13T16:00:12.272446260+01:00" level=info msg="shim reaped" id=daa0a49e82e38b1bea49327081d08c149ad59bdc8a918a4b9b4729d9a5da8108
  308. Jan 13 16:00:12 kubernetes-master2 dockerd[827]: time="2021-01-13T16:00:12.281567092+01:00" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
  309. Jan 13 16:00:12 kubernetes-master2 systemd[1]: var-lib-docker-overlay2-f7c7128e565187801897b46ee1879560d745a757fc5787c5f5e7f03dd14d43d8-merged.mount: Succeeded.
  310. Jan 13 16:00:12 kubernetes-master2 systemd[1333]: var-lib-docker-overlay2-f7c7128e565187801897b46ee1879560d745a757fc5787c5f5e7f03dd14d43d8-merged.mount: Succeeded.
  311. Jan 13 16:00:12 kubernetes-master2 systemd[1333]: var-lib-docker-overlay2-f1a4f9920b693e6107b1911fc38e9d83d280247cdf6d6cb2a5be4598229fa168\x2dinit-merged.mount: Succeeded.
  312. Jan 13 16:00:12 kubernetes-master2 containerd[822]: time="2021-01-13T16:00:12.750788166+01:00" level=info msg="shim containerd-shim started" address="/containerd-shim/moby/c864a51b59ad722f1b4f848d477a260105db5f9dfbdd6a085456e49fe83c4086/shim.sock" debug=false pid=6593
  313. Jan 13 16:00:12 kubernetes-master2 systemd[1]: Started libcontainer container c864a51b59ad722f1b4f848d477a260105db5f9dfbdd6a085456e49fe83c4086.
  314. Jan 13 16:00:12 kubernetes-master2 systemd[1333]: run-docker-runtime\x2drunc-moby-c864a51b59ad722f1b4f848d477a260105db5f9dfbdd6a085456e49fe83c4086-runc.sll3gY.mount: Succeeded.
  315. Jan 13 16:00:12 kubernetes-master2 systemd[1]: run-docker-runtime\x2drunc-moby-c864a51b59ad722f1b4f848d477a260105db5f9dfbdd6a085456e49fe83c4086-runc.sll3gY.mount: Succeeded.
  316. Jan 13 16:00:18 kubernetes-master2 kubelet[5807]: E0113 16:00:18.475466 5807 kubelet.go:1635] Failed creating a mirror pod for "kube-scheduler-kubernetes-master2_kube-system(9be8cb4627e7e5ad4c3f8acabd4b49b3)": rpc error: code = Unknown desc = context deadline exceeded
  317. Jan 13 16:00:19 kubernetes-master2 kubelet[5807]: E0113 16:00:19.637399 5807 kubelet.go:1635] Failed creating a mirror pod for "kube-apiserver-kubernetes-master2_kube-system(7a2e5860052277dd81c04d93d6d3fde5)": rpc error: code = Unknown desc = context deadline exceeded
  318. Jan 13 16:00:19 kubernetes-master2 kubelet[5807]: E0113 16:00:19.649618 5807 kubelet.go:1635] Failed creating a mirror pod for "kube-controller-manager-kubernetes-master2_kube-system(c61f75a63a6b7c302751a6cc76c53045)": rpc error: code = Unknown desc = context deadline exceeded
  319. Jan 13 16:00:21 kubernetes-master2 kubelet[5807]: W0113 16:00:21.559234 5807 status_manager.go:550] Failed to get status for pod "kube-scheduler-kubernetes-master2_kube-system(9be8cb4627e7e5ad4c3f8acabd4b49b3)": etcdserver: request timed out
  320. Jan 13 16:00:21 kubernetes-master2 kubelet[5807]: E0113 16:00:21.724436 5807 event.go:264] Server rejected event '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"etcd-kubernetes-master2.1659d2b2dfc68206", GenerateName:"", Namespace:"kube-system", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, InvolvedObject:v1.ObjectReference{Kind:"Pod", Namespace:"kube-system", Name:"etcd-kubernetes-master2", UID:"2850093eb023d5a9a20d573fb918b641", APIVersion:"v1", ResourceVersion:"", FieldPath:""}, Reason:"UnexpectedAdmissionError", Message:"Unexpected error while attempting to recover from admission failure: preemption: error finding a set of pods to preempt: no set of running pods found to reclaim resources: [(res: ephemeral-storage, q: 104857600), ]", Source:v1.EventSource{Component:"kubelet", Host:"kubernetes-master2"}, FirstTimestamp:v1.Time{Time:time.Time{wall:0xbff7e09ddb899c06, ext:7590531320, loc:(*time.Location)(0x70c9020)}}, LastTimestamp:v1.Time{Time:time.Time{wall:0xbff7e09ddb899c06, ext:7590531320, loc:(*time.Location)(0x70c9020)}}, Count:1, Type:"Warning", EventTime:v1.MicroTime{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, Series:(*v1.EventSeries)(nil), Action:"", Related:(*v1.ObjectReference)(nil), ReportingController:"", ReportingInstance:""}': 'rpc error: code = Unknown desc = context deadline exceeded' (will not retry!)
  321. Jan 13 16:00:25 kubernetes-master2 kubelet[5807]: E0113 16:00:25.709957 5807 kubelet.go:1635] Failed creating a mirror pod for "kube-scheduler-kubernetes-master2_kube-system(9be8cb4627e7e5ad4c3f8acabd4b49b3)": rpc error: code = Unknown desc = context deadline exceeded
  322. Jan 13 16:00:26 kubernetes-master2 kubelet[5807]: E0113 16:00:26.716920 5807 kubelet.go:1635] Failed creating a mirror pod for "kube-controller-manager-kubernetes-master2_kube-system(c61f75a63a6b7c302751a6cc76c53045)": rpc error: code = Unknown desc = context deadline exceeded
  323. Jan 13 16:00:26 kubernetes-master2 kubelet[5807]: E0113 16:00:26.716920 5807 kubelet.go:1635] Failed creating a mirror pod for "kube-apiserver-kubernetes-master2_kube-system(7a2e5860052277dd81c04d93d6d3fde5)": rpc error: code = Unknown desc = context deadline exceeded
  324. Jan 13 16:00:26 kubernetes-master2 kubelet[5807]: E0113 16:00:26.909299 5807 controller.go:144] failed to ensure lease exists, will retry in 200ms, error: Get "https://kubernetes-cluster.homelab01.local:8443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/kubernetes-master2?timeout=10s": net/http: request canceled (Client.Timeout exceeded while awaiting headers)
  325. Jan 13 16:00:27 kubernetes-master2 kubelet[5807]: E0113 16:00:27.176489 5807 kubelet_node_status.go:447] Error updating node status, will retry: error getting node "kubernetes-master2": Get "https://kubernetes-cluster.homelab01.local:8443/api/v1/nodes/kubernetes-master2?resourceVersion=0&timeout=10s": net/http: request canceled (Client.Timeout exceeded while awaiting headers)
  326. Jan 13 16:00:28 kubernetes-master2 kubelet[5807]: E0113 16:00:28.735615 5807 event.go:264] Server rejected event '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"kube-proxy-c4rb5.1659d2b316b78303", GenerateName:"", Namespace:"kube-system", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, InvolvedObject:v1.ObjectReference{Kind:"Pod", Namespace:"kube-system", Name:"kube-proxy-c4rb5", UID:"283b30fc-6fef-442d-96e2-d59031ee6f76", APIVersion:"v1", ResourceVersion:"1475", FieldPath:"spec.containers{kube-proxy}"}, Reason:"Pulled", Message:"Container image \"k8s.gcr.io/kube-proxy:v1.20.1\" already present on machine", Source:v1.EventSource{Component:"kubelet", Host:"kubernetes-master2"}, FirstTimestamp:v1.Time{Time:time.Time{wall:0xbff7e09e16dfd303, ext:8512295408, loc:(*time.Location)(0x70c9020)}}, LastTimestamp:v1.Time{Time:time.Time{wall:0xbff7e09e16dfd303, ext:8512295408, loc:(*time.Location)(0x70c9020)}}, Count:1, Type:"Normal", EventTime:v1.MicroTime{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, Series:(*v1.EventSeries)(nil), Action:"", Related:(*v1.ObjectReference)(nil), ReportingController:"", ReportingInstance:""}': 'rpc error: code = Unknown desc = context deadline exceeded' (will not retry!)
  327. Jan 13 16:00:28 kubernetes-master2 systemd[1]: run-docker-runtime\x2drunc-moby-c864a51b59ad722f1b4f848d477a260105db5f9dfbdd6a085456e49fe83c4086-runc.YCYhaT.mount: Succeeded.
  328. Jan 13 16:00:28 kubernetes-master2 systemd[1333]: run-docker-runtime\x2drunc-moby-c864a51b59ad722f1b4f848d477a260105db5f9dfbdd6a085456e49fe83c4086-runc.YCYhaT.mount: Succeeded.
  329. Jan 13 16:00:30 kubernetes-master2 systemd[1333]: run-docker-runtime\x2drunc-moby-c864a51b59ad722f1b4f848d477a260105db5f9dfbdd6a085456e49fe83c4086-runc.XODerj.mount: Succeeded.
  330. Jan 13 16:00:30 kubernetes-master2 systemd[1]: run-docker-runtime\x2drunc-moby-c864a51b59ad722f1b4f848d477a260105db5f9dfbdd6a085456e49fe83c4086-runc.XODerj.mount: Succeeded.
  331. Jan 13 16:00:33 kubernetes-master2 systemd[1]: docker-8d98914e6ba98f84651a2cef63cd1bc846f828b5554808adc1d32e7fada3d0db.scope: Succeeded.
  332. Jan 13 16:00:33 kubernetes-master2 systemd[1]: docker-8d98914e6ba98f84651a2cef63cd1bc846f828b5554808adc1d32e7fada3d0db.scope: Consumed 1.281s CPU time.
  333. Jan 13 16:00:33 kubernetes-master2 containerd[822]: time="2021-01-13T16:00:33.941267206+01:00" level=info msg="shim reaped" id=8d98914e6ba98f84651a2cef63cd1bc846f828b5554808adc1d32e7fada3d0db
  334. Jan 13 16:00:33 kubernetes-master2 systemd[1]: var-lib-docker-overlay2-8455342db84b6a4aba0515d1643fa6e933367bf33008265151662b4b5b199b08-merged.mount: Succeeded.
  335. Jan 13 16:00:33 kubernetes-master2 systemd[1333]: var-lib-docker-overlay2-8455342db84b6a4aba0515d1643fa6e933367bf33008265151662b4b5b199b08-merged.mount: Succeeded.
  336. Jan 13 16:00:33 kubernetes-master2 dockerd[827]: time="2021-01-13T16:00:33.954452872+01:00" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
  337. Jan 13 16:00:35 kubernetes-master2 kubelet[5807]: E0113 16:00:35.546122 5807 kubelet_node_status.go:447] Error updating node status, will retry: error getting node "kubernetes-master2": etcdserver: request timed out
  338. Jan 13 16:00:35 kubernetes-master2 kubelet[5807]: E0113 16:00:35.547901 5807 event.go:273] Unable to write event: '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"kube-proxy-c4rb5.1659d2b31ac65b01", GenerateName:"", Namespace:"kube-system", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, InvolvedObject:v1.ObjectReference{Kind:"Pod", Namespace:"kube-system", Name:"kube-proxy-c4rb5", UID:"283b30fc-6fef-442d-96e2-d59031ee6f76", APIVersion:"v1", ResourceVersion:"1475", FieldPath:"spec.containers{kube-proxy}"}, Reason:"Created", Message:"Created container kube-proxy", Source:v1.EventSource{Component:"kubelet", Host:"kubernetes-master2"}, FirstTimestamp:v1.Time{Time:time.Time{wall:0xbff7e09e1aeeab01, ext:8580377072, loc:(*time.Location)(0x70c9020)}}, LastTimestamp:v1.Time{Time:time.Time{wall:0xbff7e09e1aeeab01, ext:8580377072, loc:(*time.Location)(0x70c9020)}}, Count:1, Type:"Normal", EventTime:v1.MicroTime{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, Series:(*v1.EventSeries)(nil), Action:"", Related:(*v1.ObjectReference)(nil), ReportingController:"", ReportingInstance:""}': 'Post "https://kubernetes-cluster.homelab01.local:8443/api/v1/namespaces/kube-system/events": read tcp 192.168.255.202:54494->192.168.255.200:8443: use of closed network connection'(may retry after sleeping)
  339. Jan 13 16:00:35 kubernetes-master2 kubelet[5807]: E0113 16:00:35.548707 5807 kubelet.go:1635] Failed creating a mirror pod for "kube-apiserver-kubernetes-master2_kube-system(7a2e5860052277dd81c04d93d6d3fde5)": Post "https://kubernetes-cluster.homelab01.local:8443/api/v1/namespaces/kube-system/pods": read tcp 192.168.255.202:54494->192.168.255.200:8443: use of closed network connection
  340. Jan 13 16:00:35 kubernetes-master2 kubelet[5807]: I0113 16:00:35.548808 5807 scope.go:95] [topologymanager] RemoveContainer - Container ID: 8d98914e6ba98f84651a2cef63cd1bc846f828b5554808adc1d32e7fada3d0db
  341. Jan 13 16:00:35 kubernetes-master2 systemd[1333]: var-lib-docker-overlay2-13ea60e47ed27d9e23fcda5a883629494ac7486653fa751bd62bcbae6f3a0a90\x2dinit-merged.mount: Succeeded.
  342. Jan 13 16:00:35 kubernetes-master2 systemd[1]: var-lib-docker-overlay2-13ea60e47ed27d9e23fcda5a883629494ac7486653fa751bd62bcbae6f3a0a90\x2dinit-merged.mount: Succeeded.
  343. Jan 13 16:00:35 kubernetes-master2 systemd[1333]: var-lib-docker-overlay2-13ea60e47ed27d9e23fcda5a883629494ac7486653fa751bd62bcbae6f3a0a90-merged.mount: Succeeded.
  344. Jan 13 16:00:35 kubernetes-master2 systemd[1]: var-lib-docker-overlay2-13ea60e47ed27d9e23fcda5a883629494ac7486653fa751bd62bcbae6f3a0a90-merged.mount: Succeeded.
  345. Jan 13 16:00:35 kubernetes-master2 containerd[822]: time="2021-01-13T16:00:35.606661765+01:00" level=info msg="shim containerd-shim started" address="/containerd-shim/moby/5f1bd972c1320005f51b20080087221f9829e005feb65b82875ae448f8adca07/shim.sock" debug=false pid=6861
  346. Jan 13 16:00:35 kubernetes-master2 systemd[1]: Started libcontainer container 5f1bd972c1320005f51b20080087221f9829e005feb65b82875ae448f8adca07.
  347. Jan 13 16:00:37 kubernetes-master2 kubelet[5807]: E0113 16:00:37.110559 5807 controller.go:144] failed to ensure lease exists, will retry in 400ms, error: Get "https://kubernetes-cluster.homelab01.local:8443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/kubernetes-master2?timeout=10s": context deadline exceeded
  348. Jan 13 16:00:38 kubernetes-master2 systemd[1333]: run-docker-runtime\x2drunc-moby-c864a51b59ad722f1b4f848d477a260105db5f9dfbdd6a085456e49fe83c4086-runc.97h3su.mount: Succeeded.
  349. Jan 13 16:00:38 kubernetes-master2 systemd[1]: run-docker-runtime\x2drunc-moby-c864a51b59ad722f1b4f848d477a260105db5f9dfbdd6a085456e49fe83c4086-runc.97h3su.mount: Succeeded.
  350. Jan 13 16:00:40 kubernetes-master2 systemd[1333]: run-docker-runtime\x2drunc-moby-c864a51b59ad722f1b4f848d477a260105db5f9dfbdd6a085456e49fe83c4086-runc.YujH75.mount: Succeeded.
  351. Jan 13 16:00:40 kubernetes-master2 systemd[1]: run-docker-runtime\x2drunc-moby-c864a51b59ad722f1b4f848d477a260105db5f9dfbdd6a085456e49fe83c4086-runc.YujH75.mount: Succeeded.
  352. Jan 13 16:00:42 kubernetes-master2 kubelet[5807]: E0113 16:00:42.857593 5807 kubelet.go:1635] Failed creating a mirror pod for "kube-apiserver-kubernetes-master2_kube-system(7a2e5860052277dd81c04d93d6d3fde5)": rpc error: code = Unknown desc = context deadline exceeded
  353. Jan 13 16:00:45 kubernetes-master2 kubelet[5807]: E0113 16:00:45.548080 5807 kubelet_node_status.go:447] Error updating node status, will retry: error getting node "kubernetes-master2": Get "https://kubernetes-cluster.homelab01.local:8443/api/v1/nodes/kubernetes-master2?timeout=10s": net/http: request canceled (Client.Timeout exceeded while awaiting headers)
  354. Jan 13 16:00:45 kubernetes-master2 kubelet[5807]: E0113 16:00:45.549666 5807 event.go:273] Unable to write event: '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"kube-proxy-c4rb5.1659d2b31ac65b01", GenerateName:"", Namespace:"kube-system", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, InvolvedObject:v1.ObjectReference{Kind:"Pod", Namespace:"kube-system", Name:"kube-proxy-c4rb5", UID:"283b30fc-6fef-442d-96e2-d59031ee6f76", APIVersion:"v1", ResourceVersion:"1475", FieldPath:"spec.containers{kube-proxy}"}, Reason:"Created", Message:"Created container kube-proxy", Source:v1.EventSource{Component:"kubelet", Host:"kubernetes-master2"}, FirstTimestamp:v1.Time{Time:time.Time{wall:0xbff7e09e1aeeab01, ext:8580377072, loc:(*time.Location)(0x70c9020)}}, LastTimestamp:v1.Time{Time:time.Time{wall:0xbff7e09e1aeeab01, ext:8580377072, loc:(*time.Location)(0x70c9020)}}, Count:1, Type:"Normal", EventTime:v1.MicroTime{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, Series:(*v1.EventSeries)(nil), Action:"", Related:(*v1.ObjectReference)(nil), ReportingController:"", ReportingInstance:""}': 'Post "https://kubernetes-cluster.homelab01.local:8443/api/v1/namespaces/kube-system/events": read tcp 192.168.255.202:54742->192.168.255.200:8443: use of closed network connection'(may retry after sleeping)
  355. Jan 13 16:00:47 kubernetes-master2 kubelet[5807]: E0113 16:00:47.517759 5807 controller.go:144] failed to ensure lease exists, will retry in 800ms, error: Get "https://kubernetes-cluster.homelab01.local:8443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/kubernetes-master2?timeout=10s": context deadline exceeded
  356. Jan 13 16:00:48 kubernetes-master2 systemd[1333]: run-docker-runtime\x2drunc-moby-c864a51b59ad722f1b4f848d477a260105db5f9dfbdd6a085456e49fe83c4086-runc.mToVWT.mount: Succeeded.
  357. Jan 13 16:00:48 kubernetes-master2 systemd[1]: run-docker-runtime\x2drunc-moby-c864a51b59ad722f1b4f848d477a260105db5f9dfbdd6a085456e49fe83c4086-runc.mToVWT.mount: Succeeded.
  358. Jan 13 16:00:55 kubernetes-master2 kubelet[5807]: E0113 16:00:55.548928 5807 kubelet_node_status.go:447] Error updating node status, will retry: error getting node "kubernetes-master2": Get "https://kubernetes-cluster.homelab01.local:8443/api/v1/nodes/kubernetes-master2?timeout=10s": net/http: request canceled (Client.Timeout exceeded while awaiting headers)
  359. Jan 13 16:00:56 kubernetes-master2 systemd[1]: docker-5f1bd972c1320005f51b20080087221f9829e005feb65b82875ae448f8adca07.scope: Succeeded.
  360. Jan 13 16:00:56 kubernetes-master2 containerd[822]: time="2021-01-13T16:00:56.437699852+01:00" level=info msg="shim reaped" id=5f1bd972c1320005f51b20080087221f9829e005feb65b82875ae448f8adca07
  361. Jan 13 16:00:56 kubernetes-master2 dockerd[827]: time="2021-01-13T16:00:56.448355153+01:00" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
  362. Jan 13 16:00:56 kubernetes-master2 systemd[1333]: var-lib-docker-overlay2-13ea60e47ed27d9e23fcda5a883629494ac7486653fa751bd62bcbae6f3a0a90-merged.mount: Succeeded.
  363. Jan 13 16:00:56 kubernetes-master2 systemd[1]: var-lib-docker-overlay2-13ea60e47ed27d9e23fcda5a883629494ac7486653fa751bd62bcbae6f3a0a90-merged.mount: Succeeded.
  364. Jan 13 16:00:56 kubernetes-master2 kubelet[5807]: I0113 16:00:56.982044 5807 scope.go:95] [topologymanager] RemoveContainer - Container ID: 8d98914e6ba98f84651a2cef63cd1bc846f828b5554808adc1d32e7fada3d0db
  365. Jan 13 16:00:58 kubernetes-master2 kubelet[5807]: E0113 16:00:58.319652 5807 controller.go:144] failed to ensure lease exists, will retry in 1.6s, error: Get "https://kubernetes-cluster.homelab01.local:8443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/kubernetes-master2?timeout=10s": context deadline exceeded
  366. Jan 13 16:00:58 kubernetes-master2 systemd[1]: run-docker-runtime\x2drunc-moby-c864a51b59ad722f1b4f848d477a260105db5f9dfbdd6a085456e49fe83c4086-runc.PRPfpy.mount: Succeeded.
  367. Jan 13 16:00:58 kubernetes-master2 systemd[1333]: run-docker-runtime\x2drunc-moby-c864a51b59ad722f1b4f848d477a260105db5f9dfbdd6a085456e49fe83c4086-runc.PRPfpy.mount: Succeeded.
  368. Jan 13 16:01:00 kubernetes-master2 systemd[1333]: run-docker-runtime\x2drunc-moby-c864a51b59ad722f1b4f848d477a260105db5f9dfbdd6a085456e49fe83c4086-runc.9q8ki5.mount: Succeeded.
  369. Jan 13 16:01:00 kubernetes-master2 systemd[1]: run-docker-runtime\x2drunc-moby-c864a51b59ad722f1b4f848d477a260105db5f9dfbdd6a085456e49fe83c4086-runc.9q8ki5.mount: Succeeded.
  370. Jan 13 16:01:02 kubernetes-master2 kubelet[5807]: E0113 16:01:02.611020 5807 event.go:264] Server rejected event '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"kube-proxy-c4rb5.1659d2b31ac65b01", GenerateName:"", Namespace:"kube-system", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, InvolvedObject:v1.ObjectReference{Kind:"Pod", Namespace:"kube-system", Name:"kube-proxy-c4rb5", UID:"283b30fc-6fef-442d-96e2-d59031ee6f76", APIVersion:"v1", ResourceVersion:"1475", FieldPath:"spec.containers{kube-proxy}"}, Reason:"Created", Message:"Created container kube-proxy", Source:v1.EventSource{Component:"kubelet", Host:"kubernetes-master2"}, FirstTimestamp:v1.Time{Time:time.Time{wall:0xbff7e09e1aeeab01, ext:8580377072, loc:(*time.Location)(0x70c9020)}}, LastTimestamp:v1.Time{Time:time.Time{wall:0xbff7e09e1aeeab01, ext:8580377072, loc:(*time.Location)(0x70c9020)}}, Count:1, Type:"Normal", EventTime:v1.MicroTime{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, Series:(*v1.EventSeries)(nil), Action:"", Related:(*v1.ObjectReference)(nil), ReportingController:"", ReportingInstance:""}': 'rpc error: code = Unknown desc = context deadline exceeded' (will not retry!)
  371. Jan 13 16:01:03 kubernetes-master2 kubelet[5807]: E0113 16:01:03.549891 5807 kubelet_node_status.go:447] Error updating node status, will retry: error getting node "kubernetes-master2": etcdserver: request timed out
  372. Jan 13 16:01:03 kubernetes-master2 kubelet[5807]: E0113 16:01:03.550134 5807 kubelet_node_status.go:434] Unable to update node status: update node status exceeds retry count
  373. Jan 13 16:01:03 kubernetes-master2 kubelet[5807]: E0113 16:01:03.551065 5807 kubelet.go:1635] Failed creating a mirror pod for "kube-apiserver-kubernetes-master2_kube-system(7a2e5860052277dd81c04d93d6d3fde5)": Post "https://kubernetes-cluster.homelab01.local:8443/api/v1/namespaces/kube-system/pods": read tcp 192.168.255.202:54970->192.168.255.200:8443: use of closed network connection
  374. Jan 13 16:01:03 kubernetes-master2 kubelet[5807]: E0113 16:01:03.551896 5807 event.go:273] Unable to write event: '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"calico-node-kxvtq.1659d2b321b7a19a", GenerateName:"", Namespace:"kube-system", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, InvolvedObject:v1.ObjectReference{Kind:"Pod", Namespace:"kube-system", Name:"calico-node-kxvtq", UID:"1880486d-a3bc-4ebc-bd32-219d7f80429d", APIVersion:"v1", ResourceVersion:"1476", FieldPath:"spec.initContainers{upgrade-ipam}"}, Reason:"Pulled", Message:"Container image \"docker.io/calico/cni:v3.17.1\" already present on machine", Source:v1.EventSource{Component:"kubelet", Host:"kubernetes-master2"}, FirstTimestamp:v1.Time{Time:time.Time{wall:0xbff7e09e21dff19a, ext:8696852620, loc:(*time.Location)(0x70c9020)}}, LastTimestamp:v1.Time{Time:time.Time{wall:0xbff7e09e21dff19a, ext:8696852620, loc:(*time.Location)(0x70c9020)}}, Count:1, Type:"Normal", EventTime:v1.MicroTime{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, Series:(*v1.EventSeries)(nil), Action:"", Related:(*v1.ObjectReference)(nil), ReportingController:"", ReportingInstance:""}': 'Post "https://kubernetes-cluster.homelab01.local:8443/api/v1/namespaces/kube-system/events": read tcp 192.168.255.202:54970->192.168.255.200:8443: use of closed network connection'(may retry after sleeping)
  375. Jan 13 16:01:03 kubernetes-master2 kubelet[5807]: I0113 16:01:03.553953 5807 scope.go:95] [topologymanager] RemoveContainer - Container ID: 5f1bd972c1320005f51b20080087221f9829e005feb65b82875ae448f8adca07
  376. Jan 13 16:01:03 kubernetes-master2 kubelet[5807]: E0113 16:01:03.555410 5807 pod_workers.go:191] Error syncing pod 7a2e5860052277dd81c04d93d6d3fde5 ("kube-apiserver-kubernetes-master2_kube-system(7a2e5860052277dd81c04d93d6d3fde5)"), skipping: failed to "StartContainer" for "kube-apiserver" with CrashLoopBackOff: "back-off 10s restarting failed container=kube-apiserver pod=kube-apiserver-kubernetes-master2_kube-system(7a2e5860052277dd81c04d93d6d3fde5)"
  377. Jan 13 16:01:08 kubernetes-master2 systemd[1]: run-docker-runtime\x2drunc-moby-c864a51b59ad722f1b4f848d477a260105db5f9dfbdd6a085456e49fe83c4086-runc.hcK1ap.mount: Succeeded.
  378. Jan 13 16:01:08 kubernetes-master2 systemd[1333]: run-docker-runtime\x2drunc-moby-c864a51b59ad722f1b4f848d477a260105db5f9dfbdd6a085456e49fe83c4086-runc.hcK1ap.mount: Succeeded.
  379. Jan 13 16:01:09 kubernetes-master2 kubelet[5807]: E0113 16:01:09.920717 5807 controller.go:144] failed to ensure lease exists, will retry in 3.2s, error: Get "https://kubernetes-cluster.homelab01.local:8443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/kubernetes-master2?timeout=10s": context deadline exceeded
  380. Jan 13 16:01:10 kubernetes-master2 systemd[1333]: run-docker-runtime\x2drunc-moby-c864a51b59ad722f1b4f848d477a260105db5f9dfbdd6a085456e49fe83c4086-runc.6DDPw8.mount: Succeeded.
  381. Jan 13 16:01:10 kubernetes-master2 systemd[1]: run-docker-runtime\x2drunc-moby-c864a51b59ad722f1b4f848d477a260105db5f9dfbdd6a085456e49fe83c4086-runc.6DDPw8.mount: Succeeded.
  382. Jan 13 16:01:16 kubernetes-master2 kubelet[5807]: E0113 16:01:16.956519 5807 event.go:264] Server rejected event '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"calico-node-kxvtq.1659d2b321b7a19a", GenerateName:"", Namespace:"kube-system", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, InvolvedObject:v1.ObjectReference{Kind:"Pod", Namespace:"kube-system", Name:"calico-node-kxvtq", UID:"1880486d-a3bc-4ebc-bd32-219d7f80429d", APIVersion:"v1", ResourceVersion:"1476", FieldPath:"spec.initContainers{upgrade-ipam}"}, Reason:"Pulled", Message:"Container image \"docker.io/calico/cni:v3.17.1\" already present on machine", Source:v1.EventSource{Component:"kubelet", Host:"kubernetes-master2"}, FirstTimestamp:v1.Time{Time:time.Time{wall:0xbff7e09e21dff19a, ext:8696852620, loc:(*time.Location)(0x70c9020)}}, LastTimestamp:v1.Time{Time:time.Time{wall:0xbff7e09e21dff19a, ext:8696852620, loc:(*time.Location)(0x70c9020)}}, Count:1, Type:"Normal", EventTime:v1.MicroTime{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, Series:(*v1.EventSeries)(nil), Action:"", Related:(*v1.ObjectReference)(nil), ReportingController:"", ReportingInstance:""}': 'rpc error: code = Unknown desc = context deadline exceeded' (will not retry!)
  383. Jan 13 16:01:17 kubernetes-master2 kubelet[5807]: W0113 16:01:17.552155 5807 status_manager.go:550] Failed to get status for pod "kube-controller-manager-kubernetes-master2_kube-system(c61f75a63a6b7c302751a6cc76c53045)": etcdserver: request timed out
  384. Jan 13 16:01:17 kubernetes-master2 kubelet[5807]: E0113 16:01:17.564369 5807 kubelet.go:1635] Failed creating a mirror pod for "kube-apiserver-kubernetes-master2_kube-system(7a2e5860052277dd81c04d93d6d3fde5)": pods "kube-apiserver-kubernetes-master2" is forbidden: etcdserver: request timed out
  385. Jan 13 16:01:17 kubernetes-master2 kubelet[5807]: I0113 16:01:17.564578 5807 scope.go:95] [topologymanager] RemoveContainer - Container ID: 5f1bd972c1320005f51b20080087221f9829e005feb65b82875ae448f8adca07
  386. Jan 13 16:01:17 kubernetes-master2 systemd[1]: var-lib-docker-overlay2-9f8aa7176cde50757d3b6289286e730b3b7d4fe8867e931d314097a7fafe8e2f\x2dinit-merged.mount: Succeeded.
  387. Jan 13 16:01:17 kubernetes-master2 systemd[1333]: var-lib-docker-overlay2-9f8aa7176cde50757d3b6289286e730b3b7d4fe8867e931d314097a7fafe8e2f\x2dinit-merged.mount: Succeeded.
  388. Jan 13 16:01:17 kubernetes-master2 systemd[1]: var-lib-docker-overlay2-9f8aa7176cde50757d3b6289286e730b3b7d4fe8867e931d314097a7fafe8e2f-merged.mount: Succeeded.
  389. Jan 13 16:01:17 kubernetes-master2 systemd[1333]: var-lib-docker-overlay2-9f8aa7176cde50757d3b6289286e730b3b7d4fe8867e931d314097a7fafe8e2f-merged.mount: Succeeded.
  390. Jan 13 16:01:17 kubernetes-master2 containerd[822]: time="2021-01-13T16:01:17.640263702+01:00" level=info msg="shim containerd-shim started" address="/containerd-shim/moby/52d0a8623bb30b6f48dee93c6cfb370ab2469da5c2158b091ba46605ec71cd0f/shim.sock" debug=false pid=7414
  391. Jan 13 16:01:17 kubernetes-master2 systemd[1]: Started libcontainer container 52d0a8623bb30b6f48dee93c6cfb370ab2469da5c2158b091ba46605ec71cd0f.
  392. Jan 13 16:01:20 kubernetes-master2 dockerd[827]: time="2021-01-13T16:01:20.872943543+01:00" level=info msg="Container c864a51b59ad722f1b4f848d477a260105db5f9dfbdd6a085456e49fe83c4086 failed to exit within 2 seconds of signal 15 - using the force"
  393. Jan 13 16:01:20 kubernetes-master2 systemd[1]: docker-c864a51b59ad722f1b4f848d477a260105db5f9dfbdd6a085456e49fe83c4086.scope: Succeeded.
  394. Jan 13 16:01:20 kubernetes-master2 dockerd[827]: time="2021-01-13T16:01:20.905553470+01:00" level=error msg="stream copy error: reading from a closed fifo"
  395. Jan 13 16:01:20 kubernetes-master2 dockerd[827]: time="2021-01-13T16:01:20.905667169+01:00" level=error msg="stream copy error: reading from a closed fifo"
  396. Jan 13 16:01:20 kubernetes-master2 dockerd[827]: time="2021-01-13T16:01:20.905816801+01:00" level=error msg="Error running exec 227d85db8e5089fd1acfc86a54d59a05103f451e6f115a5bcd2d91ac8a4b8803 in container: cannot exec in a stopped state: unknown"
  397. Jan 13 16:01:20 kubernetes-master2 containerd[822]: time="2021-01-13T16:01:20.985908949+01:00" level=info msg="shim reaped" id=c864a51b59ad722f1b4f848d477a260105db5f9dfbdd6a085456e49fe83c4086
  398. Jan 13 16:01:21 kubernetes-master2 systemd[1333]: var-lib-docker-overlay2-f1a4f9920b693e6107b1911fc38e9d83d280247cdf6d6cb2a5be4598229fa168-merged.mount: Succeeded.
  399. Jan 13 16:01:21 kubernetes-master2 systemd[1]: var-lib-docker-overlay2-f1a4f9920b693e6107b1911fc38e9d83d280247cdf6d6cb2a5be4598229fa168-merged.mount: Succeeded.
  400. Jan 13 16:01:21 kubernetes-master2 dockerd[827]: time="2021-01-13T16:01:21.001319472+01:00" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
  401. Jan 13 16:01:21 kubernetes-master2 systemd[1333]: var-lib-docker-overlay2-5262837db879b4168115e25d6b3fdf2964faf5693b8f8b4189900a050a471adb\x2dinit-merged.mount: Succeeded.
  402. Jan 13 16:01:21 kubernetes-master2 systemd[1]: var-lib-docker-overlay2-5262837db879b4168115e25d6b3fdf2964faf5693b8f8b4189900a050a471adb\x2dinit-merged.mount: Succeeded.
  403. Jan 13 16:01:21 kubernetes-master2 systemd[1333]: var-lib-docker-overlay2-5262837db879b4168115e25d6b3fdf2964faf5693b8f8b4189900a050a471adb-merged.mount: Succeeded.
  404. Jan 13 16:01:21 kubernetes-master2 systemd[1]: var-lib-docker-overlay2-5262837db879b4168115e25d6b3fdf2964faf5693b8f8b4189900a050a471adb-merged.mount: Succeeded.
  405. Jan 13 16:01:21 kubernetes-master2 containerd[822]: time="2021-01-13T16:01:21.083243239+01:00" level=info msg="shim containerd-shim started" address="/containerd-shim/moby/fd301c5f206db22707ae922994e1fafcdf68ffd5b489725274dda2c2462f2475/shim.sock" debug=false pid=7535
  406. Jan 13 16:01:21 kubernetes-master2 systemd[1]: Started libcontainer container fd301c5f206db22707ae922994e1fafcdf68ffd5b489725274dda2c2462f2475.
  407. Jan 13 16:01:21 kubernetes-master2 kubelet[5807]: E0113 16:01:21.956677 5807 kubelet.go:1635] Failed creating a mirror pod for "kube-apiserver-kubernetes-master2_kube-system(7a2e5860052277dd81c04d93d6d3fde5)": pods "kube-apiserver-kubernetes-master2" is forbidden: rpc error: code = Unknown desc = OK: HTTP status code 200; transport: missing content-type field
  408. Jan 13 16:01:21 kubernetes-master2 kubelet[5807]: E0113 16:01:21.956738 5807 kubelet_node_status.go:447] Error updating node status, will retry: error getting node "kubernetes-master2": rpc error: code = Unknown desc = OK: HTTP status code 200; transport: missing content-type field
  409. Jan 13 16:01:21 kubernetes-master2 kubelet[5807]: E0113 16:01:21.957819 5807 event.go:264] Server rejected event '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"kube-proxy-c4rb5.1659d2b326103d9a", GenerateName:"", Namespace:"kube-system", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, InvolvedObject:v1.ObjectReference{Kind:"Pod", Namespace:"kube-system", Name:"kube-proxy-c4rb5", UID:"283b30fc-6fef-442d-96e2-d59031ee6f76", APIVersion:"v1", ResourceVersion:"1475", FieldPath:"spec.containers{kube-proxy}"}, Reason:"Started", Message:"Started container kube-proxy", Source:v1.EventSource{Component:"kubelet", Host:"kubernetes-master2"}, FirstTimestamp:v1.Time{Time:time.Time{wall:0xbff7e09e26388d9a, ext:8769768581, loc:(*time.Location)(0x70c9020)}}, LastTimestamp:v1.Time{Time:time.Time{wall:0xbff7e09e26388d9a, ext:8769768581, loc:(*time.Location)(0x70c9020)}}, Count:1, Type:"Normal", EventTime:v1.MicroTime{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, Series:(*v1.EventSeries)(nil), Action:"", Related:(*v1.ObjectReference)(nil), ReportingController:"", ReportingInstance:""}': 'rpc error: code = Unknown desc = OK: HTTP status code 200; transport: missing content-type field' (will not retry!)
  410. Jan 13 16:01:21 kubernetes-master2 kubelet[5807]: E0113 16:01:21.999881 5807 controller.go:144] failed to ensure lease exists, will retry in 6.4s, error: rpc error: code = Unavailable desc = transport is closing
  411. Jan 13 16:01:22 kubernetes-master2 kubelet[5807]: W0113 16:01:22.001463 5807 status_manager.go:550] Failed to get status for pod "kube-proxy-c4rb5_kube-system(283b30fc-6fef-442d-96e2-d59031ee6f76)": rpc error: code = Unavailable desc = transport is closing
  412. Jan 13 16:01:22 kubernetes-master2 kubelet[5807]: E0113 16:01:22.099819 5807 event.go:273] Unable to write event: '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"calico-node-kxvtq.1659d2b329b93009", GenerateName:"", Namespace:"kube-system", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, InvolvedObject:v1.ObjectReference{Kind:"Pod", Namespace:"kube-system", Name:"calico-node-kxvtq", UID:"1880486d-a3bc-4ebc-bd32-219d7f80429d", APIVersion:"v1", ResourceVersion:"1476", FieldPath:"spec.initContainers{upgrade-ipam}"}, Reason:"Created", Message:"Created container upgrade-ipam", Source:v1.EventSource{Component:"kubelet", Host:"kubernetes-master2"}, FirstTimestamp:v1.Time{Time:time.Time{wall:0xbff7e09e29e18009, ext:8831172342, loc:(*time.Location)(0x70c9020)}}, LastTimestamp:v1.Time{Time:time.Time{wall:0xbff7e09e29e18009, ext:8831172342, loc:(*time.Location)(0x70c9020)}}, Count:1, Type:"Normal", EventTime:v1.MicroTime{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, Series:(*v1.EventSeries)(nil), Action:"", Related:(*v1.ObjectReference)(nil), ReportingController:"", ReportingInstance:""}': 'Post "https://kubernetes-cluster.homelab01.local:8443/api/v1/namespaces/kube-system/events": http2: server sent GOAWAY and closed the connection; LastStreamID=47, ErrCode=NO_ERROR, debug=""'(may retry after sleeping)
  413. ^C
  414. wojcieh@kubernetes-master2:/var/log$
  415. Broadcast message from systemd-journald@kubernetes-master2 (Wed 2021-01-13 16:01:27 CET):
  416.  
  417. haproxy[870]: backend apiserver has no server available!
  418.  
  419.  
  420. Broadcast message from systemd-journald@kubernetes-master2 (Wed 2021-01-13 16:01:27 CET):
  421.  
  422. haproxy[870]: backend apiserver has no server available!
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement
Public Pastes
Advertisement
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!