API tools faq
paste
Advertisement
Guest User

operator_log

a guest
Aug 18th, 2021
299
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
Go 111.44 KB | None | 0 0
raw download clone embed print report
  1. 2021-08-17 22:48:40.478121 I | rookcmd: starting Rook v1.6.7 with arguments '/usr/local/bin/rook ceph operator'
  2. 2021-08-17 22:48:40.478320 I | rookcmd: flag values: --add_dir_header=false, --alsologtostderr=false, --csi-cephfs-plugin-template-path=/etc/ceph-csi/cephfs/csi-cephfsplugin.yaml, --csi-cephfs-provisioner-dep-template-path=/etc/ceph-csi/cephfs/csi-cephfsplugin-provisioner-dep.yaml, --csi-rbd-plugin-template-path=/etc/ceph-csi/rbd/csi-rbdplugin.yaml, --csi-rbd-provisioner-dep-template-path=/etc/ceph-csi/rbd/csi-rbdplugin-provisioner-dep.yaml, --enable-machine-disruption-budget=false, --help=false, --kubeconfig=, --log-flush-frequency=5s, --log-level=INFO, --log_backtrace_at=:0, --log_dir=, --log_file=, --log_file_max_size=1800, --logtostderr=true, --mon-healthcheck-interval=45s, --mon-out-timeout=10m0s, --one_output=false, --operator-image=, --service-account=, --skip_headers=false, --skip_log_headers=false, --stderrthreshold=2, --v=0, --vmodule=
  3. 2021-08-17 22:48:40.478331 I | cephcmd: starting Rook-Ceph operator
  4. 2021-08-17 22:48:42.280487 I | cephcmd: base ceph version inside the rook operator image is "ceph version 16.2.2 (e8f22dde28889481f4dda2beb8a07788204821d3) pacific (stable)"
  5. 2021-08-17 22:48:42.288102 I | op-k8sutil: ROOK_ENABLE_DISCOVERY_DAEMON="false" (env var)
  6. 2021-08-17 22:48:42.290697 I | operator: looking for secret "rook-ceph-admission-controller"
  7. 2021-08-17 22:48:42.292998 I | operator: secret "rook-ceph-admission-controller" not found. proceeding without the admission controller
  8. 2021-08-17 22:48:42.296799 I | op-k8sutil: ROOK_ENABLE_FLEX_DRIVER="false" (env var)
  9. 2021-08-17 22:48:42.296817 I | operator: watching all namespaces for ceph cluster CRs
  10. 2021-08-17 22:48:42.296884 I | operator: setting up the controller-runtime manager
  11. 2021-08-17 22:48:43.124399 I | ceph-cluster-controller: successfully started
  12. 2021-08-17 22:48:43.124523 I | ceph-cluster-controller: enabling hotplug orchestration
  13. 2021-08-17 22:48:43.124552 I | ceph-crashcollector-controller: successfully started
  14. 2021-08-17 22:48:43.124680 I | ceph-block-pool-controller: successfully started
  15. 2021-08-17 22:48:43.124781 I | ceph-object-store-user-controller: successfully started
  16. 2021-08-17 22:48:43.124898 I | ceph-object-realm-controller: successfully started
  17. 2021-08-17 22:48:43.124970 I | ceph-object-zonegroup-controller: successfully started
  18. 2021-08-17 22:48:43.125082 I | ceph-object-zone-controller: successfully started
  19. 2021-08-17 22:48:43.125261 I | ceph-object-controller: successfully started
  20. 2021-08-17 22:48:43.125388 I | ceph-file-controller: successfully started
  21. 2021-08-17 22:48:43.125486 I | ceph-nfs-controller: successfully started
  22. 2021-08-17 22:48:43.125606 I | ceph-rbd-mirror-controller: successfully started
  23. 2021-08-17 22:48:43.125728 I | ceph-client-controller: successfully started
  24. 2021-08-17 22:48:43.125808 I | ceph-filesystem-mirror-controller: successfully started
  25. 2021-08-17 22:48:43.125955 I | operator: starting the controller-runtime manager
  26. 2021-08-17 22:48:43.369013 I | clusterdisruption-controller: create event from ceph cluster CR
  27. 2021-08-17 22:48:43.572908 E | ceph-crashcollector-controller: ceph version not found for image "viavi-artefacts.local.dev:32705/nci-container-os/ceph:v15.2.13" used by cluster "kube-system". attempt to determine ceph version for the current cluster image timed out
  28. 2021-08-17 22:48:43.573006 I | ceph-cluster-controller: reconciling ceph cluster in namespace "kube-system"
  29. 2021-08-17 22:48:43.573924 E | ceph-crashcollector-controller: ceph version not found for image "viavi-artefacts.local.dev:32705/nci-container-os/ceph:v15.2.13" used by cluster "kube-system". attempt to determine ceph version for the current cluster image timed out
  30. 2021-08-17 22:48:43.574774 E | ceph-crashcollector-controller: ceph version not found for image "viavi-artefacts.local.dev:32705/nci-container-os/ceph:v15.2.13" used by cluster "kube-system". attempt to determine ceph version for the current cluster image timed out
  31. 2021-08-17 22:48:43.575761 E | ceph-crashcollector-controller: ceph version not found for image "viavi-artefacts.local.dev:32705/nci-container-os/ceph:v15.2.13" used by cluster "kube-system". attempt to determine ceph version for the current cluster image timed out
  32. 2021-08-17 22:48:43.577403 I | op-k8sutil: ROOK_ENABLE_FLEX_DRIVER="false" (env var)
  33. 2021-08-17 22:48:43.580295 I | op-mon: parsing mon endpoints: b=10.10.11.184:6789,c=10.10.11.212:6789,a=10.10.11.220:6789
  34. 2021-08-17 22:48:43.580762 I | op-mon: parsing mon endpoints: b=10.10.11.184:6789,c=10.10.11.212:6789,a=10.10.11.220:6789
  35. 2021-08-17 22:48:43.686121 I | ceph-csi: successfully created csi config map "rook-ceph-csi-config"
  36. 2021-08-17 22:48:43.690434 I | op-k8sutil: ROOK_CSI_ENABLE_RBD="true" (env var)
  37. 2021-08-17 22:48:43.693332 I | op-k8sutil: ROOK_CSI_ENABLE_CEPHFS="true" (env var)
  38. 2021-08-17 22:48:43.774737 I | op-mon: parsing mon endpoints: b=10.10.11.184:6789,c=10.10.11.212:6789,a=10.10.11.220:6789
  39. 2021-08-17 22:48:43.879007 I | op-mon: ceph mon status in namespace "kube-system" check interval "45s"
  40. 2021-08-17 22:48:43.879043 I | ceph-cluster-controller: enabling ceph mon monitoring goroutine for cluster "kube-system"
  41. 2021-08-17 22:48:43.879074 I | op-osd: ceph osd status in namespace "kube-system" check interval "1m0s"
  42. 2021-08-17 22:48:43.879080 I | ceph-cluster-controller: enabling ceph osd monitoring goroutine for cluster "kube-system"
  43. 2021-08-17 22:48:43.879096 I | ceph-cluster-controller: ceph status check interval is 1m0s
  44. 2021-08-17 22:48:43.879102 I | ceph-cluster-controller: enabling ceph status monitoring goroutine for cluster "kube-system"
  45. 2021-08-17 22:48:43.973630 I | op-k8sutil: ROOK_CSI_ALLOW_UNSUPPORTED_VERSION="false" (default)
  46. 2021-08-17 22:48:44.177141 I | op-k8sutil: ROOK_OBC_WATCH_OPERATOR_NAMESPACE="true" (env var)
  47. 2021-08-17 22:48:44.177201 I | op-bucket-prov: ceph bucket provisioner launched watching for provisioner "kube-system.ceph.rook.io/bucket"
  48. I0817 22:48:44.179792       6 manager.go:118] objectbucket.io/provisioner-manager "msg"="starting provisioner"  "name"="kube-system.ceph.rook.io/bucket"
  49. 2021-08-17 22:48:44.374924 I | op-k8sutil: ROOK_CSI_ENABLE_GRPC_METRICS="false" (env var)
  50. 2021-08-17 22:48:44.769751 I | ceph-cluster-controller: detecting the ceph image version for image viavi-artefacts.local.dev:32705/nci-container-os/ceph:v15.2.13...
  51. 2021-08-17 22:48:44.774144 I | op-k8sutil: CSI_ENABLE_HOST_NETWORK="true" (default)
  52. 2021-08-17 22:48:44.881480 I | ceph-block-pool-controller: skipping reconcile since operator is still initializing
  53. 2021-08-17 22:48:44.989840 I | ceph-file-controller: skipping reconcile since operator is still initializing
  54. 2021-08-17 22:48:45.106044 I | ceph-cluster-controller: skipping ceph status since operator is still initializing
  55. 2021-08-17 22:48:45.173605 I | op-k8sutil: ROOK_CSI_CEPH_IMAGE="viavi-artefacts.local.dev:32705/nci-container-os/cephcsi:v3.3.1" (env var)
  56. 2021-08-17 22:48:45.383323 I | op-k8sutil: Removing previous job rook-ceph-detect-version to start a new one
  57. 2021-08-17 22:48:45.392892 I | op-k8sutil: batch job rook-ceph-detect-version still exists
  58. 2021-08-17 22:48:45.575192 I | op-k8sutil: ROOK_CSI_REGISTRAR_IMAGE="viavi-artefacts.local.dev:32705/nci-container-os/csi-node-driver-registrar:v2.2.0" (env var)
  59. 2021-08-17 22:48:45.775535 I | op-k8sutil: ROOK_CSI_PROVISIONER_IMAGE="viavi-artefacts.local.dev:32705/nci-container-os/csi-provisioner:v2.2.2" (env var)
  60. 2021-08-17 22:48:45.975064 I | op-k8sutil: ROOK_CSI_ATTACHER_IMAGE="viavi-artefacts.local.dev:32705/nci-container-os/csi-attacher:v3.2.1" (env var)
  61. 2021-08-17 22:48:46.175775 I | op-k8sutil: ROOK_CSI_SNAPSHOTTER_IMAGE="viavi-artefacts.local.dev:32705/nci-container-os/csi-snapshotter:v4.1.1" (env var)
  62. 2021-08-17 22:48:46.375215 I | op-k8sutil: ROOK_CSI_KUBELET_DIR_PATH="/var/lib/kubelet" (default)
  63. 2021-08-17 22:48:46.574772 I | op-k8sutil: CSI_VOLUME_REPLICATION_IMAGE="quay.io/csiaddons/volumereplication-operator:v0.1.0" (default)
  64. 2021-08-17 22:48:46.774308 I | op-k8sutil: ROOK_CSI_CEPHFS_POD_LABELS="" (default)
  65. 2021-08-17 22:48:46.975614 I | op-k8sutil: ROOK_CSI_RBD_POD_LABELS="" (default)
  66. 2021-08-17 22:48:46.975723 I | ceph-csi: detecting the ceph csi image version for image "viavi-artefacts.local.dev:32705/nci-container-os/cephcsi:v3.3.1"
  67. 2021-08-17 22:48:47.176110 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default)
  68. 2021-08-17 22:48:47.398477 I | op-k8sutil: batch job rook-ceph-detect-version deleted
  69. 2021-08-17 22:48:49.898088 I | ceph-csi: Detected ceph CSI image version: "v3.3.1"
  70. 2021-08-17 22:48:49.916896 I | op-k8sutil: CSI_FORCE_CEPHFS_KERNEL_CLIENT="true" (env var)
  71. 2021-08-17 22:48:49.923112 I | op-k8sutil: CSI_CEPHFS_GRPC_METRICS_PORT="9091" (env var)
  72. 2021-08-17 22:48:49.926414 I | ceph-cluster-controller: detected ceph image version: "15.2.13-0 octopus"
  73. 2021-08-17 22:48:49.926430 I | ceph-cluster-controller: validating ceph version from provided image
  74. 2021-08-17 22:48:49.926521 I | op-k8sutil: CSI_CEPHFS_LIVENESS_METRICS_PORT="9081" (env var)
  75. 2021-08-17 22:48:49.928173 I | op-k8sutil: CSI_RBD_GRPC_METRICS_PORT="9090" (env var)
  76. 2021-08-17 22:48:49.931215 I | op-k8sutil: CSI_RBD_LIVENESS_METRICS_PORT="9080" (env var)
  77. 2021-08-17 22:48:50.088227 I | op-k8sutil: CSI_PLUGIN_PRIORITY_CLASSNAME="" (env var)
  78. 2021-08-17 22:48:50.286109 I | op-mon: parsing mon endpoints: b=10.10.11.184:6789,c=10.10.11.212:6789,a=10.10.11.220:6789
  79. 2021-08-17 22:48:50.487794 I | op-k8sutil: CSI_PROVISIONER_PRIORITY_CLASSNAME="" (env var)
  80. 2021-08-17 22:48:50.688023 I | cephclient: writing config file /var/lib/rook/kube-system/kube-system.config
  81. 2021-08-17 22:48:50.688128 I | cephclient: generated admin config in /var/lib/rook/kube-system
  82. 2021-08-17 22:48:50.889246 I | op-k8sutil: CSI_ENABLE_OMAP_GENERATOR="false" (env var)
  83. 2021-08-17 22:48:51.086038 I | op-k8sutil: CSI_ENABLE_RBD_SNAPSHOTTER="true" (env var)
  84. 2021-08-17 22:48:51.288451 I | op-k8sutil: CSI_ENABLE_CEPHFS_SNAPSHOTTER="true" (env var)
  85. 2021-08-17 22:48:51.487099 I | op-k8sutil: CSI_ENABLE_VOLUME_REPLICATION="false" (env var)
  86. 2021-08-17 22:48:51.674586 I | ceph-cluster-controller: cluster "kube-system": version "15.2.13-0 octopus" detected for image "viavi-artefacts.local.dev:32705/nci-container-os/ceph:v15.2.13"
  87. 2021-08-17 22:48:51.687427 I | op-k8sutil: CSI_CEPHFS_PLUGIN_UPDATE_STRATEGY="OnDelete" (env var)
  88. 2021-08-17 22:48:51.888744 I | op-k8sutil: CSI_RBD_PLUGIN_UPDATE_STRATEGY="OnDelete" (env var)
  89. 2021-08-17 22:48:51.888789 I | ceph-csi: Kubernetes version is 1.21
  90. 2021-08-17 22:48:52.112956 I | op-mon: start running mons
  91. 2021-08-17 22:48:52.287419 I | op-k8sutil: ROOK_CSI_RESIZER_IMAGE="viavi-artefacts.local.dev:32705/nci-container-os/csi-resizer:v1.2.0" (env var)
  92. 2021-08-17 22:48:52.689631 I | op-k8sutil: CSI_LOG_LEVEL="" (default)
  93. 2021-08-17 22:48:52.889015 I | op-mon: parsing mon endpoints: b=10.10.11.184:6789,c=10.10.11.212:6789,a=10.10.11.220:6789
  94. 2021-08-17 22:48:53.130442 I | ceph-csi: successfully started CSI Ceph RBD
  95. 2021-08-17 22:48:53.174837 I | ceph-csi: successfully started CSI CephFS driver
  96. 2021-08-17 22:48:53.486584 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default)
  97. 2021-08-17 22:48:53.888012 I | op-k8sutil: CSI_PROVISIONER_NODE_AFFINITY="" (default)
  98. 2021-08-17 22:48:54.089624 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"kube-system","monitors":["10.10.11.184:6789","10.10.11.212:6789","10.10.11.220:6789"]}] data:b=10.10.11.184:6789,c=10.10.11.212:6789,a=10.10.11.220:6789 mapping:{"node":{"a":{"Name":"nodename-2","Hostname":"nodename-2","Address":"10.10.11.220"},"b":{"Name":"nodename-1","Hostname":"nodename-1","Address":"10.10.11.184"},"c":{"Name":"nodename-3","Hostname":"nodename-3","Address":"10.10.11.212"}}} maxMonId:2]
  99. 2021-08-17 22:48:54.287048 I | op-k8sutil: CSI_PLUGIN_TOLERATIONS="- effect: NoSchedule\n  operator: Exists\n" (env var)
  100. 2021-08-17 22:48:54.689850 I | op-k8sutil: CSI_PLUGIN_NODE_AFFINITY="" (default)
  101. 2021-08-17 22:48:55.087262 I | op-k8sutil: CSI_RBD_PLUGIN_TOLERATIONS="" (default)
  102. 2021-08-17 22:48:55.489995 I | cephclient: writing config file /var/lib/rook/kube-system/kube-system.config
  103. 2021-08-17 22:48:55.490353 I | cephclient: generated admin config in /var/lib/rook/kube-system
  104. 2021-08-17 22:48:55.890149 I | op-k8sutil: CSI_RBD_PLUGIN_NODE_AFFINITY="" (default)
  105. 2021-08-17 22:48:56.087993 I | op-mon: parsing mon endpoints: b=10.10.11.184:6789,c=10.10.11.212:6789,a=10.10.11.220:6789
  106. 2021-08-17 22:48:56.485847 I | op-mon: parsing mon endpoints: b=10.10.11.184:6789,c=10.10.11.212:6789,a=10.10.11.220:6789
  107. 2021-08-17 22:48:56.689274 I | op-k8sutil: CSI_RBD_PLUGIN_RESOURCE="" (default)
  108. 2021-08-17 22:48:57.286672 I | op-k8sutil: CSI_RBD_PROVISIONER_TOLERATIONS="" (default)
  109. 2021-08-17 22:48:57.374511 I | ceph-block-pool-controller: creating pool "rook-ceph-ceph-pool" in namespace "kube-system"
  110. 2021-08-17 22:48:57.687467 I | op-k8sutil: CSI_RBD_PROVISIONER_NODE_AFFINITY="" (default)
  111. 2021-08-17 22:48:58.087983 I | op-k8sutil: CSI_RBD_PROVISIONER_RESOURCE="" (default)
  112. 2021-08-17 22:48:58.288039 I | op-mon: targeting the mon count 3
  113. 2021-08-17 22:48:58.293961 I | op-config: setting "global"="mon allow pool delete"="true" option to the mon configuration database
  114. 2021-08-17 22:48:59.087678 I | op-k8sutil: CSI_CEPHFS_PLUGIN_TOLERATIONS="" (default)
  115. 2021-08-17 22:48:59.373163 I | op-k8sutil: CSI_CEPHFS_PLUGIN_NODE_AFFINITY="" (default)
  116. 2021-08-17 22:48:59.488456 I | op-k8sutil: CSI_CEPHFS_PLUGIN_RESOURCE="" (default)
  117. 2021-08-17 22:48:59.688126 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_TOLERATIONS="" (default)
  118. 2021-08-17 22:48:59.969022 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_NODE_AFFINITY="" (default)
  119. 2021-08-17 22:49:00.086376 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_RESOURCE="" (default)
  120. 2021-08-17 22:49:00.891910 I | op-k8sutil: CSI_RBD_FSGROUPPOLICY="ReadWriteOnceWithFSType" (env var)
  121. 2021-08-17 22:49:00.978784 I | ceph-csi: CSIDriver object created for driver "kube-system.rbd.csi.ceph.com"
  122. 2021-08-17 22:49:01.169155 I | op-k8sutil: CSI_CEPHFS_FSGROUPPOLICY="ReadWriteOnceWithFSType" (env var)
  123. 2021-08-17 22:49:01.179977 I | ceph-csi: CSIDriver object created for driver "kube-system.cephfs.csi.ceph.com"
  124. 2021-08-17 22:49:01.772931 I | op-config: successfully set "global"="mon allow pool delete"="true" option to the mon configuration database
  125. 2021-08-17 22:49:01.772965 I | op-config: setting "global"="mon cluster log file"="" option to the mon configuration database
  126. 2021-08-17 22:49:05.179643 I | op-config: successfully set "global"="mon cluster log file"="" option to the mon configuration database
  127. 2021-08-17 22:49:05.179689 I | op-config: setting "global"="osd scrub auto repair"="true" option to the mon configuration database
  128. 2021-08-17 22:49:06.983420 I | ceph-file-controller: filesystem "rook-ceph-ceph-fs" already exists
  129. 2021-08-17 22:49:08.089947 I | op-config: successfully set "global"="osd scrub auto repair"="true" option to the mon configuration database
  130. 2021-08-17 22:49:08.090000 I | op-config: setting "global"="log to file"="false" option to the mon configuration database
  131. 2021-08-17 22:49:10.969264 I | op-config: successfully set "global"="log to file"="false" option to the mon configuration database
  132. 2021-08-17 22:49:10.969327 I | op-config: setting "global"="rbd_default_features"="3" option to the mon configuration database
  133. 2021-08-17 22:49:11.582150 I | cephclient: creating replicated pool rook-ceph-ceph-pool succeeded
  134. 2021-08-17 22:49:14.278132 I | op-config: successfully set "global"="rbd_default_features"="3" option to the mon configuration database
  135. 2021-08-17 22:49:14.278158 I | op-config: deleting "log file" option from the mon configuration database
  136. 2021-08-17 22:49:17.583417 I | op-config: successfully deleted "log file" option from the mon configuration database
  137. 2021-08-17 22:49:17.583473 I | op-mon: checking for basic quorum with existing mons
  138. 2021-08-17 22:49:17.583494 I | op-mon: setting mon endpoints for hostnetwork mode
  139. 2021-08-17 22:49:17.583508 I | op-mon: setting mon endpoints for hostnetwork mode
  140. 2021-08-17 22:49:17.583516 I | op-mon: setting mon endpoints for hostnetwork mode
  141. 2021-08-17 22:49:17.680517 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"kube-system","monitors":["10.10.11.184:6789","10.10.11.212:6789","10.10.11.220:6789"]}] data:b=10.10.11.184:6789,c=10.10.11.212:6789,a=10.10.11.220:6789 mapping:{"node":{"a":{"Name":"nodename-2","Hostname":"nodename-2","Address":"10.10.11.220"},"b":{"Name":"nodename-1","Hostname":"nodename-1","Address":"10.10.11.184"},"c":{"Name":"nodename-3","Hostname":"nodename-3","Address":"10.10.11.212"}}} maxMonId:2]
  142. 2021-08-17 22:49:17.769214 I | cephclient: writing config file /var/lib/rook/kube-system/kube-system.config
  143. 2021-08-17 22:49:17.769574 I | cephclient: generated admin config in /var/lib/rook/kube-system
  144. 2021-08-17 22:49:17.878231 I | op-mon: deployment for mon rook-ceph-mon-a already exists. updating if needed
  145. 2021-08-17 22:49:17.974586 I | op-k8sutil: updating deployment "rook-ceph-mon-a" after verifying it is safe to stop
  146. 2021-08-17 22:49:17.974622 I | op-mon: checking if we can stop the deployment rook-ceph-mon-a
  147. 2021-08-17 22:49:19.592761 I | cephclient: setting pool property "compression_mode" to "none" on pool "rook-ceph-ceph-fs-metadata"
  148. 2021-08-17 22:49:21.899004 I | cephclient: creating replicated pool rook-ceph-ceph-fs-metadata succeeded
  149. 2021-08-17 22:49:23.765442 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-nodename-2": the object has been modified; please apply your changes to the latest version and try again
  150. 2021-08-17 22:49:27.910628 I | op-k8sutil: finished waiting for updated deployment "rook-ceph-mon-a"
  151. 2021-08-17 22:49:27.910670 I | op-mon: checking if we can continue the deployment rook-ceph-mon-a
  152. 2021-08-17 22:49:27.910687 I | op-mon: waiting for mon quorum with [a b c]
  153. 2021-08-17 22:49:28.001888 I | op-mon: mons running: [a b c]
  154. 2021-08-17 22:49:28.884873 I | op-mon: Monitors in quorum: [a b c]
  155. 2021-08-17 22:49:28.968980 I | op-mon: deployment for mon rook-ceph-mon-b already exists. updating if needed
  156. 2021-08-17 22:49:28.986561 I | op-k8sutil: updating deployment "rook-ceph-mon-b" after verifying it is safe to stop
  157. 2021-08-17 22:49:28.986588 I | op-mon: checking if we can stop the deployment rook-ceph-mon-b
  158. 2021-08-17 22:49:44.528783 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-nodename-1": the object has been modified; please apply your changes to the latest version and try again
  159. 2021-08-17 22:49:44.577112 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-nodename-1": the object has been modified; please apply your changes to the latest version and try again
  160. 2021-08-17 22:49:48.070356 I | op-k8sutil: finished waiting for updated deployment "rook-ceph-mon-b"
  161. 2021-08-17 22:49:48.070397 I | op-mon: checking if we can continue the deployment rook-ceph-mon-b
  162. 2021-08-17 22:49:48.070416 I | op-mon: waiting for mon quorum with [a b c]
  163. 2021-08-17 22:49:48.269308 I | op-mon: mons running: [a b c]
  164. 2021-08-17 22:49:48.569430 I | cephclient: setting pool property "compression_mode" to "none" on pool "rook-ceph-ceph-fs-data0"
  165. 2021-08-17 22:49:55.391296 I | op-mon: Monitors in quorum: [a c]
  166. 2021-08-17 22:49:55.480573 I | op-mon: deployment for mon rook-ceph-mon-c already exists. updating if needed
  167. 2021-08-17 22:49:55.497708 I | op-k8sutil: updating deployment "rook-ceph-mon-c" after verifying it is safe to stop
  168. 2021-08-17 22:49:55.497731 I | op-mon: checking if we can stop the deployment rook-ceph-mon-c
  169. 2021-08-17 22:49:57.790803 I | util: retrying after 1m0s, last error: deployment rook-ceph-mon-c cannot be stopped: Error EBUSY: not enough monitors would be available (a) after stopping mons [c]
  170. .
  171. 2021-08-17 22:49:59.924482 I | cephclient: creating replicated pool rook-ceph-ceph-fs-data0 succeeded
  172. 2021-08-17 22:50:01.685733 I | cephclient: setting allow_standby_replay for filesystem "rook-ceph-ceph-fs"
  173. 2021-08-17 22:50:03.261826 I | ceph-file-controller: start running mdses for filesystem "rook-ceph-ceph-fs"
  174. 2021-08-17 22:50:04.200494 I | cephclient: getting or creating ceph auth key "mds.rook-ceph-ceph-fs-a"
  175. 2021-08-17 22:50:05.125828 I | op-mds: setting mds config flags
  176. 2021-08-17 22:50:05.125853 I | op-config: setting "mds.rook-ceph-ceph-fs-a"="mds_join_fs"="rook-ceph-ceph-fs" option to the mon configuration database
  177. 2021-08-17 22:50:05.988539 I | op-config: successfully set "mds.rook-ceph-ceph-fs-a"="mds_join_fs"="rook-ceph-ceph-fs" option to the mon configuration database
  178. 2021-08-17 22:50:06.005369 I | op-mds: deployment for mds "rook-ceph-mds-rook-ceph-ceph-fs-a" already exists. updating if needed
  179. 2021-08-17 22:50:06.026954 I | op-k8sutil: updating deployment "rook-ceph-mds-rook-ceph-ceph-fs-a" after verifying it is safe to stop
  180. 2021-08-17 22:50:06.026987 I | op-mon: checking if we can stop the deployment rook-ceph-mds-rook-ceph-ceph-fs-a
  181. 2021-08-17 22:50:13.370370 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-nodename-2": the object has been modified; please apply your changes to the latest version and try again
  182. 2021-08-17 22:50:13.422895 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-nodename-2": the object has been modified; please apply your changes to the latest version and try again
  183. 2021-08-17 22:50:13.449965 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-nodename-3": the object has been modified; please apply your changes to the latest version and try again
  184. 2021-08-17 22:50:16.952536 I | op-k8sutil: finished waiting for updated deployment "rook-ceph-mds-rook-ceph-ceph-fs-a"
  185. 2021-08-17 22:50:16.952582 I | op-mon: checking if we can continue the deployment rook-ceph-mds-rook-ceph-ceph-fs-a
  186. 2021-08-17 22:50:17.894424 I | util: retrying after 15s, last error: mds rook-ceph-ceph-fs-a not found in fsmap, this likely means mdss are transitioning between active and standby states
  187. 2021-08-17 22:50:33.774275 I | cephclient: mds rook-ceph-ceph-fs-a is up:standby-replay
  188. 2021-08-17 22:50:33.774358 I | cephclient: getting or creating ceph auth key "mds.rook-ceph-ceph-fs-b"
  189. 2021-08-17 22:50:34.707005 I | op-mds: setting mds config flags
  190. 2021-08-17 22:50:34.707054 I | op-config: setting "mds.rook-ceph-ceph-fs-b"="mds_join_fs"="rook-ceph-ceph-fs" option to the mon configuration database
  191. 2021-08-17 22:50:36.375198 I | op-config: successfully set "mds.rook-ceph-ceph-fs-b"="mds_join_fs"="rook-ceph-ceph-fs" option to the mon configuration database
  192. 2021-08-17 22:50:36.399717 I | op-mds: deployment for mds "rook-ceph-mds-rook-ceph-ceph-fs-b" already exists. updating if needed
  193. 2021-08-17 22:50:36.480853 I | op-k8sutil: updating deployment "rook-ceph-mds-rook-ceph-ceph-fs-b" after verifying it is safe to stop
  194. 2021-08-17 22:50:36.480880 I | op-mon: checking if we can stop the deployment rook-ceph-mds-rook-ceph-ceph-fs-b
  195. 2021-08-17 22:50:54.132006 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-nodename-1": the object has been modified; please apply your changes to the latest version and try again
  196. 2021-08-17 22:50:54.148797 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-nodename-1": the object has been modified; please apply your changes to the latest version and try again
  197. 2021-08-17 22:50:54.175305 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-nodename-1": the object has been modified; please apply your changes to the latest version and try again
  198. 2021-08-17 22:50:54.207823 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-nodename-1": the object has been modified; please apply your changes to the latest version and try again
  199. 2021-08-17 22:50:54.253637 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-nodename-1": the object has been modified; please apply your changes to the latest version and try again
  200. 2021-08-17 22:50:56.810223 I | op-k8sutil: finished waiting for updated deployment "rook-ceph-mds-rook-ceph-ceph-fs-b"
  201. 2021-08-17 22:50:56.810268 I | op-mon: checking if we can continue the deployment rook-ceph-mds-rook-ceph-ceph-fs-b
  202. 2021-08-17 22:51:01.670510 I | cephclient: mds rook-ceph-ceph-fs-b is up:standby-replay
  203. 2021-08-17 22:51:12.622100 I | op-k8sutil: finished waiting for updated deployment "rook-ceph-mon-c"
  204. 2021-08-17 22:51:12.622135 I | op-mon: checking if we can continue the deployment rook-ceph-mon-c
  205. 2021-08-17 22:51:12.622151 I | op-mon: waiting for mon quorum with [a b c]
  206. 2021-08-17 22:51:12.653991 I | op-mon: mons running: [a b c]
  207. 2021-08-17 22:51:19.417405 I | op-mon: Monitors in quorum: [a b c]
  208. 2021-08-17 22:51:19.417449 I | op-mon: mons created: 3
  209. 2021-08-17 22:51:20.318355 I | op-mon: waiting for mon quorum with [a b c]
  210. 2021-08-17 22:51:20.383298 I | op-mon: mons running: [a b c]
  211. 2021-08-17 22:51:21.193216 I | op-mon: Monitors in quorum: [a b c]
  212. 2021-08-17 22:51:21.193308 I | cephclient: getting or creating ceph auth key "client.csi-rbd-provisioner"
  213. 2021-08-17 22:51:23.477226 I | cephclient: getting or creating ceph auth key "client.csi-rbd-node"
  214. 2021-08-17 22:51:23.608196 I | op-mon: checking if multiple mons are on the same node
  215. 2021-08-17 22:51:24.496156 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-provisioner"
  216. 2021-08-17 22:51:25.405004 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-node"
  217. 2021-08-17 22:51:26.347767 I | ceph-csi: created kubernetes csi secrets for cluster "kube-system"
  218. 2021-08-17 22:51:26.347812 I | cephclient: getting or creating ceph auth key "client.crash"
  219. 2021-08-17 22:51:27.303992 I | ceph-crashcollector-controller: created kubernetes crash collector secret for cluster "kube-system"
  220. 2021-08-17 22:51:28.284368 I | cephclient: successfully enabled msgr2 protocol
  221. 2021-08-17 22:51:28.316087 I | op-mgr: start running mgr
  222. 2021-08-17 22:51:28.316152 I | cephclient: getting or creating ceph auth key "mgr.a"
  223. 2021-08-17 22:51:29.228046 I | op-mgr: deployment for mgr rook-ceph-mgr-a already exists. updating if needed
  224. 2021-08-17 22:51:29.276894 I | op-k8sutil: updating deployment "rook-ceph-mgr-a" after verifying it is safe to stop
  225. 2021-08-17 22:51:29.276935 I | op-mon: checking if we can stop the deployment rook-ceph-mgr-a
  226. 2021-08-17 22:51:48.350186 I | op-k8sutil: finished waiting for updated deployment "rook-ceph-mgr-a"
  227. 2021-08-17 22:51:48.350205 I | op-mon: checking if we can continue the deployment rook-ceph-mgr-a
  228. 2021-08-17 22:51:48.353294 I | op-mgr: setting services to point to mgr "a"
  229. 2021-08-17 22:51:48.387105 I | op-osd: start running osds in namespace "kube-system"
  230. 2021-08-17 22:51:48.387121 I | op-osd: wait timeout for healthy OSDs during upgrade or restart is "10m0s"
  231. 2021-08-17 22:51:48.470784 I | op-osd: start provisioning the OSDs on PVCs, if needed
  232. 2021-08-17 22:51:48.479771 I | op-osd: no storageClassDeviceSets defined to configure OSDs on PVCs
  233. 2021-08-17 22:51:48.479806 I | op-osd: start provisioning the OSDs on nodes, if needed
  234. 2021-08-17 22:51:48.479817 W | op-osd: useAllNodes is TRUE, but nodes are specified. NODES in the cluster CR will be IGNORED unless useAllNodes is FALSE.
  235. 2021-08-17 22:51:48.673543 I | op-osd: 3 of the 4 storage nodes are valid
  236. 2021-08-17 22:51:48.686323 I | op-k8sutil: Removing previous job rook-ceph-osd-prepare-nodename-1 to start a new one
  237. 2021-08-17 22:51:48.773556 I | op-k8sutil: batch job rook-ceph-osd-prepare-nodename-1 deleted
  238. 2021-08-17 22:51:48.782772 I | op-osd: started OSD provisioning job for node "nodename-1"
  239. 2021-08-17 22:51:48.970179 I | op-k8sutil: Removing previous job rook-ceph-osd-prepare-nodename-2 to start a new one
  240. 2021-08-17 22:51:49.070100 I | op-k8sutil: batch job rook-ceph-osd-prepare-nodename-2 still exists
  241. 2021-08-17 22:51:51.075432 I | op-k8sutil: batch job rook-ceph-osd-prepare-nodename-2 deleted
  242. 2021-08-17 22:51:51.271551 I | op-osd: started OSD provisioning job for node "nodename-2"
  243. 2021-08-17 22:51:51.369490 I | op-k8sutil: Found previous job rook-ceph-osd-prepare-nodename-3. Status={Conditions:[{Type:Complete Status:True LastProbeTime:2021-08-17 22:15:30 +0000 UTC LastTransitionTime:2021-08-17 22:15:30 +0000 UTC Reason: Message:}] StartTime:2021-08-17 22:15:14 +0000 UTC CompletionTime:2021-08-17 22:15:30 +0000 UTC Active:1 Succeeded:1 Failed:0 CompletedIndexes:}
  244. 2021-08-17 22:51:51.369520 I | op-osd: started OSD provisioning job for node "nodename-3"
  245. 2021-08-17 22:51:51.468839 I | op-osd: OSD orchestration status for node nodename-1 is "orchestrating"
  246. 2021-08-17 22:51:51.468861 I | op-osd: OSD orchestration status for node nodename-2 is "starting"
  247. 2021-08-17 22:51:51.468866 I | op-osd: OSD orchestration status for node nodename-3 is "starting"
  248. 2021-08-17 22:51:55.072728 I | op-mgr: successful modules: prometheus
  249. 2021-08-17 22:51:55.078486 I | op-mgr: successful modules: dashboard
  250. 2021-08-17 22:51:55.982181 I | op-config: setting "global"="osd_pool_default_pg_autoscale_mode"="on" option to the mon configuration database
  251. 2021-08-17 22:51:59.283796 I | op-config: successfully set "global"="osd_pool_default_pg_autoscale_mode"="on" option to the mon configuration database
  252. 2021-08-17 22:51:59.283839 I | op-config: setting "global"="mon_pg_warn_min_per_osd"="0" option to the mon configuration database
  253. 2021-08-17 22:52:01.269091 I | op-mgr: successful modules: balancer
  254. 2021-08-17 22:52:01.893548 I | op-config: successfully set "global"="mon_pg_warn_min_per_osd"="0" option to the mon configuration database
  255. 2021-08-17 22:52:01.893596 I | op-mgr: successful modules: mgr module(s) from the spec
  256. 2021-08-17 22:52:02.906143 I | op-osd: updating OSD 0 on node "nodename-3"
  257. 2021-08-17 22:52:13.226418 I | op-osd: OSD orchestration status for node nodename-1 is "completed"
  258. 2021-08-17 22:52:13.232944 I | op-osd: OSD orchestration status for node nodename-2 is "orchestrating"
  259. 2021-08-17 22:52:13.233399 I | op-osd: OSD orchestration status for node nodename-2 is "completed"
  260. 2021-08-17 22:52:16.309119 I | op-osd: OSD 1 is not ok-to-stop. will try updating it again later
  261. 2021-08-17 22:52:21.799353 I | op-osd: OSD 2 is not ok-to-stop. will try updating it again later
  262. 2021-08-17 22:52:25.108947 I | op-osd: OSD 1 is not ok-to-stop. will try updating it again later
  263. 2021-08-17 22:52:28.414788 I | op-osd: OSD 2 is not ok-to-stop. will try updating it again later
  264. 2021-08-17 22:52:31.792286 I | op-osd: updating OSD 1 on node "nodename-1"
  265. 2021-08-17 22:52:53.202399 I | op-osd: OSD 2 is not ok-to-stop. will try updating it again later
  266. 2021-08-17 22:52:53.202425 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 2 of 3 OSDs have been updated
  267. 2021-08-17 22:52:57.880975 I | op-osd: OSD 2 is not ok-to-stop. will try updating it again later
  268. 2021-08-17 22:53:01.083693 I | op-osd: OSD 2 is not ok-to-stop. will try updating it again later
  269. 2021-08-17 22:53:04.398464 I | op-osd: OSD 2 is not ok-to-stop. will try updating it again later
  270. 2021-08-17 22:53:10.426872 I | op-osd: updating OSD 2 on node "nodename-2"
  271. 2021-08-17 22:53:51.370043 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  272. 2021-08-17 22:54:51.370389 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  273. 2021-08-17 22:55:51.370510 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  274. 2021-08-17 22:56:51.370567 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  275. 2021-08-17 22:57:51.369999 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  276. 2021-08-17 22:58:51.369843 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  277. 2021-08-17 22:59:51.369987 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  278. 2021-08-17 23:00:51.370357 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  279. 2021-08-17 23:01:51.370165 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  280. 2021-08-17 23:02:51.369942 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  281. 2021-08-17 23:03:51.370689 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  282. 2021-08-17 23:04:51.370375 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  283. 2021-08-17 23:05:51.369819 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  284. 2021-08-17 23:06:51.370489 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  285. 2021-08-17 23:07:51.370232 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  286. 2021-08-17 23:08:51.370056 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  287. 2021-08-17 23:09:51.370757 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  288. 2021-08-17 23:10:51.370604 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  289. 2021-08-17 23:11:51.370379 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  290. 2021-08-17 23:12:51.369605 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  291. 2021-08-17 23:13:51.369737 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  292. 2021-08-17 23:14:51.370490 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  293. 2021-08-17 23:15:51.370311 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  294. 2021-08-17 23:16:51.370238 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  295. 2021-08-17 23:17:51.370452 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  296. 2021-08-17 23:18:51.370437 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  297. 2021-08-17 23:19:51.369948 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  298. 2021-08-17 23:20:51.369622 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  299. 2021-08-17 23:21:51.370707 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  300. 2021-08-17 23:22:51.369773 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  301. 2021-08-17 23:23:51.370114 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  302. 2021-08-17 23:24:51.370169 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  303. 2021-08-17 23:25:51.369681 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  304. 2021-08-17 23:26:51.369908 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  305. 2021-08-17 23:27:51.369732 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  306. 2021-08-17 23:28:51.370684 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  307. 2021-08-17 23:29:51.370490 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  308. 2021-08-17 23:30:51.369919 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  309. 2021-08-17 23:31:51.370544 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  310. 2021-08-17 23:32:51.370588 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  311. 2021-08-17 23:33:51.370009 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  312. 2021-08-17 23:34:51.370592 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  313. 2021-08-17 23:35:36.969071 I | op-osd: restarting watcher for OSD provisioning status ConfigMaps. the watcher closed the channel
  314. 2021-08-17 23:35:36.985681 I | op-osd: OSD orchestration status for node nodename-3 is "starting"
  315. 2021-08-17 23:35:51.369777 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  316. 2021-08-17 23:36:51.370235 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  317. 2021-08-17 23:37:51.369626 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  318. 2021-08-17 23:38:51.369609 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  319. 2021-08-17 23:39:51.370628 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  320. 2021-08-17 23:40:51.369735 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  321. 2021-08-17 23:41:51.370007 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  322. 2021-08-17 23:42:51.370004 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  323. 2021-08-17 23:43:51.370623 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  324. 2021-08-17 23:44:51.370538 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  325. 2021-08-17 23:45:51.370152 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  326. 2021-08-17 23:46:51.369877 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  327. 2021-08-17 23:47:51.370611 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  328. 2021-08-17 23:48:51.370240 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  329. 2021-08-17 23:49:51.370401 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  330. 2021-08-17 23:50:51.369705 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  331. 2021-08-17 23:51:51.369889 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  332. 2021-08-17 23:52:51.369692 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  333. 2021-08-17 23:53:51.370049 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  334. 2021-08-17 23:54:51.369967 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  335. 2021-08-17 23:55:51.370694 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  336. 2021-08-17 23:56:51.370683 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  337. 2021-08-17 23:57:51.369702 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  338. 2021-08-17 23:58:51.370525 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  339. 2021-08-17 23:59:51.370207 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  340. 2021-08-18 00:00:51.369849 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  341. 2021-08-18 00:01:51.369813 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  342. 2021-08-18 00:02:51.370074 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  343. 2021-08-18 00:03:51.370394 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  344. 2021-08-18 00:04:51.370172 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  345. 2021-08-18 00:05:51.369587 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  346. 2021-08-18 00:06:51.370601 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  347. 2021-08-18 00:07:51.369927 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  348. 2021-08-18 00:08:51.370179 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  349. 2021-08-18 00:09:51.370619 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  350. 2021-08-18 00:10:51.369676 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  351. 2021-08-18 00:11:51.369989 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  352. 2021-08-18 00:12:51.370434 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  353. 2021-08-18 00:13:51.370110 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  354. 2021-08-18 00:14:51.369976 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  355. 2021-08-18 00:15:51.370671 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  356. 2021-08-18 00:16:32.024704 I | op-osd: restarting watcher for OSD provisioning status ConfigMaps. the watcher closed the channel
  357. 2021-08-18 00:16:32.052379 I | op-osd: OSD orchestration status for node nodename-3 is "starting"
  358. 2021-08-18 00:16:51.370391 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  359. 2021-08-18 00:17:51.369754 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  360. 2021-08-18 00:18:51.369709 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  361. 2021-08-18 00:19:51.369740 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  362. 2021-08-18 00:20:51.369608 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  363. 2021-08-18 00:21:51.369743 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  364. 2021-08-18 00:22:51.369847 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  365. 2021-08-18 00:23:51.370205 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  366. 2021-08-18 00:24:51.370170 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  367. 2021-08-18 00:25:51.369947 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  368. 2021-08-18 00:26:51.369935 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  369. 2021-08-18 00:27:51.369802 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  370. 2021-08-18 00:28:51.370698 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  371. 2021-08-18 00:29:51.370890 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  372. 2021-08-18 00:30:51.369630 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  373. 2021-08-18 00:31:51.370357 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  374. 2021-08-18 00:32:51.370608 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  375. 2021-08-18 00:33:51.370008 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  376. 2021-08-18 00:34:51.369749 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  377. 2021-08-18 00:35:51.370043 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  378. 2021-08-18 00:36:51.369742 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  379. 2021-08-18 00:37:51.370683 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  380. 2021-08-18 00:38:51.370341 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  381. 2021-08-18 00:39:51.370294 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  382. 2021-08-18 00:40:51.370399 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  383. 2021-08-18 00:41:51.370555 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  384. 2021-08-18 00:42:51.373500 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  385. 2021-08-18 00:43:51.369735 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  386. 2021-08-18 00:44:51.369707 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  387. 2021-08-18 00:45:51.370014 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  388. 2021-08-18 00:46:51.370729 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  389. 2021-08-18 00:47:51.370313 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  390. 2021-08-18 00:48:51.370071 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  391. 2021-08-18 00:49:51.369867 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  392. 2021-08-18 00:50:51.369690 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  393. 2021-08-18 00:51:51.369727 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  394. 2021-08-18 00:52:51.369792 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  395. 2021-08-18 00:53:51.370103 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  396. 2021-08-18 00:54:51.370418 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  397. 2021-08-18 00:55:51.369750 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  398. 2021-08-18 00:56:51.370119 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  399. 2021-08-18 00:57:51.370676 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  400. 2021-08-18 00:58:51.369754 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  401. 2021-08-18 00:59:51.369997 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  402. 2021-08-18 01:00:51.370146 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  403. 2021-08-18 01:01:51.370579 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  404. 2021-08-18 01:02:51.370638 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  405. 2021-08-18 01:03:51.370696 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  406. 2021-08-18 01:04:51.370608 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  407. 2021-08-18 01:05:51.370532 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  408. 2021-08-18 01:06:51.369801 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  409. 2021-08-18 01:07:51.370045 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  410. 2021-08-18 01:08:51.370260 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  411. 2021-08-18 01:09:51.369826 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  412. 2021-08-18 01:10:51.369673 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  413. 2021-08-18 01:11:51.370152 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  414. 2021-08-18 01:12:51.369908 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  415. 2021-08-18 01:13:51.370223 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  416. 2021-08-18 01:14:13.511293 I | op-osd: restarting watcher for OSD provisioning status ConfigMaps. the watcher closed the channel
  417. 2021-08-18 01:14:13.525826 I | op-osd: OSD orchestration status for node nodename-3 is "starting"
  418. 2021-08-18 01:14:51.371775 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  419. 2021-08-18 01:15:51.370526 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  420. 2021-08-18 01:16:51.370682 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  421. 2021-08-18 01:17:51.370491 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  422. 2021-08-18 01:18:51.369863 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  423. 2021-08-18 01:19:51.370936 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  424. 2021-08-18 01:20:51.370085 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  425. 2021-08-18 01:21:51.370056 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  426. 2021-08-18 01:22:51.369751 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  427. 2021-08-18 01:23:51.369672 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  428. 2021-08-18 01:24:51.373630 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  429. 2021-08-18 01:25:51.370551 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  430. 2021-08-18 01:26:51.369638 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  431. 2021-08-18 01:27:51.370341 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  432. 2021-08-18 01:28:51.369842 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  433. 2021-08-18 01:29:51.369988 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  434. 2021-08-18 01:30:51.369901 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  435. 2021-08-18 01:31:51.370011 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  436. 2021-08-18 01:32:51.370256 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  437. 2021-08-18 01:33:51.369956 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  438. 2021-08-18 01:34:51.369832 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  439. 2021-08-18 01:35:51.369903 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  440. 2021-08-18 01:36:51.370141 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  441. 2021-08-18 01:37:51.370614 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  442. 2021-08-18 01:38:51.369748 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  443. 2021-08-18 01:39:51.369956 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  444. 2021-08-18 01:40:51.370713 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  445. 2021-08-18 01:41:51.369851 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  446. 2021-08-18 01:42:51.370053 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  447. 2021-08-18 01:43:51.370543 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  448. 2021-08-18 01:44:51.370003 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  449. 2021-08-18 01:45:51.370275 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  450. 2021-08-18 01:46:51.370011 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  451. 2021-08-18 01:47:51.370199 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  452. 2021-08-18 01:48:51.370709 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  453. 2021-08-18 01:49:51.370143 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  454. 2021-08-18 01:50:51.370662 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  455. 2021-08-18 01:51:51.370007 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  456. 2021-08-18 01:52:51.369954 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  457. 2021-08-18 01:53:39.374153 I | op-osd: restarting watcher for OSD provisioning status ConfigMaps. the watcher closed the channel
  458. 2021-08-18 01:53:39.390224 I | op-osd: OSD orchestration status for node nodename-3 is "starting"
  459. 2021-08-18 01:53:51.370713 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  460. 2021-08-18 01:54:51.369981 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  461. 2021-08-18 01:55:51.370363 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  462. 2021-08-18 01:56:51.370248 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  463. 2021-08-18 01:57:51.370571 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  464. 2021-08-18 01:58:51.369967 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  465. 2021-08-18 01:59:51.369971 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  466. 2021-08-18 02:00:51.370723 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  467. 2021-08-18 02:01:51.370764 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  468. 2021-08-18 02:02:51.370727 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  469. 2021-08-18 02:03:51.370000 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  470. 2021-08-18 02:04:51.370721 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  471. 2021-08-18 02:05:51.369615 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  472. 2021-08-18 02:06:51.369989 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  473. 2021-08-18 02:07:51.370480 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  474. 2021-08-18 02:08:51.369907 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  475. 2021-08-18 02:09:51.370291 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  476. 2021-08-18 02:10:51.369951 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  477. 2021-08-18 02:11:51.369638 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  478. 2021-08-18 02:12:51.370505 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  479. 2021-08-18 02:13:51.370708 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  480. 2021-08-18 02:14:51.370393 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  481. 2021-08-18 02:15:51.370606 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  482. 2021-08-18 02:16:51.370408 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  483. 2021-08-18 02:17:51.370640 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  484. 2021-08-18 02:18:51.369881 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  485. 2021-08-18 02:19:51.369783 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  486. 2021-08-18 02:20:51.370621 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  487. 2021-08-18 02:21:51.369962 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  488. 2021-08-18 02:22:51.369721 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  489. 2021-08-18 02:23:46.205471 I | op-osd: restarting watcher for OSD provisioning status ConfigMaps. the watcher closed the channel
  490. 2021-08-18 02:23:46.246840 I | op-osd: OSD orchestration status for node nodename-3 is "starting"
  491. 2021-08-18 02:23:51.370548 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  492. 2021-08-18 02:24:51.369678 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  493. 2021-08-18 02:25:51.370730 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  494. 2021-08-18 02:26:51.369689 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  495. 2021-08-18 02:27:51.370044 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  496. 2021-08-18 02:28:51.369813 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  497. 2021-08-18 02:29:51.370516 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  498. 2021-08-18 02:30:51.370701 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  499. 2021-08-18 02:31:51.369917 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  500. 2021-08-18 02:32:51.369745 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  501. 2021-08-18 02:33:51.370327 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  502. 2021-08-18 02:34:51.369916 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  503. 2021-08-18 02:35:51.370671 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  504. 2021-08-18 02:36:51.369736 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  505. 2021-08-18 02:37:51.370036 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  506. 2021-08-18 02:38:51.370589 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  507. 2021-08-18 02:39:51.370161 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  508. 2021-08-18 02:40:51.369928 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  509. 2021-08-18 02:41:51.370493 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  510. 2021-08-18 02:42:51.370168 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  511. 2021-08-18 02:43:51.369750 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  512. 2021-08-18 02:44:51.370633 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  513. 2021-08-18 02:45:51.370499 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  514. 2021-08-18 02:46:51.370684 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  515. 2021-08-18 02:47:51.370257 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  516. 2021-08-18 02:48:51.370007 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  517. 2021-08-18 02:49:51.369998 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  518. 2021-08-18 02:50:51.369815 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  519. 2021-08-18 02:51:51.369935 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  520. 2021-08-18 02:52:51.370614 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  521. 2021-08-18 02:53:51.370138 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  522. 2021-08-18 02:54:51.370094 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  523. 2021-08-18 02:55:51.370476 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  524. 2021-08-18 02:56:51.370006 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  525. 2021-08-18 02:57:51.370433 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  526. 2021-08-18 02:58:51.370389 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  527. 2021-08-18 02:59:51.370140 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  528. 2021-08-18 03:00:51.370452 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  529. 2021-08-18 03:01:51.370500 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  530. 2021-08-18 03:02:51.370626 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  531. 2021-08-18 03:03:51.369871 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  532. 2021-08-18 03:04:51.370113 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  533. 2021-08-18 03:05:51.369965 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  534. 2021-08-18 03:06:51.369974 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  535. 2021-08-18 03:07:51.369797 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  536. 2021-08-18 03:08:51.370521 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  537. 2021-08-18 03:09:51.370388 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  538. 2021-08-18 03:10:51.370234 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  539. 2021-08-18 03:11:51.370667 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  540. 2021-08-18 03:12:51.370677 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  541. 2021-08-18 03:13:51.370080 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  542. 2021-08-18 03:14:51.369842 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  543. 2021-08-18 03:15:36.793340 I | op-osd: restarting watcher for OSD provisioning status ConfigMaps. the watcher closed the channel
  544. 2021-08-18 03:15:36.869400 I | op-osd: OSD orchestration status for node nodename-3 is "starting"
  545. 2021-08-18 03:15:51.370041 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  546. 2021-08-18 03:16:51.369683 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  547. 2021-08-18 03:17:51.370330 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  548. 2021-08-18 03:18:51.370292 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  549. 2021-08-18 03:19:51.370729 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  550. 2021-08-18 03:20:51.370363 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  551. 2021-08-18 03:21:51.369726 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  552. 2021-08-18 03:22:51.370736 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  553. 2021-08-18 03:23:51.369877 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  554. 2021-08-18 03:24:51.370468 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  555. 2021-08-18 03:25:51.369751 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  556. 2021-08-18 03:26:51.370056 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  557. 2021-08-18 03:27:51.369857 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  558. 2021-08-18 03:28:51.370147 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  559. 2021-08-18 03:29:51.370686 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  560. 2021-08-18 03:30:51.369677 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  561. 2021-08-18 03:31:51.369639 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  562. 2021-08-18 03:32:51.369724 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  563. 2021-08-18 03:33:51.370690 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  564. 2021-08-18 03:34:51.370426 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  565. 2021-08-18 03:35:51.370307 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  566. 2021-08-18 03:36:51.369871 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  567. 2021-08-18 03:37:51.369970 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  568. 2021-08-18 03:38:51.370183 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  569. 2021-08-18 03:39:51.370215 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  570. 2021-08-18 03:40:51.370247 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  571. 2021-08-18 03:41:51.370315 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  572. 2021-08-18 03:42:51.369935 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  573. 2021-08-18 03:43:51.370086 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  574. 2021-08-18 03:44:51.369862 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  575. 2021-08-18 03:45:51.370136 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  576. 2021-08-18 03:46:51.370347 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  577. 2021-08-18 03:47:51.370712 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  578. 2021-08-18 03:48:48.580740 I | op-osd: restarting watcher for OSD provisioning status ConfigMaps. the watcher closed the channel
  579. 2021-08-18 03:48:48.608326 I | op-osd: OSD orchestration status for node nodename-3 is "starting"
  580. 2021-08-18 03:48:51.370040 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  581. 2021-08-18 03:49:51.369994 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  582. 2021-08-18 03:50:51.370330 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  583. 2021-08-18 03:51:51.370013 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  584. 2021-08-18 03:52:51.370670 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  585. 2021-08-18 03:53:51.370672 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  586. 2021-08-18 03:54:51.370373 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  587. 2021-08-18 03:55:51.370515 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  588. 2021-08-18 03:56:51.369994 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  589. 2021-08-18 03:57:51.369856 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  590. 2021-08-18 03:58:51.370442 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  591. 2021-08-18 03:59:51.371722 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  592. 2021-08-18 04:00:51.370303 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  593. 2021-08-18 04:01:51.369949 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  594. 2021-08-18 04:02:51.369971 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  595. 2021-08-18 04:03:51.370289 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  596. 2021-08-18 04:04:51.370092 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  597. 2021-08-18 04:05:51.370546 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  598. 2021-08-18 04:06:51.369628 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  599. 2021-08-18 04:07:51.369744 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  600. 2021-08-18 04:08:51.370190 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  601. 2021-08-18 04:09:51.369967 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  602. 2021-08-18 04:10:51.369965 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  603. 2021-08-18 04:11:51.370601 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  604. 2021-08-18 04:12:51.369725 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  605. 2021-08-18 04:13:51.369879 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  606. 2021-08-18 04:14:51.370293 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  607. 2021-08-18 04:15:51.370683 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  608. 2021-08-18 04:16:51.369738 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  609. 2021-08-18 04:17:51.369861 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  610. 2021-08-18 04:18:51.370559 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  611. 2021-08-18 04:19:51.369975 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  612. 2021-08-18 04:20:51.369602 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  613. 2021-08-18 04:21:51.369850 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  614. 2021-08-18 04:22:51.370684 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  615. 2021-08-18 04:23:51.370090 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  616. 2021-08-18 04:24:51.370501 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  617. 2021-08-18 04:25:51.370015 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  618. 2021-08-18 04:26:51.370564 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  619. 2021-08-18 04:27:51.369750 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  620. 2021-08-18 04:28:51.370381 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  621. 2021-08-18 04:29:51.369736 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  622. 2021-08-18 04:30:51.370153 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  623. 2021-08-18 04:31:51.370030 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  624. 2021-08-18 04:32:51.369865 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  625. 2021-08-18 04:33:51.369757 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  626. 2021-08-18 04:34:51.370124 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  627. 2021-08-18 04:35:51.370610 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  628. 2021-08-18 04:36:51.370700 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  629. 2021-08-18 04:37:51.369822 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  630. 2021-08-18 04:38:51.370070 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  631. 2021-08-18 04:39:51.370089 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  632. 2021-08-18 04:40:47.429333 I | op-osd: restarting watcher for OSD provisioning status ConfigMaps. the watcher closed the channel
  633. 2021-08-18 04:40:47.462627 I | op-osd: OSD orchestration status for node nodename-3 is "starting"
  634. 2021-08-18 04:40:51.370565 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  635. 2021-08-18 04:41:51.370460 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  636. 2021-08-18 04:42:51.370267 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  637. 2021-08-18 04:43:51.370599 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  638. 2021-08-18 04:44:51.370113 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  639. 2021-08-18 04:45:51.370627 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  640. 2021-08-18 04:46:51.370391 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  641. 2021-08-18 04:47:51.370046 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  642. 2021-08-18 04:48:51.369852 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  643. 2021-08-18 04:49:51.370042 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  644. 2021-08-18 04:50:51.369787 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  645. 2021-08-18 04:51:51.370418 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  646. 2021-08-18 04:52:51.370703 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  647. 2021-08-18 04:53:51.370142 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  648. 2021-08-18 04:54:51.370335 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  649. 2021-08-18 04:55:51.370317 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  650. 2021-08-18 04:56:51.370133 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  651. 2021-08-18 04:57:51.369690 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  652. 2021-08-18 04:58:51.370458 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  653. 2021-08-18 04:59:51.370576 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  654. 2021-08-18 05:00:51.370751 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  655. 2021-08-18 05:01:51.370270 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  656. 2021-08-18 05:02:51.369954 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  657. 2021-08-18 05:03:51.370447 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  658. 2021-08-18 05:04:51.369700 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  659. 2021-08-18 05:05:51.369596 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  660. 2021-08-18 05:06:51.369738 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  661. 2021-08-18 05:07:51.370404 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  662. 2021-08-18 05:08:51.370771 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  663. 2021-08-18 05:09:51.369859 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  664. 2021-08-18 05:10:51.369705 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  665. 2021-08-18 05:11:51.369742 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  666. 2021-08-18 05:12:51.369727 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  667. 2021-08-18 05:13:51.369722 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  668. 2021-08-18 05:14:51.369736 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  669. 2021-08-18 05:15:51.370487 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  670. 2021-08-18 05:16:51.369961 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  671. 2021-08-18 05:17:51.369623 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  672. 2021-08-18 05:18:51.370664 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  673. 2021-08-18 05:19:51.370695 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  674. 2021-08-18 05:20:51.369692 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  675. 2021-08-18 05:21:51.369642 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  676. 2021-08-18 05:22:51.370335 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  677. 2021-08-18 05:23:51.370294 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  678. 2021-08-18 05:24:51.370612 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  679. 2021-08-18 05:25:51.370418 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  680. 2021-08-18 05:26:51.370117 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  681. 2021-08-18 05:27:51.370120 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  682. 2021-08-18 05:28:51.370073 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  683. 2021-08-18 05:29:51.369789 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  684. 2021-08-18 05:30:51.369822 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  685. 2021-08-18 05:31:51.370612 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  686. 2021-08-18 05:32:51.369626 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  687. 2021-08-18 05:33:51.370171 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  688. 2021-08-18 05:34:51.369603 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  689. 2021-08-18 05:34:57.345031 I | op-osd: restarting watcher for OSD provisioning status ConfigMaps. the watcher closed the channel
  690. 2021-08-18 05:34:57.370975 I | op-osd: OSD orchestration status for node nodename-3 is "starting"
  691. 2021-08-18 05:35:51.369703 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  692. 2021-08-18 05:36:51.369987 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  693. 2021-08-18 05:37:51.369678 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  694. 2021-08-18 05:38:51.370545 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  695. 2021-08-18 05:39:51.370300 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  696. 2021-08-18 05:40:51.370301 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  697. 2021-08-18 05:41:51.370525 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  698. 2021-08-18 05:42:51.369994 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  699. 2021-08-18 05:43:51.369876 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  700. 2021-08-18 05:44:51.370239 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  701. 2021-08-18 05:45:51.369982 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  702. 2021-08-18 05:46:51.369954 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  703. 2021-08-18 05:47:51.370582 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  704. 2021-08-18 05:48:51.369818 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  705. 2021-08-18 05:49:51.370503 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  706. 2021-08-18 05:50:51.370502 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  707. 2021-08-18 05:51:51.370327 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  708. 2021-08-18 05:52:51.370138 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  709. 2021-08-18 05:53:51.370520 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  710. 2021-08-18 05:54:51.369951 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  711. 2021-08-18 05:55:51.370254 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  712. 2021-08-18 05:56:51.369976 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  713. 2021-08-18 05:57:51.372732 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  714. 2021-08-18 05:58:51.369907 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  715. 2021-08-18 05:59:51.370070 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  716. 2021-08-18 06:00:51.369810 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  717. 2021-08-18 06:01:51.369726 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  718. 2021-08-18 06:02:51.370224 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  719. 2021-08-18 06:03:51.369643 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  720. 2021-08-18 06:04:51.369934 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  721. 2021-08-18 06:05:51.369925 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  722. 2021-08-18 06:06:51.369811 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  723. 2021-08-18 06:07:51.370598 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  724. 2021-08-18 06:08:51.370570 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  725. 2021-08-18 06:09:51.370030 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  726. 2021-08-18 06:10:51.370181 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  727. 2021-08-18 06:11:51.370364 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  728. 2021-08-18 06:12:51.370708 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  729. 2021-08-18 06:13:51.369714 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  730. 2021-08-18 06:14:34.410167 I | op-osd: restarting watcher for OSD provisioning status ConfigMaps. the watcher closed the channel
  731. 2021-08-18 06:14:34.442529 I | op-osd: OSD orchestration status for node nodename-3 is "starting"
  732. 2021-08-18 06:14:51.369925 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  733. 2021-08-18 06:15:51.370181 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  734. 2021-08-18 06:16:51.370198 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  735. 2021-08-18 06:17:51.370729 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  736. 2021-08-18 06:18:51.370401 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  737. 2021-08-18 06:19:51.369954 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  738. 2021-08-18 06:20:51.370304 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  739. 2021-08-18 06:21:51.370158 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  740. 2021-08-18 06:22:51.370077 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  741. 2021-08-18 06:23:51.370514 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  742. 2021-08-18 06:24:51.370606 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  743. 2021-08-18 06:25:51.369732 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  744. 2021-08-18 06:26:51.370633 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  745. 2021-08-18 06:27:51.369860 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  746. 2021-08-18 06:28:51.370694 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  747. 2021-08-18 06:29:51.370092 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  748. 2021-08-18 06:30:51.370726 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  749. 2021-08-18 06:31:51.370767 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  750. 2021-08-18 06:32:51.370236 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  751. 2021-08-18 06:33:51.369706 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  752. 2021-08-18 06:34:51.370279 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  753. 2021-08-18 06:35:51.370725 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  754. 2021-08-18 06:36:51.370392 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  755. 2021-08-18 06:37:51.369825 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  756. 2021-08-18 06:38:51.369901 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  757. 2021-08-18 06:39:51.370677 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  758. 2021-08-18 06:40:51.370038 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  759. 2021-08-18 06:41:51.370109 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  760. 2021-08-18 06:42:51.369912 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  761. 2021-08-18 06:43:51.370116 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  762. 2021-08-18 06:44:51.369901 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  763. 2021-08-18 06:45:51.370595 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  764. 2021-08-18 06:46:51.370008 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  765. 2021-08-18 06:47:51.370591 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  766. 2021-08-18 06:48:51.369743 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  767. 2021-08-18 06:49:51.381917 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  768. 2021-08-18 06:50:51.370694 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  769. 2021-08-18 06:51:51.370283 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  770. 2021-08-18 06:52:51.370198 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  771. 2021-08-18 06:53:51.370526 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  772. 2021-08-18 06:54:51.369670 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  773. 2021-08-18 06:55:51.370308 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  774. 2021-08-18 06:56:51.370057 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  775. 2021-08-18 06:57:51.369995 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  776. 2021-08-18 06:58:51.369863 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  777. 2021-08-18 06:59:51.369825 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  778. 2021-08-18 07:00:51.370271 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  779. 2021-08-18 07:01:51.370340 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  780. 2021-08-18 07:02:51.370031 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  781. 2021-08-18 07:03:51.370726 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  782. 2021-08-18 07:04:51.369665 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  783. 2021-08-18 07:05:19.701803 I | op-osd: restarting watcher for OSD provisioning status ConfigMaps. the watcher closed the channel
  784. 2021-08-18 07:05:19.716313 I | op-osd: OSD orchestration status for node nodename-3 is "starting"
  785. 2021-08-18 07:05:51.369921 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  786. 2021-08-18 07:06:51.369618 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  787. 2021-08-18 07:07:51.370278 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  788. 2021-08-18 07:08:51.370572 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  789. 2021-08-18 07:09:51.370236 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  790. 2021-08-18 07:10:51.369736 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  791. 2021-08-18 07:11:51.370576 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  792. 2021-08-18 07:12:51.369745 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  793. 2021-08-18 07:13:51.369644 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  794. 2021-08-18 07:14:51.370027 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  795. 2021-08-18 07:15:51.369968 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  796. 2021-08-18 07:16:51.370476 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  797. 2021-08-18 07:17:51.370571 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  798. 2021-08-18 07:18:51.370213 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  799. 2021-08-18 07:19:51.370580 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  800. 2021-08-18 07:20:51.369945 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  801. 2021-08-18 07:21:51.370501 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  802. 2021-08-18 07:22:51.369855 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  803. 2021-08-18 07:23:51.370735 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  804. 2021-08-18 07:24:51.369881 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  805. 2021-08-18 07:25:51.370534 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  806. 2021-08-18 07:26:51.369669 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  807. 2021-08-18 07:27:51.369903 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  808. 2021-08-18 07:28:51.369716 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  809. 2021-08-18 07:29:51.369954 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  810. 2021-08-18 07:30:51.370702 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  811. 2021-08-18 07:31:51.369761 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  812. 2021-08-18 07:32:51.369852 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  813. 2021-08-18 07:33:51.370161 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  814. 2021-08-18 07:34:51.370038 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  815. 2021-08-18 07:35:51.370621 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  816. 2021-08-18 07:36:51.370023 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  817. 2021-08-18 07:37:51.370692 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  818. 2021-08-18 07:38:51.370726 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  819. 2021-08-18 07:39:51.370147 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  820. 2021-08-18 07:40:51.370370 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  821. 2021-08-18 07:41:51.370686 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  822. 2021-08-18 07:42:51.370590 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  823. 2021-08-18 07:43:51.370111 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  824. 2021-08-18 07:44:43.520163 I | op-osd: restarting watcher for OSD provisioning status ConfigMaps. the watcher closed the channel
  825. 2021-08-18 07:44:43.536322 I | op-osd: OSD orchestration status for node nodename-3 is "starting"
  826. 2021-08-18 07:44:51.370676 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  827. 2021-08-18 07:45:51.370358 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  828. 2021-08-18 07:46:51.370748 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  829. 2021-08-18 07:47:51.369687 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  830. 2021-08-18 07:48:51.370311 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  831. 2021-08-18 07:49:51.369861 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  832. 2021-08-18 07:50:51.370605 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  833. 2021-08-18 07:51:51.370721 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  834. 2021-08-18 07:52:51.370386 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  835. 2021-08-18 07:53:51.369751 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  836. 2021-08-18 07:54:51.369868 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  837. 2021-08-18 07:55:51.370269 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  838. 2021-08-18 07:56:51.370020 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  839. 2021-08-18 07:57:51.370189 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  840. 2021-08-18 07:58:51.370292 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  841. 2021-08-18 07:59:51.370983 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  842. 2021-08-18 08:00:51.370137 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  843. 2021-08-18 08:01:51.370546 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  844. 2021-08-18 08:02:51.369731 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  845. 2021-08-18 08:03:51.369747 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  846. 2021-08-18 08:04:51.370693 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  847. 2021-08-18 08:05:51.369917 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  848. 2021-08-18 08:06:51.369966 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  849. 2021-08-18 08:07:51.370573 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  850. 2021-08-18 08:08:51.369853 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  851. 2021-08-18 08:09:51.370228 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  852. 2021-08-18 08:10:51.369691 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  853. 2021-08-18 08:11:51.370689 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  854. 2021-08-18 08:12:51.370244 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  855. 2021-08-18 08:13:51.370937 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  856. 2021-08-18 08:14:51.370605 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  857. 2021-08-18 08:15:51.370618 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  858. 2021-08-18 08:16:51.369716 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  859. 2021-08-18 08:17:51.370096 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  860. 2021-08-18 08:18:51.370478 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  861. 2021-08-18 08:19:51.370086 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  862. 2021-08-18 08:20:51.369897 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  863. 2021-08-18 08:21:51.370047 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  864. 2021-08-18 08:22:51.369866 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  865. 2021-08-18 08:23:51.370060 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  866. 2021-08-18 08:24:51.369965 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  867. 2021-08-18 08:25:51.370684 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  868. 2021-08-18 08:26:51.369926 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  869. 2021-08-18 08:27:51.369604 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  870. 2021-08-18 08:28:51.369616 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  871. 2021-08-18 08:29:51.370401 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  872. 2021-08-18 08:30:51.369950 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  873. 2021-08-18 08:31:51.370453 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  874. 2021-08-18 08:32:51.370353 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  875. 2021-08-18 08:33:51.370399 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  876. 2021-08-18 08:34:51.370043 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  877. 2021-08-18 08:35:51.370238 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  878. 2021-08-18 08:36:51.370320 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  879. 2021-08-18 08:37:51.369640 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  880. 2021-08-18 08:38:51.369825 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  881. 2021-08-18 08:39:51.369989 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  882. 2021-08-18 08:40:51.370569 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
  883. 2021-08-18 08:40:58.469007 I | op-osd: restarting watcher for OSD provisioning status ConfigMaps. the watcher closed the channel
  884. 2021-08-18 08:40:58.491287 I | op-osd: OSD orchestration status for node nodename-3 is "starting"
  885. 2021-08-18 08:41:51.369721 I | op-osd: waiting... 2 of 3 OSD prepare jobs have finished processing and 3 of 3 OSDs have been updated
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement
Public Pastes
Advertisement
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!