2026/04/22 13:36:31 maxprocs: Updating GOMAXPROCS=1: determined from CPU quota
2026-04-22 13:36:31.258626 I | rookcmd: starting Rook v1.14.5 with arguments '/usr/local/bin/rook ceph operator'
2026-04-22 13:36:31.258648 I | rookcmd: flag values: --enable-machine-disruption-budget=false, --help=false, --kubeconfig=, --log-level=INFO
2026-04-22 13:36:31.258650 I | cephcmd: starting Rook-Ceph operator
2026-04-22 13:36:31.395277 I | cephcmd: base ceph version inside the rook operator image is "ceph version 18.2.2 (531c0d11a1c5d39fbfe6aa8a521f023abf3bf3e2) reef (stable)"
2026-04-22 13:36:31.403823 I | op-k8sutil: ROOK_CURRENT_NAMESPACE_ONLY="false" (env var)
2026-04-22 13:36:31.403852 I | operator: watching all namespaces for Ceph CRs
2026-04-22 13:36:31.403972 I | operator: setting up schemes
2026-04-22 13:36:31.406268 I | operator: setting up the controller-runtime manager
2026-04-22 13:36:31.406647 I | ceph-cluster-controller: successfully started
2026-04-22 13:36:31.409906 I | op-k8sutil: ROOK_DISABLE_DEVICE_HOTPLUG="false" (env var)
2026-04-22 13:36:31.409922 I | ceph-cluster-controller: enabling hotplug orchestration
2026-04-22 13:36:31.409944 I | ceph-nodedaemon-controller: successfully started
2026-04-22 13:36:31.409961 I | ceph-block-pool-controller: successfully started
2026-04-22 13:36:31.409976 I | ceph-object-store-user-controller: successfully started
2026-04-22 13:36:31.409991 I | ceph-object-realm-controller: successfully started
2026-04-22 13:36:31.410006 I | ceph-object-zonegroup-controller: successfully started
2026-04-22 13:36:31.410024 I | ceph-object-zone-controller: successfully started
2026-04-22 13:36:31.410115 I | ceph-object-controller: successfully started
2026-04-22 13:36:31.410143 I | ceph-file-controller: successfully started
2026-04-22 13:36:31.410164 I | ceph-nfs-controller: successfully started
2026-04-22 13:36:31.410191 I | ceph-rbd-mirror-controller: successfully started
2026-04-22 13:36:31.410214 I | ceph-client-controller: successfully started
2026-04-22 13:36:31.410228 I | ceph-filesystem-mirror-controller: successfully started
2026-04-22 13:36:31.410242 I | operator: rook-ceph-operator-config-controller successfully started
2026-04-22 13:36:31.410254 I | ceph-csi: rook-ceph-operator-csi-controller successfully started
2026-04-22 13:36:31.410336 I | op-bucket-prov: rook-ceph-operator-bucket-controller successfully started
2026-04-22 13:36:31.410350 I | ceph-bucket-topic: successfully started
2026-04-22 13:36:31.410358 I | ceph-bucket-notification: successfully started
2026-04-22 13:36:31.410371 I | ceph-bucket-notification: successfully started
2026-04-22 13:36:31.410380 I | ceph-fs-subvolumegroup-controller: successfully started
2026-04-22 13:36:31.410390 I | blockpool-rados-namespace-controller: successfully started
2026-04-22 13:36:31.410397 I | ceph-cosi-controller: successfully started
2026-04-22 13:36:31.410411 I | operator: starting the controller-runtime manager
2026-04-22 13:36:31.546354 I | op-k8sutil: ROOK_WATCH_FOR_NODE_FAILURE="true" (default)
2026-04-22 13:36:31.633419 I | op-k8sutil: ROOK_CEPH_COMMANDS_TIMEOUT_SECONDS="15" (configmap)
2026-04-22 13:36:31.633459 I | op-k8sutil: ROOK_LOG_LEVEL="INFO" (configmap)
2026-04-22 13:36:31.633479 I | op-k8sutil: ROOK_ENABLE_DISCOVERY_DAEMON="false" (configmap)
2026-04-22 13:36:31.636635 I | op-k8sutil: ROOK_CEPH_ALLOW_LOOP_DEVICES="false" (configmap)
2026-04-22 13:36:31.636659 I | operator: rook-ceph-operator-config-controller done reconciling
2026-04-22 13:36:31.657419 I | ceph-csi: successfully created csi config map "rook-ceph-csi-config"
2026-04-22 13:36:31.657462 I | op-k8sutil: ROOK_CSI_DISABLE_DRIVER="false" (configmap)
2026-04-22 13:36:31.658528 I | ceph-csi: CSI Ceph RBD driver disabled
2026-04-22 13:36:31.658539 I | op-k8sutil: removing daemonset csi-rbdplugin if it exists
2026-04-22 13:36:31.661095 I | op-k8sutil: removing deployment csi-rbdplugin-provisioner if it exists
2026-04-22 13:36:31.667312 I | ceph-csi: successfully removed CSI Ceph RBD driver
2026-04-22 13:36:31.667325 I | ceph-csi: CSI CephFS driver disabled
2026-04-22 13:36:31.667329 I | op-k8sutil: removing daemonset csi-cephfsplugin if it exists
2026-04-22 13:36:31.669728 I | op-k8sutil: removing deployment csi-cephfsplugin-provisioner if it exists
2026-04-22 13:36:31.680159 I | ceph-csi: successfully removed CSI CephFS driver
2026-04-22 13:36:31.680194 I | ceph-csi: CSI NFS driver disabled
2026-04-22 13:36:31.680206 I | op-k8sutil: removing daemonset csi-nfsplugin if it exists
2026-04-22 13:36:31.682215 I | op-k8sutil: removing deployment csi-nfsplugin-provisioner if it exists
2026-04-22 13:36:31.687661 I | ceph-csi: successfully removed CSI NFS driver
2026-04-22 13:36:53.534334 I | ceph-spec: adding finalizer "cephcluster.ceph.rook.io" on "ceph"
2026-04-22 13:36:53.540192 I | ceph-spec: adding finalizer "cephobjectstore.ceph.rook.io" on "ceph"
2026-04-22 13:36:53.546669 I | clusterdisruption-controller: deleted all legacy node drain canary pods
2026-04-22 13:36:53.551683 I | ceph-cluster-controller: reconciling ceph cluster in namespace "openstack"
2026-04-22 13:36:53.558753 I | ceph-spec: parsing mon endpoints: instance=10.96.240.200:6789
2026-04-22 13:36:53.558786 I | ceph-spec: updating obsolete maxMonID 0 to actual value 76846025234
2026-04-22 13:36:53.558870 I | op-k8sutil: ROOK_OBC_WATCH_OPERATOR_NAMESPACE="true" (configmap)
2026-04-22 13:36:53.558888 I | op-k8sutil: ROOK_OBC_PROVISIONER_NAME_PREFIX="" (default)
2026-04-22 13:36:53.558897 I | op-bucket-prov: ceph bucket provisioner launched watching for provisioner "openstack.ceph.rook.io/bucket"
2026-04-22 13:36:53.559388 I | op-bucket-prov: successfully reconciled bucket provisioner
I0422 13:36:53.559475       1 manager.go:135] "msg"="starting provisioner" "logger"="objectbucket.io/provisioner-manager" "name"="openstack.ceph.rook.io/bucket"
2026-04-22 13:36:53.570782 I | ceph-spec: parsing mon endpoints: instance=10.96.240.200:6789
2026-04-22 13:36:53.570811 I | ceph-spec: updating obsolete maxMonID 0 to actual value 76846025234
2026-04-22 13:36:53.570834 I | ceph-spec: found the cluster info to connect to the external cluster. will use "client.admin" to check health and monitor status. mons=map[instance:0xc00184ff50]
2026-04-22 13:36:53.570856 I | ceph-spec: detecting the ceph image version for image harbor.atmosphere.dev/quay.io/ceph/ceph:v18.2.7...
2026-04-22 13:36:53.941322 I | op-k8sutil: CSI_ENABLE_HOST_NETWORK="true" (configmap)
2026-04-22 13:36:53.941352 I | op-k8sutil: CSI_DISABLE_HOLDER_PODS="true" (configmap)
2026-04-22 13:36:54.340804 I | ceph-spec: parsing mon endpoints: instance=10.96.240.200:6789
2026-04-22 13:36:54.340834 I | ceph-spec: updating obsolete maxMonID 0 to actual value 76846025234
2026-04-22 13:36:54.540594 I | op-k8sutil: ROOK_CSI_ENABLE_RBD="false" (configmap)
2026-04-22 13:36:54.540680 I | op-k8sutil: ROOK_CSI_ENABLE_CEPHFS="false" (configmap)
2026-04-22 13:36:54.540702 I | op-k8sutil: ROOK_CSI_ENABLE_NFS="false" (configmap)
2026-04-22 13:36:54.540721 I | op-k8sutil: ROOK_CSI_ALLOW_UNSUPPORTED_VERSION="false" (default)
2026-04-22 13:36:54.540756 I | op-k8sutil: CSI_FORCE_CEPHFS_KERNEL_CLIENT="true" (configmap)
2026-04-22 13:36:54.540788 I | op-k8sutil: CSI_GRPC_TIMEOUT_SECONDS="150" (configmap)
2026-04-22 13:36:54.540809 I | op-k8sutil: CSI_CEPHFS_LIVENESS_METRICS_PORT="9081" (default)
2026-04-22 13:36:54.540828 I | op-k8sutil: CSIADDONS_PORT="9070" (default)
2026-04-22 13:36:54.540865 I | op-k8sutil: CSI_RBD_LIVENESS_METRICS_PORT="9080" (default)
2026-04-22 13:36:54.540890 I | op-k8sutil: CSI_ENABLE_LIVENESS="false" (default)
2026-04-22 13:36:54.540912 I | op-k8sutil: CSI_PLUGIN_PRIORITY_CLASSNAME="system-node-critical" (configmap)
2026-04-22 13:36:54.540935 I | op-k8sutil: CSI_PROVISIONER_PRIORITY_CLASSNAME="system-cluster-critical" (configmap)
2026-04-22 13:36:54.540974 I | op-k8sutil: CSI_ENABLE_OMAP_GENERATOR="false" (configmap)
2026-04-22 13:36:54.541002 I | op-k8sutil: CSI_ENABLE_RBD_SNAPSHOTTER="true" (configmap)
2026-04-22 13:36:54.541024 I | op-k8sutil: CSI_ENABLE_CEPHFS_SNAPSHOTTER="true" (configmap)
2026-04-22 13:36:54.541042 I | op-k8sutil: CSI_ENABLE_NFS_SNAPSHOTTER="true" (configmap)
2026-04-22 13:36:54.543290 I | op-k8sutil: CSI_ENABLE_CSIADDONS="false" (configmap)
2026-04-22 13:36:54.543338 I | op-k8sutil: CSI_ENABLE_TOPOLOGY="false" (configmap)
2026-04-22 13:36:54.543357 I | op-k8sutil: CSI_ENABLE_ENCRYPTION="false" (configmap)
2026-04-22 13:36:54.543374 I | op-k8sutil: CSI_ENABLE_METADATA="false" (configmap)
2026-04-22 13:36:54.543392 I | op-k8sutil: CSI_CEPHFS_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default)
2026-04-22 13:36:54.543426 I | op-k8sutil: CSI_CEPHFS_PLUGIN_UPDATE_STRATEGY_MAX_UNAVAILABLE="1" (default)
2026-04-22 13:36:54.543450 I | op-k8sutil: CSI_NFS_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default)
2026-04-22 13:36:54.543469 I | op-k8sutil: CSI_RBD_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default)
2026-04-22 13:36:54.543486 I | op-k8sutil: CSI_RBD_PLUGIN_UPDATE_STRATEGY_MAX_UNAVAILABLE="1" (default)
2026-04-22 13:36:54.543503 I | op-k8sutil: CSI_PLUGIN_ENABLE_SELINUX_HOST_MOUNT="false" (configmap)
2026-04-22 13:36:54.543535 I | ceph-csi: Kubernetes version is 1.28
2026-04-22 13:36:54.543586 I | op-k8sutil: CSI_LOG_LEVEL="" (default)
2026-04-22 13:36:54.543605 I | op-k8sutil: CSI_SIDECAR_LOG_LEVEL="" (default)
2026-04-22 13:36:54.543638 I | op-k8sutil: CSI_LEADER_ELECTION_LEASE_DURATION="" (default)
2026-04-22 13:36:54.543666 I | op-k8sutil: CSI_LEADER_ELECTION_RENEW_DEADLINE="" (default)
2026-04-22 13:36:54.543687 I | op-k8sutil: CSI_LEADER_ELECTION_RETRY_PERIOD="" (default)
2026-04-22 13:36:55.058032 I | op-k8sutil: ROOK_CSI_CEPH_IMAGE="quay.io/cephcsi/cephcsi:v3.11.0" (configmap)
2026-04-22 13:36:55.058080 I | op-k8sutil: ROOK_CSI_REGISTRAR_IMAGE="registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.10.1" (configmap)
2026-04-22 13:36:55.058092 I | op-k8sutil: ROOK_CSI_PROVISIONER_IMAGE="registry.k8s.io/sig-storage/csi-provisioner:v4.0.1" (configmap)
2026-04-22 13:36:55.058100 I | op-k8sutil: ROOK_CSI_ATTACHER_IMAGE="registry.k8s.io/sig-storage/csi-attacher:v4.5.1" (configmap)
2026-04-22 13:36:55.058107 I | op-k8sutil: ROOK_CSI_SNAPSHOTTER_IMAGE="registry.k8s.io/sig-storage/csi-snapshotter:v7.0.2" (configmap)
2026-04-22 13:36:55.058122 I | op-k8sutil: ROOK_CSI_RESIZER_IMAGE="registry.k8s.io/sig-storage/csi-resizer:v1.10.1" (configmap)
2026-04-22 13:36:55.058137 I | op-k8sutil: ROOK_CSI_KUBELET_DIR_PATH="/var/lib/kubelet" (default)
2026-04-22 13:36:55.058148 I | op-k8sutil: ROOK_CSIADDONS_IMAGE="quay.io/csiaddons/k8s-sidecar:v0.8.0" (configmap)
2026-04-22 13:36:55.058167 I | op-k8sutil: CSI_TOPOLOGY_DOMAIN_LABELS="" (default)
2026-04-22 13:36:55.058176 I | op-k8sutil: ROOK_CSI_CEPHFS_POD_LABELS="" (default)
2026-04-22 13:36:55.058189 I | op-k8sutil: ROOK_CSI_NFS_POD_LABELS="" (default)
2026-04-22 13:36:55.058196 I | op-k8sutil: ROOK_CSI_RBD_POD_LABELS="" (default)
2026-04-22 13:36:55.058203 I | op-k8sutil: CSI_CLUSTER_NAME="" (default)
2026-04-22 13:36:55.058210 I | op-k8sutil: ROOK_CSI_IMAGE_PULL_POLICY="IfNotPresent" (configmap)
2026-04-22 13:36:55.058216 I | op-k8sutil: CSI_CEPHFS_KERNEL_MOUNT_OPTIONS="" (default)
2026-04-22 13:36:55.058222 I | op-k8sutil: CSI_CEPHFS_ATTACH_REQUIRED="true" (configmap)
2026-04-22 13:36:55.058229 I | op-k8sutil: CSI_RBD_ATTACH_REQUIRED="true" (configmap)
2026-04-22 13:36:55.058235 I | op-k8sutil: CSI_NFS_ATTACH_REQUIRED="true" (configmap)
2026-04-22 13:36:55.058241 I | op-k8sutil: CSI_DRIVER_NAME_PREFIX="rook-ceph" (default)
2026-04-22 13:36:55.061365 I | op-k8sutil: CSI_ENABLE_VOLUME_GROUP_SNAPSHOT="true" (configmap)
2026-04-22 13:36:55.061382 I | ceph-csi: skipping csi version check, since unsupported versions are allowed or csi is disabled
2026-04-22 13:36:55.061387 I | ceph-csi: CSI Ceph RBD driver disabled
2026-04-22 13:36:55.061391 I | op-k8sutil: removing daemonset csi-rbdplugin if it exists
2026-04-22 13:36:55.064428 I | op-k8sutil: removing deployment csi-rbdplugin-provisioner if it exists
2026-04-22 13:36:55.075092 I | ceph-csi: successfully removed CSI Ceph RBD driver
2026-04-22 13:36:55.075112 I | ceph-csi: CSI CephFS driver disabled
2026-04-22 13:36:55.075119 I | op-k8sutil: removing daemonset csi-cephfsplugin if it exists
2026-04-22 13:36:55.077809 I | op-k8sutil: removing deployment csi-cephfsplugin-provisioner if it exists
2026-04-22 13:36:55.143131 I | ceph-csi: successfully removed CSI CephFS driver
2026-04-22 13:36:55.143149 I | ceph-csi: CSI NFS driver disabled
2026-04-22 13:36:55.143153 I | op-k8sutil: removing daemonset csi-nfsplugin if it exists
2026-04-22 13:36:55.145864 I | op-k8sutil: removing deployment csi-nfsplugin-provisioner if it exists
2026-04-22 13:36:55.343789 I | ceph-csi: successfully removed CSI NFS driver
2026-04-22 13:37:16.770240 I | ceph-spec: detected ceph image version: "18.2.7-0 reef"
2026-04-22 13:37:16.770263 I | ceph-cluster-controller: validating ceph version from provided image
2026-04-22 13:37:17.652233 I | ceph-spec: parsing mon endpoints: instance=10.96.240.200:6789
2026-04-22 13:37:17.652320 I | ceph-spec: updating obsolete maxMonID 0 to actual value 76846025234
2026-04-22 13:37:17.935335 I | cephclient: writing config file /var/lib/rook/openstack/openstack.config
2026-04-22 13:37:17.935532 I | cephclient: generated admin config in /var/lib/rook/openstack
2026-04-22 13:37:19.335104 E | cephver: external cluster ceph version is a major version higher "18.2.7-0 reef" than the local cluster "0.0.0-0 <unknown version>", consider upgrading
2026-04-22 13:37:22.804916 I | ceph-cluster-controller: cluster "openstack": version "18.2.7-0 reef" detected for image "harbor.atmosphere.dev/quay.io/ceph/ceph:v18.2.7"
2026-04-22 13:37:23.931527 I | ceph-cluster-controller: creating "rook-config-override" configmap
2026-04-22 13:37:24.184087 I | ceph-cluster-controller: creating "rook-ceph-config" secret
2026-04-22 13:37:24.807592 I | ceph-cluster-controller: external cluster identity established
2026-04-22 13:37:24.807613 I | cephclient: getting or creating ceph auth key "client.csi-rbd-provisioner"
2026-04-22 13:37:25.192997 I | clusterdisruption-controller: all PGs are active+clean. Restoring default OSD pdb settings
2026-04-22 13:37:25.193027 I | clusterdisruption-controller: creating the default pdb "rook-ceph-osd" with maxUnavailable=1 for all osd
2026-04-22 13:37:25.497507 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-22 13:37:26.528468 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-22 13:37:27.401304 I | cephclient: getting or creating ceph auth key "client.csi-rbd-node"
2026-04-22 13:37:27.987664 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-provisioner"
2026-04-22 13:37:28.614710 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-node"
2026-04-22 13:37:29.232555 I | ceph-csi: created kubernetes csi secrets for cluster "openstack"
2026-04-22 13:37:29.242340 I | ceph-cluster-controller: successfully updated csi config map
2026-04-22 13:37:29.242372 I | cephclient: getting or creating ceph auth key "client.crash"
2026-04-22 13:37:29.848915 I | ceph-nodedaemon-controller: created kubernetes crash collector secret for cluster "openstack"
2026-04-22 13:37:29.854686 I | ceph-cluster-controller: enabling ceph mon monitoring goroutine for cluster "openstack"
2026-04-22 13:37:29.854736 I | ceph-cluster-controller: ceph status check interval is 1m0s
2026-04-22 13:37:29.854746 I | ceph-cluster-controller: enabling ceph status monitoring goroutine for cluster "openstack"
2026-04-22 13:37:33.945207 I | ceph-spec: parsing mon endpoints: instance=10.96.240.200:6789
2026-04-22 13:37:33.945235 I | ceph-spec: updating obsolete maxMonID 0 to actual value 76846025234
2026-04-22 13:37:33.945248 I | ceph-spec: detecting the ceph image version for image harbor.atmosphere.dev/quay.io/ceph/ceph:v18.2.7...
2026-04-22 13:37:37.163672 I | ceph-spec: detected ceph image version: "18.2.7-0 reef"
2026-04-22 13:37:38.676367 I | ceph-object-controller: reconciling object store deployments
2026-04-22 13:37:38.695243 I | ceph-object-controller: ceph object store gateway service running at 10.96.199.179
2026-04-22 13:37:38.695277 I | ceph-object-controller: reconciling object store pools
2026-04-22 13:37:43.274992 I | cephclient: reconciling replicated pool ceph.rgw.control succeeded
2026-04-22 13:37:44.302404 I | cephclient: creating a new crush rule for changed deviceClass ("default"-->"") on crush rule "ceph.rgw.control"
2026-04-22 13:37:44.302438 I | cephclient: updating pool "ceph.rgw.control" failure domain from "osd" to "osd" with new crush rule "ceph.rgw.control_osd"
2026-04-22 13:37:44.302446 I | cephclient: crush rule "ceph.rgw.control" will no longer be used by pool "ceph.rgw.control"
2026-04-22 13:37:46.333176 I | cephclient: Successfully updated pool "ceph.rgw.control" failure domain to "osd"
2026-04-22 13:37:46.333210 I | cephclient: setting pool property "pg_num_min" to "8" on pool "ceph.rgw.control"
2026-04-22 13:37:53.470118 I | cephclient: reconciling replicated pool ceph.rgw.meta succeeded
2026-04-22 13:37:54.375474 I | cephclient: creating a new crush rule for changed deviceClass ("default"-->"") on crush rule "ceph.rgw.meta"
2026-04-22 13:37:54.375512 I | cephclient: updating pool "ceph.rgw.meta" failure domain from "osd" to "osd" with new crush rule "ceph.rgw.meta_osd"
2026-04-22 13:37:54.375520 I | cephclient: crush rule "ceph.rgw.meta" will no longer be used by pool "ceph.rgw.meta"
2026-04-22 13:37:56.571178 I | clusterdisruption-controller: all "host" failure domains: []. osd is down in failure domain: "". active node drains: false. pg health: "cluster is not fully clean. PGs: [{StateName:active+clean Count:11} {StateName:unknown Count:3} {StateName:creating+peering Count:3}]"
2026-04-22 13:37:56.576143 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-22 13:37:57.114058 I | cephclient: Successfully updated pool "ceph.rgw.meta" failure domain to "osd"
2026-04-22 13:37:57.114090 I | cephclient: setting pool property "pg_num_min" to "8" on pool "ceph.rgw.meta"
2026-04-22 13:37:57.125890 I | clusterdisruption-controller: all "host" failure domains: []. osd is down in failure domain: "". active node drains: false. pg health: "cluster is not fully clean. PGs: [{StateName:active+clean Count:14} {StateName:creating+peering Count:3}]"
2026-04-22 13:37:57.129675 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-22 13:38:01.150512 I | cephclient: reconciling replicated pool ceph.rgw.log succeeded
2026-04-22 13:38:02.105976 I | cephclient: creating a new crush rule for changed deviceClass ("default"-->"") on crush rule "ceph.rgw.log"
2026-04-22 13:38:02.106002 I | cephclient: updating pool "ceph.rgw.log" failure domain from "osd" to "osd" with new crush rule "ceph.rgw.log_osd"
2026-04-22 13:38:02.106006 I | cephclient: crush rule "ceph.rgw.log" will no longer be used by pool "ceph.rgw.log"
2026-04-22 13:38:04.212309 I | cephclient: Successfully updated pool "ceph.rgw.log" failure domain to "osd"
2026-04-22 13:38:04.212336 I | cephclient: setting pool property "pg_num_min" to "8" on pool "ceph.rgw.log"
2026-04-22 13:38:08.333731 I | cephclient: reconciling replicated pool ceph.rgw.buckets.index succeeded
2026-04-22 13:38:09.253894 I | cephclient: creating a new crush rule for changed deviceClass ("default"-->"") on crush rule "ceph.rgw.buckets.index"
2026-04-22 13:38:09.253928 I | cephclient: updating pool "ceph.rgw.buckets.index" failure domain from "osd" to "osd" with new crush rule "ceph.rgw.buckets.index_osd"
2026-04-22 13:38:09.253967 I | cephclient: crush rule "ceph.rgw.buckets.index" will no longer be used by pool "ceph.rgw.buckets.index"
2026-04-22 13:38:11.386403 I | cephclient: Successfully updated pool "ceph.rgw.buckets.index" failure domain to "osd"
2026-04-22 13:38:11.386434 I | cephclient: setting pool property "pg_num_min" to "8" on pool "ceph.rgw.buckets.index"
2026-04-22 13:38:16.481252 I | cephclient: reconciling replicated pool ceph.rgw.buckets.non-ec succeeded
2026-04-22 13:38:16.809320 W | op-mon: external cluster mon count is 1, consider adding new monitors.
2026-04-22 13:38:17.596530 I | cephclient: creating a new crush rule for changed deviceClass ("default"-->"") on crush rule "ceph.rgw.buckets.non-ec"
2026-04-22 13:38:17.596556 I | cephclient: updating pool "ceph.rgw.buckets.non-ec" failure domain from "osd" to "osd" with new crush rule "ceph.rgw.buckets.non-ec_osd"
2026-04-22 13:38:17.596562 I | cephclient: crush rule "ceph.rgw.buckets.non-ec" will no longer be used by pool "ceph.rgw.buckets.non-ec"
2026-04-22 13:38:19.501412 I | cephclient: Successfully updated pool "ceph.rgw.buckets.non-ec" failure domain to "osd"
2026-04-22 13:38:19.501447 I | cephclient: setting pool property "pg_num_min" to "8" on pool "ceph.rgw.buckets.non-ec"
2026-04-22 13:38:24.453962 I | cephclient: reconciling replicated pool ceph.rgw.otp succeeded
2026-04-22 13:38:25.365948 I | cephclient: creating a new crush rule for changed deviceClass ("default"-->"") on crush rule "ceph.rgw.otp"
2026-04-22 13:38:25.365988 I | cephclient: updating pool "ceph.rgw.otp" failure domain from "osd" to "osd" with new crush rule "ceph.rgw.otp_osd"
2026-04-22 13:38:25.365998 I | cephclient: crush rule "ceph.rgw.otp" will no longer be used by pool "ceph.rgw.otp"
2026-04-22 13:38:27.619026 I | cephclient: Successfully updated pool "ceph.rgw.otp" failure domain to "osd"
2026-04-22 13:38:27.619062 I | cephclient: setting pool property "pg_num_min" to "8" on pool "ceph.rgw.otp"
2026-04-22 13:38:27.714876 I | clusterdisruption-controller: all "host" failure domains: []. osd is down in failure domain: "". active node drains: false. pg health: "cluster is not fully clean. PGs: [{StateName:active+clean Count:43} {StateName:unknown Count:3} {StateName:creating+peering Count:3}]"
2026-04-22 13:38:27.719682 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-22 13:38:28.805547 I | clusterdisruption-controller: all "host" failure domains: []. osd is down in failure domain: "". active node drains: false. pg health: "cluster is not fully clean. PGs: [{StateName:active+clean Count:46} {StateName:creating+peering Count:3}]"
2026-04-22 13:38:28.809605 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-22 13:38:36.264741 I | cephclient: reconciling replicated pool .rgw.root succeeded
2026-04-22 13:38:37.249188 I | cephclient: creating a new crush rule for changed deviceClass ("default"-->"") on crush rule ".rgw.root"
2026-04-22 13:38:37.249208 I | cephclient: updating pool ".rgw.root" failure domain from "osd" to "osd" with new crush rule ".rgw.root_osd"
2026-04-22 13:38:37.249212 I | cephclient: crush rule ".rgw.root" will no longer be used by pool ".rgw.root"
2026-04-22 13:38:39.614482 I | cephclient: Successfully updated pool ".rgw.root" failure domain to "osd"
2026-04-22 13:38:39.614521 I | cephclient: setting pool property "pg_num_min" to "8" on pool ".rgw.root"
2026-04-22 13:38:43.672129 I | cephclient: reconciling replicated pool ceph.rgw.buckets.data succeeded
2026-04-22 13:38:44.537481 I | cephclient: creating a new crush rule for changed deviceClass ("default"-->"") on crush rule "ceph.rgw.buckets.data"
2026-04-22 13:38:44.537515 I | cephclient: updating pool "ceph.rgw.buckets.data" failure domain from "osd" to "osd" with new crush rule "ceph.rgw.buckets.data_osd"
2026-04-22 13:38:44.537573 I | cephclient: crush rule "ceph.rgw.buckets.data" will no longer be used by pool "ceph.rgw.buckets.data"
2026-04-22 13:38:46.727176 I | cephclient: Successfully updated pool "ceph.rgw.buckets.data" failure domain to "osd"
2026-04-22 13:38:46.727221 I | ceph-object-controller: configuring object store "ceph"
2026-04-22 13:38:47.366786 I | ceph-object-controller: Object store "ceph": realm=ceph, zonegroup=ceph, zone=ceph
2026-04-22 13:38:47.578325 I | ceph-object-controller: committing changes to RGW configuration period for CephObjectStore "openstack/ceph"
2026-04-22 13:38:48.006754 I | ceph-object-controller: configuration for object-store ceph is complete
2026-04-22 13:38:48.006795 I | ceph-object-controller: creating object store "ceph" in namespace "openstack"
2026-04-22 13:38:48.011701 I | cephclient: getting or creating ceph auth key "client.rgw.ceph.a"
2026-04-22 13:38:48.660286 I | ceph-object-controller: setting rgw config flags
2026-04-22 13:38:48.660322 I | op-config: setting "client.rgw.ceph.a"="rgw_zonegroup"="ceph" option to the mon configuration database
2026-04-22 13:38:49.117438 I | op-config: successfully set "client.rgw.ceph.a"="rgw_zonegroup"="ceph" option to the mon configuration database
2026-04-22 13:38:49.117486 I | op-config: setting "client.rgw.ceph.a"="rgw_run_sync_thread"="true" option to the mon configuration database
2026-04-22 13:38:49.579792 I | op-config: successfully set "client.rgw.ceph.a"="rgw_run_sync_thread"="true" option to the mon configuration database
2026-04-22 13:38:49.579827 I | op-config: setting "client.rgw.ceph.a"="rgw_log_nonexistent_bucket"="true" option to the mon configuration database
2026-04-22 13:38:50.034584 I | op-config: successfully set "client.rgw.ceph.a"="rgw_log_nonexistent_bucket"="true" option to the mon configuration database
2026-04-22 13:38:50.034621 I | op-config: setting "client.rgw.ceph.a"="rgw_log_object_name_utc"="true" option to the mon configuration database
2026-04-22 13:38:50.498935 I | op-config: successfully set "client.rgw.ceph.a"="rgw_log_object_name_utc"="true" option to the mon configuration database
2026-04-22 13:38:50.498975 I | op-config: setting "client.rgw.ceph.a"="rgw_enable_usage_log"="true" option to the mon configuration database
2026-04-22 13:38:50.959828 I | op-config: successfully set "client.rgw.ceph.a"="rgw_enable_usage_log"="true" option to the mon configuration database
2026-04-22 13:38:50.959865 I | op-config: setting "client.rgw.ceph.a"="rgw_zone"="ceph" option to the mon configuration database
2026-04-22 13:38:51.437945 I | op-config: successfully set "client.rgw.ceph.a"="rgw_zone"="ceph" option to the mon configuration database
2026-04-22 13:38:51.438266 I | ceph-object-controller: object store "ceph" deployment "rook-ceph-rgw-ceph-a" created
2026-04-22 13:38:51.492010 I | ceph-object-controller: enabling rgw dashboard
2026-04-22 13:38:53.524076 I | ceph-object-controller: created object store "ceph" in namespace "openstack"
2026-04-22 13:38:53.524661 I | ceph-object-controller: setting the dashboard api secret key
2026-04-22 13:38:54.124963 I | ceph-object-controller: done setting the dashboard api secret key
2026-04-22 13:38:58.288486 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-22 13:38:59.368779 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-22 13:39:02.957180 W | op-mon: external cluster mon count is 1, consider adding new monitors.
2026-04-22 13:39:23.950355 E | ceph-object-controller: failed to reconcile CephObjectStore "openstack/ceph". failed to create object store deployments: failed to get COSI user "cosi": Get "http://rook-ceph-rgw-ceph.openstack.svc:80/admin/user?format=json&uid=cosi": dial tcp 10.96.199.179:80: i/o timeout
2026-04-22 13:39:23.973642 I | ceph-spec: parsing mon endpoints: instance=10.96.240.200:6789
2026-04-22 13:39:23.973669 I | ceph-spec: updating obsolete maxMonID 0 to actual value 76846025234
2026-04-22 13:39:23.973683 I | ceph-spec: detecting the ceph image version for image harbor.atmosphere.dev/quay.io/ceph/ceph:v18.2.7...
2026-04-22 13:39:24.502670 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-22 13:39:25.075873 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-22 13:39:25.853600 I | ceph-spec: detected ceph image version: "18.2.7-0 reef"
2026-04-22 13:39:27.337576 I | ceph-object-controller: reconciling object store deployments
2026-04-22 13:39:27.371303 I | ceph-object-controller: ceph object store gateway service running at 10.96.199.179
2026-04-22 13:39:27.371333 I | ceph-object-controller: reconciling object store pools
2026-04-22 13:39:29.334981 I | cephclient: application "rgw" is already set on pool "ceph.rgw.control"
2026-04-22 13:39:29.335030 I | cephclient: reconciling replicated pool ceph.rgw.control succeeded
2026-04-22 13:39:29.521135 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-22 13:39:30.729139 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-22 13:39:31.049185 I | cephclient: setting pool property "pg_num_min" to "8" on pool "ceph.rgw.control"
2026-04-22 13:39:34.746372 I | cephclient: application "rgw" is already set on pool "ceph.rgw.meta"
2026-04-22 13:39:34.746410 I | cephclient: reconciling replicated pool ceph.rgw.meta succeeded
2026-04-22 13:39:36.216356 I | cephclient: setting pool property "pg_num_min" to "8" on pool "ceph.rgw.meta"
2026-04-22 13:39:38.911085 I | cephclient: application "rgw" is already set on pool "ceph.rgw.log"
2026-04-22 13:39:38.911131 I | cephclient: reconciling replicated pool ceph.rgw.log succeeded
2026-04-22 13:39:39.793732 I | cephclient: setting pool property "pg_num_min" to "8" on pool "ceph.rgw.log"
2026-04-22 13:39:41.960521 I | cephclient: application "rgw" is already set on pool "ceph.rgw.buckets.index"
2026-04-22 13:39:41.960617 I | cephclient: reconciling replicated pool ceph.rgw.buckets.index succeeded
2026-04-22 13:39:42.955815 I | cephclient: setting pool property "pg_num_min" to "8" on pool "ceph.rgw.buckets.index"
2026-04-22 13:39:45.031673 I | cephclient: application "rgw" is already set on pool "ceph.rgw.buckets.non-ec"
2026-04-22 13:39:45.031712 I | cephclient: reconciling replicated pool ceph.rgw.buckets.non-ec succeeded
2026-04-22 13:39:45.961505 I | cephclient: setting pool property "pg_num_min" to "8" on pool "ceph.rgw.buckets.non-ec"
2026-04-22 13:39:48.230913 I | cephclient: application "rgw" is already set on pool "ceph.rgw.otp"
2026-04-22 13:39:48.230959 I | cephclient: reconciling replicated pool ceph.rgw.otp succeeded
2026-04-22 13:39:50.404357 I | cephclient: setting pool property "pg_num_min" to "8" on pool "ceph.rgw.otp"
2026-04-22 13:39:50.616873 W | op-mon: external cluster mon count is 1, consider adding new monitors.
2026-04-22 13:39:53.150770 I | cephclient: application "rgw" is already set on pool ".rgw.root"
2026-04-22 13:39:53.150808 I | cephclient: reconciling replicated pool .rgw.root succeeded
2026-04-22 13:39:54.056138 I | cephclient: setting pool property "pg_num_min" to "8" on pool ".rgw.root"
2026-04-22 13:39:56.210748 I | cephclient: application "rgw" is already set on pool "ceph.rgw.buckets.data"
2026-04-22 13:39:56.210787 I | cephclient: reconciling replicated pool ceph.rgw.buckets.data succeeded
2026-04-22 13:39:57.177681 I | ceph-object-controller: configuring object store "ceph"
2026-04-22 13:39:57.430849 I | ceph-object-controller: Object store "ceph": realm=ceph, zonegroup=ceph, zone=ceph
2026-04-22 13:39:57.642336 I | ceph-object-controller: there are no changes to commit for RGW configuration period for CephObjectStore "openstack/ceph"
2026-04-22 13:39:57.642363 I | ceph-object-controller: configuration for object-store ceph is complete
2026-04-22 13:39:57.642369 I | ceph-object-controller: creating object store "ceph" in namespace "openstack"
2026-04-22 13:39:57.646719 I | cephclient: getting or creating ceph auth key "client.rgw.ceph.a"
2026-04-22 13:39:58.238187 I | ceph-object-controller: setting rgw config flags
2026-04-22 13:39:58.238219 I | op-config: setting "client.rgw.ceph.a"="rgw_log_object_name_utc"="true" option to the mon configuration database
2026-04-22 13:39:58.691775 I | op-config: successfully set "client.rgw.ceph.a"="rgw_log_object_name_utc"="true" option to the mon configuration database
2026-04-22 13:39:58.691821 I | op-config: setting "client.rgw.ceph.a"="rgw_enable_usage_log"="true" option to the mon configuration database
2026-04-22 13:39:59.155766 I | op-config: successfully set "client.rgw.ceph.a"="rgw_enable_usage_log"="true" option to the mon configuration database
2026-04-22 13:39:59.155799 I | op-config: setting "client.rgw.ceph.a"="rgw_zone"="ceph" option to the mon configuration database
2026-04-22 13:39:59.638821 I | op-config: successfully set "client.rgw.ceph.a"="rgw_zone"="ceph" option to the mon configuration database
2026-04-22 13:39:59.638853 I | op-config: setting "client.rgw.ceph.a"="rgw_zonegroup"="ceph" option to the mon configuration database
2026-04-22 13:40:00.805099 I | op-config: successfully set "client.rgw.ceph.a"="rgw_zonegroup"="ceph" option to the mon configuration database
2026-04-22 13:40:00.805128 I | op-config: setting "client.rgw.ceph.a"="rgw_run_sync_thread"="true" option to the mon configuration database
2026-04-22 13:40:00.809478 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-22 13:40:01.910540 I | op-config: successfully set "client.rgw.ceph.a"="rgw_run_sync_thread"="true" option to the mon configuration database
2026-04-22 13:40:01.910568 I | op-config: setting "client.rgw.ceph.a"="rgw_log_nonexistent_bucket"="true" option to the mon configuration database
2026-04-22 13:40:02.018909 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-22 13:40:02.466955 I | op-config: successfully set "client.rgw.ceph.a"="rgw_log_nonexistent_bucket"="true" option to the mon configuration database
2026-04-22 13:40:02.467151 I | ceph-object-controller: object store "ceph" deployment "rook-ceph-rgw-ceph-a" created
2026-04-22 13:40:02.484310 I | ceph-object-controller: object store "ceph" deployment "rook-ceph-rgw-ceph-a" already exists. updating if needed
2026-04-22 13:40:02.492068 I | op-k8sutil: deployment "rook-ceph-rgw-ceph-a" did not change, nothing to update
2026-04-22 13:40:02.494846 I | ceph-object-controller: config map "rook-ceph-rgw-ceph-mime-types" for object store "ceph" already exists, not overwriting
2026-04-22 13:40:02.499117 I | ceph-object-controller: enabling rgw dashboard
2026-04-22 13:40:03.694303 I | ceph-object-controller: created object store "ceph" in namespace "openstack"
2026-04-22 13:40:04.573201 I | ceph-object-controller: creating COSI user "cosi"
2026-04-22 13:40:04.675138 I | ceph-spec: created ceph *v1.Secret object "rook-ceph-object-user-ceph-cosi"
2026-04-22 13:40:05.355179 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-22 13:40:31.371685 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-22 13:40:32.582248 I | clusterdisruption-controller: all "host" failure domains: []. osd is down in failure domain: "". active node drains: false. pg health: "cluster is not fully clean. PGs: [{StateName:active+clean Count:113} {StateName:clean+premerge+peered Count:2}]"
2026-04-22 13:40:32.587020 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-22 13:40:38.219227 W | op-mon: external cluster mon count is 1, consider adding new monitors.
2026-04-22 13:41:01.961856 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-22 13:41:03.218638 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-22 13:41:24.452315 W | op-mon: external cluster mon count is 1, consider adding new monitors.
2026-04-22 13:41:32.523947 I | clusterdisruption-controller: all "host" failure domains: []. osd is down in failure domain: "". active node drains: false. pg health: "cluster is not fully clean. PGs: [{StateName:active+clean Count:127} {StateName:peering Count:1}]"
2026-04-22 13:41:32.528831 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-22 13:41:33.811684 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-22 13:42:03.172536 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-22 13:42:04.405086 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-22 13:42:10.575573 W | op-mon: external cluster mon count is 1, consider adding new monitors.
2026-04-22 13:42:33.733311 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-22 13:42:34.996530 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-22 13:42:56.708802 W | op-mon: external cluster mon count is 1, consider adding new monitors.
2026-04-22 13:43:04.297442 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-22 13:43:05.571072 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-22 13:43:34.865506 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-22 13:43:36.164602 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-22 13:43:42.881939 W | op-mon: external cluster mon count is 1, consider adding new monitors.
2026-04-22 13:44:05.450122 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-22 13:44:06.725627 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-22 13:44:29.064605 W | op-mon: external cluster mon count is 1, consider adding new monitors.
2026-04-22 13:44:36.036592 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-22 13:44:37.292868 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-22 13:45:06.599603 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-22 13:45:07.876619 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-22 13:45:15.248945 W | op-mon: external cluster mon count is 1, consider adding new monitors.
2026-04-22 13:45:37.183312 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-22 13:45:38.459650 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-22 13:46:01.419011 W | op-mon: external cluster mon count is 1, consider adding new monitors.
2026-04-22 13:46:07.759687 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-22 13:46:09.082103 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-22 13:46:38.346623 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-22 13:46:39.666169 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-22 13:46:47.639079 W | op-mon: external cluster mon count is 1, consider adding new monitors.
2026-04-22 13:47:08.927643 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-22 13:47:10.232057 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-22 13:47:33.829250 W | op-mon: external cluster mon count is 1, consider adding new monitors.
2026-04-22 13:47:39.532592 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-22 13:47:40.853453 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-22 13:48:10.118122 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-22 13:48:11.441337 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-22 13:48:19.994942 W | op-mon: external cluster mon count is 1, consider adding new monitors.