2026/04/20 06:04:04 maxprocs: Updating GOMAXPROCS=1: determined from CPU quota
2026-04-20 06:04:04.078921 I | rookcmd: starting Rook v1.14.2 with arguments '/usr/local/bin/rook ceph operator'
2026-04-20 06:04:04.078938 I | rookcmd: flag values: --enable-machine-disruption-budget=false, --help=false, --kubeconfig=, --log-level=INFO
2026-04-20 06:04:04.078940 I | cephcmd: starting Rook-Ceph operator
2026-04-20 06:04:04.209959 I | cephcmd: base ceph version inside the rook operator image is "ceph version 18.2.2 (531c0d11a1c5d39fbfe6aa8a521f023abf3bf3e2) reef (stable)"
2026-04-20 06:04:04.217051 I | op-k8sutil: ROOK_CURRENT_NAMESPACE_ONLY="false" (env var)
2026-04-20 06:04:04.217078 I | operator: watching all namespaces for Ceph CRs
2026-04-20 06:04:04.217152 I | operator: setting up schemes
2026-04-20 06:04:04.220601 I | operator: setting up the controller-runtime manager
2026-04-20 06:04:04.221521 I | ceph-cluster-controller: successfully started
2026-04-20 06:04:04.225846 I | op-k8sutil: ROOK_DISABLE_DEVICE_HOTPLUG="false" (env var)
2026-04-20 06:04:04.225856 I | ceph-cluster-controller: enabling hotplug orchestration
2026-04-20 06:04:04.225873 I | ceph-nodedaemon-controller: successfully started
2026-04-20 06:04:04.225883 I | ceph-block-pool-controller: successfully started
2026-04-20 06:04:04.225899 I | ceph-object-store-user-controller: successfully started
2026-04-20 06:04:04.225912 I | ceph-object-realm-controller: successfully started
2026-04-20 06:04:04.225926 I | ceph-object-zonegroup-controller: successfully started
2026-04-20 06:04:04.225934 I | ceph-object-zone-controller: successfully started
2026-04-20 06:04:04.225994 I | ceph-object-controller: successfully started
2026-04-20 06:04:04.226020 I | ceph-file-controller: successfully started
2026-04-20 06:04:04.226038 I | ceph-nfs-controller: successfully started
2026-04-20 06:04:04.226061 I | ceph-rbd-mirror-controller: successfully started
2026-04-20 06:04:04.226083 I | ceph-client-controller: successfully started
2026-04-20 06:04:04.226098 I | ceph-filesystem-mirror-controller: successfully started
2026-04-20 06:04:04.226115 I | operator: rook-ceph-operator-config-controller successfully started
2026-04-20 06:04:04.226128 I | ceph-csi: rook-ceph-operator-csi-controller successfully started
2026-04-20 06:04:04.226195 I | op-bucket-prov: rook-ceph-operator-bucket-controller successfully started
2026-04-20 06:04:04.226207 I | ceph-bucket-topic: successfully started
2026-04-20 06:04:04.226215 I | ceph-bucket-notification: successfully started
2026-04-20 06:04:04.226224 I | ceph-bucket-notification: successfully started
2026-04-20 06:04:04.226240 I | ceph-fs-subvolumegroup-controller: successfully started
2026-04-20 06:04:04.226246 I | blockpool-rados-namespace-controller: successfully started
2026-04-20 06:04:04.226252 I | ceph-cosi-controller: successfully started
2026-04-20 06:04:04.226270 I | operator: starting the controller-runtime manager
2026-04-20 06:04:04.347186 I | op-k8sutil: ROOK_WATCH_FOR_NODE_FAILURE="true" (default)
2026-04-20 06:04:04.442782 I | op-k8sutil: ROOK_CSI_DISABLE_DRIVER="false" (configmap)
2026-04-20 06:04:04.443661 I | op-k8sutil: ROOK_CEPH_COMMANDS_TIMEOUT_SECONDS="15" (configmap)
2026-04-20 06:04:04.443687 I | op-k8sutil: ROOK_LOG_LEVEL="INFO" (configmap)
2026-04-20 06:04:04.443712 I | op-k8sutil: ROOK_ENABLE_DISCOVERY_DAEMON="false" (configmap)
2026-04-20 06:04:04.445159 I | ceph-csi: CSI Ceph RBD driver disabled
2026-04-20 06:04:04.445187 I | op-k8sutil: removing daemonset csi-rbdplugin if it exists
2026-04-20 06:04:04.446820 I | op-k8sutil: ROOK_CEPH_ALLOW_LOOP_DEVICES="false" (configmap)
2026-04-20 06:04:04.446845 I | operator: rook-ceph-operator-config-controller done reconciling
2026-04-20 06:04:04.447214 I | op-k8sutil: removing deployment csi-rbdplugin-provisioner if it exists
2026-04-20 06:04:04.453681 I | ceph-csi: successfully removed CSI Ceph RBD driver
2026-04-20 06:04:04.453700 I | ceph-csi: CSI CephFS driver disabled
2026-04-20 06:04:04.453705 I | op-k8sutil: removing daemonset csi-cephfsplugin if it exists
2026-04-20 06:04:04.455682 I | op-k8sutil: removing deployment csi-cephfsplugin-provisioner if it exists
2026-04-20 06:04:04.461146 I | ceph-csi: successfully removed CSI CephFS driver
2026-04-20 06:04:04.461168 I | ceph-csi: CSI NFS driver disabled
2026-04-20 06:04:04.461175 I | op-k8sutil: removing daemonset csi-nfsplugin if it exists
2026-04-20 06:04:04.463027 I | op-k8sutil: removing deployment csi-nfsplugin-provisioner if it exists
2026-04-20 06:04:04.468075 I | ceph-csi: successfully removed CSI NFS driver
2026-04-20 06:04:25.437026 I | ceph-spec: adding finalizer "cephcluster.ceph.rook.io" on "ceph"
2026-04-20 06:04:25.444568 I | clusterdisruption-controller: deleted all legacy node drain canary pods
2026-04-20 06:04:25.445780 I | ceph-spec: adding finalizer "cephobjectstore.ceph.rook.io" on "ceph"
2026-04-20 06:04:25.469982 I | ceph-cluster-controller: reconciling ceph cluster in namespace "openstack"
2026-04-20 06:04:25.474218 I | ceph-spec: parsing mon endpoints: instance=10.96.240.200:6789
2026-04-20 06:04:25.474243 I | ceph-spec: updating obsolete maxMonID 0 to actual value 76846025234
2026-04-20 06:04:25.474322 I | op-k8sutil: ROOK_OBC_WATCH_OPERATOR_NAMESPACE="true" (configmap)
2026-04-20 06:04:25.474334 I | op-k8sutil: ROOK_OBC_PROVISIONER_NAME_PREFIX="" (default)
2026-04-20 06:04:25.474339 I | op-bucket-prov: ceph bucket provisioner launched watching for provisioner "openstack.ceph.rook.io/bucket"
2026-04-20 06:04:25.474769 I | op-bucket-prov: successfully reconciled bucket provisioner
I0420 06:04:25.474844       1 manager.go:135] "msg"="starting provisioner" "logger"="objectbucket.io/provisioner-manager" "name"="openstack.ceph.rook.io/bucket"
2026-04-20 06:04:25.480130 I | ceph-csi: successfully created csi config map "rook-ceph-csi-config"
2026-04-20 06:04:25.490487 I | ceph-spec: parsing mon endpoints: instance=10.96.240.200:6789
2026-04-20 06:04:25.490515 I | ceph-spec: updating obsolete maxMonID 0 to actual value 76846025234
2026-04-20 06:04:25.490534 I | ceph-spec: found the cluster info to connect to the external cluster. will use "client.admin" to check health and monitor status. mons=map[instance:0xc0011a5b60]
2026-04-20 06:04:25.490550 I | ceph-spec: detecting the ceph image version for image harbor.atmosphere.dev/quay.io/ceph/ceph:v18.2.1...
2026-04-20 06:04:25.495745 I | op-k8sutil: CSI_ENABLE_HOST_NETWORK="true" (configmap)
2026-04-20 06:04:25.495759 I | op-k8sutil: CSI_DISABLE_HOLDER_PODS="true" (configmap)
2026-04-20 06:04:26.047479 I | ceph-spec: parsing mon endpoints: instance=10.96.240.200:6789
2026-04-20 06:04:26.047512 I | ceph-spec: updating obsolete maxMonID 0 to actual value 76846025234
2026-04-20 06:04:26.247036 I | op-k8sutil: ROOK_CSI_ENABLE_RBD="false" (configmap)
2026-04-20 06:04:26.247076 I | op-k8sutil: ROOK_CSI_ENABLE_CEPHFS="false" (configmap)
2026-04-20 06:04:26.247087 I | op-k8sutil: ROOK_CSI_ENABLE_NFS="false" (configmap)
2026-04-20 06:04:26.247099 I | op-k8sutil: ROOK_CSI_ALLOW_UNSUPPORTED_VERSION="false" (default)
2026-04-20 06:04:26.247109 I | op-k8sutil: CSI_FORCE_CEPHFS_KERNEL_CLIENT="true" (configmap)
2026-04-20 06:04:26.247116 I | op-k8sutil: CSI_GRPC_TIMEOUT_SECONDS="150" (configmap)
2026-04-20 06:04:26.247124 I | op-k8sutil: CSI_CEPHFS_LIVENESS_METRICS_PORT="9081" (default)
2026-04-20 06:04:26.247133 I | op-k8sutil: CSIADDONS_PORT="9070" (default)
2026-04-20 06:04:26.247142 I | op-k8sutil: CSI_RBD_LIVENESS_METRICS_PORT="9080" (default)
2026-04-20 06:04:26.247158 I | op-k8sutil: CSI_ENABLE_LIVENESS="false" (default)
2026-04-20 06:04:26.247165 I | op-k8sutil: CSI_PLUGIN_PRIORITY_CLASSNAME="system-node-critical" (configmap)
2026-04-20 06:04:26.247173 I | op-k8sutil: CSI_PROVISIONER_PRIORITY_CLASSNAME="system-cluster-critical" (configmap)
2026-04-20 06:04:26.247185 I | op-k8sutil: CSI_ENABLE_OMAP_GENERATOR="false" (configmap)
2026-04-20 06:04:26.247198 I | op-k8sutil: CSI_ENABLE_RBD_SNAPSHOTTER="true" (configmap)
2026-04-20 06:04:26.247206 I | op-k8sutil: CSI_ENABLE_CEPHFS_SNAPSHOTTER="true" (configmap)
2026-04-20 06:04:26.247217 I | op-k8sutil: CSI_ENABLE_NFS_SNAPSHOTTER="true" (configmap)
2026-04-20 06:04:26.249347 I | op-k8sutil: CSI_ENABLE_CSIADDONS="false" (configmap)
2026-04-20 06:04:26.249374 I | op-k8sutil: CSI_ENABLE_TOPOLOGY="false" (configmap)
2026-04-20 06:04:26.249393 I | op-k8sutil: CSI_ENABLE_ENCRYPTION="false" (configmap)
2026-04-20 06:04:26.249406 I | op-k8sutil: CSI_ENABLE_METADATA="false" (configmap)
2026-04-20 06:04:26.249414 I | op-k8sutil: CSI_CEPHFS_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default)
2026-04-20 06:04:26.249423 I | op-k8sutil: CSI_CEPHFS_PLUGIN_UPDATE_STRATEGY_MAX_UNAVAILABLE="1" (default)
2026-04-20 06:04:26.249430 I | op-k8sutil: CSI_NFS_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default)
2026-04-20 06:04:26.249437 I | op-k8sutil: CSI_RBD_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default)
2026-04-20 06:04:26.249445 I | op-k8sutil: CSI_RBD_PLUGIN_UPDATE_STRATEGY_MAX_UNAVAILABLE="1" (default)
2026-04-20 06:04:26.249459 I | op-k8sutil: CSI_PLUGIN_ENABLE_SELINUX_HOST_MOUNT="false" (configmap)
2026-04-20 06:04:26.249464 I | ceph-csi: Kubernetes version is 1.28
2026-04-20 06:04:26.249471 I | op-k8sutil: CSI_LOG_LEVEL="" (default)
2026-04-20 06:04:26.249482 I | op-k8sutil: CSI_SIDECAR_LOG_LEVEL="" (default)
2026-04-20 06:04:26.249496 I | op-k8sutil: CSI_LEADER_ELECTION_LEASE_DURATION="" (default)
2026-04-20 06:04:26.249503 I | op-k8sutil: CSI_LEADER_ELECTION_RENEW_DEADLINE="" (default)
2026-04-20 06:04:26.249510 I | op-k8sutil: CSI_LEADER_ELECTION_RETRY_PERIOD="" (default)
2026-04-20 06:04:26.476357 I | op-k8sutil: ROOK_CSI_CEPH_IMAGE="quay.io/cephcsi/cephcsi:v3.11.0" (configmap)
2026-04-20 06:04:26.476392 I | op-k8sutil: ROOK_CSI_REGISTRAR_IMAGE="registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.10.0" (configmap)
2026-04-20 06:04:26.476404 I | op-k8sutil: ROOK_CSI_PROVISIONER_IMAGE="registry.k8s.io/sig-storage/csi-provisioner:v4.0.0" (configmap)
2026-04-20 06:04:26.476409 I | op-k8sutil: ROOK_CSI_ATTACHER_IMAGE="registry.k8s.io/sig-storage/csi-attacher:v4.5.0" (configmap)
2026-04-20 06:04:26.476414 I | op-k8sutil: ROOK_CSI_SNAPSHOTTER_IMAGE="registry.k8s.io/sig-storage/csi-snapshotter:v7.0.1" (configmap)
2026-04-20 06:04:26.476426 I | op-k8sutil: ROOK_CSI_RESIZER_IMAGE="registry.k8s.io/sig-storage/csi-resizer:v1.10.0" (configmap)
2026-04-20 06:04:26.476436 I | op-k8sutil: ROOK_CSI_KUBELET_DIR_PATH="/var/lib/kubelet" (default)
2026-04-20 06:04:26.476470 I | op-k8sutil: ROOK_CSIADDONS_IMAGE="quay.io/csiaddons/k8s-sidecar:v0.8.0" (configmap)
2026-04-20 06:04:26.476477 I | op-k8sutil: CSI_TOPOLOGY_DOMAIN_LABELS="" (default)
2026-04-20 06:04:26.476483 I | op-k8sutil: ROOK_CSI_CEPHFS_POD_LABELS="" (default)
2026-04-20 06:04:26.476489 I | op-k8sutil: ROOK_CSI_NFS_POD_LABELS="" (default)
2026-04-20 06:04:26.476494 I | op-k8sutil: ROOK_CSI_RBD_POD_LABELS="" (default)
2026-04-20 06:04:26.476498 I | op-k8sutil: CSI_CLUSTER_NAME="" (default)
2026-04-20 06:04:26.476504 I | op-k8sutil: ROOK_CSI_IMAGE_PULL_POLICY="IfNotPresent" (configmap)
2026-04-20 06:04:26.476509 I | op-k8sutil: CSI_CEPHFS_KERNEL_MOUNT_OPTIONS="" (default)
2026-04-20 06:04:26.476515 I | op-k8sutil: CSI_CEPHFS_ATTACH_REQUIRED="true" (configmap)
2026-04-20 06:04:26.476519 I | op-k8sutil: CSI_RBD_ATTACH_REQUIRED="true" (configmap)
2026-04-20 06:04:26.476524 I | op-k8sutil: CSI_NFS_ATTACH_REQUIRED="true" (configmap)
2026-04-20 06:04:26.476546 I | op-k8sutil: CSI_DRIVER_NAME_PREFIX="rook-ceph" (default)
2026-04-20 06:04:26.480460 I | op-k8sutil: CSI_ENABLE_VOLUME_GROUP_SNAPSHOT="true" (configmap)
2026-04-20 06:04:26.480474 I | ceph-csi: skipping csi version check, since unsupported versions are allowed or csi is disabled
2026-04-20 06:04:26.480479 I | ceph-csi: CSI Ceph RBD driver disabled
2026-04-20 06:04:26.480483 I | op-k8sutil: removing daemonset csi-rbdplugin if it exists
2026-04-20 06:04:26.483359 I | op-k8sutil: removing deployment csi-rbdplugin-provisioner if it exists
2026-04-20 06:04:26.651460 I | ceph-csi: successfully removed CSI Ceph RBD driver
2026-04-20 06:04:26.651499 I | ceph-csi: CSI CephFS driver disabled
2026-04-20 06:04:26.651510 I | op-k8sutil: removing daemonset csi-cephfsplugin if it exists
2026-04-20 06:04:26.654280 I | op-k8sutil: removing deployment csi-cephfsplugin-provisioner if it exists
2026-04-20 06:04:26.850078 I | ceph-csi: successfully removed CSI CephFS driver
2026-04-20 06:04:26.850109 I | ceph-csi: CSI NFS driver disabled
2026-04-20 06:04:26.850118 I | op-k8sutil: removing daemonset csi-nfsplugin if it exists
2026-04-20 06:04:26.852872 I | op-k8sutil: removing deployment csi-nfsplugin-provisioner if it exists
2026-04-20 06:04:27.051342 I | ceph-csi: successfully removed CSI NFS driver
2026-04-20 06:04:41.883910 I | ceph-spec: detected ceph image version: "18.2.1-0 reef"
2026-04-20 06:04:41.883931 I | ceph-cluster-controller: validating ceph version from provided image
2026-04-20 06:04:41.889748 I | ceph-spec: parsing mon endpoints: instance=10.96.240.200:6789
2026-04-20 06:04:41.889788 I | ceph-spec: updating obsolete maxMonID 0 to actual value 76846025234
2026-04-20 06:04:41.892565 I | cephclient: writing config file /var/lib/rook/openstack/openstack.config
2026-04-20 06:04:41.892690 I | cephclient: generated admin config in /var/lib/rook/openstack
2026-04-20 06:04:42.449381 E | cephver: external cluster ceph version is a major version higher "18.2.7-0 reef" than the local cluster "0.0.0-0 <unknown version>", consider upgrading
2026-04-20 06:04:43.051831 W | ceph-cluster-controller: image spec version 18.2.1-0 reef is lower than the running cluster version 18.2.7-0 reef, downgrading is not supported
2026-04-20 06:04:43.622581 I | ceph-cluster-controller: upgrading ceph cluster to "18.2.1-0 reef"
2026-04-20 06:04:43.622627 I | ceph-cluster-controller: cluster "openstack": version "18.2.1-0 reef" detected for image "harbor.atmosphere.dev/quay.io/ceph/ceph:v18.2.1"
2026-04-20 06:04:43.711554 I | ceph-cluster-controller: creating "rook-config-override" configmap
2026-04-20 06:04:43.715456 I | ceph-cluster-controller: creating "rook-ceph-config" secret
2026-04-20 06:04:43.737217 I | ceph-cluster-controller: external cluster identity established
2026-04-20 06:04:43.737252 I | cephclient: getting or creating ceph auth key "client.csi-rbd-provisioner"
2026-04-20 06:04:44.403608 I | cephclient: getting or creating ceph auth key "client.csi-rbd-node"
2026-04-20 06:04:44.989584 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-provisioner"
2026-04-20 06:04:45.556628 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-node"
2026-04-20 06:04:46.567678 I | ceph-csi: created kubernetes csi secrets for cluster "openstack"
2026-04-20 06:04:46.578307 I | ceph-cluster-controller: successfully updated csi config map
2026-04-20 06:04:46.578336 I | cephclient: getting or creating ceph auth key "client.crash"
2026-04-20 06:04:46.874577 I | clusterdisruption-controller: all PGs are active+clean. Restoring default OSD pdb settings
2026-04-20 06:04:46.874599 I | clusterdisruption-controller: creating the default pdb "rook-ceph-osd" with maxUnavailable=1 for all osd
2026-04-20 06:04:46.886095 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-20 06:04:47.766757 I | ceph-nodedaemon-controller: created kubernetes crash collector secret for cluster "openstack"
2026-04-20 06:04:47.769296 I | ceph-cluster-controller: enabling ceph mon monitoring goroutine for cluster "openstack"
2026-04-20 06:04:47.769336 I | ceph-cluster-controller: ceph status check interval is 1m0s
2026-04-20 06:04:47.769343 I | ceph-cluster-controller: enabling ceph status monitoring goroutine for cluster "openstack"
2026-04-20 06:04:48.153069 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-20 06:04:55.541997 I | ceph-spec: parsing mon endpoints: instance=10.96.240.200:6789
2026-04-20 06:04:55.542098 I | ceph-spec: updating obsolete maxMonID 0 to actual value 76846025234
2026-04-20 06:04:55.542131 I | ceph-spec: detecting the ceph image version for image harbor.atmosphere.dev/quay.io/ceph/ceph:v18.2.1...
2026-04-20 06:04:57.285167 I | ceph-spec: detected ceph image version: "18.2.1-0 reef"
2026-04-20 06:04:58.764226 I | ceph-object-controller: reconciling object store deployments
2026-04-20 06:04:58.778422 I | ceph-object-controller: ceph object store gateway service running at 10.100.51.17
2026-04-20 06:04:58.778452 I | ceph-object-controller: reconciling object store pools
2026-04-20 06:05:02.160540 I | cephclient: reconciling replicated pool ceph.rgw.control succeeded
2026-04-20 06:05:03.165776 I | cephclient: creating a new crush rule for changed deviceClass ("default"-->"") on crush rule "ceph.rgw.control"
2026-04-20 06:05:03.165814 I | cephclient: updating pool "ceph.rgw.control" failure domain from "osd" to "osd" with new crush rule "ceph.rgw.control_osd"
2026-04-20 06:05:03.165823 I | cephclient: crush rule "ceph.rgw.control" will no longer be used by pool "ceph.rgw.control"
2026-04-20 06:05:05.219909 I | cephclient: Successfully updated pool "ceph.rgw.control" failure domain to "osd"
2026-04-20 06:05:05.219946 I | cephclient: setting pool property "pg_num_min" to "8" on pool "ceph.rgw.control"
2026-04-20 06:05:09.279777 I | cephclient: reconciling replicated pool ceph.rgw.meta succeeded
2026-04-20 06:05:10.182283 I | cephclient: creating a new crush rule for changed deviceClass ("default"-->"") on crush rule "ceph.rgw.meta"
2026-04-20 06:05:10.182329 I | cephclient: updating pool "ceph.rgw.meta" failure domain from "osd" to "osd" with new crush rule "ceph.rgw.meta_osd"
2026-04-20 06:05:10.182338 I | cephclient: crush rule "ceph.rgw.meta" will no longer be used by pool "ceph.rgw.meta"
2026-04-20 06:05:12.381078 I | cephclient: Successfully updated pool "ceph.rgw.meta" failure domain to "osd"
2026-04-20 06:05:12.381112 I | cephclient: setting pool property "pg_num_min" to "8" on pool "ceph.rgw.meta"
2026-04-20 06:05:16.452931 I | cephclient: reconciling replicated pool ceph.rgw.log succeeded
2026-04-20 06:05:17.777362 I | cephclient: creating a new crush rule for changed deviceClass ("default"-->"") on crush rule "ceph.rgw.log"
2026-04-20 06:05:17.777384 I | cephclient: updating pool "ceph.rgw.log" failure domain from "osd" to "osd" with new crush rule "ceph.rgw.log_osd"
2026-04-20 06:05:17.777390 I | cephclient: crush rule "ceph.rgw.log" will no longer be used by pool "ceph.rgw.log"
2026-04-20 06:05:18.058719 I | clusterdisruption-controller: all "host" failure domains: []. osd is down in failure domain: "". active node drains: false. pg health: "cluster is not fully clean. PGs: [{StateName:active+clean Count:17} {StateName:unknown Count:8}]"
2026-04-20 06:05:18.063643 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-20 06:05:19.035798 I | clusterdisruption-controller: all "host" failure domains: []. osd is down in failure domain: "". active node drains: false. pg health: "cluster is not fully clean. PGs: [{StateName:active+clean Count:17} {StateName:unknown Count:8}]"
2026-04-20 06:05:19.039483 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-20 06:05:20.499290 I | cephclient: Successfully updated pool "ceph.rgw.log" failure domain to "osd"
2026-04-20 06:05:20.499324 I | cephclient: setting pool property "pg_num_min" to "8" on pool "ceph.rgw.log"
2026-04-20 06:05:24.557964 I | cephclient: reconciling replicated pool ceph.rgw.buckets.index succeeded
2026-04-20 06:05:25.435244 I | cephclient: creating a new crush rule for changed deviceClass ("default"-->"") on crush rule "ceph.rgw.buckets.index"
2026-04-20 06:05:25.435276 I | cephclient: updating pool "ceph.rgw.buckets.index" failure domain from "osd" to "osd" with new crush rule "ceph.rgw.buckets.index_osd"
2026-04-20 06:05:25.435283 I | cephclient: crush rule "ceph.rgw.buckets.index" will no longer be used by pool "ceph.rgw.buckets.index"
2026-04-20 06:05:27.608364 I | cephclient: Successfully updated pool "ceph.rgw.buckets.index" failure domain to "osd"
2026-04-20 06:05:27.608405 I | cephclient: setting pool property "pg_num_min" to "8" on pool "ceph.rgw.buckets.index"
2026-04-20 06:05:31.683549 I | cephclient: reconciling replicated pool ceph.rgw.buckets.non-ec succeeded
2026-04-20 06:05:32.577619 I | cephclient: creating a new crush rule for changed deviceClass ("default"-->"") on crush rule "ceph.rgw.buckets.non-ec"
2026-04-20 06:05:32.577654 I | cephclient: updating pool "ceph.rgw.buckets.non-ec" failure domain from "osd" to "osd" with new crush rule "ceph.rgw.buckets.non-ec_osd"
2026-04-20 06:05:32.577662 I | cephclient: crush rule "ceph.rgw.buckets.non-ec" will no longer be used by pool "ceph.rgw.buckets.non-ec"
2026-04-20 06:05:34.683534 W | op-mon: external cluster mon count is 1, consider adding new monitors.
2026-04-20 06:05:35.754339 I | cephclient: Successfully updated pool "ceph.rgw.buckets.non-ec" failure domain to "osd"
2026-04-20 06:05:35.754368 I | cephclient: setting pool property "pg_num_min" to "8" on pool "ceph.rgw.buckets.non-ec"
2026-04-20 06:05:39.806552 I | cephclient: reconciling replicated pool ceph.rgw.otp succeeded
2026-04-20 06:05:40.692682 I | cephclient: creating a new crush rule for changed deviceClass ("default"-->"") on crush rule "ceph.rgw.otp"
2026-04-20 06:05:40.692730 I | cephclient: updating pool "ceph.rgw.otp" failure domain from "osd" to "osd" with new crush rule "ceph.rgw.otp_osd"
2026-04-20 06:05:40.692740 I | cephclient: crush rule "ceph.rgw.otp" will no longer be used by pool "ceph.rgw.otp"
2026-04-20 06:05:42.871021 I | cephclient: Successfully updated pool "ceph.rgw.otp" failure domain to "osd"
2026-04-20 06:05:42.871069 I | cephclient: setting pool property "pg_num_min" to "8" on pool "ceph.rgw.otp"
2026-04-20 06:05:47.012300 I | cephclient: reconciling replicated pool .rgw.root succeeded
2026-04-20 06:05:47.942850 I | cephclient: creating a new crush rule for changed deviceClass ("default"-->"") on crush rule ".rgw.root"
2026-04-20 06:05:47.942895 I | cephclient: updating pool ".rgw.root" failure domain from "osd" to "osd" with new crush rule ".rgw.root_osd"
2026-04-20 06:05:47.942909 I | cephclient: crush rule ".rgw.root" will no longer be used by pool ".rgw.root"
2026-04-20 06:05:49.024889 I | clusterdisruption-controller: all "host" failure domains: []. osd is down in failure domain: "". active node drains: false. pg health: "cluster is not fully clean. PGs: [{StateName:active+clean Count:49} {StateName:unknown Count:8}]"
2026-04-20 06:05:49.029768 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-20 06:05:50.754818 I | clusterdisruption-controller: all "host" failure domains: []. osd is down in failure domain: "". active node drains: false. pg health: "cluster is not fully clean. PGs: [{StateName:active+clean Count:51} {StateName:unknown Count:6}]"
2026-04-20 06:05:50.757793 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-20 06:05:51.186761 I | cephclient: Successfully updated pool ".rgw.root" failure domain to "osd"
2026-04-20 06:05:51.186793 I | cephclient: setting pool property "pg_num_min" to "8" on pool ".rgw.root"
2026-04-20 06:05:55.256482 I | cephclient: reconciling replicated pool ceph.rgw.buckets.data succeeded
2026-04-20 06:05:56.139536 I | cephclient: creating a new crush rule for changed deviceClass ("default"-->"") on crush rule "ceph.rgw.buckets.data"
2026-04-20 06:05:56.139593 I | cephclient: updating pool "ceph.rgw.buckets.data" failure domain from "osd" to "osd" with new crush rule "ceph.rgw.buckets.data_osd"
2026-04-20 06:05:56.139604 I | cephclient: crush rule "ceph.rgw.buckets.data" will no longer be used by pool "ceph.rgw.buckets.data"
2026-04-20 06:05:58.299630 I | cephclient: Successfully updated pool "ceph.rgw.buckets.data" failure domain to "osd"
2026-04-20 06:05:58.299657 I | ceph-object-controller: configuring object store "ceph"
2026-04-20 06:05:58.915825 I | ceph-object-controller: Object store "ceph": realm=ceph, zonegroup=ceph, zone=ceph
2026-04-20 06:05:59.104093 I | ceph-object-controller: committing changes to RGW configuration period for CephObjectStore "openstack/ceph"
2026-04-20 06:05:59.462130 I | ceph-object-controller: configuration for object-store ceph is complete
2026-04-20 06:05:59.462170 I | ceph-object-controller: creating object store "ceph" in namespace "openstack"
2026-04-20 06:05:59.467084 I | cephclient: getting or creating ceph auth key "client.rgw.ceph.a"
2026-04-20 06:06:00.078568 I | ceph-object-controller: setting rgw config flags
2026-04-20 06:06:00.078600 I | op-config: setting "client.rgw.ceph.a"="rgw_run_sync_thread"="true" option to the mon configuration database
2026-04-20 06:06:00.526174 I | op-config: successfully set "client.rgw.ceph.a"="rgw_run_sync_thread"="true" option to the mon configuration database
2026-04-20 06:06:00.526204 I | op-config: setting "client.rgw.ceph.a"="rgw_log_nonexistent_bucket"="true" option to the mon configuration database
2026-04-20 06:06:00.981889 I | op-config: successfully set "client.rgw.ceph.a"="rgw_log_nonexistent_bucket"="true" option to the mon configuration database
2026-04-20 06:06:00.981922 I | op-config: setting "client.rgw.ceph.a"="rgw_log_object_name_utc"="true" option to the mon configuration database
2026-04-20 06:06:01.421721 I | op-config: successfully set "client.rgw.ceph.a"="rgw_log_object_name_utc"="true" option to the mon configuration database
2026-04-20 06:06:01.421756 I | op-config: setting "client.rgw.ceph.a"="rgw_enable_usage_log"="true" option to the mon configuration database
2026-04-20 06:06:01.880617 I | op-config: successfully set "client.rgw.ceph.a"="rgw_enable_usage_log"="true" option to the mon configuration database
2026-04-20 06:06:01.880692 I | op-config: setting "client.rgw.ceph.a"="rgw_zone"="ceph" option to the mon configuration database
2026-04-20 06:06:02.322951 I | op-config: successfully set "client.rgw.ceph.a"="rgw_zone"="ceph" option to the mon configuration database
2026-04-20 06:06:02.322991 I | op-config: setting "client.rgw.ceph.a"="rgw_zonegroup"="ceph" option to the mon configuration database
2026-04-20 06:06:02.771099 I | op-config: successfully set "client.rgw.ceph.a"="rgw_zonegroup"="ceph" option to the mon configuration database
2026-04-20 06:06:02.771420 I | ceph-object-controller: object store "ceph" deployment "rook-ceph-rgw-ceph-a" created
2026-04-20 06:06:02.812588 I | ceph-object-controller: enabling rgw dashboard
2026-04-20 06:06:04.830645 I | ceph-object-controller: created object store "ceph" in namespace "openstack"
2026-04-20 06:06:04.831389 I | ceph-object-controller: setting the dashboard api secret key
2026-04-20 06:06:05.460713 I | ceph-object-controller: done setting the dashboard api secret key
2026-04-20 06:06:19.599881 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-20 06:06:20.954677 W | op-mon: external cluster mon count is 1, consider adding new monitors.
2026-04-20 06:06:21.465066 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-20 06:06:35.233819 E | ceph-object-controller: failed to reconcile CephObjectStore "openstack/ceph". failed to create object store deployments: failed to get COSI user "cosi": Get "http://rook-ceph-rgw-ceph.openstack.svc:80/admin/user?format=json&uid=cosi": dial tcp 10.100.51.17:80: i/o timeout
2026-04-20 06:06:35.262876 I | ceph-spec: parsing mon endpoints: instance=10.96.240.200:6789
2026-04-20 06:06:35.262909 I | ceph-spec: updating obsolete maxMonID 0 to actual value 76846025234
2026-04-20 06:06:35.262927 I | ceph-spec: detecting the ceph image version for image harbor.atmosphere.dev/quay.io/ceph/ceph:v18.2.1...
2026-04-20 06:06:35.807441 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-20 06:06:36.370880 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-20 06:06:37.273826 I | ceph-spec: detected ceph image version: "18.2.1-0 reef"
2026-04-20 06:06:38.744438 I | ceph-object-controller: reconciling object store deployments
2026-04-20 06:06:38.774681 I | ceph-object-controller: ceph object store gateway service running at 10.100.51.17
2026-04-20 06:06:38.774709 I | ceph-object-controller: reconciling object store pools
2026-04-20 06:06:40.116065 I | cephclient: application "rgw" is already set on pool "ceph.rgw.control"
2026-04-20 06:06:40.116096 I | cephclient: reconciling replicated pool ceph.rgw.control succeeded
2026-04-20 06:06:40.989736 I | cephclient: setting pool property "pg_num_min" to "8" on pool "ceph.rgw.control"
2026-04-20 06:06:42.996229 I | cephclient: application "rgw" is already set on pool "ceph.rgw.meta"
2026-04-20 06:06:42.996261 I | cephclient: reconciling replicated pool ceph.rgw.meta succeeded
2026-04-20 06:06:43.876847 I | cephclient: setting pool property "pg_num_min" to "8" on pool "ceph.rgw.meta"
2026-04-20 06:06:46.070581 I | cephclient: application "rgw" is already set on pool "ceph.rgw.log"
2026-04-20 06:06:46.070615 I | cephclient: reconciling replicated pool ceph.rgw.log succeeded
2026-04-20 06:06:46.974821 I | cephclient: setting pool property "pg_num_min" to "8" on pool "ceph.rgw.log"
2026-04-20 06:06:49.115751 I | cephclient: application "rgw" is already set on pool "ceph.rgw.buckets.index"
2026-04-20 06:06:49.115787 I | cephclient: reconciling replicated pool ceph.rgw.buckets.index succeeded
2026-04-20 06:06:50.467123 I | cephclient: setting pool property "pg_num_min" to "8" on pool "ceph.rgw.buckets.index"
2026-04-20 06:06:50.786452 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-20 06:06:52.637543 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-20 06:06:54.771606 I | cephclient: application "rgw" is already set on pool "ceph.rgw.buckets.non-ec"
2026-04-20 06:06:54.771642 I | cephclient: reconciling replicated pool ceph.rgw.buckets.non-ec succeeded
2026-04-20 06:06:55.687197 I | cephclient: setting pool property "pg_num_min" to "8" on pool "ceph.rgw.buckets.non-ec"
2026-04-20 06:06:58.188132 I | cephclient: application "rgw" is already set on pool "ceph.rgw.otp"
2026-04-20 06:06:58.188169 I | cephclient: reconciling replicated pool ceph.rgw.otp succeeded
2026-04-20 06:06:59.095037 I | cephclient: setting pool property "pg_num_min" to "8" on pool "ceph.rgw.otp"
2026-04-20 06:07:01.260502 I | cephclient: application "rgw" is already set on pool ".rgw.root"
2026-04-20 06:07:01.260536 I | cephclient: reconciling replicated pool .rgw.root succeeded
2026-04-20 06:07:02.181984 I | cephclient: setting pool property "pg_num_min" to "8" on pool ".rgw.root"
2026-04-20 06:07:04.309519 I | cephclient: application "rgw" is already set on pool "ceph.rgw.buckets.data"
2026-04-20 06:07:04.309555 I | cephclient: reconciling replicated pool ceph.rgw.buckets.data succeeded
2026-04-20 06:07:05.203263 I | ceph-object-controller: configuring object store "ceph"
2026-04-20 06:07:05.450632 I | ceph-object-controller: Object store "ceph": realm=ceph, zonegroup=ceph, zone=ceph
2026-04-20 06:07:05.621724 I | ceph-object-controller: there are no changes to commit for RGW configuration period for CephObjectStore "openstack/ceph"
2026-04-20 06:07:05.621752 I | ceph-object-controller: configuration for object-store ceph is complete
2026-04-20 06:07:05.621757 I | ceph-object-controller: creating object store "ceph" in namespace "openstack"
2026-04-20 06:07:05.626224 I | cephclient: getting or creating ceph auth key "client.rgw.ceph.a"
2026-04-20 06:07:06.460058 I | ceph-object-controller: setting rgw config flags
2026-04-20 06:07:06.460084 I | op-config: setting "client.rgw.ceph.a"="rgw_enable_usage_log"="true" option to the mon configuration database
2026-04-20 06:07:07.384567 I | op-config: successfully set "client.rgw.ceph.a"="rgw_enable_usage_log"="true" option to the mon configuration database
2026-04-20 06:07:07.384603 I | op-config: setting "client.rgw.ceph.a"="rgw_zone"="ceph" option to the mon configuration database
2026-04-20 06:07:08.355915 W | op-mon: external cluster mon count is 1, consider adding new monitors.
2026-04-20 06:07:08.365633 I | op-config: successfully set "client.rgw.ceph.a"="rgw_zone"="ceph" option to the mon configuration database
2026-04-20 06:07:08.365678 I | op-config: setting "client.rgw.ceph.a"="rgw_zonegroup"="ceph" option to the mon configuration database
2026-04-20 06:07:08.837250 I | op-config: successfully set "client.rgw.ceph.a"="rgw_zonegroup"="ceph" option to the mon configuration database
2026-04-20 06:07:08.837290 I | op-config: setting "client.rgw.ceph.a"="rgw_run_sync_thread"="true" option to the mon configuration database
2026-04-20 06:07:09.280413 I | op-config: successfully set "client.rgw.ceph.a"="rgw_run_sync_thread"="true" option to the mon configuration database
2026-04-20 06:07:09.280455 I | op-config: setting "client.rgw.ceph.a"="rgw_log_nonexistent_bucket"="true" option to the mon configuration database
2026-04-20 06:07:09.726194 I | op-config: successfully set "client.rgw.ceph.a"="rgw_log_nonexistent_bucket"="true" option to the mon configuration database
2026-04-20 06:07:09.726240 I | op-config: setting "client.rgw.ceph.a"="rgw_log_object_name_utc"="true" option to the mon configuration database
2026-04-20 06:07:10.167694 I | op-config: successfully set "client.rgw.ceph.a"="rgw_log_object_name_utc"="true" option to the mon configuration database
2026-04-20 06:07:10.168816 I | ceph-object-controller: object store "ceph" deployment "rook-ceph-rgw-ceph-a" created
2026-04-20 06:07:10.184154 I | ceph-object-controller: object store "ceph" deployment "rook-ceph-rgw-ceph-a" already exists. updating if needed
2026-04-20 06:07:10.194764 I | op-k8sutil: deployment "rook-ceph-rgw-ceph-a" did not change, nothing to update
2026-04-20 06:07:10.198912 I | ceph-object-controller: config map "rook-ceph-rgw-ceph-mime-types" for object store "ceph" already exists, not overwriting
2026-04-20 06:07:10.205201 I | ceph-object-controller: enabling rgw dashboard
2026-04-20 06:07:11.387015 I | ceph-object-controller: created object store "ceph" in namespace "openstack"
2026-04-20 06:07:13.278514 I | ceph-object-controller: creating COSI user "cosi"
2026-04-20 06:07:13.373848 I | ceph-spec: created ceph *v1.Secret object "rook-ceph-object-user-ceph-cosi"
2026-04-20 06:07:13.930927 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-20 06:07:21.338709 I | clusterdisruption-controller: all "host" failure domains: []. osd is down in failure domain: "". active node drains: false. pg health: "cluster is not fully clean. PGs: [{StateName:active+clean Count:92} {StateName:creating+peering Count:5}]"
2026-04-20 06:07:21.341951 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-20 06:07:23.247248 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-20 06:07:51.899959 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-20 06:07:54.277220 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-20 06:07:55.634646 W | op-mon: external cluster mon count is 1, consider adding new monitors.
2026-04-20 06:08:22.470287 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-20 06:08:24.859757 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-20 06:08:41.770950 W | op-mon: external cluster mon count is 1, consider adding new monitors.
2026-04-20 06:08:53.046277 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-20 06:08:55.428939 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-20 06:09:23.602116 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-20 06:09:25.983377 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-20 06:09:27.923346 W | op-mon: external cluster mon count is 1, consider adding new monitors.
2026-04-20 06:09:54.154941 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-20 06:09:56.533336 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-20 06:10:14.026929 W | op-mon: external cluster mon count is 1, consider adding new monitors.
2026-04-20 06:10:24.703889 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-20 06:10:27.097660 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-20 06:10:55.265133 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-20 06:10:57.692637 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-20 06:11:00.973909 W | op-mon: external cluster mon count is 1, consider adding new monitors.
2026-04-20 06:11:25.813681 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-20 06:11:28.254261 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-20 06:11:47.086492 W | op-mon: external cluster mon count is 1, consider adding new monitors.
2026-04-20 06:11:56.368370 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-20 06:11:58.810027 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-20 06:12:26.940762 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-20 06:12:29.374570 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-20 06:12:33.192326 W | op-mon: external cluster mon count is 1, consider adding new monitors.
2026-04-20 06:12:57.518861 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-20 06:12:59.983334 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-20 06:13:19.287334 W | op-mon: external cluster mon count is 1, consider adding new monitors.
2026-04-20 06:13:28.075132 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-20 06:13:30.565503 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-20 06:13:58.641169 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-20 06:14:01.123534 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-20 06:14:05.414702 W | op-mon: external cluster mon count is 1, consider adding new monitors.
2026-04-20 06:14:29.213079 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-20 06:14:31.702241 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-20 06:14:51.554662 W | op-mon: external cluster mon count is 1, consider adding new monitors.
2026-04-20 06:14:59.779172 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-20 06:15:02.257363 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-20 06:15:30.351980 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-20 06:15:32.843479 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0