I0403 02:29:33.765222 1 cephcsi.go:171] Driver version: v3.5.1 and Git version: 00f88e58a0b08be3d7ea009a0cfde8dca23021bf I0403 02:29:33.765398 1 cephcsi.go:220] Starting driver type: controller with name: rbd.csi.ceph.com I0403 02:29:33.866628 1 request.go:597] Waited for 84.917957ms due to client-side throttling, not priority and fairness, request: GET:https://10.96.0.1:443/api/v1?timeout=32s I0403 02:29:33.872032 1 shared_informer.go:270] caches populated I0403 02:29:33.872079 1 leaderelection.go:248] attempting to acquire leader lease kube-system/rbd.csi.ceph.com-kube-system... I0403 02:29:33.918433 1 leaderelection.go:258] successfully acquired lease kube-system/rbd.csi.ceph.com-kube-system I0403 02:29:33.919451 1 reflector.go:219] Starting reflector *v1.PersistentVolume (10h8m1.029900152s) from sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:250 I0403 02:29:33.919480 1 reflector.go:255] Listing and watching *v1.PersistentVolume from sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:250 I0403 02:29:33.957241 1 leaderelection.go:278] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0403 02:29:34.019241 1 shared_informer.go:270] caches populated I0403 02:29:34.019368 1 shared_informer.go:270] caches populated I0403 02:29:36.013751 1 leaderelection.go:278] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0403 02:29:38.341508 1 leaderelection.go:278] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0403 02:29:40.609435 1 leaderelection.go:278] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0403 02:29:42.657539 1 leaderelection.go:278] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0403 02:29:44.726732 1 leaderelection.go:278] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0403 02:29:46.767127 1 leaderelection.go:278] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0403 02:29:48.872595 1 leaderelection.go:278] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0403 02:29:50.998217 1 leaderelection.go:278] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0403 02:29:53.023767 1 leaderelection.go:278] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0403 02:29:55.046465 1 leaderelection.go:278] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0403 02:29:57.081532 1 leaderelection.go:278] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0403 02:29:59.135276 1 leaderelection.go:278] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0403 02:30:01.181469 1 leaderelection.go:278] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0403 02:30:03.334880 1 leaderelection.go:278] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0403 02:30:05.404888 1 leaderelection.go:278] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0403 02:30:07.701242 1 leaderelection.go:278] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0403 02:30:09.774835 1 leaderelection.go:278] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0403 02:30:12.688710 1 leaderelection.go:278] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0403 02:30:14.802093 1 leaderelection.go:278] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0403 02:30:17.044474 1 leaderelection.go:278] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0403 02:30:19.113845 1 leaderelection.go:278] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0403 02:30:21.131793 1 leaderelection.go:278] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0403 02:30:23.150668 1 leaderelection.go:278] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0403 02:30:25.309860 1 leaderelection.go:278] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0403 02:30:27.451424 1 leaderelection.go:278] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0403 02:30:29.501019 1 leaderelection.go:278] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0403 02:30:31.530776 1 leaderelection.go:278] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0403 02:30:33.602631 1 leaderelection.go:278] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0403 02:30:35.682828 1 leaderelection.go:278] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0403 02:30:37.821269 1 leaderelection.go:278] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0403 02:30:39.843577 1 leaderelection.go:278] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0403 02:30:42.044295 1 leaderelection.go:278] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0403 02:30:44.072412 1 leaderelection.go:278] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0403 02:30:46.093217 1 leaderelection.go:278] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0403 02:30:48.169084 1 leaderelection.go:278] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0403 02:30:50.223938 1 leaderelection.go:278] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0403 02:30:52.334665 1 leaderelection.go:278] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0403 02:30:54.353764 1 leaderelection.go:278] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0403 02:30:56.620172 1 leaderelection.go:278] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0403 02:30:58.655001 1 leaderelection.go:278] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0403 02:31:00.676621 1 leaderelection.go:278] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0403 02:31:02.740756 1 leaderelection.go:278] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0403 02:31:04.892112 1 leaderelection.go:278] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0403 02:31:07.238222 1 leaderelection.go:278] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0403 02:31:09.289693 1 leaderelection.go:278] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0403 02:31:11.590941 1 leaderelection.go:278] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0403 02:31:13.614721 1 leaderelection.go:278] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0403 02:31:15.648956 1 leaderelection.go:278] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0403 02:31:17.671876 1 leaderelection.go:278] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0403 02:31:19.725704 1 leaderelection.go:278] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0403 02:31:21.859997 1 leaderelection.go:278] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0403 02:31:23.885682 1 leaderelection.go:278] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0403 02:31:25.903014 1 leaderelection.go:278] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0403 02:31:28.018912 1 leaderelection.go:278] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0403 02:31:30.087107 1 leaderelection.go:278] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0403 02:31:32.192927 1 leaderelection.go:278] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0403 02:31:34.605094 1 leaderelection.go:278] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0403 02:31:36.629167 1 leaderelection.go:278] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0403 02:31:38.647828 1 leaderelection.go:278] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0403 02:31:40.665022 1 leaderelection.go:278] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0403 02:31:42.691367 1 leaderelection.go:278] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0403 02:31:44.707944 1 leaderelection.go:278] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0403 02:31:46.724522 1 leaderelection.go:278] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0403 02:31:48.743710 1 leaderelection.go:278] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0403 02:31:50.764246 1 leaderelection.go:278] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0403 02:31:52.784206 1 leaderelection.go:278] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0403 02:31:54.864847 1 leaderelection.go:278] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0403 02:31:56.879612 1 leaderelection.go:278] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0403 02:31:58.897159 1 leaderelection.go:278] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0403 02:32:00.913413 1 leaderelection.go:278] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0403 02:32:02.935748 1 leaderelection.go:278] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0403 02:32:04.949808 1 leaderelection.go:278] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0403 02:32:06.967534 1 leaderelection.go:278] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0403 02:32:08.986530 1 leaderelection.go:278] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0403 02:32:11.004512 1 leaderelection.go:278] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0403 02:32:13.022399 1 leaderelection.go:278] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0403 02:32:15.042349 1 leaderelection.go:278] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0403 02:32:17.062736 1 leaderelection.go:278] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0403 02:32:19.079789 1 leaderelection.go:278] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0403 02:32:21.108457 1 leaderelection.go:278] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0403 02:32:23.128448 1 leaderelection.go:278] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0403 02:32:25.147225 1 leaderelection.go:278] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0403 02:32:27.167055 1 leaderelection.go:278] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0403 02:32:29.187240 1 leaderelection.go:278] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0403 02:32:31.208132 1 leaderelection.go:278] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0403 02:32:33.224041 1 leaderelection.go:278] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0403 02:32:35.246681 1 leaderelection.go:278] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0403 02:32:37.266784 1 leaderelection.go:278] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0403 02:32:39.294921 1 leaderelection.go:278] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0403 02:32:39.725957 1 reflector.go:219] Starting reflector *v1.Secret (10h47m6.411663472s) from sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:250 I0403 02:32:39.725976 1 reflector.go:255] Listing and watching *v1.Secret from sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:250 I0403 02:32:39.826497 1 shared_informer.go:270] caches populated I0403 02:32:39.846795 1 omap.go:87] got omap values: (pool="kube", namespace="", name="csi.volumes.default"): map[csi.volume.pvc-866621a7-cbd2-41f8-8328-c960277aa3ac:611fa5ec-2f05-11f1-850d-b2529849fc99] I0403 02:32:39.847689 1 omap.go:87] got omap values: (pool="kube", namespace="", name="csi.volume.611fa5ec-2f05-11f1-850d-b2529849fc99"): map[csi.imageid:37d7517aa880 csi.imagename:csi-vol-611fa5ec-2f05-11f1-850d-b2529849fc99 csi.volname:pvc-866621a7-cbd2-41f8-8328-c960277aa3ac csi.volume.owner:default] I0403 02:32:39.848592 1 omap.go:87] got omap values: (pool="kube", namespace="", name="csi.volumes.default"): map[csi.volume.pvc-866621a7-cbd2-41f8-8328-c960277aa3ac:611fa5ec-2f05-11f1-850d-b2529849fc99] I0403 02:32:39.849263 1 omap.go:87] got omap values: (pool="kube", namespace="", name="csi.volume.611fa5ec-2f05-11f1-850d-b2529849fc99"): map[csi.imageid:37d7517aa880 csi.imagename:csi-vol-611fa5ec-2f05-11f1-850d-b2529849fc99 csi.volname:pvc-866621a7-cbd2-41f8-8328-c960277aa3ac csi.volume.owner:default] I0403 02:32:41.314761 1 leaderelection.go:278] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0403 02:32:43.432011 1 leaderelection.go:278] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0403 02:32:45.733035 1 leaderelection.go:278] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0403 02:32:48.663470 1 leaderelection.go:278] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0403 02:32:51.527053 1 leaderelection.go:278] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0403 02:32:53.917450 1 leaderelection.go:278] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0403 02:32:56.151624 1 leaderelection.go:278] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0403 02:32:59.431704 1 leaderelection.go:278] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0403 02:33:03.295156 1 leaderelection.go:278] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0403 02:33:06.026223 1 leaderelection.go:278] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0403 02:33:09.694252 1 leaderelection.go:278] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0403 02:33:12.090855 1 leaderelection.go:278] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0403 02:33:14.371683 1 leaderelection.go:278] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0403 02:33:16.983856 1 leaderelection.go:278] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0403 02:33:20.681097 1 leaderelection.go:278] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0403 02:33:23.958697 1 leaderelection.go:278] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0403 02:33:26.503745 1 leaderelection.go:278] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0403 02:33:28.785396 1 leaderelection.go:278] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0403 02:33:31.883339 1 leaderelection.go:278] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0403 02:33:34.292802 1 leaderelection.go:278] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0403 02:33:37.314316 1 leaderelection.go:278] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0403 02:33:39.669675 1 leaderelection.go:278] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0403 02:33:42.071025 1 leaderelection.go:278] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0403 02:33:44.946448 1 leaderelection.go:278] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system E0403 02:33:56.947643 1 leaderelection.go:330] error retrieving resource lock kube-system/rbd.csi.ceph.com-kube-system: Get "https://10.96.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/rbd.csi.ceph.com-kube-system": context deadline exceeded I0403 02:33:56.947729 1 leaderelection.go:283] failed to renew lease kube-system/rbd.csi.ceph.com-kube-system: timed out waiting for the condition E0403 02:33:56.947782 1 controller.go:80] failed to start manager leader election lost E0403 02:33:56.947799 1 cephcsi.go:283] leader election lost