I0411 02:12:18.515156 1 options.go:220] external host was not specified, using 199.204.45.72 I0411 02:12:18.516367 1 server.go:148] Version: v1.28.13 I0411 02:12:18.516492 1 server.go:150] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" I0411 02:12:18.917526 1 shared_informer.go:311] Waiting for caches to sync for node_authorizer I0411 02:12:18.928591 1 plugins.go:158] Loaded 12 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,StorageObjectInUseProtection,RuntimeClass,DefaultIngressClass,MutatingAdmissionWebhook. I0411 02:12:18.928621 1 plugins.go:161] Loaded 13 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,PodSecurity,Priority,PersistentVolumeClaimResize,RuntimeClass,CertificateApproval,CertificateSigning,ClusterTrustBundleAttest,CertificateSubjectRestriction,ValidatingAdmissionPolicy,ValidatingAdmissionWebhook,ResourceQuota. I0411 02:12:18.928925 1 instance.go:298] Using reconciler: lease I0411 02:12:19.437862 1 handler.go:275] Adding GroupVersion apiextensions.k8s.io v1 to ResourceManager W0411 02:12:19.437925 1 genericapiserver.go:744] Skipping API apiextensions.k8s.io/v1beta1 because it has no resources. I0411 02:12:19.714390 1 handler.go:275] Adding GroupVersion v1 to ResourceManager I0411 02:12:19.715827 1 instance.go:709] API group "internal.apiserver.k8s.io" is not enabled, skipping. I0411 02:12:20.220324 1 instance.go:709] API group "resource.k8s.io" is not enabled, skipping. I0411 02:12:20.232309 1 handler.go:275] Adding GroupVersion authentication.k8s.io v1 to ResourceManager W0411 02:12:20.232413 1 genericapiserver.go:744] Skipping API authentication.k8s.io/v1beta1 because it has no resources. W0411 02:12:20.232475 1 genericapiserver.go:744] Skipping API authentication.k8s.io/v1alpha1 because it has no resources. I0411 02:12:20.233168 1 handler.go:275] Adding GroupVersion authorization.k8s.io v1 to ResourceManager W0411 02:12:20.233222 1 genericapiserver.go:744] Skipping API authorization.k8s.io/v1beta1 because it has no resources. I0411 02:12:20.234405 1 handler.go:275] Adding GroupVersion autoscaling v2 to ResourceManager I0411 02:12:20.235368 1 handler.go:275] Adding GroupVersion autoscaling v1 to ResourceManager W0411 02:12:20.235421 1 genericapiserver.go:744] Skipping API autoscaling/v2beta1 because it has no resources. W0411 02:12:20.235446 1 genericapiserver.go:744] Skipping API autoscaling/v2beta2 because it has no resources. I0411 02:12:20.237386 1 handler.go:275] Adding GroupVersion batch v1 to ResourceManager W0411 02:12:20.237432 1 genericapiserver.go:744] Skipping API batch/v1beta1 because it has no resources. I0411 02:12:20.238168 1 handler.go:275] Adding GroupVersion certificates.k8s.io v1 to ResourceManager W0411 02:12:20.238208 1 genericapiserver.go:744] Skipping API certificates.k8s.io/v1beta1 because it has no resources. W0411 02:12:20.238228 1 genericapiserver.go:744] Skipping API certificates.k8s.io/v1alpha1 because it has no resources. I0411 02:12:20.238725 1 handler.go:275] Adding GroupVersion coordination.k8s.io v1 to ResourceManager W0411 02:12:20.238762 1 genericapiserver.go:744] Skipping API coordination.k8s.io/v1beta1 because it has no resources. W0411 02:12:20.238808 1 genericapiserver.go:744] Skipping API discovery.k8s.io/v1beta1 because it has no resources. I0411 02:12:20.239326 1 handler.go:275] Adding GroupVersion discovery.k8s.io v1 to ResourceManager I0411 02:12:20.240879 1 handler.go:275] Adding GroupVersion networking.k8s.io v1 to ResourceManager W0411 02:12:20.240919 1 genericapiserver.go:744] Skipping API networking.k8s.io/v1beta1 because it has no resources. W0411 02:12:20.240934 1 genericapiserver.go:744] Skipping API networking.k8s.io/v1alpha1 because it has no resources. I0411 02:12:20.241428 1 handler.go:275] Adding GroupVersion node.k8s.io v1 to ResourceManager W0411 02:12:20.241468 1 genericapiserver.go:744] Skipping API node.k8s.io/v1beta1 because it has no resources. W0411 02:12:20.241484 1 genericapiserver.go:744] Skipping API node.k8s.io/v1alpha1 because it has no resources. I0411 02:12:20.242277 1 handler.go:275] Adding GroupVersion policy v1 to ResourceManager W0411 02:12:20.242321 1 genericapiserver.go:744] Skipping API policy/v1beta1 because it has no resources. I0411 02:12:20.243885 1 handler.go:275] Adding GroupVersion rbac.authorization.k8s.io v1 to ResourceManager W0411 02:12:20.243924 1 genericapiserver.go:744] Skipping API rbac.authorization.k8s.io/v1beta1 because it has no resources. W0411 02:12:20.243942 1 genericapiserver.go:744] Skipping API rbac.authorization.k8s.io/v1alpha1 because it has no resources. I0411 02:12:20.244446 1 handler.go:275] Adding GroupVersion scheduling.k8s.io v1 to ResourceManager W0411 02:12:20.244485 1 genericapiserver.go:744] Skipping API scheduling.k8s.io/v1beta1 because it has no resources. W0411 02:12:20.244501 1 genericapiserver.go:744] Skipping API scheduling.k8s.io/v1alpha1 because it has no resources. I0411 02:12:20.246381 1 handler.go:275] Adding GroupVersion storage.k8s.io v1 to ResourceManager W0411 02:12:20.246421 1 genericapiserver.go:744] Skipping API storage.k8s.io/v1beta1 because it has no resources. W0411 02:12:20.246436 1 genericapiserver.go:744] Skipping API storage.k8s.io/v1alpha1 because it has no resources. I0411 02:12:20.247318 1 handler.go:275] Adding GroupVersion flowcontrol.apiserver.k8s.io v1beta3 to ResourceManager I0411 02:12:20.248194 1 handler.go:275] Adding GroupVersion flowcontrol.apiserver.k8s.io v1beta2 to ResourceManager W0411 02:12:20.248232 1 genericapiserver.go:744] Skipping API flowcontrol.apiserver.k8s.io/v1beta1 because it has no resources. W0411 02:12:20.248248 1 genericapiserver.go:744] Skipping API flowcontrol.apiserver.k8s.io/v1alpha1 because it has no resources. I0411 02:12:20.251252 1 handler.go:275] Adding GroupVersion apps v1 to ResourceManager W0411 02:12:20.251294 1 genericapiserver.go:744] Skipping API apps/v1beta2 because it has no resources. W0411 02:12:20.251309 1 genericapiserver.go:744] Skipping API apps/v1beta1 because it has no resources. I0411 02:12:20.252057 1 handler.go:275] Adding GroupVersion admissionregistration.k8s.io v1 to ResourceManager W0411 02:12:20.252099 1 genericapiserver.go:744] Skipping API admissionregistration.k8s.io/v1beta1 because it has no resources. W0411 02:12:20.252113 1 genericapiserver.go:744] Skipping API admissionregistration.k8s.io/v1alpha1 because it has no resources. I0411 02:12:20.252614 1 handler.go:275] Adding GroupVersion events.k8s.io v1 to ResourceManager W0411 02:12:20.252657 1 genericapiserver.go:744] Skipping API events.k8s.io/v1beta1 because it has no resources. I0411 02:12:20.302407 1 handler.go:275] Adding GroupVersion apiregistration.k8s.io v1 to ResourceManager W0411 02:12:20.302434 1 genericapiserver.go:744] Skipping API apiregistration.k8s.io/v1beta1 because it has no resources. I0411 02:12:20.747690 1 dynamic_cafile_content.go:157] "Starting controller" name="request-header::/etc/kubernetes/pki/front-proxy-ca.crt" I0411 02:12:20.747707 1 dynamic_cafile_content.go:157] "Starting controller" name="client-ca-bundle::/etc/kubernetes/pki/ca.crt" I0411 02:12:20.747996 1 dynamic_serving_content.go:132] "Starting controller" name="serving-cert::/etc/kubernetes/pki/apiserver.crt::/etc/kubernetes/pki/apiserver.key" I0411 02:12:20.748063 1 secure_serving.go:213] Serving securely on [::]:6443 I0411 02:12:20.748138 1 tlsconfig.go:240] "Starting DynamicServingCertificateController" I0411 02:12:20.748183 1 aggregator.go:164] waiting for initial CRD sync... I0411 02:12:20.748422 1 system_namespaces_controller.go:67] Starting system namespaces controller I0411 02:12:20.748705 1 crdregistration_controller.go:111] Starting crd-autoregister controller I0411 02:12:20.748720 1 shared_informer.go:311] Waiting for caches to sync for crd-autoregister I0411 02:12:20.748756 1 apiservice_controller.go:97] Starting APIServiceRegistrationController I0411 02:12:20.748786 1 cache.go:32] Waiting for caches to sync for APIServiceRegistrationController controller I0411 02:12:20.748896 1 cluster_authentication_trust_controller.go:440] Starting cluster_authentication_trust_controller controller I0411 02:12:20.748988 1 shared_informer.go:311] Waiting for caches to sync for cluster_authentication_trust_controller I0411 02:12:20.749015 1 customresource_discovery_controller.go:289] Starting DiscoveryController I0411 02:12:20.749111 1 handler_discovery.go:412] Starting ResourceDiscoveryManager I0411 02:12:20.749134 1 gc_controller.go:78] Starting apiserver lease garbage collector I0411 02:12:20.749157 1 controller.go:78] Starting OpenAPI AggregationController I0411 02:12:20.749186 1 controller.go:85] Starting OpenAPI V3 controller I0411 02:12:20.749350 1 gc_controller.go:78] Starting apiserver lease garbage collector I0411 02:12:20.749361 1 naming_controller.go:291] Starting NamingConditionController I0411 02:12:20.749424 1 establishing_controller.go:76] Starting EstablishingController I0411 02:12:20.749433 1 dynamic_serving_content.go:132] "Starting controller" name="aggregator-proxy-cert::/etc/kubernetes/pki/front-proxy-client.crt::/etc/kubernetes/pki/front-proxy-client.key" I0411 02:12:20.749470 1 nonstructuralschema_controller.go:192] Starting NonStructuralSchemaConditionController I0411 02:12:20.749579 1 apiapproval_controller.go:186] Starting KubernetesAPIApprovalPolicyConformantConditionController I0411 02:12:20.749606 1 available_controller.go:423] Starting AvailableConditionController I0411 02:12:20.749616 1 cache.go:32] Waiting for caches to sync for AvailableConditionController controller I0411 02:12:20.749659 1 crd_finalizer.go:266] Starting CRDFinalizer I0411 02:12:20.749220 1 controller.go:134] Starting OpenAPI controller I0411 02:12:20.749189 1 controller.go:80] Starting OpenAPI V3 AggregationController I0411 02:12:20.750158 1 controller.go:116] Starting legacy_token_tracking_controller I0411 02:12:20.750188 1 shared_informer.go:311] Waiting for caches to sync for configmaps I0411 02:12:20.748459 1 apf_controller.go:374] Starting API Priority and Fairness config controller I0411 02:12:20.750719 1 dynamic_cafile_content.go:157] "Starting controller" name="client-ca-bundle::/etc/kubernetes/pki/ca.crt" I0411 02:12:20.750868 1 dynamic_cafile_content.go:157] "Starting controller" name="request-header::/etc/kubernetes/pki/front-proxy-ca.crt" I0411 02:12:20.818239 1 shared_informer.go:318] Caches are synced for node_authorizer E0411 02:12:20.823085 1 controller.go:146] "Failed to ensure lease exists, will retry" err="namespaces \"kube-system\" not found" interval="200ms" I0411 02:12:20.849381 1 shared_informer.go:318] Caches are synced for crd-autoregister I0411 02:12:20.849437 1 aggregator.go:166] initial CRD sync complete... I0411 02:12:20.849448 1 autoregister_controller.go:141] Starting autoregister controller I0411 02:12:20.849458 1 cache.go:32] Waiting for caches to sync for autoregister controller I0411 02:12:20.849507 1 cache.go:39] Caches are synced for autoregister controller I0411 02:12:20.849480 1 cache.go:39] Caches are synced for APIServiceRegistrationController controller I0411 02:12:20.849792 1 shared_informer.go:318] Caches are synced for cluster_authentication_trust_controller I0411 02:12:20.849884 1 cache.go:39] Caches are synced for AvailableConditionController controller I0411 02:12:20.850298 1 shared_informer.go:318] Caches are synced for configmaps I0411 02:12:20.850727 1 apf_controller.go:379] Running API Priority and Fairness config worker I0411 02:12:20.850750 1 apf_controller.go:382] Running API Priority and Fairness periodic rebalancing process I0411 02:12:20.851361 1 controller.go:624] quota admission added evaluator for: namespaces I0411 02:12:21.027263 1 controller.go:624] quota admission added evaluator for: leases.coordination.k8s.io I0411 02:12:21.756257 1 storage_scheduling.go:95] created PriorityClass system-node-critical with value 2000001000 I0411 02:12:21.760864 1 storage_scheduling.go:95] created PriorityClass system-cluster-critical with value 2000000000 I0411 02:12:21.760885 1 storage_scheduling.go:111] all system priority classes are created successfully or already exist. I0411 02:12:22.387520 1 controller.go:624] quota admission added evaluator for: roles.rbac.authorization.k8s.io I0411 02:12:22.435608 1 controller.go:624] quota admission added evaluator for: rolebindings.rbac.authorization.k8s.io I0411 02:12:22.567510 1 alloc.go:330] "allocated clusterIPs" service="default/kubernetes" clusterIPs={"IPv4":"10.96.0.1"} W0411 02:12:22.577319 1 lease.go:263] Resetting endpoints for master service "kubernetes" to [199.204.45.72] I0411 02:12:22.578353 1 controller.go:624] quota admission added evaluator for: endpoints I0411 02:12:22.583985 1 controller.go:624] quota admission added evaluator for: endpointslices.discovery.k8s.io I0411 02:12:23.945529 1 controller.go:624] quota admission added evaluator for: serviceaccounts I0411 02:12:24.265438 1 controller.go:624] quota admission added evaluator for: deployments.apps I0411 02:12:24.303502 1 alloc.go:330] "allocated clusterIPs" service="kube-system/kube-dns" clusterIPs={"IPv4":"10.96.0.10"} I0411 02:12:24.503080 1 controller.go:624] quota admission added evaluator for: daemonsets.apps I0411 02:12:28.988560 1 controller.go:624] quota admission added evaluator for: replicasets.apps I0411 02:12:29.035024 1 controller.go:624] quota admission added evaluator for: controllerrevisions.apps I0411 02:12:49.670957 1 handler.go:275] Adding GroupVersion gateway.networking.x-k8s.io v1alpha1 to ResourceManager I0411 02:12:49.703284 1 handler.go:275] Adding GroupVersion gateway.networking.k8s.io v1 to ResourceManager I0411 02:12:49.703320 1 handler.go:275] Adding GroupVersion gateway.networking.k8s.io v1beta1 to ResourceManager I0411 02:12:49.826329 1 handler.go:275] Adding GroupVersion gateway.networking.k8s.io v1 to ResourceManager I0411 02:12:49.826402 1 handler.go:275] Adding GroupVersion gateway.networking.k8s.io v1alpha3 to ResourceManager I0411 02:12:49.847640 1 handler.go:275] Adding GroupVersion gateway.networking.k8s.io v1beta1 to ResourceManager I0411 02:12:49.869159 1 handler.go:275] Adding GroupVersion gateway.networking.k8s.io v1alpha2 to ResourceManager I0411 02:12:49.883697 1 handler.go:275] Adding GroupVersion gateway.networking.x-k8s.io v1alpha1 to ResourceManager I0411 02:12:49.911536 1 handler.go:275] Adding GroupVersion gateway.networking.k8s.io v1alpha2 to ResourceManager I0411 02:12:49.936767 1 handler.go:275] Adding GroupVersion gateway.networking.x-k8s.io v1alpha1 to ResourceManager I0411 02:12:49.967255 1 handler.go:275] Adding GroupVersion gateway.networking.k8s.io v1alpha2 to ResourceManager I0411 02:12:49.967311 1 handler.go:275] Adding GroupVersion gateway.networking.k8s.io v1alpha3 to ResourceManager I0411 02:12:50.010324 1 handler.go:275] Adding GroupVersion gateway.networking.k8s.io v1 to ResourceManager I0411 02:12:50.010397 1 handler.go:275] Adding GroupVersion gateway.networking.k8s.io v1beta1 to ResourceManager I0411 02:12:50.058917 1 handler.go:275] Adding GroupVersion gateway.networking.k8s.io v1 to ResourceManager I0411 02:12:50.207501 1 handler.go:275] Adding GroupVersion gateway.networking.k8s.io v1 to ResourceManager I0411 02:12:50.207554 1 handler.go:275] Adding GroupVersion gateway.networking.k8s.io v1beta1 to ResourceManager I0411 02:12:50.219546 1 handler.go:275] Adding GroupVersion gateway.envoyproxy.io v1alpha1 to ResourceManager I0411 02:12:50.270702 1 handler.go:275] Adding GroupVersion gateway.envoyproxy.io v1alpha1 to ResourceManager I0411 02:12:50.293038 1 handler.go:275] Adding GroupVersion gateway.envoyproxy.io v1alpha1 to ResourceManager I0411 02:12:50.341175 1 handler.go:275] Adding GroupVersion gateway.envoyproxy.io v1alpha1 to ResourceManager I0411 02:12:50.357662 1 handler.go:275] Adding GroupVersion gateway.envoyproxy.io v1alpha1 to ResourceManager I0411 02:12:51.029531 1 handler.go:275] Adding GroupVersion gateway.envoyproxy.io v1alpha1 to ResourceManager I0411 02:12:51.031083 1 handler.go:275] Adding GroupVersion gateway.envoyproxy.io v1alpha1 to ResourceManager I0411 02:12:51.097577 1 handler.go:275] Adding GroupVersion gateway.envoyproxy.io v1alpha1 to ResourceManager I0411 02:12:55.948674 1 trace.go:236] Trace[1629943325]: "Update" accept:application/json, */*,audit-id:9879c225-ed0e-42a9-8808-6e5658549053,client:199.204.45.72,protocol:HTTP/2.0,resource:secrets,scope:resource,url:/api/v1/namespaces/envoy-gateway-system/secrets/sh.helm.release.v1.envoy-gateway.v1,user-agent:Helm/3.11.2,verb:PUT (11-Apr-2026 02:12:55.291) (total time: 657ms): Trace[1629943325]: ---"Conversion done" 23ms (02:12:55.319) Trace[1629943325]: ["GuaranteedUpdate etcd3" audit-id:9879c225-ed0e-42a9-8808-6e5658549053,key:/secrets/envoy-gateway-system/sh.helm.release.v1.envoy-gateway.v1,type:*core.Secret,resource:secrets 629ms (02:12:55.319) Trace[1629943325]: ---"Txn call completed" 575ms (02:12:55.918)] Trace[1629943325]: ---"Writing http response done" 29ms (02:12:55.948) Trace[1629943325]: [657.246079ms] [657.246079ms] END I0411 02:12:56.989859 1 controller.go:624] quota admission added evaluator for: jobs.batch I0411 02:12:57.351224 1 handler.go:275] Adding GroupVersion cilium.io v2 to ResourceManager I0411 02:12:57.359768 1 handler.go:275] Adding GroupVersion cilium.io v2alpha1 to ResourceManager I0411 02:12:57.547299 1 handler.go:275] Adding GroupVersion cilium.io v2 to ResourceManager I0411 02:12:57.749381 1 handler.go:275] Adding GroupVersion cilium.io v2alpha1 to ResourceManager I0411 02:12:57.959968 1 handler.go:275] Adding GroupVersion cilium.io v2alpha1 to ResourceManager I0411 02:12:58.145564 1 handler.go:275] Adding GroupVersion cilium.io v2alpha1 to ResourceManager I0411 02:12:59.504960 1 handler.go:275] Adding GroupVersion cilium.io v2alpha1 to ResourceManager I0411 02:12:59.523141 1 handler.go:275] Adding GroupVersion cilium.io v2 to ResourceManager I0411 02:13:00.580330 1 handler.go:275] Adding GroupVersion cilium.io v2 to ResourceManager I0411 02:13:01.154059 1 handler.go:275] Adding GroupVersion cilium.io v2 to ResourceManager I0411 02:13:01.556532 1 handler.go:275] Adding GroupVersion cilium.io v2 to ResourceManager W0411 02:13:02.244183 1 dispatcher.go:210] Failed calling webhook, failing open topology.webhook.gateway.envoyproxy.io: failed calling webhook "topology.webhook.gateway.envoyproxy.io": failed to call webhook: Post "https://envoy-gateway.envoy-gateway-system.svc:9443/inject-pod-topology?timeout=10s": service "envoy-gateway" not found E0411 02:13:02.244224 1 dispatcher.go:214] failed calling webhook "topology.webhook.gateway.envoyproxy.io": failed to call webhook: Post "https://envoy-gateway.envoy-gateway-system.svc:9443/inject-pod-topology?timeout=10s": service "envoy-gateway" not found I0411 02:13:05.848134 1 controller.go:624] quota admission added evaluator for: ciliumendpoints.cilium.io I0411 02:13:13.718230 1 alloc.go:330] "allocated clusterIPs" service="envoy-gateway-system/envoy-gateway" clusterIPs={"IPv4":"10.102.7.201"} W0411 02:13:15.053500 1 dispatcher.go:210] Failed calling webhook, failing open topology.webhook.gateway.envoyproxy.io: failed calling webhook "topology.webhook.gateway.envoyproxy.io": failed to call webhook: Post "https://envoy-gateway.envoy-gateway-system.svc:9443/inject-pod-topology?timeout=10s": dial tcp 10.102.7.201:9443: connect: connection refused E0411 02:13:15.053565 1 dispatcher.go:214] failed calling webhook "topology.webhook.gateway.envoyproxy.io": failed to call webhook: Post "https://envoy-gateway.envoy-gateway-system.svc:9443/inject-pod-topology?timeout=10s": dial tcp 10.102.7.201:9443: connect: connection refused I0411 02:13:26.226341 1 handler.go:275] Adding GroupVersion cert-manager.io v1 to ResourceManager I0411 02:13:26.237368 1 handler.go:275] Adding GroupVersion acme.cert-manager.io v1 to ResourceManager I0411 02:13:26.252367 1 handler.go:275] Adding GroupVersion cert-manager.io v1 to ResourceManager I0411 02:13:26.270689 1 alloc.go:330] "allocated clusterIPs" service="cert-manager/cert-manager" clusterIPs={"IPv4":"10.107.144.226"} I0411 02:13:26.284514 1 alloc.go:330] "allocated clusterIPs" service="cert-manager/cert-manager-webhook" clusterIPs={"IPv4":"10.102.195.178"} I0411 02:13:26.291517 1 handler.go:275] Adding GroupVersion acme.cert-manager.io v1 to ResourceManager I0411 02:13:26.343391 1 handler.go:275] Adding GroupVersion cert-manager.io v1 to ResourceManager I0411 02:13:26.388290 1 handler.go:275] Adding GroupVersion cert-manager.io v1 to ResourceManager W0411 02:13:37.341612 1 dispatcher.go:225] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/mutate?timeout=10s": dial tcp 10.102.195.178:443: connect: connection refused W0411 02:13:42.330119 1 dispatcher.go:225] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/mutate?timeout=10s": dial tcp 10.102.195.178:443: connect: connection refused E0411 02:13:42.722335 1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"context canceled"}: context canceled E0411 02:13:42.722450 1 writers.go:122] apiserver was unable to write a JSON response: http: Handler timeout E0411 02:13:42.723568 1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout E0411 02:13:42.723641 1 writers.go:135] apiserver was unable to write a fallback JSON response: http: Handler timeout I0411 02:13:42.724686 1 trace.go:236] Trace[1641439308]: "Get" accept:application/vnd.kubernetes.protobuf, */*,audit-id:93567b41-a926-4f2c-bcb6-98da26267dff,client:199.204.45.72,protocol:HTTP/2.0,resource:leases,scope:resource,url:/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/kube-scheduler,user-agent:kube-scheduler/v1.28.13 (linux/amd64) kubernetes/024ab2a/leader-election,verb:GET (11-Apr-2026 02:13:37.722) (total time: 5002ms): Trace[1641439308]: [5.002180257s] [5.002180257s] END E0411 02:13:42.724862 1 timeout.go:142] post-timeout activity - time-elapsed: 2.484753ms, GET "/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/kube-scheduler" result: E0411 02:13:42.726766 1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"context canceled"}: context canceled E0411 02:13:42.726831 1 writers.go:122] apiserver was unable to write a JSON response: http: Handler timeout E0411 02:13:42.727942 1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout E0411 02:13:42.727979 1 writers.go:135] apiserver was unable to write a fallback JSON response: http: Handler timeout I0411 02:13:42.729210 1 trace.go:236] Trace[1618560944]: "Get" accept:application/json, */*,audit-id:a45e6c45-e7db-4419-ae43-3e9624aceedd,client:199.204.45.72,protocol:HTTP/2.0,resource:leases,scope:resource,url:/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/cilium-operator-resource-lock,user-agent:cilium-operator-generic/v0.0.0 (linux/amd64) kubernetes/$Format/leader-election,verb:GET (11-Apr-2026 02:13:37.727) (total time: 5002ms): Trace[1618560944]: [5.002026345s] [5.002026345s] END E0411 02:13:42.729322 1 timeout.go:142] post-timeout activity - time-elapsed: 2.607186ms, GET "/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/cilium-operator-resource-lock" result: I0411 02:13:43.162482 1 trace.go:236] Trace[194553800]: "Update" accept:application/vnd.kubernetes.protobuf,application/json,audit-id:402a3258-a4cb-4053-81eb-875e782ac2ed,client:10.0.0.61,protocol:HTTP/2.0,resource:leases,scope:resource,url:/apis/coordination.k8s.io/v1/namespaces/envoy-gateway-system/leases/5b9825d2.gateway.envoyproxy.io,user-agent:envoy-gateway/v0.0.0 (linux/amd64) kubernetes/$Format/leader-election,verb:PUT (11-Apr-2026 02:13:38.163) (total time: 4999ms): Trace[194553800]: ["GuaranteedUpdate etcd3" audit-id:402a3258-a4cb-4053-81eb-875e782ac2ed,key:/leases/envoy-gateway-system/5b9825d2.gateway.envoyproxy.io,type:*coordination.Lease,resource:leases.coordination.k8s.io 4999ms (02:13:38.163) Trace[194553800]: ---"Txn call failed" err:context canceled 4998ms (02:13:43.162)] Trace[194553800]: [4.999359806s] [4.999359806s] END E0411 02:13:43.162635 1 wrap.go:54] timeout or abort while handling: method=PUT URI="/apis/coordination.k8s.io/v1/namespaces/envoy-gateway-system/leases/5b9825d2.gateway.envoyproxy.io?timeout=5s" audit-ID="402a3258-a4cb-4053-81eb-875e782ac2ed" E0411 02:13:43.162646 1 finisher.go:175] FinishRequest: post-timeout activity - time-elapsed: 10.42µs, panicked: false, err: context canceled, panic-reason: E0411 02:13:43.162740 1 timeout.go:142] post-timeout activity - time-elapsed: 16.161µs, PUT "/apis/coordination.k8s.io/v1/namespaces/envoy-gateway-system/leases/5b9825d2.gateway.envoyproxy.io" result: E0411 02:13:43.423032 1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"context canceled"}: context canceled E0411 02:13:43.423199 1 writers.go:122] apiserver was unable to write a JSON response: http: Handler timeout E0411 02:13:43.425124 1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout E0411 02:13:43.425170 1 writers.go:135] apiserver was unable to write a fallback JSON response: http: Handler timeout I0411 02:13:43.426342 1 trace.go:236] Trace[530487784]: "Get" accept:application/vnd.kubernetes.protobuf, */*,audit-id:713b87ef-9f12-4c86-a24f-5988812baf48,client:199.204.45.72,protocol:HTTP/2.0,resource:leases,scope:resource,url:/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/kube-controller-manager,user-agent:kube-controller-manager/v1.28.13 (linux/amd64) kubernetes/024ab2a/leader-election,verb:GET (11-Apr-2026 02:13:38.423) (total time: 5003ms): Trace[530487784]: [5.003135868s] [5.003135868s] END E0411 02:13:43.426534 1 timeout.go:142] post-timeout activity - time-elapsed: 3.512605ms, GET "/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/kube-controller-manager" result: I0411 02:13:43.473191 1 trace.go:236] Trace[1171893031]: "Update" accept:application/vnd.kubernetes.protobuf, */*,audit-id:e56e995a-9738-405c-b753-2b06a801d63b,client:199.204.45.72,protocol:HTTP/2.0,resource:endpointslices,scope:resource,url:/apis/discovery.k8s.io/v1/namespaces/cert-manager/endpointslices/cert-manager-webhook-wmgvv,user-agent:kube-controller-manager/v1.28.13 (linux/amd64) kubernetes/024ab2a/system:serviceaccount:kube-system:endpointslice-controller,verb:PUT (11-Apr-2026 02:13:37.135) (total time: 6338ms): Trace[1171893031]: ["GuaranteedUpdate etcd3" audit-id:e56e995a-9738-405c-b753-2b06a801d63b,key:/endpointslices/cert-manager/cert-manager-webhook-wmgvv,type:*discovery.EndpointSlice,resource:endpointslices.discovery.k8s.io 6337ms (02:13:37.135) Trace[1171893031]: ---"Txn call completed" 6336ms (02:13:43.472)] Trace[1171893031]: [6.338115825s] [6.338115825s] END I0411 02:13:43.497426 1 trace.go:236] Trace[761649677]: "Update" accept:application/vnd.kubernetes.protobuf, */*,audit-id:2703bee3-12a1-41ee-9530-eeb9f93bc9ff,client:199.204.45.72,protocol:HTTP/2.0,resource:replicasets,scope:resource,url:/apis/apps/v1/namespaces/cert-manager/replicasets/cert-manager-webhook-548949fc64/status,user-agent:kube-controller-manager/v1.28.13 (linux/amd64) kubernetes/024ab2a/system:serviceaccount:kube-system:replicaset-controller,verb:PUT (11-Apr-2026 02:13:37.134) (total time: 6362ms): Trace[761649677]: ["GuaranteedUpdate etcd3" audit-id:2703bee3-12a1-41ee-9530-eeb9f93bc9ff,key:/replicasets/cert-manager/cert-manager-webhook-548949fc64,type:*apps.ReplicaSet,resource:replicasets.apps 6362ms (02:13:37.134) Trace[761649677]: ---"Txn call completed" 6358ms (02:13:43.496)] Trace[761649677]: [6.362677651s] [6.362677651s] END I0411 02:13:43.502053 1 trace.go:236] Trace[157551455]: "Update" accept:application/vnd.kubernetes.protobuf, */*,audit-id:9616bcb3-d6c5-4a30-8925-7f03c16ab312,client:::1,protocol:HTTP/2.0,resource:leases,scope:resource,url:/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/apiserver-f3tcohoifagyom4bbi4wgeu7te,user-agent:kube-apiserver/v1.28.13 (linux/amd64) kubernetes/024ab2a,verb:PUT (11-Apr-2026 02:13:42.813) (total time: 688ms): Trace[157551455]: ["GuaranteedUpdate etcd3" audit-id:9616bcb3-d6c5-4a30-8925-7f03c16ab312,key:/leases/kube-system/apiserver-f3tcohoifagyom4bbi4wgeu7te,type:*coordination.Lease,resource:leases.coordination.k8s.io 688ms (02:13:42.813) Trace[157551455]: ---"Txn call completed" 687ms (02:13:43.501)] Trace[157551455]: [688.626427ms] [688.626427ms] END I0411 02:13:43.502239 1 trace.go:236] Trace[396792467]: "Create" accept:application/vnd.kubernetes.protobuf,application/json,audit-id:44736840-f215-4e68-bdb7-9eb684098e00,client:199.204.45.72,protocol:HTTP/2.0,resource:events,scope:resource,url:/api/v1/namespaces/kube-system/events,user-agent:kubelet/v1.28.13 (linux/amd64) kubernetes/024ab2a,verb:POST (11-Apr-2026 02:13:41.196) (total time: 2305ms): Trace[396792467]: ["Create etcd3" audit-id:44736840-f215-4e68-bdb7-9eb684098e00,key:/events/kube-system/kube-apiserver-instance.18a52ba645514c9e,type:*core.Event,resource:events 2305ms (02:13:41.197) Trace[396792467]: ---"Txn call succeeded" 2304ms (02:13:43.501)] Trace[396792467]: [2.305801399s] [2.305801399s] END I0411 02:13:43.502757 1 trace.go:236] Trace[1239647708]: "Get" accept:application/json, */*,audit-id:8b0fcc26-a0ec-4e48-aeb9-5aac22871964,client:127.0.0.1,protocol:HTTP/2.0,resource:leases,scope:resource,url:/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/plndr-cp-lock,user-agent:kube-vip/v0.0.0 (linux/amd64) kubernetes/$Format,verb:GET (11-Apr-2026 02:13:38.233) (total time: 5268ms): Trace[1239647708]: ---"About to write a response" 5268ms (02:13:43.502) Trace[1239647708]: [5.268766544s] [5.268766544s] END I0411 02:13:43.503622 1 trace.go:236] Trace[1976225603]: "Get" accept:application/json, */*,audit-id:780b8b4b-c860-481c-827e-fc124155da41,client:127.0.0.1,protocol:HTTP/2.0,resource:leases,scope:resource,url:/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/plndr-svcs-lock,user-agent:kube-vip/v0.0.0 (linux/amd64) kubernetes/$Format,verb:GET (11-Apr-2026 02:13:37.194) (total time: 6308ms): Trace[1976225603]: ---"About to write a response" 6308ms (02:13:43.503) Trace[1976225603]: [6.308970946s] [6.308970946s] END I0411 02:13:43.504708 1 trace.go:236] Trace[195995485]: "Get" accept:application/vnd.kubernetes.protobuf, */*,audit-id:7ae4ef42-35fe-41f4-8f82-841476f8854c,client:199.204.45.72,protocol:HTTP/2.0,resource:serviceaccounts,scope:resource,url:/api/v1/namespaces/kube-system/serviceaccounts/ttl-after-finished-controller,user-agent:kube-controller-manager/v1.28.13 (linux/amd64) kubernetes/024ab2a/kube-controller-manager,verb:GET (11-Apr-2026 02:13:43.002) (total time: 502ms): Trace[195995485]: ---"About to write a response" 502ms (02:13:43.504) Trace[195995485]: [502.50282ms] [502.50282ms] END I0411 02:13:43.517206 1 trace.go:236] Trace[791129665]: "GuaranteedUpdate etcd3" audit-id:,key:/masterleases/199.204.45.72,type:*v1.Endpoints,resource:apiServerIPInfo (11-Apr-2026 02:13:42.568) (total time: 948ms): Trace[791129665]: ---"initial value restored" 934ms (02:13:43.502) Trace[791129665]: [948.498924ms] [948.498924ms] END I0411 02:13:47.344330 1 controller.go:624] quota admission added evaluator for: certificates.cert-manager.io I0411 02:13:52.667335 1 controller.go:624] quota admission added evaluator for: certificaterequests.cert-manager.io I0411 02:14:07.014062 1 alloc.go:330] "allocated clusterIPs" service="ingress-nginx/ingress-nginx-defaultbackend" clusterIPs={"IPv4":"10.107.89.136"} I0411 02:14:07.037680 1 alloc.go:330] "allocated clusterIPs" service="ingress-nginx/ingress-nginx-controller-metrics" clusterIPs={"IPv4":"10.107.209.222"} I0411 02:14:07.061250 1 alloc.go:330] "allocated clusterIPs" service="ingress-nginx/ingress-nginx-controller-admission" clusterIPs={"IPv4":"10.108.26.58"} I0411 02:14:07.095594 1 alloc.go:330] "allocated clusterIPs" service="ingress-nginx/ingress-nginx-controller" clusterIPs={"IPv4":"10.111.80.30"} I0411 02:14:09.224700 1 trace.go:236] Trace[1385379997]: "Create" accept:application/json, */*,audit-id:f5c442cd-a005-4639-b5a2-3e31fb2e3aee,client:199.204.45.72,protocol:HTTP/2.0,resource:ciliumidentities,scope:resource,url:/apis/cilium.io/v2/ciliumidentities,user-agent:cilium-agent/1.14.8 cf6e022e 2024-03-13T12:23:35-04:00 go version go1.21.8 linux/amd64,verb:POST (11-Apr-2026 02:14:08.227) (total time: 996ms): Trace[1385379997]: ["Create etcd3" audit-id:f5c442cd-a005-4639-b5a2-3e31fb2e3aee,key:/cilium.io/ciliumidentities/961,type:*unstructured.Unstructured,resource:ciliumidentities.cilium.io 995ms (02:14:08.228) Trace[1385379997]: ---"Txn call succeeded" 994ms (02:14:09.223)] Trace[1385379997]: [996.939612ms] [996.939612ms] END I0411 02:14:20.022449 1 handler.go:275] Adding GroupVersion rabbitmq.com v1beta1 to ResourceManager I0411 02:14:20.039828 1 handler.go:275] Adding GroupVersion rabbitmq.com v1beta1 to ResourceManager I0411 02:14:20.068598 1 handler.go:275] Adding GroupVersion rabbitmq.com v1beta1 to ResourceManager I0411 02:14:20.095484 1 handler.go:275] Adding GroupVersion rabbitmq.com v1beta1 to ResourceManager I0411 02:14:20.118095 1 handler.go:275] Adding GroupVersion rabbitmq.com v1beta1 to ResourceManager I0411 02:14:20.141369 1 handler.go:275] Adding GroupVersion rabbitmq.com v1beta1 to ResourceManager I0411 02:14:20.153015 1 handler.go:275] Adding GroupVersion rabbitmq.com v1beta1 to ResourceManager I0411 02:14:20.175145 1 handler.go:275] Adding GroupVersion rabbitmq.com v1beta1 to ResourceManager I0411 02:14:20.194433 1 handler.go:275] Adding GroupVersion rabbitmq.com v1beta1 to ResourceManager I0411 02:14:20.210988 1 handler.go:275] Adding GroupVersion rabbitmq.com v1alpha1 to ResourceManager I0411 02:14:20.230786 1 handler.go:275] Adding GroupVersion rabbitmq.com v1beta1 to ResourceManager I0411 02:14:20.250332 1 handler.go:275] Adding GroupVersion rabbitmq.com v1beta1 to ResourceManager I0411 02:14:20.269140 1 handler.go:275] Adding GroupVersion rabbitmq.com v1beta1 to ResourceManager I0411 02:14:21.550440 1 handler.go:275] Adding GroupVersion rabbitmq.com v1beta1 to ResourceManager I0411 02:14:24.772496 1 controller.go:624] quota admission added evaluator for: networkpolicies.networking.k8s.io I0411 02:14:24.830719 1 alloc.go:330] "allocated clusterIPs" service="openstack/rabbitmq-messaging-topology-operator-webhook" clusterIPs={"IPv4":"10.109.12.20"} I0411 02:14:24.864375 1 controller.go:624] quota admission added evaluator for: issuers.cert-manager.io I0411 02:14:27.109101 1 handler.go:275] Adding GroupVersion pxc.percona.com v1 to ResourceManager I0411 02:14:27.152387 1 handler.go:275] Adding GroupVersion pxc.percona.com v1 to ResourceManager I0411 02:14:27.795034 1 handler.go:275] Adding GroupVersion pxc.percona.com v1-10-0 to ResourceManager I0411 02:14:27.795134 1 handler.go:275] Adding GroupVersion pxc.percona.com v1-11-0 to ResourceManager I0411 02:14:27.795240 1 handler.go:275] Adding GroupVersion pxc.percona.com v1 to ResourceManager I0411 02:14:34.122080 1 controller.go:624] quota admission added evaluator for: perconaxtradbclusters.pxc.percona.com I0411 02:14:37.884311 1 alloc.go:330] "allocated clusterIPs" service="openstack/percona-xtradb-cluster-operator" clusterIPs={"IPv4":"10.107.105.60"} I0411 02:14:38.156810 1 controller.go:624] quota admission added evaluator for: statefulsets.apps I0411 02:14:38.303970 1 alloc.go:330] "allocated clusterIPs" service="openstack/percona-xtradb-haproxy" clusterIPs={"IPv4":"10.103.87.58"} I0411 02:14:38.323496 1 alloc.go:330] "allocated clusterIPs" service="openstack/percona-xtradb-haproxy-replicas" clusterIPs={"IPv4":"10.100.226.189"} I0411 02:14:38.620751 1 controller.go:624] quota admission added evaluator for: poddisruptionbudgets.policy I0411 02:16:13.535791 1 alloc.go:330] "allocated clusterIPs" service="openstack/percona-xtradb-haproxy-metrics" clusterIPs={"IPv4":"10.107.211.56"} I0411 02:16:16.230386 1 alloc.go:330] "allocated clusterIPs" service="openstack/valkey" clusterIPs={"IPv4":"10.97.111.179"} I0411 02:16:16.249012 1 alloc.go:330] "allocated clusterIPs" service="openstack/valkey-metrics" clusterIPs={"IPv4":"10.101.63.146"} I0411 02:16:22.733749 1 alloc.go:330] "allocated clusterIPs" service="auth-system/keycloak" clusterIPs={"IPv4":"10.101.218.114"} I0411 02:16:22.743134 1 alloc.go:330] "allocated clusterIPs" service="auth-system/keycloak-metrics" clusterIPs={"IPv4":"10.101.204.37"} I0411 02:18:52.563542 1 controller.go:624] quota admission added evaluator for: ingresses.networking.k8s.io I0411 02:18:56.716201 1 handler.go:275] Adding GroupVersion nfd.k8s-sigs.io v1alpha1 to ResourceManager I0411 02:18:56.740736 1 handler.go:275] Adding GroupVersion nfd.k8s-sigs.io v1alpha1 to ResourceManager I0411 02:19:00.794568 1 handler.go:275] Adding GroupVersion secretgen.carvel.dev v1alpha1 to ResourceManager I0411 02:19:00.803937 1 handler.go:275] Adding GroupVersion secretgen.carvel.dev v1alpha1 to ResourceManager I0411 02:19:00.818947 1 handler.go:275] Adding GroupVersion secretgen.carvel.dev v1alpha1 to ResourceManager I0411 02:19:00.832599 1 handler.go:275] Adding GroupVersion secretgen.k14s.io v1alpha1 to ResourceManager I0411 02:19:00.841975 1 handler.go:275] Adding GroupVersion secretgen.k14s.io v1alpha1 to ResourceManager I0411 02:19:00.855379 1 handler.go:275] Adding GroupVersion secretgen.k14s.io v1alpha1 to ResourceManager I0411 02:19:00.866583 1 handler.go:275] Adding GroupVersion secretgen.k14s.io v1alpha1 to ResourceManager I0411 02:19:02.630346 1 controller.go:624] quota admission added evaluator for: nodefeatures.nfd.k8s-sigs.io I0411 02:19:13.201953 1 controller.go:624] quota admission added evaluator for: passwords.secretgen.k14s.io I0411 02:19:53.046113 1 controller.go:624] quota admission added evaluator for: secrettemplates.secretgen.carvel.dev I0411 02:20:08.008773 1 handler.go:275] Adding GroupVersion monitoring.coreos.com v1alpha1 to ResourceManager I0411 02:20:08.189750 1 handler.go:275] Adding GroupVersion monitoring.coreos.com v1 to ResourceManager I0411 02:20:08.217993 1 handler.go:275] Adding GroupVersion monitoring.coreos.com v1 to ResourceManager I0411 02:20:08.219458 1 handler.go:275] Adding GroupVersion monitoring.coreos.com v1 to ResourceManager I0411 02:20:08.527371 1 handler.go:275] Adding GroupVersion monitoring.coreos.com v1alpha1 to ResourceManager I0411 02:20:08.817200 1 handler.go:275] Adding GroupVersion monitoring.coreos.com v1 to ResourceManager I0411 02:20:08.825005 1 handler.go:275] Adding GroupVersion monitoring.coreos.com v1 to ResourceManager I0411 02:20:08.881922 1 handler.go:275] Adding GroupVersion monitoring.coreos.com v1alpha1 to ResourceManager I0411 02:20:08.896826 1 handler.go:275] Adding GroupVersion monitoring.coreos.com v1 to ResourceManager I0411 02:20:09.061991 1 handler.go:275] Adding GroupVersion monitoring.coreos.com v1 to ResourceManager I0411 02:20:50.049032 1 alloc.go:330] "allocated clusterIPs" service="monitoring/kube-prometheus-stack-grafana" clusterIPs={"IPv4":"10.97.44.79"} I0411 02:20:50.059118 1 alloc.go:330] "allocated clusterIPs" service="monitoring/kube-prometheus-stack-kube-state-metrics" clusterIPs={"IPv4":"10.101.67.138"} I0411 02:20:50.068302 1 alloc.go:330] "allocated clusterIPs" service="monitoring/kube-prometheus-stack-prometheus-node-exporter" clusterIPs={"IPv4":"10.97.106.203"} I0411 02:20:50.077807 1 alloc.go:330] "allocated clusterIPs" service="monitoring/kube-prometheus-stack-operator" clusterIPs={"IPv4":"10.102.77.160"} I0411 02:20:50.087459 1 alloc.go:330] "allocated clusterIPs" service="monitoring/kube-prometheus-stack-alertmanager" clusterIPs={"IPv4":"10.110.207.29"} I0411 02:20:50.097707 1 alloc.go:330] "allocated clusterIPs" service="monitoring/kube-prometheus-stack-prometheus" clusterIPs={"IPv4":"10.107.215.46"} I0411 02:20:50.144391 1 controller.go:624] quota admission added evaluator for: alertmanagers.monitoring.coreos.com I0411 02:20:50.157262 1 controller.go:624] quota admission added evaluator for: prometheusrules.monitoring.coreos.com I0411 02:20:50.198458 1 controller.go:624] quota admission added evaluator for: podmonitors.monitoring.coreos.com I0411 02:20:50.214424 1 controller.go:624] quota admission added evaluator for: servicemonitors.monitoring.coreos.com I0411 02:20:50.231078 1 controller.go:624] quota admission added evaluator for: prometheuses.monitoring.coreos.com W0411 02:20:50.240362 1 dispatcher.go:210] Failed calling webhook, failing open prometheusrulemutate.monitoring.coreos.com: failed calling webhook "prometheusrulemutate.monitoring.coreos.com": failed to call webhook: Post "https://kube-prometheus-stack-operator.monitoring.svc:443/admission-prometheusrules/mutate?timeout=10s": dial tcp 10.102.77.160:443: connect: connection refused E0411 02:20:50.240400 1 dispatcher.go:214] failed calling webhook "prometheusrulemutate.monitoring.coreos.com": failed to call webhook: Post "https://kube-prometheus-stack-operator.monitoring.svc:443/admission-prometheusrules/mutate?timeout=10s": dial tcp 10.102.77.160:443: connect: connection refused W0411 02:20:50.240402 1 dispatcher.go:210] Failed calling webhook, failing open prometheusrulemutate.monitoring.coreos.com: failed calling webhook "prometheusrulemutate.monitoring.coreos.com": failed to call webhook: Post "https://kube-prometheus-stack-operator.monitoring.svc:443/admission-prometheusrules/mutate?timeout=10s": dial tcp 10.102.77.160:443: connect: connection refused E0411 02:20:50.240418 1 dispatcher.go:214] failed calling webhook "prometheusrulemutate.monitoring.coreos.com": failed to call webhook: Post "https://kube-prometheus-stack-operator.monitoring.svc:443/admission-prometheusrules/mutate?timeout=10s": dial tcp 10.102.77.160:443: connect: connection refused W0411 02:20:50.240594 1 dispatcher.go:210] Failed calling webhook, failing open prometheusrulemutate.monitoring.coreos.com: failed calling webhook "prometheusrulemutate.monitoring.coreos.com": failed to call webhook: Post "https://kube-prometheus-stack-operator.monitoring.svc:443/admission-prometheusrules/mutate?timeout=10s": dial tcp 10.102.77.160:443: connect: connection refused E0411 02:20:50.240627 1 dispatcher.go:214] failed calling webhook "prometheusrulemutate.monitoring.coreos.com": failed to call webhook: Post "https://kube-prometheus-stack-operator.monitoring.svc:443/admission-prometheusrules/mutate?timeout=10s": dial tcp 10.102.77.160:443: connect: connection refused W0411 02:20:50.240790 1 dispatcher.go:210] Failed calling webhook, failing open prometheusrulemutate.monitoring.coreos.com: failed calling webhook "prometheusrulemutate.monitoring.coreos.com": failed to call webhook: Post "https://kube-prometheus-stack-operator.monitoring.svc:443/admission-prometheusrules/mutate?timeout=10s": dial tcp 10.102.77.160:443: connect: connection refused E0411 02:20:50.240820 1 dispatcher.go:214] failed calling webhook "prometheusrulemutate.monitoring.coreos.com": failed to call webhook: Post "https://kube-prometheus-stack-operator.monitoring.svc:443/admission-prometheusrules/mutate?timeout=10s": dial tcp 10.102.77.160:443: connect: connection refused W0411 02:20:50.241376 1 dispatcher.go:210] Failed calling webhook, failing open prometheusrulemutate.monitoring.coreos.com: failed calling webhook "prometheusrulemutate.monitoring.coreos.com": failed to call webhook: Post "https://kube-prometheus-stack-operator.monitoring.svc:443/admission-prometheusrules/mutate?timeout=10s": dial tcp 10.102.77.160:443: connect: connection refused E0411 02:20:50.241446 1 dispatcher.go:214] failed calling webhook "prometheusrulemutate.monitoring.coreos.com": failed to call webhook: Post "https://kube-prometheus-stack-operator.monitoring.svc:443/admission-prometheusrules/mutate?timeout=10s": dial tcp 10.102.77.160:443: connect: connection refused W0411 02:20:50.241493 1 dispatcher.go:210] Failed calling webhook, failing open prometheusrulemutate.monitoring.coreos.com: failed calling webhook "prometheusrulemutate.monitoring.coreos.com": failed to call webhook: Post "https://kube-prometheus-stack-operator.monitoring.svc:443/admission-prometheusrules/mutate?timeout=10s": dial tcp 10.102.77.160:443: connect: connection refused E0411 02:20:50.241517 1 dispatcher.go:214] failed calling webhook "prometheusrulemutate.monitoring.coreos.com": failed to call webhook: Post "https://kube-prometheus-stack-operator.monitoring.svc:443/admission-prometheusrules/mutate?timeout=10s": dial tcp 10.102.77.160:443: connect: connection refused W0411 02:20:50.241696 1 dispatcher.go:210] Failed calling webhook, failing open prometheusrulemutate.monitoring.coreos.com: failed calling webhook "prometheusrulemutate.monitoring.coreos.com": failed to call webhook: Post "https://kube-prometheus-stack-operator.monitoring.svc:443/admission-prometheusrules/mutate?timeout=10s": dial tcp 10.102.77.160:443: connect: connection refused E0411 02:20:50.241715 1 dispatcher.go:214] failed calling webhook "prometheusrulemutate.monitoring.coreos.com": failed to call webhook: Post "https://kube-prometheus-stack-operator.monitoring.svc:443/admission-prometheusrules/mutate?timeout=10s": dial tcp 10.102.77.160:443: connect: connection refused W0411 02:20:50.243140 1 dispatcher.go:210] Failed calling webhook, failing open prometheusrulemutate.monitoring.coreos.com: failed calling webhook "prometheusrulemutate.monitoring.coreos.com": failed to call webhook: Post "https://kube-prometheus-stack-operator.monitoring.svc:443/admission-prometheusrules/mutate?timeout=10s": dial tcp 10.102.77.160:443: connect: connection refused E0411 02:20:50.243163 1 dispatcher.go:214] failed calling webhook "prometheusrulemutate.monitoring.coreos.com": failed to call webhook: Post "https://kube-prometheus-stack-operator.monitoring.svc:443/admission-prometheusrules/mutate?timeout=10s": dial tcp 10.102.77.160:443: connect: connection refused W0411 02:20:50.243203 1 dispatcher.go:210] Failed calling webhook, failing open prometheusrulemutate.monitoring.coreos.com: failed calling webhook "prometheusrulemutate.monitoring.coreos.com": failed to call webhook: Post "https://kube-prometheus-stack-operator.monitoring.svc:443/admission-prometheusrules/mutate?timeout=10s": dial tcp 10.102.77.160:443: connect: connection refused E0411 02:20:50.243235 1 dispatcher.go:214] failed calling webhook "prometheusrulemutate.monitoring.coreos.com": failed to call webhook: Post "https://kube-prometheus-stack-operator.monitoring.svc:443/admission-prometheusrules/mutate?timeout=10s": dial tcp 10.102.77.160:443: connect: connection refused W0411 02:20:50.243254 1 dispatcher.go:210] Failed calling webhook, failing open prometheusrulemutate.monitoring.coreos.com: failed calling webhook "prometheusrulemutate.monitoring.coreos.com": failed to call webhook: Post "https://kube-prometheus-stack-operator.monitoring.svc:443/admission-prometheusrules/mutate?timeout=10s": dial tcp 10.102.77.160:443: connect: connection refused W0411 02:20:50.243361 1 dispatcher.go:210] Failed calling webhook, failing open prometheusrulemutate.monitoring.coreos.com: failed calling webhook "prometheusrulemutate.monitoring.coreos.com": failed to call webhook: Post "https://kube-prometheus-stack-operator.monitoring.svc:443/admission-prometheusrules/mutate?timeout=10s": dial tcp 10.102.77.160:443: connect: connection refused E0411 02:20:50.243388 1 dispatcher.go:214] failed calling webhook "prometheusrulemutate.monitoring.coreos.com": failed to call webhook: Post "https://kube-prometheus-stack-operator.monitoring.svc:443/admission-prometheusrules/mutate?timeout=10s": dial tcp 10.102.77.160:443: connect: connection refused W0411 02:20:50.243382 1 dispatcher.go:210] Failed calling webhook, failing open prometheusrulemutate.monitoring.coreos.com: failed calling webhook "prometheusrulemutate.monitoring.coreos.com": failed to call webhook: Post "https://kube-prometheus-stack-operator.monitoring.svc:443/admission-prometheusrules/mutate?timeout=10s": dial tcp 10.102.77.160:443: connect: connection refused E0411 02:20:50.243422 1 dispatcher.go:214] failed calling webhook "prometheusrulemutate.monitoring.coreos.com": failed to call webhook: Post "https://kube-prometheus-stack-operator.monitoring.svc:443/admission-prometheusrules/mutate?timeout=10s": dial tcp 10.102.77.160:443: connect: connection refused E0411 02:20:50.243459 1 dispatcher.go:214] failed calling webhook "prometheusrulemutate.monitoring.coreos.com": failed to call webhook: Post "https://kube-prometheus-stack-operator.monitoring.svc:443/admission-prometheusrules/mutate?timeout=10s": dial tcp 10.102.77.160:443: connect: connection refused W0411 02:20:50.243539 1 dispatcher.go:210] Failed calling webhook, failing open prometheusrulemutate.monitoring.coreos.com: failed calling webhook "prometheusrulemutate.monitoring.coreos.com": failed to call webhook: Post "https://kube-prometheus-stack-operator.monitoring.svc:443/admission-prometheusrules/mutate?timeout=10s": dial tcp 10.102.77.160:443: connect: connection refused W0411 02:20:50.243555 1 dispatcher.go:210] Failed calling webhook, failing open prometheusrulemutate.monitoring.coreos.com: failed calling webhook "prometheusrulemutate.monitoring.coreos.com": failed to call webhook: Post "https://kube-prometheus-stack-operator.monitoring.svc:443/admission-prometheusrules/mutate?timeout=10s": dial tcp 10.102.77.160:443: connect: connection refused E0411 02:20:50.243627 1 dispatcher.go:214] failed calling webhook "prometheusrulemutate.monitoring.coreos.com": failed to call webhook: Post "https://kube-prometheus-stack-operator.monitoring.svc:443/admission-prometheusrules/mutate?timeout=10s": dial tcp 10.102.77.160:443: connect: connection refused E0411 02:20:50.243629 1 dispatcher.go:214] failed calling webhook "prometheusrulemutate.monitoring.coreos.com": failed to call webhook: Post "https://kube-prometheus-stack-operator.monitoring.svc:443/admission-prometheusrules/mutate?timeout=10s": dial tcp 10.102.77.160:443: connect: connection refused W0411 02:20:50.243612 1 dispatcher.go:210] Failed calling webhook, failing open prometheusrulemutate.monitoring.coreos.com: failed calling webhook "prometheusrulemutate.monitoring.coreos.com": failed to call webhook: Post "https://kube-prometheus-stack-operator.monitoring.svc:443/admission-prometheusrules/mutate?timeout=10s": dial tcp 10.102.77.160:443: connect: connection refused E0411 02:20:50.243716 1 dispatcher.go:214] failed calling webhook "prometheusrulemutate.monitoring.coreos.com": failed to call webhook: Post "https://kube-prometheus-stack-operator.monitoring.svc:443/admission-prometheusrules/mutate?timeout=10s": dial tcp 10.102.77.160:443: connect: connection refused W0411 02:20:50.246238 1 dispatcher.go:210] Failed calling webhook, failing open prometheusrulemutate.monitoring.coreos.com: failed calling webhook "prometheusrulemutate.monitoring.coreos.com": failed to call webhook: Post "https://kube-prometheus-stack-operator.monitoring.svc:443/admission-prometheusrules/mutate?timeout=10s": dial tcp 10.102.77.160:443: connect: connection refused E0411 02:20:50.246278 1 dispatcher.go:214] failed calling webhook "prometheusrulemutate.monitoring.coreos.com": failed to call webhook: Post "https://kube-prometheus-stack-operator.monitoring.svc:443/admission-prometheusrules/mutate?timeout=10s": dial tcp 10.102.77.160:443: connect: connection refused I0411 02:21:00.249343 1 alloc.go:330] "allocated clusterIPs" service="monitoring/loki" clusterIPs={"IPv4":"10.99.108.36"} I0411 02:21:00.262071 1 alloc.go:330] "allocated clusterIPs" service="monitoring/loki-gateway" clusterIPs={"IPv4":"10.96.49.5"} I0411 02:21:04.827598 1 alloc.go:330] "allocated clusterIPs" service="monitoring/goldpinger" clusterIPs={"IPv4":"10.100.172.91"} I0411 02:21:07.732164 1 alloc.go:330] "allocated clusterIPs" service="monitoring/prometheus-pushgateway" clusterIPs={"IPv4":"10.105.57.161"} I0411 02:21:12.183663 1 alloc.go:330] "allocated clusterIPs" service="openstack/memcached" clusterIPs={"IPv4":"10.103.57.1"} I0411 02:21:13.076047 1 alloc.go:330] "allocated clusterIPs" service="openstack/memcached-metrics" clusterIPs={"IPv4":"10.108.246.138"} I0411 02:21:20.049243 1 controller.go:624] quota admission added evaluator for: rabbitmqclusters.rabbitmq.com I0411 02:21:20.216945 1 alloc.go:330] "allocated clusterIPs" service="openstack/rabbitmq-keystone" clusterIPs={"IPv4":"10.102.173.250"} I0411 02:22:00.916040 1 alloc.go:330] "allocated clusterIPs" service="openstack/keystone-api" clusterIPs={"IPv4":"10.104.54.177"} I0411 02:22:00.939624 1 controller.go:624] quota admission added evaluator for: cronjobs.batch I0411 02:24:55.890949 1 alloc.go:330] "allocated clusterIPs" service="openstack/rabbitmq-barbican" clusterIPs={"IPv4":"10.97.76.65"} I0411 02:25:29.351172 1 alloc.go:330] "allocated clusterIPs" service="openstack/barbican-api" clusterIPs={"IPv4":"10.108.195.83"} I0411 02:27:19.415974 1 handler.go:275] Adding GroupVersion ceph.rook.io v1 to ResourceManager I0411 02:27:19.430531 1 handler.go:275] Adding GroupVersion ceph.rook.io v1 to ResourceManager I0411 02:27:19.436223 1 handler.go:275] Adding GroupVersion objectbucket.io v1alpha1 to ResourceManager I0411 02:27:19.456858 1 handler.go:275] Adding GroupVersion ceph.rook.io v1 to ResourceManager I0411 02:27:19.464072 1 handler.go:275] Adding GroupVersion ceph.rook.io v1 to ResourceManager I0411 02:27:19.486050 1 handler.go:275] Adding GroupVersion objectbucket.io v1alpha1 to ResourceManager I0411 02:27:19.543161 1 handler.go:275] Adding GroupVersion ceph.rook.io v1 to ResourceManager I0411 02:27:19.564317 1 handler.go:275] Adding GroupVersion ceph.rook.io v1 to ResourceManager I0411 02:27:19.572640 1 handler.go:275] Adding GroupVersion ceph.rook.io v1 to ResourceManager I0411 02:27:19.598765 1 handler.go:275] Adding GroupVersion ceph.rook.io v1 to ResourceManager I0411 02:27:19.608076 1 handler.go:275] Adding GroupVersion ceph.rook.io v1 to ResourceManager I0411 02:27:19.628665 1 handler.go:275] Adding GroupVersion ceph.rook.io v1 to ResourceManager I0411 02:27:19.646482 1 handler.go:275] Adding GroupVersion ceph.rook.io v1 to ResourceManager I0411 02:27:19.673651 1 handler.go:275] Adding GroupVersion ceph.rook.io v1 to ResourceManager I0411 02:27:19.685520 1 handler.go:275] Adding GroupVersion ceph.rook.io v1 to ResourceManager I0411 02:27:19.725176 1 handler.go:275] Adding GroupVersion ceph.rook.io v1 to ResourceManager I0411 02:27:19.759749 1 handler.go:275] Adding GroupVersion ceph.rook.io v1 to ResourceManager I0411 02:27:19.826234 1 handler.go:275] Adding GroupVersion ceph.rook.io v1 to ResourceManager I0411 02:27:19.893658 1 handler.go:275] Adding GroupVersion ceph.rook.io v1 to ResourceManager I0411 02:28:00.584117 1 controller.go:624] quota admission added evaluator for: cephclusters.ceph.rook.io I0411 02:28:00.594379 1 controller.go:624] quota admission added evaluator for: cephobjectstores.ceph.rook.io I0411 02:28:33.807541 1 alloc.go:330] "allocated clusterIPs" service="openstack/rook-ceph-rgw-ceph" clusterIPs={"IPv4":"10.101.242.42"} I0411 02:28:34.443552 1 alloc.go:330] "allocated clusterIPs" service="openstack/rabbitmq-glance" clusterIPs={"IPv4":"10.98.116.24"} I0411 02:29:08.356687 1 alloc.go:330] "allocated clusterIPs" service="openstack/glance-api" clusterIPs={"IPv4":"10.97.33.214"} I0411 02:31:47.310506 1 alloc.go:330] "allocated clusterIPs" service="openstack/staffeln-api" clusterIPs={"IPv4":"10.96.185.137"}