{"time": "2026-07-23T23:59:21.507657+00:00", "taskName": null, "msg": "Starting collector", "level": "INFO"} {"time": "2026-07-23T23:59:21.507885+00:00", "taskName": null, "msg": "No folder annotation was provided, defaulting to k8s-sidecar-target-directory", "level": "WARNING"} {"time": "2026-07-23T23:59:21.508026+00:00", "taskName": null, "msg": "Loading incluster config ...", "level": "INFO"} {"time": "2026-07-23T23:59:21.508642+00:00", "taskName": null, "msg": "Config for cluster api at 'https://10.96.0.1:443' loaded...", "level": "INFO"} {"time": "2026-07-23T23:59:21.508738+00:00", "taskName": null, "msg": "Unique filenames will not be enforced.", "level": "INFO"} {"time": "2026-07-23T23:59:21.508811+00:00", "taskName": null, "msg": "5xx response content will not be enabled.", "level": "INFO"} {"time": "2026-07-23T23:59:21.535842+00:00", "taskName": null, "msg": "Writing /etc/grafana/provisioning/datasources/datasource.yaml (ascii)", "level": "INFO"} {"time": "2026-07-23T23:59:21.538949+00:00", "taskName": null, "msg": "Retrying (Retry(total=4, connect=9, read=5, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused')': /api/admin/provisioning/datasources/reload", "level": "WARNING"} {"time": "2026-07-23T23:59:23.740571+00:00", "taskName": null, "msg": "Retrying (Retry(total=3, connect=8, read=5, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused')': /api/admin/provisioning/datasources/reload", "level": "WARNING"} {"time": "2026-07-23T23:59:28.142505+00:00", "taskName": null, "msg": "Retrying (Retry(total=2, connect=7, read=5, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused')': /api/admin/provisioning/datasources/reload", "level": "WARNING"} {"time": "2026-07-23T23:59:36.944003+00:00", "taskName": null, "msg": "Retrying (Retry(total=1, connect=6, read=5, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused')': /api/admin/provisioning/datasources/reload", "level": "WARNING"} {"time": "2026-07-23T23:59:54.545623+00:00", "taskName": null, "msg": "Retrying (Retry(total=0, connect=5, read=5, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused')': /api/admin/provisioning/datasources/reload", "level": "WARNING"} {"time": "2026-07-24T00:41:40.118980+00:00", "taskName": null, "msg": "ProtocolError when calling kubernetes: (\"Connection broken: InvalidChunkLength(got length b'', 0 bytes read)\", InvalidChunkLength(got length b'', 0 bytes read))\n", "level": "ERROR"} {"time": "2026-07-24T00:41:40.121845+00:00", "taskName": null, "msg": "ProtocolError when calling kubernetes: (\"Connection broken: InvalidChunkLength(got length b'', 0 bytes read)\", InvalidChunkLength(got length b'', 0 bytes read))\n", "level": "ERROR"} {"time": "2026-07-24T00:41:45.121460+00:00", "taskName": null, "msg": "Retrying (Retry(total=4, connect=9, read=5, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/namespaces/monitoring/secrets?labelSelector=grafana_datasource%3D1&timeoutSeconds=60&watch=True", "level": "WARNING"} {"time": "2026-07-24T00:41:45.123925+00:00", "taskName": null, "msg": "Retrying (Retry(total=4, connect=9, read=5, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/namespaces/monitoring/configmaps?labelSelector=grafana_datasource%3D1&timeoutSeconds=60&watch=True", "level": "WARNING"} {"time": "2026-07-24T00:41:47.322419+00:00", "taskName": null, "msg": "Retrying (Retry(total=3, connect=8, read=5, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/namespaces/monitoring/secrets?labelSelector=grafana_datasource%3D1&timeoutSeconds=60&watch=True", "level": "WARNING"} {"time": "2026-07-24T00:41:47.324675+00:00", "taskName": null, "msg": "Retrying (Retry(total=3, connect=8, read=5, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/namespaces/monitoring/configmaps?labelSelector=grafana_datasource%3D1&timeoutSeconds=60&watch=True", "level": "WARNING"} {"time": "2026-07-24T00:41:51.723529+00:00", "taskName": null, "msg": "Retrying (Retry(total=2, connect=7, read=5, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/namespaces/monitoring/secrets?labelSelector=grafana_datasource%3D1&timeoutSeconds=60&watch=True", "level": "WARNING"} {"time": "2026-07-24T00:41:51.725974+00:00", "taskName": null, "msg": "Retrying (Retry(total=2, connect=7, read=5, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/namespaces/monitoring/configmaps?labelSelector=grafana_datasource%3D1&timeoutSeconds=60&watch=True", "level": "WARNING"} {"time": "2026-07-24T00:42:00.524588+00:00", "taskName": null, "msg": "Retrying (Retry(total=1, connect=6, read=5, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/namespaces/monitoring/secrets?labelSelector=grafana_datasource%3D1&timeoutSeconds=60&watch=True", "level": "WARNING"} {"time": "2026-07-24T00:42:00.527426+00:00", "taskName": null, "msg": "Retrying (Retry(total=1, connect=6, read=5, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/namespaces/monitoring/configmaps?labelSelector=grafana_datasource%3D1&timeoutSeconds=60&watch=True", "level": "WARNING"} {"time": "2026-07-24T00:42:18.125969+00:00", "taskName": null, "msg": "Retrying (Retry(total=0, connect=5, read=5, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/namespaces/monitoring/secrets?labelSelector=grafana_datasource%3D1&timeoutSeconds=60&watch=True", "level": "WARNING"} {"time": "2026-07-24T00:42:18.127188+00:00", "taskName": null, "msg": "MaxRetryError when calling kubernetes: HTTPSConnectionPool(host='10.96.0.1', port=443): Max retries exceeded with url: /api/v1/namespaces/monitoring/secrets?labelSelector=grafana_datasource%3D1&timeoutSeconds=60&watch=True (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused'))\n", "level": "ERROR"} {"time": "2026-07-24T00:42:18.128757+00:00", "taskName": null, "msg": "Retrying (Retry(total=0, connect=5, read=5, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/namespaces/monitoring/configmaps?labelSelector=grafana_datasource%3D1&timeoutSeconds=60&watch=True", "level": "WARNING"} {"time": "2026-07-24T00:42:18.129697+00:00", "taskName": null, "msg": "MaxRetryError when calling kubernetes: HTTPSConnectionPool(host='10.96.0.1', port=443): Max retries exceeded with url: /api/v1/namespaces/monitoring/configmaps?labelSelector=grafana_datasource%3D1&timeoutSeconds=60&watch=True (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused'))\n", "level": "ERROR"} {"time": "2026-07-24T00:42:23.130421+00:00", "taskName": null, "msg": "Retrying (Retry(total=4, connect=9, read=5, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/namespaces/monitoring/secrets?labelSelector=grafana_datasource%3D1&timeoutSeconds=60&watch=True", "level": "WARNING"} {"time": "2026-07-24T00:42:23.131970+00:00", "taskName": null, "msg": "Retrying (Retry(total=4, connect=9, read=5, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/namespaces/monitoring/configmaps?labelSelector=grafana_datasource%3D1&timeoutSeconds=60&watch=True", "level": "WARNING"} {"time": "2026-07-24T00:42:25.331504+00:00", "taskName": null, "msg": "Retrying (Retry(total=3, connect=8, read=5, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/namespaces/monitoring/secrets?labelSelector=grafana_datasource%3D1&timeoutSeconds=60&watch=True", "level": "WARNING"} {"time": "2026-07-24T00:42:25.332739+00:00", "taskName": null, "msg": "Retrying (Retry(total=3, connect=8, read=5, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/namespaces/monitoring/configmaps?labelSelector=grafana_datasource%3D1&timeoutSeconds=60&watch=True", "level": "WARNING"} {"time": "2026-07-24T00:42:29.732810+00:00", "taskName": null, "msg": "Retrying (Retry(total=2, connect=7, read=5, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/namespaces/monitoring/secrets?labelSelector=grafana_datasource%3D1&timeoutSeconds=60&watch=True", "level": "WARNING"} {"time": "2026-07-24T00:42:29.734066+00:00", "taskName": null, "msg": "Retrying (Retry(total=2, connect=7, read=5, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/namespaces/monitoring/configmaps?labelSelector=grafana_datasource%3D1&timeoutSeconds=60&watch=True", "level": "WARNING"} {"time": "2026-07-24T00:42:38.534105+00:00", "taskName": null, "msg": "Retrying (Retry(total=1, connect=6, read=5, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/namespaces/monitoring/secrets?labelSelector=grafana_datasource%3D1&timeoutSeconds=60&watch=True", "level": "WARNING"} {"time": "2026-07-24T00:42:38.534983+00:00", "taskName": null, "msg": "Retrying (Retry(total=1, connect=6, read=5, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/namespaces/monitoring/configmaps?labelSelector=grafana_datasource%3D1&timeoutSeconds=60&watch=True", "level": "WARNING"} {"time": "2026-07-24T00:42:56.136364+00:00", "taskName": null, "msg": "Retrying (Retry(total=0, connect=5, read=5, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/namespaces/monitoring/secrets?labelSelector=grafana_datasource%3D1&timeoutSeconds=60&watch=True", "level": "WARNING"} {"time": "2026-07-24T00:42:56.136343+00:00", "taskName": null, "msg": "Retrying (Retry(total=0, connect=5, read=5, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/namespaces/monitoring/configmaps?labelSelector=grafana_datasource%3D1&timeoutSeconds=60&watch=True", "level": "WARNING"} {"time": "2026-07-24T00:42:56.137797+00:00", "taskName": null, "msg": "MaxRetryError when calling kubernetes: HTTPSConnectionPool(host='10.96.0.1', port=443): Max retries exceeded with url: /api/v1/namespaces/monitoring/secrets?labelSelector=grafana_datasource%3D1&timeoutSeconds=60&watch=True (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused'))\n", "level": "ERROR"} {"time": "2026-07-24T00:42:56.137912+00:00", "taskName": null, "msg": "MaxRetryError when calling kubernetes: HTTPSConnectionPool(host='10.96.0.1', port=443): Max retries exceeded with url: /api/v1/namespaces/monitoring/configmaps?labelSelector=grafana_datasource%3D1&timeoutSeconds=60&watch=True (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused'))\n", "level": "ERROR"} {"time": "2026-07-24T00:43:01.139926+00:00", "taskName": null, "msg": "Retrying (Retry(total=4, connect=9, read=5, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/namespaces/monitoring/secrets?labelSelector=grafana_datasource%3D1&timeoutSeconds=60&watch=True", "level": "WARNING"} {"time": "2026-07-24T00:43:01.140381+00:00", "taskName": null, "msg": "Retrying (Retry(total=4, connect=9, read=5, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/namespaces/monitoring/configmaps?labelSelector=grafana_datasource%3D1&timeoutSeconds=60&watch=True", "level": "WARNING"} {"time": "2026-07-24T00:43:03.340683+00:00", "taskName": null, "msg": "Retrying (Retry(total=3, connect=8, read=5, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/namespaces/monitoring/secrets?labelSelector=grafana_datasource%3D1&timeoutSeconds=60&watch=True", "level": "WARNING"} {"time": "2026-07-24T00:43:03.341249+00:00", "taskName": null, "msg": "Retrying (Retry(total=3, connect=8, read=5, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/namespaces/monitoring/configmaps?labelSelector=grafana_datasource%3D1&timeoutSeconds=60&watch=True", "level": "WARNING"} {"time": "2026-07-24T00:43:07.742046+00:00", "taskName": null, "msg": "Retrying (Retry(total=2, connect=7, read=5, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/namespaces/monitoring/secrets?labelSelector=grafana_datasource%3D1&timeoutSeconds=60&watch=True", "level": "WARNING"} {"time": "2026-07-24T00:43:07.742921+00:00", "taskName": null, "msg": "Retrying (Retry(total=2, connect=7, read=5, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/namespaces/monitoring/configmaps?labelSelector=grafana_datasource%3D1&timeoutSeconds=60&watch=True", "level": "WARNING"} {"time": "2026-07-24T00:43:16.543554+00:00", "taskName": null, "msg": "Retrying (Retry(total=1, connect=6, read=5, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/namespaces/monitoring/secrets?labelSelector=grafana_datasource%3D1&timeoutSeconds=60&watch=True", "level": "WARNING"} {"time": "2026-07-24T00:43:16.544411+00:00", "taskName": null, "msg": "Retrying (Retry(total=1, connect=6, read=5, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/namespaces/monitoring/configmaps?labelSelector=grafana_datasource%3D1&timeoutSeconds=60&watch=True", "level": "WARNING"} {"time": "2026-07-24T00:43:34.145073+00:00", "taskName": null, "msg": "Retrying (Retry(total=0, connect=5, read=5, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/namespaces/monitoring/secrets?labelSelector=grafana_datasource%3D1&timeoutSeconds=60&watch=True", "level": "WARNING"} {"time": "2026-07-24T00:43:34.145717+00:00", "taskName": null, "msg": "Retrying (Retry(total=0, connect=5, read=5, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/namespaces/monitoring/configmaps?labelSelector=grafana_datasource%3D1&timeoutSeconds=60&watch=True", "level": "WARNING"} {"time": "2026-07-24T00:43:34.146212+00:00", "taskName": null, "msg": "MaxRetryError when calling kubernetes: HTTPSConnectionPool(host='10.96.0.1', port=443): Max retries exceeded with url: /api/v1/namespaces/monitoring/secrets?labelSelector=grafana_datasource%3D1&timeoutSeconds=60&watch=True (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused'))\n", "level": "ERROR"} {"time": "2026-07-24T00:43:34.146683+00:00", "taskName": null, "msg": "MaxRetryError when calling kubernetes: HTTPSConnectionPool(host='10.96.0.1', port=443): Max retries exceeded with url: /api/v1/namespaces/monitoring/configmaps?labelSelector=grafana_datasource%3D1&timeoutSeconds=60&watch=True (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused'))\n", "level": "ERROR"} {"time": "2026-07-24T00:43:41.289500+00:00", "taskName": null, "msg": "ApiException when calling kubernetes: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({'Audit-Id': '4cc5729c-bc1f-4ceb-9beb-3972a54b04ac', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'X-Kubernetes-Pf-Flowschema-Uid': '', 'X-Kubernetes-Pf-Prioritylevel-Uid': '', 'Date': 'Fri, 24 Jul 2026 00:43:41 GMT', 'Content-Length': '736'})\nHTTP response body: b'{\"kind\":\"Status\",\"apiVersion\":\"v1\",\"metadata\":{},\"status\":\"Failure\",\"message\":\"secrets is forbidden: User \\\\\"system:serviceaccount:monitoring:kube-prometheus-stack-grafana\\\\\" cannot watch resource \\\\\"secrets\\\\\" in API group \\\\\"\\\\\" in the namespace \\\\\"monitoring\\\\\": RBAC: [clusterrole.rbac.authorization.k8s.io \\\\\"system:public-info-viewer\\\\\" not found, clusterrole.rbac.authorization.k8s.io \\\\\"system:service-account-issuer-discovery\\\\\" not found, clusterrole.rbac.authorization.k8s.io \\\\\"system:discovery\\\\\" not found, clusterrole.rbac.authorization.k8s.io \\\\\"kube-prometheus-stack-grafana-clusterrole\\\\\" not found, clusterrole.rbac.authorization.k8s.io \\\\\"system:basic-user\\\\\" not found]\",\"reason\":\"Forbidden\",\"details\":{\"kind\":\"secrets\"},\"code\":403}\\n'\n\n", "level": "ERROR"} {"time": "2026-07-24T00:43:41.295439+00:00", "taskName": null, "msg": "ApiException when calling kubernetes: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({'Audit-Id': '2f55bc94-baa4-4f47-b638-a4fbbf6e4b53', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'X-Kubernetes-Pf-Flowschema-Uid': '', 'X-Kubernetes-Pf-Prioritylevel-Uid': '', 'Date': 'Fri, 24 Jul 2026 00:43:41 GMT', 'Content-Length': '745'})\nHTTP response body: b'{\"kind\":\"Status\",\"apiVersion\":\"v1\",\"metadata\":{},\"status\":\"Failure\",\"message\":\"configmaps is forbidden: User \\\\\"system:serviceaccount:monitoring:kube-prometheus-stack-grafana\\\\\" cannot watch resource \\\\\"configmaps\\\\\" in API group \\\\\"\\\\\" in the namespace \\\\\"monitoring\\\\\": RBAC: [clusterrole.rbac.authorization.k8s.io \\\\\"system:basic-user\\\\\" not found, clusterrole.rbac.authorization.k8s.io \\\\\"system:public-info-viewer\\\\\" not found, clusterrole.rbac.authorization.k8s.io \\\\\"system:service-account-issuer-discovery\\\\\" not found, clusterrole.rbac.authorization.k8s.io \\\\\"system:discovery\\\\\" not found, clusterrole.rbac.authorization.k8s.io \\\\\"kube-prometheus-stack-grafana-clusterrole\\\\\" not found]\",\"reason\":\"Forbidden\",\"details\":{\"kind\":\"configmaps\"},\"code\":403}\\n'\n\n", "level": "ERROR"}