+ exec staffeln-conductor --config-file /etc/staffeln/staffeln.conf 2026-07-15 11:40:14.021 8 INFO staffeln.conductor.manager [-] Staffeln conductor backup controller init 2026-07-15 11:40:14.022 8 INFO staffeln.conductor.manager [-] Staffeln conductor backup controller run 2026-07-15 11:40:14.023 8 INFO staffeln.conductor.manager [-] Backup manager started 1784115614.0230014 2026-07-15 11:40:14.023 8 INFO staffeln.conductor.manager [-] Staffeln conductor backup controller periodics 2026-07-15 11:40:14.030 10 INFO staffeln.conductor.manager [-] Staffeln conductor rotation controller init 2026-07-15 11:40:14.031 10 INFO staffeln.conductor.manager [-] Staffeln conductor rotation controller run 2026-07-15 11:40:14.031 10 INFO staffeln.conductor.manager [-] Staffeln conductor rotation controller rotation_engine kube_config_path not provided and default location (~/.kube/config) does not exist. Using inCluster Config. This might not work. 2026-07-15 11:40:14.052 8 INFO staffeln.conductor.manager [-] Running as puller role 2026-07-15 11:40:14.052 8 INFO staffeln.conductor.manager [-] Updating backup task queue... kube_config_path not provided and default location (~/.kube/config) does not exist. Using inCluster Config. This might not work. 2026-07-15 11:40:14.055 10 INFO staffeln.conductor.manager [-] Starting task to check rotation... 2026-07-15 11:40:14.376 10 ERROR futurist.periodics [-] Failed to call immediate 'staffeln.conductor.manager.RotationManager.rotation_engine..rotation_tasks' (it runs every 3600.00 seconds): keystoneauth1.exceptions.catalog.EndpointNotFound: internal endpoint for compute service in RegionOne region not found 2026-07-15 11:40:14.376 10 ERROR futurist.periodics Traceback (most recent call last): 2026-07-15 11:40:14.376 10 ERROR futurist.periodics File "/var/lib/openstack/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run 2026-07-15 11:40:14.376 10 ERROR futurist.periodics work() 2026-07-15 11:40:14.376 10 ERROR futurist.periodics File "/var/lib/openstack/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ 2026-07-15 11:40:14.376 10 ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) 2026-07-15 11:40:14.376 10 ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2026-07-15 11:40:14.376 10 ERROR futurist.periodics File "/var/lib/openstack/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator 2026-07-15 11:40:14.376 10 ERROR futurist.periodics return f(*args, **kwargs) 2026-07-15 11:40:14.376 10 ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ 2026-07-15 11:40:14.376 10 ERROR futurist.periodics File "/var/lib/openstack/lib/python3.12/site-packages/staffeln/conductor/manager.py", line 271, in rotation_tasks 2026-07-15 11:40:14.376 10 ERROR futurist.periodics self.controller.collect_instance_retention_map() 2026-07-15 11:40:14.376 10 ERROR futurist.periodics File "/var/lib/openstack/lib/python3.12/site-packages/staffeln/conductor/backup.py", line 488, in collect_instance_retention_map 2026-07-15 11:40:14.376 10 ERROR futurist.periodics servers = self.openstacksdk.get_servers(all_projects=True) 2026-07-15 11:40:14.376 10 ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2026-07-15 11:40:14.376 10 ERROR futurist.periodics File "/var/lib/openstack/lib/python3.12/site-packages/tenacity/__init__.py", line 338, in wrapped_f 2026-07-15 11:40:14.376 10 ERROR futurist.periodics return copy(f, *args, **kw) 2026-07-15 11:40:14.376 10 ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^ 2026-07-15 11:40:14.376 10 ERROR futurist.periodics File "/var/lib/openstack/lib/python3.12/site-packages/tenacity/__init__.py", line 477, in __call__ 2026-07-15 11:40:14.376 10 ERROR futurist.periodics do = self.iter(retry_state=retry_state) 2026-07-15 11:40:14.376 10 ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2026-07-15 11:40:14.376 10 ERROR futurist.periodics File "/var/lib/openstack/lib/python3.12/site-packages/tenacity/__init__.py", line 378, in iter 2026-07-15 11:40:14.376 10 ERROR futurist.periodics result = action(retry_state) 2026-07-15 11:40:14.376 10 ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^ 2026-07-15 11:40:14.376 10 ERROR futurist.periodics File "/var/lib/openstack/lib/python3.12/site-packages/tenacity/__init__.py", line 400, in 2026-07-15 11:40:14.376 10 ERROR futurist.periodics self._add_action_func(lambda rs: rs.outcome.result()) 2026-07-15 11:40:14.376 10 ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^ 2026-07-15 11:40:14.376 10 ERROR futurist.periodics File "/usr/lib/python3.12/concurrent/futures/_base.py", line 449, in result 2026-07-15 11:40:14.376 10 ERROR futurist.periodics return self.__get_result() 2026-07-15 11:40:14.376 10 ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^ 2026-07-15 11:40:14.376 10 ERROR futurist.periodics File "/usr/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result 2026-07-15 11:40:14.376 10 ERROR futurist.periodics raise self._exception 2026-07-15 11:40:14.376 10 ERROR futurist.periodics File "/var/lib/openstack/lib/python3.12/site-packages/tenacity/__init__.py", line 480, in __call__ 2026-07-15 11:40:14.376 10 ERROR futurist.periodics result = fn(*args, **kwargs) 2026-07-15 11:40:14.376 10 ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^ 2026-07-15 11:40:14.376 10 ERROR futurist.periodics File "/var/lib/openstack/lib/python3.12/site-packages/staffeln/common/openstack.py", line 141, in get_servers 2026-07-15 11:40:14.376 10 ERROR futurist.periodics return self.conn.compute.servers(details=details, all_projects=all_projects) 2026-07-15 11:40:14.376 10 ERROR futurist.periodics ^^^^^^^^^^^^^^^^^ 2026-07-15 11:40:14.376 10 ERROR futurist.periodics File "/var/lib/openstack/lib/python3.12/site-packages/openstack/service_description.py", line 91, in __get__ 2026-07-15 11:40:14.376 10 ERROR futurist.periodics proxy = self._make_proxy(instance) 2026-07-15 11:40:14.376 10 ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^ 2026-07-15 11:40:14.376 10 ERROR futurist.periodics File "/var/lib/openstack/lib/python3.12/site-packages/openstack/service_description.py", line 289, in _make_proxy 2026-07-15 11:40:14.376 10 ERROR futurist.periodics found_version = temp_adapter.get_api_major_version() 2026-07-15 11:40:14.376 10 ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2026-07-15 11:40:14.376 10 ERROR futurist.periodics File "/var/lib/openstack/lib/python3.12/site-packages/keystoneauth1/adapter.py", line 404, in get_api_major_version 2026-07-15 11:40:14.376 10 ERROR futurist.periodics return self.session.get_api_major_version(auth or self.auth, **kwargs) 2026-07-15 11:40:14.376 10 ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2026-07-15 11:40:14.376 10 ERROR futurist.periodics File "/var/lib/openstack/lib/python3.12/site-packages/keystoneauth1/session.py", line 1470, in get_api_major_version 2026-07-15 11:40:14.376 10 ERROR futurist.periodics return auth.get_api_major_version(self, **kwargs) 2026-07-15 11:40:14.376 10 ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2026-07-15 11:40:14.376 10 ERROR futurist.periodics File "/var/lib/openstack/lib/python3.12/site-packages/keystoneauth1/identity/base.py", line 573, in get_api_major_version 2026-07-15 11:40:14.376 10 ERROR futurist.periodics data = get_endpoint_data(discover_versions=discover_versions) 2026-07-15 11:40:14.376 10 ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2026-07-15 11:40:14.376 10 ERROR futurist.periodics File "/var/lib/openstack/lib/python3.12/site-packages/keystoneauth1/identity/base.py", line 300, in get_endpoint_data 2026-07-15 11:40:14.376 10 ERROR futurist.periodics endpoint_data = service_catalog.endpoint_data_for( 2026-07-15 11:40:14.376 10 ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2026-07-15 11:40:14.376 10 ERROR futurist.periodics File "/var/lib/openstack/lib/python3.12/site-packages/keystoneauth1/access/service_catalog.py", line 532, in endpoint_data_for 2026-07-15 11:40:14.376 10 ERROR futurist.periodics raise exceptions.EndpointNotFound(msg) 2026-07-15 11:40:14.376 10 ERROR futurist.periodics keystoneauth1.exceptions.catalog.EndpointNotFound: internal endpoint for compute service in RegionOne region not found 2026-07-15 11:40:14.376 10 ERROR futurist.periodics 2026-07-15 11:40:14.941 8 ERROR futurist.periodics [-] Failed to call immediate 'staffeln.conductor.manager.BackupManager.backup_engine..backup_tasks' (it runs every 3600.00 seconds): sqlalchemy.exc.ProgrammingError: (pymysql.err.ProgrammingError) (1146, "Table 'staffeln.queue_data' doesn't exist") [SQL: SELECT queue_data.id AS queue_data_id, queue_data.backup_id AS queue_data_backup_id, queue_data.project_id AS queue_data_project_id, queue_data.volume_id AS queue_data_volume_id, queue_data.backup_status AS queue_data_backup_status, queue_data.instance_id AS queue_data_instance_id, queue_data.volume_name AS queue_data_volume_name, queue_data.instance_name AS queue_data_instance_name, queue_data.incremental AS queue_data_incremental, queue_data.reason AS queue_data_reason, queue_data.created_at AS queue_data_created_at, queue_data.updated_at AS queue_data_updated_at FROM queue_data WHERE queue_data.backup_status = %(backup_status_1)s ORDER BY queue_data.id ASC] [parameters: {'backup_status_1': 1}] (Background on this error at: https://sqlalche.me/e/20/f405) 2026-07-15 11:40:14.941 8 ERROR futurist.periodics Traceback (most recent call last): 2026-07-15 11:40:14.941 8 ERROR futurist.periodics File "/var/lib/openstack/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1961, in _exec_single_context 2026-07-15 11:40:14.941 8 ERROR futurist.periodics self.dialect.do_execute( 2026-07-15 11:40:14.941 8 ERROR futurist.periodics File "/var/lib/openstack/lib/python3.12/site-packages/sqlalchemy/engine/default.py", line 944, in do_execute 2026-07-15 11:40:14.941 8 ERROR futurist.periodics cursor.execute(statement, parameters) 2026-07-15 11:40:14.941 8 ERROR futurist.periodics File "/var/lib/openstack/lib/python3.12/site-packages/pymysql/cursors.py", line 153, in execute 2026-07-15 11:40:14.941 8 ERROR futurist.periodics result = self._query(query) 2026-07-15 11:40:14.941 8 ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ 2026-07-15 11:40:14.941 8 ERROR futurist.periodics File "/var/lib/openstack/lib/python3.12/site-packages/pymysql/cursors.py", line 322, in _query 2026-07-15 11:40:14.941 8 ERROR futurist.periodics conn.query(q) 2026-07-15 11:40:14.941 8 ERROR futurist.periodics File "/var/lib/openstack/lib/python3.12/site-packages/pymysql/connections.py", line 563, in query 2026-07-15 11:40:14.941 8 ERROR futurist.periodics self._affected_rows = self._read_query_result(unbuffered=unbuffered) 2026-07-15 11:40:14.941 8 ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2026-07-15 11:40:14.941 8 ERROR futurist.periodics File "/var/lib/openstack/lib/python3.12/site-packages/pymysql/connections.py", line 825, in _read_query_result 2026-07-15 11:40:14.941 8 ERROR futurist.periodics result.read() 2026-07-15 11:40:14.941 8 ERROR futurist.periodics File "/var/lib/openstack/lib/python3.12/site-packages/pymysql/connections.py", line 1199, in read 2026-07-15 11:40:14.941 8 ERROR futurist.periodics first_packet = self.connection._read_packet() 2026-07-15 11:40:14.941 8 ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2026-07-15 11:40:14.941 8 ERROR futurist.periodics File "/var/lib/openstack/lib/python3.12/site-packages/pymysql/connections.py", line 775, in _read_packet 2026-07-15 11:40:14.941 8 ERROR futurist.periodics packet.raise_for_error() 2026-07-15 11:40:14.941 8 ERROR futurist.periodics File "/var/lib/openstack/lib/python3.12/site-packages/pymysql/protocol.py", line 219, in raise_for_error 2026-07-15 11:40:14.941 8 ERROR futurist.periodics err.raise_mysql_exception(self._data) 2026-07-15 11:40:14.941 8 ERROR futurist.periodics File "/var/lib/openstack/lib/python3.12/site-packages/pymysql/err.py", line 150, in raise_mysql_exception 2026-07-15 11:40:14.941 8 ERROR futurist.periodics raise errorclass(errno, errval) 2026-07-15 11:40:14.941 8 ERROR futurist.periodics pymysql.err.ProgrammingError: (1146, "Table 'staffeln.queue_data' doesn't exist") 2026-07-15 11:40:14.941 8 ERROR futurist.periodics 2026-07-15 11:40:14.941 8 ERROR futurist.periodics The above exception was the direct cause of the following exception: 2026-07-15 11:40:14.941 8 ERROR futurist.periodics 2026-07-15 11:40:14.941 8 ERROR futurist.periodics Traceback (most recent call last): 2026-07-15 11:40:14.941 8 ERROR futurist.periodics File "/var/lib/openstack/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run 2026-07-15 11:40:14.941 8 ERROR futurist.periodics work() 2026-07-15 11:40:14.941 8 ERROR futurist.periodics File "/var/lib/openstack/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ 2026-07-15 11:40:14.941 8 ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) 2026-07-15 11:40:14.941 8 ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2026-07-15 11:40:14.941 8 ERROR futurist.periodics File "/var/lib/openstack/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator 2026-07-15 11:40:14.941 8 ERROR futurist.periodics return f(*args, **kwargs) 2026-07-15 11:40:14.941 8 ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ 2026-07-15 11:40:14.941 8 ERROR futurist.periodics File "/var/lib/openstack/lib/python3.12/site-packages/staffeln/conductor/manager.py", line 177, in backup_tasks 2026-07-15 11:40:14.941 8 ERROR futurist.periodics self._update_task_queue() 2026-07-15 11:40:14.941 8 ERROR futurist.periodics File "/var/lib/openstack/lib/python3.12/site-packages/staffeln/conductor/manager.py", line 137, in _update_task_queue 2026-07-15 11:40:14.941 8 ERROR futurist.periodics current_wip_tasks = self.controller.get_queues(filters=filters) 2026-07-15 11:40:14.941 8 ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2026-07-15 11:40:14.941 8 ERROR futurist.periodics File "/var/lib/openstack/lib/python3.12/site-packages/staffeln/conductor/backup.py", line 108, in get_queues 2026-07-15 11:40:14.941 8 ERROR futurist.periodics queues = objects.Queue.list( # pylint: disable=E1120 2026-07-15 11:40:14.941 8 ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2026-07-15 11:40:14.941 8 ERROR futurist.periodics File "/var/lib/openstack/lib/python3.12/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper 2026-07-15 11:40:14.941 8 ERROR futurist.periodics result = fn(cls, context, *args, **kwargs) 2026-07-15 11:40:14.941 8 ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2026-07-15 11:40:14.941 8 ERROR futurist.periodics File "/var/lib/openstack/lib/python3.12/site-packages/staffeln/objects/queue.py", line 39, in list 2026-07-15 11:40:14.941 8 ERROR futurist.periodics db_queue = cls.dbapi.get_queue_list(context, filters=filters) 2026-07-15 11:40:14.941 8 ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2026-07-15 11:40:14.941 8 ERROR futurist.periodics File "/var/lib/openstack/lib/python3.12/site-packages/staffeln/db/sqlalchemy/api.py", line 310, in get_queue_list 2026-07-15 11:40:14.941 8 ERROR futurist.periodics return self._get_model_list( 2026-07-15 11:40:14.941 8 ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^ 2026-07-15 11:40:14.941 8 ERROR futurist.periodics File "/var/lib/openstack/lib/python3.12/site-packages/staffeln/db/sqlalchemy/api.py", line 273, in _get_model_list 2026-07-15 11:40:14.941 8 ERROR futurist.periodics return _paginate_query(model, limit, marker, sort_key, sort_dir, query) 2026-07-15 11:40:14.941 8 ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2026-07-15 11:40:14.941 8 ERROR futurist.periodics File "/var/lib/openstack/lib/python3.12/site-packages/staffeln/db/sqlalchemy/api.py", line 84, in _paginate_query 2026-07-15 11:40:14.941 8 ERROR futurist.periodics return query.all() 2026-07-15 11:40:14.941 8 ERROR futurist.periodics ^^^^^^^^^^^ 2026-07-15 11:40:14.941 8 ERROR futurist.periodics File "/var/lib/openstack/lib/python3.12/site-packages/sqlalchemy/orm/query.py", line 2704, in all 2026-07-15 11:40:14.941 8 ERROR futurist.periodics return self._iter().all() # type: ignore 2026-07-15 11:40:14.941 8 ERROR futurist.periodics ^^^^^^^^^^^^ 2026-07-15 11:40:14.941 8 ERROR futurist.periodics File "/var/lib/openstack/lib/python3.12/site-packages/sqlalchemy/orm/query.py", line 2857, in _iter 2026-07-15 11:40:14.941 8 ERROR futurist.periodics result: Union[ScalarResult[_T], Result[_T]] = self.session.execute( 2026-07-15 11:40:14.941 8 ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^ 2026-07-15 11:40:14.941 8 ERROR futurist.periodics File "/var/lib/openstack/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 2365, in execute 2026-07-15 11:40:14.941 8 ERROR futurist.periodics return self._execute_internal( 2026-07-15 11:40:14.941 8 ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^ 2026-07-15 11:40:14.941 8 ERROR futurist.periodics File "/var/lib/openstack/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 2251, in _execute_internal 2026-07-15 11:40:14.941 8 ERROR futurist.periodics result: Result[Any] = compile_state_cls.orm_execute_statement( 2026-07-15 11:40:14.941 8 ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2026-07-15 11:40:14.941 8 ERROR futurist.periodics File "/var/lib/openstack/lib/python3.12/site-packages/sqlalchemy/orm/context.py", line 306, in orm_execute_statement 2026-07-15 11:40:14.941 8 ERROR futurist.periodics result = conn.execute( 2026-07-15 11:40:14.941 8 ERROR futurist.periodics ^^^^^^^^^^^^^ 2026-07-15 11:40:14.941 8 ERROR futurist.periodics File "/var/lib/openstack/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1413, in execute 2026-07-15 11:40:14.941 8 ERROR futurist.periodics return meth( 2026-07-15 11:40:14.941 8 ERROR futurist.periodics ^^^^^ 2026-07-15 11:40:14.941 8 ERROR futurist.periodics File "/var/lib/openstack/lib/python3.12/site-packages/sqlalchemy/sql/elements.py", line 526, in _execute_on_connection 2026-07-15 11:40:14.941 8 ERROR futurist.periodics return connection._execute_clauseelement( 2026-07-15 11:40:14.941 8 ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2026-07-15 11:40:14.941 8 ERROR futurist.periodics File "/var/lib/openstack/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1635, in _execute_clauseelement 2026-07-15 11:40:14.941 8 ERROR futurist.periodics ret = self._execute_context( 2026-07-15 11:40:14.941 8 ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^ 2026-07-15 11:40:14.941 8 ERROR futurist.periodics File "/var/lib/openstack/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1840, in _execute_context 2026-07-15 11:40:14.941 8 ERROR futurist.periodics return self._exec_single_context( 2026-07-15 11:40:14.941 8 ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^ 2026-07-15 11:40:14.941 8 ERROR futurist.periodics File "/var/lib/openstack/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1980, in _exec_single_context 2026-07-15 11:40:14.941 8 ERROR futurist.periodics self._handle_dbapi_exception( 2026-07-15 11:40:14.941 8 ERROR futurist.periodics File "/var/lib/openstack/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 2346, in _handle_dbapi_exception 2026-07-15 11:40:14.941 8 ERROR futurist.periodics raise newraise.with_traceback(exc_info[2]) from e 2026-07-15 11:40:14.941 8 ERROR futurist.periodics File "/var/lib/openstack/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1961, in _exec_single_context 2026-07-15 11:40:14.941 8 ERROR futurist.periodics self.dialect.do_execute( 2026-07-15 11:40:14.941 8 ERROR futurist.periodics File "/var/lib/openstack/lib/python3.12/site-packages/sqlalchemy/engine/default.py", line 944, in do_execute 2026-07-15 11:40:14.941 8 ERROR futurist.periodics cursor.execute(statement, parameters) 2026-07-15 11:40:14.941 8 ERROR futurist.periodics File "/var/lib/openstack/lib/python3.12/site-packages/pymysql/cursors.py", line 153, in execute 2026-07-15 11:40:14.941 8 ERROR futurist.periodics result = self._query(query) 2026-07-15 11:40:14.941 8 ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ 2026-07-15 11:40:14.941 8 ERROR futurist.periodics File "/var/lib/openstack/lib/python3.12/site-packages/pymysql/cursors.py", line 322, in _query 2026-07-15 11:40:14.941 8 ERROR futurist.periodics conn.query(q) 2026-07-15 11:40:14.941 8 ERROR futurist.periodics File "/var/lib/openstack/lib/python3.12/site-packages/pymysql/connections.py", line 563, in query 2026-07-15 11:40:14.941 8 ERROR futurist.periodics self._affected_rows = self._read_query_result(unbuffered=unbuffered) 2026-07-15 11:40:14.941 8 ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2026-07-15 11:40:14.941 8 ERROR futurist.periodics File "/var/lib/openstack/lib/python3.12/site-packages/pymysql/connections.py", line 825, in _read_query_result 2026-07-15 11:40:14.941 8 ERROR futurist.periodics result.read() 2026-07-15 11:40:14.941 8 ERROR futurist.periodics File "/var/lib/openstack/lib/python3.12/site-packages/pymysql/connections.py", line 1199, in read 2026-07-15 11:40:14.941 8 ERROR futurist.periodics first_packet = self.connection._read_packet() 2026-07-15 11:40:14.941 8 ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2026-07-15 11:40:14.941 8 ERROR futurist.periodics File "/var/lib/openstack/lib/python3.12/site-packages/pymysql/connections.py", line 775, in _read_packet 2026-07-15 11:40:14.941 8 ERROR futurist.periodics packet.raise_for_error() 2026-07-15 11:40:14.941 8 ERROR futurist.periodics File "/var/lib/openstack/lib/python3.12/site-packages/pymysql/protocol.py", line 219, in raise_for_error 2026-07-15 11:40:14.941 8 ERROR futurist.periodics err.raise_mysql_exception(self._data) 2026-07-15 11:40:14.941 8 ERROR futurist.periodics File "/var/lib/openstack/lib/python3.12/site-packages/pymysql/err.py", line 150, in raise_mysql_exception 2026-07-15 11:40:14.941 8 ERROR futurist.periodics raise errorclass(errno, errval) 2026-07-15 11:40:14.941 8 ERROR futurist.periodics sqlalchemy.exc.ProgrammingError: (pymysql.err.ProgrammingError) (1146, "Table 'staffeln.queue_data' doesn't exist") 2026-07-15 11:40:14.941 8 ERROR futurist.periodics [SQL: SELECT queue_data.id AS queue_data_id, queue_data.backup_id AS queue_data_backup_id, queue_data.project_id AS queue_data_project_id, queue_data.volume_id AS queue_data_volume_id, queue_data.backup_status AS queue_data_backup_status, queue_data.instance_id AS queue_data_instance_id, queue_data.volume_name AS queue_data_volume_name, queue_data.instance_name AS queue_data_instance_name, queue_data.incremental AS queue_data_incremental, queue_data.reason AS queue_data_reason, queue_data.created_at AS queue_data_created_at, queue_data.updated_at AS queue_data_updated_at 2026-07-15 11:40:14.941 8 ERROR futurist.periodics FROM queue_data 2026-07-15 11:40:14.941 8 ERROR futurist.periodics WHERE queue_data.backup_status = %(backup_status_1)s ORDER BY queue_data.id ASC] 2026-07-15 11:40:14.941 8 ERROR futurist.periodics [parameters: {'backup_status_1': 1}] 2026-07-15 11:40:14.941 8 ERROR futurist.periodics (Background on this error at: https://sqlalche.me/e/20/f405) 2026-07-15 11:40:14.941 8 ERROR futurist.periodics kube_config_path not provided and default location (~/.kube/config) does not exist. Using inCluster Config. This might not work. 2026-07-15 12:40:23.180 10 ERROR futurist.periodics [-] Failed to call periodic 'staffeln.conductor.manager.RotationManager.rotation_engine..rotation_tasks' (it runs every 3600.00 seconds): sherlock.lock.LockException: Failed to read Lock. 2026-07-15 12:40:23.180 10 ERROR futurist.periodics Traceback (most recent call last): 2026-07-15 12:40:23.180 10 ERROR futurist.periodics File "/var/lib/openstack/lib/python3.12/site-packages/sherlock/lock.py", line 934, in _get_lease 2026-07-15 12:40:23.180 10 ERROR futurist.periodics return self.client.read_namespaced_lease( 2026-07-15 12:40:23.180 10 ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2026-07-15 12:40:23.180 10 ERROR futurist.periodics File "/var/lib/openstack/lib/python3.12/site-packages/kubernetes/client/api/coordination_v1_api.py", line 1149, in read_namespaced_lease 2026-07-15 12:40:23.180 10 ERROR futurist.periodics return self.read_namespaced_lease_with_http_info(name, namespace, **kwargs) # noqa: E501 2026-07-15 12:40:23.180 10 ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2026-07-15 12:40:23.180 10 ERROR futurist.periodics File "/var/lib/openstack/lib/python3.12/site-packages/kubernetes/client/api/coordination_v1_api.py", line 1236, in read_namespaced_lease_with_http_info 2026-07-15 12:40:23.180 10 ERROR futurist.periodics return self.api_client.call_api( 2026-07-15 12:40:23.180 10 ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^ 2026-07-15 12:40:23.180 10 ERROR futurist.periodics File "/var/lib/openstack/lib/python3.12/site-packages/kubernetes/client/api_client.py", line 348, in call_api 2026-07-15 12:40:23.180 10 ERROR futurist.periodics return self.__call_api(resource_path, method, 2026-07-15 12:40:23.180 10 ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2026-07-15 12:40:23.180 10 ERROR futurist.periodics File "/var/lib/openstack/lib/python3.12/site-packages/kubernetes/client/api_client.py", line 180, in __call_api 2026-07-15 12:40:23.180 10 ERROR futurist.periodics response_data = self.request( 2026-07-15 12:40:23.180 10 ERROR futurist.periodics ^^^^^^^^^^^^^ 2026-07-15 12:40:23.180 10 ERROR futurist.periodics File "/var/lib/openstack/lib/python3.12/site-packages/kubernetes/client/api_client.py", line 373, in request 2026-07-15 12:40:23.180 10 ERROR futurist.periodics return self.rest_client.GET(url, 2026-07-15 12:40:23.180 10 ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^ 2026-07-15 12:40:23.180 10 ERROR futurist.periodics File "/var/lib/openstack/lib/python3.12/site-packages/kubernetes/client/rest.py", line 244, in GET 2026-07-15 12:40:23.180 10 ERROR futurist.periodics return self.request("GET", url, 2026-07-15 12:40:23.180 10 ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ 2026-07-15 12:40:23.180 10 ERROR futurist.periodics File "/var/lib/openstack/lib/python3.12/site-packages/kubernetes/client/rest.py", line 238, in request 2026-07-15 12:40:23.180 10 ERROR futurist.periodics raise ApiException(http_resp=r) 2026-07-15 12:40:23.180 10 ERROR futurist.periodics kubernetes.client.exceptions.ApiException: (500) 2026-07-15 12:40:23.180 10 ERROR futurist.periodics Reason: Internal Server Error 2026-07-15 12:40:23.180 10 ERROR futurist.periodics HTTP response headers: HTTPHeaderDict({'Audit-Id': 'a8294e47-74a2-4c2e-87f9-b3bf23f460dc', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Kubernetes-Pf-Flowschema-Uid': 'cdb3d981-cd09-4835-9e88-cbddf97a2a40', 'X-Kubernetes-Pf-Prioritylevel-Uid': 'c6b405d9-edd8-4f83-8e8d-86e05012eeb7', 'Date': 'Wed, 15 Jul 2026 12:40:23 GMT', 'Content-Length': '122'}) 2026-07-15 12:40:23.180 10 ERROR futurist.periodics HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"etcdserver: request timed out","code":500} 2026-07-15 12:40:23.180 10 ERROR futurist.periodics 2026-07-15 12:40:23.180 10 ERROR futurist.periodics 2026-07-15 12:40:23.180 10 ERROR futurist.periodics 2026-07-15 12:40:23.180 10 ERROR futurist.periodics The above exception was the direct cause of the following exception: 2026-07-15 12:40:23.180 10 ERROR futurist.periodics 2026-07-15 12:40:23.180 10 ERROR futurist.periodics Traceback (most recent call last): 2026-07-15 12:40:23.180 10 ERROR futurist.periodics File "/var/lib/openstack/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run 2026-07-15 12:40:23.180 10 ERROR futurist.periodics work() 2026-07-15 12:40:23.180 10 ERROR futurist.periodics File "/var/lib/openstack/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ 2026-07-15 12:40:23.180 10 ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) 2026-07-15 12:40:23.180 10 ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2026-07-15 12:40:23.180 10 ERROR futurist.periodics File "/var/lib/openstack/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator 2026-07-15 12:40:23.180 10 ERROR futurist.periodics return f(*args, **kwargs) 2026-07-15 12:40:23.180 10 ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ 2026-07-15 12:40:23.180 10 ERROR futurist.periodics File "/var/lib/openstack/lib/python3.12/site-packages/staffeln/conductor/manager.py", line 260, in rotation_tasks 2026-07-15 12:40:23.180 10 ERROR futurist.periodics with lock.Lock(self.lock_mgt, constants.RETENTION) as retention: 2026-07-15 12:40:23.180 10 ERROR futurist.periodics File "/var/lib/openstack/lib/python3.12/site-packages/staffeln/common/lock.py", line 46, in __enter__ 2026-07-15 12:40:23.180 10 ERROR futurist.periodics self.acquired = self.lock.acquire(blocking=False) 2026-07-15 12:40:23.180 10 ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2026-07-15 12:40:23.180 10 ERROR futurist.periodics File "/var/lib/openstack/lib/python3.12/site-packages/sherlock/lock.py", line 192, in acquire 2026-07-15 12:40:23.180 10 ERROR futurist.periodics return self._acquire() 2026-07-15 12:40:23.180 10 ERROR futurist.periodics ^^^^^^^^^^^^^^^ 2026-07-15 12:40:23.180 10 ERROR futurist.periodics File "/var/lib/openstack/lib/python3.12/site-packages/sherlock/lock.py", line 986, in _acquire 2026-07-15 12:40:23.180 10 ERROR futurist.periodics lease = self._get_lease() 2026-07-15 12:40:23.180 10 ERROR futurist.periodics ^^^^^^^^^^^^^^^^^ 2026-07-15 12:40:23.180 10 ERROR futurist.periodics File "/var/lib/openstack/lib/python3.12/site-packages/sherlock/lock.py", line 942, in _get_lease 2026-07-15 12:40:23.180 10 ERROR futurist.periodics raise LockException("Failed to read Lock.") from exc 2026-07-15 12:40:23.180 10 ERROR futurist.periodics sherlock.lock.LockException: Failed to read Lock. 2026-07-15 12:40:23.180 10 ERROR futurist.periodics kube_config_path not provided and default location (~/.kube/config) does not exist. Using inCluster Config. This might not work. 2026-07-15 12:40:23.182 8 ERROR futurist.periodics [-] Failed to call periodic 'staffeln.conductor.manager.BackupManager.backup_engine..backup_tasks' (it runs every 3600.00 seconds): sherlock.lock.LockException: Failed to read Lock. 2026-07-15 12:40:23.182 8 ERROR futurist.periodics Traceback (most recent call last): 2026-07-15 12:40:23.182 8 ERROR futurist.periodics File "/var/lib/openstack/lib/python3.12/site-packages/sherlock/lock.py", line 934, in _get_lease 2026-07-15 12:40:23.182 8 ERROR futurist.periodics return self.client.read_namespaced_lease( 2026-07-15 12:40:23.182 8 ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2026-07-15 12:40:23.182 8 ERROR futurist.periodics File "/var/lib/openstack/lib/python3.12/site-packages/kubernetes/client/api/coordination_v1_api.py", line 1149, in read_namespaced_lease 2026-07-15 12:40:23.182 8 ERROR futurist.periodics return self.read_namespaced_lease_with_http_info(name, namespace, **kwargs) # noqa: E501 2026-07-15 12:40:23.182 8 ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2026-07-15 12:40:23.182 8 ERROR futurist.periodics File "/var/lib/openstack/lib/python3.12/site-packages/kubernetes/client/api/coordination_v1_api.py", line 1236, in read_namespaced_lease_with_http_info 2026-07-15 12:40:23.182 8 ERROR futurist.periodics return self.api_client.call_api( 2026-07-15 12:40:23.182 8 ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^ 2026-07-15 12:40:23.182 8 ERROR futurist.periodics File "/var/lib/openstack/lib/python3.12/site-packages/kubernetes/client/api_client.py", line 348, in call_api 2026-07-15 12:40:23.182 8 ERROR futurist.periodics return self.__call_api(resource_path, method, 2026-07-15 12:40:23.182 8 ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2026-07-15 12:40:23.182 8 ERROR futurist.periodics File "/var/lib/openstack/lib/python3.12/site-packages/kubernetes/client/api_client.py", line 180, in __call_api 2026-07-15 12:40:23.182 8 ERROR futurist.periodics response_data = self.request( 2026-07-15 12:40:23.182 8 ERROR futurist.periodics ^^^^^^^^^^^^^ 2026-07-15 12:40:23.182 8 ERROR futurist.periodics File "/var/lib/openstack/lib/python3.12/site-packages/kubernetes/client/api_client.py", line 373, in request 2026-07-15 12:40:23.182 8 ERROR futurist.periodics return self.rest_client.GET(url, 2026-07-15 12:40:23.182 8 ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^ 2026-07-15 12:40:23.182 8 ERROR futurist.periodics File "/var/lib/openstack/lib/python3.12/site-packages/kubernetes/client/rest.py", line 244, in GET 2026-07-15 12:40:23.182 8 ERROR futurist.periodics return self.request("GET", url, 2026-07-15 12:40:23.182 8 ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ 2026-07-15 12:40:23.182 8 ERROR futurist.periodics File "/var/lib/openstack/lib/python3.12/site-packages/kubernetes/client/rest.py", line 238, in request 2026-07-15 12:40:23.182 8 ERROR futurist.periodics raise ApiException(http_resp=r) 2026-07-15 12:40:23.182 8 ERROR futurist.periodics kubernetes.client.exceptions.ApiException: (500) 2026-07-15 12:40:23.182 8 ERROR futurist.periodics Reason: Internal Server Error 2026-07-15 12:40:23.182 8 ERROR futurist.periodics HTTP response headers: HTTPHeaderDict({'Audit-Id': '7848213d-02c8-41a0-b864-64a6e2a51de8', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Kubernetes-Pf-Flowschema-Uid': 'cdb3d981-cd09-4835-9e88-cbddf97a2a40', 'X-Kubernetes-Pf-Prioritylevel-Uid': 'c6b405d9-edd8-4f83-8e8d-86e05012eeb7', 'Date': 'Wed, 15 Jul 2026 12:40:23 GMT', 'Content-Length': '122'}) 2026-07-15 12:40:23.182 8 ERROR futurist.periodics HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"etcdserver: request timed out","code":500} 2026-07-15 12:40:23.182 8 ERROR futurist.periodics 2026-07-15 12:40:23.182 8 ERROR futurist.periodics 2026-07-15 12:40:23.182 8 ERROR futurist.periodics 2026-07-15 12:40:23.182 8 ERROR futurist.periodics The above exception was the direct cause of the following exception: 2026-07-15 12:40:23.182 8 ERROR futurist.periodics 2026-07-15 12:40:23.182 8 ERROR futurist.periodics Traceback (most recent call last): 2026-07-15 12:40:23.182 8 ERROR futurist.periodics File "/var/lib/openstack/lib/python3.12/site-packages/futurist/periodics.py", line 290, in run 2026-07-15 12:40:23.182 8 ERROR futurist.periodics work() 2026-07-15 12:40:23.182 8 ERROR futurist.periodics File "/var/lib/openstack/lib/python3.12/site-packages/futurist/periodics.py", line 64, in __call__ 2026-07-15 12:40:23.182 8 ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) 2026-07-15 12:40:23.182 8 ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2026-07-15 12:40:23.182 8 ERROR futurist.periodics File "/var/lib/openstack/lib/python3.12/site-packages/futurist/periodics.py", line 178, in decorator 2026-07-15 12:40:23.182 8 ERROR futurist.periodics return f(*args, **kwargs) 2026-07-15 12:40:23.182 8 ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ 2026-07-15 12:40:23.182 8 ERROR futurist.periodics File "/var/lib/openstack/lib/python3.12/site-packages/staffeln/conductor/manager.py", line 174, in backup_tasks 2026-07-15 12:40:23.182 8 ERROR futurist.periodics with lock.Lock(self.lock_mgt, constants.PULLER) as puller: 2026-07-15 12:40:23.182 8 ERROR futurist.periodics File "/var/lib/openstack/lib/python3.12/site-packages/staffeln/common/lock.py", line 46, in __enter__ 2026-07-15 12:40:23.182 8 ERROR futurist.periodics self.acquired = self.lock.acquire(blocking=False) 2026-07-15 12:40:23.182 8 ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2026-07-15 12:40:23.182 8 ERROR futurist.periodics File "/var/lib/openstack/lib/python3.12/site-packages/sherlock/lock.py", line 192, in acquire 2026-07-15 12:40:23.182 8 ERROR futurist.periodics return self._acquire() 2026-07-15 12:40:23.182 8 ERROR futurist.periodics ^^^^^^^^^^^^^^^ 2026-07-15 12:40:23.182 8 ERROR futurist.periodics File "/var/lib/openstack/lib/python3.12/site-packages/sherlock/lock.py", line 986, in _acquire 2026-07-15 12:40:23.182 8 ERROR futurist.periodics lease = self._get_lease() 2026-07-15 12:40:23.182 8 ERROR futurist.periodics ^^^^^^^^^^^^^^^^^ 2026-07-15 12:40:23.182 8 ERROR futurist.periodics File "/var/lib/openstack/lib/python3.12/site-packages/sherlock/lock.py", line 942, in _get_lease 2026-07-15 12:40:23.182 8 ERROR futurist.periodics raise LockException("Failed to read Lock.") from exc 2026-07-15 12:40:23.182 8 ERROR futurist.periodics sherlock.lock.LockException: Failed to read Lock. 2026-07-15 12:40:23.182 8 ERROR futurist.periodics